lang-core: a raw newline or tab in a double-quoted string turns off escape decoding

Open Beginner friendly
  • Is something wrong?
#1,445 2 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
78/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
typescript
Domain
backend-api-design, testing

Research direction

Start in packages/lang-core/src/parser/lexer.ts around the double-quoted string decoding near line 207, and compare it with the adjacent single-quote branch. Reproduce the cases using createParser(schema).parse(text) and createStreamingParser(schema) with the schema in spec/fixtures/1.0/library.json. Done when raw newlines and tabs preserve escape decoding, incomplete streaming escapes do not disrupt already complete escapes, and tests cover these cases.

Written by the indexing model from the issue text.

Description

Describe the bug
A double-quoted string that contains a raw newline or tab loses all of its escape decoding. \", \\, \t, \n and \uXXXX are kept as literal backslash text, so the user sees \"hi\" instead of "hi". No error is reported. The same thing happens briefly while streaming, whenever a chunk ends partway through a \u escape.

Spec 1.0 (spec/language.md, §1.3) says "Double-quoted strings use JSON escapes (\n, \t, \", \\, \uXXXX)", and §1.4 says "strings may hold raw newlines", so both together should work. Models often write multi-line Text/TextContent with quotes in it, so this shows up in normal responses.

To Reproduce
Parse cases 1 to 3 with createParser(schema).parse(text), using the schema from spec/fixtures/1.0/library.json:

  1. Escapes on one line (works):
root = Stack([a])
a = Text("He said \"hi\"")

→ He said "hi"

  1. The same string with a raw newline:
root = Stack([a])
a = Text("He said
\"hi\"")

→ He said⏎\"hi\" (backslashes kept)

  1. A raw tab (between Tab and here):
root = Stack([a])
a = Text("Tab	here \u0041")

→ Tab⇥here \u0041

  1. Streaming, with the chunk ending inside a \u escape. Push this to createStreamingParser(schema) without completing the stream:
root = Stack([a])
a = Text("say \"hi\" \u00

→ say \"hi\" \u00 (backslashes kept on the already-complete \"hi\")

Expected behavior

  • Case 2 gives He said⏎"hi".
  • Case 3 gives Tab⇥here A.
  • Case 4 gives say "hi" while the \u escape is still incomplete, then the decoded character once the escape arrives.

Actual behavior

  • Cases 2 and 3: the escapes are not decoded and errors is empty.
  • Case 4: the whole string shows its raw escapes until the next chunk completes the \u escape, then it renders correctly. So the text flickers with backslashes mid-stream.

The cause is in packages/lang-core/src/parser/lexer.ts (around line 207): the string is decoded with JSON.parse, which rejects raw control characters and incomplete \u escapes. The catch then returns the raw text with only the quotes stripped, so every escape in the string is lost, not just the one that failed.

Environment

  • Package: @openuidev/lang-core 1.0.1 (main @ 8d9c4c3b)
  • Node 25.1.0

Additional context
Happy to send a fix with tests if this direction sounds right: decode double-quoted escapes directly (like the single-quote branch next to it already does) instead of relying on JSON.parse, and add a grammar conformance fixture for multi-line strings with escapes.

Dominant language
TypeScript
Stars
10k
Forks
698
Avg merge
1d 6h
Merged PRs (30d)
148

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from thesysdev/openui

All issues in thesysdev/openui

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.