[Bug] JavaScript/TypeScript tokenizer: regex literals with `/` inside `[...]` or with the `v` flag are not recognized

Open Beginner friendly
  • Is something wrong?
#5,575 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
79/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
javascript, typescript
Domain
developer-experience, frontend

Research direction

Start in src/languages/definitions/typescript/typescript.ts and read the root regex look-ahead and regexp state rules called out in the issue. Reproduce the three examples in the Monaco playground and check how their tokens differ from the expected behavior. Done means all three literals tokenize as regexes, with v treated as a flag and the following code tokenized normally.

Written by the indexing model from the issue text.

Description

Reproducible in vscode.dev or in VS Code Desktop?
  • Not reproducible in vscode.dev or VS Code Desktop
Reproducible in the monaco editor playground?
Monaco Editor Playground Link

https://microsoft.github.io/monaco-editor/playground.html?source=v0.57.0#XQAAAAIvAQAAAAAAAABBqIlHNwwjnNQJeFChrPHvSHns3TL9eVXpwi4q_FYRsu385WtaQb2h2hSkdYWjCVbBOYYKrFJew0sDDmv-INVEx0KePLiJRxD_S9cC7PyOeQf9hySbPHztOMuUaYYu_IHNPTo2KVrHG5ymLAGrHRVruQBLu-vbxx1Z2aC_YnL1737BMMTrCsOOwBodB3KXxw0TUB7GTAZI4UaAW6SOs9Vw9IWJDrM3Ow0F7KwxBSiowcXdxcWW5_1DboJgT781vVA_TUUYLpK0sd8NJRVLOdMAWMefOuhE6ZvBJiRIf8M32WOyxtDNkLMqH_RZ__G4Uwc

Monaco Editor Playground Code
const value = [
	"if (/[/]'/.test(s)) f();",
	"if (/[a/b]/.test(s)) f();",
	"x = /a\\/b/v.test(s);",
].join("\n");

monaco.editor.create(document.getElementById("container"), {
	value,
	language: "javascript",
	automaticLayout: true,
});
Reproduction Steps

Open the link (Monaco 0.57.0) and click Load. Same result with main 5359a33.

Actual (Problematic) Behavior
  • Line 1: the regex ends at the / inside [/], and the ' then opens a string, so '/.test(s)) f(); is colored as string.invalid.
  • Line 2: the leading / is a delimiter, then /b]/ is read as a regex.
  • Line 3: no regexp tokens at all; \ becomes invalid.

Tokens from MonarchTokenizer with the built-in typescript definition:

if (/[/]'/.test(s)) f(); -> 0:keyword.ts 2: 3:delimiter.parenthesis.ts 4:regexp.ts 5:regexp.escape.control.ts 6:regexp.ts 7:delimiter.square.ts 8:string.invalid.ts
if (/[a/b]/.test(s)) f(); -> 0:keyword.ts 2: 3:delimiter.parenthesis.ts 4:delimiter.ts 5:delimiter.square.ts 6:identifier.ts 7:regexp.ts 9:regexp.escape.control.ts 10:regexp.ts 11:delimiter.ts 12:identifier.ts 16:delimiter.parenthesis.ts 17:identifier.ts 18:delimiter.parenthesis.ts 20: 21:identifier.ts 22:delimiter.parenthesis.ts 24:delimiter.ts
x = /a\/b/v.test(s); -> 0:identifier.ts 1: 2:delimiter.ts 3: 4:delimiter.ts 5:identifier.ts 6:invalid.ts 7:delimiter.ts 8:identifier.ts 9:delimiter.ts 10:identifier.ts 11:delimiter.ts 12:identifier.ts 16:delimiter.parenthesis.ts 17:identifier.ts 18:delimiter.parenthesis.ts 19:delimiter.ts
Expected Behavior

Each /.../ literal is a regexp token (with the v flag as keyword.other), and the rest of each line is ordinary code. An unescaped / is allowed inside a class and v is a valid flag (https://tc39.es/ecma262/#sec-literals-regular-expression-literals).

Additional Context

In src/languages/definitions/typescript/typescript.ts, the root look-ahead /\/(?=([^\\\/]|\\.)+\/([dgimsuy]*)(\s*)(\.|;|,|\)|\]|\}|$))/ and the regexp state's class rule (\[)(\^?)(?=(?:[^\]\\\/]|\\.)+) both exclude / inside [...], and [dgimsuy]* (in the look-ahead and in the closing rule (\/)([dgimsuy]*)) lacks v. #5552 touches the same look-ahead for a different case.

Credit: https://ucsbplse.github.io/

Dominant language
JavaScript
Stars
46.9k
Forks
4.2k
Avg merge
2d 5h
Merged PRs (30d)
28

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from microsoft/monaco-editor

All issues in microsoft/monaco-editor

Similar issues

More JavaScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.