[p5.js 2.0+ Bug Report]: splitTokens() returns [] for a caret delimiter and throws for a backslash

Open Beginner friendly
#9,126 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
76/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
javascript
Domain
frontend

Research direction

Start at the splitTokens implementation and its existing unit tests. Reproduce the caret and backslash cases, then check coverage for bracket, array-delimiter, and whitespace behavior. Done means literal special-character delimiters produce the expected tokens without breaking the existing cases.

Written by the indexing model from the issue text.

Description

Most appropriate sub-area of p5.js?

Utilities

p5.js version

2.x main (4b096e2)

Actual vs expected behavior

splitTokens builds a regex character class from the delimiter characters with only [ and ] special-cased (the 2015 fix for #1004), so other regex metacharacters corrupt the class:

splitTokens('a^b', '^');   // actual []                       expected ['a', 'b']
                           // the ^ negates the class, so every character becomes a delimiter
splitTokens('a\b', '\'); // actual SyntaxError: Invalid regular expression: /[\]/g
                           // expected ['a', 'b']

A - between delimiter characters also silently forms a range instead of matching the literal characters.

Steps to reproduce

Outputs above are from executed runs against current main.

Note

I am aware splitTokens was deprecated in 2.0 (#7624), so feel free to close this as wontfix if the function is not receiving fixes anymore. If it is, I have a small fix ready (escape \, ], ^, - when building the class) with unit tests for both broken cases plus regression coverage for the existing bracket, array-delimiter and whitespace behaviors; the two new tests fail on current main and pass with the change. Will open the PR once approved.

Dominant language
JavaScript
Stars
24k
Forks
3.8k
Avg merge
3d 9h
Merged PRs (30d)
26

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from processing/p5.js

All issues in processing/p5.js

Similar issues

More JavaScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.