re: character classes are parsed as regular expressions

Open
#914 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
76/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
javascript, python
Domain
compilers

Research direction

Start in transcrypt/modules/re/translate.py, reading shift() and countCaptureGroups() to understand how the tokenizer handles character classes. Compare the listed CPython and Transcrypt examples, including classes containing parentheses and a leading literal ]. Done means capture-group numbering matches CPython and the []x], and invalid [] cases behave as described.

Written by the indexing model from the issue text.

Description

re: character classes are parsed as regular expressions

transcrypt/modules/re/translate.py turns a Python regex into a token list. The tokenizer has no notion of character classes. Every character inside [...] becomes a token of its own. That causes two defects.

Parentheses inside a character class are counted as capture groups

countCaptureGroups() counts every token named (. A ( inside a character class is counted too. All groups behind such a class shift by one.

import re
r = re.compile(r"(?P<a>[()])(?P<b>x)")
CPython Transcrypt
r.groups 2 3
r.groupindex {'a': 1, 'b': 2} {'a': 1, 'b': 3}

r.match("(x").group("b") returns x under CPython and None under Transcrypt. The generated JavaScript is correct, only the group numbers are wrong.

A literal ] at the start of a character class is not translated

Python treats a ] right behind [ or [^ as a literal. JavaScript ends the class at it. The translator passes the class through unchanged.

re.compile(r"[]x]") matches ] and x under CPython. Transcrypt emits []x] and the RegExp constructor raises Invalid regular expression: /[]x]/u: Lone quantifier brackets. Without the u flag the pattern compiles but means an empty class followed by x], so it never matches.

The reverse case is silent. re.compile("[]") raises unterminated character set under CPython. Transcrypt accepts it and returns a pattern that never matches.

Fix

Both defects go away when shift() consumes a character class as a single token. The class content is then never parsed as a regular expression, and the literal ] can be escaped while the class is read.

Dominant language
Python
Stars
2.9k
Forks
218
No merged PRs in 30d

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from TranscryptOrg/Transcrypt

All issues in TranscryptOrg/Transcrypt

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.