[BUG] Tokenizer mishandles triple-quoted strings — keywords replaced inside them
还没有人认领这个 Issue。
评估
调研方向
The bug lives in _tokenize_preserving_strings in maithili_dsl/transpiler/transpile.py, which only matches single-line quoted strings and # comments. Reproduce by transpiling x = """क " में " ख""" and confirm में is replaced inside the literal. Extend the tokenizer regex to match triple-quoted strings ("""...""" and '''...''', spanning newlines) before the single-quote alternatives, then add tests for keywords inside triple quotes, inner quotes, and a multi-line docstring. Done when those cases pass unchanged and existing transpiler tests still pass.
由索引模型根据 Issue 内容生成。
描述
File: maithili_dsl/transpiler/transpile.py (_tokenize_preserving_strings)
The tokenizer only knows single-line "..." / '...' strings and # comments. Triple-quoted strings are parsed as pairs of empty strings, so when the content itself contains a quote, part of the string body is treated as code and keywords inside it get replaced:
Input: x = """क " में " ख"""
Output: x = """क " in " ख""" # में replaced inside a string literal
(Verified against v0.3.0. Simple triple-quoted strings without inner quotes survive only by accident.)
Fix: extend the tokenizer regex to match """...""" and '''...''' (with re.DOTALL-style content) before the single-quote alternatives. Consider f-string prefixes (f"...") while in there.
Tests: triple-quoted string with keywords, with inner quotes, multi-line docstring on a कार्य.
- 主要语言
- Python
- 星标
- 1
- 派生
- 4
- 平均合并
- 12 天 21 小时
- 30 天内合并 PR
- 1
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
alphacrack/python-maithili-dsl 的其他 Issue
-
help wanted infra P2 testing
难度 2/5 1-3 小时 新手友好度 80/100
-
documentation enhancement good first issue help wanted P3
难度 2/5 1-3 小时 新手友好度 75/100
-
documentation good first issue P3
难度 1/5 1 小时以内 新手友好度 88/100
-
area:linter enhancement good first issue P3 translation
难度 2/5 半天 新手友好度 66/100
-
area:cli enhancement good first issue P3
难度 2/5 1-3 小时 新手友好度 82/100
查看 alphacrack/python-maithili-dsl 的全部 Issue
相似的 Issue
-
documentation good first issue
难度 2/5 1-3 小时 新手友好度 88/100
JheysonJhordyHuarauyaRiquelme/sprocguard#18 ·
维护者通常 1 天内回复
-
enhancement good first issue help wanted
难度 2/5 1-3 小时 新手友好度 80/100
dlq/synthpopcan#59 ·
维护者通常 2 天内回复
-
难度 2/5 1-3 小时 新手友好度 72/100
mdn/translated-content#40220 ·
维护者通常 4 天内回复
-
enhancement
难度 2/5 1-3 小时 新手友好度 84/100
ETSInitiative/PETSIRD#243 ·
维护者通常 1 天内回复
-
deferred p2
难度 2/5 1-3 小时 新手友好度 70/100
cuadrao-ai/argus#2160 ·
维护者通常 1 天内回复