Avoid revalidating the entire remaining UTF-8 source for every lexer character
维护者通常 1 天内回复
已经有一个关联 PR 被合并了。
- #965 来自 @hiyouwhatsapp —— 已合并
评估
调研方向
从 front/lexer/src/cursor.rs 中的 advance 和 peek 方法开始，然后阅读为 #392 添加的现有 UTF-8 游标测试。检查游标偏移量和诊断列的跟踪方式，并在添加针对长输入的测试覆盖前后运行 lexer 测试。完成标准是：逐字符操作不再验证整个剩余后缀，同时 UTF-8 处理和 spans 仍然正确；记录扩展性测量结果，不添加时间断言。
由索引模型根据 Issue 内容生成。
描述
Problem
Lexer already stores its input as &str, but advance and peek repeatedly call std::str::from_utf8 on the entire remaining suffix. A long comment performs this linear suffix validation for each character, giving quadratic scanning work even though only the next character is needed.
Starting points: front/lexer/src/cursor.rs:25 and front/lexer/src/cursor.rs:47. Preserve the UTF-8 byte-offset and diagnostic-column contract; this is a cursor hot-path fix, not a lexer redesign. #392 added UTF-8 correctness tests but does not cover repeated suffix validation.
Reproduction
Audited canonical master d59bdab376bdc04edacdc8029b256cb8d55b5adf on Linux amd64 with a checkout-built Wave 0.0.1-alpha-dev, Rust 1.89.0 and LLVM 21.1.8.
With the debug compiler, checked files containing /* + N ASCII a characters + */\nfun main() {}\n, three executions per size:
| Comment bytes | Elapsed seconds |
|---|---|
| 30,000 | 0.0707, 0.0633, 0.0665 |
| 60,000 | 0.2024, 0.2087, 0.1956 |
| 120,000 | 0.7333, 0.7140, 0.7221 |
All checks succeed. Doubling the long comment approaches four times the scanning cost. These are debug-build observations, not release benchmark claims. The repeated full-suffix validation is also directly visible in the source.
Reproduce with target/debug/wavec check long-comment.wave; use time.perf_counter() around repeated subprocess invocations and keep the number of tokens constant.
Scope
Decode/look ahead within the already-valid string without scanning its full suffix again. Difficulty: easy; the existing UTF-8 cursor tests provide a starting point.
Completion criteria
- Preserve multibyte identifiers, comments, strings, CR/LF tracking and diagnostic spans.
- Remove full-suffix UTF-8 validation from per-character cursor operations.
- Add focused long-input correctness coverage and record scaling measurements; avoid a flaky wall-clock assertion in unit tests.
- 主要语言
- Rust
- 星标
- 56
- 派生
- 21
- 平均合并
- 5 小时 2 分钟
- 30 天内合并 PR
- 56
环境准备
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
wavefnd/Wave 的其他 Issue
-
bug good first issue
难度 2/5 1-3 小时 新手友好度 78/100
维护者通常 1 天内回复
-
Distinguish Rust character literals from lifetimes in the spacing helper可能已有人在做 @hiyouwhatsapp 于 4 天前认领。 未关闭bug good first issue
难度 2/5 1-3 小时 新手友好度 78/100
维护者通常 1 天内回复
-
Preserve truncation errors when bounded C-string copy uses the same buffer可能已有人在做 关联的 PR 仍在进行中或已合并。 未关闭bug good first issue
难度 2/5 1-3 小时 新手友好度 82/100
维护者通常 1 天内回复
-
enhancement good first issue needs testing
难度 2/5 1-3 小时 新手友好度 78/100
维护者通常 1 天内回复
-
Add checked floating-point byte readers and writers可能已有人在做 @LunaStev 于 1 天前认领。 未关闭enhancement good first issue needs testing
难度 2/5 1-3 小时 新手友好度 78/100
维护者通常 1 天内回复
相似的 Issue
-
难度 1/5 1 小时以内 新手友好度 68/100
ilya-zlobintsev/LACT#1235 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 78/100
FalkorDB/FalkorDB#3482 · 1 条评论 ·
维护者通常 8 天内回复
-
难度 2/5 1-3 小时 新手友好度 72/100
维护者通常 1 天内回复
-
checkpoint/verify/rebuild_indexes on a thread holding a WriteTransaction deadlock instead of INT-001未关闭bug known-issue
难度 2/5 1-3 小时 新手友好度 74/100
project-minigraf/minigraf#734 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 70/100