Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Avoid revalidating the entire remaining UTF-8 source for every lexer character

Closed Beginner friendly
  • Is something wrong?
#935 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

A pull request for this has already been merged.

  • #965 by @hiyouwhatsapp — merged

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
83/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
rust
Domain
compilers

Research direction

Start with front/lexer/src/cursor.rs at the advance and peek methods, then read the existing UTF-8 cursor tests added for #392. Check how cursor offsets and diagnostic columns are tracked, and run the lexer tests before and after focused long-input coverage. Done means per-character operations no longer validate the full remaining suffix while UTF-8 handling and spans remain correct; record scaling measurements without a timing assertion.

Written by the indexing model from the issue text.

Description

good first issue performance

Problem

Lexer already stores its input as &str, but advance and peek repeatedly call std::str::from_utf8 on the entire remaining suffix. A long comment performs this linear suffix validation for each character, giving quadratic scanning work even though only the next character is needed.

Starting points: front/lexer/src/cursor.rs:25 and front/lexer/src/cursor.rs:47. Preserve the UTF-8 byte-offset and diagnostic-column contract; this is a cursor hot-path fix, not a lexer redesign. #392 added UTF-8 correctness tests but does not cover repeated suffix validation.

Reproduction

Audited canonical master d59bdab376bdc04edacdc8029b256cb8d55b5adf on Linux amd64 with a checkout-built Wave 0.0.1-alpha-dev, Rust 1.89.0 and LLVM 21.1.8.

With the debug compiler, checked files containing /* + N ASCII a characters + */\nfun main() {}\n, three executions per size:

Comment bytes Elapsed seconds
30,000 0.0707, 0.0633, 0.0665
60,000 0.2024, 0.2087, 0.1956
120,000 0.7333, 0.7140, 0.7221

All checks succeed. Doubling the long comment approaches four times the scanning cost. These are debug-build observations, not release benchmark claims. The repeated full-suffix validation is also directly visible in the source.

Reproduce with target/debug/wavec check long-comment.wave; use time.perf_counter() around repeated subprocess invocations and keep the number of tokens constant.

Scope

Decode/look ahead within the already-valid string without scanning its full suffix again. Difficulty: easy; the existing UTF-8 cursor tests provide a starting point.

Completion criteria

  • Preserve multibyte identifiers, comments, strings, CR/LF tracking and diagnostic spans.
  • Remove full-suffix UTF-8 validation from per-character cursor operations.
  • Add focused long-input correctness coverage and record scaling measurements; avoid a flaky wall-clock assertion in unit tests.
Dominant language
Rust
Stars
56
Forks
21
Avg merge
4h 41m
Merged PRs (30d)
58

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from wavefnd/Wave

All issues in wavefnd/Wave

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.