is_ident_start reads the wrong byte for a hyphen before a backslash

Open Beginner friendly
#440 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
88/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Quiet
Tech stack
css, rust
Domain
compilers

Research direction

Start at src/tokenizer.rs:895 and run the Rust reproducer against the default features. Check the identifier-start decision for a hyphen before a backslash-newline sequence, then verify that the three reported inputs produce the expected token types and values.

Written by the indexing model from the issue text.

Description

When a - is followed by a backslash that begins an escape, the check that decides whether an identifier starts here reads the wrong byte. A backslash right before a newline is an invalid escape and cannot start an identifier, so -\<newline> is mis-tokenized.

Reproducer

cssparser 0.37.0, default features.

use cssparser::{Parser, ParserInput};

fn main() {
    let mut pi = ParserInput::new("1-\\\n");
    let mut p = Parser::new(&mut pi);
    let a = p.next().unwrap().clone();
    // cssparser: Dimension { value: 1.0, int_value: Some(1), unit: "-" }
    println!("{:?}", a);
}

Observed vs expected

Observed for 1-\<newline>: the tokens are Dimension { value: 1.0, unit: "-" } then Delim('\\'). For -\<newline> the first token is Ident("-"). For #-\<newline> the first token is IDHash("-").

Expected per CSS Syntax Level 3, matched by tinycss2: 1-\<newline> gives Number(1), Delim('-'), Delim('\\'). -\<newline> gives Delim('-') then Delim('\\'). #-\<newline> gives a hash token Hash("-") with the unrestricted type flag, not an id hash.

Root cause

src/tokenizer.rs:895. The would-start-an-identifier test reads the byte after the backslash to decide if -\ begins an identifier. A backslash directly before a newline is not a valid escape, but the offset used skips the backslash and reads the newline as though it were escaped.

Scope

A hyphen before an invalid escape produces a unit, an identifier, or an id hash that the input never held. A consumer that keys on token type, such as a linter or a minifier, then acts on tokens the source did not contain.

Dominant language
Rust
Stars
869
Forks
152
Avg merge
15h 8m
Merged PRs (30d)
12

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from servo/rust-cssparser

All issues in servo/rust-cssparser

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.