Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Difference in SIMD and non-SIMD process_token tokens

Open
  • Is something wrong?
#797 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
Half a day
Newbie friendliness
66/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
rust
Domain
performance

Research direction

Start at the step function in html5ever/src/tokenizer/mod.rs, comparing the small_char_set! used by pop_except_from (via BufferQueue) with the trailing-byte loop in data_state_simd_fast_path, which skips stopping at \n. Decide which path matches the HTML tokenizer spec, then align the two so process_token receives identical tokens with SIMD forced off (patch is_x86_feature_detected! to false). Done when both paths produce the same token stream; verify with cargo test and the tokenizer test suite.

Written by the indexing model from the issue text.

Description

Summary

SIMD and non-SIMD paths produce different process_token calls.
As the maintainers of html5ever do you think this inconsistency is a bug or normal behaviour?

How it was discovered

The lychee test fails on riscv machine:
https://github.com/lycheeverse/lychee/issues/2263

Inside lychee-lib/src/extract/html/html5ever.rs process_token recieves
different tokens (example in the issue is given). This incosistency breaks tests.

Reason behind it

The step function inside html5ever/src/tokenizer/mod.rs contains following lines:

// ...
let set = small_char_set!('\r' '\0' '&' '<' '\n' );

#[cfg(any(target_arch = "x86", target_arch = "x86_64", target_arch = "aarch64"))]
// ...
let result = unsafe { self.data_state_simd_fast_path(&mut front_buffer) };
// ...

#[cfg(not(any(
    target_arch = "x86",
    target_arch = "x86_64",
    target_arch = "aarch64"
)))]
let set_result = self.pop_except_from(input, set);
// ...
SIMD path

Note how data_state_simd_fast_path processes remaining bytes:

// ...

// Process any remaining bytes (less than STRIDE)
while let Some(c) = input.as_bytes().get(i) {
    if matches!(*c, b'<' | b'&' | b'\r' | b'\0') {
        break;
    }
    if *c == b'\n' {
        n_newlines += 1;
    }
    i += 1;
}

// ...
non-SIMD path

Note how pop_except_from processes bytes:

let d = input.pop_except_from(set);

input is BufferQueue and set is small_char_set!('\r' '\0' '&' '<' '\n' ). So countrary to the SIMD part this pop stops after \n.

How to reproduce

I was able to reproduce this issue on x86_64 machine by turning SIMD off.

  1. git clone https://github.com/lycheeverse/lychee && cd lychee
  2. cargo vendor # don't forget to change .cargo/config.toml
  3. patch html5ever
--- a/vendor/html5ever/src/tokenizer/mod.rs
+++ b/vendor/html5ever/src/tokenizer/mod.rs
@@ -1921,7 +1921,7 @@ impl<Sink: TokenSink> Tokenizer<Sink> {
     fn is_supported_simd_feature_detected() -> bool {
         #[cfg(any(target_arch = "x86", target_arch = "x86_64"))]
         {
-            is_x86_feature_detected!("sse2")
+            false
         }

         #[cfg(target_arch = "aarch64")]
  1. Fix checksums
sed -i -e 's/"files":{[^}]*}/"files":{}/' \
     ./vendor/html5ever/.cargo-checksum.json
  1. cargo test --frozen -p lychee-lib --lib html5ever::tests::test_include_verbatim
Dominant language
Rust
Stars
2.6k
Forks
291
Avg merge
8h 21m
Merged PRs (30d)
3

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from servo/html5ever

All issues in servo/html5ever

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.