Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

C++: a 16-character raw-string delimiter (the standard maximum) fails to parse — capacity guard is checked before the '(' terminator

Aperta Adatta ai principianti
  • Qualcosa non va?
#371 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
1/5
Tempo stimato
Meno di un'ora
Idoneità per principianti
90/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
cpp
Ambito
compilers

Direzione di ricerca

Inizia in src/scanner.c, in scan_raw_string_delimiter, e analizza il ciclo del delimitatore di apertura e il relativo controllo della capacità. Verifica che il parser accetti un delimitatore di 16 caratteri continuando a rifiutare i delimitatori più lunghi, quindi esegui i controlli disponibili del parser e aggiungi o esegui un caso di regressione del corpus, se supportato; la scansione differenziale esistente fornisce il comportamento previsto.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Summary

A raw-string delimiter of exactly 16 characters — the maximum [lex.string] allows — fails to parse. The whole translation unit after it collapses into ERROR nodes. 15 characters is fine, so the one length that is legal-but-rejected is exactly the spec's limit.

The cause is a check ordering in scan_raw_string_delimiter, and the fix is a one-line reorder. I have built and tested it; details and a differential sweep below.

Repro

// compiles cleanly with `g++ -std=c++17 -fsyntax-only`
const char* kTemplate = R"FFFFFFFFFFFFFFFF(
struct Ignored { int v; };
)FFFFFFFFFFFFFFFF";

int after_the_raw_string(int x) {
  return x + 1;
}

FFFFFFFFFFFFFFFF is 16 chars.

Parsed with the C API against this repo at HEAD (src/parser.c + src/scanner.c, linked with the tree-sitter runtime — no editor or downstream tool involved):

delim=14  hasError=false  function_definition=1  ERROR=0
delim=15  hasError=false  function_definition=1  ERROR=0
delim=16  hasError=true   function_definition=0  ERROR=3     <-- legal, rejected
delim=17  hasError=true   function_definition=0  ERROR=3
delim=18  hasError=true   function_definition=0  ERROR=3

after_the_raw_string disappears entirely at 16.

Cause

src/scanner.c, opening-delimiter loop:

for (;;) {
    if (scanner->delimiter_length >= MAX_DELIMITER_LENGTH || lexer->eof(lexer) || lexer->lookahead == '\\' ||
        iswspace(lexer->lookahead)) {
        return false;                              // (1) capacity guard
    }
    if (lexer->lookahead == '(') {
        return scanner->delimiter_length > 0;      // (2) legal terminator
    }
    scanner->delimiter[scanner->delimiter_length++] = lexer->lookahead;
    advance(lexer);
}

With a 16-char delimiter, the 16 d-chars are consumed and delimiter_length == 16. On the next iteration lookahead is ( — the legal terminator, check (2) — but check (1) runs first, sees the buffer is full, and returns false. The delimiter is never accepted even though nothing is wrong with it: the buffer is full, not overflowing.

15 works only because the loop still has a free slot when it reaches (.

Fix

Test the terminator before the capacity guard. A full buffer followed by ( is a complete, legal delimiter; a full buffer followed by another d-char is the genuine overflow.

     for (;;) {
-        if (scanner->delimiter_length >= MAX_DELIMITER_LENGTH || lexer->eof(lexer) || lexer->lookahead == '\\' ||
-            iswspace(lexer->lookahead)) {
-            return false;
-        }
         if (lexer->lookahead == '(') {
             // Rather than create a token for an empty delimiter, we fail and
             // let the grammar fall back to a delimiter-less rule.
             return scanner->delimiter_length > 0;
         }
+        if (scanner->delimiter_length >= MAX_DELIMITER_LENGTH || lexer->eof(lexer) || lexer->lookahead == '\\' ||
+            iswspace(lexer->lookahead)) {
+            return false;
+        }
         scanner->delimiter[scanner->delimiter_length++] = lexer->lookahead;
         advance(lexer);
     }

No bounds change: delimiter[] is still only written when delimiter_length < MAX_DELIMITER_LENGTH, so the array cannot overflow.

Verification

Built stock and patched scanners from the same sources and swept delimiter lengths 0-18, comparing function_definition counts:

length stock patched
0-15 1 1 identical
16 0 1 fixed
17, 18 0 0 identical — still rejected

The patch changes behaviour at exactly one length, the one that is legal today and rejected. 17+ stays rejected, so the spec limit is preserved, and length 0 (the empty-delimiter fallback the comment describes) is unchanged.

What I did not do: I could not run the corpus suite — npx tree-sitter-cli did not finish downloading in my environment — so this is verified by differential parsing rather than by tree-sitter test. Worth running before merging.

Happy to open a PR with the patch and a corpus case if useful.

Why it matters downstream

Files most likely to carry long descriptive delimiters — code generators, template/scaffold files, embedded shader or SQL blobs — are exactly where FILE_TEMPLATE_V1, SHADER_SOURCE_V2 or CMAKE_TEMPLATE_1 read as natural, and all are 16. When it happens the file yields no symbols at all, with no diagnostic.

Distinct from #245 (error-recovery synthesising a raw_string_delimiter, fixed in 3d8d510a) — nothing here is in error recovery; the input is well-formed C++.

Environment: this repo at HEAD, tree-sitter runtime from tree-sitter/tree-sitter HEAD, gcc 13.3.0, Ubuntu 24.04.

Lingua principale
JavaScript
Stelle
463
Fork
190
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di tree-sitter/tree-sitter-cpp

Tutte le issue di tree-sitter/tree-sitter-cpp

Issue simili

Altre issue su JavaScript

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.