Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

C++: a 16-character raw-string delimiter (the standard maximum) fails to parse — capacity guard is checked before the '(' terminator

Abierto Apto para principiantes
  • ¿Algo no está bien?
#371 1 comentario 0 reacciones 0 asignados Ver en GitHub

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
1/5
Tiempo estimado
Menos de una hora
Aptitud para principiantes
90/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
cpp
Área
compilers

Línea de trabajo

Empieza en src/scanner.c, en scan_raw_string_delimiter, e inspecciona el bucle del delimitador de apertura y su comprobación de capacidad. Verifica que el parser acepta un delimitador de 16 caracteres y sigue rechazando los delimitadores más largos; después, ejecuta las comprobaciones disponibles del parser y añade o ejecuta un caso de regresión del corpus si es compatible; el barrido diferencial existente proporciona el comportamiento esperado.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Summary

A raw-string delimiter of exactly 16 characters — the maximum [lex.string] allows — fails to parse. The whole translation unit after it collapses into ERROR nodes. 15 characters is fine, so the one length that is legal-but-rejected is exactly the spec's limit.

The cause is a check ordering in scan_raw_string_delimiter, and the fix is a one-line reorder. I have built and tested it; details and a differential sweep below.

Repro

// compiles cleanly with `g++ -std=c++17 -fsyntax-only`
const char* kTemplate = R"FFFFFFFFFFFFFFFF(
struct Ignored { int v; };
)FFFFFFFFFFFFFFFF";

int after_the_raw_string(int x) {
  return x + 1;
}

FFFFFFFFFFFFFFFF is 16 chars.

Parsed with the C API against this repo at HEAD (src/parser.c + src/scanner.c, linked with the tree-sitter runtime — no editor or downstream tool involved):

delim=14  hasError=false  function_definition=1  ERROR=0
delim=15  hasError=false  function_definition=1  ERROR=0
delim=16  hasError=true   function_definition=0  ERROR=3     <-- legal, rejected
delim=17  hasError=true   function_definition=0  ERROR=3
delim=18  hasError=true   function_definition=0  ERROR=3

after_the_raw_string disappears entirely at 16.

Cause

src/scanner.c, opening-delimiter loop:

for (;;) {
    if (scanner->delimiter_length >= MAX_DELIMITER_LENGTH || lexer->eof(lexer) || lexer->lookahead == '\\' ||
        iswspace(lexer->lookahead)) {
        return false;                              // (1) capacity guard
    }
    if (lexer->lookahead == '(') {
        return scanner->delimiter_length > 0;      // (2) legal terminator
    }
    scanner->delimiter[scanner->delimiter_length++] = lexer->lookahead;
    advance(lexer);
}

With a 16-char delimiter, the 16 d-chars are consumed and delimiter_length == 16. On the next iteration lookahead is ( — the legal terminator, check (2) — but check (1) runs first, sees the buffer is full, and returns false. The delimiter is never accepted even though nothing is wrong with it: the buffer is full, not overflowing.

15 works only because the loop still has a free slot when it reaches (.

Fix

Test the terminator before the capacity guard. A full buffer followed by ( is a complete, legal delimiter; a full buffer followed by another d-char is the genuine overflow.

     for (;;) {
-        if (scanner->delimiter_length >= MAX_DELIMITER_LENGTH || lexer->eof(lexer) || lexer->lookahead == '\\' ||
-            iswspace(lexer->lookahead)) {
-            return false;
-        }
         if (lexer->lookahead == '(') {
             // Rather than create a token for an empty delimiter, we fail and
             // let the grammar fall back to a delimiter-less rule.
             return scanner->delimiter_length > 0;
         }
+        if (scanner->delimiter_length >= MAX_DELIMITER_LENGTH || lexer->eof(lexer) || lexer->lookahead == '\\' ||
+            iswspace(lexer->lookahead)) {
+            return false;
+        }
         scanner->delimiter[scanner->delimiter_length++] = lexer->lookahead;
         advance(lexer);
     }

No bounds change: delimiter[] is still only written when delimiter_length < MAX_DELIMITER_LENGTH, so the array cannot overflow.

Verification

Built stock and patched scanners from the same sources and swept delimiter lengths 0-18, comparing function_definition counts:

length stock patched
0-15 1 1 identical
16 0 1 fixed
17, 18 0 0 identical — still rejected

The patch changes behaviour at exactly one length, the one that is legal today and rejected. 17+ stays rejected, so the spec limit is preserved, and length 0 (the empty-delimiter fallback the comment describes) is unchanged.

What I did not do: I could not run the corpus suite — npx tree-sitter-cli did not finish downloading in my environment — so this is verified by differential parsing rather than by tree-sitter test. Worth running before merging.

Happy to open a PR with the patch and a corpus case if useful.

Why it matters downstream

Files most likely to carry long descriptive delimiters — code generators, template/scaffold files, embedded shader or SQL blobs — are exactly where FILE_TEMPLATE_V1, SHADER_SOURCE_V2 or CMAKE_TEMPLATE_1 read as natural, and all are 16. When it happens the file yields no symbols at all, with no diagnostic.

Distinct from #245 (error-recovery synthesising a raw_string_delimiter, fixed in 3d8d510a) — nothing here is in error recovery; the input is well-formed C++.

Environment: this repo at HEAD, tree-sitter runtime from tree-sitter/tree-sitter HEAD, gcc 13.3.0, Ubuntu 24.04.

Lenguaje dominante
JavaScript
Estrellas
463
Forks
190
Sin PR fusionados en 30 d

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de tree-sitter/tree-sitter-cpp

Todos los issues de tree-sitter/tree-sitter-cpp

Issues similares

Más issues de JavaScript

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.