Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

TokenReader subrange bounds enforced inconsistently: peekToken/peekPreviousTokenKind/backtrackToMarker ignore embedded window

Abierto
  • ¿Algo no está bien?
#481 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
3/5
Tiempo estimado
1-2 días
Aptitud para principiantes
72/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
typescript
Área
compilers

Línea de trabajo

Empieza en tsdoc/src/parser/TokenReader.ts comparando peekToken(), peekPreviousTokenKind() y backtrackToMarker() con los métodos hermanos protegidos. Revisa el comportamiento del lector embebido y los puntos de llamada de NodeParser mencionados en el issue. Se considera terminado cuando los tres métodos respetan la ventana [_readerStartIndex, _readerEndIndex) y el comportamiento elegido para el final de la entrada está cubierto por pruebas.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Summary

TokenReader respects its embedded subrange (_readerStartIndex/_readerEndIndex) in peekTokenKind()/peekTokenAfterKind()/peekTokenAfterAfterKind(), but three sibling methods ignore those bounds: peekToken() has no end guard at all, and peekPreviousTokenKind() / backtrackToMarker() compare against 0 / allow rewinding before the subrange start. This leaks outer tokens into embedded parses and can return undefined where Token is promised.

Location

  • File: tsdoc/src/parser/TokenReader.ts
  • Class: TokenReader
  • Methods:
    • peekToken(): Token — return this.tokens[this._currentIndex]; with no _readerEndIndex check
    • peekPreviousTokenKind() — if (this._currentIndex === 0) instead of === this._readerStartIndex
    • backtrackToMarker(marker) — checks marker > this._currentIndex but not marker < this._readerStartIndex

Contrast with the guarded siblings in the same file:

public peekTokenKind(): TokenKind {
  if (this._currentIndex >= this._readerEndIndex) {
    return TokenKind.EndOfInput;
  }
  ...
}

Problem

  1. peekToken() past end returns undefined: return type claims Token, but past _readerEndIndex the index expression yields undefined. Callers doing peekToken().range / peekToken().kind then throw TypeError: Cannot read properties of undefined instead of getting a clean EndOfInput signal. Current internal hot paths in NodeParser.ts happen to call peekTokenKind() first (e.g. block/inline tag badCharacter paths), which masks the hole, but the public API contract is broken for any external consumer or future internal use.
  2. peekPreviousTokenKind() leaks across embedded start: for an embedded reader starting at index N>0, calling it at the embedded start returns tokens[N-1].kind (outer context) instead of EndOfInput. NodeParser._parseBlockTag and _parseFencedCode switch on this to decide start-of-input behavior (AtSignInWord, CodeFenceOpeningIndent); an embedded @ or fence at the subrange start can therefore be misclassified using the token before the subrange.
  3. backtrackToMarker() can rewind before subrange start: nothing prevents marker < _readerStartIndex, letting a later readToken() consume outer tokens that the embedded reader was explicitly scoped to exclude.

Trigger / Reproduction

Based on static analysis (no execution performed):

  • Construct new TokenReader(parserContext, embeddedSequence) where embeddedSequence.startIndex > 0, advance to the embedded start, and call peekPreviousTokenKind() — returns the outer predecessor kind instead of EndOfInput.
  • Call peekToken() when _currentIndex === _readerEndIndex — returns undefined instead of signaling end (compare peekTokenKind() returning EndOfInput in the same state).
  • Call backtrackToMarker(outerMarker) with outerMarker < _readerStartIndex on an embedded reader — accepted, subsequent reads escape the subrange.

Note: this is a static-analysis finding; I did not execute a parser fixture.

Expected Behavior

All TokenReader accessors/mutators should honor the same [ _readerStartIndex, _readerEndIndex ) window: peekToken() should signal end (or throw a clear parser-bug error like readToken() does) instead of returning undefined; peekPreviousTokenKind() should return EndOfInput at the subrange start; backtrackToMarker() should reject markers below the subrange start.

Actual Behavior

Subrange bounds are enforced inconsistently, so embedded parses can observe outer tokens and end-of-input handling differs by method.

Impact

  • Potential spurious AtSignInWord / fence-indent errors for embedded constructs starting at a subrange boundary.
  • undefined-token TypeErrors for API consumers using peekToken() at end, bypassing the clean EndOfInput protocol every sibling method follows.

Suggested Direction

  • Mirror the existing peekTokenKind() guard in peekToken() (return the EndOfInput token or throw a parser-bug error — maintainer's choice, but document it), change the peekPreviousTokenKind() zero-check to _readerStartIndex, and add a marker < _readerStartIndex rejection in backtrackToMarker(). No grammar changes needed.

Evidence

  • Source via API: tsdoc/src/parser/TokenReader.ts shows guarded peekTokenKind/peekTokenAfterKind/peekTokenAfterAfterKind vs unguarded peekToken/peekPreviousTokenKind/backtrackToMarker; tsdoc/src/parser/NodeParser.ts shows peekPreviousTokenKind gating AtSignInWord/CodeFenceOpeningIndent and embedded TokenReader construction for scoped parses.
  • Duplicate check: issue search for peekToken TokenReader bounds returns total_count: 0, and open issues contain no TokenReader-boundary report — no apparent duplicate. This is a non-security correctness finding, so Microsoft's private-security-disclosure requirement does not apply.

Classification

  • FACT: three TokenReader methods ignore the subrange window that three sibling methods enforce (verified in source via API).
  • INFERENCE: embedded parses can observe out-of-range tokens; end-of-input peekToken() yields undefined.
  • HYPOTHESIS: aligning all methods on the same window fixes the misclassification/TypeError risk with no grammar change.
Lenguaje dominante
TypeScript
Estrellas
5k
Forks
162
Merge medio
2 d 4 h
PR fusionados (30 d)
12

Preparar el entorno

Aún no hemos revisado los archivos de configuración de este proyecto. Empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de microsoft/tsdoc

Todos los issues de microsoft/tsdoc

Issues similares

Más issues de TypeScript

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.