Let a Dialect override tokenization

Abierto
  • ¿Algo no está bien?
#2,612 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
35/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
rust
Área
backend-api-design

Línea de trabajo

Start by reading Tokenizer::next_token, the private State character-stream implementation, and the parser dialect hooks. The issue proposes a new tokenizer override and raises unresolved questions about lookahead, custom token variants, and performance; first assess those design choices, then validate the hook against existing dialect-specific lexing and measure with sqlparser_bench. Done means the design is settled and dialects can override tokenization without consuming input when they return None.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

BigQuery DuckDB Hive MySQL PostgreSQL Snowflake

Follow-up to the discussion in #2610.

Problem

The parser lets a dialect take over at specific points (parse_prefix, parse_infix, parse_statement): it receives &mut Parser, and returning None falls back to the default. The tokenizer has no equivalent. A dialect can only answer yes/no questions the tokenizer asks (is_identifier_part, supports_triple_quoted_string, and about 20 others), so it can toggle existing lexemes but not add new ones.

Dialect-specific lexing therefore lives in Tokenizer::next_token behind dialect_of! checks:

  • b'...' byte strings (BigQuery, MySQL) and bit strings (Postgres)
  • r'...' raw strings (BigQuery)
  • // as a line comment (Snowflake) or integer division (DuckDB)
  • # as a line comment (Snowflake, BigQuery, MySQL, Hive)
  • \r ending a line comment (Postgres)

A downstream dialect cannot add a branch like these. Its only option is tokenize_with_location_into_buf_with_mapper, which rewrites one finished token at a time and cannot see characters or merge tokens.

Example: in dbt models, a Jinja comment {# ... #} breaks under every dialect that treats # as a line comment, because the tokenizer consumes the rest of the line, including #}.

Proposal

  1. Expose the character stream that next_token reads from (today the private State: peek, next, location), or a narrow public wrapper around it.
  2. Add a hook with the same contract as the parser hooks: 
    /// Dialect-specific tokenizer override, called before the default rules.
///
/// If `None` is returned, falls back to the default behavior.
fn next_token(
    &self,
    _chars: &mut State,
    _prev_token: Option<&Token>,
) -> Option<Result<Token, TokenizerError>> {
    None
}
  3. As a separate step, move the existing dialect_of! branches into the hook implementations of their dialects. That would show the hook covers syntax already in the tree, and would remove dialect_of! from the tokenizer.

Open questions

  • Lookahead: a hook that returns None must not consume input. State wraps Peekable<Chars> and allows one character of lookahead. Should State be Clone, or offer peek_nth?
  • Token is a closed enum, so a dialect-specific lexeme must map to an existing variant (Word, CustomBinaryOperator, ...). Is that enough, or should there be a Token::Custom variant?
  • Cost: one extra dynamic call per token. The tokenizer already calls several dialect methods per token, and sqlparser_bench can measure the difference.
Lenguaje dominante
Rust
Estrellas
3.5k
Forks
805
Merge medio
2 d 3 h
PR fusionados (30 d)
116

Preparar el entorno

  • Sin Dockerfile ni archivo de Docker Compose
  • Tiene una plantilla de pull request
  • Sin guía de contribución

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de apache/datafusion-sqlparser-rs

Todos los issues de apache/datafusion-sqlparser-rs

Issues similares

Más issues de Rust

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.