Let a Dialect override tokenization

Aperta
  • Qualcosa non va?
#2,612 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
35/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
rust
Ambito
backend-api-design

Direzione di ricerca

Start by reading Tokenizer::next_token, the private State character-stream implementation, and the parser dialect hooks. The issue proposes a new tokenizer override and raises unresolved questions about lookahead, custom token variants, and performance; first assess those design choices, then validate the hook against existing dialect-specific lexing and measure with sqlparser_bench. Done means the design is settled and dialects can override tokenization without consuming input when they return None.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

BigQuery DuckDB Hive MySQL PostgreSQL Snowflake

Follow-up to the discussion in #2610.

Problem

The parser lets a dialect take over at specific points (parse_prefix, parse_infix, parse_statement): it receives &mut Parser, and returning None falls back to the default. The tokenizer has no equivalent. A dialect can only answer yes/no questions the tokenizer asks (is_identifier_part, supports_triple_quoted_string, and about 20 others), so it can toggle existing lexemes but not add new ones.

Dialect-specific lexing therefore lives in Tokenizer::next_token behind dialect_of! checks:

  • b'...' byte strings (BigQuery, MySQL) and bit strings (Postgres)
  • r'...' raw strings (BigQuery)
  • // as a line comment (Snowflake) or integer division (DuckDB)
  • # as a line comment (Snowflake, BigQuery, MySQL, Hive)
  • \r ending a line comment (Postgres)

A downstream dialect cannot add a branch like these. Its only option is tokenize_with_location_into_buf_with_mapper, which rewrites one finished token at a time and cannot see characters or merge tokens.

Example: in dbt models, a Jinja comment {# ... #} breaks under every dialect that treats # as a line comment, because the tokenizer consumes the rest of the line, including #}.

Proposal

  1. Expose the character stream that next_token reads from (today the private State: peek, next, location), or a narrow public wrapper around it.
  2. Add a hook with the same contract as the parser hooks: 
    /// Dialect-specific tokenizer override, called before the default rules.
///
/// If `None` is returned, falls back to the default behavior.
fn next_token(
    &self,
    _chars: &mut State,
    _prev_token: Option<&Token>,
) -> Option<Result<Token, TokenizerError>> {
    None
}
  3. As a separate step, move the existing dialect_of! branches into the hook implementations of their dialects. That would show the hook covers syntax already in the tree, and would remove dialect_of! from the tokenizer.

Open questions

  • Lookahead: a hook that returns None must not consume input. State wraps Peekable<Chars> and allows one character of lookahead. Should State be Clone, or offer peek_nth?
  • Token is a closed enum, so a dialect-specific lexeme must map to an existing variant (Word, CustomBinaryOperator, ...). Is that enough, or should there be a Token::Custom variant?
  • Cost: one extra dynamic call per token. The tokenizer already calls several dialect methods per token, and sqlparser_bench can measure the difference.
Lingua principale
Rust
Stelle
3.5k
Fork
805
Merge medio
2g 3h
PR unite (30g)
116

Preparare l'ambiente

  • Nessun Dockerfile né file Docker Compose
  • Ha un modello di pull request
  • Nessuna guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/datafusion-sqlparser-rs

Tutte le issue di apache/datafusion-sqlparser-rs

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.