Let a Dialect override tokenization

Ouverte
  • Quelque chose ne va pas ?
#2,612 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

Personne n'a encore pris cette issue.

Évaluation

Difficulté
5/5
Temps estimé
Plus d'une semaine
Accessibilité débutants
35/100
Type d'issue
Fonctionnalité
Clarté
Plutôt claire
Activité
Active
Stack technique
rust
Domaine
backend-api-design

Piste de recherche

Start by reading Tokenizer::next_token, the private State character-stream implementation, and the parser dialect hooks. The issue proposes a new tokenizer override and raises unresolved questions about lookahead, custom token variants, and performance; first assess those design choices, then validate the hook against existing dialect-specific lexing and measure with sqlparser_bench. Done means the design is settled and dialects can override tokenization without consuming input when they return None.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

BigQuery DuckDB Hive MySQL PostgreSQL Snowflake

Follow-up to the discussion in #2610.

Problem

The parser lets a dialect take over at specific points (parse_prefix, parse_infix, parse_statement): it receives &mut Parser, and returning None falls back to the default. The tokenizer has no equivalent. A dialect can only answer yes/no questions the tokenizer asks (is_identifier_part, supports_triple_quoted_string, and about 20 others), so it can toggle existing lexemes but not add new ones.

Dialect-specific lexing therefore lives in Tokenizer::next_token behind dialect_of! checks:

  • b'...' byte strings (BigQuery, MySQL) and bit strings (Postgres)
  • r'...' raw strings (BigQuery)
  • // as a line comment (Snowflake) or integer division (DuckDB)
  • # as a line comment (Snowflake, BigQuery, MySQL, Hive)
  • \r ending a line comment (Postgres)

A downstream dialect cannot add a branch like these. Its only option is tokenize_with_location_into_buf_with_mapper, which rewrites one finished token at a time and cannot see characters or merge tokens.

Example: in dbt models, a Jinja comment {# ... #} breaks under every dialect that treats # as a line comment, because the tokenizer consumes the rest of the line, including #}.

Proposal

  1. Expose the character stream that next_token reads from (today the private State: peek, next, location), or a narrow public wrapper around it.
  2. Add a hook with the same contract as the parser hooks: 
    /// Dialect-specific tokenizer override, called before the default rules.
///
/// If `None` is returned, falls back to the default behavior.
fn next_token(
    &self,
    _chars: &mut State,
    _prev_token: Option<&Token>,
) -> Option<Result<Token, TokenizerError>> {
    None
}
  3. As a separate step, move the existing dialect_of! branches into the hook implementations of their dialects. That would show the hook covers syntax already in the tree, and would remove dialect_of! from the tokenizer.

Open questions

  • Lookahead: a hook that returns None must not consume input. State wraps Peekable<Chars> and allows one character of lookahead. Should State be Clone, or offer peek_nth?
  • Token is a closed enum, so a dialect-specific lexeme must map to an existing variant (Word, CustomBinaryOperator, ...). Is that enough, or should there be a Token::Custom variant?
  • Cost: one extra dynamic call per token. The tokenizer already calls several dialect methods per token, and sqlparser_bench can measure the difference.
Langage dominant
Rust
Étoiles
3.5k
Forks
805
Merge moyen
2 j 3 h
PR mergées (30 j)
116

Préparer son environnement

  • Aucun Dockerfile ni fichier Docker Compose
  • Propose un modèle de pull request
  • Aucun guide de contribution

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de apache/datafusion-sqlparser-rs

Toutes les issues de apache/datafusion-sqlparser-rs

Issues similaires

Plus d'issues Rust

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.