Let a Dialect override tokenization

未关闭
  • 哪里不对？
#2,612 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

还没有人认领这个 Issue。

评估

难度
5/5
预计耗时
一周以上
新手友好度
35/100
Issue 类型
功能
描述清晰度
基本清楚
活跃度
活跃
技术栈
rust
领域
backend-api-design

调研方向

Start by reading Tokenizer::next_token, the private State character-stream implementation, and the parser dialect hooks. The issue proposes a new tokenizer override and raises unresolved questions about lookahead, custom token variants, and performance; first assess those design choices, then validate the hook against existing dialect-specific lexing and measure with sqlparser_bench. Done means the design is settled and dialects can override tokenization without consuming input when they return None.

由索引模型根据 Issue 内容生成。

描述

BigQuery DuckDB Hive MySQL PostgreSQL Snowflake

Follow-up to the discussion in #2610.

Problem

The parser lets a dialect take over at specific points (parse_prefix, parse_infix, parse_statement): it receives &mut Parser, and returning None falls back to the default. The tokenizer has no equivalent. A dialect can only answer yes/no questions the tokenizer asks (is_identifier_part, supports_triple_quoted_string, and about 20 others), so it can toggle existing lexemes but not add new ones.

Dialect-specific lexing therefore lives in Tokenizer::next_token behind dialect_of! checks:

  • b'...' byte strings (BigQuery, MySQL) and bit strings (Postgres)
  • r'...' raw strings (BigQuery)
  • // as a line comment (Snowflake) or integer division (DuckDB)
  • # as a line comment (Snowflake, BigQuery, MySQL, Hive)
  • \r ending a line comment (Postgres)

A downstream dialect cannot add a branch like these. Its only option is tokenize_with_location_into_buf_with_mapper, which rewrites one finished token at a time and cannot see characters or merge tokens.

Example: in dbt models, a Jinja comment {# ... #} breaks under every dialect that treats # as a line comment, because the tokenizer consumes the rest of the line, including #}.

Proposal

  1. Expose the character stream that next_token reads from (today the private State: peek, next, location), or a narrow public wrapper around it.
  2. Add a hook with the same contract as the parser hooks: 
    /// Dialect-specific tokenizer override, called before the default rules.
///
/// If `None` is returned, falls back to the default behavior.
fn next_token(
    &self,
    _chars: &mut State,
    _prev_token: Option<&Token>,
) -> Option<Result<Token, TokenizerError>> {
    None
}
  3. As a separate step, move the existing dialect_of! branches into the hook implementations of their dialects. That would show the hook covers syntax already in the tree, and would remove dialect_of! from the tokenizer.

Open questions

  • Lookahead: a hook that returns None must not consume input. State wraps Peekable<Chars> and allows one character of lookahead. Should State be Clone, or offer peek_nth?
  • Token is a closed enum, so a dialect-specific lexeme must map to an existing variant (Word, CustomBinaryOperator, ...). Is that enough, or should there be a Token::Custom variant?
  • Cost: one extra dynamic call per token. The tokenizer already calls several dialect methods per token, and sqlparser_bench can measure the difference.
主要语言
Rust
星标
3.5k
派生
805
平均合并
2 天 3 小时
30 天内合并 PR
116

环境准备

  • 没有 Dockerfile 或 Docker Compose 文件
  • 有 Pull Request 模板
  • 没有贡献指南

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

apache/datafusion-sqlparser-rs 的其他 Issue

查看 apache/datafusion-sqlparser-rs 的全部 Issue

相似的 Issue

更多 Rust Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。