FTS: max_token_length=40 also drops tokens of exactly 40 bytes; docs say only longer tokens are removed

Aperta
  • Qualcosa non va?
#9,668 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
2/5
Tempo stimato
1-3 ore
Idoneità per principianti
52/100
Tipo di issue
Documentazione
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python, rust
Ambito
documentation, search

Direzione di ricerca

Start at src/lance/python/python/lance/dataset.py:3841-3843 and compare its max_token_length wording with the repro in $S/hunt/tokenization/r4.py. Confirm whether the team selects the docs-only option; done means the docs state that tokens at least max_token_length bytes are removed, including byte rather than character semantics, without changing tokenizer behavior.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Bug bash: Needs team input — design question

Bug bash priority: Low · critical-fix: no
From the FTS for Code bug bash (2026-10-01), findings: tokenization-4.md. Repros ran against released artifacts (pylance 12.0.0 / 13.0.0b18, lancedb 0.39.0 / 0.40.0b12, @lancedb/lancedb 0.40.0-beta.12) unless stated. $S is the bug bash scratch dir.

Decision needed:

Options:

  1. Change the check to <=. This changes which tokens existing indexes contain.
  2. Fix the docs to say "at least", and that the limit is in bytes.

Recommendation: 2, since it's safe for existing indexes.

  • Category: bug (off-by-one) / docs gap
  • Environment: pylance local, Python 3.12, Linux aarch64
  • Artifacts: pylance 13.0.0b18 and 12.0.0 (identical output)
  • Target tracker: GitHub lance-format/lance

Repro

$S/hunt/tokenization/r4.py:

import lance, pyarrow as pa, tempfile
print("pylance", lance.__version__)
for n in [39, 40]:
    for kw in [dict(analyzer="code"), {}]:
        print(n, kw, lance.tokenize("x" * n, **kw))
print("é*20 (20 chars, 40 bytes)", lance.tokenize("é" * 20, analyzer="code"))
ds = lance.write_dataset(pa.table({"id": [0, 1], "code": ["x" * 39, "x" * 40]}), tempfile.mkdtemp())
ds.create_scalar_index("code", "INVERTED", analyzer="code")
for n in [39, 40]:
    print(n, "->", ds.to_table(full_text_query="x" * n, columns=["id", "_score"])["id"].to_pylist())

Output:

pylance 13.0.0-beta.18
39 {'analyzer': 'code'} [FtsToken(text="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", position=0)]
39 {} [FtsToken(text="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", position=0)]
40 {'analyzer': 'code'} []
40 {} []
é*20 (20 chars, 40 bytes) []
39 -> [0]
40 -> []

A 40-byte identifier cannot be found. Both the code and text analyzers are affected. The limit counts UTF-8 bytes, not characters: 20 é characters are dropped. Neither point is documented.

Expected behavior

As documented, "Any token longer than this will be removed" (src/lance/python/python/lance/dataset.py:3841-3843): a 40-byte token is kept. The docs should also say the limit is in bytes.

Suspected root cause

src/lance/rust/lance-tokenizer/src/remove_long.rs:54-56: token.text.len() < self.token_length_limit keeps only tokens strictly shorter than the limit. This is inherited from Tantivy's RemoveLongFilter (remove_long.rs:4), whose limit is exclusive.

Fix direction

Either change the predicate to <= (this changes tokenization for existing indexes: new 40-byte tokens appear on append/optimize versus old segments, so it may need gating), or fix the docs to say "tokens of max_token_length bytes or more are removed". The docs-only fix is the safe one.

Lingua principale
Rust
Stelle
7.3k
Fork
924
Merge medio
3g 22h
PR unite (30g)
570

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di lance-format/lance

Tutte le issue di lance-format/lance

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.