Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

FTS code analyzer: no split at CJK→Latin or non-ASCII digit boundaries; folding after split misses x²/ｕｔｆ８

Aperta
  • Qualcosa non va?
#9,666 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 3 giorni

Nessuno ha ancora preso questa issue.

  • #9892 di @ahlag — chiusa senza merge

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
48/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python, rust
Ambito
search

Direzione di ricerca

Start with hunt/tokenization/r7.py to reproduce the token and query behavior, then inspect is_boundary in src/lance/rust/lance-tokenizer/src/word_delimiter_filter.rs:142-160 and the filter ordering in src/lance/rust/lance-index/src/scalar/inverted/tokenizer.rs:1094-1096 and :1145-1154. Done means the specified CJK, non-ASCII digit, superscript, and fullwidth cases split and match as expected, with the existing-index compatibility question resolved.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Bug bash: Needs team input — design question

Bug bash priority: Low · critical-fix: no
From the FTS for Code bug bash (2026-10-01), findings: tokenization-7.md. Repros ran against released artifacts (pylance 12.0.0 / 13.0.0b18, lancedb 0.39.0 / 0.40.0b12, @lancedb/lancedb 0.40.0-beta.12) unless stated. $S is the bug bash scratch dir.

Low priority; this is edge-case script handling. Changing tokenization affects existing indexes (see the tokenizer-compatibility question on the underscore-identifier issue). Recommendation: batch it with the other tokenizer fixes.

  • Category: bug (low)
  • Environment: pylance local, Python 3.12, Linux aarch64
  • Artifacts: pylance 13.0.0b18 and 12.0.0 (identical output)
  • Target tracker: GitHub lance-format/lance

Repro

$S/hunt/tokenization/r7.py:

import lance, pyarrow as pa, tempfile
from lance.query import MatchQuery, FullTextOperator as Op
print("pylance", lance.__version__)
S = dict(analyzer="code", split_identifiers=True)
for s in ["日本語Variable", "価格Total", "var٣Name", "x١٢٣", "x²", "ｕｔｆ８", "utf8"]:
    print(f"{s!r:16}", [(t.text, t.position) for t in lance.tokenize(s, **S)])
docs = ["日本語Variable = 1", "x² + 1", "ｕｔｆ８ = 1"]
ds = lance.write_dataset(pa.table({"id": [0, 1, 2], "code": docs}), tempfile.mkdtemp())
ds.create_scalar_index("code", "INVERTED", **S)
r = lambda q: ds.to_table(full_text_query=q, columns=["id", "_score"])["id"].to_pylist()
print("match variable ->", r("variable"))
print("AND x2   ->", r(MatchQuery("x2", "code", operator=Op.AND)))
print("AND utf8 ->", r(MatchQuery("utf8", "code", operator=Op.AND)))

Output:

pylance 13.0.0-beta.18
'日本語Variable'    [('日本語variable', 0)]
'価格Total'        [('価格total', 0)]
'var٣Name'       [('var٣name', 0)]
'x١٢٣'           [('x١٢٣', 0)]
'x²'             [('x2', 0)]
'ｕｔｆ８'           [('utf8', 0)]
'utf8'           [('utf8', 0), ('utf', 0), ('8', 1)]
match variable -> []
AND x2   -> []
AND utf8 -> []
  • 日本語Variable / 価格Total (mixed CJK + camelCase, common in Japanese/Chinese codebases) never split, so variable/total find nothing.
  • var٣Name / x١٢٣ (Arabic-Indic digits) never split, while var3Name does.
  • x² and fullwidth ｕｔｆ８ fold to x2 / utf8 after splitting, so they are indexed as a single x2/utf8 token with no subwords. The query utf8 (ASCII) becomes {utf8|utf}@0, 8@1, and AND/phrase require 8, so the row with the same folded text does not match.

Expected behavior

A switch between a caseless letter (CJK, Hangul, Arabic) and a cased letter is a boundary, as is a switch between any char::is_numeric digit and a non-digit. A document and a query that are identical after ascii_folding match each other under AND/phrase.

Evidence: ASCII equivalents behave this way (var3Name → var,3,name; utf8 matches utf8).

Suspected root cause

is_boundary (src/lance/rust/lance-tokenizer/src/word_delimiter_filter.rs:142-160) only checks is_ascii_digit and lowercase→uppercase transitions, so caseless→upper and non-ASCII digits are never boundaries. AsciiFoldingFilter runs last (src/lance/rust/lance-index/src/scalar/inverted/tokenizer.rs:1094-1096), after WordDelimiterFilter (:1145-1154), so split decisions are made on unfolded text.

Fix direction

Use is_numeric (or Nd) for the digit rule, and add a caseless-letter↔cased-letter boundary. Consider folding (or NFKC-normalizing) before the word-delimiter stage in the code profile. Low priority, but it changes tokens for existing indexes.

Lingua principale
Rust
Stelle
7.2k
Fork
913
Merge medio
5g 2h
PR unite (30g)
405

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di lance-format/lance

Tutte le issue di lance-format/lance

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.