Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

FTS code analyzer index_operators: greedy matching turns nested generics into >> and a<-1 into <-

Aperta
  • Qualcosa non va?
#9,664 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 3 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
38/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python, rust
Ambito
search

Direzione di ricerca

Start with src/lance/rust/lance-tokenizer/src/code_tokenizer.rs, especially is_operator_char, MULTI_CHAR_OPERATORS, and operator_token_end. Run test_code_lex_tokenizer_splits_adjacent_operators and the provided hunt/tokenization/r5.py reproduction to compare nested generic, comparison, and dot-operator tokenization. Done requires an agreed tokenization behavior, regression coverage, and documentation for any compatibility impact.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Bug bash: Needs team input — design question

Bug bash priority: Medium · critical-fix: no
From the FTS for Code bug bash (2026-10-01), findings: tokenization-5.md. Repros ran against released artifacts (pylance 12.0.0 / 13.0.0b18, lancedb 0.39.0 / 0.40.0b12, @lancedb/lancedb 0.40.0-beta.12) unless stated. $S is the bug bash scratch dir.

Decision needed: which operators are atomic tokens.

Options:

  1. Index single-character angle brackets and comparison operators individually.
  2. Keep longest-match but special-case >>/<< inside generics. This is not lexically decidable without language awareness.
  3. Document the current behavior.

Recommendation: 1. It also affects existing indexes (see the tokenizer-compatibility question on the underscore-identifier issue).

  • Category: bug
  • Environment: pylance local, Python 3.12, Linux aarch64
  • Artifacts: pylance 13.0.0b18 and 12.0.0 (identical output)
  • Target tracker: GitHub lance-format/lance

Repro

$S/hunt/tokenization/r5.py:

import lance, pyarrow as pa, tempfile
from lance.query import MatchQuery, PhraseQuery
print("pylance", lance.__version__)
O = dict(analyzer="code", index_operators=True)
for s in ["Vec<Vec<u8>>", "HashMap<String, Vec<i32>>", "a<-1", "x <- 1", "opt?.value", "a..b", "a...b"]:
    print(f"{s!r:30}", [t.text for t in lance.tokenize(s, **O)])
docs = ["let v: Vec<Vec<u8>> = vec![];", "let v: Vec<u8> = vec![];", "map: HashMap<String, Vec<i32>>", "x <- 1", "if a<-1 { }"]
ds = lance.write_dataset(pa.table({"id": list(range(5)), "code": docs}), tempfile.mkdtemp())
ds.create_scalar_index("code", "INVERTED", with_position=True, **O)
r = lambda q: ds.to_table(full_text_query=q, columns=["id", "_score"])["id"].to_pylist()
print('phrase "Vec<u8>"  ->', r(PhraseQuery("Vec<u8>", "code")))
print('phrase "Vec<i32>" ->', r(PhraseQuery("Vec<i32>", "code")))
print('match ">"         ->', r(MatchQuery(">", "code")))
print('match "<-"        ->', r(MatchQuery("<-", "code")))

Output:

pylance 13.0.0-beta.18
'Vec<Vec<u8>>'                 ['vec', '<', 'vec', '<', 'u8', '>>']
'HashMap<String, Vec<i32>>'    ['hashmap', '<', 'string', 'vec', '<', 'i32', '>>']
'a<-1'                         ['a', '<-', '1']
'x <- 1'                       ['x', '<-', '1']
'opt?.value'                   ['opt', '?', 'value']
'a..b'                         ['a', 'b']
'a...b'                        ['a', 'b']
phrase "Vec<u8>"  -> [1]
phrase "Vec<i32>" -> []
match ">"         -> [1]
match "<-"        -> [3, 4]

Problems, in order of impact:

  1. Closing nested generics (Vec<Vec<u8>>, Map<K, List<V>>, Option<Box<T>>) become >>, the shift operator. Phrase Vec<i32> misses row 2. Match > misses row 0, the most common place > appears in Rust/Java/C++/TS types.
  2. a<-1 (less-than negative) is tokenized as the R/Go/Haskell <- operator. A search for <- matches row 4.
  3. ?. (optional chaining in JS/TS/Kotlin/C#), ../..= (Rust ranges), ... (spread/varargs), and . are not operator characters. ?. indexes as ?, and the others are dropped entirely, so they cannot be searched.

Expected behavior

Vec<Vec<u8>> yields >,> (at minimum the closing > tokens remain findable as >). Searching Vec<i32> as a phrase finds HashMap<String, Vec<i32>>. Common multi-character operators that contain . (?., .., ...) are indexable, or the docs say they are not.

Evidence: existing test test_code_lex_tokenizer_splits_adjacent_operators (src/lance/rust/lance-tokenizer/src/code_tokenizer.rs:182-190) expects parse::<usize>() to give <, > as separate tokens. So generic brackets are meant to be single </> tokens, and the nested case simply has no test.

Suspected root cause

operator_token_end (src/lance/rust/lance-tokenizer/src/code_tokenizer.rs:96-113) takes the longest entry of MULTI_CHAR_OPERATORS (:76-80, which includes >>, >>>, >>=, <-) that prefixes the remaining text, with no context. is_operator_char (:69-74) excludes ., so ?., .. and ... cannot form tokens.

Fix direction

Triage. Options: emit >>/>>> also as > alternatives at the same position (like preserved originals), or make the shift-family tokens only bind when not closing a < opened earlier in the stream. Add .-containing operators to the list, or document the exclusions. Any change alters tokenization of existing indexes.

Lingua principale
Rust
Stelle
7.2k
Fork
900
Merge medio
5g 2h
PR unite (30g)
405

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di lance-format/lance

Tutte le issue di lance-format/lance

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.