Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

FTS code analyzer index_operators: greedy matching turns nested generics into >> and a<-1 into <-

Open
  • Is something wrong?
#9,664 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 3 days

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
38/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
python, rust
Domain
search

Research direction

Start with src/lance/rust/lance-tokenizer/src/code_tokenizer.rs, especially is_operator_char, MULTI_CHAR_OPERATORS, and operator_token_end. Run test_code_lex_tokenizer_splits_adjacent_operators and the provided hunt/tokenization/r5.py reproduction to compare nested generic, comparison, and dot-operator tokenization. Done requires an agreed tokenization behavior, regression coverage, and documentation for any compatibility impact.

Written by the indexing model from the issue text.

Description

Bug bash: Needs team input — design question

Bug bash priority: Medium · critical-fix: no
From the FTS for Code bug bash (2026-10-01), findings: tokenization-5.md. Repros ran against released artifacts (pylance 12.0.0 / 13.0.0b18, lancedb 0.39.0 / 0.40.0b12, @lancedb/lancedb 0.40.0-beta.12) unless stated. $S is the bug bash scratch dir.

Decision needed: which operators are atomic tokens.

Options:

  1. Index single-character angle brackets and comparison operators individually.
  2. Keep longest-match but special-case >>/<< inside generics. This is not lexically decidable without language awareness.
  3. Document the current behavior.

Recommendation: 1. It also affects existing indexes (see the tokenizer-compatibility question on the underscore-identifier issue).

  • Category: bug
  • Environment: pylance local, Python 3.12, Linux aarch64
  • Artifacts: pylance 13.0.0b18 and 12.0.0 (identical output)
  • Target tracker: GitHub lance-format/lance

Repro

$S/hunt/tokenization/r5.py:

import lance, pyarrow as pa, tempfile
from lance.query import MatchQuery, PhraseQuery
print("pylance", lance.__version__)
O = dict(analyzer="code", index_operators=True)
for s in ["Vec<Vec<u8>>", "HashMap<String, Vec<i32>>", "a<-1", "x <- 1", "opt?.value", "a..b", "a...b"]:
    print(f"{s!r:30}", [t.text for t in lance.tokenize(s, **O)])
docs = ["let v: Vec<Vec<u8>> = vec![];", "let v: Vec<u8> = vec![];", "map: HashMap<String, Vec<i32>>", "x <- 1", "if a<-1 { }"]
ds = lance.write_dataset(pa.table({"id": list(range(5)), "code": docs}), tempfile.mkdtemp())
ds.create_scalar_index("code", "INVERTED", with_position=True, **O)
r = lambda q: ds.to_table(full_text_query=q, columns=["id", "_score"])["id"].to_pylist()
print('phrase "Vec<u8>"  ->', r(PhraseQuery("Vec<u8>", "code")))
print('phrase "Vec<i32>" ->', r(PhraseQuery("Vec<i32>", "code")))
print('match ">"         ->', r(MatchQuery(">", "code")))
print('match "<-"        ->', r(MatchQuery("<-", "code")))

Output:

pylance 13.0.0-beta.18
'Vec<Vec<u8>>'                 ['vec', '<', 'vec', '<', 'u8', '>>']
'HashMap<String, Vec<i32>>'    ['hashmap', '<', 'string', 'vec', '<', 'i32', '>>']
'a<-1'                         ['a', '<-', '1']
'x <- 1'                       ['x', '<-', '1']
'opt?.value'                   ['opt', '?', 'value']
'a..b'                         ['a', 'b']
'a...b'                        ['a', 'b']
phrase "Vec<u8>"  -> [1]
phrase "Vec<i32>" -> []
match ">"         -> [1]
match "<-"        -> [3, 4]

Problems, in order of impact:

  1. Closing nested generics (Vec<Vec<u8>>, Map<K, List<V>>, Option<Box<T>>) become >>, the shift operator. Phrase Vec<i32> misses row 2. Match > misses row 0, the most common place > appears in Rust/Java/C++/TS types.
  2. a<-1 (less-than negative) is tokenized as the R/Go/Haskell <- operator. A search for <- matches row 4.
  3. ?. (optional chaining in JS/TS/Kotlin/C#), ../..= (Rust ranges), ... (spread/varargs), and . are not operator characters. ?. indexes as ?, and the others are dropped entirely, so they cannot be searched.

Expected behavior

Vec<Vec<u8>> yields >,> (at minimum the closing > tokens remain findable as >). Searching Vec<i32> as a phrase finds HashMap<String, Vec<i32>>. Common multi-character operators that contain . (?., .., ...) are indexable, or the docs say they are not.

Evidence: existing test test_code_lex_tokenizer_splits_adjacent_operators (src/lance/rust/lance-tokenizer/src/code_tokenizer.rs:182-190) expects parse::<usize>() to give <, > as separate tokens. So generic brackets are meant to be single </> tokens, and the nested case simply has no test.

Suspected root cause

operator_token_end (src/lance/rust/lance-tokenizer/src/code_tokenizer.rs:96-113) takes the longest entry of MULTI_CHAR_OPERATORS (:76-80, which includes >>, >>>, >>=, <-) that prefixes the remaining text, with no context. is_operator_char (:69-74) excludes ., so ?., .. and ... cannot form tokens.

Fix direction

Triage. Options: emit >>/>>> also as > alternatives at the same position (like preserved originals), or make the shift-family tokens only bind when not closing a < opened earlier in the stream. Add .-containing operators to the list, or document the exclusions. Any change alters tokenization of existing indexes.

Dominant language
Rust
Stars
7.2k
Forks
900
Avg merge
4d 22h
Merged PRs (30d)
366

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from lance-format/lance

All issues in lance-format/lance

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.