Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

FTS code analyzer: default max_token_length=40 silently drops long identifiers, so a class can't be found by its own name

Open
  • Is something wrong?
#9,667 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 3 days

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
42/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
python, rust
Domain
databases, search

Research direction

Start with apply_code_defaults in src/lance/rust/lance-index/src/scalar/inverted/tokenizer.rs and the max_token_length handling around the cited lines. Run $S/hunt/tokenization/r3.py to reproduce the dropped identifier, then review src/lance/docs/src/guide/tokenizer.md and the create_scalar_index docstring. Done means the team has chosen a code-profile default, implemented it for new indexes, and documented the resulting behavior.

Written by the indexing model from the issue text.

Description

Bug bash: Needs team input — design question

Bug bash priority: Medium · critical-fix: no
From the FTS for Code bug bash (2026-10-01), findings: tokenization-3.md. Repros ran against released artifacts (pylance 12.0.0 / 13.0.0b18, lancedb 0.39.0 / 0.40.0b12, @lancedb/lancedb 0.40.0-beta.12) unless stated. $S is the bug bash scratch dir.

Decision needed: the default token length limit for the code profile.

Options:

  1. Raise or remove the limit for the code analyzer.
  2. Keep 40 but, with splitting on, keep the subwords and drop only the original (today the original is already dropped).
  3. Keep 40 and document it prominently.

Recommendation: 1 (for example 128), since identifiers over 40 bytes are common: about 9k Rust function names in our own repos.

  • Category: feature gap (default choice) / docs gap
  • Environment: pylance local, Python 3.12, Linux aarch64
  • Artifacts: pylance 13.0.0b18 and 12.0.0 (identical output)
  • Target tracker: GitHub lance-format/lance

Repro

$S/hunt/tokenization/r3.py:

import lance, pyarrow as pa, tempfile
print("pylance", lance.__version__)
name = "AbstractSingletonProxyFactoryBeanTransactionInterceptorConfig"  # 61 bytes
print(lance.tokenize(f"class {name} {{}}", analyzer="code"))
ds = lance.write_dataset(pa.table({"id": [0], "code": [f"class {name} {{}}"]}), tempfile.mkdtemp())
ds.create_scalar_index("code", "INVERTED", analyzer="code")
for q in [name, "class"]:
    print(repr(q), "->", ds.to_table(full_text_query=q, columns=["id", "_score"])["id"].to_pylist())

Output:

pylance 13.0.0-beta.18
[FtsToken(text="class", position=0)]
'AbstractSingletonProxyFactoryBeanTransactionInterceptorConfig' -> []
'class' -> [0]

Expected behavior

The code profile keeps identifiers of realistic length by default: a larger limit, or no limit. At minimum, the code-analyzer docs warn that names over 40 bytes are not indexed unless max_token_length is raised.

Evidence: the create_scalar_index docstring documents max_token_length default 40 only as a text-profile option (src/lance/python/python/lance/dataset.py:3841-3843). The code-analyzer docs (src/lance/docs/src/guide/tokenizer.md:15-47) do not mention it. Long descriptive names are idiomatic for Java/C#/Rust tests.

Suspected root cause

apply_code_defaults (src/lance/rust/lance-index/src/scalar/inverted/tokenizer.rs:679-689) resets stem/stop words/folding but leaves max_token_length at the text default Some(40) (:641). RemoveLongFilter runs right after the word-delimiter stage (tokenizer.rs:1082-1083). The 40-byte limit was tuned for natural-language junk tokens (base64, hashes), not for identifiers.

Fix direction

Give the code profile its own max_token_length default (for example 128 or 255) in apply_code_defaults and document it. Needs a decision, because it changes the default for new indexes (existing ones persist their value).

Dominant language
Rust
Stars
7.2k
Forks
900
Avg merge
4d 22h
Merged PRs (30d)
366

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from lance-format/lance

All issues in lance-format/lance

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.