Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

FTS: max_token_length=40 also drops tokens of exactly 40 bytes; docs say only longer tokens are removed

Open
  • Is something wrong?
#9,668 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 3 days

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
52/100
Issue type
Documentation
Clarity
Mostly clear
Activity status
Active
Tech stack
python, rust
Domain
documentation, search

Research direction

Start at src/lance/python/python/lance/dataset.py:3841-3843 and compare its max_token_length wording with the repro in $S/hunt/tokenization/r4.py. Confirm whether the team selects the docs-only option; done means the docs state that tokens at least max_token_length bytes are removed, including byte rather than character semantics, without changing tokenizer behavior.

Written by the indexing model from the issue text.

Description

Bug bash: Needs team input — design question

Bug bash priority: Low · critical-fix: no
From the FTS for Code bug bash (2026-10-01), findings: tokenization-4.md. Repros ran against released artifacts (pylance 12.0.0 / 13.0.0b18, lancedb 0.39.0 / 0.40.0b12, @lancedb/lancedb 0.40.0-beta.12) unless stated. $S is the bug bash scratch dir.

Decision needed:

Options:

  1. Change the check to <=. This changes which tokens existing indexes contain.
  2. Fix the docs to say "at least", and that the limit is in bytes.

Recommendation: 2, since it's safe for existing indexes.

  • Category: bug (off-by-one) / docs gap
  • Environment: pylance local, Python 3.12, Linux aarch64
  • Artifacts: pylance 13.0.0b18 and 12.0.0 (identical output)
  • Target tracker: GitHub lance-format/lance

Repro

$S/hunt/tokenization/r4.py:

import lance, pyarrow as pa, tempfile
print("pylance", lance.__version__)
for n in [39, 40]:
    for kw in [dict(analyzer="code"), {}]:
        print(n, kw, lance.tokenize("x" * n, **kw))
print("é*20 (20 chars, 40 bytes)", lance.tokenize("é" * 20, analyzer="code"))
ds = lance.write_dataset(pa.table({"id": [0, 1], "code": ["x" * 39, "x" * 40]}), tempfile.mkdtemp())
ds.create_scalar_index("code", "INVERTED", analyzer="code")
for n in [39, 40]:
    print(n, "->", ds.to_table(full_text_query="x" * n, columns=["id", "_score"])["id"].to_pylist())

Output:

pylance 13.0.0-beta.18
39 {'analyzer': 'code'} [FtsToken(text="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", position=0)]
39 {} [FtsToken(text="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", position=0)]
40 {'analyzer': 'code'} []
40 {} []
é*20 (20 chars, 40 bytes) []
39 -> [0]
40 -> []

A 40-byte identifier cannot be found. Both the code and text analyzers are affected. The limit counts UTF-8 bytes, not characters: 20 é characters are dropped. Neither point is documented.

Expected behavior

As documented, "Any token longer than this will be removed" (src/lance/python/python/lance/dataset.py:3841-3843): a 40-byte token is kept. The docs should also say the limit is in bytes.

Suspected root cause

src/lance/rust/lance-tokenizer/src/remove_long.rs:54-56: token.text.len() < self.token_length_limit keeps only tokens strictly shorter than the limit. This is inherited from Tantivy's RemoveLongFilter (remove_long.rs:4), whose limit is exclusive.

Fix direction

Either change the predicate to <= (this changes tokenization for existing indexes: new 40-byte tokens appear on append/optimize versus old segments, so it may need gating), or fix the docs to say "tokens of max_token_length bytes or more are removed". The docs-only fix is the safe one.

Dominant language
Rust
Stars
7.2k
Forks
900
Avg merge
4d 22h
Merged PRs (30d)
366

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from lance-format/lance

All issues in lance-format/lance

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.