Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

FTS code analyzer: no split at CJK→Latin or non-ASCII digit boundaries; folding after split misses x²/ｕｔｆ８

Open
  • Is something wrong?
#9,666 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 3 days

Nobody has claimed this yet.

  • #9892 by @ahlag — closed without merging

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
48/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
python, rust
Domain
search

Research direction

Start with hunt/tokenization/r7.py to reproduce the token and query behavior, then inspect is_boundary in src/lance/rust/lance-tokenizer/src/word_delimiter_filter.rs:142-160 and the filter ordering in src/lance/rust/lance-index/src/scalar/inverted/tokenizer.rs:1094-1096 and :1145-1154. Done means the specified CJK, non-ASCII digit, superscript, and fullwidth cases split and match as expected, with the existing-index compatibility question resolved.

Written by the indexing model from the issue text.

Description

Bug bash: Needs team input — design question

Bug bash priority: Low · critical-fix: no
From the FTS for Code bug bash (2026-10-01), findings: tokenization-7.md. Repros ran against released artifacts (pylance 12.0.0 / 13.0.0b18, lancedb 0.39.0 / 0.40.0b12, @lancedb/lancedb 0.40.0-beta.12) unless stated. $S is the bug bash scratch dir.

Low priority; this is edge-case script handling. Changing tokenization affects existing indexes (see the tokenizer-compatibility question on the underscore-identifier issue). Recommendation: batch it with the other tokenizer fixes.

  • Category: bug (low)
  • Environment: pylance local, Python 3.12, Linux aarch64
  • Artifacts: pylance 13.0.0b18 and 12.0.0 (identical output)
  • Target tracker: GitHub lance-format/lance

Repro

$S/hunt/tokenization/r7.py:

import lance, pyarrow as pa, tempfile
from lance.query import MatchQuery, FullTextOperator as Op
print("pylance", lance.__version__)
S = dict(analyzer="code", split_identifiers=True)
for s in ["日本語Variable", "価格Total", "var٣Name", "x١٢٣", "x²", "ｕｔｆ８", "utf8"]:
    print(f"{s!r:16}", [(t.text, t.position) for t in lance.tokenize(s, **S)])
docs = ["日本語Variable = 1", "x² + 1", "ｕｔｆ８ = 1"]
ds = lance.write_dataset(pa.table({"id": [0, 1, 2], "code": docs}), tempfile.mkdtemp())
ds.create_scalar_index("code", "INVERTED", **S)
r = lambda q: ds.to_table(full_text_query=q, columns=["id", "_score"])["id"].to_pylist()
print("match variable ->", r("variable"))
print("AND x2   ->", r(MatchQuery("x2", "code", operator=Op.AND)))
print("AND utf8 ->", r(MatchQuery("utf8", "code", operator=Op.AND)))

Output:

pylance 13.0.0-beta.18
'日本語Variable'    [('日本語variable', 0)]
'価格Total'        [('価格total', 0)]
'var٣Name'       [('var٣name', 0)]
'x١٢٣'           [('x١٢٣', 0)]
'x²'             [('x2', 0)]
'ｕｔｆ８'           [('utf8', 0)]
'utf8'           [('utf8', 0), ('utf', 0), ('8', 1)]
match variable -> []
AND x2   -> []
AND utf8 -> []
  • 日本語Variable / 価格Total (mixed CJK + camelCase, common in Japanese/Chinese codebases) never split, so variable/total find nothing.
  • var٣Name / x١٢٣ (Arabic-Indic digits) never split, while var3Name does.
  • x² and fullwidth ｕｔｆ８ fold to x2 / utf8 after splitting, so they are indexed as a single x2/utf8 token with no subwords. The query utf8 (ASCII) becomes {utf8|utf}@0, 8@1, and AND/phrase require 8, so the row with the same folded text does not match.

Expected behavior

A switch between a caseless letter (CJK, Hangul, Arabic) and a cased letter is a boundary, as is a switch between any char::is_numeric digit and a non-digit. A document and a query that are identical after ascii_folding match each other under AND/phrase.

Evidence: ASCII equivalents behave this way (var3Name → var,3,name; utf8 matches utf8).

Suspected root cause

is_boundary (src/lance/rust/lance-tokenizer/src/word_delimiter_filter.rs:142-160) only checks is_ascii_digit and lowercase→uppercase transitions, so caseless→upper and non-ASCII digits are never boundaries. AsciiFoldingFilter runs last (src/lance/rust/lance-index/src/scalar/inverted/tokenizer.rs:1094-1096), after WordDelimiterFilter (:1145-1154), so split decisions are made on unfolded text.

Fix direction

Use is_numeric (or Nd) for the digit rule, and add a caseless-letter↔cased-letter boundary. Consider folding (or NFKC-normalizing) before the word-delimiter stage in the code profile. Low priority, but it changes tokens for existing indexes.

Dominant language
Rust
Stars
7.2k
Forks
900
Avg merge
5d 1h
Merged PRs (30d)
392

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from lance-format/lance

All issues in lance-format/lance

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.