FTS code analyzer: identifier queries are 9–53x slower than simple when split_identifiers and preserve_original are on

Start in rust/lance-index/src/scalar/inverted/index/partition.rs, especially the needs_union path around lines 1309-1311, documents.lengths, and union_posting_lists. Read the union helpers and the v3 block-format behavior, then compare the provided benchmarks and analyze_plan output. Done means selecting and validating an approach that avoids the demonstrated query-time slowdown while preserving correct matching and scoring.

Bug bash: Needs team input — design question

Bug bash priority: Medium · critical-fix: no
From the FTS for Code bug bash (2026-10-01), findings: perf-1.md. Repros ran against released artifacts (pylance 12.0.0 / 13.0.0b18, lancedb 0.39.0 / 0.40.0b12, @lancedb/lancedb 0.40.0-beta.12) unless stated. $S is the bug bash scratch dir.

Decision needed: how to evaluate same-position alternatives efficiently. The full benchmark tables are in the bug bash report.

Options:

1. Score the alternatives lazily (a WAND-friendly disjunction) instead of merging full posting lists up front.
2. Precompute a merged posting for the original plus its first subword at index time.
3. Document preserve_original=False for latency-sensitive use.

Recommendation: 1.

• Category: performance
• Environment: pylance local, warm cache (after prewarm_index), Linux aarch64 Neoverse-V2, nproc=16, 30 GiB RAM (free -g), uname -m=aarch64. Datasets on local ext4 (/dev/root). Load average was 0.8-0.9 during the runs.
• Artifacts: pylance 13.0.0b18 (lance v13.0.0-beta.18, b646eda6a)
• Target tracker: GitHub lance-format/lance

Repro

CONFIGS = {
  "simple":            dict(base_tokenizer="simple", with_position=True),
  "code":              dict(analyzer="code", with_position=True),
  "code_split":        dict(analyzer="code", split_identifiers=True, with_position=True),
  "code_split_nopres": dict(analyzer="code", split_identifiers=True, preserve_original=False, with_position=True),
}
ds.create_scalar_index("code", "INVERTED", name="fts", **CONFIGS[cfg])
# query (grouped.py): 3 warm-up + 20 timed runs, after ds.prewarm_index("fts")
ds.to_table(columns=["id"], disable_scoring_autoprojection=True,
            full_text_query=MatchQuery(q, "code"), limit=10)

RUST_LOG=error $S/venv-pre/bin/python grouped.py /home/ubuntu/bugbash-perf-scratch/chunks25. Results are p50/max ms over 20 runs, with total matching rows in parentheses. The three process-level repeats agree within 1%. Rep 1:

query               | simple            | code             | code_split          | code_split_nopres
create_scalar_index | 2.15/2.38 (505475)| 0.88/3.07 (6725) | 55.26/57.20 (503600)| 2.70/2.89 (503600)
to_string           | 0.91/1.03 (431550)| 0.95/1.04 (270500)| 86.67/90.63 (752925)| 1.36/1.48 (752925)
new_dataset         | 2.37/2.43 (667250)| 0.85/0.95 (2325) | 126.36/129.66 (670975)| 2.25/2.37 (670975)
IndexType           | 0.88/1.02 (35325) | 0.89/0.97 (35325)| 46.36/46.81 (518875)| 1.77/1.92 (518875)
get_mut             | 1.85/1.91 (559400)| 0.88/1.04 (3975) | 47.16/47.44 (569775)| 2.11/2.22 (569775)
InvertedIndexParams | 0.91/1.00 (11325) | 0.83/0.88 (11325)| 8.26/8.56 (389575) | 1.32/1.41 (389575)
RecordBatch         | 0.90/1.04 (114025)| 0.88/0.99 (113825)| 42.42/42.68 (327275)| 1.58/1.74 (327275)
self.dataset        | 2.80/2.89 (547000)| 2.56/2.63 (520400)| 2.76/2.89 (552925) | 2.93/3.08 (552925)

Phrase queries hit it too (qbench.py, 3 reps): PhraseQuery("InvertedIndexParams") limit 10 takes 13.7-13.8 ms on code_split, 1.10-1.13 ms on simple, and 1.52-1.57 ms on code_split_nopres.

It scales with corpus size. create_scalar_index limit 10 on code_split takes 2.0 ms at 11k rows, 11.7 ms at 204k rows, and 55.3 ms at 1.02M rows. On simple it takes 0.8, 1.1, and 2.2 ms.

analyze_plan for new_dataset limit 10 (everything cached, no I/O):

code_split:        MatchQuery: elapsed=128.55ms, tokenized_query=[("new_dataset", 0), ("new", 0), ("dataset", 1)], index_comparisons=156, bytes_read=0, iops=0
code_split_nopres: MatchQuery: elapsed=1.51ms,   tokenized_query=[("new", 0), ("dataset", 1)],                    index_comparisons=85.50 K, bytes_read=0

Under concurrent load (qps.py: 16 threads, 400 queries of MatchQuery("new_dataset") limit 10, warm cache, 1.02M rows, 3 reps):

chunks25_simple.lance            threads=16 qps=3121-3261 p50=4.5-4.6ms  p99=7.8-8.4ms   maxrss=534-566MB
chunks25_code_split.lance        threads=16 qps=108-109   p50=141-142ms  p99=185-190ms   maxrss=1326-1358MB
chunks25_code_split_nopres.lance threads=16 qps=3192-3310 p50=4.4-4.5ms  p99=7.6-9.8ms   maxrss=536-549MB

That is 30x lower throughput and ~800 MB extra peak RSS from the per-query union buffers.

Remote (lancedb server + 1 PE, --use-remote-fts=true, EXPLAIN shows RemoteFtsExec ... scorer=local; 204,180-row chunks table; Python remote client; p50 range over 3 reps × 15 runs). Index built with SQL CREATE INDEX ON r_split USING fts (code) WITH (base_tokenizer='code', split_identifiers=true, with_position=true):

query (limit 10)     r_simple PD   r_split PD    r_nopres PD   | r_split no-pushdown
new_dataset          4.67-4.88     27.87-27.94   4.31-4.42     | 26.91-27.36
create_scalar_index  4.37-4.61     15.19-15.29   4.84-4.95     | 14.25-14.55
PhraseQuery camel    3.82-3.97     6.56-6.70     4.54-4.70     | 5.74-5.95

The same slowdown applies with and without pushdown, so it's in the Lance index search, not in LanceDB Enterprise.

Expected behavior

The result sets are nearly identical (code_split and code_split_nopres return the same row counts, and simple is within ~1%). Top-k latency should therefore be close to the text analyzer, as it is for code_split_nopres: 1.3-2.7 ms. In fact the grouped plan does fewer WAND comparisons (156 vs 85.5K) but is 85x slower. Any identifier with a separator is slow: snake_case, camelCase, PascalCase. self.dataset is not affected because . produces two separate identifiers, so no same-position group exists. That's ordinary code search, and it's the configuration users need to enable for subword matching (PW-4).

Suspected root cause

When preserve_original=true (the default), the query tokens for a split identifier include the original and the first subword at the same position. lance rust/lance-index/src/scalar/inverted/index/partition.rs:1309-1311 detects this (needs_union), and then:

• loads the lengths of every document in the partition (documents.lengths().await, partition.rs:1380-1386);
• eagerly builds a new merged posting list for the group with union_posting_lists (partition.rs:1155-1197, the union_* helpers at :900-1160). This decodes, merges, rescores, and builds blocks over the entire posting lists of both tokens before WAND starts. The first subword (new, to, get, create) is often among the most frequent terms, so this costs O(df) for each query;
• returns impact_safe: false, exact_scoring_required: true (partition.rs:1478-1481).

The cost is all CPU in the union (elapsed_compute = elapsed, zero I/O), and it's linear in posting length, which explains the scaling with corpus size. WAND then needs only 156 comparisons.

Fix direction

Avoid building the full union for each query. Options: a lazy union iterator with a per-block max-score upper bound (the sum or max of the members' block maxima), so WAND can skip blocks; caching the merged posting for the group; or, at index time, storing the original and first subword so that no query-time union is needed. Triage which fits the v3 block format.