A pylance ≤8.x writer silently deletes a format-v3 FTS index (every code index) on its next commit

Start with rust/lance/src/index.rs, rust/lance/src/dataset.rs, and rust/lance-table/src/feature_flags.rs, then run the repro scripts in repro/r1_write.py, repro/r1_old.py, and repro/r1_check.py. Trace how unsupported indices enter the next manifest and how writer flags are checked. Done means an old writer cannot erase a format-v3 index and the migration guidance covers pre-v9 writers.

Bug bash: Needs team input — systemic

Bug bash priority: Urgent · critical-fix: yes
From the FTS for Code bug bash (2026-10-01), findings: lifecycle-1.md. Repros ran against released artifacts (pylance 12.0.0 / 13.0.0b18, lancedb 0.39.0 / 0.40.0b12, @lancedb/lancedb 0.40.0-beta.12) unless stated. $S is the bug bash scratch dir.

Released versions can't be fixed retroactively (the erase path was fixed in #8427, v12+). The question is how new writers protect v3 indexes from old writers. This is a format/compatibility decision.

Options:

1. Set a writer feature flag while a v3 FTS index is present, so ≤8.x refuses to write.
2. Document a minimum writer version for code indexes.
3. Both.

Recommendation: 3. Needs an owner for the format change.

• Category: bug (data loss: index metadata), plus docs gap
• Environment: pylance local, Python 3.12, Linux aarch64 (Ubuntu 26.04)
• Artifacts: writer of the index: pylance 13.0.0b18. Old writer: pylance 8.0.1 (7.0.0 behaves the same). Readers 9.0.0, 10.0.0, 11.0.0, 12.0.0 were also checked and are fine (see below).
• Target tracker: GitHub lance-format/lance

Repro

# repro/r1_write.py  (pylance 13.0.0b18)
import lance, pyarrow as pa, shutil
shutil.rmtree("repro/r1.lance", ignore_errors=True)
ds = lance.write_dataset(pa.table({"id": [0, 1], "code": ["def getUserName(): pass", "class HTTPServer: pass"]}), "repro/r1.lance")
ds.create_scalar_index("code", "INVERTED", analyzer="code", split_identifiers=True)
ds = lance.dataset("repro/r1.lance")
print(lance.__version__, "version", ds.version, "indices", [i.name for i in ds.describe_indices()])

# repro/r1_old.py  (pylance 8.0.1)
import lance, pyarrow as pa
ds = lance.dataset("repro/r1.lance")
print(lance.__version__, "sees indices:", ds.list_indices())
try:
    ds.to_table(full_text_query="user")
except Exception as e:
    print("FTS:", e)
ds.insert(pa.table({"id": [2], "code": ["fn fetchUserProfile() {}"]}))
print("insert ok, version", lance.dataset("repro/r1.lance").version)

# repro/r1_check.py  (pylance 13.0.0b18)
import lance
for v in lance.dataset("repro/r1.lance").versions():
    d = lance.dataset("repro/r1.lance", version=v["version"])
    print("version", v["version"], "rows", d.count_rows(), "indices", [i.name for i in d.describe_indices()])
try:
    lance.dataset("repro/r1.lance").to_table(full_text_query="user")
except Exception as e:
    print("FTS:", e)

Output:

13.0.0-beta.18 version 2 indices ['code_idx']
[2026-10-01T21:20:51Z WARN  lance::index] Index code_idx has version 3, which is not supported (<=2), ignoring it
8.0.1 sees indices: []
FTS: Invalid user input: Cannot perform full text search unless an INVERTED index has been created on at least one column, .../lance-index/src/scalar/inverted/query.rs:835:25
insert ok, version 3
version 1 rows 2 indices []
version 2 rows 2 indices ['code_idx']
version 3 rows 3 indices []
FTS: Invalid user input: Cannot perform full text search unless an INVERTED index has been created on at least one column, .../lance-index/src/scalar/inverted/query.rs:1173:20

After the 8.0.1 insert, the code index is gone from the manifest. Every later version, read by any pylance, has no FTS index. The user can only get it back by checking out version 2 or rebuilding. The 8.x reader shows only a WARN log line. Its query error says no INVERTED index exists, and it does not say the index needs a newer reader.

Related behavior from the same run:

• With 8.0.1, create_scalar_index("code", "INVERTED") (default replace=True) silently replaces the hidden code index with a text index of the same name code_idx. The 13.0 reader then sees base_tokenizer='simple'.
• A block_size=256 text index (also format v3) is dropped the same way. The bug is not specific to the code analyzer. But every code index is format v3, so every code index is exposed.
• Not affected: pylance 9.0.0, 10.0.0, 11.0.0 and 12.0.0 can each open, query, append to, and optimize_indices a 13.0.0b18 code index. Each returns the same results as 13.0. I tested with an 18-identifier corpus and 43 queries, using split, nosplit, index_operators, with_position and block_size=256 variants. 13.0 also optimizes a code index built by 9.0.0 with identical results. Correction to the research notes: v9.0.0 already contains the code analyzer (fee1f3ceb, a cherry-pick of #7681, plus #7866 3956a55c0). So the readers without code support are ≤ 8.x, not 9.0.x. This also means the migration note's "## 9.0.0" heading is correct.

Expected behavior

An older writer that cannot read an index must not delete that index from the next manifest. It should either refuse to commit, or carry the index through untouched. The old reader should say the index needs a newer Lance, not that it doesn't exist.

Evidence: lance#8427 (108f78e92, in v12.0.0) fixed exactly this erase path for current builds. Its commit message says "For an index whose version this build cannot read, that turns 'ignore it' into 'delete it' on the next commit of any kind". Releases before the fix still delete the index. docs/src/guide/migration.md:9-24 says "readers must support v3 before an index using either option is created". It does not warn that a pre-v9 writer will drop the index.

Suspected root cause

In v8.0.1, retain_supported_indices (lance v8.0.1:rust/lance/src/index.rs:1710-1733) filters out indices with index_version above the plugin max. It is called while loading the manifest (v8.0.1:rust/lance/src/dataset.rs:696). The commit path builds the next manifest from that filtered list, so any commit erases the hidden index. The current codebase fixed this by splitting load_all_indices and load_indices (lance#8427). Releases older than that can't be patched. Nothing in the manifest stops an old writer: the code index adds no writer feature flag (rust/lance-table/src/feature_flags.rs:11-88), so can_write_dataset passes on 8.x.

Fix direction

Set a writer feature flag in the manifest while a v3 (or code) FTS index is present. v8.0.1's can_write_dataset refuses any writer_flags >= 64 (v8.0.1:rust/lance-table/src/feature_flags.rs:24,84-86), so a new bit would make 8.x fail loudly instead of erasing the index. Also add a migration-guide warning that pre-v9 writers drop these indices.