Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

With LANCE_FTS_FORMAT_VERSION=1|2 set, analyzer="code" fails with "requires format_version=3, got 2" and never names the env var

Open
  • Is something wrong?
#9,672 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 3 days

@hello-peter-tang is already working on this.

Since Oct 12, 2026.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
52/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
python, rust
Domain
databases, search

Research direction

Start in src/lance/rust/lance-index/src/scalar/inverted/tokenizer.rs, reading resolve_creation_format_version, from_training_json, and validate_format_version. Run the provided Python reproduction with LANCE_FTS_FORMAT_VERSION set and unset, then inspect the full matrix behavior. Done means the code-analyzer error identifies the environment variable and format_version=3 override, after the team decides whether the analyzer bypasses the fallback.

Written by the indexing model from the issue text.

Description

Bug bash: Needs team input — design question

Bug bash priority: Medium · critical-fix: no
From the FTS for Code bug bash (2026-10-01), findings: lifecycle-3.md. Repros ran against released artifacts (pylance 12.0.0 / 13.0.0b18, lancedb 0.39.0 / 0.40.0b12, @lancedb/lancedb 0.40.0-beta.12) unless stated. $S is the bug bash scratch dir.

Decision needed: whether the code analyzer should bypass the env fallback.

Options:

  1. Have code ignore LANCE_FTS_FORMAT_VERSION (it only works on v3).
  2. Keep the precedence, and name the env var in the error.

Recommendation: 2 at minimum (an error-text fix); consider 1. A process-wide rollback setting would otherwise block every code index build.

  • Category: bug (error message / precedence UX)
  • Environment: pylance local, Python 3.12, Linux aarch64
  • Artifacts: pylance 13.0.0b18

Repro

import lance, os, pyarrow as pa, tempfile
print(lance.__version__, "LANCE_FTS_FORMAT_VERSION =", os.environ.get("LANCE_FTS_FORMAT_VERSION"))
ds = lance.write_dataset(pa.table({"code": ["def getUserName(): pass"]}), tempfile.mkdtemp())
try:
    ds.create_scalar_index("code", "INVERTED", analyzer="code", split_identifiers=True)
    print("created")
except Exception as e:
    print(type(e).__name__, e)

$ LANCE_FTS_FORMAT_VERSION=2 python r2.py
13.0.0-beta.18 LANCE_FTS_FORMAT_VERSION = 2
ValueError Invalid user input: base_tokenizer='code' requires FTS format_version=3, got 2, .../rust/lance-index/src/scalar/inverted/tokenizer.rs:1013:24
$ python r2.py
13.0.0-beta.18 LANCE_FTS_FORMAT_VERSION = None
created

The user never passed format_version, but the message says "got 2". LANCE_FTS_FORMAT_VERSION=1 gives the same result ("got 1"), as does base_tokenizer="code". Passing format_version=3 explicitly works under either env value. The full matrix run (t_fmt.py, env unset/1/2/3/bogus × format_version none/1/2/3/4/0/"3") otherwise behaved as designed:

  • Explicit format_version always wins.
  • 4 and 0 give unsupported FTS format version 4, expected 1, 2, or 3.
  • LANCE_FTS_FORMAT_VERSION=bogus names the variable.
  • optimize_indices with new data under env=1 keeps the existing v3 code index and its params.

Expected behavior

The error should say where the version came from, and how to override it. For example: base_tokenizer='code' requires FTS format_version=3, but LANCE_FTS_FORMAT_VERSION=2 is set; pass format_version=3 or unset the variable.

There is also a design question for the team. analyzer="code" is an explicit request for a v3-only feature. The env var is documented as a "rollout-wide override" fallback (src/lance/docs/src/guide/migration.md:16-19). Should a profile that requires v3 ignore the fallback (or warn about it) instead of failing? The current precedence matches the docs, so this file only asks the question.

Evidence: the bogus case shows the env-var wording already exists: invalid LANCE_FTS_FORMAT_VERSION value "bogus": ... (tokenizer.rs:588-592).

Suspected root cause

from_training_json (src/lance/rust/lance-index/src/scalar/inverted/tokenizer.rs:1063-1077) resolves the env value in resolve_creation_format_version (:580-599) and stores it in params.format_version as if the user had passed it. validate_format_version (:1009-1019) then can't tell an explicit value from an env value, so it prints only the number.

Fix direction

Track whether the format version came from the environment, and name LANCE_FTS_FORMAT_VERSION plus the format_version=3 override in the code-analyzer error. Team call on whether code should bypass the env fallback.

Dominant language
Rust
Stars
7.2k
Forks
900
Avg merge
5d 2h
Merged PRs (30d)
405

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from lance-format/lance

All issues in lance-format/lance

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.