Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

With LANCE_FTS_FORMAT_VERSION=1|2 set, analyzer="code" fails with "requires format_version=3, got 2" and never names the env var

Aperta
  • Qualcosa non va?
#9,672 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 3 giorni

@hello-peter-tang ci sta già lavorando.

Dal 12/10/2026.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
52/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python, rust
Ambito
databases, search

Direzione di ricerca

Start in src/lance/rust/lance-index/src/scalar/inverted/tokenizer.rs, reading resolve_creation_format_version, from_training_json, and validate_format_version. Run the provided Python reproduction with LANCE_FTS_FORMAT_VERSION set and unset, then inspect the full matrix behavior. Done means the code-analyzer error identifies the environment variable and format_version=3 override, after the team decides whether the analyzer bypasses the fallback.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Bug bash: Needs team input — design question

Bug bash priority: Medium · critical-fix: no
From the FTS for Code bug bash (2026-10-01), findings: lifecycle-3.md. Repros ran against released artifacts (pylance 12.0.0 / 13.0.0b18, lancedb 0.39.0 / 0.40.0b12, @lancedb/lancedb 0.40.0-beta.12) unless stated. $S is the bug bash scratch dir.

Decision needed: whether the code analyzer should bypass the env fallback.

Options:

  1. Have code ignore LANCE_FTS_FORMAT_VERSION (it only works on v3).
  2. Keep the precedence, and name the env var in the error.

Recommendation: 2 at minimum (an error-text fix); consider 1. A process-wide rollback setting would otherwise block every code index build.

  • Category: bug (error message / precedence UX)
  • Environment: pylance local, Python 3.12, Linux aarch64
  • Artifacts: pylance 13.0.0b18

Repro

import lance, os, pyarrow as pa, tempfile
print(lance.__version__, "LANCE_FTS_FORMAT_VERSION =", os.environ.get("LANCE_FTS_FORMAT_VERSION"))
ds = lance.write_dataset(pa.table({"code": ["def getUserName(): pass"]}), tempfile.mkdtemp())
try:
    ds.create_scalar_index("code", "INVERTED", analyzer="code", split_identifiers=True)
    print("created")
except Exception as e:
    print(type(e).__name__, e)

$ LANCE_FTS_FORMAT_VERSION=2 python r2.py
13.0.0-beta.18 LANCE_FTS_FORMAT_VERSION = 2
ValueError Invalid user input: base_tokenizer='code' requires FTS format_version=3, got 2, .../rust/lance-index/src/scalar/inverted/tokenizer.rs:1013:24
$ python r2.py
13.0.0-beta.18 LANCE_FTS_FORMAT_VERSION = None
created

The user never passed format_version, but the message says "got 2". LANCE_FTS_FORMAT_VERSION=1 gives the same result ("got 1"), as does base_tokenizer="code". Passing format_version=3 explicitly works under either env value. The full matrix run (t_fmt.py, env unset/1/2/3/bogus × format_version none/1/2/3/4/0/"3") otherwise behaved as designed:

  • Explicit format_version always wins.
  • 4 and 0 give unsupported FTS format version 4, expected 1, 2, or 3.
  • LANCE_FTS_FORMAT_VERSION=bogus names the variable.
  • optimize_indices with new data under env=1 keeps the existing v3 code index and its params.

Expected behavior

The error should say where the version came from, and how to override it. For example: base_tokenizer='code' requires FTS format_version=3, but LANCE_FTS_FORMAT_VERSION=2 is set; pass format_version=3 or unset the variable.

There is also a design question for the team. analyzer="code" is an explicit request for a v3-only feature. The env var is documented as a "rollout-wide override" fallback (src/lance/docs/src/guide/migration.md:16-19). Should a profile that requires v3 ignore the fallback (or warn about it) instead of failing? The current precedence matches the docs, so this file only asks the question.

Evidence: the bogus case shows the env-var wording already exists: invalid LANCE_FTS_FORMAT_VERSION value "bogus": ... (tokenizer.rs:588-592).

Suspected root cause

from_training_json (src/lance/rust/lance-index/src/scalar/inverted/tokenizer.rs:1063-1077) resolves the env value in resolve_creation_format_version (:580-599) and stores it in params.format_version as if the user had passed it. validate_format_version (:1009-1019) then can't tell an explicit value from an env value, so it prints only the number.

Fix direction

Track whether the format version came from the environment, and name LANCE_FTS_FORMAT_VERSION plus the format_version=3 override in the code-analyzer error. Team call on whether code should bypass the env fallback.

Lingua principale
Rust
Stelle
7.2k
Fork
913
Merge medio
5g 3h
PR unite (30g)
427

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di lance-format/lance

Tutte le issue di lance-format/lance

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.