Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Docs: no code-analyzer coverage for FTS; base_tokenizer list omits "code"

Aperta
  • Qualcosa non va?
#369 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
55/100
Tipo di issue
Documentazione
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python, typescript
Ambito
documentation, search

Direzione di ricerca

Start with docs/indexing/fts-index.mdx and compare its tokenizer table and Tokenizer choices section with src/lancedb/docs/src/js/type-aliases/BaseTokenizer.md and src/lancedb/python/python/lancedb/index.py. Read the referenced PW-1/PW-2 context before editing. Done means the docs list code, explain code searching and query usage, and state the identifier-splitting and stemming/stop-word limitations accurately.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

documentation

Bug bash: Ready to fix

Bug bash priority: Low · critical-fix: no
From the FTS for Code bug bash (2026-10-01), findings: sdkdocs-7.md. Repros ran against released artifacts (pylance 12.0.0 / 13.0.0b18, lancedb 0.39.0 / 0.40.0b12, @lancedb/lancedb 0.40.0-beta.12) unless stated. $S is the bug bash scratch dir.

Docs gap, not dispatched.

  • Category: docs gap
  • Environment: lancedb/docs repo, main at 7fa2ff6 (2026-09-24), cloned read-only. src/lancedb/docs at v0.40.0-beta.12. Checked against lancedb 0.40.0b12 / @lancedb/lancedb 0.40.0-beta.12
  • Artifacts: as above
  • Target tracker: GitHub lancedb/docs

Repro

$ grep -rn -i 'split_identifiers\|analyzer=\|base_tokenizer="code"\|code analyzer\|splitIdentifiers' docs tests   # in lancedb/docs
(no hits other than unrelated icon="code" attributes)

docs/indexing/fts-index.mdx:88:

| `base_tokenizer` | str | `"simple"` | Text splitting method (`simple`, `whitespace`, `raw`, `ngram`, `icu`, `jieba/*`, or `lindera/*`) |

The "Tokenizer choices" section (fts-index.mdx:107-119) has no code entry. src/lancedb/docs/src/js/type-aliases/BaseTokenizer.md:10-19 mirrors the TS union without "code". The Python FTS docstring (src/lancedb/python/python/lancedb/index.py:136-145) also lists no "code".

Even so, everything on the query side already works from both SDKs on a code index, whether pylance built it locally or LanceDB Enterprise SQL built it remotely with split_identifiers = true. My parity runs (parity_local.py, parity_remote.py, ts/parity.mjs, ts/parity_remote.mjs, scores.py, strq.py) returned identical row sets in pylance, lancedb Python local and remote, and lancedb TS local and remote. That covers match, phrase, AUTO fuzzy, AND, boolean, hybrid, string queries with code punctuation, and Table.tokenize. BM25 scores were identical too:

match 'user'         pylance=[0, 3] lancedb=[0, 3]          (remote py [0, 3], TS local/remote [0,3])
phrase 'userName'    pylance= [0, 3] lancedb= [0, 3]
user local [(0, 1.0618), (3, 1.425)] remote [(0, 1.0618), (3, 1.425)]
Table.tokenize: [('getusername', 0), ('get', 0), ('user', 1), ('name', 2), ('user_id', 3), ('user', 3), ('id', 4), ...]

Expected behavior

fts-index.mdx lists code with a short "searching source code" subsection. The subsection covers what the code tokenizer does, that stemming and stop words should be off for it (see PW-2), and how to query with MatchQuery/PhraseQuery. Until PW-1 lands, it should also say that identifier splitting is only available via pylance or LanceDB Enterprise SQL.

Evidence: parity results above; LanceDB Enterprise skill doc above.

Suspected root cause

No lancedb or lancedb/docs PR followed lance#7681 (plan "What changed").

Fix direction

Add code to the tokenizer table and a code-search subsection on the docs site, after or together with the PW-1/PW-2 SDK fixes so the documented defaults are right.

Lingua principale
Python
Stelle
11
Fork
13
Merge medio
1g 12h
PR unite (30g)
9

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di lancedb/docs

Tutte le issue di lancedb/docs

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.