Hacktoberfest 2026：维护者为十月标记出来的 issue，仍然开放、适合新手。 浏览 Hacktoberfest issue

Docs: no code-analyzer coverage for FTS; base_tokenizer list omits "code"

未关闭
  • 哪里不对？
#369 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

还没有人认领这个 Issue。

评估

难度
4/5
预计耗时
3-5 天
新手友好度
55/100
Issue 类型
文档
描述清晰度
基本清楚
活跃度
活跃
技术栈
python, typescript
领域
documentation, search

调研方向

Start with docs/indexing/fts-index.mdx and compare its tokenizer table and Tokenizer choices section with src/lancedb/docs/src/js/type-aliases/BaseTokenizer.md and src/lancedb/python/python/lancedb/index.py. Read the referenced PW-1/PW-2 context before editing. Done means the docs list code, explain code searching and query usage, and state the identifier-splitting and stemming/stop-word limitations accurately.

由索引模型根据 Issue 内容生成。

描述

documentation

Bug bash: Ready to fix

Bug bash priority: Low · critical-fix: no
From the FTS for Code bug bash (2026-10-01), findings: sdkdocs-7.md. Repros ran against released artifacts (pylance 12.0.0 / 13.0.0b18, lancedb 0.39.0 / 0.40.0b12, @lancedb/lancedb 0.40.0-beta.12) unless stated. $S is the bug bash scratch dir.

Docs gap, not dispatched.

  • Category: docs gap
  • Environment: lancedb/docs repo, main at 7fa2ff6 (2026-09-24), cloned read-only. src/lancedb/docs at v0.40.0-beta.12. Checked against lancedb 0.40.0b12 / @lancedb/lancedb 0.40.0-beta.12
  • Artifacts: as above
  • Target tracker: GitHub lancedb/docs

Repro

$ grep -rn -i 'split_identifiers\|analyzer=\|base_tokenizer="code"\|code analyzer\|splitIdentifiers' docs tests   # in lancedb/docs
(no hits other than unrelated icon="code" attributes)

docs/indexing/fts-index.mdx:88:

| `base_tokenizer` | str | `"simple"` | Text splitting method (`simple`, `whitespace`, `raw`, `ngram`, `icu`, `jieba/*`, or `lindera/*`) |

The "Tokenizer choices" section (fts-index.mdx:107-119) has no code entry. src/lancedb/docs/src/js/type-aliases/BaseTokenizer.md:10-19 mirrors the TS union without "code". The Python FTS docstring (src/lancedb/python/python/lancedb/index.py:136-145) also lists no "code".

Even so, everything on the query side already works from both SDKs on a code index, whether pylance built it locally or LanceDB Enterprise SQL built it remotely with split_identifiers = true. My parity runs (parity_local.py, parity_remote.py, ts/parity.mjs, ts/parity_remote.mjs, scores.py, strq.py) returned identical row sets in pylance, lancedb Python local and remote, and lancedb TS local and remote. That covers match, phrase, AUTO fuzzy, AND, boolean, hybrid, string queries with code punctuation, and Table.tokenize. BM25 scores were identical too:

match 'user'         pylance=[0, 3] lancedb=[0, 3]          (remote py [0, 3], TS local/remote [0,3])
phrase 'userName'    pylance= [0, 3] lancedb= [0, 3]
user local [(0, 1.0618), (3, 1.425)] remote [(0, 1.0618), (3, 1.425)]
Table.tokenize: [('getusername', 0), ('get', 0), ('user', 1), ('name', 2), ('user_id', 3), ('user', 3), ('id', 4), ...]

Expected behavior

fts-index.mdx lists code with a short "searching source code" subsection. The subsection covers what the code tokenizer does, that stemming and stop words should be off for it (see PW-2), and how to query with MatchQuery/PhraseQuery. Until PW-1 lands, it should also say that identifier splitting is only available via pylance or LanceDB Enterprise SQL.

Evidence: parity results above; LanceDB Enterprise skill doc above.

Suspected root cause

No lancedb or lancedb/docs PR followed lance#7681 (plan "What changed").

Fix direction

Add code to the tokenizer table and a code-search subsection on the docs site, after or together with the PW-1/PW-2 SDK fixes so the documented defaults are right.

主要语言
Python
星标
11
派生
14
平均合并
1 天 12 小时
30 天内合并 PR
9

环境准备

这个项目没有提供开发容器、Dockerfile 或贡献指南，环境需要你自己搭建：先看它的 README，通用步骤见我们的新手贡献指南。

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

lancedb/docs 的其他 Issue

查看 lancedb/docs 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。