Refactor RFC: 单分段重新建索引深模块化 (SegmentReindexService)
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
Research direction
Start by reading _reindex_single_segment in knowledge_bases.py and the related logic in IndexingRunner._load_indexes, then inspect any router tests covering the current helper. The work is done when SegmentReindexService owns single-segment index deletion and rebuilding, create_segment and update_segment call it, and boundary tests cover overrides, failure handling, and segment status updates.
Written by the indexing model from the issue text.
Description
问题(Problem)
描述架构摩擦点(architectural friction):
- 浅模块:
knowledge_bases.py中的_reindex_single_segment(约 60 行)在 API 路由层直接实现单分段重新建索引逻辑,接口几乎与实现同等复杂。路由层直接newLanceDBProvider、OllamaEmbeddingProvider、JiebaKeywordService,与IndexingRunner._load_indexes的向量/关键词双写逻辑高度重复。 - 集成缝隙:create_segment、update_segment 两处调用点与索引实现强耦合;删除分段时(delete_document_segments)又单独实现了一套 vector_db + keyword_service 的删除逻辑,未复用。
- 维护成本:理解「单分段重索引」需在 knowledge_bases.py、indexing_runner.py、LanceDBProvider、JiebaKeywordService 之间多次跳转;字段变更或容错策略调整需同步多处。
拟议接口(Proposed Interface)
接口签名
# core/segment_reindex_service.py
class SegmentReindexService:
def __init__(
self,
embedding: EmbeddingPort,
vector_db: VectorDBPort,
keyword: KeywordPort,
): ...
async def reindex(
self,
db: Session,
segment: DocumentSegment,
*,
kb: KnowledgeBase | None = None, # 可选,不传则从 segment 解析
doc: Document | None = None, # 可选
keywords_override: list[str] | None = None,
) -> None:
"""单分段重新建索引。kb/doc 不传时从 db 查询。"""
使用示例
# create_segment / update_segment 中
service = create_segment_reindex_service(db, kb) # 或通过 FastAPI Depends 注入
await service.reindex(db, segment, keywords_override=payload.keywords)
内部隐藏的复杂度
- index_node_id 解析与兼容(segment.index_node_id or segment.id)
- 删除旧向量/关键词索引
- Embedding 调用与向量写入
- 关键词写入(支持 keywords_override 或自动分词)
- indexing_technique 容错(high_quality 向量失败抛错,economy 关键词失败抛错)
- segment 状态更新(status、completed_at、error)
依赖策略(Dependency Strategy)
| 依赖 | 类别 | 策略 |
|---|---|---|
| LanceDBProvider | 本地可替代 | 定义 VectorDBPort,生产用 LanceDBAdapter,测试用 InMemoryVectorDBAdapter |
| OllamaEmbeddingProvider | 远程但由你方拥有 | Ports & Adapters:定义 EmbeddingPort,生产用 Ollama 适配器,测试用 InMemoryEmbeddingAdapter |
| JiebaKeywordService | 进程内 | 定义 KeywordPort,生产用 JiebaKeywordAdapter,测试用 FakeKeywordAdapter |
测试策略(Testing Strategy)
- 需要新增的边界测试:在 SegmentReindexService.reindex 接口处验证:删除旧索引后写入新索引、keywords_override 正确传递、indexing_technique 容错行为、segment 状态正确更新。
- 需要删除的旧测试:若有针对 router 内
_reindex_single_segment的单元测试,可删除。 - 测试环境需要:InMemoryEmbeddingAdapter、InMemoryVectorDBAdapter、FakeKeywordAdapter;或使用内存 SQLite + 临时 LanceDB 路径。
实现建议(Implementation Recommendations)
- 模块职责:SegmentReindexService 负责单分段索引的删除与重建,不负责 ORM 的 CRUD;router 负责参数校验、HTTP 映射、错误转 HTTP 状态码。
- 隐藏实现:调用方不感知 vector_db 路径、Ollama 配置、Jieba 分词细节。
- 接口合约:reindex 成功时更新 segment 状态;失败时抛出异常,由调用方处理。
- 迁移路径:1) 新建 core/segment_reindex_service.py 与 ports;2) 将 _reindex_single_segment 逻辑迁移到 service;3) router 中 create_segment、update_segment 改为调用 service.reindex;4) 删除 router 中的 _reindex_single_segment。
- Dominant language
- Python
- Stars
- 139
- Forks
- 28
- PR merge metrics
- No merged PRs in 30d
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Similar issues
-
documentation help wanted
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
simonw/sqlite-utils#872 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100