[19] [Bug] [Retrieval] File document IDs depend on how the path is spelled; uppercase extensions are skipped
维护者通常 1 天内回复
还没有人认领这个 Issue。
评估
调研方向
Start with the source assignment in retrieval/loaders/text_loader.py (around line 55), then csv_loader.py, json_loader.py, pdf_loader.py and pdf_ocr_loader.py, and the rglob calls in directory mode. The HuggingFace source strings are in huggingface_loader.py near lines 138 and 180. Done means the same file gives the same id through different path spellings, and SCAN.PDF-style files are picked up. Decide the base for relative paths first, since that changes existing ids.
由索引模型根据 Issue 内容生成。
描述
Description
- IDs follow the path spelling. The file loaders use
str(path)as the source (retrieval/loaders/text_loader.py:55,csv_loader.py:86,json_loader.py:185,pdf_loader.py:83), and the source becomes the document id.kb/a.txt,/abs/kb/a.txtandkb/../kb/a.txtgive three different ids for the same file. Running from cron with a different working directory, or in a container with a different mount path, re-ingests the whole corpus as new documents, and the old copies are never removed. - HuggingFace sources are incomplete. They are
"{dataset_name}/{split}#..."(retrieval/loaders/huggingface_loader.py:138,:180). A config name or revision passed throughdataset_kwargsisn't included, so two configs of the same dataset collide. - Extensions match case-sensitively. Directory mode uses
rglob("*.pdf"),"*.csv"and similar (retrieval/loaders/pdf_loader.py:155,pdf_ocr_loader.py:231,csv_loader.py:156,json_loader.py:263), soSCAN.PDFandDATA.CSVare silently skipped.
Proposed fix
- Use the path relative to the directory passed in, or a resolved absolute path, as the source. Relative is more portable.
- Include the config and revision in HuggingFace sources.
- Match extensions case-insensitively.
Acceptance criteria
- The same file loaded through different path spellings gets the same id.
- Uppercase extensions are picked up in directory mode.
- 主要语言
- Python
- 星标
- 230
- 派生
- 57
- 平均合并
- 2 天 20 小时
- 30 天内合并 PR
- 36
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
RailtownAI/railtracks 的其他 Issue
-
[08] [Bug] [Retrieval] `OpenAIEmbedding`/`AzureEmbedding` have no default batch size, so the docs examples crash可能已有人在做 @Shivani120703 于 6 天前认领。 未关闭area:retrieval
难度 2/5 1-3 小时 新手友好度 82/100
RailtownAI/railtracks#1653 · 1 条评论 ·
维护者通常 1 天内回复
-
area:retrieval
难度 2/5 1-3 小时 新手友好度 75/100
RailtownAI/railtracks#1652 ·
维护者通常 1 天内回复
-
area:retrieval
难度 2/5 1-3 小时 新手友好度 78/100
RailtownAI/railtracks#1651 ·
维护者通常 1 天内回复
-
URL attachments with a non-image extension (.mp3, .mp4, .docx) are silently sent as images可能已有人在做 @CoronRing 于 6 天前认领。 未关闭area:llm bug
难度 2/5 1-3 小时 新手友好度 82/100
RailtownAI/railtracks#1645 · 已指派 1 人 ·
维护者通常 1 天内回复
-
area:docs
难度 2/5 1-3 小时 新手友好度 72/100
RailtownAI/railtracks#1633 ·
维护者通常 1 天内回复
查看 RailtownAI/railtracks 的全部 Issue
相似的 Issue
-
Issue Triage-Needed
难度 2/5 1-3 小时 新手友好度 68/100
pulp/pulp_container#2581 ·
维护者通常 1 天内回复
-
难度 1/5 1 小时以内 新手友好度 85/100
521xueweihan/HelloGitHub#4040 ·
-
bug
难度 2/5 1-3 小时 新手友好度 64/100
ClanGenOfficial/clangen#6313 ·
维护者通常 2 天内回复
-
bug
难度 2/5 1-3 小时 新手友好度 84/100
维护者通常 1 天内回复
-
bug(trainer): CustomTrainer func_args with non literal values fail inside the TrainJob可能已有人在做 @avanishkasar 今天认领。 未关闭
难度 2/5 1-3 小时 新手友好度 78/100
维护者通常 1 天内回复