[Bug]: JSONLinesFileReader drops rows containing U+2028, U+2029 or U+0085 (uses str.splitlines)
维护者通常 5 天内回复
还没有人认领这个 Issue。
评估
调研方向
该缺陷位于 packages/graphrag-input/graphrag_input/jsonl.py 中的 read_file，其中使用 text.splitlines() 切分记录。将该切分替换为仅按 "\n 切分，并去除每行末尾的 "\r"，同时保持现有的行号计数和跳过警告行为不变。在字符串值中加入 U+2028、U+2029 和 U+0085 编写一个测试，然后运行现有的 JSONL 读取器测试。当这些行能完整读取、CRLF 文件仍能正常解析时即视为完成。
由索引模型根据 Issue 内容生成。
描述
Do you need to file an issue?
- I have searched the existing issues and this bug is not already filed.
- My model is hosted on OpenAI or Azure. If not, please look at the "model providers" issue and don't file a new one here. (Not applicable: the bug is in the input reader, before any model call.)
- I believe this is a legitimate bug, not just a question. If this is a question, please use the Discussions area.
Describe the bug
JSONLinesFileReader.read_file splits the file with text.splitlines() (jsonl.py#L38). Python's str.splitlines() also breaks on U+2028 (LINE SEPARATOR), U+2029 (PARAGRAPH SEPARATOR) and U+0085 (NEXT LINE). JSON Lines uses only \n as the record separator, and those three characters are valid unescaped inside JSON strings. A valid row containing one of them is therefore cut into two fragments, neither of which parses. Each fragment is logged as Skipping malformed JSONL row in <path> at line N, so that row is dropped from the input (the rest of the file is still read), and the only trace is a warning.
(Side note: splitlines() also splits on \x0b, \x0c, \x1c, \x1d and \x1e. Those are control characters that must be escaped in JSON, so they cannot occur unescaped in a valid row and are not part of this bug.)
Steps to reproduce
File x.jsonl, where the first row contains a raw U+2028 between a and b (shown here as <LS>):
{"text":"a<LS>b"}
{"text":"c"}
docs = await JSONLinesFileReader(storage=storage).read_files()
print([d.text for d in docs]) # ['c'] (expected ['a
b', 'c'])
# log: Skipping malformed JSONL row in x.jsonl at line 1
# Skipping malformed JSONL row in x.jsonl at line 2
json.loads('{"text":"a
b"}') parses fine on its own. The same happens with U+2029 and U+0085.
Expected Behavior
Both rows are read. Split on "\n" only and strip a trailing "\r" for CRLF files.
GraphRAG Config Used
input:
type: jsonl
Logs and screenshots
Skipping malformed JSONL row in x.jsonl at line 1
Skipping malformed JSONL row in x.jsonl at line 2
Additional Information
- GraphRAG Version: 3.3.0 (graphrag-input at 5faaaf4f)
- Operating System: macOS
- Python Version: 3.11
- Related Issues: #2424 / PR #2434 (JSONL reader skipped the whole file on blank lines). That is a different bug in the same function; the
splitlines()loop appears to have come in with that fix.
Suggested fix: for line_number, line in enumerate(text.split("\n"), start=1): line = line.rstrip("\r"), plus a test with U+2028 and U+0085 inside a string value.
- 主要语言
- Python
- 星标
- 36.6k
- 派生
- 3.9k
- 平均合并
- 4 天 11 小时
- 30 天内合并 PR
- 1
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
microsoft/graphrag 的其他 Issue
-
难度 2/5 1-3 小时 新手友好度 74/100
microsoft/graphrag#2626 · 1 条评论 ·
维护者通常 5 天内回复
-
[Bug]: response_format_json_object=True is silently dropped, so JSON mode never reaches the model未关闭bug triage
难度 2/5 1-3 小时 新手友好度 72/100
microsoft/graphrag#2625 · 1 条评论 ·
维护者通常 5 天内回复
-
难度 1/5 1 小时以内 新手友好度 95/100
维护者通常 5 天内回复
-
[Bug]: JsonCache.get() crashes with unhandled AttributeError on non-dict or invalid envelope JSON payloads可能重新可做 关联的 PR 已关闭且未合并。 未关闭
难度 2/5 1-3 小时 新手友好度 82/100
microsoft/graphrag#2614 · 2 条评论 ·
维护者通常 5 天内回复
-
[Bug]: TokenChunker never terminates when chunking overlap >= size (no validation of size/overlap)未关闭
难度 2/5 1-3 小时 新手友好度 78/100
microsoft/graphrag#2593 · 2 条评论 ·
维护者通常 5 天内回复
查看 microsoft/graphrag 的全部 Issue
相似的 Issue
-
bug
难度 1/5 1 小时以内 新手友好度 78/100
splunk/security_content#4418 ·
维护者通常 1 天内回复
-
[Bug / Enhancement]: Improve teleop_robot CLI with configurable timeout, robust input validation, and clean ROS shutdown可能已有人在做 @muskan-creates352 今天认领。 未关闭
难度 2/5 1-3 小时 新手友好度 72/100
维护者通常 3 天内回复
-
bug(pipelines): shortcode stripping deletes text between a {{< >}} shortcode and the next {{% %}}可能已有人在做 @Ashborne-lab 今天认领。 未关闭
难度 2/5 1-3 小时 新手友好度 70/100
kubeflow/docs-agent#293 ·
维护者通常 3 天内回复
-
难度 2/5 1-3 小时 新手友好度 62/100
Imbad0202/academic-research-skills#1093 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 78/100
pierreaubert/gpui-toolkit#114 ·
维护者通常 1 天内回复