[Bug]: TokenChunker cuts multi-token characters at chunk boundaries, leaving U+FFFD in text units

未关闭
  • 哪里不对？
#2,589 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 5 天内回复

还没有人认领这个 Issue。

评估

难度
3/5
预计耗时
1-2 天
新手友好度
67/100
Issue 类型
缺陷
描述清晰度
描述清楚
活跃度
活跃
技术栈
python
领域
data-engineering

调研方向

Read packages/graphrag-chunking/graphrag_chunking/token_chunker.py, especially split_text_on_tokens, and examine how its token windows are decoded. Add regression tests for the provided multilingual example, checking that chunks contain no unexpected U+FFFD and join back to the input when overlap is zero. The issue suggests keeping the existing encode/decode interface; done means boundaries preserve characters.

由索引模型根据 Issue 内容生成。

描述

Do you need to file an issue?
  • I have searched the existing issues and this bug is not already filed.
  • My model is hosted on OpenAI or Azure. If not, please look at the "model providers" issue and don't file a new one here. (Not applicable: the bug is in chunking, before any model call.)
  • I believe this is a legitimate bug, not just a question. If this is a question, please use the Discussions area.
Describe the bug

split_text_on_tokens in packages/graphrag-chunking/graphrag_chunking/token_chunker.py decodes every window of token ids on its own:

chunk_text = decode(list(chunk_tokens))

When a character is encoded as more than one token, a window boundary can fall inside it. tiktoken's decode turns the incomplete UTF-8 sequence into U+FFFD, so the chunk ends with � and the next one starts with �. With overlap: 0 that character is in no chunk at all. With an overlap it survives in the overlapping chunk, but every affected text unit still carries one or two � into extraction and embedding.

With the default o200k_base this hits emoji, rarer Han characters, Tibetan and Amharic, among others. In short samples, 5 of the 14 token boundaries in a string of ten emoji fall inside a character, and 20 of the 42 in a Tibetan phrase. With cl100k_base, common Chinese characters are affected too. That is what #596 ran into ("官方用的分块是先把文档token化，按照token数进行切分，对于中文来说容易出现乱码"). That issue was closed on the extraction question, and the chunker is unchanged since.

Steps to reproduce

graphrag-chunking 3.2.0, tiktoken 0.14:

import tiktoken
from graphrag_chunking.token_chunker import split_text_on_tokens

enc = tiktoken.get_encoding("o200k_base")
text = "关系。龘靐齉爩 是很少见的汉字。" * 3
chunks = split_text_on_tokens(text, chunk_size=7, chunk_overlap=0, encode=enc.encode, decode=enc.decode)
print([c for c in chunks if "�" in c])
print("".join(chunks) == text)

['关系。龘靐�', '�爩 是很少见', '的汉字。关系。�', '�靐齉爩']
False

One 龘 and one 齉 are missing from the joined chunks.

Expected Behavior

A chunk boundary falls between characters, so no chunk contains U+FFFD unless the input does, and with overlap: 0 the chunks join back into the input.

GraphRAG Config Used
chunking:
  type: tokens
  size: 1200
  overlap: 100
  encoding_model: o200k_base

(the defaults; the repro above calls the function directly with a small size)

Logs and screenshots

No response

Additional Information
  • GraphRAG Version: graphrag-chunking 3.2.0 (main has the same split_text_on_tokens)
  • Operating System: macOS
  • Python Version: 3.12
  • Related Issues: #596

A fix can stay inside split_text_on_tokens and keep its encode/decode interface: before cutting at token index b, check that the cut does not split a character, for example decode(tokens[b-4:b]) + decode(tokens[b:b+4]) == decode(tokens[b-4:b+4]) (a UTF-8 character spans at most four bytes), and move b back one token while it does. I can open a PR with that and a test if the approach is fine with you.

主要语言
Python
星标
36.6k
派生
3.9k
平均合并
4 天 11 小时
30 天内合并 PR
1

环境准备

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

microsoft/graphrag 的其他 Issue

查看 microsoft/graphrag 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。