[Bug]: TokenChunker cuts multi-token characters at chunk boundaries, leaving U+FFFD in text units

オープン
  • 何か問題がありますか？
#2,589 コメント 1 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 5 日以内に返信

まだ誰も着手していません。

評価

難易度
3/5
見積もり時間
1〜2日
初心者へのやさしさ
67/100
issue の種類
バグ
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
python
領域
data-engineering

調査の方向性

Read packages/graphrag-chunking/graphrag_chunking/token_chunker.py, especially split_text_on_tokens, and examine how its token windows are decoded. Add regression tests for the provided multilingual example, checking that chunks contain no unexpected U+FFFD and join back to the input when overlap is zero. The issue suggests keeping the existing encode/decode interface; done means boundaries preserve characters.

索引モデルが issue の本文から書いたものです。

説明

Do you need to file an issue?
  • I have searched the existing issues and this bug is not already filed.
  • My model is hosted on OpenAI or Azure. If not, please look at the "model providers" issue and don't file a new one here. (Not applicable: the bug is in chunking, before any model call.)
  • I believe this is a legitimate bug, not just a question. If this is a question, please use the Discussions area.
Describe the bug

split_text_on_tokens in packages/graphrag-chunking/graphrag_chunking/token_chunker.py decodes every window of token ids on its own:

chunk_text = decode(list(chunk_tokens))

When a character is encoded as more than one token, a window boundary can fall inside it. tiktoken's decode turns the incomplete UTF-8 sequence into U+FFFD, so the chunk ends with � and the next one starts with �. With overlap: 0 that character is in no chunk at all. With an overlap it survives in the overlapping chunk, but every affected text unit still carries one or two � into extraction and embedding.

With the default o200k_base this hits emoji, rarer Han characters, Tibetan and Amharic, among others. In short samples, 5 of the 14 token boundaries in a string of ten emoji fall inside a character, and 20 of the 42 in a Tibetan phrase. With cl100k_base, common Chinese characters are affected too. That is what #596 ran into ("官方用的分块是先把文档token化，按照token数进行切分，对于中文来说容易出现乱码"). That issue was closed on the extraction question, and the chunker is unchanged since.

Steps to reproduce

graphrag-chunking 3.2.0, tiktoken 0.14:

import tiktoken
from graphrag_chunking.token_chunker import split_text_on_tokens

enc = tiktoken.get_encoding("o200k_base")
text = "关系。龘靐齉爩 是很少见的汉字。" * 3
chunks = split_text_on_tokens(text, chunk_size=7, chunk_overlap=0, encode=enc.encode, decode=enc.decode)
print([c for c in chunks if "�" in c])
print("".join(chunks) == text)

['关系。龘靐�', '�爩 是很少见', '的汉字。关系。�', '�靐齉爩']
False

One 龘 and one 齉 are missing from the joined chunks.

Expected Behavior

A chunk boundary falls between characters, so no chunk contains U+FFFD unless the input does, and with overlap: 0 the chunks join back into the input.

GraphRAG Config Used
chunking:
  type: tokens
  size: 1200
  overlap: 100
  encoding_model: o200k_base

(the defaults; the repro above calls the function directly with a small size)

Logs and screenshots

No response

Additional Information
  • GraphRAG Version: graphrag-chunking 3.2.0 (main has the same split_text_on_tokens)
  • Operating System: macOS
  • Python Version: 3.12
  • Related Issues: #596

A fix can stay inside split_text_on_tokens and keep its encode/decode interface: before cutting at token index b, check that the cut does not split a character, for example decode(tokens[b-4:b]) + decode(tokens[b:b+4]) == decode(tokens[b-4:b+4]) (a UTF-8 character spans at most four bytes), and move b back one token while it does. I can open a PR with that and a test if the approach is fine with you.

主要言語
Python
スター
36.6k
フォーク
3.9k
平均マージ
4日 11時間
マージ済み PR（30日）
1

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

microsoft/graphrag のほかの issue

microsoft/graphrag の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。