[Bug]: TokenChunker cuts multi-token characters at chunk boundaries, leaving U+FFFD in text units

Đang mở
  • Có gì không ổn?
#2,589 1 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 5 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
3/5
Thời gian dự kiến
1-2 ngày
Mức phù hợp với người mới
67/100
Loại issue
Lỗi
Độ rõ ràng
Đặc tả rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
python
Lĩnh vực
data-engineering

Hướng nghiên cứu

Read packages/graphrag-chunking/graphrag_chunking/token_chunker.py, especially split_text_on_tokens, and examine how its token windows are decoded. Add regression tests for the provided multilingual example, checking that chunks contain no unexpected U+FFFD and join back to the input when overlap is zero. The issue suggests keeping the existing encode/decode interface; done means boundaries preserve characters.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

Do you need to file an issue?
  • I have searched the existing issues and this bug is not already filed.
  • My model is hosted on OpenAI or Azure. If not, please look at the "model providers" issue and don't file a new one here. (Not applicable: the bug is in chunking, before any model call.)
  • I believe this is a legitimate bug, not just a question. If this is a question, please use the Discussions area.
Describe the bug

split_text_on_tokens in packages/graphrag-chunking/graphrag_chunking/token_chunker.py decodes every window of token ids on its own:

chunk_text = decode(list(chunk_tokens))

When a character is encoded as more than one token, a window boundary can fall inside it. tiktoken's decode turns the incomplete UTF-8 sequence into U+FFFD, so the chunk ends with � and the next one starts with �. With overlap: 0 that character is in no chunk at all. With an overlap it survives in the overlapping chunk, but every affected text unit still carries one or two � into extraction and embedding.

With the default o200k_base this hits emoji, rarer Han characters, Tibetan and Amharic, among others. In short samples, 5 of the 14 token boundaries in a string of ten emoji fall inside a character, and 20 of the 42 in a Tibetan phrase. With cl100k_base, common Chinese characters are affected too. That is what #596 ran into ("官方用的分块是先把文档token化，按照token数进行切分，对于中文来说容易出现乱码"). That issue was closed on the extraction question, and the chunker is unchanged since.

Steps to reproduce

graphrag-chunking 3.2.0, tiktoken 0.14:

import tiktoken
from graphrag_chunking.token_chunker import split_text_on_tokens

enc = tiktoken.get_encoding("o200k_base")
text = "关系。龘靐齉爩 是很少见的汉字。" * 3
chunks = split_text_on_tokens(text, chunk_size=7, chunk_overlap=0, encode=enc.encode, decode=enc.decode)
print([c for c in chunks if "�" in c])
print("".join(chunks) == text)

['关系。龘靐�', '�爩 是很少见', '的汉字。关系。�', '�靐齉爩']
False

One 龘 and one 齉 are missing from the joined chunks.

Expected Behavior

A chunk boundary falls between characters, so no chunk contains U+FFFD unless the input does, and with overlap: 0 the chunks join back into the input.

GraphRAG Config Used
chunking:
  type: tokens
  size: 1200
  overlap: 100
  encoding_model: o200k_base

(the defaults; the repro above calls the function directly with a small size)

Logs and screenshots

No response

Additional Information
  • GraphRAG Version: graphrag-chunking 3.2.0 (main has the same split_text_on_tokens)
  • Operating System: macOS
  • Python Version: 3.12
  • Related Issues: #596

A fix can stay inside split_text_on_tokens and keep its encode/decode interface: before cutting at token index b, check that the cut does not split a character, for example decode(tokens[b-4:b]) + decode(tokens[b:b+4]) == decode(tokens[b-4:b+4]) (a UTF-8 character spans at most four bytes), and move b back one token while it does. I can open a PR with that and a test if the approach is fine with you.

Ngôn ngữ chính
Python
Star
36.6k
Fork
3.9k
Merge trung bình
4 ngày 11 giờ
Pull request đã merge (30 ngày)
1

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của microsoft/graphrag

Tất cả issue của microsoft/graphrag

Issue tương tự

Thêm issue về Python

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.