[Bug]: TokenChunker cuts multi-token characters at chunk boundaries, leaving U+FFFD in text units

Aperta
  • Qualcosa non va?
#2,589 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 5 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
3/5
Tempo stimato
1-2 giorni
Idoneità per principianti
67/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
python
Ambito
data-engineering

Direzione di ricerca

Read packages/graphrag-chunking/graphrag_chunking/token_chunker.py, especially split_text_on_tokens, and examine how its token windows are decoded. Add regression tests for the provided multilingual example, checking that chunks contain no unexpected U+FFFD and join back to the input when overlap is zero. The issue suggests keeping the existing encode/decode interface; done means boundaries preserve characters.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Do you need to file an issue?
  • I have searched the existing issues and this bug is not already filed.
  • My model is hosted on OpenAI or Azure. If not, please look at the "model providers" issue and don't file a new one here. (Not applicable: the bug is in chunking, before any model call.)
  • I believe this is a legitimate bug, not just a question. If this is a question, please use the Discussions area.
Describe the bug

split_text_on_tokens in packages/graphrag-chunking/graphrag_chunking/token_chunker.py decodes every window of token ids on its own:

chunk_text = decode(list(chunk_tokens))

When a character is encoded as more than one token, a window boundary can fall inside it. tiktoken's decode turns the incomplete UTF-8 sequence into U+FFFD, so the chunk ends with � and the next one starts with �. With overlap: 0 that character is in no chunk at all. With an overlap it survives in the overlapping chunk, but every affected text unit still carries one or two � into extraction and embedding.

With the default o200k_base this hits emoji, rarer Han characters, Tibetan and Amharic, among others. In short samples, 5 of the 14 token boundaries in a string of ten emoji fall inside a character, and 20 of the 42 in a Tibetan phrase. With cl100k_base, common Chinese characters are affected too. That is what #596 ran into ("官方用的分块是先把文档token化，按照token数进行切分，对于中文来说容易出现乱码"). That issue was closed on the extraction question, and the chunker is unchanged since.

Steps to reproduce

graphrag-chunking 3.2.0, tiktoken 0.14:

import tiktoken
from graphrag_chunking.token_chunker import split_text_on_tokens

enc = tiktoken.get_encoding("o200k_base")
text = "关系。龘靐齉爩 是很少见的汉字。" * 3
chunks = split_text_on_tokens(text, chunk_size=7, chunk_overlap=0, encode=enc.encode, decode=enc.decode)
print([c for c in chunks if "�" in c])
print("".join(chunks) == text)

['关系。龘靐�', '�爩 是很少见', '的汉字。关系。�', '�靐齉爩']
False

One 龘 and one 齉 are missing from the joined chunks.

Expected Behavior

A chunk boundary falls between characters, so no chunk contains U+FFFD unless the input does, and with overlap: 0 the chunks join back into the input.

GraphRAG Config Used
chunking:
  type: tokens
  size: 1200
  overlap: 100
  encoding_model: o200k_base

(the defaults; the repro above calls the function directly with a small size)

Logs and screenshots

No response

Additional Information
  • GraphRAG Version: graphrag-chunking 3.2.0 (main has the same split_text_on_tokens)
  • Operating System: macOS
  • Python Version: 3.12
  • Related Issues: #596

A fix can stay inside split_text_on_tokens and keep its encode/decode interface: before cutting at token index b, check that the cut does not split a character, for example decode(tokens[b-4:b]) + decode(tokens[b:b+4]) == decode(tokens[b-4:b+4]) (a UTF-8 character spans at most four bytes), and move b back one token while it does. I can open a PR with that and a test if the approach is fine with you.

Lingua principale
Python
Stelle
36.6k
Fork
3.9k
Merge medio
4g 11h
PR unite (30g)
1

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di microsoft/graphrag

Tutte le issue di microsoft/graphrag

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.