[Bug]: TextChunk.start_char/end_char are wrong for overlapping token chunks (and for SentenceChunker in some cases)

Aperta
  • Qualcosa non va?
#2,595 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 5 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
3/5
Tempo stimato
1-2 giorni
Idoneità per principianti
52/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python
Ambito
data

Direzione di ricerca

Start in packages/graphrag-chunking: create_chunk_results.py assigns start_char from the previous end_char, which is wrong for TokenChunker's overlapping windows in token_chunker.py. The SentenceChunker correction in sentence_chunker.py uses text.find on the transformed text. Run the TokenChunker snippet from the issue and check that text[start_char:end_char+1] equals original. Done means the offsets match for overlapping and transformed chunks, or the docstring in text_chunk.py states they only hold for non-overlapping chunks. The maintainers still need to choose between the two.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Do you need to file an issue?
  • I have searched the existing issues and this bug is not already filed.
  • My model is hosted on OpenAI or Azure. If not, please look at the "model providers" issue and don't file a new one here. (Not applicable: the bug is in chunking, before any model call.)
  • I believe this is a legitimate bug, not just a question. If this is a question, please use the Discussions area.
Describe the bug

TextChunk.start_char / end_char are documented as the position of the raw chunk text in the source document (text_chunk.py). Severity is low: nothing else in packages/graphrag reads these fields (I grepped for start_char / end_char outside graphrag-chunking), so current indexing output is not affected. It is a wrong value in a public data class.

  1. TokenChunker with overlap (the case I ran). create_chunk_results (create_chunk_results.py#L18-L31) sets each start_char to the previous end_char + 1, i.e. it assumes the chunks are consecutive and non-overlapping. TokenChunker passes it overlapping windows (token_chunker.py#L42, default overlap 100), so every chunk after the first drifts by about overlap * index and end_char can run past the end of the text.

  2. SentenceChunker with a transform (checked with a stubbed splitter only). SentenceChunker.chunk (sentence_chunker.py#L35-L47) corrects offsets with text.find(result.text, start). result.text is the transformed text (e.g. with prepend_metadata), so find returns -1, delta is negative, and the delta > 0 guard silently skips the correction. Using result.original would match the source.

  3. SentenceChunker with repeated sentences (weakest of the three). The correction only bumps results[index + 1], not later chunks, so for repeated sentences separated by wide whitespace find can match an earlier occurrence.

I did not have nltk installed, so cases 2 and 3 were checked by reading the code and running it with a stubbed sent_tokenize, not against the real tokenizer. Case 1 was run as shown below.

Steps to reproduce
from graphrag_chunking.token_chunker import TokenChunker

enc = lambda s: [ord(c) for c in s]
dec = lambda t: "".join(map(chr, t))
text = "abcdefghijklmnopqrstuvwxyz"
for c in TokenChunker(size=10, overlap=4, encode=enc, decode=dec).chunk(text):
    print(c.start_char, c.end_char, repr(text[c.start_char:c.end_char + 1]), repr(c.original))

Output:

0 9 'abcdefghij' 'abcdefghij'
10 19 'klmnopqrst' 'ghijklmnop'     <- true span is 6..15
20 29 'uvwxyz' 'mnopqrstuv'          <- true span is 12..21
30 37 '' 'stuvwxyz'                  <- start is past the end of the text

For case 2, with the splitter stubbed: "One. Two. Three." with add_metadata({"t": "x"}) gives 'Two.' the span (4, 7), which slices ' Two'. For case 3, "Go. Go. Go. Go." gives spans (0,2), (6,8), (12,14), (12,14); the 4th duplicates the 3rd and its correct start is 18.

Expected Behavior

text[chunk.start_char : chunk.end_char + 1] == chunk.original when the chunker does not strip or alter the text.

GraphRAG Config Used
chunking:
  type: tokens
  size: 1200
  overlap: 100

(Library-level behaviour; the repro calls the chunker directly with small values.)

Logs and screenshots

No errors; the offsets are just wrong.

Additional Information
  • GraphRAG Version: 3.3.0 (graphrag-chunking at 5faaaf4f)
  • Operating System: macOS
  • Python Version: 3.11
  • Related Issues: #2589 (another TokenChunker defect, about decoding at token boundaries)

Possible fix: have split_text_on_tokens also return the real character offset of each window (e.g. the length of decode(tokens[:start_idx])), and in SentenceChunker use a running cursor with text.find(result.original, cursor). If the maintainers consider the offsets unused and not worth fixing, documenting that they are only valid for non-overlapping chunks would also resolve this.

Lingua principale
Python
Stelle
36.6k
Fork
3.9k
Merge medio
4g 11h
PR unite (30g)
1

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di microsoft/graphrag

Tutte le issue di microsoft/graphrag

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.