[Bug]: TextChunk.start_char/end_char are wrong for overlapping token chunks (and for SentenceChunker in some cases)

Abierto
  • ¿Algo no está bien?
#2,595 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 5 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
3/5
Tiempo estimado
1-2 días
Aptitud para principiantes
52/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
python
Área
data

Línea de trabajo

Start in packages/graphrag-chunking: create_chunk_results.py assigns start_char from the previous end_char, which is wrong for TokenChunker's overlapping windows in token_chunker.py. The SentenceChunker correction in sentence_chunker.py uses text.find on the transformed text. Run the TokenChunker snippet from the issue and check that text[start_char:end_char+1] equals original. Done means the offsets match for overlapping and transformed chunks, or the docstring in text_chunk.py states they only hold for non-overlapping chunks. The maintainers still need to choose between the two.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Do you need to file an issue?
  • I have searched the existing issues and this bug is not already filed.
  • My model is hosted on OpenAI or Azure. If not, please look at the "model providers" issue and don't file a new one here. (Not applicable: the bug is in chunking, before any model call.)
  • I believe this is a legitimate bug, not just a question. If this is a question, please use the Discussions area.
Describe the bug

TextChunk.start_char / end_char are documented as the position of the raw chunk text in the source document (text_chunk.py). Severity is low: nothing else in packages/graphrag reads these fields (I grepped for start_char / end_char outside graphrag-chunking), so current indexing output is not affected. It is a wrong value in a public data class.

  1. TokenChunker with overlap (the case I ran). create_chunk_results (create_chunk_results.py#L18-L31) sets each start_char to the previous end_char + 1, i.e. it assumes the chunks are consecutive and non-overlapping. TokenChunker passes it overlapping windows (token_chunker.py#L42, default overlap 100), so every chunk after the first drifts by about overlap * index and end_char can run past the end of the text.

  2. SentenceChunker with a transform (checked with a stubbed splitter only). SentenceChunker.chunk (sentence_chunker.py#L35-L47) corrects offsets with text.find(result.text, start). result.text is the transformed text (e.g. with prepend_metadata), so find returns -1, delta is negative, and the delta > 0 guard silently skips the correction. Using result.original would match the source.

  3. SentenceChunker with repeated sentences (weakest of the three). The correction only bumps results[index + 1], not later chunks, so for repeated sentences separated by wide whitespace find can match an earlier occurrence.

I did not have nltk installed, so cases 2 and 3 were checked by reading the code and running it with a stubbed sent_tokenize, not against the real tokenizer. Case 1 was run as shown below.

Steps to reproduce
from graphrag_chunking.token_chunker import TokenChunker

enc = lambda s: [ord(c) for c in s]
dec = lambda t: "".join(map(chr, t))
text = "abcdefghijklmnopqrstuvwxyz"
for c in TokenChunker(size=10, overlap=4, encode=enc, decode=dec).chunk(text):
    print(c.start_char, c.end_char, repr(text[c.start_char:c.end_char + 1]), repr(c.original))

Output:

0 9 'abcdefghij' 'abcdefghij'
10 19 'klmnopqrst' 'ghijklmnop'     <- true span is 6..15
20 29 'uvwxyz' 'mnopqrstuv'          <- true span is 12..21
30 37 '' 'stuvwxyz'                  <- start is past the end of the text

For case 2, with the splitter stubbed: "One. Two. Three." with add_metadata({"t": "x"}) gives 'Two.' the span (4, 7), which slices ' Two'. For case 3, "Go. Go. Go. Go." gives spans (0,2), (6,8), (12,14), (12,14); the 4th duplicates the 3rd and its correct start is 18.

Expected Behavior

text[chunk.start_char : chunk.end_char + 1] == chunk.original when the chunker does not strip or alter the text.

GraphRAG Config Used
chunking:
  type: tokens
  size: 1200
  overlap: 100

(Library-level behaviour; the repro calls the chunker directly with small values.)

Logs and screenshots

No errors; the offsets are just wrong.

Additional Information
  • GraphRAG Version: 3.3.0 (graphrag-chunking at 5faaaf4f)
  • Operating System: macOS
  • Python Version: 3.11
  • Related Issues: #2589 (another TokenChunker defect, about decoding at token boundaries)

Possible fix: have split_text_on_tokens also return the real character offset of each window (e.g. the length of decode(tokens[:start_idx])), and in SentenceChunker use a running cursor with text.find(result.original, cursor). If the maintainers consider the offsets unused and not worth fixing, documenting that they are only valid for non-overlapping chunks would also resolve this.

Lenguaje dominante
Python
Estrellas
36.6k
Forks
3.9k
Merge medio
4 d 11 h
PR fusionados (30 d)
1

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de microsoft/graphrag

Todos los issues de microsoft/graphrag

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.