Gleaning responses are concatenated without a record delimiter, silently dropping records at the join

Abierto
  • ¿Algo no está bien?
#2,605 1 comentario 1 reacción 0 asignados Ver en GitHub

Los mantenedores suelen responder en 5 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
3/5
Tiempo estimado
Medio día
Aptitud para principiantes
50/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
python
Área
data

Línea de trabajo

Start in packages/graphrag/graphrag/index/operations/extract_graph/graph_extractor.py: _process_document joins gleaning responses with results += response_text (around line 109), and _process_result splits on ## and handles the COMPLETION_DELIMITER guard. Run the reproduction in the issue body, which calls _process_result directly with no LLM. Done when the joined case keeps ALICE CHEN and the relationship weight parses as 9 instead of 1.0.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Do you need to file an issue?

  • I have searched the existing issues and this bug is not already filed. (Closed #1183 covered part of it and was closed as not planned; see Additional Information.)

  • My model is hosted on OpenAI or Azure. If not, please look at the "model providers" section of the Get Started guide.

    Noting this one honestly: I found this against a self-hosted model, but the bug is in string handling after the response arrives and is model-independent. The repro below calls no model at all.

  • I believe this is a legitimate bug, not just a question. If this is a question, please use the Discussions area.

Describe the bug

GraphExtractor._process_document concatenates each gleaning response directly onto the previous one:

https://github.com/microsoft/graphrag/blob/main/packages/graphrag/graphrag/index/operations/extract_graph/graph_extractor.py#L109

results += response_text

No record delimiter is inserted. The shipped extraction prompt never asks the model to terminate its final record, and its own example shows the last record ending with ) followed by <|COMPLETE|> on the next line:

("relationship"<|>MARTIN SMITH<|>CENTRAL INSTITUTION<|>Martin Smith is the Chair of the Central Institution<|>9)
<|COMPLETE|>

So when _process_result splits on ##, the previous response's final record and the next response's first record are one record.

The fused record is not rejected. It parses as the first record with extra tuple fields appended, and _process_result reads only fields [1], [2], [3], so the second record's fields are silently discarded:

['"entity"', 'CENTRAL INSTITUTION', 'ORGANIZATION', 'The Central Institution ...)\n<|COMPLETE|>\n("entity"', 'ALICE CHEN', 'PERSON', 'Alice Chen is ...']
   [0]              [1]                  [2]                      [3]                                          [4]           [5]          [6]

Nothing is logged. The entity or relationship the gleaning was called to recover is simply absent from the graph. max_gleanings defaults to 1, so a default configuration can lose up to one record per text unit.

The same prompt shape causes a second, independent problem with no gleanings involved. Because the final record ends ...<|>9)\n<|COMPLETE|>, the string does not end in ), so re.sub(r"^\(|\)$", "", record) cannot strip the closing paren. record_attributes[-1] is then 9)\n<|COMPLETE|>, float() raises ValueError, and the except ValueError branch sets weight = 1.0. Every text unit's last relationship gets weight 1.0 regardless of what the model scored it, which feeds into community detection and ranking.

The record == COMPLETION_DELIMITER guard on line 139 only fires when the delimiter is on its own record, which happens only if the model terminates its last record first.

Steps to reproduce

Run extraction with max_gleanings: 1 (the default) and a model that follows the prompt's example formatting. Entities that appear only in a gleaning continuation go missing, and the last relationship of each text unit has weight 1.0.

Both halves reproduce without an LLM call, using the library's own parser and two responses shaped exactly like the prompt's Example 1 output:

from graphrag.index.operations.extract_graph.graph_extractor import GraphExtractor

BASE = (
    '("entity"<|>CENTRAL INSTITUTION<|>ORGANIZATION<|>The Central Institution is the Federal Reserve of Verdantis)\n'
    "##\n"
    '("relationship"<|>MARTIN SMITH<|>CENTRAL INSTITUTION<|>Martin Smith is the Chair of the Central Institution<|>9)\n'
    "<|COMPLETE|>"
)
GLEANING = (
    '("entity"<|>ALICE CHEN<|>PERSON<|>Alice Chen is a policy analyst at the Central Institution)\n'
    "##\n"
    '("relationship"<|>ALICE CHEN<|>CENTRAL INSTITUTION<|>Alice Chen analyses policy<|>7)\n'
    "<|COMPLETE|>"
)
ge = GraphExtractor.__new__(GraphExtractor)

# (a) weight corruption, no gleanings involved
_, r = ge._process_result(BASE, "tu1", "<|>", "##")
print(r.iloc[0]["weight"])                      # 1.0, model said 9

# (b) what _process_document returns today for one gleaning
e, _ = ge._process_result(BASE + GLEANING, "tu1", "<|>", "##")
print(sorted(e["title"]))                       # ['CENTRAL INSTITUTION'] -- ALICE CHEN is gone

# (c) the same two responses joined by a record delimiter
e, _ = ge._process_result(BASE + "\n##\n" + GLEANING, "tu1", "<|>", "##")
print(sorted(e["title"]))                       # ['ALICE CHEN', 'CENTRAL INSTITUTION']

The difference between (b) and (c) is the whole of the first bug: the two responses carry the same records, and only the join decides whether one of them survives.

Expected Behavior
  • An entity or relationship emitted by a gleaning continuation appears in the extracted graph.
  • A relationship's weight is the value the model emitted, not the 1.0 fallback.
GraphRAG Config Used
extract_graph:
  model_id: default_chat_model
  prompt: "prompts/extract_graph.txt"
  entity_types: [organization, person, geo, event]
  max_gleanings: 1
Logs and screenshots

No logs are produced. That is the core of the report: both failures are silent. The fused record satisfies record_attributes[0] == '"entity"' and len(record_attributes) >= 4, so it is accepted, and the weight failure is swallowed by except ValueError: weight = 1.0.

Additional Information
  • GraphRAG Version: main @ v3.3.0
  • Operating System: Linux / macOS
  • Python Version: 3.12
  • Related Issues: #1183 reported the concatenation half of this in September 2024 with the correct fix, and was closed as not planned in August 2025 with no discussion. The completion-delimiter half of the problem was not covered there.

I have a fix ready with a regression test and will open a PR referencing this issue.

Lenguaje dominante
Python
Estrellas
36.6k
Forks
3.9k
Merge medio
4 d 11 h
PR fusionados (30 d)
1

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de microsoft/graphrag

Todos los issues de microsoft/graphrag

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.