Gleaning responses are concatenated without a record delimiter, silently dropping records at the join

Aperta
  • Qualcosa non va?
#2,605 1 commento 1 reazione 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 5 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
3/5
Tempo stimato
Mezza giornata
Idoneità per principianti
50/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
python
Ambito
data

Direzione di ricerca

Start in packages/graphrag/graphrag/index/operations/extract_graph/graph_extractor.py: _process_document joins gleaning responses with results += response_text (around line 109), and _process_result splits on ## and handles the COMPLETION_DELIMITER guard. Run the reproduction in the issue body, which calls _process_result directly with no LLM. Done when the joined case keeps ALICE CHEN and the relationship weight parses as 9 instead of 1.0.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Do you need to file an issue?

  • I have searched the existing issues and this bug is not already filed. (Closed #1183 covered part of it and was closed as not planned; see Additional Information.)

  • My model is hosted on OpenAI or Azure. If not, please look at the "model providers" section of the Get Started guide.

    Noting this one honestly: I found this against a self-hosted model, but the bug is in string handling after the response arrives and is model-independent. The repro below calls no model at all.

  • I believe this is a legitimate bug, not just a question. If this is a question, please use the Discussions area.

Describe the bug

GraphExtractor._process_document concatenates each gleaning response directly onto the previous one:

https://github.com/microsoft/graphrag/blob/main/packages/graphrag/graphrag/index/operations/extract_graph/graph_extractor.py#L109

results += response_text

No record delimiter is inserted. The shipped extraction prompt never asks the model to terminate its final record, and its own example shows the last record ending with ) followed by <|COMPLETE|> on the next line:

("relationship"<|>MARTIN SMITH<|>CENTRAL INSTITUTION<|>Martin Smith is the Chair of the Central Institution<|>9)
<|COMPLETE|>

So when _process_result splits on ##, the previous response's final record and the next response's first record are one record.

The fused record is not rejected. It parses as the first record with extra tuple fields appended, and _process_result reads only fields [1], [2], [3], so the second record's fields are silently discarded:

['"entity"', 'CENTRAL INSTITUTION', 'ORGANIZATION', 'The Central Institution ...)\n<|COMPLETE|>\n("entity"', 'ALICE CHEN', 'PERSON', 'Alice Chen is ...']
   [0]              [1]                  [2]                      [3]                                          [4]           [5]          [6]

Nothing is logged. The entity or relationship the gleaning was called to recover is simply absent from the graph. max_gleanings defaults to 1, so a default configuration can lose up to one record per text unit.

The same prompt shape causes a second, independent problem with no gleanings involved. Because the final record ends ...<|>9)\n<|COMPLETE|>, the string does not end in ), so re.sub(r"^\(|\)$", "", record) cannot strip the closing paren. record_attributes[-1] is then 9)\n<|COMPLETE|>, float() raises ValueError, and the except ValueError branch sets weight = 1.0. Every text unit's last relationship gets weight 1.0 regardless of what the model scored it, which feeds into community detection and ranking.

The record == COMPLETION_DELIMITER guard on line 139 only fires when the delimiter is on its own record, which happens only if the model terminates its last record first.

Steps to reproduce

Run extraction with max_gleanings: 1 (the default) and a model that follows the prompt's example formatting. Entities that appear only in a gleaning continuation go missing, and the last relationship of each text unit has weight 1.0.

Both halves reproduce without an LLM call, using the library's own parser and two responses shaped exactly like the prompt's Example 1 output:

from graphrag.index.operations.extract_graph.graph_extractor import GraphExtractor

BASE = (
    '("entity"<|>CENTRAL INSTITUTION<|>ORGANIZATION<|>The Central Institution is the Federal Reserve of Verdantis)\n'
    "##\n"
    '("relationship"<|>MARTIN SMITH<|>CENTRAL INSTITUTION<|>Martin Smith is the Chair of the Central Institution<|>9)\n'
    "<|COMPLETE|>"
)
GLEANING = (
    '("entity"<|>ALICE CHEN<|>PERSON<|>Alice Chen is a policy analyst at the Central Institution)\n'
    "##\n"
    '("relationship"<|>ALICE CHEN<|>CENTRAL INSTITUTION<|>Alice Chen analyses policy<|>7)\n'
    "<|COMPLETE|>"
)
ge = GraphExtractor.__new__(GraphExtractor)

# (a) weight corruption, no gleanings involved
_, r = ge._process_result(BASE, "tu1", "<|>", "##")
print(r.iloc[0]["weight"])                      # 1.0, model said 9

# (b) what _process_document returns today for one gleaning
e, _ = ge._process_result(BASE + GLEANING, "tu1", "<|>", "##")
print(sorted(e["title"]))                       # ['CENTRAL INSTITUTION'] -- ALICE CHEN is gone

# (c) the same two responses joined by a record delimiter
e, _ = ge._process_result(BASE + "\n##\n" + GLEANING, "tu1", "<|>", "##")
print(sorted(e["title"]))                       # ['ALICE CHEN', 'CENTRAL INSTITUTION']

The difference between (b) and (c) is the whole of the first bug: the two responses carry the same records, and only the join decides whether one of them survives.

Expected Behavior
  • An entity or relationship emitted by a gleaning continuation appears in the extracted graph.
  • A relationship's weight is the value the model emitted, not the 1.0 fallback.
GraphRAG Config Used
extract_graph:
  model_id: default_chat_model
  prompt: "prompts/extract_graph.txt"
  entity_types: [organization, person, geo, event]
  max_gleanings: 1
Logs and screenshots

No logs are produced. That is the core of the report: both failures are silent. The fused record satisfies record_attributes[0] == '"entity"' and len(record_attributes) >= 4, so it is accepted, and the weight failure is swallowed by except ValueError: weight = 1.0.

Additional Information
  • GraphRAG Version: main @ v3.3.0
  • Operating System: Linux / macOS
  • Python Version: 3.12
  • Related Issues: #1183 reported the concatenation half of this in September 2024 with the correct fix, and was closed as not planned in August 2025 with no discussion. The completion-delimiter half of the problem was not covered there.

I have a fix ready with a regression test and will open a PR referencing this issue.

Lingua principale
Python
Stelle
36.6k
Fork
3.9k
Merge medio
4g 11h
PR unite (30g)
1

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di microsoft/graphrag

Tutte le issue di microsoft/graphrag

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.