replace_source retires a source's facts when one LLM call fails

Aperta
#356 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
55/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
python, typescript
Ambito
ai, databases

Direzione di ricerca

The issue identifies Insert.text, _apply_facts, and _ops.replace_source as the relevant entry points; inspect how extraction partial failures are represented and handled in both SDKs. Reproduce the supplied Kuzu scenario and trace how failed calls affect retirement. Done when a failed LLM call during replacement does not silently retire facts, with the behavior covered in both SDKs.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

area: extraction area: write bug effort: M P1

What happens

insert.text(..., source_id=..., replace_source=True) retires every relation the earlier versions of the source stated and the new text "no longer yields". When one LLM call fails on a chunk (a timeout, a rate limit, a provider error), extraction carries on with an extraction_partial_failure warning, and the relations that call would have returned count as no longer stated. They are retired with fact_retired_by_source.

So a transient provider error during a re-ingest silently removes valid facts from the current graph. Both SDKs do this.

Reproduce

On main (5a277c9), embedded Kuzu:

import asyncio, tempfile
from selfgraph import KuzuStore, SelfGraph

REPLY = {
    "entities": [{"type": "Person", "name": "Ann"}, {"type": "Company", "name": "Acme"}],
    "relations": [{"from_name": "Ann", "type": "WORKS_AT", "to_name": "Acme"}],
}

class LLM:
    fail = False

    async def generate(self, *, system, user, output):
        kind = next(iter(output.model_fields))
        if self.fail and kind == "relations":
            raise TimeoutError("provider timed out")
        return {kind: REPLY.get(kind, [])}

async def main():
    with tempfile.TemporaryDirectory() as tmp:
        llm = LLM()
        async with SelfGraph(store=KuzuStore(f"{tmp}/g.kuzu"), llm=llm) as g:
            await g.insert.text("Ann works at Acme.", source_id="doc", replace_source=True)
            llm.fail = True
            r = await g.insert.text("Ann works at Acme. She likes it.", source_id="doc", replace_source=True)
            print([w.code for w in r.warnings], r.source_diff)
            print([(x.type, x.is_current) for x in (await g.relations.list(include_invalidated=True)).items])

asyncio.run(main())
['extraction_partial_failure', 'fact_retired_by_source'] kept=[] added=[] retired=['relation:a4b698967fceb2c72cc81f83']
[('WORKS_AT', False)]

The TypeScript SDK gives the same warnings and source_diff for the same client.

Cause

Insert.text passes the extraction to _apply_facts with replace_source unchanged, and _ops.replace_source retires prior - produced without knowing that some LLM calls failed. When extraction_partial_failure comes from a failed call (not from dropping one invalid item), replacing the source should either fail or skip retirement for the affected chunks.

Lingua principale
TypeScript
Stelle
173
Fork
58
Merge medio
6h
PR unite (30g)
120

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di hydra-db/selfgraph

Tutte le issue di hydra-db/selfgraph

Issue simili

Altre issue su TypeScript

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.