[Bug]: Incremental knowledge-graph compile cites far fewer new sources than a full compile of the same files (Shuffle history recall capped at 4, Reduce related_outputs [:12])

Aperta
  • Qualcosa non va?
#7,299 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
45/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python
Ambito
data-engineering

Direzione di ricerca

Read bot/vikingbot/compile/ops/shuffle.py and bot/vikingbot/compile/ops/reduce.py, starting with the cited history and related-output limits and the Reduce prompt. Then inspect how incremental runs record source coverage in pipeline/coverage.json. Done means incremental knowledge-graph runs either achieve comparable coverage or report when coverage falls short; the issue offers a reproducible 41-file comparison.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Version
  • OpenViking core v0.5.3 (178a8cdfb)
  • Compile: bot/vikingbot/compile from main 4b92b30656c821712e2cf5f053963dedc66206d9 (the limits below are unchanged on main 3fab17054)
  • LiteLLM 1.102.4 (OpenViking's pin)
Setup

Directory compile with the bundled knowledge-graph Skill, routing mode all, one Reduce. Incremental mode: the new batch in full_dirs, earlier batches (already compiled into the same target) in diff_dirs. Inputs are plain-text documents, 25 to 45 files per batch.

Minimal reproduction
  1. Compile batch A into a knowledge-graph target in full mode.
  2. Add batch B (41 files) incrementally: B in full_dirs, A in diff_dirs, same target.
  3. Count how many of B's 41 files are cited by any entity or edge in the resulting graph.
  4. For comparison, compile B alone into an empty target in full mode and count again.
Observed
Run Map records Reduce output B originals cited
Incremental (B added to A) 136 5 files (4 new entities and one relations.jsonl patch) 8 of 41
Full mode, B alone n/a 54 entities, 78 edges 31 of 41

The task completed successfully. The stage review itself reported that integration "cannot be confirmed", but nothing failed or flagged the coverage drop.

Expected

An incremental compile should integrate the new sources at roughly the coverage a full compile achieves on the same files, or report the shortfall (for example as incomplete coverage) instead of completing silently.

Likely causes (at 4b92b3065)
  • bot/vikingbot/compile/ops/shuffle.py, candidates(): history recall is r.client.find(text, target_uri=r.target, limit=4) (line 290), the text-search fallback uses grep(..., node_limit=4) (line 301), and the results are then truncated with entries[:4]. When a match is a directory, its children are listed with node_limit=4 (line 332). Each record therefore sees at most 4 historical matches, so a new source that relates to more than a few existing entities is routed against a small slice of the graph.
  • bot/vikingbot/compile/ops/reduce.py (lines 262 to 272): related_outputs is sorted by path (not relevance) and truncated to [:12]; related_subjects is also truncated to [:12] in record order.
  • The Reduce prompt states "References establish provenance, not semantic completeness" (reduce.py line 37), and nothing requires the output to cover every input, so Reduce can fold 136 records into a handful of edits and still pass.
Workaround

We stopped using incremental mode for knowledge graphs and compile one full-mode graph per batch, then check source coverage externally.

Ask
  1. Is incremental compile intended to keep source coverage comparable to a full compile? If yes:
  2. Make the Shuffle history recall limit (and the node_limit) and the Reduce related-output cap configurable per Skill or target, or raise them for knowledge-graph targets.
  3. Add a coverage requirement or check for incremental runs (for example, fail or mark incomplete when the share of new inputs cited falls below a threshold), using the data already in pipeline/coverage.json.

Happy to run a patched build against the same batch and report the numbers.

Lingua principale
Python
Stelle
41.8k
Fork
3.4k
Merge medio
19h 29m
PR unite (30g)
500

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di volcengine/OpenViking

Tutte le issue di volcengine/OpenViking

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.