[Bug]: Compile commits partial output after a job fails and reports the task completed

Aperta
  • Qualcosa non va?
#7,303 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
48/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python
Ambito
api, backend

Direzione di ricerca

Start with bot/vikingbot/compile/ops/map.py, ops/reduce.py, pipeline.py, and service.py, tracing how recorded job failures reach finalization and determine the task status. Check the existing tests for partial output and task status, then add coverage for the requested behavior: a task with failed jobs should report incomplete or failed, or support disabling salvage. Done means partial graph output cannot be reported as fully completed, with regression tests covering the chosen behavior.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Version

  • OpenViking core v0.5.3 (178a8cdfb)
  • Compile (bot/vikingbot/compile) from main 4b92b3065; the same behavior is still on current main (3f6aa65ac), with service.py line numbers shifted by about four
  • LiteLLM 1.102.4 (the current pin)

Summary

When a pipeline job fails (for example a Reduce job that still gets 429 after its provider and job retries), Compile finalizes whatever artifacts remain, commits them to the target, and marks the task completed. The only signals are a warning string, pipeline/coverage.json, and prepared: false in the pipeline summary. For a knowledge-graph target, the result is a partial graph that a client can't tell apart from a complete one unless it reads the coverage file.

Reproduction

  1. Run a directory compile with the knowledge-graph Skill over 25 to 45 source files into an empty target.
  2. Force one Map or Reduce job to fail partway through. A model rate limit that outlasts the provider retries and the job-level retries (MAX_JOB_ATTEMPTS) does this reliably.
  3. Poll the task until it finishes.

Observed: task status: completed, error: null. result.warnings includes a Partial output; ... warning, and the target holds entity pages and edges from only the jobs that finished. pipeline/coverage.json lists the missing sources as failed or pending.

Expected: a failed job gives a failed or incomplete task, or at least a way to stop partial output from being published to that target.

Code path (at 4b92b3065)

There are two routes to the same result.

Job failures inside Map or Reduce:

  • bot/vikingbot/compile/ops/reduce.py:85-91 and ops/map.py:148-154 call bounded_jobs(..., failures=runtime.failures), so a failed job is recorded in pipeline.failures instead of raising.
  • bot/vikingbot/compile/pipeline.py:251-259: the finalize node still runs on the remaining inputs and adds Partial output; unfinished jobs: .... The summary records prepared: false.

Exceptions that escape the pipeline (timeouts, shuffle or stage errors):

  • bot/vikingbot/compile/pipeline.py:319-324: any non-CompileFailure exception becomes CompileFailure("COMPILE_INCOMPLETE", ...).
  • bot/vikingbot/compile/service.py:1169-1176: except (CompileFailure, TimeoutError) runs finalize_op on pipeline.artifacts whenever any artifacts exist and adds Partial output; Compile did not finish: .... The only failure code that skips this salvage is SKILL_REPAIR_FAILED.

Both routes end at:

  • bot/vikingbot/compile/service.py:1242: status is completed whenever anything was published, whether or not pipeline.failures is empty.
  • bot/vikingbot/compile/models.py:49: salvage_grace_seconds is the only salvage setting. Partial publication can't be turned off per target or per Skill.

Salvage was added in #3948 for Resource targets that hit the runtime deadline or the iteration limit. That makes sense for wiki pages. A graph is different: missing sources quietly drop entities and relations, and later incremental compiles build on that partial state.

Impact

Over a run of 105 input batches, two batches committed partial graphs this way while the task reported completed. Our external coverage check caught both before anything used them.

Workaround

We compile each batch into its own target. After the task finishes, we read pipeline/coverage.json and keep the batch only if every input is delivered. Otherwise we discard the target and redo the batch in a fresh target.

Request

Either of these would fix it:

  1. A per-request or per-Skill option (for example salvage: false) that makes any job failure fail the task without committing, or
  2. Report the task as incomplete or failed (with the partial result still attached) whenever pipeline.failures is non-empty, so completed always means full coverage.

We're happy to test a patch against our workload.

Lingua principale
Python
Stelle
41.8k
Fork
3.4k
Merge medio
19h 29m
PR unite (30g)
500

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di volcengine/OpenViking

Tutte le issue di volcengine/OpenViking

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.