FFmpeg: public result hides whether failure occurred before correctness checks

Aperta
  • Qualcosa non va?
#57 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

@loopx-agent ci sta già lavorando.

Dal 6/11/2026.

  • #58 di @loopx-agent — aperta

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
35/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Ferma
Stack tecnologico
docker, python
Ambito
testing-qa

Direzione di ricerca

Trace how the verifier writes reward.json.additional_data.reason, how the shim reads the reason, and how the task grader builds the failed overall-score summary. Record the failure stage at its producing branch and forward only a fixed public message into the existing summary or detail. Verify with same-host old and new runs for source rejection, build failure, constructor failure, and a correctness failure; confirm scores and validity remain unchanged.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Task: ffmpeg_swscale_reimplementation 1.0.0, source ba13e33842bd24e62fe0a361c5d41b2b21dd3010.

The public result loses the distinction between a source-policy rejection, a failed build, and an actual pixel-correctness failure. The verifier writes a reason under reward.json.additional_data.reason; the shim reads a top-level reason, and the task grader returns only the failed overall-score row. A solver can consequently debug pixel math even when compilation never ran.

Reproduced on the published amd64 judge seededge/edgebench.judge.ffmpeg_swscale_reimplementation:e938bf6f485e (digest sha256:711b595fe8a28783b6450afd900a2d4a52fa0c409efdb78e94deed012c0ab9f6), with networking disabled, 4 CPUs and 8 GiB. An empty archive, a Makefile whose release target fails, a library whose context constructor returns null, and a research JSON mentioning verifier-related text all produce valid score 0 and the identical summary: 0/1 tests passed. Failed: tests/eval.py::overall_score.

This is missing feedback, not a measured numerical-score error. The research-note example also exercises the existing broad source scan; blanket JSON/text exemptions would be unsafe because those files may be build inputs. The proposed fix records the stage at its producing branch and forwards only a fixed public message into the existing summary/detail. Scores, validity, test counts, thresholds, workloads and policy decisions remain unchanged. Raw reasons, paths, exceptions and workload details are not forwarded.

A source patch and same-host old/new verification are in preparation. Score injection, root build isolation, the legacy category-hint script and dev-tool/baseline mismatch remain separate known issues.

Lingua principale
Python
Stelle
474
Fork
20
Merge medio
3g 6h
PR unite (30g)
3

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di ByteDance-Seed/EdgeBench

Tutte le issue di ByteDance-Seed/EdgeBench

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.