Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

judge-validity: validate the judge in the setting arms are scored — persist judge material by digest

Aperta
  • Qualcosa non va?
#48 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
22/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python
Ambito
ai, testing

Direzione di ricerca

Start with ArmRunner.judge and the eval judge-validity command, then the arms.v1 artifact writer and its evaluation records, which currently keep only 200-character snippets. The change adds a digest-keyed judge_materials store and a --context stored option, so a first step is to read how the in-memory material store is populated during an answering run. Done means both settings appear per variant in the report, and re-judging a stored artifact resolves material from the store rather than from snippets.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Problem

eval judge-validity checks the judge with no context. The arms run judges skill and docs arms with the arm's full treatment material in the judge's context. So the setting the skill-arm deltas come from is never validated.

That gap matters here. On 81 skill-arm answers from the perf golden set (Haiku 4.5 judge, 24 bases, deterministic degradations, one run), the same pairs were judged twice: once with the 200-character report snippet, once with the full material.

snippet full material
mean intact score 85.9 89.9
pairwise accuracy, overall 0.869 0.808
contradiction 0.938 1.000 (drop +11.5, 20/24 pairs, sign p = 0.002)
wrong_tools 0.891 0.674 (+1/−6, p = 0.125)
wrong_numbers 0.750 0.722

Scores go up and discrimination doesn't, and it moves differently per defect. A gate that passes no context cannot see any of it.

Why the gate can't just pass the context today

The material reaches the judge from an in-memory store that lives only for the answering run. arms.v1 keeps 200-char snippets, and skill arms record no content digest. So nothing reading an artifact can rebuild what the judge saw, or prove that a rebuild matches it. That is the same limitation that makes re-judging an answers file fall back to snippets (#13 now labels such runs instead of hiding it).

Proposal

Persist the material once, by digest, and let every consumer use it.

 arms.v1
+  judge_materials: {<sha256>: <text>}        # one copy per distinct material
   evaluations[].scores.<arm>
     judge_context_source: stored | snippet_fallback
+    judge_context_sha256: <sha256>           # which material this score saw
  • Size. A skill arm's material is identical for every question, so it is stored once (~15k chars). Docs arms differ per question, about 15k × questions. That's about 1.2 MB for an 81-question docs arm. Acceptable, and it could be gzip-optional.
  • eval judge-validity --context stored reads the material for each base's arm and passes it to the scorer alongside the degraded answer. The report gets one column set per setting (no context / stored), so the two can be compared.
  • Re-judging becomes faithful. ArmRunner.judge on a loaded artifact resolves from judge_materials, not from snippets. snippet_fallback remains only for pre-digest artifacts, and cross_judge / eval pairwise can score stored arms on the same evidence.
  • Schema: additive. Old artifacts validate, and consumers treat a missing digest as "unknown material".

Alternatives considered

  • Rebuild from the arm spec (skill:<path> → as_context + manifest). Works only for skill arms, can't verify the version (no digest), and does nothing for docs arms.
  • Run the gate inside arms (arms --validity). Uses the exact material, but every check costs a full answering run, and it can't be re-run against a stored artifact.

Acceptance

  • Gate on the perf golden set reports both settings per variant, with the length-only baseline. Expect the table above to reproduce within run-to-run spread.
  • Re-judging a stored artifact with the same judge gives judge_context_source: stored for every non-baseline score, and its aggregates sit within run-to-run spread of the original.
  • Artifact size increase is reported in the PR.

🤖 Generated with Claude Code

Lingua principale
Python
Stelle
8
Fork
7
Merge medio
1g 55m
PR unite (30g)
126

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di intel/dokimi

Tutte le issue di intel/dokimi

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.