Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

judge-validity: validate the judge in the setting arms are scored — persist judge material by digest

Abierto
  • ¿Algo no está bien?
#48 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
22/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
python
Área
ai, testing

Línea de trabajo

Start with ArmRunner.judge and the eval judge-validity command, then the arms.v1 artifact writer and its evaluation records, which currently keep only 200-character snippets. The change adds a digest-keyed judge_materials store and a --context stored option, so a first step is to read how the in-memory material store is populated during an answering run. Done means both settings appear per variant in the report, and re-judging a stored artifact resolves material from the store rather than from snippets.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Problem

eval judge-validity checks the judge with no context. The arms run judges skill and docs arms with the arm's full treatment material in the judge's context. So the setting the skill-arm deltas come from is never validated.

That gap matters here. On 81 skill-arm answers from the perf golden set (Haiku 4.5 judge, 24 bases, deterministic degradations, one run), the same pairs were judged twice: once with the 200-character report snippet, once with the full material.

snippet full material
mean intact score 85.9 89.9
pairwise accuracy, overall 0.869 0.808
contradiction 0.938 1.000 (drop +11.5, 20/24 pairs, sign p = 0.002)
wrong_tools 0.891 0.674 (+1/−6, p = 0.125)
wrong_numbers 0.750 0.722

Scores go up and discrimination doesn't, and it moves differently per defect. A gate that passes no context cannot see any of it.

Why the gate can't just pass the context today

The material reaches the judge from an in-memory store that lives only for the answering run. arms.v1 keeps 200-char snippets, and skill arms record no content digest. So nothing reading an artifact can rebuild what the judge saw, or prove that a rebuild matches it. That is the same limitation that makes re-judging an answers file fall back to snippets (#13 now labels such runs instead of hiding it).

Proposal

Persist the material once, by digest, and let every consumer use it.

 arms.v1
+  judge_materials: {<sha256>: <text>}        # one copy per distinct material
   evaluations[].scores.<arm>
     judge_context_source: stored | snippet_fallback
+    judge_context_sha256: <sha256>           # which material this score saw
  • Size. A skill arm's material is identical for every question, so it is stored once (~15k chars). Docs arms differ per question, about 15k × questions. That's about 1.2 MB for an 81-question docs arm. Acceptable, and it could be gzip-optional.
  • eval judge-validity --context stored reads the material for each base's arm and passes it to the scorer alongside the degraded answer. The report gets one column set per setting (no context / stored), so the two can be compared.
  • Re-judging becomes faithful. ArmRunner.judge on a loaded artifact resolves from judge_materials, not from snippets. snippet_fallback remains only for pre-digest artifacts, and cross_judge / eval pairwise can score stored arms on the same evidence.
  • Schema: additive. Old artifacts validate, and consumers treat a missing digest as "unknown material".

Alternatives considered

  • Rebuild from the arm spec (skill:<path> → as_context + manifest). Works only for skill arms, can't verify the version (no digest), and does nothing for docs arms.
  • Run the gate inside arms (arms --validity). Uses the exact material, but every check costs a full answering run, and it can't be re-run against a stored artifact.

Acceptance

  • Gate on the perf golden set reports both settings per variant, with the length-only baseline. Expect the table above to reproduce within run-to-run spread.
  • Re-judging a stored artifact with the same judge gives judge_context_source: stored for every non-baseline score, and its aggregates sit within run-to-run spread of the original.
  • Artifact size increase is reported in the PR.

🤖 Generated with Claude Code

Lenguaje dominante
Python
Estrellas
8
Forks
7
Merge medio
1 d 55 min
PR fusionados (30 d)
126

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de intel/dokimi

Todos los issues de intel/dokimi

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.