eval reference: stamp a digest of the questions file so headroom catches an edit at the same path

Aperta
  • Qualcosa non va?
#356 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
3/5
Tempo stimato
1-2 giorni
Idoneità per principianti
32/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python
Ambito
cli

Direzione di ricerca

Start with where eval reference writes reference_scoring into the arms file, and where eval headroom's pooling guard compares the scorer kind and questions path. The issue names no source files, so locate them from those command names. Before coding, read the PR #279 thread (PRRT_kwDOVBg7g86yYDIs), since the issue calls the digest decision open. Done means questions_sha256 is written, a digest mismatch exits 2, and files without the digest still pass the path check.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Problem

eval reference writes reference_scoring into an arms file: the scorer kind and the questions file it
was graded against. eval headroom (#279) compares the recorded scorer and questions path across inputs,
so two reference-scored files that name the same path are pooled as one measurement even when the file was
edited in between. The file's grading content is not in the record, so the edit is not seen.

Example: two runs scored against questions.json before and after an edit that changes the exact target of
one question. Both carry the same reference_scoring, the pool averages 100 and 0 and can hide a ceiling.

Proposal

Have eval reference stamp a digest of the questions file (the grading content, canonicalized) into
reference_scoring in the arms file (an arms.v1 field), and have eval headroom compare that digest across
inputs and refuse on a mismatch (exit 2), as it does for the scorer kind now.

Changes: eval reference writes questions_sha256; eval headroom refuses a mismatch in the pooling guard;
old files without the digest keep the path check only.

Context: PR #279, thread PRRT_kwDOVBg7g86yYDIs (the open decision).

Lingua principale
Python
Stelle
9
Fork
9
Merge medio
1g 8h
PR unite (30g)
282

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di intel/dokimi

Tutte le issue di intel/dokimi

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.