eval reference: stamp a digest of the questions file so headroom catches an edit at the same path
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 32/100
Research direction
Start with where eval reference writes reference_scoring into the arms file, and where eval headroom's pooling guard compares the scorer kind and questions path. The issue names no source files, so locate them from those command names. Before coding, read the PR #279 thread (PRRT_kwDOVBg7g86yYDIs), since the issue calls the digest decision open. Done means questions_sha256 is written, a digest mismatch exits 2, and files without the digest still pass the path check.
Written by the indexing model from the issue text.
Description
Problem
eval reference writes reference_scoring into an arms file: the scorer kind and the questions file it
was graded against. eval headroom (#279) compares the recorded scorer and questions path across inputs,
so two reference-scored files that name the same path are pooled as one measurement even when the file was
edited in between. The file's grading content is not in the record, so the edit is not seen.
Example: two runs scored against questions.json before and after an edit that changes the exact target of
one question. Both carry the same reference_scoring, the pool averages 100 and 0 and can hide a ceiling.
Proposal
Have eval reference stamp a digest of the questions file (the grading content, canonicalized) into
reference_scoring in the arms file (an arms.v1 field), and have eval headroom compare that digest across
inputs and refuse on a mismatch (exit 2), as it does for the scorer kind now.
Changes: eval reference writes questions_sha256; eval headroom refuses a mismatch in the pooling guard;
old files without the digest keep the path check only.
Context: PR #279, thread PRRT_kwDOVBg7g86yYDIs (the open decision).
- Dominant language
- Python
- Stars
- 9
- Forks
- 9
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 282
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from intel/dokimi
-
repo budget: Copilot profiles get Claude Code's root CLAUDE* rule, so a root AGENTS.md reads offOpen
Difficulty 3/5 Half a day Newbie friendliness 42/100
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 18/100
Maintainers usually reply within 1 day
-
Difficulty 3/5 1-2 days Newbie friendliness 45/100
Maintainers usually reply within 1 day
-
Difficulty 5/5 Over a week Newbie friendliness 8/100
Maintainers usually reply within 1 day
-
Harbor features unsupported: MCP servers, compose sidecar services, separate verifier environmentOpen
Difficulty 5/5 Over a week Newbie friendliness 12/100
Maintainers usually reply within 1 day
Similar issues
-
cost:cheap practicality:seen-in-the-wild severity:medium
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
fairagro/m4.2_advanced_middleware_api#620 ·
Maintainers usually reply within 1 day
-
area:security effort:S priority:P2 provider:local-libvirt status:ready type:docs
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
randomparity/kdive#3495 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
microsoft/markitdown#2820 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 2 days