record replay provenance per answer in arms artifacts so headroom can exclude cached repeats
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 45/100
Research direction
Start with the in-process usage record where the replayed flag is set when DOKIMI_RESPONSE_CACHE serves an answer, then find where arms.v1 artifacts are written per answer. Add a replayed provenance field to that schema and make eval headroom skip or flag replayed repeats. Done when a cached rerun produces artifacts marked replayed and headroom excludes them in its counts.
Written by the indexing model from the issue text.
Description
Headroom counts repeats of a run as independent samples. With DOKIMI_RESPONSE_CACHE on, repeated arms runs replay the same answers and judgments, so cached repeats look like fresh runs and can make an item non-provisional.
Ask: record replay provenance per answer in arms.v1 artifacts (for example a replayed flag stored with each answer), so eval headroom can exclude cached repeats or warn about them.
Today the replayed flag lives only in the in-process usage record. Storing it is an arms.v1 schema change. Context: PR #279 (documents the limitation until this lands).
- Dominant language
- Python
- Stars
- 9
- Forks
- 9
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 282
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from intel/dokimi
-
repo budget: Copilot profiles get Claude Code's root CLAUDE* rule, so a root AGENTS.md reads offOpen
Difficulty 3/5 Half a day Newbie friendliness 42/100
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 18/100
Maintainers usually reply within 1 day
-
Difficulty 5/5 Over a week Newbie friendliness 8/100
Maintainers usually reply within 1 day
-
eval reference: stamp a digest of the questions file so headroom catches an edit at the same pathOpen
Difficulty 3/5 1-2 days Newbie friendliness 32/100
Maintainers usually reply within 1 day
-
Harbor features unsupported: MCP servers, compose sidecar services, separate verifier environmentOpen
Difficulty 5/5 Over a week Newbie friendliness 12/100
Maintainers usually reply within 1 day
Similar issues
-
ci-rerun: collect.py fails a whole pr-babysit cycle on one transient GitHub 5xx and never retriesOpenfollow-up
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
-
Built-in QUBO fallback applies bitwise xor to floating binary statesPossibly taken @iloveeyjafjalla claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
qboson/kaiwu-pytorch-plugin#1702 ·
Maintainers usually reply within 3 days
-
bug v2
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
modelcontextprotocol/python-sdk#3811 ·
Maintainers usually reply within 1 day
-
cost:cheap practicality:seen-in-the-wild severity:medium
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
fairagro/m4.2_advanced_middleware_api#620 ·
Maintainers usually reply within 1 day