Stop a gate run early once the verdict is settled
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- python
- Domain
- machine-learning, testing-qa
Research direction
Read src/meridian/gate/decide.py and the relevant modules under src/meridian/stats/ to understand how gates currently decide verdicts. First discuss an anytime-valid sequential testing method in the issue, since it changes verdict semantics; then prototype behind a flag if the design is agreed. Done means the design is agreed and, if implemented, early-stopped verdicts match full runs on the self-eval.
Written by the indexing model from the issue text.
Description
Context
Every gate runs all n trials even when the verdict is already clear, which costs model calls. Anytime-valid sequential tests can stop early without inflating error rates.
What to do
Discuss the method in this issue first (it changes what a verdict means), then prototype it behind a flag.
Done when
A design is agreed and, if built, the early-stopped verdict matches the full run on the self-eval.
Where to look
src/meridian/gate/decide.py, src/meridian/stats/
New here? Read CONTRIBUTING.md and comment on this issue to say you are taking it.
Generated by Claude Code
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- Avg merge
- 1h 54m
- Merged PRs (30d)
- 14
Getting set up
- Ships a Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from patsypppe/meridian
-
area: security area: tests size: S tests
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
Maintainers usually reply within 1 day
-
area: dashboard area: tests good first issue size: XS tests
Difficulty 1/5 1-3 hours Newbie friendliness 88/100
Maintainers usually reply within 1 day
-
area: dashboard area: tests good first issue size: S tests
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 1 day
-
area: dashboard area: tests good first issue size: XS tests
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
Maintainers usually reply within 1 day
-
area: gate area: tests good first issue size: S tests
Difficulty 1/5 1-3 hours Newbie friendliness 85/100
Maintainers usually reply within 1 day
All issues in patsypppe/meridian
Similar issues
-
fuzz fuzz-unconfirmed
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
Maintainers usually reply within 1 day
-
ready-for-agent
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
neilsh/stunt-reel#220 ·
Maintainers usually reply within 1 day
-
[Issue]: use_fp4_cache is accepted but ignored; the launcher always writes FP8 recordsPossibly taken @0z5a claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
area: cli priority: p3 size: S type: bug
Difficulty 2/5 1-3 hours Newbie friendliness 83/100
finos/open-resource-broker#757 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
Maintainers usually reply within 6 days