Option to grade a trial's final state after a timeout
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 48/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- python
- Domain
- testing-qa
Research direction
Read src/meridian/runtime/trial_runner.py at _verdict to trace how timed-out trials are scored and how the verifier result is handled. The issue asks for a decision about a flag and reporting, so first clarify those choices in the discussion before implementation. If adopted, add tests for the chosen behavior; done means the decision is recorded and the adopted change is implemented with tests.
Written by the indexing model from the issue text.
Description
Context
A timed-out trial is scored timeout without looking at its final state. Harbor runs the verifier anyway. In one run the agent had already written the right file when it was killed.
What to do
Discuss whether this belongs behind a flag, and how to report it (for example pass-after-timeout counted as a fail for pass^k but shown separately).
Done when
A decision is recorded, and if adopted, implemented with tests.
Where to look
src/meridian/runtime/trial_runner.py (_verdict)
New here? Read CONTRIBUTING.md and comment on this issue to say you are taking it.
Generated by Claude Code
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- Avg merge
- 1h 54m
- Merged PRs (30d)
- 14
Getting set up
- Ships a Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from patsypppe/meridian
-
area: security area: tests size: S tests
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
Maintainers usually reply within 1 day
-
area: dashboard area: tests good first issue size: XS tests
Difficulty 1/5 1-3 hours Newbie friendliness 88/100
Maintainers usually reply within 1 day
-
area: dashboard area: tests good first issue size: S tests
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 1 day
-
area: dashboard area: tests good first issue size: XS tests
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
Maintainers usually reply within 1 day
-
area: gate area: tests good first issue size: S tests
Difficulty 1/5 1-3 hours Newbie friendliness 85/100
Maintainers usually reply within 1 day
All issues in patsypppe/meridian
Similar issues
-
bug severity:low
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Maintainers usually reply within 2 days
-
curriculum documentation quality
Difficulty 2/5 1-3 hours Newbie friendliness 77/100
githubnext/gh-aw-workshop#5367 ·
Maintainers usually reply within 3 days
-
OAuth documentation for GET /api/v2/users/me/ appears inconsistent with required global read scopeOpen
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Maintainers usually reply within 1 day
-
Create Noise Latents ignores randomize_seed and produces the same noise every runPossibly taken @griptapeops claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
griptape-ai/griptape-nodes-library-diffusers#127 ·
Maintainers usually reply within 1 day
-
Jenkins capacity report: metrics history is lost when archiving a month failsPossibly taken @RaphaelFakhri claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
adoptium/infrastructure#4568 ·
Maintainers usually reply within 1 day