Detect flaky tasks across runs
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 55/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- python
- Domain
- testing-qa
Research direction
Start by reading the run history storage under src/meridian/store/ and the task model in src/meridian/models/task.py, then trace how reports are produced. Determine what evidence the stored runs provide for comparing a task’s pass rate with chance variation. Done means a report identifies flaky tasks and includes the evidence supporting each flag.
Written by the indexing model from the issue text.
Description
Context
Some tasks pass in one run and fail in the next with the same agent. They add noise to every verdict.
What to do
Using stored run history, flag tasks whose pass rate varies more than chance would explain, and suggest quarantining them (tasks already support state: quarantined).
Done when
A report lists flaky tasks with the evidence.
Where to look
src/meridian/store/, src/meridian/models/task.py
New here? Read CONTRIBUTING.md and comment on this issue to say you are taking it.
Generated by Claude Code
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- Avg merge
- 1h 54m
- Merged PRs (30d)
- 14
Getting set up
- Ships a Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from patsypppe/meridian
-
area: security area: tests size: S tests
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
Maintainers usually reply within 1 day
-
area: dashboard area: tests good first issue size: XS tests
Difficulty 1/5 1-3 hours Newbie friendliness 88/100
Maintainers usually reply within 1 day
-
area: dashboard area: tests good first issue size: S tests
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 1 day
-
area: dashboard area: tests good first issue size: XS tests
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
Maintainers usually reply within 1 day
-
area: gate area: tests good first issue size: S tests
Difficulty 1/5 1-3 hours Newbie friendliness 85/100
Maintainers usually reply within 1 day
All issues in patsypppe/meridian
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
huggingface/deep-rl-class#723 ·
-
[tool] bug: Qwen3 XML tool parser raises on `...` argument values (Ellipsis is not JSON serializable)Possibly taken @super-yuki claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 87/100
verl-project/verl#8559 ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
TencentCloud/octop-harness#228 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
google/osv-scalibr#2585 ·
Maintainers usually reply within 2 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
NousResearch/hermes-agent#149323 ·
Maintainers usually reply within 1 day