J11/J12: distinct inconclusive verdict and blocking policy (AC-J11-002, AC-J12-002)
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 43/100
Research direction
Start with the eval judge-validity --sequential entry point and trace how it reports a reached sample cap; the issue gives no file or test names. Define how the inconclusive or budget-stopped verdict is represented and how configured policy controls blocking, while preserving the rule that budget-limited results never pass. Done when the cap produces a distinct verdict and policy can determine whether it blocks.
Written by the indexing model from the issue text.
Description
Parent: #95
- AC-J11-002 (partial): a sample cap that ends without a decision must be reported as inconclusive.
eval judge-validity --sequentialreports a reached cap as FAIL. That is not a pass (AC-J07-005 is enforced), but it is not a distinct inconclusive verdict either. - AC-J12-002 (proposed): an inconclusive or budget-stopped check is reported as such, and a configured policy decides whether it blocks.
Neither may turn a budget-limited result into a pass.
- Dominant language
- Python
- Stars
- 9
- Forks
- 9
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 231
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from intel/dokimi
-
pip-audit: new advisory against the dependency locksPossibly taken @antonwolfy claimed this today. Opendependencies
Difficulty 2/5 1-3 hours Newbie friendliness 60/100
Maintainers usually reply within 1 day
-
eval cross-judge drops the provider when splitting the judge specPossibly taken @vlad-perevezentsev claimed this 2 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
Maintainers usually reply within 1 day
-
Difficulty 5/5 Over a week Newbie friendliness 42/100
Maintainers usually reply within 1 day
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
Maintainers usually reply within 1 day
-
Difficulty 5/5 Over a week Newbie friendliness 38/100
Maintainers usually reply within 1 day
Similar issues
-
enhancement help wanted
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Git-Hospital/githealthcheck#61 ·
Maintainers usually reply within 1 day
-
changelog investigate
Difficulty 2/5 1-3 hours Newbie friendliness 67/100
ramnes/notion-sdk-py#433 ·
-
Partial method descriptors raise on class accessPossibly taken @Gambit-Checkmate claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 19 days
-
docs pkg:core
Difficulty 1/5 Under an hour Newbie friendliness 84/100
pydantic/pydantic-ai#11929 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
intake/intake-esm#789 ·