J7: draft tasks, negative controls and coverage gaps (AC-J07-004, AC-J07-006, AC-J07-007)
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- python
- Domain
- ai-infra-agents, testing
Research direction
Start with parent issue #95 and the existing grader self-check for AC-J07-001 through AC-J07-003. Decide how proposed AC-J07-006 and AC-J07-007 should be represented, and whether AC-J07-004 can be measured by a report-level assertion. Record the design rationale here or in the implementing PR; done means the rationale and chosen criteria address task hypotheses and sources, coverage gaps, and the limits of self-check evidence.
Written by the indexing model from the issue text.
Description
Parent: #95
J7 separates four claims about evals: schema validity, grader correctness, task relevance and coverage. The grader self-check (AC-J07-001..003) is enforced. Still open:
- AC-J07-006 (proposed): draft tasks from the repository's code, docs, issues and traces. Present them as hypotheses with their source, never as confirmed user intent.
- AC-J07-007 (proposed): an eval set's report lists the areas it has no task for.
- AC-J07-004 (manual): a passing self-check must not be reported as evidence of relevance or coverage. Decide whether a report-level assertion can make this measurable.
The design rationale goes in this issue or in the PR that implements it.
- Dominant language
- Python
- Stars
- 8
- Forks
- 7
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 187
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from intel/dokimi
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
Maintainers usually reply within 1 day
-
Consolidate duplicated subsystems (stats, LLM routing, judge loop, containers, digests, writers)Open
Difficulty 5/5 Over a week Newbie friendliness 25/100
Maintainers usually reply within 1 day
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
Maintainers usually reply within 1 day
-
Difficulty 5/5 Over a week Newbie friendliness 24/100
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 48/100
Maintainers usually reply within 1 day
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
UKGovernmentBEIS/optstop#10 ·
-
maintenance
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
pollen-robotics/reachy_mini#1508 ·
Maintainers usually reply within 1 day
-
[BUG] `cloud-keys.env` values retain enclosing quotes causing 401 Unauthorized, and missing utf-8 encoding crashes on WindowsPossibly taken @divyanshu07-code claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 77/100
open-jarvis/OpenJarvis#1192 · 1 comment ·
Maintainers usually reply within 2 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
sylvainmetayer/homelab#851 ·
Maintainers usually reply within 1 day