J2: join the novelty diagnosis to a task-level effect (AC-J02-004, AC-J02-005)
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 48/100
Research direction
Start by reading the J2 requirements and the existing arms run and tasks run entry points; the payload does not name specific files or tests. Work out how to re-measure an edited skill and its original on the same task set, model, and repetitions, then report the task-level effect with uncertainty. Done when one step produces that comparison and allows null or negative effects.
Written by the indexing model from the issue text.
Description
Parent: #95
In J2, novelty is a diagnostic. It does not show an improvement. The novelty steps are enforced (AC-J02-001, AC-J02-002, AC-J14-001..004). Still open:
- AC-J02-004 (partial): today the docs state that novelty is not task impact, but no report joins the novelty diagnosis to a task-level effect.
- AC-J02-005 (proposed): re-measure an edited skill and its original on the same task set, model and repetitions in one step, and report the effect with its uncertainty. Today this takes separate
arms runortasks runinvocations.
A null or negative effect is a valid outcome (AC-J02-003).
- Dominant language
- Python
- Stars
- 8
- Forks
- 7
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 187
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from intel/dokimi
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
Maintainers usually reply within 1 day
-
Consolidate duplicated subsystems (stats, LLM routing, judge loop, containers, digests, writers)Open
Difficulty 5/5 Over a week Newbie friendliness 25/100
Maintainers usually reply within 1 day
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
Maintainers usually reply within 1 day
-
Difficulty 5/5 Over a week Newbie friendliness 24/100
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 48/100
Maintainers usually reply within 1 day
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
scikit-learn-contrib/category_encoders#553 ·
Maintainers usually reply within 1 day
-
tests/strategies_test.py:174 goes uncovered on about 1 run in 100, failing the coverage ratchetOpenflaky-watch
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
btclib-org/btclib#2839 ·
Maintainers usually reply within 1 day
-
BarrelList constructor drops false-valued iterable contentsPossibly taken @Gambit-Checkmate claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 19 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 77/100
vllm-project/vllm#63863 ·
Maintainers usually reply within 1 day