repo tasks: authoring replay tasks from merged PRs
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 42/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- python
- Domain
- ai-infra-agents, testing-qa
Research direction
Start by reading the existing repo tasks replay implementation and the task contract’s section 3 to see how review replay builds tasks and how feature or fix tasks are graded. Then inspect the manifest handling and repo tasks check path, along with the network-free fixture tests. Done means authoring tasks from merged PRs are accepted by repo tasks check, with the untouched base failing and the PR diff passing as the oracle.
Written by the indexing model from the issue text.
Description
Problem
repo tasks replay builds dokimi:review_replay tasks, which measure what guidance does to an agent reviewing code. Most guidance is read while writing code. Of the eight AGENTS.md files in IntelPython/dpnp#3099, most lines say where a new function's kernel and dispatch helpers go, how tests are laid out, and what a changelog entry looks like. The review replay of that PR exercised little of it.
dokimi can already grade such work. A task.json with family: feature or fix and the repository's own grade entry point does it (contract §3). But those tasks are written by hand, so this measurement is skipped too, just as review counting was before repo tasks replay.
Proposal
Build authoring tasks from merged PRs, with the same input and the same rules for what to leave out as review replay: repo tasks replay --kind authoring, or a sibling subcommand. Per merged PR:
- Base: the merge base of the PR with
mergeCommit^1, as for review replay. - Prompt: the PR's title and description, with links to its commits and diff and anything quoting the change removed.
--prompt-filereplaces the default. - Hidden tests: the test files the PR added or changed. They are held out of the workspace and applied only at grade time.
- Grade: the repository's test command from the manifest, on the hidden tests; the gate is that they pass.
oracleis the PR's own diff, sorepo tasks checkcan show that the base fails and the oracle passes. - Left out: guidance files and
eval_dir, as review replay leaves them out.
Open questions
- Which PRs qualify: with tests, under a size limit, not only refactors.
- Leaked tests: a description that names the test functions.
- Build cost: dpnp needs its SYCL extensions built for every cell. How much can the manifest's caches and a prebuilt base image save?
- Partial credit: some hidden tests passing. Should it count, as a score rather than a gate?
- Flaky tests: run the oracle twice in
tasks check?
Done when
On a window of merged PRs of a real repository, the command writes tasks that repo tasks check accepts (the untouched base fails, the oracle passes). The tests use a fixture repository with no network.
- Dominant language
- Python
- Stars
- 9
- Forks
- 9
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 231
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from intel/dokimi
-
pip-audit: new advisory against the dependency locksPossibly taken @antonwolfy claimed this 1 day ago. Opendependencies
Difficulty 2/5 1-3 hours Newbie friendliness 60/100
Maintainers usually reply within 1 day
-
eval cross-judge drops the provider when splitting the judge specPossibly taken @vlad-perevezentsev claimed this 3 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
Maintainers usually reply within 1 day
-
novelty: spread the atom cap across the artifact instead of its openingPossibly taken A pull request linked to this issue is open or already merged. Open
Difficulty 5/5 Over a week Newbie friendliness 35/100
Maintainers usually reply within 1 day
-
repo compare: an interval that agrees with the sign-flip pPossibly taken A pull request linked to this issue is open or already merged. Open
Difficulty 5/5 Over a week Newbie friendliness 38/100
Maintainers usually reply within 1 day
-
review_replay: optional judged recall over the credited hitsPossibly taken A pull request linked to this issue is open or already merged. Open
Difficulty 5/5 Over a week Newbie friendliness 34/100
Maintainers usually reply within 1 day
Similar issues
-
bug
Difficulty 1/5 Under an hour Newbie friendliness 86/100
awslabs/visual-asset-management-system#937 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
-
follow-up
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 83/100
Maintainers usually reply within 3 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
Maintainers usually reply within 2 days