Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

repo tasks: authoring replay tasks from merged PRs

Aperta
  • Qualcosa non va?
#294 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
42/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python
Ambito
ai-infra-agents, testing-qa

Direzione di ricerca

Start by reading the existing repo tasks replay implementation and the task contract’s section 3 to see how review replay builds tasks and how feature or fix tasks are graded. Then inspect the manifest handling and repo tasks check path, along with the network-free fixture tests. Done means authoring tasks from merged PRs are accepted by repo tasks check, with the untouched base failing and the PR diff passing as the oracle.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Problem

repo tasks replay builds dokimi:review_replay tasks, which measure what guidance does to an agent reviewing code. Most guidance is read while writing code. Of the eight AGENTS.md files in IntelPython/dpnp#3099, most lines say where a new function's kernel and dispatch helpers go, how tests are laid out, and what a changelog entry looks like. The review replay of that PR exercised little of it.

dokimi can already grade such work. A task.json with family: feature or fix and the repository's own grade entry point does it (contract §3). But those tasks are written by hand, so this measurement is skipped too, just as review counting was before repo tasks replay.

Proposal

Build authoring tasks from merged PRs, with the same input and the same rules for what to leave out as review replay: repo tasks replay --kind authoring, or a sibling subcommand. Per merged PR:

  • Base: the merge base of the PR with mergeCommit^1, as for review replay.
  • Prompt: the PR's title and description, with links to its commits and diff and anything quoting the change removed. --prompt-file replaces the default.
  • Hidden tests: the test files the PR added or changed. They are held out of the workspace and applied only at grade time.
  • Grade: the repository's test command from the manifest, on the hidden tests; the gate is that they pass. oracle is the PR's own diff, so repo tasks check can show that the base fails and the oracle passes.
  • Left out: guidance files and eval_dir, as review replay leaves them out.

Open questions

  • Which PRs qualify: with tests, under a size limit, not only refactors.
  • Leaked tests: a description that names the test functions.
  • Build cost: dpnp needs its SYCL extensions built for every cell. How much can the manifest's caches and a prebuilt base image save?
  • Partial credit: some hidden tests passing. Should it count, as a score rather than a gate?
  • Flaky tests: run the oracle twice in tasks check?

Done when

On a window of merged PRs of a real repository, the command writes tasks that repo tasks check accepts (the untouched base fails, the oracle passes). The tests use a fixture repository with no network.

Lingua principale
Python
Stelle
9
Fork
9
Merge medio
1g 7h
PR unite (30g)
231

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di intel/dokimi

Tutte le issue di intel/dokimi

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.