Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

repo tasks: build review_replay tasks from recorded review threads

Chiusa
  • Qualcosa non va?
#260 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@antonwolfy ci sta già lavorando.

Dal 28/10/2026.

  • #265 di @antonwolfy — aperta

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
38/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python
Ambito
testing-qa, tooling

Direzione di ricerca

Start by reading docs/preparing-a-repository.md and the existing review replay tasks and repo tasks check implementation to learn the task format and thread mining rules. Then trace how repo tasks subcommands generate task directories, and inspect tests for offline repository commands. Done means recorded threads produce tasks accepted by repo tasks check and usable by a two-arm repo run, with tests requiring no network.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Problem

How much of a repository's past review its guidance would have caught is counted by hand today. A recent review of an AGENTS.md PR in IntelPython/dpnp did that over the 25 PRs merged after it was opened and their 16 substantive review comments. The count went wrong in ways a tool would not:

  • comments the author meant or declined with a reason (a CI matrix kept apart on purpose, an overflow NumPy has too) counted as misses;
  • one comment counted both as covered by a rule and as an example of the gap;
  • a fix cited as made in review had been undone by a later merge of main, and is still missing on the default branch.

dokimi:review_replay already measures this. It replays a merged PR at the revision humans reviewed and scores an agent's findings against the human threads, and recall_addressed counts only the threads the code changed for (contract §4.5). Its input is the missing part: gold.json and the task directories are written by hand, so the measurement is skipped and the review falls back to counting.

Proposal

A repo tasks subcommand that turns recorded review threads into replay tasks, for example:

dokimi repo tasks replay <source> --threads threads.json --out <tasks dir>
  • Input: a JSON file of merged PRs with their review threads, as gh api graphql returns them. Per PR: number, author, baseRefOid, mergeCommit. Per thread: path, line, originalLine, originalCommit, isOutdated, and comments (author, body, createdAt). dokimi makes no GitHub call, so the tests stay offline and a private repository works the same way. The docs give the gh query.
  • Which threads: the mining rules in docs/preparing-a-repository.md: the first comment by someone other than the PR author, no bots, anchored on a line. A review body with no line is listed in the output but is not gold.
  • Per PR, one task:
    • task.json: grader: dokimi:review_replay, score: {metric: recall, max: 1.0}, the base and the reviewed revision (the originalCommit most threads anchor on), and the review prompt of the existing replay tasks;
    • gold.json: one entry per thread, {id, file, line, kind, comment, addressed}, plus the author's reply.
  • addressed: true when the anchored file changed within window lines of the anchor between the reviewed revision and the PR's merge commit, false otherwise. A maintainer reads reply and can overrule it before a run: "it's intended" or "no reason to update" means the comment was declined.
  • kind: unknown, for the maintainer to set.

A two-arm repo run --arm guidance:none,guidance:raw, or guidance:rev:<sha> for a guidance PR, then reports recall_addressed per arm. That is the measured form of "the rules would have caught N of these comments".

Open questions

  • A thread on a revision the PR later rewrote: anchor on originalLine at originalCommit, and drop outdated threads whose file is gone?
  • A PR that merges main in after review: is addressed decided at the PR's merge commit, so a fix a later merge undid reads as not addressed?
  • Is "the file changed near the line" enough for addressed, or should a declining reply override it on its own?
  • One task per PR, or one per reviewed revision when threads anchor on several?

Done when

On a window of merged PRs of a real repository, the command writes tasks that repo tasks check accepts, and a two-arm repo run reports recall_addressed per arm. The tests use a recorded threads file, with no network.

Lingua principale
Python
Stelle
9
Fork
9
Merge medio
1g 7h
PR unite (30g)
231

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di intel/dokimi

Tutte le issue di intel/dokimi

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.