Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

repo tasks: build review_replay tasks from recorded review threads

Open
  • Is something wrong?
#260 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
38/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
python
Domain
testing-qa, tooling

Research direction

Start by reading docs/preparing-a-repository.md and the existing review replay tasks and repo tasks check implementation to learn the task format and thread mining rules. Then trace how repo tasks subcommands generate task directories, and inspect tests for offline repository commands. Done means recorded threads produce tasks accepted by repo tasks check and usable by a two-arm repo run, with tests requiring no network.

Written by the indexing model from the issue text.

Description

Problem

How much of a repository's past review its guidance would have caught is counted by hand today. A recent review of an AGENTS.md PR in IntelPython/dpnp did that over the 25 PRs merged after it was opened and their 16 substantive review comments. The count went wrong in ways a tool would not:

  • comments the author meant or declined with a reason (a CI matrix kept apart on purpose, an overflow NumPy has too) counted as misses;
  • one comment counted both as covered by a rule and as an example of the gap;
  • a fix cited as made in review had been undone by a later merge of main, and is still missing on the default branch.

dokimi:review_replay already measures this. It replays a merged PR at the revision humans reviewed and scores an agent's findings against the human threads, and recall_addressed counts only the threads the code changed for (contract §4.5). Its input is the missing part: gold.json and the task directories are written by hand, so the measurement is skipped and the review falls back to counting.

Proposal

A repo tasks subcommand that turns recorded review threads into replay tasks, for example:

dokimi repo tasks replay <source> --threads threads.json --out <tasks dir>
  • Input: a JSON file of merged PRs with their review threads, as gh api graphql returns them. Per PR: number, author, baseRefOid, mergeCommit. Per thread: path, line, originalLine, originalCommit, isOutdated, and comments (author, body, createdAt). dokimi makes no GitHub call, so the tests stay offline and a private repository works the same way. The docs give the gh query.
  • Which threads: the mining rules in docs/preparing-a-repository.md: the first comment by someone other than the PR author, no bots, anchored on a line. A review body with no line is listed in the output but is not gold.
  • Per PR, one task:
    • task.json: grader: dokimi:review_replay, score: {metric: recall, max: 1.0}, the base and the reviewed revision (the originalCommit most threads anchor on), and the review prompt of the existing replay tasks;
    • gold.json: one entry per thread, {id, file, line, kind, comment, addressed}, plus the author's reply.
  • addressed: true when the anchored file changed within window lines of the anchor between the reviewed revision and the PR's merge commit, false otherwise. A maintainer reads reply and can overrule it before a run: "it's intended" or "no reason to update" means the comment was declined.
  • kind: unknown, for the maintainer to set.

A two-arm repo run --arm guidance:none,guidance:raw, or guidance:rev:<sha> for a guidance PR, then reports recall_addressed per arm. That is the measured form of "the rules would have caught N of these comments".

Open questions

  • A thread on a revision the PR later rewrote: anchor on originalLine at originalCommit, and drop outdated threads whose file is gone?
  • A PR that merges main in after review: is addressed decided at the PR's merge commit, so a fix a later merge undid reads as not addressed?
  • Is "the file changed near the line" enough for addressed, or should a declining reply override it on its own?
  • One task per PR, or one per reviewed revision when threads anchor on several?

Done when

On a window of merged PRs of a real repository, the command writes tasks that repo tasks check accepts, and a two-arm repo run reports recall_addressed per arm. The tests use a recorded threads file, with no network.

Dominant language
Python
Stars
8
Forks
7
Avg merge
1d 4h
Merged PRs (30d)
199

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from intel/dokimi

All issues in intel/dokimi

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.