repo compare: flag replay tasks a guidance revision may have seen

Open
  • Is something wrong?
#290 1 comment 0 reactions 1 assignee View on GitHub

Maintainers usually reply within 1 day

@antonwolfy is already working on this.

Since Nov 1, 2026.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
48/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
python
Domain
data, testing

Research direction

Start by reading the repo tasks replay, repo run, and repo compare entry points, plus contract sections 4.6 and 9 and the fixture tests. Trace how replay tasks store gold thread data and how compare reads arm metadata; then implement and test the requested review time and guidance history flags. Done means fixture tests cover PR-name and after-review-date flags, unrelated tasks remain unflagged, and the docs explain both flags.

Written by the indexing model from the issue text.

Description

enhancement

Problem

A guidance:rev:<sha> arm runs a revision of a repository's guidance against review_replay tasks built from PRs merged earlier. When the guidance was written or edited after a task's review, it can encode that review, and the arm then "catches" the thread because a rule was written from it. Nothing in a batch shows this.

A replay of IntelPython/dpnp#3099 (eight AGENTS.md files; 26 tasks from dpnp PRs merged 2026-09-01..10-28; guidance:none vs guidance:rev:d8fb283) ran into it:

  • The cut looked clean. The guidance's first commit (7781481, 14 Oct 18:45Z) was mined from reviews before it. So a split by review date looked like an out-of-sample test: 9 tasks reviewed after the cut, recall +0.111 against −0.025 for the 17 before.
  • It was not clean. The evaluated revision has later guidance commits that name three of those 9 tasks' PRs:
    • c9b533d "Update test guidance for the fixes in #3103";
    • a9f79ed "Update CMake guidance for the extension helper in #3111";
    • 441690c, with a rule taken from #3098.
  • And 441690c came after every one of them. It was committed after the first review comment of all 9.
  • The batch can't show it. task.json has no review time, and meta.json records the arm's SHA but not its guidance history. The split, and finding the leak, were done by hand.

Proposal

  • Review time: repo tasks replay writes reviewed_at into task.json: the createdAt of the earliest gold thread's first comment. The threads file already holds it.
  • Guidance history: repo run, for a guidance:rev:<rev> arm, records the commits reachable from <rev> that touch the manifest's guidance globs, in meta.json arm_rev.guidance_commits ({sha, date, subject}, newest first, capped).
  • Report it: repo compare flags the tasks such an arm may have seen.
    • By date: some guidance commit is dated after the task's reviewed_at.
    • By name: a guidance commit's message names the task's source_pr (#<n>).
    • The output says how many tasks each flag covers and names the named ones. The rest can be compared on their own (with a repo compare --task filter, or an option for it).
  • Arms with no history: a patch: arm has none, so it reads "unknown". guidance:raw is the base's own guidance and is never flagged.

Open questions

  • The date rule is strict. Here it flags all 26 tasks for d8fb283: the last guidance commit came after every review. Is a per-file rule better, counting only commits to a guidance file the task's diff makes relevant?
  • Which date? Author date or committer date, given that a rebase moves the committer date.
  • Content: should a commit that names no PR, but adds text matching a gold comment, be found too? That is a text-similarity question, likely out of scope.

Done when

The fixture tests cover two cases: a guidance commit that names a task's PR, and one dated after a task's review. repo compare flags both, and leaves the other tasks unflagged. The docs (contract §4.6 and §9) say what the flags mean.

Dominant language
Python
Stars
9
Forks
9
Avg merge
1d 8h
Merged PRs (30d)
269

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from intel/dokimi

All issues in intel/dokimi

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.