fix(aidd-dev): review leaves out-of-diff phases and the verified percent undefined

Open
#858 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
74/100
Issue type
Feature
Clarity
Clearly specified
Activity status
Active
Domain
documentation

Research direction

Start with plugins/aidd-dev/skills/05-review/SKILL.md and actions/02-review-functional.md, then compare review-template.md and references/review-rubric.md against the stated scope. Update the Test section to cover out-of-scope phases, deterministic reruns, and percentage calculation. Done means the four acceptance scenarios pass and all four documents agree without contradiction.

Written by the indexing model from the issue text.

Description

Problem

05-review does not say what happens to plan phases outside the reviewed diff, nor how the verified percent is computed. Reviewing one phase of a multi-phase plan, three models on the same skill produced three incompatible review.md files.

The skill contradicts itself:

  • SKILL.md:30 says review.md is overwritten each run: "a snapshot of the current diff, not a history".
  • 02-review-functional.md:23 says the Phases section holds "one block per plan phase".
  • Nothing says whether the previous review.md is read before writing.
  • review-template.md:33 renders {{pct}}% ({{n_checked}}/{{n_total}}), but nothing says which phases enter n_total. not-applicable is only defined as "neither a finding nor a blocker" (review-rubric.md:19), never for the percent, and never when to use it.

Observed on a 7-phase plan, phase 4 reviewed in three worktrees, aidd-dev 2.4.1 (05-review is unchanged in 2.5.0):

Tool + model Diff Phase blocks Verified line
Mistral Vibe CLI + Mistral Medium 6623000...HEAD 7 100% (6/6), with 5 of 6 phase-4 boxes checked
Mistral Vibe CLI + GLM 6623000...69e491a 1 (phase 4 only) 100% (6/6)
OpenCode + Qwen 0bb8973...HEAD 7 53% (16/34), while the file renders 21 of 36 boxes checked, and 16/34 is 47%
  • Mistral Medium and Qwen have word-for-word identical blocks for phases 1, 2, 3, 5, 6 and 7: both copied the previous review.md. Mistral's phase 2 still says "ajoutés par ce diff" while its own Files checked lists only phase-4 files. Copied evidence reads as fresh verification.
  • GLM dropped the six other phases.
  • Qwen tagged the 15 criteria of out-of-diff phases not-applicable, Mistral used the same tag for a partially verifiable phase-4 criterion: one tag, two meanings.

Scope

  • review.md stays a snapshot: the skill never reads the previous review.md, and says so.
  • Every run renders one block per plan phase, re-derived from the plan.
  • A phase the diff does not touch keeps its block, marked out of scope in its heading, with its criteria unchecked and no evidence. Nothing is copied from an earlier review.
  • A phase the diff touches is verified as today: [x] with evidence, or [ ] tagged fix, not-applicable or fixed.
  • Define not-applicable: a criterion of a phase in scope that this change cannot satisfy by nature (for example, it depends on a later phase). It is never used to mark a phase out of scope.
  • The verified percent counts only the criteria of phases in scope, minus not-applicable ones, and matches the boxes rendered.
  • Update the Test section of 02-review-functional.md to check the four rules above.

Acceptance criteria

  • Reviewing phase 4 of a 7-phase plan renders 7 phase blocks, 6 of them marked out of scope.
  • An out-of-scope block holds no [x] and no file:line.
  • Re-running the review on the same diff gives the same Phases and Verification, whatever the content of the previous review.md.
  • With 6 criteria in scope, 5 checked and 1 not-applicable, the report shows 100% (5/5).
  • SKILL.md, 02-review-functional.md, review-template.md and review-rubric.md agree on these rules, with no contradiction left.

Prior art in this repository

Out of scope

  • How "the diff against the repository default branch" resolves (5 occurrences in 05-review, never defined). A clone whose origin/HEAD points at a feature branch diffs against it. Separate issue.
  • Enforcing review-validator.yml: a section outside the list survived in the three reviews above. Separate issue.

Reported on Discord: https://discord.com/channels/1173363373115723796/1547134531188822047

Dominant language
TypeScript
Stars
481
Forks
45
Avg merge
14h 36m
Merged PRs (30d)
110

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from ai-driven-dev/framework

All issues in ai-driven-dev/framework

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.