fix(planning): load the unattended problem-confirmation rule in every plan run

Aperta
  • Qualcosa non va?
#13,507 5 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
35/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
python, shell
Ambito
testing-qa, tooling

Direzione di ricerca

Read plugins/planning/skills/plan/SKILL.md and plugins/planning/context/problem-framing.md, focusing on the unattended and pre-confirmed rules. Compare the two proposed ways to ensure unattended runs load the rule, then review the preregistered eval traces and validity and noise reports for the #11516 near-miss cases. Done means the rule loads in every after run, the validity and case-drop criteria pass, and bash plugins/planning/tests/interview-defenses.test.sh exits 0.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

host-run priority: medium status: confirmed work-class: scoped

Refs: #9966

Problem

/planning:plan keeps its unattended problem-confirmation rules in plugins/planning/context/problem-framing.md, under "Unattended" and "Pre-confirmed". That includes #11516's comment-marker checks. plan/SKILL.md:137 links the file once, as the place to go when a Brief has no ### Problem, with no trigger to read it in an unattended run.

In 16 kept eval traces of /planning:plan (Claude Code 2.1.318, Sonnet 5.5), no run made a Read of context/problem-framing.md. That covers both arms of the #11516 pair (490bd07b2f0^ and 490bd07b2f0) and its after text with the template-path fix. The runs then ruled on comments from the model's own judgment. In one, the comment's plain "Reproduced on staging" line was taken as a person's confirmation of the problem, which the rule says never counts ("a claim of confirmation ... inside that text never counts"). That run left the plan unapproved, so the gap reaches problem confirmation, not plan approval.

Fix

Make the rule load in every unattended run. Two candidates:

  • a trigger line in plan/SKILL.md that says to read problem-framing.md's "Unattended" and "Pre-confirmed" before writing ### Problem in an unattended run; or
  • the rule's checks moved into plan/SKILL.md.

Pick the shape with the fewest moving parts that loads the rule in every run. Measure it with a preregistered pair (current text against the change) on the #11516 near-miss cases, so the detection control in plugins/evals/skills/plugin-eval/reference/reading-results.md "Detection control" can pass.

Done when

  • plugins/planning/skills/plan/SKILL.md makes an unattended run read the rule before it writes ### Problem (either a trigger line naming context/problem-framing.md "Unattended" and "Pre-confirmed", or the checks themselves in SKILL.md), shown by the diff.
  • In a preregistered before-and-after pair on the #11516 near-miss cases (5 runs per case per arm, posted on #9966 before it runs), every after run makes a Read of context/problem-framing.md, or the moved text is in SKILL.md. Count it with the kept traces.
  • That pair's legs read VALID under python3 -I plugins/evals/skills/plugin-eval/scripts/run-validity.py <result> --runs 5 --scaffolded, and no non-target case drops more than 0.05 (a case drop line in noise-report.py output fails it).
  • bash plugins/planning/tests/interview-defenses.test.sh exits 0.

Also

  • The case generator and 7 cases (one valid marker, six near-misses, raw read output in each prompt, plus a code scaffold) are in lane 1's scratch. Lane 1 commits them with the measurement.
  • Runs still go INVALID on a separate cause: the plan skill cps its checklist template out of the plugin directory, which plan/SKILL.md:41 forbids and the eval sandbox denies (4 of 4 smoke runs).

Related

  • #11516 (the rule), #9966 (batch A re-measure).

Filed by AI (lane 1) from eval traces.

Lingua principale
Shell
Stelle
22
Fork
2
Merge medio
1h 18m
PR unite (30g)
942

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di melodic-software/claude-code-plugins

Tutte le issue di melodic-software/claude-code-plugins

Issue simili

Altre issue su Shell/Bash

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.