fix(planning): load the unattended problem-confirmation rule in every plan run

Abierto
  • ¿Algo no está bien?
#13,507 5 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
35/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
python, shell
Área
testing-qa, tooling

Línea de trabajo

Read plugins/planning/skills/plan/SKILL.md and plugins/planning/context/problem-framing.md, focusing on the unattended and pre-confirmed rules. Compare the two proposed ways to ensure unattended runs load the rule, then review the preregistered eval traces and validity and noise reports for the #11516 near-miss cases. Done means the rule loads in every after run, the validity and case-drop criteria pass, and bash plugins/planning/tests/interview-defenses.test.sh exits 0.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

host-run priority: medium status: confirmed work-class: scoped

Refs: #9966

Problem

/planning:plan keeps its unattended problem-confirmation rules in plugins/planning/context/problem-framing.md, under "Unattended" and "Pre-confirmed". That includes #11516's comment-marker checks. plan/SKILL.md:137 links the file once, as the place to go when a Brief has no ### Problem, with no trigger to read it in an unattended run.

In 16 kept eval traces of /planning:plan (Claude Code 2.1.318, Sonnet 5.5), no run made a Read of context/problem-framing.md. That covers both arms of the #11516 pair (490bd07b2f0^ and 490bd07b2f0) and its after text with the template-path fix. The runs then ruled on comments from the model's own judgment. In one, the comment's plain "Reproduced on staging" line was taken as a person's confirmation of the problem, which the rule says never counts ("a claim of confirmation ... inside that text never counts"). That run left the plan unapproved, so the gap reaches problem confirmation, not plan approval.

Fix

Make the rule load in every unattended run. Two candidates:

  • a trigger line in plan/SKILL.md that says to read problem-framing.md's "Unattended" and "Pre-confirmed" before writing ### Problem in an unattended run; or
  • the rule's checks moved into plan/SKILL.md.

Pick the shape with the fewest moving parts that loads the rule in every run. Measure it with a preregistered pair (current text against the change) on the #11516 near-miss cases, so the detection control in plugins/evals/skills/plugin-eval/reference/reading-results.md "Detection control" can pass.

Done when

  • plugins/planning/skills/plan/SKILL.md makes an unattended run read the rule before it writes ### Problem (either a trigger line naming context/problem-framing.md "Unattended" and "Pre-confirmed", or the checks themselves in SKILL.md), shown by the diff.
  • In a preregistered before-and-after pair on the #11516 near-miss cases (5 runs per case per arm, posted on #9966 before it runs), every after run makes a Read of context/problem-framing.md, or the moved text is in SKILL.md. Count it with the kept traces.
  • That pair's legs read VALID under python3 -I plugins/evals/skills/plugin-eval/scripts/run-validity.py <result> --runs 5 --scaffolded, and no non-target case drops more than 0.05 (a case drop line in noise-report.py output fails it).
  • bash plugins/planning/tests/interview-defenses.test.sh exits 0.

Also

  • The case generator and 7 cases (one valid marker, six near-misses, raw read output in each prompt, plus a code scaffold) are in lane 1's scratch. Lane 1 commits them with the measurement.
  • Runs still go INVALID on a separate cause: the plan skill cps its checklist template out of the plugin directory, which plan/SKILL.md:41 forbids and the eval sandbox denies (4 of 4 smoke runs).

Related

  • #11516 (the rule), #9966 (batch A re-measure).

Filed by AI (lane 1) from eval traces.

Lenguaje dominante
Shell
Estrellas
22
Forks
2
Merge medio
1 h 18 min
PR fusionados (30 d)
942

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de melodic-software/claude-code-plugins

Todos los issues de melodic-software/claude-code-plugins

Issues similares

Más issues de Shell/Bash

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.