test(planning): run the seven interview eval-first cases and land the rules whose case fails

Geschlossen
  • Stimmt etwas nicht?
#7,269 2 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 1 Tag

Dieses Issue hat noch niemand übernommen.

Bewertung

Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Anfängerfreundlichkeit
35/100
Issue-Typ
Bug
Klarheit
Größtenteils klar
Aktivitätsstatus
Aktiv
Tech-Stack
github, shell
Bereich
testing-qa

Rechercherichtung

Start with the /evals:plugin-eval Cost guidance and inspect PR #7247 to identify the seven eval cases. Run cases one or two at a time against a copy of plugins/planning outside git, then use run-validity.py and noise-report.py to review valid results; inspect kept traces for the two cases that prohibit Bash. Done means rules are added to the interview skill's context spokes only for cases that fail on main, with each change citing its result.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

needs-triage

Summary

PR #7247 adds seven eval-first cases for /planning:interview behaviors, a to g: read-back of a dictated answer, active-reply final confirmation, changes-only final brief, a checkable basis and flip condition on each recommendation, the "something else" closing probe, a MoSCoW Must-test scope cut before /planning:wayfind, and one crux question on a rejected recommendation. Under Instruction economy, the rule for each case lands only if its case fails on main.

State

  • All seven llm judges were calibrated against an opus judge at 100% agreement.
  • The first full pass stopped at its cost ceiling (partial: true). Five cases finished, interview-recommendation-basis-and-flip finished partly, and interview-scope-cut-moscow did not run. Under the plugin-eval reading rules, a partial run's scores are not comparable, so they are not used.
  • Per-run cost came out well above the fresh-suite headroom estimate. Size the next pass from that pass's own costUsd (see /evals:plugin-eval, Cost).

Fix

  1. Run the cases one or two at a time (--case <name>, --runs 3, --judge-model opus, --keep-temp) on a copy of plugins/planning outside git, because the repository has too many linked worktrees for the CLI to screen.
  2. Read each result with run-validity.py and noise-report.py. For final-confirm-active-reply and final-brief-changes-only, read the kept traces: those cases allow no Bash, so the Step 3 register-gate script cannot run in them.
  3. For each case that fails on main, add its rule to the interview skill's context spokes in a PR that cites the result.

Related

  • #7247
  • #6986 and #7036 (the eval-first precedent)
Vorherrschende Sprache
Shell
Sterne
22
Forks
2
Ø Merge
1 Std. 38 Min.
Gemergte PRs (30 T.)
911

Entwicklungsumgebung

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus melodic-software/claude-code-plugins

Alle Issues in melodic-software/claude-code-plugins

Ähnliche Issues

Weitere Issues zu Shell/Bash

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.