Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

test(planning): run the seven interview eval-first cases and land the rules whose case fails

Aperta
  • Qualcosa non va?
#7,269 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
35/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
github, shell
Ambito
testing-qa

Direzione di ricerca

Start with the /evals:plugin-eval Cost guidance and inspect PR #7247 to identify the seven eval cases. Run cases one or two at a time against a copy of plugins/planning outside git, then use run-validity.py and noise-report.py to review valid results; inspect kept traces for the two cases that prohibit Bash. Done means rules are added to the interview skill's context spokes only for cases that fail on main, with each change citing its result.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

needs-triage

Summary

PR #7247 adds seven eval-first cases for /planning:interview behaviors, a to g: read-back of a dictated answer, active-reply final confirmation, changes-only final brief, a checkable basis and flip condition on each recommendation, the "something else" closing probe, a MoSCoW Must-test scope cut before /planning:wayfind, and one crux question on a rejected recommendation. Under Instruction economy, the rule for each case lands only if its case fails on main.

State

  • All seven llm judges were calibrated against an opus judge at 100% agreement.
  • The first full pass stopped at its cost ceiling (partial: true). Five cases finished, interview-recommendation-basis-and-flip finished partly, and interview-scope-cut-moscow did not run. Under the plugin-eval reading rules, a partial run's scores are not comparable, so they are not used.
  • Per-run cost came out well above the fresh-suite headroom estimate. Size the next pass from that pass's own costUsd (see /evals:plugin-eval, Cost).

Fix

  1. Run the cases one or two at a time (--case <name>, --runs 3, --judge-model opus, --keep-temp) on a copy of plugins/planning outside git, because the repository has too many linked worktrees for the CLI to screen.
  2. Read each result with run-validity.py and noise-report.py. For final-confirm-active-reply and final-brief-changes-only, read the kept traces: those cases allow no Bash, so the Step 3 register-gate script cannot run in them.
  3. For each case that fails on main, add its rule to the interview skill's context spokes in a PR that cites the result.

Related

  • #7247
  • #6986 and #7036 (the eval-first precedent)
Lingua principale
Shell
Stelle
22
Fork
2
Merge medio
7h 42m
PR unite (30g)
923

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di melodic-software/claude-code-plugins

Tutte le issue di melodic-software/claude-code-plugins

Issue simili

Altre issue su Shell/Bash

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.