Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

test(planning): run the seven interview eval-first cases and land the rules whose case fails

Abierto
  • ¿Algo no está bien?
#7,269 2 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
35/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
github, shell
Área
testing-qa

Línea de trabajo

Start with the /evals:plugin-eval Cost guidance and inspect PR #7247 to identify the seven eval cases. Run cases one or two at a time against a copy of plugins/planning outside git, then use run-validity.py and noise-report.py to review valid results; inspect kept traces for the two cases that prohibit Bash. Done means rules are added to the interview skill's context spokes only for cases that fail on main, with each change citing its result.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

needs-triage

Summary

PR #7247 adds seven eval-first cases for /planning:interview behaviors, a to g: read-back of a dictated answer, active-reply final confirmation, changes-only final brief, a checkable basis and flip condition on each recommendation, the "something else" closing probe, a MoSCoW Must-test scope cut before /planning:wayfind, and one crux question on a rejected recommendation. Under Instruction economy, the rule for each case lands only if its case fails on main.

State

  • All seven llm judges were calibrated against an opus judge at 100% agreement.
  • The first full pass stopped at its cost ceiling (partial: true). Five cases finished, interview-recommendation-basis-and-flip finished partly, and interview-scope-cut-moscow did not run. Under the plugin-eval reading rules, a partial run's scores are not comparable, so they are not used.
  • Per-run cost came out well above the fresh-suite headroom estimate. Size the next pass from that pass's own costUsd (see /evals:plugin-eval, Cost).

Fix

  1. Run the cases one or two at a time (--case <name>, --runs 3, --judge-model opus, --keep-temp) on a copy of plugins/planning outside git, because the repository has too many linked worktrees for the CLI to screen.
  2. Read each result with run-validity.py and noise-report.py. For final-confirm-active-reply and final-brief-changes-only, read the kept traces: those cases allow no Bash, so the Step 3 register-gate script cannot run in them.
  3. For each case that fails on main, add its rule to the interview skill's context spokes in a PR that cites the result.

Related

  • #7247
  • #6986 and #7036 (the eval-first precedent)
Lenguaje dominante
Shell
Estrellas
22
Forks
2
Merge medio
6 h 42 min
PR fusionados (30 d)
902

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de melodic-software/claude-code-plugins

Todos los issues de melodic-software/claude-code-plugins

Issues similares

Más issues de Shell/Bash

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.