test(planning): run the seven interview eval-first cases and land the rules whose case fails
维护者通常 1 天内回复
还没有人认领这个 Issue。
评估
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 新手友好度
- 35/100
- Issue 类型
- 缺陷
- 描述清晰度
- 基本清楚
- 活跃度
- 活跃
- 技术栈
- github, shell
- 领域
- testing-qa
调研方向
Start with the /evals:plugin-eval Cost guidance and inspect PR #7247 to identify the seven eval cases. Run cases one or two at a time against a copy of plugins/planning outside git, then use run-validity.py and noise-report.py to review valid results; inspect kept traces for the two cases that prohibit Bash. Done means rules are added to the interview skill's context spokes only for cases that fail on main, with each change citing its result.
由索引模型根据 Issue 内容生成。
描述
Summary
PR #7247 adds seven eval-first cases for /planning:interview behaviors, a to g: read-back of a dictated answer, active-reply final confirmation, changes-only final brief, a checkable basis and flip condition on each recommendation, the "something else" closing probe, a MoSCoW Must-test scope cut before /planning:wayfind, and one crux question on a rejected recommendation. Under Instruction economy, the rule for each case lands only if its case fails on main.
State
- All seven llm judges were calibrated against an opus judge at 100% agreement.
- The first full pass stopped at its cost ceiling (
partial: true). Five cases finished,interview-recommendation-basis-and-flipfinished partly, andinterview-scope-cut-moscowdid not run. Under the plugin-eval reading rules, a partial run's scores are not comparable, so they are not used. - Per-run cost came out well above the fresh-suite headroom estimate. Size the next pass from that pass's own
costUsd(see/evals:plugin-eval, Cost).
Fix
- Run the cases one or two at a time (
--case <name>,--runs 3,--judge-model opus,--keep-temp) on a copy ofplugins/planningoutside git, because the repository has too many linked worktrees for the CLI to screen. - Read each result with
run-validity.pyandnoise-report.py. Forfinal-confirm-active-replyandfinal-brief-changes-only, read the kept traces: those cases allow no Bash, so the Step 3 register-gate script cannot run in them. - For each case that fails on main, add its rule to the interview skill's context spokes in a PR that cites the result.
Related
- #7247
- #6986 and #7036 (the eval-first precedent)
- 主要语言
- Shell
- 星标
- 22
- 派生
- 2
- 平均合并
- 7 小时 42 分钟
- 30 天内合并 PR
- 923
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 没有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
melodic-software/claude-code-plugins 的其他 Issue
-
priority: medium
难度 2/5 1-3 小时 新手友好度 68/100
melodic-software/claude-code-plugins#7351 · 2 条评论 ·
维护者通常 1 天内回复
-
priority: high
难度 2/5 1-3 小时 新手友好度 74/100
melodic-software/claude-code-plugins#7312 · 2 条评论 ·
维护者通常 1 天内回复
-
priority: medium
难度 2/5 1-3 小时 新手友好度 76/100
melodic-software/claude-code-plugins#7288 · 2 条评论 ·
维护者通常 1 天内回复
-
knowledge: docpage-digest dual verification defines no verdict vocabulary or pass/fail threshold未关闭needs-human
难度 2/5 1-3 小时 新手友好度 78/100
melodic-software/claude-code-plugins#7286 · 2 条评论 ·
维护者通常 1 天内回复
-
priority: low
难度 2/5 1-3 小时 新手友好度 76/100
melodic-software/claude-code-plugins#7282 · 2 条评论 ·
维护者通常 1 天内回复
查看 melodic-software/claude-code-plugins 的全部 Issue
相似的 Issue
-
难度 1/5 1 小时以内 新手友好度 82/100
-
[Bug]: Three default prefs with JSON values are never set (override.sh escapes their quotes twice)未关闭bug
难度 2/5 1-3 小时 新手友好度 78/100
Floorp-Projects/Floorp#2913 ·
维护者通常 1 天内回复
-
darwin zsh
难度 1/5 1 小时以内 新手友好度 90/100
carapace-sh/carapace-bin#3846 ·
维护者通常 1 天内回复
-
out-of-date
难度 1/5 1 小时以内 新手友好度 65/100
CachyOS/CachyOS-PKGBUILDS#1981 ·
维护者通常 1 天内回复
-
area: backend bug
难度 2/5 1-3 小时 新手友好度 68/100
snapotter-hq/SnapOtter#2415 ·
维护者通常 1 天内回复