test(songwriting): grade eight-plus-candidates with an llm judge in the rhyme menu cases
维护者通常 1 天内回复
已经有一个关联 PR 被合并了。
- #12376 来自 @kyle-sexton —— 已合并
评估
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 新手友好度
- 40/100
- Issue 类型
- 重构
- 描述清晰度
- 基本清楚
- 活跃度
- 活跃
- 技术栈
- yaml
- 领域
- testing-qa
调研方向
Start with the eight-plus-candidates regex grader in plugins/songwriting/evals/rh-window-menu-no-winner/case.yaml and plugins/songwriting/evals/rh-release-identity-menu/case.yaml, and look at how the existing llm graders are written in the repo's other eval cases. Keep every current regex sample as a labeled sample, add the three new shapes (tier table, several candidates per list line, inline comma-separated), and calibrate the judge against them. Done when the llm grader passes all labeled samples and the eight-plus check no longer depends on line shape.
由索引模型根据 Issue 内容生成。
描述
Observed
The eight-plus-candidates regex grader in plugins/songwriting/evals/rh-window-menu-no-winner/case.yaml and rh-release-identity-menu/case.yaml counts list lines and table rows, not candidates. A reply that offers eight or more candidates on fewer lines fails it.
In the #8431 item 2 measured pass (freeze edef3f69, Sonnet 5.5, --ablation none, 3 runs), it gave two false fails, each read in full against the reply:
- L1 (before) rh-release run 1: about 20 labeled candidates in a 4-row tier table.
- L2 (after) rh-release: 12 candidates on 3 list lines.
The grader checks a semantic count by line shape, the same pattern that made no-single-winner keep opening new edges (#12153, #12194, #12209; moved to an llm grader in #12195).
Fix wanted
Replace eight-plus-candidates in both cases with an llm grader: "the reply offers at least eight distinct rhyme candidates, however they are laid out (list, table, or inline)". Keep every current regex sample as a labeled sample, add the three shapes above (a tier table with several candidates per row, several candidates per list line, inline comma-separated candidates), and calibrate the judge on them, rather than another regex round.
Related
- #8431, #12195
- 主要语言
- Shell
- 星标
- 22
- 派生
- 2
- 平均合并
- 1 小时 29 分钟
- 30 天内合并 PR
- 895
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 没有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
melodic-software/claude-code-plugins 的其他 Issue
-
agent-ready priority: medium status: ready work-class: scoped
难度 1/5 1 小时以内 新手友好度 75/100
melodic-software/claude-code-plugins#12762 ·
维护者通常 1 天内回复
-
agent-ready priority: medium status: ready work-class: scoped
难度 2/5 1-3 小时 新手友好度 62/100
melodic-software/claude-code-plugins#12716 · 3 条评论 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 67/100
melodic-software/claude-code-plugins#9179 · 1 条评论 ·
维护者通常 1 天内回复
-
priority: low status: confirmed
难度 2/5 1-3 小时 新手友好度 72/100
melodic-software/claude-code-plugins#8799 ·
维护者通常 1 天内回复
-
needs-human priority: low work-class: read-only
难度 2/5 1-3 小时 新手友好度 62/100
melodic-software/claude-code-plugins#5696 · 1 条评论 ·
维护者通常 1 天内回复
查看 melodic-software/claude-code-plugins 的全部 Issue
相似的 Issue
-
难度 1/5 1 小时以内 新手友好度 85/100
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 65/100
openSUSE/rpm-config-SUSE#98 ·
-
package-update
难度 2/5 1-3 小时 新手友好度 70/100
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 75/100
ISKME/Open-Metadata-Exchange#425 ·
维护者通常 1 天内回复
-
难度 1/5 1 小时以内 新手友好度 82/100
suitenumerique/find#205 ·
维护者通常 2 天内回复