test(songwriting): grade eight-plus-candidates with an llm judge in the rhyme menu cases

Cerrado
  • ¿Algo no está bien?
#12,299 4 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Ya se ha fusionado un pull request relacionado.

  • #12376 de @kyle-sexton — fusionado

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
40/100
Tipo de issue
Refactorización
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
yaml
Área
testing-qa

Línea de trabajo

Start with the eight-plus-candidates regex grader in plugins/songwriting/evals/rh-window-menu-no-winner/case.yaml and plugins/songwriting/evals/rh-release-identity-menu/case.yaml, and look at how the existing llm graders are written in the repo's other eval cases. Keep every current regex sample as a labeled sample, add the three new shapes (tier table, several candidates per list line, inline comma-separated), and calibrate the judge against them. Done when the llm grader passes all labeled samples and the eight-plus check no longer depends on line shape.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

admission-deferred priority: medium status: ready work-class: scoped

Observed

The eight-plus-candidates regex grader in plugins/songwriting/evals/rh-window-menu-no-winner/case.yaml and rh-release-identity-menu/case.yaml counts list lines and table rows, not candidates. A reply that offers eight or more candidates on fewer lines fails it.

In the #8431 item 2 measured pass (freeze edef3f69, Sonnet 5.5, --ablation none, 3 runs), it gave two false fails, each read in full against the reply:

  • L1 (before) rh-release run 1: about 20 labeled candidates in a 4-row tier table.
  • L2 (after) rh-release: 12 candidates on 3 list lines.

The grader checks a semantic count by line shape, the same pattern that made no-single-winner keep opening new edges (#12153, #12194, #12209; moved to an llm grader in #12195).

Fix wanted

Replace eight-plus-candidates in both cases with an llm grader: "the reply offers at least eight distinct rhyme candidates, however they are laid out (list, table, or inline)". Keep every current regex sample as a labeled sample, add the three shapes above (a tier table with several candidates per row, several candidates per list line, inline comma-separated candidates), and calibrate the judge on them, rather than another regex round.

Related

  • #8431, #12195
Lenguaje dominante
Shell
Estrellas
22
Forks
2
Merge medio
1 h 29 min
PR fusionados (30 d)
895

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de melodic-software/claude-code-plugins

Todos los issues de melodic-software/claude-code-plugins

Issues similares

Más issues de Shell/Bash

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.