Live eval harness: run every skill's refusal evals against a model (split rubric, CI-gated)

Abierto
  • ¿Algo no está bien?
#61 7 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
20/100
Tipo de issue
Nueva funcionalidad
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
github-actions, python
Área
ai, ci-cd, security, testing-qa

Línea de trabajo

Start with scripts/test-ci-gates.sh, docs/roadmap.md, and the existing .claude/skills/*/evals/evals.json files to understand the current validation and eval formats. Then map out the schema changes, live eval runner, scheduled and dispatch workflow, and offline PR gate described in the issue. Done means all evals are covered, hard and soft axes use their specified checks, spend and judge separation are enforced, and a known-bad fixture fails the gate.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

enhancement

The suite's load-bearing safety property — per-skill refusal evals — is only shape-validated today (scripts/test-ci-gates.sh), never executed against a model. Turns docs/roadmap.md Loop 2 (deterministic gates for boundary/non-fabrication + a decorrelated LLM-judge for soft axes) into built infrastructure.

Where: new scripts/run-evals.* consuming every .claude/skills/*/evals/evals.json; a scheduled/dispatch workflow; a schema extension (axis tag hard|soft + a machine-checkable predicate per hard expectation) across all evals + the shape validator.
Done when: harness runs each eval prompt against a configured subject model; deterministic predicate on hard axes, LLM-judge on soft only; aborts when judge==subject model id (decorrelation by identifier, no vendor name); a per-run spend cap is enforced before each call; writes a dated results artifact (per-eval pass/fail, model ids, spend); model choice in env/config only — no vendor/model name in any skill's prose.
Verify: live run on schedule/dispatch against main (where the secret lives), never on PR; a separate deterministic offline PR gate asserts the results artifact exists, is not stale, and covers every eval id — failing closed (not skip-when-key-absent). A golden known-bad fixture must be rejected by its predicate, proving the gate can fail.
Why: today a skill could silently start fabricating a TAM, CWE, or compliance verdict and every gate stays green. Single biggest hole in the safety story. Grounded by two independent review agents this session.

Lenguaje dominante
Python
Estrellas
2
Forks
1
Merge medio
1 h 29 min
PR fusionados (30 d)
551

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de protocol/perun

Todos los issues de protocol/perun

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.