section_present rejects numbered or annotated headings, and the optimizer 'fixes' it by forbidding the skill's own heading format
Nadie ha tomado este issue todavía.
Evaluación
- Dificultad
- 4/5
- Tiempo estimado
- 3-5 días
- Aptitud para principiantes
- 48/100
- Tipo de issue
- Error
- Claridad
- Bastante claro
- Estado de actividad
- Tranquilo
- Stack tecnológico
- python
- Área
- testing-qa
Línea de trabajo
Comienza en judges._section_present y reproduce la discrepancia hasta score_rule_judge con los dos encabezados mostrados en la issue. Revisa los Judge-Tests y Task-Sets existentes para determinar la semántica prevista de los encabezados y la compatibilidad del scoring; el trabajo estará terminado cuando el comportamiento elegido esté cubierto por pruebas de regresión sin romper las comprobaciones de Strict-Form existentes.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
Problem
judges._section_present anchors \s*$ immediately after the section name:
pat = re.compile(r"(?im)^\s{0,3}(#{1,6}\s*.*%s|\*\*.*%s.*\*\*\s*:?)\s*$" % (name, name))
So a heading passes only when the name is the last thing on the line:
1 ## Key Risks
1 **Key Risks:**
0 ### 1. Key Risks (Риски) — обзор ← numbered + translated + subtitle
0 ### Key Risks (Risks)
Numbered headings, bilingual headings, and Heading — subtitle are all common in real skill documents, and none of them survive.
Why this is more than cosmetic
My skill's own body demonstrates headings in exactly the rejected style, in its "structure" section:
### 1. Пратигья (Pratijñā) — Тезис
### 2. Хету (Hetu) — Причина
The model faithfully reproduced that format, so all five section_present checks failed on an otherwise correct answer (soft 0.55 with every content check passing). The optimizer then read those failures and proposed:
OVERRIDE: You MUST use EXACTLY these section titles […]. Do NOT append numbers, Latin transliterations, or descriptive text to the headings (e.g. output
### Пратигья, NOT### 1. Пратигья (Pratijñā) — Тезис).
The gate accepted it: 0.682 → 0.852. So a judge artifact produced a rule that forbids the format the skill itself teaches, and it would have been written into the skill permanently had I not read the per-task diff. As a side effect the same rule made the model drop a required citation from another task, which the mean hid (filed separately as the no-regression issue).
This is a concrete instance of the Goodhart pattern discussed in #154, arising purely from a strictness mismatch in one operator.
Possible directions
I did not send a patch because any change here alters scoring for existing task sets, so it seems like a maintainer call:
- Relax the anchor — allow trailing text after the name (
^#{1,6}[^\n]*<name>). Most faithful to what "section present" means, but existing sets that relied on the strict form would start passing more. - Add a
section_containsop and leavesection_presentuntouched. No behaviour change; authors opt in. - Document the strictness in the operator list, so authors know a numbered heading will not match.
My own workaround was replacing every section_present with (?im)^\s{0,3}(?:#{1,6}|\*\*)[^\n]*<name>, which behaves as I expected the built-in to.
Repro
from skillopt_sleep.judges import score_rule_judge
j = {"kind": "rule", "checks": [{"op": "section_present", "arg": "Key Risks"}]}
score_rule_judge(j, "## Key Risks")[0] # 1.0
score_rule_judge(j, "### 1. Key Risks (Риски) — обзор")[0] # 0.0
main @ fdeebaf, Python 3.11.
- Lenguaje dominante
- Python
- Estrellas
- 17.3k
- Forks
- 1.6k
- Merge medio
- 2 d 23 h
- PR fusionados (30 d)
- 9
Guía de contribución
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de microsoft/SkillOpt
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 84/100
-
项目还在迭代嘛? Abierto
Dificultad 5/5 Más de una semana Aptitud para principiantes 10/100
-
Dificultad 5/5 Más de una semana Aptitud para principiantes 42/100
-
Dificultad 3/5 1-2 días Aptitud para principiantes 72/100
-
Proposal: optional offline typed-decision judge for rubric scoring (SemIf / NanoJev pattern) Abierto
Dificultad 5/5 Más de una semana Aptitud para principiantes 28/100
Todos los issues de microsoft/SkillOpt
Issues similares
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 75/100
anthropics/skills#1811 · 1 comentario ·
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 75/100
speaches-ai/speaches#678 ·
-
bug
Dificultad 2/5 1-3 horas Aptitud para principiantes 75/100
datalayer/mcp-compose#42 ·
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 75/100
conda-forge/spacy-feedstock#177 ·
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 70/100
UKGovernmentBEIS/inspect_evals#2523 ·