Pin batch-size invariance of safety and judge generation in fp32 (`batch_size` 1 / 2 / 4 / 8)
Nadie ha tomado este issue todavía.
Evaluación
- Dificultad
- 3/5
- Tiempo estimado
- Medio día
- Aptitud para principiantes
- 62/100
Línea de trabajo
Start with the batched generation helpers in forgelm/safety/_generate.py and forgelm/judge.py, then find the existing batch_size tests, which only check input validation. Add a tiny-model fp32 test that runs safety and judge generation at batch sizes 1, 2, 4 and 8 and asserts identical outputs. Done means that test passes, the row at docs/roadmap/risks-and-decisions.md:91 is removed, and the CHANGELOG names F-PR29-A8-10.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
Deferred work ·
F-PR29-A8-10· severity Low · cost ~1-2 h · recorded indocs/roadmap/risks-and-decisions.md:91
Summary
Both gates generate the answers they score with left-padded, batched greedy decoding (forgelm/safety/_generate.py, forgelm/judge.py), but every test stubs the batched helpers, so a change that made results depend on batch_size would ship without a failing test. The PR #29 review deferred the determinism test. #677 shows bf16 output legitimately varies with batch size, so the contract to pin is fp32.
Condition or budget
None recorded (the row still names v0.6.x).
State today (re-checked 2026-10-09)
No determinism test exists; the only batch_size tests check input validation.
Related
#676 asks for a safety-side batched-versus-single test (and accepts a weaker padding-side fix for the judge); #677 documents the bf16 variance.
Acceptance criteria
- A tiny-model fp32 test runs safety generation and judge generation at
batch_size1, 2, 4 and 8 and asserts identical outputs. - The row is removed from the deferral table and the CHANGELOG names its ID (the table's removal contract).
Recorded in docs/roadmap/risks-and-decisions.md (read at f94595f) and re-checked against the code on 2026-10-09. The row leaves that table only when the fix is on main, the CHANGELOG names its ID and any promised guard exists.
- Lenguaje dominante
- Python
- Estrellas
- 9
- Forks
- 1
- Merge medio
- 8 h 10 min
- PR fusionados (30 d)
- 5
Preparar el entorno
- Incluye un Dockerfile o un archivo de Docker Compose
- Tiene una plantilla de pull request
- Leer la guía de contribución
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de HodeTech/ForgeLM
-
area: dev-tooling bug severity: low source: roadmap wave: 4
Dificultad 2/5 1-3 horas Aptitud para principiantes 70/100
-
Site: the theme-locked --code-comment grey (#6a6f80) on the dark code surface is 3.8:1, below WCAG AA, in both themes (F-W1-SITE-06)Posiblemente ocupada @vortsghost2025 la tomó hace 1 día. Abiertoarea: site bug good first issue severity: low source: review-2026-09 wave: 4
Dificultad 2/5 1-3 horas Aptitud para principiantes 66/100
-
Choosing-a-trainer table documents KTO rows as {prompt, response, label}, but the KTO loader requires a completion column (F-W1-DOCS-TRAIN-02)Posiblemente ocupada @vortsghost2025 la tomó hoy. Abiertodocumentation good first issue severity: medium source: review-2026-09 wave: 3
Dificultad 2/5 1-3 horas Aptitud para principiantes 75/100
-
documentation good first issue severity: low source: review-2026-09 wave: 4
Dificultad 1/5 1-3 horas Aptitud para principiantes 80/100
-
documentation severity: medium source: review-2026-09 wave: 3
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
Todos los issues de HodeTech/ForgeLM
Issues similares
-
md/asciidoc/csv: an explicit `encoding="utf-8"` keeps a leading BOM, the auto-detect path drops itAbierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 82/100
docling-project/docling#4846 ·
Los mantenedores suelen responder en 1 día
-
area/sessions comp/dashboard comp/desktop comp/gateway comp/tui duplicate P2 sweeper:risk-session-state type/bug
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
NousResearch/hermes-agent#138914 · 1 comentario ·
Los mantenedores suelen responder en 1 día
-
bot-triaged module: activation checkpointing module: inductor oncall: pt2 triaged
Dificultad 2/5 1-3 horas Aptitud para principiantes 62/100
Los mantenedores suelen responder en 1 día
-
[oblt-aw][autodoc] Update PR Actions Detective E2E prerequisites and fix dashboard-format table typoAbiertodocumentation
Dificultad 2/5 1-3 horas Aptitud para principiantes 75/100
elastic/oblt-aw#2382 · 1 comentario ·
Los mantenedores suelen responder en 1 día
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 74/100
Los mantenedores suelen responder en 1 día