Pin batch-size invariance of safety and judge generation in fp32 (`batch_size` 1 / 2 / 4 / 8)
Nessuno ha ancora preso questa issue.
Valutazione
- Difficoltà
- 3/5
- Tempo stimato
- Mezza giornata
- Idoneità per principianti
- 62/100
Direzione di ricerca
Start with the batched generation helpers in forgelm/safety/_generate.py and forgelm/judge.py, then find the existing batch_size tests, which only check input validation. Add a tiny-model fp32 test that runs safety and judge generation at batch sizes 1, 2, 4 and 8 and asserts identical outputs. Done means that test passes, the row at docs/roadmap/risks-and-decisions.md:91 is removed, and the CHANGELOG names F-PR29-A8-10.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Deferred work ·
F-PR29-A8-10· severity Low · cost ~1-2 h · recorded indocs/roadmap/risks-and-decisions.md:91
Summary
Both gates generate the answers they score with left-padded, batched greedy decoding (forgelm/safety/_generate.py, forgelm/judge.py), but every test stubs the batched helpers, so a change that made results depend on batch_size would ship without a failing test. The PR #29 review deferred the determinism test. #677 shows bf16 output legitimately varies with batch size, so the contract to pin is fp32.
Condition or budget
None recorded (the row still names v0.6.x).
State today (re-checked 2026-10-09)
No determinism test exists; the only batch_size tests check input validation.
Related
#676 asks for a safety-side batched-versus-single test (and accepts a weaker padding-side fix for the judge); #677 documents the bf16 variance.
Acceptance criteria
- A tiny-model fp32 test runs safety generation and judge generation at
batch_size1, 2, 4 and 8 and asserts identical outputs. - The row is removed from the deferral table and the CHANGELOG names its ID (the table's removal contract).
Recorded in docs/roadmap/risks-and-decisions.md (read at f94595f) and re-checked against the code on 2026-10-09. The row leaves that table only when the fix is on main, the CHANGELOG names its ID and any promised guard exists.
- Lingua principale
- Python
- Stelle
- 9
- Fork
- 1
- Merge medio
- 4h 3m
- PR unite (30g)
- 3
Preparare l'ambiente
- Include un Dockerfile o un file Docker Compose
- Ha un modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di HodeTech/ForgeLM
-
area: dev-tooling bug severity: low source: roadmap wave: 4
Difficoltà 2/5 1-3 ore Idoneità per principianti 70/100
-
area: site bug good first issue severity: low source: review-2026-09 wave: 4
Difficoltà 2/5 1-3 ore Idoneità per principianti 66/100
-
documentation good first issue severity: medium source: review-2026-09 wave: 3
Difficoltà 2/5 1-3 ore Idoneità per principianti 75/100
-
documentation good first issue severity: low source: review-2026-09 wave: 4
Difficoltà 1/5 1-3 ore Idoneità per principianti 80/100
-
documentation severity: medium source: review-2026-09 wave: 3
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
Tutte le issue di HodeTech/ForgeLM
Issue simili
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 63/100
GAM-team/got-your-back#547 ·
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 75/100
UKGovernmentBEIS/inspect_ai#5821 ·
I maintainer di solito rispondono entro 1 giorno
-
docs help wanted
Difficoltà 1/5 1-3 ore Idoneità per principianti 75/100
-
evidence wanted good first issue help wanted
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
open-coder-ai/agentseam#195 ·
I maintainer di solito rispondono entro 1 giorno
-
bug
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
I maintainer di solito rispondono entro 1 giorno