Pin batch-size invariance of safety and judge generation in fp32 (`batch_size` 1 / 2 / 4 / 8)
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- Half a day
- Newbie friendliness
- 62/100
Research direction
Start with the batched generation helpers in forgelm/safety/_generate.py and forgelm/judge.py, then find the existing batch_size tests, which only check input validation. Add a tiny-model fp32 test that runs safety and judge generation at batch sizes 1, 2, 4 and 8 and asserts identical outputs. Done means that test passes, the row at docs/roadmap/risks-and-decisions.md:91 is removed, and the CHANGELOG names F-PR29-A8-10.
Written by the indexing model from the issue text.
Description
Deferred work ·
F-PR29-A8-10· severity Low · cost ~1-2 h · recorded indocs/roadmap/risks-and-decisions.md:91
Summary
Both gates generate the answers they score with left-padded, batched greedy decoding (forgelm/safety/_generate.py, forgelm/judge.py), but every test stubs the batched helpers, so a change that made results depend on batch_size would ship without a failing test. The PR #29 review deferred the determinism test. #677 shows bf16 output legitimately varies with batch size, so the contract to pin is fp32.
Condition or budget
None recorded (the row still names v0.6.x).
State today (re-checked 2026-10-09)
No determinism test exists; the only batch_size tests check input validation.
Related
#676 asks for a safety-side batched-versus-single test (and accepts a weaker padding-side fix for the judge); #677 documents the bf16 variance.
Acceptance criteria
- A tiny-model fp32 test runs safety generation and judge generation at
batch_size1, 2, 4 and 8 and asserts identical outputs. - The row is removed from the deferral table and the CHANGELOG names its ID (the table's removal contract).
Recorded in docs/roadmap/risks-and-decisions.md (read at f94595f) and re-checked against the code on 2026-10-09. The row leaves that table only when the fix is on main, the CHANGELOG names its ID and any promised guard exists.
- Dominant language
- Python
- Stars
- 9
- Forks
- 1
- Avg merge
- 8h 10m
- Merged PRs (30d)
- 5
Getting set up
- Ships a Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from HodeTech/ForgeLM
-
area: dev-tooling bug severity: low source: roadmap wave: 4
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
-
Site: the theme-locked --code-comment grey (#6a6f80) on the dark code surface is 3.8:1, below WCAG AA, in both themes (F-W1-SITE-06)Possibly taken @vortsghost2025 claimed this today. Openarea: site bug good first issue severity: low source: review-2026-09 wave: 4
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
-
documentation good first issue severity: medium source: review-2026-09 wave: 3
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
documentation good first issue severity: low source: review-2026-09 wave: 4
Difficulty 1/5 1-3 hours Newbie friendliness 80/100
-
documentation severity: medium source: review-2026-09 wave: 3
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
All issues in HodeTech/ForgeLM
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
task
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
danparshall/claude_researcher#87 ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
datajoint/datajoint-python#1586 ·
Maintainers usually reply within 1 day
-
1D intercept of length 1 is accepted as `fix_params` in population GLM with more than 1 unitPossibly taken @dhvani7014 claimed this today. Openbug good first issue
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
flatironinstitute/nemos#642 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 78/100
uniswap-python/uniswap-python#487 ·