[32B Dense] Debug 64k increasing loss
@luukkonenr ci sta già lavorando.
Dal 2/9/2026.
Valutazione
Questa issue non è ancora stata valutata.
Descrizione
While training the 32B dense flag model we encountered an increasing loss around 64k:
See:
From: https://wandb.ai/openeurollm-project/oellm_32b_dense
For live debug session: https://github.com/OpenEuroLLM/oellm-autoexp/blob/prod/oellm_32b_dense/docs/64k-debug/DEBUG.md
The Problem:
The training loss was going down like it should. Then around iteration 65,000
it turned around and started going up, and it keeps getting worse.
- Run:
oellm_32b_dense_prod_dataopt5_gbs4096_lr3e-4 - Last job:
1537344→
wandb,
stopped 2026-08-30 22:43 at iteration 75,125 - Training loss: 1.5434 at ~65,000 → 1.587 at 75,125
This is not a wobble. It is a steady climb that is speeding up — all of it visible
in job 1537344
(wandb, which
covers 66,626 → 75,125, i.e. the whole broken stretch):
| iterations | loss change per 1000 steps | where to see it |
|---|---|---|
| 50k–56k | −0.0018 (good, going down) | job 1530865 |
| 62k–66.6k | −0.0005 (nearly stopped) | jobs 1535676, 1536694 |
| 66.6k–71k | +0.0023 (going up) | 1537344 |
| 71k–75k | +0.0062 (going up faster) | 1537344 |
Why this is a big deal: we are only 8.4% into a 15T-token run
(75,125 of 894,000 iterations, ~1.26T tokens). If the model gets worse the longer
we train, the other 92% cannot be run this way. Rough cost so far: ~11,000
iterations spent making it worse ≈ 185B tokens ≈ 27,500 GPU-h.
How we found the exact spot
Fitting the healthy part of the curve (12k–34k) and extrapolating, the run stays
on track until ~66,000 and then breaks away: +0.012 at 68k, +0.024 at 72k,
+0.038 at 74k. The break is at ~66,625.
Held-out scores confirm the model really got worse (lower = better):
| checkpoint | score | job |
|---|---|---|
| 60,000 | 1.500890 | 1548575 |
| 64,000 | 1.495481 ← best | 1573075 |
| 68,000 | 1.511213 | 1563495 |
| 72,000 | 1.519173 | 1543640 |
Normally later = better. Here it flips after 64,000. That flip is the bug.
This could be due to a variety of things
- FP8
- NaN weights/activations
- Switching containers
- Turning on inter-document attention masking
- Data quality
- RMSNorm off-zero gamma
- Lingua principale
- Nessun dato sulla lingua
- Stelle
- 3
- Fork
- 0
- Metriche di merge delle PR
- Nessuna PR unita negli ultimi 30g
Preparare l'ambiente
Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di OpenEuroLLM/Taskboard
-
[nucleus sparse] Experiment setupForse già presa @aaronkl l’ha presa 8 giorni fa. ApertaT4.1 - Optimization HPC T4.2 - model exploration
OpenEuroLLM/Taskboard#439 · 1 reazione · 1 assegnatario ·
-
Qwen3.5 35B-A3B hybrid MoE weak and strong scalingForse già presa @ibaucells l’ha presa 8 giorni fa. ApertaT4.1 - Optimization HPC
OpenEuroLLM/Taskboard#438 · 2 commenti · 1 assegnatario ·
-
Synthetic Generation for Multilingual Instruction TuningForse già presa @abhash-er l’ha presa 9 giorni fa. Aperta4.6 post-training
OpenEuroLLM/Taskboard#437 · 1 assegnatario ·
-
Safety and alignment evaluationForse già presa @BirgerMoell l’ha presa 9 giorni fa. Aperta4.6 post-training
OpenEuroLLM/Taskboard#436 · 1 assegnatario ·
-
Finding optimal dataset mixtures for DPOForse già presa @geoalgo l’ha presa 9 giorni fa. Aperta4.6 post-training
OpenEuroLLM/Taskboard#435 · 3 assegnatari ·