[32B Dense] Debug 64k increasing loss
@luukkonenr ya está trabajando en esto.
Desde el 2/9/2026.
Evaluación
Este issue todavía no se ha evaluado.
Descripción
While training the 32B dense flag model we encountered an increasing loss around 64k:
See:
From: https://wandb.ai/openeurollm-project/oellm_32b_dense
For live debug session: https://github.com/OpenEuroLLM/oellm-autoexp/blob/prod/oellm_32b_dense/docs/64k-debug/DEBUG.md
The Problem:
The training loss was going down like it should. Then around iteration 65,000
it turned around and started going up, and it keeps getting worse.
- Run:
oellm_32b_dense_prod_dataopt5_gbs4096_lr3e-4 - Last job:
1537344→
wandb,
stopped 2026-08-30 22:43 at iteration 75,125 - Training loss: 1.5434 at ~65,000 → 1.587 at 75,125
This is not a wobble. It is a steady climb that is speeding up — all of it visible
in job 1537344
(wandb, which
covers 66,626 → 75,125, i.e. the whole broken stretch):
| iterations | loss change per 1000 steps | where to see it |
|---|---|---|
| 50k–56k | −0.0018 (good, going down) | job 1530865 |
| 62k–66.6k | −0.0005 (nearly stopped) | jobs 1535676, 1536694 |
| 66.6k–71k | +0.0023 (going up) | 1537344 |
| 71k–75k | +0.0062 (going up faster) | 1537344 |
Why this is a big deal: we are only 8.4% into a 15T-token run
(75,125 of 894,000 iterations, ~1.26T tokens). If the model gets worse the longer
we train, the other 92% cannot be run this way. Rough cost so far: ~11,000
iterations spent making it worse ≈ 185B tokens ≈ 27,500 GPU-h.
How we found the exact spot
Fitting the healthy part of the curve (12k–34k) and extrapolating, the run stays
on track until ~66,000 and then breaks away: +0.012 at 68k, +0.024 at 72k,
+0.038 at 74k. The break is at ~66,625.
Held-out scores confirm the model really got worse (lower = better):
| checkpoint | score | job |
|---|---|---|
| 60,000 | 1.500890 | 1548575 |
| 64,000 | 1.495481 ← best | 1573075 |
| 68,000 | 1.511213 | 1563495 |
| 72,000 | 1.519173 | 1543640 |
Normally later = better. Here it flips after 64,000. That flip is the bug.
This could be due to a variety of things
- FP8
- NaN weights/activations
- Switching containers
- Turning on inter-document attention masking
- Data quality
- RMSNorm off-zero gamma
- Lenguaje dominante
- Sin datos de lenguaje
- Estrellas
- 3
- Forks
- 0
- Métricas de merge de PR
- Sin PR fusionados en 30 d
Preparar el entorno
Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de OpenEuroLLM/Taskboard
-
[nucleus sparse] Experiment setupPosiblemente ocupada @aaronkl la tomó hace 8 días. AbiertoT4.1 - Optimization HPC T4.2 - model exploration
OpenEuroLLM/Taskboard#439 · 1 reacción · 1 asignado ·
-
Qwen3.5 35B-A3B hybrid MoE weak and strong scalingPosiblemente ocupada @ibaucells la tomó hace 8 días. AbiertoT4.1 - Optimization HPC
OpenEuroLLM/Taskboard#438 · 2 comentarios · 1 asignado ·
-
Synthetic Generation for Multilingual Instruction TuningPosiblemente ocupada @abhash-er la tomó hace 9 días. Abierto4.6 post-training
OpenEuroLLM/Taskboard#437 · 1 asignado ·
-
Safety and alignment evaluationPosiblemente ocupada @BirgerMoell la tomó hace 9 días. Abierto4.6 post-training
OpenEuroLLM/Taskboard#436 · 1 asignado ·
-
Finding optimal dataset mixtures for DPOPosiblemente ocupada @geoalgo la tomó hace 9 días. Abierto4.6 post-training
OpenEuroLLM/Taskboard#435 · 3 asignados ·