[32B Dense] Debug 64k increasing loss
@luukkonenr 已经在做这个了。
开始于 2026年9月2日。
评估
这个 Issue 还没有评估数据。
描述
While training the 32B dense flag model we encountered an increasing loss around 64k:
See:
From: https://wandb.ai/openeurollm-project/oellm_32b_dense
For live debug session: https://github.com/OpenEuroLLM/oellm-autoexp/blob/prod/oellm_32b_dense/docs/64k-debug/DEBUG.md
The Problem:
The training loss was going down like it should. Then around iteration 65,000
it turned around and started going up, and it keeps getting worse.
- Run:
oellm_32b_dense_prod_dataopt5_gbs4096_lr3e-4 - Last job:
1537344→
wandb,
stopped 2026-08-30 22:43 at iteration 75,125 - Training loss: 1.5434 at ~65,000 → 1.587 at 75,125
This is not a wobble. It is a steady climb that is speeding up — all of it visible
in job 1537344
(wandb, which
covers 66,626 → 75,125, i.e. the whole broken stretch):
| iterations | loss change per 1000 steps | where to see it |
|---|---|---|
| 50k–56k | −0.0018 (good, going down) | job 1530865 |
| 62k–66.6k | −0.0005 (nearly stopped) | jobs 1535676, 1536694 |
| 66.6k–71k | +0.0023 (going up) | 1537344 |
| 71k–75k | +0.0062 (going up faster) | 1537344 |
Why this is a big deal: we are only 8.4% into a 15T-token run
(75,125 of 894,000 iterations, ~1.26T tokens). If the model gets worse the longer
we train, the other 92% cannot be run this way. Rough cost so far: ~11,000
iterations spent making it worse ≈ 185B tokens ≈ 27,500 GPU-h.
How we found the exact spot
Fitting the healthy part of the curve (12k–34k) and extrapolating, the run stays
on track until ~66,000 and then breaks away: +0.012 at 68k, +0.024 at 72k,
+0.038 at 74k. The break is at ~66,625.
Held-out scores confirm the model really got worse (lower = better):
| checkpoint | score | job |
|---|---|---|
| 60,000 | 1.500890 | 1548575 |
| 64,000 | 1.495481 ← best | 1573075 |
| 68,000 | 1.511213 | 1563495 |
| 72,000 | 1.519173 | 1543640 |
Normally later = better. Here it flips after 64,000. That flip is the bug.
This could be due to a variety of things
- FP8
- NaN weights/activations
- Switching containers
- Turning on inter-document attention masking
- Data quality
- RMSNorm off-zero gamma
- 主要语言
- 没有语言数据
- 星标
- 3
- 派生
- 0
- PR 合并指标
- 30 天内没有已合并 PR
环境准备
这个项目没有提供开发容器、Dockerfile 或贡献指南，环境需要你自己搭建：先看它的 README，通用步骤见我们的新手贡献指南。
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
OpenEuroLLM/Taskboard 的其他 Issue
-
[nucleus sparse] Experiment setup可能已有人在做 @aaronkl 于 10 天前认领。 未关闭T4.1 - Optimization HPC T4.2 - model exploration
OpenEuroLLM/Taskboard#439 · 1 条评论 · 1 个 reaction · 已指派 1 人 ·
-
Qwen3.5 35B-A3B hybrid MoE weak and strong scaling可能已有人在做 @ibaucells 于 10 天前认领。 未关闭T4.1 - Optimization HPC
OpenEuroLLM/Taskboard#438 · 2 条评论 · 已指派 1 人 ·
-
Synthetic Generation for Multilingual Instruction Tuning可能已有人在做 @abhash-er 于 11 天前认领。 未关闭4.6 post-training
OpenEuroLLM/Taskboard#437 · 已指派 1 人 ·
-
Safety and alignment evaluation可能已有人在做 @BirgerMoell 于 11 天前认领。 未关闭4.6 post-training
OpenEuroLLM/Taskboard#436 · 已指派 1 人 ·
-
Finding optimal dataset mixtures for DPO可能已有人在做 @geoalgo 于 11 天前认领。 未关闭4.6 post-training
OpenEuroLLM/Taskboard#435 · 已指派 3 人 ·