[BUG] ZenFlow: loss values becomes NaN after 'update_interval' number of steps.
Maintainers usually reply within 3 days
@Antlera is already working on this.
Since Jan 4, 2026.
Assessment
This issue has not been assessed yet.
Description
I was training a Llama model using the script released in the Deepspeed-Examples repository using ZenFlow. Interestingly, I noticed the loss values becoming NaN after the preconfigured (in the deepspeed config) update_interval number of steps. Ex: the update interval is set to 4 in the deepspeed config, loss becomes nan from the 5th step. The code used is available here: https://github.com/deepspeedai/DeepSpeedExamples/tree/master/training/DeepSpeed-ZenFlow/finetuning
My software configurations and versions:
torch==2.5.0+cu118
Transformers==4.57.3
Cloned the latest version of deepspeed from github
Datasets==4.4.1
The jobs are run with DGX H200 GPUs and AMD EPYC 7742 processor.
I wish to understand why the loss turns NaN and more importantly why it happens only after the update_interval step. Can I please get some help with respect to this ?
- Dominant language
- Python
- Stars
- 43.3k
- Forks
- 5.1k
- Avg merge
- 4d 20h
- Merged PRs (30d)
- 69
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from deepspeedai/DeepSpeed
-
[BUG] Universal checkpoint resume puts CPU `step` into fused/capturable optimizer statePossibly taken @ArjunPakhan claimed this 3 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
deepspeedai/DeepSpeed#8884 · 1 comment ·
Maintainers usually reply within 3 days
-
init_inference rejects dtype=torch.int8 on every acceleratorMay be free again A pull request for this issue was closed without being merged. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
deepspeedai/DeepSpeed#8577 ·
Maintainers usually reply within 3 days
-
[BUG] AutoTP checkpoint load raises AttributeError on a model containing nn.InstanceNorm1dPossibly taken @vineethsaivs claimed this 41 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
deepspeedai/DeepSpeed#8514 ·
Maintainers usually reply within 3 days
-
[BUG] Python checkpoint writers fail to reload full ZeRO-3 state with default torch.loadPossibly taken @ai-yang claimed this 42 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
deepspeedai/DeepSpeed#8500 ·
Maintainers usually reply within 3 days
-
[BUG] set_norm_for_param_grad_in_gpu's grad_accum fallback is dead in both accumulation modesPossibly taken @LOGO127 claimed this 51 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
deepspeedai/DeepSpeed#8371 ·
Maintainers usually reply within 3 days
All issues in deepspeedai/DeepSpeed
Similar issues
-
bug P2 rust
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
jscott3201/rusty-bacnet#2085 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
evarness-ai/iris-harness#401 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
sistemicorp/scripts#62 ·
-
area/mac help wanted needs-qa
Difficulty 1/5 Under an hour Newbie friendliness 76/100