Delta weight sync (NCCL) produces NaN weights on Qwen3.5-122B MoE → "probability tensor contains inf/nan" crash
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 42/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- python
- Domain
- distributed-systems, machine-learning
Research direction
Start with the delta-sync apply path selected by --update-weight-mode delta and --update-weight-transport nccl, comparing it with the stable full-sync path. Reproduce on the Qwen3.5-122B-A10B grouped-MoE configuration and inspect the grouped-MoE parameter update after the checksum passes. Done means the first delta apply leaves no inf or NaN weights and the next rollout completes without the probability-tensor crash.
Written by the indexing model from the issue text.
Description
Summary
With --update-weight-mode delta --update-weight-transport nccl, the first delta apply on Qwen3.5-122B-A10B (grouped MoE) leaves NaN/inf in the rollout engine's weights. The next rollout crashes in sampling. Full-sync mode (--update-weight-mode full) on the same model/config is stable. Reproduced 4/4 runs.
Symptom
TensorCompare.cu:109 _assert_async_cuda_kernel: Assertion `probability tensor contains either inf, nan or element < 0` failed
Subprocess scheduler_0 crashed with exit code -3. Triggering SIGQUIT for cleanup...
Timeline: initial full seed sync ✓ → step-1 train ✓ → step-2 delta sync applied (checksum passes) ✓ → next rollout crashes.
Config
- slime with delta sync (#1806); SGLang v0.5.13 +
docker/patch/latest/sglang.patch - Qwen3.5-122B-A10B:
--num-experts 256 --moe-grouped-gemm --moe-token-dispatcher-type alltoall --attention-backend flash(not flashinfer_trtllm) --update-weight-encoding indices, per-sync density ~1%- Reproduced on both TP2/PP2/EP16 and TP2/PP4/EP8 (parallelism-independent)
Notes
- Checksum passes, so the
(positions, values)payload arrives intact — corruption is in the apply into grouped-MoE params, not on the wire. - Possibly related: #2193 (grouped-MoE GLU rechunk expert-axis), #2104 (failed apply not surfaced) — but neither reports this crash signature.
- Dominant language
- Python
- Stars
- 8.8k
- Forks
- 1.3k
- Avg merge
- 20h 16m
- Merged PRs (30d)
- 24
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from THUDM/slime
-
[Bug] Math reward accepts distinct large integers after float normalizationPossibly taken @lll-hhh claimed this 2 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
gpqa: when no option letter can be extracted, a single-letter gold label is substring-matched against the whole reply, so a reply that never answers can score 1.0Possibly taken @shaurya416 claimed this 39 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
Maintainers usually reply within 1 day
-
[Bug] shipped dynamic-sampling filters crash on fan-out groups (list[list[Sample]]) from multi-turn agent rolloutsPossibly taken @QIU-Guanzong claimed this 50 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
Maintainers usually reply within 1 day
-
[Bug] W&B initialization fails with wandb 0.28.2 because wandb.util.generate_id was removedPossibly taken @cyyecao-lappland claimed this 68 days ago. Openbug
Difficulty 1/5 1-3 hours Newbie friendliness 90/100
THUDM/slime#2332 · 1 comment ·
Maintainers usually reply within 1 day
-
[Bug] solution_str = sample.prompt + sample.responsePossibly taken @mvanhorn claimed this 132 days ago. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
THUDM/slime#1829 · 1 comment ·
Maintainers usually reply within 1 day
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
pytest-dev/pyfakefs#1367 · 1 comment ·
Maintainers usually reply within 2 days
-
IFEvalG exclude_word_harder misses the keyword next to punctuation, at the start or end of the text, and in other letter casePossibly taken @damienen claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
allenai/open-instruct#2151 ·
Maintainers usually reply within 1 day
-
feat(plan): support convertorLabels and convertorAffinity in Plan resourcePossibly taken @AmenB claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
RedHatQE/openshift-python-wrapper#2849 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
Maintainers usually reply within 25 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
libnativeapi/nativeapi#21 ·