zero.Init partitioning of large fused MoE-expert tensors spikes a single GPU (transient full materialization) -> OOM during load even when the sharded model fits
Maintainers usually reply within 3 days
Nobody has claimed this yet.
- #8103 by @Achyuthan-S — closed without merging
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 45/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Quiet
Research direction
Reproduce the MiniMaxAI/MiniMax-M3 load with the stated DeepSpeed, PyTorch, and transformers environment, then trace the from_pretrained init path into zero.Init partitioning. Inspect how the giant fused MoE parameter is materialized and how remote_device='cpu' and offload_param are handled. Done means partitioning avoids a full single-GPU materialization and the sharded model loads without the transient OOM.
Written by the indexing model from the issue text.
Description
Env: transformers 5.12.1, deepspeed 0.18.9, torch 2.12.0+cu130, peft 0.19.1, bitsandbytes 0.49.2, accelerate 1.14.0; 8x B200 (178GB) / 2TB RAM; model MiniMaxAI/MiniMax-M3 (428B sparse MoE VL, minimax_m3_vl).
During the correctly-sharded (world_size=8) load, a single GPU spikes to ~180GB vs the ~94GB steady partition and OOMs. The giant fused MoE-expert parameter (128 experts) appears to be materialized in full on a GPU before being scattered. This is a load-time transient — independent of sequence length — and offload_param: {device: cpu} does not prevent the GPU spike during the from_pretrained init path.
Ask: stream/scatter very large parameters during partitioning without a full single-GPU materialization, and honor remote_device='cpu' during the from_pretrained init path so the load can stage through CPU.
- Dominant language
- Python
- Stars
- 43.3k
- Forks
- 5.1k
- Avg merge
- 4d 21h
- Merged PRs (30d)
- 64
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from deepspeedai/DeepSpeed
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
deepspeedai/DeepSpeed#8925 · 1 comment ·
Maintainers usually reply within 3 days
-
[BUG] Universal checkpoint resume puts CPU `step` into fused/capturable optimizer statePossibly taken @ArjunPakhan claimed this 6 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
deepspeedai/DeepSpeed#8884 · 1 comment ·
Maintainers usually reply within 3 days
-
init_inference rejects dtype=torch.int8 on every acceleratorMay be free again A pull request for this issue was closed without being merged. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
deepspeedai/DeepSpeed#8577 ·
Maintainers usually reply within 3 days
-
[BUG] AutoTP checkpoint load raises AttributeError on a model containing nn.InstanceNorm1dPossibly taken @vineethsaivs claimed this 44 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
deepspeedai/DeepSpeed#8514 ·
Maintainers usually reply within 3 days
-
[BUG] Python checkpoint writers fail to reload full ZeRO-3 state with default torch.loadPossibly taken @ai-yang claimed this 45 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
deepspeedai/DeepSpeed#8500 ·
Maintainers usually reply within 3 days
All issues in deepspeedai/DeepSpeed
Similar issues
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
glossyio/traffic-monitor#264 ·
Maintainers usually reply within 1 day
-
Add self-hostOpen
Difficulty 1/5 Under an hour Newbie friendliness 75/100
Atarity/deploy-your-own-saas#102 ·
-
[Bug]: RDFlex silently drops the covariate adjustment when the outcome or treatment column is integer-typedPossibly taken @vrjdev claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
DoubleML/doubleml-for-py#424 ·
Maintainers usually reply within 1 day
-
agentic-dmx docs enhancement
Difficulty 1/5 1-3 hours Newbie friendliness 86/100
-
Difficulty 1/5 Under an hour Newbie friendliness 72/100
521xueweihan/HelloGitHub#4150 ·