Multi-GPU interactive mode hangs during model initialization on some machines
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 35/100
- Issue type
- Bug
- Clarity
- Needs clarification
- Activity status
- Quiet
- Tech stack
- python
- Domain
- distributed-systems, machine-learning
Research direction
Start with generate.py and trace the interactive torchrun path through the dist.barrier() preceding WanModel.from_pretrained(). Compare logs and initialization behavior between the working and hanging machine setups, using the reported 4-rank command and NCCL settings. Done means identifying a reproducible cause and documenting or fixing the hang so model initialization completes.
Written by the indexing model from the issue text.
Description
Running Matrix-Game 3.0 interactive mode with torchrun on 4x A100 SXM 80GB.
Stock generate.py with --interactive works on some machines but silently hangs
during model initialization on others.
Setup:
- torch 2.10.0, FA2, CUDA 12.8, Python 3.12
- torchrun --nproc_per_node=4 with --ulysses_size 4, --use_int8, --t5_cpu
- Docker image based on nvidia/cuda:12.8.0-devel-ubuntu22.04
What happens:
- NCCL init succeeds on all ranks
- T5 encoder loads successfully on all ranks
- All 4 ranks log "Initializing Model (DiT)..."
- Then hangs forever: 0% CPU, 0% GPU VRAM (633MB CUDA context only)
- No error, no timeout, no crash
What we've tried:
- NCCL_P2P_DISABLE=1
- NCCL_DEBUG=INFO (init shows COMPLETE, no errors)
- device_id in init_process_group()
- 2 GPUs instead of 4
- Different volume sizes (75GB, 200GB)
Key observation:
- Works on some RunPod machines (e.g. vqnzttsdu44t, c250s631h6rc)
- Hangs on others (e.g. cmg18urp1kg5, hleuzkw9vyyn)
- Same Docker image, same code, same config
The hang appears to be inside WanModel.from_pretrained() or the
dist.barrier() that precedes it. Has anyone else seen this?
- Dominant language
- Python
- Stars
- 2.3k
- Forks
- 254
- PR merge metrics
- No merged PRs in 30d
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from SkyworkAI/Matrix-Game
-
Difficulty 1/5 Under an hour Newbie friendliness 90/100
SkyworkAI/Matrix-Game#82 ·
-
Difficulty 1/5 Under an hour Newbie friendliness 88/100
SkyworkAI/Matrix-Game#80 ·
-
Difficulty 1/5 Under an hour Newbie friendliness 88/100
SkyworkAI/Matrix-Game#77 ·
-
Difficulty 3/5 1-2 days Newbie friendliness 76/100
SkyworkAI/Matrix-Game#86 ·
-
Third person support Open
Difficulty 5/5 Over a week Newbie friendliness 35/100
SkyworkAI/Matrix-Game#76 · 1 comment ·
All issues in SkyworkAI/Matrix-Game
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
anthropics/skills#1811 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
speaches-ai/speaches#678 ·
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
datalayer/mcp-compose#42 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
conda-forge/spacy-feedstock#177 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
UKGovernmentBEIS/inspect_evals#2523 ·