nightly timeout: cpu-torch-multi-latest multi-rank suite exceeded its time budget
Maintainers usually reply within 3 days
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 35/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Active
- Domain
- ci-cd, testing-qa
Research direction
Start from the nightly run linked in the issue and read the cpu-torch-multi-latest job log to see which phase hit the 120m or 25m budget. Find the GitHub Actions workflow that defines this job and its timeout settings, then compare them with how long the multi-rank tests take. Done means either a justified budget change or a split suite, and the nightly job finishing within its limit. Raising budgets versus splitting is a maintainer decision, so confirm it in the thread first.
Written by the indexing model from the issue text.
Description
The nightly multi-rank CPU run died at its time budget (class: timeout) at 64915cb41bfe12830c618856cb95186799be0099.
The multi-rank suite likely outgrew its 120m/25m phase budgets; consider raising them or splitting the suite. Run: https://github.com/deepspeedai/DeepSpeed/actions/runs/38849719748
- Dominant language
- Python
- Stars
- 43.3k
- Forks
- 5.1k
- Avg merge
- 4d 21h
- Merged PRs (30d)
- 66
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from deepspeedai/DeepSpeed
-
[BUG] Universal checkpoint resume puts CPU `step` into fused/capturable optimizer statePossibly taken @ArjunPakhan claimed this 3 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
deepspeedai/DeepSpeed#8884 · 1 comment ·
Maintainers usually reply within 3 days
-
init_inference rejects dtype=torch.int8 on every acceleratorMay be free again A pull request for this issue was closed without being merged. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
deepspeedai/DeepSpeed#8577 ·
Maintainers usually reply within 3 days
-
[BUG] AutoTP checkpoint load raises AttributeError on a model containing nn.InstanceNorm1dPossibly taken @vineethsaivs claimed this 41 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
deepspeedai/DeepSpeed#8514 ·
Maintainers usually reply within 3 days
-
[BUG] Python checkpoint writers fail to reload full ZeRO-3 state with default torch.loadPossibly taken @ai-yang claimed this 43 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
deepspeedai/DeepSpeed#8500 ·
Maintainers usually reply within 3 days
-
[BUG] set_norm_for_param_grad_in_gpu's grad_accum fallback is dead in both accumulation modesPossibly taken @LOGO127 claimed this 52 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
deepspeedai/DeepSpeed#8371 ·
Maintainers usually reply within 3 days
All issues in deepspeedai/DeepSpeed
Similar issues
-
Difficulty 1/5 Under an hour Newbie friendliness 85/100
yzhao062/awesome-auditable-ai#45 ·
Maintainers usually reply within 3 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
NVIDIA/flare-local-inference#90 ·
Maintainers usually reply within 1 day
-
area/streaming comp/gateway P2 platform/telegram type/bug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
NousResearch/hermes-agent#144897 ·
Maintainers usually reply within 1 day
-
request-theme
Difficulty 1/5 Under an hour Newbie friendliness 66/100
LizardByte/ThemerrDB#9575 ·
Maintainers usually reply within 1 day