Why is deepspeed enabled in the Bloom training script?
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 25/100
- Issue type
- Documentation
- Clarity
- Needs clarification
- Activity status
- Stale
- Tech stack
- shell
- Domain
- distributed-systems, machine-learning
Research direction
Start with the Bloom training script and the shown DEEPSPEED_ARGS block; inspect how ZERO_STAGE and config_json are set and compare runs with and without DeepSpeed. Verify the reported Zero-State value and whether the loss curves align, then document the explanation and results.
Written by the indexing model from the issue text.
Description
Why is the value of Zero-State 0 when deepspeed is enabled in the Bloom training script? Can the Bloom model be trained and the loss curve is aligned when deepspeed is disabled? Thanks very much.
DEEPSPEED_ARGS=" \
--deepspeed \
--deepspeed_config ${config_json} \
--zero-stage ${ZERO_STAGE} \
--deepspeed-activation-checkpointing \
"
- Dominant language
- Shell
- Stars
- 1k
- Forks
- 102
- PR merge metrics
- No merged PRs in 30d
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from bigscience-workshop/bigscience
-
Difficulty 2/5 1-3 hours Newbie friendliness 35/100
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
-
Difficulty 1/5 Under an hour Newbie friendliness 25/100
-
Difficulty 3/5 1-2 days Newbie friendliness 28/100
bigscience-workshop/bigscience#66 · 1 comment ·
-
eval opt-175B Open
Difficulty 4/5 3-5 days Newbie friendliness 20/100
bigscience-workshop/bigscience#63 · 1 comment ·
All issues in bigscience-workshop/bigscience
Similar issues
-
align on terminology Open
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
CycloneDX/transparency-exchange-api#393 · 1 comment ·
-
module/agent platform/macos type/bug/regression
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 1/5 Under an hour Newbie friendliness 92/100
CachyOS/cachyos-aur-derived#754 ·
-
Difficulty 1/5 Under an hour Newbie friendliness 90/100
CrowdStrike/falcon-scripts#528 ·
-
bug(cli): hapi doctor inline-media prints a fabricated B:\ helper-script path in packaged installs Open
Difficulty 2/5 1-3 hours Newbie friendliness 70/100