Zero_Stage=1 results in higher TFLOPS?
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 35/100
- Issue type
- Documentation
- Clarity
- Mostly clear
- Activity status
- Stale
- Domain
- documentation, machine-learning, performance
Research direction
Start with the 48-node contenders table in chronicles-prequel.md, linked from the issue, and compare the Zero Stage 0 and 1 rows and their recorded settings. Determine why the reported TFLOPS differ despite matching parameters, then clarify the explanation or correct the documentation so the result is understandable.
Written by the indexing model from the issue text.
Description
Description
I am learning the chronicles_prequel, and I find the last table in the chapter indicates the higher TFLOPS is achieved with Zero_Stage = 1.
Trying with ZeRO_STAGE=0/1
Zero_stage=1 could reduce the memory cost, but how come it increases the performance with other parameter being the same?
| Nodes | Size | ZS | DP | TP | PP | MBS | GBS | Mem | Sec/it | TFLOPs | Notes |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 48 | 181B | 1 | 4 | 8 | 12 | 2 | 2048 | 37GB | 120.29 | 134.02 | 02-21 |
| 48 | 181B | 0 | 4 | 8 | 12 | 2 | 2048 | 72GB | 137.34 | 113.02 | 02-21 |
- Dominant language
- Shell
- Stars
- 1k
- Forks
- 102
- PR merge metrics
- No merged PRs in 30d
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from bigscience-workshop/bigscience
-
Difficulty 4/5 3-5 days Newbie friendliness 25/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 35/100
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
-
Difficulty 1/5 Under an hour Newbie friendliness 25/100
-
Difficulty 3/5 1-2 days Newbie friendliness 28/100
bigscience-workshop/bigscience#66 · 1 comment ·
All issues in bigscience-workshop/bigscience
Similar issues
-
Bob Shell support Openenhancement
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
-
tooling
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
openSUSE/python-rpm-macros#219 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
YunoHost-Apps/adguardhome_ynh#258 ·
-
steam-short-session-tracker: incorrect registry backup path resets settings during automatic repair Open
Difficulty 2/5 1-3 hours Newbie friendliness 80/100
ValveSoftware/SteamOS#2829 ·