Silently slower attention kernel when training MiniMax-H3 LoRA on B200
Maintainers usually reply within 2 days
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 55/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- python
- Domain
- machine-learning, performance
Research direction
Start with examples/minimax_h3/model_training/lora/MiniMax-H3-FL2VA.sh at stage 2, the installation README, and the experiment recorded in #1680. Reproduce the attention implementation choice with DIFFSYNTH_ATTENTION_IMPLEMENTATION=torch and inspect the existing logging or selection path. Done means the selected implementation is visible to users or the installation guidance clearly prevents the silent slowdown.
Written by the indexing model from the issue text.
Description
I ran the repo's MiniMax-H3 LoRA training example (examples/minimax_h3/model_training/lora/MiniMax-H3-FL2VA.sh, stage 2) on 8x B200, following the installation README, and got 6.32 s per training step. A profile showed attention running on the FA2 kernel, which is an sm80-era design and much slower on this GPU than torch SDPA's cuDNN backend. With the kernel switched, the same step took 3.80 s. Nothing in the logs said which implementation had been picked, so I only found it by profiling.
Setting DIFFSYNTH_ATTENTION_IMPLEMENTATION=torch avoids this, as @mi804 noted in #1680. A line in the install docs, or a one-line log of the chosen implementation at import, would have saved me the profile.
I opened #1680 to record my experiment: setup, measurements, traces and some potential fixes. A different fix may also well suit the codebase.
- Dominant language
- Python
- Stars
- 13.1k
- Forks
- 1.3k
- Avg merge
- 12h 53m
- Merged PRs (30d)
- 42
Getting set up
We have not checked this project's setup files yet. Start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from modelscope/DiffSynth-Studio
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
modelscope/DiffSynth-Studio#1707 · 1 comment ·
Maintainers usually reply within 2 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
modelscope/DiffSynth-Studio#1702 ·
Maintainers usually reply within 2 days
-
Difficulty 1/5 1-3 hours Newbie friendliness 78/100
modelscope/DiffSynth-Studio#1668 ·
Maintainers usually reply within 2 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
modelscope/DiffSynth-Studio#1499 · 1 comment ·
Maintainers usually reply within 2 days
-
Difficulty 1/5 Under an hour Newbie friendliness 78/100
modelscope/DiffSynth-Studio#1373 · 5 comments · 1 reaction ·
Maintainers usually reply within 2 days
All issues in modelscope/DiffSynth-Studio
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
PedestrianDynamics/pyFDS-Evac#199 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
521xueweihan/HelloGitHub#3790 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
sandialabs/atlas-ui-3#978 ·
Maintainers usually reply within 1 day
-
area: tests perceived difficulty: 2
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Nitjsefnie-Harness-Commons/daedalus#1255 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
EleutherAI/lm-evaluation-harness#4256 ·
Maintainers usually reply within 1 day