[NPU] Track reverting Ascend Atlas 950 workarounds after triton-ascend upstream fixes (#1464)
Maintainers usually reply within 3 days
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Refactor
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- python
- Domain
- compilers, machine-learning, performance
Research direction
Start with PR #1464 and inspect the listed Ascend files: group_norm.py, grpo_loss.py, and fused_moe_kernels.py. Check release status for triton-ascend issues #2239, #2246, and #2247, then on A5 revert one corresponding workaround at a time and run the named pytest tests, including Mixtral experts. Done means the original form compiles, matches numerically, and a Liger PR removes or narrows that workaround.
Written by the indexing model from the issue text.
Description
Summary
PR #1464 landed Ascend A3/Atlas 950 UB tile capping plus temporary kernel workarounds so group norm, GRPO/CISPO/SAPO, and fused MoE pass on Atlas 950 while triton-ascend compiler bugs are still open.
This issue tracks removing or narrowing those workarounds once a released triton-ascend build contains the upstream fixes. It does not track the ub_manager 4/3 auto-multi-buffer cap (8ba7d90); that is a Liger tiling policy, not a compiler workaround.
Upstream bugs (filed, workarounds in #1464)
| triton-ascend issue | A5 symptom | Liger workaround | Location |
|---|---|---|---|
| #2239 | BiShengHIR memref.extract_strided_metadata when dynamic task_base + tl.sum(..., axis=1) on padded 2-D tiles (e.g. group norm (1,1,1,3)) |
Use full-tile tl.sum(X_vals) / tl.sum(X_vals * X_vals) instead of axis=1 (equivalent for cpg=1) |
src/liger_kernel/ops/backends/_ascend/ops/group_norm.py (_group_norm_forward_single_task_kernel) |
| #2246 | Compile-time UB overflow on large-vocab scans with tl.static_range + bf16→fp32 online softmax (V=151936) |
Inner vocab loops: tl.static_range → tl.range(0, N, BLOCK_N) |
src/liger_kernel/ops/backends/_ascend/ops/grpo_loss.py (fwd / softmax / bwd) |
| #2247 | 2-D reduction over [TPT, K_POW2] drops lanes when K_POW2 < 8 (MoE router histogram undercount; Mixtral K=2 → 256 instead of 1024) |
Pad inner dim to HIST_K_POW2 = max(8, K_POW2), keep k < K mask |
src/liger_kernel/ops/backends/_ascend/ops/fused_moe_kernels.py (_moe_router_histogram_kernel) |
These are A5 pipeline / compiler bugs, not application-logic errors. A3 (A2/A3 pipeline) did not need the same kernel rewrites for the cases we tested.
Additional fused-MoE A5 workarounds (same commit, not covered by #2239/#2246/#2247)
These are still in tree and should be tracked separately (or new triton-ascend issues) before we fully restore the original kernels:
| Item | Symptom (A5) | Workaround in Liger |
|---|---|---|
| Token gather H=128 | TILE_H == BLOCK_H == 128 gather/reduce wrong; pad to BLOCK_H + BLOCK_H//2 restores |
TILE_H padding |
Bwd down-proj tl.dot |
Multi-row M-tile + tl.dot incorrect |
One row per program (pid_base / row_mask) |
Bwd dX_expanded |
Nested H+I tl.range in one program fails |
Launch (num_rows, n_h_tiles) (H on grid) |
bf16 tl.sum(..., axis=0) on [K, H] |
K-reduce unsafe | Explicit mask weights |
Exit criteria (when to close this issue)
For each of #2239 / #2246 / #2247:
- Upstream issue is fixed in a released
triton-ascend(record version). - On A5, revert only that workaround and confirm:
- the original Triton form compiles and matches the current kernel numerically;
pytestfor the corresponding files is green (test_group_norm.py,test_grpo_loss.py,test_fused_moe.py/ Mixtral experts).
- Open a Liger PR that drops or narrows that workaround (prefer one PR per upstream fix). Do not wait to revert all three at once.
Suggested verification:
# GroupNorm: original axis=1 reduce
pytest test/transformers/test_group_norm.py
# GRPO: original tl.static_range vocab scan (include V=151936)
pytest test/transformers/test_grpo_loss.py
# MoE histogram: original K_POW2 (no HIST_K_POW2 pad); run fused_moe in a subprocess
pytest test/transformers/test_fused_moe.py
pytest test/transformers/test_swiglu.py::test_correctness_mixtralexperts
Keep ub_manager 4/3 UB cap unless a later measurement shows it is no longer needed on A3/A5.
Out of scope
- A3 vs A5 pytest skip/pass count differences caused by
transformers/ HBM skip marks (not kernel bugs). - Running
test_fused_moe.pyin-process with the rest of the suite (can poison the NPU); that is test isolation, not a compiler fix.
Links
- Merged PR: https://github.com/linkedin/Liger-Kernel/pull/1464
- Merge commit:
09f7feb - triton-ascend: #2239, #2246, #2247
- Dominant language
- Python
- Stars
- 6.7k
- Forks
- 656
- Avg merge
- 4d 12h
- Merged PRs (30d)
- 27
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from linkedin/Liger-Kernel
-
Difficulty 2/5 1-3 hours Newbie friendliness 77/100
linkedin/Liger-Kernel#1658 ·
Maintainers usually reply within 3 days
-
LigerPolyNorm defaults to in_place=True (docstring says False) and overwrites shared gradientsPossibly taken @nazek52 claimed this 1 day ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
linkedin/Liger-Kernel#1647 ·
Maintainers usually reply within 3 days
-
[Bug] Experimental embedding kernels overflow int32 row offsets (illegal memory access past 2^31 output elements)Possibly taken @0xDexe claimed this 8 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
linkedin/Liger-Kernel#1570 · 3 comments ·
Maintainers usually reply within 3 days
-
Difficulty 3/5 1-2 days Newbie friendliness 68/100
linkedin/Liger-Kernel#1661 ·
Maintainers usually reply within 3 days
-
Difficulty 4/5 3-5 days Newbie friendliness 55/100
linkedin/Liger-Kernel#1660 ·
Maintainers usually reply within 3 days
All issues in linkedin/Liger-Kernel
Similar issues
-
author-page correction move-papers
Difficulty 1/5 Under an hour Newbie friendliness 78/100
acl-org/acl-anthology#10978 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
GoogleCloudPlatform/professional-services-data-validator#1899 ·
Maintainers usually reply within 4 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
HKUDS/CLI-Anything#652 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Project-N-E-K-O/N.E.K.O#3813 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Dao-AILab/flash-attention#3191 ·
Maintainers usually reply within 4 days