[NPU] Track reverting Ascend Atlas 950 workarounds after triton-ascend upstream fixes (#1464)

Open
#1,481 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 3 days

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Refactor
Clarity
Mostly clear
Activity status
Active
Tech stack
python

Research direction

Start with PR #1464 and inspect the listed Ascend files: group_norm.py, grpo_loss.py, and fused_moe_kernels.py. Check release status for triton-ascend issues #2239, #2246, and #2247, then on A5 revert one corresponding workaround at a time and run the named pytest tests, including Mixtral experts. Done means the original form compiles, matches numerically, and a Liger PR removes or narrows that workaround.

Written by the indexing model from the issue text.

Description

Summary

PR #1464 landed Ascend A3/Atlas 950 UB tile capping plus temporary kernel workarounds so group norm, GRPO/CISPO/SAPO, and fused MoE pass on Atlas 950 while triton-ascend compiler bugs are still open.

This issue tracks removing or narrowing those workarounds once a released triton-ascend build contains the upstream fixes. It does not track the ub_manager 4/3 auto-multi-buffer cap (8ba7d90); that is a Liger tiling policy, not a compiler workaround.

Upstream bugs (filed, workarounds in #1464)

triton-ascend issue A5 symptom Liger workaround Location
#2239 BiShengHIR memref.extract_strided_metadata when dynamic task_base + tl.sum(..., axis=1) on padded 2-D tiles (e.g. group norm (1,1,1,3)) Use full-tile tl.sum(X_vals) / tl.sum(X_vals * X_vals) instead of axis=1 (equivalent for cpg=1) src/liger_kernel/ops/backends/_ascend/ops/group_norm.py (_group_norm_forward_single_task_kernel)
#2246 Compile-time UB overflow on large-vocab scans with tl.static_range + bf16→fp32 online softmax (V=151936) Inner vocab loops: tl.static_range → tl.range(0, N, BLOCK_N) src/liger_kernel/ops/backends/_ascend/ops/grpo_loss.py (fwd / softmax / bwd)
#2247 2-D reduction over [TPT, K_POW2] drops lanes when K_POW2 < 8 (MoE router histogram undercount; Mixtral K=2 → 256 instead of 1024) Pad inner dim to HIST_K_POW2 = max(8, K_POW2), keep k < K mask src/liger_kernel/ops/backends/_ascend/ops/fused_moe_kernels.py (_moe_router_histogram_kernel)

These are A5 pipeline / compiler bugs, not application-logic errors. A3 (A2/A3 pipeline) did not need the same kernel rewrites for the cases we tested.

Additional fused-MoE A5 workarounds (same commit, not covered by #2239/#2246/#2247)

These are still in tree and should be tracked separately (or new triton-ascend issues) before we fully restore the original kernels:

Item Symptom (A5) Workaround in Liger
Token gather H=128 TILE_H == BLOCK_H == 128 gather/reduce wrong; pad to BLOCK_H + BLOCK_H//2 restores TILE_H padding
Bwd down-proj tl.dot Multi-row M-tile + tl.dot incorrect One row per program (pid_base / row_mask)
Bwd dX_expanded Nested H+I tl.range in one program fails Launch (num_rows, n_h_tiles) (H on grid)
bf16 tl.sum(..., axis=0) on [K, H] K-reduce unsafe Explicit mask weights

Exit criteria (when to close this issue)

For each of #2239 / #2246 / #2247:

  1. Upstream issue is fixed in a released triton-ascend (record version).
  2. On A5, revert only that workaround and confirm:
    • the original Triton form compiles and matches the current kernel numerically;
    • pytest for the corresponding files is green (test_group_norm.py, test_grpo_loss.py, test_fused_moe.py / Mixtral experts).
  3. Open a Liger PR that drops or narrows that workaround (prefer one PR per upstream fix). Do not wait to revert all three at once.

Suggested verification:

# GroupNorm: original axis=1 reduce
pytest test/transformers/test_group_norm.py

# GRPO: original tl.static_range vocab scan (include V=151936)
pytest test/transformers/test_grpo_loss.py

# MoE histogram: original K_POW2 (no HIST_K_POW2 pad); run fused_moe in a subprocess
pytest test/transformers/test_fused_moe.py
pytest test/transformers/test_swiglu.py::test_correctness_mixtralexperts

Keep ub_manager 4/3 UB cap unless a later measurement shows it is no longer needed on A3/A5.

Out of scope

  • A3 vs A5 pytest skip/pass count differences caused by transformers / HBM skip marks (not kernel bugs).
  • Running test_fused_moe.py in-process with the rest of the suite (can poison the NPU); that is test isolation, not a compiler fix.

Links

Dominant language
Python
Stars
6.7k
Forks
656
Avg merge
4d 12h
Merged PRs (30d)
27

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from linkedin/Liger-Kernel

All issues in linkedin/Liger-Kernel

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.