[New backend] cuTile backend for Liger kernels β performance data on B200 + integration interest
Maintainers usually reply within 3 days
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 25/100
- Issue type
- Feature
- Clarity
- Needs clarification
- Activity status
- Quiet
- Tech stack
- python
- Domain
- backend, performance
Research direction
Start by reviewing the proposed optional backend layout under liger_kernel/ops/backends/cutile/ and the jsd proof-of-concept scope. Use the listed benchmark command and B200 results as context, then resolve the questions about optional tilegym installation, testing policy, and supported platforms. Done means the team has agreed on an integration shape and contribution requirements.
Written by the indexing model from the issue text.
Description
Background
Hi Liger team π
We've been building TileGym, helpful kernel tutorials and examples for tile-based GPU programming.
Over the past few weeks, we have implemented cuTile Python versions of most Liger kernels internally and want to release them in TileGym. We wanted to share some early performance numbers and start a conversation about whether there might be an opportunity to integrate a cuTile backend into Liger down the road.
What we've implemented
We have working internal implementations of the following kernels (tested on NVIDIA B200, bfloat16, Llama-3-8B config β hidden_size=4096, intermediate_size=14336):
cross_entropy, dyt, fused_add_rms_norm, fused_linear_cross_entropy, fused_linear_jsd, fused_neighborhood_attention, geglu, group_norm, jsd, kl_div, layer_norm, llama4_rope, poly_norm, qwen2vl_mrope, rms_norm, rope, softmax, sparsemax, swiglu, tiled_geglu, tiled_swiglu, tvd
We currently have tested cuTile-Python backends for Liger-equivalent kernels internally(tested on NVIDIA B200, 1650 MHz, bfloat16, Llama-3-8B config, triton version: 3.7.0+git7d075612). Below are some of our very promising results.
We use the following command to test it.
python benchmark_${kernel}.py --model llama_3_8b --sweep-mode token_length
e.g. jsd
Full (Fwd + Bwd) Speed + Memory
βββββββββ¬ββββββββββββββ¬βββββββββββββββ¬βββββββββββββββ¬ββββββββββ¬ββββββββββββ¬βββββββββββββ¬ββββββββββββ
β BΓT β Torch Speed β Liger Speed β CuTile Speed β Speedup β Liger Mem β CuTile Mem β Mem Ratio β
β β β β β(vs Liger)β β β β
βββββββββΌββββββββββββββΌβββββββββββββββΌβββββββββββββββΌββββββββββΌββββββββββββΌβββββββββββββΌββββββββββββ€
β 1,024 β 6.12 ms β 6.10 ms β 1.39 ms β 4.38Γ β 3,012 MB β 3,012 MB β 1.00Γ β
β 2,048 β 12.24 ms β 10.56 ms β 2.57 ms β 4.11Γ β 6,012 MB β 6,012 MB β 1.00Γ β
β 4,096 β 24.40 ms β 21.13 ms β 4.95 ms β 4.27Γ β 12,024 MB β 12,024 MB β 1.00Γ β
β 8,192 β 48.89 ms β 42.14 ms β 9.79 ms β 4.30Γ β 24,048 MB β 24,048 MB β 1.00Γ β
βββββββββ΄ββββββββββββββ΄βββββββββββββββ΄βββββββββββββββ΄ββββββββββ΄ββββββββββββ΄βββββββββββββ΄ββββββββββββ
Consistent 4Γ speedup; memory bit-for-bit identical. Here we use a different BLOCK_SIZE for cuTile
Proposed Integration Plan
Step 1 β TileGym side (we own this entirely)
Release the CuTile implementations under tilegym/suites/liger/, publish to PyPI as part of tilegym. No changes needed in Liger for this step.
Step 2 β Liger side (we'd like to contribute a PR, pending your approval)
Add an optional CuTile backend that activates when tilegym is installed:
liger_kernel/ops/backends/cutile/
__init__.py # try: import tilegym; except: pass
group_norm.py # thin dispatch wrappers
jsd.py
...
Questions for the team
- Is the pip install tilegym + optional dispatch model acceptable to you? Or would you prefer a different integration shape?
- What are your requirements for adding an optional dependency (testing policy, supported platforms)?
- Happy to start small β would you be open to a draft PR with just jsd as a proof of concept?
Thanks for all the great work on Liger
- Dominant language
- Python
- Stars
- 6.7k
- Forks
- 656
- Avg merge
- 4d 12h
- Merged PRs (30d)
- 27
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up β it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from linkedin/Liger-Kernel
-
Difficulty 2/5 1-3 hours Newbie friendliness 77/100
linkedin/Liger-Kernel#1658 Β·
Maintainers usually reply within 3 days
-
LigerPolyNorm defaults to in_place=True (docstring says False) and overwrites shared gradientsPossibly taken @nazek52 claimed this 1 day ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
linkedin/Liger-Kernel#1647 Β·
Maintainers usually reply within 3 days
-
[Bug] Experimental embedding kernels overflow int32 row offsets (illegal memory access past 2^31 output elements)Possibly taken @0xDexe claimed this 8 days ago. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
linkedin/Liger-Kernel#1570 Β· 3 comments Β·
Maintainers usually reply within 3 days
-
Difficulty 3/5 1-2 days Newbie friendliness 68/100
linkedin/Liger-Kernel#1661 Β·
Maintainers usually reply within 3 days
-
Difficulty 4/5 3-5 days Newbie friendliness 55/100
linkedin/Liger-Kernel#1660 Β·
Maintainers usually reply within 3 days
All issues in linkedin/Liger-Kernel
Similar issues
-
author-page correction move-papers
Difficulty 1/5 Under an hour Newbie friendliness 78/100
acl-org/acl-anthology#10978 Β· 1 comment Β·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
GoogleCloudPlatform/professional-services-data-validator#1899 Β·
Maintainers usually reply within 4 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
HKUDS/CLI-Anything#652 Β·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Project-N-E-K-O/N.E.K.O#3813 Β·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Dao-AILab/flash-attention#3191 Β·
Maintainers usually reply within 4 days