vllm-project/vllm

[Feature]: Add Triton implementation of NVFP4 GEMM

Open

#21,014 opened on Jul 15, 2025

View on GitHub
 (10 comments) (1 reaction) (0 assignees)Python (80,034 stars) (16,816 forks)batch import
feature requestgood first issueperformanceunstale

Description

🚀 The feature, motivation and pitch

Currently we only have NVFP4 GEMMs written in CUTLASS for SM100, which means we have no support for SM120. While we still expect tuned CUTLASS kernels to provide the best performance, it would be nice to have a reference Triton implementation available as a fallback if no other kernels are available.

It seems Triton has supported FP4 formats for several months so we should have a new enough version of Triton https://github.com/triton-lang/triton/blob/620237edd282d3fa275e7f931af2018423108c4a/python/test/unit/language/test_matmul.py#L652

A good starting point would be to add the triton kernel directly to https://github.com/vllm-project/vllm/blob/main/benchmarks/kernels/bench_nvfp4_gemm.py and compare the performance to the CUTLASS kernel, before integrating with vLLM as a whole

Alternatives

No response

Additional context

No response

Before submitting a new issue...

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

Contributor guide

Tech stack
python
Domain
performancemachine learning
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
4
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
over 1 week
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
Triton understandingCUDA basicsFP4 formatsvLLM build setup
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
15
Research direction
Start by studying the existing CUTLASS NVFP4 GEMM implementation in the benchmarks/kernels/bench nvfp4 gemm.py file. Then, write a Triton kernel that achieves the same functionality, referring to the Triton FP4 matmul test at the provided URL. Compare performance with the CUTLASS kernel before integrating into the broader vLLM codebase. Check the discussion in comments for any additional guidance or constraints.