Repository metrics
- Stars
- (4,990 stars)
- PR merge metrics
- (PR metrics pending)
Description
Motivation.
Mixture-of-Experts (MoE) layers are a large fraction of end-to-end latency for MoE diffusion models served by vLLM-Omni (e.g. HunyuanImage-3.0). On our L20X / Hopper stack we profiled HunyuanImage TP=4 (tests/dfx/perf/tests/test_hunyuan_image_tp4.json workload shape: 1024×1024, 8 steps) with torch.profiler:
Therefore, optimizing MoE is critical to improving overall performance.
Proposed Change.
This RFC is a tracking issue. Concrete PRs can link back here.
(1) vLLM-Omni: fine-tuned MoE JSON support
we use ncu to profiling fused_moe_kernel, and found that sm in low utilazation:
- vllm-omni with tuned moe tiling: https://github.com/vllm-project/vllm-omni/pull/5190
- vllm benchmark_moe.py support more models: https://github.com/vllm-project/vllm/pull/49088
After this, the ncu profling is:
(2) Upstream vLLM: further fused_moe optimization under TMA
On SM90+, use tl.make_tensor_descriptor (TMA) for expert weight (B) loads in fused_moe_kernel, keeping a pointer-load fallback for older arches. Goal: raise effective memory throughput toward compute-bound when configs are already reasonable.
- vllm fused_moe_optimazation: https://github.com/vllm-project/vllm/pull/49078
After this, the ncu profling is:
Call for contributions
Please keep profiling / NCU notes and new ideas on this issue — Happy to co-triage and turn solid findings into follow-up PRs.
Feedback Period.
No response
CC List.
No response
Any Other Things.
No response
Before submitting a new issue...
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.