vllm-project/vllm

[torch.compile] E2E correctness testing for fusions

Open

#39428 opened on Apr 9, 2026

View on GitHub
 (6 comments) (0 reactions) (0 assignees)Python (80,034 stars) (16,816 forks)batch import
help wantedtorch.compile

Description

E2E tests for fusions (tests/compile/fusions_e2e) have done a great job preventing fusion regressions where model/forward code changes break a custom torch.compile fusion pass. However, we currently have no way of testing correctness for these fusion configurations.

It would be good to investigate an approach where we only run a few layers of a model and compare the outputs. This would be helpful for correctness testing in general, and we could compare the outputs to both a baseline vLLM configuration and the huggingface baseline.

This would likely require some work to fix weight loading for models like DeepSeek when --hf-overrides.num_hidden_layers is overriden.

Contributor guide

Tech stack
pythonpytorch
Domain
backendtesting
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
4
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
3-5 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
needs investigation
Prerequisites
PyTorch compilationvLLM fusion passesModel weight loading
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
35
Research direction
Start by examining the existing test suite at `tests/compile/fusions e2e` to understand the structure. Consider using a subset of layers with weight loading overrides via ` hf overrides.num hidden layers`, especially for models like DeepSeek. Develop a method to compare outputs against both vLLM baseline and Hugging Face reference, possibly using a small model configuration. Address any weight loading issues that arise from overriding the number of layers.