Repository metrics
- Stars
- (4,990 stars)
- PR merge metrics
- (PR metrics pending)
Description
Nightly Pipeline RFC
Background
This RFC proposes adding and improving nightly test coverage for vLLM-Omni core models.
The goal is to make the nightly pipeline more stable, observable, and useful for catching functionality, performance, and accuracy regressions early.
Pipeline Plan
-
Initial pipeline & Wan2.2 functionality nightly tests
PR: https://github.com/vllm-project/vllm-omni/pull/3480 -
Ready pipeline
Status: TBD -
Merge pipeline
Status: TBD -
Full nightly pipeline
Status: TBD
Core Models Monitor
The following core models should be covered by the nightly monitoring pipeline.
For each model, we should track functionality, performance, and accuracy where applicable.
| Model | Functionality Test | Performance Test | Accuracy Test | Status | Notes |
|---|---|---|---|---|---|
| Wan2.2 | ✅ | TBD | TBD | Planned | Initial functionality coverage is being added in PR #3480 |
| Hunyuan-Image 3.0 | TBD | TBD | TBD | Planned | Image generation model coverage |
| Qwen-Image | TBD | TBD | TBD | Planned | Image generation model coverage |
| Qwen3-TTS | ✅ #3600 | TBD | TBD | Planned | TTS model coverage |
| Qwen3-Omni | TBD | TBD | TBD | Planned | Omni / multimodal model coverage |
CI Challenges
Ideally, after a vLLM release, vllm-ascend would promptly publish a corresponding rc1 release along with a Docker image for downstream projects like vllm-omni. In practice, however, there are gaps in this process. Taking v0.20.0 as an example, the timeline can be broken down into three phases:
1. Break Period
vLLM releases v0.20.0. vllm-omni adapts to v0.20.0 on day 0 or day 1, publishing v0.20.0rc1. Meanwhile, the main branch of vllm-ascend typically lags by 1–2 days, causing CI to break.
2. Downstream CI Installation Gap
Downstream CI installs vllm-ascend from main. Since the v0.20.0rc1 release of vllm-ascend is often delayed (usually arriving only 1–2 days before the next vLLM release), no vllm-ascend image that is compatible with vLLM v0.20.0 is available. Consequently, downstream projects must install vllm-ascend manually, facing three options:
- Option 2.1: Continuously pull and build from
vllm-ascendmain. Build time is approximately 30 minutes. - Option 2.2: Use the
vllm-ascendnightly Docker image (built frommain). This requires pulling the latest image daily, resulting in a longer first-trigger PR time. - Option 2.3:
vllm-ascendquickly publishes anrc0image immediately after adoptingv0.20.0(ideal but not yet standard practice).
Current choice: We are using the vllm-ascend nightly image built from the main branch (Option 2.2).
3. Normal Operation
After vllm-ascend v0.20.0rc1 is officially released, downstream projects can use a fixed-version Docker image directly. Build time is reduced to approximately 10 minutes.