Align and expand official LTX-2 / LTX-2.3 pipeline support

Motivation

vLLM-Omni currently supports one-stage LTX-2 and LTX-2.3 text-to-video and image-to-video generation, together with an initial LTX-2 two-stage path.

The upstream LTX family now exposes several distinct inference recipes that share most model components but have different checkpoint composition, denoising schedules, guidance behavior, output-resolution semantics, and audio ownership. Treating all two-stage execution as one conditional path makes these contracts difficult to reason about and has already left some branches unreachable or only partially aligned with upstream behavior.

This issue tracks the work required to:

• share the common LTX-2/LTX-2.3 runtime without forcing the two model versions to have identical numerical behavior;
• represent each official inference recipe explicitly;
• align guidance, conditioning, latent preparation, schedules, and stage outputs with the upstream implementation;
• keep T2V and I2V as different public tasks backed by the same stage orchestration;
• preserve vLLM-Omni distributed execution, offload, compilation, batching, and serving features;

Upstream pipeline model

The official LTX implementation has four core generation recipes. T2V and I2V use the same model and stage structure; image conditioning changes request preparation rather than checkpoint selection.

Official recipe	Upstream inference behavior	Completed in vLLM-Omni main	Not yet supported / needs correction
Dev one-stage	Run the dev model directly at the requested output resolution with the model-version-specific schedule and multimodal guidance	LTX-2/LTX-2.3 T2V and I2V entry points; dev checkpoint loading; CFG execution	Official STG, modality guidance, per-modality scales/rescale, and full upstream numerical alignment
Standard two-stage	Run dev at half of the requested width/height with multimodal guidance; spatially upsample latents; refine video for three positive-only steps with the same dev model plus distilled LoRA; keep Stage 1 audio	Reusable one-stage execution and latent upsampling components exist	End-to-end recipe; explicit auxiliary-checkpoint contract; safe LoRA activation/reset; final-resolution request semantics; Stage 1 audio ownership; LTX-2.3 coverage
HQ two-stage	Apply distilled LoRA in both stages with recipe-specific strengths; use the second-order sampler and HQ defaults; keep Stage 1 audio	None	LTX-2.3 HQ recipe, sampler, defaults, stage-specific adapter strengths, accuracy coverage
Full-distilled two-stage	Use the full distilled model in both stages; run fixed 8-step then 3-step positive-only schedules around spatial upsampling; return Stage 2 video and audio	Initial LTX-2 T2V/I2V two-stage execution with a full-distilled componentized checkpoint	Explicit distilled identity; positive-only Stage 1 by default; final-resolution request semantics; LTX-2.3 support; canonical checkpoint loading; accuracy coverage

What distillation saves

Distillation in this family does not reduce the Transformer parameter count: the dev and full-distilled checkpoints use the same model scale. It reduces inference compute through two mechanisms:

1. fixed, much shorter denoising schedules;
2. positive-only prediction instead of the additional CFG, STG, and isolated-modality passes used by the full guidance recipe.

The distilled LoRA similarly does not create a smaller Transformer. It changes the dev model's behavior so the high-resolution refinement stage can use the short positive-only schedule. Weight quantization is a separate optimization responsible for reducing checkpoint bytes and model-weight memory.

Current gaps

The existing LTX-2 two-stage path should be treated as an early full-distilled implementation, not as the standard upstream two-stage recipe.

• Model mode is inferred from the model path instead of an explicit recipe/checkpoint contract.
• Only the full-distilled model is accepted; the existing dev-plus-LoRA branch is unreachable.
• Stage 1 defaults to CFG even though the official full-distilled recipe is positive-only.
• Requested width and height are currently interpreted as Stage 1 dimensions, so upsampling changes the final output size. Upstream two-stage APIs interpret them as final dimensions.
• The dormant LoRA path assumes a fixed local layout and does not reset adapter state before the next request's Stage 1.
• Standard two-stage, HQ, and LTX-2.3 two-stage entry points are missing.
• Current one-stage execution is CFG-only and does not yet expose the full official multimodal-guidance behavior.
• Accuracy coverage does not yet independently guard dev one-stage, standard two-stage, and full-distilled outputs.

Checkpoint contracts

Recipe	Required checkpoint roles	Completed	Not yet supported / needs correction
Dev one-stage	Dev model components and text encoder	Componentized loading for both model versions	Optional official single-file loading is not required for initial support, but provenance should remain traceable to official weights
Standard two-stage	Dev model, distilled LoRA, spatial latent upsampler	Dev and upsampler loaders exist independently	Explicit auxiliary paths/sources, cross-source composition, adapter lifecycle, version-specific recipe selection
HQ two-stage	Same roles as standard two-stage	None	Stage-specific LoRA strengths and HQ schedule metadata
Full-distilled two-stage	Full-distilled model and spatial latent upsampler	One componentized LTX-2 path works	Explicit pipeline identity, official source support or a verified conversion, LTX-2.3 path

The implementation should allow the base components, distilled adapter, and upsampler to come from separate sources. This is necessary for LTX-2.3 and avoids encoding repository layout assumptions into pipeline logic.

Proposed development order

Phase 1: Shared LTX runtime foundation

• Consolidate common LTX-2/LTX-2.3 phase execution, request normalization, conditioning, latent operations, component discovery, and postprocessing.
• Keep model-version profiles explicit for architecture/config differences, schedule defaults, STG blocks, and numerical behavior.
• Preserve existing public one-stage T2V/I2V names and runtime feature integration.

Phase 2: Official dev one-stage behavior

• Add official video/audio CFG, STG, modality guidance, rescale, and per-request overrides.
• Align prompt conditioning, latent sampling, denoising math, and decode inputs with upstream.

Phase 3: Make two-stage recipes explicit

• Introduce a stage recipe describing model kind, schedule, guidance, adapter state, upsampling, and output audio source.
• Correct and expose the full-distilled two-stage recipe with positive-only execution and final-resolution semantics.
• Implement the standard dev-plus-distilled-LoRA two-stage recipe for LTX-2 T2V/I2V.
• Generalize the standard and full-distilled recipes to LTX-2.3.

Phase 4: HQ and runtime hardening

• Add the LTX-2.3 HQ two-stage recipe and second-order sampler.
• Add stage-specific distilled-LoRA strengths and defaults.
• Validate compile/offload and distributed-feature compatibility for each stage topology.

Other Specialized official pipelines

These use the same components and should build on the core stage/recipe abstraction rather than introduce independent model implementations.

Official specialized pipeline	Purpose	Not supported
Text-to-audio one-stage	Generate audio without video	Pipeline entry, audio-only latent/schedule/output path
Keyframe interpolation	Generate between supplied keyframes	Guiding-latent conditioning and two-stage recipe
Audio-to-video two-stage	Generate video while preserving input audio	Audio conditioning/freeze semantics and two-stage output policy
IC-LoRA pipeline	Image/video-controlled generation	Reference conditioning, IC-LoRA lifecycle, two-stage integration
Retake	Regenerate a selected temporal region	Source-video masking/conditioning and range handling
HDR IC-LoRA	Produce HDR video from a reference	HDR conditioning, decode/output contract, specialized adapter
LipDub	Re-voice/lip-sync using video and audio references	Joint reference conditioning, frozen reference tokens, specialized adapter

Specialized pipelines are follow-up scope. The immediate priority is making the four core recipes correct and maintainable.

Constraints

• T2V and I2V may remain separate public classes, but must share stage orchestration and checkpoint recipes.
• Standard, HQ, and full-distilled two-stage recipes may share executors but must remain independently testable contracts.
• A stage must explicitly set its complete guidance and adapter state.
• Requested dimensions must consistently mean final output dimensions for two-stage APIs.
• Existing vLLM-Omni runtime features should be preserved unless a limitation is explicitly documented and tested.

References

• Upstream pipeline overview
• Upstream core model overview