vllm-project/vllm-omni

[Help Wanted][Examples] Ratchet Python examples toward shared task runners

Open

#6,260 opened on Aug 17, 2026

 (7 comments) (0 reactions) (2 assignees)Python (1,067 forks)github user discovery
CI/CDdiffusiondocumentationgood first issuehelp wantedrefactortts

Repository metrics

Stars
 (4,990 stars)
PR merge metrics
 (Avg merge 7d 6h) (387 merged PRs in 30d)

Description

Motivation

#6046 added a diff-scoped policy that blocks new model-specific Python examples in the precheck/review skills. Its own impact statement correctly notes that the policy is advisory and does not add a hard CI gate.

Since that policy merged, six net-new model-specific Python paths and one policy-relevant rename have landed under examples/. This is a process gap, not a criticism of those contributions: a rule that exists only in review guidance is easy to miss.

#6076 demonstrated the intended migration pattern for LingBot-Video: preserve model defaults and behavior behind the shared text_to_video.py runner, use the canonical request envelope and model_extras contracts, add model-neutral tests and hardware parity evidence, repoint documentation, then retire the dedicated script.

This issue turns that pattern into an enforceable, incremental cleanup plan and asks the community to help with independently claimable PRs. It complements #6116, which governs examples documentation and navigation. #6116 may catalog grandfathered examples, but it does not exempt new model-specific Python files.

Target state

  • One maintained Python entrypoint per user task or protocol.
  • Model-specific defaults, prompt/request adaptation, and output contracts live in production model modules or vllm_omni/model_extras/.
  • Model commands, hardware notes, and validation evidence live in task documentation or recipes.
  • Existing example debt is grandfathered and migrated deliberately; there is no bulk deletion.

Work plan

Please follow this dependency order. Comment before starting and claim one unchecked item. If a migration exposes a missing generic capability, add that capability in a small prerequisite PR instead of adding model-name branches to the shared runner.

1. Make the examples policy enforceable

  • Add a small CI ratchet for Python paths introduced under examples/.

Acceptance criteria:

  • Compare the PR head with its merge base and inspect added, copied, and renamed destination paths (ACR) under examples/**/*.py.
  • Grandfather paths that exist when the ratchet lands. Modifications and deletions pass; the grandfathered baseline may only shrink.
  • Reject every newly introduced Python path by default, including copies and renames.
  • Permit a genuinely model-neutral, user-facing task/protocol runner only through an exact-path exception with a written justification and explicit maintainer review. Do not allow directory globs or name-based loopholes.
  • Test additions, copies, renames, modifications, deletions, non-Python files, and approved exceptions.
  • Ensure CI has enough base history and fails clearly rather than silently skipping the check.
  • Point failures toward shared runners, production adapters/model_extras, recipes, tests, tools, apps, or benchmarks as appropriate.
  • Do not migrate or delete existing examples in the ratchet PR.

2. Harden the existing X-to-video/audio runner

The current runner silently resamples audio to 16 kHz and keeps only seconds 1–3, branches request construction on DreamID versus Magi, and assumes model-specific frame layouts, ranges, FPS, and audio sample rates during export. No CPU test directly exercises the shared runner.

Acceptance criteria:

  • Define an importable canonical request builder: prompt, optional negative_prompt, multi_modal_data containing only supplied image/audio, and optional additional_information. Model knobs use explicit config/extra channels, not model-name conditionals.
  • Remove implicit audio cropping and resampling. Any trim or resample must be explicit, documented, and tested.
  • Resolve one canonical media representation. Preferred: decoded RGB images and (waveform, sample_rate) audio with cardinality preserved. An acceptable alternative is unmodified paths with all decoding owned by production model processors. Do not silently mix representations by model.
  • Normalize OmniRequestOutput by documented structure and metadata rather than model name. Cover PIL frame sequences, THWC uint8, THWC float [0, 1], and CTHW float [-1, 1]; reject ambiguous shapes/ranges.
  • Read generated audio, FPS, and sample rate from multimodal_output, with explicit CLI fallbacks. Muxing conditioning audio must be opt-in, not inferred from a model name.
  • Extract pure request, sampling, frame-normalization, and mux-metadata helpers; keep orchestration thin and close Omni in finally.
  • Add model-neutral CPU tests for text/image/audio combinations, repeated media, untrimmed audio, output variants, optional audio, metadata precedence, invalid output, and one fake-Omni CLI path.
  • Keep the documented DreamID and Magi commands working except for the intentional removal of hidden audio cropping. Retain existing model E2E coverage.
  • Do not migrate Wan or LongCat in this PR.

3. Migrate one model per PR

For each model migration:

  • Preserve supported modes, defaults, and user-visible behavior through presets, production adapters, and explicit extras. Do not grow a model-dispatch ladder in the shared runner.
  • Add CPU request/output contract tests and retain the relevant model E2E tests.
  • Record a reproducible hardware smoke/parity command with model, commit, seed, sampling settings, media metadata, and result.
  • Repoint README, recipe, generated-doc, benchmark, and test references before removing the dedicated entrypoint.
  • Keep the PR limited to one model and include a DCO Signed-off-by trailer.

Wan should go first because its request is already close to the canonical prompt / negative_prompt / multi_modal_data envelope. LongCat is a later, maintainer-assisted migration: its multi-speaker JSON cases, conditioning-audio merge, and mux behavior must become production-owned or genuinely reusable capabilities, not copied into the generic runner.

TTS is a separate follow-up project

The offline text_to_speech hub currently has 17 model-specific folders. Do not bundle those migrations into this tracker. Open a separate design issue for a shared text_to_speech.py contract first, land the model-neutral runner and CPU contract tests without deleting model scripts, then migrate exactly one model per PR.

Coordination

  • Comment with the checklist item you want to claim and wait for confirmation so work is not duplicated.
  • Start a migration only after its shared-runner prerequisite has merged.
  • Link this issue from each PR and report any generic-contract gap before implementing a model-specific workaround.
  • Small reviewable PRs are preferred; unrelated cleanup belongs elsewhere.

Contributor guide