Repository metrics
- Stars
- (4,990 stars)
- PR merge metrics
- (PR metrics pending)
Description
Motivation.
This PR proposes a template for the offline inference example document and respective test script implementation.
🙋: help wanted!
| Examples Docs Refinement | Test script | Assignee | PR |
|---|---|---|---|
examples/offline_inference/text_to_image/README.md |
tests/examples/offline_inference/test_text_to_image.py |
@fhfuih | #1910 |
examples/offline_inference/text_to_video/text_to_video.md |
tests/examples/offline_inference/test_text_to_video.py |
@jackywangno007-cyber | |
examples/offline_inference/text_to_audio/README.md |
tests/examples/offline_inference/test_text_to_audio.py |
@mistcoversmyeyes | |
examples/offline_inference/image_to_image/image_to_image.md |
tests/examples/offline_inference/test_image_to_image.py |
@wtomin | #2417 |
examples/offline_inference/image_to_video/README.md |
tests/examples/offline_inference/test_image_to_video.py |
@loveysuby | #3573 |
examples/offline_inference/text_to_audio/README.md |
tests/examples/offline_inference/test_text_to_audio.py |
🙋 |
1. Offline Inference Example Document
All .md files under docs/user_guide/examples/ are auto-generated at build time and should not be edited manually. The true sources are the corresponding README.md or standalone .md files in the examples/ directory.
The offline inference documents under docs/user_guideexamples/offline_inference/ about diffusion models are five:
text_to_audio/*.mdimage_to_video/*.mdimage_to_image/*.mdtext_to_video/*.mdtext_to_image/*.md
In addtion, there are some independent models offline inference document, for example, examples/offline_inference/bagel/README.md.
Each document has different structure. Some information might be missing. It is better to follow a unified template for these document
2. Offline Inference Example Test
For each task, there is supposed to be one test script that verify the functionality of all the examples in the current example document. For example:
tests/examples/offline_inference/test_text_to_image.py
One related PR is #1910. test_text_to_image.py will automatically extract the code snippets from the respective readme document (examples/offline_inference/text_to_image/README.md), and run them to verify its functionality.
Proposed Change.
I have a suggested template for offline inference example:
# Task Name
Brief introduction.
## Table of Content
## Overview
Task introduction
### Supported Models
| Model | Image/Video Shape | Peak VRAM (GiB) | Model Weights (GiB) |
| `Model/Name-A` | height x width | xxx| xxx|
| `Model/Name-B` | height x width | xxx | xxx|
!!! info
Peak VRAM: based on basic single-card usage, batch size =1, without any acceleration/optimization features. Some models weights cannot fit into one card with 80GiB VRAM, which may need to use cpu offloading.
Model Weights: the VRAM consumption of model weights (usually BF16) is printed as `Model loading took xxx GiB and xxx seconds` in the logging.
Default model : `Model/Name-A`
## Prerequisites (Optional)
install special dependencies, needs to login HF to access gated model, etc.
For some latest models, it may require special diffusers version.
## Quick Start
Use the default model
### Python API
For example:
from vllm_omni.entrypoints.omni import Omni
if __name__ == "__main__":
omni = Omni(model="Qwen/Qwen-Image")
prompt = "a cup of coffee on the table"
outputs = omni.generate(prompt)
images = outputs[0].request_output[0].images
images[0].save("coffee.png")
### Local CLI Usage
The same example, but in local CLI usage
python text_to_image.py \
--model "Qwen/Qwen-Image" \
--prompt "a cup of coffee on the table" \
--output coffee.png
## Key Arguments
A table for parameters, including Argument、Type、Default、Description
| Argument | Type | Default | Description |
| --------- | ---- | ------- | ------------- |
| `height` | int | `{default}` | Output image height in pixels |
| `width` | int | `{default}` | Output image width in pixels |
## More CLI Examples
change the model names, change parameters, batch request (optional)
### Advanced Features (Optional)
e.g., CFG-parallel, maybe provide link to `user_guide/diffusion_acceleration.md#supported-models`.
e.g., inference with LoRA weight
## Web UI Demo (Optional)
Gradio demo launching method
## FAQ (Optional)
Feedback Period.
No response
CC List.
No response
Any Other Things.
@hsliuustc0106 @ZJY0516 @SamitHuang @dongbo910220 @lishunyang12 @mxuax @fhfuih
Before submitting a new issue...
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.