[New Model] PrismAudio (Video-to-Audio Generation) · vllm-project/vllm-omni#2140

2026-03-24T15:00:05.000Z

### Model [PrismAudio](https://prismaudio-project.github.io/) — a video-to-audio generation framework from FunAudioLLM. Generates environment sounds and sound effects synchronized with video content (e.g. footsteps, rain, metal impacts). 518M parameters, generates 9s audio in 0.63s. Published at ICLR 2026. ### Architecture Diffusion-based model with decomposed chain-of-thought reasoning. Uses four reward models (semantic, temporal, aesthetic, spatial) for multi-objective optimization via reinforcement learning (Fast-GRPO). Key components: - Diffusion backbone for audio generation - Chain-of-thought decomposition for multi-dimensional planning (semantic, temporal, aesthetic, spatial) - Multi-reward RL training with Fast-GRPO for efficient optimization ### Fit with vllm-omni Single-stage diffusion pipeline, similar to Stable Audio and AudioX. Could reuse the existing diffusion infrastructure (DiffusionAttention, pipeline base classes). ### References - Project page: https://prismaudio-project.github.io/ - Paper: arXiv:2511.18833 - HuggingFace: https://huggingface.co/FunAudioLLM/PrismAudio - ModelScope: https://www.modelscope.cn/models/iic/PrismAudio - Demo: https://huggingface.co/spaces/FunAudioLLM/PrismAudio

(4 comments) (0 reactions) (1 assignee)Python (1.067 forks)github user discovery

good first issuehelp wantednew model

Métricas do repositório

Stars: (4.990 stars)
Métricas de merge de PR: (Métricas PR pendentes)

Description

Model

PrismAudio — a video-to-audio generation framework from FunAudioLLM. Generates environment sounds and sound effects synchronized with video content (e.g. footsteps, rain, metal impacts). 518M parameters, generates 9s audio in 0.63s. Published at ICLR 2026.

Architecture

Diffusion-based model with decomposed chain-of-thought reasoning. Uses four reward models (semantic, temporal, aesthetic, spatial) for multi-objective optimization via reinforcement learning (Fast-GRPO).

Key components:

Diffusion backbone for audio generation
Chain-of-thought decomposition for multi-dimensional planning (semantic, temporal, aesthetic, spatial)
Multi-reward RL training with Fast-GRPO for efficient optimization

Fit with vllm-omni

Single-stage diffusion pipeline, similar to Stable Audio and AudioX. Could reuse the existing diffusion infrastructure (DiffusionAttention, pipeline base classes).

References

Project page: https://prismaudio-project.github.io/
Paper: arXiv:2511.18833
HuggingFace: https://huggingface.co/FunAudioLLM/PrismAudio
ModelScope: https://www.modelscope.cn/models/iic/PrismAudio
Demo: https://huggingface.co/spaces/FunAudioLLM/PrismAudio

Guia do colaborador

Direção de pesquisa: Estude o artigo e o código do PrismAudio e integre o modelo no pipeline de difusão do vllm omni reutilizando a infraestrutura existente, como DiffusionAttention e classes base do pipeline.
Pilha de tecnologia: pythonpytorch
Domain: machine learningai
Tipo Issue: Funcionalidade
Difficulty: 4
Tempo estimado: 3-5 dias
Status da atividade: Ativo
Clarity: Claro
Prerequisites: PythonPyTorchDiffusion Models
Simpatia para novatos: 30