[RFC]: Support Deterministic Rollout for RL Training of Omni-Modal & Diffusion Models

Motivation.

In Reinforcement Learning (RL) training for generative models, especially for complex multi-modal and diffusion models, stability is a critical factor for successful convergence. Non-deterministic behavior during rollout generation can introduce significant variance in the reward signals, destabilizing the PPO (or other RL algorithms) training process.

Currently, generation might have subtle non-determinism (due to random seed handling across batches, CUDA non-determinism in certain operations, or floating point accumulations). We need to explicitly support deterministic rollout generation for RL training to ensure reproducible and stable generation given the same prompts and latent seeds.

Crucially, this includes ensuring batch invariance (generation results for a sequence must be identical regardless of other sequences in the same batch) and bit-wise alignment (strict numerical reproducibility across runs given the same seed, inputs, and environment).

This is critical for:

1. Omni-modality models like qwen3-omni
2. Diffusion-only models like qwen-image

Proposed Change.

Introduce a robust mechanism to guarantee deterministic rollout generation during RL training.

Specifically:

• Batch Invariance: Explicit per-sequence seed control in SamplingParams that strictly isolates random state per request, ensuring that batch composition, padding, or concurrent requests do not affect individual sequence outputs.
• Bit-wise Alignment: Provide an option or backend configuration to enforce deterministic CUDA operations (e.g., torch.backends.cudnn.deterministic = True and setting torch.use_deterministic_algorithms(True)) to eliminate floating-point non-determinism (e.g., from atomic additions or non-deterministic reductions).
• Consistent random state management for diffusion models (qwen-image) during the denoising steps (e.g., in DDIM/Euler schedulers).
• Consistent random state management for multi-modal token generation in qwen3-omni.

Feedback Period.

1-2 weeks.

CC List.

@hsliuustc0106 @Gaohan123 @princepride @knlnguyen1802 @zhtmike

Any Other Things.

This feature will significantly improve the stability of RLHF/DPO training pipelines for our newer multi-modal models.

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.