vllm-project/vllm-omni

[Bug]: weekly CI, tests/e2e/online_serving/test_flux_kontext_expansion.py, OOM

Open

#5,747 opened on Aug 4, 2026

View on GitHubContributor guide
 (0 comments) (0 reactions) (0 assignees)Python (1,067 forks)github user discovery
bugci-failurehelp wantedlow priority

Repository metrics

Stars
 (4,990 stars)
PR merge metrics
 (PR metrics pending)

Description

Your current environment

CI env

Your code version

0.26.0

67c54777bb22e9e7e08fdf7c47a64f06b566fc47

🐛 Describe the bug

https://buildkite.com/vllm/vllm-omni/builds/13066/canvas?sid=019fbf5a-afed-4c75-908f-648905799da6&tab=output

[2026-08-01T22:42:47Z] ERROR 08-01 22:42:47 [diffusion_worker.py:1399] Error executing method 'execute_model'. This might cause issues in distributed execution.
Actions for L9649
9649
[2026-08-01T22:42:47Z] ERROR 08-01 22:42:47 [diffusion_worker.py:1399] Traceback (most recent call last):
Actions for L9650
9650
[2026-08-01T22:42:47Z] ERROR 08-01 22:42:47 [diffusion_worker.py:1399]   File "/workdir/vllm_omni/diffusion/worker/diffusion_worker.py", line 1395, in execute_method
Actions for L9651
9651
[2026-08-01T22:42:47Z] ERROR 08-01 22:42:47 [diffusion_worker.py:1399]     return func(*args, **kwargs)
Actions for L9652
9652
[2026-08-01T22:42:47Z] ERROR 08-01 22:42:47 [diffusion_worker.py:1399]            ^^^^^^^^^^^^^^^^^^^^^
Actions for L9653
9653
[2026-08-01T22:42:47Z] ERROR 08-01 22:42:47 [diffusion_worker.py:1399]   File "/workdir/vllm_omni/diffusion/worker/diffusion_worker.py", line 458, in execute_model
Actions for L9654
9654
[2026-08-01T22:42:47Z] ERROR 08-01 22:42:47 [diffusion_worker.py:1399]     output = self.model_runner.execute_model(req, kv_prefetch_job=kv_prefetch_job)
Actions for L9655
9655
[2026-08-01T22:42:47Z] ERROR 08-01 22:42:47 [diffusion_worker.py:1399]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Actions for L9656
9656
[2026-08-01T22:42:47Z] ERROR 08-01 22:42:47 [diffusion_worker.py:1399]   File "/workdir/vllm_omni/diffusion/worker/diffusion_model_runner.py", line 564, in execute_model
Actions for L9657
9657
[2026-08-01T22:42:47Z] ERROR 08-01 22:42:47 [diffusion_worker.py:1399]     runner_output = self._execute_request_list(
Actions for L9658
9658
[2026-08-01T22:42:47Z] ERROR 08-01 22:42:47 [diffusion_worker.py:1399]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^
Actions for L9659
9659
[2026-08-01T22:42:47Z] ERROR 08-01 22:42:47 [diffusion_worker.py:1399]   File "/workdir/vllm_omni/diffusion/worker/diffusion_model_runner.py", line 505, in _execute_request_list
Actions for L9660
9660
[2026-08-01T22:42:47Z] ERROR 08-01 22:42:47 [diffusion_worker.py:1399]     raw_outputs = self.pipeline.forward(batch)
Actions for L9661
9661
[2026-08-01T22:42:47Z] ERROR 08-01 22:42:47 [diffusion_worker.py:1399]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Actions for L9662
9662
[2026-08-01T22:42:47Z] ERROR 08-01 22:42:47 [diffusion_worker.py:1399]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context
Actions for L9663
9663
[2026-08-01T22:42:47Z] ERROR 08-01 22:42:47 [diffusion_worker.py:1399]     return func(*args, **kwargs)
Actions for L9664
9664
[2026-08-01T22:42:47Z] ERROR 08-01 22:42:47 [diffusion_worker.py:1399]            ^^^^^^^^^^^^^^^^^^^^^
Actions for L9665
9665
[2026-08-01T22:42:47Z] ERROR 08-01 22:42:47 [diffusion_worker.py:1399]   File "/workdir/vllm_omni/diffusion/models/flux/pipeline_flux_kontext.py", line 787, in forward
Actions for L9666
9666
[2026-08-01T22:42:47Z] ERROR 08-01 22:42:47 [diffusion_worker.py:1399]     latents = self.diffuse(
Actions for L9667
9667
[2026-08-01T22:42:47Z] ERROR 08-01 22:42:47 [diffusion_worker.py:1399]               ^^^^^^^^^^^^^
Actions for L9668
9668
[2026-08-01T22:42:47Z] ERROR 08-01 22:42:47 [diffusion_worker.py:1399]   File "/workdir/vllm_omni/diffusion/models/flux/pipeline_flux_kontext.py", line 555, in diffuse
Actions for L9669
9669
[2026-08-01T22:42:47Z] ERROR 08-01 22:42:47 [diffusion_worker.py:1399]     noise_pred = self.predict_noise_maybe_with_cfg(
Actions for L9670
9670
[2026-08-01T22:42:47Z] ERROR 08-01 22:42:47 [diffusion_worker.py:1399]                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Actions for L9671
9671
[2026-08-01T22:42:47Z] ERROR 08-01 22:42:47 [diffusion_worker.py:1399]   File "/workdir/vllm_omni/diffusion/distributed/cfg_parallel.py", line 157, in predict_noise_maybe_with_cfg
Actions for L9672
9672
[2026-08-01T22:42:47Z] ERROR 08-01 22:42:47 [diffusion_worker.py:1399]     pred = self.predict_noise(**positive_kwargs)
Actions for L9673
9673
[2026-08-01T22:42:47Z] ERROR 08-01 22:42:47 [diffusion_worker.py:1399]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Actions for L9674
9674
[2026-08-01T22:42:47Z] ERROR 08-01 22:42:47 [diffusion_worker.py:1399]   File "/workdir/vllm_omni/diffusion/models/flux/pipeline_flux_kontext.py", line 487, in predict_noise
Actions for L9675
9675
[2026-08-01T22:42:47Z] ERROR 08-01 22:42:47 [diffusion_worker.py:1399]     return self.transformer(
Actions for L9676
9676
[2026-08-01T22:42:47Z] ERROR 08-01 22:42:47 [diffusion_worker.py:1399]            ^^^^^^^^^^^^^^^^^
Actions for L9677
9677
[2026-08-01T22:42:47Z] ERROR 08-01 22:42:47 [diffusion_worker.py:1399]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
Actions for L9678
9678
[2026-08-01T22:42:47Z] ERROR 08-01 22:42:47 [diffusion_worker.py:1399]     return self._call_impl(*args, **kwargs)
Actions for L9679
9679
[2026-08-01T22:42:47Z] ERROR 08-01 22:42:47 [diffusion_worker.py:1399]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Actions for L9680
9680
[2026-08-01T22:42:47Z] ERROR 08-01 22:42:47 [diffusion_worker.py:1399]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1790, in _call_impl
Actions for L9681
9681
[2026-08-01T22:42:47Z] ERROR 08-01 22:42:47 [diffusion_worker.py:1399]     return forward_call(*args, **kwargs)
Actions for L9682
9682
[2026-08-01T22:42:47Z] ERROR 08-01 22:42:47 [diffusion_worker.py:1399]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Actions for L9683
9683
[2026-08-01T22:42:47Z] ERROR 08-01 22:42:47 [diffusion_worker.py:1399]   File "/workdir/vllm_omni/diffusion/hooks/base.py", line 141, in __call__
Actions for L9684
9684
[2026-08-01T22:42:47Z] ERROR 08-01 22:42:47 [diffusion_worker.py:1399]     return registry.dispatch(*args, **kwargs)
Actions for L9685
9685
[2026-08-01T22:42:47Z] ERROR 08-01 22:42:47 [diffusion_worker.py:1399]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Actions for L9686
9686
[2026-08-01T22:42:47Z] ERROR 08-01 22:42:47 [diffusion_worker.py:1399]   File "/workdir/vllm_omni/diffusion/hooks/base.py", line 308, in dispatch
Actions for L9687
9687
[2026-08-01T22:42:47Z] ERROR 08-01 22:42:47 [diffusion_worker.py:1399]     args, kwargs = hook.pre_forward(self.module, *args, **kwargs)
Actions for L9688
9688
[2026-08-01T22:42:47Z] ERROR 08-01 22:42:47 [diffusion_worker.py:1399]                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Actions for L9689
9689
[2026-08-01T22:42:47Z] ERROR 08-01 22:42:47 [diffusion_worker.py:1399]   File "/workdir/vllm_omni/diffusion/offloader/sequential_backend.py", line 106, in pre_forward
Actions for L9690
9690
[2026-08-01T22:42:47Z] ERROR 08-01 22:42:47 [diffusion_worker.py:1399]     self._to_gpu(module)
Actions for L9691
9691
[2026-08-01T22:42:47Z] ERROR 08-01 22:42:47 [diffusion_worker.py:1399]   File "/workdir/vllm_omni/diffusion/offloader/sequential_backend.py", line 98, in _to_gpu
Actions for L9692
9692
[2026-08-01T22:42:47Z] ERROR 08-01 22:42:47 [diffusion_worker.py:1399]     self._move_params(module, self.device, non_blocking=False)
Actions for L9693
9693
[2026-08-01T22:42:47Z] ERROR 08-01 22:42:47 [diffusion_worker.py:1399]   File "/workdir/vllm_omni/diffusion/offloader/sequential_backend.py", line 59, in _move_params
Actions for L9694
9694
[2026-08-01T22:42:47Z] ERROR 08-01 22:42:47 [diffusion_worker.py:1399]     data = p.data.to(target_device, non_blocking=non_blocking)
Actions for L9695
9695
[2026-08-01T22:42:47Z] ERROR 08-01 22:42:47 [diffusion_worker.py:1399]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Actions for L9696
9696
[2026-08-01T22:42:47Z] ERROR 08-01 22:42:47 [diffusion_worker.py:1399] torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 54.00 MiB. GPU 0 has a total capacity of 22.05 GiB of which 33.00 MiB is free. Process 20 has 260.00 MiB memory in use. Including non-PyTorch memory, this process has 21.72 GiB memory in use. Of the allocated memory 21.47 GiB is allocated by PyTorch, and 5.65 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)

Before submitting a new issue...

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

Contributor guide

Research direction
Investigate the CUDA OOM by reproducing the failing test locally, profiling GPU memory usage with PyTorch's memory profiler or nvidia smi, and examining the sequential offloader's parameter transfer logic in diffusion/offloader/sequential backend.py to identify where memory is being exhausted.
Tech stack
pythonpytorch
Domain
backendperformancetesting qa
Issue type
Bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
4
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-2 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Fresh
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
GitPythonPyTorch
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
30