[SGLang] Decoding on a non-default stream still waits for work queued on the default stream

Open
  • Is something wrong?
#162 0 comments 0 reactions 1 assignee View on GitHub

Maintainers usually reply within 3 days

@eromomon is already working on this.

Since Nov 2, 2026.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
8/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
cpp, python
Domain
audio-video-rtc, performance

Research direction

Start by running the attached host_sync_stall_repro.py with --device xpu to reproduce the stall, then compare it with --device cuda. The cause is likely a host-side synchronization point in the XPU decode path that waits on the default stream, so trace how torch.xpu.stream is handled in the decode code. Done means decoding on its own stream stays near the idle time even while default-stream work is queued, matching the CUDA numbers.

Written by the indexing model from the issue text.

Description

Environment

torch 2.14.0+xpu, torchcodec 0.17.0 (+cpu wheel from download.pytorch.org/whl/xpu), torchcodec-xpu 0.17.0 (PyPI), FFmpeg 7.1 (libavcodec 61), Python 3.12, Intel Arc Pro B60 (BMG, 8086:E211, 2 codec engines), xe kernel driver (Linux 6.14), intel-media-driver iHD 25.4.5, libva 2.22, Level Zero loader 1.26.

Summary

While another thread in the same process keeps queuing GPU work on the default stream, XPU decoding slows in proportion to that work even when the decode runs on a stream of its own (with torch.xpu.stream(s):). On CUDA, the same script shows decoding on its own stream unaffected by work on the default stream, so an application can isolate decode there but not on XPU.

Reproduction

Single file, attached: host_sync_stall_repro.py. It generates its own 3 min 720p H.264 test clip (needs the ffmpeg CLI) and decodes 512 sampled frames while a background thread runs an image-processor-like compute op (bilinear resize + normalize of 64 frames of 720p; it does not use the media engine) in batches of 1, 3 or 10 ops, waiting for its stream after each batch. The decode runs on the default stream or on a stream of its own. The two streams share no tensors and the script adds no synchronization between them.

python host_sync_stall_repro.py --device xpu    # or --device cuda

Results

XPU
# Arc Pro B60, one card of 8 (ZE_AFFINITY_MASK=4), torchcodec 0.17.0+cpu + torchcodec-xpu 0.17.0, ffmpeg version 7.1.1-1ubuntu1.3
$ python host_sync_stall_repro.py --device xpu
torch 2.14.0+xpu, device Intel(R) Arc(TM) Pro B60 Graphics, 512 frames sampled from 5400, background op 17 ms
idle                                           5.2 s    1.0x idle
load on its own stream, 1 op per wait         13.9 s    2.7x idle
load on the same stream, 1 op(s) per wait     22.6 s    4.3x idle
load on the same stream, 3 op(s) per wait     50.1 s    9.6x idle
load on the same stream, 10 op(s) per wait    83.1 s   15.8x idle
decode on its own stream, 1 op(s) per wait    13.6 s    2.6x idle
decode on its own stream, 3 op(s) per wait    20.0 s    3.8x idle
decode on its own stream, 10 op(s) per wait    66.4 s   12.7x idle
CUDA, same script
# NVIDIA L40S, driver 580.178.04, torchcodec 0.17.0+cu130 (NVDEC backend), ffmpeg version 7.1.1
$ python host_sync_stall_repro.py --device cuda
torch 2.14.0+cu130, device NVIDIA L40S, 512 frames sampled from 5400, background op 7 ms
idle                                           2.9 s    1.0x idle
load on its own stream, 1 op per wait          5.3 s    1.8x idle
load on the same stream, 1 op(s) per wait     14.5 s    5.0x idle
load on the same stream, 3 op(s) per wait     41.0 s   14.1x idle
load on the same stream, 10 op(s) per wait    96.8 s   33.3x idle
decode on its own stream, 1 op(s) per wait     5.3 s    1.8x idle
decode on its own stream, 3 op(s) per wait     5.4 s    1.9x idle
decode on its own stream, 10 op(s) per wait     5.4 s    1.9x idle
Background ops queued on the default stream per wait B60, decode on default stream B60, decode on own stream L40S, decode on default stream L40S, decode on own stream
1 4.3x idle 2.6x 5.0x 1.8x
3 9.6x 3.8x 14.1x 1.9x
10 15.8x 12.7x 33.3x 1.9x

On CUDA a decode on its own stream stays at about 1.9x idle (the GPU is busy) however much work waits on the default stream. On XPU it still grows with that work.

Impact

We are moving request-video decode in SGLang serving from the CPU onto the XPU media engine. In a serving process other requests' image processing runs on the default stream, and putting decode on its own stream does not protect it on XPU, so with dense frame sampling individual hardware decodes took 100-300 s and SGLang falls back to CPU decode for such videos.

CC: @eromomon , @dvrogozh

Dominant language
C++
Stars
5
Forks
9
Avg merge
3d 10h
Merged PRs (30d)
11

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from intel/torchlib-xpu

All issues in intel/torchlib-xpu

Similar issues

More C++ issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.