Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

[Regression] 2026.3.1-gpu hangs with 200% CPU in sched_yield / libopencl-clang on dynamic shape embeddings (working fine in 2026.3-gpu)

Abierto
  • ¿Algo no está bien?
#4,510 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 2 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
35/100
Tipo de issue
Error
Claridad
Necesita aclaración
Estado de actividad
Activo
Stack tecnológico
cpp, docker, linux
Área
backend, machine-learning, operating-systems

Línea de trabajo

Empieza reproduciendo el bloqueo con el comando de Docker proporcionado, solicitudes de embeddings de forma dinámica, varios --rest_workers y la imagen 2026.3.1-gpu; compáralo con 2026.3-gpu. Usa los backtraces de GDB proporcionados para investigar libigdrcl.so, libopencl-clang2.so.16 y libopenvino_intel_gpu_plugin.so. Se considera terminado cuando las solicitudes concurrentes se completan sin un deadlock, un spinning persistente de sched_yield ni un servidor que deje de responder.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Describe the bug
In openvino/model_server:2026.3.1-gpu, serving embedding models on Intel iGPU with multiple workers (--rest_workers > 1, e.g. 4 or 16) causes OVMS to hang indefinitely when handling variable-length texts (or concurrent dynamic shape requests).

The container CPU stays pegged at ~200% indefinitely (two threads busy-spinning at 100% each in sched_yield) and stops responding to HTTP requests.

Attaching GDB to the hanging process reveals a spinlock deadlock between libigdrcl.so (Intel OpenCL runtime) and libopencl-clang2.so.16:

  • Thread 8 (LWP 96) is stuck inside libopencl-clang2.so.16 (Compile()).
  • Worker threads (e.g. Thread 22 LWP 82) are stuck in a busy spinloop in sched_yield() inside libigdrcl.so.

To Reproduce
Steps to reproduce the behavior:

  1. Model: OpenVINO/Qwen3-Embedding-0.6B-int8-ov (or any dynamic shape embedding model).
  2. Launch OVMS on Intel GPU with multiple workers: 
    docker run --device /dev/dri/renderD128:/dev/dri/renderD128 \
  -p 9200:9200 \
  openvino/model_server:2026.3.1-gpu \
  --model_repository_path /models \
  --source_model OpenVINO/Qwen3-Embedding-0.6B-int8-ov \
  --task embeddings \
  --pooling LAST \
  --target_device GPU \
  --rest_workers 4 \
  --rest_port 9200 \
  --model_name qwen3-embedding-0.6b
  3. Send variable-length text requests in quick succession or concurrently: 
    curl -X POST http://localhost:9200/v3/embeddings \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen3-embedding-0.6b", "input": "testing variable length dynamic shape sentence"}'
  4. OVMS hangs, container CPU pins at 200%, and requests never return.

Expected behavior
The server should properly synchronize dynamic shape OpenCL compilation without deadlocking or busy-spinning in sched_yield.

GDB Backtrace
GDB backtrace of the spinning process in 2026.3.1-gpu:

Thread 8 (LWP 96 - Compilation):

#0  0x00007fc... in ... from /usr/local/lib/libopencl-clang2.so.16
#1  0x00007fc... in Compile () from /usr/local/lib/libopencl-clang2.so.16
#2  0x00007fc... in ... from /usr/lib/x86_64-linux-gnu/intel-opencl/libigdrcl.so

Thread 22 (LWP 82 - Worker spinning in sched_yield):

#0  0x00007fc... in sched_yield () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007fc... in ... from /usr/lib/x86_64-linux-gnu/intel-opencl/libigdrcl.so
#2  0x00007fc... in ... from /ovms/lib/libopenvino_intel_gpu_plugin.so

Environment & Hardware Configuration

  1. Host CPU: 12th Gen Intel Core i5-1235U (10 cores, 12 threads)
  2. GPU: Intel Iris Xe Graphics (Alder Lake-UP3 GT2, 80 EU, PCI ID [8086:46a8])
  3. RAM: 64 GB
  4. Host OS: Linux 6.6.x (Debian 12)
  5. Model: OpenVINO/Qwen3-Embedding-0.6B-int8-ov
  6. Docker Image: openvino/model_server:2026.3.1-gpu (Broken) vs openvino/model_server:2026.3-gpu (Working)

Root Cause Analysis & Verified Workarounds
The root trigger is that when multiple worker threads (--rest_workers > 1) handle concurrent embedding requests with variable-length text, multiple threads simultaneously call into the Intel OpenCL driver (libigdrcl.so) and trigger JIT compilation in libopencl-clang2.so.16, resulting in a spinlock deadlock (sched_yield). Furthermore, in 2026.3.1-gpu, once dynamic compilation is triggered, background compiler threads can remain permanently spinning at 200% CPU even after HTTP calls complete.

Workarounds:

  1. Reverting to openvino/model_server:2026.3-gpu with --rest_workers 1 and --truncate true: requests execute smoothly at ~40ms latency and CPU drops back to 0.07% immediately upon completion.
  2. Under 2026.3.1-gpu, setting --rest_workers 1 and --truncate true serializes requests at the HTTP layer, but libopencl-clang threads may still leak CPU loops after JIT compilation.
Lenguaje dominante
C++
Estrellas
952
Forks
285
Merge medio
3 d 2 h
PR fusionados (30 d)
44

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de openvinotoolkit/model_server

Todos los issues de openvinotoolkit/model_server

Issues similares

Más issues de C++

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.