[GSD-13927] Deadlock in zeCommandListAppendLaunchKernelWithArguments with concurrent launches: SVMAllocsManager::containerLockedById shared across threads (setKernelState)

Read CommandList::setKernelState(), SVMAllocsManager::obtainReadContainerLock(), and getSVMAlloc() in the compute-runtime code, then trace the nested KernelImp::setArgBuffer() call described in the report. Reproduce with the linked l0_mt_submit.cpp steps and compare against the workaround setting. Done means concurrent launches complete without deadlock while the USM reuse cleaner is active.

Pre-submission Checklist

• I am using the latest GPU driver version (download center)
• I have searched for similar issues and found none

GPU Hardware

Intel Arc Pro B70 (4 cards; one card is enough to reproduce)

GPU Detailed Information

Name : Intel(R) Arc(TM) Pro B70 Graphics
DriverVersion : 32.0.101.8976
DriverDate : 20260828000000.000000-000
VideoProcessor : Intel(R) Arc(TM) Pro B70 Graphics Family
AdapterRAM : 4293918720
PNPDeviceID : PCI\VEN_8086&DEV_E223&SUBSYS_0105172F&REV_00\6&2C26ECC2&0&00080009

Name : Intel(R) Arc(TM) Pro B70 Graphics
DriverVersion : 32.0.101.8976
DriverDate : 20260828000000.000000-000
VideoProcessor : Intel(R) Arc(TM) Pro B70 Graphics Family
AdapterRAM : 4293918720
PNPDeviceID : PCI\VEN_8086&DEV_E223&SUBSYS_60251849&REV_00\6&A0CECBF&0&00080019

Name : Intel(R) Arc(TM) Pro B70 Graphics
DriverVersion : 32.0.101.8976
DriverDate : 20260828000000.000000-000
VideoProcessor : Intel(R) Arc(TM) Pro B70 Graphics Family
AdapterRAM : 4293918720
PNPDeviceID : PCI\VEN_8086&DEV_E223&SUBSYS_60251849&REV_00\6&2CA3521A&0&00080009

Name : Intel(R) Arc(TM) Pro B70 Graphics
DriverVersion : 32.0.101.8976
DriverDate : 20260828000000.000000-000
VideoProcessor : Intel(R) Arc(TM) Pro B70 Graphics Family
AdapterRAM : 4293918720
PNPDeviceID : PCI\VEN_8086&DEV_E223&SUBSYS_17018086&REV_00\6&1D38AD12&0&00080009

Driver Installation Details

Driver 32.0.101.8976 (2026-08-27), installed from Intel's Arc Pro driver package. Level Zero version reported by SYCL: 1.15.39183. Also reproduced on 32.0.101.8805 (L0 1.15.37858).

Windows Build Information

WindowsProductName : Windows 10 Pro
WindowsVersion : 2009
OsHardwareAbstractionLayer : 10.0.26100.1

Actual Behavior

With several host threads launching kernels at the same time (through SYCL / Unified Runtime L0 v2 adapter, which uses zeCommandListAppendLaunchKernelWithArguments), the process deadlocks inside ze_intel_gpu64.dll within seconds. All blocked threads wait in RtlAcquireSRWLockShared / RtlAcquireSRWLockExclusive under zeCommandListAppendLaunchKernelWithArguments, plus one driver-internal thread (the USM reuse cleaner) waiting exclusive. It only happens when something frees USM (oneMKL gemm scratch in our reproducer), which keeps the cleaner thread taking SVMAllocsManager::mtx exclusive.

Expected Behavior

Concurrent kernel launches from multiple threads complete without deadlock.

Reproduction Rate

Always reproduces - 100%

Steps to Reproduce

1. Build the reproducer (single file, linked below):
   icx -fsycl /O2 /EHsc /std:c++17 /DUSE_MKL /DMKL_ILP64 /Qmkl l0_mt_submit.cpp /Fe:l0_mt_submit_mkl.exe /link mkl_sycl_blas_dll.lib
2. set ONEAPI_DEVICE_SELECTOR=level_zero:0
3. l0_mt_submit_mkl.exe 4 60 3 32
   (4 threads, 60 s, mode 3 = kernels + USM memcpy + oneMKL gemm, flag 32 = warm-up)
4. Within ~5 s, per-thread progress stops; the program reports no progress for 30 s and exits with code 2 (or stays hung).
5. With UR_L0_V2_DISABLE_ZE_LAUNCH_KERNEL_WITH_ARGS=1 set, the same command completes: 10 of 10 runs, ~1M iterations per thread.

Is this a regression?

• Yes, this is a regression - functionality that previously worked is now broken

Last Known Working Driver Version

Unknown: Introduced by commit 6d6715bebf ("performance: Lock svm allocs mutex once in appendLaunchKernelWithArgs", NEO-15774, first in 25.44.36015). Not bisected across driver releases; we have no older driver result.

First Known Failing Driver Version

32.0.101.8805

API Call Logs

No response

Windows Event Viewer Logs

N/A - hang, not a crash

Crash Dumps / Minidumps

N/A - stacks-repro-utf8.txt

Source Code / Reproducer

https://github.com/Captain-Tripps/b70-lab/tree/main/reproducers/l0-deadlock (l0_mt_submit.cpp, single file ~200 lines; stacks and run logs in the same folder)

Command Line / Application Details

steps 1–3 from above

oneAPI Version (if applicable)

oneAPI version: Intel oneAPI DPC++/C++ Compiler 2026.0.0 (Build 20260331), oneMKL 2026.0; runtime DLLs ur_adapter_level_zero_v2.dll, sycl9.dll, mkl_sycl_blas.6.dll
System configuration: AMD EPYC 7402P, 4x Arc Pro B70 32 GB (you can skip the RAM and board details)
Screenshots / SSU report: skip both.

Additional notes:
Root cause found in compute-runtime (NEO). CommandList::setKernelState() (the zeCommandListAppendLaunchKernelWithArguments path) holds SVMAllocsManager::mtx (std::shared_mutex, which is an SRW lock on Windows) shared via obtainReadContainerLock(). That helper records the owner in SVMAllocsManager::containerLockedById, a single non-atomic std::thread::id for the whole driver. getSVMAlloc() skips locking only if containerLockedById == this_thread. With two threads in setKernelState() at once, the second overwrites (or, on exit, clears) the first one's id. The first thread's nested KernelImp::setArgBuffer() -> getSVMAlloc() then calls lock_shared() on a lock it already holds shared. If another thread is queued for mtx exclusive at that moment, the SRW lock blocks the recursive shared request and the process deadlocks. In our hang that thread was the USM reuse-cleaner thread in trimOldAllocs -> freeSVMAllocImpl -> freeSVMData, which also holds mtxForIndirectAccess. This was introduced by commit 6d6715bebf ("Lock svm allocs mutex once in appendLaunchKernelWithArgs", NEO-15774) and is still on master.

Suggested fix: make the owner marker per-thread (for example a thread_local record of which manager this thread has locked) instead of a shared member.

Confirmed workaround: UR_L0_V2_DISABLE_ZE_LAUNCH_KERNEL_WITH_ARGS=1. With our reproducer, the baseline hung in 6 of 6 runs and the workaround ran 10 of 10 without a hang (Arc Pro B70, driver 32.0.101.8976, Windows 11). The same NEO code is used on Linux, so Linux is likely affected too (not tested).

Also reported to intel/llvm#23530 (UR side, now known to be a driver issue) and community.intel.com/t/144119.

System Configuration

CsProcessors : {AMD EPYC 7402P 24-Core Processor }
CsTotalPhysicalMemory : 102941720576
CsManufacturer : To Be Filled By O.E.M.
CsModel : ROMED8-2T

Screenshots / Video

No response

Intel System Support Utility Report

No response

Additional Notes

No response