Intermittent GPU hang on ARL-H executing reduced OpenCL kernel

Start with the attached DebugOpenCLFindBlocksReduced.cpp and DebugOpenCLFindBlocksReduced.cl reproducer, then compile it with g++ DebugOpenCLFindBlocksReduced.cpp -lOpenCL and run it while checking dmesg for the i915 GPU hang. Compare the reproducer and IGC shader dumps across runs where the hang occurs and where it does not. Done means the reduced kernel runs reliably without a GPU hang.

Pre-submission Checklist

• I am using the latest GPU driver version (releases)
• I have searched for similar issues and found none

GPU Hardware

Intel Arc 140T

DRI Devices Information

lrwxrwxrwx. 1 root root   8 Nov  6 14:43 pci-0000:00:02.0-card -> ../card1
lrwxrwxrwx. 1 root root  13 Nov  6 14:43 pci-0000:00:02.0-render -> ../renderD128

GPU Detailed Information (lspci output)

00:02.0 VGA compatible controller: Intel Corporation Arrow Lake-P [Arc Pro 130T/140T] (rev 03) (prog-if 00 [VGA controller])
        DeviceName: To Be Filled by O.E.M.
        Subsystem: Lenovo Device 3fa0
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupts: unknown pin routed to IRQ 195, MSI(X) routed to IRQ 195
        IOMMU group: 0
        Region 0: Memory at 4818000000 (64-bit, prefetchable) [size=16M]
        Region 2: Memory at 4000000000 (64-bit, prefetchable) [size=256M]
        Expansion ROM at 000c0000 [virtual] [disabled] [size=128K]
        Capabilities: [40] Vendor Specific Information: Intel Capabilities v1
                CapA: Peg60Dis- Peg12Dis- Peg11Dis- Peg10Dis- PeLWUDis- DmiWidth=x4
                      EccDis- ForceEccEn- VTdDis- DmiG2Dis- PegG2Dis- DDRMaxSize=Unlimited
                      1NDis- CDDis- DDPCDis- X2APICEn- PDCDis- IGDis- CDID=0 CRID=0
                      DDROCCAP+ OCEn- DDRWrtVrefEn+ DDR3LEn+
                CapB: ImguDis- OCbySSKUCap- OCbySSKUEn- SMTCap- CacheSzCap 0x0
                      SoftBinCap- DDR3MaxFreqWithRef100=Disabled PegG3Dis-
                      PkgTyp- AddGfxEn- AddGfxCap- PegX16Dis- DmiG3Dis- GmmDis-
                      DDR3MaxFreq=2932MHz LPDDR3En-
        Capabilities: [70] Express (v2) Root Complex Integrated Endpoint, IntMsgNum 0
                DevCap: MaxPayload 128 bytes, PhantFunc 0
                        ExtTag+ RBE+ FLReset+ TEE-IO-
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
                        MaxPayload 128 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
                DevCap2: Completion Timeout: Range B, TimeoutDis+ NROPrPrP- LTR+
                         10BitTagComp+ 10BitTagReq+ OBFF Not Supported, ExtFmt+ EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
                         AtomicOpsCtl: ReqEn-
                         IDOReq- IDOCompl- LTR- EmergencyPowerReductionReq-
                         10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
        Capabilities: [ac] MSI: Enable+ Count=1/1 Maskable+ 64bit+
                Address: 00000000fee00018  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [d0] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [100 v1] Null
        Capabilities: [110 v1] Process Address Space ID (PASID)
                PASIDCap: Exec- Priv-, Max PASID Width: 14
                PASIDCtl: Enable- Exec- Priv-
        Capabilities: [200 v1] Address Translation Service (ATS)
                ATSCap: Invalidate Queue Depth: 00
                ATSCtl: Enable+, Smallest Translation Unit: 00
        Capabilities: [420 v1] Physical Resizable BAR
                BAR 2: current size: 256MB, supported: 256MB
        Capabilities: [320 v1] Single Root I/O Virtualization (SR-IOV)
                IOVCap: Migration- 10BitTagReq+ IntMsgNum 0
                IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy- 10BitTagReq-
                IOVSta: Migration-
                Initial VFs: 7, Total VFs: 7, Number of VFs: 0, Function Dependency Link: 00
                VF offset: 1, stride: 1, Device ID: 7d51
                Supported Page Size: 00000553, System Page Size: 00000001
                Region 0: Memory at 000000481a000000 (64-bit, prefetchable)
                VF Migration: offset: 00000000, BIR: 0
        Capabilities: [400 v1] Latency Tolerance Reporting
                Max snoop latency: 0ns
                Max no snoop latency: 0ns
        Kernel driver in use: i915
        Kernel modules: i915, xe

Driver Version

26.35.39758.21

Installed GPU Driver Packages

libsigc++20-2.12.2-1.fc44.x86_64
libsigc++30-3.8.1-1.fc44.x86_64
oclock-1.0.6-4.fc44.x86_64
opencl-filesystem-1.0-24.fc44.noarch
intel-gmmlib-22.10.2-1.fc44.x86_64
intel-gmmlib-devel-22.10.2-1.fc44.x86_64
oneapi-level-zero-1.33.1-3.fc44.x86_64
intel-opencl-clang-22.1.4-2.fc44.x86_64
oneapi-level-zero-zello_world-1.33.1-3.fc44.x86_64
oneapi-level-zero-devel-1.33.1-3.fc44.x86_64
opencl-headers-3.1-1.20260512git6fe718c.fc44.noarch
wine-opencl-11.20-1.0.1.2T.fc44.x86_64
intel-igc-2.41.16-2.fc44.x86_64
intel-igc-libs-2.41.16-2.fc44.x86_64
intel-level-zero-26.35.39758.21-2.fc44.x86_64
intel-ocloc-26.35.39758.21-2.fc44.x86_64
intel-opencl-26.35.39758.21-2.fc44.x86_64

Driver Installation Details

Stock Fedora 44 packages (installed intel-opencl), tested with versions reported above, currently in the updates-testing repo (no apparent difference from versions already in stable Fedora 44)

Linux Distribution

Fedora / RHEL

Other Linux Distribution

No response

Kernel Version & Boot Parameters

$ uname -r
7.2.15-200.fc44.x86_64
$ cat /proc/cmdline 
root=/dev/mapper/electra-btrfs ro rd.lvm.lv=electra/btrfs rd.lvm.lv=electra/btrfsB rd.luks.uuid=luks-097c6111-9eb5-49ff-a142-cd010e6c2147 rd.luks.uuid=luks-3d6c8dc9-5dd2-480d-b075-b55d6d45553d rd.lvm.lv=electra/swap resume=/dev/electra/swap quiet zswap.enabled=1 zswap.zpool=zsmalloc zswap.compressor=zstd crashkernel=2G-64G:256M,64G-:512M
$ lsmod | grep -E 'i915|xe'
xe                   5070848  0
drm_ttm_helper         20480  1 xe
drm_suballoc_helper    24576  1 xe
gpu_sched              77824  1 xe
drm_gpuvm              57344  1 xe
drm_exec               16384  2 drm_gpuvm,xe
drm_gpusvm_helper      69632  1 xe
i915                 5623808  87
drm_buddy              12288  2 xe,i915
i2c_algo_bit           24576  2 xe,i915
ttm                   167936  3 drm_ttm_helper,xe,i915
drm_display_helper    368640  2 xe,i915
video                  81920  3 ideapad_laptop,xe,i915
intel_vsec             28672  4 intel_pmc_ssram_telemetry,intel_pmc_pwrm_telemetry,pmt_telemetry,xe
cec                   110592  3 drm_display_helper,xe,i915

Actual Behavior

The attached OpenCL kernel (isolated and reduced from the NonbondedForce kernels in the OpenMM computational chemistry library, which also hangs reliably when running its tests) occasionally hangs when executed, with the following message in the Linux kernel log:

i915 0000:00:02.0: [drm] GPU HANG: ecode 12:10:85def5fa, in DebugOpenCLFind [50069]
i915 0000:00:02.0: [drm] DebugOpenCLFind[50069] context reset due to GPU hang

The issue seems very sensitive to seemingly irrelevant details in the kernel code - just removing some unused arguments can make the issue go away; semantically identical code shows very different hang/pass rate.

$ ./a.out 
OpenCL device: Intel(R) Arc(TM) Graphics
Atoms: 117649, blocks: 3677
Kernel resources: workGroup=1024 local=672 private=0
Repeat 0: completed
Repeat 1: completed
Repeat 2: completed
Repeat 3: completed
exception: clFinish(findBlocksWithInteractions) failed with OpenCL status -5

(number of completed repeats varies a lot between runs)

Expected Behavior

Kernel executing reliably without hanging.

Reproduction Rate

Always reproduces - 100%

Steps to Reproduce

1. Place attached reproducer (two files) in the same directory
2. Compile using g++ DebugOpenCLFindBlocksReduced.cpp -lOpenCL
3. Execute using ./a.out
4. Observe hang - exception in clFinish and message in dmesg

Is this a regression?

• Yes, this is a regression - functionality that previously worked is now broken

Last Known Working Driver Version

No response

First Known Failing Driver Version

No response

API Call Logs

No response

strace Logs

No response

System Logs / dmesg Output

Full kernel log from a reproduction after a fresh reboot: dmesg.log

i915 0000:00:02.0: [drm] GPU HANG: ecode 12:10:85def5fa, in DebugOpenCLFind [46987]
i915 0000:00:02.0: [drm] GPU error state saved to /sys/class/drm/card1/error
i915 0000:00:02.0: [drm] DebugOpenCLFind[46987] context reset due to GPU hang

Original GPU error state capture and version decoded using intel_error_decode from igt-gpu-tools-2.4-1.fc44.x86_64:
i915-decoded-error.txt
i915-error.txt

Backtrace (if crash or hang occurred)

No response

Source Code / Reproducer

• Compilation command: g++ DebugOpenCLFindBlocksReduced.cpp -lOpenCL
• Source files: DebugOpenCLFindBlocksReduced.cpp DebugOpenCLFindBlocksReduced.cl.txt (remove .txt extension after downloading)

Command Line / Application Details

./a.out

oneAPI Version (if applicable)

No response

Screenshots / Video

No response

Additional Notes

• Dumps produced using IGC_ShaderDumpEnable=1 on the reproducer: DebugOpenCLFindBlocksReduce_2235882.tar.gz