Float64 miscompile with FP64 emulation on Arc A750 at sub-group size 32 (NEO 26.35 / IGC 2.41.5); KernelAbstractions kernels hit it by default

Start with the reproducer and compare the reported results for Float64 and Float32 at subgroup sizes 8, 16, and 32. The report identifies IGC 2.41.5 SIMD32 code generation with FP64 emulation as the failing path and points to KI.kernel_function and device_limits().sub_group_size as the oneAPI.jl entry points to inspect. Done means the wrong result is explained and the reported failure is addressed or documented; the issue notes that forwarding the regression to IGC may be needed.

On an Arc A750 (no native FP64, so Float64 is emulated), this kernel gives a wrong Float64 result when compiled with sub_group_size=32. Sub-group sizes 8 and 16, the default @oneapi launch, and Float32 are all correct.

In oneAPI.jl 2.10 the KernelAbstractions backend compiles every kernel for sub-group size 32 (KI.kernel_function → device_limits().sub_group_size). So with oneAPI.jl 2.10, any KA kernel with this pattern returns wrong results on this device. We found it through wrong Hessians in ExaModels.jl (madsuite-org/ExaModels.jl#347).

Reproducer

using oneAPI

# `a * b`, except that a zero `a` gives zero (branchless)
@inline amul(a, b) = (p = a * b; ifelse(iszero(a), zero(p), p))

@noinline function body(y, x, adj, adj2)
    @inbounds a = x[1]; @inbounds b = x[2]
    s = a + 1; c = b + 1
    v = amul(adj2, b * b)
    @inbounds y[1] += amul(adj * b, -s) + v * c * c
    @inbounds y[2] += (adj + adj2 * s * b)
    return
end

function kernel(y, x, adj, adj2)
    body(y, x, adj, adj2)
    return
end

for T in (Float32, Float64)
    x = T[-1.2, 1.0]; adj = T(2); adj2 = T(0)
    ref = zeros(T, 2); body(ref, x, adj, adj2)
    println(T, "  cpu: ", ref)
    for sg in (nothing, 8, 16, 32)
        y = oneAPI.zeros(T, 2)
        if sg === nothing
            @oneapi kernel(y, oneArray(x), adj, adj2)
        else
            @oneapi sub_group_size=sg kernel(y, oneArray(x), adj, adj2)
        end
        gpu = Array(y)
        println("  sub_group_size=", rpad(something(sg, "default"), 8), gpu, gpu ≈ ref ? "" : "   <-- WRONG")
    end
end

Output (oneAPI.jl 2.10.0, fresh environment):

Float32  cpu: Float32[0.4000001, 2.0]
  sub_group_size=default Float32[0.4000001, 2.0]
  sub_group_size=8       Float32[0.4000001, 2.0]
  sub_group_size=16      Float32[0.4000001, 2.0]
  sub_group_size=32      Float32[0.4000001, 2.0]
Float64  cpu: [0.3999999999999999, 2.0]
  sub_group_size=default [0.3999999999999999, 2.0]
  sub_group_size=8       [0.3999999999999999, 2.0]
  sub_group_size=16      [0.3999999999999999, 2.0]
  sub_group_size=32      [2.0, 2.0]   <-- WRONG

With adj2 = 0, y[1] should be -adj*b*(a+1) = 0.4. Instead it gets 2.0, the value computed for y[2] (adj + 0). Changing adj changes the wrong y[1] to match: it is always the y[2] value.

The same KA kernel (@kernel function k(y, x, adj, adj2); body(y, x, adj, adj2); end) gives [2.0, 2.0] for any workgroupsize, and also with oneAPIBackend(; always_inline = true).

It is very sensitive to the code shape

Small changes make it go away:

• inlining body into the kernel
• writing amul out by hand
• dropping either store, or using = instead of +=
• dropping * c * c

The trigger is not specific to any math function: the version above has only FP64 add/mul/neg, compare and select, and the original case used sin/cos.

Regression in the driver, not the SPIR-V

• SPIR-V is the same on both stacks. Dumped with @device_code, the SPIR-V for this kernel is the same with oneAPI.jl 2.9.2 and 2.10.0 except for one type declaration: the kernel state struct is {ptr, i64, i64} instead of [1 x ptr], and the code never uses it. The SPIR-V body is below.
• The old driver is correct at SIMD32. oneAPI.jl 2.9.2 (NEO 26.18.38308 / IGC 2.34.4) gives the correct result for this kernel. That driver already compiles it at sub-group size 32 by default (get_sub_group_size() returns 32), and IGC_ForceOCLSIMDWidth=32 doesn't change the result.
• The new driver is wrong at SIMD32 however it's selected. On oneAPI.jl 2.10.0 (NEO 26.35.39758 / IGC 2.41.5), the default launch picks sub-group size 8 and is correct. IGC_ForceOCLSIMDWidth=32 makes the default launch wrong too, so it's the SIMD32 code generation, not the sub_group_size plumbing.
• The SPIR-V back-end doesn't matter. The same wrong result happens with the Khronos translator (oneAPI.oneL0.LTS[] = true).

So the same SPIR-V is correct with IGC 2.34.4 and wrong with IGC 2.41.5 at SIMD32 with FP64 emulation. This looks like an IGC regression to forward to intel/intel-graphics-compiler. On the oneAPI.jl side, it might be worth not forcing sub-group size 32 for KA kernels on devices without native FP64, or at least documenting it. KA kernels can't currently opt out, since KI.kernel_function throws for a different sub_group_size.

%julia_body = OpFunction %void DontInline %10
%y = OpFunctionParameter %_ptr_Function__ptr_CrossWorkgroup_uchar
%x = OpFunctionParameter %_ptr_Function__ptr_CrossWorkgroup_uchar
%adj = OpFunctionParameter %double
%adj2 = OpFunctionParameter %double
%top = OpLabel
%34 = OpBitcast %_ptr_Function__ptr_CrossWorkgroup_double %x
%35 = OpLoad %_ptr_CrossWorkgroup_double %34 Aligned 8
%36 = OpBitcast %_ptr_CrossWorkgroup_uchar %35
%37 = OpLoad %double %35 Aligned 8
%38 = OpInBoundsPtrAccessChain %_ptr_CrossWorkgroup_uchar %36 %ulong_8
%39 = OpBitcast %_ptr_CrossWorkgroup_double %38
%40 = OpLoad %double %39 Aligned 8
%41 = OpFAdd %double %37 %double_1
%42 = OpFAdd %double %40 %double_1
%43 = OpFMul %double %40 %40
%44 = OpFMul %double %adj2 %43
%45 = OpFUnordNotEqual %bool %adj2 %14
%46 = OpSelect %double %45 %44 %14
%47 = OpBitcast %_ptr_Function__ptr_CrossWorkgroup_double %y
%48 = OpLoad %_ptr_CrossWorkgroup_double %47 Aligned 8
%49 = OpBitcast %_ptr_CrossWorkgroup_uchar %48
%50 = OpLoad %double %48 Aligned 8
%51 = OpFMul %double %adj %40
%52 = OpFNegate %double %41
%53 = OpFMul %double %51 %52
%54 = OpFUnordNotEqual %bool %51 %14
%55 = OpSelect %double %54 %53 %14
%56 = OpFMul %double %42 %46
%57 = OpFMul %double %42 %56
%58 = OpFAdd %double %55 %57
%59 = OpFAdd %double %50 %58
OpStore %48 %59 Aligned 8
%60 = OpInBoundsPtrAccessChain %_ptr_CrossWorkgroup_uchar %49 %ulong_8
%61 = OpBitcast %_ptr_CrossWorkgroup_double %60
%62 = OpLoad %double %61 Aligned 8
%63 = OpFMul %double %adj2 %41
%64 = OpFMul %double %40 %63
%65 = OpFAdd %double %adj %64
%66 = OpFAdd %double %65 %62
OpStore %61 %66 Aligned 8
OpReturn
OpFunctionEnd

Versions

Bad (oneAPI.jl 2.10.0):

Binary dependencies:
- NEO: 26.35.39758+2
- libigc: 2.41.5+2
- gmmlib: 22.10.0+0
- SPIRV_LLVM_Backend: 23.1.1+8
- SPIRV_LLVM_Translator: 23.1.1+3
- SPIRV_Tools: 2025.4.0+0
- oneAPI_Support: 0.11.0+0 (oneMKL v2026.1.0)

Toolchain:
- Julia: 1.13.1
- LLVM: 20.1.8

Julia packages:
- oneAPI.jl: 2.10.0
- GPUArrays: 12.1.1
- GPUCompiler: 2.18.0
- KernelAbstractions: 0.10.3
- KernelInterface: 0.4.3
- LLVM: 10.4.2
- SPIRVIntrinsics: 1.5.1

1 driver:
- 00000000-0000-0000-18e2-6b5701039b4e (v1.3.39758, API v1.17.0)

1 device:
- Intel(R) Arc(TM) A750 Graphics

Good (oneAPI.jl 2.9.2, same machine):

Binary dependencies:
- NEO: 26.18.38308+0
- libigc: 2.34.4+5
- gmmlib: 22.10.0+0
- SPIRV_LLVM_Backend: 23.1.1+8
- SPIRV_LLVM_Translator: 23.1.1+3
- SPIRV_Tools: 2025.4.0+0
- oneAPI_Support: 0.10.0+0 (oneMKL v2025.3.0)

Julia packages:
- oneAPI.jl: 2.9.2
- GPUArrays: 11.5.16
- GPUCompiler: 2.10.0
- KernelAbstractions: 0.9.45
- LLVM: 9.13.2
- SPIRVIntrinsics: 1.1.4

1 driver:
- 00000000-0000-0000-18e2-6bb1010395a4 (v1.3.38308, API v1.15.0)

Linux 7.0.0-38-generic. The results are the same with or without OverrideDefaultFP64Settings=1 IGC_EnableDPEmulation=1, and all runs used NEO_CACHE_PERSISTENT=0.