Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

A failed SVM allocation returns a CLArray at address 0, and kernels that write to it fault the GPU

Open
  • Is something wrong?
#601 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
74/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
julia
Domain
backend, devtools

Research direction

Start with lib/cl/memory/svm.jl, where clSVMAlloc is called and its result is checked, then read the retry_reclaim implementation and its tests to understand the existing garbage collection and retry behavior. Reproduce the failure with the allocation loop in the issue. Done means failed SVM allocations trigger reclamation and then report an out-of-memory error instead of producing a CLArray at address 0.

Written by the indexing model from the issue text.

Description

When the device runs out of memory, clSVMAlloc returns NULL. cl.svm_alloc doesn't catch that, so the allocation becomes a CLArray at address 0. The first kernel that writes to it triggers an MMU fault (Xid 31 on NVIDIA), and every later OpenCL call fails with CL_UNKNOWN_ERROR_CODE (-9999). OpenCL.jl doesn't run the GC to free memory before an SVM allocation fails, so ordinary allocation-heavy loops end up there. In AcceleratedKernels' benchmarks on an RTX 5080 (16 GB), repeatedly calling AK.accumulate(op, x; dims=2) on 10^8 elements faulted after about 40 calls, each of which allocates a 400 MB result.

using OpenCL                      # NVIDIA's ICD: OCL_ICD_FILENAMES=libnvidia-opencl.so.1
keep = CLArray{Float32}[]
while isempty(keep) || pointer(keep[end]) != OpenCL.cl.CL_NULL
    push!(keep, CLArray{Float32}(undef, 2^28))   # 1 GiB each, until the device is full
end
x = keep[end]
@show length(keep) pointer(x)
x .= 1f0                                         # a kernel writes through the NULL pointer
sum(Array(x))

length(keep) = 15
pointer(x) = CLPtr{Float32}(0x0000000000000000)
ERROR: LoadError: CLError(code=-30, CL_INVALID_VALUE)     # from clEnqueueSVMMemcpy in Array(x)
error in running finalizer: OpenCL.cl.CLError(code=Int32(-9999), desc=:CL_UNKNOWN_ERROR_CODE)   # 15 times

NVRM: Xid (PCI:0000:01:00): 31, pid=2137476, name=julia, intr 00000000. MMU Fault: ENGINE GRAPHICS GPC1 GPCCLIENT_T1_10 faulted @ 0x0_00005000. Fault is of type FAULT_PDE ACCESS_TYPE_VIRT_WRITE

The cause is in lib/cl/memory/svm.jl (unchanged on main, fe5410b):

ptr = clSVMAlloc(ctx, flags, bytesize, alignment)
@assert ptr != C_NULL

clSVMAlloc returns a CLPtr{Cvoid}, and a CLPtr never compares equal to the Ptr{Cvoid} C_NULL, so the assertion always passes (reinterpret(CLPtr{Cvoid}, C_NULL) != C_NULL is true; comparing with CL_NULL works). Because clSVMAlloc reports failure through its return value and not an error code, it also bypasses retry_reclaim. That is the path where the checked wrappers run GC.gc and retry after CL_OUT_OF_RESOURCES or CL_MEM_OBJECT_ALLOCATION_FAILURE. So the SVM path neither frees garbage nor reports the failure.

Suggested fix: retry the allocation after collecting garbage, and throw an out-of-memory error if it still fails, as CUDA.jl's allocator does:

ptr = retry_reclaim(isequal(CL_NULL)) do
    clSVMAlloc(ctx, flags, bytesize, alignment)
end
ptr == CL_NULL && throw(OutOfGPUMemoryError(bytesize))   # or a CLError, since this lives in lib/cl

Environment: OpenCL.jl 1.0.0 (main has the same code), Julia 1.12.7, NVIDIA driver 615.78.08 (OpenCL platform "CUDA 13.4.96"), RTX 5080 16 GB, Linux 7.2. On this driver, allocations of a few MB come from a separate pool and kept succeeding, so only large allocations come back NULL. The same benchmark on an RTX 6000 Ada (48 GB, same driver) never filled the device and never faulted.

How this was ruled out as an AcceleratedKernels bug

The fault first showed up as AK scans along dims crashing on large inputs ((8, n÷8) along dims=2 and (n÷8, 8) along dims=1, at n = 10^7 and 10^8), with a write fault at a small address (0x0 to 0x20c000). It doesn't depend on the shape or the kernel:

  • Run on their own, the same scans are correct on NVIDIA OpenCL: 200 repetitions at 10^7 and 40 at 10^8, with a synchronization after each.
  • The same shapes are correct on CUDA.jl with --check-bounds=yes, which makes AK's @inbounds kernels check bounds (a deliberate out-of-bounds access in such a kernel does throw), and on PoCL and Intel's CPU runtime.
  • Repeating the dims=2 scan at 10^8 without a GC: the results land at increasing addresses while nvidia-smi climbs to 15.9 GB. The 39th result is at CLPtr(0x0), and its kernel faults with Xid 31 at 0x20c000.
  • In the original reports, other processes shared the GPU, which explains why it happened at 10^7 there.
Dominant language
Julia
Stars
313
Forks
50
Avg merge
1d 8h
Merged PRs (30d)
44

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from JuliaGPU/OpenCL.jl

All issues in JuliaGPU/OpenCL.jl

Similar issues

More Julia issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.