Device-side backtraces make kernels 10x slower to build at -g2 on Intel GPUs
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 52/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Domain
- compilers, performance
Research direction
Start with the store_cstring! loop and the report_exception_frame calls emitted for -g2, then inspect finish_linked_module! to understand where the linked kernel and its string data are registered. Prototype compile-time string interning so the device stores integer IDs and the host resolves the corresponding names and files. Done means the device no longer copies strings byte by byte, the -g2 build times and ISA size return near the stubbed baseline, and the OpenCL test suite passes.
Written by the indexing model from the issue text.
Description
At -g2, the device-side backtrace recording from #478 makes bounds-checked kernels an order of magnitude slower to build on Intel's GPU driver, and about 3x of that comes from store_cstring!, the byte loop that copies each frame's function and file name into the mailbox. At the default -g1 there is no such regression.
Same issue as in oneAPI.jl (JuliaGPU/oneAPI.jl#731), where the per-frame string copies blew up IGC's output 20x.
Measurements
Arc A770, Intel OpenCL Graphics (system NEO), NEO_CACHE_PERSISTENT=0. SPIR-V dumped with JULIA_OPENCL_DUMP_DIR from a workload (a plain a[i] = b[i] + 1f0 kernel, and A[I, :] for GPUArrays' getindex kernel), then clBuildProgram timed on each module separately (min of 5). Variants: base is cee0834^ (before #478), main is 0739da8, stub is main with the store_cstring! loop removed (it keeps the calls and writes only the terminator, which approximates recording just a pointer).
| kernel | -g |
base | main | stub |
|---|---|---|---|---|
| simple | 1 | 15 ms, 631 ISA lines | 28 ms, 1133 | 28 ms, 1133 |
| getindex | 1 | 34 ms, 2847 | 37 ms, 2062 | 38 ms, 2062 |
| simple | 2 | 15 ms, 631 | 203 ms, 19061 | 80 ms, 6742 |
| getindex | 2 | 34 ms, 2847 | 373 ms, 27727 | 98 ms, 8946 |
Binary sizes follow the same pattern (simple at -g2: 12 KB base, 158 KB main, 70 KB stub; getindex: 30 KB, 228 KB, 92 KB). Broadcast kernels without checks are unaffected.
The SPIR-V contains only one DontInline copy of store_cstring! per buffer size, and IGC keeps the calls (27 in simple, 59 in getindex), but the two byte loops still add 12-19k lines of SIMD32 ISA per kernel.
The full test suite on the A770 (--platform=graphics --jobs=8) takes 5m47s at -g1 and 8m00s at -g2 on main, vs 6m10s at -g2 with the stub.
PoCL (pocl_jll, CPU) barely notices: -g2 getindex builds in 16 ms (12 ms base, 13 ms stub).
Cause
GPUCompiler emits a report_exception_frame(idx, func, file, line) call per frame at every throw site at -g2, and each one copies two strings byte by byte from program-scope constants into the host-visible mailbox. -g1 is fine because @gputhrow stores literal words and report_exception returns early.
Even with the loops stubbed, -g2 stays ~5x slower to build than before #478 (the per-frame claim and stores across many call sites), which may be worth a look separately.
Fix
CUDA.jl (JuliaGPU/CUDA.jl#3243) and oneAPI.jl record only the string pointers on the device and let the host copy the bytes, bounding each read with cuMemGetAddressRange / zeMemGetAddressRange. OpenCL has no portable way for the host to read program-scope constants or to bound a device pointer, so this needs a different host-side mechanism. Options:
- Intern the strings at compile time: in
finish_linked_module!, replace thefunc/file(and exception name) operands of the reporting calls with integer IDs, and keep the ID-to-string table with the compiled kernel (e.g. keyed by a hash of the string, registered in a session-wide table when the kernel is linked, so cached objects keep working). The device stores an integer per field and the host resolves it. Portable across all drivers and mailbox backends. - Record the pointers and resolve them through
clGetDeviceGlobalVariablePointerINTEL(cl_intel_global_variable_pointers) or USM memcpy. Intel-only, and needs the globals to be host-accessible. - Keep copying but limit it, e.g. word-sized copies or fewer frames. Smaller gain and keeps a loop at every throw site.
The first one seems the most robust.
- Dominant language
- Julia
- Stars
- 313
- Forks
- 50
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 41
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from JuliaGPU/OpenCL.jl
-
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 48/100
Maintainers usually reply within 1 day
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
Maintainers usually reply within 1 day
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
Maintainers usually reply within 1 day
-
performance runtime
Difficulty 5/5 Over a week Newbie friendliness 35/100
Maintainers usually reply within 1 day
All issues in JuliaGPU/OpenCL.jl
Similar issues
-
io
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
JuliaGPU/GPUArrays.jl#837 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 88/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
bmad-sim/BeamTracking.jl#382 ·
Maintainers usually reply within 1 day