Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Back-ends each compile their own copy of inference and the pipeline

Open
  • Is something wrong?
#1,011 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
12/100
Issue type
Refactor
Clarity
Mostly clear
Activity status
Active
Tech stack
docker, julia
Domain
compilers

Research direction

Start with GPUInterpreter and its method_table_view field, then the CC. methods that dispatch on it, and src/precompile.jl where REPL is imported. Measure precompile time before and after with the workload enabled and disabled. Done means a single native-target interpreter type reused across back-ends with precompile time reduced, and the Enzyme and cuTile hook overrides still work.

Written by the indexing model from the issue text.

Description

latency

Every back-end compiles its own copy of Julia's inference and of GPUCompiler's pipeline during precompilation. None of it is shared with GPUCompiler's precompile workload or with other back-ends. This is most of KernelAbstractions 0.10's precompile time: about 20 s locally against 2.3 s for 0.9, and about 100 s on CUDA.jl's CI. With the workload disabled it is about 3 s. It is presumably also a large part of CUDACore's. Measured on GPUCompiler 2.14.0 and Julia 1.12/1.13.

Back-ends customize GPUCompiler only through data: method_table(s) and the params hooks. CUDA, AMDGPU, Metal, oneAPI, OpenCL.jl and KA's POCL back-end don't override get_interpreter or add Core.Compiler methods. Packages that do customize inference through dispatch, like Enzyme and cuTile, bring their own interpreter through get_interpreter. So nothing requires the compiled code to differ per back-end. It differs only because of the following.

1. The interpreter type depends on the method-table stack

GPUInterpreter{MTV} is parametric in the type of its method-table view, and StackedMethodTable nests once per table. The interpreter type therefore encodes how many tables a back-end stacks:

tables used by
1 GPUCompiler's own workload (the native target leaves out SHARED_METHOD_TABLE)
2 CUDA, AMDGPU
3 Metal, KA/POCL, OpenCL and oneAPI with Float64
4 OpenCL and oneAPI without Float64

Every interpreter type gets its own native code for inference (typeinf_local, abstract_call_*, ...). In a toy back-end, that is about 560 Compiler method instances and about 10 s for the first compile of a job with a new interpreter type. A new job type with the same interpreter type costs about 1 s. GPUCompiler's own workload compiles inference for the depth-1 type, which no GPU back-end uses.

Before #494, GPUInterpreter was not parametric (method_table::CachedMethodTable{OverlayMethodTable}), so all back-ends shared one type. The parameter is only used as the type of the method_table_view field. Every CC. method GPUCompiler defines dispatches on the bare GPUInterpreter.

Proposal: make GPUInterpreter non-parametric again, with method_table_view::CC.MethodTableView, and have the precompile workload compile a job whose view has the same shape as the back-ends'. This costs one dynamic dispatch per findall/findsup, and CachedMethodTable already caches those. The view object itself should not change (no wrapper type): Enzyme inspects CC.method_table(interp) and method_table_view(job) by type, for its cache token and for lookup_world.

2. The pipeline is specialized per job type

Most of the pipeline takes @nospecialize(job), but closures that capture job store its concrete type. Inference then follows the concrete job through the code they call. On 1.10–1.12, the report_failure(job) do ... end in compile (src/driver.jl:86) re-infers compile_unhooked, emit_llvm, irgen, optimize! and mcgen for every job type: about 157 method instances on 1.12. 1.13 no longer specializes that closure. Other places:

  • src/driver.jl: the get!(jobs, dyn_job) do closure, and Dict{CompilerJob,String}(job => ...)
  • src/optim.jl: optimize! runs its whole body in a with_llvm_machine(...) do closure
  • src/mcgen.jl: the cleanup closure in prepare_execution!
  • src/validation.jl: check_ir and check_ir! don't use @nospecialize

Nothing in the pipeline benefits from being specialized on the job. The work happens in LLVM, and the hooks are cheap calls.

3. Back-end hooks invalidate the precompiled pipeline

Most hooks have a single method in GPUCompiler: method_tables, method_table, kernel_state_type, link_libraries!, prepare_job!, get_interpreter, inference_params, optimization_params, julia_ir_passes, can_throw, can_safepoint, ... A call with an abstract job therefore resolves statically, and a back-end defining a method invalidates the caller. In a toy back-end, defining one method_tables method invalidated 73 method instances, including get_interpreter, check_method, irgen, emit_llvm and compile_unhooked. The first compile went from 1.2 s to 7.7 s. Loading KA invalidates method_tables, runtime_module and kernel_state_type this way.

The fix has to keep the hooks overridable by dispatch; Enzyme overrides compile_unhooked and get_interpreter, for example. It also has to keep inference from binding the pipeline to the hooks' current methods. How best to do that needs prototyping.

4. On 1.12, import REPL comes too late

src/precompile.jl imports REPL so that REPL's ::AbstractInterpreter compiler code doesn't invalidate the precompiled pipeline. But precompile.jl is included after jlgen.jl has defined CC.InferenceParams(::GPUInterpreter) and the rest. The workload's code then links to REPL code that is invalidated at load in the next session. Even GPUCompiler's own native-target workload loses its typeinf_local that way.

Moving import REPL to the top of GPUCompiler.jl, before any method definition, brings the first native compile from 5.7 s to 1.5 s on 1.12. 1.13 and 1.14 aren't affected (JuliaLang/julia#61714 is on 1.14). Loading another package with its own interpreter afterwards can still invalidate it on 1.12/1.13 (JuliaLang/julia#61711).

Expected effect

With 1–3, GPUCompiler's own workload would cover inference and the pipeline for every back-end. A back-end's workload would then only compile its own hooks and its kernels' device code. I'd expect that to save about 10 s or more per back-end, for KA's workload and CUDACore's alike. That is an estimate from the toy numbers above, not measured yet.

Dominant language
Julia
Stars
189
Forks
71
Avg merge
1d 12h
Merged PRs (30d)
72

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from JuliaGPU/GPUCompiler.jl

All issues in JuliaGPU/GPUCompiler.jl

Similar issues

More Julia issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.