Cache the memory back-end per device instead of per task

Open
  • Is something wrong?
#524 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
55/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
julia
Domain
backend-api-design, performance, testing-qa

Research direction

Start with memory_backend(), unified_memory_backend(), default_memory_backend(dev), and device_contexts in lib/cl/state.jl; compare their task-local caching with the global per-device context pattern. Verify that the result is shared per device and unified flag, then add the concurrent kernel-launch coverage described for cl.Kernel, using test/pocl_session.jl as the reference. Done means fresh tasks avoid repeated backend queries and the threaded launch test passes.

Written by the indexing model from the issue text.

Description

performance runtime

cl.memory_backend() and cl.unified_memory_backend() are cached in task-local storage, so every new task works out its memory back-end again. default_memory_backend(dev) queries the device's USM, SVM and BDA support and reads the default_memory_backend preference. On PoCL that takes about 54 µs per task. It's the main cost of the first allocation in a fresh task:

using OpenCL, pocl_jll, Statistics
cl.platform!("pocl")
CLArray{Float32}(undef, 16)
t(f) = (t0 = time_ns(); f(); (time_ns() - t0) / 1e3)
same = [t(() -> CLArray{Float32}(undef, 16)) for _ in 1:200]
newt = [fetch(Threads.@spawn (cl.platform!("pocl"); cl.context(); cl.queue();
                              t(() -> CLArray{Float32}(undef, 16)))) for _ in 1:200]
median(same), median(newt)   # (11.4, 69.7) µs, OpenCL.jl main (937fc5e), PoCL 7.2, 2 threads

This adds up for code that allocates from many short-lived tasks (Threads.@threads loops, @spawn-per-chunk patterns, Dagger). The result only depends on the device, the unified flag and a load-time preference, so it can be cached once per (device, unified) in a global dict under a lock, like device_contexts in lib/cl/state.jl. The task-local entries can stay as the fast path, or be dropped. The maxlog=1 fallback warning in memory_backend() then also needs no task-local state.

Everything else per task is cheap: cl.context() and cl.queue() cost well under a microsecond in a fresh task, thanks to the global per-device contexts.

Also, while comparing with KernelAbstractions' PoCL back-end (which is moving to the same shared-context design): cl.Kernel's lock (75a01d4, "Serialize shared kernel launches") has no test. A test that launches one kernel from many threads with different arguments catches the race reliably. Without the equivalent lock, KA's version of that test segfaults inside PoCL within a few hundred launches on 4 threads. See test/pocl_session.jl on KernelAbstractions.jl's tb/pocl-shared-context branch.

Dominant language
Julia
Stars
313
Forks
50
Avg merge
1d 8h
Merged PRs (30d)
44

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from JuliaGPU/OpenCL.jl

All issues in JuliaGPU/OpenCL.jl

Similar issues

More Julia issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.