[Engine] 0.1.52 hip-rx9070 ships no amdgcn/bitcode: HIP cannot build its blit kernels and the engine SEGVs at startup

Start by tracing how the HIP engine component is assembled and how its files are listed in engine-component.json; compare the hip-rx9070 package with the hip-rx7900 report in #3263. Then inspect the bundled HIP/COMGR device-library lookup and the failure path after Could not create BlitManager!. Done means the component has compatible device libraries or reports a clean startup error instead of crashing; validate with the supplied reproduction and logs.

Area

[Engine]

Desktop app, Terminal or source?

Desktop app (Windows, Linux or Mac) — Linux AppImage 0.2.7, engine installed through Update engine.

What happened

Engine 0.1.52-linux-x86_64-hip-rx9070 dies with SIGSEGV a few seconds after it starts, before the model is ready. strata-vision-hip dies at the same instruction a second earlier.

The crash is not in a kernel and not in the driver. It is a null-pointer dereference inside the bundled libamdhip64.so.7, releasing a reference count of an object that was never created, because the HIP runtime could not build its blit kernels:

error: cannot find ROCm device library; provide its path via '--rocm-path'
       or '--rocm-device-lib-path', or pass '-nogpulib' to build without ROCm device library
Couldn't create blit kernels!
Could not create BlitManager!
releaseQueue refCount:0x... (0)

The component ships libamdhip64.so.7, libhsa-runtime64.so.1, libamd_comgr.so.3, libLLVM.so.22.0git, rocblas/ and hipblaslt/, but no amdgcn/bitcode. At HIP init, ROCclr JIT-links its blit kernels against the ROCm device libraries (ockl.bc, ocml.bc, opencl.bc, hip.bc); without them the link fails, BlitManager is never created, and the first strided host→device copy dereferences it.

Expected instead: bundle the device libraries that match the bundled toolchain, or fail with the cannot find ROCm device library message instead of a segmentation fault.

Smallest steps or command that reproduce it

1. Install Strata 0.2.7 (AppImage) and engine 0.1.52-linux-x86_64-hip-rx9070 with Update engine.
2. Start Qwen3.8-Flash-Next IQ3_XXS with pictures on.
3. The window reports Strata engine 0.1.52 is installed; no model is loaded yet, then both engine processes die within ~5 s and coredumpctl list shows the SIGSEGV entries.

Without the app, with the same arguments the app writes into strata-<model>.json:

cd "$HOME/.local/share/strata/components/engine/0.1.52-linux-x86_64-hip-rx9070"

AMD_LOG_LEVEL=3 ./strata --serve \
  --pack "$HOME/Strata/packs/iq3_xxs" \
  --native "$HOME/Strata/models/IQ3_XXS/....-00001-of-00002.gguf" \
  --ple-gguf "$HOME/Strata/models/IQ3_XXS/....-00002-of-00002.gguf" \
  --expert-profile /tmp/.mount_XXXXXXXX/usr/lib/Strata/control/data/expert-profile.bin \
  --expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5 \
  --mtp "$HOME/Strata/mtp/rt" --max-context 131072 --kv int8 --vision \
  --pool-workers 13 --resident-budget-gib 27 --vram-reserve-mib 6533
# exit code 139 (SIGSEGV)

--vision, --mtp, --spec and the context size are not involved: with --max-context 2048, no --vision, no --mtp and no --spec, the log still prints Couldn't create blit kernels! and the process dies the same way.

What matters is which HIP library is loaded. strata has RUNPATH $ORIGIN, and the app's launcher (serve/server.py, child_env()) puts the component directory first in LD_LIBRARY_PATH, so the bundled HIP 7.2.53210 / COMGR 3.0 are always used on the app path. Running the same binary with /opt/rocm/lib first in LD_LIBRARY_PATH loads the system runtime instead and does not crash (workaround below).

Reproduction status

Reproduced locally. 15 coredumps in 12 minutes, all at the same instruction, in two binaries (strata and strata-vision-hip).

Strata version

Strata 0.2.7 · engine 0.1.52 · variant rx9070 · build_id f-amd-vision a42cc3d7df70+rocm · engine-component.json schema 2 · hipBLASLt 100200

Graphics card, memory and system

• GPU: AMD Radeon AI PRO R9700 32 GB (Navi 48, 1002:7551, gfx1201), no display attached
• Second GPU: Radeon 610M iGPU (gfx1036); KFD node 1 = gfx1201, node 2 = gfx1036, so HIP device 0 is the R9700
• CPU: AMD Ryzen 9 9950X3D (16 cores) · RAM: 64 GB
• OS: CachyOS rolling (Arch-based), kernel 7.3.0-5-cachyos, amdgpu 3.64.0, linux-firmware 1:20261019-1
• Userspace graphics: mesa-git 27.0.0_devel.232438.f7dd61f78f8-1, xf86-video-amdgpu 25.0.0
• System ROCm: 10.1.0 (rocm-bin 10.1.0-1) at /opt/rocm; /opt/rocm/amdgcn/bitcode has the 75 .bc files including oclc_isa_version_1201.bc, opencl.bc, ockl.bc, ocml.bc, hip.bc
• Bundled in the component: HIP 7.2.53210.41ce572d55, COMGR 3.0, LLVM 22.0git
• In /opt/rocm/lib: HIP 7.16.26385, COMGR 3.5

No amdgpu reset, ring timeout or VM fault in the kernel log around the crashes.

Model, size and settings

Qwen3.8-Flash-Next IQ3_XXS native pack (2 shards, ~70.6 GiB) + mmproj-…-BF16.gguf (865 MiB) · --max-context 131072 · --kv int8 · pictures on · --spec 4 --spec-min-p 0.5 · --mtp rt · --pool-workers 13 · --resident-budget-gib 27 · --vram-reserve-mib 6533. VRAM free at the time: ~31.8 GiB of 31.86 GiB.

Logs

AMD_LOG_LEVEL=3, last lines before the death (timestamps removed):

:3:hip_context.cpp  :62  :  HIP Version: 7.2.53210.41ce572d55, Direct Dispatch: 1
:3:os_posix.cpp     :934 :  HIP Library Path: ./libamdhip64.so.7
:3:rocdevice.cpp    :1611:  Gfx Major/Minor/Stepping: 12/0/1
:1:devprogram.cpp   :1390:  error: cannot find ROCm device library; ...
error: cannot find ROCm device library; ...
ld.lld: error: undefined hidden symbol: __amd_fillBufferAligned2D
ld.lld: error: undefined hidden symbol: __amd_copyBufferAligned
ld.lld: error: undefined hidden symbol: __amd_copyBufferRect
ld.lld: error: undefined hidden symbol: __amd_fillImage
ld.lld: error: undefined hidden symbol: __amd_copyImageToBuffer
ld.lld: error: undefined hidden symbol: __ockl_dm_init_v1
Error: Creating the executable from LLVM IRs failed.
:1:rocdevice.cpp    :730 :  Couldn't create blit kernels!
:1:rocvirtual.cpp   :1725:  Could not create BlitManager!
:3:rocdevice.cpp    :3078:  releaseQueue refCount:0x7f...000 (0)
Segmentation fault (core dumped)

Coredump, main thread (paths shortened):

Signal: 11 (SEGV)  si_code: SEGV_MAPERR
#0  libamdhip64.so.7 + 0x4b71c9
#1  libamdhip64.so.7 + 0x4a7aba
#2  libamdhip64.so.7 + 0x3b7ca6
#3  libamdhip64.so.7 + 0x4b71db
#4  libamdhip64.so.7 + 0x3b7c69
#5  libamdhip64.so.7 + 0x3b8cd9
#6  main (strata)

The faulting instruction is an atomic refcount decrement on rdi (the this pointer), and rdi is 0:

(gdb) info registers
rdi            0x0
(gdb) x/6i $rip-6
   mov    $0xffffffff,%ebx
=> lock xadd %ebx,0x8(%rdi)      <-- fault at address 0x8
   dec    %ebx

Kernel log, same for both processes:

kernel: strata[<pid>]: segfault at 8 ip ...c9 error 6 in libamdhip64.so.7[4b61c9,...]
kernel: strata-vision-h[<pid>]: segfault at 8 ip ...c9 error 6 in libamdhip64.so.7[4b61c9,...]

strata-vision-hip keeps its symbols, so the path that reaches the broken copy is visible:

#9  ggml_backend_cuda_buffer_set_tensor
#10 clip_model_loader::load_tensors
#11 clip_init
#12 mtmd_context::mtmd_context
#13 mtmd_init_from_file
#14 main

Why the device library is not found

The component directory has no amdgcn/ at all:

$ ls ~/.local/share/strata/components/engine/0.1.52-linux-x86_64-hip-rx9070/
BUILD.json  engine-component.json  hipblaslt/  libamdhip64.so.7  libamd_comgr.so.3
libhsa-runtime64.so.1  libLLVM.so.22.0git  librocblas.so.5  rocblas/  strata  strata-vision  strata-vision-hip  ...

strace -f -e trace=openat,newfstatat,stat,access of the failing run: the bundled COMGR's clang driver reads /opt/rocm/share/hip/version, accepts the directory given by HIP_DEVICE_LIB_PATH (/opt/rocm/lib/llvm/amdgcn/bitcode, openat(..., O_DIRECTORY) = 5), and still fails the link with cannot find ROCm device library. It also probes its own resource directory, which the component does not ship:

access("lib/clang/22/lib/amdgcn/bitcode", F_OK) = -1 ENOENT
stat("/lib/clang/22/lib/amdgcn/amdhsa", ...) = -1 ENOENT
stat("/opt/rocm/lib/llvm/amdgcn/bitcode", {st_mode=S_IFDIR|0755, ...}) = 0
openat(AT_FDCWD, "/opt/rocm/lib/llvm/amdgcn/bitcode", O_RDONLY|O_DIRECTORY) = 5

So the variables are read; the bundled LLVM 22 / COMGR 3.0 does not accept the ROCm 10.1 device libraries, and it has no bitcode of its own to fall back to. These do not change the failure:

ROCM_PATH=/opt/rocm ./strata ...                          -> same error
HIP_DEVICE_LIB_PATH=/opt/rocm/lib/llvm/amdgcn/bitcode     -> same error
ln -s /opt/rocm/core/lib/llvm/amdgcn <component>/amdgcn   -> same error

Workaround (verified)

Run the engine on the system ROCm runtime instead of the bundled one. Tested with the app's own engine arguments, pictures on, 8192 context.

1. Put /opt/rocm/lib before the component directory:

   LD_LIBRARY_PATH=/opt/rocm/lib:/opt/rocm/lib64:/opt/rocm/llvm/lib \
     ~/.local/share/strata/components/engine/0.1.52-linux-x86_64-hip-rx9070/strata --serve ...
   

2. Or keep the launcher's LD_LIBRARY_PATH and preload the system runtime:

   LD_PRELOAD="/opt/rocm/lib/libamdhip64.so.7 \
               /opt/rocm/lib/libamd_comgr.so.3 \
               /opt/rocm/lib/libhsa-runtime64.so.1" ./strata --serve ...
   

For the desktop app, variant 2 needs no change inside the component: the engine config ~/.local/share/strata/install/strata-<model>.json carries an env object that the launcher applies to both the engine and strata-vision-hip:

"env": {
  "LD_PRELOAD": "/opt/rocm/lib/libamdhip64.so.7 /opt/rocm/lib/libamd_comgr.so.3 /opt/rocm/lib/libhsa-runtime64.so.1"
}

Result with that in place, starting the model from the app (or serve/server.py with the same config):

:3:comgrctx.cpp  :88  :  Loaded COMGR library version 3.5.
:3:hip_context.cpp :42 :  HIP Version: 7.16.26385, Direct Dispatch: 1
strata generate: PCIe probe: 28.7 GB/s host->device (best of 28.7 28.7 28.7 28.7) -> pcie_frac 0.15
[strata] filling the GPU's expert cache (15326 experts, 24.86 GiB of VRAM) ...
[strata] note: 560 MiB of VRAM free with the model loaded.
ready: http://127.0.0.1:8080/v1  (OpenAI: /v1/chat/completions, Anthropic: /v1/messages, context 8192 tokens, images on)

No Couldn't create blit kernels, no Could not create BlitManager, no SIGSEGV, no new coredump, and the GPU picture reader starts instead of falling back to the CPU.

Notes:

• The bundled librocblas.so.5, libhipblas.so.3 and libhipblaslt.so.1 are still loaded from the component and work against the system HIP; only libamdhip64, libamd_comgr and libhsa-runtime64 come from /opt/rocm.
• It requires a system ROCm with matching device libraries. On a machine without ROCm the crash is unavoidable, because the package ships neither the runtime's bitcode nor a fallback.
• It is a runtime substitution, not a fix, and it mixes HIP 7.16 with the 7.2 user-space libraries of the component.

Suggested fix

1. Ship amdgcn/bitcode from the ROCm toolchain the engine was built with (build_id …+rocm) inside the component package and list it in engine-component.json so updates keep it. The hip-rx7900 report in #3263 shows the packaging already varies between builds, so this looks like the same class of gap.
2. When the bundled directory is absent, fall back to HIP_DEVICE_LIB_PATH and then ROCM_PATH, and make that fallback actually work with the bundled COMGR, which today reads those paths and still reports cannot find ROCm device library.
3. Do not continue after Could not create BlitManager!: report the cannot find ROCm device library line as a normal engine error and exit, instead of dereferencing the missing object later on the first strided copy.

Validation

• Passed: SIGSEGV reproduced with the engine alone (AMD_LOG_LEVEL=3, exit 139); the JIT failure that precedes it captured; same failure without --vision; ROCM_PATH, HIP_DEVICE_LIB_PATH and the amdgcn symlink do not help; both workarounds remove the JIT failure, load the weights and reach READY / ready: http://127.0.0.1:8080/v1, including through the app's own launcher with pictures on.
• Not run: a full 128K-context load; a request through the API after the workaround; a machine without ROCm installed (no second machine available).