Eval bug: CUDA kernel stall during model execution, killed by watchdog

Name and Version

version: 0.1.0-dev (build 10438, commit 9d57ce456) built with GNU 16.1.1 for Linux x86_64

Operating systems

Linux

GGML backends

CUDA

Hardware

RTX Pro 6000 Blackwell MAX-Q

Models

unsloth/Qwen3.8-27B-GGUF, UD-Q8_K_XL

Problem description & steps to reproduce

During sustained token generation, llama-server crashes with a CUDA launch timeout. The NVIDIA driver's RC watchdog detects a locked GPU channel (XID 8) and terminates the process. The issue is non-deterministic — it did not occur on every run, and has so far only reproduced on a Blackwell GPU.

First Bad Commit

No response

Relevant log output

llama-server -hf unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL --host 0.0.0.0 --jinja
--batch-size 8192 --parallel 1 -fa on --main-gpu 0 --split-mode none
--spec-type draft-mtp,ngram-mod --spec-draft-n-max 3
--spec-ngram-mod-n-match 40 --spec-ngram-mod-n-min 0 --spec-ngram-mod-n-max 16
--temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0
--reasoning on --reasoning-preserve --cache-type-k q8_0 --cache-type-v q8_0 --fit on --kv_unified --load-mode none

ggml-cuda.cu:106: CUDA error CUDA error: the launch timed out and was terminated current device: 1, in function ggml_backend_cuda_synchronize at ggml-cuda.cu:2530 cudaStreamSynchronize(cuda_ctx->stream())

NVRM: krcWatchdog_IMPL: RC watchdog: GPU is probably locked! Notify Timeout Seconds: 7 NVRM: GPU at PCI:0000:03:00 NVRM: GPU Board Serial Number: 1794125074179 NVRM: Xid (PCI:0000:03:00): 8, pid=45087, name=llama-server, channel 0x0000000d audit: ANOM_ABEND pid=45087 comm="llama-server" sig=6 res=1

#0 __syscall_cancel_arch (libc.so.6) #3 __poll (libc.so.6) #4 n/a (libcuda.so.1 + 0x35d3f7) #5 n/a (libcuda.so.1 + 0x41f0c5) #6 n/a (libcuda.so.1 + 0x341953)