server: prompt cache is populated by rerank tasks that can never read it back (cache_idle_slots missing the SERVER_TASK_TYPE_COMPLETION guard)

Start in server-context.cpp at the idle-slot save loop around lines 2398-2414 and compare it with the completion guard around lines 1617-1618. Ensure idle prompt caching is limited to completion tasks, then run the rerank reproduction with distinct documents and confirm memory no longer grows from prompt-cache entries.

Summary

On a rerank-only server, host RAM grows by ~23 MB per /v1/rerank call until it reaches the --cache-ram limit (8192 MiB by default), and is never released while the process lives. The growth is the prompt cache being populated by rerank tasks that can never read it back - for SERVER_TASK_TYPE_RERANK the cache is write-only.

This is not the same as #25740 (closed as not-planned). There the cache was doing its job on the completion path and the reporter simply hadn't set -cram 0. Here the entries are provably never read, so the memory is pure waste plus a per-request serialisation cost.

Version / environment

• b4d6c7d8ff69c2e05e4e8ee7e6e710a08abd7b45 (b10091), also present on master at time of writing
• CUDA, aarch64 (NVIDIA GB10), driver 580.173.02
• Model: Qwen3-Reranker-0.6B Q8_0 GGUF
• Server: llama-server --reranking --pooling rank -c 8192 -np 2 -ngl 999 -fa on --no-mmap
• Reproduced identically on two hosts

The path

1. POST /v1/rerank creates independent SERVER_TASK_TYPE_RERANK tasks with no parent.
2. server_slot::release() clears the prompt only for child tasks (server-context.cpp:488-491):

   // do not keep context of the child slots - the parent's context is enough
   if (task->is_child()) {
       prompt_clear();
   }
   

   Rerank tasks are not children, so slot.prompt and its KV state survive release.
3. On the next task launch, the idle-slot pass saves every idle slot to the prompt cache (server-context.cpp:2398-2414) - with no task-type guard:

   if (params_base.cache_idle_slots) {
       for (auto & slot : slots) {
           if (!slot.is_processing()) {
               if (slot.prompt_save(*prompt_cache)) {
   

4. The other call site does have the guard it needs (server-context.cpp:1617-1618):

   // cache prompts only for completion tasks
   update_cache = update_cache && task.type == SERVER_TASK_TYPE_COMPLETION;
   

5. prompt_load is only reached from inside that completion-gated branch, so nothing ever reads a rerank entry back.

With -np 2, kv_unified is false, so the [TAG_IDLE_SLOT_CLEAR] path does not run and the slot keeps its prompt after the save as well.

Measurements

Fresh restart each arm, 40 calls, resident memory of the service:

workload	growth
40 calls, identical query+docs	+68 MB, then flat from call ~10
40 calls, 20 docs, 40 distinct doc lengths	+3032 MB by call 20, climbing until capped
same, with --cache-ram 0	+626 MB
same, --cache-ram 0 + GGML_CUDA_DISABLE_GRAPHS=1	+620 MB (graph cache not involved)
same 40 lengths descending	+620 MB immediately, then flat

Two separate effects: the cache (dominant, content-keyed, deduped only on full prefix match, so every distinct query+document pair is a new entry) and a benign high-water-mark buffer allocation of ~620 MB that stops growing once the largest request has been seen.

The arithmetic matches: Qwen3-0.6B is 28 layers x 8 KV heads x 128 head_dim, f16 → 112 KiB/token, so ~23 MB/call ≈ 210 tokens of KV state - a query plus a few short documents. In production the process sat at 9137 MB with zero busy slots, consistent with ~1.1 GB baseline plus the 8 GiB cap.

Reproduce

llama-server -m Qwen3-Reranker-0.6B-Q8_0.gguf --reranking --pooling rank -c 8192 -np 2
# then POST /v1/rerank repeatedly with DIFFERENT documents each time and watch RSS.
# Identical documents will not reproduce it - the cache dedups on full prefix match.

Suggested fix

Apply the same guard the completion path already uses, so idle slots are only saved for task types that can read the cache back:

if (params_base.cache_idle_slots) {
    for (auto & slot : slots) {
        if (!slot.is_processing() && slot.task && slot.task->type == SERVER_TASK_TYPE_COMPLETION) {

Alternatively (or additionally) clear the prompt on release for task types that never reuse it.

Workaround

--cache-ram 0 (which also auto-disables cache_idle_slots), or --no-cache-idle-slots. For a rerank/embedding-only server this costs nothing, since those entries are never read. Confirmed: 9137 MB → ~1.1 GB steady state on our fleet.

Note on scope

/v1/embeddings builds non-child SERVER_TASK_TYPE_EMBEDDING tasks through the same send_embedding → release() path, so it looks structurally exposed. I could not reproduce growth on an embedding-only server (Qwen3-Embedding-0.6B, 30 distinct input lengths, +20 MB and flat), so I am reporting only what I measured rather than assuming.