Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

DeepSeek-V4.1-Flash (PR #30817): single-token output corruption when a cached prompt prefix is reused (Engram n-gram history)

Open
  • Is something wrong?
#30,944 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
38/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
cpp
Domain
backend, machine-learning

Research direction

Start with the Engram history path described in the issue, especially engram_hash and get_prev_tokens, then trace how server prompt-prefix reuse restores cached KV state. Compare the token history used for Engram row lookups after a reused prefix with a fresh prefill, using the linked reproduction harness and its cache_prompt: false control. Done means reused-prefix requests no longer produce the reported single-token substitutions.

Written by the indexing model from the issue text.

Description

This is against the deepseek41 branch of #30817 (draft), not master — filing as an Issue so it doesn't clutter the PR thread.

Trigger isolated: KV prompt-prefix reuse corrupts output with the Engram (both --lazy-mode settings)

Name and Version

Build of the deepseek41 branch @ b72050d (the branch behind PR #30817); this commit is also the first bad commit.

Operating systems

Linux

GGML backends

CUDA

Hardware

2× RTX PRO 6000 Blackwell (96 GB) + 251 GB RAM, WRX90E-SAGE-SE

Models

unsloth/DeepSeek-V4.1-Flash-1017-GGUF, UD-IQ3_S (8 shards)

Setup
llama-server \
  -m DeepSeek-V4.1-Flash-1017-UD-IQ3_S-00001-of-00008.gguf \
  --alias ds41f \
  -ngl 99 \
  -ncmoe 16 \
  -ts 27,13 \
  -c 131072 -fa on -ub 2048 -b 4096 --parallel 1 \
  -t 32 \
  --load-mode mmap --lazy-mode on \
  --override-tensor "per_layer_token_embd.weight=CPU" \
  --jinja \
  --reasoning on --reasoning-format deepseek \
  --chat-template-kwargs '{"reasoning_effort":75}' \
  --temp 1.0 --top-p 1.0 --top-k 0 --min-p 0.0

CUDA 13.2, sm_120. The --lazy-mode off variant was run on the same build with the Engram tensor resident (no CPU override).

We ran a controlled discriminant with a literal-copy harness (20 hex + 20 prose items, temp 0, forced tool-call), each pass being the full harness against the same server process:

server mode pass 1 (fresh prefixes) pass 2 (reused prefixes) pass 3 ("cache_prompt": false)
--lazy-mode on 1/20 hex, 0/20 prose 6/20 hex, 2/20 prose 0/20 hex, 0/20 prose
--lazy-mode off 0/20 hex, 0/20 prose 8/20 hex, 0/20 prose —

Pass 1 already reuses the shared system-prompt prefix across items, which likely explains the 1/20 in lazy-on pass 1.

Corruption in pass 2 accumulates progressively in our runs (first items clean, grows monotonically) — consistent with reused KV state.

Mechanism (appears to be): engram_hash builds the n-gram row-ids from the token history reconstructed via get_prev_tokens. When the server reuses a cached prompt prefix (slot selected by LCP similarity, f_keep), the n-gram history of the first tokens after the reused prefix diverges from what the original forward pass saw, so the Engram row lookups land on wrong rows and single-token substitutions appear in the output, growing with the reused-prefix length.

Reproduction: run the harness once against a fresh server (clean), run it again immediately (hex corruption jumps to 6-8/20), run a third time with "cache_prompt": false on every request (clean again).

Harness: https://gist.github.com/bertusmt/cd1bba69bc92836e338b2fb5e35291ad (iso_2x20.py; defaults to localhost:8000 — pass your port as the first argument).

Workaround: send "cache_prompt": false on requests that must be bit-exact (cost: full re-prefill).

In a separate 500-item tool-calling battery with CJK content on the same model we observed the same class of single-token substitutions when prefixes were reused; the same mitigation applied.

Reference: PR #30817.

Relevant log output

Slot selection on every harness call (f_sim_best/f_keep from the server log):

oct 20 23:47:53 llama-ds41dbg[3977692]: slot get_availabl: id  0 | task -1 | selected slot by LCP similarity, f_sim_best = 0.910 (> 0.100 thold), f_keep = 0.398
oct 20 23:48:16 llama-ds41dbg[3977692]: slot get_availabl: id  0 | task -1 | selected slot by LCP similarity, f_sim_best = 0.910 (> 0.100 thold), f_keep = 0.281
oct 20 23:48:24 llama-ds41dbg[3977692]: slot get_availabl: id  0 | task -1 | selected slot by LCP similarity, f_sim_best = 0.899 (> 0.100 thold), f_keep = 0.573
oct 20 23:48:47 llama-ds41dbg[3977692]: slot get_availabl: id  0 | task -1 | selected slot by LCP similarity, f_sim_best = 0.899 (> 0.100 thold), f_keep = 0.280

(376 slot-selection lines of this kind across the two-pass run.)

Example of observed output diffs on hex items (expected vs produced, pass 2, from iso2x20_hex_raw.json):

iso/hex02.bin  char 0:   expected '1', produced '9'
iso/hex05.bin  char 47:  expected '7', produced ' '
iso/hex07.bin  char 0:   expected '6', produced 'x'
iso/hex11.bin  char 150: expected '3', produced ' '
iso/hex17.bin  char 0:   expected '9', produced 'x'
iso/hex12.bin  char 171: expected '9', produced ' '
iso/hex15.bin  char 0:   expected 'a', produced 'c'
iso/hex19.bin  char 194: expected '8', produced ' '

(8/20 corrupted hex items in this pass; prose items in the same pass: 0/20 corrupted.)

Dominant language
C++
Stars
132k
Forks
24.6k
Avg merge
2d 1h
Merged PRs (30d)
500

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from ggml-org/llama.cpp

All issues in ggml-org/llama.cpp

Similar issues

More C++ issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.