DeepSeek V4.1 Flash gb202 TP4: next-token logits intermittently corrupt mid-reply (stray "##", "-", "<ds_safety>" classifier text in answers)

Start by reading the Flare server's Chat Completions decoding path and the code handling speculative decoding, especially around --mtp-draft-tokens; the issue does not name source files or tests. Compare a target-only run with speculative decoding enabled and inspect logprobs at the failing token to locate where the distribution diverges. Done means repeated identical requests produce stable, clean answers without leaked classifier or markup tokens.

Summary

Serving local-inference-lab/DeepSeek-V4.1-Flash-1017-FLARE (gb202, TP4), short replies intermittently contain stray tokens. Sometimes it's a DeepSeek safety-classifier block (<ds_safety>[用户未成年]否\n[分类]其他\n[判定]…) written into the visible answer. Logprobs show this isn't a sampling or detokenizing glitch: at the failing position the whole next-token distribution is wrong. After "It's great to" a clean run gives meet (logprob −0.07). A bad run gives a near-flat spread over markup tokens (\n −2.5, < −2.77, # −2.88, ## −3.22).

It reproduces on release-6fa01a4-main-20261101 and release-18b16e1-main-20261104. How often it happens varies a lot between loads of the same image and package.

Environment

• Package: local-inference-lab/DeepSeek-V4.1-Flash-1017-FLARE, revision gb202 (cefb84d2b80f7cdb16762a666c507b048d7a1336), full C32 runtime.
• Images:
  • ghcr.io/local-inference-lab/flare:release-18b16e1-main-20261104@sha256:e2f83d5a06383bf8bde5e2cf4f69c13f077b1319f2b78b1786efcad2fbb1a21a
  • ghcr.io/local-inference-lab/flare:release-6fa01a4-main-20261101@sha256:895b3aebf585dcfd1bc4df1aced74d4034875009bab9c3f95d7f5aca353bff86
• 4× RTX PRO 6000 Blackwell Workstation Edition 96 GB (SM120), x86-64, 125 GiB host RAM. The host also has an RTX 3090, which is not visible to the container.
• Disclosure: NVIDIA driver 615.71.09 with the community P2P patch (aikitoria fork), GPUs power-capped to 425 W. Not a stock driver; mentioning it in case peer access matters here.
• Launch (flags as served):

docker run --rm --runtime=nvidia --network host --init --shm-size=32g \
  --security-opt seccomp=unconfined --ulimit memlock=-1:-1 \
  -e NVIDIA_VISIBLE_DEVICES=<4 Blackwell UUIDs> \
  -v .../DeepSeek-V4.1-Flash-1017-FLARE:/models:ro -v .../flare-home:/home/flare \
  ghcr.io/local-inference-lab/flare:release-18b16e1-main-20261104@sha256:e2f83d5a... \
  run /models/model.safetensors --publisher-keys /models/publisher.json \
  --gpus 0,1,2,3 --host 127.0.0.1 --port 10004 --served-model-name deepseek4.1-763b-flare \
  --max-concurrency 32 --mtp-draft-tokens 7 --scheduler-policy balanced \
  --kv-cache-limit memory --embedding-table-storage disk --embedding-table-cache-size 8G \
  --trust-store /home/flare/.config/flare/trust.toml ...

Engram is on disk (125 GiB host RAM is below the card's RAM-mode requirement). Package loads cleanly on both images; startup reports 1048576 tokens/request.

Reproduction

All requests were sequential (concurrency 1), sent directly to the Flare server's Chat Completions endpoint, at temperature: 0.

curl -s http://127.0.0.1:10004/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model":"deepseek4.1-763b-flare",
  "messages":[{"role":"user","content":"Say hello to me."}],
  "max_tokens":120,"temperature":0,"logprobs":true,"top_logprobs":5}'

Eight identical requests on the current 18b16e1 load:

Count	Reply
3	Hello! 👋 It's great to meet you. What can I help you with today?
2	Hello! 👋 Great to meet you — how can I help you today?
2	Hello! 👋 It's great to## to meet you. How can I help you today?
1	Hello! 👋 It's great to<ds_safety>[用户未成年]否\n[分类]其他\n[判定]用户仅要求简单…

The leaked block reads, in English, "[user is a minor] no / [category] other / [verdict] the user only asked for a simple greeting; the reply is a friendly greeting with no political or sexual content". It looks like an internal safety-classifier output format, written straight into content.

Logprobs at the failing position

Bad run:

' great'   -0.400  top: [' great' -0.4, ' nice' -1.11, ' good' -7.66, ' lovely' -8.97]
' to'       0.000  top: [' to' 0.0, ' meeting' -20.19, ' that' -20.36, ' for' -20.72]
'##'       -3.217  top: ['\n' -2.5, '<' -2.77, '#' -2.88, '##' -3.22]      <-- flat, markup-only
' to'      -0.850  top: [' to' -0.85, ' Heading' -1.55, 'to' -2.34, ' Step' -2.77]
' meet'    -0.068

Clean run, same visible prefix:

' great'   -0.301  top: [' great' -0.3, ' nice' -1.35, ...]
' to'       0.000  top: [' to' 0.0, ' that' -20.27, ...]
' meet'    -0.088  top: [' meet' -0.09, ' see' -2.49, ' hear' -7.08, ' have' -7.64]

meet doesn't appear in the bad run's top 5 at all. The next position's top candidates are Heading / Step, as if the model's state briefly turned into "markdown document". It then recovers. The reasoning before the answer is short (10 tokens) in both runs, but I can't rule out that it differed between them.

It varies between loads

• 18b16e1, first load: "Say hi." (max_tokens: 64) skipped reasoning (reasoning_tokens: 0) and was garbled in all 7 requests (buffered, streamed, and native /v1/messages). Examples:
  • "Hi! 😊 How can-chan here! What can I<dy> I help you with today?"
  • "Hi! 😊 How can- I help you today?" ×3
  • "Hi! 😊 How can-chan I help you today?"
• 6fa01a4, fresh load (same package, same flags): "Say hi." gave "Hi! 👋 How can// can I help you today?" ×3 and "Hi there! 👋 What can Ig help you with today?".
• 18b16e1, second load (current): "Say hi." reasons first and is clean in 92 of 92 runs across max_tokens 32–200, temperature 0/1, with and without logprobs. The "Say hello to me." failure above is from this load.

In every failure I saw, the stray token comes right after an emoji and a few words, inside a short answer.

Expected

Stable answers without control or markup tokens. <ds_safety>, <dy> and the like should never reach content. A request that's identical at temperature: 0 shouldn't produce a corrupted next-token distribution on some runs and not others.

Side observation, possibly intended

temperature: 0 isn't greedy on this package. With logprobs, the chosen token is regularly not the top-1:

• first token We (−1.10) chosen over </think> (−0.62)
• What (−1.08) chosen over How (−0.42)

Identical requests give different text. If #167's DeepSeek V4.1 sampling ranges clamp temperature up from 0, that explains it, but it also makes this bug harder to pin down. A way to force deterministic greedy decoding for debugging would help.

Not tested yet

• --mtp-draft-tokens 0 (target-only). I can run it if that would help separate speculative verification from the target forward pass.
• --embedding-table-storage ram (not enough host RAM here).
• A stock (non-P2P-patched) driver.

Happy to collect more data: full logprob dumps, startup logs, or a run with any debug flags you'd like.