Llama decode wrong for SmolLM2-135M: early <|im_end|>, long-prompt degeneration, '1 2 3…9' → ' 1 1 1 1' — positions/RoPE suspected, tokenizer ruled out

Start by tracing OptimizedLLMRuntime.generateUntilStop into the llama decode path and inspect how token positions reach RoPE and the KV cache; the issue points to both as possible causes but does not identify a file or test. Run the supplied counting continuation and ChatML reproductions, then compare greedy output with llama.cpp. Done means all three decode symptoms are fixed and the SKaiNET-cartridge-blueprints goldens are regenerated.

Summary

Greedy decode of SmolLM2-135M-Instruct through OptimizedLLMRuntime + generateUntilStop produces wrong tokens. Three symptoms, one fingerprint:

1. Early EOS. ChatML prompt "What is the capital of France?" (37 prompt tokens) yields The capital of France (4 tokens, then <|im_end|>). The llama.cpp reference on the F16 GGUF of the same checkpoint yields The capital of France is Paris.
2. Length-dependent degeneration. A ~200-token ChatML prompt (RAG agent transcript) collapses into single-token lines: You\nYou\nYou\n… under greedy.
3. Counting failure. Raw continuation (no chat template) of "1 2 3 4 5 6 7 8 9" produces " 1 1 1 1". Short idiomatic continuations survive: "Paris is the capital of" → " France.", "The quick brown fox jumps over the lazy" → " dog." — then drift within a few tokens.

A model that completes idioms but cannot continue a counting sequence is seeing a bag of tokens: this points at attention positions (RoPE application or KV-cache position bookkeeping) in the llama decode path, not at sampling and not at the tokenizer.

Found while wiring the generator-smollm2-skainet cartridge into SKaiNET-rag's Generator seam; downstream the output gate rejects everything the model produces, so the host degrades to quoting passages.

Environment

• skainet-transformers 0.57.1, skainet BOM 0.57.0, OptimizedLLMMode.DIRECT, FP32, DirectCpuExecutionContext
• Cartridge: generator-smollm2-skainet 0.1.0-SNAPSHOT (SKaiNET-cartridge-blueprints @ e055789)
• Checkpoint: HuggingFaceTB/SmolLM2-135M-Instruct @ 12fd25f77366fa6b3b4b768ec3050bf629380bac (safetensors, bf16 → f32 on load), ropeFreqBase pinned to 100 000 by the cartridge and in the repro below
• macOS arm64 (Darwin 27), JVM 21 with jdk.incubator.vector
• Reference: llama.cpp b10621, bartowski/SmolLM2-135M-Instruct-GGUF F16, --temp 0

Ruled out

• Tokenizer (relevant to the #471 parity work): TokenizerFactory.fromTokenizerJsonString on this checkpoint encodes the ChatML specials correctly — <|im_start|> → [1], <|im_end|> → [2], full template round-trips to the expected id sequence ([1, 4093, 198, 26843, 2, 198, 1, 520, 9531, 198] for <|im_start|>user\nHi<|im_end|>\n<|im_start|>assistant\n), and decode(1)/decode(2) return the special strings. The prompt ids entering the model are right, so the tokenizer refactor alone will not close this issue — but the repro below is a ready-made end-to-end check for both.
• Double BOS: generateUntilStop does not prepend bos; the prompt goes in as encoded.
• Prefill strategy: PrefillStrategy.Autoregressive and PrefillStrategy.Batched() produce token-identical output ([504, 3575, 282, 4649]), so the divergence is not batched-prefill-specific — both paths share the wrong result.

Reproduction

With the blueprint snapshot fetched (~/.cache/skainet/models/HuggingFaceTB_SmolLM2-135M-Instruct):

val configJson = File("$dir/config.json").readText()
val metadata = LlamaConfigParser.parse(configJson).copy(ropeFreqBase = 100_000f)
val ctx = DirectCpuExecutionContext()
val model = LlamaNetworkLoader.fromSafeTensors(
    metadata = metadata,
    randomAccessProvider = { openRandomAccessSource("$dir/model.safetensors")!! },
    tiedEmbeddings = LlamaConfigParser.isTiedEmbeddings(configJson),
).load<FP32, Float>(ctx)
val runtime = OptimizedLLMRuntime(model, ctx, OptimizedLLMMode.DIRECT, FP32::class, bos = metadata.bosTokenId)
val tokenizer = TokenizerFactory.fromTokenizerJsonString(
    File("$dir/tokenizer.json").readText(),
    File("$dir/tokenizer_config.json").readText(),
)

// Case A: ChatML greedy — expect "The capital of France is Paris.", get early <|im_end|> after 4 tokens
val chat = tokenizer.encode(
    "<|im_start|>system\nYou are a helpful AI assistant named SmolLM, trained by Hugging Face<|im_end|>\n" +
    "<|im_start|>user\nWhat is the capital of France?<|im_end|>\n<|im_start|>assistant\n"
)
runtime.generateUntilStop(chat, maxTokens = 24, eosTokenIds = setOf(2), temperature = 0f)

// Case B: positions canary — expect " 10 11 12 …", get " 1 1 1 1"
runtime.reset()
runtime.generateUntilStop(tokenizer.encode("1 2 3 4 5 6 7 8 9"), maxTokens = 8, eosTokenIds = setOf(2), temperature = 0f)

Reference, same machine:

llama-cli -m SmolLM2-135M-Instruct-f16.gguf -p "What is the capital of France?" --temp 0 -n 24 -st
# → The capital of France is Paris.

Side effect to fix together with this

SmolLm2GoldenTest in SKaiNET-cartridge-blueprints pins the broken transcript as the cartridge's behavioral identity (GOLDEN_TEXT = "The capital of France", GOLDEN_TOKENS = [504, 3575, 282, 4649]). The goldens prove cross-target determinism, not correctness against the checkpoint's reference — once the decode is fixed they must be regenerated (and the cartridge republished), not treated as a regression.

Acceptance

• Case A greedy transcript is The capital of France is Paris. (token-identical to llama.cpp F16 greedy / HF transformers greedy)
• Case B continues the counting sequence (10 11 12 …)
• A ~200-token ChatML prompt decodes coherent text instead of single-token lines
• Blueprints goldens regenerated from the fixed path