Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

TextLLMRunner degenerates on an LFM2.5 export that steps correctly token by token

Aperta
  • Qualcosa non va?
#21,858 0 commenti 0 reazioni 1 assegnatario Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@JakeStevens ci sta già lavorando.

Dal 15/8/2026.

Valutazione

Questa issue non è ancora stata valutata.

Descrizione

module: llm

TextLLMRunner produces degenerate output for an LFM2.5 export that generates correctly when the same .pte is stepped token by token. Reproduces on macOS with the pip wheel and on Android through LlmModule, and does not depend on the export configuration.

Environment: executorch 1.4.0 (pip), torch 2.13.0, macOS arm64; also reproduced on a Pixel 8a with an AAR built from main.

What happens

Same .pte, same prompt, same machine, two ways of driving it:

token-by-token, explicit input_pos   "Here are three colors:\n\n1. **Red**\n2. **Green**"
TextLLMRunner                        "s::::<|im_end|>"

Shorter prompts make the shape clearer — the runner repeats one token:

prompt "Hi"                    -> "HiHiHiHiHiHiHiHiHiHiHiHiHiHiHiHi"
prompt "Name three colours."   -> "<|im_end|><|im_end|>"

Repro

from executorch.extension.llm.custom_ops import custom_ops  # noqa
from executorch.kernels import quantized  # noqa
from executorch.extension.llm.runner import TextLLMRunner, GenerationConfig

r = TextLLMRunner("lfm2_5_350m_xnnpack_8da4w.pte", "tokenizer.json")
out = []
r.generate("Hi", GenerationConfig(max_new_tokens=16, echo=False), lambda t: out.append(t))
print("".join(out))     # HiHiHiHiHiHiHiHiHiHiHiHiHiHiHiHi

The same file through a loop that steps one token at a time answers normally:

import torch
from executorch.extension.pybindings.portable_lib import _load_for_executorch

model = _load_for_executorch("lfm2_5_350m_xnnpack_8da4w.pte")
for i, tok in enumerate(prompt_tokens):
    logits = model.forward((torch.tensor([[tok]]), torch.tensor([i])))[0]
# then greedy-decode from logits, advancing input_pos each step

The .pte is public: https://huggingface.co/mlboydaisuke/LFM2.5-350M-ExecuTorch — exported with
executorch.extension.llm.export.export_llm, qmode: 8da4w, embedding_quantize: "8,0",
XNNPACK.

What it is not

I re-exported the model to remove each candidate in turn and drove every build both ways:

build forward input enable_dynamic_shape token-by-token TextLLMRunner
shipped (use_sdpa_with_kv_cache: True) [1, 2047] True correct degenerate
control (use_sdpa_with_kv_cache: False, enable_dynamic_shape: False) [1, 1] False correct degenerate

So it is not the custom SDPA path, and not dynamic versus static shape — a static [1, 1] export,
which the runner should drive sequentially, fails the same way.

It is not the tokenizer either: the same tokenizer.json encodes the prompt to the expected ids
through a standalone HFTokenizer harness on the same machine, special tokens included, and the
.pte reports get_bos_id = 1, get_eos_ids = [7], matching the tokenizer.

And it is not the runner in general — Qwen3.5-0.8B exported the same way (8da4w,
embedding_quantize: "8,0", XNNPACK) generates correctly through TextLLMRunner on the same
build, so whatever this is interacts with something specific to LFM2.5.

A guess I could not confirm

LFM2.5 is a hybrid: short-convolution blocks interleaved with attention. Convolution blocks carry
state along the sequence, so anything that drives positions differently from a plain
one-token-at-a-time walk — a warmup pass, a re-prefill, a batched first step — would corrupt that
state while leaving a pure-attention model like Qwen unharmed. I could not confirm which of those
the runner does, so I am reporting the isolation rather than a diagnosis.

Happy to run further experiments against this model, or to test a patch.

This report was written with Claude.

cc @larryliu0820 @mergennachin @cccclai @helunwencser @jackzhxng @digantdesai

Lingua principale
Python
Stelle
5k
Fork
1.2k
Merge medio
2g 5h
PR unite (30g)
491

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di pytorch/executorch

Tutte le issue di pytorch/executorch

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.