Hacktoberfest 2026: die Issues, die Maintainer für den Oktober markiert haben – offen und einsteigerfreundlich. Hacktoberfest-Issues durchsuchen

TextLLMRunner degenerates on an LFM2.5 export that steps correctly token by token

Offen
  • Stimmt etwas nicht?
#21,858 0 Kommentare 0 Reaktionen 1 zugewiesene Person Auf GitHub ansehen

Maintainer antworten meist innerhalb von 1 Tag

@JakeStevens arbeitet bereits daran.

Seit 15.8.2026.

Bewertung

Dieses Issue wurde noch nicht bewertet.

Beschreibung

module: llm

TextLLMRunner produces degenerate output for an LFM2.5 export that generates correctly when the same .pte is stepped token by token. Reproduces on macOS with the pip wheel and on Android through LlmModule, and does not depend on the export configuration.

Environment: executorch 1.4.0 (pip), torch 2.13.0, macOS arm64; also reproduced on a Pixel 8a with an AAR built from main.

What happens

Same .pte, same prompt, same machine, two ways of driving it:

token-by-token, explicit input_pos   "Here are three colors:\n\n1. **Red**\n2. **Green**"
TextLLMRunner                        "s::::<|im_end|>"

Shorter prompts make the shape clearer — the runner repeats one token:

prompt "Hi"                    -> "HiHiHiHiHiHiHiHiHiHiHiHiHiHiHiHi"
prompt "Name three colours."   -> "<|im_end|><|im_end|>"

Repro

from executorch.extension.llm.custom_ops import custom_ops  # noqa
from executorch.kernels import quantized  # noqa
from executorch.extension.llm.runner import TextLLMRunner, GenerationConfig

r = TextLLMRunner("lfm2_5_350m_xnnpack_8da4w.pte", "tokenizer.json")
out = []
r.generate("Hi", GenerationConfig(max_new_tokens=16, echo=False), lambda t: out.append(t))
print("".join(out))     # HiHiHiHiHiHiHiHiHiHiHiHiHiHiHiHi

The same file through a loop that steps one token at a time answers normally:

import torch
from executorch.extension.pybindings.portable_lib import _load_for_executorch

model = _load_for_executorch("lfm2_5_350m_xnnpack_8da4w.pte")
for i, tok in enumerate(prompt_tokens):
    logits = model.forward((torch.tensor([[tok]]), torch.tensor([i])))[0]
# then greedy-decode from logits, advancing input_pos each step

The .pte is public: https://huggingface.co/mlboydaisuke/LFM2.5-350M-ExecuTorch — exported with
executorch.extension.llm.export.export_llm, qmode: 8da4w, embedding_quantize: "8,0",
XNNPACK.

What it is not

I re-exported the model to remove each candidate in turn and drove every build both ways:

build forward input enable_dynamic_shape token-by-token TextLLMRunner
shipped (use_sdpa_with_kv_cache: True) [1, 2047] True correct degenerate
control (use_sdpa_with_kv_cache: False, enable_dynamic_shape: False) [1, 1] False correct degenerate

So it is not the custom SDPA path, and not dynamic versus static shape — a static [1, 1] export,
which the runner should drive sequentially, fails the same way.

It is not the tokenizer either: the same tokenizer.json encodes the prompt to the expected ids
through a standalone HFTokenizer harness on the same machine, special tokens included, and the
.pte reports get_bos_id = 1, get_eos_ids = [7], matching the tokenizer.

And it is not the runner in general — Qwen3.5-0.8B exported the same way (8da4w,
embedding_quantize: "8,0", XNNPACK) generates correctly through TextLLMRunner on the same
build, so whatever this is interacts with something specific to LFM2.5.

A guess I could not confirm

LFM2.5 is a hybrid: short-convolution blocks interleaved with attention. Convolution blocks carry
state along the sequence, so anything that drives positions differently from a plain
one-token-at-a-time walk — a warmup pass, a re-prefill, a batched first step — would corrupt that
state while leaving a pure-attention model like Qwen unharmed. I could not confirm which of those
the runner does, so I am reporting the isolation rather than a diagnosis.

Happy to run further experiments against this model, or to test a patch.

This report was written with Claude.

cc @larryliu0820 @mergennachin @cccclai @helunwencser @jackzhxng @digantdesai

Vorherrschende Sprache
Python
Sterne
5k
Forks
1.2k
Ø Merge
2 T. 7 Std.
Gemergte PRs (30 T.)
521

Entwicklungsumgebung

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus pytorch/executorch

Alle Issues in pytorch/executorch

Ähnliche Issues

Weitere Issues zu Python

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.