Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

TextLLMRunner degenerates on an LFM2.5 export that steps correctly token by token

Abierto
  • ¿Algo no está bien?
#21,858 0 comentarios 0 reacciones 1 asignado Ver en GitHub

Los mantenedores suelen responder en 1 día

@JakeStevens ya está trabajando en esto.

Desde el 15/8/2026.

Evaluación

Este issue todavía no se ha evaluado.

Descripción

module: llm

TextLLMRunner produces degenerate output for an LFM2.5 export that generates correctly when the same .pte is stepped token by token. Reproduces on macOS with the pip wheel and on Android through LlmModule, and does not depend on the export configuration.

Environment: executorch 1.4.0 (pip), torch 2.13.0, macOS arm64; also reproduced on a Pixel 8a with an AAR built from main.

What happens

Same .pte, same prompt, same machine, two ways of driving it:

token-by-token, explicit input_pos   "Here are three colors:\n\n1. **Red**\n2. **Green**"
TextLLMRunner                        "s::::<|im_end|>"

Shorter prompts make the shape clearer — the runner repeats one token:

prompt "Hi"                    -> "HiHiHiHiHiHiHiHiHiHiHiHiHiHiHiHi"
prompt "Name three colours."   -> "<|im_end|><|im_end|>"

Repro

from executorch.extension.llm.custom_ops import custom_ops  # noqa
from executorch.kernels import quantized  # noqa
from executorch.extension.llm.runner import TextLLMRunner, GenerationConfig

r = TextLLMRunner("lfm2_5_350m_xnnpack_8da4w.pte", "tokenizer.json")
out = []
r.generate("Hi", GenerationConfig(max_new_tokens=16, echo=False), lambda t: out.append(t))
print("".join(out))     # HiHiHiHiHiHiHiHiHiHiHiHiHiHiHiHi

The same file through a loop that steps one token at a time answers normally:

import torch
from executorch.extension.pybindings.portable_lib import _load_for_executorch

model = _load_for_executorch("lfm2_5_350m_xnnpack_8da4w.pte")
for i, tok in enumerate(prompt_tokens):
    logits = model.forward((torch.tensor([[tok]]), torch.tensor([i])))[0]
# then greedy-decode from logits, advancing input_pos each step

The .pte is public: https://huggingface.co/mlboydaisuke/LFM2.5-350M-ExecuTorch — exported with
executorch.extension.llm.export.export_llm, qmode: 8da4w, embedding_quantize: "8,0",
XNNPACK.

What it is not

I re-exported the model to remove each candidate in turn and drove every build both ways:

build forward input enable_dynamic_shape token-by-token TextLLMRunner
shipped (use_sdpa_with_kv_cache: True) [1, 2047] True correct degenerate
control (use_sdpa_with_kv_cache: False, enable_dynamic_shape: False) [1, 1] False correct degenerate

So it is not the custom SDPA path, and not dynamic versus static shape — a static [1, 1] export,
which the runner should drive sequentially, fails the same way.

It is not the tokenizer either: the same tokenizer.json encodes the prompt to the expected ids
through a standalone HFTokenizer harness on the same machine, special tokens included, and the
.pte reports get_bos_id = 1, get_eos_ids = [7], matching the tokenizer.

And it is not the runner in general — Qwen3.5-0.8B exported the same way (8da4w,
embedding_quantize: "8,0", XNNPACK) generates correctly through TextLLMRunner on the same
build, so whatever this is interacts with something specific to LFM2.5.

A guess I could not confirm

LFM2.5 is a hybrid: short-convolution blocks interleaved with attention. Convolution blocks carry
state along the sequence, so anything that drives positions differently from a plain
one-token-at-a-time walk — a warmup pass, a re-prefill, a batched first step — would corrupt that
state while leaving a pure-attention model like Qwen unharmed. I could not confirm which of those
the runner does, so I am reporting the isolation rather than a diagnosis.

Happy to run further experiments against this model, or to test a patch.

This report was written with Claude.

cc @larryliu0820 @mergennachin @cccclai @helunwencser @jackzhxng @digantdesai

Lenguaje dominante
Python
Estrellas
5k
Forks
1.2k
Merge medio
2 d 5 h
PR fusionados (30 d)
491

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de pytorch/executorch

Todos los issues de pytorch/executorch

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.