Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

Streaming: all deltas labeled 'reasoning' when the rendered prompt contains a (pre-closed) thinking marker — non-streaming parses the same output correctly

Abierto
  • ¿Algo no está bien?
#11,135 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 3 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
3/5
Tiempo estimado
1-2 días
Aptitud para principiantes
52/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Tranquilo
Stack tecnológico
go
Área
ai, api, backend

Línea de trabajo

El issue no nombra ningún archivo fuente ni prueba; reproduce el comportamiento con el YAML del modelo proporcionado y una solicitud curl de streaming, y luego localiza el parser de streaming y compara su clasificación con la ruta sin streaming. Se considera terminado cuando la salida generada desde un canal de thinking cerrado de antemano produce deltas de contenido en lugar de deltas de razonamiento, mientras que la salida de thinking normal sigue clasificándose correctamente.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

area/api area/llama.cpp bug unconfirmed

LocalAI version: v4.7.1 (b224c96), localai/localai:latest-gpu-vulkan (docker), llama-cpp backend
Model: Gemma-4-12B-it (GGUF, Q8_0)

Summary

Gemma-4's official chat template disables thinking by ending the generation prompt with a pre-closed thought channel:

<|turn>model
<|channel>thought
<channel|>

With the channel already closed, the model emits plain answer text only — no thinking tokens at all. The non-streaming endpoint handles this correctly (message.content filled, no reasoning field). But in streaming mode, every single delta is emitted as delta.reasoning and delta.content stays null for the entire stream. The stream parser apparently enters the reasoning state because the prompt contains <|channel>thought, without checking that it is immediately closed — and never leaves that state, since the model (correctly) never emits a close marker itself.

Any correctly-templated "thinking disabled" Gemma-4 request triggers this, so OpenAI-compatible clients that read delta.content receive a completely empty stream while the server generates a full answer.

Reproduction

Model YAML (explicit template mirroring the GGUF's embedded jinja for the thinking-off case):

name: gemma-4-12b-it
backend: llama-cpp
parameters:
    model: gemma-4-12b-it-Q8_0.gguf
gpu_layers: 99
stopwords:
- "<turn|>"
template:
    chat_message: |-
        <|turn>{{if eq .RoleName "assistant"}}model{{else}}{{.RoleName}}{{end}}
        {{ if .Content }}{{.Content}}{{ end }}<turn|>
    chat: |-
        {{.Input -}}
        <|turn>model
        <|channel>thought
        <channel|>

Non-streaming — correct:

curl -s http://localhost:8080/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"gemma-4-12b-it","max_tokens":40,"messages":[{"role":"user","content":"Say READY."}]}'
# → "message":{"role":"assistant","content":"READY. …"}   (no reasoning field, finish_reason stop)

Streaming — broken (same request + "stream": true):

"delta":{"role":"assistant","content":null}
"delta":{"content":null,"reasoning":"READY"}
"delta":{"content":null,"reasoning":"."}
...            ← 100 % of tokens arrive as reasoning; not a single content delta
Expected

The stream parser should only enter the reasoning state when the model emits a thinking-open marker (or should recognize that the marker in the prompt is immediately followed by its close <channel|>). Streaming and non-streaming should classify identical output identically.

Workarounds tried (no effect on the stream labeling)
  • request level: chat_template_kwargs: {enable_thinking: false}, thinking: false, reasoning_format: "none", disable_thinking: true
  • model YAML: disable_thinking: true, thinking_start_tokens: ["<NEVER_EMITTED>"], logit_bias (appears to be ignored on the llama-cpp path)
  • Removing the pre-close from the template is not viable: Gemma-4 then opens the thought channel itself (that pre-close is the model's official "thinking off" mechanism).
Lenguaje dominante
Go
Estrellas
49.2k
Forks
4.5k
Merge medio
1 d 7 h
PR fusionados (30 d)
340

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de mudler/LocalAI

Todos los issues de mudler/LocalAI

Issues similares

Más issues de Go

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.