Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

Vision/multimodal silently dropped with use_tokenizer_template on the python backends (sglang, vllm): images reach the engine but the prompt carries no media placeholder

Abierto
  • ¿Algo no está bien?
#11,621 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 3 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
55/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
go, python
Área
ai, api, backend

Línea de trabajo

Lea backend/python/sglang/backend.py y backend/python/vllm/backend.py, especialmente _messages_to_dicts() y _build_prompt(), y luego inspeccione core/http/middleware/request.go, core/schema/message.go y backend.proto. Reproduzca la solicitud de tokenizer-template y verifique que sglang y vLLM reciban marcadores de posición multimedia vinculados a las imágenes reenviadas, mientras que las solicitudes que solo contienen texto no se vean afectadas.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

area/backends area/vllm bug confirmed
LocalAI version

v4.7.1, image localai/localai:v4.7.1-nvidia-l4t-arm64-cuda-13.
Backends cuda13-nvidia-l4t-arm64-sglang (sglang 0.5.17) and cuda13-nvidia-l4t-arm64-vllm (vLLM 0.24.0), both from the gallery.

Environment, CPU architecture, OS, and Version

NVIDIA DGX Spark (GB10, Blackwell sm_121), arm64, Ubuntu 24.04, CUDA 13, driver 595.71.05, Docker with the NVIDIA runtime.

Describe the bug

For a model configured with template.use_tokenizer_template: true, an image sent as an image_url content part is silently ignored. No error, no warning — the model simply answers as if no image had been attached. The same model, image and prompt work correctly when the engine is driven directly (standalone sglang.launch_server and its own /v1/chat/completions), so it is neither the model nor the engine.

The images do reach the backend. What does not reach it is the media placeholder in the prompt:

  1. core/http/middleware/request.go decodes the image_url parts into Messages[i].StringImages, and then — for UseTokenizerTemplate — deliberately writes only the text back into StringContent:
// When the backend handles templating itself (UseTokenizerTemplate),
// it also injects media markers server-side (see
// oaicompat_chat_params_parse in llama.cpp). ...
if config.TemplateConfig.UseTokenizerTemplate {
    input.Messages[i].StringContent = textContent
} else {
    input.Messages[i].StringContent, _ = templates.TemplateMultiModal(...)
}

That assumption holds for llama.cpp's server, which injects the markers itself. It does not hold for the python backends: they call tokenizer.apply_chat_template() on plain string content, and a chat template only emits vision tokens when the content is a list of parts.

  1. core/schema/message.go (Messages.ToProto()) then drops the image parts entirely and keeps only the concatenated .text.

  2. message Message in backend.proto has no media field at all, so images can only travel out-of-band in the global PredictOptions.Images — the image↔message association is lost on the wire.

  3. In backend/python/sglang/backend.py, _messages_to_dicts() builds {"role": …, "content": msg.content or ""} and _build_prompt() renders that through apply_chat_template(). For a Qwen3.5-VL model the rendered prompt therefore contains no <|vision_start|><|image_pad|><|vision_end|>.

  4. The images themselves are forwarded correctly — backend.py does image_data = list(request.Images) → llm.async_generate(..., image_data=image_data). But sglang's multimodal processor locates images by scanning the prompt for the model's image token (sglang/srt/multimodal/processors/qwen_vl.py: image_token="<|vision_start|><|image_pad|><|vision_end|>" plus the matching regex). With no placeholder present nothing is split out and image_data is discarded without a message.

backend/python/vllm/backend.py has the identical gap: its _messages_to_dicts() is the same string-content version and apply_chat_template() is applied the same way. Its load_image() / multi_modal_data / LimitImagePerPrompt machinery only carries the pixels, not the placeholder.

Rendering the model's own chat template confirms the mechanism directly (Qwen3.5-VL template, jinja2, no model load):

message content rendered user turn
"Wie hoch steht das Wasser?" (what the backend builds today) <|im_start|>user\nWie hoch steht das Wasser?<|im_end|> — no placeholder
[{"type":"image"},{"type":"text","text":"Wie hoch steht das Wasser?"}] <|im_start|>user\n<|vision_start|><|image_pad|><|vision_end|>Wie hoch steht das Wasser?<|im_end|>
To Reproduce
  1. Serve any VLM through the sglang backend with the tokenizer template:
name: vlm
backend: cuda13-nvidia-l4t-arm64-sglang
parameters:
  model: <a Qwen3.5-VL-family checkpoint>
template:
  use_tokenizer_template: true
engine_args:
  model_path: /models/<checkpoint>
  trust_remote_code: true
  1. POST /v1/chat/completions with an image_url content part (data URI) and a question about the image.
  2. → the model replies that no image was attached. HTTP 200, nothing in the log.
  3. Control: send the same request to a standalone python3 -m sglang.launch_server with the same checkpoint → correct answer about the image.
Expected behavior

The image is coupled to the prompt and the model sees it — with use_tokenizer_template: true, on the sglang and vllm backends, the same way it already works on llama-cpp.

Additional context

Two ways to fix it, and they are not mutually exclusive:

(a) Backend-local, small, no protocol change. In _build_prompt(), rebuild the OpenAI content parts for the last user message from request.Images / request.Videos before templating, so the chat template emits the model's own placeholders. The pixels keep travelling via image_data / multi_modal_data:

n_img = len(request.Images) if request.Images else 0
n_vid = len(request.Videos) if request.Videos else 0
if n_img or n_vid:
    idx = next((i for i in range(len(messages_dicts) - 1, -1, -1)
                if messages_dicts[i].get("role") == "user"), None)
    if idx is not None:
        text = messages_dicts[idx].get("content") or ""
        parts = [{"type": "image"}] * n_img + [{"type": "video"}] * n_vid
        if text:
            parts.append({"type": "text", "text": text})
        messages_dicts[idx]["content"] = parts

The existing except TypeError around apply_chat_template() needs widening to except Exception so that a text-only template falls back to string content instead of failing the request. Text-only requests are unaffected — with no images the whole path is a no-op. The same patch applies verbatim to the vllm backend. This covers every single-image and last-turn request, which is effectively all real vision traffic.

(b) Protocol-level, complete. Add repeated string images (and videos/audios) to message Message in backend.proto, stop discarding the parts in Messages.ToProto(), and let the backends read them per message. This is the only way to get multi-turn conversations with images in different turns right, and it fixes every python backend at once.

Happy to send a PR for (a) — that is the change we are running locally.

Related: #10945 (same class of failure — marker↔bitmap coupling — but on the llama-cpp/mtmd path).

Lenguaje dominante
Go
Estrellas
49.2k
Forks
4.5k
Merge medio
1 d 7 h
PR fusionados (30 d)
340

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de mudler/LocalAI

Todos los issues de mudler/LocalAI

Issues similares

Más issues de Go

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.