Vision/multimodal silently dropped with use_tokenizer_template on the python backends (sglang, vllm): images reach the engine but the prompt carries no media placeholder

backend/python/sglang/backend.py と backend/python/vllm/backend.py、特に _messages_to_dicts() と _build_prompt() を読み、その後 core/http/middleware/request.go、core/schema/message.go、backend.proto を調べてください。tokenizer-template リクエストを再現し、sglang と vLLM が転送された画像に対応付けられたメディアプレースホルダーを受け取ることを確認してください。一方、テキストのみのリクエストには影響がないことも確認してください。

LocalAI version

v4.7.1, image localai/localai:v4.7.1-nvidia-l4t-arm64-cuda-13.
Backends cuda13-nvidia-l4t-arm64-sglang (sglang 0.5.17) and cuda13-nvidia-l4t-arm64-vllm (vLLM 0.24.0), both from the gallery.

Environment, CPU architecture, OS, and Version

NVIDIA DGX Spark (GB10, Blackwell sm_121), arm64, Ubuntu 24.04, CUDA 13, driver 595.71.05, Docker with the NVIDIA runtime.

Describe the bug

For a model configured with template.use_tokenizer_template: true, an image sent as an image_url content part is silently ignored. No error, no warning — the model simply answers as if no image had been attached. The same model, image and prompt work correctly when the engine is driven directly (standalone sglang.launch_server and its own /v1/chat/completions), so it is neither the model nor the engine.

The images do reach the backend. What does not reach it is the media placeholder in the prompt:

1. core/http/middleware/request.go decodes the image_url parts into Messages[i].StringImages, and then — for UseTokenizerTemplate — deliberately writes only the text back into StringContent:

// When the backend handles templating itself (UseTokenizerTemplate),
// it also injects media markers server-side (see
// oaicompat_chat_params_parse in llama.cpp). ...
if config.TemplateConfig.UseTokenizerTemplate {
    input.Messages[i].StringContent = textContent
} else {
    input.Messages[i].StringContent, _ = templates.TemplateMultiModal(...)
}

That assumption holds for llama.cpp's server, which injects the markers itself. It does not hold for the python backends: they call tokenizer.apply_chat_template() on plain string content, and a chat template only emits vision tokens when the content is a list of parts.

2. core/schema/message.go (Messages.ToProto()) then drops the image parts entirely and keeps only the concatenated .text.

3. message Message in backend.proto has no media field at all, so images can only travel out-of-band in the global PredictOptions.Images — the image↔message association is lost on the wire.

4. In backend/python/sglang/backend.py, _messages_to_dicts() builds {"role": …, "content": msg.content or ""} and _build_prompt() renders that through apply_chat_template(). For a Qwen3.5-VL model the rendered prompt therefore contains no <|vision_start|><|image_pad|><|vision_end|>.

5. The images themselves are forwarded correctly — backend.py does image_data = list(request.Images) → llm.async_generate(..., image_data=image_data). But sglang's multimodal processor locates images by scanning the prompt for the model's image token (sglang/srt/multimodal/processors/qwen_vl.py: image_token="<|vision_start|><|image_pad|><|vision_end|>" plus the matching regex). With no placeholder present nothing is split out and image_data is discarded without a message.

backend/python/vllm/backend.py has the identical gap: its _messages_to_dicts() is the same string-content version and apply_chat_template() is applied the same way. Its load_image() / multi_modal_data / LimitImagePerPrompt machinery only carries the pixels, not the placeholder.

Rendering the model's own chat template confirms the mechanism directly (Qwen3.5-VL template, jinja2, no model load):

To Reproduce

1. Serve any VLM through the sglang backend with the tokenizer template:

name: vlm
backend: cuda13-nvidia-l4t-arm64-sglang
parameters:
  model: <a Qwen3.5-VL-family checkpoint>
template:
  use_tokenizer_template: true
engine_args:
  model_path: /models/<checkpoint>
  trust_remote_code: true

2. POST /v1/chat/completions with an image_url content part (data URI) and a question about the image.
3. → the model replies that no image was attached. HTTP 200, nothing in the log.
4. Control: send the same request to a standalone python3 -m sglang.launch_server with the same checkpoint → correct answer about the image.

Expected behavior

The image is coupled to the prompt and the model sees it — with use_tokenizer_template: true, on the sglang and vllm backends, the same way it already works on llama-cpp.

Additional context

Two ways to fix it, and they are not mutually exclusive:

(a) Backend-local, small, no protocol change. In _build_prompt(), rebuild the OpenAI content parts for the last user message from request.Images / request.Videos before templating, so the chat template emits the model's own placeholders. The pixels keep travelling via image_data / multi_modal_data:

n_img = len(request.Images) if request.Images else 0
n_vid = len(request.Videos) if request.Videos else 0
if n_img or n_vid:
    idx = next((i for i in range(len(messages_dicts) - 1, -1, -1)
                if messages_dicts[i].get("role") == "user"), None)
    if idx is not None:
        text = messages_dicts[idx].get("content") or ""
        parts = [{"type": "image"}] * n_img + [{"type": "video"}] * n_vid
        if text:
            parts.append({"type": "text", "text": text})
        messages_dicts[idx]["content"] = parts

The existing except TypeError around apply_chat_template() needs widening to except Exception so that a text-only template falls back to string content instead of failing the request. Text-only requests are unaffected — with no images the whole path is a no-op. The same patch applies verbatim to the vllm backend. This covers every single-image and last-turn request, which is effectively all real vision traffic.

(b) Protocol-level, complete. Add repeated string images (and videos/audios) to message Message in backend.proto, stop discarding the parts in Messages.ToProto(), and let the backends read them per message. This is the only way to get multi-turn conversations with images in different turns right, and it fixes every python backend at once.

Happy to send a PR for (a) — that is the change we are running locally.

Related: #10945 (same class of failure — marker↔bitmap coupling — but on the llama-cpp/mtmd path).