GGUF vision towers (mmproj): survey and plan

Start with the completed shared mmproj work and family implementations in #803, then inspect the model architecture mappings referenced here, such as #721 for GLM-5.3-Flash and #725 for MiniMax-M3. Pick one unchecked family and verify its listed tensor transforms, defaults, and image processor behavior against the payload’s findings. Done means that family loads image input from its separate mmproj and its checks pass; the issue does not name test files.

Sub-issue of #671 (GGUF → Vision towers). It records what we found about loading llama.cpp vision towers (mmproj) for the families FreeToken serves images for. Checked against public main 546c5fe and llama.cpp master 339f3242c.

Where it stands

• #803 (merged): image input from an mmproj for Qwen3.5 / 3.6 / 3.8 and Gemma-4 (26B-A4B, 31B, 12B unified); the behavior is under Behavior.
• On the same base, all merged: GLM-5.3-Flash (glm5v) #810, DeepSeek-V4-Flash-Vision-Exp (deepseek4v) #816, Qwen3-VL (qwen3vl_merger with DeepStack) #827, MiniMax-M3 (minimax_m3) #829, Muse-Glimmer (muse-glimmer, images only) #830.
• Every family that has GGUF text support and a llama.cpp vision tower now takes image input. DeepSeek-V4.1 has no GGUF text support.
• llama.cpp always writes the vision tower to a separate file, mmproj-*.gguf:
  • general.architecture = clip, general.type = mmproj.
  • Tensors are named v.* / mm.*; metadata sits under clip.vision.*.
  • The projector type is in clip.projector_type or clip.vision.projector_type.
• llama.cpp loads it with --mmproj FILE. When it downloads a model with -hf, it also picks the sibling *mmproj*.gguf whose quant bits are closest to the text model.
• Old Ollama GGUFs put v.* / mm.* inside the text file. Ollama is migrating them to split files (compatmigrate). None of the HF repos we checked ship that layout.

Per family

Checked by comparing a BF16 mmproj with the HF vision weights after undoing llama.cpp's transforms (CPU only).

Notes:

• Where mmprojs are published:
  • unsloth, ggml-org, bartowski and lmstudio publish one for every family above except MiniMax-M3.
  • For MiniMax-M3, 9 of 26 GGUF repos ship one (bartowski, AesSedai, ...); unsloth does not.
• Weight types:
  • Most mmprojs are BF16, F16 or F32.
  • ggml-org also publishes Q8_0.
  • lmstudio's Muse-Glimmer mmproj is Q4_K_M.
• Image token ids: the mmproj does not store image / boi / eoi token ids. They come from the text GGUF's tokenizer, looked up by token string.
• DeepSeek-V4 specifics:
  • The text GGUF has no image tokens. The img_start / pad / end embeddings live in the mmproj.
  • ggml-org's chat template renders no image placeholder (unsloth's does), so image requests on GGUFs that carry it fail (see Open).
• Image processors: for Qwen, Gemma-4, GLM-5.3, MiniMax-M3 (20 images) and Muse-Glimmer (19 images, two budgets), an image processor built from the mmproj metadata plus the defaults above gives the same output as AutoImageProcessor on the HF release. DeepSeek-V4 has its own preprocessing.

Precision

Step 1 does not quantize the vision tower. Every mmproj is dequantized to bf16 at load.

Why not quantize:

• Every FreeToken tower is plain bf16. Qwen's tower takes config.quant, but GGUFConfig maps no visual.* names.
• The FP8 / NVFP4 / MXFP8 HF releases we checked all keep vision in bf16.
• The towers are 0.1–3.7 GB, and host placement already covers memory pressure.
• vLLM and SGLang:
  • Neither has a feature that quantizes ViT weights on its own initiative; the tower follows the checkpoint.
  • SGLang hardcodes quant_config=None for the Qwen3-VL / Qwen3.5 and MiniMax-M3 towers.
  • vLLM's GGUF plugin handles mmproj for Gemma3 only, and runs it at the file's types.

Behavior

As shipped in #803; the families added later (#810, #816, #827, #829, #830) follow it.

• Only separate llama.cpp mmproj files are read. A text GGUF that carries v.* / mm.* tensors itself (old Ollama layout) serves text, with a warning that those tensors are not read.
• --mmproj FILE picks the file. It must be a llama.cpp vision mmproj and needs a .gguf model. A file the model cannot use (another tower type, another output width, image tokens missing from the text tokenizer) stops startup with an error.
• Without --mmproj, FreeToken looks for *mmproj*.gguf in the model file's own folder only, as llama-server does for a --models-dir entry. It does not look in parent folders, so a repo that keeps each quant in a subfolder and the mmproj at the top needs --mmproj (17 of the 200 most downloaded Qwen GGUF repos).
  • It skips a file that cannot be read, whose output width differs from the text model, or that fails the family's checks. Unreadable files and failed checks are logged; a width mismatch is not (see Open).
  • When the files belong to several models, it takes the one named like the text model. If none matches, the model serves text, with a warning.
  • Among the rest it prefers BF16, then F32, then F16, then quantized files. llama.cpp -hf picks the file whose quant bits are closest to the text model instead.
• No usable mmproj: the model serves text and rejects image requests. --mm-disable vision and --text-model-only leave the mmproj unloaded the same way.
• Families without GGUF image input do not look for an mmproj, and --mmproj has no effect on them.
• Log: one line per process names the mmproj, how it was found, its projector type, its file type -> bf16, and its size. Another line lists the files not used. A quantized file gets a warning. Without an mmproj, the log says vision is disabled and why.
• Precision: the tower runs in bf16. A quantized mmproj is dequantized at load and keeps the quantization error of the file.
• Config: comes from the GGUF files alone, with no HF repo. The mmproj metadata, the text GGUF tokenizer (image token ids) and per-projector defaults supply the values. Image tokens, budgets, --image-min-tokens / --image-max-tokens and --mm-processor-kwargs work as for the HF checkpoint of the same model.
• FTW: an FTW converted from a GGUF serves text only.

Open

• FTW converted from a GGUF keeps no vision tower.
• Discovery drops a folder mmproj without a log line when its output width differs from the text model, or when the several-models name rule filters it out (unsloth Muse-Glimmer's mmproj-kquant.gguf is missing from the "not used" line).
• DeepSeek-V4 GGUFs whose chat template renders no image placeholder (ggml-org, vultr, prometheusAIR, huihui-ai and others: 25 of the 35 files checked with a tower in #816; unsloth's template maps image parts to <｜deepseek_image｜>). FreeToken passes the message content to the template as a list of parts; these templates print it as text, so the request fails with "prompt renders 0 image placeholders" (a 400, or an error chunk when streaming). llama.cpp turns image parts into <__media__> markers first, and vLLM flattens the content for string-content templates.

Work items

• Shared: mmproj discovery, logging, GGUF config with the vision sections, image processor (#803)
• Qwen3.5 / 3.6 / 3.8 (qwen3vl_merger) (#803)
• GLM-5.3-Flash (glm5v): #810
• Gemma-4 26B-A4B / 31B (gemma4v), 12B unified (gemma4uv, vision only) (#803)
• MiniMax-M3 (minimax_m3) (#829)
• Muse-Glimmer (muse-glimmer, images only) (#830)
• DeepSeek-V4-Flash-Vision-Exp (deepseek4v) (#816)
• Qwen3-VL (qwen3vl_merger with DeepStack; text archs in #807) (#827)