Performance profile: where does all the time go?

Start by reviewing the MLX core quantized GEMM and gather_qmm paths, then use the project's benchmarking method: warm up and compare medians across repeated, like-for-like runs. Capture the execution shapes for the 2,048-token prefill chunks before assessing kernel coverage or tuning. Done means a reproducible improvement on the identified long-prompt workloads, with results outside the stated noise band.

I periodically run a sequence of 40-60 models using mlx/mlx-vlm on a large image, asking it to provide a caption for the image and, in some cases, giving it a fairly lengthy hint as to what the current image is.

My harness records performance data, so I asked my bot to tell me what that showed. It produced the analysis below. This sets out areas that might be profitable for performance improvement (and which would not be), for my workflow. I thought that it might be of wider interest.

---

A profile of the 30 October 2026 sweep (43 cached vision-language models, one
63 MP photograph, one metadata prompt, one generation per model) on an Apple
M5 Max, identifying where engineering effort would visibly reduce wall time
for this workflow — and where it would not. These are observations with
evidence, not requests; every number is from one timed generation per model,
so individual ratios carry the sweep's usual ±15% noise band.

Headline. The sweep spends 55% of its wall time prefilling prompts and
only 18% decoding. Eight models that tokenize the test image into ~16,600
tokens account for 282 s of prefill — 43% of the entire sweep — running at
280–550 tok/s through 4-bit and 2-bit weights. Long-image-prompt quantized
prefill is where essentially all the recoverable time lives. The image-token
budget itself is probably a larger lever, but it is a checkpoint/processor
property, not a runtime one, and is listed separately because changing it
changes the measurement.

Environment and workload

• Apple M5 Max (40-core GPU, applegpu_g17s, Metal 4), 128 GB, macOS 27.2
• mlx 0.32.5.dev20261030+1ba7f969a (local build), mlx-vlm 0.7.11 @ 918763c68,
  transformers 5.22.1, Python 3.14.7
• Input: one 6,656 x 9,483 JPEG (63.1 MP, 60.5 MB); prompt ~320 text tokens;
  max_tokens 1,000; prefill_step_size 2,048; greedy for 25 of 43 models,
  checkpoint sampling defaults elsewhere; one model per process-equivalent
  cycle, no warm-up, no repeats
• Neural-accelerator (NAX) quantized kernels were active: the detection gate
  (macOS >= 26.2, GPU gen >= 17) passes on this machine, and mlx routes
  quantized matmul to qmm_nax/qmv_nax variants when it does

Where the time goes

Wall time 10 m 58 s (658 s), split by harness phase, with generation further
split using each model's recorded prompt and decode rates:

Phase	Time	Share
Generation: prompt prefill	359 s	55%
Generation: token decode	116 s	18%
Model load	86 s	13%
Image preprocessing	35 s	5%
Cleanup + outside the model loop	27 s	4%

Prefill concentration: the eight models whose processors tokenize the image
into more than 10,000 tokens consume 282 s of the 359 s prefill. For each of
them, prefill is 78–96% of generation time:

Model	Prompt tok	Prefill	Rate	Decode
Ternary-Bonsai-2-27B-mlx-2bit	16,594	59.5 s	279 tok/s	3.5 s
Qwen3.8-27B-nvfp4	16,594	49.9 s	333 tok/s	4.1 s
AREX-2-4bit	16,594	46.2 s	360 tok/s	3.8 s
Qwen3-VL-8B-Instruct-4bit	16,578	33.1 s	501 tok/s	1.3 s
Qwen3.5-35B-A3B-4bit	16,594	31.4 s	529 tok/s	1.5 s
Qwen3-VL-30B-A3B-Instruct-4bit	16,578	30.2 s	548 tok/s	2.0 s
Qwen3-Omni-30B-A3B-Instruct-4bit	12,683	17.9 s	708 tok/s	1.4 s
X-Reasoner-7B-8bit	16,589	13.5 s	1,226 tok/s	3.9 s

Area 1 — quantized prefill at long sequence length (owner: mlx core)

This is the profitable area. The operations that matter for this workflow are
the quantized GEMMs of long prefills: dense 4-bit affine (group 32/64), 2-bit,
nvfp4/mxfp4, and MoE expert gathers (gather_qmm) for the A3B-style models.
Prompts here total 12–17k tokens, but that is not the GEMM M: the sweep
requested 2,048-token prefill chunks, and mlx-vlm splits prefill into chunks
for models that support it, so for those models each call probably sees
M ≤ 2,048. The execution shapes actually run were not captured; any kernel
recommendation needs them from a profiled run first.

Evidence worth weighing:

• Observed rates for these prompts are 280–550 tok/s for 4-bit 8–35B models
  and 279 tok/s for the 2-bit 27B, against ~2,700–2,800 tok/s for the same
  class of hardware on 2–3k-token prompts of small/medium models. Rates blend
  vision-tower and language-model time, so they are workload rates, not pure
  GEMM rates.
• The NAX path is already in place and active here: staged affine_qmm_t_nax
  split-K kernels landed 20 October (PR 4682), qmv_mma/qmv_nax for 6–16
  rows on 22 October (PR 4712), an fp4 fast dequant route on 21 October
  (PR 4709). Four sweeps spanning 23–30 October (two directly comparable
  pairs) measured prefill flat to within the noise band across all of it. So
  for these models the remaining profit is in coverage and tuning at the
  shapes these prefills actually execute, not in enabling the hardware.
• One suggestive but confounded pair: at the same ~16.6k-token prompt and
  similar parameter count, an 8-bit model (X-Reasoner-7B, Qwen2.5-VL lineage)
  prefills at 1,226 tok/s while a 4-bit one (Qwen3-VL-8B) does 501 tok/s.
  Weight bandwidth predicts the opposite ordering. The two differ in vision
  tower and architecture, so this is not attributable to dequantization cost
  alone — but it bounds how much the 4-bit path might be leaving on the
  table on long prompts. A counter-datapoint from an August sweep:
  Ministral-3-14B in mxfp4 vs nvfp4 prefilled within 1% of each other, so
  format alone does not dominate.

Profit for this workflow: these eight models' prefill is 43% of the sweep;
each 2x on it returns ~140 s (~21% of wall time).

Area 2 — decode at long context (owner: mlx core; open question)

Decode is only 18% of the sweep, so gains here are bounded. The context
effect is visible but confounded by architecture: two ~8B 4-bit models decode
at 101 tok/s (2k context) vs 67 tok/s (16.6k context), but a third 8B model at
1.3k context decodes at 71 tok/s. KV-cache bandwidth reduction (for example
8-bit KV) is the standard lever at 16k context; this data cannot isolate its
value. The one clear beneficiary of any decode improvement is
Muse-Glimmer-30B-OptiQ (1,000 tokens at 24 tok/s = 41 s, two-thirds of its
runtime).

Area 3 — long-output thinking models (owner: mlx-vlm)

Three models decode 750+ tokens (ERNIE-4.5-VL: 855 tok / 7.3 s;
Muse-Glimmer: 1,000 tok / 41 s; MiniCPM-V-4.6: 752 tok / 3.1 s). mlx-vlm
has speculative-decoding implementations with generation integration; this
sweep did not check which of these models have compatible support, so its
value here is untested. For this sweep it could save under a minute at most
(the three models decode for ~51 s in total); for interactive use of
thinking VLMs, speedups of 1.5–3x are commonly reported, but none were
measured here.

Area 4 — image preprocessing (owner: mlx-vlm / harness; minor)

35 s of CPU-side decode/resize of the 63 MP JPEG across 43 models (~0.8 s per
model, serial with everything else). Overlapping it with model load, or
caching the decoded image across models, would return most of it. Small but
unambiguous.

Not profitable

• Model load: 86 s total, median 1.3 s per model (max 15 s for the 73 GB
  119B model). Already dominated by disk/page-in; little headroom.
• Cleanup and loop overhead: 27 s combined.

The image-token budget is not a runtime optimization

Eight different checkpoints (Qwen3-VL/3.5/3.8/Omni family tokenizers, AREX,
Ternary-Bonsai, X-Reasoner) tokenize this 63 MP image into ~16,600 tokens,
far more than the other models in the sweep. The sweep did not record the
image dimensions their processors actually produced, so how close to native
resolution they run is not known. Capping image resolution for those
processors could cut their prefill substantially, possibly by more than
kernel work can, but this sweep does not measure by how much: that needs the
processed dimensions per model and a controlled comparison of the same models
on the same image at two or three resize levels, recording image-token count
and prefill time at each. It is excluded from the areas above deliberately:
changing the input tokens changes model outputs, so for a comparison harness
it is a measurement-policy decision, not an optimization. Any discussion of
prefill cost for these models should state the token count first.

Calibration: what this week's data already shows

Between 28 and 30 October (identical image, prompt and settings), mlx moved
11 commits, mlx-vlm 21 commits (0.7.10 → 0.7.11) and transformers
5.21 → 5.22.1: 42 of 43 models produced byte-identical text, generation
throughput ratio was 0.980 (median) and prefill 0.984 — flat within the
harness's ±15% single-generation noise band. The 23 and 25 October runs are
excluded from trend claims: the former is a recorded fast-decode outlier
sweep, the latter ran entirely on battery power. Any claim built on these
profiles should be re-verified with the project's benchmarking method
(median of N runs after warm-up, like-for-like A/B across builds) before
being treated as more than an observation.