Vulkan LLMs: what would make decode competitive

Aperta
  • Qualcosa non va?
#1,478 0 commenti 0 reazioni 1 assegnatario Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@msluszniak ci sta già lavorando.

Dal 16/9/2026.

Valutazione

Questa issue non è ancora stata valutata.

Descrizione

idea performance platform: android

Vulkan LLMs already work, but the win is TTFT and memory, not decode. Recording what would change that.

Measured, Qwen3-0.6B vs published XNNPACK:

Adreno 840 Mali-G76
prefill 2.7x faster 4.5x slower
decode 0.86x 1.7x faster, 3.4x at 773 tokens
RSS 303 vs 1088 MiB 1038 vs 1028 MiB

Decode is batch-1 GEMV, so it is bandwidth bound and the GPU has no FLOPs to spend. That explains both columns, including why Mali inverts: prefill per token there is about equal to decode per token, so batching buys nothing.

1. 4-bit weights on GPU

The biggest lever. Decode is bandwidth bound, so weight precision is the bottleneck. int8 is fixed (pytorch/executorch#22429, #22430) and measured faster than fp16 on the mpnet embedders. linear_qcs4w is broken the same way and untouched, so a Vulkan LLM decodes fp16 weights while XNNPACK decodes 4-bit. That is roughly a 4x bandwidth advantage to XNNPACK in exactly the phase Vulkan loses.

### 2. Adreno nondeterminism (pytorch/executorch#22327, #21938)

Correctness, not performance. One nn.Linear(384,384) at M=1500 is wrong on 23/100 executions, error 3x the signal, nothing reported. It is why LFM2.5-350M emitted garbage from a byte-identical cached file. Not root caused, no PR.

3. One artifact does not fit both vendors

force_fp16 is +14% decode on Adreno and loses on both axes on Mali. Vulkan int8 tiled kernels are Adreno tuned and pathological on Mali (5.4x on lfm2.5-embedding, 5-8x on mpnet). Needs per-vendor variant selection or half our users get a regression.

4. Dispatch overhead and occupancy

reduce_gwg hardcodes 16 threads, softmax_buffer hardcodes 4. LFM2.5 text TTFT was 80 vs 56 ms on short prompts, mostly fixed per-call cost.

Known good recipe

Keep for any new Vulkan LLM export: plain VulkanPartitioner so causal_sdpa fuses (57 delegates to 1, decode 41.5 to 86.2 tok/s), and enable_dynamic_shape=True, without which ET prefills one token at a time.

Lingua principale
TypeScript
Stelle
1.8k
Fork
101
Merge medio
2g 4h
PR unite (30g)
58

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di software-mansion/react-native-executorch

Tutte le issue di software-mansion/react-native-executorch

Issue simili

Altre issue su TypeScript

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.