Vulkan LLMs: what would make decode competitive
维护者通常 1 天内回复
@msluszniak 已经在做这个了。
开始于 2026年9月16日。
评估
这个 Issue 还没有评估数据。
描述
Vulkan LLMs already work, but the win is TTFT and memory, not decode. Recording what would change that.
Measured, Qwen3-0.6B vs published XNNPACK:
| Adreno 840 | Mali-G76 | |
|---|---|---|
| prefill | 2.7x faster | 4.5x slower |
| decode | 0.86x | 1.7x faster, 3.4x at 773 tokens |
| RSS | 303 vs 1088 MiB | 1038 vs 1028 MiB |
Decode is batch-1 GEMV, so it is bandwidth bound and the GPU has no FLOPs to spend. That explains both columns, including why Mali inverts: prefill per token there is about equal to decode per token, so batching buys nothing.
1. 4-bit weights on GPU
The biggest lever. Decode is bandwidth bound, so weight precision is the bottleneck. int8 is fixed (pytorch/executorch#22429, #22430) and measured faster than fp16 on the mpnet embedders. linear_qcs4w is broken the same way and untouched, so a Vulkan LLM decodes fp16 weights while XNNPACK decodes 4-bit. That is roughly a 4x bandwidth advantage to XNNPACK in exactly the phase Vulkan loses.
### 2. Adreno nondeterminism (pytorch/executorch#22327, #21938)
Correctness, not performance. One nn.Linear(384,384) at M=1500 is wrong on 23/100 executions, error 3x the signal, nothing reported. It is why LFM2.5-350M emitted garbage from a byte-identical cached file. Not root caused, no PR.
3. One artifact does not fit both vendors
force_fp16 is +14% decode on Adreno and loses on both axes on Mali. Vulkan int8 tiled kernels are Adreno tuned and pathological on Mali (5.4x on lfm2.5-embedding, 5-8x on mpnet). Needs per-vendor variant selection or half our users get a regression.
4. Dispatch overhead and occupancy
reduce_gwg hardcodes 16 threads, softmax_buffer hardcodes 4. LFM2.5 text TTFT was 80 vs 56 ms on short prompts, mostly fixed per-call cost.
Known good recipe
Keep for any new Vulkan LLM export: plain VulkanPartitioner so causal_sdpa fuses (57 delegates to 1, decode 41.5 to 86.2 tok/s), and enable_dynamic_shape=True, without which ET prefills one token at a time.
- 主要语言
- TypeScript
- 星标
- 1.8k
- 派生
- 101
- 平均合并
- 2 天 5 小时
- 30 天内合并 PR
- 57
环境准备
- 没有 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
software-mansion/react-native-executorch 的其他 Issue
-
Require react-native-blob-util >=1.0.0 (Xcode 27 iOS build failure)可能已有人在做 @msluszniak 于 1 天前认领。 未关闭dependencies platform: ios
难度 1/5 1-3 小时 新手友好度 78/100
software-mansion/react-native-executorch#1617 · 已指派 1 人 ·
维护者通常 1 天内回复
-
Peer react-native >=0.83可能已有人在做 @msluszniak 于 1 天前认领。 未关闭dependencies dev exp
难度 1/5 1 小时以内 新手友好度 25/100
software-mansion/react-native-executorch#1623 · 已指派 1 人 ·
维护者通常 1 天内回复
-
Docs: iOS < 17 fails pod install on the bare template可能已有人在做 @msluszniak 于 1 天前认领。 未关闭documentation platform: ios
难度 1/5 1 小时以内 新手友好度 20/100
software-mansion/react-native-executorch#1622 · 已指派 1 人 ·
维护者通常 1 天内回复
-
Docs: dynamic frameworks are supported可能已有人在做 @msluszniak 于 1 天前认领。 未关闭documentation platform: ios
难度 1/5 1-3 小时 新手友好度 35/100
software-mansion/react-native-executorch#1621 · 已指派 1 人 ·
维护者通常 1 天内回复
-
Docs: drop the Expo babel.config.js step可能已有人在做 @msluszniak 于 1 天前认领。 未关闭documentation
难度 1/5 1 小时以内 新手友好度 25/100
software-mansion/react-native-executorch#1620 · 已指派 1 人 ·
维护者通常 1 天内回复
查看 software-mansion/react-native-executorch 的全部 Issue
相似的 Issue
-
Add: ESP 2 RO未关闭approved check:passed streams:add
难度 2/5 1-3 小时 新手友好度 66/100
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 68/100
维护者通常 1 天内回复
-
ipaas: Configure form shows "Deploy to apply." and "Overrides selected Link" when they don't apply未关闭
难度 2/5 1-3 小时 新手友好度 67/100
wso2/integration-control-plane#1118 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 68/100
UTDNebula/utd-notebook#311 ·
维护者通常 2 天内回复
-
难度 2/5 1-3 小时 新手友好度 78/100
维护者通常 1 天内回复