Metal: a lane matmul with an input tile in device memory, also transposed, as llama.cpp's flash attention reads K and V
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- rust
- Domain
- performance
Research direction
Start with fitsSimdgroupMatrices and the E6009/E6010 capacity checks in src/dialect/VxLowering.cpp, then trace how lane matmul inputs are represented and printed. The issue proposes loading eligible f16 or f32 HBM views with simdgroup_load, including transposed views. Done means those inputs need no shared memory in the capacity check and can be used by a Vx lane matmul.
Written by the indexing model from the issue text.
Description
The lane matmul (acc = acc + a @ b) uses the simdgroup matrices of the Apple GPU only when the two inputs are tiles in shared memory (fitsSimdgroupMatrices in src/dialect/VxLowering.cpp needs memory space 3 and rows with stride 1). llama.cpp's flash attention for a prompt loads the tiles of K and V straight from the KV cache in device memory into simdgroup matrices, K with a transposed load:
simdgroup_load(mk, pk + 8*i, NS10, 0, true); // K from device memory, transposed
simdgroup_multiply_accumulate(mqk, mq, mk, mqk);
...
simdgroup_load(mv, pv + 8*i, NS20, 0, false); // V from device memory
(kernels/fa_common.metal in llama.cpp.) So K and V never go through shared memory, and the 32 KB of shared memory holds only Q (f16), the scores and the output sums of 8 queries.
In Vx, a kernel must first copy each tile of K and V into shared memory. For head_dim 128 and f16, a step of 64 positions needs 16 KB for K and 16 KB for V, which does not fit next to Q and the scores. So a Vx kernel takes steps of 16 or 32 positions, with a copy and more barriers in each step. In vx-lang/qwen-vx, attention of a prompt of 8,000 tokens takes about half of the time of the prompt (vx-lang/qwen-vx#43), and its time grows twice as fast with the length as that of llama.cpp.
Proposal
Let an input of a lane matmul be a view of a tensor in Memory::HBM, when:
- its element type is f16 (or f32),
- its shape is static and a multiple of 8 in each dimension,
- one of its strides is 1 (row-major), or the other one is (a transposed view), with the other stride a run-time value.
The printer then loads its 8 x 8 pieces with simdgroup_load(m, ptr, stride, 0, transposed) from device memory. A transposed view would need a way to write it in Vx, for example k_tile.t() or a transpose argument of the matmul.
The capacity check (E6009/E6010) then counts nothing in shared memory for such an input.
What it would allow
A port of llama.cpp's prompt attention: 8 queries for each block, steps of 64 positions, K and V read once from the cache for each step, the softmax in all the lanes of a subgroup. The same form also gives a matrix-vector product with simdgroup matrices from device memory, as llama.cpp's mul_mv_mma does.
🤖 Generated with Claude Code
- Dominant language
- Rust
- Stars
- 35
- Forks
- 5
- Avg merge
- 12h 26m
- Merged PRs (30d)
- 799
Getting set up
We have not checked this project's setup files yet. Start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from vx-lang/Vx
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day
-
bug
Difficulty 1/5 Under an hour Newbie friendliness 88/100
Maintainers usually reply within 1 day
-
bug testing
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
`pub struct` gives "Expected 'fn' after 'pub'": the error should say that a struct needs no `pub`Open
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
Maintainers usually reply within 1 day
-
bug devex
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
longbridge/gpui-kit#4031 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
neuralmagic/crucible#465 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 1 day
-
compiler P3
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
HerringtonDarkholme/hd-lang#2162 ·
Maintainers usually reply within 1 day
-
embedded SQL: pin correlated-guard behaviour when a closed variable is reassigned inside a branchOpenpriority: low
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
jonathanong/no-mistakes#3627 ·
Maintainers usually reply within 1 day