Metal: a lane matmul with an input tile in device memory, also transposed, as llama.cpp's flash attention reads K and V

Open
#3,552 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
rust
Domain
performance

Research direction

Start with fitsSimdgroupMatrices and the E6009/E6010 capacity checks in src/dialect/VxLowering.cpp, then trace how lane matmul inputs are represented and printed. The issue proposes loading eligible f16 or f32 HBM views with simdgroup_load, including transposed views. Done means those inputs need no shared memory in the capacity check and can be used by a Vx lane matmul.

Written by the indexing model from the issue text.

Description

The lane matmul (acc = acc + a @ b) uses the simdgroup matrices of the Apple GPU only when the two inputs are tiles in shared memory (fitsSimdgroupMatrices in src/dialect/VxLowering.cpp needs memory space 3 and rows with stride 1). llama.cpp's flash attention for a prompt loads the tiles of K and V straight from the KV cache in device memory into simdgroup matrices, K with a transposed load:

simdgroup_load(mk, pk + 8*i, NS10, 0, true);   // K from device memory, transposed
simdgroup_multiply_accumulate(mqk, mq, mk, mqk);
...
simdgroup_load(mv, pv + 8*i, NS20, 0, false);  // V from device memory

(kernels/fa_common.metal in llama.cpp.) So K and V never go through shared memory, and the 32 KB of shared memory holds only Q (f16), the scores and the output sums of 8 queries.

In Vx, a kernel must first copy each tile of K and V into shared memory. For head_dim 128 and f16, a step of 64 positions needs 16 KB for K and 16 KB for V, which does not fit next to Q and the scores. So a Vx kernel takes steps of 16 or 32 positions, with a copy and more barriers in each step. In vx-lang/qwen-vx, attention of a prompt of 8,000 tokens takes about half of the time of the prompt (vx-lang/qwen-vx#43), and its time grows twice as fast with the length as that of llama.cpp.

Proposal

Let an input of a lane matmul be a view of a tensor in Memory::HBM, when:

  • its element type is f16 (or f32),
  • its shape is static and a multiple of 8 in each dimension,
  • one of its strides is 1 (row-major), or the other one is (a transposed view), with the other stride a run-time value.

The printer then loads its 8 x 8 pieces with simdgroup_load(m, ptr, stride, 0, transposed) from device memory. A transposed view would need a way to write it in Vx, for example k_tile.t() or a transpose argument of the matmul.

The capacity check (E6009/E6010) then counts nothing in shared memory for such an input.

What it would allow

A port of llama.cpp's prompt attention: 8 queries for each block, steps of 64 positions, K and V read once from the cache for each step, the softmax in all the lanes of a subgroup. The same form also gives a matrix-vector product with simdgroup matrices from device memory, as llama.cpp's mul_mv_mma does.

🤖 Generated with Claude Code

https://claude.ai/code/session_014U9gTxfQzAEpmg4WJhYQqw

Dominant language
Rust
Stars
35
Forks
5
Avg merge
12h 26m
Merged PRs (30d)
799

Getting set up

We have not checked this project's setup files yet. Start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from vx-lang/Vx

All issues in vx-lang/Vx

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.