GPU performance: tracking the open leads on the CUDA and HIP builds

This is a tracking list, not a single change, so start by reading the ledger sections in docs/dev/benchmarks.md and the measurement steps in docs/dev/benchmarking.md. Pick one linked issue whose "Done when" you can meet on the named hardware, and run python -m benchmarks ab --device with three rounds. Done means that issue closes or this list gets a note explaining why the lead was dropped.

This issue tracks every open lead on the speed of the native backend's GPU builds (CUDA and HIP), so that someone can pick up the next one without reading the history first. Each lead with an issue carries its own detail and "Done when". Tick a box here when its issue closes, and add new leads here as well as opening an issue for them.

Where things stand

PR #59 was the last GPU performance campaign. The ledger's device sections hold the measurements: transfers and the Kerr stage, launches and the Adams passes, graph replay, the four-mode and radial transforms and split real rows. In short, a data-centre GPU's trial is limited by launching 2 to 4 µs kernels from the host, while an NVIDIA A40 is limited by binary64 arithmetic. GPU comes after accuracy and CPU performance in the project's priorities, so this list has no deadline.

Leads, in a suggested order

The order is a suggestion: what makes other measurements trustworthy, then what is cheap, then what is large. Nothing here has been re-measured for this issue.

First, so that the rest can be measured

• #151: no CI lane builds a device target, so device builds break unnoticed.
• #366: some device runs take one of two speeds from process to process, which blurs any A/B ratio.

Host work, which needs little or no GPU time to change

• #346: a saved plane of a large Cartesian field costs about 3 s of host work per plane, mostly copies. It is the only host cost that repeats.
• #347: the fixed host cost of a run, about 6 s on kerr_beam_384.

Launches and copies

• #357: device steps are limited by kernel launches and buffer copies.
• #103: per-trial mirrors, the accept pass and unused plasma buffers.
• #361: node chunk size. Re-measure before changing anything, see the note below.
• #365: the low-storage march's register traffic.
• rocFFT's kernels per transform, no issue yet. On an AMD MI325X rocFFT runs each of a trial's transforms as four kernels, and what remains there is mostly the launch rate, while cuFFT on an H200 uses one. Unmeasured: whether a different plan or a custom kernel for these shapes reduces that. Open an issue if a trace shows the kernels are worth it. Read, not run.
• Radial graph replay on HIP, no issue yet. The MI325X's radial source still holds rocBLAS, which ends the graph region, while the A40's replays through its own GEMM kernel. Unmeasured: a device GEMM kernel on HIP. My suspicion is that the gain is small, since a trace put the saving from recording vendor GEMMs in the graph at about 1 % of a radial trial. Trace first, and open an issue only if it pays.

Kernels

• #352: fusing the band column's kernels, and coalescing the split's transposes.
• #364: the far-field pass's accumulators in global memory.
• #359: the on_axis_* statistics copy the whole state to the host.
• #341: forming fewer propagators where binary64 is slow.

Larger work, each likely its own design

• #356: a batch axis for lockstep batches of small runs. The roadmap expected its own SEP.
• #362: an Adams controller held on the device.
• #355: the carrier frame on devices, refused until it has parity tests and timings.

Related, not speed itself

#353 (the next device qualification), #354 (a scheduled device test lane), #360 (the stepper gallery's device rows), #363 (z tables on GPU builds), #310 (device memory in the run record), #303 (GPU geometries in execution tuning) and SEP 4.

Already tried: do not repeat without new evidence

#59's rejected list and the ledger record these as measured and dropped:

• Eight rows a thread in the compute capability 8.6 GEMM (1.054 against four).
• A team per chunk of the quadrature partials: 39 to 7.8 µs, but the score's sum took most of it back (0.989 on rdw, 0.994 and 1.005 on the single-mode runs).
• Band column tilings: half-tile teams and 512-sample tiles moved nothing on an A40, whose performance counters were not accessible, so what limits those kernels is unknown (#352).
• Adams trial blocks of 64 and 32.
• The split real-row transform on an H200, which lost 4 to 5 %.
• Padding the radial GEMM's row pitch, and recording vendor GEMMs in the graph: the first changed nothing, and the second was not tried, since the trace put its saving at about 1 % of a radial trial.
• A device whole-node chunk, which #59 lists as rejected. #361's older numbers from SEP 16 predate that.

#59 also names "FP32 device paths (findings only)" as a lead, but I found no such finding in the ledger. The nearest record is SEP 36's mixed precision, measured and withdrawn, and #342 holds what single precision still refuses. So it has no box here.

How to measure

• Compare two builds with python -m benchmarks ab --device, three rounds, as the ledger's tables do. Record a run with python -m benchmarks track --device.
• Time on the hardware an item names, since the limits differ: an NVIDIA A40 (compute capability 8.6, slow binary64), an H200 or H100, and an AMD MI325X (HIP).
• Keep the same steps and fields within the floating-point rules of SEP 25, run the gpu-marked tests, and add the result to the ledger.

Done when

Every box is ticked, each by a closed issue or a note here saying why the lead was dropped, with its measurement in the ledger.

Checked against commit 8a9f764. Read, not run: this issue only gathers what the records say.