test(mdla): fold test/multigpu into the MDLA suite and lift the 2-partition caps

Start with test/multigpu/README.md, test/multigpu/mdla_3partition.jl, runtests.jl, and test/mdla.jl, including the MFO_TEST_MULTIGPU and MFO_TEST_MDLA gating paths. Verify the runner's multi-GPU P2P health, choose and implement the gating approach, and confirm the official suite includes the three-partition tests without changing the CPU baseline or producing errors on hosts with fewer GPUs.

Follow-up to #16. Commit 3fe8b0f added test/multigpu/ — exploratory distributed tests needing >= 3 physical GPUs — deliberately not included from runtests.jl or test/mdla.jl, since the standard suite must stay runnable on 0–2 GPU machines. This issue tracks folding them into the MDLA suite properly.

Status (2026-09-30): upstream blockers are cleared (kylebeggs/MultiDeviceLinearAlgebra.jl#21 and #23 both closed). Scope is narrowed to the fold-in, lifting the 2-partition caps in test/mdla_gpu.jl, and documenting the >= 3-GPU run as a manual pre-release check. Provisioning a CI runner is now a non-goal.

Why >= 3 partitions is worth having

Two partitions can never produce a slab cut on both faces; three can. That middle slab is the first case where MDLA's local_x ghost section holds planes from two different owners at once, so it is the first real exercise of the owner-ascending / plane-ascending ordering rule _slab_ghost_layout (src/partitioning.jl) replays from MDLA's _compute_ghost_topology. test/partitioning.jl proves this CPU-side against emulated exchange semantics; test/multigpu/mdla_3partition.jl runs it on real scatter!/reduce!.

Status: written and passing

Verified on a 9x A30 host. test/multigpu/mdla_3partition.jl passes 23/23, and the existing test/mdla.jl passes 34/34 with zero skips. Bitwise == forward parity holds at both 2 and 3 partitions — it was not loosened. The extension itself needed no changes; none of the three pre-agreed fallbacks (stream sync, CG iteration-count tolerance, Krylov.cg dispatch) were needed, and allequal(niters) passed first try.

So this is not a correctness gap — it is purely about making the coverage run as part of the MDLA suite.

The suite that already runs is capped at 2 partitions too

Beyond test/multigpu/ not being included, the tests that do run are capped below the device count. Four testsets in test/mdla_gpu.jl (:363, :387, :568, :605) loop

for nd in 1:min(NGPUS_MDLA, 2)

with NGPUS_MDLA = length(CUDA.devices()) at :9. So those distributed Krylov.cg parity, boundary_rhs, and RHS-assembly testsets can never exercise 3 partitions no matter how many GPUs the host has — the same gap, just inside the suite that already runs. Lifting these needs no new infrastructure.

Upstream blockers: cleared

Those results required a session-local workaround for kylebeggs/MultiDeviceLinearAlgebra.jl#21: MDLA's scatter!/reduce! move ghost data with direct device-to-device copyto!, which on some hosts silently returns zeros with no error. On such a host every >=2-partition test fails while single-partition tests pass, for entirely environmental reasons.

That is resolved. #21 turned out to be a host IOMMU misconfiguration, not an MDLA bug — fixed on sasquatch with intel_iommu=off + reboot, validated 36/36 peer pairs (closed 2026-07-27). MDLA additionally now probes each ordered device pair at GhostExchange construction (kylebeggs/MultiDeviceLinearAlgebra.jl#22) and host-stages any pair that fails, so a similarly broken host produces correct numbers plus a loud one-time warning instead of silent zeros. kylebeggs/MultiDeviceLinearAlgebra.jl#23 (raise MDLA's own device-count caps to NGPUS and cover cross-NUMA pairs) closed 2026-07-28. Nothing upstream blocks this any more.

Any host must still be checked before its results mean anything — the fallback is a safety net, not a fix. test/multigpu/README.md documents the hazard with a four-line check:

using CUDA
CUDA.device!(1); src = CuArray([10.0,20.0,30.0,40.0]); CUDA.synchronize()
CUDA.device!(0); dst = CUDA.zeros(Float64,4); copyto!(dst, src); CUDA.synchronize()
Array(dst)   # [10,20,30,40] = healthy;  [0,0,0,0] = broken P2P

Work items

• Land the MDLA fix (kylebeggs/MultiDeviceLinearAlgebra.jl#21), or confirm the runner's P2P is healthy (IOMMU/ACS is the usual culprit). — Done: #21 was a host IOMMU misconfiguration, fixed on sasquatch; MDLA now probes each device pair and host-stages failures loudly. kylebeggs/MultiDeviceLinearAlgebra.jl#23 (raise the device-count caps upstream) is also closed. Nothing upstream blocks this any more.
• Fold test/multigpu/mdla_3partition.jl into the MDLA suite (it is written as ordinary @testsets so it lifts verbatim) and drop the separate directory. Gating: include it from the MDLA path so it sits behind the existing MFO_TEST_MDLA gate, and rely on its NGPUS_MP >= 3 self-skip. The self-skip path is verified working — under CUDA_VISIBLE_DEVICES=0,1 the suite degrades to a clean skip with no errors.
• Lift the four min(NGPUS_MDLA, 2) caps in test/mdla_gpu.jl so those testsets run at every available partition count.
• Document the >= 3-GPU run as a manual pre-release check, with the P2P health check above as its preflight so a broken host is caught up front rather than discovered through MDLA's host-staging warning (or, on an MDLA predating the probe, 44 mystery failures). The Running and P2P-hazard content in test/multigpu/README.md needs a new home once the directory goes — docs/pages/distributed.qmd ("Running it takes CUDA, an unregistered package, and an env var") already covers the side env and the hazard, and currently points at both test/multigpu/mdla_3partition.jl and test/multigpu/README.md, so those references need updating either way.

Non-goal: provisioning a >= 3-GPU CI runner. The coverage is proven by hand and the host-side P2P fault is now caught by MDLA itself; a manual pre-release check is enough, and runner work can be picked up separately if it ever becomes worth it.

Note that folding these in must not disturb the CPU baseline: 4657 pass / 0 fail / 3 broken (the three env-gated skips), confirmed unchanged by 3fe8b0f.

—
🤖 Claude cut this down from 'provision hardware' to 'move some tests'.
_{claude-opus-5-5[1m] (high) · claude-code 2.1.285 · unreviewed}