metal: run the kernel gate under Metal API validation and shader validation

Start with tools/mac-gate.sh and the gate clauses in crates/metal/src/bin/gate_metal_qwen35moe_kernels/clauses_matvec.rs and clauses_experts.rs; first run plain just lab-metal, then run the two specified gates on a Mac with the listed validation variables. Record the chip, macOS version, commit, exit codes, validation start lines, message counts, and each message with its clause. Done when one comment contains those results, including zero-message results and any panic report header if a panic occurs.

Part of #593. Related: #860.

Metal has two checking layers that an application turns on with environment variables. API validation checks calls to the Metal API. Shader validation instruments the kernels and reports bad memory accesses while they run. The gate uses the second layer in one place only: the clause row_tail_reads runs the row-tail launches in a child process with MTL_SHADER_VALIDATION=1 and counts Invalid Metal usage blocks (crates/metal/src/bin/gate_metal_qwen35moe_kernels/clauses_matvec.rs:158-244). The reason is in mutants.rs: a read that no output shows is seen only by this layer. Nothing runs the other kernels under it. Nothing uses API validation.

What to ask for. Run the Metal device gate and the kernel gate, with both layers on, on one Mac. Report every validation message with the clause it came from. A run with no message is a result too.

The variables. Apple documents them on two pages. If a name below is not on those pages it is marked [unverified].

• API validation: https://developer.apple.com/documentation/xcode/validating-your-apps-metal-api-usage. It says: "The API Validation layer checks for code that calls the Metal API incorrectly, including errors in creating resources, encoding Metal commands, and performing other common tasks." The page lists MTL_DEBUG_LAYER=1 ("Enables all API Validation tests"). Its MTL_DEBUG_LAYER_ERROR_MODE is assert unless set: the layer logs and then asserts, so the first error ends the process.
• Shader validation: https://developer.apple.com/documentation/xcode/validating-your-apps-metal-shader-usage. It says: "You can also configure Shader Validation to copy its messages to the app's standard error stream by setting environment variables MTL_SHADER_VALIDATION=1 and MTL_SHADER_VALIDATION_REPORT_TO_STDERR=1." The page also lists MTL_SHADER_VALIDATION_GLOBAL_MEMORY=1, MTL_SHADER_VALIDATION_THREADGROUP_MEMORY=1 and MTL_SHADER_VALIDATION_TENSOR_VALIDATION (on by default; "Checks tensor operations for invalid arguments"). The kernel gate's tensor-op GEMM clauses use the tensor ops. By default an invalid read returns 0 and an invalid write is dropped (MTL_SHADER_VALIDATION_FAIL_MODE=zerofill), so the run continues, and a clause may then FAIL on its numbers. The page says man MetalValidation has the full list.
• MTL_SHADER_VALIDATION_ENABLE_VERBOSE_LOGGING, which clauses_matvec.rs:172 sets, is not on those pages [unverified]. Do not set it. If you cannot find a name on Apple's pages or in man MetalValidation, say so.
• Whether API validation needs Xcode or the Metal Toolchain installed is [unverified]. Say what you have.

What to run. One process at a time, nothing else on the GPU, on AC power.

export BLOOMERY_GATE_BOUND=3600
cargo build --release -p bloomery-metal --bin gate_metal_device --bin gate_metal_qwen35moe_kernels
export MTL_DEBUG_LAYER=1 MTL_SHADER_VALIDATION=1 MTL_SHADER_VALIDATION_REPORT_TO_STDERR=1 \
       MTL_SHADER_VALIDATION_GLOBAL_MEMORY=1 MTL_SHADER_VALIDATION_THREADGROUP_MEMORY=1
./tools/mac-gate.sh gate_metal_device > validation-device.log 2>&1; echo "rc=$?"
./tools/mac-gate.sh gate_metal_qwen35moe_kernels > validation-kernels.log 2>&1; echo "rc=$?"

These are the two gate steps of just lab-metal (justfile:98). Do not run the whole recipe under the variables. Its metal_scan step builds Metal binary archives, and Apple's shader-validation page says the layer "is incompatible with Metal Binary Archives". Run just lab-metal once without the variables first if you have not, so we know the plain run passes on your Mac. Never run lab-metal-e2e.

The bound is BLOOMERY_GATE_BOUND seconds per gate, 900 unset (tools/mac-gate.sh header). Apple says shader validation costs GPU time and compile time. If a gate ends with exit 124, rerun with a larger bound and say so.

Optional, after both logs are saved: ./tools/mac-gate.sh gate_metal_qwen35moe_kernels --mutants-all under the same variables. Each mutant is a deliberately broken kernel. A message in the child that runs tail_read is expected (it is the control of row_tail_reads). Report the rest, and any child that dies by abort.

Reading the logs. The shader layer prints Metal GPU Validation Enabled when it starts; the gate's own clause looks for that line and for Invalid Metal usage (clauses_matvec.rs:191-192). Count them: grep -c 'Invalid Metal usage' validation-kernels.log. A report of zero messages counts only if the start line is in the log. The gate prints one line per clause (clause <name>: PASS|SKIP|FAIL). A message belongs to the clause line that follows it. If you find one, run that clause alone with ./tools/mac-gate.sh gate_metal_qwen35moe_kernels --only <clause> and paste that log. The clause names are in ./tools/mac-gate.sh gate_metal_qwen35moe_kernels --list.

Where to look first. The tensor_* clauses (Metal Performance Primitives; the page lists "Misuse of Metal Performance Primitives"). The clauses that launch on resident weights (the page lists "Resource not resident"; the open commits a residency set, and generate_metal prints residency_requests=). The clause pair_id: it launches ids past the stack on purpose, and its note says the existing validation clause does not see that launch (clauses_experts.rs:603-612,677). A message there would be the case the clause cannot prove today.

Read this before you run it. On 2026-10-25 the maintainers' Mac (an M1 Pro, macOS 27.0) had three kernel panics. Each was a kernel data abort in com.apple.iokit.IOGPUFamily, and the panicked task was gate_metal_qwen35moe_e2e each time. Two happened while many Metal processes ran at once. The third happened with the e2e gate running alone. We have not found the trigger. That is why this issue asks for the kernel gate only, in one process, and not for lab-metal-e2e. If your Mac panics, stop, do not retry, and paste the panic report's header lines (macOS build, the panicked task, the abort line).

Done looks like. One comment with: the chip, sw_vers -productVersion, the commit, whether plain just lab-metal passed, the exact variables you set, the exit code of each gate, the Metal GPU Validation Enabled line, grep -c 'Invalid Metal usage' for each log, and every message with its clause name (the whole block, from the line of ===== to the next). A message is a finding to triage. Do not change a kernel or a band to silence one (AGENTS.md, "Never").

AI use: drafted with Claude Code.