Optional active GPU memory test for nodes without DCGM (opt-in, like AUDIT_DCGM_DIAG_R2)

Start with host-check.sh and the existing dcgmi/AUDIT_DCGM_DIAG_R2 handling to understand how audit results are emitted. Review tests/audit/ and the cmax audit entry point, then verify that the opt-in path produces each listed status while the default remains skipped and existing audit behavior is unchanged.

We maintain Pantheon, an open-source (Apache-2.0) GPU diagnostics suite, so please read this as a suggestion from an interested party. We would like to know if it fits before we write a pull request.

What we noticed in cmax audit

The active GPU diagnostic that host-check.sh runs is dcgmi diag -r 1, with -r 2 behind AUDIT_DCGM_DIAG_R2=true. That leaves two cases with no active result:

1. A node where dcgmi is not installed. The audit records WORKER_DCGM_DIAG_R1=no-dcgmi.
2. An AMD node. The audit records whether rvs is installed, its version and its config directory, and does not run it.

In both cases the audit can say which tools the operator installed. It cannot say if the memory of the cards in front of it returns what was written to it.

What we suggest

One more opt-in block in the same pattern, off by default so the audit stays read-only:

AUDIT_PANTHEON=true cmax audit

It runs one memory pattern test (March C-) for 30 seconds on every GPU and records one value:

What it adds next to the vendor tools:

• The same test and the same result format on NVIDIA and AMD.
• It runs as the tenant. It needs no root access, no daemon and no kernel module (pipx install pantheon-gpu).
• The report is a JSON file that names each card by UUID, so a result can be tied to a card and repeated later by someone else.

What it does not do, so that nobody is surprised:

• It needs the CUDA or ROCm compiler on the node, because it compiles its workloads for the cards it finds. The first run took about 80 seconds longer on our machine for that reason. A node without a compiler gives no-gpu-backend, never pass.
• It is not a replacement for dcgmi diag or RVS. Those test more of the system (PCIe, power, pulse tests under the vendor's thresholds). This is a second opinion from outside the vendor's stack.
• It loads the GPUs for 30 seconds each, which is why it should be opt-in.

Sketch of the block

We ran this block as it is on a machine with two NVIDIA GPUs. It gave skipped by default, pass on the two cards (39 seconds), no-gpu-backend without a GPU compiler, not-found without Pantheon, and fail when the command exits with an error. We have no card with uncorrectable errors, so we tested that branch with a report edited by hand.

# --- Pantheon (opt-in active GPU memory test, the same on NVIDIA and AMD) ---
if [[ "${AUDIT_PANTHEON:-false}" == "true" || "${AUDIT_PANTHEON:-0}" == "1" ]]; then
    if command -v pantheon &>/dev/null; then
        _pantheon_dir=$(mktemp -d)
        (cd "$_pantheon_dir" && timeout 300 pantheon --test march_test --duration 30 --gpu all >/dev/null 2>&1)
        PANTHEON_RC=$?
        if [[ $PANTHEON_RC -eq 124 ]]; then
            echo "WORKER_PANTHEON_MARCH=timeout"
        elif grep -rqs '"type": "MOCK"' "$_pantheon_dir/database"; then
            # No GPU compiler on the node: the workload ran on the CPU backend.
            echo "WORKER_PANTHEON_MARCH=no-gpu-backend"
        elif [[ $PANTHEON_RC -ne 0 ]]; then
            echo "WORKER_PANTHEON_MARCH=fail"
        elif grep -rqs '"RAS Status": "ERROR"' "$_pantheon_dir/database"; then
            # The exit code covers the workload. Counted errors are in the report.
            echo "WORKER_PANTHEON_MARCH=fail"
        elif ! ls "$_pantheon_dir"/database/*.json &>/dev/null; then
            echo "WORKER_PANTHEON_MARCH=no-report"
        else
            echo "WORKER_PANTHEON_MARCH=pass"
        fi
        rm -rf "$_pantheon_dir"
    else
        echo "WORKER_PANTHEON_MARCH=not-found"
    fi
else
    echo "WORKER_PANTHEON_MARCH=skipped"
fi

One row of the report it reads from, from a real run (the card's UUID is in the same file). The warning in this row is the PCIe link changing its power state, which Pantheon reports and does not count as a failure:

{
  "Test Name": "march_test",
  "GPU ID": 0,
  "Score": 2915520000.0,
  "Unit": "march-ops/s",
  "Status": null,
  "Failure Stage": null,
  "Max Temp (C)": 55.0,
  "Limit Reason": "None",
  "RAS Status": "WARNING",
  "RAS Error Delta": "vendor_ras.pcie.l0_to_recovery +1",
  "Throttle Time (s)": 0.0
}

If this is useful we will send a pull request with the block, the report lines in the harness scripts, and tests in tests/audit/. If it is out of scope for cmax, that is a fine answer too, and we would still be glad to hear what an auditor would need from a test like this.

This issue was drafted with the help of an AI assistant. We ran the block and the workload ourselves, on our hardware.