[Issue]: Dual Radeon AI PRO R9700 – intermittent full-system resets during ROCm inference, Data Fabric Sync Flood 0x08000800

Start by reviewing the attached kernel logs, PCIe dumps, and ROCm diagnostics, then compare the reported reset messages with the dual-GPU configuration and workload details. The issue names no source files or tests; progress would mean identifying whether the failures point to a driver, GPU, or PCIe problem and documenting useful diagnostics or a confirmed workaround.

Problem Description

Dear AMD ROCm Engineering Team,

I am experiencing intermittent hard system resets during local LLM inference on a workstation equipped with two Radeon AI PRO R9700 GPUs.

These are complete machine resets, not just application crashes. The system may restart without a preceding kernel panic or a recoverable GPU reset.

System configuration

CPU: AMD Ryzen 9 7900

Motherboard: ASUS ProArt X870E-CREATOR WIFI

BIOS: 2503, AGESA 1.3.0.1d

GPU 1: Radeon AI PRO R9700 32 GB, 0000:03:00.0

GPU 2: Radeon AI PRO R9700 32 GB, 0000:07:00.0

GPU architecture: gfx1201

Operating system: Ubuntu Linux

ROCm driver: 7.1.9.31600000

amdgpu-dkms: 7.1.9.31600000

amdgpu packages: 26.14.26140000

Workload: llama.cpp / llama-server with local Qwen LLMs, including Qwen3-8B and a 27B model

Observed failure

The system resets unexpectedly during GPU-accelerated inference. The problem has occurred during model processing and other GPU-intensive operations, but is not strictly limited to the largest prompts.

After reboot, the kernel reports:

x86/amd: Previous system reset reason [0x08000800]:
an uncorrected error caused a data fabric sync flood event

In earlier failure logs, I have also observed:

uncorrectable hardware error (ERREVENT_ATHUB_INTERRUPT)
device lost from bus!

Some previous logs also contained high GPU UMC error counts and subsequent GPU initialization failures. These messages were not necessarily captured immediately before every reset.

Mitigations already attempted

Disabled PCIe ASPM via pcie_aspm=off

Disabled GFXOFF for both discrete GPUs

Limited each GPU to 240 W (1200W power supply)

Selected the COMPUTE power profile

Set PCIe runtime power control to on

Bios updates

Tested with EXPO disabled

The full-system resets have continued despite these changes.

Additional PCIe observation

The first CPU PCIe root port negotiates Gen5 x8, while the second currently negotiates Gen4 x8 even though its target speed is Gen5:

00:01.1: 32 GT/s x8
00:01.3: 16 GT/s x8

The second GPU endpoint itself reports a Gen5 x16 link to its downstream PCIe switch.

I am reporting the link negotiation issue separately to ASUS. I do not know whether it is related to the resets.

Questions

Are Data Fabric Sync Flood resets or ATHUB errors known with dual Radeon AI PRO R9700 configurations?

Are there known stability issues with ROCm 7.1.3 and llama.cpp inference on gfx1201?

Could the earlier ATHUB and UMC errors indicate a GPU, PCIe, memory-management or driver issue?

What additional logs or diagnostic tests would help isolate the failure?

Is there a recommended driver version, firmware update or workaround for this configuration?

I can provide kernel logs, PCIe capability/status dumps, ROCm diagnostics and the llama-server configuration.

Thank you for investigating.

Operating System

Ubuntu 26.04.1 LTS (Resolute Raccoon)

CPU

AMD Ryzen 9 7900 12-Core Processor

GPU

2 x AMD Ryzen 9 7900 12-Core Processor

ROCm Version

7.1.3.31500000 / 7.1.9.31600000

Installation Method

Built from source

Installed ROCm Packages / Versions

:$ apt list --installed 2>/dev/null | grep -iE "rocm|hip|amdgpu"
amdgpu-core/now 1:26.14.26140000-2390957.26.04 all [Installiert,aktualisierbar auf: 1:26.16.26160000-2407865.26.04]
amdgpu-dkms-firmware/now 1:31.50.0.0.31500000-2390945.26.04 all [Installiert,aktualisierbar auf: 1:31.60.0.0.31600000-2407772.26.04]
amdgpu-dkms/resolute,resolute,resolute,resolute,now 1:7.1.9.31600000-2407772.26.04 all [installiert]
amdgpu-install/resolute,resolute,resolute,resolute,now 31.60.0.0.31600000-2407772.26.04 all [installiert]
amdgpu-lib/now 1:26.14.26140000-2390957.26.04 amd64 [Installiert,aktualisierbar auf: 1:26.16.26160000-2407865.26.04]
amdgpu-multimedia/now 1:26.14.26140000-2390957.26.04 amd64 [Installiert,aktualisierbar auf: 1:26.16.26160000-2407865.26.04]
amdrocm-amdsmi10.0/now 10.0.0pre4-32424980240 amd64 [Installiert,lokal]
amdrocm-base10.0/now 10.0.0pre4-32424980240 amd64 [Installiert,lokal]
amdrocm-blas-host10.0/now 10.0.0pre4-32424980240 amd64 [Installiert,lokal]
amdrocm-blas10.0-gfx1201/now 10.0.0pre4-32424980240 amd64 [Installiert,lokal]
amdrocm-ck10.0/now 10.0.0pre4-32424980240 amd64 [Installiert,lokal]
amdrocm-core10.0-gfx1201/now 10.0.0pre4-32424980240 amd64 [Installiert,lokal]
amdrocm-decode10.0/now 10.0.0pre4-32424980240 amd64 [Installiert,lokal]
amdrocm-dnn-host10.0/now 10.0.0pre4-32424980240 amd64 [Installiert,lokal]
amdrocm-dnn10.0-gfx1201/now 10.0.0pre4-32424980240 amd64 [Installiert,lokal]
amdrocm-fft-host10.0/now 10.0.0pre4-32424980240 amd64 [Installiert,lokal]
amdrocm-fft10.0-gfx1201/now 10.0.0pre4-32424980240 amd64 [Installiert,lokal]
amdrocm-hipfile10.0/now 10.0.0pre4-32424980240 amd64 [Installiert,lokal]
amdrocm-hipify10.0/now 10.0.0pre4-32424980240 amd64 [Installiert,lokal]
amdrocm-jpeg10.0/now 10.0.0pre4-32424980240 amd64 [Installiert,lokal]
amdrocm-llvm10.0/now 10.0.0pre4-32424980240 amd64 [Installiert,lokal]
amdrocm-math-common10.0/now 10.0.0pre4-32424980240 amd64 [Installiert,lokal]
amdrocm-opencl10.0/now 10.0.0pre4-32424980240 amd64 [Installiert,lokal]
amdrocm-opencl/now 10.0.0pre4-32424980240 amd64 [Installiert,aktualisierbar auf: 10.1.0-3]
amdrocm-profiler-base10.0/now 10.0.0pre4-32424980240 amd64 [Installiert,lokal]
amdrocm-rand-host10.0/now 10.0.0pre4-32424980240 amd64 [Installiert,lokal]
amdrocm-rand10.0-gfx1201/now 10.0.0pre4-32424980240 amd64 [Installiert,lokal]
amdrocm-rccl-host10.0/now 10.0.0pre4-32424980240 amd64 [Installiert,lokal]
amdrocm-rccl10.0-gfx1201/now 10.0.0pre4-32424980240 amd64 [Installiert,lokal]
amdrocm-rocshmem10.0/now 10.0.0pre4-32424980240 amd64 [Installiert,lokal]
amdrocm-runtime10.0/now 10.0.0pre4-32424980240 amd64 [Installiert,lokal]
amdrocm-runtime/now 10.0.0pre4-32424980240 amd64 [Installiert,aktualisierbar auf: 10.1.0-3]
amdrocm-solver-host10.0/now 10.0.0pre4-32424980240 amd64 [Installiert,lokal]
amdrocm-solver10.0-gfx1201/now 10.0.0pre4-32424980240 amd64 [Installiert,lokal]
amdrocm-sparse-host10.0/now 10.0.0pre4-32424980240 amd64 [Installiert,lokal]
amdrocm-sparse10.0-gfx1201/now 10.0.0pre4-32424980240 amd64 [Installiert,lokal]
amdrocm-sysdeps10.0/now 10.0.0pre4-32424980240 amd64 [Installiert,lokal]
amdrocm10.0-gfx1201/now 10.0.0~pre4-32424980240 amd64 [Installiert,lokal]
hip-utils/resolute,now 7.1.0-0ubuntu2 amd64 [Installiert,automatisch]
hipcc/resolute,now 7.1.1+dfsg-0ubuntu1 amd64 [Installiert,automatisch]
hipify-perl/resolute,now 7.1.1+dfsg-0ubuntu1 all [Installiert,automatisch]
libamdhip64-7/resolute,now 7.1.0-0ubuntu2 amd64 [Installiert,automatisch]
libamdhip64-dev/resolute,now 7.1.0-0ubuntu2 amd64 [Installiert,automatisch]
libdrm-amdgpu-amdgpu1/now 1:2.4.131.26140000-2390957.26.04 amd64 [Installiert,aktualisierbar auf: 1:2.4.134.26160000-2407865.26.04]
libdrm-amdgpu-common/now 1.0.0.26140000-2390957.26.04 all [Installiert,aktualisierbar auf: 1.0.0.26160000-2407865.26.04]
libdrm-amdgpu1/resolute,now 2.4.131-1 amd64 [Installiert,automatisch]
libdrm2-amdgpu/now 1:2.4.131.26140000-2390957.26.04 amd64 [Installiert,aktualisierbar auf: 1:2.4.134.26160000-2407865.26.04]
libegl1-amdgpu-mesa-drivers/now 1:26.1.3.26140000-2390957.26.04 amd64 [Installiert,aktualisierbar auf: 1:26.2.0.26160000-2407865.26.04]
libegl1-amdgpu-mesa/now 1:26.1.3.26140000-2390957.26.04 amd64 [Installiert,aktualisierbar auf: 1:26.2.0.26160000-2407865.26.04]
libgbm1-amdgpu/now 1:26.1.3.26140000-2390957.26.04 amd64 [Installiert,aktualisierbar auf: 1:26.2.0.26160000-2407865.26.04]
libgl1-amdgpu-mesa-dri/now 1:26.1.3.26140000-2390957.26.04 amd64 [Installiert,aktualisierbar auf: 1:26.2.0.26160000-2407865.26.04]
libgl1-amdgpu-mesa-glx/now 1:26.1.3.26140000-2390957.26.04 amd64 [Installiert,aktualisierbar auf: 1:26.2.0.26160000-2407865.26.04]
libhipblas-common-dev/resolute,now 7.1.1-0ubuntu1 amd64 [Installiert,automatisch]
libhipblas-dev/resolute,now 7.1.0-0ubuntu5 amd64 [Installiert,automatisch]
libhipblas3-tests-data/resolute,now 7.1.0-0ubuntu5 all [Installiert,automatisch]
libhipblas3-tests/resolute,now 7.1.0-0ubuntu5 amd64 [Installiert,automatisch]
libhipblas3/resolute,now 7.1.0-0ubuntu5 amd64 [Installiert,automatisch]
libhipcub-dev/resolute,now 7.1.0-0ubuntu3 all [Installiert,automatisch]
libhipcub-tests/resolute,now 7.1.0-0ubuntu3 amd64 [Installiert,automatisch]
libhipfft-dev/resolute,now 7.1.1-0ubuntu1 amd64 [Installiert,automatisch]
libhipfft0-tests/resolute,now 7.1.1-0ubuntu1 amd64 [Installiert,automatisch]
libhipfft0/resolute,now 7.1.1-0ubuntu1 amd64 [Installiert,automatisch]
libhipfftw0/resolute,now 7.1.1-0ubuntu1 amd64 [Installiert,automatisch]
libhiprand-dev/resolute,now 7.1.1-0ubuntu1 amd64 [Installiert,automatisch]
libhiprand1-tests/resolute,now 7.1.1-0ubuntu1 amd64 [Installiert,automatisch]
libhiprand1/resolute,now 7.1.1-0ubuntu1 amd64 [Installiert,automatisch]
libhiprtc-builtins7/resolute,now 7.1.0-0ubuntu2 amd64 [Installiert,automatisch]
libhiprtc7/resolute,now 7.1.0-0ubuntu2 amd64 [Installiert,automatisch]
libhipsolver-dev/resolute,now 7.1.0-0ubuntu4 amd64 [Installiert,automatisch]
libhipsolver-fortran-dev/resolute,now 7.1.0-0ubuntu4 amd64 [Installiert,automatisch]
libhipsolver-fortran1/resolute,now 7.1.0-0ubuntu4 amd64 [Installiert,automatisch]
libhipsolver1-tests-data/resolute,now 7.1.0-0ubuntu4 all [Installiert,automatisch]
libhipsolver1-tests/resolute,now 7.1.0-0ubuntu4 amd64 [Installiert,automatisch]
libhipsolver1/resolute,now 7.1.0-0ubuntu4 amd64 [Installiert,automatisch]
libhipsparse-dev/resolute,now 7.1.0-0ubuntu1 amd64 [Installiert,automatisch]
libhipsparse4-tests/resolute,now 7.1.0-0ubuntu1 amd64 [Installiert,automatisch]
libhipsparse4/resolute,now 7.1.0-0ubuntu1 amd64 [Installiert,automatisch]
libllvm22.1-amdgpu/now 1:22.1.26140000-2390957.26.04 amd64 [Installiert,aktualisierbar auf: 1:22.1.26160000-2407865.26.04]
librocm-dbgapi0/resolute,now 7.1.1-0ubuntu1 amd64 [Installiert,automatisch]
librocm-smi-dev/resolute,now 7.1.1-0ubuntu1 amd64 [Installiert,automatisch]
librocm-smi64-7/resolute,now 7.1.1-0ubuntu1 amd64 [Installiert,automatisch]
mesa-amdgpu-libgallium/now 1:26.1.3.26140000-2390957.26.04 amd64 [Installiert,aktualisierbar auf: 1:26.2.0.26160000-2407865.26.04]
mesa-amdgpu-va-drivers/now 1:26.1.3.26140000-2390957.26.04 amd64 [Installiert,aktualisierbar auf: 1:26.2.0.26160000-2407865.26.04]
rocm-dev/resolute,now 7.1.0-0ubuntu6 amd64 [Installiert,automatisch]
rocm-device-libs-21/resolute,now 7.1.1+dfsg-0ubuntu1 amd64 [Installiert,automatisch]
rocm-tests/resolute,now 7.1.0-0ubuntu6 amd64 [Installiert,automatisch]
rocm/resolute,now 7.1.0-0ubuntu6 amd64 [installiert]
rocminfo/resolute,now 7.1.1-0ubuntu1 amd64 [Installiert,automatisch]
whiptail/resolute,now 0.52.25-1ubuntu3 amd64 [Installiert,automatisch]

ROCm Component

No response

Steps to Reproduce

AI worloads, it does not matter with olama /llama / comfy UI

(Optional for Linux users) Output of rocminfo --support

Paste output here

Additional Information

kernel.txt
kernel-cmdline.txt
kernel-current-boot.txt
kernel-previous-boot.txt
lspci-full.txt
pcie-tree.txt
README-anonymization.txt
rocminfo.txt
rocm-smi.txt
ubuntu-version.txt
amd-packages.txt
bios.txt
dkms.txt