[ZimaCube N100][1.8.0] Repeated hard lockups: NMI watchdog panic in pci_pme_list_scan -> pci_conf1_read (kernel 6.18.9); stable after rollback to 1.6.1

This is a kernel-level hard lockup in the PCI PME polling path (pci_pme_list_scan, pci_conf1_read) on one hardware platform, with the lock contention rooted in the kernel rather than ZimaOS scripts. Reproduction needs a ZimaCube N100 and the crash-dump archive, which the reporter keeps privately. Done would mean a confirmed regression cause or a cmdline/quirk workaround verified across reboots.

Summary

After updating a ZimaCube (Intel N100) from ZimaOS 1.6.1 to 1.8.0 the whole system hard-freezes every few hours: no network, no console, nothing written to the journal, and only a hard power cycle brings it back. Four freezes in 4.5 days (uptimes 18 h, 78 h, 6.5 h, 39 min).

With kernel.hardlockup_panic=1 enabled, the fourth freeze produced a crash dump in EFI pstore (archived by zimaos-crash-archive.service, which worked well). It shows a hard lockup on CPU 0 inside the PCI PME polling worker (pci_pme_list_scan → pci_check_pme_status → pci_conf1_read) spinning on the PCI config-space spinlock while another CPU was holding it inside a config-space access that never completed. That hangs the platform with interrupts off, which is why no freeze ever left a log line.

Rolling back to 1.6.1 with rauc status mark-active other is the current workaround. 1.6.1 (kernel 6.12.25) previously ran 38+ days on the same hardware and BIOS without a freeze.

Environment

Item	Value
Device	ZimaCube, IceWhale "ADL-N" board, Intel N100 (4 cores), 8 GB RAM, BIOS 5.27 (05/06/2024)
ZimaOS	v1.8.0 (installed 2026-10-24), kernel 6.18.9 #1 SMP PREEMPT_DYNAMIC Tue Apr 21 21:05:00 UTC 2026, systemd 258.7
Previous	v1.6.1, kernel 6.12.25, systemd 256.17 (no freezes)
Storage	Kingston OM8PGP4256Q-A0 NVMe (system + DATA, 05:00.0); 5× HDD RAID5 md0 (22 TB) on ASMedia ASM1166 (0d:00.0, root port 00:1d.0)
PCIe switches	ASMedia ASM2806 at 01:00.0 (downstream 02:00.0 / 02:02.0 / 02:06.0 / 02:0e.0 → 2× Intel I226-V igc 03:00.0 + 04:00.0, NVMe 05:00.0); second ASM2806 at 07:00.0 behind root port 00:1c.6 with four empty downstream ports 08:00.0 / 08:02.0 / 08:06.0 / 08:0e.0
Other	Display attached to the iGPU (i915), Realtek RTL8188EU USB Wi-Fi (unused), Docker apps, ZimaOS Photos (1.8.0), no VMs
Kernel cmdline	BOOT_IMAGE=(hd9,gpt2)/bzImage root=PARTUUID=… rootwait net.naming-scheme=v250 fsck.repair=yes console=tty1 quiet splash loglevel=3 systemd.show_status=1 rd.udev.log_level=3 net.ifnames=0 biosdevname=0 thunderbolt.host_reset=false rauc.slot=A

Timeline on 1.8.0 (local time, PDT)

Boot	Freeze	Uptime	Notes
10-24 13:33 (right after the 1.8.0 update)	10-25 07:41	18 h	journal just stops; next boot: "system.journal corrupted or uncleanly shut down", md0 "not clean"
10-25 11:11	10-28 17:24	78 h	same
10-28 17:54	10-29 00:26	6.5 h	same, /sys/fs/pstore empty afterwards
10-29 05:04	10-29 05:42	39 min	hardlockup_panic=1 was active → pstore dump (below). Automatic reboot after the panic did not come back (RAUC showed slot A "bad" afterwards, i.e. boot attempts exhausted); needed a power cycle 3 h later

Every freeze: the zimaos-user 10-second self-poll keeps its exact cadence to the last line, then silence. No OOM, no I/O errors, no thermal messages, no NIC hang messages in any of the four boots.

Crash evidence (EFI pstore, 2026-10-29 12:42:48 UTC)

[ 2350.849584] watchdog: CPU0: Watchdog detected hard LOCKUP on cpu 0
[ 2350.849588] Modules linked in: ... xe ... i915 ... igc ... raid456 ... zfs(PO) spl(O) ... iTCO_wdt ...
[ 2350.849633] CPU: 0 UID: 0 PID: 195 Comm: kworker/0:2 Tainted: P  O  6.18.9 #1 PREEMPT(voluntary)
[ 2350.849637] Hardware name: IceWhale Technology CO,.LTD ADL-N/ADL-N, BIOS 5.27 05/06/2024
[ 2350.849639] Workqueue: events_freezable pci_pme_list_scan
[ 2350.849643] RIP: 0010:native_queued_spin_lock_slowpath+0x67/0x2a0
[ 2350.849650] RAX: 0000000000000001 RBX: ffffffffaaf0a2b8 RCX: 0000000000000044
[ 2350.849652] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffffffffaaf0a2b8
[ 2350.849653] RBP: ffffceab80943ddc R08: 0000000000000002 R09: ffffceab80943ddc
[ 2350.849660] Call Trace:
[ 2350.849661]  <TASK>
[ 2350.849663]  _raw_spin_lock_irqsave+0x34/0x40
[ 2350.849666]  pci_conf1_read+0x45/0xf0
[ 2350.849670]  pci_bus_read_config_word+0x43/0x80
[ 2350.849672]  pci_check_pme_status+0x5e/0x90
[ 2350.849674]  pci_pme_list_scan+0x1a9/0x1e0
[ 2350.849676]  process_one_work+0x15a/0x320
[ 2350.849679]  worker_thread+0x2ca/0x410
[ 2350.849683]  kthread+0xf2/0x1e0
[ 2350.849691]  ret_from_fork+0x182/0x1b0
[ 2350.849695]  ret_from_fork_asm+0x1a/0x30
[ 2350.849699]  </TASK>
[ 2350.849700] Kernel panic - not syncing: Hard LOCKUP
[ 2351.884237] Shutting down cpus with NMI

Reading the argument registers (RSI = bus 1, RDX = devfn 0, RCX = reg 0x44, R8 = len 2): the poller was reading the Power Management Control/Status register of 0000:01:00.0 (ASM2806 upstream port) through the legacy CF8/CFC mechanism and could not take pci_config_lock. The lock holder (another CPU, inside a config access with IRQs disabled) never returned, so the whole machine stopped. Full dmesg of the crashed boot up to the panic is in the pstore records (attached).

Platform facts that look relevant

From the 6.18.9 boot log:

ACPI FADT declares the system doesn't support PCIe ASPM, so disable it
acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI EDR HPX-Type3]
acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug SHPCHotplug PME]
acpi PNP0A08:00: _OSC: OS now controls [AER PCIeCapability LTR DPC]
acpi PNP0A08:00: FADT indicates ASPM is unsupported, using BIOS configuration
igc 0000:03:00.0: can't disable ASPM; OS doesn't have ASPM control
igc 0000:04:00.0: can't disable ASPM; OS doesn't have ASPM control

From the 6.12.25 (1.6.1) boot log on the same box:

acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
acpi PNP0A08:00: _OSC: platform does not support [SHPCHotplug PME]
acpi PNP0A08:00: _OSC: OS now controls [PCIeCapability LTR]

So on this BIOS: no native PME (every sleeping device is polled by config reads every 2 s), no OS ASPM control (link states are whatever the BIOS set, and the igc driver's L1.2 workaround cannot apply), and 6.18 additionally takes native AER + DPC control (and requests EDR), which 6.12 did not. 1.8.0 users have also started reporting new AER "Correctable error" floods (forum thread "Correctable error message received").

Runtime-suspended PCI functions at the time (/sys/bus/pci/devices/*/power/runtime_status): 00:1c.6 (root port), 07:00.0 ASM2806 and all four empty downstream ports 08:00.0 / 08:02.0 / 08:06.0 / 08:0e.0, 02:0e.0, 00:16.0 mei, 00:1a.0 sdhci, 00:1e.0 / 00:1e.3 lpss, 00:1f.4 smbus. The same set is suspended on 6.12.25, so port runtime PM by itself is not new; the way 6.18 handles those sleeping ports (PME polling + native AER/DPC/EDR services, plus the 6.18 PCI PM/ASPM changes) is the difference to look at.

Ruled out before the dump was captured

• Memory (≥5 GB available, no OOM), thermal (no throttling messages), disks/RAID (clean, 31–39 °C), docker churn, cron/timers.
• Deep C-states: C6/C8/C10 disabled via cpuidle/state*/disable → still froze.
• USB Wi-Fi: rtl8xxxu blacklisted and unloaded → still froze.
• ZimaOS Photos: its searchd restarts happened shortly before freezes, but the trace is in the PCI core; it probably only adds background activity.
• Hardware watchdog (iTCO_wdt, armed via RuntimeWatchdogSec=60s): did not recover the box, because after the panic the reboot stalled before userspace.

Current workaround

rauc status mark-active other → back on 1.6.1 (kernel 6.12.25). Note: 1.6.1's kernel has CONFIG_HARDLOCKUP_DETECTOR, CONFIG_SOFTLOCKUP_DETECTOR and CONFIG_ITCO_WDT disabled, so if it ever locks up there will be no dump; and 1.6.1's systemd 256 cannot read the 1.8.0 journal files ("Protocol not supported"), so the 1.8.0 logs are only readable on systemd ≥ 258.

Suggestions / things to test on a ZimaCube N100 with 1.8.x

1. Reproduce with pcie_port_pm=off, with pcie_ports=compat (no native AER/DPC/hotplug/PME port services), with pci=noaer, and with all PCI devices held at power/control=on. Whichever prevents the recurrence narrows the regression (6.18 PCI PM/ASPM/PME changes vs. native AER/DPC on this firmware).
2. Consider a DMI quirk for the ZimaCube board (IceWhale Technology CO,.LTD ADL-N) if a cmdline workaround is confirmed.
3. Keep CONFIG_HARDLOCKUP_DETECTOR + kernel.hardlockup_panic=1 + zimaos-crash-archive in future releases: that combination is what finally produced usable evidence.
4. Investigate why the post-panic reboot (kernel.panic=30) did not complete on this firmware (EFI pstore write path?) and why RAUC then marked the slot bad.
5. Kernel 6.18.9 also shows ACPI BIOS Error (bug): Could not resolve symbol [\_SB.PC00.TXHC.RHUB.SS01/SS02] at boot (DSDT references a Thunderbolt xHCI hub that is not present). Harmless by itself but noted for completeness.

Attachments

Scrubbed log files (full kernel log of the crashed boot from the EFI pstore records, the journal tail before the freeze, PCI runtime-PM state on 6.18.9 vs 6.12.25, lspci) will be attached in a follow-up comment. The raw /DATA/.crash pstore archive and the raw systemd journal files of all 1.8.0 boots are kept and can be provided privately on request.