[1.7.1][Bug] Btrfs COW amplification exhausts transaction reserve, forces Main Storage read-only and stops Docker

Start with the ZimaOS 6.18.9 kernel build and the fs/btrfs/delayed-inode.c trace, then compare it with upstream commits f9a48549a15aa369d42cebc08a6a72b71a53d547 and 67bd829a14b7d71ffea28147b98dadf9ee684568. Done means confirming whether both commits or an equivalent backport are present and documenting the appropriate kernel upgrade or backport path.

Bug report: Btrfs transaction-reserve exhaustion forces Main Storage read-only and stops Docker

Summary

On a ZimaCube running ZimaOS v1.7.1, the Btrfs filesystem on the healthy Main Storage RAID repeatedly aborts a transaction with errno=-28 (No space left) and forces the filesystem read-only, even though multiple terabytes remain available.

Once Main Storage becomes read-only, Docker can no longer write its volume/network metadata. The Docker daemon exits and every container becomes unavailable. Rebooting temporarily restores service, but the same failure returns later.

The kernel trace and Btrfs reservation state closely match the upstream issue described as COW amplification during btrfs_search_slot(), where the transaction block reserve can be exhausted under memory/writeback pressure despite ample unallocated space:

https://www.spinics.net/lists/linux-btrfs/msg162178.html

This has now happened at least twice with the same kernel signature.

---

Severity

Critical / data-service outage

• Main Storage is forced read-only.
• Docker fails completely.
• All containers and container-backed websites/services become unavailable.
• A reboot is required to restore operation temporarily.
• The issue is recurring.

No filesystem repair or remount was attempted after the abort; the system was left read-only until a controlled reboot.

---

Environment

• Hardware: ZimaCube
• OS: ZimaOS v1.7.1
• Build date: 2026-08-21 08:17:47 UTC
• Build classification from /etc/os-release: development
• Kernel: Linux 6.18.9 #4 PREEMPT(voluntary)
• Architecture: amd64
• Storage: mdadm RAID5 (md0) with 7 active members
• Filesystem: Btrfs on /dev/md0
• Mount point: /media/Main-Storage
• Docker data root: /media/Main-Storage/docker
• Workload: multiple Docker containers, including PostgreSQL databases and browser-based services

Relevant /etc/os-release values:

NAME="ZimaOS"
BOARD="ZimaCube"
VERSION="v1.7.1"
VERSION_ID=1.7.1
PRETTY_NAME="ZimaOS v1.7.1"
CPE_NAME=cpe:2.3:o:icewhale:zimaos:1.7.1:*:development:*:*:*:zimacube:*
BUILD_DATE="2026/08/21 08:17:47 UTC"

---

Recurrence

The same failure was found in persistent kernel journals on two separate boots:

1. 2026-09-14 00:01:03 CEST
2. 2026-09-21 07:13:46 CEST

Both events show:

• postgres as the process performing the final metadata operation;
• warning at fs/btrfs/delayed-inode.c:1572;
• btrfs_insert_delayed_dir_index();
• btrfs_add_link() and btrfs_rename();
• transaction abort with error -28;
• empty trans_block_rsv;
• fully reserved global_block_rsv;
• metadata space still reported as not full;
• filesystem forced read-only.

PostgreSQL appears to be the final triggering writer, not necessarily the root cause. The upstream report demonstrates the same failure from ordinary file creation/rename under memory/writeback pressure.

---

Actual result

Btrfs aborts the current transaction and changes Main Storage to read-only:

BTRFS: Transaction aborted (error -28)
BTRFS: error (device md0 state A) in btrfs_add_link:6752: errno=-28 No space left
BTRFS info (device md0 state EA): forced readonly
BTRFS: error (device md0 state EA) in btrfs_rename:8624: errno=-28 No space left

Docker then fails because its data root is on that filesystem:

failed to start daemon: error while opening volume store metadata database
(/media/Main-Storage/docker/volumes/metadata.db):
attempt to write a readonly database

All published container ports stop responding, while the ZimaOS host and its non-Docker web interface remain reachable.

---

Expected result

• Btrfs must not report ENOSPC or abort a transaction while substantial data and metadata allocation capacity remains.
• Temporary transaction-reserve exhaustion must not force the entire filesystem read-only under a normal container workload.
• If this is a known kernel defect, ZimaOS should ship a kernel containing the applicable upstream fix/backport.
• ZimaOS should alert clearly when Main Storage is remounted read-only instead of only showing all applications as stopped.

---

Evidence that the array/filesystem was not actually full

At the time of diagnosis after the failure:

Filesystem      Size  Used Avail Use%
/dev/md0        6.0T  1.8T  3.8T  33%

Btrfs allocation summary:

Device size:                 5.46 TiB
Device allocated:            1.72 TiB
Device unallocated:          4.10 TiB
Data, single:                1.70 TiB
Metadata, DUP:               6.00 GiB
Metadata used:               approximately 5.72 GiB

RAID state:

md0 : active raid5 ...
      [7/7] [UUUUUUU]

Therefore, this is not conventional data-space exhaustion and not a degraded RAID event.

---

Kernel trace from the most recent occurrence

2026-09-21T07:13:46+02:00 ZimaOS kernel: WARNING: CPU: 3 PID: 12554 at fs/btrfs/delayed-inode.c:1572 btrfs_insert_delayed_dir_index+0x1e0/0x200
2026-09-21T07:13:46+02:00 ZimaOS kernel: CPU: 3 UID: 999 PID: 12554 Comm: postgres Tainted: P S         O        6.18.9 #4 PREEMPT(voluntary)
2026-09-21T07:13:46+02:00 ZimaOS kernel: btrfs_add_link+0x98/0x460
2026-09-21T07:13:46+02:00 ZimaOS kernel: btrfs_rename+0x627/0xcf0
2026-09-21T07:13:46+02:00 ZimaOS kernel: btrfs_rename2+0x25/0x60
2026-09-21T07:13:46+02:00 ZimaOS kernel: BTRFS: Transaction aborted (error -28)
2026-09-21T07:13:46+02:00 ZimaOS kernel: BTRFS info (device md0 state A): space_info DATA (sub-group id 0) has 17780453376 free, is not full
2026-09-21T07:13:46+02:00 ZimaOS kernel: BTRFS info (device md0 state A): space_info total=1871540387840, used=1853759741952, pinned=0, reserved=208896, may_use=741376, readonly=65536 zone_unusable=0
2026-09-21T07:13:46+02:00 ZimaOS kernel: BTRFS info (device md0 state A): space_info METADATA (sub-group id 0) has 181096448 free, is not full
2026-09-21T07:13:46+02:00 ZimaOS kernel: BTRFS info (device md0 state A): space_info total=6442450944, used=5720768512, pinned=5013504, reserved=1114112, may_use=534511616, readonly=65536 zone_unusable=0
2026-09-21T07:13:46+02:00 ZimaOS kernel: BTRFS info (device md0 state A): global_block_rsv: size 536870912 reserved 536870912
2026-09-21T07:13:46+02:00 ZimaOS kernel: BTRFS info (device md0 state A): trans_block_rsv: size 3538944 reserved 0
2026-09-21T07:13:46+02:00 ZimaOS kernel: BTRFS info (device md0 state A): delayed_block_rsv: size 720896 reserved 720896
2026-09-21T07:13:46+02:00 ZimaOS kernel: BTRFS: error (device md0 state A) in btrfs_add_link:6752: errno=-28 No space left
2026-09-21T07:13:46+02:00 ZimaOS kernel: BTRFS info (device md0 state EA): forced readonly
2026-09-21T07:13:46+02:00 ZimaOS kernel: BTRFS: error (device md0 state EA) in btrfs_rename:8624: errno=-28 No space left

---

Previous occurrence showing the same signature

2026-09-14T00:01:03+02:00 ZimaOS kernel: WARNING: CPU: 2 PID: 880737 at fs/btrfs/delayed-inode.c:1572 btrfs_insert_delayed_dir_index+0x1e0/0x200
2026-09-14T00:01:03+02:00 ZimaOS kernel: CPU: 2 UID: 999 PID: 880737 Comm: postgres Tainted: P S         O        6.18.9 #4 PREEMPT(voluntary)
2026-09-14T00:01:03+02:00 ZimaOS kernel: BTRFS: Transaction aborted (error -28)
2026-09-14T00:01:04+02:00 ZimaOS kernel: BTRFS info (device md0 state A): space_info DATA (sub-group id 0) has 18867826688 free, is not full
2026-09-14T00:01:04+02:00 ZimaOS kernel: BTRFS info (device md0 state A): space_info METADATA (sub-group id 0) has 320733184 free, is not full
2026-09-14T00:01:04+02:00 ZimaOS kernel: BTRFS info (device md0 state A): space_info total=6442450944, used=5580685312, pinned=3424256, reserved=147456, may_use=537395200, readonly=65536 zone_unusable=0
2026-09-14T00:01:04+02:00 ZimaOS kernel: BTRFS info (device md0 state A): global_block_rsv: size 536870912 reserved 536870912
2026-09-14T00:01:04+02:00 ZimaOS kernel: BTRFS info (device md0 state A): trans_block_rsv: size 3407872 reserved 0
2026-09-14T00:01:04+02:00 ZimaOS kernel: BTRFS: error (device md0 state A) in btrfs_add_link:6752: errno=-28 No space left
2026-09-14T00:01:04+02:00 ZimaOS kernel: BTRFS info (device md0 state EA): forced readonly

---

Comparison with upstream Btrfs report and merged fix

The upstream report describes filesystems receiving ENOSPC while having more than 100 GB unallocated. Its sample trace contains the same key sequence:

btrfs_insert_delayed_dir_index
btrfs_add_link
Transaction aborted (error -28)
space_info METADATA ... is not full
global_block_rsv ... fully reserved
trans_block_rsv ... reserved 0
forced readonly

The reported mechanism is repeated COW of the same B-tree path during btrfs_search_slot() under sustained/global memory pressure. This invalidates the assumption that the transaction reservation represents the worst-case requirement, allowing the reserve to become exhausted.

The corresponding fix was subsequently merged upstream:

1. f9a48549a15aa369d42cebc08a6a72b71a53d547 — btrfs: inhibit extent buffer writeback to prevent COW amplification
   https://github.com/torvalds/linux/commit/f9a48549a15aa369d42cebc08a6a72b71a53d547
2. 67bd829a14b7d71ffea28147b98dadf9ee684568 — btrfs: replace writeback inhibition xarray with a fixed inline buffer
   https://github.com/torvalds/linux/commit/67bd829a14b7d71ffea28147b98dadf9ee684568

The second commit is an important refinement of the first: it removes a reported write-IOPS regression and an unbounded cleanup walk. Therefore, backporting only the original xarray implementation is not recommended; IceWhale should use the final upstream implementation or upgrade to a kernel that contains both commits.

Please confirm whether ZimaOS kernel 6.18.9 contains these changes or an equivalent backport. The runtime behavior strongly indicates that this build remains affected.

Existing ZimaOS issue search

No exact duplicate was found among open or closed IceWhaleTech/ZimaOS issues or pull requests. The closest reports are related but technically different:

• #560: recurring read-only storage and Docker failures on kernel 6.18.9, but on ext4 with EROFS/EIO journal failures;
• #585: Btrfs/md-RAID on 1.7.1, but a boot-time btrfs_bio_counter_sub() NULL-dereference panic;
• #601: a duplicate storage-database row that only makes the UI report read-only while the filesystem remains rw;
• #326: deleted space held by icewhale-files-backup, without a Btrfs transaction abort or forced-readonly transition.

This report therefore describes a separate kernel-level failure and should be tracked as a new issue.

---

Steps to reproduce

The failure is intermittent and has not been intentionally reproduced because this is a production NAS and the result forces the primary filesystem read-only.

Observed conditions:

1. Start ZimaOS v1.7.1 with Btrfs Main Storage on md0.
2. Run a normal multi-container workload for several days.
3. Include concurrent metadata-heavy activity, PostgreSQL checkpoints/renames, and periods of memory/writeback pressure.
4. Eventually a directory-index insertion/rename exhausts trans_block_rsv.
5. Btrfs aborts the transaction and forces md0 read-only.
6. Docker subsequently exits because its data root is no longer writable.

The two observed incidents occurred about one week apart.

---

Temporary workaround

A controlled reboot remounts Main Storage read-write and allows Docker to start again. This only clears the transient reservation/error state; it does not prevent recurrence.

A direct read-write remount was deliberately not attempted after the aborted transaction.

---

Additional hardware observation

There is a separate storage-link warning that should be investigated independently. The kernel logged repeated ATA interface errors for one RAID member, a Crucial CT1000BX500SSD1 exposed as sdb:

failed command: READ FPDMA QUEUED
Emask 0x10 (ATA bus error)
status: { DRDY DF ERR }
error: { ICRC ABRT }
SATA link down
hard resetting link
limiting SATA link speed to 3.0 Gbps

The drive repeatedly reconnects and mdadm continues to report [UUUUUUU]. The ICRC, bus-reset, and speed-downgrade pattern suggests a SATA/backplane/power/device-link issue rather than proving Btrfs data-space exhaustion.

This hardware warning should not obscure the separate reproducible Btrfs failure signature: both read-only events show transaction-reserve exhaustion with substantial filesystem capacity still available and match the upstream Btrfs report closely.

---

Requested action from IceWhale

Please:

1. Confirm whether this is a known issue in ZimaOS v1.7.1 / kernel 6.18.9.
2. Confirm whether the upstream btrfs_search_slot() COW-amplification fix is present in this kernel build.
3. Provide a patched kernel or safe ZimaOS update/backport.
4. Advise whether upgrading or rolling back to a specific stable ZimaOS version is the recommended mitigation.
5. Add a prominent ZimaOS alert when a storage filesystem is forced read-only.
6. Consider preventing Docker restart loops after its data root becomes read-only, while preserving diagnostic logs.
7. Advise which diagnostic bundle or full journal should be supplied for further analysis.

Because this is recurring and takes every Docker service offline, a kernel-level resolution or supported rollback path is required; rebooting is not an adequate permanent workaround.