[Bug]: Native macOS NFS mount fails `cp` with `Operation not permitted` once 4,096 AppleDouble companions accumulate (companion table cap)

The issue reports a companion table limit in engine/crates/layerfs-nfs/src/apple_double.rs and links PR #636, which implements the proposed fix. Read that file and the pull request to understand the change; the report says the fix was verified with 30,000 copies and retained all tested attributes. The open question about persisting xattrs spans several crates and needs maintainer direction.

Description

When copying files within a native macOS loopback NFS workspace (e.g. running cp -r or repeated cp commands), once the workspace accumulates 4,096 files carrying extended attributes, all subsequent cp operations fail with:

cp: could not copy extended attributes to <destination>: Operation not permitted

The command terminates with exit code 1.

This behavior:

• Depends on the number of accumulated companion files, not on file size: with the table below its cap, 1 MiB, 8 MiB and 64 MiB sources each copied 50/50 times and 6 concurrent copies ran 300/300, all with 0 failures;
• Affects any file where macOS automatically attaches metadata (including com.apple.provenance, which the macOS kernel assigns to files on host APFS filesystems);
• Is not a regression introduced by #604–#607 (independently reproduced on clean baseline commit aa131e394);
• Has two further caps at baseline: each companion body is limited to FILE_BYTES = 1 MiB (NFS3ERR_FBIG), and all bodies together to TOTAL_BYTES = 64 MiB (NFS3ERR_NOSPC), which a 100 KiB xattr reaches at the 631st copy, before the count cap.

---

Steps to Reproduce

1. Initialize a project and workspace on the native macOS machine (NFS route, SQLite store, Disposable profile):

   layerfs store create main
   layerfs machine update native --daemon "$BIN/layerfs-daemon"
   layerfs project init main/proj ./src
   layerfs branch fork main/proj work main
   layerfs sandbox create main/proj s1 native
   layerfs workspace mount s1 work
   

2. In the mounted directory, sequentially copy a file carrying extended attributes (or any file created on host APFS):

   cd /tmp/mnt
   for i in $(seq 1 4200); do
       cp /path/to/host/file "file_$i" || { echo "Failed at $i"; break; }
   done
   

3. Observe:
   • Copies 1 through 4,095 succeed without error.
   • At copy 4,096, cp fails immediately with Operation not permitted (exit code 1).
   • Every subsequent copy (4,097, 4,098, ...) continues to fail (100% failure rate).
   • If an existing file is removed (rm file_1), exactly one subsequent cp is permitted before failing again.

---

Root Cause Analysis

1. AppleDouble Sidecars on NFS: On NFSv3 the macOS XNU client keeps a file's extended attributes and resource fork in a ._<name> companion file instead of a protocol-level xattr call (engine/crates/layerfs-nfs/src/apple_double.rs:1-12), so copying a file with attributes into the workspace creates ._file_1 and so on. LayerFS keeps those names out of the Workspace and in the export's own table.
2. Hardcoded Server Companion Limit: Inside engine/crates/layerfs-nfs/src/apple_double.rs, the daemon manages companion files in a private per-export table:

   /// Companion index entries held at once.
   pub const HELD: usize = 4096;             // line 41 at 716197c49
   pub const FILE_BYTES: u64 = 1 << 20;      // line 43
   pub const TOTAL_BYTES: usize = 64 << 20;  // line 45
   
   pub(crate) struct Companions {
       ...
       held_limit: usize,   // initialized from HELD
   }
   

3. Error Code Translation: When the companion table reaches the cap, CREATE ._<name> is refused with NFS3ERR_ACCES (13). The macOS kernel hands ACCES to user space as EPERM (1). /bin/cp then fails in its extended-attribute phase and exits 1. At the count cap the file content was already written (a 4 KiB source landed with a matching sha256 and an empty xattr -l), but a source carrying com.apple.quarantine was observed to leave a 0-byte destination when the same refusal happened, so the data outcome is not uniform.
4. RPC Verification: Using a transparent loopback ONC RPC proxy, we captured the exact RPC traffic at the failure point. The server directly returns status code 13 (NFS3ERR_ACCES) for CREATE ._<filename>, confirming the failure originates from the server table cap rather than client-side heuristics.

---

Fix Implementation & Trade-offs

Why In-Memory LRU Eviction Was Rejected

An initial prototype attempted to evict the least-recently-used (LRU) companion entry when reaching the 4,096 cap. Across 4,300 sequential copies it kept every copy succeeding, but 208 of the 4,300 4 KiB files (and 204 of the 4,300 1 MiB files) silently lost their extended attributes — the boundary was exact (a-00000…a-00207) — so the prototype was dropped.

Chosen Solution: Memory Index + Disk Spilling

We implemented a disk-spilling architecture in https://github.com/Ephemeral-AI-Lab/layerfs/pull/636:

1. Compact In-Memory Index: In-memory metadata tracks all active companion records (serial, parent inode, name hash, size, file flags).
2. Disk Spilling: Companion bodies beyond a 32 MiB resident budget (RESIDENT_BYTES = 32 << 20) are spilled to one file per companion under the export's own scratch directory (<mounts>/.appledouble/<namespace>, directory 0700, files 0600), written as <serial>.<n>.tmp and renamed on adoption; no descriptor is kept open for a spilled body.
3. Increased Capacity: Raised the per-export companion cap from 4,096 to 65,536 entries (HELD = 65_536; index ≈294 B per entry, ≈20 MB per export), and removed the 64 MiB global TOTAL_BYTES cap in favour of the 32 MiB spill budget. FILE_BYTES = 1 MiB per companion is unchanged.
4. Lifecycle Linkage & Orphan Reclamation:
   • Primary file deletion (REMOVE, RMDIR, overwrite RENAME, silly rename) cascades to immediately release associated companion index slots and delete spilled disk files.
   • An orphan garbage collector (reclaim_orphans) scans and reclaims unreferenced companions when approaching limits.
   • On daemon startup, stale companion directories from prior abnormal terminations are purged.
5. Two Concurrency & Link Boundary Fixes:
   • Bug A (Duplicate Serial Race): Two concurrent CREATE requests for the same companion name briefly released locks, creating two active serials where one orphaned slot became un-reclaimable. Fixed by consolidating name checking, capacity checks, orphan reclamation, and insertion into a single critical section.
   • Bug B (Hard Link Reclamation): The server previously tracked only the most recent filename per inode in by_name. When hard links existed, removing one alias caused the companion for remaining aliases to be misclassified as an orphan and discarded. Fixed by retaining all seen directory-entry aliases.
6. Error Code & Logging: When reaching the expanded 65,536 limit, the server returns NFS3ERR_NOSPC (28) instead of ACCES, and logs a rate-limited warning to stderr (throttled to at most once per 60 seconds per export).

---

Verification & Test Results

Tested on MacBook Air M2 (macOS 27.2) and Linux:

Metric / Scenario	Baseline (716197c49 / aa131e394)	LRU Eviction Prototype	Spilling Fix (https://github.com/Ephemeral-AI-Lab/layerfs/pull/636)
Sequential Copies (4 KiB files)	Fails from No. 4,096 on (243/243 attempts after the cap failed; the accumulating run was 208/208)	4,300/4,300 passed	4,300/4,300 passed (0 failed)
Sequential Copies (1 MiB files)	Fails from No. 4,096 on	4,300/4,300 passed	4,300/4,300 passed (0 failed)
Attribute Retention (full scan after the 60 s acregmin cache expired)	N/A (copy failed)	208 (4 KiB) / 204 (1 MiB) files lost xattrs	4,300/4,300 and 4,300/4,300 kept, 0 lost
30,000 Continuous Copy Endurance (cp exit codes + non-OK CREATE ._* only)	N/A	Not tested	30,000/30,000 passed, 0 non-OK CREATE ._* replies
Spilled Payload Volume	0 B	0 B	29,394 files / 1,587,276 KiB (≈1.5 GB) after the 30,000-copy run
File Descriptor Hygiene	Normal	Normal	lsof -p constant at 26 lines (12–25 fds) under ulimit -n 256 while spills grew to 4,522 (measured on the previous commit ca153a7f6)
Linux Unit Tests (ulimit -n 1024)	—	Not tested	layerfs-nfs 32/32 passed (including the unit test that lowers RLIMIT to 512 and creates 2,048 spilled companions)
Linux non-mount daemon tests	—	Not tested	layerfs-daemon 51/51 passed (3 ignored)
Merge with Upstream 833d04cf4	—	N/A	0 conflicts via git merge-tree; layerfs-nfs 32/32 on the merged tree

---

Known Limitations

1. Mount-Scoped Ephemeral Metadata: Companion data remains ephemeral to the active mount session. LayerFS Commit continues to persist only POSIX mode and mtime into the global Store; AppleDouble companions are not committed.
2. Client-Side Error Mapping at Capacity: When the 65,536 ceiling is reached, the server returns NFS3ERR_NOSPC (28). However, the macOS XNU client still presents this to user space as EPERM during /bin/cp. The advantage of NOSPC is clean protocol semantics and rate-limited server logging.
3. Abnormal Termination Recovery: If the daemon process is forcefully killed (kill -9), uncleaned spill files remain on host disk until the workspace sandbox is closed or the mount is re-established. The startup purge only runs when a daemon is started on the same mounts again.
4. False-Positive EPERM under Concurrent Scanning:
   • In a fresh workspace holding ~4,100 companions (6% of the cap), running background stat or getxattr loops against the destination folder concurrently with /bin/cp produced 846 EPERM failures in 2,000 copies; the same load without the scanner was 0/2,000.
   • A user-space RPC proxy confirmed the server answered NFS3_OK for all 230,576 RPC calls in that window. Reading the 846 failed destinations back: 821 had both data and extended attributes correctly on disk, and 25 were really missing the 100 KiB com.example.big (the earliest runs).
   • A separate pure-setxattr group that failed 422 times did hit the cap (65,158 prefilled companions + 378 successes = 65,536), so that group is the capacity path, not a client race.
   • The concurrent-scan error is reported by the macOS client; the server side returns success.
5. Remaining Count Ceiling: 65,536 companions means roughly 65,000 attributed files per mount. Past that, the copied file still lands but its xattrs fail (the client reports EPERM).
6. Per-Companion Size Cap Unchanged: Each companion body is still limited to FILE_BYTES = 1 MiB (NFS3ERR_FBIG). A file whose AppleDouble data (for example a large resource fork) exceeds 1 MiB still fails to copy its attributes. This PR neither changes nor tests that path.

---

Open Architectural Question

Should LayerFS natively parse ._* AppleDouble files into real extended attributes and persist them in the Store across Commits?

• Current Architecture:
  • The LayerFS Store model (InodeValue) stores the file kind, the namespace ref count, a content root and a metadata root; the portable metadata carries only mode and mtime, and the length lives with the content side. There is no xattr field.
  • The Linux FUSE route returns ENOSYS for the extended-attribute family (getxattr, setxattr, listxattr), sticky per connection.
  • AppleDouble companions on NFS are kept outside the Workspace: they never reach the inode table, READDIR, or a Commit.
• Estimated Scope for Native xattrs:
  • Requires defining xattr serialization in layerfs-content / layerfs-storage.
  • Requires metadata table schema updates in layerfs-persistence.
  • Requires implementing xattr handlers across both layerfs-fuse and layerfs-nfs.
  • Estimated implementation size is ~800–1,200+ LOC and introduces a store schema format evolution.
• Request: We invite maintainer feedback on whether the current ephemeral spilling approach (expanded to 65,536 files) is sufficient for agent workloads, or if true committed xattrs should be planned on the LayerFS roadmap.