FUSE cold-read path: single-flight and async read-ahead, one BLAKE3 check (~1.9x); profile tuning alone ≤1.32x

This is a performance design thread, not a starter task. Read the report at core/docs/issues/fuse-profile-tuning/README.md and the receipts/ directory, then the CanonicalClient read path and the Store resolver where the BLAKE3 re-check happens. Done means a maintainer-agreed design for single-flight demands and a single BLAKE3 check, with the benchmark rerun on the same cold-read setup.

Summary

Diagnostic work on the 2.0 line (layerfs_v2_0_0 at 8cfb6e2734) asked where LayerFS's cold large-file read time goes, and how much of the FUSE I/O gap (cold read about 150 MB/s, against several hundred MB/s to GB/s for kernel overlay approaches) is profile tuning rather than design.

Answer: mostly design. Within the current contract (kernel write-back off), changing the FUSE profile alone gives at most 1.32×. Three read-path changes give about 1.9×.

• Side branch, not for merge: issue-fuse-profile-tuning
• Report: core/docs/issues/fuse-profile-tuning/README.md, with raw receipts in receipts/.

Method: one sample per arm, Disposable Store, write-back off, fuser unpatched, Linux 6.12 aarch64 in the Docker VM. The guest page cache was evicted and verified before each cold read; the macOS host cache was not controlled, so throughput is diagnostic and request counts are exact.

Findings

1. Where a cold 500 MiB read goes (current profile: 128 KiB requests, 1 in flight)

2. Profile alone (request size × max_background, 9 arms)

• Larger requests (512 KiB, 1 MiB) with one in flight: no cold-read gain (147–162 MB/s against 152). READ frames fall 4,000 → 500.
  • The kernel caps the INIT readahead at 128 KiB. Larger READs only appear after raising the mount's bdi read_ahead_kb.
• More requests in flight (4/16) at 512 KiB / 1 MiB: +32%, but only by fetching every object twice (27k → 54k demands) and using 50–90% more callback time.
• Writes:
  • Sequential: possibly +10–20%, within the noise of one sample.
  • 4 KiB random: unchanged.
• Memory: receive buffers grow +0.75 MiB per mount at 512 KiB and +1.75 MiB at 1 MiB.

3. Read-path changes, quantified (round 2)

All arms of this round ran on the same CPU state. Control 206 MB/s.

How to read it:

• Next ceiling: after these changes, the Workspace's single Store read session (one lock, one fetch thread) is busy 94% of the time. That caps the design at about 410 MB/s; going further needs parallel Store reads within one Workspace.
• Async with more in flight: asynchronous read-ahead is slower with 4 in flight than single-flight alone, because it is a third claimant on that lock.
• The failed run: the first run with async read-ahead at 512 KiB / 4 FAILED. Its background fetch reused the request's StorePorts, which refuses a concurrent demand, and the split-READ path turned the refusal into a short read. The fix uses StorePorts::fresh(); the failure and its diagnosis are kept in the receipts.

4. Small files and ls -l

• FLUSH elision saves nothing. FLUSH already returns ENOSYS, which the kernel remembers per connection, so it is sent once per mount.
• Creates: a create costs 5 requests. The GETATTR per create is about 20% of create time; I suspect parent re-validation under default_permissions, but that is not verified.
• READDIRPLUS (estimate): ls -l of 2,000 entries would drop from 2,022 requests to about 84.

Proposed work (ranked)

• Single-flight object demands in CanonicalClient: concurrent demands wait for an in-flight fetch instead of re-demanding.
• Authenticate once per trust boundary. Drop the client BLAKE3 re-check of objects the in-process Store just authenticated (~370 ms per 500 MiB). Whether pack SHA-256 and object BLAKE3 are both needed is an owner decision.
• Asynchronous read-ahead off the request path, over its own Store custody. Use it only while Store fetches are serialized per Workspace, or together with the next item.
• Parallel Store reads within one Workspace (more than one read session or connection), the next ceiling.
• Then revisit the profile: 512 KiB with 4 in flight. It needs:
  • a single-visit composed READ
  • a defined multi-window WRITE contract
  • a daemon-side bdi read_ahead_kb write after mount
  • a larger MAX_INPUT_BYTES
  • 0.75 MiB more receive buffer per mount
• Small files:
  • Identify the GETATTR per create.
  • Evaluate adaptive READDIRPLUS.

Not done or not verified

• Clippy: not run. The substitute rust:1.85.1-bookworm image has no Clippy; the registered build image was removed by another session's cleanup.
• Existing mount tests: not run against the prototype.
• Correctness proofs for the prototype read switches (concurrent writers, unmount during a background fetch, failure custody): not run.
• Prototype limits: split READs make several owner visits and split WRITEs publish several windows before one reply. Neither is merge quality.