FUSE cold-read path: single-flight and async read-ahead, one BLAKE3 check (~1.9x); profile tuning alone ≤1.32x

Aperta
  • Qualcosa non va?
#684 10 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
8/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
linux, rust, sqlite
Ambito
backend, performance

Direzione di ricerca

This is a performance design thread, not a starter task. Read the report at core/docs/issues/fuse-profile-tuning/README.md and the receipts/ directory, then the CanonicalClient read path and the Store resolver where the BLAKE3 re-check happens. Done means a maintainer-agreed design for single-flight demands and a single BLAKE3 check, with the benchmark rerun on the same cold-read setup.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Summary

Diagnostic work on the 2.0 line (layerfs_v2_0_0 at 8cfb6e2734) asked where LayerFS's cold large-file read time goes, and how much of the FUSE I/O gap (cold read about 150 MB/s, against several hundred MB/s to GB/s for kernel overlay approaches) is profile tuning rather than design.

Answer: mostly design. Within the current contract (kernel write-back off), changing the FUSE profile alone gives at most 1.32×. Three read-path changes give about 1.9×.

Method: one sample per arm, Disposable Store, write-back off, fuser unpatched, Linux 6.12 aarch64 in the Docker VM. The guest page cache was evicted and verified before each cold read; the macOS host cache was not controlled, so throughput is diagnostic and request counts are exact.

Findings

1. Where a cold 500 MiB read goes (current profile: 128 KiB requests, 1 in flight)
Part Share of wall time
Synchronous 8 MiB read-ahead inside the READ request (one request in 64 waits ~42 ms) 75%
Hashing the same bytes three times: pack SHA-256, BLAKE3 in the Store resolver, BLAKE3 again in CanonicalClient 38% (overlaps the row above)
SQLite locate and pack reads 27%
Kernel round trip outside the callback 11%
Reply copy into /dev/fuse 7.6%
Overlay owner visit 1.3%
zstd decode ~0% (random data is stored raw)
2. Profile alone (request size × max_background, 9 arms)
  • Larger requests (512 KiB, 1 MiB) with one in flight: no cold-read gain (147–162 MB/s against 152). READ frames fall 4,000 → 500.
    • The kernel caps the INIT readahead at 128 KiB. Larger READs only appear after raising the mount's bdi read_ahead_kb.
  • More requests in flight (4/16) at 512 KiB / 1 MiB: +32%, but only by fetching every object twice (27k → 54k demands) and using 50–90% more callback time.
  • Writes:
    • Sequential: possibly +10–20%, within the noise of one sample.
    • 4 KiB random: unchanged.
  • Memory: receive buffers grow +0.75 MiB per mount at 512 KiB and +1.75 MiB at 1 MiB.
3. Read-path changes, quantified (round 2)

All arms of this round ran on the same CPU state. Control 206 MB/s.

Change 128 KiB / 1 512 KiB / 4
Control 206.0 279.9 (duplicate fetches)
One BLAKE3 check (skip the client re-check) 257.3 —
Single-flight (concurrent demands share one fetch) 219.9 386.0 (objects 53,771 → 27,214)
Asynchronous read-ahead (own thread, own Store custody) 283.4 278.0
Async + one BLAKE3 check 361–383 340.0
Single-flight + one BLAKE3 check — 392.5

How to read it:

  • Next ceiling: after these changes, the Workspace's single Store read session (one lock, one fetch thread) is busy 94% of the time. That caps the design at about 410 MB/s; going further needs parallel Store reads within one Workspace.
  • Async with more in flight: asynchronous read-ahead is slower with 4 in flight than single-flight alone, because it is a third claimant on that lock.
  • The failed run: the first run with async read-ahead at 512 KiB / 4 FAILED. Its background fetch reused the request's StorePorts, which refuses a concurrent demand, and the split-READ path turned the refusal into a short read. The fix uses StorePorts::fresh(); the failure and its diagnosis are kept in the receipts.
4. Small files and ls -l
  • FLUSH elision saves nothing. FLUSH already returns ENOSYS, which the kernel remembers per connection, so it is sent once per mount.
  • Creates: a create costs 5 requests. The GETATTR per create is about 20% of create time; I suspect parent re-validation under default_permissions, but that is not verified.
  • READDIRPLUS (estimate): ls -l of 2,000 entries would drop from 2,022 requests to about 84.

Proposed work (ranked)

  • Single-flight object demands in CanonicalClient: concurrent demands wait for an in-flight fetch instead of re-demanding.
  • Authenticate once per trust boundary. Drop the client BLAKE3 re-check of objects the in-process Store just authenticated (~370 ms per 500 MiB). Whether pack SHA-256 and object BLAKE3 are both needed is an owner decision.
  • Asynchronous read-ahead off the request path, over its own Store custody. Use it only while Store fetches are serialized per Workspace, or together with the next item.
  • Parallel Store reads within one Workspace (more than one read session or connection), the next ceiling.
  • Then revisit the profile: 512 KiB with 4 in flight. It needs:
    • a single-visit composed READ
    • a defined multi-window WRITE contract
    • a daemon-side bdi read_ahead_kb write after mount
    • a larger MAX_INPUT_BYTES
    • 0.75 MiB more receive buffer per mount
  • Small files:
    • Identify the GETATTR per create.
    • Evaluate adaptive READDIRPLUS.

Not done or not verified

  • Clippy: not run. The substitute rust:1.85.1-bookworm image has no Clippy; the registered build image was removed by another session's cleanup.
  • Existing mount tests: not run against the prototype.
  • Correctness proofs for the prototype read switches (concurrent writers, unmount during a background fetch, failure custody): not run.
  • Prototype limits: split READs make several owner visits and split WRITEs publish several windows before one reply. Neither is merge quality.
Lingua principale
Rust
Stelle
2.1k
Fork
110
Merge medio
23h 52m
PR unite (30g)
138

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di Ephemeral-AI-Lab/layerfs

Tutte le issue di Ephemeral-AI-Lab/layerfs

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.