Slow ledger fsync or chunk creation blocks `GET /node/ledger_chunk` on `Ledger::state_lock`, which can use up the worker threads and stop the node answering any request

Start in src/host/ledger.h, where Ledger::commit(), LedgerFile::commit(), Ledger::write_entry() and committed_ledger_path_with_idx() all take state_lock around fsync and chunk creation. Then read the ledger_chunk handler in src/node/rpc/file_serving_handlers.h, which calls the lookup synchronously on a worker thread. Done means a delayed fsync no longer makes /node/version or a ledger_chunk request block all workers. The issue lists three candidate fixes, so the design choice needs maintainer input first.

Describe the bug

Since 7.0.19, ledger writes no longer go through the host-enclave ringbuffer (#8405), and ledger file I/O runs on the task worker threads. This works as intended for a slow disk on its own. When one ledger fsync or chunk open is blocked for 30 s, a 7.0.23 node keeps answering /node/version and /node/state, while a 7.0.17 node stops answering completely.

One case still freezes the node:

• Ledger::commit() takes state_lock (ledger.h#L2023-L2051), and LedgerFile::commit() then fsyncs the completed chunk while the lock is held (ledger.h#L1086-L1105). The comment there says this is deliberate, so that committed_ledger_path_with_idx() only returns complete files. Ledger::write_entry() also holds the lock (ledger.h#L1845-L1850) while it creates the next chunk file.
• The handler for GET /node/ledger_chunk?since=N (file_serving_handlers.h#L1178-L1179) calls committed_ledger_path_with_idx() synchronously on a worker thread (file_serving_handlers.h#L803-L804). That function takes state_lock (ledger.h#L2078-L2081).

So while an fsync is slow, every ledger_chunk request holds a worker thread until the fsync finishes. With the default worker_threads: 1 (two task workers), one request is enough: the other worker is the one running the fsync. No worker is left, and the node stops answering every endpoint, including /node/version, until the fsync returns. With worker_threads: 3 (four workers), the node froze when the third concurrent ledger_chunk request arrived.

In Kubernetes, a liveness probe on a /node endpoint then restarts a node that is only waiting for slow storage. Any client that polls ledger_chunk, for example to archive committed chunks, makes this likely during storage latency spikes.

To Reproduce

1. Start a 3-node network with the release logging sample app and the devel RPM (virtual build), --ledger-chunk-bytes 5KB and the default worker_threads: 1. Send about 20 POST /app/log/public per second.
2. On one node, delay the next ledger fsync by 30 s: strace -f -p <cchost pid> -e trace=fsync,openat -e inject=fsync:delay_enter=30s:when=1. The node log confirms it: Operation took too long ( 30.02s): Committing ledger file - fsync(ledger_609).
3. Poll GET /node/version and GET /node/state on that node every 0.5 s, with a 2 s timeout. All of them succeed during the delay.
4. Repeat, and during the delay send one GET /node/ledger_chunk?since=100 to the same node. From that request until the fsync returns, every poll times out. The ledger_chunk request itself takes about 27 s.

Expected behavior

A slow ledger fsync shouldn't stop the node serving endpoints that don't need the ledger, and /node endpoints should stay responsive.

Environment information

• CCF 7.0.23. src/host/ledger.h is identical on main at 9cdc86cc78a42ee27ea167a7a347c9d924bf4841.
• Reproduced with the virtual build in Azure Linux 3.0 containers. Production runs C++ applications on the SNP platform (confidential Azure Container Instances), with the ledger on an SMB file share, where single fsync or file-create calls sometimes take 40-55 s.

Frequency

Every time a ledger_chunk request reaches a node while its ledger fsync or chunk creation is blocked. In our deployment that depends on storage latency spikes, which happen several times a week on some file shares.

Additional context / possible fixes

1. Don't hold state_lock for the fsync in Ledger::commit(). For example, mark the file as being committed under the lock, fsync and rename it outside the lock, then publish it as committed under the lock. committed_ledger_path_with_idx() would only return published files, which keeps today's guarantee.
2. Or don't make ledger_chunk lookups wait for an in-progress commit. For example, try the lock and answer 503 with Retry-After when it's held, or keep the committed-chunk index under a separate lock.
3. Or serve ledger_chunk, and other readers of state_lock such as historical reads of committed ranges, without blocking a worker thread.

Related (not necessarily a bug): with 7.0.23, a primary whose ledger fsync or chunk create stalls loses leadership within about 3 s, although its HTTP endpoints stay up. Outbound node-to-node messages, including heartbeats, wait for all ledger mutations submitted before them (node_connections.h#L413-L423, ledger_subsystem.h#L210-L221). So a storage latency spike on the primary still causes an election.