Slow ledger fsync or chunk creation blocks `GET /node/ledger_chunk` on `Ledger::state_lock`, which can use up the worker threads and stop the node answering any request

Aperta
  • Qualcosa non va?
#8,655 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
8/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
cpp
Ambito
distributed-systems

Direzione di ricerca

Start in src/host/ledger.h, where Ledger::commit(), LedgerFile::commit(), Ledger::write_entry() and committed_ledger_path_with_idx() all take state_lock around fsync and chunk creation. Then read the ledger_chunk handler in src/node/rpc/file_serving_handlers.h, which calls the lookup synchronously on a worker thread. Done means a delayed fsync no longer makes /node/version or a ledger_chunk request block all workers. The issue lists three candidate fixes, so the design choice needs maintainer input first.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Describe the bug

Since 7.0.19, ledger writes no longer go through the host-enclave ringbuffer (#8405), and ledger file I/O runs on the task worker threads. This works as intended for a slow disk on its own. When one ledger fsync or chunk open is blocked for 30 s, a 7.0.23 node keeps answering /node/version and /node/state, while a 7.0.17 node stops answering completely.

One case still freezes the node:

So while an fsync is slow, every ledger_chunk request holds a worker thread until the fsync finishes. With the default worker_threads: 1 (two task workers), one request is enough: the other worker is the one running the fsync. No worker is left, and the node stops answering every endpoint, including /node/version, until the fsync returns. With worker_threads: 3 (four workers), the node froze when the third concurrent ledger_chunk request arrived.

In Kubernetes, a liveness probe on a /node endpoint then restarts a node that is only waiting for slow storage. Any client that polls ledger_chunk, for example to archive committed chunks, makes this likely during storage latency spikes.

To Reproduce

  1. Start a 3-node network with the release logging sample app and the devel RPM (virtual build), --ledger-chunk-bytes 5KB and the default worker_threads: 1. Send about 20 POST /app/log/public per second.
  2. On one node, delay the next ledger fsync by 30 s: strace -f -p <cchost pid> -e trace=fsync,openat -e inject=fsync:delay_enter=30s:when=1. The node log confirms it: Operation took too long ( 30.02s): Committing ledger file - fsync(ledger_609).
  3. Poll GET /node/version and GET /node/state on that node every 0.5 s, with a 2 s timeout. All of them succeed during the delay.
  4. Repeat, and during the delay send one GET /node/ledger_chunk?since=100 to the same node. From that request until the fsync returns, every poll times out. The ledger_chunk request itself takes about 27 s.
Version Delayed for 30 s Polls during the delay that failed
7.0.17 fsync (primary or backup), chunk create (primary) 112/112 in each test
7.0.23 fsync (primary or backup), chunk create (primary) 0/112 (max latency 0.33 s)
7.0.23 fsync + 1 ledger_chunk request (two runs, two nodes) 100/112, all after the request
7.0.23, worker_threads: 3 fsync + 3 ledger_chunk requests 4 s apart 68/112, all after the 3rd request

Expected behavior

A slow ledger fsync shouldn't stop the node serving endpoints that don't need the ledger, and /node endpoints should stay responsive.

Environment information

  • CCF 7.0.23. src/host/ledger.h is identical on main at 9cdc86cc78a42ee27ea167a7a347c9d924bf4841.
  • Reproduced with the virtual build in Azure Linux 3.0 containers. Production runs C++ applications on the SNP platform (confidential Azure Container Instances), with the ledger on an SMB file share, where single fsync or file-create calls sometimes take 40-55 s.

Frequency

Every time a ledger_chunk request reaches a node while its ledger fsync or chunk creation is blocked. In our deployment that depends on storage latency spikes, which happen several times a week on some file shares.

Additional context / possible fixes

  1. Don't hold state_lock for the fsync in Ledger::commit(). For example, mark the file as being committed under the lock, fsync and rename it outside the lock, then publish it as committed under the lock. committed_ledger_path_with_idx() would only return published files, which keeps today's guarantee.
  2. Or don't make ledger_chunk lookups wait for an in-progress commit. For example, try the lock and answer 503 with Retry-After when it's held, or keep the committed-chunk index under a separate lock.
  3. Or serve ledger_chunk, and other readers of state_lock such as historical reads of committed ranges, without blocking a worker thread.

Related (not necessarily a bug): with 7.0.23, a primary whose ledger fsync or chunk create stalls loses leadership within about 3 s, although its HTTP endpoints stay up. Outbound node-to-node messages, including heartbeats, wait for all ledger mutations submitted before them (node_connections.h#L413-L423, ledger_subsystem.h#L210-L221). So a storage latency spike on the primary still causes an election.

Lingua principale
C++
Stelle
876
Fork
264
Merge medio
1g 16h
PR unite (30g)
127

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di microsoft/CCF

Tutte le issue di microsoft/CCF

Issue simili

Altre issue su C++

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.