Slow ledger fsync or chunk creation blocks `GET /node/ledger_chunk` on `Ledger::state_lock`, which can use up the worker threads and stop the node answering any request

Abierto
  • ¿Algo no está bien?
#8,655 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
8/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
cpp
Área
distributed-systems

Línea de trabajo

Start in src/host/ledger.h, where Ledger::commit(), LedgerFile::commit(), Ledger::write_entry() and committed_ledger_path_with_idx() all take state_lock around fsync and chunk creation. Then read the ledger_chunk handler in src/node/rpc/file_serving_handlers.h, which calls the lookup synchronously on a worker thread. Done means a delayed fsync no longer makes /node/version or a ledger_chunk request block all workers. The issue lists three candidate fixes, so the design choice needs maintainer input first.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Describe the bug

Since 7.0.19, ledger writes no longer go through the host-enclave ringbuffer (#8405), and ledger file I/O runs on the task worker threads. This works as intended for a slow disk on its own. When one ledger fsync or chunk open is blocked for 30 s, a 7.0.23 node keeps answering /node/version and /node/state, while a 7.0.17 node stops answering completely.

One case still freezes the node:

So while an fsync is slow, every ledger_chunk request holds a worker thread until the fsync finishes. With the default worker_threads: 1 (two task workers), one request is enough: the other worker is the one running the fsync. No worker is left, and the node stops answering every endpoint, including /node/version, until the fsync returns. With worker_threads: 3 (four workers), the node froze when the third concurrent ledger_chunk request arrived.

In Kubernetes, a liveness probe on a /node endpoint then restarts a node that is only waiting for slow storage. Any client that polls ledger_chunk, for example to archive committed chunks, makes this likely during storage latency spikes.

To Reproduce

  1. Start a 3-node network with the release logging sample app and the devel RPM (virtual build), --ledger-chunk-bytes 5KB and the default worker_threads: 1. Send about 20 POST /app/log/public per second.
  2. On one node, delay the next ledger fsync by 30 s: strace -f -p <cchost pid> -e trace=fsync,openat -e inject=fsync:delay_enter=30s:when=1. The node log confirms it: Operation took too long ( 30.02s): Committing ledger file - fsync(ledger_609).
  3. Poll GET /node/version and GET /node/state on that node every 0.5 s, with a 2 s timeout. All of them succeed during the delay.
  4. Repeat, and during the delay send one GET /node/ledger_chunk?since=100 to the same node. From that request until the fsync returns, every poll times out. The ledger_chunk request itself takes about 27 s.
Version Delayed for 30 s Polls during the delay that failed
7.0.17 fsync (primary or backup), chunk create (primary) 112/112 in each test
7.0.23 fsync (primary or backup), chunk create (primary) 0/112 (max latency 0.33 s)
7.0.23 fsync + 1 ledger_chunk request (two runs, two nodes) 100/112, all after the request
7.0.23, worker_threads: 3 fsync + 3 ledger_chunk requests 4 s apart 68/112, all after the 3rd request

Expected behavior

A slow ledger fsync shouldn't stop the node serving endpoints that don't need the ledger, and /node endpoints should stay responsive.

Environment information

  • CCF 7.0.23. src/host/ledger.h is identical on main at 9cdc86cc78a42ee27ea167a7a347c9d924bf4841.
  • Reproduced with the virtual build in Azure Linux 3.0 containers. Production runs C++ applications on the SNP platform (confidential Azure Container Instances), with the ledger on an SMB file share, where single fsync or file-create calls sometimes take 40-55 s.

Frequency

Every time a ledger_chunk request reaches a node while its ledger fsync or chunk creation is blocked. In our deployment that depends on storage latency spikes, which happen several times a week on some file shares.

Additional context / possible fixes

  1. Don't hold state_lock for the fsync in Ledger::commit(). For example, mark the file as being committed under the lock, fsync and rename it outside the lock, then publish it as committed under the lock. committed_ledger_path_with_idx() would only return published files, which keeps today's guarantee.
  2. Or don't make ledger_chunk lookups wait for an in-progress commit. For example, try the lock and answer 503 with Retry-After when it's held, or keep the committed-chunk index under a separate lock.
  3. Or serve ledger_chunk, and other readers of state_lock such as historical reads of committed ranges, without blocking a worker thread.

Related (not necessarily a bug): with 7.0.23, a primary whose ledger fsync or chunk create stalls loses leadership within about 3 s, although its HTTP endpoints stay up. Outbound node-to-node messages, including heartbeats, wait for all ledger mutations submitted before them (node_connections.h#L413-L423, ledger_subsystem.h#L210-L221). So a storage latency spike on the primary still causes an election.

Lenguaje dominante
C++
Estrellas
876
Forks
264
Merge medio
1 d 16 h
PR fusionados (30 d)
127

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de microsoft/CCF

Todos los issues de microsoft/CCF

Issues similares

Más issues de C++

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.