Slow ledger fsync or chunk creation blocks `GET /node/ledger_chunk` on `Ledger::state_lock`, which can use up the worker threads and stop the node answering any request

Đang mở
  • Có gì không ổn?
#8,655 0 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức phù hợp với người mới
8/100
Loại issue
Lỗi
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
cpp
Lĩnh vực
distributed-systems

Hướng nghiên cứu

Start in src/host/ledger.h, where Ledger::commit(), LedgerFile::commit(), Ledger::write_entry() and committed_ledger_path_with_idx() all take state_lock around fsync and chunk creation. Then read the ledger_chunk handler in src/node/rpc/file_serving_handlers.h, which calls the lookup synchronously on a worker thread. Done means a delayed fsync no longer makes /node/version or a ledger_chunk request block all workers. The issue lists three candidate fixes, so the design choice needs maintainer input first.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

Describe the bug

Since 7.0.19, ledger writes no longer go through the host-enclave ringbuffer (#8405), and ledger file I/O runs on the task worker threads. This works as intended for a slow disk on its own. When one ledger fsync or chunk open is blocked for 30 s, a 7.0.23 node keeps answering /node/version and /node/state, while a 7.0.17 node stops answering completely.

One case still freezes the node:

So while an fsync is slow, every ledger_chunk request holds a worker thread until the fsync finishes. With the default worker_threads: 1 (two task workers), one request is enough: the other worker is the one running the fsync. No worker is left, and the node stops answering every endpoint, including /node/version, until the fsync returns. With worker_threads: 3 (four workers), the node froze when the third concurrent ledger_chunk request arrived.

In Kubernetes, a liveness probe on a /node endpoint then restarts a node that is only waiting for slow storage. Any client that polls ledger_chunk, for example to archive committed chunks, makes this likely during storage latency spikes.

To Reproduce

  1. Start a 3-node network with the release logging sample app and the devel RPM (virtual build), --ledger-chunk-bytes 5KB and the default worker_threads: 1. Send about 20 POST /app/log/public per second.
  2. On one node, delay the next ledger fsync by 30 s: strace -f -p <cchost pid> -e trace=fsync,openat -e inject=fsync:delay_enter=30s:when=1. The node log confirms it: Operation took too long ( 30.02s): Committing ledger file - fsync(ledger_609).
  3. Poll GET /node/version and GET /node/state on that node every 0.5 s, with a 2 s timeout. All of them succeed during the delay.
  4. Repeat, and during the delay send one GET /node/ledger_chunk?since=100 to the same node. From that request until the fsync returns, every poll times out. The ledger_chunk request itself takes about 27 s.
Version Delayed for 30 s Polls during the delay that failed
7.0.17 fsync (primary or backup), chunk create (primary) 112/112 in each test
7.0.23 fsync (primary or backup), chunk create (primary) 0/112 (max latency 0.33 s)
7.0.23 fsync + 1 ledger_chunk request (two runs, two nodes) 100/112, all after the request
7.0.23, worker_threads: 3 fsync + 3 ledger_chunk requests 4 s apart 68/112, all after the 3rd request

Expected behavior

A slow ledger fsync shouldn't stop the node serving endpoints that don't need the ledger, and /node endpoints should stay responsive.

Environment information

  • CCF 7.0.23. src/host/ledger.h is identical on main at 9cdc86cc78a42ee27ea167a7a347c9d924bf4841.
  • Reproduced with the virtual build in Azure Linux 3.0 containers. Production runs C++ applications on the SNP platform (confidential Azure Container Instances), with the ledger on an SMB file share, where single fsync or file-create calls sometimes take 40-55 s.

Frequency

Every time a ledger_chunk request reaches a node while its ledger fsync or chunk creation is blocked. In our deployment that depends on storage latency spikes, which happen several times a week on some file shares.

Additional context / possible fixes

  1. Don't hold state_lock for the fsync in Ledger::commit(). For example, mark the file as being committed under the lock, fsync and rename it outside the lock, then publish it as committed under the lock. committed_ledger_path_with_idx() would only return published files, which keeps today's guarantee.
  2. Or don't make ledger_chunk lookups wait for an in-progress commit. For example, try the lock and answer 503 with Retry-After when it's held, or keep the committed-chunk index under a separate lock.
  3. Or serve ledger_chunk, and other readers of state_lock such as historical reads of committed ranges, without blocking a worker thread.

Related (not necessarily a bug): with 7.0.23, a primary whose ledger fsync or chunk create stalls loses leadership within about 3 s, although its HTTP endpoints stay up. Outbound node-to-node messages, including heartbeats, wait for all ledger mutations submitted before them (node_connections.h#L413-L423, ledger_subsystem.h#L210-L221). So a storage latency spike on the primary still causes an election.

Ngôn ngữ chính
C++
Star
876
Fork
264
Merge trung bình
1 ngày 16 giờ
Pull request đã merge (30 ngày)
127

Chuẩn bị môi trường

Mở trong Codespaces

Khởi chạy dev container của dự án ngay trên trình duyệt, bằng tài khoản GitHub của bạn.

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của microsoft/CCF

Tất cả issue của microsoft/CCF

Issue tương tự

Thêm issue về C++

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.