Backup node aborts on `CompactedVersionConflict` when reading the Nodes table

Đã đóng
  • Có gì không ổn?
#8,650 1 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Một pull request liên quan đã được merge.

  • #8652 của @achamayou — đã merge
  • #8656 của @eddyashton — đã đóng, không merge

Đánh giá

Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức phù hợp với người mới
18/100
Loại issue
Lỗi
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
cpp
Lĩnh vực
backend

Hướng nghiên cứu

Start in BackupSnapshotFetch::do_task_implementation in src/node/node_state.h, where the Nodes lookup can throw, and in try_do_task in src/tasks/worker.h, which aborts on the unknown exception. CompactedVersionConflict is in src/kv/compacted_version_conflict.h. A fix needs a test that forces compaction between the read-version capture and create_change_set. Done means the race logs and retries or skips instead of aborting. Open PR #8652 already targets this, so check it first.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

Describe the bug

With snapshots.backup_fetch.enabled = true, backup nodes occasionally abort (exit code 134) at the moment the primary produces a snapshot.

The snapshot-evidence commit hook schedules a BackupSnapshotFetch task. That task runs on a worker thread and opens a read-only transaction on the Nodes table to find the primary's RPC address (node_state.h#L294-L303):

auto primary_id = owner->consensus->primary();
...
auto tx = owner->network.tables->create_read_only_tx();
auto* nodes = tx.ro<ccf::Nodes>(Tables::NODES);

The read version is captured lazily, on the first ro() call. If the store compacts past that version before the change set is created, create_change_set() returns nullptr. BaseTx::get_handle_by_name then calls compacted_version_conflict(), which throws ccf::kv::CompactedVersionConflict (tx.h#L98-L100, tx.cpp#L116-L128). Compaction happens concurrently on the consensus side.

Nothing in BackupSnapshotFetch::do_task_implementation catches it. CompactedVersionConflict does not derive from std::exception (compacted_version_conflict.h#L9), so try_do_task reaches catch (...), logs BackupSnapshotFetch task failed with unknown exception and calls std::abort() (worker.h#L42-L48).

So a best-effort optimisation, keeping a recent snapshot on backups, can take a backup node down. If another node is already unavailable, the service loses quorum.

Stack trace (CCF 7.0.10; this code path is unchanged in 7.0.17, 7.0.23 and main at 9cdc86cc):

fatal CCF/src/tasks/worker.cpp:181 BackupSnapshotFetch task failed with unknown exception
fatal CCF/src/tasks/worker.cpp:186 Stack trace:
  #0: __cxa_throw
  #1: ccf::kv::BaseTx::compacted_version_conflict(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
  #2: ccf::kv::MapHandle<ccf::EntityId<ccf::NodeIdFormatter>, ccf::NodeInfo, ccf::kv::serialisers::BlitSerialiser<ccf::EntityId<ccf::NodeIdFormatter> >, ccf::kv::serialisers::JsonSerialiser<ccf::NodeInfo> >* ccf::kv::BaseTx::get_handle_by_name<ccf::kv::MapHandle<...> >(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool)
  #3: ccf::NodeState::BackupSnapshotFetch::do_task_implementation()
  #4: ccf::tasks::BaseTask::do_task()
  #5: ccf::tasks::try_do_task(ccf::tasks::BaseTask&, bool)
  #6: ccf::Enclave::run_worker()
  #7: ccf::enclave_run()
  #8: std::thread::_State_impl<...run_enclave_threads(host::HostConfig const&)::$_0, unsigned int...>::_M_run()
  #9: execute_native_thread_routine
  #10: start_thread
  #11: __clone3

To Reproduce

We have not reproduced it in isolation; it is a timing race. What we observe in production:

  1. A 3- or 5-node service with snapshots.backup_fetch.enabled = true (we use max_attempts: 3, retry_interval: 1000ms, target_rpc_interface: operator) and periodic snapshots (tx_count: 10000, time_interval: 24h).
  2. When a snapshot's evidence commits, every backup schedules BackupSnapshotFetch (node_state.h#L3654-L3686).
  3. Rarely, one backup aborts in the same second. In one case, one backup logged Snapshot evidence detected on backup - scheduling snapshot fetch from primary and BackupSnapshotFetch: Received snapshot ..., while the other backup of the same service exited with code 134 at that second. The aborting node's own last log lines are lost, because its log shipping dies with it.

A unit test could force compaction between the read-version capture and create_change_set in a read-only transaction over Nodes, then run the task.

This happens roughly 1 in 5,000 to 10,000 fetches.

Expected behavior

A failure in the backup snapshot fetch should be logged and retried, or skipped until the next snapshot evidence. It should not abort the node.

Environment information

  • CCF 7.0.17 (cdcb74c7), also seen on 7.0.10. Code is unchanged in 7.0.23 and on main at 9cdc86cc78a42ee27ea167a7a347c9d924bf4841 (node_state.h#L309-L318 on main).
  • C++ application on the SNP platform (confidential Azure Container Instances), Linux.
  • worker_threads: 1 in the host config, so two task worker threads.

Additional context / possible fixes

  1. In BackupSnapshotFetch::do_task_implementation, catch ccf::kv::CompactedVersionConflict around the Nodes lookup and retry with a fresh read-only transaction (a few attempts), or log and return. CCF already handles this exception in other places: JS endpoint re-execution (#8289), opening a recovered service (#8450), QuickJS (#7281).
  2. Make CompactedVersionConflict derive from std::exception, or catch it explicitly in try_do_task, so the message is logged instead of "unknown exception".
  3. Consider whether best-effort tasks like BackupSnapshotFetch should run with abort_on_throw = false.

Related, possibly worth a separate issue: the fetch is scheduled when the snapshot evidence commits. That can be before the primary has finished writing and fsyncing the snapshot. For a 435 MB snapshot we saw backups receive Peer has no suitable snapshot at +0.14 s and +1.4 s, while the primary committed the file at +2.4 s. Large snapshots (about 1-2 GB) take 5-6 s, or more under storage latency. A fixed max_attempts x retry_interval window therefore misses them, and the retry sleeps on a task worker thread (fetch_from_peer uses std::this_thread::sleep_for, fetch.h#L450-L480).

Ngôn ngữ chính
C++
Star
876
Fork
264
Merge trung bình
1 ngày 16 giờ
Pull request đã merge (30 ngày)
127

Chuẩn bị môi trường

Mở trong Codespaces

Khởi chạy dev container của dự án ngay trên trình duyệt, bằng tài khoản GitHub của bạn.

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của microsoft/CCF

Tất cả issue của microsoft/CCF

Issue tương tự

Thêm issue về C++

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.