Backup node aborts on `CompactedVersionConflict` when reading the Nodes table

Start in BackupSnapshotFetch::do_task_implementation in src/node/node_state.h, where the Nodes lookup can throw, and in try_do_task in src/tasks/worker.h, which aborts on the unknown exception. CompactedVersionConflict is in src/kv/compacted_version_conflict.h. A fix needs a test that forces compaction between the read-version capture and create_change_set. Done means the race logs and retries or skips instead of aborting. Open PR #8652 already targets this, so check it first.

Describe the bug

With snapshots.backup_fetch.enabled = true, backup nodes occasionally abort (exit code 134) at the moment the primary produces a snapshot.

The snapshot-evidence commit hook schedules a BackupSnapshotFetch task. That task runs on a worker thread and opens a read-only transaction on the Nodes table to find the primary's RPC address (node_state.h#L294-L303):

auto primary_id = owner->consensus->primary();
...
auto tx = owner->network.tables->create_read_only_tx();
auto* nodes = tx.ro<ccf::Nodes>(Tables::NODES);

The read version is captured lazily, on the first ro() call. If the store compacts past that version before the change set is created, create_change_set() returns nullptr. BaseTx::get_handle_by_name then calls compacted_version_conflict(), which throws ccf::kv::CompactedVersionConflict (tx.h#L98-L100, tx.cpp#L116-L128). Compaction happens concurrently on the consensus side.

Nothing in BackupSnapshotFetch::do_task_implementation catches it. CompactedVersionConflict does not derive from std::exception (compacted_version_conflict.h#L9), so try_do_task reaches catch (...), logs BackupSnapshotFetch task failed with unknown exception and calls std::abort() (worker.h#L42-L48).

So a best-effort optimisation, keeping a recent snapshot on backups, can take a backup node down. If another node is already unavailable, the service loses quorum.

Stack trace (CCF 7.0.10; this code path is unchanged in 7.0.17, 7.0.23 and main at 9cdc86cc):

fatal CCF/src/tasks/worker.cpp:181 BackupSnapshotFetch task failed with unknown exception
fatal CCF/src/tasks/worker.cpp:186 Stack trace:
  #0: __cxa_throw
  #1: ccf::kv::BaseTx::compacted_version_conflict(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
  #2: ccf::kv::MapHandle<ccf::EntityId<ccf::NodeIdFormatter>, ccf::NodeInfo, ccf::kv::serialisers::BlitSerialiser<ccf::EntityId<ccf::NodeIdFormatter> >, ccf::kv::serialisers::JsonSerialiser<ccf::NodeInfo> >* ccf::kv::BaseTx::get_handle_by_name<ccf::kv::MapHandle<...> >(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool)
  #3: ccf::NodeState::BackupSnapshotFetch::do_task_implementation()
  #4: ccf::tasks::BaseTask::do_task()
  #5: ccf::tasks::try_do_task(ccf::tasks::BaseTask&, bool)
  #6: ccf::Enclave::run_worker()
  #7: ccf::enclave_run()
  #8: std::thread::_State_impl<...run_enclave_threads(host::HostConfig const&)::$_0, unsigned int...>::_M_run()
  #9: execute_native_thread_routine
  #10: start_thread
  #11: __clone3

To Reproduce

We have not reproduced it in isolation; it is a timing race. What we observe in production:

1. A 3- or 5-node service with snapshots.backup_fetch.enabled = true (we use max_attempts: 3, retry_interval: 1000ms, target_rpc_interface: operator) and periodic snapshots (tx_count: 10000, time_interval: 24h).
2. When a snapshot's evidence commits, every backup schedules BackupSnapshotFetch (node_state.h#L3654-L3686).
3. Rarely, one backup aborts in the same second. In one case, one backup logged Snapshot evidence detected on backup - scheduling snapshot fetch from primary and BackupSnapshotFetch: Received snapshot ..., while the other backup of the same service exited with code 134 at that second. The aborting node's own last log lines are lost, because its log shipping dies with it.

A unit test could force compaction between the read-version capture and create_change_set in a read-only transaction over Nodes, then run the task.

This happens roughly 1 in 5,000 to 10,000 fetches.

Expected behavior

A failure in the backup snapshot fetch should be logged and retried, or skipped until the next snapshot evidence. It should not abort the node.

Environment information

• CCF 7.0.17 (cdcb74c7), also seen on 7.0.10. Code is unchanged in 7.0.23 and on main at 9cdc86cc78a42ee27ea167a7a347c9d924bf4841 (node_state.h#L309-L318 on main).
• C++ application on the SNP platform (confidential Azure Container Instances), Linux.
• worker_threads: 1 in the host config, so two task worker threads.

Additional context / possible fixes

1. In BackupSnapshotFetch::do_task_implementation, catch ccf::kv::CompactedVersionConflict around the Nodes lookup and retry with a fresh read-only transaction (a few attempts), or log and return. CCF already handles this exception in other places: JS endpoint re-execution (#8289), opening a recovered service (#8450), QuickJS (#7281).
2. Make CompactedVersionConflict derive from std::exception, or catch it explicitly in try_do_task, so the message is logged instead of "unknown exception".
3. Consider whether best-effort tasks like BackupSnapshotFetch should run with abort_on_throw = false.

Related, possibly worth a separate issue: the fetch is scheduled when the snapshot evidence commits. That can be before the primary has finished writing and fsyncing the snapshot. For a 435 MB snapshot we saw backups receive Peer has no suitable snapshot at +0.14 s and +1.4 s, while the primary committed the file at +2.4 s. Large snapshots (about 1-2 GB) take 5-6 s, or more under storage latency. A fixed max_attempts x retry_interval window therefore misses them, and the retry sleeps on a task worker thread (fetch_from_peer uses std::this_thread::sleep_for, fetch.h#L450-L480).