replicated: whole-cluster SIGKILL discards acked messages already in every node's segment files (recovery truncates to a frontier only checkpoints publish)

Start with core/integration/tests/cluster/crash_durability.rs and run the failing replicated recovery test using the command in the issue. Then trace the issue’s named recovery paths, PartitionPrepareJournal::open, recover_unpublished_tail, recover_segment_files, and PartitionPersistence::append_batch. Done means the replicated test recovers the acknowledged payloads after whole-cluster SIGKILL, with recovery still stopping at an invalid or torn record.

Bug description

With durability = replicated, a whole-cluster SIGKILL loses acknowledged messages that were already written to every node's segment files. On restart, each replica truncates its segment back to the last published WAL frontier and logs:

WARN journal::partition_journal::segments: discarding unpublished segment bytes beyond the durable WAL frontier group=0 incarnation=1 start_offset=0 durable_bytes=0 discarded_bytes=9510

Buffered appends do not publish the frontier. Only PartitionPrepareJournal::sync(), checkpoint rewrites and a few control operations (segment enable, view certification) do. PartitionPersistence::append_batch calls sync() only for durable appends, so a replicated partition's frontier stays where its last checkpoint or control operation left it. Recovery trusts only the frontier: when both slots verify, PartitionPrepareJournal::open discards everything after it, as the comment on recover_unpublished_tail says ("the tail is discarded as it always was"). recover_segment_files then truncates the segment to segments.tail.

SIGKILL does not lose a completed write(), because the page cache outlives the process. That is the premise of crash_durability.rs, and the README states the same contract:

A crash (SIGKILL, OOM kill) loses the WAL writes not yet handed to the OS, and power loss also loses what the OS had not written to disk. Under replicated, q or more such crashes at once can lose the most recently acknowledged writes.

The second sentence alone does not bound the loss, but the first one does: SIGKILL loses the writes not yet handed to the OS. Expected: a whole-cluster SIGKILL loses at most the writes still in process memory. Actual: it loses everything after the last frontier publication, even though those bytes are on disk on all three nodes. Checkpoints, durable appends, persisted consumer-offset stores, the shutdown drain and a few control operations publish the frontier. replicated appends do not, so on a busy replicated partition the published frontier can sit far behind the data.

Failing test

PR #4632 adds given_replicated_eager_flush_topic_when_the_whole_cluster_is_killed_should_recover_flushed_data to core/integration/tests/cluster/crash_durability.rs. It is the replicated twin of the existing given_eager_flush_topic_when_the_whole_cluster_is_killed_should_recover_flushed_data:

1. Create a 1-partition topic with messages_required_to_save = 1 and durability = Replicated.
2. Send 30 single-message batches and record each confirmed offset.
3. Wait until every node's segment files contain all 30 payloads (wait_until_payloads_installed).
4. kill_cluster() (SIGKILL on every node), then restart_cluster().
5. Poll from offset 0.

On master b15c8e1e9 it fails with:

replicated acked data already in every node's segments must survive a whole-cluster SIGKILL: 0 of 30 acked offsets readable (0 polled, in order: true), first missing or payload-mismatched offset: Some(0)

The persisted twin passes in the same run. All three nodes log durable_bytes=0 discarded_bytes=9510, which is 30 messages of 317 bytes.

Real-process numbers

A 3-node cluster in Docker (apache/iggy:edge, revision 44c7dc834), 2 CPUs and 4 GiB per node, 2 shards, data on volumes. Six producers send about 2,000 msg/s in batches of 20 to a topic with 6 partitions, replicated, and default flush thresholds. Producers retry every send until it is acknowledged. All three nodes get SIGKILL 20 s into the run and restart 15 s later. A verifier then reads every partition from offset 0.

• Lost: 33,860 of 179,760 acknowledged messages. About 40,000 were sent before the kill, so recovery kept roughly the first 15%.
• Order and gaps: no reordering, no offset gaps, no duplicates. The lost messages are a suffix of each partition.
• Discarded bytes: each node logs the discard line for every partition, with the same numbers on all three nodes. For example, partition 5 kept durable_bytes=525039 and dropped discarded_bytes=2677984. The discarded bytes summed over the 6 partitions (about 10.6 MB at 319 bytes per message) equal the lost count.
• persisted: 0 lost in the same scenario.

The same scenario on f6dfbdbb3, before #4387 added the frontier-owned segment recovery, lost 2,829 of 179,400 (1.6%). We reported that on #4266.

Possible directions (for discussion, not in the PR)

1. On open, let a replicated partition adopt the WAL records after the published frontier that verify as an unbroken chain (op sequence, parent checksum, segment reservation), and stop at the first record that does not verify. A torn tail after power loss stops the walk. The log stays untrusted for the election as #4500 already does, so a quorum of trusted replicas still decides the view.
2. If this loss is intended, change the README to say that a whole-cluster crash under replicated loses everything since the last frontier publication, which on a busy partition can be most of the data written since the cluster started.

Versions

• master b15c8e1e9 (integration test, built from source)
• apache/iggy:edge revision 44c7dc834 (real-process run)

Reproduction

cargo build --bin iggy-server
cargo test -p integration --test mod -- --ignored given_replicated_eager_flush_topic_when_the_whole_cluster_is_killed_should_recover_flushed_data

Related: #4266 (recovery contract for Replicated, closed by #4500), #4092 (durability policies), #4387 (frontier-owned segment recovery).

Found and reproduced with an AI agent (Claude Code), following CONTRIBUTING's "Bugs Found by an Agent".