Jepsen: define consistent outcome and Harness failure semantics

Background

An unsuccessful operation is not necessarily an error in a Jepsen test. Timeouts, connection failures, version conflicts, and skipped fault injection may be expected and necessary observations under faults. Calling all of them "errors" obscures the boundary between valid test evidence and a broken test harness.

We have already crossed this boundary in several different ways:

• #1861 found that a raw InterruptedException could be converted into a generic :client-error outcome.
• #1934 found that pause recovery history did not accurately describe what the Nemesis had done, and that teardown failure could prevent analysis.
• #1965 allowed analysis to continue after partition teardown cleanup failed.
• #1967 then found that the broad cleanup handler swallowed additional forms of interruption.
• The current Client still has a catch-all :client-error fallback for unknown exceptions, while some teardown paths treat every non-interruption exception as a cleanup warning.

These fixes address individual symptoms, but they do not establish one rule for classifying future failures. A later change can therefore solve the same class of problem differently and move the harness away from a consistent model.

Why Does It Matter?

Producing the appropriate unsuccessful outcome is part of validating OpenRaft. For example, a timed-out write may or may not have taken effect. It should enter the history as an indeterminate operation so the linearizability checker can interpret it together with later reads and writes.

Conversely, an unknown exception caused by a Client or Nemesis implementation bug is not an SUT observation. Converting it into an ordinary operation outcome allows a broken harness to continue and may produce an untrustworthy verdict.

The opposite mistake is also harmful: treating an expected network or SUT condition as a harness failure terminates a valid experiment and discards useful evidence.

We should therefore use the neutral term outcome for expected observations and reserve Harness failure for failures of the test implementation or its environment.

Proposed Semantic Model

Category	Meaning	Primary evidence
Workload outcome	What a Client operation observed from OpenRaft	history.edn
Nemesis outcome	Whether a requested fault was installed, skipped, failed, or became indeterminate	history.edn
Checker verdict	A decision about a property, coverage, or final recovery	results.edn
Harness failure	A failure of test code, configuration, dependencies, setup, teardown, storage, or checker execution	run-level state and jepsen.log
Interruption	A lifecycle cancellation signal	propagated to the Jepsen coordinator

There is no separate catch-all "Client error" or "Nemesis error" category. The classification depends on what the event means, not on whether it happened inside a Client or Nemesis call.

Expected Behavior

1. A Client converts only recognized transport, HTTP, and OpenRaft API conditions into :ok, :fail, or :info outcomes.
2. A Nemesis converts only explicitly modeled operational conditions into structured outcomes.
3. An unknown Client or Nemesis implementation exception is a Harness failure, not a generic operation outcome.
4. The first Harness failure triggers controlled fail-fast:
   • retain the first exception and its source as run-level state;
   • stop scheduling new Client and Nemesis runtime operations;
   • retain completions from operations already in flight;
   • attempt final best-effort fault cleanup;
   • preserve available artifacts and run applicable checkers;
   • exit nonzero.
5. Interruption is never converted into :fail, :info, or a cleanup warning. Client and Nemesis code restores the interrupt flag and rethrows it.
6. Teardown does not manufacture successful cleanup outcomes. An expected best-effort cleanup failure may be logged as a warning; an unexpected teardown implementation exception remains a Harness failure.

Jepsen normally finishes by exhausting the generator, draining outstanding operations, and stopping its workers. An abnormal interpreter exit may then interrupt workers as a secondary cleanup action. The harness should therefore propagate interruption without trying to infer its origin. Controlled fail-fast needs run-level first-failure state, not cancellation-origin fields in history.edn.

Verdict Semantics

A Harness failure makes the whole run unacceptable, but it must not erase evidence already established by a nested checker.

• A successful nested property verdict is inconclusive because the Harness may not have completed the intended experiment.
• A failed linearizability verdict retains a concrete counterexample and must not be discarded.
• The counterexample establishes an OpenRaft failure only after confirming that the Harness failure did not compromise the history, model, or checker that produced it.

The top-level :valid? therefore answers only whether the whole run can be accepted. It does not independently describe OpenRaft correctness or Harness health.

Questions for Discussion

1. Do we agree with using outcomes to distinguish expected observations from Harness failures?
2. Do we agree that a Harness failure should stop new runtime work while still preserving cleanup, artifacts, and applicable analysis?
3. Do we agree to preserve established counterexamples while requiring an evidence-integrity check before attributing them to OpenRaft?

Non-Goals

• Defining a generic classifier that converts arbitrary SSH or control-layer exceptions into expected Nemesis outcomes.
• Adding cancellation-origin fields to operation history.
• Changing OpenRaft's safety or liveness guarantees.

This issue first seeks agreement on the semantic boundary. The implementation can then be split into small, independently reviewable changes.