[rpc] Qualify the complete RPC stack under combined faults and sustained load

Start with the exact verified commits for #216, #217, and #218, then read docs/plans/moonpool-rpc/architecture.md and parity.md. Inspect the moonpool-sim runner files, existing determinism and recovery tests, and crates/xtask/src/main.rs before running the listed focused cargo tests. Done means the qualification campaign, evidence, documentation, platform checks, independent review, and one passing package PR satisfy every included contract.

Goal

The already tested feature set has independent cross-feature evidence, bounded fault campaigns, performance/resource baselines and an honest release-readiness record for the agreed scope.

Part of #212. This is one substantial implementation package, with multiple coherent local commits and one independently reviewed PR. Design: docs/plans/moonpool-rpc/{README,architecture,parity}.md. These documents are prepared in the planning change; this issue's contracts and validation are self-contained so work does not wait for a documentation merge.

Included scope

Primary ownership: moonpool-rpc-sim and release documentation.

• Combined recruitment/reboots, reliable ambiguity, streaming/backpressure, load balancing and authorization/version scenarios.
• Independent semantic oracles, mandatory scenario coverage inventory, deterministic canary plus semantic replay, process/run isolation and recovery after faults stop.
• Longer bounded campaigns, resource soak, real-network performance and platform/security qualification, book/facade packaging and release-readiness evidence.
• Recalibrate limits and package findings using the P1 measurements; fix discovered root causes without weakening assertions.

Contracts

• Every parity row has passing owned evidence or an explicitly approved exclusion; a green coverage plateau alone cannot satisfy unobserved scenarios.
• Stale I1 never reaches I2 even with delayed registrations, reliable retransmission, active streams and balancing. Participant/configuration identity stays application-owned.
• No task, registration, buffer, model reservation or retained request leaks across process or simulation-run lifetimes. Recovery campaigns regain service after faults stop within declared scenario bounds.
• Faults occur at valid boundaries: partial I/O, close/partition, peer reboot, overload, cancellation and scheduling races. Corruption is a separate explicit network-model campaign, not arbitrary message loss on healthy TCP.
• No exactly-once claim, stream resumption, transparent incarnation refresh or mTLS completion claim appears in release documentation.

FoundationDB references

Exact inspected SHA: c0c44752df676e4a2d532b5cdb4bf96728a30b78. The linked current checkout is the behavioral reference; retained .actor.* snapshots may differ.

• fdbrpc/FlowTests.actor.cpp: /fdbrpc/waitValueOrSignal/peerDisconnect; /noPeerFallback; /retryOnDisconnect.
• fdbserver/clustercontroller/ClusterController.cpp: workerAvailabilityWatch; registerWorker; registration generation tests.
• fdbserver/storageserver/storageserver.cpp: getKeyValuesStreamQ; reply.setByteLimit; reply.onReady; end_of_stream.
• fdbclient/NativeAPI.actor.cpp: getReplyStream and loadBalance call sites; range-stream consumption and application-level recovery.
• fdbrpc/include/fdbrpc/LoadBalance.actor.h: loadBalance; basicLoadBalance; RequestData; ModelHolder; LoadBalanceRequestHooks; AtMostOnce.
• fdbrpc/tests/AuthzTlsTest.cpp: SessionProbeReceiver; getExpectedResult; runHost.
• fdbrpc/sim2.cpp: Sim2Conn::{receiver,whenReadable,whenWritable}; destroyProcess; killProcess_internal; rebootProcess.

Moonpool references

Inspected SHA: 70bc7d6317a76f337cbce0835cf6e63bb4070821.

• docs/plans/moonpool-rpc/architecture.md and parity.md: agreed contracts, adaptation differences and capability ownership.
• crates/moonpool-core/src/{providers,network,time,task,random,select,metrics}.rs: static-generic providers; futures::io; provider time/RNG; handles detach on drop; core's deterministic-select feature is installed by sim.
• crates/moonpool-sim/src/runner/{process_manager,orchestrator,fault_injector,builder}.rs, network/sim/, executor/, and observability/: process factories, actual crash/connection cleanup, deterministic scheduling, scripted faults and trace invariants.
• crates/moonpool-sim/tests/{determinism,determinism_canary,scripted_faults,recovery_mode}.rs; crates/moonpool-buggify/; crates/moonpool-assertions/.
• crates/xtask/src/main.rs uses an explicit SIM_BINARIES registry with package and sancov crate names; register the harness instead of assuming automatic discovery. .github/workflows/rust.yml contains the existing gates.
• Closed #61 (decode loss), #169 (explicit shutdown), #174 (fault-family control), #208 (hidden wall-clock dependency) are lessons, not code/merge dependencies. Do not reopen or restore the removed transport.

Implementation checklist

• Map every parity row and mandatory scenario to test identifiers and campaign entry points; use independent workload histories rather than transport representation as the oracle.
• Combine Paros interface publication with reply loss/retries, slow/abandoned streams, hedge exhaustion/late losers, key rotation, version mismatch and shutdown.
• Add bounded run budgets and explicit recovery phase, mandatory sometimes/reachable coverage, fixed-seed reproduction commands, semantic trace comparison and check_determinism.
• Run real network soak/performance on Linux/macOS with documented offered load, payloads, limits, concurrency, hardware/toolchain and latency/throughput/allocation/resource results.
• Audit dependency trees, process-scoped ownership, external clock/entropy reads, malformed input and operational diagnostics; independently review and fix root causes.
• Add sim-rpc-qualification to CI, publish reproducible evidence in the repository/book and close the tracking matrix only for proven contracts.

Dependencies

• CODE DEPENDENCY: #216, #217, #218.
• MERGE DEPENDENCY: #216, #217, #218. Referenced predecessor implementation must be merged before this package lands; documentation publication is not a code gate.
• CI DEPENDENCY: this package's required remote checks and independent review gate merge. Remote checks do not gate continued local work on a verified predecessor commit.
• Parallel work: Campaign/oracle design can proceed from the established contracts earlier. Full code/merge dependencies are P4/P5/P6; integration can use their exact independently verified commits while remote CI runs, with one recorded integration base.

Depends on #216, #217, #218.

Validation

• Run all earlier unit and real-network suites; none are postponed to this package or replaced by campaign smoke tests.
• Independent execution/publication/consumption ledger validates lost replies after execution, permitted reliable duplicates, no unintended single-attempt retransmit, Paros same-address reboot and third-party interface transmission.
• Mixed campaigns cover reply/disconnect races, timeout ambiguity, caller/stream abandonment, ACK exhaustion, load-balancer fallback, exhausted hedge budgets, late losers, auth denial, version mismatch and shutdown.
• Run repeated process death and repeated complete simulations; assert resource baselines and semantic trace equality in addition to RNG canary. Repeat under varying host speed to expose hidden wall-clock decisions.
• Measure throughput and latency tails at bounded loads, retained bytes and task/endpoint counts under overload/recovery; record budgets before acceptance and explain any measured regression against P1.
• Real TLS/JWT interoperability, cross-version pairings, Linux/macOS runtime variants and wasm/no-native builds; mandatory scenario coverage plus bounded campaign termination and post-fault recovery.

For every introduced fault contract, specify a bounded deterministic scenario that must fire, then supplement with UntilCoverageStable (default plateau 10, maximum 1000). Use .check_determinism() and semantic trace comparison. Fixed failing seeds use .set_debug_seeds(vec![seed]).set_iterations(1); repair the root cause and restore the full campaign. Never disable assertions/tests, reduce required fault coverage, or stretch timeouts to hide a failure.

Local integration strategy

1. Read repository and applicable crate AGENTS.md instructions; start from the listed predecessors' exact verified commits.
2. Implement the checklist as coherent local commits; run focused unit/real-network tests after each behavior change.
3. Run nix develop --command cargo nextest run -p moonpool-rpc -p moonpool-rpc-sim and nix develop --command cargo xtask sim run rpc-qualification (add that campaign in this package).
4. Complete package simulations, the checks below, and independent review before opening one package PR. Internal checkboxes do not require remote PRs/merges.
5. Record commit SHA, predecessor SHAs, review result, commands and campaign seeds/configuration in the PR. While remote CI runs, proceed with the parallel work above. If a predecessor changes, rebase/update, rerun affected tests/campaigns and repeat affected review. Avoid deep provisional stacks: normally at most one unmerged prerequisite layer; return to independent work if more would accumulate.

nix develop --command cargo fmt
nix develop --command cargo clippy -- --deny warnings
nix develop --command cargo nextest run
nix develop --command cargo nextest run -p moonpool-core --features select
nix develop --command cargo clippy -p moonpool-sim --no-default-features --all-targets -- --deny warnings
nix develop --command cargo nextest run -p moonpool-sim --no-default-features
nix develop --command cargo check --target wasm32-unknown-unknown -p moonpool-sim --no-default-features
nix develop --command cargo check --target wasm32-unknown-unknown -p moonpool-rpc --no-default-features
nix develop --command mdbook build book/

Run the relevant real-network tests on Linux/macOS and both Tokio runtime flavors. Verify cargo tree -p moonpool-rpc --no-default-features -e no-dev excludes sim/explorer; after facade integration also inspect the lean facade and optional rpc trees. Match .github/workflows/rust.yml, including book/wasm-demo and feature isolation gates, before merging. Commands mentioning RPC targets apply once this package or its predecessors create them.

Definition of done

• Every included contract has passing unit, real-network and deterministic scenario evidence appropriate to it, including failure/cancellation and resource cleanup.
• Mandatory scenarios were observed; no ignored mandatory tests, placeholders, disabled assertions or undocumented missing behavior.
• Semantic replay and RNG canary pass; all owned work is scoped to process/runtime lifetimes.
• Public items and corresponding book chapters are documented; no RPC-specific core vocabulary or forbidden direct runtime calls. Improve Moonpool when a genuine provider/simulation gap is demonstrated; justify reusable core changes and qualify only existing Moonpool platform/provider support.
• Relevant feature/platform/security checks pass; measurements and residuals are recorded honestly.
• Independent review completed; one substantial PR passes remote CI; update parity evidence and tracking issue.

Out of scope

Restoring moonpool-transport; virtual actors, membership, Paros/Matchmaker Paxos algorithms, consensus identity/fencing, durable RPC queues, exactly-once, transparent dynamic endpoint refresh, automatic stream resumption, FDB wire compatibility, HTTP/2 RPC, unrelated FDB file/HTTP/gRPC subsystems. Mutual TLS is deferred by explicit user decision. Later package capabilities are not required here beyond the boundaries needed to keep this implementation correct.