sim: a dropped AcceptFuture forfeits its accept latency, so a select!-embedded accept starves under load

Open
#210 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
68/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
rust

Research direction

Start in crates/moonpool-sim/src/network/sim/stream.rs, tracing AcceptFuture, poll_accept, store_pending, cancel_accept, and return_pending_connection. Add coverage for a select!-embedded accept and for dropping an AcceptFuture mid-delay; done means the queued connection keeps its original readiness time, while existing uncontended accept-latency tests still pass.

Written by the indexing model from the issue text.

Description

Downstream evidence

paros hunt seed 17898267817771645730 on paros 3484b13 (moonpool pin 2482fc7): a three-node cluster whose Phase 1 needed every acceptor never elected a leader in 60 s of recovery tail. One node was answering a client retry storm (18,451 requests in 60 s on one connection, ~3 ms apart) from its driver's main select! loop, whose accept arm is the ordinary accepted = listener.accept() => …. At that node's listener the sim recorded 14,497 cancel_accept calls and no completed accept after the chaos window; the peer's connection was created at the TCP level (create_connection_pair), its bytes were delivered into the server-side receive buffer, and nobody ever read them, so every request on it timed out and hyper's keep-alive killed the connection every 3 s.

Mechanism, from crates/moonpool-sim/src/network/sim/stream.rs (AcceptFuture): the first poll reserves the pending connection (poll_accept) and starts a network_delay(accept_latency) (1–10 ms per seed under random_latency_for_seed); Drop returns the reservation to the listener's queue (return_pending_connection) and cancels the waiter. select! drops and re-creates its branch futures on every pass, so a hot sibling arm that fires faster than the accept latency restarts the delay on every pass and the accept never completes. Polling the same future until it completes (let mut accept = Box::pin(listener.accept()) outside the loop, accepted = &mut accept => { accept = Box::pin(listener.accept()); … }) makes the seed green; paros ships that as its own fix (PierreZ/paros#151).

Why it is a sim fidelity issue, not only a caller's mistake

A kernel completes the TCP handshake whether or not an accept() is pending: the connection sits in the accept queue fully established, and a later accept() returns it at once. Under tokio, tokio::select! { conn = listener.accept() => … } in a hot loop is a common and correct idiom for exactly that reason. In the sim the same idiom is a starvation bug, and only under load — a divergence that hides in production and appears as a liveness failure in DST.

Smallest requested behaviour

Charge the accept latency to the connection, not to the accept() call: sample the delay when the connection is queued at the listener (store_pending), schedule its completion then, and let a later poll_accept return a connection whose delay has elapsed immediately (or wait out the remainder of that connection's delay, never a fresh one). Dropping an AcceptFuture must not reset a queued connection's clock. (Equivalent: keep per-connection ready_at in pending_connections and have poll_accept hand out the first ready one.)

Determinism constraints

  • The delay draw happens once per connection at queue time instead of once per accept call: the RNG call sequence changes, so every seed's schedule shifts (as any change to the draw tree does). No draw should depend on wall time or on how many times the future was polled.
  • cancel_accept / return_pending_connection keep the connection's ready_at; no reordering of the pending queue on cancel.

Acceptance criteria

  • A test where a listener's accept is embedded in a select! with a sibling arm firing every 1 ms, under an accept_latency of 5–10 ms: the connection is accepted within one accept latency of being queued.
  • A test that dropping an AcceptFuture mid-delay and creating a new one accepts the connection at the originally scheduled instant, not original + latency.
  • Existing accept-latency tests keep their expectations for the uncontended case.

Reported from paros; paros keeps the persistent-future shape in its drivers either way.

Dominant language
Rust
Stars
49
Forks
3
Avg merge
1h 4m
Merged PRs (30d)
41

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from PierreZ/moonpool

All issues in PierreZ/moonpool

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.