sim: a dropped AcceptFuture forfeits its accept latency, so a select!-embedded accept starves under load
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 68/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- rust
- Domain
- distributed-systems, networking, testing-qa
Research direction
Start in crates/moonpool-sim/src/network/sim/stream.rs, tracing AcceptFuture, poll_accept, store_pending, cancel_accept, and return_pending_connection. Add coverage for a select!-embedded accept and for dropping an AcceptFuture mid-delay; done means the queued connection keeps its original readiness time, while existing uncontended accept-latency tests still pass.
Written by the indexing model from the issue text.
Description
Downstream evidence
paros hunt seed 17898267817771645730 on paros 3484b13 (moonpool pin 2482fc7): a three-node cluster whose Phase 1 needed every acceptor never elected a leader in 60 s of recovery tail. One node was answering a client retry storm (18,451 requests in 60 s on one connection, ~3 ms apart) from its driver's main select! loop, whose accept arm is the ordinary accepted = listener.accept() => …. At that node's listener the sim recorded 14,497 cancel_accept calls and no completed accept after the chaos window; the peer's connection was created at the TCP level (create_connection_pair), its bytes were delivered into the server-side receive buffer, and nobody ever read them, so every request on it timed out and hyper's keep-alive killed the connection every 3 s.
Mechanism, from crates/moonpool-sim/src/network/sim/stream.rs (AcceptFuture): the first poll reserves the pending connection (poll_accept) and starts a network_delay(accept_latency) (1–10 ms per seed under random_latency_for_seed); Drop returns the reservation to the listener's queue (return_pending_connection) and cancels the waiter. select! drops and re-creates its branch futures on every pass, so a hot sibling arm that fires faster than the accept latency restarts the delay on every pass and the accept never completes. Polling the same future until it completes (let mut accept = Box::pin(listener.accept()) outside the loop, accepted = &mut accept => { accept = Box::pin(listener.accept()); … }) makes the seed green; paros ships that as its own fix (PierreZ/paros#151).
Why it is a sim fidelity issue, not only a caller's mistake
A kernel completes the TCP handshake whether or not an accept() is pending: the connection sits in the accept queue fully established, and a later accept() returns it at once. Under tokio, tokio::select! { conn = listener.accept() => … } in a hot loop is a common and correct idiom for exactly that reason. In the sim the same idiom is a starvation bug, and only under load — a divergence that hides in production and appears as a liveness failure in DST.
Smallest requested behaviour
Charge the accept latency to the connection, not to the accept() call: sample the delay when the connection is queued at the listener (store_pending), schedule its completion then, and let a later poll_accept return a connection whose delay has elapsed immediately (or wait out the remainder of that connection's delay, never a fresh one). Dropping an AcceptFuture must not reset a queued connection's clock. (Equivalent: keep per-connection ready_at in pending_connections and have poll_accept hand out the first ready one.)
Determinism constraints
- The delay draw happens once per connection at queue time instead of once per accept call: the RNG call sequence changes, so every seed's schedule shifts (as any change to the draw tree does). No draw should depend on wall time or on how many times the future was polled.
cancel_accept/return_pending_connectionkeep the connection'sready_at; no reordering of the pending queue on cancel.
Acceptance criteria
- A test where a listener's accept is embedded in a
select!with a sibling arm firing every 1 ms, under anaccept_latencyof 5–10 ms: the connection is accepted within one accept latency of being queued. - A test that dropping an
AcceptFuturemid-delay and creating a new one accepts the connection at the originally scheduled instant, notoriginal + latency. - Existing accept-latency tests keep their expectations for the uncontended case.
Reported from paros; paros keeps the persistent-future shape in its drivers either way.
- Dominant language
- Rust
- Stars
- 49
- Forks
- 3
- Avg merge
- 1h 4m
- Merged PRs (30d)
- 41
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from PierreZ/moonpool
-
bug simulation
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
-
bug github_actions
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
-
bug simulation
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
bug simulation
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
-
documentation release-blocker
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
All issues in PierreZ/moonpool
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
Eynzof/Hermes-CN-Desktop#610 ·
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
gitbutlerapp/gitbutler#15998 · 1 comment ·
-
bug triage:deciding
Difficulty 1/5 Under an hour Newbie friendliness 88/100
open-telemetry/otel-arrow#4132 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100