`handle_timeout` never returns after a long stall (machine sleep): the PTO re-arms in the past
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 45/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- rust
- Domain
- networking
Research direction
The loop is Connection::handle_timeout in src/connection/mod.rs, which calls TimerTable::expire_before in src/connection/timer.rs; the loss-detection re-arm is in set_loss_detection_timer. Start by running the proposed handle_timeout_returns_after_long_stall test from the issue (add it as src/tests/stall.rs with mod stall; in src/tests/mod.rs) to reproduce the spin. Done means that test passes, the PTO fires once per call, and the existing lib tests and clippy stay clean.
Written by the indexing model from the issue text.
Description
After my Windows PC sleeps for a few hours, Connection::handle_timeout never returns.
- A packet goes out at 06:00:51, then the PC sleeps before the ACK arrives.
- On wake (09:11:55) the PTO fires and picks the next timeout based on the previous send time (06:00:51). The wait is capped at
pto_base * 2^16(~89 min here), so the new deadline is 07:29 at the latest, already past. - So
handle_timeoutkeeps going: the PTO fires again, forever. - The probe that would move last send to now only goes out after
handle_timeoutreturns, so that never happens.
The fix is simple: fire each timer at most once per call, like quinn does. Then the probe goes out and the next deadline lands in the future.
// was: while let Some((timer, _)) = self.timers.expire_before(now, &self.qlog) {
for timer in self.timers.expired(now) {
if !self.timers.expire(timer, now, &self.qlog) {
continue; // an earlier timer stopped or moved it
}
// ...unchanged
}
expired and expire are two small new TimerTable methods (full diff below).
repro test and claude written details r here:
Repro, cause and fix (AI-assisted)
Repro (fails on 1.3.0, passes with the fix; add mod stall; to src/tests/mod.rs)
use std::{sync::mpsc, time::Duration};
use super::util::ConnPair;
use crate::{Dir, Side::*};
/// Lost data is unacked when the machine sleeps for 3 h; a peer packet arrives on wake.
#[test]
fn handle_timeout_returns_after_long_stall() {
let mut pair = ConnPair::builder().connect();
let s = pair.streams(Client).open(Dir::Uni).unwrap();
pair.send_stream(Client, s).write(&[0; 1000]).unwrap();
pair.drive_client();
pair.server.inbound.clear(); // lost
let s = pair.streams(Server).open(Dir::Uni).unwrap();
pair.send_stream(Server, s).write(&[0; 100]).unwrap();
pair.drive_server(); // held until wake
pair.time += Duration::from_secs(3 * 3600);
let now = pair.time;
let held: Vec<_> = std::iter::from_fn(|| pair.client.inbound.pop_first()).collect();
for (_, packet) in held {
pair.client.inbound.push(now, packet);
}
pair.client.drive_incoming(now);
// Driver order: packets first (resets the idle timer), then timers.
let ch = *pair.client.connections.keys().next().unwrap();
let mut conn = pair.client.connections.remove(&ch).unwrap();
for event in pair.client.conn_events.drain().flat_map(|(_, e)| e) {
conn.handle_event(event);
}
let (tx, rx) = mpsc::channel();
std::thread::spawn(move || {
conn.handle_timeout(now);
tx.send(()).unwrap();
});
rx.recv_timeout(Duration::from_secs(10))
.expect("handle_timeout never returned");
}
Cause
handle_timeout loops while let Some((timer, _)) = self.timers.expire_before(now, ..). For PathTimer::LossDetection:
on_loss_detection_timeoutincrementspto_countand callsset_loss_detection_timer.- That re-arms the timer at
time_of_last_ack_eliciting_packet + duration.durationstops growing atpto_base * 2^MAX_BACKOFF_EXPONENT, because of the exponent cap and themax_intervalstep inpto_time_and_space. time_of_last_ack_eliciting_packetonly moves when the probes are sent, which happens afterhandle_timeoutreturns.
So once now - time_of_last_ack_eliciting_packet > pto_base * 2^16, every re-armed deadline is still <= now, and expire_before returns LossDetection again, forever. Each pass also costs O(pto_count) in pto_time_and_space, so it gets slower as it goes.
Two other timers would normally end the loop, but neither can here:
- Connection
Idle:ConnectionDriver::pollrunsprocess_conn_eventsbeforedrive_timer. A packet that arrives on wake resetsIdletonow + 30 s, so it isn't due. - The path's own
PathIdle: it is due, butexpire_beforechecksLossDetectionfirst (PathTimer::VALUESorder), so it's never reached.
quinn-proto doesn't have this problem: its handle_timeout fires each timer at most once per call (for &timer in &Timer::VALUES).
Numbers from the stuck process
Read from memory of the live process (multipath connection; the stuck path predates the sleep, and the wake packet came in on a newer path):
| path RTT | smoothed 53.3 ms, var 0.77 ms; max_ack_delay 25 ms → pto_base 81.3 ms |
cap pto_base * 2^16 |
5 329.9 s |
| last ack-eliciting send on the path | 06:00:51.4 (16 packets in flight) |
now passed to handle_timeout |
09:11:55.9 |
| latest possible PTO deadline | 07:29:41, before now |
connection Idle deadline |
09:12:25.9, reset by the wake packet |
that path's PathIdle deadline |
06:01:11, due but never fired |
| after ~30 min | pto_count = 881 542, loss_probes = 1 763 070, none sent |
Fix
diff --git a/src/connection/mod.rs b/src/connection/mod.rs
index 7971c74..bdcd187 100644
--- a/src/connection/mod.rs
+++ b/src/connection/mod.rs
@@ -2451,7 +2451,13 @@ impl Connection {
/// `Instant` that was output by `poll_timeout`; however spurious extra calls will simply
/// no-op and therefore are safe.
pub fn handle_timeout(&mut self, now: Instant) {
- while let Some((timer, _time)) = self.timers.expire_before(now, &self.qlog) {
+ // Each timer fires at most once per call. A timeout can re-arm its own timer at or before
+ // `now`: loss detection measures the PTO from the last ack-eliciting send, which only moves
+ // once the probes it queued go out. Firing it again here, before they can, never ends.
+ for timer in self.timers.expired(now) {
+ if !self.timers.expire(timer, now, &self.qlog) {
+ continue;
+ }
let span = match timer {
Timer::Conn(timer) => trace_span!("timeout", scope = "conn", ?timer),
Timer::PerPath(path_id, timer) => {
diff --git a/src/connection/timer.rs b/src/connection/timer.rs
index d4524dd..851f5ee 100644
--- a/src/connection/timer.rs
+++ b/src/connection/timer.rs
@@ -97,7 +97,7 @@ impl std::hash::Hash for PathIdKey {
/// Keeps track of the nearest timeout for each `Timer`
///
-/// The [`TimerTable`] is advanced with [`TimerTable::expire_before`].
+/// The [`TimerTable`] is advanced with [`TimerTable::expired`] and [`TimerTable::expire`].
#[derive(Debug, Clone, Default)]
pub(crate) struct TimerTable {
generic: [Option<Instant>; ConnTimer::VALUES.len()],
@@ -185,7 +185,6 @@ where
self.heap.as_mut().and_then(|h| h.get_mut(key))
}
- #[cfg(test)]
fn iter(&self) -> impl Iterator<Item = (&K, &V)> {
let a = self
.stack
@@ -201,6 +200,7 @@ where
a.chain(b)
}
+ #[cfg(test)]
fn iter_mut(&mut self) -> impl Iterator<Item = (&K, &mut V)> {
let a = self
.stack
@@ -256,19 +256,6 @@ impl PathTimerTable {
fn stop(&mut self, timer: PathTimer) {
self.timers[timer as usize] = None;
}
-
- /// Remove the next timer up until `now`, including it
- fn expire_before(&mut self, now: Instant) -> Option<(PathTimer, Instant)> {
- for timer in PathTimer::VALUES {
- if self.timers[timer as usize].is_some()
- && self.timers[timer as usize].expect("checked") <= now
- {
- return self.timers[timer as usize].take().map(|time| (timer, time));
- }
- }
-
- None
- }
}
impl TimerTable {
@@ -294,7 +281,6 @@ impl TimerTable {
qlog.emit_timer_set(timer, time);
}
- #[allow(unused)]
pub(super) fn get(&self, timer: Timer) -> Option<Instant> {
match timer {
Timer::Conn(timer) => self.generic[timer as usize],
@@ -357,42 +343,46 @@ impl TimerTable {
}
}
- /// Remove the next timer up until `now`, including it
- pub(super) fn expire_before(
- &mut self,
- now: Instant,
- qlog: &QlogSink,
- ) -> Option<(Timer, Instant)> {
- let (timer, instant) = self.expire_before_inner(now)?;
- qlog.with_time(now).emit_timer_expire(timer);
- Some((timer, instant))
- }
-
- fn expire_before_inner(&mut self, now: Instant) -> Option<(Timer, Instant)> {
+ /// The timers due at `now`, in the order they fire: the connection's, then each path's.
+ pub(super) fn expired(&mut self, now: Instant) -> Vec<Timer> {
// TODO: this is currently linear in the number of paths
- for timer in ConnTimer::VALUES {
- if self.generic[timer as usize].is_some()
- && self.generic[timer as usize].expect("checked") <= now
- {
- return self.generic[timer as usize]
- .take()
- .map(|time| (Timer::Conn(timer), time));
- }
- }
-
- let mut res = None;
- for (PathIdKey(path_id), timers) in self.path_timers.iter_mut() {
- if let Some((timer, time)) = timers.expire_before(now) {
- res = Some((Timer::PerPath(*path_id, timer), time));
- break;
- }
- }
-
// clear out old timers
self.path_timers
.retain(|_path_id, timers| timers.timers.iter().any(|t| t.is_some()));
- res
+
+ let due = |time: Option<Instant>| time.is_some_and(|time| time <= now);
+ let conn = ConnTimer::VALUES
+ .into_iter()
+ .filter(|&timer| due(self.generic[timer as usize]))
+ .map(Timer::Conn);
+ let paths = self
+ .path_timers
+ .iter()
+ .flat_map(|(&PathIdKey(path_id), timers)| {
+ PathTimer::VALUES
+ .into_iter()
+ .filter(move |&timer| due(timers.get(timer)))
+ .map(move |timer| Timer::PerPath(path_id, timer))
+ });
+ conn.chain(paths).collect()
+ }
+
+ /// Removes `timer` if it's still due at `now`: an earlier timeout may have stopped or moved it.
+ pub(super) fn expire(&mut self, timer: Timer, now: Instant, qlog: &QlogSink) -> bool {
+ if !self.get(timer).is_some_and(|time| time <= now) {
+ return false;
+ }
+ match timer {
+ Timer::Conn(timer) => self.generic[timer as usize] = None,
+ Timer::PerPath(path_id, timer) => {
+ if let Some(timers) = self.path_timers.get_mut(&PathIdKey(path_id)) {
+ timers.stop(timer);
+ }
+ }
+ }
+ qlog.with_time(now).emit_timer_expire(timer);
+ true
}
pub(super) fn reset(&mut self) {
@@ -449,15 +439,13 @@ mod tests {
);
assert_eq!(timers.peek(), Some(now - 3 * sec));
- assert_eq!(
- timers.expire_before(now, &QlogSink::default()),
- Some((Timer::Conn(ConnTimer::Idle), now - 3 * sec))
- );
- assert_eq!(
- timers.expire_before(now, &QlogSink::default()),
- Some((Timer::Conn(ConnTimer::Close), now - 2 * sec))
- );
- assert_eq!(timers.expire_before(now, &QlogSink::default()), None);
+ let idle = Timer::Conn(ConnTimer::Idle);
+ let close = Timer::Conn(ConnTimer::Close);
+ assert_eq!(timers.expired(now), [idle, close]);
+ assert!(timers.expire(idle, now, &QlogSink::default()));
+ assert!(!timers.expire(idle, now, &QlogSink::default()));
+ assert!(timers.expire(close, now, &QlogSink::default()));
+ assert!(timers.expired(now).is_empty());
}
#[test]
- Dominant language
- Rust
- Stars
- 454
- Forks
- 98
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 19
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from n0-computer/noq
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
n0-computer/noq#892 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
n0-computer/noq#873 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
n0-computer/noq#872 ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 55/100
n0-computer/noq#904 ·
Maintainers usually reply within 1 day
-
Difficulty 3/5 Half a day Newbie friendliness 68/100
n0-computer/noq#895 ·
Maintainers usually reply within 1 day
Similar issues
-
area:web kind:bug priority:p2 ready
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 80/100
Maintainers usually reply within 1 day
-
bug
Difficulty 1/5 Under an hour Newbie friendliness 84/100
sims1253/r-typeshed#140 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
storytold/craft-libs#2867 ·
Maintainers usually reply within 1 day
-
help wanted needs investigation priority: low
Difficulty 2/5 1-3 hours Newbie friendliness 62/100