`handle_timeout` never returns after a long stall (machine sleep): the PTO re-arms in the past

Open
#841 4 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
45/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
rust
Domain
networking

Research direction

The loop is Connection::handle_timeout in src/connection/mod.rs, which calls TimerTable::expire_before in src/connection/timer.rs; the loss-detection re-arm is in set_loss_detection_timer. Start by running the proposed handle_timeout_returns_after_long_stall test from the issue (add it as src/tests/stall.rs with mod stall; in src/tests/mod.rs) to reproduce the spin. Done means that test passes, the PTO fires once per call, and the existing lib tests and clippy stay clean.

Written by the indexing model from the issue text.

Description

After my Windows PC sleeps for a few hours, Connection::handle_timeout never returns.

  1. A packet goes out at 06:00:51, then the PC sleeps before the ACK arrives.
  2. On wake (09:11:55) the PTO fires and picks the next timeout based on the previous send time (06:00:51). The wait is capped at pto_base * 2^16 (~89 min here), so the new deadline is 07:29 at the latest, already past.
  3. So handle_timeout keeps going: the PTO fires again, forever.
  4. The probe that would move last send to now only goes out after handle_timeout returns, so that never happens.

The fix is simple: fire each timer at most once per call, like quinn does. Then the probe goes out and the next deadline lands in the future.

// was: while let Some((timer, _)) = self.timers.expire_before(now, &self.qlog) {
for timer in self.timers.expired(now) {
    if !self.timers.expire(timer, now, &self.qlog) {
        continue; // an earlier timer stopped or moved it
    }
    // ...unchanged
}

expired and expire are two small new TimerTable methods (full diff below).

repro test and claude written details r here:

Repro, cause and fix (AI-assisted)
Repro (fails on 1.3.0, passes with the fix; add mod stall; to src/tests/mod.rs)
use std::{sync::mpsc, time::Duration};

use super::util::ConnPair;
use crate::{Dir, Side::*};

/// Lost data is unacked when the machine sleeps for 3 h; a peer packet arrives on wake.
#[test]
fn handle_timeout_returns_after_long_stall() {
    let mut pair = ConnPair::builder().connect();
    let s = pair.streams(Client).open(Dir::Uni).unwrap();
    pair.send_stream(Client, s).write(&[0; 1000]).unwrap();
    pair.drive_client();
    pair.server.inbound.clear(); // lost
    let s = pair.streams(Server).open(Dir::Uni).unwrap();
    pair.send_stream(Server, s).write(&[0; 100]).unwrap();
    pair.drive_server(); // held until wake

    pair.time += Duration::from_secs(3 * 3600);
    let now = pair.time;
    let held: Vec<_> = std::iter::from_fn(|| pair.client.inbound.pop_first()).collect();
    for (_, packet) in held {
        pair.client.inbound.push(now, packet);
    }
    pair.client.drive_incoming(now);

    // Driver order: packets first (resets the idle timer), then timers.
    let ch = *pair.client.connections.keys().next().unwrap();
    let mut conn = pair.client.connections.remove(&ch).unwrap();
    for event in pair.client.conn_events.drain().flat_map(|(_, e)| e) {
        conn.handle_event(event);
    }
    let (tx, rx) = mpsc::channel();
    std::thread::spawn(move || {
        conn.handle_timeout(now);
        tx.send(()).unwrap();
    });
    rx.recv_timeout(Duration::from_secs(10))
        .expect("handle_timeout never returned");
}
Cause

handle_timeout loops while let Some((timer, _)) = self.timers.expire_before(now, ..). For PathTimer::LossDetection:

  1. on_loss_detection_timeout increments pto_count and calls set_loss_detection_timer.
  2. That re-arms the timer at time_of_last_ack_eliciting_packet + duration. duration stops growing at pto_base * 2^MAX_BACKOFF_EXPONENT, because of the exponent cap and the max_interval step in pto_time_and_space.
  3. time_of_last_ack_eliciting_packet only moves when the probes are sent, which happens after handle_timeout returns.

So once now - time_of_last_ack_eliciting_packet > pto_base * 2^16, every re-armed deadline is still <= now, and expire_before returns LossDetection again, forever. Each pass also costs O(pto_count) in pto_time_and_space, so it gets slower as it goes.

Two other timers would normally end the loop, but neither can here:

  • Connection Idle: ConnectionDriver::poll runs process_conn_events before drive_timer. A packet that arrives on wake resets Idle to now + 30 s, so it isn't due.
  • The path's own PathIdle: it is due, but expire_before checks LossDetection first (PathTimer::VALUES order), so it's never reached.

quinn-proto doesn't have this problem: its handle_timeout fires each timer at most once per call (for &timer in &Timer::VALUES).

Numbers from the stuck process

Read from memory of the live process (multipath connection; the stuck path predates the sleep, and the wake packet came in on a newer path):

path RTT smoothed 53.3 ms, var 0.77 ms; max_ack_delay 25 ms → pto_base 81.3 ms
cap pto_base * 2^16 5 329.9 s
last ack-eliciting send on the path 06:00:51.4 (16 packets in flight)
now passed to handle_timeout 09:11:55.9
latest possible PTO deadline 07:29:41, before now
connection Idle deadline 09:12:25.9, reset by the wake packet
that path's PathIdle deadline 06:01:11, due but never fired
after ~30 min pto_count = 881 542, loss_probes = 1 763 070, none sent
Fix
diff --git a/src/connection/mod.rs b/src/connection/mod.rs
index 7971c74..bdcd187 100644
--- a/src/connection/mod.rs
+++ b/src/connection/mod.rs
@@ -2451,7 +2451,13 @@ impl Connection {
     /// `Instant` that was output by `poll_timeout`; however spurious extra calls will simply
     /// no-op and therefore are safe.
     pub fn handle_timeout(&mut self, now: Instant) {
-        while let Some((timer, _time)) = self.timers.expire_before(now, &self.qlog) {
+        // Each timer fires at most once per call. A timeout can re-arm its own timer at or before
+        // `now`: loss detection measures the PTO from the last ack-eliciting send, which only moves
+        // once the probes it queued go out. Firing it again here, before they can, never ends.
+        for timer in self.timers.expired(now) {
+            if !self.timers.expire(timer, now, &self.qlog) {
+                continue;
+            }
             let span = match timer {
                 Timer::Conn(timer) => trace_span!("timeout", scope = "conn", ?timer),
                 Timer::PerPath(path_id, timer) => {
diff --git a/src/connection/timer.rs b/src/connection/timer.rs
index d4524dd..851f5ee 100644
--- a/src/connection/timer.rs
+++ b/src/connection/timer.rs
@@ -97,7 +97,7 @@ impl std::hash::Hash for PathIdKey {
 
 /// Keeps track of the nearest timeout for each `Timer`
 ///
-/// The [`TimerTable`] is advanced with [`TimerTable::expire_before`].
+/// The [`TimerTable`] is advanced with [`TimerTable::expired`] and [`TimerTable::expire`].
 #[derive(Debug, Clone, Default)]
 pub(crate) struct TimerTable {
     generic: [Option<Instant>; ConnTimer::VALUES.len()],
@@ -185,7 +185,6 @@ where
         self.heap.as_mut().and_then(|h| h.get_mut(key))
     }
 
-    #[cfg(test)]
     fn iter(&self) -> impl Iterator<Item = (&K, &V)> {
         let a = self
             .stack
@@ -201,6 +200,7 @@ where
         a.chain(b)
     }
 
+    #[cfg(test)]
     fn iter_mut(&mut self) -> impl Iterator<Item = (&K, &mut V)> {
         let a = self
             .stack
@@ -256,19 +256,6 @@ impl PathTimerTable {
     fn stop(&mut self, timer: PathTimer) {
         self.timers[timer as usize] = None;
     }
-
-    /// Remove the next timer up until `now`, including it
-    fn expire_before(&mut self, now: Instant) -> Option<(PathTimer, Instant)> {
-        for timer in PathTimer::VALUES {
-            if self.timers[timer as usize].is_some()
-                && self.timers[timer as usize].expect("checked") <= now
-            {
-                return self.timers[timer as usize].take().map(|time| (timer, time));
-            }
-        }
-
-        None
-    }
 }
 
 impl TimerTable {
@@ -294,7 +281,6 @@ impl TimerTable {
         qlog.emit_timer_set(timer, time);
     }
 
-    #[allow(unused)]
     pub(super) fn get(&self, timer: Timer) -> Option<Instant> {
         match timer {
             Timer::Conn(timer) => self.generic[timer as usize],
@@ -357,42 +343,46 @@ impl TimerTable {
         }
     }
 
-    /// Remove the next timer up until `now`, including it
-    pub(super) fn expire_before(
-        &mut self,
-        now: Instant,
-        qlog: &QlogSink,
-    ) -> Option<(Timer, Instant)> {
-        let (timer, instant) = self.expire_before_inner(now)?;
-        qlog.with_time(now).emit_timer_expire(timer);
-        Some((timer, instant))
-    }
-
-    fn expire_before_inner(&mut self, now: Instant) -> Option<(Timer, Instant)> {
+    /// The timers due at `now`, in the order they fire: the connection's, then each path's.
+    pub(super) fn expired(&mut self, now: Instant) -> Vec<Timer> {
         // TODO: this is currently linear in the number of paths
 
-        for timer in ConnTimer::VALUES {
-            if self.generic[timer as usize].is_some()
-                && self.generic[timer as usize].expect("checked") <= now
-            {
-                return self.generic[timer as usize]
-                    .take()
-                    .map(|time| (Timer::Conn(timer), time));
-            }
-        }
-
-        let mut res = None;
-        for (PathIdKey(path_id), timers) in self.path_timers.iter_mut() {
-            if let Some((timer, time)) = timers.expire_before(now) {
-                res = Some((Timer::PerPath(*path_id, timer), time));
-                break;
-            }
-        }
-
         // clear out old timers
         self.path_timers
             .retain(|_path_id, timers| timers.timers.iter().any(|t| t.is_some()));
-        res
+
+        let due = |time: Option<Instant>| time.is_some_and(|time| time <= now);
+        let conn = ConnTimer::VALUES
+            .into_iter()
+            .filter(|&timer| due(self.generic[timer as usize]))
+            .map(Timer::Conn);
+        let paths = self
+            .path_timers
+            .iter()
+            .flat_map(|(&PathIdKey(path_id), timers)| {
+                PathTimer::VALUES
+                    .into_iter()
+                    .filter(move |&timer| due(timers.get(timer)))
+                    .map(move |timer| Timer::PerPath(path_id, timer))
+            });
+        conn.chain(paths).collect()
+    }
+
+    /// Removes `timer` if it's still due at `now`: an earlier timeout may have stopped or moved it.
+    pub(super) fn expire(&mut self, timer: Timer, now: Instant, qlog: &QlogSink) -> bool {
+        if !self.get(timer).is_some_and(|time| time <= now) {
+            return false;
+        }
+        match timer {
+            Timer::Conn(timer) => self.generic[timer as usize] = None,
+            Timer::PerPath(path_id, timer) => {
+                if let Some(timers) = self.path_timers.get_mut(&PathIdKey(path_id)) {
+                    timers.stop(timer);
+                }
+            }
+        }
+        qlog.with_time(now).emit_timer_expire(timer);
+        true
     }
 
     pub(super) fn reset(&mut self) {
@@ -449,15 +439,13 @@ mod tests {
         );
 
         assert_eq!(timers.peek(), Some(now - 3 * sec));
-        assert_eq!(
-            timers.expire_before(now, &QlogSink::default()),
-            Some((Timer::Conn(ConnTimer::Idle), now - 3 * sec))
-        );
-        assert_eq!(
-            timers.expire_before(now, &QlogSink::default()),
-            Some((Timer::Conn(ConnTimer::Close), now - 2 * sec))
-        );
-        assert_eq!(timers.expire_before(now, &QlogSink::default()), None);
+        let idle = Timer::Conn(ConnTimer::Idle);
+        let close = Timer::Conn(ConnTimer::Close);
+        assert_eq!(timers.expired(now), [idle, close]);
+        assert!(timers.expire(idle, now, &QlogSink::default()));
+        assert!(!timers.expire(idle, now, &QlogSink::default()));
+        assert!(timers.expire(close, now, &QlogSink::default()));
+        assert!(timers.expired(now).is_empty());
     }
 
     #[test]
Dominant language
Rust
Stars
454
Forks
98
Avg merge
2d 8h
Merged PRs (30d)
19

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from n0-computer/noq

All issues in n0-computer/noq

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.