Multi-second stall after a short blackout: PTO backoff (and occasional RTT sample spanning the blackout) delays recovery by up to one probe interval

Open
#863 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
rust
Domain
networking

Research direction

Read the PTO re-arming discussion in #841 and the path scheduling discussion in #516, then trace the QUIC PTO and congestion-control probe logic using the supplied timer_fired and PATH_ACK traces. The report asks whether configuration or an endpoint API can recover promptly after a blackout; a minimal reproduction is only available on request. Done means establishing the cause and a supported recovery approach that avoids delayed probing and misleading RTT samples.

Written by the indexing model from the issue text.

Description

Versions: iroh 1.3.0, noq 1.3.0 / noq-proto 1.3.0. Linux (Docker lab, netem).

Scenario: two endpoints on a direct path (NAT'd, hole-punched). One connection carries audio as 100 datagrams/s plus a reliable control stream. A total blackout of 5 s (netem loss 100% on both routers), after which the network comes back.

Expected: traffic resumes within ~1 RTT plus up to one PTO base after the network returns.

Observed (30 blackouts):

  • 5 of them took 4.2–4.8 s to resume, while other connections on the same path resumed earlier.
  • Traces (timer_fired): the probe interval grows during the blackout: count 5 at −1.45 s, count 6 at +0.72 s, count 7 at +3.89 s (relative to the network coming back). Until the next probe, the path stays blocked by congestion control, with the cwnd full of packets sent during the blackout.
  • Sometimes the first ACK after the blackout acknowledges packets sent before the blackout (PATH_ACK … delay: 915949µs). That yields an RTT sample about as long as the blackout, inflates srtt/rttvar, and pushes the PTO to 4–6 s (the connection stayed stuck for ~9 s in one trace).

What we tried (local patches, discarded):

  1. Skipping RTT samples whose largest acked packet was sent before the current PTO episode: no measurable improvement.
  2. Capping the absolute probe interval at max_interval: recovery broke completely (the connection never resumed). We did not dig into why.

We also tried, at the application level, tearing down and re-establishing the stalled connection as soon as another connection to the same peer received data again. It made things worse (occasional long gaps), so we dropped it.

Question: is there a recommended configuration for short blackouts (e.g. Wi-Fi drops)? Or could the endpoint expose a way to reset the PTO backoff and probe immediately when any packet from the same remote arrives on another connection or path, which signals that the network is back? Endpoint::network_change() exists, but it seems aimed at interface changes and may abandon or reopen paths on the client.

Possibly related: #841 (PTO re-arming after a long stall) and #516 (switching scheduling to other paths faster).

Reproduction: the code is in a private repository, but we can share a minimal reproduction (Docker Compose with NAT'd routers and netem) and the traces on request.

Dominant language
Rust
Stars
454
Forks
98
Avg merge
2d 8h
Merged PRs (30d)
19

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from n0-computer/noq

All issues in n0-computer/noq

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.