Address change with relays off: the new path is closed as redundant, and the connection stalls ~6 s without a default route

Open
#4,657 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
50/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
rust

Research direction

Start with the issue’s reproduction and client/server path logs, then trace apply_selected_path and handle_network_change, including the default-route wait described around #4039. Reproduce the no-default-route case and inspect the path selection and abandonment behavior on both endpoints. Done means the new validated path stays usable after the old local address disappears and the server stops sending on a path the client abandoned; no test file is named in the issue.

Written by the indexing model from the issue text.

Description

multipath

Describe the bug

Relays disabled, two endpoints on one LAN segment. When the client's IP address changes under an open connection (a new address added, the old one removed), the connection delivers nothing for about 6 s if the client's network has no default route, and about 1.1 s if it has one. Plain QUIC (quinn 0.11.12) in the same setup carries on within 0.1 s.

What the debug logs show without a default route, on the client:

  1. iroh sees the link change about 0.25 s after the address change and holepunches from the new address: path 1 is open about 60 ms later.
  2. The client closes path 1 straight away with APPLICATION_ABANDON_PATH, as a redundant IP path (apply_selected_path), because path 0, from the address that no longer exists, is still the selected path.
  3. Path 0 is dropped (UnusableAfterNetworkChange) only when the network change reaches noq, and handle_network_change first waits for a default route, up to PendingNetworkChangeNotify::MAX_WAIT (5 s, 5.5 s with the backoff steps; #4039).

On the server: once path 0 was abandoned, the server selected path 1, which the client had already closed, and kept sending on it until the client's PATH_ABANDON for it arrived 6.5 s later. That showed up as a second gap of 6.45 s in 4 of the 6 runs where the client was still sending by then.

With a default route (put back right after the change, as DHCP would), the change reaches noq at once and a path from the new address is selected within about 60 ms of the link change; the remaining 1.1 s gap comes after that, and we haven't looked into it.

Calling Endpoint::network_change() right after the change didn't shorten the 6 s, which fits it going through the same wait. A network without a default route is, for example, a LAN with no gateway.

client's network longest gap between echoes after the change
no default route (7 runs of the reproduction below) 5.84 s; in 4 of the 6 that sent long enough, a second gap of 6.45 s
the same, in the larger harness it was cut from (6 runs) 5.8–6.0 s
the same, with network_change() called right after (6 runs) 5.9–6.0 s
default route put back at once (3 runs) 1.08–1.09 s
quinn 0.11.12, no default route (6 runs) 0.09–0.14 s

Relevant Logs

In the details at the end (RUST_LOG=iroh=debug,noq_proto=debug, one 16 s run each way).

Expected behavior

The connection continues on the new address within a round trip or two, as QUIC migration does: a validated path from the new address isn't closed while the selected path's local address is gone, and the server doesn't keep sending on a path the client has closed.

Iroh

Version: 1.4.1 (noq 1.4.0), default features (fast-apple-datapath, metrics, portmapper, tls-ring).

Endpoint configuration: presets::Minimal, RelayMode::Disabled, PortmapperConfig::Disabled, no address lookup, default transport config. The server binds one address; the client binds 0.0.0.0:0 and dials EndpointAddr::from_parts(id, [TransportAddr::Ip(server)]).

Platform(s)
Desktop:

  • OS: Linux, GitHub Actions ubuntu-24.04 runners (x86_64): two network namespaces on one veth pair, 10 ms netem delay each way, no NAT

Additional Context / Screenshots / GIFs

Reproduction (Rust, plus a script that sets up the namespaces; Linux, as root)

Cargo.toml dependencies: iroh = "=1.4.1", tokio = { version = "1", features = ["full"] }, anyhow = "1", tracing-subscriber = { version = "0.3", features = ["env-filter"] }.

use std::time::{Duration, Instant};

use anyhow::Result;
use iroh::endpoint::{presets, PortmapperConfig};
use iroh::{Endpoint, EndpointAddr, RelayMode, TransportAddr};

const ALPN: &[u8] = b"repro/echo";

async fn bind(addr: &str) -> Result<Endpoint> {
    let ep = Endpoint::builder(presets::Minimal)
        .alpns(vec![ALPN.to_vec()])
        .relay_mode(RelayMode::Disabled)
        .portmapper_config(PortmapperConfig::Disabled)
        .clear_ip_transports()
        .bind_addr(addr.parse::<std::net::SocketAddr>()?)?
        .bind()
        .await?;
    Ok(ep)
}

#[tokio::main]
async fn main() -> Result<()> {
    tracing_subscriber::fmt()
        .with_env_filter(tracing_subscriber::EnvFilter::from_default_env())
        .with_writer(std::io::stderr)
        .init();
    let args: Vec<String> = std::env::args().collect();
    match args.get(1).map(String::as_str) {
        Some("serve") => {
            let ep = bind(&args[2]).await?;
            println!("{}", ep.id());
            while let Some(incoming) = ep.accept().await {
                tokio::spawn(async move {
                    let conn = incoming.accept()?.await?;
                    let (mut send, mut recv) = conn.accept_bi().await?;
                    tokio::io::copy(&mut recv, &mut send).await?;
                    send.finish()?;
                    // Keep the connection until the client closes it, so the last echoes arrive.
                    conn.closed().await;
                    anyhow::Ok(())
                });
            }
        }
        Some("ping") => {
            let secs: u64 = args[4].parse()?;
            let ep = bind("0.0.0.0:0").await?;
            let addr =
                EndpointAddr::from_parts(args[3].parse()?, [TransportAddr::Ip(args[2].parse()?)]);
            let conn = ep.connect(addr, ALPN).await?;
            let (mut send, mut recv) = conn.open_bi().await?;
            let start = Instant::now();
            let total = secs * 20;
            tokio::spawn(async move {
                let mut tick = tokio::time::interval(Duration::from_millis(50));
                for _ in 0..total {
                    tick.tick().await;
                    send.write_all(&[0u8; 64]).await?;
                }
                anyhow::Ok(())
            });
            // Replies, and every wait of over a second between two of them (from, to), in seconds.
            let (mut got, mut last, mut gaps) = (0, 0.0f64, Vec::new());
            let mut buf = [0u8; 64];
            while got < total {
                match tokio::time::timeout(Duration::from_secs(20), recv.read_exact(&mut buf)).await
                {
                    Ok(Ok(())) => got += 1,
                    _ => break,
                }
                let now = start.elapsed().as_secs_f64();
                if now - last > 1.0 {
                    gaps.push(format!(
                        "{:.2} s from {last:.2} s to {now:.2} s",
                        now - last
                    ));
                }
                last = now;
            }
            println!(
                "sent for {secs} s: {got}/{total} echoes back; gaps over 1 s: {}",
                if gaps.is_empty() {
                    "none".to_string()
                } else {
                    gaps.join(", ")
                }
            );
            conn.close(0u32.into(), b"done");
            ep.close().await;
        }
        _ => eprintln!(
            "usage: repro serve <ip:port> | ping <ip:port> <endpoint id> <seconds>"
        ),
    }
    Ok(())
}
#!/usr/bin/env bash
# sudo ./run.sh ./target/release/repro   (Linux; client namespace c, server namespace s)
set -uo pipefail
BIN="$(readlink -f "$1")"
cleanup() { ip netns del c 2>/dev/null; ip netns del s 2>/dev/null; return 0; }
trap cleanup EXIT

# run <seconds the client sends for> <route|noroute> [RUST_LOG]
run() {
  local SECS=$1 ROUTE=$2 LOGS=${3:-}
  cleanup
  ip netns add c; ip netns add s
  ip link add vc type veth peer name vs
  ip link set vc netns c; ip link set vs netns s
  ip -n c link set lo up; ip -n s link set lo up
  # Without promote_secondaries, deleting the client's first address also deletes the second.
  ip netns exec c sysctl -qw net.ipv4.conf.all.promote_secondaries=1 net.ipv4.conf.vc.promote_secondaries=1
  ip -n c addr add 10.0.0.1/24 dev vc
  ip -n s addr add 10.0.0.2/24 dev vs
  ip -n c link set vc up; ip -n s link set vs up
  if [ "$ROUTE" = route ]; then ip -n c route add default via 10.0.0.2 dev vc; fi
  ip netns exec c tc qdisc add dev vc root netem delay 10ms
  ip netns exec s tc qdisc add dev vs root netem delay 10ms

  local TMP SLOG=/dev/null CLOG=/dev/null
  TMP="$(mktemp -d)"
  if [ -n "$LOGS" ]; then SLOG="$SECS-$ROUTE-server.log"; CLOG="$SECS-$ROUTE-client.log"; fi
  ip netns exec s env RUST_LOG="$LOGS" "$BIN" serve 10.0.0.2:47000 >"$TMP/id" 2>"$SLOG" &
  local SERVER=$!
  for _ in $(seq 100); do [ -s "$TMP/id" ] && break; sleep 0.1; done
  ip netns exec c env RUST_LOG="$LOGS" "$BIN" ping 10.0.0.2:47000 "$(cat "$TMP/id")" "$SECS" >"$TMP/result" 2>"$CLOG" &
  local CLIENT=$!
  # Three seconds in, the client gets a new address and loses the old one; with a default
  # route, the route is put back at once.
  sleep 3
  ip -n c addr add 10.0.0.3/24 dev vc
  ip -n c addr del 10.0.0.1/24 dev vc
  if [ "$ROUTE" = route ]; then ip -n c route replace default via 10.0.0.2 dev vc; fi
  wait "$CLIENT"; kill "$SERVER" 2>/dev/null; wait "$SERVER" 2>/dev/null
  echo "$ROUTE: $(cat "$TMP/result")"
  rm -rf "$TMP"
}

run 16 noroute
run 30 noroute
run 16 noroute "iroh=debug,noq_proto=debug"
run 16 route
run 30 route
run 16 route "iroh=debug,noq_proto=debug"

Output from one CI run:

noroute: sent for 16 s: 320/320 echoes back; gaps over 1 s: 5.84 s from 2.92 s to 8.76 s
noroute: sent for 30 s: 600/600 echoes back; gaps over 1 s: 5.84 s from 2.92 s to 8.76 s, 6.45 s from 8.78 s to 15.23 s
route: sent for 16 s: 320/320 echoes back; gaps over 1 s: 1.09 s from 3.38 s to 4.47 s
route: sent for 30 s: 600/600 echoes back; gaps over 1 s: 1.08 s from 3.38 s to 4.46 s
Logs without a default route (path events only; the address changes at about 07:56:51.9)

Client:

07:56:52.183 iroh::socket: link change detected is_major=true
07:56:52.184 iroh::_events::direct_addrs: addrs={DirectAddr { addr: 10.0.0.3:57451, typ: Local }}
07:56:52.184 iroh::_events::qnt::init: local_candidates={10.0.0.3:57451} remote_candidates={10.0.0.2:47000}
07:56:52.224 noq_proto::connection: path added validated=false path_id=1 network_path=(local: 10.0.0.3, remote: [::ffff:10.0.0.2]:47000)
07:56:52.244 iroh::_events::path::open: path_id=1 network_path=Ip(10.0.0.3->10.0.0.2:47000)
07:56:52.245 iroh::_events::path::abandoned: path_id=1 network_path=Ip(10.0.0.3->10.0.0.2:47000) reason=ApplicationClosed { error_code: 62 }
07:56:57.694 noq_proto::connection: network changed
07:56:57.694 noq_proto::connection: path added validated=true path_id=2 network_path=(local: <unspecified>, remote: [::ffff:10.0.0.2]:47000)
07:56:57.694 iroh::_events::path::abandoned: path_id=0 network_path=Ip(10.0.0.1->10.0.0.2:47000) reason=UnusableAfterNetworkChange
07:56:57.714 iroh::_events::path::selected: network_path=Ip(10.0.0.3->10.0.0.2:47000) prev_network_path=Ip(10.0.0.1->10.0.0.2:47000)

Server:

07:56:52.255 iroh::_events::path::open: path_id=1 network_path=Ip(10.0.0.2->10.0.0.3:57451)
07:56:57.704 iroh::_events::path::abandoned: path_id=0 network_path=Ip(10.0.0.2->10.0.0.1:57451) reason=RemoteAbandoned { error_code: 15990 }
07:56:57.704 iroh::_events::path::selected: network_path=Ip(10.0.0.2->10.0.0.3:57451) prev_network_path=Ip(10.0.0.2->10.0.0.1:57451)
07:56:57.704 iroh::_events::path::set_status: path_id=1 network_path=Ip(10.0.0.2->10.0.0.3:57451) status=Available prev_status=Backup
07:57:04.192 iroh::_events::path::abandoned: path_id=1 network_path=Ip(10.0.0.2->10.0.0.3:57451) reason=RemoteAbandoned { error_code: 62 }

With a default route, client (in both cases iroh also opens a path from the old address right after abandoning path 0):

08:25:46.985 iroh::socket: link change detected is_major=true
08:25:46.985 noq_proto::connection: network changed
08:25:46.985 noq_proto::connection: path added validated=true path_id=1 network_path=(local: <unspecified>, remote: [::ffff:10.0.0.2]:47000)
08:25:46.985 iroh::_events::path::abandoned: path_id=0 network_path=Ip(10.0.0.1->10.0.0.2:47000) reason=UnusableAfterNetworkChange
08:25:46.985 noq_proto::connection: path added validated=false path_id=2 network_path=(local: 10.0.0.1, remote: [::ffff:10.0.0.2]:47000)
08:25:47.026 noq_proto::connection: path added validated=false path_id=3 network_path=(local: 10.0.0.3, remote: [::ffff:10.0.0.2]:47000)
08:25:47.046 iroh::_events::path::selected: network_path=Ip(10.0.0.3->10.0.0.2:47000) prev_network_path=Ip(10.0.0.1->10.0.0.2:47000)

This came out of testing iroh for a desktop app. This issue was written and posted by an AI agent on my behalf, from those CI runs, so the two boxes below are left unchecked; the numbers, logs and reproduction are from the runs described.

lolwut

  • This issue was created by a human that thought critically about
    the issue being reported and wrote this as concisely and clearly
    as they could. Taking full responsibility for the issue being
    accurate.
  • This issue isn't slop, and is backed up by evidence.
Dominant language
Rust
Stars
13k
Forks
771
Avg merge
4d 7h
Merged PRs (30d)
20

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from n0-computer/iroh

All issues in n0-computer/iroh

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.