A send window that opens a little wakes every stream blocked on it

Open
#889 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
56/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
rust
Domain
networking

Research direction

Read noq-proto/src/connection/streams/state.rs around StreamsState::poll and noq/src/connection.rs around forward_app_events to trace how blocked writers are woken. The issue describes a FIFO, one-writer-at-a-time approach and notes the change spans two files under noq-proto/src/connection/streams/. Reproduce the wakeup counts with the test described for noq-proto/src/tests/mod.rs; done means writers wake in blocking order as the send window can serve them, with the no-in-flight case handled.

Written by the indexing model from the issue text.

Description

Describe the bug

When many streams on one connection are blocked on the connection's send window, every one of them is woken each time the window opens, even if it only opened by a packet's worth.

StreamsState::poll pops the whole connection_blocked list while write_limit() > 0:
https://github.com/n0-computer/noq/blob/04807b674cff617cbac323256416dc4779ccf7c2/noq-proto/src/connection/streams/state.rs#L749-L773

forward_app_events drains poll() and wakes each writer before any of them runs:
https://github.com/n0-computer/noq/blob/04807b674cff617cbac323256416dc4779ccf7c2/noq/src/connection.rs#L1762

So write_limit() stays above zero for the whole drain, every blocked writer wakes, the first one or two fill the window, and the rest block again. The cost per ACK grows with the number of blocked streams. The list is also popped from the back, so the stream that blocked last is woken first.

We hit this with 10,000 concurrent 64 KiB transfers on one connection:

noq-proto transfers/s CPU (user + sys)
1.4.0 861 to 944 121 to 165 s
1.4.0 with a fix that wakes one writer at a time 1,709 to 1,736 16 to 19 s

Three runs each, Linux, both ends in network namespaces joined by a 1 Gbit link.

Relevant Logs

n/a: the cost is in wakeups, not errors; the numbers below are from a reproduction.

Expected behavior

A window that opens a little wakes as many writers as it can serve, in the order they blocked.

noq

Version: 1.4.0 (through iroh 1.4.1, default features); the same code is on main at 04807b674.

Platform(s)

Desktop: Linux x86_64 (kernel 7.0, Docker).

Additional Context

Possible fix:

Keep connection_blocked as a FIFO and wake one writer at a time: the next one is woken when the woken one writes, or when an ACK or MAX_DATA grows the window. With nothing in flight no ACK will come, so in that case it wakes everyone as today. The diff is small (two files in noq-proto/src/connection/streams/), and noq-proto's and noq's tests pass with it. Happy to open a PR if this direction looks right. The same code is in quinn, quinn-proto/src/connection/streams/state.rs#L837, at aa61ce99.

Reproduction in noq-proto's test style, and its output

A test added to noq-proto/src/tests/mod.rs. It drains Connection::poll() before letting the woken streams write, as forward_app_events does. Each stream writes 4 × 1 KiB with a 16 KiB send window.

fn writable_wakeups(streams: usize) -> (u64, u64, u64) {
    const CHUNK: usize = 1024;
    const PER_STREAM: usize = 4 * CHUNK;
    let server = ServerConfig {
        transport: Arc::new(TransportConfig {
            max_concurrent_uni_streams: VarInt::from_u32(streams as u32),
            receive_window: VarInt::MAX,
            stream_receive_window: VarInt::from_u32(PER_STREAM as u32),
            ..TransportConfig::default()
        }),
        ..server_config()
    };
    let mut pair = Pair::new(Default::default(), server);
    let (client_ch, _server_ch) = pair.connect();
    pair.client_conn_mut(client_ch).set_send_window(16 * CHUNK as u64);
    let chunk = vec![0xAB; CHUNK];
    let mut left = vec![PER_STREAM; streams];
    let ids = (0..streams)
        .map(|_| pair.client_streams(client_ch).open(Dir::Uni).unwrap())
        .collect::<Vec<_>>();
    let index = |id: crate::StreamId| id.index() as usize;
    let write = |pair: &mut Pair, id: crate::StreamId, left: &mut [usize]| -> u64 {
        let mut written = 0;
        while left[index(id)] > 0 {
            let n = left[index(id)].min(CHUNK);
            match pair.client_send(client_ch, id).write(&chunk[..n]) {
                Ok(n) => {
                    left[index(id)] -= n;
                    written += n as u64;
                }
                Err(WriteError::Blocked) => break,
                Err(e) => panic!("{e}"),
            }
        }
        written
    };
    for &id in &ids {
        write(&mut pair, id, &mut left);
    }
    let (mut wakeups, mut useful, mut bytes) = (0, 0, 0);
    while left.iter().any(|&l| l > 0) {
        pair.step();
        let mut woken = Vec::new();
        while let Some(event) = pair.client_conn_mut(client_ch).poll() {
            if let Event::Stream(StreamEvent::Writable { id }) = event {
                woken.push(id);
            }
        }
        for id in woken {
            wakeups += 1;
            let written = write(&mut pair, id, &mut left);
            if written > 0 {
                useful += 1;
                bytes += written;
            }
        }
    }
    (wakeups, useful, bytes)
}

cargo test --release -p noq-proto --lib, at 04807b674 and with the fix:

blocked streams Writable events, main found room, main Writable events, fix found room, fix time, main → fix
10 13 8 6 6 0.9 → 0.9 ms
100 2,380 143 96 96 1.5 → 1.5 ms
1,000 248,804 1,493 996 996 16 → 11 ms
10,000 24,988,043 14,985 9,996 9,996 863 → 105 ms

noq-perf (--uni-requests 100 and 10000, 64 KiB uploads, loopback) showed no difference beyond its run-to-run noise (±15%), we think because its requests rarely leave thousands of writers blocked at once.

This issue was written with an AI agent (Claude) for the account owner, who read it and asked for it to be posted. The tick boxes below are left for the owner.

lolwut

  • This issue was created by a human that thought critically about
    the issue being reported and wrote this as concisely and clearly
    as they could. Taking full responsibility for the issue being
    accurate.
  • This issue isn't slop, and is backed up by evidence.
Dominant language
Rust
Stars
454
Forks
98
Avg merge
2d 8h
Merged PRs (30d)
19

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from n0-computer/noq

All issues in n0-computer/noq

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.