A send window that opens a little wakes every stream blocked on it
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 56/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- rust
- Domain
- networking
Research direction
Read noq-proto/src/connection/streams/state.rs around StreamsState::poll and noq/src/connection.rs around forward_app_events to trace how blocked writers are woken. The issue describes a FIFO, one-writer-at-a-time approach and notes the change spans two files under noq-proto/src/connection/streams/. Reproduce the wakeup counts with the test described for noq-proto/src/tests/mod.rs; done means writers wake in blocking order as the send window can serve them, with the no-in-flight case handled.
Written by the indexing model from the issue text.
Description
Describe the bug
When many streams on one connection are blocked on the connection's send window, every one of them is woken each time the window opens, even if it only opened by a packet's worth.
StreamsState::poll pops the whole connection_blocked list while write_limit() > 0:
https://github.com/n0-computer/noq/blob/04807b674cff617cbac323256416dc4779ccf7c2/noq-proto/src/connection/streams/state.rs#L749-L773
forward_app_events drains poll() and wakes each writer before any of them runs:
https://github.com/n0-computer/noq/blob/04807b674cff617cbac323256416dc4779ccf7c2/noq/src/connection.rs#L1762
So write_limit() stays above zero for the whole drain, every blocked writer wakes, the first one or two fill the window, and the rest block again. The cost per ACK grows with the number of blocked streams. The list is also popped from the back, so the stream that blocked last is woken first.
We hit this with 10,000 concurrent 64 KiB transfers on one connection:
| noq-proto | transfers/s | CPU (user + sys) |
|---|---|---|
| 1.4.0 | 861 to 944 | 121 to 165 s |
| 1.4.0 with a fix that wakes one writer at a time | 1,709 to 1,736 | 16 to 19 s |
Three runs each, Linux, both ends in network namespaces joined by a 1 Gbit link.
Relevant Logs
n/a: the cost is in wakeups, not errors; the numbers below are from a reproduction.
Expected behavior
A window that opens a little wakes as many writers as it can serve, in the order they blocked.
noq
Version: 1.4.0 (through iroh 1.4.1, default features); the same code is on main at 04807b674.
Platform(s)
Desktop: Linux x86_64 (kernel 7.0, Docker).
Additional Context
Possible fix:
Keep connection_blocked as a FIFO and wake one writer at a time: the next one is woken when the woken one writes, or when an ACK or MAX_DATA grows the window. With nothing in flight no ACK will come, so in that case it wakes everyone as today. The diff is small (two files in noq-proto/src/connection/streams/), and noq-proto's and noq's tests pass with it. Happy to open a PR if this direction looks right. The same code is in quinn, quinn-proto/src/connection/streams/state.rs#L837, at aa61ce99.
Reproduction in noq-proto's test style, and its output
A test added to noq-proto/src/tests/mod.rs. It drains Connection::poll() before letting the woken streams write, as forward_app_events does. Each stream writes 4 × 1 KiB with a 16 KiB send window.
fn writable_wakeups(streams: usize) -> (u64, u64, u64) {
const CHUNK: usize = 1024;
const PER_STREAM: usize = 4 * CHUNK;
let server = ServerConfig {
transport: Arc::new(TransportConfig {
max_concurrent_uni_streams: VarInt::from_u32(streams as u32),
receive_window: VarInt::MAX,
stream_receive_window: VarInt::from_u32(PER_STREAM as u32),
..TransportConfig::default()
}),
..server_config()
};
let mut pair = Pair::new(Default::default(), server);
let (client_ch, _server_ch) = pair.connect();
pair.client_conn_mut(client_ch).set_send_window(16 * CHUNK as u64);
let chunk = vec![0xAB; CHUNK];
let mut left = vec![PER_STREAM; streams];
let ids = (0..streams)
.map(|_| pair.client_streams(client_ch).open(Dir::Uni).unwrap())
.collect::<Vec<_>>();
let index = |id: crate::StreamId| id.index() as usize;
let write = |pair: &mut Pair, id: crate::StreamId, left: &mut [usize]| -> u64 {
let mut written = 0;
while left[index(id)] > 0 {
let n = left[index(id)].min(CHUNK);
match pair.client_send(client_ch, id).write(&chunk[..n]) {
Ok(n) => {
left[index(id)] -= n;
written += n as u64;
}
Err(WriteError::Blocked) => break,
Err(e) => panic!("{e}"),
}
}
written
};
for &id in &ids {
write(&mut pair, id, &mut left);
}
let (mut wakeups, mut useful, mut bytes) = (0, 0, 0);
while left.iter().any(|&l| l > 0) {
pair.step();
let mut woken = Vec::new();
while let Some(event) = pair.client_conn_mut(client_ch).poll() {
if let Event::Stream(StreamEvent::Writable { id }) = event {
woken.push(id);
}
}
for id in woken {
wakeups += 1;
let written = write(&mut pair, id, &mut left);
if written > 0 {
useful += 1;
bytes += written;
}
}
}
(wakeups, useful, bytes)
}
cargo test --release -p noq-proto --lib, at 04807b674 and with the fix:
| blocked streams | Writable events, main |
found room, main | Writable events, fix |
found room, fix | time, main → fix |
|---|---|---|---|---|---|
| 10 | 13 | 8 | 6 | 6 | 0.9 → 0.9 ms |
| 100 | 2,380 | 143 | 96 | 96 | 1.5 → 1.5 ms |
| 1,000 | 248,804 | 1,493 | 996 | 996 | 16 → 11 ms |
| 10,000 | 24,988,043 | 14,985 | 9,996 | 9,996 | 863 → 105 ms |
noq-perf (--uni-requests 100 and 10000, 64 KiB uploads, loopback) showed no difference beyond its run-to-run noise (±15%), we think because its requests rarely leave thousands of writers blocked at once.
This issue was written with an AI agent (Claude) for the account owner, who read it and asked for it to be posted. The tick boxes below are left for the owner.
lolwut
- This issue was created by a human that thought critically about
the issue being reported and wrote this as concisely and clearly
as they could. Taking full responsibility for the issue being
accurate. - This issue isn't slop, and is backed up by evidence.
- Dominant language
- Rust
- Stars
- 454
- Forks
- 98
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 19
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from n0-computer/noq
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
n0-computer/noq#892 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
n0-computer/noq#873 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
n0-computer/noq#872 ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 55/100
n0-computer/noq#904 ·
Maintainers usually reply within 1 day
-
Difficulty 3/5 Half a day Newbie friendliness 68/100
n0-computer/noq#895 ·
Maintainers usually reply within 1 day
Similar issues
-
area:web kind:bug priority:p2 ready
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 80/100
Maintainers usually reply within 1 day
-
bug
Difficulty 1/5 Under an hour Newbie friendliness 84/100
sims1253/r-typeshed#140 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
storytold/craft-libs#2867 ·
Maintainers usually reply within 1 day
-
help wanted needs investigation priority: low
Difficulty 2/5 1-3 hours Newbie friendliness 62/100