Assembler can still reject stream data with no gap when chunk sizes are mixed (after #866)
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 35/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- rust
- Domain
- networking
Research direction
The bug lives in noq-proto/src/connection/assembler.rs, in Assembler::insert and defragment, where chunks are kept or merged into runs before the MAX_CHUNKS bound is checked. Start by running the 13-line mixed_sizes_without_a_gap_are_not_rejected test quoted in the issue and reading how defragment decides which chunks to keep. Done means that test passes, the existing TooManyChunks tests still pass for real gaps, and the bound is applied only to chunks remaining after contiguous runs are merged.
Written by the indexing model from the issue text.
Description
Describe the bug
#866 fixed a bug that could lead to TooManyChunks when contiguous chunks of the same size accumulated in the buffer. Even without a gap, Assembler::insert can still return TooManyChunks if enough large chunks are inserted, each followed by a few small ones. The reason is that defragment now keeps a chunk if defragmented and at least min_chunk_size bytes, and all contiguous chunks between such chunks are merged together, but not with the kept chunks. Once the buffer has about 512 such chunks, each followed by a few small chunks, compaction can leave > MAX_CHUNKS contiguous chunks, leading to the connection being closed with too many gaps in stream buffer.
#866 also mentions quinn-rs/quinn#2955 as not covered, about contiguous data behind a single gap when there's sustained packet loss. It's the same underlying mechanism, but doesn't even require a gap, or any loss. (13 line failing unit test on main below.)
We only hit this while constructing a test, by making a reader fall behind while small frames are being sent. I have no idea how likely this is in practice.
Relevant Logs
From the test below, on main:
rejected at offset 67072: 1025 chunks
End to end, reason given by receiving side for closing connection:
TransportError(Error { code: INTERNAL_ERROR, frame: None, reason: "too many gaps in stream buffer", crypto: None })
Expected behavior
Data that is all contiguous, within the receive window, is not rejected; the bound is only on real gaps.
noq
Version: on main at bb9a5edc for the unit test. For the end-to-end test, on crates.io noq-proto 1.4.0, and 1.4.0 with 3ff2822f (#866) and b70f1f2e (#867), used by iroh 1.4.1.
Platform(s)
Desktop:
- OS: macOS (arm64)
- Version 27.2.1
Additional Context / Screenshots / GIFs
#891 reports the same error on 1.4.0, after a stall with a relay. I don't know whether it was due to this, due to chunks of one size, as fixed in #866, or due to real gaps in the data.
quinn-rs/quinn#2955 suggests a fix: if defragment still leaves more than MAX_CHUNKS chunks, only apply the bound to chunks remaining after merging each run of contiguous chunks into one buffer. This ensures the bound is on real gaps between data. I have a patch for noq along those lines, and its tests still expect TooManyChunks when there are too many real gaps in the chunks. (The patch was written by AI, and so was this report.) If you'd like a PR for this, please label this issue ready-for-pr and I'll file one.
Test and measurement (AI-assisted)
defragment keeps a chunk if defragmented and at least max(self.buffered.div_ceil(MAX_CHUNKS), MIN_RETAINED_CHUNK_SIZE) bytes; all other nonempty chunks are copied into a run, and each run ends at the next kept chunk, at a gap, or at the end of the buffered data.
in noq-proto/src/connection/assembler.rs, in the test module:
#[test]
fn mixed_sizes_without_a_gap_are_not_rejected() {
// `defragment` keeps each 128-byte chunk apart, and so the runs of one-byte chunks between
// them, although every chunk starts where the one before it ends.
let mut x = Assembler::new();
let mut offset = 0;
let lengths = core::iter::once(1).chain((0..512).flat_map(|_| [128, 1, 1, 1]));
for len in lengths {
let result = x.insert(offset, Bytes::from(vec![0; len]), len);
assert!(result.is_ok(), "rejected at offset {offset}: {} chunks", x.data.len());
offset += len as u64;
}
}
On main at bb9a5edc, this fails at the last of the 2049 inserts, with rejected at offset 67072: 1025 chunks. Compaction keeps the 512 128 byte chunks, and leaves 513 runs of 1 byte chunks.
End to end, two iroh 1.4.1 endpoints communicating over loopback, with the frames on one stream. The writer writes alternating frames of length 604, 104 and 104 (including a 4 byte length prefix). After writing its nth frame, it doesn't write the next one until the receiving connection's count of STREAM frames has gone up by n. The reader reads one frame, then doesn't read any more until the receiving connection has counted 2100 STREAM frames or the connection closes. Tested three times each, on macOS:
- With
noq-proto1.4.0 from crates.io, the test fails. The receiver reads 2050 of the 2400 frames, then the read fails because the connection is closed, with the reason given above. - With
noq-proto1.4.0 patched with #866 and #867, the test also fails, for the same reason. - With my patch on top, the test passes.
The same test, with frames of one size (204 bytes) fails with noq-proto 1.4.0, but passes with #866 and #867 applied.
lolwut
-
This issue was created by a human that thought critically about
the issue being reported and wrote this as concisely and clearly
as they could. Taking full responsibility for the issue being
accurate.This report was written and filed by an AI assistant that has access to my repos. The tests above were run by AI agents running on my machine. That's why the box above isn't ticked.
-
This issue isn't slop, and is backed up by evidence.
- Dominant language
- Rust
- Stars
- 454
- Forks
- 98
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 19
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from n0-computer/noq
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
n0-computer/noq#892 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
n0-computer/noq#873 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
n0-computer/noq#872 ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 55/100
n0-computer/noq#904 ·
Maintainers usually reply within 1 day
-
Difficulty 3/5 Half a day Newbie friendliness 68/100
n0-computer/noq#895 ·
Maintainers usually reply within 1 day
Similar issues
-
area:web kind:bug priority:p2 ready
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 80/100
Maintainers usually reply within 1 day
-
bug
Difficulty 1/5 Under an hour Newbie friendliness 84/100
sims1253/r-typeshed#140 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
storytold/craft-libs#2867 ·
Maintainers usually reply within 1 day
-
help wanted needs investigation priority: low
Difficulty 2/5 1-3 hours Newbie friendliness 62/100