Assembler can still reject stream data with no gap when chunk sizes are mixed (after #866)

Open
#893 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
rust
Domain
networking

Research direction

The bug lives in noq-proto/src/connection/assembler.rs, in Assembler::insert and defragment, where chunks are kept or merged into runs before the MAX_CHUNKS bound is checked. Start by running the 13-line mixed_sizes_without_a_gap_are_not_rejected test quoted in the issue and reading how defragment decides which chunks to keep. Done means that test passes, the existing TooManyChunks tests still pass for real gaps, and the bound is applied only to chunks remaining after contiguous runs are merged.

Written by the indexing model from the issue text.

Description

Describe the bug

#866 fixed a bug that could lead to TooManyChunks when contiguous chunks of the same size accumulated in the buffer. Even without a gap, Assembler::insert can still return TooManyChunks if enough large chunks are inserted, each followed by a few small ones. The reason is that defragment now keeps a chunk if defragmented and at least min_chunk_size bytes, and all contiguous chunks between such chunks are merged together, but not with the kept chunks. Once the buffer has about 512 such chunks, each followed by a few small chunks, compaction can leave > MAX_CHUNKS contiguous chunks, leading to the connection being closed with too many gaps in stream buffer.

#866 also mentions quinn-rs/quinn#2955 as not covered, about contiguous data behind a single gap when there's sustained packet loss. It's the same underlying mechanism, but doesn't even require a gap, or any loss. (13 line failing unit test on main below.)

We only hit this while constructing a test, by making a reader fall behind while small frames are being sent. I have no idea how likely this is in practice.

Relevant Logs

From the test below, on main:

rejected at offset 67072: 1025 chunks

End to end, reason given by receiving side for closing connection:

TransportError(Error { code: INTERNAL_ERROR, frame: None, reason: "too many gaps in stream buffer", crypto: None })

Expected behavior

Data that is all contiguous, within the receive window, is not rejected; the bound is only on real gaps.

noq

Version: on main at bb9a5edc for the unit test. For the end-to-end test, on crates.io noq-proto 1.4.0, and 1.4.0 with 3ff2822f (#866) and b70f1f2e (#867), used by iroh 1.4.1.

Platform(s)

Desktop:

  • OS: macOS (arm64)
  • Version 27.2.1

Additional Context / Screenshots / GIFs

#891 reports the same error on 1.4.0, after a stall with a relay. I don't know whether it was due to this, due to chunks of one size, as fixed in #866, or due to real gaps in the data.

quinn-rs/quinn#2955 suggests a fix: if defragment still leaves more than MAX_CHUNKS chunks, only apply the bound to chunks remaining after merging each run of contiguous chunks into one buffer. This ensures the bound is on real gaps between data. I have a patch for noq along those lines, and its tests still expect TooManyChunks when there are too many real gaps in the chunks. (The patch was written by AI, and so was this report.) If you'd like a PR for this, please label this issue ready-for-pr and I'll file one.

Test and measurement (AI-assisted)

defragment keeps a chunk if defragmented and at least max(self.buffered.div_ceil(MAX_CHUNKS), MIN_RETAINED_CHUNK_SIZE) bytes; all other nonempty chunks are copied into a run, and each run ends at the next kept chunk, at a gap, or at the end of the buffered data.

in noq-proto/src/connection/assembler.rs, in the test module:

    #[test]
    fn mixed_sizes_without_a_gap_are_not_rejected() {
        // `defragment` keeps each 128-byte chunk apart, and so the runs of one-byte chunks between
        // them, although every chunk starts where the one before it ends.
        let mut x = Assembler::new();
        let mut offset = 0;
        let lengths = core::iter::once(1).chain((0..512).flat_map(|_| [128, 1, 1, 1]));
        for len in lengths {
            let result = x.insert(offset, Bytes::from(vec![0; len]), len);
            assert!(result.is_ok(), "rejected at offset {offset}: {} chunks", x.data.len());
            offset += len as u64;
        }
    }

On main at bb9a5edc, this fails at the last of the 2049 inserts, with rejected at offset 67072: 1025 chunks. Compaction keeps the 512 128 byte chunks, and leaves 513 runs of 1 byte chunks.

End to end, two iroh 1.4.1 endpoints communicating over loopback, with the frames on one stream. The writer writes alternating frames of length 604, 104 and 104 (including a 4 byte length prefix). After writing its nth frame, it doesn't write the next one until the receiving connection's count of STREAM frames has gone up by n. The reader reads one frame, then doesn't read any more until the receiving connection has counted 2100 STREAM frames or the connection closes. Tested three times each, on macOS:

  • With noq-proto 1.4.0 from crates.io, the test fails. The receiver reads 2050 of the 2400 frames, then the read fails because the connection is closed, with the reason given above.
  • With noq-proto 1.4.0 patched with #866 and #867, the test also fails, for the same reason.
  • With my patch on top, the test passes.

The same test, with frames of one size (204 bytes) fails with noq-proto 1.4.0, but passes with #866 and #867 applied.

lolwut

  • This issue was created by a human that thought critically about
    the issue being reported and wrote this as concisely and clearly
    as they could. Taking full responsibility for the issue being
    accurate.

    This report was written and filed by an AI assistant that has access to my repos. The tests above were run by AI agents running on my machine. That's why the box above isn't ticked.

  • This issue isn't slop, and is backed up by evidence.

Dominant language
Rust
Stars
454
Forks
98
Avg merge
2d 8h
Merged PRs (30d)
19

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from n0-computer/noq

All issues in n0-computer/noq

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.