perf(cluster): a read buffer in the QUIC frame receiver, +30 % on Linux and +95 % on macOS for pipelined tells

Open
  • Is something wrong?
#283 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
55/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
rust
Domain
networking, performance

Research direction

Start with tellus/src/cluster/transport/quic.rs, especially QuicFrameReceiver::recv at lines 562–584 and the existing send/send_batch buffering at lines 495–533. Review the described prototype and its unit cases for partial reads, empty and oversized frames, stream fragments, and cancellation safety. Done means the receiver buffers reads correctly and tests cover those cases; choose a buffer size that accounts for the stated per-peer memory cost.

Written by the indexing model from the issue text.

Description

QuicFrameReceiver::recv reads every frame with two read_exact calls, one for the length and one for the body (quic.rs 562-584). Each call locks quinn's connection state, which the connection driver holds while it handles packets. The sending half does not pay that per frame: send and send_batch assemble lengths and frames in one buffer and write once (495-533).

A prototype keeps a read buffer in the receiver: one read per refill, and frames are handed out of the buffer until it holds no complete one. The pipelined row of just bench-cluster, with BENCH_PINGS raised to 3,000,000 so that the phase lasts several seconds, three alternating rounds each, medians:

Pipelined, round trips/s main 16 KiB read buffer
Linux x86_64 (EPYC 9R14, 8 vCPU) 563k 745k
Linux arm64 (Graviton3, 8 vCPU) 390k 502k
macOS (Apple silicon) 400k 779k

The ask rows gain 12 to 16 % on Linux and nothing clear on macOS. The serial rows and the bulk row do not change on any of the three.

Where the time goes: on macOS I sampled both processes during the pipelined phase. On the node which receives the pings, a quarter of the samples outside idle waits are blocked in __psynch_mutexwait, and 519 of those 1,121 samples come from RecvStream::poll_read_buf under read_frames. With the buffer that falls from 130 to 6 samples per second. On Linux x86_64, quinn's share of all perf samples in that phase falls from 11 % to 4 %.

The buffer size matters, measured on macOS, two runs each: 2 KiB gives 601k and 646k round trips/s, 4 KiB 700k and 687k, 16 KiB 788k and 802k, 64 KiB 841k and 915k.

bench-streams with 16 streams shows no difference on macOS in two runs each: bulk 101 to 102 MiB/s, and the ask latencies within noise.

What I'd do: what the prototype does. The buffer is a small type of its own, so it has unit tests without a connection: random frames cut into random reads, empty frames and frames larger than the buffer among them, a fragment at the end of a stream, and a frame beyond the limit. recv is then also safe to drop while it waits, since what was read stays in the buffer.

One open point: the buffer is memory per receiving stream, and an inbound connection has 17 streams with the defaults. 16 KiB is 272 KiB per peer, 4 KiB is 68 KiB. It could also start small and grow.

Happy to do it as a PR if you want it, no hurry.

Dominant language
Rust
Stars
37
Forks
1
Avg merge
6h 36m
Merged PRs (30d)
123

Getting set up

  • Ships a Dockerfile or Docker Compose file
  • No pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from tellus-rs/tellus

All issues in tellus-rs/tellus

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.