BBRv3 stuck at 40-50 Mbit/s on any path with 1 ms or more of delay; sender stays below its pacing rate

Open
#882 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
22/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
rust
Domain
networking

Research direction

The suspected code is connection/pacing.rs: start at Pacer::delay_at_rate and the .min(self.capacity) token cap, and check why the bucket is full on wake-ups. Reproduce with the netem setup in the issue, using noq-perf with --congestion bbr3 and the 5 ms delay, 600 Mbit/s path. Done means BBRv3 on that path reaches roughly CUBIC's rate without changing TARGET_BURST_INTERVAL, and the cause is explained.

Written by the indexing model from the issue text.

Description

Describe the bug

BBRv3 settles at 40-50 Mbit/s on a loss-free 10 ms, 600 Mbit/s path (netem), where Quinn 0.11's BBR does about 480 and noq's CUBIC about 480-500. Any added delay of 1 ms or more triggers it. A rate limit with no delay does not.

It looks pacing-limited: a large window, little in flight, and delivery below the pacing rate, so max_bw never climbs. Changing TARGET_BURST_INTERVAL from 2 ms to 8 ms in connection/pacing.rs brings it to roughly 275-411 Mbit/s. That is a workaround, not a proposed fix, because we did not work out why the bucket is full on those wake-ups.

Expected behavior

BBRv3 reaches about the same rate as CUBIC on a clean path.

noq

Version: 1.4.0 and main at 70fdbfa (noq-perf was built from main).

Platform(s)

Desktop: Linux container (kernel 7.0.14-orbstack, OrbStack on macOS 27, both ends on lo), native arm64 and amd64 under Rosetta.

Additional Context

This issue was written by an AI agent (Claude) for the account owner, who asked for it to be posted. The tick boxes below are left for the owner, who has not independently reproduced this.

Repro with noq-perf, measurements and traces (AI-assisted)
ip link add ifb0 type ifb && ip link set ifb0 up
tc qdisc add dev lo handle ffff: ingress
tc filter add dev lo parent ffff: protocol ip u32 match u32 0 0 action mirred egress redirect dev ifb0
tc qdisc add dev ifb0 root netem delay 5ms rate 600mbit limit 2000
noq-perf server --listen 127.0.0.1:4433 --congestion bbr3 &
noq-perf client --congestion bbr3 --download-size 2M --upload-size 0 --bi-requests 4 --duration 4 127.0.0.1:4433

noq-perf, average per-stream download:

path cubic bbr3 bbr3, 8 ms bucket
delay 5 ms, rate 600 Mbit/s 122 Mb/s 51 Mb/s not run
delay 5 ms, no rate cap not run 61-70 Mb/s (instrumented builds) 122-127 Mb/s

Our own HTTP range workload (4 streams, 8 MiB ranges) gave the 40-50 Mbit/s above. The 275-411 is from smoke runs at 0-1 % loss, with two outliers at 63 and 186 Mbit/s.

Per-round trace in steady state: cwnd 17-23 MB, 33-60 KB in flight, pacing_rate 52.9 Mbit/s in PROBE_UP, delivery samples 37 Mbit/s. STARTUP had reached about 290 Mbit/s.

Instrumenting Pacer::delay_at_rate, on wake-ups after a pacing delay: mean 0.33 ms late, max 4.8 ms, and 36 % of the tokens earned were discarded by .min(self.capacity). So the timer looks on time and the bucket is already full. Possibly related: MAX_TRANSMIT_DATAGRAMS = 20 and the send_quantum cap. Not confirmed.

Separately, with the 8 ms bucket BBRv3 had more runs below 100 Mbit/s than Quinn 0.11's BBR under modelled loss (5/119 against 1/119). That may just be BBRv3's loss response, so it is not part of this report.

lolwut

  • This issue was created by a human that thought critically about
    the issue being reported and wrote this as concisely and clearly
    as they could. Taking full responsibility for the issue being
    accurate.
  • This issue isn't slop, and is backed up by evidence.
Dominant language
Rust
Stars
454
Forks
98
Avg merge
2d 8h
Merged PRs (30d)
19

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from n0-computer/noq

All issues in n0-computer/noq

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.