BBRv3 stuck at 40-50 Mbit/s on any path with 1 ms or more of delay; sender stays below its pacing rate
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 22/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- rust
- Domain
- networking
Research direction
The suspected code is connection/pacing.rs: start at Pacer::delay_at_rate and the .min(self.capacity) token cap, and check why the bucket is full on wake-ups. Reproduce with the netem setup in the issue, using noq-perf with --congestion bbr3 and the 5 ms delay, 600 Mbit/s path. Done means BBRv3 on that path reaches roughly CUBIC's rate without changing TARGET_BURST_INTERVAL, and the cause is explained.
Written by the indexing model from the issue text.
Description
Describe the bug
BBRv3 settles at 40-50 Mbit/s on a loss-free 10 ms, 600 Mbit/s path (netem), where Quinn 0.11's BBR does about 480 and noq's CUBIC about 480-500. Any added delay of 1 ms or more triggers it. A rate limit with no delay does not.
It looks pacing-limited: a large window, little in flight, and delivery below the pacing rate, so max_bw never climbs. Changing TARGET_BURST_INTERVAL from 2 ms to 8 ms in connection/pacing.rs brings it to roughly 275-411 Mbit/s. That is a workaround, not a proposed fix, because we did not work out why the bucket is full on those wake-ups.
Expected behavior
BBRv3 reaches about the same rate as CUBIC on a clean path.
noq
Version: 1.4.0 and main at 70fdbfa (noq-perf was built from main).
Platform(s)
Desktop: Linux container (kernel 7.0.14-orbstack, OrbStack on macOS 27, both ends on lo), native arm64 and amd64 under Rosetta.
Additional Context
This issue was written by an AI agent (Claude) for the account owner, who asked for it to be posted. The tick boxes below are left for the owner, who has not independently reproduced this.
Repro with noq-perf, measurements and traces (AI-assisted)
ip link add ifb0 type ifb && ip link set ifb0 up
tc qdisc add dev lo handle ffff: ingress
tc filter add dev lo parent ffff: protocol ip u32 match u32 0 0 action mirred egress redirect dev ifb0
tc qdisc add dev ifb0 root netem delay 5ms rate 600mbit limit 2000
noq-perf server --listen 127.0.0.1:4433 --congestion bbr3 &
noq-perf client --congestion bbr3 --download-size 2M --upload-size 0 --bi-requests 4 --duration 4 127.0.0.1:4433
noq-perf, average per-stream download:
| path | cubic | bbr3 | bbr3, 8 ms bucket |
|---|---|---|---|
| delay 5 ms, rate 600 Mbit/s | 122 Mb/s | 51 Mb/s | not run |
| delay 5 ms, no rate cap | not run | 61-70 Mb/s (instrumented builds) | 122-127 Mb/s |
Our own HTTP range workload (4 streams, 8 MiB ranges) gave the 40-50 Mbit/s above. The 275-411 is from smoke runs at 0-1 % loss, with two outliers at 63 and 186 Mbit/s.
Per-round trace in steady state: cwnd 17-23 MB, 33-60 KB in flight, pacing_rate 52.9 Mbit/s in PROBE_UP, delivery samples 37 Mbit/s. STARTUP had reached about 290 Mbit/s.
Instrumenting Pacer::delay_at_rate, on wake-ups after a pacing delay: mean 0.33 ms late, max 4.8 ms, and 36 % of the tokens earned were discarded by .min(self.capacity). So the timer looks on time and the bucket is already full. Possibly related: MAX_TRANSMIT_DATAGRAMS = 20 and the send_quantum cap. Not confirmed.
Separately, with the 8 ms bucket BBRv3 had more runs below 100 Mbit/s than Quinn 0.11's BBR under modelled loss (5/119 against 1/119). That may just be BBRv3's loss response, so it is not part of this report.
lolwut
- This issue was created by a human that thought critically about
the issue being reported and wrote this as concisely and clearly
as they could. Taking full responsibility for the issue being
accurate. - This issue isn't slop, and is backed up by evidence.
- Dominant language
- Rust
- Stars
- 454
- Forks
- 98
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 19
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from n0-computer/noq
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
n0-computer/noq#892 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
n0-computer/noq#873 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
n0-computer/noq#872 ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 55/100
n0-computer/noq#904 ·
Maintainers usually reply within 1 day
-
Difficulty 3/5 Half a day Newbie friendliness 68/100
n0-computer/noq#895 ·
Maintainers usually reply within 1 day
Similar issues
-
area:web kind:bug priority:p2 ready
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 80/100
Maintainers usually reply within 1 day
-
bug
Difficulty 1/5 Under an hour Newbie friendliness 84/100
sims1253/r-typeshed#140 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
storytold/craft-libs#2867 ·
Maintainers usually reply within 1 day
-
help wanted needs investigation priority: low
Difficulty 2/5 1-3 hours Newbie friendliness 62/100