[Detail Bug] Downloader: overload fallback can burn a retry credit without sending any request
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 68/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- rust
- Domain
- backend, distributed-systems
Research direction
Start in src/object_store/downloader/replicas.rs at Followups::next, then trace admission handling into BucketOperation::execute in src/object_store/downloader.rs. Run or recreate the described admission saturation scenario and check whether an overload fallback can consume a retry credit without sending a request. Done means the regression test confirms the credit remains available when the copy cannot start.
Written by the indexing model from the issue text.
Description
Detail Bug Report
Introduced in #139 by @shikhar on Sep 9, 2026
Summary
- Context: The downloader launches replica copies in
Followups::next(src/object_store/downloader/replicas.rs), where a shared retry budget (src/object_store/budget.rs) gates extra copies while all buckets are overloaded, and admission permits (src/object_store/admission.rs) gate each copy before it sends an HTTP request. - Bug: When a fallback copy is launched while the routing snapshot reports every bucket overloaded, the copy spends a retry credit but is launched without an admission permit, so it can fail inside
BucketOperation::executewithAdmissionTimeout— from the local admission queue — without ever sending a request, burning the scarce credit for a copy that never started. - Actual vs. expected: The code's own invariant comment says a credit cannot be spent on a copy that "run[s] out of time before it starts", and the code carefully enforces that for hedges and body retries by pre-acquiring admission; but the third credit-spending case, the overload fallback, skips pre-acquisition and can burn a credit on a copy that never starts.
- Impact: During a widespread-overload window — exactly when the budget is designed to regulate extra attempts and credits replenish slowest (+10 per successful download, 1 attempt per 10 successes) — each affected copy wastes one of the downloader's shared retry grants — one of up to 16 attempts' worth the budget can hold (100 credits per attempt, replenishing up to 1600 at MAX_HEDGES 16 × ATTEMPT_COST 100) — denying retries to other concurrent downloads and reducing the overload window's recovery ability.
Code with Bug
src/object_store/downloader/replicas.rs (Followups::next):
let overloaded = routing.all_overloaded();
let retrying = !early && self.states.buckets[index].tried;
// Hedges and retries never queue for admission, so a retry cannot spend a credit
// and then run out of time before it starts.
let admission = if early || retrying {
Some(downloader.admission.try_acquire(self.request.bytes)?)
} else {
None // <-- BUG 🔴 overload-fallback copies launch with no admission permit
};
if (overloaded || retrying) && !downloader.attempt_budget.try_retry() {
self.retry_budget_exhausted |= overloaded; // <-- BUG 🔴 overload path spends credit even though it may still need to queue
return None;
}
src/object_store/downloader.rs (BucketOperation::execute):
let _admission = if let Some(permit) = permit {
permit
} else {
// A copy queues while it can still finish by the deadline. Only a bucket's first
// copy queues, so no bucket deadline applies.
let deadline = self.deadline.download();
let allowance = self.timeout_allowance();
let latest_start = deadline.checked_sub(allowance).unwrap_or_else(Instant::now);
self.downloader
.admission
.acquire(self.request.bytes, latest_start, allowance)
.await? // <-- BUG 🔴 can return AdmissionTimeout after retry credit was already spent
};
Explanation
Followups::next spends a retry credit via attempt_budget.try_retry() when routing.all_overloaded() is true (or when retrying). For early hedges and same-bucket retries, the code first pre-acquires admission via try_acquire(), ensuring those credit-spending copies never queue and therefore cannot fail with AdmissionTimeout before making any request.
For overload-triggered fallback/rescue copies, admission is None, so the copy is launched and later attempts to acquire admission inside BucketOperation::execute (queueing with a deadline). If admission cannot be obtained before latest_start, acquire() returns AdmissionTimeout and the copy aborts without sending any HTTP request—yet the retry credit was already consumed at launch.
Failing Test
An automated reproduction was added as an uncommitted test-only change: an_overload_fallback_burns_a_retry_credit_for_a_copy_that_never_starts in src/object_store/downloader.rs. It sets both buckets to overloaded, saturates admission (max_inflight_requests: 1) past the fallback’s latest start, and shows:
- no request is ever sent to the fallback bucket (script has no response and would panic if called; request counter stays at 1),
- but the shared budget’s single retry allowance is still consumed (
try_retry()becomesfalseafter the run).
Recommended Fix
Apply the existing invariant uniformly by pre-acquiring (non-queueing) admission for overload-triggered copies too:
let admission = if early || retrying || overloaded {
Some(downloader.admission.try_acquire(self.request.bytes)?)
} else {
None
};
History
This bug was introduced in commit c7c9dbd (fix!: prioritize replica availability and bound downloads (#139)). Later refactors (e.g., b0782c2 / PR #182) added the invariant comment and pre-acquisition for hedges and body retries, but left the overload-triggered credit-spending path unchanged, making overload fallback/rescue the only case that can spend a retry credit and then queue/fail admission before starting.
- Dominant language
- Rust
- Stars
- 622
- Forks
- 18
- Avg merge
- 5h 2m
- Merged PRs (30d)
- 33
Getting set up
- Ships a Dockerfile or Docker Compose file
- No pull request template
- No contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from s2-streamstore/cachey
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
s2-streamstore/cachey#225 ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 52/100
s2-streamstore/cachey#223 ·
Maintainers usually reply within 1 day
-
Difficulty 3/5 1-2 days Newbie friendliness 71/100
s2-streamstore/cachey#222 ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 55/100
s2-streamstore/cachey#217 ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 15/100
s2-streamstore/cachey#6 · 2 comments ·
Maintainers usually reply within 1 day
All issues in s2-streamstore/cachey
Similar issues
-
<input type="number"> misses text-entry leaf sizing in layoutPossibly taken @GoCoder7 claimed this today. Open
Difficulty 1/5 Under an hour Newbie friendliness 88/100
DioxusLabs/blitz#1675 ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
metasequoiaime/msime#11666 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 77/100
Maintainers usually reply within 1 day
-
bug drawings help wanted ui
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
userFRM/egui-charts#3479 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
spellshift/realm#2509 ·
Maintainers usually reply within 1 day