[Detail Bug] Downloader: overload fallback can burn a retry credit without sending any request

Open
#224 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
68/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
rust

Research direction

Start in src/object_store/downloader/replicas.rs at Followups::next, then trace admission handling into BucketOperation::execute in src/object_store/downloader.rs. Run or recreate the described admission saturation scenario and check whether an overload fallback can consume a retry credit without sending a request. Done means the regression test confirms the credit remains available when the copy cannot start.

Written by the indexing model from the issue text.

Description

Detail Bug Report

https://app.detail.dev/org_89d327b3-b883-4365-b6a3-46b6701342a9/bugs/bug_5cd3d26e-8f67-40c7-b044-8d7d105ce372

Introduced in #139 by @shikhar on Sep 9, 2026

Summary

  • Context: The downloader launches replica copies in Followups::next (src/object_store/downloader/replicas.rs), where a shared retry budget (src/object_store/budget.rs) gates extra copies while all buckets are overloaded, and admission permits (src/object_store/admission.rs) gate each copy before it sends an HTTP request.
  • Bug: When a fallback copy is launched while the routing snapshot reports every bucket overloaded, the copy spends a retry credit but is launched without an admission permit, so it can fail inside BucketOperation::execute with AdmissionTimeout — from the local admission queue — without ever sending a request, burning the scarce credit for a copy that never started.
  • Actual vs. expected: The code's own invariant comment says a credit cannot be spent on a copy that "run[s] out of time before it starts", and the code carefully enforces that for hedges and body retries by pre-acquiring admission; but the third credit-spending case, the overload fallback, skips pre-acquisition and can burn a credit on a copy that never starts.
  • Impact: During a widespread-overload window — exactly when the budget is designed to regulate extra attempts and credits replenish slowest (+10 per successful download, 1 attempt per 10 successes) — each affected copy wastes one of the downloader's shared retry grants — one of up to 16 attempts' worth the budget can hold (100 credits per attempt, replenishing up to 1600 at MAX_HEDGES 16 × ATTEMPT_COST 100) — denying retries to other concurrent downloads and reducing the overload window's recovery ability.

Code with Bug

src/object_store/downloader/replicas.rs (Followups::next):

        let overloaded = routing.all_overloaded();
        let retrying = !early && self.states.buckets[index].tried;
        // Hedges and retries never queue for admission, so a retry cannot spend a credit
        // and then run out of time before it starts.
        let admission = if early || retrying {
            Some(downloader.admission.try_acquire(self.request.bytes)?)
        } else {
            None  // <-- BUG 🔴 overload-fallback copies launch with no admission permit
        };
        if (overloaded || retrying) && !downloader.attempt_budget.try_retry() {
            self.retry_budget_exhausted |= overloaded;  // <-- BUG 🔴 overload path spends credit even though it may still need to queue
            return None;
        }

src/object_store/downloader.rs (BucketOperation::execute):

        let _admission = if let Some(permit) = permit {
            permit
        } else {
            // A copy queues while it can still finish by the deadline. Only a bucket's first
            // copy queues, so no bucket deadline applies.
            let deadline = self.deadline.download();
            let allowance = self.timeout_allowance();
            let latest_start = deadline.checked_sub(allowance).unwrap_or_else(Instant::now);
            self.downloader
                .admission
                .acquire(self.request.bytes, latest_start, allowance)
                .await?   // <-- BUG 🔴 can return AdmissionTimeout after retry credit was already spent
        };

Explanation

Followups::next spends a retry credit via attempt_budget.try_retry() when routing.all_overloaded() is true (or when retrying). For early hedges and same-bucket retries, the code first pre-acquires admission via try_acquire(), ensuring those credit-spending copies never queue and therefore cannot fail with AdmissionTimeout before making any request.

For overload-triggered fallback/rescue copies, admission is None, so the copy is launched and later attempts to acquire admission inside BucketOperation::execute (queueing with a deadline). If admission cannot be obtained before latest_start, acquire() returns AdmissionTimeout and the copy aborts without sending any HTTP request—yet the retry credit was already consumed at launch.

Failing Test

An automated reproduction was added as an uncommitted test-only change: an_overload_fallback_burns_a_retry_credit_for_a_copy_that_never_starts in src/object_store/downloader.rs. It sets both buckets to overloaded, saturates admission (max_inflight_requests: 1) past the fallback’s latest start, and shows:

  • no request is ever sent to the fallback bucket (script has no response and would panic if called; request counter stays at 1),
  • but the shared budget’s single retry allowance is still consumed (try_retry() becomes false after the run).

Recommended Fix

Apply the existing invariant uniformly by pre-acquiring (non-queueing) admission for overload-triggered copies too:

        let admission = if early || retrying || overloaded {
            Some(downloader.admission.try_acquire(self.request.bytes)?)
        } else {
            None
        };

History

This bug was introduced in commit c7c9dbd (fix!: prioritize replica availability and bound downloads (#139)). Later refactors (e.g., b0782c2 / PR #182) added the invariant comment and pre-acquisition for hedges and body retries, but left the overload-triggered credit-spending path unchanged, making overload fallback/rescue the only case that can spend a retry credit and then queue/fail admission before starting.

Dominant language
Rust
Stars
622
Forks
18
Avg merge
5h 2m
Merged PRs (30d)
33

Getting set up

  • Ships a Dockerfile or Docker Compose file
  • No pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from s2-streamstore/cachey

All issues in s2-streamstore/cachey

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.