ghac: cache writes are dropped on HTTP 429 because GHACache::build installs no RetryLayer
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 78/100
Research direction
Start in src/cache/gha.rs at GHACache::build and inspect the opendal dependency declarations, including the existing logging layer setup. Add the published retry layer dependency and configure it on this Operator; done means temporary HTTP 429 cache-write errors are retried while non-temporary failures remain errors.
Written by the indexing model from the issue text.
Description
GHACache::build (src/cache/gha.rs) builds its opendal Operator with only a logging layer:
let op = Operator::new(builder)?
.with_context(OperationContext::new().with_http_transport(set_user_agent()))
.layer(LoggingLayer::default());
GitHub's Actions cache service rate limits per workflow run, shared across every job in it, and
answers a burst of writes with HTTP 429. opendal marks that error temporary, but RetryLayer
appears nowhere in the repository, so nothing acts on it: sccache increments cache_write_errors
and the compilation result is never stored. Confirmed unchanged on main as of 2026-08-23 and on
v0.17.0.
What it costs
A dropped write is not slow, it is gone, until that object is recompiled and re-offered on some
later run that happens to win the rate limit.
We moved a six-job CI matrix from a tarball cache to SCCACHE_GHA_ENABLED=on. Two consecutive
runs, same code, all six jobs writing at once (run 1,
run 2). Run 2's hit rate reads out
what run 1 managed to store:
| job family | run 1 stored / attempted | run 2 Rust hit rate |
|---|---|---|
| windows-latest | 337 / 610 (55%) | 41.29% |
| macos-latest | 369 / 905 (41%) | 39.20% |
| ubuntu-24.04-arm | 274 / 895 (31%) | 34.37% |
| Android (cargo-ndk) | 128 / 819 (16%) | 23.92% |
Because sccache keys on the whole compiler invocation, each family's objects are private to it
(different target triples, cargo check vs cargo build, different RUSTFLAGS), so a write one
job loses is never covered by another job's write. The loss rate is an upper bound on the
achievable hit rate, not a warm-up effect.
The control: one job family ran ghac alone for seven consecutive runs before the others migrated,
and sustained 96.79% Rust hit rate with zero write errors, read errors, cache errors and timeouts
in every one of them. Same repo, same sccache version, same backend. Adding five concurrent
writers took write loss from 0% to 45-90%.
Why it can't be worked around
SCCACHE_GHA_RW_MODE only turns writing off. The limit is shared across the whole run, so no
single job can observe it or throttle against it, and retrying at the CI level is impossible
because the put happens inside the compile request.
Suggested fix
sccache pins opendal 0.58.1 and already pulls layers as separate crates (opendal-layer-logging).
opendal-layer-retry is published at the same version, so this is a dependency plus a layer:
.layer(RetryLayer::new().with_jitter())
RetryLayer retries only errors opendal marks temporary, so it does not paper over real failures.
Given the service asks for retry-after: 1, even a small max_times should recover most of
these. The default matters more than a tuning knob: the failure is silent, so people are far more
likely to report "GHA cache builds are surprisingly slow" than "my cache is 40% smaller than it
should be".
Possibly related: #1485 reports the same 429s against the older v1 cache service, diagnosed as far
as Cache write error: ... 429 Too Many Requests, and never closed.
Environment
sccache 0.17.0, SCCACHE_GHA_ENABLED=on, ACTIONS_CACHE_SERVICE_V2=true, SCCACHE_DIRECT=true,
GitHub-hosted runners: ubuntu-latest, ubuntu-24.04-arm, macos-latest, windows-latest.
- Dominant language
- Rust
- Stars
- 7.7k
- Forks
- 748
- Avg merge
- 4d 7h
- Merged PRs (30d)
- 21
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from mozilla/sccache
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
-
generate-jwt-hs256-key can produce a value that breaks generate-jwt-hs256-server-token CLI parsing Open
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
Eynzof/Hermes-CN-Desktop#610 ·
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
gitbutlerapp/gitbutler#15998 · 1 comment ·
-
bug triage:deciding
Difficulty 1/5 Under an hour Newbie friendliness 88/100
open-telemetry/otel-arrow#4132 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100