[Detail Bug] Disk cache silently drops entries for 1 TiB+ objects due to object_size serialization limit
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 55/100
Research direction
Start with CacheValueHeader serialization and deserialization in src/cache.rs, then inspect the object-size flow in src/object_store/downloader.rs and src/service/mod.rs. Run the existing header and range-boundary tests, including the 2^40 case. Done means supported 1 TiB objects persist through the disk cache, with tests covering the boundary and invalid sizes handled clearly.
Written by the indexing model from the issue text.
Description
Detail Bug Report
Introduced in 2c1010efa2295931ede9857d021f0368526e1d63 by @shikhar on Sep 5, 2025
Summary
- Context:
cacheyis an S3 range-read caching service. Each fetched byte range is cached as aCacheValuein a foyer hybrid cache: served from memory, and persisted to the disk cache when the entry is evicted. The addressable object space isPAGE_SIZE (16 MiB) * 65536 pages = 2^40bytes = exactly 1 TiB, delimited byMAX_RANGE_END. - Bug: Disk serialization of
CacheValuerejectsobject_size >= 2^40(5-byte header field), but objects of exactly 1 TiB are fully supported and addressable by the service (larger objects are addressable up to that same limit), so, whenever the disk cache is enabled, every cache entry of any object >= 1 TiB fails to persist to disk and is silently discarded. - Actual vs. expected: The entry is evicted from memory, fails disk encoding inside foyer's block engine, is silently dropped (neither memory nor disk), and only a foyer
warnlog plus a misattributedbuffer_overflowmetric are emitted; expected behavior is either that 1 TiB objects persist like every other object, or that such reads are rejected up front — not a silent permanent disk-cache miss. - Impact: For every object >= 1 TiB served with the disk cache enabled, the disk cache never retains anything: after every eviction all its pages must be re-downloaded from S3 forever, with no error ever surfaced to the operator beyond a cryptic foyer warning per eviction. Memory-only deployments (no
--disk-path) use foyer's NoopEngine, which never invokes the encoder and is unaffected.
Code with Bug
// src/cache.rs
impl CacheValueHeader {
fn new(...) -> Result<Self, &'static str> {
...
if object_size >= (1 << 40) {
return Err("Object size exceeds limit"); // <-- BUG 🔴 rejects 2^40 even though service addresses up to 2^40
}
...
bytes[1..6].copy_from_slice(&object_size.to_be_bytes()[3..]); // only low 5 bytes kept
// src/object_store/downloader.rs (Downloader::read_response)
let object_size = content_range.complete_length; // no upper-bound check
// src/service/mod.rs (EntryGetExecutor::execute)
Ok::<_, ServiceError>(CacheValue {
...
object_size: out.piece.object_size, // <-- BUG 🔴 can be >= 2^40 and will later fail disk encoding
...
})
// src/service/mod.rs
pub const PAGE_SIZE: u64 = 16 * 1024 * 1024;
pub const MAX_RANGE_END: u64 = PAGE_SIZE * (PageId::MAX as u64 + 1); // = 2^40 = 1 TiB
// foyer-storage-0.22.6/src/engine/block/buffer.rs
let info = match EntrySerializer::serialize(key, value, compression, ...) {
Ok(info) => info,
Err(e) => {
if e.kind() != ErrorKind::BufferSizeLimit {
tracing::warn!(?e, "[blob writer]: serialize entry kv error");
}
return false; // <-- BUG 🔴 encoding failure drops entry; nothing propagates to cachey
}
};
Explanation
- The on-disk
CacheValueHeaderstoresobject_sizein 40 bits and explicitly errors onobject_size >= 2^40. - The service can fetch and address objects up to exactly
2^40bytes (MAX_RANGE_END), andobject_sizeflows from S3Content-RangeintoCacheValuewithout validation. - With disk cache enabled and
WriteOnEviction, the value is evicted from memory, then encoding fails during foyer’s flush. Foyer logs a warning and returnsfalsefrom its buffer append path, silently dropping the entry; no API surface returns an error (insert/eviction pipeline returns(),wait()has no error). - Repro evidence showed
object_size = 2^40 - 1round-trips to disk, whileobject_size = 2^40is evicted and thencache.get()returnsNone, confirming the entry is lost from both tiers.
Codebase Inconsistency
MAX_RANGE_ENDand existing tests assert that ranges covering a full 1 TiB object are valid, but the disk format forbidsobject_size == 2^40. This makes disk caching permanently ineffective specifically for 1 TiB objects, despite the rest of the system treating them as supported.
Recommended Fix
- Widen the serialized
object_sizefield to include the2^40value (e.g., store an extra bit by repurposing the currently-unused reserved bit in the header flags byte, and relax the corresponding validation inCacheValueHeader::from_bytes). - Additionally, validate
object_sizewhen creatingCacheValue(e.g. inEntryGetExecutor::execute) so an out-of-range size fails early with a clear error instead of silently degrading disk cache behavior.
History
This bug was introduced in commit 2c1010e (the initial "init" commit of the repository). That commit created the on-disk CacheValueHeader format with a 5-byte (40-bit) object_size field and an explicit if object_size >= (1 << 40) { return Err("Object size exceeds limit") } check in CacheValueHeader::new — even shipping a test asserting 1 << 40 is rejected — while simultaneously flowing S3's Content-Range complete-length into CacheValue unvalidated and addressing an object space of PAGE_SIZE * 65535 = 1 TiB - 16 MiB, so the boundary sat exactly at the top of the intended space with no up-front validation; the failure only ever surfaced later inside foyer's storage encode path.
- Dominant language
- Rust
- Stars
- 622
- Forks
- 18
- Avg merge
- 5h 2m
- Merged PRs (30d)
- 33
Getting set up
- Ships a Dockerfile or Docker Compose file
- No pull request template
- No contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from s2-streamstore/cachey
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
s2-streamstore/cachey#225 ·
Maintainers usually reply within 1 day
-
Difficulty 3/5 1-2 days Newbie friendliness 68/100
s2-streamstore/cachey#224 ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 52/100
s2-streamstore/cachey#223 ·
Maintainers usually reply within 1 day
-
Difficulty 3/5 1-2 days Newbie friendliness 71/100
s2-streamstore/cachey#222 ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 15/100
s2-streamstore/cachey#6 · 2 comments ·
Maintainers usually reply within 1 day
All issues in s2-streamstore/cachey
Similar issues
-
<input type="number"> misses text-entry leaf sizing in layoutPossibly taken @GoCoder7 claimed this today. Open
Difficulty 1/5 Under an hour Newbie friendliness 88/100
DioxusLabs/blitz#1675 ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
metasequoiaime/msime#11666 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 77/100
Maintainers usually reply within 1 day
-
bug drawings help wanted ui
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
userFRM/egui-charts#3479 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
spellshift/realm#2509 ·
Maintainers usually reply within 1 day