[Detail Bug] Disk cache silently drops entries for 1 TiB+ objects due to object_size serialization limit

Open
#217 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
55/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
rust
Domain
backend

Research direction

Start with CacheValueHeader serialization and deserialization in src/cache.rs, then inspect the object-size flow in src/object_store/downloader.rs and src/service/mod.rs. Run the existing header and range-boundary tests, including the 2^40 case. Done means supported 1 TiB objects persist through the disk cache, with tests covering the boundary and invalid sizes handled clearly.

Written by the indexing model from the issue text.

Description

Detail Bug Report

https://app.detail.dev/org_89d327b3-b883-4365-b6a3-46b6701342a9/bugs/bug_ac89f23a-a7cc-4bbe-ae37-0805d290ef3a

Introduced in 2c1010efa2295931ede9857d021f0368526e1d63 by @shikhar on Sep 5, 2025

Summary

  • Context: cachey is an S3 range-read caching service. Each fetched byte range is cached as a CacheValue in a foyer hybrid cache: served from memory, and persisted to the disk cache when the entry is evicted. The addressable object space is PAGE_SIZE (16 MiB) * 65536 pages = 2^40 bytes = exactly 1 TiB, delimited by MAX_RANGE_END.
  • Bug: Disk serialization of CacheValue rejects object_size >= 2^40 (5-byte header field), but objects of exactly 1 TiB are fully supported and addressable by the service (larger objects are addressable up to that same limit), so, whenever the disk cache is enabled, every cache entry of any object >= 1 TiB fails to persist to disk and is silently discarded.
  • Actual vs. expected: The entry is evicted from memory, fails disk encoding inside foyer's block engine, is silently dropped (neither memory nor disk), and only a foyer warn log plus a misattributed buffer_overflow metric are emitted; expected behavior is either that 1 TiB objects persist like every other object, or that such reads are rejected up front — not a silent permanent disk-cache miss.
  • Impact: For every object >= 1 TiB served with the disk cache enabled, the disk cache never retains anything: after every eviction all its pages must be re-downloaded from S3 forever, with no error ever surfaced to the operator beyond a cryptic foyer warning per eviction. Memory-only deployments (no --disk-path) use foyer's NoopEngine, which never invokes the encoder and is unaffected.

Code with Bug

// src/cache.rs
impl CacheValueHeader {
    fn new(...) -> Result<Self, &'static str> {
        ...
        if object_size >= (1 << 40) {
            return Err("Object size exceeds limit"); // <-- BUG 🔴 rejects 2^40 even though service addresses up to 2^40
        }
        ...
        bytes[1..6].copy_from_slice(&object_size.to_be_bytes()[3..]); // only low 5 bytes kept
// src/object_store/downloader.rs (Downloader::read_response)
let object_size = content_range.complete_length; // no upper-bound check
// src/service/mod.rs (EntryGetExecutor::execute)
Ok::<_, ServiceError>(CacheValue {
    ...
    object_size: out.piece.object_size, // <-- BUG 🔴 can be >= 2^40 and will later fail disk encoding
    ...
})
// src/service/mod.rs
pub const PAGE_SIZE: u64 = 16 * 1024 * 1024;
pub const MAX_RANGE_END: u64 = PAGE_SIZE * (PageId::MAX as u64 + 1); // = 2^40 = 1 TiB
// foyer-storage-0.22.6/src/engine/block/buffer.rs
let info = match EntrySerializer::serialize(key, value, compression, ...) {
    Ok(info) => info,
    Err(e) => {
        if e.kind() != ErrorKind::BufferSizeLimit {
            tracing::warn!(?e, "[blob writer]: serialize entry kv error");
        }
        return false; // <-- BUG 🔴 encoding failure drops entry; nothing propagates to cachey
    }
};

Explanation

  • The on-disk CacheValueHeader stores object_size in 40 bits and explicitly errors on object_size >= 2^40.
  • The service can fetch and address objects up to exactly 2^40 bytes (MAX_RANGE_END), and object_size flows from S3 Content-Range into CacheValue without validation.
  • With disk cache enabled and WriteOnEviction, the value is evicted from memory, then encoding fails during foyer’s flush. Foyer logs a warning and returns false from its buffer append path, silently dropping the entry; no API surface returns an error (insert/eviction pipeline returns (), wait() has no error).
  • Repro evidence showed object_size = 2^40 - 1 round-trips to disk, while object_size = 2^40 is evicted and then cache.get() returns None, confirming the entry is lost from both tiers.

Codebase Inconsistency

  • MAX_RANGE_END and existing tests assert that ranges covering a full 1 TiB object are valid, but the disk format forbids object_size == 2^40. This makes disk caching permanently ineffective specifically for 1 TiB objects, despite the rest of the system treating them as supported.

Recommended Fix

  • Widen the serialized object_size field to include the 2^40 value (e.g., store an extra bit by repurposing the currently-unused reserved bit in the header flags byte, and relax the corresponding validation in CacheValueHeader::from_bytes).
  • Additionally, validate object_size when creating CacheValue (e.g. in EntryGetExecutor::execute) so an out-of-range size fails early with a clear error instead of silently degrading disk cache behavior.

History

This bug was introduced in commit 2c1010e (the initial "init" commit of the repository). That commit created the on-disk CacheValueHeader format with a 5-byte (40-bit) object_size field and an explicit if object_size >= (1 << 40) { return Err("Object size exceeds limit") } check in CacheValueHeader::new — even shipping a test asserting 1 << 40 is rejected — while simultaneously flowing S3's Content-Range complete-length into CacheValue unvalidated and addressing an object space of PAGE_SIZE * 65535 = 1 TiB - 16 MiB, so the boundary sat exactly at the top of the intended space with no up-front validation; the failure only ever surfaced later inside foyer's storage encode path.

Dominant language
Rust
Stars
622
Forks
18
Avg merge
5h 2m
Merged PRs (30d)
33

Getting set up

  • Ships a Dockerfile or Docker Compose file
  • No pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from s2-streamstore/cachey

All issues in s2-streamstore/cachey

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.