Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Bound cache staleness with the ETag witness, in both auth modes

Open
  • Is something wrong?
#72 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
42/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
aws, rust
Domain
backend, cloud, infrastructure, performance

Research direction

Start with crates/pacer-cache/src/slot.rs and crates/pacer-cache/src/store.rs, especially SlotHeader and verified(), to understand how ETags survive disk demotion. Then trace FillCtx::to_cached and FillCtx::witness_matches and the node-mode read path. Done means node-mode cached reads perform the configured per-object check, refresh or evict on the stated outcomes, and preserve the witness through the append-only slot format.

Written by the indexing model from the issue text.

Description

Requester mode already has the mechanism this needs: every GET carries a probe to S3 under
the caller's signature (ADR-0041 § 1), the chunk is tagged with the ETag that probe returned
(FillCtx::to_cached), and a read serves a cached chunk only when its tag matches the
current probe's ETag (FillCtx::witness_matches). A mismatch is a miss, re-read and
re-filled under the new tag. That already bounds staleness to zero for requester mode: it is
checked on every GET, not on a timer.

Node mode has no such check. ADR-0007/0044/0046 make an in-cluster Invalidate reach every
chunk this node holds, but that is the only way a stale copy is ever removed — a write that
reaches S3 around PACER, or an invalidation that failed and was only logged (ADR-0007's
"bounded stale window... self-limiting" argument), leaves a chunk cached until eviction, which
for a hot, rarely-evicted checkpoint chunk can be indefinite. That is not eventual consistency;
it is consistency with no bound at all.

Proposed fix: extend the requester-mode witness to node mode, sourcing the "current" ETag
differently since node mode has no per-GET probe:

  • Add a per-object last_checked to the cached header (ADR-0015/0042). On a chunk hit whose
    header is older than PACER_STALE_CHECK_PERIOD_SECS (default 86,400 — 24h), issue one
    HeadObject, compare its ETag to the header's, and only then serve.
  • Default 24h, not the shorter refresh intervals elsewhere in this codebase (the 300s
    reannounce interval, EFA client TTL): the target workload is WORM checkpoint objects, where a
    rewrite is the rare, deliberate case rather than the common one, and this codebase already
    uses 24h as a ceiling for a similar knob (MAX_CLIENT_TTL_SECS). A shorter default would pay
    a HeadObject per object per period against multi-GiB checkpoint chunks for a rewrite that
    is not expected to happen.
  • A separate PACER_STALE_CHECK_ENABLED (default true) turns the check off entirely, for a
    workload that is not WORM or wants ADR-0007's invalidation-only guarantee unchanged. Kept as
    its own boolean rather than a sentinel value of the period (0 already means "use the
    default" for every duration-with-a-default env var in this codebase — PACER_REANNOUNCE_INTERVAL_SECS,
    PACER_SCATTER_STAGING_TTL_SECS — and overloading it here to also mean "never" would break
    that convention for one knob only). Parsed with the same off-string set as
    PACER_DELIVERY_ENABLED/PACER_RDMA_AFFINITY ("0"|"false"|"off"|"no", ADR-0026).
  • On a match: refresh last_checked, serve from cache, no extra chunk I/O.
  • On a mismatch: update the header to the new ETag/length, evict this node's chunks of the
    object (the length-derived range ADR-0042 already computes), and serve this read as a miss.
    No refill beyond what the miss path already does — the new version repopulates from ordinary
    reads, not eagerly.
  • This is per-node and per-object; nothing fans out. Each node catches its own stale copy at
    its own next check, same as requester mode's per-GET probe.

This closes on its own the layer-1 admit race on a non-home node (a stale copy admitted from a
peer that itself raced a write, outside the sharer set an Invalidate reaches) without needing
a fence for that specific race.

Blocking prerequisite: ChunkStore's on-disk SlotHeader (crates/pacer-cache/src/slot.rs)
has no ETag field, so any chunk demoted to disk loses its witness and rebuilds as
CachedChunk::new (crates/pacer-cache/src/store.rs, verified()) — untagged, and in
requester mode a permanent miss once demoted. The slot format needs the ETag added
(append-only, per ADR-0045's precedent, so an old build still decodes and treats it as a miss
rather than panicking). See #73 for whether this is already costing hit rate today.

24h is a proposed default for a WORM-shaped workload, not a measured one — open to discussion
for a workload that rewrites more often.

Dominant language
Rust
Stars
16
Forks
1
Avg merge
7h 19m
Merged PRs (30d)
29

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from aws-samples/sample-pacer

All issues in aws-samples/sample-pacer

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.