Add S3 storage backend for indexed blocks

Open
#29 0 comments 0 reactions 1 assignee View on GitHub

@pthmas is already working on this.

Since Mar 24, 2026.

Assessment

This issue has not been assessed yet.

Description

Motivation

Currently all indexed data is written directly to PostgreSQL and served from there. Adding S3 as a storage backend gives us:

  • Redundancy — blocks, transactions, and address data persisted to durable object storage independent of the database
  • Cost efficiency — cold data in S3 is significantly cheaper than keeping everything in PG at scale
  • Disaster recovery — can rebuild the database from S3 if needed

Proposal

Write path

The indexer writes all indexed data (blocks, transactions, address state) to both PostgreSQL (as today) and S3 (new). S3 writes can be async/best-effort with retries so they don't block the indexing pipeline.

Read path — in-memory cache for recent data

Hold the last N blocks (and their transactions) in an in-memory cache. Serve recent queries directly from memory, bypassing both PG and S3. This covers the hot path — most explorer traffic hits recent data.

Cache eviction: LRU or ring buffer keyed by block number.

Read path — fallback
  1. Check in-memory cache
  2. Miss → query PostgreSQL (current behavior)
  3. Optionally: if PG is down/slow, fall back to S3
Open questions
  • S3 object layout — one object per block (including its txs)? Separate prefixes for blocks/txs/addresses? Batched ranges?
  • Serialization format — JSON, bincode, protobuf?
  • Consistency — how to handle S3 write failures? Retry queue? Reconciliation job?
  • Cache size — what's a reasonable default for N? Should it be memory-bounded (bytes) or count-bounded (blocks)?
  • Existing data backfill — do we need a one-time migration to populate S3 from PG?
  • Address data granularity — store snapshots per block or cumulative state?
  • Configuration — S3_BUCKET, S3_REGION, S3_PREFIX, BLOCK_CACHE_SIZE env vars
Dominant language
Rust
Stars
2
Forks
1
No merged PRs in 30d

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from evstack/atlas

All issues in evstack/atlas

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.