feat: optional ClickHouse dual-sink for analytics
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 25/100
Research direction
Start by reading the referenced sync/sink.rs, sync/ch_sink.rs, and clickhouse.rs implementations from tidx, then inspect Atlas's existing sink and analytics entry points. Break the work into the dual-sink writer, ClickHouse schema and routing, backfill tool, configuration, and Docker Compose profile. Done means optional ClickHouse writes, reads, retries, backfill, and deployment configuration satisfy the listed requirements without changing PostgreSQL point lookups.
Written by the indexing model from the issue text.
Description
Summary
Add optional ClickHouse support as a parallel write sink alongside PostgreSQL. PG continues to handle point lookups (get block, get tx by hash); ClickHouse handles analytical queries (aggregations, time-range scans, charts, top-N rankings).
Motivation
Atlas's analytics endpoints (daily tx charts, gas price history, token holder rankings) all hit PostgreSQL today. As chain history grows into hundreds of millions of rows, these queries will degrade — ClickHouse is purpose-built for this workload and can be orders of magnitude faster for full-column scans and aggregations.
Design
Dual-Sink Writer
- Add a
ClickHouseSinkthat writes blocks, txs, logs, and receipts using theclickhousecrate (RowBinary format + LZ4 compression) - Wrap PG + CH sinks in a
SinkSetabstraction that fans out writes viatokio::try_join!(parallel writes, both must succeed) - CH is optional — enabled via
ENABLE_CLICKHOUSE=true+CLICKHOUSE_URL - CH sink should chunk large batches (e.g., 10K rows per INSERT) to bound memory
- Use
ReplacingMergeTreeengine so re-inserts after crashes are safe (duplicates merged away) - CH sink gets its own retry logic (3 attempts, exponential backoff)
ClickHouse Schema
- Mirror the core PG tables:
blocks,transactions,event_logs,receipts - Partition by month:
toYYYYMM(timestamp)for efficient time-range pruning - Bloom filter indexes on
event_logs(topic0, address)for fast event filtering - Order keys optimized for analytical access patterns (e.g., logs ordered by
(address, topic0, block_number))
Query Routing
- Analytics API endpoints (charts, top holders, aggregations) route to ClickHouse when available
- Point lookups (get block by number, get tx by hash) stay on PostgreSQL
- Could expose an
?engine=clickhousequery param for explicit routing
PG → CH Backfill
- One-time migration tool that reads historical data from PG and bulk-loads into CH
- Allows adding ClickHouse to an existing Atlas deployment without re-indexing from the chain
- Should run as a background task with configurable concurrency and throttling
Configuration
| Var | Default | Description |
|---|---|---|
ENABLE_CLICKHOUSE |
false |
Enable ClickHouse dual-sink |
CLICKHOUSE_URL |
— | ClickHouse HTTP endpoint (required if enabled) |
CLICKHOUSE_DATABASE |
atlas |
Database name |
CLICKHOUSE_USER |
default |
Auth user |
CLICKHOUSE_PASSWORD |
— | Auth password |
CH_INSERT_CHUNK_SIZE |
10000 |
Rows per INSERT batch |
Docker Compose
- Add optional ClickHouse service (clickhouse/clickhouse-server)
- Only started when ENABLE_CLICKHOUSE=true profile is active
References
- tidx implementation:
sync/sink.rs(SinkSet fan-out),sync/ch_sink.rs(ClickHouse writer with chunked inserts + retry),clickhouse.rs(read engine with failover) - tidx uses
ReplacingMergeTree+ monthly partitioning + bloom filter indexes on logs
- Dominant language
- Rust
- Stars
- 2
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from evstack/atlas
-
enhancement
Difficulty 3/5 1-2 days Newbie friendliness 68/100
-
Difficulty 4/5 3-5 days Newbie friendliness 55/100
-
feat: reorg handling Open
Difficulty 5/5 Over a week Newbie friendliness 25/100
-
feat: admin API Open
Difficulty 4/5 3-5 days Newbie friendliness 55/100
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
Eynzof/Hermes-CN-Desktop#610 ·
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
gitbutlerapp/gitbutler#15998 · 1 comment ·
-
bug triage:deciding
Difficulty 1/5 Under an hour Newbie friendliness 88/100
open-telemetry/otel-arrow#4132 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100