feat: pipelined fetch/write for indexer
@pthmas is already working on this.
Since Apr 14, 2026.
Assessment
This issue has not been assessed yet.
Description
Summary
Currently the indexer fetches a batch of blocks, then writes them to the database, then fetches the next batch. The RPC is idle during writes and the DB is idle during fetches. Pipelining would overlap these operations: fetch batch N+1 while writing batch N.
Motivation
This is the single biggest throughput improvement available for indexing speed. Based on analysis of tidx, which implements this pattern, estimated 30-50% faster sync during both initial sync and steady-state head tracking.
Design
- After fetching batch N, spawn the DB write as a background task
- Immediately begin fetching batch N+1
- Before writing batch N+1, await completion of the batch N write
- Errors in either fetch or write should still propagate correctly
Key considerations:
- The reorder buffer (
BTreeMap) already collects results in sequence — pipelining fits naturally after the reorder step - The binary COPY connection is exclusive during writes, so only one write can be in-flight at a time
- SSE publishing (via
HeadTracker) currently happens before the DB write, so pipelining doesn't affect SSE timing
References
- tidx implementation:
sync/engine.rs—tick_realtimefetches batch N+1 while writing batch N
- Dominant language
- Rust
- Stars
- 2
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Getting set up
- Ships a Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from evstack/atlas
-
enhancement
Difficulty 3/5 1-2 days Newbie friendliness 68/100
-
Difficulty 4/5 3-5 days Newbie friendliness 55/100
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
-
feat: reorg handlingOpen
Difficulty 5/5 Over a week Newbie friendliness 25/100
-
feat: admin APIOpen
Difficulty 4/5 3-5 days Newbie friendliness 55/100
Similar issues
-
Change output crossing a compactsize boundary leaves the fee slightly below the requested feerateOpenbug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
bitcoindevkit/bdk_wallet#578 ·
Maintainers usually reply within 8 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
Maintainers usually reply within 2 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
521xueweihan/HelloGitHub#3832 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
canonical/opentelemetry-collector-operator#409 ·
Maintainers usually reply within 1 day