Storage: add native multi-get support (QMDB + API)

Open
#21 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
30/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Stale
Tech stack
rust
Domain
backend, databases

Research direction

No files or tests are named. Start by tracing the ReadonlyKV/storage-wrapper integration and commonware_storage::qmdb backend, then inspect warm_cache as the first consumer; done means ordered duplicate-safe reads, explicit cache handling, and runnable cold/warm benchmarks with bounded memory.

Written by the indexing model from the issue text.

Description

Context

We experimented with adding multi_get at the ReadonlyKV/storage-wrapper level in evolve_storage, but the current backend (commonware_storage::qmdb in use here) exposes single-key get and write batching, not native read batching.

That means wrapper-level multi_get can reduce some overhead, but remains fundamentally N x single-key reads.

Problem

Hot read paths (cache warming, account/state lookups, RPC-adjacent storage reads) need deterministic, ordered batch reads without paying per-key API/runtime overhead repeatedly.

Proposal

  1. Add native read-side batch API support in storage backend integration:
    • Target API shape in our layer: multi_get(&[Vec<u8>]) -> Vec<Option<Vec<u8>>> (input-order preserving).
    • If upstream QMDB adds a native batch read, use it directly.
    • If not, implement the best possible pipelined/parallel strategy with bounded concurrency and deterministic output ordering.
  2. Keep cache behavior explicit:
    • Resolve cache hits first.
    • Fetch misses in batch path.
    • Backfill positive and negative cache deterministically.
  3. Adopt in first consumer path:
    • cache warming (warm_cache)
    • then one RPC/state query hot path
  4. Add benchmarks + regression gates:
    • compare get loop vs multi_get for cold/warm cache
    • include key cardinality and hit-ratio variants

Acceptance Criteria

  • Deterministic output order for any key vector (including duplicates).
  • No behavior regressions vs existing single-key get.
  • Measurable latency improvement on representative read-heavy workloads.
  • Benchmarks checked into repo and runnable in CI/dev.

Notes

  • This should be done as small reversible steps: API -> backend impl -> one consumer -> benchmarks.
  • Keep memory bounds explicit for any parallel/pipelined implementation.
Dominant language
Rust
Stars
4
Forks
0
PR merge metrics
No merged PRs in 30d

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from evstack/ev-rs

All issues in evstack/ev-rs

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.