Storage: add native multi-get support (QMDB + API)
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 30/100
Research direction
No files or tests are named. Start by tracing the ReadonlyKV/storage-wrapper integration and commonware_storage::qmdb backend, then inspect warm_cache as the first consumer; done means ordered duplicate-safe reads, explicit cache handling, and runnable cold/warm benchmarks with bounded memory.
Written by the indexing model from the issue text.
Description
Context
We experimented with adding multi_get at the ReadonlyKV/storage-wrapper level in evolve_storage, but the current backend (commonware_storage::qmdb in use here) exposes single-key get and write batching, not native read batching.
That means wrapper-level multi_get can reduce some overhead, but remains fundamentally N x single-key reads.
Problem
Hot read paths (cache warming, account/state lookups, RPC-adjacent storage reads) need deterministic, ordered batch reads without paying per-key API/runtime overhead repeatedly.
Proposal
- Add native read-side batch API support in storage backend integration:
- Target API shape in our layer:
multi_get(&[Vec<u8>]) -> Vec<Option<Vec<u8>>>(input-order preserving). - If upstream QMDB adds a native batch read, use it directly.
- If not, implement the best possible pipelined/parallel strategy with bounded concurrency and deterministic output ordering.
- Target API shape in our layer:
- Keep cache behavior explicit:
- Resolve cache hits first.
- Fetch misses in batch path.
- Backfill positive and negative cache deterministically.
- Adopt in first consumer path:
- cache warming (
warm_cache) - then one RPC/state query hot path
- cache warming (
- Add benchmarks + regression gates:
- compare
getloop vsmulti_getfor cold/warm cache - include key cardinality and hit-ratio variants
- compare
Acceptance Criteria
- Deterministic output order for any key vector (including duplicates).
- No behavior regressions vs existing single-key
get. - Measurable latency improvement on representative read-heavy workloads.
- Benchmarks checked into repo and runnable in CI/dev.
Notes
- This should be done as small reversible steps: API -> backend impl -> one consumer -> benchmarks.
- Keep memory bounds explicit for any parallel/pipelined implementation.
- Dominant language
- Rust
- Stars
- 4
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from evstack/ev-rs
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
-
enhancement
Difficulty 5/5 Over a week Newbie friendliness 25/100
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
-
Difficulty 5/5 Over a week Newbie friendliness 20/100
-
enhancement
Difficulty 5/5 Over a week Newbie friendliness 20/100
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
Eynzof/Hermes-CN-Desktop#610 ·
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
gitbutlerapp/gitbutler#15998 · 1 comment ·
-
bug triage:deciding
Difficulty 1/5 Under an hour Newbie friendliness 88/100
open-telemetry/otel-arrow#4132 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100