Expose RocksDB MultiGet: read several keys in one call
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 45/100
Research direction
Start by reading the existing get/getSync bindings for Store, Database, and Transaction, plus src/stats.ts for the exposed MultiGet tickers; then check RocksDB's MultiGet and transaction APIs. Define the batch behavior against the acceptance cases for snapshots, uncommitted writes, deletions, duplicates, and synchronous cached reads. Done means the batch API covers the requested variants and a microbenchmark compares per-key reads with warm and cold cache.
Written by the indexing model from the issue text.
Description
Summary
rocksdb-js has no way to read several keys in one call. Store, Database and Transaction expose get / getSync / getBinary* for a single key, plus getRange. RocksDB's MultiGet isn't bound. Its statistics tickers are already surfaced (rocksdb.db.multiget.micros, rocksdb.multiget.io.batch.size, … in src/stats.ts), but nothing in the binding can call it.
I'd like a batched point read, so Harper can offer a multi-key Table read (HarperFast/harper#3428) without paying the per-key binding cost N times.
Why
The use case is a request that does many independent primary-key reads. In our ecommerce reference app (ecommerce-store), a cart quote does 60–150 point reads (variants, products, and SKU × location inventory rows), and a product view fill does a few dozen. Today each key costs:
- a separate key encode and N-API crossing,
- a separate memtable and SST lookup,
- on a block-cache miss, its own queued async read and completion callback.
MultiGet amortizes the per-call work across the batch and lets RocksDB group the lookups (sorted keys, shared memtable/SST passes, batched I/O for the misses). The separated stack we compare against batches the same reads with Redis MGET and set-based SQL. The details are in ecommerce-store#19 and its write-up in ecommerce-store#22.
Proposed API (open to a different shape)
getMany(keys, options?):MaybePromise<Array<value | undefined>>in key order, withundefinedfor a missing key. Same contract asget: if every key is in memory, return synchronously; otherwise one promise for the batch, not one per key.getManySync(keys, options?)- Binary variants mirroring
getBinary/getBinaryFast, if that's how Harper's record decoder wants them. - On
Transactiontoo, reading through the transaction's snapshot and its own uncommitted writes. RocksDB hasTransaction::MultiGetandWriteBatchWithIndex::MultiGetFromBatchAndDBfor this.
Acceptance
- Inside a transaction,
getMany(keys)returns exactly whatkeys.map(get)returns, including its own uncommitted writes, deleted keys, and duplicate keys in the input. - Fully cached batches resolve synchronously; a batch with misses settles once.
- A microbenchmark comparing per-key cost against N
gets, with the keys in the block cache and with the cache cold.
Out of scope
The Harper Table API on top of this is tracked in HarperFast/harper#3428. No benchmark result depends on this yet. It removes per-call work we can identify in the source, but nobody has measured that work's share of a request.
sent with Claude Opus 5.5
- Dominant language
- C++
- Stars
- 24
- Forks
- 3
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 49
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from HarperFast/rocksdb-js
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
HarperFast/rocksdb-js#945 ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 55/100
HarperFast/rocksdb-js#956 ·
Maintainers usually reply within 1 day
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
HarperFast/rocksdb-js#955 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 1-2 days Newbie friendliness 45/100
HarperFast/rocksdb-js#946 ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 25/100
HarperFast/rocksdb-js#944 ·
Maintainers usually reply within 1 day
All issues in HarperFast/rocksdb-js
Similar issues
-
Remove source files from packaging for deb, zip, etcPossibly taken A pull request linked to this issue is open or already merged. Open
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
OpenSpace/OpenSpace#4353 · 1 comment ·
Maintainers usually reply within 5 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
justcallmekoko/ESP32Marauder#1656 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Decompiler: convertCharRef() in xml.y overflows its int4 accumulator on long character referencesPossibly taken @kaayzouee claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
NationalSecurityAgency/ghidra#10009 ·
Maintainers usually reply within 2 days