Export column families at one snapshot and import them into another database
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 30/100
Research direction
Start with the database binding’s checkpoint and column family import code, especially checkpoint.cpp and db_descriptor.cpp, then trace how column families are registered and tracked in operationsInFlight. Review RocksDB’s SstFileWriter and CreateColumnFamilyWithImport APIs and their metadata requirements. Done means adding consistent multi-column-family export and import with a manifest, safe cleanup and rollback, plus tests for snapshots, blob values, reopen behavior, and concurrent close or drop.
Written by the indexing model from the issue text.
Description
Summary
Add a way to export a chosen set of column families from a live database, all at one point in time, and import them into another database as new column families.
This is needed for Harper's table-scoped backups (HarperFast/harper#4435). A Harper table is spread over several column families: its primary store plus one per indexed attribute. Today db.backup() and db.createCheckpoint() can only copy the whole database.
RocksDB's native column family export can't be used as is
RocksDB has a native pair for this: Checkpoint::ExportColumnFamily and DB::CreateColumnFamilyWithImport. It doesn't work for databases created by rocksdb-js:
- rocksdb-js turns on blob files for every column family. It sets
enable_blob_files = truewithmin_blob_size = 2048(db_descriptor.cpp:243), so any value of 2 KB or more is stored in a blob file and the SST keeps only a reference to it. - The native export copies SST files only, never blob files. In RocksDB 11.13.2 (what rocksdb-js ships), exporting a column family that has blob references succeeds. The imported column family then points at blob files that don't exist in the target database, and reads of those keys fail later. Nothing reports the problem at export or import time.
- Upstream now rejects this case. meta/rocksdb#15652 (on
mainasfd7408e, not yet in a tagged release) makes both calls returnNotSupportedwhen blob references are present. After the next upgrade, the native export fails for any column family holding a value of 2 KB or more. - No consistency across column families.
ExportColumnFamilyflushes and exports one column family per call, and takes no snapshot. Exporting several column families one after another gives a different point in time for each.
Proposal: logical export through SstFileWriter
Export by reading each column family under one snapshot and writing the rows with SstFileWriter. Values are written inline, so the exported files hold no blob references. Import them with CreateColumnFamilyWithImport. Its header lists files from SstFileWriter as a supported input, and upstream confirms that importing files without blob references into a column family with blob files enabled still works.
This solves both problems: there are no blob references, and one snapshot covers every exported column family. The cost is that every row is read and written again instead of hard-linked. That is acceptable here, because a streamed backup is already a full copy.
API sketch
Names are placeholders:
// Takes one snapshot, then writes SST files for each named column family plus
// a JSON manifest to exportDir.
const manifest = await db.exportColumnFamilies(['dog/', 'dog/name', 'dog/breed'], exportDir, {
maxFileSize, // split a column family's rows across several SST files
});
// Creates each column family from the exported files. Rejects if any of the
// names already exists.
await target.importColumnFamilies(exportDir, {
names, // optional subset, or a rename map
moveFiles, // ImportColumnFamilyOptions::move_files
});
Requirements
- One snapshot for every column family in the call. Iterating in key order already gives
SstFileWriterthe sorted input it needs. - A manifest that can be stored and read back later. Import runs in a different process, and possibly on a different host, after the export has gone through a tar archive. Write the per-file metadata
CreateColumnFamilyWithImportneeds to JSON (column family name, comparator name, and for each file its name, key range and sequence numbers), and rebuildExportImportFilesMetaDatafrom it on import. - Imported column families get registered. Once imported, a column family must behave as if
db.use(name)had created it: on other threads, after a reopen, and instats. - Safe against concurrent close and drop. Register both operations in
operationsInFlight, ascreateCheckpointdoes (checkpoint.cpp). Define what happens if a column family being exported is dropped partway through. - Fail cleanly. A failed export removes its partial directory. A failed import leaves no column family behind; RocksDB already guarantees this for a single column family, but a multi-column-family call needs to roll back the ones it already created.
- The transaction log is out of scope. It covers the whole database, and Harper starts a fresh log for a restored table.
Open questions
- Streaming: should export write straight into a tar stream, the way
db.backup(stream)does, instead of to a directory? Directory output is simpler, and Harper can put the files in a tar itself. - A native fast path: keep
ExportColumnFamilyfor column families with no blob references? It would only help tables whose every value is under 2 KB. Probably not worth maintaining two paths. - Sizing: large column families need their output split into several files (
maxFileSize) to keep memory and import time bounded.
🤖 Filed with Claude Code (Claude Opus 5.5)
- Dominant language
- C++
- Stars
- 24
- Forks
- 3
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 94
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from HarperFast/rocksdb-js
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
HarperFast/rocksdb-js#1118 ·
Maintainers usually reply within 1 day
-
CLI crashes with ENOTCONN instead of exiting 0 when stdout closes on macOSPossibly taken @cb1kenobi claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 83/100
HarperFast/rocksdb-js#1112 ·
Maintainers usually reply within 1 day
-
A durable commit is reported as failed when aftercommit hits a database closed after the commitOpen
Difficulty 4/5 3-5 days Newbie friendliness 48/100
HarperFast/rocksdb-js#1117 ·
Maintainers usually reply within 1 day
-
bug
Difficulty 4/5 1-2 days Newbie friendliness 38/100
HarperFast/rocksdb-js#1114 ·
Maintainers usually reply within 1 day
-
Release builds abort the process when an internal native assert failsPossibly taken @cb1kenobi claimed this today. Openbug
HarperFast/rocksdb-js#1113 · 1 assignee ·
Maintainers usually reply within 1 day
All issues in HarperFast/rocksdb-js
Similar issues
-
bug triage
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Maintainers usually reply within 1 day
-
26.04 bug dark-mode Easy Hack impress symptom: visual
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
CollaboraOnline/online#16421 · 1 comment ·
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 63/100
telegramdesktop/tdesktop#31653 ·
Maintainers usually reply within 1 day
-
area/cs bug
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
ydb-platform/ydb#59881 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Maintainers usually reply within 1 day