Export column families at one snapshot and import them into another database

Open
#1,111 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
30/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
cpp, node.js
Domain
database

Research direction

Start with the database binding’s checkpoint and column family import code, especially checkpoint.cpp and db_descriptor.cpp, then trace how column families are registered and tracked in operationsInFlight. Review RocksDB’s SstFileWriter and CreateColumnFamilyWithImport APIs and their metadata requirements. Done means adding consistent multi-column-family export and import with a manifest, safe cleanup and rollback, plus tests for snapshots, blob values, reopen behavior, and concurrent close or drop.

Written by the indexing model from the issue text.

Description

enhancement

Summary

Add a way to export a chosen set of column families from a live database, all at one point in time, and import them into another database as new column families.

This is needed for Harper's table-scoped backups (HarperFast/harper#4435). A Harper table is spread over several column families: its primary store plus one per indexed attribute. Today db.backup() and db.createCheckpoint() can only copy the whole database.

RocksDB's native column family export can't be used as is

RocksDB has a native pair for this: Checkpoint::ExportColumnFamily and DB::CreateColumnFamilyWithImport. It doesn't work for databases created by rocksdb-js:

  • rocksdb-js turns on blob files for every column family. It sets enable_blob_files = true with min_blob_size = 2048 (db_descriptor.cpp:243), so any value of 2 KB or more is stored in a blob file and the SST keeps only a reference to it.
  • The native export copies SST files only, never blob files. In RocksDB 11.13.2 (what rocksdb-js ships), exporting a column family that has blob references succeeds. The imported column family then points at blob files that don't exist in the target database, and reads of those keys fail later. Nothing reports the problem at export or import time.
  • Upstream now rejects this case. meta/rocksdb#15652 (on main as fd7408e, not yet in a tagged release) makes both calls return NotSupported when blob references are present. After the next upgrade, the native export fails for any column family holding a value of 2 KB or more.
  • No consistency across column families. ExportColumnFamily flushes and exports one column family per call, and takes no snapshot. Exporting several column families one after another gives a different point in time for each.

Proposal: logical export through SstFileWriter

Export by reading each column family under one snapshot and writing the rows with SstFileWriter. Values are written inline, so the exported files hold no blob references. Import them with CreateColumnFamilyWithImport. Its header lists files from SstFileWriter as a supported input, and upstream confirms that importing files without blob references into a column family with blob files enabled still works.

This solves both problems: there are no blob references, and one snapshot covers every exported column family. The cost is that every row is read and written again instead of hard-linked. That is acceptable here, because a streamed backup is already a full copy.

API sketch

Names are placeholders:

// Takes one snapshot, then writes SST files for each named column family plus
// a JSON manifest to exportDir.
const manifest = await db.exportColumnFamilies(['dog/', 'dog/name', 'dog/breed'], exportDir, {
	maxFileSize, // split a column family's rows across several SST files
});

// Creates each column family from the exported files. Rejects if any of the
// names already exists.
await target.importColumnFamilies(exportDir, {
	names, // optional subset, or a rename map
	moveFiles, // ImportColumnFamilyOptions::move_files
});
Requirements
  • One snapshot for every column family in the call. Iterating in key order already gives SstFileWriter the sorted input it needs.
  • A manifest that can be stored and read back later. Import runs in a different process, and possibly on a different host, after the export has gone through a tar archive. Write the per-file metadata CreateColumnFamilyWithImport needs to JSON (column family name, comparator name, and for each file its name, key range and sequence numbers), and rebuild ExportImportFilesMetaData from it on import.
  • Imported column families get registered. Once imported, a column family must behave as if db.use(name) had created it: on other threads, after a reopen, and in stats.
  • Safe against concurrent close and drop. Register both operations in operationsInFlight, as createCheckpoint does (checkpoint.cpp). Define what happens if a column family being exported is dropped partway through.
  • Fail cleanly. A failed export removes its partial directory. A failed import leaves no column family behind; RocksDB already guarantees this for a single column family, but a multi-column-family call needs to roll back the ones it already created.
  • The transaction log is out of scope. It covers the whole database, and Harper starts a fresh log for a restored table.
Open questions
  • Streaming: should export write straight into a tar stream, the way db.backup(stream) does, instead of to a directory? Directory output is simpler, and Harper can put the files in a tar itself.
  • A native fast path: keep ExportColumnFamily for column families with no blob references? It would only help tables whose every value is under 2 KB. Probably not worth maintaining two paths.
  • Sizing: large column families need their output split into several files (maxFileSize) to keep memory and import time bounded.

🤖 Filed with Claude Code (Claude Opus 5.5)

Dominant language
C++
Stars
24
Forks
3
Avg merge
1d 17h
Merged PRs (30d)
94

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from HarperFast/rocksdb-js

All issues in HarperFast/rocksdb-js

Similar issues

More C++ issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.