Export column families at one snapshot and import them into another database

Aperta
#1,111 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
30/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
cpp, node.js
Ambito
database

Direzione di ricerca

Start with the database binding’s checkpoint and column family import code, especially checkpoint.cpp and db_descriptor.cpp, then trace how column families are registered and tracked in operationsInFlight. Review RocksDB’s SstFileWriter and CreateColumnFamilyWithImport APIs and their metadata requirements. Done means adding consistent multi-column-family export and import with a manifest, safe cleanup and rollback, plus tests for snapshots, blob values, reopen behavior, and concurrent close or drop.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

enhancement

Summary

Add a way to export a chosen set of column families from a live database, all at one point in time, and import them into another database as new column families.

This is needed for Harper's table-scoped backups (HarperFast/harper#4435). A Harper table is spread over several column families: its primary store plus one per indexed attribute. Today db.backup() and db.createCheckpoint() can only copy the whole database.

RocksDB's native column family export can't be used as is

RocksDB has a native pair for this: Checkpoint::ExportColumnFamily and DB::CreateColumnFamilyWithImport. It doesn't work for databases created by rocksdb-js:

  • rocksdb-js turns on blob files for every column family. It sets enable_blob_files = true with min_blob_size = 2048 (db_descriptor.cpp:243), so any value of 2 KB or more is stored in a blob file and the SST keeps only a reference to it.
  • The native export copies SST files only, never blob files. In RocksDB 11.13.2 (what rocksdb-js ships), exporting a column family that has blob references succeeds. The imported column family then points at blob files that don't exist in the target database, and reads of those keys fail later. Nothing reports the problem at export or import time.
  • Upstream now rejects this case. meta/rocksdb#15652 (on main as fd7408e, not yet in a tagged release) makes both calls return NotSupported when blob references are present. After the next upgrade, the native export fails for any column family holding a value of 2 KB or more.
  • No consistency across column families. ExportColumnFamily flushes and exports one column family per call, and takes no snapshot. Exporting several column families one after another gives a different point in time for each.

Proposal: logical export through SstFileWriter

Export by reading each column family under one snapshot and writing the rows with SstFileWriter. Values are written inline, so the exported files hold no blob references. Import them with CreateColumnFamilyWithImport. Its header lists files from SstFileWriter as a supported input, and upstream confirms that importing files without blob references into a column family with blob files enabled still works.

This solves both problems: there are no blob references, and one snapshot covers every exported column family. The cost is that every row is read and written again instead of hard-linked. That is acceptable here, because a streamed backup is already a full copy.

API sketch

Names are placeholders:

// Takes one snapshot, then writes SST files for each named column family plus
// a JSON manifest to exportDir.
const manifest = await db.exportColumnFamilies(['dog/', 'dog/name', 'dog/breed'], exportDir, {
	maxFileSize, // split a column family's rows across several SST files
});

// Creates each column family from the exported files. Rejects if any of the
// names already exists.
await target.importColumnFamilies(exportDir, {
	names, // optional subset, or a rename map
	moveFiles, // ImportColumnFamilyOptions::move_files
});
Requirements
  • One snapshot for every column family in the call. Iterating in key order already gives SstFileWriter the sorted input it needs.
  • A manifest that can be stored and read back later. Import runs in a different process, and possibly on a different host, after the export has gone through a tar archive. Write the per-file metadata CreateColumnFamilyWithImport needs to JSON (column family name, comparator name, and for each file its name, key range and sequence numbers), and rebuild ExportImportFilesMetaData from it on import.
  • Imported column families get registered. Once imported, a column family must behave as if db.use(name) had created it: on other threads, after a reopen, and in stats.
  • Safe against concurrent close and drop. Register both operations in operationsInFlight, as createCheckpoint does (checkpoint.cpp). Define what happens if a column family being exported is dropped partway through.
  • Fail cleanly. A failed export removes its partial directory. A failed import leaves no column family behind; RocksDB already guarantees this for a single column family, but a multi-column-family call needs to roll back the ones it already created.
  • The transaction log is out of scope. It covers the whole database, and Harper starts a fresh log for a restored table.
Open questions
  • Streaming: should export write straight into a tar stream, the way db.backup(stream) does, instead of to a directory? Directory output is simpler, and Harper can put the files in a tar itself.
  • A native fast path: keep ExportColumnFamily for column families with no blob references? It would only help tables whose every value is under 2 KB. Probably not worth maintaining two paths.
  • Sizing: large column families need their output split into several files (maxFileSize) to keep memory and import time bounded.

🤖 Filed with Claude Code (Claude Opus 5.5)

Lingua principale
C++
Stelle
24
Fork
3
Merge medio
1g 17h
PR unite (30g)
94

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di HarperFast/rocksdb-js

Tutte le issue di HarperFast/rocksdb-js

Issue simili

Altre issue su C++

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.