Export column families at one snapshot and import them into another database
I maintainer di solito rispondono entro 1 giorno
Nessuno ha ancora preso questa issue.
Valutazione
- Difficoltà
- 5/5
- Tempo stimato
- Più di una settimana
- Idoneità per principianti
- 30/100
Direzione di ricerca
Start with the database binding’s checkpoint and column family import code, especially checkpoint.cpp and db_descriptor.cpp, then trace how column families are registered and tracked in operationsInFlight. Review RocksDB’s SstFileWriter and CreateColumnFamilyWithImport APIs and their metadata requirements. Done means adding consistent multi-column-family export and import with a manifest, safe cleanup and rollback, plus tests for snapshots, blob values, reopen behavior, and concurrent close or drop.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Summary
Add a way to export a chosen set of column families from a live database, all at one point in time, and import them into another database as new column families.
This is needed for Harper's table-scoped backups (HarperFast/harper#4435). A Harper table is spread over several column families: its primary store plus one per indexed attribute. Today db.backup() and db.createCheckpoint() can only copy the whole database.
RocksDB's native column family export can't be used as is
RocksDB has a native pair for this: Checkpoint::ExportColumnFamily and DB::CreateColumnFamilyWithImport. It doesn't work for databases created by rocksdb-js:
- rocksdb-js turns on blob files for every column family. It sets
enable_blob_files = truewithmin_blob_size = 2048(db_descriptor.cpp:243), so any value of 2 KB or more is stored in a blob file and the SST keeps only a reference to it. - The native export copies SST files only, never blob files. In RocksDB 11.13.2 (what rocksdb-js ships), exporting a column family that has blob references succeeds. The imported column family then points at blob files that don't exist in the target database, and reads of those keys fail later. Nothing reports the problem at export or import time.
- Upstream now rejects this case. meta/rocksdb#15652 (on
mainasfd7408e, not yet in a tagged release) makes both calls returnNotSupportedwhen blob references are present. After the next upgrade, the native export fails for any column family holding a value of 2 KB or more. - No consistency across column families.
ExportColumnFamilyflushes and exports one column family per call, and takes no snapshot. Exporting several column families one after another gives a different point in time for each.
Proposal: logical export through SstFileWriter
Export by reading each column family under one snapshot and writing the rows with SstFileWriter. Values are written inline, so the exported files hold no blob references. Import them with CreateColumnFamilyWithImport. Its header lists files from SstFileWriter as a supported input, and upstream confirms that importing files without blob references into a column family with blob files enabled still works.
This solves both problems: there are no blob references, and one snapshot covers every exported column family. The cost is that every row is read and written again instead of hard-linked. That is acceptable here, because a streamed backup is already a full copy.
API sketch
Names are placeholders:
// Takes one snapshot, then writes SST files for each named column family plus
// a JSON manifest to exportDir.
const manifest = await db.exportColumnFamilies(['dog/', 'dog/name', 'dog/breed'], exportDir, {
maxFileSize, // split a column family's rows across several SST files
});
// Creates each column family from the exported files. Rejects if any of the
// names already exists.
await target.importColumnFamilies(exportDir, {
names, // optional subset, or a rename map
moveFiles, // ImportColumnFamilyOptions::move_files
});
Requirements
- One snapshot for every column family in the call. Iterating in key order already gives
SstFileWriterthe sorted input it needs. - A manifest that can be stored and read back later. Import runs in a different process, and possibly on a different host, after the export has gone through a tar archive. Write the per-file metadata
CreateColumnFamilyWithImportneeds to JSON (column family name, comparator name, and for each file its name, key range and sequence numbers), and rebuildExportImportFilesMetaDatafrom it on import. - Imported column families get registered. Once imported, a column family must behave as if
db.use(name)had created it: on other threads, after a reopen, and instats. - Safe against concurrent close and drop. Register both operations in
operationsInFlight, ascreateCheckpointdoes (checkpoint.cpp). Define what happens if a column family being exported is dropped partway through. - Fail cleanly. A failed export removes its partial directory. A failed import leaves no column family behind; RocksDB already guarantees this for a single column family, but a multi-column-family call needs to roll back the ones it already created.
- The transaction log is out of scope. It covers the whole database, and Harper starts a fresh log for a restored table.
Open questions
- Streaming: should export write straight into a tar stream, the way
db.backup(stream)does, instead of to a directory? Directory output is simpler, and Harper can put the files in a tar itself. - A native fast path: keep
ExportColumnFamilyfor column families with no blob references? It would only help tables whose every value is under 2 KB. Probably not worth maintaining two paths. - Sizing: large column families need their output split into several files (
maxFileSize) to keep memory and import time bounded.
🤖 Filed with Claude Code (Claude Opus 5.5)
- Lingua principale
- C++
- Stelle
- 24
- Fork
- 3
- Merge medio
- 1g 17h
- PR unite (30g)
- 94
Preparare l'ambiente
- Nessun Dockerfile né file Docker Compose
- Nessun modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di HarperFast/rocksdb-js
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 68/100
HarperFast/rocksdb-js#1118 ·
I maintainer di solito rispondono entro 1 giorno
-
CLI crashes with ENOTCONN instead of exiting 0 when stdout closes on macOSForse già presa @cb1kenobi l’ha presa oggi. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 83/100
HarperFast/rocksdb-js#1112 ·
I maintainer di solito rispondono entro 1 giorno
-
A durable commit is reported as failed when aftercommit hits a database closed after the commitAperta
Difficoltà 4/5 3-5 giorni Idoneità per principianti 48/100
HarperFast/rocksdb-js#1117 ·
I maintainer di solito rispondono entro 1 giorno
-
bug
Difficoltà 4/5 1-2 giorni Idoneità per principianti 38/100
HarperFast/rocksdb-js#1114 ·
I maintainer di solito rispondono entro 1 giorno
-
Release builds abort the process when an internal native assert failsForse già presa @cb1kenobi l’ha presa oggi. Apertabug
HarperFast/rocksdb-js#1113 · 1 assegnatario ·
I maintainer di solito rispondono entro 1 giorno
Tutte le issue di HarperFast/rocksdb-js
Issue simili
-
Issue-Docs Needs-Triage
Difficoltà 2/5 1-3 ore Idoneità per principianti 70/100
microsoft/winget-cli#6692 ·
I maintainer di solito rispondono entro 2 giorni
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 75/100
stratum-mining/sv2-tp#167 ·
I maintainer di solito rispondono entro 1 giorno
-
Sign up button opens the Sign in page instead of the Sign up page for Enterprise edition serversApertaauth bug ui/ux
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
MerginMaps/mobile#4795 ·
I maintainer di solito rispondono entro 2 giorni
-
bug triage
Difficoltà 2/5 1-3 ore Idoneità per principianti 75/100
I maintainer di solito rispondono entro 1 giorno
-
26.04 bug dark-mode Easy Hack impress symptom: visual
Difficoltà 2/5 1-3 ore Idoneità per principianti 68/100
CollaboraOnline/online#16421 · 1 commento ·