feature: StagedSnapshot, a safe multi-step Transaction V2 builder
Los mantenedores suelen responder en 2 días
Nadie ha tomado este issue todavía.
Evaluación
- Dificultad
- 5/5
- Tiempo estimado
- Más de una semana
- Aptitud para principiantes
- 25/100
Línea de trabajo
Read the Transaction V2 implementation stack (#8644, #8645, #8646) and the proposed crate split: put the pure StagedOperation core in lance-table and the IO-backed StagedSnapshot in lance. Start by tracing the existing transaction actions, manifest application, and writer entry points; the issue names no specific files or tests. Done means witnesses, leases, staged views, file cleanup, and commit/cancel follow the stated invariants, with #10386 able to build on the API.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
Problem
A Transaction V2 CompositeOperation is a list of actions, and today every writer that wants one assembles those actions by hand. Each action is a plain struct with public fields, and many of them make claims that nothing at commit time can check:
AddDataFileclaims its file exists and holds exactly the fragment'sphysical_rowsrows, with columns matchingfield_ids.SetDeletionFileclaims its offsets are in range and include every earlier deletion.AddIndexSegmentandAdjustIndexCoverageclaim the segment's contents cover the fragments listed.AssertUniqueKeysclaims its filter covers every key written.
apply does no IO, so a wrong claim commits silently and corrupts the table. The more code builds actions by hand, the more places can get this wrong. There is also no API for composing several writes into one commit, where a later step reads what earlier steps staged.
Proposal
Add StagedSnapshot, a multi-step transaction builder, and make it the only ordinary way to produce actions that make claims about storage.
Three kinds of invariants
| Kind | Examples | Where it is enforced |
|---|---|---|
| Intrinsic: one action, no IO | AddDataFile.file.fields is empty, no field or column is bound twice |
The action's constructor and its protobuf decode |
| Extrinsic: claims about stored bytes | the file exists, its row count, its schema, the contents of an index segment | The writer that produced the bytes, through a witness (below) |
| Relational: against the table and other actions | the fragment or field exists, a field is not already backed, an index segment is not left stale by a later step in the same operation | apply, plus a finalize pass over the whole operation |
Relational checks stay in apply because a rebase replays the actions onto a different version, where they must be checked again. Actions work like events, not commands: the command side (StagedSnapshot) checks the extrinsic invariants once, when the files are written.
Action constructors
- Action fields become private.
- Actions with no claims about storage get ordinary constructors that check intrinsic invariants.
- Actions with such claims get
new_unchecked, whose docs list the invariants the caller must uphold. These areAddFragment,AddDataFile,SetDeletionFile,AddIndexSegment,AdjustIndexCoverage,AddOverlays,UpdateCompactedSsTablesandAssertUniqueKeys. new_uncheckedis a safe function, notunsafe. A false claim corrupts the table, which readers must already tolerate; it is not undefined behavior. Python and Java expose these objects too, and have no notion ofunsafe.- Within the workspace,
clippy.tomldisallowed-methodsdenies thenew_uncheckedconstructors. Each trusted call site (wire decode, the writers, compaction, test support) opts in with#[expect(clippy::disallowed_methods, reason = "...")]. data_change: falseis not checked byapply. It comes fromnew_unchecked, or from a dedicated, well-tested compaction method that builds its transaction itself.
StagedSnapshot
The read version, plus the operation composed so far, plus a read-only view of the table that operation would produce.
let mut snapshot = StagedSnapshot::try_new_experimental(Arc::new(dataset)).await?;
// A lease serializes, so a worker in another process can write against it.
let lease = snapshot.lease()?;
let mut writer = lease.new_fragment(&schema)?;
writer.write(batch).await?;
let written: WrittenFragment = writer.finish().await?;
// Actions are built from witnesses here, never by the caller. Fragment
// tokens are minted here too, and returned as handles.
let (snapshot, fragments) = snapshot
.stage(Step::new("append batch", vec![written.into()]))
.await?;
let dataset = snapshot.commit().await?;
- Witnesses, not actions. Writers return witnesses (
WrittenFragment,WrittenColumn), andstage()turns them into actions. Callers never hold an action they could put in some other operation. - Two writers. For a new fragment,
physical_rowscomes from the rows written, soAddFragmentand itsAddDataFileagree by construction. For a new column on an existing fragment,physical_rowsis fixed, andfinish()fails if the row count differs. - Fragment references are
Committed(id)(checked against the view) orStaged(StagedFragment). AStagedFragmentis an opaque handle carrying the snapshot id, so a handle from one snapshot is rejected by another. - Leases. A
WriteLeaseholds the snapshot and lease ids, the table URI, the read version, and the operation composed so far (so a worker can rebuild the view). It does not reserve fragment tokens, so a worker can write any number of fragments without knowing the count up front. Local writers use leases too, so there is one code path. - Tokens are minted at
stage(). A worker needs a fragment's provisional id only if it writes something that stores it. Data files do not (their paths are UUIDs, and row ids are assigned at commit). Index segments do, but they are built afterstage(), against the staged view. A witness names its fragments by UUID;stage()mints a token for each and returnsStagedFragmenthandles. If a remote worker ever needs ids before staging (for example, a distributed index build), a lease can own a token namespace, with the lease's sequence number in the high bits, still without a count. - Trusted writers. Witnesses come only from Lance's writers, so
stage()does not re-read files. It checks only provenance: the snapshot and lease ids match, and no witness or fragment UUID is staged twice. - Read-only view. The view's manifest is speculative: its version number is the one the next real commit will take. Nothing may commit through it.
- File ledger and cancel. The snapshot records every file path its witnesses report.
cancel()deletes them. Files left by a crashed worker or a dropped snapshot are unreferenced, socleanup_old_versionsremoves them. - Index segments. Adding an index segment over staged fragments is #10386. Distributed index builds are out of scope here.
Crate placement
The pure core goes in lance-table, which starts compiling far earlier than lance (see #10244). Call it StagedOperation:
- witness and lease types
- token allocation and the file ledger
- turning witnesses into actions and applying them to the manifest
- the action constructors
lance keeps a thin StagedSnapshot around it, holding the IO parts:
- the data writers (today tied to
DatasetandWriteParams) - the speculative
Datasetview and its caches commit()andcancel()
Moving the data writers into lance-table is a separate refactor.
Sequencing
- A follow-up to the Transaction V2 implementation stack (#8644, #8645, #8646): private fields, intrinsic checks moved into constructors and decode,
new_unchecked, and the clippy rule. StagedOperationinlance-table, andStagedSnapshotinlance, with local and distributed data writes.- #10386 builds on it.
Open questions
- Should the lease carry the whole composed operation? It is expected to be small. If it grows large, it could be written to a staged location and loaded by the worker.
- Lenguaje dominante
- Rust
- Estrellas
- 7.3k
- Forks
- 929
- Merge medio
- 2 d 20 h
- PR fusionados (30 d)
- 575
Preparar el entorno
Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.
- Incluye un Dockerfile o un archivo de Docker Compose
- Sin plantilla de pull request
- Leer la guía de contribución
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de lance-format/lance
-
bug(python): read_transaction raises TypeError on a Rewrite with rewritten indicesPosiblemente ocupada @Borg93 la tomó hace 1 día. Abiertobug
Dificultad 2/5 1-3 horas Aptitud para principiantes 75/100
lance-format/lance#11117 ·
Los mantenedores suelen responder en 2 días
-
bug
Dificultad 2/5 1-3 horas Aptitud para principiantes 75/100
lance-format/lance#10782 ·
Los mantenedores suelen responder en 2 días
-
Upgrade DataFusion for the unparser stack-overflow fixQuizá libre de nuevo Un pull request para esta issue se cerró sin fusionarse. Abierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
lance-format/lance#8225 ·
Los mantenedores suelen responder en 2 días
-
enhancement
Dificultad 1/5 Menos de una hora Aptitud para principiantes 84/100
lance-format/lance#6425 · 2 comentarios ·
Los mantenedores suelen responder en 2 días
-
Expose compute_pq_distanceQuizá libre de nuevo Un pull request para esta issue se cerró sin fusionarse. Abiertoenhancement
Dificultad 1/5 1-3 horas Aptitud para principiantes 62/100
lance-format/lance#5675 · 1 comentario ·
Los mantenedores suelen responder en 2 días
Todos los issues de lance-format/lance
Issues similares
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
oxc-project/tsrs#1010 ·
Los mantenedores suelen responder en 1 día
-
Link is deadAbierto
Dificultad 1/5 Menos de una hora Aptitud para principiantes 63/100
nicholas-ochoa/OpenSC2K#104 ·
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
Los mantenedores suelen responder en 1 día
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
libracoder/kiri#445 ·
Los mantenedores suelen responder en 1 día
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 68/100
DioxusLabs/blitz#1614 ·
Los mantenedores suelen responder en 1 día