feature: StagedSnapshot, a safe multi-step Transaction V2 builder

Abierto
#10,429 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 2 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
25/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
rust
Área
databases

Línea de trabajo

Read the Transaction V2 implementation stack (#8644, #8645, #8646) and the proposed crate split: put the pure StagedOperation core in lance-table and the IO-backed StagedSnapshot in lance. Start by tracing the existing transaction actions, manifest application, and writer entry points; the issue names no specific files or tests. Done means witnesses, leases, staged views, file cleanup, and commit/cancel follow the stated invariants, with #10386 able to build on the API.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

bug feature

Problem

A Transaction V2 CompositeOperation is a list of actions, and today every writer that wants one assembles those actions by hand. Each action is a plain struct with public fields, and many of them make claims that nothing at commit time can check:

  • AddDataFile claims its file exists and holds exactly the fragment's physical_rows rows, with columns matching field_ids.
  • SetDeletionFile claims its offsets are in range and include every earlier deletion.
  • AddIndexSegment and AdjustIndexCoverage claim the segment's contents cover the fragments listed.
  • AssertUniqueKeys claims its filter covers every key written.

apply does no IO, so a wrong claim commits silently and corrupts the table. The more code builds actions by hand, the more places can get this wrong. There is also no API for composing several writes into one commit, where a later step reads what earlier steps staged.

Proposal

Add StagedSnapshot, a multi-step transaction builder, and make it the only ordinary way to produce actions that make claims about storage.

Three kinds of invariants
Kind Examples Where it is enforced
Intrinsic: one action, no IO AddDataFile.file.fields is empty, no field or column is bound twice The action's constructor and its protobuf decode
Extrinsic: claims about stored bytes the file exists, its row count, its schema, the contents of an index segment The writer that produced the bytes, through a witness (below)
Relational: against the table and other actions the fragment or field exists, a field is not already backed, an index segment is not left stale by a later step in the same operation apply, plus a finalize pass over the whole operation

Relational checks stay in apply because a rebase replays the actions onto a different version, where they must be checked again. Actions work like events, not commands: the command side (StagedSnapshot) checks the extrinsic invariants once, when the files are written.

Action constructors
  • Action fields become private.
  • Actions with no claims about storage get ordinary constructors that check intrinsic invariants.
  • Actions with such claims get new_unchecked, whose docs list the invariants the caller must uphold. These are AddFragment, AddDataFile, SetDeletionFile, AddIndexSegment, AdjustIndexCoverage, AddOverlays, UpdateCompactedSsTables and AssertUniqueKeys.
  • new_unchecked is a safe function, not unsafe. A false claim corrupts the table, which readers must already tolerate; it is not undefined behavior. Python and Java expose these objects too, and have no notion of unsafe.
  • Within the workspace, clippy.toml disallowed-methods denies the new_unchecked constructors. Each trusted call site (wire decode, the writers, compaction, test support) opts in with #[expect(clippy::disallowed_methods, reason = "...")].
  • data_change: false is not checked by apply. It comes from new_unchecked, or from a dedicated, well-tested compaction method that builds its transaction itself.
StagedSnapshot

The read version, plus the operation composed so far, plus a read-only view of the table that operation would produce.

let mut snapshot = StagedSnapshot::try_new_experimental(Arc::new(dataset)).await?;

// A lease serializes, so a worker in another process can write against it.
let lease = snapshot.lease()?;
let mut writer = lease.new_fragment(&schema)?;
writer.write(batch).await?;
let written: WrittenFragment = writer.finish().await?;

// Actions are built from witnesses here, never by the caller. Fragment
// tokens are minted here too, and returned as handles.
let (snapshot, fragments) = snapshot
    .stage(Step::new("append batch", vec![written.into()]))
    .await?;
let dataset = snapshot.commit().await?;
  • Witnesses, not actions. Writers return witnesses (WrittenFragment, WrittenColumn), and stage() turns them into actions. Callers never hold an action they could put in some other operation.
  • Two writers. For a new fragment, physical_rows comes from the rows written, so AddFragment and its AddDataFile agree by construction. For a new column on an existing fragment, physical_rows is fixed, and finish() fails if the row count differs.
  • Fragment references are Committed(id) (checked against the view) or Staged(StagedFragment). A StagedFragment is an opaque handle carrying the snapshot id, so a handle from one snapshot is rejected by another.
  • Leases. A WriteLease holds the snapshot and lease ids, the table URI, the read version, and the operation composed so far (so a worker can rebuild the view). It does not reserve fragment tokens, so a worker can write any number of fragments without knowing the count up front. Local writers use leases too, so there is one code path.
  • Tokens are minted at stage(). A worker needs a fragment's provisional id only if it writes something that stores it. Data files do not (their paths are UUIDs, and row ids are assigned at commit). Index segments do, but they are built after stage(), against the staged view. A witness names its fragments by UUID; stage() mints a token for each and returns StagedFragment handles. If a remote worker ever needs ids before staging (for example, a distributed index build), a lease can own a token namespace, with the lease's sequence number in the high bits, still without a count.
  • Trusted writers. Witnesses come only from Lance's writers, so stage() does not re-read files. It checks only provenance: the snapshot and lease ids match, and no witness or fragment UUID is staged twice.
  • Read-only view. The view's manifest is speculative: its version number is the one the next real commit will take. Nothing may commit through it.
  • File ledger and cancel. The snapshot records every file path its witnesses report. cancel() deletes them. Files left by a crashed worker or a dropped snapshot are unreferenced, so cleanup_old_versions removes them.
  • Index segments. Adding an index segment over staged fragments is #10386. Distributed index builds are out of scope here.
Crate placement

The pure core goes in lance-table, which starts compiling far earlier than lance (see #10244). Call it StagedOperation:

  • witness and lease types
  • token allocation and the file ledger
  • turning witnesses into actions and applying them to the manifest
  • the action constructors

lance keeps a thin StagedSnapshot around it, holding the IO parts:

  • the data writers (today tied to Dataset and WriteParams)
  • the speculative Dataset view and its caches
  • commit() and cancel()

Moving the data writers into lance-table is a separate refactor.

Sequencing

  1. A follow-up to the Transaction V2 implementation stack (#8644, #8645, #8646): private fields, intrinsic checks moved into constructors and decode, new_unchecked, and the clippy rule.
  2. StagedOperation in lance-table, and StagedSnapshot in lance, with local and distributed data writes.
  3. #10386 builds on it.

Open questions

  • Should the lease carry the whole composed operation? It is expected to be small. If it grows large, it could be written to a staged location and loaded by the worker.
Lenguaje dominante
Rust
Estrellas
7.3k
Forks
929
Merge medio
2 d 20 h
PR fusionados (30 d)
575

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de lance-format/lance

Todos los issues de lance-format/lance

Issues similares

Más issues de Rust

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.