Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

feature: append data and index it in one Transaction V2 commit

Aperta
  • Qualcosa non va?
#10,386 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 3 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
25/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
rust
Ambito
databases

Direzione di ricerca

Read the draft PR #10384 and the Transaction V2 definitions in transaction/actions.proto and table.proto; the issue describes a prototype that covers BTree indexing and synchronous fragment reuse. Start by tracing how provisional fragment IDs are translated during commit and how readers compose that mapping with fragment reuse. Done means a single commit can append fragments and add an index segment over them, including after a concurrent append rebase, while lookups return committed addresses; vector indices and legacy writer conflicts remain unresolved.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

feature

Depends on #10429.

Problem

Today, appending data and indexing it takes two commits: one adds the fragments, a second adds an index segment over them. In between, readers see the new rows unindexed, so queries fall back to a flat scan of them. A writer that wants new data to show up already indexed can't get that atomically.

Transaction V2 lets one commit carry several steps, and a step can already name a fragment added earlier in the same commit with a Local reference. The index files are what stop this. An index segment stores row addresses (fragment_id << 32 | row_offset) in its files, and a new fragment has no id until the commit allocates one. Reserving ids up front (ReserveFragmentIds) costs an extra commit, and a write that loses a race still has to commit somewhere else.

Proposal

Let an index segment store provisional fragment ids for fragments added in the same commit, and have the commit record how to translate them.

  • A fragment added under local token t gets the provisional id u32::MAX - 1 - t. Provisional ids count down from just below the tombstone fragment id (u32::MAX), and committed ids count up from 0.
  • The writer builds the segment against a speculative view of the table in which the new fragments already carry their provisional ids. The segment's files hold those addresses.
  • At commit, apply records provisional -> committed on the segment's metadata. Readers translate addresses through this map before any fragment reuse translation. A rebase onto a newer version just produces a different map; the index files never change.
Spec changes
 // transaction/actions.proto
 message AddIndexSegment {
   ...
   optional uint64 dataset_version = 11;
+  /* The segment's files address fragments added by this same operation by
+   * provisional id `u32::MAX - 1 - local_token`. When set, every
+   * `covered_fragments` entry must be Local; apply records
+   * IndexMetadata.fragment_id_map, and rejects a table whose committed
+   * fragment ids reach the provisional band. */
+  bool local_fragment_addresses = 13;
 }

 // table.proto
 message IndexMetadata {
   ...
   map<string, string> metadata = 13;
+  /* Provisional -> committed fragment id for the row addresses stored in this
+   * segment's files. Ids absent from the map are already committed. Applied
+   * before any fragment reuse translation. Requires feature flag 1 << 18. */
+  map<uint32, uint32> fragment_id_map = 14;
 }

 // feature flags (reader and writer)
+FLAG_LOCAL_FRAGMENT_ADDRESSES = 1 << 18   // some segment has a fragment_id_map

A reader that ignored the map would return addresses that name no fragment, so both reader and writer flags are required.

API

Builds on StagedSnapshot (#10429), which composes several writes into one commit. A later step sees what earlier steps staged, so an index segment can be built over fragments appended in the same snapshot:

let snapshot = StagedSnapshot::try_new_experimental(Arc::new(dataset)).await?;

// Step 1: write new fragments. They have provisional ids in the snapshot.
let (snapshot, appended) = snapshot
    .append("append batch", batches)
    .await?;

// Step 2: build a segment of the existing BTree index over just those
// fragments, against the snapshot's view of the table.
let snapshot = snapshot
    .add_index_segment("index batch", "a_idx", "a", IndexType::BTree, &params, &appended)
    .await?;

// The snapshot can already be queried through the new segment.
snapshot.dataset().scan().filter("a = 105")?.try_into_batch().await?;

// One commit: fragments + segment. Concurrent commits are rebased over as usual.
let dataset = snapshot.commit().await?;

The snapshot owns local token allocation, so a segment's provisional ids always match the fragments it names.

Prototype

Draft PR #10384 (not for merge). It implements the above for BTree end to end. The test appends to a table, adds a segment to an existing BTree index over the appended fragment, loses a race to a concurrent append (so the committed id is neither the provisional id nor the one it would have had at read time), commits, and checks that an index lookup returns the committed address.

Open questions

  • Caching. Index caches hold decoded content with addresses already translated, so a cache key needs more than the segment uuid. The prototype adds a hash of fragment_id_map to the key: the same segment is read untranslated in the snapshot and translated once committed. More generally, the snapshot's version number is the same one the next real commit will get, so every cache keyed by version needs another way to tell the two apart.
  • Fragment reuse. The translation has to compose with both the synchronous remapper (legacy fragment reuse) and the asynchronous batch remapper (tagged fragment reuse). The prototype handles only the first.
  • Vector indices. Not yet supported. IVF_PQ is the next test: its training sample, takes by address and two-pass reads all run against the speculative view.
  • Legacy writers. A Transaction V2 commit currently conflicts with any concurrent legacy operation. That is planned to be resolved separately, but this feature depends on it.
  • Later steps over indexed fragments. A step that modifies a fragment after a segment in the same snapshot covers it would leave the segment stale. The plan is to reject that at first.
Lingua principale
Rust
Stelle
7.2k
Fork
913
Merge medio
4g 19h
PR unite (30g)
453

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di lance-format/lance

Tutte le issue di lance-format/lance

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.