feature: let Python and Java turn on non-reusable field IDs, including at dataset creation

Aperta
#10,684 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
52/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
java, python, rust

Direzione di ricerca

Start by reading Dataset::migrate_to_non_reusable_field_ids and the Python and Java bindings for it, then trace WriteParams and the lance.write_dataset entry point. Check the existing stable-row-ID option for how create-time flags are exposed. Done means Python and Java can invoke the migration, and new datasets can enable non-reusable field IDs at creation; run the relevant binding and dataset-creation tests.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

feature

Field IDs are reused by default. When the highest-ID column is dropped and its files go away, the next column added gets the same ID. On pylance 14.0.2:

import lance
import pyarrow as pa

ds = lance.write_dataset(pa.table({"a": [1, 2, 3]}), "/tmp/t.lance")
ds.add_columns({"b": "a * 10"})
ds.drop_columns(["b"])
ds.add_columns({"c": "a * 7"})
print([(f.name(), f.id()) for f in ds.lance_schema.fields()])
# [('a', 0), ('c', 1)]: c gets b's old ID

So a field ID can't serve as a column's identity for a writer that stages files before a schema change and commits them after it. For example, a long-running column backfill writes DataFiles for field 1 and commits them with DataReplacement. If b is dropped and re-added in between, the staged files fill the new column with the old column's values. #9436 catches the drop when it falls inside the commit's rebase window. But a writer that checks other writers' commits itself and then commits at the latest version never shows the drop to the conflict resolver, and its staged field IDs name a column that exists.

#8658 solves this with non-reusable field IDs, but only Rust can turn them on:

  • Dataset::migrate_to_non_reusable_field_ids has no Python or Java binding. Both bindings honor the contract once it is on.
  • A dataset can't be created with it on, the way WriteParams::enable_stable_row_ids works for stable row IDs. A new dataset has no older writers, so the migration's warning about clients that ignore writer feature flags doesn't apply to it.

Ask

  1. Python and Java bindings for migrate_to_non_reusable_field_ids.
  2. A create-time option, for example WriteParams::enable_non_reusable_field_ids and lance.write_dataset(..., enable_non_reusable_field_ids=True).
  3. Later, possibly, making it the default for new datasets. Lance writers from before 14.0 don't know FLAG_NON_REUSABLE_FIELD_IDS and refuse to write to a dataset that sets it (checked on v10.0.0), so older clients can't reuse IDs on such a dataset.
Lingua principale
Rust
Stelle
7.3k
Fork
929
Merge medio
2g 22h
PR unite (30g)
572

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di lance-format/lance

Tutte le issue di lance-format/lance

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.