feature: let Python and Java turn on non-reusable field IDs, including at dataset creation

Abierto
#10,684 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 2 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
52/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
java, python, rust

Línea de trabajo

Start by reading Dataset::migrate_to_non_reusable_field_ids and the Python and Java bindings for it, then trace WriteParams and the lance.write_dataset entry point. Check the existing stable-row-ID option for how create-time flags are exposed. Done means Python and Java can invoke the migration, and new datasets can enable non-reusable field IDs at creation; run the relevant binding and dataset-creation tests.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

feature

Field IDs are reused by default. When the highest-ID column is dropped and its files go away, the next column added gets the same ID. On pylance 14.0.2:

import lance
import pyarrow as pa

ds = lance.write_dataset(pa.table({"a": [1, 2, 3]}), "/tmp/t.lance")
ds.add_columns({"b": "a * 10"})
ds.drop_columns(["b"])
ds.add_columns({"c": "a * 7"})
print([(f.name(), f.id()) for f in ds.lance_schema.fields()])
# [('a', 0), ('c', 1)]: c gets b's old ID

So a field ID can't serve as a column's identity for a writer that stages files before a schema change and commits them after it. For example, a long-running column backfill writes DataFiles for field 1 and commits them with DataReplacement. If b is dropped and re-added in between, the staged files fill the new column with the old column's values. #9436 catches the drop when it falls inside the commit's rebase window. But a writer that checks other writers' commits itself and then commits at the latest version never shows the drop to the conflict resolver, and its staged field IDs name a column that exists.

#8658 solves this with non-reusable field IDs, but only Rust can turn them on:

  • Dataset::migrate_to_non_reusable_field_ids has no Python or Java binding. Both bindings honor the contract once it is on.
  • A dataset can't be created with it on, the way WriteParams::enable_stable_row_ids works for stable row IDs. A new dataset has no older writers, so the migration's warning about clients that ignore writer feature flags doesn't apply to it.

Ask

  1. Python and Java bindings for migrate_to_non_reusable_field_ids.
  2. A create-time option, for example WriteParams::enable_non_reusable_field_ids and lance.write_dataset(..., enable_non_reusable_field_ids=True).
  3. Later, possibly, making it the default for new datasets. Lance writers from before 14.0 don't know FLAG_NON_REUSABLE_FIELD_IDS and refuse to write to a dataset that sets it (checked on v10.0.0), so older clients can't reuse IDs on such a dataset.
Lenguaje dominante
Rust
Estrellas
7.3k
Forks
929
Merge medio
2 d 22 h
PR fusionados (30 d)
572

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de lance-format/lance

Todos los issues de lance-format/lance

Issues similares

Más issues de Rust

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.