feature: let Python and Java turn on non-reusable field IDs, including at dataset creation

オープン
#10,684 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 2 日以内に返信

まだ誰も着手していません。

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
52/100
issue の種類
機能追加
明瞭さ
おおむね明確
活発さ
活発
技術スタック
java, python, rust

調査の方向性

Start by reading Dataset::migrate_to_non_reusable_field_ids and the Python and Java bindings for it, then trace WriteParams and the lance.write_dataset entry point. Check the existing stable-row-ID option for how create-time flags are exposed. Done means Python and Java can invoke the migration, and new datasets can enable non-reusable field IDs at creation; run the relevant binding and dataset-creation tests.

索引モデルが issue の本文から書いたものです。

説明

feature

Field IDs are reused by default. When the highest-ID column is dropped and its files go away, the next column added gets the same ID. On pylance 14.0.2:

import lance
import pyarrow as pa

ds = lance.write_dataset(pa.table({"a": [1, 2, 3]}), "/tmp/t.lance")
ds.add_columns({"b": "a * 10"})
ds.drop_columns(["b"])
ds.add_columns({"c": "a * 7"})
print([(f.name(), f.id()) for f in ds.lance_schema.fields()])
# [('a', 0), ('c', 1)]: c gets b's old ID

So a field ID can't serve as a column's identity for a writer that stages files before a schema change and commits them after it. For example, a long-running column backfill writes DataFiles for field 1 and commits them with DataReplacement. If b is dropped and re-added in between, the staged files fill the new column with the old column's values. #9436 catches the drop when it falls inside the commit's rebase window. But a writer that checks other writers' commits itself and then commits at the latest version never shows the drop to the conflict resolver, and its staged field IDs name a column that exists.

#8658 solves this with non-reusable field IDs, but only Rust can turn them on:

  • Dataset::migrate_to_non_reusable_field_ids has no Python or Java binding. Both bindings honor the contract once it is on.
  • A dataset can't be created with it on, the way WriteParams::enable_stable_row_ids works for stable row IDs. A new dataset has no older writers, so the migration's warning about clients that ignore writer feature flags doesn't apply to it.

Ask

  1. Python and Java bindings for migrate_to_non_reusable_field_ids.
  2. A create-time option, for example WriteParams::enable_non_reusable_field_ids and lance.write_dataset(..., enable_non_reusable_field_ids=True).
  3. Later, possibly, making it the default for new datasets. Lance writers from before 14.0 don't know FLAG_NON_REUSABLE_FIELD_IDS and refuse to write to a dataset that sets it (checked on v10.0.0), so older clients can't reuse IDs on such a dataset.
主要言語
Rust
スター
7.3k
フォーク
929
平均マージ
2日 22時間
マージ済み PR(30日)
572

環境構築

Codespaces で開く

このプロジェクトの開発コンテナを、あなたの GitHub アカウントでブラウザ上に起動します。

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

lance-format/lance のほかの issue

lance-format/lance の issue をすべて見る

似ている issue

Rust の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。