Hacktoberfest 2026:メンテナが10月に向けて印を付けた、オープンで初心者向けの issue。 Hacktoberfest の issue を見る

Support merge-on-read deletes on v2 tables

オープン
#3,618 コメント 1 件 リアクション 1 件 担当者 0 名 GitHub で見る

まだ誰も着手していません。

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
38/100
issue の種類
機能追加
明瞭さ
明確に書かれている
活発さ
静か
技術スタック
python
領域
databases

調査の方向性

Transaction.delete() の merge-on-read ブランチから始め、POSITIONAL_DELETE_SCHEMA、既存の delete-file index、v2 manifest writer を調査する。manifests と snapshots がどのように生成されるかを追跡し、その後、position resolution が一致する data-file rows を physical order で読み取ることを確認する。完了条件は、v2 opt-in deletes が DELETES manifests に exact-bound position-delete files を作成し、v1、v3、デフォルトの copy-on-write behavior は変更されないことである。

索引モデルが issue の本文から書いたものです。

説明

Feature Request / Improvement

Transaction.delete() is copy-on-write: deleting rows rewrites every data file that holds a matched row, so it costs ~O(table) per operation and grows with the table. Add v2 merge-on-read position-delete writes so a delete instead writes a small position-delete file, costing O(rows deleted), with the data files left untouched. v2 position deletes are readable by the existing query engines today, so this is broadly adoptable.

Even as v3 merge-on-read (deletion vectors) support lands, v2 position-delete writes remain valuable, because format version is a per-table property and the two encodings are mutually exclusive: a v2 table can never use a deletion vector. v3 write support therefore does nothing for the large existing v2 fleet, whose tables stay v2 until deliberately upgraded and often must stay v2 to remain readable by every engine that consumes them (v3 deletion vectors require newer readers). For those tables, position deletes are the only efficient delete path, and they are also the format Java/Spark Iceberg already writes, so this also closes a read/write asymmetry in PyIceberg. Deletion vectors are the right default for new v3 tables; v2 position deletes serve the installed base.

What already exists (no work needed)
  • The read path already applies position deletes at scan time (the delete-file index / position-delete application).
  • POSITIONAL_DELETE_SCHEMA (file_path, pos), DataFileContent.POSITION_DELETES, ManifestContent.DELETES, and the full snapshot / manifest-list / commit machinery.
  • Transaction.delete() already recognizes write.delete.mode=merge-on-read. It currently just warns "not yet supported" and falls back to copy-on-write. That fallback is the hook point.
The gap to close
  1. Widen pos in POSITIONAL_DELETE_SCHEMA from int to long. The Iceberg spec types pos as long; keeping it int mismatches what other engines write/read and overflows on files with more than 2^31 rows. The read path treats pos as a generic index, so widening is safe.
  2. A DELETES manifest writer. The v2 manifest writer hardcodes manifest content to data; a variant is needed that writes content deletes.
  3. A snapshot producer that adds position-delete files (writes a DELETES manifest for them, grouped by partition spec, and carries existing manifests forward), committing an overwrite snapshot.
  4. A position-delete file writer that records exact file_path lower/upper bounds. The read-side delete-file index pins a delete to a single data file only when its file_path bound is exact; the default truncate(16) write-metrics mode would truncate that bound and mis-route (or over-apply) the delete.
  5. Predicate to positions resolution. For each data file a delete predicate touches, read it in physical order and collect the ordinal row positions that match, then write one position-delete file per touched data file.
  6. Wire 1 through 5 into the merge-on-read branch of Transaction.delete(), gated behind write.delete.mode=merge-on-read on v2 tables; copy-on-write stays the default so nothing changes unless a user opts in. v1 (cannot store delete manifests) and v3 (deletion vectors, out of scope) keep the copy-on-write fallback. A size-based heuristic (rewrite tiny files instead of writing delete files) can come later.
主要言語
Python
スター
1.1k
フォーク
589
平均マージ
1日 20時間
マージ済み PR(30日)
68

コントリビューションガイド

このリポジトリのコントリビューションガイドは索引されていません

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

apache/iceberg-python のほかの issue

apache/iceberg-python の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。