Support merge-on-read deletes on v2 tables
Nadie ha tomado este issue todavía.
Evaluación
- Dificultad
- 5/5
- Tiempo estimado
- Más de una semana
- Aptitud para principiantes
- 38/100
Línea de trabajo
Comienza en la rama merge-on-read de Transaction.delete() e inspecciona POSITIONAL_DELETE_SCHEMA, el delete-file index existente y el v2 manifest writer. Traza cómo se producen manifests y snapshots y, después, verifica que position resolution lea las data-file rows coincidentes en physical order. Se considera terminado cuando v2 opt-in deletes creen exact-bound position-delete files en DELETES manifests, mientras que el comportamiento predeterminado de copy-on-write de v1 y v3 permanece sin cambios.
Escrito por el modelo de indexación a partir del texto del issue.
Descripción
Feature Request / Improvement
Transaction.delete() is copy-on-write: deleting rows rewrites every data file that holds a matched row, so it costs ~O(table) per operation and grows with the table. Add v2 merge-on-read position-delete writes so a delete instead writes a small position-delete file, costing O(rows deleted), with the data files left untouched. v2 position deletes are readable by the existing query engines today, so this is broadly adoptable.
Even as v3 merge-on-read (deletion vectors) support lands, v2 position-delete writes remain valuable, because format version is a per-table property and the two encodings are mutually exclusive: a v2 table can never use a deletion vector. v3 write support therefore does nothing for the large existing v2 fleet, whose tables stay v2 until deliberately upgraded and often must stay v2 to remain readable by every engine that consumes them (v3 deletion vectors require newer readers). For those tables, position deletes are the only efficient delete path, and they are also the format Java/Spark Iceberg already writes, so this also closes a read/write asymmetry in PyIceberg. Deletion vectors are the right default for new v3 tables; v2 position deletes serve the installed base.
What already exists (no work needed)
- The read path already applies position deletes at scan time (the delete-file index / position-delete application).
POSITIONAL_DELETE_SCHEMA(file_path,pos),DataFileContent.POSITION_DELETES,ManifestContent.DELETES, and the full snapshot / manifest-list / commit machinery.Transaction.delete()already recognizeswrite.delete.mode=merge-on-read. It currently just warns "not yet supported" and falls back to copy-on-write. That fallback is the hook point.
The gap to close
- Widen
posinPOSITIONAL_DELETE_SCHEMAfrominttolong. The Iceberg spec typesposaslong; keeping itintmismatches what other engines write/read and overflows on files with more than 2^31 rows. The read path treatsposas a generic index, so widening is safe. - A DELETES manifest writer. The v2 manifest writer hardcodes manifest content to
data; a variant is needed that writes contentdeletes. - A snapshot producer that adds position-delete files (writes a DELETES manifest for them, grouped by partition spec, and carries existing manifests forward), committing an overwrite snapshot.
- A position-delete file writer that records exact
file_pathlower/upper bounds. The read-side delete-file index pins a delete to a single data file only when itsfile_pathbound is exact; the defaulttruncate(16)write-metrics mode would truncate that bound and mis-route (or over-apply) the delete. - Predicate to positions resolution. For each data file a delete predicate touches, read it in physical order and collect the ordinal row positions that match, then write one position-delete file per touched data file.
- Wire 1 through 5 into the merge-on-read branch of
Transaction.delete(), gated behindwrite.delete.mode=merge-on-readon v2 tables; copy-on-write stays the default so nothing changes unless a user opts in. v1 (cannot store delete manifests) and v3 (deletion vectors, out of scope) keep the copy-on-write fallback. A size-based heuristic (rewrite tiny files instead of writing delete files) can come later.
- Lenguaje dominante
- Python
- Estrellas
- 1.1k
- Forks
- 589
- Merge medio
- 2 d 2 h
- PR fusionados (30 d)
- 70
Guía de contribución
No hay ninguna guía de contribución indexada para este repositorio
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de apache/iceberg-python
-
kind:bug
Dificultad 1/5 Menos de una hora Aptitud para principiantes 92/100
apache/iceberg-python#4006 ·
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
apache/iceberg-python#3996 ·
-
Deletion vector bitmap count is read from the blob and used as a loop bound without validation Abiertobug
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
apache/iceberg-python#3979 ·
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
apache/iceberg-python#3885 ·
-
[Bug] PyArrowFileIO fails to propagate s3.ssl.ca-cert to pyarrow.fs.S3FileSystem tls_ca_file_path Abierto
Dificultad 2/5 1-3 horas Aptitud para principiantes 76/100
apache/iceberg-python#3866 · 1 comentario ·
Todos los issues de apache/iceberg-python
Issues similares
-
bug confirmed issue
Dificultad 2/5 1-3 horas Aptitud para principiantes 75/100
open-webui/open-webui#30750 · 1 comentario ·
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 75/100
-
enhancement
Dificultad 2/5 1-3 horas Aptitud para principiantes 75/100
OpenwaterHealth/openmotion-bloodflow-app#604 · 1 comentario ·
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 70/100
-
good first issue
Dificultad 1/5 Menos de una hora Aptitud para principiantes 90/100