Hacktoberfest 2026: as issues que os mantenedores marcaram para outubro, abertas e boas para iniciantes. Ver issues do Hacktoberfest

Feature: Add metadata-only replace API to Table for REPLACE snapshot operations

Aberta
  • Algo errado?
#3,130 1 comentário 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 1 dia

@qzyu999 já está trabalhando nisso.

Desde 9/3/2026.

  • #3131 de @qzyu999 — aberto

Avaliação

Dificuldade
5/5
Tempo estimado
Mais de uma semana
Facilidade para iniciantes
28/100
Tipo de issue
Funcionalidade
Clareza
Razoavelmente clara
Status de atividade
Pouca atividade
Stack de tecnologia
python
Domínio
databases

Direção de pesquisa

Comece por pyiceberg/table/update/snapshot.py e pelas APIs de snapshot existentes de Table e Transaction e, em seguida, compare o comportamento proposto com a interface RewriteFiles do Java. Revise tests/table/test_snapshots.py, especialmente test_invalid_operation(), e adicione cobertura focada para trocas de arquivos, números de sequência e operation=REPLACE. Considera-se concluído quando a substituição somente de metadados funcionar atomicamente com entradas Iterable[DataFile] e evitar a serialização Parquet.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

Feature Request / Improvement

Description

This issue proposes implementing a metadata-only replace API in PyIceberg, enabling orchestrators to submit a set of DataFiles to delete and a set of DataFiles to append in a single atomic transaction.

This functionality is critical for maintenance operations such as data compaction (the "small files" problem), ensuring the logical state of the table remains unaltered while physical data layout is optimized.

Background

In a current PR (#3124, part of #1092), PyIceberg's replace semantics are tightly coupled with PyArrow dataframes (def replace(self, df: pa.Table)). This approach introduces several architectural flaws:

  1. Coupling Physical Serialization with Metadata: It forces a .parquet write serialization hook directly into the snapshot commit transaction, increasing the risk of schema degradation and blocking network topologies.
  2. Missing Operation.REPLACE: The current system uses primitives that log as APPEND or OVERWRITE, muddying the table history and complicating snapshot expiry/maintenance.
  3. Java Inconsistency: This severely drifts from Java Iceberg's native org.apache.iceberg.RewriteFiles specification, which strictly isolates the builder into accepting purely DataFile pointers.
Proposed Solution

To fix this and achieve logical equivalence, we must implement an exact port of Java's RewriteFiles builder API into PyIceberg's native _SnapshotProducer engine.

  1. Introduce _RewriteFiles Snapshot Producer:
    Add a new _RewriteFiles class that specifically targets replacing existing files. This class will implement:

    • _deleted_entries(): To find the existing target files and re-emit them as DELETED entries, defensively keeping their ancestral sequence_numbers completely intact for time travel compatibility.
    • _existing_manifests(): To scavenge unchanged manifests natively, skipping deep rewrites and only mutating manifests impacted by the deleted files.

  2. Builder Hook Implementation:
    Implement UpdateSnapshot().replace() which configures the transaction with Operation.REPLACE.

  3. Expose Shorthands on Table & Transaction:
    Add replace APIs on both Table and Transaction taking Iterable[DataFile] arguments to elegantly wrap the snapshot mutation:

    def replace(
    self,
    files_to_delete: Iterable[DataFile],
    files_to_add: Iterable[DataFile],
    snapshot_properties: dict[str, str] = EMPTY_DICT,
    branch: str | None = MAIN_BRANCH,
) -> None:
    ...
Notable canges
  • replace() API implemented on both Table and Transaction using Iterable[DataFile].
  • PyArrow .parquet write logic decoupled from the metadata transaction.
  • _RewriteFiles correctly copies ancestral sequence_number pointers for DELETED and EXISTING manifest entries.
  • Snapshots committed via the replace() hook possess a Summary containing operation=Operation.REPLACE.
  • Unit tests pass simulating data file swaps and summary verifications.
Related Java API

Inspired heavily by Java's builder interface: https://github.com/apache/iceberg/blob/main/api/src/main/java/org/apache/iceberg/RewriteFiles.java

AI Disclosure

AI was used to help understand the code base and draft code changes. All code changes have been thoroughly reviewed, ensuring that the code changes are in line with a broader understanding of the codebase.

  • Worth deeper review after AI-assistance:
  • The test_invalid_operation() in tests/table/test_snapshots.py previously used Operation.REPLACE as a value to test invalid operations, but with this change Operation.REPLACE becomes valid. In place I just put a dummy Operation.
  • The _RewriteFiles in pyiceberg/table/update/snapshot.py overrides the _deleted_entries and _existing_manifests functions. I sought to test this thoroughly that it was done correctly. I am thinking it's possible to improve the test suite to make this more rigorous. I am open to suggestions on how that could be done.
Linguagem predominante
Python
Estrelas
1.2k
Forks
618
Merge médio
1d 10h
PRs com merge (30d)
71

Preparar o ambiente

  • Sem Dockerfile nem arquivo Docker Compose
  • Tem um modelo de pull request
  • Sem guia de contribuição

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de apache/iceberg-python

Todas as issues de apache/iceberg-python

Issues semelhantes

Mais issues de Python

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.