_DeleteFiles.delete_data_file() silently drops explicit file references

Inizia in pyiceberg/table/update/snapshot.py, in _DeleteFiles._compute_deletes, intorno alla riga 598, e segui il modo in cui delete_data_file() registra i file specificati esplicitamente. Esegui la riproduzione dell’issue, quindi verifica che un file di dati referenziato esplicitamente venga eliminato dopo il commit, mentre l’eliminazione basata su predicato continui a funzionare. Il lavoro è completato quando la tabella è vuota invece di conservare [1, 2, 3].

Description

_DeleteFiles.delete_data_file() silently drops explicit file references because _compute_deletes resets self._deleted_data_files = set() before scanning manifests by predicate.

The delete_data_file() method is inherited from _SnapshotProducer and adds to self._deleted_data_files. However, when _DeleteFiles._compute_deletes runs (triggered by _deleted_entries()), it resets this field to an empty set and repopulates it only from files matched by the predicate evaluator. Any explicitly added files are silently lost.

While the high-level table.delete() API only uses predicates (so this isn't hit in normal usage), the low-level update_snapshot().delete() API exposes delete_data_file() as a public method. Calling it produces no error and no effect.

Reproduction

import pyarrow as pa
from pyiceberg.catalog import load_catalog
from pyiceberg.schema import Schema
from pyiceberg.types import LongType, NestedField

catalog = load_catalog("default")
catalog.create_namespace("default")
table = catalog.create_table(
    "default.delete_explicit",
    Schema(NestedField(1, "x", LongType(), required=False)),
)
table.append(pa.table({"x": [1, 2, 3]}))

data_file = next(iter(table.scan().plan_files())).file

# This should delete the file but silently does nothing
with table.transaction() as tx:
    delete_snapshot = tx.update_snapshot().delete()
    delete_snapshot.delete_data_file(data_file)

# Bug: table still has [1, 2, 3]
print(table.scan().to_arrow()["x"].to_pylist())

Expected: table is empty after commit.
Actual: table still has [1, 2, 3].

Root cause

In pyiceberg/table/update/snapshot.py, _DeleteFiles._compute_deletes (line ~598):

self._deleted_data_files = set()  # <-- overwrites any explicit files added via delete_data_file()

Then it repopulates from predicate matches only. Since no predicate was set (AlwaysFalse by default), nothing matches, nothing is deleted.

Suggested fix

Before resetting, preserve explicit files and include them in the deletion scan:

# Preserve files explicitly requested for deletion
explicit_deletes = set(self._deleted_data_files)
self._deleted_data_files = set()

# ... existing predicate-based scan ...
# After the scan, also mark explicitly requested files as deleted:
for entry in manifest_entries:
    if entry.data_file in explicit_deletes:
        # mark as deleted

Alternatively, prevent calling delete_data_file() on _DeleteFiles entirely by raising NotImplementedError.

Related

Follow-up observation from #3818 review. The _OverwriteFiles path handles delete_data_file() correctly because it does not reset the field.