Hacktoberfest 2026 : les issues que les mainteneurs ont marquées pour octobre, ouvertes et accessibles aux débutants. Parcourir les issues Hacktoberfest

Segfault on large multi-column Iceberg upserts

Ouverte
  • Quelque chose ne va pas ?
#3,508 1 commentaire 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

@steven-winfield-quantohm y travaille déjà.

Depuis le 15/6/2026.

  • #3509 par @steven-winfield-quantohm — ouverte
  • #4027 par @azwanzuharimi — ouverte

Évaluation

Difficulté
4/5
Temps estimé
3-5 jours
Accessibilité débutants
64/100
Type d'issue
Bug
Clarté
Clairement spécifiée
Activité
Calme
Stack technique
python
Domaine
data-engineering, databases

Piste de recherche

Commencez dans pyiceberg.table.upsert_util.create_match_filter et exécutez iceberg_upsert_segfault_repro.py pour reproduire le crash de l’upsert multi-colonne, en utilisant pyiceberg-stacktrace.txt pour confirmer le chemin de canonicalisation. Regroupez les tuples de clés en moins de disjonctions tout en préservant les correspondances exactes, puis vérifiez que le cas synthétique à faible cardinalité ne provoque plus de segfault et examinez la réserve concernant la forte cardinalité.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

Apache Iceberg version

0.11.0 (latest release)

Please describe the bug 🐞

When upserting into an Iceberg table, PyIceberg first scans the target table to
find which existing rows match the source rows' key columns. It builds that
"matching" predicate in pyiceberg.table.upsert_util.create_match_filter:

  • For a single join column it emits one flat In(col, [v1, v2, ...]).
    PyArrow lowers this to a single is_in compute node, no matter how many
    values it contains — so single-column upserts of huge tables are fine.

  • For a multi-column key it instead emits one disjunct per distinct key
    tuple::

    Or(And(c1 == v1, c2 == w1),
   And(c1 == v2, c2 == w2),
   ...)                          # ONE disjunct PER ROW

PyIceberg builds that Or as a balanced tree, so the Python side copes.
But when the expression is handed to PyArrow's dataset scanner as a filter, the
C++ expression engine canonicalises it: Dataset::GetFragments calls
SimplifyWithGuarantee → Canonicalize, which flattens the associative
or_kleene chain and then recurses over it. With tens of thousands of
disjuncts that recursion overflows the C++ call stack and the process
segfaults (SIGSEGV) — typically after several minutes of work, with a
backtrace full of arrow::compute::Canonicalize / ModifyExpression
frames.

Reference: https://github.com/apache/iceberg-python/issues/3272

Note that apache/iceberg-python#3448 addresses a different upsert segfault (a
per-batch Acero re-filter in _task_to_record_batches, mostly observed on
Apple Silicon). It does not touch the GetFragments canonicalisation path
exercised here, so it does not help with this crash.

The fix

Produce a predicate that matches exactly the same rows, but with far fewer
disjuncts. Group the key tuples and emit a single In over whichever column
collapses to the fewest distinct "prefix" combinations (choosing that column
makes the result independent of the caller's column ordering)::

Or(And(c1 == v1, c2 IN [w, x, y]),
   And(c1 == v2, c2 IN [z]),
   ...)                            # one disjunct per distinct PREFIX

The disjunct count drops from "number of rows" to "number of distinct prefix
values". In the synthetic data below there are 50 000 unique ids spread over
just 50 group values, so the predicate shrinks from 50 000 disjuncts to 50 —
shallow enough that PyArrow's canonicaliser no longer overflows.

Caveat

This helps whenever at least one key column is low-cardinality (or, equivalently,
one column is near-unique and can be folded into the In). A genuinely
high-cardinality composite key — where every column is near-unique and all of
them are needed to identify a row — still produces roughly one disjunct per row
even after grouping, and can still overflow. For that pathological case the
only robust option is to upsert in smaller batches.

pyiceberg-stacktrace.txt

iceberg_upsert_segfault_repro.py

Willingness to contribute
  • I can contribute a fix for this bug independently
  • I would be willing to contribute a fix for this bug with guidance from the Iceberg community
  • I cannot contribute a fix for this bug at this time
Langage dominant
Python
Étoiles
1.2k
Forks
618
Merge moyen
1 j 18 h
PR mergées (30 j)
70

Préparer son environnement

  • Aucun Dockerfile ni fichier Docker Compose
  • Propose un modèle de pull request
  • Aucun guide de contribution

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de apache/iceberg-python

Toutes les issues de apache/iceberg-python

Issues similaires

Plus d'issues Python

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.