Hacktoberfest 2026 : les issues que les mainteneurs ont marquées pour octobre, ouvertes et accessibles aux débutants. Parcourir les issues Hacktoberfest

Null list<struct<...>> is written and read as an empty list

Ouverte
  • Quelque chose ne va pas ?
#3,833 1 commentaire 1 réaction 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

@DG47 y travaille déjà.

Depuis le 2/10/2026.

  • #3834 par @slachiewicz — fermée sans fusion
  • #4065 par @DG47 — ouverte

Évaluation

Difficulté
2/5
Temps estimé
Une demi-journée
Accessibilité débutants
45/100
Type d'issue
Bug
Clarté
Clairement spécifiée
Activité
Active
Stack technique
python
Domaine
data-engineering, databases

Piste de recherche

Commencez à pyiceberg/io/pyarrow.py:2078, dans ArrowProjectionVisitor.list, et examinez tests/integration/test_reads.py::test_null_list_and_map. Vérifiez que la reproduction préserve les null struct lists dans le fichier Parquet écrit et lors de la lecture, puis exécutez le test d’intégration ciblé et le test unitaire pertinent ; le travail est terminé lorsque l’assertion restaurée réussit sans régression du traitement de primitive-list.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

Apache Iceberg version

main (development)

Please describe the bug 🐞

A null list<struct<...>> is silently rebuilt as an empty list. This is known on
the read path — tests/integration/test_reads.py::test_null_list_and_map currently
asserts the corrupted value, with the correct assertion commented out pending
apache/arrow#38809:

# This should be:
# assert arrow_table["col_list_with_struct"].to_pylist() == [None, [{'test': 1}]]
# Once https://github.com/apache/arrow/issues/38809 has been fixed
assert arrow_table["col_list_with_struct"].to_pylist() == [[], [{"test": 1}]]

Two things seem worth reporting on top of that.

It also affects the write path, where the consequence is worse. The Parquet file
pyiceberg writes contains an empty list, so the null is gone at rest and no reader —
pyiceberg, Spark, Trino — can recover it. On read the file is at least still correct.

It does not depend on the upstream Arrow fix. pa.LargeListArray.from_arrays
takes a mask argument — since well before pyiceberg's pyarrow>=18.0.0 floor — so
this particular null loss can be fixed independently of apache/arrow#38809, which is
still open.

This is the array<struct<>> case from #251. That issue was closed in March 2025 on
the strength of this test existing, but the assertion it makes is the corrupted one;
the array<int> case in the issue body was genuinely fixed by #252, while the
array<struct<test:int>> case in the issue title — which @HonahX flagged as
remaining broken in
https://github.com/apache/iceberg-python/pull/252#discussion_r1467065763 — was not.

Reproduction (write path)

pyiceberg 0.11.1, pyarrow 25.0.1:

import os, shutil, glob
import pyarrow as pa, pyarrow.parquet as pq
from pyiceberg.catalog.sql import SqlCatalog

WH = "/tmp/wh"; shutil.rmtree(WH, ignore_errors=True); os.makedirs(WH)

sch = pa.schema([
    pa.field("id", pa.int32(), nullable=False),
    pa.field("l_struct", pa.list_(pa.field("element", pa.struct([pa.field("x", pa.int32())]), nullable=True)), nullable=True),
    pa.field("l_int",    pa.list_(pa.field("element", pa.int32(), nullable=True)), nullable=True),
])
tbl = pa.table({"id": [1, 2, 3, 4],
                "l_struct": [[{"x": 1}], [], None, [{"x": 3}]],
                "l_int":    [[1],        [], None, [3]]}, schema=sch)

cat = SqlCatalog("r", uri=f"sqlite:///{WH}/c.db", warehouse=f"file://{WH}")
cat.create_namespace("ns")
it = cat.create_table("ns.t", schema=tbl.schema)
it.append(tbl)

out = it.scan().to_arrow()
for c in ("l_struct", "l_int"):
    print(f"{c:9s} in={tbl.column(c).to_pylist()!s:35s} out={out.column(c).to_pylist()}")

# the loss is already in the file on disk, not in the read path
f = glob.glob(f"{WH}/**/*.parquet", recursive=True)[0]
print("raw parquet:", pq.read_table(f).column("l_struct").to_pylist())

Output:

l_struct  in=[[{'x': 1}], [], None, [{'x': 3}]]  out=[[{'x': 1}], [], [], [{'x': 3}]]
l_int     in=[[1], [], None, [3]]                out=[[1], [], None, [3]]
raw parquet: [[{'x': 1}], [], [], [{'x': 3}]]

l_int round-trips correctly, and writing the same pa.Table with pq.write_table
preserves the null, so the loss is not pyarrow's.

Cause

ArrowProjectionVisitor.list rebuilds the array when the element is a struct
(pyiceberg/io/pyarrow.py:2078 on main @ 7539661):

if isinstance(value_array, pa.StructArray):
    # This can be removed once this has been fixed:
    # https://github.com/apache/arrow/issues/38809
    list_array = pa.LargeListArray.from_arrays(list_array.offsets, value_array)

from_arrays receives the offsets buffer alone, which cannot express a null list, and
no mask, so the validity bitmap is dropped. That is also why only this one shape is
affected: the struct visitor passes mask=struct_array.is_null(), the map visitor
does not rebuild at all, and a list whose element is a primitive never enters this
branch. The visitor runs on both paths, which is why the same root cause shows up as
the read-side assertion above and as the write-side corruption here.

Fix

Carrying the mask over is enough:

list_array = pa.LargeListArray.from_arrays(list_array.offsets, value_array, mask=list_array.is_null())

With that change the reproduction above returns None for both columns, and
test_null_list_and_map passes with its commented-out assertion restored. I have not
looked at int32-offset or sliced-array handling of this call, which the existing line
already relies on; that appears independent of the mask.

I have this on a branch with a unit test covering the write path and the integration
assertion un-commented, and can open a PR.

Found while testing a third-party Iceberg writer against pyiceberg as a reader.

Willingness to contribute
  • I can contribute a fix for this bug independently
Langage dominant
Python
Étoiles
1.1k
Forks
606
Merge moyen
1 j 11 h
PR mergées (30 j)
76

Préparer son environnement

  • Aucun Dockerfile ni fichier Docker Compose
  • Propose un modèle de pull request
  • Aucun guide de contribution

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de apache/iceberg-python

Toutes les issues de apache/iceberg-python

Issues similaires

Plus d'issues Python

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.