Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

DataFusion: aggregating a dictionary column across partitions overflows the key type ("Dictionary key bigger than the key type")

Abierto
  • ¿Algo no está bien?
#220 5 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
43/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Tranquilo
Stack tecnológico
python
Área
data

Línea de trabajo

The payload identifies repros/datafusion/dict_key_overflow.py on a linked branch as a minimal reproduction; start by running it with the stated DataFusion 54.0.0 and PyArrow 23.0.0 environment. The issue says a paste-in Rust test is in that repro's README, but gives no DataFusion source file to change. Done means the grouped aggregation succeeds when partitions have disjoint dictionary values that exceed the Int8 key range.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Tracking issue for an upstream DataFusion bug (to file against apache/datafusion). One of two blockers for dictionary-encoding coordinate columns (#217).

Summary

GROUP BY over a dictionary-typed column fails when the combined dictionary across partitions exceeds the per-batch key width:

DataFusion error: Arrow error: Dictionary key bigger than the key type

Each input partition is a valid Dictionary(Int8, Int64) — its dictionary has ≤128 distinct values, so an Int8 key is legal per batch. But the partitions carry disjoint values, so when the aggregate combines them the combined dictionary exceeds 128 entries and the Int8 key overflows.

Minimal repro

Pure datafusion + pyarrow, no xarray-sql, no network: repros/datafusion/dict_key_overflow.py (branch claude/datafusion-upstream-repros-fs1bqv; a paste-in Rust test is in the README).

# 50 partitions, each Dictionary(Int8, Int64) of 100 DISJOINT values
#   -> combined cardinality 5000 >> Int8 max (127)
ctx.sql("SELECT k, SUM(v) FROM t GROUP BY k")  # -> "Dictionary key bigger than the key type"

The disjoint values matter: when every partition shares the same dictionary values, arrow unifies them and there is no overflow — which is why this is intermittent and version-sensitive in the wild (it does not reproduce on arrow-rs 58.3 when the values coincide).

Expected

Combining dictionary-typed columns across partitions should not fail on a key type that was valid for each input batch. It should widen the key (Int8 → Int16 → …) or decode, since a producer cannot know per batch how large the combined dictionary will become downstream.

Environment

datafusion 54.0.0 / pyarrow 23.0.0 (arrow-rs 58.3.0).

Impact on xarray-sql

Blocks dictionary-encoding coordinate columns (#217): an unchunked coordinate repeated across partitions, or a chunked coordinate whose partitions hold disjoint slices, can exceed a narrow key under streaming aggregation. Reported downstream in #217 by @ghostiee-11.

Lenguaje dominante
Python
Estrellas
142
Forks
19
Merge medio
1 d 9 h
PR fusionados (30 d)
10

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de xqlsystems/xarray-sql

Todos los issues de xqlsystems/xarray-sql

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.