Clustered (fanout disabled) writes fail on sorted input because batches are split in HashMap order

Abierto
  • ¿Algo no está bien?
#39 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

@CaptainAni187 ya está trabajando en esto.

Desde el 20/10/2026.

  • #81 de @CaptainAni187 — abierto

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
55/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
rust
Área
database

Línea de trabajo

Start with crates/datafusion/src/task_writer.rs at the linked splitter and ClusteredWriter call sites, then inspect crates/iceberg/src/arrow/record_batch_partition_splitter.rs. Reproduce by executing IcebergTableProvider::insert_into directly with fanout disabled, batch_size 8, target_partitions 1, and 200 rows. Done means sorted partitioned input writes successfully without the previously closed partition error.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

bug
Describe the bug

With fanout disabled, TaskWriter splits each batch with iceberg-rust's RecordBatchPartitionSplitter::split and hands the pieces to ClusteredWriter (task_writer.rs#L129-L135, task_writer.rs#L217-L237).

The splitter groups rows in a HashMap, so the partitions within a batch come out in arbitrary order. ClusteredWriter closes the current partition when it sees a different one, and rejects any partition it has already closed. Input that is correctly sorted by partition therefore fails whenever a partition spans a batch boundary. For example, batch N holds [A, B] but is emitted as B then A, and batch N+1 starts with B.

To Reproduce

Execute the plan returned by IcebergTableProvider::insert_into directly, so the SortExec is still present (#38 explains why the optimized plan loses it). Use a table partitioned by identity(cat) with fanout disabled, 200 rows with cat = id % 10, batch_size = 8 and target_partitions = 1.

9 of 10 runs fail with:

The input is not sorted! Cannot write to partition that was previously closed
Expected behavior

Input sorted by partition is written successfully.

Additional context

Fixing #38 alone won't make fanout-disabled writes work. The splitter lives in iceberg-rust (crates/iceberg/src/arrow/record_batch_partition_splitter.rs). Either it emits partitions in first-occurrence order, or TaskWriter orders each batch's pieces by row position before passing them to ClusteredWriter.

Lenguaje dominante
Rust
Estrellas
30
Forks
30
Merge medio
3 d 19 h
PR fusionados (30 d)
15

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de apache/datafusion-iceberg

Todos los issues de apache/datafusion-iceberg

Issues similares

Más issues de Rust

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.