SanityCheckPlan fails after ProjectionPushdown for GROUP BY over a mark join on a struct field (regression from #26747)

Abierto
  • ¿Algo no está bien?
#27,318 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

@cetra3 ya está trabajando en esto.

Desde el 11/11/2026.

  • #27319 de @cetra3 — abierto

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
20/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Estancado
Stack tecnológico
rust, sql
Área
backend, databases

Línea de trabajo

Start by tracing the equivalence handling described in the issue through EnsureRequirements and the ProjectionPushdown physical optimizer rule, then inspect the related regression tests around #26747. Reproduce the SQL query from the issue and check how the join’s output partitioning is represented after projection pushdown. Done means the query plans and returns a and b without the final sanity check rejecting its distribution requirements.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Describe the bug

This query fails to plan on main:

SET datafusion.execution.target_partitions = 2;
SET datafusion.optimizer.hash_join_single_partition_threshold_rows = 0;
COPY (SELECT named_struct('k', 'a') AS s) TO 'data/t/1.parquet';
COPY (SELECT named_struct('k', 'b') AS s) TO 'data/t/2.parquet';
CREATE EXTERNAL TABLE t STORED AS PARQUET LOCATION 'data/t/';

SELECT s['k'] FROM t WHERE s['k'] = 'a' OR s['k'] IN (SELECT s['k'] FROM t) GROUP BY 1;

SanityCheckPlan: order and distribution requirements stopped holding after physical optimizer rule 'ProjectionPushdown'
caused by
Error during planning: Plan: [
AggregateExec: mode=SinglePartitioned, gby=[__datafusion_extracted_2@0 as __datafusion_extracted_2], aggr=[]
  FilterExec: __datafusion_extracted_3@1 = a OR mark@0, projection=[__datafusion_extracted_2@2]
    ProjectionExec: expr=[mark@0 as mark, get_field(s@1, k) as __datafusion_extracted_3, get_field(s@1, k) as __datafusion_extracted_2]
      HashJoinExec: mode=Partitioned, join_type=LeftMark, on=[(__datafusion_extracted_4@0, t.s[k]@0)], projection=[mark@2, s@1]
        RepartitionExec: partitioning=Hash([__datafusion_extracted_4@0], 2), input_partitions=2
          DataSourceExec: file_groups={...}, projection=[get_field(s@0, k) as __datafusion_extracted_4, s], file_type=parquet
        RepartitionExec: partitioning=Hash([t.s[k]@0], 2), input_partitions=2
          DataSourceExec: file_groups={...}, projection=[get_field(s@0, k) as t.s[k]], file_type=parquet
] does not satisfy distribution requirements: KeyPartitioned[[__datafusion_extracted_2@0]]). Child-0 output partitioning: Hash([__datafusion_extracted_4@0], 2)

The settings make the join Partitioned on this small data set.

Expected behavior

The query returns a and b, as it does at 0d3aac0bde.

Additional context

A bisect points to #26747 (99e723dd47). Its parent ce0e8293f9 plans the query.

#26747 records get_field(s, k) ≡ __datafusion_extracted_4 when projecting equivalences. That lets EnsureRequirements use the join's hash partitioning for the group key and skip a repartition, which is correct. Then ProjectionPushdown moves get_field(s, k) into the DataSourceExec projection and drops __datafusion_extracted_4 through the join's embedded projection. Neither step records the equivalence again. So the join's output partitioning becomes an unknown column, and the final check rejects the plan.

No rows move in that rewrite, so the query fails to plan and never returns wrong results.

Claude Code (an AI coding assistant) investigated this and wrote this issue on behalf of cetra3. A PR with a fix follows.

Lenguaje dominante
Rust
Estrellas
9.6k
Forks
2.6k
Merge medio
3 d 4 h
PR fusionados (30 d)
345

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de apache/datafusion

Todos los issues de apache/datafusion

Issues similares

Más issues de Rust

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.