REGRESSION: SanityCheckPlan error for GROUP BY ... HAVING on a CASE key with parquet filter pushdown

Aperta
  • Qualcosa non va?
#27,287 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@cetra3 ci sta già lavorando.

Dal 10/11/2026.

  • #27288 di @cetra3 — aperta

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
35/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Ferma
Stack tecnologico
rust, sql
Ambito
backend, databases

Direzione di ricerca

Start with the regression identified by git bisect at 99e723dd47 and trace how EnsureRequirements handles orderings and constants across the ProjectionExec inserted by common subexpression elimination. Reproduce with the SQL in the issue, using parquet filter pushdown; the fix is done when planning succeeds and the query returns e = 'y', count = 1. A linked open pull request already addresses this issue.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Posted by Claude (an AI assistant) on behalf of cetra3.

Describe the bug

With datafusion.execution.parquet.pushdown_filters = true, a query that groups by a CASE expression and pins the group key with HAVING fails to plan with SanityCheckPlan. The CASE repeats a subexpression, so common subexpression elimination adds a ProjectionExec above the scan.

The HAVING filter is pushed into the parquet predicate, which makes the group key constant. The partial AggregateExec relies on that and requires its input ordered by the key. But the ProjectionExec between the scan and the aggregate rewrites a = 'x' to __common_expr_1, and the constant does not survive that projection, so the final plan check rejects the plan.

This plans and returns the right row on 55.2.0.

To Reproduce

datafusion-cli at main (c8265b6872, 2026-11-10):

SET datafusion.execution.parquet.pushdown_filters = true;
COPY (SELECT * FROM (VALUES ('x', 'y'), ('x', 'z'), ('w', 'y')) AS v(a, b)) TO 't.parquet';
CREATE EXTERNAL TABLE t STORED AS PARQUET LOCATION 't.parquet';
SELECT CASE WHEN a = 'x' AND b = 'y' THEN b WHEN a = 'x' THEN 'z' END AS e, count(*)
FROM t
GROUP BY e
HAVING e = 'y';

SanityCheckPlan: the plan produced by physical optimizer rule 'EnsureRequirements' does not satisfy the order and distribution requirements
caused by
Error during planning: Plan: ["AggregateExec: mode=Partial, gby=[CASE WHEN __common_expr_1@0 AND b@1 = y THEN b@1 WHEN __common_expr_1@0 THEN z END as CASE WHEN t.a = Utf8(\"x\") AND t.b = Utf8(\"y\") THEN t.b WHEN t.a = Utf8(\"x\") THEN Utf8(\"z\") END], aggr=[count(Int64(1))]", "  ProjectionExec: expr=[a@0 = x as __common_expr_1, b@1 as b]", "    RepartitionExec: partitioning=RoundRobinBatch(32), input_partitions=1", "      DataSourceExec: file_groups={1 group: [[.../t.parquet]]}, projection=[a, b], file_type=parquet, predicate=CASE WHEN a@0 = x AND b@1 = y THEN b@1 WHEN a@0 = x THEN z END = y"] does not satisfy order requirements: [CASE WHEN __common_expr_1@0 AND b@1 = y THEN b@1 WHEN __common_expr_1@0 THEN z END ASC]. Child-0 order: []
Expected behavior

One row, e = 'y', count = 1, as 55.2.0 returns.

Additional context
  • git bisect between 55.2.0's merge base with main and main points at 99e723dd47, "fix: retain only valid orderings after order-preserving repartition" (#26747). Its parent ce0e8293f9 returns the right row.
  • It also fails with target_partitions = 1.
  • Each of these makes it plan and return the right row:
    • pushdown_filters = false.
    • An in-memory table (CREATE TABLE t AS VALUES ...) in place of parquet.
    • A CASE with no repeated subexpression, so CSE adds no projection, for example CASE WHEN a = 'x' THEN b ELSE 'z' END.
    • WHERE <the same CASE> = 'y' ... ORDER BY e with no GROUP BY.
  • This looks related to #26624 (fixed by #27123), where a constant was lost when equivalence properties were projected through a sort-merge join. That fix is in main, and this query still fails. The open PR #27191 does not fix it either. I built its head (434a1272bb) and got the same error.
Lingua principale
Rust
Stelle
9.6k
Fork
2.6k
Merge medio
3g 4h
PR unite (30g)
345

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/datafusion

Tutte le issue di apache/datafusion

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.