Support disabling output batch coalescing in HashJoinExec (and LimitedBatchCoalescer)

Aperta
  • Qualcosa non va?
#27,109 3 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
48/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
rust
Ambito
backend, databases, performance

Direzione di ricerca

Start with datafusion/physical-plan/src/joins/hash_join/stream.rs, where HashJoinStream uses LimitedBatchCoalescer, and inspect the coalescer implementation and configuration wiring for datafusion.execution.coalesce_batches. Then trace how HashJoinExec exposes operator settings and how other operators use the coalescer. Done means the setting can disable buffering while preserving empty-batch handling and fetch enforcement, with the execution configuration wired into the relevant operators.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Is your feature request related to a problem or challenge?

HashJoinStream unconditionally coalesces matched output rows across probe batches up to datafusion.execution.batch_size (8192) via LimitedBatchCoalescer:
https://github.com/apache/datafusion/blob/a7c7c08365a37f0b46bce81b1fd45515f3f18045/datafusion/physical-plan/src/joins/hash_join/stream.rs#L586-L587

When the probe side is expensive to pull from and already ordered for a downstream limit, coalescing across probe batches forces unnecessary work on the probe input before downstream operators can see already-matched rows. For example, when an incremental top-K probe scan feeds a chain of HashJoinExecs under ORDER BY ... LIMIT K, LimitPushdown can set fetch = Some(K) only on the outermost join; inner joins have fetch = None, so their LimitedBatchCoalescer buffers matches and repeatedly pulls from the probe scan until 8192 matches accumulate or the scan is exhausted.

Relatedly, datafusion.execution.coalesce_batches became a no-op after FilterExec (#18630), HashJoinExec (#18972), and RepartitionExec (#19449) internalized LimitedBatchCoalescer and #19622 removed the CoalesceBatches rule:
https://github.com/apache/datafusion/blob/a7c7c08365a37f0b46bce81b1fd45515f3f18045/datafusion/common/src/config.rs#L928-L932

Describe the solution you'd like

Support disabling coalescing on LimitedBatchCoalescer (via biggest_coalesce_batch_size = Some(0) so non-empty batches bypass buffering while empty batches are skipped and fetch is still enforced), exposed via:

  1. A per-operator HashJoinExec::with_coalesce_batches(bool) setting (so physical optimizer rules can disable coalescing selectively on demand-driven joins).
  2. Wiring datafusion.execution.coalesce_batches into operators that use LimitedBatchCoalescer.
Describe alternatives you've considered
  • Lowering datafusion.execution.batch_size globally degrades bulk operators and build-side scans in the same query.
  • A per-operator batch_size override on HashJoinExec still buffers batches of <= batch_size / 2 rows and shrinks the per-step match cap in lookup_join_hashmap.
Additional context

None.

Lingua principale
Rust
Stelle
9.6k
Fork
2.6k
Merge medio
3g 4h
PR unite (30g)
345

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/datafusion

Tutte le issue di apache/datafusion

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.