Support disabling output batch coalescing in HashJoinExec (and LimitedBatchCoalescer)

Aberta
  • Algo errado?
#27,109 3 comentários 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 1 dia

Ninguém assumiu esta issue ainda.

Avaliação

Dificuldade
4/5
Tempo estimado
3-5 dias
Facilidade para iniciantes
48/100
Tipo de issue
Funcionalidade
Clareza
Razoavelmente clara
Status de atividade
Ativa
Stack de tecnologia
rust
Domínio
backend, databases, performance

Direção de pesquisa

Start with datafusion/physical-plan/src/joins/hash_join/stream.rs, where HashJoinStream uses LimitedBatchCoalescer, and inspect the coalescer implementation and configuration wiring for datafusion.execution.coalesce_batches. Then trace how HashJoinExec exposes operator settings and how other operators use the coalescer. Done means the setting can disable buffering while preserving empty-batch handling and fetch enforcement, with the execution configuration wired into the relevant operators.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

Is your feature request related to a problem or challenge?

HashJoinStream unconditionally coalesces matched output rows across probe batches up to datafusion.execution.batch_size (8192) via LimitedBatchCoalescer:
https://github.com/apache/datafusion/blob/a7c7c08365a37f0b46bce81b1fd45515f3f18045/datafusion/physical-plan/src/joins/hash_join/stream.rs#L586-L587

When the probe side is expensive to pull from and already ordered for a downstream limit, coalescing across probe batches forces unnecessary work on the probe input before downstream operators can see already-matched rows. For example, when an incremental top-K probe scan feeds a chain of HashJoinExecs under ORDER BY ... LIMIT K, LimitPushdown can set fetch = Some(K) only on the outermost join; inner joins have fetch = None, so their LimitedBatchCoalescer buffers matches and repeatedly pulls from the probe scan until 8192 matches accumulate or the scan is exhausted.

Relatedly, datafusion.execution.coalesce_batches became a no-op after FilterExec (#18630), HashJoinExec (#18972), and RepartitionExec (#19449) internalized LimitedBatchCoalescer and #19622 removed the CoalesceBatches rule:
https://github.com/apache/datafusion/blob/a7c7c08365a37f0b46bce81b1fd45515f3f18045/datafusion/common/src/config.rs#L928-L932

Describe the solution you'd like

Support disabling coalescing on LimitedBatchCoalescer (via biggest_coalesce_batch_size = Some(0) so non-empty batches bypass buffering while empty batches are skipped and fetch is still enforced), exposed via:

  1. A per-operator HashJoinExec::with_coalesce_batches(bool) setting (so physical optimizer rules can disable coalescing selectively on demand-driven joins).
  2. Wiring datafusion.execution.coalesce_batches into operators that use LimitedBatchCoalescer.
Describe alternatives you've considered
  • Lowering datafusion.execution.batch_size globally degrades bulk operators and build-side scans in the same query.
  • A per-operator batch_size override on HashJoinExec still buffers batches of <= batch_size / 2 rows and shrinks the per-step match cap in lookup_join_hashmap.
Additional context

None.

Linguagem predominante
Rust
Estrelas
9.6k
Forks
2.6k
Merge médio
3d 4h
PRs com merge (30d)
345

Preparar o ambiente

Abrir no Codespaces

Inicia o contêiner de desenvolvimento do projeto no navegador, com a sua própria conta do GitHub.

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de apache/datafusion

Todas as issues de apache/datafusion

Issues semelhantes

Mais issues de Rust

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.