Support disabling output batch coalescing in HashJoinExec (and LimitedBatchCoalescer)

未关闭
  • 哪里不对？
#27,109 3 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

还没有人认领这个 Issue。

评估

难度
4/5
预计耗时
3-5 天
新手友好度
48/100
Issue 类型
功能
描述清晰度
基本清楚
活跃度
活跃
技术栈
rust
领域
backend, databases, performance

调研方向

Start with datafusion/physical-plan/src/joins/hash_join/stream.rs, where HashJoinStream uses LimitedBatchCoalescer, and inspect the coalescer implementation and configuration wiring for datafusion.execution.coalesce_batches. Then trace how HashJoinExec exposes operator settings and how other operators use the coalescer. Done means the setting can disable buffering while preserving empty-batch handling and fetch enforcement, with the execution configuration wired into the relevant operators.

由索引模型根据 Issue 内容生成。

描述

Is your feature request related to a problem or challenge?

HashJoinStream unconditionally coalesces matched output rows across probe batches up to datafusion.execution.batch_size (8192) via LimitedBatchCoalescer:
https://github.com/apache/datafusion/blob/a7c7c08365a37f0b46bce81b1fd45515f3f18045/datafusion/physical-plan/src/joins/hash_join/stream.rs#L586-L587

When the probe side is expensive to pull from and already ordered for a downstream limit, coalescing across probe batches forces unnecessary work on the probe input before downstream operators can see already-matched rows. For example, when an incremental top-K probe scan feeds a chain of HashJoinExecs under ORDER BY ... LIMIT K, LimitPushdown can set fetch = Some(K) only on the outermost join; inner joins have fetch = None, so their LimitedBatchCoalescer buffers matches and repeatedly pulls from the probe scan until 8192 matches accumulate or the scan is exhausted.

Relatedly, datafusion.execution.coalesce_batches became a no-op after FilterExec (#18630), HashJoinExec (#18972), and RepartitionExec (#19449) internalized LimitedBatchCoalescer and #19622 removed the CoalesceBatches rule:
https://github.com/apache/datafusion/blob/a7c7c08365a37f0b46bce81b1fd45515f3f18045/datafusion/common/src/config.rs#L928-L932

Describe the solution you'd like

Support disabling coalescing on LimitedBatchCoalescer (via biggest_coalesce_batch_size = Some(0) so non-empty batches bypass buffering while empty batches are skipped and fetch is still enforced), exposed via:

  1. A per-operator HashJoinExec::with_coalesce_batches(bool) setting (so physical optimizer rules can disable coalescing selectively on demand-driven joins).
  2. Wiring datafusion.execution.coalesce_batches into operators that use LimitedBatchCoalescer.
Describe alternatives you've considered
  • Lowering datafusion.execution.batch_size globally degrades bulk operators and build-side scans in the same query.
  • A per-operator batch_size override on HashJoinExec still buffers batches of <= batch_size / 2 rows and shrinks the per-step match cap in lookup_join_hashmap.
Additional context

None.

主要语言
Rust
星标
9.6k
派生
2.6k
平均合并
3 天 4 小时
30 天内合并 PR
345

环境准备

在 Codespaces 中打开

在浏览器里用你自己的 GitHub 账号启动这个项目的开发容器。

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

apache/datafusion 的其他 Issue

查看 apache/datafusion 的全部 Issue

相似的 Issue

更多 Rust Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。