apache/beam
在 GitHub 查看Multiple batch joins on the same key results in multiple shuffles
Open
#19,097 创建于 2022年6月3日
P3dslgood first issueimprovementsql
仓库指标
- Star
- (7,313 star)
- PR 合并指标
- (平均合并 6天) (30 天内合并 243 个 PR)
描述
The query like this:
SELECT a.*, b.*, c.* FROM a JOIN b ON a.some_id = b.some_id JOIN c ON a.some_id = c.some_id;
results in two shuffles. Can probably be optimized.
Relevant code:
-
BeamJoinRel implements Join in SQL: https://github.com/apache/beam/blob/1675b0f843ed34de8ba6f3676f794db80b40139d/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRel.java#L194
-
CoGBK Join implementation: https://github.com/apache/beam/blob/279a05604b83a54e8e5a79e13d8761f94841f326/sdks/java/extensions/join-library/src/main/java/org/apache/beam/sdk/extensions/joinlibrary/Join.java#L36
Imported from Jira BEAM-5049. Original Jira may contain additional context. Reported by: kedin.