Can't use side inputs with Combine.PerKeyWithHotKeyFanout · apache/beam#20637 | Good First Issue

(3 评论) (0 反应) (1 负责人)Java (4,097 fork)batch import

P3bugcoregood first issuejava

仓库指标

Star: (7,313 star)
PR 合并指标: (平均合并 6天) (30 天内合并 243 个 PR)

描述

PerKeyWithHotKeyFanout doesn't allow a user to specify a side input and access it in the CombineFn. When the function withHotKeyFanout is called, any sideinputs stored in the Combine.PerKey are dropped. (https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Combine.java#L1571):


    public PerKeyWithHotKeyFanout<K, InputT, OutputT> withHotKeyFanout(
        SerializableFunction<?
super K, Integer> hotKeyFanout) {
      return new PerKeyWithHotKeyFanout<>(fn, fnDisplayData, hotKeyFanout);

   }

We would like to be able to use side inputs and hot key fanout.

Imported from Jira BEAM-11005. Original Jira may contain additional context. Reported by: miagilepner.

贡献者指南

研究方向: 研究 `Combine.java` 第 1571 行的 `withHotKeyFanout` 方法，该方法会丢弃侧输入。确定在创建 `PerKeyWithHotKeyFanout` 时，如何保留侧输入并将其传递给 `CombineFn`。查看其他 `Combine` 变体以及 `WithSideInputs` 类中现有的侧输入处理方式。该问题要求为热键扇出添加侧输入支持，需要修改 `PerKeyWithHotKeyFanout` 的构造函数或方法以接受并存储侧输入，然后在评估期间将其传播给 `CombineFn`。
技术栈: java
领域: backend
议题类型: 缺陷
难度: 2
预计时间: 1-3 小时
活动状态: 活跃
清晰度: 清晰
前置要求: JavaApache Beam
新手友好度: 70