Can't use side inputs with Combine.PerKeyWithHotKeyFanout · apache/beam#20637 | Good First Issue

(3 留言) (0 反應) (1 負責人)Java (4,097 fork)batch import

P3bugcoregood first issuejava

倉庫指標

Star: (7,313 star)
PR 合併指標: (平均合併 6天) (30 天內合併 243 個 PR)

描述

PerKeyWithHotKeyFanout doesn't allow a user to specify a side input and access it in the CombineFn. When the function withHotKeyFanout is called, any sideinputs stored in the Combine.PerKey are dropped. (https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Combine.java#L1571):


    public PerKeyWithHotKeyFanout<K, InputT, OutputT> withHotKeyFanout(
        SerializableFunction<?
super K, Integer> hotKeyFanout) {
      return new PerKeyWithHotKeyFanout<>(fn, fnDisplayData, hotKeyFanout);

   }

We would like to be able to use side inputs and hot key fanout.

Imported from Jira BEAM-11005. Original Jira may contain additional context. Reported by: miagilepner.

貢獻者指南

研究方向: 研究 `Combine.java` 第 1571 行的 `withHotKeyFanout` 方法，該方法會丟棄側輸入。確定在建立 `PerKeyWithHotKeyFanout` 時，如何保留側輸入並將其傳遞給 `CombineFn`。查看其他 `Combine` 變體以及 `WithSideInputs` 類別中現有的側輸入處理方式。該問題要求為熱鍵扇出添加側輸入支援，需要修改 `PerKeyWithHotKeyFanout` 的建構函式或方法以接受並儲存側輸入，然後在評估期間將其傳播給 `CombineFn`。
技術棧: java
領域: backend
議題類型: 錯誤
難度: 2
預計時間: 1-3 小時
活動狀態: 活躍
清晰度: 清晰
前置要求: JavaApache Beam
新手友善度: 70