apache/beam

Can't use side inputs with Combine.PerKeyWithHotKeyFanout

Open

#20,637 opened on 2022年6月4日

GitHub で見る
 (3 comments) (0 reactions) (1 assignee)Java (4,097 forks)batch import
P3bugcoregood first issuejava

Repository metrics

Stars
 (7,313 stars)
PR merge metrics
 (平均マージ 6d) (30d で 243 merged PRs)

説明

PerKeyWithHotKeyFanout doesn't allow a user to specify a side input and access it in the CombineFn. When the function withHotKeyFanout is called, any sideinputs stored in the Combine.PerKey are dropped. (https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Combine.java#L1571): 


    public PerKeyWithHotKeyFanout<K, InputT, OutputT> withHotKeyFanout(
        SerializableFunction<?
super K, Integer> hotKeyFanout) {
      return new PerKeyWithHotKeyFanout<>(fn, fnDisplayData, hotKeyFanout);

   }

We would like to be able to use side inputs and hot key fanout.

Imported from Jira BEAM-11005. Original Jira may contain additional context. Reported by: miagilepner.

コントリビューターガイド