apache/beam

DoFn.process should raise exception if something other than a List is returned

Open

#18,712 创建于 2022年6月3日

在 GitHub 查看
 (4 评论) (0 反应) (1 负责人)Java (4,097 fork)batch import
P3coregood first issueimprovementpython

仓库指标

Star
 (7,313 star)
PR 合并指标
 (平均合并 6天) (30 天内合并 243 个 PR)

描述

The process method of DoFns can either return values or yield values. In the case of returning values, it expects a List of elements to be returned. When returning a single value, it is easy to forget this, and return the value instead.

Correct way:

class SomeDoFn(beam.DoFn)   def process(self, elem):     return ['a']

Incorrect way:

class SomeDoFn(beam.DoFn)   def process(self, elem):     return 'a'

A pipeline with the incorrect DoFn will fail will a cryptic error message without a direct indication that the actual error is due to SomeDoFn returning an element instead of a List containing that element. This issue is very time-consuming to track down.

It would be good if the pipeline could raise an exception or otherwise indicate that the DoFn is incorrectly returning an element instead of a List to make it easier to identify the error.

Imported from Jira BEAM-3530. Original Jira may contain additional context. Reported by: chuanyu.

贡献者指南