apache/beam

Trigger example in the cookbook failed to run on Dataflow

Open

#18,767 opened on Jun 3, 2022

View on GitHub
 (2 comments) (0 reactions) (1 assignee)Java (4,097 forks)batch import
P3bugexamplesgood first issuejava

Repository metrics

Stars
 (7,313 stars)
PR merge metrics
 (Avg merge 6d) (243 merged PRs in 30d)

Description

The example org.apache.beam.examples.cookbook.TriggerExample failed to run on Dataflow. Big query returns failure code 409 indicating it's trying to write to an non-empty table but WriteDisposition is set to WRITE_EMPTY (the default value). The example fails with this error even if the output table does not exist before the example is run. 

In fact, the four pipelines corresponding to the four trigger types write to the same table. The pipeline that finishes first will create the table and insert data into it. But the remaining ones can only append to that table. Hence, the problem can be avoided by setting WriteDisposition to WRITE_APPEND.

Imported from Jira BEAM-3844. Original Jira may contain additional context. Reported by: Zhaozhi.

Contributor guide

Research direction
Change WriteDisposition from WRITE EMPTY to WRITE APPEND in the TriggerExample to allow multiple pipelines to write to the same table.
Tech stack
java
Domain
backend
Issue type
Bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
2
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
Under 1 hour
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
JavaApache Beam
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
70