apache/beam

[Task]: Update BigQueryIO.setTriggeringFrequency documentation

Open

#23,949 opened on Nov 2, 2022

View on GitHub
 (20 comments) (0 reactions) (3 assignees)Java (7,313 stars) (4,097 forks)batch import
P2bigquerygcpgood first issueiojavatask

Description

What needs to happen?

BigQueryIO currently documents triggering frequency as:

This is only applicable when the write method is set to {@link Method#FILE_LOADS}, and only when writing an unbounded {@link PCollection}.

However, we use triggering frequency for STORAGE_WRITE_API and STORAGE_API_AT_LEAST_ONCE as well. This documentation should be updated to reflect this.

In addition, during the validation component of BigQueryIO.write().expand(), we check for unbounded && (file loads || storage write api) to see if a triggering frequency is set. This should probably also be checked if storage write at least once is set.

Issue Priority

Priority: 2

Issue Component

Component: io-java-gcp

Contributor guide

Tech stack
java
Domain
documentationbackenddata
Issue type
documentation
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
blocked
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
JavaApache Beam SDKBigQueryIO basics
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
60
Research direction
Investigate the BigQueryIO source code, specifically the setTriggeringFrequency method documentation and the validation logic in BigQueryIO.write().expand(). The issue comments may indicate further context. Update the Javadoc to mention STORAGE WRITE API and STORAGE API AT LEAST ONCE, and potentially add a validation check for those methods.