apache/beam

[Feature Request]: Use unique id for Python BigQueryIO

Open

#22,733 opened on Aug 15, 2022

View on GitHub
 (9 comments) (0 reactions) (1 assignee)Java (7,313 stars) (4,097 forks)batch import
P2gcpgood first issueionew featurepython

Description

What would you like to happen?

There can be a collision when 2 pipelines using templates are loading to BigQuery at the same time to the same temp_location.

@baeminbo found there is code to use unique_id [1], but it seems that templates can re-use the same uuid. It is fixed for Java[2] by moving the UUID generation into a DoFn of a ParDo

[1] https://github.com/apache/beam/blob/v2.34.0/sdks/python/apache_beam/io/gcp/bigquery.py#L2399 [2] https://github.com/apache/beam/blob/v2.34.0/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java#L1094-L1105

Issue Priority

Priority: 2

Issue Component

Component: io-py-gcp

Contributor guide

Tech stack
pythonjava
Domain
datacloud
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-2 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
PythonApache BeamBigQueryUUID generation
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
45
Research direction
The issue describes a collision problem in Python BigQueryIO when two pipelines use templates with the same temp location. It references existing code in apache beam/io/gcp/bigquery.py around line 2399 and a Java fix in BigQueryIO.java at lines 1094-1105. The solution involves moving UUID generation into a DoFn within a ParDo. A newcomer should examine the Python file, understand the current UUID usage, and replicate the Java fix pattern. The issue has an assignee (not specified) and comments, so check for any existing work before starting.
[Feature Request]: Use unique id for Python BigQueryIO · apache/beam#22733 | Good First Issue