[Feature Request]: Use unique id for Python BigQueryIO · apache/beam#22733

(9 comments) (0 reactions) (1 assignee)Java (7,313 stars) (4,097 forks)batch import

P2gcpgood first issueionew featurepython

説明

What would you like to happen?

There can be a collision when 2 pipelines using templates are loading to BigQuery at the same time to the same temp_location.

@baeminbo found there is code to use unique_id [1], but it seems that templates can re-use the same uuid. It is fixed for Java[2] by moving the UUID generation into a DoFn of a ParDo

[1] https://github.com/apache/beam/blob/v2.34.0/sdks/python/apache_beam/io/gcp/bigquery.py#L2399 [2] https://github.com/apache/beam/blob/v2.34.0/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java#L1094-L1105

Issue Priority

Priority: 2

Issue Component

Component: io-py-gcp

コントリビューターガイド

技術スタック: pythonjava
領域: datacloud
Issue 種別: feature
難度: 3
推定時間: 1-2 days
活動状況: active
明確さ: clear
前提条件: PythonApache BeamBigQueryUUID generation
初心者向け度: 45
調査方針: The issue describes a collision problem in Python BigQueryIO when two pipelines use templates with the same temp location. It references existing code in apache beam/io/gcp/bigquery.py around line 2399 and a Java fix in BigQueryIO.java at lines 1094-1105. The solution involves moving UUID generation into a DoFn within a ParDo. A newcomer should examine the Python file, understand the current UUID usage, and replicate the Java fix pattern. The issue has an assignee (not specified) and comments, so check for any existing work before starting.