[Feature Request]: Use unique id for Python BigQueryIO · apache/beam#22733

(9 评论) (0 反应) (1 负责人)Java (7,313 star) (4,097 fork)batch import

P2gcpgood first issueionew featurepython

描述

What would you like to happen?

There can be a collision when 2 pipelines using templates are loading to BigQuery at the same time to the same temp_location.

@baeminbo found there is code to use unique_id [1], but it seems that templates can re-use the same uuid. It is fixed for Java[2] by moving the UUID generation into a DoFn of a ParDo

[1] https://github.com/apache/beam/blob/v2.34.0/sdks/python/apache_beam/io/gcp/bigquery.py#L2399 [2] https://github.com/apache/beam/blob/v2.34.0/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java#L1094-L1105

Issue Priority

Priority: 2

Issue Component

Component: io-py-gcp

贡献者指南

技术栈: pythonjava
领域: datacloud
议题类型: feature
难度: 3
预计时间: 1-2 days
活动状态: active
清晰度: clear
前置要求: PythonApache BeamBigQueryUUID generation
新手友好度: 45
研究方向: The issue describes a collision problem in Python BigQueryIO when two pipelines use templates with the same temp location. It references existing code in apache beam/io/gcp/bigquery.py around line 2399 and a Java fix in BigQueryIO.java at lines 1094-1105. The solution involves moving UUID generation into a DoFn within a ParDo. A newcomer should examine the Python file, understand the current UUID usage, and replicate the Java fix pattern. The issue has an assignee (not specified) and comments, so check for any existing work before starting.