[Feature Request]: Honor --quota_project_id in Python GCS/Pub/Sub IO and the Java-backed BigQuery STORAGE_WRITE_API path

未关闭
  • 哪里不对？
#40,555 2 条评论 0 个 reaction 已指派 1 人 在 GitHub 查看

维护者通常 1 天内回复

@shahar1 已经在做这个了。

开始于 2026年10月13日。

评估

这个 Issue 还没有评估数据。

描述

dataflow io new feature P2 python
What would you like to happen?

Follow-up to #37431 / #39203.

#39203 adds a --quota_project_id pipeline option to Python GoogleCloudOptions, which attributes quota/billing of API requests to a chosen GCP project (the x-goog-user-project header). The option lives on the GCP-wide options class, but currently only the Python BigQuery IO transforms honor it (ReadFromBigQuery, ReadAllFromBigQuery, WriteToBigQuery with FILE_LOADS/STREAMING_INSERTS, ReadBigQueryChangeHistory).

It should be honored by the rest of the Python GCP IO so the option means what its name says. The client libraries already support it, so this is plumbing only:

  • GCS (apache_beam/io/gcp/gcsio.py): storage.Client(...) accepts client_options={'quota_project_id': ...} (honored by google.cloud.client.Client); resourcemanager_v3.ProjectsClient used by _get_project_number takes client_options too. GcsIO already receives pipeline_options, so the value is available where the client is built.
  • Pub/Sub (apache_beam/io/gcp/pubsub.py, runners/direct/transform_evaluator.py): pubsub.PublisherClient(...) / SubscriberClient(...) forward client_options to the GAPIC client, which accepts quota_project_id.
  • WriteToBigQuery(method=STORAGE_WRITE_API): this path expands to the Java StorageWriteToBigQuery transform. Java already supports --bigQueryQuotaProjectId (BigQueryOptions), but the Python option never reaches it: PipelineOptionsTranslation.fromProto converts lower_underscore → lowerCamel, so quota_project_id arrives as quotaProjectId, which no Java option reads. Options: have the Python transform/expansion forward it under the Java name, or have Java's BigQueryOptions.getBigQueryQuotaProjectId() fall back to a generic quotaProjectId. (Note: on Dataflow, options reach the Java worker in the legacy {"options": {...}} format without case conversion, so a solution must cover that path as well.)
  • Other GCP IOs that build clients (Bigtable, Spanner, …) should be checked the same way.

Until then, the option's help text says it is currently honored by BigQuery IO only.

Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

Issue Components
  • Component: Python SDK
  • Component: Java SDK
  • Component: Go SDK
  • Component: Typescript SDK
  • Component: IO connector
  • Component: Beam YAML
  • Component: Beam examples
  • Component: Beam playground
  • Component: Beam katas
  • Component: Website
  • Component: Infrastructure
  • Component: Spark Runner
  • Component: Flink Runner
  • Component: Prism Runner
  • Component: Twister2 Runner
  • Component: Hazelcast Jet Runner
  • Component: Google Cloud Dataflow Runner
主要语言
Java
星标
8.7k
派生
4.8k
平均合并
2 天 12 小时
30 天内合并 PR
371

环境准备

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

apache/beam 的其他 Issue

查看 apache/beam 的全部 Issue

相似的 Issue

更多 Java Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。