[Feature Request]: Honor --quota_project_id in Python GCS/Pub/Sub IO and the Java-backed BigQuery STORAGE_WRITE_API path

オープン
  • 何か問題がありますか？
#40,555 コメント 2 件 リアクション 0 件 担当者 1 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

@shahar1 がすでに取り組んでいます。

2026年10月13日 から。

評価

この issue はまだ評価されていません。

説明

dataflow io new feature P2 python
What would you like to happen?

Follow-up to #37431 / #39203.

#39203 adds a --quota_project_id pipeline option to Python GoogleCloudOptions, which attributes quota/billing of API requests to a chosen GCP project (the x-goog-user-project header). The option lives on the GCP-wide options class, but currently only the Python BigQuery IO transforms honor it (ReadFromBigQuery, ReadAllFromBigQuery, WriteToBigQuery with FILE_LOADS/STREAMING_INSERTS, ReadBigQueryChangeHistory).

It should be honored by the rest of the Python GCP IO so the option means what its name says. The client libraries already support it, so this is plumbing only:

  • GCS (apache_beam/io/gcp/gcsio.py): storage.Client(...) accepts client_options={'quota_project_id': ...} (honored by google.cloud.client.Client); resourcemanager_v3.ProjectsClient used by _get_project_number takes client_options too. GcsIO already receives pipeline_options, so the value is available where the client is built.
  • Pub/Sub (apache_beam/io/gcp/pubsub.py, runners/direct/transform_evaluator.py): pubsub.PublisherClient(...) / SubscriberClient(...) forward client_options to the GAPIC client, which accepts quota_project_id.
  • WriteToBigQuery(method=STORAGE_WRITE_API): this path expands to the Java StorageWriteToBigQuery transform. Java already supports --bigQueryQuotaProjectId (BigQueryOptions), but the Python option never reaches it: PipelineOptionsTranslation.fromProto converts lower_underscore → lowerCamel, so quota_project_id arrives as quotaProjectId, which no Java option reads. Options: have the Python transform/expansion forward it under the Java name, or have Java's BigQueryOptions.getBigQueryQuotaProjectId() fall back to a generic quotaProjectId. (Note: on Dataflow, options reach the Java worker in the legacy {"options": {...}} format without case conversion, so a solution must cover that path as well.)
  • Other GCP IOs that build clients (Bigtable, Spanner, …) should be checked the same way.

Until then, the option's help text says it is currently honored by BigQuery IO only.

Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

Issue Components
  • Component: Python SDK
  • Component: Java SDK
  • Component: Go SDK
  • Component: Typescript SDK
  • Component: IO connector
  • Component: Beam YAML
  • Component: Beam examples
  • Component: Beam playground
  • Component: Beam katas
  • Component: Website
  • Component: Infrastructure
  • Component: Spark Runner
  • Component: Flink Runner
  • Component: Prism Runner
  • Component: Twister2 Runner
  • Component: Hazelcast Jet Runner
  • Component: Google Cloud Dataflow Runner
主要言語
Java
スター
8.7k
フォーク
4.8k
平均マージ
2日 9時間
マージ済み PR（30日）
368

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

apache/beam のほかの issue

apache/beam の issue をすべて見る

似ている issue

Java の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。