feat(bigquery): Support queryResultsFormat and compressionCodec in query_and_wait()

未关闭
  • 哪里不对？
#18,047 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

还没有人认领这个 Issue。

评估

难度
4/5
预计耗时
3-5 天
新手友好度
52/100
Issue 类型
功能
描述清晰度
基本清楚
活跃度
冷清
技术栈
python
领域
api, data

调研方向

从 Client.query_and_wait() 开始，然后查看 _job_helpers.py 中的 BigQuery 请求负载，以及 table.py 中的 RowIterator 和 Arrow 解析。对比相关的 PR 18027，并验证两个选项都能被接受并发送到 API，同时压缩的 Arrow 结果支持所述的 to_arrow() 和 to_dataframe() 使用场景。

由索引模型根据 Issue 内容生成。

描述

api: bigquery priority: p3 type: feature request
Determine this is the right repository
  • I determined this is the correct repository in which to report this feature request.
Summary of the feature request
Feature Description

Add support for query_results_format (e.g., "ARROW") and compression_codec (e.g., "LZ4_FRAME") in Client.query_and_wait() and related job helper functions in google-cloud-bigquery.

Use Case & Problem Statement

Currently, Client.query_and_wait() executes a query and fetches results, but does not allow callers to request the underlying query results format as Apache Arrow streams or specify compression options directly during the fetch process.

For high-performance data processing pipelines using Apache Arrow or PyArrow, requesting query results in Arrow format with compression significantly reduces network payload size, deserialization overhead, and total query result fetch latency.

Proposed Solution
  1. Update google.cloud.bigquery.client.Client.query_and_wait() to accept optional parameters:
    • query_results_format: Optional string specifying result serialization format (e.g. "ARROW").
    • compression_codec: Optional string specifying compression algorithm
  2. Update underlying helper functions in _job_helpers.py to populate these options in the request payload sent to the BigQuery API.
  3. Update RowIterator and Arrow parsing logic in table.py to seamlessly handle Arrow record batch streams when query_results_format="ARROW" is specified.
Alternative Approaches Considered
  • Manually constructing custom QueryJob instances and extracting Arrow record batches post-execution, which requires boilerplate code and loses the convenience of query_and_wait().
Additional Context

Related PR: https://github.com/googleapis/google-cloud-python/pull/18027

Desired code experience

from google.cloud import bigquery

client = bigquery.Client()

query = "SELECT * FROM bigquery-public-data.usa_names.usa_1910_2013 LIMIT 10000"

Request query results directly in Apache Arrow format with LZ4_FRAME compression

results = client.query_and_wait(
query,
query_results_format="ARROW",
compression_codec="LZ4_FRAME",
)

Convert directly to PyArrow Table or Pandas DataFrame

arrow_table = results.to_arrow()
df = results.to_dataframe()

Expected results
  1. Client.query_and_wait() accepts query_results_format and compression_codec parameters.
  2. The BigQuery API receives these parameters in the job execution request.
  3. Query results are returned and parsed efficiently via compressed Apache Arrow streams into the resulting RowIterator / Arrow table without client errors.
API client name and version

No response

Use case

e.g. This feature would be useful in a scenario where I need to process large amounts of data efficiently.

Additional context

https://github.com/googleapis/google-cloud-python/pull/18027

主要语言
Python
星标
5.4k
派生
1.8k
平均合并
2 天 7 小时
30 天内合并 PR
173

环境准备

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

googleapis/google-cloud-python 的其他 Issue

查看 googleapis/google-cloud-python 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。