storage: zero-copy BidiReadObjectResponse parsing and off-loop CRC32C verification for async multi-range reads

未關閉
  • 哪裡不對？
#18,629 0 則留言 0 個 reaction 已指派 1 人 在 GitHub 檢視

維護者通常 1 天內回覆

@chandra-siri 已經在處理了。

開始於 2026年10月12日。

評估

這個 Issue 還沒有評估資料。

描述

api: storage priority: p2 type: feature request

Summary of the feature request

AsyncMultiRangeDownloader reads from Rapid (zonal) buckets over the BidiReadObject gRPC stream. Per process, throughput is bound by the asyncio event-loop thread: every 2 MiB BidiReadObjectResponse is processed on that thread, under the GIL, before the next message can be received, and all streams of a process queue behind it.

Profiling one process reading 16 streams at 2.0 GB/s (c4-standard-192, bucket in the same zone, py-spy on the loop thread) gives 318 ms of loop-thread time per GB:

Step on the loop thread ms per GB
gRPC receive copy 123
BidiReadObjectResponse.deserialize (upb copies the payload: copy 1) 69
checksummed_data.content (proto-plus returns a new bytes: copy 2) 58
google_crc32c.value() inline 38
asyncio overhead 30

Two of the three large items are avoidable:

  1. The payload is copied twice by the generated parser. A parser that reads the message in place over the bytes object gRPC hands to the deserializer can give the read path a memoryview of the received message instead (257 us per 2 MiB message today, 5 us in place).
  2. CRC32C verification runs on the loop thread even though google-crc32c already releases the GIL for inputs of 1 MiB or more; hashing on a small worker pool overlaps it with the next receive.

Both need google_crc32c.value() to accept a memoryview; today it parses its argument with y# and raises TypeError: argument 1 must be read-only bytes-like object, so a view into the gRPC buffer cannot be verified without copying it back to bytes.

Proposed changes (three PRs, independent except that the parser depends on the google-crc32c release):

  • #18626 google-crc32c: value() / extend() / Checksum.update() accept any contiguous bytes-like object; the GIL is released for bytes-owned memory of 1 MiB or more (a bytes or a memoryview of one).
  • #18627 google-cloud-storage: opt-in zero-copy parser, create_mrd(..., zero_copy=True) (or GOOGLE_CLOUD_STORAGE_ZERO_COPY_BIDI_READ=1); only active when the installed google-crc32c can hash a memoryview, otherwise the generated parser is used. With it on, buffer.write() receives a read-only memoryview of the received message.
  • #18628 google-cloud-storage: chunks of 1 MiB or more are hashed on a 2-thread pool and written to their buffer only after the checksum matches, in arrival order; on by default, GOOGLE_CLOUD_STORAGE_CRC32C_OFFLOAD_MIN_BYTES=0 restores inline verification.

fsspec/gcsfs is the main consumer of this path and would opt in; its sinks copy from the view into the caller's buffer.

Desired code experience

import io
from google.cloud.storage.asyncio.async_grpc_client import AsyncGrpcClient
from google.cloud.storage.asyncio.async_multi_range_downloader import AsyncMultiRangeDownloader

client = AsyncGrpcClient()
mrd = await AsyncMultiRangeDownloader.create_mrd(
    client, "my-rapid-bucket", "my-object", zero_copy=True
)
buffers = [io.BytesIO() for _ in range(4)]
# buffer.write() receives memoryview chunks that alias the received gRPC
# message; BytesIO, binary files and bytearray sinks accept them unchanged.
await mrd.download_ranges([(i * 16 << 20, 16 << 20, buf) for i, buf in enumerate(buffers)])

Off-loop verification needs no code change; it applies to download_ranges with enable_checksum=True.

Expected results

Same machine and bucket, 16 streams per process, 4 x 16 MiB per download_ranges call into preallocated buffers, CRC32C verification on, aggregate GB/s:

Processes main zero-copy parser (#18627, median of 3 interleaved runs) off-loop CRC32C (#18628, median of 9)
1 1.55 2.62 about 2.0 for both arms (unchanged)
8 9.76 11.29 8.17 -> 9.59 (+17%, led in 7 of 9 paired runs)

With all three changes together, one process goes from 1.6 to 3.5 GB/s and eight processes from 9.8 to 13.5 GB/s; loop-thread time per GB drops from 318 ms to 195 ms.

Environment: google-cloud-storage at 053dbc9872ad, google-crc32c 1.9.0 (main arm), grpcio 1.84.0, protobuf 7.36 (upb backend), Python 3.13, c4-standard-192 in us-central1-c.

主要語言
Python
星號
5.4k
分支
1.8k
平均合併
2 天 7 小時
30 天內合併 PR
173

環境準備

從這裡開始

  1. 先讀完整個 Issue，再讀專案的貢獻指南。
  2. 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
  3. Fork 儲存庫，在一個分支上完成修改。
  4. 送出 Pull Request，並在描述裡引用這個 Issue 編號。

googleapis/google-cloud-python 的其他 Issue

查看 googleapis/google-cloud-python 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。