storage: zero-copy BidiReadObjectResponse parsing and off-loop CRC32C verification for async multi-range reads

Đang mở
  • Có gì không ổn?
#18,629 0 bình luận 0 reaction 1 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

@chandra-siri đang làm issue này rồi.

Từ ngày 12/10/2026.

Đánh giá

Issue này chưa được đánh giá.

Mô tả

api: storage priority: p2 type: feature request

Summary of the feature request

AsyncMultiRangeDownloader reads from Rapid (zonal) buckets over the BidiReadObject gRPC stream. Per process, throughput is bound by the asyncio event-loop thread: every 2 MiB BidiReadObjectResponse is processed on that thread, under the GIL, before the next message can be received, and all streams of a process queue behind it.

Profiling one process reading 16 streams at 2.0 GB/s (c4-standard-192, bucket in the same zone, py-spy on the loop thread) gives 318 ms of loop-thread time per GB:

Step on the loop thread ms per GB
gRPC receive copy 123
BidiReadObjectResponse.deserialize (upb copies the payload: copy 1) 69
checksummed_data.content (proto-plus returns a new bytes: copy 2) 58
google_crc32c.value() inline 38
asyncio overhead 30

Two of the three large items are avoidable:

  1. The payload is copied twice by the generated parser. A parser that reads the message in place over the bytes object gRPC hands to the deserializer can give the read path a memoryview of the received message instead (257 us per 2 MiB message today, 5 us in place).
  2. CRC32C verification runs on the loop thread even though google-crc32c already releases the GIL for inputs of 1 MiB or more; hashing on a small worker pool overlaps it with the next receive.

Both need google_crc32c.value() to accept a memoryview; today it parses its argument with y# and raises TypeError: argument 1 must be read-only bytes-like object, so a view into the gRPC buffer cannot be verified without copying it back to bytes.

Proposed changes (three PRs, independent except that the parser depends on the google-crc32c release):

  • #18626 google-crc32c: value() / extend() / Checksum.update() accept any contiguous bytes-like object; the GIL is released for bytes-owned memory of 1 MiB or more (a bytes or a memoryview of one).
  • #18627 google-cloud-storage: opt-in zero-copy parser, create_mrd(..., zero_copy=True) (or GOOGLE_CLOUD_STORAGE_ZERO_COPY_BIDI_READ=1); only active when the installed google-crc32c can hash a memoryview, otherwise the generated parser is used. With it on, buffer.write() receives a read-only memoryview of the received message.
  • #18628 google-cloud-storage: chunks of 1 MiB or more are hashed on a 2-thread pool and written to their buffer only after the checksum matches, in arrival order; on by default, GOOGLE_CLOUD_STORAGE_CRC32C_OFFLOAD_MIN_BYTES=0 restores inline verification.

fsspec/gcsfs is the main consumer of this path and would opt in; its sinks copy from the view into the caller's buffer.

Desired code experience

import io
from google.cloud.storage.asyncio.async_grpc_client import AsyncGrpcClient
from google.cloud.storage.asyncio.async_multi_range_downloader import AsyncMultiRangeDownloader

client = AsyncGrpcClient()
mrd = await AsyncMultiRangeDownloader.create_mrd(
    client, "my-rapid-bucket", "my-object", zero_copy=True
)
buffers = [io.BytesIO() for _ in range(4)]
# buffer.write() receives memoryview chunks that alias the received gRPC
# message; BytesIO, binary files and bytearray sinks accept them unchanged.
await mrd.download_ranges([(i * 16 << 20, 16 << 20, buf) for i, buf in enumerate(buffers)])

Off-loop verification needs no code change; it applies to download_ranges with enable_checksum=True.

Expected results

Same machine and bucket, 16 streams per process, 4 x 16 MiB per download_ranges call into preallocated buffers, CRC32C verification on, aggregate GB/s:

Processes main zero-copy parser (#18627, median of 3 interleaved runs) off-loop CRC32C (#18628, median of 9)
1 1.55 2.62 about 2.0 for both arms (unchanged)
8 9.76 11.29 8.17 -> 9.59 (+17%, led in 7 of 9 paired runs)

With all three changes together, one process goes from 1.6 to 3.5 GB/s and eight processes from 9.8 to 13.5 GB/s; loop-thread time per GB drops from 318 ms to 195 ms.

Environment: google-cloud-storage at 053dbc9872ad, google-crc32c 1.9.0 (main arm), grpcio 1.84.0, protobuf 7.36 (upb backend), Python 3.13, c4-standard-192 in us-central1-c.

Ngôn ngữ chính
Python
Star
5.4k
Fork
1.8k
Merge trung bình
2 ngày 7 giờ
Pull request đã merge (30 ngày)
173

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của googleapis/google-cloud-python

Tất cả issue của googleapis/google-cloud-python

Issue tương tự

Thêm issue về Python

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.