storage: zero-copy BidiReadObjectResponse parsing and off-loop CRC32C verification for async multi-range reads

Aperta
  • Qualcosa non va?
#18,629 0 commenti 0 reazioni 1 assegnatario Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@chandra-siri ci sta già lavorando.

Dal 12/10/2026.

Valutazione

Questa issue non è ancora stata valutata.

Descrizione

api: storage priority: p2 type: feature request

Summary of the feature request

AsyncMultiRangeDownloader reads from Rapid (zonal) buckets over the BidiReadObject gRPC stream. Per process, throughput is bound by the asyncio event-loop thread: every 2 MiB BidiReadObjectResponse is processed on that thread, under the GIL, before the next message can be received, and all streams of a process queue behind it.

Profiling one process reading 16 streams at 2.0 GB/s (c4-standard-192, bucket in the same zone, py-spy on the loop thread) gives 318 ms of loop-thread time per GB:

Step on the loop thread ms per GB
gRPC receive copy 123
BidiReadObjectResponse.deserialize (upb copies the payload: copy 1) 69
checksummed_data.content (proto-plus returns a new bytes: copy 2) 58
google_crc32c.value() inline 38
asyncio overhead 30

Two of the three large items are avoidable:

  1. The payload is copied twice by the generated parser. A parser that reads the message in place over the bytes object gRPC hands to the deserializer can give the read path a memoryview of the received message instead (257 us per 2 MiB message today, 5 us in place).
  2. CRC32C verification runs on the loop thread even though google-crc32c already releases the GIL for inputs of 1 MiB or more; hashing on a small worker pool overlaps it with the next receive.

Both need google_crc32c.value() to accept a memoryview; today it parses its argument with y# and raises TypeError: argument 1 must be read-only bytes-like object, so a view into the gRPC buffer cannot be verified without copying it back to bytes.

Proposed changes (three PRs, independent except that the parser depends on the google-crc32c release):

  • #18626 google-crc32c: value() / extend() / Checksum.update() accept any contiguous bytes-like object; the GIL is released for bytes-owned memory of 1 MiB or more (a bytes or a memoryview of one).
  • #18627 google-cloud-storage: opt-in zero-copy parser, create_mrd(..., zero_copy=True) (or GOOGLE_CLOUD_STORAGE_ZERO_COPY_BIDI_READ=1); only active when the installed google-crc32c can hash a memoryview, otherwise the generated parser is used. With it on, buffer.write() receives a read-only memoryview of the received message.
  • #18628 google-cloud-storage: chunks of 1 MiB or more are hashed on a 2-thread pool and written to their buffer only after the checksum matches, in arrival order; on by default, GOOGLE_CLOUD_STORAGE_CRC32C_OFFLOAD_MIN_BYTES=0 restores inline verification.

fsspec/gcsfs is the main consumer of this path and would opt in; its sinks copy from the view into the caller's buffer.

Desired code experience

import io
from google.cloud.storage.asyncio.async_grpc_client import AsyncGrpcClient
from google.cloud.storage.asyncio.async_multi_range_downloader import AsyncMultiRangeDownloader

client = AsyncGrpcClient()
mrd = await AsyncMultiRangeDownloader.create_mrd(
    client, "my-rapid-bucket", "my-object", zero_copy=True
)
buffers = [io.BytesIO() for _ in range(4)]
# buffer.write() receives memoryview chunks that alias the received gRPC
# message; BytesIO, binary files and bytearray sinks accept them unchanged.
await mrd.download_ranges([(i * 16 << 20, 16 << 20, buf) for i, buf in enumerate(buffers)])

Off-loop verification needs no code change; it applies to download_ranges with enable_checksum=True.

Expected results

Same machine and bucket, 16 streams per process, 4 x 16 MiB per download_ranges call into preallocated buffers, CRC32C verification on, aggregate GB/s:

Processes main zero-copy parser (#18627, median of 3 interleaved runs) off-loop CRC32C (#18628, median of 9)
1 1.55 2.62 about 2.0 for both arms (unchanged)
8 9.76 11.29 8.17 -> 9.59 (+17%, led in 7 of 9 paired runs)

With all three changes together, one process goes from 1.6 to 3.5 GB/s and eight processes from 9.8 to 13.5 GB/s; loop-thread time per GB drops from 318 ms to 195 ms.

Environment: google-cloud-storage at 053dbc9872ad, google-crc32c 1.9.0 (main arm), grpcio 1.84.0, protobuf 7.36 (upb backend), Python 3.13, c4-standard-192 in us-central1-c.

Lingua principale
Python
Stelle
5.4k
Fork
1.8k
Merge medio
2g 7h
PR unite (30g)
173

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di googleapis/google-cloud-python

Tutte le issue di googleapis/google-cloud-python

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.