storage: zero-copy BidiReadObjectResponse parsing and off-loop CRC32C verification for async multi-range reads

Offen
  • Stimmt etwas nicht?
#18,629 0 Kommentare 0 Reaktionen 1 zugewiesene Person Auf GitHub ansehen

Maintainer antworten meist innerhalb von 1 Tag

@chandra-siri arbeitet bereits daran.

Seit 12.10.2026.

Bewertung

Dieses Issue wurde noch nicht bewertet.

Beschreibung

api: storage priority: p2 type: feature request

Summary of the feature request

AsyncMultiRangeDownloader reads from Rapid (zonal) buckets over the BidiReadObject gRPC stream. Per process, throughput is bound by the asyncio event-loop thread: every 2 MiB BidiReadObjectResponse is processed on that thread, under the GIL, before the next message can be received, and all streams of a process queue behind it.

Profiling one process reading 16 streams at 2.0 GB/s (c4-standard-192, bucket in the same zone, py-spy on the loop thread) gives 318 ms of loop-thread time per GB:

Step on the loop thread ms per GB
gRPC receive copy 123
BidiReadObjectResponse.deserialize (upb copies the payload: copy 1) 69
checksummed_data.content (proto-plus returns a new bytes: copy 2) 58
google_crc32c.value() inline 38
asyncio overhead 30

Two of the three large items are avoidable:

  1. The payload is copied twice by the generated parser. A parser that reads the message in place over the bytes object gRPC hands to the deserializer can give the read path a memoryview of the received message instead (257 us per 2 MiB message today, 5 us in place).
  2. CRC32C verification runs on the loop thread even though google-crc32c already releases the GIL for inputs of 1 MiB or more; hashing on a small worker pool overlaps it with the next receive.

Both need google_crc32c.value() to accept a memoryview; today it parses its argument with y# and raises TypeError: argument 1 must be read-only bytes-like object, so a view into the gRPC buffer cannot be verified without copying it back to bytes.

Proposed changes (three PRs, independent except that the parser depends on the google-crc32c release):

  • #18626 google-crc32c: value() / extend() / Checksum.update() accept any contiguous bytes-like object; the GIL is released for bytes-owned memory of 1 MiB or more (a bytes or a memoryview of one).
  • #18627 google-cloud-storage: opt-in zero-copy parser, create_mrd(..., zero_copy=True) (or GOOGLE_CLOUD_STORAGE_ZERO_COPY_BIDI_READ=1); only active when the installed google-crc32c can hash a memoryview, otherwise the generated parser is used. With it on, buffer.write() receives a read-only memoryview of the received message.
  • #18628 google-cloud-storage: chunks of 1 MiB or more are hashed on a 2-thread pool and written to their buffer only after the checksum matches, in arrival order; on by default, GOOGLE_CLOUD_STORAGE_CRC32C_OFFLOAD_MIN_BYTES=0 restores inline verification.

fsspec/gcsfs is the main consumer of this path and would opt in; its sinks copy from the view into the caller's buffer.

Desired code experience

import io
from google.cloud.storage.asyncio.async_grpc_client import AsyncGrpcClient
from google.cloud.storage.asyncio.async_multi_range_downloader import AsyncMultiRangeDownloader

client = AsyncGrpcClient()
mrd = await AsyncMultiRangeDownloader.create_mrd(
    client, "my-rapid-bucket", "my-object", zero_copy=True
)
buffers = [io.BytesIO() for _ in range(4)]
# buffer.write() receives memoryview chunks that alias the received gRPC
# message; BytesIO, binary files and bytearray sinks accept them unchanged.
await mrd.download_ranges([(i * 16 << 20, 16 << 20, buf) for i, buf in enumerate(buffers)])

Off-loop verification needs no code change; it applies to download_ranges with enable_checksum=True.

Expected results

Same machine and bucket, 16 streams per process, 4 x 16 MiB per download_ranges call into preallocated buffers, CRC32C verification on, aggregate GB/s:

Processes main zero-copy parser (#18627, median of 3 interleaved runs) off-loop CRC32C (#18628, median of 9)
1 1.55 2.62 about 2.0 for both arms (unchanged)
8 9.76 11.29 8.17 -> 9.59 (+17%, led in 7 of 9 paired runs)

With all three changes together, one process goes from 1.6 to 3.5 GB/s and eight processes from 9.8 to 13.5 GB/s; loop-thread time per GB drops from 318 ms to 195 ms.

Environment: google-cloud-storage at 053dbc9872ad, google-crc32c 1.9.0 (main arm), grpcio 1.84.0, protobuf 7.36 (upb backend), Python 3.13, c4-standard-192 in us-central1-c.

Vorherrschende Sprache
Python
Sterne
5.4k
Forks
1.8k
Ø Merge
2 T. 7 Std.
Gemergte PRs (30 T.)
173

Entwicklungsumgebung

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus googleapis/google-cloud-python

Alle Issues in googleapis/google-cloud-python

Ähnliche Issues

Weitere Issues zu Python

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.