storage: zero-copy BidiReadObjectResponse parsing and off-loop CRC32C verification for async multi-range reads

Abierto
  • ¿Algo no está bien?
#18,629 0 comentarios 0 reacciones 1 asignado Ver en GitHub

Los mantenedores suelen responder en 1 día

@chandra-siri ya está trabajando en esto.

Desde el 12/10/2026.

Evaluación

Este issue todavía no se ha evaluado.

Descripción

api: storage priority: p2 type: feature request

Summary of the feature request

AsyncMultiRangeDownloader reads from Rapid (zonal) buckets over the BidiReadObject gRPC stream. Per process, throughput is bound by the asyncio event-loop thread: every 2 MiB BidiReadObjectResponse is processed on that thread, under the GIL, before the next message can be received, and all streams of a process queue behind it.

Profiling one process reading 16 streams at 2.0 GB/s (c4-standard-192, bucket in the same zone, py-spy on the loop thread) gives 318 ms of loop-thread time per GB:

Step on the loop thread ms per GB
gRPC receive copy 123
BidiReadObjectResponse.deserialize (upb copies the payload: copy 1) 69
checksummed_data.content (proto-plus returns a new bytes: copy 2) 58
google_crc32c.value() inline 38
asyncio overhead 30

Two of the three large items are avoidable:

  1. The payload is copied twice by the generated parser. A parser that reads the message in place over the bytes object gRPC hands to the deserializer can give the read path a memoryview of the received message instead (257 us per 2 MiB message today, 5 us in place).
  2. CRC32C verification runs on the loop thread even though google-crc32c already releases the GIL for inputs of 1 MiB or more; hashing on a small worker pool overlaps it with the next receive.

Both need google_crc32c.value() to accept a memoryview; today it parses its argument with y# and raises TypeError: argument 1 must be read-only bytes-like object, so a view into the gRPC buffer cannot be verified without copying it back to bytes.

Proposed changes (three PRs, independent except that the parser depends on the google-crc32c release):

  • #18626 google-crc32c: value() / extend() / Checksum.update() accept any contiguous bytes-like object; the GIL is released for bytes-owned memory of 1 MiB or more (a bytes or a memoryview of one).
  • #18627 google-cloud-storage: opt-in zero-copy parser, create_mrd(..., zero_copy=True) (or GOOGLE_CLOUD_STORAGE_ZERO_COPY_BIDI_READ=1); only active when the installed google-crc32c can hash a memoryview, otherwise the generated parser is used. With it on, buffer.write() receives a read-only memoryview of the received message.
  • #18628 google-cloud-storage: chunks of 1 MiB or more are hashed on a 2-thread pool and written to their buffer only after the checksum matches, in arrival order; on by default, GOOGLE_CLOUD_STORAGE_CRC32C_OFFLOAD_MIN_BYTES=0 restores inline verification.

fsspec/gcsfs is the main consumer of this path and would opt in; its sinks copy from the view into the caller's buffer.

Desired code experience

import io
from google.cloud.storage.asyncio.async_grpc_client import AsyncGrpcClient
from google.cloud.storage.asyncio.async_multi_range_downloader import AsyncMultiRangeDownloader

client = AsyncGrpcClient()
mrd = await AsyncMultiRangeDownloader.create_mrd(
    client, "my-rapid-bucket", "my-object", zero_copy=True
)
buffers = [io.BytesIO() for _ in range(4)]
# buffer.write() receives memoryview chunks that alias the received gRPC
# message; BytesIO, binary files and bytearray sinks accept them unchanged.
await mrd.download_ranges([(i * 16 << 20, 16 << 20, buf) for i, buf in enumerate(buffers)])

Off-loop verification needs no code change; it applies to download_ranges with enable_checksum=True.

Expected results

Same machine and bucket, 16 streams per process, 4 x 16 MiB per download_ranges call into preallocated buffers, CRC32C verification on, aggregate GB/s:

Processes main zero-copy parser (#18627, median of 3 interleaved runs) off-loop CRC32C (#18628, median of 9)
1 1.55 2.62 about 2.0 for both arms (unchanged)
8 9.76 11.29 8.17 -> 9.59 (+17%, led in 7 of 9 paired runs)

With all three changes together, one process goes from 1.6 to 3.5 GB/s and eight processes from 9.8 to 13.5 GB/s; loop-thread time per GB drops from 318 ms to 195 ms.

Environment: google-cloud-storage at 053dbc9872ad, google-crc32c 1.9.0 (main arm), grpcio 1.84.0, protobuf 7.36 (upb backend), Python 3.13, c4-standard-192 in us-central1-c.

Lenguaje dominante
Python
Estrellas
5.4k
Forks
1.8k
Merge medio
2 d 7 h
PR fusionados (30 d)
173

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de googleapis/google-cloud-python

Todos los issues de googleapis/google-cloud-python

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.