Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

Stream multipart UploadPart on Workers (avoid buffering parts in WASM memory)

Abierto
  • ¿Algo no está bien?
#89 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
68/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Tranquilo
Stack tecnológico
rust
Área
api, backend

Línea de trabajo

Lee crates/cf-workers/src/backend.rs, comparando send_raw con la plantilla de forward, y luego sigue execute_multipart y la rama NeedsBody en crates/core/src/proxy.rs. Sigue S3RequestSigner para el presign de los parámetros de consulta; se considera terminado cuando UploadPart se transmite a través de Forward sin almacenamiento en búfer del cuerpo de WASM ni cálculo del hash del payload, mientras las demás operaciones multipart conservan el comportamiento indicado.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

enhancement

Problem

Multipart UploadPart flows through the NeedsBody → send_raw path, which buffers the entire part body in WASM linear memory on the Cloudflare Workers runtime:

  1. collect_js_body copies the request stream JS→WASM (array_buffer() + to_vec())
  2. execute_multipart computes SHA256 over the whole part for the outbound signature (hash_payload)
  3. send_raw copies WASM→JS (js_sys::Uint8Array::from) to hand to fetch

So each part incurs two full body copies across the JS/WASM boundary + an O(N) SHA256, and is held in the worker's 128 MB memory heap. Single-object PutObject does not have this problem — it uses the zero-copy forward() path (the ReadableStream is handed straight to fetch, signed with UNSIGNED-PAYLOAD so no hashing).

For typical part sizes (~8 MiB) this is a few ms of CPU and fine. But the 128 MB memory limit caps the maximum multipart_chunksize a client can use before the worker OOMs.

Proposed fix

Route UploadPart through the streaming forward() path instead of send_raw:

  • Generate a presigned URL for PUT /{key}?partNumber=N&uploadId=X. object_store's Signer::signed_url only takes a Path (no query params), so presign manually via S3RequestSigner — it already builds SigV4 canonical query strings (incl. value-less params, fixed in #88).
  • Return HandlerAction::Forward so the ReadableStream streams straight through fetch: no WASM buffering, no payload hash.

CompleteMultipartUpload stays on send_raw (its small XML body must be signed/forwarded), but that body is tiny. CreateMultipartUpload/AbortMultipartUpload are body-less and could also move to the forward path opportunistically.

Interaction with Cloudflare's request-body size limit

Even with streaming, every request through a Worker is bounded by the plan's request-body limit (100 MB Free/Pro, 200 MB Business, 500 MB Enterprise default). Streaming changes which limit binds:

  • Today (buffered): a part is capped by the smaller of the 128 MB memory limit and the plan body limit.
  • After streaming: a part is capped only by the plan body limit (no WASM buffering).

So streaming lets clients use the full plan body limit for part sizes. It does not lift the body limit itself — that's a hard platform constraint (see separate doc note). Large objects must still be uploaded as multipart with part sizes ≤ the plan limit.

Impact

  • Removes the 128 MB part-size ceiling on Workers (raises it to the plan body limit).
  • Eliminates 2 body copies + 1 SHA256 per part → lower CPU.
  • Brings UploadPart to parity with the zero-copy PutObject path.

References

  • crates/cf-workers/src/backend.rs — send_raw (buffers); forward is the zero-copy template
  • crates/core/src/proxy.rs — execute_multipart, dispatch_operation (the NeedsBody arm)
  • Deferred from #88 to keep that PR scoped to correctness.
Lenguaje dominante
Rust
Estrellas
20
Forks
1
Merge medio
1 d 8 h
PR fusionados (30 d)
10

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de developmentseed/multistore

Todos los issues de developmentseed/multistore

Issues similares

Más issues de Rust

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.