Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Stream multipart UploadPart on Workers (avoid buffering parts in WASM memory)

Aperta
  • Qualcosa non va?
#89 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
68/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Tranquilla
Stack tecnologico
rust
Ambito
api, backend

Direzione di ricerca

Leggi crates/cf-workers/src/backend.rs, confrontando send_raw con il template forward, quindi segui execute_multipart e il ramo NeedsBody in crates/core/src/proxy.rs. Segui S3RequestSigner per il presigning dei parametri di query; il lavoro è completo quando UploadPart viene trasmesso tramite Forward senza buffering del body WASM né hashing del payload, mentre le altre operazioni multipart mantengono il comportamento dichiarato.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

enhancement

Problem

Multipart UploadPart flows through the NeedsBody → send_raw path, which buffers the entire part body in WASM linear memory on the Cloudflare Workers runtime:

  1. collect_js_body copies the request stream JS→WASM (array_buffer() + to_vec())
  2. execute_multipart computes SHA256 over the whole part for the outbound signature (hash_payload)
  3. send_raw copies WASM→JS (js_sys::Uint8Array::from) to hand to fetch

So each part incurs two full body copies across the JS/WASM boundary + an O(N) SHA256, and is held in the worker's 128 MB memory heap. Single-object PutObject does not have this problem — it uses the zero-copy forward() path (the ReadableStream is handed straight to fetch, signed with UNSIGNED-PAYLOAD so no hashing).

For typical part sizes (~8 MiB) this is a few ms of CPU and fine. But the 128 MB memory limit caps the maximum multipart_chunksize a client can use before the worker OOMs.

Proposed fix

Route UploadPart through the streaming forward() path instead of send_raw:

  • Generate a presigned URL for PUT /{key}?partNumber=N&uploadId=X. object_store's Signer::signed_url only takes a Path (no query params), so presign manually via S3RequestSigner — it already builds SigV4 canonical query strings (incl. value-less params, fixed in #88).
  • Return HandlerAction::Forward so the ReadableStream streams straight through fetch: no WASM buffering, no payload hash.

CompleteMultipartUpload stays on send_raw (its small XML body must be signed/forwarded), but that body is tiny. CreateMultipartUpload/AbortMultipartUpload are body-less and could also move to the forward path opportunistically.

Interaction with Cloudflare's request-body size limit

Even with streaming, every request through a Worker is bounded by the plan's request-body limit (100 MB Free/Pro, 200 MB Business, 500 MB Enterprise default). Streaming changes which limit binds:

  • Today (buffered): a part is capped by the smaller of the 128 MB memory limit and the plan body limit.
  • After streaming: a part is capped only by the plan body limit (no WASM buffering).

So streaming lets clients use the full plan body limit for part sizes. It does not lift the body limit itself — that's a hard platform constraint (see separate doc note). Large objects must still be uploaded as multipart with part sizes ≤ the plan limit.

Impact

  • Removes the 128 MB part-size ceiling on Workers (raises it to the plan body limit).
  • Eliminates 2 body copies + 1 SHA256 per part → lower CPU.
  • Brings UploadPart to parity with the zero-copy PutObject path.

References

  • crates/cf-workers/src/backend.rs — send_raw (buffers); forward is the zero-copy template
  • crates/core/src/proxy.rs — execute_multipart, dispatch_operation (the NeedsBody arm)
  • Deferred from #88 to keep that PR scoped to correctness.
Lingua principale
Rust
Stelle
20
Fork
1
Merge medio
2g 3h
PR unite (30g)
11

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di developmentseed/multistore

Tutte le issue di developmentseed/multistore

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.