A store allocates about 10 MiB up front, however small the upload

Ouverte
  • Quelque chose ne va pas ?
#48 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 2 jours

Personne n'a encore pris cette issue.

Évaluation

Difficulté
4/5
Temps estimé
3-5 jours
Accessibilité débutants
48/100
Type d'issue
Fonctionnalité
Clarté
Plutôt claire
Activité
Active
Stack technique
typescript
Domaine
backend, performance

Piste de recherche

Start with collect() in src/streams.ts and uploadBlob() in src/store.ts, then trace their callers and the existing size limits. The issue proposes growing both buffers as data arrives, up to those limits; check how the current tests cover small and maximum-sized uploads. Done means uploads retain the stated limits while buffers no longer allocate their full capacity for small inputs.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

What happens. Every store allocates its buffers at their limits, whatever the upload's size:

  • collect() in src/streams.ts does new Uint8Array(limit) for the metadata part, so 5 MiB with the default limits;
  • uploadBlob() in src/store.ts allocates a 5 MiB part buffer before reading the blob, and a small blob ends up as one put of a slice of it.

So each store in flight holds about 10 MiB however small it is, and Admission has to cap stores at 2 per isolate to stay inside 128 MiB. When several tasks finish at once, the 503 concurrency_limit refusals that follow aren't retried by the client, so those entries are lost.

What our entries weigh. From one deployment's generations table, 1,897 ready rows (Vite+ 1.5.0, a monorepo's build, lint and test tasks):

Average Under 64 KiB Over 1 MiB Largest
value 669 KB 129 268 3.99 MB
blob 141 KB 1,754 74 13.1 MB

Most blobs are far under one part. The values average about 0.7 MB, against the 5 MiB buffer each one gets.

Suggested fix. Grow both buffers as bytes arrive, up to their limits, rather than allocating the limit up front. For the part buffer, start small and grow to PART_SIZE only when a blob gets that far. Memory then follows what's actually in flight, and the admission caps could rise a lot, leaving the rate limit to bound traffic.

Also worth a look: some values are close to the 4 MiB value limit (largest 3.99 MB), and those are task logs. A task with a slightly longer log would be refused as too large, and that's silent on the client side too.

We run a vendored copy at 5b06c2c with the caps raised to 6 stores and 8 reads. We're making the buffer change in our copy and can send it as a PR if that's welcome.

Langage dominant
TypeScript
Étoiles
8
Forks
0
Merge moyen
2 j 10 h
PR mergées (30 j)
40

Préparer son environnement

Ce projet ne fournit ni conteneur de développement, ni Dockerfile, ni guide de contribution : l'installation est à votre charge. Commencez par son README, et consultez notre guide de la première contribution pour les étapes générales.

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de voidzero-dev/vite-plus-remote-cache-cloudflare

Toutes les issues de voidzero-dev/vite-plus-remote-cache-cloudflare

Issues similaires

Plus d'issues TypeScript

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.