Hacktoberfest 2026 : les issues que les mainteneurs ont marquées pour octobre, ouvertes et accessibles aux débutants. Parcourir les issues Hacktoberfest

Capture memory is bounded in uncompressed bytes: bound the crawler's queue and compress as the capture reads, so the cap means stored size

Ouverte
  • Quelque chose ne va pas ?
#235 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

Personne n'a encore pris cette issue.

Évaluation

Difficulté
5/5
Temps estimé
Plus d'une semaine
Accessibilité débutants
38/100
Type d'issue
Fonctionnalité
Clarté
Plutôt claire
Activité
Active
Stack technique
javascript
Domaine
backend, performance

Piste de recherche

Start in util/rawCache.js, especially teeForCapture and collectBody, and review the stalled-reader test described in #229. First resolve the queue-wait versus capture-abandon and option-naming questions, then verify the stated done conditions: bounded memory, a 1.5 MB uncompressed 404 stored under a 512 KB cap, and a failing revert check without the queue bound.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

enhancement performance

The raw and negative caches' maxBytes counts a body as received. So an origin that sends error pages uncompressed needs a cap sized for the uncompressed body, though the copy that is stored is gzipped 7-13x smaller (#229). This issue is about making the cap bound what is held and stored, not what crossed the wire.

Problem

  • Why the cap counts received bytes. It is what bounds a capture's memory: about 2x maxBytes each.
    • One copy is the captured chunks.
    • The other is what tee() keeps for the crawler's unread branch, because the capture reads in a tight loop and runs ahead (teeForCapture / collectBody in util/rawCache.js; measured in #229).
  • What it costs, on one production origin (observed, 24h of prerender_ops negative_cache over 4 nodes):
    • Its retired-product 404s arrive uncompressed at 190 KB-1.5 MB.
    • At a 1 MB cap, 6.5k were refused as oversize, about 3.4% of the 404s stored. These are recorded counts; Harper's analytics record about 2/3 of real.
    • Each would have stored at roughly 80-210 KB (inferred from the 7-13x gzip ratio).
  • The workaround. The deployment raised render.negative.maxBytes to 2 MB. That raises the per-capture memory bound to about 4 MB, to admit bodies that store at about 150 KB. It is affordable there only because few captures run at once.

Why compressing the capture alone is not enough

#229 considered capping on compressed bytes and kept the received-bytes cap for this reason:

  • Gzipping the captured branch as it reads shrinks only the captured half.
  • The tee would still hold everything the crawler has not read yet, uncompressed.
  • With the cap on compressed bytes, a slow crawler could make one capture hold about 10x the cap in that other half.

Proposal

  1. Bound the crawler's queue separately from maxBytes. Put a high-water mark (say 256 KB) on the unread bytes held for the crawler's branch. ReadableStream.tee() has no such limit, so this needs a small custom tee. Past the mark, one of two things happens:
    • a. The capture waits for the crawler. This restores backpressure to the origin, but the capture slot is held for as long as the crawler takes to read.
    • b. The capture is abandoned, counted under a new outcome such as capture-slow.
  2. Compress the captured branch as it reads when the origin sent no encoding (zlib's streaming gzip, level 6, which runs on the thread pool). Count the cap on the compressed output. A body the origin already encoded is captured as it is today.
  3. Result: per-capture memory is about high-water + compressed bytes + deflate state (~256 KB at zlib defaults). It no longer depends on the uncompressed size, so the cap can be a stored-size cap.

Open questions

  • 1a or 1b? Slot occupancy suggests waiting is affordable: in the same 24h, raw capture-busy 4 against ~858k stored, negative 0. This is a hypothesis; the share of slow crawlers is unmeasured.
  • Keep maxBytes or add an option? Redefining maxBytes to mean compressed bytes would silently let an existing config admit bodies ~10x larger. A new option, for example maxStoredBytes, alongside a queue high-water option, is safer.
  • Can the store step be dropped? storableBody gzips once on the detached store path today. With compression at capture time it would only pass bodies through, but rows stored before the change still need the re-encode on the serve path.

Done when

  • A stalled-reader test, like the one in #229, shows memory per capture bounded independently of the body's uncompressed size.
  • A 1.5 MB uncompressed 404 is stored under a 512 KB cap. A deployment that raised the cap to admit such bodies can lower it again.
  • A revert check: with the queue bound removed, the stalled-reader test fails.

🤖 Generated with Claude Code

Langage dominant
JavaScript
Étoiles
0
Forks
0
Merge moyen
3 h 1 min
PR mergées (30 j)
70

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de HarperFast/prerender-plugin

Toutes les issues de HarperFast/prerender-plugin

Issues similaires

Plus d'issues JavaScript

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.