Hacktoberfest 2026: as issues que os mantenedores marcaram para outubro, abertas e boas para iniciantes. Ver issues do Hacktoberfest

Capture memory is bounded in uncompressed bytes: bound the crawler's queue and compress as the capture reads, so the cap means stored size

Aberta
  • Algo errado?
#235 0 comentários 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 1 dia

Ninguém assumiu esta issue ainda.

Avaliação

Dificuldade
5/5
Tempo estimado
Mais de uma semana
Facilidade para iniciantes
38/100
Tipo de issue
Funcionalidade
Clareza
Razoavelmente clara
Status de atividade
Ativa
Stack de tecnologia
javascript
Domínio
backend, performance

Direção de pesquisa

Start in util/rawCache.js, especially teeForCapture and collectBody, and review the stalled-reader test described in #229. First resolve the queue-wait versus capture-abandon and option-naming questions, then verify the stated done conditions: bounded memory, a 1.5 MB uncompressed 404 stored under a 512 KB cap, and a failing revert check without the queue bound.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

enhancement performance

The raw and negative caches' maxBytes counts a body as received. So an origin that sends error pages uncompressed needs a cap sized for the uncompressed body, though the copy that is stored is gzipped 7-13x smaller (#229). This issue is about making the cap bound what is held and stored, not what crossed the wire.

Problem

  • Why the cap counts received bytes. It is what bounds a capture's memory: about 2x maxBytes each.
    • One copy is the captured chunks.
    • The other is what tee() keeps for the crawler's unread branch, because the capture reads in a tight loop and runs ahead (teeForCapture / collectBody in util/rawCache.js; measured in #229).
  • What it costs, on one production origin (observed, 24h of prerender_ops negative_cache over 4 nodes):
    • Its retired-product 404s arrive uncompressed at 190 KB-1.5 MB.
    • At a 1 MB cap, 6.5k were refused as oversize, about 3.4% of the 404s stored. These are recorded counts; Harper's analytics record about 2/3 of real.
    • Each would have stored at roughly 80-210 KB (inferred from the 7-13x gzip ratio).
  • The workaround. The deployment raised render.negative.maxBytes to 2 MB. That raises the per-capture memory bound to about 4 MB, to admit bodies that store at about 150 KB. It is affordable there only because few captures run at once.

Why compressing the capture alone is not enough

#229 considered capping on compressed bytes and kept the received-bytes cap for this reason:

  • Gzipping the captured branch as it reads shrinks only the captured half.
  • The tee would still hold everything the crawler has not read yet, uncompressed.
  • With the cap on compressed bytes, a slow crawler could make one capture hold about 10x the cap in that other half.

Proposal

  1. Bound the crawler's queue separately from maxBytes. Put a high-water mark (say 256 KB) on the unread bytes held for the crawler's branch. ReadableStream.tee() has no such limit, so this needs a small custom tee. Past the mark, one of two things happens:
    • a. The capture waits for the crawler. This restores backpressure to the origin, but the capture slot is held for as long as the crawler takes to read.
    • b. The capture is abandoned, counted under a new outcome such as capture-slow.
  2. Compress the captured branch as it reads when the origin sent no encoding (zlib's streaming gzip, level 6, which runs on the thread pool). Count the cap on the compressed output. A body the origin already encoded is captured as it is today.
  3. Result: per-capture memory is about high-water + compressed bytes + deflate state (~256 KB at zlib defaults). It no longer depends on the uncompressed size, so the cap can be a stored-size cap.

Open questions

  • 1a or 1b? Slot occupancy suggests waiting is affordable: in the same 24h, raw capture-busy 4 against ~858k stored, negative 0. This is a hypothesis; the share of slow crawlers is unmeasured.
  • Keep maxBytes or add an option? Redefining maxBytes to mean compressed bytes would silently let an existing config admit bodies ~10x larger. A new option, for example maxStoredBytes, alongside a queue high-water option, is safer.
  • Can the store step be dropped? storableBody gzips once on the detached store path today. With compression at capture time it would only pass bodies through, but rows stored before the change still need the re-encode on the serve path.

Done when

  • A stalled-reader test, like the one in #229, shows memory per capture bounded independently of the body's uncompressed size.
  • A 1.5 MB uncompressed 404 is stored under a 512 KB cap. A deployment that raised the cap to admit such bodies can lower it again.
  • A revert check: with the queue bound removed, the stalled-reader test fails.

🤖 Generated with Claude Code

Linguagem predominante
JavaScript
Estrelas
0
Forks
0
Merge médio
3h 1min
PRs com merge (30d)
70

Preparar o ambiente

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de HarperFast/prerender-plugin

Todas as issues de HarperFast/prerender-plugin

Issues semelhantes

Mais issues de JavaScript

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.