Hacktoberfest 2026：維護者為十月標記出來的 issue，仍然開放、適合新手。 瀏覽 Hacktoberfest issue

Capture memory is bounded in uncompressed bytes: bound the crawler's queue and compress as the capture reads, so the cap means stored size

未關閉
  • 哪裡不對？
#235 0 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視

維護者通常 1 天內回覆

還沒有人認領這個 Issue。

評估

難度
5/5
預估耗時
一週以上
新手友好度
38/100
Issue 類型
功能
描述清晰度
基本清楚
活躍度
活躍
技術堆疊
javascript
領域
backend, performance

研究方向

Start in util/rawCache.js, especially teeForCapture and collectBody, and review the stalled-reader test described in #229. First resolve the queue-wait versus capture-abandon and option-naming questions, then verify the stated done conditions: bounded memory, a 1.5 MB uncompressed 404 stored under a 512 KB cap, and a failing revert check without the queue bound.

由索引模型根據 Issue 內容生成。

描述

enhancement performance

The raw and negative caches' maxBytes counts a body as received. So an origin that sends error pages uncompressed needs a cap sized for the uncompressed body, though the copy that is stored is gzipped 7-13x smaller (#229). This issue is about making the cap bound what is held and stored, not what crossed the wire.

Problem

  • Why the cap counts received bytes. It is what bounds a capture's memory: about 2x maxBytes each.
    • One copy is the captured chunks.
    • The other is what tee() keeps for the crawler's unread branch, because the capture reads in a tight loop and runs ahead (teeForCapture / collectBody in util/rawCache.js; measured in #229).
  • What it costs, on one production origin (observed, 24h of prerender_ops negative_cache over 4 nodes):
    • Its retired-product 404s arrive uncompressed at 190 KB-1.5 MB.
    • At a 1 MB cap, 6.5k were refused as oversize, about 3.4% of the 404s stored. These are recorded counts; Harper's analytics record about 2/3 of real.
    • Each would have stored at roughly 80-210 KB (inferred from the 7-13x gzip ratio).
  • The workaround. The deployment raised render.negative.maxBytes to 2 MB. That raises the per-capture memory bound to about 4 MB, to admit bodies that store at about 150 KB. It is affordable there only because few captures run at once.

Why compressing the capture alone is not enough

#229 considered capping on compressed bytes and kept the received-bytes cap for this reason:

  • Gzipping the captured branch as it reads shrinks only the captured half.
  • The tee would still hold everything the crawler has not read yet, uncompressed.
  • With the cap on compressed bytes, a slow crawler could make one capture hold about 10x the cap in that other half.

Proposal

  1. Bound the crawler's queue separately from maxBytes. Put a high-water mark (say 256 KB) on the unread bytes held for the crawler's branch. ReadableStream.tee() has no such limit, so this needs a small custom tee. Past the mark, one of two things happens:
    • a. The capture waits for the crawler. This restores backpressure to the origin, but the capture slot is held for as long as the crawler takes to read.
    • b. The capture is abandoned, counted under a new outcome such as capture-slow.
  2. Compress the captured branch as it reads when the origin sent no encoding (zlib's streaming gzip, level 6, which runs on the thread pool). Count the cap on the compressed output. A body the origin already encoded is captured as it is today.
  3. Result: per-capture memory is about high-water + compressed bytes + deflate state (~256 KB at zlib defaults). It no longer depends on the uncompressed size, so the cap can be a stored-size cap.

Open questions

  • 1a or 1b? Slot occupancy suggests waiting is affordable: in the same 24h, raw capture-busy 4 against ~858k stored, negative 0. This is a hypothesis; the share of slow crawlers is unmeasured.
  • Keep maxBytes or add an option? Redefining maxBytes to mean compressed bytes would silently let an existing config admit bodies ~10x larger. A new option, for example maxStoredBytes, alongside a queue high-water option, is safer.
  • Can the store step be dropped? storableBody gzips once on the detached store path today. With compression at capture time it would only pass bodies through, but rows stored before the change still need the re-encode on the serve path.

Done when

  • A stalled-reader test, like the one in #229, shows memory per capture bounded independently of the body's uncompressed size.
  • A 1.5 MB uncompressed 404 is stored under a 512 KB cap. A deployment that raised the cap to admit such bodies can lower it again.
  • A revert check: with the queue bound removed, the stalled-reader test fails.

🤖 Generated with Claude Code

主要語言
JavaScript
星號
0
分支
0
平均合併
4 小時 29 分鐘
30 天內合併 PR
40

環境準備

  • 沒有 Dockerfile 或 Docker Compose 檔案
  • 沒有 Pull Request 範本
  • 閱讀貢獻指南

從這裡開始

  1. 先讀完整個 Issue，再讀專案的貢獻指南。
  2. 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
  3. Fork 儲存庫，在一個分支上完成修改。
  4. 送出 Pull Request，並在描述裡引用這個 Issue 編號。

HarperFast/prerender-plugin 的其他 Issue

查看 HarperFast/prerender-plugin 的全部 Issue

相似的 Issue

更多 JavaScript Issue

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。