Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

writeSchedule lowers only the writing node's claim floor, but rows are residency-routed to their owner — ~75% of writes can strand rows silently

Aperta
  • Qualcosa non va?
#111 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
35/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
javascript
Ambito
backend, distributed-systems

Direzione di ricerca

Inizia da writeSchedule e segui lowerFloorFor attraverso leaseTable().lowerFloorTo(...) e il coordination.SharedBuffer locale. Poi leggi resources/Target.js, Sitemap.js e getResidencyByUrl per confrontare la gestione esistente per URL con la titolarità della residency. Done deve includere una direzione concordata, la copertura delle normali scritture nel plugin e la dimostrazione che le righe non rimangano al di sotto del floor del nodo proprietario.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Rewritten 2026-08-19. The first version of this issue blamed replication. That was wrong —
RenderSchedule is not replicated; rows are residency-routed and each key lives on exactly one
node (verified: a 27-key probe returned 1/16/10/0 across four nodes, perfectly disjoint). The
actual mechanism is below, and it is worse, because it fires from ordinary in-plugin writes.

Summary

writeSchedule lowers the claim floor of the node executing the write. But the row it just
wrote is residency-routed to whichever node owns that cacheKey. On a 4-node cluster ~75% of
writes therefore lower a floor belonging to the wrong node, and the owner's floor is never lowered
at all.

Any row filed with a due minute older than queue.claimFloor.guard (default 5 min) lands below
its owner's floor and is never claimed again — silently, permanently, from a fully funnel-routed
write.

The mechanism

export const writeSchedule = async (cacheKey, { nextRenderTime, fromSitemap } = {}) => {
	...
	await scheduleTable().put(cacheKey, { nextRenderTime, fromSitemap });
	lowerFloorFor(nextRenderTime);   // ← local SharedBuffer, i.e. THIS node's floor
};

lowerFloorFor → leaseTable().lowerFloorTo(...) is an Atomics CAS-min against the local
coordination.SharedBuffer (replicate: false, node-local by construction). The put, however,
is residency-routed — and per the module comment on writeSchedule, deliberately does not block
on the owner:

a write to a residency-pinned key this node does not own does NOT block on the owner (measured:
500 writes in 10.7 ms, mean 0.021 ms, against residency pinned to a node that does not exist)

Ownership is getResidencyByUrl → fnv1a32(\${url}|${node}`)` rendezvous hashing, so for any
given writer 3 of 4 keys are owned elsewhere.

This is already documented in the tree

resources/Target.js, in revalidate's phase 2 — the hazard is known and currently mitigated
only by writers remembering to recompute the minute per URL:

THE CURRENT MINUTE, PER URL — never captured once for the whole sweep. Phase 2 writes up to
scan.collectCap × devices rows with a PrerenderedPage.get per key, which at scale takes tens
of minutes. Rows are residency-routed, so ~75% land on nodes whose claim floor this process
cannot lower and which hold it at nowMinute − queue.claimFloor.guard: every row filed with a
minute more than the guard band old lands BELOW the owner's floor and is never claimed again —
silently, from a fully funnel-routed in-plugin write, and permanently where resetInterval: 0.
Sitemap.js already computes it per entry for the same reason.

So the invariant "never let more than guard elapse between computing a due minute and writing
it" is load-bearing across every schedule writer, enforced nowhere, and unobservable when broken.
Two call sites currently carry hand-written comments to remember it.

Impact

Observed on a 4-node production cluster (plugin 0.49.0 / harper-pro 5.2.3). One node accumulated
92,873 rows below its floor across 2,101 distinct due-minutes, oldest due 2026-08-01 — 18 days
of a growing slice of its shard silently not rendering, while the node reported healthy and kept
serving. Peer nodes were at 507 / 3 / 1 overdue.

Downstream, those keys held damaged blobs that could only be replaced by a re-render, so
replication base copies from the affected node latched indefinitely (harper-pro#699 territory) —
a queue bug surfacing as a storage symptom.

Why the nominated recovery does not cover it

queue.claimFloor.resetInterval's docstring names the out-of-plugin write paths it exists to
cover — the operations API and the exported REST surface. Residency-routed in-plugin writes
are not mentioned, and they are the high-volume path. The reset is also worker-0-gated inside
syncQueueState (see the open question in #110).

Possible directions

  1. Lower the owner's floor, not the writer's. The write already routes to the owner; the floor
    lowering needs to ride with it rather than being applied locally. That is the fix that makes
    the invariant unnecessary.
  2. Bound the forward advance so a floor can never outrun rows the pass never scanned (#110).
    That alone would stop the stranding even if the lowering stays local.
  3. Make the below-floor condition observable. The backlog snapshot already computes
    below_floor; a node with essentially its whole due backlog below the floor should alarm.
  4. Failing 1, at minimum assert the invariant at the funnel — reject or warn on a writeSchedule
    whose nextRenderTime minute is already older than guard, so a slow batch fails loudly
    instead of silently stranding rows on a peer.
Lingua principale
JavaScript
Stelle
0
Fork
0
Merge medio
9h 6m
PR unite (30g)
54

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di HarperFast/prerender-plugin

Tutte le issue di HarperFast/prerender-plugin

Issue simili

Altre issue su JavaScript

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.