changeProbe after the endpoint switch: anchored nightly pass + rule fingerprint shipped in #159; follow-ups: field-class lanes, sitemap-membership availability, probe identity

Leggi prima i follow-up e i criteri di accettazione rimanenti; esamina ProbeState, SitemapRefresh, config.yaml e l’entry point ingress.routes[].departureAction indicato nell’issue. Separa il lavoro sulla membership della sitemap non ancora mergiato dalle lane per field-class, identità del probe, canonical-slug e dalle attività della console. È completato quando i controlli di accettazione pertinenti hanno esito positivo senza regressioni nel comportamento del probe rilasciato.

Context

A customer deployment reported the change probe's request volume against their per-product price/availability JSON API (the same call the PDP's client JS makes on load) and asked for it to go away. Measured across the four nodes (2026-09-14): ~85% of what they saw was changeProbe (mode: continuous, 12h cycle, ~1.4M PDP URLs); ~15% was the PDP's own client JS during renders (#153). Their edge began throttling that path the same evening (probe failures 1-2% -> 33-52%, all connection timeouts and resets), so the probe was already half-blind.

The customer supplied the facts that decide the design:

1. Prices change only at their local midnight, never intraday; a refresh right after midnight is good for the day. Our data agrees (83% of canary-detected change events in the 23:00-02:59 local window; promo nights repriced 7.8% and 18.9% of products).
2. Availability changes intraday, and sold-out products leave the product sitemap the same day (57/58 sold-out products absent from that day's sitemap, 77/77 in-stock present). Sitemap membership is a daily availability signal; per-URL <lastmod> is not (every entry carries the build date).
3. They offered a better endpoint (2026-09-16): a per-product details API that is cached at the edge with the PDP's own lifetime (purged on known changes, expired at their midnight) and carries price, per-variant availability, canonical slug, promotions, badges, title/meta, images and breadcrumbs. Verified: 8/8 sitemap products and 46/46 variants agree with the PDP's JSON-LD on price and availability; two fetches 2.5 s apart differ in zero fields; sold-out is a 200 with a status word and no price object, not an error status; an unknown id is a 400 with a stable error code; the edge requires a Referer header (any value) or serves a cached 404 page. They also confirmed the page's own price call can be dropped for bot renders.

Resolved

The probe stays; it moved endpoints. With a cached, PDP-coherent endpoint the probe costs the origin one product-service read per product per pass and detects nearly every rich-result field, which removes the reason for the fleet-side "conditional render" this issue originally designed (see "Dropped" below).

• Live switch, 2026-09-16, via config-override rows: changeProbe.rules -> the details endpoint (GET, referer header, 20 extract slots: price core, status word, price set, all-variants flag, canonical slug, sale status, percentage off, promotion, badges, new/best-seller/top-rated flags, title, brand, meta title/description, main image, breadcrumbs), with dryRun: true and reprobeAfter: 0 for one full cycle because the signature shape changed for every URL and the plugin had no way to tell a rule edit from a content change (the canary would have read 100% changed and invalidated the route). The in-flight old-rule passes were cancelled with a momentary enabled: false so nothing kept polling the old path.
• #159 (prerender-v0.65.0) makes that procedure unnecessary and ships the generic pieces:
  • ProbeState.ruleFingerprint: a baseline records the rule that made it; a mismatch re-baselines (stored, not compared, not triggered, outside the canary verdict; rebaselined stat, probe_rebaselined metric). A rule edit costs one pass without detection for that rule and no dry-run cycle.
  • pageCheck reads availability words (In Stock, IN_STOCK, schema.org forms) and [*] per-variant lists, with per-rule availableValues / unavailableValues over the built-in vocabulary.
  • [*] projection in extract paths, so a rule can watch variants[*].availability without signing whole variant objects.
  • mode: anchored (anchorTime, anchorTimezone, anchorWindow): one full pass a day starting right after the origin's scheduled change, at the ceiling by default; no boot sweep; the canary keeps its cadence. This is the "nightly pass" of the original design, as configuration.
• Render-side price call (#153): an A/B render on the deployment's fleet config with the price API added to block.urlPatterns left JSON-LD offers (12/12), aggregate rating, price markup and error-string counts identical on desktop and mobile. That is a fleet configuration change, no code.

Deployment sequence for the customer: release 0.65.0 -> bump the component and port the rule into config.yaml (identical fingerprint to the override rule, verified) -> clear the override rows (changeProbe.rules, .dryRun, .reprobeAfter, .cycleTarget) once the re-baseline pass has finished on every node -> fleet config with the block pattern.

Dropped: conditional render in the fleet (former stages 2-3)

Pre-fetching the SSR document in the worker, extracting a signature, and seeding #157's document cache was designed to replace price polling with a fetch the render needed anyway. Against a cached endpoint that costs the origin ~nothing and answers in 15-35 KB, that machinery no longer buys origin relief. What it still would buy — ~6-15% render throughput from hiding the document fetch, and ~80% fewer blob writes from making cadence renders conditional — is a render-pipeline optimization that belongs with #155/#156/#157 on its own merits, not to this issue. Its measurements stay valid and are kept here for that work:

• Per-device SSR: desktop vs mobile product documents differ in 3 cosmetic hunks, Vary: Accept-Encoding only, so one document fetch per product is valid.
• Render is ~11.7 s wall / ~9.6 s Chrome CPU per variant (median 11.5 s, p95 16.3 s); origin SSR 465 ms mean, p95 1.26 s; PDP blob 610 KB raw / 53 KB gzipped, replicated x4; ~1.0M render attempts/day.
• Cookie handling (blocking for #157 on this deployment): the storefront buckets its PDP frontend by a cookie the edge sets on the document response, and the server honours the bucket for API routing; a pricing XHR without it is answered by a legacy backend with 400 for live products. #157's replayed variant strips Set-Cookie, so its scripts would run without the bucket and its sampleEvery check (SSR vs SSR) cannot see it. Rules: pin listed cookies on every variant (cookies.pin), never cross session/cart/visitor/bot-manager cookies, allowlist deterministic bucket cookies per job, compare structuredOffers between variants on sample jobs, and enable documentReuse on that deployment only after one PDP render shows the replayed variant matching.

Remaining follow-ups (this issue stays open for these)

1. Field-class lanes. Widening the signature to ratings would re-render a product per new review. Give extract slots a lane (immediate for price/availability/canonical, nightly for copy, images, ratings, breadcrumbs) so the anchored pass absorbs slow-moving classes and the canary/immediate path stays about price. Run the wider signature in dry-run first and read the changed-share split (probe_changed) before turning classes into renders.

2. Sitemap-membership availability — BUILT, awaiting merge (#164, prerender-v0.68.0). Per-route ingress.routes[].departureAction (none | expire | render): a PDP target that becomes unlinked has its pages HARD-expired (past page.swrTtl, so they stop serving rather than serving stale — a plain Date.now() expiry leaves the page swr and still serving, which is the trap here) and is filed to render at the current minute. Route-scoped as this item specified, so catalog pages are excluded: a product URL leaving a product sitemap says something about that product, while a listing URL leaving a paginated sitemap usually means the catalog was re-bucketed. Never deleted — the origin still serves a 200 with out-of-stock markup, so this is a re-check and the render's own verdict decides. Capped per walk (sitemap.departure.maxActions), and dryRun defaults TRUE because the count a real walk would act on is not yet known (see the measurement note below).

   The trap that shaped the design: the check must run AFTER the whole walk, not at prune time. The product sitemap is 17 children of exactly 50,000 entries ordered by ascending prd- id, so it shears across boundaries — a URL moving to an EARLIER child is re-attached before the child it left is pruned and never looks departed, but one moving to a LATER child is pruned first and looks departed until the child that now claims it is reached. Acting at prune time would fire on every URL after an insertion. Candidates are re-read once the walk ends; departure_reattached counts the shear, and a large share there means the raw removed count is not a departure count.

   Still unmeasured: how many PDPs actually depart per day. The SitemapRefresh row keeps only the last run, and the pass I read (18:00 ET: created 65 / updated 3 / removed 38 / skipped 886,471) is not the one that absorbs the midnight-CT rebuild — that is the 06:00 ET pass. That number sizes maxActions, which is why the feature ships in dry run.

3. Distinct probe identity. changeProbe.userAgent (or a header) so an origin's CDN can allowlist verification traffic separately from renders; today the probe rides origin.userAgents.desktop.

4. Canonical slug as a signal. With the canonical in the signature, a slug change can retarget without rendering (today it surfaces as a canonical-mismatch suppressed render, ~153k/day on this deployment).

5. Console. Surface rebaselined and nextAnchoredRunAt; alert on a steady probe_rebaselined (a rule that keeps changing).

Load (measured on the deployment)

• Probe keys are never shopper-warmed (the PDP's client does not call the details endpoint), so each probe is an edge miss and one origin product-service read: ~1.4M/day at the 24h cycle, the same count at one anchored pass a day, delivered at the ratePerSecond ceiling (10/s/node) for ~12 h rather than spread.
• Signature ~1.2-1.6 KB/URL (was ~40 B): ProbeState grows to ~600 MB per node, node-local, written only on change after the re-baseline.
• The flagged price API drops to the render-side XHR share (~0.6M/day) at the switch and to zero once the fleet block ships.

Acceptance

• Zero probe requests to the price API; probe_failed back at the 1-2% floor on the details endpoint (first dry-run canary: 500 probed, 0 failed).
• Anchored pass starts within a minute of the anchor and finishes inside the ceiling-rate window; nextAnchoredRunAt published.
• A rule edit produces one pass of probe_rebaselined and no canary trip.
• Pages of products that leave the sitemap re-render within 24 h of the refresh (follow-up 2). Gated on one dry-run walk reporting departure_would_render and departure_reattached before sitemap.departure.dryRun is turned off.

Notes / risks

• Status signals on the details endpoint: sold-out is a body state, not a status, so the "unavailable" signal is dormant there; the id-not-found signal is now trustworthy (the old endpoint also answered it for live products when the cohort cookie was missing).
• Analytics: sweep probe_* rows emit per finished pass (smeared); use canary rows or the Invalidation table for flip timing. probe_* are value metrics (mean x count).
• Related: #138 (compare origin against the rendered page), #153 (the page's own JS during renders), #80 (render priority lanes), #159 (the release above).

Versions: prerender-v0.65.0 = #159 (also claimed by #154; whichever merges second renumbers). The former stage-2 browser version (1.25.0) is released back to the #155-#157 line.