Answer a variant spelling of an entity URL from the canonical's cached render, instead of proxying or raw-caching each spelling

Start with the entity index proposed in #166 and trace the existing product-route miss path, probe confirmation, and bot_serve reporting. Review the snapshot post-processing area for the self-reference rewrite. Done means the route option and every listed fall-through are tested, the rewrite has a test, and entity serving is visible in bot_serve.

On one production origin, most misses on the product route are bots asking for another spelling of a product we already have rendered: /product/prd-<id>/<any slug>. The origin answers every such spelling with the same document, declaring the canonical. This issue proposes answering those misses from the canonical URL's cached render, instead of proxying them or storing a raw copy per spelling. It builds on the entity index proposed in #166.

Evidence

Everything below was measured on 2026-10-01 against one production origin. Product pages there were raw-cached for about 25 minutes as a test (rawCache + rawFacts on the product route).

• Most product misses are variant spellings (observed, from the raw rows' stored facts):
  • First read: 584 of 626 stored product documents (93%) sat at a spelling other than their own canonical, all of them the same product. Later: 1,869 of 1,984.
  • 0 named a different product.
  • Examples: /prd-N/email.jsp, /prd-N/product.jsp, capitalised or truncated slugs.
  • Canonical spellings rarely reach the raw cache because they are render targets, so they are cache hits.
• The origin serves one document per product (verified):
  • 70 products: the stored variant document (fetched through the normal miss path) against the origin's canonical document, both through documentFactsOf.
  • Identical canonical, title, description, name, brand, image, every offer (sku, price, currency, availability) and breadcrumbs for 68 of 70.
  • The 2 differences were the origin changing between fetches: a rename that added a SKU, and a description that flapped. Refetching every spelling back to back gave identical documents.
  • The CDN in front keeps one copy per product, ignoring both the slug and the query string: a random cache-busting query changed nothing.
  • The only place the spelling shows is an analytics pageUrl prop, and it carries whichever spelling filled the edge cache, so the canonical's own copy can name a variant.
  • <link rel=canonical>, og:url and JSON-LD url/@id always name the canonical.
  • Documents differ only in one per-response timing script line, which also differs between two fetches of the same URL.
• The canonical's render is almost always cached (measured): 199 of 200 sampled variant spellings, each a different product, had a fresh, indexable, 200 render of their canonical cached for both devices.
  • Render age: median 28h, p90 40h, max 48h, on a 96h render interval.
  • The one miss was a product whose canonical slug had just changed. No spelling of it had a servable page, so it would fall through.
• The choice is never ambiguous (measured): for all 199, exactly one spelling of the product has a servable render, and it is the canonical the variant document declares.
• The origin never 404s one spelling of a product it serves at another (measured over stored 404 bodies):
  • 450 product ids sampled from three nodes: 449 have no render at all for that id (retired or never-registered products).
  • One has an old render at the same URL that is no longer fresh, which the "fresh" guard below excludes.
  • None is a 404 spelling of a product with a live render. A 404 is per product, not per spelling.
• Every product spelling in real traffic is a .jsp path, and none redirects (measured):
  • 1,932 stored 200 variants: lowercase slugs 1,605, one-word slugs (email.jsp, product.jsp) 302, capitals 20, collection ids 5.
  • 16,000 stored 404s: all .jsp.
  • On the product route, bot_miss redirect was 0 of 361k misses in 24h across four nodes.
  • Empty slugs, missing slashes, extra segments and encoded characters never occur. Synthetic requests for those shapes cannot be tested from our side: the origin's bot mitigation answers a non-crawler 403.
• Collection pages behave the same (measured, n=5): each variant, often a member product's slug under the collection id, declared the collection's canonical. That canonical had a cached render on both devices, and the facts were identical.

Proposal

A per-route option, off by default (for example entityServe: 'canonical'), on routes that declare entityPrefix:

1. On a miss for a URL on such a route that has no Target of its own, resolve its entity id, then the entity's canonical target. This is the #166 index.
   • A spelling with a Target is the render path's: a new canonical arriving from the sitemap, a suppressed duplicate.
2. Serve that target's cached page for this device only if all of these hold:
   • it is fresh, indexable and 200, and its own canonical names itself;
   • that canonical was confirmed against the origin since the last anchor, by the probe's canonical slot or a serve-time check.
     Then serve it as source entity with its own cache headers, and count it under bot_serve.
3. Otherwise fall through to the existing miss path: proxy, or raw.

Why the confirmation guard. The origin re-spells slugs; one wave moved canonical-mismatch from 4% to 45% of render outcomes in 2.5h. Between a re-spell and the re-render of the old page, the old page still names its old slug as canonical, while the origin already declares the new one at every spelling. Serving the old page at more spellings would spread that contradiction. Confirmation since the anchor bounds it to what the page itself already serves.

Compared with raw-caching every spelling:

• It answers the first request for a spelling, not only repeats.
• It stores nothing, and therefore replicates nothing. Raw rows are replicated to every node, which is the expensive part on that cluster.
• It serves the rendered page rather than the unrendered origin document.

Freshness is the canonical's. Probe passes and serve-time checks already cover that key. A serve-time check on a page served this way should check the canonical's key, not the requested spelling.

A known difference to fix first

The snapshot is rendered at the canonical URL, so anything a script builds from location carries the canonical. Examined on two snapshots: every product URL in them is the canonical, and there are no share links. The exception is a third-party reviews widget, which builds two things from the page URL:

• SVG fills, fill="url(<page URL>#rating_star_filled)". Served at another URL, an absolute self-reference becomes an external reference, so the stars may render unfilled. That is a hypothesis, untested.
• Review pagination links, ?bvstate=pg:2/ct:r. These pointing at the canonical is harmless.

This already happens today: a route with queryParams: [] serves one snapshot to every query-string variant of the URL. Fix: in snapshot post-processing, rewrite url(<this page's own URL>#id) to url(#id). That is correct at any URL.

Open questions

• The entity lookup on the miss path. Is it one local point read, and does the #166 index exist by then? Without it, a lookup by id prefix over the target table is too slow for the serve path.
• The confirmation guard's reach. Probe passes cover every target, but a serve-time check covers only pages bots ask for. The pass reaches a page anywhere up to about 9h after the anchor, the length of the pass, so until then a page no bot has asked for is unconfirmed and falls through.
• Site-specific. The measurements are from one origin. Another site may serve different content per spelling, which is why the option is off by default.

Done when

• A route-level option serves a variant spelling from the canonical's cached render behind the guards above, and is tested for each fall-through: a Target exists, the page is stale or non-indexable, the canonical is unconfirmed since the anchor, or there is no render.
• The post-processing rewrite makes absolute self-references in url() fragment-only, with a test.
• bot_serve reports the new source, so its share of misses is visible.

🤖 Generated with Claude Code