Plugin redesign: site logic in host-supplied code, knobs in config, Harper conventions where they fit

Wait for #240 and #241 to merge, then survey changeProbe.js, changeProbeSpec.js, configSchema.js, config.js, the resource modules, and the adjacent util and test files. Start by tracing the existing request source, probe scheduling, and resource loading through the named entry points. Done means a target architecture and phased plan covering the extension contracts, config, API, metrics, and invariants, with maintainer agreement before implementation.

Status: proposed 2026-10-02, not started. The goal is a plugin that could serve as the example of how a Harper plugin is built, so a full refactor and redesign is invited: module layout, config shape, tables, admin API and extension points may all change, within the invariants in Section 5. Start after #240 and #241 merge, because both change the probe and entity code this rewrites.

TL;DR

• The dividing line: config holds knobs (rates, schedules, switches, caps, thresholds). Code holds what is specific to a site: how to ask its origin about a page, and what a field means. A config value that is really a program moves into code the host app supplies. Examples are a URL template, a path language, a positional field map, and matching text in response bodies.
• Extension points are modules the host app supplies, following Harper's Resource contract where it fits. The plugin ships generic defaults that need no code.
• Use Harper's conventions where they fit, and record why where they don't. The main misfit is a sourcedFrom caching table for the plugin's scheduled refreshes (Section 3).
• Already designed: the first change. The change probe's request rule language becomes an entity source, a module the host app supplies (Section 3).
• Everything else is open. Survey the plugin, propose a target architecture and a phase plan on this issue, and get the maintainer's agreement before writing code (Sections 4 and 6).

1. What "exemplary" means: acceptance

1. A site whose pages carry schema.org markup runs on domains, routes and knobs alone, with no host code.
2. A site with an API, or with its catalog in Harper, supplies one small module per extension point instead of YAML.
3. A site operator can set every config option without reading the plugin's code. No option is a program.
4. Each module holds one concept. Today seven files are over 1,500 lines (Section 2).
5. The extension contracts, config, admin API and metrics are documented as the plugin's public API.
6. The bot path is no slower and allocates no more. This is measured before and after any change to it.
7. Each phase ships on its own and removes what it replaces in the same PR, with no shadow path.

2. Where the weight is today

Measured on the 0.101.0 branch with walkOptions from configSchema.js, wc -l and the schema:

What	Size
Config options	253: render 60, changeProbe 47, sitemap 29, ingress 16, management 14, queue 14, the rest under 12 each
util/changeProbe.js	4,084 lines
configSchema.js	3,529 lines
resources/PrerenderAdmin.js	2,353 lines, one resource for almost all of the management API
resources/Sitemap.js	1,984 lines
resources/RenderQueue.js	1,769 lines
util/documentFacts.js	1,631 lines
util/changeProbeSpec.js	1,527 lines
Source and tests	39.8k lines in src; 38.6k lines in 106 test files (103 suites)
Tables	23, across 12 databases

• Observed: on one production deployment, the host app's config.yaml is 1,659 lines. Most of it is comments recording the experiment behind each setting. That history is useful, but it also shows the settings carry meaning their names don't convey.
• Verified: no resource module loads without a running Harper. Target, Sitemap and PrerenderedPage subclass a table when imported, and RenderQueue, PrerenderAdmin and QueueState extend Harper's global Resource. So logic is moved out into util/ to be testable; the header of util/sitemapConditional.js says so for Sitemap.js.

3. Designed: the change probe's entity source

Today

A source: request rule is a small program written in YAML. The measured deployment's rule has:

• a pathPattern capture, and a urlTemplate with headers;
• 21 extract paths in the plugin's own path language, which has grown tuple projections (ITEMS[*].{id,stock,price.min});
• statusSignals that match text in 400 response bodies;
• a positional slot map. pageCheck.fields (slot, fact, comparator), priceFrom, availableFrom and ignoreChanges all refer to it by number.

Because the signature is positional, the plugin also carries rule fingerprints, prefix fingerprints and an upgrade for appended slots. They exist so that editing a rule does not read as a mass change (ruleFingerprint, prefixFingerprints and signatureUnderPrefix in changeProbeSpec.js). Judging by the module's layout, roughly half of changeProbeSpec.js serves the rule language. That is a reading of its structure, not a trial deletion.

Proposed

The host app supplies a source module, and the route or rule names it:

// <host app>/sources/products.js, loaded by path from config (not listed in jsResource)
export default class Products extends Resource {
	// id: the entity key (#166), e.g. https://www.example.com/products/123/
	static async get(id) {
		const productId = new URL(id).pathname.split('/')[2];
		const res = await fetch(`https://api.example.com/products/${productId}`, {
			headers: { accept: 'application/json' },
		});
		if (res.status === 404) return null; // no such entity
		if (!res.ok) throw Object.assign(new Error(`HTTP ${res.status}`), { statusCode: res.status });
		const p = await res.json();
		return {
			// keyed by PAGE_FACTS name (flat, dotted), not the renderer's nested pageFacts object
			'canonical': p.url,
			'title': p.seoTitle,
			'product.name': p.name,
			'h1': p.name,
			'product.offers': p.skus.map((s) => ({ sku: s.id, price: s.price, availability: s.availability })),
		};
	}
}

• Named facts replace slots.
  • The source returns values under the fact names the renderer already extracts (PAGE_FACTS in changeProbeSpec.js: canonical, title, metaDescription, h1, product.*, breadcrumbs). The plugin keeps one comparator per fact.
  • That removes the slot numbers, pageCheck.fields, priceFrom / availableFrom and ignoreChanges: a field that doesn't matter is simply not returned.
  • The fingerprints go too. A fact that appears is seeded by name, and one that disappears is no longer compared.
  • A source that changes what an existing fact means says so with a static version, and a new version re-baselines.
• null means no such entity, and a throw means the probe failed. This replaces statusSignals.
• The plugin keeps the hard parts:
  • the anchored pass, with its pacing, resume and canary;
  • the mapping guard and the comparators;
  • the Entity writes and canonical adoption (#166, #241);
  • the serve-time checks.
• Sites without code keep source: document. The plugin fetches the page itself and reads its JSON-LD Product offers. The request language is deleted, not kept alongside the module.
• A source can wrap a table. A host that already holds its catalog in Harper wraps that table, and the probe asks no origin at all.
  • Open: a source with subscribe() could push changes instead of waiting for the nightly pass.

How the plugin finds the module

Verified in the Harper 5.0.28 and 5.3.0 source. Not checked on 5.2.14, which the measured deployment runs.

• jsResource registers every export that has a get, put, post or delete as a resource (resources/jsResource.ts, handleApplication). A source exported from the host app's jsResource file would therefore become a REST endpoint that makes Harper call the origin. It must not be one.
• A plugin's scope.directory is the host app's folder: components/componentLoader.ts builds the scope with the component's directory, and components/Scope.ts returns it from get directory. scope.import() loads through the app's own module loader (Scope.ts, import).
• So config names a module path relative to the host app, and the plugin loads it with scope.import(join(scope.directory, path)).
• When the plugin calls the class's static get(id) itself, the override receives the raw id; Harper's transactional wrapper is not involved (resources/Resource.ts). Resource is a global inside a module loaded this way.

Why the plugin calls the source itself, not Entity.sourcedFrom(source)

These behaviours were observed in a local spike on Harper 5.3.0, on one node. The measured production deployment runs 5.2.14.

Behaviour of a sourcedFrom table	What it would break here
search() refetches expired rows from the source	A probe walk over Entity becomes an unpaced burst of origin requests
A nullish result from the source deletes the stored record. The spike returned undefined; 5.0.28 treats any falsy result the same (Table.ts, if (resolvedData))	A gone product loses the rest of its Entity row: the adoption memory and firstSeenAt
A refresh writes no audit entry. Per one reading of the harper-pro source, replication and change events read the audit log (untested on a cluster)	Other nodes keep reading the old canonical
During an outage, waiting readers retry the source one at a time	A burst of serve-time checks queues up behind it
A fill that takes over 10 s is force-unlocked and runs twice	Rules out rendered pages (Section 4)

sourcedFrom is built for refresh on read. The probe refreshes on a schedule, at a capped rate, behind a canary. So the plugin calls get() under its own pacer and writes its own rows. The module still follows Harper's Resource contract for an external data source. The host app can also make it the sourcedFrom source of a table of its own (Harper calls source.get(id, context)), where the behaviours above apply.

Still open in this part

• The request budget. Who owns the timeout, the abort when a pass is cancelled, and the origin bypass token? Today the plugin adds the token only when the endpoint shares the probed page's origin. Options:
  • pass the plugin's fetch and an AbortSignal as a second argument;
  • let the source use its own fetch while the plugin races it against a deadline.
• The value shape of each fact, above all product.offers. One list has to yield the price set, the per-SKU comparison and the price/availability claim (apiClaimOf).
• Where the entity key is defined. It could stay route config (entityPrefix, one regex on the bot path) or move into the source as keyOf(url). Either way it gets one home, not two.

4. Invited: the rest of the plugin

Survey before proposing anything. These are candidates, each with the question it raises. None is decided.

Area	Today	Question
Config and override layer	253 options; configSchema.js and config.js are 4.5k lines together. Observed: on the measured deployment, stored override rows, not config.yaml, are the live source of truth.	Which options are knobs and which are programs? Which groups can collapse?
Change probe	changeProbe.js (4,084 lines) holds pass scheduling and resume, probing, comparison, the canary and the mapping guard. probePacer.js, changeActions.js and serveCheck.js sit beside it.	Split by concept, around the source contract
Management API	One PrerenderAdmin resource dispatches almost every action; RenderQueue also takes pause and resume. The console proxies to both.	Resources per concern, shaped like REST
Sitemap	resources/Sitemap.js holds the resource, arrivals, departures and the refresh scheduler. Five util/sitemap*.js modules hold the decisions that need tests.	Split the same way. Its conditional fetch, on Last-Modified because one edge ignored If-None-Match, does what a sourcedFrom source's 304 revalidation does. Moving gains little, since what matters is the arrival and departure walk, not the cached XML.
Caches	Rendered pages; raw origin HTML (RawPage); 404s (NegativePage)	Rendered pages can't be sourcedFrom, since a bot never waits on a render. Raw HTML and 404s are read-through caches in shape. But the serve path streams the origin's answer and decides afterwards whether to store it, and the outage behaviour above would queue bots. They may belong in a general HTML cache layer, designed separately.
Testability	Resource modules can't load without Harper (Section 2)	Inject the tables, so a resource's logic is testable where it lives
Route classes, cache keys, visit filter	Regexes and per-route switches in config	Is this fine as declarative config, or is some of it site logic?
Open bugs	For example #218: the store mutex is not exclusive. Some open bugs predate the 0.93.0 queue rewrite and may be stale.	Confirm each is still current, then fix it or design it out

5. Invariants a redesign keeps

The residency, TLS and token lessons are recorded in CLAUDE.md, "Hard-won lessons". The others are recorded in the code's comments, or are this project's standing rules.

• Residency. A read of a key this node does not own can hang forever, so pass replicateFrom: false. A write does not forward and needs no deadline.
• TLS. Use https to every origin except localhost.
• The origin bypass token goes on every request to the page's own origin, and never to a third party.
• A bot never waits for a render. On any failure, the serve path falls back to the origin.
• A bad stored override fails open. The plugin runs its file config and logs a warning.
• Tables hold production data. One deployment has 1.69M Target rows (counted 2026-09-24, #166). A schema change needs a migration path, and a change to a @sealed table needs a worker restart.
• Override rows name today's config paths, and they outlive any one release. A renamed option needs its stored rows migrated, or an alias (aliasPaths exists). This is the one place a compatibility path is expected.
• The plugin, the render fleet and the console are released and deployed separately. A change to the queue protocol, the page facts or the management API needs a release order that never breaks a running fleet or console.
• The bot path runs at crawler volume. It makes no allocation it can avoid, and does no work at all for a feature that is off.
• Fix forward, with no transitional fallbacks. Each phase deletes what it replaces. The exceptions are stored override rows (above) and the release order between packages (below).
• This repo is public. No customer names, hostnames or IPs.

6. How to run it

1. Survey. Read the code, README.md, METRICS.md and CLAUDE.md. Measure the baselines the acceptance list needs: bot-path latency and allocation, and the probe pass rate.
2. Propose the target architecture and a phase plan as a comment on this issue. Once the maintainer agrees, rewrite this body to match it.
3. Ship in phases, as separate PRs in order. Each one ships on its own and removes what it replaces. Reserve each version number in its PR. A host app's change for a phase, such as its first source module, ships after that phase's release.
4. For each PR: tests that fail without the change (mutation-checked), docs, and review.

Related

• #244: the operator outcomes scorecard. What a site needs to see to know this works and what to change; the console half of "exemplary".

• #166, the entity table. Its phase 2 step "the probe walks Entity" goes through the source contract in Section 3.

• #241 (phase 1 of #166) and #240 (entity serve, #237) land before this starts.

🤖 Generated with Claude Code