Spec: topic research with local-model agents, wiki versioning and status

The spec derives topics from wiki dirs projects/, learning/, plans/ and names agents FactFinder, ThreadWeaver, Pleb, Scout. First read the existing agent pipeline and wiki storage code, then plan git commits per agent run and a Status page. Done means auto-curated topics, hourly SearXNG/HN search, versioned backups, and a deny-list UI.

Problem Statement

The knowledge base only knows what the author writes down. The projects, plans and learning topics in the wiki go stale: releases, announcements, discussions and articles relevant to them appear on the web, and the author has to go and find them by hand. There is no way for the wiki to keep itself informed about the subjects the author cares about, and no safe way to let outside content into a pipeline that today only ever processes the author's own notes.

There is also no safety net for the wiki itself: agents edit pages in place, with no history to inspect or revert, and no off-machine backup. Letting web content (and therefore possible prompt injection) into the pipeline makes that gap urgent. Nor is there anywhere in the app to see whether background machinery (backups, model server, scheduled work) is healthy; the Aspire dashboard is only reachable over an SSH tunnel.

Solution

A research capability that keeps wiki topics informed without the author maintaining a topic list:

• Topics are derived from the wiki. Every top-level entry under projects/, learning/ and plans/ is a research topic. When the ThreadWeaver changes a topic, a new local-model agent, the Pleb, decides whether the topic's search queries should change, and writes up to three queries plus a short, de-identified brief.
• Searching is plain code. An hourly job picks topics that are due, runs their queries against a self-hosted SearXNG instance and the Hacker News Algolia API, filters out already-seen and old results, and fetches the remaining pages as clean markdown through an SSRF-guarded fetcher.
• Triage is a second local-model agent, the Scout. It judges one fetched result at a time against the topic brief, keeping or discarding it. Kept results for a topic are combined into a single research note dropped into the normal notes inbox, so the existing FactFinder → ThreadWeaver pipeline files them, with citations.
• Freshness follows real value. If the ThreadWeaver actually wrote something from a research note, the topic is checked again in 24 hours; if not (everything was noise), the interval doubles, up to 30 days.
• Privacy is enforced in code. Names of people and organisations never go into search queries: a deny-list is built automatically from the author's own notes (FactFinder now classifies subjects), and the author can override any entry from the UI.
• The wiki is versioned and backed up. Every wiki-writing agent run ends with a git commit attributed to the run, pushed to a private backup repository with a deploy key.
• A Status page in the app shows whether backups, the model server, search sources and the research loop are healthy, with a header badge when anything is degraded.

Pleb and Scout run on the OpenAI-compatible provider (llama.cpp's llama-server with a small Qwen model); the existing agents keep their configured providers.

User Stories

Wiki versioning and backup

1. As the author, I want every wiki change made by an agent to be committed to git, so that I can see what changed and when.
2. As the author, I want each commit message to name the agent, the run and (for research) the topic and origin, so that I can tell why a page changed.
3. As the author, I want commits to happen even when an agent run fails, so that I can revert partial edits from a failed run.
4. As the author, I want commits to be taken under the wiki lock, so that a commit never captures a half-written page.
5. As the author, I want no empty commits when a run changed nothing, so that history stays meaningful.
6. As the author, I want an existing deployment to initialise the wiki repository on its next start, so that I don't have to migrate anything by hand.
7. As the author, I want only the wiki pages versioned (not notes, facts, runs or conversations), so that history isn't drowned in pipeline churn.
8. As the author, I want commits pushed to a private backup repository, so that a dead disk doesn't lose my wiki.
9. As the author, I want the push to authenticate with a per-repository SSH deploy key, so that there is no expiring token to renew and a leak exposes only the backup repository.
10. As the author, I want the host key of the backup remote pinned, so that the push can't be redirected.
11. As the author, I want a failed push to be logged and retried on the next commit without failing the agent run, so that a network blip doesn't break the pipeline.
12. As the author, I want backup to be optional, so that a clone without a backup remote still works.

Status

13. As the author, I want a Status page showing the last commit, last successful push and last backup error, so that I notice a silently failing backup.
14. As the author, I want the Status page to show whether the model server is reachable, so that I know why research has stalled.
15. As the author, I want the Status page to show whether each search source is reachable, so that I can tell a broken source from a quiet week.
16. As the author, I want the Status page to show how many topics are due, awaiting ingestion, backed off, muted or failing, and when the next cycle runs, so that I can see the loop is alive.
17. As the author, I want a badge in the app header when anything is degraded, so that I notice without visiting the Status page.
18. As the author, I want the Status page to update live, so that I don't have to refresh it.
19. As an operator, I want each status signal to also be a health check, so that /health and the Aspire dashboard agree with the Status page.

Provenance and the deny-list

20. As the author, I want the FactFinder to classify each subject as a person, an organisation or other, so that the system knows which subjects are private.
21. As the author, I want every fact document to record whether it came from my note or from research, so that research content is treated as third-party claims.
22. As the author, I want each research observation to carry the URLs it came from, so that the ThreadWeaver can cite them.
23. As the author, I want the ThreadWeaver to attribute and cite research-derived content in the wiki, so that a blog's claim never reads as something I know.
24. As the author, I want people and organisations from my own notes to be added to a deny-list automatically, so that their names never leave my machine in a search query.
25. As the author, I want subjects from research notes never to be auto-denied, so that public companies in articles don't silently degrade my queries.
26. As the author, I want the existing people in the wiki to be seeded into the deny-list once, so that people captured before this feature are protected too.
27. As the author, I want to see the deny-list in the UI with each entry's origin (automatic or manual) and state (denied or allowed), so that I know what is being filtered.
28. As the author, I want to toggle any entry between denied and allowed, so that I can allow a public organisation I do want to research.
29. As the author, I want my manual toggles to be permanent, so that automatic classification never reverses a decision I made.
30. As the author, I want a research-origin ThreadWeaver run to be unable to delete or move pages, so that injected instructions in web content can't destroy my wiki.
31. As the author, I want a research-origin ThreadWeaver run to be unable to write under me/ or people/, so that web content never ends up in personal sections.
32. As the author, I want those refusals to come back as explicit tool errors, so that the agent understands why and continues with what it may do.

Topics and the Pleb

33. As the author, I want topics derived from the top-level entries of projects/, learning/ and plans/, so that I never maintain a separate topic list.
34. As the author, I want a project folder and all its sub-pages to be one topic, so that reshuffling pages inside a project doesn't reset its research state.
35. As the author, I want a new topic to appear when the ThreadWeaver creates a new top-level entry, so that new interests are researched automatically.
36. As the author, I want a topic's state to be dropped when its page is deleted or moved, and the new location to start fresh, so that orphaned state doesn't accumulate.
37. As the author, I want the Pleb to revisit a topic's queries when the ThreadWeaver changes it, so that queries follow what the wiki knows.
38. As the author, I want Editor-only changes to update topic lifecycle but not trigger query curation, so that restructuring doesn't waste model time.
39. As the author, I want repeated changes to the same topic to collapse into one curation, so that a burst of notes on one project doesn't queue five model runs.
40. As the author, I want the Pleb to keep the current queries when nothing relevant changed, so that the staleness signal stays comparable between cycles.
41. As the author, I want at most three queries per topic, so that one topic can't exhaust the search and model budget.
42. As the author, I want queries rejected in code when they contain a denied name, so that privacy doesn't depend on a small model following instructions.
43. As the author, I want the rejection to tell the Pleb exactly what was wrong, so that it can correct the query and retry.
44. As the author, I want the Pleb to write a short brief of the topic without names or organisations, so that the web-facing Scout never sees my personal wiki content.
45. As the author, I want every existing topic curated once on first start, drained gradually, so that adopting the feature doesn't saturate the model server.
46. As the author, I want to see each topic in the UI with its brief, queries, state, interval and last useful result, so that I can judge whether the queries are any good.
47. As the author, I want to mute a topic from the UI, so that I can stop researching something without moving it in the wiki.
48. As the author, I want to trigger research for a topic immediately from the UI, so that I don't wait for its interval.

Searching and fetching

49. As the author, I want topics searched on an hourly cycle when they are due, so that the wiki stays current without manual effort.
50. As the author, I want results from a self-hosted SearXNG instance, so that general web search costs nothing and needs no API key.
51. As the author, I want results from Hacker News, so that discussions and launches in my areas show up.
52. As the author, I want results I've already seen for a topic skipped, so that the same article isn't triaged twice.
53. As the author, I want results older than the topic's last search skipped (or older than 30 days when undated), so that only news is considered.
54. As the author, I want manually blocked domains never fetched, so that content farms I've identified stay out.
55. As the author, I want fetched pages reduced to their main content as markdown, so that triage sees the article and not the navigation.
56. As the author, I want the fetcher to refuse loopback, private, link-local and metadata addresses, including after redirects, so that a malicious search result can't reach services on the VM.
57. As the author, I want fetches limited in size, time and content type, with an honest user agent and robots.txt respected, so that the fetcher is safe and polite.
58. As the author, I want a cap on topics and model runs per cycle, so that research shares the VM and model server fairly with other work.
59. As the author, I want the cycle skipped when the model server is unhealthy, so that an outage doesn't produce a pile of failed runs.
60. As the author, I want to run search and fetch with triage disabled, so that I can inspect raw results before the loop closes.

Triage and the Scout

61. As the author, I want each fetched result judged on its own against the topic brief, so that the prompt fits the model's small context.
62. As the author, I want noise (rants, beginner questions, SEO spam, restatements of what I know) discarded, so that it doesn't cost the pipeline anything.
63. As the author, I want the Scout to keep anything plausibly useful when unsure, so that the ThreadWeaver makes the final call on borderline items.
64. As the author, I want kept results combined into one research note per topic per cycle, so that the pipeline processes one note rather than many.
65. As the author, I want research notes to contain the Scout's summaries, short length-capped quotes and source URLs, never raw page text, so that injected instructions are not passed along verbatim.
66. As the author, I want a Scout run that ends without a verdict to count as a discard and be recorded as failed, so that a confused model doesn't stall the cycle.
67. As the author, I want Pleb and Scout runs to appear on the existing Runs page, so that I can inspect what they did.
68. As the author, I want prompts that exceed the model's budget to fail clearly, so that the model server never silently truncates input.

Freshness

69. As the author, I want a topic to be researched again after 24 hours when the ThreadWeaver filed something from its last note, so that active topics stay current.
70. As the author, I want the interval to double after each cycle that yields nothing filed (24h, 48h, 4d, 8d, 16d, capped at 30d), so that quiet topics cost little.
71. As the author, I want a cycle where the Scout discarded everything to count as empty immediately, so that noise doesn't wait on the pipeline.
72. As the author, I want a topic awaiting ingestion to be skipped by the scheduler, so that it isn't researched twice before its outcome is known.
73. As the author, I want a research note that fails in the pipeline to leave the topic's interval unchanged, so that a broken pipeline doesn't make topics look stale.
74. As the author, I want a topic whose notes fail three times in a row to show as failing on the Status page, so that I can investigate.
75. As the author, I want every research step to survive a restart, so that an interrupted cycle resumes where it was.
76. As the author, I want seen URLs to expire after 180 days and completed batches after 30 days, so that research state doesn't grow forever.

Implementation Decisions

Delivery order

The work ships in slices, each safe on its own:

1. Wiki versioning and backup.
2. Status page and header badge.
3. Provenance (subject kind, fact origin and sources), ThreadWeaver citation rules and run-scoped tool restrictions, deny-list with auto-deny and backfill, Deny-list UI.
4. Pleb, topic discovery and curation, Topics UI (with mute). Watch the queries for a week before anything searches.
5. SearXNG and Hacker News sources, the fetcher, the search cycle as a dry run with triage disabled.
6. Scout, note aggregation, outcome resolution and back-off: the loop closes.

Wiki versioning

• The wiki directory (only) is a git repository, initialised by the wiki structure startup job if missing.
• A versioning step runs when any wiki-writing agent run (ThreadWeaver, Editor) ends, whatever its status, under the existing wiki lock. It commits only if something changed.
• The commit message carries the agent, run id, origin and topic (when research) and the run's summary line.
• Git is reached through an IGitClient interface wrapping the git CLI; the API image installs git.
• An optional backup remote (SSH URL) is pushed to after each commit. Authentication is an ed25519 deploy key generated on the VM, mounted read-only into the API container, with a pinned known_hosts. The key never goes into GitHub secrets, CI or the repository. New AppHost parameters (backup remote, key path) follow the usual path: AppHost, CD workflow env:, setup script.
• A push failure is logged and reported on Status; it never fails the commit or the run.

Status

• A status endpoint under /api returns each signal (backup, model server, search sources, research loop) with detail; each signal is also an ASP.NET health check.
• Status changes are pushed over the existing server-sent event stream as a new area; the UI shows a Status page and a header badge when anything is degraded.

Pipeline contract changes

• Fact subjects gain a Kind: Person, Organisation or Other.
• Fact documents gain an Origin (Author or Research) and, for research, the topic. Observations gain Sources (URLs).
• pipeline.md describes these additions in generic terms (it is public). Personal wording stays in configuration.
• A ThreadWeaver run over research-origin facts has DeleteFile and MoveFile refused, and writes under me/ and people/ refused, by the tool layer for that run, with explicit errors. The current run's origin flows with the run (as the run tracker already does for the current run).
• ThreadWeaver instructions: research-origin facts are third-party claims, always attributed with a citation, never a reason to restructure or remove existing content.

Deny-list

• Entries: name, kind, source (Auto or Manual), state (Denied or Allowed). Automatic classification never overwrites a Manual entry; a UI toggle always writes Manual.
• Auto-deny comes only from Author-origin fact documents. A one-off backfill seeds entries from existing people/ folder names.
• Query validation rejects any query containing a denied name, case-insensitively, as a whole-word match.

Research feature and storage

• A new Research feature area in the API, following the existing conventions: command/query records with singleton handlers returning Result<T, Error>, sealed errors, source-generated logging, minimal API endpoints with OpenAPI metadata, extension-block registration.
• All research state lives under a new research/ directory in the wiki root (outside the versioned wiki directory), behind an IResearchStore interface:

research/
  topics/<slug>.json              brief, queries, interval, last searched, last positive, seen URLs, mute
  curation/requests/<slug>.json   Pleb queue; overwriting debounces
  curation/failed/
  batches/pending/<slug>.json     search output awaiting the Scout
  batches/awaiting/<slug>.json    note written, waiting on the ThreadWeaver
  batches/completed/ | failed/
  denylist.json, domains.json

• A topic's state is the folder its batch sits in, so restarts resume:

Due → Searching → Triaging (batches/pending) → AwaitingIngestion (batches/awaiting) → Idle(next due)

A pending batch records which results the Scout has already judged, and a resumed batch skips them. An awaiting batch is resolved from the run store when its note's ThreadWeaver run is found.

• Topic identity is its wiki path: projects/foo for a folder, learning/bar.md for a single page. The slug replaces / with --, and any slug supplied by the UI is resolved back and checked to stay inside the wiki.

Topic discovery and the Pleb

• Trigger: the end of a ThreadWeaver or Editor run. The run's recorded file tool calls give the changed paths, which map deterministically to topics. Index pages and other sections are ignored.
• Lifecycle is code: unknown topic → create state and request curation; deleted or moved topic → prune state; known topic changed by the ThreadWeaver → request curation. Editor-only changes do not request curation.
• On first start with no research state, curation is requested for every existing topic.
• The Pleb has no read tools. Code assembles its prompt: the topic's index page (truncated), the changed paths with their first headings, and the current brief and queries. Its tools are SetTopicQueries(topic, brief, queries) and KeepQueries(topic).
• SetTopicQueries validates in code: at most three queries, length limits, no denied names in queries or brief. Errors are explicit enough for a small model to correct the call.

Search, fetch and scheduling

• An hourly job selects topics where last searched + interval <= now, that are not muted, and have no batch in flight, most overdue first, up to a per-cycle topic cap.
• Sources implement ISearchSource: SearXNG (a container added in the AppHost and published to Compose, configured with engines that tolerate metasearch) and the Hacker News Algolia API.
• Filters, in order: manual domain block, seen URL (normalised), recency (newer than the topic's last search; undated results within 30 days). Seen URLs are recorded whatever the verdict.
• The fetcher uses HttpClient with Readability-style main-content extraction and HTML-to-markdown conversion, truncated to a configurable token budget. It resolves DNS and refuses loopback, private, link-local and metadata ranges, re-checking every redirect; it also enforces size, time and content-type limits, a descriptive user agent, robots.txt and per-domain politeness.
• When the model server's health endpoint is not OK, the Pleb and Scout phases skip the cycle.
• A single in-process gate (configurable concurrency, default 1) wraps every Pleb and Scout run, with the Scout taking priority over the Pleb. Per-cycle caps limit Scout runs and curations (low defaults, tuned from recorded run durations). Model client timeouts are generous (five minutes or more), and a timeout leaves the work queued.
• A setting disables the Scout, so the search cycle can run as a dry run.

The Scout and outcomes

• One Scout run per result. Code assembles its prompt: a fixed system prompt and tools first (byte-stable, for prefix caching), then the topic brief, then the fetched content inside clear delimiters, marked as data, not instructions.
• Its tools are KeepResult(summary, quotes) (quotes length-capped in code) and DiscardResult(reason). A run without a verdict counts as a discard and is recorded as failed.
• Kept results for a topic are combined in code into one research note, written to the new-notes folder with a research header (origin, topic, source URLs).
• Outcome: a cycle is positive when the ThreadWeaver run for that note made at least one CreateFile or PatchFile call (from the recorded tool calls). Positive resets the interval to 24h; empty doubles it, capped at 30 days. A cycle with every result discarded is empty immediately. A note that ends in the failed-notes folder leaves the interval unchanged and counts toward a three-strike failing flag.
• Seen URLs expire after 180 days, and completed batches are pruned after 30 days.

Agents and configuration

• Two new agents, Pleb and Scout, registered like the existing ones (known agents, startup registration, the AppHost agents list, provider defaults set to OpenAICompatible, instruction templates in config.template/). Neither gets the wiki rules. Both run through the run tracker.
• A model option disables thinking per request for the OpenAI-compatible provider (chat_template_kwargs.enable_thinking = false). That provider-specific code stays in the agent configuration.
• Target model: Qwen3.5-4B on llama-server with an 8k–16k context and a single slot. Prompt budgets are configuration, so a larger context only needs a settings change.
• New settings: cycle interval, per-cycle caps, model concurrency, triage input token budget, Scout enabled, SearXNG endpoint, backup remote, deploy key path.

UI

• New pages: Research › Topics (brief, queries, state, interval, last positive; mute and research-now actions), Research › Deny-list (toggle), and Status, with the header badge.
• Data through TanStack Query and the API client. Live updates come from new research and status areas on the event stream.

Testing Decisions

• Tests exercise external behaviour through the highest available seam: the research command and query handlers' Handle methods, asserting on the returned Result and on the resulting store state, not on internal calls.
• Ports are substituted with NSubstitute:
  • IResearchStore, so tests stay pure with no disk.
  • ISearchSource.
  • HttpMessageHandler for the fetcher, including redirects for the SSRF guard.
  • IGitClient.
  • AIAgent for the Pleb and Scout.
  • TimeProvider.
• Pure policies are tested directly, without seams:
  • Back-off schedule and due-topic selection.
  • Path-to-topic mapping and slugs.
  • Query and brief validation against the deny-list.
  • SSRF address classification.
  • Deny-list merge rules (Manual beats Auto).
  • Run-scoped tool restrictions.
  • Outcome classification from recorded tool calls.
• Modules under test: versioning (commit-on-run-end, push failure handling), deny-list, topic discovery and curation, search cycle, fetcher, triage aggregation, outcome resolution, tool restrictions.
• Style follows AGENTS.md: TUnit, Shouldly, NSubstitute; one test class per class under test, subject built in the constructor; scenario-named methods that arrange and act, then call ItShould... local functions that only assert; expected exceptions captured with Catch.Exception.
• Prior art: CreateFileHandlerTests (handler validation returning Result/ValidationError), WikiPathsTests (path containment, the model for slug and SSRF classification tests), WikiLockTests and NoteCatalogTests (pure policies with [Arguments]).
• Unit tests make no LLM, network or real-disk calls. A labelled fixture of 20–30 real search results (useful or noise) checks Scout agreement against a real llama-server in a separate integration test project that CI does not run. It is rerun after any model or prompt change.

Out of Scope

• The Brave Search API and any paid search source.
• A headless browser (Playwright) fetch tier.
• RSS feed registration, polling and topic inboxes.
• Domain reputation scoring and the Domains UI (the manual domain blocklist store exists, but its UI comes later).
• Per-app cloudflared tunnels and exposing the Aspire dashboard publicly.
• Tuning llama-server beyond documenting recommended flags.
• A wiki history or revert UI (the git CLI is enough for now).
• Push notifications for degraded status.
• Coordinating llama-server use with other applications on the VM.

Further Notes

• Recommended llama-server flags for a 4-core ARM (Neoverse-N1) VM: -c 16384 -np 1 -t 4 -fa on --cache-type-k q8_0 --cache-type-v q8_0 --jinja, keeping MTP speculative decoding. Drop the image flags and projector. Choose between a Q4_0 quant (ARM runtime repacking) and the MTP K-quant by running llama-bench.
• Verify on the target box before relying on them: prefix-cache reuse on Qwen3.5's hybrid architecture, and tool-call reliability with thinking disabled (20–30 sample calls per agent).
• Move to Qwen3.5-9B only if the Scout fixture shows poor agreement; it roughly halves throughput on this hardware.
• Search queries leave the machine, so the deny-list and the Pleb's de-identified brief are the privacy boundary. Treat any change to them as security-sensitive.
• A follow-up spec covers RSS feeds with topic inboxes (an inbox overrides back-off), domain reputation as a triage hint with reserved slots for unknown domains, and optional Brave and Playwright.