docs: rework documentation from scratch - structure and content

Start by reading the retained audits and the docs tree, then inspect api_server.py, tools/generate_api_docs.py, and generate-mdx-docs.ts alongside the publishing and webui MDX sync paths. Decide the documentation structure, audiences, and cross-repository sources of truth before planning the rewrite. Done means the reorganized documentation is accurate and maintainable, with reference material and inter-document links working through the publishing pipeline.

Rescoped. This began as an audit producing a list of fixes. The intent now is to review and
rewrite the documentation completely - structure as well as content - rather than repair it
incrementally. The audit findings below are retained as input to that rewrite, not as a task
list.

Why a rewrite rather than fixes

The documentation has drifted in a way that patching does not address. Two audits have now been
run against it, roughly three months apart, and the pattern is consistent: individually wrong
statements get corrected, while structural incompleteness grows with the codebase.

From the original audit, the "incorrect and misleading" findings have largely been fixed:

• The glossary's definition of ARIA is now correct.
• References to tox -e docs, simulate_msg and simulate-messages.sh are gone.
• The stale baseline migration ID and "three migrations" claim are gone.
• The smartem-decisions README no longer calls the system a proof of concept.

The completeness gaps, however, have widened rather than closed:

Document	At audit time	Now
backend/api-documentation.md	~8 endpoints documented, 60+ existed	~17 documented, 89 routes in api_server.py
backend/database.md	~5 tables implied, 22 existed	~17 entries, 26 tables (table=True)
agent/cli-reference.md	5 entry points missing	still none of them mentioned
operations/environment-variables.md	CORS_ALLOWED_ORIGINS, SMARTEM_BACKEND_CONFIG, appconfig.yml missing	still none of them mentioned

Documentation was last substantively touched on 2026-07-11.

This is the signature of hand-maintained reference material tracking a moving codebase. Another
round of corrections would produce the same result again.

Prior audit findings, for reuse

Two audits exist and neither needs re-running.

April 2026 (this issue, above) - a targeted audit against smartem-decisions,
smartem-frontend and smartem-devtools, grouped into incorrect / severely incomplete / stale.
Its structural conclusion still holds: the documentation is well organised, and the ADRs are
genuinely useful; it is the reference material that fails.

July 2026 - systematic drift verification. The docs/ tree was decomposed into
approximately 2,560 content atoms, of which the 1,506 how-to, reference and tutorial atoms were
resolved into 960 checkable claims and verified individually against the repositories. Results:

• 79% true and reusable
• 12% misleading, overwhelmingly from repository-split path drift - scripts and tooling that
  moved between smartem-decisions and smartem-devtools while the docs kept the old locations
• 6% incorrect, including a set of Athena documents describing a package deleted under
  ADR-0015
• 2% unverifiable

The tooling and per-claim data are retained locally and backed up.

The headline number matters for the rewrite: roughly four fifths of existing content is
sound, so this is a restructure-and-regenerate exercise, not a blank page. The single largest
category of error is path drift from the repository split, which is mechanical and can be fixed
in bulk.

Questions the rework should settle

Structure. Whether to keep the current organisation, and whether the how-to / reference /
tutorial / explanation split is serving readers or merely categorising. This is the decision
that determines everything else, and it should be made explicitly rather than inherited.

What is generated versus written. Reference material that mirrors code - endpoints, tables,
environment variables, CLI entry points - is exactly what has decayed twice. It should be
generated from the source of truth. Note that tools/generate_api_docs.py already exists in
smartem-decisions; it appears in the original audit only as an undocumented tool, which is a
fair summary of the problem.

Audience. The documentation currently serves developers on the project, prospective
contributors, and operators, without distinguishing between them. A rewrite is the moment to
decide whether that is one document set or several.

Sources of truth for cross-repo material. The largest error category came from content
describing files in another repository. The rewrite should decide where cross-repo material
lives so the split cannot silently invalidate it again.

Rendering-layer work, currently deprioritised

Two issues covering how documentation is rendered in the webui have been closed as not current
priority, and are noted here so the rework does not rediscover them from scratch:

• #219 - render a "View source on GitHub" link on each MDX-rendered page. An enhancement.
  Its one durable design point: derive the base URL from a single constant, so a change of
  source-of-truth host does not break every page at once.
• #220 - rewrite relative .md links and repo-absolute GitHub URLs to SPA routes. This one
  is a live defect: generate-mdx-docs.ts renames .md to .mdx but never rewrites link
  targets inside content, so links between documents 404 in the rendered webui today.

Any documentation structure will contain links between documents, so #220's problem will recur
under this rework regardless of how the content is reorganised. It should be solved as part of
whatever rendering pipeline the rework settles on.

Also relevant: the documentation is published automatically, so any restructuring must keep the
publishing pipeline and the webui's MDX sync working. That constrains how freely directories can
be rearranged.