🎃 Generate llms.txt and llms-full.txt from the repo docs so AI agents can read them

Start with scripts/validate-adapters.test.mjs, .github/workflows/ci.yml:48-75, and scripts/sync-guides-to-website.sh:57,81; inspect docs/, content/guides/en/, and the sample guide’s front matter. Run the existing script tests with node --test and review the catalog CI check. Done means deterministic generated files, passing generator tests and --check in CI, website sync support, and the requested docs update.

Part of the October Challenge #846.

Why

People increasingly ask Claude, Cursor or ChatGPT "how do I connect weclapp to AnythingMCP?" and the agent fetches our docs. HTML pages with navigation and JS are a poor input; the llms.txt convention gives agents a short index (/llms.txt) and a single plain-Markdown dump (/llms-full.txt). Mintlify, Cloudflare, Anthropic and Qonto (https://docs.qonto.com/llms.txt) already publish one. We have the source Markdown in this repo; we just never assemble it.

Self-hosted vs Cloud

• Self-hosted: not affected at runtime. This is a build-time script that produces two text files from the repo; nothing ships in the Docker image.
• Cloud (DEPLOYMENT_MODE=cloud): not affected. The files describe public docs only; they contain no tenant data.
• Who can use it: anyone (public files). Publishing on anythingmcp.com is done by the website team: the website lives in a separate repository and pulls content from here with scripts/sync-guides-to-website.sh.

Decision: do not serve /llms.txt from the app (frontend/backend). The app on cloud.anythingmcp.com or a self-hosted instance is a product, not the documentation site, and an agent connected to it already gets workspace instructions over MCP. The canonical location is https://anythingmcp.com/llms.txt.

What to build

1. scripts/generate-llms-txt.mjs (plain Node ≥22, no new dependencies), writing into content/llms/:
   • llms.txt: follows llmstxt.org. H1 # AnythingMCP, a one-paragraph > summary (reuse the README tagline), then H2 sections with - [Title](url): one-line description links:
     • Docs: every docs/**/*.md except pages about running our Cloud: docs/operations/signup-attribution.md, docs/operations/slo.md, plus docs/ADOPTERS.md and docs/AUTHORS.md. Keep backup-restore.md, disaster-recovery.md and observability.md, which self-hosters need. Keep the exclusion list in one constant at the top of the script. Link to the GitHub blob URL (https://github.com/HelpCode-ai/anythingmcp/blob/main/docs/...); the title is the file's first # heading.
     • Adapter guides: one line per content/guides/en/*.mdx, title and description from the front matter, linked as https://anythingmcp.com/guides/<file-without-extension>. Ask in the PR if the website URL pattern differs.
     • Optional: the de/it guides (agents may skip these).
   • llms-full.txt: the full Markdown of the "Docs" section plus the English guides. For each file: strip MDX front matter, write # <title> and Source: <url>, then the body. Rewrite relative links (../README.md, connectors/rest.md) to absolute GitHub URLs so they still resolve.
   • content/llms/guides/<locale>/<slug>.md: the plain-Markdown version of each .mdx guide (front matter turned into a title and description header), so the website can serve /guides/<slug>.md next to the HTML page.
2. A --check flag that exits 1 when the committed output is stale. Same pattern as the catalog check at .github/workflows/ci.yml:67.
3. A CI step in the backend job of .github/workflows/ci.yml, next to Validate adapter catalog (line 48): node scripts/generate-llms-txt.mjs --check.
4. Extend scripts/sync-guides-to-website.sh to also copy content/llms/llms.txt, content/llms/llms-full.txt and content/llms/guides/** into the website (target directory set by a new WEBSITE_PUBLIC_DIR env var; skip with a message when it is unset). The website team wires the route.
5. Unit tests with node --test in scripts/generate-llms-txt.test.mjs (see scripts/validate-adapters.test.mjs): front-matter stripping, relative link rewriting, deterministic order (sort file names), and the excluded paths.

Where to look

• docs/ — the public Markdown docs (connectors, integrations, tool-definition, api-reference, sso, deployment…).
• docs/guides.md:5 — says per-adapter guides are published at anythingmcp.com/guides.
• content/guides/{en,de,it}/*.mdx — adapter guides with YAML front matter (title, description, adapterSlug); see content/guides/en/billomat-to-mcp.mdx.
• scripts/sync-guides-to-website.sh:57 — loop that copies the .mdx guides into the website repo; :81 copies assets. This is where step 4 goes.
• .github/workflows/ci.yml:48-75 — existing "generate then fail if diff" checks to copy.
• scripts/validate-adapters.test.mjs — node --test style used for scripts.

Acceptance criteria

• node scripts/generate-llms-txt.mjs produces content/llms/llms.txt, llms-full.txt and guides/<locale>/*.md; running it twice gives byte-identical output.
• llms.txt follows llmstxt.org (H1, blockquote, H2 link lists) and every link is absolute.
• The excluded Cloud-internal pages and anything resembling a .env file stay out of the output (a test asserts this).
• --check fails CI when a doc changes without regeneration.
• node --test scripts/generate-llms-txt.test.mjs passes and runs in CI.
• Docs updated: a short "For AI agents" paragraph in docs/guides.md pointing to https://anythingmcp.com/llms.txt, and a comment in sync-guides-to-website.sh explaining the new files.

Out of scope

• Any route in packages/frontend or packages/backend serving llms.txt (see the decision above).
• Changes to the website repository.
• Translating docs, or adding the generated adapter catalog (≈260 adapters) to llms-full.txt. A follow-up can add it from catalog.ts.

Size

M (1–2 days)

How to claim

Comment "I'd like to work on this" and we'll assign you. Rules in #846.