Unify AWF diagnosis knowledge, discovery, and maintenance

Problem

AWF diagnostic knowledge is useful but fragmented: runner failure modes live in .github/workflows/shared/self-hosted-failure-modes.md and are copied into .github/agents/self-hosted-runner-doctor.md; auth support facts live in docs/auth-matrix.md while auth-doctor-updater.md edits documentation rather than diagnosing an incident; network tooling lives in .claude/skills/debug-firewall and awf-debug-tools, CI diagnosis in ci-doctor.md and debugging-workflows, and broad troubleshooting in docs/troubleshooting.md. Agents must already know which surface to find. Runner updater proposes issues requiring manual edits to multiple copies; auth updater makes separate documentation PRs. There is no shared contract for recording, updating, or consuming a diagnosis finding.

Goal

An agent given an AWF error, run URL, or symptom can discover one entry point, identify the failing boundary, retrieve version-appropriate, evidence-backed findings, run the smallest safe probe, and report a cause/fix or uncertainty. Canonical diagnosis state must be in one obvious, versioned location in this repository, and every producer must update it through the same documented primitives and review process. No credential values or private incident logs belong in that state.

Proposed architecture

1. One canonical, searchable diagnosis registry

Create docs/diagnostics/ as the public home, with README.md as the human/agent index, schema.json (or another repository-standard validated schema), patterns.md documenting the update and consumption protocol, and findings/ containing small machine-readable records grouped by boundary (runner, runtime, network, auth, ci, security). The exact serialization can be decided during implementation; it must support deterministic validation and rendering, not require an agent to parse a giant free-form prompt. README.md maps symptoms/boundaries to the relevant records and detailed existing docs.

Each record has a stable ID (retain the existing A/B/C/D runner IDs), concise symptom signatures, discriminating conditions, affected runtime/runner/provider/auth mode, introduced/fixed versions or commits when known, status (fixed, workaround, unresolved, needs-evidence), root cause, smallest read-only and secret-safe probe, recommended action, provenance links (issue/merged PR and implementation/tests), and owner/review date. Use explicit unknown/unset for unverified version ranges; do not assert a shipped fix based only on an open PR or provider documentation. Protect against collisions and ambiguous matching; findings can cross-link rather than duplicate a cause across categories. The registry is the canonical diagnosis knowledge state; per-run logs, cache-memory, and incident reports are evidence or working memory, not competing sources of truth.

Keep docs/auth-matrix.md authoritative for supported auth combinations and docs/compatibility.md for supported platforms; findings link to those references and describe failures/probes rather than copy entire matrices. Code/tests on the default branch remain authoritative for actual AWF behavior.

2. One discoverable agent entry point

Add .github/skills/diagnose-awf/SKILL.md with an explicit description covering auth, ARC/DinD and self-hosted runners, alternative runtimes, Squid/DNS/egress, CI, and security regressions. Expose it to the existing .claude/skills layout as appropriate. The short entry point reads docs/diagnostics/README.md, fingerprints AWF version and topology, classifies the failed phase/trust boundary, loads only matching records and specialist references, and returns a consistent report: observed symptom, matched finding ID and evidence, affected version/topology, safe next probe, fix/workaround, and citations. When no record matches, say so and collect the smallest missing evidence; never invent a fix. For use without a clone, maintain a portable agent artifact generated from the canonical registry, with a published version/tag recommendation; it is an output, not a second editable catalog.

Auth routing must distinguish (a) AWF API-proxy sidecar/provider token exchange, (b) GitHub/Copilot enterprise and BYOK routing, and (c) gh-aw-launched mcpg HTTP MCP GitHub OIDC. Check configuration presence/shape, route and health status, and redacted error classes, not keys, JWTs, authorization headers, environment dumps, inference probes, or token exchanges. Security regressions should route to the security-review process instead of recommending --env-all, disabling isolation, or broadening an allowlist by default. Reuse debug-firewall, awf-debug-tools, debugging-workflows, runner catalog, and existing docs as specialist references instead of copying their commands wholesale.

3. Shared diagnosis primitives and update pattern

Implement a small shared tooling layer (for example scripts/diagnostics/) with documented commands/APIs for validate, index/search, render, and check-sync. validate enforces schema, unique/stable IDs, required provenance, safe probes, status/version consistency, valid references, and no credential-bearing sample values. index/search supports bounded lookup by signature and dimensions; render deterministically produces the gh-aw runtime-importable Markdown catalog, portable agent knowledge section, and navigation/index; check-sync fails CI if generated outputs differ from canonical state. Preserve authored playbooks separately from generated findings so no generator overwrites human instructions. Prefer existing repository build/test tooling; generated .lock.yml files continue to come from gh aw compile rather than hand edits.

Document the common observe → classify → verify → propose → review → publish protocol in docs/diagnostics/patterns.md:

• Observation: issue/run/PR/provider-doc change is a candidate, not a diagnosis. Strip secrets and record links plus minimal redacted evidence.
• Classification: match an existing ID or allocate a new stable ID in the correct namespace; distinguish root cause from symptom and mark unresolved work honestly.
• Verification: confirm current-main implementation/tests and fix status, version scope, a safe discriminating probe, and primary citations; check auth support against the matrix/provider guidance without treating guidance alone as shipped support.
• Proposal: use a shared issue template/structured payload or a bounded docs PR containing exact canonical-record changes. Deduplicate by finding ID plus source citation; overlapping scan windows must not create duplicate findings.
• Review/publish: a human-reviewed PR changes the canonical record and any necessary support docs together. CI validates schema, generated parity, links/reference shape, safety constraints, and representative routing fixtures before publishing. Never let an issue, cache-memory, or an updater's unreviewed text silently alter runtime diagnosis knowledge.
• Maintenance: assign an owner and review cadence per boundary; supersede/deprecate findings in place without recycling IDs. Document how to retract a wrong finding and regenerate consumers.

4. Bring agentic workflows under the same contract

Create a reusable gh-aw shared diagnosis-maintenance component (within .github/workflows/shared/) that defines scan-window handling, candidate extraction, evidence/provenance requirements, matching/deduplication, proposal format, and noop behavior; put deterministic validation/rendering in the shared tooling, not copied prompt text. Make the Runner Doctor Updater and Auth Doctor Updater use those primitives and propose canonical registry changes. Choose and document a single proposal path: preferably bounded, reviewed PRs for verified documentation/registry edits, with issues for uncertain findings requiring investigation. Keep domain-specific source queries and official-doc allowlists in thin runner/auth adapters. Extend or add network/runtime/CI discovery only after the common path works. CI Doctor remains an incident investigator and may link existing finding IDs or propose candidates; it must not become an unreviewed writer. Security Guard/Review and dependency monitoring retain their distinct review scopes and feed verified diagnosis candidates where appropriate.

The Self-Hosted Runner Doctor workflow should import generated diagnosis content through gh-aw's runtime import and keep its read-only playbook. The portable agent should be regenerated from the same source. Recompile affected .github/workflows/*.lock.yml with gh aw compile; never edit lock files by hand. Document permissions, safe outputs, triggers, dedupe keys, and failure/noop behavior for the maintenance workflows.

Implementation sequence

1. Define schema, namespace/status/version rules, patterns document, and discovery index; add tooling and fixtures before migrating data.
2. Import the existing runner A/B/C/D catalog and lookup table without renumbering IDs; preserve citations, unresolved items, and version caveats. Generate the workflow import and portable copy; add parity CI and remove manual-copy instructions.
3. Add focused auth findings and the secret-safe auth diagnostic path, cross-linking the existing auth matrix, architecture, and API-proxy docs. Do not duplicate the support matrix.
4. Add the diagnose-awf skill and routes for runner/runtime, network, auth, CI, and security; link it from README, docs/diagnosing-awf-failures.md, troubleshooting, and relevant existing skills.
5. Refactor runner/auth updater workflows to the common proposal contract and validate/recompile their lock files. Integrate CI Doctor as a read-only consumer. Expand candidate sources later rather than building multiple uncoordinated updaters at once.
6. Add CI checks for registry schema, source/generated parity, secret-safe probes, expected routing, and compiled workflow contracts; remove obsolete duplicated diagnosis material only after migration passes.

Acceptance criteria

• An agent can discover diagnose-awf from its skill description and docs/diagnostics/README.md without knowing a workflow name.
• One canonical edit updates indexed findings, the runner-doctor workflow import, and portable agent output deterministically; CI catches a stale generated artifact or changed stable ID.
• Representative fixtures route an ARC split-filesystem bind failure, a Squid/DNS denial, an enterprise Copilot header failure, an API-proxy OIDC configuration error, an mcpg OIDC boundary failure, and a CI safe-output error to the right record/reference; ambiguous input yields a small safe probe rather than a speculative fix.
• Maintainers can trace each diagnosis to evidence, shipped version scope, owner/status, and a documented update/retraction process. Runner, auth, and later domain updaters use the same proposal/validation/deduplication primitives.
• No diagnostic or generated guidance asks for credentials, token exchange, full environment dumps, or a security bypass. Existing documented support and runner IDs remain intact; affected gh-aw sources compile and their .lock.yml outputs stay synchronized.