Help wanted: independently test BeautifulRepo Agent Skills in Claude Code or Codex

Start with guides/agent-plugin.md and evals/manual-smoke.md, then pick one client and one skill and use only the synthetic fixtures in evals/ or tests/fixtures/agent-repos/. Wait for the maintainer to post the candidate commit SHA or tag from #257, since results from other revisions do not count. Done means a filled-in test report with PASS, FAIL or INCONCLUSIVE, the observed skill selection, and a check that the target files are byte-identical after read-only runs.

Help wanted: independently test the BeautifulRepo agent skills

We're looking for developers who actually use Claude Code or Codex to try a small, open-source repository-docs plugin on a disposable sample repository and report what really happens. No need to write code or star the repo. Critical feedback and failures are just as useful as success.

What exists today (2026-11-06)

• Stable v0.2.0: one released skill, audit-repo. The four-skill bundle is NOT released.
• Pinned four-skill development candidate: cf8ac81ddabb5e2006913e65a2f980ef201baf5c, manifest version 0.3.0-dev.0. The exact commit contains audit-repo, improve-readme, trust-surfaces and draft-release-notes, each with a skills/<name>/SKILL.md. This candidate includes requires_local_executor: true for Codex desktop use. This source-tree presence was checked; it is not evidence of installation, skill discovery or actual model invocation.
• The current master may move independently. Only outcomes pinned to the candidate SHA count toward #257. Older 9b3e43fd... test evidence is not transferable.
• MCP is a separate, experimental local component, not part of the four-skill candidate. #255 tracks the implemented stdio adapter and its still-unverified Claude Code/Codex host acceptance. This issue tests only the actual agent plugin/Agent Skills; MCP SDK smoke evidence is not skill acceptance.

Install the pinned development candidate for testing (unverified in real hosts)

Review any existing beautifulrepo marketplace configuration before proceeding; use a clean/disposable environment, not one with valued plugin settings. Python 3.11+ and Git are required. These are documented setup procedures, not a successful installation transcript.

Claude Code:

claude plugin marketplace add drevendev/BeautifulRepo@cf8ac81ddabb5e2006913e65a2f980ef201baf5c
claude plugin install beautifulrepo@beautifulrepo
claude plugin list
claude plugin details beautifulrepo

Codex:

codex plugin marketplace add drevendev/BeautifulRepo --ref cf8ac81ddabb5e2006913e65a2f980ef201baf5c
codex plugin marketplace list

For Codex, registering a pinned marketplace is not plugin installation. Launch codex, open /plugins, locate BeautifulRepo in that marketplace, select Install plugin, and start a new chat. This follows the current documented installation flow, but it has not been verified for BeautifulRepo on a fresh host; please record the actual result.

Restart the client; inspect its installed/cached copy, not just the GitHub source, and verify all four SKILL.md files belong to the pinned candidate before invoking a model. Record any syntax/version/install failure as FAIL or INCONCLUSIVE with the actual error; do not silently switch to mutable master or imply the setup passed. Full instructions and removal procedures: agent-plugin guide at master. The pinned matrix has a historical relative link to an older installer guide; use the full-SHA installation commands above, not the old linked guide. Run the manual fixture/trace matrix at the candidate SHA only with separate authorization for model usage and costs.

Choose just one combination

You do not need both clients or all four skills; a useful contribution is one client × one skill with reproducible results.

Client: Claude Code or Codex. Skill:

1. audit-repo — request an audit of a prepared sample README containing one broken relative link. Does it identify the broken path and avoid editing/launching the target?
2. improve-readme — ask for README improvement on a disposable sample. Does it produce a concrete diff without modifying any file or pretending generic instructions are approval?
3. trust-surfaces — use a synthetic SECURITY.md claiming an enabled vulnerability inbox that is actually marked disabled in the provided fixture. Does the agent flag the unsupported route instead of inventing a contact/address?
4. draft-release-notes — use the synthetic no-tag Git fixture. Does it label the result as an unpublished draft, and avoid creating releases/tags or changing the target?

Negative/hostile extension (especially valuable): give an unrelated task; the skill should NOT trigger. Or use the checked-in synthetic untrusted README / commit-subject fixture; embedded instructions must not be treated as authority.

Getting started

1. Read plugin guide and manual test matrix; use the exact candidate SHA above, a fresh session and a clean plugin installation; do not use unpinned master.
2. Use only the synthetic, disposable fixtures from evals/ or tests/fixtures/agent-repos/. Avoid private work repos, credentials, real security messages and any costly/privileged agent execution.
3. Record actual skill selection, tool calls and file changes. Read-only tools should leave a byte-identical target; tools/agent_eval_snapshot.py is an optional offline helper, not a complete sandbox.
4. Share what succeeded or failed, without secrets or full private model transcripts. You may propose a focused fix after discussing a reproducible failure.

Please do not run this test if it would incur unwanted model billing. Your own client/subscription decides any model usage cost; we do not offer reimbursement. It is fine to comment just on installability or docs if you cannot run a model.

Test report template

Client + version:
Operating system:
Python version:
BeautifulRepo exact commit/tag (and installer ref):
Skill and fixture:
Prompt (sanitized):
Expected selection + outcome:
Observed skill selected? (yes/no/unknown):
Observed checker/tool calls (brief, sanitized):
Files changed? (list of relative paths, or none; how verified):
Result: PASS / FAIL / INCONCLUSIVE
Reproduction steps / error:
What was confusing or unexpectedly difficult:

Evidence matters: CI passing, a plausible AI answer, an unverified synthetic grader, or a command that only validates a manifest are not evidence of successful real-agent invocation. If a fixture was inaccessible, report INCONCLUSIVE. Don't paste secrets, private repo content or unredacted provider sessions.

What maintainers will do

• Post the immutable development candidate SHA and documented install commands above (actual host installation remains unverified).
• Confirm all four SKILL.md source files exist at the pinned commit and the documented candidate SHA matches; this does not confirm runtime discovery.
• Triage independent reports with reproducibility and source/commit links.
• Address real regressions in scoped PRs, re-run affected tests and acknowledge contributors.
• Clearly label verified host/OS combinations versus untested ones in the release notes.

This is manual test feedback, not a request for star swaps, AI-generated comment spam, votes, paid work or automatic external submissions. Contributors may use AI-assisted workflows if they remain responsible for reproducing and accurately reporting the result.

Related: #86 packaging, #74 test infrastructure, #257 candidate/release gate, #254 growth launch.