Spec: self-testing loops from the Self-Testing AI Agents course

Start by reading the implementation decisions and identify the gstack skills, playbooks, principles, scripts, and references named there; inspect their indexes and go-help and README entry points. Check the reused course examples and vendor-specific claims against the cited commands and current documentation. Done means the specified materials and tests are added, each flagged example is reproduced before and after any fix, and the verification records are included in the PR description.

Problem

Agents using gstack still make the human their feedback loop. gstack says to prove work against the real artifact, but it gives the agent few concrete procedures for doing so. It has no procedure for a flaky test, so agents raise retries or skip the test. A red build gives the agent a wall of runner output instead of a short readable account of the failure. Nothing helps a repo owner see which feedback loops their project has and which it lacks. Nothing tells an agent which test layer fits a change either. Asked to test a React login flow, an agent following gstack today writes a component test with mocked fetch calls. That test never sees the cookies, redirects, and sessions where login actually breaks.

The Self-Testing AI Agents course has the missing procedures, rules, and scripts. Its examples also contain bugs. A model review flagged them without running anything, so some of the flags may be wrong.

Solution

gstack gains three skills, one playbook, and one principle, and four existing skills get sharper. Together they let an agent check its own work at the right layer, read a failure without help, handle a flake without hiding it, and set up the missing loops in a project.

Every course example that gstack reuses is checked before it ships. Each flagged bug is reproduced against the example as written. A confirmed bug is fixed in the gstack copy, and the reproduction runs again to show the fix. A claim that does not reproduce is dropped.

User stories

Choosing a test layer

1. As an agent testing a change, I want a rule that picks the lowest layer where the bug can actually occur, so that I test pure logic with unit tests, component behavior with component tests, and anything across the browser-server line with one real end-to-end test.
2. As an agent testing a login flow, I want the rule to name auth, sessions, routing, and redirects as browser-server concerns, so that I write one end-to-end test of the main path instead of only mocked form tests.
3. As an agent, I want to know when each layer runs, so that I run sub-second checks continuously, targeted checks on every meaningful edit, slow suites before declaring done, and leave CI-only suites to CI.

Handling a flaky test

4. As an agent facing a flaky test, I want to run it ten times and read the failure count, so that I know whether it fails only in CI (0 of 10), is a real flake (1 to 3), or is broken (4 or more). The reruns use the project's Playwright test runner, not an ad hoc browser script.
5. As an agent, I want to classify a flake as a timing race, leaked state, an ambiguous locator, or a config or auth mismatch, with evidence from the trace, before I change code.
6. As a repo owner, I want agents never to raise retries to make a flake pass, so that retries stay a ceiling and not a fix.
7. As a repo owner, I want a quarantined test marked as an expected failure with a linked issue, so that it fails the run once it starts passing and someone removes the quarantine.

Reading a failure

8. As an agent with a red test run, I want a short markdown dossier per failure with the error, location, screenshot, trace path, and a reproduce command, so that I can fix it without a human explaining the failure.
9. As an agent, I want to read the trace before changing code, and to start from the action that took far longer than the others, so that the fix follows evidence.
10. As a repo owner, I want the option to generate the dossier automatically when a test command fails, so that the agent reads it without being told to.
11. As a gstack user, I want the reproduce command to run exactly the failing test even when its title contains $, backticks, or quotes, so that pasting it never runs the wrong test or expands shell syntax.
12. As a gstack user, I want each failure's message shown whether the Playwright JSON report puts it in error or in errors, so that no failure appears with an empty message.

Auditing a project's loops

13. As a repo owner, I want an audit that lists each feedback loop with its real command or "missing, proposed: …", so that I see the loops my project has and the loops it lacks. The loops are agent rules, static checks, git hooks, end-to-end proof, failure output, CI, nightly checks, and post-deploy smoke checks.
14. As a repo owner, I want the audit to name the weakest loop and build the first fix, so that the audit ends in a change and not a report.
15. As a repo owner, I want each check placed by its speed, so that staged-file checks under ten seconds run on pre-commit, checks under two minutes run on pre-push, CI is the clean-environment superset, and nightly runs only slow checks that catch drift.
16. As a repo owner, I want each nightly failure to become an issue, a PR, or a file an agent can read, so that nightly failures do not sit unread.
17. As a repo owner, I want a post-deploy smoke check with stop-ship triggers written before the deploy, so that a bad deploy is caught and rolled back by rule.
18. As a repo owner copying the post-deploy workflow, I want the smoke job to receive the deploy URL, so that the smoke test runs against the deployment and not against an empty base URL.
19. As a repo owner using the structured triage pipeline, I want the "needs human review" count to equal the number of low-confidence failures, so that the routing decision is correct.
20. As a repo owner turning triage output into CI annotations, I want each annotation to land on the right file and line whatever its message contains.

Keeping agents inside the guardrails

21. As a repo owner, I want git commit -n, git -C <path> commit --no-verify, and the env prefixes HUSKY=0, LEFTHOOK=0, LEFTHOOK=false, and LEFTHOOK_EXCLUDE= denied, so that an agent cannot skip git hooks with a short flag or another spelling.
22. As a repo owner, I want ordinary commands that contain -n, such as git log -n 5 and grep -n, allowed, so that the guard does not block normal work.
23. As a repo owner, I want the files that hold the guardrails themselves protected, so that an agent cannot pass a check by editing hook configs, CI workflows, or lint config. The audit also flags quieter bypasses: a narrowed glob, a removed or renamed check, and a step moved to a later stage.
24. As a repo owner, I want behavior that must happen every time enforced by a hook, so that correct reaches for a hook when a rule alone keeps being skipped.

Learning from review

25. As a repo owner, I want a review-bot finding that shows up in a third PR promoted to an agent rule, and to a lint or a test if it keeps happening, so that review comments stop repeating.
26. As a repo owner writing a reviewer config, I want a structure to follow, so that the reviewer knows what to flag and where, what to ignore, and to name the line and a specific fix in plain words.
27. As a repo owner, I want to test a reviewer config by planting a bug that passes every local check, so that I know whether the config catches it and whether an agent can fix it from the comment alone.

Proving behavior in a running app

28. As an agent proving a change in a project's verification skill, I want any new console error to block done, so that a change that throws is never reported as working.
29. As an agent, I want accessibility violations on changed routes to block done, with results that axe marks incomplete going to a human to review.
30. As an agent, I want visual baselines changed only in the same commit as an intentional UI change, so that a baseline is never regenerated to make a test pass.
31. As an agent working on a web app, I want the course's Playwright rules available for reference, so that my tests use stable locators and real waits. The rules cover the locator order, banned waits, storage-state auth, HAR recording, and test.step. The reference opens by saying that Playwright drives the browser to prove a change, and that Chrome DevTools MCP is for debugging, not for proof.

Implementation decisions

New skills, playbook, and principle

• Test-layer principle. A new principle states the rule in stories 1 to 3. It joins the Verification group in the go-mode principle index. The tdd skill and the Feature playbook point to it. To generate a new end-to-end suite, the principle points to Playwright Test Agents (npx playwright init-agents) instead of having the agent write the suite by hand.
• Flaky-test playbook. A new go-mode playbook covers stories 4 to 7. It sits beside Bug fix and Trace forensics in the go-mode playbook index, with one routing line.
• dossier skill. It ships a script that reads the runner's JSON report and writes one markdown file per failure (stories 8 to 12). The skill body holds the trace-reading procedure and the optional hook that runs the script when a test command fails. The script targets Playwright's JSON report first.
• loop-audit skill. It generalizes the course's lesson on porting the loop to another stack (stories 13 to 23). Its references hold the stage placement table, the post-deploy smoke example, the nightly example, the structured triage pipeline, and the anti-bypass checklist. It sends static-layer setup to the user's project-guardrails skill when that skill is installed, and it does not duplicate that skill.
• Bypass guard. A hook script ships with loop-audit (stories 21 and 22). It splits the command into separate git invocations and checks flags only on git commit, as the course's Cursor variant does. A plain regex over the whole command cannot tell commit -n from log -n.

Changes to existing skills

• correct. Add hooks as the level for behavior that must happen every time, between types and tests (story 24). Add the review-bot third-PR threshold as a source of mistake classes (story 25). Add protecting the guardrail files themselves (story 23).
• Review-bot triage reference in go-mode. Link the third-PR threshold to correct. Add the reviewer config structure and the planted-bug check (stories 26 and 27).
• create-verification-skill. Add stories 28 to 30 to the proof standards in the skill it generates. Add a Playwright reference with the course's rules (story 31), linked only for web surfaces. The reference names the driving tools: the project's Playwright test runner for proof, and @playwright/cli for an interactive loop when the agent works in a shell. Playwright MCP is the fallback when the host has no shell loop. The debugging side, Chrome DevTools and the control-ui routing, is a separate spec.
• go-help and README. Each new skill, the playbook, and the principle get a go-help row and a README line, per the gstack convention.

Checking reused course examples

• Each flagged bug follows the same order. Run the course example as written and record the failure. Apply the fix. Run it again and record the pass. Both records go in the PR description.
• A claim that does not reproduce is dropped. The PR lists it with the command that disproved it.
• Fixes land in the gstack copies only. No patches go to the course repo.
• The dossier builds the reproduce command with POSIX single-quote escaping, not JSON.stringify.
• The workflow fix declares the deploy URL as a job-level output. actionlint reports references to undeclared job outputs, so it confirms both the bug and the fix.
• The jq fix collects low-confidence failures into one array before counting them.
• The annotation fix follows GitHub's workflow-command escaping: %, \r, and \n in the message, plus , and : in property values. The review claimed that commas and colons in the message also break annotations. GitHub's rules say they do not, so that claim is checked like the others.
• Vendor-specific facts from the course, such as CLI flags, hook payload shapes, and action versions, are checked against current docs before they go in a skill. The course dates from April 2026. The same check covers the facts from Steve Kinney's article "Playwright vs. Chrome DevTools MCP": @playwright/cli, npx playwright test --debug=cli, and npx playwright init-agents.

Left to the build

• The test-layer principle's name.
• Which Playwright version the dossier fixture report comes from.
• Whether the dossier script supports a second runner's JSON format now or later.

Testing

• Dossier script. A node --test regression test runs the script on a fixture JSON report with hostile test titles and both error shapes. It catches quoting and missing-message regressions. It misses report-format changes in later Playwright versions. Prior art: the doctor tests spawn the real script on fixture files.
• Bypass guard. A node --test regression test pipes sample PreToolUse payloads into the hook script. It asserts a deny for every bypass form in story 21 and an allow for every look-alike in story 22. It misses changes to the harness payload shape. Prior art: the same doctor tests.

Everything else is verified by running it. Each new or changed skill passes the authoring-a-skill validation and the build. loop-audit runs once on a real project. The post-deploy workflow passes actionlint, the jq fix runs on a sample triage stream, and one annotation renders in a GitHub Actions run.

Out of scope

• Performance budgets. The course's checker sums module gzip sizes, not emitted files, and never reads its runtime budget. It needs its own design.
• Vendor walkthroughs for Bugbot, Copilot, and Codex.
• The capstone grading checklist.
• The Codex edit guard and the custom MCP probe. gstack does not reuse them.
• Contradictions in the course prose, such as the default retry count and whether the starter ships a Lefthook config.
• Upstreaming fixes to the course repo.