The agentic layer's repeatable steps moved from the model into scripts

Open
  • Is something wrong?
#112 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
10/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
angular, playwright, python
Domain
testing-qa, tooling

Research direction

This is an epic with several sub-tasks, so start with the first one: re-audit the four skills and two scripts under agentic/.claude/skills/ against the 2026-09 plan. Then read living-doc-shared/lib/ and the compute_gaps.py and trace_impact.py scripts, plus agentic/CONTRIBUTING.md and agentic/docs/testing/eval-protocol.md. Done means the audit is recorded and a single sub-task is ready to start, not the whole epic.

Written by the indexing model from the issue text.

Description

type:epic
Background

Several skills in agentic/ ask the model to do work a script could do the same way every time: comparing browser
snapshots, resolving routes, linting step definitions, matching steps to definitions. Every step runs on whatever model
the session uses, so a cheap scan costs as much as a judgement, and answers vary between runs where a script would give
one result. A plan written in 2026-09, when the layer still lived in agentic-toolkit, audited four skills and
specified their scripts. Its paths are from before the move to agentic/.claude/skills/, and some skills have changed
since.

Goals
  • Every script prints one JSON envelope that says whether its result is final, ambiguous (with narrowed candidates,
    not a raw dump) or an error, so a skill knows when the model is needed.
  • The four skills' mechanical steps run as scripts with tests.
  • Each step names the model it needs: a small model scans, a large model judges.
  • Token cost and answer quality are measured before and after, with a go/no-go before the pattern spreads.
Subtasks

In landing order. The script specs below come from the 2026-09 audit; the first child re-checks them.

  • Re-audit the four skills and the two scripts to retrofit, against agentic/.claude/skills/ today.
  • Escalation contract: the envelope's schema and reference page, plus a helper module in living-doc-shared/lib/.
    living-doc-gap-finder's compute_gaps.py and living-doc-impact-analysis's trace_impact.py adopt it
    first, and agentic/CONTRIBUTING.md says every new script speaks it.
  • living-doc-pageobject-scan: summarize_snapshot.py. It condenses one MCP Playwright accessibility snapshot
    into a per-route element table and a new-versus-known diff, so the crawl stops re-reading raw snapshots. It
    appends each iteration's counts to a scratch log and flags a coverage plateau (no new elements or routes over N
    iterations); the plateau check lives here, not in manifest_diff.py.
  • data-cy-instrument: two scripts. resolve_gap_targets.py (Angular only) maps a manifest coverage gap and its
    route to the component file through the routing modules ({ path, component } and lazy loadComponent
    entries, route parameters, feature-flag variants). validate_test_id_name.py is framework-agnostic and checks a
    proposed test id: kebab-case, then the naming rules.
  • gherkin-step: lint_steps.py, a static lint of step definitions.
  • gherkin-living-doc-sync: the step matching in bdd-maintain's find_unused_steps.py (cucumber_to_regex,
    StepDefinition, collect_step_definitions with a file-glob parameter) moves into
    living-doc-shared/lib/step_matcher.py. A new detect_step_drift.py finds .feature step lines with no
    matching definition and reports, for each, the most similar definition (a difflib ratio of at least 0.5) and
    the PageObject method it calls.
  • Model tiering: the agent's mode dispatch names the model per step. A two-phase pattern (cheap scan, expensive
    judgement) goes into pageobject-scan's create and re-scan, bdd-maintain's dead-code audit and gap-finder's
    audit, and a guide states the pattern.
  • Measurement: define benchmark sessions, capture a baseline before tiering and a capture after it, then a
    comparison with a go/no-go, and a retro with the follow-ups.
Related / References
  • agentic/docs/testing/eval-protocol.md, agentic/docs/testing/baseline.md (how a change is measured).
  • Related issues: living-doc-gap-finder's compute_gaps.py defects (fix before the retrofit); the eval tooling.
  • Not in this epic: PageObject generation and healing, Playwright crawling and step-definition authoring. They stay
    out, and none of the agentic layer is ever a pipeline dependency.
Dominant language
Python
Stars
0
Forks
0
Avg merge
6h 14m
Merged PRs (30d)
27

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from AbsaOSS/living-doc

All issues in AbsaOSS/living-doc

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.