Three skill regressions from the last fix pass

Open
  • Is something wrong?
#122 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
62/100
Issue type
Documentation
Clarity
Clearly specified
Activity status
Active
Tech stack
json, markdown, python
Domain
documentation, testing-qa

Research direction

Start at agentic/.claude/skills/living-doc-impact-analysis/SKILL.md line 63 and the matching expected output in evals/evals.json line 194, then check the wording at living-doc-scenario-creator/SKILL.md lines 222 and 232-235 and the surface_type text in living-doc-create-feature/SKILL.md. Re-measure evals impact-analysis 15, scenario-creator 8 and 11, and create-feature 6 and 8 per agentic/docs/testing/eval-protocol.md. Done when the grep for living-doc-create-feature in the impact-analysis SKILL.md returns a match and those evals pass on two runs.

Written by the indexing model from the issue text.

Description

type:tech-debt
Description of Technical Debt

The last eval run measured three criteria that passed before the previous fix pass and fail after it (0 of 2 runs each):

  • living-doc-impact-analysis lost its route to living-doc-create-feature. The entity-ID rewrite turned
    SKILL.md:63 into a pointer to living-doc-update's "Split or merge a Feature" only, so the skill no longer names
    living-doc-create-feature. Eval 15's routing criterion fails, and its expected output (evals/evals.json:194) still
    names the retired "Rename a Feature" workflow.
  • living-doc-scenario-creator drops the coverage table when no script ran. SKILL.md:222 requires it after the
    .feature block. The newer wording at :232-235 led both runs to leave it out (eval 8).
  • living-doc-create-feature drafts surface_type: "API" without saying only UI and API exist when it infers
    the type (eval 6).
Impact of Technical Debt
  • Wrong routing for new Features; reports missing their mandated part; an unexplained inferred type.
Category

Code Quality / Refactoring

Priority

High - Actively causing problems

Proposed Solution

At impact-analysis :63, say that a new Feature is created with living-doc-create-feature and registered here, and
keep the split/merge sentence; fix eval 15's expected output. In scenario-creator, state that with no script run the
derived summary is still appended, labelled "derived from your description, not from a coverage_report.py run". In
create-feature, require the draft to state the two-type rule whenever surface_type is inferred. Re-measure the named evals by agentic/docs/testing/eval-protocol.md (two runs, one grader), and run the regression gate in agentic/docs/testing/baseline.md when a script or canon copy changes. Evals:
impact-analysis 15, scenario-creator 8 and 11, create-feature 6 and 8.

Effort Estimate

1 day

Dependencies / Related

No response

Additional Context

Measured 2026-11-12 at living-doc 2df73dc. Re-check:

grep -n "living-doc-create-feature" agentic/.claude/skills/living-doc-impact-analysis/SKILL.md is empty while this is open.

Dominant language
Python
Stars
0
Forks
0
Avg merge
6h 14m
Merged PRs (30d)
27

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from AbsaOSS/living-doc

All issues in AbsaOSS/living-doc

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.