Weak answers the evals measured on unchanged skill text

Open
  • Is something wrong?
#125 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
1-2 days
Newbie friendliness
35/100
Issue type
Refactor
Clarity
Mostly clear
Activity status
Active
Tech stack
json, markdown
Domain
testing

Research direction

Start with the named skill files and their evals.json, for example gherkin-living-doc-sync, living-doc-update (SKILL.md lines 100-102) and scenario-creator (SKILL.md lines 35 and 406). Re-run the affected evals to see which assertions fail on the current text. Done means the skill text states the expected output, the weak assertions are fixed or removed, and the re-run no longer shows the listed misses.

Written by the indexing model from the issue text.

Description

type:tech-debt
Description of Technical Debt
  • gherkin-living-doc-sync drops its structured output when the prompt is a question (evals 3, 10, 11, 12): no
    sync direction, no block, no DRIFT section, no confirmation prompt.
  • living-doc-update has no instruction for an invariant it cannot check because the entity is absent
    (SKILL.md:100-102). Eval 2's assertion says "five" where the skill lists four rows.
  • Weak or ambiguous assertions. Gap-finder eval 12's third assertion is taught by no sentence of the skill.
    Scenario-creator eval 5: the skill never says where @AC:STANDALONE goes (SKILL.md:35, :406). Bdd-maintain
    evals.json:14 is a negative that a refusal satisfies, and :50 expects a sentence the skill lacks. Pageobject-scan
    evals.json:27 needs an id the prompt never gives, :41 conflicts with SKILL.md:319-321, and :169 / :184
    cannot be observed in a text answer.
  • Misses on unchanged text, possibly variance. Impact-analysis evals 10 and 13; gap-finder evals 17 and 18;
    scenario-creator eval 10.
Impact of Technical Debt
  • Lower answer quality, and scores that measure the assertion rather than the skill.
Category

Testing / Test Coverage

Priority

Low - Nice to have

Proposed Solution

Add an output rule near the top of gherkin-living-doc-sync (every answer names the direction, shows the block it would emit and ends with the confirmation prompt). Add the missing living-doc-update rule and fix the count. Fix or drop each weak assertion, saying in the skill what an assertion expects. For the last group, fix only what a re-run shows again.

Effort Estimate

2 days

Dependencies / Related

No response

Additional Context

Measured 2026-11-12 at living-doc 2df73dc.

Dominant language
Python
Stars
0
Forks
0
Avg merge
6h 14m
Merged PRs (30d)
27

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from AbsaOSS/living-doc

All issues in AbsaOSS/living-doc

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.