`living-doc-gap-finder`: make `compute_gaps.py` deterministic and true to the skill

Open
  • Is something wrong?
#123 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
55/100
Issue type
Refactor
Clarity
Clearly specified
Activity status
Active
Tech stack
json, python
Domain
testing, tooling

Research direction

Start with scripts/compute_gaps.py in the living-doc-gap-finder skill: the set iteration near line 137 that assigns GAP ids, and the .get("catalog", {}) defaults. Re-run the PYTHONHASHSEED loop and the echo '{}' check from the issue to reproduce both defects. Done means identical output across seeds, UNTESTED_AC for Functionality with ac_count > 0, a non-zero exit on malformed input, and eval 1 corrected in evals/evals.json.

Written by the indexing model from the issue text.

Description

type:tech-debt
Description of Technical Debt

Five defects in one skill, found by the last eval run (line numbers as measured then; re-check before editing):

  • GAP ids are not deterministic. scripts/compute_gaps.py:137 iterates a set of AC ids while ids come from a
    running counter, so the order, and with it every GAP id, depends on PYTHONHASHSEED. SKILL.md:27 and the docstring
    promise "deterministically".
  • The output contradicts the skill's taxonomy. The script emits UNDOCUMENTED_FUNCTIONALITY for every
    Functionality without tests, whatever its ac_count. The skill says a Functionality with criteria and no tests is
    UNTESTED_AC only (SKILL.md:46-47), so a correct report depends on the model fixing the output by hand. Eval 1's
    assertion (evals/evals.json:16) says FUNC-002 "has no ACs of its own", while its fixture gives ac_count: 5.
  • Wrong-shaped input passes. .get("catalog", {}), .get("inventory", {}) and .get("known_test_links", {})
    default to empty, so {} reports "0/0 ACs covered" with exit 0.
  • The only worked example of the input is an eval fixture. SKILL.md:38-39 points at
    evals/files/catalog-snapshot.json.
  • "all 9 gap types" (SKILL.md:27, compute_gaps.py:3, evals/evals.json:199) against "Ten gap types … at
    nine priority levels"     (SKILL.md:68).
Impact of Technical Debt
  • The same snapshot gives different GAP ids per run.
  • Reports are right only if the model corrects the script.
  • A malformed snapshot gives an all-clear.
Category

Code Quality / Refactoring

Priority

High - Actively causing problems

Proposed Solution

Iterate in sorted order, with a test that compares two runs under different seeds. Emit UNTESTED_AC for a
Functionality with ac_count > 0, and correct eval 1. Exit non-zero, naming the missing sections, when none of the
three exists. Move a minimal input example into the skill or its references/ and point the docstring at it. Say "ten
gap types at nine priority levels" everywhere. Re-measure the named evals by agentic/docs/testing/eval-protocol.md (two runs, one grader), and run the regression gate in agentic/docs/testing/baseline.md when a script or canon copy changes. Eval: gap-finder 1.

Effort Estimate

1 day

Dependencies / Related

No response

Additional Context

Measured 2026-11-12 at living-doc 2df73dc. Re-check:

for s in 1 2 3; do PYTHONHASHSEED=$s python3 agentic/.claude/skills/living-doc-gap-finder/scripts/compute_gaps.py agentic/.claude/skills/living-doc-gap-finder/evals/files/catalog-snapshot.json | md5; done gives three equal hashes when fixed. echo '{}' > /tmp/x.json && python3 agentic/.claude/skills/living-doc-gap-finder/scripts/compute_gaps.py /tmp/x.json; echo rc=$? is non-zero when fixed.

Dominant language
Python
Stars
0
Forks
0
Avg merge
6h 14m
Merged PRs (30d)
27

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from AbsaOSS/living-doc

All issues in AbsaOSS/living-doc

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.