Prove a green suite was earned: changed-line coverage and mutation checks

トランスクリプトの Edit payload と、その oldString/newString および structuredPatch データから始め、次に Stop hook がモデル呼び出しなしでカバレッジレポートを検査できる仕組みを追跡します。coverage.xml など、1 つの形式をエンドツーエンドで実装し、レポートが最後の編集より新しい場合にのみ変更行がマッピングされることを確認します。変更行がカバーされていなかった場合に unsupported result が報告され、mutation は backcheck --mutate による opt-in のままであることが完了条件です。

Today backcheck can tell you the suite passed and that nobody disabled a test to get there.
It cannot tell you the passing suite actually exercises the code that changed — the most
common way a green bar means nothing.

An agent edits src/billing.py, runs the suite, everything passes. If no test touches the new
branch, the green is real and meaningless.

What would close the gap

Changed-line coverage. The agent edited these files; the coverage report says these lines
ran. Were any of the changed lines covered?

✗ unsupported  tests pass
    `pytest` passed, but none of the 14 lines changed in src/billing.py were executed
    by the suite (coverage.xml)

This is tractable because coverage output is a file on disk with a stable format, and
backcheck already knows exactly which lines were edited — the Edit payloads in the
transcript carry oldString/newString and a structuredPatch.

• parse coverage.xml / .coverage (Python)
• parse lcov.info (JS, Rust via tarpaulin/llvm-cov)
• parse cobertura.xml (widely emitted)
• map changed lines from the transcript's patches onto the coverage report
• report only when a coverage file exists and is newer than the last edit

Mutation signal (harder, later). A test that runs a line but asserts nothing about it still
passes when the line is broken. Scoped mutation testing on changed lines only is the rigorous
version of "was this green earned", and is
discussed
as the real answer to agents gaming test suites. It is slow and invasive, so it would have to be
opt-in (backcheck --mutate) and probably delegated to mutmut/cargo-mutants/stryker
rather than reimplemented.

Design constraint

Coverage parsing keeps the no-model-calls rule intact and stays deterministic, which is why it
is the right first step. Anything that needs to run code (mutation) must be explicitly opt-in
and must never happen inside the Stop hook — the hook has to stay fast enough to be invisible.

Start with one coverage format end-to-end rather than a general framework.