Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Prove a green suite was earned: changed-line coverage and mutation checks

Open
  • Is something wrong?
#8 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
50/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Quiet
Tech stack
javascript, python, rust
Domain
cli, devtools, testing-qa

Research direction

Start with the transcript Edit payloads and their oldString/newString and structuredPatch data, then trace how the Stop hook can inspect coverage reports without model calls. Implement one format end to end, such as coverage.xml, and verify that changed lines are mapped only when the report is newer than the last edit. Done means an unsupported result is reported when changed lines were not covered, while mutation remains opt-in via backcheck --mutate.

Written by the indexing model from the issue text.

Description

accuracy enhancement help wanted

Today backcheck can tell you the suite passed and that nobody disabled a test to get there.
It cannot tell you the passing suite actually exercises the code that changed — the most
common way a green bar means nothing.

An agent edits src/billing.py, runs the suite, everything passes. If no test touches the new
branch, the green is real and meaningless.

What would close the gap

Changed-line coverage. The agent edited these files; the coverage report says these lines
ran. Were any of the changed lines covered?

✗ unsupported  tests pass
    `pytest` passed, but none of the 14 lines changed in src/billing.py were executed
    by the suite (coverage.xml)

This is tractable because coverage output is a file on disk with a stable format, and
backcheck already knows exactly which lines were edited — the Edit payloads in the
transcript carry oldString/newString and a structuredPatch.

  • parse coverage.xml / .coverage (Python)
  • parse lcov.info (JS, Rust via tarpaulin/llvm-cov)
  • parse cobertura.xml (widely emitted)
  • map changed lines from the transcript's patches onto the coverage report
  • report only when a coverage file exists and is newer than the last edit

Mutation signal (harder, later). A test that runs a line but asserts nothing about it still
passes when the line is broken. Scoped mutation testing on changed lines only is the rigorous
version of "was this green earned", and is
discussed
as the real answer to agents gaming test suites. It is slow and invasive, so it would have to be
opt-in (backcheck --mutate) and probably delegated to mutmut/cargo-mutants/stryker
rather than reimplemented.

Design constraint

Coverage parsing keeps the no-model-calls rule intact and stays deterministic, which is why it
is the right first step. Anything that needs to run code (mutation) must be explicitly opt-in
and must never happen inside the Stop hook — the hook has to stay fast enough to be invisible.

Start with one coverage format end-to-end rather than a general framework.

Dominant language
Rust
Stars
0
Forks
2
No merged PRs in 30d

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from VectorInstitute/backcheck

All issues in VectorInstitute/backcheck

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.