Verify claims about things other than tests: migrations, deploys, docs, dependencies

Open
#9 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
68/100
Issue type
Feature
Clarity
Clearly specified
Activity status
Quiet
Tech stack
rust
Domain
devtools, testing-qa

Research direction

Start with ClaimKind and all_patterns() in src/claims.rs, then trace evidence collection in src/evidence.rs and verification in src/verify.rs. Choose one claim, add its evidence handling and verifier, and add tests for both supported evidence and missing or hedged evidence, including the output line used for the verdict.

Written by the indexing model from the issue text.

Description

enhancement good first issue help wanted

The claim taxonomy in src/claims.rs covers tests, type checks,
lint, build, commit, push, and file writes. Agents routinely assert plenty else, and each of
these has recorded evidence that could back or refute it.

Each item below is a self-contained contribution: one claim pattern, one verifier, two tests.

  • "I ran the migration" — evidence: alembic upgrade, prisma migrate, rails db:migrate,
    django manage.py migrate, and what they printed
  • "I installed the dependency" — evidence: a package-manager command and the manifest
    actually changing (package.json, pyproject.toml, Cargo.toml)
  • "I updated the docs" — evidence: a write to a docs path; flag "documented in the README"
    when the README was never touched
  • "I deployed" / "it's live" — evidence: a deploy command that succeeded
  • "I reverted that" — evidence: git revert/git checkout/git restore, or an edit
    restoring the earlier content
  • "I removed the debug logging" — evidence: no console.log/print/dbg! remains in
    the added lines. This one is checkable directly from the edit payloads.
  • "the endpoint returns 200" — evidence: a curl/httpie call and its status
  • "I opened a PR" — evidence: gh pr create and the URL it printed
  • "no secrets were committed" — evidence: added lines containing key-shaped strings

How to add one

  1. Add a variant to ClaimKind and a pattern in all_patterns().
  2. Extend the evidence ledger in src/evidence.rs if the
    evidence is not already collected.
  3. Add a match arm in src/verify.rs returning a verdict and the
    line of output it rests on    .
  4. Test both directions: the claim with evidence, and the claim without.

The trap to avoid

The patterns must reject intentions and hypotheticals. "I'll run the migration next" and "once
the migration runs" are not claims, and matching them makes the tool noisy enough that people
turn it off. is_hedged() and is_negated() exist for exactly this — extend them alongside any
new pattern, and add a test proving the hedged form is ignored.

Dominant language
Rust
Stars
0
Forks
2
No merged PRs in 30d

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from VectorInstitute/backcheck

All issues in VectorInstitute/backcheck

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.