Ship a GitHub Action so agent PRs are verified in CI

Open
#7 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
45/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Quiet
Tech stack
github-actions, rust
Domain
ci-cd, cli, devops, testing

Research direction

Start with the existing backcheck --json schema and the CLI entry points for --format github, --format markdown, and fail-on. The main action and transcript transport still need a design decision among the three options described. A smaller contribution is done when one listed output mode or the documented exit-code contract has defined behavior and supporting coverage.

Written by the indexing model from the issue text.

Description

enhancement help wanted

Locally, backcheck runs as a Stop hook and catches an unsupported claim the moment it is made.
That only protects the person who installed it. The claims that do the most damage are the ones
in a pull request description that a reviewer takes at face value.

The idea

- uses: VectorInstitute/backcheck@v1
  with:
    transcript: .backcheck/session.jsonl   # committed or uploaded as an artifact
    fail-on: unsupported                   # unsupported | contradicted | never
    comment: true                          # post findings as a PR review comment

The action would post something a reviewer can act on:

backcheck — 1 claim not fully supported

  • tests pass: pytest passed, but only a subset ran (tests/test_billing.py)
  • test integrity: a skip marker was added to tests/test_billing.py

Why it needs design first

The hard part is not the action, it is getting the transcript to CI. Options, none obviously
right:

  1. A SessionEnd hook writes a redacted summary into the repo, committed with the work.
  2. The agent uploads the transcript as a workflow artifact.
  3. backcheck emits a signed, minimal "evidence receipt" (claims + verdicts + the commands
    that ran, no file contents) small enough to live in a commit trailer or PR body.

Option 3 is the interesting one and the most privacy-preserving — a transcript is far too
sensitive to commit, but the conclusions are not.

Worth discussing the shape here before anyone writes code. backcheck --json already produces
a stable schema that could be the basis for the receipt.

Smaller pieces anyone can pick up

  • a --format github output mode producing workflow annotations (::warning file=...)
  • a --format markdown mode suitable for a PR comment
  • a documented exit-code contract for fail-on behaviour
Dominant language
Rust
Stars
0
Forks
2
No merged PRs in 30d

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from VectorInstitute/backcheck

All issues in VectorInstitute/backcheck

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.