CI should notice a snapshot edited around the gate: rules that vanish, stamps that advance without a run

Open
  • Is something wrong?
#239 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
55/100
Issue type
Feature
Clarity
Clearly specified
Activity status
Active
Tech stack
github-actions, python
Domain
ci-cd, devtools

Research direction

Start by reading the shipped workflow and its three copies, then trace how okl export --drift, okl verify, and drift.stamp_problem represent snapshots and receipts. Add the pull request comparison as a separate logged step, preserving byte-identical workflows and a standard-library-only workflow test. Done when the acceptance cases pass, including the push skip and older snapshots without retired.

Written by the indexing model from the issue text.

Description

enhancement

Problem

CI's gate reads the committed snapshot okl-drift.json and judges each rule as the snapshot describes it. A branch can change the snapshot as well as the code: advance a rule's verification by hand, or drop a rule from the snapshot, and the gate on that branch is green. The repository's principle is that removal is recorded, never deleted (retire with a reason), and nothing in CI holds it.

Where it stands (verified 2026-10-20)

  • The snapshot is written by okl export --drift and refreshed by okl verify; CI never writes it. It carries files, id, scope, title, type, verified_at, verified_by (the evidence receipt: command, expected signal, commit, minute; _stored_check parses the command back out) and verified_commit per rule. It carries no retirements and no check runs.
  • drift.stamp_problem refuses a snapshot whose stamp is internally inconsistent, so a verified_commit that disagrees with its receipt is caught. A receipt rewritten wholesale is not, and a rule deleted from the snapshot leaves no trace.
  • The workflow runs on pull_request (base branch reachable; fetch-depth: 0) and on push to main, where there is no base.

Proposal

In the shipped workflow, after the gate and only on pull_request, compare the PR's snapshot with the base branch's:

  1. A rule that disappeared must be explained: the snapshot gains a retired list (id, kind, reason, when) written by okl export --drift from the retirement log, and CI fails when an id present in the base snapshot is absent from both the PR's rules and its retired list. Removing files from a lesson is the other legitimate exit and is listed the same way (unwatched).
  2. A verification that advanced (verified_at later than the base's) must carry a receipt whose commit is an ancestor of the PR head and whose command equals the command in the base snapshot's receipt; a changed check is a deliberate change that must show in the diff. CI fails otherwise. This extends what stamp_problem checks today.
  3. The comparison is a separate step with its own line in the job log, so a failure says which rule and which condition.

Older snapshots without the retired list pass the first check with a note, so existing repos are not broken on upgrade.

Interaction with #213: a rule that one teammate's clone never held would now fail CI as "disappeared without retirement" when that clone exports. That is the intended tightening; #213's guard (refuse an export that drops rules the committed snapshot holds) is the local half of the same rule.

Acceptance

  • A PR that deletes a rule from the snapshot fails CI; the same PR with a retirement in the list passes.
  • A PR that edits verified_at by hand fails CI; a PR whose re-verification ran okl verify passes.
  • The step is skipped on push to main and says so.
  • The three workflow copies stay byte-identical; any test that reads the workflow keeps to the standard library.

Related: #213 (the snapshot rebuilt from one clone's store), #231 (TTL in the snapshot).

Dominant language
Python
Stars
1
Forks
1
Avg merge
5h 22m
Merged PRs (30d)
119

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from emeraldleaf/okl

All issues in emeraldleaf/okl

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.