Pipeline controls: planted defects prove each gate can still go red

Open
  • Is something wrong?
#7,155 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Feature
Clarity
Needs clarification
Activity status
Active
Tech stack
github-actions, python, yaml
Domain
ci-cd, testing, tooling

Research direction

The issue mentions det_mutation.py and mutation.py and the CI rules in .claude/rules/ci-workflows.md. First read those and enumerate gates from shepherd-*.yml and other workflows. For the cheapest first step, add per-gate fixtures that must fail and a suite check requiring them; a gate with no fixture fails. Full completion spans control PRs and review experiments needing deep CI familiarity.

Written by the indexing model from the issue text.

Description

enhancement priority: high scope:infra

Why

A green gate only means something if that gate can go red. Three failures from one week looked healthy until someone checked:

  • CodeQL: every PR read "No new alerts" while PR scans held 0 results, and open alerts went from 34 to 459 (#7153).
  • ci_busy_hours: read an impossible 45 h on Oct 23, and nothing noticed.
  • promoter-bot #89: passed every test, yet would have stopped every host's dashboard push once factory.json passed 128 KB.

We test code heavily, but we don't test whether the tests and gates still work. det_mutation.py and mutation.py measure whether tests catch code changes. Nothing measures whether a bad PR is actually stopped.

The idea

Run lab-style controls on the pipeline itself:

  • Negative control: a known-bad change. Its gate must go red. If it stays green, that gate is blind, which matters more than any single bug.
  • Positive control: a known-clean change. Everything must stay green. This measures false alarms and fix-loop cost.

The output is a detection matrix: each gate against each planted defect, with pass or miss and how long it took. A miss is treated as a high-priority failure of the gate, not of the code.

Plan, cheapest first

1. Gate self-tests (no extra CI runs)
  • List every gate and what it claims to catch
  • Give each gate at least one fixture it must fail on, and run these in the existing suite
  • Add a check that fails when a gate has no fixture
2. Weekly control PR (one full CI run a week)
  • A scheduled job opens a real, ready-for-review PR from a canary/ branch, carrying one small planted defect per gate
  • A results job compares which gates went red with which should have, posts the matrix, then closes the PR and deletes the branch
  • Also run a clean control PR, to measure false alarms
  • A miss opens a priority: high issue
3. Controls on the review layer (an experiment, not a default)

Gates catch rules. Reviewers catch judgement. The #5274 import cycles passed every gate, and a reviewer was the only layer that could have questioned them.

  • Hide planted defects in realistic diffs, such as a swallowed exception, a test that can't fail, or a green-by-skipping check
  • Measure catch rate and false alarms for the Sonnet review, and compare with Opus on the same set
  • The reviewer must not know a diff is a control. Run through the experiment lane: Opus design, owner OK, logged in temporal-crucible, cost-vs-reward gate first
4. Learn from escapes
  • Every defect that reaches main adds a new control, so the set grows from real failures
  • A monthly Opus session reviews misses, false alarms and escapes, and proposes new control classes
  • Add the detection matrix to the squid-bot dashboard

Safety rules

  • A control PR can never merge: canary/ branches carry a hold that ci-gate always fails, whatever else is green, and the merge label is refused
  • Planted defects are inert: no real vulnerabilities, secrets, network calls, or changes to workflows or release paths
  • Control runs stay out of the factory's own numbers: merges, first-pass rate, lessons and agent memory
  • A control PR that fails to close alerts the owner

How to judge it

  • Cost to measure first: the runner-minutes of one control PR (.claude/rules/ci-workflows.md) and the tokens for a review session.
  • Worth continuing if it finds gaps that weekly CI doesn't. Part 2 should show its value within about 4 runs, or the cadence drops.

Open questions

  • Should part 2 be weekly, or follow gate changes only?
  • Do control PRs count against the 20-runner pool at peak hours, or run off-peak?
  • Can controls for owner-only workflows (shepherd-*.yml) run here, or do they need separate approval?
Dominant language
Python
Stars
57
Forks
11
Avg merge
1h 35m
Merged PRs (30d)
975

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from squid-protocol/gitgalaxy

All issues in squid-protocol/gitgalaxy

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.