Pipeline controls: planted defects prove each gate can still go red

オープン
  • 何か問題がありますか？
#7,155 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
25/100
issue の種類
機能追加
明瞭さ
説明が足りない
活発さ
活発
技術スタック
github-actions, python, yaml
領域
ci-cd, testing, tooling

調査の方向性

The issue mentions det_mutation.py and mutation.py and the CI rules in .claude/rules/ci-workflows.md. First read those and enumerate gates from shepherd-*.yml and other workflows. For the cheapest first step, add per-gate fixtures that must fail and a suite check requiring them; a gate with no fixture fails. Full completion spans control PRs and review experiments needing deep CI familiarity.

索引モデルが issue の本文から書いたものです。

説明

enhancement priority: high scope:infra

Why

A green gate only means something if that gate can go red. Three failures from one week looked healthy until someone checked:

  • CodeQL: every PR read "No new alerts" while PR scans held 0 results, and open alerts went from 34 to 459 (#7153).
  • ci_busy_hours: read an impossible 45 h on Oct 23, and nothing noticed.
  • promoter-bot #89: passed every test, yet would have stopped every host's dashboard push once factory.json passed 128 KB.

We test code heavily, but we don't test whether the tests and gates still work. det_mutation.py and mutation.py measure whether tests catch code changes. Nothing measures whether a bad PR is actually stopped.

The idea

Run lab-style controls on the pipeline itself:

  • Negative control: a known-bad change. Its gate must go red. If it stays green, that gate is blind, which matters more than any single bug.
  • Positive control: a known-clean change. Everything must stay green. This measures false alarms and fix-loop cost.

The output is a detection matrix: each gate against each planted defect, with pass or miss and how long it took. A miss is treated as a high-priority failure of the gate, not of the code.

Plan, cheapest first

1. Gate self-tests (no extra CI runs)
  • List every gate and what it claims to catch
  • Give each gate at least one fixture it must fail on, and run these in the existing suite
  • Add a check that fails when a gate has no fixture
2. Weekly control PR (one full CI run a week)
  • A scheduled job opens a real, ready-for-review PR from a canary/ branch, carrying one small planted defect per gate
  • A results job compares which gates went red with which should have, posts the matrix, then closes the PR and deletes the branch
  • Also run a clean control PR, to measure false alarms
  • A miss opens a priority: high issue
3. Controls on the review layer (an experiment, not a default)

Gates catch rules. Reviewers catch judgement. The #5274 import cycles passed every gate, and a reviewer was the only layer that could have questioned them.

  • Hide planted defects in realistic diffs, such as a swallowed exception, a test that can't fail, or a green-by-skipping check
  • Measure catch rate and false alarms for the Sonnet review, and compare with Opus on the same set
  • The reviewer must not know a diff is a control. Run through the experiment lane: Opus design, owner OK, logged in temporal-crucible, cost-vs-reward gate first
4. Learn from escapes
  • Every defect that reaches main adds a new control, so the set grows from real failures
  • A monthly Opus session reviews misses, false alarms and escapes, and proposes new control classes
  • Add the detection matrix to the squid-bot dashboard

Safety rules

  • A control PR can never merge: canary/ branches carry a hold that ci-gate always fails, whatever else is green, and the merge label is refused
  • Planted defects are inert: no real vulnerabilities, secrets, network calls, or changes to workflows or release paths
  • Control runs stay out of the factory's own numbers: merges, first-pass rate, lessons and agent memory
  • A control PR that fails to close alerts the owner

How to judge it

  • Cost to measure first: the runner-minutes of one control PR (.claude/rules/ci-workflows.md) and the tokens for a review session.
  • Worth continuing if it finds gaps that weekly CI doesn't. Part 2 should show its value within about 4 runs, or the cadence drops.

Open questions

  • Should part 2 be weekly, or follow gate changes only?
  • Do control PRs count against the 20-runner pool at peak hours, or run off-peak?
  • Can controls for owner-only workflows (shepherd-*.yml) run here, or do they need separate approval?
主要言語
Python
スター
57
フォーク
11
平均マージ
1時間 35分
マージ済み PR（30日）
975

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

squid-protocol/gitgalaxy のほかの issue

squid-protocol/gitgalaxy の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。