Pipeline controls: planted defects prove each gate can still go red

Abierto
  • ¿Algo no está bien?
#7,155 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
25/100
Tipo de issue
Nueva funcionalidad
Claridad
Necesita aclaración
Estado de actividad
Activo
Stack tecnológico
github-actions, python, yaml
Área
ci-cd, testing, tooling

Línea de trabajo

The issue mentions det_mutation.py and mutation.py and the CI rules in .claude/rules/ci-workflows.md. First read those and enumerate gates from shepherd-*.yml and other workflows. For the cheapest first step, add per-gate fixtures that must fail and a suite check requiring them; a gate with no fixture fails. Full completion spans control PRs and review experiments needing deep CI familiarity.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

enhancement priority: high scope:infra

Why

A green gate only means something if that gate can go red. Three failures from one week looked healthy until someone checked:

  • CodeQL: every PR read "No new alerts" while PR scans held 0 results, and open alerts went from 34 to 459 (#7153).
  • ci_busy_hours: read an impossible 45 h on Oct 23, and nothing noticed.
  • promoter-bot #89: passed every test, yet would have stopped every host's dashboard push once factory.json passed 128 KB.

We test code heavily, but we don't test whether the tests and gates still work. det_mutation.py and mutation.py measure whether tests catch code changes. Nothing measures whether a bad PR is actually stopped.

The idea

Run lab-style controls on the pipeline itself:

  • Negative control: a known-bad change. Its gate must go red. If it stays green, that gate is blind, which matters more than any single bug.
  • Positive control: a known-clean change. Everything must stay green. This measures false alarms and fix-loop cost.

The output is a detection matrix: each gate against each planted defect, with pass or miss and how long it took. A miss is treated as a high-priority failure of the gate, not of the code.

Plan, cheapest first

1. Gate self-tests (no extra CI runs)
  • List every gate and what it claims to catch
  • Give each gate at least one fixture it must fail on, and run these in the existing suite
  • Add a check that fails when a gate has no fixture
2. Weekly control PR (one full CI run a week)
  • A scheduled job opens a real, ready-for-review PR from a canary/ branch, carrying one small planted defect per gate
  • A results job compares which gates went red with which should have, posts the matrix, then closes the PR and deletes the branch
  • Also run a clean control PR, to measure false alarms
  • A miss opens a priority: high issue
3. Controls on the review layer (an experiment, not a default)

Gates catch rules. Reviewers catch judgement. The #5274 import cycles passed every gate, and a reviewer was the only layer that could have questioned them.

  • Hide planted defects in realistic diffs, such as a swallowed exception, a test that can't fail, or a green-by-skipping check
  • Measure catch rate and false alarms for the Sonnet review, and compare with Opus on the same set
  • The reviewer must not know a diff is a control. Run through the experiment lane: Opus design, owner OK, logged in temporal-crucible, cost-vs-reward gate first
4. Learn from escapes
  • Every defect that reaches main adds a new control, so the set grows from real failures
  • A monthly Opus session reviews misses, false alarms and escapes, and proposes new control classes
  • Add the detection matrix to the squid-bot dashboard

Safety rules

  • A control PR can never merge: canary/ branches carry a hold that ci-gate always fails, whatever else is green, and the merge label is refused
  • Planted defects are inert: no real vulnerabilities, secrets, network calls, or changes to workflows or release paths
  • Control runs stay out of the factory's own numbers: merges, first-pass rate, lessons and agent memory
  • A control PR that fails to close alerts the owner

How to judge it

  • Cost to measure first: the runner-minutes of one control PR (.claude/rules/ci-workflows.md) and the tokens for a review session.
  • Worth continuing if it finds gaps that weekly CI doesn't. Part 2 should show its value within about 4 runs, or the cadence drops.

Open questions

  • Should part 2 be weekly, or follow gate changes only?
  • Do control PRs count against the 20-runner pool at peak hours, or run off-peak?
  • Can controls for owner-only workflows (shepherd-*.yml) run here, or do they need separate approval?
Lenguaje dominante
Python
Estrellas
57
Forks
11
Merge medio
1 h 35 min
PR fusionados (30 d)
975

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de squid-protocol/gitgalaxy

Todos los issues de squid-protocol/gitgalaxy

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.