CI structure amplifies flakes: fail-fast matrix + test-complete mirror check
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 72/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Quiet
- Tech stack
- github-actions
- Domain
- ci-cd
Research direction
Start in .github/workflows/test-pr.yaml at the test matrix around line 34 and the test-complete gate around lines 211-218. Review how matrix results are propagated, then verify that one failed or flaky language leg no longer cancels sibling legs and that the aggregate status remains understandable during triage.
Written by the indexing model from the issue text.
Description
What happens
Two aspects of the fixture-test workflow (.github/workflows/test-pr.yaml) inflate the blast radius of every flaky job and hide the results of unrelated languages.
-
fail-fast: trueon the fixture matrix. Thetestjob's matrix (strategy.fail-fast: true, line 34) runs every language fixture as a matrix leg. When any one leg fails — including a pure infra flake — GitHub cancels all sibling legs. A single flaky job therefore turns into a wall of "cancelled" jobs, and we lose the signal of whether the other languages would have passed. -
test-completeonly mirrors the matrix. Thetest-completejob (line 211) runsif: cancelled() || contains(needs.*.result, 'cancelled') || contains(needs.*.result, 'failure')and simply prints "Some workflows have failed!" andexit 1. It never fails independently — it only reflects whatever the matrix did. During triage its failure carries no information beyond "some matrix leg failed or was cancelled."
Evidence
.github/workflows/test-pr.yamlline 34:fail-fast: trueon thetestmatrix..github/workflows/test-pr.yamllines 211-218:test-completegate that mirrors the matrix result.- Today's triage of 31 failing PRs: several PRs showed many "cancelled" sibling jobs stemming from a single flaky leg, and
test-complete"Some workflows have failed!" was never an independent failure — it only mirrored whichever matrix job failed.
Impact
- One flaky job (Scala Bloop timeout, Elm warmup deadlock, cjson download reset) cancels the entire language matrix.
- Triage is harder: the real failing leg is buried among cancellations, and passing languages never report.
- The
test-completered X is noise on its own; you must open the matrix to learn anything.
Suggested solutions
- Set
fail-fast: falseon the fixture matrix so each language leg runs to completion independently and one flake stops masking the rest. - During triage, treat
test-completeas a mirror only — inspect the individual matrix legs, not the aggregate gate.
Related flaky areas
These are the specific flakes this structure amplifies:
- #3063 — scala3-upickle: Bloop startup timeout + maven-nightlies download failures
- #3064 — elm: Warmup.elm setup races the compiler (file locks / MVar deadlock)
- #3065 — cjson: cJSON.c downloaded from raw.githubusercontent.com at test time
Interim policy
Per CLAUDE.md ("Known CI flakiness"), we accept these flakes for now and retry the failed jobs (gh run rerun <run-id> --failed) when they occur.
- Dominant language
- TypeScript
- Stars
- 13.9k
- Forks
- 1.2k
- Avg merge
- 8h 14m
- Merged PRs (30d)
- 368
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from glideapps/quicktype
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
-
Difficulty 5/5 Over a week Newbie friendliness 45/100
-
[BUG]: Openbug
Difficulty 4/5 3-5 days Newbie friendliness 35/100
-
Difficulty 3/5 1-2 days Newbie friendliness 58/100
-
bug
Difficulty 4/5 3-5 days Newbie friendliness 48/100
All issues in glideapps/quicktype
Similar issues
-
calcite-components needs triage refactor
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Esri/calcite-design-system#15203 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 91/100
-
community first-timers-only good first issue hacktoberfest help wanted low hanging fruit up-for-grabs
Difficulty 1/5 Under an hour Newbie friendliness 95/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Automattic/studio#4908 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 90/100