CI structure amplifies flakes: fail-fast matrix + test-complete mirror check

Open Beginner friendly
#3,066 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
72/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Quiet
Tech stack
github-actions
Domain
ci-cd

Research direction

Start in .github/workflows/test-pr.yaml at the test matrix around line 34 and the test-complete gate around lines 211-218. Review how matrix results are propagated, then verify that one failed or flaky language leg no longer cancels sibling legs and that the aggregate status remains understandable during triage.

Written by the indexing model from the issue text.

Description

CI

What happens

Two aspects of the fixture-test workflow (.github/workflows/test-pr.yaml) inflate the blast radius of every flaky job and hide the results of unrelated languages.

  1. fail-fast: true on the fixture matrix. The test job's matrix (strategy.fail-fast: true, line 34) runs every language fixture as a matrix leg. When any one leg fails — including a pure infra flake — GitHub cancels all sibling legs. A single flaky job therefore turns into a wall of "cancelled" jobs, and we lose the signal of whether the other languages would have passed.

  2. test-complete only mirrors the matrix. The test-complete job (line 211) runs if: cancelled() || contains(needs.*.result, 'cancelled') || contains(needs.*.result, 'failure') and simply prints "Some workflows have failed!" and exit 1. It never fails independently — it only reflects whatever the matrix did. During triage its failure carries no information beyond "some matrix leg failed or was cancelled."

Evidence

  • .github/workflows/test-pr.yaml line 34: fail-fast: true on the test matrix.
  • .github/workflows/test-pr.yaml lines 211-218: test-complete gate that mirrors the matrix result.
  • Today's triage of 31 failing PRs: several PRs showed many "cancelled" sibling jobs stemming from a single flaky leg, and test-complete "Some workflows have failed!" was never an independent failure — it only mirrored whichever matrix job failed.

Impact

  • One flaky job (Scala Bloop timeout, Elm warmup deadlock, cjson download reset) cancels the entire language matrix.
  • Triage is harder: the real failing leg is buried among cancellations, and passing languages never report.
  • The test-complete red X is noise on its own; you must open the matrix to learn anything.

Suggested solutions

  • Set fail-fast: false on the fixture matrix so each language leg runs to completion independently and one flake stops masking the rest.
  • During triage, treat test-complete as a mirror only — inspect the individual matrix legs, not the aggregate gate.

Related flaky areas

These are the specific flakes this structure amplifies:

  • #3063 — scala3-upickle: Bloop startup timeout + maven-nightlies download failures
  • #3064 — elm: Warmup.elm setup races the compiler (file locks / MVar deadlock)
  • #3065 — cjson: cJSON.c downloaded from raw.githubusercontent.com at test time

Interim policy

Per CLAUDE.md ("Known CI flakiness"), we accept these flakes for now and retry the failed jobs (gh run rerun <run-id> --failed) when they occur.

Dominant language
TypeScript
Stars
13.9k
Forks
1.2k
Avg merge
8h 14m
Merged PRs (30d)
368

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from glideapps/quicktype

All issues in glideapps/quicktype

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.