CI flakiness inventory: Oct 2-16, 2026

This is a CI flakiness census, not a code change. Start from the signature table and the Azure DevOps pipeline definition 333 it cites, then read the linked PRs #13136 (TMPDIR for macOS mutex failures) and #13068 (retained-GREF controls) to see what each fix covers. The issue sets no single deliverable, so any useful first step needs a maintainer to define the scope first.

Android framework version

net11.0-android (Preview); the CI matrix also exercises net10.0-android test flavors.

Affected platform version

dotnet/android public Azure DevOps dotnet-android pipeline (dnceng-public, project public, definition 333), macOS/Linux/Windows. PR builds queued 2026-10-02 07:18:00 through 2026-10-16 07:18:00 UTC.

Description

Follow-up to #12929, the September 19-26 inventory, which followed #12704. This inventory supersedes #12929 as the current tracking snapshot; closing the predecessor does not mean its unresolved problems are fixed.

Observed flake/infrastructure incidence worsened within this fortnight, despite a slightly lower final red-build rate. A new macOS startup outage accounts for much, but not all, of the increase. Several scoped fixes appear to be holding, while recurring agent, checkout, file-lock, device-lab and finalizer failures remain.

Scope and trend

The census contains 305 PR builds across 157 PRs: 303 completed, non-canceled builds (174 succeeded / 129 failed), one canceled and one in progress. Published data contains 16,914 test runs, 2,834,814 test executions and 7,240 failed-result entries spanning 558 automated test names. These are not 558 independently flaky tests.

The metric below is a lower-bound distinct-build incidence: a completed build containing at least one confirmed flaky-test or infrastructure signature. It includes recovered green builds and mixed-cause red builds. It is not a count of builds red solely due to flakes, nor a regression-excluded first-attempt rate. Signature counts overlap and must not be added as unique builds.

Metric	Oct 2-9	Oct 9-16	Total
Completed, non-canceled PR builds	124	179	303
Confirmed flake/infra-affected builds	20 (16.1%)	57 (31.8%)	77 (25.4%)
Affected red builds, possibly mixed-cause	13	38	51
Affected green builds	7	19	26
Final red builds, all causes	55 (44.4%)	74 (41.3%)	129 (42.6%)
Builds with stage retries	16 (12.9%)	33 (18.4%)	49 (16.2%)
Affected builds excluding the new mutex outage	20 (16.1%)	40 (22.3%)	60 (19.8%)

The lower-bound incident rate roughly doubled. Even excluding the mutex outage, it increased from 16.1% to 22.3%. Final green status hides recovered incidents; it is not a sufficient reliability measure.

Historical comparison limits: #12929 included 241 mixed-trigger builds (232 PR, 7 scheduled, 2 manual), whereas this census is PR-only. #12704's 46/155 = 29.7% measured flake-red first attempts after excluding PR-regression sequences, a different denominator. Queries for September 18-October 2 returned scheduled builds but no older PR records, including with deletedFilter=includeDeleted; an equivalent earlier PR baseline is unavailable, not evidence of no prior CI activity. Matrix exposure also changed: #12973 removed Mono coverage, #13011 removed redundant CoreCLRTrimmable coverage, and #13068 removed unstable retained-GREF controls. These are descriptive build-level rates, not independent-trial causal estimates.

Most common confirmed problems

Counts are distinct completed builds per signature. Recovered and mixed-cause builds are included.

Signature	Builds (week 1 / week 2)	Evidence / assessment
macOS NuGet-Migrations mutex / empty TMPDIR	17 (0 / 17)	Plus one canceled investigation build. lstat("", ...) == ENOENT prevents normal execution. Example 1634008. #13136 appears to hold on builds actually containing it; see below.
Hosted-agent disconnect	8 (3 / 5)	Explicit "We stopped hearing from agent". Example 1632397. No corrective repository PR identified.
Git checkout/submodule transport	8 (2 / 6)	DNS, RPC/transport timeouts and submodule failures. Example 1633598.
BuildBasicApplicationThenMoveIt sharing violation	8 (2 / 6)	Exact IOException; three same-test/configuration fail-to-pass recoveries: 1623340, 1629376, 1630461. Other same-name build failures are not automatically classified as this flake.
Helix install / UID assignment	7 (3 / 4)	Eight failed work-item entries in seven builds. Remote consoles confirm DELETE_FAILED_INTERNAL_ERROR, then insufficient storage / inability to assign UID. Example 1632651. #12658 remains relevant.
macOS PowerShell procargs errno 5	5 (2 / 3)	Continuing #12652; two eventually-green builds. Example 1635746.
Azure cache/results HTTP 429	4 (1 / 3)	Includes cache post-job incidents; one build also hit results-upload throttling. Example 1637091.
SDK/artifact download DNS	4 (1 / 3)	Actual failed-task evidence, not telemetry-only DNS warnings. Example 1632547.
JavaObjectTest.Dispose_Finalized	4 (2 / 2)	Exact tracker #10358. Same Release configuration failed then passed in 1636541.
Incremental-install timing assertions	3 (2 / 1)	IncrementalFastDeployment("aab"); same-config recovery in 1628262.
Azure task-download timeout	2 (0 / 2)	PublishTestResults / Cache task downloads. Example 1633943.
Gradle corrupt ZIP	2 (0 / 2)	zip END header not found: 1633619, 1635155. First case recovered in the same configuration.
Gradle dependency connection reset	2 (1 / 1)	1623855, 1635133; both recovered unchanged in the same configuration.
DotNetRunWaitForExit missing expected output	2 (0 / 2)	Exact same-test/config recoveries in 1633594, 1633678.
TryFindClass GREF delta	2 (1 / 1)	Both recovered in the same configuration; neither build contains #13014. See branch/ancestry details below.
Retained-GREF positive control, Delta=99	2 (1 / 1)	Unstable exact-100 controls removed by #13068; the apparently later occurrence lacks that removal.
PingLocalhost timeout	1 (0 / 1)	Same-config recovery in 1633537.
GetObjectArray existing-peer mismatch	1 (0 / 1)	1633710 contains #12650. Stronger recurrence evidence than the PR-overlapping case in #12929, without proving the race mechanism is identical.

Fixes since the previous probe: holding, scoped, or not observed

Change	Current evidence and limitation
MERGED #13136, valid TMPDIR globally	Merged October 14, 12:57 UTC; sets TMPDIR=$(Agent.TempDirectory) for the .NET named-mutex startup failure exposed on shared macOS agents. Zero exact recurrences in 30 completed main-targeting builds verified to contain commit 347bedca60b8838f4c6d6ec414634e44a67c5f15. Apparently later failure 1635460 used a stale merge ref without the fix. Early evidence is holding; this is only about two days of exposure.
MERGED #13014, completed-GC-bridge accounting	Merged October 6. Adds bounded accounting to both samples while retaining the zero-growth threshold. The UTF-8 failure 1623091 preceded it. String failure 1633921, on #13128 / release/11.0.1xx, does not contain it. No observed real TryFindClass or JavaSideActivation leak-check failure on fixed main in this census; do not infer the release branch is fixed.
MERGED #13068, remove unstable retained-GREF positive controls	Merged October 12. Removes two controls asserting exact Delta=100, sometimes observed as 99; real leak checks remain. 1626120 already contained #13014, so measurement synchronization alone did not eliminate control instability. Later 1633269, on #13105's stale stacked base, does not contain #13068. This is removal of unstable positive-control coverage, not a runtime leak fix.
MERGED #12957, APK-size decreases inconclusive	Decreases no longer gate; actual increases and comparison-tool crashes remain failures. Current cases do not establish regression of that policy.
Earlier exact-signature fixes #12712, #12714, #12718, #12710, #12886 and #12515	Original cancellation-rewind, pre-TLS certificate, benchmark-stability, embedded-desugar, Android SDK corrupt-archive and old apkdiff-timeout signatures were not found recurring outside the broad PR-related failure burst. Same-named tests failing for other reasons are not evidence those fixes regressed. Absence is not proof of permanent resolution.
MERGED #12708, bounded/non-gating diagnostic logcat capture	Stronger containment than #12471/#12620. It addresses diagnostic capture, not every testhost hang or job-cap timeout.

Remaining problems despite attempted PRs

• Helix device-lab recovery: #12666 and #12715 are now both CLOSED UNMERGED. The UID/install path under #12658 remains. #12641 fixed a different fork-token HTTP 401 path; UID failures are not evidence that authentication fix regressed.
• Gradle family: there are six non-burst GradleFBProj builds: the four diagnosed cases above, one missing platforms;android-37.0 / repository-connectivity case (1623500), and one generic case without available attachments (1628440). MERGED #12707 state isolation is present in build 1635155; MERGED #12902 distribution-download retry is present in 1635133. The broader family persists, but dependency resets and corrupt artifacts are not automatically the original distribution-download problem. Recovery proposal #12719 is CLOSED UNMERGED. Gradle ZIP failures do not establish regression of #12886's Android SDK archive fix.
• Cache/network resilience: #12705 is CLOSED UNMERGED; HTTP 429 continues. No corrective repository PR was identified for #12652's PowerShell startup failures or the hosted-agent disconnects.
• JNI measurement: #12709's earlier isolation did not eliminate old-protocol lookup failures. However, current apparent recurrences do not contain the later #13014 fix. Trackers #12031 and #13010 remain relevant.
• Peer identity: 1633710, on #13124 / release/11.0.1xx, reports "Expected existing Context peer, got Android.AppTests.App" and contains #12650. #13124 changes obsolete-attribute diagnostics, not implicated peer-runtime code. The #10973 assertion remains observable after the prior fix; one recurrence does not prove the mechanism is unchanged.
• Finalization: #10358's exact JavaObjectTest.Dispose_Finalized signature persists. Three additional RegisteredPeer_Dispose_Finalized builds are kept separate from the confirmed metric because causality/recovery evidence is weaker. #13016's native control-block lifetime fix does not establish that all finalizer timing or GREF assertions are solved.
• SDK directory removal: zero exact current-window MSB3231 occurrences. #12767 atomic replacement was CLOSED UNMERGED; classify this as not observed, not fixed by a landed PR.
• Triage/retry tooling: #12926 improves diagnosis and targeted retries; it is not a root-cause fix for the problems above.

New or newly specific failures, with deterministic/unknown cases kept separate

1. macOS mutex outage: the largest new environmental failure; #13136 appears to hold on actual fixed builds as described above.
2. APK comparison crash: 32 failed entries in four builds on October 11: 1631312, 1631340, 1631357, 1631364. Exits 134 / -532462766. The attached apkdiff.log in 1631364 shows KeyNotFoundException for lib/arm64-v8a/libxamarin-app.so in apkdiff.ApkDescription.Compare. This is a deterministic layout/baseline/tool-compatibility problem, not demonstrated random flakiness, and is excluded from the confirmed metric. Other exit-code-3 cases include actual size increases such as libbinary_blobs.so (+20,112 bytes), not failures of #12957's decrease-only policy.
3. Transferred-reference cleanup assertions: 81 entries across seven builds report "The partially constructed wrapper's own copies must be disposed separately." Representative 1630724, 1630732, and 1631037 contain MERGED #13052. #13015, #13016 and #13052 fix distinct ownership/registration defects; do not declare all ownership problems solved or presume this is the same original defect. Repeated cross-configuration assertions are not established flakes and are excluded from the metric. #13010 remains relevant.
4. Newly separated intermittent symptoms: incremental-install timing and DotNetRunWaitForExit missing output have direct same-config recovery evidence, as tabulated above. A separate keyguard-command failure in 1637091 remains unclassified. Two SharedPreferencesTest.PutAndGetManyValues OOM cases need logcat and are excluded.
5. Timeouts: four job-cap timeout builds, including one 240-minute cap, remain separate from proven infrastructure roots; not all timeouts are flakes.

Avoid misleading rankings and counts

Build 1631604, dependency-update PR #13076, contributed 5,620 failed entries across 70 failed test runs, including repeated attempts: 77.6% of all failed entries. These broad PR-correlated failures are not 5,620 independently flaky tests and were excluded from the confirmed nuisance catalog. Other R8/NativeAOT development branches also contain correlated regressions.

There were 22 green builds with published failed-test results, but 26 green builds with confirmed test or infrastructure incidents: these measures differ because infrastructure failures can recover without publishing failed tests. Conversely, 55 red builds had no published failed-test result. Final status or failed-test APIs alone miss important incidents.

Warning-only test-run-summary retries in 74 builds and telemetry DNS/timeouts in 80 builds were not counted as gating outages or confirmed flakes. Broad candidate scans also contain intentional device offline test inputs; only actual root evidence is classified.

Follow-up

• Monitor #13136 on builds actually containing the fix; refresh stale merge refs before calling later mutex failures regressions.
• Address recurrent file-sharing failures in BuildBasicApplicationThenMoveIt, preserving same-config retry evidence.
• Continue #12658 device-lab cleanup/recovery work; previous recovery proposals did not merge.
• Investigate Gradle corrupt ZIPs and dependency-reset recovery separately from distribution-download retries.
• Follow #12652 and hosted-agent/checkout/download incidents with agent-pool/service evidence rather than blanket test retries.
• Revisit #10973/#12650 using build 1633710; track #10358 separately from ownership-lifetime fixes.
• Check release-branch exposure to #13014; do not relax real leak assertions or confuse removal of controls with fixing a leak.
• Investigate APK-layout/baseline compatibility and transferred-reference cleanup assertions as separate deterministic/unknown problems, not presumed flakes.
• Preserve a consistent PR-only, build-level incident metric, including recovered green builds, for the next probe.

Steps to Reproduce

1. Enumerate pipeline definition 333 via https://dev.azure.com/dnceng-public/public/_apis/build/builds, with reasonFilter=pullRequest, queue-time bounds 2026-10-02T07:18:00Z to 2026-10-16T07:18:00Z, and all continuation pages. Preserve results/status at the observation cutoff; later retries or retention can change the returned population.
2. Read each root build timeline, follow previousAttempts[].timelineId, and inspect failed task issues/logs. Do not infer cause from GitHub red-check counts.
3. Collect build test-run summaries and each failed run's detailed failed results. ResultsByBuild can repeat its first 1,000 rows despite $skip; use per-run retrieval for large failure bursts. The completed collection reconciled failed-result counts with every build's run summaries and had no collection/enrichment errors.
4. Identify the exact error and test storage/configuration. Confirm recoveries using the same test, assembly and configuration; an eventual green build alone is not proof that a test passed on retry. Inspect Helix remote consoles and Gradle/APK attachments where generic result errors are insufficient.
5. Compare source SHA ancestry against the relevant fix commit, not just queue time versus merge time. Separate release branches and stale stacked/merge bases from main containing the fix. Inspect the PR's changed files before classifying correlated failures as flakes.
6. Count each confirmed signature at most once per completed build, union signatures for affected-build incidence, and split at 2026-10-09T07:18:00Z. Exclude canceled/in-progress builds from completed ratios; retain recovered green and mixed-cause builds. The linked examples above provide reproducible source evidence.

Did you find any workaround?

For a confirmed transient agent/network/device-lab incident, a targeted rerun of the affected job/stage can unblock the PR. It is not a root-cause fix and should not mask deterministic regressions or generic failures whose logs have not been classified.

For the macOS mutex startup failure, use a revision containing #13136 and a valid TMPDIR; a queued-after-merge build can still use an older base. For GREF measurements, distinguish main containing #13014/#13068 from release/stale bases and preserve real zero-growth assertions. No blanket workaround is established for file-lock failures, finalization/peer-identity issues, APK comparison compatibility or transferred-reference cleanup assertions.

Relevant log output

System.IO.IOException ... 'NuGet-Migrations' ... lstat("", ...) == -1; errno == ENOENT
We stopped hearing from agent
Call to 'procargs' failed with errno 5
DELETE_FAILED_INTERNAL_ERROR
INSTALL_FAILED_INSUFFICIENT_STORAGE
java.util.zip.ZipException: zip END header not found
GetObjectArray: Expected existing Context peer, got Android.AppTests.App
KeyNotFoundException: ... 'lib/arm64-v8a/libxamarin-app.so' ...
The partially constructed wrapper's own copies must be disposed separately.

These are identifying excerpts; the linked Azure builds contain the full task logs and selected test/Helix attachments.