CI flakiness inventory: Oct 16-27, 2026 (follow-up to #13168)

Read the linked build evidence and the previous inventory, #13168, then compare the documented signatures and fixes against the October 16–27 pipeline snapshot. Done means the issue provides a reconciled inventory of affected builds, persistent and newly diagnosed failures, and the status of fixes; it explicitly notes that the bounded-copy repair has no completed post-merge CI evidence yet.

Android framework version

Other: net12.0-android on main; net11.0-android on release/11.0.1xx. The matrix also exercises stable net10.0-android test flavors.

Affected platform version

dotnet/android public Azure DevOps dotnet-android pipeline (dnceng-public, project public, definition 333), macOS/Linux/Windows. PR builds queued 2026-10-16 07:18:00 through 2026-10-27 13:10:20 UTC. Results below preserve the investigation snapshot; later retries/completions can change live build results.

Description

Follow-up to #13168, the October 2-16 inventory, which followed #12929 and #12704. This issue supersedes #13168 as the current tracking snapshot. Closing the predecessor does not mean its unresolved flaky-test or infrastructure problems are fixed.

The macOS mutex fix is holding, but CI has not improved overall. Retries nearly doubled and on-device test-process crashes became the dominant failure family. Some old problems are quieter; newly diagnosed test failures and reduced MAUI coverage make headline results less reassuring.

Population, methodology and trend

The current census contains 113 retained PR builds across 87 PRs: 104 completed (57 succeeded / 47 failed) and nine in progress. A query including deleted records found 50 additional builds, all canceled; they do not alter the completed-build denominator. The 104 completed builds published 5,754 test runs, 1,046,236 test executions and 115 failed-result entries. Detailed failed-result totals reconcile with every build's run summaries. Collection and enrichment completed without outstanding errors.

Metric	Oct 2-16 (#13168)	Oct 16-27
Completed, non-canceled PR builds	303	104
Final red builds, all causes	129 (42.6%)	47 (45.2%)
Builds with stage retries	49 (16.2%)	33 (31.7%)
Confirmed catalogued test/infra incidents	77 (25.4%)	42 (40.4%)
Builds with explicit test-process crashes, separately measured	6 (2.0%)	36 (34.6%)

Metric definitions and comparison limits:

• Confirmed incident incidence is a lower bound of distinct completed builds containing at least one verified catalogued test or infrastructure signature. It includes recovered green builds and mixed-cause red builds. It is not "red solely due to flakes" or a regression-excluded first-attempt rate. Signature categories overlap and must not be added as unique builds.
• The current catalog adds newly verified signatures. Freezing the previous catalog gives 22/104 = 21.2%, versus the previously published 77/303 = 25.4%. Excluding the old mutex outage, the previous rate was 60/303 = 19.8%. The older problem set is not materially better once that resolved outage is separated.
• The crash row applies the same explicit task-log crash signature retrospectively to both periods. It is separate from the confirmed-incident catalog: not every crash is proven flaky, and they do not all have one cause. It overlaps the other rows.
• 20 of 42 confirmed-incident builds eventually went green, as did 15 of 36 crash-affected builds. These overlap; do not add them. Final green status conceals recovered/non-gating incidents.
• Periods differ in duration, traffic, branch/runtime mix and exposure. The nine current running builds are excluded from completed ratios. These are descriptive build-level rates, not independent-trial causal estimates.
• MAUI coverage dropped sharply, as detailed below. Zero Helix failures cannot be interpreted as a fixed device lab.

The modest final-red change understates the recovery burden. Conversely, the apparent drop from 7,240 to 115 published failed entries is not proof of improved reliability: the older count included a huge PR-correlated burst, while current crashing runs explicitly report missing/incomplete TRX output.

Dominant current problem: on-device test-process crashes

83 crashing task executions across 36 completed builds, compared with 15 crashing task logs across six completed builds in the earlier snapshot. The current active-test groups include:

Active test when the crash was reported	Completed builds	Interpretation
UploadConnectionResetThrowsHttpRequestException(ResponseContentRead)	27	A major cross-PR crash family; one concrete Java-heap cause is diagnosed below.
UnreferencedInstanceIsCollected	20	A crash reported here is not the same as the collection assertion in #13275.
ConcurrentFirstUsePublishesSingleFieldAndStaticMethodCaches	8	Preserve as a separate lead; do not presume the active method caused every crash.

Groups overlap by build. There is also an early SharedPreferences/bitmap OOM/crash case. The active test is not necessarily the origin of the failure.

A concrete root was verified from the NativeAOT logcat in build 1648837:

1. An upload-reset test attempts a 16,777,232-byte Java allocation against a 16,777,216-byte heap growth limit.
2. The allocation throws OutOfMemoryError.
3. A later JNI FindClass encounters the pending exception and aborts with SIGABRT.
4. The adapter reports the crash while UnreferencedInstanceIsCollected is active, then reports that the Android adapter lacks ITrxReportCapability, leaving missing/incomplete TRX results.

The upload-reset tests were introduced near the end of the earlier window by MERGED #13125, so this period has substantially more exposure. Do not equate this new fixture failure with a sudden worsening of every underlying infrastructure service.

The bounded-copy test repair landed in MERGED #13110 on October 27 at 11:07:51 UTC. It prevents MemoryStream.CopyToAsync() from marshaling the entire 16 MiB backing array in one write while preserving the full payload and assertions. The standalone extraction #13270 CLOSED UNMERGED because the parent landed, not because the repair was abandoned.

There are no completed post-merge builds in this snapshot. The PR records prior local low-heap validation, but CI recovery is not yet demonstrated. Do not attribute all 36 crashing builds to this one cause: concurrent-first-use crashes and early Java heap exhaustion still need differentiation.

New or newly dominant exact signatures

Signature	Current completed builds	Evidence / status
ResolveSdkTiming two-second guard	13, 17 failed entries	Six builds have verified same-configuration fail-to-pass evidence; other cases repeatedly measure about 3.4 seconds against a two-second limit. MERGED #13265 removes the unreliable timing assertion while preserving correctness/output checks. Only three completed builds contain the merge; this is removing an unsuitable correctness gate, not evidence that SDK resolution became faster. Example 1648837.
Upload-reset OperationCanceledException results	6, plus one running build excluded from the count	Two builds recovered the same test/configuration: 1640755, 1644515. This exact result already appeared in one build at the end of the old window; it is newly catalogued, not necessarily newly introduced here. Keep cancellation results distinct from proven OOM/SIGABRT and verify the effect of #13110.
UnreferencedInstanceIsCollected collection assertion	1	1643260 failed then passed the same configuration. Exact tracker OPEN #13275 has two verified builds, but its other build queued before this window. These are assertion failures, not the process-crash outcomes reported under the same test name.
Gradle AGP plugin resolution	2	1640664, 1644966: plugin com.android.application:9.1.1 could not resolve from the configured repository; both passed the same configuration later. Distinct from Gradle distribution-download failure.
Emulator package service unavailable	1	1640294: actual ADB0010 installation failure, cmd: Can't find service: package. Intentional device offline inputs in host unit tests are not counted as incidents.

The exact collection assertion and Dispose_Finalized failures below occur after the earlier finalizer-test stabilization in dotnet/java-interop#1408. Same-SHA recovery establishes nondeterminism, not correctness of runtime/reference lifetime behavior.

Fixes that are holding, newly landed, or scoped hardening

Change	Current assessment
MERGED #13136, valid TMPDIR	Holding: zero exact mutex recurrences in 81 additional completed main-targeted builds ancestry-verified to contain the fix, versus 17 affected completed builds previously. This is the clearest sustained improvement.
MERGED #13265, remove SDK timing gate	Merged October 26. Removes the assertion responsible for 13 affected builds above. Small post-fix cohort; no claim of improved SDK performance.
MERGED #13110, bounded upload-test copying	Diagnosed fixture repair merged October 27. No completed post-merge cohort yet; verify all affected configurations and release-branch exposure.
MERGED #13228, preserve Android run name	Stops MAUI from overwriting the parent Android build number. A correction to misleading run names, not a device-lab recovery fix.
MERGED #13172, OpenJDK verification; MERGED #13204, SDK archive checksum enforcement	Useful download/install safety hardening. Do not credit these with solving Gradle corruption, network transport, or Helix UID failures.

Persistent problems, including post-fix recurrences

Problem	Previous / current completed builds	Evidence and assessment
JavaObjectTest.Dispose_Finalized	4 / 7	Four exact same-configuration recoveries: 1638624, 1640187, 1645848, 1646356. Exact tracker #10358 remains relevant. More affected builds despite the smaller population.
TryFindClass GREF delta	2 / 1	Important post-fix recurrence: 1646300, on #13244, contains both #13014 and #13068 but reports Global=20 to 21, Delta=1. The PR only removes region directives in PermissionAttribute.Partial.cs. Unlike the old inventory's release/stale-base cases, this cannot be dismissed as lacking the measurement fix. No same-build recovery was found. Real retention versus measurement instability remains unresolved under #12031/#13010; do not relax the real leak assertion.
Incremental-install timing	3 / 3	Same-configuration recoveries in 1641777 and 1646270. No demonstrated root fix.
BuildBasicApplicationThenMoveIt sharing violation	8 / 2	1641831, 1642138; the latter recovered the same configuration. Quieter, still present, no new corrective PR identified.
Gradle corrupt ZIP	2 / 1	1642138: zip END header not found, then same-config pass. Together with the two plugin-resolution cases above, all three current GradleFBProj cases recovered unchanged. #12707 state isolation and #12902 distribution-download retry have not eliminated the broader family; do not treat all three roots as the same wrapper-download problem. #12719 remains closed unmerged.
Git checkout/submodule transport	8 / 6	Plus one current in-progress build excluded from completed counts. Includes DNS and RPC/submodule transport failures. Example 1645418. Lower raw count but a higher proportion of the smaller population.
macOS PowerShell procargs	5 / 2	1642228, 1644271. #12652 remains open.
Hosted-agent disconnects	8 / 2	1642228, 1650443.
SDK/artifact download DNS	4 / 2	Actual SDK installation-task failures in 1638624, 1639760, not telemetry-only warnings.

Helix: zero failures with substantially less coverage is not a fix

There were zero observed Helix UID/install failures, compared with seven previously. However, the latest-attempt Run MAUI R2R APKs on Helix task executed in only 15 completed builds, versus 205 previously. All 15 current executions were on release/11.0.1xx; none were on main.

Main's MAUI Integration job is explicitly disabled with condition: 'false' during the .NET 12 transition until MAUI's matching branch exists, tracked by OPEN #13088. The disabling landed with #13076 near the end of the previous window. Stage-level success alone conceals the skipped integration job.

The device-lab problem under #12658 must remain unresolved. Previous recovery proposals #12666/#12715 closed unmerged. The disappearance of failures partly reflects lost exposure, not demonstrated healthier devices. Restore coverage independently of the test-stabilization work.

Quiet, excluded, or still unknown

Not observed, without claiming permanent resolution: the previous APK-comparison missing-libxamarin-app.so crash, RuntimeExceptionLookupFailureReleasesInput cleanup-assertion burst, GetObjectArray existing-peer mismatch, cache/results HTTP 429, and SDK-directory-removal MSB3231. The retained-GREF Delta=99 positive controls remain absent as intended after their earlier removal in #13068; that was removal of unstable control coverage, not a runtime leak fix.

Not automatically flakes: five builds hit job-cap timeouts; two DotNetRunWithDeviceParameter missing-message failures lack same-config recovery; other missing SDK/runtime-package, generated-contract, tooling-input, NativeAOT packaging and device-shard failures need their own root classification. PR-correlated failures are not charged to infrastructure just because another configuration passed.

Warning-only noise is separate: 31 completed builds had test-run-summary retry warnings and 30 had telemetry publication warnings. These counts are not added as gating outages or confirmed incidents.

Follow-up priorities

• Verify the newly merged #13110 upload-memory repair across affected configurations and release-branch exposure; separate remaining crash signatures rather than calling every active test flaky.
• Investigate finalization/reference reliability: #10358, the exact collection assertion in #13275, and the post-#13014 GREF failure in build 1646300 under #12031/#13010. Preserve real leak/collection checks.
• Restore main MAUI Integration under #13088; keep #12658 unresolved until meaningful device-lab exposure and evidence support closure.
• Monitor #13265 with correctness assertions retained; do not replace the removed timing gate with an arbitrary higher timeout.
• Diagnose Gradle plugin-resolution and corrupt-cache cases separately from distribution-download retry; retain exact same-configuration recovery evidence.
• Continue file-lock, install-timing, agent, checkout and download work. Preserve a consistent build-level metric, recovered green builds, and matrix/exposure accounting for the next inventory.

Steps to Reproduce

1. Enumerate public Azure pipeline definition 333 builds through https://dev.azure.com/dnceng-public/public/_apis/build/builds, using queue-time bounds 2026-10-16T07:18:00Z through 2026-10-27T13:10:20Z, queryOrder=queueTimeDescending, all continuation pages, and PR-source filtering. Query deletedFilter=includeDeleted separately to quantify missing/deleted records. Preserve the observed snapshot before later retries/completions change it.
2. Fetch each root timeline and follow previous-attempt timelines. Inspect actual failed task issues/logs, including recovered builds, rather than relying on GitHub aggregate checks or the latest attempt alone.
3. Collect test-run summaries and detailed per-run failed results, reconciling counts for every build. Do not rely on ResultsByBuild pagination for large bursts: it can repeat the first 1,000 rows despite $skip.
4. Match exact test identity, assembly/storage, configuration and source SHA to establish fail-to-pass recovery. Inspect attached Gradle build logs, device-output logs and native logcat; green builds and sibling configurations alone do not prove a flaky method.
5. Search task logs for explicit The test process crashed: and keep these outcomes separate from assertion failures and published TRX counts. Apply the same crash scan to the preserved earlier census. For build 1648837, inspect the NativeAOT logcat in the Test Results - APKs .NET Release - macOS 2 artifact to trace the upload allocation failure through the pending JNI exception and abort.
6. Verify fix ancestry against the build's actual source SHA, not queue date versus merge date. In particular, build 1646300 contains #13014/#13068; 81 additional completed main-targeted builds contain #13136. Do not confuse release/stale bases with fixed main or assume the newly merged upload repair already has a completed CI cohort.
7. Count each signature once per completed build and union catalogued incidents. Report process crashes separately without presuming a common cause. Preserve a frozen previous-catalog comparison when adding new signatures.
8. Check job/task execution exposure, not just stage result: main's MAUI stage can appear successful while the integration job is disabled. Compare executed Helix tasks before interpreting zero failures as improvement.

Did you find any workaround?

A targeted rerun may unblock a confirmed transient network/agent/device incident or a verified intermittent test, but repeated reruns hide the cost and do not establish correctness. Do not blanket-retry deterministic PR regressions, unknown crashes or every generic test failure.

Use a revision containing #13136 for the old mutex failure. #13265 removes the unsuitable SDK-resolution timing gate while preserving correctness checks. #13110 now contains the bounded upload-copy repair, but its CI effectiveness still needs a completed post-merge cohort; do not treat raising Java heap limits or disabling assertions as an established general fix.

There is no validated blanket workaround for finalizer/GREF assertions, remaining crash families, file locks or Gradle plugin/cache failures. MAUI integration's temporary skip is a coverage gap, not a device-lab workaround to count as a fix.

Relevant log output

Task should not take more than 2 seconds to run.
Expected: less than or equal to 00:00:02
But was: 00:00:03.4285870

Expected JavaDisposedObject.Dispose(disposing: false) to run.
Expected the unreferenced instance to be collected.

Operation should not leak global references after 100 iterations.
Before=(Generation=4, Global=20, Weak=0)
After=(Generation=6, Global=21, Weak=0), Delta=1

java.lang.OutOfMemoryError: Failed to allocate a 16777232 byte allocation ... growth limit 16777216
JNI DETECTED ERROR IN APPLICATION: JNI FindClass called with pending exception java.lang.OutOfMemoryError
Fatal signal 6 (SIGABRT)
The test process crashed: Process crashed.
The test framework 'Android Instrumentation Test Adapter' ... does not support the 'ITrxReportCapability' leading to missing or incomplete TRX reports

Plugin [id: 'com.android.application', version: '9.1.1', apply: false] was not found
java.util.zip.ZipException: zip END header not found

Excerpts identify distinct signatures, not one combined failure. Linked Azure builds and their artifacts contain the full logs.