CI flakiness inventory: Sep 19-26, 2026 (follow-up to #12704)

Start by reading the unresolved-signature evidence and the linked trackers, especially #12658, #12652, #12916, and #12927. Inspect the cited build logs, dotnet-run-device-output.log, and the publish binlog where applicable; done means separating correlated failures from independent flakes and recording evidence without treating every named test as a distinct issue.

Android framework version

net11.0-android (Preview); the CI matrix also exercises net10.0-android test flavors.

Affected platform version

dotnet/android public Azure DevOps dotnet-android pipeline (dnceng-public, definition 333), macOS/Linux/Windows. This follow-up covers builds queued 2026-09-19 21:25:43 through 2026-09-26 21:25:43 UTC.

Description

Follow-up to the closed previous flaky-CI inventory #12704. That issue covered PR builds from the earlier interval; this inventory includes PR, scheduled main, and manual builds, so its percentages are not directly comparable with #12704.

Population at the cutoff: 241 builds (232 PR, 7 scheduled, 2 manual): 94 succeeded, 145 failed, 1 canceled, 1 in progress. The in-progress build 1613531 was canceled after the cutoff. The 239 completed, non-canceled builds published 18,191 test runs and 2,754,737 test executions, including 4,387 failed-result entries spanning 550 automated test names. Nine green builds still published 27 non-gating failed results; 65 red builds had no published failed-test result. These are not 550 independently flaky tests: #12903 alone accounts for 2,766 failed-result entries (63.0%), and #12903/#12887/#12889 together account for 3,848 (87.7%), much of it correlated with changes on those branches.

Counts below are distinct affected builds per signature, unless explicitly called results or tasks. One build may have several causes; warning counts, test-result counts, and red-build counts must not be added together. An otherwise-green build with a failed test is a lead, not by itself proof that the case passed on retry. Source builds and the prior issue are linked for reproducibility.

Ongoing or resurfaced since #12704

New to this inventory, or newly specific and still unresolved

Fixed or mitigated signatures: distinguish the exact cause from the test name

Unclassified/excluded rather than declared fixed: FastTimingTests.ConcurrentEventsCanGrowAndDump still appears on code-changing PR builds after MERGED #12492; its current error may differ from the old logcat race and needs logs. The 21 generated-Java cannot find symbol build failures on the still-open #12890–#12895 stack are product/stack failures, not infrastructure flakes. A previous BuildArtifactsOutputPaths test fix #11839 CLOSED unmerged; this week's cases fail during build, so that earlier missing-output-path fix cannot be assumed sufficient. OPEN #12926 improves classification and targeted retries, but does not itself fix any root cause above.

Proposed follow-up

• Re-evaluate #12650/#10973 against build 1605734 on #12851; preserve the distinction between PR regression and recurring flake.
• Rework #12767's atomic SDK replacement with the review issues addressed; follow #12652 for the pwsh host failures.
• Continue #12658's device-lab work and evaluate OPEN #12666 XHarness recovery. The four other Helix work-item failures, all in build 1605352, are app crashes from missing jni_remapping_type_replacement_count; their correction is already in OPEN #12847 and must not be charged to device-lab UID failures.
• Investigate the #12916/#12927 Windows testhost hang and the Ref.37 assertion using task logs/binlogs before changing timeout limits or workload installation.
• Revisit #12031's leak-test isolation after the two post-#12709 failures; update #10832 with the new DotNetRunWithDeviceParameter output logs. Consider only classified targeted retries for confirmed external errors.

Steps to Reproduce

1. Enumerate public dotnet-android definition 333 builds queued in the exact UTC window above, recording each build's status at the cutoff. Include scheduled main and manual runs as well as PRs; canceled build 1613531 may no longer appear in a later default build-list query.
2. For each build, read its Azure timeline, failed/canceled job and task issues, and the root task log—not just the GitHub aggregate check or failed-test API.
3. Query ResultsByBuild?outcomes=Failed, the build's test runs, and the failed run's exact result title/error. Compare repeated results in the same configuration and unchanged-source reruns; green results can contain non-gating failed attempts.
4. Compare the exact error signature against #12704 and the linked fix PR, not merely the test method name. Check changed files and stacked PR bases before attributing correlated test failures to infrastructure.
5. Reproduce specific examples using the linked Azure builds. For device crashes or hangs, inspect the matching published logcat/Helix work-item artifacts; for generic FailedBuildException, inspect the attached build log/binlog before selecting a retry or a code fix.

Did you find any workaround?

A targeted rerun of the affected job/stage can unblock a confirmed transient DNS, hosted-agent, or device-lab failure, but it must not treat deterministic PR regressions, all Helix work-item failures, or all generic test failures as flaky. #12926 is an OPEN proposal for classified, confirmation-gated targeted retries, not a landed root-cause fix. #12705's CLOSED unmerged bounded cache-task retry is a small mitigation for HTTP 429. For SDK cleanup, three task attempts already failed: the unmerged atomic-replacement design in #12767 needs correction rather than more retries. For Helix UID failures, #12666 proposes XHarness recovery while device-lab cleanup is investigated.

Relevant log output

Call to 'procargs' failed with errno 5
error MSB3231: Unable to remove directory ".../platforms/android-36.1". Directory not empty: '.../data/res'
DownloadPipelineArtifact: nodename nor servname provided, or not known (dev.azure.com:443)
GetObjectArray: Expected existing Context peer, got Android.AppTests.App

Full task logs, case-level errors, and retries are linked to representative builds in the Description; the lines above are only short identifying signatures.