Coverage collection is unreliable, and blocks the coverlet 10 upgrade

Read eng/Test.targets and the listed test project files to trace coverage collection, output paths, and thresholds; compare the failing workflow runs across all three target frameworks. Investigate the collector and coverlet 10 branch-counting behavior, then verify that coverage is reproducible, the #6 update can be merged or has a recorded reason, and no threshold was lowered for a collection defect.

Summary

How we collect coverage is unreliable, and it is now blocking a dependency update. There are two symptoms with one underlying subject — how branch and line coverage are gathered and counted — so they are worth solving together rather than separately.

Neither symptom is a real loss of test coverage. In every case below the tests themselves passed.

Symptom 1 — coverlet 6.0.4 drops coverage data non-deterministically

coverlet.msbuild 6.0.4 intermittently reports coverage below threshold, or fails outright, on one target framework while the other two report the correct figure for the same tests on the same commit. It has surfaced four times in the last week, across three different test projects and both Linux and Windows:

Every one cleared on gh run rerun <id> --failed with no code change.

The last row is the concerning one. The earlier failures announced themselves as obviously bogus — 0%, or a stream exception. 99.95% does not. It reads exactly like somebody genuinely missed a line, and the only thing distinguishing it from a real regression is that two other frameworks reported 100% for identical tests. That is a subtle tell to rely on, and it is the sort of thing that eventually gets "fixed" by quietly lowering a threshold.

It is also not free: it failed the 9.0.0 release build and needed a manual re-run before publish-nuget would go.

Symptom 2 — coverlet 10.0.1 counts branches differently, so we cannot take the update

#6 bumps coverlet.msbuild from 6.0.4 to 10.0.1 and fails:

error : The minimum branch coverage is below the specified 100
        [test/Paramore.Fences.Core.Tests::TargetFramework=net10.0]

This one is not a flake — it reproduces, and it is a real behavioural change: coverlet 10 identifies branches the 6.x line did not. So we are pinned to a version that flakes, and the upgrade that might fix the flakiness is gated behind deciding what our branch-coverage numbers should actually be.

Why this is one piece of work

Taking #6 means answering "is 100% branch coverage on Paramore.Fences.Core still the right gate, under a tool that counts branches differently?" That is a coverage-policy question, not a dependency bump, and it should not be settled as a side effect of merging a Dependabot PR.

Where the configuration lives

• eng/Test.targets — CollectCoverage, CoverletOutputFormat, ExcludeByAttribute, ReportGenerator wiring
• test/Paramore.Fences.Core.Tests/*.csproj, Extensions.Tests, RateLimiting.Tests, Testing.Tests — <Threshold>100</Threshold>
• test/Paramore.Fences.Specs/*.csproj — <Threshold>94,94,91</Threshold>

Worth investigating

Not proposals, just the threads that look most promising:

• CollectCoverage=true uses coverlet's MSBuild in-process collector. The coverlet.collector datacollector (--collect:"XPlat Code Coverage") is generally the more robust path, and the in-process collector writing per-framework output during a multi-targeted build is a plausible source of the truncated/partial results above. Switching how we pick up coverage may make symptom 1 disappear regardless of version.
• Whether the three target frameworks are racing over shared coverage output paths.
• What coverlet 10 actually counts that 6.x did not on Core.Tests, and whether those branches deserve tests or an exclusion.
• Whether a hard 100 threshold is the right instrument, given it makes any collection wobble a build failure.

Acceptance

• Coverage figures are reproducible across all three target frameworks for the same commit.
• #6 can be merged, or closed with a recorded reason.
• No threshold is lowered merely to accommodate a collection defect.