Microsoft.Testing.Extensions.CodeCoverage: line hits missing from cobertura output for passing tests when test process spawns a child dotnet process via Process.Start in Azure Dev Ops Pipeline

最小限の再現スケッチから始め、Microsoft.Testing.Extensions.CodeCoverage 18.6.2 を使用して、指定された dotnet test コマンドを実行し、Process.Start の子プロセスがある場合とない場合の Cobertura 出力を比較します。プロファイラー環境の継承と報告されたカバレッジ出力を調査します。完了の条件は、別プロジェクトの回避策を必要とせず、成功したテストで無関係なクラスの行ヒットが維持されることです。

Environment

• OS: Ubuntu 24.04.4 LTS (Linux CI agent), Docker container based on mcr.microsoft.com/dotnet/sdk:10.0-noble
• CPU/RAM: 12-core Ryzen, 32 GB RAM
• .NET SDK: 10.0.203 (also reproduces with .NET 9 SDK side-by-side installed)
• Test framework: xUnit v3 via xunit.v3.mtp-v2
• MTP coverage extension: Microsoft.Testing.Extensions.CodeCoverage 18.6.2
• Test runner: dotnet test ... --coverage --coverage-output-format cobertura --coverage-output
  coverage.cobertura.xml --coverage-settings .runsettings
• Reproduces both via Azure DevOps DotNetCoreCLI@2 task and raw pwsh dotnet test invocation (verified — task
  wrapper is not the cause).

Summary (revised)

A test class containing [Fact]s that spawn another dotnet .dll as a subprocess via
Process.Start is correlated with the parent test process's published cobertura output dropping line hits for many
classes that the same tests still pass against. The drop pattern is binary at the class level, not gradient at
the method level.

Concretely, on a single Linux CI run reproducing the issue (Microsoft.Testing.Extensions.CodeCoverage 18.6.2,
.NET 10.0.203, xUnit v3 via xunit.v3.mtp-v2), 73 classes that were ≥99% covered in the prior run dropped to ≤9%
in the current run, distributed:

┌───────────────────────────┬────────────────┐
│ Post-drop coverage bucket │ Class count │
├───────────────────────────┼────────────────┤
│ 0–9% │ 73 │
├───────────────────────────┼────────────────┤
│ 10–19% │ 12 │
├───────────────────────────┼────────────────┤
│ 20–29% │ 4 │
├───────────────────────────┼────────────────┤
│ 30–39% │ 0 │
├───────────────────────────┼────────────────┤
│ 40–89% │ 14 (scattered) │
├───────────────────────────┼────────────────┤
│ 90–99% │ 3 │
└───────────────────────────┴────────────────┘

51 of the 73 classes went from ≥99% to exactly 0% — every line uncovered, no partial coverage. Test results are
unchanged: ≥99% of tests passed, including tests known to exercise the now-zero classes.

The bimodal distribution (most affected classes either keep full coverage or are fully zeroed, with a small
intermediate tail) is inconsistent with a per-method hit-count race, which would produce a gradient. It is
consistent with per-class instrumentation registration being lost at JIT time. Smaller classes (most of the
zeroed set are 1–30 coverable lines) JIT in a single brief window; if the perturbation overlaps that window,
every method on the class misses instrumentation. Larger classes (e.g., one we examined has 392 coverable lines
across many methods) JIT method-by-method over a longer interval and end up with partial coverage — some methods
registered, others not.

Cobertura lines-valid, method counts, and per-class metadata (sequence points, complexity) are identical between
healthy and broken runs. Only lines-covered differs. PDB-driven instrumentation enumeration is fine; runtime hit
recording is what fails.

Effects observed on real CI runs:

• Overall coverage drops from a stable baseline of ~89% to ~73–78%.
• Specific classes that are well-covered by tests that pass show 0% line coverage in the published Cobertura
  output.
• Cobertura lines-valid / method counts are identical to a healthy run; only lines-covered differs. Per-class
  metadata (sequence points, complexity) is fully present, so PDB scanning/instrumentation enumeration is fine —
  only runtime hit recording is dropped.
• Test results are clean (>99% passed). Tests covering the affected classes did execute and assert successfully —
  they just didn't get credit.

Minimal Repro Sketch

public sealed class SubprocessSmokeFixture
{
public string ChildDllPath { get; } = /* path to any built net10.0 console app dll */;

  public async Task<int> RunAsync(params string[] args)
  {
      var psi = new ProcessStartInfo("dotnet")
      {
          RedirectStandardOutput = true,
          RedirectStandardError = true,
          UseShellExecute = false,
          CreateNoWindow = true,
      };
      psi.ArgumentList.Add(ChildDllPath);
      foreach (var a in args) psi.ArgumentList.Add(a);

      using var p = new Process { StartInfo = psi };
      p.Start();
      await p.WaitForExitAsync();
      return p.ExitCode;
  }

}

public sealed class SmokeTests : IClassFixture
{
private readonly SubprocessSmokeFixture _spawner;
public SmokeTests(SubprocessSmokeFixture s) => _spawner = s;

  [Fact] public async Task RunsChildSubprocess() => Assert.Equal(0, await _spawner.RunAsync("--version"));
  [Fact] public async Task RunsChildSubprocessAgain() => Assert.Equal(0, await _spawner.RunAsync("--help"));

}

Add a few hundred unrelated tests in the same project that exercise unrelated production code. Run with dotnet
test --coverage. The unrelated production code's class-level coverage will be partially zeroed despite all tests
passing.

What We Verified

1. Tests run and pass — confirmed via TRX. Test result counts and outcomes are identical between healthy and
   broken runs.
2. Static IL not modified — captured SHA256 of every project assembly in the test bin before vs. after the test
   step on Linux CI. All hashes identical. So MS Code Coverage on Linux CI uses dynamic / JIT-time profiler
   instrumentation, not static rewrite. Hit recording depends on the runtime profiler — and that's what's failing.
3. Excluding the subprocess-spawning test classes restores coverage to baseline (deterministic).
4. Child inherits profiler env vars by default. Captured the env of two PIDs in the test process tree:
   - Parent test orchestrator: no profiler env vars set.
   - Test host (the child of the orchestrator that loads tests): CORECLR_ENABLE_PROFILING=1,
   CORECLR_PROFILER={324F817A-7420-4E6D-B3C1-143FBED6D855}, CORECLR_PROFILER_PATH_64=…/libInstrumentationEngine.so,
   plus MicrosoftInstrumentationEngine_ConfigPath64_VanguardInstrumentationProfiler and
   MicrosoftInstrumentationEngine_DisableCodeSignatureValidation=1.
   - When the test host spawns dotnet … via Process.Start, the child inherits these by default. The child loads
   the same profiler shim and instrumentation method as the parent, contending for shared bin-dir state
   (runtimes/linux-x64/native/Cov_x64.config is the same file).
5. Scrubbing profiler env in child psi.Environment helps but doesn't fully fix it.
   Removed in child's psi.Environment before Process.Start:
   CORECLR_ENABLE_PROFILING (set to "0")
   CORECLR_PROFILER
   CORECLR_PROFILER_PATH / 32 / 64
   DOTNET_STARTUP_HOOKS
   MicrosoftInstrumentationEngine*
   Microsoft_VisualStudio_TraceDataCollector*
   MicrosoftCodeCoverage_*
   MTPCC_*
   VSTEST_*
6. With this scrub: regression goes from ~16pp → ~6pp. So env inheritance accounts for ~10pp of the bug; the
   remaining ~6pp comes from Process.Start itself. (Likely fork()/exec() on Linux briefly perturbs profiler
   thread/signal state, or threadpool callbacks for OutputDataReceived/ErrorDataReceived async readers compete with
   profiler hit-recording threads. Speculation; we don't have a definitive mechanism for the residual.)
7. DotNetCoreCLI@2 Azure DevOps task is not the cause. Replaced with raw pwsh dotnet test, identical
   reproduction.
8. Reproducibility: consistent on the Azure DevOps Linux agent. We could not reproduce in a manually constructed
   Linux Docker container with the same SDK + workloads on a high-spec workstation — likely a timing-window artifact
   rather than an environmental difference (we ran the local container only a handful of times; CI runs every
   commit).

Expected Behavior

A test that spawns a dotnet child process should not affect the coverage profiler's hit recording in the parent
test host process for unrelated classes.

Workaround We Implemented

In our subprocess fixture, scrub profiler-related env from the child ProcessStartInfo before Start():

private static void ScrubProfilerEnv(ProcessStartInfo psi)
{
psi.Environment["CORECLR_ENABLE_PROFILING"] = "0";
psi.Environment.Remove("CORECLR_PROFILER");
psi.Environment.Remove("CORECLR_PROFILER_PATH");
psi.Environment.Remove("CORECLR_PROFILER_PATH_32");
psi.Environment.Remove("CORECLR_PROFILER_PATH_64");
psi.Environment.Remove("DOTNET_STARTUP_HOOKS");
string[] prefixes =
{
"MicrosoftInstrumentationEngine_",
"MicrosoftTestPlatform_",
"Microsoft_VisualStudio_TraceDataCollector_",
"MicrosoftCodeCoverage_",
"MTPCC_",
"VSTEST_"
};
var keys = psi.Environment.Keys
.Where(k => prefixes.Any(p => k.StartsWith(p, StringComparison.Ordinal)))
.ToList();
foreach (var key in keys) psi.Environment.Remove(key);
}

This recovers ~10pp of the regression. The remaining ~6pp we've worked around by moving subprocess-spawning tests
into a separate test project (separate dotnet test invocation, separate test host process), which removes them
from the same coverage-instrumented host.