[quality-improver] CiRunSummaryAggregateReader.ValidateModule provenance cross-check guards are untested on the throwing path

The issue is about adding unit tests for the provenance guards in CiRunSummaryAggregateReader.ValidateModule. Start by reading the existing test file at test/UnitTests/Microsoft.Testing.Extensions.UnitTests/CiRunSummaryAggregationTests.cs and the helper method CreateInput. Then, modify the helper to accept optional parameters for mismatched provenance values. Write new tests that pass mismatched InputArtifact values for ProducingTestModule, TargetFramework, Architecture, and ExecutionId/AttemptNumber, and assert that FormatException is thrown with the expected message. Run the tests to ensure they pass and cover the previously untested throwing paths.

🎯 Repository Quality Improvement Report — CI Run Summary Provenance Guard Coverage Gap

Analysis Date: 2026-09-23
Focus Area: ci-run-summary-provenance-guard-coverage-gap
Strategy Type: Custom

Executive Summary

Microsoft.Testing.Extensions.SharedExtensionHelpers recently gained a new CiRunSummaryAggregation/CiRunSummaryAggregateReader pair that aggregates per-module CI run summary fragments (coverage, flaky tests, history, dependencies) shared by both the AzureDevOpsReport and GitHubActionsReport extensions. CiRunSummaryAggregateReader.ValidateModule includes four distinct "provenance" guards that cross-check a deserialized fragment's ModulePath, TargetFramework, Architecture, and ExecutionId/AttemptNumber against the InputArtifact metadata supplied by the platform's artifact post-processing pipeline, throwing FormatException on any mismatch. These guards exist specifically to catch fragments that were mixed up between modules/frameworks/architectures/retry attempts in CI — exactly the kind of defect that is silent until multi-module or multi-attempt runs corrupt an aggregate report.

test/UnitTests/Microsoft.Testing.Extensions.UnitTests/CiRunSummaryAggregationTests.cs is a substantial 972-line, 23-test suite that already covers schema-version mismatches, duplicate identities, null bounded collections, and null history/dependency entries — but every test constructs its InputArtifact via the CreateInput helper, which always derives producingTestModule, targetFramework, architecture, and executionId directly from the same module object being validated. As a result the four provenance-mismatch branches (lines ~183–192 of CiRunSummaryAggregateReader.cs) are never exercised on their throwing path — a regression that silently disabled cross-module corruption detection (e.g. reverting a != to ==, or dropping a branch) would compile and pass the entire existing suite.

The task below adds targeted tests that pass a mismatched InputArtifact for each of the four provenance dimensions plus the RetryAttempts attempt-number variant, asserting the specific FormatException message for each.

🤖 Suggested Improvement Tasks

Task 1: Add tests for ValidateModule's ProducingTestModule provenance mismatch

Priority: Medium
Estimated Effort: Small

In test/UnitTests/Microsoft.Testing.Extensions.UnitTests/CiRunSummaryAggregationTests.cs, add a test that writes a valid fragment for a module (e.g. via CreateModule/WriteFragmentAsync), then calls CiRunSummaryAggregation.ReadAndAggregate with an InputArtifact whose producingTestModule argument is a different path than module.ModulePath (while input.Path still points at the correct fragment file). Assert Assert.ThrowsExactly<FormatException>(...) and, ideally, assert on the exception message containing "provenance does not match" to pin the specific guard. This exercises src/Platform/SharedExtensionHelpers/CiRunSummaryAggregateReader.cs's if (input.ProducingTestModule is not null && !PathsEqual(...)) branch.

Task 2: Add tests for TargetFramework and Architecture provenance mismatch

Priority: Medium
Estimated Effort: Small

Add two tests (or one [DataRow]-parameterized test) analogous to Task 1, but instead pass an InputArtifact with a targetFramework or architecture value that differs from the fragment's module. Both branches are in ValidateModule immediately after the ProducingTestModule check. Consider extracting a small helper overload of CreateInput that accepts explicit override values for producingTestModule, targetFramework, architecture, and executionId instead of always deriving them from module, to keep the new tests concise and to make future provenance tests easy to add.

Task 3: Add tests for ExecutionId/AttemptNumber provenance mismatch in both post-processing modes

Priority: Medium
Estimated Effort: Small

Add tests covering both branches of the ternary in the ExecutionId guard:

• ArtifactPostProcessingMode.RetryAttempts: pass an input.ExecutionId that either fails int.TryParse or parses to a value different from module.AttemptNumber.
• Default mode: pass an input.ExecutionId string that differs from module.ExecutionId.

Both should throw FormatException with the execution-provenance message. This directly complements the existing GitHubAggregate_ForRetryAttempts_UsesInitialAttemptCoverage test, which exercises the happy path of RetryAttempts mode but not its mismatch guard.

Task 4: Extend the shared CreateInput test helper to support mismatched provenance construction

Priority: Low
Estimated Effort: Small

Refactor CreateInput(string path, CiRunSummaryModule module) (line ~851) to add optional parameters (producingTestModule, targetFramework, architecture, executionId) defaulting to the module's own values, so Tasks 1–3 can construct mismatched InputArtifacts without duplicating the existing helper's boilerplate. Keep the existing call sites unchanged by relying on the new parameters' defaults.

📊 Historical Context

🎯 Recommendations

Immediate Actions (This Week)

1. Add the four provenance-mismatch tests (Tasks 1–3) — Priority: Medium

Short-term Actions (This Month)

1. Consider adding basic unit tests for NamedPipeServerFactory, PipeNameEnvironmentVariableProviderBase, CiRunSummaryFragmentIo, and ReportFileNameHelper in SharedExtensionHelpers — Priority: Low

Next analysis: 2026-09-24 — Focus area selected based on diversity algorithm

🤖 Automated content by GitHub Copilot. Generated by the Repository Quality Improver workflow. · copilot · auto · 113.4 AIC · ⌖ 6.87 AIC · ⊞ 16.5K · [◷]( · ◷)

gh aw add githubnext/agentics/workflows/repository-quality-improver.md@main

• expires on Sep 25, 2026, 10:27 PM UTC