[quality-improver] Azure DevOps summary reports raw numeric exit codes while GitHub Actions explains them

Chiusa
  • Qualcosa non va?
#11,961 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@Evangelink ci sta già lavorando.

Dal 16/10/2026.

  • #11970 di @copilot-swe-agent — integrata

Valutazione

Difficoltà
3/5
Tempo stimato
1-2 giorni
Idoneità per principianti
45/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
csharp
Ambito
devtools, testing-qa

Direzione di ricerca

Start with AppendStatusStrip and AppendModuleDiagnostics in src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/AzureDevOpsSummaryReporter.Markdown.cs, and compare them with GitHubActionsSummaryReporter.Markdown.PerProject.cs, which already names the exit code. The issue leaves open whether to share GitHubActionsExitCode or duplicate its resources, so confirm that choice with maintainers first. Done means the Azure DevOps summary shows the exit code name and reason, and a unit test in AzureDevOpsLivePublishingTests.cs asserts it.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

type/automation type/tech-debt
🎯 Repository Quality Improvement Report — CI Reporter Diagnostic Parity

Analysis Date: 2026-10-15
Focus Area: Cross-reporter exit-code diagnostic parity
Strategy Type: Custom

Executive Summary

Microsoft.Testing.Extensions.AzureDevOpsReport and Microsoft.Testing.Extensions.GitHubActionsReport share nearly identical infrastructure — both compile the same SharedExtensionHelpers sources (CiRunSummaryModels.cs, CiRunSummaryAggregation.cs, SummaryReporterHelpers.cs, etc.) and both model a run's outcome through the same CiRunSummaryModule.ExitCode field. Yet only the GitHub Actions reporter turns that exit code into a human-readable explanation.

GitHubActionsExitCode.GetReason(int) maps every documented ExitCode value (InvalidPlatformSetup, ZeroTests, MinimumExpectedTestsPolicyViolation, TestAdapterTestSessionFailure, IncompatibleProtocolVersion, TestExecutionStoppedForMaxFailedTests, CoverageThresholdFailed, TestExecutionStoppedAtDeadline, ...) to a localized sentence drawn from GitHubActionsResources.resx (16 ExitCodeReason* entries), and GetName/IsTestResultOutcome/IndicatesFailure are used throughout GitHubActionsAnnotationReporter, GitHubActionsSummaryReporter.Markdown.Aggregate.cs, and GitHubActionsSummaryReporter.Markdown.PerProject.cs to surface a readable callout (e.g. "exit code 9 (MinimumExpectedTestsPolicyViolation)") whenever a non-test-result failure occurs.

AzureDevOpsSummaryReporter has no equivalent. AppendStatusStrip and AppendModuleDiagnostics in AzureDevOpsSummaryReporter.Markdown.cs render only the bare integer — · exit code **9** or > Module exit code: `9` — with no name and no explanation. A developer reading an Azure DevOps pipeline summary for a run that hit, say, the minimum-expected-tests policy or a coverage-threshold failure sees an opaque number and has to go look up docs/Changelog-Platform.md/MS Learn exit-code tables to understand what happened, while the exact same scenario in a GitHub Actions summary spells it out inline. This is a straightforward parity gap between two reporters that otherwise deliberately share their aggregation and formatting helpers.

Full Analysis Report
Focus Area: Cross-reporter exit-code diagnostic parity
Current State Assessment

Metrics Collected:

Metric Value Status
ExitCodeReason* resource entries in GitHubActionsResources.resx 16 ✅
Equivalent entries in AzureDevOpsResources.resx 0 ❌
GitHubActionsSummaryReporter call sites using GetReason/GetName/IsTestResultOutcome 11 ✅
AzureDevOpsSummaryReporter call sites rendering a named/explained exit code 0 ❌
Shared SharedExtensionHelpers source files linked by both projects 10+ (identical set) ✅
Findings
Strengths
  • GitHubActionsExitCode is a clean, fully-covered mapping from every ExitCode enum member to a localized string, exercised consistently across annotation, aggregate-summary, and per-project-summary rendering paths.
  • Both reporters already share the exact same CiRunSummaryModule/CiRunSummaryAggregation model, so there is no structural barrier to reusing the same mapping logic for Azure DevOps — only the presentation layer differs today.
Areas for Improvement
  • 🟡 Medium — AzureDevOpsSummaryReporter.Markdown.cs (AppendStatusStrip, lines ~404-429, and AppendModuleDiagnostics, lines ~179-192) prints only module.ExitCode/exitCode as a bare integer with no name or reason, even though the identical information is available and already explained one directory over in GitHubActionsExitCode.GetReason/GetName.
🤖 Suggested Improvement Tasks
Task 1: Extract GitHubActionsExitCode into a shared, reporter-agnostic helper

Priority: Medium
Estimated Effort: Medium

Move GetName/GetReason/IsTestResultOutcome/IndicatesFailure (currently internal in src/Platform/Microsoft.Testing.Extensions.GitHubActionsReport/GitHubActionsExitCode.cs) into src/Platform/SharedExtensionHelpers/ as a linked source file (following the existing pattern already used for SummaryReporterHelpers.cs, CiRunSummaryModels.cs, etc.), with the reason strings expressed through a small parameterized resource contract each reporter project supplies, or duplicated as resx entries per project (matching how other shared-helper resources are already handled). Keep the existing GitHubActionsExitCode call sites working unchanged.

Task 2: Add ExitCodeReason* resource entries and rendering to AzureDevOpsResources.resx / AzureDevOpsSummaryReporter

Priority: Medium
Estimated Effort: Small

Add the same 16 ExitCodeReason* keys (or localized equivalents) to src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/Resources/AzureDevOpsResources.resx, and update AppendStatusStrip and AppendModuleDiagnostics in AzureDevOpsSummaryReporter.Markdown.cs to render exit code **9** (MinimumExpectedTestsPolicyViolation — Fewer tests ran than required by the minimum-expected-tests policy.) instead of the bare number, mirroring GitHubActionsSummaryReporter.Markdown.PerProject.cs's existing pattern (lines ~88-95, ~219-222).

Task 3: Add regression tests asserting the explained-exit-code text appears in Azure DevOps summaries

Priority: Medium
Estimated Effort: Small

In test/UnitTests/Microsoft.Testing.Extensions.UnitTests/AzureDevOpsLivePublishingTests.cs (or a sibling markdown-rendering test file), add a case that builds a CiRunSummaryModule with a known non-zero ExitCode (e.g. (int)ExitCode.ZeroTests) and asserts the rendered markdown contains the human-readable name/reason, not just the integer — mirroring whatever assertion style the equivalent GitHub Actions tests already use for GitHubActionsExitCode.GetReason.

Task 4: Audit other CI-facing reporters (CTRF, HTML, JUnit, TRX) for the same gap

Priority: Low
Estimated Effort: Small

CtrfReportEngine, HtmlReportEngine, and TrxReportEngine all persist a raw exitCode field/number in their respective output formats (JSON "exitCode", HTML exitCode pair, TRX via TrxReportEngine.Metadata.cs) without an accompanying name/reason either. Decide whether machine-readable formats should stay numeric-only (reasonable for JSON/XML consumers) or whether any of their human-facing renderings (e.g. the HTML report's visual summary) should also surface the explained reason once the shared helper from Task 1 exists.

📊 Historical Context
Previous Focus Areas (last 5 runs)
Date Focus Area Type
2026-10-08 experimental-api-diagnostic-url-consistency Custom
2026-10-09 central-package-management-dead-entries Custom
2026-10-12 mutation-testing-manifest-coverage-gap Custom
2026-10-13 aotsg-diagnostic-documentation-parity Custom
2026-10-14 browser-wasm-blocking-wait-contract-compliance Custom
🎯 Recommendations
Immediate Actions (This Week)
  1. Extract the exit-code name/reason mapping into a shared helper reusable by both reporters — Priority: Medium
Short-term Actions (This Month)
  1. Wire the shared mapping into AzureDevOpsSummaryReporter and add its resource entries — Priority: Medium
  2. Add a regression test locking in the explained-exit-code rendering — Priority: Medium
  3. Audit CTRF/HTML/JUnit/TRX reporters for the same gap in their human-facing output — Priority: Low

Next analysis: 2026-10-16 — Focus area selected based on diversity algorithm

🤖 Automated content by GitHub Copilot. Generated by the Repository Quality Improver workflow. · copilot · auto · 149.3 AIC · ⌖ 17.9 AIC · ⊞ 17.5K · [◷]( · ◷)

Add this agentic workflow to your repo

To install this agentic workflow, run

gh aw add githubnext/agentics/workflows/repository-quality-improver.md@main
  • expires on Oct 17, 2026, 10:31 PM UTC
Lingua principale
C#
Stelle
1.1k
Fork
312
Merge medio
11h 24m
PR unite (30g)
328

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di microsoft/testfx

Tutte le issue di microsoft/testfx

Issue simili

Altre issue su C#

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.