Retry extension: final attempt's .trx is duplicated in results directory, causing double-counted attempts in Azure DevOps

クローズ
  • 何か問題がありますか？
#12,228 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

関連するプルリクエストがすでにマージされています。

  • #12241 @Evangelink による — マージ済み

評価

難易度
3/5
見積もり時間
1〜2日
初心者へのやさしさ
57/100
issue の種類
バグ
明瞭さ
おおむね明確
活発さ
活発
技術スタック
csharp
領域
testing

調査の方向性

Start by tracing how Microsoft.Testing.Extensions.Retry writes per-attempt TRX files and copies the final attempt into the results directory; the issue does not name source files or tests. Reproduce with dotnet test --test-modules **/*.Tests.dll --results-directory "<dir>" --report-trx --retry-failed-tests 1 and an always-failing test. Done when publishing the generated files with **/*.trx reports each actual attempt once.

索引モデルが issue の本文から書いたものです。

説明

needs/triage

Describe the bug

When using Microsoft.Testing.Extensions.Retry together with --report-trx, the retry extension writes one .trx per attempt to <results-directory>/Retries/<id>/<attempt>/, and additionally copies the .trx of the last attempt to <results-directory> itself.

When results are published to Azure DevOps with the common pattern **/*.trx (PublishTestResults@2), the last attempt is published twice. For failed or retried tests, Azure DevOps then shows one more attempt than actually happened:

  • a test that was retried once shows three attempts, the last two being identical;
  • a failed test that was not retried (e.g. because --retry-failed-tests-max-percentage was exceeded) shows Attempt 0 and Attempt 1 with identical results.

Tests that pass on the first try are shown only once.

This makes the test history misleading, especially for detecting flaky tests.

Version used

  • .NET SDK 10.x, dotnet test with "test": { "runner": "Microsoft.Testing.Platform" } in global.json
  • NUnit via VSTestBridge (also applicable to MSTest/xUnit v3/TUnit but not tested)
  • NUnit 5.0.0; NUnit3TestAdapter 6.3.1
  • Microsoft.Testing.Extensions.Retry version: 2.5.2
  • Azure DevOps PublishTestResults@2, testResultsFormat: VSTest

Steps To Reproduce

dotnet test --test-modules **/*.Tests.dll --results-directory "<dir>" --report-trx --retry-failed-tests 1

with one test that always fails. Resulting files:

<dir>\MyTests_net10.0_x64.trx                 <- copy of last attempt
<dir>\Retries\Fx7uV\1\MyTests_net10.0_x64.trx  <- attempt 1
<dir>\Retries\Fx7uV\2\MyTests_net10.0_x64.trx  <- attempt 2

Note that the copy in <dir> is not listed under "In process file artifacts produced" in the console output.

Expected behavior

Test has Attempt# 0 and Attempt# 1 when retried

Actual behavior

Test has Attempt# 0, Attempt# 1 ánd Attempt# 2 when retried

Additional context

Questions:

  1. Is copying the final attempt's results to the root of the results directory intended behavior? If so, could this be documented?
  2. What is the recommended way to publish results to Azure DevOps so that every actual attempt is reported exactly once? Publishing Retries/**/*.trx when retries are enabled and *.trx otherwise works, but requires conditional logic in every pipeline.
  3. Would you consider an option to control this, for example writing only the per-attempt files, or only a single merged .trx that contains all attempts?
主要言語
C#
スター
1.1k
フォーク
314
平均マージ
11時間 9分
マージ済み PR（30日）
340

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

microsoft/testfx のほかの issue

microsoft/testfx の issue をすべて見る

似ている issue

C# の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。