scripts: the `issue_workspace` exact-match test fails when a fixture commit's SHA contains 570 (flaky NVX CLI)

オープン 初心者向け
#1,665 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
1/5
見積もり時間
1時間未満
初心者へのやさしさ
90/100
issue の種類
バグ
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
git, github-actions, python
領域
ci-cd, testing

調査の方向性

scripts/test_issue_workspace.pyを読み、特にIssueWorkspaceTests.test_find_groups_exact_issue_matches_from_every_sourceと、283行目付近のアサーションを確認してください。アサーションを変更し、単独のissue番号ではなく、おとりのブランチ名を確認するようにしてから、そのunittestを実行してください。issueでは日付を固定した再現手順が説明されており、おとりのブランチとpull requestのチェックが引き続きカバーされることを想定しています。

索引モデルが issue の本文から書いたものです。

説明

area: ci

Summary

scripts.test_issue_workspace.IssueWorkspaceTests.test_find_groups_exact_issue_matches_from_every_source fails whenever the 12-character SHA prefix of either commit that its fixture creates contains 570. Each time, it fails with AssertionError: True is not false at self.assertFalse(any("570" in name for name in names)), and nothing else in the job fails. In the CI runs created from 2026-11-06 to 2026-11-16, it failed 6 of the 1,554 NVX CLI jobs that completed, 4 on Linux and 2 on Windows. That is about 1 in 260, and 6 of the 21 NVX CLI jobs that failed.

The test checks that issue_workspace.find_issue, when it looks for issue 57, reports none of the fixture's decoys for issue 570. It does so by searching every candidate's name for 570. But the tool names a commit candidate after its SHA prefix, and the fixture's commit SHAs change with the time at which the test runs. A prefix contains 570 with a probability of about 0.24%, so with two commits the test fails about once in 205 runs. Pinning the fixture's commit dates reproduces the failure every time. The tool excludes the decoys correctly, so the defect is in the test.

Incident

  • Workflow: CI run 43902030265, attempt 1, push to feature/nvx-refactor (#1651) at 263d9ee8dff.
  • Failed job: NVX CLI / Windows, step Validate NVX CLI. It failed in the python -m unittest command of the validate-nvx action's Windows step (action.yml#L25-L34). Of 2,213 tests, 87 were skipped, and this test was the only failure. The run's NVX CLI / Linux job failed for an unrelated reason, which 2b8b51581aa fixed.
  • Runner: GitHub-hosted, image windows-2025-vs2026 20261103.298.1, runner 2.338.2, Python 3.12.10.
  • The test and scripts/issue_workspace.py are byte-identical at all six failing heads and on dev at 28a0c326f5e.

Decisive output:

======================================================================
FAIL: test_find_groups_exact_issue_matches_from_every_source (scripts.test_issue_workspace.IssueWorkspaceTests.test_find_groups_exact_issue_matches_from_every_source)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\a\nvx\nvx\scripts\test_issue_workspace.py", line 283, in test_find_groups_exact_issue_matches_from_every_source
    self.assertFalse(any("570" in name for name in names))
AssertionError: True is not false

----------------------------------------------------------------------
Ran 2213 tests in 679.383s

FAILED (failures=1, skipped=87)

History

I classified the failed NVX CLI jobs of the 1,172 CI runs created from 2026-11-06 to 2026-11-16 18:43 UTC:

  • 783 NVX CLI / Linux jobs completed. 14 failed, 4 of them on this assertion.
  • 771 NVX CLI / Windows jobs completed. 7 failed, 2 of them on this assertion.
  • In each of the 6, this assertion was the job's only failure.
  • The window starts before #1337 added the assertion on 2026-11-07, so 6 in 1,554 slightly understates the rate.
Created (UTC) Run Event Head Job Outcome
2026-11-10 01:25 42735627387 push, dev f5ef2dceaee Linux Run failed.
2026-11-14 18:05 43624611183 pull_request, #1591 3e4c9f59ebe Linux Attempt 2 passed; #1591 merged.
2026-11-15 08:26 43694553887 push, dev 45081e3e553 Linux Run failed.
2026-11-16 00:45 43782274623 push, dev 52da532f5e3 Linux Run failed.
2026-11-16 04:23 43803791787 pull_request, #1646 181da97bea4 Windows Attempt 2 passed; #1646 merged.
2026-11-16 15:57 43902030265 push, feature/nvx-refactor (#1651) 263d9ee8dff Windows This incident.

Root cause

Permalinks point to dev at 28a0c326f5e.

  1. The tool names a commit candidate after its SHA. _append_commit_candidates names each commit that mentions the issue f"{commit[:12]} {subject}" (issue_workspace.py#L272-L287). The test's fixture creates two such commits, Fix #57 on dev and Work on issues/57 (test_issue_workspace.py#L236-L239).
  2. The fixture's SHAs change with the clock. _commit (L39-L43) commits with a fixed identity but at the current time, so the SHAs of both commits differ from run to run.
  3. The assertion searches every candidate's name for a bare 570. self.assertFalse(any("570" in name for name in names)) (L283) is meant to catch the decoys: the branch issue-570-unrelated (L241) and pull request #11, Fix #570 (L174-L181). But it also matches a SHA prefix that happens to contain 570. Each of the 10 three-character windows of a 12-character prefix is 570 with a probability of 1 in 4,096, so a prefix contains 570 with a probability of about 0.24%, and either of two prefixes with about 0.49%: about 1 run in 205. The worktree candidate's name, a path under the fixture's random temporary directory (L243), adds about 0.01%.
  4. The tool excludes the decoys. _number_pattern (issue_workspace.py#L227-L228) requires the number to stand alone, so the branch scan (L396-L412) skips issue-570-unrelated. _reference_pattern (L223-L224) does the same for the pull requests' titles and bodies (L326-L338), as the commit searches do for their messages (L432-L438). The test checks pull request #11 separately, with "#11 " (test_issue_workspace.py#L282). The tool keeps a pull request's head ref, here fix-570, only in the candidate's details (issue_workspace.py#L256-L267).

Reproduction

Pinning the fixture's commit dates makes its SHAs deterministic. On Linux (Ubuntu under WSL, Python 3.14.4, Git 2.53.0), with the two files as they are on dev:

GIT_AUTHOR_DATE='1794787201 +0000' GIT_COMMITTER_DATE='1794787201 +0000' \
  python3 -m unittest scripts.test_issue_workspace.IssueWorkspaceTests.test_find_groups_exact_issue_matches_from_every_source

This fails 3 times out of 3 with the CI's assertion at line 283, because these dates give the Fix #57 on dev commit the prefix 02e570e6ec14. With 1794787200 +0000, the test passes. Other platforms may produce other SHAs for the same dates.

Impact

  • The failure fails Required status check, the only required check, so a pull request that hits it needs a rerun: #1591 and #1646 passed on attempt 2.
  • A dev push that hits it publishes no development release and records no performance results, because Publish development release and Record performance results need both NVX CLI jobs to succeed (ci.yml#L1598-L1610, ci.yml#L1847-L1857). Both jobs were skipped in the three dev pushes in the table.

Suggested fix

Match the decoy branch's name instead of the bare number:

-        self.assertFalse(any("570" in name for name in names))
+        self.assertFalse(any("issue-570" in name for name in names))
  • issue-570 cannot occur in a hexadecimal SHA prefix, and a temporary directory's random name holds only lowercase letters, digits, and _.
  • With this change, the test passes at both pinned dates above. It still fails at line 283 when _number_pattern loses its (?![0-9]) guard, because the decoy branch then matches issue 57. The "#11 " check on line 282 still covers pull request #11.
  • The change modifies an existing test, so its pull request needs the tests-modified label.

Pinning the commit dates in _commit would also stop the flake, but the assertion would still depend on whichever SHAs those dates produce.

Related work

  • #1337 added the test and this assertion in 6de09c84e24.
  • #1591 and #1646 needed a rerun because of it, and #1651 hit it in a push run.
主要言語
Rust
スター
848
フォーク
63
平均マージ
13時間 37分
マージ済み PR(30日)
571

環境構築

  • Dockerfile・Docker Compose ファイルなし
  • プルリクエストのテンプレートあり
  • コントリビューションガイドなし

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

microsoft/nvx のほかの issue

microsoft/nvx の issue をすべて見る

似ている issue

Rust の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。