scripts: the `issue_workspace` exact-match test fails when a fixture commit's SHA contains 570 (flaky NVX CLI)
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 1/5
- Estimated time
- Under an hour
- Newbie friendliness
- 90/100
Research direction
Read scripts/test_issue_workspace.py, especially IssueWorkspaceTests.test_find_groups_exact_issue_matches_from_every_source and its assertion near line 283. Change the assertion so it checks for the decoy branch name instead of the bare issue number, then run that unittest; the issue describes a pinned-date reproduction and expects the decoy branch and pull request checks to remain covered.
Written by the indexing model from the issue text.
Description
Summary
scripts.test_issue_workspace.IssueWorkspaceTests.test_find_groups_exact_issue_matches_from_every_source fails whenever the 12-character SHA prefix of either commit that its fixture creates contains 570. Each time, it fails with AssertionError: True is not false at self.assertFalse(any("570" in name for name in names)), and nothing else in the job fails. In the CI runs created from 2026-11-06 to 2026-11-16, it failed 6 of the 1,554 NVX CLI jobs that completed, 4 on Linux and 2 on Windows. That is about 1 in 260, and 6 of the 21 NVX CLI jobs that failed.
The test checks that issue_workspace.find_issue, when it looks for issue 57, reports none of the fixture's decoys for issue 570. It does so by searching every candidate's name for 570. But the tool names a commit candidate after its SHA prefix, and the fixture's commit SHAs change with the time at which the test runs. A prefix contains 570 with a probability of about 0.24%, so with two commits the test fails about once in 205 runs. Pinning the fixture's commit dates reproduces the failure every time. The tool excludes the decoys correctly, so the defect is in the test.
Incident
- Workflow: CI run 43902030265, attempt 1,
pushtofeature/nvx-refactor(#1651) at263d9ee8dff. - Failed job:
NVX CLI / Windows, stepValidate NVX CLI. It failed in thepython -m unittestcommand of thevalidate-nvxaction's Windows step (action.yml#L25-L34). Of 2,213 tests, 87 were skipped, and this test was the only failure. The run'sNVX CLI / Linuxjob failed for an unrelated reason, which2b8b51581aafixed. - Runner: GitHub-hosted, image
windows-2025-vs202620261103.298.1, runner 2.338.2, Python 3.12.10. - The test and
scripts/issue_workspace.pyare byte-identical at all six failing heads and ondevat28a0c326f5e.
Decisive output:
======================================================================
FAIL: test_find_groups_exact_issue_matches_from_every_source (scripts.test_issue_workspace.IssueWorkspaceTests.test_find_groups_exact_issue_matches_from_every_source)
----------------------------------------------------------------------
Traceback (most recent call last):
File "D:\a\nvx\nvx\scripts\test_issue_workspace.py", line 283, in test_find_groups_exact_issue_matches_from_every_source
self.assertFalse(any("570" in name for name in names))
AssertionError: True is not false
----------------------------------------------------------------------
Ran 2213 tests in 679.383s
FAILED (failures=1, skipped=87)
History
I classified the failed NVX CLI jobs of the 1,172 CI runs created from 2026-11-06 to 2026-11-16 18:43 UTC:
- 783
NVX CLI / Linuxjobs completed. 14 failed, 4 of them on this assertion. - 771
NVX CLI / Windowsjobs completed. 7 failed, 2 of them on this assertion. - In each of the 6, this assertion was the job's only failure.
- The window starts before #1337 added the assertion on 2026-11-07, so 6 in 1,554 slightly understates the rate.
| Created (UTC) | Run | Event | Head | Job | Outcome |
|---|---|---|---|---|---|
| 2026-11-10 01:25 | 42735627387 | push, dev |
f5ef2dceaee |
Linux | Run failed. |
| 2026-11-14 18:05 | 43624611183 | pull_request, #1591 |
3e4c9f59ebe |
Linux | Attempt 2 passed; #1591 merged. |
| 2026-11-15 08:26 | 43694553887 | push, dev |
45081e3e553 |
Linux | Run failed. |
| 2026-11-16 00:45 | 43782274623 | push, dev |
52da532f5e3 |
Linux | Run failed. |
| 2026-11-16 04:23 | 43803791787 | pull_request, #1646 |
181da97bea4 |
Windows | Attempt 2 passed; #1646 merged. |
| 2026-11-16 15:57 | 43902030265 | push, feature/nvx-refactor (#1651) |
263d9ee8dff |
Windows | This incident. |
Root cause
Permalinks point to dev at 28a0c326f5e.
- The tool names a commit candidate after its SHA.
_append_commit_candidatesnames each commit that mentions the issuef"{commit[:12]} {subject}"(issue_workspace.py#L272-L287). The test's fixture creates two such commits,Fix #57 on devandWork on issues/57(test_issue_workspace.py#L236-L239). - The fixture's SHAs change with the clock.
_commit(L39-L43) commits with a fixed identity but at the current time, so the SHAs of both commits differ from run to run. - The assertion searches every candidate's name for a bare
570.self.assertFalse(any("570" in name for name in names))(L283) is meant to catch the decoys: the branchissue-570-unrelated(L241) and pull request #11,Fix #570(L174-L181). But it also matches a SHA prefix that happens to contain570. Each of the 10 three-character windows of a 12-character prefix is570with a probability of 1 in 4,096, so a prefix contains570with a probability of about 0.24%, and either of two prefixes with about 0.49%: about 1 run in 205. The worktree candidate's name, a path under the fixture's random temporary directory (L243), adds about 0.01%. - The tool excludes the decoys.
_number_pattern(issue_workspace.py#L227-L228) requires the number to stand alone, so the branch scan (L396-L412) skipsissue-570-unrelated._reference_pattern(L223-L224) does the same for the pull requests' titles and bodies (L326-L338), as the commit searches do for their messages (L432-L438). The test checks pull request #11 separately, with"#11 "(test_issue_workspace.py#L282). The tool keeps a pull request's head ref, herefix-570, only in the candidate's details (issue_workspace.py#L256-L267).
Reproduction
Pinning the fixture's commit dates makes its SHAs deterministic. On Linux (Ubuntu under WSL, Python 3.14.4, Git 2.53.0), with the two files as they are on dev:
GIT_AUTHOR_DATE='1794787201 +0000' GIT_COMMITTER_DATE='1794787201 +0000' \
python3 -m unittest scripts.test_issue_workspace.IssueWorkspaceTests.test_find_groups_exact_issue_matches_from_every_source
This fails 3 times out of 3 with the CI's assertion at line 283, because these dates give the Fix #57 on dev commit the prefix 02e570e6ec14. With 1794787200 +0000, the test passes. Other platforms may produce other SHAs for the same dates.
Impact
- The failure fails
Required status check, the only required check, so a pull request that hits it needs a rerun: #1591 and #1646 passed on attempt 2. - A
devpush that hits it publishes no development release and records no performance results, becausePublish development releaseandRecord performance resultsneed bothNVX CLIjobs to succeed (ci.yml#L1598-L1610, ci.yml#L1847-L1857). Both jobs were skipped in the threedevpushes in the table.
Suggested fix
Match the decoy branch's name instead of the bare number:
- self.assertFalse(any("570" in name for name in names))
+ self.assertFalse(any("issue-570" in name for name in names))
issue-570cannot occur in a hexadecimal SHA prefix, and a temporary directory's random name holds only lowercase letters, digits, and_.- With this change, the test passes at both pinned dates above. It still fails at line 283 when
_number_patternloses its(?![0-9])guard, because the decoy branch then matches issue 57. The"#11 "check on line 282 still covers pull request #11. - The change modifies an existing test, so its pull request needs the
tests-modifiedlabel.
Pinning the commit dates in _commit would also stop the flake, but the assertion would still depend on whichever SHAs those dates produce.
Related work
- #1337 added the test and this assertion in
6de09c84e24. - #1591 and #1646 needed a rerun because of it, and #1651 hit it in a push run.
- Dominant language
- Rust
- Stars
- 831
- Forks
- 56
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 571
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- No contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from microsoft/nvx
-
area: ci
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
openvmm: virtiofs: aggregate synthetic root st_nlink counts single-file shares as subdirectoriesOpenarea: openvmm
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
microsoft/nvx#1632 · 1 comment ·
Maintainers usually reply within 1 day
-
area: openvmm
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
microsoft/nvx#1628 · 1 comment ·
Maintainers usually reply within 1 day
-
area: openvmm
Difficulty 2/5 1-3 hours Newbie friendliness 80/100
microsoft/nvx#1629 · 1 comment ·
Maintainers usually reply within 1 day
-
area: openvmm
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
microsoft/nvx#1631 · 1 comment ·
Maintainers usually reply within 1 day
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
oxc-project/tsrs#1010 ·
Maintainers usually reply within 1 day
-
Link is deadOpen
Difficulty 1/5 Under an hour Newbie friendliness 63/100
nicholas-ochoa/OpenSC2K#104 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
libracoder/kiri#445 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
DioxusLabs/blitz#1614 ·
Maintainers usually reply within 1 day