Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

ci: a cancelled job leaves its scratch directories behind and can leave OpenVMM running

Closed
  • Is something wrong?
#946 0 comments 0 reactions 1 assignee View on GitHub

Maintainers usually reply within 1 day

@ppenna is already working on this.

Since Oct 28, 2026.

  • #975 by @ppenna — merged

Assessment

This issue has not been assessed yet.

Description

confirmed

Summary

When a CI job is cancelled, the test harness never learns of it. The runner signals only the step's shell (#779), so the harness runs on until, a few seconds later, the runner's orphan cleanup ends it together with the OpenVMM processes that the cleanup lists. The harness's own cleanup never runs, so:

  • every scratch directory that a running scenario created stays in /tmp, where they accumulate: the ten Linux runners hold 64 of them; and
  • a VM that the harness starts after the orphan cleanup has listed the job's processes keeps running, as one did for 8 days on azure-mshv-1 (#768). Since #779, validate-runner's Terminate OpenVMM left by earlier jobs step ends such a VM when the runner's next job starts, but not before.

This issue takes over the test-harness work that #768 left open.

Evidence

On 2026-10-28 at about 01:05Z, the ten Linux runners held 64 /tmp/nvx-* and /tmp/openvmm-* directories, the oldest from 2026-10-12, when the runners booted, and ran no OpenVMM process. The Windows runners weren't examined. Each of the 18 directories left since #779 merged on 2026-10-23 was last modified during a cancelled pull-request job on the same runner, 2 to 46 s before the job ended:

Runner Directory Last modified Cancelled job
azure-kvm-2 nvx-snapshot-tier-workload-start-fg351kca 2026-10-23 14:50Z 120478619377, microVM tests, KVM, shard 1 of 2
azure-mshv-5 nvx-public-proxy-z7jnr9mi 2026-10-23 14:51Z 120478927919, microVM tests, MSHV, guests and sandboxes
azure-mshv-1 nvx-restore-downtime-5tedq1dy 2026-10-23 15:06Z 120485806283, microVM tests, MSHV, shard 1 of 2
azure-mshv-1 nvx-restore-processors-to3s5zfi 2026-10-23 18:26Z 120581648372, microVM tests, MSHV, shard 1 of 2
azure-mshv-2 nvx-public-proxy-h4tu2olo 2026-10-24 18:54Z 120957638702, microVM tests, MSHV, guests and sandboxes
azure-mshv-4 openvmm-shell-snapshot-r00c41xk 2026-10-24 18:54Z 120957286685, Platform, MSHV
azure-kvm-5 nvx-smp-snapshot-g_zyq8x1 2026-10-26 20:55Z 121838557038, microVM tests, KVM, shard 1 of 2
azure-kvm-1 nvx-console-snapshot-tecvbao1 2026-10-26 23:27Z 121911099990, microVM tests, KVM, shard 1 of 2
azure-kvm-5 nvx-smp-snapshot-23yjf35o 2026-10-26 23:27Z 121911099016, microVM tests, KVM, debug kernel
azure-mshv-5 nvx-endpoint-policy-ilkcz9xb 2026-10-26 23:27Z 121911099282, microVM tests, MSHV, shard 2 of 2
azure-kvm-3 openvmm-device-io-rrjbu4k8 2026-10-27 01:35Z 121951173833, Platform, KVM
azure-kvm-4 nvx-restore-processors-augqq4m7 2026-10-27 05:58Z 122034111191, microVM tests, KVM, debug kernel
azure-kvm-2 nvx-restore-downtime-duijv83o 2026-10-27 18:17Z 122377689293, microVM tests, KVM, shard 1 of 2
azure-kvm-3 openvmm-device-io-ep0lqoc8 2026-10-27 18:31Z 122385894460, Platform, KVM
azure-mshv-1 nvx-network-snapshot-rxh02rpf 2026-10-27 18:31Z 122385895095, microVM tests, MSHV, shard 1 of 2
azure-kvm-4 nvx-smp-snapshot-6vnh7pum 2026-10-27 18:31Z 122385894891, microVM tests, KVM, debug kernel
azure-kvm-5 nvx-platform-filesystem-xh935389 2026-10-27 21:51Z 122488000487, microVM tests, KVM, shard 2 of 2
azure-mshv-1 nvx-platform-network-policy-fcjyevtx 2026-10-27 22:06Z 122494839394, microVM tests, MSHV, shard 2 of 2

The log of the last one shows the sequence. The step was reported cancelled at 22:06:08Z, but the harness kept running: the scenario's directory changed at 22:06:09Z, and the orphan cleanup ended the harness and OpenVMM at 22:06:10Z.

2026-10-27T22:06:07.5640040Z Running microVM platform-snapshot network policy replacement on OpenVMM/mshv
2026-10-27T22:06:08.4039977Z ##[error]The operation was canceled.
2026-10-27T22:06:10.5337607Z Cleaning up orphan processes
2026-10-27T22:06:10.5579309Z Terminate orphan process: pid (1452196) (python3)
2026-10-27T22:06:10.5613531Z Terminate orphan process: pid (1452681) (sccache)
2026-10-27T22:06:10.5639517Z Terminate orphan process: pid (1454966) (openvmm)

Job 121951173833's log on azure-kvm-3 shows the same for the device I/O benchmark. None of these cancellations left a VM running: since #779, none of the 3,705 Linux jobs on the Azure runners has validate-runner's warning for an OpenVMM process among its annotations.

Where the cleanup is, and why it doesn't run:

  • The microVM scenarios remove their directories with tempfile.TemporaryDirectory context managers, such as this one, and the sandbox acceptances in run_acceptance. A killed harness runs neither. run_acceptance also cleans up only on Exception, so a KeyboardInterrupt would skip it as well.
  • Run NVX microVM tests on Linux runs python3 scripts/nvx.py test-microvm as a child of bash --noprofile --norc -e -o pipefail. The benchmark action's Linux steps, from Run Linux acceptance test on, run the harness among other commands.
  • OpenvmmProcess starts OpenVMM as a plain child, so nothing ends OpenVMM when the harness dies. InteractiveProcess(contain_process_tree=True) wouldn't on Linux either: its supervisor, process_supervisor.py, runs supervise_linux_process, a child subreaper that forwards SIGHUP, SIGINT, and SIGTERM but has no parent-death signal, so a killed harness leaves it running with its VM.
  • OpenvmmProcess also builds TimeAbiMonitor(command) after OpenVMM has started, outside its cleanup try, as #720 noted, although that constructor can't fail on a valid command.

Next steps

  • Let the cancellation reach the harness, so that its cleanup stops OpenVMM and removes its directories: exec the harness where it's a step's only command, as in Run NVX microVM tests on Linux, and forward SIGINT and SIGTERM to it where a step runs other commands, as the benchmark action's steps do. Cleanup that handles only Exception, such as run_acceptance's, must also run on KeyboardInterrupt.
  • Tie each OpenVMM process's lifetime to the harness on Linux, so that a harness that is killed outright, as by the orphan cleanup, takes its VMs with it: for example, a parent-death signal (PR_SET_PDEATHSIG) on OpenVMM or on the containment supervisor. The signal fires when the thread that started the child exits, not the process.
  • Remove the scratch directories that cancelled jobs leave behind, including the 64 already there: for example, have validate-runner delete its account's /tmp/nvx-* and /tmp/openvmm-* directories as it ends leftover OpenVMM processes, or create them under RUNNER_TEMP, which the runner empties at the start and end of each job.
  • Build TimeAbiMonitor before OpenVMM starts, or inside OpenvmmProcess's cleanup try.

Also carried over from #768, independent of the harness:

  • Optionally, record a host-change signal in runner validation, such as a digest of the Hyper-V KVP PhysicalHostName on the Azure runners, so that a live migration like azure-mshv-1's on 2026-10-18 shows up next to H4's result. #526's benchmark host record holds nothing that a live migration changes.

Related issues

  • #768: the H4 failures that #779 fixed, and the 8-day VM leak that this work would prevent.
  • #720 (closed): noted OpenvmmProcess's TimeAbiMonitor ordering.
  • #526 (closed): benchmark host-change detection, which a live migration bypasses.
Dominant language
Rust
Stars
758
Forks
48
Avg merge
18h 27m
Merged PRs (30d)
446

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from microsoft/nvx

All issues in microsoft/nvx

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.