CI: maximize workflow parallelism and reduce turnaround time

Start by reading the linked workflow changes in PR #318, the existing OpenVMM pipeline commands, and CI documentation describing runner pools and artifact handoffs. Use the reference run and sampled timing baselines to identify a bounded workstream, then measure comparable full-matrix runs before and after. Done means the acceptance criteria are met without losing required coverage, fail-closed gates, or benchmark comparability.

Summary

Track changes that maximize GitHub Actions parallelism and reduce full CI turnaround without weakening test coverage, determinism, release gating, or performance comparability.

Recent full-matrix runs usually complete in about 24-27 minutes. The investigation found that the dominant constraint is serialized build and scenario work inside long jobs, not runner scheduling: sampled self-hosted jobs had a median queue time of only about 5-6 seconds.

Reference run: 37635396339.

Current baseline

Seven sampled successful full-matrix runs had a 25.6-minute median wall time; six completed in 23.5-26.5 minutes.

Lane	Median start	Median duration	Approximate finish
OpenVMM VMM tests / Windows / WHP	0.8m	22.4m	23.2m
OpenVMM binary / Windows, then NVX microVM / Windows	10.4m	12.3m after build	22.7m
OpenVMM unit tests / Windows / WHP	0.2m	18.6m	18.8m
OpenVMM binary / Windows, then platform benchmark	10.4m	9.5m after build	20.0m
OpenVMM VMM tests / Linux / KVM	0.7m	16.8m	17.5m

In the reference Windows VMM job, the test-openvmm step took 20.27 minutes, but Nextest executed its 29 selected tests in 288 seconds. Their individual durations sum to 821 seconds, so test execution is already parallelized; roughly 14-15 minutes are build and preparation work.

The standard NVX microVM command is sequential across 30 scenarios. In the reference Windows lane:

• Default Alpine suite: 7m45s
• Ubuntu smoke suite: 54s
• Azure Linux smoke suite: 60s
• Whole job median: 12.3m

Goals

• Reduce the median full-matrix workflow from about 25.6 minutes to no more than 20 minutes on an OpenVMM test-content cache miss.
• Target 12-15 minutes when the OpenVMM revision and exact reusable test bundle are cached.
• Preserve all required tests, supported platforms, failure diagnostics, and fail-closed behavior.
• Preserve full-matrix validation for dev pushes.
• Preserve benchmark host/series comparability; do not distribute samples for one benchmark result across hosts.
• Continue using same-run workflow artifacts, rather than caches, for required producer/consumer handoffs.

Workstreams

1. Preserve runner capacity and current fan-out

PR #318 moves several GitHub-hosted jobs onto one Linux PR utility runner and makes the debug-kernel job wait for shared artifacts. With one runner for that label, the current parallel artifact and debug-kernel builds would become serial, adding approximately 5-6 minutes on cold-cache runs.

• Before adopting that topology, provide at least two Linux PR utility runners or separate lightweight and artifact-builder labels with sufficient capacity.
• Prefer enough capacity to retain the current initial fan-out of quality, change detection, shared artifacts, and debug kernel.
• Record runner-pool capacity as a CI prerequisite and update stale documentation that says each backend has three runners.
• Measure queue time after every fan-out change; adding shards without available capacity is not useful.

2. Build OpenVMM VMM test content once and execute it in parallel

The pinned OpenVMM pipeline already supports vmm-tests-run --build-only and vmm-tests-run-target --dir --filter.

• Create one build-only producer per compatible target OS/architecture.
• Upload the complete revision-bound test content as a same-run workflow artifact.
• Run at least two duration-balanced Windows filter shards on separate WHP runners.
• Reuse one Linux build bundle for KVM and MSHV consumers if target and artifact compatibility are verified.
• Cache the exact built bundle by OpenVMM SHA, Rust version, target, feature/build mode, and bundle schema.
• Keep workflow artifacts as the authoritative same-run handoff when a cache misses.
• Give every shard unique failure-diagnostic artifact names, including run ID and attempt.
• Measure bundle size and upload/download overhead before fixing the final shard count.

3. Shard the sequential NVX microVM suite

• Emit machine-readable per-scenario setup and execution durations.
• Partition standard scenarios by observed duration, not count.
• Start with separate restore/snapshot/time-ABI and network/filesystem/lifecycle groups.
• Evaluate running Ubuntu and Azure Linux smoke coverage as an independent shard.
• Preserve each scenario exactly once per required guest, backend, kernel type, and processor-count configuration.
• Keep debug-kernel coverage and its platform rules intact.
• Aggregate shard results into the existing required status, release, and performance-persistence gates.

4. Remove backend-independent duplication and free hypervisor runners

The NVX CLI validation action ran 15 times in the reference dev workflow, consuming about 10.4 aggregate runner-minutes. test-openvmm-unit accepts no backend, while the KVM and MSHV matrix entries run the same Linux command.

• Run NVX CLI validation once per OS and make platform jobs depend on the matching result.
• Verify that OpenVMM unit/doc-test coverage is OS-dependent rather than backend-dependent.
• If equivalent, run OpenVMM unit/doc tests once on Linux and once on Windows instead of once per backend.
• Move backend-independent jobs to appropriately isolated utility runners with enough capacity.
• Do not shard OpenVMM unit execution itself: 4,469 tests took only 45 seconds in the reference Windows run; compilation dominates.

This intentionally revisits the backend duplication retained by #137 using current timing and command behavior.

5. Refine pull-request change classification

The current detector exposes only run-tests and run-workloads, so every non-documentation change can trigger binary builds, platform benchmarks, OpenVMM tests, and NVX microVM tests together.

• Evaluate separate fail-safe outputs for OpenVMM unit tests, OpenVMM VMM tests, NVX microVM tests, platform benchmarks, guest artifacts, and OpenVMM binaries.
• Default to the full matrix whenever classification is missing, ambiguous, or fails.
• Keep every dev push on the full required matrix.
• Add tests covering every path class and fallback behavior.
• Confirm that isolated CI/tooling/package changes retain all validation relevant to their transitive effects.

6. Reduce cold guest-artifact tail latency

The shared guest-artifact action restores or builds the kernel, Alpine, Ubuntu, Azure Linux, and Ubuntu distro layer sequentially. Warm runs take about 0.4 minutes, while the sampled cold run took 8.7 minutes.

• Measure misses by component over multiple runs.
• If cold misses materially affect p90/p99, split independent components into parallel producers.
• Make consumers depend only on the artifact components they use.
• Retain the separate debug-kernel lane and provenance/determinism checks.

Acceptance criteria

• Record before/after timing from at least five comparable successful full-matrix runs for each major topology change.
• Median full-matrix wall time is at most 20 minutes on comparable cache-miss runs, or improves by at least 20% with the remaining critical path documented.
• Exact OpenVMM test-bundle cache-hit runs target 12-15 minutes.
• No required scenario, backend, guest, kernel type, processor count, unit test, doctest, quality check, or benchmark gate is lost.
• Required-status, release, performance-gate, and performance-persistence behavior remains fail closed.
• Fork, draft, path-filter, cancellation, and rerun behavior remains correct.
• Self-hosted queue time and capacity are included in the results.
• CI documentation describes the final DAG, runner pools, shards, and artifact handoffs.

Non-goals

• Pinning jobs to individual runner names.
• Weakening assertions, reducing benchmark samples, or changing performance thresholds to improve elapsed time.
• Splitting samples for one performance result across different hosts.
• Treating Actions caches as reliable required handoffs.
• Adding runner capacity without adding runnable parallel work.

Related work

• #137 split OpenVMM VMM and unit suites into parallel matrices.
• #318 proposes routing hosted jobs to isolated Azure utility runners and changes artifact/debug-kernel ordering.