Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

nvx-rs: replace the golden CLI scenarios with structured scenario tests

Open
  • Is something wrong?
#728 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Refactor
Clarity
Clearly specified
Activity status
Active
Tech stack
linux, python, rust
Domain
cli, testing-qa

Research direction

Start with tests/cli.rs, tests/fixtures/cli.json, and the replay in nvx_rs_reproduces_the_recorded_scenarios; inspect the shared setup and scenario records before designing the harness under tests/cli/. Map all 281 scenarios and listed regressions to the requested tests, then run the Linux and Windows suites and the specified mutation checks. Done means every scenario is covered and the behavior preserving pull request passes its gate; retirement is a later pull request.

Written by the indexing model from the issue text.

Description

Part of #722. Step 2. Its gate needs #726.

Goal

Replace tests/fixtures/cli.json and its replay, nvx_rs_reproduces_the_recorded_scenarios in tests/cli.rs, with scenario tests that check what callers rely on: exit statuses, error codes and stable fragments, the state directory and its parsed JSON files, file modes, and the arguments that OpenVMM receives, rather than transcripts recorded from Python.

Background
  • cli.json holds 141 POSIX and 140 Windows scenarios, in 426 steps, recorded from nvx.py sandbox, egress compile, and artifacts, with nvx-fake-openvmm as OpenVMM. Each step compares the exit status, standard output and error, and state files byte for byte, after replacing the root, process IDs, and random identifiers with placeholders.
  • On Windows, only the replay reaches 533 lines and 130 branches, mostly in src/host/cli/sandbox/ and src/testing/fake_openvmm.rs, and all of egress compile.
  • nvx.py sandbox passes the streams and status of nvx-rs through, and the hypervisor suites match message fragments and parse runtime.json and outcome reports.
Tasks

Replacement tests, in a pull request that doesn't change behavior:

  • A harness in tests/cli/ that reuses the root and fake-OpenVMM setup of tests/cli.rs, and checks exit statuses, error codes (#724) or stable fragments, the entries of the state directory, its parsed JSON files, POSIX file modes, and the arguments that the fake OpenVMM received.
  • Table-driven tests:
    • usage_errors_exit_2_and_name_the_option
    • validation_errors_exit_1_before_any_side_effect: the state directory is unchanged, and OpenVMM didn't run.
    • dry_runs_print_the_openvmm_command_and_launch_nothing: the >> line, split into arguments, holds the sandbox blocks, workload identity, shares, egress rules, and kernel command line.
    • one_shot_runs_exit_with_the_workload_or_openvmm_status: statuses such as 0, 3, 7, 124, 127, and 137, the workload's output byte for byte, and the schema of the outcome report.
    • lifecycles_of_every_configuration: the state after each of provision, start, exec, stop, and deprovision.
    • refusals_of_lifecycle_operations
    • configurations_of_every_format_start_and_exec, from hand-written config.json files of formats 1 to 6 in tests/fixtures/state/.
    • egress_compile_prints_the_library_lowering
    • artifacts_list_every_guest_in_order
    • A property that the >> line splits back into the arguments, with CommandLineToArgvW on Windows and a shell-word splitter on POSIX.
  • A temporary table that maps each of the 281 scenarios, and the regressions of #369, #373, #464, and #513, to tests, and a test that fails on a scenario that maps to none.
  • Keep the other tests of tests/cli.rs, including the Ctrl-C tests.

Retirement, in the next pull request:

  • Delete cli.json, the replay and its normalization helpers, and the mapping test.
  • Write only the newest config.json format, and keep reading formats 1 to 6. Drop the README's promise that an older nvx.py can operate a sandbox that nvx-rs provisioned.
  • Docs: replace the README's "Golden scenarios" section, and update its Testing section and doc/project-structure.md.
Gate
  • The lines and branches that only the replay reaches, on Windows and Linux (#726), are reached by the new tests or deleted.
  • cargo mutants on src/host/cli/sandbox/mod.rs, lifecycle.rs, and state.rs misses no mutant with the new tests that the replay catches.
  • Every scenario maps to a test.
Coordination
  • nvx-rs run (#638) gets its scenarios as tests of this harness, not as records from Python.
  • Until cli.json is deleted, changes such as those of #620 and #622 may edit its records by hand, but add no recorded scenarios.
Acceptance criteria
  • cli.json is gone, and the scenario tests pass on Linux and Windows.
Dominant language
Python
Stars
408
Forks
28
Avg merge
15h 15m
Merged PRs (30d)
322

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from microsoft/nvx

All issues in microsoft/nvx

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.