Track guest managed agent bugs: output limit forwards later output after a dropped chunk

Leggi nvx-rs/src/guest/managed_agent/exec.rs, in particolare stream() e la gestione di Limited nel ciclo di esecuzione; confronta lo svuotamento delle pipe con il percorso delle sessioni perse. Poi esamina il banco di test loopback in nvx-rs/src/guest/managed_agent/loopback.rs e aggiungi un controllo di regressione che verifichi che l’output successivo a un chunk scartato non venga mai inoltrato. Il lavoro è completato quando ogni stream ricevuto rimane un prefisso dell’output del workload dopo il raggiungimento del limite di 1 MiB.

Bug-hunt tracking issue for the guest managed agent (nvx-rs/src/guest/managed_agent/), specifically how its exec loop streams workload output and enforces the 1 MiB output limit.

Baseline: origin/dev at 7eb9005f12afae31a9a63437133d05890256c134. Every permalink below uses this SHA, except the links into the deleted C agent, which use its last revision, d8316433c5e39bd215bddbeeeda71050868242b7.

Hunt: time-boxed to about 45 minutes, starting at 18:00 -03:00 on 2026-10-21. It stopped at about 18:25, when this bug was confirmed. No repository files were modified and nothing was committed. The reproduction ran in a scratch copy of nvx-rs outside the repository.

Selection: none of the bug-hunt issues covers the guest agent's exec loop. #733 (open) covers sandbox resource limits. #513 covers async output collection on the host, in aci_edge_sandboxes. #736 covers guest boot and the test harness, #720 the host VMM supervisor, and #464 the managed lifecycle's control pipe writes. The remaining closed ones cover other components, so this issue is new.

Findings

1. After the output limit is hit, the agent keeps forwarding later output, so the client receives a stream with a silent gap

Severity: Medium. The client receives corrupted output, not just truncated output: up to one 32 KiB read chunk goes missing from the middle of a stream, and later bytes of the same stream are joined onto the bytes before it. Nothing tells the client where the gap is. This happens on the documented output-limit path. The exit category (output-limit, status 125) is still correct, so only workloads that write more than 1 MiB are affected.

Where

• stream() handles a read chunk that would cross the cap at L467-L470. It drops the whole chunk, including the bytes that still fit, leaves output_bytes unchanged, and returns Limited.
• The exec loop (L104) drops a pipe only when it is Closed (L149-L159). Once a stream is Limited, the agent kills the workload, but read_output (L142-L148) keeps calling stream() on the same pipes on every later iteration, until EOF. So any later chunk that fits the remaining room is still sent to the host.
• After a lost session, the same closure switches to drain(), which discards output. The output-limited path has no such switch.
• The execution then ends with output-limit (L229-L230).
• The Rust agent inherited this from the C agent, which had the same cap check (nvx-managed-agent.c#L1628-L1631) and the same loop handling (L2035-L2038).
• The contract in doc/run.md#L1070-L1073 and 4.4-host-guest-control-interface.md#L48 says that exec forwards at most 1 MiB and that the agent kills a workload that writes more. The acceptance checks only check how many bytes arrive, not their content: the sandbox-lifecycle scenario (doc/run.md#L1126-L1128) and the loopback unit test (loopback.rs#L422-L429). That is why this has gone unnoticed.

Trigger: the combined stdout and stderr of a workload exceeds 1 MiB. One read, of up to 32 KiB, crosses the cap while R > 0 bytes of room are left, and the same pipe later yields a chunk of at most R bytes. That chunk can be bytes still buffered in the pipe when the agent kills the workload, or output from a descendant that survives the kill: sandbox mode kills only the process group.

Expected: for an output-limit execution, each stream that the client receives is a prefix of what the workload wrote to it, ideally filled up to the 1 MiB cap. Once the cap is hit, the agent forwards nothing more.

Actual: the agent drops the chunk that crosses the cap and kills the workload. It then forwards later output of the same stream that fits the remaining room, so bytes written after a missing chunk of up to 32 KiB reach the client.

Reproduction: the crate's end-to-end loopback harness, control_session_broker_and_real_agent_loop_back, runs the real direct-mode agent (dispatch::run) against the real host ControlSession through the loopback broker. In a scratch copy of nvx-rs outside the repository, I added the test below, which has the same setup. Its workload writes 1,048,476 a, sleeps 1 s so that the agent forwards all of them, leaving 100 bytes of room, and then writes 32,768 b and 50 c in one write():

head -c 1048476 /dev/zero | tr '\0' a; sleep 1; dd if=CHUNK bs=65536 count=1 2>/dev/null; sleep 10

$ cargo test --locked --all-features --lib bughunt_output_limit -- --nocapture
BUGHUNT exit=Exit { category: OutputLimit, status: 125 } stdout_len=1048526 a=1048476 b=0 c=50 stderr_len=0 tail="aaaaaaaaaacccccccccccccccccccccccccccccccccccccccccccccccccc"
assertion `left == right` failed: output after a dropped chunk was forwarded
  left: 50
  right: 0

All 5 of 5 runs gave this result. The agent's first read takes exactly 32 KiB of b, which crosses the cap, so it drops that read and kills the workload. On its next iteration, it reads the 50 c bytes that were still in the pipe, and they fit the 100 bytes of room, so it forwards them. The host's stdout ends …aaaa + cccc…, although the workload never wrote c right after a. A clean truncation would end in a, followed by at most 100 b.

#[test]
fn bughunt_output_limit_forwards_output_after_a_dropped_chunk() {
    if super::super::isolate_process_test(
        "NVX_AGENT_BUGHUNT_HOLE_CHILD",
        "guest::managed_agent::loopback::tests::bughunt_output_limit_forwards_output_after_a_dropped_chunk",
    ) {
        return;
    }
    let _process = super::super::PROCESS_TEST_LOCK.lock().unwrap();
    let (temporary, mut system) = TestSystem::with_cgroup();
    system.paths.hostfs_dir = temporary.path().join("hostfs");
    std::fs::create_dir_all(system.paths.hostfs_dir.join("root/0/source")).unwrap();
    system.paths.setpriv = setpriv_shim(temporary.path(), ":");
    system.spawner = Some(spawn_as_privileged);
    let target = temporary.path().join("mapped");
    let (broker_guest, agent) = UnixStream::pair().unwrap();
    let agent_descriptor = FileDescriptor::from_owned(agent.into());
    let agent_thread = thread::spawn(move || {
        let mut system = system;
        let config = AgentConfig {
            rootfs: c"-".to_owned(),
            hostname: c"loopback".to_owned(),
            uid: c"1000".to_owned(),
            gid: c"1000".to_owned(),
            user: c"workload".to_owned(),
            home: c"/".to_owned(),
            direct: true,
        };
        let mut session = Session::new(agent_descriptor);
        let mut mappings = HostMappings { expected: 1, ..HostMappings::default() };
        let outcome = dispatch::run(&mut system, &mut session, &config, &mut mappings).unwrap();
        (outcome, system)
    });
    let core = Arc::new(Mutex::new(BrokerCore::new(broker_guest)));
    let broker = BrokerHandle { core };
    let capability = [0x5a; CAPABILITY_LEN];
    let mut session = ControlSession::attach(broker.transport(), &capability, deadline()).unwrap();
    session.ping(deadline()).unwrap();
    session
        .map_host_paths(
            &[HostMap {
                source: "0/source".to_owned(),
                target: target.to_string_lossy().into_owned(),
                read_only: true,
            }],
            deadline(),
        )
        .unwrap();
    // 32 KiB of 'b', which is one full read of the agent, then 50 bytes of 'c', in one write.
    let chunk = temporary.path().join("chunk");
    let mut bytes = vec![b'b'; 32 * 1024];
    bytes.extend_from_slice(&[b'c'; 50]);
    std::fs::write(&chunk, &bytes).unwrap();
    let room = 100usize;
    let script = format!(
        "head -c {} /dev/zero | tr '\\0' a; sleep 1; dd if={} bs=65536 count=1 2>/dev/null; sleep 10",
        crate::protocol::MAX_OUTPUT_BYTES - room,
        chunk.display()
    );
    let workload = Workload {
        argv: vec!["/bin/sh".to_owned(), "-c".to_owned(), script],
        timeout_ms: 0,
        cwd: None,
        environment: WorkloadEnvironment::Default,
    };
    let (stdout, stderr, exit) = run_exec(&mut session, &workload);
    let count = |byte: u8| stdout.iter().filter(|&&each| each == byte).count();
    let tail = String::from_utf8_lossy(&stdout[stdout.len().saturating_sub(60)..]).into_owned();
    println!(
        "BUGHUNT exit={exit:?} stdout_len={} a={} b={} c={} stderr_len={} tail={tail:?}",
        stdout.len(), count(b'a'), count(b'b'), count(b'c'), stderr.len()
    );
    session.stop(deadline()).unwrap();
    drop(session);
    let _ = agent_thread.join().unwrap();
    // A clean truncation leaves a prefix of what the workload wrote: 'a's, then at most
    // `room` 'b's. Bytes of 'c' follow 32 KiB of 'b' that the host never received.
    assert_eq!(count(b'c'), 0, "output after a dropped chunk was forwarded");
}

Suggested fix (small and local, which suits a first contribution):

• Make the limit end forwarding. Once either stream returns Limited, read what is left on both pipes with drain(), as the lost-session path already does, for example Some(pipe) if readable && (session_lost || output_limited) => drain(pipe).
• Keep output_limited set once it is set (output_limited |= …). The plain assignment at L156-L157 would clear an earlier Limited if a later iteration returned only Failed.
• Optionally, forward the part of the crossing chunk that still fits, so that the client receives exactly 1 MiB. The acceptance wording in doc/run.md ("more than 1 MiB less one 32 KiB read chunk") would then need an update.
• Add a regression test along the lines of the one above that requires each stream to be a prefix of the workload's output. Consider checking content, not only length, in sandbox-lifecycle too.

Explored and verified (no bug)

1. Host NVXS role and control session (#641, #680): HostConnection keeps partial records across timeouts, and HostSession enforces sequence and identity. I checked both against OpenVMM's broker at the pinned gitlink 9f04d05c. It numbers each attachment from WAIT at sequence 0 and READY at 1, refuses a bad capability by closing the stream, and defines ERROR with a 4-byte code, all of which the new strict host role matches. The CLI Session matches the deleted Python control_session.py, including the single deadline for a whole exec.
2. Guest NVXS role: the window and the credit are checked before any state changes. RecordParser::needed() never returns 0 inside a record, so GuestConnection never reads past the record in progress, and polling stays accurate. Dispatch ends the session on every DataFailure, so no credit leaks.
3. Execution::pump: its poll slices, deadlines, and cancellation are correct. Each execution has its own session, so no stale events cross requests.
4. Platform transports: Linux clamps socket timeouts to at least 1 ms. Windows handles the race between an overlapped pipe operation and its cancellation, and clamps waits to at least 1 ms.
5. Host mapping chunking (encode_maps_payloads): every entry is checked against the header and the limit before a payload is flushed.
6. Egress policy compiler, including the #682 refactor of legacy rules: I compared it with the deleted Python egress_policy.py on 3,028 random cases, 4,000 cases biased toward valid policies, and 6,000 byte-mutated JSON cases. Results and error messages never diverged.
7. #681 and #679: the time probe's invalid-option diagnostics and the shared guest main behave as before.
8. Guest cgroup handles (#655): exclusive creation, removals that take turns, and parsing of canonical pids.current values.
9. Filter for performance history resets (#658): history files are only ever appended to, so a drop in rows means a reset, and the side-branch and merge cases are handled.
10. #664 Ubuntu manifest mismatch messages: no expected value can be None, so the behavior is unchanged.

All 398 nvx-rs library tests pass on Windows, and the existing loopback test passes on Linux under WSL2.

Checked, not filed

• Deadline::after in nvx-rs/src/host/cli/deadline.rs turns negative or NaN seconds into a deadline that never passes, although its documentation mentions only overflow. This can't happen: --timeout and the exec response timeout are both validated as finite and positive.
• openvmm_arguments returns no options for a posture that denies all egress, ingress, and host loopback, without validating the deny rules. Invalid deny rules, or more than 256 of them, are accepted silently, but no network device is attached then, so they have no effect. Low impact.
• cli::session::Session::reply doesn't close the session on a mismatched request ID. Each CLI command uses a fresh session and exits on an error, so this has no impact.

Validation limits

• I reproduced the bug on Linux under WSL2 (Ubuntu, kernel 6.18.33.2-microsoft-standard-WSL2, Rust 1.95.0) through the crate's loopback harness. It used the real agent loop, real /bin/sh workloads, a fake cgroup tree (TestSystem::with_cgroup), and the setpriv shim, and ran in direct mode only. Sandbox mode uses the same stream() and loop, but I didn't run it.
• I did not reproduce it in a microVM: no KVM, MSHV, or WHP scenario ran. I neither implemented nor tested the fix.