Two VMs created at the same moment by two processes share a shadow root, and disposing one deletes the other's files

Read create_vm_shadow_root in crates/native-sidecar/src/vm.rs and compare its directory allocation with create_vm_unix_socket_host_dir. The issue says the fix uses 16 random bytes and exclusive create_dir, retrying on AlreadyExists; it also names a unit test that checks concurrent allocations. Done means each VM gets a distinct shadow root and concurrent creation passes that test. Pull request #2093 is already open.

Written by an AI agent (Claude) on behalf of @jasonkuhrt.

Problem

On macOS, two VMs created at the same moment by two processes, each with its own sidecar, can get the same host shadow root. When one VM is disposed, the other VM's files are deleted: its next read of a file it wrote fails with ENOENT.

We hit this on 0.4.6 in integration tests that run in parallel processes: a VM failed with kernel_error: ENOENT: storage stat for '/workspace/export.ts' failed a few calls after writing that file.

Reproduction

macOS 27.2 arm64, Node v26.13.0, @rivet-dev/agentos-core 0.4.7 with its runtime sidecar. The sidecar's code is the same on main 785e6441: v0.4.7..main changes only docs and one example.

repro.mjs starts two processes at once, each creating one VM, 20 times. Each VM writes a file named after its process. VM 0 is disposed after 0.2 s, then VM 1 reads its own file back.

// Usage: npm install @rivet-dev/[email protected] && node repro.mjs
import { fork } from 'node:child_process';
import { existsSync, readdirSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';

const R = Number(process.env.R ?? 20);
const script = new URL(import.meta.url).pathname;

if (process.argv[2] === 'child') {
  const { AgentOs } = await import('@rivet-dev/agentos-core');
  process.send('ready');
  await new Promise((resolve) => process.once('message', resolve));

  const vm = await AgentOs.create();
  const file = `workspace/owner-${process.pid}.txt`;
  await vm.filesystem.writeFile(`/${file}`, String(process.pid));
  const roots = readdirSync(tmpdir()).filter(
    (name) =>
      name.startsWith('agentos-native-sidecar-shadow-') &&
      existsSync(join(tmpdir(), name, file)),
  );
  let outcome = 'ok';
  if (process.argv[3] === '0') {
    await new Promise((resolve) => setTimeout(resolve, 200));
  } else {
    await new Promise((resolve) => setTimeout(resolve, 1000));
    try {
      await vm.filesystem.readFile(`/${file}`);
    } catch (error) {
      outcome = error.message;
    }
  }
  await vm.dispose();
  process.send({ roots, outcome }, () => process.exit(0));
} else {
  let shared = 0;
  let failed = 0;
  for (let round = 1; round <= R; round++) {
    const children = [0, 1].map((i) => fork(script, ['child', String(i)]));
    await Promise.all(
      children.map((child) => new Promise((resolve) => child.once('message', resolve))),
    );
    const reports = Promise.all(
      children.map(
        (child) =>
          new Promise((resolve) =>
            child.once('message', (report) => resolve({ pid: child.pid, ...report })),
          ),
      ),
    );
    for (const child of children) child.send('go');
    const [vm0, vm1] = await reports;
    if (vm0.roots.length !== 1 || vm1.roots.length !== 1) {
      throw new Error(`round ${round}: expected one shadow root per VM: ${JSON.stringify([vm0, vm1])}`);
    }
    const sameRoot = vm0.roots[0] === vm1.roots[0];
    if (sameRoot) shared++;
    if (vm1.outcome !== 'ok') failed++;
    console.log(
      `round ${round}: ${sameRoot ? `one shadow root, ${vm0.roots[0]}` : 'two shadow roots'}`,
    );
    if (vm1.outcome !== 'ok') console.log(`  VM 1 read its own file: ${vm1.outcome}`);
  }
  console.log(`${shared} of ${R} rounds shared a shadow root; VM 1 failed in ${failed}`);
  process.exitCode = shared || failed ? 1 : 0;
}

$ node repro.mjs
round 1: two shadow roots
…
round 6: one shadow root, agentos-native-sidecar-shadow-vm-1-1793891804462806000
  VM 1 read its own file: sidecar rejected request 8: kernel_error: ENOENT: no such file: /workspace/owner-93650.txt
round 7: two shadow roots
round 8: one shadow root, agentos-native-sidecar-shadow-vm-1-1793891807777946000
  VM 1 read its own file: sidecar rejected request 8: kernel_error: ENOENT: no such file: /workspace/owner-93673.txt
…
4 of 20 rounds shared a shadow root; VM 1 failed in 4

Sidecar	Rounds sharing a shadow root	Rounds where VM 1 got ENOENT
published 0.4.7	4 of 20	4 of 20, the same rounds
main with the fix below, via AGENTOS_SIDECAR_BIN	0 of 20	0 of 20

How many rounds share a root varies between runs (4 to 10 of 20 in ours); VM 1 failed in exactly the shared rounds every time.

Cause

create_vm_shadow_root names the directory after the VM's id and the clock, and adopts a directory that already exists (vm.rs#L4576-L4585):

fn create_vm_shadow_root(vm_id: &str) -> Result<PathBuf, SidecarError> {
    let nonce = SystemTime::now()
        .duration_since(UNIX_EPOCH)
        .map_err(|error| SidecarError::Io(format!("failed to compute shadow-root nonce: {error}")))?
        .as_nanos();
    let root = std::env::temp_dir().join(format!("agentos-native-sidecar-shadow-{vm_id}-{nonce}"));
    fs::create_dir_all(&root)
        .map_err(|error| SidecarError::Io(format!("failed to create VM shadow root: {error}")))?;
    initialize_vm_shadow_root(root)
}

• Every sidecar process numbers its VMs from vm-1.
• macOS's clock counts microseconds, so nonce ends in 000 and two processes can read the same value.
• create_dir_all succeeds on a directory that already exists.

We have not run this on Linux; the name is still not unique by construction there.

Two VMs that share the directory then lose each other's files:

sequenceDiagram
    participant V0 as VM 0 (sidecar process 1)
    participant D as $TMPDIR/agentos-native-sidecar-shadow-vm-1-1793891804462806000
    participant V1 as VM 1 (sidecar process 2)
    V0->>D: create_vm_shadow_root("vm-1")
    V1->>D: create_vm_shadow_root("vm-1") in the same microsecond, adopted
    V0->>D: writeFile /workspace/owner-A.txt
    V1->>D: writeFile /workspace/owner-B.txt
    V0->>D: dispose, remove_dir_all(&vm.cwd)
    V1->>D: readFile /workspace/owner-B.txt
    Note over V1,D: sync_vm_shadow_root_to_kernel finds the shadow copy gone
    Note over V1,D: propagate_shadow_deletions_to_kernel deletes the kernel file
    D-->>V1: ENOENT

• Disposal removes the whole directory: vm.rs#L1364.
• A read reconciles first: sync_guest_filesystem_shadow_before_call runs sync_active_shadow_path_to_kernel for ReadFile, Stat and similar calls, and its root walk ends in propagate_shadow_deletions_to_kernel.

The same collision shows inside one process. A unit test that releases 16 threads at once into create_vm_shadow_root("vm-1") fails on main in its first round on this Mac:

$ cargo test -p agentos-native-sidecar --lib create_vm_shadow_root_gives_vms
assertion `left == right` failed: VMs with the same ID must have distinct shadow roots in round 0: [Ok("/private/var/folders/…/T/agentos-native-sidecar-shadow-vm-1-1793…"), …]
  left: 14
 right: 16

Fix

The fix names the shadow root with 16 random bytes and creates it with an exclusive create_dir, retrying on AlreadyExists. That is how create_vm_unix_socket_host_dir already allocates each VM's Unix socket directory. Fix: #2093.