Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Firecracker backend: verify on KVM that forks of one tag never share a writable rootfs

Open
  • Is something wrong?
#317 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
linux, rust
Domain
backend, infrastructure

Research direction

Start by reading crates/forkd-vmm/src/lib.rs around BootConfig::ext4_rw and ext4_ro, then trace how restore_many_with handles the rootfs and how Snapshot records metadata. The issue asks maintainers to decide on the fix design before implementation, so first seek agreement on an option and any Snapshot changes. Done means restored children cannot concurrently corrupt a shared writable rootfs; verify with a concurrent-restore test.

Written by the indexing model from the issue text.

Description

backend:firecracker

Maintainer update, 2026-10-29: what this issue tracks now. The original report is kept unchanged below.

The code this report is about has left the tree. sandstorm-vmm, sandstorm-cli and restore_many_with were removed in #362, along with the rest of the v0.5 Firecracker runtime; they live on only in the v0.5.3 tag. #321, which fixed it on that path, was closed unmerged.

Firecracker is back as an optional node backend (sandstormd --backend firecracker, crates/sandstorm-core/src/node/firecracker.rs, #370, ADR 11). It is built so this bug cannot happen:

  • Every image carries its own rootfs.ext4, reflinked while the VM is paused, so memory and disk always come from the same moment.
  • Each sandbox gets a private reflinked copy in layers/<id>/disk/. The drive path in the vmstate is the relative rootfs.ext4, and Firecracker runs in that directory, so a restore opens its own copy and never the parent's.
  • A restore re-points the drive anyway before resuming, in the order #321 worked out: snapshot/load with resume_vm: false, then PATCH /drives/rootfs, then resume, and the VM is killed on any failure (tests a_load_repoints_the_disk_before_it_resumes, a_failed_repoint_never_resumes).
  • The backend refuses to start on a work directory that cannot reflink, rather than falling back to a shared or fully copied disk.

None of that has run on KVM yet. None of the project's machines has /dev/kvm. So this stays open as the check that the design really holds.

Done when, on a KVM host with --backend firecracker:

  • Fork one tag into N ≥ 2 children concurrently. lsof shows each Firecracker holding a different inode under its own layers/<id>/disk/rootfs.ext4, and none holds the template's or the tag's image.
  • Have every child write to the same paths for a while, then checkpoint and restore each one. Each sees only its own writes, and e2fsck -n is clean on every disk and on the tag's image.
  • The tag's rootfs.ext4 hashes the same before and after the run.
Original report (v0.5, forkd-vmm)

Restores are not safe to run concurrently against one snapshot, and the failure is silent corruption rather than an error. Filing with the evidence because it cost us ~12h to attribute, and because the fix is a design decision I would rather ask you about than assume.

Mechanism

  1. An rw bake boots the parent with rootfs_read_only: false (BootConfig::ext4_rw, crates/forkd-vmm/src/lib.rs:159; the ro variant ext4_ro at :139 sets true).
  2. Firecracker serialises the drive's path_on_host + is_read_only into the vmstate, and reopens that path verbatim on /snapshot/load.
  3. snapshot.json pins the rootfs as an absolute shared path, not per-snapshot or per-child:
{ "vmstate": ".../elixir-release/vmstate",
  "memory":  ".../elixir-release/memory.bin",
  "rootfs":  "/var/cache/forkd/elixir-release-tools-local.ext4" }
  1. restore_many_with copies/layers memory only — the v0.5 chain assembly stages memory-assembled.bin; the rootfs path is untouched.

So every restored child of a tag opens the same ext4 read-write. Two children, two guest kernels, one filesystem, no coordinator:

$ lsof | grep firecrack
firecrack 2588644 root 30u REG 252,1 12884901888 4628036 /var/cache/forkd/elixir-release-tools-local.ext4
firecrack 2600514 root 30u REG 252,1 12884901888 4628036 /var/cache/forkd/elixir-release-tools-local.ext4

Same inode, both 30u (read-write).

Observable damage

Random, different victims each run, and the base ext4 still hashes byte-identical to its source image in the same session — so nothing points at the image:

  • /usr/bin/uname holding another file's bytes — a valid ELF of plausible size, so an "it still executes" check passes; only an identity check catches it (uname --version → uniq (GNU coreutils) 9.1). /usr/bin/tr likewise.
  • libapt-pkg.so.6.0: invalid ELF header, libidn2.so.0: cannot dynamically load position-independent executable.
  • EBADMSG on /var/lib/apt/lists/* and /var/lib/dpkg/triggers/File — corrupt directory entries.
  • The ext4's mtime changing while no bake is running.

Two consequences worth separating:

  • Re-baking "fixes" it until the next concurrent job, which is what makes it look like a bad image rather than a shared writer. Restarting the service does not help either: long-lived pooled sandboxes are separate from the runner process, and it re-attaches to them, so the pool is the stale-content holder and nothing invalidates it.
  • ENOSPC is an amplifier, not a second cause. On a full filesystem the diff/CoW writes fail mid-block, which is why freeing disk appeared to fix it and why it returns the moment concurrency does.

On the documented non-goal

DESIGN-v0.5-diff-snapshot-chains.md (Non-goals) says chains apply to memory.bin only and that "the rootfs gets the apt / pip-installed bytes via the install step's writes to ext4; we keep copying the rootfs whole; filesystem CoW (overlayfs / btrfs) is a separate piece of work". The stated safety assumption — the rootfs is copied whole — is not what restore_many_with does; it copies nothing. Per-child FS CoW was deferred as an optimisation, but with n-way restore it is load-bearing for correctness, which moves it in priority rather than scope.

Worth noting the same root cause explains two other reports: from-image has no content identity to key a cache on (the tag string is all there is), and the warm pool has no key to evict against — so cache staleness and pool staleness are the same missing identity.

Options

1. Read-only base + per-child writable upper (my preference). Flip the bake to ext4_ro, give each child a private upper (tmpfs, or a small per-child ext4/erofs overlaid on the ro base), and let callers pass a capacity hint so a build can size its layer. No host-side per-spawn copy, so spawn latency is unchanged and disk stays bounded by the pool rather than by accumulated writes.

2. Host-side per-child reflink. reflink_copy the base per child (12.88 GiB apparent / ~5 GiB allocated here, so near-instant on XFS/btrfs), or a dm-snapshot/qcow2 diff. More general — writes land in a real file you can inspect and GC — but it materialises per spawn and costs a lot of disk at pool scale.

3. Interim invariant. Refuse to let a second live child of an rw-rootfs snapshot exist. I want to flag the cost honestly rather than present it as free: from-image prints fork --tag X -n N --per-child-netns with rw hardcoded, and restore_many_with is built to share the inode across n children, so this rejects n > 1 on rw snapshots — a behaviour break, not just a guard. It also needs the drive's ro/rw state at restore time, which Snapshot does not persist today (the vmstate carries it, but forkd never parses vmstate).

Asks

  1. Which option, or a different shape? I am happy to write the DESIGN amendment and implement, and I would rather have that decided before producing a diff you then have to redirect.
  2. Independently of the choice: should Snapshot persist the drive's read-only flag? Nothing in forkd metadata records it today, so any guard or diagnostic has to infer it from the .ext4 extension — and the daemon's bake path records no rootfs at all (snapshot_to returns rootfs: None).

Until one of these lands the only workable mode on a shared-rw tag is to keep a single writer, and nothing enforces that today. The caps that approximate it (a runner-side concurrency limit, a warm pool of one) cover only runner-driven sandboxes: anything spawning the same tag through the backend API, a CLI, or an agent is an extra writer regardless, so the rule is discipline rather than enforcement and I would not call it safe.

Dominant language
Rust
Stars
2.9k
Forks
227
No merged PRs in 30d

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from deeplethe/forkd

All issues in deeplethe/forkd

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.