Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

Firecracker backend: the 5 s API-socket wait under a burst of starts

Open
  • Is something wrong?
#301 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
28/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
rust
Domain
backend, infrastructure

Research direction

Start with Registry::reconcile() in state.rs and the wait_for_sock timeout and restore_many_with paths in lib.rs; the issue gives approximate line references for these entry points. Read the related tests and trace how restore failures and controller restart affect registered sandboxes. Done means meeting the five acceptance criteria, including per-VM failure reporting, orphan cleanup, and avoiding mass-pruning live VMs; a restart-recovery change is already in progress on another branch.

Written by the indexing model from the issue text.

Description

backend:firecracker bug

Maintainer update, 2026-10-29: what this issue tracks now. The original report is kept unchanged below.

Everything this report names is gone from dev. forkd-controller and its Registry::reconcile() were removed in #328. sandstorm-vmm with restore_many_with and wait_for_sock, the CLI, and the vendored Firecracker were removed in #362. They are only in the v0.5.3 tag.

Three of the four failure areas do not carry over to the node runtime:

  • Mass prune on restart: the master keeps desired state, and a node's VMs outlive sandstormd. They are re-adopted from bundles/<id>/, not pruned (ADR 11).
  • Orphans: each VM is in its own cgroup, so cgroup.kill takes all of it. A kill checks the socket path in /proc/<pid>/cmdline first, so it never hits another sandbox's Firecracker.
  • One failure losing the batch: each sandbox is created, restored and reported on its own, and a failed one gets its own error and retry.

What is still open is the socket wait itself, now in the new backend. Firecracker::spawn (crates/sandstorm-core/src/node/firecracker.rs) waits a fixed 5 s for the API socket and kills the VM if it is late. That is shorter than v0.5's 10 s, and nobody has measured it, because no project machine has /dev/kvm. "Restore several snapshots in parallel on one node" is still an open item in the README. Once it lands, it will put exactly the burst this issue saw onto that 5 s.

Done when, on a KVM host with --backend firecracker:

  • Time from spawn to API socket is recorded as a p50/p99 for one start, and for a burst of N starts on one node (N = 10, 50, 100).
  • The budget is chosen from those numbers, or scaled with the number of starts in flight, and a late socket logs how long it took.
  • A burst with no orphans and no leftover taps does not fail any start on the socket wait.
Original report (v0.5, forkd-vmm + forkd-controller)

Problem

Intermittent CI failures across the stack trace back to a multi-layer reliability gap in the forkd controller's snapshot restore path. The failure chain:

  1. restore_many_with fails — firecracker VMs fail to start during restore, logging socket /tmp/forkd-daemon-<tag>-<id>/child-N.sock never appeared within 10s (forkd-vmm wait_for_sock, lib.rs ~line 902). Observed to be systemic (affects multiple image tags, not one).
  2. Controller restart prunes ALL sandboxes — on startup, Registry::reconcile() (state.rs:213) prunes every sandbox entry whose PID is no longer alive. After a watchdog-triggered restart, every running sandbox is pruned (observed: 22 pruned in one incident), invalidating all upstream lease references in a single stroke.
  3. Orphaned firecracker processes survive backend restarts — when the upstream lease manager restarts, its Shutdown() path does not fully kill all sandbox VMs via the forkd API, leaving orphaned firecracker processes. These orphans cause resource contention (tap device conflicts, netns conflicts, fd limits) that prevents new restore_many_with operations from succeeding, producing a cascade.
Root cause analysis
  • The 10-second wait_for_sock timeout (Duration::from_secs(10) at lib.rs:980 and :1486) is a fixed budget with no load-awareness. Under contention from orphaned VMs or parallel restore bursts, firecracker processes take longer than 10s to open their API socket, and the wait bails out.
  • reconcile() is a blunt instrument: it cannot distinguish "VM died unexpectedly" from "VM is alive but the controller just restarted." On restart, all live VMs become unreconcilable (no live_vms handles can be reconstructed from registry data alone) and are pruned.
  • There is no pre-restore orphan detection: restore_many_with does not check whether firecracker processes from a prior (crashed) run still hold tap devices or netns slots before attempting a new restore.
Resource exhaustion ruled out

The host has ample capacity during failures: 62 GB RAM (58 GB available), 22 GB disk free, 32 GB tmpfs. The 22 firecracker processes and 4m37s CPU / 2.7 GB peak of the controller are well within limits. The failures are not resource exhaustion — they are startup-path contention and blunt-restart side effects.

Proposed fix areas

1. Restore-path resilience
  • Investigate why firecracker processes fail to open their API socket within 10s under load — tap/netns conflicts from orphaned VMs, fd limits, or controller-internal serialization.
  • Make the wait_for_sock timeout configurable and/or load-aware (longer budget under concurrent restore bursts), rather than a hardcoded 10s.
  • Add per-VM error reporting so partial restore_many_with failures surface which specific child failed, rather than a single bail that loses all children.
2. Graceful reconcile / restart recovery
  • Distinguish "controller crashed with surviving VMs" from "VM died" during reconcile(). Surviving firecracker processes with valid sockets are manageable orphans, not stale entries to prune — either reattach (if feasible) or kill targeted PIDs before pruning the registry.
  • (In progress on branch fix/controller-restart-recovery — kill_orphans() kills orphaned firecracker processes on startup after reconcile(), with PID-reuse guards and bounded death polling. Tracked separately; this issue covers the broader restore-path resilience.)
3. Orphan detection before restore
  • Before restore_many_with, detect and reap orphaned firecracker processes (processes not tracked by a live live_vms entry) holding tap devices or netns slots, so a new restore is not doomed by a prior crash's leftovers.
4. Watchdog: targeted recovery instead of mass kill
  • The operational watchdog (forkd-spawn-watchdog.sh, deployed on the server, not in this repo) currently pkill -9 firecracker — killing all running VMs, not just the failed ones. A single-VM restore failure becomes a full-pool outage.
  • A graduated response: first attempt soft restart of just the failed restore, only escalate to controller restart on repeated failures. Kill firecracker processes associated with the failed daemon dir only, not all firecracker processes.
  • (The watchdog script is operational tooling living outside this repo; this issue tracks the controller-side changes that would enable targeted recovery — per-VM error reporting, orphan detection, graceful reconcile.)

Acceptance criteria

  1. Single VM restore failure does NOT prune all running sandboxes
  2. restore_many_with reports per-VM failures so partial successes are not lost
  3. Orphaned firecracker processes from a prior crash are detected and reaped before a new restore is attempted
  4. wait_for_sock timeout is no longer a hardcoded 10s under load
  5. A controller restart does not mass-prune VMs that are still alive

Related

  • The upstream lease manager has a companion issue for its side of the cascade (runner retry on 410 Gone, graceful drain before restart, orphan detection on backend startup).
Dominant language
Rust
Stars
2.9k
Forks
227
No merged PRs in 30d

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from deeplethe/forkd

All issues in deeplethe/forkd

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.