orchestrator: NBDProvider.Close() syncs device before disconnecting, causing spurious EIO on sandbox cleanup after VM crash
Maintainers usually reply within 3 days
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 75/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Quiet
- Tech stack
- go
- Domain
- infrastructure
Research direction
Start in packages/orchestrator/pkg/sandbox/rootfs/nbd.go, focusing on NBDProvider.Close() and sync(). Verify cleanup behavior after a VM crash: EIO from sync() should be logged at WARN without propagating as a cleanup error, while other sync errors should still propagate.
Written by the indexing model from the issue text.
Description
Describe the bug
When a Firecracker VM exits abnormally (OOM-kill, crash, etc.), NBDProvider.Close() in packages/orchestrator/pkg/sandbox/rootfs/nbd.go calls sync() before mnt.Close(). Because the NBD device is already in an error state after the VM dies, the ioctl(BLKFLSBUF) + fsync inside sync() returns EIO. This causes every sandbox cleanup after an abnormal VM exit to log:
failed to cleanup sandbox: error flushing cow device: failed to fsync path: input/output error
even though no data loss has occurred — the error is an expected consequence of the VM being dead, not a real storage failure.
Root cause
NBDProvider.Close() (line 130 of rootfs/nbd.go):
func (o *NBDProvider) Close(ctx context.Context) error {
var errs []error
err := o.sync(ctx) // ← calls BLKFLSBUF ioctl + fsync on /dev/nbdX
if err != nil {
errs = append(errs, fmt.Errorf("error flushing cow device: %w", err))
}
err = o.mnt.Close(ctx) // ← disconnects the NBD device (too late)
...
}
sync() (line 162) opens /dev/nbdX, issues BLKFLSBUF, then calls flush() which does syscall.Fsync. When the VM has already crashed, the NBD device is in error state and Fsync returns syscall.EIO. The error propagates up through Sandbox.Close() → setupSandboxLifecycle() which logs it at ERROR level.
Expected behavior
EIO from sync() during sandbox cleanup should be treated as an expected condition (VM died, no data to flush) and logged at WARN level rather than propagating as an error that triggers the "failed to cleanup sandbox" ERROR log.
Impact
- Spurious
"failed to cleanup sandbox"ERROR log on every abnormal VM exit - Noise in production alerting / on-call signals
Suggested fix
In NBDProvider.Close(), detect syscall.EIO from sync() and downgrade to a warning:
err := o.sync(ctx)
if err != nil {
if errors.Is(err, syscall.EIO) {
logger.L().Warn(ctx, "error flushing cow device (VM likely crashed, ignoring)", zap.Error(err))
} else {
errs = append(errs, fmt.Errorf("error flushing cow device: %w", err))
}
}
Environment
- Bare-metal deployment with Nomad
- Observed under high sandbox creation load (thundering herd from client retries)
- Dominant language
- Go
- Stars
- 1.6k
- Forks
- 448
- PR merge metrics
- No merged PRs in 30d
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from e2b-dev/runtime
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
Maintainers usually reply within 3 days
-
sandbox cache: StartRemoving state transition not broadcast, all allocations see stale Running stateOpen
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
Maintainers usually reply within 3 days
-
Difficulty 1/5 Under an hour Newbie friendliness 86/100
Maintainers usually reply within 3 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
Maintainers usually reply within 3 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
Maintainers usually reply within 3 days
Similar issues
-
flaky-test
Difficulty 1/5 Under an hour Newbie friendliness 88/100
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 88/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
Maintainers usually reply within 2 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
SocialGouv/iterion#2108 ·
Maintainers usually reply within 1 day