Linux: container overlay mounts leak across a daemon restart — nothing can start until manually unmounted

Open
#230 3 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
42/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
linux, rust

Research direction

Start by tracing how the native Linux engine mounts container root filesystems and how daemon shutdown handles those mounts; the issue identifies /var/lib/kiri/containers/*/rootfs and reports leaks after systemctl restart kiri. Compare that lifecycle with the failed builder start described in issue #227. Confirm the fix by restarting the daemon and verifying containers start without manually unmounting their root filesystems.

Written by the indexing model from the issue text.

Description

Kiri version and system

0.2.0 (kiri-desktop-bin from the AUR) on CachyOS (Arch Linux), kernel 7.3.4-1-cachyos, systemd 262. Native Linux engine, no VM.

What you did

Have containers running (any containers — a plain permitio/pdp-v2, a registry:3, a privileged kind control-plane node), then restart the service:

$ sudo systemctl restart kiri
$ docker --context kiri start smartmix-control-plane kind-registry permit-pdp
What happened

Every previously-running container fails to start, with the same error as each other:

Error response from daemon: run 95faa159919846f67170028a855b1c4d9af254ec43a4ef4416ef8762f56d472d: mount /var/lib/kiri/containers/95faa159919846f67170028a855b1c4d9af254ec43a4ef4416ef8762f56d472d/rootfs: Resource busy (os error 16)

journalctl -u kiri:

container did not start id=c2641120bfe4 image=kindest/node:v1.32.2 ... error=run c2641120bfe4...: mount /var/lib/kiri/containers/c2641120bfe4.../rootfs: Resource busy (os error 16)

The reason is visible in /proc/mounts: the overlay mounts of the containers from the previous daemon run are still mounted in the host mount namespace after the daemon stopped, so the new daemon cannot mount the same rootfs again:

$ grep -c '/var/lib/kiri' /proc/mounts
3      # one overlay per previously-running container, still attached

restart: always containers do not come back on their own after this, either.

What you expected instead

What dockerd does: when the daemon stops, its container mounts are cleaned up (or at least reclaimed on start), and after a systemctl restart docker the containers start again (and restart: always containers come up on their own).

Anything else

Recovery that works for me:

$ sudo sh -c 'umount /var/lib/kiri/containers/*/rootfs'
$ docker --context kiri start smartmix-control-plane kind-registry permit-pdp   # now works

Reproduced twice, deterministically, on two consecutive systemctl restart kiri cycles. The mounts appear to leak into the host namespace with the daemon gone, so nothing unmounts them when the containers stop.

Possibly the same mechanism as the daemon-spawned builder container in #227 ("Linux: every docker build fails — Kiri's internal builder container cannot start"), where a mount is left behind on a failed start and blocks the next attempt.

Dominant language
Rust
Stars
7.5k
Forks
329
Avg merge
7h 48m
Merged PRs (30d)
105

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from libracoder/kiri

All issues in libracoder/kiri

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.