Linux: container overlay mounts leak across a daemon restart — nothing can start until manually unmounted
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 42/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- linux, rust
- Domain
- infrastructure
Research direction
Start by tracing how the native Linux engine mounts container root filesystems and how daemon shutdown handles those mounts; the issue identifies /var/lib/kiri/containers/*/rootfs and reports leaks after systemctl restart kiri. Compare that lifecycle with the failed builder start described in issue #227. Confirm the fix by restarting the daemon and verifying containers start without manually unmounting their root filesystems.
Written by the indexing model from the issue text.
Description
Kiri version and system
0.2.0 (kiri-desktop-bin from the AUR) on CachyOS (Arch Linux), kernel 7.3.4-1-cachyos, systemd 262. Native Linux engine, no VM.
What you did
Have containers running (any containers — a plain permitio/pdp-v2, a registry:3, a privileged kind control-plane node), then restart the service:
$ sudo systemctl restart kiri
$ docker --context kiri start smartmix-control-plane kind-registry permit-pdp
What happened
Every previously-running container fails to start, with the same error as each other:
Error response from daemon: run 95faa159919846f67170028a855b1c4d9af254ec43a4ef4416ef8762f56d472d: mount /var/lib/kiri/containers/95faa159919846f67170028a855b1c4d9af254ec43a4ef4416ef8762f56d472d/rootfs: Resource busy (os error 16)
journalctl -u kiri:
container did not start id=c2641120bfe4 image=kindest/node:v1.32.2 ... error=run c2641120bfe4...: mount /var/lib/kiri/containers/c2641120bfe4.../rootfs: Resource busy (os error 16)
The reason is visible in /proc/mounts: the overlay mounts of the containers from the previous daemon run are still mounted in the host mount namespace after the daemon stopped, so the new daemon cannot mount the same rootfs again:
$ grep -c '/var/lib/kiri' /proc/mounts
3 # one overlay per previously-running container, still attached
restart: always containers do not come back on their own after this, either.
What you expected instead
What dockerd does: when the daemon stops, its container mounts are cleaned up (or at least reclaimed on start), and after a systemctl restart docker the containers start again (and restart: always containers come up on their own).
Anything else
Recovery that works for me:
$ sudo sh -c 'umount /var/lib/kiri/containers/*/rootfs'
$ docker --context kiri start smartmix-control-plane kind-registry permit-pdp # now works
Reproduced twice, deterministically, on two consecutive systemctl restart kiri cycles. The mounts appear to leak into the host namespace with the daemon gone, so nothing unmounts them when the containers stop.
Possibly the same mechanism as the daemon-spawned builder container in #227 ("Linux: every docker build fails — Kiri's internal builder container cannot start"), where a mount is left behind on a failed start and blocks the next attempt.
- Dominant language
- Rust
- Stars
- 7.5k
- Forks
- 329
- Avg merge
- 7h 48m
- Merged PRs (30d)
- 105
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from libracoder/kiri
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
libracoder/kiri#364 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
libracoder/kiri#289 ·
Maintainers usually reply within 1 day
-
bug
Difficulty 4/5 3-5 days Newbie friendliness 47/100
libracoder/kiri#365 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 35/100
libracoder/kiri#362 ·
Maintainers usually reply within 1 day
-
bug
Difficulty 3/5 1-2 days Newbie friendliness 47/100
libracoder/kiri#359 ·
Maintainers usually reply within 1 day
Similar issues
-
benchmark-review
Difficulty 2/5 1-3 hours Newbie friendliness 63/100
Maintainers usually reply within 1 day
-
area:filesystem bug need-triage
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
Captions panel does not offer to show an existing transcriptPossibly taken @thedavidweng claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
CapSoftware/Cap#4276 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
matrix-org/matrix-rust-sdk#7495 ·
Maintainers usually reply within 1 day