`unless-stopped` containers are not restarted after dockerd exits uncleanly: restore() marks the containers it stops as manually stopped

Open
  • Is something wrong?
#54,065 3 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@akasakariko is already working on this.

Since Oct 30, 2026.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
68/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
docker, go
Domain
backend

Research direction

Start by reading the restore path in daemon/daemon.go alongside shutdownContainer() and the HasBeenManuallyStopped assignment in daemon/kill.go; then check how daemon/internal/restartmanager/restartmanager.go uses that flag. Done means a container stopped during restore after an unclean daemon exit is not treated as manually stopped, and an unless-stopped container restarts as expected; verify that clean-shutdown behavior remains unchanged.

Written by the indexing model from the issue text.

Description

Description

When dockerd exits without a clean shutdown (crash, OOM kill, or SIGKILL from systemd after its stop timeout) with live-restore disabled, the container processes keep running under their shims. On the next start, restore() finds them alive and stops them. That stop goes through killWithSignal, which sets HasBeenManuallyStopped = true because the daemon is not shutting down. The restart manager then refuses to restart unless-stopped containers. They stay Exited (137) until someone starts them by hand. always containers come back.

A clean daemon restart is not affected. Shutdown() sets daemon.shutdown = true before stopping containers, so the flag is not set, and unless-stopped containers are restarted on the next start.

As a result, unless-stopped behaves like no in exactly the situation where automatic recovery matters most, even though nobody stopped these containers.

Code path (docker-v29.8.2):

Reproduce

containerd runs as its own process, as with the usual containerd.service + docker.service setup:

docker run -d --name us --restart unless-stopped busybox sleep 86400
docker run -d --name al --restart always busybox sleep 86400
kill -9 $(pidof dockerd)        # container processes keep running
dockerd ...                     # start the daemon again (systemd does this with Restart=always)
docker ps -a                    # "us" is Exited (137), "al" is running

Results on Docker 29.8.2 / containerd v2.3.6 (docker:29.8.2-dind, containerd started separately; full script and output below):

Scenario unless-stopped always
A. dockerd SIGKILL, live-restore off exited 137, HasBeenManuallyStopped=true restarted
B. clean dockerd restart (SIGTERM, wait), live-restore off restarted restarted
C. dockerd SIGKILL 6 s into shutdown (container ignores SIGTERM, stop-timeout 20 s), live-restore off exited 137, HasBeenManuallyStopped=true restarted
D. dockerd SIGKILL, live-restore on still running, same PID still running, same PID

The daemon logs the stop during restore:

level=debug msg="shutting down container considered alive by containerd" container=d2d12ab9… paused=false restarting=false running=true
Reproduction script 
#!/bin/sh
# Reproduction: unless-stopped containers after dockerd crash / restart (Docker 29.8.2, containerd 2.3.6)
set -u
log() { echo "[$(date -u +%H:%M:%S)] $*"; }
FLAGS="--containerd=/run/containerd/containerd.sock --storage-driver=vfs --iptables=false --ip6tables=false --bridge=none --debug"
start_dockerd() {
  dockerd $FLAGS "$@" >>/var/log/dockerd.log 2>&1 &
  for i in $(seq 1 120); do docker info >/dev/null 2>&1 && return 0; sleep 1; done
  log "dockerd did not become ready"; tail -40 /var/log/dockerd.log; exit 1
}
graceful_stop() { pid=$(pidof dockerd); kill -TERM "$pid"; while kill -0 "$pid" 2>/dev/null; do sleep 0.5; done; }
crash() { kill -KILL "$(pidof dockerd)"; sleep 1; }
show() {
  for c in "$@"; do
    id=$(docker inspect -f '{{.Id}}' "$c")
    m=$(grep -o '"HasBeenManuallyStopped":[a-z]*' /var/lib/docker/containers/$id/config.v2.json | cut -d: -f2)
    printf '    %-12s policy=%-15s status=%-8s exit=%-4s pid=%-6s HasBeenManuallyStopped=%s\n' "$c" \
      "$(docker inspect -f '{{.HostConfig.RestartPolicy.Name}}' "$c")" "$(docker inspect -f '{{.State.Status}}' "$c")" \
      "$(docker inspect -f '{{.State.ExitCode}}' "$c")" "$(docker inspect -f '{{.State.Pid}}' "$c")" "$m"
  done
}
pair() { # $1 prefix, rest: extra run args + command
  p=$1; shift
  docker run -d --network none --name "$p-us" --restart unless-stopped "$@" >/dev/null
  docker run -d --network none --name "$p-al" --restart always "$@" >/dev/null
}

containerd >/var/log/containerd.log 2>&1 &
sleep 2
start_dockerd
docker version --format 'Docker {{.Server.Version}}, containerd {{(index .Server.Components 1).Version}}'
docker pull -q busybox:1.37 >/dev/null && log "busybox pulled"

log "=== A. dockerd crash (SIGKILL), live-restore OFF ==="
pair A busybox:1.37 sleep 86400; show A-us A-al
crash; log "dockerd killed; container processes still alive: $(pgrep -f 'sleep 86400' | wc -l)"
start_dockerd; sleep 3; log "after dockerd restart:"; show A-us A-al

log "=== B. graceful dockerd restart (SIGTERM, waits for exit), live-restore OFF ==="
pair B busybox:1.37 sleep 86400
graceful_stop; log "dockerd stopped gracefully"; start_dockerd; sleep 3; log "after dockerd restart:"; show B-us B-al

log "=== C. dockerd killed mid-shutdown (like systemd TimeoutStopSec), live-restore OFF ==="
pair C --stop-timeout 20 busybox:1.37 sh -c 'trap "" TERM; while true; do sleep 1; done'
kill -TERM "$(pidof dockerd)"; sleep 6; crash; log "dockerd SIGKILLed 6 s into shutdown (containers ignore SIGTERM, stop-timeout 20 s)"
start_dockerd; sleep 3; log "after dockerd restart:"; show C-us C-al

log "=== D. dockerd crash (SIGKILL), live-restore ON ==="
graceful_stop; start_dockerd --live-restore; docker rm -f A-us A-al B-us B-al C-us C-al >/dev/null 2>&1
pair D busybox:1.37 sleep 86400; show D-us D-al
crash; start_dockerd --live-restore; sleep 3; log "after dockerd restart:"; show D-us D-al

log "=== daemon log lines about restore ==="
grep -E 'shutting down container considered alive|Loading containers: (start|done)' /var/log/dockerd.log | sed 's/^/    /' | tail -15
Full output 
Docker 29.8.2, containerd v2.3.6
[17:47:46] busybox pulled
[17:47:46] === A. dockerd crash (SIGKILL), live-restore OFF ===
    A-us         policy=unless-stopped  status=running  exit=0    pid=187    HasBeenManuallyStopped=false
    A-al         policy=always          status=running  exit=0    pid=243    HasBeenManuallyStopped=false
[17:47:51] dockerd killed; container processes still alive: 2
[17:48:07] after dockerd restart:
    A-us         policy=unless-stopped  status=exited   exit=137  pid=0      HasBeenManuallyStopped=true
    A-al         policy=always          status=running  exit=0    pid=533    HasBeenManuallyStopped=false
[17:48:08] === B. graceful dockerd restart (SIGTERM, waits for exit), live-restore OFF ===
[17:48:21] dockerd stopped gracefully
[17:48:27] after dockerd restart:
    B-us         policy=unless-stopped  status=running  exit=0    pid=1081   HasBeenManuallyStopped=false
    B-al         policy=always          status=running  exit=0    pid=1057   HasBeenManuallyStopped=false
[17:48:28] === C. dockerd killed mid-shutdown (like systemd TimeoutStopSec), live-restore OFF ===
[17:48:36] dockerd SIGKILLed 6 s into shutdown (containers ignore SIGTERM, stop-timeout 20 s)
[17:49:02] after dockerd restart:
    C-us         policy=unless-stopped  status=exited   exit=137  pid=0      HasBeenManuallyStopped=true
    C-al         policy=always          status=running  exit=0    pid=1822   HasBeenManuallyStopped=false
[17:49:03] === D. dockerd crash (SIGKILL), live-restore ON ===
    D-us         policy=unless-stopped  status=running  exit=0    pid=2504   HasBeenManuallyStopped=false
    D-al         policy=always          status=running  exit=0    pid=2560   HasBeenManuallyStopped=false
[17:49:35] after dockerd restart:
    D-us         policy=unless-stopped  status=running  exit=0    pid=2504   HasBeenManuallyStopped=false
    D-al         policy=always          status=running  exit=0    pid=2560   HasBeenManuallyStopped=false
[17:49:37] === daemon log lines about restore ===
    time="2026-10-29T17:47:51.830970074Z" level=debug msg="shutting down container considered alive by containerd" container=d2d12ab9381bfe111ef3d81f0848075d1e57b3c594d68ccbbcfd9bd14af2f403 paused=false restarting=false running=true
    time="2026-10-29T17:48:03.547977366Z" level=info msg="Loading containers: done."
    time="2026-10-29T17:48:22.241021412Z" level=info msg="Loading containers: start."
    time="2026-10-29T17:48:23.319430308Z" level=info msg="Loading containers: done."
    time="2026-10-29T17:48:37.787761376Z" level=info msg="Loading containers: start."
    time="2026-10-29T17:48:37.897514583Z" level=debug msg="shutting down container considered alive by containerd" container=e001e0be5f8c8942e74d11cdfe712d1165490f0ff59243d70f725dcea4181b53 paused=false restarting=false running=true
    time="2026-10-29T17:48:37.898568132Z" level=debug msg="shutting down container considered alive by containerd" container=66a413eb24575a727196dd6371c8198bc2c25fefdbfe0e2dce93f7130098a9c4 paused=false restarting=false running=true
    time="2026-10-29T17:48:37.915650455Z" level=debug msg="shutting down container considered alive by containerd" container=bffeb038730d0d377d0b71011a9b2916bf0bdd8a732cdf4a501b552e24802ac4 paused=false restarting=false running=true
    time="2026-10-29T17:48:37.917730223Z" level=debug msg="shutting down container considered alive by containerd" container=9b4ce6b32f07f5a28e6768ea714d65d057a0326ede38b3b5ec7a5bf4e4e8e52d paused=false restarting=false running=true
    time="2026-10-29T17:48:37.927318991Z" level=debug msg="shutting down container considered alive by containerd" container=1d1d00ab7af11c871bd235eaa273034ffffeb8586e6db373fe292ac60e6f93af paused=false restarting=false running=true
    time="2026-10-29T17:48:58.967052265Z" level=info msg="Loading containers: done."
    time="2026-10-29T17:49:24.137523014Z" level=info msg="Loading containers: start."
    time="2026-10-29T17:49:25.231805147Z" level=info msg="Loading containers: done."
    time="2026-10-29T17:49:31.405805634Z" level=info msg="Loading containers: start."
    time="2026-10-29T17:49:31.469102953Z" level=info msg="Loading containers: done."
Expected behavior

A stop that the daemon issues itself during restore(), for a container that survived an unclean daemon exit, should not count as a manual stop. unless-stopped containers should then be restarted, as they are after a clean daemon restart.

One possible approach: in the restore path, keep the container's previous HasBeenManuallyStopped value, or stop it through a path that does not set the flag. I have not checked whether other callers rely on the current behaviour.

Impact

We hit this in production. dockerd 29.6.1 aborted with BuildKit's concurrent map iteration and map write (moby/buildkit#6915, fixed in 29.7.0), and systemd restarted it about 30 s later. Two unless-stopped application containers stayed down, one of them for three weeks. Coolify, for example, uses unless-stopped for every application container it manages.

Scenario C suggests a second trigger. A package-driven or manual systemctl restart docker can hit the same path when a container's stop timeout is longer than systemd's stop timeout. The shipped docker.service does not set TimeoutStopSec, so systemd's default (90 s) applies, while the daemon's own shutdown timeout follows the containers' stop timeouts.

Related
  • docker/for-linux#652: the same symptom, with a comment describing a daemon crash.
  • #11065, #41302, #47792, and PR #53415: the same flag set by docker kill with non-terminating signals. That is a different trigger, and #53415 would not cover this path, because restore uses the container's stop signal.
  • #41817: an earlier change in the same restore code.
docker version / docker info

Output from the reproduction environment (docker:29.8.2-dind under rootless Podman, containerd started separately). Production is Ubuntu 26.04.1, kernel 7.0.0-38, Docker CE packages 29.8.2 / containerd.io 2.3.6. The incident happened on 29.6.1; the code path is the same on 29.8.2.

docker version / docker info 
$ docker version
Client:
 Version:           29.8.2
 API version:       1.56
 Go version:        go1.26.8
 Git commit:        7fc2dff
 Built:             Wed Sep 30 19:31:45 2026
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          29.8.2
  API version:      1.56 (minimum version 1.40)
  Go version:       go1.26.8
  Git commit:       8af9fe3
  Built:            Wed Sep 30 19:34:22 2026
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v2.3.6
  GitCommit:        ee2735368117d2eb259779949d5e75cdafec9761
 runc:
  Version:          1.5.2
  GitCommit:        v1.5.2-0-g29dd3dc
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

$ docker info
Client:
 Version:    29.8.2
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.37.2
    Path:     /usr/local/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v5.6.0
    Path:     /usr/local/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 29.8.2
 Storage Driver: vfs
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ee2735368117d2eb259779949d5e75cdafec9761
 runc version: v1.5.2-0-g29dd3dc
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 7.0.0-38-generic
 Operating System: Alpine Linux v3.24
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 14.97GiB
 Name: e3d850c20b2d
 ID: 8d8791e0-6c75-4e48-9d81-bc0fdeb492f0
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
  ::1/128
 Live Restore Enabled: false
 Product License: Community Engine
 Firewall Backend: iptables
  EnableUserlandProxy: true
  UserlandProxyPath: /usr/local/bin/docker-proxy
Dominant language
Go
Stars
72.2k
Forks
19.3k
Avg merge
1d 18h
Merged PRs (30d)
157

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from moby/moby

All issues in moby/moby

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.