Manage Docker containers as engines — Berth-owned start/stop + keep-durably-out when the card is claimed

Start by reading issue #96 and the existing engine configuration and lease drain/grant flow; the payload does not name source files or tests. Decide how container lifecycle commands, Compose services, restart policies, and unload-only behavior fit the engine model, then implement and test the chosen design. Done means a container can be kept stopped while another tenant holds the card and is accounted for through its engine entry.

The problem

Berth's value comes from owning the card's admission: a grant, a drain, a claim.
But it can only govern workloads it can control — either a child process it launched
(berth run) or, since the design being queued for #96, an engine that exposes its own
load_url/unload_url.

A Docker container that allocates VRAM on its own schedule is in the blind spot. Concretely,
on my box the immich-pet-tagger container (a scheduled image-embedding service,
restart: unless-stopped) loads ~2 GB of CUDA models on every startup and on schedule (in example every ~20 min) and unloads them when the cycle ends. Berth sees that VRAM — it shows up in the stowaway list — but has no control channel to it. Two consequences, both measured:

1. The card is not gated. An unleased container will load its models the moment its own
   scheduler fires, regardless of what else is using the card. With the 27 B model fully
   resident (15,330 MiB used / 664 MiB free — the hardest state on this 16,303 MiB card), a
   live manual trigger produced this (2 s cadence):

   t	VRAM	event
   11:13:12	15,330	trigger; 27 B serving an active 58,907-token prompt
   11:13:22–11:14:14	15,483 → 15,567 peak	tagger's YOLO (2 s) + CLIP (45 s) load; 27 B yields ~0.75 GB
   11:14:15	—	STATS clean, no OOM, no errors
   11:14:18	11,043 (4,953 free)	tagger models released; 27 B still displaced (~4.3 GB in RAM)
   11:14:55	15,301, util 90–92 %	27 B re-paginated in <2 s

   It's self-healing — but "coexists by displacement" is not the same as "the card is
   protected." For a gaming session or a long LLM batch job (OCR, transcribe, tag-a-disk)
   the operator would want the opposite: *while the card is held by another tenant, this workload
   stays out of VRAM. That would be the value of Berth in this scenario. * There is no configuration
   that gives that today; the stowaway list is report-only.
   While self-healing it still causes effectively system lock-up for a while (unresponsive desktop, stuttering,
   possible timeouts of certain jobs and Berth would be here valuable solution to remove manual lifecycle
   management.

2. Manual start/stop is still needed. Killing the berth run wrapper releases only the
   lease; it does not stop the container (daemon-owned, unless-stopped, compose up
   without -d only attaches logs). So the operator is back to hand-running docker stop /
   docker start to keep a stowaway workload out — exactly the manual step this feature removes.

The precedent this builds on (llama-swap + #96)

The llama-swap wiring works because the drain is the engine's own exit: Berth holds the
lease and, on a competing grant, calls llama-swap's own unload endpoint
(/api/models/unload/<model>) — it never signals the process. The 27 B unloads, VRAM frees,
the next engine is admitted.

Issue #96 generalizes this and the maintainer has confirmed the shape: Berth never signals a
process it did not start; it uses the engine's own way out (a load_url/unload_url on the
[[engine]] entry, plus a match list so the engine's VRAM is counted as the engine's, not a
stowaway).

A Docker container fits that design exactly — its "own way out" is the container runtime's
documented lifecycle:

• unload = docker stop <name> (or docker compose -f <file> stop <svc>) — sends
  SIGTERM, the container's graceful path, releases the CUDA context and its VRAM.
• load = docker start <name> (or docker compose -f <file> up -d <svc>) — brings it back.

docker stop is a graceful stop, not a kill -9; the container can trap it and clean up
(unload models), so it is the container's own exit. Berth invokes the runtime's lifecycle API —
it does not signal an arbitrary PID. This is the same principle as the llama-swap endpoint, with
a shell command standing in for the HTTP URL.

One load-bearing Docker behavior that makes this a stable knob (per the documented restart
policies): with restart: unless-stopped, a container that is explicitly stopped via the
CLI/API is not auto-restarted. So when Berth runs docker stop, the container stays stopped
until Berth (or the operator) starts it again. That is what lets Berth durably keep a workload
out of the card while a competing tenant holds it — instead of a transient stop that the
daemon immediately reverts. (Contrast restart: always, which would revert an explicit stop;
see "Open questions".)

How this could work

Extend the [[engine]] entry so an engine's load/unload can be commands, not only HTTP
URLs — with a kind = "docker" / kind = "compose" preset that fills them in. Then a container
is governed like any other engine.

Sketch (there may be better ways to do it via docker desktop API?):

[[engine]]
name = "pet-tagger"
kind = "docker"                 # or "compose"; sets load/unload to the lifecycle below
# --- docker lifecycle (the "own way out") ---
name_docker = "immich-pet-tagger"
# load   -> docker start immich-pet-tagger        (or: docker compose -f F:/AI/immich-pet-tagger/docker-compose.yml up -d immich-pet-tagger)
# unload -> docker stop  immich-pet-tagger        (or: docker compose ... stop immich-pet-tagger)
# --- accounting (the #96 "match") ---
match = ["immich-pet-tagger"]   # process / container identity whose VRAM is the engine's, not a stowaway
needs_mib = 2560                # measured peak (see evidence); drives admission math
# --- optional ---
# ready = "..."                 # readiness probe after load (e.g. the app's HTTP port)

Behaviour that follows:

1. Per-container load/unload controlled by Berth. On a grant, Berth runs load (start the
   container); on a drain (a higher-priority grant, or a whole-card claim), Berth runs unload
   (docker stop), waits for the driver to show the memory free, and only then releases the
   lease / admits the next tenant. Symmetric to llama-swap.
2. Admission gating (the "keep out while busy" the operator actually wants). While a
   whole-card claim/hold (e.g. a game) or a higher-priority lease is active, a drained Docker
   engine stays stopped — Berth does not restart it on its own schedule, because the card is
   held. On release, it reloads only if its own lease/claim is granted. This is what makes
   "card busy with gaming/OCR → tagger stays out of VRAM" actually hold, rather than
   "coexists by displacement."
3. match accounting. The container's VRAM is attributed to its engine entry (counted as the
   engine's, with needs_mib), not buried in the stowaway list — so the stowaway list stays
   honest and the admission math sees the real number.

The design deliberately reuses #96's invariants:

• Berth never signals a process it didn't start — it runs the container runtime's own lifecycle.

• If unload is refused or the memory doesn't come down, Berth waits for the next boundary and
  reports the lease stuck; it never grants the card on top of a loaded model.

  I am not yet sure how it would ideally work in practice, but due to different nature of ML containers

  • some of them load models on startup some only lazy-load - it would be useful to have "unload-only" option
  • where Berth evicts a certain container, but does not restart it when memory is free.

Evidence

• Displacement coexistence, hardest state (unleased): peak 15,567 MiB < 16,303 card, no OOM,
  no lockup, no stuck state; the 27 B kept serving through the whole scan window; ~4.3 GB of its
  weights went to RAM for the ~65 s scan; re-pagination <2 s. So the coexistence is safe — the
  missing piece is the gate, which this feature adds.
• Leased run works end-to-end: berth run --as pet-tagger --priority foreground --needs-mib ... drained the 27 B lease (<5 s), granted the tagger, ran a clean poll cycle (VRAM peak
  2,098 MiB), and released on wrapper exit — no stuck leases. So the broker path that this
  feature reuses (drain → grant → release) is already proven; we're extending what can be an
  engine (a container) and how it's drained (its lifecycle).
• needs_mib measured: peak 2,098 MiB during a clean cycle → configure 2560.
• Manual trigger exists: POST http://<container>:8000/api/scan (port not host-mapped; call
  via docker exec) — used to force a cycle for the measurements above.

Environment

Windows 11, Docker Desktop (WSL2 backend), RTX 5070 Ti 16,303 MiB (governed), berth 0.3.x.
Containers run inside the WSL2 VM; the GPU is passed through via the NVIDIA runtime, so a stopped
container's CUDA context and VRAM are released back to the driver. (Other backends should be
equivalent for the lifecycle semantics.)

Open questions

1. Restart policy. unless-stopped (explicit stop holds) is the policy under which Berth's
   docker stop is a stable "keep out" knob. Should the entry require/verify a non-always
   policy, or should Berth treat "container came back without my load" as a stowaway and
   re-drain it?
2. Lease semantics for a service. The tagger is a long-running scheduled service, not a
   run-to-completion job. Is the intended model "lease = container running (Berth keeps it up
   while granted, stops it on drain)" rather than "lease = one scan"?
3. Command vs. Docker preset. Do you want load_cmd/unload_cmd (generic, any engine with a
   start/stop) plus a kind="docker" sugar, or a dedicated container entry? Either reuses #96's
   match + stuck-drain logic.
4. Compose granularity. docker compose stop <svc> preserves the container (and state/volumes);
   down removes it. Presuming stop/start (not down) is the right option.
5. is docker CLI even the best lever here - maybe there is a batter one I don't know in docker ecosystem? Especially one that would interact with app INSIDE the container.

Note on scope / relation to #96

Not a duplicate of #96. #96 is about adopting an engine Berth did not start (Strata) via its
HTTP API. This is the Docker-container instantiation of the same "engine's own way out"
design, where the "way out" is the container runtime's lifecycle instead of an HTTP endpoint. If
load_url/unload_url ships first, this is just the command-based sibling of it.