The warm pool learns what to keep warm, with no configuration

Start by locating the warm pool module, its configuration for warmPool.targets, and the worker's lease history and persistence code; the issue does not name file paths or tests. The proposed behavior spans demand learning, persistence across restarts, queue scheduling, capacity budgeting, and status reporting, with several open questions and completion conditions still unwritten. Done means the worker learns and restores warm targets from recent leases, boots queued requests within limits, reports actionable refusals, and preserves explicit target overrides.

Request: none

Problem

A worker boots nothing ahead of demand unless an operator writes warmPool.targets into the config file and restarts the daemon. Nobody does, so the first lease of every model, OS and mode on a worker pays a cold boot (about 60 s for an iOS simulator, about 160 s slim, about 70 s for an Android emulator). After a lease ends, the pool reboots the device only if the lease ended less than idle.shutdownAfterMs ago and a slot is free, one boot at a time; the next request often arrives before that lands, and a restart starts cold. A queued request gets a boot only when it is the queue head and a shut-down device already matches; a burst of five agents asking for one kind pays five boots one after the other. The pool learns nothing from what was leased.

Two reliability holes found on 2026-10-20 sit beside this: a keep boot the RAM budget refuses is declined silently on every pass, and a target is never reported short while any waiter sits on its platform. Two more findings outside the pool were filed on their own (see Non-goals).

Who it is for

The maintainer of a worker that serves parallel coding agents, who installs Simlock and never edits its config. The agents that lease from it, which today wait a cold boot on every new kind and on every burst.

Outcome

With no configuration, a worker keeps what it has leased: one device per kind it leased recently, as many as that kind was held at once in the last 24 hours counting the ones leased right now, newest kinds first within the capacity budget, booted back when released, and booted again after a restart. Every queued request gets a device started for it ahead of its turn, several at once. A kind the worker cannot keep says why in status. Explicit warmPool.targets stay as an override and are never displaced. simlock nuke forgets what was learned.

How it works for the user

sequenceDiagram
  participant A as agent
  participant W as worker
  participant P as warm pool
  A->>W: lease iPhone 16 / 18.4
  W-->>A: granted, source provisioned (60 s)
  W->>P: lease.granted: kind learned
  A->>W: release
  W->>P: lease ended; reclaim erases, shuts down
  P->>P: pass: kind wants 1, has 0: boot it back
  A->>W: lease iPhone 16 / 18.4 (later, or after a restart)
  W-->>A: granted, source warm (under 1 s)
  A->>W: 5 leases of one kind at once
  P->>P: pass: one boot or creation per queued request, up to the boot cap
  W-->>A: each granted as its device is ready

# a fresh worker, no config.json
$ simlock lease --platform ios --device "iPhone 16" --os 18.4 --agent-id a1
{"lease":{"id":"..."},"device":{...},"timing":{"estimatedBootMs":60000,...},"source":"provisioned"}
$ simlock release <id>
$ simlock status
warm pool: enabled, reserve ios 0 android 0
  iPhone 16 / 18.4 / full   learned   wanted 1  ready 0  booting 1
$ simlock daemon stop; xcrun simctl --set ~/.simlock/devices/ios shutdown all; simlock status   # a minute later
warm pool: enabled, reserve ios 0 android 0
  iPhone 16 / 18.4 / full   learned   wanted 1  ready 1  booting 0
$ simlock lease --platform ios --device "iPhone 16" --os 18.4 --agent-id a2
{..., "source":"warm"}
$ simlock nuke --yes
$ simlock status
warm pool: enabled, reserve ios 0 android 0

Today the same session shows no warm pool line after the release, source: booted or provisioned on the second lease, and nothing ready after a restart.

Examples

What the pool keeps on a worker with two iOS running slots and no warmPool.targets:

Leases on this worker	Today	After
one iPhone 16 / 18.4, ended 5 min ago	1 ready until 10 min idle	1 ready, kept
one iPhone 16 / 18.4, ended 2 h ago	0	1 ready, kept
three iPhone 16 / 18.4 held at once this morning, all released	0	2 ready; the third short running-limit
two iPhone 16 / 18.4 held right now, peak 2	nothing extra	nothing extra: the kind has two
iPhone 16 / 18.4 at 09:00, then iPhone 15 / 17.5 at 10:00	0	one of each
three kinds, one lease each, the newest an hour ago	0	the two newest ready; the oldest short, its idle device displaced
a kind whose runtime was removed	—	forgotten on the next pass
a kind that failed to boot three times in a row	—	forgotten, one warning logged
after simlock nuke	nothing booted back	nothing booted back, record empty
warmPool.enabled: false	nothing kept	nothing kept, nothing learned
warmPool.targets names iPhone 17 / 26.0 count 1, and it was held 2 at once today	1 ready	2 ready, one line, origin configured

Five requests for one kind queued at once on a 12-CPU worker with one ready device:

	Today	After
boots started at once	1	3 (maxConcurrentBoots defaults to max(1, floor(cpu / 4)))
time until the fifth is granted, 60 s boots	about 4 min	about 2 min

What could go wrong

• Devices boot back that nobody asked for yet, holding RAM on a quiet machine: by design, within the same running limits as today; nothing extra is made while a burst is held.
• Two kinds that alternate under a budget of one trade places, one boot each time: the LRU rule the maintainer chose.
• A kind keeps failing to boot: it is tried three times in a row, then forgotten with a warning, not forever.
• A request in a burst is granted a device the pool created rather than one it started itself: the same device either way; the rare race where both create leaves one idle device the idle rules remove.

Words used

• kind: a virtual device's exact spec: platform, model, OS version, mode (slim or full) and image tag, e.g. iPhone 16 / 18.4 / full.
• learned kind: a kind this worker granted a lease of, kept ready like a target; shown with origin learned.
• configured target: a warmPool.targets entry; origin configured. A kind that is both is one entry, origin configured, with the larger count.
• count: how many devices of a kind the pool wants, ready or leased: the most leases of it held at the same moment in the last 24 hours, at least one. A configured target's count is ready devices only, as today.
• budget: the running slots, device limit, RAM budget and boot cap the pool may fill, as today (ADR 0017 §4).
• boot cap: warmPool.maxConcurrentBoots, the most pool boots and creations running at once.
• displaced: an older learned kind's idle device shut down so a newer learned kind can take its slot.
• on its way: a device being created or booted for nobody in particular, which a request that fits waits for instead of starting another.
• forgotten: a learned kind removed from the record; its next lease learns it again.

Non-goals

• Fleet-level targets or learning on the gateway. Each worker learns its own demand; the gateway keeps routing by warm hits.
• Any change to lease.identity: a learned kind under fresh pre-creates its replacement as a configured target does.
• Changing configured targets without a restart.
• A new config key: warmPool.enabled stays the one switch.
• Android's quoted boot time on a snapshot restore, and shut-down devices counting against the RAM budget: filed as #778 and #779.
• A creation for the queue head: leasing's own, as today.

Completion conditions

• With no config, after a lease of a kind ends, the next lease of that kind on that worker is granted source: warm while the budget allows.
• Three leases of one kind held at once make nothing extra while held, and once released leave three devices of it ready while the budget allows.
• After simlock daemon stop and a start, the kinds last leased are ready again before any lease.
• Five requests for one kind queued at once start maxConcurrentBoots boots together, and each request behind the head is granted the device started for it.
• warmPool.maxConcurrentBoots defaults to max(1, floor(cpu / 4)), as simlock config shows.
• simlock status and the console list each learned kind with origin learned, wanted, ready, booting, short and its image tag when it has one; a configured target shows origin configured.
• simlock nuke clears the learned record: nothing is booted back after it until the next lease.
• A learned kind whose runtime was removed disappears from status without a restart.
• A boot the RAM budget refuses is not proposed again every pass.

Open questions

Decisions

• ADR 0038 — The warm pool learns its kinds from leases, and boots for every queued request (PR #772): what a kind is, the 24-hour peak, newest-first filling and displacement, forgetting, the record as the pool's own state.db tables, queue work for every waiter, the boot cap's default, and what status shows.
• ADR 0036 — Daemon state is a SQLite database, and each owner keeps its own tables (PR #755, #754): why the record is a table and not a JSON key.

Shared values

Value or state	Rule	Written by	Read by	Source
kind identity	platform, model, osVersion, mode (slim or absent), imageTag, what sameSpec compares	#776 (the record)	#776, #777	src/core/domain.ts:93-102
WaitingDemand.spec	the waiter's resolved VirtualSpec, present on every entry waitingDemand() lists	#774	#774 (the pool creates it); #776 keeps boots() as #774 leaves it	src/core/domain.ts:135-141; src/leasing/lease-acquisition-coordinator.ts:635-652
proposal reasons	boots and creations: waiting-request (for a waiter, #774), target (a configured target or a learned kind, told apart by the report's origin); shutdowns add displaced (#776); reasons are logged, never on an event payload (device.shutdown carries deviceId and initiator only)	#774, #776	the daemon log; the device.shutdown catalog prose (#776)	src/core/warm-pool/policy.ts:40-42, :149-158, :263, :303, :428; src/bus/index.ts:200
the boot cap	warmPool.maxConcurrentBoots, default max(1, floor(cpuCount / 4)) from SystemStats.cpuCount(), a file value wins; counts the pool's flights plus each waiting request whose own boot or creation is in flight	#774	#776 (learned fills under it), #777 docs	ADR 0038 §7; src/ports/system-stats.ts:9-10
the retry schedule	one per spec, shared: a failed boot or creation for a waiter, a configured target or a learned kind of that spec holds the spec back 1, 2, 4 ... minutes for all three and counts toward a learned kind's three failures in a row	#774 (waiter failures), #776 (forgetting)	the policy's mayAttempt	src/core/warm-pool/retry.ts:24-43
a learned kind's count and what it has	count = peak concurrent leases of the kind over the last 24 h from now, a lease held across the window's start counts, at least 1; has = ready idle + leased + reclaiming devices of the kind; a merged entry needs the larger of (configured count − ready) and (learned count − has)	#776	#777 (reported as count; the event uses the configured count)	ADR 0038 §2, §3
fill order	configured targets in config order, then learned kinds by last lease, newest first; a kind in both is one entry, count the larger, in the configured position	#776	#777 (the one line)	ADR 0038 §3
forgetting	catalog lacks model or OS, or no driver; 3 failed boots in a row; 33rd kind of a platform, oldest dropped; simlock nuke	#776	#777 docs	ADR 0038 §4
owner warm-pool	SqliteSchemaOwner version 1; tables warm_kinds, warm_kind_leases as #776 lists them; in the worker's owner list after registry and the device-data owners	#776	the state module (#761)	ADR 0036 §2; #754 Shared values "Owner lists"
origin and imageTag	configured or learned on every status.get warm pool entry, and the kind's image tag when it has one; on warm-pool.target-missed origin is always configured; the entry list is bounded to 320 (WARM_POOL_ENTRIES_MAX: 256 configured + 32 learned per platform)	#776 sets origin on reports; #777 puts both on the wire	CLI, console, doctor, gateway view (#777)	ADR 0038 §8; src/contract/schemas.ts:522, :539-557
protocol bump	#777 bumps by one for origin; #774 and #776 change no wire shape	#777	—	src/contract/protocol.ts:13

Tasks

flowchart LR
  T761["#761 state.db (feature #754)"] --> B["#776 learned kinds"]
  A["#774 queue boots and the boot cap"] --> B
  B --> C["#777 status, doctor, console, docs"]
  ADR["ADR 0038, PR #772"] -.-> A

• #774 The warm pool boots or creates one device per queued request, and the boot cap defaults from the CPU — a burst of requests starts several boots at once, and a request behind the head is granted the device started for it. Risk: two devices created for one request, or more boots at once than the machine takes.
• #776 The warm pool learns the kinds it leases and keeps them ready, across restarts — a worker with no config keeps what it leased ready, replenishes it, reboots it after a restart, forgets it on nuke. Risk: devices booting that nobody asked for, or two kinds trading places in a loop.
• #777 Status, doctor, the console and the docs show learned kinds with their origin — an operator sees what holds the worker's RAM and why a kind is short. Risk: a status block that hides learned kinds.

Completion conditions and the task whose Done when delivers them:

Condition	Task
next lease of a kind is warm	#776
three held at once: nothing extra, then three ready	#776
ready again after a restart	#776 (fast e2e and the slow lane)
five queued, maxConcurrentBoots boots at once	#774
maxConcurrentBoots default from the CPU	#774
status and console show origin	#777
nuke clears the record	#776
removed runtime forgets the kind	#776 (the pass), #777 (gone from status)
a RAM-refused boot is not proposed every pass	#774

Open work in the same area:

• #754 state.db, tasks #761 (the state module, the registry's tables), #757, #758, #759, #760: #776 adds an owner to the module #761 builds, in the same worker owner list #758 adds tokensOwner to. Depends on: #776 on #761 and #758.
• #766 (task:ready, feature #763's console) renames ui/src/screens/workers/ and rewrites the worker page's Warm pool tab: the lines #777 edits. Depends on: #777 on #766.
• #764 (task:ready) bumps the protocol: contract shape changes overlap (toolchain.md). Depends on: #777 on #764.
• #387 (request) one waiting line per platform: #774's per-request boots lessen the stall it describes; no dependency.
• #484 (request) low-RAM Android devices: a learned kind would carry whatever spec such a device has; no dependency.
• #327 (feature:spec) named baselines: changes what a reclaim restores, not what is kept warm; no dependency.
• PR #762 (ADR 0037, console on Base UI): the ADR behind #766; covered by that dependency.

Written by an agent.