Simulation submission stops on prod AND dev when the shared SLURM controller gets slow

Start in CommandServiceSshNative.java (host rotation and penalty box around lines 110-155) to see where the SSH command timeout is applied and where exhausted all hostnames is logged. Then trace PropertyLoader.cmdSrvcSshCmdTimeoutMS / vcell.ssh.cmd.cmdtimeout and check kustomize/config/prod/{submit,sched}.env and the dev overlays to confirm the property is unset. Done looks like: the read-only discovery calls (sinfo -N -h -p vcell) tolerate a slow controller, the node list is no longer re-enumerated per submission (via VCELL_HTC_NODELIST or caching), and there is a signal that fires on exhausted all hostnames — verify by reading the code paths rather than reproducing a cluster slowdown, which is out of a newcomer's reach.

Updated after checking the cluster itself. The first version blamed the single SSH submit host; that is real but is not the whole story, and it pointed at a fix that may not help. The current account is below; the original analysis and the cluster investigation are in the comments.

Symptom

Two Better Stack incidents, both ~6 minutes, both recovering together — which is what made it worth chasing:

VCell Release Health - Sim - Warning    GET https://vcell.cam.uchc.edu/api/v0/health?check=sim
  {"nagiosStatusName":"Warning","nagiosStatusCode":1,"elapsedTime_MS":342210}

VCell Alpha Health - Sim                GET https://vcell-dev.cam.uchc.edu/api/v0/health?check=sim
  {"nagiosStatusName":"Critical","nagiosStatusCode":2,
   "message":"simulation failed (1181): simulation took longer than 480000 …"}

Window: 2026-09-02 00:08–00:21 UTC (Better Stack renders Eastern). The prod figure pins it exactly — elapsedTime_MS=342210 is sim 323813748 at 343.2s. An earlier cluster hit 2026-09-01 23:04–23:40 UTC.

What happened

On the cluster. An administrative operation made a set of nodes unresponsive:

$ sinfo -R -o '%60E | %12u | %20H | %N'
REASON                         | USER   | TIMESTAMP            | NODELIST
move-server : Not responding   | slurm  | 2026-09-01T19:39:39  | mantis-[018-020,022]
Kill task failed               | root   | 2026-09-01T19:44:40  | mantis-027

At 2026-09-01 19:39:39 EDT (23:39:39 UTC), 44 jobs NODE_FAILed within one second across 20 distinct nodes — mantis-003 011 013 015 016 019 023 024 025 028 031 035 037 038 045 047 048 049 050 064. Twenty nodes failing in the same second is one event, not twenty. Individual NODE_FAILs scatter from 19:27 to 19:44, so the disruption ran roughly 19:27–19:44 EDT: the first incident cluster, with the second in its aftermath (a further NODE_FAIL at 20:09:31).

On the VCell side. VCell does not talk to SLURM directly. CommandServiceSshNative SSHes into a submit host and runs sinfo / sbatch / squeue / sacct. That SSH timed out:

Command "ssh" encountered a problem: Process timed out
Unexpected error: Process timed out
failed execution, using mantis-svcsub.cam.uchc.edu,
  command=[sinfo, -N, -h, -p, vcell, --Format='nodelist', |, xargs, |, tr, ' ', ',']
failed, exhausted all hostnames, command=[sinfo …]

The important part: VCell's own compute was never affected

The vcell partition is mantis-[040-044], and not one of them appears in the failure list. They were healthy throughout:

$ sinfo -p vcell
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
vcell        up   infinite      1    mix mantis-041
vcell        up   infinite      4   idle mantis-[040,042-044]

VCell did not lose capacity. It lost the ability to talk to the scheduler — sinfo -N -h -p vcell exceeded its 10-second timeout while the controller dealt with a 20-node failure on a partition VCell does not use.

So the exposure is broader than a shared submit host: VCell is coupled to the health of the shared SLURM controller, not merely to its own nodes. A neighbouring event elsewhere on the cluster is enough to stop simulation submission on both sites.

Both sites fail together because they share one door

# kustomize/config/prod/{submit,sched}.env  AND  kustomize/config/dev/{submit,sched}.env
VCELL_HTC_HOSTS=mantis-svcsub.cam.uchc.edu
VCELL_HTC_USER=vcell
VCELL_SLURM_PARTITION=vcell

"exhausted all hostnames" is literal. CommandServiceSshNative rotates through VCELL_HTC_HOSTS with a penalty box for failing hosts (CommandServiceSshNative.java:110-155), but with one host there is nothing to rotate to.

It was submission, not queueing

Worth being precise, because "sim health check slow" reads like a busy cluster and it was not:

• Dispatch was instant — none of these jobs waited in a SLURM queue.
• The timed-out jobs report execStatus=[submit-6cf57bfdc9-tmcph, …, null, false] — the host is still the submit pod, end date null, no data. Accepted, never reached a compute node.
• The one that did land shows execStatus=[mantis-042, 20:09:37, 20:14:45, true] — 5m08s for a job that normally finishes in seconds.
• Baseline: across 1,679 health-check simulations in 36h, the slowest outside these clusters is 14.4 seconds.

Most of these never alerted

exhausted all hostnames, last ~36h, prod + dev — 29 events:

hour (UTC)	events
2026-09-01 15:00	25
2026-09-01 18:00	2
2026-09-01 23:00	1
2026-09-02 00:00	1

The 25-event burst raised no incident at all, and has the same shape: 11:41:03–11:42:24 EDT, an 81-second window. Nothing failed then — the cluster was simply busy, with 426 job records touching the surrounding 15 minutes and bursts of starts from three users (59, 14, 11 jobs). The controller got slow, sinfo exceeded 10s, and VCell had nowhere to fall back to. (The 48-job mass cancellation at 11:57:21 EDT is 15 minutes later and is not the cause.)

Monitoring only noticed the two occasions when a health-check simulation happened to overlap a failure. User simulations submitted during any of these windows hit the same wall silently.

Chain, end to end

1. The shared controller becomes slow — a 20-node move-server failure at 19:39, or ordinary scheduling churn at 11:41.
2. VCell's sinfo -N -h -p vcell over SSH exceeds vcell.ssh.cmd.cmdtimeout (10s default, unset in every overlay).
3. VCELL_HTC_HOSTS lists one host, so CommandServiceSshNative logs exhausted all hostnames and gives up.
4. prod and dev share that host, so both stop submitting at the same instant and recover together.
5. Health-check simulations time out; Better Stack alerts on both sites.

Suggested directions, in the order I would try them

1. Raise the SSH timeout for the read-only discovery calls. Both incidents were controller slowdowns of tens of seconds, not outages. PropertyLoader.cmdSrvcSshCmdTimeoutMS (vcell.ssh.cmd.cmdtimeout) defaults to 10000ms and is not set anywhere. This looks like the single change most likely to have absorbed both events.
2. Stop asking the controller to enumerate nodes on every submission. VCELL_HTC_NODELIST is empty, so sinfo -N -h -p vcell runs each time. That list changes rarely — setting it explicitly, or caching it, removes a controller round-trip from the hot path.
3. Alert on exhausted all hostnames. A precise, low-noise signal that fires before users notice, and it already caught an 81-second outage nothing else did.
4. More submit hosts — but check first. The island overlay uses hpc-ext-1..4, so the failover machinery is in use elsewhere. It helps only if those are genuinely separate paths to the controller; if slurmctld is the bottleneck, more SSH front doors will not help.

Not determined

Why move-server made those nodes unresponsive, and whether it was planned — that belongs to whoever runs the cluster.

sshd logs on mantis-svcsub were not read (the available account has standard privileges only), but sshd throttling looks unlikely: the host is healthy — load 0.15, up 41 days, SSH round-trip 0.355s — and both bursts line up with controller-side events rather than connection-rate limits.

🤖 Generated with Claude Code

https://claude.ai/code/session_01D71LBYmQNf5J94wPqr81Jx