A single unresponsive/untrusted SSH host crashes unrelated plugins via unhandled "no valid actor" rejections, eventually disabling core plugins (including the UI layout) through the 5-crashes-in-10-minutes breaker

Open
#1,944 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
18/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
docker, typescript
Domain
backend, devops

Research direction

The failure starts in the host-metrics stats poller, which logs stats_connect_failed and stats_host_key_verification_failed for hosts 5 and 7-11. The unhandled rejection surfaces in the sandbox RPC layer, plugins/sandbox/rpc.js (fromWireError and Endpoint.dispatch), and the breaker lives in plugin_runtime. Start by reproducing with one unreachable host and checking whether a rejected stats promise reaches the sandbox dispatcher. Done means host-metrics errors stay inside that plugin and no unrelated plugin is restarted or disabled.

Written by the indexing model from the issue text.

Description

bug platform-docker
What happened?

Environment

  • Termix version: 26.11.3 (ghcr.io/termix-ssh/termix:latest)
  • Deployment: Docker

Note on scope

This is a different bug from the plugin-sandbox mass-restart issue previously reported and fixed. That fix is confirmed working in 26.11.3 — the supervisor now correctly logs "N of N plugins stopped answering at once; waiting instead of restarting them, since core or its channel to them was busy" instead of SIGKILLing the whole fleet simultaneously. This report is about a new failure mode that surfaced afterward.

Summary

A single host that's slow to respond over SSH, or has an untrusted/unverified host key, causes the host-metrics background stats poller to repeatedly fail against it. These failures appear to trigger unhandled promise rejections in unrelated plugins (observed in ssh-terminal: "This plugin call has no valid actor"), which crash those plugin processes. Because this happens repeatedly within a short window, several plugins individually hit Termix's "stopped 5 times in 10 minutes" circuit breaker and get permanently disabled (not just restarted) — including, eventually, the active UI layout plugin itself, taking down the entire web UI with no automatic recovery.

Evidence

host-metrics repeatedly failing against problem hosts

[3:36:29 PM] [WARN] [🖥️] Host has no saved key and nobody to ask [op:host_key_unverified,user:j2gj0GtMvFuXqXn6rLFwZ,host:7] [3:36:29 PM] [ERROR] [🧩] Stats collector host key verification failed {"operation":"stats_host_key_verification_failed","hostId":7}: This host's key is not trusted yet. Open a terminal to the host once to check and trust it. [op:plugin:host-metrics] ... [3:37:12 PM] [ERROR] [🧩] Stats collector connection failed {"operation":"stats_connect_failed","hostId":5}: SSH connection timeout [op:plugin:host-metrics] ... [3:49:47 PM] [ERROR] [🧩] Stats collector connection failed {"operation":"stats_connect_failed","hostId":5}: SSH connection timeout [op:plugin:host-metrics]

Host 5 (slow/unreachable over SSH) and hosts 7–11 (untrusted keys) produced these errors repeatedly over the ~15 minute window captured.

Unrelated plugin throwing an unhandled rejection shortly after

[3:43:20 PM] [WARN] [🧩] Plugin ssh-terminal: Unhandled rejection: Error: This plugin call has no valid actor [op:plugin_sandbox] [3:43:20 PM] [WARN] [🧩] Plugin ssh-terminal: at fromWireError (file:///app/dist/backend/backend/plugins/sandbox/rpc.js:606:19) [op:plugin_sandbox] [3:43:20 PM] [WARN] [🧩] Plugin ssh-terminal: at Endpoint.dispatch (file:///app/dist/backend/backend/plugins/sandbox/rpc.js:501:35) [op:plugin_sandbox]

Individual plugins crash-looping independently (not a mass-restart event)

[3:48:09 PM] [ERROR] [🧩] Plugin vault stopped [op:plugin_runtime] Error: The plugin process stopped 5 times in 10 minutes: exited (SIGKILL) [3:48:09 PM] [ERROR] [🧩] Plugin alerts stopped [op:plugin_runtime] Error: The plugin process stopped 5 times in 10 minutes: exited (SIGKILL) [3:51:06 PM] [ERROR] [🧩] Plugin totp stopped [op:plugin_runtime] Error: The plugin process stopped 5 times in 10 minutes: exited (SIGKILL) [3:51:06 PM] [ERROR] [🧩] Plugin webauthn stopped [op:plugin_runtime] Error: The plugin process stopped 5 times in 10 minutes: exited (SIGKILL) [3:51:06 PM] [ERROR] [🧩] Plugin ssh-terminal stopped [op:plugin_runtime] Error: The plugin process stopped 5 times in 10 minutes: exited (SIGKILL)

Shortly after, the active layout plugin itself tripped the same breaker, taking down the whole UI (screenshot available on request — shows Termix's own "This layout could not be shown" recovery screen, with the same "stopped 5 times in 10 minutes" message).

Expected behavior

A background polling failure against one slow or not-yet-trusted host should be caught and handled entirely within host-metrics (e.g., logged, skipped, backed off) and must never be able to crash unrelated plugin processes — especially not ssh-terminal, authentication plugins (totp, webauthn), or the active UI layout plugin. None of these have any direct relationship to a single host's SSH reachability.

Actual behavior

A single flaky/untrusted host caused a cascade of unrelated individual plugin crashes, several of which tripped the 10-minute circuit breaker and were permanently disabled (no automatic recovery) — including, eventually, the UI layout plugin, which took the entire web interface down with no path back except Termix's manual "Start Classic Layout again" recovery button, plus manually re-enabling the other disabled plugins (ssh-terminal, totp, webauthn, vault, alerts) from Settings → Plugins.

Workaround

Removing the problem host (ID 5) from the host list stopped the recurrence. This only removes the trigger, not the underlying issue: any future flaky/unreachable/untrusted host could reproduce this and take down the UI again.

Impact

A routine, expected condition (one slow or newly-added host) was able to cause a full outage of the web UI requiring multiple manual recovery steps, with no self-healing.

Steps to reproduce
  1. Add a host that is slow/unreachable over SSH (or whose SSH host key has never been trusted via an interactive terminal session).
  2. Leave Termix running with host-metrics enabled so it periodically polls that host in the background.
  3. Watch docker logs -f <container> — within a few minutes, unrelated plugins (not just host-metrics) start logging stopped answering, restarting it / exited (SIGKILL) individually, and eventually:
    [ERROR] [🧩] Plugin stopped [op:plugin_runtime] Error: The plugin process stopped 5 times in 10 minutes: exited (SIGKILL)
  4. If the active layout plugin (classic, formerly classic-layout) is among the plugins that trip this breaker, the entire UI goes down with:

This layout could not be shown — The layout failed to start.
classic | Classic Layout 1.2.0
The plugin process stopped 5 times in 10 minutes: exited (SIGKILL)

What did you expect?

No response

Termix version

26.11.3

Platform

Web browser

Server install

Docker

Environment

No response

Logs

Anything else?

No response

Before you submit
  • I searched open and closed issues and this is not a duplicate
  • I am on the latest version, or I said which version I am on
Dominant language
TypeScript
Stars
16.1k
Forks
776
Avg merge
1d 5h
Merged PRs (30d)
57

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Termix-SSH/Termix

All issues in Termix-SSH/Termix

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.