nesquena/hermes-webui

bug(shutdown): 'Stop server' button no-ops on ctl.sh-managed daemons (SIGINT inherited SIG_IGN from non-interactive bash '&')

Ouverte

#7 078 ouverte le 16 août 2026

 (2 commentaires) (0 réaction) (0 personne assignée)Python (2 449 forks)github user discovery
bughelp wantedsprint-candidate

Métriques du dépôt

Stars
 (17 688 étoiles)
Métriques de merge PR
 (Merge moyen 14h 31m) (314 PRs mergées en 30 j)

Description

Symptom

The Stop server button in Settings (POST /api/shutdown) returns 200 {"status": "shutting_down"} but the WebUI process keeps running — for daemons launched via ./ctl.sh start (including the launchd-managed setup where the plist calls ctl.sh start).

Downstream consequences for the user:

  • Every subsequent ./ctl.sh start spawns a new instance that dies with [!!] FATAL: Another server is already responding on 127.0.0.1:8787. Stop the existing instance first. (_abort_if_already_serving, server.py:529) — the old process still owns the port.
  • The pid/state files (~/.hermes/webui.pid, ~/.hermes/webui.ctl.env) keep pointing at the just-died new PIDs, so ctl.sh stop also cannot reach the real process.
  • Result: the only way to actually stop the daemon is kill <pid> from a terminal.

Root cause

Three facts combine:

  1. ctl.sh start spawns the server from a non-interactive bash background job (( cd ...; exec nohup python bootstrap.py ... ) >> log 2>&1 &). POSIX/bash rule: "If job control is not in effect, asynchronous commands ignore SIGINT and SIGQUIT in addition to SIGHUP" — the child inherits SIGINT = SIG_IGN.

    Minimal proof:

    $ bash -c 'python3 -c "import signal; print(repr(signal.getsignal(signal.SIGINT)))" & wait'
    <Handlers.SIG_IGN: 1>
    
  2. CPython preserves an inherited SIG_IGN for SIGINT — it does not install its KeyboardInterrupt handler when SIGINT was ignored at interpreter startup.

  3. _handle_shutdown (api/routes.py) commits suicide with SIGINT:

    def _do_shutdown():
        import time
        time.sleep(0.3)
        os.kill(os.getpid(), signal.SIGINT)
    

    For a ctl.sh-managed daemon this signal is a silent no-op at the OS level.

    Meanwhile server.py only registers a graceful handler for SIGTERM (server.py:717 signal.signal(signal.SIGTERM, _request_shutdown)) — which is exactly why kill -TERM stops the daemon cleanly while the button's SIGINT does nothing.

Reproduction

  1. ./ctl.sh start (any non-interactive shell).
  2. curl -X POST http://127.0.0.1:8787/api/shutdown (authenticated session) → 200 {"status":"shutting_down"}.
  3. Process still alive:
    $ kill -0 <pid> && echo alive
    alive
    
  4. Signal-level equivalent:
    $ kill -INT <pid>   # no effect, process stays alive
    $ kill -TERM <pid>  # graceful shutdown (shutdown-audit + drain in the serve_forever() finally block)
    
    Note: _log_shutdown_audit() (server.py:492) has never fired in my environment's webui.log despite many stop attempts — consistent with no shutdown signal ever being processed.

Suggested fix

In server.py main(), register the existing _request_shutdown handler for SIGINT as well as SIGTERM:

try:
    signal.signal(signal.SIGTERM, _request_shutdown)
    signal.signal(signal.SIGINT, _request_shutdown)
except (ValueError, OSError):
    ...

This matches the handler's own docstring intent ("The handler is idempotent and guards against double-shutdown (e.g. repeated SIGTERM/SIGINT)") and makes the Settings stop button, external kill -INT, and foreground Ctrl-C all take the same graceful path (the finally block around serve_forever() already drains sessions either way).

Alternative (narrower): have _do_shutdown send SIGTERM instead of SIGINT. This fixes the button but leaves process-level SIGINT ignored for ctl.sh daemons.

Verification performed

  • signal.getsignal(SIGINT) in a bash-&-spawned child → SIG_IGN (mechanism).
  • Isolated instance (separate HERMES_HOME/port 8789, launched via ctl.sh): kill -INT → alive; kill -TERM → clean exit with [crash-visibility] process exit logged.
  • Production daemon (PID up 19 days, launched via launchd→ctl.sh): /api/shutdown returned 200 at 20:39:51, process still alive 12+ minutes later; all ctl.sh start attempts in between died with the port-in-use FATAL.
  • Confirmed against current upstream master (2026-08-15): server.py still registers only SIGTERM; routes.py _do_shutdown still uses SIGINT.

Related: #6396 assumes the Control Center stop works for ctl.sh-managed daemons — this bug is why the restart half of that story matters.

Guide contributeur