PrometheusSimple: per-host connection slot leaks when TimeoutThread lands in the close handler

Start with hermod-tracing-prometheus/src/Hermod/Tracing/Prometheus/NetworkRun.hs, especially the forkFinally close handler and the connection-limiter paths at L143–L164; then read how withHandleKillThread and gracefulClose handle timeout exceptions. The issue reports no deterministic reproduction or named test, so first establish a reproducible case. Done means a late timeout cannot leak a per-host or global connection slot and subsequent scrapes are not refused.

cardano-node 11.2.0.164 (git rev b004fe0c), trace-dispatcher 2.13.1, network 3.2.9.0, time-manager 0.3.2; Leios prototype testnet, block producer.

Observed

After brief whole-process pauses (major GC under memory pressure), the node's stderr shows:

cardano-node: Thread killed by timeout manager

The BP logged this line five times in one day. After the fifth, every scrape from our Prometheus host is refused at once (read: connection reset by peer, ~2 ms). The same endpoint answers 200 to any other host. Inside the container's network namespace, ss -tn '( sport = :12798 )' shows only the listener and no open connections. The refusal persists (35+ minutes so far) while the node keeps running normally. A same-version node on another host with no pauses logged zero such lines and had zero failed scrapes.

Code path (trace-dispatcher-2.13.1, Cardano/Logging/Prometheus/NetworkRun.hs; unchanged on hermod-tracing master in hermod-tracing-prometheus/src/Hermod/Tracing/Prometheus/NetworkRun.hs)

• L90: forkFinally (runServer mgr conn) (const $ gclose conn >> releasePeer peer). releasePeer runs only if gclose returns normally.
• L96: runServer uses T.withHandleKillThread. In time-manager 0.3.2, the kill fires as forkIO $ E.throwTo tid' TimeoutThread (registerKillThread). cancel does not recall a throw already forked.
• The ignore TimeoutThread handler covers only the body of withHandleKillThread. A late throw is delivered in the forkFinally handler, which runs under mask (interruptible). gracefulClose blocks in atomically (network 3.2.9.0, recvEOFevent), which is an interruptible point. The exception then aborts gclose, skips releasePeer, and escapes the thread. That escape is the stderr line above.
• L159/L164: with runConnLimitPerHost = 5, the fifth leaked slot for a peer address means canServeThisPeer returns False. L91 then calls close conn on every later connection from that address, which the client sees as RST/EOF.

A process-wide pause makes the race likely: the inactivity timer and the client's close are both pending when the RTS resumes.

Impact

Metrics from one monitoring host are lost until the node restarts, while the node itself is healthy. With connCountPerHost = 0 and a non-zero global limit, the same leak would count against runConnLimitGlobal (L153), so every host would be refused instead.

Workaround (from code; not tested on a live node)

Set "TracePrometheusSimpleRun": {"connCountGlobal": 0, "connCountPerHost": 0}. mkConnLimiter 0 0 (L143) disables the limiter. cardano-node passes this to runPrometheusSimpleWith (cardano-node/src/Cardano/Node/Tracing/API.hs:175-177 at b004fe0c). Alternatively, restart the node.

Not verified

• The exact interleaving. It is inferred from the code and from the one-to-one match between the stderr lines and the leaked slots. There is no deterministic reproduction yet.
• The workaround on a running node.

Related: IntersectMBO/cardano-node#6389 (same symptom; closed after #6396 added withHandleKillThread).