Pull-wake: stream reconnect path does not fire when agents-server becomes unreachable (Ctrl-C and kill -9 both verified)
Nessuno ha ancora preso questa issue.
Valutazione
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Idoneità per principianti
- 55/100
- Tipo di issue
- Bug
- Chiarezza
- Abbastanza chiara
- Stato di attività
- Tranquilla
- Stack tecnologico
- node.js, typescript
- Ambito
- backend, networking, observability
Direzione di ricerca
Inizia in packages/agents-runtime/src/pull-wake-runner.ts, soprattutto in consumeWakeStream e nella gestione dell’heartbeat collegata a agents/src/server.ts:324. Riproduci entrambi i casi di arresto del server e controlla l’endpoint di health mentre tracci come i fallimenti dello stream e dell’heartbeat influiscono sulla macchina a stati. Il lavoro è completato quando il runner rileva l’interruzione, entra in running.reconnecting, esegue il backoff, incrementa reconnect_count e reimposta stream_connected_since dopo il ripristino.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Summary
When the agents-server becomes unreachable (verified for both Ctrl-C graceful shutdown and kill -9 hard kill), the pull-wake runner's stream reconnect path never fires. The runner reports stream_connected: true, reconnect_count: 0, client_status: streaming and stream_connected_since unchanged for the entire duration of the outage. The state machine never transitions to running.reconnecting, exponential backoff never runs, and reconnect_count is never incremented.
Heartbeats (which are separate, short-lived HTTP POSTs) correctly observe the outage and fail with ECONNREFUSED every 30s — so the runner is aware on one path that the server is unreachable. But the stream reader sees no error and concludes the stream is fine.
Surfaced while testing #4339 (pull-wake runner lifecycle hardening). The PR adds the state machine, exponential backoff (1s → 30s cap), reconnect_count tracking, and running.reconnecting state — but none of these engage under any tested failure mode.
Updated: original framing was specific to graceful shutdown. A follow-up test with
kill -9(which generates an RST instead of FIN) produced the same broken behavior — the bug is not about TCP shutdown signal type, it is about the stream reader not surfacing any server failure as a stream error, regardless of how the server dies.
Reproductions
Case A — graceful shutdown (Ctrl-C)
Setup:
- Local agents-server running via
pnpm --filter @electric-ax/agents-server dev(tsx watch onsrc/entrypoint.ts) - Desktop app connected, pull-wake runner registered and healthy
Steps:
- Confirm baseline via the new health endpoint:
Expected:curl -sS -H "electric-principal: system:dev-local" \ "http://127.0.0.1:4437/_electric/runners/<RUNNER_ID>/health" \ | jq '{rc:.client.reconnect_count, status:.client.status, stream:.client.stream_connected, since:.client.stream_connected_since}'rc:0, status:"streaming", stream:true, since:"<some prior time>" - Stop the server gracefully —
Ctrl-Cthepnpm devprocess (orkill -SIGINT <pid>). - Watch the desktop log at
~/Library/Application Support/Electric Agents/logs/builtin-agents-*.jsonl. Heartbeat failures appear every 30s:[builtin-agents] pull-wake runner failed TypeError: fetch failed: connect ECONNREFUSED 127.0.0.1:4437 - Leave the server down for several minutes (verified for 17 minutes in original repro).
- Restart the server.
- Re-query the health endpoint after the next heartbeat succeeds.
Case B — hard kill (kill -9)
Same setup. Same procedure but kill -9 <pid> instead of Ctrl-C. RST should be sent immediately. Identical observed behavior:
kill at 10:32:45.3Z
first log entry at 10:33:01.7Z (the regularly-scheduled 30s heartbeat — NOT a reconnect attempt)
+30s next heartbeat failure
+30s next heartbeat failure
+30s next heartbeat failure
All entries arrive on the 30s heartbeat schedule. No additional log entries at exponential-backoff intervals (1s, 2s, 4s, 8s, 16s) — which is what we'd expect to see if the stream-reconnect catch branch was firing.
After recovery, reconnect_count is still 0 and stream_connected_since is unchanged from before the kill. The runner thinks nothing happened.
Observed payload (post-recovery, after both Case A and Case B)
{
"client": {
"status": "streaming",
"reconnect_count": 0,
"stream_connected": true,
"stream_connected_since": "2026-05-18T08:11:00.913Z", // unchanged — was the value pre-outage
"last_heartbeat_at": "2026-05-18T10:35:01.684Z",
"last_heartbeat_ok": true
},
"health": { "status": "healthy", "issues": [] }
}
stream_connected_since survived BOTH outages (Ctrl-C followed by hard kill) without changing. The runner has held the same conceptual stream object since before either outage.
Root cause analysis
The stream consumption loop in packages/agents-runtime/src/pull-wake-runner.ts opens a long-lived HTTP fetch and reads chunks until it ends:
await consumeWakeStream(signal, runGeneration)
if (!signal.aborted) {
state = `running.reconnecting`
// ... backoff/sleep ...
}
When the agents-server dies (regardless of shutdown mechanism):
- The OS should send a FIN (Ctrl-C) or RST (kill -9) on the established TCP connection
- Node's fetch streaming reader does not surface this as a read error when the stream is idle (no in-flight wakes) — the reader just sits on the dead socket
- The connection effectively enters a half-open state from the application's perspective
- Wakes published by the server during downtime (if any) cannot be received
Plausible cause: the stream uses long-polling / Server-Sent-Events-style chunked transfer over fetch. When the underlying socket is closed by the peer but no data is in flight, the AsyncIterator returned by response.body may not yield or error promptly. It depends on:
- Whether Node observes the FIN/RST before the next read attempt is made
- Whether the AsyncIterator emits an end-of-stream that the code then handles in the success branch (no
reportError, noreconnectCount++)
So consumeWakeStream either doesn't return at all (most likely) or returns "cleanly" without an error. Either way:
- The state machine never transitions to
running.reconnecting reconnect_countis never incrementedreportErroris never called from the stream pathstream_connectedandstream_connected_sinceare never updated
Heartbeats run on a separate timer and use individual fetch calls — those fail with ECONNREFUSED immediately because each one initiates a new TCP connection. That's why we see clean 30s-cadence heartbeat failures in the log but no stream-level activity. Heartbeat failures call reportError (via onError in agents/src/server.ts:324) and set last_heartbeat_ok: false, but they do not invalidate the stream connection state on the client side.
Expected behavior
The runner should recognize that its stream is dead — at minimum, this should be visible in the diagnostics. Ideal end state:
- State machine transitions
running.streaming → running.reconnectingwhen the stream stops delivering OR when N consecutive heartbeats fail (whichever comes first). reconnect_countincrements each time a reconnect is attempted.stream_connected_sinceis reset on successful reconnect (the PR's "reset on success" claim depends on this firing).- Exponential backoff path is exercised (and observable via
reconnect_countgrowing faster than the heartbeat cadence).
Suggested investigation / fixes
Pick one or combine; (a) is the cheapest:
(a) Tie stream liveness to heartbeat failures (cheap)
After K consecutive heartbeat failures (e.g. K=2), abort the current stream connection and re-enter consumeWakeStream. This bridges the two signal paths and uses the already-working heartbeat detection to drive the (already-implemented) reconnect machinery. This is now the strongest recommendation given that the bug is universal, not specific to graceful shutdown — a stream-side detector won't help if the stream never sees the failure.
(b) Application-level keepalive on the stream
Have the server periodically push a no-op ping event over the wake stream (e.g. every 30s). On the client, treat absence-of-pings (>2× ping interval) as a stream failure and reconnect.
(c) TCP-level keepalive
Set SO_KEEPALIVE on the underlying socket with aggressive timing (idle=10s, interval=5s, count=3). Standard fetch in Node doesn't expose this directly; would require a custom http2 agent or undici Dispatcher.
(d) Write-side heartbeat on the stream connection itself
If the stream is bidirectional or supports a client→server keepalive, send periodic empty writes. Failure to write surfaces a socket error → triggers reconnect.
Given the Case B result, (a) is now clearly the highest-leverage fix: heartbeat failures are the only observable signal the runner has when the server is gone, so heartbeat-driven stream reset is the only path that doesn't depend on the stream layer somehow noticing what it currently doesn't notice.
Severity
Reliability — the PR advertises a feature (graceful reconnect with backoff) that does not engage under any tested failure mode (server Ctrl-C or kill -9 — verified). The runner eventually recovers because heartbeats start working again — but during the outage there is no visibility, no proactive recovery attempt, and wakes published during the outage cannot be received until the stream actually gets reset by some other mechanism.
The PR's running.reconnecting state, reconnect_count counter, and exponential backoff implementation appear to be unreachable code in practice. This is not a corner case — it's the default behavior on every observed server-down scenario.
Related
- #4339 — PR that introduced the state machine and reconnect logic
- #4340 — claim not released after dispatch
- #4341 —
lease_expires_at: nullon materialized claims - #4342 — Local Runtime UI shows stale state during outage (compounded by this bug, since the UI's stream indicator reads
client.stream_connected: truewhich is exactly the wrong value)
- Lingua principale
- TypeScript
- Stelle
- 10.4k
- Fork
- 399
- Metriche di merge delle PR
- Nessuna PR unita negli ultimi 30g
Preparare l'ambiente
- Nessun Dockerfile né file Docker Compose
- Nessun modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di electric-sql/electric
-
camelToSnake and snakeCamelMapper() drop the underscore when a digit precedes a camelCase boundaryAperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 85/100
electric-sql/electric#4855 ·
-
ShapeStream does not notify subscribers when it gives up on a non-Error valueForse già presa @Tyagiquamar l’ha presa 19 giorni fa. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
electric-sql/electric#4830 ·
-
@electric-sql/client throws SecurityError on import when the browser blocks site storage (typeof localStorage outside try)Forse già presa @mohanedmoh l’ha presa 37 giorni fa. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
electric-sql/electric#4826 · 1 commento ·
-
agents-chat-starter: spawn PUT goes to the durable-streams proxy, so no entity is created and agents never wakeForse già presa @adityavkk l’ha presa 133 giorni fa. Aperta
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
electric-sql/electric#4687 ·
-
Listen IP addressApertafeature request triage
Difficoltà 2/5 1-3 ore Idoneità per principianti 68/100
electric-sql/electric#4150 · 1 commento ·
Tutte le issue di electric-sql/electric
Issue simili
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 82/100
software-mansion/argent#1946 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 63/100
podman-desktop/podman-desktop#20216 ·
I maintainer di solito rispondono entro 1 giorno
-
[Type] Bug
Difficoltà 2/5 1-3 ore Idoneità per principianti 76/100
WordPress/ai#1386 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
I maintainer di solito rispondono entro 5 giorni
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 65/100
omdsh-dev/DSH-better-sidebar#1205 ·
I maintainer di solito rispondono entro 3 giorni