Don't route to a restarted upstream until it serves, not just accepts

Open
#84 0 comments 0 reactions 1 assignee View on GitHub

Maintainers usually reply within 1 day

@kriszyp is already working on this.

Since Nov 5, 2026.

Assessment

This issue has not been assessed yet.

Description

When a local Harper restarts, symphony marks a failover route up again about 1 s before Harper serves on it. Every connection in that window is dropped.

Observed (symphony 1.6.0, two-node Fabric stage cluster, Harper 5.3.5)

The route is 443, TLS-terminating, with a uds upstream (<instance>-sockets/0-9926.sock). The local container was stopped and then started again, with a new 443 connection every 0.25 s:

After docker start New 443 connections
0–3.1 s forwarded to the peer (localUp: false)
3.1 s the probe clears the mark (localUp: true, refusingSlots: 0)
3.3–4.1 s 4 of 4 dropped, 4–7 ms each, counted as route error stream
4.4 s onward served locally

The upstream accepted the connection and then closed it without a response. Every restart of a node with failover therefore loses about 1 s of new connections on each route, the same exposure #79 addressed for the stop side.

Cause (partly confirmed)

failover::probe (src/failover.rs) is a connect-and-close. Its own comment says it "proves the socket accepts, not that the application behind it is ready", and a successful probe clears the slot. Harper's socket accepts about 1 s before Harper serves on it.

Not yet known: what accepts during that window, and whether Harper binds its per-thread sockets before its HTTP handler is ready. The socket files' mtimes in that run were later than the first drops, which suggests an earlier bind or a rebind.

What to settle

  1. Where readiness belongs. If Harper binds or accepts on these sockets before it can serve, the root fix is in Harper: bind only when ready. symphony can't recover a TLS-terminated connection after its handshake, so a symphony-only fix can't be complete there. If Harper is correct and something else accepts, find it.
  2. What symphony should do either way. For example: require the probe to see the socket accepting continuously for a hold-down period before clearing a slot that was marked down, or have the probe wait for a protocol-level signal. Harper writes a .yaml beside each socket; check whether it carries a readiness signal.

Tests

  • symphony: a UDS upstream that accepts and immediately closes for its first N connections after coming back must not have new connections routed to it during that period, or they must be retried at a peer.
  • End to end on a two-node setup: restart one node's Harper under steady connection load, and no connection fails.

Found during Fabric stage failover testing. — Claude Opus 5.5, drafted with Kris

Dominant language
TypeScript
Stars
1
Forks
0
Avg merge
11h 49m
Merged PRs (30d)
12

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from HarperFast/symphony

All issues in HarperFast/symphony

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.