Forward a connection to a peer when its local upstream connect fails
Maintainers usually reply within 1 day
@kriszyp is already working on this.
Since Nov 5, 2026.
Assessment
This issue has not been assessed yet.
Description
On a route with peer failover, a connection whose local upstream connect fails is dropped instead of forwarded to a peer. Peer failover only covers connections that arrive after the route is marked down, so the connections that discover an outage are lost.
Observed (symphony 1.6.0, two-node Fabric stage cluster)
The local Harper container on one node was stopped, and new TLS connections were sent to that node's public ports about 2 s later:
| Port | 1st connection | Next connections |
|---|---|---|
| 443 | forwarded | forwarded |
| 8883 | forwarded | forwarded |
| 9925 | dropped after 6 ms, upstream_connect error on the local node |
forwarded to the peer's 19925 |
| 9926 | dropped after 7 ms | forwarded to the peer's 10443 |
A packet capture on the peer shows no ingress connection for the dropped attempts. Earlier traffic had most likely already marked the 443 and 8883 routes down, which is why their first connection was forwarded.
Cause
src/proxy_conn.rs (step 3d) makes the failover decision before connecting: failover::decide() forwards only when Failover::locally_down() (src/failover.rs) reports every slot already refusing. When the decision is Local and upstream::connect then fails, proxy_raw / proxy_via_tls return ErrorKind::UpstreamConnect and the client is closed. No peer is tried. A slot is marked refusing either by such a failed connect or by the next probe (every probeIntervalMs, default 1 s).
Impact: a planned restart is unaffected, because host-manager drains the routes first. A crash, an OOM kill or an unplanned stop drops every connection that arrives before the route is marked, which is roughly one probe interval of traffic per route on a busy node.
Change
On a route with a failover block, not suspended, on a client listener (not ingress) and in passthrough mode (terminateTls: false, which is every Fabric route): when the local upstream connect fails, call forward_to_peers in ForwardMode::Down instead of closing. Nothing has been consumed from the client yet (step 1 only peeks the ClientHello), so the connection can still be replayed to a peer, as a Decision::Forward connection is.
- Count the local failure as today (
upstream_connect, and the slot's refused mark), then count the forward's own outcome. - Bound the total: the local connect plus peer attempts must stay within the handshake budget the
Downpath already uses. A local connect that times out (rather than being refused) can spend most of that budget, so pass the time left toforward_to_peersas theSpillfallback does. - If every peer fails, close as today.
- TLS-terminating routes are out of scope: by the time they connect upstream, the handshake has consumed the ClientHello and the raw bytes can no longer be replayed. Note this in the docs.
Tests
- Passthrough route with a peer, local upstream refusing and not yet marked down: the first connection is served by the peer. This test fails on v1.6.0.
- Same with a UDS upstream whose socket file still exists but nothing listens on it (a stopped container), and with a missing socket file.
- Local connect times out: the peer forward gets the remaining budget, and the total stays within the handshake timeout.
- All peers refuse: the client is closed, with
upstream_connectcounted. - Ingress listener: a failed local connect still closes with no bytes, with no forward to a peer.
- TLS-terminating route: behavior unchanged.
— Claude Opus 5.5, drafted with Kris
- Dominant language
- TypeScript
- Stars
- 1
- Forks
- 0
- Avg merge
- 11h 49m
- Merged PRs (30d)
- 12
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from HarperFast/symphony
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
HarperFast/symphony#91 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
HarperFast/symphony#89 ·
Maintainers usually reply within 1 day
-
Mixed-case SNI forwarded to an older peer during a rolling upgrade can reach its wildcard routeOpen
Difficulty 4/5 3-5 days Newbie friendliness 49/100
HarperFast/symphony#93 ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 42/100
HarperFast/symphony#92 ·
Maintainers usually reply within 1 day
-
Difficulty 3/5 1-2 days Newbie friendliness 72/100
HarperFast/symphony#90 ·
Maintainers usually reply within 1 day
All issues in HarperFast/symphony
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Maintainers usually reply within 1 day
-
feature request needs triage
Difficulty 1/5 1-3 hours Newbie friendliness 69/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
solidjs/solid-docs#1716 ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 80/100
thecodacus/pithagoras#380 ·
Maintainers usually reply within 1 day