Forward a connection to a peer when its local upstream connect fails

Open
#79 0 comments 0 reactions 1 assignee View on GitHub

Maintainers usually reply within 1 day

@kriszyp is already working on this.

Since Nov 5, 2026.

Assessment

This issue has not been assessed yet.

Description

On a route with peer failover, a connection whose local upstream connect fails is dropped instead of forwarded to a peer. Peer failover only covers connections that arrive after the route is marked down, so the connections that discover an outage are lost.

Follows Peer failover: forward a route's new connections to other cluster nodes while its local upstream is down (#70).

Observed (symphony 1.6.0, two-node Fabric stage cluster)

The local Harper container on one node was stopped, and new TLS connections were sent to that node's public ports about 2 s later:

Port 1st connection Next connections
443 forwarded forwarded
8883 forwarded forwarded
9925 dropped after 6 ms, upstream_connect error on the local node forwarded to the peer's 19925
9926 dropped after 7 ms forwarded to the peer's 10443

A packet capture on the peer shows no ingress connection for the dropped attempts. Earlier traffic had most likely already marked the 443 and 8883 routes down, which is why their first connection was forwarded.

Cause

src/proxy_conn.rs (step 3d) makes the failover decision before connecting: failover::decide() forwards only when Failover::locally_down() (src/failover.rs) reports every slot already refusing. When the decision is Local and upstream::connect then fails, proxy_raw / proxy_via_tls return ErrorKind::UpstreamConnect and the client is closed. No peer is tried. A slot is marked refusing either by such a failed connect or by the next probe (every probeIntervalMs, default 1 s).

Impact: a planned restart is unaffected, because host-manager drains the routes first. A crash, an OOM kill or an unplanned stop drops every connection that arrives before the route is marked, which is roughly one probe interval of traffic per route on a busy node.

Change

On a route with a failover block, not suspended, on a client listener (not ingress) and in passthrough mode (terminateTls: false, which is every Fabric route): when the local upstream connect fails, call forward_to_peers in ForwardMode::Down instead of closing. Nothing has been consumed from the client yet (step 1 only peeks the ClientHello), so the connection can still be replayed to a peer, as a Decision::Forward connection is.

  • Count the local failure as today (upstream_connect, and the slot's refused mark), then count the forward's own outcome.
  • Bound the total: the local connect plus peer attempts must stay within the handshake budget the Down path already uses. A local connect that times out (rather than being refused) can spend most of that budget, so pass the time left to forward_to_peers as the Spill fallback does.
  • If every peer fails, close as today.
  • TLS-terminating routes are out of scope: by the time they connect upstream, the handshake has consumed the ClientHello and the raw bytes can no longer be replayed. Note this in the docs.

Tests

  • Passthrough route with a peer, local upstream refusing and not yet marked down: the first connection is served by the peer. This test fails on v1.6.0.
  • Same with a UDS upstream whose socket file still exists but nothing listens on it (a stopped container), and with a missing socket file.
  • Local connect times out: the peer forward gets the remaining budget, and the total stays within the handshake timeout.
  • All peers refuse: the client is closed, with upstream_connect counted.
  • Ingress listener: a failed local connect still closes with no bytes, with no forward to a peer.
  • TLS-terminating route: behavior unchanged.

— Claude Opus 5.5, drafted with Kris

Dominant language
TypeScript
Stars
1
Forks
0
Avg merge
11h 49m
Merged PRs (30d)
12

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from HarperFast/symphony

All issues in HarperFast/symphony

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.