Forward a connection to a peer when its local upstream connect fails

Abierto
  • ¿Algo no está bien?
#79 0 comentarios 0 reacciones 1 asignado Ver en GitHub

Los mantenedores suelen responder en 1 día

@kriszyp ya está trabajando en esto.

Desde el 5/11/2026.

Evaluación

Este issue todavía no se ha evaluado.

Descripción

On a route with peer failover, a connection whose local upstream connect fails is dropped instead of forwarded to a peer. Peer failover only covers connections that arrive after the route is marked down, so the connections that discover an outage are lost.

Follows Peer failover: forward a route's new connections to other cluster nodes while its local upstream is down (#70).

Observed (symphony 1.6.0, two-node Fabric stage cluster)

The local Harper container on one node was stopped, and new TLS connections were sent to that node's public ports about 2 s later:

Port 1st connection Next connections
443 forwarded forwarded
8883 forwarded forwarded
9925 dropped after 6 ms, upstream_connect error on the local node forwarded to the peer's 19925
9926 dropped after 7 ms forwarded to the peer's 10443

A packet capture on the peer shows no ingress connection for the dropped attempts. Earlier traffic had most likely already marked the 443 and 8883 routes down, which is why their first connection was forwarded.

Cause

src/proxy_conn.rs (step 3d) makes the failover decision before connecting: failover::decide() forwards only when Failover::locally_down() (src/failover.rs) reports every slot already refusing. When the decision is Local and upstream::connect then fails, proxy_raw / proxy_via_tls return ErrorKind::UpstreamConnect and the client is closed. No peer is tried. A slot is marked refusing either by such a failed connect or by the next probe (every probeIntervalMs, default 1 s).

Impact: a planned restart is unaffected, because host-manager drains the routes first. A crash, an OOM kill or an unplanned stop drops every connection that arrives before the route is marked, which is roughly one probe interval of traffic per route on a busy node.

Change

On a route with a failover block, not suspended, on a client listener (not ingress) and in passthrough mode (terminateTls: false, which is every Fabric route): when the local upstream connect fails, call forward_to_peers in ForwardMode::Down instead of closing. Nothing has been consumed from the client yet (step 1 only peeks the ClientHello), so the connection can still be replayed to a peer, as a Decision::Forward connection is.

  • Count the local failure as today (upstream_connect, and the slot's refused mark), then count the forward's own outcome.
  • Bound the total: the local connect plus peer attempts must stay within the handshake budget the Down path already uses. A local connect that times out (rather than being refused) can spend most of that budget, so pass the time left to forward_to_peers as the Spill fallback does.
  • If every peer fails, close as today.
  • TLS-terminating routes are out of scope: by the time they connect upstream, the handshake has consumed the ClientHello and the raw bytes can no longer be replayed. Note this in the docs.

Tests

  • Passthrough route with a peer, local upstream refusing and not yet marked down: the first connection is served by the peer. This test fails on v1.6.0.
  • Same with a UDS upstream whose socket file still exists but nothing listens on it (a stopped container), and with a missing socket file.
  • Local connect times out: the peer forward gets the remaining budget, and the total stays within the handshake timeout.
  • All peers refuse: the client is closed, with upstream_connect counted.
  • Ingress listener: a failed local connect still closes with no bytes, with no forward to a peer.
  • TLS-terminating route: behavior unchanged.

— Claude Opus 5.5, drafted with Kris

Lenguaje dominante
Rust
Estrellas
1
Forks
0
Merge medio
19 h 12 min
PR fusionados (30 d)
13

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de HarperFast/symphony

Todos los issues de HarperFast/symphony

Issues similares

Más issues de Rust

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.