Forward to a peer when the local upstream connect fails on a TLS-terminating failover route

Abierto
  • ¿Algo no está bien?
#83 0 comentarios 0 reacciones 1 asignado Ver en GitHub

Los mantenedores suelen responder en 1 día

@kriszyp ya está trabajando en esto.

Desde el 5/11/2026.

Evaluación

Este issue todavía no se ha evaluado.

Descripción

On a TLS-terminating route with peer failover, the first connections after the local upstream stops are still dropped. Forward a connection to a peer when its local upstream connect fails (#79), fixed by #81, covers only passthrough routes. That was meant to be every Fabric route, but it isn't.

Observed (symphony 1.6.0, two-node Fabric stage cluster, Harper 5.3.5)

On Harper versions with per-thread Unix sockets, host-manager configures every failover route (443, 9925, 9926, 8883) as terminateTls: true with a uds upstream. Passthrough is used only for TCP upstreams (older Harper versions).

The local Harper container was stopped, and 2 s later a TLS connection went to 9926. It was dropped after 6 ms, counted as upstream_connect, with no forward to the peer. The connections after it were forwarded, because that failure had marked the slot refusing.

symphony 1.7.1 behaves the same on these routes: #81's forward runs only in the passthrough branch of handle (src/proxy_conn.rs).

Cause

A TLS-terminating route completes the client handshake (tls_accept) before proxy_via_tls connects upstream. Once the handshake has consumed the ClientHello, the connection can no longer be replayed raw to a peer's failoverIngress. failover::decide() runs before the handshake but forwards only a route already marked down.

Proposed change

On a TLS-terminating failover route, on a client listener, not suspended: connect upstream before the TLS handshake. If the connect fails, forward the untouched hello with forward_to_peers (ForwardMode::Down) exactly as #81 does for passthrough. If it succeeds, hand the connected upstream to proxy_via_tls after the handshake. A Unix-socket connect costs well under 1 ms, so the extra latency is negligible.

The open design question is h1 vs h2. proxy_via_tls picks destination or destination_h2 from the ALPN the handshake negotiates, which isn't known before the handshake. Options:

  1. Predict the negotiated protocol from the ClientHello's ALPN offer and the route's ServerConfig ALPN list (rustls picks by server preference), then connect to that destination. If the handshake negotiates something else, connect the other destination afterward, as today.
  2. Probe both destinations with a connect-and-close before the handshake, then connect for real afterward. Simpler, but adds an extra accept and close on Harper for every connection.
  3. Connect the h1 destination first and keep it only when h1 is negotiated.

Prefer option 1 unless the investigation shows a problem with it.

Constraints:

  • Keep the per-connection future within its 4 KiB budget (the_per_connection_future_stays_within_budget). An UpstreamStream held across the handshake may need boxing.
  • Reuse #81's budget handling: a local connect capped at half of hs_timeout, and peers get the remainder.
  • upstream_connect accounting stays the same as #81.

Tests

  • TLS-terminating route with a peer, local UDS refusing and not yet marked down: the first connection is served by the peer. This test fails on 1.7.1.
  • Same with a stale socket file (no listener) and with a missing socket file.
  • An h2-offering client on a route with destination_h2: the correct destination is used.
  • All peers refuse: the client is closed, and upstream_connect is counted.
  • Ingress listener and suspended routes: unchanged.

Found during Fabric stage failover testing. — Claude Opus 5.5, drafted with Kris

Lenguaje dominante
Rust
Estrellas
1
Forks
0
Merge medio
19 h 12 min
PR fusionados (30 d)
13

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de HarperFast/symphony

Todos los issues de HarperFast/symphony

Issues similares

Más issues de Rust

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.