Forward to a peer when the local upstream connect fails on a TLS-terminating failover route

オープン
  • 何か問題がありますか？
#83 コメント 0 件 リアクション 0 件 担当者 1 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

@kriszyp がすでに取り組んでいます。

2026年11月5日 から。

評価

この issue はまだ評価されていません。

説明

On a TLS-terminating route with peer failover, the first connections after the local upstream stops are still dropped. Forward a connection to a peer when its local upstream connect fails (#79), fixed by #81, covers only passthrough routes. That was meant to be every Fabric route, but it isn't.

Observed (symphony 1.6.0, two-node Fabric stage cluster, Harper 5.3.5)

On Harper versions with per-thread Unix sockets, host-manager configures every failover route (443, 9925, 9926, 8883) as terminateTls: true with a uds upstream. Passthrough is used only for TCP upstreams (older Harper versions).

The local Harper container was stopped, and 2 s later a TLS connection went to 9926. It was dropped after 6 ms, counted as upstream_connect, with no forward to the peer. The connections after it were forwarded, because that failure had marked the slot refusing.

symphony 1.7.1 behaves the same on these routes: #81's forward runs only in the passthrough branch of handle (src/proxy_conn.rs).

Cause

A TLS-terminating route completes the client handshake (tls_accept) before proxy_via_tls connects upstream. Once the handshake has consumed the ClientHello, the connection can no longer be replayed raw to a peer's failoverIngress. failover::decide() runs before the handshake but forwards only a route already marked down.

Proposed change

On a TLS-terminating failover route, on a client listener, not suspended: connect upstream before the TLS handshake. If the connect fails, forward the untouched hello with forward_to_peers (ForwardMode::Down) exactly as #81 does for passthrough. If it succeeds, hand the connected upstream to proxy_via_tls after the handshake. A Unix-socket connect costs well under 1 ms, so the extra latency is negligible.

The open design question is h1 vs h2. proxy_via_tls picks destination or destination_h2 from the ALPN the handshake negotiates, which isn't known before the handshake. Options:

  1. Predict the negotiated protocol from the ClientHello's ALPN offer and the route's ServerConfig ALPN list (rustls picks by server preference), then connect to that destination. If the handshake negotiates something else, connect the other destination afterward, as today.
  2. Probe both destinations with a connect-and-close before the handshake, then connect for real afterward. Simpler, but adds an extra accept and close on Harper for every connection.
  3. Connect the h1 destination first and keep it only when h1 is negotiated.

Prefer option 1 unless the investigation shows a problem with it.

Constraints:

  • Keep the per-connection future within its 4 KiB budget (the_per_connection_future_stays_within_budget). An UpstreamStream held across the handshake may need boxing.
  • Reuse #81's budget handling: a local connect capped at half of hs_timeout, and peers get the remainder.
  • upstream_connect accounting stays the same as #81.

Tests

  • TLS-terminating route with a peer, local UDS refusing and not yet marked down: the first connection is served by the peer. This test fails on 1.7.1.
  • Same with a stale socket file (no listener) and with a missing socket file.
  • An h2-offering client on a route with destination_h2: the correct destination is used.
  • All peers refuse: the client is closed, and upstream_connect is counted.
  • Ingress listener and suspended routes: unchanged.

Found during Fabric stage failover testing. — Claude Opus 5.5, drafted with Kris

主要言語
Rust
スター
1
フォーク
0
平均マージ
19時間 12分
マージ済み PR（30日）
13

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

HarperFast/symphony のほかの issue

HarperFast/symphony の issue をすべて見る

似ている issue

Rust の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。