Forward to a peer when the local upstream connect fails on a TLS-terminating failover route

Đang mở
  • Có gì không ổn?
#83 0 bình luận 0 reaction 1 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

@kriszyp đang làm issue này rồi.

Từ ngày 5/11/2026.

Đánh giá

Issue này chưa được đánh giá.

Mô tả

On a TLS-terminating route with peer failover, the first connections after the local upstream stops are still dropped. Forward a connection to a peer when its local upstream connect fails (#79), fixed by #81, covers only passthrough routes. That was meant to be every Fabric route, but it isn't.

Observed (symphony 1.6.0, two-node Fabric stage cluster, Harper 5.3.5)

On Harper versions with per-thread Unix sockets, host-manager configures every failover route (443, 9925, 9926, 8883) as terminateTls: true with a uds upstream. Passthrough is used only for TCP upstreams (older Harper versions).

The local Harper container was stopped, and 2 s later a TLS connection went to 9926. It was dropped after 6 ms, counted as upstream_connect, with no forward to the peer. The connections after it were forwarded, because that failure had marked the slot refusing.

symphony 1.7.1 behaves the same on these routes: #81's forward runs only in the passthrough branch of handle (src/proxy_conn.rs).

Cause

A TLS-terminating route completes the client handshake (tls_accept) before proxy_via_tls connects upstream. Once the handshake has consumed the ClientHello, the connection can no longer be replayed raw to a peer's failoverIngress. failover::decide() runs before the handshake but forwards only a route already marked down.

Proposed change

On a TLS-terminating failover route, on a client listener, not suspended: connect upstream before the TLS handshake. If the connect fails, forward the untouched hello with forward_to_peers (ForwardMode::Down) exactly as #81 does for passthrough. If it succeeds, hand the connected upstream to proxy_via_tls after the handshake. A Unix-socket connect costs well under 1 ms, so the extra latency is negligible.

The open design question is h1 vs h2. proxy_via_tls picks destination or destination_h2 from the ALPN the handshake negotiates, which isn't known before the handshake. Options:

  1. Predict the negotiated protocol from the ClientHello's ALPN offer and the route's ServerConfig ALPN list (rustls picks by server preference), then connect to that destination. If the handshake negotiates something else, connect the other destination afterward, as today.
  2. Probe both destinations with a connect-and-close before the handshake, then connect for real afterward. Simpler, but adds an extra accept and close on Harper for every connection.
  3. Connect the h1 destination first and keep it only when h1 is negotiated.

Prefer option 1 unless the investigation shows a problem with it.

Constraints:

  • Keep the per-connection future within its 4 KiB budget (the_per_connection_future_stays_within_budget). An UpstreamStream held across the handshake may need boxing.
  • Reuse #81's budget handling: a local connect capped at half of hs_timeout, and peers get the remainder.
  • upstream_connect accounting stays the same as #81.

Tests

  • TLS-terminating route with a peer, local UDS refusing and not yet marked down: the first connection is served by the peer. This test fails on 1.7.1.
  • Same with a stale socket file (no listener) and with a missing socket file.
  • An h2-offering client on a route with destination_h2: the correct destination is used.
  • All peers refuse: the client is closed, and upstream_connect is counted.
  • Ingress listener and suspended routes: unchanged.

Found during Fabric stage failover testing. — Claude Opus 5.5, drafted with Kris

Ngôn ngữ chính
Rust
Star
1
Fork
0
Merge trung bình
19 giờ 12 phút
Pull request đã merge (30 ngày)
13

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của HarperFast/symphony

Tất cả issue của HarperFast/symphony

Issue tương tự

Thêm issue về Rust

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.