Forward a connection to a peer when its local upstream connect fails
メンテナーはふだん 1 日以内に返信
@kriszyp がすでに取り組んでいます。
2026年11月5日 から。
評価
この issue はまだ評価されていません。
説明
On a route with peer failover, a connection whose local upstream connect fails is dropped instead of forwarded to a peer. Peer failover only covers connections that arrive after the route is marked down, so the connections that discover an outage are lost.
Observed (symphony 1.6.0, two-node Fabric stage cluster)
The local Harper container on one node was stopped, and new TLS connections were sent to that node's public ports about 2 s later:
| Port | 1st connection | Next connections |
|---|---|---|
| 443 | forwarded | forwarded |
| 8883 | forwarded | forwarded |
| 9925 | dropped after 6 ms, upstream_connect error on the local node |
forwarded to the peer's 19925 |
| 9926 | dropped after 7 ms | forwarded to the peer's 10443 |
A packet capture on the peer shows no ingress connection for the dropped attempts. Earlier traffic had most likely already marked the 443 and 8883 routes down, which is why their first connection was forwarded.
Cause
src/proxy_conn.rs (step 3d) makes the failover decision before connecting: failover::decide() forwards only when Failover::locally_down() (src/failover.rs) reports every slot already refusing. When the decision is Local and upstream::connect then fails, proxy_raw / proxy_via_tls return ErrorKind::UpstreamConnect and the client is closed. No peer is tried. A slot is marked refusing either by such a failed connect or by the next probe (every probeIntervalMs, default 1 s).
Impact: a planned restart is unaffected, because host-manager drains the routes first. A crash, an OOM kill or an unplanned stop drops every connection that arrives before the route is marked, which is roughly one probe interval of traffic per route on a busy node.
Change
On a route with a failover block, not suspended, on a client listener (not ingress) and in passthrough mode (terminateTls: false, which is every Fabric route): when the local upstream connect fails, call forward_to_peers in ForwardMode::Down instead of closing. Nothing has been consumed from the client yet (step 1 only peeks the ClientHello), so the connection can still be replayed to a peer, as a Decision::Forward connection is.
- Count the local failure as today (
upstream_connect, and the slot's refused mark), then count the forward's own outcome. - Bound the total: the local connect plus peer attempts must stay within the handshake budget the
Downpath already uses. A local connect that times out (rather than being refused) can spend most of that budget, so pass the time left toforward_to_peersas theSpillfallback does. - If every peer fails, close as today.
- TLS-terminating routes are out of scope: by the time they connect upstream, the handshake has consumed the ClientHello and the raw bytes can no longer be replayed. Note this in the docs.
Tests
- Passthrough route with a peer, local upstream refusing and not yet marked down: the first connection is served by the peer. This test fails on v1.6.0.
- Same with a UDS upstream whose socket file still exists but nothing listens on it (a stopped container), and with a missing socket file.
- Local connect times out: the peer forward gets the remaining budget, and the total stays within the handshake timeout.
- All peers refuse: the client is closed, with
upstream_connectcounted. - Ingress listener: a failed local connect still closes with no bytes, with no forward to a peer.
- TLS-terminating route: behavior unchanged.
— Claude Opus 5.5, drafted with Kris
- 主要言語
- Rust
- スター
- 1
- フォーク
- 0
- 平均マージ
- 19時間 12分
- マージ済み PR（30日）
- 13
環境構築
- Dockerfile・Docker Compose ファイルなし
- プルリクエストのテンプレートなし
- コントリビューションガイドを読む
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
HarperFast/symphony のほかの issue
-
Don't route to a restarted upstream until it serves, not just accepts対応中かも @kriszyp が 5 日前に担当しました。 オープン
HarperFast/symphony#84 · 担当者 1 名 ·
メンテナーはふだん 1 日以内に返信
-
Forward to a peer when the local upstream connect fails on a TLS-terminating failover route対応中かも @kriszyp が 5 日前に担当しました。 オープン
HarperFast/symphony#83 · 担当者 1 名 ·
メンテナーはふだん 1 日以内に返信
-
難易度 4/5 3〜5日 初心者へのやさしさ 48/100
HarperFast/symphony#80 ·
メンテナーはふだん 1 日以内に返信
-
難易度 4/5 3〜5日 初心者へのやさしさ 35/100
HarperFast/symphony#77 ·
メンテナーはふだん 1 日以内に返信
-
Spill a share of new connections to a less-loaded peer when local Harper threads are busy対応中かも @kriszyp が 12 日前に担当しました。 オープン
HarperFast/symphony#73 · 担当者 1 名 ·
メンテナーはふだん 1 日以内に返信
HarperFast/symphony の issue をすべて見る
似ている issue
-
area: cli kind:docs priority: P3
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
メンテナーはふだん 1 日以内に返信
-
bug
難易度 2/5 1〜3時間 初心者へのやさしさ 88/100
swc-project/swc-ai-issues#164 ·
-
難易度 2/5 1〜3時間 初心者へのやさしさ 85/100
pop-os/cosmic-player#367 ·
-
good first issue help wanted
難易度 2/5 1〜3時間 初心者へのやさしさ 72/100
servo/rust-smallvec#873 ·