Spill a share of new connections to a less-loaded peer when local Harper threads are busy

オープン
  • 何か問題がありますか？
#73 コメント 0 件 リアクション 0 件 担当者 1 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

@kriszyp がすでに取り組んでいます。

2026年10月29日 から。

評価

この issue はまだ評価されていません。

説明

Why

Symphony already balances each new connection across the local Harper worker threads by load, and since Peer failover (HarperFast/symphony#71) it forwards a route's new connections to peer nodes while the local upstream is down or drained. It does not move load between nodes while both are up.

Today, load between nodes is balanced only at the DNS layer: Akamai GTM polls each node's /current_load_for_gtm (HarperFast/akamai-status) every 30 s and shifts DNS answers. DNS setups we expect to use for some clusters have no load feedback at all: latency or geoproximity records without health checks (for example Route 53 or Google Cloud DNS). On those, a hot node keeps receiving its full share of traffic no matter how busy it is. Even with GTM, the reaction time is the 30 s poll plus DNS TTL plus client DNS caching.

Symphony can react in seconds and needs no DNS support: when this node's Harper threads are much busier than a peer's, forward a share of new connections to that peer through the existing failover path.

What exists to build on

  • Local load signal: src/balancer.rs samples each UDS slot's Harper thread CPU from /proc/{pid}/task/{tid}/stat every 250 ms (cpu_util_permille). host-manager passes pid/tid for every Harper worker socket.
  • Forwarding: failover::forward_to_peers sends the peeked ClientHello plus a PROXY v2 header to a peer's failover ingress listener; the peer terminates TLS. The forwarding node only copies bytes after that, so a spilled connection costs it very little.
  • No re-forwarding: a connection that arrived on the failover ingress is never forwarded again (src/proxy_conn.rs, step 3d). Spill-over must keep this: only connections from clients can spill.

Proposed behavior

  1. Node load: a smoothed (a few seconds) CPU utilization of the local Harper threads, from the existing samples. Not connection count: MQTT and WebSocket connections are mostly idle, so count says little about CPU.
  2. Peer load: each node learns its peers' load. Proposed transport: the failover ingress already requires a PROXY v2 header from a trusted source; a PROXY v2 LOCAL command (the spec's health-check form, rejected today) could be answered with the node's load and closed. Polled by the existing probe task, at probeIntervalMs. This needs no new port or channel.
  3. Spill rule, per failover route, on client connections only: when local load is above a high-water mark and above the least-loaded ready peer's load by a margin, forward a fraction of new connections to that peer. The fraction rises in small steps while the condition holds and decays when it clears, capped at a maximum, so two nodes don't swing traffic back and forth. Drain and local-down keep their current behavior (forward everything).
  4. Do nothing when every node is busy: if no peer is below the margin, keep serving locally; relaying would add a hop without adding capacity.
  5. Observability: counters for spilled connections per route, a gauge for the current spill fraction, and the last-known load of each peer.

Tradeoffs

  • A spilled connection is relayed through the hot node for its whole life, so it pays an extra network hop. Across regions that adds latency and egress cost. Where GTM load feedback exists it remains the better steering, because the client connects to the other node directly; spill-over complements it for short spikes and replaces it where DNS has no load feedback.
  • Only new connections move. Long-lived connections stay where they landed.

Open design questions

  • Peer load transport: PROXY v2 LOCAL on the ingress (above), or something else.
  • Load scope: one figure per node, or per destination (routes can point at different worker sets).
  • Config surface on failover: defaults for high-water mark, margin, step and maximum fraction, and whether spill-over is opt-in per route.
  • Whether to limit spilling to peers below a connect-RTT ceiling, to avoid relaying across regions.
  • Whether the thread CPU samples read correctly from host-manager's pid/tid on a host where Harper runs in a container (PID namespaces).

Enabling it from host-manager is a separate follow-up.

— Claude Opus 5.5, drafted with Kris

主要言語
Rust
スター
1
フォーク
0
平均マージ
19時間 12分
マージ済み PR（30日）
13

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

HarperFast/symphony のほかの issue

HarperFast/symphony の issue をすべて見る

似ている issue

Rust の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。