Spill a share of new connections to a less-loaded peer when local Harper threads are busy

Đang mở
  • Có gì không ổn?
#73 0 bình luận 0 reaction 1 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

@kriszyp đang làm issue này rồi.

Từ ngày 29/10/2026.

Đánh giá

Issue này chưa được đánh giá.

Mô tả

Why

Symphony already balances each new connection across the local Harper worker threads by load, and since Peer failover (HarperFast/symphony#71) it forwards a route's new connections to peer nodes while the local upstream is down or drained. It does not move load between nodes while both are up.

Today, load between nodes is balanced only at the DNS layer: Akamai GTM polls each node's /current_load_for_gtm (HarperFast/akamai-status) every 30 s and shifts DNS answers. DNS setups we expect to use for some clusters have no load feedback at all: latency or geoproximity records without health checks (for example Route 53 or Google Cloud DNS). On those, a hot node keeps receiving its full share of traffic no matter how busy it is. Even with GTM, the reaction time is the 30 s poll plus DNS TTL plus client DNS caching.

Symphony can react in seconds and needs no DNS support: when this node's Harper threads are much busier than a peer's, forward a share of new connections to that peer through the existing failover path.

What exists to build on

  • Local load signal: src/balancer.rs samples each UDS slot's Harper thread CPU from /proc/{pid}/task/{tid}/stat every 250 ms (cpu_util_permille). host-manager passes pid/tid for every Harper worker socket.
  • Forwarding: failover::forward_to_peers sends the peeked ClientHello plus a PROXY v2 header to a peer's failover ingress listener; the peer terminates TLS. The forwarding node only copies bytes after that, so a spilled connection costs it very little.
  • No re-forwarding: a connection that arrived on the failover ingress is never forwarded again (src/proxy_conn.rs, step 3d). Spill-over must keep this: only connections from clients can spill.

Proposed behavior

  1. Node load: a smoothed (a few seconds) CPU utilization of the local Harper threads, from the existing samples. Not connection count: MQTT and WebSocket connections are mostly idle, so count says little about CPU.
  2. Peer load: each node learns its peers' load. Proposed transport: the failover ingress already requires a PROXY v2 header from a trusted source; a PROXY v2 LOCAL command (the spec's health-check form, rejected today) could be answered with the node's load and closed. Polled by the existing probe task, at probeIntervalMs. This needs no new port or channel.
  3. Spill rule, per failover route, on client connections only: when local load is above a high-water mark and above the least-loaded ready peer's load by a margin, forward a fraction of new connections to that peer. The fraction rises in small steps while the condition holds and decays when it clears, capped at a maximum, so two nodes don't swing traffic back and forth. Drain and local-down keep their current behavior (forward everything).
  4. Do nothing when every node is busy: if no peer is below the margin, keep serving locally; relaying would add a hop without adding capacity.
  5. Observability: counters for spilled connections per route, a gauge for the current spill fraction, and the last-known load of each peer.

Tradeoffs

  • A spilled connection is relayed through the hot node for its whole life, so it pays an extra network hop. Across regions that adds latency and egress cost. Where GTM load feedback exists it remains the better steering, because the client connects to the other node directly; spill-over complements it for short spikes and replaces it where DNS has no load feedback.
  • Only new connections move. Long-lived connections stay where they landed.

Open design questions

  • Peer load transport: PROXY v2 LOCAL on the ingress (above), or something else.
  • Load scope: one figure per node, or per destination (routes can point at different worker sets).
  • Config surface on failover: defaults for high-water mark, margin, step and maximum fraction, and whether spill-over is opt-in per route.
  • Whether to limit spilling to peers below a connect-RTT ceiling, to avoid relaying across regions.
  • Whether the thread CPU samples read correctly from host-manager's pid/tid on a host where Harper runs in a container (PID namespaces).

Enabling it from host-manager is a separate follow-up.

— Claude Opus 5.5, drafted with Kris

Ngôn ngữ chính
Rust
Star
1
Fork
0
Merge trung bình
19 giờ 12 phút
Pull request đã merge (30 ngày)
13

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của HarperFast/symphony

Tất cả issue của HarperFast/symphony

Issue tương tự

Thêm issue về Rust

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.