Spill a share of new connections to a less-loaded peer when local Harper threads are busy

Open
#73 0 comments 0 reactions 1 assignee View on GitHub

Maintainers usually reply within 1 day

@kriszyp is already working on this.

Since Oct 29, 2026.

Assessment

This issue has not been assessed yet.

Description

Why

Symphony already balances each new connection across the local Harper worker threads by load, and since Peer failover (HarperFast/symphony#71) it forwards a route's new connections to peer nodes while the local upstream is down or drained. It does not move load between nodes while both are up.

Today, load between nodes is balanced only at the DNS layer: Akamai GTM polls each node's /current_load_for_gtm (HarperFast/akamai-status) every 30 s and shifts DNS answers. DNS setups we expect to use for some clusters have no load feedback at all: latency or geoproximity records without health checks (for example Route 53 or Google Cloud DNS). On those, a hot node keeps receiving its full share of traffic no matter how busy it is. Even with GTM, the reaction time is the 30 s poll plus DNS TTL plus client DNS caching.

Symphony can react in seconds and needs no DNS support: when this node's Harper threads are much busier than a peer's, forward a share of new connections to that peer through the existing failover path.

What exists to build on

  • Local load signal: src/balancer.rs samples each UDS slot's Harper thread CPU from /proc/{pid}/task/{tid}/stat every 250 ms (cpu_util_permille). host-manager passes pid/tid for every Harper worker socket.
  • Forwarding: failover::forward_to_peers sends the peeked ClientHello plus a PROXY v2 header to a peer's failover ingress listener; the peer terminates TLS. The forwarding node only copies bytes after that, so a spilled connection costs it very little.
  • No re-forwarding: a connection that arrived on the failover ingress is never forwarded again (src/proxy_conn.rs, step 3d). Spill-over must keep this: only connections from clients can spill.

Proposed behavior

  1. Node load: a smoothed (a few seconds) CPU utilization of the local Harper threads, from the existing samples. Not connection count: MQTT and WebSocket connections are mostly idle, so count says little about CPU.
  2. Peer load: each node learns its peers' load. Proposed transport: the failover ingress already requires a PROXY v2 header from a trusted source; a PROXY v2 LOCAL command (the spec's health-check form, rejected today) could be answered with the node's load and closed. Polled by the existing probe task, at probeIntervalMs. This needs no new port or channel.
  3. Spill rule, per failover route, on client connections only: when local load is above a high-water mark and above the least-loaded ready peer's load by a margin, forward a fraction of new connections to that peer. The fraction rises in small steps while the condition holds and decays when it clears, capped at a maximum, so two nodes don't swing traffic back and forth. Drain and local-down keep their current behavior (forward everything).
  4. Do nothing when every node is busy: if no peer is below the margin, keep serving locally; relaying would add a hop without adding capacity.
  5. Observability: counters for spilled connections per route, a gauge for the current spill fraction, and the last-known load of each peer.

Tradeoffs

  • A spilled connection is relayed through the hot node for its whole life, so it pays an extra network hop. Across regions that adds latency and egress cost. Where GTM load feedback exists it remains the better steering, because the client connects to the other node directly; spill-over complements it for short spikes and replaces it where DNS has no load feedback.
  • Only new connections move. Long-lived connections stay where they landed.

Open design questions

  • Peer load transport: PROXY v2 LOCAL on the ingress (above), or something else.
  • Load scope: one figure per node, or per destination (routes can point at different worker sets).
  • Config surface on failover: defaults for high-water mark, margin, step and maximum fraction, and whether spill-over is opt-in per route.
  • Whether to limit spilling to peers below a connect-RTT ceiling, to avoid relaying across regions.
  • Whether the thread CPU samples read correctly from host-manager's pid/tid on a host where Harper runs in a container (PID namespaces).

Enabling it from host-manager is a separate follow-up.

— Claude Opus 5.5, drafted with Kris

Dominant language
TypeScript
Stars
1
Forks
0
Avg merge
11h 49m
Merged PRs (30d)
12

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from HarperFast/symphony

All issues in HarperFast/symphony

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.