Spill a share of new connections to a less-loaded peer when local Harper threads are busy
Maintainers usually reply within 1 day
@kriszyp is already working on this.
Since Oct 29, 2026.
Assessment
This issue has not been assessed yet.
Description
Why
Symphony already balances each new connection across the local Harper worker threads by load, and since Peer failover (HarperFast/symphony#71) it forwards a route's new connections to peer nodes while the local upstream is down or drained. It does not move load between nodes while both are up.
Today, load between nodes is balanced only at the DNS layer: Akamai GTM polls each node's /current_load_for_gtm (HarperFast/akamai-status) every 30 s and shifts DNS answers. DNS setups we expect to use for some clusters have no load feedback at all: latency or geoproximity records without health checks (for example Route 53 or Google Cloud DNS). On those, a hot node keeps receiving its full share of traffic no matter how busy it is. Even with GTM, the reaction time is the 30 s poll plus DNS TTL plus client DNS caching.
Symphony can react in seconds and needs no DNS support: when this node's Harper threads are much busier than a peer's, forward a share of new connections to that peer through the existing failover path.
What exists to build on
- Local load signal:
src/balancer.rssamples each UDS slot's Harper thread CPU from/proc/{pid}/task/{tid}/statevery 250 ms (cpu_util_permille). host-manager passespid/tidfor every Harper worker socket. - Forwarding:
failover::forward_to_peerssends the peeked ClientHello plus a PROXY v2 header to a peer's failover ingress listener; the peer terminates TLS. The forwarding node only copies bytes after that, so a spilled connection costs it very little. - No re-forwarding: a connection that arrived on the failover ingress is never forwarded again (
src/proxy_conn.rs, step 3d). Spill-over must keep this: only connections from clients can spill.
Proposed behavior
- Node load: a smoothed (a few seconds) CPU utilization of the local Harper threads, from the existing samples. Not connection count: MQTT and WebSocket connections are mostly idle, so count says little about CPU.
- Peer load: each node learns its peers' load. Proposed transport: the failover ingress already requires a PROXY v2 header from a trusted source; a PROXY v2
LOCALcommand (the spec's health-check form, rejected today) could be answered with the node's load and closed. Polled by the existing probe task, atprobeIntervalMs. This needs no new port or channel. - Spill rule, per failover route, on client connections only: when local load is above a high-water mark and above the least-loaded ready peer's load by a margin, forward a fraction of new connections to that peer. The fraction rises in small steps while the condition holds and decays when it clears, capped at a maximum, so two nodes don't swing traffic back and forth. Drain and local-down keep their current behavior (forward everything).
- Do nothing when every node is busy: if no peer is below the margin, keep serving locally; relaying would add a hop without adding capacity.
- Observability: counters for spilled connections per route, a gauge for the current spill fraction, and the last-known load of each peer.
Tradeoffs
- A spilled connection is relayed through the hot node for its whole life, so it pays an extra network hop. Across regions that adds latency and egress cost. Where GTM load feedback exists it remains the better steering, because the client connects to the other node directly; spill-over complements it for short spikes and replaces it where DNS has no load feedback.
- Only new connections move. Long-lived connections stay where they landed.
Open design questions
- Peer load transport: PROXY v2
LOCALon the ingress (above), or something else. - Load scope: one figure per node, or per destination (routes can point at different worker sets).
- Config surface on
failover: defaults for high-water mark, margin, step and maximum fraction, and whether spill-over is opt-in per route. - Whether to limit spilling to peers below a connect-RTT ceiling, to avoid relaying across regions.
- Whether the thread CPU samples read correctly from host-manager's
pid/tidon a host where Harper runs in a container (PID namespaces).
Enabling it from host-manager is a separate follow-up.
— Claude Opus 5.5, drafted with Kris
- Dominant language
- TypeScript
- Stars
- 1
- Forks
- 0
- Avg merge
- 11h 49m
- Merged PRs (30d)
- 12
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from HarperFast/symphony
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
HarperFast/symphony#91 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
HarperFast/symphony#89 ·
Maintainers usually reply within 1 day
-
Mixed-case SNI forwarded to an older peer during a rolling upgrade can reach its wildcard routeOpen
Difficulty 4/5 3-5 days Newbie friendliness 49/100
HarperFast/symphony#93 ·
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 42/100
HarperFast/symphony#92 ·
Maintainers usually reply within 1 day
-
Difficulty 3/5 1-2 days Newbie friendliness 72/100
HarperFast/symphony#90 ·
Maintainers usually reply within 1 day
All issues in HarperFast/symphony
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Maintainers usually reply within 1 day
-
feature request needs triage
Difficulty 1/5 1-3 hours Newbie friendliness 69/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
solidjs/solid-docs#1716 ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 80/100
thecodacus/pithagoras#380 ·
Maintainers usually reply within 1 day