Don't route to a restarted upstream until it serves, not just accepts

Đang mở
  • Có gì không ổn?
#84 0 bình luận 0 reaction 1 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

@kriszyp đang làm issue này rồi.

Từ ngày 5/11/2026.

Đánh giá

Issue này chưa được đánh giá.

Mô tả

When a local Harper restarts, symphony marks a failover route up again about 1 s before Harper serves on it. Every connection in that window is dropped.

Observed (symphony 1.6.0, two-node Fabric stage cluster, Harper 5.3.5)

The route is 443, TLS-terminating, with a uds upstream (<instance>-sockets/0-9926.sock). The local container was stopped and then started again, with a new 443 connection every 0.25 s:

After docker start New 443 connections
0–3.1 s forwarded to the peer (localUp: false)
3.1 s the probe clears the mark (localUp: true, refusingSlots: 0)
3.3–4.1 s 4 of 4 dropped, 4–7 ms each, counted as route error stream
4.4 s onward served locally

The upstream accepted the connection and then closed it without a response. Every restart of a node with failover therefore loses about 1 s of new connections on each route, the same exposure #79 addressed for the stop side.

Cause (partly confirmed)

failover::probe (src/failover.rs) is a connect-and-close. Its own comment says it "proves the socket accepts, not that the application behind it is ready", and a successful probe clears the slot. Harper's socket accepts about 1 s before Harper serves on it.

Not yet known: what accepts during that window, and whether Harper binds its per-thread sockets before its HTTP handler is ready. The socket files' mtimes in that run were later than the first drops, which suggests an earlier bind or a rebind.

What to settle

  1. Where readiness belongs. If Harper binds or accepts on these sockets before it can serve, the root fix is in Harper: bind only when ready. symphony can't recover a TLS-terminated connection after its handshake, so a symphony-only fix can't be complete there. If Harper is correct and something else accepts, find it.
  2. What symphony should do either way. For example: require the probe to see the socket accepting continuously for a hold-down period before clearing a slot that was marked down, or have the probe wait for a protocol-level signal. Harper writes a .yaml beside each socket; check whether it carries a readiness signal.

Tests

  • symphony: a UDS upstream that accepts and immediately closes for its first N connections after coming back must not have new connections routed to it during that period, or they must be retried at a peer.
  • End to end on a two-node setup: restart one node's Harper under steady connection load, and no connection fails.

Found during Fabric stage failover testing. — Claude Opus 5.5, drafted with Kris

Ngôn ngữ chính
Rust
Star
1
Fork
0
Merge trung bình
19 giờ 12 phút
Pull request đã merge (30 ngày)
13

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của HarperFast/symphony

Tất cả issue của HarperFast/symphony

Issue tương tự

Thêm issue về Rust

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.