hyper-util legacy client: an HTTP/1 request can hang forever when the connection closes while the request is being queued
Maintainers usually reply within 4 days
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 62/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- rust
- Domain
- backend-api-design, networking
Research direction
Start at hyper_util::client::legacy::Client::try_send_request and PoolClient::poll_ready, then inspect the HTTP/1 dispatcher close path described in the issue. Use the deterministic regression tests in the linked hyper-util patch tree as the starting test reference. Done means a dispatcher close releases the held sender and the request completes with the existing retry or cancellation behavior instead of hanging.
Written by the indexing model from the issue text.
Description
Summary
hyper_util::client::legacy::Client can leave an HTTP/1 request future pending
forever. This happens when the pooled connection's dispatcher closes (the peer
sends RST, or a FIN on an idle keep-alive connection) at the moment the request
is being enqueued. No error is returned and the request is not retried. The
caller sees a hang that ends only when its own timeout fires.
Versions: hyper 1.9.0, hyper-util 0.1.20, tokio 1.52.3. hyper and hyper-util
master have the same code.
Mechanism
Client::try_send_requestchecks out an HTTP/1Pooled<PoolClient>and
callspooled.try_send_request(req).await. It holdspooledacross the
await.hyper::client::dispatch::Sender::try_sendcalls tokio's
UnboundedSender::send, which works in two steps:
inc_num_messages()checks the closed bit and reserves a slot, then
chan.send()publishes the value.- Suppose the requester is preempted between those two steps while the
connection task sees the peer's reset.proto::h1::dispatch::Client::recv_msg(Err)
finds no callback. It callsrx.close(), and thenrx.try_recv(). tokio's
recvreturnsPendingwhile a send is in progress (the semaphore is not
idle), andtry_recvusesnow_or_never, so it returnsNone. The
dispatcher returnsErr, and hyper-util logs
client connection error: ... ConnectionReset. On a graceful FIN, the
dispatcher finishesOkinstead and drops its receiver. - Dropping
UnboundedReceiverdrains only published values. - The requester resumes and publishes into a channel nobody will read.
Envelope::dropwould fail the callback withCanceledand give back the
request. But it runs only when the channel is dropped, and the channel
stays alive whilepooledholds its only sender.pooledis held until the
response arrives, and the response never arrives.
tokio documents that a value can be sent after the receiver is dropped
(tokio-rs/tokio#7714, tokio-rs/tokio#8278).
Evidence
In CI, a server that accepts a connection and immediately resets it
(SO_LINGER=0) produces this trace:
hyper_util::client::legacy::client: connection is ready
hyper_util::client::legacy::pool: checkout dropped
<server accepts, resets>
hyper_util::client::legacy::client: client connection error: hyper::Error(Io, Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" })
hyper_util::client::legacy::client: sending connection error to error channel
<nothing until the caller's 5 s timeout>
The client connection error line appears only when the dispatcher had no
callback and try_recv found nothing. Every other ordering fails the request
in microseconds. The hang needs a preemption inside a window of a few
instructions, so it shows up only on loaded, many-threaded runners. We saw it
five times in eight days of CI across nine variants of that test, and never in
local repetition.
Proposed fix (hyper-util)
Do not keep holding the only HTTP/1 sender after its dispatcher has closed.
While awaiting the response, try_send_request also polls
PoolClient::poll_ready for HTTP/1 connections:
Err(thewanttaker was canceled, which happens onrx.close()and on
receiver drop): captureis_reused()andconn_info, then droppooled.
That drops the last sender. The channel destructor then drops any stranded
envelope, and hyper fails the callback withCanceledplus the request. The
existingRetryablepath then retries a reused connection or returns
Canceledfor a fresh one, which is exactly what happens when the dispatcher
does see the queued request.Ok(the dispatcher asked for more work after the request was published):
it saw the request, so stop watching.
The response future is polled first on every wake, so a delivered response or
error always wins. The change does not allocate and changes no API.
The patch we currently carry against hyper-util 0.1.20 is here:
https://github.com/ferrum-edge/ferrum-edge/blob/d1136ab60f0c74221bae49cc224b265bdcb24575/docs/upstream-hyper-util-patches/001-release-h1-sender-on-dispatch-close/hyper-util-release-h1-sender-on-dispatch-close.patch
(with deterministic regression tests in the same vendored tree). We are happy to open a PR if this direction looks right.
An alternative fix in hyper: after rx.close(), keep the dispatcher polling
rx until it returns Ready(None), and cancel anything that arrives. Do the
same before dropping the receiver on a graceful shutdown. That fixes every
SendRequest user, including code that drives client::conn::http1 directly
and holds its sender while waiting. But it keeps the connection task alive
after close until the preempted sender runs again.
- Dominant language
- Rust
- Stars
- 16.3k
- Forks
- 1.8k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 11
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from hyperium/hyper
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
hyperium/hyper#4206 · 2 comments ·
Maintainers usually reply within 4 days
-
C-feature
Difficulty 1/5 Under an hour Newbie friendliness 65/100
hyperium/hyper#2652 · 4 reactions ·
Maintainers usually reply within 4 days
-
Difficulty 4/5 3-5 days Newbie friendliness 55/100
Maintainers usually reply within 4 days
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
Maintainers usually reply within 4 days
-
C-feature
Difficulty 5/5 Over a week Newbie friendliness 35/100
hyperium/hyper#4186 · 1 comment ·
Maintainers usually reply within 4 days
Similar issues
-
backend::vllm diffusion multimodal
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
lambdaclass/ethrex#7329 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 88/100
shadowsocks/shadowsocks-rust#2186 · 1 comment ·
-
C-bug S-awaiting-triage
Difficulty 1/5 Under an hour Newbie friendliness 92/100
juspay/hyperswitch#14479 ·
Maintainers usually reply within 1 day