hyper-util legacy client: an HTTP/1 request can hang forever when the connection closes while the request is being queued
Nessuno ha ancora preso questa issue.
Valutazione
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Idoneità per principianti
- 62/100
- Tipo di issue
- Bug
- Chiarezza
- Abbastanza chiara
- Stato di attività
- Attiva
- Stack tecnologico
- rust
- Ambito
- backend-api-design, networking
Direzione di ricerca
Start at hyper_util::client::legacy::Client::try_send_request and PoolClient::poll_ready, then inspect the HTTP/1 dispatcher close path described in the issue. Use the deterministic regression tests in the linked hyper-util patch tree as the starting test reference. Done means a dispatcher close releases the held sender and the request completes with the existing retry or cancellation behavior instead of hanging.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Summary
hyper_util::client::legacy::Client can leave an HTTP/1 request future pending
forever. This happens when the pooled connection's dispatcher closes (the peer
sends RST, or a FIN on an idle keep-alive connection) at the moment the request
is being enqueued. No error is returned and the request is not retried. The
caller sees a hang that ends only when its own timeout fires.
Versions: hyper 1.9.0, hyper-util 0.1.20, tokio 1.52.3. hyper and hyper-util
master have the same code.
Mechanism
Client::try_send_requestchecks out an HTTP/1Pooled<PoolClient>and
callspooled.try_send_request(req).await. It holdspooledacross the
await.hyper::client::dispatch::Sender::try_sendcalls tokio's
UnboundedSender::send, which works in two steps:
inc_num_messages()checks the closed bit and reserves a slot, then
chan.send()publishes the value.- Suppose the requester is preempted between those two steps while the
connection task sees the peer's reset.proto::h1::dispatch::Client::recv_msg(Err)
finds no callback. It callsrx.close(), and thenrx.try_recv(). tokio's
recvreturnsPendingwhile a send is in progress (the semaphore is not
idle), andtry_recvusesnow_or_never, so it returnsNone. The
dispatcher returnsErr, and hyper-util logs
client connection error: ... ConnectionReset. On a graceful FIN, the
dispatcher finishesOkinstead and drops its receiver. - Dropping
UnboundedReceiverdrains only published values. - The requester resumes and publishes into a channel nobody will read.
Envelope::dropwould fail the callback withCanceledand give back the
request. But it runs only when the channel is dropped, and the channel
stays alive whilepooledholds its only sender.pooledis held until the
response arrives, and the response never arrives.
tokio documents that a value can be sent after the receiver is dropped
(tokio-rs/tokio#7714, tokio-rs/tokio#8278).
Evidence
In CI, a server that accepts a connection and immediately resets it
(SO_LINGER=0) produces this trace:
hyper_util::client::legacy::client: connection is ready
hyper_util::client::legacy::pool: checkout dropped
<server accepts, resets>
hyper_util::client::legacy::client: client connection error: hyper::Error(Io, Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" })
hyper_util::client::legacy::client: sending connection error to error channel
<nothing until the caller's 5 s timeout>
The client connection error line appears only when the dispatcher had no
callback and try_recv found nothing. Every other ordering fails the request
in microseconds. The hang needs a preemption inside a window of a few
instructions, so it shows up only on loaded, many-threaded runners. We saw it
five times in eight days of CI across nine variants of that test, and never in
local repetition.
Proposed fix (hyper-util)
Do not keep holding the only HTTP/1 sender after its dispatcher has closed.
While awaiting the response, try_send_request also polls
PoolClient::poll_ready for HTTP/1 connections:
Err(thewanttaker was canceled, which happens onrx.close()and on
receiver drop): captureis_reused()andconn_info, then droppooled.
That drops the last sender. The channel destructor then drops any stranded
envelope, and hyper fails the callback withCanceledplus the request. The
existingRetryablepath then retries a reused connection or returns
Canceledfor a fresh one, which is exactly what happens when the dispatcher
does see the queued request.Ok(the dispatcher asked for more work after the request was published):
it saw the request, so stop watching.
The response future is polled first on every wake, so a delivered response or
error always wins. The change does not allocate and changes no API.
The patch we currently carry against hyper-util 0.1.20 is here:
https://github.com/ferrum-edge/ferrum-edge/blob/d1136ab60f0c74221bae49cc224b265bdcb24575/docs/upstream-hyper-util-patches/001-release-h1-sender-on-dispatch-close/hyper-util-release-h1-sender-on-dispatch-close.patch
(with deterministic regression tests in the same vendored tree). We are happy to open a PR if this direction looks right.
An alternative fix in hyper: after rx.close(), keep the dispatcher polling
rx until it returns Ready(None), and cancel anything that arrives. Do the
same before dropping the receiver on a graceful shutdown. That fixes every
SendRequest user, including code that drives client::conn::http1 directly
and holds its sender while waiting. But it keeps the connection task alive
after close until the preempted sender runs again.
- Lingua principale
- Rust
- Stelle
- 16.3k
- Fork
- 1.8k
- Merge medio
- 2g 20h
- PR unite (30g)
- 14
Guida per i contributori
Apri la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di hyperium/hyper
-
C-feature
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 65/100
-
C-feature
Difficoltà 5/5 Più di una settimana Idoneità per principianti 35/100
-
C-bug S-waiting-on-author
-
A-http2 C-bug
-
C-bug
Difficoltà 4/5 3-5 giorni Idoneità per principianti 48/100
Tutte le issue di hyperium/hyper
Issue simili
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 88/100
nautechsystems/nautilus_trader#5095 ·
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
stellar/rs-soroban-env#1739 ·
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 76/100
-
bug good first issue package: quic
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100