hyper-util legacy client: an HTTP/1 request can hang forever when the connection closes while the request is being queued
メンテナーはふだん 3 日以内に返信
まだ誰も着手していません。
評価
- 難易度
- 4/5
- 見積もり時間
- 3〜5日
- 初心者へのやさしさ
- 62/100
- issue の種類
- バグ
- 明瞭さ
- おおむね明確
- 活発さ
- 活発
- 技術スタック
- rust
調査の方向性
Start at hyper_util::client::legacy::Client::try_send_request and PoolClient::poll_ready, then inspect the HTTP/1 dispatcher close path described in the issue. Use the deterministic regression tests in the linked hyper-util patch tree as the starting test reference. Done means a dispatcher close releases the held sender and the request completes with the existing retry or cancellation behavior instead of hanging.
索引モデルが issue の本文から書いたものです。
説明
Summary
hyper_util::client::legacy::Client can leave an HTTP/1 request future pending
forever. This happens when the pooled connection's dispatcher closes (the peer
sends RST, or a FIN on an idle keep-alive connection) at the moment the request
is being enqueued. No error is returned and the request is not retried. The
caller sees a hang that ends only when its own timeout fires.
Versions: hyper 1.9.0, hyper-util 0.1.20, tokio 1.52.3. hyper and hyper-util
master have the same code.
Mechanism
Client::try_send_requestchecks out an HTTP/1Pooled<PoolClient>and
callspooled.try_send_request(req).await. It holdspooledacross the
await.hyper::client::dispatch::Sender::try_sendcalls tokio's
UnboundedSender::send, which works in two steps:
inc_num_messages()checks the closed bit and reserves a slot, then
chan.send()publishes the value.- Suppose the requester is preempted between those two steps while the
connection task sees the peer's reset.proto::h1::dispatch::Client::recv_msg(Err)
finds no callback. It callsrx.close(), and thenrx.try_recv(). tokio's
recvreturnsPendingwhile a send is in progress (the semaphore is not
idle), andtry_recvusesnow_or_never, so it returnsNone. The
dispatcher returnsErr, and hyper-util logs
client connection error: ... ConnectionReset. On a graceful FIN, the
dispatcher finishesOkinstead and drops its receiver. - Dropping
UnboundedReceiverdrains only published values. - The requester resumes and publishes into a channel nobody will read.
Envelope::dropwould fail the callback withCanceledand give back the
request. But it runs only when the channel is dropped, and the channel
stays alive whilepooledholds its only sender.pooledis held until the
response arrives, and the response never arrives.
tokio documents that a value can be sent after the receiver is dropped
(tokio-rs/tokio#7714, tokio-rs/tokio#8278).
Evidence
In CI, a server that accepts a connection and immediately resets it
(SO_LINGER=0) produces this trace:
hyper_util::client::legacy::client: connection is ready
hyper_util::client::legacy::pool: checkout dropped
<server accepts, resets>
hyper_util::client::legacy::client: client connection error: hyper::Error(Io, Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" })
hyper_util::client::legacy::client: sending connection error to error channel
<nothing until the caller's 5 s timeout>
The client connection error line appears only when the dispatcher had no
callback and try_recv found nothing. Every other ordering fails the request
in microseconds. The hang needs a preemption inside a window of a few
instructions, so it shows up only on loaded, many-threaded runners. We saw it
five times in eight days of CI across nine variants of that test, and never in
local repetition.
Proposed fix (hyper-util)
Do not keep holding the only HTTP/1 sender after its dispatcher has closed.
While awaiting the response, try_send_request also polls
PoolClient::poll_ready for HTTP/1 connections:
Err(thewanttaker was canceled, which happens onrx.close()and on
receiver drop): captureis_reused()andconn_info, then droppooled.
That drops the last sender. The channel destructor then drops any stranded
envelope, and hyper fails the callback withCanceledplus the request. The
existingRetryablepath then retries a reused connection or returns
Canceledfor a fresh one, which is exactly what happens when the dispatcher
does see the queued request.Ok(the dispatcher asked for more work after the request was published):
it saw the request, so stop watching.
The response future is polled first on every wake, so a delivered response or
error always wins. The change does not allocate and changes no API.
The patch we currently carry against hyper-util 0.1.20 is here:
https://github.com/ferrum-edge/ferrum-edge/blob/d1136ab60f0c74221bae49cc224b265bdcb24575/docs/upstream-hyper-util-patches/001-release-h1-sender-on-dispatch-close/hyper-util-release-h1-sender-on-dispatch-close.patch
(with deterministic regression tests in the same vendored tree). We are happy to open a PR if this direction looks right.
An alternative fix in hyper: after rx.close(), keep the dispatcher polling
rx until it returns Ready(None), and cancel anything that arrives. Do the
same before dropping the receiver on a graceful shutdown. That fixes every
SendRequest user, including code that drives client::conn::http1 directly
and holds its sender while waiting. But it keeps the connection task alive
after close until the preempted sender runs again.
- 主要言語
- Rust
- スター
- 16.3k
- フォーク
- 1.8k
- 平均マージ
- 2日 5時間
- マージ済み PR(30日)
- 12
環境構築
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
hyperium/hyper のほかの issue
-
難易度 2/5 1〜3時間 初心者へのやさしさ 88/100
メンテナーはふだん 3 日以内に返信
-
C-feature
難易度 1/5 1時間未満 初心者へのやさしさ 65/100
hyperium/hyper#2652 · リアクション 4 件 ·
メンテナーはふだん 3 日以内に返信
-
C-feature
難易度 5/5 1週間以上 初心者へのやさしさ 35/100
hyperium/hyper#4186 · コメント 1 件 ·
メンテナーはふだん 3 日以内に返信
-
HTTP/1 client connection is never closed or pooled when a response completes while the request body is still unsent ((Reading::KeepAlive, Writing::Body) is a terminal state)対応中かも @BlackRabbitCoder が 13 日前に担当しました。 オープンC-bug S-waiting-on-author
hyperium/hyper#4176 · コメント 4 件 · 担当者 1 名 ·
メンテナーはふだん 3 日以内に返信
-
`Change graceful_shutdown function behavior` PR can cause tonic servers to hang `serve_with_incoming_shutdown` in two different cases再び着手できるかも @seanmonstar が 30 日前に担当しましたが、オープン中のプルリクエストはありません。 オープンA-http2 C-bug
hyperium/hyper#4170 · コメント 4 件 · 担当者 1 名 ·
メンテナーはふだん 3 日以内に返信
似ている issue
-
area:casework bug criticality:p3 triage:needs-implementation
難易度 2/5 1〜3時間 初心者へのやさしさ 84/100
registrystack/registry-stack#1623 ·
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 72/100
DioxusLabs/anyrender#98 ·
-
難易度 2/5 1〜3時間 初心者へのやさしさ 76/100
leptos-rs/leptos#4885 · コメント 1 件 ·
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 82/100
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
longbridge/gpui-kit#3276 ·
メンテナーはふだん 1 日以内に返信