`Change graceful_shutdown function behavior` PR can cause tonic servers to hang `serve_with_incoming_shutdown` in two different cases
@seanmonstar is already working on this.
Since Aug 27, 2026.
Assessment
This issue has not been assessed yet.
Description
Version
Foudn in hyper 1.5.1 and later
Platform
25.5.0 (macOS), though it is also applying on our linux bots
Summary
At OneSignal we have a number of tonic gRPC services we maintain. After a recent update of dependencies on one of them, we noticed that it "randomly" would hang during shutdown for an hour or more.
After a week of chasing the wild goose, we have isolated it to a single PR introduced in hyper 1.5.1:
https://github.com/hyperium/hyper/pull/3729
In these cases a tonic server:
Server::builder()
.add_service(GreeterServer::new(MyGreeter::default()))
.serve_with_incoming_shutdown(incoming, async move {
shutdown_grpc_token.cancelled().await
})
.await
.unwrap();
would never return and block for an hour or more after the shutdown_grpc_token CancellationToken was canceled.
We have a primary example non_upgrade_connections_can_block_shutdown and a secondary one we found along the way client_not_responding_example in this self contained example:
https://github.com/chamons/hyper_hung_client_example
With a patch.crates-io line, we can resolve this branch with the PR reverted and both of these example begin acting as expected.
Expected Behavior
It is not expected that passive tonic gRPC connections to a server that are unused would block shutdown. Before 1.5.1 they did not.
Actual Behavior
We block until the upstream instance restarts or otherwise drops all of its connections.
Additional Context
No response
- Dominant language
- Rust
- Stars
- 16.3k
- Forks
- 1.8k
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 13
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from hyperium/hyper
-
C-feature
Difficulty 1/5 Under an hour Newbie friendliness 65/100
-
C-feature
Difficulty 5/5 Over a week Newbie friendliness 35/100
-
C-bug S-waiting-on-author
-
C-bug
Difficulty 4/5 3-5 days Newbie friendliness 48/100
-
A-client C-bug E-pr-welcome K-hyper-util
Difficulty 4/5 3-5 days Newbie friendliness 48/100
Similar issues
-
bug github_actions
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
registrystack/registry-stack#1393 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
longbridge/gpui-kit#3223 ·
-
bug engine
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
rocky-data/rocky#2181 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
oasisprotocol/oasis-sdk#2523 ·
-
[indexer] [QA] Add a focused test for the new NonRetryableError / assertSocketAlive() behavior. Openbot:ai-assisted component:indexer QA-roadmap status:untriaged
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
midnightntwrk/midnight-indexer#1557 ·