Connection hangs/does not timeout after 350 seconds.

Open
  • Is something wrong?
#798 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Quiet
Tech stack
python
Domain
backend, networking

Research direction

Start by reproducing the AsyncClient HTTP/2 request through an AWS VPC NAT gateway, using the timeout and periodic healthcheck details in the report. Investigate how the connection handles the NAT gateway's RST after 350 seconds; done means the behavior is addressed by timing out at the appropriate point or maintaining the connection with keepalive traffic.

Written by the indexing model from the issue text.

Description

Originally opened by @mittalsuraj18 on 2023-12-04 06:24:56 in encode/httpx

Discussed in https://github.com/encode/httpx/discussions/2968

Originally posted by mittalsuraj18 December 1, 2023
When an application is running on an AWS VPC and using NAT gateway to communicate, there is a specific behavior in NAT gateway which is described here https://docs.aws.amazon.com/vpc/latest/userguide/nat-gateway-troubleshooting.html as such

Internet connection drops after 350 seconds
Problem

Your instances can access the internet, but the connection drops after 350 seconds.
Cause

If a connection that's using a NAT gateway is idle for 350 seconds or more, the connection times out.

When a connection times out, a NAT gateway returns an RST packet to any resources behind the NAT gateway that attempt to continue the connection (it does not send a FIN packet).
Solution

To prevent the connection from being dropped, you can initiate more traffic over the connection. Alternatively, you can enable TCP keepalive on the instance with a value less than 350 seconds.

For example, if i have a single connection. and do something like,

client = AsyncClient(base_url=BASE_URL, http2=True)
_ = await client.get("/", timeout=1200)

The connection does not respond to RST packet. And times out after 1200 seconds instead of 350 seconds.

The behavior should have been either of the following

  1. Timeout at 350 seconds
  2. Periodically send keepalive packet to make sure NAT gateway does not timeout.

How i currently have solved it is by doing a periodic call every 60 seconds, which just calls a healtcheck endpoint of the same baseurl, to use the same connection.

Dominant language
Python
Stars
1.6k
Forks
103
Avg merge
1d 19h
Merged PRs (30d)
25

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from pydantic/httpx2

All issues in pydantic/httpx2

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.