Let callers bound or skip the intermediate bodies read while following redirects and auth flows

Open
  • Is something wrong?
#1,249 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

@rkuce is already working on this.

Since Oct 7, 2026.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
42/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
python
Domain
backend-api-design, networking

Research direction

Start in _client.py at _send_handling_redirects and _send_handling_auth, including the async implementations at the referenced locations, and reproduce the redirect plus DigestAuth flow with MockTransport. Compare the proposed handling of intermediate response bodies, history, and auth flows; done should establish an agreed opt-in or general limit and cover the behavior without breaking existing defaults.

Written by the indexing model from the issue text.

Description

Raised as an issue directly: the template points to "Ideas" and "Potential Issue" discussion categories, but this repo's discussions only have Announcements and Q&A.

When Client.send() follows redirects or runs a multi-step auth flow, it reads every intermediate response body in full before moving on. This happens even with stream=True:

  • _send_handling_redirects: response.read() on each redirect when follow_redirects=True (_client.py:1056, async :1895 in 2.12.0)
  • _send_handling_auth: response.read() on each response the auth flow answers, such as a DigestAuth 401 challenge or a token-refresh retry (_client.py:1020, async :1858)

Most callers never look at those bodies. But a client that talks to untrusted or misbehaving servers, and bounds memory by streaming and capping the final body, can't bound these reads. A server can send a huge body on a 3xx or 401 and the client buffers all of it before the caller sees anything.

Reproduction (httpx2 2.12.0):

import httpx2

pulled = 0


def huge_body():
    global pulled
    for _ in range(1024):
        pulled += 1024
        yield b"x" * 1024


def handler(request: httpx2.Request) -> httpx2.Response:
    if request.url.path == "/redirect":
        return httpx2.Response(302, headers={"location": "/digest"}, content=huge_body())
    if not request.headers.get("authorization", "").startswith("Digest "):
        challenge = 'Digest realm="api", nonce="abc", qop="auth"'
        return httpx2.Response(401, headers={"www-authenticate": challenge}, content=huge_body())
    return httpx2.Response(200, content=b"ok")


with httpx2.Client(
    transport=httpx2.MockTransport(handler),
    auth=httpx2.DigestAuth("user", "pass"),
    follow_redirects=True,
) as client:
    with client.stream("GET", "https://example.test/redirect") as response:
        print(response.status_code, f"{pulled // 1024} KiB of intermediate bodies read before the stream opened")

Output: 200 3072 KiB of intermediate bodies read before the stream opened. That is the 302, the 401, and the 302 again when DigestAuth re-sends the original request.

The only workaround today is to re-implement both loops outside the client: send with follow_redirects=False and a no-op auth, follow response.next_request, and drive auth.sync_auth_flow() by hand. httpware, a client library built on httpx2, does this to enforce its response body cap. It handles redirects in https://github.com/modern-python/httpware/pull/152 and auth flows in https://github.com/modern-python/httpware/pull/153. That works with public API, but it means copying redirect counting, history handling and the private _build_request_auth fallback to URL credentials. Those copies can drift from httpx2.

Possible directions, in increasing scope:

  1. Close intermediate bodies unread when nothing needs them. Redirect bodies are never used by httpx2. Auth-step bodies are only needed when auth.requires_response_body is set. The catch: response.history[i].content would raise ResponseNotRead, so this probably has to be opt-in.
  2. An opt-in client option for (1), keeping today's default.
  3. A general response-size limit on the client, applied to these reads and to read(). This is related to encode/httpx#1392 (#521 here), which asked for an easy way to bound body size.

Would any of these be welcome?

Dominant language
Python
Stars
1.6k
Forks
103
Avg merge
1d 19h
Merged PRs (30d)
25

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from pydantic/httpx2

All issues in pydantic/httpx2

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.