PreparedRequests can't bypass URL normalization when proxies are used

Open
  • Is something wrong?
#6,830 3 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
48/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Quiet
Tech stack
python
Domain
networking

Research direction

Start at requests.Session.send and the PreparedRequest path used when proxies are supplied; compare it with the direct send path in the reproduction. Trace where prep.url is transformed before the proxy request, then add a regression test covering /../something.txt through a proxy and confirm the proxy sees the unnormalized URL.

Written by the indexing model from the issue text.

Description

Related to #5289, where akmalhisyam found a way to bypass URL normalization using PreparedRequests, however, the solution doesn't work when you have proxies provided.

Expected Result

This should be able to explicitly set the request URL without getting normalized (from /../something.txt to /something.txt)

url = "http://example.com/../something.txt"
s = requests.Session()
req = requests.Request(method='POST' ,url=url, headers=headers, data=data)
prep = req.prepare()
prep.url = url
r = s.send(prep, proxies={"http": "http://127.0.0.1"}, verify=False)

Actual Result

The code above doesn't work, this one works though:

url = "http://example.com/../something.txt"
s = requests.Session()
req = requests.Request(method='POST' ,url=url, headers=headers, data=data)
prep = req.prepare()
prep.url = url
r = s.send(prep, verify=False)

Reproduction Steps

Use the code in Expected Result and check your proxy request log, you will see it doesn't work

System Information

$ python -m requests.help

{
  "chardet": {
    "version": "5.2.0"
  },
  "charset_normalizer": {
    "version": "2.0.12"
  },
  "cryptography": {
    "version": "38.0.4"
  },
  "idna": {
    "version": "3.4"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.11.4"
  },
  "platform": {
    "release": "4.4.0-19041-Microsoft",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "30000080",
    "version": "21.0.0"
  },
  "requests": {
    "version": "2.32.3"
  },
  "system_ssl": {
    "version": "30000030"
  },
  "urllib3": {
    "version": "2.0.4"
  },
  "using_charset_normalizer": false,
  "using_pyopenssl": true
}
Dominant language
Python
Stars
54.7k
Forks
10.8k
Avg merge
18m
Merged PRs (30d)
1

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from psf/requests

All issues in psf/requests

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.