UnicodeDecodeError after following a chain of redirects
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 42/100
Research direction
Start with requests/sessions.py at get_redirect_target and requests_internal_utils.py at to_native_string, then run the provided requests.get reproduction URL. Trace how the redirect Location bytes are decoded and determine the expected handling for the shown non-UTF-8 byte. Done means the reproduction no longer raises UnicodeDecodeError and the redirect is handled consistently.
Written by the indexing model from the issue text.
Description
#6006
Something confuses requests (or urllib3?) along the way
Actual Result
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf1 in position 92: invalid continuation byte
Reproduction Steps
import requests
requests.get("https://www.lavozdegalicia.es/noticia/deportes/2021/12/13/psg-juve-united-nuevos-rivales-espa%C3%B1oles-champions/00031639396272418389372.htm")
System Information
$ python -m requests.help
{
"chardet": {
"version": "4.0.0"
},
"charset_normalizer": {
"version": "2.0.9"
},
"cryptography": {
"version": "36.0.0"
},
"idna": {
"version": "3.3"
},
"implementation": {
"name": "CPython",
"version": "3.8.10"
},
"platform": {
"release": "4.4.0-17763-Microsoft",
"system": "Linux"
},
"pyOpenSSL": {
"openssl_version": "101010cf",
"version": "21.0.0"
},
"requests": {
"version": "2.26.0"
},
"system_ssl": {
"version": "1010106f"
},
"urllib3": {
"version": "1.26.7"
},
"using_charset_normalizer": false,
"using_pyopenssl": true
}
@sigmavirus24 you have been too harsh on this one.
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\Ahmed\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\api.py", line 76, in get
return request('get', url, params=params, **kwargs)
File "C:\Users\Ahmed\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Users\Ahmed\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "C:\Users\Ahmed\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\sessions.py", line 677, in send
history = [resp for resp in gen]
File "C:\Users\Ahmed\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\sessions.py", line 677, in
history = [resp for resp in gen]
File "C:\Users\Ahmed\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\sessions.py", line 150, in resolve_redirects
url = self.get_redirect_target(resp)
File "C:\Users\Ahmed\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\sessions.py", line 116, in get_redirect_target
return to_native_string(location, 'utf8')
File "C:\Users\Ahmed\AppData\Local\Programs\Python\Python37\lib\site-packages\requests_internal_utils.py", line 25, in to_native_string
out = string.decode(encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf1 in position 92: invalid continuation byte
The guilty part there is return to_native_string(location, 'utf8') Trying to decode the URL to native utf8 when it should rather "URL encode" it. I am not an HTTP expert, but this exception should be handled more gracefully anyway.
The location given by the remote for redirection is as follow:
Should the \xf1 be decoded as follow %F1 ?
I can see that this behavior is already followed by Chrome.
- Dominant language
- Python
- Stars
- 54.7k
- Forks
- 10.8k
- Avg merge
- 18m
- Merged PRs (30d)
- 1
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from psf/requests
-
Difficulty 1/5 Under an hour Newbie friendliness 85/100
-
`iter_lines(decode_unicode=True, delimiter=<str>)` raises `TypeError` when `Response.encoding` is `None`May be free again A pull request for this issue was closed without being merged. Open
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
-
get_encoding_from_headers treats any content type containing 'text' as text/* (ISO-8859-1)Possibly taken @suyashgitt claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Invalid `params` type (e.g. int) raises misleading "TypeError: Cannot mix str and non-str arguments"Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Similar issues
-
plot_graph crashes when a mesh node has no m2m edgePossibly taken @tygvi claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
mllam/neural-lam#883 · 1 comment ·
Maintainers usually reply within 2 days
-
enhancement possible-duplicate
Difficulty 1/5 1-3 hours Newbie friendliness 78/100
AOSSIE-Org/PictoPy#1715 · 2 comments ·
Maintainers usually reply within 6 days
-
SQLite connector omits generated columns from the catalog, and drops relationships sourced on themOpen
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
NVIDIA/auto-ontology#511 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
ethereum/execution-specs#3996 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
cpp-best-practices/project_validator#267 ·
Maintainers usually reply within 1 day