Incorrectly strips trailing dot from absolute hostnames, causing TooManyRedirects

オープン
  • 何か問題がありますか？
#6,131 コメント 1 件 リアクション 0 件 担当者 0 名 GitHub で見る

まだ誰も着手していません。

評価

難易度
3/5
見積もり時間
1〜2日
初心者へのやさしさ
38/100
issue の種類
バグ
明瞭さ
おおむね明確
活発さ
停滞
技術スタック
python
領域
api, networking

調査の方向性

requests/api.py と requests/sessions.py から始め、特に traceback に示されている request と redirect のパスを確認し、その後 urllib3 が TLS SNI と Host ヘッダーをどのように構築するかを追ってください。requests.get("https://pyropus.ca./") で問題を再現し、リクエストがリダイレクトループなしで完了する一方、Host ヘッダーが末尾のドットを保持することを確認してください。

索引モデルが issue の本文から書いたものです。

説明

requests appears to be incorrectly stripping the trailing dot on absolute hostnames (i.e. explicitly marked as in the root DNS namespace) in URLs when constructing the Host: request header, like pyropus.ca.. This causes redirect loops when the site chooses the absolute hostname as canonical and redirects requests without the trailing dot.

The example site redirects requests with a host header of Host: pyropus.ca to the same URL with the host name changed to the canonical absolute version:

HTTP/1.1 301 Moved Permanently
[...]
Location: https://pyropus.ca./

requests then strips the dot and resends the request, resulting in a redirect loop.

The trailing dot needs to be stripped from the SNI header for https requests (required by TLS/etc spec) but it should not be stripped from the Host: header value.

When fixing the SNI trailing dot issue, curl had this bug because they changed it to also affect the Host: header value, but they've fixed it by reverting that part of the change:
https://github.com/curl/curl/issues/8290

Many other common user agents that I've tested handle this properly:

  • all GUI browers, to my knowledge, preserve the absolute domain name in the Host: header - I've tested Firefox, Chromium, Vivaldi, Konqueror, and Safari
  • wget also preserves the absolute domain name on redirect URLs
  • curl is now fixed and handles it

A few less common ones have the same buggy behaviour, or did -- I haven't re-checked:

  • lynx, links, and elinks handle it like curl, erroring out on a redirect loop

Expected Result

Successful 200 response with content.

Actual Result

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File ".../test/.venv/lib/python3.9/site-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File ".../test/.venv/lib/python3.9/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File ".../test/.venv/lib/python3.9/site-packages/requests/sessions.py", line 529, in request
    resp = self.send(prep, **send_kwargs)
  File ".../test/.venv/lib/python3.9/site-packages/requests/sessions.py", line 667, in send
    history = [resp for resp in gen]
  File ".../test/.venv/lib/python3.9/site-packages/requests/sessions.py", line 667, in <listcomp>
    history = [resp for resp in gen]
  File ".../test/.venv/lib/python3.9/site-packages/requests/sessions.py", line 166, in resolve_redirects
    raise TooManyRedirects('Exceeded {} redirects.'.format(self.max_redirects), response=resp)
requests.exceptions.TooManyRedirects: Exceeded 30 redirects.

Reproduction Steps

import requests

requests.get("https://pyropus.ca./")

System Information

$ python -m requests.help

{
  "chardet": {
    "version": null
  },
  "charset_normalizer": {
    "version": "2.0.12"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "3.3"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.9.2"
  },
  "platform": {
    "release": "5.17.5",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.27.1"
  },
  "system_ssl": {
    "version": "101010bf"
  },
  "urllib3": {
    "version": "1.26.9"
  },
  "using_charset_normalizer": true,
  "using_pyopenssl": false
}

Edit: added a couple words to the summary to clarify.

主要言語
Python
スター
54.7k
フォーク
10.8k
平均マージ
18分
マージ済み PR（30日）
1

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

psf/requests のほかの issue

psf/requests の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。