Fetching the document database often times-out

devdocs2zim/client.py の _get_text()、read_doc_file()、get_db() から始め、次に generator.py からのそれらの使用箇所を追跡します。documents.devdocs.io に対して文書化された読み取りタイムアウトを再現し、連続するタスクがどのように処理されるかを調べます。一時的なドキュメントデータベースの取得タイムアウトによって生成が予期せず失敗しなくなれば完了です。関連するテストまたは生成の繰り返し実行で確認してください。

See e.g. https://farm.openzim.org/pipeline/761e418d-483b-4909-b6f7-a92c9fdccca3/debug, but we have ten to twenty occurrences every quarter.

Restarting the recipe is enough to solve the problem, sometimes twice.

[devdocs2zim::2026-06-02 15:16:43,323] INFO:  Fetching the index...
[devdocs2zim::2026-06-02 15:16:43,825] INFO:  Fetching the document database...
[devdocs2zim::2026-06-02 15:16:58,864] ERROR:HTTPSConnectionPool(host='documents.devdocs.io', port=443): Read timed out. (read timeout=15)
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 534, in _make_request
    response = conn.getresponse()
               ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/urllib3/connection.py", line 571, in getresponse
    httplib_response = super().getresponse()
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/http/client.py", line 1430, in getresponse
    response.begin()
  File "/usr/local/lib/python3.12/http/client.py", line 331, in begin
    version, status, reason = self._read_status()
                              ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/http/client.py", line 292, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/socket.py", line 720, in readinto
    return self._sock.recv_into(b)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/ssl.py", line 1251, in recv_into
    return self.read(nbytes, buffer)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/ssl.py", line 1103, in read
    return self._sslobj.read(len, buffer)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TimeoutError: The read operation timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 841, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 490, in increment
    raise reraise(type(error), error, _stacktrace)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/urllib3/util/util.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 536, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 367, in _raise_timeout
    raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='documents.devdocs.io', port=443): Read timed out. (read timeout=15)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/devdocs2zim/entrypoint.py", line 104, in main
    ).run()
      ^^^^^
  File "/usr/local/lib/python3.12/site-packages/devdocs2zim/generator.py", line 454, in run
    self.generate_zim(
  File "/usr/local/lib/python3.12/site-packages/devdocs2zim/generator.py", line 506, in generate_zim
    db = self.devdocs_client.get_db(doc_metadata.slug)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/devdocs2zim/client.py", line 329, in get_db
    file_contents = self.read_doc_file(doc_slug, "db.json")
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/devdocs2zim/client.py", line 308, in read_doc_file
    return self._get_text(f"{self.documents_url}/{doc_slug}/{file_name}")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/devdocs2zim/client.py", line 264, in _get_text
    resp = requests.get(
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 713, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='documents.devdocs.io', port=443): Read timed out. (read timeout=15)
[devdocs2zim::2026-06-02 15:16:58,890] ERROR:Generation failed with the following error: HTTPSConnectionPool(host='documents.devdocs.io', port=443): Read timed out. (read timeout=15)

Looks like it is very related to the same worker starting two tasks in a row.