Fetching the document database often times-out

Inizia in devdocs2zim/client.py, in _get_text(), read_doc_file() e get_db(), quindi segui il loro utilizzo da generator.py. Riproduci il timeout di lettura documentato verso documents.devdocs.io e analizza come vengono gestiti i task consecutivi. Il lavoro è completato quando i timeout transitori durante il recupero del database dei documenti non causano più il fallimento imprevisto della generazione; verifica con i test pertinenti o con un’esecuzione ripetuta della generazione.

See e.g. https://farm.openzim.org/pipeline/761e418d-483b-4909-b6f7-a92c9fdccca3/debug, but we have ten to twenty occurrences every quarter.

Restarting the recipe is enough to solve the problem, sometimes twice.

[devdocs2zim::2026-06-02 15:16:43,323] INFO:  Fetching the index...
[devdocs2zim::2026-06-02 15:16:43,825] INFO:  Fetching the document database...
[devdocs2zim::2026-06-02 15:16:58,864] ERROR:HTTPSConnectionPool(host='documents.devdocs.io', port=443): Read timed out. (read timeout=15)
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 534, in _make_request
    response = conn.getresponse()
               ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/urllib3/connection.py", line 571, in getresponse
    httplib_response = super().getresponse()
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/http/client.py", line 1430, in getresponse
    response.begin()
  File "/usr/local/lib/python3.12/http/client.py", line 331, in begin
    version, status, reason = self._read_status()
                              ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/http/client.py", line 292, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/socket.py", line 720, in readinto
    return self._sock.recv_into(b)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/ssl.py", line 1251, in recv_into
    return self.read(nbytes, buffer)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/ssl.py", line 1103, in read
    return self._sslobj.read(len, buffer)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TimeoutError: The read operation timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 841, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 490, in increment
    raise reraise(type(error), error, _stacktrace)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/urllib3/util/util.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 536, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 367, in _raise_timeout
    raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='documents.devdocs.io', port=443): Read timed out. (read timeout=15)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/devdocs2zim/entrypoint.py", line 104, in main
    ).run()
      ^^^^^
  File "/usr/local/lib/python3.12/site-packages/devdocs2zim/generator.py", line 454, in run
    self.generate_zim(
  File "/usr/local/lib/python3.12/site-packages/devdocs2zim/generator.py", line 506, in generate_zim
    db = self.devdocs_client.get_db(doc_metadata.slug)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/devdocs2zim/client.py", line 329, in get_db
    file_contents = self.read_doc_file(doc_slug, "db.json")
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/devdocs2zim/client.py", line 308, in read_doc_file
    return self._get_text(f"{self.documents_url}/{doc_slug}/{file_name}")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/devdocs2zim/client.py", line 264, in _get_text
    resp = requests.get(
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 713, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='documents.devdocs.io', port=443): Read timed out. (read timeout=15)
[devdocs2zim::2026-06-02 15:16:58,890] ERROR:Generation failed with the following error: HTTPSConnectionPool(host='documents.devdocs.io', port=443): Read timed out. (read timeout=15)

Looks like it is very related to the same worker starting two tasks in a row.