Fetching the document database often times-out

Empieza en devdocs2zim/client.py, en _get_text(), read_doc_file() y get_db(), y luego sigue su uso desde generator.py. Reproduce el timeout de lectura documentado contra documents.devdocs.io e inspecciona cómo se gestionan las tareas consecutivas. La tarea estará terminada cuando los timeouts transitorios al obtener datos de la base de datos de documentos ya no hagan que la generación falle inesperadamente; verifícalo con las pruebas relevantes o con una ejecución repetida de la generación.

See e.g. https://farm.openzim.org/pipeline/761e418d-483b-4909-b6f7-a92c9fdccca3/debug, but we have ten to twenty occurrences every quarter.

Restarting the recipe is enough to solve the problem, sometimes twice.

[devdocs2zim::2026-06-02 15:16:43,323] INFO:  Fetching the index...
[devdocs2zim::2026-06-02 15:16:43,825] INFO:  Fetching the document database...
[devdocs2zim::2026-06-02 15:16:58,864] ERROR:HTTPSConnectionPool(host='documents.devdocs.io', port=443): Read timed out. (read timeout=15)
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 534, in _make_request
    response = conn.getresponse()
               ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/urllib3/connection.py", line 571, in getresponse
    httplib_response = super().getresponse()
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/http/client.py", line 1430, in getresponse
    response.begin()
  File "/usr/local/lib/python3.12/http/client.py", line 331, in begin
    version, status, reason = self._read_status()
                              ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/http/client.py", line 292, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/socket.py", line 720, in readinto
    return self._sock.recv_into(b)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/ssl.py", line 1251, in recv_into
    return self.read(nbytes, buffer)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/ssl.py", line 1103, in read
    return self._sslobj.read(len, buffer)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TimeoutError: The read operation timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 841, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 490, in increment
    raise reraise(type(error), error, _stacktrace)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/urllib3/util/util.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 536, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 367, in _raise_timeout
    raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='documents.devdocs.io', port=443): Read timed out. (read timeout=15)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/devdocs2zim/entrypoint.py", line 104, in main
    ).run()
      ^^^^^
  File "/usr/local/lib/python3.12/site-packages/devdocs2zim/generator.py", line 454, in run
    self.generate_zim(
  File "/usr/local/lib/python3.12/site-packages/devdocs2zim/generator.py", line 506, in generate_zim
    db = self.devdocs_client.get_db(doc_metadata.slug)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/devdocs2zim/client.py", line 329, in get_db
    file_contents = self.read_doc_file(doc_slug, "db.json")
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/devdocs2zim/client.py", line 308, in read_doc_file
    return self._get_text(f"{self.documents_url}/{doc_slug}/{file_name}")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/devdocs2zim/client.py", line 264, in _get_text
    resp = requests.get(
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 713, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='documents.devdocs.io', port=443): Read timed out. (read timeout=15)
[devdocs2zim::2026-06-02 15:16:58,890] ERROR:Generation failed with the following error: HTTPSConnectionPool(host='documents.devdocs.io', port=443): Read timed out. (read timeout=15)

Looks like it is very related to the same worker starting two tasks in a row.