ReadTimeout

Start in apps/creatives_manager/processors/base.py at _get_files_from_dropbox and inspect the files_list_folder_continue(result.cursor) call shown in the traceback. Reproduce the listing with a folder containing nearly 100k files and determine why continuation times out despite completing; done means the behavior is fixed or its expected handling is clearly established.

There is an error while retrieving list of files from folder with almost 100k files. But all files were retrieved successfully.

Traceback (most recent call last):
  File ".pyenv/versions/marketing-ua-reports/lib/python3.6/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File ".pyenv/versions/marketing-ua-reports/lib/python3.6/site-packages/urllib3/connectionpool.py", line 382, in _make_request
    httplib_response = conn.getresponse()
  File ".pyenv/versions/3.6.4/lib/python3.6/http/client.py", line 1331, in getresponse
    response.begin()
  File ".pyenv/versions/3.6.4/lib/python3.6/http/client.py", line 297, in begin
    version, status, reason = self._read_status()
  File ".pyenv/versions/3.6.4/lib/python3.6/http/client.py", line 258, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File ".pyenv/versions/3.6.4/lib/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
  File ".pyenv/versions/marketing-ua-reports/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py", line 291, in recv_into
    raise timeout('The read operation timed out')
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File ".pyenv/versions/marketing-ua-reports/lib/python3.6/site-packages/requests/adapters.py", line 445, in send
    timeout=timeout
  File ".pyenv/versions/marketing-ua-reports/lib/python3.6/site-packages/urllib3/connectionpool.py", line 649, in urlopen
    _stacktrace=sys.exc_info()[2])
  File ".pyenv/versions/marketing-ua-reports/lib/python3.6/site-packages/urllib3/util/retry.py", line 357, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File ".pyenv/versions/marketing-ua-reports/lib/python3.6/site-packages/urllib3/packages/six.py", line 686, in reraise
    raise value
  File ".pyenv/versions/marketing-ua-reports/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File ".pyenv/versions/marketing-ua-reports/lib/python3.6/site-packages/urllib3/connectionpool.py", line 388, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File ".pyenv/versions/marketing-ua-reports/lib/python3.6/site-packages/urllib3/connectionpool.py", line 308, in _raise_timeout
    raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='api.dropboxapi.com', port=443): Read timed out. (read timeout=30)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "apps/creatives_manager/processors/base.py", line 144, in _get_files_from_dropbox
    result = self.dropbox.files_list_folder_continue(result.cursor)
  File ".pyenv/versions/marketing-ua-reports/lib/python3.6/site-packages/dropbox/base.py", line 1583, in files_list_folder_continue
    None,
  File ".pyenv/versions/marketing-ua-reports/lib/python3.6/site-packages/dropbox/dropbox.py", line 274, in request
    timeout=timeout)
  File ".pyenv/versions/marketing-ua-reports/lib/python3.6/site-packages/dropbox/dropbox.py", line 365, in request_json_string_with_retry
    timeout=timeout)
  File ".pyenv/versions/marketing-ua-reports/lib/python3.6/site-packages/dropbox/dropbox.py", line 449, in request_json_string
    timeout=timeout,
  File ".pyenv/versions/marketing-ua-reports/lib/python3.6/site-packages/requests/sessions.py", line 559, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File ".pyenv/versions/marketing-ua-reports/lib/python3.6/site-packages/requests/sessions.py", line 512, in request
    resp = self.send(prep, **send_kwargs)
  File ".pyenv/versions/marketing-ua-reports/lib/python3.6/site-packages/requests/sessions.py", line 622, in send
    r = adapter.send(request, **kwargs)
  File ".pyenv/versions/marketing-ua-reports/lib/python3.6/site-packages/requests/adapters.py", line 526, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='api.dropboxapi.com', port=443): Read timed out. (read timeout=30)

Repeating of call self.dropbox.files_list_folder_continue(result.cursor) and enlarging timeout don't help.