datalad/datalad

Should sleep/retry a few times upon 'Network is unreachable'

Aberta

#5.886 aberto em 12 de ago. de 2021

 (0 comentário) (0 reação) (0 responsável)Python (128 forks)github user discovery
DXgood first issuetests

Métricas do repositório

Stars
 (640 estrelas)
Métricas de merge de PR
 (Mesclagem média 10d 18h) (8 fundiu PRs em 30d)

Description

During local tests got following test failure

======================================================================
ERROR: datalad.downloaders.tests.test_s3.test_restricted_bucket_on_NDA('s3://NDAR_Central_4/submission_23075/README', 'error', 'BIDS')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.9/urllib/request.py", line 1346, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/usr/lib/python3.9/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.9/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.9/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.9/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/usr/lib/python3.9/http/client.py", line 950, in send
    self.connect()
  File "/usr/lib/python3.9/http/client.py", line 1417, in connect
    super().connect()
  File "/usr/lib/python3.9/http/client.py", line 921, in connect
    self.sock = self._create_connection(
  File "/usr/lib/python3.9/socket.py", line 843, in create_connection
    raise err
  File "/usr/lib/python3.9/socket.py", line 831, in create_connection
    sock.connect(sa)
OSError: [Errno 101] Network is unreachable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/yoh/proj/datalad/datalad-maint/datalad/tests/utils.py", line 764, in _wrap_with_tempfile
    return t(*(arg + (filename,)), **kw)
  File "/home/yoh/proj/datalad/datalad-maint/datalad/downloaders/tests/test_http.py", line 279, in check_download_external_url
    downloaded_path = downloader.download(url, path=d)
  File "/home/yoh/proj/datalad/datalad-maint/datalad/downloaders/base.py", line 520, in download
    return self.access(self._download, url, path=path, **kwargs)
  File "/home/yoh/proj/datalad/datalad-maint/datalad/downloaders/base.py", line 165, in access
    used_old_session = self._establish_session(url, allow_old=allow_old_session)
  File "/home/yoh/proj/datalad/datalad-maint/datalad/downloaders/s3.py", line 228, in _establish_session
    self._bucket = try_multiple_dec_s3(self.authenticator.authenticate)(
  File "/home/yoh/proj/datalad/datalad-maint/datalad/utils.py", line 2032, in _wrap_try_multiple_dec
    return f(*args, **kwargs)
  File "/home/yoh/proj/datalad/datalad-maint/datalad/downloaders/s3.py", line 106, in authenticate
    credentials = credential()
  File "/home/yoh/proj/datalad/datalad-maint/datalad/downloaders/credentials.py", line 310, in __call__
    next_fields = adapter(self, **fields)
  File "/home/yoh/proj/datalad/datalad-maint/datalad/downloaders/credentials.py", line 324, in _nda_adapter
    token = gen.generate_token(user, password)
  File "/home/yoh/proj/datalad/datalad-maint/datalad/support/third/nda_aws_token_generator.py", line 34, in generate_token
    return self.__make_request(request_xml)
  File "/home/yoh/proj/datalad/datalad-maint/datalad/support/third/nda_aws_token_generator.py", line 81, in __make_request
    response = urllib_request.urlopen(request)
  File "/usr/lib/python3.9/urllib/request.py", line 214, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.9/urllib/request.py", line 517, in open
    response = self._open(req, data)
  File "/usr/lib/python3.9/urllib/request.py", line 534, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/usr/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.9/urllib/request.py", line 1389, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "/usr/lib/python3.9/urllib/request.py", line 1349, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 101] Network is unreachable>

and in that try_multiple_dec_s3 we only retry on:

                exceptions=S3ResponseError,
                exceptions_filter=_check_S3ResponseError,

where adding urllib.error.URLError to exceptions might do the trick, but _check_S3ResponseError might need tune up to ensure that it returns True (should be retried) on that one.

Test should mock urllib_request.urlopen so it first fails with that unreachable error and then proceeds on subsequent run. URL to try on could be any of the ones we already use, e.g. s3://datalad-test0-versioned/2versions-nonversioned1.txt although this one would not trigger auth code path like here, so we should try on multiple , e.g. also on 3://NDAR_Central_4/submission_23075/README used here (but requires NDA authorization for this one)

Guia do colaborador