RuntimeError: cannot schedule new futures after shutdown thrown during cluster shutdown

Inizia riproducendo l’esempio main() fornito con FargateCluster e confronta il comportamento dell’arresto tra Python 3.8 e 3.10. Traccia client.close() e cluster.close() attraverso lo stack di aiobotocore e aiohttp mostrato nel traceback. Il lavoro è completato quando l’arresto non genera alcun RuntimeError e le risorse AWS come TaskDefinitionLists e IAMRole non rimangono più presenti.

What happened:
RuntimeErrors are thrown at the end of execution when the cluster is being killed. As a result, cloud resources such as TaskDefinitionLists and IAMRole seem to linger on.
What you expected to happen:
No errors thrown during the shutdown procedure. I tested the same minimal example in a py3.8 env and could not reproduce this issue; however, I could reproduce the same issue in a py3.9 environment.

Environment:

• Dask version: 2022.6.1
• Python version: 3.10.5
• Operating System: macOS 12.4 (21F79)
• Install method (conda, pip, source): conda

Py3.10 environment:

name: test
channels:
  - conda-forge
dependencies:
  - aiobotocore=2.3.4=pyhd8ed1ab_0
  - aiohttp=3.8.1=py310h1961e1f_1
  - aioitertools=0.10.0=pyhd8ed1ab_0
  - aiosignal=1.2.0=pyhd8ed1ab_0
  - async-timeout=4.0.2=pyhd8ed1ab_0
  - attrs=21.4.0=pyhd8ed1ab_0
  - bokeh=2.4.3=py310h2ec42d9_0
  - botocore=1.24.21=pyhd8ed1ab_1
  - brotlipy=0.7.0=py310h1961e1f_1004
  - bzip2=1.0.8=h0d85af4_4
  - ca-certificates=2022.6.15=h033912b_0
  - certifi=2022.6.15=py310h2ec42d9_0
  - cffi=1.15.1=py310h96bbf6e_0
  - charset-normalizer=2.1.0=pyhd8ed1ab_0
  - click=8.1.3=py310h2ec42d9_0
  - cloudpickle=2.1.0=pyhd8ed1ab_0
  - cryptography=37.0.2=py310h52c3658_0
  - cytoolz=0.11.2=py310h1961e1f_2
  - dask=2022.6.1=pyhd8ed1ab_0
  - dask-cloudprovider=2022.5.0=pyhd8ed1ab_0
  - dask-core=2022.6.1=pyhd8ed1ab_0
  - distributed=2022.6.1=pyhd8ed1ab_0
  - freetype=2.10.4=h4cff582_1
  - frozenlist=1.3.0=py310h1961e1f_1
  - fsspec=2022.5.0=pyhd8ed1ab_0
  - giflib=5.2.1=hbcb3906_2
  - heapdict=1.0.1=py_0
  - idna=3.3=pyhd8ed1ab_0
  - jinja2=3.1.2=pyhd8ed1ab_1
  - jmespath=1.0.1=pyhd8ed1ab_0
  - jpeg=9e=hac89ed1_2
  - lcms2=2.12=h577c468_0
  - lerc=3.0=he49afe7_0
  - libblas=3.9.0=15_osx64_openblas
  - libcblas=3.9.0=15_osx64_openblas
  - libcxx=14.0.6=hce7ea42_0
  - libdeflate=1.12=hac89ed1_0
  - libffi=3.4.2=h0d85af4_5
  - libgfortran=5.0.0=9_3_0_h6c81a4c_23
  - libgfortran5=9.3.0=h6c81a4c_23
  - liblapack=3.9.0=15_osx64_openblas
  - libopenblas=0.3.20=openmp_hb3cd9ec_0
  - libpng=1.6.37=h7cec526_2
  - libtiff=4.4.0=h9847915_1
  - libwebp=1.2.2=h28dabe5_0
  - libwebp-base=1.2.2=h0d85af4_1
  - libxcb=1.13=h0d85af4_1004
  - libzlib=1.2.12=hfe4f2af_1
  - llvm-openmp=14.0.4=ha654fa7_0
  - locket=1.0.0=pyhd8ed1ab_0
  - lz4=4.0.0=py310h2e8d50f_2
  - lz4-c=1.9.3=he49afe7_1
  - markupsafe=2.1.1=py310h1961e1f_1
  - msgpack-python=1.0.4=py310habb735a_0
  - multidict=6.0.2=py310h1961e1f_1
  - ncurses=6.3=h96cf925_1
  - numpy=1.23.0=py310ha3f357c_0
  - openjpeg=2.4.0=h6e7aa92_1
  - openssl=1.1.1p=hfe4f2af_0
  - packaging=21.3=pyhd8ed1ab_0
  - pandas=1.4.3=py310h3099161_0
  - partd=1.2.0=pyhd8ed1ab_0
  - pillow=9.1.1=py310hb3240ae_1
  - pip=22.1.2=pyhd8ed1ab_0
  - psutil=5.9.1=py310h6c45266_0
  - pthread-stubs=0.4=hc929b4f_1001
  - pycparser=2.21=pyhd8ed1ab_0
  - pyopenssl=22.0.0=pyhd8ed1ab_0
  - pyparsing=3.0.9=pyhd8ed1ab_0
  - pysocks=1.7.1=py310h2ec42d9_5
  - python=3.10.5=hdaaf3db_0_cpython
  - python-dateutil=2.8.2=pyhd8ed1ab_0
  - python_abi=3.10=2_cp310
  - pytz=2022.1=pyhd8ed1ab_0
  - pyyaml=6.0=py310h1961e1f_4
  - readline=8.1.2=h3899abd_0
  - setuptools=63.1.0=py310h2ec42d9_0
  - six=1.16.0=pyh6c4a22f_0
  - sortedcontainers=2.4.0=pyhd8ed1ab_0
  - sqlite=3.39.0=hd9f0692_0
  - tblib=1.7.0=pyhd8ed1ab_0
  - tk=8.6.12=h5dbffcc_0
  - toolz=0.11.2=pyhd8ed1ab_0
  - tornado=6.2=py310h6c45266_0
  - typing-extensions=4.3.0=hd8ed1ab_0
  - typing_extensions=4.3.0=pyha770c72_0
  - tzdata=2022a=h191b570_0
  - urllib3=1.26.9=pyhd8ed1ab_0
  - wheel=0.37.1=pyhd8ed1ab_0
  - wrapt=1.14.1=py310h6c45266_0
  - xorg-libxau=1.0.9=h35c211d_0
  - xorg-libxdmcp=1.1.3=h35c211d_0
  - xz=5.2.5=haf1e3a3_1
  - yaml=0.2.5=h0d85af4_2
  - yarl=1.7.2=py310h1961e1f_2
  - zict=2.2.0=pyhd8ed1ab_0
  - zlib=1.2.12=hfe4f2af_1
  - zstd=1.5.2=ha9df2e0_2

Py3.8 environment:

name: test
channels:
  - conda-forge
dependencies:
  - aiobotocore=2.3.4=pyhd8ed1ab_0
  - aiohttp=3.8.1=py38hed1de0f_1
  - aioitertools=0.10.0=pyhd8ed1ab_0
  - aiosignal=1.2.0=pyhd8ed1ab_0
  - async-timeout=4.0.2=pyhd8ed1ab_0
  - attrs=21.4.0=pyhd8ed1ab_0
  - bokeh=2.4.3=py38h50d1736_0
  - botocore=1.24.21=pyhd8ed1ab_1
  - brotlipy=0.7.0=py38hed1de0f_1004
  - bzip2=1.0.8=h0d85af4_4
  - ca-certificates=2022.6.15=h033912b_0
  - certifi=2022.6.15=py38h50d1736_0
  - cffi=1.15.1=py38h86886aa_0
  - charset-normalizer=2.1.0=pyhd8ed1ab_0
  - click=8.1.3=py38h50d1736_0
  - cloudpickle=2.1.0=pyhd8ed1ab_0
  - cryptography=37.0.2=py38h0f8513e_0
  - cytoolz=0.11.2=py38hed1de0f_2
  - dask=2022.6.1=pyhd8ed1ab_0
  - dask-cloudprovider=2022.5.0=pyhd8ed1ab_0
  - dask-core=2022.6.1=pyhd8ed1ab_0
  - distributed=2022.6.1=pyhd8ed1ab_0
  - freetype=2.10.4=h4cff582_1
  - frozenlist=1.3.0=py38hed1de0f_1
  - fsspec=2022.5.0=pyhd8ed1ab_0
  - giflib=5.2.1=hbcb3906_2
  - heapdict=1.0.1=py_0
  - idna=3.3=pyhd8ed1ab_0
  - jinja2=3.1.2=pyhd8ed1ab_1
  - jmespath=1.0.1=pyhd8ed1ab_0
  - jpeg=9e=hac89ed1_2
  - lcms2=2.12=h577c468_0
  - lerc=3.0=he49afe7_0
  - libblas=3.9.0=15_osx64_openblas
  - libcblas=3.9.0=15_osx64_openblas
  - libcxx=14.0.6=hce7ea42_0
  - libdeflate=1.12=hac89ed1_0
  - libffi=3.4.2=h0d85af4_5
  - libgfortran=5.0.0=9_3_0_h6c81a4c_23
  - libgfortran5=9.3.0=h6c81a4c_23
  - liblapack=3.9.0=15_osx64_openblas
  - libopenblas=0.3.20=openmp_hb3cd9ec_0
  - libpng=1.6.37=h7cec526_2
  - libtiff=4.4.0=h9847915_1
  - libwebp=1.2.2=h28dabe5_0
  - libwebp-base=1.2.2=h0d85af4_1
  - libxcb=1.13=h0d85af4_1004
  - libzlib=1.2.12=hfe4f2af_1
  - llvm-openmp=14.0.4=ha654fa7_0
  - locket=1.0.0=pyhd8ed1ab_0
  - lz4=4.0.0=py38h5cd37e2_2
  - lz4-c=1.9.3=he49afe7_1
  - markupsafe=2.1.1=py38hed1de0f_1
  - msgpack-python=1.0.4=py38hf58141a_0
  - multidict=6.0.2=py38hed1de0f_1
  - ncurses=6.3=h96cf925_1
  - numpy=1.23.0=py38h604f2a5_0
  - openjpeg=2.4.0=h6e7aa92_1
  - openssl=1.1.1p=hfe4f2af_0
  - packaging=21.3=pyhd8ed1ab_0
  - pandas=1.4.3=py38h2b30649_0
  - partd=1.2.0=pyhd8ed1ab_0
  - pillow=9.1.1=py38h21af888_1
  - pip=22.1.2=pyhd8ed1ab_0
  - psutil=5.9.1=py38h0dd4459_0
  - pthread-stubs=0.4=hc929b4f_1001
  - pycparser=2.21=pyhd8ed1ab_0
  - pyopenssl=22.0.0=pyhd8ed1ab_0
  - pyparsing=3.0.9=pyhd8ed1ab_0
  - pysocks=1.7.1=py38h50d1736_5
  - python=3.8.13=h394c593_0_cpython
  - python-dateutil=2.8.2=pyhd8ed1ab_0
  - python_abi=3.8=2_cp38
  - pytz=2022.1=pyhd8ed1ab_0
  - pyyaml=6.0=py38hed1de0f_4
  - readline=8.1.2=h3899abd_0
  - setuptools=63.1.0=py38h50d1736_0
  - six=1.16.0=pyh6c4a22f_0
  - sortedcontainers=2.4.0=pyhd8ed1ab_0
  - sqlite=3.39.0=hd9f0692_0
  - tblib=1.7.0=pyhd8ed1ab_0
  - tk=8.6.12=h5dbffcc_0
  - toolz=0.11.2=pyhd8ed1ab_0
  - tornado=6.2=py38h0dd4459_0
  - typing-extensions=4.3.0=hd8ed1ab_0
  - typing_extensions=4.3.0=pyha770c72_0
  - tzdata=2022a=h191b570_0
  - urllib3=1.26.9=pyhd8ed1ab_0
  - wheel=0.37.1=pyhd8ed1ab_0
  - wrapt=1.14.1=py38h0dd4459_0
  - xorg-libxau=1.0.9=h35c211d_0
  - xorg-libxdmcp=1.1.3=h35c211d_0
  - xz=5.2.5=haf1e3a3_1
  - yaml=0.2.5=h0d85af4_2
  - yarl=1.7.2=py38hed1de0f_2
  - zict=2.2.0=pyhd8ed1ab_0
  - zlib=1.2.12=hfe4f2af_1
  - zstd=1.5.2=ha9df2e0_2

Using the following script:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import logging
from time import sleep

from dask_cloudprovider.aws import FargateCluster, ECSCluster
from distributed import Client

###############################################################################

logging.basicConfig(
    level=logging.INFO,
    format="[%(levelname)4s: %(module)s:%(lineno)4s %(asctime)s] %(message)s",
)
log = logging.getLogger(__name__)

###############################################################################


def add_one(val: int) -> int:
    sleep(0.1)
    return val + 1


def multiply_by_ten(val: int) -> int:
    sleep(0.1)
    return val * 10


def main():
    # Cluster setup
    # https://docs.aws.amazon.com/AmazonECS/latest/developerguide/AWS_Fargate.html#fargate-tasks-size
    cluster = FargateCluster(fargate_spot=True, worker_mem=512, worker_cpu=256, n_workers=3)

    log.info(f"Cluster name is {cluster.name}")
    log.info(f"Dashboard available at: {cluster.dashboard_link}")
    client = Client(cluster.scheduler_address)

    # Actual work
    initial_values = list(range(100))
    log.info(f"Values range: [{initial_values[0]}-{initial_values[-1]}]")

    # Add
    addition_futures = client.map(add_one, initial_values)
    values = client.gather(addition_futures)

    # Multiply
    multiplication_futures = client.map(multiply_by_ten, values)
    values = client.gather(multiplication_futures)

    # Sort
    values = sorted(values)

    # Results
    log.info(f"Values range: [{values[0]}-{values[-1]}]")

    log.info("All done!")
    client.close()
    cluster.close()


if __name__ == "__main__":
    main()

Full Stacktrace:

Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/distributed/client.py:1274: VersionMismatchWarning: Mismatched versions found

+---------+----------------+----------------+----------------+
| Package | client         | scheduler      | workers        |
+---------+----------------+----------------+----------------+
| python  | 3.10.5.final.0 | 3.8.13.final.0 | 3.8.13.final.0 |
| tornado | 6.2            | 6.1            | 6.1            |
+---------+----------------+----------------+----------------+
  warnings.warn(version_module.VersionMismatchWarning(msg[0]["warning"]))
[INFO: client:  45 2022-07-04 16:30:16,057] Values range: [0-99]
[INFO: client:  59 2022-07-04 16:30:23,609] Values range: [10-1000]
[INFO: client:  61 2022-07-04 16:30:23,609] All done!
Traceback (most recent call last):
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/httpsession.py", line 178, in send
    response = await self._session.request(
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/client.py", line 535, in _request
    conn = await self._connector.connect(
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/connector.py", line 542, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/connector.py", line 907, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/connector.py", line 1154, in _create_direct_connection
    hosts = await asyncio.shield(host_resolved)
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/connector.py", line 880, in _resolve_host
    addrs = await self._resolver.resolve(host, port, family=self._family)
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/resolver.py", line 33, in resolve
    infos = await self._loop.getaddrinfo(
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/asyncio/base_events.py", line 860, in getaddrinfo
    return await self.run_in_executor(
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/asyncio/base_events.py", line 818, in run_in_executor
    executor.submit(func, *args), loop=self)
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/concurrent/futures/thread.py", line 167, in submit
    raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/weakref.py", line 667, in _exitfunc
    f()
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/weakref.py", line 591, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/distributed/utils.py", line 336, in sync
    return sync(
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/distributed/utils.py", line 403, in sync
    raise exc.with_traceback(tb)
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/distributed/utils.py", line 376, in f
    result = yield future
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/tornado/gen.py", line 769, in run
    value = future.result()
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/dask_cloudprovider/aws/ecs.py", line 1313, in _delete_worker_task_definition_arn
    await ecs.deregister_task_definition(
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/client.py", line 251, in _make_api_call
    http, parsed_response = await self._make_request(
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/client.py", line 271, in _make_request
    return await self._endpoint.make_request(operation_model, request_dict)
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 80, in _send_request
    while await self._needs_retry(attempts, operation_model,
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 213, in _needs_retry
    responses = await self._event_emitter.emit(
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/hooks.py", line 50, in _emit
    response = await resolve_awaitable(handler(**kwargs))
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/_helpers.py", line 11, in resolve_awaitable
    return await obj
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/retryhandler.py", line 89, in _call
    if await resolve_awaitable(self._checker(**checker_kwargs)):
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/_helpers.py", line 11, in resolve_awaitable
    return await obj
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/retryhandler.py", line 107, in _call
    should_retry = await self._should_retry(attempt_number, response,
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/retryhandler.py", line 129, in _should_retry
    return await resolve_awaitable(
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/_helpers.py", line 11, in resolve_awaitable
    return await obj
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/retryhandler.py", line 146, in _call
    checker(attempt_number, response,
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/botocore/retryhandler.py", line 231, in __call__
    return self._check_caught_exception(
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/botocore/retryhandler.py", line 374, in _check_caught_exception
    raise caught_exception
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 149, in _do_get_response
    http_response = await self._send(request)
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 229, in _send
    return await self.http_session.send(request)
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/httpsession.py", line 220, in send
    raise HTTPClientError(error=e)
botocore.exceptions.HTTPClientError: An HTTP Client raised an unhandled exception: cannot schedule new futures after shutdown
Traceback (most recent call last):
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/httpsession.py", line 178, in send
    response = await self._session.request(
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/client.py", line 535, in _request
    conn = await self._connector.connect(
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/connector.py", line 542, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/connector.py", line 907, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/connector.py", line 1154, in _create_direct_connection
    hosts = await asyncio.shield(host_resolved)
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/connector.py", line 880, in _resolve_host
    addrs = await self._resolver.resolve(host, port, family=self._family)
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/resolver.py", line 33, in resolve
    infos = await self._loop.getaddrinfo(
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/asyncio/base_events.py", line 860, in getaddrinfo
    return await self.run_in_executor(
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/asyncio/base_events.py", line 818, in run_in_executor
    executor.submit(func, *args), loop=self)
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/concurrent/futures/thread.py", line 167, in submit
    raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/weakref.py", line 667, in _exitfunc
    f()
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/weakref.py", line 591, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/distributed/utils.py", line 336, in sync
    return sync(
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/distributed/utils.py", line 403, in sync
    raise exc.with_traceback(tb)
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/distributed/utils.py", line 376, in f
    result = yield future
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/tornado/gen.py", line 769, in run
    value = future.result()
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/dask_cloudprovider/aws/ecs.py", line 1240, in _delete_scheduler_task_definition_arn
    await ecs.deregister_task_definition(
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/client.py", line 251, in _make_api_call
    http, parsed_response = await self._make_request(
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/client.py", line 271, in _make_request
    return await self._endpoint.make_request(operation_model, request_dict)
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 80, in _send_request
    while await self._needs_retry(attempts, operation_model,
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 213, in _needs_retry
    responses = await self._event_emitter.emit(
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/hooks.py", line 50, in _emit
    response = await resolve_awaitable(handler(**kwargs))
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/_helpers.py", line 11, in resolve_awaitable
    return await obj
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/retryhandler.py", line 89, in _call
    if await resolve_awaitable(self._checker(**checker_kwargs)):
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/_helpers.py", line 11, in resolve_awaitable
    return await obj
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/retryhandler.py", line 107, in _call
    should_retry = await self._should_retry(attempt_number, response,
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/retryhandler.py", line 129, in _should_retry
    return await resolve_awaitable(
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/_helpers.py", line 11, in resolve_awaitable
    return await obj
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/retryhandler.py", line 146, in _call
    checker(attempt_number, response,
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/botocore/retryhandler.py", line 231, in __call__
    return self._check_caught_exception(
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/botocore/retryhandler.py", line 374, in _check_caught_exception
    raise caught_exception
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 149, in _do_get_response
    http_response = await self._send(request)
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 229, in _send
    return await self.http_session.send(request)
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/httpsession.py", line 220, in send
    raise HTTPClientError(error=e)
botocore.exceptions.HTTPClientError: An HTTP Client raised an unhandled exception: cannot schedule new futures after shutdown
Traceback (most recent call last):
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/httpsession.py", line 178, in send
    response = await self._session.request(
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/client.py", line 535, in _request
    conn = await self._connector.connect(
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/connector.py", line 542, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/connector.py", line 907, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/connector.py", line 1154, in _create_direct_connection
    hosts = await asyncio.shield(host_resolved)
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/connector.py", line 880, in _resolve_host
    addrs = await self._resolver.resolve(host, port, family=self._family)
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/resolver.py", line 33, in resolve
    infos = await self._loop.getaddrinfo(
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/asyncio/base_events.py", line 860, in getaddrinfo
    return await self.run_in_executor(
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/asyncio/base_events.py", line 818, in run_in_executor
    executor.submit(func, *args), loop=self)
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/concurrent/futures/thread.py", line 167, in submit
    raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/weakref.py", line 667, in _exitfunc
    f()
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/weakref.py", line 591, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/distributed/utils.py", line 336, in sync
    return sync(
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/distributed/utils.py", line 403, in sync
    raise exc.with_traceback(tb)
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/distributed/utils.py", line 376, in f
    result = yield future
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/tornado/gen.py", line 769, in run
    value = future.result()
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/dask_cloudprovider/aws/ecs.py", line 1130, in _delete_role
    attached_policies = (await iam.list_attached_role_policies(RoleName=role))[
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/client.py", line 251, in _make_api_call
    http, parsed_response = await self._make_request(
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/client.py", line 271, in _make_request
    return await self._endpoint.make_request(operation_model, request_dict)
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 80, in _send_request
    while await self._needs_retry(attempts, operation_model,
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 213, in _needs_retry
    responses = await self._event_emitter.emit(
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/hooks.py", line 50, in _emit
    response = await resolve_awaitable(handler(**kwargs))
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/_helpers.py", line 11, in resolve_awaitable
    return await obj
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/retryhandler.py", line 89, in _call
    if await resolve_awaitable(self._checker(**checker_kwargs)):
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/_helpers.py", line 11, in resolve_awaitable
    return await obj
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/retryhandler.py", line 107, in _call
    should_retry = await self._should_retry(attempt_number, response,
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/retryhandler.py", line 129, in _should_retry
    return await resolve_awaitable(
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/_helpers.py", line 11, in resolve_awaitable
    return await obj
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/retryhandler.py", line 146, in _call
    checker(attempt_number, response,
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/botocore/retryhandler.py", line 231, in __call__
    return self._check_caught_exception(
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/botocore/retryhandler.py", line 374, in _check_caught_exception
    raise caught_exception
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 149, in _do_get_response
    http_response = await self._send(request)
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 229, in _send
    return await self.http_session.send(request)
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/httpsession.py", line 220, in send
    raise HTTPClientError(error=e)
botocore.exceptions.HTTPClientError: An HTTP Client raised an unhandled exception: cannot schedule new futures after shutdown
Traceback (most recent call last):
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/httpsession.py", line 178, in send
    response = await self._session.request(
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/client.py", line 535, in _request
    conn = await self._connector.connect(
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/connector.py", line 542, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/connector.py", line 907, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/connector.py", line 1154, in _create_direct_connection
    hosts = await asyncio.shield(host_resolved)
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/connector.py", line 880, in _resolve_host
    addrs = await self._resolver.resolve(host, port, family=self._family)
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/resolver.py", line 33, in resolve
    infos = await self._loop.getaddrinfo(
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/asyncio/base_events.py", line 860, in getaddrinfo
    return await self.run_in_executor(
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/asyncio/base_events.py", line 818, in run_in_executor
    executor.submit(func, *args), loop=self)
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/concurrent/futures/thread.py", line 167, in submit
    raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/weakref.py", line 667, in _exitfunc
    f()
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/weakref.py", line 591, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/distributed/utils.py", line 336, in sync
    return sync(
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/distributed/utils.py", line 403, in sync
    raise exc.with_traceback(tb)
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/distributed/utils.py", line 376, in f
    result = yield future
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/tornado/gen.py", line 769, in run
    value = future.result()
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/dask_cloudprovider/aws/ecs.py", line 1130, in _delete_role
    attached_policies = (await iam.list_attached_role_policies(RoleName=role))[
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/client.py", line 251, in _make_api_call
    http, parsed_response = await self._make_request(
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/client.py", line 271, in _make_request
    return await self._endpoint.make_request(operation_model, request_dict)
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 80, in _send_request
    while await self._needs_retry(attempts, operation_model,
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 213, in _needs_retry
    responses = await self._event_emitter.emit(
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/hooks.py", line 50, in _emit
    response = await resolve_awaitable(handler(**kwargs))
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/_helpers.py", line 11, in resolve_awaitable
    return await obj
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/retryhandler.py", line 89, in _call
    if await resolve_awaitable(self._checker(**checker_kwargs)):
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/_helpers.py", line 11, in resolve_awaitable
    return await obj
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/retryhandler.py", line 107, in _call
    should_retry = await self._should_retry(attempt_number, response,
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/retryhandler.py", line 129, in _should_retry
    return await resolve_awaitable(
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/_helpers.py", line 11, in resolve_awaitable
    return await obj
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/retryhandler.py", line 146, in _call
    checker(attempt_number, response,
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/botocore/retryhandler.py", line 231, in __call__
    return self._check_caught_exception(
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/botocore/retryhandler.py", line 374, in _check_caught_exception
    raise caught_exception
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 149, in _do_get_response
    http_response = await self._send(request)
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 229, in _send
    return await self.http_session.send(request)
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/httpsession.py", line 220, in send
    raise HTTPClientError(error=e)
botocore.exceptions.HTTPClientError: An HTTP Client raised an unhandled exception: cannot schedule new futures after shutdown
Traceback (most recent call last):
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/httpsession.py", line 178, in send
    response = await self._session.request(
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/client.py", line 535, in _request
    conn = await self._connector.connect(
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/connector.py", line 542, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/connector.py", line 907, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/connector.py", line 1154, in _create_direct_connection
    hosts = await asyncio.shield(host_resolved)
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/connector.py", line 880, in _resolve_host
    addrs = await self._resolver.resolve(host, port, family=self._family)
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiohttp/resolver.py", line 33, in resolve
    infos = await self._loop.getaddrinfo(
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/asyncio/base_events.py", line 860, in getaddrinfo
    return await self.run_in_executor(
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/asyncio/base_events.py", line 818, in run_in_executor
    executor.submit(func, *args), loop=self)
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/concurrent/futures/thread.py", line 167, in submit
    raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/weakref.py", line 667, in _exitfunc
    f()
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/weakref.py", line 591, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/distributed/utils.py", line 336, in sync
    return sync(
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/distributed/utils.py", line 403, in sync
    raise exc.with_traceback(tb)
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/distributed/utils.py", line 376, in f
    result = yield future
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/tornado/gen.py", line 769, in run
    value = future.result()
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/dask_cloudprovider/aws/ecs.py", line 1049, in _delete_cluster
    async for page in ecs.get_paginator("list_tasks").paginate(
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/paginate.py", line 32, in __anext__
    response = await self._make_request(current_kwargs)
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/client.py", line 251, in _make_api_call
    http, parsed_response = await self._make_request(
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/client.py", line 271, in _make_request
    return await self._endpoint.make_request(operation_model, request_dict)
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 80, in _send_request
    while await self._needs_retry(attempts, operation_model,
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 213, in _needs_retry
    responses = await self._event_emitter.emit(
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/hooks.py", line 50, in _emit
    response = await resolve_awaitable(handler(**kwargs))
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/_helpers.py", line 11, in resolve_awaitable
    return await obj
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/retryhandler.py", line 89, in _call
    if await resolve_awaitable(self._checker(**checker_kwargs)):
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/_helpers.py", line 11, in resolve_awaitable
    return await obj
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/retryhandler.py", line 107, in _call
    should_retry = await self._should_retry(attempt_number, response,
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/retryhandler.py", line 129, in _should_retry
    return await resolve_awaitable(
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/_helpers.py", line 11, in resolve_awaitable
    return await obj
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/retryhandler.py", line 146, in _call
    checker(attempt_number, response,
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/botocore/retryhandler.py", line 231, in __call__
    return self._check_caught_exception(
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/botocore/retryhandler.py", line 374, in _check_caught_exception
    raise caught_exception
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 149, in _do_get_response
    http_response = await self._send(request)
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/endpoint.py", line 229, in _send
    return await self.http_session.send(request)
  File "/Users/ak/mambaforge/envs/test/lib/python3.10/site-packages/aiobotocore/httpsession.py", line 220, in send
    raise HTTPClientError(error=e)
botocore.exceptions.HTTPClientError: An HTTP Client raised an unhandled exception: cannot schedule new futures after shutdown