Hacktoberfest 2026：メンテナが10月に向けて印を付けた、オープンで初心者向けの issue。 Hacktoberfest の issue を見る

SpecCluster._correct_state_internal closes workers with Nanny's default 5s timeout; shutdown often fails on LocalCluster(processes=True)

オープン
  • 何か問題がありますか？
#9,300 コメント 3 件 リアクション 0 件 担当者 0 名 GitHub で見る

@abhisek343 がすでに取り組んでいます。

2026年9月28日 から。

  • #9368 @abhisek343 による — オープン

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
48/100
issue の種類
バグ
明瞭さ
おおむね明確
活発さ
静か
技術スタック
python
領域
distributed-systems

調査の方向性

トレースバックは distributed/deploy/spec.py と SpecCluster._correct_state_internal() を指している。まず、完了した futures を保持したまま、LocalCluster(processes=True) と client.shutdown() のサンプルで問題を再現する。次に retirement と AMM の経路を追跡し、その後、クラスター全体の teardown によって不要な retirement 処理が回避され、TimeoutError が発生しなくなることを示す回帰チェックを確立する。

索引モデルが issue の本文から書いたものです。

説明

needs triage

TimeoutError during client.shutdown() due to unnecessary AMM retirement when futures are not explicitly released

Description

When shutting down a dedicated LocalCluster via client.shutdown() after batch processing is complete, Dask still executes the full graceful worker retirement path (retire_workers + Active Memory Manager replication/drop), even though all workers are being removed and the results are no longer needed.

If the client has not released its futures first, completed task data remains in the scheduler's state memory on the workers. During shutdown, the AMM attempts to replicate or drop those keys across workers that are simultaneously shutting down. This causes unnecessary network/memory overhead and slows down worker teardown. Consequently, this can exceed the Nanny’s default process.join timeout (~4s) and surface as a TimeoutError / Tornado ERROR log during SpecCluster._correct_state_internal().

Releasing all client-held keys before cluster.close() avoids the problem in practice, which suggests the current shutdown path is doing redundant work that a full-cluster teardown should not require.

Sample Code from My Project
Create cluster and client
cls.cluster = LocalCluster(
    name="cluster", 
    n_workers=cls.n_workers, 
    threads_per_worker=cls.threads_per_worker, 
    memory_limit=0, 
    dashboard_address=dashboard_address
)
cls.client = Client(
    name="client", 
    address=cls.cluster, 
    direct_to_workers=True
)

Shutdown
@classmethod
def system_teardown(cls):
    with suppress(Exception):
        cls.client.shutdown()
Error Stack Trace

Occasionally, the following error occurs during shutdown:

2026-06-10 14:56:47,407 - tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOMainLoop object at 0x00000288583A1D30>>, <Task finished name='Task-1174981' coro=<Spec Cluster._correct_state_internal() done, defined at .venvLibsite-packagesdistributeddeployspec.py:346> exception=TimeoutError()>)
Traceback (most recent call last):
  File ".venvLibsite-packagesdistributedutils.py", line 1910, in wait_for
    return await fut
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File ".venvLibsite-packagestornadoioloop.py", line 758, in _run_callback
    ret = callback()
  File ".venvLibsite-packagestornadoioloop.py", line 782, in _discard_future_result
    future.result()
TimeoutError
Root Cause Analysis

When client.shutdown() is called without prior key release, completed task data remains in the scheduler's state memory on the workers. During the shutdown sequence, the Active Memory Manager (AMM) attempts to replicate or drop these keys across workers that are concurrently being torn down. This triggers unnecessary network/memory operations and significantly slows down the worker teardown process. Ultimately, this exceeds the Nanny’s default process.join timeout (~4s), resulting in a TimeoutError within SpecCluster._correct_state_internal().

Suggestions / Recommendations

Optimize Full-Cluster Teardown: Consider bypassing or short-circuiting the AMM retirement logic (retire_workers, data replication/drop) when a full-cluster teardown is initiated via client.shutdown(). Since all workers are being removed anyway, preserving or migrating their data is redundant.

Auto-Release on Shutdown: Alternatively, automatically release all client-held keys before initiating the shutdown sequence to prevent the AMM from acting on stale future references.

Documentation: In the meantime, it might be helpful to document this behavior and recommend users explicitly release their futures (e.g., via client.release() or iterating over client.futures) before calling shutdown() as a best practice.

Thanks!

主要言語
Python
スター
1.7k
フォーク
781
30日以内にマージされた PR はありません

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

dask/distributed のほかの issue

dask/distributed の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。