Hacktoberfest 2026：メンテナが10月に向けて印を付けた、オープンで初心者向けの issue。 Hacktoberfest の issue を見る

Priorities are ignored when stealing tasks

オープン
  • 何か問題がありますか？
#8,992 コメント 5 件 リアクション 0 件 担当者 0 名 GitHub で見る

まだ誰も着手していません。

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
35/100
issue の種類
バグ
明瞭さ
おおむね明確
活発さ
停滞
技術スタック
python
領域
distributed-systems

調査の方向性

まず、client.submit、priorities、worker-saturation、task stealing を使用して、説明されている 8 タスク・2 ワーカーのシナリオを再現します。最初に scheduler と worker の task stealing の動作を調査します。完了の条件は、作業が盗まれたときに優先度付きタスクが期待どおりの順序で選択・実行され、報告された環境が回帰テストでカバーされていることです。

索引モデルが issue の本文から書いたものです。

説明

bug help wanted scheduling stealing

Describe the issue:

There are a few issues actually (two of them related to stealing), so let's go through the whole process.

I use dask and distributed packages to run automation in my project.
To speed up test execution (the tests cannot be run on a single environment in parallel), I spin up multiple VMs (dask workers) and then create a queue of tests (tasks) and assign them to workers until the tasks are depleted. Each task is unique and run once unless it failed (then it’s rerun).
This works perfectly but I noticed that using priorities may speed up execution significantly.

I tried to use dask priorities but ultimately failed. Here are my findings (the first issue is not related to the topic but is a problem nevertheless):

  1. All of the tasks are consumed immediately by the first VM created; I would like to have no more than one task assigned to a VM at once - this would ensure that priorities are balanced better.
  2. Dask dashboard shows that priorities have been assigned correctly, but only tasks on the first VM are run according to the priority. The priorities seems to be ignored on the second VM (the one that stole tasks from the first).
  3. The VM that stole the tasks grab random tasks, not the highest priority ones.

I would prefer assigning only one task per VM instead of stealing, but if I were really to use stealing, than it should grab the highest priority tasks that are not being processed (now it grabs random tasks)
To resolve the first problem, I set distributed.scheduler.worker-saturation to 1.0 but it didn’t help, all of the tasks are consumed at once.
I tried using fifo-timeout to deal with the second one, but it wasn’t helpful (I thought that fifo-timeout is taken into consideration when stealing).

The whole process looks as follows:

  1. The agent (host) VM creates tasks (not dask → asyncio.create_task()) to create VMs from snapshots (every VM is identical). Please note that spinning up a VM and then setting it up takes several minutes, so the tasks are created before the workers exist.
    1.1. Register set up VMs as workers with cmd (nthreads=1).
  2. In the meantime, the agent schedules tests asynchronously:
    2.1. Create a list of futures
    2.2. Append (client.submit(test)) each test to the list of futures
    2.3. Run as_completed to gather futures; if there are failed tests, readd them to the futures list
  3. If there are any existing workers, they start to pick up the tasks

With the current behaviour I haven’t encountered a single situation where a lowest priority task was run last. My task lengths vary from between 20 seconds to 3 minutes; the whole runtime can increase significantly if the tasks are ordered improperly.

Example:
I tested it on a small scale (8 tasks, 2 VMs).
The priorities are equal to an estimated task runtime in seconds (actual runtime is about priority + 40 seconds). That ensures that the slowest tasks have the highest priority.

The first worker consumed all of the tasks immediately.
Image

As soon as the second VM was created, it stole several tasks from the first worker.
Image

Then the workers kept stealing work from each other.
Tasks on the second VM were run in random order.
After the first task was executed on the first VM, the other tasks (which were re-stolen from the second VM) were run in order they were stolen.
Image

Environment:

  • Dask version: 2024.12.1
  • Distributed version: 2024.12.1
  • Python version: 3.13.1
  • Operating System: Windows Server 2022 on both scheduler and workers
  • Install method: pip

Conclusion

What I only want to achieve here is to make prioritization work.
I am not sure if all of the work consumed by the first VM I create is due to a flaw in my setup process; if this can be fixed, then I could skip stealing altogether.
Still, priorities seem to be ignored when stealing tasks which is a real issue.

主要言語
Python
スター
1.7k
フォーク
778
30日以内にマージされた PR はありません

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

dask/distributed のほかの issue

dask/distributed の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。