Hacktoberfest 2026：维护者为十月标记出来的 issue，仍然开放、适合新手。 浏览 Hacktoberfest issue

Priorities are ignored when stealing tasks

未关闭
  • 哪里不对？
#8,992 5 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

评估

难度
4/5
预计耗时
3-5 天
新手友好度
35/100
Issue 类型
缺陷
描述清晰度
基本清楚
活跃度
停滞
技术栈
python
领域
distributed-systems

调研方向

首先使用 client.submit、priorities、worker-saturation 和 task stealing 重现所描述的八任务、双 worker 场景。先检查 scheduler 和 worker 的任务窃取行为；完成的标准是：发生任务窃取时，优先任务会按预期顺序被选择并执行，并且报告的环境由回归测试覆盖。

由索引模型根据 Issue 内容生成。

描述

bug help wanted scheduling stealing

Describe the issue:

There are a few issues actually (two of them related to stealing), so let's go through the whole process.

I use dask and distributed packages to run automation in my project.
To speed up test execution (the tests cannot be run on a single environment in parallel), I spin up multiple VMs (dask workers) and then create a queue of tests (tasks) and assign them to workers until the tasks are depleted. Each task is unique and run once unless it failed (then it’s rerun).
This works perfectly but I noticed that using priorities may speed up execution significantly.

I tried to use dask priorities but ultimately failed. Here are my findings (the first issue is not related to the topic but is a problem nevertheless):

  1. All of the tasks are consumed immediately by the first VM created; I would like to have no more than one task assigned to a VM at once - this would ensure that priorities are balanced better.
  2. Dask dashboard shows that priorities have been assigned correctly, but only tasks on the first VM are run according to the priority. The priorities seems to be ignored on the second VM (the one that stole tasks from the first).
  3. The VM that stole the tasks grab random tasks, not the highest priority ones.

I would prefer assigning only one task per VM instead of stealing, but if I were really to use stealing, than it should grab the highest priority tasks that are not being processed (now it grabs random tasks)
To resolve the first problem, I set distributed.scheduler.worker-saturation to 1.0 but it didn’t help, all of the tasks are consumed at once.
I tried using fifo-timeout to deal with the second one, but it wasn’t helpful (I thought that fifo-timeout is taken into consideration when stealing).

The whole process looks as follows:

  1. The agent (host) VM creates tasks (not dask → asyncio.create_task()) to create VMs from snapshots (every VM is identical). Please note that spinning up a VM and then setting it up takes several minutes, so the tasks are created before the workers exist.
    1.1. Register set up VMs as workers with cmd (nthreads=1).
  2. In the meantime, the agent schedules tests asynchronously:
    2.1. Create a list of futures
    2.2. Append (client.submit(test)) each test to the list of futures
    2.3. Run as_completed to gather futures; if there are failed tests, readd them to the futures list
  3. If there are any existing workers, they start to pick up the tasks

With the current behaviour I haven’t encountered a single situation where a lowest priority task was run last. My task lengths vary from between 20 seconds to 3 minutes; the whole runtime can increase significantly if the tasks are ordered improperly.

Example:
I tested it on a small scale (8 tasks, 2 VMs).
The priorities are equal to an estimated task runtime in seconds (actual runtime is about priority + 40 seconds). That ensures that the slowest tasks have the highest priority.

The first worker consumed all of the tasks immediately.
Image

As soon as the second VM was created, it stole several tasks from the first worker.
Image

Then the workers kept stealing work from each other.
Tasks on the second VM were run in random order.
After the first task was executed on the first VM, the other tasks (which were re-stolen from the second VM) were run in order they were stolen.
Image

Environment:

  • Dask version: 2024.12.1
  • Distributed version: 2024.12.1
  • Python version: 3.13.1
  • Operating System: Windows Server 2022 on both scheduler and workers
  • Install method: pip

Conclusion

What I only want to achieve here is to make prioritization work.
I am not sure if all of the work consumed by the first VM I create is due to a flaw in my setup process; if this can be fixed, then I could skip stealing altogether.
Still, priorities seem to be ignored when stealing tasks which is a real issue.

主要语言
Python
星标
1.7k
派生
780
30 天内没有已合并 PR

环境准备

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

dask/distributed 的其他 Issue

查看 dask/distributed 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。