JuliaParallel/Dagger.jl

Remove scheduler plugin machinery, make Sch programmable

Open

#166 建立於 2020年11月17日

在 GitHub 查看
 (1 留言) (0 反應) (0 負責人)Julia (86 fork)batch import
help wantedprocessorsscheduler

倉庫指標

Star
 (706 star)
PR 合併指標
 (平均合併 12天 5小時) (30 天內合併 4 個 PR)

描述

The scheduler plugin system is (to my knowledge) unused by any modern users of Dagger. However, the presence of the potential for multiple external schedulers makes changing the scheduler API technically a breaking change, which would slow Dagger's development were that actually a concern. Of course, we still want to support different kinds of scheduling algorithms and optimizations that best suit a user's use case and DAG structure, so we should make the Sch scheduler user-programmable by making the current worker pressure algorithm optional, and having schedule! call a user-defined callable.

The option to change the internal scheduler algorithm is intended to only be used by adventurous users who understand that deadlocks/livelocks/hangs/etc. are all possible when changing the default scheduler, although we should expose "safe" semi-internal APIs that can perform common tasks correctly (which the default scheduler should also use whenever possible). Hopefully this change can spur developers and adventurous users to experiment with (and contribute) new scheduling algorithms which are better than the default scheduler for certain classes of workloads, with Dagger itself becoming the foundation for user-defined scheduling of distributed Julia code.

貢獻者指南