[feat] Make the CPU reservation of the controller and SimpleStorageUnit configurable

未关闭 适合新手
  • 哪里不对？
#207 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

还没有人认领这个 Issue。

评估

难度
2/5
预计耗时
1-3 小时
新手友好度
78/100
Issue 类型
功能
描述清晰度
描述清楚
活跃度
活跃
技术栈
python
领域
performance

调研方向

两个调用点定义了 actors：transfer_queue/controller.py 大约在第 1107 行，以及 transfer_queue/storage/simple_storage.py 大约在第 900 行，二者目前都是 @ray.remote(num_cpus=1)。将它们与 TransferQueueLockManager 一起阅读，后者已经使用 num_cpus=0，然后读取两个环境变量，并通过 .options(num_cpus=...) 将值传入 .remote() 调用，在未设置时默认为 1。完成的样子是一个构造 actors 并断言其请求的 num_cpus 的测试，并且在变量缺失时行为不变。

由索引模型根据 Issue 内容生成。

描述

Problem

TransferQueueController and SimpleStorageUnit are declared with @ray.remote(num_cpus=1) (controller.py#L1107, simple_storage.py#L900), so each one reserves a full CPU for as long as it lives. A framework that starts a controller and several storage units per job reserves that many CPUs per job before any training work runs. verl's v1 trainer starts one controller and 8 storage units by default, so 9 CPUs per job (see verl#7398, where a 16-CPU Ray cluster was left with 7 CPUs after the units came up). When jobs queue for a GPU, their TransferQueue actors already hold CPUs and can keep the jobs ahead of them from being scheduled; #164 is the same pressure showing up as a startup hang.

The reservation is far above the usage. One local Ray 2.57.0 run (torch 2.13.0, Python 3.12) with 8 storage units, 126 kv_put calls per second of two small fields plus a kv_list per batch, for 30 seconds, measured through psutil CPU times of the actor processes:

controller    0.148 cores (average)
8 units       0.012 cores each (average)

TransferQueueLockManager is already num_cpus=0, so a zero reservation for these actors is not a new pattern here.

Proposal

Read the CPU request from environment variables, TQ_CONTROLLER_NUM_CPUS and TQ_STORAGE_UNIT_NUM_CPUS, and pass it through .options(num_cpus=...) where the actors are created. Keep the default at 1 so nothing changes for existing users. I can send a PR with a test that checks the requested resources of the created actors.

Not covered

The numbers above come from a small local loop, not from a training run, so they show the order of magnitude and not the load of a real job.

主要语言
Python
星标
174
派生
57
平均合并
1 天 17 小时
30 天内合并 PR
15

环境准备

这个项目没有提供开发容器、Dockerfile 或贡献指南，环境需要你自己搭建：先看它的 README，通用步骤见我们的新手贡献指南。

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

Ascend/TransferQueue 的其他 Issue

查看 Ascend/TransferQueue 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。