Batch acquire: take up to N tasks per call across non-empty queues
还没有人认领这个 Issue。
评估
- 难度
- 5/5
- 预计耗时
- 一周以上
- 新手友好度
- 30/100
- Issue 类型
- 功能
- 描述清晰度
- 基本清楚
- 活跃度
- 活跃
- 技术栈
- python, redis
调研方向
先阅读 threadmill/backends/redis.py 和 acquire.lua,然后跟踪 WorkerProcess 和 WorkerThread,以了解当前按线程执行的 acquire 流程。定义 fetch-and-dispatch 如何处理 lease 计时、优雅关闭和预取任务,同时将 pop 和 move 保持在一次 script 调用中。完成标准是:可以从非空队列中获取最多 N 个任务,且不会有工作被搁置或过早失败。
由索引模型根据 Issue 内容生成。
描述
Split out of #48 during the fair multi-queue scheduling change (codingjoe-fair-multi-queue-scheduling).
defer: Batch acquire (fetch up to N tasks per call across non-empty queues). Needs executor fetch-and-dispatch; the lease would tick while tasks wait in-process. [threadmill/backends/redis.py]
Why
acquire() pops a single task per script call, so a process with --threads 4 performs four Redis round trips to fill its threads. A batch acquire that spreads up to N tasks across the non-empty queues would cut round trips and drain several queues in parallel.
What makes it non-trivial
WorkerProcessruns oneWorkerThreadper thread, each callingacquire()itself. A batch would need a fetch-and-dispatch step, since a single thread cannot execute N tasks at once.acquire.luamoves the task into the running set withdeadline = now + lease_ttl. Tasks prefetched into an in-process queue would burn lease budget while waiting, so the reaper could fail them, or the lease clock has to start at execution instead.- Graceful shutdown must return prefetched, unstarted tasks to the ready set instead of leaving them to be reaped as FAILED.
Pop and move must stay inside one script call.
- 主要语言
- Python
- 星标
- 12
- 派生
- 1
- 平均合并
- 1 天 1 小时
- 30 天内合并 PR
- 10
贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
codingjoe/threadmill 的其他 Issue
-
难度 2/5 半天 新手友好度 84/100
codingjoe/threadmill#52 ·
-
WorkerProcess passes None poll intervals to the backend, breaking acquire in the consumer thread 未关闭
难度 2/5 1-3 小时 新手友好度 82/100
codingjoe/threadmill#53 ·
-
难度 1/5 1 小时以内 新手友好度 90/100
codingjoe/threadmill#51 ·
-
难度 2/5 1-3 小时 新手友好度 78/100
codingjoe/threadmill#47 ·
-
难度 3/5 1-2 天 新手友好度 68/100
codingjoe/threadmill#55 · 1 条评论 ·
查看 codingjoe/threadmill 的全部 Issue
相似的 Issue
-
triage/confirmed
难度 2/5 1-3 小时 新手友好度 88/100
agentscope-ai/agentscope#2775 ·
-
comp/desktop P3 type/bug
难度 1/5 1 小时以内 新手友好度 92/100
NousResearch/hermes-agent#118866 ·
-
bug
难度 1/5 1 小时以内 新手友好度 90/100
apache/cloudstack#14222 ·
-
难度 2/5 1-3 小时 新手友好度 76/100
-
bug
难度 2/5 1-3 小时 新手友好度 82/100