Bound the batch acquire count because --prefetch-count is unbounded
維護者通常 1 天內回覆
還沒有人認領這個 Issue。
評估
- 難度
- 5/5
- 預估耗時
- 一週以上
- 新手友好度
- 35/100
- Issue 類型
- 功能
- 描述清晰度
- 需要釐清
- 活躍度
- 活躍
- 技術堆疊
- python, redis
研究方向
Start with the --prefetch-count option, the worker's queue.Queue and acquire call, then trace RedisTaskBackend.acquire through acquire.lua and the README prefetching section. Confirm how the count reaches the queue and Redis script, and use the real redis-server reproduction described in the issue. Done requires a maintainer-selected ceiling to be consistently enforced and documented.
由索引模型根據 Issue 內容生成。
描述
Deferred from the task-prefetching review (crew run) rather than fixed in that branch, because choosing a ceiling is a maintainer decision.
What
count is unbounded end to end:
--prefetch-countaccepts any int>= 1(the only validation is the lower bound), defaulting to4 × threads- it becomes
queue.Queue(maxsize=count)andacquire(..., count=count)in the worker RedisTaskBackend.acquireforwards it asARGV[8], andacquire.luapops up to that many tasks inside one atomicEVAL
Why it matters
- one worker can lock the entire backlog under a single lease, starving every other worker until the shared deadline expires, at which point the reaper records the whole batch FAILED with no retry
- process memory grows with the number of buffered payloads
- the single-threaded Redis server is blocked for work proportional to
countper call (oneZPOPMIN/HGET/cjson/ZADD/HSETpass per task, inside one script execution)
Reproduced against a real redis-server: one call with ARGV[8]=6 moved all six ready tasks to the running set with an identical deadline and one worker id, leaving the ready set empty.
Options
- cap the effective count at a documented maximum (or at a new dedicated backend option)
- clamp
prefetch_countto the queue's backlog bound, if one exists - derive the ceiling from
lease_ttland the expected worst-case task duration, so a buffer cannot outlive its lease
Related: the buffer dwell counts against lease_ttl (the reserve happens at fetch time), so an unbounded buffer also makes false lease expiries more likely. That ceiling is documented in the branch's README section on prefetching.
- 主要語言
- Python
- 星號
- 21
- 分支
- 2
- 平均合併
- 12 小時 38 分鐘
- 30 天內合併 PR
- 27
環境準備
- 沒有 Dockerfile 或 Docker Compose 檔案
- 沒有 Pull Request 範本
- 閱讀貢獻指南
從這裡開始
- 先讀完整個 Issue，再讀專案的貢獻指南。
- 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
- Fork 儲存庫，在一個分支上完成修改。
- 送出 Pull Request，並在描述裡引用這個 Issue 編號。
相似的 Issue
-
難度 1/5 1 小時以內 新手友好度 85/100
521xueweihan/HelloGitHub#4115 ·
-
get_lldp_neighbors_detail: query asks for 'max-address' (typo for 'mac-address'), so remote_chassis_id is always empty可能已有人在做 @Jah-yee 今天認領。 未關閉
難度 1/5 1 小時以內 新手友好度 88/100
-
lesson-submission pending-review
難度 1/5 1 小時以內 新手友好度 74/100
Ikalus1988/MisakaNet#3899 ·
維護者通常 1 天內回覆
-
難度 2/5 1-3 小時 新手友好度 68/100
aws-samples/sample-per-model-bedrock#549 ·
維護者通常 1 天內回覆
-
難度 1/5 1 小時以內 新手友好度 90/100