Threadmill uses 4 to 5 times more Redis CPU than dramatiq for each task
維護者通常 1 天內回覆
已經有一個關聯 PR 被合併了。
- #69 來自 @codingjoe —— 已合併
評估
- 難度
- 5/5
- 預估耗時
- 一週以上
- 新手友好度
- 35/100
- Issue 類型
- 重構
- 描述清晰度
- 基本清楚
- 活躍度
- 活躍
- 技術堆疊
- python, redis
- 領域
- backend, performance
研究方向
Start by tracing the Redis fetch and ack paths, then inspect how the reaper and inspector depend on lease visibility. Compare the alternatives against the benchmark measurements and verify that the chosen design preserves lease behavior while reducing server CPU per task.
由索引模型根據 Issue 內容生成。
描述
Measured while the benchmark chart was explained. The chart ranking is not a story about the number of commands. Both queues issue about eleven Redis commands for each task (threadmill 11.03, dramatiq 11.15). The dramatiq number includes its five-command Results store. The difference is the cost of those commands.
| threadmill | dramatiq | |
|---|---|---|
| commands per task (small payload) | 11.03 | 11.15 |
| server CPU per task | 54.3 µs | 11.7 µs |
of which the script EVAL |
43.4 µs | 9.5 µs |
| server CPU per task, 5 KB payload | 79.9 µs (+47%) | 21.2 µs (+81%) |
Two full JSON passes for each task explain the script cost of threadmill:
- Fetch rewrites the task payload inside the script:
HGET,cjson.decode, stampRUNNING/last_attempted_at/worker_ids,cjson.encode,HSET. The dramatiq fetch usesLPOPandSADDfor each id, plus oneHMGETfor each batch. It never touches the payload. - Ack writes the full serialized
TaskResult(args, kwargs, errors, worker ids, return value) as oneSET. It also writes a historyZADD, an evictionZREMRANGEBYSCORE, and a telemetryPUBLISH.
Redis has one thread. Therefore this CPU load makes every other client wait. 2,000 tasks use about 108 ms of server time in threadmill against 23 ms in dramatiq.
Options, in order of cost:
- Stamp the leased state outside the payload (a small hash field, or a separate status key or lease key). Then the fetch script stops the decode and encode of the payload.
- Remove the telemetry
PUBLISHfor each task, or batch the events. The inspector only needs counts. - Amortize the history
ZADDand the eviction. Do not run both for each ack. - Think again about the content of the stored result. The serialized
TaskResultis the largest single write for each task.
The fetch-side change is the larger one. It also interacts with the lease visibility that the reaper and the inspector use. It is a design change, not a small correction. Filed out of scope for the prefetching branch, where it was found.
- 主要語言
- Python
- 星號
- 21
- 分支
- 2
- 平均合併
- 12 小時 38 分鐘
- 30 天內合併 PR
- 27
環境準備
- 沒有 Dockerfile 或 Docker Compose 檔案
- 沒有 Pull Request 範本
- 閱讀貢獻指南
從這裡開始
- 先讀完整個 Issue，再讀專案的貢獻指南。
- 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
- Fork 儲存庫，在一個分支上完成修改。
- 送出 Pull Request，並在描述裡引用這個 Issue 編號。
相似的 Issue
-
難度 1/5 1 小時以內 新手友好度 72/100
slavakurilyak/awesome-ai-agents#845 ·
維護者通常 1 天內回覆
-
tests: pin that the atomic writers in utils/paths.py stage the temporary file in the target's directory可能已有人在做 關聯的 PR 仍在進行中或已合併。 未關閉difficulty:2 enhancement good first issue
難度 1/5 1-3 小時 新手友好度 92/100
MakazhanAlpamys/Soup#2178 ·
維護者通常 1 天內回覆
-
[BUG] dcg.waggr permutation p-values are folded twice, so near-zero scores get the smallest p-values未關閉
難度 2/5 1-3 小時 新手友好度 75/100
scverse/rapids-singlecell#854 ·
維護者通常 1 天內回覆
-
Slang input: int8_t4_packed and uint8_t4_packed are copied into CrossGL可能已有人在做 @ninglinLiu 於 1 天前認領。 未關閉area:frontend backend:slang bug direction:input
難度 2/5 1-3 小時 新手友好度 70/100
維護者通常 1 天內回覆
-
random_elements(unique=True) on an OrderedDict: ZeroDivisionError once only zero weights remain, AssertionError with use_weighting=False可能已有人在做 關聯的 PR 仍在進行中或已合併。 未關閉
難度 2/5 1-3 小時 新手友好度 85/100
維護者通常 13 天內回覆