Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Threadmill uses 4 to 5 times more Redis CPU than dramatiq for each task

Chiusa
  • Qualcosa non va?
#66 1 commento 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Una pull request collegata è già stata integrata.

  • #69 di @codingjoe — integrata

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
35/100
Tipo di issue
Refactoring
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python, redis
Ambito
backend, performance

Direzione di ricerca

Start by tracing the Redis fetch and ack paths, then inspect how the reaper and inspector depend on lease visibility. Compare the alternatives against the benchmark measurements and verify that the chosen design preserves lease behavior while reducing server CPU per task.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

perf real

Measured while the benchmark chart was explained. The chart ranking is not a story about the number of commands. Both queues issue about eleven Redis commands for each task (threadmill 11.03, dramatiq 11.15). The dramatiq number includes its five-command Results store. The difference is the cost of those commands.

threadmill dramatiq
commands per task (small payload) 11.03 11.15
server CPU per task 54.3 µs 11.7 µs
of which the script EVAL 43.4 µs 9.5 µs
server CPU per task, 5 KB payload 79.9 µs (+47%) 21.2 µs (+81%)

Two full JSON passes for each task explain the script cost of threadmill:

  • Fetch rewrites the task payload inside the script: HGET, cjson.decode, stamp RUNNING/last_attempted_at/worker_ids, cjson.encode, HSET. The dramatiq fetch uses LPOP and SADD for each id, plus one HMGET for each batch. It never touches the payload.
  • Ack writes the full serialized TaskResult (args, kwargs, errors, worker ids, return value) as one SET. It also writes a history ZADD, an eviction ZREMRANGEBYSCORE, and a telemetry PUBLISH.

Redis has one thread. Therefore this CPU load makes every other client wait. 2,000 tasks use about 108 ms of server time in threadmill against 23 ms in dramatiq.

Options, in order of cost:

  • Stamp the leased state outside the payload (a small hash field, or a separate status key or lease key). Then the fetch script stops the decode and encode of the payload.
  • Remove the telemetry PUBLISH for each task, or batch the events. The inspector only needs counts.
  • Amortize the history ZADD and the eviction. Do not run both for each ack.
  • Think again about the content of the stored result. The serialized TaskResult is the largest single write for each task.

The fetch-side change is the larger one. It also interacts with the lease visibility that the reaper and the inspector use. It is a design change, not a small correction. Filed out of scope for the prefetching branch, where it was found.

Lingua principale
Python
Stelle
21
Fork
2
Merge medio
12h 38m
PR unite (30g)
27

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.