Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

Threadmill uses 4 to 5 times more Redis CPU than dramatiq for each task

Cerrado
  • ¿Algo no está bien?
#66 1 comentario 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Ya se ha fusionado un pull request relacionado.

  • #69 de @codingjoe — fusionado

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
35/100
Tipo de issue
Refactorización
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
python, redis
Área
backend, performance

Línea de trabajo

Start by tracing the Redis fetch and ack paths, then inspect how the reaper and inspector depend on lease visibility. Compare the alternatives against the benchmark measurements and verify that the chosen design preserves lease behavior while reducing server CPU per task.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

perf real

Measured while the benchmark chart was explained. The chart ranking is not a story about the number of commands. Both queues issue about eleven Redis commands for each task (threadmill 11.03, dramatiq 11.15). The dramatiq number includes its five-command Results store. The difference is the cost of those commands.

threadmill dramatiq
commands per task (small payload) 11.03 11.15
server CPU per task 54.3 µs 11.7 µs
of which the script EVAL 43.4 µs 9.5 µs
server CPU per task, 5 KB payload 79.9 µs (+47%) 21.2 µs (+81%)

Two full JSON passes for each task explain the script cost of threadmill:

  • Fetch rewrites the task payload inside the script: HGET, cjson.decode, stamp RUNNING/last_attempted_at/worker_ids, cjson.encode, HSET. The dramatiq fetch uses LPOP and SADD for each id, plus one HMGET for each batch. It never touches the payload.
  • Ack writes the full serialized TaskResult (args, kwargs, errors, worker ids, return value) as one SET. It also writes a history ZADD, an eviction ZREMRANGEBYSCORE, and a telemetry PUBLISH.

Redis has one thread. Therefore this CPU load makes every other client wait. 2,000 tasks use about 108 ms of server time in threadmill against 23 ms in dramatiq.

Options, in order of cost:

  • Stamp the leased state outside the payload (a small hash field, or a separate status key or lease key). Then the fetch script stops the decode and encode of the payload.
  • Remove the telemetry PUBLISH for each task, or batch the events. The inspector only needs counts.
  • Amortize the history ZADD and the eviction. Do not run both for each ack.
  • Think again about the content of the stored result. The serialized TaskResult is the largest single write for each task.

The fetch-side change is the larger one. It also interacts with the lease visibility that the reaper and the inspector use. It is a design change, not a small correction. Filed out of scope for the prefetching branch, where it was found.

Lenguaje dominante
Python
Estrellas
21
Forks
2
Merge medio
12 h 38 min
PR fusionados (30 d)
27

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.