Hacktoberfest 2026: as issues que os mantenedores marcaram para outubro, abertas e boas para iniciantes. Ver issues do Hacktoberfest

Threadmill uses 4 to 5 times more Redis CPU than dramatiq for each task

Fechada
  • Algo errado?
#66 1 comentário 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 1 dia

Um pull request relacionado já foi integrado.

  • #69 de @codingjoe — integrado

Avaliação

Dificuldade
5/5
Tempo estimado
Mais de uma semana
Facilidade para iniciantes
35/100
Tipo de issue
Refatoração
Clareza
Razoavelmente clara
Status de atividade
Ativa
Stack de tecnologia
python, redis
Domínio
backend, performance

Direção de pesquisa

Start by tracing the Redis fetch and ack paths, then inspect how the reaper and inspector depend on lease visibility. Compare the alternatives against the benchmark measurements and verify that the chosen design preserves lease behavior while reducing server CPU per task.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

perf real

Measured while the benchmark chart was explained. The chart ranking is not a story about the number of commands. Both queues issue about eleven Redis commands for each task (threadmill 11.03, dramatiq 11.15). The dramatiq number includes its five-command Results store. The difference is the cost of those commands.

threadmill dramatiq
commands per task (small payload) 11.03 11.15
server CPU per task 54.3 µs 11.7 µs
of which the script EVAL 43.4 µs 9.5 µs
server CPU per task, 5 KB payload 79.9 µs (+47%) 21.2 µs (+81%)

Two full JSON passes for each task explain the script cost of threadmill:

  • Fetch rewrites the task payload inside the script: HGET, cjson.decode, stamp RUNNING/last_attempted_at/worker_ids, cjson.encode, HSET. The dramatiq fetch uses LPOP and SADD for each id, plus one HMGET for each batch. It never touches the payload.
  • Ack writes the full serialized TaskResult (args, kwargs, errors, worker ids, return value) as one SET. It also writes a history ZADD, an eviction ZREMRANGEBYSCORE, and a telemetry PUBLISH.

Redis has one thread. Therefore this CPU load makes every other client wait. 2,000 tasks use about 108 ms of server time in threadmill against 23 ms in dramatiq.

Options, in order of cost:

  • Stamp the leased state outside the payload (a small hash field, or a separate status key or lease key). Then the fetch script stops the decode and encode of the payload.
  • Remove the telemetry PUBLISH for each task, or batch the events. The inspector only needs counts.
  • Amortize the history ZADD and the eviction. Do not run both for each ack.
  • Think again about the content of the stored result. The serialized TaskResult is the largest single write for each task.

The fetch-side change is the larger one. It also interacts with the lease visibility that the reaper and the inspector use. It is a design change, not a small correction. Filed out of scope for the prefetching branch, where it was found.

Linguagem predominante
Python
Estrelas
21
Forks
2
Merge médio
12h 38min
PRs com merge (30d)
27

Preparar o ambiente

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Issues semelhantes

Mais issues de Python

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.