Batch acquire: take up to N tasks per call across non-empty queues
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 30/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- python, redis
- Domain
- backend, distributed-systems
Research direction
Read threadmill/backends/redis.py and acquire.lua first, then trace WorkerProcess and WorkerThread to understand the current per-thread acquire flow. Define how fetch-and-dispatch handles lease timing, graceful shutdown, and prefetched tasks, while keeping pop and move in one script call. Done means up to N tasks can be acquired across non-empty queues without stranded or prematurely failed work.
Written by the indexing model from the issue text.
Description
Split out of #48 during the fair multi-queue scheduling change (codingjoe-fair-multi-queue-scheduling).
defer: Batch acquire (fetch up to N tasks per call across non-empty queues). Needs executor fetch-and-dispatch; the lease would tick while tasks wait in-process. [threadmill/backends/redis.py]
Why
acquire() pops a single task per script call, so a process with --threads 4 performs four Redis round trips to fill its threads. A batch acquire that spreads up to N tasks across the non-empty queues would cut round trips and drain several queues in parallel.
What makes it non-trivial
WorkerProcessruns oneWorkerThreadper thread, each callingacquire()itself. A batch would need a fetch-and-dispatch step, since a single thread cannot execute N tasks at once.acquire.luamoves the task into the running set withdeadline = now + lease_ttl. Tasks prefetched into an in-process queue would burn lease budget while waiting, so the reaper could fail them, or the lease clock has to start at execution instead.- Graceful shutdown must return prefetched, unstarted tasks to the ready set instead of leaving them to be reaped as FAILED.
Pop and move must stay inside one script call.
- Dominant language
- Python
- Stars
- 12
- Forks
- 1
- Avg merge
- 19h 11m
- Merged PRs (30d)
- 12
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from codingjoe/threadmill
-
Difficulty 2/5 Half a day Newbie friendliness 84/100
codingjoe/threadmill#52 ·
-
WorkerProcess passes None poll intervals to the backend, breaking acquire in the consumer thread Open
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
codingjoe/threadmill#53 ·
-
Difficulty 1/5 Under an hour Newbie friendliness 90/100
codingjoe/threadmill#51 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
codingjoe/threadmill#47 ·
-
Difficulty 3/5 1-2 days Newbie friendliness 68/100
codingjoe/threadmill#55 · 1 comment ·
All issues in codingjoe/threadmill
Similar issues
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
canonical/paas-charm#368 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
tech debt
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
Difficulty 1/5 Under an hour Newbie friendliness 90/100
StevenBlack/hosts#3256 ·
-
Difficulty 1/5 Under an hour Newbie friendliness 90/100
qualcomm/qai-appbuilder#275 ·