Bogdanp/dramatiq

Feature Request: iterate over a group as actors complete

Open

#278 opened on Feb 27, 2020

View on GitHubContributor guide
 (3 comments) (0 reactions) (0 assignees)Python (277 forks)batch import
enhancementhelp wanted

Repository metrics

Stars
 (3,884 stars)
PR merge metrics
 (Avg merge 20d 20h) (10 merged PRs in 30d)

Description

concurrent.futures (and probably other Python stdlib stuff) has the idea of iterating over results from concurrent processing as they are completed:

https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.as_completed

I'm currently using the Group.get_results() method to iterate over a group of results, but it would be nice to be able to do that as the results come back instead of in the order they were queued. That way, if I have a network request hang at the very beginning, I can get other work done while that action times out and is retried.

Contributor guide

Research direction
Study Dramatiq's Group.get results() implementation and the concurrent.futures.as completed function. Design an as completed method for Group that yields results as they become available, likely by polling result backends (e.g., Redis) for each message ID.
Tech stack
python
Domain
backend
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
2
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
Half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
PythonDramatiq
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
60