Parallel streams with buffers
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 35/100
- Issue type
- Feature
- Clarity
- Needs clarification
- Activity status
- Stale
- Tech stack
- python
- Domain
- stream-processing
Research direction
Start with the provided Stream(asynchronous=True) example and trace how map, buffer, and sink schedule their work. Compare the stream's behavior with the stated 30-second runtime and expected 21-second runtime; done means the issue's intended parallel download and processing behavior is clearly established, including whether Dask is appropriate.
Written by the indexing model from the issue text.
Description
In a simple use case like downloading files and process them on a single machine, how could one achieve parallelization of downloads and processes with buffers?
Example:
import time
from streamz import Stream
from tornado.ioloop import IOLoop
def download_file(file_id: int):
time.sleep(1)
print(f"Downloaded file: {file_id}")
return file_id
def process_file(file_id: int):
time.sleep(2)
print(f"Processed file : {file_id}")
return file_id
async def streamz_run():
s = Stream(asynchronous=True)
s.map(download_file).buffer(4).sink(process_file)
for i in range(10):
await s.emit(i)
if __name__ == '__main__':
start = time.time()
IOLoop().run_sync(streamz_run)
print(f"Streamz run took: {time.time() - start}s")
The download_file is properly buffered but not executed at the same time as process_file. The whole thing takes ~30s to run while we could expect 21s with parallel downloads/processes. Is using Dask the intended way in that case?
- Dominant language
- Python
- Stars
- 1.3k
- Forks
- 149
- PR merge metrics
- No merged PRs in 30d
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from python-streamz/streamz
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
python-streamz/streamz#481 · 4 comments ·
-
Combining the streamz.Stream.filenames() and streamz.Stream.from_textfile() using dask scatter? Open
Difficulty 5/5 Over a week Newbie friendliness 25/100
python-streamz/streamz#480 · 2 comments ·
-
Difficulty 5/5 Over a week Newbie friendliness 20/100
python-streamz/streamz#479 · 1 comment ·
-
Difficulty 5/5 Over a week Newbie friendliness 20/100
python-streamz/streamz#478 · 6 comments ·
-
Difficulty 4/5 3-5 days Newbie friendliness 10/100
python-streamz/streamz#476 · 17 comments · 2 reactions ·
All issues in python-streamz/streamz
Similar issues
-
documentation help wanted
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
simonw/sqlite-utils#872 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100