Speed tests with a Stream?
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 25/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- python
- Domain
- performance, stream-processing
Research direction
Start by running the two standalone benchmark scripts, comparing the generator loop with Stream.from_iterable() and sink_to_list(). Check the streamz documentation and implementation of source.start() and sink_to_list() to determine whether the polling loop is required. Done means documenting the expected performance and the correct standalone-script usage, or identifying a reproducible defect.
Written by the indexing model from the issue text.
Description
I am trying to compare the speed of operation when a Stream is introduced into the code. These are both scripts that need to be run.
A 'plain' Python example:
from timeit import default_timer as timer
LIMIT = 100000
def sequencer(limit=LIMIT):
num = 0
while num < limit:
yield num
num += 1
L = []
start = timer()
for i in sequencer():
L.append(i)
end = timer()
print(end - start)
This takes about 17 milliseconds to run.
An attempt to replicate the above with streamz:
from timeit import default_timer as timer
from streamz import Stream
LIMIT = 100000
def sequencer(limit=LIMIT):
num = 0
while num < limit:
yield num
num += 1
source = Stream.from_iterable(sequencer())
L = source.sink_to_list()
start = timer()
source.start()
while True:
if len(L) >= LIMIT:
break
end = timer()
print(end - start)
This takes about 1200 milliseconds to run. Is this the expected slowdown (two orders-of-magnitude)?
I am not sure though if the streamz code is correct? Its based off of the examples in the docs, but those examples all seem geared towards use in the shell rather than in standalone scripts. If you omit the while True: section, then no data ends up in the output list?
- Dominant language
- Python
- Stars
- 1.3k
- Forks
- 149
- PR merge metrics
- No merged PRs in 30d
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from python-streamz/streamz
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
python-streamz/streamz#481 · 4 comments ·
-
Combining the streamz.Stream.filenames() and streamz.Stream.from_textfile() using dask scatter? Open
Difficulty 5/5 Over a week Newbie friendliness 25/100
python-streamz/streamz#480 · 2 comments ·
-
Difficulty 5/5 Over a week Newbie friendliness 20/100
python-streamz/streamz#479 · 1 comment ·
-
Difficulty 5/5 Over a week Newbie friendliness 20/100
python-streamz/streamz#478 · 6 comments ·
-
Difficulty 4/5 3-5 days Newbie friendliness 10/100
python-streamz/streamz#476 · 17 comments · 2 reactions ·
All issues in python-streamz/streamz
Similar issues
-
documentation help wanted
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
simonw/sqlite-utils#872 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100