Collect does not allow awaitable sinks
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 48/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- python
- Domain
- stream-processing
Research direction
Start by locating the Python collect stream class and its flush method, then run the issue's awaitable-sink example. Check how flush propagates results to downstream sinks and compare that with regular synchronous sinks. Done means awaitable sinks complete correctly without breaking existing collection and flush behavior, with tests covering both cases.
Written by the indexing model from the issue text.
Description
The collect class does not allow for awaitable sinks.
Small example:
async def sink_to_something(x):
print(x)
return await asyncio.sleep(1)
source = streamz.Source()
collector = source.collect()
collector.sink(sink_to_something)
for i in range(10):
source.emit(i)
collector.flush()
Changing def flush in the collect class from:
@Stream.register_api()
class collect(Stream):
...
def flush(self, _=None):
out = tuple(self.cache)
metadata = list(self.metadata_cache)
self._emit(out, metadata)
...
To:
@Stream.register_api()
class collect(Stream):
...
def flush(self, _=None):
out = tuple(self.cache)
metadata = list(self.metadata_cache)
# change self._emit to self.emit (self.emit waits for awaitable results from downstream)
self.emit(out, metadata=metadata)
...
Fixed this problem, but I'm not sure if this has any drawbacks.
- Dominant language
- Python
- Stars
- 1.3k
- Forks
- 149
- Avg merge
- 17h 39m
- Merged PRs (30d)
- 1
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from python-streamz/streamz
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
python-streamz/streamz#481 · 4 comments ·
-
Combining the streamz.Stream.filenames() and streamz.Stream.from_textfile() using dask scatter? Open
Difficulty 5/5 Over a week Newbie friendliness 25/100
python-streamz/streamz#480 · 2 comments ·
-
Difficulty 5/5 Over a week Newbie friendliness 20/100
python-streamz/streamz#479 · 1 comment ·
-
Difficulty 5/5 Over a week Newbie friendliness 20/100
python-streamz/streamz#478 · 6 comments ·
-
Difficulty 4/5 3-5 days Newbie friendliness 10/100
python-streamz/streamz#476 · 17 comments · 2 reactions ·
All issues in python-streamz/streamz
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
syfoud/Simulated_Scepter#172 ·
-
A cancelled tests run makes the coverage comment workflow fail and reports it as a red check on main Openarea: ci bug perceived difficulty: 3
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Nitjsefnie-Harness-Commons/daedalus#921 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
EleutherAI/lm-evaluation-harness#4207 ·
-
Difficulty 1/5 Under an hour Newbie friendliness 92/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
ClickHouse/clickhouse-connect#1057 ·