[Q] stream asyncio dataframes to_dataframe usage
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 25/100
- Issue type
- Bug
- Clarity
- Needs clarification
- Activity status
- Stale
- Tech stack
- pandas, python
- Domain
- data-engineering, stream-processing
Research direction
Start by reproducing the supplied coroutine using Stream.to_dataframe, source.emit(readField()), and sdf.tail(). Inspect the to_dataframe path and asynchronous emission behavior to determine why the resulting dataframe is empty. Done means the behavior is explained and, if it is a defect, covered by a focused regression test.
Written by the indexing model from the issue text.
Description
Hey guys,
Great work here, I just have a problem understanding how we translate a stream of dataframe so we can use tail() etc.
I have a coroutine, that emits dataframes to my stream.
from streamz import Stream
from tornado.ioloop import IOLoop
@gen.coroutine
def f():
source = Stream(asynchronous=True) # tell the stream we're working asynchronously
example = pd.DataFrame({'x': []})
sdf = source.to_dataframe(example=example)
for x in range(10):
yield gen.sleep(0.1)
yield source.emit(readField())
sdf.tail()
sdf.cumsum()
print(sdf)
IOLoop().run_sync(f)
with:
@asyncio.async
def readField():
"""
asynchronously - Blocking IO operation
"""
df = pd.DataFrame({'x': [SomeValues]})
return df
As output I get:
DataFrame - elements like:
Empty DataFrame
Columns: [x]
Index: []
I am not sure if thats the way we are looking to use the to_DataFrame.
I basically want to send my dataframes to a stream, translate it to bigger dataframes (window etc.) and perform some dynamic printing on the resulting frames. But somehow it seems, that my data is not transmitted to my stream.
Any idea/help would be appreciated.
- Dominant language
- Python
- Stars
- 1.3k
- Forks
- 149
- Avg merge
- 17h 39m
- Merged PRs (30d)
- 1
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from python-streamz/streamz
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
python-streamz/streamz#481 · 4 comments ·
-
Combining the streamz.Stream.filenames() and streamz.Stream.from_textfile() using dask scatter? Open
Difficulty 5/5 Over a week Newbie friendliness 25/100
python-streamz/streamz#480 · 2 comments ·
-
Difficulty 5/5 Over a week Newbie friendliness 20/100
python-streamz/streamz#479 · 1 comment ·
-
Difficulty 5/5 Over a week Newbie friendliness 20/100
python-streamz/streamz#478 · 6 comments ·
-
Difficulty 4/5 3-5 days Newbie friendliness 10/100
python-streamz/streamz#476 · 17 comments · 2 reactions ·
All issues in python-streamz/streamz
Similar issues
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
canonical/paas-charm#368 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
tech debt
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
Difficulty 1/5 Under an hour Newbie friendliness 90/100
StevenBlack/hosts#3256 ·
-
Difficulty 1/5 Under an hour Newbie friendliness 90/100
qualcomm/qai-appbuilder#275 ·