Silent fail / exit with streamz >= 0.4.0 when pipeline is being run in a daemon process
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 25/100
- Issue type
- Bug
- Clarity
- Needs clarification
- Activity status
- Stale
- Tech stack
- elasticsearch, kafka, python
- Domain
- data-engineering, stream-processing
Research direction
Reproduce the pipeline under python-daemon while comparing streamz 0.3.0 and 0.4.0, using Kafka input and Elasticsearch output. Investigate whether the Tornado processes introduced in later versions cause the daemon to stop after its first message. Done means the daemon continues processing messages and exposes an error if processing fails.
Written by the indexing model from the issue text.
Description
I've been using streamz to consume events from kafka. For that I built a pipeline which I run in a daemon process, managed by the python daemon package (https://pypi.org/project/python-daemon/). The pipeline ultimately writes to some elasticsearch instance.
This worked great with streamz version 0.3.0 with which I started developing.
Now I wanted to update to v0.5.0 and discovered that while I can still perfectly run the streamz pipeline in the current python process, the daemon running the pipeline suddenly stops without any further detectable error in the pipeline itself after consuming the first message. I downgraded to v0.4.0, then v0.3.0 and observed that the fail is introduced by the 0.3.0 -> 0.4.0 step.
Does anybody has an idea what introduced the problem at hand?
Is there an easy way to fix the issue with the later versions?
.. I somehow suspect that it has something to do with the tornado processes started ..?
- Dominant language
- Python
- Stars
- 1.3k
- Forks
- 149
- Avg merge
- 17h 39m
- Merged PRs (30d)
- 1
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from python-streamz/streamz
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
python-streamz/streamz#481 · 4 comments ·
-
Combining the streamz.Stream.filenames() and streamz.Stream.from_textfile() using dask scatter? Open
Difficulty 5/5 Over a week Newbie friendliness 25/100
python-streamz/streamz#480 · 2 comments ·
-
Difficulty 5/5 Over a week Newbie friendliness 20/100
python-streamz/streamz#479 · 1 comment ·
-
Difficulty 5/5 Over a week Newbie friendliness 20/100
python-streamz/streamz#478 · 6 comments ·
-
Difficulty 4/5 3-5 days Newbie friendliness 10/100
python-streamz/streamz#476 · 17 comments · 2 reactions ·
All issues in python-streamz/streamz
Similar issues
-
essnmx good first issue
Difficulty 1/5 Under an hour Newbie friendliness 95/100
-
[Feature] 奇物选择添加优先级 Open
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
syfoud/Simulated_Scepter#174 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Giskard-AI/giskard-oss#2840 · 1 comment ·
-
A claim comment carrying the issue number is silently declined while the workflow reports success Openarea: repo bug perceived difficulty: 2
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
yeti-platform/yeti#1380 ·