Dask leaks memory with Batched Kafka and cudf
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 28/100
- Issue type
- Bug
- Clarity
- Needs clarification
- Activity status
- Stale
- Tech stack
- kafka, python
- Domain
- performance, stream-processing
Research direction
No files or tests are named. Start by reproducing the long-running Batched Kafka workload with the listed Dask, distributed, and cuDF versions and worker configuration, then measure memory against batch size over time. Done means the source of the worker memory growth is identified and a regression test or verified fix is available.
Written by the indexing model from the issue text.
Description
Dask workers' memory shooting up gradually for long running jobs and eventually job crashes when the memory of workers exceeds 80%(or around). Refer to the image below:
Dask is able to process the data at the input rate(600 mbps) and certainly not keeping processed futures in memory as we can infer from the image. I am using 10 secs window for reading messages from Kafka. So, the size of each batch would be ~6GB. But the workers seem to use much higher memory than that.
Dask configuration:
2 nodes (24 core CPU and 1 T4 GPU)
8 workers and 3 threads on each node
dask - 1.2.2
distributed - 1.28.1
cudf - 0.8
- Dominant language
- Python
- Stars
- 1.3k
- Forks
- 149
- PR merge metrics
- No merged PRs in 30d
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from python-streamz/streamz
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
python-streamz/streamz#481 · 4 comments ·
-
Combining the streamz.Stream.filenames() and streamz.Stream.from_textfile() using dask scatter? Open
Difficulty 5/5 Over a week Newbie friendliness 25/100
python-streamz/streamz#480 · 2 comments ·
-
Difficulty 5/5 Over a week Newbie friendliness 20/100
python-streamz/streamz#479 · 1 comment ·
-
Difficulty 5/5 Over a week Newbie friendliness 20/100
python-streamz/streamz#478 · 6 comments ·
-
Difficulty 4/5 3-5 days Newbie friendliness 10/100
python-streamz/streamz#476 · 17 comments · 2 reactions ·
All issues in python-streamz/streamz
Similar issues
-
documentation help wanted
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
simonw/sqlite-utils#872 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100