Enhancement: allow iterating signals in chunks of dataframes
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 30/100
Research direction
Start with the to_dataframe() implementation introduced in pull request 380 and review how it currently loads waveform signals. Define an approach for lazy, chunked dataframe processing that avoids loading the full signal into memory; done means large records such as those from MIMIC-III can be processed in memory-sized parts.
Written by the indexing model from the issue text.
Description
I'm using the new to_dataframe() function that was implemented in https://github.com/MIT-LCP/wfdb-python/pull/380
One issue that I'm seeing is that when loading some of the waveform signals from https://physionet.org/content/mimic3wdb-matched/1.0/ using to_dataframe() it eats up a lot of memory. Specifically, on the machine I'm running on which has 96gb of memory, reading the record and calling to_dataframe runs out of memory.
I would like to lazy load the signal data into a chunked dataframe which would allow me to process the waveform signals in parts that could fit into memory, rather than loading it all into memory.
- Dominant language
- Jupyter Notebook
- Stars
- 853
- Forks
- 322
- PR merge metrics
- No merged PRs in 30d
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from MIT-LCP/wfdb-python
-
Difficulty 4/5 3-5 days Newbie friendliness 45/100
MIT-LCP/wfdb-python#568 ·
-
Difficulty 3/5 1-2 days Newbie friendliness 48/100
MIT-LCP/wfdb-python#557 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 58/100
MIT-LCP/wfdb-python#554 ·
-
WFDB path ignored Open
Difficulty 4/5 3-5 days Newbie friendliness 35/100
MIT-LCP/wfdb-python#545 ·
-
Difficulty 5/5 Over a week Newbie friendliness 30/100
MIT-LCP/wfdb-python#540 ·
All issues in MIT-LCP/wfdb-python
Similar issues
-
bug triage
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
fivetran/great_expectations#12253 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
correction metadata
Difficulty 1/5 Under an hour Newbie friendliness 78/100
acl-org/acl-anthology#10071 · 1 comment ·
-
data:css needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
mdn/browser-compat-data#30618 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
basedosdados/pipelines#2095 ·