Memory Exception When Merging Large Volumes of Waveform Data Files Using wrdb.wrsamp()
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 25/100
- Issue type
- Bug
- Clarity
- Needs clarification
- Activity status
- Stale
- Tech stack
- python
- Domain
- data-engineering
Research direction
Start by reproducing the reported memory exception with wrdb.wrsamp() on multiple .dat files and inspect how the current read, combine, and write flow handles the arrays. Done means establishing whether incremental writing is supported and identifying the implementation and regression test needed if it is not; the issue names no files or tests.
Written by the indexing model from the issue text.
Description
I'm trying to merge multiple waveform data (.dat) files into a single file. I'm using the wrdb.wrsamp() function for this task. The total number of files is approximately 10,000 and each one has 3 channels. I've tried several times, but every attempt results in a memory exception, requiring more than 40GB of memory. I'm unsure if I am doing something incorrect.
I've been unable to find a method to write the files incrementally. My current approach is to read each sample, combine all signals into an array, and write them. While this works fine with a small number of files, I'm having difficulties when it comes to larger datasets. Each file contains over 6 minutes of data.
Any assistance insights or suggestions on this matter would be highly appreciated.
- Dominant language
- Jupyter Notebook
- Stars
- 853
- Forks
- 322
- PR merge metrics
- No merged PRs in 30d
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from MIT-LCP/wfdb-python
-
Difficulty 4/5 3-5 days Newbie friendliness 45/100
MIT-LCP/wfdb-python#568 ·
-
Difficulty 3/5 1-2 days Newbie friendliness 48/100
MIT-LCP/wfdb-python#557 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 58/100
MIT-LCP/wfdb-python#554 ·
-
WFDB path ignored Open
Difficulty 4/5 3-5 days Newbie friendliness 35/100
MIT-LCP/wfdb-python#545 ·
-
Difficulty 5/5 Over a week Newbie friendliness 30/100
MIT-LCP/wfdb-python#540 ·
All issues in MIT-LCP/wfdb-python
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
openml/openml-python#1749 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 83/100
-
documentation
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
desihub/desisurveyops#486 · 3 comments ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
scverse/spatialdata#1256 ·