PsiStore operations on very big stores leads to running out of memory.
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 35/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- csharp
- Domain
- performance, stream-processing
Research direction
Start with the PsiStore.Copy call shown in the report and inspect how the stream filter processes very large stores. Reproduce the copy using a large store or the reported 426 GB scenario, then determine whether memory remains bounded while copying only the selected streams. Done means the operation completes without exhausting available memory and does not require real-time playback.
Written by the indexing model from the issue text.
Description
I have a 2 hour length, 426 GB PsiStore. As part of my post study process, I wanted to remove a stream and save the store in another location for deep storage. The code is basically as following:
var outputStorePath = Path.Combine(@"D:\Hallway-Recording-Clean", datasetID, sessionID);
PsiStore.Copy((partition.StoreName, partition.StorePath), (partition.StoreName,outputStorePath), null, s => s.Name.Contains("ir"), false);
When I run the code, the memory usage slowly goes up and eventually my system run out of memory (I have 64 GB of RAM) and the application crashes. I think I could just change the pipeline to use real time which should work but it means I have to wait 2 hours for it to finish.
Is there any ways to make the system run as fast as possible without reading the store so fast that it runs out of memory? Thanks!
- Dominant language
- C#
- Stars
- 574
- Forks
- 104
- Avg merge
- 19m
- Merged PRs (30d)
- 1
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from microsoft/psi
-
Difficulty 4/5 3-5 days Newbie friendliness 25/100
-
Difficulty 1/5 1-3 hours Newbie friendliness 42/100
-
feature request help wanted
Difficulty 4/5 3-5 days Newbie friendliness 35/100
-
help wanted
Difficulty 4/5 3-5 days Newbie friendliness 25/100
-
Difficulty 5/5 Over a week Newbie friendliness 15/100
Similar issues
-
type/automation type/tech-debt
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
t/bug
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
ci-failure-cause test-failure
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
area:auth FE mvp P3
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
klasolsson81/jobbliggaren#1788 ·