Function determining dataset store backing incorrectly assumes new numpy array is always created by reader
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 68/100
Research direction
Read httomo/data/dataset_store.py at the linked reader and writer sections, then inspect determine_store_backing() in httomo/runner/dataset_store_backing.py. Trace the zero-padding and reslice cases, and verify that backing-size accounting reflects whether reader creation actually allocates a new NumPy array.
Written by the indexing model from the issue text.
Description
In the case of a section that is not the last section, a reader for section n+1 will be created from the writer used in section n. The creation of a reader only causes a new numpy array to be created if there is non-zero padding required for section n+1: https://github.com/DiamondLightSource/httomo/blob/f9bbccb3e915f4edfa51f77d5ab6fc195e4d5592/httomo/data/dataset_store.py#L301-L303
However, the determine_store_backing() function assumes that the creation of the reader will always create a new numpy array, so it accounts for the size of the numpy array even though it'll only exist in the case of non-zero padding: https://github.com/DiamondLightSource/httomo/blob/f9bbccb3e915f4edfa51f77d5ab6fc195e4d5592/httomo/runner/dataset_store_backing.py#L174-L178
Extra info
For some more info on why I think that there's no new numpy array created for the reader of section n+1 when there is zero padding: when that case occurs, the reader's self._data attribute is assigned to the writer's self._data attribute (which is a numpy array) and nothing else will happen to the reader's self._data: https://github.com/DiamondLightSource/httomo/blob/f9bbccb3e915f4edfa51f77d5ab6fc195e4d5592/httomo/data/dataset_store.py#L284
Meaning, I think that in the case of zero padding, the reader of section n+1 simply gets a reference to the numpy array from the writer of section n and nothing else (ie, no copy is made, no new array is created) so there's no reason for more memory to be allocated when creating the reader for section n+1.
Note that the above info is excluding the case of a reslice: in the case of a reslice, stuff will of course happen in the reslice algorithm to cause allocations, but that is separate from purely what the writer and reader are doing with the numpy arrays that represent the chunks associated with a section.
- Dominant language
- Python
- Stars
- 10
- Forks
- 5
- Avg merge
- 5d 17h
- Merged PRs (30d)
- 1
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from DiamondLightSource/httomo
-
ci
Difficulty 1/5 Under an hour Newbie friendliness 88/100
DiamondLightSource/httomo#714 ·
-
framework minor
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
DiamondLightSource/httomo#699 ·
-
documentation question
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
DiamondLightSource/httomo#697 ·
-
Difficulty 1/5 Under an hour Newbie friendliness 72/100
DiamondLightSource/httomo#609 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
DiamondLightSource/httomo#607 ·
All issues in DiamondLightSource/httomo
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
EleutherAI/lm-evaluation-harness#4207 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
ClickHouse/clickhouse-connect#1057 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
open-telemetry/sig-end-user#406 ·
-
bug ci good first issue
Difficulty 2/5 1-3 hours Newbie friendliness 88/100