Inconsistent creation of blocks by test loader compared to real loader
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 55/100
Research direction
Start in tests/testing_utils.py at the test loader implementation around lines 86-98, then inspect httomo/runner/task_runner.py around lines 158-161 and the failing test in tests/runner/test_task_runner.py. Update the loader so each produced block owns copied array data, and run the affected task-runner tests to confirm multiple blocks no longer fail after block data is deleted.
Written by the indexing model from the issue text.
Description
The real loader creates a new numpy array (into which a subset of the input data in the hdf5 file is copied): https://github.com/DiamondLightSource/httomo/blob/4e4b9d9bdf6e69d97aff840554ec9630b048e2c4/httomo/loaders/standard_tomo_loader.py#L171
which then eventually gets put into a DataSetBlock: https://github.com/DiamondLightSource/httomo/blob/4e4b9d9bdf6e69d97aff840554ec9630b048e2c4/httomo/loaders/standard_tomo_loader.py#L304-L313
The test loader defined in testing_utils.py which can be configured to produce one or more mock blocks:
- takes in a
DataSetBlock - and then uses slices of the numpy array within that
DataSetBlock
to create new blocks: https://github.com/DiamondLightSource/httomo/blob/4e4b9d9bdf6e69d97aff840554ec9630b048e2c4/tests/testing_utils.py#L86-L98
Ie, if running httomo within a test that wants multiple blocks to be read from the test loader, the loader is providing multiple blocks containing references to a single numpy array. But if running httomo not in a test, the real loader provides multiple blocks where each block has its own data, rather than referencing a single central numpy array.
This can cause different behaviour between tests and real runs if more than one block is in a section due to the del block.data line in the task runner: https://github.com/DiamondLightSource/httomo/blob/4e4b9d9bdf6e69d97aff840554ec9630b048e2c4/httomo/runner/task_runner.py#L158-L161
For the case of multiple blocks in a test produced by the test loader, the reference count of the single central numpy array drops to 0 due to the del block.data, and subsequent attempts to read blocks fails due to the single central numpy array not being there anymore:
File "/httomo/tests/runner/test_task_runner.py", line 735, in test_minimum_block_length_passed_to_intermediate_data_wrapper
t._execute_section(s[0])
File "/httomo/httomo/runner/task_runner.py", line 160, in _execute_section
del block.data
^^^^^^^^^^
File "/opt/conda/lib/python3.12/unittest/mock.py", line 823, in __delattr__
raise AttributeError(name)
AttributeError: data
The test loader should be modified to:
- take a plain numpy array rather than a
DataSetBlock - copy data from that numpy array into a new numpy, before putting the new numpy array into a new
DataSetBlock
in order to be consistent with how a real loader works when multiple blocks are loaded from it.
- Dominant language
- Python
- Stars
- 10
- Forks
- 5
- Avg merge
- 5d 17h
- Merged PRs (30d)
- 1
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from DiamondLightSource/httomo
-
ci
Difficulty 1/5 Under an hour Newbie friendliness 88/100
DiamondLightSource/httomo#714 ·
-
framework minor
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
DiamondLightSource/httomo#699 ·
-
documentation question
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
DiamondLightSource/httomo#697 ·
-
Difficulty 1/5 Under an hour Newbie friendliness 72/100
DiamondLightSource/httomo#609 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
DiamondLightSource/httomo#607 ·
All issues in DiamondLightSource/httomo
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100