Inconsistent creation of blocks by test loader compared to real loader

Open
#579 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
55/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Stale
Tech stack
numpy, python
Domain
testing

Research direction

Start in tests/testing_utils.py at the test loader implementation around lines 86-98, then inspect httomo/runner/task_runner.py around lines 158-161 and the failing test in tests/runner/test_task_runner.py. Update the loader so each produced block owns copied array data, and run the affected task-runner tests to confirm multiple blocks no longer fail after block data is deleted.

Written by the indexing model from the issue text.

Description

testing

The real loader creates a new numpy array (into which a subset of the input data in the hdf5 file is copied): https://github.com/DiamondLightSource/httomo/blob/4e4b9d9bdf6e69d97aff840554ec9630b048e2c4/httomo/loaders/standard_tomo_loader.py#L171

which then eventually gets put into a DataSetBlock: https://github.com/DiamondLightSource/httomo/blob/4e4b9d9bdf6e69d97aff840554ec9630b048e2c4/httomo/loaders/standard_tomo_loader.py#L304-L313

The test loader defined in testing_utils.py which can be configured to produce one or more mock blocks:

  • takes in a DataSetBlock
  • and then uses slices of the numpy array within that DataSetBlock

to create new blocks: https://github.com/DiamondLightSource/httomo/blob/4e4b9d9bdf6e69d97aff840554ec9630b048e2c4/tests/testing_utils.py#L86-L98

Ie, if running httomo within a test that wants multiple blocks to be read from the test loader, the loader is providing multiple blocks containing references to a single numpy array. But if running httomo not in a test, the real loader provides multiple blocks where each block has its own data, rather than referencing a single central numpy array.

This can cause different behaviour between tests and real runs if more than one block is in a section due to the del block.data line in the task runner: https://github.com/DiamondLightSource/httomo/blob/4e4b9d9bdf6e69d97aff840554ec9630b048e2c4/httomo/runner/task_runner.py#L158-L161

For the case of multiple blocks in a test produced by the test loader, the reference count of the single central numpy array drops to 0 due to the del block.data, and subsequent attempts to read blocks fails due to the single central numpy array not being there anymore:

  File "/httomo/tests/runner/test_task_runner.py", line 735, in test_minimum_block_length_passed_to_intermediate_data_wrapper
    t._execute_section(s[0])
  File "/httomo/httomo/runner/task_runner.py", line 160, in _execute_section
    del block.data
        ^^^^^^^^^^
  File "/opt/conda/lib/python3.12/unittest/mock.py", line 823, in __delattr__
    raise AttributeError(name)
AttributeError: data

The test loader should be modified to:

  • take a plain numpy array rather than a DataSetBlock
  • copy data from that numpy array into a new numpy, before putting the new numpy array into a new DataSetBlock

in order to be consistent with how a real loader works when multiple blocks are loaded from it.

Dominant language
Python
Stars
10
Forks
5
Avg merge
5d 17h
Merged PRs (30d)
1

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from DiamondLightSource/httomo

All issues in DiamondLightSource/httomo

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.