Handle numpy `_ArrayMemoryError` in reslice `np.concatenate()` call to avoid deadlock in multi-process runs

Open
#602 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
45/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Stale
Tech stack
numpy, python
Domain
distributed-systems

Research direction

Start at httomo/data/mpiutil.py line 67, where the reslice calls np.concatenate(), and reproduce the issue with two MPI processes using the referenced test data on ws582. Trace what happens when only one rank raises _ArrayMemoryError. Done means the multi-process run does not deadlock and returns control to the terminal when the allocation fails.

Written by the indexing model from the issue text.

Description

bug

For single process runs there is no issue, but with multi-process runs with MPI a deadlock can occur if one or more but not all processes run into an _ArrayMemoryError at this line in the reslice: https://github.com/DiamondLightSource/httomo/blob/8ccf66304dfb7af0a77495e4bcf974c92f324953/httomo/data/mpiutil.py#L67

The output looks like the following, where some printing has been inserted to show the rank of the processes that run into the error:

Pipeline has been separated into 3 sections
See the full log file at: /scratch/yousef/outdata/17-07-2025_15_56_54_output/user.log
Running loader (pattern=projection): standard_tomo...
    Finished loader: standard_tomo (httomo) Took 10000.46ms
Section 0 (pattern=projection) with the following methods:
    data_reducer (httomolib)
    find_center_vo (httomolibgpu)
    remove_outlier (httomolibgpu)
    normalize (httomolibgpu)
     0%|          | 0/4 [00:26<?, ?block/s]
    25%|##5       | 1/4 [00:52<01:45, 35.00s/block]
    50%|#####     | 2/4 [01:07<00:57, 28.89s/block]
    75%|#######5  | 3/4 [01:15<00:22, 22.70s/block]
    --->The center of rotation is 1275.0
    Finished processing last block
Rank 0 got an error: Unable to allocate 20.6 GiB for an array with shape (5529600000,) and data type float32

Note how only one of the two ranks report the error, and the fact that control is not given back to the terminal after this printing, the terminal is "hanging" and Ctrl-C needs to be done to kill both MPI processes. The specific rank running into the error can vary across different runs, but with a specific machine (ws582) and specific test data this deadlock can be consistently reproduced running httomo with 2 MPI processes.

Dominant language
Python
Stars
10
Forks
5
Avg merge
5d 17h
Merged PRs (30d)
1

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from DiamondLightSource/httomo

All issues in DiamondLightSource/httomo

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.