Fatal error in internal_Allreduce
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 42/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- c
- Domain
- distributed-systems
Research direction
The report points to MPIDI_POSIX_mpi_release_gather_release timing out during MPI_Allreduce with GPU-Aware MPI at large node counts, but does not identify a source file or reproducer. Start by tracing that POSIX release-gather path and checking existing Allreduce tests for GPU-Aware MPI coverage. Done means reproducing or isolating the timeout and verifying a fix with the reported multi-node conditions.
Written by the indexing model from the issue text.
Description
The following are tested on Aurora and many jobs failed as follows:
Application: kynema-sgf on top of AMRexX
MPICH versions: mpich/prd/5.0.0.aurora_test.b76b754, aurora_test_7980
Number of nodes: 128, 256, 512 and 1024
Runs without using GPU-Aware MPI from the application didn't fail.
The failure happened at random places and the following errors are from one of the cases:
Abort(137440271) on node 3130 (rank 3130 in comm 496): Fatal error in internal_Allreduce: Other MPI error, error stack:
internal_Allreduce(152)......................: MPI_Allreduce(sendbuf=MPI_IN_PLACE, recvbuf=0x7ffca2851248, count=1, MPI_DOUBLE, MPI_SUM, comm=0x84000001) failed
MPID_Allreduce(483)..........................:
MPIDI_Allreduce_allcomm_composition_json(400):
MPIDI_Allreduce_intra_composition_alpha(568).:
MPIDI_POSIX_mpi_reduce(681)..................:
MPIDI_POSIX_mpi_reduce_release_gather(235)...:
MPIDI_POSIX_mpi_release_gather_release(205)..: Progress TIMEOUT due to MPIR_CVAR_PROGRESS_TIMEOUT set to 600 (seconds). Reset it to 0 to disable progress TIMEOUT.
/opt/aurora/26.181.0/spack/unified/1.1.1/install/linux-x86_64/mpich-5.0.0.aurora_test.b76b754-juntkad/lib/libmpi.so.12(+0x27dad36) [0x14691fd57d36]
/opt/aurora/26.181.0/spack/unified/1.1.1/install/linux-x86_64/mpich-5.0.0.aurora_test.b76b754-juntkad/lib/libmpi.so.12(+0x1f12a7e) [0x14691f48fa7e]
/opt/aurora/26.181.0/spack/unified/1.1.1/install/linux-x86_64/mpich-5.0.0.aurora_test.b76b754-juntkad/lib/libmpi.so.12(PMPI_Allreduce+0xde3) [0x14691d5db943]
./kynema_sgf() [0xe76d7b]
- Dominant language
- C
- Stars
- 688
- Forks
- 337
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 51
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from pmodels/mpich
-
Difficulty 2/5 1-3 hours Newbie friendliness 60/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
Maintainers usually reply within 1 day
-
Difficulty 4/5 3-5 days Newbie friendliness 55/100
Maintainers usually reply within 1 day
-
Difficulty 3/5 1-2 days Newbie friendliness 68/100
Maintainers usually reply within 1 day
-
aurora
Difficulty 4/5 3-5 days Newbie friendliness 48/100
pmodels/mpich#8066 · 1 comment ·
Maintainers usually reply within 1 day
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
JustAResearcher/gpu-unlock-ada#124 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
-
inflate: total_in uses plain += on c_ulong (32-bit on Windows), panics in debug for streams > 4 GiBOpen
Difficulty 1/5 Under an hour Newbie friendliness 85/100
trifectatechfoundation/zlib-rs#570 · 1 comment ·
-
vim.filetype.match() prefixes a URL buffer name with the cwd before passing it to filetype functionsOpen
Difficulty 2/5 1-3 hours Newbie friendliness 73/100
Maintainers usually reply within 1 day
-
compilation errorOpen
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
COVESA/dlt-daemon#925 · 1 reaction ·