BiocParallel fail to start with MPI

Start with the reported foo.R script and compare the interactive mpiexec -np 1 R invocation with the sbatch command using mpiexec -np 1 Rscript foo.R. Read the SnowParam MPI configuration and Rmpi startup behavior, then reproduce the MPI_INIT timeout in the batch environment. Done means identifying the cause of the differing startup behavior and documenting a verified resolution.

Hello Everyone,
We are having trouble running BiocParallel within our SLURM cluster environment.

The foo.R script we are trying to run is

library("BiocParallel")
library("Rmpi")

param <- SnowParam(workers = 3, type = "MPI")
FUN <- function(i) system("hostname", intern=TRUE)
bplapply(1:6, FUN, BPPARAM = param)

If we request an interactive job allocation, for example with salloc -p mpi -N 2 -n 4 -t 1:00:00 and then start R with:
mpiexec -np 1 R --no-save and run the above script from this interactive shell we have as expected:

> library("BiocParallel")
library("BiocParallel")
> library("Rmpi")
library("Rmpi")
> param <- SnowParam(workers = 3, type = "MPI")
param <- SnowParam(workers = 3, type = "MPI")
> FUN <- function(i) system("hostname", intern=TRUE)
FUN <- function(i) system("hostname", intern=TRUE)
> bplapply(1:6, FUN, BPPARAM = param)
bplapply(1:6, FUN, BPPARAM = param)
	3 slaves are spawned successfully. 0 failed.
[[1]]
[1] "compute-a-16-21"

[[2]]
[1] "compute-a-16-21"

[[3]]
[1] "compute-a-16-22"

[[4]]
[1] "compute-a-16-22"

[[5]]
[1] "compute-a-16-22"

[[6]]
[1] "compute-a-16-22"

However if we try to run the same R script from within a sbatch job with:

#!/bin/bash

#SBATCH -p mpi
#SBATCH -N 2
#SBATCH -n 4
#SBATCH -t 2:00:00

mpiexec -np 1 Rscript foo.R  # or R CMD BATCH foo.R 

The execution hangs for several seconds and eventually fails with the MPI error:

[compute-a-16-21:10780] OPAL ERROR: Timeout in file base/pmix_base_fns.c at line 193
--------------------------------------------------------------------------
It looks like MPI_INIT failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during MPI_INIT; some of which are due to configuration or environment
problems.  This failure appears to be an internal failure; here's some
additional information (which may only be relevant to an Open MPI
developer):

  ompi_dpm_dyn_init() failed
  --> Returned "Timeout" (-15) instead of "Success" (0)
--------------------------------------------------------------------------
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)

Does anyone have any idea of why the primary R process is failing to start the other tasks?

Thank you
Raffaele