Non-blocking, local evaluation of future_lapply()?

Inizia leggendo gli entry point future_lapply(), futureOf(), resolved() e plan(multiprocess) citati nel report. Riproduci entrambi gli esempi su Linux, confrontando i piani multiprocess diretti e annidati con mclapply. Il lavoro è completo quando il comportamento di blocco previsto è documentato o corretto, mentre la distribuzione multicore e la risoluzione dei future si comportano in modo coerente.

Is future_lapply() intended to be non-blocking? I thought it was, but it's not working for me.

Here is a minimal reproducible example (with a simple function that serves no purpose except to take ~10 seconds to evaluate on my machine so I could test blocking and multicore behavior). I'm running Linux, so my understanding is that multiprocess here implies multicore. (Output shown as comments.)

# future_lapply() blocks, even in multiprocess. You can see that resolved()
# does not get evaluated until future_lapply() has finished.
# But it successfully distributes this across 2 cores.
plan(multiprocess)
date()
### [1] "Fri May 24 14:29:51 2019"
a <- future_lapply(rep(50000000, 2), function(i) rnorm(i)*rnorm(i))
resolved(futureOf(a))
### Error: Future (‘a’) not found in environment ‘R_GlobalEnv’: ‘a’
date()
### [1] "Fri May 24 14:30:03 2019"
head(a[[1]])
### [1] -1.2233054  0.1918043 -0.4650852  0.5335259 -0.2493615 -0.8267408
date()
### [1] "Fri May 24 14:30:03 2019"

Note that I get an error when trying to call resolved(futureOf(a)), because a has already been resolved before it gets called, and no future exists because it was implicit. The calls to date() are in there to show that it blocked for 12 seconds while it was evaluating future_lapply().

Based on your response in #1, I tried assigning future_lapply as an implicit future and using nested multiprocess evaluation (though that was intended for someone running SGE). This also blocks, and now future_lapply() is evaluated sequentially, not on multiple cores. I watched process allocation happening, but you can see that it now blocks for twice as long: 24 seconds.

# Try nesting it in an implicit future call. Still blocks. But now this gets
# evaluated sequentially rather than distributed across 2 cores.
plan(list(multiprocess, multiprocess))
date()
### "Fri May 24 14:37:15 2019"
a %<-% future_lapply(rep(50000000, 2), function(i) rnorm(i)*rnorm(i))
resolved(futureOf(a))
### [1] FALSE
date()
### [1] "Fri May 24 14:37:15 2019"
head(a[[1]])
### [1] -0.9747142 -0.1586670 -0.1039924  4.5885303 -0.4779900  0.3339059
date()
### [1] "Fri May 24 14:37:39 2019"

I ran into this issue because I'm trying to switch from mclapply to future_lapply (for the great parallel RNG!), and I do get non-blocking behavior using an implicit future with mclapply (resolved() and date() are both executed immediately after the mclapply call without blocking):

# This works as expected: Setting mc.cores explicitly does distribute across
# multiple cores, and it's non-blocking.
library(parallel)
plan(multiprocess)
date()
### [1] "Fri May 24 14:51:31 2019"
a %<-% mclapply(rep(50000000, 2), function(i) rnorm(i)*rnorm(i), mc.cores=2)
resolved(futureOf(a))
### [1] FALSE
date()
### [1] "Fri May 24 14:51:31 2019"
head(a[[1]])
### [1]  0.968440961 -0.015869658  0.321415096 -0.609809739  0.005155251
date()
### [1] "Fri May 24 14:51:44 2019"

Incidentally, if I replace the call to explicitly set mc.cores=2 with mc.cores=future::availableCores(), I still get non-blocking behavior, but now mclapply gets executed sequentially instead of being distributed across cores. (If I run mc.cores=future::availableCores() I get 16.) I'm not sure if this is a bug, and I didn't explore it thoroughly, but it's not what I expected.

Thanks so much for your help and for all your work to bring R into the future!