Problem forwarding batchtools resources to individual futures

Inizia dal punto di ingresso BatchtoolsFuture() e dalla chiamata future() nell’esempio riproducibile, quindi esamina come le risorse arrivano all’invio di batchtools. Usa sge_batchtools.tmpl e l’esempio SGE per riprodurre il valore mancante di slots. Il lavoro è completato quando resources = list(slots = 2) raggiunge l’invio, così il job richiede due slots.

Summary

When I supply a resources list to a batchtools future, the resources seem to be ignored. I was alerted to this via targets, https://github.com/ropensci/targets/discussions/562#discussioncomment-1304379 and https://github.com/ropensci/targets/discussions/632 (cc @wresch, @sdechaumet)

Reproducible example

Using an SGE cluster with this template:

#!/bin/bash
#$ -cwd
#$ -j y
#$ -o <%= log.file %>
#$ -V
#$ -N <%= job.name %>
#$ -pe smp <%= resources$slots %>
module load R/4.0.3
Rscript -e 'batchtools::doJobCollection("<%= uri %>")'
exit 0

R console:

library(future)
library(future.batchtools)
plan(batchtools_sge, template = "sge_batchtools.tmpl")
f <- future(Sys.sleep(5), resources = list(slots = 2))
# Error: Fatal error occurred: 101. Command 'qsub' produced exit code 2. Output: 'Unable to read script file because of error: ERROR! -pe option must have range as 2nd argument'

When I run debug(BatchtoolsFuture) on this example, I see that resources = list(slots = 2) is not actually passed to the BatchtoolsFuture() function. BatchtoolsFuture() appears to receive the default resources = list().

Expected behavior

An SGE job with 2 slots should be submitted.

Session information

I am using the following commits of future and future.batchtools:

• https://github.com/HenrikBengtsson/future/commit/344a97f1cc85283095a43fe01cfabf3fb33fdf70
• https://github.com/HenrikBengtsson/future.batchtools/commit/ea3b86cbc6b30f62f78cfd9ef7c5a64f5d42e683

> sessionInfo()
R version 4.0.3 (2020-10-10)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Red Hat Enterprise Linux

Matrix products: default
BLAS/LAPACK: <CENSORED>

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] future.batchtools_0.10.0-9000 future_1.22.1-9000           

loaded via a namespace (and not attached):
 [1] parallelly_1.28.1 magrittr_2.0.1    hms_1.1.0         progress_1.2.2   
 [5] rappdirs_0.3.3    debugme_1.1.0     R6_2.5.1          brew_1.0-6       
 [9] rlang_0.4.11      fansi_0.5.0       globals_0.14.0    tools_4.0.3      
[13] parallel_4.0.3    checkmate_2.0.0   data.table_1.14.0 utf8_1.2.2       
[17] withr_2.4.2       ellipsis_0.3.2    base64url_1.4     digest_0.6.27    
[21] tibble_3.1.4      lifecycle_1.0.0   crayon_1.4.1      fs_1.5.0         
[25] vctrs_0.3.8       batchtools_0.9.15 codetools_0.2-18  stringi_1.7.4    
[29] pillar_1.6.2      compiler_4.0.3    backports_1.2.1   prettyunits_1.1.1
[33] listenv_0.8.0     pkgconfig_2.0.3