Problem forwarding batchtools resources to individual futures

Start with the BatchtoolsFuture() entry point and the future() call in the reproducible example, then inspect how resources move into the batchtools submission. Use sge_batchtools.tmpl and the SGE example to reproduce the missing slots value. Done means resources = list(slots = 2) reaches the submission so the job requests two slots.

Summary

When I supply a resources list to a batchtools future, the resources seem to be ignored. I was alerted to this via targets, https://github.com/ropensci/targets/discussions/562#discussioncomment-1304379 and https://github.com/ropensci/targets/discussions/632 (cc @wresch, @sdechaumet)

Reproducible example

Using an SGE cluster with this template:

#!/bin/bash
#$ -cwd
#$ -j y
#$ -o <%= log.file %>
#$ -V
#$ -N <%= job.name %>
#$ -pe smp <%= resources$slots %>
module load R/4.0.3
Rscript -e 'batchtools::doJobCollection("<%= uri %>")'
exit 0

R console:

library(future)
library(future.batchtools)
plan(batchtools_sge, template = "sge_batchtools.tmpl")
f <- future(Sys.sleep(5), resources = list(slots = 2))
# Error: Fatal error occurred: 101. Command 'qsub' produced exit code 2. Output: 'Unable to read script file because of error: ERROR! -pe option must have range as 2nd argument'

When I run debug(BatchtoolsFuture) on this example, I see that resources = list(slots = 2) is not actually passed to the BatchtoolsFuture() function. BatchtoolsFuture() appears to receive the default resources = list().

Expected behavior

An SGE job with 2 slots should be submitted.

Session information

I am using the following commits of future and future.batchtools:

• https://github.com/HenrikBengtsson/future/commit/344a97f1cc85283095a43fe01cfabf3fb33fdf70
• https://github.com/HenrikBengtsson/future.batchtools/commit/ea3b86cbc6b30f62f78cfd9ef7c5a64f5d42e683

> sessionInfo()
R version 4.0.3 (2020-10-10)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Red Hat Enterprise Linux

Matrix products: default
BLAS/LAPACK: <CENSORED>

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] future.batchtools_0.10.0-9000 future_1.22.1-9000           

loaded via a namespace (and not attached):
 [1] parallelly_1.28.1 magrittr_2.0.1    hms_1.1.0         progress_1.2.2   
 [5] rappdirs_0.3.3    debugme_1.1.0     R6_2.5.1          brew_1.0-6       
 [9] rlang_0.4.11      fansi_0.5.0       globals_0.14.0    tools_4.0.3      
[13] parallel_4.0.3    checkmate_2.0.0   data.table_1.14.0 utf8_1.2.2       
[17] withr_2.4.2       ellipsis_0.3.2    base64url_1.4     digest_0.6.27    
[21] tibble_3.1.4      lifecycle_1.0.0   crayon_1.4.1      fs_1.5.0         
[25] vctrs_0.3.8       batchtools_0.9.15 codetools_0.2-18  stringi_1.7.4    
[29] pillar_1.6.2      compiler_4.0.3    backports_1.2.1   prettyunits_1.1.1
[33] listenv_0.8.0     pkgconfig_2.0.3