Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

setting 1 job per worker core after chunking the elements

Open
#39 14 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
32/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Stale
Tech stack
r
Domain
distributed-systems, hpc

Research direction

Start with the future::tweak(... batchtools_sge ...) plan and furrr::future_map example in the issue, then inspect how resources[["slots"]] is passed through batchtools.sge-mrg.tmpl. Done means determining whether native chunking can allocate 16 elements across workers and cores, and recording the supported behavior or requirements.

Written by the indexing model from the issue text.

Description

I am using an sge template.

Is there a native way in future plan to have elements of a list be allocated to each available core in preset number of workers (ie chunking the list)?

ie if i have a workers with 8 cores and a vector (x) of length 16, i want future to send each element to a core on the workers to run in parallel. (assuming i am sending out jobs from the remote master)

sge <- future::tweak(
    future.batchtools::batchtools_sge,
    template = 'batchtools.sge-mrg.tmpl',
    workers = 2,
    resources = list(slots = 1)
  )
  
  future::plan(list(sge))
  
  x <- vector('list',16)

  ret <- furrr::future_map(x, .f = foo)
  

#!/bin/bash

## The name of the job, can be anything, simply used when displaying the list of running jobs
#$ -N <%= job.name %>

## Combining output/error messages into one file
#$ -j y

## Giving the name of the output log file
#$ -o <%= log.file %>

## One needs to tell the queue system to use the current directory as the working directory
## Or else the script may fail as it will execute in your top level home directory /home/username
#$ -cwd

## Use environment variables
#$ -V

## Use correct queue
##$ -q all.q

#$ -pe smp <%= resources[["slots"]] %>

## Export value of DEBUGME environemnt var to slave
export DEBUGME=<%= Sys.getenv("DEBUGME") %>

<%= sprintf("export OMP_NUM_THREADS=%i", resources$omp.threads) -%>
<%= sprintf("export OPENBLAS_NUM_THREADS=%i", resources$blas.threads) -%>
<%= sprintf("export MKL_NUM_THREADS=%i", resources$blas.threads) -%>


Rscript -e 'batchtools::doJobCollection("<%= uri %>")'
exit 0

cc @wlandau

Dominant language
R
Stars
87
Forks
10
No merged PRs in 30d

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from futureverse/future.batchtools

All issues in futureverse/future.batchtools

Similar issues

More R issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.