nextflow-io/nextflow

How to set the computing resources (cpus, memory) in total submitted jobs

Open

#5,702 opened on Jan 23, 2025

View on GitHub
 (6 comments) (0 reactions) (0 assignees)Groovy (3,382 stars) (784 forks)batch import
executor/slurmgood first issuestale

Description

Use cases: I have a SLURM HPC with small computing resources and a few users. What I want is that when l run a nextflow pipeline that it can limit the total cpus and memory. For example, I have 10 jobs, each job requires 4 cpus and 8.GB memory. However, I want to run only with overall cpus in total is 8 cpus and 16 GB memory. As a result, it will only submit the jobs with 2 jobs. It will wait on done to submit next 4 times. However, nextflow only limits cpus and memory in each process that I can not control this.

One possible solution is using the queueSize and resourceLimits. I can set the queue sizes to be 2, the max cpus and max memory is 4. However, it takes time if I have the too many small resources processes (1 cpus, 2GB memory), it can be run 4 processes each time however, due to the queue size, it can only submit with 2 processes. Is there any alternative solution that I may miss via the nextflow documentation?

Contributor guide

Tech stack
groovy
Domain
infrastructureperformance
Issue type
research
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
2
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
mostly clear
Prerequisites
Nextflow basicsSLURM HPCresource limiting concepts
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
60
Research direction
Review Nextflow documentation on resourceLimits and queueSize in the official docs. Check the repository's existing issues and pull requests for any related feature requests (e.g., issues about global resource limits). Participate in the discussion to clarify the desired behavior. Look at the source code for process scheduling to understand current limitations. Files to examine: nextflow source files related to executor and process scheduling (e.g., modules/nextflow/src/main/groovy/nextflow/executor/).