nextflow-io/nextflow

How to set the computing resources (cpus, memory) in total submitted jobs

Open

#5,702 创建于 2025年1月23日

在 GitHub 查看
 (6 评论) (0 反应) (0 负责人)Groovy (784 fork)batch import
executor/slurmgood first issuestale

仓库指标

Star
 (3,382 star)
PR 合并指标
 (平均合并 3天 18小时) (30 天内合并 38 个 PR)

描述

Use cases: I have a SLURM HPC with small computing resources and a few users. What I want is that when l run a nextflow pipeline that it can limit the total cpus and memory. For example, I have 10 jobs, each job requires 4 cpus and 8.GB memory. However, I want to run only with overall cpus in total is 8 cpus and 16 GB memory. As a result, it will only submit the jobs with 2 jobs. It will wait on done to submit next 4 times. However, nextflow only limits cpus and memory in each process that I can not control this.

One possible solution is using the queueSize and resourceLimits. I can set the queue sizes to be 2, the max cpus and max memory is 4. However, it takes time if I have the too many small resources processes (1 cpus, 2GB memory), it can be run 4 processes each time however, due to the queue size, it can only submit with 2 processes. Is there any alternative solution that I may miss via the nextflow documentation?

贡献者指南