nextflow-io/nextflow

Support reservation when provisioning VM/ accelerators on Google Batch

Open

#4,262 建立於 2023年9月1日

在 GitHub 查看
 (1 留言) (0 反應) (0 負責人)Groovy (784 fork)batch import
executor/google-batchgood first issuestale

倉庫指標

Star
 (3,382 star)
PR 合併指標
 (平均合併 16天 9小時) (30 天內合併 54 個 PR)

描述

New feature

Google Cloud supports provisioning Compute Engine instances using reservations (e.g. dedicated compute resources CPUs/GPUs/Memory etc). https://cloud.google.com/compute/docs/instances/reservations-consume

This is a configuration available via Google LifeSciences API and Google Batch.

reservation stringIf specified, VMs will consume only the specified reservation. If not specified (default), VMs will consume any applicable reservation.
reservation stringIf specified, the VM will only be allocated inside the matching reservation. It will fail if the VM parameters don't match the reservation.

Usage scenario

For example a user may configure a reservation for GPU resources to ensure dedicated resources are available to be provisioned (instead of waiting for GPUs to be available on demand).

Reservations can be made and applied automatically but it can be useful to be able to explicitly specify a reservation in case it's only used by a certain user or for a certain project.

Then the nextflow user would want to be able to configure a particular Nextflow process to use that dedicated reservation when provisioning a task that uses a GPU.

Suggest implementation

Allow for a Nextflow process to specify reservation configuration along with machineType/accelerators etc.

process foo {
    accelerator 4, type: 'nvidia-tesla-k80'
    reservation: 'userx-gpu-reservation'

    script:
    """
    your_gpu_enabled --command --line
    """
}

貢獻者指南