nextflow-io/nextflow

Support container streaming for Google Batch

Open

#4 719 ouverte le 7 févr. 2024

Voir sur GitHub
 (3 commentaires) (0 réactions) (0 assignés)Groovy (784 forks)batch import
executor/google-batchgood first issue

Métriques du dépôt

Stars
 (3 382 stars)
Métriques de merge PR
 (Merge moyen 3j 18h) (38 PRs mergées en 30 j)

Description

Google Batch now supports container streaming (i.e. lazy loading container images) which improves startup time and reduces overhead, especially for fine-grained workloads.

https://cloud.google.com/batch/docs/use-image-streaming

For us to support it in Nextflow we just need to add a config option for enableImageStreaming.

Note that container streaming is currently only supported for container images in Artifact Registry. This is likely because they need to modify the container image format to support the lazy loading, so this requirement is unlikely to change.

Guide contributeur