nextflow-io/nextflow

Support container streaming for Google Batch

Open

#4719 aperta il 7 feb 2024

Vedi su GitHub
 (3 commenti) (0 reazioni) (0 assegnatari)Groovy (784 fork)batch import
executor/google-batchgood first issue

Metriche repository

Star
 (3382 star)
Metriche merge PR
 (Merge medio 3g 18h) (38 PR mergiate in 30 g)

Descrizione

Google Batch now supports container streaming (i.e. lazy loading container images) which improves startup time and reduces overhead, especially for fine-grained workloads.

https://cloud.google.com/batch/docs/use-image-streaming

For us to support it in Nextflow we just need to add a config option for enableImageStreaming.

Note that container streaming is currently only supported for container images in Artifact Registry. This is likely because they need to modify the container image format to support the lazy loading, so this requirement is unlikely to change.

Guida contributor