nextflow-io/nextflow

Support container streaming for Google Batch

Aperta

#4719 aperta il 7 feb 2024

 (3 commenti) (0 reazioni) (0 assegnatari)Groovy (784 fork)batch import
executor/google-batchgood first issue

Metriche repository

Star
 (3382 stelle)
Metriche merge PR
 (Merge medio 16g 9h) (54 PR mergiate in 30 g)

Descrizione

Google Batch now supports container streaming (i.e. lazy loading container images) which improves startup time and reduces overhead, especially for fine-grained workloads.

https://cloud.google.com/batch/docs/use-image-streaming

For us to support it in Nextflow we just need to add a config option for enableImageStreaming.

Note that container streaming is currently only supported for container images in Artifact Registry. This is likely because they need to modify the container image format to support the lazy loading, so this requirement is unlikely to change.

Guida contributor