nextflow-io/nextflow

Support container streaming for Google Batch

Open

#4,719 opened on Feb 7, 2024

View on GitHub
 (3 comments) (0 reactions) (0 assignees)Groovy (3,382 stars) (784 forks)batch import
executor/google-batchgood first issue

Description

Google Batch now supports container streaming (i.e. lazy loading container images) which improves startup time and reduces overhead, especially for fine-grained workloads.

https://cloud.google.com/batch/docs/use-image-streaming

For us to support it in Nextflow we just need to add a config option for enableImageStreaming.

Note that container streaming is currently only supported for container images in Artifact Registry. This is likely because they need to modify the container image format to support the lazy loading, so this requirement is unlikely to change.

Contributor guide

Tech stack
None
Domain
cloudinfrastructure
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
fresh
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
Understanding of Nextflow configurationBasic knowledge of Google Batch
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
60
Research direction
The issue requests adding a configuration option `enableImageStreaming` for Google Batch. Start by examining Nextflow's configuration loading mechanism, likely in files related to Google Batch executor or general configuration. Review the linked Google Cloud documentation on image streaming. Check comments on the issue for any additional context or implementation ideas. There are no linked PRs yet, so the implementation is open. The change involves adding a boolean config key and passing it to the Google Batch API when creating a job.
Support container streaming for Google Batch · nextflow-io/nextflow#4719 | Good First Issue