nextflow-io/nextflow

Support container streaming for Google Batch

Open

#4,719 建立於 2024年2月7日

在 GitHub 查看
 (3 留言) (0 反應) (0 負責人)Groovy (3,382 star) (784 fork)batch import
executor/google-batchgood first issue

描述

Google Batch now supports container streaming (i.e. lazy loading container images) which improves startup time and reduces overhead, especially for fine-grained workloads.

https://cloud.google.com/batch/docs/use-image-streaming

For us to support it in Nextflow we just need to add a config option for enableImageStreaming.

Note that container streaming is currently only supported for container images in Artifact Registry. This is likely because they need to modify the container image format to support the lazy loading, so this requirement is unlikely to change.

貢獻者指南

技術棧
無
領域
cloudinfrastructure
議題類型
feature
難度面向新貢獻者的預計實作難度，1 表示很小改動，5 表示專家級工作。
3
預計時間有經驗貢獻者完成調查、實作、測試並準備 pull request 的粗略時間範圍。
1-3 hours
活動狀態議題目前的可參與程度：新鮮、活躍、陳舊、阻塞或等待維護者輸入。
fresh
清晰度議題是否清楚說明預期改動、驗收標準和下一步。
clear
前置要求
Understanding of Nextflow configurationBasic knowledge of Google Batch
新手友善度1-100 的估計分數，表示該議題對首次貢獻者的友善程度。
60
研究方向
The issue requests adding a configuration option `enableImageStreaming` for Google Batch. Start by examining Nextflow's configuration loading mechanism, likely in files related to Google Batch executor or general configuration. Review the linked Google Cloud documentation on image streaming. Check comments on the issue for any additional context or implementation ideas. There are no linked PRs yet, so the implementation is open. The change involves adding a boolean config key and passing it to the Google Batch API when creating a job.