skypilot-org/skypilot

[GCP] Move from `gsutil cp/rsync` to `gcloud storage cp/rsync`

Open

#8457 opened on Dec 30, 2025

View on GitHub
 (3 comments) (0 reactions) (0 assignees)Python (4,859 stars) (311 forks)batch import
good first issuegood starter issues

Description

According to https://docs.cloud.google.com/storage/docs/working-with-big-data, gcloud storage cp is much faster due to optimizations in automatic parallelism. We can migrate the usage gsutil cp in https://github.com/skypilot-org/skypilot/blob/e6a41b121ae27c33e1815ecf316858752dcc9982/sky/data/storage.py#L2370.

I manually tried a few file downloads with gcloud storage cp, it easily saturates the 300MB/s throughput, i.e. it worth a try.

Contributor guide

Tech stack
pythonshellgcp
Domain
cloudperformance
Issue type
performance
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
fresh
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
Basic PythonGCP storage concepts
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
70
Research direction
Examine the file sky/data/storage.py at line 2370 to understand how gsutil cp/rsync is currently used. Then, replace those commands with gcloud storage cp/rsync following the reference mentioned in the issue (https://docs.cloud.google.com/storage/docs/working with big data). After making the change, test the performance improvement by downloading files with gcloud storage cp and compare throughput. No additional changes to the logic are required.