skypilot-org/skypilot

[GCP] Move from `gsutil cp/rsync` to `gcloud storage cp/rsync`

Open

#8,457 opened on Dec 30, 2025

View on GitHub
 (3 comments) (0 reactions) (0 assignees)Python (311 forks)batch import
good first issuegood starter issues

Repository metrics

Stars
 (4,859 stars)
PR merge metrics
 (Avg merge 2d 23h) (146 merged PRs in 30d)

Description

According to https://docs.cloud.google.com/storage/docs/working-with-big-data, gcloud storage cp is much faster due to optimizations in automatic parallelism. We can migrate the usage gsutil cp in https://github.com/skypilot-org/skypilot/blob/e6a41b121ae27c33e1815ecf316858752dcc9982/sky/data/storage.py#L2370.

I manually tried a few file downloads with gcloud storage cp, it easily saturates the 300MB/s throughput, i.e. it worth a try.

Contributor guide

Research direction
Inspect the file sky/data/storage.py around line 2370 to understand the current gsutil cp/rsync usage. Replace with equivalent gcloud storage cp/rsync commands, ensuring the same behavior and optimizing for parallelism. Test with sample files to verify functionality and performance improvement.
Tech stack
pythonshell
Domain
backendcloud
Issue type
Refactor
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
2
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
GitPythonGoogle Cloud SDK
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
70