TPU.scp_to() has hardcoded 300s timeout — no way to override for large transfers

Open Beginner friendly
#14 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
67/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Quiet
Tech stack
google-cloud, python
Domain
cloud

Research direction

Start in tpuz/tpu.py at scp_to around line 360, then inspect _gcloud around line 116 and the related scp_from, push, and pull entry points. Trace how each transfer invokes gcloud and verify that the chosen timeout behavior is consistently exposed through the public methods without changing unrelated command handling.

Written by the indexing model from the issue text.

Description

TPU.scp_to() in tpuz/tpu.py:360 calls self._gcloud(args, timeout=300) with a hardcoded 300 second timeout. There's no way to pass a longer timeout through the public API, so any transfer larger than roughly 100–200 MB at typical residential upload speeds hits subprocess.TimeoutExpired even though the underlying gcloud scp is making forward progress.

Repro

from tpuz import TPU
tpu = TPU("some-tpu", accelerator="v6e-8", zone="europe-west4-a", preemptible=True)
tpu.up_queued(timeout_hours=2)
# flaxchat repo is ~1 GB (mostly models/ and exports/). scp_to will time out at 300s:
tpu.scp_to("/Users/me/code/flaxchat/", "/home/me/workdir/flaxchat")

Traceback

File ".../tpuz/tpu.py", line 360, in scp_to
    self._gcloud(args, timeout=300)
File ".../tpuz/tpu.py", line 116, in _gcloud
    result = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout)
...
subprocess.TimeoutExpired: Command '['gcloud', 'compute', 'tpus', 'tpu-vm', 'scp',
'/Users/me/code/flaxchat/', 'some-tpu:/home/me/workdir/flaxchat',
'--zone=europe-west4-a', '--worker=0', '--recurse']' timed out after 300 seconds

Fix

Add a timeout kwarg to scp_to/scp_from/push/pull:

def scp_to(self, local, remote, worker=0, timeout=300):
    ...
    self._gcloud(args, timeout=timeout)

Also consider defaulting to None (no timeout) for scp, since transfer time is inherently unpredictable and the operation is interruptible. Timeouts make sense for ssh where a hang indicates a dead command, not for bulk data movement.

Environment

  • tpuz==0.1.11
  • Python 3.13 on macOS (client), TPU v6e-8 europe-west4-a (remote)

Workaround

Use git clone on the TPU for code, and scp_to only for small untracked files:

tpu.ssh("git clone --depth 1 https://github.com/org/repo /home/me/workdir/repo", timeout=180)
tpu.scp_to("/local/untracked_script.py", "/home/me/workdir/repo/scripts/untracked_script.py")
Dominant language
Python
Stars
0
Forks
0
PR merge metrics
No merged PRs in 30d

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from mlnomadpy/tpuz

All issues in mlnomadpy/tpuz

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.