skypilot-org/skypilot

[Spot] An option for keeping failed spot job for a while before termination

Open

#1163 opened on Sep 11, 2022

View on GitHub
 (4 comments) (0 reactions) (0 assignees)Python (4,859 stars) (311 forks)batch import
P1feature-requesthelp wanted

Description

When something wrong happens with the spot job, it would be nice to be able to log into the spot cluster to take a look at the problem. As proposed by @lhqing, having an option like --keep-minutes-after-error 60 for the spot launch can be useful for debugging.

Contributor guide

Tech stack
python
Domain
clouddevops
Issue type
feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
2
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
fresh
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
PythonSkyPilot CLI basics
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
70
Research direction
The issue requests a keep minutes after error flag for spot jobs. Look at the spot job launch code in the SkyPilot repository, specifically the spot module (e.g., sky/spot/). Check existing command line argument parsing in sky/cli.py or similar. Implement the flag to delay termination after failure, likely modifying the spot job termination logic. Verify the proposal in the issue comments for additional context.