Slurm template: Signal SIGINT by default to allow R code to catch it gracefully
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 66/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Quiet
- Tech stack
- r
- Domain
- distributed-systems, infrastructure
Research direction
Locate the Slurm template and the resource handling behind plan(..., resources = list(signal = ...)); start by checking how scheduler-specific defaults and explicit signal values are currently passed through. Done means the Slurm default uses B:INT@60 while an explicitly supplied signal remains effective, with relevant existing tests updated or added if the project has coverage for templates.
Written by the indexing model from the issue text.
Description
When a job is approaching it's maximum run-time limit, Slurm sends a SIGTERM 30 seconds before sending a SIGKILL (abrupt; not capturable).
R code cannot handle SIGTERM signals, only SIGINT. If Slurm would signal SIGINT instead, we could capture it internally as an interrupt condition, and using tryCatch(..., interrupt = ...), on.exit(), and likes to gracefully exit, e.g. close connections, checkpoint intermediate results, etc.
Slurm allows us to declare what type of signal, and when, to signal when we approach the run-time limit. This can be done by declaring, e.g. --signal=B:INT@60.
Idea
First, should be able to control the signal explicitly via:
plan(..., resources = list(signal = "INT@60"))
already today.
Second, we could update the default to be signal = "INT@60" by adding the following to the template:
## Resources needed
<%
## Default to sending SIGINT 60 seconds before walltime limit
## to allow graceful R-level cleanup/checkpointing
if (is.null(resources[["signal"]])) {
resources[["signal"]] <- "B:INT@60"
}
...
%>
Third, alternative to a Slurm-specific resource name, we might harmonize the signal type and signal grace period with what is used by other job schedulers.
- Dominant language
- R
- Stars
- 87
- Forks
- 10
- PR merge metrics
- No merged PRs in 30d
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from futureverse/future.batchtools
-
feature/resources scheduler/lsf
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
futureverse/future.batchtools#105 ·
-
feature/resources scheduler/sge
Difficulty 4/5 3-5 days Newbie friendliness 50/100
futureverse/future.batchtools#104 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 55/100
futureverse/future.batchtools#102 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 15/100
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
All issues in futureverse/future.batchtools
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
robjhyndman/forecast#1220 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
JamesHWade/deputy#192 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
bug triage_needed
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
pharmaverse/rtables#1123 · 1 comment · 1 reaction ·