SGE template: Signal SIGINT by default to allow R to catch it gracefully
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 50/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Quiet
- Tech stack
- bash, r
- Domain
- distributed-systems
Research direction
Start with the default SGE template and trace how rscript_call, uri, and batchtools::doJobCollection launch the job. Review the proposed Bash SIGUSR2 trap and process-wait flow, then verify that an approaching time limit reaches R as SIGINT and permits graceful cleanup before termination.
Written by the indexing model from the issue text.
Description
When a job is approaching it's maximum run-time limit, and --notify has been specified, SGE sends a SIGUSR2 60 seconds before sending a SIGKILL (abrupt; not capturable).
R code cannot handle SIGUSR2 signals, only SIGINT. If SGE would signal SIGINT instead, we could capture it internally as an interrupt condition, and using tryCatch(..., interrupt = ...), on.exit(), and likes to gracefully exit, e.g. close connections, checkpoint intermediate results, etc.
Idea
Update the default SGE template to signal SIGINT. Unlike Slurm, there is no user-facing option to change the type of signal in SGE. But we could use Bash traps to capture the SIGUSR2 and re-signal it as SIGINT. Something like:
#! /bin/bash
...
## Request SGE to send warning signals (SIGUSR2) before run-time limit termination
#$ -notify
...
# Function to catch SGE run-time signals and re-signal as SIGINT
forward_sigint() {
echo "Caught SGE run-time signal. Re-signal as SIGINT to R..."
kill -2 "${R_PID}" 2>/dev/null || true
wait "${R_PID}"
res=$?
echo " - exit code (from signal handler): ${res}"
exit "${res}"
}
# Trap SGE run-time limit signals
trap forward_sigint SIGUSR2
<%= rscript_call %> -e 'batchtools::doJobCollection("<%= uri %>")' &
R_PID=$!
wait "${R_PID}"
res=$?
# Clear the trap on normal exit
trap - SIGUSR2
- Dominant language
- R
- Stars
- 87
- Forks
- 10
- PR merge metrics
- No merged PRs in 30d
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from futureverse/future.batchtools
-
feature/resources scheduler/lsf
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
futureverse/future.batchtools#105 ·
-
feature/resources scheduler/slurm
Difficulty 3/5 1-2 days Newbie friendliness 66/100
futureverse/future.batchtools#103 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 55/100
futureverse/future.batchtools#102 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 15/100
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
All issues in futureverse/future.batchtools
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
robjhyndman/forecast#1220 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
JamesHWade/deputy#192 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
bug triage_needed
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
pharmaverse/rtables#1123 · 1 comment · 1 reaction ·