Slurm template: Signal SIGINT by default to allow R code to catch it gracefully

[HenrikBengtsson]

When a job is approaching it's maximum run-time limit, Slurm sends a SIGTERM 30 seconds before sending a SIGKILL (abrupt; not capturable).

R code cannot handle SIGTERM signals, only SIGINT. If Slurm would signal SIGINT instead, we could capture it internally as an interrupt condition, and using tryCatch(..., interrupt = ...), on.exit(), and likes to gracefully exit, e.g. close connections, checkpoint intermediate results, etc.

Slurm allows us to declare what type of signal, and when, to signal when we approach the run-time limit. This can be done by declaring, e.g. --signal=B:INT@60.

Idea

First, should be able to control the signal explicitly via:

plan(..., resources = list(signal = "INT@60"))

already today.

Second, we could update the default to be signal = "INT@60" by adding the following to the template:

## Resources needed
<%
 ## Default to sending SIGINT 60 seconds before walltime limit 
 ## to allow graceful R-level cleanup/checkpointing
 if (is.null(resources[["signal"]])) {
   resources[["signal"]] <- "B:INT@60"
 }
 ...
%>

Third, alternative to a Slurm-specific resource name, we might harmonize the signal type and signal grace period with what is used by other job schedulers.