Add R worker cleanup trap and HPC doctor tooling

[smjenness]

RSPH runs ProctrackType=proctrack/linuxproc with no cgroup containment, so SLURM finds a job's processes by walking parent-PID chains. R's PSOCK workers reparent to init, severing that chain. On scancel, TIME_LIMIT or preemption the master dies and the workers survive, still pinned to their original core mask, while SLURM marks those cores free and schedules new work onto them. Two process sets then share the same cores, so the next task gets about half a core, runs many times slower, and is eventually killed at its walltime, leaking more workers in turn.

Measured 2026-07-18: cancelling one calibration array left 163 orphans on 15 nodes for about 2.5 hours (roughly 400 core-hours). Two unrelated projects (India_COVID19_model and LA-PrEP-2026) were degraded on the same node in the same two-hour window.

What this adds

swf_cleanup_r_workers() returns the TERM/EXIT trap that reaps a task's own workers. swf_configs_rsph() now appends it to r_loader by default via a new cleanup_workers argument (set FALSE to opt out).

This matters beyond the mechanism: the trap already existed, hand-written, in LA-PrEP-2026's hpc_configs.R, and was simply absent from India_COVID19_model. Neither project calls swf_configs_rsph(); both hardcode their own node setup. That divergence is why the leak went unnoticed for months. Putting it here lets projects stop maintaining private copies.

inst/hpc_doctor/ ships four shell tools, located via hpc_doctor_script():

script	role
probe_node_cpu.sh	node-side; instantaneous CPU per SLURM task, one ssh per node
degen_watch.sh	classifies running tasks, requeues degenerate ones
deploy_doctor.sh	1-CPU companion job for the life of a campaign, self-terminating
term_orphans.sh	SIGTERMs genuinely orphaned workers, safe on shared nodes

Detect by CPU, not by runtime

Healthy workers sit at ~99% CPU, starved ones at ~50%. That signal is distribution-free: no runtime history, so no cold-start window on a new deploy, and no per-project calibration. Critically it cannot mistake a legitimately long scenario for a broken one, since a slow task doing real work still reads ~99% and is spared. A runtime threshold cannot make that distinction at all.

Two things worth reviewing carefully

1. PPID=1 is not an orphan test. parallelly::makeClusterPSOCK reparents healthy workers to init, so a live worker legitimately shows PPID=1. Verified: 88 such processes across six nodes were all healthy workers of live jobs. term_orphans.sh instead tests whether the process's SLURM_JOB_ID still resolves to a job alive in squeue. Getting this wrong destroys running work.

2. Three load-bearing details in the trap, each a bug in an earlier draft: grep -qxz (not -qz, or an empty SLURM_JOB_ID matches every R process the user owns, including other live jobs); the [ -n "${SLURM_JOB_ID:-}" ] guard; and the trailing : so set -e steps do not exit non-zero on a normal exit.

Scope and limits

The trap is containment, not protection: it stops a dying job leaving orphans, but does nothing to protect a job from orphans already squatting on cores it was handed. It does not cover SIGKILL or, reliably, preemption. The root fix is administrative (proctrack/cgroup + task/cgroup with ConstrainCores=yes); everything here is mitigation until that lands.

The doctor deliberately does not submit workflow slices early (pipelining). That would need exclusive ownership of slice progression, and if it submitted while a step's built-in array_max chaining also submitted, two tasks would write the same output file concurrently and corrupt it.

Notes

• man/ was regenerated with roxygen 8.0.0. I reverted the unrelated .Rd churn and the RoxygenNote bump so the diff stays reviewable; you may want to regenerate with the project's pinned version.
• Regenerating also surfaced a pre-existing stale doc in merge_simfiles.Rd (control.netsim should be control.net). Left out of this PR.
• The shell tools are RSPH-shaped (they assume pgrep -x R, compute-to-compute ssh, and this cluster's no-cgroup behaviour). They should grow capability guards before being relied on elsewhere.