Skip to content

Hard-bound heartbeat cleanup when a command ignores SIGTERM #17

Description

@rmcdaniel

Status Summary

The shared heartbeat cleanup runner derives Docker, Compose, inventory, and sandbox-removal command timeouts from one remaining budget. Those commands still execute through synchronous spawnSync, whose timeout sends SIGTERM and waits for the child to exit. A cleanup command that handles or ignores SIGTERM can therefore keep the host runner blocked beyond DW_HEARTBEATS_CLEANUP_TIMEOUT_SECONDS, before deadline_exhausted can be recorded or final resource diagnostics can run.

The current hung-command regression uses a fake Docker process with default signal handling, so it exits when the timeout sends SIGTERM and does not cover this failure mode.

How to fix

  • Run each cleanup command under a bounded lifecycle that escalates and reaps the command and its descendants within the single shared cleanup deadline.
  • Preserve bounded stdout, stderr, exit status, signal, and deadline diagnostics when escalation is required or fails.
  • Add a focused regression whose fake cleanup command deliberately ignores SIGTERM; prove the cleanup runner returns non-passing within a tight bound, does not start later commands after budget exhaustion, and still writes decisive final evidence.

Delete when

Delete when all owned cleanup subprocesses and descendants are settled within the shared deadline even when a command ignores SIGTERM, and an exact-published-artifact heartbeat wave passes while leaving zero project-owned processes, containers, volumes, networks, or sandbox artifacts after the host runner exits.

Metadata

Metadata

Assignees

No one assigned

    Labels

    authority:githubGitHub is the authoritative lifecycle record for this workbeta:blockerMust be resolved before the first 2.0 betacompletion:evidence-requiredClose only after all explicit acceptance and operational evidence is publickind:release-blockerA release cannot proceed until this issue is resolvedpriority:P1High-priority product or release riskrepo:serverOwned by the standalone server repositorystatus:readyReady for implementation

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions