tests: reframe IPC hang docs as test-side bug rather than CS bug

Reword the inline comment in ci/tools/setup-sanitizer and the docstrings in
helpers/child_processes.py and memory_ipc/conftest.py to make clear that the
underlying problem is insufficient guards in the IPC tests when child
processes spawn slowly (>30s under compute-sanitizer). The sanitizer change
and the new helpers are mitigations / safety nets; the durable fix is making
the tests handle slow children correctly.

Author: Andy-Jost

ci/tools/setup-sanitizer

@@ -14,11 +14,12 @@ if [[ "${SETUP_SANITIZER}" == 1 ]]; then
   COMPUTE_SANITIZER_VERSION=$(${COMPUTE_SANITIZER} --version | grep -Eo "[0-9]{4}\.[0-9]\.[0-9]" | sed -e 's/\.//g')
   # --target-processes=application-only: attach the sanitizer to the parent
   # pytest process only. Spawned multiprocessing.Process children run without
-  # the sanitizer. This avoids a class of CI hangs where compute-sanitizer's
-  # IPC teardown analysis wedges a child on certain CUDA driver / toolkit
-  # combinations (see issue #2004). The parent process is still fully
-  # sanitized, which is where most of the interesting host-side IPC plumbing
-  # runs anyway.
+  # the sanitizer. This aims to mitigate a class of CI hangs where child
+  # processes take an extreme amount of time to spawn (>30 seconds). Test bugs
+  # triggered by that specific condition are typically uncovered only in CI,
+  # where they become emergencies and are difficult to debug. The parent
+  # process is still fully sanitized, which is where most of the interesting
+  # host-side IPC plumbing runs anyway.
   SANITIZER_CMD="${COMPUTE_SANITIZER} --target-processes=application-only --launch-timeout=0 --tool=memcheck --error-exitcode=1 --report-api-errors=no"
   if [[ "$COMPUTE_SANITIZER_VERSION" -ge 202111 ]]; then
     SANITIZER_CMD="${SANITIZER_CMD} --padding=32"

cuda_core/tests/helpers/child_processes.py

@@ -4,11 +4,10 @@
 """Helpers for tests that spawn ``multiprocessing.Process`` children.
 
 These exist primarily to defend IPC tests against a class of CI hang where a
-child process gets stuck during teardown (e.g., compute-sanitizer's IPC
-teardown analysis on certain CUDA driver / toolkit combinations -- see issue
-#2004). Without intervention, a zombie child holds an IPC memory handle and
-blocks the parent's ``mr.close()`` in fixture teardown, wedging the GHA runner
-for hours.
+child process spawns too slowly and the parent does not implement proper guards
+for that (see issue #2004). Without intervention, a zombie child holds an IPC
+memory handle and blocks the parent's ``mr.close()`` in fixture teardown,
+leading to deadlock and wedging the test runner for hours.
 """
 
 import contextlib

cuda_core/tests/memory_ipc/conftest.py

@@ -6,8 +6,8 @@
 Applies an outer-guard ``pytest.mark.timeout`` to every test in this directory.
 Individual tests still drive their own per-process waits using
 ``child_timeout_sec()`` from ``helpers.child_processes``; this marker is the
-final fallback so that no IPC test can wedge the CI runner for hours if some
-new driver / sanitizer / IPC interaction defeats every other layer.
+final fallback so that no IPC test can wedge the CI runner for hours if
+deadlock occurs.
 """
 
 import pathlib