tests: kill zombie IPC child processes after join timeout

When Python 3.12 CI runs, env-vars enables compute-sanitizer with
--target-processes=all, which attaches to every mp.Process child the
tests spawn. On CUDA 12.9.1 the sanitizer analysis of IPC buffer teardown
gets stuck, so child processes never exit. The existing
join(timeout=CHILD_TIMEOUT_SEC) returns but leaves the child alive.

That zombie keeps its IPC handle open. When pytest teardown runs
ipc_memory_resource's mr.close(), it blocks waiting for the handle to
be released -- tying up the runner for hours until GitHub Actions
force-cancels the job. This is the exact pattern in issue #2004
(always Python 3.12 + CUDA 12.9.1 local).

Fix: after join(timeout=...), kill any process still alive so the IPC
handle is released before fixture teardown. A stderr warning is printed
when kill() fires so the failure is clearly attributable to the
sanitizer/IPC deadlock rather than appearing as a generic exitcode != 0.
Tests still fail (exit code is non-zero or completed is False), just in
seconds rather than hours.

Fixes #2004

Author: aryanputta

cuda_core/tests/memory_ipc/test_send_buffers.py

@@ -2,6 +2,7 @@
 # SPDX-License-Identifier: Apache-2.0
 
 import multiprocessing as mp
+import sys
 from itertools import cycle
 
 import pytest
@@ -40,10 +41,19 @@ def test_main(self, ipc_device, nmrs):
             # Wait for the child process.
             process.join(timeout=CHILD_TIMEOUT_SEC)
             if process.is_alive():
-                # Child is stuck (CUDA context teardown can hang under certain
-                # driver/Python combos — see issue #2004). Kill it so the
-                # IPC handle is released and the fixture teardown doesn't
-                # block the runner for hours.
+                # Child did not exit within the timeout. Under compute-sanitizer
+                # with --target-processes=all (active for Python 3.12 + local
+                # CTK on Linux), IPC memory teardown inside the sanitizer can
+                # deadlock on CUDA 12.9.1, leaving the child alive indefinitely.
+                # SIGKILL forces the kernel to reclaim all IPC handles so that
+                # fixture teardown (mr.close()) does not block the runner for
+                # hours. See issue #2004.
+                print(
+                    f"[WARN] child process {process.pid} still alive after "
+                    f"{CHILD_TIMEOUT_SEC}s — sending SIGKILL "
+                    f"(likely compute-sanitizer IPC deadlock, see issue #2004)",
+                    file=sys.stderr,
+                )
                 process.kill()
                 process.join()
             assert process.exitcode == 0
@@ -108,11 +118,19 @@ def test_main(self, ipc_device, ipc_memory_resource):
         proc_b.join(timeout=CHILD_TIMEOUT_SEC)
         proc_c.join(timeout=CHILD_TIMEOUT_SEC)
 
-        # Kill any processes that are still alive. Without this, a child stuck
-        # in CUDA context teardown (issue #2004: Python 3.12 + CUDA 12.9.1)
-        # holds IPC handles and blocks fixture teardown indefinitely.
+        # Kill any processes that are still alive. Under compute-sanitizer with
+        # --target-processes=all, IPC teardown deadlocks on CUDA 12.9.1 so
+        # children never exit. SIGKILL forces kernel IPC handle release so
+        # fixture teardown (mr.close()) does not block the runner for hours.
+        # See issue #2004.
         for p in (proc_b, proc_c):
             if p.is_alive():
+                print(
+                    f"[WARN] child process {p.pid} still alive after "
+                    f"{CHILD_TIMEOUT_SEC}s — sending SIGKILL "
+                    f"(likely compute-sanitizer IPC deadlock, see issue #2004)",
+                    file=sys.stderr,
+                )
                 p.kill()
                 p.join()