Node reuse is non-functional on macOS — three bugs cause orphaned node accumulation

[JakeRadMSFT]

Note: The details in this issue were gathered during an investigative Copilot session. Line numbers and code references are based on the current main branch and may have shifted. The findings and repro steps have been validated manually.

Summary

MSBuild's out-of-process node reuse is non-functional on macOS. Three independent bugs combine so that:

1. Node reuse always fails — every idle node times out during the handshake
2. Cross-terminal reuse is impossible — handshake mismatch due to SessionId / getsid() semantics on Unix
3. Orphaned nodes accumulate indefinitely — failed reuse probes keep nodes alive far past their configured idle timeout

In a typical multi-build workflow (e.g., multiple VS Code windows or git worktrees), hundreds of MSBuild worker nodes accumulate and never exit, consuming memory with no possibility of reuse.

Environment: macOS 15 (ARM64), .NET SDK 10.0.103

Bug 1: 50ms Handshake Timeout (Primary — Reuse Always Fails)

When the scheduler tries to reuse an existing idle node, it calls TryConnectToProcess with timeout 0:

// NodeProviderOutOfProcBase.cs:321
nodeToReuse.TryConnectToProcess(nodeToReuse.Id, 0 /* poll */, ...);

On Unix, this is clamped to 50ms:

// CommunicationsUtilities.cs:883
timeout = Math.Max(timeout, 50);

50ms is insufficient for a sleeping node to wake, read the handshake from the pipe, validate it, and respond. Reuse always times out.

Trace Evidence (MSBUILDDEBUGCOMM=1)

Writing handshake part 1 (options)   to pipe 27254... (Timeout 50ms)
Writing handshake part 2 (salt)      to pipe 27254...
Writing handshake part 3-6 (version) to pipe 27254...
Writing handshake part 7 (sessionId) to pipe 27254...
Did not receive return handshake in 50ms.  Probably not an MSBuild process.

Handshake is written successfully (pipe connects to a valid idle node), but the response read times out at 50ms. This happens for every single idle node.

Suggested Fix

Add a dedicated reuse timeout (~1000ms). Busy nodes fail instantly (pipe occupied), so this won't slow builds:

private const int TimeoutForNodeReuse = 1000;
nodeToReuse.TryConnectToProcess(nodeToReuse.Id, TimeoutForNodeReuse, ...);

Bug 2: SessionId / getsid() Breaks Cross-Terminal Reuse

The handshake includes a SessionId component. On Unix, Process.SessionId calls getsid(), which returns the session leader PID:

// CommunicationsUtilities.cs:290-294
sessionId = currentProcess.SessionId;

On macOS, each terminal window and VS Code integrated terminal has a different session leader PID. Nodes spawned from one terminal can never be reused by builds in another — the handshake always mismatches.

# Terminal 1:  SESS=1937
# Terminal 2:  SESS=2504
# VS Code:     SESS=3102

Suggested Fix

On non-Windows, set sessionId = 0. Unix doesn't need the RDP session isolation that motivated this field:

if (!NativeMethodsShared.IsWindows)
    sessionId = 0;
else
    sessionId = currentProcess.SessionId;

Bug 3: ClientConnectTimeout Keeps Nodes Alive Indefinitely

When a reuse probe (Bug 1) connects to an idle node's pipe, writes the handshake, then disconnects after 50ms, the node side is stuck reading the handshake:

// NodeEndpointOutOfProcBase.cs:48
private const int ClientConnectTimeout = 60000; // 60 seconds!

The node's PacketPumpProc calls TryReadIntForHandshake(..., ClientConnectTimeout) which blocks for up to 60 seconds per field, even though the client disconnected. With multiple concurrent builds, each probing idle nodes, a node gets continuously probed: each failed probe blocks it for up to 60s, preventing the idle connection timeout (MSBUILDNODECONNECTIONTIMEOUT) from ever being checked.

This is why MSBUILDNODECONNECTIONTIMEOUT=60000 has no effect in practice.

Suggested Fix

Reduce ClientConnectTimeout to match the reuse probe timeout, or detect broken pipes early and abort the handshake read.

How the Bugs Interact

Build starts
  → Scheduler finds 50+ idle nodes from previous builds
  → Tries to connect to each with 0ms timeout (clamped to 50ms)  [Bug 1]
  → Handshakes also fail due to SessionId mismatch                [Bug 2]
  → All reuse attempts fail → spawns fresh nodes
  → Failed probes keep idle nodes stuck in 60s handshake reads    [Bug 3]
  → Idle nodes never reach their timeout check → never exit
  → Next build repeats with even MORE idle nodes
  → System accumulates hundreds of orphaned MSBuild processes

Workaround Assessment

Workaround	Effective?	Downside
MSBUILDDISABLENODEREUSE=1	Yes — nodes exit after build	10+ concurrent builds spawn too many total processes, machine freezes
MSBUILDNODECONNECTIONTIMEOUT=60000	No — Bug 3 prevents timeout from firing	-
External watchdog script (kill idle nodes)	Yes	Requires extra tooling
-m:N to limit nodes per build	Partially	Reduces peak count but doesn't fix the accumulation

No good client-side workaround exists. These bugs need to be fixed in MSBuild.

Repro Steps

1. macOS with .NET SDK installed
2. Clone a non-trivial .NET solution (e.g., 20+ projects)
3. Open 3+ terminal windows
4. Run dotnet build in each terminal
5. After builds complete, check: pgrep -f 'MSBuild.dll.*nodemode' | wc -l
6. Observe: nodes from all builds are still running, none were reused
7. Wait 15+ minutes — nodes still running (Bug 3 keeps them alive if any builds ran concurrently)

Source References

Bug	File	Line	Code
1	CommunicationsUtilities.cs	883	Math.Max(timeout, 50)
1	NodeProviderOutOfProcBase.cs	321	TryConnectToProcess(..., 0 /* poll */)
2	CommunicationsUtilities.cs	290	sessionId = currentProcess.SessionId
3	NodeEndpointOutOfProcBase.cs	48	ClientConnectTimeout = 60000
3	NodeEndpointOutOfProcBase.cs	377-410	PacketPumpProc wait loop