Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions internal/llm/subprocess.go
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,14 @@ type LaunchedProcess struct {
processGroup *processGroup
}

// subprocessWaitDelay is the grace, after the process context ends (task
// timeout, abort, or normal completion), before stdout/stderr are force-closed.
// A claude worker that escapes the process group (it becomes its own group
// leader) survives cmd.Cancel's group kill and keeps a pipe open; a consumer's
// synchronous read then blocks forever and cmd.Wait never runs — the run wedges
// at 0% CPU holding the ledger. A var (not const) so tests can shorten it.
var subprocessWaitDelay = 10 * time.Second

// LaunchProcess starts an adapter subprocess with shared cancellation, logging,
// and platform-specific process-group handling.
func LaunchProcess(ctx context.Context, command string, args []string, dir string, env []string, timeout time.Duration, logPath string, cleanup func() error, withStdin bool) (*LaunchedProcess, error) {
Expand All @@ -48,6 +56,9 @@ func LaunchProcess(ctx context.Context, command string, args []string, dir strin
return nil, err
}
cmd.Cancel = func() error { return procGroup.kill(cmd) }
// Backstop for cmd.Wait if the pipes are held open by a worker that escaped
// the group after the process exits.
cmd.WaitDelay = subprocessWaitDelay

var stdin io.WriteCloser
if withStdin {
Expand Down Expand Up @@ -102,6 +113,16 @@ func LaunchProcess(ctx context.Context, command string, args []string, dir strin
process.Abort(cleanup)
return nil, err
}
// If a worker escapes the process group and holds stdout/stderr open, a
// consumer's synchronous read blocks forever and cmd.Wait never runs. When
// the context ends, give the pipes a grace to EOF cleanly, then force them
// closed so any pending read unblocks and the task fails instead of wedging.
go func() {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The goroutine that force-closes stdout/stderr fires on every context end — including normal successful completion — not only the stuck-pipe scenario it was added for.

When a task finishes cleanly the adapter eventually calls the derived cancel(), which immediately satisfies <-procCtx.Done(). The goroutine then sleeps subprocessWaitDelay (10 s) and closes pipes the process already closed. Every subprocess invocation adds a 10-second goroutine tail, and at the same moment cmd.WaitDelay (set to the same value) triggers Go's exec internals to also close those pipes — so both paths race to close identical *os.File descriptors. Individual Close() errors are suppressed, so there is no crash, but concurrent double-close on the same fd is confusing to future readers, and the always-on goroutine pattern will compound as more subprocess adapter types are added.

cmd.WaitDelay and the goroutine sleep also share one variable (subprocessWaitDelay) for two semantically distinct timeouts: the exec-internal backstop for cmd.Wait vs. the grace period before a consumer-side io.ReadAll is unblocked. Coupling them prevents tuning either independently.

Concrete fix: Introduce a pipesDrained channel that the consumer closes after its read loop exits, and gate the force-close goroutine on that signal:

pipesDrained := make(chan struct{})
go func() {
    select {
    case <-pipesDrained:
        // consumer finished normally; nothing to force-close
    case <-procCtx.Done():
        time.Sleep(subprocessWaitDelay)
        _ = stdout.Close()
        _ = stderr.Close()
    }
}()

Return pipesDrained from LaunchProcess so the adapter can close it after its scan loop. This eliminates the 10-second goroutine tail on the happy path and makes the guard conditional rather than always-on. Split subprocessWaitDelay into subprocessPipeGrace (goroutine) and subprocessWaitDelay (cmd.WaitDelay) if different tuning is ever needed.

Reply inline to this comment.

<-procCtx.Done()
time.Sleep(subprocessWaitDelay)
_ = stdout.Close()
_ = stderr.Close()
}()
return process, nil
}

Expand Down
62 changes: 62 additions & 0 deletions internal/llm/subprocess_launch_unix_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
//go:build !windows

package llm

import (
"bufio"
"context"
"io"
"os/exec"
"path/filepath"
"strings"
"testing"
"time"
)

// TestLaunchProcessUnblocksReadOnCanceledContext reproduces the subprocess
// wedge: a worker that escapes the process group (becomes its own group leader)
// holds stdout open, so cmd.Cancel's group kill misses it and a consumer's
// synchronous read of Stdout() never sees EOF. LaunchProcess must force the
// pipes closed a grace after the context ends so the read unblocks. Without the
// on-cancel pipe close this test times out.
func TestLaunchProcessUnblocksReadOnCanceledContext(t *testing.T) {
if _, err := exec.LookPath("python3"); err != nil {
t.Skip("python3 required to spawn a process-group-escaping worker")
}
prev := subprocessWaitDelay
subprocessWaitDelay = 100 * time.Millisecond
defer func() { subprocessWaitDelay = prev }()

ctx, cancel := context.WithCancel(context.Background())
// The worker os.setsid()s out of the process group, prints READY on the
// inherited stdout, then holds it open. Reading READY proves it has escaped
// and owns the pipe. The parent sleeps so it (and the group kill) can't
// close stdout on its own.
script := `python3 -c 'import os,sys,time; os.setsid(); print("READY"); sys.stdout.flush(); time.sleep(30)' & sleep 20`

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Python worker's time.sleep(30) far outlasts the 5-second test deadline. Because the worker has called os.setsid() it escapes the process-group kill and becomes an orphan reparented to init; it will run for up to 30 seconds after the test exits. Across repeated or parallel test runs this accumulates stray Python processes. A 3–5 second sleep is enough to hold the pipe open through the 5-second test timeout and the 100 ms grace period without leaving long-lived orphans. Change the script to time.sleep(5) (or similar) to keep the scenario valid while bounding the orphan lifetime.

Reply inline to this comment.

logPath := filepath.Join(t.TempDir(), "subprocess.log")
p, err := LaunchProcess(ctx, "/bin/sh", []string{"-c", script}, "", nil, 0, logPath, func() error { return nil }, false)
if err != nil {
t.Fatalf("LaunchProcess: %v", err)
}

rd := bufio.NewReader(p.Stdout())
line, err := rd.ReadString('\n')
if err != nil || !strings.Contains(line, "READY") {
t.Fatalf("worker did not signal READY: %q err=%v", line, err)
}

cancel() // simulate task-timeout / abort; the escaped worker still holds stdout

done := make(chan struct{})
go func() {
_, _ = io.ReadAll(rd)
close(done)
}()
select {
case <-done:
// unblocked as expected
case <-time.After(5 * time.Second):
t.Fatal("stdout read wedged after context cancel: pipes were not force-closed")
}
_ = p.Command().Wait()
}
Loading