fix(watcher): reap finished issue-processing task handles [DISCORD-15363517]

[claudear]

Problem

appwrite/claudear gets OOM-killed intermittently. The daemon leaks memory in proportion to the number of issues it processes.

Watcher.spawn_handles is a Vec<JoinHandle<()>> that is push-only in production. dispatch_lane pushes a handle for every spawned process_issue task, and the only code that removes entries is drain_spawned_tasks(), whose four call sites are all inside #[cfg(test)] mod tests. Nothing in the run loop, the housekeeping loop, or the stop path ever reaps them.

Tokio frees a task's allocation only once both the scheduler and the last JoinHandle are dropped, so every completed issue-processing task stayed resident for the process lifetime. That allocation is not a 24-byte handle: process_issue awaited IssueProcessor::run inline with no boxing anywhere, so each retained handle pinned the entire inlined pipeline state machine.

RSS therefore climbed monotonically across days of polling until the kernel OOM-killed the container, which then restarted with a fresh (empty) Vec — matching the reported "sometimes gets OOM killed" pattern.

Agent spawning itself is not out of bounds: the per-source/per-lane gating in dispatch_lane is correct, ProcessingState::remove decrements properly, and the intent-classification path is explicitly sequential.

Fix

• Reap finished handles once per poll cycle (top of poll_source, before the rate-limit early return so an idle or paused watcher still frees them) and on every dispatch, so the list is bounded by concurrency rather than by issues-processed-ever.
• Drain the spawned tasks in stop_and_drain (bounded by the existing 30s budget) so shutdown waits for their teardown too.
• Box::pin the processor.run(...) await so each spawned task allocation carries a pointer instead of the fully inlined pipeline state machine. This shrinks the per-task footprint while tasks are running, not just after they finish.

Test

test_watcher_reaps_finished_spawn_handles seeds spawn_handles with the handles of 64 completed tasks, waits for them to finish, then runs a normal poll cycle and asserts nothing is retained. It fails on main with retained 64 handles and passes with this change.

Full suite: 1186 passed, 0 failed (cargo test -p claudear-engine --lib --features sqlite). cargo fmt --all -- --check and cargo clippy --workspace --all-targets -- -D warnings are clean.

Reported in Discord.

🤖 Generated with Claude Code

[greptile-apps]

Greptile Summary

This PR reaps completed issue-processing JoinHandles during polling and dispatch, drains tracked tasks during graceful shutdown, and boxes the processor future to reduce the spawned task’s inline state-machine footprint.

• Bounds retained task handles by active concurrency rather than lifetime issue count.
• Adds shutdown joining under the existing 30-second budget.
• Adds a regression test covering completed-handle reaping.

Confidence Score: 4/5

The shutdown synchronization gap should be fixed before merging because graceful shutdown can still return with an untracked issue-processing task running.

Spawning a task and inserting its handle are not synchronized with the one-shot shutdown snapshot, so shutdown can observe no active work, drain an empty vector, and then miss a newly inserted handle.

Files Needing Attention: crates/claudear-engine/src/watcher.rs

Important Files Changed

Filename	Overview
crates/claudear-engine/src/watcher.rs	The runtime handle leak is addressed, but the one-shot shutdown drain can miss a concurrently spawned task recorded after the drain snapshot.

Prompt To Fix All With AI

### Issue 1
crates/claudear-engine/src/watcher.rs:1185-1188
**Shutdown misses concurrently recorded tasks**

When shutdown overlaps a dispatch that has passed its running check but has not yet recorded its spawned task, `drain_spawned_tasks` takes the current handle vector before the new handle is inserted. The one-shot drain then returns without joining that task, causing `stop_and_drain` to report a graceful stop while issue processing continues to mutate tracker, notifier, or agent state.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

_{Reviews (1): Last reviewed commit: "fix(watcher): reap finished issue-proces..." | Re-trigger Greptile}

[greptile-apps]

Shutdown misses concurrently recorded tasks

When shutdown overlaps a dispatch that has passed its running check but has not yet recorded its spawned task, drain_spawned_tasks takes the current handle vector before the new handle is inserted. The one-shot drain then returns without joining that task, causing stop_and_drain to report a graceful stop while issue processing continues to mutate tracker, notifier, or agent state.

Prompt To Fix With AI

This is a comment left during a code review.
Path: crates/claudear-engine/src/watcher.rs
Line: 1185-1188

Comment:
**Shutdown misses concurrently recorded tasks**

When shutdown overlaps a dispatch that has passed its running check but has not yet recorded its spawned task, `drain_spawned_tasks` takes the current handle vector before the new handle is inserted. The one-shot drain then returns without joining that task, causing `stop_and_drain` to report a graceful stop while issue processing continues to mutate tracker, notifier, or agent state.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[claudear]

Fix Confidence: 88/100

High confidence the leak is real and fixed: the push-only Vec was verified directly in code (grep over crates/ and src/ returns only the field decl, init, the test-only drain, and the push), the new test fails on main and passes with the change, and all 1186 engine unit tests plus all 16 PR checks are green. Deducted for two things I could not verify end-to-end: (1) I did not measure actual RSS of a running daemon before/after, so the OOM kills could have a second contributing cause beyond this leak; (2) the stop_and_drain addition and the Box::pin change are covered only by the existing suite, not by targeted new assertions — both are low-risk (the drain is bounded by the existing 30s budget and the box is semantics-preserving), but neither is directly proven by a new test.

[ArnabChatterjee20k]

@claudear
This is a comment left during a code review.
Path: crates/claudear-engine/src/watcher.rs
Line: 1185-1188

Comment:
Shutdown misses concurrently recorded tasks

When shutdown overlaps a dispatch that has passed its running check but has not yet recorded its spawned task, drain_spawned_tasks takes the current handle vector before the new handle is inserted. The one-shot drain then returns without joining that task, causing stop_and_drain to report a graceful stop while issue processing continues to mutate tracker, notifier, or agent state.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.