feat(orchestrator): Surface waiting background work

[mwolson]

Summary

• Show a Waiting state after a root turn settles while finite provider
  background work remains.
• Reuse projected command, dynamic-tool, and subagent lifecycle for Codex, Grok,
  and Claude subagents.
• Project Claude's SDK background-task roster for opaque work such as background
  Bash.
• Present that roster as a static, muted Waiting status in the active classic
  web sidebar while preserving Working for an active turn.

Relationship to #4415

This is the orchestrator-v2 counterpart to Theo's
#4415. That PR owns the current
SidebarV2 and mobile-v2 Waiting presentation. This PR carries the equivalent
provider-thread and projection contract for orchestrator v2.

The active desktop build still renders the classic Sidebar.tsx list, which
does not consume #4415's SidebarV2 resolver. This PR adds only the shared
classic status-pill bridge required to display the already-normalized v2 roster.
It does not modify SidebarV2 or the native mobile list.

Problem and Fix

Problem and Why it Happened	Fix
A completed root turn can still own finite background work, but the v2 UI had no normalized post-settlement signal and could look idle.	Derive normalized pending background work from existing projected turn items and Claude's authoritative SDK task roster.
The visible classic web sidebar ignored that normalized roster and fell back to a timestamp.	Render a static, muted Waiting pill from pendingBackgroundTasks, after active Working and Connecting precedence.
Closing the provider event subscription as soon as a root run settled could miss the final task-roster update.	Keep the subscription open while that provider thread still has finite background work, then persist the empty roster that clears Waiting.
A session-wide pending-work probe could couple unrelated provider threads on a shared runtime.	Use a thread-scoped probe for run ingestion while retaining the session-wide probe for idle release.
Existing stored thread shells do not contain the new field.	Decode pendingBackgroundTasks with an empty-array default for backward compatibility.

Defensive Fixes

Problem and Why it Happened	Fix
Claude SDK roster levels and completion edges can arrive in either order.	Track authoritative roster state separately from wake eligibility and replay classification so Waiting clears without losing exactly-once continuation.
A finite Monitor can emit a terminal notification without Claude starting a native wake turn.	Buffer opaque notifications for replay, but request their continuation only after native user, assistant, or result output proves Claude actually began the wake.
Replacing a Claude query kills the old CLI process before it can clear its roster.	Clear the dead thread's process-scoped roster and wake state, preserve only explicitly classified buffered Bash notifications for same-thread replacement, and keep buffered subagent notifications on the subagent lifecycle.
A Grok subagent can outlive the root turn through adapter carryover.	Include live carryover subagents in the ACP pending-work probe so idle release cannot kill work that is still projected as Waiting.
Persisted background state can survive a process restart without a provider capable of completing it.	Clear persisted rosters, cancel stale background-capable items, and attribute recovery events to the run, subagent, or matching provider thread.

Validation

• Existing focused server, contract, shared, client-runtime, and web logic
  suites pass. The final Claude adapter suite passes 53 tests, and the
  classic-sidebar suite passes 87 tests.
• Typechecks passed for server, web, shared, contracts, and client-runtime.
• Targeted formatting and lint pass for the final sidebar files, the web package
  typecheck passes, and git diff --check passes.
• Real-provider experiment packs passed for Claude background Bash and
  subagents, Codex background exec and subagents, and Grok continuations and
  subagents.
• Isolated integrated web verification with a real Claude background Bash task:
  the classic sidebar row displayed static, muted Waiting after root settlement,
  then returned to its timestamp after the roster cleared.
• A deterministic private-server Opus replay reproduced a nested finite Monitor
  that ended with notification-only traffic. Waiting cleared after run 2, no
  empty run 3 appeared, and the provider stayed idle through a 35-second
  residual window.
• Packaged in-vivo round against the built AppImage covering the
  notification-only path and its regression guards. Two independent scenarios
  drove a real nonpersistent Monitor whose terminal notification arrived with no
  native wake frames: both cleared Waiting without opening a run, and each held
  at two completed runs, idle with an empty roster, through a residual window of
  about 176 seconds. The prior defect left an empty run active for 225 seconds.
• Regression guards passed in the same round: background Bash still wakes exactly
  once across a separate follow-up turn, a resumed subagent still wakes, and Stop
  during a blocking TaskOutput still leaves two distinct clean recovery turns.
  Every wake log marker in the round was attributable to an expected wake, with
  none unattributed.
• Packaged classic-sidebar verification against the running desktop app's own web
  server: the row rendered a static muted Waiting pill with dot class
  bg-sidebar-muted-foreground and no pulse while the roster was nonempty, and
  title plus timestamp only after it cleared. An adjacent thread rendered Working
  in the same view, confirming the states are visually distinct.
• Patch application check passed on fix(orchestrator/grok): Prevent spurious wake run after in-turn monitors #4218. fix(orchestrator): Preserve claude/codex post-interrupt recovery state #4229 has an expected
  RunExecutionService overlap that will need conflict resolution if the
  branches are combined.

The failing Check job is the known Vite+ stdout panic ("This is a bug in Vite+,
not your code") raised while printing pre-existing warnings, not a lint or type
error from this branch. vp check on this exact tree reports 0 errors and 70
pre-existing warnings, and Test, Release Smoke, Mobile Native Static Analysis,
and both Macroscope checks pass on this head.

This branch is based on the live t3code/codex-turn-mapping tip. It does not
semantically depend on #4218.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 0e986ced-7747-4d1a-a8ef-27c11ea064c7

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[macroscopeapp]

🟡 Medium

t3code/apps/server/src/orchestration-v2/ProviderRuntimeRecoveryService.ts

Line 304 in e46af0a

const recoveredNonterminalRunIds = new Set(runs.map((run) => run.id));

isBackgroundCapableTurnItemType causes replayable waiting runs to have their background-capable turn items (command_execution, dynamic_tool, subagent) cancelled, even though those runs were intentionally skipped from terminalization because a pending/running checkpoint effect still exists. When a waiting run has a replayable checkpoint, reconcileProjection omits it from runs, so its ID is absent from recoveredNonterminalRunIds. The later stale-item loop then matches that run's items via isBackgroundCapableTurnItemType + isNonterminalTurnItemStatus and emits turn-item.updated cancellation events, destroying items that belong to a still-recoverable run and checkpoint. The stale-item loop needs to also exclude turn items whose runId belongs to a replayable waiting run, not just those already in recoveredNonterminalRunIds.

🤖 Copy this AI Prompt to have your agent fix this:

In file @apps/server/src/orchestration-v2/ProviderRuntimeRecoveryService.ts around line 304:

`isBackgroundCapableTurnItemType` causes replayable `waiting` runs to have their background-capable turn items (command_execution, dynamic_tool, subagent) cancelled, even though those runs were intentionally skipped from terminalization because a pending/running checkpoint effect still exists. When a `waiting` run has a replayable checkpoint, `reconcileProjection` omits it from `runs`, so its ID is absent from `recoveredNonterminalRunIds`. The later stale-item loop then matches that run's items via `isBackgroundCapableTurnItemType` + `isNonterminalTurnItemStatus` and emits `turn-item.updated` cancellation events, destroying items that belong to a still-recoverable run and checkpoint. The stale-item loop needs to also exclude turn items whose `runId` belongs to a replayable `waiting` run, not just those already in `recoveredNonterminalRunIds`.

[mwolson]

Thanks for the careful trace. The control flow you describe is accurate, but I don't think the cancellation is a defect, so I'm leaving this as-is.

A replayable checkpoint.capture cannot revive the provider process or the background work it owned. Its only run-level action is finalizing the already-settled root run: CheckpointCaptureService emits run.updated with status: "completed". So on restart, the command_execution / dynamic_tool / subagent items belonging to that run are orphaned regardless, because the process that could complete them is gone.

That makes cancelling them the correct cleanup rather than the destruction of recoverable work. The skip set exists to avoid emitting duplicate cancellation events for runs already terminalized in the loop above, not to exempt preserved runs from item cleanup. Exempting them as suggested would leave permanently nonterminal cards for work that can never finish, which is a worse outcome than the current behavior: a stuck spinner with no path to resolution.

The sequence on restart during that window is: background items are cancelled, the roster clears, and the durable checkpoint effect replays and completes the root run. That matches "the server died while background work was in flight," which is what happened.

I did act on your neighbouring comment about the same loop (the missing subagent.updated for settled runs); that one is a real gap and is being fixed.

[macroscopeapp]

Sorry, I'm unable to act on this request because you do not have permissions within this repository.