Skip to content

fix(always-on): extract report from LLM text when report tool is not invoked#153

Open
Gucc111 wants to merge 1 commit into
mainfrom
fix/148-report-fallback-from-events
Open

fix(always-on): extract report from LLM text when report tool is not invoked#153
Gucc111 wants to merge 1 commit into
mainfrom
fix/148-report-fallback-from-events

Conversation

@Gucc111
Copy link
Copy Markdown
Collaborator

@Gucc111 Gucc111 commented Jun 4, 2026

Summary

  • When the report-phase agent produces text but does not call always_on_report, the run was incorrectly marked as failed with reason report_tool_not_invoked. Now the runtime extracts assistant_text_delta events from the report session, runs them through parseReportMarkdown for validation/completion, and persists the result — so the outcome reflects the actual execution success rather than a missed tool call.
  • The existing tool-call path is completely unchanged; the new logic only activates as a fallback when reportCtx.report is unset and there is no reportError.

Changes

File: src/always-on/runtime/DiscoveryFire.ts

  1. Added extractAssistantText() helper that concatenates all assistant_text_delta events into a single string.
  2. In both fire() and run() methods, after drainTurn completes and before outcome determination: if the report tool was not called and there is no error, attempt to extract LLM text from events. If non-empty, parse it via parseReportMarkdown and write to disk, setting reportCtx.report so the outcome resolves to "executed".
  3. If the LLM produced no text output either, the original writeFallbackReport path is preserved (outcome = "failed").

Priority: tool call > LLM text > fallback template

Scenario Outcome
Tool called successfully executed (unchanged)
Tool not called, LLM text present executed (new)
Tool not called, no LLM text failed (unchanged)

Closes #148

Made with Cursor

…invoked (#148)

When the report-phase agent produces text but does not call
always_on_report, the run was incorrectly marked as failed with
reason "report_tool_not_invoked". Now the runtime extracts
assistant_text_delta events, runs them through parseReportMarkdown,
and persists the result — so the outcome reflects the actual
execution success rather than a missed tool call.

Co-authored-by: Cursor <cursoragent@cursor.com>
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the Always-On runtime to avoid incorrectly marking a run as failed when the report-phase agent produces a valid textual report but does not invoke the always_on_report tool. It adds a fallback path that reconstructs the report from streamed assistant_text_delta events, validates/completes it via parseReportMarkdown, and persists it so the run outcome reflects execution success.

Changes:

  • Capture report-phase gateway events (reportEvents) and extract concatenated assistant text when the report tool was not invoked and no report error occurred.
  • Parse the extracted text with parseReportMarkdown and persist the normalized report, setting reportCtx.report so the outcome becomes "executed".
  • Preserve the existing fallback-template behavior when no tool call occurs and the assistant produced no report text.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +1160 to +1167
function extractAssistantText(events: GatewayEvent[]): string {
let text = "";
for (const event of events) {
if (event.type === "assistant_text_delta") {
text += event.text;
}
}
return text.trim();
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Always-On run marked failed when report tool is not invoked despite successful execution

2 participants