Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/go-logger.lock.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 5 additions & 3 deletions .github/workflows/go-logger.md
Original file line number Diff line number Diff line change
Expand Up @@ -244,9 +244,11 @@ After adding logging to **all selected files**, validate your changes before cre

After validating your changes:

1. The safe-outputs create-pull-request will automatically create a PR
2. Ensure your changes follow the guidelines above
3. The PR title will automatically have the "[log] " prefix
1. Choose exactly one terminal outcome: `create_pull_request` after successful changes, `noop` when no changes are needed, or `report_incomplete` when a blocking failure prevents completion.
2. Call the chosen safe-output command exactly once, as your final action. Do not call any other safe-output command before or after it.
3. Do not probe safe outputs with `which`, `type`, `--help`, or schema-inspection commands.
4. If the safe-output gateway rejects the call, stop immediately and surface its exact rejection message. Do not retry the call or switch to another terminal safe output.
5. The PR title will automatically have the "[log] " prefix.

## Quality Checklist

Expand Down
11 changes: 11 additions & 0 deletions pkg/cli/audit_mcp_tool_usage_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,17 @@ func TestExtractMCPToolUsageData(t *testing.T) {
wantToolCalls: 2,
wantErr: false,
},
{
name: "tool discovery is not tool usage",
// Discovery traffic identifies the contacted server but does not constitute tool usage.
logContent: `{"timestamp":"2024-01-12T10:00:00Z","level":"info","type":"request","event":"rpc_call","server_name":"safeoutputs","method":"tools/list","duration":50.0,"status":"success"}
{"timestamp":"2024-01-12T10:00:01Z","level":"info","type":"request","event":"request","server_name":"safeoutputs","method":"tools/list","duration":50.0,"status":"success"}
`,
wantServers: 1,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/tdd] The new test case sets wantServers: 1 for a log that contains only tools/list events, but after the fix, discovery-only traffic should arguably not register a server at all — or the test should document why a server entry is still expected.

💡 Detail

If wantServers is 1 because the ServerName field in a tools/list entry still creates a server bucket, the test is correct but the assertion silently documents that the server counter is not subject to the same filtering as tool calls. Consider adding a comment explaining this intentional asymmetry, or reconsider whether discovery-only traffic should increment the server counter.

@copilot please address this.

wantTools: 0,
wantToolCalls: 0,
wantErr: false,
},
{
name: "no gateway.jsonl file",
logContent: "",
Expand Down
29 changes: 28 additions & 1 deletion pkg/cli/audit_report.go
Original file line number Diff line number Diff line change
Expand Up @@ -845,7 +845,7 @@ func extractGHErrorLines(filePath string) []string {
for line := range strings.SplitSeq(string(content), "\n") {
if strings.Contains(line, "##[error]") {
stripped := stripGHALogTimestamps(line)
if stripped != "" {
if stripped != "" && !isAgentToolResultAnnotation(stripped) {
errorLines = append(errorLines, stripped)
}
}
Expand All @@ -854,6 +854,33 @@ func extractGHErrorLines(filePath string) []string {
return errorLines
}

func isAgentToolResultAnnotation(line string) bool {
_, payload, found := strings.Cut(line, "##[error]")
if !found {
return false
}

var event struct {
Type string `json:"type"`
Message struct {
Content []struct {
Type string `json:"type"`
} `json:"content"`
} `json:"message"`
}
if err := json.Unmarshal([]byte(strings.TrimSpace(payload)), &event); err != nil || event.Type != "user" {
return false
}
hasToolResult := false
for _, content := range event.Message.Content {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/diagnosing-bugs] If a ##[error] line contains mixed content — a tool_result alongside another content type (e.g. "type":"text") — the entire line is silently suppressed, hiding a real error.

💡 Suggested guard

Return true only when all content items are tool_result, not when any one of them is:

hasToolResult := false
hasOtherContent := false
for _, c := range event.Message.Content {
    if c.Type == "tool_result" {
        hasToolResult = true
    } else {
        hasOtherContent = true
    }
}
return hasToolResult && !hasOtherContent

This keeps suppression for pure annotation messages while surfacing mixed content that includes real error text.

@copilot please address this.

if content.Type != "tool_result" {
return false
}
hasToolResult = true
}
return hasToolResult
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

L857-880: yagni: full JSON struct decode (nested Type/Message/Content) just to check for a tool_result content type. strings.Contains(payload, "tool_result") gets the same filtering result in ~2 lines, no struct needed.


func extractAgentFailureError(agentRan bool, agentStdioPath string, maxMessageLen int) []ValidationIssue {
if !agentRan {
return nil
Expand Down
29 changes: 29 additions & 0 deletions pkg/cli/audit_report_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -1448,6 +1448,35 @@ func TestExtractPreAgentStepErrors(t *testing.T) {
assert.Contains(t, errors[0].Message, "Lockdown mode is enabled", "Should include actionable ##[error] text")
})

t.Run("ignores annotated tool results and surfaces the runner failure", func(t *testing.T) {
dir := testutil.TempDir(t, "audit-step-*")
require.NoError(t, os.WriteFile(filepath.Join(dir, "agent-stdio.log"), []byte("agent output"), 0600))
workflowLogsDir := filepath.Join(dir, "workflow-logs", "agent")
require.NoError(t, os.MkdirAll(workflowLogsDir, 0755))
toolResult := `{"type":"user","message":{"content":[{"type":"tool_result","content":"raw Go source"}]}}`
logContent := "2026-08-13T04:03:11Z ##[error]" + toolResult + "\n" +
"2026-08-13T04:11:07Z ##[error]The action 'Execute Claude Code CLI' has timed out after 15 minutes."
require.NoError(t, os.WriteFile(filepath.Join(workflowLogsDir, "10_Execute Claude Code CLI.txt"), []byte(logContent), 0600))

errors := extractPreAgentStepErrors(dir)
require.Len(t, errors, 1)
assert.Contains(t, errors[0].Message, "timed out after 15 minutes")
assert.NotContains(t, errors[0].Message, "raw Go source")
})

t.Run("preserves mixed tool result annotations", func(t *testing.T) {
dir := testutil.TempDir(t, "audit-step-*")
workflowLogsDir := filepath.Join(dir, "workflow-logs", "agent")
require.NoError(t, os.MkdirAll(workflowLogsDir, 0755))
mixedContent := `{"type":"user","message":{"content":[{"type":"tool_result","content":"raw Go source"},{"type":"text","text":"runner failure"}]}}`
logContent := "2026-08-13T04:03:11Z ##[error]" + mixedContent
require.NoError(t, os.WriteFile(filepath.Join(workflowLogsDir, "10_Execute Claude Code CLI.txt"), []byte(logContent), 0600))

errors := extractPreAgentStepErrors(dir)
require.Len(t, errors, 1)
assert.Contains(t, errors[0].Message, "runner failure")
})

t.Run("returns nil when workflow-logs directory missing", func(t *testing.T) {
dir := testutil.TempDir(t, "audit-step-*")
// No agent-stdio.log and no workflow-logs directory
Expand Down
4 changes: 2 additions & 2 deletions pkg/cli/gateway_logs_mcp.go
Original file line number Diff line number Diff line change
Expand Up @@ -123,8 +123,8 @@ func extractToolCallsFromGatewayLog(gatewayLogPath string, mcpData *MCPToolUsage
continue // Skip malformed lines
}

// Only process tool call events
if entry.Event == "tool_call" || entry.Event == "rpc_call" || entry.Event == "request" {
// Only process actual tool invocations, not protocol requests such as tools/list.
if entry.Event == "tool_call" || entry.Method == "tools/call" {
toolName := entry.ToolName
if toolName == "" {
toolName = entry.Method
Expand Down
4 changes: 2 additions & 2 deletions pkg/cli/gateway_logs_parsing.go
Original file line number Diff line number Diff line change
Expand Up @@ -190,8 +190,8 @@ func processGatewayLogEntry(entry *GatewayLogEntry, metrics *GatewayMetrics, ver
metrics.TotalDuration += entry.Duration
}

// Track tool calls
if entry.ToolName != "" || entry.Method != "" {
// Track only actual tool invocations, not protocol requests such as tools/list.
if entry.Event == "tool_call" || entry.Method == "tools/call" {
toolName := entry.ToolName
if toolName == "" {
toolName = entry.Method
Expand Down
10 changes: 5 additions & 5 deletions pkg/cli/gateway_logs_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ func TestParseGatewayLogs(t *testing.T) {
`,
wantServers: 3,
wantRequests: 3,
wantToolCalls: 3,
wantToolCalls: 0,
wantErrors: 0,
wantErr: false,
},
Expand Down Expand Up @@ -447,14 +447,14 @@ func TestGatewayLogsWithMethodField(t *testing.T) {

assert.Len(t, metrics.Servers, 1)
assert.Equal(t, 2, metrics.TotalRequests)
assert.Equal(t, 2, metrics.TotalToolCalls)
assert.Equal(t, 1, metrics.TotalToolCalls)

server := metrics.Servers["github"]
require.NotNil(t, server)
assert.Len(t, server.Tools, 2)
assert.Len(t, server.Tools, 1)

// Check that methods were tracked as tools
assert.Contains(t, server.Tools, "tools/list")
// Protocol discovery remains a request but is not counted as tool usage.
assert.NotContains(t, server.Tools, "tools/list")
assert.Contains(t, server.Tools, "tools/call")
}

Expand Down
28 changes: 28 additions & 0 deletions pkg/workflow/prompts_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -276,6 +276,34 @@ func TestDailyFunctionNamerUsesConcreteClaudeModelsForExperiment(t *testing.T) {
}
}

func TestGoLoggerDefinesSingleTerminalSafeOutputContract(t *testing.T) {
repoRoot, err := findRepoRoot()
if err != nil {
t.Fatalf("Failed to find repo root: %v", err)
}

workflowFile := filepath.Join(repoRoot, ".github", "workflows", "go-logger.md")
content, err := os.ReadFile(workflowFile)
if err != nil {
t.Fatalf("Failed to read workflow file: %v", err)
}

workflow := string(content)
for _, keyword := range []string{
"exactly one terminal outcome",
"`create_pull_request`",
"`noop`",
"`report_incomplete`",
"exactly once, as your final action",
"Do not probe safe outputs",
"Do not retry the call or switch to another terminal safe output",
} {
if !strings.Contains(workflow, keyword) {
t.Fatalf("Expected go-logger workflow to include safe-output contract keyword %q", keyword)
}
}
}

func TestDailyCavemanOptimizerUsesConcreteClaudeModelsForExperiment(t *testing.T) {
repoRoot, err := findRepoRoot()
if err != nil {
Expand Down
Loading