feat(eval): imperative online-insight commands + BYO execution role (draft)

[jariy17]

Draft — Part 1. Deploy bug-bash on the exploratory account: PASSED (results below). Scope TODOs remain.

Command structure

agentcore eval online-insight create --help
  --name <name>                     (required)
  --execution-role-arn <arn>        (required) IAM role the config assumes — NO auto-provisioning
  --agent <agent>                   harness ID or runtime ID whose traffic to sample
  --endpoint <endpoint>             endpoint qualifier (default DEFAULT; requires --agent)
  --data-source-config <json>       traces to evaluate (alt. to --agent)
  --insight <insight...>            (required) Builtin.Insight.* IDs or ARNs
  --clustering-frequency <freq...>  DAILY | WEEKLY | MONTHLY
  --sampling-rate <rate>            (required) 0.01-100
  --session-timeout-minutes <n>     1-1440 (default 15)
  --filters <json>                  trace filters
  --enable-on-create <true|false>   default true
  --description <text>

Also: eval online-insight get | list | pause | resume | delete (by --id).

What this does

• New eval online-insight command family (service-direct), sibling of eval online-eval.
• Dedicated client methods on CoreEvalClient: createOnlineInsight, getOnlineInsight, listOnlineInsights, setOnlineInsightExecutionStatus, deleteOnlineInsight.
• create sets the SDK insights (+ optional clusteringConfig), never evaluators, and requires --execution-role-arn — insight configs are not auto-provisioned a role.
• get/list/pause/resume/delete delegate to the online-eval control-plane ops (same underlying OnlineEvaluationConfig resource).

Service client (CoreEvalClient)

CreateOnlineInsightInput: name, samplingRate, insightIds (required), evaluationExecutionRoleArn (required), clusteringConfig?, sessionTimeoutMinutes?, filters?, enableOnCreate?, description?, and a source union (agent+endpoint? | dataSourceConfig). The impl maps insightIds → insights[{insightId}] on CreateOnlineEvaluationConfigCommand and passes the caller role straight through (no grantOnlineEvalScope).

Decisions made (flag for review)

1. New flag named --execution-role-arn (net-new, non-breaking). Online-eval's --role-arn rename is deferred.
2. Insight methods live on CoreEvalClient (not a separate class file).

TODO (remaining scope)

• eval online-insight update
• Make eval online-eval --role-arn required + remove shared auto-provisioning (onlineEvalExecutionRole.tsx, the ?? autoProvision fallback, --update-role, role-scope warning)
• Deploy bug-bash on exploratory account — PASSED

Verification

tsc --noEmit clean for these changes. Unit test added (online-insight.test.tsx). Note: bun test/unbundled bun run for the eval chain can't execute in the current sandbox (pre-existing missing @opentelemetry/otlp-transformer deep-import that only the bundler resolves) — bug-bash uses the built bundle.

Bug bash — exploratory account (PASSED)

Bundled CLI vs account 725476964917 (us-west-2). Evidence: bugbash-evidence/OI-IMPERATIVE.log.

• create (--data-source-config + BYO role AgentCoreEvalsSDK-… + --insight Builtin.Insight.FailureAnalysis --clustering-frequency DAILY --sampling-rate 50) → created oiImperativeBugBash-Dak7Bc4nAu, executionStatus=ENABLED.
• GET (raw wire) confirms insights:[{insightId:"Builtin.Insight.FailureAnalysis"}], evaluators:[], and evaluationExecutionRoleArn == the supplied role → BYO role honored, not auto-provisioned. sampling / clustering / data-source round-tripped exactly.
• lifecycle: list ✓ · pause → DISABLED ✓ · resume → ENABLED ✓ · delete → ResourceNotFound after ✓.
• negative: create without --execution-role-arn → exit 1 (required option '--execution-role-arn'), no resource created.
• cleanup confirmed; reused role + log group left intact.

Notes: (1) command uses the global --region + AWS_PROFILE env — there is no --profile flag (doc-worthy). (2) A stale node_modules (@opentelemetry/otlp-transformer 0.221.0 vs pinned 0.213.0) breaks the unbundled/dev build; bun install fixes it — environment drift, not this PR.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 80.28169% with 56 lines in your changes missing coverage. Please review.
✅ Project coverage is 97.03%. Comparing base (38c30ae) to head (b3ae3a3).

Files with missing lines	Patch %	Lines
src/core/eval.tsx	11.36%	39 Missing ⚠️
src/handlers/eval/online-insight/resume/index.tsx	52.17%	11 Missing ⚠️
src/handlers/eval/online-insight/list/index.tsx	70.00%	6 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##           refactor    #2067      +/-   ##
============================================
- Coverage     97.24%   97.03%   -0.22%     
============================================
  Files           396      403       +7     
  Lines         24019    24305     +286     
============================================
+ Hits          23357    23584     +227     
- Misses          662      721      +59     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.