feat(evaluator): add derived evaluators (reuse a base metric on your own model)

[jariy17]

What this does — 1 line

agentcore add evaluator --type derived --base-evaluator-id <id> --model <m> — create an evaluator that reuses a managed base metric's prompt/scoring on your own model. The base is a 3P library metric (ThirdParty.<Provider>.<Metric>) or a built-in (Builtin.<Metric>).

Try it

# derive from a 3P metric (level auto-resolves to SESSION)
agentcore add evaluator --name my_tool_use \
  --type derived --base-evaluator-id ThirdParty.DeepEval.ToolUse \
  --model us.anthropic.claude-sonnet-4-6

# derive from a built-in (level auto-resolves to TRACE)
agentcore add evaluator --name my_helpfulness \
  --type derived --base-evaluator-id Builtin.Helpfulness \
  --model us.anthropic.claude-sonnet-4-6

Output:

Added evaluator 'my_tool_use' (derived from ThirdParty.DeepEval.ToolUse evaluator)

add evaluator --help (new options)

  --type <type>              llm-as-a-judge (default) | code-based | derived        ← now includes derived
  --base-evaluator-id <id>   [derived] "ThirdParty.<Provider>.<Metric>" or "Builtin.<Metric>"   ← new
  --level <level>            SESSION | TRACE | TOOL_CALL (auto-resolved from the base for derived)
  --model <model>            [LLM] Bedrock/OpenResponses model; [derived] your judge model

--instructions / --rating-scale / --model-provider are rejected for --type derived (the base owns the prompt + scale).

Additions to agentcore.json

{
  "name": "my_tool_use",
  "level": "SESSION",
  "config": { "derived": { "baseEvaluatorId": "ThirdParty.DeepEval.ToolUse", "model": "us.anthropic.claude-sonnet-4-6" } }
}

How level is set

The API requires the derived evaluator's level to match the base metric's level. The CLI resolves it for you at add time via GetEvaluator(baseEvaluatorId) and bakes it into agentcore.json — you never type it. --level stays available as an offline override.

Changes

1. schema — new EvaluatorConfig.derived arm { baseEvaluatorId, model }; config is exactly-one-of llmAsAJudge | codeBased | derived; level required for every evaluator.
2. CLI — --type derived + --base-evaluator-id; auto-level via GetEvaluator; derived success message.
3. telemetry — EvaluatorType gains derived.

Verification

• Schema + primitive unit tests pass; regenerated JSON schema.
• Verified end-to-end on gamma (us-west-2): both 3P (ThirdParty.DeepEval.ToolUse → SESSION) and built-in (Builtin.Helpfulness → TRACE) auto-resolve their level and write the right config.derived.
• The add online-eval TUI evaluator picker lists the managed ThirdParty.* evaluators alongside Builtin.*.

Companion CDK change (emits EvaluatorConfig.Derived into the CFN Evaluator): aws/agentcore-l3-cdk-constructs#323