Skip to content

fix: Correct eval model_config and RBAC pass-through for CD pipeline#2

Merged
ericchansen merged 1 commit into
mainfrom
agents/it-s-been-a-long-time-since-i-cd30301f
May 28, 2026
Merged

fix: Correct eval model_config and RBAC pass-through for CD pipeline#2
ericchansen merged 1 commit into
mainfrom
agents/it-s-been-a-long-time-since-i-cd30301f

Conversation

@ericchansen
Copy link
Copy Markdown
Owner

  • Fix GroundednessEvaluator model_config validation failure by adding
    azure_deployment and credential params (was missing both)
  • Extract base AI Services endpoint from project endpoint URL
    (evaluators need account-level endpoint, not project-scoped)
  • Add configurable evaluation.model field to per-env agent configs
  • Pass deployerPrincipalId to foundryAccount Bicep module (was missing,
    preventing RBAC assignment for pipeline identity)
  • Update config SCHEMA.md with new evaluation.model field

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

- Fix GroundednessEvaluator model_config validation failure by adding
  azure_deployment and credential params (was missing both)
- Extract base AI Services endpoint from project endpoint URL
  (evaluators need account-level endpoint, not project-scoped)
- Add configurable evaluation.model field to per-env agent configs
- Pass deployerPrincipalId to foundryAccount Bicep module (was missing,
  preventing RBAC assignment for pipeline identity)
- Update config SCHEMA.md with new evaluation.model field

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings May 28, 2026 18:32
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes evaluation configuration for real Foundry evaluator runs and passes the pipeline deployer identity into the Foundry account module for RBAC assignment.

Changes:

  • Adds base AI Services endpoint extraction and configurable evaluator model wiring.
  • Passes deployerPrincipalId into the Foundry account Bicep module.
  • Documents and sets evaluation.model across environment configs.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
src/scripts/run_evaluation.py Adds evaluator endpoint/model configuration for real evaluations.
infra/main.bicep Passes deployer principal ID to the Foundry account module.
config/SCHEMA.md Documents the new optional evaluation.model field.
config/agent-config.dev.json Sets the evaluation scoring model for dev.
config/agent-config.test.json Sets the evaluation scoring model for test.
config/agent-config.prod.json Sets the evaluation scoring model for prod.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/scripts/run_evaluation.py
@ericchansen ericchansen merged commit 53a930f into main May 28, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants