Skip to content

[evaluation] Enable groundedness for Bing and OpenAPI tool calls - #48305

Open
mmkawale wants to merge 2 commits into
mainfrom
copilot/groundedness-enable-bing-openapi-sdk-clean2
Open

[evaluation] Enable groundedness for Bing and OpenAPI tool calls#48305
mmkawale wants to merge 2 commits into
mainfrom
copilot/groundedness-enable-bing-openapi-sdk-clean2

Conversation

@mmkawale

Copy link
Copy Markdown
Contributor

Summary

Enable groundedness validator support for Bing and OpenAPI tool calls while keeping stricter rejection for the remaining unsupported tool families.

Changes

  • Update groundedness-specific unsupported tool list to allow:
    • bing_grounding
    • bing_custom_search
    • openapi_call
  • Keep groundedness-only rejection for:
    • browser_automation
    • code_interpreter_call
    • computer_call
    • web_search
  • Update groundedness validator tests to assert new allow/reject behavior.
  • Refresh groundedness evaluator inline comment to match policy.

Validation

  • PYTHONPATH=sdk/evaluation/azure-ai-evaluation python -m pytest sdk/evaluation/azure-ai-evaluation/tests/unittests/test_common_validators.py -k "UnsupportedToolsListGroundednessValidator or UnsupportedToolsListConversationValidator or ConversationValidatorRejectsStillUnsupportedTools"
  • Result: 24 passed

Diff hygiene

This PR is a low-noise replacement for #48304 with reduced whitespace churn.

Copilot AI review requested due to automatic review settings July 28, 2026 19:19
@mmkawale
mmkawale requested a review from a team as a code owner July 28, 2026 19:19
@github-actions github-actions Bot added the Evaluation Issues related to the client library for Azure AI Evaluation label Jul 28, 2026
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
9 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Enables groundedness validation for Bing and OpenAPI tool calls while retaining restrictions for unsupported tool families.

Changes:

  • Narrows groundedness’s unsupported-tool list.
  • Updates validator tests and policy comments.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
_conversation_validator.py Updates groundedness tool restrictions.
_groundedness.py Refreshes evaluator policy documentation.
test_common_validators.py Tests allowed and rejected tool families.

Comment on lines +490 to 494
UNSUPPORTED_TOOLS: List[str] = [
tool_name
for tool_name in ConversationValidator.UNSUPPORTED_TOOLS
if tool_name not in {"bing_custom_search", "bing_grounding", "openapi_call"}
]
Copilot AI review requested due to automatic review settings July 28, 2026 20:22
@mmkawale

Copy link
Copy Markdown
Contributor Author

Addressed: updated the stale class-level policy comment in _conversation_validator.py so it no longer says Groundedness rejects structured grounding tools pending a context extractor. It now states the current policy: those structured grounding tools are accepted, and Groundedness only keeps stricter guardrails for the remaining unsupported tool families.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_common/_validators/_conversation_validator.py:488

  • The subclass does not preserve stricter guardrails; it relaxes the base policy by removing three entries. Its purpose is to scope that relaxation to Groundedness without changing other evaluators, so the class documentation currently describes the relationship in reverse.
    A dedicated subclass is still kept so Groundedness can preserve stricter
    guardrails for the remaining unsupported tool families without changing
    behavior of other evaluators.

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_groundedness/_groundedness.py:123

  • This says the groundedness policy is stricter than the shared validator, but the new list is a subset of ConversationValidator.UNSUPPORTED_TOOLS; both validators reject these four remaining families. Please avoid documenting a stricter relative policy that the code does not implement.
        # keeps stricter guardrails than the shared base for the remaining
        # unsupported tool families, while allowing grounding/search/openapi
        # tool calls supported by this evaluator.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Evaluation Issues related to the client library for Azure AI Evaluation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants