[evaluation] Enable groundedness for Bing and OpenAPI tool calls - #48305
[evaluation] Enable groundedness for Bing and OpenAPI tool calls#48305mmkawale wants to merge 2 commits into
Conversation
|
Azure Pipelines: Successfully started running 1 pipeline(s). 9 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
There was a problem hiding this comment.
Pull request overview
Enables groundedness validation for Bing and OpenAPI tool calls while retaining restrictions for unsupported tool families.
Changes:
- Narrows groundedness’s unsupported-tool list.
- Updates validator tests and policy comments.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
_conversation_validator.py |
Updates groundedness tool restrictions. |
_groundedness.py |
Refreshes evaluator policy documentation. |
test_common_validators.py |
Tests allowed and rejected tool families. |
| UNSUPPORTED_TOOLS: List[str] = [ | ||
| tool_name | ||
| for tool_name in ConversationValidator.UNSUPPORTED_TOOLS | ||
| if tool_name not in {"bing_custom_search", "bing_grounding", "openapi_call"} | ||
| ] |
|
Addressed: updated the stale class-level policy comment in _conversation_validator.py so it no longer says Groundedness rejects structured grounding tools pending a context extractor. It now states the current policy: those structured grounding tools are accepted, and Groundedness only keeps stricter guardrails for the remaining unsupported tool families. |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (2)
sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_common/_validators/_conversation_validator.py:488
- The subclass does not preserve stricter guardrails; it relaxes the base policy by removing three entries. Its purpose is to scope that relaxation to Groundedness without changing other evaluators, so the class documentation currently describes the relationship in reverse.
A dedicated subclass is still kept so Groundedness can preserve stricter
guardrails for the remaining unsupported tool families without changing
behavior of other evaluators.
sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_groundedness/_groundedness.py:123
- This says the groundedness policy is stricter than the shared validator, but the new list is a subset of
ConversationValidator.UNSUPPORTED_TOOLS; both validators reject these four remaining families. Please avoid documenting a stricter relative policy that the code does not implement.
# keeps stricter guardrails than the shared base for the remaining
# unsupported tool families, while allowing grounding/search/openapi
# tool calls supported by this evaluator.
Summary
Enable groundedness validator support for Bing and OpenAPI tool calls while keeping stricter rejection for the remaining unsupported tool families.
Changes
Validation
PYTHONPATH=sdk/evaluation/azure-ai-evaluation python -m pytest sdk/evaluation/azure-ai-evaluation/tests/unittests/test_common_validators.py -k "UnsupportedToolsListGroundednessValidator or UnsupportedToolsListConversationValidator or ConversationValidatorRejectsStillUnsupportedTools"Diff hygiene
This PR is a low-noise replacement for #48304 with reduced whitespace churn.