[Evaluation] Fix unhashable list crash in binary aggregation#46743
Open
mmkawale wants to merge 5 commits intoAzure:mainfrom
Open
[Evaluation] Fix unhashable list crash in binary aggregation#46743mmkawale wants to merge 5 commits intoAzure:mainfrom
mmkawale wants to merge 5 commits intoAzure:mainfrom
Conversation
Wrap value_counts().to_dict() in _aggregation_binary_output with try/except TypeError. Columns matching outputs.*_result whose values are unhashable (e.g. lists) are now skipped with a warning instead of aborting the entire evaluate() call with EvaluationException: (InternalError) unhashable type: 'list'. Adds a unit test covering a mixed DataFrame (valid pass/fail column + list-valued column) and a CHANGELOG entry under 1.16.7 (Unreleased).
github-actions
Bot
added
Community Contribution
Contributor
|
Thank you for your contribution @mmkawale! We will review the pull request and get back to you soon. |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR hardens evaluation metric aggregation in azure-ai-evaluation so malformed list-valued outputs.*_result columns no longer abort aggregation. It updates the binary aggregation path, adds a targeted unit test, and documents the fix in the package changelog.
Changes:
- Wrapped binary
_resultaggregation intry/except TypeErrorso unhashable values are skipped with a warning instead of failing the run. - Added a unit test covering a valid binary result column alongside an unhashable list-valued
_resultcolumn. - Added a
1.16.7 (Unreleased)changelog entry for the regression fix.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluate/_evaluate.py |
Adds defensive handling for unhashable _result values during binary aggregation. |
sdk/evaluation/azure-ai-evaluation/tests/unittests/test_evaluate.py |
Adds unit coverage for skipping malformed list-valued _result columns without crashing aggregation. |
sdk/evaluation/azure-ai-evaluation/CHANGELOG.md |
Documents the unhashable-list aggregation fix in the unreleased changelog section. |
posaninagendra
approved these changes
May 6, 2026
Author
|
@microsoft-github-policy-service agree [company="Microsoft"] |
Author
|
@microsoft-github-policy-service agree company="Microsoft" |
posaninagendra
approved these changes
May 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Wrap
value_counts().to_dict()in_aggregation_binary_outputwithtry/except TypeError. Columns matchingoutputs.*_resultwhose values are unhashable (e.g. lists) are now skipped with a warning instead of aborting the entireevaluate()call withEvaluationException: (InternalError) unhashable type: 'list'.Changes
try/except TypeErroraroundvalue_counts().to_dict()in_aggregation_binary_output.