Skip to content

feat: retrival eval add open-eval mode#439

Merged
e06084 merged 2 commits into
MigoXLab:devfrom
e06084:dev
Jun 16, 2026
Merged

feat: retrival eval add open-eval mode#439
e06084 merged 2 commits into
MigoXLab:devfrom
e06084:dev

Conversation

@e06084

@e06084 e06084 commented Jun 16, 2026

Copy link
Copy Markdown
Collaborator

No description provided.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces an LLM-as-Judge 'Open Eval' phase (Exa-style pointwise grading) to the retrieval executor, supporting both standalone evaluation on custom queries and post-MTEB evaluation. Key additions include configuration schemas, CLI options, the LLMSearchResultRelevance grader, and corresponding unit tests. The review feedback highlights several critical and medium-severity issues: a concurrency bug in _run_open_eval where out-of-order thread completion misaligns grades with queries, ignoring the user-configured aggregation method for the summary score, potential parsing crashes on malformed LLM JSON responses, issues with OpenAI client initialization when api_key is None, and unhandled aggregation methods like 'ndcg'.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread dingo/exec/retrieval.py Outdated
Comment thread dingo/exec/retrieval.py Outdated
Comment thread dingo/model/llm/llm_search_result_relevance.py Outdated
Comment thread dingo/model/llm/llm_search_result_relevance.py Outdated
Comment thread dingo/model/llm/llm_search_result_relevance.py
@e06084 e06084 merged commit 2bd25e1 into MigoXLab:dev Jun 16, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant