feat: retrival add raw api metrics calculate#437
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces the computation and tracking of raw API metrics from search traces during retrieval execution. It adds helper functions to build ranked document ID lists preserving raw API ranks and prefixes these metrics accordingly. Feedback on the changes suggests using the loop index instead of the rank in build_raw_api_ranked_doc_ids to guarantee unique placeholders for unmatched or duplicate results, and adopting a more defensive pattern in _compute_raw_api_metrics_from_search_traces to prevent potential TypeError exceptions when the queries key is explicitly set to None.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| ranked: list[str] = [] | ||
| seen_resolved_ids: set[str] = set() | ||
| for index, result in enumerate(top_api_results, start=1): | ||
| rank = result.get("rank") or index | ||
| resolved_id = str(result.get("resolved_corpus_id") or "").strip() | ||
| if not resolved_id: | ||
| ranked.append(f"__raw_api_unmatched__{rank}") | ||
| continue | ||
| if resolved_id in seen_resolved_ids: | ||
| ranked.append(f"__raw_api_duplicate__{rank}") | ||
| continue | ||
| seen_resolved_ids.add(resolved_id) | ||
| ranked.append(resolved_id) |
There was a problem hiding this comment.
In build_raw_api_ranked_doc_ids, the unique suffixes for unmatched and duplicate placeholders are constructed using rank. However, if the input top_api_results contains duplicate or zero ranks, these placeholders might not be unique, which could cause them to collapse and shift subsequent results incorrectly during metric evaluation. Using the loop index (which is guaranteed to be strictly increasing and unique) instead of rank is much more robust and also allows simplifying the function by removing the unused rank variable entirely.
| ranked: list[str] = [] | |
| seen_resolved_ids: set[str] = set() | |
| for index, result in enumerate(top_api_results, start=1): | |
| rank = result.get("rank") or index | |
| resolved_id = str(result.get("resolved_corpus_id") or "").strip() | |
| if not resolved_id: | |
| ranked.append(f"__raw_api_unmatched__{rank}") | |
| continue | |
| if resolved_id in seen_resolved_ids: | |
| ranked.append(f"__raw_api_duplicate__{rank}") | |
| continue | |
| seen_resolved_ids.add(resolved_id) | |
| ranked.append(resolved_id) | |
| ranked: list[str] = [] | |
| seen_resolved_ids: set[str] = set() | |
| for index, result in enumerate(top_api_results, start=1): | |
| resolved_id = str(result.get("resolved_corpus_id") or "").strip() | |
| if not resolved_id: | |
| ranked.append(f"__raw_api_unmatched__{index}") | |
| continue | |
| if resolved_id in seen_resolved_ids: | |
| ranked.append(f"__raw_api_duplicate__{index}") | |
| continue | |
| seen_resolved_ids.add(resolved_id) | |
| ranked.append(resolved_id) | |
| return ranked |
| for trace in traces: | ||
| if trace.get("task") != task_name: | ||
| continue | ||
| for query in trace.get("queries", []): |
There was a problem hiding this comment.
Using trace.get("queries", []) can raise a TypeError: 'NoneType' object is not iterable if the "queries" key is explicitly set to None in the trace dictionary. Using trace.get("queries") or [] is more robust and consistent with the defensive pattern used on line 350 (query.get("raw_api_metrics") or {}).
| for query in trace.get("queries", []): | |
| for query in (trace.get("queries") or []): |
1 participant
No description provided.