Description
PR #6136 fixed ASCII-only tokenization in ROUGE-1 evaluation for non-English languages. However, the eval module structure has a broader issue: all evaluators are imported eagerly regardless of whether their optional dependencies are installed. Missing dependencies cause import-time failures that are caught and silently converted to zero scores.
Where
google/adk-python eval/ directory — metric registration and import system.
Impact
- Users get zero evaluation scores without realizing an optional dependency is missing
- No warning that
rouge-score, bert-score, or other eval deps weren't installed
- Makes evaluation results silently unreliable
Expected Behavior
- Missing optional dependencies should raise clear errors at eval time, not silently return zero
- Or: evaluators should be lazily imported with user-friendly error messages
Related
Description
PR #6136 fixed ASCII-only tokenization in ROUGE-1 evaluation for non-English languages. However, the eval module structure has a broader issue: all evaluators are imported eagerly regardless of whether their optional dependencies are installed. Missing dependencies cause import-time failures that are caught and silently converted to zero scores.
Where
google/adk-pythoneval/ directory — metric registration and import system.Impact
rouge-score,bert-score, or other eval deps weren't installedExpected Behavior
Related