feat: add LiteLLM as AI gateway provider#440
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces BaseLiteLLM, a new base class that routes LLM evaluators through LiteLLM to support multiple providers. It also adds the litellm dependency to setup.py and implements comprehensive unit tests for the new provider. The review feedback highlights two important improvements: first, create_client should initialize the embedding client to maintain compatibility with RAG-related evaluators inheriting from BaseLiteLLM; second, defensive checks should be added to handle empty response choices and prevent returning the literal string 'None' when the message content is null.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| def create_client(cls): | ||
| if not cls.dynamic_config.model: | ||
| raise ValueError("model cannot be empty in llm config.") | ||
| try: | ||
| import litellm # noqa: F401 | ||
| except ImportError as exc: | ||
| raise ImportError( | ||
| "litellm is not installed. Run: pip install 'dingo-python[litellm]'" | ||
| ) from exc | ||
| # Use cls.client as an initialisation sentinel (no real client object needed). | ||
| cls.client = True |
There was a problem hiding this comment.
The BaseLiteLLM class overrides create_client but completely omits the initialization of the embedding client (cls.embedding_client and cls.embedding_model). This breaks compatibility with RAG-related evaluators that inherit from BaseLiteLLM or use embedding_config in EvaluatorLLMArgs. To ensure BaseLiteLLM is a true drop-in replacement for BaseOpenAI, we should replicate the embedding initialization logic from BaseOpenAI.create_client.
@classmethod
def create_client(cls):
if not cls.dynamic_config.model:
raise ValueError("model cannot be empty in llm config.")
try:
import litellm # noqa: F401
except ImportError as exc:
raise ImportError(
"litellm is not installed. Run: pip install 'dingo-python[litellm]'"
) from exc
# Use cls.client as an initialisation sentinel (no real client object needed).
cls.client = True
# If embedding_config is configured, initialize the embedding client
if cls.dynamic_config.embedding_config:
from openai import OpenAI
from dingo.config.input_args import EmbeddingConfigArgs
embedding_cfg = cls.dynamic_config.embedding_config
# Handle embedding_config being a dict or object
if isinstance(embedding_cfg, dict):
embedding_cfg = EmbeddingConfigArgs(**embedding_cfg)
if not embedding_cfg.api_url:
raise ValueError("embedding_config must provide api_url")
if not embedding_cfg.model:
raise ValueError("embedding_config must provide model")
# Create independent Embedding client
cls.embedding_client = OpenAI(
api_key=embedding_cfg.key or 'dummy-key',
base_url=embedding_cfg.api_url
)
cls.embedding_model = {
'model_name': embedding_cfg.model,
'client': cls.embedding_client
}| finish_reason = response.choices[0].finish_reason # type: ignore[union-attr] | ||
| if finish_reason == "length": | ||
| raise ExceedMaxTokens( | ||
| f"Exceed max tokens: {extra_params.get('max_tokens', 4000)}" | ||
| ) | ||
|
|
||
| return str(response.choices[0].message.content) # type: ignore[union-attr] |
There was a problem hiding this comment.
Defensive checks should be added to handle cases where response.choices is empty or None, which can happen with certain API providers or due to content filtering. Additionally, if message.content is None, calling str(None) will return the literal string 'None', which will cause JSON parsing to fail in process_response. We should return an empty string "" instead.
| finish_reason = response.choices[0].finish_reason # type: ignore[union-attr] | |
| if finish_reason == "length": | |
| raise ExceedMaxTokens( | |
| f"Exceed max tokens: {extra_params.get('max_tokens', 4000)}" | |
| ) | |
| return str(response.choices[0].message.content) # type: ignore[union-attr] | |
| if not response.choices: | |
| raise ValueError("LiteLLM returned an empty response choices list.") | |
| choice = response.choices[0] | |
| finish_reason = choice.finish_reason # type: ignore[union-attr] | |
| if finish_reason == "length": | |
| raise ExceedMaxTokens( | |
| f"Exceed max tokens: {extra_params.get('max_tokens', 4000)}" | |
| ) | |
| content = choice.message.content # type: ignore[union-attr] | |
| return str(content) if content is not None else "" |
|
@RheagalFire thanks for add LiteLLM，pls have a look at the gemini-code-assist comment and failed CI check。 |
2 participants
Summary
BaseLiteLLM— a drop-in base class for Dingo's LLM evaluators that routes calls through LiteLLM, giving access to 100+ providers (Anthropic, Gemini, Bedrock, Groq, Cohere, Mistral, etc.) via a single unified interfacelitellmis an optional extra (pip install "dingo-python[litellm]") — base install is unaffectedBaseOpenAIare untouched;BaseLiteLLMis purely additiveMotivation
Dingo's
BaseOpenAIevaluators require an OpenAI-compatibleapi_urlendpoint. Users who want to run evaluations against Anthropic, Google Gemini, AWS Bedrock, Groq, or any other provider today must either maintain a separate proxy or have no supported path.BaseLiteLLMremoves that friction — specify a provider-prefixed model string and LiteLLM handles the routing automatically.Changes
dingo/model/llm/base_litellm.py— newBaseLiteLLMclass extendingBaseOpenAI; overridescreate_client()andsend_messages()to uselitellm.completion()withdrop_params=Truesetup.py— addedlitellm>=1.35.0,<2.0underextras_require['litellm']and included it in'all'test/scripts/model/llm/test_litellm.py— 13 unit tests covering dispatch,drop_params, key/url forwarding, extra params, length error, and response parsingImplementation notes
drop_params=Trueby default — silently drops per-provider-unsupported kwargs (e.g.seed,strict,frequency_penaltyfail on Anthropic/Gemini without this)litellmis imported inside method bodies so the base install works without itcls.client = Trueafter init so theeval()loop'sif cls.client is Noneguard doesn't re-run initialization on every callapi_urlmaps toapi_base— supports LiteLLM proxy deployments;keymaps toapi_keyTests
1. Unit tests:
pytest test/scripts/model/llm/test_litellm.py -vFull suite (excluding pyspark which isn't installed in CI without the extras):
543 passed, 27 skipped2. Live E2E against Anthropic (via Azure Foundry):
Full chain verified:
eval()→create_client()→build_messages()→send_messages()→litellm.completion()→ Anthropic REST →process_response()→EvalDetail.3. Import syntax check:
python .github/scripts/check_imports.py— 158 files checked, all pass.4. Pin resolution:
litellm>=1.35.0,<2.0resolves cleanly (litellm 1.89.1 is current stable).Risk / Compatibility
setup.pyextraslitellmstays optional; base install (dingo-python) is unaffecteddrop_params=Trueis scoped toBaseLiteLLMonly;BaseOpenAIbehavior is unchangedExample usage
Install:
pip install "dingo-python[litellm]"