Skip to content

feat: add [ LiteLLM as AI gateway ] provider#2016

Open
RheagalFire wants to merge 3 commits into
FoundationAgents:mainfrom
RheagalFire:feat/add-litellm-provider
Open

feat: add [ LiteLLM as AI gateway ] provider#2016
RheagalFire wants to merge 3 commits into
FoundationAgents:mainfrom
RheagalFire:feat/add-litellm-provider

Conversation

@RheagalFire
Copy link
Copy Markdown

@RheagalFire RheagalFire commented Apr 21, 2026

Features

  • Add LiteLLM provider in metagpt/provider/litellm_api.py implementing BaseLLM against the litellm.acompletion() API
  • Add LLMType.LITELLM = "litellm" to metagpt/configs/llm_config.py and register in metagpt/provider/init.py
  • Add tests/metagpt/provider/test_litellm_api.py covering completion, streaming, and config kwarg mapping
  • Add litellm>=1.55,<1.65 to requirements.txt (version range chosen to resolve cleanly against MetaGPT's openai~=1.64 pin).
  • Add aask_code / _achat_completion_function / get_choice_function_arguments so Engineer / QAEngineer / DataAnalyst roles work on LiteLLM-routed models.

Feature Docs

Users set api_type: litellm in config2.yaml and route to 100+ providers (OpenAI, Anthropic, Bedrock, Vertex, Ollama, OpenRouter, Groq, DeepSeek, etc.) via a single interface:

llm:
api_type: "litellm"
model: "anthropic/claude-3-5-sonnet-20241022"
api_key: "sk-..."

LiteLLM routes based on the model-name prefix. Full provider list: https://docs.litellm.ai/docs/providers

Influence

  • Unblocks MetaGPT on any LiteLLM-supported provider without writing a new provider class per provider, reduces maintenance burden of per-provider files going forward
  • Additive only existing providers (OpenAI, Anthropic, Bedrock, etc.) are untouched
  • One new required dependency (litellm>=1.55,<1.65). Range pinned to stay compatible with MetaGPT's base install (newer litellm requires openai>=1.66; MetaGPT pins openai~=1.64)
  • Matches existing provider conventions (header block, @register_provider decorator, async-first)
  • Reuses OpenAI-shaped response parsing since litellm.ModelResponse follows OpenAI format
  • Thinking/reasoning-mode passthrough matches anthropic_api.py

Result

$ pytest tests/metagpt/provider/test_litellm_api.py -v
tests/metagpt/provider/test_litellm_api.py::test_litellm_acompletion PASSED
tests/metagpt/provider/test_litellm_api.py::test_cons_kwargs_maps_config PASSED
tests/metagpt/provider/test_litellm_api.py::test_cons_kwargs_forwards_custom_base_url PASSED
tests/metagpt/provider/test_litellm_api.py::test_aask_code_forwards_tools_and_parses_arguments PASSED
tests/metagpt/provider/test_litellm_api.py::test_aask_code_respects_user_supplied_tools PASSED
======================== 5 passed in 5.32s =========================

Existing provider tests (test_openai.py) also pass, no regressions.

E2E smoke test against Azure OpenAI (azure/gpt-4o):

→ Model: azure/gpt-4o
→ Endpoint: https://<redacted>.cognitiveservices.azure.com                                                                                                                                                                                                                                                                                                              
                                                                                                                                                                                                                                                                                                                                                                        
[Test 1] acompletion, prompt 'What is 2+2?'                                                                                                                                                                                                                                                                                                                             
  response: '4'                                                                                                                                                                                                                                                                                                                                                         
                                                                                                                                                                                                                                                                                                                                                                        
[Test 2] aask with system message, prompt 'What is 10 - 3?'                                                                                                                                                                                                                                                                                                             
  response: '7'
                                                                                                                                                                                                                                                                                                                                                                        
MetaGPT LiteLLM live test PASSED.                                                                                                                                        

Exercised the full call path: LLMConfig(api_type=LITELLM) to LLM_REGISTRY[LITELLM] to LiteLLM._cons_kwargs to litellm.acompletion (real Azure endpoint) to response parsing.

Other

  • Forwards api_key / base_url / proxy / reasoning config fields through to litellm.acompletion
  • Default OpenAI base_url is intentionally not forwarded so LiteLLM can route by model-name prefix
  • Cost tracking via _update_costs LiteLLM returns an OpenAI-shaped usage object
  • aask_code / tool-use support lets MetaGPT's code-generating roles (Engineer, QAEngineer, DataAnalyst) work unchanged on LiteLLM-routed models. LiteLLM forwards tools=[...] to all providers that support native function calling and returns OpenAI-shape tool_calls regardless of backend, so the same parsing logic from openai_api.py applies.

@RheagalFire RheagalFire changed the title feat: add LiteLLM as AI gateway provider feat: add [ LiteLLM as AI gateway ] provider Apr 21, 2026
@RheagalFire
Copy link
Copy Markdown
Author

cc @geekan @garylin2099

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant