From 98a7a3065ca4da1fce4e690f54d9cee050d6a806 Mon Sep 17 00:00:00 2001 From: "Anna.Zhdan" Date: Mon, 2 Feb 2026 19:23:05 +0100 Subject: [PATCH 01/15] Add custom-llm-endpoint.mdx --- docs/rfds/custom-llm-endpoint.mdx | 291 ++++++++++++++++++++++++++++++ 1 file changed, 291 insertions(+) create mode 100644 docs/rfds/custom-llm-endpoint.mdx diff --git a/docs/rfds/custom-llm-endpoint.mdx b/docs/rfds/custom-llm-endpoint.mdx new file mode 100644 index 00000000..1e60479a --- /dev/null +++ b/docs/rfds/custom-llm-endpoint.mdx @@ -0,0 +1,291 @@ +--- +title: "Custom LLM Endpoint Configuration" +--- + +- Author(s): [@anna239](https://github.com/anna239) + +> **Note:** This RFD is very preliminary and intended to start a dialog about this feature. The proposed design may change significantly based on feedback and further discussion. + +## Elevator pitch + +> What are you proposing to change? + +Add the ability for clients to pass a custom LLM endpoint URL and authentication credentials to agents during initialization. This allows organizations to route all LLM requests through their own infrastructure (proxies, gateways, or self-hosted models) without agents needing to know about this configuration in advance. + +## Status quo + +> How do things work today and what problems does this cause? Why would we change things? + +Currently, agents are configured with their own LLM endpoints and credentials, typically through environment variables or configuration files. This creates problems for: + +- **Client proxies**: Clients want to route agent traffic through their own proxies, f.i. for setting additional headers or logging +- **Enterprise deployments**: Organizations want to route LLM traffic through their own proxies for compliance, logging, or cost management +- **Self-hosted models**: Users running local LLM servers (vLLM, Ollama, etc.) cannot easily redirect agent traffic +- **API gateways**: Organizations using LLM gateways for rate limiting, caching, or multi-provider routing + +## Shiny future + +> How will things play out once this feature exists? + +Clients will be able to: +1. Pass a custom LLM endpoint URL and authentication header +2. Have all agent LLM requests automatically routed through that endpoint + +## Implementation details and plan + +> Tell me more about your implementation. What is your detailed implementation plan? + +We present three alternative approaches for discussion. + +### Alternative A: Pass in `initialize` request + +Add a new optional `llmEndpoint` property to `InitializeRequest`: + +```typescript +interface InitializeRequest { + // ... existing fields ... + + /** + * Custom LLM endpoint configuration. + * When provided, the agent should route all LLM requests through this endpoint. + * This configuration is per-process and should not be persisted to disk. + */ + llmEndpoint?: LlmEndpointConfig | null; +} + +interface LlmEndpointConfig { + /** Base URL for LLM API requests (e.g., "https://llm-proxy.corp.example.com/v1") */ + url: string; + + /** + * Value for the Authorization header. + * Will be sent as-is (e.g., "Bearer sk-..." or "Basic ..."). + */ + authHeader: string; + + /** Extension metadata */ + _meta?: Record; +} + +interface InitializeResponse { + // ... existing fields ... + + /** + * Echoed back when the agent accepts the custom LLM endpoint. + * If present in the response, the agent confirms it will use this endpoint. + * If absent (and was provided in request), the agent did not accept it. + */ + llmEndpoint?: LlmEndpointConfig | null; +} +``` + +#### JSON Schema Additions + +```json +{ + "$defs": { + "LlmEndpointConfig": { + "description": "Configuration for a custom LLM endpoint. This configuration is per-process and should not be persisted to disk.", + "properties": { + "url": { + "type": "string", + "description": "Base URL for LLM API requests." + }, + "authHeader": { + "type": ["string"], + "description": "Value for the Authorization header. Sent as-is." + }, + "_meta": { + "additionalProperties": true, + "type": ["object", "null"] + } + }, + "required": ["url", "authHeader"], + "type": "object" + } + } +} +``` + +#### Example Exchange + +**Initialize Request:** +```json +{ + "jsonrpc": "2.0", + "id": 1, + "method": "initialize", + "params": { + "clientInfo": { + "name": "MyIDE", + "version": "1.0.0" + }, + "protocolVersion": "2025-01-01", + "llmEndpoint": { + "url": "https://llm-gateway.corp.example.com/v1", + "authHeader": "Bearer temporary-token-abc123" + } + } +} +``` + +**Initialize Response (endpoint accepted):** +```json +{ + "jsonrpc": "2.0", + "id": 1, + "result": { + "protocolVersion": "2025-01-01", + "agentInfo": { + "name": "MyAgent", + "version": "2.0.0" + }, + "agentCapabilities": { + "sessionCapabilities": {} + }, + "llmEndpoint": { + "url": "https://llm-gateway.corp.example.com/v1", + "authHeader": "Bearer temporary-token-abc123" + } + } +} +``` + +**Initialize Response (endpoint not accepted):** +```json +{ + "jsonrpc": "2.0", + "id": 1, + "result": { + "protocolVersion": "2025-01-01", + "agentInfo": { + "name": "MyAgent", + "version": "2.0.0" + }, + "agentCapabilities": { + "sessionCapabilities": {} + } + } +} +``` + +#### Behavior + +1. **Confirmation via response**: When the agent accepts the `llmEndpoint`, it MUST echo it back in the `InitializeResponse`. If `llmEndpoint` is absent from the response, the client should assume the agent did not accept the custom endpoint configuration. + +2. **Per-process scope**: The `llmEndpoint` configuration applies to the entire agent process lifetime. It should not be stored to disk or persist beyond the process. + +3. **All LLM requests**: When `llmEndpoint` is accepted, the agent should route ALL LLM API requests through the specified URL, using the provided `authHeader` for authentication. + +4. **No mid-execution changes**: The endpoint cannot be changed after initialization. If a different endpoint is needed, a new agent process must be started. + +5. **Agent discretion**: If an agent cannot support custom endpoints (e.g., uses a proprietary API), it should omit `llmEndpoint` from the response or return an error during initialization. + +### Alternative B: Dedicated `setLlmEndpoint` method + +Instead of passing the endpoint in `initialize`, introduce a dedicated method that can be called after initialization but before session creation. + +```typescript +interface SetLlmEndpointRequest { + /** Base URL for LLM API requests */ + url: string; + + /** Value for the Authorization header */ + authHeader: string; + + /** Extension metadata */ + _meta?: Record; +} + +interface SetLlmEndpointResponse { + /** Whether the endpoint was accepted */ + accepted: boolean; + + /** Extension metadata */ + _meta?: Record; +} +``` + +**Example flow:** +1. Client calls `initialize` +2. Client calls `setLlmEndpoint` with URL and auth header +3. Agent confirms acceptance +4. Client creates session + +**Trade-offs:** +- (+) Cleaner separation of concerns - initialization vs configuration +- (+) Could potentially allow changing endpoint between sessions (if desired) +- (-) Different sessions might want different endpoints, but we want a single endpoint per process + +### Alternative C: Registry-declared environment variables + +Declare in the Agent Registry which environment variables or command-line arguments should be used to pass the URL and token when starting the agent process. + +```json +{ + "name": "MyAgent", + "command": "my-agent", + "llmEndpointConfig": { + "urlEnvVar": "LLM_BASE_URL", + "authEnvVar": "LLM_AUTH_TOKEN" + } +} +``` + +Or with command-line arguments: + +```json +{ + "name": "MyAgent", + "command": "my-agent", + "llmEndpointConfig": { + "urlArg": "--llm-url", + "authArg": "--llm-token" + } +} +``` + +The client would then start the agent process with: +```bash +LLM_BASE_URL="https://gateway.example.com/v1" LLM_AUTH_TOKEN="Bearer sk-..." my-agent +# or +my-agent --llm-url "https://gateway.example.com/v1" --llm-token "Bearer sk-..." +``` + +**Trade-offs:** +- (+) Agents in the Registry can be marked as supporting custom endpoints +- (+) Easier to implement for agents that already read these values from environment variables +- (+) No protocol changes needed - uses existing process spawning mechanisms +- (+) Works even before ACP connection is established +- (-) Requires Registry support +- (-) Less dynamic - configuration is fixed at process start + +## Open questions + +### How should model availability be handled? + +When a custom endpoint is provided, it may only support a subset of models. For example, a self-hosted vLLM server might only have `llama-3-70b` available, while the agent normally advertises `claude-3-opus`, `gpt-4`, etc. + +### Should the agent advertise support for custom endpoints? + +Should there be a capability flag indicating the agent supports `llmEndpoint`? This would let clients know whether passing this configuration will have any effect. + +## Frequently asked questions + +> What questions have arisen over the course of authoring this document? + +### Why not pass endpoint when selecting a model? + +One option would be to pass the endpoint URL and credentials when the user selects a model (e.g., in `session/new` or a model selection method). + +Many agents throw authentication errors before the model selection happens. This makes the flow unreliable. + +### What if the agent doesn't support custom endpoints? + +If the agent doesn't support custom endpoints, it should omit `llmEndpoint` from the `InitializeResponse`. The client can then detect this by comparing the request and response: +- If `llmEndpoint` was sent but not returned, the agent did not accept it +- The client can then decide whether to proceed (using agent's default endpoint) or abort + +## Revision history + +- 2026-02-02: Initial draft - preliminary proposal to start discussion From 8c26ce79191724de00709bed4ad2ac64d47b2d57 Mon Sep 17 00:00:00 2001 From: "Anna.Zhdan" Date: Mon, 2 Feb 2026 19:32:26 +0100 Subject: [PATCH 02/15] make authHeader optional --- docs/rfds/custom-llm-endpoint.mdx | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/rfds/custom-llm-endpoint.mdx b/docs/rfds/custom-llm-endpoint.mdx index 1e60479a..5d7b8d14 100644 --- a/docs/rfds/custom-llm-endpoint.mdx +++ b/docs/rfds/custom-llm-endpoint.mdx @@ -61,7 +61,7 @@ interface LlmEndpointConfig { * Value for the Authorization header. * Will be sent as-is (e.g., "Bearer sk-..." or "Basic ..."). */ - authHeader: string; + authHeader?: string | null; /** Extension metadata */ _meta?: Record; @@ -92,7 +92,7 @@ interface InitializeResponse { "description": "Base URL for LLM API requests." }, "authHeader": { - "type": ["string"], + "type": ["string", "null"], "description": "Value for the Authorization header. Sent as-is." }, "_meta": { @@ -100,7 +100,7 @@ interface InitializeResponse { "type": ["object", "null"] } }, - "required": ["url", "authHeader"], + "required": ["url"], "type": "object" } } @@ -175,7 +175,7 @@ interface InitializeResponse { 2. **Per-process scope**: The `llmEndpoint` configuration applies to the entire agent process lifetime. It should not be stored to disk or persist beyond the process. -3. **All LLM requests**: When `llmEndpoint` is accepted, the agent should route ALL LLM API requests through the specified URL, using the provided `authHeader` for authentication. +3. **All LLM requests**: When `llmEndpoint` is accepted, the agent should route ALL LLM API requests through the specified URL, using the provided `authHeader` for authentication (if provided). 4. **No mid-execution changes**: The endpoint cannot be changed after initialization. If a different endpoint is needed, a new agent process must be started. @@ -191,7 +191,7 @@ interface SetLlmEndpointRequest { url: string; /** Value for the Authorization header */ - authHeader: string; + authHeader?: string | null; /** Extension metadata */ _meta?: Record; From 6b8332f257c6ac80df78174175f8f05be2420907 Mon Sep 17 00:00:00 2001 From: "Anna.Zhdan" Date: Tue, 3 Feb 2026 18:08:08 +0100 Subject: [PATCH 03/15] support for multiple providers --- docs/rfds/custom-llm-endpoint.mdx | 166 ++++++++++++++---------------- 1 file changed, 79 insertions(+), 87 deletions(-) diff --git a/docs/rfds/custom-llm-endpoint.mdx b/docs/rfds/custom-llm-endpoint.mdx index 5d7b8d14..a824f022 100644 --- a/docs/rfds/custom-llm-endpoint.mdx +++ b/docs/rfds/custom-llm-endpoint.mdx @@ -10,7 +10,7 @@ title: "Custom LLM Endpoint Configuration" > What are you proposing to change? -Add the ability for clients to pass a custom LLM endpoint URL and authentication credentials to agents during initialization. This allows organizations to route all LLM requests through their own infrastructure (proxies, gateways, or self-hosted models) without agents needing to know about this configuration in advance. +Add the ability for clients to pass custom LLM endpoint URLs and authentication credentials to agents during initialization, with support for multiple providers. This allows clients to route LLM requests through their own infrastructure (proxies, gateways, or self-hosted models) without agents needing to know about this configuration in advance. ## Status quo @@ -28,29 +28,33 @@ Currently, agents are configured with their own LLM endpoints and credentials, t > How will things play out once this feature exists? Clients will be able to: -1. Pass a custom LLM endpoint URL and authentication header -2. Have all agent LLM requests automatically routed through that endpoint +1. Pass custom LLM endpoint URLs and authentication headers for different providers +2. Have agent LLM requests automatically routed through the appropriate endpoint based on provider ## Implementation details and plan > Tell me more about your implementation. What is your detailed implementation plan? -We present three alternative approaches for discussion. +We present two alternative approaches for discussion. ### Alternative A: Pass in `initialize` request -Add a new optional `llmEndpoint` property to `InitializeRequest`: +Add a new optional `llmEndpoints` property to `InitializeRequest` that maps provider identifiers to endpoint configurations: ```typescript +/** Well-known LLM provider identifiers */ +type LlmProvider = "anthropic" | "openai" | "google" | "amazon" | "will be added later"; + interface InitializeRequest { // ... existing fields ... /** - * Custom LLM endpoint configuration. - * When provided, the agent should route all LLM requests through this endpoint. + * Custom LLM endpoint configurations per provider. + * When provided, the agent should route LLM requests to the appropriate endpoint + * based on the provider being used. * This configuration is per-process and should not be persisted to disk. */ - llmEndpoint?: LlmEndpointConfig | null; + llmEndpoints?: Record | null; } interface LlmEndpointConfig { @@ -71,11 +75,11 @@ interface InitializeResponse { // ... existing fields ... /** - * Echoed back when the agent accepts the custom LLM endpoint. - * If present in the response, the agent confirms it will use this endpoint. - * If absent (and was provided in request), the agent did not accept it. + * Echoed back with the providers the agent accepts. + * Only includes providers that the agent will actually use. + * If absent (and was provided in request), the agent did not accept any endpoints. */ - llmEndpoint?: LlmEndpointConfig | null; + llmEndpoints?: Record | null; } ``` @@ -85,7 +89,7 @@ interface InitializeResponse { { "$defs": { "LlmEndpointConfig": { - "description": "Configuration for a custom LLM endpoint. This configuration is per-process and should not be persisted to disk.", + "description": "Configuration for a custom LLM endpoint.", "properties": { "url": { "type": "string", @@ -102,6 +106,13 @@ interface InitializeResponse { }, "required": ["url"], "type": "object" + }, + "LlmEndpoints": { + "description": "Map of provider identifiers to endpoint configurations. This configuration is per-process and should not be persisted to disk.", + "type": "object", + "additionalProperties": { + "$ref": "#/$defs/LlmEndpointConfig" + } } } } @@ -121,15 +132,21 @@ interface InitializeResponse { "version": "1.0.0" }, "protocolVersion": "2025-01-01", - "llmEndpoint": { - "url": "https://llm-gateway.corp.example.com/v1", - "authHeader": "Bearer temporary-token-abc123" + "llmEndpoints": { + "anthropic": { + "url": "https://llm-gateway.corp.example.com/anthropic/v1", + "authHeader": "Bearer anthropic-token-abc123" + }, + "openai": { + "url": "https://llm-gateway.corp.example.com/openai/v1", + "authHeader": "Bearer openai-token-xyz789" + } } } } ``` -**Initialize Response (endpoint accepted):** +**Initialize Response (endpoints accepted):** ```json { "jsonrpc": "2.0", @@ -143,15 +160,19 @@ interface InitializeResponse { "agentCapabilities": { "sessionCapabilities": {} }, - "llmEndpoint": { - "url": "https://llm-gateway.corp.example.com/v1", - "authHeader": "Bearer temporary-token-abc123" + "llmEndpoints": { + "anthropic": { + "url": "https://llm-gateway.corp.example.com/anthropic/v1", + "authHeader": "Bearer anthropic-token-abc123" + } } } } ``` -**Initialize Response (endpoint not accepted):** +In this example, the agent only uses Anthropic, so it only echoes back that provider's configuration. + +**Initialize Response (endpoints not accepted):** ```json { "jsonrpc": "2.0", @@ -171,35 +192,32 @@ interface InitializeResponse { #### Behavior -1. **Confirmation via response**: When the agent accepts the `llmEndpoint`, it MUST echo it back in the `InitializeResponse`. If `llmEndpoint` is absent from the response, the client should assume the agent did not accept the custom endpoint configuration. +1. **Confirmation via response**: When the agent accepts any `llmEndpoints`, it MUST echo back the ones it will use in the `InitializeResponse`. If `llmEndpoints` is absent from the response, the client should assume the agent did not accept any custom endpoint configurations. -2. **Per-process scope**: The `llmEndpoint` configuration applies to the entire agent process lifetime. It should not be stored to disk or persist beyond the process. +2. **Per-process scope**: The `llmEndpoints` configuration applies to the entire agent process lifetime. It should not be stored to disk or persist beyond the process. -3. **All LLM requests**: When `llmEndpoint` is accepted, the agent should route ALL LLM API requests through the specified URL, using the provided `authHeader` for authentication (if provided). +3. **Provider-based routing**: The agent should route LLM requests to the appropriate endpoint based on the provider. If the agent uses a provider not in the provided map, it uses its default endpoint for that provider. -4. **No mid-execution changes**: The endpoint cannot be changed after initialization. If a different endpoint is needed, a new agent process must be started. +4. **No mid-execution changes**: The endpoints cannot be changed after initialization. If different endpoints are needed, a new agent process must be started. -5. **Agent discretion**: If an agent cannot support custom endpoints (e.g., uses a proprietary API), it should omit `llmEndpoint` from the response or return an error during initialization. +5. **Agent discretion**: If an agent cannot support custom endpoints (e.g., uses a proprietary API), it should omit `llmEndpoints` from the response or return an error during initialization. -### Alternative B: Dedicated `setLlmEndpoint` method +### Alternative B: Dedicated `setLlmEndpoints` method -Instead of passing the endpoint in `initialize`, introduce a dedicated method that can be called after initialization but before session creation. +Instead of passing the endpoints in `initialize`, introduce a dedicated method that can be called after initialization but before session creation. ```typescript -interface SetLlmEndpointRequest { - /** Base URL for LLM API requests */ - url: string; - - /** Value for the Authorization header */ - authHeader?: string | null; +interface SetLlmEndpointsRequest { + /** Map of provider identifiers to endpoint configurations */ + endpoints: Record; /** Extension metadata */ _meta?: Record; } -interface SetLlmEndpointResponse { - /** Whether the endpoint was accepted */ - accepted: boolean; +interface SetLlmEndpointsResponse { + /** Map of accepted provider endpoints */ + accepted: Record; /** Extension metadata */ _meta?: Record; @@ -208,59 +226,23 @@ interface SetLlmEndpointResponse { **Example flow:** 1. Client calls `initialize` -2. Client calls `setLlmEndpoint` with URL and auth header -3. Agent confirms acceptance +2. Client calls `setLlmEndpoints` with provider -> endpoint map +3. Agent confirms which providers were accepted 4. Client creates session **Trade-offs:** - (+) Cleaner separation of concerns - initialization vs configuration -- (+) Could potentially allow changing endpoint between sessions (if desired) -- (-) Different sessions might want different endpoints, but we want a single endpoint per process - -### Alternative C: Registry-declared environment variables - -Declare in the Agent Registry which environment variables or command-line arguments should be used to pass the URL and token when starting the agent process. - -```json -{ - "name": "MyAgent", - "command": "my-agent", - "llmEndpointConfig": { - "urlEnvVar": "LLM_BASE_URL", - "authEnvVar": "LLM_AUTH_TOKEN" - } -} -``` +- (+) Could potentially allow changing endpoints between sessions (if desired) +- (-) If we want a single set of endpoints per process, this is kind of confusing -Or with command-line arguments: - -```json -{ - "name": "MyAgent", - "command": "my-agent", - "llmEndpointConfig": { - "urlArg": "--llm-url", - "authArg": "--llm-token" - } -} -``` - -The client would then start the agent process with: -```bash -LLM_BASE_URL="https://gateway.example.com/v1" LLM_AUTH_TOKEN="Bearer sk-..." my-agent -# or -my-agent --llm-url "https://gateway.example.com/v1" --llm-token "Bearer sk-..." -``` +## Open questions -**Trade-offs:** -- (+) Agents in the Registry can be marked as supporting custom endpoints -- (+) Easier to implement for agents that already read these values from environment variables -- (+) No protocol changes needed - uses existing process spawning mechanisms -- (+) Works even before ACP connection is established -- (-) Requires Registry support -- (-) Less dynamic - configuration is fixed at process start +### How should provider identifiers be standardized? -## Open questions +We need to define a standard set of provider identifiers (e.g., `"anthropic"`, `"openai"`, `"google"`, `"amazon"`). Should this be: +- A fixed enum in the protocol specification? +- An extensible set with well-known values and support for custom strings? +- Defined in a separate registry/document that can be updated independently? ### How should model availability be handled? @@ -268,7 +250,7 @@ When a custom endpoint is provided, it may only support a subset of models. For ### Should the agent advertise support for custom endpoints? -Should there be a capability flag indicating the agent supports `llmEndpoint`? This would let clients know whether passing this configuration will have any effect. +Should there be a capability flag indicating the agent supports `llmEndpoints`? This would let clients know whether passing this configuration will have any effect. ## Frequently asked questions @@ -280,11 +262,21 @@ One option would be to pass the endpoint URL and credentials when the user selec Many agents throw authentication errors before the model selection happens. This makes the flow unreliable. +### Why not use environment variables or command-line arguments? + +One option would be to pass endpoint configuration via environment variables (like `OPENAI_API_BASE`) or command-line arguments when starting the agent process. + +This approach has significant drawbacks: +- With multiple providers, the configuration becomes complex JSON that is awkward to pass via command-line arguments +- Environment variables may be logged or visible to other processes, creating security concerns +- Requires knowledge of agent-specific variable names or argument formats +- No standardized way to confirm the agent accepted the configuration + ### What if the agent doesn't support custom endpoints? -If the agent doesn't support custom endpoints, it should omit `llmEndpoint` from the `InitializeResponse`. The client can then detect this by comparing the request and response: -- If `llmEndpoint` was sent but not returned, the agent did not accept it -- The client can then decide whether to proceed (using agent's default endpoint) or abort +If the agent doesn't support custom endpoints, it should omit `llmEndpoints` from the `InitializeResponse`. The client can then detect this by comparing the request and response: +- If `llmEndpoints` was sent but not returned, the agent did not accept any +- The client can then decide whether to proceed (using agent's default endpoints) or abort ## Revision history From 42ddd4cd192883794c6a35093886e65810a71526 Mon Sep 17 00:00:00 2001 From: Evgeniy Stepanov Date: Wed, 4 Mar 2026 22:14:34 +0200 Subject: [PATCH 04/15] docs: revise custom LLM endpoint RFD to use dedicated setLlmEndpoints method --- docs/rfds/custom-llm-endpoint.mdx | 238 ++++++++++++++++-------------- 1 file changed, 124 insertions(+), 114 deletions(-) diff --git a/docs/rfds/custom-llm-endpoint.mdx b/docs/rfds/custom-llm-endpoint.mdx index a824f022..28ba843e 100644 --- a/docs/rfds/custom-llm-endpoint.mdx +++ b/docs/rfds/custom-llm-endpoint.mdx @@ -2,7 +2,7 @@ title: "Custom LLM Endpoint Configuration" --- -- Author(s): [@anna239](https://github.com/anna239) +- Author(s): [@anna239](https://github.com/anna239), [@xtmq](https://github.com/xtmq) > **Note:** This RFD is very preliminary and intended to start a dialog about this feature. The proposed design may change significantly based on feedback and further discussion. @@ -10,7 +10,7 @@ title: "Custom LLM Endpoint Configuration" > What are you proposing to change? -Add the ability for clients to pass custom LLM endpoint URLs and authentication credentials to agents during initialization, with support for multiple providers. This allows clients to route LLM requests through their own infrastructure (proxies, gateways, or self-hosted models) without agents needing to know about this configuration in advance. +Add the ability for clients to pass custom LLM endpoint URLs and authentication credentials to agents via a dedicated `setLlmEndpoints` method, with support for multiple providers. This allows clients to route LLM requests through their own infrastructure (proxies, gateways, or self-hosted models) without agents needing to know about this configuration in advance. ## Status quo @@ -28,58 +28,120 @@ Currently, agents are configured with their own LLM endpoints and credentials, t > How will things play out once this feature exists? Clients will be able to: -1. Pass custom LLM endpoint URLs and authentication headers for different providers -2. Have agent LLM requests automatically routed through the appropriate endpoint based on provider +1. Discover whether an agent supports custom LLM endpoints via capabilities during initialization +2. Perform agent configuration, including authorization based on this knowledge +3. Pass custom LLM endpoint URLs and headers for different providers via a dedicated method +4. Have agent LLM requests automatically routed through the appropriate endpoint based on provider ## Implementation details and plan > Tell me more about your implementation. What is your detailed implementation plan? -We present two alternative approaches for discussion. +### Intended flow -### Alternative A: Pass in `initialize` request +The design uses a two-step approach: capability discovery during initialization, followed by endpoint configuration via a dedicated method. This enables the following flow: -Add a new optional `llmEndpoints` property to `InitializeRequest` that maps provider identifiers to endpoint configurations: +```mermaid +sequenceDiagram + participant Client + participant Agent -```typescript -/** Well-known LLM provider identifiers */ -type LlmProvider = "anthropic" | "openai" | "google" | "amazon" | "will be added later"; + Client->>Agent: initialize + Note right of Agent: Agent reports capabilities,
including llmEndpoints support + Agent-->>Client: initialize response
(agentCapabilities.llmEndpoints = true) + + Note over Client: Client sees llmEndpoints capability.
Performs configuration / authorization
based on this knowledge. + + Client->>Agent: setLlmEndpoints + Agent-->>Client: setLlmEndpoints response
(accepted providers) + + Note over Client,Agent: Ready for session setup + Client->>Agent: session/new +``` -interface InitializeRequest { +1. **Initialization**: The client calls `initialize`. The agent responds with its capabilities, including an `llmEndpoints` flag indicating support for custom endpoint configuration. +2. **Client-side decision**: The client inspects the capability. If the agent supports `llmEndpoints`, the client can perform authorization, resolve credentials, or configure endpoints accordingly. If the agent does not support it, the client falls back to a different authorization and configuration strategy. +3. **Endpoint configuration**: The client calls `setLlmEndpoints` with provider-to-endpoint mappings. The agent responds with the subset of providers it accepted. +4. **Session creation**: The client proceeds to create a session. + +### Capability advertisement + +The agent advertises support for custom LLM endpoints via a new capability flag in `agentCapabilities`: + +```typescript +interface AgentCapabilities { // ... existing fields ... - /** - * Custom LLM endpoint configurations per provider. - * When provided, the agent should route LLM requests to the appropriate endpoint - * based on the provider being used. - * This configuration is per-process and should not be persisted to disk. - */ - llmEndpoints?: Record | null; + /** Whether the agent supports custom LLM endpoint configuration via setLlmEndpoints */ + llmEndpoints?: boolean; } +``` + +**Initialize Response example:** +```json +{ + "jsonrpc": "2.0", + "id": 0, + "result": { + "protocolVersion": 1, + "agentInfo": { + "name": "MyAgent", + "version": "2.0.0" + }, + "agentCapabilities": { + "llmEndpoints": true, + "sessionCapabilities": {} + } + } +} +``` + +### `setLlmEndpoints` method + +A dedicated method that can be called after initialization but before session creation. + +```typescript +/** Well-known LLM provider identifiers */ +type LlmProvider = "anthropic" | "openai" | "google" | "amazon" | "will be added later"; interface LlmEndpointConfig { /** Base URL for LLM API requests (e.g., "https://llm-proxy.corp.example.com/v1") */ url: string; /** - * Value for the Authorization header. - * Will be sent as-is (e.g., "Bearer sk-..." or "Basic ..."). + * Additional HTTP headers to include in LLM API requests. + * Each entry is a header name mapped to its value. + * Common use cases include Authorization, custom routing, or tracing headers. */ - authHeader?: string | null; + headers?: Record | null; /** Extension metadata */ _meta?: Record; } -interface InitializeResponse { - // ... existing fields ... +interface SetLlmEndpointsRequest { + /** + * Custom LLM endpoint configurations per provider. + * When provided, the agent should route LLM requests to the appropriate endpoint + * based on the provider being used. + * This configuration is per-process and should not be persisted to disk. + */ + endpoints: Record; + + /** Extension metadata */ + _meta?: Record; +} +interface SetLlmEndpointsResponse { /** * Echoed back with the providers the agent accepts. * Only includes providers that the agent will actually use. - * If absent (and was provided in request), the agent did not accept any endpoints. + * If empty, the agent did not accept any endpoints. */ - llmEndpoints?: Record | null; + accepted: Record; + + /** Extension metadata */ + _meta?: Record; } ``` @@ -95,9 +157,12 @@ interface InitializeResponse { "type": "string", "description": "Base URL for LLM API requests." }, - "authHeader": { - "type": ["string", "null"], - "description": "Value for the Authorization header. Sent as-is." + "headers": { + "type": ["object", "null"], + "description": "Additional HTTP headers to include in LLM API requests.", + "additionalProperties": { + "type": "string" + } }, "_meta": { "additionalProperties": true, @@ -120,120 +185,66 @@ interface InitializeResponse { #### Example Exchange -**Initialize Request:** +**setLlmEndpoints Request:** ```json { "jsonrpc": "2.0", - "id": 1, - "method": "initialize", + "id": 2, + "method": "setLlmEndpoints", "params": { - "clientInfo": { - "name": "MyIDE", - "version": "1.0.0" - }, - "protocolVersion": "2025-01-01", - "llmEndpoints": { + "endpoints": { "anthropic": { "url": "https://llm-gateway.corp.example.com/anthropic/v1", - "authHeader": "Bearer anthropic-token-abc123" + "headers": { + "Authorization": "Bearer anthropic-token-abc123", + "X-Request-Source": "my-ide" + } }, "openai": { "url": "https://llm-gateway.corp.example.com/openai/v1", - "authHeader": "Bearer openai-token-xyz789" + "headers": { + "Authorization": "Bearer openai-token-xyz789" + } } } } } ``` -**Initialize Response (endpoints accepted):** +**setLlmEndpoints Response:** ```json { "jsonrpc": "2.0", - "id": 1, + "id": 2, "result": { - "protocolVersion": "2025-01-01", - "agentInfo": { - "name": "MyAgent", - "version": "2.0.0" - }, - "agentCapabilities": { - "sessionCapabilities": {} - }, - "llmEndpoints": { + "accepted": { "anthropic": { "url": "https://llm-gateway.corp.example.com/anthropic/v1", - "authHeader": "Bearer anthropic-token-abc123" + "headers": { + "Authorization": "Bearer anthropic-token-abc123", + "X-Request-Source": "my-ide" + } } } } } ``` -In this example, the agent only uses Anthropic, so it only echoes back that provider's configuration. - -**Initialize Response (endpoints not accepted):** -```json -{ - "jsonrpc": "2.0", - "id": 1, - "result": { - "protocolVersion": "2025-01-01", - "agentInfo": { - "name": "MyAgent", - "version": "2.0.0" - }, - "agentCapabilities": { - "sessionCapabilities": {} - } - } -} -``` +In this example, the agent only uses Anthropic, so it only accepts that provider's configuration. #### Behavior -1. **Confirmation via response**: When the agent accepts any `llmEndpoints`, it MUST echo back the ones it will use in the `InitializeResponse`. If `llmEndpoints` is absent from the response, the client should assume the agent did not accept any custom endpoint configurations. - -2. **Per-process scope**: The `llmEndpoints` configuration applies to the entire agent process lifetime. It should not be stored to disk or persist beyond the process. - -3. **Provider-based routing**: The agent should route LLM requests to the appropriate endpoint based on the provider. If the agent uses a provider not in the provided map, it uses its default endpoint for that provider. - -4. **No mid-execution changes**: The endpoints cannot be changed after initialization. If different endpoints are needed, a new agent process must be started. - -5. **Agent discretion**: If an agent cannot support custom endpoints (e.g., uses a proprietary API), it should omit `llmEndpoints` from the response or return an error during initialization. - -### Alternative B: Dedicated `setLlmEndpoints` method - -Instead of passing the endpoints in `initialize`, introduce a dedicated method that can be called after initialization but before session creation. - -```typescript -interface SetLlmEndpointsRequest { - /** Map of provider identifiers to endpoint configurations */ - endpoints: Record; +1. **Capability discovery**: The agent MUST advertise `llmEndpoints: true` in `agentCapabilities` if it supports the `setLlmEndpoints` method. Clients SHOULD check this capability before calling the method. - /** Extension metadata */ - _meta?: Record; -} +2. **Timing**: The `setLlmEndpoints` method MUST be called after `initialize` and before `session/new`. Calling it during an active session is undefined behavior. All subsequent sessions will use the newly configured endpoints. -interface SetLlmEndpointsResponse { - /** Map of accepted provider endpoints */ - accepted: Record; +3. **Confirmation via response**: The agent MUST respond with the `accepted` map containing only the providers it will actually use. If the agent accepts none, the `accepted` map SHOULD be empty. - /** Extension metadata */ - _meta?: Record; -} -``` +4. **Per-process scope**: The endpoint configuration applies to the entire agent process lifetime. It should not be stored to disk or persist beyond the process. -**Example flow:** -1. Client calls `initialize` -2. Client calls `setLlmEndpoints` with provider -> endpoint map -3. Agent confirms which providers were accepted -4. Client creates session +5. **Provider-based routing**: The agent should route LLM requests to the appropriate endpoint based on the provider. If the agent uses a provider not in the provided map, it uses its default endpoint for that provider. -**Trade-offs:** -- (+) Cleaner separation of concerns - initialization vs configuration -- (+) Could potentially allow changing endpoints between sessions (if desired) -- (-) If we want a single set of endpoints per process, this is kind of confusing +6. **Agent discretion**: If an agent cannot support custom endpoints (e.g., uses a proprietary API), it should not advertise the `llmEndpoints` capability. ## Open questions @@ -248,14 +259,14 @@ We need to define a standard set of provider identifiers (e.g., `"anthropic"`, ` When a custom endpoint is provided, it may only support a subset of models. For example, a self-hosted vLLM server might only have `llama-3-70b` available, while the agent normally advertises `claude-3-opus`, `gpt-4`, etc. -### Should the agent advertise support for custom endpoints? - -Should there be a capability flag indicating the agent supports `llmEndpoints`? This would let clients know whether passing this configuration will have any effect. - ## Frequently asked questions > What questions have arisen over the course of authoring this document? +### Why not pass endpoints in the `initialize` request? + +Passing endpoints directly in `initialize` would require the client to have already resolved credentials and configured endpoints before knowing whether the agent supports this feature. In practice, the client needs to inspect the agent's capabilities first to decide its authorization strategy — for example, whether to route through a corporate proxy or use direct credentials. A dedicated method after initialization solves this chicken-and-egg problem and keeps capability negotiation separate from endpoint configuration. + ### Why not pass endpoint when selecting a model? One option would be to pass the endpoint URL and credentials when the user selects a model (e.g., in `session/new` or a model selection method). @@ -274,10 +285,9 @@ This approach has significant drawbacks: ### What if the agent doesn't support custom endpoints? -If the agent doesn't support custom endpoints, it should omit `llmEndpoints` from the `InitializeResponse`. The client can then detect this by comparing the request and response: -- If `llmEndpoints` was sent but not returned, the agent did not accept any -- The client can then decide whether to proceed (using agent's default endpoints) or abort +If the agent doesn't support custom endpoints, it will not advertise `llmEndpoints: true` in `agentCapabilities` during initialization. The client can detect this and choose an alternative authorization and configuration strategy, or proceed using the agent's default endpoints. ## Revision history +- 2026-03-04: Revised to use dedicated `setLlmEndpoints` method with capability advertisement - 2026-02-02: Initial draft - preliminary proposal to start discussion From f7548d9b805d85257570fbecfe70b26fab0d45cd Mon Sep 17 00:00:00 2001 From: Evgeniy Stepanov Date: Thu, 5 Mar 2026 10:26:42 +0200 Subject: [PATCH 05/15] docs: update custom LLM endpoint RFD to include provider-based capabilities --- docs/rfds/custom-llm-endpoint.mdx | 79 +++++++++++++++---------------- 1 file changed, 39 insertions(+), 40 deletions(-) diff --git a/docs/rfds/custom-llm-endpoint.mdx b/docs/rfds/custom-llm-endpoint.mdx index 28ba843e..773b5eda 100644 --- a/docs/rfds/custom-llm-endpoint.mdx +++ b/docs/rfds/custom-llm-endpoint.mdx @@ -48,32 +48,50 @@ sequenceDiagram Client->>Agent: initialize Note right of Agent: Agent reports capabilities,
including llmEndpoints support - Agent-->>Client: initialize response
(agentCapabilities.llmEndpoints = true) + Agent-->>Client: initialize response
(agentCapabilities.llmEndpoints.providers) - Note over Client: Client sees llmEndpoints capability.
Performs configuration / authorization
based on this knowledge. + Note over Client: Client sees supported providers.
Performs configuration / authorization
based on this knowledge. Client->>Agent: setLlmEndpoints - Agent-->>Client: setLlmEndpoints response
(accepted providers) + Agent-->>Client: setLlmEndpoints response Note over Client,Agent: Ready for session setup Client->>Agent: session/new ``` -1. **Initialization**: The client calls `initialize`. The agent responds with its capabilities, including an `llmEndpoints` flag indicating support for custom endpoint configuration. -2. **Client-side decision**: The client inspects the capability. If the agent supports `llmEndpoints`, the client can perform authorization, resolve credentials, or configure endpoints accordingly. If the agent does not support it, the client falls back to a different authorization and configuration strategy. -3. **Endpoint configuration**: The client calls `setLlmEndpoints` with provider-to-endpoint mappings. The agent responds with the subset of providers it accepted. +1. **Initialization**: The client calls `initialize`. The agent responds with its capabilities, including an `llmEndpoints` object listing supported providers. +2. **Client-side decision**: The client inspects the supported providers list. If the agent lists providers in `llmEndpoints.providers`, the client can perform authorization, resolve credentials, or configure endpoints for those specific providers. If `llmEndpoints` is absent or the providers list is empty, the client falls back to a different authorization and configuration strategy. +3. **Endpoint configuration**: The client calls `setLlmEndpoints` with endpoint configurations for the supported providers. 4. **Session creation**: The client proceeds to create a session. ### Capability advertisement -The agent advertises support for custom LLM endpoints via a new capability flag in `agentCapabilities`: +The agent advertises support for custom LLM endpoints and lists its supported providers via a new `llmEndpoints` capability in `agentCapabilities`: ```typescript +/** Well-known LLM provider identifiers */ +type LlmProvider = "anthropic" | "openai" | "google" | "amazon" | "will be added later"; + +interface LlmEndpointsCapability { + /** + * Map of supported provider identifiers. + * The client should only configure endpoints for providers listed here. + */ + providers: Record; + + /** Extension metadata */ + _meta?: Record; +} + interface AgentCapabilities { // ... existing fields ... - /** Whether the agent supports custom LLM endpoint configuration via setLlmEndpoints */ - llmEndpoints?: boolean; + /** + * Custom LLM endpoint support. + * If present with a non-empty providers map, the agent supports the setLlmEndpoints method. + * If absent or providers is empty, the agent does not support custom endpoints. + */ + llmEndpoints?: LlmEndpointsCapability; } ``` @@ -89,7 +107,12 @@ interface AgentCapabilities { "version": "2.0.0" }, "agentCapabilities": { - "llmEndpoints": true, + "llmEndpoints": { + "providers": { + "anthropic": {}, + "openai": {} + } + }, "sessionCapabilities": {} } } @@ -101,9 +124,6 @@ interface AgentCapabilities { A dedicated method that can be called after initialization but before session creation. ```typescript -/** Well-known LLM provider identifiers */ -type LlmProvider = "anthropic" | "openai" | "google" | "amazon" | "will be added later"; - interface LlmEndpointConfig { /** Base URL for LLM API requests (e.g., "https://llm-proxy.corp.example.com/v1") */ url: string; @@ -133,13 +153,6 @@ interface SetLlmEndpointsRequest { } interface SetLlmEndpointsResponse { - /** - * Echoed back with the providers the agent accepts. - * Only includes providers that the agent will actually use. - * If empty, the agent did not accept any endpoints. - */ - accepted: Record; - /** Extension metadata */ _meta?: Record; } @@ -216,35 +229,21 @@ interface SetLlmEndpointsResponse { { "jsonrpc": "2.0", "id": 2, - "result": { - "accepted": { - "anthropic": { - "url": "https://llm-gateway.corp.example.com/anthropic/v1", - "headers": { - "Authorization": "Bearer anthropic-token-abc123", - "X-Request-Source": "my-ide" - } - } - } - } + "result": {} } ``` -In this example, the agent only uses Anthropic, so it only accepts that provider's configuration. - #### Behavior -1. **Capability discovery**: The agent MUST advertise `llmEndpoints: true` in `agentCapabilities` if it supports the `setLlmEndpoints` method. Clients SHOULD check this capability before calling the method. +1. **Capability discovery**: The agent MUST list its supported providers in `agentCapabilities.llmEndpoints.providers` if it supports the `setLlmEndpoints` method. Clients SHOULD only send endpoint configurations for providers listed there. 2. **Timing**: The `setLlmEndpoints` method MUST be called after `initialize` and before `session/new`. Calling it during an active session is undefined behavior. All subsequent sessions will use the newly configured endpoints. -3. **Confirmation via response**: The agent MUST respond with the `accepted` map containing only the providers it will actually use. If the agent accepts none, the `accepted` map SHOULD be empty. - -4. **Per-process scope**: The endpoint configuration applies to the entire agent process lifetime. It should not be stored to disk or persist beyond the process. +3. **Per-process scope**: The endpoint configuration applies to the entire agent process lifetime. It should not be stored to disk or persist beyond the process. -5. **Provider-based routing**: The agent should route LLM requests to the appropriate endpoint based on the provider. If the agent uses a provider not in the provided map, it uses its default endpoint for that provider. +4. **Provider-based routing**: The agent should route LLM requests to the appropriate endpoint based on the provider. If the agent uses a provider not in the provided map, it uses its default endpoint for that provider. -6. **Agent discretion**: If an agent cannot support custom endpoints (e.g., uses a proprietary API), it should not advertise the `llmEndpoints` capability. +5. **Agent discretion**: If an agent cannot support custom endpoints (e.g., uses a proprietary API), it should omit `llmEndpoints` from capabilities or return an empty providers map. ## Open questions @@ -285,7 +284,7 @@ This approach has significant drawbacks: ### What if the agent doesn't support custom endpoints? -If the agent doesn't support custom endpoints, it will not advertise `llmEndpoints: true` in `agentCapabilities` during initialization. The client can detect this and choose an alternative authorization and configuration strategy, or proceed using the agent's default endpoints. +If the agent doesn't support custom endpoints, `llmEndpoints` will be absent from `agentCapabilities` (or its `providers` map will be empty). The client can detect this and choose an alternative authorization and configuration strategy, or proceed using the agent's default endpoints. ## Revision history From a9c4fadf341c54da7ce5d534657d00a88f598e1a Mon Sep 17 00:00:00 2001 From: Evgeniy Stepanov Date: Thu, 5 Mar 2026 19:50:56 +0200 Subject: [PATCH 06/15] docs: update custom LLM endpoint RFD - wording --- docs/rfds/custom-llm-endpoint.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/rfds/custom-llm-endpoint.mdx b/docs/rfds/custom-llm-endpoint.mdx index 773b5eda..dd98eaf8 100644 --- a/docs/rfds/custom-llm-endpoint.mdx +++ b/docs/rfds/custom-llm-endpoint.mdx @@ -237,7 +237,7 @@ interface SetLlmEndpointsResponse { 1. **Capability discovery**: The agent MUST list its supported providers in `agentCapabilities.llmEndpoints.providers` if it supports the `setLlmEndpoints` method. Clients SHOULD only send endpoint configurations for providers listed there. -2. **Timing**: The `setLlmEndpoints` method MUST be called after `initialize` and before `session/new`. Calling it during an active session is undefined behavior. All subsequent sessions will use the newly configured endpoints. +2. **Timing**: The `setLlmEndpoints` method MUST be called after `initialize` and before `session/new`. Calling this MAY NOT affect currently running sessions. Agents MUST apply these settings to any sessions created or loaded after this has been called. 3. **Per-process scope**: The endpoint configuration applies to the entire agent process lifetime. It should not be stored to disk or persist beyond the process. From 4091d1dd94ba030f3d8f39090f216d2fa04cae8c Mon Sep 17 00:00:00 2001 From: Evgeniy Stepanov Date: Sat, 7 Mar 2026 15:24:14 +0200 Subject: [PATCH 07/15] docs: rename llm provider to llm protocol in custom LLM endpoint RFD --- docs/rfds/custom-llm-endpoint.mdx | 59 +++++++++++++++++-------------- 1 file changed, 32 insertions(+), 27 deletions(-) diff --git a/docs/rfds/custom-llm-endpoint.mdx b/docs/rfds/custom-llm-endpoint.mdx index dd98eaf8..6647fafc 100644 --- a/docs/rfds/custom-llm-endpoint.mdx +++ b/docs/rfds/custom-llm-endpoint.mdx @@ -10,7 +10,7 @@ title: "Custom LLM Endpoint Configuration" > What are you proposing to change? -Add the ability for clients to pass custom LLM endpoint URLs and authentication credentials to agents via a dedicated `setLlmEndpoints` method, with support for multiple providers. This allows clients to route LLM requests through their own infrastructure (proxies, gateways, or self-hosted models) without agents needing to know about this configuration in advance. +Add the ability for clients to pass custom LLM endpoint URLs and authentication credentials to agents via a dedicated `setLlmEndpoints` method, with support for multiple LLM protocols. This allows clients to route LLM requests through their own infrastructure (proxies, gateways, or self-hosted models) without agents needing to know about this configuration in advance. ## Status quo @@ -30,8 +30,8 @@ Currently, agents are configured with their own LLM endpoints and credentials, t Clients will be able to: 1. Discover whether an agent supports custom LLM endpoints via capabilities during initialization 2. Perform agent configuration, including authorization based on this knowledge -3. Pass custom LLM endpoint URLs and headers for different providers via a dedicated method -4. Have agent LLM requests automatically routed through the appropriate endpoint based on provider +3. Pass custom LLM endpoint URLs and headers for different LLM protocols via a dedicated method +4. Have agent LLM requests automatically routed through the appropriate endpoint based on the LLM protocol ## Implementation details and plan @@ -48,9 +48,9 @@ sequenceDiagram Client->>Agent: initialize Note right of Agent: Agent reports capabilities,
including llmEndpoints support - Agent-->>Client: initialize response
(agentCapabilities.llmEndpoints.providers) + Agent-->>Client: initialize response
(agentCapabilities.llmEndpoints.protocols) - Note over Client: Client sees supported providers.
Performs configuration / authorization
based on this knowledge. + Note over Client: Client sees supported protocols.
Performs configuration / authorization
based on this knowledge. Client->>Agent: setLlmEndpoints Agent-->>Client: setLlmEndpoints response @@ -59,25 +59,30 @@ sequenceDiagram Client->>Agent: session/new ``` -1. **Initialization**: The client calls `initialize`. The agent responds with its capabilities, including an `llmEndpoints` object listing supported providers. -2. **Client-side decision**: The client inspects the supported providers list. If the agent lists providers in `llmEndpoints.providers`, the client can perform authorization, resolve credentials, or configure endpoints for those specific providers. If `llmEndpoints` is absent or the providers list is empty, the client falls back to a different authorization and configuration strategy. -3. **Endpoint configuration**: The client calls `setLlmEndpoints` with endpoint configurations for the supported providers. +1. **Initialization**: The client calls `initialize`. The agent responds with its capabilities, including an `llmEndpoints` object listing supported LLM protocols. +2. **Client-side decision**: The client inspects the supported protocols list. If the agent lists protocols in `llmEndpoints.protocols`, the client can perform authorization, resolve credentials, or configure endpoints for those specific protocols. If `llmEndpoints` is absent or the protocols list is empty, the client falls back to a different authorization and configuration strategy. +3. **Endpoint configuration**: The client calls `setLlmEndpoints` with endpoint configurations for the supported protocols. 4. **Session creation**: The client proceeds to create a session. ### Capability advertisement -The agent advertises support for custom LLM endpoints and lists its supported providers via a new `llmEndpoints` capability in `agentCapabilities`: +The agent advertises support for custom LLM endpoints and lists its supported LLM protocols via a new `llmEndpoints` capability in `agentCapabilities`: ```typescript -/** Well-known LLM provider identifiers */ -type LlmProvider = "anthropic" | "openai" | "google" | "amazon" | "will be added later"; +/** + * Well-known LLM protocol identifiers. + * Each identifier represents an API compatibility level, not a specific vendor. + * For example, "openai" means any endpoint implementing the OpenAI-compatible API + * (including proxies, gateways, and self-hosted servers like vLLM or Ollama). + */ +type LlmProtocol = "anthropic" | "openai" | "google" | "amazon" | "will be added later"; interface LlmEndpointsCapability { /** - * Map of supported provider identifiers. - * The client should only configure endpoints for providers listed here. + * Map of supported LLM protocol identifiers. + * The client should only configure endpoints for protocols listed here. */ - providers: Record; + protocols: Record; /** Extension metadata */ _meta?: Record; @@ -88,8 +93,8 @@ interface AgentCapabilities { /** * Custom LLM endpoint support. - * If present with a non-empty providers map, the agent supports the setLlmEndpoints method. - * If absent or providers is empty, the agent does not support custom endpoints. + * If present with a non-empty protocols map, the agent supports the setLlmEndpoints method. + * If absent or protocols is empty, the agent does not support custom endpoints. */ llmEndpoints?: LlmEndpointsCapability; } @@ -108,7 +113,7 @@ interface AgentCapabilities { }, "agentCapabilities": { "llmEndpoints": { - "providers": { + "protocols": { "anthropic": {}, "openai": {} } @@ -141,12 +146,12 @@ interface LlmEndpointConfig { interface SetLlmEndpointsRequest { /** - * Custom LLM endpoint configurations per provider. + * Custom LLM endpoint configurations per LLM protocol. * When provided, the agent should route LLM requests to the appropriate endpoint - * based on the provider being used. + * based on the protocol being used. * This configuration is per-process and should not be persisted to disk. */ - endpoints: Record; + endpoints: Record; /** Extension metadata */ _meta?: Record; @@ -186,7 +191,7 @@ interface SetLlmEndpointsResponse { "type": "object" }, "LlmEndpoints": { - "description": "Map of provider identifiers to endpoint configurations. This configuration is per-process and should not be persisted to disk.", + "description": "Map of LLM protocol identifiers to endpoint configurations. This configuration is per-process and should not be persisted to disk.", "type": "object", "additionalProperties": { "$ref": "#/$defs/LlmEndpointConfig" @@ -235,21 +240,21 @@ interface SetLlmEndpointsResponse { #### Behavior -1. **Capability discovery**: The agent MUST list its supported providers in `agentCapabilities.llmEndpoints.providers` if it supports the `setLlmEndpoints` method. Clients SHOULD only send endpoint configurations for providers listed there. +1. **Capability discovery**: The agent MUST list its supported protocols in `agentCapabilities.llmEndpoints.protocols` if it supports the `setLlmEndpoints` method. Clients SHOULD only send endpoint configurations for protocols listed there. 2. **Timing**: The `setLlmEndpoints` method MUST be called after `initialize` and before `session/new`. Calling this MAY NOT affect currently running sessions. Agents MUST apply these settings to any sessions created or loaded after this has been called. 3. **Per-process scope**: The endpoint configuration applies to the entire agent process lifetime. It should not be stored to disk or persist beyond the process. -4. **Provider-based routing**: The agent should route LLM requests to the appropriate endpoint based on the provider. If the agent uses a provider not in the provided map, it uses its default endpoint for that provider. +4. **Protocol-based routing**: The agent should route LLM requests to the appropriate endpoint based on the LLM protocol. If the agent uses a protocol not in the provided map, it uses its default endpoint for that protocol. -5. **Agent discretion**: If an agent cannot support custom endpoints (e.g., uses a proprietary API), it should omit `llmEndpoints` from capabilities or return an empty providers map. +5. **Agent discretion**: If an agent cannot support custom endpoints (e.g., uses a proprietary API), it should omit `llmEndpoints` from capabilities or return an empty protocols map. ## Open questions -### How should provider identifiers be standardized? +### How should protocol identifiers be standardized? -We need to define a standard set of provider identifiers (e.g., `"anthropic"`, `"openai"`, `"google"`, `"amazon"`). Should this be: +We need to define a standard set of LLM protocol identifiers (e.g., `"anthropic"`, `"openai"`, `"google"`, `"amazon"`). Should this be: - A fixed enum in the protocol specification? - An extensible set with well-known values and support for custom strings? - Defined in a separate registry/document that can be updated independently? @@ -284,7 +289,7 @@ This approach has significant drawbacks: ### What if the agent doesn't support custom endpoints? -If the agent doesn't support custom endpoints, `llmEndpoints` will be absent from `agentCapabilities` (or its `providers` map will be empty). The client can detect this and choose an alternative authorization and configuration strategy, or proceed using the agent's default endpoints. +If the agent doesn't support custom endpoints, `llmEndpoints` will be absent from `agentCapabilities` (or its `protocols` map will be empty). The client can detect this and choose an alternative authorization and configuration strategy, or proceed using the agent's default endpoints. ## Revision history From d5fbf9676247defe7a77cd90b3a37b0482deb8f5 Mon Sep 17 00:00:00 2001 From: Evgeniy Stepanov Date: Sat, 7 Mar 2026 15:35:39 +0200 Subject: [PATCH 08/15] docs: make LlmProtocol an open string type with well-known values in custom LLM endpoint RFD --- docs/rfds/custom-llm-endpoint.mdx | 47 +++++++++++++++++++++++++------ 1 file changed, 39 insertions(+), 8 deletions(-) diff --git a/docs/rfds/custom-llm-endpoint.mdx b/docs/rfds/custom-llm-endpoint.mdx index 6647fafc..08dc1a0e 100644 --- a/docs/rfds/custom-llm-endpoint.mdx +++ b/docs/rfds/custom-llm-endpoint.mdx @@ -70,12 +70,14 @@ The agent advertises support for custom LLM endpoints and lists its supported LL ```typescript /** - * Well-known LLM protocol identifiers. - * Each identifier represents an API compatibility level, not a specific vendor. + * LLM protocol identifier representing an API compatibility level, not a specific vendor. * For example, "openai" means any endpoint implementing the OpenAI-compatible API * (including proxies, gateways, and self-hosted servers like vLLM or Ollama). + * + * Well-known values: "anthropic", "openai", "google", "amazon". + * Custom protocol identifiers are allowed for regional or emerging LLM APIs not covered by the well-known set. */ -type LlmProtocol = "anthropic" | "openai" | "google" | "amazon" | "will be added later"; +type LlmProtocol = string; interface LlmEndpointsCapability { /** @@ -252,12 +254,9 @@ interface SetLlmEndpointsResponse { ## Open questions -### How should protocol identifiers be standardized? +### ~~How should protocol identifiers be standardized?~~ Resolved -We need to define a standard set of LLM protocol identifiers (e.g., `"anthropic"`, `"openai"`, `"google"`, `"amazon"`). Should this be: -- A fixed enum in the protocol specification? -- An extensible set with well-known values and support for custom strings? -- Defined in a separate registry/document that can be updated independently? +Protocol identifiers are plain strings with a set of well-known values (`"anthropic"`, `"openai"`, `"google"`, `"amazon"`). Custom identifiers are allowed for regional or emerging LLM APIs not covered by the well-known set. Agents advertise the protocol identifiers they understand; clients match against them. ### How should model availability be handled? @@ -267,6 +266,38 @@ When a custom endpoint is provided, it may only support a subset of models. For > What questions have arisen over the course of authoring this document? +### Why is `LlmProtocol` a plain string instead of a fixed enum? + +The protocol wire format uses plain strings to keep the specification stable and avoid blocking adoption of new LLM APIs. A fixed enum would require a spec update every time a new protocol emerges. + +Instead, well-known protocol identifiers are provided by SDKs as convenience constants: + +**TypeScript** — an open string type with predefined values: +```typescript +type LlmProtocol = string & {}; + +const LlmProtocols = { + Anthropic: "anthropic", + OpenAI: "openai", + Google: "google", + Amazon: "amazon", +} as const; +``` + +**Kotlin** — a string wrapper with predefined companion objects: +```kotlin +value class LlmProtocol(val value: String) { + companion object { + val Anthropic = LlmProtocol("anthropic") + val OpenAI = LlmProtocol("openai") + val Google = LlmProtocol("google") + val Amazon = LlmProtocol("amazon") + } +} +``` + +This way SDK users get discoverability and autocomplete for well-known protocols while remaining free to use any custom string for protocols not yet in the predefined set. + ### Why not pass endpoints in the `initialize` request? Passing endpoints directly in `initialize` would require the client to have already resolved credentials and configured endpoints before knowing whether the agent supports this feature. In practice, the client needs to inspect the agent's capabilities first to decide its authorization strategy — for example, whether to route through a corporate proxy or use direct credentials. A dedicated method after initialization solves this chicken-and-egg problem and keeps capability negotiation separate from endpoint configuration. From 09a8de0ee3813701fb0595932f2224b414a65484 Mon Sep 17 00:00:00 2001 From: Evgeniy Stepanov Date: Sat, 7 Mar 2026 17:58:16 +0200 Subject: [PATCH 09/15] docs: resolve model availability open question in custom LLM endpoint RFD --- docs/rfds/custom-llm-endpoint.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/rfds/custom-llm-endpoint.mdx b/docs/rfds/custom-llm-endpoint.mdx index 08dc1a0e..b1b22cf5 100644 --- a/docs/rfds/custom-llm-endpoint.mdx +++ b/docs/rfds/custom-llm-endpoint.mdx @@ -258,9 +258,9 @@ interface SetLlmEndpointsResponse { Protocol identifiers are plain strings with a set of well-known values (`"anthropic"`, `"openai"`, `"google"`, `"amazon"`). Custom identifiers are allowed for regional or emerging LLM APIs not covered by the well-known set. Agents advertise the protocol identifiers they understand; clients match against them. -### How should model availability be handled? +### ~~How should model availability be handled?~~ Resolved -When a custom endpoint is provided, it may only support a subset of models. For example, a self-hosted vLLM server might only have `llama-3-70b` available, while the agent normally advertises `claude-3-opus`, `gpt-4`, etc. +Model availability is not a concern of this RFD. Most agents can discover available models from the configured endpoint at runtime. Agents that cannot should report an error if a user selects a model not supported by the custom endpoint. ## Frequently asked questions From fd3aa2c95cb643827e099080c811c2fd29f1a5b5 Mon Sep 17 00:00:00 2001 From: Evgeniy Stepanov Date: Sat, 7 Mar 2026 18:13:48 +0200 Subject: [PATCH 10/15] docs: update revision history in custom LLM endpoint RFD --- docs/rfds/custom-llm-endpoint.mdx | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/rfds/custom-llm-endpoint.mdx b/docs/rfds/custom-llm-endpoint.mdx index b1b22cf5..8249656f 100644 --- a/docs/rfds/custom-llm-endpoint.mdx +++ b/docs/rfds/custom-llm-endpoint.mdx @@ -4,8 +4,6 @@ title: "Custom LLM Endpoint Configuration" - Author(s): [@anna239](https://github.com/anna239), [@xtmq](https://github.com/xtmq) -> **Note:** This RFD is very preliminary and intended to start a dialog about this feature. The proposed design may change significantly based on feedback and further discussion. - ## Elevator pitch > What are you proposing to change? @@ -324,5 +322,6 @@ If the agent doesn't support custom endpoints, `llmEndpoints` will be absent fro ## Revision history +- 2026-03-07: Rename "provider" to "protocol" to reflect API compatibility level; make `LlmProtocol` an open string type with well-known values; resolve open questions on identifier standardization and model availability - 2026-03-04: Revised to use dedicated `setLlmEndpoints` method with capability advertisement - 2026-02-02: Initial draft - preliminary proposal to start discussion From 8e93892a1fc50379e2ca03675703a49800c361c6 Mon Sep 17 00:00:00 2001 From: Evgeniy Stepanov Date: Mon, 9 Mar 2026 10:41:04 +0200 Subject: [PATCH 11/15] docs: formatting for custom LLM endpoint RFD --- docs/rfds/custom-llm-endpoint.mdx | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/rfds/custom-llm-endpoint.mdx b/docs/rfds/custom-llm-endpoint.mdx index 8249656f..3b6a9532 100644 --- a/docs/rfds/custom-llm-endpoint.mdx +++ b/docs/rfds/custom-llm-endpoint.mdx @@ -26,6 +26,7 @@ Currently, agents are configured with their own LLM endpoints and credentials, t > How will things play out once this feature exists? Clients will be able to: + 1. Discover whether an agent supports custom LLM endpoints via capabilities during initialization 2. Perform agent configuration, including authorization based on this knowledge 3. Pass custom LLM endpoint URLs and headers for different LLM protocols via a dedicated method @@ -101,6 +102,7 @@ interface AgentCapabilities { ``` **Initialize Response example:** + ```json { "jsonrpc": "2.0", @@ -204,6 +206,7 @@ interface SetLlmEndpointsResponse { #### Example Exchange **setLlmEndpoints Request:** + ```json { "jsonrpc": "2.0", @@ -230,6 +233,7 @@ interface SetLlmEndpointsResponse { ``` **setLlmEndpoints Response:** + ```json { "jsonrpc": "2.0", @@ -271,6 +275,7 @@ The protocol wire format uses plain strings to keep the specification stable and Instead, well-known protocol identifiers are provided by SDKs as convenience constants: **TypeScript** — an open string type with predefined values: + ```typescript type LlmProtocol = string & {}; @@ -283,6 +288,7 @@ const LlmProtocols = { ``` **Kotlin** — a string wrapper with predefined companion objects: + ```kotlin value class LlmProtocol(val value: String) { companion object { @@ -311,6 +317,7 @@ Many agents throw authentication errors before the model selection happens. This One option would be to pass endpoint configuration via environment variables (like `OPENAI_API_BASE`) or command-line arguments when starting the agent process. This approach has significant drawbacks: + - With multiple providers, the configuration becomes complex JSON that is awkward to pass via command-line arguments - Environment variables may be logged or visible to other processes, creating security concerns - Requires knowledge of agent-specific variable names or argument formats From 5eec689fe6d8bc7ceaee7906f9e585b406f29af9 Mon Sep 17 00:00:00 2001 From: Evgeniy Stepanov Date: Sat, 21 Mar 2026 13:05:55 +0200 Subject: [PATCH 12/15] docs(rfd): redesign custom LLM endpoint proposal into providers list/set/remove API --- docs/rfds/custom-llm-endpoint.mdx | 400 +++++++++++++++--------------- 1 file changed, 201 insertions(+), 199 deletions(-) diff --git a/docs/rfds/custom-llm-endpoint.mdx b/docs/rfds/custom-llm-endpoint.mdx index 3b6a9532..437454bc 100644 --- a/docs/rfds/custom-llm-endpoint.mdx +++ b/docs/rfds/custom-llm-endpoint.mdx @@ -1,5 +1,5 @@ --- -title: "Custom LLM Endpoint Configuration" +title: "Configurable LLM Providers" --- - Author(s): [@anna239](https://github.com/anna239), [@xtmq](https://github.com/xtmq) @@ -8,18 +8,33 @@ title: "Custom LLM Endpoint Configuration" > What are you proposing to change? -Add the ability for clients to pass custom LLM endpoint URLs and authentication credentials to agents via a dedicated `setLlmEndpoints` method, with support for multiple LLM protocols. This allows clients to route LLM requests through their own infrastructure (proxies, gateways, or self-hosted models) without agents needing to know about this configuration in advance. +Add the ability for clients to discover and configure agent LLM providers (identified by `id`) via dedicated provider methods: + +- `providers/list` +- `providers/set` +- `providers/remove` + +This allows clients to route LLM requests through their own infrastructure (proxies, gateways, or self-hosted models) without agents needing to know about this configuration in advance. ## Status quo > How do things work today and what problems does this cause? Why would we change things? -Currently, agents are configured with their own LLM endpoints and credentials, typically through environment variables or configuration files. This creates problems for: +ACP does not currently define a standard method for configuring LLM providers. + +In practice, provider configuration is usually done via environment variables or agent-specific config files. That creates several problems: + +- No standard way for clients to discover what providers an agent exposes +- No standard way to update one specific provider by id +- No standard way to remove provider configuration at runtime +- Secret-bearing values in headers are difficult to handle safely when configuration must be round-tripped -- **Client proxies**: Clients want to route agent traffic through their own proxies, f.i. for setting additional headers or logging -- **Enterprise deployments**: Organizations want to route LLM traffic through their own proxies for compliance, logging, or cost management -- **Self-hosted models**: Users running local LLM servers (vLLM, Ollama, etc.) cannot easily redirect agent traffic -- **API gateways**: Organizations using LLM gateways for rate limiting, caching, or multi-provider routing +This particularly affects: + +- **Client proxies**: clients want to route agent traffic through their own proxies, for example to add headers or logging +- **Enterprise deployments**: organizations want to route LLM traffic through internal gateways for compliance, logging, and cost controls +- **Self-hosted models**: users running local servers (vLLM, Ollama, etc.) need to redirect agent traffic to local infrastructure +- **API gateways**: organizations using multi-provider routing, rate limiting, and caching need standardized endpoint configuration ## Shiny future @@ -27,10 +42,12 @@ Currently, agents are configured with their own LLM endpoints and credentials, t Clients will be able to: -1. Discover whether an agent supports custom LLM endpoints via capabilities during initialization -2. Perform agent configuration, including authorization based on this knowledge -3. Pass custom LLM endpoint URLs and headers for different LLM protocols via a dedicated method -4. Have agent LLM requests automatically routed through the appropriate endpoint based on the LLM protocol +1. Understand whether an agent supports client-managed LLM routing +2. See where the agent is currently sending LLM requests (for example in settings UI) +3. Route agent LLM traffic through their own infrastructure (enterprise proxy, gateway, self-hosted stack) +4. Update routing settings from the client instead of relying on agent-specific env vars +5. Remove custom routing when needed and return to agent defaults +6. Apply these settings before starting new work in sessions ## Implementation details and plan @@ -38,297 +55,282 @@ Clients will be able to: ### Intended flow -The design uses a two-step approach: capability discovery during initialization, followed by endpoint configuration via a dedicated method. This enables the following flow: - ```mermaid sequenceDiagram participant Client participant Agent Client->>Agent: initialize - Note right of Agent: Agent reports capabilities,
including llmEndpoints support - Agent-->>Client: initialize response
(agentCapabilities.llmEndpoints.protocols) + Agent-->>Client: initialize response (agentCapabilities.providers = true) - Note over Client: Client sees supported protocols.
Performs configuration / authorization
based on this knowledge. + Client->>Agent: providers/list + Agent-->>Client: providers/list response - Client->>Agent: setLlmEndpoints - Agent-->>Client: setLlmEndpoints response + Client->>Agent: providers/set (id = "main") + Agent-->>Client: providers/set response + + Client->>Agent: providers/remove (optional) + Agent-->>Client: providers/remove response - Note over Client,Agent: Ready for session setup Client->>Agent: session/new ``` -1. **Initialization**: The client calls `initialize`. The agent responds with its capabilities, including an `llmEndpoints` object listing supported LLM protocols. -2. **Client-side decision**: The client inspects the supported protocols list. If the agent lists protocols in `llmEndpoints.protocols`, the client can perform authorization, resolve credentials, or configure endpoints for those specific protocols. If `llmEndpoints` is absent or the protocols list is empty, the client falls back to a different authorization and configuration strategy. -3. **Endpoint configuration**: The client calls `setLlmEndpoints` with endpoint configurations for the supported protocols. -4. **Session creation**: The client proceeds to create a session. +1. Client initializes and checks `agentCapabilities.providers`. +2. Client calls `providers/list` to discover available providers, their current routing targets, and supported protocol types. +3. Client calls `providers/set` to apply new (required) configuration for a specific provider id. +4. Client may call `providers/remove` when a provider configuration should be removed. +5. Client creates or loads sessions. ### Capability advertisement -The agent advertises support for custom LLM endpoints and lists its supported LLM protocols via a new `llmEndpoints` capability in `agentCapabilities`: +The agent advertises support with a single boolean capability: ```typescript -/** - * LLM protocol identifier representing an API compatibility level, not a specific vendor. - * For example, "openai" means any endpoint implementing the OpenAI-compatible API - * (including proxies, gateways, and self-hosted servers like vLLM or Ollama). - * - * Well-known values: "anthropic", "openai", "google", "amazon". - * Custom protocol identifiers are allowed for regional or emerging LLM APIs not covered by the well-known set. - */ -type LlmProtocol = string; - -interface LlmEndpointsCapability { +interface AgentCapabilities { + // ... existing fields ... + /** - * Map of supported LLM protocol identifiers. - * The client should only configure endpoints for protocols listed here. + * Provider configuration support. + * If true, the agent supports providers/list, providers/set, and providers/remove. */ - protocols: Record; + providers?: boolean; +} +``` + +If `providers` is absent or `false`, clients must treat provider methods as unsupported. + +### Types + +```typescript +/** Well-known API protocol identifiers. */ +type LlmProtocol = "anthropic" | "openai" | "azure" | "vertex" | "bedrock"; + +interface ProviderInfo { + /** Provider identifier, for example "main" or "openai". */ + id: string; + + /** Protocol currently used by this provider. */ + apiType: LlmProtocol; + + /** Base URL currently used by this provider. */ + baseUrl: string; + + /** Supported protocol types for this provider. */ + supported: LlmProtocol[]; /** Extension metadata */ _meta?: Record; } +``` -interface AgentCapabilities { - // ... existing fields ... +### `providers/list` - /** - * Custom LLM endpoint support. - * If present with a non-empty protocols map, the agent supports the setLlmEndpoints method. - * If absent or protocols is empty, the agent does not support custom endpoints. - */ - llmEndpoints?: LlmEndpointsCapability; +```typescript +interface ProvidersListRequest { + /** Extension metadata */ + _meta?: Record; } -``` -**Initialize Response example:** +interface ProvidersListResponse { + /** Configurable providers with current routing info suitable for UI display. */ + providers: ProviderInfo[]; -```json -{ - "jsonrpc": "2.0", - "id": 0, - "result": { - "protocolVersion": 1, - "agentInfo": { - "name": "MyAgent", - "version": "2.0.0" - }, - "agentCapabilities": { - "llmEndpoints": { - "protocols": { - "anthropic": {}, - "openai": {} - } - }, - "sessionCapabilities": {} - } - } + /** Extension metadata */ + _meta?: Record; } ``` -### `setLlmEndpoints` method +### `providers/set` -A dedicated method that can be called after initialization but before session creation. +`providers/set` updates the full configuration for one provider id. ```typescript -interface LlmEndpointConfig { - /** Base URL for LLM API requests (e.g., "https://llm-proxy.corp.example.com/v1") */ - url: string; +interface ProvidersSetRequest { + /** Provider id to configure. */ + id: string; + + /** Protocol type for this provider. */ + apiType: LlmProtocol; + + /** Base URL for requests sent through this provider. */ + baseUrl: string; /** - * Additional HTTP headers to include in LLM API requests. - * Each entry is a header name mapped to its value. - * Common use cases include Authorization, custom routing, or tracing headers. + * Full headers map for this provider. + * May include authorization, routing, or other integration-specific headers. */ - headers?: Record | null; + headers: Record; /** Extension metadata */ _meta?: Record; } -interface SetLlmEndpointsRequest { - /** - * Custom LLM endpoint configurations per LLM protocol. - * When provided, the agent should route LLM requests to the appropriate endpoint - * based on the protocol being used. - * This configuration is per-process and should not be persisted to disk. - */ - endpoints: Record; +interface ProvidersSetResponse { + /** Extension metadata */ + _meta?: Record; +} +``` + +### `providers/remove` + +```typescript +interface ProvidersRemoveRequest { + /** Provider id to remove. */ + id: string; /** Extension metadata */ _meta?: Record; } -interface SetLlmEndpointsResponse { +interface ProvidersRemoveResponse { /** Extension metadata */ _meta?: Record; } ``` -#### JSON Schema Additions +### Example exchange + +**initialize Response:** ```json { - "$defs": { - "LlmEndpointConfig": { - "description": "Configuration for a custom LLM endpoint.", - "properties": { - "url": { - "type": "string", - "description": "Base URL for LLM API requests." - }, - "headers": { - "type": ["object", "null"], - "description": "Additional HTTP headers to include in LLM API requests.", - "additionalProperties": { - "type": "string" - } - }, - "_meta": { - "additionalProperties": true, - "type": ["object", "null"] - } - }, - "required": ["url"], - "type": "object" + "jsonrpc": "2.0", + "id": 0, + "result": { + "protocolVersion": 1, + "agentInfo": { + "name": "MyAgent", + "version": "2.0.0" }, - "LlmEndpoints": { - "description": "Map of LLM protocol identifiers to endpoint configurations. This configuration is per-process and should not be persisted to disk.", - "type": "object", - "additionalProperties": { - "$ref": "#/$defs/LlmEndpointConfig" - } + "agentCapabilities": { + "providers": true, + "sessionCapabilities": {} } } } ``` -#### Example Exchange +**providers/list Request:** -**setLlmEndpoints Request:** +```json +{ + "jsonrpc": "2.0", + "id": 1, + "method": "providers/list", + "params": {} +} +``` + +**providers/list Response:** ```json { "jsonrpc": "2.0", - "id": 2, - "method": "setLlmEndpoints", - "params": { - "endpoints": { - "anthropic": { - "url": "https://llm-gateway.corp.example.com/anthropic/v1", - "headers": { - "Authorization": "Bearer anthropic-token-abc123", - "X-Request-Source": "my-ide" - } + "id": 1, + "result": { + "providers": [ + { + "id": "main", + "apiType": "anthropic", + "baseUrl": "http://localhost/anthropic", + "supported": ["bedrock", "vertex", "azure", "anthropic"] }, - "openai": { - "url": "https://llm-gateway.corp.example.com/openai/v1", - "headers": { - "Authorization": "Bearer openai-token-xyz789" - } + { + "id": "openai", + "apiType": "openai", + "baseUrl": "http://localhost/openai", + "supported": ["openai"] } - } + ] } } ``` -**setLlmEndpoints Response:** +**providers/set Request:** ```json { "jsonrpc": "2.0", "id": 2, - "result": {} + "method": "providers/set", + "params": { + "id": "main", + "apiType": "anthropic", + "baseUrl": "https://llm-gateway.corp.example.com/anthropic/v1", + "headers": { + "X-Request-Source": "my-ide" + } + } } ``` -#### Behavior - -1. **Capability discovery**: The agent MUST list its supported protocols in `agentCapabilities.llmEndpoints.protocols` if it supports the `setLlmEndpoints` method. Clients SHOULD only send endpoint configurations for protocols listed there. - -2. **Timing**: The `setLlmEndpoints` method MUST be called after `initialize` and before `session/new`. Calling this MAY NOT affect currently running sessions. Agents MUST apply these settings to any sessions created or loaded after this has been called. - -3. **Per-process scope**: The endpoint configuration applies to the entire agent process lifetime. It should not be stored to disk or persist beyond the process. - -4. **Protocol-based routing**: The agent should route LLM requests to the appropriate endpoint based on the LLM protocol. If the agent uses a protocol not in the provided map, it uses its default endpoint for that protocol. - -5. **Agent discretion**: If an agent cannot support custom endpoints (e.g., uses a proprietary API), it should omit `llmEndpoints` from capabilities or return an empty protocols map. - -## Open questions - -### ~~How should protocol identifiers be standardized?~~ Resolved - -Protocol identifiers are plain strings with a set of well-known values (`"anthropic"`, `"openai"`, `"google"`, `"amazon"`). Custom identifiers are allowed for regional or emerging LLM APIs not covered by the well-known set. Agents advertise the protocol identifiers they understand; clients match against them. +**providers/set Response:** -### ~~How should model availability be handled?~~ Resolved - -Model availability is not a concern of this RFD. Most agents can discover available models from the configured endpoint at runtime. Agents that cannot should report an error if a user selects a model not supported by the custom endpoint. - -## Frequently asked questions - -> What questions have arisen over the course of authoring this document? - -### Why is `LlmProtocol` a plain string instead of a fixed enum? - -The protocol wire format uses plain strings to keep the specification stable and avoid blocking adoption of new LLM APIs. A fixed enum would require a spec update every time a new protocol emerges. - -Instead, well-known protocol identifiers are provided by SDKs as convenience constants: +```json +{ + "jsonrpc": "2.0", + "id": 2, + "result": {} +} +``` -**TypeScript** — an open string type with predefined values: +**providers/remove Request:** -```typescript -type LlmProtocol = string & {}; - -const LlmProtocols = { - Anthropic: "anthropic", - OpenAI: "openai", - Google: "google", - Amazon: "amazon", -} as const; +```json +{ + "jsonrpc": "2.0", + "id": 3, + "method": "providers/remove", + "params": { + "id": "openai" + } +} ``` -**Kotlin** — a string wrapper with predefined companion objects: +**providers/remove Response:** -```kotlin -value class LlmProtocol(val value: String) { - companion object { - val Anthropic = LlmProtocol("anthropic") - val OpenAI = LlmProtocol("openai") - val Google = LlmProtocol("google") - val Amazon = LlmProtocol("amazon") - } +```json +{ + "jsonrpc": "2.0", + "id": 3, + "result": {} } ``` -This way SDK users get discoverability and autocomplete for well-known protocols while remaining free to use any custom string for protocols not yet in the predefined set. +### Behavior -### Why not pass endpoints in the `initialize` request? +1. **Capability discovery**: agents that support provider methods MUST advertise `agentCapabilities.providers: true` in `initialize`. Clients SHOULD only call `providers/*` when this capability is present and `true`. +2. **Timing and session impact**: provider methods MUST be called after `initialize`. Clients SHOULD configure providers before creating or loading sessions. Agents MAY choose not to apply changes to already running sessions, but SHOULD apply them to sessions created or loaded after the change. +3. **List semantics**: `providers/list` returns configurable providers, their current routing (`apiType`, `baseUrl`), and supported protocol types. +4. **Set semantics and validation**: `providers/set` replaces the full configuration for the target `id` (`apiType`, `baseUrl`, full `headers`). If `id` is unknown, `apiType` is unsupported for that provider, or params are malformed, agents SHOULD return `invalid_params`. +5. **Remove semantics**: `providers/remove` removes runtime configuration for the target `id`. Removing an unknown `id` SHOULD be treated as success (idempotent behavior). +6. **Scope and persistence**: provider configuration is process-scoped and SHOULD NOT be persisted to disk. -Passing endpoints directly in `initialize` would require the client to have already resolved credentials and configured endpoints before knowing whether the agent supports this feature. In practice, the client needs to inspect the agent's capabilities first to decide its authorization strategy — for example, whether to route through a corporate proxy or use direct credentials. A dedicated method after initialization solves this chicken-and-egg problem and keeps capability negotiation separate from endpoint configuration. +## Frequently asked questions + +> What questions have arisen over the course of authoring this document? -### Why not pass endpoint when selecting a model? +### Why not a single `providers/update` method for full list replacement? -One option would be to pass the endpoint URL and credentials when the user selects a model (e.g., in `session/new` or a model selection method). +A full-list update means the client must send complete configuration (including `headers`) for all providers every time. -Many agents throw authentication errors before the model selection happens. This makes the flow unreliable. +If the client wants to change only one provider, it may not know headers for the others. In that case it cannot safely build a correct full-list payload. -### Why not use environment variables or command-line arguments? +Also, `providers/list` does not return headers, so the client cannot simply "take what the agent returned" and send it back with one edit. -One option would be to pass endpoint configuration via environment variables (like `OPENAI_API_BASE`) or command-line arguments when starting the agent process. +Per-provider methods (`set` and `remove`) avoid this problem and keep updates explicit. -This approach has significant drawbacks: +### Why doesn't `providers/list` return headers? -- With multiple providers, the configuration becomes complex JSON that is awkward to pass via command-line arguments -- Environment variables may be logged or visible to other processes, creating security concerns -- Requires knowledge of agent-specific variable names or argument formats -- No standardized way to confirm the agent accepted the configuration +Header values may contain secrets and should not be echoed by the agent. `providers/list` is intentionally limited to non-secret routing information (`apiType`, `baseUrl`). -### What if the agent doesn't support custom endpoints? +### Why is this separate from `initialize` params? -If the agent doesn't support custom endpoints, `llmEndpoints` will be absent from `agentCapabilities` (or its `protocols` map will be empty). The client can detect this and choose an alternative authorization and configuration strategy, or proceed using the agent's default endpoints. +Clients need capability discovery first, then provider discovery, then configuration. A dedicated method family keeps initialization focused on negotiation and leaves provider mutation to explicit steps. ## Revision history +- 2026-03-21: Initial draft of provider configuration API (`providers/list`, `providers/set`, `providers/remove`) - 2026-03-07: Rename "provider" to "protocol" to reflect API compatibility level; make `LlmProtocol` an open string type with well-known values; resolve open questions on identifier standardization and model availability - 2026-03-04: Revised to use dedicated `setLlmEndpoints` method with capability advertisement - 2026-02-02: Initial draft - preliminary proposal to start discussion From c2a9777b5bb6bb59eab9386191447a77d34ad9a7 Mon Sep 17 00:00:00 2001 From: Evgeniy Stepanov Date: Sun, 22 Mar 2026 00:10:26 +0200 Subject: [PATCH 13/15] docs(rfd): finalized provider disable semantics --- docs/rfds/custom-llm-endpoint.mdx | 100 +++++++++++++++++++++--------- 1 file changed, 69 insertions(+), 31 deletions(-) diff --git a/docs/rfds/custom-llm-endpoint.mdx b/docs/rfds/custom-llm-endpoint.mdx index 437454bc..2bd4b7a4 100644 --- a/docs/rfds/custom-llm-endpoint.mdx +++ b/docs/rfds/custom-llm-endpoint.mdx @@ -12,7 +12,7 @@ Add the ability for clients to discover and configure agent LLM providers (ident - `providers/list` - `providers/set` -- `providers/remove` +- `providers/disable` This allows clients to route LLM requests through their own infrastructure (proxies, gateways, or self-hosted models) without agents needing to know about this configuration in advance. @@ -26,7 +26,7 @@ In practice, provider configuration is usually done via environment variables or - No standard way for clients to discover what providers an agent exposes - No standard way to update one specific provider by id -- No standard way to remove provider configuration at runtime +- No standard way to disable a specific provider at runtime while preserving provider discoverability - Secret-bearing values in headers are difficult to handle safely when configuration must be round-tripped This particularly affects: @@ -46,7 +46,7 @@ Clients will be able to: 2. See where the agent is currently sending LLM requests (for example in settings UI) 3. Route agent LLM traffic through their own infrastructure (enterprise proxy, gateway, self-hosted stack) 4. Update routing settings from the client instead of relying on agent-specific env vars -5. Remove custom routing when needed and return to agent defaults +5. Disable a provider when needed and later re-enable it explicitly 6. Apply these settings before starting new work in sessions ## Implementation details and plan @@ -69,16 +69,16 @@ sequenceDiagram Client->>Agent: providers/set (id = "main") Agent-->>Client: providers/set response - Client->>Agent: providers/remove (optional) - Agent-->>Client: providers/remove response + Client->>Agent: providers/disable (optional) + Agent-->>Client: providers/disable response Client->>Agent: session/new ``` 1. Client initializes and checks `agentCapabilities.providers`. -2. Client calls `providers/list` to discover available providers, their current routing targets, and supported protocol types. +2. Client calls `providers/list` to discover available providers, their current routing targets (or disabled state), supported protocol types, and whether they are required. 3. Client calls `providers/set` to apply new (required) configuration for a specific provider id. -4. Client may call `providers/remove` when a provider configuration should be removed. +4. Client may call `providers/disable` when a non-required provider should be disabled. 5. Client creates or loads sessions. ### Capability advertisement @@ -91,7 +91,7 @@ interface AgentCapabilities { /** * Provider configuration support. - * If true, the agent supports providers/list, providers/set, and providers/remove. + * If true, the agent supports providers/list, providers/set, and providers/disable. */ providers?: boolean; } @@ -105,19 +105,33 @@ If `providers` is absent or `false`, clients must treat provider methods as unsu /** Well-known API protocol identifiers. */ type LlmProtocol = "anthropic" | "openai" | "azure" | "vertex" | "bedrock"; -interface ProviderInfo { - /** Provider identifier, for example "main" or "openai". */ - id: string; - +interface ProviderCurrentConfig { /** Protocol currently used by this provider. */ apiType: LlmProtocol; /** Base URL currently used by this provider. */ baseUrl: string; +} + +interface ProviderInfo { + /** Provider identifier, for example "main" or "openai". */ + id: string; /** Supported protocol types for this provider. */ supported: LlmProtocol[]; + /** + * Whether this provider is mandatory and cannot be disabled via providers/disable. + * If true, clients must not call providers/disable for this id. + */ + required: boolean; + + /** + * Current effective non-secret routing config. + * Null means provider is disabled. + */ + current: ProviderCurrentConfig | null; + /** Extension metadata */ _meta?: Record; } @@ -171,18 +185,18 @@ interface ProvidersSetResponse { } ``` -### `providers/remove` +### `providers/disable` ```typescript -interface ProvidersRemoveRequest { - /** Provider id to remove. */ +interface ProvidersDisableRequest { + /** Provider id to disable. */ id: string; /** Extension metadata */ _meta?: Record; } -interface ProvidersRemoveResponse { +interface ProvidersDisableResponse { /** Extension metadata */ _meta?: Record; } @@ -231,15 +245,18 @@ interface ProvidersRemoveResponse { "providers": [ { "id": "main", - "apiType": "anthropic", - "baseUrl": "http://localhost/anthropic", - "supported": ["bedrock", "vertex", "azure", "anthropic"] + "supported": ["bedrock", "vertex", "azure", "anthropic"], + "required": true, + "current": { + "apiType": "anthropic", + "baseUrl": "http://localhost/anthropic" + } }, { "id": "openai", - "apiType": "openai", - "baseUrl": "http://localhost/openai", - "supported": ["openai"] + "supported": ["openai"], + "required": false, + "current": null } ] } @@ -274,20 +291,20 @@ interface ProvidersRemoveResponse { } ``` -**providers/remove Request:** +**providers/disable Request:** ```json { "jsonrpc": "2.0", "id": 3, - "method": "providers/remove", + "method": "providers/disable", "params": { "id": "openai" } } ``` -**providers/remove Response:** +**providers/disable Response:** ```json { @@ -301,15 +318,29 @@ interface ProvidersRemoveResponse { 1. **Capability discovery**: agents that support provider methods MUST advertise `agentCapabilities.providers: true` in `initialize`. Clients SHOULD only call `providers/*` when this capability is present and `true`. 2. **Timing and session impact**: provider methods MUST be called after `initialize`. Clients SHOULD configure providers before creating or loading sessions. Agents MAY choose not to apply changes to already running sessions, but SHOULD apply them to sessions created or loaded after the change. -3. **List semantics**: `providers/list` returns configurable providers, their current routing (`apiType`, `baseUrl`), and supported protocol types. -4. **Set semantics and validation**: `providers/set` replaces the full configuration for the target `id` (`apiType`, `baseUrl`, full `headers`). If `id` is unknown, `apiType` is unsupported for that provider, or params are malformed, agents SHOULD return `invalid_params`. -5. **Remove semantics**: `providers/remove` removes runtime configuration for the target `id`. Removing an unknown `id` SHOULD be treated as success (idempotent behavior). -6. **Scope and persistence**: provider configuration is process-scoped and SHOULD NOT be persisted to disk. +3. **List semantics**: `providers/list` returns configurable providers, their supported protocol types, current effective routing, and `required` flag. Providers SHOULD remain discoverable in list after `providers/disable`. +4. **Client behavior for required providers**: clients SHOULD NOT call `providers/disable` for providers where `required: true`. +5. **Disabled state encoding**: in `providers/list`, `current: null` means the provider is disabled and MUST NOT be used by the agent for LLM calls. +6. **Set semantics and validation**: `providers/set` replaces the full configuration for the target `id` (`apiType`, `baseUrl`, full `headers`). If `id` is unknown, `apiType` is unsupported for that provider, or params are malformed, agents SHOULD return `invalid_params`. +7. **Disable semantics**: `providers/disable` disables the target provider at runtime. A disabled provider MUST appear in `providers/list` with `current: null`. If target provider has `required: true`, agents MUST return `invalid_params`. Disabling an unknown `id` SHOULD be treated as success (idempotent behavior). +8. **Scope and persistence**: provider configuration is process-scoped and SHOULD NOT be persisted to disk. ## Frequently asked questions > What questions have arisen over the course of authoring this document? +### What does `null` mean in `providers/list`? + +`current: null` means the provider is disabled. + +When disabled, the agent MUST NOT route LLM calls through that provider until the client enables it again with `providers/set`. + +### Why is there a `required` flag? + +Some providers are mandatory for agent operation and must not be disabled. + +`required` lets clients hide or disable the provider-disable action in UI and avoid calling `providers/disable` for those ids. + ### Why not a single `providers/update` method for full list replacement? A full-list update means the client must send complete configuration (including `headers`) for all providers every time. @@ -318,11 +349,17 @@ If the client wants to change only one provider, it may not know headers for the Also, `providers/list` does not return headers, so the client cannot simply "take what the agent returned" and send it back with one edit. -Per-provider methods (`set` and `remove`) avoid this problem and keep updates explicit. +Per-provider methods (`set` and `disable`) avoid this problem and keep updates explicit. ### Why doesn't `providers/list` return headers? -Header values may contain secrets and should not be echoed by the agent. `providers/list` is intentionally limited to non-secret routing information (`apiType`, `baseUrl`). +Header values may contain secrets and should not be echoed by the agent. `providers/list` is intentionally limited to non-secret routing information (`current.apiType`, `current.baseUrl`). + +### Why are `providers/list` and `providers/set` payloads different? + +`providers/set` accepts `headers`, including secrets, and is write-oriented. + +`providers/list` is read-oriented and returns only non-secret routing summary (`current`) for UI and capability discovery. ### Why is this separate from `initialize` params? @@ -330,6 +367,7 @@ Clients need capability discovery first, then provider discovery, then configura ## Revision history +- 2026-03-22: Finalized provider disable semantics - `providers/remove` renamed to `providers/disable`, required providers are non-disableable, and disabled state is represented as `current: null` - 2026-03-21: Initial draft of provider configuration API (`providers/list`, `providers/set`, `providers/remove`) - 2026-03-07: Rename "provider" to "protocol" to reflect API compatibility level; make `LlmProtocol` an open string type with well-known values; resolve open questions on identifier standardization and model availability - 2026-03-04: Revised to use dedicated `setLlmEndpoints` method with capability advertisement From b6cf31900a44b45598849f484eb2ce3798803581 Mon Sep 17 00:00:00 2001 From: Evgeniy Stepanov Date: Sun, 22 Mar 2026 00:24:23 +0200 Subject: [PATCH 14/15] docs(rfd): add FAQ on why providers/* is preferred over session-config --- docs/rfds/custom-llm-endpoint.mdx | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/docs/rfds/custom-llm-endpoint.mdx b/docs/rfds/custom-llm-endpoint.mdx index 2bd4b7a4..a9af0848 100644 --- a/docs/rfds/custom-llm-endpoint.mdx +++ b/docs/rfds/custom-llm-endpoint.mdx @@ -365,6 +365,18 @@ Header values may contain secrets and should not be echoed by the agent. `provid Clients need capability discovery first, then provider discovery, then configuration. A dedicated method family keeps initialization focused on negotiation and leaves provider mutation to explicit steps. +### Why not use `session-config` with a `provider` category instead? + +`session-config` is a possible alternative, and we may revisit it as the spec evolves. + +We did not choose it as the primary approach in this proposal because provider routing here needs dedicated semantics that are difficult to express with today's session config model: + +- Multiple providers identified by `id`, each with its own lifecycle +- Structured payloads (`apiType`, `baseUrl`, full `headers` map) rather than simple scalar values +- Explicit discoverable (`providers/list`) and disable (`providers/disable`) semantics + +Today, `session-config` values are effectively string-oriented and do not define a standard multi-value/structured model for this use case. + ## Revision history - 2026-03-22: Finalized provider disable semantics - `providers/remove` renamed to `providers/disable`, required providers are non-disableable, and disabled state is represented as `current: null` From 840d7be3c5af1e238a92af33cde95e286f5e04a5 Mon Sep 17 00:00:00 2001 From: Ben Brandt Date: Tue, 24 Mar 2026 16:00:43 +0100 Subject: [PATCH 15/15] Align capabilities --- docs/rfds/custom-llm-endpoint.mdx | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/rfds/custom-llm-endpoint.mdx b/docs/rfds/custom-llm-endpoint.mdx index a9af0848..a5b86729 100644 --- a/docs/rfds/custom-llm-endpoint.mdx +++ b/docs/rfds/custom-llm-endpoint.mdx @@ -61,7 +61,7 @@ sequenceDiagram participant Agent Client->>Agent: initialize - Agent-->>Client: initialize response (agentCapabilities.providers = true) + Agent-->>Client: initialize response (agentCapabilities.providers = {}) Client->>Agent: providers/list Agent-->>Client: providers/list response @@ -83,7 +83,7 @@ sequenceDiagram ### Capability advertisement -The agent advertises support with a single boolean capability: +The agent advertises support with an empty object capability: ```typescript interface AgentCapabilities { @@ -91,13 +91,13 @@ interface AgentCapabilities { /** * Provider configuration support. - * If true, the agent supports providers/list, providers/set, and providers/disable. + * If present, the agent supports providers/list, providers/set, and providers/disable. */ - providers?: boolean; + providers?: {}; } ``` -If `providers` is absent or `false`, clients must treat provider methods as unsupported. +If `providers` is absent, clients must treat provider methods as unsupported. ### Types @@ -217,7 +217,7 @@ interface ProvidersDisableResponse { "version": "2.0.0" }, "agentCapabilities": { - "providers": true, + "providers": {}, "sessionCapabilities": {} } } @@ -316,7 +316,7 @@ interface ProvidersDisableResponse { ### Behavior -1. **Capability discovery**: agents that support provider methods MUST advertise `agentCapabilities.providers: true` in `initialize`. Clients SHOULD only call `providers/*` when this capability is present and `true`. +1. **Capability discovery**: agents that support provider methods MUST advertise `agentCapabilities.providers: {}` in `initialize`. Clients SHOULD only call `providers/*` when this capability is present. 2. **Timing and session impact**: provider methods MUST be called after `initialize`. Clients SHOULD configure providers before creating or loading sessions. Agents MAY choose not to apply changes to already running sessions, but SHOULD apply them to sessions created or loaded after the change. 3. **List semantics**: `providers/list` returns configurable providers, their supported protocol types, current effective routing, and `required` flag. Providers SHOULD remain discoverable in list after `providers/disable`. 4. **Client behavior for required providers**: clients SHOULD NOT call `providers/disable` for providers where `required: true`.