Can't connect to open-webui API

[mahenning]

Before submitting your bug report

• I believe this is a bug. I'll try to join the Continue Discord for questions
• I'm not able to find an open issue that reports the same bug
• I've seen the troubleshooting guide on the Continue Docs

Relevant environment info

- OS: Ubuntu 22.04
- Continue version: 0.8.66
- IDE version: VS Code 1.96.4
- Model: any, but in the example mostly TechxGenus/Mistral-Large-Instruct-2407-GPTQ
- config.json:
  
{
  "models": [
    {
      "title": "Mistral Large vLLM",
      "model": "TechxGenus/Mistral-Large-Instruct-2407-GPTQ",
      "provider": "vllm",
      "apiBase": "http://<open-webui-server>:8080/v1",
    },
    {
      "title": "Mistral Large webui",
      "model": "TechxGenus/Mistral-Large-Instruct-2407-GPTQ",
      "provider": "openai",
      "apiBase": "https://<open-webui-server>/api",
      "apiKey": "sk-<apikey>",
      "useLegacyCompletionsEndpoint": false   
    },
    {
      "title": "Mistral Large webui localhost",
      "model": "TechxGenus/Mistral-Large-Instruct-2407-GPTQ",
      "provider": "openai",
      "apiBase": "http://localhost:3000/api",
      "apiKey": "sk-<local apikey>",
      "useLegacyCompletionsEndpoint": false
    }
  ],

  "contextProviders": [
    {
      "name": "code",
      "params": {}
    },
    {
      "name": "docs",
      "params": {}
    },
    {
      "name": "diff",
      "params": {}
    },
    {
      "name": "terminal",
      "params": {}
    },
    {
      "name": "problems",
      "params": {}
    },
    {
      "name": "folder",
      "params": {}
    },
    {
      "name": "codebase",
      "params": {}
    }
  ],
  "slashCommands": [
    {
      "name": "share",
      "description": "Export the current chat session to markdown"
    },
    {
      "name": "cmd",
      "description": "Generate a shell command"
    },
    {
      "name": "commit",
      "description": "Generate a git commit message"
    }
  ]
}

Description

I got Open-WebUI running with three backends for models: Ollama, vLLM, and Llama.cpp. All 4 services are running in separate docker containers, the "Mistral Large" model is running in vLLM. In the config, In the Continue config, I have an API call directly to vllm (this works), a call to an Open-WebUI instance running behind a nginx server, and a locally hosted Open-WebUI to test without nginx.
The Problem is that with the above config, I get errors when trying to chat with the Open-WebUI "models" in the VS Code Continue plugin window (the errors are different to each other, depending on the nginx server or localhost).
Localhost (VS Code Dev Tools window):

[Extension Host] Error handling webview message: {
  "msg": {
    "messageId": "0e06bbbb-f108-4b16-af64-6cbbdc813f20",
    "messageType": "llm/streamChat",
    "data": {
      "messages": [
        {
          "role": "user",
          "content": [
            {
              "type": "text",
              "text": "test"
            }
          ]
        },
        {
          "role": "assistant",
          "content": ""
        }
      ],
      "title": "Mistral Large webui local",
      "completionOptions": {}
    }
  }
}

Error: Request timed out.

The time-out takes some time, in this time the vLLM engine/model is responsive to other requests.

nginx-server (VS Code Dev Tools window):

[Extension Host] Error handling webview message: {
  "msg": {
    "messageId": "450d58ba-f9ae-4b5a-b62a-116e57b6694d",
    "messageType": "llm/streamChat",
    "data": {
      "messages": [
        {
          "role": "user",
          "content": [
            {
              "type": "text",
              "text": "test"
            }
          ]
        },
        {
          "role": "assistant",
          "content": ""
        }
      ],
      "title": "Mistral Large webui",
      "completionOptions": {}
    }
  }
}

Error: Connection error.

notificationsAlerts.ts:42 The request failed with "FetchError": request to https://<open-webui-server>/api/chat/completions failed, reason: Parse Error: Invalid header value char. If you're having trouble setting up Continue, please see the troubleshooting guide for help.

This error comes pretty quick. So at least I know from the second request, that the correct API point (/api/chat/completions) is used. Unfortunately, I don't really know what to do with this error message.

The weird thing is, that I can connect to both APIs with a simple bash script (nginx-server example here):

curl -v -X POST --location 'https://<open-webui-server>/api/chat/completions' \
    -H 'Content-Type: application/json' \
    -H 'Authorization: Bearer sk-<apikey>' \
    -d '{
        "model": "TechxGenus/Mistral-Large-Instruct-2407-GPTQ",
        "messages": [
            {
            "role": "user",
            "content": "Write 50 words."
            }
        ]
    }'

This produces an answer. I get the same problem with Ollama models, and with Llama.cpp models. I can use Continue with the backend apis directly, and I can curl to every model through Open-WebUI (and the backends), but Continue can't use the Open-WebUI API. Because I have changing models/backends and I don't want to expose the backends directly to users, which want to use the API, it would be really great to use the unified Open-WebUI API calls.

PS: I tried to append header information to the model config but it didn't change the outcome:

"requestOptions": {
  "headers": {
    "Authorization":"Bearer sk-5111770f0a6a4fb08c3d7fd1673f4309",
    "Content-Type": "application/json"
  }
}

PPS: I found #3620 and #1380, but both use the ollama endpoint, which I can't use with the vLLM/Llama.cpp models.

To reproduce

No response

Log output

[jknapp-illuminate]

Your configuration looks right, for Continue.dev. For what it is worth, I am using OpenWebUI at home through HAProxy and at work through Fargate and ALBs and both work fine with Continue.dev (we are testing it currently at work as an alternative to CoPilot).

I would suggest looking at looking at nginx and ensure it isn't doing anything like compression of the headers.

I can share my configuration for HAProxy if you want to see if that works better for you.

To troubleshoot the localhost issue, I want to confirm, you are running OWUI on your local system with that large model?

Could you provide the OS and specs of your system along with how you are running OWUI (docker, WSL, direct)?

[mahenning]

Thanks for the suggestions. I also think that the nginx config has some potential problems because of the 2 different error messages. That's why I currently first try to fix the localhost part. And afaik, we will stay at nginx, but thank you for offering sharing your config.

For the localhost version, both vLLM and OWUI are running in separate docker containers on the same system (headless server), and the VS Code IDE with the continue plugin is running on the same VM via ssh remote. The VM is also Ubuntu 22.04. Do you need additional information?

[jknapp-illuminate]

Are the containers configured to expose the ports to the local system?
I've never used the ssh remote feature, but I would suggest trying running it directly through the web interface to see if it works that way, removing the possibility of something acting up via the ssh remote.

Is the Continue Plugin installed on the server or the local VS Code?

[mahenning]

Well, I don't know what changed, but it works now for the localhost variant.. Maybe I missed that the forwarded port was not the exact port number at this time. Sometimes VS Code forwards e.g. port 3000 as port 3001. And I think then I need to use port 3001 for Continue, as it works in my local VS Code. It is now port 3000 also in the forwarding and it works now.

The Continue plugin is installed on the local VS Code, and as far as I'm aware it's up to the plugin where it installs itself. I need port forwarding for the OWUI port for it to work with Continue, as the Continue plugin most likely works on my local "localhost". As it's working now, the Problem with the nginx version is most likely a nginx miss-configuration by me. Thank you for the help!

[jax0m]

Worth noting for others that OpenWebUI allows for you to "rename" models. However when called via the API it still uses the given name from the backend.

Examples

Working: Local direct call of an OpenAI backend (vLLM)

"models": [
    {
      "model": "/models/phi-4-Q4_K_M.gguf",
      "provider": "openai",
      "title": "phi-4_vllm",
      "apiBase": "http://X.X.X.X:8000/v1"
    },
]

---

Didnt Work: Continue attempting to call OpenWebUI via openai protocol using a name customized in OpenWebUI:

I had added this model into OpenWebUi with a slightly more human name, example "phi4_vllm" and configured Continue as such:

"models": [
    {
      "model": "phi4_vllm", 
      "provider": "openai",
      "title": "phi4_vllm",
      "apiKey": "{key}",
      "apiBase": "https://sub.domain.tld/api/"
    }
  ]

Results in Continue giving popup window Title: '400 Error' Message: '400 status code (no body)` and a link to logging which didnt have a ton of clarity.

Note: Possibly this return could be more like the ollama return in the "bonus" at the bottom of this post

---

Working: Continue configured to use the "model" name that OpenWebUI calls for backend:

Looking at the logs for the OpenWebUI container I could see: "POST /api/chat/completions HTTP/1.1" 400 Bad Request"

After trying some iterations I found that once I made the configuration for the "model" portion of the call via this interface match the backend name for the model it worked right away:

"models": [
    {
      "model": "/models/phi-4-Q4_K_M.gguf", 
      "provider": "openai",
      "title": "phi4_vllm",
      "apiKey": "{key}",
      "apiBase": "https://sub.domain.tld/api/"
    }
  ]

---

Bonus test: Attempting to use ollama protocol via OpenWebUI via Continue with a OpenAI backend (Didnt work):

Interestingly while OpenWebUi does support the ollama style of API as well when called from continue one or the other is adding a ":latest" to the end of it:

Example Configuration:

"models": [
    {
      "model": "/models/phi-4-Q4_K_M.gguf", 
      "provider": "ollama",
      "title": "phi4_vllm",
      "apiKey": "{key}",
      "apiBase": "https://sub.domain.tld/ollama"
    }
  ]

For which continue provides a more useful error return Title:Error Message: HTTP 400 Bad Request from https://sub.domain.tld/ollama/api/chat {"detail":"Model '/models/phi-4-Q4_K_M.gguf:latest' was not found"} and OpenWebUi throws: "POST /ollama/api/chat HTTP/1.1" 400 Bad Request

[mahenning]

Good point jax0m. I used the model TechxGenus/Mistral-Large-Instruct-2407-GPTQ in my example, and in OWUI it's renamed to "Mistral Large", which does not work on API calls.

I also just wanted to say that I managed to get it working with nginx and OWUI. I think my nginx config was not configured correctly with API calls, and I had no websockets enabled for them.

[SKJoy]

This is an extension bug without doubt, at least till this moment when I am using.

Continue configuration

  - name: Default agent
    provider: openai
    model: MODEL_NAME:latest
    apiBase: http://OLLAMA_OPEN_WEBUI_HOST:PORT/api
    apiKey: OLLAMA-OPEN-WEBUI-USER-API-KEY
    defaultCompletionOptions:
      stream: false

What is causing this?

Upon setting up, the extension is making an invalid request path like /api/completions to the OLLAMA_OPEN_WEBUI_HOST while it should have been /api/chat/completions.

How to fix this?

The easiest way is to rewrite the request URL path for the back end OLLAMA_OPEN_WEBUI_HOST server block, until Continue fixes this.

• NginX

  location /api/completions {
      proxy_pass http://OLLAMA_OPEN_WEBUI_UPSTREAM:PORT/api/chat/completions;
  }
  

• Apache: Have not tried yet, but should be simpler than NginX

Note: For low end OLLaMA host, you may need to consider increasing the proxy timeout limits for NginX/Apache.

[mahenning]

Thats strange. From my understanding, the /chat/completions endpoint should be default, because the /completions endpoint is deprecated. You can try to set useLegacyCompletionsEndpoint: false explicitly in your model config to try and force the /chat/completions endpoint.

Edit: Is that a model for tabAutocomplete? It seems that that uses the /completions endpoint by default, at least for me. According to #1830 (comment) you can force the chat endpoint by setting the useLegacyCompletionsEndpoint: false in the model.

[SKJoy]

Thats strange. From my understanding, the /chat/completions endpoint should be default, because the /completions endpoint is deprecated. You can try to set useLegacyCompletionsEndpoint: false explicitly in your model config to try and force the /chat/completions endpoint.

Edit: Is that a model for tabAutocomplete? It seems that that uses the /completions endpoint by default, at least for me. According to #1830 (comment) you can force the chat endpoint by setting the useLegacyCompletionsEndpoint: false in the model.

For some reason the useLegacyCompletionsEndpoint: false didn't work for me, neither I wasted much time into that. I just created a small PHP script to act as a bridge in between to intercept and correct the wrong end point, and then decided to work it out with NginX configuration. Now everything is working without a flaw :)

Also, I currently am using it for Chat only, not for Autocomplete, because it is a trial VPS without a GPU and Autocomplete simply causes a destructive overload with likely hundreds of pending requests in a chain :(