diff --git a/README.md b/README.md index 3e3e553a81..cf8c509cb2 100644 --- a/README.md +++ b/README.md @@ -1,88 +1,192 @@ # OpenVINO™ Model Server -Model Server hosts models and makes them accessible to software components over standard network protocols: a client sends a request to the model server, which performs model inference and sends a response back to the client. Model Server offers many advantages for efficient model deployment: -- Remote inference enables using lightweight clients with only the necessary functions to perform API calls to edge or cloud deployments. -- Applications are independent of the model framework, hardware device, and infrastructure. -- Client applications in any programming language that supports REST or gRPC calls can be used to run inference remotely on the model server. -- Clients require fewer updates since client libraries change very rarely. -- Model topology and weights are not exposed directly to client applications, making it easier to control access to the model. -- Ideal architecture for microservices-based applications and deployments in cloud environments – including Kubernetes and OpenShift clusters. -- Efficient resource utilization with horizontal and vertical inference scaling. +**High-performance model serving for Generative AI and classic deep learning — powered by [OpenVINO](https://github.com/openvinotoolkit/openvino) and optimized for Intel hardware.** -![OVMS diagram](docs/ovms_diagram.png) - -OpenVINO™ Model Server (OVMS) is a high-performance system for serving models. Implemented in C++ for scalability and optimized for deployment on Intel architectures. It uses the [generative API](https://docs.openvino.ai/2026/model-server/ovms_docs_clients_genai.html) like OpenAI and Cohere, [KServe](https://docs.openvino.ai/2026/model-server/ovms_docs_clients_kfs.html) and [TensorFlow Serving](https://docs.openvino.ai/2026/model-server/ovms_docs_clients_tfs.html) and while applying OpenVINO for inference execution. Inference service is provided via gRPC or REST API, making deploying new algorithms and AI experiments easy. - -![OVMS picture](docs/ovms_high_level.png) - -The models used by the server can be stored locally, hosted remotely by object storage services or pulled from HuggingFace Hub. For more details, refer to [Preparing Model Repository](https://docs.openvino.ai/2026/model-server/ovms_docs_models_repository.html) and [Deployment](https://docs.openvino.ai/2026/model-server/ovms_docs_deploying_server.html) documentation. -Model server works inside Docker containers, Bare Metal and in Kubernetes environment. - -Start using OpenVINO Model Server with a fast-forward serving example from the [QuickStart guide](https://docs.openvino.ai/2026/model-server/ovms_docs_quick_start_guide.html) or [LLM QuickStart guide](https://docs.openvino.ai/2026/model-server/ovms_docs_llm_quickstart.html). +[![Apache License](https://img.shields.io/badge/license-Apache%202.0-green.svg)](https://github.com/openvinotoolkit/model_server/blob/main/LICENSE) +[![Docker Pulls](https://img.shields.io/docker/pulls/openvino/model_server.svg)](https://hub.docker.com/r/openvino/model_server) +[![GitHub Release](https://img.shields.io/github/v/release/openvinotoolkit/model_server)](https://github.com/openvinotoolkit/model_server/releases) +[![Platform](https://img.shields.io/badge/platform-Linux%20%7C%20Windows-blue)](https://docs.openvino.ai/2026/model-server/ovms_docs_deploying_server.html) -Read [release notes](https://github.com/openvinotoolkit/model_server/releases) to find out what’s new. - -### Key features: -- **[NEW]** [Speech Generation and Speech Recognition with OpenAI API](https://docs.openvino.ai/2026/model-server/ovms_demos_audio.html) -- **[NEW]** [Support for AI agents](https://docs.openvino.ai/2026/model-server/ovms_demos_continuous_batching_agent.html) -- **[NEW]** [Image generation compatible with OpenAI API](https://docs.openvino.ai/2026/model-server/ovms_demos_image_generation.html) -- Native Windows support. Check updated [deployment guide](https://docs.openvino.ai/2026/model-server/ovms_docs_deploying_server_baremetal.html) -- [Text Embeddings compatible with OpenAI API](https://docs.openvino.ai/2026/model-server/ovms_demos_embeddings.html) -- [Reranking compatible with Cohere API](https://docs.openvino.ai/2026/model-server/ovms_demos_rerank.html) -- [Efficient Text Generation via OpenAI API](https://docs.openvino.ai/2026/model-server/ovms_demos_continuous_batching.html) -- [Python code execution](docs/python_support/reference.md) -- [gRPC streaming](docs/streaming_endpoints.md) -- [MediaPipe graphs serving](docs/mediapipe.md) -- Model management - including [model versioning](docs/model_version_policy.md) and [model updates in runtime](docs/online_config_changes.md) -- [Dynamic model inputs](docs/shape_batch_size_and_layout.md) -- [Directed Acyclic Graph Scheduler](docs/dag_scheduler.md) along with [custom nodes in DAG pipelines](docs/custom_node_development.md) -- [Metrics](docs/metrics.md) - metrics compatible with Prometheus standard -- Support for multiple frameworks, such as TensorFlow, PaddlePaddle and ONNX -- Support for [AI accelerators](./docs/accelerators.md) +--- -Check full list of [features](./docs/features.md) +## What is OVMS? -**Note:** OVMS has been tested on RedHat, Ubuntu and Windows. -Public docker images are stored in: -- [Dockerhub](https://hub.docker.com/r/openvino/model_server) -- [RedHat Ecosystem Catalog](https://catalog.redhat.com/software/containers/intel/openvino-model-server/607833052937385fc98515de) -Binary packages for Linux and Windows are on [Github](https://github.com/openvinotoolkit/model_server/releases) +OpenVINO Model Server (OVMS) is a production-grade, C++ inference server that exposes ML models over standard network APIs. It serves both **Generative AI** workloads (LLMs, VLMs, image generation, audio) and **classic deep learning** models (object detection, classification, OCR, and more). -## Run OpenVINO Model Server +- **OpenAI-compatible API** for text generation, embeddings, image generation, and audio +- **KServe** APIs for classic model inference +- **Runs anywhere** — Docker, bare metal, Kubernetes/OpenShift, Windows +- **Intel-optimized** — CPU, GPU, NPU acceleration via OpenVINO -A demonstration on how to use OpenVINO Model Server can be found in our [quick-start guide for vision use case](docs/ovms_quickstart.md) and [LLM text generation](docs/llm/quickstart.md). +![OVMS diagram](docs/ovms_diagram.png) -Check also other instructions: +--- -[Preparing model repository](https://docs.openvino.ai/2026/model-server/ovms_docs_models_repository.html) +## Quick Start + +### Serve an LLM with OpenAI-compatible API + +**On Linux (Docker):** +```bash +# Model is downloaded automatically from HuggingFace +docker run --rm -p 8000:8000 \ + openvino/model_server:latest \ + --source_model OpenVINO/Qwen3-4B-int4-ov \ + --model_repository_path /tmp/models \ + --rest_port 8000 +``` +> For GPU acceleration, use the `latest-gpu` image tag and pass `--device /dev/dri --group-add $(stat -c '%g' /dev/dri/render* | head -n1)` to expose the Intel GPU device. + +**On Windows (binary package):** +```bat +mkdir c:\models +ovms.exe --source_model OpenVINO/Qwen3-4B-int4-ov --model_repository_path c:\models --rest_port 8000 +``` + +**Query the model:** +```console +pip install openai +``` +```python +from openai import OpenAI + +client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused") +stream = client.chat.completions.create( + model="OpenVINO/Qwen3-4B-int4-ov", + messages=[{"role": "user", "content": "What are the 3 main tourist attractions in Paris?"}], + stream=True, + extra_body={"chat_template_kwargs": {"enable_thinking": False}} +) +for chunk in stream: + if chunk.choices[0].delta.content: + print(chunk.choices[0].delta.content, end="", flush=True) +``` +> [LLM QuickStart](https://docs.openvino.ai/2026/model-server/ovms_docs_llm_quickstart.html) + + + +### Serve a Classic Model with KServe API + +**Download the model:** +```console +curl -L https://huggingface.co/OpenVINO/resnet50-int8-ov/resolve/main/resnet50.bin -O +curl -L https://huggingface.co/OpenVINO/resnet50-int8-ov/resolve/main/resnet50.xml -O +``` + +**On Linux (Docker):** +```bash +docker run --rm -d -u $(id -u) -v ${PWD}:/models -p 9000:9000 \ + openvino/model_server:latest \ + --model_name resnet --model_path /models/resnet50.xml \ + --mean "[123.675,116.28,103.53]" --scale "[58.395,57.12,57.375]" --layout "NHWC:NCHW" \ + --port 9000 +``` +> For GPU acceleration, use the `latest-gpu` image tag and pass `--device /dev/dri --group-add $(stat -c '%g' /dev/dri/render* | head -n1)` to expose the Intel GPU device. + +**Windows (binary package):** +```bat +ovms --model_name resnet --model_path resnet50.xml --mean "[123.675,116.28,103.53]" --scale "[58.395,57.12,57.375]" --layout "NHWC:NCHW" --port 9000 +``` + +Run inference with a sample client +```console +pip install numpy tritonclient[grpc] +curl -o image.jpeg https://github.com/openvinotoolkit/model_server/blob/main/demos/common/static/images/bee.jpeg?raw=true +``` +```python +import numpy as np +import tritonclient.grpc as grpcclient +with open("image.jpeg", "rb") as f: + image_bytes = f.read() +client = grpcclient.InferenceServerClient(url="localhost:9000") +inputs = [grpcclient.InferInput("image", [1], "BYTES")] +inputs[0].set_data_from_numpy(np.array([image_bytes], dtype=object)) +outputs = [grpcclient.InferRequestedOutput("output")] +result = client.infer(model_name="resnet", inputs=inputs, outputs=outputs) +output = result.as_numpy("output") # (1, 1000) FP32 +print("Top-1 class index:", int(np.argmax(output[0]))) +``` + +> [Vision model QuickStart](https://docs.openvino.ai/2026/model-server/ovms_docs_quick_start_guide.html) -[Deployment](https://docs.openvino.ai/2026/model-server/ovms_docs_deploying_server.html) +--- -[Writing client code](https://docs.openvino.ai/2026/model-server/ovms_docs_server_app.html) +## Features + +### Generative AI +- [LLM text generation](https://docs.openvino.ai/2026/model-server/ovms_demos_continuous_batching.html) — continuous batching, streaming, structured output, speculative decoding +- [VLM (Vision Language Models)](https://docs.openvino.ai/2026/model-server/ovms_demos_continuous_batching_vlm.html) +- [AI Agents with MCP servers](https://docs.openvino.ai/2026/model-server/ovms_demos_continuous_batching_agent.html) +- [Text embeddings](https://docs.openvino.ai/2026/model-server/ovms_demos_embeddings.html) — OpenAI-compatible `/v1/embeddings` +- [Reranking](https://docs.openvino.ai/2026/model-server/ovms_demos_rerank.html) — Cohere-compatible API +- [Image generation](https://docs.openvino.ai/2026/model-server/ovms_demos_image_generation.html) — OpenAI-compatible `/v1/images/generations` +- [Speech recognition and TTS](https://docs.openvino.ai/2026/model-server/ovms_demos_audio.html) — OpenAI-compatible audio API +- [GGUF model support](https://docs.openvino.ai/2026/model-server/ovms_demos_gguf.html) + +### Classic Models & Pipelines +- TensorFlow, ONNX, PaddlePaddle, OpenVINO IR model formats +- [MediaPipe graphs](https://docs.openvino.ai/2026/model-server/ovms_docs_mediapipe.html) +- [Python execution nodes](https://docs.openvino.ai/2026/model-server/ovms_docs_python_support_reference.html) +- [Dynamic input shapes](https://docs.openvino.ai/2026/model-server/ovms_docs_shape_batch_size_and_layout.html) + +### Deployment & Integration +- [Docker](docs/deploying_server_docker.md), [bare metal (Linux & Windows)](docs/deploying_server_baremetal.md), [Kubernetes / OpenShift](docs/deploying_server_kubernetes.md) +- [Model repository](https://docs.openvino.ai/2026/model-server/ovms_docs_models_repository.html): local storage, S3, GCS, Azure Blob, HuggingFace Hub +- [Model versioning](https://docs.openvino.ai/2026/model-server/ovms_docs_model_version_policy.html) and [hot-reload](https://docs.openvino.ai/2026/model-server/ovms_docs_online_config_changes.html) +- [Prometheus-compatible metrics](https://docs.openvino.ai/2026/model-server/ovms_docs_metrics.html) +- [gRPC streaming](https://docs.openvino.ai/2026/model-server/ovms_docs_streaming_endpoints.html) +- [C API](https://docs.openvino.ai/2026/model-server/ovms_docs_c_api.html) for embedding OVMS in native applications + +### Hardware Acceleration +- CPU (x86, including Xeon), Intel integrated and discrete GPU, NPU +- See [supported accelerators](https://docs.openvino.ai/2026/model-server/ovms_docs_accelerators.html) + +[→ Full feature list](https://docs.openvino.ai/2026/model-server/ovms_docs_features.html) -[Demos](https://docs.openvino.ai/2026/model-server/ovms_docs_demos.html) +--- +## Documentation +| Topic | Link | +|---|---| +| Deployment | [Deploying the server](https://docs.openvino.ai/2026/model-server/ovms_docs_deploying_server.html) | +| Model repository | [Preparing models](https://docs.openvino.ai/2026/model-server/ovms_docs_models_repository.html) | +| Client libraries | [Writing client code](https://docs.openvino.ai/2026/model-server/ovms_docs_server_app.html) | +| Demos & examples | [Demos](https://docs.openvino.ai/2026/model-server/ovms_docs_demos.html) | +| Release notes | [GitHub Releases](https://github.com/openvinotoolkit/model_server/releases) | -## References +--- -* [OpenVINO™](https://software.intel.com/en-us/openvino-toolkit) +## Get the Server -* [ADVANCING GENAI WITH CPU OPTIMIZATION](https://cdrdv2-public.intel.com/864404/vFINAL_Intel%20SLM%20Whitepaper.pdf) +**Docker images** (recommended): +```bash +docker pull openvino/model_server:latest # Intel CPU +docker pull openvino/model_server:latest-gpu # Intel CPU,GPU,NPU +``` -* [Manage deep learning models with OpenVINO Model Server](https://developers.redhat.com/articles/2024/07/03/manage-deep-learning-models-openvino-model-server#) +- [Docker Hub](https://hub.docker.com/r/openvino/model_server) +- [Red Hat Ecosystem Catalog](https://catalog.redhat.com/software/containers/intel/openvino-model-server/607833052937385fc98515de) -* [RAG building blocks made easy and affordable with OpenVINO Model Server](https://medium.com/openvino-toolkit/rag-building-blocks-made-easy-and-affordable-with-openvino-model-server-e7b03da5012b) +**Binary packages** (Linux & Windows): [GitHub Releases](https://github.com/openvinotoolkit/model_server/releases) -* [Simple deployment with KServe API](https://blog.openvino.ai/blog-posts/kserve-api) +--- -* [Benchmarking results](https://docs.openvino.ai/2026/about-openvino/performance-benchmarks.html) +## Contributing +Contributions are welcome! Please open an issue or pull request on GitHub. +See [security policy](security.md) for responsible disclosure. -## Contact +--- -If you have a question, a feature request, or a bug report, feel free to submit a Github issue. +## References +- [OpenVINO Toolkit](https://software.intel.com/en-us/openvino-toolkit) +- [Performance benchmarks](https://docs.openvino.ai/2026/about-openvino/performance-benchmarks.html) +- [GenAI with CPU optimization — Intel whitepaper](https://cdrdv2-public.intel.com/864404/vFINAL_Intel%20SLM%20Whitepaper.pdf) +- [RAG with OpenVINO Model Server — blog post](https://medium.com/openvino-toolkit/rag-building-blocks-made-easy-and-affordable-with-openvino-model-server-e7b03da5012b) +- [AIPC turned into a mighty assistant](https://medium.com/openvino-toolkit/ai-pc-turned-into-a-mighty-ai-assistant-with-local-models-and-openvino-model-server-1f41913252c9) --- + \* Other names and brands may be claimed as the property of others. diff --git a/demos/README.md b/demos/README.md index c8a808fa1c..169ab529c0 100644 --- a/demos/README.md +++ b/demos/README.md @@ -5,112 +5,88 @@ maxdepth: 1 hidden: --- -ovms_demos_continuous_batching -ovms_demos_integration_with_open_webui -ovms_demos_code_completion_vsc -ovms_demos_audio -ovms_demos_rerank -ovms_demos_embeddings -ovms_demos_continuous_batching_vlm -ovms_demos_image_generation -ovms_demo_clip_image_classification -ovms_demo_age_gender_guide -ovms_demo_horizontal_text_detection -ovms_demo_optical_character_recognition -ovms_demo_face_detection -ovms_demo_face_blur_pipeline -ovms_demo_capi_inference_demo -ovms_demo_single_face_analysis_pipeline -ovms_demo_multi_faces_analysis_pipeline -ovms_docs_demo_ensemble -ovms_docs_demo_mediapipe_image_classification -ovms_docs_demo_mediapipe_multi_model -ovms_docs_demo_mediapipe_object_detection -ovms_docs_demo_mediapipe_holistic -ovms_docs_demo_mediapipe_iris -ovms_docs_image_classification -ovms_demo_using_onnx_model -ovms_demo_tf_classification -ovms_demo_person_vehicle_bike_detection -ovms_demo_vehicle_analysis_pipeline -ovms_demo_real_time_stream_analysis -ovms_demo_using_paddlepaddle_model -ovms_demo_bert -ovms_demo_universal-sentence-encoder -ovms_string_output_model_demo -ovms_demos_gguf +Text generation +Image generation +Audio +Text Embeddings +Text Reranking +Classic models +MediaPipe +Python Node +Integrations ``` -OpenVINO Model Server demos have been created to showcase the usage of the model server as well as demonstrate it’s capabilities. -### Check Out New Generative AI Demos +OpenVINO Model Server demos have been created to showcase the usage of the model server as well as demonstrate its capabilities. + +## Text Generation | Demo | Description | |---|---| -|[AI Agents with MCP servers and serving language models](./continuous_batching/agentic_ai/README.md)|OpenAI agents with MCP servers and serving LLM models| -|[Integration with Open WebUI](integration_with_OpenWebUI/README.md)|Using OpenWeb UI with OVMS as inference provider. Shows text and image generation as well as usage with RAG and tools| -|[LLM Text Generation with continuous batching](continuous_batching/README.md)|Generate text with LLM models and continuous batching pipeline| -|[VLM Text Generation with continuous batching](continuous_batching/vlm/README.md)|Generate text with VLM models and continuous batching pipeline| -|[OpenAI API text embeddings ](embeddings/README.md)|Get text embeddings via endpoint compatible with OpenAI API| -|[Reranking with Cohere API](rerank/README.md)| Rerank documents via endpoint compatible with Cohere| -|[RAG with OpenAI API endpoint and langchain](https://github.com/openvinotoolkit/model_server/blob/main/demos/continuous_batching/rag/rag_demo.ipynb)| Example how to use RAG with model server endpoints| -|[LLM on NPU](./llm_npu/README.md)| Generate text with LLM models and NPU acceleration| -|[VLM on NPU](./vlm_npu/README.md)| Generate text with VLM models and NPU acceleration| -|[Long context LLMs](./continuous_batching/long_context/README.md)| Recommendations for handling very long context in LLM models| -|[Visual Studio Code assistant](./code_local_assistant/README.md)|Use Continue extension to Visual Studio Code with local OVMS serving| -|[Image Generation](image_generation/README.md)|Generate images| -|[GGUF models support](gguf/README.md)|Serve GGUF models with OVMS| - +|[LLM Text Generation with continuous batching](continuous_batching/README.md)|Generate text with LLM models and continuous batching pipeline.| +|[VLM Text Generation with continuous batching](continuous_batching/vlm/README.md)|Generate text with VLM models and continuous batching pipeline.| +|[AI Agents with MCP servers](./continuous_batching/agentic_ai/README.md)|OpenAI agents with MCP servers and serving LLM models.| +|[RAG with OpenAI API endpoint and langchain](continuous_batching/rag/README.md)|Example how to use RAG with model server endpoints.| +|[Long context LLMs](./continuous_batching/long_context/README.md)|Recommendations for handling very long context in LLM models.| +|[Structured output](./continuous_batching/structured_output/README.md)|Generate structured (JSON) output from LLM models.| +|[Speculative decoding](./continuous_batching/speculative_decoding/README.md)|Speed up LLM inference with speculative decoding.| +|[LLM on NPU](./llm_npu/README.md)|Generate text with LLM models and NPU acceleration.| +|[Scaling on multi CPU and GPU](./continuous_batching/scaling/README.md)|Scale LLM serving across multiple CPUs and GPUs.| +|[Loading models in GGUF](gguf/README.md)|Serve GGUF models with OVMS.| -Check out the list below to see complete step-by-step examples of using OpenVINO Model Server with real world use cases: -## With Traditional Models +## Image Generation | Demo | Description | |---|---| -|[Image Classification](image_classification/python/README.md)|Run prediction on a JPEG image using image classification model via gRPC API.| -|[Using ONNX Model](using_onnx_model/python/README.md)|Run prediction on a JPEG image using image classification ONNX model via gRPC API in two preprocessing variants. This demo uses [pipeline](../docs/dag_scheduler.md) with [image_transformation custom node](https://github.com/openvinotoolkit/model_server/tree/main/src/custom_nodes/image_transformation). | -|[Using TensorFlow Model](image_classification_using_tf_model/python/README.md)|Run image classification using directly imported TensorFlow model. | -|[Age gender recognition](age_gender_recognition/python/README.md) | Run prediction on a JPEG image using age gender recognition model via gRPC API.| -|[Face Detection](face_detection/python/README.md)|Run prediction on a JPEG image using face detection model via gRPC API.| -|[Classification with PaddlePaddle](classification_using_paddlepaddle_model/python/README.md)| Perform classification on an image with a PaddlePaddle model. | -|[Natural Language Processing with BERT](bert_question_answering/python/README.md)|Provide a knowledge source and a query and use BERT model for question answering use case via gRPC API. This demo uses dynamic shape feature. | -|[Using inputs data in string format with universal-sentence-encoder model](universal-sentence-encoder/README.md)| Handling AI model with text as the model input. | -|[Person, Vehicle, Bike Detection](person_vehicle_bike_detection/python/README.md)|Run prediction on a video file or camera stream using person, vehicle, bike detection model via gRPC API.| -|[Benchmark App](benchmark/python/README.md)|Generate traffic and measure performance of the model served in OpenVINO Model Server.| +|[Image Generation](image_generation/README.md)|Generate images with diffusion models.| -## With Python Nodes +## Audio | Demo | Description | |---|---| -|[CLIP image classification](python_demos/clip_image_classification/README.md) | Classify image according to provided labels using CLIP model embedded in a multi-node MediaPipe graph.| +|[Audio demos](audio/README.md)|Text-to-speech and automatic speech recognition demos.| -## With MediaPipe Graphs +## Text Embeddings | Demo | Description | |---|---| -|[Real Time Stream Analysis](real_time_stream_analysis/python/README.md)| Analyze RTSP video stream in real time with generic application template for custom pre and post processing routines as well as simple results visualizer for displaying predictions in the browser. | -|[Image classification](./mediapipe/image_classification/README.md)| Basic example with a single inference node. | -|[Chain of models](./mediapipe/image_classification/README.md)| A chain of models in a graph. | -|[Object detection](./mediapipe/object_detection/README.md)| A pipeline implementing object detection | -|[Iris demo](./mediapipe/object_detection/README.md)| A pipeline implementing iris detection | -|[Holistic demo](./mediapipe/holistic_tracking/README.md)| A complex pipeline linking several image analytical models and image transformations | +|[OpenAI API text embeddings](embeddings/README.md)|Get text embeddings via endpoint compatible with OpenAI API.| -## With DAG Pipelines +## Text Reranking | Demo | Description | |---|---| -|[Horizontal Text Detection in Real-Time](horizontal_text_detection/python/README.md) | Run prediction on camera stream using a horizontal text detection model via gRPC API. This demo uses [pipeline](../docs/dag_scheduler.md) with [horizontal_ocr custom node](https://github.com/openvinotoolkit/model_server/tree/main/src/custom_nodes/horizontal_ocr) and [demultiplexer](../docs/demultiplexing.md). | -|[Optical Character Recognition Pipeline](optical_character_recognition/python/README.md) | Run prediction on a JPEG image using a pipeline of text recognition and text detection models with a custom node for intermediate results processing via gRPC API. This demo uses [pipeline](../docs/dag_scheduler.md) with [east_ocr custom node](https://github.com/openvinotoolkit/model_server/tree/main/src/custom_nodes/east_ocr) and [demultiplexer](../docs/demultiplexing.md). | -|[Single Face Analysis Pipeline](single_face_analysis_pipeline/python/README.md)|Run prediction on a JPEG image using a simple pipeline of age-gender recognition and emotion recognition models via gRPC API to analyze image with a single face. This demo uses [pipeline](../docs/dag_scheduler.md) | -|[Multi Faces Analysis Pipeline](multi_faces_analysis_pipeline/python/README.md)|Run prediction on a JPEG image using a pipeline of age-gender recognition and emotion recognition models via gRPC API to extract multiple faces from the image and analyze all of them. This demo uses [pipeline](../docs/dag_scheduler.md) with [model_zoo_intel_object_detection custom node](https://github.com/openvinotoolkit/model_server/tree/main/src/custom_nodes/model_zoo_intel_object_detection) and [demultiplexer](../docs/demultiplexing.md) | -|[Model Ensemble Pipeline](model_ensemble/python/README.md)|Combine multiple image classification models into one [pipeline](../docs/dag_scheduler.md) and aggregate results to improve classification accuracy. | -|[Face Blur Pipeline](face_blur/python/README.md)|Detect faces and blur image using a pipeline of object detection models with a custom node for intermediate results processing via gRPC API. This demo uses [pipeline](../docs/dag_scheduler.md) with [face_blur custom node](https://github.com/openvinotoolkit/model_server/tree/main/src/custom_nodes/face_blur). | -|[Vehicle Analysis Pipeline](vehicle_analysis_pipeline/python/README.md)|Detect vehicles and recognize their attributes using a pipeline of vehicle detection and vehicle attributes recognition models with a custom node for intermediate results processing via gRPC API. This demo uses [pipeline](../docs/dag_scheduler.md) with [model_zoo_intel_object_detection custom node](https://github.com/openvinotoolkit/model_server/tree/main/src/custom_nodes/model_zoo_intel_object_detection). | +|[Reranking with Cohere API](rerank/README.md)|Rerank documents via endpoint compatible with Cohere.| -## With C++ Client +## Classic Models | Demo | Description | |---|---| -|[C API applications](c_api_minimal_app/README.md)|How to use C API from the OpenVINO Model Server to create C and C++ application.| +|[Image Classification](image_classification/python/README.md)|Run prediction on a JPEG image using image classification model via gRPC API.| +|[Using ONNX Model](using_onnx_model/python/README.md)|Run prediction on a JPEG image using image classification ONNX model via gRPC API in two preprocessing variants. This demo uses [pipeline](../docs/dag_scheduler.md) with [image_transformation custom node](https://github.com/openvinotoolkit/model_server/tree/main/src/custom_nodes/image_transformation).| +|[Using TensorFlow Model](image_classification_using_tf_model/python/README.md)|Run image classification using directly imported TensorFlow model.| +|[Classification with PaddlePaddle](classification_using_paddlepaddle_model/python/README.md)|Perform classification on an image with a PaddlePaddle model.| +|[Age gender recognition](age_gender_recognition/python/README.md)|Run prediction on a JPEG image using age gender recognition model via gRPC API.| +|[Face Detection](face_detection/python/README.md)|Run prediction on a JPEG image using face detection model via gRPC API.| +|[Person, Vehicle, Bike Detection](person_vehicle_bike_detection/python/README.md)|Run prediction on a video file or camera stream using person, vehicle, bike detection model via gRPC API.| +|[Using input strings](universal-sentence-encoder/README.md)|Handling AI model with text as the model input.| +|[Using output strings](image_classification_with_string_output/README.md)|Handling AI model with string output.| +|[Natural Language Processing with BERT](bert_question_answering/python/README.md)|Provide a knowledge source and a query and use BERT model for question answering via gRPC API. This demo uses dynamic shape feature.| +|[Benchmark App](benchmark/python/README.md)|Generate traffic and measure performance of the model served in OpenVINO Model Server.| -## With Go Client +## MediaPipe | Demo | Description | |---|---| -|[Image Classification](image_classification/go/README.md)|Run prediction on a JPEG image using image classification model via gRPC API.| +|[Object Detection](./mediapipe/object_detection/README.md)|A pipeline implementing object detection.| +|[Iris](./mediapipe/iris_tracking/README.md)|A pipeline implementing iris detection.| +|[Holistic](./mediapipe/holistic_tracking/README.md)|A complex pipeline linking several image analytical models and image transformations.| +|[Realtime Stream Analysis](real_time_stream_analysis/python/README.md)|Analyze RTSP video stream in real time with generic application template for custom pre and post processing routines.| +|[Image classification](./mediapipe/image_classification/README.md)|Basic example with a single inference node.| +|[Chain of models](./mediapipe/multi_model_graph/README.md)|A chain of models in a graph.| +|[CLIP image classification](python_demos/clip_image_classification/README.md)|Classify image according to provided labels using CLIP model embedded in a multi-node MediaPipe graph.| +## Python Node +| Demo | Description | +|---|---| +|[OpenClip with python execution](./python_demos/clip_image_classification/README.md)|A pipeline implementing OpenClip classification in Python Node.| +## Integrations +| Demo | Description | +|---|---| +|[Integration with Open WebUI](integration_with_OpenWebUI/README.md)|Using Open WebUI with OVMS as inference provider. Shows text and image generation as well as usage with RAG and tools.| +|[Visual Studio Code assistant](./code_local_assistant/README.md)|Use Continue or Cline extension to Visual Studio Code with local OVMS serving.| diff --git a/demos/classic_models_demos.md b/demos/classic_models_demos.md new file mode 100644 index 0000000000..9462a754d5 --- /dev/null +++ b/demos/classic_models_demos.md @@ -0,0 +1,17 @@ +# Classic models {#ovms_demos_classic_models} + +```{toctree} +--- +maxdepth: 1 +--- + +Image classification +ONNX +TensorFlow +PaddlePaddle +Age gender classification +Face detection +Person Detection +Using input strings +Using output strings +``` diff --git a/demos/continuous_batching/README.md b/demos/continuous_batching/README.md index 04c89aaa08..abb207a246 100644 --- a/demos/continuous_batching/README.md +++ b/demos/continuous_batching/README.md @@ -1,20 +1,5 @@ # LLM models via OpenAI API {#ovms_demos_continuous_batching} -```{toctree} ---- -maxdepth: 1 -hidden: ---- -ovms_demos_continuous_batching_agent -ovms_demos_continuous_batching_rag -ovms_demos_continuous_batching_scaling -ovms_demos_continuous_batching_speculative_decoding -ovms_structured_output -ovms_demo_long_context -ovms_demos_llm_npu -ovms_demos_continuous_batching_accuracy -``` - This demo shows how to deploy LLM models in the OpenVINO Model Server using continuous batching and paged attention algorithms. Text generation use case is exposed via OpenAI API `chat/completions`, `completions` and `responses` endpoints. That makes it easy to use and efficient especially on Intel® Xeon® processors and ARC GPUs. diff --git a/demos/continuous_batching/vlm/README.md b/demos/continuous_batching/vlm/README.md index 5f1145f73c..ef97337585 100644 --- a/demos/continuous_batching/vlm/README.md +++ b/demos/continuous_batching/vlm/README.md @@ -1,13 +1,5 @@ # VLM models via OpenAI API {#ovms_demos_continuous_batching_vlm} -```{toctree} ---- -maxdepth: 1 -hidden: ---- -ovms_demos_vlm_npu -``` - This demo shows how to deploy Vision Language Models in the OpenVINO Model Server. Text generation use case is exposed via OpenAI API `chat/completions` and `responses` endpoints. diff --git a/demos/genai_integrations.md b/demos/genai_integrations.md new file mode 100644 index 0000000000..54cd10985b --- /dev/null +++ b/demos/genai_integrations.md @@ -0,0 +1,13 @@ +# OpenAI API Integrations {#ovms_demos_integrations} + +```{toctree} +--- +maxdepth: 1 +--- + +OpenWebUI +VSC Continue dev +``` + + + diff --git a/demos/mediapipe_demos.md b/demos/mediapipe_demos.md new file mode 100644 index 0000000000..ca2b0b5b3b --- /dev/null +++ b/demos/mediapipe_demos.md @@ -0,0 +1,14 @@ +# MediaPipe demos {#ovms_demos_mediapipe} + +```{toctree} +--- +maxdepth: 1 +--- + +MediaPipe Object Detection Demo +Iris +Holistic +Realtime Stream Analysis +``` + + diff --git a/demos/text_generation.md b/demos/text_generation.md new file mode 100644 index 0000000000..c7587e5039 --- /dev/null +++ b/demos/text_generation.md @@ -0,0 +1,18 @@ +# Text generation demos {#ovms_text_generation} + +```{toctree} +--- +maxdepth: 1 +--- + +LLM demo +VLM demo +Agentic demo +RAG +Long Context +Structured output +Speculative decoding +LLM on NPU +Scaling on multi CPU and GPU +Loading models in GGUF +``` diff --git a/docs/home.md b/docs/home.md index 207273b7ae..b603bea96a 100644 --- a/docs/home.md +++ b/docs/home.md @@ -17,53 +17,182 @@ ovms_docs_demos ovms_docs_troubleshooting ``` -Model Server hosts models and makes them accessible to software components over standard network protocols: a client sends a request to the model server, which performs model inference and sends a response back to the client. Model Server offers many advantages for efficient model deployment: -- Remote inference enables using lightweight clients with only the necessary functions to perform API calls to edge or cloud deployments. -- Applications are independent of the model framework, hardware device, and infrastructure. -- Client applications in any programming language that supports REST or gRPC calls can be used to run inference remotely on the model server. -- Clients require fewer updates since client libraries change very rarely. -- Model topology and weights are not exposed directly to client applications, making it easier to control access to the model. -- Ideal architecture for microservices-based applications and deployments in cloud environments – including Kubernetes and OpenShift clusters. -- Efficient resource utilization with horizontal and vertical inference scaling. +**High-performance model serving for Generative AI and classic deep learning — powered by [OpenVINO](https://github.com/openvinotoolkit/openvino) and optimized for Intel hardware.** + +--- + +## What is OVMS? + +OpenVINO Model Server (OVMS) is a production-grade, C++ inference server that exposes ML models over standard network APIs. It serves both **Generative AI** workloads (LLMs, VLMs, image generation, audio) and **classic deep learning** models (object detection, classification, OCR, and more). + +- **OpenAI-compatible API** for text generation, embeddings, image generation, and audio +- **KServe** APIs for classic model inference +- **Runs anywhere** — Docker, bare metal, Kubernetes/OpenShift, Windows +- **Intel-optimized** — CPU, GPU, NPU acceleration via OpenVINO ![OVMS diagram](ovms_diagram.png) -## Serving with OpenVINO Model Server +--- + +## Quick Start + +### Serve an LLM with OpenAI-compatible API + +**On Linux (Docker):** +```bash +# Model is downloaded automatically from HuggingFace +docker run --rm -p 8000:8000 \ + openvino/model_server:latest \ + --source_model OpenVINO/Qwen3-4B-int4-ov \ + --model_repository_path /tmp/models \ + --rest_port 8000 +``` +> For GPU acceleration, use the `latest-gpu` image tag and pass `--device /dev/dri --group-add $(stat -c '%g' /dev/dri/render* | head -n1)` to expose the Intel GPU device. + +**On Windows (binary package):** +```bat +mkdir c:\models +ovms.exe --source_model OpenVINO/Qwen3-4B-int4-ov --model_repository_path c:\models --rest_port 8000 +``` + +**Query the model:** +```python +from openai import OpenAI + +client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused") +stream = client.chat.completions.create( + model="OpenVINO/Qwen3-4B-int4-ov", + messages=[{"role": "user", "content": "What are the 3 main tourist attractions in Paris?"}], + stream=True, + extra_body={"chat_template_kwargs": {"enable_thinking": False}} +) +for chunk in stream: + if chunk.choices[0].delta.content: + print(chunk.choices[0].delta.content, end="", flush=True) +``` +> [LLM QuickStart](llm/quickstart.md) + + + +### Serve a Classic Model with KServe API -OpenVINO™ Model Server (OVMS) is a high-performance system for serving models. Implemented in C++ for scalability and optimized for deployment on Intel architectures. It uses the same API as [OpenAI](./genai.md), [Cohere](./model_server_rest_api_rerank.md) and [KServe](./model_server_grpc_api_kfs.md) while applying OpenVINO for inference execution. Inference service is provided via gRPC or REST API, making deploying new algorithms and AI experiments easy. +**Download the model:** +```text +curl -L https://huggingface.co/OpenVINO/resnet50-int8-ov/resolve/main/resnet50.bin -O +curl -L https://huggingface.co/OpenVINO/resnet50-int8-ov/resolve/main/resnet50.xml -O +``` + +**On Linux (Docker):** +```bash +docker run --rm -d -u $(id -u) -v ${PWD}:/models -p 9000:9000 \ + openvino/model_server:latest \ + --model_name resnet --model_path /models/resnet50.xml \ + --mean "[123.675,116.28,103.53]" --scale "[58.395,57.12,57.375]" --layout "NHWC:NCHW" \ + --port 9000 +``` +> For GPU acceleration, use the `latest-gpu` image tag and pass `--device /dev/dri --group-add $(stat -c '%g' /dev/dri/render* | head -n1)` to expose the Intel GPU device. -Check how to write the client applications using [generative endpoints](./clients_genai.md). +**Windows (binary package):** +```bat +ovms --model_name resnet --model_path resnet50.xml --mean "[123.675,116.28,103.53]" --scale "[58.395,57.12,57.375]" --layout "NHWC:NCHW" --port 9000 +``` -![OVMS picture](ovms_high_level.png) +Run inference with a sample python client +```python +import numpy as np +import tritonclient.grpc as grpcclient +with open("image.jpeg", "rb") as f: + image_bytes = f.read() +client = grpcclient.InferenceServerClient(url="localhost:9000") +inputs = [grpcclient.InferInput("image", [1], "BYTES")] +inputs[0].set_data_from_numpy(np.array([image_bytes], dtype=object)) +outputs = [grpcclient.InferRequestedOutput("output")] +result = client.infer(model_name="resnet", inputs=inputs, outputs=outputs) +output = result.as_numpy("output") # (1, 1000) FP32 +print("Top-1 class index:", int(np.argmax(output[0]))) +``` -The models used by the server need to be stored locally or hosted remotely by object storage services. For more details, refer to [Preparing Model Repository](./models_repository.md) documentation. Model server works inside [Docker containers](./deploying_server.md), on [Bare Metal](deploying_server.md), and in [Kubernetes environment](deploying_server.md). -Start using OpenVINO Model Server with a fast-forward serving example from the [QuickStart guide](ovms_quickstart.md) or [LLM QuickStart guide](./llm/quickstart.md). +> [Vision model QuickStart](ovms_quickstart.md) -### Key features: -- **[NEW]** [Speech Generation and Speech Recognition with OpenAI API](../demos/audio/README.md) -- **[NEW]** [Support for AI agents](../demos/continuous_batching/agentic_ai/README.md) -- **[NEW]** [Image generation and editing](../demos/image_generation/README.md) -- Native Windows support. Check updated [deployment guide](./deploying_server.md) -- [Embeddings endpoint compatible with OpenAI API](../demos/embeddings/README.md) -- [Reranking compatible with Cohere API](../demos/rerank/README.md) -- [Efficient Text Generation with OpenAI API](../demos/continuous_batching/README.md) -- [Python code execution](python_support/reference.md) +--- + +## Features + +### Generative AI +- [LLM text generation](../demos/continuous_batching/README.md) — continuous batching, streaming, structured output, speculative decoding +- [VLM (Vision Language Models)](../demos/continuous_batching/vlm/README.md) +- [AI Agents with MCP servers](../demos/continuous_batching/agentic_ai/README.md) +- [Text embeddings](../demos/embeddings/README.md) — OpenAI-compatible `/v1/embeddings` +- [Reranking](../demos/rerank/README.md) — Cohere-compatible API +- [Image generation](../demos/image_generation/README.md) — OpenAI-compatible `/v1/images/generations` +- [Speech recognition and TTS](../demos/audio/README.md) — OpenAI-compatible audio API +- [GGUF model support](../demos/gguf/README.md) + +### Classic Models & Pipelines +- TensorFlow, ONNX, PaddlePaddle, OpenVINO IR model formats +- [DAG pipelines](dag_scheduler.md) with [custom nodes](custom_node_development.md) +- [MediaPipe graphs](mediapipe.md) +- [Python execution nodes](python_support/reference.md) +- [Dynamic input shapes](shape_batch_size_and_layout.md) + +### Deployment & Integration +- Docker, bare metal (Linux & Windows), Kubernetes / OpenShift +- [Model repository](models_repository.md): local storage, S3, GCS, Azure Blob, HuggingFace Hub +- [Model versioning](model_version_policy.md) and [hot-reload](online_config_changes.md) +- [Prometheus-compatible metrics](metrics.md) - [gRPC streaming](streaming_endpoints.md) -- [MediaPipe graphs serving](mediapipe.md) -- Model management - including [model versioning](model_version_policy.md) and [model updates in runtime](online_config_changes.md) -- [Dynamic model inputs](shape_batch_size_and_layout.md) -- [Directed Acyclic Graph Scheduler](dag_scheduler.md) along with [custom nodes in DAG pipelines](custom_node_development.md) -- [Metrics](metrics.md) - metrics compatible with Prometheus standard -- Support for multiple frameworks, such as TensorFlow, PaddlePaddle and ONNX -- Support for [AI accelerators](./accelerators.md) +- [C API](model_server_c_api.md) for embedding OVMS in native applications + +### Hardware Acceleration +- CPU (x86, including Xeon), Intel integrated and discrete GPU, NPU +- See [supported accelerators](accelerators.md) + +[→ Full feature list](features.md) -## Additional Resources -* [ADVANCING GENAI WITH CPU OPTIMIZATION](https://cdrdv2-public.intel.com/864404/vFINAL_Intel%20SLM%20Whitepaper.pdf) +--- + +## Documentation + +| Topic | Link | +|---|---| +| Deployment | [Deploying the server](deploying_server.md) | +| Model repository | [Preparing models](models_repository.md) | +| Client libraries | [Writing client code](writing_app.md) | +| Demos & examples | [Demos](../demos/README.md) | +| Release notes | [GitHub Releases](https://github.com/openvinotoolkit/model_server/releases) | -* [Manage deep learning models with OpenVINO Model Server](https://developers.redhat.com/articles/2024/07/03/manage-deep-learning-models-openvino-model-server#) +--- + +## Get the Server -* [RAG building blocks made easy and affordable with OpenVINO Model Server](https://medium.com/openvino-toolkit/rag-building-blocks-made-easy-and-affordable-with-openvino-model-server-e7b03da5012b) +**Docker images** (recommended): +```bash +docker pull openvino/model_server:latest # Intel CPU +docker pull openvino/model_server:latest-gpu # Intel CPU,GPU,NPU +``` -* [Simple deployment with KServe API](https://blog.openvino.ai/blog-posts/kserve-api) +- [Docker Hub](https://hub.docker.com/r/openvino/model_server) +- [Red Hat Ecosystem Catalog](https://catalog.redhat.com/software/containers/intel/openvino-model-server/607833052937385fc98515de) + +**Binary packages** (Linux & Windows): [GitHub Releases](https://github.com/openvinotoolkit/model_server/releases) + +--- + +## Contributing + +Contributions are welcome! Please open an issue or pull request on GitHub. +See [security policy](security.md) for responsible disclosure. + +--- + +## References + +- [OpenVINO Toolkit](https://software.intel.com/en-us/openvino-toolkit) +- [Performance benchmarks](https://docs.openvino.ai/2026/about-openvino/performance-benchmarks.html) +- [GenAI with CPU optimization — Intel whitepaper](https://cdrdv2-public.intel.com/864404/vFINAL_Intel%20SLM%20Whitepaper.pdf) +- [RAG with OpenVINO Model Server — blog post](https://medium.com/openvino-toolkit/rag-building-blocks-made-easy-and-affordable-with-openvino-model-server-e7b03da5012b) +- [AIPC turned into a mighty assistant](https://medium.com/openvino-toolkit/ai-pc-turned-into-a-mighty-ai-assistant-with-local-models-and-openvino-model-server-1f41913252c9) + +--- -* [Benchmarking results](https://docs.openvino.ai/2026/about-openvino/performance-benchmarks.html) +\* Other names and brands may be claimed as the property of others. diff --git a/docs/ovms_diagram.png b/docs/ovms_diagram.png index 2adfc99881..9034d27aae 100644 Binary files a/docs/ovms_diagram.png and b/docs/ovms_diagram.png differ