diff --git a/admin_manual/ai/app_assistant.rst b/admin_manual/ai/app_assistant.rst index 92b35b1fc4a..5f66636e7f0 100644 --- a/admin_manual/ai/app_assistant.rst +++ b/admin_manual/ai/app_assistant.rst @@ -6,7 +6,7 @@ Nextcloud Assistant Nextcloud assistant is the primary graphical user interface for interacting with artificial intelligence features in Nextcloud. -It offers the graphical user interface for the unified AI Task processing API offering features like summarizing text, generating headlines, asking arbitrary questions, transcription of media files, image generation and it integrates with the context_chat app to offer in-context answers about your own data stored in Nextcloud. The assistant app also offers a chat interface to interact with the chosen language model. Nextcloud can provide customer support upon request, please talk to your account manager for the possibilities. +It offers the graphical user interface for the unified AI Task processing API offering features like summarizing text, generating headlines, asking arbitrary questions, transcription of media files, image generation and it integrates with the context_chat app to offer in-context answers about your own data stored in Nextcloud. The assistant app also offers a chat interface to interact with the chosen language model, including multimodal chat with file attachments when a compatible backend is configured. Nextcloud can provide customer support upon request, please talk to your account manager for the possibilities. Find the user documentation here: `AI assistant user documentation `_ @@ -106,6 +106,15 @@ In order to make use of our "Chat with AI" feature you will need any one of the * :ref:`llm2` - Runs open source AI language models locally on your own server hardware (Customer support available upon request) * `OpenAI and LocalAI integration (via OpenAI API) `_ - Integrates with the OpenAI API to provide AI functionality from OpenAI servers (Customer support available upon request; see :ref:`AI as a Service`) +Multimodal chat +~~~~~~~~~~~~~~~ + +Multimodal chat extends "Chat with AI" so users can attach files for the language model to analyze in the conversation, such as images and, with some models, audio and video. Text files can also be attached as context. + +In order to make use of multimodal chat you will need any one of the following backends with a multimodal-capable model: + +* :ref:`llm2` - Use a multimodal model such as the shipped Qwen 3.5 9B (vision) or Gemma 4 E4B (vision and audio) (Customer support available upon request) +* `OpenAI and LocalAI integration (via OpenAI API) `_ - Integrates with multimodal models via the OpenAI API (Customer support available upon request; see :ref:`AI as a Service`) Voice Chat ~~~~~~~~~~ diff --git a/admin_manual/ai/app_llm2.rst b/admin_manual/ai/app_llm2.rst index 35151ab7c97..098af083ddd 100644 --- a/admin_manual/ai/app_llm2.rst +++ b/admin_manual/ai/app_llm2.rst @@ -8,18 +8,33 @@ The *llm2* app is one of the apps that provide text processing functionality usi This app uses `llama.cpp `_ under the hood and is thus compatible with any model in *gguf* format. -However, we only test with Llama 3.1. Output quality will differ depending on which model you use and downstream tasks like summarization or Context Chat may not work on other models. +However, we primarily test with the models listed below. Output quality will differ depending on which model you use and downstream tasks like summarization or Context Chat may not work on other models. We thus recommend the following models: -* `Llama3.1 8b Instruct `_ (reasonable quality; fast; good acclaim; comes shipped with the app) -* `Llama3.1 70B Instruct `_ (good quality; good acclaim) +* `Qwen 3.5 9B `_ (good quality; multimodal vision; comes shipped with the app) +* `Gemma 4 E4B `_ (fast; multimodal vision and audio; comes shipped with the app) +* `OLMo 3 7B Instruct `_ (fully open; green Ethical AI rating; text only; comes shipped with the app; see addendum below) + +Multimodal chat +--------------- + +With a multimodal-capable model, *llm2* supports multimodal chat in the :ref:`Nextcloud Assistant app`: users can attach images (and with Gemma 4, also audio) to chat messages so the model can reason about their content. Text files can also be attached as context. + +The following shipped models support multimodality: + +* **Qwen 3.5 9B** - vision (images) +* **Gemma 4 E4B** - vision (images) and audio + +Vision-capable models also register the *Analyze images* and *OCR* task types in addition to the usual text-processing tasks. + +Multimodal models need a matching multimodal projector (``mmproj``) *gguf* file next to the model file. The projector files for the shipped Qwen 3.5 and Gemma 4 models are downloaded automatically with the models. If you supply your own multimodal model, place its ``mmproj`` file in ``/nc_app_llm2_data`` as well and set ``mmproj_path`` in the model's JSON configuration (see :ref:`Configuring alternate models ` below). Multilinguality --------------- This app supports input and output in languages other than English if the underlying model supports the language. -Llama 3.1 `supports the following languages: `_ +Qwen 3.5 and Gemma 4 offer broad multilingual support. Llama 3.1 `supports the following languages: `_ * English * Portuguese @@ -30,7 +45,7 @@ Llama 3.1 `supports the following languages: