diff --git a/guide/18-working-with-AI-capabilities/introduction_to_ai_utility_services.ipynb b/guide/18-working-with-AI-capabilities/introduction_to_ai_utility_services.ipynb new file mode 100644 index 0000000000..87c8c7455c --- /dev/null +++ b/guide/18-working-with-AI-capabilities/introduction_to_ai_utility_services.ipynb @@ -0,0 +1,1167 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "b1fdca13-eb69-4584-82f7-016dd9e622a2", + "metadata": {}, + "source": [ + "## Introduction to AI-Powered services in ArcGIS Online\n", + "\n", + "ArcGIS Online comes equipped with AI-powered services, supported in the ArcGIS API for Python via the [`arcgis.ai`](https://developers.arcgis.com/python/latest/api-reference/arcgis.ai.html) module. It is designed to extract valuable information and help automate tasks involving unstructured content, such as text, images, and documents.\n", + "\n", + "The API supports three services or methods:\n", + "1. [`analyze_image()`](https://developers.arcgis.com/python/latest/api-reference/arcgis.ai.html#analyze-image) - Analyzes images based on the user provided instructions.\n", + "2. [`analyze_text()`](https://developers.arcgis.com/python/latest/api-reference/arcgis.ai.html#analyze-text) - Analyzes provided text against the specified context.\n", + "3. [`translate()`](https://developers.arcgis.com/python/latest/api-reference/arcgis.ai.html#translate) - Translates language strings.\n", + "\n", + "We will explore these methods through a generic example, as well as examples involving ArcGIS items to see potential uses of these AI services applicable to our ArcGIS data.\n", + "\n", + "> Note: Using these services consumes credits:\n", + "> 1. `analyze_image()` - __5 images per credit__\n", + "> 2. `analyze_text()` - __5,000 characters per credit__\n", + "> 3. `translate()` - __1,000 characters per credit__" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "17d8b94b-d89b-477d-baa9-133ae71c6943", + "metadata": {}, + "outputs": [], + "source": [ + "from arcgis.gis import GIS\n", + "from arcgis.ai import analyze_image, analyze_text, translate, AIUtilsResponse" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "36fe7886-0c9d-47dc-897d-3f1b0a344463", + "metadata": {}, + "outputs": [], + "source": [ + "gis = GIS(profile='your_online_profile')" + ] + }, + { + "cell_type": "markdown", + "id": "3258c4f5-f5fe-4d3f-89bd-23459d1e64f7", + "metadata": {}, + "source": [ + "### __Analyze Image__\n", + "\n", + "This tool helps analyze images using AI-powered models for object detection, text extraction, image description, and segmentation. Currently supported image formats are: JPEG, PNG and WEBP.\n", + "\n", + "> Note: Input image could be provided as an URL or Base64. Only images hosted on *.arcgis.com domains are supported via url input. \n", + "\n", + "\n", + "Let us start with this image hosted in ArcGIS Online." + ] + }, + { + "cell_type": "markdown", + "id": "d76c1db3-2237-45f9-bf6b-ed72a2782dba", + "metadata": {}, + "source": [ + "" + ] + }, + { + "cell_type": "code", + "execution_count": 25, + "id": "9ae25c09-2957-44e3-b9bf-f4c801377b0f", + "metadata": {}, + "outputs": [], + "source": [ + "image_url = \"https://geosaurus.maps.arcgis.com/sharing/rest/content/items/6e9ff93df28a4231b4bd6e729ce42898/data\"" + ] + }, + { + "cell_type": "markdown", + "id": "b7dc71ee-6c1f-4a44-a1ee-768721f180f2", + "metadata": {}, + "source": [ + "Prompts to the `analyze_image()` method may be provided as string, list or dictionary inputs. We will see examples for all 3 below. " + ] + }, + { + "cell_type": "markdown", + "id": "f3a09af8-2751-419d-8717-ca21f9d6890e", + "metadata": {}, + "source": [ + "#### 1. Prompt as `str` input" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "id": "16dbdf69-e1cd-4b3c-acd8-43342de3bca5", + "metadata": {}, + "outputs": [], + "source": [ + "prompt_str = \"List all the colors present in the image.\"" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "id": "d26f9264-5ab2-4ecd-8913-32a953fa92cd", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "AIUtilsResponse(success=True, meta=AIUtilsResponseMetaData(usage={'inputTokens': 1831, 'outputTokens': 62, 'totalTokens': 1893}, model='gpt-5.4'), results=[AnalyzeResult(success=True, value=['green', 'dark green', 'light green', 'brown', 'dark brown', 'orange', 'yellow', 'red', 'pink', 'purple', 'blue', 'black', 'gray', 'white'], key='unique_string_1')])" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "colors = analyze_image(image = image_url, prompt = prompt_str)\n", + "colors" + ] + }, + { + "cell_type": "markdown", + "id": "c913fa32-9d71-4b1f-8a63-9aa189d60204", + "metadata": {}, + "source": [ + "As we will continue to see through this guide, the results generated from running these tools in the ArcGIS API for Python are of [`AIUtilsResponse`](https://developers.arcgis.com/python/latest/api-reference/arcgis.ai.html#arcgis.ai.AIUtilsResponse) type. \n", + "\n", + "You can verify if the request was successful using the `success` field of the response as shown below. " + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "id": "a022d63d-f1bd-47e3-b8f7-17fd8627d6df", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "True" + ] + }, + "execution_count": 23, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "colors.success" + ] + }, + { + "cell_type": "markdown", + "id": "bc82764b-5614-452e-bf6e-020a5184e242", + "metadata": {}, + "source": [ + "Since `AIUtilsResponse` is a pydantic model, you can access the specific results of the response as shown below." + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "id": "7bf338af-8e3a-457f-8539-f95a50bb38bc", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['green',\n", + " 'dark green',\n", + " 'light green',\n", + " 'brown',\n", + " 'dark brown',\n", + " 'orange',\n", + " 'yellow',\n", + " 'red',\n", + " 'pink',\n", + " 'purple',\n", + " 'blue',\n", + " 'black',\n", + " 'gray',\n", + " 'white']" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "colors.results[0].value" + ] + }, + { + "cell_type": "markdown", + "id": "f7c09fc8-84cb-4f48-853e-21ef26a10204", + "metadata": {}, + "source": [ + "The service has correctly identified the colors in this image.\n", + "\n", + "We will now proceed to see examples involving the other input types through an ArcGIS item. This is a [map image](https://www.arcgis.com/home/item.html?id=36c34ac9e2c5402ab94762f97f3349af#overview) representing the hurricane history for Virginia. " + ] + }, + { + "cell_type": "markdown", + "id": "e22cf3ba-c198-4a86-8a48-343f9a9dfa76", + "metadata": {}, + "source": [ + "" + ] + }, + { + "cell_type": "code", + "execution_count": 30, + "id": "1dcdeb90-54db-4080-ab89-a6910740c5a8", + "metadata": {}, + "outputs": [], + "source": [ + "virginia_map_url = \"https://geosaurus.maps.arcgis.com/sharing/rest/content/items/36c34ac9e2c5402ab94762f97f3349af/data\"" + ] + }, + { + "cell_type": "markdown", + "id": "339ed176-0dbe-44f4-96f6-7623a9a0cef2", + "metadata": {}, + "source": [ + "#### 2. Prompt as `list` input" + ] + }, + { + "cell_type": "code", + "execution_count": 44, + "id": "4e647c79-771b-49b6-ad2f-c9035805baec", + "metadata": {}, + "outputs": [], + "source": [ + "prompt_list = [\n", + " \"Describe the image.\",\n", + " \"Identify any hurricane related text in the image.\"\n", + "]" + ] + }, + { + "cell_type": "code", + "execution_count": 45, + "id": "43ec5bdf-c5d8-45b7-977b-94ac67484347", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "AIUtilsResponse(success=True, meta=AIUtilsResponseMetaData(usage={'inputTokens': 1822, 'outputTokens': 448, 'totalTokens': 2270}, model='gpt-5.4'), results=[AnalyzeResult(success=True, value='A grayscale map of Virginia showing hurricane, tropical storm, and tropical depression tracks across the state and nearby Atlantic coast. The title reads \"Virginia Hurricanes, Tropical Storms, and Tropical Depressions by Date 1852-2016, 2017-2021.\" Many storm paths are drawn in light blue for 1852-2016 and in orange/red shades for 2017-2021. County boundaries, neighboring states, a mileage scale, north arrow, legend, and source information are also shown.', key='unique_string_0'), AnalyzeResult(success=True, value=['Virginia Hurricanes, Tropical Storms, and Tropical Depressions by Date 1852-2016, 2017-2021', 'Hurricane Category 2, 2017-2021', 'Tropical Storm, 2017-2021', 'Tropical Depression 2017-2021', 'Hurricane Category 4, 1852-2016', 'Hurricane Category 3, 1852-2016', 'Hurricane Category 2, 1852-2016', 'Hurricane Category 1, 1852-2016', 'Tropical Storm, 1852-2016', 'Tropical Depression, 1852-2016', 'FLORENCE, 2018', 'IDA, 2021', 'ZETA, 2020', 'BERTHA, 2020', 'MICHAEL, 2018', 'ELSA, 2021', 'ISAIAS, 2020', 'FAY, 2020', 'CLAUDETTE, 2021', 'DORIAN, 2019', 'FRED, 2021'], key='unique_string_1')])" + ] + }, + "execution_count": 45, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "result_list = analyze_image(image = virginia_map_url, prompt = prompt_list)\n", + "result_list" + ] + }, + { + "cell_type": "markdown", + "id": "345e81cb-8b3f-48a9-beae-691a8c7d487c", + "metadata": {}, + "source": [ + "We can access results for the individual prompts via the prompt index provided to the `results` field as demonstrated below." + ] + }, + { + "cell_type": "code", + "execution_count": 46, + "id": "e3779890-7eb3-405b-8c4e-626289fe9615", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "'A grayscale map of Virginia showing hurricane, tropical storm, and tropical depression tracks across the state and nearby Atlantic coast. The title reads \"Virginia Hurricanes, Tropical Storms, and Tropical Depressions by Date 1852-2016, 2017-2021.\" Many storm paths are drawn in light blue for 1852-2016 and in orange/red shades for 2017-2021. County boundaries, neighboring states, a mileage scale, north arrow, legend, and source information are also shown.'" + ] + }, + "execution_count": 46, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "result_list.results[0].value" + ] + }, + { + "cell_type": "code", + "execution_count": 47, + "id": "126503e6-217a-481b-87d6-b2a541d3e609", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Virginia Hurricanes, Tropical Storms, and Tropical Depressions by Date 1852-2016, 2017-2021',\n", + " 'Hurricane Category 2, 2017-2021',\n", + " 'Tropical Storm, 2017-2021',\n", + " 'Tropical Depression 2017-2021',\n", + " 'Hurricane Category 4, 1852-2016',\n", + " 'Hurricane Category 3, 1852-2016',\n", + " 'Hurricane Category 2, 1852-2016',\n", + " 'Hurricane Category 1, 1852-2016',\n", + " 'Tropical Storm, 1852-2016',\n", + " 'Tropical Depression, 1852-2016',\n", + " 'FLORENCE, 2018',\n", + " 'IDA, 2021',\n", + " 'ZETA, 2020',\n", + " 'BERTHA, 2020',\n", + " 'MICHAEL, 2018',\n", + " 'ELSA, 2021',\n", + " 'ISAIAS, 2020',\n", + " 'FAY, 2020',\n", + " 'CLAUDETTE, 2021',\n", + " 'DORIAN, 2019',\n", + " 'FRED, 2021']" + ] + }, + "execution_count": 47, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "result_list.results[1].value" + ] + }, + { + "cell_type": "markdown", + "id": "8f70d85b-c2e6-46df-9e1d-e3b6c869ac09", + "metadata": {}, + "source": [ + "The response provides a succinct summary of this map and extracts key information regarding hurricanes referenced. " + ] + }, + { + "cell_type": "markdown", + "id": "5dc8a7e9-33a3-4684-a2f4-e463ff340556", + "metadata": {}, + "source": [ + "#### 3. Prompt as `dict` input " + ] + }, + { + "cell_type": "markdown", + "id": "d0067f71-e2e1-415f-a686-01e31effd616", + "metadata": {}, + "source": [ + "In order to provide prompts as a dictionary, it is essential to provide the values for the `key` and `context` keys of the dictionary for each prompt that you wish to provide. " + ] + }, + { + "cell_type": "code", + "execution_count": 40, + "id": "bc6d62b5-2780-46e7-8228-c200ffd708cf", + "metadata": {}, + "outputs": [], + "source": [ + "prompt_dicts = [\n", + " {\"key\": \"images_prompt\", \"context\": \"How many sub-boundaries of Virginia are included in the image?\"},\n", + " {\"key\": \"text_prompt\", \"context\": \"Identify the states named in the map.\"}\n", + "]" + ] + }, + { + "cell_type": "code", + "execution_count": 41, + "id": "f94354b5-5251-4edb-bd10-a8927bba9873", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "AIUtilsResponse(success=True, meta=AIUtilsResponseMetaData(usage={'inputTokens': 1826, 'outputTokens': 117, 'totalTokens': 1943}, model='gpt-5.4'), results=[AnalyzeResult(success=True, value=95, key='images_prompt'), AnalyzeResult(success=True, value=['Ohio', 'Pennsylvania', 'Maryland', 'Delaware', 'Kentucky', 'Tennessee', 'North Carolina', 'Virginia'], key='text_prompt')])" + ] + }, + "execution_count": 41, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "result_dicts = analyze_image(image = virginia_map_url, prompt = prompt_dicts)\n", + "result_dicts" + ] + }, + { + "cell_type": "code", + "execution_count": 42, + "id": "bda129d9-807a-4d34-9a73-57f677a77ed1", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "95" + ] + }, + "execution_count": 42, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "result_dicts.results[0].value" + ] + }, + { + "cell_type": "code", + "execution_count": 43, + "id": "95980b7f-7a17-437c-8a0d-b0c00d751fc8", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Ohio',\n", + " 'Pennsylvania',\n", + " 'Maryland',\n", + " 'Delaware',\n", + " 'Kentucky',\n", + " 'Tennessee',\n", + " 'North Carolina',\n", + " 'Virginia']" + ] + }, + "execution_count": 43, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "dicts_result.results[1].value" + ] + }, + { + "cell_type": "markdown", + "id": "c1395c40-3097-4a9a-9d20-2635ee53a9c1", + "metadata": {}, + "source": [ + "The tool here currently identifies 95 sub-boundaries (counties) for Virginia, and detects all the states mentioned in the map image. " + ] + }, + { + "cell_type": "markdown", + "id": "efc61dca-97f4-4e72-bcbe-63a6e618f48a", + "metadata": {}, + "source": [ + "### __Analyze Text__\n", + "\n", + "This tool analyzes the provided text against the given prompt(s) using AI capabilities. Users are allowed to submit text for analysis, which can include tasks such as sentiment analysis, entity recognition, and other natural language processing operations.\n", + "\n", + "Similar to the `analyze_image()` method, prompts to the `analyze_text()` method may be provided as string, list or dictionary inputs. We will see examples for all 3 input types below. " + ] + }, + { + "cell_type": "markdown", + "id": "a568c69f-fdc2-4b3f-959b-528bbd8d16d4", + "metadata": {}, + "source": [ + "#### 1. Prompt as `str` input" + ] + }, + { + "cell_type": "code", + "execution_count": 48, + "id": "0235aef9-9bb7-416f-ba64-88516e53e7ff", + "metadata": {}, + "outputs": [], + "source": [ + "sample_text = \"During our recent travels, we explored bustling city centers like London and Prague, \\\n", + " wandered through the ancient, narrow streets of Rome, and relaxed on the sunny beaches of \\\n", + " Alanya near the historic Alanya Castle. We visited the majestic Taj Mahal in Agra, toured the \\\n", + " expansive Amber Palace in Jaipur, and took a relaxing break by the calm waters of the Dim River.\"\n", + "\n", + "prompt_str = \"Detect all locations in this text\"" + ] + }, + { + "cell_type": "code", + "execution_count": 50, + "id": "6f14b6d6-3ec0-4f1e-b99b-7865addc5eb3", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "AIUtilsResponse(success=True, meta=AIUtilsResponseMetaData(usage={'inputTokens': 750, 'outputTokens': 55, 'totalTokens': 805}, model='gpt-5.4'), results=[AnalyzeResult(success=True, value=['London', 'Prague', 'Rome', 'Alanya', 'Alanya Castle', 'Taj Mahal', 'Agra', 'Amber Palace', 'Jaipur', 'Dim River'], key='key:1')])" + ] + }, + "execution_count": 50, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "locations = analyze_text(text = sample_text, prompt = prompt_str)\n", + "locations" + ] + }, + { + "cell_type": "code", + "execution_count": 51, + "id": "c69102c3-84a0-4587-b406-41ff70fd7050", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['London',\n", + " 'Prague',\n", + " 'Rome',\n", + " 'Alanya',\n", + " 'Alanya Castle',\n", + " 'Taj Mahal',\n", + " 'Agra',\n", + " 'Amber Palace',\n", + " 'Jaipur',\n", + " 'Dim River']" + ] + }, + "execution_count": 51, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "locations.results[0].value" + ] + }, + { + "cell_type": "markdown", + "id": "41532a46-9e05-43c6-802b-e3ac73720b80", + "metadata": {}, + "source": [ + "#### 2. Prompt as `list` input" + ] + }, + { + "cell_type": "code", + "execution_count": 60, + "id": "f754e903-f258-4dff-8a07-ebc3be38296a", + "metadata": {}, + "outputs": [], + "source": [ + "prompt_list = [\n", + " \"Detect the sentiment of the text.\",\n", + " \"Identify countries in the text.\"\n", + "]" + ] + }, + { + "cell_type": "code", + "execution_count": 61, + "id": "53a34b53-6881-41fc-91ab-f98f69f5bf35", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "AIUtilsResponse(success=True, meta=AIUtilsResponseMetaData(usage={'inputTokens': 768, 'outputTokens': 54, 'totalTokens': 822}, model='gpt-5.4'), results=[AnalyzeResult(success=True, value='positive', key='unique_string_0'), AnalyzeResult(success=True, value=['United Kingdom', 'Czech Republic', 'Italy', 'Turkey', 'India'], key='unique_string_1')])" + ] + }, + "execution_count": 61, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "result_list = analyze_text(text = sample_text, prompt = prompt_list)\n", + "result_list" + ] + }, + { + "cell_type": "markdown", + "id": "c930517e-7a48-4b0e-8a4d-05894ab0bf59", + "metadata": {}, + "source": [ + "We can access results for the individual prompts via the prompt index provided to the `results` field as demonstrated below." + ] + }, + { + "cell_type": "code", + "execution_count": 62, + "id": "fea6b707-b448-4e18-b725-6bf782f82743", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "'positive'" + ] + }, + "execution_count": 62, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "result_list.results[0].value" + ] + }, + { + "cell_type": "code", + "execution_count": 63, + "id": "0275cf94-021d-4d95-913c-74bd9accc9d5", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['United Kingdom', 'Czech Republic', 'Italy', 'Turkey', 'India']" + ] + }, + "execution_count": 63, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "result_list.results[1].value" + ] + }, + { + "cell_type": "markdown", + "id": "bbed4555-ab32-413e-8f6b-0d7c9cf2faec", + "metadata": {}, + "source": [ + "#### 3. Prompt as `dict` input\n", + "\n", + "We will now look at an example that uses a dictionary of prompt inputs using an ArcGIS item. This is a [document](https://www.arcgis.com/home/item.html?id=c7b4e9240149481ea69af628513962fe#overview) about Esri's Water Resources. Let us learn more about this item. " + ] + }, + { + "cell_type": "code", + "execution_count": 66, + "id": "50fedaa5-909b-43c6-aec6-8ab7276341b8", + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "