├── 1. Vectors embeddings with Azure Computer Vision.ipynb ├── 2. Visual search - Azure Cognitive Search index creation.ipynb ├── 2D_masterCategory.png ├── 2D_season.png ├── 3. Visual search - examples.ipynb ├── 3D_masterCategory.png ├── 4. Visual search - webapp.ipynb ├── 5. Visual Search - UMAP.ipynb ├── 6. Visual search - Azure Open AI.ipynb ├── 7. Visual search - Index management.ipynb ├── README.md ├── acs1.png ├── azure.env ├── demovideo.jpg ├── images ├── car1.jpg ├── car2.jpg ├── car3.jpg ├── car4.jpg ├── car5.jpg ├── car6.jpg ├── car7.jpg ├── image1.jpg ├── image2.jpg ├── image3.jpg ├── image4.jpg ├── image5.jpg ├── image6.jpg ├── image7.jpg └── image8.jpg ├── img ├── image_search (1).jpg ├── image_search (2).jpg ├── image_search (3).jpg ├── prompt search (1).jpg ├── prompt search (2).jpg └── prompt search (3).jpg ├── logo.jpg ├── screenshots.gif └── styles.xlsx /2D_masterCategory.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/retkowsky/visual-search-azureAI/65cc91881486532e1bc48467076d24a2c8e1457e/2D_masterCategory.png -------------------------------------------------------------------------------- /2D_season.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/retkowsky/visual-search-azureAI/65cc91881486532e1bc48467076d24a2c8e1457e/2D_season.png -------------------------------------------------------------------------------- /3D_masterCategory.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/retkowsky/visual-search-azureAI/65cc91881486532e1bc48467076d24a2c8e1457e/3D_masterCategory.png -------------------------------------------------------------------------------- /4. Visual search - webapp.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "\n", 8 | "\n", 9 | "# Fashion Visual Search demo\n", 10 | "\n", 11 | "## 4. Visual search webapp\n", 12 | "\n", 13 | "This code demonstrates how to use **Azure Cognitive Search** with **Cognitive Services Florence Vision API** and Azure Python SDK for visual search.\n", 14 | "\n", 15 | "\n", 16 | "## Steps\n", 17 | "- Connect to a blob storage where your catalog images are\n", 18 | "- Use Azure Computer Vision 4 to embed all these images\n", 19 | "- Create an Azure Cognitive search vector store index\n", 20 | "- Upload the embeddings into an Azure Cognitive Search index\n", 21 | "- Do some visual search using a prompt or an image\n", 22 | "\n", 23 | "\n", 24 | "## Visual search with vector embeddings\n", 25 | "Vector embeddings are a way of representing content such as text or images as vectors of real numbers in a high-dimensional space. These embeddings are often learned from large amounts of textual and visual data using machine learning algorithms like neural networks. Each dimension of the vector corresponds to a different feature or attribute of the content, such as its semantic meaning, syntactic role, or context in which it commonly appears. By representing content as vectors, we can perform mathematical operations on them to compare their similarity or use them as inputs to machine learning models.\n", 26 | "\n", 27 | "## Process\n", 28 | "\n", 29 | "\n", 30 | "## Business applications\n", 31 | "- Digital asset management: Image retrieval can be used to manage large collections of digital images, such as in museums, archives, or online galleries. Users can search for images based on visual features and retrieve the images that match their criteria.\n", 32 | "- Medical image retrieval: Image retrieval can be used in medical imaging to search for images based on their diagnostic features or disease patterns. This can help doctors or researchers to identify similar cases or track disease progression.\n", 33 | "- Security and surveillance: Image retrieval can be used in security and surveillance systems to search for images based on specific features or patterns, such as in, people & object tracking, or threat detection.\n", 34 | "- Forensic image retrieval: Image retrieval can be used in forensic investigations to search for images based on their visual content or metadata, such as in cases of cyber-crime.\n", 35 | "- E-commerce: Image retrieval can be used in online shopping applications to search for similar products based on their features or descriptions or provide recommendations based on previous purchases.\n", 36 | "- Fashion and design: Image retrieval can be used in fashion and design to search for images based on their visual features, such as color, pattern, or texture. This can help designers or retailers to identify similar products or trends.\n", 37 | "\n", 38 | "## To learn more\n", 39 | "- https://learn.microsoft.com/en-us/azure/cognitive-services/computer-vision/concept-image-retrieval\n", 40 | "- https://learn.microsoft.com/en-us/azure/search/search-what-is-azure-search\n", 41 | "\n", 42 | "In this notebook we took some samples fashion images are taken from this link:
\n", 43 | "https://www.kaggle.com/datasets/paramaggarwal/fashion-product-images-dataset\n", 44 | "\n", 45 | "> Note: Image retrieval is curently in public preview" 46 | ] 47 | }, 48 | { 49 | "cell_type": "markdown", 50 | "metadata": {}, 51 | "source": [ 52 | "## 1. Python librairies" 53 | ] 54 | }, 55 | { 56 | "cell_type": "code", 57 | "execution_count": 1, 58 | "metadata": {}, 59 | "outputs": [], 60 | "source": [ 61 | "import datetime\n", 62 | "import io\n", 63 | "import json\n", 64 | "import os\n", 65 | "import requests\n", 66 | "import sys\n", 67 | "import gradio as gr\n", 68 | "\n", 69 | "from dotenv import load_dotenv\n", 70 | "from PIL import Image\n", 71 | "\n", 72 | "from azure.core.credentials import AzureKeyCredential\n", 73 | "from azure.search.documents import SearchClient\n", 74 | "from azure.search.documents.indexes import SearchIndexClient\n", 75 | "from azure.search.documents.models import Vector\n", 76 | "\n", 77 | "from azure.storage.blob import BlobServiceClient" 78 | ] 79 | }, 80 | { 81 | "cell_type": "code", 82 | "execution_count": 2, 83 | "metadata": {}, 84 | "outputs": [ 85 | { 86 | "data": { 87 | "text/plain": [ 88 | "'3.8.5 (default, Sep 4 2020, 07:30:14) \\n[GCC 7.3.0]'" 89 | ] 90 | }, 91 | "execution_count": 2, 92 | "metadata": {}, 93 | "output_type": "execute_result" 94 | } 95 | ], 96 | "source": [ 97 | "sys.version" 98 | ] 99 | }, 100 | { 101 | "cell_type": "code", 102 | "execution_count": 3, 103 | "metadata": {}, 104 | "outputs": [ 105 | { 106 | "name": "stdout", 107 | "output_type": "stream", 108 | "text": [ 109 | "Today is 2023-06-30 07:49:41.488610\n" 110 | ] 111 | } 112 | ], 113 | "source": [ 114 | "print(\"Today is\", datetime.datetime.today())" 115 | ] 116 | }, 117 | { 118 | "cell_type": "markdown", 119 | "metadata": {}, 120 | "source": [ 121 | "## 2. Azure AI Services" 122 | ] 123 | }, 124 | { 125 | "cell_type": "code", 126 | "execution_count": 4, 127 | "metadata": {}, 128 | "outputs": [], 129 | "source": [ 130 | "load_dotenv(\"azure.env\")\n", 131 | "\n", 132 | "# Azure Computer Vision 4\n", 133 | "acv_key = os.getenv(\"acv_key\")\n", 134 | "acv_endpoint = os.getenv(\"acv_endpoint\")\n", 135 | "\n", 136 | "# Azure Cognitive Search\n", 137 | "acs_endpoint = os.getenv(\"acs_endpoint\")\n", 138 | "acs_key = os.getenv(\"acs_key\")" 139 | ] 140 | }, 141 | { 142 | "cell_type": "code", 143 | "execution_count": 5, 144 | "metadata": {}, 145 | "outputs": [], 146 | "source": [ 147 | "# Ensure that the azure endpoints should not finished a /\n", 148 | "if acv_endpoint.endswith(\"/\"):\n", 149 | " acv_endpoint = acv_endpoint[:-1]\n", 150 | "\n", 151 | "if acs_endpoint.endswith(\"/\"):\n", 152 | " acs_endpoint = acv_endpoint[:-1]" 153 | ] 154 | }, 155 | { 156 | "cell_type": "code", 157 | "execution_count": 6, 158 | "metadata": {}, 159 | "outputs": [], 160 | "source": [ 161 | "# Azure storage account\n", 162 | "blob_connection_string = os.getenv(\"blob_connection_string\")\n", 163 | "container_name = os.getenv(\"container_name\")" 164 | ] 165 | }, 166 | { 167 | "cell_type": "markdown", 168 | "metadata": {}, 169 | "source": [ 170 | "## 3. Functions & parameters" 171 | ] 172 | }, 173 | { 174 | "cell_type": "code", 175 | "execution_count": 7, 176 | "metadata": {}, 177 | "outputs": [], 178 | "source": [ 179 | "# Azure Cognitive Search index name to create\n", 180 | "index_name = \"azure-fashion-demo\"\n", 181 | "\n", 182 | "# Azure Cognitive Search api version\n", 183 | "api_version = \"2023-02-01-preview\"" 184 | ] 185 | }, 186 | { 187 | "cell_type": "code", 188 | "execution_count": 8, 189 | "metadata": {}, 190 | "outputs": [], 191 | "source": [ 192 | "def text_embedding(prompt):\n", 193 | " \"\"\"\n", 194 | " Text embedding using Azure Computer Vision 4.0\n", 195 | " \"\"\"\n", 196 | " version = \"?api-version=\" + api_version + \"&modelVersion=latest\"\n", 197 | " vec_txt_url = f\"{acv_endpoint}/computervision/retrieval:vectorizeText{version}\"\n", 198 | " headers = {\"Content-type\": \"application/json\", \"Ocp-Apim-Subscription-Key\": acv_key}\n", 199 | "\n", 200 | " payload = {\"text\": prompt}\n", 201 | " response = requests.post(vec_txt_url, json=payload, headers=headers)\n", 202 | "\n", 203 | " if response.status_code == 200:\n", 204 | " text_emb = response.json().get(\"vector\")\n", 205 | " return text_emb\n", 206 | "\n", 207 | " else:\n", 208 | " print(f\"Error: {response.status_code} - {response.text}\")\n", 209 | " return None" 210 | ] 211 | }, 212 | { 213 | "cell_type": "code", 214 | "execution_count": 9, 215 | "metadata": {}, 216 | "outputs": [], 217 | "source": [ 218 | "def index_stats(index_name):\n", 219 | " \"\"\"\n", 220 | " Get statistics about Azure Cognitive Search index\n", 221 | " \"\"\"\n", 222 | " url = (\n", 223 | " acs_endpoint\n", 224 | " + \"/indexes/\"\n", 225 | " + index_name\n", 226 | " + \"/stats?api-version=2021-04-30-Preview\"\n", 227 | " )\n", 228 | " headers = {\n", 229 | " \"Content-Type\": \"application/json\",\n", 230 | " \"api-key\": acs_key,\n", 231 | " }\n", 232 | " response = requests.get(url, headers=headers)\n", 233 | " print(\"Azure Cognitive Search index status for:\", index_name, \"\\n\")\n", 234 | "\n", 235 | " if response.status_code == 200:\n", 236 | " res = response.json()\n", 237 | " print(json.dumps(res, indent=2))\n", 238 | "\n", 239 | " else:\n", 240 | " print(\"Request failed with status code:\", response.status_code)" 241 | ] 242 | }, 243 | { 244 | "cell_type": "code", 245 | "execution_count": 10, 246 | "metadata": {}, 247 | "outputs": [], 248 | "source": [ 249 | "def index_status(index_name):\n", 250 | " \"\"\"\n", 251 | " Azure Cognitive Search index status\n", 252 | " \"\"\"\n", 253 | " print(\"Azure Cognitive Search Index:\", index_name, \"\\n\")\n", 254 | "\n", 255 | " headers = {\"Content-Type\": \"application/json\", \"api-key\": acs_key}\n", 256 | " params = {\"api-version\": \"2021-04-30-Preview\"}\n", 257 | " index_status = requests.get(\n", 258 | " acs_endpoint + \"/indexes/\" + index_name, headers=headers, params=params\n", 259 | " )\n", 260 | " try:\n", 261 | " print(json.dumps((index_status.json()), indent=5))\n", 262 | " except:\n", 263 | " print(\"Request failed\")" 264 | ] 265 | }, 266 | { 267 | "cell_type": "code", 268 | "execution_count": 11, 269 | "metadata": {}, 270 | "outputs": [ 271 | { 272 | "name": "stdout", 273 | "output_type": "stream", 274 | "text": [ 275 | "URL of the first blob: https://azurestorageaccountsr.blob.core.windows.net/fashionimages/fashion/0390469004.jpg\n" 276 | ] 277 | } 278 | ], 279 | "source": [ 280 | "# Connect to Blob Storage\n", 281 | "blob_service_client = BlobServiceClient.from_connection_string(blob_connection_string)\n", 282 | "container_client = blob_service_client.get_container_client(container_name)\n", 283 | "blobs = container_client.list_blobs()\n", 284 | "\n", 285 | "first_blob = next(blobs)\n", 286 | "blob_url = container_client.get_blob_client(first_blob).url\n", 287 | "print(f\"URL of the first blob: {blob_url}\")" 288 | ] 289 | }, 290 | { 291 | "cell_type": "markdown", 292 | "metadata": {}, 293 | "source": [ 294 | "## 4. Azure Cognitive Search" 295 | ] 296 | }, 297 | { 298 | "cell_type": "code", 299 | "execution_count": 12, 300 | "metadata": {}, 301 | "outputs": [ 302 | { 303 | "name": "stdout", 304 | "output_type": "stream", 305 | "text": [ 306 | "Setting the Azure Cognitive Search client\n", 307 | "Done\n", 308 | "\n" 309 | ] 310 | } 311 | ], 312 | "source": [ 313 | "try:\n", 314 | " # Setting the Azure Cognitive Search client\n", 315 | " print(\"Setting the Azure Cognitive Search client\")\n", 316 | " search_client = SearchIndexClient(\n", 317 | " endpoint=acs_endpoint, credential=AzureKeyCredential(acs_key)\n", 318 | " )\n", 319 | " print(\"Done\")\n", 320 | " print(search_client)\n", 321 | "\n", 322 | "except:\n", 323 | " print(\"Request failed. Cannot create Azure Cognitive Search client:\", acs_endpoint)" 324 | ] 325 | }, 326 | { 327 | "cell_type": "markdown", 328 | "metadata": {}, 329 | "source": [ 330 | "### Azure Cognitive Search index status" 331 | ] 332 | }, 333 | { 334 | "cell_type": "code", 335 | "execution_count": 13, 336 | "metadata": {}, 337 | "outputs": [ 338 | { 339 | "name": "stdout", 340 | "output_type": "stream", 341 | "text": [ 342 | "Azure Cognitive Search index status for: azure-fashion-demo \n", 343 | "\n", 344 | "{\n", 345 | " \"@odata.context\": \"https://azurecogsearcheastussr.search.windows.net/$metadata#Microsoft.Azure.Search.V2021_04_30_Preview.IndexStatistics\",\n", 346 | " \"documentCount\": 10226,\n", 347 | " \"storageSize\": 155596726\n", 348 | "}\n" 349 | ] 350 | } 351 | ], 352 | "source": [ 353 | "index_stats(index_name)" 354 | ] 355 | }, 356 | { 357 | "cell_type": "code", 358 | "execution_count": 14, 359 | "metadata": {}, 360 | "outputs": [ 361 | { 362 | "name": "stdout", 363 | "output_type": "stream", 364 | "text": [ 365 | "Azure Cognitive Search Index: azure-fashion-demo \n", 366 | "\n", 367 | "{\n", 368 | " \"@odata.context\": \"https://azurecogsearcheastussr.search.windows.net/$metadata#indexes/$entity\",\n", 369 | " \"@odata.etag\": \"\\\"0x8DB7639E6632A9E\\\"\",\n", 370 | " \"name\": \"azure-fashion-demo\",\n", 371 | " \"defaultScoringProfile\": null,\n", 372 | " \"fields\": [\n", 373 | " {\n", 374 | " \"name\": \"idfile\",\n", 375 | " \"type\": \"Edm.String\",\n", 376 | " \"searchable\": false,\n", 377 | " \"filterable\": false,\n", 378 | " \"retrievable\": true,\n", 379 | " \"sortable\": false,\n", 380 | " \"facetable\": false,\n", 381 | " \"key\": true,\n", 382 | " \"indexAnalyzer\": null,\n", 383 | " \"searchAnalyzer\": null,\n", 384 | " \"analyzer\": null,\n", 385 | " \"normalizer\": null,\n", 386 | " \"synonymMaps\": []\n", 387 | " },\n", 388 | " {\n", 389 | " \"name\": \"imagefile\",\n", 390 | " \"type\": \"Edm.String\",\n", 391 | " \"searchable\": true,\n", 392 | " \"filterable\": false,\n", 393 | " \"retrievable\": true,\n", 394 | " \"sortable\": false,\n", 395 | " \"facetable\": false,\n", 396 | " \"key\": false,\n", 397 | " \"indexAnalyzer\": null,\n", 398 | " \"searchAnalyzer\": null,\n", 399 | " \"analyzer\": null,\n", 400 | " \"normalizer\": null,\n", 401 | " \"synonymMaps\": []\n", 402 | " },\n", 403 | " {\n", 404 | " \"name\": \"imagevector\",\n", 405 | " \"type\": \"Collection(Edm.Single)\",\n", 406 | " \"searchable\": true,\n", 407 | " \"filterable\": false,\n", 408 | " \"retrievable\": true,\n", 409 | " \"sortable\": false,\n", 410 | " \"facetable\": false,\n", 411 | " \"key\": false,\n", 412 | " \"indexAnalyzer\": null,\n", 413 | " \"searchAnalyzer\": null,\n", 414 | " \"analyzer\": null,\n", 415 | " \"normalizer\": null,\n", 416 | " \"synonymMaps\": []\n", 417 | " }\n", 418 | " ],\n", 419 | " \"scoringProfiles\": [],\n", 420 | " \"corsOptions\": null,\n", 421 | " \"suggesters\": [],\n", 422 | " \"analyzers\": [],\n", 423 | " \"normalizers\": [],\n", 424 | " \"tokenizers\": [],\n", 425 | " \"tokenFilters\": [],\n", 426 | " \"charFilters\": [],\n", 427 | " \"encryptionKey\": null,\n", 428 | " \"similarity\": {\n", 429 | " \"@odata.type\": \"#Microsoft.Azure.Search.BM25Similarity\",\n", 430 | " \"k1\": null,\n", 431 | " \"b\": null\n", 432 | " },\n", 433 | " \"semantic\": {\n", 434 | " \"defaultConfiguration\": null,\n", 435 | " \"configurations\": [\n", 436 | " {\n", 437 | " \"name\": \"my-semantic-config\",\n", 438 | " \"prioritizedFields\": {\n", 439 | " \"titleField\": {\n", 440 | " \"fieldName\": \"idfile\"\n", 441 | " },\n", 442 | " \"prioritizedContentFields\": [],\n", 443 | " \"prioritizedKeywordsFields\": []\n", 444 | " }\n", 445 | " }\n", 446 | " ]\n", 447 | " }\n", 448 | "}\n" 449 | ] 450 | } 451 | ], 452 | "source": [ 453 | "index_status(index_name)" 454 | ] 455 | }, 456 | { 457 | "cell_type": "markdown", 458 | "metadata": {}, 459 | "source": [ 460 | "## 5. Webapp" 461 | ] 462 | }, 463 | { 464 | "cell_type": "code", 465 | "execution_count": 15, 466 | "metadata": {}, 467 | "outputs": [], 468 | "source": [ 469 | "topn = 5\n", 470 | "imgsize = 360\n", 471 | "\n", 472 | "footnote = \"Powered by Azure Computer Vision & Azure Cognitive Search\"\n", 473 | "header_prompt = \"Visual Search with Azure AI using a prompt\"\n", 474 | "header_images = \"Visual Search with Azure AI using a prompt\"\n", 475 | "\n", 476 | "logo = \"https://github.com/retkowsky/visual-search-azureAI/blob/main/logo.jpg?raw=true\"\n", 477 | "image = \"
\".format(logo)\n", 478 | "\n", 479 | "# Themes: https://huggingface.co/spaces/gradio/theme-gallery\n", 480 | "theme = \"snehilsanyal/scikit-learn\"" 481 | ] 482 | }, 483 | { 484 | "cell_type": "code", 485 | "execution_count": 16, 486 | "metadata": {}, 487 | "outputs": [], 488 | "source": [ 489 | "def prompt_search_gradio(prompt, topn=5):\n", 490 | " \"\"\"\n", 491 | " Prompt search for the gradio webapp\n", 492 | " \"\"\"\n", 493 | " results_list = []\n", 494 | " images_list = []\n", 495 | "\n", 496 | " # Initialize the Azure Cognitive Search client\n", 497 | " search_client = SearchClient(acs_endpoint, index_name, AzureKeyCredential(acs_key))\n", 498 | " # Perform vector search\n", 499 | " vector = Vector(value=text_embedding(prompt), k=topn, fields=\"imagevector\")\n", 500 | " response = search_client.search(\n", 501 | " search_text=prompt, vector=vector, select=[\"idfile\", \"imagefile\"]\n", 502 | " )\n", 503 | "\n", 504 | " for result in response:\n", 505 | " image_file = result[\"imagefile\"]\n", 506 | " results_list.append(image_file)\n", 507 | "\n", 508 | " for image_file in results_list:\n", 509 | " blob_client = container_client.get_blob_client(image_file)\n", 510 | " blob_image = blob_client.download_blob().readall()\n", 511 | " img = Image.open(io.BytesIO(blob_image)).resize((imgsize, imgsize))\n", 512 | " images_list.append(img)\n", 513 | "\n", 514 | " return images_list" 515 | ] 516 | }, 517 | { 518 | "cell_type": "code", 519 | "execution_count": 17, 520 | "metadata": {}, 521 | "outputs": [], 522 | "source": [ 523 | "prompt_examples = [\n", 524 | " \"a red dress\",\n", 525 | " \"a red dress with long sleeves\",\n", 526 | " \"Articles with birds\",\n", 527 | " \"Hair products\",\n", 528 | " \"Ray-Ban\",\n", 529 | " \"NYC cap\",\n", 530 | " \"Camo products\",\n", 531 | " \"A blue shirt with stripped lines\",\n", 532 | " \"Ray ban with leopard frames\",\n", 533 | " \"Show me some articles with Italian names\",\n", 534 | " \"Show me some articles with Japanese on it\",\n", 535 | " \"Show me some tie-dye articles\",\n", 536 | "]\n", 537 | "\n", 538 | "prompt = gr.components.Textbox(\n", 539 | " lines=2,\n", 540 | " label=\"What do you want to search?\",\n", 541 | " placeholder=\"Enter your prompt for the visual search...\",\n", 542 | ")\n", 543 | "\n", 544 | "topn_list_prompt = [\"\"] * topn\n", 545 | "\n", 546 | "list_img_results_image = [\n", 547 | " gr.components.Image(label=f\"Top {i+1}: {topn_list_prompt[i]}\", type=\"filepath\")\n", 548 | " for i in range(topn)\n", 549 | "]\n", 550 | "\n", 551 | "webapp_prompt = gr.Interface(\n", 552 | " prompt_search_gradio,\n", 553 | " prompt,\n", 554 | " list_img_results_image,\n", 555 | " title=header_prompt,\n", 556 | " examples=prompt_examples,\n", 557 | " theme=theme,\n", 558 | " description=image,\n", 559 | " article=footnote,\n", 560 | ")" 561 | ] 562 | }, 563 | { 564 | "cell_type": "code", 565 | "execution_count": 18, 566 | "metadata": {}, 567 | "outputs": [], 568 | "source": [ 569 | "# webapp_prompt.launch(share=True)" 570 | ] 571 | }, 572 | { 573 | "cell_type": "code", 574 | "execution_count": 19, 575 | "metadata": {}, 576 | "outputs": [], 577 | "source": [ 578 | "def image_embedding(imagefile):\n", 579 | " \"\"\"\n", 580 | " Image embedding using Azure Computer Vision 4.0\n", 581 | " \"\"\"\n", 582 | " session = requests.Session()\n", 583 | "\n", 584 | " version = \"?api-version=\" + api_version + \"&modelVersion=latest\"\n", 585 | " vec_img_url = acv_endpoint + \"/computervision/retrieval:vectorizeImage\" + version\n", 586 | " headers = {\n", 587 | " \"Content-type\": \"application/octet-stream\",\n", 588 | " \"Ocp-Apim-Subscription-Key\": acv_key,\n", 589 | " }\n", 590 | "\n", 591 | " try:\n", 592 | " with open(imagefile, \"rb\") as f:\n", 593 | " data = f.read()\n", 594 | " response = session.post(vec_img_url, data=data, headers=headers)\n", 595 | " response.raise_for_status()\n", 596 | "\n", 597 | " image_emb = response.json()[\"vector\"]\n", 598 | " return image_emb\n", 599 | "\n", 600 | " except requests.exceptions.RequestException as e:\n", 601 | " print(f\"Request Exception: {e}\")\n", 602 | " except Exception as ex:\n", 603 | " print(f\"Error: {ex}\")\n", 604 | "\n", 605 | " return None" 606 | ] 607 | }, 608 | { 609 | "cell_type": "code", 610 | "execution_count": 20, 611 | "metadata": {}, 612 | "outputs": [], 613 | "source": [ 614 | "def image_search_gradio(imagefile, topn=5):\n", 615 | " \"\"\"\n", 616 | " Image search for the gradio webapp\n", 617 | " \"\"\"\n", 618 | " results_list = []\n", 619 | " images_list = []\n", 620 | "\n", 621 | " # Initialize the Azure Cognitive Search client\n", 622 | " search_client = SearchClient(acs_endpoint, index_name, AzureKeyCredential(acs_key))\n", 623 | "\n", 624 | " # Perform vector search\n", 625 | " response = search_client.search(\n", 626 | " search_text=\"\",\n", 627 | " vector=Vector(value=image_embedding(imagefile), k=topn, fields=\"imagevector\"),\n", 628 | " select=[\"idfile\", \"imagefile\"],\n", 629 | " )\n", 630 | "\n", 631 | " for result in response:\n", 632 | " image_file = result[\"imagefile\"]\n", 633 | " results_list.append(image_file)\n", 634 | "\n", 635 | " for image_file in results_list:\n", 636 | " blob_client = container_client.get_blob_client(image_file)\n", 637 | " blob_image = blob_client.download_blob().readall()\n", 638 | " img = Image.open(io.BytesIO(blob_image)).resize((imgsize, imgsize))\n", 639 | " images_list.append(img)\n", 640 | "\n", 641 | " return images_list" 642 | ] 643 | }, 644 | { 645 | "cell_type": "code", 646 | "execution_count": 21, 647 | "metadata": {}, 648 | "outputs": [], 649 | "source": [ 650 | "images_examples = [\n", 651 | " \"images/image1.jpg\",\n", 652 | " \"images/image2.jpg\",\n", 653 | " \"images/image3.jpg\",\n", 654 | " \"images/image4.jpg\",\n", 655 | " \"images/image5.jpg\",\n", 656 | " \"images/image6.jpg\",\n", 657 | " \"images/image7.jpg\",\n", 658 | " \"images/image8.jpg\",\n", 659 | "]\n", 660 | "\n", 661 | "refimage = gr.components.Image(label=\"Your image:\", type=\"filepath\", shape=((imgsize, imgsize)))\n", 662 | "topn_list_prompt = [\"\"] * topn\n", 663 | "\n", 664 | "list_img_results_image = [\n", 665 | " gr.components.Image(label=f\"Top {i+1} : {topn_list_prompt[i]}\", type=\"filepath\")\n", 666 | " for i in range(topn)\n", 667 | "]\n", 668 | "\n", 669 | "webapp_image = gr.Interface(\n", 670 | " image_search_gradio,\n", 671 | " refimage,\n", 672 | " list_img_results_image,\n", 673 | " title=header_images,\n", 674 | " examples=images_examples,\n", 675 | " theme=theme,\n", 676 | " description=image,\n", 677 | " article=footnote,\n", 678 | ")" 679 | ] 680 | }, 681 | { 682 | "cell_type": "code", 683 | "execution_count": 22, 684 | "metadata": {}, 685 | "outputs": [], 686 | "source": [ 687 | "# webapp_image.launch(share=True)" 688 | ] 689 | }, 690 | { 691 | "cell_type": "markdown", 692 | "metadata": {}, 693 | "source": [ 694 | "### Combining the two gradio apps into one" 695 | ] 696 | }, 697 | { 698 | "cell_type": "code", 699 | "execution_count": 23, 700 | "metadata": {}, 701 | "outputs": [ 702 | { 703 | "name": "stderr", 704 | "output_type": "stream", 705 | "text": [ 706 | "/anaconda/envs/jupyter_env/lib/python3.8/site-packages/gradio/blocks.py:922: UserWarning: api_name predict already exists, using predict_1\n", 707 | " warnings.warn(\n", 708 | "/anaconda/envs/jupyter_env/lib/python3.8/site-packages/gradio/blocks.py:922: UserWarning: api_name load_example already exists, using load_example_1\n", 709 | " warnings.warn(\n" 710 | ] 711 | } 712 | ], 713 | "source": [ 714 | "visual_search_webapp = gr.TabbedInterface(\n", 715 | " [webapp_prompt, webapp_image],\n", 716 | " [\n", 717 | " \"1) Visual search from a prompt\",\n", 718 | " \"2) Visual search from an image\"\n", 719 | " ],\n", 720 | " css=\"body {background-color: black}\",\n", 721 | " theme=theme,\n", 722 | ")" 723 | ] 724 | }, 725 | { 726 | "cell_type": "code", 727 | "execution_count": 24, 728 | "metadata": {}, 729 | "outputs": [ 730 | { 731 | "name": "stdout", 732 | "output_type": "stream", 733 | "text": [ 734 | "Running on local URL: http://127.0.0.1:7860\n", 735 | "Running on public URL: https://828c9920912687a0b6.gradio.live\n", 736 | "\n", 737 | "This share link expires in 72 hours. For free permanent hosting and GPU upgrades, run `gradio deploy` from Terminal to deploy to Spaces (https://huggingface.co/spaces)\n" 738 | ] 739 | }, 740 | { 741 | "data": { 742 | "text/html": [ 743 | "
" 744 | ], 745 | "text/plain": [ 746 | "" 747 | ] 748 | }, 749 | "metadata": {}, 750 | "output_type": "display_data" 751 | }, 752 | { 753 | "data": { 754 | "text/plain": [] 755 | }, 756 | "execution_count": 24, 757 | "metadata": {}, 758 | "output_type": "execute_result" 759 | } 760 | ], 761 | "source": [ 762 | "visual_search_webapp.launch(share=True)" 763 | ] 764 | }, 765 | { 766 | "cell_type": "markdown", 767 | "metadata": {}, 768 | "source": [ 769 | "## 6. Post processing\n", 770 | "\n", 771 | "We can delete the index if needed" 772 | ] 773 | }, 774 | { 775 | "cell_type": "code", 776 | "execution_count": 25, 777 | "metadata": {}, 778 | "outputs": [ 779 | { 780 | "name": "stdout", 781 | "output_type": "stream", 782 | "text": [ 783 | "Azure Cognitive Search Index: azure-fashion-demo \n", 784 | "\n", 785 | "{\n", 786 | " \"@odata.context\": \"https://azurecogsearcheastussr.search.windows.net/$metadata#indexes/$entity\",\n", 787 | " \"@odata.etag\": \"\\\"0x8DB7639E6632A9E\\\"\",\n", 788 | " \"name\": \"azure-fashion-demo\",\n", 789 | " \"defaultScoringProfile\": null,\n", 790 | " \"fields\": [\n", 791 | " {\n", 792 | " \"name\": \"idfile\",\n", 793 | " \"type\": \"Edm.String\",\n", 794 | " \"searchable\": false,\n", 795 | " \"filterable\": false,\n", 796 | " \"retrievable\": true,\n", 797 | " \"sortable\": false,\n", 798 | " \"facetable\": false,\n", 799 | " \"key\": true,\n", 800 | " \"indexAnalyzer\": null,\n", 801 | " \"searchAnalyzer\": null,\n", 802 | " \"analyzer\": null,\n", 803 | " \"normalizer\": null,\n", 804 | " \"synonymMaps\": []\n", 805 | " },\n", 806 | " {\n", 807 | " \"name\": \"imagefile\",\n", 808 | " \"type\": \"Edm.String\",\n", 809 | " \"searchable\": true,\n", 810 | " \"filterable\": false,\n", 811 | " \"retrievable\": true,\n", 812 | " \"sortable\": false,\n", 813 | " \"facetable\": false,\n", 814 | " \"key\": false,\n", 815 | " \"indexAnalyzer\": null,\n", 816 | " \"searchAnalyzer\": null,\n", 817 | " \"analyzer\": null,\n", 818 | " \"normalizer\": null,\n", 819 | " \"synonymMaps\": []\n", 820 | " },\n", 821 | " {\n", 822 | " \"name\": \"imagevector\",\n", 823 | " \"type\": \"Collection(Edm.Single)\",\n", 824 | " \"searchable\": true,\n", 825 | " \"filterable\": false,\n", 826 | " \"retrievable\": true,\n", 827 | " \"sortable\": false,\n", 828 | " \"facetable\": false,\n", 829 | " \"key\": false,\n", 830 | " \"indexAnalyzer\": null,\n", 831 | " \"searchAnalyzer\": null,\n", 832 | " \"analyzer\": null,\n", 833 | " \"normalizer\": null,\n", 834 | " \"synonymMaps\": []\n", 835 | " }\n", 836 | " ],\n", 837 | " \"scoringProfiles\": [],\n", 838 | " \"corsOptions\": null,\n", 839 | " \"suggesters\": [],\n", 840 | " \"analyzers\": [],\n", 841 | " \"normalizers\": [],\n", 842 | " \"tokenizers\": [],\n", 843 | " \"tokenFilters\": [],\n", 844 | " \"charFilters\": [],\n", 845 | " \"encryptionKey\": null,\n", 846 | " \"similarity\": {\n", 847 | " \"@odata.type\": \"#Microsoft.Azure.Search.BM25Similarity\",\n", 848 | " \"k1\": null,\n", 849 | " \"b\": null\n", 850 | " },\n", 851 | " \"semantic\": {\n", 852 | " \"defaultConfiguration\": null,\n", 853 | " \"configurations\": [\n", 854 | " {\n", 855 | " \"name\": \"my-semantic-config\",\n", 856 | " \"prioritizedFields\": {\n", 857 | " \"titleField\": {\n", 858 | " \"fieldName\": \"idfile\"\n", 859 | " },\n", 860 | " \"prioritizedContentFields\": [],\n", 861 | " \"prioritizedKeywordsFields\": []\n", 862 | " }\n", 863 | " }\n", 864 | " ]\n", 865 | " }\n", 866 | "}\n" 867 | ] 868 | } 869 | ], 870 | "source": [ 871 | "index_status(index_name)" 872 | ] 873 | }, 874 | { 875 | "cell_type": "code", 876 | "execution_count": 26, 877 | "metadata": {}, 878 | "outputs": [ 879 | { 880 | "name": "stdout", 881 | "output_type": "stream", 882 | "text": [ 883 | "Azure Cognitive Search index status for: azure-fashion-demo \n", 884 | "\n", 885 | "{\n", 886 | " \"@odata.context\": \"https://azurecogsearcheastussr.search.windows.net/$metadata#Microsoft.Azure.Search.V2021_04_30_Preview.IndexStatistics\",\n", 887 | " \"documentCount\": 10226,\n", 888 | " \"storageSize\": 155596726\n", 889 | "}\n" 890 | ] 891 | } 892 | ], 893 | "source": [ 894 | "index_stats(index_name)" 895 | ] 896 | }, 897 | { 898 | "cell_type": "code", 899 | "execution_count": 27, 900 | "metadata": {}, 901 | "outputs": [], 902 | "source": [ 903 | "# delete_index(index_name)" 904 | ] 905 | }, 906 | { 907 | "cell_type": "code", 908 | "execution_count": null, 909 | "metadata": {}, 910 | "outputs": [], 911 | "source": [] 912 | } 913 | ], 914 | "metadata": { 915 | "kernelspec": { 916 | "display_name": "Python 3 (ipykernel)", 917 | "language": "python", 918 | "name": "python3" 919 | }, 920 | "language_info": { 921 | "codemirror_mode": { 922 | "name": "ipython", 923 | "version": 3 924 | }, 925 | "file_extension": ".py", 926 | "mimetype": "text/x-python", 927 | "name": "python", 928 | "nbconvert_exporter": "python", 929 | "pygments_lexer": "ipython3", 930 | "version": "3.8.5" 931 | } 932 | }, 933 | "nbformat": 4, 934 | "nbformat_minor": 2 935 | } 936 | -------------------------------------------------------------------------------- /7. Visual search - Index management.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "\n", 8 | "\n", 9 | "# Fashion Visual Search demo\n", 10 | "\n", 11 | "## 7. Visual search - Index management\n", 12 | "\n", 13 | "This code demonstrates how to use **Azure Cognitive Search** with **Cognitive Services Florence Vision API** and Azure Python SDK for visual search.\n", 14 | "\n", 15 | "## Visual search with vector embeddings\n", 16 | "Vector embeddings are a way of representing content such as text or images as vectors of real numbers in a high-dimensional space. These embeddings are often learned from large amounts of textual and visual data using machine learning algorithms like neural networks. Each dimension of the vector corresponds to a different feature or attribute of the content, such as its semantic meaning, syntactic role, or context in which it commonly appears. By representing content as vectors, we can perform mathematical operations on them to compare their similarity or use them as inputs to machine learning models.\n", 17 | "\n", 18 | "## Process\n", 19 | "\n", 20 | "\n", 21 | "## Business applications\n", 22 | "- Digital asset management: Image retrieval can be used to manage large collections of digital images, such as in museums, archives, or online galleries. Users can search for images based on visual features and retrieve the images that match their criteria.\n", 23 | "- Medical image retrieval: Image retrieval can be used in medical imaging to search for images based on their diagnostic features or disease patterns. This can help doctors or researchers to identify similar cases or track disease progression.\n", 24 | "- Security and surveillance: Image retrieval can be used in security and surveillance systems to search for images based on specific features or patterns, such as in, people & object tracking, or threat detection.\n", 25 | "- Forensic image retrieval: Image retrieval can be used in forensic investigations to search for images based on their visual content or metadata, such as in cases of cyber-crime.\n", 26 | "- E-commerce: Image retrieval can be used in online shopping applications to search for similar products based on their features or descriptions or provide recommendations based on previous purchases.\n", 27 | "- Fashion and design: Image retrieval can be used in fashion and design to search for images based on their visual features, such as color, pattern, or texture. This can help designers or retailers to identify similar products or trends.\n", 28 | "\n", 29 | "## To learn more\n", 30 | "- https://learn.microsoft.com/en-us/azure/cognitive-services/computer-vision/concept-image-retrieval\n", 31 | "- https://learn.microsoft.com/en-us/azure/search/search-what-is-azure-search\n", 32 | "\n", 33 | "In this notebook we took some samples fashion images are taken from this link:
\n", 34 | "https://www.kaggle.com/datasets/paramaggarwal/fashion-product-images-dataset\n", 35 | "\n", 36 | "> Note: Image retrieval is curently in public preview" 37 | ] 38 | }, 39 | { 40 | "cell_type": "markdown", 41 | "metadata": {}, 42 | "source": [ 43 | "## 1. Python librairies" 44 | ] 45 | }, 46 | { 47 | "cell_type": "code", 48 | "execution_count": 1, 49 | "metadata": {}, 50 | "outputs": [], 51 | "source": [ 52 | "import datetime\n", 53 | "import io\n", 54 | "import json\n", 55 | "import glob\n", 56 | "import math\n", 57 | "import os\n", 58 | "import requests\n", 59 | "import sys\n", 60 | "import time\n", 61 | "\n", 62 | "from azure.core.credentials import AzureKeyCredential\n", 63 | "from azure.search.documents import SearchClient\n", 64 | "from azure.search.documents.indexes import SearchIndexClient\n", 65 | "from azure.search.documents.indexes import SearchIndexerClient\n", 66 | "from azure.search.documents.indexes.models import (\n", 67 | " PrioritizedFields,\n", 68 | " SearchableField,\n", 69 | " SearchField,\n", 70 | " SearchFieldDataType,\n", 71 | " SearchIndex,\n", 72 | " SearchIndexerDataContainer,\n", 73 | " SearchIndexerDataSourceConnection,\n", 74 | " SemanticConfiguration,\n", 75 | " SemanticField,\n", 76 | " SemanticSettings,\n", 77 | " SimpleField,\n", 78 | " VectorSearch,\n", 79 | " VectorSearchAlgorithmConfiguration,\n", 80 | ")\n", 81 | "from azure.storage.blob import BlobServiceClient\n", 82 | "from dotenv import load_dotenv\n", 83 | "from io import BytesIO\n", 84 | "from PIL import Image" 85 | ] 86 | }, 87 | { 88 | "cell_type": "code", 89 | "execution_count": 2, 90 | "metadata": {}, 91 | "outputs": [ 92 | { 93 | "data": { 94 | "text/plain": [ 95 | "'3.8.5 (default, Sep 4 2020, 07:30:14) \\n[GCC 7.3.0]'" 96 | ] 97 | }, 98 | "execution_count": 2, 99 | "metadata": {}, 100 | "output_type": "execute_result" 101 | } 102 | ], 103 | "source": [ 104 | "sys.version" 105 | ] 106 | }, 107 | { 108 | "cell_type": "code", 109 | "execution_count": 3, 110 | "metadata": {}, 111 | "outputs": [ 112 | { 113 | "name": "stdout", 114 | "output_type": "stream", 115 | "text": [ 116 | "Today is 2023-10-18 13:39:42.340307\n" 117 | ] 118 | } 119 | ], 120 | "source": [ 121 | "print(\"Today is\", datetime.datetime.today())" 122 | ] 123 | }, 124 | { 125 | "cell_type": "markdown", 126 | "metadata": {}, 127 | "source": [ 128 | "## 2. Azure AI Services" 129 | ] 130 | }, 131 | { 132 | "cell_type": "code", 133 | "execution_count": 4, 134 | "metadata": {}, 135 | "outputs": [], 136 | "source": [ 137 | "load_dotenv(\"azure.env\")\n", 138 | "\n", 139 | "# Azure Computer Vision 4\n", 140 | "acv_key = os.getenv(\"acv_key\")\n", 141 | "acv_endpoint = os.getenv(\"acv_endpoint\")\n", 142 | "\n", 143 | "# Azure Cognitive Search\n", 144 | "acs_endpoint = os.getenv(\"acs_endpoint\")\n", 145 | "acs_key = os.getenv(\"acs_key\")" 146 | ] 147 | }, 148 | { 149 | "cell_type": "code", 150 | "execution_count": 5, 151 | "metadata": {}, 152 | "outputs": [], 153 | "source": [ 154 | "# Ensure that the azure endpoints should not finished a /\n", 155 | "if acv_endpoint.endswith(\"/\"):\n", 156 | " acv_endpoint = acv_endpoint[:-1]\n", 157 | "\n", 158 | "if acs_endpoint.endswith(\"/\"):\n", 159 | " acs_endpoint = acv_endpoint[:-1]" 160 | ] 161 | }, 162 | { 163 | "cell_type": "code", 164 | "execution_count": 6, 165 | "metadata": {}, 166 | "outputs": [], 167 | "source": [ 168 | "# Azure Cognitive Search index name to create\n", 169 | "index_name = \"azure-fashion-demo\"\n", 170 | "\n", 171 | "# Azure Cognitive Search api version\n", 172 | "api_version = \"2023-02-01-preview\"" 173 | ] 174 | }, 175 | { 176 | "cell_type": "markdown", 177 | "metadata": {}, 178 | "source": [ 179 | "## 3. Functions" 180 | ] 181 | }, 182 | { 183 | "cell_type": "code", 184 | "execution_count": 7, 185 | "metadata": {}, 186 | "outputs": [], 187 | "source": [ 188 | "session = requests.Session()\n", 189 | "\n", 190 | "\n", 191 | "def image_embedding(imagefile):\n", 192 | " \"\"\"\n", 193 | " Image embedding using Azure Computer Vision 4.0\n", 194 | " \"\"\"\n", 195 | " version = \"?api-version=\" + api_version + \"&modelVersion=latest\"\n", 196 | " vec_img_url = acv_endpoint + \"/computervision/retrieval:vectorizeImage\" + version\n", 197 | " headers = {\n", 198 | " \"Content-type\": \"application/octet-stream\",\n", 199 | " \"Ocp-Apim-Subscription-Key\": acv_key,\n", 200 | " }\n", 201 | "\n", 202 | " try:\n", 203 | " blob_service_client = BlobServiceClient.from_connection_string(\n", 204 | " blob_connection_string\n", 205 | " )\n", 206 | " container_client = blob_service_client.get_container_client(container_name)\n", 207 | "\n", 208 | " blob_client = container_client.get_blob_client(imagefile)\n", 209 | " stream = BytesIO()\n", 210 | " blob_data = blob_client.download_blob()\n", 211 | " blob_data.readinto(stream)\n", 212 | "\n", 213 | " stream.seek(0) # Reset stream position to the beginning\n", 214 | "\n", 215 | " response = session.post(vec_img_url, data=stream, headers=headers)\n", 216 | " response.raise_for_status() # Raise an exception if response is not 200\n", 217 | "\n", 218 | " image_emb = response.json()[\"vector\"]\n", 219 | " return image_emb\n", 220 | "\n", 221 | " except requests.exceptions.RequestException as e:\n", 222 | " print(f\"Request Exception: {e}\")\n", 223 | " except Exception as ex:\n", 224 | " print(f\"Error: {ex}\")\n", 225 | "\n", 226 | " return None" 227 | ] 228 | }, 229 | { 230 | "cell_type": "code", 231 | "execution_count": 16, 232 | "metadata": {}, 233 | "outputs": [], 234 | "source": [ 235 | "def index_status(index_name):\n", 236 | " \"\"\"\n", 237 | " Azure Cognitive Search index status\n", 238 | " \"\"\"\n", 239 | " print(\"Azure Cognitive Search Index:\", index_name, \"\\n\")\n", 240 | "\n", 241 | " headers = {\"Content-Type\": \"application/json\", \"api-key\": acs_key}\n", 242 | " params = {\"api-version\": \"2021-04-30-Preview\"}\n", 243 | " index_status = requests.get(\n", 244 | " acs_endpoint + \"/indexes/\" + index_name, headers=headers, params=params\n", 245 | " )\n", 246 | " try:\n", 247 | " print(json.dumps((index_status.json()), indent=5))\n", 248 | " except:\n", 249 | " print(\"Request failed\")\n", 250 | "\n", 251 | "\n", 252 | "def index_stats(index_name):\n", 253 | " \"\"\"\n", 254 | " Get statistics about Azure Cognitive Search index\n", 255 | " \"\"\"\n", 256 | " url = (\n", 257 | " acs_endpoint\n", 258 | " + \"/indexes/\"\n", 259 | " + index_name\n", 260 | " + \"/stats?api-version=2021-04-30-Preview\"\n", 261 | " )\n", 262 | " headers = {\n", 263 | " \"Content-Type\": \"application/json\",\n", 264 | " \"api-key\": acs_key,\n", 265 | " }\n", 266 | " response = requests.get(url, headers=headers)\n", 267 | " print(\"Azure Cognitive Search index status for:\", index_name, \"\\n\")\n", 268 | "\n", 269 | " if response.status_code == 200:\n", 270 | " res = response.json()\n", 271 | " print(json.dumps(res, indent=2))\n", 272 | " document_count = res[\"documentCount\"]\n", 273 | " storage_size = res[\"storageSize\"]\n", 274 | "\n", 275 | " else:\n", 276 | " print(\"Request failed with status code:\", response.status_code)\n", 277 | "\n", 278 | " return document_count, storage_size\n", 279 | "\n", 280 | "\n", 281 | "def acs_service_statistics(index_name):\n", 282 | " \"\"\"\n", 283 | " Azure Cognitive Search service statistics\n", 284 | " \"\"\"\n", 285 | " url = os.getenv(\"acs_endpoint\") + \"/servicestats?api-version=2021-04-30-Preview\"\n", 286 | " headers = {\n", 287 | " \"Content-Type\": \"application/json\",\n", 288 | " \"api-key\": os.getenv(\"acs_key\"),\n", 289 | " }\n", 290 | " response = requests.get(url, headers=headers)\n", 291 | " print(\"Azure Cognitive Search index status for:\", index_name, \"\\n\")\n", 292 | "\n", 293 | " if response.status_code == 200:\n", 294 | " res = response.json()\n", 295 | " print(json.dumps(res, indent=2))\n", 296 | "\n", 297 | " else:\n", 298 | " print(\"Request failed with status code:\", response.status_code)\n", 299 | " \n", 300 | " \n", 301 | "def delete_acs_document_by_id(document_id):\n", 302 | " \"\"\"\n", 303 | " Delete Azure Cognitive Search document\n", 304 | " \"\"\"\n", 305 | " # Search client\n", 306 | " search_client = SearchClient(acs_endpoint, index_name, AzureKeyCredential(acs_key))\n", 307 | "\n", 308 | " # Delete document by its id number\n", 309 | " print(f\"Document with id {document_id} will be deleted from index {index_name}\")\n", 310 | " document_to_delete = {\"idfile\": document_id} # Delete the entry based on the idfile\n", 311 | " search_client.delete_documents([document_to_delete])\n", 312 | "\n", 313 | " print(\"Done\")" 314 | ] 315 | }, 316 | { 317 | "cell_type": "markdown", 318 | "metadata": {}, 319 | "source": [ 320 | "## 4. Azure Cognitive search index informations" 321 | ] 322 | }, 323 | { 324 | "cell_type": "code", 325 | "execution_count": 9, 326 | "metadata": {}, 327 | "outputs": [ 328 | { 329 | "name": "stdout", 330 | "output_type": "stream", 331 | "text": [ 332 | "Azure Cognitive Search index status for: azure-fashion-demo \n", 333 | "\n", 334 | "{\n", 335 | " \"@odata.context\": \"https://azurecogsearcheastussr.search.windows.net/$metadata#Microsoft.Azure.Search.V2021_04_30_Preview.ServiceStatistics\",\n", 336 | " \"counters\": {\n", 337 | " \"documentCount\": {\n", 338 | " \"usage\": 33242,\n", 339 | " \"quota\": null\n", 340 | " },\n", 341 | " \"indexesCount\": {\n", 342 | " \"usage\": 21,\n", 343 | " \"quota\": 50\n", 344 | " },\n", 345 | " \"indexersCount\": {\n", 346 | " \"usage\": 1,\n", 347 | " \"quota\": 50\n", 348 | " },\n", 349 | " \"dataSourcesCount\": {\n", 350 | " \"usage\": 3,\n", 351 | " \"quota\": 50\n", 352 | " },\n", 353 | " \"storageSize\": {\n", 354 | " \"usage\": 1026951396,\n", 355 | " \"quota\": 26843545600\n", 356 | " },\n", 357 | " \"synonymMaps\": {\n", 358 | " \"usage\": 0,\n", 359 | " \"quota\": 5\n", 360 | " },\n", 361 | " \"skillsetCount\": {\n", 362 | " \"usage\": 0,\n", 363 | " \"quota\": 50\n", 364 | " },\n", 365 | " \"aliasesCount\": {\n", 366 | " \"usage\": 0,\n", 367 | " \"quota\": 100\n", 368 | " }\n", 369 | " },\n", 370 | " \"limits\": {\n", 371 | " \"maxFieldsPerIndex\": 3000,\n", 372 | " \"maxFieldNestingDepthPerIndex\": 10,\n", 373 | " \"maxComplexCollectionFieldsPerIndex\": 40,\n", 374 | " \"maxComplexObjectsInCollectionsPerDocument\": 3000\n", 375 | " }\n", 376 | "}\n" 377 | ] 378 | } 379 | ], 380 | "source": [ 381 | "acs_service_statistics(index_name)" 382 | ] 383 | }, 384 | { 385 | "cell_type": "code", 386 | "execution_count": 10, 387 | "metadata": {}, 388 | "outputs": [ 389 | { 390 | "name": "stdout", 391 | "output_type": "stream", 392 | "text": [ 393 | "Azure Cognitive Search Index: azure-fashion-demo \n", 394 | "\n", 395 | "{\n", 396 | " \"@odata.context\": \"https://azurecogsearcheastussr.search.windows.net/$metadata#indexes/$entity\",\n", 397 | " \"@odata.etag\": \"\\\"0x8DB79721D17ED54\\\"\",\n", 398 | " \"name\": \"azure-fashion-demo\",\n", 399 | " \"defaultScoringProfile\": null,\n", 400 | " \"fields\": [\n", 401 | " {\n", 402 | " \"name\": \"idfile\",\n", 403 | " \"type\": \"Edm.String\",\n", 404 | " \"searchable\": false,\n", 405 | " \"filterable\": false,\n", 406 | " \"retrievable\": true,\n", 407 | " \"sortable\": false,\n", 408 | " \"facetable\": false,\n", 409 | " \"key\": true,\n", 410 | " \"indexAnalyzer\": null,\n", 411 | " \"searchAnalyzer\": null,\n", 412 | " \"analyzer\": null,\n", 413 | " \"normalizer\": null,\n", 414 | " \"synonymMaps\": []\n", 415 | " },\n", 416 | " {\n", 417 | " \"name\": \"imagefile\",\n", 418 | " \"type\": \"Edm.String\",\n", 419 | " \"searchable\": true,\n", 420 | " \"filterable\": false,\n", 421 | " \"retrievable\": true,\n", 422 | " \"sortable\": false,\n", 423 | " \"facetable\": false,\n", 424 | " \"key\": false,\n", 425 | " \"indexAnalyzer\": null,\n", 426 | " \"searchAnalyzer\": null,\n", 427 | " \"analyzer\": null,\n", 428 | " \"normalizer\": null,\n", 429 | " \"synonymMaps\": []\n", 430 | " },\n", 431 | " {\n", 432 | " \"name\": \"imagevector\",\n", 433 | " \"type\": \"Collection(Edm.Single)\",\n", 434 | " \"searchable\": true,\n", 435 | " \"filterable\": false,\n", 436 | " \"retrievable\": true,\n", 437 | " \"sortable\": false,\n", 438 | " \"facetable\": false,\n", 439 | " \"key\": false,\n", 440 | " \"indexAnalyzer\": null,\n", 441 | " \"searchAnalyzer\": null,\n", 442 | " \"analyzer\": null,\n", 443 | " \"normalizer\": null,\n", 444 | " \"synonymMaps\": []\n", 445 | " }\n", 446 | " ],\n", 447 | " \"scoringProfiles\": [],\n", 448 | " \"corsOptions\": null,\n", 449 | " \"suggesters\": [],\n", 450 | " \"analyzers\": [],\n", 451 | " \"normalizers\": [],\n", 452 | " \"tokenizers\": [],\n", 453 | " \"tokenFilters\": [],\n", 454 | " \"charFilters\": [],\n", 455 | " \"encryptionKey\": null,\n", 456 | " \"similarity\": {\n", 457 | " \"@odata.type\": \"#Microsoft.Azure.Search.BM25Similarity\",\n", 458 | " \"k1\": null,\n", 459 | " \"b\": null\n", 460 | " },\n", 461 | " \"semantic\": {\n", 462 | " \"defaultConfiguration\": null,\n", 463 | " \"configurations\": [\n", 464 | " {\n", 465 | " \"name\": \"my-semantic-config\",\n", 466 | " \"prioritizedFields\": {\n", 467 | " \"titleField\": {\n", 468 | " \"fieldName\": \"idfile\"\n", 469 | " },\n", 470 | " \"prioritizedContentFields\": [],\n", 471 | " \"prioritizedKeywordsFields\": []\n", 472 | " }\n", 473 | " }\n", 474 | " ]\n", 475 | " }\n", 476 | "}\n" 477 | ] 478 | } 479 | ], 480 | "source": [ 481 | "index_status(index_name)" 482 | ] 483 | }, 484 | { 485 | "cell_type": "code", 486 | "execution_count": 17, 487 | "metadata": {}, 488 | "outputs": [ 489 | { 490 | "name": "stdout", 491 | "output_type": "stream", 492 | "text": [ 493 | "Azure Cognitive Search index status for: azure-fashion-demo \n", 494 | "\n", 495 | "{\n", 496 | " \"@odata.context\": \"https://azurecogsearcheastussr.search.windows.net/$metadata#Microsoft.Azure.Search.V2021_04_30_Preview.IndexStatistics\",\n", 497 | " \"documentCount\": 10219,\n", 498 | " \"storageSize\": 155597306\n", 499 | "}\n" 500 | ] 501 | } 502 | ], 503 | "source": [ 504 | "document_count, storage_size = index_stats(index_name)" 505 | ] 506 | }, 507 | { 508 | "cell_type": "code", 509 | "execution_count": 18, 510 | "metadata": {}, 511 | "outputs": [ 512 | { 513 | "name": "stdout", 514 | "output_type": "stream", 515 | "text": [ 516 | "Number of documents in the index = 10,219\n", 517 | "Size of the index = 148.39 MB\n" 518 | ] 519 | } 520 | ], 521 | "source": [ 522 | "print(\"Number of documents in the index =\", f\"{document_count:,}\")\n", 523 | "print(\"Size of the index =\", round(storage_size / (1024 * 1024), 2), \"MB\")" 524 | ] 525 | }, 526 | { 527 | "cell_type": "markdown", 528 | "metadata": {}, 529 | "source": [ 530 | "## 5. Quick search on a document" 531 | ] 532 | }, 533 | { 534 | "cell_type": "code", 535 | "execution_count": 19, 536 | "metadata": {}, 537 | "outputs": [], 538 | "source": [ 539 | "search_client = SearchClient(acs_endpoint, index_name, AzureKeyCredential(acs_key))" 540 | ] 541 | }, 542 | { 543 | "cell_type": "code", 544 | "execution_count": 22, 545 | "metadata": {}, 546 | "outputs": [ 547 | { 548 | "name": "stdout", 549 | "output_type": "stream", 550 | "text": [ 551 | "Id file: 3062\n", 552 | "Filename: fashion/0627214001.jpg\n" 553 | ] 554 | } 555 | ], 556 | "source": [ 557 | "text = \"0627214001\"\n", 558 | "response = search_client.search(search_text=text)\n", 559 | "\n", 560 | "for result in response:\n", 561 | " print(\"Id file:\", result[\"idfile\"])\n", 562 | " print(\"Filename:\", result[\"imagefile\"])" 563 | ] 564 | }, 565 | { 566 | "cell_type": "code", 567 | "execution_count": 23, 568 | "metadata": {}, 569 | "outputs": [ 570 | { 571 | "name": "stdout", 572 | "output_type": "stream", 573 | "text": [ 574 | "Let's query the index with text = 0628469001 \n", 575 | "\n", 576 | "Id file: 3267\n", 577 | "Filename: fashion/0628469001.jpg\n" 578 | ] 579 | } 580 | ], 581 | "source": [ 582 | "text = \"0628469001\"\n", 583 | "print(\"Let's query the index with text =\", text, \"\\n\")\n", 584 | "\n", 585 | "response = search_client.search(search_text=text)\n", 586 | "\n", 587 | "for result in response:\n", 588 | " print(\"Id file:\", result[\"idfile\"])\n", 589 | " print(\"Filename:\", result[\"imagefile\"])" 590 | ] 591 | }, 592 | { 593 | "cell_type": "markdown", 594 | "metadata": {}, 595 | "source": [ 596 | "## 6. Let's delete some documents from the index" 597 | ] 598 | }, 599 | { 600 | "cell_type": "code", 601 | "execution_count": 24, 602 | "metadata": {}, 603 | "outputs": [ 604 | { 605 | "name": "stdout", 606 | "output_type": "stream", 607 | "text": [ 608 | "Azure Cognitive Search index status for: azure-fashion-demo \n", 609 | "\n", 610 | "{\n", 611 | " \"@odata.context\": \"https://azurecogsearcheastussr.search.windows.net/$metadata#Microsoft.Azure.Search.V2021_04_30_Preview.IndexStatistics\",\n", 612 | " \"documentCount\": 10221,\n", 613 | " \"storageSize\": 155633480\n", 614 | "}\n" 615 | ] 616 | } 617 | ], 618 | "source": [ 619 | "document_count, storage_size = index_stats(index_name)" 620 | ] 621 | }, 622 | { 623 | "cell_type": "code", 624 | "execution_count": 25, 625 | "metadata": {}, 626 | "outputs": [ 627 | { 628 | "name": "stdout", 629 | "output_type": "stream", 630 | "text": [ 631 | "Document with id 3062 will be deleted from index azure-fashion-demo\n", 632 | "Done\n" 633 | ] 634 | } 635 | ], 636 | "source": [ 637 | "document_id = \"3062\"\n", 638 | "delete_acs_document_by_id(document_id)" 639 | ] 640 | }, 641 | { 642 | "cell_type": "code", 643 | "execution_count": 29, 644 | "metadata": {}, 645 | "outputs": [ 646 | { 647 | "name": "stdout", 648 | "output_type": "stream", 649 | "text": [ 650 | "Azure Cognitive Search index status for: azure-fashion-demo \n", 651 | "\n", 652 | "{\n", 653 | " \"@odata.context\": \"https://azurecogsearcheastussr.search.windows.net/$metadata#Microsoft.Azure.Search.V2021_04_30_Preview.IndexStatistics\",\n", 654 | " \"documentCount\": 10220,\n", 655 | " \"storageSize\": 155618254\n", 656 | "}\n" 657 | ] 658 | } 659 | ], 660 | "source": [ 661 | "document_count, storage_size = index_stats(index_name)" 662 | ] 663 | }, 664 | { 665 | "cell_type": "code", 666 | "execution_count": 30, 667 | "metadata": {}, 668 | "outputs": [ 669 | { 670 | "name": "stdout", 671 | "output_type": "stream", 672 | "text": [ 673 | "Document with id 3267 will be deleted from index azure-fashion-demo\n", 674 | "Done\n" 675 | ] 676 | } 677 | ], 678 | "source": [ 679 | "document_id = \"3267\"\n", 680 | "delete_acs_document_by_id(document_id)" 681 | ] 682 | }, 683 | { 684 | "cell_type": "code", 685 | "execution_count": 38, 686 | "metadata": {}, 687 | "outputs": [ 688 | { 689 | "name": "stdout", 690 | "output_type": "stream", 691 | "text": [ 692 | "Azure Cognitive Search index status for: azure-fashion-demo \n", 693 | "\n", 694 | "{\n", 695 | " \"@odata.context\": \"https://azurecogsearcheastussr.search.windows.net/$metadata#Microsoft.Azure.Search.V2021_04_30_Preview.IndexStatistics\",\n", 696 | " \"documentCount\": 10219,\n", 697 | " \"storageSize\": 155603130\n", 698 | "}\n" 699 | ] 700 | } 701 | ], 702 | "source": [ 703 | "document_count, storage_size = index_stats(index_name)" 704 | ] 705 | }, 706 | { 707 | "cell_type": "markdown", 708 | "metadata": {}, 709 | "source": [ 710 | "> We have deleted 2 documents from the index as we can see" 711 | ] 712 | }, 713 | { 714 | "cell_type": "markdown", 715 | "metadata": {}, 716 | "source": [ 717 | "## 7. Let's add some new images to the index\n", 718 | "\n", 719 | "Let's use a new blob to use that contains some new images to add" 720 | ] 721 | }, 722 | { 723 | "cell_type": "code", 724 | "execution_count": 39, 725 | "metadata": {}, 726 | "outputs": [], 727 | "source": [ 728 | "# Azure storage account\n", 729 | "blob_connection_string = \"tobereplaced\"\n", 730 | "container_name = \"tobereplaced\"" 731 | ] 732 | }, 733 | { 734 | "cell_type": "code", 735 | "execution_count": 40, 736 | "metadata": {}, 737 | "outputs": [ 738 | { 739 | "name": "stdout", 740 | "output_type": "stream", 741 | "text": [ 742 | "URL of the first blob: https://azurestorageaccountsr.blob.core.windows.net/fashion-images-new/shirt1.jpg\n" 743 | ] 744 | } 745 | ], 746 | "source": [ 747 | "# Connect to Blob Storage\n", 748 | "blob_service_client = BlobServiceClient.from_connection_string(blob_connection_string)\n", 749 | "container_client = blob_service_client.get_container_client(container_name)\n", 750 | "blobs = container_client.list_blobs()\n", 751 | "\n", 752 | "first_blob = next(blobs)\n", 753 | "blob_url = container_client.get_blob_client(first_blob).url\n", 754 | "print(f\"URL of the first blob: {blob_url}\")" 755 | ] 756 | }, 757 | { 758 | "cell_type": "code", 759 | "execution_count": 41, 760 | "metadata": {}, 761 | "outputs": [ 762 | { 763 | "name": "stdout", 764 | "output_type": "stream", 765 | "text": [ 766 | "Done. Data source 'azure-fashion-demo-blob' has been created or updated.\n" 767 | ] 768 | } 769 | ], 770 | "source": [ 771 | "# Create a data source\n", 772 | "ds_client = SearchIndexerClient(acs_endpoint, AzureKeyCredential(acs_key))\n", 773 | "container = SearchIndexerDataContainer(name=container_name)\n", 774 | "data_source_connection = SearchIndexerDataSourceConnection(\n", 775 | " name=f\"{index_name}-blob\",\n", 776 | " type=\"azureblob\",\n", 777 | " connection_string=blob_connection_string,\n", 778 | " container=container,\n", 779 | ")\n", 780 | "data_source = ds_client.create_or_update_data_source_connection(data_source_connection)\n", 781 | "\n", 782 | "print(f\"Done. Data source '{data_source.name}' has been created or updated.\")" 783 | ] 784 | }, 785 | { 786 | "cell_type": "code", 787 | "execution_count": 42, 788 | "metadata": {}, 789 | "outputs": [ 790 | { 791 | "name": "stdout", 792 | "output_type": "stream", 793 | "text": [ 794 | "Total number of images = 2 in blob: fashion-images-new\n" 795 | ] 796 | } 797 | ], 798 | "source": [ 799 | "blob_service_client = BlobServiceClient.from_connection_string(blob_connection_string)\n", 800 | "container_client = blob_service_client.get_container_client(container_name)\n", 801 | "number_images = len(list(container_client.list_blobs()))\n", 802 | "\n", 803 | "print(\"Total number of images =\", number_images, \"in blob:\", container_name)" 804 | ] 805 | }, 806 | { 807 | "cell_type": "code", 808 | "execution_count": 43, 809 | "metadata": {}, 810 | "outputs": [], 811 | "source": [ 812 | "EMBEDDINGS_DIR = \"embeddings\"\n", 813 | "\n", 814 | "os.makedirs(EMBEDDINGS_DIR, exist_ok=True)" 815 | ] 816 | }, 817 | { 818 | "cell_type": "code", 819 | "execution_count": 44, 820 | "metadata": {}, 821 | "outputs": [], 822 | "source": [ 823 | "list_of_images = container_client.list_blobs()\n", 824 | "\n", 825 | "new_images_list = []\n", 826 | "\n", 827 | "for image in list_of_images:\n", 828 | " imagefile = image[\"name\"]\n", 829 | " new_images_list.append(imagefile)" 830 | ] 831 | }, 832 | { 833 | "cell_type": "code", 834 | "execution_count": 45, 835 | "metadata": {}, 836 | "outputs": [ 837 | { 838 | "data": { 839 | "text/plain": [ 840 | "['shirt1.jpg', 'shirt2.jpg']" 841 | ] 842 | }, 843 | "execution_count": 45, 844 | "metadata": {}, 845 | "output_type": "execute_result" 846 | } 847 | ], 848 | "source": [ 849 | "new_images_list" 850 | ] 851 | }, 852 | { 853 | "cell_type": "code", 854 | "execution_count": 61, 855 | "metadata": {}, 856 | "outputs": [ 857 | { 858 | "name": "stdout", 859 | "output_type": "stream", 860 | "text": [ 861 | "-rwxrwxrwx 1 root root 98 Oct 18 13:55 embeddings/list_of_images_new.json\r\n" 862 | ] 863 | } 864 | ], 865 | "source": [ 866 | "startindex = 100003\n", 867 | "\n", 868 | "data = [\n", 869 | " {\"idfile\": str(i + startindex +1), \"imagefile\": image} for i, image in enumerate(new_images_list)\n", 870 | "]\n", 871 | "\n", 872 | "with open(os.path.join(EMBEDDINGS_DIR, \"list_of_images_new.json\"), \"w\") as f:\n", 873 | " json.dump(data, f)\n", 874 | " \n", 875 | "!ls $EMBEDDINGS_DIR/list_of_images_new.json -lh" 876 | ] 877 | }, 878 | { 879 | "cell_type": "code", 880 | "execution_count": 62, 881 | "metadata": {}, 882 | "outputs": [ 883 | { 884 | "data": { 885 | "text/plain": [ 886 | "[{'idfile': '100004', 'imagefile': 'shirt1.jpg'},\n", 887 | " {'idfile': '100005', 'imagefile': 'shirt2.jpg'}]" 888 | ] 889 | }, 890 | "execution_count": 62, 891 | "metadata": {}, 892 | "output_type": "execute_result" 893 | } 894 | ], 895 | "source": [ 896 | "data" 897 | ] 898 | }, 899 | { 900 | "cell_type": "code", 901 | "execution_count": 63, 902 | "metadata": {}, 903 | "outputs": [ 904 | { 905 | "name": "stdout", 906 | "output_type": "stream", 907 | "text": [ 908 | "Running the image files embeddings...\n", 909 | "Total number of images to embed = 2 \n", 910 | "\n", 911 | "\n", 912 | "Done\n", 913 | "\n", 914 | "Elapsed time: 00:00:01.166330\n", 915 | "Time per image = 0.58317 seconds\n" 916 | ] 917 | } 918 | ], 919 | "source": [ 920 | "batch_size = 1\n", 921 | "\n", 922 | "start = time.time()\n", 923 | "print(\"Running the image files embeddings...\")\n", 924 | "print(\"Total number of images to embed =\", len(new_images_list), \"\\n\")\n", 925 | "\n", 926 | "with open(\n", 927 | " os.path.join(EMBEDDINGS_DIR, \"list_of_images_new.json\"), \"r\", encoding=\"utf-8\"\n", 928 | ") as file:\n", 929 | " input_data = json.load(file)\n", 930 | "\n", 931 | "image_count = len(input_data)\n", 932 | "processed_count = 0\n", 933 | "\n", 934 | "for batch_start in range(0, image_count, batch_size):\n", 935 | " batch_end = min(batch_start + batch_size, image_count)\n", 936 | " batch_data = input_data[batch_start:batch_end]\n", 937 | "\n", 938 | " for idx, item in enumerate(batch_data, start=batch_start + 1):\n", 939 | " imgindex = item[\"idfile\"]\n", 940 | " imgfile = item[\"imagefile\"]\n", 941 | " item[\"imagevector\"] = image_embedding(imgfile)\n", 942 | "\n", 943 | " if idx % batch_size == 1:\n", 944 | " pctdone = round(idx / image_count * 100)\n", 945 | " dt = datetime.datetime.today().strftime(\"%d-%b-%Y %H:%M:%S\")\n", 946 | " print(\n", 947 | " dt,\n", 948 | " f\"Number of processed image files = {idx:06} of {image_count:06} | Done: {pctdone}%\",\n", 949 | " )\n", 950 | "\n", 951 | " processed_count += len(batch_data)\n", 952 | "\n", 953 | "elapsed = time.time() - start\n", 954 | "print(\"\\nDone\")\n", 955 | "print(\n", 956 | " \"\\nElapsed time: \"\n", 957 | " + time.strftime(\n", 958 | " \"%H:%M:%S.{}\".format(str(elapsed % 1)[2:])[:15], time.gmtime(elapsed)\n", 959 | " )\n", 960 | ")\n", 961 | "print(\"Time per image =\", round(elapsed / processed_count, 5), \"seconds\")" 962 | ] 963 | }, 964 | { 965 | "cell_type": "code", 966 | "execution_count": 64, 967 | "metadata": {}, 968 | "outputs": [ 969 | { 970 | "name": "stdout", 971 | "output_type": "stream", 972 | "text": [ 973 | "Saving the results into a json file...\n", 974 | "Done. Elapsed time: 0.05 secs\n" 975 | ] 976 | } 977 | ], 978 | "source": [ 979 | "# Save embeddings to documents.json file\n", 980 | "start = time.time()\n", 981 | "\n", 982 | "print(\"Saving the results into a json file...\")\n", 983 | "with open(os.path.join(EMBEDDINGS_DIR, \"documents_new.json\"), \"w\") as f:\n", 984 | " json.dump(input_data, f)\n", 985 | "\n", 986 | "print(\"Done. Elapsed time:\", round(time.time() - start, 2), \"secs\")" 987 | ] 988 | }, 989 | { 990 | "cell_type": "code", 991 | "execution_count": 65, 992 | "metadata": {}, 993 | "outputs": [ 994 | { 995 | "name": "stdout", 996 | "output_type": "stream", 997 | "text": [ 998 | "Size of the documents to load = 2\n" 999 | ] 1000 | } 1001 | ], 1002 | "source": [ 1003 | "with open(os.path.join(EMBEDDINGS_DIR, \"documents_new.json\"), \"r\") as file:\n", 1004 | " documents = json.load(file)\n", 1005 | "\n", 1006 | "print(\"Size of the documents to load =\", len(documents))" 1007 | ] 1008 | }, 1009 | { 1010 | "cell_type": "code", 1011 | "execution_count": 66, 1012 | "metadata": {}, 1013 | "outputs": [], 1014 | "source": [ 1015 | "def loading_documents(documents):\n", 1016 | " \"\"\"\n", 1017 | " Loading documents into the Azure Cognitive Search index\n", 1018 | " \"\"\"\n", 1019 | " # Upload some documents to the index\n", 1020 | " print(\"Uploading the documents into the index\", index_name, \"...\")\n", 1021 | "\n", 1022 | " # Setting the Azure Cognitive Search client\n", 1023 | " search_client = SearchClient(\n", 1024 | " endpoint=acs_endpoint,\n", 1025 | " index_name=index_name,\n", 1026 | " credential=AzureKeyCredential(acs_key),\n", 1027 | " )\n", 1028 | " response = search_client.upload_documents(documents)\n", 1029 | " print(\n", 1030 | " f\"\\nDone. Uploaded {len(documents)} documents into the Azure Cognitive Search index.\\n\"\n", 1031 | " )\n", 1032 | " return len(documents)" 1033 | ] 1034 | }, 1035 | { 1036 | "cell_type": "code", 1037 | "execution_count": 67, 1038 | "metadata": {}, 1039 | "outputs": [ 1040 | { 1041 | "name": "stdout", 1042 | "output_type": "stream", 1043 | "text": [ 1044 | "Uploading the documents into the index azure-fashion-demo ...\n", 1045 | "\n", 1046 | "Done. Uploaded 2 documents into the Azure Cognitive Search index.\n", 1047 | "\n" 1048 | ] 1049 | }, 1050 | { 1051 | "data": { 1052 | "text/plain": [ 1053 | "2" 1054 | ] 1055 | }, 1056 | "execution_count": 67, 1057 | "metadata": {}, 1058 | "output_type": "execute_result" 1059 | } 1060 | ], 1061 | "source": [ 1062 | "loading_documents(documents)" 1063 | ] 1064 | }, 1065 | { 1066 | "cell_type": "code", 1067 | "execution_count": 73, 1068 | "metadata": {}, 1069 | "outputs": [ 1070 | { 1071 | "name": "stdout", 1072 | "output_type": "stream", 1073 | "text": [ 1074 | "Azure Cognitive Search index status for: azure-fashion-demo \n", 1075 | "\n", 1076 | "{\n", 1077 | " \"@odata.context\": \"https://azurecogsearcheastussr.search.windows.net/$metadata#Microsoft.Azure.Search.V2021_04_30_Preview.IndexStatistics\",\n", 1078 | " \"documentCount\": 10221,\n", 1079 | " \"storageSize\": 155639272\n", 1080 | "}\n" 1081 | ] 1082 | } 1083 | ], 1084 | "source": [ 1085 | "document_count, storage_size = index_stats(index_name)" 1086 | ] 1087 | }, 1088 | { 1089 | "cell_type": "markdown", 1090 | "metadata": {}, 1091 | "source": [ 1092 | "> This is the new size of the index" 1093 | ] 1094 | }, 1095 | { 1096 | "cell_type": "markdown", 1097 | "metadata": {}, 1098 | "source": [ 1099 | "## 8. Post processing" 1100 | ] 1101 | }, 1102 | { 1103 | "cell_type": "code", 1104 | "execution_count": 56, 1105 | "metadata": {}, 1106 | "outputs": [ 1107 | { 1108 | "name": "stdout", 1109 | "output_type": "stream", 1110 | "text": [ 1111 | "Azure Cognitive Search Index: azure-fashion-demo \n", 1112 | "\n", 1113 | "{\n", 1114 | " \"@odata.context\": \"https://azurecogsearcheastussr.search.windows.net/$metadata#indexes/$entity\",\n", 1115 | " \"@odata.etag\": \"\\\"0x8DB79721D17ED54\\\"\",\n", 1116 | " \"name\": \"azure-fashion-demo\",\n", 1117 | " \"defaultScoringProfile\": null,\n", 1118 | " \"fields\": [\n", 1119 | " {\n", 1120 | " \"name\": \"idfile\",\n", 1121 | " \"type\": \"Edm.String\",\n", 1122 | " \"searchable\": false,\n", 1123 | " \"filterable\": false,\n", 1124 | " \"retrievable\": true,\n", 1125 | " \"sortable\": false,\n", 1126 | " \"facetable\": false,\n", 1127 | " \"key\": true,\n", 1128 | " \"indexAnalyzer\": null,\n", 1129 | " \"searchAnalyzer\": null,\n", 1130 | " \"analyzer\": null,\n", 1131 | " \"normalizer\": null,\n", 1132 | " \"synonymMaps\": []\n", 1133 | " },\n", 1134 | " {\n", 1135 | " \"name\": \"imagefile\",\n", 1136 | " \"type\": \"Edm.String\",\n", 1137 | " \"searchable\": true,\n", 1138 | " \"filterable\": false,\n", 1139 | " \"retrievable\": true,\n", 1140 | " \"sortable\": false,\n", 1141 | " \"facetable\": false,\n", 1142 | " \"key\": false,\n", 1143 | " \"indexAnalyzer\": null,\n", 1144 | " \"searchAnalyzer\": null,\n", 1145 | " \"analyzer\": null,\n", 1146 | " \"normalizer\": null,\n", 1147 | " \"synonymMaps\": []\n", 1148 | " },\n", 1149 | " {\n", 1150 | " \"name\": \"imagevector\",\n", 1151 | " \"type\": \"Collection(Edm.Single)\",\n", 1152 | " \"searchable\": true,\n", 1153 | " \"filterable\": false,\n", 1154 | " \"retrievable\": true,\n", 1155 | " \"sortable\": false,\n", 1156 | " \"facetable\": false,\n", 1157 | " \"key\": false,\n", 1158 | " \"indexAnalyzer\": null,\n", 1159 | " \"searchAnalyzer\": null,\n", 1160 | " \"analyzer\": null,\n", 1161 | " \"normalizer\": null,\n", 1162 | " \"synonymMaps\": []\n", 1163 | " }\n", 1164 | " ],\n", 1165 | " \"scoringProfiles\": [],\n", 1166 | " \"corsOptions\": null,\n", 1167 | " \"suggesters\": [],\n", 1168 | " \"analyzers\": [],\n", 1169 | " \"normalizers\": [],\n", 1170 | " \"tokenizers\": [],\n", 1171 | " \"tokenFilters\": [],\n", 1172 | " \"charFilters\": [],\n", 1173 | " \"encryptionKey\": null,\n", 1174 | " \"similarity\": {\n", 1175 | " \"@odata.type\": \"#Microsoft.Azure.Search.BM25Similarity\",\n", 1176 | " \"k1\": null,\n", 1177 | " \"b\": null\n", 1178 | " },\n", 1179 | " \"semantic\": {\n", 1180 | " \"defaultConfiguration\": null,\n", 1181 | " \"configurations\": [\n", 1182 | " {\n", 1183 | " \"name\": \"my-semantic-config\",\n", 1184 | " \"prioritizedFields\": {\n", 1185 | " \"titleField\": {\n", 1186 | " \"fieldName\": \"idfile\"\n", 1187 | " },\n", 1188 | " \"prioritizedContentFields\": [],\n", 1189 | " \"prioritizedKeywordsFields\": []\n", 1190 | " }\n", 1191 | " }\n", 1192 | " ]\n", 1193 | " }\n", 1194 | "}\n" 1195 | ] 1196 | } 1197 | ], 1198 | "source": [ 1199 | "index_status(index_name)" 1200 | ] 1201 | }, 1202 | { 1203 | "cell_type": "code", 1204 | "execution_count": null, 1205 | "metadata": {}, 1206 | "outputs": [], 1207 | "source": [ 1208 | "#delete_index(index_name)" 1209 | ] 1210 | }, 1211 | { 1212 | "cell_type": "code", 1213 | "execution_count": null, 1214 | "metadata": {}, 1215 | "outputs": [], 1216 | "source": [] 1217 | } 1218 | ], 1219 | "metadata": { 1220 | "kernelspec": { 1221 | "display_name": "Python 3 (ipykernel)", 1222 | "language": "python", 1223 | "name": "python3" 1224 | }, 1225 | "language_info": { 1226 | "codemirror_mode": { 1227 | "name": "ipython", 1228 | "version": 3 1229 | }, 1230 | "file_extension": ".py", 1231 | "mimetype": "text/x-python", 1232 | "name": "python", 1233 | "nbconvert_exporter": "python", 1234 | "pygments_lexer": "ipython3", 1235 | "version": "3.8.5" 1236 | } 1237 | }, 1238 | "nbformat": 4, 1239 | "nbformat_minor": 2 1240 | } 1241 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Visual search with Azure Computer Vision and Azure Cognitive Search 2 | 3 | 4 | 5 | We will use vectors embeddings generation using **Azure Computer Vision 4** and **Azure Cognitive Search** and its new vectors support capabilities to build a visual search application. 6 | 7 | 8 | ## Visual search with vector embeddings 9 | **Vector embeddings** are a way of representing content such as text or images as vectors of real numbers in a high-dimensional space. These embeddings are often learned from large amounts of textual and visual data using machine learning algorithms like neural networks. Each dimension of the vector corresponds to a different feature or attribute of the content, such as its semantic meaning, syntactic role, or context in which it commonly appears. By representing content as vectors, we can perform mathematical operations on them to compare their similarity or use them as inputs to machine learning models. 10 | 11 | **Azure Cognitive Search** today doesn't provide a way to vectorize documents and queries, leaving it up to you to pick the best embedding model for your data. 12 | In this example we will use **Azure Computer Vision** to generate the embeddings. 13 | 14 | 15 | 16 | ## Notebooks 17 | 18 | - Introduction to vectors embeddings with Azure Computer Vision 4:
19 | 1. Vectors embeddings with Azure Computer Vision 20 | - Creation of a fashion visual search demo using Azure Computer Vision and Azure Cognitive search:
21 | 2. Visual search - Azure Cognitive Search index creation 22 | - Some visual search examples:
23 | 3. Visual search - examples 24 | - A gradio webapp for the search:
25 | 4. Visual search - webapp 26 | - UMAP analysis on the vectors embeddings:
27 | 5. Visual Search - UMAP 28 | - Visual search with Azure Open AI, Azure Speech Services and Azure Cognitive Search:
29 | 6. Visual search - Azure Open AI 30 | 31 | Note: You need to write your Azure AI credentials in the **azure.env** file: azure.env 32 | 33 | ## Demo video 34 | YouTube demo video 35 | 36 | 37 | ## Some examples of a prompt search 38 | 39 | 40 | 41 | 42 | ## Some examples of an image search 43 | 44 | 45 | 46 | 47 | ## Steps 48 | 1. Connect to a blob storage where your catalog images are 49 | 2. Use Azure Computer Vision 4 to embed all these images 50 | 3. Create an Azure Cognitive search vector store index 51 | 4. Upload the embeddings into an Azure Cognitive Search index 52 | 5. Do some visual search using a prompt or an image 53 | 54 | ## Business applications 55 | - Digital asset management: Image retrieval can be used to manage large collections of digital images, such as in museums, archives, or online galleries. Users can search for images based on visual features and retrieve the images that match their criteria. 56 | - Medical image retrieval: Image retrieval can be used in medical imaging to search for images based on their diagnostic features or disease patterns. This can help doctors or researchers to identify similar cases or track disease progression. 57 | - Security and surveillance: Image retrieval can be used in security and surveillance systems to search for images based on specific features or patterns, such as in, people & object tracking, or threat detection. 58 | - Forensic image retrieval: Image retrieval can be used in forensic investigations to search for images based on their visual content or metadata, such as in cases of cyber-crime. 59 | - E-commerce: Image retrieval can be used in online shopping applications to search for similar products based on their features or descriptions or provide recommendations based on previous purchases. 60 | - Fashion and design: Image retrieval can be used in fashion and design to search for images based on their visual features, such as color, pattern, or texture. This can help designers or retailers to identify similar products or trends. 61 | 62 | ## To learn more 63 | - https://learn.microsoft.com/en-us/azure/cognitive-services/computer-vision/concept-image-retrieval 64 | - https://learn.microsoft.com/en-us/azure/search/search-what-is-azure-search 65 | - https://techcommunity.microsoft.com/t5/azure-ai-services-blog/announcing-vector-search-in-azure-cognitive-search-public/ba-p/3872868 66 | 67 | ## Fashion images 68 | We used some samples fashion images that are available here: 69 | https://www.kaggle.com/datasets/paramaggarwal/fashion-product-images-dataset 70 | 71 |
72 |
73 | 22-June-2023
74 | Updated 30-June-2023
75 |
76 | Serge Retkowsky | serge.retkowsky@microsoft.com | https://www.linkedin.com/in/serger/ 77 | -------------------------------------------------------------------------------- /acs1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/retkowsky/visual-search-azureAI/65cc91881486532e1bc48467076d24a2c8e1457e/acs1.png -------------------------------------------------------------------------------- /azure.env: -------------------------------------------------------------------------------- 1 | # Azure Cognitive Search 2 | acs_endpoint = "Your Azure Cognitive Search endpoint" 3 | acs_key = "Your Azure Cognitive Search key" 4 | 5 | # Azure Computer Vision 4 Florence for the vectors embeddings generation 6 | acv_endpoint = "Your Azure Computer Vision endpoint" 7 | acv_key = "Your Azure Computer Vision key" 8 | 9 | # Azure Storage Account for the catalog images to analyze 10 | container_name = "Your container name" 11 | blob_connection_string = "Your connection string" 12 | 13 | # Azure Open AI 14 | openai.api_type = "azure" 15 | openai.api_base = "Your Azure Open AI endpoint" 16 | openai.api_version = "2023-03-15-preview" 17 | openai.api_key = "Your Azure Open AI key" 18 | 19 | # Azure Speech Services 20 | asps_key = "Your Azure Speech Services key" 21 | asps_region = "Your Azure Speech Services region" 22 | 23 | -------------------------------------------------------------------------------- /demovideo.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/retkowsky/visual-search-azureAI/65cc91881486532e1bc48467076d24a2c8e1457e/demovideo.jpg -------------------------------------------------------------------------------- /images/car1.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/retkowsky/visual-search-azureAI/65cc91881486532e1bc48467076d24a2c8e1457e/images/car1.jpg -------------------------------------------------------------------------------- /images/car2.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/retkowsky/visual-search-azureAI/65cc91881486532e1bc48467076d24a2c8e1457e/images/car2.jpg -------------------------------------------------------------------------------- /images/car3.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/retkowsky/visual-search-azureAI/65cc91881486532e1bc48467076d24a2c8e1457e/images/car3.jpg -------------------------------------------------------------------------------- /images/car4.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/retkowsky/visual-search-azureAI/65cc91881486532e1bc48467076d24a2c8e1457e/images/car4.jpg -------------------------------------------------------------------------------- /images/car5.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/retkowsky/visual-search-azureAI/65cc91881486532e1bc48467076d24a2c8e1457e/images/car5.jpg -------------------------------------------------------------------------------- /images/car6.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/retkowsky/visual-search-azureAI/65cc91881486532e1bc48467076d24a2c8e1457e/images/car6.jpg -------------------------------------------------------------------------------- /images/car7.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/retkowsky/visual-search-azureAI/65cc91881486532e1bc48467076d24a2c8e1457e/images/car7.jpg -------------------------------------------------------------------------------- /images/image1.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/retkowsky/visual-search-azureAI/65cc91881486532e1bc48467076d24a2c8e1457e/images/image1.jpg -------------------------------------------------------------------------------- /images/image2.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/retkowsky/visual-search-azureAI/65cc91881486532e1bc48467076d24a2c8e1457e/images/image2.jpg -------------------------------------------------------------------------------- /images/image3.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/retkowsky/visual-search-azureAI/65cc91881486532e1bc48467076d24a2c8e1457e/images/image3.jpg -------------------------------------------------------------------------------- /images/image4.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/retkowsky/visual-search-azureAI/65cc91881486532e1bc48467076d24a2c8e1457e/images/image4.jpg -------------------------------------------------------------------------------- /images/image5.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/retkowsky/visual-search-azureAI/65cc91881486532e1bc48467076d24a2c8e1457e/images/image5.jpg -------------------------------------------------------------------------------- /images/image6.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/retkowsky/visual-search-azureAI/65cc91881486532e1bc48467076d24a2c8e1457e/images/image6.jpg -------------------------------------------------------------------------------- /images/image7.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/retkowsky/visual-search-azureAI/65cc91881486532e1bc48467076d24a2c8e1457e/images/image7.jpg -------------------------------------------------------------------------------- /images/image8.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/retkowsky/visual-search-azureAI/65cc91881486532e1bc48467076d24a2c8e1457e/images/image8.jpg -------------------------------------------------------------------------------- /img/image_search (1).jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/retkowsky/visual-search-azureAI/65cc91881486532e1bc48467076d24a2c8e1457e/img/image_search (1).jpg -------------------------------------------------------------------------------- /img/image_search (2).jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/retkowsky/visual-search-azureAI/65cc91881486532e1bc48467076d24a2c8e1457e/img/image_search (2).jpg -------------------------------------------------------------------------------- /img/image_search (3).jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/retkowsky/visual-search-azureAI/65cc91881486532e1bc48467076d24a2c8e1457e/img/image_search (3).jpg -------------------------------------------------------------------------------- /img/prompt search (1).jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/retkowsky/visual-search-azureAI/65cc91881486532e1bc48467076d24a2c8e1457e/img/prompt search (1).jpg -------------------------------------------------------------------------------- /img/prompt search (2).jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/retkowsky/visual-search-azureAI/65cc91881486532e1bc48467076d24a2c8e1457e/img/prompt search (2).jpg -------------------------------------------------------------------------------- /img/prompt search (3).jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/retkowsky/visual-search-azureAI/65cc91881486532e1bc48467076d24a2c8e1457e/img/prompt search (3).jpg -------------------------------------------------------------------------------- /logo.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/retkowsky/visual-search-azureAI/65cc91881486532e1bc48467076d24a2c8e1457e/logo.jpg -------------------------------------------------------------------------------- /screenshots.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/retkowsky/visual-search-azureAI/65cc91881486532e1bc48467076d24a2c8e1457e/screenshots.gif -------------------------------------------------------------------------------- /styles.xlsx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/retkowsky/visual-search-azureAI/65cc91881486532e1bc48467076d24a2c8e1457e/styles.xlsx --------------------------------------------------------------------------------