├── .github └── ISSUE_TEMPLATE │ ├── bug_report.md │ └── feature_request.md ├── .gitignore ├── CODEOWNERS ├── CODE_OF_CONDUCT.md ├── LICENSE ├── README.md ├── SECURITY.md ├── SUPPORT.md ├── ai-services ├── ai-landing-zone │ └── README.md ├── doc-intelligence-in-a-box │ └── README.md └── gpt-video-analysis-in-a-box │ └── README.md ├── edge-ai ├── README.MD ├── aml-edge-in-a-box │ └── readme.md └── customvision-edge-in-a-box │ └── readme.md ├── gen-ai ├── Assistants │ ├── README.md │ ├── bot-in-a-box │ │ └── README.md │ └── notebooks │ │ ├── .env-sample │ │ ├── README.md │ │ ├── autogen │ │ ├── .gitignore │ │ ├── README.md │ │ ├── data │ │ │ └── .gitkeep │ │ └── gpt_assistant_agent.ipynb │ │ ├── failed_banks │ │ ├── README.md │ │ ├── assistant-failed_banks.ipynb │ │ └── data │ │ │ └── failed_banks.csv │ │ ├── file_search │ │ ├── .gitignore │ │ ├── README.md │ │ ├── assistant-file-search.ipynb │ │ └── data │ │ │ └── .gitkeep │ │ ├── function_calling │ │ ├── README.md │ │ └── assistants_function_calling_with_bing_search.ipynb │ │ ├── math_tutor │ │ ├── README.md │ │ └── assistant-math_tutor.ipynb │ │ ├── personal_finance │ │ ├── README.md │ │ ├── assistant-personal_finance.ipynb │ │ └── data │ │ │ └── portfolio.csv │ │ ├── readme_assets │ │ ├── assistantsapi-flow-overview.svg │ │ └── banner-assistants-api-in-a-box.png │ │ ├── requirements.txt │ │ ├── sales_assistant │ │ ├── .gitignore │ │ ├── README.md │ │ ├── assistant_sales.ipynb │ │ └── data │ │ │ ├── customers.csv │ │ │ ├── inventory.csv │ │ │ ├── orders.csv │ │ │ └── sellers.csv │ │ ├── sales_copilot │ │ ├── AgentProxy.py │ │ ├── AgentRegistration.py │ │ ├── AgentSettings.py │ │ ├── ArgumentException.py │ │ ├── AssistantAgent.py │ │ ├── README.md │ │ ├── data │ │ │ ├── customers.csv │ │ │ ├── inventory.csv │ │ │ ├── orders.csv │ │ │ └── sellers.csv │ │ ├── images │ │ │ └── proxy-two-agent-diagram.png │ │ ├── sales_agent.py │ │ ├── trading_agent.py │ │ └── two_agent_copilot_proxy.ipynb │ │ └── wind_farm │ │ ├── README.md │ │ ├── assistant-wind_farm.ipynb │ │ └── data │ │ └── turbines.csv ├── nlp-sql-in-a-box │ └── readme.md └── semantic-kernel-bot-in-a-box │ └── README.md ├── guidance ├── genai-security │ └── README.md ├── responsible-ai │ └── README.md └── scaling │ └── README.md ├── machine-learning └── ml-ops-in-a-box │ └── README.md └── media ├── Assets └── AI-in-a-Box-PatternPool.vsdx └── images ├── AI-in-a-Box.jpeg ├── OWASP-Security-Risks-LLM-Apps.png ├── Pattern1.A.MLOps.png ├── Pattern2.A.EdgeAI.png ├── Pattern3.A.1.png ├── Pattern3.A.App.png ├── Pattern3.A.Landing.png ├── Pattern4.A.png ├── Pattern4.Landing.png ├── Rai-mitigation-layers.png ├── ai-in-a-box.png ├── ai-in-a-box.svg ├── aibxtable.png ├── aibxtable.svg ├── aiinabox.png ├── aiservices-in-a-box.png ├── aiservices-in-a-box.svg ├── aoai-in-a-box.png ├── aoai-in-a-box.svg ├── assistantsapi-flow-overview.svg ├── banner-ai-landing-zone.png ├── banner-aio-with-ai-in-a-box.png ├── banner-aivision-edge-in-a-box.png ├── banner-aml-edge-in-a-box.png ├── banner-aoai-video-analysis-in-a-box.png ├── banner-assistants-api-bot-in-a-box.png ├── banner-assistants-api-in-a-box.png ├── banner-assistants-in-a-box.png ├── banner-customvision-edge-in-a-box.png ├── banner-doc-intelligence-in-a-box.png ├── banner-edgeai-in-a-box.png ├── banner-genai-security.png ├── banner-mlops-in-a-box.png ├── banner-nlp-to-sql-in-a-box.png ├── banner-rai.png ├── bluebox.svg ├── buildingblocks.png ├── edgeai-customvision-architecture.png ├── edgeai-in-a-box.png ├── edgeai-in-a-box.svg ├── edgeai-tree-options.png ├── justthebox.png ├── justthebox.svg ├── logo.png ├── mindmap.png ├── ml-in-a-box.png ├── ml-in-a-box.svg └── sk-bot-in-a-box.png /.github/ISSUE_TEMPLATE/bug_report.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Bug report 3 | about: Create a report to help us improve 4 | title: "[BUG] Issue with ..." 5 | labels: bug 6 | assignees: '' 7 | 8 | --- 9 | 10 | **Solution Accelerators** 11 | This repository contains multiple solution accelerators. Please tell us which ones are involved in your report. (Replace the space in between square brackets with an x) 12 | - [ ] Cognitive Services Landing Zone 13 | - [ ] Semantic Kernel Bot 14 | - [ ] Azure ML Operationalization 15 | - [ ] Edge AI 16 | 17 | **Describe the bug** 18 | A clear and concise description of what the bug is. 19 | 20 | **To Reproduce** 21 | Steps to reproduce the behavior: 22 | 1. Go to '...' 23 | 2. Click on '....' 24 | 3. Scroll down to '....' 25 | 4. See error 26 | 27 | **Expected behavior** 28 | A clear and concise description of what you expected to happen. 29 | 30 | **Screenshots** 31 | If applicable, add screenshots to help explain your problem. 32 | 33 | **Please complete the following information:** 34 | - OS: [e.g. Windows] 35 | - Browser [e.g. chrome, safari] 36 | - Version [e.g. 22] 37 | 38 | **Additional context** 39 | Add any other context about the problem here. 40 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/feature_request.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Feature request 3 | about: Suggest an idea for this project 4 | title: "[FEATURE REQUEST] Support for ..." 5 | labels: enhancement 6 | assignees: '' 7 | 8 | --- 9 | 10 | **Solution Accelerators** 11 | This repository contains multiple solution accelerators. Please tell us which ones are involved in your report. (Replace the space in between square brackets with an x) 12 | - [ ] Cognitive Services Landing Zone 13 | - [ ] Semantic Kernel Bot 14 | - [ ] Azure ML Operationalization 15 | - [ ] Edge AI 16 | 17 | **Is your feature request related to a problem? Please describe.** 18 | A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] 19 | 20 | **Describe the solution you'd like** 21 | A clear and concise description of what you want to happen. 22 | 23 | **Describe alternatives you've considered** 24 | A clear and concise description of any alternative solutions or features you've considered. 25 | 26 | **Additional context** 27 | Add any other context or screenshots about the feature request here. 28 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | .DS_Store 2 | 3 | # dotnet 4 | bin 5 | obj 6 | appsettings.json 7 | 8 | .azure 9 | 10 | .env 11 | *_db 12 | 13 | __pycache__ 14 | 15 | .cache -------------------------------------------------------------------------------- /CODEOWNERS: -------------------------------------------------------------------------------- 1 | # Owners for this repo 2 | * @AndresPad @codebytes @neerajjhaveri @MarcoABCardoso 3 | 4 | /ai-services/* @jehayesms @AndresPad @MarcoABCardoso 5 | /machine-learning/ml-ops-in-a-box/* @Welasco @AndresPad 6 | 7 | /gen-ai/Assistants/* @msalemor @AndresPad 8 | /gen-ai/semantic-kernel-bot-in-a-box/* @MarcoABCardoso 9 | 10 | /guidance/* @codebytes @KBaroni @AndresPad -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | # Microsoft Open Source Code of Conduct 2 | 3 | This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). 4 | 5 | Resources: 6 | 7 | - [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/) 8 | - [Microsoft Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) 9 | - Contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with questions or concerns 10 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) Microsoft Corporation. 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # AI-in-a-Box 2 | 3 |

4 | FTA AI-in-a-Box: Deployment Accelerator 5 |

6 | 7 | *AI-in-a-Box* leverages the collective expertise of Microsoft Customer Engineers and Architects across the globe to develop and provide AI and ML solutions to the technical community. 8 | 9 | Our intent is to present a curated collection of solution accelerators that can help engineers establish their AI/ML environments and solutions rapidly and with minimal friction, while maintaining the highest standards of quality and efficiency. 10 | 11 | As we continue to learn from the market, the contributors will look to equip the community with the tools and resources necessary to succeed in the ever-evolving AI and ML landscape. 12 | 13 | ## Why AI-in-a-Box? 14 | 15 | * Accelerated Deployment: Speed up your solutions with our proven, ready-to-use patterns. 16 | * Cost Savings: Maximize your budget by reusing existing code and patterns. 17 | * Enhanced Quality & Reliability: Trust in our solutions, validated through real-world scenarios. 18 | * Competitive Advantage: Outpace competitors by accelerating solution deployment. 19 | 20 |

21 | 22 | ![FTA AI-in-a-Box: Deployment Accelerator](/media/images/aibxtable.png) 23 | 24 |

25 | 26 | ## Available Guidance 27 | 28 | |Topic|Description| 29 | |---|---| 30 | |[Responsible AI](./guidance/responsible-ai/) | This provides essential guidance on the responsible use of AI and LLM technologies. | 31 | |[Security for Generative AI Applications](./guidance/genai-security/)| This document provides specific security guidance for Generative AI (GenAI) applications. | 32 | |[Scaling OpenAI Applications](./guidance/scaling/)| This document contains best practices for scaling OpenAI applications within Azure. | 33 | 34 | ## Available “-in-a-Box” accelerators 35 | 36 | |Pattern|Description|Supported Use Cases and Features| 37 | |---|---|---| 38 | |[Azure ML Operationalization in-a-box](https://github.com/Azure-Samples/ml-ops-in-a-box)|Boilerplate Data Science project from model development to deployment and monitoring|
  • End-to-end MLOps project template
  • Outer Loop (infrastructure setup)
  • Inner Loop (model creation and deployment lifecycle)| 39 | |[Edge AI in-a-box](./edge-ai/)|Edge AI from model creation to deployment on Edge Device(s) |
  • Create a model and deploy to Edge Device.
  • Outer Loop Infrastructure Setup (IoT Hub, IoT Edge, Edge VM, Container Registry, Azure ML)
  • Inner Loop (model creation and deployment)| 40 | |[AML Edge in-a-box](https://github.com/Azure-Samples/aml-edge-in-a-box)|Edge AI from model creation to deployment on Edge Device(s) |Orchestrate the entire Edge AI model lifecycle—from creation to deployment—using Azure ML, IoT Edge, and IoT Hub, while leveraging Azure ML CLI V2 for streamlined management.| 41 | |[Custom Vision Edge in-a-box](https://github.com/Azure-Samples/customvision-edge-in-a-box)|Edge AI from model creation to deployment on Edge Device(s) |Edge AI mitigates cloud latency by shifting analysis closer to the data source for faster responses. This accelerator demonstrates using [Custom Vision](https://www.customvision.ai/) to train a model and export it in formats like ONNX or Dockerfile for edge deployment.| 42 | |[Doc Intelligence in-a-box](https://github.com/Azure-Samples/doc-intelligence-in-a-box) | This accelerator enables companies to automate PDF form processing, modernize operations, save time, and cut costs as part of their digital transformation journey.|
  • Receive PDF Forms
  • Function App and Logic App for Orchestration
  • Document Intelligence Model creation for form processing and content extraction
  • Saves PDF data in Azure Cosmos DB | 43 | |[Image and Video Analysis in-a-box](https://github.com/Azure-Samples/gpt-video-analysis-in-a-box) | Extracts information from images and videos with Azure AI Vision and sends the results along with the prompt and system message to Azure GPT-4 Turbo with Vision. |
  • Orchestration through Azure Data Factory
  • Low code solution, easily extensible for your own use cases through ADF parameters
  • Reuse same solution and deployed resources for many different scenarios
  • Saves GPT4-V results to Azure CosmosDB| 44 | |[Cognitive Services Landing Zone in-a-box](https://github.com/Azure-Samples/ai-landing-zone-in-a-box)|Minimal enterprise-ready networking and AI Services setup to support most Cognitive Services scenarios in a secure environment|
  • Hub-and-Spoke Vnet setup and peering
  • Cognitive Service deployment
  • Private Endpoint setup
  • Private DNS integration with PaaS DNS resolver| 45 | |[Semantic Kernel Bot in-a-box](https://github.com/Azure-Samples/gen-ai-bot-in-a-box)|Extendable solution accelerator for advanced Azure OpenAI Bots|
  • Deploy Azure OpenAI bot to multiple channels (Web, Teams, Slack, etc)
  • Built-in Retrieval-Augmented Generation (RAG) support
  • Implement custom AI Plugins| 46 | [NLP to SQL in-a-box](https://github.com/Azure-Samples/nlp-sql-in-a-box)|Unleash the power of a cutting-edge speech-enabled SQL query system with Azure OpenAI, Semantic Kernel, and Azure Speech Services. Simply speak your data requests in natural language, and let the magic happen.|
  • Allows users to verbally express natural language queries
  • Translate into SQL statements using Azure Speech & AOAI
  • Execute on an Azure SQL DB | 47 | |[Assistants API notebooks](./gen-ai/Assistants/notebooks)|Harnessing the simplicity of the Assistants API, developers can seamlessly integrate assistants with diverse functionalities, from executing code to retrieving data, empowering users with versatile and dynamic digital assistants tailored to their needs.|
  • Offers three main capabilities: Code Interpreter (tech tasks), Retrieval (finding info), and Function calling (task execution)
  • These powers combine to form a versatile super-assistant for handling diverse tasks | 48 | |[Assistants API Bot in-a-box](https://github.com/Azure-Samples/gen-ai-bot-in-a-box)|This sample provides a step-by-step guide on how to deploy a virtual assistant leveraging the Azure OpenAI Assistants API. It covers the infrastructure deployment, configuration on the AI Studio and Azure Portal, and end-to-end testing examples.|
  • Deploy the necessary infrastructure to support an Azure OpenAI Assistant
  • Configure as Assistant with the required tools.
  • Connect a Bot Framework application to your Assistant to deploy the chat to multiple channels | 49 | 50 | ## Key Contacts & Contributors 51 | 52 | If you have any questions or would like to contribute please reach out to: `aibox@microsoft.com` 53 | 54 | | Contact | GitHub ID | Email | 55 | |--------------|------|-----------| 56 | | Alex Morales | @msalemor | `alemor@microsoft.com` | 57 | | Andre Dewes | @andredewes | `andredewes@microsoft.com` | 58 | | Andrés Padilla | @AndresPad | `andres.padilla@microsoft.com` | 59 | | Chris Ayers | @codebytes | `chrisayers@microsoft.com` | 60 | | Eduardo Noriega | @EduardoN | `ednorieg@microsoft.com` | 61 | | Franklin Guimaraes | @franklinlindemberg | `fguimaraes@microsoft.com` | 62 | | Jean Hayes | @jehayesms | `jean.hayes@microsoft.com` | 63 | | Marco Aurélio Bigélli Cardoso | @MarcoABCardoso | `macardoso@microsoft.com` | 64 | | Maria Vrabie | @MariaVrabie | `mavrabie@microsoft.com` | 65 | | Neeraj Jhaveri | @neerajjhaveri | `neeraj.jhaveri@microsoft.com` | 66 | | Thiago Rotta | @rottathiago | `thiago.rotta@microsoft.com` | 67 | | Victor Santana | @Welasco | `vsantana@microsoft.com` | 68 | | Sabyasachi Samaddar | @ssamadda | `ssamadda@microsoft.com` | 69 | -------------------------------------------------------------------------------- /SECURITY.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | ## Security 4 | 5 | Microsoft takes the security of our software products and services seriously, which includes all source code repositories managed through our GitHub organizations, which include [Microsoft](https://github.com/microsoft), [Azure](https://github.com/Azure), [DotNet](https://github.com/dotnet), [AspNet](https://github.com/aspnet), [Xamarin](https://github.com/xamarin), and [our GitHub organizations](https://opensource.microsoft.com/). 6 | 7 | If you believe you have found a security vulnerability in any Microsoft-owned repository that meets [Microsoft's definition of a security vulnerability](https://aka.ms/opensource/security/definition), please report it to us as described below. 8 | 9 | ## Reporting Security Issues 10 | 11 | **Please do not report security vulnerabilities through public GitHub issues.** 12 | 13 | Instead, please report them to the Microsoft Security Response Center (MSRC) at [https://msrc.microsoft.com/create-report](https://aka.ms/opensource/security/create-report). 14 | 15 | If you prefer to submit without logging in, send email to [secure@microsoft.com](mailto:secure@microsoft.com). If possible, encrypt your message with our PGP key; please download it from the [Microsoft Security Response Center PGP Key page](https://aka.ms/opensource/security/pgpkey). 16 | 17 | You should receive a response within 24 hours. If for some reason you do not, please follow up via email to ensure we received your original message. Additional information can be found at [microsoft.com/msrc](https://aka.ms/opensource/security/msrc). 18 | 19 | Please include the requested information listed below (as much as you can provide) to help us better understand the nature and scope of the possible issue: 20 | 21 | * Type of issue (e.g. buffer overflow, SQL injection, cross-site scripting, etc.) 22 | * Full paths of source file(s) related to the manifestation of the issue 23 | * The location of the affected source code (tag/branch/commit or direct URL) 24 | * Any special configuration required to reproduce the issue 25 | * Step-by-step instructions to reproduce the issue 26 | * Proof-of-concept or exploit code (if possible) 27 | * Impact of the issue, including how an attacker might exploit the issue 28 | 29 | This information will help us triage your report more quickly. 30 | 31 | If you are reporting for a bug bounty, more complete reports can contribute to a higher bounty award. Please visit our [Microsoft Bug Bounty Program](https://aka.ms/opensource/security/bounty) page for more details about our active programs. 32 | 33 | ## Preferred Languages 34 | 35 | We prefer all communications to be in English. 36 | 37 | ## Policy 38 | 39 | Microsoft follows the principle of [Coordinated Vulnerability Disclosure](https://aka.ms/opensource/security/cvd). 40 | 41 | 42 | -------------------------------------------------------------------------------- /SUPPORT.md: -------------------------------------------------------------------------------- 1 | # TODO: The maintainer of this repo has not yet edited this file 2 | 3 | **REPO OWNER**: Do you want Customer Service & Support (CSS) support for this product/project? 4 | 5 | - **No CSS support:** Fill out this template with information about how to file issues and get help. 6 | - **Yes CSS support:** Fill out an intake form at [aka.ms/onboardsupport](https://aka.ms/onboardsupport). CSS will work with/help you to determine next steps. 7 | - **Not sure?** Fill out an intake as though the answer were "Yes". CSS will help you decide. 8 | 9 | *Then remove this first heading from this SUPPORT.MD file before publishing your repo.* 10 | 11 | # Support 12 | 13 | ## How to file issues and get help 14 | 15 | This project uses GitHub Issues to track bugs and feature requests. Please search the existing 16 | issues before filing new issues to avoid duplicates. For new issues, file your bug or 17 | feature request as a new Issue. 18 | 19 | For help and questions about using this project, please **REPO MAINTAINER: INSERT INSTRUCTIONS HERE 20 | FOR HOW TO ENGAGE REPO OWNERS OR COMMUNITY FOR HELP. COULD BE A STACK OVERFLOW TAG OR OTHER 21 | CHANNEL. WHERE WILL YOU HELP PEOPLE?**. 22 | 23 | ## Microsoft Support Policy 24 | 25 | Support for this **PROJECT or PRODUCT** is limited to the resources listed above. 26 | -------------------------------------------------------------------------------- /ai-services/ai-landing-zone/README.md: -------------------------------------------------------------------------------- 1 | # Cognitive Services Landing Zone in-a-box 2 | 3 | ![Banner](/media/images/banner-ai-landing-zone.png) 4 | 5 | ## What's in this box? 6 | 7 | This deployment template contains a minimal Landing Zone configuration that enables: 8 | 9 | - Cognitive services that implement a wide range of Natural Language functionality, including Azure OpenAI, Document Intelligence and AI Search; 10 | - Hub-and-Spoke network configuration; 11 | - Private endpoint connectivity between Azure services; 12 | - DNS Resolver and Private DNS Zones to enable domain name resolution within the Vnets; 13 | - App Services with Vnet integration to host custom applications which rely on Cognitive Services. 14 | 15 | ## MOVED 16 | 17 | We have moved the solution to a new repository. Please visit the new repository [https://github.com/Azure-Samples/ai-landing-zone-in-a-box](https://github.com/Azure-Samples/ai-landing-zone-in-a-box). 18 | -------------------------------------------------------------------------------- /ai-services/doc-intelligence-in-a-box/README.md: -------------------------------------------------------------------------------- 1 | # Doc-Intelligence-in-a-Box 2 | 3 | ![Banner](../../media/images/banner-doc-intelligence-in-a-box.png) 4 | 5 | This solution leverages Azure AI Document Intelligence and Bicep to automate document processing tasks. It deploys necessary Azure resources using Bicep scripts and trains custom machine learning models for document extraction. Azure Data Lake Storage is used for storing sample data, and the trained models are integrated into Azure Functions for processing documents. This setup allows for easy customization and scalability for various document processing use cases. 6 | 7 | ## MOVED 8 | 9 | We have moved the solution to a new repository. Please visit the new repository [https://github.com/Azure-Samples/doc-intelligence-in-a-box](https://github.com/Azure-Samples/doc-intelligence-in-a-box). 10 | -------------------------------------------------------------------------------- /ai-services/gpt-video-analysis-in-a-box/README.md: -------------------------------------------------------------------------------- 1 | # Image and Video Analysis-Azure Open AI in-a-box 2 | 3 | ![banner](../../media/images/banner-aoai-video-analysis-in-a-box.png) 4 | This solution examines videos and image of vehicles for damage using Azure Open AI GPT-4 Turbo with Vision and Azure AI Vision Image Analysis 4.0. All orchestration is done with Azure Data Factory, allowing this solution to be easily customized for your own use cases. 5 | 6 | ## MOVED 7 | 8 | We have moved the solution to a new repository. Please visit the new repository [https://github.com/Azure-Samples/gpt-video-analysis-in-a-box](https://github.com/Azure-Samples/gpt-video-analysis-in-a-box). 9 | -------------------------------------------------------------------------------- /edge-ai/README.MD: -------------------------------------------------------------------------------- 1 | # EdgeAI in-a-box 2 | 3 | ![Banner](../media/images/banner-edgeai-in-a-box.png) 4 | 5 | ## Why Edge AI? 6 | 7 | Edge AI excels in diverse applications like image classification, object detection, body, face, and gesture analysis, as well as image manipulation. Moreover, in this swiftly evolving Generative AI (Gen AI) landscape, the move from cloud-based Gen AI to Edge Gen AI is becoming more prominent. This shift is not just desirable but increasingly necessary, propelled by demands for privacy, security, hyper-personalization, accuracy, cost-effectiveness, energy efficiency, and more. While the commercialization of today's cloud-based Gen AI is in full swing, there are efforts underway to optimize models to run on power-sipping edge devices with efficient mobile GPUs, Neural and Tensor processors (NPU and TPU). 8 | 9 | With this in mind, we aim to simplify your understanding of creating a model, packaging it, and deploying it to the edge. The series kicks off by establishing the baseline of creating and deploying a model with Azure ML and IoT Edge. We'll then explore more specific scenarios, such as deploying a model with AKS and AKS Edge, dive into creating a model with ONNX Runtime, and finally guide you into the realm of Edge Gen AI. Our ultimate goal is to provide you with a clearer understanding of your options within Azure for Edge AI scenarios. 10 | 11 | ## Why does AI-in-a-Box have an Edge AI Section? 12 | 13 | ### Because we want to show you the options of crafting and deploying a model anywhere 14 | 15 | **Edge AI** plays a crucial role in expanding AI and ML capabilities by bringing them directly to edge devices. This emphasizes the importance of running AI models on these devices and underscores the essential interplay between various Azure services. This series introduces a progressive flow designed to enhance your comprehension of model creation and deployment on Edge devices or Edge-compatible containers. It guides you through the journey, starting with fundamental concepts, advancing towards AKS deployments, and eventually delving into more advanced GenAI Edge scenarios. These practical examples aim to deepen your understanding of the possibilities within Azure for Edge AI scenarios, providing valuable insights into the diverse applications of this technology. 16 | 17 | ![Banner](../media/images/edgeai-tree-options.png) 18 | 19 | The series will encompass: 20 | 21 | 1. [AML Edge](https://github.com/Azure-Samples/aml-edge-in-a-box) - Creating a model in Azure ML, leveraging AutoML, packaging it correctly and deploying with IoT Edge to an Edge Device. 22 | 2. [Custom Vision Edge](https://github.com/Azure-Samples/customvision-edge-in-a-box) - Creating a model with Azure's [Custom Vision](https://learn.microsoft.com/en-us/azure/ai-services/custom-vision-service/overview) and deploying with IoT Edge and running that AI model in a docker container. 23 | 24 | 3. [AI Vision](https://learn.microsoft.com/en-us/azure/ai-services/computer-vision/overview) - Azure AI services provides several Docker containers that let you use the same APIs that are available in Azure, on-premises. Using these containers gives you the flexibility to bring Azure AI services closer to your data for compliance, security or other operational reasons. 25 | - Optical Character Recognition (OCR) - The Read OCR container allows you to extract printed and handwritten text from images and documents with support for JPEG, PNG, BMP, PDF, and TIFF file formats. For more information, see the [Read API documentation](https://learn.microsoft.com/en-us/azure/ai-services/computer-vision/overview-ocr). This container can also [run in disconnected environments](https://learn.microsoft.com/en-us/azure/ai-services/containers/disconnected-containers). 26 | 27 | We are actively advancing this series to showcase the creation and deployment of models within Azure-Arc enabled Kubernetes and [Azure IoT Operations](https://learn.microsoft.com/en-us/azure/iot-operations/get-started/overview-iot-operations) scenarios. 28 | 29 | 4. [Azure IoT Operations (AIO)](https://aka.ms/aio-edgeai): 30 | - **Azure ML to AIO** - Creating a model and deploying that model in Azure-Arc enabled Kubernetes and [Azure IoT Operations](https://learn.microsoft.com/en-us/azure/iot-operations/get-started/overview-iot-operations) scenarios with the [Azure ML Extension](https://learn.microsoft.com/en-us/azure/machine-learning/how-to-attach-kubernetes-anywhere?view=azureml-api-2). 31 | - **AIO with Open AI** - Using ***[Cerebral](https://github.com/Azure/arc_jumpstart_drops/tree/main/sample_app/cerebral_genai)***, we will demonstrate how to seamlessly integrate Azure OpenAI and Natural Language Processing (NLP) into an AIO deployment. This setup will highlight how to implement Retrieval-Augmented Generation (RAG) for efficient document retrieval and actionable insights. By optimizing the use of local AIO resources alongside Azure OpenAI, this enhancement delivers faster, more contextually aware responses directly from the network edge. 32 | - **AIO with SLMs (RAG) on Edge** - We will demonstrate how to optimize your [AIO](https://learn.microsoft.com/en-us/azure/iot-operations/get-started/overview-iot-operations)/***[Cerebral](https://github.com/Azure/arc_jumpstart_drops/tree/main/sample_app/cerebral_genai)*** setup by integrating Small Language Models (SLMs) from various model catalogs for flexible edge deployments, enabling Retrieval-Augmented Generation (RAG) at the Edge. By localizing key GenAI processing tasks, this approach reduces dependence on cloud resources, allowing RAG to function seamlessly in disconnected/hybrid environments. 33 | 34 | Additionally, we plan to expand the series by offering valuable knowledge and key building blocks to help you navigate GenAI scenarios at the edge. This includes practical insights into integrating ChatGPT with Azure AI Vision, performing video analysis using Azure Video Indexer, and leveraging GPT-4 Turbo with Vision, GPT-4o, or Phi-3 Vision for tasks such as OCR, object detection, video insights, and image analysis. 35 | 36 | 5. [GenAI Edge]() - Working with GenAI at the edge scenarios -> Work in Progress. 37 | 38 | Stay tuned for more exciting accelerators in the pipeline, as there's much more to come! 39 | 40 | ## Quick Note: Model customization in Azure Custom Vision and/or Azure AI Services 41 | 42 | You can train a custom model using either the [Custom Vision](https://learn.microsoft.com/en-us/azure/ai-services/custom-vision-service/overview) service or the Image Analysis 4.0 service (within [Azure AI Services](https://learn.microsoft.com/en-us/azure/ai-services/computer-vision/overview)) with model customization. The [following table](https://learn.microsoft.com/en-us/azure/ai-services/computer-vision/concept-model-customization) compares the two services. 43 | 44 | |Areas|Custom Vision service|Image Analysis 4.0 service| 45 | |---|---|---| 46 | |Tasks | Image classification
    Object detection | Image classification
    Object detection | 47 | |Base model | CNN | Transformer model | 48 | |Labeling | [CustomVision.ai](https://www.customvision.ai/) | [AML Studio](https://ml.azure.com/) | 49 | |Web Portal | [CustomVision.ai](https://www.customvision.ai/) | [Vision Studio](http://aka.ms/VisionStudio) | 50 | |Libraries | REST, SDK | REST, Python Sample | 51 | |Minimum training data needed | 15 images per category |2-5 images per category | 52 | |Training data storage | Uploaded to service | Customer’s blob storage account | 53 | |Model hosting | Cloud and **Edge** | Cloud hosting only,
    ***Edge container hosting to come*** | 54 | 55 | ## Additional Resources 56 | 57 | ### Check out our Mindmap 58 | 59 | [![ml edge mindmap](/media/images/mindmap.png) ML/AI Edge MindMap](https://aka.ms/mledge-mm) 60 | -------------------------------------------------------------------------------- /edge-ai/aml-edge-in-a-box/readme.md: -------------------------------------------------------------------------------- 1 | # AML Edge in-a-box 2 | 3 | ![Banner](/media/images/banner-aml-edge-in-a-box.png) 4 | 5 | Orchestration of resources for the entire Edge AI model lifecycle, including creation, deployment, and proper packaging through Azure ML and IoT Edge. This involves leveraging key components such as IoT Hub, IoT Edge, Azure ML, and Azure ML CLI V2. 6 | 7 | ## MOVED 8 | 9 | We have moved the solution to a new repository. Please visit the new repository [https://github.com/Azure-Samples/aml-edge-in-a-box](https://github.com/Azure-Samples/aml-edge-in-a-box). 10 | -------------------------------------------------------------------------------- /edge-ai/customvision-edge-in-a-box/readme.md: -------------------------------------------------------------------------------- 1 | # Custom Vision Edge in-a-box 2 | 3 | ![Banner](/media/images/banner-customvision-edge-in-a-box.png) 4 | 5 | In real-time decision scenarios with cloud latency challenges, Edge AI enables us to shift analysis closer to the data source, extracting responsive results. These advantages are exemplified in the architecture below. In this accelerator, we aim to demonstrate how easy it is to leverage [Custom Vision](https://www.customvision.ai/) for training a classification model and exporting it to a specific format, such as ONNX or a Dockerfile, ready for deployment to an edge container. 6 | 7 | ## MOVED 8 | 9 | We have moved the solution to a new repository. Please visit the new repository [https://github.com/Azure-Samples/customvision-edge-in-a-box](https://github.com/Azure-Samples/customvision-edge-in-a-box). 10 | -------------------------------------------------------------------------------- /gen-ai/Assistants/README.md: -------------------------------------------------------------------------------- 1 | # Assistants in-a-Box 2 | ![Banner](../../media/images/banner-assistants-in-a-box.png) 3 | 4 | ## Overview 5 | The Assistants API enables you to create AI assistants within your own apps. These assistants have instructions and can use models, tools, and knowledge to answer user questions. The API currently offers three types of tools: Code Interpreter, Retrieval, and Function calling. 6 | 7 | ## Key Features and Benefits 8 | - **Code Interpreter Tool:** The Code Interpreter feature of the Assistants API enables the writing and execution of Python code within a secure environment. This functionality is capable of handling various types of files, including those with different data and formatting, and can produce output files containing data and graphical representations. 9 | - **Function Calling Tool:** The Assistants API, like the Chat Completions API, enables function calling. With function calling, you can define functions for the Assistants and it will intelligently provide the necessary functions to be called, along with their respective arguments. 10 | - **Knowledge Retrieval:** The Assistant can enhance its knowledge by incorporating information from external sources, such as exclusive product details or documents shared by users. When a file is uploaded and given to the Assistant, OpenAI will automatically break down the documents into smaller parts, organize and store the embeddings, and utilize vector search to find relevant content that can be used to respond to user inquiries. 11 | - **Threads:** A Thread represents a conversation. The Assistant will ensure that requests to the model fit within the maximum context window, using relevant optimization techniques such as truncation 12 | 13 | ## Use Case 14 | Envision an all-in-one assistant aiding with finances, fetching information, and handling tasks effortlessly. The Assistants API offers developers a toolkit to enrich their apps. Key features like ***Function Calling*** enable seamless integration for tasks like messaging and device control. With ***Knowledge Retrieval***, developers can tap into vast data, ensuring accurate user responses. The API's ***Code Interpreter*** provides calculation abilities, while blending these powers crafts a versatile, assistant for diverse tasks. 15 | 16 | ## How It Works 17 | The Assistants API operates through a straightforward flow: 18 | 19 | 20 | ![Banner](../../media/images/assistantsapi-flow-overview.svg) 21 | 22 | 1. **Assistant Creation:** Developers define an assistant with custom instructions and select a model, along with enabling specific tools like Code Interpreter, Retrieval, and Function calling. 23 | 2. **Thread Creation:** When a user initiates a conversation, a thread is created to facilitate the interaction. 24 | 3. **Message Exchange:** Users interact with the assistant by sending messages, which are added to the thread. 25 | 4. **Assistant Execution:** The assistant processes the messages within the thread, triggering relevant tools as necessary. 26 | 5. **Response Handling:** Once processing is complete, the assistant generates responses, which are then delivered back to the user via the thread. 27 | 28 | ## Samples 29 | 30 | Check out the provided samples to get started with integrating the Assistants API into your application. The code showcases foundational concepts such as Assistants, Threads, Messages, Runs, and the Assistant lifecycle, offering a clear starting point for implementation. 31 | 32 | | Topic | Description | 33 | |----------------------|--------------------------------------------------| 34 | | [Math Tutor](./notebooks/math_tutor/assistant-math_tutor.ipynb) | Showcases the foundational concepts of Assistants such as Threads, Messages, Runs, Tools, and lifecycle management. | 35 | | [Financial Assistant](./notebooks/personal_finance/assistant-personal_finance.ipynb) | Function Calling with Yfinance to get latest stock prices. Summarization of user provided article. Extract country info from article, extract country, capital and other aspects, and call an API to get more information about each country. | 36 | | [Failed Banks](./notebooks/failed_banks/assistant-failed_banks.ipynb) | Failed Banks is an Assistant designed to analyze and extract data concerning failed banks. It can provide insights into questions like identifying failed banks within specific states during a given time frame and generate charts illustrating bank failures across the US. | 37 | | [Wind Farm](./notebooks/wind_farm/assistant-wind_farm.ipynb) | Utilizing Assistant tools such as the Code Interpreter and Function calling, this bot is capable of retrieving a CSV file that illustrates turbine wind speed, voltage, and the last maintenance date. It assists you in reviewing the file contents and aids in determining whether a specific turbine is in need of maintenance. | 38 | | [Sales Assistant](./notebooks/sales_assistant/assistant_sales.ipynb) | Showcases how you can create an Assistant adept at managing various tasks, such as handling relational data across different files and supporting multiple users simultaneously within the same Assistant across distinct threads. | 39 | | [Autogen Assistants](./notebooks/autogen/gpt_assistant_agent.ipynb) | Get started with Assistants on Autogen, with examples of basic assistants, code interpreter, retrieval and function calling, entirely managed by the Autogen framework. | 40 | | [File Search Assistant](./notebooks/file_search/assistant-file-search.ipynb) | Leverage File Search with Assistants, with examples of retrieval over a document containing benefits information. | 41 | | [Assistants Bot-in-a-Box](./bot-in-a-box/) | The Assistants API Bot in-a-box automates virtual assistant deployment on Azure, facilitating seamless communication between users and assistants across various messaging channels. It streamlines message processing and response delivery through Azure Bot Services and the Assistants API. | 42 | 43 | ## Prerequisites 44 | Before using the Assistants API, ensure you have: 45 | 46 | * An [Azure subscription](https://azure.microsoft.com/en-us/free/). 47 | * Created and deployed an [Azure OpenAI Service](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/create-resource) Resource. 48 | * Set up the necessary environment variables containing API credentials. (i.e. [.env]()) 49 | * Installed Python 3.7+ for running the provided sample code. -------------------------------------------------------------------------------- /gen-ai/Assistants/bot-in-a-box/README.md: -------------------------------------------------------------------------------- 1 | # Assistants API Bot in-a-Box 2 | 3 | ![Banner](../../../media/images/banner-assistants-api-bot-in-a-box.png) 4 | 5 | ## Overview 6 | 7 | This sample provides a step-by-step guide on how to deploy a virtual assistant leveraging the Azure OpenAI Assistants API. It covers the infrastructure deployment, configuration on the AI Studio and Azure Portal, and end-to-end testing examples. 8 | 9 | ## MOVED and ARCHIVED 10 | 11 | This solution is archived at: [https://github.com/Azure-Samples/aoai-assistants-bot-in-a-box](https://github.com/Azure-Samples/aoai-assistants-bot-in-a-box). 12 | 13 | ## SUCCESSOR 14 | 15 | The successor to this solution, which is actively maintained and includes enhanced capabilities, is available at: [https://github.com/Azure-Samples/gen-ai-bot-in-a-box](https://github.com/Azure-Samples/gen-ai-bot-in-a-box). 16 | 17 | The new repository builds upon the foundation of the original Assistants Bot in-a-box, offering additional features, improved performance, and expanded integration options. It is designed to provide a more robust and flexible framework for developing AI-powered bots using the latest advancements in generative AI and Azure services. 18 | 19 | By migrating to the new repository, you will benefit from continuous updates, new feature additions, and ongoing support from the development community. Whether you are looking to enhance your existing bot or start a new project, the [Gen AI Bot in-a-box](https://github.com/Azure-Samples/gen-ai-bot-in-a-box) offers a comprehensive solution that evolves with the latest AI technologies. 20 | -------------------------------------------------------------------------------- /gen-ai/Assistants/notebooks/.env-sample: -------------------------------------------------------------------------------- 1 | OPENAI_URI=https://.openai.azure.com/ 2 | BASE_URL=https://.openai.azure.com/openai 3 | OPENAI_KEY= 4 | OPENAI_VERSION=2024-05-01-preview 5 | OPENAI_GPT_DEPLOYMENT= 6 | -------------------------------------------------------------------------------- /gen-ai/Assistants/notebooks/README.md: -------------------------------------------------------------------------------- 1 | # Assistants API in-a-Box 2 | 3 | ![Banner](./readme_assets/banner-assistants-api-in-a-box.png) 4 | 5 | ## Use Case 6 | 7 | Imagine an assistant that helps you figure out your finances, retrieves useful info, and does special functions—all in one go! 8 | 9 | Using the Assistants API is simple. Just tell your assistant what to do, like having a conversation. It can understand different file formats, execute code (super techy!), and do all sorts of cool tricks. As Azure OpenAI adds more tools, your assistant can do even cooler things. It's a fun and easy way to add some brainpower to your GenAI app! 10 | 11 | By leveraging the Assitants API, developers can create assistants with easy-to-follow instructions that use special tools to get things done. Right now, it has three superpowers: Code Interpreter (for doing techy stuff), Retrieval (for finding info), and Function calling (for making things happen). You can even mix these powers to create a super-assistant that can handle all sorts of tasks. 12 | 13 | In this accelerator, our goal is to demonstrate the simplicity of integrating the **Assistants API**. We aim to empower you with the knowledge to grasp fundamental concepts and explore the full range of capabilities that the API offers. 14 | 15 | ## Assistants API Flow 16 | 17 | At a high level, a typical integration of the Assistants API has the following flow: 18 | 19 | 1. Create an [Assistant](https://platform.openai.com/docs/api-reference/assistants/createAssistant) in the API by defining its custom instructions and picking a model. If helpful, enable tools like Code Interpreter, Retrieval, and Function calling. 20 | 2. Create a [Thread](https://platform.openai.com/docs/api-reference/threads) when a user starts a conversation. 21 | 3. Add [Messages](https://platform.openai.com/docs/api-reference/messages) to the Thread as the user ask questions. 22 | 4. [Run](https://platform.openai.com/docs/api-reference/runs) the Assistant on the Thread to trigger responses. This automatically calls the relevant tools. 23 | 24 | ![Banner](./readme_assets/assistantsapi-flow-overview.svg) 25 | 26 | ## Benefits of using assistants 27 | 28 | | Feature | Description | 29 | |--------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| 30 | | Threads | Contains Messages and automatically handles the truncation of content to fit within the context of a model. | 31 | | Files | Allows importing file content of different file formats. Can be used in tools such as Retrieval scenarios and analysis with Code Interpreter. | 32 | | Tools | Includes Code Interpreter, Knowledge Retrieval, and Function calling.
     - Code Interpreter allows executing code snippets.
     - Knowledge Retrieval automatically chunks and embeds content in files for augmented retrieval scenarios.
     - Function calling enables calling functions. | 33 | | Tool composition | Enables using multiple tools in one Assistant. | 34 | 35 | ## Foundational concepts 36 | 37 | | Term | Definition | 38 | |--------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| 39 | | Assistant | An AI system specifically designed to utilize OpenAI's models and call tools. | 40 | | Thread | A session of conversation between a user and an Assistant. Threads store Messages and automatically handle truncation to ensure that content fits within the context of the model. | 41 | | Message | A piece of communication generated by either an Assistant or a user. Messages can contain text, images, and other files. Messages are stored as a list within a Thread. | 42 | | Run | An instance of an Assistant being invoked on a Thread. The Assistant utilizes its configuration and the Messages within the Thread to perform tasks by calling models and tools. During a Run, the Assistant appends Messages to the Thread. | 43 | | Run Step | A detailed record of the individual actions taken by the Assistant during a Run. These steps can include calling tools or generating Messages. Examining Run Steps provides insight into how the Assistant arrives at its final results. | 44 | 45 | ## Lifecycle 46 | 47 | | Status | Description | 48 | |--------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| 49 | | queued | When Runs are initially created or when the required action is completed, they are placed in a queued status. They should quickly transition to an in_progress status. | | in_progress | While in_progress, the Assistant utilizes the model and tools to perform steps. The progress of the Run can be monitored by examining the Run Steps. | 50 | | completed | The Run has successfully finished! You can now access all the Messages added by the Assistant to the Thread, as well as the steps taken by the Run. You can also continue the conversation by adding more user Messages to the Thread and initiating another Run. | 51 | | requires_action | When using the Function calling tool, the Run will transition to a required_action state once the model determines the names and arguments of the functions to be called. You must then execute those functions and submit the outputs before the Run can proceed. If the outputs are not provided before the expires_at timestamp (approximately 10 minutes after creation), the Run will move to an expired status. | 52 | | expired | This occurs when the function calling outputs were not submitted before the expires_at timestamp and the Run expires. Additionally, if the Run takes too long to execute and exceeds the time specified in expires_at, our systems will mark the Run as expired. | 53 | | cancelling | You can attempt to cancel an in_progress Run by using the Cancel Run endpoint. Once the cancellation attempt is successful, the status of the Run will change to cancelled. However, cancellation is not guaranteed and may not always be possible. | 54 | | cancelled | The Run was successfully cancelled. | 55 | | failed | The reason for the failure can be viewed by examining the last_error object in the Run. The timestamp for the failure will be recorded under failed_at. | 56 | 57 | ### Reference Sample code 58 | 59 | Reference: 60 | 61 | - [OpenAI Assistant Sample](https://github.com/openai/openai-python/blob/main/examples/assistant.py) 62 | 63 | **Note:** the following code showcases the foundational Assistant concepts such as Assistants, Threads, Messages, Runs and the Assistant lifecycle. 64 | 65 | ```python 66 | import time 67 | from dotenv import load_dotenv 68 | from openai import AzureOpenAI 69 | 70 | # Load the environment variables - These are secrets. 71 | load_dotenv() 72 | 73 | api_URI = os.getenv("OPENAI_URI") 74 | api_KEY = os.getenv("OPENAI_KEY") 75 | api_version = os.getenv("OPENAI_VERSION") 76 | deployment_name = os.getenv("OPENAI_DEPLOYMENT_NAME") 77 | 78 | # Create an OpenAI Azure client 79 | client = AzureOpenAI(api_key=api_key, 80 | api_version=api_version, 81 | azure_endpoint=api_endpoint) 82 | 83 | # Create an Asssitant with the code_interpreter tool 84 | assistant = client.beta.assistants.create( 85 | name="Math Tutor", 86 | instructions="You are a personal math tutor. Write and run code to answer math questions.", 87 | tools=[{"type": "code_interpreter"}], 88 | model=deployment_name, 89 | ) 90 | 91 | # Create a Thread 92 | thread = client.beta.threads.create() 93 | 94 | # Create a Message 95 | message = client.beta.threads.messages.create( 96 | thread_id=thread.id, 97 | role="user", 98 | content="I need to solve the equation `3x + 11 = 14`. Can you help me?", 99 | ) 100 | 101 | # Create a Run 102 | run = client.beta.threads.runs.create( 103 | thread_id=thread.id, 104 | assistant_id=assistant.id, 105 | instructions="Please address the user as Jane Doe. The user has a premium account.", 106 | ) 107 | 108 | # Check the status of a Run 109 | print("checking assistant status. ") 110 | while True: 111 | # Retrieve a Run by Thread and Run id 112 | run = client.beta.threads.runs.retrieve(thread_id=thread.id, run_id=run.id) 113 | 114 | 115 | if run.status == "completed": 116 | # Get the messages for a thread 117 | messages = client.beta.threads.messages.list(thread_id=thread.id) 118 | # Print the messages 119 | print("messages: ") 120 | for message in messages: 121 | assert message.content[0].type == "text" 122 | print({"role": message.role, "message": message.content[0].text.value}) 123 | # Dispose of the assistant 124 | client.beta.assistants.delete(assistant.id) 125 | # Dispose of the thread 126 | client.beta.threads.delete(thread.id) 127 | 128 | break 129 | elif run.status = "requires_action": 130 | # handle funcation calling and continue with the execution 131 | pass 132 | elif run.status = "expired" or run.status=="failed" or run.status=="cancelled": 133 | # run failed, expired, or was cancelled 134 | break 135 | else: 136 | print("in progress...") 137 | time.sleep(5) 138 | ``` 139 | 140 | ## Sample Assistants - Jupyter notebooks 141 | 142 | **Note:** Some Assistants in the notebooks folder have the ability to send emails and create appointments in Outlook. This functionality is achieved by using Function calling, which in turn triggers an HTTP Azure Logic App. The process of creating the Logic App is not covered in this guide. However, if you are interested in learning more about creating Azure Logic Apps, you can follow this [link](https://learn.microsoft.com/en-us/azure/connectors/connectors-create-api-office365-outlook?tabs=consumption). 143 | 144 | | Topic | Description | 145 | |----------------------|--------------------------------------------------| 146 | | [Math Tutor](math_tutor/assistant-math_tutor.ipynb) | Showcases the foundational concepts of Assistants such as Threads, Messages, Runs, Tools, and lifecycle management. | 147 | | [Financial Assistant](personal_finance/assistant-personal_finance.ipynb) | Function Calling with Yfinance to get latest stock prices. Summarization of user provided article. Extract country info from article, extract country, capital and other aspects, and call an API to get more information about each country. | 148 | | [Failed Banks](failed_banks/assistant-failed_banks.ipynb) | Using Assistant tools Code Interpreter and Function calling, this Assistant can get a CSV file, gather a list of failed banks by state, and generate a chart to visually represent the data. | 149 | | [Wind Farm](wind_farm/assistant-wind_farm.ipynb) | Utilizing Assistant tools such as the Code Interpreter and Function calling, this bot is capable of retrieving a CSV file that illustrates turbine wind speed, voltage, and the last maintenance date. It assists you in reviewing the file contents and aids in determining whether a specific turbine is in need of maintenance. | 150 | | [Sales Assistant](sales_assistant/assistant_sales.ipynb) | Showcases how you can create an Assistant adept at managing various tasks, such as handling relational data across different files and supporting multiple users simultaneously within the same Assistant across distinct threads. | 151 | | [Autogen Assistants](autogen/gpt_assistant_agent.ipynb) | Get started with Assistants on Autogen, with examples of basic assistants, code interpreter, retrieval and function calling, entirely managed by the Autogen framework. | 152 | | [File Search Assistant](file_search/assistant-file-search.ipynb) | Leverage File Search with Assistants, with examples of retrieval over a document containing benefits information. | 153 | 154 | ## Prerequisites 155 | 156 | - An [Azure subscription](https://azure.microsoft.com/en-us/free/) 157 | - Create and deploy an [Azure OpenAI Service Resource](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/create-resource) 158 | - In Azure create OpenAI endpoints for: 159 | - Model version: GPT 4 (1106-Preview) 160 | - API version: 2024-02-15-preview 161 | - Copy the deployment name, endpoint, and API key information 162 | - Clone this repo 163 | - Make sure the VS Code is installed as well as the VS Code Polyglot extension 164 | - Python 3.7+ 165 | - Create/rename `\.env` file and insert your Azure Open AI credentials: 166 | 167 | ``` 168 | OPENAI_URI=https://.openai.azure.com/ 169 | OPENAI_KEY= 170 | OPENAI_VERSION=2024-02-15-preview 171 | OPENAI_GPT_DEPLOYMENT= 172 | OPENAI_ADA_DEPLOYMENT= 173 | ``` 174 | 175 | - Open a Notebook and click play on the cells 176 | -------------------------------------------------------------------------------- /gen-ai/Assistants/notebooks/autogen/.gitignore: -------------------------------------------------------------------------------- 1 | data/* 2 | !data/.gitkeep 3 | -------------------------------------------------------------------------------- /gen-ai/Assistants/notebooks/autogen/README.md: -------------------------------------------------------------------------------- 1 | 2 | ## Autogen Assistant 3 | 4 | ### Overview 5 | 6 | Autogen supports Azure OpenAI Assistants, allowing you to manage the entire lifecycle of an Assistant, from creation to thread management to tool integration. 7 | 8 | ### Objective 9 | 10 | This notebook covers the basics of using Azure OpenAI Assistants in Autogen. The examples will show you how to configure and connect to your Azure OpenAI resource, create assistants, and leverage different tools, such as the Code Interpreter, File Search and Function Calling tools. 11 | 12 | ### Programming Languages 13 | 14 | - Python 15 | 16 | ### Estimated Runtime: 5 mins -------------------------------------------------------------------------------- /gen-ai/Assistants/notebooks/autogen/data/.gitkeep: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/AI-in-a-Box/52ff38a2333df2ae1d918f0db4da18e9e717ca57/gen-ai/Assistants/notebooks/autogen/data/.gitkeep -------------------------------------------------------------------------------- /gen-ai/Assistants/notebooks/failed_banks/README.md: -------------------------------------------------------------------------------- 1 | 2 | ## Failed Banks Information Assistant 3 | 4 | ### Overview 5 | 6 | Using Assistant tools Code Interpreter and Function calling, this bot can get a CSV file, gather a list of failed banks by state, and generate a chart to visually represent the data. 7 | 8 | ### Objective 9 | 10 | The objective of the provided notebook file is to create an Azure OpenAI Assistant named "Failed Banks Assistant" using the Azure OpenAI API. The assistant is designed to assist users in obtaining information about failed banks based on a provided CSV file. The assistant is equipped with the capability to process user messages, interact with the provided file, and respond with relevant information. 11 | 12 | ### Programming Languages 13 | 14 | - Python 15 | 16 | ### Estimated Runtime: 5 mins -------------------------------------------------------------------------------- /gen-ai/Assistants/notebooks/failed_banks/assistant-failed_banks.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Failed Banks Information Assistant" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "## Objective\n", 15 | "\n", 16 | "This notebook demonstrates the following:\n", 17 | "\n", 18 | "- Using Assistant tools Code Interpreter and Function calling, this bot can get a CSV file, gather a list of failed banks by state, and generate a chart to visually represent the data.\n", 19 | "\n", 20 | "This tutorial uses the following Azure AI services:\n", 21 | "- Access to Azure OpenAI Service - you can apply for access [here](https://aka.ms/oai/access)\n", 22 | "- Azure OpenAI service - you can create it from instructions [here](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/create-resource)\n", 23 | "- Azure OpenAI Studio - go to [https://oai.azure.com/](https://oai.azure.com/) to work with the Assistants API Playground\n", 24 | "- A connection to the Azure OpenAI Service with a [Key and Endpoint](https://learn.microsoft.com/en-us/azure/ai-services/openai/chatgpt-quickstart)\n", 25 | "\n", 26 | "Reference:\n", 27 | "- Learn more about how to use Assistants with our [How-to guide on Assistants](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/assistant)\n", 28 | "- [Assistants OpenAI Overview](https://platform.openai.com/docs/assistants/overview)" 29 | ] 30 | }, 31 | { 32 | "cell_type": "markdown", 33 | "metadata": {}, 34 | "source": [ 35 | "## Time\n", 36 | "\n", 37 | "You should expect to spend 5-10 minutes running this sample. " 38 | ] 39 | }, 40 | { 41 | "cell_type": "markdown", 42 | "metadata": {}, 43 | "source": [ 44 | "## About this example\n", 45 | "\n", 46 | "This sample demonstrates the creation of an Azure OpenAI Assistant named \"Failed Banks Assistant\" through the Azure OpenAI API. The assistant is tailored to aid users in retrieving information about failed banks from a CSV file, the assistant efficiently processes user messages, interacts with the designated file, and promptly delivers pertinent information in response.\n", 47 | "\n", 48 | "### Data\n", 49 | "This sample uses files from the folder [`data/`](./data/) in this repo. You can clone this repo or copy this folder to make sure you have access to these files when running the sample." 50 | ] 51 | }, 52 | { 53 | "cell_type": "markdown", 54 | "metadata": {}, 55 | "source": [ 56 | "## Before you begin" 57 | ] 58 | }, 59 | { 60 | "cell_type": "markdown", 61 | "metadata": {}, 62 | "source": [ 63 | "### Installation\n", 64 | "\n", 65 | "Install the following packages required to execute this notebook." 66 | ] 67 | }, 68 | { 69 | "cell_type": "code", 70 | "execution_count": null, 71 | "metadata": {}, 72 | "outputs": [], 73 | "source": [ 74 | "# Install the packages\n", 75 | "%pip install -r ../requirements.txt" 76 | ] 77 | }, 78 | { 79 | "cell_type": "markdown", 80 | "metadata": {}, 81 | "source": [ 82 | "### Parameters" 83 | ] 84 | }, 85 | { 86 | "cell_type": "code", 87 | "execution_count": null, 88 | "metadata": {}, 89 | "outputs": [], 90 | "source": [ 91 | "import os\n", 92 | "\n", 93 | "from dotenv import load_dotenv\n", 94 | "\n", 95 | "load_dotenv(\"../.env\") # make sure to have the .env file in the root directory of the project\n", 96 | "\n", 97 | "api_endpoint = os.getenv(\"OPENAI_URI\")\n", 98 | "api_key = os.getenv(\"OPENAI_KEY\")\n", 99 | "api_version = os.getenv(\"OPENAI_VERSION\")\n", 100 | "api_deployment_name = os.getenv(\"OPENAI_GPT_DEPLOYMENT\")\n", 101 | "email_URI = os.getenv(\"EMAIL_URI\")\n", 102 | "\n", 103 | "should_cleanup: bool = True" 104 | ] 105 | }, 106 | { 107 | "cell_type": "markdown", 108 | "metadata": {}, 109 | "source": [ 110 | "## Run this Example" 111 | ] 112 | }, 113 | { 114 | "cell_type": "markdown", 115 | "metadata": {}, 116 | "source": [ 117 | "### Load the required libraries" 118 | ] 119 | }, 120 | { 121 | "cell_type": "code", 122 | "execution_count": null, 123 | "metadata": {}, 124 | "outputs": [], 125 | "source": [ 126 | "import io\n", 127 | "import time\n", 128 | "from datetime import datetime\n", 129 | "from pathlib import Path\n", 130 | "from typing import Iterable\n", 131 | "\n", 132 | "from openai import AzureOpenAI\n", 133 | "from openai.types import FileObject\n", 134 | "from openai.types.beta.threads.text_content_block import TextContentBlock\n", 135 | "from openai.types.beta.threads.image_file_content_block import ImageFileContentBlock\n", 136 | "from PIL import Image" 137 | ] 138 | }, 139 | { 140 | "cell_type": "markdown", 141 | "metadata": {}, 142 | "source": [ 143 | "### Create an Azure OpenAI client" 144 | ] 145 | }, 146 | { 147 | "cell_type": "code", 148 | "execution_count": null, 149 | "metadata": {}, 150 | "outputs": [], 151 | "source": [ 152 | "client = AzureOpenAI(api_key=api_key, api_version=api_version, azure_endpoint=api_endpoint)" 153 | ] 154 | }, 155 | { 156 | "cell_type": "markdown", 157 | "metadata": {}, 158 | "source": [ 159 | "### Prepare the tools for function calling" 160 | ] 161 | }, 162 | { 163 | "cell_type": "code", 164 | "execution_count": null, 165 | "metadata": {}, 166 | "outputs": [], 167 | "source": [ 168 | "tools_list = [\n", 169 | " {\"type\": \"code_interpreter\"},\n", 170 | "]" 171 | ] 172 | }, 173 | { 174 | "cell_type": "code", 175 | "execution_count": null, 176 | "metadata": {}, 177 | "outputs": [], 178 | "source": [ 179 | "DATA_FOLDER = \"data/\"\n", 180 | "\n", 181 | "def upload_file(client: AzureOpenAI, path: str) -> FileObject:\n", 182 | " with Path(path).open(\"rb\") as f:\n", 183 | " return client.files.create(file=f, purpose=\"assistants\")\n", 184 | "\n", 185 | "arr = os.listdir(DATA_FOLDER)\n", 186 | "assistant_files = []\n", 187 | "for file in arr:\n", 188 | " filePath = DATA_FOLDER + file\n", 189 | " assistant_files.append(upload_file(client, filePath))\n", 190 | "\n", 191 | "file_ids = [file.id for file in assistant_files]" 192 | ] 193 | }, 194 | { 195 | "cell_type": "markdown", 196 | "metadata": {}, 197 | "source": [ 198 | "### Create an Assistant and a Thread" 199 | ] 200 | }, 201 | { 202 | "cell_type": "code", 203 | "execution_count": null, 204 | "metadata": {}, 205 | "outputs": [], 206 | "source": [ 207 | "assistant = client.beta.assistants.create(\n", 208 | " name=\"Failed Banks Assistant\",\n", 209 | " instructions=\"You are an assistant that can help find information about failed banks. \"\n", 210 | " + \"Use the provided file only.\",\n", 211 | " tools=tools_list,\n", 212 | " model=api_deployment_name,\n", 213 | " tool_resources={'code_interpreter':{'file_ids': file_ids}},\n", 214 | ")\n", 215 | "\n", 216 | "thread = client.beta.threads.create()" 217 | ] 218 | }, 219 | { 220 | "cell_type": "markdown", 221 | "metadata": {}, 222 | "source": [ 223 | "### Format and display the Assistant Messages for text and images" 224 | ] 225 | }, 226 | { 227 | "cell_type": "code", 228 | "execution_count": null, 229 | "metadata": {}, 230 | "outputs": [], 231 | "source": [ 232 | "def read_assistant_file(file_id:str):\n", 233 | " response_content = client.files.content(file_id)\n", 234 | " return response_content.read()\n", 235 | "\n", 236 | "def print_messages(messages) -> None:\n", 237 | " message_list = []\n", 238 | "\n", 239 | " # Get all the messages till the last user message\n", 240 | " for message in messages:\n", 241 | " message_list.append(message)\n", 242 | " if message.role == \"user\":\n", 243 | " break\n", 244 | "\n", 245 | " # Reverse the messages to show the last user message first\n", 246 | " message_list.reverse()\n", 247 | "\n", 248 | " # Print the user or Assistant messages or images\n", 249 | " for message in message_list:\n", 250 | " for item in message.content:\n", 251 | " # Determine the content type\n", 252 | " if isinstance(item, TextContentBlock):\n", 253 | " print(f\"{message.role}:\\n{item.text.value}\\n\")\n", 254 | " file_annotations = item.text.annotations\n", 255 | " if file_annotations:\n", 256 | " for annotation in file_annotations:\n", 257 | " file_id = annotation.file_path.file_id\n", 258 | " content = read_assistant_file(file_id)\n", 259 | " print(f\"Annotation Content:\\n{str(content)}\\n\")\n", 260 | " elif isinstance(item, ImageFileContentBlock):\n", 261 | " # Retrieve image from file id \n", 262 | " data_in_bytes = read_assistant_file(item.image_file.file_id)\n", 263 | " # Convert bytes to image\n", 264 | " readable_buffer = io.BytesIO(data_in_bytes)\n", 265 | " image = Image.open(readable_buffer)\n", 266 | " # Resize image to fit in terminal\n", 267 | " width, height = image.size\n", 268 | " image = image.resize((width // 2, height // 2), Image.LANCZOS)\n", 269 | " # Display image\n", 270 | " image.show()" 271 | ] 272 | }, 273 | { 274 | "cell_type": "markdown", 275 | "metadata": {}, 276 | "source": [ 277 | "### Process the user messages" 278 | ] 279 | }, 280 | { 281 | "cell_type": "code", 282 | "execution_count": null, 283 | "metadata": {}, 284 | "outputs": [], 285 | "source": [ 286 | "def process_prompt(prompt: str) -> None:\n", 287 | " client.beta.threads.messages.create(thread_id=thread.id, role=\"user\", content=prompt)\n", 288 | " run = client.beta.threads.runs.create(\n", 289 | " thread_id=thread.id,\n", 290 | " assistant_id=assistant.id,\n", 291 | " instructions=\"Please address the user as Jane Doe. The user has a premium account. Be assertive, accurate, and polite. Ask if the user has further questions. \"\n", 292 | " + \"The current date and time is: \"\n", 293 | " + datetime.now().strftime(\"%x %X\")\n", 294 | " + \". \",\n", 295 | " )\n", 296 | " print(\"processing ...\")\n", 297 | " while True:\n", 298 | " run = client.beta.threads.runs.retrieve(thread_id=thread.id, run_id=run.id)\n", 299 | " if run.status == \"completed\":\n", 300 | " # Handle completed\n", 301 | " messages = client.beta.threads.messages.list(thread_id=thread.id)\n", 302 | " print_messages(messages)\n", 303 | " break\n", 304 | " if run.status == \"failed\":\n", 305 | " messages = client.beta.threads.messages.list(thread_id=thread.id)\n", 306 | " answer = messages.data[0].content[0].text.value\n", 307 | " print(f\"Failed User:\\n{prompt}\\nAssistant:\\n{answer}\\n\")\n", 308 | " # Handle failed\n", 309 | " break\n", 310 | " if run.status == \"expired\":\n", 311 | " # Handle expired\n", 312 | " print(run)\n", 313 | " break\n", 314 | " if run.status == \"cancelled\":\n", 315 | " # Handle cancelled\n", 316 | " print(run)\n", 317 | " break\n", 318 | " if run.status == \"requires_action\":\n", 319 | " # Handle function calling and continue processing\n", 320 | " pass\n", 321 | " else:\n", 322 | " time.sleep(5)" 323 | ] 324 | }, 325 | { 326 | "cell_type": "markdown", 327 | "metadata": {}, 328 | "source": [ 329 | "### Process user requests" 330 | ] 331 | }, 332 | { 333 | "cell_type": "code", 334 | "execution_count": null, 335 | "metadata": {}, 336 | "outputs": [], 337 | "source": [ 338 | "process_prompt(\"Create a chart of failed banks by state.\")" 339 | ] 340 | }, 341 | { 342 | "cell_type": "code", 343 | "execution_count": null, 344 | "metadata": {}, 345 | "outputs": [], 346 | "source": [ 347 | "process_prompt(\"What was the last bank to fail?\")" 348 | ] 349 | }, 350 | { 351 | "cell_type": "code", 352 | "execution_count": null, 353 | "metadata": {}, 354 | "outputs": [], 355 | "source": [ 356 | "process_prompt(\"Which state has the most bank failures?\")" 357 | ] 358 | }, 359 | { 360 | "cell_type": "code", 361 | "execution_count": null, 362 | "metadata": {}, 363 | "outputs": [], 364 | "source": [ 365 | "process_prompt(\"Which banks failed in Florida in between 2020-2023?\")" 366 | ] 367 | }, 368 | { 369 | "cell_type": "markdown", 370 | "metadata": {}, 371 | "source": [ 372 | "### Cleaning up" 373 | ] 374 | }, 375 | { 376 | "cell_type": "code", 377 | "execution_count": null, 378 | "metadata": {}, 379 | "outputs": [], 380 | "source": [ 381 | "if should_cleanup:\n", 382 | " client.beta.assistants.delete(assistant.id)\n", 383 | " client.beta.threads.delete(thread.id)\n", 384 | " for file in assistant_files:\n", 385 | " client.files.delete(file.id)" 386 | ] 387 | }, 388 | { 389 | "cell_type": "code", 390 | "execution_count": null, 391 | "metadata": {}, 392 | "outputs": [], 393 | "source": [] 394 | } 395 | ], 396 | "metadata": { 397 | "kernelspec": { 398 | "display_name": "Python 3 (ipykernel)", 399 | "language": "python", 400 | "name": "python3" 401 | }, 402 | "language_info": { 403 | "codemirror_mode": { 404 | "name": "ipython", 405 | "version": 3 406 | }, 407 | "file_extension": ".py", 408 | "mimetype": "text/x-python", 409 | "name": "python", 410 | "nbconvert_exporter": "python", 411 | "pygments_lexer": "ipython3", 412 | "version": "3.11.9" 413 | } 414 | }, 415 | "nbformat": 4, 416 | "nbformat_minor": 4 417 | } 418 | -------------------------------------------------------------------------------- /gen-ai/Assistants/notebooks/file_search/.gitignore: -------------------------------------------------------------------------------- 1 | data/* 2 | !data/.gitkeep 3 | -------------------------------------------------------------------------------- /gen-ai/Assistants/notebooks/file_search/README.md: -------------------------------------------------------------------------------- 1 | 2 | ## Assistants File Search 3 | 4 | ### Overview 5 | 6 | With File Search, Azure Open AI manages document chunking, vectorization, indexing and retrieval to create an end-to-end chat experience. 7 | 8 | ### Objective 9 | 10 | This notebook covers the basics of using File Search in Azure OpenAI Assistants. The examples will show you how to configure and connect to your Azure OpenAI resource, create a file search-enabled assistant, and ask questions about the contents of your documents. 11 | 12 | ### Programming Languages 13 | 14 | - Python 15 | 16 | ### Estimated Runtime: 5 mins -------------------------------------------------------------------------------- /gen-ai/Assistants/notebooks/file_search/assistant-file-search.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# File Search Assistant" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "## Objective\n", 15 | "\n", 16 | "This notebook demonstrates the following:\n", 17 | "\n", 18 | "- Using Assistant File Search tool, this bot can answer questions about Contoso's employee benefits \n", 19 | "\n", 20 | "This tutorial uses the following Azure AI services:\n", 21 | "- Access to Azure OpenAI Service - you can apply for access [here](https://aka.ms/oai/access)\n", 22 | "- Azure OpenAI service - you can create it from instructions [here](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/create-resource)\n", 23 | "- Azure OpenAI Studio - go to [https://oai.azure.com/](https://oai.azure.com/) to work with the Assistants API Playground\n", 24 | "- A connection to the Azure OpenAI Service with a [Key and Endpoint](https://learn.microsoft.com/en-us/azure/ai-services/openai/chatgpt-quickstart)\n", 25 | "\n", 26 | "Reference:\n", 27 | "- Learn more about how to use Assistants with our [How-to guide on Assistants](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/assistant)\n", 28 | "- [Assistants OpenAI Overview](https://platform.openai.com/docs/assistants/overview)" 29 | ] 30 | }, 31 | { 32 | "cell_type": "markdown", 33 | "metadata": {}, 34 | "source": [ 35 | "## Time\n", 36 | "\n", 37 | "You should expect to spend 5-10 minutes running this sample. " 38 | ] 39 | }, 40 | { 41 | "cell_type": "markdown", 42 | "metadata": {}, 43 | "source": [ 44 | "## About this example\n", 45 | "\n", 46 | "This sample demonstrates the creation of an Azure OpenAI Assistant named \"Contoso Benefits Assistant\" through the Azure OpenAI API. The assistant is tailored to aid users in retrieving information about benefits for employees, the assistant automatically handles chunking, vectorization and retrieval on information, and promptly delivers pertinent information in response.\n", 47 | "\n", 48 | "### Data\n", 49 | "This sample will automatically download the benefits policy from a sample repository from Azure Samples." 50 | ] 51 | }, 52 | { 53 | "cell_type": "markdown", 54 | "metadata": {}, 55 | "source": [ 56 | "## Before you begin" 57 | ] 58 | }, 59 | { 60 | "cell_type": "markdown", 61 | "metadata": {}, 62 | "source": [ 63 | "### Installation\n", 64 | "\n", 65 | "Install the following packages required to execute this notebook." 66 | ] 67 | }, 68 | { 69 | "cell_type": "code", 70 | "execution_count": null, 71 | "metadata": {}, 72 | "outputs": [], 73 | "source": [ 74 | "# Install the packages\n", 75 | "%pip install -r ../requirements.txt" 76 | ] 77 | }, 78 | { 79 | "cell_type": "markdown", 80 | "metadata": {}, 81 | "source": [ 82 | "### Parameters" 83 | ] 84 | }, 85 | { 86 | "cell_type": "code", 87 | "execution_count": null, 88 | "metadata": {}, 89 | "outputs": [], 90 | "source": [ 91 | "import os\n", 92 | "\n", 93 | "from dotenv import load_dotenv\n", 94 | "\n", 95 | "load_dotenv(\"../.env\") # make sure to have the .env file in the root directory of the project\n", 96 | "\n", 97 | "api_endpoint = os.getenv(\"OPENAI_URI\")\n", 98 | "api_key = os.getenv(\"OPENAI_KEY\")\n", 99 | "api_version = os.getenv(\"OPENAI_VERSION\")\n", 100 | "api_deployment_name = os.getenv(\"OPENAI_GPT_DEPLOYMENT\")\n", 101 | "email_URI = os.getenv(\"EMAIL_URI\")\n", 102 | "\n", 103 | "should_cleanup: bool = True" 104 | ] 105 | }, 106 | { 107 | "cell_type": "markdown", 108 | "metadata": {}, 109 | "source": [ 110 | "## Run this Example" 111 | ] 112 | }, 113 | { 114 | "cell_type": "markdown", 115 | "metadata": {}, 116 | "source": [ 117 | "### Load the required libraries" 118 | ] 119 | }, 120 | { 121 | "cell_type": "code", 122 | "execution_count": null, 123 | "metadata": {}, 124 | "outputs": [], 125 | "source": [ 126 | "import io\n", 127 | "import time\n", 128 | "from datetime import datetime\n", 129 | "from pathlib import Path\n", 130 | "from typing import Iterable\n", 131 | "\n", 132 | "from openai import AzureOpenAI\n", 133 | "from openai.types import FileObject\n", 134 | "from openai.types.beta.threads.text_content_block import TextContentBlock\n", 135 | "from openai.types.beta.threads.image_file_content_block import ImageFileContentBlock" 136 | ] 137 | }, 138 | { 139 | "cell_type": "markdown", 140 | "metadata": {}, 141 | "source": [ 142 | "### Create an Azure OpenAI client" 143 | ] 144 | }, 145 | { 146 | "cell_type": "code", 147 | "execution_count": null, 148 | "metadata": {}, 149 | "outputs": [], 150 | "source": [ 151 | "client = AzureOpenAI(api_key=api_key, api_version=api_version, azure_endpoint=api_endpoint)" 152 | ] 153 | }, 154 | { 155 | "cell_type": "markdown", 156 | "metadata": {}, 157 | "source": [ 158 | "### Prepare the tools for function calling" 159 | ] 160 | }, 161 | { 162 | "cell_type": "code", 163 | "execution_count": null, 164 | "metadata": {}, 165 | "outputs": [], 166 | "source": [ 167 | "tools_list = [\n", 168 | " {\"type\": \"file_search\"},\n", 169 | "]" 170 | ] 171 | }, 172 | { 173 | "cell_type": "code", 174 | "execution_count": null, 175 | "metadata": {}, 176 | "outputs": [], 177 | "source": [ 178 | "!curl -L \"https://github.com/Azure-Samples/cognitive-services-sample-data-files/raw/master/openai/contoso_benefits_document_example.pdf\" > data/contoso_benefits_document_example.pdf" 179 | ] 180 | }, 181 | { 182 | "cell_type": "code", 183 | "execution_count": null, 184 | "metadata": {}, 185 | "outputs": [], 186 | "source": [ 187 | "# Create a vector store caled \"Contoso Benefits\"\n", 188 | "vector_store = client.beta.vector_stores.create(name=\"Contoso Benefits\")\n", 189 | " \n", 190 | "# Ready the files for upload to OpenAI\n", 191 | "file_paths = [\"data/contoso_benefits_document_example.pdf\"]\n", 192 | "file_streams = [open(path, \"rb\") for path in file_paths]\n", 193 | " \n", 194 | "# Use the upload and poll SDK helper to upload the files, add them to the vector store,\n", 195 | "# and poll the status of the file batch for completion.\n", 196 | "file_batch = client.beta.vector_stores.file_batches.upload_and_poll(\n", 197 | " vector_store_id=vector_store.id, files=file_streams\n", 198 | ")" 199 | ] 200 | }, 201 | { 202 | "cell_type": "markdown", 203 | "metadata": {}, 204 | "source": [ 205 | "### Create an Assistant and a Thread" 206 | ] 207 | }, 208 | { 209 | "cell_type": "code", 210 | "execution_count": null, 211 | "metadata": {}, 212 | "outputs": [], 213 | "source": [ 214 | "assistant = client.beta.assistants.create(\n", 215 | " name=\"File Search Assistant\",\n", 216 | " instructions=\"You are an assistant that can help find information about Contoso's benefits policies. \"\n", 217 | " + \"Use information retrieved from the policy only.\",\n", 218 | " tools=tools_list,\n", 219 | " model=api_deployment_name,\n", 220 | " tool_resources={\"file_search\": {\"vector_store_ids\": [vector_store.id]}},\n", 221 | ")\n", 222 | "\n", 223 | "thread = client.beta.threads.create()" 224 | ] 225 | }, 226 | { 227 | "cell_type": "markdown", 228 | "metadata": {}, 229 | "source": [ 230 | "### Format and display the Assistant Messages for text and images" 231 | ] 232 | }, 233 | { 234 | "cell_type": "code", 235 | "execution_count": null, 236 | "metadata": {}, 237 | "outputs": [], 238 | "source": [ 239 | "def read_assistant_file(file_id:str):\n", 240 | " response_content = client.files.content(file_id)\n", 241 | " return response_content.read()\n", 242 | "\n", 243 | "def print_messages(messages) -> None:\n", 244 | " message_list = []\n", 245 | "\n", 246 | " # Get all the messages till the last user message\n", 247 | " for message in messages:\n", 248 | " message_list.append(message)\n", 249 | " if message.role == \"user\":\n", 250 | " break\n", 251 | "\n", 252 | " # Reverse the messages to show the last user message first\n", 253 | " message_list.reverse()\n", 254 | "\n", 255 | " # Print the user or Assistant messages or images\n", 256 | " for message in message_list:\n", 257 | " for item in message.content:\n", 258 | " # Determine the content type\n", 259 | " if isinstance(item, TextContentBlock):\n", 260 | " print(f\"{message.role}:\\n{item.text.value}\\n\")\n", 261 | " file_annotations = item.text.annotations\n", 262 | " if file_annotations:\n", 263 | " for annotation in file_annotations:\n", 264 | " file_id = annotation.file_path.file_id\n", 265 | " content = read_assistant_file(file_id)\n", 266 | " print(f\"Annotation Content:\\n{str(content)}\\n\")\n", 267 | " elif isinstance(item, ImageFileContentBlock):\n", 268 | " # Retrieve image from file id \n", 269 | " data_in_bytes = read_assistant_file(item.image_file.file_id)\n", 270 | " # Convert bytes to image\n", 271 | " readable_buffer = io.BytesIO(data_in_bytes)\n", 272 | " image = Image.open(readable_buffer)\n", 273 | " # Resize image to fit in terminal\n", 274 | " width, height = image.size\n", 275 | " image = image.resize((width // 2, height // 2), Image.LANCZOS)\n", 276 | " # Display image\n", 277 | " image.show()" 278 | ] 279 | }, 280 | { 281 | "cell_type": "markdown", 282 | "metadata": {}, 283 | "source": [ 284 | "### Process the user messages" 285 | ] 286 | }, 287 | { 288 | "cell_type": "code", 289 | "execution_count": null, 290 | "metadata": {}, 291 | "outputs": [], 292 | "source": [ 293 | "def process_prompt(prompt: str) -> None:\n", 294 | " client.beta.threads.messages.create(thread_id=thread.id, role=\"user\", content=prompt)\n", 295 | " run = client.beta.threads.runs.create(\n", 296 | " thread_id=thread.id,\n", 297 | " assistant_id=assistant.id,\n", 298 | " instructions=\"Please address the user as Jane Doe. The user has a premium account. Be assertive, accurate, and polite. Ask if the user has further questions. \"\n", 299 | " + \"The current date and time is: \"\n", 300 | " + datetime.now().strftime(\"%x %X\")\n", 301 | " + \". \",\n", 302 | " )\n", 303 | " print(\"processing ...\")\n", 304 | " while True:\n", 305 | " run = client.beta.threads.runs.retrieve(thread_id=thread.id, run_id=run.id)\n", 306 | " if run.status == \"completed\":\n", 307 | " # Handle completed\n", 308 | " messages = client.beta.threads.messages.list(thread_id=thread.id)\n", 309 | " print_messages(messages)\n", 310 | " break\n", 311 | " if run.status == \"failed\":\n", 312 | " messages = client.beta.threads.messages.list(thread_id=thread.id)\n", 313 | " answer = messages.data[0].content[0].text.value\n", 314 | " print(f\"Failed User:\\n{prompt}\\nAssistant:\\n{answer}\\n\")\n", 315 | " # Handle failed\n", 316 | " break\n", 317 | " if run.status == \"expired\":\n", 318 | " # Handle expired\n", 319 | " print(run)\n", 320 | " break\n", 321 | " if run.status == \"cancelled\":\n", 322 | " # Handle cancelled\n", 323 | " print(run)\n", 324 | " break\n", 325 | " if run.status == \"requires_action\":\n", 326 | " # Handle function calling and continue processing\n", 327 | " pass\n", 328 | " else:\n", 329 | " time.sleep(5)" 330 | ] 331 | }, 332 | { 333 | "cell_type": "markdown", 334 | "metadata": {}, 335 | "source": [ 336 | "### Process user requests" 337 | ] 338 | }, 339 | { 340 | "cell_type": "code", 341 | "execution_count": null, 342 | "metadata": {}, 343 | "outputs": [], 344 | "source": [ 345 | "process_prompt(\"Does Contoso offer dental insurance?\")" 346 | ] 347 | }, 348 | { 349 | "cell_type": "code", 350 | "execution_count": null, 351 | "metadata": {}, 352 | "outputs": [], 353 | "source": [ 354 | "process_prompt(\"How do the costs for each plan compare?\")" 355 | ] 356 | }, 357 | { 358 | "cell_type": "markdown", 359 | "metadata": {}, 360 | "source": [ 361 | "### Cleaning up" 362 | ] 363 | }, 364 | { 365 | "cell_type": "code", 366 | "execution_count": null, 367 | "metadata": {}, 368 | "outputs": [], 369 | "source": [ 370 | "if should_cleanup:\n", 371 | " client.beta.assistants.delete(assistant.id)\n", 372 | " client.beta.threads.delete(thread.id)\n", 373 | " client.beta.vector_stores.delete(vector_store.id)" 374 | ] 375 | }, 376 | { 377 | "cell_type": "code", 378 | "execution_count": null, 379 | "metadata": {}, 380 | "outputs": [], 381 | "source": [] 382 | } 383 | ], 384 | "metadata": { 385 | "kernelspec": { 386 | "display_name": "Python 3 (ipykernel)", 387 | "language": "python", 388 | "name": "python3" 389 | }, 390 | "language_info": { 391 | "codemirror_mode": { 392 | "name": "ipython", 393 | "version": 3 394 | }, 395 | "file_extension": ".py", 396 | "mimetype": "text/x-python", 397 | "name": "python", 398 | "nbconvert_exporter": "python", 399 | "pygments_lexer": "ipython3", 400 | "version": "3.11.9" 401 | } 402 | }, 403 | "nbformat": 4, 404 | "nbformat_minor": 4 405 | } 406 | -------------------------------------------------------------------------------- /gen-ai/Assistants/notebooks/file_search/data/.gitkeep: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/AI-in-a-Box/52ff38a2333df2ae1d918f0db4da18e9e717ca57/gen-ai/Assistants/notebooks/file_search/data/.gitkeep -------------------------------------------------------------------------------- /gen-ai/Assistants/notebooks/function_calling/README.md: -------------------------------------------------------------------------------- 1 | ## Assistants function calling with Bing Search 2 | 3 | ### Overview 4 | 5 | In this notebook, we'll show how you can use the [Bing Search APIs](https://www.microsoft.com/bing/apis/llm) and [function calling](https://learn.microsoft.com/azure/ai-services/openai/how-to/function-calling?tabs=python) to ground Azure OpenAI models on data from the web. This is a great way to give the model access to up to date data from the web. 6 | 7 | This sample will be useful to developers and for data scientists who want to learn about function calling capabilities and search based grounding. 8 | 9 | ### Objective 10 | 11 | The main objective of this tutorial is to demonstrate how to use function calling with LLMs for enhanced functionality. 12 | 13 | This tutorial uses the following services: 14 | 15 | - Azure OpenAI Service - you can apply for access [here](https://go.microsoft.com/fwlink/?linkid=2222006) 16 | - Bing Search - [Bing Search resource](https://learn.microsoft.com/en-us/bing/search-apis/bing-web-search/create-bing-search-service-resource) 17 | 18 | ### Programming Languages 19 | 20 | - Python 21 | 22 | ### Estimated Runtime: 10 mins -------------------------------------------------------------------------------- /gen-ai/Assistants/notebooks/function_calling/assistants_function_calling_with_bing_search.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Assistants function calling with Bing Search\n", 8 | "In this notebook, we'll show how you can use the [Bing Search APIs](https://www.microsoft.com/bing/apis/llm) and [function calling](https://learn.microsoft.com/azure/ai-services/openai/how-to/function-calling?tabs=python) to ground Azure OpenAI models on data from the web. This is a great way to give the model access to up to date data from the web.\n", 9 | "\n", 10 | "You'll need to create a [Bing Search resource](https://learn.microsoft.com/en-us/bing/search-apis/bing-web-search/create-bing-search-service-resource) before you begin." 11 | ] 12 | }, 13 | { 14 | "cell_type": "markdown", 15 | "metadata": {}, 16 | "source": [ 17 | "### Time\n", 18 | "\n", 19 | "You should expect to spend 10 minutes running this sample." 20 | ] 21 | }, 22 | { 23 | "cell_type": "markdown", 24 | "metadata": {}, 25 | "source": [ 26 | "### Before you begin\n", 27 | "#### Installation\n", 28 | "The following packages are required to execute this notebook." 29 | ] 30 | }, 31 | { 32 | "cell_type": "code", 33 | "execution_count": null, 34 | "metadata": {}, 35 | "outputs": [], 36 | "source": [ 37 | "# Install the packages\\\n", 38 | "%pip install requests openai~=1.10" 39 | ] 40 | }, 41 | { 42 | "cell_type": "markdown", 43 | "metadata": {}, 44 | "source": [ 45 | "### Parameters\n", 46 | "Update the following config to include details of your Azure OpenAI and Bing Search resources" 47 | ] 48 | }, 49 | { 50 | "cell_type": "code", 51 | "execution_count": null, 52 | "metadata": {}, 53 | "outputs": [], 54 | "source": [ 55 | "azure_endpoint = \"https://.openai.azure.com\"\n", 56 | "api_version = \"2024-02-15-preview\"\n", 57 | "aoai_api_key = \"\"\n", 58 | "deployment_name = \"\"\n", 59 | "bing_search_subscription_key = \"\"\n", 60 | "bing_search_url = \"https://api.bing.microsoft.com/v7.0/search\"" 61 | ] 62 | }, 63 | { 64 | "cell_type": "markdown", 65 | "metadata": {}, 66 | "source": [ 67 | "### Define a function to call the Bing Search APIs\n", 68 | "To learn more about using the Bing Search APIs with Azure OpenAI, see [Bing Search APIs, with your LLM](https://learn.microsoft.com/bing/search-apis/bing-web-search/use-display-requirements-llm)." 69 | ] 70 | }, 71 | { 72 | "cell_type": "code", 73 | "execution_count": null, 74 | "metadata": {}, 75 | "outputs": [], 76 | "source": [ 77 | "import json\n", 78 | "import requests\n", 79 | "import time\n", 80 | "\n", 81 | "from openai import AzureOpenAI\n", 82 | "from pathlib import Path\n", 83 | "from typing import Optional\n", 84 | "\n", 85 | "\n", 86 | "def search(query: str) -> list:\n", 87 | " \"\"\"\n", 88 | " Perform a bing search against the given query\n", 89 | "\n", 90 | " @param query: Search query\n", 91 | " @return: List of search results\n", 92 | "\n", 93 | " \"\"\"\n", 94 | " headers = {\"Ocp-Apim-Subscription-Key\": bing_search_subscription_key}\n", 95 | " params = {\"q\": query, \"textDecorations\": False}\n", 96 | " response = requests.get(bing_search_url, headers=headers, params=params)\n", 97 | " response.raise_for_status()\n", 98 | " search_results = response.json()\n", 99 | "\n", 100 | " output = []\n", 101 | "\n", 102 | " for result in search_results[\"webPages\"][\"value\"]:\n", 103 | " output.append({\"title\": result[\"name\"], \"link\": result[\"url\"], \"snippet\": result[\"snippet\"]})\n", 104 | "\n", 105 | " return json.dumps(output)" 106 | ] 107 | }, 108 | { 109 | "cell_type": "code", 110 | "execution_count": null, 111 | "metadata": {}, 112 | "outputs": [], 113 | "source": [ 114 | "search(\"where will the 2032 olympics be held?\")" 115 | ] 116 | }, 117 | { 118 | "cell_type": "markdown", 119 | "metadata": {}, 120 | "source": [ 121 | "### Get things running end to end\n", 122 | "In the following cells, we will define some functions essential for assistants with function calling. All these functions come together in our final cell, where we will define a new web search assistant, give it instructions on its functionality and ask a question." 123 | ] 124 | }, 125 | { 126 | "cell_type": "code", 127 | "execution_count": null, 128 | "metadata": {}, 129 | "outputs": [], 130 | "source": [ 131 | "def poll_run_till_completion(\n", 132 | " client: AzureOpenAI,\n", 133 | " thread_id: str,\n", 134 | " run_id: str,\n", 135 | " available_functions: dict,\n", 136 | " verbose: bool,\n", 137 | " max_steps: int = 10,\n", 138 | " wait: int = 3,\n", 139 | ") -> None:\n", 140 | " \"\"\"\n", 141 | " Poll a run until it is completed or failed or exceeds a certain number of iterations (MAX_STEPS)\n", 142 | " with a preset wait in between polls\n", 143 | "\n", 144 | " @param client: OpenAI client\n", 145 | " @param thread_id: Thread ID\n", 146 | " @param run_id: Run ID\n", 147 | " @param assistant_id: Assistant ID\n", 148 | " @param verbose: Print verbose output\n", 149 | " @param max_steps: Maximum number of steps to poll\n", 150 | " @param wait: Wait time in seconds between polls\n", 151 | "\n", 152 | " \"\"\"\n", 153 | "\n", 154 | " if (client is None and thread_id is None) or run_id is None:\n", 155 | " print(\"Client, Thread ID and Run ID are required.\")\n", 156 | " return\n", 157 | " try:\n", 158 | " cnt = 0\n", 159 | " while cnt < max_steps:\n", 160 | " run = client.beta.threads.runs.retrieve(thread_id=thread_id, run_id=run_id)\n", 161 | " if verbose:\n", 162 | " print(\"Poll {}: {}\".format(cnt, run.status))\n", 163 | " cnt += 1\n", 164 | " if run.status == \"requires_action\":\n", 165 | " tool_responses = []\n", 166 | " if (\n", 167 | " run.required_action.type == \"submit_tool_outputs\"\n", 168 | " and run.required_action.submit_tool_outputs.tool_calls is not None\n", 169 | " ):\n", 170 | " tool_calls = run.required_action.submit_tool_outputs.tool_calls\n", 171 | "\n", 172 | " for call in tool_calls:\n", 173 | " if call.type == \"function\":\n", 174 | " if call.function.name not in available_functions:\n", 175 | " raise Exception(\"Function requested by the model does not exist\")\n", 176 | " function_to_call = available_functions[call.function.name]\n", 177 | " tool_response = function_to_call(**json.loads(call.function.arguments))\n", 178 | " tool_responses.append({\"tool_call_id\": call.id, \"output\": tool_response})\n", 179 | "\n", 180 | " run = client.beta.threads.runs.submit_tool_outputs(\n", 181 | " thread_id=thread_id, run_id=run.id, tool_outputs=tool_responses\n", 182 | " )\n", 183 | " if run.status == \"failed\":\n", 184 | " print(\"Run failed.\")\n", 185 | " break\n", 186 | " if run.status == \"completed\":\n", 187 | " break\n", 188 | " time.sleep(wait)\n", 189 | "\n", 190 | " except Exception as e:\n", 191 | " print(e)" 192 | ] 193 | }, 194 | { 195 | "cell_type": "code", 196 | "execution_count": null, 197 | "metadata": {}, 198 | "outputs": [], 199 | "source": [ 200 | "def create_message(\n", 201 | " client: AzureOpenAI,\n", 202 | " thread_id: str,\n", 203 | " role: str = \"\",\n", 204 | " content: str = \"\",\n", 205 | " file_ids: Optional[list] = None,\n", 206 | " metadata: Optional[dict] = None,\n", 207 | " message_id: Optional[str] = None,\n", 208 | ") -> any:\n", 209 | " \"\"\"\n", 210 | " Create a message in a thread using the client.\n", 211 | "\n", 212 | " @param client: OpenAI client\n", 213 | " @param thread_id: Thread ID\n", 214 | " @param role: Message role (user or assistant)\n", 215 | " @param content: Message content\n", 216 | " @param file_ids: Message file IDs\n", 217 | " @param metadata: Message metadata\n", 218 | " @param message_id: Message ID\n", 219 | " @return: Message object\n", 220 | "\n", 221 | " \"\"\"\n", 222 | " if metadata is None:\n", 223 | " metadata = {}\n", 224 | " if file_ids is None:\n", 225 | " file_ids = []\n", 226 | "\n", 227 | " if client is None:\n", 228 | " print(\"Client parameter is required.\")\n", 229 | " return None\n", 230 | "\n", 231 | " if thread_id is None:\n", 232 | " print(\"Thread ID is required.\")\n", 233 | " return None\n", 234 | "\n", 235 | " try:\n", 236 | " if message_id is not None:\n", 237 | " return client.beta.threads.messages.retrieve(thread_id=thread_id, message_id=message_id)\n", 238 | "\n", 239 | " if file_ids is not None and len(file_ids) > 0 and metadata is not None and len(metadata) > 0:\n", 240 | " return client.beta.threads.messages.create(\n", 241 | " thread_id=thread_id, role=role, content=content, file_ids=file_ids, metadata=metadata\n", 242 | " )\n", 243 | "\n", 244 | " if file_ids is not None and len(file_ids) > 0:\n", 245 | " return client.beta.threads.messages.create(\n", 246 | " thread_id=thread_id, role=role, content=content, file_ids=file_ids\n", 247 | " )\n", 248 | "\n", 249 | " if metadata is not None and len(metadata) > 0:\n", 250 | " return client.beta.threads.messages.create(\n", 251 | " thread_id=thread_id, role=role, content=content, metadata=metadata\n", 252 | " )\n", 253 | "\n", 254 | " return client.beta.threads.messages.create(thread_id=thread_id, role=role, content=content)\n", 255 | "\n", 256 | " except Exception as e:\n", 257 | " print(e)\n", 258 | " return None" 259 | ] 260 | }, 261 | { 262 | "cell_type": "code", 263 | "execution_count": null, 264 | "metadata": {}, 265 | "outputs": [], 266 | "source": [ 267 | "def retrieve_and_print_messages(\n", 268 | " client: AzureOpenAI, thread_id: str, verbose: bool, out_dir: Optional[str] = None\n", 269 | ") -> any:\n", 270 | " \"\"\"\n", 271 | " Retrieve a list of messages in a thread and print it out with the query and response\n", 272 | "\n", 273 | " @param client: OpenAI client\n", 274 | " @param thread_id: Thread ID\n", 275 | " @param verbose: Print verbose output\n", 276 | " @param out_dir: Output directory to save images\n", 277 | " @return: Messages object\n", 278 | "\n", 279 | " \"\"\"\n", 280 | "\n", 281 | " if client is None and thread_id is None:\n", 282 | " print(\"Client and Thread ID are required.\")\n", 283 | " return None\n", 284 | " try:\n", 285 | " messages = client.beta.threads.messages.list(thread_id=thread_id)\n", 286 | " display_role = {\"user\": \"User query\", \"assistant\": \"Assistant response\"}\n", 287 | "\n", 288 | " prev_role = None\n", 289 | "\n", 290 | " if verbose:\n", 291 | " print(\"\\n\\nCONVERSATION:\")\n", 292 | " for md in reversed(messages.data):\n", 293 | " if prev_role == \"assistant\" and md.role == \"user\" and verbose:\n", 294 | " print(\"------ \\n\")\n", 295 | "\n", 296 | " for mc in md.content:\n", 297 | " # Check if valid text field is present in the mc object\n", 298 | " if mc.type == \"text\":\n", 299 | " txt_val = mc.text.value\n", 300 | " # Check if valid image field is present in the mc object\n", 301 | " elif mc.type == \"image_file\":\n", 302 | " image_data = client.files.content(mc.image_file.file_id)\n", 303 | " if out_dir is not None:\n", 304 | " out_dir_path = Path(out_dir)\n", 305 | " if out_dir_path.exists():\n", 306 | " image_path = out_dir_path / (mc.image_file.file_id + \".png\")\n", 307 | " with image_path.open(\"wb\") as f:\n", 308 | " f.write(image_data.read())\n", 309 | "\n", 310 | " if verbose:\n", 311 | " if prev_role == md.role:\n", 312 | " print(txt_val)\n", 313 | " else:\n", 314 | " print(\"{}:\\n{}\".format(display_role[md.role], txt_val))\n", 315 | " prev_role = md.role\n", 316 | " return messages\n", 317 | " except Exception as e:\n", 318 | " print(e)\n", 319 | " return None" 320 | ] 321 | }, 322 | { 323 | "cell_type": "code", 324 | "execution_count": null, 325 | "metadata": {}, 326 | "outputs": [], 327 | "source": [ 328 | "name = \"websearch-assistant\"\n", 329 | "instructions = \"\"\"You are an assistant designed to help people answer questions.\n", 330 | "\n", 331 | "You have access to query the web using Bing Search. You should call bing search whenever a question requires up to date information or could benefit from web data.\n", 332 | "\"\"\"\n", 333 | "\n", 334 | "message = {\"role\": \"user\", \"content\": \"How tall is mount rainier?\"}\n", 335 | "\n", 336 | "\n", 337 | "tools = [\n", 338 | " {\n", 339 | " \"type\": \"function\",\n", 340 | " \"function\": {\n", 341 | " \"name\": \"search_bing\",\n", 342 | " \"description\": \"Searches bing to get up-to-date information from the web.\",\n", 343 | " \"parameters\": {\n", 344 | " \"type\": \"object\",\n", 345 | " \"properties\": {\n", 346 | " \"query\": {\n", 347 | " \"type\": \"string\",\n", 348 | " \"description\": \"The search query\",\n", 349 | " }\n", 350 | " },\n", 351 | " \"required\": [\"query\"],\n", 352 | " },\n", 353 | " },\n", 354 | " }\n", 355 | "]\n", 356 | "\n", 357 | "available_functions = {\"search_bing\": search}\n", 358 | "verbose_output = True\n", 359 | "\n", 360 | "client = AzureOpenAI(api_key=aoai_api_key, api_version=api_version, azure_endpoint=azure_endpoint)\n", 361 | "\n", 362 | "assistant = client.beta.assistants.create(\n", 363 | " name=name, description=\"\", instructions=instructions, tools=tools, model=deployment_name\n", 364 | ")\n", 365 | "\n", 366 | "thread = client.beta.threads.create()\n", 367 | "create_message(client, thread.id, message[\"role\"], message[\"content\"])\n", 368 | "\n", 369 | "\n", 370 | "run = client.beta.threads.runs.create(thread_id=thread.id, assistant_id=assistant.id, instructions=instructions)\n", 371 | "poll_run_till_completion(\n", 372 | " client=client, thread_id=thread.id, run_id=run.id, available_functions=available_functions, verbose=verbose_output\n", 373 | ")\n", 374 | "messages = retrieve_and_print_messages(client=client, thread_id=thread.id, verbose=verbose_output)" 375 | ] 376 | } 377 | ], 378 | "metadata": { 379 | "kernelspec": { 380 | "display_name": "base", 381 | "language": "python", 382 | "name": "python3" 383 | }, 384 | "language_info": { 385 | "codemirror_mode": { 386 | "name": "ipython", 387 | "version": 3 388 | }, 389 | "file_extension": ".py", 390 | "mimetype": "text/x-python", 391 | "name": "python", 392 | "nbconvert_exporter": "python", 393 | "pygments_lexer": "ipython3" 394 | } 395 | }, 396 | "nbformat": 4, 397 | "nbformat_minor": 2 398 | } 399 | -------------------------------------------------------------------------------- /gen-ai/Assistants/notebooks/math_tutor/README.md: -------------------------------------------------------------------------------- 1 | 2 | ## Math Tutor Assistant 3 | 4 | ### Overview 5 | 6 | Showcases the foundational concepts of Assistants such as Threads, Messages, Runs, Tools, and lifecycle management. 7 | 8 | ### Objective 9 | 10 | The objective of the provided notebook file is to create an Azure OpenAI Assistant named "Math Tutor" using the Azure OpenAI API. The assistant is designed to function as a personal math tutor, capable of answering math questions through code interpretation. The script initiates a conversation with the assistant, guiding it through various mathematical queries and scenarios to showcase its capabilities. 11 | 12 | ### Programming Languages 13 | 14 | - Python 15 | 16 | ### Estimated Runtime: 5 mins -------------------------------------------------------------------------------- /gen-ai/Assistants/notebooks/math_tutor/assistant-math_tutor.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Math Tutor Assistant" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "## Objective\n", 15 | "\n", 16 | "This notebook demonstrates the following:\n", 17 | "\n", 18 | "- Showcases the foundational concepts of Assistants such as Threads, Messages, Runs, Tools, and lifecycle management.\n", 19 | "\n", 20 | "This tutorial uses the following Azure AI services:\n", 21 | "- Access to Azure OpenAI Service - you can apply for access [here](https://aka.ms/oai/access)\n", 22 | "- Azure OpenAI service - you can create it from instructions [here](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/create-resource)\n", 23 | "- Azure OpenAI Studio - go to [https://oai.azure.com/](https://oai.azure.com/) to work with the Assistants API Playground\n", 24 | "- A connection to the Azure OpenAI Service with a [Key and Endpoint](https://learn.microsoft.com/en-us/azure/ai-services/openai/chatgpt-quickstart)\n", 25 | "\n", 26 | "Reference:\n", 27 | "- Learn more about how to use Assistants with our [How-to guide on Assistants](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/assistant)\n", 28 | "- [Assistants OpenAI Overview](https://platform.openai.com/docs/assistants/overview)\n", 29 | "- [Github-OpenAI Python/examples/Assistant demo notebook](https://github.com/openai/openai-python/blob/main/examples/assistant.py)" 30 | ] 31 | }, 32 | { 33 | "cell_type": "markdown", 34 | "metadata": {}, 35 | "source": [ 36 | "## Time\n", 37 | "\n", 38 | "You should expect to spend 5-10 minutes running this sample." 39 | ] 40 | }, 41 | { 42 | "cell_type": "markdown", 43 | "metadata": {}, 44 | "source": [ 45 | "## About this example\n", 46 | "\n", 47 | "This sample shows users how to create an Azure OpenAI Assistant named \"Math Tutor\" using the Azure OpenAI API. The assistant is designed to function as a personal math tutor, capable of answering math questions through code interpretation. The script initiates a conversation with the assistant, guiding it through various mathematical queries and scenarios to showcase its capabilities.\n", 48 | "\n", 49 | "This sample provides developers with a clear demonstration of how to leverage the core concepts of the Assistants API into their projects, highlighting its simplicity and effectiveness in leveraging foundational concepts." 50 | ] 51 | }, 52 | { 53 | "cell_type": "markdown", 54 | "metadata": {}, 55 | "source": [ 56 | "## Before you begin" 57 | ] 58 | }, 59 | { 60 | "cell_type": "markdown", 61 | "metadata": {}, 62 | "source": [ 63 | "### Installation\n", 64 | "\n", 65 | "Install the following packages required to execute this notebook. \n", 66 | "\n" 67 | ] 68 | }, 69 | { 70 | "cell_type": "code", 71 | "execution_count": null, 72 | "metadata": {}, 73 | "outputs": [], 74 | "source": [ 75 | "# Install the packages\n", 76 | "%pip install -r ../requirements.txt" 77 | ] 78 | }, 79 | { 80 | "cell_type": "markdown", 81 | "metadata": {}, 82 | "source": [ 83 | "### Parameters" 84 | ] 85 | }, 86 | { 87 | "cell_type": "code", 88 | "execution_count": null, 89 | "metadata": {}, 90 | "outputs": [], 91 | "source": [ 92 | "import os\n", 93 | "\n", 94 | "from dotenv import load_dotenv\n", 95 | "\n", 96 | "load_dotenv(\"../.env\") # make sure to have the .env file in the root directory of the project\n", 97 | "\n", 98 | "api_endpoint = os.getenv(\"OPENAI_URI\")\n", 99 | "api_key = os.getenv(\"OPENAI_KEY\")\n", 100 | "api_version = os.getenv(\"OPENAI_VERSION\")\n", 101 | "api_deployment_name = os.getenv(\"OPENAI_GPT_DEPLOYMENT\")\n", 102 | "\n", 103 | "should_cleanup: bool = True" 104 | ] 105 | }, 106 | { 107 | "cell_type": "markdown", 108 | "metadata": {}, 109 | "source": [ 110 | "## Run this Example" 111 | ] 112 | }, 113 | { 114 | "cell_type": "markdown", 115 | "metadata": {}, 116 | "source": [ 117 | "### Load the required libraries" 118 | ] 119 | }, 120 | { 121 | "cell_type": "code", 122 | "execution_count": null, 123 | "metadata": {}, 124 | "outputs": [], 125 | "source": [ 126 | "import io\n", 127 | "import time\n", 128 | "from datetime import datetime\n", 129 | "from typing import Iterable\n", 130 | "\n", 131 | "from openai import AzureOpenAI\n", 132 | "from openai.types.beta.threads.text_content_block import TextContentBlock\n", 133 | "from openai.types.beta.threads.image_file_content_block import ImageFileContentBlock\n", 134 | "from PIL import Image" 135 | ] 136 | }, 137 | { 138 | "cell_type": "markdown", 139 | "metadata": {}, 140 | "source": [ 141 | "### Create an Azure OpenAI client" 142 | ] 143 | }, 144 | { 145 | "cell_type": "code", 146 | "execution_count": null, 147 | "metadata": {}, 148 | "outputs": [], 149 | "source": [ 150 | "client = AzureOpenAI(api_key=api_key, api_version=api_version, azure_endpoint=api_endpoint)" 151 | ] 152 | }, 153 | { 154 | "cell_type": "markdown", 155 | "metadata": {}, 156 | "source": [ 157 | "### Create an Assistant and a Thread" 158 | ] 159 | }, 160 | { 161 | "cell_type": "code", 162 | "execution_count": null, 163 | "metadata": {}, 164 | "outputs": [], 165 | "source": [ 166 | "assistant = client.beta.assistants.create(\n", 167 | " name=\"Math Tutor\",\n", 168 | " instructions=\"You are a personal math tutor. Write and run code to answer math questions.\",\n", 169 | " tools=[{\"type\": \"code_interpreter\"}],\n", 170 | " model=api_deployment_name,\n", 171 | ")\n", 172 | "\n", 173 | "thread = client.beta.threads.create()" 174 | ] 175 | }, 176 | { 177 | "cell_type": "markdown", 178 | "metadata": {}, 179 | "source": [ 180 | "### Format and display the Assistant Messages for text and images" 181 | ] 182 | }, 183 | { 184 | "cell_type": "code", 185 | "execution_count": null, 186 | "metadata": {}, 187 | "outputs": [], 188 | "source": [ 189 | "def read_assistant_file(file_id:str):\n", 190 | " response_content = client.files.content(file_id)\n", 191 | " return response_content.read()\n", 192 | "\n", 193 | "def print_messages(messages) -> None:\n", 194 | " message_list = []\n", 195 | "\n", 196 | " # Get all the messages till the last user message\n", 197 | " for message in messages:\n", 198 | " message_list.append(message)\n", 199 | " if message.role == \"user\":\n", 200 | " break\n", 201 | "\n", 202 | " # Reverse the messages to show the last user message first\n", 203 | " message_list.reverse()\n", 204 | "\n", 205 | " # Print the user or Assistant messages or images\n", 206 | " for message in message_list:\n", 207 | " for item in message.content:\n", 208 | " # Determine the content type\n", 209 | " if isinstance(item, TextContentBlock):\n", 210 | " print(f\"{message.role}:\\n{item.text.value}\\n\")\n", 211 | " file_annotations = item.text.annotations\n", 212 | " if file_annotations:\n", 213 | " for annotation in file_annotations:\n", 214 | " file_id = annotation.file_path.file_id\n", 215 | " content = read_assistant_file(file_id)\n", 216 | " print(f\"Annotation Content:\\n{str(content)}\\n\")\n", 217 | " elif isinstance(item, ImageFileContentBlock):\n", 218 | " # Retrieve image from file id \n", 219 | " data_in_bytes = read_assistant_file(item.image_file.file_id)\n", 220 | " # Convert bytes to image\n", 221 | " readable_buffer = io.BytesIO(data_in_bytes)\n", 222 | " image = Image.open(readable_buffer)\n", 223 | " # Resize image to fit in terminal\n", 224 | " width, height = image.size\n", 225 | " image = image.resize((width // 2, height // 2), Image.LANCZOS)\n", 226 | " # Display image\n", 227 | " image.show()" 228 | ] 229 | }, 230 | { 231 | "cell_type": "markdown", 232 | "metadata": {}, 233 | "source": [ 234 | "### Process the user messages" 235 | ] 236 | }, 237 | { 238 | "cell_type": "code", 239 | "execution_count": null, 240 | "metadata": {}, 241 | "outputs": [], 242 | "source": [ 243 | "def process_prompt(prompt: str) -> None:\n", 244 | " client.beta.threads.messages.create(thread_id=thread.id, role=\"user\", content=prompt)\n", 245 | " run = client.beta.threads.runs.create(\n", 246 | " thread_id=thread.id,\n", 247 | " assistant_id=assistant.id,\n", 248 | " instructions=\"Please address the user as Jane Doe. The user has a premium account. Be assertive, accurate, and polite. Ask if the user has further questions. \"\n", 249 | " + \"The current date and time is: \"\n", 250 | " + datetime.now().strftime(\"%x %X\")\n", 251 | " + \". \",\n", 252 | " )\n", 253 | " print(\"processing ...\")\n", 254 | " while True:\n", 255 | " run = client.beta.threads.runs.retrieve(thread_id=thread.id, run_id=run.id)\n", 256 | " if run.status == \"completed\":\n", 257 | " # Handle completed\n", 258 | " messages = client.beta.threads.messages.list(thread_id=thread.id)\n", 259 | " print_messages(messages)\n", 260 | " break\n", 261 | " if run.status == \"failed\":\n", 262 | " messages = client.beta.threads.messages.list(thread_id=thread.id)\n", 263 | " answer = messages.data[0].content[0].text.value\n", 264 | " print(f\"Failed User:\\n{prompt}\\nAssistant:\\n{answer}\\n\")\n", 265 | " # Handle failed\n", 266 | " break\n", 267 | " if run.status == \"expired\":\n", 268 | " # Handle expired\n", 269 | " print(run)\n", 270 | " break\n", 271 | " if run.status == \"cancelled\":\n", 272 | " # Handle cancelled\n", 273 | " print(run)\n", 274 | " break\n", 275 | " if run.status == \"requires_action\":\n", 276 | " # Handle function calling and continue processing\n", 277 | " pass\n", 278 | " else:\n", 279 | " time.sleep(5)" 280 | ] 281 | }, 282 | { 283 | "cell_type": "markdown", 284 | "metadata": {}, 285 | "source": [ 286 | "### Have a conversation with the Assistant" 287 | ] 288 | }, 289 | { 290 | "cell_type": "code", 291 | "execution_count": null, 292 | "metadata": {}, 293 | "outputs": [], 294 | "source": [ 295 | "process_prompt(\"What is the linear equation when two (x,y) points are (1,1) and (5,10)?\")" 296 | ] 297 | }, 298 | { 299 | "cell_type": "code", 300 | "execution_count": null, 301 | "metadata": {}, 302 | "outputs": [], 303 | "source": [ 304 | "process_prompt(\"I need to solve the equation `3x + 11 = 14`. Can you help me?\")" 305 | ] 306 | }, 307 | { 308 | "cell_type": "code", 309 | "execution_count": null, 310 | "metadata": {}, 311 | "outputs": [], 312 | "source": [ 313 | "process_prompt(\"\"\"x=r*cos(u)sin(v), y=r*sin(u)sin(v), r=2+sin(7*u+5*v) for 0 str: 26 | prompt_template = """system: 27 | You are an agent that can determine intent from the following list of intents and return the intent that best matches the user's question or statement. 28 | 29 | List of intents: 30 | 31 | OtherAgent: any other question 32 | 33 | user: 34 | 35 | 36 | Output in ONE word.""" 37 | 38 | intents = "" 39 | for reg_agent in self.registered_agents: 40 | intents += f"{reg_agent.intent}: {reg_agent.intent_desc}\n" 41 | 42 | full_prompt = prompt_template.replace( 43 | "", intents).replace("", prompt) 44 | completion = self.client.chat.completions.create( 45 | model=self.settings.model_deployment, 46 | messages=[ 47 | { 48 | "role": "user", 49 | "content": full_prompt, 50 | }, 51 | ], 52 | max_tokens=2, 53 | temperature=0.1 54 | ) 55 | try: 56 | intent = completion.choices[0].message.content 57 | return intent 58 | except: 59 | return "Unknown" 60 | 61 | def process_for_intent(self, user_name, user_id, prompt: str) -> str: 62 | intent = self.__semantic_intent(prompt) 63 | print(f'Intent: {intent}') 64 | if intent is None or intent == "OtherAgent" or intent == "Unknown": 65 | completion = self.client.chat.completions.create( 66 | model=self.settings.model_deployment, 67 | messages=[ 68 | { 69 | "role": "user", 70 | "content": prompt, 71 | } 72 | ] 73 | ) 74 | print(completion.choices[0].message.content) 75 | else: 76 | for registered_agent in self.registered_agents: 77 | if registered_agent.intent == intent: 78 | return registered_agent.agent.process_prompt(user_name, user_id, prompt) 79 | -------------------------------------------------------------------------------- /gen-ai/Assistants/notebooks/sales_copilot/AgentRegistration.py: -------------------------------------------------------------------------------- 1 | from AgentSettings import AgentSettings 2 | from openai import AzureOpenAI 3 | from ArgumentException import ArgumentExceptionError 4 | from AssistantAgent import AssistantAgent 5 | 6 | 7 | class AgentRegistration: 8 | """This function is to hold the agent registration information""" 9 | 10 | def __init__(self, settings=None, client=None, intent: str = None, intent_desc: str = None, agent: AssistantAgent = None): 11 | self.settings = settings 12 | self.client = client 13 | self.agent = agent 14 | self.intent = intent 15 | self.intent_desc = intent_desc 16 | 17 | if intent is None: 18 | raise ArgumentExceptionError("intent parameter is missing") 19 | if intent_desc is None: 20 | raise ArgumentExceptionError("intent_desc parameter is missing") 21 | 22 | if settings is None: 23 | self.settings = AgentSettings() 24 | 25 | if client is None: 26 | client = AzureOpenAI( 27 | api_key=self.settings.api_key, 28 | api_version=self.settings.api_version, 29 | azure_endpoint=self.settings.api_endpoint) 30 | -------------------------------------------------------------------------------- /gen-ai/Assistants/notebooks/sales_copilot/AgentSettings.py: -------------------------------------------------------------------------------- 1 | from dotenv import load_dotenv 2 | import os 3 | 4 | 5 | class AgentSettings: 6 | def __init__(self): 7 | load_dotenv() 8 | self.api_endpoint = os.getenv("OPENAI_URI") 9 | self.api_key = os.getenv("OPENAI_KEY") 10 | self.api_version = os.getenv("OPENAI_VERSION") 11 | self.model_deployment = os.getenv("OPENAI_GPT_DEPLOYMENT") 12 | self.email_URI = os.getenv("EMAIL_URI") 13 | -------------------------------------------------------------------------------- /gen-ai/Assistants/notebooks/sales_copilot/ArgumentException.py: -------------------------------------------------------------------------------- 1 | class ArgumentExceptionError(Exception): 2 | def __init__(self, msg): 3 | self.msg = msg 4 | -------------------------------------------------------------------------------- /gen-ai/Assistants/notebooks/sales_copilot/AssistantAgent.py: -------------------------------------------------------------------------------- 1 | from typing import Iterable 2 | import os 3 | import io 4 | import time 5 | from datetime import datetime 6 | from pathlib import Path 7 | 8 | 9 | from AgentSettings import AgentSettings 10 | 11 | from openai.types.beta.threads.text_content_block import TextContentBlock 12 | from openai.types.beta.threads.image_file_content_block import ImageFileContentBlock 13 | from openai.types.beta.threads.messages import MessageFile 14 | from openai.types import FileObject 15 | from PIL import Image 16 | from ArgumentException import ArgumentExceptionError 17 | 18 | 19 | class AssistantAgent: 20 | def __init__(self, settings, client, name, instructions, data_folder, tools_list, keep_state: bool = False, fn_calling_delegate=None): 21 | if name is None: 22 | raise ArgumentExceptionError("name parameter missing") 23 | if instructions is None: 24 | raise ArgumentExceptionError("instructions parameter missing") 25 | if tools_list is None: 26 | raise ArgumentExceptionError("tools_list parameter missing") 27 | 28 | self.assistant = None 29 | self.settings = settings 30 | self.client = client 31 | self.name = name 32 | self.instructions = instructions 33 | self.data_folder = data_folder 34 | self.tools_list = tools_list 35 | self.fn_calling_delegate = fn_calling_delegate 36 | self.keep_state = keep_state 37 | self.ai_threads = [] 38 | self.ai_files = [] 39 | self.file_ids = [] 40 | self.get_agent() 41 | 42 | def upload_file(self, path: str) -> FileObject: 43 | print(path) 44 | with Path(path).open("rb") as f: 45 | return self.client.files.create(file=f, purpose="assistants") 46 | 47 | def upload_all_files(self): 48 | files_in_folder = os.listdir(self.data_folder) 49 | local_file_list = [] 50 | for file in files_in_folder: 51 | filePath = self.data_folder + file 52 | assistant_file = self.upload_file(filePath) 53 | self.ai_files.append(assistant_file) 54 | local_file_list.append(assistant_file) 55 | self.file_ids = [file.id for file in local_file_list] 56 | 57 | def get_agent(self): 58 | if self.data_folder is not None: 59 | self.upload_all_files() 60 | self.assistant = self.client.beta.assistants.create( 61 | name=self.name, # "Sales Assistant", 62 | # "You are a sales assistant. You can answer questions related to customer orders.", 63 | instructions=self.instructions, 64 | tools=self.tools_list, 65 | model=self.settings.model_deployment, 66 | file_ids=self.file_ids 67 | ) 68 | else: 69 | self.assistant = self.client.beta.assistants.create( 70 | name=self.name, # "Sales Assistant", 71 | # "You are a sales assistant. You can answer questions related to customer orders.", 72 | instructions=self.instructions, 73 | tools=self.tools_list, 74 | model=self.settings.model_deployment 75 | ) 76 | 77 | def process_prompt(self, user_name: str, user_id: str, prompt: str) -> None: 78 | 79 | # if keep_state: 80 | # thread_id = check_if_thread_exists(user_id) 81 | 82 | # # If a thread doesn't exist, create one and store it 83 | # if thread_id is None: 84 | # print(f"Creating new thread for {name} with user_id {user_id}") 85 | # thread = self.client.beta.threads.create() 86 | # store_thread(user_id, thread) 87 | # thread_id = thread.id 88 | # # Otherwise, retrieve the existing thread 89 | # else: 90 | # print( 91 | # f"Retrieving existing thread for {name} with user_id {user_id}") 92 | # thread = self.client.beta.threads.retrieve(thread_id) 93 | # add_thread(thread) 94 | # else: 95 | thread = self.client.beta.threads.create() 96 | 97 | self.client.beta.threads.messages.create( 98 | thread_id=thread.id, role="user", content=prompt) 99 | 100 | run = self.client.beta.threads.runs.create( 101 | thread_id=thread.id, 102 | assistant_id=self.assistant.id, 103 | instructions="Please address the user as Jane Doe. The user has a premium account. Be assertive, accurate, and polite. Ask if the user has further questions. Do not provide explanations for the answers." 104 | + "The current date and time is: " 105 | + datetime.now().strftime("%x %X") 106 | + ". ", 107 | ) 108 | 109 | print("processing ...") 110 | while True: 111 | run = self.client.beta.threads.runs.retrieve( 112 | thread_id=thread.id, run_id=run.id) 113 | if run.status == "completed": 114 | # Handle completed 115 | messages = self.client.beta.threads.messages.list( 116 | thread_id=thread.id) 117 | self.print_messages(user_name, messages) 118 | break 119 | if run.status == "failed": 120 | messages = self.client.beta.threads.messages.list( 121 | thread_id=thread.id) 122 | self.print_messages(user_name, messages) 123 | # Handle failed 124 | break 125 | if run.status == "expired": 126 | # Handle expired 127 | break 128 | if run.status == "cancelled": 129 | # Handle cancelled 130 | break 131 | if run.status == "requires_action": 132 | if self.fn_calling_delegate: 133 | self.fn_calling_delegate(self.client, thread, run) 134 | else: 135 | time.sleep(5) 136 | if not self.keep_state: 137 | self.client.beta.threads.delete(thread.id) 138 | print("Deleted thread: ", thread.id) 139 | 140 | def read_assistant_file(self, file_id: str): 141 | response_content = self.client.files.content(file_id) 142 | return response_content.read() 143 | 144 | def print_messages(self, name: str, messages: Iterable[MessageFile]) -> None: 145 | message_list = [] 146 | 147 | # Get all the messages till the last user message 148 | for message in messages: 149 | message_list.append(message) 150 | if message.role == "user": 151 | break 152 | 153 | # Reverse the messages to show the last user message first 154 | message_list.reverse() 155 | 156 | # Print the user or Assistant messages or images 157 | for message in message_list: 158 | for item in message.content: 159 | # Determine the content type 160 | if isinstance(item, TextContentBlock): 161 | if message.role == "user": 162 | print(f"user: {name}:\n{item.text.value}\n") 163 | else: 164 | print(f"{message.role}:\n{item.text.value}\n") 165 | file_annotations = item.text.annotations 166 | if file_annotations: 167 | for annotation in file_annotations: 168 | file_id = annotation.file_path.file_id 169 | content = self.read_assistant_file(file_id) 170 | print(f"Annotation Content:\n{str(content)}\n") 171 | elif isinstance(item, ImageFileContentBlock): 172 | # Retrieve image from file id 173 | data_in_bytes = self.read_assistant_file( 174 | item.image_file.file_id) 175 | # Convert bytes to image 176 | readable_buffer = io.BytesIO(data_in_bytes) 177 | image = Image.open(readable_buffer) 178 | # Resize image to fit in terminal 179 | width, height = image.size 180 | image = image.resize( 181 | (width // 2, height // 2), Image.LANCZOS) 182 | # Display image 183 | image.show() 184 | 185 | def cleanup(self): 186 | print(self.client.beta.assistants.delete(self.assistant.id)) 187 | print("Deleting: ", len(self.ai_threads), " threads.") 188 | for thread in self.ai_threads: 189 | print(self.client.beta.threads.delete(thread.id)) 190 | print("Deleting: ", len(self.ai_files), " files.") 191 | for file in self.ai_files: 192 | print(self.client.files.delete(file.id)) 193 | -------------------------------------------------------------------------------- /gen-ai/Assistants/notebooks/sales_copilot/README.md: -------------------------------------------------------------------------------- 1 | # Two Agent Copilot
    with Agent Proxy 2 | 3 | This is a sample of two Assistants being used as agents in a complex Copilot solution. 4 | 5 | ## Diagram 6 | 7 | ![Two agent Assistant Diagram](images/proxy-two-agent-diagram.png) 8 | 9 | ## Agents 10 | 11 | - **Sales Agent:** It is implemented with an Assistants API that can perform calculations and can provide the user information customer, sellers, customer orders, and inventory. 12 | - **Trading Agent:** It is implemented with an Assistant API with function calling and code interpreter to get the latest stock prices and can provide the user with up to date prices and perform portfolio calculations. 13 | 14 | ## Orchestration 15 | 16 | - **Agent Proxy:** This agent coordinates which agent to call depending on user's intent. 17 | 18 | ## Possible messages handling options 19 | 20 | 1. No handling: a la LLM model 21 | 2. Manual handling: user write logic to coordinate messages between two agent 22 | 3. Hybrid: One agent keeps state the other just used as LLM 23 | -------------------------------------------------------------------------------- /gen-ai/Assistants/notebooks/sales_copilot/data/customers.csv: -------------------------------------------------------------------------------- 1 | CustomerID,Name,State 2 | 1000,John Doe,FL 3 | 1001,Jane Smith,NY 4 | 1002,Michael Johnson,TX 5 | 1003,Sarah Williams,CA 6 | 1004,Robert Brown,IL 7 | 1005,Emily Davis,OH 8 | 1006,David Wilson,FL 9 | 1007,Jennifer Taylor,GA 10 | 1008,Christopher Anderson,MI 11 | 1009,Lisa Martinez,NC 12 | -------------------------------------------------------------------------------- /gen-ai/Assistants/notebooks/sales_copilot/data/inventory.csv: -------------------------------------------------------------------------------- 1 | ItemID,Description 2 | 1000,Intel i7 Latop 3 | 1001,AMD Ryzen 7 Desktop -------------------------------------------------------------------------------- /gen-ai/Assistants/notebooks/sales_copilot/data/orders.csv: -------------------------------------------------------------------------------- 1 | OrderID,CustomerID,SellerID,ItemID,Qty,Price 2 | 1000,1000,1000,1000,10,150.25 3 | 1001,1001,1001,1001,20,250.50 -------------------------------------------------------------------------------- /gen-ai/Assistants/notebooks/sales_copilot/data/sellers.csv: -------------------------------------------------------------------------------- 1 | SellerID,Name,State 2 | 1000,Carlos Lopez,CA 3 | 1001,Deepak Usman,CA 4 | -------------------------------------------------------------------------------- /gen-ai/Assistants/notebooks/sales_copilot/images/proxy-two-agent-diagram.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/AI-in-a-Box/52ff38a2333df2ae1d918f0db4da18e9e717ca57/gen-ai/Assistants/notebooks/sales_copilot/images/proxy-two-agent-diagram.png -------------------------------------------------------------------------------- /gen-ai/Assistants/notebooks/sales_copilot/sales_agent.py: -------------------------------------------------------------------------------- 1 | from openai import AzureOpenAI 2 | 3 | from AgentSettings import AgentSettings 4 | from AssistantAgent import AssistantAgent 5 | 6 | tools_list = [ 7 | {"type": "code_interpreter"} 8 | ] 9 | 10 | DATA_FOLDER = "data/" 11 | 12 | 13 | def get_agent(settings=None, client=None): 14 | """This function creates a Sales Assistants API agent""" 15 | 16 | if settings is None: 17 | settings = AgentSettings() 18 | 19 | if client is None: 20 | client = AzureOpenAI( 21 | api_key=settings.api_key, 22 | api_version=settings.api_version, 23 | azure_endpoint=settings.api_endpoint) 24 | 25 | agent = AssistantAgent(settings, 26 | client, 27 | "Sales Assistant", 28 | "You are an Assistant that can help answer questions and perform calculations related to customers, customer orders, inventory, and sellers with the provided CSV files.", 29 | DATA_FOLDER, 30 | tools_list) 31 | return agent 32 | -------------------------------------------------------------------------------- /gen-ai/Assistants/notebooks/sales_copilot/trading_agent.py: -------------------------------------------------------------------------------- 1 | from AgentSettings import AgentSettings 2 | from AssistantAgent import AssistantAgent 3 | from openai import AzureOpenAI 4 | import yfinance as yf 5 | import requests 6 | import html 7 | 8 | email_URI = None 9 | 10 | 11 | def send_logic_apps_email(to: str, content: str) -> None: 12 | """This function sends an email using an Http endpoint implemented with Logic Apps""" 13 | # In this demo email was implemented using a Logic App with an HTTP trigger and M365 email action 14 | # Here are instructions on how to create an email endpoint using Logic Apps 15 | # https://learn.microsoft.com/en-us/azure/app-service/tutorial-send-email?tabs=python 16 | try: 17 | json_payload = {"to": to, "content": html.unescape(content)} 18 | headers = {"Content-Type": "application/json"} 19 | response = requests.post(email_URI, json=json_payload, headers=headers) 20 | if response.status_code == 202: 21 | print("Email sent to: " + json_payload["to"]) 22 | except: 23 | print("Failed to send email via Logic Apps") 24 | 25 | 26 | def get_stock_price(symbol: str) -> float: 27 | """This function gets the latest stock price based on a ticker symbol""" 28 | stock = yf.Ticker(symbol) 29 | return stock.history(period="1d")["Close"].iloc[-1] 30 | 31 | 32 | def call_functions(client, thread, run): 33 | """This function is delegate function that is called when the Assistant needs to make function calls""" 34 | print("Function Calling") 35 | required_actions = run.required_action.submit_tool_outputs.model_dump() 36 | print(required_actions) 37 | tool_outputs = [] 38 | import json 39 | for action in required_actions["tool_calls"]: 40 | func_name = action['function']['name'] 41 | arguments = json.loads(action['function']['arguments']) 42 | 43 | if func_name == "get_stock_price": 44 | output = get_stock_price(symbol=arguments['symbol']) 45 | tool_outputs.append({ 46 | "tool_call_id": action['id'], 47 | "output": output 48 | }) 49 | elif func_name == "send_email": 50 | print("Sending email...") 51 | email_to = arguments['to'] 52 | email_content = arguments['content'] 53 | send_logic_apps_email(email_to, email_content) 54 | 55 | tool_outputs.append({ 56 | "tool_call_id": action['id'], 57 | "output": "Email sent" 58 | }) 59 | else: 60 | raise ValueError(f"Unknown function: {func_name}") 61 | 62 | print("Submitting outputs back to the Assistant...") 63 | client.beta.threads.runs.submit_tool_outputs( 64 | thread_id=thread.id, 65 | run_id=run.id, 66 | tool_outputs=tool_outputs 67 | ) 68 | 69 | 70 | tools_list = [ 71 | {"type": "code_interpreter"}, 72 | {"type": "function", 73 | "function": { 74 | "name": "get_stock_price", 75 | "description": "Retrieve the latest closing price of a stock using its ticker symbol.", 76 | "parameters": { 77 | "type": "object", 78 | "properties": { 79 | "symbol": { 80 | "type": "string", 81 | "description": "The ticker symbol of the stock" 82 | } 83 | }, 84 | "required": ["symbol"] 85 | } 86 | } 87 | }, 88 | ] 89 | 90 | DATA_FOLDER = None 91 | 92 | 93 | def get_agent(settings=None, client=None): 94 | """This function creates a trading Assistants API agent""" 95 | 96 | if settings is None: 97 | settings = AgentSettings() 98 | 99 | if client is None: 100 | client = AzureOpenAI( 101 | api_key=settings.api_key, 102 | api_version=settings.api_version, 103 | azure_endpoint=settings.api_endpoint) 104 | 105 | email_URI = settings.email_URI 106 | 107 | agent = AssistantAgent(settings, 108 | client, 109 | "Trading Agent", "You are an agent that can help get the latest stock prices and perform investment related calculations.", 110 | DATA_FOLDER, tools_list, fn_calling_delegate=call_functions) 111 | 112 | return agent 113 | -------------------------------------------------------------------------------- /gen-ai/Assistants/notebooks/sales_copilot/two_agent_copilot_proxy.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Two Agent Copilot
    with Agent Proxy\n", 8 | "\n", 9 | "## Objective\n", 10 | "\n", 11 | "This is a sample of two Assistants being used as agents in a complex Copilot solution.\n", 12 | "\n", 13 | "## Agents\n", 14 | "\n", 15 | "- **Sales Agent:** It is implemented with an Assistants API that can perform calculations and can provide the user information customer, sellers, customer orders, and inventory.\n", 16 | "- **Trading Agent:** It is implemented with an Assistant API with function calling and code interpreter to get the latest stock prices and can provide the user with up to date prices and perform portfolio calculations.\n", 17 | "\n", 18 | "## Orchestration\n", 19 | "\n", 20 | "- **Agent Proxy:** This agent coordinates which agent to call depending on user's intent.\n", 21 | "\n", 22 | "## Possible messages handling options\n", 23 | "\n", 24 | "1. No handling: a la LLM model\n", 25 | "2. Manual handling: user write logic to coordinate messages between two agent\n", 26 | "3. Hybrid: One agent keeps state the other just used as LLM\n", 27 | "\n", 28 | "\n", 29 | "This tutorial uses the following Azure AI services:\n", 30 | "- Access to Azure OpenAI Service - you can apply for access [here](https://aka.ms/oai/access)\n", 31 | "- Azure OpenAI service - you can create it from instructions [here](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/create-resource)\n", 32 | "- Azure OpenAI Studio - go to [https://oai.azure.com/](https://oai.azure.com/) to work with the Assistants API Playground\n", 33 | "- A connection to the Azure OpenAI Service with a [Key and Endpoint](https://learn.microsoft.com/en-us/azure/ai-services/openai/chatgpt-quickstart)\n", 34 | "\n", 35 | "Reference:\n", 36 | "- Learn more about how to use Assistants with our [How-to guide on Assistants](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/assistant)\n", 37 | "- [Assistants OpenAI Overview](https://platform.openai.com/docs/assistants/overview)\n", 38 | "\n", 39 | "## Time\n", 40 | "\n", 41 | "You should expect to spend 15-30 minutes running this sample. \n", 42 | "\n", 43 | "## About this example\n", 44 | "\n", 45 | "The objective of the provided Python file is to create an multi-agent solution called \"Sales Called\" using the Azure OpenAI API. The assistant is designed to act act as a bot that can answer questions related to sales information including queries such as. Based on the user's intent the copilot will direct a user's request to:\n", 46 | "\n", 47 | "- A sales agent\n", 48 | "- A trading agent\n", 49 | "- Or handle the request itself\n", 50 | "\n", 51 | "### Data\n", 52 | "\n", 53 | "This sample uses files in the `data/` folder. You can clone this repo or copy this folder to make sure you have access to these files when running the sample.\n", 54 | "\n", 55 | "\n", 56 | "### Load the required libraries" 57 | ] 58 | }, 59 | { 60 | "cell_type": "code", 61 | "execution_count": null, 62 | "metadata": {}, 63 | "outputs": [], 64 | "source": [ 65 | "import sales_agent\n", 66 | "import trading_agent\n", 67 | "from AgentProxy import AgentProxy\n", 68 | "from AgentRegistration import AgentRegistration\n", 69 | "from AgentSettings import AgentSettings\n", 70 | "from openai import AzureOpenAI" 71 | ] 72 | }, 73 | { 74 | "cell_type": "markdown", 75 | "metadata": {}, 76 | "source": [ 77 | "### Load the environment variables" 78 | ] 79 | }, 80 | { 81 | "cell_type": "code", 82 | "execution_count": null, 83 | "metadata": {}, 84 | "outputs": [], 85 | "source": [ 86 | "settings = AgentSettings()" 87 | ] 88 | }, 89 | { 90 | "cell_type": "code", 91 | "execution_count": null, 92 | "metadata": {}, 93 | "outputs": [], 94 | "source": [ 95 | "client = AzureOpenAI(api_key=settings.api_key,\n", 96 | " api_version=settings.api_version,\n", 97 | " azure_endpoint=settings.api_endpoint)" 98 | ] 99 | }, 100 | { 101 | "cell_type": "markdown", 102 | "metadata": {}, 103 | "source": [ 104 | "### Create the AzureOpenAI client" 105 | ] 106 | }, 107 | { 108 | "cell_type": "code", 109 | "execution_count": null, 110 | "metadata": {}, 111 | "outputs": [], 112 | "source": [ 113 | "trading_agent = trading_agent.get_agent(settings,client)" 114 | ] 115 | }, 116 | { 117 | "cell_type": "code", 118 | "execution_count": null, 119 | "metadata": {}, 120 | "outputs": [], 121 | "source": [ 122 | "sales_agent = sales_agent.get_agent(settings,client)" 123 | ] 124 | }, 125 | { 126 | "cell_type": "code", 127 | "execution_count": null, 128 | "metadata": {}, 129 | "outputs": [], 130 | "source": [ 131 | "sales_agent_registration = AgentRegistration(settings, client, \"SalesIntent\",\"questions or operations related to calculations, customers, sales, orders, inventory, items, and sellers.\",sales_agent)" 132 | ] 133 | }, 134 | { 135 | "cell_type": "code", 136 | "execution_count": null, 137 | "metadata": {}, 138 | "outputs": [], 139 | "source": [ 140 | "trading_agent_registration = AgentRegistration(settings, client, \"TradingIntent\",\"questions or operations related to trading, stocks, investments, and financial markets.\",trading_agent)" 141 | ] 142 | }, 143 | { 144 | "cell_type": "code", 145 | "execution_count": null, 146 | "metadata": {}, 147 | "outputs": [], 148 | "source": [ 149 | "proxy = AgentProxy(settings, client, [sales_agent_registration, trading_agent_registration])" 150 | ] 151 | }, 152 | { 153 | "cell_type": "markdown", 154 | "metadata": {}, 155 | "source": [ 156 | "### Have a conversation with the Assistant" 157 | ] 158 | }, 159 | { 160 | "cell_type": "code", 161 | "execution_count": null, 162 | "metadata": {}, 163 | "outputs": [], 164 | "source": [ 165 | "proxy.process_for_intent(\"Suzan\", \"user_123\", \"What is the speed of light?\")" 166 | ] 167 | }, 168 | { 169 | "cell_type": "code", 170 | "execution_count": null, 171 | "metadata": {}, 172 | "outputs": [], 173 | "source": [ 174 | "proxy.process_for_intent(\"John\", \"user_123\", \"Chart the products sold by State?\")" 175 | ] 176 | }, 177 | { 178 | "cell_type": "code", 179 | "execution_count": null, 180 | "metadata": {}, 181 | "outputs": [], 182 | "source": [ 183 | "proxy.process_for_intent(\"John\", \"user_123\", \"What is the current MSFT price?\")" 184 | ] 185 | }, 186 | { 187 | "cell_type": "markdown", 188 | "metadata": {}, 189 | "source": [ 190 | "### Cleanup" 191 | ] 192 | }, 193 | { 194 | "cell_type": "code", 195 | "execution_count": null, 196 | "metadata": {}, 197 | "outputs": [], 198 | "source": [ 199 | "sales_agent.cleanup()\n", 200 | "trading_agent.cleanup()" 201 | ] 202 | } 203 | ], 204 | "metadata": { 205 | "kernelspec": { 206 | "display_name": "Python 3 (ipykernel)", 207 | "language": "python", 208 | "name": "python3" 209 | }, 210 | "language_info": { 211 | "codemirror_mode": { 212 | "name": "ipython", 213 | "version": 3 214 | }, 215 | "file_extension": ".py", 216 | "mimetype": "text/x-python", 217 | "name": "python", 218 | "nbconvert_exporter": "python", 219 | "pygments_lexer": "ipython3", 220 | "version": "3.11.8" 221 | } 222 | }, 223 | "nbformat": 4, 224 | "nbformat_minor": 4 225 | } 226 | -------------------------------------------------------------------------------- /gen-ai/Assistants/notebooks/wind_farm/README.md: -------------------------------------------------------------------------------- 1 | 2 | ## Wind Turbine Farm Management Assistant 3 | 4 | ### Overview 5 | 6 | Utilizing Assistant tools such as the Code Interpreter and Function calling, this bot is capable of retrieving a CSV file that illustrates turbine wind speed, voltage, and the last maintenance date. It assists you in reviewing the file contents and aids in determining whether a specific turbine is in need of maintenance. 7 | 8 | ### Objective 9 | 10 | The objective of the provided notebook file is to create an Azure OpenAI Assistant named "Turbine Management Assistant" using the Azure OpenAI API. The assistant is designed to function as a management tool for wind turbine farms, providing information, charts, and recommendations related to the status and performance of individual turbines within the farm. The script initiates a conversation with the assistant, guiding it through various queries and scenarios to showcase its capabilities. 11 | 12 | ### Programming Languages 13 | 14 | - Python 15 | 16 | ### Estimated Runtime: 5 mins 17 | -------------------------------------------------------------------------------- /gen-ai/Assistants/notebooks/wind_farm/assistant-wind_farm.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Wind Turbine Farm Management Assistant" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "## Objective\n", 15 | "\n", 16 | "This notebook demonstrates the following:\n", 17 | "\n", 18 | "- Utilizing Assistant tools such as the Code Interpreter and Function calling, this bot is capable of retrieving a CSV file that illustrates turbine wind speed, voltage, and the last maintenance date. It assists you in reviewing the file contents and aids in determining whether a specific turbine is in need of maintenance.\n", 19 | "\n", 20 | "This tutorial uses the following Azure AI services:\n", 21 | "- Access to Azure OpenAI Service - you can apply for access [here](https://aka.ms/oai/access)\n", 22 | "- Azure OpenAI service - you can create it from instructions [here](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/create-resource)\n", 23 | "- Azure OpenAI Studio - go to [https://oai.azure.com/](https://oai.azure.com/) to work with the Assistants API Playground\n", 24 | "- A connection to the Azure OpenAI Service with a [Key and Endpoint](https://learn.microsoft.com/en-us/azure/ai-services/openai/chatgpt-quickstart)\n", 25 | "\n", 26 | "Reference:\n", 27 | "- Learn more about how to use Assistants with our [How-to guide on Assistants](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/assistant).\n", 28 | "- [Assistants OpenAI Overview](https://platform.openai.com/docs/assistants/overview)" 29 | ] 30 | }, 31 | { 32 | "cell_type": "markdown", 33 | "metadata": {}, 34 | "source": [ 35 | "## Time\n", 36 | "\n", 37 | "You should expect to spend 5-10 minutes running this sample." 38 | ] 39 | }, 40 | { 41 | "cell_type": "markdown", 42 | "metadata": {}, 43 | "source": [ 44 | "## About this example\n", 45 | "\n", 46 | "This sample demonstrates the creation of an Azure OpenAI Assistant named \"Turbine Management Assistant\" utilizing the Azure OpenAI API. The assistant functions as a management tool for wind turbine farms, providing data, charts, and actionable insights regarding the status and performance of each turbine within the farm. The notebook/script orchestrates a dialogue with the assistant, navigating through different inquiries and scenarios to demonstrate its capabilities.\n", 47 | "\n", 48 | "### Data\n", 49 | "This sample uses files from the folder [`data/`](./data/) in this repo. You can clone this repo or copy this folder to make sure you have access to these files when running the sample." 50 | ] 51 | }, 52 | { 53 | "cell_type": "markdown", 54 | "metadata": {}, 55 | "source": [ 56 | "## Before you begin" 57 | ] 58 | }, 59 | { 60 | "cell_type": "markdown", 61 | "metadata": {}, 62 | "source": [ 63 | "### Installation\n", 64 | "\n", 65 | "Install the following packages required to execute this notebook. \n", 66 | "\n" 67 | ] 68 | }, 69 | { 70 | "cell_type": "code", 71 | "execution_count": null, 72 | "metadata": {}, 73 | "outputs": [], 74 | "source": [ 75 | "# Install the packages\n", 76 | "%pip install -r ../requirements.txt" 77 | ] 78 | }, 79 | { 80 | "cell_type": "markdown", 81 | "metadata": {}, 82 | "source": [ 83 | "### Parameters" 84 | ] 85 | }, 86 | { 87 | "cell_type": "code", 88 | "execution_count": null, 89 | "metadata": {}, 90 | "outputs": [], 91 | "source": [ 92 | "import os\n", 93 | "\n", 94 | "from dotenv import load_dotenv\n", 95 | "\n", 96 | "load_dotenv(\"../.env\") # make sure to have the .env file in the root directory of the project\n", 97 | "\n", 98 | "api_endpoint = os.getenv(\"OPENAI_URI\")\n", 99 | "api_key = os.getenv(\"OPENAI_KEY\")\n", 100 | "api_version = os.getenv(\"OPENAI_VERSION\")\n", 101 | "api_deployment_name = os.getenv(\"OPENAI_GPT_DEPLOYMENT\")\n", 102 | "email_URI = os.getenv(\"EMAIL_URI\")\n", 103 | "\n", 104 | "should_cleanup: bool = True" 105 | ] 106 | }, 107 | { 108 | "cell_type": "markdown", 109 | "metadata": {}, 110 | "source": [ 111 | "## Run this Example" 112 | ] 113 | }, 114 | { 115 | "cell_type": "markdown", 116 | "metadata": {}, 117 | "source": [ 118 | "### Load the required libraries" 119 | ] 120 | }, 121 | { 122 | "cell_type": "code", 123 | "execution_count": null, 124 | "metadata": {}, 125 | "outputs": [], 126 | "source": [ 127 | "import io\n", 128 | "import time\n", 129 | "from datetime import datetime\n", 130 | "from pathlib import Path\n", 131 | "from typing import Iterable\n", 132 | "\n", 133 | "from openai import AzureOpenAI\n", 134 | "from openai.types import FileObject\n", 135 | "from openai.types.beta.threads.text_content_block import TextContentBlock\n", 136 | "from openai.types.beta.threads.image_file_content_block import ImageFileContentBlock\n", 137 | "from PIL import Image" 138 | ] 139 | }, 140 | { 141 | "cell_type": "markdown", 142 | "metadata": {}, 143 | "source": [ 144 | "### Create an Azure OpenAI client" 145 | ] 146 | }, 147 | { 148 | "cell_type": "code", 149 | "execution_count": null, 150 | "metadata": {}, 151 | "outputs": [], 152 | "source": [ 153 | "client = AzureOpenAI(api_key=api_key, api_version=api_version, azure_endpoint=api_endpoint)" 154 | ] 155 | }, 156 | { 157 | "cell_type": "markdown", 158 | "metadata": {}, 159 | "source": [ 160 | "### Define the Assistant tools" 161 | ] 162 | }, 163 | { 164 | "cell_type": "code", 165 | "execution_count": null, 166 | "metadata": {}, 167 | "outputs": [], 168 | "source": [ 169 | "tools_list = [\n", 170 | " {\"type\": \"code_interpreter\"},\n", 171 | "]" 172 | ] 173 | }, 174 | { 175 | "cell_type": "markdown", 176 | "metadata": {}, 177 | "source": [ 178 | "### Upload the file(s)" 179 | ] 180 | }, 181 | { 182 | "cell_type": "code", 183 | "execution_count": null, 184 | "metadata": {}, 185 | "outputs": [], 186 | "source": [ 187 | "DATA_FOLDER = \"data/\"\n", 188 | "\n", 189 | "def upload_file(client: AzureOpenAI, path: str) -> FileObject:\n", 190 | " with Path(path).open(\"rb\") as f:\n", 191 | " return client.files.create(file=f, purpose=\"assistants\")\n", 192 | "\n", 193 | "arr = os.listdir(DATA_FOLDER)\n", 194 | "assistant_files = []\n", 195 | "for file in arr:\n", 196 | " filePath = DATA_FOLDER + file\n", 197 | " assistant_files.append(upload_file(client, filePath))\n", 198 | "\n", 199 | "file_ids = [file.id for file in assistant_files]" 200 | ] 201 | }, 202 | { 203 | "cell_type": "markdown", 204 | "metadata": {}, 205 | "source": [ 206 | "### Create an Assistant and a Thread" 207 | ] 208 | }, 209 | { 210 | "cell_type": "code", 211 | "execution_count": null, 212 | "metadata": {}, 213 | "outputs": [], 214 | "source": [ 215 | "assistant = client.beta.assistants.create(\n", 216 | " name=\"Portfolio Management Assistant\",\n", 217 | " instructions=\"You are an assistant that can help manage wind turbine farm. \"\n", 218 | " + \"The turbines operating ranges are output voltages of 33kv-35kv and RPM of 15-25. Wind speed is measured in miles per hour.\"\n", 219 | " + \"Maintenance should occur every 12 months. Greet the user by saying, 'Welcome Turbine Management Assistant.'\",\n", 220 | " tools=tools_list,\n", 221 | " model=api_deployment_name,\n", 222 | " tool_resources={'code_interpreter':{'file_ids': file_ids}},\n", 223 | ")\n", 224 | "\n", 225 | "thread = client.beta.threads.create()" 226 | ] 227 | }, 228 | { 229 | "cell_type": "markdown", 230 | "metadata": {}, 231 | "source": [ 232 | "### Format and display the Assistant Messages for text and image" 233 | ] 234 | }, 235 | { 236 | "cell_type": "code", 237 | "execution_count": null, 238 | "metadata": {}, 239 | "outputs": [], 240 | "source": [ 241 | "def read_assistant_file(file_id:str):\n", 242 | " response_content = client.files.content(file_id)\n", 243 | " return response_content.read()" 244 | ] 245 | }, 246 | { 247 | "cell_type": "code", 248 | "execution_count": null, 249 | "metadata": {}, 250 | "outputs": [], 251 | "source": [ 252 | "def print_messages(messages) -> None:\n", 253 | " message_list = []\n", 254 | "\n", 255 | " # Get all the messages till the last user message\n", 256 | " for message in messages:\n", 257 | " message_list.append(message)\n", 258 | " if message.role == \"user\":\n", 259 | " break\n", 260 | "\n", 261 | " # Reverse the messages to show the last user message first\n", 262 | " message_list.reverse()\n", 263 | "\n", 264 | " # Print the user or Assistant messages or images\n", 265 | " for message in message_list:\n", 266 | " for item in message.content:\n", 267 | " # Determine the content type\n", 268 | " if isinstance(item, TextContentBlock):\n", 269 | " print(f\"{message.role}:\\n{item.text.value}\\n\")\n", 270 | " file_annotations = item.text.annotations\n", 271 | " if file_annotations:\n", 272 | " for annotation in file_annotations:\n", 273 | " file_id = annotation.file_path.file_id\n", 274 | " content = read_assistant_file(file_id)\n", 275 | " print(f\"Annotation Content:\\n{str(content)}\\n\")\n", 276 | " elif isinstance(item, ImageFileContentBlock):\n", 277 | " # Retrieve image from file id \n", 278 | " data_in_bytes = read_assistant_file(item.image_file.file_id)\n", 279 | " # Convert bytes to image\n", 280 | " readable_buffer = io.BytesIO(data_in_bytes)\n", 281 | " image = Image.open(readable_buffer)\n", 282 | " # Resize image to fit in terminal\n", 283 | " width, height = image.size\n", 284 | " image = image.resize((width // 2, height // 2), Image.LANCZOS)\n", 285 | " # Display image\n", 286 | " image.show()" 287 | ] 288 | }, 289 | { 290 | "cell_type": "markdown", 291 | "metadata": {}, 292 | "source": [ 293 | "### Process the user messages" 294 | ] 295 | }, 296 | { 297 | "cell_type": "code", 298 | "execution_count": null, 299 | "metadata": {}, 300 | "outputs": [], 301 | "source": [ 302 | "def process_prompt(prompt: str) -> None:\n", 303 | " client.beta.threads.messages.create(thread_id=thread.id, role=\"user\", content=prompt)\n", 304 | " run = client.beta.threads.runs.create(\n", 305 | " thread_id=thread.id,\n", 306 | " assistant_id=assistant.id,\n", 307 | " instructions=\"Please address the user as Jane Doe. The user has a premium account. Be assertive, accurate, and polite. Ask if the user has further questions. \"\n", 308 | " + \"The current date and time is: \"\n", 309 | " + datetime.now().strftime(\"%x %X\")\n", 310 | " + \". \",\n", 311 | " )\n", 312 | " print(\"processing ...\")\n", 313 | " while True:\n", 314 | " run = client.beta.threads.runs.retrieve(thread_id=thread.id, run_id=run.id)\n", 315 | " if run.status == \"completed\":\n", 316 | " # Handle completed\n", 317 | " messages = client.beta.threads.messages.list(thread_id=thread.id)\n", 318 | " print_messages(messages)\n", 319 | " break\n", 320 | " if run.status == \"failed\":\n", 321 | " messages = client.beta.threads.messages.list(thread_id=thread.id)\n", 322 | " answer = messages.data[0].content[0].text.value\n", 323 | " print(f\"Failed User:\\n{prompt}\\nAssistant:\\n{answer}\\n\")\n", 324 | " # Handle failed\n", 325 | " break\n", 326 | " if run.status == \"expired\":\n", 327 | " # Handle expired\n", 328 | " print(run)\n", 329 | " break\n", 330 | " if run.status == \"cancelled\":\n", 331 | " # Handle cancelled\n", 332 | " print(run)\n", 333 | " break\n", 334 | " if run.status == \"requires_action\":\n", 335 | " # Handle function calling and continue processing\n", 336 | " pass\n", 337 | " else:\n", 338 | " time.sleep(5)" 339 | ] 340 | }, 341 | { 342 | "cell_type": "markdown", 343 | "metadata": {}, 344 | "source": [ 345 | "### Have a conversation with the Assistant" 346 | ] 347 | }, 348 | { 349 | "cell_type": "code", 350 | "execution_count": null, 351 | "metadata": {}, 352 | "outputs": [], 353 | "source": [ 354 | "process_prompt(\"What is the status of turbine 1001 and 1003?\")" 355 | ] 356 | }, 357 | { 358 | "cell_type": "code", 359 | "execution_count": null, 360 | "metadata": {}, 361 | "outputs": [], 362 | "source": [ 363 | "process_prompt(\"Generate a chart of the normal operating ranges?\")" 364 | ] 365 | }, 366 | { 367 | "cell_type": "code", 368 | "execution_count": null, 369 | "metadata": {}, 370 | "outputs": [], 371 | "source": [ 372 | "process_prompt(\n", 373 | " \"What turbines are operating outside normal ranges?\\n\\n\"\n", 374 | " + 'Sample: [{\"turbine\":\"\",\"voltage\":30,\"RPM\":15,\"reason\":\"Voltage outside of normal range.\"},\\n'\n", 375 | " + '{\"turbine\":\"\",\"voltage\":30,\"rpm\":10,\"reason\":\"RPM and Voltage outside of normal ranges.\"},\\n'\n", 376 | " + '{\"turbine\":\"\",\"voltage\":33,\"rpm\":5,\"reason\":\"RPM outside of normal range.\"}]\\n\\n'\n", 377 | " + 'Output format: [{\"turbine\":\"\",\"voltage\":0,\"RPM\":0,\"reason\":\"\"}]\\n\\n'\n", 378 | " + \"Output in JSON format only.\"\n", 379 | ")" 380 | ] 381 | }, 382 | { 383 | "cell_type": "code", 384 | "execution_count": null, 385 | "metadata": {}, 386 | "outputs": [], 387 | "source": [ 388 | "process_prompt(\"What turbines need maintenance?\")" 389 | ] 390 | }, 391 | { 392 | "cell_type": "markdown", 393 | "metadata": {}, 394 | "source": [ 395 | "### Cleaning up" 396 | ] 397 | }, 398 | { 399 | "cell_type": "code", 400 | "execution_count": null, 401 | "metadata": {}, 402 | "outputs": [], 403 | "source": [ 404 | "if should_cleanup:\n", 405 | " client.beta.assistants.delete(assistant.id)\n", 406 | " client.beta.threads.delete(thread.id)\n", 407 | " for file in assistant_files:\n", 408 | " client.files.delete(file.id)" 409 | ] 410 | }, 411 | { 412 | "cell_type": "code", 413 | "execution_count": null, 414 | "metadata": {}, 415 | "outputs": [], 416 | "source": [] 417 | } 418 | ], 419 | "metadata": { 420 | "kernelspec": { 421 | "display_name": "Python 3 (ipykernel)", 422 | "language": "python", 423 | "name": "python3" 424 | }, 425 | "language_info": { 426 | "codemirror_mode": { 427 | "name": "ipython", 428 | "version": 3 429 | }, 430 | "file_extension": ".py", 431 | "mimetype": "text/x-python", 432 | "name": "python", 433 | "nbconvert_exporter": "python", 434 | "pygments_lexer": "ipython3", 435 | "version": "3.11.9" 436 | } 437 | }, 438 | "nbformat": 4, 439 | "nbformat_minor": 4 440 | } 441 | -------------------------------------------------------------------------------- /gen-ai/Assistants/notebooks/wind_farm/data/turbines.csv: -------------------------------------------------------------------------------- 1 | Turbine_ID,Wind_Speed,RPM,Voltage,Maintenance_Date 2 | 1000,34,20,34,2023-08-25 3 | 1001,32,18,33,2023-10-12 4 | 1002,36,22,35,2023-07-05 5 | 1003,30,16,32,2023-11-30 6 | 1004,38,24,36,2022-06-15 7 | 1005,28,14,31,2023-12-20 8 | 1006,33,19,34,2023-09-10 9 | 1007,31,17,33,2023-10-30 10 | 1008,37,23,35,2023-07-25 11 | 1009,29,15,32,2023-12-05 12 | 1010,40,26,37,2023-05-30 13 | 1011,27,13,30,2024-01-10 14 | 1012,35,21,34,2023-08-10 15 | 1013,33,19,33,2023-09-30 16 | 1014,39,25,36,2023-06-05 17 | 1015,31,17,31,2023-11-15 18 | 1016,37,23,35,2023-07-10 19 | 1017,30,16,32,2022-12-30 20 | 1018,38,24,36,2023-05-15 21 | 1019,32,18,33,2023-10-05 22 | 1020,34,20,34,2023-08-05 23 | -------------------------------------------------------------------------------- /gen-ai/nlp-sql-in-a-box/readme.md: -------------------------------------------------------------------------------- 1 | # NLP to SQL in-a-box 2 | ![banner](../../media/images/banner-nlp-to-sql-in-a-box.png) 3 | Build a cutting-edge speech-enabled SQL query system using Azure Open AI, Semantic Kernel, and Azure AI Speech Service 4 | 5 | We will use the power of Azure Open AI and Semantic Kernel to translate your natural language queries into SQL statements that can be executed against an SQL Server database. This will allow you to interact with your data in a more intuitive and user-friendly way. No more struggling with complex SQL syntax – just speak your query and let the system do the rest! 6 | 7 | And with Azure Speech Services, we will convert your speech into text and synthesize the results as speech. This means that you can hear the results of your query spoken back to you, making it easier to understand and digest the information. 8 | 9 | ## MOVED 10 | 11 | We have moved the solution to a new repository. Please visit the new repository [https://github.com/Azure-Samples/nlp-sql-in-a-box](https://github.com/Azure-Samples/nlp-sql-in-a-box). 12 | -------------------------------------------------------------------------------- /gen-ai/semantic-kernel-bot-in-a-box/README.md: -------------------------------------------------------------------------------- 1 | # Semantic Kernel Bot in-a-box (Archived) 2 | 3 | ![Banner](../../media/images/sk-bot-in-a-box.png) 4 | 5 | This solution deployed an extensible Semantic Kernel bot template to Azure, enabling sophisticated interactions through messaging channels such as Web, PowerBI dashboards, or Teams. 6 | 7 | ## MOVED and ARCHIVED 8 | 9 | This solution is archived at: [https://github.com/Azure-Samples/semantic-kernel-bot-in-a-box](https://github.com/Azure-Samples/semantic-kernel-bot-in-a-box). 10 | 11 | ## SUCCESSOR 12 | 13 | The successor to this solution, which is actively maintained and includes enhanced capabilities, is available at: [https://github.com/Azure-Samples/gen-ai-bot-in-a-box](https://github.com/Azure-Samples/gen-ai-bot-in-a-box). 14 | 15 | The new repository builds upon the foundation of the original Semantic Kernel Bot in-a-box, offering additional features, improved performance, and expanded integration options. It is designed to provide a more robust and flexible framework for developing AI-powered bots using the latest advancements in generative AI and Azure services. 16 | 17 | By migrating to the new repository, you will benefit from continuous updates, new feature additions, and ongoing support from the development community. Whether you are looking to enhance your existing bot or start a new project, the [Gen AI Bot in-a-box](https://github.com/Azure-Samples/gen-ai-bot-in-a-box) offers a comprehensive solution that evolves with the latest AI technologies. 18 | -------------------------------------------------------------------------------- /guidance/genai-security/README.md: -------------------------------------------------------------------------------- 1 | # Guidance: Security for Generative AI (GenAI) Applications 2 | ![GenAI Security](../../media/images/banner-genai-security.png) 3 | 4 | ## Introduction 5 | As LLMs become more easily available and integrated into our work and personal lives, the promise of the technology is tempered by the potential for it to be misused. And the potential for misuse becomes even more significant when you realize LLMs can be combined with other powerful software components and agents to orchestrate a pipeline of actions. OR combined with proprietary and personal data to introduce new avenues for data disclosure and leakage. 6 | 7 | The intention for this page is not to reiterate security guidance that is generally available for more traditional or cloud software applications but to focus on guidance specific to *GenAI* applications and the unique characteristics and challenges of LLMs. 8 | 9 | ## Threats & Risks 10 | The security threats and risks with traditional software applications are familiar and understood. *GenAI* and LLMs introduce new and unique security risks including: 11 | * **AI responses are based on statistical probabilities** or the best chance for correct output. LLMs generate convincing human-like responses by predicting what words come next in a phrase. While they can be great at helping with tasks like summarizing a document or explaining complicated concepts or boosting creativity, there can be issues like responses being inaccurate, incomplete, inappropriate, or completely fabricated. You may be familiar with one well known example where ChatGPT provided non-existent legal citations that lawyers presented in court: [Here's what happens when your lawyer uses ChatGPT](https://www.nytimes.com/2023/05/27/nyregion/avianca-airline-lawsuit-chatgpt.html). 12 | * *GenAI* is **by design a non-deterministic technology** which means that given identical inputs, responses and output may differ. 13 | * *GenAI* applications **can be extended with agents, plugins, and even external APIs that can significantly expand the attack surface** for a *GenAI* application. For instance, an LLM may implicitly trust a plugin or 3rd party component that is malicious. 14 | * Another challenge with GenAI is that it **currently it is not possible to enforce an isolation boundary between the data and the control planes**. This means that LLMs are not always able to differentiate between data being submitted as content or an adversarial instruction submitted as content. Think about a SQL databases: instructions are supplied through query language and validated with a parser before data is queried, manipulated, or provided as output. With a SQL injection attack, a malicious instruction can piggyback on an ambiguously phrased language construct but it can be mitigated with a parameterized query. *GenAI*/LLMs do not have that boundary between syntax (control plane) and data so other mechanisms are needed. 15 | 16 | The diagram below is from [OWASP Top 10 for Large Language Model Applications](https://owasp.org/www-project-top-10-for-large-language-model-applications/assets/PDF/OWASP-Top-10-for-LLMs-2023-v1_1.pdf) and depicts the potential security risks for a hypothetical LLM app: 17 | 18 | ![GenAI Security](../../media/images/OWASP-Security-Risks-LLM-Apps.png) 19 | 20 | ## Security Strategies 21 | Infrastructure plays an indispensable role in helping create a secure landscape for *GenAI* applications, particularly cloud environments. Below are strategies tht can help ensure the security of a *GenAI* environment: 22 | * **Threat Modeling** Include *GenAI* apps in your threat modeling practice. Understand that *GenAI* can extend attack surface with access to underlying or referenced data sources, access to model API keys, workflow orchestration, and agents and plugins. Learn more about what can go wrong with [The AI Attack Surface Map v1.0](https://danielmiessler.com/p/the-ai-attack-surface-map-v1-0/). 23 | * **Architecture strategies** help ensure a secure, scalable, and available environment. 24 | * [Baseline OpenAI end-to-end chat reference architecture](https://learn.microsoft.com/en-us/azure/architecture/ai-ml/architecture/baseline-openai-e2e-chat): a baseline architecture for building and deploying enterprise chat apps that use Azure OpenAI. 25 | * [OpenAI end-to-end baseline reference implementation](https://github.com/azure-Samples/openai-end-to-end-baseline): Author and run a chat app in a single region with Azure ML and OpenAI. 26 | * **Network strategies** help ensure that the cloud infrastructure is properly segmented and that access is controlled and monitored. Consider network segmentation, using secure protocols, enforcing Secure APIs and endpoints. For GenAI specific recommendations see: [Cognitive Services Landing Zone in-a-box](https://github.com/Azure/AI-in-a-Box/blob/main/ai-services/ai-landing-zone/README.md) 27 | * **Access and Identity strategies** to enforce user verification and provide a barrier to malicious access. When possible, use managed identities and RBAC to authenticate and authorize access and avoid use of *GenAI* service API keys for access. Another consideration to keep in mind is that access patterns like role or row level access to indexes may not be natively supported. See: 28 | * [Authentication & Authorization in GenAI Apps with Entra ID & Search](https://techcommunity.microsoft.com/t5/fasttrack-for-azure/authentication-and-authorization-in-generative-ai-applications/ba-p/4022277) 29 | * [Azure AI Search - Restricting access to indexes](https://learn.microsoft.com/en-us/azure/search/search-security-overview#restricting-access-to-documents) 30 | * **Application strategies** help ensure the application is configured securely and vulnerabilities are identified and addressed: 31 | * Use App front end services to manage access and throughput. See: [Azure OpenAI Service Load Balancing with Azure API Management](https://learn.microsoft.com/en-us/samples/azure-samples/azure-openai-apim-load-balancing/azure-openai-service-load-balancing-with-azure-api-management/) and [Smart load balancing for OpenAI endpoints and Azure API Management](https://techcommunity.microsoft.com/t5/fasttrack-for-azure/smart-load-balancing-for-openai-endpoints-and-azure-api/ba-p/3991616) 32 | * Ensure related services are deployed securely (AI Search, Cosmos DB, etc) 33 | * Secure and validate training data and ingestion pipelines 34 | * **Governance strategies** help ensure the infrastructure is being used is meeting security and compliance requirements and that policies and procedures are in place to manage risk and accountability: 35 | * Become familiar with Responsible AI principles and frameworks and integrate them early in the development of your application. More here: [Responsible AI](./responsible-ai) 36 | * Leverage platform capabilities for logging, auditing, and monitoring *GenAI* apps. See: [Implement logging and monitoring for Azure OpenAI models](https://learn.microsoft.com/en-us/azure/architecture/ai-ml/openai/architecture/log-monitor-azure-openai). 37 | 38 | ## Adversarial Prompting 39 | 40 | #### Attacks 41 | An adversarial prompt attack is when a prompt is used to manipulate an LLM in order to generate a malicious or unintended response. A sneaky user can tamper with words or sentence structure to exploit nuances or sentiment in language models. You may be familiar with some types of prompt attacks: 42 | * [Prompt injection](https://www.promptingguide.ai/risks/adversarial#prompt-injection): prompt input, output, or instructions are manipulated to lead to unintended behavior. 43 | * [Prompt leaking](https://www.promptingguide.ai/risks/adversarial#prompt-leaking): is intended to cause the model to leak confidential or proprietary information. 44 | * [Jailbreaking](https://www.promptingguide.ai/risks/adversarial#jailbreaking): is a technique to bypass model safety mechanisms to generate illegal or unethical content. 45 | * [DAN](https://www.promptingguide.ai/risks/adversarial#dan): is an acronym for **D**o **A**nything **N**ow and is another technique intended to circumvent model safety guardrails and force it to comply with requests that generate unfiltered responses. 46 | * [Multi-prompt](https://www.lakera.ai/blog/guide-to-prompt-injection): a series of prompts are used to extract private or sensitive information. 47 | * Multi-language: although LLMs are trained in multiple languages, performance is superior for English. This technique involves submitting a request in languages other than English to cause the model to overlook or bypass security checks. 48 | * Obfuscation (token smuggling): a technique to present data in an unexpected format to avoid detection. 49 | 50 | *Note*: Details about these and other adversarial techniques can be found here: 51 | * [Prompt Engineering Guide](https://www.promptingguide.ai/) 52 | * [The EL15 Guide to Prompt Injection: Techniques, Prevention Methods & Tools](https://www.lakera.ai/blog/guide-to-prompt-injection) 53 | 54 | #### Mitigations 55 | As a mitigation strategy for adversarial prompt attacks, consider advanced [prompt engineering techniques](https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/advanced-prompt-engineering). There is a growing list of specific techniques that can be used that include enriching prompts with specific instructions, formatting, and providing examples of the kind of output content that is intended. Below are some techniques to consider: 56 | * **Defensive Instructions**: Guide model response with explicit instructions. Structure the [system message](https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/advanced-prompt-engineering?pivots=programming-language-chat-completions#system-message) with context and instructions. See also: [Add defense in the instruction](https://github.com/dair-ai/Prompt-Engineering-Guide/blob/main/guides/prompts-adversarial.md#add-defense-in-the-instruction). 57 | * **Determine intent**: Use techniques like [few-shot learning](https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/advanced-prompt-engineering?pivots=programming-language-chat-completions#few-shot-learning) to provide content to the model and help set intent. 58 | * **Monitor for degradation in output quality**: a decline in output quality can be an indication that a prompt has been tampered with. Monitor the model for output quality using metrics to measure and evaluate the prompt or human-in-the-loop to evaluate feedback or adversarial test cases to confirm prompt resilience. [Azure Machine Learning prompt flow](https://learn.microsoft.com/en-us/azure/machine-learning/prompt-flow/overview-what-is-prompt-flow?view=azureml-api-2) has built-in evaluation flows that enable users to assess the quality and effectiveness of prompts. 59 | 60 | * **Use other models or dedicated services to process requests**. [Azure AI Content Safety](https://learn.microsoft.com/en-us/azure/ai-services/content-safety/) is an azure service that provides [content filtering](https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/content-filter?tabs=warning%2Cpython). AI models are used to detect and classify categories of harm from AI-generated content. Content filters are more contextually aware than blocklists and can provide broad coverage without the manual creation of rules or lists. 61 | * **Use inbound/outbound block/allow lists or filters or rules**. When there is a need to screen for items specific to a use case, blocklists can be helpful and can be implemented as part of the AI Content Safety service. See: [Use a blocklist in Azure OpenAI](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/use-blocklists). 62 | * Use the native power of models to **steer zero- or few-shot prompting** strategies. See [promptbase](https://github.com/microsoft/promptbase) for a growing collection of resources, best practices, and sample scripts. 63 | 64 | See [Exploring Adversarial Prompting and Mitigations in AI-Infused Applications](https://www.linkedin.com/pulse/exploring-adversarial-prompting-mitigations-alex-morales-3sqne/) for more specifics on these types of attacks and defense tactics. 65 | 66 | ## Resources & References 67 | * [OWASP Top 10 for LLM applications](https://owasp.org/www-project-top-10-for-large-language-model-applications/) and the [downloadable whitepaper](https://www.llmtop10.com/assets/downloads/OWASP-Top-10-for-LLM-Applications-v1_1.pdf) 68 | * [OWASP LLM AI Security & Governance Checklist](https://owasp.org/www-project-top-10-for-large-language-model-applications/llm-top-10-governance-doc/LLM_AI_Security_and_Governance_Checklist.pdf) 69 | * [Security Best Practices for GenAI Applications in Azure](https://techcommunity.microsoft.com/t5/azure-architecture-blog/security-best-practices-for-genai-applications-openai-in-azure/ba-p/4027885) 70 | * [Steering at the Frontier: Extending the Power of Prompting](https://www.microsoft.com/en-us/research/blog/steering-at-the-frontier-extending-the-power-of-prompting/) 71 | * [Planning red teaming for large language models (LLMs) and their applications](https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/red-teaming) 72 | 73 | -------------------------------------------------------------------------------- /guidance/responsible-ai/README.md: -------------------------------------------------------------------------------- 1 | # Guidance: Responsible AI (RAI) 2 | ![Responsible AI (RAI)](../../media/images/banner-rai.png) 3 | 4 | ## Introduction 5 | There is a lot to be excited about with recent advances in AI and LLM technology, but every day there are examples in the media about where and how it has gone wrong. As AI is integrated into more of our daily work and personal lives it can cause minor inconveniences, such as mistakenly canceled appointments, to more serious issues, such as potential job displacement and privacy compromises — and may even compound already existing social or economic inequities. All of us who design, develop, and deploy AI have a responsibility to confront the risks that the technology introduces. 6 | 7 | **If your team makes use of AI APIs or AI systems or designs, develops, or deploys AI, please consider joining us in the commitment to innovate responsibly.** This page contains recommendations and guidance for how to get started. 8 | 9 | ## Get Started 10 | RAI sounds promising. Who wouldn't want their product to be responsible or trustworthy? But there can be challenges: 11 | * AI systems are complex and require a diversity of teams, skills, and tools 12 | * Potential harms and risks of AI systems are different from those of traditional software systems 13 | * Leadership support is crucial for a sustainable RAI practice 14 | 15 | The recommendations that follow are based on lessons learned from rolling out a responsible AI practice across one of the organizations within Microsoft. As you review the recommendations, keep in mind they should be *adapted to fit your organization, circumstances, and product plans*. 16 | 17 | The approach used was grounded in principles, standards, practices that were used internally and are published here: 18 | * **[Microsoft AI principles](https://www.microsoft.com/en-us/ai/principles-and-approach)** of fairness, reliability and safety, privacy and security, Inclusiveness, Transparency, and Accountability. However your organization defines and prioritizes the principles, they become the foundation for RAI standards and practices. 19 | * **[Microsoft Responsible AI Standard, v2](https://query.prod.cms.rt.microsoft.com/cms/api/am/binary/RE5cmFl)**: The RAI standard is the first step in putting the principles into practice. It answers the question... *How are we going to execute on a responsible AI practice...?*. For each principle, the standard sets out concrete steps and outcomes. 20 | * **[Microsoft Responsible AI Impact Assessment Template](https://query.prod.cms.rt.microsoft.com/cms/api/am/binary/RE5cmFk)** and **[Impact Assessment Guide](https://blogs.microsoft.com/wp-content/uploads/prod/sites/5/2022/06/Microsoft-RAI-Impact-Assessment-Guide.pdf)** provides documentation and guidance for how a team can capture and document the benefits, potential harms and mitigations for an AI system. 21 | * **RAI tools and guidance** such as the Responsible AI Maturity Model, HAX Toolkit, RAI Toolbox, and more (link [here](https://aka.ms/rai)). 22 | 23 | ## Complete an impact assessment 24 | A *Responsible AI impact assessment* is the process a product team follows to identify and assess the potential risks and harms of an AI system. It is a new process, and some organizations may be reluctant to consider it, giving reasons such as: 25 | * *It is still too early in AI lifecycle to do impact assessments. RAI is mostly academic*. 26 | * *AI is so new. How can we expect product teams to know about potential risks and harms?* 27 | * *Discussions around RAI will only devolve into messy disagreements and take time away from design/development/deployment timelines.* 28 | 29 | An RAI impact assessment is the primary method for guiding a team through the process of examining an AI system and aligning it to responsible AI principles and standards. The questions it examines include: *What are the use cases for the AI system? Who are the stakeholders? How do we monitor and measure AI? Who might be harmed and how? How do we prevent these harms?* 30 | 31 | The purpose of a good RAI impact assessment is to identify the potential risks and harms of an AI system and introduce mitigations to reduce negative consequences. The templates and guidance introduced above can help a team put responsible AI principles to work. Consider adjustments to better align with your organizational requirements and product team processes. 32 | 33 | Plan to update an impact assessment on an ongoing basis as use cases change, new functionality or technology is introduced, or even as the statistical property of the training data changes over time (data drift). 34 | 35 | ## Strategies to help 36 | The pace of AI is moving quickly and a variety of methods are needed to help measure and advance the responsible design, development, and deployment of AI systems. 37 | 38 | #### Adopt a layered mitigation approach for RAI 39 | At Microsoft, we recommend a layered mitigation approach that combines technical, operational, and governance measures to reduce the potential harms of LLMs. A layered approach applies different measures at different stages of development and deployment as documented in the article [Deploy large language models responsibly with Azure AI](https://techcommunity.microsoft.com/t5/ai-machine-learning-blog/deploy-large-language-models-responsibly-with-azure-ai/ba-p/3876792) and depicted below: 40 | ![Responsible AI (RAI)](../../media/images/Rai-mitigation-layers.png) 41 | See [Harms mitigation strategies with Azure AI](https://learn.microsoft.com/en-us/azure/ai-studio/concepts/evaluation-improvement-strategies) for a walk-through of the mitigation layers. 42 | 43 | #### Get started with RAI metrics 44 | Measuring RAI involves both *technical attributes*, like accuracy and security, and *socio-technical attributes* such as fairness, representational harms, or safety and reliability. A starter set of metrics could include platform, usage, and operational level metrics like: 45 | * *resource consumption* (GPU, CPU, memory utilization, response times), 46 | * *response/request tracking* (number of unique/active users, requests, prompts, token usage), and 47 | * *operational metrics* (quota, capacity, latency, throttling). See [Monitoring Azure OpenAI Service](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/monitoring). 48 | 49 | Add to this a layer of metrics to measure the potential for RAI harms identified through the impact assessment process. Consider measuring the number of input prompts blocked, number of responses blocked, number of jailbreak attempts, or metrics related to quality of generated output such as groundedness, relevance, and similarity. 50 | 51 | #### Tools to support RAI 52 | Tools can be a particular challenge because the RAI tools landscape is relatively new and rapidly evolving. While there are libraries and building blocks, the overall landscape can seem fragmented, making it difficult to establish an end-to-end workflow. Some of the challenges include: 53 | * There is no single tool that can holistically assess potential AI harms. 54 | * Tools often do not map directly to an RAI principle or standard, such as *fairness* or *transparency*, making it difficult to determine an appropriate tool. 55 | * Components and libraries built to support distinct aspects of RAI practice often require additional effort to use them in a complementary way. 56 | 57 | Tools to support an RAI practice for *GenAI* applications focus on model selection, prompt engineering, and model output. Recommendations include: 58 | * **[Azure AI Content Safety](https://learn.microsoft.com/en-us/azure/ai-services/content-safety/)**: An azure service that provides [content filtering](https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/content-filter?tabs=warning%2Cpython). AI models are used to detect and classify categories of harm from AI-generated content. Content filters are more contextually aware than blocklists and can provide broad coverage without the manual creation of rules or lists. 59 | * **Blocklists**: When there is a need to screen for items specific to a use case, blocklists can be helpful and can be implemented as part of the AI Content Safety service. See: [Use a blocklist in Azure OpenAI](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/use-blocklists). 60 | * **Meta-prompt best practices**: To mitigate harms, apply recommended prompt engineering practices. See: [Introduction to prompt engineering](https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/prompt-engineering) and [Prompt engineering techniques](https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/advanced-prompt-engineering). 61 | * **Prompt template recommendations**: Example templates to help write effective system messages to guide AI system behavior: See [System message framework and template recommendations for Large Language Models (LLMs)](https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/system-message?source=recommendations). 62 | * **Prompt flow**: [Azure Machine Learning prompt flow](https://learn.microsoft.com/en-us/azure/machine-learning/prompt-flow/overview-what-is-prompt-flow?view=azureml-api-2) is a development tool designed to streamline the entire development cycle of AI applications. Specific to RAI are the built-in evaluation flows that enable users to assess the quality and effectiveness of prompts. 63 | 64 | ## Resources & References 65 | Here are additional RAI resources: 66 | * [Microsoft AI principles](https://www.microsoft.com/en-us/ai/principles-and-approach) 67 | * [Microsoft Responsible AI Standard, v2](https://query.prod.cms.rt.microsoft.com/cms/api/am/binary/RE5cmFl) 68 | * [Microsoft Responsible AI Impact Assessment Template](https://query.prod.cms.rt.microsoft.com/cms/api/am/binary/RE5cmFk) 69 | * [Responsible AI tools and guidance](https://aka.ms/rai) and [HAX Toolkit](https://www.microsoft.com/en-us/haxtoolkit/) 70 | * Video: [An introduction to responsible AI (RAI) process](https://learn.microsoft.com/en-us/shows/learn-live/fasttrack-for-azure-season-3-ep07-an-introduction-to-responsible-ai-rai-process) 71 | * Medium articles: [Responsible AI in action, Part 1: Get started](https://medium.com/data-science-at-microsoft/responsible-ai-in-action-part-1-get-started-ee50bebbdff3?source=friends_link&sk=3a9ad40230116d9fc4c66fdf7ab56de2), and [Responsible AI in action, Part 2: Complete an impact assessment](https://medium.com/data-science-at-microsoft/responsible-ai-in-action-part-2-complete-an-impact-assessment-9b792409e8db?source=friends_link&sk=6e68eb938a2be1d776748cc55a89b663) and [Responsible AI in action, Part 3: Tools to help](https://medium.com/data-science-at-microsoft/responsible-ai-in-action-part-3-tools-to-help-969e45cac11b?source=friends_link&sk=69bfea1ae66e2b7272d58e28a49cafe4) 72 | 73 | 74 | -------------------------------------------------------------------------------- /guidance/scaling/README.md: -------------------------------------------------------------------------------- 1 | # Guidance on Scaling OpenAI Applications with Azure Services 2 | 3 | ## Introduction 4 | 5 | This guidance document contains best practices for scaling OpenAI applications within Azure, detailing resource organization, quota management, rate limiting, and the strategic use of Provisioned Throughput Units (PTUs) and Azure API Management (APIM) for efficient load balancing. 6 | 7 | ## Best Practices for Azure OpenAI Resources 8 | 9 | - Consolidate Azure OpenAI workloads under a **single Azure subscription** to streamline management and cost optimization. 10 | - Treat Azure OpenAI resources as a **shared service** to ensure efficient usage of PTU and PAYG resources. 11 | - Utilize separate subscriptions only for distinct development and production environments or for geographic requirements. 12 | - Prefer **resource groups for regional isolation**, which simplifies scaling and management compared to multiple subscriptions. 13 | - Maintain a **single Azure OpenAI resource per region**, allowing up to 30 enabled regions within a single subscription. 14 | - Create both PAYG and PTU deployments within each Azure OpenAI resource for each model to ensure flexible scaling. 15 | - Leverage PTUs for business critical usage and PAYG for traffic that exceeds the PTU allocation. 16 | 17 | ## Quotas and Rate Limiting 18 | 19 | ### Tokens 20 | - Tokens are basic text units processed by OpenAI models. Efficient token management is crucial for cost and load balancing. 21 | 22 | ### Quotas 23 | - OpenAI sets API quotas based on subscription plans, dictating API usage within specific time frames. 24 | - Quotas are per model, per region, and per subscription. 25 | - Proactively monitor quotas to prevent unexpected service disruptions. 26 | - **Quotas do not guarantee capacity**, and traffic may be throttled if the service is overloaded. 27 | - During peak traffic, the service may throttle requests even if the quota has not been reached. 28 | 29 | ### Rate Limiting 30 | - Rate limiting ensures equitable API access and system stability. 31 | - Rate Limits are imposed on the number of requests per minute (RPM) and the number of tokens per minute (TPM). 32 | - Implement backoff strategies to handle rate limit errors effectively. 33 | 34 | ### PTUs 35 | - Utilize PTUs for baseline usage of OpenAI workloads to guarantee consistent throughput. 36 | - PAYG deployments should handle traffic that exceeds the PTU allocation. 37 | 38 | ## Load Balancing with Azure API Management (APIM) 39 | 40 | - APIM plays a pivotal role in managing, securing, and analyzing APIs. 41 | - Policies within APIM can be used to manage traffic, secure APIs and enforce usage quotas. 42 | - **Load Balancing** within APIM distributes traffic evenly, ensuring no single instance is overwhelmed. 43 | - **Circuit Breaker** policies in APIM prevent cascading failures and improve system resilience. 44 | - **Smart Load Balancing** with APIM ensures prioritized traffic distribution across multiple OpenAI resources. 45 | 46 | ## Security and High Availability 47 | 48 | - Use Azure API Management to route traffic, ensuring centralized security and compliance. 49 | - Implement private endpoints to secure OpenAI resources and prevent unauthorized access. 50 | - Leverage Managed Identity to secure access to OpenAI resources and other Azure services. 51 | 52 | ## Addressing Special Cases and Limitations 53 | 54 | - Fine-tuned models should be treated as unique domains. Create separate Azure OpenAI resources for each model family based on specific requirements. 55 | 56 | ## Conclusion 57 | 58 | This guidance outlines a strategy for leveraging Azure OpenAI resources at an enterprise level. By centralizing OpenAI resources and adopting smart load balancing with APIM, organizations can maximize their investment in OpenAI, ensuring scalability, cost-effectiveness, and performance across a wide range of applications and use cases. 59 | 60 | ## Additional Resources 61 | 62 | - [Smart load balancing with Azure API Management](https://github.com/Azure-Samples/openai-apim-lb) 63 | - [Smart load balancing with Azure Container Apps](https://github.com/Azure-Samples/openai-aca-lb) 64 | - [Using Azure API Management Circuit Breaker and Load balancing with Azure OpenAI Service](https://techcommunity.microsoft.com/t5/fasttrack-for-azure/using-azure-api-management-circuit-breaker-and-load-balancing/ba-p/4041003) 65 | 66 | For more detailed information on OpenAI's capabilities, tokens, quotas, rate limits, and PTUs, visit the [Azure OpenAI documentation](https://docs.microsoft.com/en-us/azure/cognitive-services/openai/). 67 | -------------------------------------------------------------------------------- /machine-learning/ml-ops-in-a-box/README.md: -------------------------------------------------------------------------------- 1 | # MLOps in-a-box 2 | 3 | ![Banner](/media/images/banner-mlops-in-a-box.png) 4 | 5 | End-to-end operationalization of ML model through Azure ML and GitHub actions leveraging Azure ML CLI V2 6 | 7 | ## MOVED 8 | 9 | We have moved the solution to a new repository. Please visit the new repository [https://github.com/Azure-Samples/ml-ops-in-a-box](https://github.com/Azure-Samples/ml-ops-in-a-box). 10 | -------------------------------------------------------------------------------- /media/Assets/AI-in-a-Box-PatternPool.vsdx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/AI-in-a-Box/52ff38a2333df2ae1d918f0db4da18e9e717ca57/media/Assets/AI-in-a-Box-PatternPool.vsdx -------------------------------------------------------------------------------- /media/images/AI-in-a-Box.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/AI-in-a-Box/52ff38a2333df2ae1d918f0db4da18e9e717ca57/media/images/AI-in-a-Box.jpeg -------------------------------------------------------------------------------- /media/images/OWASP-Security-Risks-LLM-Apps.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/AI-in-a-Box/52ff38a2333df2ae1d918f0db4da18e9e717ca57/media/images/OWASP-Security-Risks-LLM-Apps.png -------------------------------------------------------------------------------- /media/images/Pattern1.A.MLOps.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/AI-in-a-Box/52ff38a2333df2ae1d918f0db4da18e9e717ca57/media/images/Pattern1.A.MLOps.png -------------------------------------------------------------------------------- /media/images/Pattern2.A.EdgeAI.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/AI-in-a-Box/52ff38a2333df2ae1d918f0db4da18e9e717ca57/media/images/Pattern2.A.EdgeAI.png -------------------------------------------------------------------------------- /media/images/Pattern3.A.1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/AI-in-a-Box/52ff38a2333df2ae1d918f0db4da18e9e717ca57/media/images/Pattern3.A.1.png -------------------------------------------------------------------------------- /media/images/Pattern3.A.App.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/AI-in-a-Box/52ff38a2333df2ae1d918f0db4da18e9e717ca57/media/images/Pattern3.A.App.png -------------------------------------------------------------------------------- /media/images/Pattern3.A.Landing.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/AI-in-a-Box/52ff38a2333df2ae1d918f0db4da18e9e717ca57/media/images/Pattern3.A.Landing.png -------------------------------------------------------------------------------- /media/images/Pattern4.A.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/AI-in-a-Box/52ff38a2333df2ae1d918f0db4da18e9e717ca57/media/images/Pattern4.A.png -------------------------------------------------------------------------------- /media/images/Pattern4.Landing.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/AI-in-a-Box/52ff38a2333df2ae1d918f0db4da18e9e717ca57/media/images/Pattern4.Landing.png -------------------------------------------------------------------------------- /media/images/Rai-mitigation-layers.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/AI-in-a-Box/52ff38a2333df2ae1d918f0db4da18e9e717ca57/media/images/Rai-mitigation-layers.png -------------------------------------------------------------------------------- /media/images/ai-in-a-box.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/AI-in-a-Box/52ff38a2333df2ae1d918f0db4da18e9e717ca57/media/images/ai-in-a-box.png -------------------------------------------------------------------------------- /media/images/aibxtable.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/AI-in-a-Box/52ff38a2333df2ae1d918f0db4da18e9e717ca57/media/images/aibxtable.png -------------------------------------------------------------------------------- /media/images/aiinabox.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/AI-in-a-Box/52ff38a2333df2ae1d918f0db4da18e9e717ca57/media/images/aiinabox.png -------------------------------------------------------------------------------- /media/images/aiservices-in-a-box.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/AI-in-a-Box/52ff38a2333df2ae1d918f0db4da18e9e717ca57/media/images/aiservices-in-a-box.png -------------------------------------------------------------------------------- /media/images/aoai-in-a-box.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/AI-in-a-Box/52ff38a2333df2ae1d918f0db4da18e9e717ca57/media/images/aoai-in-a-box.png -------------------------------------------------------------------------------- /media/images/banner-ai-landing-zone.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/AI-in-a-Box/52ff38a2333df2ae1d918f0db4da18e9e717ca57/media/images/banner-ai-landing-zone.png -------------------------------------------------------------------------------- /media/images/banner-aio-with-ai-in-a-box.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/AI-in-a-Box/52ff38a2333df2ae1d918f0db4da18e9e717ca57/media/images/banner-aio-with-ai-in-a-box.png -------------------------------------------------------------------------------- /media/images/banner-aivision-edge-in-a-box.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/AI-in-a-Box/52ff38a2333df2ae1d918f0db4da18e9e717ca57/media/images/banner-aivision-edge-in-a-box.png -------------------------------------------------------------------------------- /media/images/banner-aml-edge-in-a-box.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/AI-in-a-Box/52ff38a2333df2ae1d918f0db4da18e9e717ca57/media/images/banner-aml-edge-in-a-box.png -------------------------------------------------------------------------------- /media/images/banner-aoai-video-analysis-in-a-box.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/AI-in-a-Box/52ff38a2333df2ae1d918f0db4da18e9e717ca57/media/images/banner-aoai-video-analysis-in-a-box.png -------------------------------------------------------------------------------- /media/images/banner-assistants-api-bot-in-a-box.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/AI-in-a-Box/52ff38a2333df2ae1d918f0db4da18e9e717ca57/media/images/banner-assistants-api-bot-in-a-box.png -------------------------------------------------------------------------------- /media/images/banner-assistants-api-in-a-box.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/AI-in-a-Box/52ff38a2333df2ae1d918f0db4da18e9e717ca57/media/images/banner-assistants-api-in-a-box.png -------------------------------------------------------------------------------- /media/images/banner-assistants-in-a-box.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/AI-in-a-Box/52ff38a2333df2ae1d918f0db4da18e9e717ca57/media/images/banner-assistants-in-a-box.png -------------------------------------------------------------------------------- /media/images/banner-customvision-edge-in-a-box.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/AI-in-a-Box/52ff38a2333df2ae1d918f0db4da18e9e717ca57/media/images/banner-customvision-edge-in-a-box.png -------------------------------------------------------------------------------- /media/images/banner-doc-intelligence-in-a-box.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/AI-in-a-Box/52ff38a2333df2ae1d918f0db4da18e9e717ca57/media/images/banner-doc-intelligence-in-a-box.png -------------------------------------------------------------------------------- /media/images/banner-edgeai-in-a-box.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/AI-in-a-Box/52ff38a2333df2ae1d918f0db4da18e9e717ca57/media/images/banner-edgeai-in-a-box.png -------------------------------------------------------------------------------- /media/images/banner-genai-security.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/AI-in-a-Box/52ff38a2333df2ae1d918f0db4da18e9e717ca57/media/images/banner-genai-security.png -------------------------------------------------------------------------------- /media/images/banner-mlops-in-a-box.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/AI-in-a-Box/52ff38a2333df2ae1d918f0db4da18e9e717ca57/media/images/banner-mlops-in-a-box.png -------------------------------------------------------------------------------- /media/images/banner-nlp-to-sql-in-a-box.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/AI-in-a-Box/52ff38a2333df2ae1d918f0db4da18e9e717ca57/media/images/banner-nlp-to-sql-in-a-box.png -------------------------------------------------------------------------------- /media/images/banner-rai.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/AI-in-a-Box/52ff38a2333df2ae1d918f0db4da18e9e717ca57/media/images/banner-rai.png -------------------------------------------------------------------------------- /media/images/bluebox.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | -------------------------------------------------------------------------------- /media/images/buildingblocks.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/AI-in-a-Box/52ff38a2333df2ae1d918f0db4da18e9e717ca57/media/images/buildingblocks.png -------------------------------------------------------------------------------- /media/images/edgeai-customvision-architecture.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/AI-in-a-Box/52ff38a2333df2ae1d918f0db4da18e9e717ca57/media/images/edgeai-customvision-architecture.png -------------------------------------------------------------------------------- /media/images/edgeai-in-a-box.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/AI-in-a-Box/52ff38a2333df2ae1d918f0db4da18e9e717ca57/media/images/edgeai-in-a-box.png -------------------------------------------------------------------------------- /media/images/edgeai-tree-options.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/AI-in-a-Box/52ff38a2333df2ae1d918f0db4da18e9e717ca57/media/images/edgeai-tree-options.png -------------------------------------------------------------------------------- /media/images/justthebox.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/AI-in-a-Box/52ff38a2333df2ae1d918f0db4da18e9e717ca57/media/images/justthebox.png -------------------------------------------------------------------------------- /media/images/logo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/AI-in-a-Box/52ff38a2333df2ae1d918f0db4da18e9e717ca57/media/images/logo.png -------------------------------------------------------------------------------- /media/images/mindmap.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/AI-in-a-Box/52ff38a2333df2ae1d918f0db4da18e9e717ca57/media/images/mindmap.png -------------------------------------------------------------------------------- /media/images/ml-in-a-box.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/AI-in-a-Box/52ff38a2333df2ae1d918f0db4da18e9e717ca57/media/images/ml-in-a-box.png -------------------------------------------------------------------------------- /media/images/sk-bot-in-a-box.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/AI-in-a-Box/52ff38a2333df2ae1d918f0db4da18e9e717ca57/media/images/sk-bot-in-a-box.png --------------------------------------------------------------------------------