├── .github ├── CODE_OF_CONDUCT.md ├── ISSUE_TEMPLATE.md ├── PULL_REQUEST_TEMPLATE.md └── workflows │ └── bicep-audit.yml ├── .gitignore ├── CHANGELOG.md ├── CONTRIBUTING.md ├── LICENSE.md ├── README.ja.md ├── README.md ├── advanced-logging └── README.md ├── assets ├── EnterpriseAOAI-Architecture-Adv-Flow.svg ├── EnterpriseAOAI-Architecture-Adv.svg ├── EnterpriseAOAI-Architecture.png ├── EnterpriseLogging_0.mp4 ├── apim-config-adv-0.png ├── apim-config-apikey.png ├── apim-logger.json ├── apim-policy-event-hub-logging.xml ├── apim_config_0.0.png ├── apim_config_0.1.png ├── apim_config_0.png ├── apim_config_1.png ├── apim_config_2.png ├── apim_config_3.png ├── apim_config_4.png ├── apim_regen_keys.png ├── event-hub-explorer.png ├── monitor_0.png ├── monitor_1.png └── video.png ├── deploy-terraform ├── .gitignore ├── README.md ├── apim.tf ├── cognitive.tf ├── eventhub.tf ├── keyvault.tf ├── main.tf ├── network.tf └── variables.tf └── deploy ├── README.md ├── main.bicep ├── main.parameters.json └── modules └── cognitiveservices.bicep /.github/CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | # Microsoft Open Source Code of Conduct 2 | 3 | This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). 4 | 5 | Resources: 6 | 7 | - [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/) 8 | - [Microsoft Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) 9 | - Contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with questions or concerns 10 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE.md: -------------------------------------------------------------------------------- 1 | 4 | > Please provide us with the following information: 5 | > --------------------------------------------------------------- 6 | 7 | ### This issue is for a: (mark with an `x`) 8 | ``` 9 | - [ ] bug report -> please search issues before submitting 10 | - [ ] feature request 11 | - [ ] documentation issue or request 12 | - [ ] regression (a behavior that used to work and stopped in a new release) 13 | ``` 14 | 15 | ### Minimal steps to reproduce 16 | > 17 | 18 | ### Any log messages given by the failure 19 | > 20 | 21 | ### Expected/desired behavior 22 | > 23 | 24 | ### OS and Version? 25 | > Windows 7, 8 or 10. Linux (which distribution). macOS (Yosemite? El Capitan? Sierra?) 26 | 27 | ### Versions 28 | > 29 | 30 | ### Mention any other details that might be useful 31 | 32 | > --------------------------------------------------------------- 33 | > Thanks! We'll be in touch soon. 34 | -------------------------------------------------------------------------------- /.github/PULL_REQUEST_TEMPLATE.md: -------------------------------------------------------------------------------- 1 | ## Purpose 2 | 3 | * ... 4 | 5 | ## Does this introduce a breaking change? 6 | 7 | ``` 8 | [ ] Yes 9 | [ ] No 10 | ``` 11 | 12 | ## Pull Request Type 13 | What kind of change does this Pull Request introduce? 14 | 15 | 16 | ``` 17 | [ ] Bugfix 18 | [ ] Feature 19 | [ ] Code style update (formatting, local variables) 20 | [ ] Refactoring (no functional changes, no api changes) 21 | [ ] Documentation content changes 22 | [ ] Other... Please describe: 23 | ``` 24 | 25 | ## How to Test 26 | * Get the code 27 | 28 | ``` 29 | git clone [repo-address] 30 | cd [repo-name] 31 | git checkout [branch-name] 32 | npm install 33 | ``` 34 | 35 | * Test the code 36 | 37 | ``` 38 | ``` 39 | 40 | ## What to Check 41 | Verify that the following are valid 42 | * ... 43 | 44 | ## Other Information 45 | -------------------------------------------------------------------------------- /.github/workflows/bicep-audit.yml: -------------------------------------------------------------------------------- 1 | name: Validate bicep templates 2 | on: 3 | push: 4 | branches: 5 | - main 6 | paths: 7 | - "**/*.bicep" 8 | pull_request: 9 | branches: 10 | - main 11 | paths: 12 | - "**/*.bicep" 13 | workflow_dispatch: 14 | 15 | jobs: 16 | build: 17 | runs-on: ubuntu-latest 18 | permissions: 19 | security-events: write 20 | steps: 21 | - name: Checkout 22 | uses: actions/checkout@v4 23 | 24 | - name: Run Microsoft Security DevOps Analysis 25 | uses: microsoft/security-devops-action@preview 26 | id: msdo 27 | continue-on-error: true 28 | with: 29 | tools: templateanalyzer 30 | 31 | - name: Upload alerts to Security tab 32 | uses: github/codeql-action/upload-sarif@v3 33 | if: github.repository_owner == 'Azure-Samples' 34 | with: 35 | sarif_file: ${{ steps.msdo.outputs.sarifFile }} 36 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | ## Ignore Visual Studio temporary files, build results, and 2 | ## files generated by popular Visual Studio add-ons. 3 | ## 4 | ## Get latest from https://github.com/github/gitignore/blob/master/VisualStudio.gitignore 5 | 6 | # User-specific files 7 | *.rsuser 8 | *.suo 9 | *.user 10 | *.userosscache 11 | *.sln.docstates 12 | 13 | # User-specific files (MonoDevelop/Xamarin Studio) 14 | *.userprefs 15 | 16 | # Mono auto generated files 17 | mono_crash.* 18 | 19 | # Build results 20 | [Dd]ebug/ 21 | [Dd]ebugPublic/ 22 | [Rr]elease/ 23 | [Rr]eleases/ 24 | x64/ 25 | x86/ 26 | [Aa][Rr][Mm]/ 27 | [Aa][Rr][Mm]64/ 28 | bld/ 29 | [Bb]in/ 30 | [Oo]bj/ 31 | [Ll]og/ 32 | [Ll]ogs/ 33 | 34 | # Visual Studio 2015/2017 cache/options directory 35 | .vs/ 36 | # Uncomment if you have tasks that create the project's static files in wwwroot 37 | #wwwroot/ 38 | 39 | # Visual Studio 2017 auto generated files 40 | Generated\ Files/ 41 | 42 | # MSTest test Results 43 | [Tt]est[Rr]esult*/ 44 | [Bb]uild[Ll]og.* 45 | 46 | # NUnit 47 | *.VisualState.xml 48 | TestResult.xml 49 | nunit-*.xml 50 | 51 | # Build Results of an ATL Project 52 | [Dd]ebugPS/ 53 | [Rr]eleasePS/ 54 | dlldata.c 55 | 56 | # Benchmark Results 57 | BenchmarkDotNet.Artifacts/ 58 | 59 | # .NET Core 60 | project.lock.json 61 | project.fragment.lock.json 62 | artifacts/ 63 | 64 | # StyleCop 65 | StyleCopReport.xml 66 | 67 | # Files built by Visual Studio 68 | *_i.c 69 | *_p.c 70 | *_h.h 71 | *.ilk 72 | *.meta 73 | *.obj 74 | *.iobj 75 | *.pch 76 | *.pdb 77 | *.ipdb 78 | *.pgc 79 | *.pgd 80 | *.rsp 81 | *.sbr 82 | *.tlb 83 | *.tli 84 | *.tlh 85 | *.tmp 86 | *.tmp_proj 87 | *_wpftmp.csproj 88 | *.log 89 | *.vspscc 90 | *.vssscc 91 | .builds 92 | *.pidb 93 | *.svclog 94 | *.scc 95 | 96 | # Chutzpah Test files 97 | _Chutzpah* 98 | 99 | # Visual C++ cache files 100 | ipch/ 101 | *.aps 102 | *.ncb 103 | *.opendb 104 | *.opensdf 105 | *.sdf 106 | *.cachefile 107 | *.VC.db 108 | *.VC.VC.opendb 109 | 110 | # Visual Studio profiler 111 | *.psess 112 | *.vsp 113 | *.vspx 114 | *.sap 115 | 116 | # Visual Studio Trace Files 117 | *.e2e 118 | 119 | # TFS 2012 Local Workspace 120 | $tf/ 121 | 122 | # Guidance Automation Toolkit 123 | *.gpState 124 | 125 | # ReSharper is a .NET coding add-in 126 | _ReSharper*/ 127 | *.[Rr]e[Ss]harper 128 | *.DotSettings.user 129 | 130 | # TeamCity is a build add-in 131 | _TeamCity* 132 | 133 | # DotCover is a Code Coverage Tool 134 | *.dotCover 135 | 136 | # AxoCover is a Code Coverage Tool 137 | .axoCover/* 138 | !.axoCover/settings.json 139 | 140 | # Visual Studio code coverage results 141 | *.coverage 142 | *.coveragexml 143 | 144 | # NCrunch 145 | _NCrunch_* 146 | .*crunch*.local.xml 147 | nCrunchTemp_* 148 | 149 | # MightyMoose 150 | *.mm.* 151 | AutoTest.Net/ 152 | 153 | # Web workbench (sass) 154 | .sass-cache/ 155 | 156 | # Installshield output folder 157 | [Ee]xpress/ 158 | 159 | # DocProject is a documentation generator add-in 160 | DocProject/buildhelp/ 161 | DocProject/Help/*.HxT 162 | DocProject/Help/*.HxC 163 | DocProject/Help/*.hhc 164 | DocProject/Help/*.hhk 165 | DocProject/Help/*.hhp 166 | DocProject/Help/Html2 167 | DocProject/Help/html 168 | 169 | # Click-Once directory 170 | publish/ 171 | 172 | # Publish Web Output 173 | *.[Pp]ublish.xml 174 | *.azurePubxml 175 | # Note: Comment the next line if you want to checkin your web deploy settings, 176 | # but database connection strings (with potential passwords) will be unencrypted 177 | *.pubxml 178 | *.publishproj 179 | 180 | # Microsoft Azure Web App publish settings. Comment the next line if you want to 181 | # checkin your Azure Web App publish settings, but sensitive information contained 182 | # in these scripts will be unencrypted 183 | PublishScripts/ 184 | 185 | # NuGet Packages 186 | *.nupkg 187 | # NuGet Symbol Packages 188 | *.snupkg 189 | # The packages folder can be ignored because of Package Restore 190 | **/[Pp]ackages/* 191 | # except build/, which is used as an MSBuild target. 192 | !**/[Pp]ackages/build/ 193 | # Uncomment if necessary however generally it will be regenerated when needed 194 | #!**/[Pp]ackages/repositories.config 195 | # NuGet v3's project.json files produces more ignorable files 196 | *.nuget.props 197 | *.nuget.targets 198 | 199 | # Microsoft Azure Build Output 200 | csx/ 201 | *.build.csdef 202 | 203 | # Microsoft Azure Emulator 204 | ecf/ 205 | rcf/ 206 | 207 | # Windows Store app package directories and files 208 | AppPackages/ 209 | BundleArtifacts/ 210 | Package.StoreAssociation.xml 211 | _pkginfo.txt 212 | *.appx 213 | *.appxbundle 214 | *.appxupload 215 | 216 | # Visual Studio cache files 217 | # files ending in .cache can be ignored 218 | *.[Cc]ache 219 | # but keep track of directories ending in .cache 220 | !?*.[Cc]ache/ 221 | 222 | # Others 223 | ClientBin/ 224 | ~$* 225 | *~ 226 | *.dbmdl 227 | *.dbproj.schemaview 228 | *.jfm 229 | *.pfx 230 | *.publishsettings 231 | orleans.codegen.cs 232 | 233 | # Including strong name files can present a security risk 234 | # (https://github.com/github/gitignore/pull/2483#issue-259490424) 235 | #*.snk 236 | 237 | # Since there are multiple workflows, uncomment next line to ignore bower_components 238 | # (https://github.com/github/gitignore/pull/1529#issuecomment-104372622) 239 | #bower_components/ 240 | 241 | # RIA/Silverlight projects 242 | Generated_Code/ 243 | 244 | # Backup & report files from converting an old project file 245 | # to a newer Visual Studio version. Backup files are not needed, 246 | # because we have git ;-) 247 | _UpgradeReport_Files/ 248 | Backup*/ 249 | UpgradeLog*.XML 250 | UpgradeLog*.htm 251 | ServiceFabricBackup/ 252 | *.rptproj.bak 253 | 254 | # SQL Server files 255 | *.mdf 256 | *.ldf 257 | *.ndf 258 | 259 | # Business Intelligence projects 260 | *.rdl.data 261 | *.bim.layout 262 | *.bim_*.settings 263 | *.rptproj.rsuser 264 | *- [Bb]ackup.rdl 265 | *- [Bb]ackup ([0-9]).rdl 266 | *- [Bb]ackup ([0-9][0-9]).rdl 267 | 268 | # Microsoft Fakes 269 | FakesAssemblies/ 270 | 271 | # GhostDoc plugin setting file 272 | *.GhostDoc.xml 273 | 274 | # Node.js Tools for Visual Studio 275 | .ntvs_analysis.dat 276 | node_modules/ 277 | 278 | # Visual Studio 6 build log 279 | *.plg 280 | 281 | # Visual Studio 6 workspace options file 282 | *.opt 283 | 284 | # Visual Studio 6 auto-generated workspace file (contains which files were open etc.) 285 | *.vbw 286 | 287 | # Visual Studio LightSwitch build output 288 | **/*.HTMLClient/GeneratedArtifacts 289 | **/*.DesktopClient/GeneratedArtifacts 290 | **/*.DesktopClient/ModelManifest.xml 291 | **/*.Server/GeneratedArtifacts 292 | **/*.Server/ModelManifest.xml 293 | _Pvt_Extensions 294 | 295 | # Paket dependency manager 296 | .paket/paket.exe 297 | paket-files/ 298 | 299 | # FAKE - F# Make 300 | .fake/ 301 | 302 | # CodeRush personal settings 303 | .cr/personal 304 | 305 | # Python Tools for Visual Studio (PTVS) 306 | __pycache__/ 307 | *.pyc 308 | 309 | # Cake - Uncomment if you are using it 310 | # tools/** 311 | # !tools/packages.config 312 | 313 | # Tabs Studio 314 | *.tss 315 | 316 | # Telerik's JustMock configuration file 317 | *.jmconfig 318 | 319 | # BizTalk build output 320 | *.btp.cs 321 | *.btm.cs 322 | *.odx.cs 323 | *.xsd.cs 324 | 325 | # OpenCover UI analysis results 326 | OpenCover/ 327 | 328 | # Azure Stream Analytics local run output 329 | ASALocalRun/ 330 | 331 | # MSBuild Binary and Structured Log 332 | *.binlog 333 | 334 | # NVidia Nsight GPU debugger configuration file 335 | *.nvuser 336 | 337 | # MFractors (Xamarin productivity tool) working folder 338 | .mfractor/ 339 | 340 | # Local History for Visual Studio 341 | .localhistory/ 342 | 343 | # BeatPulse healthcheck temp database 344 | healthchecksdb 345 | 346 | # Backup folder for Package Reference Convert tool in Visual Studio 2017 347 | MigrationBackup/ 348 | 349 | # Ionide (cross platform F# VS Code tools) working folder 350 | .ionide/ 351 | -------------------------------------------------------------------------------- /CHANGELOG.md: -------------------------------------------------------------------------------- 1 | ## [project-title] Changelog 2 | 3 | 4 | # x.y.z (yyyy-mm-dd) 5 | 6 | *Features* 7 | * ... 8 | 9 | *Bug Fixes* 10 | * ... 11 | 12 | *Breaking Changes* 13 | * ... 14 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing to [project-title] 2 | 3 | This project welcomes contributions and suggestions. Most contributions require you to agree to a 4 | Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us 5 | the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com. 6 | 7 | When you submit a pull request, a CLA bot will automatically determine whether you need to provide 8 | a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions 9 | provided by the bot. You will only need to do this once across all repos using our CLA. 10 | 11 | This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). 12 | For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or 13 | contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments. 14 | 15 | - [Code of Conduct](#coc) 16 | - [Issues and Bugs](#issue) 17 | - [Feature Requests](#feature) 18 | - [Submission Guidelines](#submit) 19 | 20 | ## Code of Conduct 21 | Help us keep this project open and inclusive. Please read and follow our [Code of Conduct](https://opensource.microsoft.com/codeofconduct/). 22 | 23 | ## Found an Issue? 24 | If you find a bug in the source code or a mistake in the documentation, you can help us by 25 | [submitting an issue](#submit-issue) to the GitHub Repository. Even better, you can 26 | [submit a Pull Request](#submit-pr) with a fix. 27 | 28 | ## Want a Feature? 29 | You can *request* a new feature by [submitting an issue](#submit-issue) to the GitHub 30 | Repository. If you would like to *implement* a new feature, please submit an issue with 31 | a proposal for your work first, to be sure that we can use it. 32 | 33 | * **Small Features** can be crafted and directly [submitted as a Pull Request](#submit-pr). 34 | 35 | ## Submission Guidelines 36 | 37 | ### Submitting an Issue 38 | Before you submit an issue, search the archive, maybe your question was already answered. 39 | 40 | If your issue appears to be a bug, and hasn't been reported, open a new issue. 41 | Help us to maximize the effort we can spend fixing issues and adding new 42 | features, by not reporting duplicate issues. Providing the following information will increase the 43 | chances of your issue being dealt with quickly: 44 | 45 | * **Overview of the Issue** - if an error is being thrown a non-minified stack trace helps 46 | * **Version** - what version is affected (e.g. 0.1.2) 47 | * **Motivation for or Use Case** - explain what are you trying to do and why the current behavior is a bug for you 48 | * **Browsers and Operating System** - is this a problem with all browsers? 49 | * **Reproduce the Error** - provide a live example or a unambiguous set of steps 50 | * **Related Issues** - has a similar issue been reported before? 51 | * **Suggest a Fix** - if you can't fix the bug yourself, perhaps you can point to what might be 52 | causing the problem (line of code or commit) 53 | 54 | You can file new issues by providing the above information at the corresponding repository's issues link: https://github.com/[organization-name]/[repository-name]/issues/new]. 55 | 56 | ### Submitting a Pull Request (PR) 57 | Before you submit your Pull Request (PR) consider the following guidelines: 58 | 59 | * Search the repository (https://github.com/[organization-name]/[repository-name]/pulls) for an open or closed PR 60 | that relates to your submission. You don't want to duplicate effort. 61 | 62 | * Make your changes in a new git fork: 63 | 64 | * Commit your changes using a descriptive commit message 65 | * Push your fork to GitHub: 66 | * In GitHub, create a pull request 67 | * If we suggest changes then: 68 | * Make the required updates. 69 | * Rebase your fork and force push to your GitHub repository (this will update your Pull Request): 70 | 71 | ```shell 72 | git rebase master -i 73 | git push -f 74 | ``` 75 | 76 | That's it! Thank you for your contribution! 77 | -------------------------------------------------------------------------------- /LICENSE.md: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) Microsoft Corporation. 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE -------------------------------------------------------------------------------- /README.ja.md: -------------------------------------------------------------------------------- 1 | # エンタープライズ向けAzure OpenAIアプリ基盤 2 | 3 | エンタープライズ向けAzure OpenAIアプリ基盤の参照アーキテクチャのデプロイ方法の詳細を示すリポジトリ。 4 | 5 | ## ソリューションの主な利点: 6 | * 責任あるAI利用のためのプロンプトと応答監視。 ログ情報には、ユーザーがモデルに送信しているテキストと、モデルから受信されているテキストが含まれます。これにより、モデルが企業環境内およびサービスの承認されたユースケース内で責任を持って使用されるようになります。 7 | * 利用制御とスロットリング制御により、 さまざまなユーザグループに対してきめ細かなアクセス制御が可能になります。 8 | * 高可用性構成により、 トラフィックが 1 つの Azure OpenAI サービスの制限を超えた場合でも、ユーザー要求が満たされるようにします。 9 | * 高セキュリティを実現するために、最小特権の原則に従ってAzure Active DirectoryのRBACを活用します。 10 | 11 | [![video](assets/video.png)](https://clipchamp.com/watch/WX92A7nDyR4 'link') 12 | 13 | ## リファレンスアーキテクチャ 14 | ![img](/assets/EnterpriseAOAI-Architecture.png) 15 | 16 | ## 機能 17 | 18 | このプロジェクト フレームワークは、次の機能を提供します: 19 | 20 | * OpenAI 使用状況メトリックのエンタープライズログ: 21 | * トークン利用 22 | * モデル利用 23 | * プロンプト入力 24 | * ユーザー統計 25 | * プロンプト応答 26 | * リージョンのフェールオーバーによるOpenAI サービスの高可用性 27 | * 最新のOpenAIライブラリとの統合 28 | * [OpenAI](https://github.com/openai/openai-python/) 29 | * [LangChain](https://python.langchain.com/en/latest/) 30 | * [Llama-index](https://gpt-index.readthedocs.io/en/latest/) 31 | 32 | ## はじめ方 33 | 34 | ### 前提条件 35 | - [Azure サブスクリプション](https://azure.microsoft.com/en-us/get-started/) 36 | - [Azure OpenAI 利用許可](https://aka.ms/oai/access) 37 | - 38 | ### インストール 39 | アーティファクトのプロビジョニングは、まず、次に示すソリューションアーティファクトをプロビジョニングします: 40 | 41 | - [Azure OpenAI Cognitive Service]( https://azure.microsoft.com/en-us/products/cognitive-services/openai-service/) 42 | - [Azure API Management](https://azure.microsoft.com/services/api-management/) 43 | - [Azure Monitor](https://azure.microsoft.com/services/monitor/) 44 | - [Azure Application Gateway](https://azure.microsoft.com/services/application-gateway/) 45 | - [Azure Virtual Network](https://azure.microsoft.com/services/virtual-network/) 46 | 47 | ### マネージドサービス 48 | - [Azure Key Vault](https://azure.microsoft.com/services/key-vault/) 49 | - [Azure Storage](https://azure.microsoft.com/services/storage/) 50 | - [Azure Active Directory](https://azure.microsoft.com/services/active-directory/) 51 | 52 | ## 構成 53 | 54 | ### Azure OpenAI 55 | - まずは、Azure OpenAIのリソースをプロビジョニングします。現在のプライマリリージョンは米国東部であり新しいモデルと容量は他のリージョンよりも先にこの場所でプロビジョニングされることに注意してください。 [リソースのプロビジョニング](https://portal.azure.com/?microsoft_azure_marketplace_ItemHideKey=microsoft_openai_tip#create/Microsoft.CognitiveServicesOpenAI) 56 | 57 | - リソースがプロビジョニングされたら、選択したモデルを使用してDeploymentを作成します: [モデルのデプロイ](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal#deploy-a-model) 58 | 59 | - モデルがデプロイされたら、OpenAI スタジオに遷移して、新しく作成したモデルをスタジオのプレイグラウンドでテストします。 [oai.azure.com/portal](oai.azure.com/portal) 60 | 61 | 62 | ### API Management 63 | 64 | - Azure ポータルを使用してプロビジョニングできます :[リソースのプロビジョニング](https://learn.microsoft.com/en-us/azure/api-management/get-started-create-service-instance) 65 | - API Management サービスがプロビジョニングされたら、サービスの OpenAPI 仕様を使用して OpenAI API レイヤーをインポートできます。 66 | - [インポート手順](https://learn.microsoft.com/en-us/azure/api-management/import-and-publish#go-to-your-api-management-instance) 67 | - [APIM - API] ブレードを開き、既存の API の [ インポート ] オプションを選択します。 68 | ![img](/assets/apim_config_0.0.png) 69 | - [ 更新 ] オプションを選択して、API を現在の OpenAI 仕様に更新します。 70 | - Completions OpenAPI - https://raw.githubusercontent.com/Azure/azure-rest-api-specs/main/specification/cognitiveservices/data-plane/AzureOpenAI/inference/preview/2023-03-15-preview/inference.json 71 | ![img](/assets/apim_config_0.1.png) 72 | - すべての API 操作のために: 73 | - 設定 で、OpenAI ライブラリの仕様と一致するように サブスクリプション - ヘッダー名] を "api-key" に設定します。 74 | ![img](assets/apim-config-apikey.png) 75 | - "set-headers" のinbound ruleを構成して、"api-key" ヘッダーパラメーターを OpenAI サービスからの API シークレットキーの値で追加/上書きします。OpenAIキーを見つけるための手順はここにあります: [キーを取得する](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/quickstart?pivots=programming-language-python#retrieve-key-and-endpoint) 76 | ![img](/assets/apim_config_1.png) 77 | - デプロイされたOpenAIサービスのエンドポイントにバックエンドサービスを`/openai`として構成し、既存のエンドポイントを必ず上書きしてください。 78 | - 例: https://< yourservicename >.openai.azure.com/openai 79 | - [エンドポイントの取得](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/quickstart?pivots=programming-language-python#retrieve-key-and-endpoint) 80 | ![img](/assets/apim_config_2.png) 81 | - 診断ログの設定を構成します: 82 | - Sampling rateを100%に設定します 83 | - 「Number of payload bytes to log」を最大値に設定します。 84 | ![img](/assets/apim_config_3.png) 85 | 86 | - APIのテスト 87 | - "デプロイ ID"、"API バージョン"、およびサンプル プロンプトを指定して、エンドポイントをテストします: 88 | ![img](/assets/apim_config_4.png) 89 | 90 | 91 | #### サブスクリプションのアクセス制御 92 | API Management を使用すると、API プロバイダーは API を不正使用から保護し、さまざまな API 製品レベルの価値を生み出すことができます。API 管理レイヤーを使用して受信要求を調整することは、Azure API 管理の重要な役割です。要求の速度または転送された要求/データの合計を制御することによって。 93 |
APIM レイヤーの構成の詳細: https://learn.microsoft.com/en-us/azure/api-management/api-management-sample-flexible-throttling 94 | 95 | ### OpenAIのログ記録 96 | - API 管理レイヤーを構成したら、サブスクリプション キー パラメーターをcompletion要求に追加することで、API レイヤーを使用するように既存の OpenAI Python コードを構成できます。 例： 97 | ```python 98 | import openai 99 | 100 | openai.api_type = "azure" 101 | openai.api_base = "https://xxxxxxxxx.azure-api.net/" # APIM Endpoint 102 | openai.api_version = "2023-03-15-preview" 103 | openai.api_key = "APIM SUBSCRIPTION KEY" #DO NOT USE ACTUAL AZURE OPENAI SERVICE KEY 104 | 105 | 106 | response = openai.Completion.create(engine="modelname", 107 | prompt="prompt text", temperature=1, 108 | max_tokens=200, top_p=0.5, 109 | frequency_penalty=0, 110 | presence_penalty=0, 111 | stop=None) 112 | 113 | ``` 114 | 115 | 116 | 117 | ## デモ 118 | 119 | - OpenAI 要求が Azure Monitor サービスへのログ記録を開始したら、Log Analytics クエリを使用してサービスの使用状況の分析を開始できます。 120 | - [Log Analytics チュートリアル](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/log-analytics-tutorial) 121 | - テーブルの名前は "ApiManagementGatewayLogs"テーブルの名前は 122 | - BackendResponseBody フィールドには、テキスト補完、トークンおよびモデル情報を含む OpenAI サービスからの json 応答が含まれています。 123 | - IP とモデルによってトークンの使用状況を識別するクエリの例: 124 | ```kusto 125 | ApiManagementGatewayLogs 126 | | where OperationId == 'completions_create' 127 | | extend modelkey = substring(parse_json(BackendResponseBody)['model'], 0, indexof(parse_json(BackendResponseBody)['model'], '-', 0, -1, 2)) 128 | | extend model = tostring(parse_json(BackendResponseBody)['model']) 129 | | extend prompttokens = parse_json(parse_json(BackendResponseBody)['usage'])['prompt_tokens'] 130 | | extend completiontokens = parse_json(parse_json(BackendResponseBody)['usage'])['completion_tokens'] 131 | | extend totaltokens = parse_json(parse_json(BackendResponseBody)['usage'])['total_tokens'] 132 | | extend ip = CallerIpAddress 133 | | where model != '' 134 | | summarize 135 | sum(todecimal(prompttokens)), 136 | sum(todecimal(completiontokens)), 137 | sum(todecimal(totaltokens)), 138 | avg(todecimal(totaltokens)) 139 | by ip, model 140 | ``` 141 | ![img](/assets/monitor_0.png) 142 | - プロンプトの完了を監視するクエリの例: 143 | ```kusto 144 | ApiManagementGatewayLogs 145 | | where OperationId == 'completions_create' 146 | | extend model = tostring(parse_json(BackendResponseBody)['model']) 147 | | extend prompttokens = parse_json(parse_json(BackendResponseBody)['usage'])['prompt_tokens'] 148 | | extend prompttext = substring(parse_json(parse_json(BackendResponseBody)['choices'])[0], 0, 100) 149 | ``` 150 | ![img](/assets/monitor_1.png) 151 | 152 | 153 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Enterprise Azure OpenAI 2 | 3 | Repository detailing the deployment of an Enterprise Azure OpenAI reference architecture. 4 |
Link: [Azure Architecture Center - Monitor OpenAI Models](https://learn.microsoft.com/en-us/azure/architecture/example-scenario/ai/log-monitor-azure-openai) 5 | 6 | An [advanced pattern](/advanced-logging/README.md) is available for customers using models with larger token sizes, want to perform advanced analytics on the prompts and responses, or have requirements to send these events to another type of data store. 7 | 8 | ## Key Solution Advantages: 9 | * Comprehensive logging of Azure OpenAI model execution tracked to Source IP address. Log information includes what text users are submitting to the model as well as text being received back from the model. This ensures models are being used responsibly within the corporate environment and within the approved use cases of the service. 10 | * Advanced Usage and Throttling controls allow fine-grained access controls for different user groups without allowing access to underlying service keys. 11 | * High availability of the model APIs to ensure user requests are met even if the traffic exceeds the limits of a single Azure OpenAI Service. 12 | * Secure use of the service by ensuring role-based access managed via Azure Active Directory follows principle of least privilege. 13 | 14 |
15 | 16 |
17 | 18 | ## Reference Architecture 19 | ![img](/assets/EnterpriseAOAI-Architecture.png) 20 | 21 | 22 | 23 | 27 | 28 | 29 | 30 | 31 |
1. Client applications can access Azure OpenAI endpoints to perform text generation (completions) and model training (fine-tuning) endpoints to leverage the power of large language models.
24 |
25 | 2. Next-Gen Firewall Appliance (Optional) - Provides deep packet level inspection for network traffic to the OpenAI Models. 26 | 		3. API Management Gateway enables security controls, auditing, and monitoring of the Azure OpenAI models.  Security access is granted via AAD Groups with subscription based access permissions in APIM.  Auditing is enabled via Azure Monitor request logging for all interactions with the models.  Monitoring enables detailed AOAI model usage KPIs/Metrics.4. API Management Gateway connects to all Azure resources via Private Link to ensure all traffic is secured by private endpoints and contained to private network.

5. Multiple Azure OpenAI instances enable scale out of API usage to ensure high-availability and disaster recovery for the service.
32 | 33 | ## Features 34 | 35 | This project framework provides the following features: 36 | 37 | * Enterprise logging of OpenAI usage metrics: 38 | * Token Usage 39 | * Model Usage 40 | * Prompt Input 41 | * User statistics 42 | * Prompt Response 43 | * High Availability of OpenAI service with region failover. 44 | * Integration with latest OpenAI libraries- 45 | * [OpenAI](https://github.com/openai/openai-python/) 46 | * [LangChain](https://python.langchain.com/en/latest/) 47 | * [Llama-index](https://gpt-index.readthedocs.io/en/latest/) 48 | 49 | ## Getting Started 50 | 51 | ### Prerequisites 52 | - [Azure Subscription](https://azure.microsoft.com/en-us/get-started/) 53 | - [Azure OpenAI Application](https://aka.ms/oai/access) 54 | 55 | ### Installation 56 | Provisioning artifacts, begin by provisioning the solution artifacts listed below: 57 | 58 | - [Azure OpenAI Cognitive Service]( https://azure.microsoft.com/en-us/products/cognitive-services/openai-service/) 59 | - [Azure API Management](https://azure.microsoft.com/services/api-management/) 60 | - [Azure Monitor](https://azure.microsoft.com/services/monitor/) 61 | 62 | (Optional) 63 | - Next-Gen Firewall Appliance 64 | - [Azure Application Gateway](https://azure.microsoft.com/services/application-gateway/) 65 | - [Azure Virtual Network](https://azure.microsoft.com/services/virtual-network/) 66 | 67 | ### Managed Services 68 | - [Azure Key Vault](https://azure.microsoft.com/services/key-vault/) 69 | - [Azure Storage](https://azure.microsoft.com/services/storage/) 70 | - [Azure Active Directory](https://azure.microsoft.com/services/active-directory/) 71 | 72 | ## Configuration 73 | 74 | ### Azure OpenAI 75 | - To begin, provision a resource for Azure OpenAI in your preferred region: [Provision resource](https://portal.azure.com/?microsoft_azure_marketplace_ItemHideKey=microsoft_openai_tip#create/Microsoft.CognitiveServicesOpenAI) 76 | 77 | - Once the resource is provisioned, create a deployment with model of choice: [Deploy Model](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal#deploy-a-model) 78 | 79 | - After the model has been deployed, go to the OpenAI studio to test your newly created model with the studio playground: [oai.azure.com/portal](oai.azure.com/portal) 80 | 81 | - Note down Key1 from the Azure OpenAI instance by opening the Azure OpenAI instance, then from the Resource Management section of the left menu, select Keys and Endpoints. 82 | 83 | ### Azure Key Vault 84 | Provision an Azure Key Vault Resource: [Deploy Key Vault](https://portal.azure.com/#create/Microsoft.KeyVault) 85 | 86 | Once deployed, add Key1 from the Azure OpenAI instance as a secret: [Add a Secret](https://learn.microsoft.com/en-us/azure/key-vault/secrets/quick-create-portal) 87 | 88 | ### API Management Config 89 | 90 | - API Management can be provisioned through Azure Portal :[Provision resource](https://learn.microsoft.com/en-us/azure/api-management/get-started-create-service-instance) 91 | - Once the API Management service has been provisioned, follow this documentation to configure access permissions for the APIM instance on the Azure Key Vaults secrets. 92 | 93 | - Named Value Setup 94 | - Follow this documentation to create a named value linked to Key1 in the Azure Key Vault created earlier: [Add a plain or secret value to API Management](https://learn.microsoft.com/en-us/azure/api-management/api-management-howto-properties?tabs=azure-portal#add-a-plain-or-secret-value-to-api-management) 95 | 96 | - Backend Setup 97 | - From the left menu in API Management select Backends, then create a new backend. 98 | - Configure the backend service to the endpoint of your deployed OpenAI service with /openai as the path: 99 | - Example: https://< yourservicename >.openai.azure.com/openai 100 | - [Retrieve endpoint](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/quickstart?pivots=programming-language-python#retrieve-key-and-endpoint) 101 | ![image](https://github.com/SOE-YoungS/openai-python-enterprise-logging/assets/95053834/00dca41d-cd4b-42dc-a296-a20517f83348) 102 | 103 | - Under Authorization for the backend, set a new header named "api-key" and set its value to the created named value, then save the config. 104 | ![image](https://github.com/SOE-YoungS/openai-python-enterprise-logging/assets/95053834/b8b4c267-792d-4769-a162-e0f3d77dfa40) 105 | 106 | - [API Import instructions](https://learn.microsoft.com/en-us/azure/api-management/import-and-publish#go-to-your-api-management-instance) 107 | - Open the APIM - API blade and Select the Import option for an existing API. 108 | ![img](/assets/apim_config_0.0.png) 109 | - Select the Update option to update the API to the current OpenAI specifications. 110 | - Completions OpenAPI - https://raw.githubusercontent.com/Azure/azure-rest-api-specs/main/specification/cognitiveservices/data-plane/AzureOpenAI/inference/stable/2023-05-15/inference.json 111 | ![img](/assets/apim_config_0.1.png) 112 | - (Optional) For Semantic Kernel compatibility "Update" the following Authoring API endpoints: 113 | - Authoring OpenAPI - https://raw.githubusercontent.com/Azure/azure-rest-api-specs/c183bb012de8e9e1d0d2e67a0994748df4747d2c/specification/cognitiveservices/data-plane/AzureOpenAI/authoring/stable/2022-12-01/azureopenai.json 114 | - For All API Operations: 115 | - Build your API inbound policy as below. 116 | ![image](https://github.com/SOE-YoungS/openai-python-enterprise-logging/assets/95053834/6f76ba77-c148-46c1-ab71-e800f992e984) 117 | - Configure the Diagnostic Logs settings: 118 | - Set the sampling rate to 100% 119 | - Set the "Number of payload bytes to log" as the maximum. 120 | ![img](/assets/apim_config_3.png) 121 | 122 | - Test API 123 | - Test the endpoint by providing the "deployment-id", "api-version" and a sample prompt: 124 | ![img](/assets/apim_config_4.png) 125 | 126 | 127 | #### (Optional) Subscription Access Control 128 | API Management allows API providers to protect their APIs from abuse and create value for different API product tiers. Use of API Management layer to throttle incoming requests is a key role of Azure API Management. Either by controlling the rate of requests or the total requests/data transferred. 129 | - Details for configuring APIM Layer : https://learn.microsoft.com/en-us/azure/api-management/api-management-sample-flexible-throttling 130 | 131 | - Details for enabling Subscription based access to API's: [API Management Subscriptions](https://learn.microsoft.com/en-us/azure/api-management/api-management-subscriptions) 132 | - Note: To enable API usage via existing libraries, such as Semantic Kernel etc... you can also adjust the "Subscription" settings for the API to the following, 133 |
![image](https://github.com/SOE-YoungS/openai-python-enterprise-logging/assets/95053834/97a30a2d-bb71-456c-ba8c-f376c5c255fa) 134 |
In the calling client (using the library), you then set the "OpenAI / Azure OpenAI" URL & Key to the values for your API base URL / APIM subscription key. 135 | 136 | ### Logging OpenAI completions 137 | - Once the API Management layer has been configured, you can configure existing OpenAI python code to use the API layer by adding the subscription key parameter to the completion request: 138 | Example: 139 | ```python 140 | import openai 141 | 142 | openai.api_type = "azure" 143 | openai.api_base = "https://xxxxxxxxx.azure-api.net/" # APIM Endpoint 144 | openai.api_version = "2023-05-15" 145 | openai.api_key = "APIM SUBSCRIPTION KEY" #DO NOT USE ACTUAL AZURE OPENAI SERVICE KEY 146 | 147 | 148 | response = openai.Completion.create(engine="modelname", 149 | prompt="prompt text", temperature=1, 150 | max_tokens=200, top_p=0.5, 151 | frequency_penalty=0, 152 | presence_penalty=0, 153 | stop=None) 154 | 155 | ``` 156 | 157 | 158 | 159 | ## Demo 160 | 161 | - Once OpenAI requests begin to log to the Azure Monitor service, you can begin to analyze the service usage using Log Analytics queries. 162 | - [Log Analytics Tutorial](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/log-analytics-tutorial) 163 | - The table should be named "ApiManagementGatewayLogs" 164 | - The BackendResponseBody field contains the json response from the OpenAI service which includes the text completion as well as the token and model information. 165 | - Example query to identify token usage by ip and model: 166 | ```kusto 167 | ApiManagementGatewayLogs 168 | | where tolower(OperationId) in ('completions_create','chatcompletions_create') 169 | | where ResponseCode == '200' 170 | | extend modelkey = substring(parse_json(BackendResponseBody)['model'], 0, indexof(parse_json(BackendResponseBody)['model'], '-', 0, -1, 2)) 171 | | extend model = tostring(parse_json(BackendResponseBody)['model']) 172 | | extend prompttokens = parse_json(parse_json(BackendResponseBody)['usage'])['prompt_tokens'] 173 | | extend completiontokens = parse_json(parse_json(BackendResponseBody)['usage'])['completion_tokens'] 174 | | extend totaltokens = parse_json(parse_json(BackendResponseBody)['usage'])['total_tokens'] 175 | | extend ip = CallerIpAddress 176 | | where model != '' 177 | | summarize 178 | sum(todecimal(prompttokens)), 179 | sum(todecimal(completiontokens)), 180 | sum(todecimal(totaltokens)), 181 | avg(todecimal(totaltokens)) 182 | by ip, model 183 | ``` 184 | ![img](/assets/monitor_0.png) 185 | - Example query to monitor prompt completions: 186 | ```kusto 187 | ApiManagementGatewayLogs 188 | | where tolower(OperationId) in ('completions_create','chatcompletions_create') 189 | | where ResponseCode == '200' 190 | | extend model = tostring(parse_json(BackendResponseBody)['model']) 191 | | extend prompttokens = parse_json(parse_json(BackendResponseBody)['usage'])['prompt_tokens'] 192 | | extend prompttext = substring(parse_json(parse_json(BackendResponseBody)['choices'])[0], 0, 100) 193 | ``` 194 | ![img](/assets/monitor_1.png) 195 | 196 | ## Resources 197 | - Azure API Management Policies for Azure OpenAI: https://github.com/mattfeltonma/azure-openai-apim 198 | - Advanced Retry Policies: https://github.com/ian-t-adams/azure-openai-api-m-retry/ 199 | 200 | ## Frequently Asked Questions 201 | - Where is the "Deploy to Azure" button? 202 | - In our experience, most enterprise cloud administrators first need to understand the solution before deploying it into an enterprise environment. The steps in this repo show how each component is deployed and configured so that they can be integrated into your existing deployment scripts. We do have [bicep templates](deploy) available to accelerate your development once you are familiar with the architecture. 203 | - Does the solution work with Private Endpoints? 204 | - Yes, to configure the solution to work with private endpoints you will need to: 205 | - Configure your OpenAI instance to use a [private endpoint](https://learn.microsoft.com/en-us/azure/cognitive-services/cognitive-services-virtual-networks?tabs=portal#use-private-endpoints). 206 | - Ensure API Management can resolve the private endpoint, if they are in different virtual networks this may require [vnet link](https://learn.microsoft.com/en-us/azure/dns/private-dns-virtual-network-links) 207 | - Configure API Management to use [internal networking](https://learn.microsoft.com/en-us/azure/api-management/api-management-using-with-internal-vnet?tabs=stv2#enable-vnet-connection) 208 | - Ensure that API Management endpoints are accessible by your client [link](https://learn.microsoft.com/en-us/azure/api-management/private-endpoint) 209 | - How do I secure my Azure OpenAI endpoints once this solution is deployed? 210 | - Option 1: Rotate all OpenAI Service keys once API Management is configured.
![image](https://github.com/SOE-YoungS/openai-python-enterprise-logging/assets/95053834/ea6fd8b6-3dff-461e-9bd6-ba267e2e2430) 211 | - Option 2: Disable key based access to Azure OpenAI Instance 212 | - Will impact Azure OpenAI Studio tool 213 | -------------------------------------------------------------------------------- /advanced-logging/README.md: -------------------------------------------------------------------------------- 1 | # Enterprise Azure OpenAI Advanced Logging 2 | 3 | Repository detailing an advanced logging pattern for the Azure OpenAI Service. 4 | 5 | ## Key Solution Advantages: 6 | * Supports models with larger token sizes The advanced logging pattern supports [capturing an event up to 200KB](https://learn.microsoft.com/en-us/azure/api-management/api-management-howto-log-event-hubs?tabs=PowerShell) while the [basic logging pattern](../README.md) supports a [maximum size of 8,192 bytes](https://learn.microsoft.com/en-us/azure/api-management/api-management-howto-app-insights). This allows the pattern to support capturing prompts and responses from models that support [larger token sizes such as GPT4](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/concepts/models#gpt-4-models-1). 7 | 8 | * Enforce strong identity controls and audit logging Authentication to the Azure OpenAI Service resource is restricted to Azure Active Directory identities such as service principals and managed identities. This is accomplished through the usage of an [Azure API Management custom policy](https://learn.microsoft.com/en-us/azure/api-management/validate-jwt-policy). The identity of the application making the request is captured in the logs streamed to the Azure Event Hub. 9 | 10 | * Log the information important to you [Azure API Management custom policies](https://learn.microsoft.com/en-us/azure/api-management/log-to-eventhub-policy) can be used to filter the information captured in the event to what is important to your organization. This can include prompts and responses, the number of tokens used, the Azure Active Directory identity making the call, or the model response time. This information can be used for both compliance and chargeback purposes. 11 | 12 | * Deliver events to a wide variety of data stores Events in this pattern are streamed to an [Azure Event Hub](https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-about). These events can be [further processed by the integration with Azure Stream Analytics](https://learn.microsoft.com/en-us/azure/event-hubs/process-data-azure-stream-analytics) and then delivered to data stores such as Azure SQL, Azure CosmosDB, or a PowerBI Dataset. 13 | 14 | ## Reference Architecture 15 | ![img](../assets/EnterpriseAOAI-Architecture-Adv.svg) 16 | 17 | 18 | ## Request Flow 19 | ![img](../assets/EnterpriseAOAI-Architecture-Adv-Flow.svg) 20 | 21 | ## Features 22 | 23 | This project framework provides the following features: 24 | 25 | * Enterprise logging of OpenAI usage metrics: 26 | * Prompt Input 27 | * Prompt Response 28 | * Token Usage 29 | * Model Usage 30 | * Application Usage 31 | * Model Response Times 32 | 33 | * High Availability of OpenAI service with region failover. 34 | 35 | * Integration with latest OpenAI libraries- 36 | * [OpenAI](https://github.com/openai/openai-python/) 37 | * [LangChain](https://python.langchain.com/en/latest/) 38 | * [Llama-index](https://gpt-index.readthedocs.io/en/latest/) 39 | 40 | ## Getting Started 41 | 42 | ### Prerequisites 43 | - [Azure Subscription](https://azure.microsoft.com/en-us/get-started/) 44 | - [Azure OpenAI Application](https://aka.ms/oai/access) 45 | 46 | ### Installation 47 | Provisioning artifacts, begin by provisioning the solution artifacts listed below: 48 | 49 | - [Azure OpenAI Cognitive Service]( https://azure.microsoft.com/en-us/products/cognitive-services/openai-service/) 50 | - [Azure API Management](https://azure.microsoft.com/services/api-management/) 51 | - [Azure Monitor](https://azure.microsoft.com/services/monitor/) 52 | 53 | (Optional) 54 | - Next-Gen Firewall Appliance 55 | - [Azure Virtual Network](https://azure.microsoft.com/services/virtual-network/) 56 | 57 | ### Managed Services 58 | - [Azure Key Vault](https://azure.microsoft.com/services/key-vault/) 59 | - [Azure Storage](https://azure.microsoft.com/services/storage/) 60 | - [Azure Active Directory](https://azure.microsoft.com/services/active-directory/) 61 | 62 | ## Deployment 63 | 64 | ### Azure OpenAI 65 | - To begin, provision a resource for Azure OpenAI in your preferred region. Please note the current primary region is East US, new models and capacity will be provisioned in this location before others: [Provision resource](https://portal.azure.com/?microsoft_azure_marketplace_ItemHideKey=microsoft_openai_tip#create/Microsoft.CognitiveServicesOpenAI) 66 | 67 | - Once the resource is provisioned, create a deployment with model of choice: [Deploy Model](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/how-to/create-resource?pivots=web-portal#deploy-a-model) 68 | 69 | - After the model has been deployed, go to the OpenAI studio to test your newly created model with the studio playground: [oai.azure.com/portal](oai.azure.com/portal) 70 | 71 | ### API Management Deployment 72 | 1. API Management can be provisioned through Azure Portal using the instructions at this [link](https://learn.microsoft.com/en-us/azure/api-management/get-started-create-service-instance). 73 | 74 | 2. Once the API Management service has been provisioned you must import your OpenAI API layer using the OpenAPI specification for the service. 75 | 76 | ### [API Import Instructions](https://learn.microsoft.com/en-us/azure/api-management/import-and-publish#go-to-your-api-management-instance) 77 | 78 | 1. Open the APIM - API blade and Select the Import option for an existing API. 79 | ![img](/assets/apim_config_0.0.png) 80 | 81 | 2. Select the Update option to update the API to the current OpenAI specifications. The Completions OpenAPI specification is found at https://raw.githubusercontent.com/Azure/azure-rest-api-specs/main/specification/cognitiveservices/data-plane/AzureOpenAI/inference/stable/2023-05-15/inference.json. 82 | ![img](/assets/apim_config_0.1.png) 83 | 84 | 3. Test the endpoint to validate the API Management instance resource can communicate with the Azure OpenAI Service resource. Provide the "deployment-id", "api-version" and a sample prompt as seen in the screenshot below. The deployment-id is the name of the model deployment you created in the Azure OpenAI Service resource. 85 | ![img](/assets/apim_config_4.png) 86 | 87 | ### **Event Log Deployment** 88 | 89 | 1. You must deploy an Event Hub Namespace resource and Event Hub. This [can be done in the Azure Portal](https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-create). 90 | 91 | 2. Record the information listed below. It will be required when creating the API Management Logger. 92 | 93 | * Event Hub Namespace FQDN. Example mynamespace.servicebus.windows.net 94 | 95 | * Event Hub Name. This is the name of the Event Hub you created within the Event Hub Namespace. 96 | 97 | * Resource Id of the Event Hub Namespace. 98 | 99 | ### **User-assigned Managed Identity** 100 | 101 | 1. It is recommended to use a user-assigned managed identity to authenticate the API Management resource to the Event Hub. You can create the User-assigned Managed Identity in the [Azure Portal using these instructions](https://learn.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/how-manage-user-assigned-managed-identities?pivots=identity-mi-methods-azp). 102 | 103 | 2. Record the client id of the user-assigned managed identity. It will be required when creating the API Management Logger. 104 | 105 | ### **Assign permissions to the Event Hub to the User-Assigned Managed Identity** 106 | 107 | 1. Assign the user-assigned managed identity you created in the earlier step to the [Azure Event Hubs Data Sender Azure RBAC role](https://learn.microsoft.com/en-us/azure/api-management/api-management-howto-log-event-hubs?tabs=PowerShell#option-2-configure-api-management-managed-identity). 108 | 109 | 2. You can assign the role using the [Azure Portal using these instructions](https://learn.microsoft.com/en-us/azure/role-based-access-control/role-assignments-portal). 110 | 111 | 3. Wait at least 15 minutes for the role to propagate throughout Azure. If you do not wait at least 15 minutes, you may encounter an error when creating the API Management Logger. 112 | 113 | ### **Add the User-Assigned Managed Identity to the Azure API Management Resource** 114 | 115 | 1. You must add the user-assigned managed identity you created to the API Management resource. You can do this through the [Azure Portal using these instructions](https://learn.microsoft.com/en-us/azure/api-management/api-management-howto-use-managed-service-identity#create-a-user-assigned-managed-identity). 116 | 117 | ### **Create the API Management Logger** 118 | 119 | 1. The API Management Logger can only be created through CLI or an ARM template. A [sample ARM template](../assets/apim-logger.json) is provided in this repository. You can [deploy the ARM template using the Azure Portal using these instructions](https://learn.microsoft.com/en-us/azure/azure-resource-manager/templates/deploy-portal). You must provide the relevant information collected in the previous step. Take care to provide the exact information detailed in the ARM template. If you do not provide the exact information you will encounter non-descriptive errors. 120 | 121 | 2. Record the name that you assign to the logger. This will be required in the next step. 122 | 123 | ### **Create the custom API Management Policy** 124 | 125 | 1. In Designsection of the API in the API Management resource select the link in the Inbound policies section as seen in the screenshot below. 126 | ![img](../assets/apim-config-adv-0.png) 127 | 128 | 2. Copy and paste the custom Azure API Management Policy [provided in this repository](../assets/apim-policy-event-hub-logging.xml). You must modify the variables in the comment section of the policy with the values that match your implementation. The policy will create two events, one for the request and one for the response. The events are correlated with the message-id property which a unique GUID generated for each message to the API. 129 | 130 | 3. When complete click Save to commit the policy. If you receive any errors, likely you missed a variable or added a character. 131 | 132 | ### **Test the configuration** 133 | 134 | Test the configuration to ensure it is working as intended. Recall that the API Management policy restricts to Azure Active Directory identities so you must pass an valid access token to API Management instance. The context of the identity (such as a service principal) included in the access token must have appropriate Azure RBAC permissions on the Azure OpenAI Service resource. 135 | 136 | Sample code in Python using a service principal can be found at this link https://github.com/mattfeltonma/demo-openai-python. You should provide the API Management FQDN for the API as the OPENAI_BASE variable. 137 | 138 | If you receive errors double-check that the service principal has the appropriate permissions on the Azure OpenAI Service resource. 139 | 140 | You can also test that messages are being received from the Azure Event Hub using the [Azure Event Hub Explorer Visual Studio Code Add-In](https://marketplace.visualstudio.com/items?itemName=Summer.azure-event-hub-explorer). 141 | 142 | ![img](/assets/event-hub-explorer.png) 143 | 144 | ### **Analyzing Event Hub Messages** 145 | 146 | After you have verified requests and responses are being captured by the Azure Event Hub, you can capture those events in a number of ways. The integration with Azure Stream Analytics provides a number of simple ways to extract, transform, and load events from an Azure Event Hub to a data store for further analytics. 147 | 148 | [Review the documentation](https://learn.microsoft.com/en-us/azure/event-hubs/process-data-azure-stream-analytics) and select the option that works best for your organization. 149 | 150 | -------------------------------------------------------------------------------- /assets/EnterpriseAOAI-Architecture-Adv-Flow.svg: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /assets/EnterpriseAOAI-Architecture.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure-Samples/openai-python-enterprise-logging/562e4cf7d3b14d5f4a5d04dab3d5787f5b883f2c/assets/EnterpriseAOAI-Architecture.png -------------------------------------------------------------------------------- /assets/EnterpriseLogging_0.mp4: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure-Samples/openai-python-enterprise-logging/562e4cf7d3b14d5f4a5d04dab3d5787f5b883f2c/assets/EnterpriseLogging_0.mp4 -------------------------------------------------------------------------------- /assets/apim-config-adv-0.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure-Samples/openai-python-enterprise-logging/562e4cf7d3b14d5f4a5d04dab3d5787f5b883f2c/assets/apim-config-adv-0.png -------------------------------------------------------------------------------- /assets/apim-config-apikey.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure-Samples/openai-python-enterprise-logging/562e4cf7d3b14d5f4a5d04dab3d5787f5b883f2c/assets/apim-config-apikey.png -------------------------------------------------------------------------------- /assets/apim-logger.json: -------------------------------------------------------------------------------- 1 | { 2 | "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#", 3 | "contentVersion": "1.0.0.0", 4 | "parameters": { 5 | "apiInstanceName": { 6 | "type": "string", 7 | "metadata": { 8 | "description": "The name of the API Management instance" 9 | } 10 | }, 11 | "apimLoggerName": { 12 | "type": "string", 13 | "metadata": { 14 | "description": "The name to be assigned to the API Managmeent logger object" 15 | } 16 | }, 17 | "eventHubName": { 18 | "type": "string", 19 | "metadata": { 20 | "description": "The name of the specific Event Hub in the Event Hub Namespace that messages will be delivered to" 21 | } 22 | }, 23 | "eventHubNamespaceName": { 24 | "type": "string", 25 | "metadata": { 26 | "description": "The FQDN of the Event Hub namespace. Example: mynamespace.servicebus.windows.net" 27 | } 28 | }, 29 | "eventHubResourceId": { 30 | "type": "string", 31 | "metadata": { 32 | "description": "The resource id of the Event Hub Namespace" 33 | } 34 | }, 35 | "umiClientId": { 36 | "type": "string", 37 | "metadata": { 38 | "description": "The client id of the user-assigned managed identity with permissions over the Event Hub" 39 | } 40 | } 41 | }, 42 | "variables": {}, 43 | "resources": [ 44 | { 45 | "type": "Microsoft.ApiManagement/service/loggers", 46 | "apiVersion": "2022-04-01-preview", 47 | "name": "[concat(parameters('apiInstanceName'), '/', parameters('apimLoggerName'))]", 48 | "properties": { 49 | "loggerType": "azureEventHub", 50 | "description": "Event hub logger with user-assigned managed identity", 51 | "resourceId": "[parameters('eventHubResourceId')]", 52 | "credentials": { 53 | "endpointAddress": "[parameters('eventHubNamespaceName')]", 54 | "identityClientId": "[parameters('umiClientId')]", 55 | "name": "[parameters('eventHubName')]" 56 | } 57 | } 58 | } 59 | ] 60 | } -------------------------------------------------------------------------------- /assets/apim-policy-event-hub-logging.xml: -------------------------------------------------------------------------------- 1 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | https://sts.windows.net/{{TENANT_ID}}/ 23 | 24 | 25 | 26 | https://cognitiveservices.azure.com 27 | 28 | 29 | 30 | 31 | @{ 32 | 33 | var requestBody = context.Request.Body?.As(true); 34 | 35 | string prompt = string.Empty; 36 | string messages = string.Empty; 37 | string model = string.Empty; 38 | if(requestBody != null) 39 | { 40 | prompt = requestBody["prompt"]?.ToString(); 41 | messages = requestBody["messages"]?.ToString(); 42 | model = requestBody["model"]?.ToString(); 43 | } 44 | 45 | string operation = context.Operation.Id; 46 | string result = string.Empty; 47 | 48 | switch(operation) 49 | { 50 | case "get-a-generated-image-result": 51 | result = new JObject( 52 | new JProperty("event-type", "Request"), 53 | new JProperty("event-time", DateTime.UtcNow.ToString()), 54 | new JProperty("backend", context.Request.Url.Host.ToString()), 55 | new JProperty("message-id", context.Variables["message-id"]), 56 | new JProperty("operation", operation) 57 | ).ToString(); 58 | break; 59 | default: 60 | result = new JObject( 61 | new JProperty("event-type", "Request"), 62 | new JProperty("event-time", DateTime.UtcNow.ToString()), 63 | new JProperty("backend", context.Request.Url.Host.ToString()), 64 | new JProperty("message-id", context.Variables["message-id"]), 65 | new JProperty("operation", operation), 66 | new JProperty("model", model), 67 | new JProperty("prompt", prompt), 68 | new JProperty("messages", messages) 69 | ).ToString(); 70 | break; 71 | } 72 | return result; 73 | } 74 | 75 | 76 | 77 | 78 | 79 | 80 | @{ 81 | 82 | var responseBody = context.Response.Body?.As(true); 83 | var operation = context.Operation.Id; 84 | string response = responseBody["choices"]?.ToString(); 85 | 86 | string result = string.Empty; 87 | 88 | switch(operation) 89 | { 90 | case "ChatCompletions_Create": 91 | case "Completions_Create": 92 | result = new JObject( 93 | new JProperty("event-type", "Response"), 94 | new JProperty("event-time", DateTime.UtcNow.ToString()), 95 | new JProperty("backend", context.Request.Url.Host.ToString()), 96 | new JProperty("message-id", context.Variables["message-id"]), 97 | new JProperty("choices",response), 98 | new JProperty("operation", operation), 99 | new JProperty("apiId", context.Api.Id), 100 | new JProperty("productId", context.Product.Id), 101 | new JProperty("model", responseBody["model"].ToString()), 102 | new JProperty("modeltime", context.Response.Headers.GetValueOrDefault("Openai-Processing-Ms",string.Empty)), 103 | new JProperty("completion_tokens", responseBody["usage"]["completion_tokens"].ToString()), 104 | new JProperty("prompt_tokens", responseBody["usage"]["prompt_tokens"].ToString()), 105 | new JProperty("total_tokens", responseBody["usage"]["total_tokens"].ToString()) 106 | ).ToString(); 107 | break; 108 | 109 | case "embeddings_create": 110 | result = new JObject( 111 | new JProperty("event-type", "Response"), 112 | new JProperty("event-time", DateTime.UtcNow.ToString()), 113 | new JProperty("backend", context.Request.Url.Host.ToString()), 114 | new JProperty("message-id", context.Variables["message-id"]), 115 | new JProperty("operation", operation), 116 | new JProperty("apiId", context.Api.Id), 117 | new JProperty("productId", context.Product.Id), 118 | new JProperty("model", responseBody["model"].ToString()), 119 | new JProperty("modeltime", context.Response.Headers.GetValueOrDefault("Openai-Processing-Ms",string.Empty)), 120 | new JProperty("prompt_tokens", responseBody["usage"]["prompt_tokens"].ToString()), 121 | new JProperty("total_tokens", responseBody["usage"]["total_tokens"].ToString()) 122 | ).ToString(); 123 | break; 124 | 125 | default: 126 | result = new JObject( 127 | new JProperty("event-type", "Response"), 128 | new JProperty("event-time", DateTime.UtcNow.ToString()), 129 | new JProperty("backend", context.Request.Url.Host.ToString()), 130 | new JProperty("message-id", context.Variables["message-id"]), 131 | new JProperty("operation", operation), 132 | new JProperty("apiId", context.Api.Id), 133 | new JProperty("productId", context.Product.Id), 134 | new JProperty("modeltime", context.Response.Headers.GetValueOrDefault("Openai-Processing-Ms",string.Empty)) 135 | ).ToString(); 136 | break; 137 | } 138 | return result; 139 | 140 | } 141 | 142 | 143 | 144 | 145 | 146 | 147 | -------------------------------------------------------------------------------- /assets/apim_config_0.0.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure-Samples/openai-python-enterprise-logging/562e4cf7d3b14d5f4a5d04dab3d5787f5b883f2c/assets/apim_config_0.0.png -------------------------------------------------------------------------------- /assets/apim_config_0.1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure-Samples/openai-python-enterprise-logging/562e4cf7d3b14d5f4a5d04dab3d5787f5b883f2c/assets/apim_config_0.1.png -------------------------------------------------------------------------------- /assets/apim_config_0.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure-Samples/openai-python-enterprise-logging/562e4cf7d3b14d5f4a5d04dab3d5787f5b883f2c/assets/apim_config_0.png -------------------------------------------------------------------------------- /assets/apim_config_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure-Samples/openai-python-enterprise-logging/562e4cf7d3b14d5f4a5d04dab3d5787f5b883f2c/assets/apim_config_1.png -------------------------------------------------------------------------------- /assets/apim_config_2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure-Samples/openai-python-enterprise-logging/562e4cf7d3b14d5f4a5d04dab3d5787f5b883f2c/assets/apim_config_2.png -------------------------------------------------------------------------------- /assets/apim_config_3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure-Samples/openai-python-enterprise-logging/562e4cf7d3b14d5f4a5d04dab3d5787f5b883f2c/assets/apim_config_3.png -------------------------------------------------------------------------------- /assets/apim_config_4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure-Samples/openai-python-enterprise-logging/562e4cf7d3b14d5f4a5d04dab3d5787f5b883f2c/assets/apim_config_4.png -------------------------------------------------------------------------------- /assets/apim_regen_keys.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure-Samples/openai-python-enterprise-logging/562e4cf7d3b14d5f4a5d04dab3d5787f5b883f2c/assets/apim_regen_keys.png -------------------------------------------------------------------------------- /assets/event-hub-explorer.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure-Samples/openai-python-enterprise-logging/562e4cf7d3b14d5f4a5d04dab3d5787f5b883f2c/assets/event-hub-explorer.png -------------------------------------------------------------------------------- /assets/monitor_0.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure-Samples/openai-python-enterprise-logging/562e4cf7d3b14d5f4a5d04dab3d5787f5b883f2c/assets/monitor_0.png -------------------------------------------------------------------------------- /assets/monitor_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure-Samples/openai-python-enterprise-logging/562e4cf7d3b14d5f4a5d04dab3d5787f5b883f2c/assets/monitor_1.png -------------------------------------------------------------------------------- /assets/video.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure-Samples/openai-python-enterprise-logging/562e4cf7d3b14d5f4a5d04dab3d5787f5b883f2c/assets/video.png -------------------------------------------------------------------------------- /deploy-terraform/.gitignore: -------------------------------------------------------------------------------- 1 | # Local .terraform directories 2 | **/.terraform/* 3 | 4 | # .tfstate files 5 | *.tfstate 6 | *.tfstate.* 7 | 8 | # Crash log files 9 | crash.log 10 | 11 | # Ignore any .tfvars files that are generated automatically for each Terraform run. Most 12 | # .tfvars files are managed as part of configuration and so should be included in 13 | # version control. 14 | # 15 | # example.tfvars 16 | 17 | # Ignore override files as they are usually used to override resources locally and so 18 | # are not checked in 19 | override.tf 20 | override.tf.json 21 | *_override.tf 22 | *_override.tf.json 23 | 24 | # Include override files you do wish to add to version control using negated pattern 25 | # 26 | # !example_override.tf 27 | 28 | # Include tfplan files to ignore the plan output of command: terraform plan -out=tfplan 29 | # example: *tfplan* 30 | **/.terraform.lock.hcl -------------------------------------------------------------------------------- /deploy-terraform/README.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure-Samples/openai-python-enterprise-logging/562e4cf7d3b14d5f4a5d04dab3d5787f5b883f2c/deploy-terraform/README.md -------------------------------------------------------------------------------- /deploy-terraform/apim.tf: -------------------------------------------------------------------------------- 1 | resource "azurerm_api_management" "apim" { 2 | name = "apim-${local.name}" 3 | location = azurerm_resource_group.rg.location 4 | resource_group_name = azurerm_resource_group.rg.name 5 | publisher_name = "openai-python-enterprise-logging" 6 | publisher_email = "nothing@example.com" 7 | virtual_network_type = "External" 8 | virtual_network_configuration { 9 | subnet_id = azurerm_subnet.api.id 10 | } 11 | policy = [ 12 | { 13 | xml_content = <<-EOT 14 | 15 | 16 | 17 | 18 | https://apim-${local.name}.developer.azure-api.net 19 | 20 | 21 | * 22 | 23 | 24 |
*
25 | 26 | 27 |
*
28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | @( context.Operation.Name ) 37 | 38 | 39 | @( context.Operation.Method ) 40 | 41 | 42 | @( context.Operation.UrlTemplate ) 43 | 44 | 45 | @( context.Api.Name ) 46 | 47 | 48 | @( context.Api.Path ) 49 | 50 | 51 | 52 | 53 | @( context.Operation.Name ) 54 | 55 | 56 | @( context.Operation.Method ) 57 | 58 | 59 | @( context.Operation.UrlTemplate ) 60 | 61 | 62 | @( context.Api.Name ) 63 | 64 | 65 | @( context.Api.Path ) 66 | 67 | 68 | @( context.LastError.Message ) 69 | 70 | 71 | 72 | EOT 73 | xml_link = null 74 | }, 75 | ] 76 | zones = [] 77 | sku_name = "Developer_1" 78 | 79 | identity { 80 | type = "SystemAssigned" 81 | } 82 | 83 | tags = local.tags 84 | } 85 | 86 | resource "azurerm_api_management_api" "this" { 87 | name = "openai-api" 88 | resource_group_name = azurerm_resource_group.rg.name 89 | api_management_name = azurerm_api_management.apim.name 90 | revision = "1" 91 | display_name = "OpenAI API" 92 | path = "openai" 93 | protocols = ["https"] 94 | 95 | import { 96 | content_format = "openapi+json-link" 97 | content_value = "https://raw.githubusercontent.com/Azure/azure-rest-api-specs/main/specification/cognitiveservices/data-plane/AzureOpenAI/inference/stable/2023-05-15/inference.json" 98 | } 99 | 100 | subscription_key_parameter_names { 101 | header = "api-key" 102 | query = "api-key" 103 | } 104 | } 105 | 106 | resource "azurerm_api_management_backend" "this" { 107 | name = "${local.name}-backend" 108 | resource_group_name = azurerm_resource_group.rg.name 109 | api_management_name = azurerm_api_management.apim.name 110 | protocol = "http" 111 | url = "${azurerm_cognitive_account.this.endpoint}/openai" 112 | 113 | credentials { 114 | header = { 115 | api-key = "{{openaikey}}" 116 | } 117 | } 118 | } 119 | 120 | resource "azurerm_api_management_api_policy" "this" { 121 | api_name = azurerm_api_management_api.this.name 122 | api_management_name = azurerm_api_management_api.this.api_management_name 123 | resource_group_name = azurerm_api_management_api.this.resource_group_name 124 | 125 | xml_content = < 127 | 128 | 129 | @{ 130 | var body = context.Request.Body?.As(true); 131 | if (body != null && body.Length > 1024) 132 | { 133 | body = body.Substring(0, 1024); 134 | } 135 | 136 | var headers = context.Request.Headers 137 | .Where(h => h.Key != "Authorization" && h.Key != "Ocp-Apim-Subscription-Key" && h.Key != "api-key") 138 | .Select(h => string.Format("{0}: {1}", h.Key, String.Join(", ", h.Value))) 139 | .ToArray(); 140 | var requestIdHeader = context.Request.Headers.GetValueOrDefault("Request-Id", ""); 141 | return new JObject( 142 | new JProperty("Type", "request"), 143 | new JProperty("Headers", headers), 144 | new JProperty("EventTime", DateTime.UtcNow.ToString()), 145 | new JProperty("ServiceName", context.Deployment.ServiceName), 146 | new JProperty("requestIdHeader", requestIdHeader), 147 | new JProperty("RequestId", context.RequestId), 148 | new JProperty("RequestIp", context.Request.IpAddress), 149 | new JProperty("RequestMethod", context.Request.Method), 150 | new JProperty("RequestPath", context.Request.Url.Path), 151 | new JProperty("RequestQuery", context.Request.Url.QueryString), 152 | new JProperty("RequestBody", body), 153 | new JProperty("OperationName", context.Operation.Name), 154 | new JProperty("OperationMethod", context.Operation.Method), 155 | new JProperty("OperationUrl", context.Operation.UrlTemplate), 156 | new JProperty("ApiName", context.Api.Name), 157 | new JProperty("ApiPath", context.Api.Path) 158 | 159 | ).ToString(); 160 | 161 | } 162 | 163 | 164 | 165 | 166 | 167 | 168 | 169 | @{ 170 | 171 | var body = ""; 172 | var headers = context.Response.Headers 173 | .Select(h => string.Format("{0}: {1}", h.Key, String.Join(", ", h.Value))) 174 | .ToArray(); 175 | 176 | var respContentType = context.Response.Headers.GetValueOrDefault("Content-Type", ""); 177 | if( respContentType.Equals("text/event-stream") ){ 178 | body = "streaming"; 179 | }else{ 180 | body = "not-streaming"; 181 | 182 | } 183 | var requestIdHeader = context.Request.Headers.GetValueOrDefault("Request-Id", ""); 184 | var requestBody = context.Request.Body?.As(true); 185 | return new JObject( 186 | new JProperty("Type", "response"), 187 | new JProperty("Headers", headers), 188 | new JProperty("EventTime", DateTime.UtcNow.ToString()), 189 | new JProperty("ServiceName", context.Deployment.ServiceName), 190 | new JProperty("requestIdHeader", requestIdHeader), 191 | new JProperty("RequestId", context.RequestId), 192 | new JProperty("RequestIp", context.Request.IpAddress), 193 | new JProperty("RequestMethod", context.Request.Method), 194 | new JProperty("ResponseStatusCode", context.Response.StatusCode), 195 | new JProperty("ResponseStatusReason", context.Response.StatusReason), 196 | new JProperty("ResponseBody", body), 197 | new JProperty("OperationName", context.Operation.Name), 198 | new JProperty("OperationMethod", context.Operation.Method), 199 | new JProperty("OperationUrl", context.Operation.UrlTemplate), 200 | new JProperty("ApiName", context.Api.Name), 201 | new JProperty("ApiPath", context.Api.Path), 202 | new JProperty("RequestBody", requestBody), 203 | new JProperty("Duration", context.Elapsed) 204 | 205 | ).ToString(); 206 | 207 | } 208 | 209 | 210 | 211 | @{ 212 | var requestIdHeader = context.Request.Headers.GetValueOrDefault("Request-Id", ""); 213 | return new JObject( 214 | new JProperty("Type", "error"), 215 | new JProperty("EventTime", DateTime.UtcNow.ToString()), 216 | new JProperty("ServiceName", context.Deployment.ServiceName), 217 | new JProperty("requestIdHeader", requestIdHeader), 218 | new JProperty("RequestId", context.RequestId), 219 | new JProperty("RequestIp", context.Request.IpAddress), 220 | new JProperty("LastErrorMessage", context.LastError.Message), 221 | new JProperty("OperationName", context.Operation.Name), 222 | new JProperty("OperationMethod", context.Operation.Method), 223 | new JProperty("OperationUrl", context.Operation.UrlTemplate), 224 | new JProperty("ApiName", context.Api.Name), 225 | new JProperty("ApiPath", context.Api.Path), 226 | new JProperty("Duration", context.Elapsed) 227 | ).ToString(); 228 | } 229 | 230 | 231 | XML 232 | } 233 | 234 | resource "azurerm_api_management_logger" "this" { 235 | name = "ehlogger" 236 | api_management_name = azurerm_api_management.apim.name 237 | resource_group_name = azurerm_resource_group.rg.name 238 | 239 | 240 | eventhub { 241 | name = azurerm_eventhub.this.name 242 | connection_string = azurerm_eventhub_namespace.this.default_primary_connection_string 243 | } 244 | } 245 | 246 | resource "azurerm_api_management_named_value" "openaikey" { 247 | name = "openaikey" 248 | resource_group_name = azurerm_resource_group.rg.name 249 | api_management_name = azurerm_api_management.apim.name 250 | display_name = "openaikey" 251 | secret = true 252 | value_from_key_vault { 253 | secret_id = azurerm_key_vault_secret.openaikey.id 254 | } 255 | } 256 | -------------------------------------------------------------------------------- /deploy-terraform/cognitive.tf: -------------------------------------------------------------------------------- 1 | resource "azurerm_cognitive_account" "this" { 2 | name = local.name 3 | location = azurerm_resource_group.rg.location 4 | resource_group_name = azurerm_resource_group.rg.name 5 | kind = "OpenAI" 6 | sku_name = "S0" 7 | custom_subdomain_name = local.name 8 | tags = local.tags 9 | } 10 | 11 | resource "azurerm_cognitive_deployment" "this" { 12 | name = "model-${local.name}" 13 | cognitive_account_id = azurerm_cognitive_account.this.id 14 | model { 15 | format = "OpenAI" 16 | name = var.model 17 | version = var.model_version 18 | } 19 | scale { 20 | type = "Standard" 21 | } 22 | } -------------------------------------------------------------------------------- /deploy-terraform/eventhub.tf: -------------------------------------------------------------------------------- 1 | resource "azurerm_eventhub_namespace" "this" { 2 | name = "ehn${local.name}" 3 | location = azurerm_resource_group.rg.location 4 | resource_group_name = azurerm_resource_group.rg.name 5 | sku = "Basic" 6 | capacity = 1 7 | 8 | tags = local.tags 9 | } 10 | 11 | resource "azurerm_eventhub" "this" { 12 | name = "apimlogger" 13 | namespace_name = azurerm_eventhub_namespace.this.name 14 | resource_group_name = azurerm_resource_group.rg.name 15 | partition_count = 2 16 | message_retention = 1 17 | } -------------------------------------------------------------------------------- /deploy-terraform/keyvault.tf: -------------------------------------------------------------------------------- 1 | resource "azurerm_key_vault" "kv" { 2 | name = "kv-${local.name}" 3 | location = azurerm_resource_group.rg.location 4 | resource_group_name = azurerm_resource_group.rg.name 5 | tenant_id = data.azurerm_client_config.current.tenant_id 6 | sku_name = "standard" 7 | soft_delete_retention_days = 7 8 | purge_protection_enabled = false 9 | 10 | } 11 | 12 | resource "azurerm_key_vault_access_policy" "sp" { 13 | key_vault_id = azurerm_key_vault.kv.id 14 | tenant_id = data.azurerm_client_config.current.tenant_id 15 | object_id = data.azurerm_client_config.current.object_id 16 | 17 | key_permissions = [ 18 | "Create", 19 | "Get", 20 | "Purge", 21 | "Recover", 22 | "Delete" 23 | ] 24 | 25 | secret_permissions = [ 26 | "Set", 27 | "Purge", 28 | "Get", 29 | "List", 30 | "Delete" 31 | ] 32 | 33 | certificate_permissions = [ 34 | "Purge" 35 | ] 36 | 37 | storage_permissions = [ 38 | "Purge" 39 | ] 40 | 41 | } 42 | 43 | resource "azurerm_key_vault_access_policy" "apim" { 44 | key_vault_id = azurerm_key_vault.kv.id 45 | tenant_id = data.azurerm_client_config.current.tenant_id 46 | object_id = azurerm_api_management.apim.identity[0].principal_id 47 | 48 | key_permissions = [ 49 | ] 50 | 51 | secret_permissions = [ 52 | "Get", 53 | "List", 54 | ] 55 | 56 | certificate_permissions = [ 57 | ] 58 | 59 | storage_permissions = [ 60 | ] 61 | 62 | } 63 | 64 | resource "azurerm_key_vault_secret" "openaikey" { 65 | depends_on = [ azurerm_key_vault_access_policy.sp, azurerm_key_vault_access_policy.apim ] 66 | name = "openaikey" 67 | value = azurerm_cognitive_account.this.primary_access_key 68 | key_vault_id = azurerm_key_vault.kv.id 69 | } -------------------------------------------------------------------------------- /deploy-terraform/main.tf: -------------------------------------------------------------------------------- 1 | terraform { 2 | required_providers { 3 | azurerm = { 4 | source = "hashicorp/azurerm" 5 | version = "=3.66.0" 6 | } 7 | random = { 8 | source = "hashicorp/random" 9 | version = "=3.1.0" 10 | } 11 | azapi = { 12 | source = "Azure/azapi" 13 | version = "1.7.0" 14 | } 15 | } 16 | } 17 | 18 | provider "azurerm" { 19 | features { 20 | resource_group { 21 | prevent_deletion_if_contains_resources = false 22 | } 23 | } 24 | } 25 | 26 | locals { 27 | name = "openai${random_string.unique.result}" 28 | location = var.location 29 | 30 | tags = { 31 | "managed_by" = "terraform" 32 | "repo" = "openai-python-enterprise-logging" 33 | } 34 | } 35 | 36 | data "azurerm_client_config" "current" {} 37 | 38 | resource "random_string" "unique" { 39 | length = 8 40 | special = false 41 | upper = false 42 | } 43 | 44 | resource "azurerm_resource_group" "rg" { 45 | name = "rg-${local.name}-${local.location}" 46 | location = local.location 47 | tags = local.tags 48 | } 49 | 50 | -------------------------------------------------------------------------------- /deploy-terraform/network.tf: -------------------------------------------------------------------------------- 1 | resource "azurerm_virtual_network" "gateway" { 2 | name = "vnet-gateway" 3 | location = azurerm_resource_group.rg.location 4 | resource_group_name = azurerm_resource_group.rg.name 5 | address_space = ["10.0.0.0/16"] 6 | 7 | tags = local.tags 8 | } 9 | 10 | resource "azurerm_subnet" "gateway" { 11 | name = "snet-gateway" 12 | resource_group_name = azurerm_resource_group.rg.name 13 | virtual_network_name = azurerm_virtual_network.gateway.name 14 | address_prefixes = ["10.0.1.0/24"] 15 | } 16 | 17 | resource "azurerm_network_security_group" "gateway" { 18 | name = "nsg-gateway" 19 | location = azurerm_resource_group.rg.location 20 | resource_group_name = azurerm_resource_group.rg.name 21 | 22 | security_rule { 23 | name = "AllowGatewayManager" 24 | priority = 2702 25 | direction = "Inbound" 26 | access = "Allow" 27 | protocol = "*" 28 | source_address_prefix = "GatewayManager" 29 | source_port_range = "*" 30 | destination_address_prefix = "*" 31 | destination_port_range = "65200-65535" 32 | } 33 | 34 | tags = local.tags 35 | } 36 | 37 | resource "azurerm_subnet_network_security_group_association" "gateway" { 38 | subnet_id = azurerm_subnet.gateway.id 39 | network_security_group_id = azurerm_network_security_group.gateway.id 40 | } 41 | 42 | resource "azurerm_virtual_network" "app" { 43 | name = "vnet-app" 44 | location = azurerm_resource_group.rg.location 45 | resource_group_name = azurerm_resource_group.rg.name 46 | address_space = ["10.1.0.0/16"] 47 | 48 | tags = local.tags 49 | } 50 | 51 | resource "azurerm_subnet" "api" { 52 | name = "snet-api" 53 | resource_group_name = azurerm_resource_group.rg.name 54 | virtual_network_name = azurerm_virtual_network.app.name 55 | address_prefixes = ["10.1.1.0/24"] 56 | } 57 | 58 | resource "azurerm_subnet" "endpoints" { 59 | name = "snet-endpoints" 60 | resource_group_name = azurerm_resource_group.rg.name 61 | virtual_network_name = azurerm_virtual_network.app.name 62 | address_prefixes = ["10.1.2.0/24"] 63 | } 64 | 65 | resource "azurerm_network_security_group" "app" { 66 | name = "nsg-api" 67 | location = azurerm_resource_group.rg.location 68 | resource_group_name = azurerm_resource_group.rg.name 69 | 70 | security_rule { 71 | name = "Allow-3443-Inbound" 72 | priority = 1010 73 | direction = "Inbound" 74 | access = "Allow" 75 | protocol = "Tcp" 76 | source_address_prefix = "*" 77 | source_port_range = "*" 78 | destination_address_prefix = "*" 79 | destination_port_range = "3443" 80 | } 81 | 82 | security_rule { 83 | name = "Allow-443-Inbound" 84 | priority = 1020 85 | direction = "Inbound" 86 | access = "Allow" 87 | protocol = "Tcp" 88 | source_address_prefix = "*" 89 | source_port_range = "*" 90 | destination_address_prefix = "*" 91 | destination_port_range = "443" 92 | } 93 | 94 | security_rule { 95 | name = "Allow-3443-Outbound" 96 | priority = 1030 97 | direction = "Outbound" 98 | access = "Allow" 99 | protocol = "Tcp" 100 | source_address_prefix = "*" 101 | source_port_range = "*" 102 | destination_address_prefix = "*" 103 | destination_port_range = "3443" 104 | } 105 | 106 | tags = local.tags 107 | } 108 | 109 | resource "azurerm_subnet_network_security_group_association" "api" { 110 | subnet_id = azurerm_subnet.api.id 111 | network_security_group_id = azurerm_network_security_group.app.id 112 | } 113 | 114 | resource "azurerm_subnet_network_security_group_association" "endpoints" { 115 | subnet_id = azurerm_subnet.endpoints.id 116 | network_security_group_id = azurerm_network_security_group.app.id 117 | } 118 | 119 | resource "azurerm_virtual_network_peering" "gw-to-app" { 120 | name = "gw-to-app" 121 | resource_group_name = azurerm_resource_group.rg.name 122 | virtual_network_name = azurerm_virtual_network.gateway.name 123 | remote_virtual_network_id = azurerm_virtual_network.app.id 124 | } 125 | 126 | resource "azurerm_virtual_network_peering" "app-to-gw" { 127 | name = "app-to-gw" 128 | resource_group_name = azurerm_resource_group.rg.name 129 | virtual_network_name = azurerm_virtual_network.app.name 130 | remote_virtual_network_id = azurerm_virtual_network.gateway.id 131 | } 132 | 133 | resource "azurerm_public_ip" "gateway" { 134 | name = "pip-gateway-openai" 135 | resource_group_name = azurerm_resource_group.rg.name 136 | location = azurerm_resource_group.rg.location 137 | allocation_method = "Static" 138 | 139 | tags = local.tags 140 | } 141 | 142 | resource "azurerm_public_ip" "api" { 143 | name = "pip-apim-openai" 144 | resource_group_name = azurerm_resource_group.rg.name 145 | location = azurerm_resource_group.rg.location 146 | allocation_method = "Static" 147 | domain_name_label = "apim${local.name}" 148 | tags = local.tags 149 | } 150 | 151 | -------------------------------------------------------------------------------- /deploy-terraform/variables.tf: -------------------------------------------------------------------------------- 1 | variable "location" { 2 | type = string 3 | default = "eastus" 4 | } 5 | 6 | variable "model" { 7 | type = string 8 | default = "gpt-35-turbo" 9 | } 10 | 11 | variable "model_version" { 12 | type = string 13 | default = "0301" 14 | } -------------------------------------------------------------------------------- /deploy/README.md: -------------------------------------------------------------------------------- 1 | # Deployment steps using infrastructure-as-code 2 | While the reference architecture can be deployed using the Azure Portal, leveraging infrastructure-as-code can make the process quicker and more repeatable. The following outlines how to leverage the Bicep code in this repository to deploy the reference architecture. 3 | 4 | ## Prerequisites 5 | - [Azure Subscription](https://azure.microsoft.com/en-us/get-started/) 6 | - [Azure OpenAI Application](https://aka.ms/oai/access) 7 | 8 | ## Provision the Azure resources 9 | ### Provision the Azure resources that are defined in the Bicep code 10 | 1. Sign in to the Azure Portal, and open the Cloud Shell. The following steps will assume the use of Bash 11 | 2. `git clone` this repository 12 | 3. `cd` to the `deploy` directory 13 | 4. Create a variable called `RG` and assign a resource group name you prefer (e.g., `RG=rg-openaiapp`) 14 | 5. Create a vailable called `LOC` and assign the name of the Azure region you prefer (e.g., `LOC=eastus`) 15 | 6. Create the resource group: `az group create -l $LOC -n $RG` 16 | 7. Preview the changes that will be made with the Bicep code: `az deployment group what-if --resource-group $RG --template-file main.bicep --parameters @main.parameters.json` (If asked to specify parameters `suffix` and `customSubDomainName` then set a unique value of your choice) 17 | 8. Apply the Bicep code: `az deployment group create --resource-group $RG --template-file main.bicep --parameters @main.parameters.json` (If asked to specify parameters `suffix` and `customSubDomainName` then set a unique value of your choice) 18 | 19 | ### Provision the remaining Azure resources 20 | The current version of the Bicep does not deploy the following Azure resources, do they need to be deployed using other means such as through the Azure Portal GUI. 21 | - Azure Log Analytics 22 | - Azure Key Vault 23 | - Azure Storage 24 | 25 | ## Configuration 26 | The current version of the Bicep does not include all the configuration that is needed. Therefore follow the steps outlined [here] 27 | (https://github.com/Azure-Samples/openai-python-enterprise-logging#configuration), and apply the configurations that are missings -------------------------------------------------------------------------------- /deploy/main.bicep: -------------------------------------------------------------------------------- 1 | // Define the parameter for location 2 | param location string 3 | param email string 4 | param publisherName string 5 | param suffix string 6 | param openAiLocation string 7 | param customSubDomainName string 8 | param openai_model_deployments array = [] 9 | 10 | // Network Security Group for the App Gateway subnet 11 | resource nsggateway 'Microsoft.Network/networkSecurityGroups@2020-11-01' = { 12 | name: 'nsg-gateway' 13 | location: location 14 | properties: { 15 | securityRules: [ 16 | { 17 | name: 'AllowGatewayManager' 18 | properties: { 19 | description: 'Allow GatewayManager' 20 | priority: 2702 21 | protocol: '*' 22 | access: 'Allow' 23 | direction: 'Inbound' 24 | sourceAddressPrefix: 'GatewayManager' 25 | sourcePortRange: '*' 26 | destinationAddressPrefix: '*' 27 | destinationPortRange: '65200-65535' 28 | } 29 | } 30 | ] 31 | } 32 | } 33 | 34 | // Virtual Network for the App Gateway subnet 35 | resource vnetgateway 'Microsoft.Network/virtualNetworks@2020-11-01' = { 36 | name: 'vnet-gateway' 37 | location: location 38 | properties: { 39 | addressSpace: { 40 | addressPrefixes: [ 41 | '10.0.0.0/16' 42 | ] 43 | } 44 | subnets: [ 45 | { 46 | name: 'snet-gateway' 47 | properties: { 48 | addressPrefix: '10.0.1.0/24' 49 | networkSecurityGroup: { id: nsggateway.id } 50 | } 51 | } 52 | ] 53 | } 54 | } 55 | 56 | // Subnet that for the Application Gateway 57 | resource snetgateway 'Microsoft.Network/virtualNetworks/subnets@2020-11-01' existing = { 58 | parent: vnetgateway 59 | name: 'snet-gateway' 60 | } 61 | 62 | // Network Security Group for the API Management subnet 63 | resource nsgapi 'Microsoft.Network/networkSecurityGroups@2020-11-01' = { 64 | name: 'nsg-api' 65 | location: location 66 | properties: { 67 | securityRules: [ 68 | { 69 | name: 'Allow-3443-Inbound' 70 | properties: { 71 | priority: 1010 72 | protocol: 'Tcp' 73 | access: 'Allow' 74 | direction: 'Inbound' 75 | sourceAddressPrefix: '*' 76 | sourcePortRange: '*' 77 | destinationAddressPrefix: '*' 78 | destinationPortRange: '3443' 79 | } 80 | } 81 | { 82 | name: 'Allow-443-Inbound' 83 | properties: { 84 | priority: 1020 85 | protocol: 'Tcp' 86 | access: 'Allow' 87 | direction: 'Inbound' 88 | sourceAddressPrefix: '*' 89 | sourcePortRange: '*' 90 | destinationAddressPrefix: '*' 91 | destinationPortRange: '443' 92 | } 93 | } 94 | { 95 | name: 'Allow-3443-Outbound' 96 | properties: { 97 | priority: 1030 98 | protocol: 'Tcp' 99 | access: 'Allow' 100 | direction: 'Outbound' 101 | sourceAddressPrefix: '*' 102 | sourcePortRange: '*' 103 | destinationAddressPrefix: '*' 104 | destinationPortRange: '3443' 105 | } 106 | } 107 | ] 108 | } 109 | } 110 | 111 | // Virtual Network for workload 112 | resource vnetapp 'Microsoft.Network/virtualNetworks@2020-11-01' = { 113 | name: 'vnet-app' 114 | location: location 115 | properties: { 116 | addressSpace: { 117 | addressPrefixes: [ 118 | '10.1.0.0/16' 119 | ] 120 | } 121 | subnets: [ 122 | { 123 | name: 'snet-api' 124 | properties: { 125 | addressPrefix: '10.1.1.0/24' 126 | networkSecurityGroup: { id: nsgapi.id } 127 | } 128 | } 129 | { 130 | name: 'snet-endpoints' 131 | properties: { 132 | addressPrefix: '10.1.2.0/24' 133 | networkSecurityGroup: { id: nsgapi.id } 134 | } 135 | } 136 | ] 137 | } 138 | } 139 | 140 | // Subnet for API Management 141 | resource snetapi 'Microsoft.Network/virtualNetworks/subnets@2020-11-01' existing = { 142 | parent: vnetapp 143 | name: 'snet-api' 144 | } 145 | 146 | // Subnet for Private Links 147 | resource snetendpoints 'Microsoft.Network/virtualNetworks/subnets@2020-11-01' existing = { 148 | parent: vnetapp 149 | name: 'snet-endpoints' 150 | } 151 | 152 | // Application Gateway Public IP 153 | resource publicIPAddress 'Microsoft.Network/publicIPAddresses@2021-08-01' = { 154 | name: 'pip-gateway-openai' 155 | location: location 156 | sku: { 157 | name: 'Standard' 158 | } 159 | properties: { 160 | publicIPAddressVersion: 'IPv4' 161 | publicIPAllocationMethod: 'Static' 162 | idleTimeoutInMinutes: 4 163 | } 164 | } 165 | 166 | // Craete Application Gateway 167 | resource appgateway 'Microsoft.Network/applicationGateways@2021-08-01' = { 168 | name: 'gateway-openai' 169 | location: location 170 | properties: { 171 | sku: { 172 | name: 'Standard_v2' 173 | tier: 'Standard_v2' 174 | capacity: 2 175 | } 176 | gatewayIPConfigurations: [ 177 | { 178 | name: 'appGatewayIpConfig' 179 | properties: { 180 | subnet: { 181 | id: snetgateway.id 182 | } 183 | } 184 | } 185 | ] 186 | frontendIPConfigurations: [ 187 | { 188 | name: 'appGatewayFrontendIP' 189 | properties: { 190 | privateIPAllocationMethod: 'Dynamic' 191 | publicIPAddress: { 192 | id: publicIPAddress.id 193 | } 194 | } 195 | } 196 | ] 197 | frontendPorts: [ 198 | { 199 | name: 'http' 200 | properties: { 201 | port: 80 202 | } 203 | } 204 | ] 205 | backendAddressPools: [ 206 | { 207 | name: 'appGatewayBackendPool' 208 | } 209 | ] 210 | backendHttpSettingsCollection: [ 211 | { 212 | name: 'appGatewayBackendHttpSettings' 213 | properties: { 214 | port: 80 215 | protocol: 'Http' 216 | cookieBasedAffinity: 'Disabled' 217 | pickHostNameFromBackendAddress: false 218 | requestTimeout: 30 219 | } 220 | } 221 | ] 222 | httpListeners: [ 223 | { 224 | name: 'appGatewayHttpListener' 225 | properties: { 226 | frontendIPConfiguration: { 227 | //id: appgateway.properties.frontendIPConfigurations[0].id 228 | id: resourceId('Microsoft.Network/applicationGateways/frontendIPConfigurations', 'gateway-openai', 'appGatewayFrontendIP') 229 | } 230 | frontendPort: { 231 | //id: appgateway.properties.frontendPorts[1].id 232 | id: resourceId('Microsoft.Network/applicationGateways/frontendPorts', 'gateway-openai', 'http') 233 | } 234 | } 235 | } 236 | ] 237 | requestRoutingRules: [ 238 | { 239 | name: 'rule1' 240 | properties: { 241 | ruleType: 'Basic' 242 | priority: 10 243 | httpListener: { 244 | id: resourceId('Microsoft.Network/applicationGateways/httpListeners', 'gateway-openai', 'appGatewayHttpListener') 245 | } 246 | backendAddressPool: { 247 | id: resourceId('Microsoft.Network/applicationGateways/backendAddressPools', 'gateway-openai', 'appGatewayBackendPool') 248 | } 249 | backendHttpSettings: { 250 | id: resourceId('Microsoft.Network/applicationGateways/backendHttpSettingsCollection', 'gateway-openai', 'appGatewayBackendHttpSettings') 251 | } 252 | } 253 | } 254 | ] 255 | } 256 | } 257 | 258 | // Create API Management 259 | resource apim 'Microsoft.ApiManagement/service@2020-06-01-preview' = { 260 | name: 'apim-openai-${suffix}' 261 | location: location 262 | sku: { 263 | name: 'Developer' 264 | capacity: 1 265 | } 266 | properties: { 267 | publisherEmail: email 268 | publisherName: publisherName 269 | virtualNetworkType: 'External' 270 | virtualNetworkConfiguration: { 271 | subnetResourceId: snetapi.id 272 | } 273 | } 274 | } 275 | 276 | // OpenAI Account + Model 277 | module openAi 'modules/cognitiveservices.bicep' = { 278 | name: 'my-openai-account' 279 | scope: resourceGroup() 280 | params: { 281 | name: 'openai' 282 | openaiLocation: openAiLocation 283 | sku: { 284 | name: 'S0' 285 | } 286 | customSubDomainName: customSubDomainName 287 | deployments: openai_model_deployments 288 | } 289 | } 290 | -------------------------------------------------------------------------------- /deploy/main.parameters.json: -------------------------------------------------------------------------------- 1 | { 2 | "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentParameters.json#", 3 | "contentVersion": "1.0.0.0", 4 | "parameters": { 5 | "location": { 6 | "value": "japaneast" 7 | }, 8 | "email": { 9 | "value": "name@domain.com" 10 | }, 11 | "publisherName": { 12 | "value": "name" 13 | }, 14 | "openAiLocation": { 15 | "value": "eastus" 16 | }, 17 | "openai_model_deployments": { 18 | "value": [ 19 | { 20 | "name": "myChatGptModelDeployment", 21 | "model": { 22 | "format": "OpenAI", 23 | "name": "gpt-35-turbo", 24 | "version": "0301" 25 | }, 26 | "scaleSettings": { 27 | "scaleType": "Standard" 28 | } 29 | } 30 | ] 31 | } 32 | } 33 | } 34 | -------------------------------------------------------------------------------- /deploy/modules/cognitiveservices.bicep: -------------------------------------------------------------------------------- 1 | param name string 2 | param openaiLocation string = 'eastus' 3 | 4 | param customSubDomainName string 5 | param deployments array = [] 6 | param kind string = 'OpenAI' 7 | param publicNetworkAccess string = 'Disabled' 8 | param sku object = { 9 | name: 'S0' 10 | } 11 | 12 | resource account 'Microsoft.CognitiveServices/accounts@2022-10-01' = { 13 | name: name 14 | location: openaiLocation 15 | kind: kind 16 | properties: { 17 | customSubDomainName: customSubDomainName 18 | publicNetworkAccess: publicNetworkAccess 19 | } 20 | sku: sku 21 | } 22 | 23 | @batchSize(1) 24 | resource deployment 'Microsoft.CognitiveServices/accounts/deployments@2022-10-01' = [for deployment in deployments: { 25 | parent: account 26 | name: deployment.name 27 | properties: { 28 | model: deployment.model 29 | raiPolicyName: contains(deployment, 'raiPolicyName') ? deployment.raiPolicyName : null 30 | scaleSettings: deployment.scaleSettings 31 | } 32 | }] 33 | 34 | output endpoint string = account.properties.endpoint 35 | output id string = account.id 36 | output name string = account.name 37 | --------------------------------------------------------------------------------