├── docs ├── media │ ├── pr_1.png │ ├── pr_2.png │ ├── fed_1.png │ ├── fed_2.png │ ├── template_repo.png │ └── template_repo_2.png └── Getting-Started.md ├── infra └── bicep │ ├── modules │ ├── rg.bicep │ ├── roleAssignment.bicep │ ├── subRoleAssignment.bicep │ ├── bingGrounding.bicep │ ├── logAnalytics.bicep │ ├── deploymentScript.bicep │ └── azureAIServices.bicep │ ├── agentsSetup.bicepparam │ ├── workbooks │ ├── MetroAgents.txt │ └── MetroAgents.json │ ├── agentInstructions │ ├── defaultProxyAgentInstructions.txt │ ├── AzurePolicyDefinitionDeveloperAgent.txt │ ├── MetroAIOrhcestratorAgent.txt │ ├── AzurePolicyTesterAgent.txt │ └── azurePolicyAgent.json │ ├── scriptContent │ ├── azurePolicyAgent.ps1 │ └── proxyToAgents.ps1 │ └── agentsSetup.bicep ├── SECURITY.md ├── .devcontainer ├── Dockerfile └── devcontainer.json ├── .github ├── scripts │ ├── get-changed-files.sh │ ├── deploy-policies.ps1 │ └── test-policies.ps1 └── workflows │ └── PolicyAgent.yml ├── LICENSE ├── utilities └── policyAgent │ ├── deployDef.ps1 │ ├── policyDef.bicep │ └── policyDef.parameters.json ├── policyDefinitions └── allowedLocations.json.sample ├── CONTRIBUTING.md ├── README.md └── .gitignore /docs/media/pr_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/AzurePolicyAgents/HEAD/docs/media/pr_1.png -------------------------------------------------------------------------------- /docs/media/pr_2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/AzurePolicyAgents/HEAD/docs/media/pr_2.png -------------------------------------------------------------------------------- /docs/media/fed_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/AzurePolicyAgents/HEAD/docs/media/fed_1.png -------------------------------------------------------------------------------- /docs/media/fed_2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/AzurePolicyAgents/HEAD/docs/media/fed_2.png -------------------------------------------------------------------------------- /docs/media/template_repo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/AzurePolicyAgents/HEAD/docs/media/template_repo.png -------------------------------------------------------------------------------- /docs/media/template_repo_2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/AzurePolicyAgents/HEAD/docs/media/template_repo_2.png -------------------------------------------------------------------------------- /infra/bicep/modules/rg.bicep: -------------------------------------------------------------------------------- 1 | targetScope = 'subscription' 2 | 3 | param rgName string = '' 4 | 5 | param location string = deployment().location 6 | 7 | resource rg 'Microsoft.Resources/resourceGroups@2021-04-01' = { 8 | name: rgName 9 | location: location 10 | } 11 | -------------------------------------------------------------------------------- /infra/bicep/modules/roleAssignment.bicep: -------------------------------------------------------------------------------- 1 | targetScope = 'resourceGroup' 2 | 3 | param objectId string = '' 4 | 5 | param roleDefinitionId string = '' 6 | 7 | param principalType string = '' 8 | 9 | resource roleAssignment 'Microsoft.Authorization/roleAssignments@2022-04-01' = { 10 | name: guid(subscription().subscriptionId, objectId, roleDefinitionId, resourceGroup().name) 11 | scope: resourceGroup() 12 | properties: { 13 | principalId: objectId 14 | roleDefinitionId: roleDefinitionId 15 | principalType: principalType 16 | } 17 | } 18 | -------------------------------------------------------------------------------- /infra/bicep/modules/subRoleAssignment.bicep: -------------------------------------------------------------------------------- 1 | targetScope = 'subscription' 2 | 3 | param objectId string = '' 4 | 5 | param roleDefinitionId string = '' 6 | 7 | param principalType string = '' 8 | 9 | resource roleAssignment 'Microsoft.Authorization/roleAssignments@2022-04-01' = { 10 | name: guid(subscription().subscriptionId, objectId, roleDefinitionId, subscription().displayName) 11 | scope: subscription() 12 | properties: { 13 | principalId: objectId 14 | roleDefinitionId: roleDefinitionId 15 | principalType: principalType 16 | } 17 | } 18 | -------------------------------------------------------------------------------- /SECURITY.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | ## Security 4 | 5 | Microsoft takes the security of our software products and services seriously, which 6 | includes all source code repositories in our GitHub organizations. 7 | 8 | **Please do not report security vulnerabilities through public GitHub issues.** 9 | 10 | For security reporting information, locations, contact information, and policies, 11 | please review the latest guidance for Microsoft repositories at 12 | [https://aka.ms/SECURITY.md](https://aka.ms/SECURITY.md). 13 | 14 | -------------------------------------------------------------------------------- /infra/bicep/modules/bingGrounding.bicep: -------------------------------------------------------------------------------- 1 | targetScope = 'resourceGroup' 2 | param resourceName string = '' 3 | 4 | resource bingGrounding 'Microsoft.Bing/accounts@2020-06-10' = { 5 | name: resourceName 6 | location: 'global' 7 | kind: 'Bing.Grounding' 8 | sku: { 9 | name: 'G1' 10 | } 11 | } 12 | 13 | // Retrieving the API Key for the Bing Grounding to be added as API key for connected services to the AI hubs 14 | #disable-next-line BCP037 15 | output bingKeys string = resourceName == '' ? '' : listKeys(bingGrounding.id, '2020-06-10').key1 16 | output bingResourceId string = resourceName == '' ? '' : bingGrounding.id 17 | -------------------------------------------------------------------------------- /infra/bicep/agentsSetup.bicepparam: -------------------------------------------------------------------------------- 1 | using './agentsSetup.bicep' 2 | 3 | // Basic resource parameters 4 | param rgName = 'AzKnPolicyAgent' // Name of the resource group for all resources 5 | param resourceName = 'AzKnPolicyFoundry' // Resource name prefix for all resources 6 | param location = 'swedencentral' // check for model availability and capacity in the model specific parameters below 7 | 8 | // AI agent parameters 9 | param agentModelCapacity = 150 // Model capacity 10 | param agentModelName = 'gpt-4.1' // Model name 11 | param agentModelVersion = '2025-04-14' // Model version 12 | param agentModelDeploymentName = 'gpt-4.1' // Model deployment 13 | param agentModelSkuName = 'GlobalStandard' // Model SKU name 14 | 15 | // Additional param for agent tooling 16 | param addKnowledge = 'groundingWithBing' // Add knowledge to the AI Agents - 'none', 'groundingWithBing', or 'aiSearch' 17 | -------------------------------------------------------------------------------- /.devcontainer/Dockerfile: -------------------------------------------------------------------------------- 1 | # Base image switched to azure-powershell 2 | FROM mcr.microsoft.com/azure-powershell:latest 3 | 4 | # Install essential packages and tools 5 | RUN apt-get update \ 6 | && apt-get install -y --no-install-recommends curl unzip ca-certificates zsh jq git sudo python3-pip \ 7 | && curl -sL https://aka.ms/InstallAzureCLIDeb | bash \ 8 | && curl -Lo /usr/local/bin/bicep https://github.com/Azure/bicep/releases/latest/download/bicep-linux-x64 \ 9 | && chmod +x /usr/local/bin/bicep \ 10 | && pip3 install pre-commit \ 11 | && rm -rf /var/lib/apt/lists/* 12 | 13 | # Non-root user setup 14 | ARG USERNAME=vscode 15 | ARG USER_UID=1000 16 | ARG USER_GID=$USER_UID 17 | 18 | RUN groupadd --gid $USER_GID $USERNAME \ 19 | && useradd -s /bin/pwsh --uid $USER_UID --gid $USER_GID -m $USERNAME \ 20 | && echo "$USERNAME ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers 21 | 22 | # Set user context 23 | USER $USERNAME -------------------------------------------------------------------------------- /infra/bicep/modules/logAnalytics.bicep: -------------------------------------------------------------------------------- 1 | targetScope = 'resourceGroup' 2 | param location string = resourceGroup().location 3 | param resourceName string = 'logAnalytics' 4 | param skuName string = 'PerGB2018' 5 | param retentionInDays int = 30 6 | 7 | resource logAnalyticsWorkspace 'Microsoft.OperationalInsights/workspaces@2023-09-01' = { 8 | name: resourceName 9 | location: location 10 | properties: { 11 | sku: { 12 | name: skuName 13 | } 14 | retentionInDays: retentionInDays 15 | features: { enableLogAccessUsingOnlyResourcePermissions: true } 16 | } 17 | } 18 | 19 | resource workbook 'Microsoft.Insights/workbooks@2022-04-01' = { 20 | name: guid(resourceName) 21 | location: location 22 | kind: 'Shared' 23 | properties: { 24 | displayName: 'MetroAgents' 25 | category: 'workbook' 26 | version: '1.0' 27 | sourceId: logAnalyticsWorkspace.id 28 | serializedData: loadTextContent('../workbooks/MetroAgents.txt') 29 | } 30 | } 31 | 32 | output logAnalyticsWorkspaceId string = logAnalyticsWorkspace.id 33 | -------------------------------------------------------------------------------- /infra/bicep/workbooks/MetroAgents.txt: -------------------------------------------------------------------------------- 1 | "{\"version\":\"Notebook/1.0\",\"items\":[{\"type\":1,\"content\":{\"json\":\"## MetroAgents\\n---\\n\\nThis workbook will give you line of sight to the [Metro Agent setup](https://github.com/Azure/MetroAgents)\",\"style\":\"success\"},\"name\":\"text\"},{\"type\":3,\"content\":{\"version\":\"KqlItem/1.0\",\"query\":\"AzureDiagnostics\\n| where ResourceProvider == \\\"MICROSOFT.COGNITIVESERVICES\\\"\\n| where Category == \\\"RequestResponse\\\"\\n| extend properties = parse_json(properties_s)\\n| extend apiVersion = tostring(properties.apiName)\\n| extend modelDeploymentName = tostring(properties.modelDeploymentName)\\n| extend modelName = tostring(properties.modelName)\\n| extend modelVersion = tostring(properties.modelVersion)\\n| extend objectId = tostring(properties.objectId)\\n| project TimeGenerated, OperationName, ResultSignature, Resource, apiVersion, objectId, modelDeploymentName, modelName, modelVersion\",\"size\":1,\"timeContext\":{\"durationMs\":86400000},\"queryType\":0,\"resourceType\":\"microsoft.operationalinsights/workspaces\"},\"name\":\"query\"}],\"isLocked\":false,\"fallbackResourceIds\":[]}" -------------------------------------------------------------------------------- /.github/scripts/get-changed-files.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Get changed files using git diff 4 | if [ "$GITHUB_EVENT_NAME" = "pull_request" ]; then 5 | # For pull requests, compare against the base branch 6 | BASE_SHA="$PR_BASE_SHA" 7 | HEAD_SHA="$PR_HEAD_SHA" 8 | CHANGED_FILES=$(git diff --name-only $BASE_SHA..$HEAD_SHA) 9 | else 10 | # For push events, compare against previous commit 11 | CHANGED_FILES=$(git diff --name-only HEAD~1 HEAD) 12 | fi 13 | 14 | echo "Changed files:" 15 | echo "$CHANGED_FILES" 16 | 17 | # Filter for JSON files in policyDefinitions directory 18 | JSON_FILES=$(echo "$CHANGED_FILES" | grep '^policyDefinitions/.*\.json$' || true) 19 | 20 | echo "Changed JSON files in policyDefinitions:" 21 | echo "$JSON_FILES" 22 | 23 | # Export as space-separated string for compatibility with existing code 24 | ALL_CHANGED_FILES=$(echo "$CHANGED_FILES" | tr '\n' ' ') 25 | JSON_FILES_LIST=$(echo "$JSON_FILES" | tr '\n' ' ') 26 | 27 | echo "all_changed_files=$ALL_CHANGED_FILES" >> $GITHUB_OUTPUT 28 | echo "json_files=$JSON_FILES_LIST" >> $GITHUB_OUTPUT 29 | echo "json_files_count=$(echo "$JSON_FILES" | grep -c . || echo "0")" >> $GITHUB_OUTPUT -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) Microsoft Corporation. 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE -------------------------------------------------------------------------------- /.devcontainer/devcontainer.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "Powershell and Azure CLI 2.0", 3 | "build": { 4 | "dockerfile": "Dockerfile" 5 | }, 6 | 7 | // Set *default* container specific settings.json values on container create. 8 | "settings": { 9 | "terminal.integrated.shell.linux": "/bin/pwsh", 10 | "files.eol": "\n", 11 | }, 12 | 13 | // Add the IDs of extensions you want installed when the container is created. 14 | // Note that some extensions may not work in Alpine Linux. See https://aka.ms/vscode-remote/linux. 15 | "extensions": [ 16 | "github.vscode-pull-request-github", 17 | "ms-azuretools.vscode-docker", 18 | "ms-vscode.azure-account", 19 | "ms-vscode.azurecli", 20 | "ms-azuretools.vscode-bicep" 21 | ], 22 | 23 | // Use 'forwardPorts' to make a list of ports inside the container available locally. 24 | // "forwardPorts": [], 25 | 26 | // Use 'postCreateCommand' to run commands after the container is created. 27 | "postCreateCommand": "uname -a", 28 | 29 | // Uncomment when using a ptrace-based debugger like C++, Go, and Rust 30 | // "runArgs": [ "--cap-add=SYS_PTRACE", "--security-opt", "seccomp=unconfined" ], 31 | 32 | // Uncomment to connect as a non-root user. See https://aka.ms/vscode-remote/containers/non-root. 33 | // "remoteUser": "vscode" 34 | } -------------------------------------------------------------------------------- /utilities/policyAgent/deployDef.ps1: -------------------------------------------------------------------------------- 1 | # Paths for params and deployment inputs 2 | param ( 3 | [string]$PolicyDefinitionFilePath = "./policyDefinitions/allowedLocations.json", 4 | [string]$BicepFilePath = "./utilities/policyAgent/policyDef.bicep", 5 | [string]$ParameterFilePath = "./utilities/policyAgent/policyDef.parameters.json", 6 | [string]$DeploymentName = "YourDeploymentName", 7 | [string]$Location = "eastus" 8 | ) 9 | 10 | # Step 1: Read policy definition JSON and create parameter file 11 | $policyDefinition = Get-Content -Path $policyDefinitionFilePath -Raw | ConvertFrom-Json 12 | 13 | $parameterContent = @{ 14 | "$schema" = "https://schema.management.azure.com/schemas/2019-04-01/deploymentParameters.json#" 15 | "contentVersion" = "1.0.0.0" 16 | "parameters" = @{ 17 | "policyDefinition" = @{ 18 | "value" = $policyDefinition 19 | } 20 | } 21 | } 22 | 23 | $parameterContent | ConvertTo-Json -Depth 10 | Set-Content -Path $parameterFilePath 24 | 25 | # Step 2: Deploy the compiled ARM template with parameters 26 | New-AzSubscriptionDeployment -Location $Location ` 27 | -Name $DeploymentName ` 28 | -TemplateFile $BicepFilePath ` 29 | -TemplateParameterFile $ParameterFilePath ` 30 | -Verbose -------------------------------------------------------------------------------- /infra/bicep/workbooks/MetroAgents.json: -------------------------------------------------------------------------------- 1 | { 2 | "version": "Notebook/1.0", 3 | "items": [ 4 | { 5 | "type": 1, 6 | "content": { 7 | "json": "## MetroAgents\n---\n\nThis workbook will give you line of sight to the [Metro Agent setup](https://github.com/Azure/MetroAgents)", 8 | "style": "success" 9 | }, 10 | "name": "text" 11 | }, 12 | { 13 | "type": 3, 14 | "content": { 15 | "version": "KqlItem/1.0", 16 | "query": "AzureDiagnostics\n| where ResourceProvider == \"MICROSOFT.COGNITIVESERVICES\"\n| where Category == \"RequestResponse\"\n| extend properties = parse_json(properties_s)\n| extend apiVersion = tostring(properties.apiName)\n| extend modelDeploymentName = tostring(properties.modelDeploymentName)\n| extend modelName = tostring(properties.modelName)\n| extend modelVersion = tostring(properties.modelVersion)\n| extend objectId = tostring(properties.objectId)\n| project TimeGenerated, OperationName, ResultSignature, Resource, apiVersion, objectId, modelDeploymentName, modelName, modelVersion", 17 | "size": 1, 18 | "timeContext": { 19 | "durationMs": 86400000 20 | }, 21 | "queryType": 0, 22 | "resourceType": "microsoft.operationalinsights/workspaces" 23 | }, 24 | "name": "query" 25 | } 26 | ], 27 | "fallbackResourceIds": [ 28 | ], 29 | "$schema": "https://github.com/Microsoft/Application-Insights-Workbooks/blob/master/schema/workbook.json" 30 | } -------------------------------------------------------------------------------- /infra/bicep/agentInstructions/defaultProxyAgentInstructions.txt: -------------------------------------------------------------------------------- 1 | You are a proxy AI Agent and your primary purpose is to interacts with, and manage specialized AI Agents to solve complex tasks. 2 | 3 | If you have to create a new agent, you will do that instead of trying to figure out the answer yourself. 4 | 5 | When creating new agents: 6 | 7 | * Check if there is already a suitable agent that can solve the task. 8 | * If you do not have internet directly with Grounding with Bing tool, and are asked to find information on the internet, check first for available agent with Bing tool, if not, you will create an agent with Bing tool to carry out the task. 9 | * Always use model gpt-4o when creating new agents. 10 | * You will always check which agents you have available, and explain if they are more suitable to perform the task given their expertise and access to other tools. 11 | * Provide a name, description, and give instructions as best effort based on the need you have identified. 12 | * Poll the thread until the run has returned as completed. 13 | 14 | When interacting with agents: 15 | 16 | * Always include the agent context in your responses 17 | * Poll the thread until the run has returned as completed 18 | 19 | General behavior: 20 | 21 | * If the topic changes and you have not completed a previous task, always poll the latest information and check for status before starting on something new. 22 | * If a user insist in starting something new, accept the keyword *reset message* to start a new thread and ignore the follow-up from previous tasks. 23 | 24 | Be brief, and always end with a positive spin in your interactions with an emoji or two - leeeezzz goooo! -------------------------------------------------------------------------------- /utilities/policyAgent/policyDef.bicep: -------------------------------------------------------------------------------- 1 | targetScope = 'subscription' 2 | 3 | @description('The policy definition object to be deployed.') 4 | param policyDefinition object 5 | 6 | resource resPolicyDefinitions 'Microsoft.Authorization/policyDefinitions@2021-06-01' = { 7 | name: policyDefinition.name // Ensure the name is unique for the policy definition 8 | properties: { 9 | description: policyDefinition.properties.description 10 | displayName: policyDefinition.properties.displayName 11 | metadata: policyDefinition.properties.metadata 12 | mode: policyDefinition.properties.mode 13 | parameters: policyDefinition.properties.parameters 14 | policyType: policyDefinition.properties.policyType 15 | policyRule: policyDefinition.properties.policyRule 16 | } 17 | } 18 | 19 | resource resPolicyAssignments 'Microsoft.Authorization/policyAssignments@2021-06-01' = { 20 | name: '${resPolicyDefinitions.name}-assignment' 21 | properties: { 22 | displayName: resPolicyDefinitions.properties.displayName 23 | policyDefinitionId: resPolicyDefinitions.id 24 | } 25 | } 26 | 27 | output policyDefinitionId string = resPolicyDefinitions.id 28 | output policyDefinitionName string = resPolicyDefinitions.name 29 | output policyDefinitionObject object = resPolicyDefinitions.properties 30 | output policyDefinitionParameters object = resPolicyDefinitions.properties.parameters 31 | output policyDefinitionRule object = resPolicyDefinitions.properties.policyRule 32 | output policyAssignmentId string = resPolicyAssignments.id 33 | output policyAssignmentName string = resPolicyAssignments.name 34 | output policyAssignmentObject object = resPolicyAssignments.properties 35 | -------------------------------------------------------------------------------- /policyDefinitions/allowedLocations.json.sample: -------------------------------------------------------------------------------- 1 | { 2 | "name": "Allowed-Locations-Resources", 3 | "type": "Microsoft.Authorization/policyDefinitions", 4 | "apiVersion": "2021-06-01", 5 | "scope": null, 6 | "properties": { 7 | "displayName": "Allowed locations for workloads", 8 | "policyType": "Custom", 9 | "mode": "Indexed", 10 | "description": "This policy enables you to restrict the locations your organization can specify when deploying resources. Use to enforce your geo-compliance requirements. Excludes resource groups, Microsoft.AzureActiveDirectory/b2cDirectories, Microsoft.Resources/deployments for DINE, and resources that use the 'global' region.", 11 | "metadata": { 12 | "category": "Ringfencing" 13 | }, 14 | "parameters": { 15 | "listOfAllowedLocations": { 16 | "type": "Array", 17 | "defaultValue": ["eastus"], 18 | "metadata": { 19 | "displayName": "Allowed locations", 20 | "description": "The list of locations that can be specified when deploying resources.", 21 | "strongType": "location" 22 | } 23 | } 24 | }, 25 | "policyRule": { 26 | "if": { 27 | "allOf": [ 28 | { 29 | "field": "location", 30 | "notIn": "[parameters('listOfAllowedLocations')]" 31 | }, 32 | { 33 | "field": "location", 34 | "notEquals": "global" 35 | }, 36 | { 37 | "field": "type", 38 | "notEquals": "Microsoft.AzureActiveDirectory/b2cDirectories" 39 | }, 40 | { 41 | "field": "type", 42 | "notEquals": "Microsoft.Resources/deployments" 43 | } 44 | ] 45 | }, 46 | "then": { 47 | "effect": "deny" 48 | } 49 | } 50 | } 51 | } -------------------------------------------------------------------------------- /infra/bicep/modules/deploymentScript.bicep: -------------------------------------------------------------------------------- 1 | targetScope = 'resourceGroup' 2 | 3 | param location string = resourceGroup().location 4 | param resourceName string = 'depScript' 5 | // param azAIAgentUri string 6 | // param modelDeploymentName string = 'gpt-4.1' 7 | // param scriptContent string = loadTextContent('../scriptContent/azurePolicyAgent.ps1') 8 | 9 | 10 | resource userAssignedIdentity 'Microsoft.ManagedIdentity/userAssignedIdentities@2018-11-30' = { 11 | name: '${resourceName}-uai' 12 | location: location 13 | } 14 | 15 | module roleAssignmentAIAgents 'roleAssignment.bicep' = { 16 | name: 'roleAssignmentAIAgents' 17 | scope: resourceGroup() 18 | params: { 19 | objectId: userAssignedIdentity.properties.principalId 20 | roleDefinitionId: '${subscription().id}/providers/Microsoft.Authorization/roleDefinitions/53ca6127-db72-4b80-b1b0-d745d6d5456d' 21 | principalType: 'ServicePrincipal' 22 | } 23 | } 24 | 25 | // resource initialize 'Microsoft.Resources/deploymentScripts@2020-10-01' = { 26 | // name: 'initialize-azurePolicyAgents' 27 | // location: location 28 | // kind: 'AzurePowerShell' 29 | // dependsOn: [ 30 | // roleAssignmentAIAgents 31 | // ] 32 | // identity: { 33 | // type: 'UserAssigned' 34 | // userAssignedIdentities: { 35 | // '${userAssignedIdentity.id}': {} 36 | // } 37 | // } 38 | // properties: { 39 | // azPowerShellVersion: '7.4' 40 | // retentionInterval: 'PT1H' 41 | // scriptContent: scriptContent 42 | // cleanupPreference: 'OnSuccess' 43 | // timeout: 'PT1H' 44 | // arguments: '-ModelDeploymentName "${modelDeploymentName}" -AIAgentEndpoint "${azAIAgentUri}"' 45 | // } 46 | // } 47 | 48 | // output arguments string = initialize.properties.arguments 49 | output userAssignedIdentityId string = userAssignedIdentity.id 50 | output userAssignedClientId string = userAssignedIdentity.properties.clientId 51 | output userAssignedIdentityObjectId string = userAssignedIdentity.properties.principalId 52 | // output agentId string = initialize.properties.outputs.agentId 53 | // output agentName string = initialize.properties.outputs.agentName 54 | // output deploymentStatus string = initialize.properties.outputs.status 55 | // output deploymentTimestamp string = initialize.properties.outputs.timestamp 56 | 57 | output tenantId string = subscription().tenantId 58 | output subscriptionId string = subscription().subscriptionId 59 | -------------------------------------------------------------------------------- /.github/scripts/deploy-policies.ps1: -------------------------------------------------------------------------------- 1 | param( 2 | [Parameter(Mandatory = $true)] 3 | [string]$JsonFiles, 4 | 5 | [Parameter(Mandatory = $true)] 6 | [string]$JsonFilesCount 7 | ) 8 | 9 | Write-Host "JSON files changed: $JsonFiles" 10 | Write-Host "Number of JSON files: $JsonFilesCount" 11 | 12 | if ([string]::IsNullOrWhiteSpace($JsonFiles) -or $JsonFilesCount -eq "0") { 13 | Write-Warning "No JSON files found in the 'policyDefinitions' directory" 14 | Write-Output "::set-output name=policyContentBase64::" 15 | exit 0 16 | } 17 | 18 | $JsonFilesList = $JsonFiles -split " " | Where-Object { -not [string]::IsNullOrWhiteSpace($_) } 19 | 20 | $AllPolicyContents = @() 21 | $DeploymentResults = @() 22 | 23 | foreach ($JsonFile in $JsonFilesList) { 24 | Write-Host "Processing policy definition file: $JsonFile" 25 | 26 | try { 27 | # Deploy each policy definition 28 | ./utilities/policyAgent/deployDef.ps1 ` 29 | -PolicyDefinitionFilePath $JsonFile ` 30 | -BicepFilePath './utilities/policyAgent/policyDef.bicep' ` 31 | -ParameterFilePath './utilities/policyAgent/policyDef.parameters.json' ` 32 | -Verbose 33 | 34 | Write-Host "Policy definition deployment complete for: $JsonFile" 35 | 36 | # Read and store policy content 37 | $PolicyContent = Get-Content -Path $JsonFile -Raw 38 | $PolicyContentInfo = @{ 39 | FilePath = $JsonFile 40 | Content = $PolicyContent 41 | Base64Content = [Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes($PolicyContent)) 42 | } 43 | 44 | $AllPolicyContents += $PolicyContentInfo 45 | $DeploymentResults += "✅ Successfully deployed: $JsonFile" 46 | 47 | Write-Host "Policy Content Length for ${JsonFile}: $($PolicyContent.Length)" 48 | 49 | } 50 | catch { 51 | Write-Error "Failed to deploy policy definition for: $JsonFile - $($_.Exception.Message)" 52 | $DeploymentResults += "❌ Failed to deploy: $JsonFile - $($_.Exception.Message)" 53 | } 54 | } 55 | 56 | # Convert all policy contents to JSON and then to Base64 for passing to next job 57 | $AllPolicyContentsJson = $AllPolicyContents | ConvertTo-Json -Depth 10 -Compress 58 | $AllPolicyContentsBase64 = [Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes($AllPolicyContentsJson)) 59 | 60 | Write-Host "Deployment Summary:" 61 | $DeploymentResults | ForEach-Object { Write-Host $_ } 62 | 63 | Write-Output "::set-output name=policyContentBase64::$AllPolicyContentsBase64" 64 | Write-Output "::set-output name=processedFilesCount::$($AllPolicyContents.Count)" -------------------------------------------------------------------------------- /infra/bicep/scriptContent/azurePolicyAgent.ps1: -------------------------------------------------------------------------------- 1 | # Starting the Azure Policy Agent deployment using the provided Bicep template 2 | # Remember to modify the agentsSetup.bicepparam file to match your environment with unique naming for the project endpoint 3 | 4 | # Login to Azure and set the subscription context 5 | 6 | Connect-AzAccount 7 | 8 | # Set the subscription ID - replace with your actual subscription ID 9 | 10 | $SubscriptionId = "fcbf5e04-699b-436a-88ef-d72081f7ad52" 11 | Set-AzContext -SubscriptionId $SubscriptionId 12 | 13 | $AzureDeploymentLocation = "swedencentral" 14 | $AzurePolicyAgentDeployment = New-AzSubscriptionDeployment ` 15 | -Name "AzurePolicyAgentDeployment" ` 16 | -Location $AzureDeploymentLocation ` 17 | -TemplateFile './infra/bicep/agentsSetup.bicep' ` 18 | -TemplateParameterFile './infra/bicep/agentsSetup.bicepparam' ` 19 | -Verbose 20 | 21 | # Installing Metro-AI Powershell module for declarative management of Azure AI Agent (and to bypass current limmitations of deploymentScripts) 22 | 23 | Install-Module -Name Metro.AI -Force 24 | 25 | # Setting Metro AI agent context using the deployment outputs 26 | 27 | Set-MetroAIContext -Endpoint $AzurePolicyAgentDeployment.Outputs.agentEndpoint.value -ApiType Agent 28 | 29 | # Creating the Azure Policy Agent using the provided JSON definition 30 | 31 | $AgentDefinition = Invoke-RestMethod -uri "https://gist.githubusercontent.com/krnese/c4ee2c9db19cdd09028d3e7da4ff8141/raw/4507b8cff32fbf4c56e72265da84c4977b4a834f/azurePolicyAgent.json" 32 | 33 | try { 34 | $NewAgent = New-MetroAIAgent -Name "Azure Policy Agent" -InputObject $AgentDefinition 35 | Write-Host "Azure Policy Agent has been created successfully with the ID: $($NewAgent.id)" 36 | } catch { 37 | Write-Host "Failed to create agent. Error: $($_.Exception.Message)" 38 | Write-Host "Agent definition: $($AgentDefinition | ConvertTo-Json -Depth 3)" 39 | throw 40 | } 41 | 42 | # Update AI Agent with Microsoft Docs as MCP server 43 | Set-MetroAIAgent -AssistantId $NewAgent.id ` 44 | -EnableMcp -McpServerLabel 'Microsoft_Learn_MCP' ` 45 | -McpServerUrl 'https://learn.microsoft.com/api/mcp' ` 46 | -Temperature 0.2 ` 47 | -Verbose 48 | 49 | # Writing required outputs to the console for further use in GitHub Actions using Federated Credentials 50 | Write-Host "Repository secrets:" 51 | Write-Host "AZURE_CLIENT_ID: $($AzurePolicyAgentDeployment.Outputs.azureClientId.Value)" 52 | Write-Host "TENANT_ID: $($AzurePolicyAgentDeployment.Outputs.azureTenantId.Value)" 53 | Write-Host "SUBSCRIPTION_ID: $($AzurePolicyAgentDeployment.Outputs.azureSubscriptionId.value)" 54 | 55 | Write-Host "Repository variables:" 56 | Write-Host "PROJECT_ENDPOINT: $($AzurePolicyAgentDeployment.Outputs.projectEndpoint.value)" 57 | Write-Host "Assistant_ID: $($NewAgent.id)" -------------------------------------------------------------------------------- /utilities/policyAgent/policyDef.parameters.json: -------------------------------------------------------------------------------- 1 | { 2 | "": "https://schema.management.azure.com/schemas/2019-04-01/deploymentParameters.json#", 3 | "parameters": { 4 | "policyDefinition": { 5 | "value": { 6 | "name": "Allowed-Locations-Resources", 7 | "type": "Microsoft.Authorization/policyDefinitions", 8 | "apiVersion": "2021-06-01", 9 | "scope": null, 10 | "properties": { 11 | "displayName": "Allowed locations for workloads", 12 | "policyType": "Custom", 13 | "mode": "Indexed", 14 | "description": "This policy enables you to restrict the locations your organization can specify when deploying resources. Use to enforce your geo-compliance requirements. Excludes resource groups, Microsoft.AzureActiveDirectory/b2cDirectories, Microsoft.Resources/deployments for DINE, and resources that use the 'global' region.", 15 | "metadata": { 16 | "category": "Foobar124" 17 | }, 18 | "parameters": { 19 | "listOfAllowedLocations": { 20 | "type": "Array", 21 | "defaultValue": [ 22 | "eastus" 23 | ], 24 | "metadata": { 25 | "displayName": "Allowed locations", 26 | "description": "The list of locations that can be specified when deploying resources.", 27 | "strongType": "location" 28 | } 29 | } 30 | }, 31 | "policyRule": { 32 | "if": { 33 | "allOf": [ 34 | { 35 | "field": "location", 36 | "notIn": "[parameters('listOfAllowedLocations')]" 37 | }, 38 | { 39 | "field": "location", 40 | "notEquals": "global" 41 | }, 42 | { 43 | "field": "type", 44 | "notEquals": "Microsoft.AzureActiveDirectory/b2cDirectories" 45 | }, 46 | { 47 | "field": "type", 48 | "notEquals": "Microsoft.Resources/deployments" 49 | } 50 | ] 51 | }, 52 | "then": { 53 | "effect": "deny" 54 | } 55 | } 56 | } 57 | } 58 | } 59 | }, 60 | "contentVersion": "1.0.0.0" 61 | } -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing 2 | 3 | This project welcomes contributions and suggestions. Most contributions require you to agree to a 4 | Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us 5 | the rights to use your contribution. For details, visit . 6 | 7 | When you submit a pull request, a CLA bot will automatically determine whether you need to provide 8 | a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions 9 | provided by the bot. You will only need to do this once across all repos using our CLA. 10 | 11 | This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). 12 | For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or 13 | contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments. 14 | 15 | - [Code of Conduct](#coc) 16 | - [Issues and Bugs](#issue) 17 | - [Feature Requests](#feature) 18 | - [Submission Guidelines](#submit) 19 | 20 | ## Code of Conduct 21 | 22 | Help us keep this project open and inclusive. Please read and follow our [Code of Conduct](https://opensource.microsoft.com/codeofconduct/). 23 | 24 | ## Found an Issue? 25 | 26 | If you find a bug in the source code or a mistake in the documentation, you can help us by 27 | [submitting an issue](#submit-issue) to the GitHub Repository. Even better, you can 28 | [submit a Pull Request](#submit-pr) with a fix. 29 | 30 | ## Want a Feature? 31 | 32 | You can *request* a new feature by [submitting an issue](#submit-issue) to the GitHub 33 | Repository. If you would like to *implement* a new feature, please submit an issue with 34 | a proposal for your work first, to be sure that we can use it. 35 | 36 | - **Small Features** can be crafted and directly [submitted as a Pull Request](#submit-pr). 37 | 38 | ## Submission Guidelines 39 | 40 | ### Submitting an Issue 41 | 42 | Before you submit an issue, search the archive, maybe your question was already answered. 43 | 44 | If your issue appears to be a bug, and hasn't been reported, open a new issue. 45 | Help us to maximize the effort we can spend fixing issues and adding new 46 | features, by not reporting duplicate issues. Providing the following information will increase the 47 | chances of your issue being dealt with quickly: 48 | 49 | - **Overview of the Issue** - if an error is being thrown a non-minified stack trace helps 50 | - **Version** - what version is affected (e.g. 0.1.2) 51 | - **Motivation for or Use Case** - explain what are you trying to do and why the current behavior is a bug for you 52 | - **Browsers and Operating System** - is this a problem with all browsers? 53 | - **Reproduce the Error** - provide a live example or an unambiguous set of steps 54 | - **Related Issues** - has a similar issue been reported before? 55 | - **Suggest a Fix** - if you can't fix the bug yourself, perhaps you can point to what might be 56 | causing the problem (line of code or commit) 57 | 58 | ### Submitting a Pull Request (PR) 59 | 60 | Before you submit your Pull Request (PR) consider the following guidelines: 61 | 62 | - Search the repository (<) for an open or closed PR 63 | that relates to your submission. You don't want to duplicate effort. 64 | 65 | - Make your changes in a new git fork: 66 | 67 | - Commit your changes using a descriptive commit message 68 | - Push your fork to GitHub: 69 | - In GitHub, create a pull request 70 | - If we suggest changes then: 71 | - Make the required updates. 72 | - Rebase your fork and force push to your GitHub repository (this will update your Pull Request): 73 | 74 | That's it! Thank you for your contribution! -------------------------------------------------------------------------------- /infra/bicep/scriptContent/proxyToAgents.ps1: -------------------------------------------------------------------------------- 1 | param( 2 | [string] $modelDeploymentName, 3 | [string] $azAIProxyInstructions, 4 | [string] $azAIProxyUri, 5 | [string] $openApiDefinitionUri, 6 | [string] $azAIAgentUri, 7 | [string] $azAgentName, 8 | [string] $tenantId 9 | ) 10 | 11 | try { 12 | Write-Host "Requesting Azure access token..." 13 | $TokenRequest = (Get-AzAccessToken -ResourceUrl "https://ai.azure.com/").Token 14 | Write-Host "Access token retrieved successfully." 15 | 16 | Write-Host "Validating input" 17 | Write-Host "Agent name is: $azAgentName" 18 | Write-Host "Agent instruction is: $azAIProxyInstructions" 19 | Write-Host "OpenAPI definition URI is: $openApiDefinitionUri" 20 | Write-Host "Azure AI Proxy URI is: $azAIProxyUri" 21 | Write-Host "Azure AI Agent URI is: $azAIAgentUri" 22 | Write-Host "Tenant ID is: $tenantId" 23 | 24 | Write-Host "Downloading Swagger file from $openApiDefinitionUri..." 25 | $swaggerDefinition = Invoke-RestMethod -Uri $openApiDefinitionUri -UseBasicParsing 26 | 27 | Write-Host "Modifying the URL in the Swagger file..." 28 | $swaggerDefinition.servers[0].url = $azAIAgentUri 29 | Write-Host "Successfully modified the URL." 30 | 31 | Write-Host "Updating the tokenUrl in the Swagger file with tenantId..." 32 | $swaggerDefinition.components.securitySchemes.oauth2ManagedIdentity.flows.clientCredentials.tokenUrl = "https://login.microsoftonline.com/$tenantId/oauth2/v2.0/token" 33 | Write-Host "Successfully updated the tokenUrl with tenantId." 34 | 35 | Write-Host "Converting the modified Swagger definition to JSON..." 36 | $modifiedSwaggerJson = $swaggerDefinition | ConvertTo-Json -Depth 100 37 | Write-Host "Successfully converted the Swagger definition to JSON." 38 | 39 | $localPath = 'modifiedSwagger.json' 40 | Write-Host "Saving the modified Swagger file to $localPath..." 41 | $modifiedSwaggerJson | Set-Content -Path $localPath -Encoding utf8 42 | Write-Host "Successfully saved the modified Swagger file." 43 | 44 | Write-Host "Creating the agent..." 45 | $authHeader = @{ 46 | Authorization = "Bearer $($TokenRequest)" 47 | "x-ms-enable-preview" = "true" 48 | } 49 | 50 | $bodyCreateAgent = @{ 51 | instructions = $azAIProxyInstructions 52 | name = $azAgentName 53 | model = $modelDeploymentName 54 | } | ConvertTo-Json -Depth 100 55 | 56 | $agentUri = "$azAIProxyUri/assistants?api-version=2025-05-15-preview" 57 | 58 | $createAgentResponse = Invoke-RestMethod -Uri $agentUri -Method Post -Headers $authHeader -ContentType "application/json" -Body $bodyCreateAgent 59 | Write-Host "Agent creation response: $($createAgentResponse | ConvertTo-Json -Depth 100)" 60 | 61 | Write-Host "Adding the OpenAPI definition to the agent..." 62 | $openAPISpec = Get-Content -Path $localPath -Raw 63 | 64 | $body = @{ 65 | tools = @( 66 | @{ 67 | type = "openapi" 68 | openapi = @{ 69 | name = "AzureAIAgentAPI" 70 | description = "Open API for Azure AI Agent" 71 | auth = @{ 72 | type = "managed_identity" 73 | security_scheme = @{ 74 | audience = "https://ai.azure.com/" 75 | } 76 | } 77 | spec = $openAPISpec | ConvertFrom-Json 78 | } 79 | } 80 | ) 81 | } | ConvertTo-Json -Depth 100 82 | 83 | $agentUri = "$azAIProxyUri/assistants/$($createAgentResponse.id)?api-version=2025-05-15-preview" 84 | $response = Invoke-RestMethod -Uri $agentUri -Method Post -Headers $authHeader -ContentType "application/json" -Body $body 85 | Write-Host "Successfully added OpenAPI definition to the agent. Response: $($response | ConvertTo-Json -Depth 100)" 86 | } 87 | catch { 88 | Write-Host "Error occurred: $_" 89 | exit 1 90 | } -------------------------------------------------------------------------------- /infra/bicep/agentInstructions/AzurePolicyDefinitionDeveloperAgent.txt: -------------------------------------------------------------------------------- 1 | # System Prompt: Azure Policy Definition Creation Agent 2 | 3 | You are an Azure Policy expert whose job is to create Azure Policy definitions based on user requests. 4 | 5 | Your output **must be a valid Azure Policy definition** that adheres strictly to the following requirements, so that the resulting policy can be consumed by an automated policy test agent. 6 | 7 | --- 8 | 9 | ## Output Format 10 | 11 | The Azure Policy definition you generate **must**: 12 | 13 | 1. Be valid JSON. 14 | 2. Follow the Azure Policy definition schema as shown below. 15 | 3. Include all required top-level properties: `name`, `type`, `apiVersion`, `properties`. 16 | 4. Under `properties`, include at least: `displayName`, `policyType`, `mode`, `description`, `metadata`, `parameters` (if needed), and `policyRule`. 17 | 5. The `policyRule` property must define the policy logic, including targeted resource types and the policy effect (`deny`, `audit`, etc.). 18 | 6. If parameters are included, ensure the `parameters` object contains `type`, `defaultValue`, and a `metadata` object with `displayName` and `description`. 19 | 7. Ensure parameter usage within `policyRule` follows Azure Policy template language and references parameters with `[parameters('paramName')]`. 20 | 8. Exclude optional or unnecessary properties unless required for the scenario. 21 | 9. Use clear, concise descriptions for `displayName`, `description`, and `metadata`. 22 | 10. Do **not** include any PowerShell, Bash, or scripting content—**output only the JSON policy definition**. 23 | 11. Use your tool "bing_grounding" to validate and ensure accurate schema, API version, resource properties, and policy aliases are correctly used. 24 | 25 | --- 26 | 27 | ## Example Strict Policy Definition Format 28 | 29 | ```json 30 | { 31 | "name": "Allowed-Locations-Resources", 32 | "type": "Microsoft.Authorization/policyDefinitions", 33 | "apiVersion": "2021-06-01", 34 | "scope": null, 35 | "properties": { 36 | "displayName": "Allowed locations for workloads", 37 | "policyType": "Custom", 38 | "mode": "Indexed", 39 | "description": "This policy enables you to restrict the locations your organization can specify when deploying resources. Use to enforce your geo-compliance requirements. Excludes resource groups, Microsoft.AzureActiveDirectory/b2cDirectories, Microsoft.Resources/deployments for DINE, and resources that use the 'global' region.", 40 | "metadata": { 41 | "category": "General" 42 | }, 43 | "parameters": { 44 | "listOfAllowedLocations": { 45 | "type": "Array", 46 | "defaultValue": ["eastus"], 47 | "metadata": { 48 | "displayName": "Allowed locations", 49 | "description": "The list of locations that can be specified when deploying resources.", 50 | "strongType": "location" 51 | } 52 | } 53 | }, 54 | "policyRule": { 55 | "if": { 56 | "allOf": [ 57 | { 58 | "field": "location", 59 | "notIn": "[parameters('listOfAllowedLocations')]" 60 | }, 61 | { 62 | "field": "location", 63 | "notEquals": "global" 64 | }, 65 | { 66 | "field": "type", 67 | "notEquals": "Microsoft.AzureActiveDirectory/b2cDirectories" 68 | }, 69 | { 70 | "field": "type", 71 | "notEquals": "Microsoft.Resources/deployments" 72 | } 73 | ] 74 | }, 75 | "then": { 76 | "effect": "deny" 77 | } 78 | } 79 | } 80 | } 81 | ``` 82 | 83 | # Key Instructions 84 | 85 | * Analyze the user’s requirements and scenario. If uncertain, ask for clarification before proceeding to generate the policy JSON. 86 | * Generate a single, valid Azure Policy definition in JSON format that enforces the required policy logic. 87 | * For specific values (e.g., certain resource kinds, ports, skus etc., ensure they are defined as defaultValues in corresponding parameters.) 88 | * Your output must be directly usable as input to the automated policy-testing agent. 89 | * Do not include any comments, explanations, or non-JSON output. 90 | 91 | Your only output should be a valid Azure Policy definition JSON, matching the schema and style shown above. -------------------------------------------------------------------------------- /.github/workflows/PolicyAgent.yml: -------------------------------------------------------------------------------- 1 | name: Azure Policy Agent 2 | 3 | on: 4 | push: 5 | branches: 6 | - main 7 | pull_request: 8 | branches: 9 | - main 10 | paths: 11 | - "policyDefinitions/*.json" 12 | 13 | permissions: 14 | id-token: write 15 | contents: write 16 | pull-requests: write 17 | 18 | jobs: 19 | PolicyDefinition: 20 | if: github.event_name == 'pull_request' && github.event.action != 'closed' # Only run on pull requests 21 | runs-on: ubuntu-latest 22 | outputs: 23 | policyContentBase64: ${{ steps.deploy-policy.outputs.policyContentBase64 }} 24 | processedFilesCount: ${{ steps.deploy-policy.outputs.processedFilesCount }} 25 | steps: 26 | - name: Checkout repository 27 | uses: actions/checkout@v2 28 | with: 29 | fetch-depth: 0 30 | 31 | - name: Azure Login 32 | uses: azure/login@v1 33 | with: 34 | client-id: ${{ secrets.AZURE_CLIENT_ID }} 35 | tenant-id: ${{ secrets.AZURE_TENANT_ID }} 36 | subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }} 37 | enable-AzPSSession: true 38 | 39 | - name: Get changed files 40 | id: changed-files 41 | shell: bash 42 | env: 43 | GITHUB_EVENT_NAME: ${{ github.event_name }} 44 | PR_BASE_SHA: ${{ github.event.pull_request.base.sha }} 45 | PR_HEAD_SHA: ${{ github.event.pull_request.head.sha }} 46 | run: | 47 | chmod +x ./.github/scripts/get-changed-files.sh 48 | ./.github/scripts/get-changed-files.sh 49 | 50 | - name: Deploy Azure Policy Definition and Assignment 51 | id: deploy-policy 52 | uses: azure/powershell@v2 53 | with: 54 | azPSVersion: latest 55 | inlineScript: | 56 | ./.github/scripts/deploy-policies.ps1 ` 57 | -JsonFiles "${{ steps.changed-files.outputs.json_files }}" ` 58 | -JsonFilesCount "${{ steps.changed-files.outputs.json_files_count }}" 59 | 60 | PolicyAgent: 61 | runs-on: ubuntu-latest 62 | if: needs.PolicyDefinition.outputs.policyContentBase64 != '' 63 | needs: PolicyDefinition 64 | env: 65 | PROJECT_ENDPOINT: ${{ vars.PROJECT_ENDPOINT }} 66 | ASSISTANT_ID: ${{ vars.ASSISTANT_ID }} 67 | AZUREPS_HOST_ENVIRONMENT: "GitHubActions" 68 | steps: 69 | - name: Checkout repository 70 | uses: actions/checkout@v2 71 | with: 72 | fetch-depth: 0 73 | # 74 | - name: Azure Login 75 | uses: azure/login@v1 76 | with: 77 | client-id: ${{ secrets.AZURE_CLIENT_ID }} 78 | tenant-id: ${{ secrets.AZURE_TENANT_ID }} 79 | subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }} 80 | enable-AzPSSession: true 81 | 82 | - name: Get Policy Content from Previous Job 83 | id: get-policy-content 84 | run: | 85 | echo "Retrieved PolicyContentBase64:" 86 | echo "${{ needs.PolicyDefinition.outputs.policyContentBase64 }}" 87 | echo "Number of processed files: ${{ needs.PolicyDefinition.outputs.processedFilesCount }}" 88 | 89 | # Decode the Base64 content which contains JSON array of all policy files 90 | echo "${{ needs.PolicyDefinition.outputs.policyContentBase64 }}" | base64 --decode > allPolicyContents.json 91 | 92 | echo "Contents of allPolicyContents.json:" 93 | cat allPolicyContents.json 94 | 95 | # Extract individual policy files for processing 96 | jq -r '.[] | @base64' allPolicyContents.json > policyFiles.txt || echo "[]" > policyFiles.txt 97 | shell: bash 98 | 99 | - name: Policy Testing (AI Foundry Agent Service) 100 | uses: azure/powershell@v2 101 | with: 102 | azPSVersion: latest 103 | inlineScript: | 104 | ./.github/scripts/test-policies.ps1 ` 105 | -Endpoint $env:PROJECT_ENDPOINT ` 106 | -AssistantId $env:ASSISTANT_ID 107 | 108 | - name: "Post Result as Comment" 109 | if: ${{ needs.PolicyDefinition.result == 'success' }} 110 | shell: bash 111 | run: | 112 | if [ -f /tmp/RESULT.md ]; then 113 | gh pr comment ${{ github.event.pull_request.number }} --body-file /tmp/RESULT.md 114 | cat /tmp/RESULT.md 115 | else 116 | echo "There are no results, so no comments are posted. For further information, please look into the validate step." 117 | fi 118 | env: 119 | GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} -------------------------------------------------------------------------------- /infra/bicep/modules/azureAIServices.bicep: -------------------------------------------------------------------------------- 1 | targetScope = 'resourceGroup' 2 | 3 | param location string = resourceGroup().location 4 | 5 | param resourceName string = 'knff' 6 | 7 | param modelDeploymentName string = 'gpt-4o' 8 | 9 | param modelSkuName string = 'DataZoneStandard' 10 | 11 | param modelCapacity int = 20 12 | 13 | param modelName string = 'gpt-4o' 14 | 15 | param modelVersion string = '2024-05-13' 16 | 17 | param skuName string = 'S0' 18 | 19 | param logAnaltyicsWorkspaceId string = '' 20 | 21 | param bingGroundingKey string = '' 22 | 23 | param bingGroundingResourceId string = '' 24 | 25 | // required roleDefinitions for RBAC'ing the AI project to the AI services 26 | var roleDefinitions = [ 27 | 'a97b65f3-24c7-4388-baec-2e87135dc908' 28 | '5e0bd9bd-7b93-4f28-af87-19fc36ad61bd' 29 | '25fbc0a9-bd7c-42a3-aa1a-3b75d497ee68' 30 | ] 31 | 32 | resource aiServices 'Microsoft.CognitiveServices/accounts@2025-04-01-preview' = { 33 | name: '${resourceName}-ais' 34 | location: location 35 | sku: { 36 | name: skuName 37 | } 38 | identity: { 39 | type: 'SystemAssigned' 40 | } 41 | kind: 'AIServices' 42 | properties: { 43 | allowProjectManagement: true 44 | customSubDomainName: '${resourceName}-ais' 45 | publicNetworkAccess: 'Enabled' 46 | disableLocalAuth: true 47 | restrictOutboundNetworkAccess: false 48 | } 49 | } 50 | 51 | resource aiDeployment 'Microsoft.CognitiveServices/accounts/deployments@2025-04-01-preview' = { 52 | name: modelDeploymentName 53 | parent: aiServices 54 | 55 | sku: { 56 | capacity: modelCapacity 57 | name: modelSkuName 58 | } 59 | properties: { 60 | model: { 61 | format: 'OpenAI' 62 | name: modelName 63 | version: modelVersion 64 | } 65 | versionUpgradeOption: 'OnceNewDefaultVersionAvailable' 66 | } 67 | } 68 | 69 | resource project 'Microsoft.CognitiveServices/accounts/projects@2025-04-01-preview' = { 70 | name: '${resourceName}-project' 71 | parent: aiServices 72 | location: location 73 | identity: { 74 | type: 'SystemAssigned' 75 | } 76 | properties: { 77 | description: '${resourceName} AI Project' 78 | displayName: '${resourceName} AI Project' 79 | } 80 | } 81 | 82 | resource accountCapabilityHost 'Microsoft.CognitiveServices/accounts/capabilityHosts@2025-04-01-preview' = { 83 | name: '${resourceName}-accountCapabilityHost' 84 | parent: aiServices 85 | properties: { 86 | capabilityHostKind: 'Agents' 87 | } 88 | } 89 | 90 | resource projectCapabilityHost 'Microsoft.CognitiveServices/accounts/projects/capabilityHosts@2025-04-01-preview' = { 91 | name: '${resourceName}-projectCapabilityHost' 92 | parent: project 93 | properties: { 94 | capabilityHostKind: 'Agents' 95 | } 96 | dependsOn: [ 97 | accountCapabilityHost 98 | ] 99 | } 100 | 101 | resource groundingWithBingConnection 'Microsoft.CognitiveServices/accounts/connections@2025-04-01-preview' = if (!empty(bingGroundingKey) && !empty(bingGroundingResourceId)) { 102 | name: '${resourceName}-bing-grounding' 103 | parent: aiServices 104 | properties: { 105 | category: 'ApiKey' 106 | target: 'https://api.bing.microsoft.com/' 107 | authType: 'ApiKey' 108 | credentials: { 109 | key: bingGroundingKey 110 | } 111 | isSharedToAll: true 112 | metadata: { 113 | ApiType: 'Azure' 114 | type: 'bing_grounding' 115 | ResourceId: bingGroundingResourceId 116 | } 117 | } 118 | } 119 | 120 | resource roleAssignmentLoop 'Microsoft.Authorization/roleAssignments@2022-04-01' = [ 121 | for roleDefinition in roleDefinitions: { 122 | name: guid( 123 | subscription().subscriptionId, 124 | '${subscription().id}/providers/Microsoft.Authorization/roleDefinitions/${roleDefinition}', 125 | '${resourceName}' 126 | ) 127 | scope: aiServices 128 | properties: { 129 | principalId: project.identity.principalId 130 | roleDefinitionId: resourceId('Microsoft.Authorization/roleDefinitions', roleDefinition) 131 | principalType: 'ServicePrincipal' 132 | } 133 | } 134 | ] 135 | 136 | // Enabling diagnostics for the AI Service account (Microsoft.CognitiveServices) 137 | resource diagnosticSetting 'Microsoft.Insights/diagnosticSettings@2021-05-01-preview' = if (!(empty(logAnaltyicsWorkspaceId))) { 138 | name: 'diag' 139 | scope: aiServices 140 | properties: { 141 | workspaceId: logAnaltyicsWorkspaceId 142 | logs: [ 143 | { 144 | categoryGroup: 'allLogs' 145 | enabled: true 146 | } 147 | ] 148 | metrics: [ 149 | { 150 | category: 'AllMetrics' 151 | enabled: true 152 | } 153 | ] 154 | } 155 | } 156 | 157 | output endpoint string = aiServices.properties.endpoint 158 | output identityObjectId string = aiServices.identity.principalId 159 | output agentEndpoint string = project.properties.endpoints['AI Foundry API'] 160 | output modelName string = aiDeployment.properties.model.name 161 | output modelDeploymentName string = aiDeployment.name 162 | output projectResourceId string = project.id 163 | output aiHubName string = aiServices.name 164 | output aiProjectName string = project.name 165 | output bingGroundingConnectionId string = groundingWithBingConnection.id 166 | -------------------------------------------------------------------------------- /infra/bicep/agentsSetup.bicep: -------------------------------------------------------------------------------- 1 | targetScope = 'subscription' 2 | 3 | @description('Location for all resources.') 4 | param location string = deployment().location 5 | 6 | @description('Resource group name') 7 | param rgName string = '' 8 | 9 | @description('Resource name prefix') 10 | param resourceName string = '' 11 | 12 | @description('Model name') 13 | param agentModelName string = 'gpt-4o' 14 | 15 | @description('Model version') 16 | param agentModelVersion string = '2024-05-13' 17 | 18 | @description('Model deployment name') 19 | param agentModelDeploymentName string = 'gpt-4o' 20 | 21 | @description('Agent model SKU name') 22 | param agentModelSkuName string = 'DataZoneStandard' 23 | 24 | @description('Model capacity') 25 | param agentModelCapacity int = 150 26 | 27 | @description('Add knowledge to the AI Agents') 28 | @allowed([ 29 | 'none' 30 | 'groundingWithBing' 31 | 'aiSearch' 32 | ]) 33 | param addKnowledge string = 'none' 34 | 35 | module rg './modules/rg.bicep' = { 36 | name: 'rg-${location}' 37 | params: { 38 | rgName: rgName 39 | location: location 40 | } 41 | } 42 | 43 | // Deploy Log Analytics workspace 44 | module logAnalyticsWorkspace './modules/logAnalytics.bicep' = { 45 | name: 'logAnalyticsWorkspace' 46 | dependsOn: [ 47 | rg 48 | ] 49 | scope: resourceGroup(rgName) 50 | params: { 51 | location: location 52 | resourceName: '${resourceName}-logAnalytics' 53 | skuName: 'PerGB2018' 54 | retentionInDays: 30 55 | } 56 | } 57 | 58 | // Optionally add Grounding with Bing 59 | module groundingWithBing './modules/bingGrounding.bicep' = if (addKnowledge == 'groundingWithBing') { 60 | name: 'groundingWithBing' 61 | scope: resourceGroup(rgName) 62 | dependsOn: [ 63 | rg 64 | ] 65 | params: { 66 | resourceName: '${resourceName}-bingGrounding' 67 | } 68 | } 69 | 70 | // Deploy Azure AI Agents 71 | module azureAIAgents './modules/azureAIServices.bicep' = { 72 | name: 'azureAIAgents' 73 | dependsOn: [ 74 | rg 75 | ] 76 | scope: resourceGroup(rgName) 77 | params: { 78 | location: location 79 | resourceName: '${resourceName}-agents' 80 | modelCapacity: agentModelCapacity 81 | modelName: agentModelName 82 | modelVersion: agentModelVersion 83 | modelSkuName: agentModelSkuName 84 | modelDeploymentName: agentModelDeploymentName 85 | logAnaltyicsWorkspaceId: logAnalyticsWorkspace.outputs.logAnalyticsWorkspaceId 86 | bingGroundingKey: addKnowledge == 'groundingWithBing' ? groundingWithBing.outputs.bingKeys : '' 87 | bingGroundingResourceId: addKnowledge == 'groundingWithBing' ? groundingWithBing.outputs.bingResourceId : '' 88 | } 89 | } 90 | 91 | module roleAssignments './modules/roleAssignment.bicep' = { 92 | name: 'roleAssignments' 93 | scope: resourceGroup(rgName) 94 | params: { 95 | objectId: azureAIAgents.outputs.identityObjectId 96 | principalType: 'ServicePrincipal' 97 | roleDefinitionId: '${subscription().id}/providers/Microsoft.Authorization/roleDefinitions/64702f94-c441-49e6-a78b-ef80e0188fee' 98 | } 99 | } 100 | 101 | // Role assignment for the callerId submitting the deployment 102 | module callerIdRoleAssignments './modules/roleAssignment.bicep' = { 103 | name: 'userRoleAssignments' 104 | scope: resourceGroup(rgName) 105 | dependsOn: [ 106 | rg 107 | ] 108 | params: { 109 | objectId: deployer().objectId 110 | principalType: 'User' 111 | roleDefinitionId: '${subscription().id}/providers/Microsoft.Authorization/roleDefinitions/53ca6127-db72-4b80-b1b0-d745d6d5456d' 112 | } 113 | } 114 | 115 | // Deployment script to initialize the Policy Agent 116 | module initializeAgentSetup './modules/deploymentScript.bicep' = { 117 | name: 'initializeAgentProxySetup' 118 | dependsOn: [ 119 | rg 120 | roleAssignments 121 | ] 122 | scope: resourceGroup(rgName) 123 | params: { 124 | location: location 125 | resourceName: resourceName 126 | } 127 | } 128 | 129 | module umiRoleAssignment './modules/subRoleAssignment.bicep' = { 130 | name: 'umiRoleAssignment' 131 | scope: subscription() 132 | params: { 133 | objectId: initializeAgentSetup.outputs.userAssignedIdentityObjectId 134 | principalType: 'ServicePrincipal' 135 | roleDefinitionId: '${subscription().id}/providers/Microsoft.Authorization/roleDefinitions/8e3af657-a8ff-443c-a75c-2fe8c4bcb635' 136 | } 137 | } 138 | 139 | // Azure AI Agents Outputs 140 | output agentEndpoint string = azureAIAgents.outputs.agentEndpoint 141 | output agentModelName string = azureAIAgents.outputs.modelName 142 | output agentModelDeploymentName string = azureAIAgents.outputs.modelDeploymentName 143 | output agentProjectResourceId string = azureAIAgents.outputs.projectResourceId 144 | output userAssignedIdentityObjectId string = initializeAgentSetup.outputs.userAssignedIdentityObjectId 145 | 146 | // Shared Infrastructure Outputs 147 | output resourceGroupName string = rgName 148 | 149 | // Bing Grounding Outputs 150 | output bingConnectionId string = azureAIAgents.outputs.bingGroundingConnectionId 151 | 152 | // Github output for secrets and variables 153 | output azureClientId string = initializeAgentSetup.outputs.userAssignedClientId 154 | output azureTenantId string = initializeAgentSetup.outputs.tenantId 155 | output azureSubscriptionId string = initializeAgentSetup.outputs.subscriptionId 156 | output projectEndpoint string = azureAIAgents.outputs.agentEndpoint 157 | -------------------------------------------------------------------------------- /infra/bicep/agentInstructions/MetroAIOrhcestratorAgent.txt: -------------------------------------------------------------------------------- 1 | You are a **AI Agent Orchestrator** specializing in **customer support** for Microsoft products. Your role is to efficiently **route inquiries** to specialized AI agents, ensuring **precise, structured, and complete** responses. 2 | 3 | You manage a team of **specialized AI agents** who handle complex tasks across **M365, Azure, Power Platform, D365, Copilot Studio, Microsoft Graph, Xbox, Copilot Studio Agent, Log Analytics, PurviewExpertAgent, CustomerCaseAgent, and DLPExpertAgent**. Each agent has access to **real-time web search (Grounding with Bing)** and uses the **GPT-4o** model. 4 | 5 | You will **NEVER respond to the question directly, interpret agent output, or summarize agent answers**. Your job is to route, monitor, and return results **as-is** from agents with proper structure and formatting. 6 | 7 | --- 8 | 9 | ## Core Directives: 10 | - **Only respond if the inquiry relates to Microsoft products.** 11 | - **NEVER create new agents — only use the connected agents.** 12 | - **NEVER perform web searches yourself—only agents do that.** 13 | - **ALWAYS poll until ALL agent runs are completed before responding.** 14 | - **ALWAYS maintain exact formatting and content from agent responses.** 15 | - **DO NOT interpret or synthesize agent replies — preserve original structure.** 16 | 17 | --- 18 | 19 | ## Available AI Agents: 20 | | **Agent** | **Expertise** | 21 | |-----------|--------------| 22 | | **AzureAgent** | Azure, cloud computing, partner ecosystem, IaC, and CLI tooling | 23 | | **D365Agent** | Dynamics 365, Finance & Operations, Contact Center, Dataverse, licensing | 24 | | **M365Agent** | Microsoft 365, Copilot, EntraID, Word, Excel, SharePoint, Visio, Loop | 25 | | **MicrosoftGraphAgent** | Microsoft Graph REST APIs for users, groups, Teams, devices, and more | 26 | | **PowerPlatformAgent** | Power Platform (Power Apps, Automate, Desktop Flows, AI Builder, DLP) | 27 | | **XboxAgent** | Xbox platform, features, support, gaming trends | 28 | | **PurviewExpertAgent** | Microsoft Purview, compliance, encryption, audit, data classification | 29 | | **ResourceGraphAgent** | Azure Resource Graph queries, RBAC and policy assessment, regional usage. Always request `subscriptionId` from user. Provide raw output with queries shown. | 30 | | **LogAnalyticsAgent** | Azure Monitor, Log Analytics, KQL, agent health and diagnostics | 31 | | **CopilotProxyAgent** | Copilot Studio Agent orchestration and coordination | 32 | | **CustomerCaseAgent** | Customer Case Agent that helps to look up information in Dataverse for accounts, incidents, incident details, and more. This agent is also used to compile and send required emails to customers when asked for, using the information generated by the other agents that you will interact with, and you MUST provide to this agent when users are asking for email summary| 33 | | **AzurePolicyDefinitionDeveloperAgent** | Azure Policy Definition Developer Agent that helps to create and develope Azure Policy Definitions. The output from this AI Agent will always be in JSON format, which my may sometimes share with the **AzurePolicyTesterAgent** to generate automated scripts for download.| 34 | | **AzurePolicyTesterAgent** | Azure Policy Tester Agent that helps to test Azure Policy Definitions. The output from this AI Agent will always be in PowerShell script format, which my will provide back to the user for file download. | 35 | 36 | --- 37 | 38 | ## Handling Agent Interactions via API: 39 | 1. If multiple agents are needed, **run all simultaneously**. 40 | 2. **Poll all threads** and ensure completion before responding. 41 | 3. **Preserve threadId** across conversations unless user requests reset. 42 | 43 | --- 44 | 45 | ## File Handling: 46 | - If files are provided: 47 | 1. **Assess content type** (e.g., flow.json, compliance report, KQL export). 48 | 2. **Route to the most appropriate agent** based on domain. 49 | 3. Include a **short summary of the file** when handing it off. 50 | 4. Agents must be given access to file content along with the prompt. 51 | 52 | --- 53 | 54 | ## Response Guidelines: 55 | 56 | ### Step 1: Confirm Agent Selection 57 | - Say: **"Routing the question to {AgentName}."** 58 | - Explain briefly **why** this agent is being selected (only on first occurrence per topic). 59 | 60 | ### Step 2: Categorized Response Format 61 | - **Response from {Agent Name}:** Include exact agent response with no modification. 62 | - If multiple agents are used, clearly separate their replies. 63 | 64 | ### Step 3: Completion Verification 65 | - **Poll all threads until they are fully complete.** Never respond early. 66 | 67 | ### Step 4: Provide Attribution 68 | - At the bottom of each response include: 69 | - **Tools Used:** Web Search 🔍, Internal KB 📄, Live API 🔗 70 | 71 | ### Step 5: Closing Message 72 | - Ask: **"Did this solve your issue? Do you need further help?"** 73 | - Close with a warm tone and a fitting emoji or two. 🎯✅🔥 74 | 75 | ### Step 6: Thread Continuity 76 | - If continuing a conversation: **retain the same threadId**. 77 | - If user confirms issue is resolved, ask whether to **start a new thread or continue**. 78 | - If the topic changes but user has not confirmed resolution, ask explicitly what to do. 79 | 80 | --- 81 | 82 | ## General Behavior Rules: 83 | - **Never synthesize agent content.** 84 | - **Always maintain structured format and show tool attribution.** 85 | - If an agent reports uncertainty, **do not override** or reinterpret it. 86 | - If the user says "reset message" → clear thread context and confirm reset. 87 | - If asked to transfer to human support: 88 | - Stop further runs. 89 | - **Generate a summary** of relevant context and share it with the supervisor. 90 | - Observe the rest of the conversation but do not intervene. 91 | - If agents are interacting with each other, **preserve continuity via shared threadId**. 92 | 93 | --- 94 | 95 | ## Final Enhancements: 96 | - Strong emphasis on structured formatting, tool attribution, and orchestration discipline 97 | - Clear thread continuity and file handling rules 98 | - Support escalation rules and resolution checkpoints 99 | - Fun but reliable tone on closing 🎉🚀 -------------------------------------------------------------------------------- /infra/bicep/agentInstructions/AzurePolicyTesterAgent.txt: -------------------------------------------------------------------------------- 1 | # System Prompt: Azure Policy Test Generation Agent 2 | 3 | You are an Azure Policy expert that generates Azure policy tests based on the provided prompts. 4 | 5 | The prompts provided are always in the following format, where the key information to consider is the "parameters" object with its defaultValues, and the "policyRule" object, which shows the logic, targeted resource types, and the policy effect. 6 | 7 | ```json 8 | { 9 | "name": "Allowed-Locations-Resources", 10 | "type": "Microsoft.Authorization/policyDefinitions", 11 | "apiVersion": "2021-06-01", 12 | "scope": null, 13 | "properties": { 14 | "displayName": "Allowed locations for workloads", 15 | "policyType": "Custom", 16 | "mode": "Indexed", 17 | "description": "This policy enables you to restrict the locations your organization can specify when deploying resources. Use to enforce your geo-compliance requirements. Excludes resource groups, Microsoft.AzureActiveDirectory/b2cDirectories, Microsoft.Resources/deployments for DINE, and resources that use the 'global' region.", 18 | "metadata": { 19 | "category": "General" 20 | }, 21 | "parameters": { 22 | "listOfAllowedLocations": { 23 | "type": "Array", 24 | "defaultValue": ["eastus"], 25 | "metadata": { 26 | "displayName": "Allowed locations", 27 | "description": "The list of locations that can be specified when deploying resources.", 28 | "strongType": "location" 29 | } 30 | } 31 | }, 32 | "policyRule": { 33 | "if": { 34 | "allOf": [ 35 | { 36 | "field": "location", 37 | "notIn": "[parameters('listOfAllowedLocations')]" 38 | }, 39 | { 40 | "field": "location", 41 | "notEquals": "global" 42 | }, 43 | { 44 | "field": "type", 45 | "notEquals": "Microsoft.AzureActiveDirectory/b2cDirectories" 46 | }, 47 | { 48 | "field": "type", 49 | "notEquals": "Microsoft.Resources/deployments" 50 | } 51 | ] 52 | }, 53 | "then": { 54 | "effect": "deny" 55 | } 56 | } 57 | } 58 | } 59 | ``` 60 | 61 | The goal is to generate a single test that validates the policy rule that will be non-compliant. The test should be in the following format: 62 | 63 | ```powerShell 64 | Invoke-AzRestMethod -uri "https://management.azure.com/subscriptions/67c92640-33e0-40e6-b8d9-6518d1b4fadc1/resourceGroups/kneast-rg-eastus/providers/{provider subject to what is defined in the policy rule}/{resource type subject to what is defined in the policy rule}/{random name}?api-version=2020-06-01" -Payload $payload -Method PUT 65 | ``` 66 | 67 | Where `$payload` is a JSON object that represents the resource being created, and it should be structured according to the policy rule's requirements. 68 | 69 | 70 | 1. You will generate the required PowerShell script that will test and validate the Azure Policy that was provided from the user. 71 | 2. Some resourceTypes may require the “kind” property to be set. If so, set it to to a valid kind. 72 | 3. Whenever a location must be specified, always use “eastus”, unless the intent is to validate the location itself. 73 | 4. If a policy is targeting resourceTypes, never use “Microsoft.Compute/virtualMachines” as an example, and use storage accounts with account type defined as much as possible as it is lightweight, with unique storage account name that does not exceed the limit. 74 | 5. If the policy rule is targeting child resources, e.g., ‘Microsoft.Automation/automationAccounts/variables’, then you must first create the parent resource with the minimum required valid request body, and the subsequent request must be child to that parent resource with the correct resource name, minimum required valid request body, and only the request of the child resource must be written to the logging.json file. 75 | 6. You MUST produce a valid JSON for the payload, and not add any # characters for Powershell documentation purposes, as it will break the JSON. 76 | 7. ALWAYS validate with latest available operations, API versions etc., policy aliases, REST API schemas, using first, your MCP Server tool which provides grounding on Microsoft Docs, and secondly your "bing_grounding" tool to get the latest information and ensure the generated PowerShell script is valid and executable. 77 | 78 | Provide the output in the exact format as shown in the below example, into dedicated PowerShell file so I can download from you directly. The filename should always be named in a consistent format, such as “policy-{azureservice}.ps1” me to easily download: 79 | 80 | ```powershell 81 | # Define the JSON payload as a string 82 | $Payload = @' 83 | { 84 | "location": "eastus2", 85 | "properties": { 86 | "supportsHttpsTrafficOnly": true 87 | }, 88 | "sku": { 89 | "name": "Standard_LRS" 90 | }, 91 | "kind": "StorageV2" 92 | } 93 | '@ 94 | 95 | # Display the JSON payload (optional for debugging purposes) 96 | Write-Host "Payload for creating a storage account:" 97 | Write-Host $Payload 98 | 99 | # Define the REST API URI 100 | $subscriptionId = (Get-AzContext).Subscription.Id 101 | $resourceGroupName = "AIPolicy" # Always use AIPolicy as the resource group name 102 | $storageAccountName = "knfq234fds" # Create unique storage account name subject to the constraints and requirements 103 | $apiVersion = "2021-04-01" # Always use the valid and latest API version for the resourceProvider that is in scope for the payload 104 | $Uri = "https://management.azure.com/subscriptions/$($subscriptionId)/resourceGroups/$($resourceGroupName)/providers/Microsoft.Storage/storageAccounts/$($storageAccountName)?api-version=$($2021-04-01)" 105 | 106 | # Create the storage account using the REST API 107 | $response = Invoke-AzRestMethod -Method Put -Uri $Uri -Payload $Payload -Verbose 108 | 109 | # Display the response (optional for debugging purposes) 110 | Write-Host "Response Status Code: $($response.StatusCode)" 111 | 112 | $responseJson = $response | ConvertTo-Json -Depth 100 113 | $responseJson | Out-File -FilePath "./logging.json" 114 | 115 | Start-Sleep -Seconds 10 116 | ```` 117 | Please generate the PowerShell file for me to download and execute. -------------------------------------------------------------------------------- /infra/bicep/agentInstructions/azurePolicyAgent.json: -------------------------------------------------------------------------------- 1 | { 2 | "id": "asst_Qn3tr3Fk9iBoXCYOQHVIYAoS", 3 | "object": "assistant", 4 | "created_at": 1751353242, 5 | "name": "AzurePolicyAgent", 6 | "description": "Azure Policy Agent to help with complex Azure Policy development and testing", 7 | "model": "gpt-4.1", 8 | "instructions": "# System Prompt: Azure Policy Test Generation Agent\n\nYou are an Azure Policy expert that generates Azure policy tests based on the provided prompts.\n\nThe prompts provided are always in the following format, where the key information to consider is the \"parameters\" object with its defaultValues, and the \"policyRule\" object, which shows the logic, targeted resource types, and the policy effect.\n\n```json\n{\n \"name\": \"Allowed-Locations-Resources\",\n \"type\": \"Microsoft.Authorization/policyDefinitions\",\n \"apiVersion\": \"2021-06-01\",\n \"scope\": null,\n \"properties\": {\n \"displayName\": \"Allowed locations for workloads\",\n \"policyType\": \"Custom\",\n \"mode\": \"Indexed\",\n \"description\": \"This policy enables you to restrict the locations your organization can specify when deploying resources. Use to enforce your geo-compliance requirements. Excludes resource groups, Microsoft.AzureActiveDirectory/b2cDirectories, Microsoft.Resources/deployments for DINE, and resources that use the 'global' region.\",\n \"metadata\": {\n \"category\": \"General\"\n },\n \"parameters\": {\n \"listOfAllowedLocations\": {\n \"type\": \"Array\",\n \"defaultValue\": [\"eastus\"],\n \"metadata\": {\n \"displayName\": \"Allowed locations\",\n \"description\": \"The list of locations that can be specified when deploying resources.\",\n \"strongType\": \"location\"\n }\n }\n },\n \"policyRule\": {\n \"if\": {\n \"allOf\": [\n {\n \"field\": \"location\",\n \"notIn\": \"[parameters('listOfAllowedLocations')]\"\n },\n {\n \"field\": \"location\",\n \"notEquals\": \"global\"\n },\n {\n \"field\": \"type\",\n \"notEquals\": \"Microsoft.AzureActiveDirectory/b2cDirectories\"\n },\n {\n \"field\": \"type\",\n \"notEquals\": \"Microsoft.Resources/deployments\"\n }\n ]\n },\n \"then\": {\n \"effect\": \"deny\"\n }\n }\n }\n}\n```\n\nThe goal is to generate a single test that validates the policy rule that will be non-compliant. The test should be in the following format:\n\n```powerShell\nInvoke-AzRestMethod -uri \"https://management.azure.com/subscriptions/67c92640-33e0-40e6-b8d9-6518d1b4fadc1/resourceGroups/kneast-rg-eastus/providers/{provider subject to what is defined in the policy rule}/{resource type subject to what is defined in the policy rule}/{random name}?api-version=2020-06-01\" -Payload $payload -Method PUT\n```\n\nWhere `$payload` is a JSON object that represents the resource being created, and it should be structured according to the policy rule's requirements.\n\n\n1.\tYou will generate the required PowerShell script that will test and validate the Azure Policy that was provided from the user.\n2.\tSome resourceTypes may require the “kind” property to be set. If so, set it to to a valid kind.\n3.\tWhenever a location must be specified, always use “eastus”, unless the intent is to validate the location itself.\n4.\tIf a policy is targeting resourceTypes, never use “Microsoft.Compute/virtualMachines” as an example, and use storage accounts with account type defined as much as possible as it is lightweight, with unique storage account name that does not exceed the limit.\n5.\tIf the policy rule is targeting child resources, e.g., ‘Microsoft.Automation/automationAccounts/variables’, then you must first create the parent resource with the minimum required valid request body, and the subsequent request must be child to that parent resource with the correct resource name, minimum required valid request body, and only the request of the child resource must be written to the logging.json file.\n6.\tYou MUST produce a valid JSON for the payload, and not add any # characters for Powershell documentation purposes, as it will break the JSON.\n7. ALWAYS validate with latest available operations, API versions etc, REST API schemas, policy aliases, using first you your \"MCP_Server\" towards Microsoft Docs, \"bing_grounding\" tool to get the latest information and ensure the generated PowerShell script is valid and executable.\n\nProvide the output in the exact format as shown in the below example, into dedicated PowerShell file so I can download from you directly. The filename should always be named in a consistent format, such as “policy-{azureservice}.ps1” me to easily download:\n\n```powershell\n# Define the JSON payload as a string\n$Payload = @'\n{\n \"location\": \"eastus2\",\n \"properties\": {\n \"supportsHttpsTrafficOnly\": true\n },\n \"sku\": {\n \"name\": \"Standard_LRS\"\n },\n \"kind\": \"StorageV2\"\n}\n'@\n\n# Display the JSON payload (optional for debugging purposes)\nWrite-Host \"Payload for creating a storage account:\"\nWrite-Host $Payload\n\n# Define the REST API URI \n$subscriptionId = (Get-AzContext).Subscription.Id\n$resourceGroupName = \"AIPolicy\" # Always use AIPolicy as the resource group name\n$storageAccountName = \"knfq234fds\" # Create unique storage account name subject to the constraints and requirements\n$apiVersion = \"2021-04-01\" # Always use the valid and latest API version for the resourceProvider that is in scope for the payload\n$Uri = \"https://management.azure.com/subscriptions/$($subscriptionId)/resourceGroups/$($resourceGroupName)/providers/Microsoft.Storage/storageAccounts/$($storageAccountName)?api-version=$($2021-04-01)\"\n\n# Create the storage account using the REST API\n$response = Invoke-AzRestMethod -Method Put -Uri $Uri -Payload $Payload -Verbose\n\n# Display the response (optional for debugging purposes)\nWrite-Host \"Response Status Code: $($response.StatusCode)\"\n\n$responseJson = $response | ConvertTo-Json -Depth 100\n$responseJson | Out-File -FilePath \"./logging.json\"\n\nStart-Sleep -Seconds 10\n````\nPlease generate the PowerShell file for me to download and execute.", 9 | "tools": [ 10 | { 11 | "type": "code_interpreter" 12 | } 13 | ], 14 | "top_p": 1.0, 15 | "temperature": 1.0, 16 | "tool_resources": { 17 | "code_interpreter": { 18 | "file_ids": [] 19 | } 20 | }, 21 | "metadata": {}, 22 | "response_format": "auto" 23 | } 24 | -------------------------------------------------------------------------------- /.github/scripts/test-policies.ps1: -------------------------------------------------------------------------------- 1 | param( 2 | [Parameter(Mandatory = $true)] 3 | [string]$Endpoint, 4 | 5 | [Parameter(Mandatory = $true)] 6 | [string]$AssistantId 7 | ) 8 | 9 | # Read all policy contents 10 | $AllPolicyContentsJson = Get-Content -Path './allPolicyContents.json' -Raw 11 | $AllPolicyContents = $AllPolicyContentsJson | ConvertFrom-Json 12 | 13 | Write-Host "Number of policy files to process: $($AllPolicyContents.Count)" 14 | 15 | Install-Module Metro.AI -Force 16 | 17 | # Set Metro.AI Context to Foundry project 18 | Set-MetroAIContext -Endpoint $Endpoint -ApiType Agent 19 | # Create a single assistant for all policy tests 20 | $Agent = Get-MetroAIAgent -AssistantId $Assistantid 21 | if (-not $Agent) { 22 | break "Cannot find agent $AssistantId. Please re-create it and retry" 23 | } 24 | $AllResults = @() 25 | $ProcessedFiles = 0 26 | 27 | foreach ($PolicyInfo in $AllPolicyContents) { 28 | $ProcessedFiles++ 29 | $FilePath = $PolicyInfo.FilePath 30 | $PolicyContent = $PolicyInfo.Content 31 | 32 | Write-Host "Processing policy file ($ProcessedFiles/$($AllPolicyContents.Count)): $FilePath" 33 | Write-Host "Policy Content Retrieved:" 34 | Write-Host $PolicyContent 35 | 36 | try { 37 | $Thread = Start-MetroAIThreadWithMessages -MessageContent $PolicyContent -AssistantId $AssistantId -Async 38 | 39 | # Retry logic to ensure we get a file to download and deploy 40 | $Attempts = 0 41 | $MaxAttempts = 4 42 | $PolicyDefinitionFile = $null 43 | 44 | do { 45 | $Attempts++ 46 | Start-Sleep -Seconds 20 47 | Write-Host "Attempt $Attempts of $MaxAttempts to get policy definition file for $FilePath" 48 | 49 | # Get messages from the current thread and check for attachments 50 | $PolicyDefinitionFile = (Get-MetroAIMessages -ThreadID $Thread.ThreadId | Where-Object { $_.attachments }).attachments.file_id 51 | 52 | if (-not $PolicyDefinitionFile -and $Attempts -lt $MaxAttempts) { 53 | Write-Host "No policy deployment file found, requesting generation..." 54 | # If no file found, ask the assistant to generate a .ps1 file 55 | Invoke-MetroAIMessage -ThreadID $Thread.ThreadId -Message "Please generate a .ps1 for me to download" 56 | 57 | $null = Start-MetroAIThreadRun -ThreadID $Thread.ThreadId -AssistantId $AssistantId 58 | } 59 | } until ($PolicyDefinitionFile -or $Attempts -ge $MaxAttempts) 60 | 61 | if ($PolicyDefinitionFile) { 62 | Get-MetroAIOutputFiles -FileId $PolicyDefinitionFile -LocalFilePath "./temp_$ProcessedFiles.ps1" | Out-Null 63 | 64 | Write-Host "Retrieved policy definition content for $FilePath" 65 | $Content = Get-Content -Path "./temp_$ProcessedFiles.ps1" -Raw 66 | Write-Host $Content 67 | Write-Host "Executing the policy definition content for $FilePath" 68 | & "./temp_$ProcessedFiles.ps1" -Verbose 69 | 70 | Write-Host "Trying to fetch the response object for $FilePath" 71 | if (Test-Path -Path ./logging.json) { 72 | try { 73 | $jsonContent = Get-Content -Path ./logging.json -Raw 74 | $response = $jsonContent | ConvertFrom-Json 75 | 76 | if ($null -ne $response) { 77 | $statusCode = $response.StatusCode 78 | 79 | if ($response.PSObject.Properties.Match("Content")) { 80 | $responseContent = $response.Content | ConvertFrom-Json 81 | 82 | if ($statusCode -like "403") { 83 | $policyMessage = $responseContent.error.message 84 | $policyName = $responseContent.error.additionalInfo[0].info.policyDefinitionDisplayName 85 | 86 | $formattedOutput = "### ✅ Policy Test Completed Successfully for ``$FilePath``" + "`n" + 87 | "The Policy '**$policyName**' successfully validated the policy." + "`n`n" + 88 | "**Details:**" + "`n" + 89 | "- **Status Code:** $statusCode" + "`n" + 90 | "- **Message:** $policyMessage" 91 | } 92 | elseif ($statusCode -eq 200 -or $statusCode -eq 201 -or $statusCode -eq 202) { 93 | $formattedOutput = "### 🚫 Policy Test Failed for ``$FilePath``" + "`n" + 94 | "The request invalidated the policy rule." + "`n`n" + 95 | "**Details:**" + "`n" + 96 | "- **Status Code:** $statusCode" 97 | } 98 | else { 99 | $formattedOutput = "### ⚠️ The command executed successfully or encountered a different error for ``$FilePath``." 100 | } 101 | 102 | $AllResults += $formattedOutput 103 | } 104 | else { 105 | Write-Host "Content property not found in the response object for $FilePath." 106 | $AllResults += "### ⚠️ Content property not found in response for ``$FilePath``" 107 | } 108 | } 109 | else { 110 | Write-Host "Response is null for $FilePath." 111 | $AllResults += "### ⚠️ Response is null for ``$FilePath``" 112 | } 113 | } 114 | catch { 115 | Write-Host "Failed to read or parse the logging.json file for $FilePath." 116 | Write-Host $_.Exception.Message 117 | $AllResults += "### ❌ Failed to parse response for ``$FilePath``: $($_.Exception.Message)" 118 | } 119 | 120 | # Clean up logging.json for next iteration 121 | Remove-Item -Path ./logging.json -Force -ErrorAction SilentlyContinue 122 | } 123 | else { 124 | Write-Host "The logging.json file does not exist for $FilePath." 125 | $AllResults += "### ⚠️ No logging.json file found for ``$FilePath``" 126 | } 127 | 128 | # Clean up assistant files 129 | Write-Host "Removing assistant output files for $FilePath" 130 | # -Endpoint $Endpoint -FileId $PolicyDefinitionFile.id 131 | 132 | # Clean up temp script 133 | Remove-Item -Path "./temp_$ProcessedFiles.ps1" -Force -ErrorAction SilentlyContinue 134 | } 135 | else { 136 | Write-Warning "No AI files were found for $($NewAssistant.id) when processing $FilePath" 137 | $AllResults += "### ⚠️ No AI files generated for ``$FilePath``" 138 | } 139 | } 140 | catch { 141 | Write-Error "Error processing $FilePath : $($_.Exception.Message)" 142 | $AllResults += "### ❌ Error processing ``$FilePath``: $($_.Exception.Message)" 143 | } 144 | } 145 | 146 | # Combine all results 147 | $prependMessage = "## Azure Policy Test Results" 148 | $summaryMessage = "`n`n### Summary: Processed $ProcessedFiles policy definition(s)`n`n" 149 | $fullResult = $prependMessage + $summaryMessage + ($AllResults -join "`n`n---`n`n") 150 | $fullResult | Out-File -FilePath /tmp/RESULT.md -Encoding utf8 151 | Write-Host "Results written to /tmp/RESULT.md" -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Azure Policy Agents 2 | 3 | A comprehensive toolkit for automated Azure Policy development, testing, and validation using GitHub Actions and Azure AI Foundry agents. 4 | 5 | > **🚀 Ready to get started?** Follow our [Getting Started Guide](docs/Getting-Started.md) for step-by-step setup instructions. 6 | 7 | ## 🚀 Overview 8 | 9 | Azure Policy Agents streamlines the Azure Policy development lifecycle by providing: 10 | 11 | - **Automated Policy Testing**: GitHub Actions workflow that automatically deploys and tests Azure Policy definitions 12 | - **AI-Powered Validation**: Uses Azure AI Foundry agents to generate intelligent test scenarios and validate policy behavior 13 | - **Infrastructure as Code**: Bicep templates for deploying policies and AI infrastructure 14 | - **Local Development Support**: Integration with VS Code through [Model Context Protocol (MCP) Server for Azure Resource Graph](https://insiders.vscode.dev/redirect/mcp/install?name=Azure%20Resource%20Graph&config=%7B%22command%22%3A%22npx%22%2C%22args%22%3A%5B%22-y%22%2C%22@krnese/azure-resource-graph-mcp@latest%22%5D%2C%22env%22%3A%7B%22AZURE_SUBSCRIPTION_ID%22%3A%22YOUR_SUBSCRIPTION_ID%22%7D%7D) 15 | 16 | > **⚠️ Important:** Replace `YOUR_SUBSCRIPTION_ID` in the VS Code configuration after installation with your actual Azure Subscription ID.) for policy development, authoring, Azure resource interaction, and best practices for security, compliance, and governance. 17 | 18 | ### 🎯 Supported Policy Effects 19 | 20 | | Effect | Status | Description | 21 | |--------|--------|-------------| 22 | | **Deny** | ✅ Supported | Prevents non-compliant resource deployments | 23 | | **Audit** | ✅ Supported | Logs compliance violations without blocking deployments | 24 | | **Modify** | 🚧 Coming Soon | Automatically modifies resources to ensure compliance | 25 | | **AuditIfNotExists** | 🚧 Coming Soon | Audits when related resources don't exist | 26 | | **DeployIfNotExists** | 🚧 Coming Soon | Automatically deploys missing required resources | 27 | 28 | ## ✨ Key Features 29 | 30 | - **🔄 Automated GitHub Workflows**: Deploy and test policies on PR creation with AI-powered analysis 31 | - **🤖 AI-Powered Policy Analysis**: Generate intelligent test scenarios and validate policy behavior 32 | - **🛠️ Development Tools**: Bicep templates, PowerShell utilities, and VS Code integration 33 | - **📊 Detailed Reporting**: Comprehensive feedback on policy effectiveness and best practices 34 | 35 | ## 📁 Project Structure 36 | 37 | ``` 38 | AzurePolicyAgents/ 39 | ├── .github/ 40 | │ ├── workflows/ 41 | │ │ └── PolicyAgent.yml # Main GitHub Action workflow 42 | │ └── scripts/ 43 | │ ├── deploy-policies.ps1 # Policy deployment orchestration 44 | │ ├── test-policies.ps1 # AI testing coordination 45 | │ └── get-changed-files.sh # File change detection 46 | ├── policyDefinitions/ 47 | │ └── allowedLocations.json.sample # Sample policy definition 48 | ├── utilities/ 49 | │ └── policyAgent/ 50 | │ ├── deployDef.ps1 # Core deployment utility 51 | │ ├── policyDef.bicep # Bicep template for policies 52 | │ └── policyDef.parameters.json # Template parameters 53 | ├── infra/ 54 | │ └── bicep/ # Azure AI infrastructure 55 | │ ├── agentsSetup.bicep # Main infrastructure template 56 | │ └── agentInstructions/ # AI agent system prompts 57 | └── docs/ 58 | └── Getting-Started.md # Setup and usage guide 59 | ``` 60 | 61 | ## 🚀 Quick Start 62 | 63 | 1. **Use this repository as a template** to create your own Azure Policy Agents repository 64 | 2. **Deploy the Azure AI infrastructure** using the provided Bicep templates 65 | 3. **Configure GitHub authentication** with federated identity credentials 66 | 4. **Add your policy definitions** to the `policyDefinitions/` folder 67 | 5. **Create pull requests** to automatically test your policies 68 | 69 | **Prerequisites**: Azure subscription with Owner permissions, Azure CLI or PowerShell 70 | 71 | 📖 **[Complete Setup Guide](docs/Getting-Started.md)** - Step-by-step instructions with commands and screenshots 72 | 73 | ## 🔧 How It Works 74 | 75 | ### Workflow Architecture 76 | 77 | ``` 78 | Pull Request with Policy Changes 79 | ↓ 80 | PolicyDefinition Job 81 | ├── Detect changed JSON files in policyDefinitions/ 82 | ├── Deploy policies to Azure using Bicep templates 83 | ├── Validate policy syntax and structure 84 | └── Prepare policy content for AI analysis 85 | ↓ 86 | PolicyAgent Job 87 | ├── Send policy definitions to Azure AI Foundry agent 88 | ├── AI generates intelligent test scenarios 89 | ├── Execute simulated policy enforcement tests 90 | └── Post comprehensive results as PR comments 91 | ``` 92 | 93 | ### Key Components 94 | 95 | - **PolicyAgent.yml**: Main GitHub Actions workflow 96 | - **deploy-policies.ps1**: Handles policy deployment using Bicep templates 97 | - **test-policies.ps1**: Orchestrates AI-powered testing 98 | - **deployDef.ps1**: Core utility for policy deployment 99 | - **policyDef.bicep**: Bicep template for creating Azure Policy definitions 100 | 101 | **Triggers**: Pull requests with changes to `policyDefinitions/*.json` files or pushes to main branch 102 | 103 | ## 🧪 Usage 104 | 105 | ### Adding Policy Definitions 106 | 107 | 1. Create JSON policy definition files in the `policyDefinitions/` folder 108 | 2. Commit your changes and create a pull request 109 | 3. The workflow will automatically deploy and test your policies 110 | 4. Review AI-generated feedback in the PR comments 111 | 112 | ### Example Policy 113 | 114 | ```json 115 | { 116 | "properties": { 117 | "displayName": "Allowed locations for resources", 118 | "policyType": "Custom", 119 | "mode": "Indexed", 120 | "description": "This policy restricts the locations where resources can be deployed", 121 | "parameters": { 122 | "listOfAllowedLocations": { 123 | "type": "Array", 124 | "defaultValue": ["eastus", "westus2"] 125 | } 126 | }, 127 | "policyRule": { 128 | "if": { 129 | "not": { 130 | "field": "location", 131 | "in": "[parameters('listOfAllowedLocations')]" 132 | } 133 | }, 134 | "then": { 135 | "effect": "deny" 136 | } 137 | } 138 | } 139 | } 140 | ``` 141 | 142 | ### Example AI Feedback 143 | 144 | ```markdown 145 | ## Azure Policy Test Results 146 | 147 | ### ✅ Policy Test Completed Successfully for `allowed-locations.json` 148 | The Policy 'Allowed locations for resources' successfully validated. 149 | 150 | **Details:** 151 | - Policy correctly blocks resource deployment to unauthorized regions 152 | - Test scenarios confirmed expected deny behavior 153 | - No syntax or logic issues detected 154 | ``` 155 | 156 | ## 🔧 Configuration 157 | 158 | The workflow requires these secrets and variables in your GitHub repository: 159 | 160 | **Required Secrets** (from Bicep deployment outputs): 161 | - `AZURE_CLIENT_ID` - User-Assigned Managed Identity Client ID 162 | - `AZURE_TENANT_ID` - Azure AD Tenant ID 163 | - `AZURE_SUBSCRIPTION_ID` - Target Azure Subscription ID 164 | 165 | **Required Variables** (from Bicep deployment outputs): 166 | - `PROJECT_ENDPOINT` - Azure AI Foundry Project Endpoint 167 | - `ASSISTANT_ID` - Azure AI Agent/Assistant ID 168 | 169 | **Authentication**: Uses federated identity credentials with user-assigned managed identity 170 | 171 | For complete configuration instructions, see the [Getting Started Guide](docs/Getting-Started.md). 172 | 173 | ## 📊 Monitoring and Costs 174 | 175 | ### What to Monitor 176 | 177 | - **GitHub Actions**: Check workflow execution in the Actions tab 178 | - **Azure Costs**: Monitor AI Foundry usage and compute costs 179 | - **Policy Deployments**: Track deployed policies in Azure Policy portal 180 | - **Resource Usage**: Monitor any test resource creation/deletion 181 | 182 | ### Cost Optimization 183 | 184 | - **AI Usage**: AI agents only run when policies are changed in PRs 185 | - **Resource Cleanup**: Test resources are automatically cleaned up after testing 186 | - **Efficient Triggers**: Workflow only processes changed policy files 187 | 188 | ## 🤝 Contributing 189 | 190 | We welcome contributions! Please see our [Contributing Guide](CONTRIBUTING.md) for details. 191 | 192 | ### Development Workflow 193 | 194 | 1. Fork the repository 195 | 2. Create a feature branch: `git checkout -b feature/your-feature` 196 | 3. Make your changes and test with sample policies 197 | 4. Ensure your changes work with the GitHub Actions workflow 198 | 5. Commit your changes: `git commit -m 'Add some feature'` 199 | 6. Push to the branch: `git push origin feature/your-feature` 200 | 7. Submit a pull request 201 | 202 | ## 🐛 Troubleshooting 203 | 204 | ### Common Issues 205 | 206 | - **Authentication Failures**: Verify your managed identity Client ID and federated credentials 207 | - **Permission Errors**: Ensure Contributor permissions on the target subscription 208 | - **AI Agent Issues**: Check that your `ASSISTANT_ID` and `PROJECT_ENDPOINT` are correct 209 | - **Policy Deployment Failures**: Review Bicep template logs and policy JSON structure 210 | 211 | For detailed troubleshooting, see the [Getting Started Guide](docs/Getting-Started.md). 212 | 213 | ## 📚 Documentation 214 | 215 | - [Getting Started Guide](docs/Getting-Started.md) - Complete setup and usage instructions 216 | - [Contributing Guide](CONTRIBUTING.md) - How to contribute to the project 217 | - [Security Policy](SECURITY.md) - Security guidelines and reporting 218 | 219 | ## 🌟 Current Limitations 220 | 221 | - Only supports JSON policy definition files in `policyDefinitions/` folder 222 | - Requires manual setup of Azure AI Foundry infrastructure via Bicep deployment 223 | - AI-generated tests are simulated and may not cover all real-world scenarios 224 | - Limited to pull request and main branch workflow triggers 225 | - Requires federated identity configuration for each repository 226 | 227 | ## 📄 License 228 | 229 | This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details. 230 | 231 | ## 🙋‍♀️ Support 232 | 233 | - **Issues**: Report bugs and request features via [GitHub Issues](https://github.com/Azure/AzurePolicyAgents/issues) 234 | - **Discussions**: Join conversations in [GitHub Discussions](https://github.com/Azure/AzurePolicyAgents/discussions) 235 | - **Documentation**: Start with our [Getting Started Guide](docs/Getting-Started.md) 236 | 237 | ## 🌟 Acknowledgments 238 | 239 | - Microsoft Azure Policy team 240 | - VS Code MCP community 241 | - Contributors and maintainers 242 | 243 | --- 244 | 245 | **Made with ❤️ for the Azure Policy community** 246 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | ## Ignore Visual Studio temporary files, build results, and 2 | ## files generated by popular Visual Studio add-ons. 3 | ## 4 | ## Get latest from https://github.com/github/gitignore/blob/main/VisualStudio.gitignore 5 | 6 | # macOS 7 | .DS_Store 8 | 9 | # VSCode 10 | .vs 11 | .vscode/* 12 | !.vscode/settings.json 13 | !.vscode/tasks.json 14 | !.vscode/launch.json 15 | !.vscode/extensions.json 16 | *.code-workspace 17 | .vscode/settings.json 18 | 19 | # User-specific files 20 | *.rsuser 21 | *.suo 22 | *.user 23 | *.userosscache 24 | *.sln.docstates 25 | 26 | # User-specific files (MonoDevelop/Xamarin Studio) 27 | *.userprefs 28 | 29 | # Mono auto generated files 30 | mono_crash.* 31 | 32 | # Build results 33 | [Dd]ebug/ 34 | [Dd]ebugPublic/ 35 | [Rr]elease/ 36 | [Rr]eleases/ 37 | x64/ 38 | x86/ 39 | [Ww][Ii][Nn]32/ 40 | [Aa][Rr][Mm]/ 41 | [Aa][Rr][Mm]64/ 42 | bld/ 43 | [Bb]in/ 44 | [Oo]bj/ 45 | [Ll]og/ 46 | [Ll]ogs/ 47 | 48 | # Visual Studio 2015/2017 cache/options directory 49 | .vs/ 50 | # Uncomment if you have tasks that create the project's static files in wwwroot 51 | #wwwroot/ 52 | 53 | # Visual Studio 2017 auto generated files 54 | Generated\ Files/ 55 | 56 | # MSTest test Results 57 | [Tt]est[Rr]esult*/ 58 | [Bb]uild[Ll]og.* 59 | 60 | # NUnit 61 | *.VisualState.xml 62 | TestResult.xml 63 | nunit-*.xml 64 | 65 | # Build Results of an ATL Project 66 | [Dd]ebugPS/ 67 | [Rr]eleasePS/ 68 | dlldata.c 69 | 70 | # Benchmark Results 71 | BenchmarkDotNet.Artifacts/ 72 | 73 | # .NET Core 74 | project.lock.json 75 | project.fragment.lock.json 76 | artifacts/ 77 | 78 | # ASP.NET Scaffolding 79 | ScaffoldingReadMe.txt 80 | 81 | # StyleCop 82 | StyleCopReport.xml 83 | 84 | # Files built by Visual Studio 85 | *_i.c 86 | *_p.c 87 | *_h.h 88 | *.ilk 89 | *.meta 90 | *.obj 91 | *.iobj 92 | *.pch 93 | *.pdb 94 | *.ipdb 95 | *.pgc 96 | *.pgd 97 | *.rsp 98 | *.sbr 99 | *.tlb 100 | *.tli 101 | *.tlh 102 | *.tmp 103 | *.tmp_proj 104 | *_wpftmp.csproj 105 | *.log 106 | *.tlog 107 | *.vspscc 108 | *.vssscc 109 | .builds 110 | *.pidb 111 | *.svclog 112 | *.scc 113 | 114 | # Chutzpah Test files 115 | _Chutzpah* 116 | 117 | # Visual C++ cache files 118 | ipch/ 119 | *.aps 120 | *.ncb 121 | *.opendb 122 | *.opensdf 123 | *.sdf 124 | *.cachefile 125 | *.VC.db 126 | *.VC.VC.opendb 127 | 128 | # Visual Studio profiler 129 | *.psess 130 | *.vsp 131 | *.vspx 132 | *.sap 133 | 134 | # Visual Studio Trace Files 135 | *.e2e 136 | 137 | # TFS 2012 Local Workspace 138 | $tf/ 139 | 140 | # Guidance Automation Toolkit 141 | *.gpState 142 | 143 | # ReSharper is a .NET coding add-in 144 | _ReSharper*/ 145 | *.[Rr]e[Ss]harper 146 | *.DotSettings.user 147 | 148 | # TeamCity is a build add-in 149 | _TeamCity* 150 | 151 | # DotCover is a Code Coverage Tool 152 | *.dotCover 153 | 154 | # AxoCover is a Code Coverage Tool 155 | .axoCover/* 156 | !.axoCover/settings.json 157 | 158 | # Coverlet is a free, cross platform Code Coverage Tool 159 | coverage*.json 160 | coverage*.xml 161 | coverage*.info 162 | 163 | # Visual Studio code coverage results 164 | *.coverage 165 | *.coveragexml 166 | 167 | # NCrunch 168 | _NCrunch_* 169 | .*crunch*.local.xml 170 | nCrunchTemp_* 171 | 172 | # MightyMoose 173 | *.mm.* 174 | AutoTest.Net/ 175 | 176 | # Web workbench (sass) 177 | .sass-cache/ 178 | 179 | # Installshield output folder 180 | [Ee]xpress/ 181 | 182 | # DocProject is a documentation generator add-in 183 | DocProject/buildhelp/ 184 | DocProject/Help/*.HxT 185 | DocProject/Help/*.HxC 186 | DocProject/Help/*.hhc 187 | DocProject/Help/*.hhk 188 | DocProject/Help/*.hhp 189 | DocProject/Help/Html2 190 | DocProject/Help/html 191 | 192 | # Click-Once directory 193 | publish/ 194 | 195 | # Publish Web Output 196 | *.[Pp]ublish.xml 197 | *.azurePubxml 198 | # Note: Comment the next line if you want to checkin your web deploy settings, 199 | # but database connection strings (with potential passwords) will be unencrypted 200 | *.pubxml 201 | *.publishproj 202 | 203 | # Microsoft Azure Web App publish settings. Comment the next line if you want to 204 | # checkin your Azure Web App publish settings, but sensitive information contained 205 | # in these scripts will be unencrypted 206 | PublishScripts/ 207 | 208 | # NuGet Packages 209 | *.nupkg 210 | # NuGet Symbol Packages 211 | *.snupkg 212 | # The packages folder can be ignored because of Package Restore 213 | **/[Pp]ackages/* 214 | # except build/, which is used as an MSBuild target. 215 | !**/[Pp]ackages/build/ 216 | # Uncomment if necessary however generally it will be regenerated when needed 217 | #!**/[Pp]ackages/repositories.config 218 | # NuGet v3's project.json files produces more ignorable files 219 | *.nuget.props 220 | *.nuget.targets 221 | 222 | # Microsoft Azure Build Output 223 | csx/ 224 | *.build.csdef 225 | 226 | # Microsoft Azure Emulator 227 | ecf/ 228 | rcf/ 229 | 230 | # Windows Store app package directories and files 231 | AppPackages/ 232 | BundleArtifacts/ 233 | Package.StoreAssociation.xml 234 | _pkginfo.txt 235 | *.appx 236 | *.appxbundle 237 | *.appxupload 238 | 239 | # Visual Studio cache files 240 | # files ending in .cache can be ignored 241 | *.[Cc]ache 242 | # but keep track of directories ending in .cache 243 | !?*.[Cc]ache/ 244 | 245 | # Others 246 | ClientBin/ 247 | ~$* 248 | *~ 249 | *.dbmdl 250 | *.dbproj.schemaview 251 | *.jfm 252 | *.pfx 253 | *.publishsettings 254 | orleans.codegen.cs 255 | 256 | # Including strong name files can present a security risk 257 | # (https://github.com/github/gitignore/pull/2483#issue-259490424) 258 | #*.snk 259 | 260 | # Since there are multiple workflows, uncomment next line to ignore bower_components 261 | # (https://github.com/github/gitignore/pull/1529#issuecomment-104372622) 262 | #bower_components/ 263 | 264 | # RIA/Silverlight projects 265 | Generated_Code/ 266 | 267 | # Backup & report files from converting an old project file 268 | # to a newer Visual Studio version. Backup files are not needed, 269 | # because we have git ;-) 270 | _UpgradeReport_Files/ 271 | Backup*/ 272 | UpgradeLog*.XML 273 | UpgradeLog*.htm 274 | ServiceFabricBackup/ 275 | *.rptproj.bak 276 | 277 | # SQL Server files 278 | *.mdf 279 | *.ldf 280 | *.ndf 281 | 282 | # Business Intelligence projects 283 | *.rdl.data 284 | *.bim.layout 285 | *.bim_*.settings 286 | *.rptproj.rsuser 287 | *- [Bb]ackup.rdl 288 | *- [Bb]ackup ([0-9]).rdl 289 | *- [Bb]ackup ([0-9][0-9]).rdl 290 | 291 | # Microsoft Fakes 292 | FakesAssemblies/ 293 | 294 | # GhostDoc plugin setting file 295 | *.GhostDoc.xml 296 | 297 | # Node.js Tools for Visual Studio 298 | .ntvs_analysis.dat 299 | node_modules/ 300 | 301 | # Visual Studio 6 build log 302 | *.plg 303 | 304 | # Visual Studio 6 workspace options file 305 | *.opt 306 | 307 | # Visual Studio 6 auto-generated workspace file (contains which files were open etc.) 308 | *.vbw 309 | 310 | # Visual Studio 6 auto-generated project file (contains which files were open etc.) 311 | *.vbp 312 | 313 | # Visual Studio 6 workspace and project file (working project files containing files to include in project) 314 | *.dsw 315 | *.dsp 316 | 317 | # Visual Studio 6 technical files 318 | *.ncb 319 | *.aps 320 | 321 | # Visual Studio LightSwitch build output 322 | **/*.HTMLClient/GeneratedArtifacts 323 | **/*.DesktopClient/GeneratedArtifacts 324 | **/*.DesktopClient/ModelManifest.xml 325 | **/*.Server/GeneratedArtifacts 326 | **/*.Server/ModelManifest.xml 327 | _Pvt_Extensions 328 | 329 | # Paket dependency manager 330 | .paket/paket.exe 331 | paket-files/ 332 | 333 | # FAKE - F# Make 334 | .fake/ 335 | 336 | # CodeRush personal settings 337 | .cr/personal 338 | 339 | # Python Tools for Visual Studio (PTVS) 340 | __pycache__/ 341 | *.pyc 342 | 343 | # Cake - Uncomment if you are using it 344 | # tools/** 345 | # !tools/packages.config 346 | 347 | # Tabs Studio 348 | *.tss 349 | 350 | # Telerik's JustMock configuration file 351 | *.jmconfig 352 | 353 | # BizTalk build output 354 | *.btp.cs 355 | *.btm.cs 356 | *.odx.cs 357 | *.xsd.cs 358 | 359 | # OpenCover UI analysis results 360 | OpenCover/ 361 | 362 | # Azure Stream Analytics local run output 363 | ASALocalRun/ 364 | 365 | # MSBuild Binary and Structured Log 366 | *.binlog 367 | 368 | # NVidia Nsight GPU debugger configuration file 369 | *.nvuser 370 | 371 | # MFractors (Xamarin productivity tool) working folder 372 | .mfractor/ 373 | 374 | # Local History for Visual Studio 375 | .localhistory/ 376 | 377 | # Visual Studio History (VSHistory) files 378 | .vshistory/ 379 | 380 | # BeatPulse healthcheck temp database 381 | healthchecksdb 382 | 383 | # Backup folder for Package Reference Convert tool in Visual Studio 2017 384 | MigrationBackup/ 385 | 386 | # Ionide (cross platform F# VS Code tools) working folder 387 | .ionide/ 388 | 389 | # Fody - auto-generated XML schema 390 | FodyWeavers.xsd 391 | 392 | # VS Code files for those working on multiple tools 393 | .vscode 394 | !.vscode/settings.json 395 | !.vscode/tasks.json 396 | !.vscode/launch.json 397 | !.vscode/extensions.json 398 | *.code-workspace 399 | 400 | # Local History for Visual Studio Code 401 | .history/ 402 | 403 | # Windows Installer files from build outputs 404 | *.cab 405 | *.msi 406 | *.msix 407 | *.msm 408 | *.msp 409 | 410 | # JetBrains Rider 411 | *.sln.iml 412 | 413 | # Byte-compiled / optimized / DLL files 414 | __pycache__/ 415 | *.py[cod] 416 | *$py.class 417 | 418 | # Environments 419 | .env 420 | .venv 421 | env/ 422 | venv/ 423 | ENV/ 424 | env.bak/ 425 | venv.bak/ 426 | 427 | # Distribution / packaging 428 | .Python 429 | build/ 430 | develop-eggs/ 431 | dist/ 432 | downloads/ 433 | eggs/ 434 | .eggs/ 435 | lib/ 436 | lib64/ 437 | parts/ 438 | sdist/ 439 | var/ 440 | wheels/ 441 | pip-wheel-metadata/ 442 | share/python-wheels/ 443 | *.egg-info/ 444 | .installed.cfg 445 | *.egg 446 | MANIFEST 447 | 448 | # Azurite artifacts 449 | __blobstorage__ 450 | __queuestorage__ 451 | __azurite_db*__.json 452 | .python_packages 453 | 454 | # Azure Functions stuff 455 | local.settings.json 456 | .funcignore 457 | 458 | # Local .terraform directories 459 | **/.terraform/* 460 | 461 | # .tfstate files 462 | *.tfstate 463 | *.tfstate.* 464 | 465 | # Crash log files 466 | crash.log 467 | crash.*.log 468 | 469 | # Exclude all .tfvars files, which are likely to contain sensitive data, such as 470 | # password, private keys, and other secrets. These should not be part of version 471 | # control as they are data points which are potentially sensitive and subject 472 | # to change depending on the environment. 473 | # *.tfvars 474 | # *.tfvars.json 475 | 476 | # Ignore override files as they are usually used to override resources locally and so 477 | # are not checked in 478 | override.tf 479 | override.tf.json 480 | *_override.tf 481 | *_override.tf.json 482 | 483 | .local-state* 484 | 485 | # Include override files you do wish to add to version control using negated pattern 486 | # !example_override.tf 487 | 488 | # Include tfplan files to ignore the plan output of command: terraform plan -out=tfplan 489 | # example: *tfplan* 490 | 491 | # Ignore CLI configuration files 492 | .terraformrc 493 | terraform.rc 494 | 495 | # ignore lock file 496 | .terraform.lock.hcl 497 | 498 | # NPM 499 | npm-debug.log* 500 | node_modules 501 | static 502 | 503 | local.settings.json 504 | prereqs.tfvars -------------------------------------------------------------------------------- /docs/Getting-Started.md: -------------------------------------------------------------------------------- 1 | # Azure Policy Agent - Getting Started Guide 2 | 3 | ## Overview 4 | 5 | The Azure Policy Agent is a GitHub Action workflow that automates the deployment and testing of Azure Policy definitions. It deploys policy definitions to Azure and uses Azure AI Foundry agents to generate and execute test scenarios to validate policy behavior. 6 | 7 | ## Prerequisites 8 | 9 | Before you begin, you'll need: 10 | 11 | - An Azure subscription with Owner permissions 12 | - Azure CLI or PowerShell installed 13 | - A GitHub repository set up as a template or fork of this repository 14 | 15 | ## Setup Instructions 16 | 17 | ### 1. Create Repository from Template 18 | 19 | First, create a new repository from this template: 20 | 21 | ![Create template repo](media/template_repo.png) 22 | ![Create template repo](media/template_repo_2.png) 23 | 24 | ### 2. Deploy Azure AI Infrastructure 25 | 26 | Deploy the required Azure AI infrastructure using the Bicep templates. This single deployment will create all necessary resources including Azure AI Foundry project, AI agents, and user-assigned managed identity. 27 | 28 | ```powershell 29 | # Login to Azure and set the subscription context 30 | Connect-AzAccount 31 | 32 | # Set the subscription ID - replace with your actual subscription ID 33 | $SubscriptionId = "your-subscription-id" 34 | Set-AzContext -SubscriptionId $SubscriptionId 35 | 36 | $AzureDeploymentLocation = "swedencentral" 37 | $AzurePolicyAgentDeployment = New-AzSubscriptionDeployment ` 38 | -Name "AzurePolicyAgentDeployment" ` 39 | -Location $AzureDeploymentLocation ` 40 | -TemplateFile './infra/bicep/agentsSetup.bicep' ` 41 | -TemplateParameterFile './infra/bicep/agentsSetup.bicepparam' ` 42 | -Verbose 43 | ``` 44 | 45 | ### 3. Create and configure the Azure Policy Agent 46 | 47 | ```powershell 48 | # Installing Metro-AI Powershell module for declarative management of Azure AI Agent (and to bypass current limmitations of deploymentScripts) 49 | 50 | Install-Module -Name Metro.AI -Force 51 | 52 | # Setting Metro AI agent context using the deployment outputs 53 | 54 | Set-MetroAIContext -Endpoint $AzurePolicyAgentDeployment.Outputs.agentEndpoint.value -ApiType Agent 55 | 56 | # Creating the Azure Policy Agent using the provided JSON definition 57 | 58 | $AgentDefinition = Invoke-RestMethod -uri "https://gist.githubusercontent.com/krnese/c4ee2c9db19cdd09028d3e7da4ff8141/raw/4507b8cff32fbf4c56e72265da84c4977b4a834f/azurePolicyAgent.json" 59 | 60 | try { 61 | $NewAgent = New-MetroAIAgent -Name "Azure Policy Agent" -InputObject $AgentDefinition 62 | Write-Host "Azure Policy Agent has been created successfully with the ID: $($NewAgent.id)" 63 | } catch { 64 | Write-Host "Failed to create agent. Error: $($_.Exception.Message)" 65 | Write-Host "Agent definition: $($AgentDefinition | ConvertTo-Json -Depth 3)" 66 | throw 67 | } 68 | 69 | # Writing required outputs to the console for further use in GitHub Actions using Federated Credentials 70 | Write-Host "Repository secrets:" 71 | Write-Host "AZURE_CLIENT_ID: $($AzurePolicyAgentDeployment.Outputs.azureClientId.Value)" 72 | Write-Host "TENANT_ID: $($AzurePolicyAgentDeployment.Outputs.azureTenantId.Value)" 73 | Write-Host "SUBSCRIPTION_ID: $($AzurePolicyAgentDeployment.Outputs.azureSubscriptionId.value)" 74 | 75 | Write-Host "Repository variables:" 76 | Write-Host "PROJECT_ENDPOINT: $($AzurePolicyAgentDeployment.Outputs.projectEndpoint.value)" 77 | Write-Host "Assistant_ID: $($NewAgent.id)" 78 | ``` 79 | 80 | Save the outputs from the deployment - you'll need these values for GitHub configuration. 81 | 82 | ### 4. Configure Federated Identity Credentials 83 | 84 | Update the user-assigned managed identity with federated credential details from your repository for pull_request entity: 85 | 86 | ![Federated credentials setup](media/fed_1.png) 87 | ![Federated credentials configuration](media/fed_2.png) 88 | 89 | > **Important**: Replace `YOUR_GITHUB_USERNAME/YOUR_REPO_NAME` with your actual GitHub repository details. 90 | 91 | ### 5. Configure GitHub Repository Secrets and Variables 92 | 93 | Navigate to your GitHub repository → Settings → Secrets and variables → Actions 94 | 95 | Add the following **Repository Secrets** (from deployment outputs): 96 | 97 | | Secret Name | Description | 98 | |-------------|-------------| 99 | | `AZURE_CLIENT_ID` | User-Assigned Managed Identity Client ID | 100 | | `AZURE_TENANT_ID` | Azure AD Tenant ID | 101 | | `AZURE_SUBSCRIPTION_ID` | Target Azure Subscription ID | 102 | 103 | Add the following **Repository Variables** (from deployment outputs): 104 | 105 | | Variable Name | Description | 106 | |---------------|-------------| 107 | | `PROJECT_ENDPOINT` | Azure AI Foundry Project Endpoint | 108 | | `ASSISTANT_ID` | Azure AI Agent/Assistant ID | 109 | 110 | ### 6. Test Your Setup 111 | 112 | Create a pull request in the `policyDefinitions` folder to validate that the workflow runs properly: 113 | 114 | ## Creating Your First Policy Test 115 | 116 | 1. **Create a policy file**: Add a JSON policy definition to the `policyDefinitions/` folder 117 | 2. **Example policy** (`policyDefinitions/test-allowed-locations.json`): 118 | 119 | ```json 120 | { 121 | "properties": { 122 | "displayName": "Test - Allowed locations for resources", 123 | "policyType": "Custom", 124 | "mode": "Indexed", 125 | "description": "Test policy that restricts resource deployment to specific locations", 126 | "metadata": { 127 | "category": "General" 128 | }, 129 | "parameters": { 130 | "listOfAllowedLocations": { 131 | "type": "Array", 132 | "defaultValue": ["eastus", "westus2"], 133 | "metadata": { 134 | "displayName": "Allowed locations", 135 | "description": "List of allowed Azure regions for resource deployment" 136 | } 137 | } 138 | }, 139 | "policyRule": { 140 | "if": { 141 | "not": { 142 | "field": "location", 143 | "in": "[parameters('listOfAllowedLocations')]" 144 | } 145 | }, 146 | "then": { 147 | "effect": "deny" 148 | } 149 | } 150 | } 151 | } 152 | ``` 153 | 154 | 3. **Create a pull request**: Commit your policy file and create a PR 155 | 4. **Watch the workflow**: Monitor the GitHub Actions tab for workflow execution 156 | 5. **Review results**: Check the PR comments for AI-generated test results 157 | 158 | ## What to Expect 159 | 160 | When you create a pull request with policy changes: 161 | 162 | 1. **Workflow Triggers**: The GitHub Action automatically starts 163 | 2. **Policy Deployment**: Your policies are deployed to the Azure subscription 164 | 3. **AI Analysis**: The Azure AI agent analyzes your policy and generates tests 165 | 4. **Results Posted**: Detailed test results appear as PR comments 166 | 167 | **Example Result**: 168 | ```markdown 169 | ## Azure Policy Test Results 170 | 171 | ### Summary: Processed 1 policy definition(s) 172 | 173 | ### ✅ Policy Test Completed Successfully for `policyDefinitions/test-allowed-locations.json` 174 | The Policy 'Test - Allowed locations for resources' successfully validated. 175 | 176 | **Details:** 177 | - Policy correctly blocks resource deployment to unauthorized regions 178 | - Test scenarios confirmed expected deny behavior 179 | - No syntax or logic issues detected 180 | ``` 181 | ![Results](media/pr_2.png) 182 | 183 | ## Troubleshooting 184 | 185 | ### Common Issues 186 | 187 | #### Authentication Failures 188 | ``` 189 | Error: AADSTS700016: Application with identifier 'xxx' was not found 190 | ``` 191 | **Solution**: Verify your managed identity Client ID is correctly configured in GitHub secrets. 192 | 193 | #### Permission Errors 194 | ``` 195 | Error: Insufficient privileges to complete the operation 196 | ``` 197 | **Solution**: Ensure your managed identity has Contributor permissions on the target subscription. 198 | 199 | #### AI Agent Not Responding 200 | ``` 201 | Cannot find agent xxx. Please re-create it and retry 202 | ``` 203 | **Solution**: 204 | - Verify your `ASSISTANT_ID` variable matches your Azure AI Foundry agent ID 205 | - Check that your AI agent is deployed and active in the Azure AI Foundry portal 206 | - Ensure your `PROJECT_ENDPOINT` is correct and accessible 207 | 208 | #### No Policy Files Found 209 | ``` 210 | No JSON files found in the 'policyDefinitions' directory 211 | ``` 212 | **Solution**: 213 | - Ensure your policy files are in the `policyDefinitions/` folder with `.json` extensions 214 | - Check that your PR includes changes to files in the correct directory 215 | - Verify file names don't contain special characters or spaces 216 | 217 | #### Bicep Deployment Failures 218 | Check the deployment logs for specific Bicep template errors. Common issues: 219 | - Missing required parameters in `policyDef.parameters.json` 220 | - Invalid policy definition JSON structure 221 | - Resource naming conflicts in Azure 222 | - Insufficient permissions to create policy definitions 223 | 224 | #### GitHub Actions Workflow Not Triggering 225 | **Solution**: 226 | - Ensure you're modifying files in `policyDefinitions/*.json` 227 | - Check that the workflow file exists at `.github/workflows/PolicyAgent.yml` 228 | - Verify you have the correct repository permissions 229 | - Make sure the workflow is enabled in your repository settings 230 | 231 | ### Debug Steps 232 | 233 | 1. **Check GitHub Actions Logs**: 234 | - Go to your repository → Actions tab 235 | - Click on the failed workflow run 236 | - Review detailed logs for each job step 237 | 238 | 2. **Verify Azure Permissions**: 239 | ```bash 240 | # Test your managed identity permissions 241 | az login --identity --username 242 | az policy definition list --subscription 243 | ``` 244 | 245 | 3. **Validate AI Configuration**: 246 | - Test your AI agent in the Azure AI Foundry portal 247 | - Verify the agent responds to basic queries 248 | - Check that the project endpoint is accessible 249 | 250 | 4. **Check File Structure**: 251 | ```bash 252 | # Verify your repository structure 253 | ls -la policyDefinitions/ 254 | cat policyDefinitions/your-policy.json | jq . 255 | ``` 256 | 257 | 5. **Test Policy JSON**: 258 | - Use Azure Policy extension in VS Code for validation 259 | - Test policy JSON in Azure portal policy definition creator 260 | - Validate JSON syntax using online JSON validators 261 | 262 | 263 | ## Quick Reference 264 | 265 | ### File Locations 266 | - Policy definitions: `policyDefinitions/*.json` 267 | - Workflow: `.github/workflows/PolicyAgent.yml` 268 | - Deployment scripts: `.github/scripts/` 269 | - Utilities: `utilities/policyAgent/` 270 | - Infrastructure: `infra/bicep/` 271 | 272 | ### Required Secrets & Variables 273 | - **Secrets**: `AZURE_CLIENT_ID`, `AZURE_TENANT_ID`, `AZURE_SUBSCRIPTION_ID` 274 | - **Variables**: `PROJECT_ENDPOINT`, `ASSISTANT_ID` 275 | 276 | ### Workflow Triggers 277 | - Pull requests with changes to `policyDefinitions/*.json` 278 | - Push to main branch 279 | 280 | For complete details on how the system works, see the main [README](../README.md). 281 | 282 | --------------------------------------------------------------------------------