├── .gitignore ├── Ingress+Cloud ├── backend-echo.yml ├── deploy-ingress.cmd ├── gateway-apis.png ├── gateway-config.png ├── ingress-deployment-cloud.yml ├── ingress-deployment.yml ├── ingress-tls.yml ├── ingress.yml └── readme.md ├── Ingress-only ├── backend-echo.yml ├── deploy-ingress.cmd ├── ingress-deployment.yml ├── ingress-tls.yml ├── ingress.yml └── readme.md ├── LICENSE ├── README.md └── SECURITY.md /.gitignore: -------------------------------------------------------------------------------- 1 | ## Ignore Visual Studio temporary files, build results, and 2 | ## files generated by popular Visual Studio add-ons. 3 | ## 4 | ## Get latest from https://github.com/github/gitignore/blob/master/VisualStudio.gitignore 5 | 6 | # User-specific files 7 | *.rsuser 8 | *.suo 9 | *.user 10 | *.userosscache 11 | *.sln.docstates 12 | 13 | # User-specific files (MonoDevelop/Xamarin Studio) 14 | *.userprefs 15 | 16 | # Mono auto generated files 17 | mono_crash.* 18 | 19 | # Build results 20 | [Dd]ebug/ 21 | [Dd]ebugPublic/ 22 | [Rr]elease/ 23 | [Rr]eleases/ 24 | x64/ 25 | x86/ 26 | [Aa][Rr][Mm]/ 27 | [Aa][Rr][Mm]64/ 28 | bld/ 29 | [Bb]in/ 30 | [Oo]bj/ 31 | [Ll]og/ 32 | [Ll]ogs/ 33 | 34 | # Visual Studio 2015/2017 cache/options directory 35 | .vs/ 36 | # Uncomment if you have tasks that create the project's static files in wwwroot 37 | #wwwroot/ 38 | 39 | # Visual Studio 2017 auto generated files 40 | Generated\ Files/ 41 | 42 | # MSTest test Results 43 | [Tt]est[Rr]esult*/ 44 | [Bb]uild[Ll]og.* 45 | 46 | # NUnit 47 | *.VisualState.xml 48 | TestResult.xml 49 | nunit-*.xml 50 | 51 | # Build Results of an ATL Project 52 | [Dd]ebugPS/ 53 | [Rr]eleasePS/ 54 | dlldata.c 55 | 56 | # Benchmark Results 57 | BenchmarkDotNet.Artifacts/ 58 | 59 | # .NET Core 60 | project.lock.json 61 | project.fragment.lock.json 62 | artifacts/ 63 | 64 | # StyleCop 65 | StyleCopReport.xml 66 | 67 | # Files built by Visual Studio 68 | *_i.c 69 | *_p.c 70 | *_h.h 71 | *.ilk 72 | *.meta 73 | *.obj 74 | *.iobj 75 | *.pch 76 | *.pdb 77 | *.ipdb 78 | *.pgc 79 | *.pgd 80 | *.rsp 81 | *.sbr 82 | *.tlb 83 | *.tli 84 | *.tlh 85 | *.tmp 86 | *.tmp_proj 87 | *_wpftmp.csproj 88 | *.log 89 | *.vspscc 90 | *.vssscc 91 | .builds 92 | *.pidb 93 | *.svclog 94 | *.scc 95 | 96 | # Chutzpah Test files 97 | _Chutzpah* 98 | 99 | # Visual C++ cache files 100 | ipch/ 101 | *.aps 102 | *.ncb 103 | *.opendb 104 | *.opensdf 105 | *.sdf 106 | *.cachefile 107 | *.VC.db 108 | *.VC.VC.opendb 109 | 110 | # Visual Studio profiler 111 | *.psess 112 | *.vsp 113 | *.vspx 114 | *.sap 115 | 116 | # Visual Studio Trace Files 117 | *.e2e 118 | 119 | # TFS 2012 Local Workspace 120 | $tf/ 121 | 122 | # Guidance Automation Toolkit 123 | *.gpState 124 | 125 | # ReSharper is a .NET coding add-in 126 | _ReSharper*/ 127 | *.[Rr]e[Ss]harper 128 | *.DotSettings.user 129 | 130 | # TeamCity is a build add-in 131 | _TeamCity* 132 | 133 | # DotCover is a Code Coverage Tool 134 | *.dotCover 135 | 136 | # AxoCover is a Code Coverage Tool 137 | .axoCover/* 138 | !.axoCover/settings.json 139 | 140 | # Visual Studio code coverage results 141 | *.coverage 142 | *.coveragexml 143 | 144 | # NCrunch 145 | _NCrunch_* 146 | .*crunch*.local.xml 147 | nCrunchTemp_* 148 | 149 | # MightyMoose 150 | *.mm.* 151 | AutoTest.Net/ 152 | 153 | # Web workbench (sass) 154 | .sass-cache/ 155 | 156 | # Installshield output folder 157 | [Ee]xpress/ 158 | 159 | # DocProject is a documentation generator add-in 160 | DocProject/buildhelp/ 161 | DocProject/Help/*.HxT 162 | DocProject/Help/*.HxC 163 | DocProject/Help/*.hhc 164 | DocProject/Help/*.hhk 165 | DocProject/Help/*.hhp 166 | DocProject/Help/Html2 167 | DocProject/Help/html 168 | 169 | # Click-Once directory 170 | publish/ 171 | 172 | # Publish Web Output 173 | *.[Pp]ublish.xml 174 | *.azurePubxml 175 | # Note: Comment the next line if you want to checkin your web deploy settings, 176 | # but database connection strings (with potential passwords) will be unencrypted 177 | *.pubxml 178 | *.publishproj 179 | 180 | # Microsoft Azure Web App publish settings. Comment the next line if you want to 181 | # checkin your Azure Web App publish settings, but sensitive information contained 182 | # in these scripts will be unencrypted 183 | PublishScripts/ 184 | 185 | # NuGet Packages 186 | *.nupkg 187 | # NuGet Symbol Packages 188 | *.snupkg 189 | # The packages folder can be ignored because of Package Restore 190 | **/[Pp]ackages/* 191 | # except build/, which is used as an MSBuild target. 192 | !**/[Pp]ackages/build/ 193 | # Uncomment if necessary however generally it will be regenerated when needed 194 | #!**/[Pp]ackages/repositories.config 195 | # NuGet v3's project.json files produces more ignorable files 196 | *.nuget.props 197 | *.nuget.targets 198 | 199 | # Microsoft Azure Build Output 200 | csx/ 201 | *.build.csdef 202 | 203 | # Microsoft Azure Emulator 204 | ecf/ 205 | rcf/ 206 | 207 | # Windows Store app package directories and files 208 | AppPackages/ 209 | BundleArtifacts/ 210 | Package.StoreAssociation.xml 211 | _pkginfo.txt 212 | *.appx 213 | *.appxbundle 214 | *.appxupload 215 | 216 | # Visual Studio cache files 217 | # files ending in .cache can be ignored 218 | *.[Cc]ache 219 | # but keep track of directories ending in .cache 220 | !?*.[Cc]ache/ 221 | 222 | # Others 223 | ClientBin/ 224 | ~$* 225 | *~ 226 | *.dbmdl 227 | *.dbproj.schemaview 228 | *.jfm 229 | *.pfx 230 | *.publishsettings 231 | orleans.codegen.cs 232 | 233 | # Including strong name files can present a security risk 234 | # (https://github.com/github/gitignore/pull/2483#issue-259490424) 235 | #*.snk 236 | 237 | # Since there are multiple workflows, uncomment next line to ignore bower_components 238 | # (https://github.com/github/gitignore/pull/1529#issuecomment-104372622) 239 | #bower_components/ 240 | 241 | # RIA/Silverlight projects 242 | Generated_Code/ 243 | 244 | # Backup & report files from converting an old project file 245 | # to a newer Visual Studio version. Backup files are not needed, 246 | # because we have git ;-) 247 | _UpgradeReport_Files/ 248 | Backup*/ 249 | UpgradeLog*.XML 250 | UpgradeLog*.htm 251 | ServiceFabricBackup/ 252 | *.rptproj.bak 253 | 254 | # SQL Server files 255 | *.mdf 256 | *.ldf 257 | *.ndf 258 | 259 | # Business Intelligence projects 260 | *.rdl.data 261 | *.bim.layout 262 | *.bim_*.settings 263 | *.rptproj.rsuser 264 | *- [Bb]ackup.rdl 265 | *- [Bb]ackup ([0-9]).rdl 266 | *- [Bb]ackup ([0-9][0-9]).rdl 267 | 268 | # Microsoft Fakes 269 | FakesAssemblies/ 270 | 271 | # GhostDoc plugin setting file 272 | *.GhostDoc.xml 273 | 274 | # Node.js Tools for Visual Studio 275 | .ntvs_analysis.dat 276 | node_modules/ 277 | 278 | # Visual Studio 6 build log 279 | *.plg 280 | 281 | # Visual Studio 6 workspace options file 282 | *.opt 283 | 284 | # Visual Studio 6 auto-generated workspace file (contains which files were open etc.) 285 | *.vbw 286 | 287 | # Visual Studio LightSwitch build output 288 | **/*.HTMLClient/GeneratedArtifacts 289 | **/*.DesktopClient/GeneratedArtifacts 290 | **/*.DesktopClient/ModelManifest.xml 291 | **/*.Server/GeneratedArtifacts 292 | **/*.Server/ModelManifest.xml 293 | _Pvt_Extensions 294 | 295 | # Paket dependency manager 296 | .paket/paket.exe 297 | paket-files/ 298 | 299 | # FAKE - F# Make 300 | .fake/ 301 | 302 | # CodeRush personal settings 303 | .cr/personal 304 | 305 | # Python Tools for Visual Studio (PTVS) 306 | __pycache__/ 307 | *.pyc 308 | 309 | # Cake - Uncomment if you are using it 310 | # tools/** 311 | # !tools/packages.config 312 | 313 | # Tabs Studio 314 | *.tss 315 | 316 | # Telerik's JustMock configuration file 317 | *.jmconfig 318 | 319 | # BizTalk build output 320 | *.btp.cs 321 | *.btm.cs 322 | *.odx.cs 323 | *.xsd.cs 324 | 325 | # OpenCover UI analysis results 326 | OpenCover/ 327 | 328 | # Azure Stream Analytics local run output 329 | ASALocalRun/ 330 | 331 | # MSBuild Binary and Structured Log 332 | *.binlog 333 | 334 | # NVidia Nsight GPU debugger configuration file 335 | *.nvuser 336 | 337 | # MFractors (Xamarin productivity tool) working folder 338 | .mfractor/ 339 | 340 | # Local History for Visual Studio 341 | .localhistory/ 342 | 343 | # BeatPulse healthcheck temp database 344 | healthchecksdb 345 | 346 | # Backup folder for Package Reference Convert tool in Visual Studio 2017 347 | MigrationBackup/ 348 | 349 | # Ionide (cross platform F# VS Code tools) working folder 350 | .ionide/ 351 | 352 | # certificates 353 | *.pfx 354 | *.cer 355 | *.key -------------------------------------------------------------------------------- /Ingress+Cloud/backend-echo.yml: -------------------------------------------------------------------------------- 1 | apiVersion: apps/v1 2 | kind: Deployment 3 | metadata: 4 | name: httpecho-deployment 5 | spec: 6 | replicas: 1 7 | selector: 8 | matchLabels: 9 | app: echo-pod 10 | template: 11 | metadata: 12 | labels: 13 | app: echo-pod 14 | spec: 15 | containers: 16 | - name: echo-container 17 | image: mendhak/http-https-echo:17 18 | resources: 19 | limits: 20 | memory: "128Mi" 21 | cpu: "1m" 22 | ports: 23 | - containerPort: 8080 24 | name: http 25 | --- 26 | apiVersion: v1 27 | kind: Service 28 | metadata: 29 | name: echo-service 30 | spec: 31 | type: ClusterIP 32 | selector: 33 | app: echo-pod 34 | ports: 35 | - port: 8080 36 | targetPort: 8080 37 | -------------------------------------------------------------------------------- /Ingress+Cloud/deploy-ingress.cmd: -------------------------------------------------------------------------------- 1 | kubectl apply -f .\ingress.yml -n=gw 2 | kubectl apply -f .\backend-echo.yml -n=gw 3 | kubectl apply -f .\ingress-deployment.yml -n=gw 4 | kubectl get all -n=gw -------------------------------------------------------------------------------- /Ingress+Cloud/gateway-apis.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/api-management-self-hosted-gateway-ingress/7eeeeda3d8e7a256e14586914fae1b7b56dac0a9/Ingress+Cloud/gateway-apis.png -------------------------------------------------------------------------------- /Ingress+Cloud/gateway-config.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/api-management-self-hosted-gateway-ingress/7eeeeda3d8e7a256e14586914fae1b7b56dac0a9/Ingress+Cloud/gateway-config.png -------------------------------------------------------------------------------- /Ingress+Cloud/ingress-deployment-cloud.yml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Secret 3 | metadata: 4 | name: my-gw-token 5 | type: Opaque 6 | stringData: 7 | value: "GatewayKey my-gw&202103281825&QxxxxxxxxxxxxxxxxxQykD2gAupAsOCCo6GgaVekN+2gkmzI1lReQ0yHDKfw741xxxxxxxxxx==" 8 | --- 9 | apiVersion: v1 10 | kind: ConfigMap 11 | metadata: 12 | name: azure-apim-gateway-environment 13 | data: 14 | config.service.endpoint: "https://apim-hello-world.management.azure-api.net/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/myResourceGroup/providers/Microsoft.ApiManagement/service/apim-hello-world?api-version=2019-12-01" 15 | --- 16 | apiVersion: apps/v1 17 | kind: Deployment 18 | metadata: 19 | name: apim-ingress-pod 20 | spec: 21 | replicas: 1 22 | selector: 23 | matchLabels: 24 | app: apim-ingress-pod 25 | template: 26 | metadata: 27 | labels: 28 | app: apim-ingress-pod 29 | spec: 30 | containers: 31 | - name: azure-apim-gateway 32 | image: mcr.microsoft.com/azure-api-management/gateway:2.4.0 33 | ports: 34 | - name: http 35 | containerPort: 8080 36 | - name: https 37 | containerPort: 8081 38 | resources: 39 | limits: 40 | memory: "128Mi" 41 | cpu: "100m" 42 | env: 43 | - name: k8s.ingress.namespace 44 | valueFrom: 45 | fieldRef: 46 | fieldPath: metadata.namespace 47 | - name: k8s.ingress.enabled 48 | value: "true" 49 | - name: config.service.auth 50 | valueFrom: 51 | secretKeyRef: 52 | name: my-gw-token 53 | key: value 54 | envFrom: 55 | - configMapRef: 56 | name: azure-apim-gateway-environment 57 | --- 58 | apiVersion: v1 59 | kind: Service 60 | metadata: 61 | name: apim-ingress-service 62 | spec: 63 | selector: 64 | app: apim-ingress-pod 65 | type: LoadBalancer 66 | ports: 67 | - port: 80 68 | name: ingress-http 69 | targetPort: 8080 70 | - port: 443 71 | name: ingress-https 72 | targetPort: 8081 73 | 74 | --- 75 | apiVersion: rbac.authorization.k8s.io/v1beta1 76 | kind: ClusterRoleBinding 77 | metadata: 78 | name: ingress-rbac 79 | subjects: 80 | - kind: ServiceAccount 81 | name: default 82 | namespace: gw 83 | roleRef: 84 | kind: ClusterRole 85 | name: cluster-admin 86 | apiGroup: rbac.authorization.k8s.io 87 | -------------------------------------------------------------------------------- /Ingress+Cloud/ingress-deployment.yml: -------------------------------------------------------------------------------- 1 | apiVersion: apps/v1 2 | kind: Deployment 3 | metadata: 4 | name: apim-ingress-pod 5 | spec: 6 | replicas: 1 7 | selector: 8 | matchLabels: 9 | app: apim-ingress-pod 10 | template: 11 | metadata: 12 | labels: 13 | app: apim-ingress-pod 14 | spec: 15 | containers: 16 | - name: azure-apim-gateway 17 | image: mcr.microsoft.com/azure-api-management/gateway:2.4.0 18 | ports: 19 | - name: http 20 | containerPort: 8080 21 | - name: https 22 | containerPort: 8081 23 | resources: 24 | limits: 25 | memory: "128Mi" 26 | cpu: "100m" 27 | env: 28 | - name: k8s.ingress.namespace 29 | valueFrom: 30 | fieldRef: 31 | fieldPath: metadata.namespace 32 | - name: k8s.ingress.enabled 33 | value: "true" 34 | --- 35 | apiVersion: v1 36 | kind: Service 37 | metadata: 38 | name: apim-ingress-service 39 | spec: 40 | selector: 41 | app: apim-ingress-pod 42 | type: LoadBalancer 43 | ports: 44 | - port: 80 45 | name: ingress-http 46 | targetPort: 8080 47 | - port: 443 48 | name: ingress-https 49 | targetPort: 8081 50 | --- 51 | apiVersion: networking.k8s.io/v1 52 | kind: IngressClass 53 | metadata: 54 | name: apim-ingress 55 | annotations: 56 | ingressclass.kubernetes.io/is-default-class: "true" 57 | spec: 58 | controller: azure-api-management/gateway 59 | --- 60 | apiVersion: rbac.authorization.k8s.io/v1beta1 61 | kind: ClusterRoleBinding 62 | metadata: 63 | name: ingress-rbac 64 | subjects: 65 | - kind: ServiceAccount 66 | name: default 67 | namespace: gw 68 | roleRef: 69 | kind: ClusterRole 70 | name: cluster-admin 71 | apiGroup: rbac.authorization.k8s.io 72 | 73 | -------------------------------------------------------------------------------- /Ingress+Cloud/ingress-tls.yml: -------------------------------------------------------------------------------- 1 | apiVersion: networking.k8s.io/v1 2 | kind: Ingress 3 | metadata: 4 | name: ingress 5 | spec: 6 | tls: 7 | - hosts: 8 | - www.contoso.com 9 | secretName: tls-www-contoso-com 10 | ingressClassName: "azure-api-management/gateway" 11 | rules: 12 | - host: "www.contoso.com" 13 | http: 14 | paths: 15 | - path: /echo-tls 16 | pathType: Prefix 17 | backend: 18 | service: 19 | name: echo-service 20 | port: 21 | number: 8080 -------------------------------------------------------------------------------- /Ingress+Cloud/ingress.yml: -------------------------------------------------------------------------------- 1 | apiVersion: networking.k8s.io/v1 2 | kind: Ingress 3 | metadata: 4 | name: ingress 5 | spec: 6 | ingressClassName: "azure-api-management/gateway" 7 | rules: 8 | - http: 9 | paths: 10 | - path: /echo 11 | pathType: Prefix 12 | backend: 13 | service: 14 | name: echo-service 15 | port: 16 | number: 8080 -------------------------------------------------------------------------------- /Ingress+Cloud/readme.md: -------------------------------------------------------------------------------- 1 | # Kubernetes Ingress and Cloud configuration in Self-Hosted Gateway 2 | 3 | This is a continuation of the [Ingress Only](../Ingress-only) walkthrough where an API Management self-hosted gateway was deployed and configured to read Ingress rules in a Kubernetes cluster. This example adds a cloud configuration for API Management. This support is currently experimental. 4 | 5 | If you are starting clean, please execute the following commands to get to the latest state: 6 | 7 | ``` 8 | kubectl create namespace gw 9 | kubectl apply -f backend-echo.yml -n=gw 10 | kubectl apply -f ingress-deployment.yml -n=gw 11 | kubectl apply -f ingress.yml -n=gw 12 | ``` 13 | 14 | ## Walkthrough 15 | At this point, a self-hosted gateway should be deployed and configured according to Ingress Rules to expose the `Echo` service. 16 | 17 | Now create a self-hosted gateway resource in your API Management instance and prepare to deploy the self-hosted gateway component to your Kubernetes cluster. Follow steps 1 - 4 (**only**) in [Deploy a self-hosted gateway to Kubernetes](https://docs.microsoft.com/azure/api-management/how-to-deploy-self-hosted-gateway-kubernetes). 18 | 19 | Continue with the following steps: 20 | 21 | 1. In the [Azure portal](https://portal.azure.com), select your gateway resource. 22 | 1. On the **Deployment** page, take note of the values of **Token** and **Configuration URL**. 23 | ![Self-hosted gateway configuration](gateway-config.png) 24 | 1. Download the [deployment template](ingress-deployment-cloud.yml) `ingress-deployment-cloud.yml`. 25 | 1. Before deploying, be sure to replace the following values for your endpoint, noted previously. 26 | 1. `config.service.auth` - [line 7](ingress-deployment-cloud.yml#L7) 27 | 1. `config.service.endpoint` - [line 14](ingress-deployment-cloud.yml#L14) 28 | 1. Deploy the gateway. 29 | ``` 30 | kubectl apply -f ingress-deployment-cloud.yml -n=gw 31 | ``` 32 | ### Test HTTP calls 33 | To test a runtime call, APIs need to be assigned to the gateway. 34 | 35 | The following example uses [Petstore APIs](https://petstore.swagger.io/) imported into API Management service and assigned to the self-hosted gateway. 36 | 37 | In the Azure portal the configuration looks like this: 38 | 39 | ![APIs in self-hosted gateway](gateway-apis.png) 40 | 41 | #### Get external IP address 42 | 43 | The following command returns the external IP address to which we can test the call. 44 | ``` 45 | > kubectl get service apim-ingress-service -n=gw4 46 | NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE 47 | apim-ingress-service LoadBalancer 10.0.100.231 40.125.75.211 80:31171/TCP,443:31959/TCP 27m 48 | ``` 49 | 50 | Now that we have everything configured, let's make an HTTP call. Using an external IP address `40.125.75.211` shown in the preceding case, and passing the subscription key in a header: 51 | ``` 52 | > curl -H "Ocp-Apim-Subscription-Key: " http://40.125.75.211/pet/store/inventory 53 | {"sold":8,"SOLD":1,"string":306,"pending":3,"available":663,"avalible":1} 54 | ``` 55 | 56 | ### Troubleshooting 57 | In case of difficulties, there are several things to check in the logs for the pod: 58 | 59 | #### Ensure k8s.ingress.enabled flag is set 60 | The following excerpt from the logs would provide this clue. 61 | ``` 62 | [Info] 2021-02-8T11:01:48.803 [ConfigIngressName], message: Starting gateway 5f56585f12db2b1b302ee653 in namespace gw4 and using prefix 4b26700a for all events to disambiguate K8 events., source: KubernetesConfigurationRepositoryProvider 63 | [Warn] 2021-02-8T11:01:49.309 [LoadBackupLocationNotFound], message: /apim/config, source: FileBackupProvider 64 | ... 65 | [Info] 2021-02-8T11:01:51.206 [ConfigStarted], source: CompositConfigurationRepositoryProvider 66 | [Info] 2021-02-8T11:01:51.206 [ConfigStarted], message: Kubernetes Ingress is enabled. 'k8s.ingress.enabled':True, source: CompositConfigurationRepositoryProvider 67 | [Info] 2021-02-8T11:01:51.303 [ConfigIngressStarted], source: KubernetesConfigurationRepositoryProvider 68 | ... 69 | [Info] 2021-02-8T11:01:51.807 [ReadingCurrentIngressList], source: KubernetesConfigurationRepositoryProvider 70 | [Info] 2021-02-8T11:01:53.107 [MatchingIngressFound], message: namespace: gw4, ingressClassName: azure-api-management/gateway, count: 1, resourceVersion: 147765571, source: KubernetesConfigurationRepositoryProvider 71 | [Info] 2021-02-8T11:01:53.109 [IngressAdded], message: namespace: gw4, ingress.class: azure-api-management/gateway, name: ingress, resourceVersion: 147762114, source: KubernetesConfigurationRepositoryProvider 72 | ... 73 | [Info] 2021-02-8T11:01:58.709 [IngressSuccessfullyProcessed], source: KubernetesConfigurationRepositoryProvider 74 | [Info] 2021-02-8T11:01:58.709 [CurrentIngressListProcessed], message: 147765571, source: KubernetesConfigurationRepositoryProvider 75 | [Info] 2021-02-8T11:01:58.907 [IngressWatcherInitialized], message: 147765571, source: KubernetesConfigurationRepositoryProvider 76 | ``` 77 | 78 | #### Ensure the gateway is connected to API Management 79 | The following traces show cloud configuration. 80 | ``` 81 | 82 | [Info] 2021-02-8T11:01:59.007 [ConfigStarted], message: Connected service is enabled. 'config.service.endpoint' is set., source: CompositConfigurationRepositoryProvider 83 | [Info] 2021-02-8T11:01:59.109 [ConfigInitialSyncStarted], source: ConfigurationRepositoryProvider 84 | [Info] 2021-02-8T11:02:02.405 [EventSnapshotElected], message: provider: storage, uri: https://[redacted].blob.core.windows.net/[redacted]/snapshot-2019-10-11.Proxy.Host.WebRole_IN_0.json.gzip, rev: 610, source: events.snapshot 85 | [Info] 2021-02-8T11:02:02.409 [ConfigurationRetrieving], message: https://[redacted].blob.core.windows.net/[redacted]/snapshot-2019-10-11.Proxy.Host.WebRole_IN_0.json.gzip, source: events.snapshot.storage.private 86 | [Info] 2021-02-8T11:02:02.810 [ConfigurationLoaded], message: https://[redacted].blob.core.windows.net/[redacted]/snapshot-2019-10-11.Proxy.Host.WebRole_IN_0.json.gzip, source: events.snapshot.storage.private 87 | [Info] 2021-02-8T11:02:05.603 [ConfigurationRetrieved], message: https://[redacted].blob.core.windows.net/[redacted]/snapshot-2019-10-11.Proxy.Host.WebRole_IN_0.json.gzip, source: events.snapshot.storage.private 88 | [Info] 2021-02-8T11:02:07.111 [OperationRouteTableRebuildStarted], message: echo-api;rev=1, source: ApiRouter 89 | [Info] 2021-02-8T11:02:07.411 [OperationRouteTableRebuildCompleted], message: echo-api;rev=1, source: ApiRouter 90 | [Info] 2021-02-8T11:02:07.411 [OperationRouteTableRebuildStarted], message: swagger-petstore;rev=1, source: ApiRouter 91 | [Info] 2021-02-8T11:02:07.507 [OperationRouteTableRebuildCompleted], message: swagger-petstore;rev=1, source: ApiRouter 92 | [Info] 2021-02-8T11:02:07.508 [OperationRouteTableRebuildStarted], message: httpbin;rev=1, source: ApiRouter 93 | [Info] 2021-02-8T11:02:07.508 [OperationRouteTableRebuildCompleted], message: httpbin;rev=1, source: ApiRouter 94 | [Info] 2021-02-8T11:02:07.810 [ManagedIdentityAddedToMap], message: systemAssigned, source: ManagedIdentityResolver 95 | ... 96 | [Info] 2021-02-8T11:02:10.305 [BootstrapperStarted], source: Bootstrapper 97 | [Info] 2021-02-8T11:02:10.307 [TenantStarted], message: 6021c2d47edd480001aed27e 00:00:29.9972171 98 | ``` 99 | 100 | -------------------------------------------------------------------------------- /Ingress-only/backend-echo.yml: -------------------------------------------------------------------------------- 1 | apiVersion: apps/v1 2 | kind: Deployment 3 | metadata: 4 | name: httpecho-deployment 5 | spec: 6 | replicas: 1 7 | selector: 8 | matchLabels: 9 | app: echo-pod 10 | template: 11 | metadata: 12 | labels: 13 | app: echo-pod 14 | spec: 15 | containers: 16 | - name: echo-container 17 | image: mendhak/http-https-echo:17 18 | resources: 19 | limits: 20 | memory: "128Mi" 21 | cpu: "1m" 22 | ports: 23 | - containerPort: 8080 24 | name: http 25 | --- 26 | apiVersion: v1 27 | kind: Service 28 | metadata: 29 | name: echo-service 30 | spec: 31 | type: ClusterIP 32 | selector: 33 | app: echo-pod 34 | ports: 35 | - port: 8080 36 | targetPort: 8080 37 | -------------------------------------------------------------------------------- /Ingress-only/deploy-ingress.cmd: -------------------------------------------------------------------------------- 1 | kubectl apply -f .\ingress.yml -n=gw 2 | kubectl apply -f .\backend-echo.yml -n=gw 3 | kubectl apply -f .\ingress-deployment.yml -n=gw 4 | kubectl get all -n=gw -------------------------------------------------------------------------------- /Ingress-only/ingress-deployment.yml: -------------------------------------------------------------------------------- 1 | apiVersion: apps/v1 2 | kind: Deployment 3 | metadata: 4 | name: apim-ingress-pod 5 | spec: 6 | replicas: 1 7 | selector: 8 | matchLabels: 9 | app: apim-ingress-pod 10 | template: 11 | metadata: 12 | labels: 13 | app: apim-ingress-pod 14 | spec: 15 | containers: 16 | - name: azure-apim-gateway 17 | image: mcr.microsoft.com/azure-api-management/gateway:2.4.0 18 | ports: 19 | - name: http 20 | containerPort: 8080 21 | - name: https 22 | containerPort: 8081 23 | resources: 24 | limits: 25 | memory: "128Mi" 26 | cpu: "100m" 27 | env: 28 | - name: k8s.ingress.namespace 29 | valueFrom: 30 | fieldRef: 31 | fieldPath: metadata.namespace 32 | - name: k8s.ingress.enabled 33 | value: "true" 34 | --- 35 | apiVersion: v1 36 | kind: Service 37 | metadata: 38 | name: apim-ingress-service 39 | spec: 40 | selector: 41 | app: apim-ingress-pod 42 | type: LoadBalancer 43 | ports: 44 | - port: 80 45 | name: ingress-http 46 | targetPort: 8080 47 | - port: 443 48 | name: ingress-https 49 | targetPort: 8081 50 | --- 51 | apiVersion: networking.k8s.io/v1 52 | kind: IngressClass 53 | metadata: 54 | name: apim-ingress 55 | annotations: 56 | ingressclass.kubernetes.io/is-default-class: "true" 57 | spec: 58 | controller: azure-api-management/gateway 59 | --- 60 | apiVersion: rbac.authorization.k8s.io/v1beta1 61 | kind: ClusterRoleBinding 62 | metadata: 63 | name: ingress-rbac 64 | subjects: 65 | - kind: ServiceAccount 66 | name: default 67 | namespace: gw 68 | roleRef: 69 | kind: ClusterRole 70 | name: cluster-admin 71 | apiGroup: rbac.authorization.k8s.io 72 | 73 | -------------------------------------------------------------------------------- /Ingress-only/ingress-tls.yml: -------------------------------------------------------------------------------- 1 | apiVersion: networking.k8s.io/v1 2 | kind: Ingress 3 | metadata: 4 | name: ingress 5 | annotations: 6 | spec: 7 | tls: 8 | - hosts: 9 | - www.contoso.com 10 | secretName: tls-www-contoso-com 11 | ingressClassName: "azure-api-management/gateway" 12 | rules: 13 | - host: "www.contoso.com" 14 | http: 15 | paths: 16 | - path: /echo-tls 17 | pathType: Prefix 18 | backend: 19 | service: 20 | name: echo-service 21 | port: 22 | number: 8080 -------------------------------------------------------------------------------- /Ingress-only/ingress.yml: -------------------------------------------------------------------------------- 1 | apiVersion: networking.k8s.io/v1 2 | kind: Ingress 3 | metadata: 4 | name: ingress 5 | spec: 6 | ingressClassName: "azure-api-management/gateway" 7 | rules: 8 | - http: 9 | paths: 10 | - path: /echo 11 | pathType: Prefix 12 | backend: 13 | service: 14 | name: echo-service 15 | port: 16 | number: 8080 -------------------------------------------------------------------------------- /Ingress-only/readme.md: -------------------------------------------------------------------------------- 1 | # Kubernetes Ingress support in Self-Hosted Gateway 2 | 3 | This sample and walkthrough show how to enable Kubernetes Ingress in an Azure API Management self-hosted gateway. See [Kubernetes Ingress support in Self-Hosted Gateway](https://github.com/Azure/api-management-self-hosted-gateway-ingress) for background and other samples. This support is currently experimental. 4 | 5 | To enable Ingress support, the following environment variables are set up using a [deployment template](ingress-deployment.yml#L29-L34): 6 | 7 | - `k8s.ingress.enabled` 8 | - Ingress object should include the annotation `kubernetes.io/ingress.class: "azure-api-management/gateway"` 9 | - `k8s.ingress.namespace` - optional namespace where Ingress is read from 10 | 11 | ## Walkthrough 12 | In this example, we set up a namespace `gw` to contain all resources. The self-hosted gateway configuration includes [RBAC configuration](ingress-deployment.yml#L54) that specifies this namespace, so we use the namespace consistently. 13 | 14 | One of the deployments is the [mendhak/http-https-echo:17](https://github.com/mendhak/docker-http-https-echo) container, which echoes back all HTTP/HTTPS traffic. See the latest container image tags on [Docker Hub](https://hub.docker.com/r/mendhak/http-https-echo/tags?page=1&ordering=last_updated). 15 | 16 | ### Deploy backend API service 17 | Let's start by creating the namespace and deploying the backend and service defined in [backend-echo.yml](backend-echo.yml). 18 | 19 | ``` 20 | kubectl create namespace gw 21 | kubectl apply -f backend-echo.yml -n=gw 22 | ``` 23 | 24 | ### Deploy API Management self-hosted gateway 25 | Next, let's deploy the self-hosted gateway from [ingress-deployment.yml](ingress-deployment.yml). 26 | 27 | ``` 28 | kubectl apply -f ingress-deployment.yml -n=gw 29 | ``` 30 | 31 | At that point your namespace should look like this: 32 | 33 | ``` 34 | > kubectl get all -n=gw 35 | NAME READY STATUS RESTARTS AGE 36 | pod/apim-ingress-pod-69864d749b-s6nnm 1/1 Running 0 40s 37 | pod/httpecho-deployment-57c4686bdb-rwzfq 1/1 Running 0 29s 38 | 39 | NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE 40 | service/apim-ingress-service LoadBalancer 10.0.11.47 80:30879/TCP,443:32764/TCP 40s 41 | service/echo-service ClusterIP 10.0.142.199 8443/TCP,8480/TCP 29s 42 | 43 | NAME READY UP-TO-DATE AVAILABLE AGE 44 | deployment.apps/apim-ingress-pod 1/1 1 1 40s 45 | deployment.apps/httpecho-deployment 1/1 1 1 29s 46 | 47 | NAME DESIRED CURRENT READY AGE 48 | replicaset.apps/apim-ingress-pod-69864d749b 1 1 1 40s 49 | replicaset.apps/httpecho-deployment-57c4686bdb 1 1 1 29s 50 | ``` 51 | 52 | Note the line below. If you are using Azure Kubernetes Service, it might take a few minutes until you see the `EXTERNAL-IP` address filled. 53 | ``` 54 | NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE 55 | service/apim-ingress-service LoadBalancer 10.0.234.184 40.125.75.211 80:31394/TCP,443:31170/TCP 4m33s 56 | service/echo-service ClusterIP 10.0.26.176 8080/TCP 4m20s 57 | ``` 58 | `apim-ingress-service` is exposed publicly and thus has an external IP address, while `echo-service` isn't accessible outside the cluster directly. 59 | 60 | ### Deploy Ingress rules 61 | Now let's expose `echo-service` application via Ingress rules. 62 | ``` 63 | > kubectl apply -f ingress.yml -n=gw 64 | ingress.networking.k8s.io/ingress created 65 | ``` 66 | 67 | ### Test HTTP calls 68 | Now that we have everything configured, let's make an HTTP call. Using an external IP address `40.125.75.211` in the preceding example: 69 | ``` 70 | > curl http://40.125.75.211/echo/hello/ingress 71 | { 72 | "path": "/echo/hello/ingress", 73 | "headers": { 74 | "accept": "*/*", 75 | "user-agent": "curl/7.55.1", 76 | "host": "echo-service.gw.svc.cluster.local:8080", 77 | "x-forwarded-for": "10.240.0.5" 78 | }, 79 | "method": "GET", 80 | "body": "", 81 | "fresh": false, 82 | "hostname": "echo-service.gw.svc.cluster.local", 83 | "ip": "10.240.0.5", 84 | "ips": [ 85 | "10.240.0.5" 86 | ], 87 | "protocol": "http", 88 | "query": {}, 89 | "subdomains": [ 90 | "svc", 91 | "gw", 92 | "echo-service" 93 | ], 94 | "xhr": false, 95 | "os": { 96 | "hostname": "httpecho-deployment-7758b7747f-c4dbk" 97 | }, 98 | "connection": {} 99 | } 100 | ``` 101 | 102 | ## Configure SSL and hostname 103 | To confgure hostname we need to do following steps: 104 | 1. Configure TLS certificate 105 | 1. Configure hostname in the Ingress 106 | 1. Set up DNS server to the external IP address 107 | 108 | ### 1. Generate TLS certificate 109 | The following commands will generate a TLS certificate and upload it as a secret to the Kubernetes cluster, where the self-hosted gateway can access it. For this purpose we can use [OpenSSL](https://github.com/openssl/openssl#download). 110 | 111 | ``` 112 | openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout www.contoso.com.key -out www.contoso.com.cer -subj "/CN=www.contoso.com/O=www.contoso.com" 113 | 114 | kubectl create secret tls tls-www-contoso-com --key www.contoso.com.key --cert www.contoso.com.cer -n=gw 115 | ``` 116 | 117 | ### 2. Configure hostname in the Ingress object 118 | There is a separate file [ingress-tls.yml](ingress-tls.yml) which has the full configuration, but the main changes are in [this section](ingress-tls.yml#L9-L12). Let's apply new rules. 119 | 120 | ``` 121 | > kubectl apply -f .\ingress-tls.yml -n=gw 122 | ingress.networking.k8s.io/ingress configured 123 | ``` 124 | 125 | ### 3. Configure DNS server 126 | Let's map `www.contoso.com` to `40.125.75.211`: 127 | - For Windows, this can be done in [host file](https://gist.github.com/zenorocha/18b10a14b2deb214dc4ce43a2d2e2992) 128 | - For Linux use [/etc/hosts](https://linuxize.com/post/how-to-edit-your-hosts-file) 129 | 130 | Now test the call again. Note the `--insecure` flag because the certificate is self-signed and will not be trusted. This call uses the `echo-tls` path: 131 | ``` 132 | > curl --insecure https://www.contoso.com/echo-tls/hello/ingress 133 | { 134 | "path": "/echo/hello/ingress", 135 | "headers": { 136 | "accept": "*/*", 137 | "user-agent": "curl/7.55.1", 138 | "host": "echo-service.gw.svc.cluster.local:8080", 139 | "x-forwarded-for": "10.244.3.1" 140 | }, 141 | "method": "GET", 142 | "body": "", 143 | "fresh": false, 144 | "hostname": "echo-service.gw.svc.cluster.local", 145 | "ip": "10.244.3.1", 146 | "ips": [ 147 | "10.244.3.1" 148 | ], 149 | "protocol": "http", 150 | "query": {}, 151 | "subdomains": [ 152 | "svc", 153 | "gw", 154 | "echo-service" 155 | ], 156 | "xhr": false, 157 | "os": { 158 | "hostname": "httpecho-deployment-7758b7747f-c4dbk" 159 | }, 160 | "connection": {} 161 | } 162 | ``` 163 | 164 | Optionally continue to [Kubernetes Ingress with API Management cloud configuration](../Ingress%2BCloud). -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) Microsoft Corporation. 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Azure API Management Self-Hosted Gateway Ingress samples 2 | 3 | The samples in this repository show how to set up an Azure API Management self-hosted gateway to provide Kubernetes Ingress support. Kubernetes Ingress support in the self-hosted gateway is currently experimental. 4 | 5 | * [Kubernetes Ingress only](../../tree/main/Ingress-only) 6 | * [Kubernetes Ingress with API Management cloud configuration](../../tree/main/Ingress%2BCloud) 7 | 8 | ## What is Azure API Management self-hosted gateway? 9 | 10 | [Self-hosted gateway](https://aka.ms/apim/sputnik/overview) is a feature of [Azure API Management](https://aka.ms/apimrocks). The self-hosted gateway, a containerized version of the API Management gateway component, expands API Management support for hybrid and multi-cloud environments. It allows customers to manage all of their APIs using a single API management solution without compromising security, compliance, or performance. Customers can deploy the self-hosted gateways to the same environments where they host their APIs while continuing to manage them from an associated API Management service in Azure. 11 | 12 | See more [Azure API Management self-hosted gateway samples](https://github.com/Azure/api-management-self-hosted-gateway). 13 | 14 | ## Kubernetes Ingress support in self-hosted gateway 15 | 16 | [Kubernetes Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/) resource is an API object that manages external access to the services in a cluster, typically over HTTP. 17 | 18 | With this experimental support in the Azure API Management Gateway, the following features can be configured in the Ingress object using the networking.k8s.io/v1 API: 19 | 20 | * SSL termination 21 | * API route exposure 22 | * Support for both [exact and prefix](https://v1-27.docs.kubernetes.io/docs/concepts/services-networking/ingress/#path-types) path types 23 | 24 | To enable Ingress support, the following environment variables need to be set up (see [deployment template](../../tree/main/Ingress-only/ingress-deployment.yml#L29-L34)): 25 | 26 | * `k8s.ingress.enabled` 27 | * Ingress object should include the annotation `kubernetes.io/ingress.class: "azure-api-management/gateway"` 28 | * `k8s.ingress.namespace` - optional namespace where Ingress is read from 29 | 30 | See [deployment sample and walkthrough](../../tree/main/Ingress-only). 31 | 32 | ## Hybrid support with a cloud configuration 33 | 34 | Normally, the self-hosted gateway would have the following environment variables which are needed to communicate with the Azure API Management service deployed to the cloud: 35 | 36 | * `config.service.endpoint` 37 | * `config.service.auth` 38 | 39 | Here is a [snippet](https://github.com/Azure/api-management-self-hosted-gateway/blob/master/examples/self-hosted-gateway-with-configuration-backup.yaml#L39-L47) of the configuration. For details, see [Deploy a self-hosted gateway to Kubernetes](https://docs.microsoft.com/azure/api-management/how-to-deploy-self-hosted-gateway-kubernetes). 40 | 41 | Combining these environment variables with the environment variables for Ingress configures basic routes via Kubernetes Ingress and the full power of Azure API Management policies and transformations via cloud configuration. 42 | 43 | See [deployment sample and walkthrough](../../tree/main/Ingress%2BCloud). 44 | 45 | ### How is configuration applied? 46 | 47 | Upon starting a new instance of the self-hosted gateway container, it looks for Ingress in the namespace passed as an environment variable `k8s.ingress.namespace` and creates routes, certificates, hostnames, and gateway entities. 48 | 49 | Next, looking at `config.service.*` settings, the gateway connects to the cloud service to fetch a cloud configuration snapshot and starts listening for configuration changes. 50 | 51 | From that point, the gateway is initialized and configuration changes from Ingress and cloud configuration are applied in a **last one wins** strategy. 52 | 53 | Periodically, the gateway creates a snapshot of the most recent effective confguration from the cloud to be able to load faster on next boot. Kubernetes Ingress objects are not added to the snapshot, because those are cluster specific and might differ on the next gateway boot. 54 | 55 | **Important**: Conflicting snapshot and configuration change events from two sources can cause discrepancies in the active configuration. For example, consider the following order: 56 | 57 | 1. Self-hosted gateway boots on a Kubernetes cluster in hybrid configuration mode. 58 | 1. Cloud configuration adds a `/users` API. 59 | 1. Ingress adds a new `/users` route, overwriting the cloud API. 60 | 1. All runtime calls will be executed as per Ingress confguration. 61 | 62 | Now if the pod is torn down or nodes reboot, configuration will be loaded as: 63 | 64 | 1. Load the Ingress object, which has `/users` route. 65 | 1. Cloud confguration will overwrite `/users` route from snapshot. 66 | -------------------------------------------------------------------------------- /SECURITY.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | ## Security 4 | 5 | Microsoft takes the security of our software products and services seriously, which includes all source code repositories managed through our GitHub organizations, which include [Microsoft](https://github.com/Microsoft), [Azure](https://github.com/Azure), [DotNet](https://github.com/dotnet), [AspNet](https://github.com/aspnet), [Xamarin](https://github.com/xamarin), and [our GitHub organizations](https://opensource.microsoft.com/). 6 | 7 | If you believe you have found a security vulnerability in any Microsoft-owned repository that meets [Microsoft's definition of a security vulnerability](https://aka.ms/opensource/security/definition), please report it to us as described below. 8 | 9 | ## Reporting Security Issues 10 | 11 | **Please do not report security vulnerabilities through public GitHub issues.** 12 | 13 | Instead, please report them to the Microsoft Security Response Center (MSRC) at [https://msrc.microsoft.com/create-report](https://aka.ms/opensource/security/create-report). 14 | 15 | If you prefer to submit without logging in, send email to [secure@microsoft.com](mailto:secure@microsoft.com). If possible, encrypt your message with our PGP key; please download it from the [Microsoft Security Response Center PGP Key page](https://aka.ms/opensource/security/pgpkey). 16 | 17 | You should receive a response within 24 hours. If for some reason you do not, please follow up via email to ensure we received your original message. Additional information can be found at [microsoft.com/msrc](https://aka.ms/opensource/security/msrc). 18 | 19 | Please include the requested information listed below (as much as you can provide) to help us better understand the nature and scope of the possible issue: 20 | 21 | * Type of issue (e.g. buffer overflow, SQL injection, cross-site scripting, etc.) 22 | * Full paths of source file(s) related to the manifestation of the issue 23 | * The location of the affected source code (tag/branch/commit or direct URL) 24 | * Any special configuration required to reproduce the issue 25 | * Step-by-step instructions to reproduce the issue 26 | * Proof-of-concept or exploit code (if possible) 27 | * Impact of the issue, including how an attacker might exploit the issue 28 | 29 | This information will help us triage your report more quickly. 30 | 31 | If you are reporting for a bug bounty, more complete reports can contribute to a higher bounty award. Please visit our [Microsoft Bug Bounty Program](https://aka.ms/opensource/security/bounty) page for more details about our active programs. 32 | 33 | ## Preferred Languages 34 | 35 | We prefer all communications to be in English. 36 | 37 | ## Policy 38 | 39 | Microsoft follows the principle of [Coordinated Vulnerability Disclosure](https://aka.ms/opensource/security/cvd). 40 | 41 | 42 | --------------------------------------------------------------------------------