├── .gitignore ├── Azure AD SAML SSO ├── Elasticsearch │ ├── Dockerfile │ ├── ES_ConfigMap.yaml │ ├── ES_PersistentVolumeClaim.yaml │ ├── ES_Service.yaml │ ├── ES_StatefulSet.yaml │ └── ES_StatefulSet_V2.yaml ├── Kibana │ ├── Dockerfile │ ├── Kibana_ConfigMap.yaml │ ├── Kibana_Deployment.yaml │ ├── Kibana_Deployment_V2.yaml │ └── Kibana_Service.yaml ├── Logstash │ ├── Dockerfile │ ├── Logstash_ConfigMap.yaml │ ├── Logstash_Deployment.yaml │ ├── Logstash_Deployment_V2.yaml │ └── Logstash_Service.yaml └── README.md ├── AzureRedisCacheSample ├── AzureRedisCacheSample.csproj └── Program.cs ├── ELKSamples.sln ├── ELKSamples └── AzureEventHubSample │ ├── AzureEventHubSample.csproj │ └── Program.cs ├── ES_ConfigMap.yaml ├── ES_PersistentVolumeClaim.yaml ├── ES_Service.yaml ├── ES_StatefulSet.yaml ├── Kibana_ConfigMap.yaml ├── Kibana_Deployment.yaml ├── Kibana_Service.yaml ├── LICENSE ├── Logstash_ConfigMap.yaml ├── Logstash_Deployment.yaml ├── Logstash_Service.yaml └── README.md /.gitignore: -------------------------------------------------------------------------------- 1 | ## Ignore Visual Studio temporary files, build results, and 2 | ## files generated by popular Visual Studio add-ons. 3 | ## 4 | ## Get latest from https://github.com/github/gitignore/blob/master/VisualStudio.gitignore 5 | 6 | # User-specific files 7 | *.suo 8 | *.user 9 | *.userosscache 10 | *.sln.docstates 11 | 12 | # User-specific files (MonoDevelop/Xamarin Studio) 13 | *.userprefs 14 | 15 | # Build results 16 | [Dd]ebug/ 17 | [Dd]ebugPublic/ 18 | [Rr]elease/ 19 | [Rr]eleases/ 20 | x64/ 21 | x86/ 22 | bld/ 23 | [Bb]in/ 24 | [Oo]bj/ 25 | [Ll]og/ 26 | 27 | # Visual Studio 2015/2017 cache/options directory 28 | .vs/ 29 | # Uncomment if you have tasks that create the project's static files in wwwroot 30 | #wwwroot/ 31 | 32 | # Visual Studio 2017 auto generated files 33 | Generated\ Files/ 34 | 35 | # MSTest test Results 36 | [Tt]est[Rr]esult*/ 37 | [Bb]uild[Ll]og.* 38 | 39 | # NUNIT 40 | *.VisualState.xml 41 | TestResult.xml 42 | 43 | # Build Results of an ATL Project 44 | [Dd]ebugPS/ 45 | [Rr]eleasePS/ 46 | dlldata.c 47 | 48 | # Benchmark Results 49 | BenchmarkDotNet.Artifacts/ 50 | 51 | # .NET Core 52 | project.lock.json 53 | project.fragment.lock.json 54 | artifacts/ 55 | **/Properties/launchSettings.json 56 | 57 | # StyleCop 58 | StyleCopReport.xml 59 | 60 | # Files built by Visual Studio 61 | *_i.c 62 | *_p.c 63 | *_i.h 64 | *.ilk 65 | *.meta 66 | *.obj 67 | *.iobj 68 | *.pch 69 | *.pdb 70 | *.ipdb 71 | *.pgc 72 | *.pgd 73 | *.rsp 74 | *.sbr 75 | *.tlb 76 | *.tli 77 | *.tlh 78 | *.tmp 79 | *.tmp_proj 80 | *.log 81 | *.vspscc 82 | *.vssscc 83 | .builds 84 | *.pidb 85 | *.svclog 86 | *.scc 87 | 88 | # Chutzpah Test files 89 | _Chutzpah* 90 | 91 | # Visual C++ cache files 92 | ipch/ 93 | *.aps 94 | *.ncb 95 | *.opendb 96 | *.opensdf 97 | *.sdf 98 | *.cachefile 99 | *.VC.db 100 | *.VC.VC.opendb 101 | 102 | # Visual Studio profiler 103 | *.psess 104 | *.vsp 105 | *.vspx 106 | *.sap 107 | 108 | # Visual Studio Trace Files 109 | *.e2e 110 | 111 | # TFS 2012 Local Workspace 112 | $tf/ 113 | 114 | # Guidance Automation Toolkit 115 | *.gpState 116 | 117 | # ReSharper is a .NET coding add-in 118 | _ReSharper*/ 119 | *.[Rr]e[Ss]harper 120 | *.DotSettings.user 121 | 122 | # JustCode is a .NET coding add-in 123 | .JustCode 124 | 125 | # TeamCity is a build add-in 126 | _TeamCity* 127 | 128 | # DotCover is a Code Coverage Tool 129 | *.dotCover 130 | 131 | # AxoCover is a Code Coverage Tool 132 | .axoCover/* 133 | !.axoCover/settings.json 134 | 135 | # Visual Studio code coverage results 136 | *.coverage 137 | *.coveragexml 138 | 139 | # NCrunch 140 | _NCrunch_* 141 | .*crunch*.local.xml 142 | nCrunchTemp_* 143 | 144 | # MightyMoose 145 | *.mm.* 146 | AutoTest.Net/ 147 | 148 | # Web workbench (sass) 149 | .sass-cache/ 150 | 151 | # Installshield output folder 152 | [Ee]xpress/ 153 | 154 | # DocProject is a documentation generator add-in 155 | DocProject/buildhelp/ 156 | DocProject/Help/*.HxT 157 | DocProject/Help/*.HxC 158 | DocProject/Help/*.hhc 159 | DocProject/Help/*.hhk 160 | DocProject/Help/*.hhp 161 | DocProject/Help/Html2 162 | DocProject/Help/html 163 | 164 | # Click-Once directory 165 | publish/ 166 | 167 | # Publish Web Output 168 | *.[Pp]ublish.xml 169 | *.azurePubxml 170 | # Note: Comment the next line if you want to checkin your web deploy settings, 171 | # but database connection strings (with potential passwords) will be unencrypted 172 | *.pubxml 173 | *.publishproj 174 | 175 | # Microsoft Azure Web App publish settings. Comment the next line if you want to 176 | # checkin your Azure Web App publish settings, but sensitive information contained 177 | # in these scripts will be unencrypted 178 | PublishScripts/ 179 | 180 | # NuGet Packages 181 | *.nupkg 182 | # The packages folder can be ignored because of Package Restore 183 | **/[Pp]ackages/* 184 | # except build/, which is used as an MSBuild target. 185 | !**/[Pp]ackages/build/ 186 | # Uncomment if necessary however generally it will be regenerated when needed 187 | #!**/[Pp]ackages/repositories.config 188 | # NuGet v3's project.json files produces more ignorable files 189 | *.nuget.props 190 | *.nuget.targets 191 | 192 | # Microsoft Azure Build Output 193 | csx/ 194 | *.build.csdef 195 | 196 | # Microsoft Azure Emulator 197 | ecf/ 198 | rcf/ 199 | 200 | # Windows Store app package directories and files 201 | AppPackages/ 202 | BundleArtifacts/ 203 | Package.StoreAssociation.xml 204 | _pkginfo.txt 205 | *.appx 206 | 207 | # Visual Studio cache files 208 | # files ending in .cache can be ignored 209 | *.[Cc]ache 210 | # but keep track of directories ending in .cache 211 | !*.[Cc]ache/ 212 | 213 | # Others 214 | ClientBin/ 215 | ~$* 216 | *~ 217 | *.dbmdl 218 | *.dbproj.schemaview 219 | *.jfm 220 | *.pfx 221 | *.publishsettings 222 | orleans.codegen.cs 223 | 224 | # Including strong name files can present a security risk 225 | # (https://github.com/github/gitignore/pull/2483#issue-259490424) 226 | #*.snk 227 | 228 | # Since there are multiple workflows, uncomment next line to ignore bower_components 229 | # (https://github.com/github/gitignore/pull/1529#issuecomment-104372622) 230 | #bower_components/ 231 | 232 | # RIA/Silverlight projects 233 | Generated_Code/ 234 | 235 | # Backup & report files from converting an old project file 236 | # to a newer Visual Studio version. Backup files are not needed, 237 | # because we have git ;-) 238 | _UpgradeReport_Files/ 239 | Backup*/ 240 | UpgradeLog*.XML 241 | UpgradeLog*.htm 242 | ServiceFabricBackup/ 243 | *.rptproj.bak 244 | 245 | # SQL Server files 246 | *.mdf 247 | *.ldf 248 | *.ndf 249 | 250 | # Business Intelligence projects 251 | *.rdl.data 252 | *.bim.layout 253 | *.bim_*.settings 254 | *.rptproj.rsuser 255 | 256 | # Microsoft Fakes 257 | FakesAssemblies/ 258 | 259 | # GhostDoc plugin setting file 260 | *.GhostDoc.xml 261 | 262 | # Node.js Tools for Visual Studio 263 | .ntvs_analysis.dat 264 | node_modules/ 265 | 266 | # Visual Studio 6 build log 267 | *.plg 268 | 269 | # Visual Studio 6 workspace options file 270 | *.opt 271 | 272 | # Visual Studio 6 auto-generated workspace file (contains which files were open etc.) 273 | *.vbw 274 | 275 | # Visual Studio LightSwitch build output 276 | **/*.HTMLClient/GeneratedArtifacts 277 | **/*.DesktopClient/GeneratedArtifacts 278 | **/*.DesktopClient/ModelManifest.xml 279 | **/*.Server/GeneratedArtifacts 280 | **/*.Server/ModelManifest.xml 281 | _Pvt_Extensions 282 | 283 | # Paket dependency manager 284 | .paket/paket.exe 285 | paket-files/ 286 | 287 | # FAKE - F# Make 288 | .fake/ 289 | 290 | # JetBrains Rider 291 | .idea/ 292 | *.sln.iml 293 | 294 | # CodeRush 295 | .cr/ 296 | 297 | # Python Tools for Visual Studio (PTVS) 298 | __pycache__/ 299 | *.pyc 300 | 301 | # Cake - Uncomment if you are using it 302 | # tools/** 303 | # !tools/packages.config 304 | 305 | # Tabs Studio 306 | *.tss 307 | 308 | # Telerik's JustMock configuration file 309 | *.jmconfig 310 | 311 | # BizTalk build output 312 | *.btp.cs 313 | *.btm.cs 314 | *.odx.cs 315 | *.xsd.cs 316 | 317 | # OpenCover UI analysis results 318 | OpenCover/ 319 | 320 | # Azure Stream Analytics local run output 321 | ASALocalRun/ 322 | 323 | # MSBuild Binary and Structured Log 324 | *.binlog 325 | 326 | # NVidia Nsight GPU debugger configuration file 327 | *.nvuser 328 | 329 | # MFractors (Xamarin productivity tool) working folder 330 | .mfractor/ 331 | -------------------------------------------------------------------------------- /Azure AD SAML SSO/Elasticsearch/Dockerfile: -------------------------------------------------------------------------------- 1 | # Elasticsearch 2 | FROM docker.elastic.co/elasticsearch/elasticsearch:6.4.1 3 | COPY --chown=elasticsearch:elasticsearch elastic-stack-ca.p12 /usr/share/elasticsearch/config/ 4 | COPY --chown=elasticsearch:elasticsearch elastic-certificates.p12 /usr/share/elasticsearch/config/ 5 | COPY --chown=elasticsearch:elasticsearch Elasticsearch.xml /usr/share/elasticsearch/config/ 6 | 7 | RUN /bin/bash -c "echo Password1$ | /usr/share/elasticsearch/bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password -xf" 8 | RUN /bin/bash -c "echo Password1$ | /usr/share/elasticsearch/bin/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password -xf" 9 | RUN /bin/bash -c "echo Password1$ | /usr/share/elasticsearch/bin/elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password -xf" 10 | RUN /bin/bash -c "echo Password1$ | /usr/share/elasticsearch/bin/elasticsearch-keystore add xpack.security.http.ssl.truststore.secure_password -xf;" -------------------------------------------------------------------------------- /Azure AD SAML SSO/Elasticsearch/ES_ConfigMap.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: ConfigMap 3 | metadata: 4 | name: sample-elasticsearch-configmap 5 | namespace: default 6 | data: 7 | elasticsearch.yml: | 8 | cluster.name: "sample-elasticsearch-cluster" 9 | network.host: 0.0.0.0 10 | discovery.zen.minimum_master_nodes: 1 11 | node.max_local_storage_nodes: 1 12 | xpack.security.enabled: true 13 | xpack.monitoring.collection.enabled: true 14 | xpack.security.transport.ssl.enabled: true 15 | xpack.security.transport.ssl.verification_mode: certificate 16 | xpack.license.self_generated.type: trial 17 | xpack.ssl.keystore.type: PKCS12 18 | xpack.ssl.keystore.path: "elastic-certificates.p12" 19 | xpack.ssl.keystore.password: Password1$ 20 | xpack.ssl.truststore.type: PKCS12 21 | xpack.ssl.truststore.path: "elastic-certificates.p12" 22 | xpack.ssl.truststore.password: Password1$ 23 | xpack.security.transport.ssl.keystore.path: "elastic-certificates.p12" 24 | xpack.security.transport.ssl.truststore.path: "elastic-certificates.p12" 25 | xpack.security.http.ssl.enabled: true 26 | xpack.security.http.ssl.keystore.path: "elastic-certificates.p12" 27 | xpack.security.http.ssl.truststore.path: "elastic-certificates.p12" 28 | xpack.ssl.verification_mode: certificate 29 | http.cors.enabled: true 30 | http.cors.allow-origin: "*" 31 | http.max_header_size: 16kb 32 | xpack: 33 | security: 34 | authc: 35 | realms: 36 | native1: 37 | type: native 38 | order: 0 39 | saml1: 40 | type: saml 41 | order: 2 42 | idp.metadata.path: "/usr/share/elasticsearch/config/Elasticsearch.xml" 43 | idp.entity_id: "AZURE_AD_IDENTIFIER_URL" 44 | sp.entity_id: "https://KIBANA_URL:5601/" 45 | sp.acs: "https://KIBANA_URL:5601/api/security/v1/saml" 46 | sp.logout: "https://KIBANA_URL:5601/logout" 47 | attributes.principal: "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/name" 48 | attributes.groups: "http://schemas.microsoft.com/ws/2008/06/identity/claims/role" 49 | role_mapping.yml: | 50 | 51 | 52 | -------------------------------------------------------------------------------- /Azure AD SAML SSO/Elasticsearch/ES_PersistentVolumeClaim.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: PersistentVolumeClaim 3 | metadata: 4 | name: sample-elasticsearch-data-claim 5 | spec: 6 | accessModes: 7 | - ReadWriteOnce 8 | resources: 9 | requests: 10 | storage: 5Gi -------------------------------------------------------------------------------- /Azure AD SAML SSO/Elasticsearch/ES_Service.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Service 3 | metadata: 4 | name: sample-elasticsearch 5 | labels: 6 | service: sample-elasticsearch 7 | spec: 8 | type: ClusterIP 9 | selector: 10 | service: sample-elasticsearch 11 | ports: 12 | - name: http 13 | port: 9200 14 | targetPort: 9200 15 | protocol: TCP 16 | - name: transport 17 | port: 9300 18 | targetPort: 9300 19 | protocol: TCP 20 | 21 | --- 22 | 23 | apiVersion: v1 24 | kind: Service 25 | metadata: 26 | name: sample-elasticsearch-discovery 27 | spec: 28 | clusterIP: None 29 | selector: 30 | service: sample-elasticsearch 31 | ports: 32 | - name: transport 33 | port: 9300 -------------------------------------------------------------------------------- /Azure AD SAML SSO/Elasticsearch/ES_StatefulSet.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: apps/v1 2 | kind: StatefulSet 3 | metadata: 4 | name: sample-elasticsearch 5 | labels: 6 | service: sample-elasticsearch 7 | spec: 8 | serviceName: sample-elasticsearch-discovery 9 | # Number of PODS 10 | replicas: 1 11 | selector: 12 | matchLabels: 13 | service: sample-elasticsearch 14 | template: 15 | metadata: 16 | labels: 17 | service: sample-elasticsearch 18 | spec: 19 | terminationGracePeriodSeconds: 300 20 | initContainers: 21 | # https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html 22 | - name: increase-the-vm-max-map-count 23 | image: busybox 24 | command: 25 | - sysctl 26 | - -w 27 | - vm.max_map_count=262144 28 | securityContext: 29 | privileged: true 30 | - name: fix-the-volume-permission 31 | image: busybox 32 | command: ["sh", "-c", "chown -R 1000:1000 /usr/share/elasticsearch/data"] 33 | securityContext: 34 | privileged: true 35 | volumeMounts: 36 | - name: sample-elasticsearch-data-claim 37 | mountPath: /usr/share/elasticsearch/data 38 | containers: 39 | - name: sample-elasticsearch 40 | image: YOUR_ELASTICSEARCH_IMAGE 41 | securityContext: 42 | runAsUser: 1000 43 | resources: 44 | limits: 45 | cpu: 2000m 46 | memory: 2Gi 47 | requests: 48 | cpu: 100m 49 | memory: 1Gi 50 | ports: 51 | - containerPort: 9200 52 | name: http 53 | - containerPort: 9300 54 | name: tcp 55 | env: 56 | - name: cluster.name 57 | value: "sample-elasticsearch-cluster" 58 | - name: "xpack.security.enabled" 59 | value: "true" 60 | - name: "TAKE_FILE_OWNERSHIP" 61 | value: “1” 62 | - name: node.name 63 | valueFrom: 64 | fieldRef: 65 | fieldPath: metadata.name 66 | - name: discovery.zen.ping.unicast.hosts 67 | value: "sample-elasticsearch-discovery:9300" 68 | volumeMounts: 69 | - name: sample-elasticsearch-data-claim 70 | mountPath: /usr/share/elasticsearch/data 71 | - name: sample-elasticsearch-configmap 72 | mountPath: /usr/share/elasticsearch/config/elasticsearch.yml 73 | subPath: elasticsearch.yml 74 | - name: sample-elasticsearch-configmap 75 | mountPath: /usr/share/elasticsearch/config/role_mapping.yml 76 | subPath: role_mapping.yml 77 | volumes: 78 | - name: sample-elasticsearch-configmap 79 | configMap: 80 | name: sample-elasticsearch-configmap 81 | volumeClaimTemplates: 82 | - metadata: 83 | name: sample-elasticsearch-data-claim 84 | spec: 85 | accessModes: [ "ReadWriteOnce" ] 86 | storageClassName: default 87 | resources: 88 | requests: 89 | storage: 2Gi -------------------------------------------------------------------------------- /Azure AD SAML SSO/Elasticsearch/ES_StatefulSet_V2.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: apps/v1 2 | kind: StatefulSet 3 | metadata: 4 | name: sample-elasticsearch 5 | labels: 6 | service: sample-elasticsearch 7 | spec: 8 | serviceName: sample-elasticsearch-discovery 9 | # Number of PODS 10 | replicas: 1 11 | selector: 12 | matchLabels: 13 | service: sample-elasticsearch 14 | template: 15 | metadata: 16 | labels: 17 | service: sample-elasticsearch 18 | spec: 19 | terminationGracePeriodSeconds: 300 20 | initContainers: 21 | # https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html 22 | - name: increase-the-vm-max-map-count 23 | image: busybox 24 | command: 25 | - sysctl 26 | - -w 27 | - vm.max_map_count=262144 28 | securityContext: 29 | privileged: true 30 | - name: fix-the-volume-permission 31 | image: busybox 32 | command: ["sh", "-c", "chown -R 1000:1000 /usr/share/elasticsearch/data"] 33 | securityContext: 34 | privileged: true 35 | volumeMounts: 36 | - name: sample-elasticsearch-data-claim 37 | mountPath: /usr/share/elasticsearch/data 38 | containers: 39 | - name: sample-elasticsearch 40 | image: docker.elastic.co/elasticsearch/elasticsearch:6.4.1 41 | securityContext: 42 | runAsUser: 1000 43 | resources: 44 | limits: 45 | cpu: 2000m 46 | memory: 2Gi 47 | requests: 48 | cpu: 100m 49 | memory: 1Gi 50 | ports: 51 | - containerPort: 9200 52 | name: http 53 | - containerPort: 9300 54 | name: tcp 55 | command: ["/bin/bash","-c", "echo $PASSWORD | /usr/share/elasticsearch/bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password -xf; \ 56 | echo $PASSWORD | /usr/share/elasticsearch/bin/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password -xf; \ 57 | echo $PASSWORD | /usr/share/elasticsearch/bin/elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password -xf ; \ 58 | echo $PASSWORD | /usr/share/elasticsearch/bin/elasticsearch-keystore add xpack.security.http.ssl.truststore.secure_password -xf ; \ 59 | echo $PASSWORD | /usr/share/elasticsearch/bin/elasticsearch-keystore add bootstrap.password -xf; \ 60 | sh -c /usr/local/bin/docker-entrypoint.sh eswrapper;"] 61 | env: 62 | - name: cluster.name 63 | value: "sample-elasticsearch-cluster" 64 | - name: "xpack.security.enabled" 65 | value: "true" 66 | - name: "TAKE_FILE_OWNERSHIP" 67 | value: “1” 68 | - name: PASSWORD 69 | value: Password1$ 70 | - name: node.name 71 | valueFrom: 72 | fieldRef: 73 | fieldPath: metadata.name 74 | - name: discovery.zen.ping.unicast.hosts 75 | value: "sample-elasticsearch-discovery:9300" 76 | volumeMounts: 77 | - name: sample-elasticsearch-data-claim 78 | mountPath: /usr/share/elasticsearch/data 79 | - name: sample-elasticsearch-configmap 80 | mountPath: /usr/share/elasticsearch/config/elasticsearch.yml 81 | subPath: elasticsearch.yml 82 | - name: sample-elasticsearch-configmap 83 | mountPath: /usr/share/elasticsearch/config/role_mapping.yml 84 | subPath: role_mapping.yml 85 | - name: sample-elasticsearch-elasticsearchsecret 86 | mountPath: /usr/share/elasticsearch/config/elastic-stack-ca.p12 87 | subPath: elastic-stack-ca.p12 88 | - name: sample-elasticsearch-elasticsearchsecret 89 | mountPath: /usr/share/elasticsearch/config/elastic-certificates.p12 90 | subPath: elastic-certificates.p12 91 | - name: sample-elasticsearch-elasticsearchsecret 92 | mountPath: /usr/share/elasticsearch/config/Elasticsearch.xml 93 | subPath: Elasticsearch.xml 94 | volumes: 95 | - name: sample-elasticsearch-configmap 96 | configMap: 97 | name: sample-elasticsearch-configmap 98 | - name: sample-elasticsearch-elasticsearchsecret 99 | secret: 100 | secretName: elasticsearchsecret 101 | volumeClaimTemplates: 102 | - metadata: 103 | name: sample-elasticsearch-data-claim 104 | spec: 105 | accessModes: [ "ReadWriteOnce" ] 106 | storageClassName: default 107 | resources: 108 | requests: 109 | storage: 2Gi 110 | -------------------------------------------------------------------------------- /Azure AD SAML SSO/Kibana/Dockerfile: -------------------------------------------------------------------------------- 1 | # KIBANA 2 | FROM docker.elastic.co/kibana/kibana:6.4.1 3 | COPY instance.crt /usr/share/kibana/config/ 4 | COPY instance.key /usr/share/kibana/config/ 5 | COPY elastic-stack-ca.pem /usr/share/kibana/config/ -------------------------------------------------------------------------------- /Azure AD SAML SSO/Kibana/Kibana_ConfigMap.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: ConfigMap 3 | metadata: 4 | name: sample-kibana-configmap 5 | namespace: default 6 | data: 7 | kibana.yml: | 8 | server.name: sample-kibana 9 | server.host: "0.0.0.0" 10 | elasticsearch.url: https://sample-elasticsearch:9200 11 | xpack.monitoring.ui.container.elasticsearch.enabled: true 12 | elasticsearch.username: kibana 13 | elasticsearch.password: Password1$ 14 | xpack.security.encryptionKey: "something_at_least_32_characters" 15 | elasticsearch.ssl.certificateAuthorities: "/usr/share/kibana/config/elastic-stack-ca.pem" 16 | server.ssl.enabled: true 17 | server.ssl.key: "/usr/share/kibana/config/instance.key" 18 | server.ssl.certificate: "/usr/share/kibana/config/instance.crt" 19 | xpack.monitoring.elasticsearch.ssl.verificationMode: certificate 20 | elasticsearch.ssl.verificationMode: certificate 21 | xpack.security.authProviders: [saml, basic] 22 | server.xsrf.whitelist: [/api/security/v1/saml] 23 | xpack.security.public: 24 | protocol: https 25 | hostname: KIBANA_URL 26 | -------------------------------------------------------------------------------- /Azure AD SAML SSO/Kibana/Kibana_Deployment.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: apps/v1beta1 2 | kind: Deployment 3 | metadata: 4 | name: sample-kibana 5 | labels: 6 | component: sample-kibana 7 | spec: 8 | replicas: 1 9 | selector: 10 | matchLabels: 11 | component: sample-kibana 12 | template: 13 | metadata: 14 | labels: 15 | component: sample-kibana 16 | spec: 17 | containers: 18 | - name: sample-kibana 19 | image: YOUR_KIBANA_IMAGE 20 | resources: 21 | limits: 22 | cpu: 1000m 23 | requests: 24 | cpu: 100m 25 | ports: 26 | - containerPort: 5601 27 | name: http 28 | volumeMounts: 29 | - name: sample-kibana-configmap 30 | mountPath: /usr/share/kibana/config/kibana.yml 31 | subPath: kibana.yml 32 | resources: {} 33 | terminationMessagePath: "/dev/termination-log" 34 | terminationMessagePolicy: File 35 | imagePullPolicy: Always 36 | securityContext: 37 | privileged: false 38 | volumes: 39 | - name: sample-kibana-configmap 40 | configMap: 41 | name: sample-kibana-configmap 42 | restartPolicy: Always 43 | terminationGracePeriodSeconds: 5 44 | dnsPolicy: ClusterFirst 45 | securityContext: 46 | privileged: true 47 | schedulerName: default-scheduler -------------------------------------------------------------------------------- /Azure AD SAML SSO/Kibana/Kibana_Deployment_V2.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: apps/v1beta1 2 | kind: Deployment 3 | metadata: 4 | name: sample-kibana 5 | labels: 6 | component: sample-kibana 7 | spec: 8 | replicas: 1 9 | selector: 10 | matchLabels: 11 | component: sample-kibana 12 | template: 13 | metadata: 14 | labels: 15 | component: sample-kibana 16 | spec: 17 | containers: 18 | - name: sample-kibana 19 | image: docker.elastic.co/kibana/kibana:6.4.1 20 | resources: 21 | limits: 22 | cpu: 1000m 23 | requests: 24 | cpu: 100m 25 | ports: 26 | - containerPort: 5601 27 | name: http 28 | volumeMounts: 29 | - name: sample-kibana-configmap 30 | mountPath: /usr/share/kibana/config/kibana.yml 31 | subPath: kibana.yml 32 | - name: sample-kibana-kibanasecret 33 | mountPath: /usr/share/kibana/config/elastic-stack-ca.pem 34 | subPath: elastic-stack-ca.pem 35 | - name: sample-kibana-kibanasecret 36 | mountPath: /usr/share/kibana/config/instance.crt 37 | subPath: instance.crt 38 | - name: sample-kibana-kibanasecret 39 | mountPath: /usr/share/kibana/config/instance.key 40 | subPath: instance.key 41 | resources: {} 42 | terminationMessagePath: "/dev/termination-log" 43 | terminationMessagePolicy: File 44 | imagePullPolicy: Always 45 | securityContext: 46 | privileged: false 47 | volumes: 48 | - name: sample-kibana-configmap 49 | configMap: 50 | name: sample-kibana-configmap 51 | - name: sample-kibana-kibanasecret 52 | secret: 53 | secretName: kibanasecret 54 | restartPolicy: Always 55 | terminationGracePeriodSeconds: 5 56 | dnsPolicy: ClusterFirst 57 | securityContext: 58 | privileged: true 59 | schedulerName: default-scheduler -------------------------------------------------------------------------------- /Azure AD SAML SSO/Kibana/Kibana_Service.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Service 3 | metadata: 4 | name: sample-kibana 5 | labels: 6 | component: sample-kibana 7 | spec: 8 | type: LoadBalancer 9 | selector: 10 | component: sample-kibana 11 | ports: 12 | - name: http 13 | port: 5601 14 | targetPort: http -------------------------------------------------------------------------------- /Azure AD SAML SSO/Logstash/Dockerfile: -------------------------------------------------------------------------------- 1 | #LOGSTASH 2 | FROM docker.elastic.co/logstash/logstash:6.4.1 3 | COPY elastic-stack-ca.pem /usr/share/logstash/config/ -------------------------------------------------------------------------------- /Azure AD SAML SSO/Logstash/Logstash_ConfigMap.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: ConfigMap 3 | metadata: 4 | name: sample-logstash-configmap 5 | namespace: default 6 | data: 7 | 8 | logstash.yml: | 9 | xpack.monitoring.elasticsearch.url: https://sample-elasticsearch:9200 10 | dead_letter_queue.enable: true 11 | xpack.monitoring.enabled: true 12 | xpack.monitoring.elasticsearch.username: logstash_system 13 | xpack.monitoring.elasticsearch.password: Password1$ 14 | xpack.monitoring.elasticsearch.ssl.ca: "/usr/share/logstash/config/elastic-stack-ca.pem" 15 | 16 | pipelines.yml: | 17 | - pipeline.id: azureeventhubs 18 | path.config: "/usr/share/logstash/azureeventhubs.cfg" 19 | - pipeline.id: azureredis 20 | path.config: "/usr/share/logstash/azureredis.cfg" 21 | 22 | azureeventhubs.cfg: | 23 | input { 24 | azure_event_hubs { 25 | event_hub_connections => ["{AZURE_EVENT_HUB_CONNECTION_STRING};EntityPath=logstash"] 26 | threads => 2 27 | decorate_events => true 28 | consumer_group => "$Default" 29 | storage_connection => "{STORAGE_ACCOUNT_CONNECTION_STRING}" 30 | storage_container => "logstash" 31 | } 32 | } 33 | filter { 34 | } 35 | 36 | output { 37 | elasticsearch { 38 | hosts => ["sample-elasticsearch:9200" ] 39 | user => "elastic" 40 | password => "Password1$" 41 | index => "azureeventhub-%{+YYYY.MM.dd}" 42 | ssl => true 43 | cacert => "/usr/share/logstash/config/elastic-stack-ca.pem" 44 | } 45 | } 46 | 47 | azureredis.cfg: | 48 | input { 49 | redis { 50 | host => "{YOUR_REDIS_HOST_NAME}" 51 | key => "messages" 52 | data_type => "channel" 53 | password => "{YOUR_REDIS_SECONDARY_KEY}" 54 | port => 6380 55 | ssl => true 56 | } 57 | } 58 | 59 | filter { 60 | } 61 | 62 | output { 63 | elasticsearch { 64 | hosts => ["sample-elasticsearch:9200" ] 65 | user => "elastic" 66 | password => "Password1$" 67 | index => "azureredis-%{+YYYY.MM.dd}" 68 | ssl => true 69 | cacert => "/usr/share/logstash/config/elastic-stack-ca.pem" 70 | } 71 | } 72 | 73 | logstash.conf: | -------------------------------------------------------------------------------- /Azure AD SAML SSO/Logstash/Logstash_Deployment.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: apps/v1beta1 2 | kind: Deployment 3 | metadata: 4 | name: sample-logstash 5 | labels: 6 | component: sample-logstash 7 | spec: 8 | replicas: 1 9 | selector: 10 | matchLabels: 11 | component: sample-logstash 12 | template: 13 | metadata: 14 | labels: 15 | component: sample-logstash 16 | spec: 17 | containers: 18 | - name: sample-logstash 19 | image: YOUR_LOGSTASH_IMAGE 20 | volumeMounts: 21 | - name: sample-logstash-configmap 22 | mountPath: /usr/share/logstash/config/logstash.yml 23 | subPath: logstash.yml 24 | - name: sample-logstash-configmap 25 | mountPath: /usr/share/logstash/pipeline/logstash.conf 26 | subPath: logstash.conf 27 | - name: sample-logstash-configmap 28 | mountPath: /usr/share/logstash/azureeventhubs.cfg 29 | subPath: azureeventhubs.cfg 30 | - name: sample-logstash-configmap 31 | mountPath: /usr/share/logstash/azureredis.cfg 32 | subPath: azureredis.cfg 33 | - name: sample-logstash-configmap 34 | mountPath: /usr/share/logstash/config/pipelines.yml 35 | subPath: pipelines.yml 36 | resources: 37 | limits: 38 | cpu: 1000m 39 | requests: 40 | cpu: 100m 41 | ports: 42 | - containerPort: 5601 43 | name: http 44 | volumes: 45 | - name: sample-logstash-configmap 46 | configMap: 47 | name: sample-logstash-configmap -------------------------------------------------------------------------------- /Azure AD SAML SSO/Logstash/Logstash_Deployment_V2.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: apps/v1beta1 2 | kind: Deployment 3 | metadata: 4 | name: sample-logstash 5 | labels: 6 | component: sample-logstash 7 | spec: 8 | replicas: 1 9 | selector: 10 | matchLabels: 11 | component: sample-logstash 12 | template: 13 | metadata: 14 | labels: 15 | component: sample-logstash 16 | spec: 17 | containers: 18 | - name: sample-logstash 19 | image: docker.elastic.co/logstash/logstash:6.4.1 20 | volumeMounts: 21 | - name: sample-logstash-configmap 22 | mountPath: /usr/share/logstash/config/logstash.yml 23 | subPath: logstash.yml 24 | - name: sample-logstash-configmap 25 | mountPath: /usr/share/logstash/pipeline/logstash.conf 26 | subPath: logstash.conf 27 | - name: sample-logstash-configmap 28 | mountPath: /usr/share/logstash/azureeventhubs.cfg 29 | subPath: azureeventhubs.cfg 30 | - name: sample-logstash-configmap 31 | mountPath: /usr/share/logstash/azureredis.cfg 32 | subPath: azureredis.cfg 33 | - name: sample-logstash-configmap 34 | mountPath: /usr/share/logstash/config/pipelines.yml 35 | subPath: pipelines.yml 36 | - name: sample-logstash-logstashsecret 37 | mountPath: /usr/share/logstash/config/elastic-stack-ca.pem 38 | subPath: elastic-stack-ca.pem 39 | resources: 40 | limits: 41 | cpu: 1000m 42 | requests: 43 | cpu: 100m 44 | ports: 45 | - containerPort: 5601 46 | name: http 47 | volumes: 48 | - name: sample-logstash-configmap 49 | configMap: 50 | name: sample-logstash-configmap 51 | - name: sample-logstash-logstashsecret 52 | secret: 53 | secretName: logstashsecret -------------------------------------------------------------------------------- /Azure AD SAML SSO/Logstash/Logstash_Service.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Service 3 | metadata: 4 | name: sample-logstash 5 | labels: 6 | component: sample-logstash 7 | spec: 8 | type: ClusterIP 9 | selector: 10 | component: sample-logstash 11 | ports: 12 | - name: http 13 | port: 80 14 | targetPort: http -------------------------------------------------------------------------------- /Azure AD SAML SSO/README.md: -------------------------------------------------------------------------------- 1 | This is code samples for second part of the series on deploying Elasticsearch, Logstash and Kibana (ELK) to Azure Kubernetes Service cluster. These samples are created to enable Azure AD SAML based single sign on to secure Elasticsearch and Kibana cluster hosted in AKS. I will also cover the steps needed to encrypt communications in ELK cluster. 2 | 3 | 4 | Using SAML SSO for Elasticsearch with AAD means that Elasticsearch does not need to be seeded with any user accounts from the directory. Instead, Elasticsearch is able to rely on the claims sent within a SAML token in response to successful authentication to determine identity and privileges. 5 | 6 | Kibana, as the user facing component, interacts with the user’s browser and receives all the SAML messages that the Identity Provider sends to the Elastic Stack Service Provider. Elasticsearch implements most of the functionality a SAML Service Provider needs. It holds all SAML related configuration in the form of an authentication realm and it also generates all SAML messages required and passes them to Kibana to be relayed to the user’s browser. It finally consumes all SAML Responses that Kibana relays to it, verifies them, extracts the necessary authentication information and creates the internal authentication tokens based on that. The component diagram has been updated to add Azure AD integration. 7 | 8 | You can read more at msdn 9 | 10 | -------------------------------------------------------------------------------- /AzureRedisCacheSample/AzureRedisCacheSample.csproj: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | Exe 5 | netcoreapp2.1 6 | 7 | 8 | 9 | 10 | 11 | 12 | -------------------------------------------------------------------------------- /AzureRedisCacheSample/Program.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using StackExchange.Redis; 3 | 4 | namespace AzureRedisCacheSample 5 | { 6 | class Program 7 | { 8 | static string redisConnectionString = "YOUR_STACKEXCHANGE.REDIS_CONNECTION_STRING"; 9 | static string channel = "messages"; 10 | 11 | static void Main(string[] args) 12 | { 13 | Console.WriteLine("Starting Redis Publisher!"); 14 | 15 | ConnectionMultiplexer redis = ConnectionMultiplexer.Connect(redisConnectionString); 16 | ISubscriber sub = redis.GetSubscriber(); 17 | 18 | sub.Subscribe(channel, (channelName, msg) => 19 | { 20 | Console.WriteLine($"Local receiver: { (string)msg}"); 21 | }); 22 | 23 | string message = string.Empty; 24 | 25 | do 26 | { 27 | Console.WriteLine("Enter message to send to Channel."); 28 | message = Console.ReadLine(); 29 | 30 | if (string.IsNullOrWhiteSpace(message)) 31 | { 32 | Console.WriteLine("Stopping ..."); 33 | break; 34 | } 35 | 36 | sub.Publish(channel, message); 37 | } 38 | while (!string.IsNullOrWhiteSpace(message)); 39 | 40 | Console.WriteLine("Publisher stopped."); 41 | } 42 | } 43 | } -------------------------------------------------------------------------------- /ELKSamples.sln: -------------------------------------------------------------------------------- 1 | 2 | Microsoft Visual Studio Solution File, Format Version 12.00 3 | # Visual Studio 2012 4 | Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Solution Items", "Solution Items", "{599FF4D8-F69B-4D63-982C-BFD3FEFF347B}" 5 | EndProject 6 | Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "ElasticSearch", "ElasticSearch", "{5F838F70-3D22-438B-B937-3D6E88C8DE35}" 7 | ProjectSection(SolutionItems) = preProject 8 | ES_ConfigMap.yaml = ES_ConfigMap.yaml 9 | ES_PersistentVolumeClaim.yaml = ES_PersistentVolumeClaim.yaml 10 | ES_Service.yaml = ES_Service.yaml 11 | ES_StatefulSet.yaml = ES_StatefulSet.yaml 12 | EndProjectSection 13 | EndProject 14 | Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Kibana", "Kibana", "{5EAEE26B-ED40-4556-B9C1-085864535DFC}" 15 | ProjectSection(SolutionItems) = preProject 16 | Kibana_ConfigMap.yaml = Kibana_ConfigMap.yaml 17 | Kibana_Deployment.yaml = Kibana_Deployment.yaml 18 | Kibana_Service.yaml = Kibana_Service.yaml 19 | EndProjectSection 20 | EndProject 21 | Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Logstash", "Logstash", "{196B6317-B762-4D69-A885-AC8EE76C1B69}" 22 | ProjectSection(SolutionItems) = preProject 23 | Logstash_Deployment.yaml = Logstash_Deployment.yaml 24 | Logstash_Service.yaml = Logstash_Service.yaml 25 | Logstash_ConfigMap.yaml = Logstash_ConfigMap.yaml 26 | EndProjectSection 27 | EndProject 28 | Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "AzureEventHubSample", "ELKSamples\AzureEventHubSample\AzureEventHubSample.csproj", "{FDAD8723-DD8E-4292-9A68-02098BAF0A33}" 29 | EndProject 30 | Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "ELK with Azure AD SAML SSO Integration", "ELK with Azure AD SAML SSO Integration", "{A10E3350-F4EB-4573-A72A-EF42B6C705AF}" 31 | EndProject 32 | Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Elasticsearch", "Elasticsearch", "{A70590F8-4C48-4F54-AAEE-3BA8B55C009A}" 33 | ProjectSection(SolutionItems) = preProject 34 | Azure AD SAML SSO\Elasticsearch\Dockerfile = Azure AD SAML SSO\Elasticsearch\Dockerfile 35 | Azure AD SAML SSO\Elasticsearch\ES_ConfigMap.yaml = Azure AD SAML SSO\Elasticsearch\ES_ConfigMap.yaml 36 | Azure AD SAML SSO\Elasticsearch\ES_Service.yaml = Azure AD SAML SSO\Elasticsearch\ES_Service.yaml 37 | Azure AD SAML SSO\Elasticsearch\ES_StatefulSet.yaml = Azure AD SAML SSO\Elasticsearch\ES_StatefulSet.yaml 38 | Azure AD SAML SSO\Elasticsearch\elastic-stack-ca.p12 = Azure AD SAML SSO\Elasticsearch\elastic-stack-ca.p12 39 | Azure AD SAML SSO\Elasticsearch\elastic-certificates.p12 = Azure AD SAML SSO\Elasticsearch\elastic-certificates.p12 40 | Azure AD SAML SSO\Elasticsearch\Elasticsearch.xml = Azure AD SAML SSO\Elasticsearch\Elasticsearch.xml 41 | Azure AD SAML SSO\Elasticsearch\ES_PersistentVolumeClaim.yaml = Azure AD SAML SSO\Elasticsearch\ES_PersistentVolumeClaim.yaml 42 | Azure AD SAML SSO\Elasticsearch\ES_StatefulSet_V2.yaml = Azure AD SAML SSO\Elasticsearch\ES_StatefulSet_V2.yaml 43 | EndProjectSection 44 | EndProject 45 | Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Kibana", "Kibana", "{3700D424-4D70-4801-9790-06148FD6C062}" 46 | ProjectSection(SolutionItems) = preProject 47 | Azure AD SAML SSO\Kibana\Dockerfile = Azure AD SAML SSO\Kibana\Dockerfile 48 | Azure AD SAML SSO\Kibana\Kibana_ConfigMap.yaml = Azure AD SAML SSO\Kibana\Kibana_ConfigMap.yaml 49 | Azure AD SAML SSO\Kibana\Kibana_Service.yaml = Azure AD SAML SSO\Kibana\Kibana_Service.yaml 50 | Azure AD SAML SSO\Kibana\Kibana_Deployment.yaml = Azure AD SAML SSO\Kibana\Kibana_Deployment.yaml 51 | Azure AD SAML SSO\Kibana\instance.crt = Azure AD SAML SSO\Kibana\instance.crt 52 | Azure AD SAML SSO\Kibana\instance.key = Azure AD SAML SSO\Kibana\instance.key 53 | Azure AD SAML SSO\Kibana\elastic-stack-ca.pem = Azure AD SAML SSO\Kibana\elastic-stack-ca.pem 54 | Azure AD SAML SSO\Kibana\Kibana_Deployment_V2.yaml = Azure AD SAML SSO\Kibana\Kibana_Deployment_V2.yaml 55 | EndProjectSection 56 | EndProject 57 | Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Logstash", "Logstash", "{03EEF0F4-BDFF-412A-8E48-C31C20AF5F1E}" 58 | ProjectSection(SolutionItems) = preProject 59 | Azure AD SAML SSO\Logstash\Dockerfile = Azure AD SAML SSO\Logstash\Dockerfile 60 | Azure AD SAML SSO\Logstash\Logstash_ConfigMap.yaml = Azure AD SAML SSO\Logstash\Logstash_ConfigMap.yaml 61 | Azure AD SAML SSO\Logstash\Logstash_Deployment.yaml = Azure AD SAML SSO\Logstash\Logstash_Deployment.yaml 62 | Azure AD SAML SSO\Logstash\Logstash_Service.yaml = Azure AD SAML SSO\Logstash\Logstash_Service.yaml 63 | Azure AD SAML SSO\Logstash\elastic-stack-ca.pem = Azure AD SAML SSO\Logstash\elastic-stack-ca.pem 64 | Azure AD SAML SSO\Logstash\Logstash_Deployment_V2.yaml = Azure AD SAML SSO\Logstash\Logstash_Deployment_V2.yaml 65 | EndProjectSection 66 | EndProject 67 | Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "AzureRedisCacheSample", "AzureRedisCacheSample\AzureRedisCacheSample.csproj", "{301E567B-27A8-4A10-B9E6-0931B1416875}" 68 | EndProject 69 | Global 70 | GlobalSection(SolutionConfigurationPlatforms) = preSolution 71 | Debug|Any CPU = Debug|Any CPU 72 | Release|Any CPU = Release|Any CPU 73 | EndGlobalSection 74 | GlobalSection(ProjectConfigurationPlatforms) = postSolution 75 | {FDAD8723-DD8E-4292-9A68-02098BAF0A33}.Debug|Any CPU.ActiveCfg = Debug|Any CPU 76 | {FDAD8723-DD8E-4292-9A68-02098BAF0A33}.Debug|Any CPU.Build.0 = Debug|Any CPU 77 | {FDAD8723-DD8E-4292-9A68-02098BAF0A33}.Release|Any CPU.ActiveCfg = Release|Any CPU 78 | {FDAD8723-DD8E-4292-9A68-02098BAF0A33}.Release|Any CPU.Build.0 = Release|Any CPU 79 | {301E567B-27A8-4A10-B9E6-0931B1416875}.Debug|Any CPU.ActiveCfg = Debug|Any CPU 80 | {301E567B-27A8-4A10-B9E6-0931B1416875}.Debug|Any CPU.Build.0 = Debug|Any CPU 81 | {301E567B-27A8-4A10-B9E6-0931B1416875}.Release|Any CPU.ActiveCfg = Release|Any CPU 82 | {301E567B-27A8-4A10-B9E6-0931B1416875}.Release|Any CPU.Build.0 = Release|Any CPU 83 | EndGlobalSection 84 | GlobalSection(NestedProjects) = preSolution 85 | {5F838F70-3D22-438B-B937-3D6E88C8DE35} = {599FF4D8-F69B-4D63-982C-BFD3FEFF347B} 86 | {5EAEE26B-ED40-4556-B9C1-085864535DFC} = {599FF4D8-F69B-4D63-982C-BFD3FEFF347B} 87 | {196B6317-B762-4D69-A885-AC8EE76C1B69} = {599FF4D8-F69B-4D63-982C-BFD3FEFF347B} 88 | {A10E3350-F4EB-4573-A72A-EF42B6C705AF} = {599FF4D8-F69B-4D63-982C-BFD3FEFF347B} 89 | {A70590F8-4C48-4F54-AAEE-3BA8B55C009A} = {A10E3350-F4EB-4573-A72A-EF42B6C705AF} 90 | {3700D424-4D70-4801-9790-06148FD6C062} = {A10E3350-F4EB-4573-A72A-EF42B6C705AF} 91 | {03EEF0F4-BDFF-412A-8E48-C31C20AF5F1E} = {A10E3350-F4EB-4573-A72A-EF42B6C705AF} 92 | EndGlobalSection 93 | EndGlobal 94 | -------------------------------------------------------------------------------- /ELKSamples/AzureEventHubSample/AzureEventHubSample.csproj: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | Exe 5 | netcoreapp2.1 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | -------------------------------------------------------------------------------- /ELKSamples/AzureEventHubSample/Program.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using System.Text; 3 | using System.Threading; 4 | using System.Threading.Tasks; 5 | using Microsoft.Azure.EventHubs; 6 | using Newtonsoft.Json; 7 | 8 | namespace EventHubSample 9 | { 10 | class Program 11 | { 12 | // Update connection string : "AZURE_EVENT_HUB_CONNECTION_STRING;EntityPath=EventHubName" 13 | static string connectionString = "Endpoint={AZURE_EVENT_HUB_CONNECTION_STRING};EntityPath=logstash"; 14 | static bool isRunning; 15 | 16 | static void Main(string[] args) 17 | { 18 | Console.WriteLine("Starting sending messages to Azure event hub"); 19 | isRunning = true; 20 | SendingRandomMessages(); 21 | Console.ReadLine(); 22 | isRunning = false; 23 | Console.WriteLine("Stopping sending messages. Press any key to exit!"); 24 | Console.ReadLine(); 25 | } 26 | 27 | async static Task SendingRandomMessages() 28 | { 29 | var eventHubClient = EventHubClient.CreateFromConnectionString(connectionString); 30 | Random random = new Random(1); 31 | 32 | while (isRunning) 33 | { 34 | try 35 | { 36 | var message = JsonConvert.SerializeObject(new SampleData() { ID = random.Next(), Name = $"Sample Name {random.Next()}" }); 37 | Console.WriteLine("{0} > Sending message: {1}", DateTime.Now, message); 38 | await eventHubClient.SendAsync(new EventData(Encoding.UTF8.GetBytes(message))); 39 | } 40 | catch (Exception exception) 41 | { 42 | Console.ForegroundColor = ConsoleColor.Red; 43 | Console.WriteLine("{0} > Exception: {1}", DateTime.Now, exception.Message); 44 | Console.ResetColor(); 45 | } 46 | 47 | Thread.Sleep(500); 48 | } 49 | } 50 | } 51 | } 52 | 53 | public class SampleData 54 | { 55 | public int ID { get; set; } 56 | public string Name { get; set; } 57 | } 58 | -------------------------------------------------------------------------------- /ES_ConfigMap.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: ConfigMap 3 | metadata: 4 | name: sample-elasticsearch-configmap 5 | namespace: default 6 | data: 7 | elasticsearch.yml: | 8 | cluster.name: "sample-elasticsearch-cluster" 9 | network.host: 0.0.0.0 10 | discovery.zen.minimum_master_nodes: 1 11 | xpack.license.self_generated.type: trial 12 | node.max_local_storage_nodes: 1 13 | xpack.security.enabled: true 14 | xpack.monitoring.collection.enabled: true 15 | 16 | role_mapping.yml: | 17 | -------------------------------------------------------------------------------- /ES_PersistentVolumeClaim.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: PersistentVolumeClaim 3 | metadata: 4 | name: sample-elasticsearch-data-claim 5 | spec: 6 | accessModes: 7 | - ReadWriteOnce 8 | resources: 9 | requests: 10 | storage: 5Gi -------------------------------------------------------------------------------- /ES_Service.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Service 3 | metadata: 4 | name: sample-elasticsearch 5 | labels: 6 | service: sample-elasticsearch 7 | spec: 8 | type: ClusterIP 9 | selector: 10 | service: sample-elasticsearch 11 | ports: 12 | - name: http 13 | port: 9200 14 | targetPort: 9200 15 | protocol: TCP 16 | - name: transport 17 | port: 9300 18 | targetPort: 9300 19 | protocol: TCP 20 | 21 | --- 22 | 23 | apiVersion: v1 24 | kind: Service 25 | metadata: 26 | name: sample-elasticsearch-discovery 27 | spec: 28 | clusterIP: None 29 | selector: 30 | service: sample-elasticsearch 31 | ports: 32 | - name: transport 33 | port: 9300 34 | -------------------------------------------------------------------------------- /ES_StatefulSet.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: apps/v1 2 | kind: StatefulSet 3 | metadata: 4 | name: sample-elasticsearch 5 | labels: 6 | service: sample-elasticsearch 7 | spec: 8 | serviceName: sample-elasticsearch-discovery 9 | # Number of PODS 10 | replicas: 1 11 | selector: 12 | matchLabels: 13 | service: sample-elasticsearch 14 | template: 15 | metadata: 16 | labels: 17 | service: sample-elasticsearch 18 | spec: 19 | terminationGracePeriodSeconds: 15 20 | initContainers: 21 | # https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html 22 | - name: increase-the-vm-max-map-count 23 | image: busybox 24 | command: 25 | - sysctl 26 | - -w 27 | - vm.max_map_count=262144 28 | securityContext: 29 | privileged: true 30 | - name: fix-the-volume-permission 31 | image: busybox 32 | command: ["sh", "-c", "chown -R 1000:1000 /usr/share/elasticsearch/data"] 33 | securityContext: 34 | privileged: true 35 | volumeMounts: 36 | - name: sample-elasticsearch-data-claim 37 | mountPath: /usr/share/elasticsearch/data 38 | containers: 39 | - name: sample-elasticsearch 40 | image: docker.elastic.co/elasticsearch/elasticsearch:6.4.1 41 | resources: 42 | limits: 43 | cpu: 2000m 44 | memory: 2Gi 45 | requests: 46 | cpu: 100m 47 | memory: 1Gi 48 | ports: 49 | - containerPort: 9200 50 | name: http 51 | - containerPort: 9300 52 | name: tcp 53 | env: 54 | - name: cluster.name 55 | value: "sample-elasticsearch-cluster" 56 | - name: "xpack.security.enabled" 57 | value: "true" 58 | - name: "TAKE_FILE_OWNERSHIP" 59 | value: “1” 60 | - name: node.name 61 | valueFrom: 62 | fieldRef: 63 | fieldPath: metadata.name 64 | - name: discovery.zen.ping.unicast.hosts 65 | value: "sample-elasticsearch-discovery:9300" 66 | volumeMounts: 67 | - name: sample-elasticsearch-data-claim 68 | mountPath: /usr/share/elasticsearch/data 69 | - name: sample-elasticsearch-configmap 70 | mountPath: /usr/share/elasticsearch/config/elasticsearch.yml 71 | subPath: elasticsearch.yml 72 | - name: sample-elasticsearch-configmap 73 | mountPath: /usr/share/elasticsearch/config/role_mapping.yml 74 | subPath: role_mapping.yml 75 | volumes: 76 | - name: sample-elasticsearch-configmap 77 | configMap: 78 | name: sample-elasticsearch-configmap 79 | volumeClaimTemplates: 80 | - metadata: 81 | name: sample-elasticsearch-data-claim 82 | spec: 83 | accessModes: [ "ReadWriteOnce" ] 84 | storageClassName: default 85 | resources: 86 | requests: 87 | storage: 2Gi 88 | -------------------------------------------------------------------------------- /Kibana_ConfigMap.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: ConfigMap 3 | metadata: 4 | name: sample-kibana-configmap 5 | namespace: default 6 | data: 7 | kibana.yml: | 8 | server.name: sample-kibana 9 | server.host: "0" 10 | elasticsearch.url: http://sample-elasticsearch:9200 11 | xpack.monitoring.ui.container.elasticsearch.enabled: true 12 | elasticsearch.username: kibana 13 | elasticsearch.password: Password1$ -------------------------------------------------------------------------------- /Kibana_Deployment.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: apps/v1beta1 2 | kind: Deployment 3 | metadata: 4 | name: sample-kibana 5 | labels: 6 | component: sample-kibana 7 | spec: 8 | replicas: 1 9 | selector: 10 | matchLabels: 11 | component: sample-kibana 12 | template: 13 | metadata: 14 | labels: 15 | component: sample-kibana 16 | spec: 17 | containers: 18 | - name: sample-kibana 19 | image: docker.elastic.co/kibana/kibana:6.4.1 20 | resources: 21 | limits: 22 | cpu: 1000m 23 | requests: 24 | cpu: 100m 25 | ports: 26 | - containerPort: 5601 27 | name: http 28 | volumeMounts: 29 | - name: sample-kibana-configmap 30 | mountPath: /usr/share/kibana/config/kibana.yml 31 | subPath: kibana.yml 32 | resources: {} 33 | terminationMessagePath: "/dev/termination-log" 34 | terminationMessagePolicy: File 35 | imagePullPolicy: Always 36 | securityContext: 37 | privileged: false 38 | volumes: 39 | - name: sample-kibana-configmap 40 | configMap: 41 | name: sample-kibana-configmap 42 | restartPolicy: Always 43 | terminationGracePeriodSeconds: 5 44 | dnsPolicy: ClusterFirst 45 | securityContext: {} 46 | schedulerName: default-scheduler -------------------------------------------------------------------------------- /Kibana_Service.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Service 3 | metadata: 4 | name: sample-kibana 5 | labels: 6 | component: sample-kibana 7 | spec: 8 | type: LoadBalancer 9 | selector: 10 | component: sample-kibana 11 | ports: 12 | - name: http 13 | port: 80 14 | targetPort: http -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2018 Atul Verma 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /Logstash_ConfigMap.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: ConfigMap 3 | metadata: 4 | name: sample-logstash-configmap 5 | namespace: default 6 | data: 7 | 8 | logstash.yml: | 9 | xpack.monitoring.elasticsearch.url: http://sample-elasticsearch:9200 10 | dead_letter_queue.enable: true 11 | xpack.monitoring.enabled: true 12 | xpack.monitoring.elasticsearch.username: logstash_system 13 | xpack.monitoring.elasticsearch.password: Password1$ 14 | 15 | pipelines.yml: | 16 | - pipeline.id: azureeventhubs 17 | path.config: "/usr/share/logstash/azureeventhubs.cfg" 18 | 19 | azureeventhubs.cfg: | 20 | input { 21 | azure_event_hubs { 22 | event_hub_connections => ["{AZURE_EVENT_HUB_CONNECTION_STRING};EntityPath=logstash"] 23 | threads => 2 24 | decorate_events => true 25 | consumer_group => "$Default" 26 | storage_connection => "{STORAGE_ACCOUNT_CONNECTION_STRING}" 27 | storage_container => "logstash" 28 | } 29 | } 30 | filter { 31 | } 32 | output { 33 | elasticsearch { 34 | hosts => [ "sample-elasticsearch:9200" ] 35 | user => "elastic" 36 | password => "Password1$" 37 | index => "azureeventhub-%{+YYYY.MM.dd}" 38 | } 39 | } 40 | 41 | logstash.conf: | -------------------------------------------------------------------------------- /Logstash_Deployment.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: apps/v1beta1 2 | kind: Deployment 3 | metadata: 4 | name: sample-logstash 5 | labels: 6 | component: sample-logstash 7 | spec: 8 | replicas: 1 9 | selector: 10 | matchLabels: 11 | component: sample-logstash 12 | template: 13 | metadata: 14 | labels: 15 | component: sample-logstash 16 | spec: 17 | containers: 18 | - name: sample-logstash 19 | image: docker.elastic.co/logstash/logstash:6.4.1 20 | volumeMounts: 21 | - name: sample-logstash-configmap 22 | mountPath: /usr/share/logstash/config/logstash.yml 23 | subPath: logstash.yml 24 | - name: sample-logstash-configmap 25 | mountPath: /usr/share/logstash/pipeline/logstash.conf 26 | subPath: logstash.conf 27 | - name: sample-logstash-configmap 28 | mountPath: /usr/share/logstash/azureeventhubs.cfg 29 | subPath: azureeventhubs.cfg 30 | - name: sample-logstash-configmap 31 | mountPath: /usr/share/logstash/config/pipelines.yml 32 | subPath: pipelines.yml 33 | resources: 34 | limits: 35 | cpu: 1000m 36 | requests: 37 | cpu: 100m 38 | ports: 39 | - containerPort: 5601 40 | name: http 41 | volumes: 42 | - name: sample-logstash-configmap 43 | configMap: 44 | name: sample-logstash-configmap -------------------------------------------------------------------------------- /Logstash_Service.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Service 3 | metadata: 4 | name: sample-logstash 5 | labels: 6 | component: sample-logstash 7 | spec: 8 | type: ClusterIP 9 | selector: 10 | component: sample-logstash 11 | ports: 12 | - name: http 13 | port: 80 14 | targetPort: http -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # AzureKubernetesService-ELK 2 | 3 | # PART 1: 4 | 5 | This is the sample source code for setting up Elasticsearch, Logstash and Kibana cluster in Azure Kubernetes Service (AKS) and consume messages from Event Hub. 6 | 7 | A sample client App (e.g. IOT device) will be publishing messages to Event Hub and these messages will be ingested into Elasticsearch using 'Azure Event Hub' plugin of Logstash. The dev tools used to develop these components are Visual Studio for Mac/Visual Studio 2017, AKS Dashboard as well as kubectl commands are used to create/manager Kubernetes resources in AKS. 8 | 9 | You can read details @ https://www.appdevmusings.com/azure-kubernetes-service-aks-deploying-elasticsearch-logstash-and-kibana-elk-and-consume-messages-from-azure-event-hub/ 10 | 11 | # PART 2: 12 | 13 | Code samples for second part of the series on deploying Elasticsearch, Logstash and Kibana (ELK) to Azure Kubernetes Service cluster under 'AZURE AD SAML SSO' folder. These samples are created to enable Azure AD SAML based single sign on to secure Elasticsearch and Kibana cluster hosted in AKS. I will also cover the steps needed to encrypt communications in ELK cluster. 14 | 15 | Using SAML SSO for Elasticsearch with AAD means that Elasticsearch does not need to be seeded with any user accounts from the directory. Instead, Elasticsearch is able to rely on the claims sent within a SAML token in response to successful authentication to determine identity and privileges. 16 | 17 | You can read details @ https://www.appdevmusings.com/azure-kubernetes-service-aks-azure-ad-saml-based-single-sign-on-to-secure-elasticsearch-and-kibana-and-securing-communications-in-elk/ 18 | 19 | # PART 3: 20 | 21 | Azure Redis Cache is based on the popular open-source Redis cache. It is typically used as a cache to improve the performance and scalability of systems that rely heavily on backend data-stores. Logstash's Redis plugin will read events from Redis instance. I will create a Logstash event processing pipeline where I will define Redis as input and Elasticsearch as output. 22 | 23 | You can read details @ 24 | https://www.appdevmusings.com/ingest-azure-redis-cache-messages-into-elasticsearch-logstash-and-kibana-cluster-deployed-in-azure-kubernetes-service-aks/ 25 | --------------------------------------------------------------------------------