├── infra ├── modules │ ├── kured │ │ ├── outputs.tf │ │ ├── kured.tf │ │ └── inputs.tf │ ├── prometheus │ │ ├── manifest.json │ │ ├── outputs.tf │ │ ├── inputs.tf │ │ └── prometheus.tf │ ├── kibana │ │ ├── kibana.yaml.tmpl │ │ ├── kibana.yaml │ │ ├── input.tf │ │ └── kibana.tf │ ├── jaeger │ │ ├── output.tf │ │ ├── input.tf │ │ └── jaeger.tf │ ├── elasticsearch │ │ ├── outputs.tf │ │ ├── inputs.tf │ │ └── elasticsearch.tf │ ├── fluentd │ │ ├── input.tf │ │ └── fluentd.tf │ ├── istio │ │ ├── gateway.yaml │ │ ├── inputs.tf │ │ └── istio.tf │ └── grafana │ │ ├── inputs.tf │ │ ├── grafana.yaml │ │ ├── grafana.yaml.tmpl │ │ └── grafana.tf ├── environments │ ├── dev │ │ ├── grafana-dashboards.yaml │ │ ├── apply │ │ ├── destroy │ │ ├── inputs.tf │ │ ├── init │ │ └── main.tf │ └── prod │ │ ├── grafana-dashboards.yaml │ │ ├── tls │ │ ├── wildcard.domain.com.crt │ │ └── wildcard.domain.com.key │ │ ├── destroy │ │ ├── inputs.tf │ │ ├── apply │ │ ├── init │ │ └── main.tf └── stacks │ └── cloud-native │ ├── inputs.tf │ └── cloud-native.tf ├── services ├── common │ ├── common.tfvars │ └── deploy-service ├── environments │ ├── common │ │ └── common.tfvars │ └── dev │ │ ├── inputs.tf │ │ ├── apply │ │ ├── init │ │ └── main.tf └── modules │ ├── simple-service │ ├── inputs.tf │ ├── simple-service.tf │ └── deploy-simple-service │ └── services │ └── main.tf ├── cluster ├── environments │ ├── common │ │ └── common.tfvars │ ├── azure-aks │ │ ├── apply │ │ ├── init │ │ ├── inputs.tf │ │ └── main.tf │ └── azure-acs-engine │ │ ├── apply │ │ ├── inputs.tf │ │ ├── init │ │ └── main.tf └── providers │ ├── azure-aks │ ├── tiller.yaml │ ├── inputs.tf │ └── aks.tf │ └── azure-acs-engine │ ├── tiller.yaml │ ├── inputs.tf │ ├── acs-engine-template.json │ ├── acs-engine-cluster.json │ └── acs-engine.tf ├── docs └── images │ ├── grafana.png │ ├── jaeger.png │ ├── kibana.png │ └── traefik.png ├── .gitignore ├── deploy ├── tools ├── k8s-dashboard ├── kiali ├── jaeger ├── traefik ├── prom ├── grafana ├── kibana └── check-prereqs ├── Dockerfile ├── LICENSE └── README.md /infra/modules/kured/outputs.tf: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /services/common/common.tfvars: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /cluster/environments/common/common.tfvars: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /services/environments/common/common.tfvars: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /infra/environments/dev/grafana-dashboards.yaml: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /infra/environments/prod/grafana-dashboards.yaml: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /infra/modules/prometheus/manifest.json: -------------------------------------------------------------------------------- 1 | {} 2 | -------------------------------------------------------------------------------- /infra/environments/prod/tls/wildcard.domain.com.crt: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /infra/environments/prod/tls/wildcard.domain.com.key: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /infra/environments/dev/apply: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | terraform apply -auto-approve 4 | -------------------------------------------------------------------------------- /docs/images/grafana.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tikyau/bedrock/master/docs/images/grafana.png -------------------------------------------------------------------------------- /docs/images/jaeger.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tikyau/bedrock/master/docs/images/jaeger.png -------------------------------------------------------------------------------- /docs/images/kibana.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tikyau/bedrock/master/docs/images/kibana.png -------------------------------------------------------------------------------- /docs/images/traefik.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tikyau/bedrock/master/docs/images/traefik.png -------------------------------------------------------------------------------- /infra/environments/dev/destroy: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | terraform destroy -state=terraform.tfstate 4 | -------------------------------------------------------------------------------- /services/environments/dev/inputs.tf: -------------------------------------------------------------------------------- 1 | variable "container_repo" { 2 | type = "string" 3 | } 4 | -------------------------------------------------------------------------------- /infra/environments/dev/inputs.tf: -------------------------------------------------------------------------------- 1 | variable "grafana_admin_password" { 2 | type = "string" 3 | } 4 | -------------------------------------------------------------------------------- /infra/environments/prod/destroy: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | terraform destroy -state=terraform.tfstate 4 | -------------------------------------------------------------------------------- /infra/environments/prod/inputs.tf: -------------------------------------------------------------------------------- 1 | variable "grafana_admin_password" { 2 | type = "string" 3 | } 4 | -------------------------------------------------------------------------------- /infra/environments/prod/apply: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | terraform apply -var-file="../common/common.tfvars" -auto-approve -------------------------------------------------------------------------------- /services/environments/dev/apply: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | terraform apply -var-file="../common/common.tfvars" -auto-approve -------------------------------------------------------------------------------- /cluster/environments/azure-aks/apply: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | terraform apply -var-file="../common/common.tfvars" -auto-approve -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | .terraform 2 | deployments 3 | terraform.tfstate 4 | terraform.tfstate.backup 5 | translations 6 | istio-1.0.4 7 | -------------------------------------------------------------------------------- /cluster/environments/azure-acs-engine/apply: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | terraform apply -var-file="../common/common.tfvars" -auto-approve -------------------------------------------------------------------------------- /infra/modules/kibana/kibana.yaml.tmpl: -------------------------------------------------------------------------------- 1 | files: 2 | kibana.yml: 3 | elasticsearch.url: http://${elasticsearch_client_endpoint}:9200 -------------------------------------------------------------------------------- /infra/modules/jaeger/output.tf: -------------------------------------------------------------------------------- 1 | output "agent_endpoint" { 2 | value = "${var.name}-agent.${var.namespace}.svc.cluster.local" 3 | } 4 | -------------------------------------------------------------------------------- /deploy: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | cd infra 4 | if ! ./deploy $1; then 5 | echo "ERROR: deploying infra failed" 6 | exit 1 7 | fi 8 | cd .. 9 | -------------------------------------------------------------------------------- /tools/k8s-dashboard: -------------------------------------------------------------------------------- 1 | kubectl proxy & 2 | open http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/ 3 | -------------------------------------------------------------------------------- /infra/modules/kibana/kibana.yaml: -------------------------------------------------------------------------------- 1 | files: 2 | kibana.yml: 3 | elasticsearch.url: http://elasticsearch-client.elasticsearch.svc.cluster.local:9200 4 | -------------------------------------------------------------------------------- /infra/modules/prometheus/outputs.tf: -------------------------------------------------------------------------------- 1 | output "prometheus_service_endpoint" { 2 | value = "${var.name}-server.${var.namespace}.svc.cluster.local" 3 | } 4 | -------------------------------------------------------------------------------- /infra/modules/elasticsearch/outputs.tf: -------------------------------------------------------------------------------- 1 | output "elasticsearch_client_endpoint" { 2 | value = "${var.name}-client.${var.namespace}.svc.cluster.local" 3 | } 4 | -------------------------------------------------------------------------------- /services/modules/simple-service/inputs.tf: -------------------------------------------------------------------------------- 1 | variable "container_repo" { 2 | type = "string" 3 | } 4 | 5 | variable "environment" { 6 | type = "string" 7 | } 8 | -------------------------------------------------------------------------------- /infra/environments/dev/init: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | rm -rf .terraform 4 | rm terraform.tfstate 5 | rm terraform.tfstate.backup 6 | 7 | helm init 8 | terraform init 9 | -------------------------------------------------------------------------------- /infra/environments/prod/init: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | rm -rf .terraform 4 | rm terraform.tfstate 5 | rm terraform.tfstate.backup 6 | 7 | helm init 8 | terraform init -var-file="../common/common.tfvars" -------------------------------------------------------------------------------- /tools/kiali: -------------------------------------------------------------------------------- 1 | `sleep 2 && open http://localhost:20001` & 2 | kubectl port-forward -n istio-system $(kubectl get pod -n istio-system -l app=kiali -ojsonpath='{.items[0].metadata.name}') 20001:20001 3 | -------------------------------------------------------------------------------- /services/environments/dev/init: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | rm -rf .terraform 4 | rm terraform.tfstate 5 | rm terraform.tfstate.backup 6 | 7 | helm init 8 | terraform init -var-file="../common/common.tfvars" 9 | -------------------------------------------------------------------------------- /tools/jaeger: -------------------------------------------------------------------------------- 1 | export JAEGER_POD=$(kubectl get pods -l "jaeger-component=query" -o jsonpath="{.items[0].metadata.name}") 2 | `sleep 1 && open http://localhost:16686/` & 3 | kubectl port-forward $JAEGER_POD 16686 4 | -------------------------------------------------------------------------------- /services/modules/services/main.tf: -------------------------------------------------------------------------------- 1 | resource "kubernetes_namespace" "services_namespace" { 2 | metadata { 3 | name = "services" 4 | 5 | labels { 6 | istio-injection = "enabled" 7 | } 8 | } 9 | } 10 | -------------------------------------------------------------------------------- /cluster/environments/azure-acs-engine/inputs.tf: -------------------------------------------------------------------------------- 1 | variable "client_id" { 2 | type = "string" 3 | } 4 | 5 | variable "client_secret" { 6 | type = "string" 7 | } 8 | 9 | variable "ssh_public_key" { 10 | type = "string" 11 | } 12 | -------------------------------------------------------------------------------- /tools/traefik: -------------------------------------------------------------------------------- 1 | export TRAEFIK_POD=$(kubectl get pods -n kube-system -l "app=traefik" -o jsonpath="{.items[0].metadata.name}") 2 | `sleep 1 && open http://localhost:8080` & 3 | kubectl --namespace kube-system port-forward $TRAEFIK_POD 8080 4 | -------------------------------------------------------------------------------- /services/modules/simple-service/simple-service.tf: -------------------------------------------------------------------------------- 1 | resource "null_resource" "deploy_simple_service" { 2 | provisioner "local-exec" { 3 | command = "${path.module}/deploy-simple-service ${var.environment} ${var.container_repo}" 4 | } 5 | } 6 | -------------------------------------------------------------------------------- /cluster/environments/azure-aks/init: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | rm -rf .terraform 4 | rm terraform.tfstate 5 | rm terraform.tfstate.backup 6 | 7 | rm -rf deployment 8 | mkdir deployment 9 | 10 | terraform init -var-file="../common/common.tfvars" 11 | -------------------------------------------------------------------------------- /cluster/environments/azure-acs-engine/init: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | rm -rf .terraform 4 | rm terraform.tfstate 5 | rm terraform.tfstate.backup 6 | 7 | rm -rf deployment 8 | mkdir deployment 9 | 10 | terraform init -var-file="../common/common.tfvars" 11 | -------------------------------------------------------------------------------- /tools/prom: -------------------------------------------------------------------------------- 1 | export PROM_POD=$(kubectl get pods --namespace prometheus -l "app=prometheus,component=server" -o jsonpath="{.items[0].metadata.name}") 2 | `sleep 1 && open http://localhost:9090` & 3 | kubectl port-forward --namespace prometheus $PROM_POD 9090 4 | -------------------------------------------------------------------------------- /tools/grafana: -------------------------------------------------------------------------------- 1 | export GRAFANA_POD=$(kubectl get pods -n grafana -o jsonpath="{.items[0].metadata.name}") 2 | `sleep 1 && open http://localhost:3000/d/4XuMd2Iiz/kubernetes-cluster-prometheus?orgId=1` & 3 | kubectl --namespace grafana port-forward $GRAFANA_POD 3000 4 | -------------------------------------------------------------------------------- /tools/kibana: -------------------------------------------------------------------------------- 1 | export KIBANA_POD=$(kubectl get pods --namespace kibana -l "app=kibana,release=kibana" -o jsonpath="{.items[0].metadata.name}") 2 | `sleep 1 && open http://localhost:5601/app/kibana#/discover` & 3 | kubectl port-forward --namespace kibana $KIBANA_POD 5601 4 | -------------------------------------------------------------------------------- /services/environments/dev/main.tf: -------------------------------------------------------------------------------- 1 | module "services" { 2 | source = "../../modules/services" 3 | } 4 | 5 | module "simple-service" { 6 | source = "../../modules/simple-service" 7 | 8 | environment = "dev" 9 | container_repo = "${var.container_repo}" 10 | } 11 | -------------------------------------------------------------------------------- /services/modules/simple-service/deploy-simple-service: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | REPO_ROOT=https://github.com/timfpark \ 4 | REPO_NAME=simple-service \ 5 | SERVICE_NAME=simple-service \ 6 | NAMESPACE=services \ 7 | CONTAINER_REPO=$2 \ 8 | VALUES_FILE=devops/$1.yaml \ 9 | ../../common/deploy-service $1 10 | -------------------------------------------------------------------------------- /infra/modules/kured/kured.tf: -------------------------------------------------------------------------------- 1 | resource "helm_release" "kured" { 2 | chart = "stable/kured" 3 | name = "kured" 4 | namespace = "kube-system" 5 | timeout = "${var.helm_install_timeout}" 6 | 7 | set { 8 | name = "extraArgs.prometheus-url" 9 | value = "http://${var.prometheus_service_endpoint}" 10 | } 11 | } 12 | -------------------------------------------------------------------------------- /infra/modules/kibana/input.tf: -------------------------------------------------------------------------------- 1 | variable "elasticsearch_client_endpoint" { 2 | type = "string" 3 | } 4 | 5 | variable "helm_install_timeout" { 6 | default = 1800 7 | } 8 | 9 | variable "name" { 10 | type = "string" 11 | default = "kibana" 12 | } 13 | 14 | variable "namespace" { 15 | type = "string" 16 | default = "kibana" 17 | } 18 | -------------------------------------------------------------------------------- /infra/modules/fluentd/input.tf: -------------------------------------------------------------------------------- 1 | variable "elasticsearch_client_endpoint" { 2 | type = "string" 3 | } 4 | 5 | variable "name" { 6 | type = "string" 7 | default = "fluentd" 8 | } 9 | 10 | variable "namespace" { 11 | type = "string" 12 | default = "fluentd" 13 | } 14 | 15 | variable "helm_install_timeout" { 16 | default = 1800 17 | } 18 | -------------------------------------------------------------------------------- /infra/modules/istio/gateway.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: networking.istio.io/v1alpha3 2 | kind: Gateway 3 | metadata: 4 | name: default-gateway 5 | namespace: istio-system 6 | spec: 7 | selector: 8 | istio: ingressgateway 9 | servers: 10 | - hosts: 11 | - '*' 12 | port: 13 | name: http 14 | number: 80 15 | protocol: HTTP 16 | -------------------------------------------------------------------------------- /infra/modules/fluentd/fluentd.tf: -------------------------------------------------------------------------------- 1 | resource "helm_release" "fluentd" { 2 | chart = "stable/fluentd-elasticsearch" 3 | name = "${var.name}" 4 | namespace = "${var.namespace}" 5 | timeout = "${var.helm_install_timeout}" 6 | 7 | set { 8 | name = "elasticsearch.host" 9 | value = "${var.elasticsearch_client_endpoint}" 10 | } 11 | } 12 | -------------------------------------------------------------------------------- /infra/modules/jaeger/input.tf: -------------------------------------------------------------------------------- 1 | variable "name" { 2 | default = "jaeger" 3 | } 4 | 5 | variable "namespace" { 6 | default = "jaeger" 7 | } 8 | 9 | variable "helm_install_timeout" { 10 | default = 1800 11 | } 12 | 13 | variable "elasticsearch_client_endpoint" { 14 | type = "string" 15 | } 16 | 17 | variable "elasticsearch_port" { 18 | default = 9200 19 | } 20 | -------------------------------------------------------------------------------- /infra/modules/kured/inputs.tf: -------------------------------------------------------------------------------- 1 | variable "helm_install_timeout" { 2 | default = 1800 3 | } 4 | 5 | variable "name" { 6 | type = "string" 7 | default = "kured" 8 | } 9 | 10 | variable "namespace" { 11 | type = "string" 12 | default = "kube-system" 13 | } 14 | 15 | variable "prometheus_service_endpoint" { 16 | type = "string" 17 | default = "default" 18 | } 19 | -------------------------------------------------------------------------------- /cluster/providers/azure-aks/tiller.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: ServiceAccount 3 | metadata: 4 | name: tiller 5 | namespace: kube-system 6 | --- 7 | apiVersion: rbac.authorization.k8s.io/v1 8 | kind: ClusterRoleBinding 9 | metadata: 10 | name: tiller 11 | roleRef: 12 | apiGroup: rbac.authorization.k8s.io 13 | kind: ClusterRole 14 | name: cluster-admin 15 | subjects: 16 | - kind: ServiceAccount 17 | name: tiller 18 | namespace: kube-system -------------------------------------------------------------------------------- /cluster/providers/azure-acs-engine/tiller.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: ServiceAccount 3 | metadata: 4 | name: tiller 5 | namespace: kube-system 6 | --- 7 | apiVersion: rbac.authorization.k8s.io/v1 8 | kind: ClusterRoleBinding 9 | metadata: 10 | name: tiller 11 | roleRef: 12 | apiGroup: rbac.authorization.k8s.io 13 | kind: ClusterRole 14 | name: cluster-admin 15 | subjects: 16 | - kind: ServiceAccount 17 | name: tiller 18 | namespace: kube-system -------------------------------------------------------------------------------- /cluster/environments/azure-aks/inputs.tf: -------------------------------------------------------------------------------- 1 | /* 2 | variable "aad_server_app_id" { 3 | type = "string" 4 | } 5 | 6 | variable "aad_server_app_secret" { 7 | type = "string" 8 | } 9 | 10 | variable "aad_client_app_id" { 11 | type = "string" 12 | } 13 | 14 | variable "aad_tenant_id" { 15 | type = "string" 16 | } 17 | */ 18 | variable "client_id" { 19 | type = "string" 20 | } 21 | 22 | variable "client_secret" { 23 | type = "string" 24 | } 25 | 26 | variable "ssh_public_key" { 27 | type = "string" 28 | } 29 | -------------------------------------------------------------------------------- /cluster/environments/azure-aks/main.tf: -------------------------------------------------------------------------------- 1 | module "azure_aks" { 2 | source = "../../providers/azure-aks" 3 | 4 | cluster_name = "my-dev-cluster" 5 | agent_vm_count = "3" 6 | agent_vm_size = "Standard_DS3_v2" 7 | 8 | location = "eastus2" 9 | admin_user = "ops" 10 | 11 | subnet_address_space = "10.200.0.0/16" 12 | vnet_address_space = "10.200.0.0/16" 13 | first_master_ip = "10.200.255.239" 14 | 15 | client_id = "${var.client_id}" 16 | client_secret = "${var.client_secret}" 17 | 18 | ssh_public_key = "${var.ssh_public_key}" 19 | } 20 | -------------------------------------------------------------------------------- /infra/modules/istio/inputs.tf: -------------------------------------------------------------------------------- 1 | variable "helm_install_timeout" { 2 | default = 1800 3 | } 4 | 5 | variable "istio_version" { 6 | type = "string" 7 | default = "1.0.4" 8 | } 9 | 10 | variable "kiala_admin_username" { 11 | type = "string" 12 | default = "ops" 13 | } 14 | 15 | variable "kiala_admin_password" { 16 | type = "string" 17 | } 18 | 19 | variable "name" { 20 | type = "string" 21 | default = "istio" 22 | } 23 | 24 | variable "namespace" { 25 | type = "string" 26 | default = "istio-system" 27 | } 28 | 29 | variable "prometheus_service_endpoint" { 30 | type = "string" 31 | } 32 | -------------------------------------------------------------------------------- /infra/modules/prometheus/inputs.tf: -------------------------------------------------------------------------------- 1 | variable "helm_install_timeout" { 2 | default = 1800 3 | } 4 | 5 | variable "name" { 6 | type = "string" 7 | default = "prometheus" 8 | } 9 | 10 | variable "namespace" { 11 | type = "string" 12 | default = "prometheus" 13 | } 14 | 15 | variable "prometheus_alertmanager_storage_class" { 16 | type = "string" 17 | default = "default" 18 | } 19 | 20 | variable "prometheus_server_storage_class" { 21 | type = "string" 22 | default = "default" 23 | } 24 | 25 | variable "prometheus_server_storage_size" { 26 | type = "string" 27 | default = "default" 28 | } 29 | -------------------------------------------------------------------------------- /infra/modules/prometheus/prometheus.tf: -------------------------------------------------------------------------------- 1 | resource "helm_release" "prometheus" { 2 | chart = "stable/prometheus" 3 | name = "${var.name}" 4 | namespace = "${var.namespace}" 5 | timeout = "${var.helm_install_timeout}" 6 | 7 | set { 8 | name = "alertmanager.persistentVolume.storageClass" 9 | value = "${var.prometheus_alertmanager_storage_class}" 10 | } 11 | 12 | set { 13 | name = "server.persistentVolume.storageClass" 14 | value = "${var.prometheus_server_storage_class}" 15 | } 16 | 17 | set { 18 | name = "server.persistentVolume.size" 19 | value = "${var.prometheus_server_storage_size}" 20 | } 21 | } 22 | -------------------------------------------------------------------------------- /cluster/environments/azure-acs-engine/main.tf: -------------------------------------------------------------------------------- 1 | module "azure_acs_engine" { 2 | source = "../../providers/azure-acs-engine" 3 | 4 | cluster_name = "my-dev-cluster" 5 | agent_vm_count = "3" 6 | agent_vm_size = "Standard_DS3_v2" 7 | 8 | master_vm_count = "1" 9 | master_vm_size = "Standard_DS1_v2" 10 | 11 | location = "eastus2" 12 | admin_user = "ops" 13 | 14 | subnet_address_space = "10.201.0.0/16" 15 | vnet_address_space = "10.201.0.0/16" 16 | first_master_ip = "10.201.255.239" 17 | 18 | client_id = "${var.client_id}" 19 | client_secret = "${var.client_secret}" 20 | ssh_public_key = "${var.ssh_public_key}" 21 | } 22 | -------------------------------------------------------------------------------- /cluster/providers/azure-acs-engine/inputs.tf: -------------------------------------------------------------------------------- 1 | variable "agent_vm_count" { 2 | type = "string" 3 | } 4 | 5 | variable "agent_vm_size" { 6 | default = "Standard_DS3_v2" 7 | } 8 | 9 | variable "kubernetes_version" { 10 | default = "1.11.4" 11 | } 12 | 13 | variable "master_vm_count" { 14 | default = "3" 15 | } 16 | 17 | variable "master_vm_size" { 18 | default = "Standard_DS1_v2" 19 | } 20 | 21 | variable "admin_user" {} 22 | 23 | variable "ssh_public_key" {} 24 | 25 | variable "location" {} 26 | 27 | variable "subnet_address_space" {} 28 | 29 | variable "vnet_address_space" {} 30 | 31 | variable "first_master_ip" {} 32 | 33 | variable "cluster_name" {} 34 | 35 | variable "client_id" {} 36 | variable "client_secret" {} 37 | -------------------------------------------------------------------------------- /infra/modules/elasticsearch/inputs.tf: -------------------------------------------------------------------------------- 1 | variable "name" { 2 | type = "string" 3 | default = "elasticsearch" 4 | } 5 | 6 | variable "namespace" { 7 | type = "string" 8 | default = "elasticsearch" 9 | } 10 | 11 | variable "elasticsearch_master_storage_class" { 12 | type = "string" 13 | default = "default" 14 | } 15 | 16 | variable "elasticsearch_master_storage_size" { 17 | type = "string" 18 | default = "4Gi" 19 | } 20 | 21 | variable "elasticsearch_data_storage_class" { 22 | type = "string" 23 | default = "default" 24 | } 25 | 26 | variable "elasticsearch_data_storage_size" { 27 | type = "string" 28 | default = "31Gi" 29 | } 30 | 31 | variable "helm_install_timeout" { 32 | default = 1800 33 | } 34 | -------------------------------------------------------------------------------- /infra/modules/elasticsearch/elasticsearch.tf: -------------------------------------------------------------------------------- 1 | resource "helm_release" "elasticsearch" { 2 | chart = "stable/elasticsearch" 3 | name = "elasticsearch" 4 | namespace = "elasticsearch" 5 | timeout = "${var.helm_install_timeout}" 6 | 7 | set { 8 | name = "master.persistence.storageClass" 9 | value = "${var.elasticsearch_master_storage_class}" 10 | } 11 | 12 | set { 13 | name = "master.persistence.size" 14 | value = "${var.elasticsearch_master_storage_size}" 15 | } 16 | 17 | set { 18 | name = "data.persistence.storageClass" 19 | value = "${var.elasticsearch_data_storage_class}" 20 | } 21 | 22 | set { 23 | name = "data.persistence.size" 24 | value = "${var.elasticsearch_data_storage_size}" 25 | } 26 | } 27 | -------------------------------------------------------------------------------- /infra/environments/dev/main.tf: -------------------------------------------------------------------------------- 1 | module "infra" { 2 | source = "../../stacks/cloud-native" 3 | 4 | elasticsearch_master_storage_class = "default" 5 | elasticsearch_data_storage_class = "default" 6 | elasticsearch_data_storage_size = "4Gi" 7 | 8 | grafana_admin_username = "ops" 9 | grafana_admin_password = "${var.grafana_admin_password}" 10 | grafana_dashboard_yaml = "${file("./grafana-dashboards.yaml")}" 11 | 12 | kiala_admin_username = "ops" 13 | kiala_admin_password = "${var.grafana_admin_password}" 14 | 15 | prometheus_alertmanager_storage_class = "default" 16 | prometheus_server_storage_class = "default" 17 | prometheus_server_storage_size = "4Gi" 18 | 19 | traefik_ssl_enabled = "false" 20 | traefik_ssl_enforced = "false" 21 | ingress_replica_count = "3" 22 | } 23 | -------------------------------------------------------------------------------- /infra/modules/grafana/inputs.tf: -------------------------------------------------------------------------------- 1 | variable "admin_user" { 2 | type = "string" 3 | } 4 | 5 | variable "admin_password" { 6 | type = "string" 7 | } 8 | 9 | variable "dashboard_yaml" { 10 | type = "string" 11 | default = "" 12 | } 13 | 14 | variable "helm_install_timeout" { 15 | default = 1800 16 | } 17 | 18 | variable "name" { 19 | type = "string" 20 | default = "grafana" 21 | } 22 | 23 | variable "namespace" { 24 | type = "string" 25 | default = "grafana" 26 | } 27 | 28 | variable "persistence_enabled" { 29 | type = "string" 30 | default = "true" 31 | } 32 | 33 | variable "prometheus_service_endpoint" { 34 | type = "string" 35 | } 36 | 37 | variable "storage_class" { 38 | type = "string" 39 | default = "default" 40 | } 41 | 42 | variable "storage_size" { 43 | type = "string" 44 | default = "4Gi" 45 | } 46 | -------------------------------------------------------------------------------- /infra/modules/kibana/kibana.tf: -------------------------------------------------------------------------------- 1 | data "template_file" "kibana_yaml" { 2 | template = "${file("${path.module}/kibana.yaml.tmpl")}" 3 | 4 | vars { 5 | elasticsearch_client_endpoint = "${var.elasticsearch_client_endpoint}" 6 | } 7 | } 8 | 9 | resource "null_resource" "generate_kibana_yaml" { 10 | provisioner "local-exec" { 11 | command = "echo '${data.template_file.kibana_yaml.rendered}' > ${path.module}/kibana.yaml" 12 | } 13 | 14 | depends_on = ["data.template_file.kibana_yaml"] 15 | } 16 | 17 | resource "helm_release" "kibana" { 18 | chart = "stable/kibana" 19 | name = "${var.name}" 20 | namespace = "${var.namespace}" 21 | timeout = "${var.helm_install_timeout}" 22 | 23 | values = [ 24 | "${file("${path.module}/kibana.yaml")}", 25 | ] 26 | 27 | depends_on = ["null_resource.generate_kibana_yaml"] 28 | } 29 | -------------------------------------------------------------------------------- /services/common/deploy-service: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | mkdir -p deployments/$1 4 | cd deployments/$1 5 | 6 | REPO_URL=$REPO_ROOT/$REPO_NAME 7 | echo "cloning $REPO_URL" 8 | 9 | rm -rf $REPO_NAME 10 | 11 | if ! git clone $REPO_URL; then 12 | echo "ERROR: failed to clone $REPO_URL" 13 | exit 1 14 | fi 15 | 16 | cd $REPO_NAME 17 | 18 | DATE_TAG=`date -u +"%Y%m%dT%H%M%SZ"` 19 | RELEASE_TAG=$CONTAINER_REPO/$SERVICE_NAME:$DATE_TAG 20 | 21 | if ! docker build -t $RELEASE_TAG .; then 22 | echo "ERROR: failed to docker build" 23 | exit 1 24 | fi 25 | 26 | if ! docker push $RELEASE_TAG; then 27 | echo "ERROR: failed to docker push" 28 | exit 1 29 | fi 30 | 31 | if ! helm upgrade $SERVICE_NAME --install --namespace $NAMESPACE --set image=$RELEASE_TAG --values=$VALUES_FILE --wait devops; then 32 | echo "ERROR: failed to helm upgrade --install" 33 | exit 1 34 | fi 35 | 36 | cd ../.. 37 | -------------------------------------------------------------------------------- /infra/environments/prod/main.tf: -------------------------------------------------------------------------------- 1 | module "infra" { 2 | source = "../../stacks/cloud-native" 3 | 4 | grafana_admin_username = "${var.grafana_admin_username}" 5 | grafana_admin_password = "${var.grafana_admin_password}" 6 | grafana_dashboard_yaml = "${file("./grafana-dashboards.yaml")}" 7 | 8 | elasticsearch_master_storage_class = "managed-premium" 9 | elasticsearch_data_storage_class = "managed-premium" 10 | elasticsearch_data_storage_size = "16Gi" 11 | 12 | prometheus_alertmanager_storage_class = "managed-premium" 13 | prometheus_server_storage_class = "managed-premium" 14 | prometheus_server_storage_size = "8Gi" 15 | 16 | traefik_ssl_enabled = "true" 17 | traefik_ssl_enforced = "true" 18 | ingress_replica_count = "3" 19 | 20 | ssl_cert_base64 = "${base64encode(file("./tls/wildcard.domain.com.crt"))}" 21 | ssl_key_base64 = "${base64encode(file("./tls/wildcard.domain.com.key"))}" 22 | } 23 | -------------------------------------------------------------------------------- /infra/modules/jaeger/jaeger.tf: -------------------------------------------------------------------------------- 1 | resource "helm_repository" "incubator" { 2 | name = "incubator" 3 | url = "http://storage.googleapis.com/kubernetes-charts-incubator" 4 | } 5 | 6 | resource "helm_release" "jaeger" { 7 | chart = "incubator/jaeger" 8 | name = "${var.name}" 9 | namespace = "${var.namespace}" 10 | timeout = "${var.helm_install_timeout}" 11 | 12 | set { 13 | name = "tag" 14 | value = "1.8.0" 15 | } 16 | 17 | set { 18 | name = "provisionDataStore.cassandra" 19 | value = "false" 20 | } 21 | 22 | set { 23 | name = "provisionDataStore.elasticsearch" 24 | value = "false" 25 | } 26 | 27 | set { 28 | name = "storage.type" 29 | value = "elasticsearch" 30 | } 31 | 32 | set { 33 | name = "storage.elasticsearch.host" 34 | value = "${var.elasticsearch_client_endpoint}" 35 | } 36 | 37 | set { 38 | name = "storage.elasticsearch.port" 39 | value = "${var.elasticsearch_port}" 40 | } 41 | } 42 | -------------------------------------------------------------------------------- /cluster/providers/azure-aks/inputs.tf: -------------------------------------------------------------------------------- 1 | variable "agent_vm_count" { 2 | type = "string" 3 | } 4 | 5 | variable "agent_vm_size" { 6 | default = "Standard_DS3_v2" 7 | } 8 | 9 | variable "kubernetes_version" { 10 | default = "1.11.4" 11 | } 12 | 13 | variable "master_vm_count" { 14 | default = "3" 15 | } 16 | 17 | variable "master_vm_size" { 18 | default = "Standard_DS1_v2" 19 | } 20 | 21 | variable "admin_user" {} 22 | 23 | variable "ssh_public_key" {} 24 | 25 | variable "location" {} 26 | 27 | variable "subnet_address_space" {} 28 | 29 | variable "vnet_address_space" {} 30 | 31 | variable "first_master_ip" {} 32 | 33 | variable "cluster_name" {} 34 | 35 | /* 36 | variable "aad_server_app_id" { 37 | type = "string" 38 | } 39 | 40 | variable "aad_server_app_secret" { 41 | type = "string" 42 | } 43 | 44 | variable "aad_client_app_id" { 45 | type = "string" 46 | } 47 | 48 | variable "aad_tenant_id" { 49 | type = "string" 50 | } 51 | */ 52 | 53 | variable "client_id" {} 54 | 55 | variable "client_secret" {} 56 | -------------------------------------------------------------------------------- /infra/modules/grafana/grafana.yaml: -------------------------------------------------------------------------------- 1 | datasources: 2 | datasources.yaml: 3 | apiVersion: 1 4 | datasources: 5 | - name: Prometheus 6 | type: prometheus 7 | url: http://prometheus-server.prometheus.svc.cluster.local 8 | access: proxy 9 | isDefault: true 10 | dashboardProviders: 11 | dashboardproviders.yaml: 12 | apiVersion: 1 13 | providers: 14 | - name: default 15 | orgId: 1 16 | folder: 17 | type: file 18 | disableDeletion: false 19 | editable: true 20 | options: 21 | path: /var/lib/grafana/dashboards/default 22 | dashboards: 23 | default: 24 | cluster-metrics: 25 | gnetId: 6417 26 | revision: 1 27 | datasource: Prometheus 28 | persistent-volumes: 29 | gnetId: 6739 30 | revision: 1 31 | datasource: Prometheus 32 | 33 | persistence: 34 | enabled: true 35 | storageClassName: default 36 | accessModes: 37 | - ReadWriteOnce 38 | size: 4Gi 39 | plugins: 40 | - digrich-bubblechart-panel 41 | - grafana-clock-panel 42 | -------------------------------------------------------------------------------- /Dockerfile: -------------------------------------------------------------------------------- 1 | FROM alpine:3.8 2 | 3 | ENV TERRAFORM_VERSION="0.11.10" 4 | ENV HELM_VERSION="v2.11.0" 5 | 6 | WORKDIR /bedrock 7 | 8 | RUN apk add --no-cache ca-certificates curl bash git 9 | 10 | RUN curl -L https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl -o /usr/local/bin/kubectl && \ 11 | curl https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip > terraform_${TERRAFORM_VERSION}_linux_amd64.zip && \ 12 | unzip terraform_${TERRAFORM_VERSION}_linux_amd64.zip -d /usr/local/bin && \ 13 | curl -L https://storage.googleapis.com/kubernetes-helm/helm-${HELM_VERSION}-linux-amd64.tar.gz | tar xz && \ 14 | mv linux-amd64/helm /usr/local/bin/helm && \ 15 | chmod +x /usr/local/bin/kubectl && \ 16 | chmod +x /usr/local/bin/helm && \ 17 | chmod +x /usr/local/bin/terraform && \ 18 | rm -f terraform_${TERRAFORM_VERSION}_linux_amd64.zip && \ 19 | rm -rf linux-amd64 20 | 21 | COPY . . 22 | 23 | CMD [ "/bin/bash" ] 24 | -------------------------------------------------------------------------------- /infra/modules/grafana/grafana.yaml.tmpl: -------------------------------------------------------------------------------- 1 | datasources: 2 | datasources.yaml: 3 | apiVersion: 1 4 | datasources: 5 | - name: Prometheus 6 | type: prometheus 7 | url: http://${prometheus_service_endpoint} 8 | access: proxy 9 | isDefault: true 10 | dashboardProviders: 11 | dashboardproviders.yaml: 12 | apiVersion: 1 13 | providers: 14 | - name: 'default' 15 | orgId: 1 16 | folder: '' 17 | type: file 18 | disableDeletion: false 19 | editable: true 20 | options: 21 | path: /var/lib/grafana/dashboards/default 22 | dashboards: 23 | default: 24 | cluster-metrics: 25 | gnetId: 6417 26 | revision: 1 27 | datasource: Prometheus 28 | persistent-volumes: 29 | gnetId: 6739 30 | revision: 1 31 | datasource: Prometheus 32 | ${dashboard_yaml} 33 | persistence: 34 | enabled: ${persistence_enabled} 35 | storageClassName: '${storage_class}' 36 | accessModes: 37 | - ReadWriteOnce 38 | size: ${storage_size} 39 | plugins: 40 | - digrich-bubblechart-panel 41 | - grafana-clock-panel -------------------------------------------------------------------------------- /tools/check-prereqs: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | echo "Checking tool prereqs:" 4 | 5 | if ! terraform version > /dev/null; then 6 | echo "ERROR: terraform is not installed. Please install the latest version from https://www.terraform.io/intro/getting-started/install.html and place it in your path." 7 | exit 1 8 | else 9 | echo "terraform [ok]" 10 | fi 11 | 12 | if ! kubectl version --client > /dev/null; then 13 | echo "ERROR: kubectl is not installed. Please install the latest version from https://kubernetes.io/docs/tasks/tools/install-kubectl/ and place it in your path." 14 | exit 1 15 | else 16 | echo "kubectl [ok]" 17 | fi 18 | 19 | if ! helm > /dev/null; then 20 | echo "ERROR: helm not installed. Please install the latest version from https://docs.helm.sh/using_helm/#installing-helm and place it in your path." 21 | exit 1 22 | else 23 | echo "helm [ok]" 24 | fi 25 | 26 | printf "\nChecking current Kubernetes cluster:\n" 27 | 28 | if ! kubectl cluster-info; then 29 | echo "ERROR: kubectl cluster-info failed. Do you have a running current Kubernetes cluster?" 30 | exit 1 31 | fi 32 | 33 | printf "\nReady for deploy!\n" 34 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) Microsoft Corporation. All rights reserved. 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE 22 | -------------------------------------------------------------------------------- /infra/stacks/cloud-native/inputs.tf: -------------------------------------------------------------------------------- 1 | variable "elasticsearch_master_storage_class" { 2 | type = "string" 3 | } 4 | 5 | variable "elasticsearch_data_storage_class" { 6 | type = "string" 7 | } 8 | 9 | variable "elasticsearch_data_storage_size" { 10 | type = "string" 11 | } 12 | 13 | variable "grafana_admin_username" { 14 | type = "string" 15 | } 16 | 17 | variable "grafana_admin_password" { 18 | type = "string" 19 | } 20 | 21 | variable "grafana_dashboard_yaml" { 22 | type = "string" 23 | } 24 | 25 | variable "kiala_admin_username" { 26 | type = "string" 27 | } 28 | 29 | variable "kiala_admin_password" { 30 | type = "string" 31 | } 32 | 33 | variable "prometheus_alertmanager_storage_class" { 34 | type = "string" 35 | } 36 | 37 | variable "prometheus_server_storage_class" { 38 | type = "string" 39 | } 40 | 41 | variable "prometheus_server_storage_size" { 42 | type = "string" 43 | } 44 | 45 | variable "ingress_replica_count" { 46 | type = "string" 47 | } 48 | 49 | variable "traefik_ssl_enabled" { 50 | type = "string" 51 | } 52 | 53 | variable "traefik_ssl_enforced" { 54 | type = "string" 55 | } 56 | 57 | variable "ssl_cert_base64" { 58 | type = "string" 59 | default = "" 60 | } 61 | 62 | variable "ssl_key_base64" { 63 | type = "string" 64 | default = "" 65 | } 66 | -------------------------------------------------------------------------------- /infra/modules/grafana/grafana.tf: -------------------------------------------------------------------------------- 1 | data "template_file" "template_grafana_yaml" { 2 | template = "${file("${path.module}/grafana.yaml.tmpl")}" 3 | 4 | vars { 5 | dashboard_yaml = "${var.dashboard_yaml}" 6 | persistence_enabled = "${var.persistence_enabled}" 7 | prometheus_service_endpoint = "${var.prometheus_service_endpoint}" 8 | storage_class = "${var.storage_class}" 9 | storage_size = "${var.storage_size}" 10 | } 11 | } 12 | 13 | resource "null_resource" "generate_grafana_yaml" { 14 | provisioner "local-exec" { 15 | command = "echo '${data.template_file.template_grafana_yaml.rendered}' > ${path.module}/grafana.yaml" 16 | } 17 | } 18 | 19 | resource "helm_release" "grafana" { 20 | chart = "stable/grafana" 21 | name = "${var.name}" 22 | namespace = "${var.namespace}" 23 | timeout = "${var.helm_install_timeout}" 24 | 25 | values = [ 26 | "${file("${path.module}/grafana.yaml")}", 27 | ] 28 | 29 | set { 30 | name = "adminUser" 31 | value = "${var.admin_user}" 32 | } 33 | 34 | set { 35 | name = "adminPassword" 36 | value = "${var.admin_password}" 37 | } 38 | 39 | set { 40 | name = "persistance_enabled" 41 | value = "${var.persistence_enabled}" 42 | } 43 | 44 | set { 45 | name = "persistance_storage_class" 46 | value = "${var.storage_class}" 47 | } 48 | } 49 | -------------------------------------------------------------------------------- /infra/modules/istio/istio.tf: -------------------------------------------------------------------------------- 1 | resource "null_resource" "download_istio" { 2 | provisioner "local-exec" { 3 | command = "curl -sL https://github.com/istio/istio/releases/download/${var.istio_version}/istio-${var.istio_version}-linux.tar.gz | tar xz -C ${path.module}" 4 | } 5 | } 6 | 7 | resource "helm_release" "istio" { 8 | chart = "${path.module}/istio-${var.istio_version}/install/kubernetes/helm/istio" 9 | name = "${var.name}" 10 | namespace = "${var.namespace}" 11 | timeout = "${var.helm_install_timeout}" 12 | 13 | set { 14 | name = "global.controlPlaneSecurityEnabled" 15 | value = "true" 16 | } 17 | 18 | set { 19 | name = "kiali.enabled" 20 | value = "true" 21 | } 22 | 23 | set { 24 | name = "prometheus.enabled" 25 | value = "false" 26 | } 27 | 28 | set { 29 | name = "servicegraph.prometheusAddr" 30 | value = "http://${var.prometheus_service_endpoint}:9090" 31 | } 32 | 33 | set { 34 | name = "mtls.enabled" 35 | value = "true" 36 | } 37 | 38 | set { 39 | name = "kiali.dashboard.username" 40 | value = "${var.kiala_admin_username}" 41 | } 42 | 43 | set { 44 | name = "kiali.dashboard.passphrase" 45 | value = "${var.kiala_admin_password}" 46 | } 47 | 48 | set { 49 | name = "tracing.enabled" 50 | value = "true" 51 | } 52 | 53 | depends_on = ["null_resource.download_istio"] 54 | } 55 | 56 | resource "null_resource" "install_default_gateway" { 57 | provisioner "local-exec" { 58 | command = "kubectl apply -f ${path.module}/gateway.yaml" 59 | } 60 | 61 | depends_on = ["helm_release.istio"] 62 | } 63 | -------------------------------------------------------------------------------- /cluster/providers/azure-acs-engine/acs-engine-template.json: -------------------------------------------------------------------------------- 1 | { 2 | "apiVersion": "vlabs", 3 | "properties": { 4 | "orchestratorProfile": { 5 | "orchestratorType": "Kubernetes", 6 | "orchestratorVersion": "${version}", 7 | "kubernetesConfig": { 8 | "addons": [ 9 | { 10 | "name": "keyvault-flexvolume", 11 | "enabled": true 12 | }, 13 | { 14 | "name": "tiller", 15 | "enabled" : false 16 | } 17 | ], 18 | "networkPolicy": "azure" 19 | } 20 | }, 21 | "masterProfile": { 22 | "count": ${master_vm_count}, 23 | "dnsPrefix": "${dns_prefix}", 24 | "vmSize": "${master_vm_size}", 25 | "storageProfile": "ManagedDisks", 26 | "vnetSubnetId": "${subnet_id}", 27 | "firstConsecutiveStaticIP": "${first_master_ip}", 28 | "vnetCidr": "${vnet_cidr}" 29 | }, 30 | "agentPoolProfiles": [ 31 | { 32 | "name": "agents", 33 | "count": ${agent_vm_count}, 34 | "vmSize": "${agent_vm_size}", 35 | "vnetSubnetId": "${subnet_id}", 36 | "availabilityProfile": "AvailabilitySet", 37 | "storageProfile": "ManagedDisks", 38 | "extensions": [] 39 | } 40 | ], 41 | "linuxProfile": { 42 | "adminUsername": "${admin_user}", 43 | "ssh": { 44 | "publicKeys": [ 45 | { 46 | "keyData": "${ssh_key}" 47 | } 48 | ] 49 | } 50 | }, 51 | "servicePrincipalProfile": { 52 | "clientId": "${client_id}", 53 | "secret": "${client_secret}" 54 | } 55 | } 56 | } 57 | -------------------------------------------------------------------------------- /infra/stacks/cloud-native/cloud-native.tf: -------------------------------------------------------------------------------- 1 | module "prometheus" { 2 | source = "../../modules/prometheus" 3 | 4 | prometheus_alertmanager_storage_class = "${var.prometheus_alertmanager_storage_class}" 5 | prometheus_server_storage_class = "${var.prometheus_server_storage_class}" 6 | prometheus_server_storage_size = "${var.prometheus_server_storage_size}" 7 | } 8 | 9 | module "grafana" { 10 | source = "../../modules/grafana" 11 | 12 | admin_user = "${var.grafana_admin_username}" 13 | admin_password = "${var.grafana_admin_password}" 14 | 15 | prometheus_service_endpoint = "${module.prometheus.prometheus_service_endpoint}" 16 | 17 | dashboard_yaml = "${var.grafana_dashboard_yaml}" 18 | } 19 | 20 | module "kured" { 21 | source = "../../modules/kured" 22 | 23 | prometheus_service_endpoint = "${module.prometheus.prometheus_service_endpoint}" 24 | } 25 | 26 | module "elasticsearch" { 27 | source = "../../modules/elasticsearch" 28 | 29 | elasticsearch_master_storage_class = "${var.elasticsearch_master_storage_class}" 30 | elasticsearch_data_storage_class = "${var.elasticsearch_data_storage_class}" 31 | elasticsearch_data_storage_size = "${var.elasticsearch_data_storage_size}" 32 | } 33 | 34 | module "fluentd" { 35 | source = "../../modules/fluentd" 36 | 37 | elasticsearch_client_endpoint = "${module.elasticsearch.elasticsearch_client_endpoint}" 38 | } 39 | 40 | module "kibana" { 41 | source = "../../modules/kibana" 42 | 43 | elasticsearch_client_endpoint = "${module.elasticsearch.elasticsearch_client_endpoint}" 44 | } 45 | 46 | module "jaeger" { 47 | source = "../../modules/jaeger" 48 | 49 | elasticsearch_client_endpoint = "${module.elasticsearch.elasticsearch_client_endpoint}" 50 | } 51 | 52 | module "istio" { 53 | source = "../../modules/istio" 54 | 55 | kiala_admin_username = "${var.kiala_admin_username}" 56 | kiala_admin_password = "${var.kiala_admin_password}" 57 | 58 | prometheus_service_endpoint = "${module.prometheus.prometheus_service_endpoint}" 59 | } 60 | -------------------------------------------------------------------------------- /cluster/providers/azure-acs-engine/acs-engine-cluster.json: -------------------------------------------------------------------------------- 1 | { 2 | "apiVersion": "vlabs", 3 | "properties": { 4 | "orchestratorProfile": { 5 | "orchestratorType": "Kubernetes", 6 | "orchestratorVersion": "1.11.3", 7 | "kubernetesConfig": { 8 | "addons": [ 9 | { 10 | "name": "keyvault-flexvolume", 11 | "enabled": true 12 | }, 13 | { 14 | "name": "tiller", 15 | "enabled" : false 16 | } 17 | ], 18 | "networkPolicy": "azure" 19 | } 20 | }, 21 | "masterProfile": { 22 | "count": 3, 23 | "dnsPrefix": "rhom-magenta", 24 | "vmSize": "Standard_DS1_v2", 25 | "storageProfile": "ManagedDisks", 26 | "vnetSubnetId": "/subscriptions/1d3bc944-c31f-41a9-a1ac-cafea961eba5/resourceGroups/rhom-magenta-rg/providers/Microsoft.Network/virtualNetworks/rhom-magenta-vnet/subnets/rhom-magenta-subnet", 27 | "firstConsecutiveStaticIP": "10.3.255.239", 28 | "vnetCidr": "10.3.0.0/16" 29 | }, 30 | "agentPoolProfiles": [ 31 | { 32 | "name": "agents", 33 | "count": 3, 34 | "vmSize": "Standard_DS3_v2", 35 | "vnetSubnetId": "/subscriptions/1d3bc944-c31f-41a9-a1ac-cafea961eba5/resourceGroups/rhom-magenta-rg/providers/Microsoft.Network/virtualNetworks/rhom-magenta-vnet/subnets/rhom-magenta-subnet", 36 | "availabilityProfile": "AvailabilitySet", 37 | "storageProfile": "ManagedDisks", 38 | "extensions": [] 39 | } 40 | ], 41 | "linuxProfile": { 42 | "adminUsername": "var.admin_user", 43 | "ssh": { 44 | "publicKeys": [ 45 | { 46 | "keyData": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDHeqz5mrAA5Y+ZkAp3ZX31gHi954JBNNZU8MUA65qkPEMZ8YF5+MfoMZBiB9YBsP2G6wkn/zWOZsS299613dZZzIjkRCsJV4jqem7SK1vGI8sEXIr4H1oThtpKmHfLvoF7xYggT2NWyBxHkAaQLaSNenrCKYjyaNGpq7sYDEJ2t9xth4yAbFOYcrxx/s7lD/hE3z5OHTFk/Lqm7nSSx9zahJEU1RxZ0+JZwtWTUfwxktmWfDGmlBzN9ZZs2Znv4Zm9mJqTfUIufQrwwgFb9FhqxLiXwE49yokAT2j9XfvDnnOMGUqVuY15m2RFIouo/HU9NsgdZDxmB/lrJCf8lH30uIUbRjxnXQ2sKA4Ba/NdqP9Rdtj/Mov4kUNnScUUf8iYMSpfCL0BDY/N9oCN86fBhiREZqVqFn7iwLbQTwRZ7Fmk0/UNGpWeVbqYSm17rLiqmIs02PMepGmb6Ok5o+habgI4nXqw3+5nB8yaTAwIQcn43S4+dLvsmbONNk/gxk7UvI8J3ZALStEVRimuy72aHuseBKLnWWsDo8M1p2eXvABE84IgVNrYklgodNP7GExxNLSLqcsZa9ZALc+P3FRjgYbLC/qMWtkzPH5TEHPU4P5KLbHr4ZN3kV2MiARTtjWOlYMnMnrGu6NYxCmjHsbZxfhhZ2rU3uIEvjUBo9rdtQ== timfpark@gmail.com" 47 | } 48 | ] 49 | } 50 | }, 51 | "servicePrincipalProfile": { 52 | "clientId": "c83715a7-59b2-4b8d-b0f2-8421aa4527fa", 53 | "secret": "6fb78a8e-dcaa-4558-a055-4e5dce63e57e" 54 | } 55 | } 56 | } 57 | 58 | -------------------------------------------------------------------------------- /cluster/providers/azure-acs-engine/acs-engine.tf: -------------------------------------------------------------------------------- 1 | resource "azurerm_resource_group" "cluster" { 2 | name = "${var.cluster_name}-rg" 3 | location = "${var.location}" 4 | } 5 | 6 | resource "azurerm_virtual_network" "cluster" { 7 | name = "${var.cluster_name}-vnet" 8 | address_space = ["${var.vnet_address_space}"] 9 | location = "${var.location}" 10 | resource_group_name = "${azurerm_resource_group.cluster.name}" 11 | } 12 | 13 | resource "azurerm_subnet" "cluster" { 14 | name = "${var.cluster_name}-subnet" 15 | resource_group_name = "${azurerm_resource_group.cluster.name}" 16 | address_prefix = "${var.subnet_address_space}" 17 | virtual_network_name = "${azurerm_virtual_network.cluster.name}" 18 | } 19 | 20 | data "template_file" "acs_engine_config" { 21 | template = "${file("${path.module}/acs-engine-template.json")}" 22 | 23 | vars { 24 | version = "${var.kubernetes_version}" 25 | 26 | master_vm_count = "${var.master_vm_count}" 27 | master_vm_size = "${var.master_vm_size}" 28 | agent_vm_count = "${var.agent_vm_count}" 29 | agent_vm_size = "${var.agent_vm_size}" 30 | 31 | admin_user = "${var.admin_user}" 32 | ssh_key = "${trimspace(var.ssh_public_key)}" 33 | 34 | client_id = "${var.client_id}" 35 | client_secret = "${var.client_secret}" 36 | 37 | dns_prefix = "${var.cluster_name}" 38 | subnet_id = "${azurerm_subnet.cluster.id}" 39 | first_master_ip = "${var.first_master_ip}" 40 | vnet_cidr = "${var.subnet_address_space}" 41 | } 42 | } 43 | 44 | resource "null_resource" "generate_acs_engine_config" { 45 | provisioner "local-exec" { 46 | command = "echo '${data.template_file.acs_engine_config.rendered}' > deployment/acs-engine-cluster.json" 47 | } 48 | 49 | depends_on = ["data.template_file.acs_engine_config"] 50 | } 51 | 52 | # Locally run the ACS Engine to produce the Azure Resource Template for the K8s cluster 53 | resource "null_resource" "generate_acs_engine_deployment" { 54 | provisioner "local-exec" { 55 | command = "acs-engine generate deployment/acs-engine-cluster.json --output-directory deployment/acs-engine" 56 | } 57 | 58 | depends_on = ["null_resource.generate_acs_engine_config"] 59 | } 60 | 61 | # Locally run the Azure 2.0 CLI to create the resource deployment 62 | resource "null_resource" "cluster" { 63 | provisioner "local-exec" { 64 | command = "az group deployment create --name ${var.cluster_name} --resource-group ${var.cluster_name}-rg --template-file ./deployment/acs-engine/azuredeploy.json --parameters @./deployment/acs-engine/azuredeploy.parameters.json" 65 | } 66 | 67 | depends_on = ["null_resource.generate_acs_engine_deployment"] 68 | } 69 | 70 | # Merge k8s config from acs-engine 71 | resource "null_resource" "kubectl" { 72 | provisioner "local-exec" { 73 | command = "KUBECONFIG=./deployment/acs-engine/kubeconfig/kubeconfig.${var.location}.json:~/.kube/config kubectl config view --flatten > merged-config && mv merged-config ~/.kube/config" 74 | } 75 | 76 | depends_on = ["null_resource.cluster"] 77 | } 78 | 79 | resource "null_resource" "helm" { 80 | provisioner "local-exec" { 81 | command = "kubectl apply -f ${path.module}/tiller.yaml && helm init --service-account tiller --upgrade --wait" 82 | } 83 | 84 | depends_on = ["null_resource.kubectl"] 85 | } 86 | -------------------------------------------------------------------------------- /cluster/providers/azure-aks/aks.tf: -------------------------------------------------------------------------------- 1 | resource "azurerm_resource_group" "cluster" { 2 | name = "${var.cluster_name}-rg" 3 | location = "${var.location}" 4 | } 5 | 6 | resource "azurerm_virtual_network" "cluster" { 7 | name = "${var.cluster_name}-vnet" 8 | address_space = ["${var.vnet_address_space}"] 9 | location = "${var.location}" 10 | resource_group_name = "${azurerm_resource_group.cluster.name}" 11 | } 12 | 13 | resource "azurerm_subnet" "cluster" { 14 | name = "${var.cluster_name}-subnet" 15 | resource_group_name = "${azurerm_resource_group.cluster.name}" 16 | address_prefix = "${var.subnet_address_space}" 17 | virtual_network_name = "${azurerm_virtual_network.cluster.name}" 18 | } 19 | 20 | /* 21 | 22 | NOTE: Currently you can not enable RBAC without a backing AAD service principal. In the meantime, use the az command line. 23 | TODO: This support is expected within a couple of releases -- switch back when its available. 24 | 25 | resource "azurerm_kubernetes_cluster" "cluster" { 26 | name = "${var.cluster_name}" 27 | location = "${azurerm_resource_group.cluster.location}" 28 | resource_group_name = "${azurerm_resource_group.cluster.name}" 29 | dns_prefix = "${var.cluster_name}" 30 | kubernetes_version = "${var.kubernetes_version}" 31 | 32 | linux_profile { 33 | admin_username = "${var.admin_user}" 34 | 35 | ssh_key { 36 | key_data = "${var.ssh_public_key}" 37 | } 38 | } 39 | 40 | agent_pool_profile { 41 | name = "default" 42 | count = "${var.agent_vm_count}" 43 | vm_size = "${var.agent_vm_size}" 44 | os_type = "Linux" 45 | os_disk_size_gb = 30 46 | vnet_subnet_id = "${azurerm_subnet.cluster.id}" 47 | } 48 | 49 | network_profile { 50 | network_plugin = "azure" 51 | } 52 | 53 | role_based_access_control { 54 | azure_active_directory { 55 | server_app_id = "${var.aad_server_app_id}" 56 | server_app_secret = "${var.aad_server_app_secret}" 57 | client_app_id = "${var.aad_client_app_id}" 58 | tenant_id = "${var.aad_tenant_id}" 59 | } 60 | } 61 | 62 | service_principal { 63 | client_id = "${var.client_id}" 64 | client_secret = "${var.client_secret}" 65 | } 66 | } 67 | 68 | */ 69 | 70 | resource "null_resource" "create_cluster" { 71 | provisioner "local-exec" { 72 | command = "az aks create -g ${azurerm_resource_group.cluster.name} -n ${var.cluster_name} -l ${azurerm_resource_group.cluster.location} --kubernetes-version ${var.kubernetes_version} --node-count ${var.agent_vm_count} --node-vm-size ${var.agent_vm_size} --network-plugin azure --vnet-subnet-id ${azurerm_subnet.cluster.id}" 73 | } 74 | 75 | depends_on = ["azurerm_subnet.cluster"] 76 | } 77 | 78 | resource "null_resource" "cluster_credentials" { 79 | provisioner "local-exec" { 80 | command = "az aks get-credentials --resource-group ${azurerm_resource_group.cluster.name} --name ${var.cluster_name} --overwrite-existing" 81 | } 82 | 83 | //depends_on = ["azurerm_kubernetes_cluster.cluster"] 84 | depends_on = ["null_resource.create_cluster"] 85 | } 86 | 87 | resource "null_resource" "helm" { 88 | provisioner "local-exec" { 89 | command = "kubectl apply -f ${path.module}/tiller.yaml && helm init --service-account tiller --upgrade --wait" 90 | } 91 | 92 | depends_on = ["null_resource.cluster_credentials"] 93 | } 94 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # bedrock 2 | 3 | The cloud native ecosystem is in a virtual cambrian explosion of platforms and projects that individually promise to greatly improve our lives as developers. At the same time, even as an experienced developer in this space, it is difficult to start from stratch and stitch all of these projects together into a coherent whole without having to do a substantial amount of research and work. 4 | 5 | This project is our humble attempt to combine the collective wisdom of our cloud native community for building best practice cloud native Kubernetes clusters. It is based on the real world experience that we have of deploying cloud native applications with our largest customers. 6 | 7 | ## What's in the box? 8 | 9 | Bedrock is a set of devops scripts for automated deployment of the common elements of a production-ready cloud native Kubernetes cluster. It includes: 10 | 11 | Cluster Management 12 | 13 | - [Kured](https://github.com/weaveworks/kured) (automatic cordon/drain/reboot after node level patches are applied) 14 | 15 | Monitoring 16 | 17 | - [Prometheus](https://prometheus.io/) metrics monitoring and aggregation 18 | - [Grafana](https://grafana.com/) metrics visualization with Kubernetes monitoring dashboards preconfigured 19 | 20 | Log Management 21 | 22 | - [Fluentd](https://www.fluentd.org/) collection and forwarding 23 | - [Elasticsearch](https://www.elastic.co/) aggregation 24 | - [Kibana](https://www.elastic.co/products/kibana) querying and visualization 25 | 26 | Ingress and Service Mesh 27 | 28 | - [Istio](https://istio.io/) ingress gateway and service mesh 29 | 30 | Distributed Tracing 31 | 32 | - [Jaeger](https://www.jaegertracing.io/) end to end distributed request tracing. 33 | 34 | ## Quick Start 35 | 36 | ### Deploy a Cluster 37 | 38 | If you already have a Kubernetes cluster running and its context is the default, you can skip ahead to the "Deploying Infrastructure" section. 39 | 40 | We've included Terraform scripts for building a Kubernetes cluster with Azure AKS or ACS Engine, but would welcome pull requests for other cloud providers. 41 | 42 | To deploy a cluster, 43 | 44 | 1. Ensure you have the [az cli](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest) installed, in your path, and logged in to your subscription. 45 | 2. Edit cluster/environments/azure-aks/main.tf and adjust the name of the cluster and, if desired, any of the sizing or network parameters. 46 | 47 | 3. Deploy the cluster using: 48 | 49 | ``` 50 | $ cd cluster/environments/azure-aks 51 | $ ./init 52 | $ ./apply 53 | ``` 54 | 55 | ### Deploy Infrastructure 56 | 57 | 1. Install docker locally and confirm that it is in your path. We've encapsulated the rest of the dependencies in a Docker image, but you can also run these tools locally by installing the following set of tools locally as well: 58 | 59 | - [docker](https://docs.docker.com/docker-for-mac/install/) 60 | - [helm](https://helm.sh/) 61 | - [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) 62 | - [terraform](https://www.terraform.io/intro/getting-started/install.html) 63 | 64 | 2. If you haven't, create a new Kubernetes cluster with RBAC enabled and switch to it such that it is the default context `kubectl` is using. 65 | 66 | 3. Clone this project locally: 67 | 68 | ``` 69 | $ git clone https://github.com/Microsoft/bedrock 70 | ``` 71 | 72 | 4. Choose a password for your grafana deployment and set an environmental variable with it. 73 | 74 | ``` 75 | $ export TF_VAR_grafana_admin_password="SECRETpass" 76 | ``` 77 | 78 | 5. Deploy the dev configuration: 79 | 80 | ``` 81 | bash-4.4# cd infra/environments/dev 82 | bash-4.4# ./init 83 | bash-4.4# ./apply 84 | bash-4.4# cd ../../.. 85 | ``` 86 | 87 | 6. Take it for a test spin! 88 | 89 | ``` 90 | bash-4.4# tools/grafana 91 | 92 | NOTE: By default the credentials for grafana are 'ops' and the password you chose above. 93 | ``` 94 | 95 | Grafana provides a visualization of the metrics being collected by our cluster's Prometheus service -- and we've included a couple of Kubernetes related dashboards out of the box. 96 | 97 | ![Grafana Image](./docs/images/grafana.png) 98 | 99 | ``` 100 | bash-4.4# tools/kibana 101 | ``` 102 | 103 | Fluentd, Elasticsearch, and Kibana are installed and integrated with each other and your cluster -- ready for you to start querying and visualizing text logs immediately. 104 | 105 | ![Kibana Image](./docs/images/kibana.png) 106 | 107 | ``` 108 | bash-4.4# tools/traefik 109 | ``` 110 | 111 | Ingress traffic to the cluster is managed by Traefik, which includes a management console for monitoring the health and performance of your externally exposed services. 112 | 113 | ![Traefik Image](./docs/images/traefik.png) 114 | 115 | ``` 116 | bash-4.4# tools/jaeger 117 | ``` 118 | 119 | Jaeger provides distributed tracing of requests through your system so you can discover and optimize performance hotspots. 120 | 121 | ![Jaeger Image](./docs/images/jaeger.png) 122 | 123 | #### Using the Docker image 124 | 125 | If you'd like to avoid installing the tool dependencies, you can use our Docker container with these dependencies already installed: 126 | 127 | 1. Build the image locally: 128 | 129 | ``` 130 | $ docker build -t bedrock:latest . 131 | ``` 132 | 133 | 2. Choose a password for your grafana deployment and then start the container with your grafana password as an environmental variable and the kube config as a volume mount (the typical path for your kube config is ~/.kube/config below): 134 | 135 | ``` 136 | $ docker run --rm -it -v /config:/.kube/config -e TF_VAR_grafana_admin_password="SECRETpass" bedrock:latest /bin/bash 137 | 138 | bash-4.4# 139 | ``` 140 | 141 | From here, rejoin the quick start steps above. 142 | 143 | ### Deploy a Service 144 | 145 | We have also included terraform devops scripts for a [simple node.js service](https://github.com/timfpark/simple-service), giving you both a starting point for your own services, but also enabling you to see how all of the parts of the system fit together with a real service. 146 | 147 | Deploying it is as simple as: 148 | 149 | ``` 150 | $ set TF_VAR_container_repo=docker.io/timfpark <-- adjust this to your container repo 151 | $ cd services/environments/dev 152 | $ ./init && ./apply 153 | ``` 154 | 155 | You can then access the service externally by noting the public IP address of the Traefik service: 156 | 157 | ``` 158 | $ kubectl get services -n kube-system 159 | NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)... 160 | ... 161 | istio-ingressgateway LoadBalancer 10.0.184.127 40.123.43.171 80:31380/TCP,443:31390/TCP,31400:31400/TCP,15011:32347/TCP,8060:30113/TCP,853:31090/TCP,15030:32107/TCP,15031:30020/TCP 1h 162 | ... 163 | ``` 164 | 165 | and then using `curl` to inject the Host header for your request so Istio knows how to route it: 166 | 167 | ``` 168 | $ curl -HHost:simple.bedrock.tools http://40.123.43.171:80 169 | Your lucky number is 58 (instance id 65300 at Wed Nov 28 2018 21:46:46 GMT+0000 (UTC)) 170 | ``` 171 | 172 | ## Contributing 173 | 174 | We do not claim to have all the answers and would greatly appreciate your ideas and pull requests. 175 | 176 | This project welcomes contributions and suggestions. Most contributions require you to agree to a 177 | Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us 178 | the rights to use your contribution. For details, visit https://cla.microsoft.com. 179 | 180 | When you submit a pull request, a CLA-bot will automatically determine whether you need to provide 181 | a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions 182 | provided by the bot. You will only need to do this once across all repos using our CLA. 183 | 184 | This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). 185 | For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or 186 | contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments. 187 | 188 | For project related questions or comments, please contact (Tim Park)[https://github.com/timfpark]. 189 | 190 | ``` 191 | 192 | ``` 193 | --------------------------------------------------------------------------------