├── README.md ├── USER-GUIDE.md ├── bootstrapping ├── bootstrap-cluster-services.sh ├── cert-manager.clusterissuers.yaml ├── extra_userdata ├── fluent-bit.configmap.yaml ├── helm-values │ ├── cert-manager.yaml │ ├── efk.yaml │ ├── elasticsearch-curator.yaml │ ├── elasticsearch-operator.yaml │ └── nginx-ingress.yaml ├── pv.gotpl ├── storage-classes │ ├── gp2-storage-class.yaml │ ├── io1-slow-storage-class.yaml │ └── local-storage-class.yaml └── tiller-admin-rbac.yaml ├── generate_terraform_outputs.sh ├── terraform-aws-eks ├── backend.tf-sample ├── eks-cluster.tf ├── eks-worker-nodes.tf ├── outputs.tf-sample ├── providers.tf ├── variables.tf └── vpc.tf ├── tiller-admin-rbac-config.yaml ├── tiller.rbac.yaml └── your-values.yaml /README.md: -------------------------------------------------------------------------------- 1 | # Swarm on Kubernetes 2 | 3 | This document is targeted at developers who want to create a Kubernetes environment with Swarm and Geth applications running on it. 4 | 5 | Users of this setup should follow the [USER-GUIDE.md](https://github.com/ethersphere/swarm-kubernetes/blob/master/USER-GUIDE.md) 6 | 7 | Note that this setup is currently AWS specific. 8 | 9 | ## Table of Contents 10 | 1. [Kubernetes service](#kubernetes-service) 11 | 2. [Add Kubernetes Dashboard to cluster](#add-kubernetes-dashboard-to-cluster) 12 | 3. [Bootstrap auxiliary services](#bootstrap-auxiliary-services) 13 | 4. [Ethersphere Helm Charts](#ethersphere-helm-charts) 14 | 5. [Cluster monitoring with Prometheus and Grafana](#cluster-monitoring) 15 | 16 | 17 | ## Kubernetes service 18 | 19 | ### Terraform EKS playbooks 20 | 21 | Terraform playbooks for an AWS EKS (AWS Managed Kubernetes Service) with all related AWS resources (VPC, launch configurations, auto-scaling groups, security groups, etc.) 22 | 23 | 1. Update values in `backend.tf-sample`, and rename it to `backend.tf`. 24 | 25 | 2. Update users in `outputs.tf-sample`, and rename it to `outputs.tf`. The sample users `arn:aws:iam::123456789012:user/alice` and `arn:aws:iam::123456789012:user/bob` are added a admin for your Kubernetes environment. 26 | 27 | ``` 28 | mapUsers: | 29 | - userarn: arn:aws:iam::123456789012:user/alice 30 | username: alice 31 | groups: 32 | - system:masters 33 | - userarn: arn:aws:iam::123456789012:user/bob 34 | username: bob 35 | groups: 36 | - system:masters 37 | ``` 38 | 39 | 3. Review and update `variables.tf`. Note that this setup is running on AWS spot instances that could be terminated by AWS at any time. You will have to amend the Terraform scripts if you want to run on-demand instances. 40 | 41 | 4. Initialise Terraform and create the infrastructure. These Terraform EKS templates are heavily influenced by https://github.com/codefresh-io/eks-installer 42 | 43 | ``` 44 | terraform init 45 | 46 | terraform plan 47 | 48 | terraform apply 49 | ``` 50 | 51 | ### Update kubeconfig 52 | 53 | ``` 54 | export CLUSTER_NAME=your-cluster-name 55 | 56 | ./generate_terraform_outputs.sh 57 | 58 | aws eks update-kubeconfig --name $CLUSTER_NAME 59 | 60 | kubectl apply -f ./outputs/config-map-aws-auth.yaml 61 | ``` 62 | 63 | ## Add Kubernetes Dashboard to cluster 64 | 65 | ``` 66 | kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml 67 | kubectl apply -f eks-admin-and-cluster-role-binding.yaml 68 | 69 | echo "http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/" 70 | 71 | kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep eks-admin | awk '{print $1}') 72 | ``` 73 | 74 | ## Bootstrap auxiliary services 75 | 76 | Run `bootstrap-cluster-services.sh` in order to: 77 | 78 | 1. Setup storage classes 79 | 2. Add local persistence volumes to every i3.large instance in the cluster 80 | 3. Install tiller on kube-system to manage system wide components. 81 | 4. Install nginx ingress controller 82 | 5. Install cert-manager and cert issuers 83 | 6. Install logging stack 84 | 85 | ``` 86 | ./bootstrapping/bootstrap-cluster-services.sh 87 | ``` 88 | 89 | 90 | ## Ethersphere Helm Charts 91 | 92 | - [geth](https://github.com/ethersphere/helm-charts/tree/master/geth) 93 | - [swarm](https://github.com/ethersphere/helm-charts/tree/master/swarm) 94 | - [swarm-private](https://github.com/ethersphere/helm-charts/tree/master/swarm-private) 95 | 96 | ### Requirements 97 | 98 | You'll need access to a k8s cluster and the following binaries on your system: 99 | 100 | - [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) 101 | - [helm](https://github.com/helm/helm) 102 | 103 | ### Using the Helm charts 104 | 105 | To be able to use the [Ethersphere Helm charts](https://github.com/ethersphere/helm-charts), you need to load them from our registry first: 106 | 107 | ```sh 108 | helm repo add ethersphere-charts https://raw.githubusercontent.com/ethersphere/helm-charts-artifacts/master/ 109 | helm repo list 110 | ``` 111 | 112 | ### Create a namespace and deployer Tiller 113 | 114 | Tiller is the server portion of Helm and runs inside your Kubernetes cluster. 115 | 116 | We need to create a dedicated k8s namespace and deploy tiller there with proper RBAC 117 | to avoid that Tiller has full controll over our k8s cluster, 118 | 119 | This can be done like: 120 | 121 | ```sh 122 | export NAMESPACE=your-namespace 123 | 124 | # Create the namespace 125 | kubectl create namespace $NAMESPACE 126 | 127 | # Apply tiller Role Based Access Controlls to your namespace only 128 | kubectl -n $NAMESPACE apply -f tiller.rbac.yaml 129 | 130 | # Start tiller in your namespace 131 | helm init --service-account tiller --tiller-namespace $NAMESPACE 132 | ``` 133 | 134 | ### Deploy your chart 135 | 136 | Check out some examples on how to deploy your charts 137 | 138 | ```sh 139 | 140 | # Deploy the geth chart with default values 141 | helm --tiller-namespace=$NAMESPACE \ 142 | --namespace=$NAMESPACE \ 143 | --name=geth install ethersphere-charts/geth 144 | 145 | # Deploy the geth chart by providing your own custom-values.yaml file. 146 | # This will overwrite the default values. 147 | helm --tiller-namespace=$NAMESPACE \ 148 | --namespace=$NAMESPACE \ 149 | --name=geth install ethersphere-charts/geth \ 150 | -f custom-values.yaml 151 | 152 | ``` 153 | 154 | 155 | ## Cluster monitoring with Prometheus and Grafana 156 | 157 | Based on https://sysdig.com/blog/kubernetes-monitoring-prometheus-operator-part3/ 158 | 159 | ``` 160 | git clone https://github.com/coreos/prometheus-operator.git 161 | git clone https://github.com/mateobur/prometheus-monitoring-guide.git 162 | 163 | kubectl create -f prometheus-operator/contrib/kube-prometheus/manifests/ 164 | ``` 165 | -------------------------------------------------------------------------------- /USER-GUIDE.md: -------------------------------------------------------------------------------- 1 | # Swarm on Kubernetes - User Guide 2 | 3 | This document is targeted at developers who want to use a Kubernetes environment documented in the [README.md](https://github.com/ethersphere/swarm-kubernetes/blob/master/README.md) 4 | 5 | ## Table of Contents 6 | 1. [Requirements](#requirements) 7 | 2. [Configure kubectl](#configure-kubectl) 8 | 3. [Confirm access to K8s cluster](#confirm-access) 9 | 4. [Understand basic K8s/Tiller/Helm concepts](#basic-stuff) 10 | 5. [Deploy sample Swarm](#deploy-sample-swarm) 11 | 6. [Useful commands](#useful-commands) 12 | 13 | 14 | ### Requirements 15 | 16 | #### Install aws-cli 17 | 18 | https://aws.amazon.com/cli/ 19 | 20 | #### Setup your AWS Credentials 21 | 22 | ``` 23 | aws configure 24 | 25 | export AWS_ACCESS_KEY_ID="xxx" 26 | export AWS_SECRET_ACCESS_KEY="xxx" 27 | ``` 28 | 29 | #### Install aws-iam-authenticator 30 | 31 | https://github.com/kubernetes-sigs/aws-iam-authenticator 32 | 33 | Make sure you copy the aws-iam-authenticator binary in your PATH. 34 | 35 | #### Make sure your AWS user is added to the respective AWS EKS cluster 36 | 37 | Ask cluster administrator to apply this to the K8S cluster. 38 | 39 | - userarn: arn:aws:iam::123456789012:user/your-name 40 | username: your-name 41 | groups: 42 | - system:masters 43 | 44 | #### Set your default namespace 45 | 46 | ``` 47 | export NAMESPACE=your-namespace 48 | ``` 49 | 50 | #### Install `kubectl` and `helm` 51 | 52 | - [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) 53 | - [helm](https://github.com/helm/helm) 54 | 55 | ## Configure kubectl 56 | 57 | ``` 58 | aws eks update-kubeconfig --name your-cluster-name 59 | ``` 60 | 61 | ## Confirm access to K8s cluster 62 | 63 | ``` 64 | # list all nodes in the cluster 65 | kubectl get nodes 66 | ``` 67 | 68 | ## Understand basic K8s / Tiller / Helm concepts 69 | 70 | * kubectl, kube contexts (access to multiple clusters) 71 | 72 | * tiller 73 | 74 | * helm, charts (helm applications), configurations (values.yaml) 75 | 76 | ## Ethersphere Helm Charts 77 | 78 | To be able to use the [Ethersphere Helm charts](https://github.com/ethersphere/helm-charts), you need to load them from our registry first: 79 | 80 | ```sh 81 | helm repo add ethersphere-charts https://raw.githubusercontent.com/ethersphere/helm-charts-artifacts/master/ 82 | helm repo list 83 | ``` 84 | 85 | ## Configure your sample Swarm deployment 86 | 87 | ``` 88 | # your-values.yaml 89 | 90 | swarm: 91 | metricsEnabled: true 92 | tracingEnabled: false 93 | profilingEnabled: false 94 | image: 95 | repository: ethdevops/swarm 96 | tag: latest 97 | replicaCount: 2 98 | config: 99 | ens_api: http://mainnet-geth-geth.geth:8545 100 | verbosity: 3 101 | debug: true 102 | maxpeers: 25 103 | bzznetworkid: 3 104 | bootnodes: [] 105 | secrets: 106 | password: qwerty 107 | persistence: 108 | enabled: false 109 | ingress: 110 | domain: your-domain.com 111 | enabled: true 112 | tls: 113 | acmeEnabled: true 114 | ``` 115 | 116 | ## Deploy sample Swarm chart 117 | ```sh 118 | export NAMESPACE=your-name 119 | 120 | # Create the namespace 121 | kubectl create namespace $NAMESPACE 122 | 123 | # Apply tiller Role Based Access Controlls to your namespace only 124 | kubectl -n $NAMESPACE apply -f tiller.rbac.yaml 125 | 126 | # Start tiller in your namespace 127 | helm init --service-account tiller --tiller-namespace $NAMESPACE 128 | 129 | # Install sample Swarm chart 130 | helm --tiller-namespace=$NAMESPACE \ 131 | --namespace=$NAMESPACE \ 132 | --name=swarm install ethersphere-charts/swarm \ 133 | -f your-values.yaml 134 | 135 | # Tear-down Swarm deployment 136 | helm del --purge swarm --tiller-namespace $NAMESPACE 137 | 138 | # ... or remove k8s namespace altogether 139 | kubectl delete namespace $NAMESPACE 140 | 141 | # Upgrade sample Swarm chart 142 | helm --tiller-namespace=$NAMESPACE \ 143 | --namespace=$NAMESPACE \ 144 | upgrade swarm ethersphere-charts/swarm \ 145 | -f your-values.yaml 146 | 147 | # List helm applications 148 | helm list --tiller-namespace $NAMESPACE 149 | ``` 150 | 151 | ## Useful commands 152 | 153 | ```sh 154 | # list all nodes in the cluster 155 | kubectl get nodes 156 | 157 | # list all pods in the cluster 158 | kubectl get pods --all-namespaces 159 | 160 | # list all pods sorted by node 161 | kubectl get pods -o wide --sort-by="{.spec.nodeName}" --all-namespaces 162 | 163 | # list all namespaces 164 | kubectl get namespaces 165 | 166 | # attach to a given swarm container and to its javascript console 167 | kubectl exec -n $NAMESPACE -ti swarm-0 -- sh 168 | /geth attach /root/.ethereum/bzzd.ipc 169 | 170 | # open Kubernetes dashboard 171 | 172 | 1. get secret 173 | kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep eks-admin | awk '{print $1}') 174 | 175 | 2. run proxy 176 | kubectl proxy 177 | 178 | 3. open browser 179 | http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/ 180 | 181 | 182 | # port forwarding to k8s grafana service (credentials- admin:swarm) 183 | kubectl -n monitoring port-forward service/grafana 3000 184 | 185 | # port forwarding to kibana service 186 | kubectl -n logging port-forward service/efk-kibana 8443:443 187 | 188 | # port forwarding to access pprof on a swarm pod 189 | kubectl -n your_namespace port-forward swarm-0 6060 190 | 191 | # port forwarding to access grafana on a swarm stack 192 | kubectl -n your_namespace port-forward service/swarm-grafana 3001:80 193 | ``` 194 | -------------------------------------------------------------------------------- /bootstrapping/bootstrap-cluster-services.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | set -o errexit 3 | set -o pipefail 4 | set -o nounset 5 | set -o xtrace 6 | 7 | # Setup storage classes 8 | kubectl apply -f storage-classes/ 9 | 10 | # Add local persistence volumes to every i3.large instance in the cluster 11 | temp="$(mktemp)" 12 | kubectl get nodes -l 'beta.kubernetes.io/instance-type in (i3.large)' \ 13 | -o go-template --template="$(cat pv.gotpl)" > $temp 14 | kubectl apply -f $temp 15 | 16 | # Install tiller on kube-system to manage system wide components. 17 | kubectl apply -f tiller-admin-rbac.yaml 18 | helm init --tiller-namespace=kube-system --service-account tiller 19 | kubectl -n kube-system wait --timeout=120s --for condition=ready pod -l app=tiller 20 | 21 | 22 | # Add helm repos 23 | helm repo add incubator https://kubernetes-charts-incubator.storage.googleapis.com/ 24 | 25 | # Update information of available charts locally from chart repositories 26 | helm repo update 27 | 28 | # Install nginx ingress controller 29 | helm install --namespace=nginx-ingress \ 30 | --name=nginx-ingress stable/nginx-ingress --version=0.29.2 \ 31 | --values helm-values/nginx-ingress.yaml 32 | 33 | # Install cert-manager and cert issuers 34 | helm install --namespace kube-system \ 35 | --name cert-manager stable/cert-manager --version=v0.5.0 \ 36 | --values helm-values/cert-manager.yaml 37 | kubectl apply -f cert-manager.clusterissuers.yaml 38 | 39 | # Install logging stack 40 | helm repo add akomljen-charts \ 41 | https://raw.githubusercontent.com/komljen/helm-charts/master/charts/ 42 | 43 | helm install --namespace logging \ 44 | --name es-operator akomljen-charts/elasticsearch-operator --version=0.1.5 \ 45 | --values helm-values/elasticsearch-operator.yaml 46 | 47 | kubectl -n logging wait --timeout=120s --for condition=ready pod -l app=elasticsearch-operator 48 | 49 | kubectl apply -n logging -f fluent-bit.configmap.yaml 50 | helm install --namespace logging \ 51 | --name efk akomljen-charts/efk --version=0.1.2 \ 52 | --values helm-values/efk.yaml 53 | 54 | kubectl -n logging wait --timeout=120s --for condition=ready pod -l role=data,component=elasticsearch-efk-cluster 55 | 56 | helm install --namespace logging \ 57 | --name elasticsearch-curator stable/elasticsearch-curator --version=1.0.1 \ 58 | --values helm-values/elasticsearch-curator.yaml 59 | 60 | # Install spot instance termination handler 61 | helm install --namespace kube-system \ 62 | --name spot-termination incubator/kube-spot-termination-notice-handler --version=0.4.0 63 | 64 | # TODO: Install monitoring stack 65 | -------------------------------------------------------------------------------- /bootstrapping/cert-manager.clusterissuers.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: certmanager.k8s.io/v1alpha1 2 | kind: ClusterIssuer 3 | metadata: 4 | name: letsencrypt-staging 5 | spec: 6 | acme: 7 | # The ACME server URL 8 | server: https://acme-staging-v02.api.letsencrypt.org/directory 9 | # Email address used for ACME registration 10 | email: rafael@ethereum.org 11 | # Name of a secret used to store the ACME account private key 12 | privateKeySecretRef: 13 | name: letsencrypt-staging 14 | # Enable the HTTP-01 challenge provider 15 | http01: {} 16 | 17 | --- 18 | apiVersion: certmanager.k8s.io/v1alpha1 19 | kind: ClusterIssuer 20 | metadata: 21 | name: letsencrypt-prod 22 | spec: 23 | acme: 24 | # The ACME production server URL 25 | server: https://acme-v02.api.letsencrypt.org/directory 26 | # Email address used for ACME registration 27 | email: rafael@ethereum.org 28 | # Name of a secret used to store the ACME account private key 29 | privateKeySecretRef: 30 | name: letsencrypt-prod 31 | # Enable the HTTP-01 challenge provider 32 | http01: {} 33 | -------------------------------------------------------------------------------- /bootstrapping/extra_userdata: -------------------------------------------------------------------------------- 1 | #!/bin/bash -xe 2 | 3 | echo '{ "log-driver": "json-file", "log-opts": { "max-size": "20m", "max-file": "10" }}' > /etc/docker/daemon.json 4 | systemctl restart docker 5 | 6 | sed -i s,MAX_PODS,30,g /etc/systemd/system/kubelet.service 7 | 8 | INTERNAL_IP=$(curl -s http://169.254.169.254/latest/meta-data/local-ipv4) 9 | sed -i s,INTERNAL_IP,$INTERNAL_IP,g /etc/systemd/system/kubelet.service 10 | 11 | DNS_CLUSTER_IP=10.100.0.10 12 | if [[ $INTERNAL_IP == 10.* ]] ; then DNS_CLUSTER_IP=172.20.0.10; fi 13 | sed -i s,DNS_CLUSTER_IP,$DNS_CLUSTER_IP,g /etc/systemd/system/kubelet.service 14 | -------------------------------------------------------------------------------- /bootstrapping/fluent-bit.configmap.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: ConfigMap 3 | metadata: 4 | name: efk-fluent-bit-custom-config 5 | labels: 6 | app: efk-fluent-bit 7 | data: 8 | fluent-bit.conf: |- 9 | [SERVICE] 10 | Flush 1 11 | Daemon Off 12 | Log_Level info 13 | Parsers_File parsers.conf 14 | 15 | [INPUT] 16 | Name tail 17 | Path /var/log/containers/*.log 18 | Parser docker 19 | Tag kube.* 20 | Refresh_Interval 5 21 | Mem_Buf_Limit 5MB 22 | Skip_Long_Lines On 23 | DB /tail-db/tail-containers-state.db 24 | DB.Sync Normal 25 | 26 | [INPUT] 27 | Name systemd 28 | Tag systemd.kubelet.* 29 | Path /var/log/journal 30 | Systemd_Filter _SYSTEMD_UNIT=kubelet.service 31 | Strip_Underscores true 32 | DB /tail-db/systemd-kubelet-state.db 33 | DB.Sync Normal 34 | 35 | [INPUT] 36 | Name systemd 37 | Tag systemd.docker.* 38 | Path /var/log/journal 39 | Systemd_Filter _SYSTEMD_UNIT=docker.service 40 | Strip_Underscores true 41 | DB /tail-db/systemd-docker-state.db 42 | DB.Sync Normal 43 | 44 | [FILTER] 45 | Name kubernetes 46 | Match kube.* 47 | Kube_URL https://kubernetes.default.svc:443 48 | Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt 49 | Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token 50 | 51 | [OUTPUT] 52 | Name es 53 | Match kube.* 54 | Host elasticsearch-efk-cluster 55 | Port 9200 56 | Logstash_Format On 57 | Retry_Limit False 58 | Type flb_type 59 | Logstash_Prefix kubernetes_cluster 60 | 61 | [OUTPUT] 62 | Name es 63 | Match systemd.* 64 | Host elasticsearch-efk-cluster 65 | Port 9200 66 | Logstash_Format On 67 | Retry_Limit False 68 | Type flb_type 69 | Logstash_Prefix systemd 70 | parsers.conf: "" 71 | -------------------------------------------------------------------------------- /bootstrapping/helm-values/cert-manager.yaml: -------------------------------------------------------------------------------- 1 | ingressShim: 2 | defaultIssuerName: "letsencrypt-prod" 3 | defaultIssuerKind: "ClusterIssuer" 4 | -------------------------------------------------------------------------------- /bootstrapping/helm-values/efk.yaml: -------------------------------------------------------------------------------- 1 | # Default values for efk. 2 | 3 | # https://github.com/komljen/helm-charts/blob/master/kibana/values.yaml 4 | kibana: 5 | image: 6 | repository: "docker.elastic.co/kibana/kibana-oss" 7 | tag: "6.4.2" 8 | env: 9 | # All Kibana configuration options are adjustable via env vars. 10 | # To adjust a config option to an env var uppercase + replace `.` with `_` 11 | # Ref: https://www.elastic.co/guide/en/kibana/current/settings.html 12 | # 13 | ELASTICSEARCH_URL: http://elasticsearch-efk-cluster:9200 14 | SERVER_PORT: 5601 15 | LOGGING_VERBOSE: "true" 16 | SERVER_DEFAULTROUTE: "/app/kibana" 17 | 18 | # https://github.com/komljen/helm-charts/blob/master/elasticsearch/values.yaml 19 | elasticsearch: 20 | image: 21 | repository: quay.io/pires/docker-elasticsearch-kubernetes 22 | tag: "6.4.2" 23 | name: efk-cluster 24 | clientReplicas: 1 25 | masterReplicas: 1 26 | dataReplicas: 1 27 | dataVolumeSize: 100Gi 28 | javaOpts: "-Xms1024m -Xmx1024m" 29 | resources: 30 | requests: 31 | memory: 1024Mi 32 | cpu: 200m 33 | limits: 34 | memory: 2048Mi 35 | cpu: '1' 36 | storage: 37 | type: gp2 38 | classProvisioner: kubernetes.io/aws-ebs 39 | reclaimPolicy: Delete 40 | cerebro: 41 | enabled: true 42 | image: upmcenterprises/cerebro:0.6.8 43 | 44 | # https://github.com/kubernetes/charts/blob/master/stable/fluent-bit/values.yaml 45 | fluent-bit: 46 | image: 47 | fluent_bit: 48 | repository: fluent/fluent-bit 49 | tag: 0.14.7 50 | existingConfigMap: efk-fluent-bit-custom-config 51 | trackOffsets: true 52 | backend: 53 | type: es 54 | es: 55 | host: elasticsearch-efk-cluster 56 | port: 9200 57 | index: kubernetes_cluster 58 | logstash_prefix: kubernetes_cluster 59 | rbac: 60 | create: true 61 | -------------------------------------------------------------------------------- /bootstrapping/helm-values/elasticsearch-curator.yaml: -------------------------------------------------------------------------------- 1 | cronjob: 2 | # At 01:00 every day 3 | schedule: "0 1 * * *" 4 | concurrencyPolicy: "Forbid" 5 | failedJobsHistoryLimit: "3" 6 | successfulJobsHistoryLimit: "3" 7 | 8 | pod: 9 | annotations: {} 10 | 11 | image: 12 | repository: quay.io/pires/docker-elasticsearch-curator 13 | tag: 5.5.4 14 | pullPolicy: IfNotPresent 15 | 16 | config: 17 | elasticsearch: 18 | hosts: 19 | - elasticsearch-efk-cluster 20 | port: 9200 21 | 22 | configMaps: 23 | action_file_yml: |- 24 | --- 25 | actions: 26 | 1: 27 | action: delete_indices 28 | description: "Clean up ES by deleting old indices" 29 | options: 30 | timeout_override: 31 | continue_if_exception: False 32 | disable_action: False 33 | ignore_empty_list: True 34 | filters: 35 | - filtertype: age 36 | source: name 37 | direction: older 38 | timestring: '%Y.%m.%d' 39 | unit: days 40 | unit_count: 7 41 | field: 42 | stats_result: 43 | epoch: 44 | exclude: False 45 | -------------------------------------------------------------------------------- /bootstrapping/helm-values/elasticsearch-operator.yaml: -------------------------------------------------------------------------------- 1 | image: 2 | repository: upmcenterprises/elasticsearch-operator 3 | tag: 0.0.12 4 | pullPolicy: IfNotPresent 5 | resources: 6 | limits: 7 | cpu: 100m 8 | memory: 128Mi 9 | requests: 10 | cpu: 100m 11 | memory: 128Mi 12 | 13 | rbac: 14 | create: true 15 | 16 | serviceAccount: 17 | create: true 18 | -------------------------------------------------------------------------------- /bootstrapping/helm-values/nginx-ingress.yaml: -------------------------------------------------------------------------------- 1 | controller: 2 | replicaCount: 2 3 | minAvailable: 1 4 | resources: 5 | limits: 6 | cpu: 100m 7 | memory: 256Mi 8 | requests: 9 | cpu: 100m 10 | memory: 256Mi 11 | service: 12 | enableHttp: true 13 | enableHttps: true 14 | externalTrafficPolicy: "Local" 15 | targetPorts: 16 | http: http 17 | https: https 18 | type: LoadBalancer 19 | autoscaling: 20 | enabled: false 21 | minReplicas: 2 22 | maxReplicas: 4 23 | targetCPUUtilizationPercentage: 50 24 | targetMemoryUtilizationPercentage: 50 25 | extraContainers: [] 26 | extraInitContainers: [] 27 | extraVolumes: [] 28 | extraVolumeMounts: [] 29 | stats: 30 | enabled: false 31 | metrics: 32 | enabled: false 33 | 34 | defaultBackend: 35 | enabled: true 36 | name: default-backend 37 | image: 38 | repository: k8s.gcr.io/defaultbackend 39 | tag: "1.4" 40 | pullPolicy: IfNotPresent 41 | resources: 42 | limits: 43 | cpu: 10m 44 | memory: 20Mi 45 | requests: 46 | cpu: 10m 47 | memory: 20Mi 48 | 49 | rbac: 50 | create: true 51 | -------------------------------------------------------------------------------- /bootstrapping/pv.gotpl: -------------------------------------------------------------------------------- 1 | {{- range .items }} 2 | --- 3 | apiVersion: v1 4 | kind: PersistentVolume 5 | metadata: 6 | name: data-{{.metadata.name}} 7 | spec: 8 | capacity: 9 | storage: 450Gi 10 | accessModes: 11 | - ReadWriteOnce 12 | persistentVolumeReclaimPolicy: Retain 13 | storageClassName: local-storage 14 | local: 15 | path: /datadrive 16 | nodeAffinity: 17 | required: 18 | nodeSelectorTerms: 19 | - matchExpressions: 20 | - key: beta.kubernetes.io/instance-type 21 | operator: In 22 | values: 23 | - i3.large 24 | - key: kubernetes.io/hostname 25 | operator: In 26 | values: 27 | - {{.metadata.name}} 28 | {{end}} 29 | -------------------------------------------------------------------------------- /bootstrapping/storage-classes/gp2-storage-class.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: storage.k8s.io/v1 2 | kind: StorageClass 3 | metadata: 4 | name: gp2 5 | annotations: 6 | storageclass.kubernetes.io/is-default-class: "true" 7 | provisioner: kubernetes.io/aws-ebs 8 | parameters: 9 | type: gp2 10 | reclaimPolicy: Retain 11 | mountOptions: 12 | - debug 13 | -------------------------------------------------------------------------------- /bootstrapping/storage-classes/io1-slow-storage-class.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: storage.k8s.io/v1 2 | kind: StorageClass 3 | metadata: 4 | name: io1-slow 5 | provisioner: kubernetes.io/aws-ebs 6 | parameters: 7 | type: io1 8 | iopsPerGB: "10" 9 | fsType: ext4 10 | -------------------------------------------------------------------------------- /bootstrapping/storage-classes/local-storage-class.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: storage.k8s.io/v1 2 | kind: StorageClass 3 | metadata: 4 | name: local-storage 5 | provisioner: kubernetes.io/no-provisioner 6 | volumeBindingMode: WaitForFirstConsumer 7 | -------------------------------------------------------------------------------- /bootstrapping/tiller-admin-rbac.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: ServiceAccount 3 | metadata: 4 | name: tiller 5 | namespace: kube-system 6 | --- 7 | apiVersion: rbac.authorization.k8s.io/v1 8 | kind: ClusterRoleBinding 9 | metadata: 10 | name: tiller 11 | roleRef: 12 | apiGroup: rbac.authorization.k8s.io 13 | kind: ClusterRole 14 | name: cluster-admin 15 | subjects: 16 | - kind: ServiceAccount 17 | name: tiller 18 | namespace: kube-system 19 | -------------------------------------------------------------------------------- /generate_terraform_outputs.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | mkdir ./outputs 3 | 4 | pushd terraform-aws-eks 5 | 6 | terraform output config-map-aws-auth > ../outputs/config-map-aws-auth.yaml 7 | 8 | popd 9 | -------------------------------------------------------------------------------- /terraform-aws-eks/backend.tf-sample: -------------------------------------------------------------------------------- 1 | terraform { 2 | backend "s3" { 3 | bucket = "XXX" 4 | key = "us-east-1" 5 | region = "us-east-1" 6 | } 7 | } 8 | 9 | -------------------------------------------------------------------------------- /terraform-aws-eks/eks-cluster.tf: -------------------------------------------------------------------------------- 1 | # EKS Cluster Resources 2 | # * IAM Role to allow EKS service to manage other AWS services 3 | # * EC2 Security Group to allow networking traffic with EKS cluster 4 | # * EKS Cluster 5 | 6 | resource "aws_iam_role" "swarm-cluster" { 7 | name = "${var.cluster-name}-cluster" 8 | 9 | assume_role_policy = < /etc/docker/daemon.json 129 | systemctl restart docker 130 | 131 | CA_CERTIFICATE_DIRECTORY=/etc/kubernetes/pki 132 | CA_CERTIFICATE_FILE_PATH=$CA_CERTIFICATE_DIRECTORY/ca.crt 133 | mkdir -p $CA_CERTIFICATE_DIRECTORY 134 | echo "${aws_eks_cluster.swarm.certificate_authority.0.data}" | base64 -d > $CA_CERTIFICATE_FILE_PATH 135 | INTERNAL_IP=$(curl -s http://169.254.169.254/latest/meta-data/local-ipv4) 136 | sed -i s,MASTER_ENDPOINT,${aws_eks_cluster.swarm.endpoint},g /var/lib/kubelet/kubeconfig 137 | sed -i s,CLUSTER_NAME,${var.cluster-name},g /var/lib/kubelet/kubeconfig 138 | sed -i s,REGION,${data.aws_region.current.name},g /etc/systemd/system/kubelet.service 139 | sed -i s,MAX_PODS,20,g /etc/systemd/system/kubelet.service 140 | sed -i s,MASTER_ENDPOINT,${aws_eks_cluster.swarm.endpoint},g /etc/systemd/system/kubelet.service 141 | sed -i s,INTERNAL_IP,$INTERNAL_IP,g /etc/systemd/system/kubelet.service 142 | DNS_CLUSTER_IP=10.100.0.10 143 | if [[ $INTERNAL_IP == 10.* ]] ; then DNS_CLUSTER_IP=172.20.0.10; fi 144 | sed -i s,DNS_CLUSTER_IP,$DNS_CLUSTER_IP,g /etc/systemd/system/kubelet.service 145 | sed -i s,CERTIFICATE_AUTHORITY_FILE,$CA_CERTIFICATE_FILE_PATH,g /var/lib/kubelet/kubeconfig 146 | sed -i s,CLIENT_CA_FILE,$CA_CERTIFICATE_FILE_PATH,g /etc/systemd/system/kubelet.service 147 | systemctl daemon-reload 148 | systemctl restart kubelet 149 | USERDATA 150 | 151 | geth-node-userdata = <> /etc/fstab 158 | mount /datadrive 159 | 160 | echo '{ "log-driver": "json-file", "log-opts": { "max-size": "20m", "max-file": "10" }}' > /etc/docker/daemon.json 161 | systemctl restart docker 162 | 163 | CA_CERTIFICATE_DIRECTORY=/etc/kubernetes/pki 164 | CA_CERTIFICATE_FILE_PATH=$CA_CERTIFICATE_DIRECTORY/ca.crt 165 | mkdir -p $CA_CERTIFICATE_DIRECTORY 166 | echo "${aws_eks_cluster.swarm.certificate_authority.0.data}" | base64 -d > $CA_CERTIFICATE_FILE_PATH 167 | INTERNAL_IP=$(curl -s http://169.254.169.254/latest/meta-data/local-ipv4) 168 | sed -i s,MASTER_ENDPOINT,${aws_eks_cluster.swarm.endpoint},g /var/lib/kubelet/kubeconfig 169 | sed -i s,CLUSTER_NAME,${var.cluster-name},g /var/lib/kubelet/kubeconfig 170 | sed -i s,REGION,${data.aws_region.current.name},g /etc/systemd/system/kubelet.service 171 | sed -i s,MAX_PODS,20,g /etc/systemd/system/kubelet.service 172 | sed -i s,MASTER_ENDPOINT,${aws_eks_cluster.swarm.endpoint},g /etc/systemd/system/kubelet.service 173 | sed -i s,INTERNAL_IP,$INTERNAL_IP,g /etc/systemd/system/kubelet.service 174 | DNS_CLUSTER_IP=10.100.0.10 175 | if [[ $INTERNAL_IP == 10.* ]] ; then DNS_CLUSTER_IP=172.20.0.10; fi 176 | sed -i s,DNS_CLUSTER_IP,$DNS_CLUSTER_IP,g /etc/systemd/system/kubelet.service 177 | sed -i s,CERTIFICATE_AUTHORITY_FILE,$CA_CERTIFICATE_FILE_PATH,g /var/lib/kubelet/kubeconfig 178 | sed -i s,CLIENT_CA_FILE,$CA_CERTIFICATE_FILE_PATH,g /etc/systemd/system/kubelet.service 179 | systemctl daemon-reload 180 | systemctl restart kubelet 181 | USERDATA 182 | } 183 | 184 | resource "aws_launch_configuration" "swarm" { 185 | iam_instance_profile = "${aws_iam_instance_profile.swarm-node.name}" 186 | image_id = "${data.aws_ami.eks-worker.id}" 187 | instance_type = "${var.cluster-swarm-instance-type}" 188 | name_prefix = "${var.cluster-name}" 189 | security_groups = ["${aws_security_group.swarm-node.id}"] 190 | user_data_base64 = "${base64encode(local.swarm-node-userdata)}" 191 | key_name = "${var.cluster-key-name}" 192 | spot_price = "${var.cluster-swarm-spot-bid-price}" 193 | associate_public_ip_address = true 194 | 195 | lifecycle { 196 | create_before_destroy = true 197 | } 198 | } 199 | 200 | resource "aws_autoscaling_group" "swarm" { 201 | desired_capacity = "${var.swarm-desired-capacity}" 202 | launch_configuration = "${aws_launch_configuration.swarm.id}" 203 | min_size = "${var.swarm-desired-capacity}" 204 | max_size = "${var.swarm-desired-capacity}" 205 | name = "${var.cluster-name}" 206 | vpc_zone_identifier = ["${aws_subnet.swarm.*.id}"] 207 | 208 | tag { 209 | key = "Name" 210 | value = "${var.cluster-name}" 211 | propagate_at_launch = true 212 | } 213 | 214 | tag { 215 | key = "kubernetes.io/cluster/${var.cluster-name}" 216 | value = "owned" 217 | propagate_at_launch = true 218 | } 219 | } 220 | 221 | resource "aws_launch_configuration" "geth" { 222 | iam_instance_profile = "${aws_iam_instance_profile.swarm-node.name}" 223 | image_id = "${data.aws_ami.eks-worker.id}" 224 | instance_type = "${var.cluster-geth-instance-type}" 225 | name_prefix = "${var.cluster-name}-geth" 226 | security_groups = ["${aws_security_group.swarm-node.id}"] 227 | user_data_base64 = "${base64encode(local.geth-node-userdata)}" 228 | key_name = "${var.cluster-key-name}" 229 | spot_price = "${var.cluster-geth-spot-bid-price}" 230 | associate_public_ip_address = true 231 | 232 | lifecycle { 233 | create_before_destroy = true 234 | } 235 | } 236 | 237 | resource "aws_autoscaling_group" "geth" { 238 | desired_capacity = "${var.geth-desired-capacity}" 239 | launch_configuration = "${aws_launch_configuration.geth.id}" 240 | min_size = "${var.geth-desired-capacity}" 241 | max_size = "${var.geth-desired-capacity}" 242 | name = "${var.cluster-name}-geth" 243 | vpc_zone_identifier = ["${aws_subnet.swarm.*.id}"] 244 | 245 | tag { 246 | key = "Name" 247 | value = "${var.cluster-name}-geth" 248 | propagate_at_launch = true 249 | } 250 | 251 | tag { 252 | key = "kubernetes.io/cluster/${var.cluster-name}" 253 | value = "owned" 254 | propagate_at_launch = true 255 | } 256 | } 257 | -------------------------------------------------------------------------------- /terraform-aws-eks/outputs.tf-sample: -------------------------------------------------------------------------------- 1 | locals { 2 | config-map-aws-auth = <