├── .gitignore ├── README.md ├── ansible ├── LICENSE ├── README.md ├── ansible.cfg ├── inventory ├── k8s-remove.yml ├── k8s.retry ├── k8s.yml └── roles │ ├── common │ └── tasks │ │ ├── create_sshkey.yml │ │ └── main.yml │ ├── k8s-remove │ └── tasks │ │ ├── delete_context.yml │ │ ├── delete_inv.yml │ │ ├── delete_secgroup.yml │ │ ├── delete_secgroup_rules.yml │ │ ├── delete_vm.yml │ │ └── main.yml │ └── k8s │ ├── tasks │ ├── create_context.yml │ ├── create_inv.yml │ ├── create_secgroup.yml │ ├── create_secgroup_rules.yml │ ├── create_vm.yml │ └── main.yml │ └── templates │ ├── inventory.j2 │ ├── k8s-master-rbac.j2 │ ├── k8s-master.j2 │ └── k8s-node.j2 ├── kubernetes-dashboard ├── kubernetes-dashboard-deployment.yaml └── kubernetes-dashboard-service.yaml ├── logging ├── _logging-namespace.yaml ├── dashboards │ └── elk-v1.json ├── elasticsearch-deployment.yaml ├── elasticsearch-service.yaml ├── fluentd-daemonset.yaml ├── kibana-deployment.yaml └── kibana-service.yaml ├── media ├── grafana-k8s-pod-resources1.JPG ├── grafana-k8s-pod-resources2.JPG ├── grafana-prometheus-stats.JPG ├── grafana2-cluster1.JPG ├── grafana2-cluster2.JPG ├── grafana2-cluster3.JPG ├── grafana2-pods.JPG ├── k8s-infra1.JPG ├── k8s-kibana.JPG ├── kubernetes-dashboard-addon.JPG ├── prometheus.JPG └── traefik.JPG ├── monitoring ├── _monitoring-namespace.yaml ├── grafana-deployment.yaml ├── grafana-import-dashboards-job.yaml ├── grafana-service.yaml ├── kube-state-metrics-deployment.yaml ├── kube-state-metrics-service.yaml ├── node-directory-size-metrics-daemonset.yaml ├── prometheus-deployment.yaml ├── prometheus-node-exporter-daemonset.yaml ├── prometheus-node-exporter-service.yaml └── prometheus-service.yaml ├── monitoring2 ├── _monitoring2-namespace.yaml ├── grafana-deployment.yaml ├── grafana-service.yaml ├── heapster-deployment.yaml ├── heapster-service.yaml ├── influxdb-deployment.yaml └── influxdb-service.yaml ├── service-loadbalancer-daemonset.yaml ├── tls ├── ca-config.json ├── ca-csr.json ├── ca │ ├── ca-key.pem │ └── ca.pem ├── kube-apiserver-server-csr.json ├── kube-proxy-client-csr.json ├── kubectl │ ├── kubernetes-admin-user-key.pem │ └── kubernetes-admin-user.pem ├── kubernetes-admin-user.csr.json ├── master │ ├── ca.pem │ ├── kube-apiserver-server-key.pem │ ├── kube-apiserver-server.pem │ └── policy.jsonl └── minion │ ├── ca.pem │ ├── kube-proxy │ ├── kube-proxy-client-key.pem │ ├── kube-proxy-client.pem │ └── proxy.kubeconfig │ └── kubelet │ ├── kubelet-client-key.pem │ ├── kubelet-client.pem │ └── kubelet.kubeconfig ├── traefik ├── kube-system-ingress.yaml ├── logging-ingress.yaml ├── monitoring-ingress.yaml ├── monitoring2-ingress.yaml ├── traefik-daemonset.yaml └── traefik-service.yaml └── utils ├── kubedns-svc.yaml ├── postgres.yaml └── ubuntu.yaml /.gitignore: -------------------------------------------------------------------------------- 1 | tls/* 2 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Deploy kubernetes via ansible (on cloudstack servers) with logging (efk) & monitoring (prometheus) support # 2 | 3 | ![k8s_Infra1.jpg](https://github.com/gregbkr/kubernetes-ansible-logging-monitoring/raw/master/media/k8s-infra1.JPG) 4 | 5 | ## Updates ;-) 6 | An even better setup with kargo (deploy to aws, azure, baremetal, and hot k8s migration) is available here --> https://github.com/gregbkr/kubernetes-kargo-logging-monitoring 7 | 8 | ## What you will get: 9 | - 1 master node running : k8s for container orchestration, it will pilot and gives work to the minions 10 | - 2(or more) minion/slave/worker nodes : running the actual containers and doing the actual work 11 | - Efk: we will send all k8s container logs to an elasticsearch DB, via fluentd, and visualize dashboards with kibana 12 | - Prometheus will monitoring all this infra, with grafana dashbaord 13 | - Heapster is an alternative for monitoring your k8s cluster 14 | - K8s dashboard addon (not efk dashboard), where you can visualize k8s component in a GUI 15 | - Service-loadbalancer (static haproxy): which is the public gateway to access your internal k8s services (kibana, grafana) 16 | - Dynamic loadbalancer (traefik): an alternative to haproxy, quite powerful with its dynamic service discovery and auto certification 17 | 18 | *Prerequisit:* 19 | - Cloudstack cloud provider (ex: exoscale) / but you can deploy anywhere else with a bit of adaptation in ansible. Deploying logging and monitoring stay the same at the moment you have k8s running 20 | - A vm (ubuntu) with ansible installed, where you will run recipes, and manage k8s with kubeclt 21 | 22 | More info: you can find an overview of that setup on my blog: https://greg.satoshi.tech/ 23 | 24 | # 1. Deploy kubernetes 25 | 26 | ### 1.1 Clone repo 27 | 28 | git clone https://github.com/gregbkr/kubernetes-ansible-logging-monitoring.git k8s && cd k8s 29 | 30 | ### 1.2 Install kubectl 31 | 32 | Kubeclt is your admin local client to pilot the k8s cluster. 33 | Please use the same version as server. You will be able to talk and pilot k8s with this tool. 34 | 35 | curl -O https://storage.googleapis.com/kubernetes-release/release/v1.4.6/bin/linux/amd64/kubectl 36 | chmod +x kubectl 37 | mv kubectl /usr/local/bin/kubectl 38 | 39 | Autocompletion 40 | 41 | source <(kubectl completion bash) 42 | kubeclt get nod +[TAB] 43 | 44 | If issues, see troubleshooting section. 45 | 46 | 47 | ### 1.3 Deploy k8s for a cloudstack infra 48 | 49 | I will use the nice setup made by Seb: https://www.exoscale.ch/syslog/2016/05/09/kubernetes-ansible/ 50 | I just added few lines in file: ansible/roles/k8s/templates/k8s-node.j2 to be able to collect logs with fluentd 51 | (# In order to have logs in /var/log/containers to be pickup by fluentd 52 | Environment="RKT_OPTS=--volume dns,kind=host,source=/etc/resolv.conf --mount volume=dns,target=/etc/resolv.conf --volume var-log,kind=host,source=/var/log --mount volume=var-log,target=/var/log" ) 53 | 54 | nano ansible/k8s.yml <-- edit k8s version, num_node, ssh_key if you want to use your own 55 | 56 | Next step will create firewall rules k8s, master and minion nodes, and install k8s components 57 | Run recipe: 58 | 59 | ansible-playbook ansible/k8s.yml 60 | watch kubectl get node <-- wait for the nodes to be up 61 | 62 | ### 1.4 Checks: 63 | 64 | kubectl get all --all-namespaces <-- should have no error here 65 | 66 | # 2. Deploy logging (efk) to collect k8s & containers events 67 | 68 | ### 2.1 Deploy elasticsearch, fluentd, kibana 69 | 70 | kubectl apply -f logging <-- all deployment declarations and configurations are here 71 | 72 | kubectl get all --all-namespaces <-- if you see elasticsearch container restarting, please restart all nodes one time only (setting vm.max_map_count, see troubleshooting section) 73 | 74 | ### 2.2 Access services 75 | 76 | From here, you should be able to access our services from your laptop, as long as your cloud server ip are public: 77 | 78 | - kibana: http://any_minion_node_ip:30601 79 | - ES: http://any_minion_node_ip:30200 80 | 81 | To enable that access, we had set Type=NodePort and nodePort:35601/39200 in kibana/elasticsearch-service.yaml, to make it easier to learn at this point. 82 | Because we want to control how and from where we should be accessing our public services, we will set in a later section a loadbalancer. 83 | 84 | ### 2.3 See logs in kibana 85 | 86 | Check logs coming in kibana, you just need to refresh, select Time-field name : @timestamps + create 87 | 88 | Load and view your first dashboard: management > Saved Object > Import > logging/dashboards/elk-v1.json 89 | 90 | ![k8s-kibana.jpg](https://github.com/gregbkr/kubernetes-ansible-logging-monitoring/raw/master/media/k8s-kibana.JPG) 91 | 92 | 93 | # 3. Monitoring services and containers 94 | 95 | It seems like two schools are gently fighting for container monitoring: 96 | 97 | - Heapster: this new player now comes as a kind of k8s addon (you can deploy it via a simple switch in some setup). It seems to be better integrated at the moment, and even more in the future with k8s component depending on it, but still young and few features 98 | - Prometheus: it has been around for some times, lots of nice features (alerting, application metrics) and community resources available (see the public dashboards for example) 99 | 100 | More on which one to choose: https://github.com/kubernetes/heapster/issues/645 101 | 102 | ### 3.1 Monitoring with prometheus 103 | 104 | Create monitoring containers 105 | 106 | kubectl apply -f monitoring 107 | kubectl get all --namespace=monitoring 108 | 109 | **Prometheus** 110 | 111 | Access the gui: http://any_minion_node_ip:30090 112 | 113 | Go to status > target : you should see only some green. 114 | *We got one false positive error scaping k8s-node with 2 ports 9102 and 80. As long as 9102 is good, we got the data. 115 | If you got some "context deadline exceeded" or "getsockopt connection refused", you will have to open firewall rule between the nodes. For exemple in security group k8s, you need to open 9100 and 10255.* 116 | 117 | Try a query: "node_memory_Active" > Execute > Graph --> you should see 2 lines representing both nodes. 118 | 119 | ![prometheus.jpg](https://github.com/gregbkr/kubernetes-ansible-logging-monitoring/raw/master/media/prometheus.JPG) 120 | 121 | 122 | 123 | **Grafana** 124 | 125 | Login to the interface with login:admin | pass:admin) : http://any_minion_node_ip:30000 126 | Load some dashboards: dashboard > home 127 | 128 | **Kubernetes pod resources** 129 | ![grafana-k8s-pod-resources1.jpg](https://github.com/gregbkr/kubernetes-ansible-logging-monitoring/raw/master/media/grafana-k8s-pod-resources1.JPG) 130 | ![grafana-k8s-pod-resources2.jpg](https://github.com/gregbkr/kubernetes-ansible-logging-monitoring/raw/master/media/grafana-k8s-pod-resources2.JPG) 131 | 132 | 133 | **Prometheus stats** 134 | ![grafana-prometheus-stats.jpg](https://github.com/gregbkr/kubernetes-ansible-logging-monitoring/raw/master/media/grafana-prometheus-stats.JPG) 135 | 136 | **Load other public dashboards** 137 | 138 | Grafana GUI > Dashboards > Import 139 | 140 | Already loaded: 141 | - prometheus stats: https://grafana.net/dashboards/2 142 | - kubernetes cluster : https://grafana.net/dashboards/162 143 | 144 | Other good dashboards : 145 | 146 | - node exporter: https://grafana.net/dashboards/704 - https://grafana.net/dashboards/22 147 | 148 | - deployment: pod metrics: https://grafana.net/dashboards/747 - pod resources: https://grafana.net/dashboards/737 149 | 150 | ### 3.2 Monitoring2 with heapster 151 | 152 | kubectl apply -f monitoring2 153 | kubectl get all --namespace=monitoring2 154 | 155 | **Access services** 156 | 157 | - Grafana2: http://any_minion_node_ip:30002 158 | 159 | You can load Cluster or Pods dashboards. When viewing Pods, type manually "namespace=monitoring2" to view stats for the related containers. 160 | 161 | ![grafana2-pods.jpg](https://github.com/gregbkr/kubernetes-ansible-logging-monitoring/raw/master/media/grafana2-pods.JPG) 162 | ![grafana2-cluster1.jpg](https://github.com/gregbkr/kubernetes-ansible-logging-monitoring/raw/master/media/grafana2-cluster1.JPG) 163 | ![grafana2-cluster1.jpg](https://github.com/gregbkr/kubernetes-ansible-logging-monitoring/raw/master/media/grafana2-cluster2.JPG) 164 | ![grafana2-cluster1.jpg](https://github.com/gregbkr/kubernetes-ansible-logging-monitoring/raw/master/media/grafana2-cluster3.JPG) 165 | 166 | # 4. Kubenetes dashboard addon (not logging efk) 167 | 168 | Dashboard addon let you see k8s services and containers via a nice GUI. 169 | 170 | kubectl apply -f kubernetes-dashboard 171 | kubectl get all --namespace=kube-system <-- carefull dashboard is running in namespace=kube-system 172 | 173 | Access GUI: http://any_minion_node_ip:30999 174 | 175 | ![kubernetes-dashboard.jpg](https://github.com/gregbkr/kubernetes-ansible-logging-monitoring/raw/master/media/kubernetes-dashboard-addon.JPG) 176 | 177 | # 5. LoadBalancers 178 | 179 | If you are on aws or google cloud, these provider we automatically set a loadbalancer matching the *-ingress.yaml configuration. For all other cloud provider and baremetal, you will have to take care of that step. Luckyly, I will present you two types of loadlancer below ;-) 180 | - service-loadbalancer (static haproxy) https://github.com/kubernetes/contrib/tree/master/service-loadbalancer 181 | - traefik (dynamic proxy) https://github.com/containous/traefik 182 | 183 | ### 5.1 Service-loadbalancer 184 | 185 | Create the load-balancer to be able to connect your service from the internet. 186 | Give 1 or more nodes the loadbalancer role: 187 | 188 | kubectl label node 185.19.30.121 role=loadbalancer 189 | kubectl apply -f service-loadbalancer-daemonset.yaml 190 | 191 | *If you change the config, use "kubectl delete -f service-loadbalancer.yaml" to force a delete/create, then the discovery of the newly created service. 192 | Add/remove services? please edit service-loadbalancer.yaml* 193 | 194 | **Access services** 195 | 196 | - kibana (logging): http://lb_node_ip:5601 197 | - grafana (monitoring): http://lb_node_ip:3000 (admin/admin) 198 | - prometheus (monitoring): http://lb_node_ip:3000 199 | - grafana2 (monitoring2): http://lb_node_ip:3002 200 | - kubernetes-dashboard: http://lb_node_ip:9999 201 | 202 | ### 5.2 Traefik 203 | 204 | Any news services, exposed by *-ingress.yaml, will be caught by traefik and made available without restart. 205 | 206 | To experience the full power of traefik, please purchase a domain name (ex: satoshi.tech), and point that record to the node you choose to be the lb. This record will help create the automatic certificate via the acme standard. 207 | 208 | - satoshi.tech --> lb_node_ip 209 | 210 | Then for each services you will use, create a dns A record: 211 | 212 | - kibana.satoshi.tech --> lb_node_ip 213 | - grafana.satoshi.tech --> lb_node_ip 214 | - prometheus.satoshi.tech --> lb_node_ip 215 | - grafana2.satoshi.tech --> lb_node_ip 216 | - kubernetes-dashboard.satoshi.tech --> lb_node_ip 217 | - traefik-console.satoshi.tech --> lb_node_ip 218 | 219 | Based on which name you use to access the lb_node, traefik will forward to the right k8s service. 220 | 221 | Now you need to edit the configuration: 222 | 223 | nano traefik/traefik-daemonset.yaml 224 | [acme] <-- set your data for auto certification 225 | 226 | Create the dynamic proxy to be able to connect your service from the internet. 227 | 228 | kubectl apply -f traefik <-- if error, probably because you didn't deploy other namespaces, so can ignore 229 | kubectl get all --all-namespaces <-- if traefik pod can't get created, probably issue with port 443 on loadbalancer --> see troubleshooting section 230 | 231 | **Access services** 232 | If set in traefik, please use login/pass: test/test 233 | You can use http or https 234 | 235 | - kibana (logging): http://kibana.satoshi.tech 236 | - grafana (monitoring): http:grafana.satoshi.tech (admin/admin) 237 | - prometheus (monitoring): http://prometheus.satoshi.tech 238 | - grafana2 (monitoring2): http://grafana2.satoshi.tech 239 | - kubernetes-dashboard: http://kubernetes-dashboard.satoshi.tech 240 | - traefik-console: http://traefik-console.satoshi.tech or http://any_minion_node_ip:30080 241 | 242 | ![traefik.jpg](https://github.com/gregbkr/kubernetes-ansible-logging-monitoring/raw/master/media/traefik.JPG) 243 | 244 | ### 5.3 Security considerations 245 | 246 | These lb nodes are some kind of DMZ servers where you could balance later your DNS queries. 247 | For production environment, I would recommend that only DMZ services (service-loadbalancer, traefik, nginx, ...) could run in here, because these servers will apply some less restrictive firewall rules (ex: open 80, 433, 5601, 3000) than other minion k8s nodes. 248 | So I would create a second security group (sg): k8s-dmz with same rules as k8s, and rules between both zone, so k8s services can talk to k8s and k8s-dmz. Then open 80, 433, 5601, 3000 for k8s-dmz only. Like this, k8s sg still protect more sensitive containers from direct public access/scans. 249 | 250 | The same applies for the master node. I would create a new sg for it: k8s-master, so only this group will permit access from kubeclt (port 80, 443). 251 | 252 | Then you should remove all NodePort from the services configuration, so no service will be available when scanning a classic minion. For that please comment the section "# type: NodePort" for all *-service.yaml 253 | 254 | ### 5.4 Scaling loadbalancers 255 | 256 | Add more loadbalancers, by adding more loadbalancers nodes. Because we use Daemonset type of job, all new nodes tagged with loadbalancer will spawn a loadbalancer container. 257 | 258 | Use ansible to add a node 259 | 260 | nano ansible/k8s.yml <-- edit: k8s_num_nodes: 3 261 | ansible-playbook ansible/k8s.yml 262 | kubeclt get node <-- wait for it! 263 | 264 | Label it as a loadbalancer node 265 | 266 | kubectl label node your_new_lb_minion_node role=loadbalancer 267 | 268 | Then just check the new containers getting created 269 | kubectl get all --all-namespaces 270 | 271 | For service-loadbalancer, try to access new_lb_minion_ip:5601 272 | 273 | For trafik, add a dns A-record kibana.satoshi.tech --> new_lb_minion_ip so we will balance dns resolution to the old and new lb_node. 274 | Test some ping, and access kibana.satoshi.tech few times... 275 | 276 | 277 | # 6. Secure your k8s access with certificates (optional demonstration) 278 | 279 | kubectl pilot k8s via the api server already on a secured port 443 in https. 280 | We will now create a certicate autority, to issue a certificate for the api, and for your admin client, to get even higher level of authentification. 281 | 282 | The lines below are based on the work of Kelsey and the cloudflare easy CA. 283 | https://github.com/kelseyhightower/docker-kubernetes-tls-guide 284 | https://github.com/cloudflare/cfssl 285 | 286 | Install your own PKI infra tool: cfssl 287 | You need GO 1.6+ and the GOPATH set 288 | 289 | apt install golang-go 290 | export GOPATH=$HOME/work 291 | export PATH=$PATH:/usr/local/go/bin 292 | export PATH=$PATH:$GOPATH/bin 293 | 294 | go get -u github.com/cloudflare/cfssl/cmd/cfssl 295 | go get -u github.com/cloudflare/cfssl/cmd/... 296 | 297 | cd tls 298 | mkdir kubectl master minion 299 | 300 | Declare your master Ip (or domain) to the server cert 301 | 302 | nano kube-apiserver-server-csr.json <-- add your master_ip in hosts section 303 | 304 | **Initialize a CA** 305 | 306 | cfssl gencert -initca ca-csr.json | cfssljson -bare ca/ca 307 | 308 | **Create an api server cert** 309 | 310 | ``` 311 | cfssl gencert \ 312 | -ca=ca/ca.pem \ 313 | -ca-key=ca/ca-key.pem \ 314 | -config=ca-config.json \ 315 | -profile=server \ 316 | kube-apiserver-server-csr.json | cfssljson -bare master/kube-apiserver-server 317 | ``` 318 | 319 | cp ca/ca.pem master 320 | 321 | **Create kubeclt client cert** 322 | 323 | ``` 324 | cfssl gencert \ 325 | -ca=ca/ca.pem \ 326 | -ca-key=ca/ca-key.pem \ 327 | -config=ca-config.json \ 328 | -profile=client \ 329 | kubernetes-admin-user.csr.json | cfssljson -bare kubectl/kubernetes-admin-user 330 | ``` 331 | 332 | kubectl config set-cluster secure --server=https://185.19.30.189:443 --certificate-authority=master/ca.pem --embed-certs=true 333 | 334 | ``` 335 | kubectl config set-credentials admin \ 336 | --client-key=kubectl/kubernetes-admin-user-key.pem \ 337 | --client-certificate=kubectl/kubernetes-admin-user.pem \ 338 | --embed-certs=true 339 | ``` 340 | 341 | kubectl config set-context secure --cluster=secure --user=admin 342 | kubectl config use-context secure 343 | 344 | Copy tls/master folder to node master 345 | 346 | scp -r -i ~/.ssh/id_rsa_foobar master core@185.19.30.189:/home/core 347 | 348 | Edit master 349 | 350 | ssh -i ~/.ssh/id_rsa_foobar core@185.19.30.189 351 | 352 | mkdir /etc/kubernetes 353 | mv master/* /etc/kubernetes/. 354 | 355 | ``` 356 | sudo vim /etc/systemd/system/kube-apiserver.service 357 | 358 | --client-ca-file=/etc/kubernetes/ca.pem \ 359 | --tls-cert-file=/etc/kubernetes/kube-api-server-server.pem \ 360 | --tls-private-key-file=/etc/kubernetes/kube-apiserver-server-key.pem \ 361 | ``` 362 | 363 | sudo systemctl daemon-reload 364 | sudo systemctl restart kube-apiserver.service 365 | 366 | *Note: this systemclt config will be erased at the next reboot. If you want to use cert, edit ansible master node template with the right config, then deploy a new master. Then generate certs...* 367 | 368 | Test 369 | 370 | curl --cert kubectl/kubernetes-admin-user.pem --key kubectl/kubernetes-admin-user-key.pem --cacert master/ca.pem https://185.19.30.189/api -v 371 | kubectl get node 372 | 373 | 374 | **Create kubelet client cert** 375 | 376 | ``` 377 | cfssl gencert \ 378 | -ca=ca/ca.pem \ 379 | -ca-key=ca/ca-key.pem \ 380 | -config=ca-config.json \ 381 | -profile=client \ 382 | kubelet-client-csr.json | cfssljson -bare minion/kubelet/kubelet-client 383 | ``` 384 | 385 | Edit minion node 386 | 387 | Copy and mv the file: 388 | 389 | scp -r -i ~/.ssh/id_rsa_foobar minion core@185.19.30.189:/home/core 390 | ssh -i ~/.ssh/id_rsa_foobar core@185.19.3.31 391 | 392 | mkdir /etc/kubernetes 393 | mv minion/* /etc/kubernetes/. 394 | 395 | ``` 396 | sudo vim /etc/systemd/system/kube-kubelet.service 397 | --api-servers=https://185.19.30.189:443 \ 398 | --kubeconfig=/etc/kubernetes/kubelet/kubelet.kubeconfig \ 399 | ``` 400 | 401 | sudo systemctl daemon-reload 402 | sudo systemctl restart kube-apiserver.service 403 | 404 | Check logs: 405 | 406 | journalctl --since "10 minutes ago" -u kube-kubelet --no-pager 407 | 408 | Try 409 | 410 | kubectl get node 411 | 412 | *All services (kube-proxy, kube-client, kube-controller) can be set to use certificate. But this is a subject for another setup.* 413 | 414 | # 7. Troubleshooting 415 | 416 | ### Kubectl autocompletion not working 417 | 418 | Probably your custom bash profile is interfering with the sourcing. 419 | - Use kubectl completion bash > /etc/bash_completion.d/kubectl 420 | - And make sure that lines are uncomment in 421 | ``` 422 | nano ~/.bashrc 423 | if [ -f /etc/bash_completion ] && ! shopt -oq posix; then 424 | . /etc/bash_completion 425 | fi 426 | ``` 427 | Then logon and try again. 428 | 429 | ### If problem starting elasticsearch v5: (fix present in roles/k8s/templates/k8s-node.j2) 430 | - manually on all node: fix an issue with hungry es v5 431 | ``` 432 | ssh -i ~/.ssh/id_rsa_foobar core@185.19.29.212 433 | sudo sysctl -w vm.max_map_count=262144 434 | ``` 435 | 436 | - make it persistent: 437 | ``` 438 | sudo vi /etc/sysctl.d/elasticsearch.conf 439 | vm.max_map_count=262144 440 | sudo sysctl --system 441 | ``` 442 | 443 | ### If issue connecting to svc (for example elasticsearch), use ubuntu container: 444 | - First see if ubuntu will be in the same namespace as the service you want to check: 445 | 446 | ``` 447 | nano utils/ubuntu.yaml 448 | kubectl apply -f utils/ubuntu.yaml 449 | ``` 450 | 451 | - Depending in which namespace ubuntu runs, you can check services with one of these commands: 452 | 453 | ``` 454 | kubectl exec ubuntu -- curl elasticseach:9200 <-- should returns ... "cluster_name" : "elasticsearch"... 455 | kubectl exec ubuntu -- curl kibana:5601 <-- should returns ... var defaultRoute = '/app/kibana'... 456 | 457 | kubectl exec ubuntu -- curl elasticsearch.logging.svc.cluster.local:9200 <-- ubuntu in default namespace 458 | kubectl exec ubuntu --namespace=logging -- nslookup elasticsearch <-- ubuntu in logging namespace 459 | kubectl exec ubuntu --namespace=logging -- nslookup kubernetes.default.svc.cluster.local <-- ubuntu in logging namespace 460 | ``` 461 | 462 | - Check port 9200 on the node running elasticsearch container: ssh -i ~/.ssh/id_rsa_foobar core@185.19.29.212 netstat -plunt 463 | - Uncomment type: NodePort and nodePort: 39200 if you want to access elasticsearch from any node_ip 464 | - Check data in elasticsearch 465 | kubectl exec ubuntu -- curl es:9200/_search?q=* 466 | curl node_ip:39200/_search?q=* <-- if type: NodePort set in es.yaml 467 | 468 | ### No log coming in kibana: 469 | - check that there are file in node: ssh -i ~/.ssh/id_rsa_foobar core@185.19.29.212 ls /var/log/containers 470 | 471 | Can't connect to k8s-dashboard addon: 472 | - Carefull, addon are in kube-system namespace! 473 | If stuck use type: NodePort and 474 | - Find the node public port: kubectl describe service kubernetes-dashboard --namespace=kube-system 475 | - Access it from nodes : http://185.19.30.220:31224/ 476 | 477 | ### DNS resolution not working? Svc kube-dns.kube-system should take care of the resolution 478 | 479 | kubectl exec ubuntu -- nslookup google.com 480 | kubectl exec ubuntu -- nslookup kubernetes 481 | kubectl exec ubuntu -- nslookup kubernetes.default 482 | kubectl exec ubuntu -- nslookup kubernetes-dashboard.kube-system 483 | 484 | ### Pod can't get created? See more logs: 485 | 486 | kubectl describe po/elastcisearch 487 | kubectl logs -f elasticsearch-ret5zg 488 | 489 | ### Prometheus can't scrape node_exporter 490 | Possibly firewall issues! 491 | You need to open firewall internal rules between all nodes port 9100 (endpoint) and 10255 (node) 492 | 493 | ### Check influxdb 494 | 495 | kubectl exec ubuntu --namespace=monitoring2 -- curl -sl -I influxdb:8086/ping 496 | 497 | ### Traefik pod cant get created: port 443 already used 498 | 499 | Not very sure yet what the issue on lb_node, but try to start traefik on another minion: 500 | 501 | kubectl label node 185.19.29.192 role=minion <-- give another node the minion role 502 | nano traefik/traefik-deployment.yaml <-- and edit section spec > nodeSelector: > role: minion 503 | 504 | Then delete and recreate traefik, should be all good. 505 | 506 | ### Check traefik protected access 507 | 508 | apt install httpie 509 | http --verify=no --auth test:test https://kibana.satoshi.tech -v 510 | 511 | # 8. Annexes 512 | 513 | ### Shell Alias for K8s 514 | ``` 515 | alias k='kubectl' 516 | alias kk='kubectl get all' 517 | alias wk='watch kubectl get all' 518 | alias ka='kubectl get all --all-namespaces' 519 | alias kc='kubectl create -f' 520 | alias kdel='kubectl delete -f' 521 | alias kcdel='kubectl delete configmap' 522 | alias kd='kubectl describe' 523 | alias kl='kubectl logs' 524 | 525 | ``` 526 | 527 | ### Need another slave node? 528 | Edit ansible-cloudstack/k8s.yml and run again the deploy 529 | 530 | ### Want to start from scrash? 531 | 532 | Delete the corresponding namespace, all related containers/services will be destroyed. 533 | 534 | kubectl delete namespace monitoring 535 | kubectl delete namespace logging 536 | 537 | # 9. Future work 538 | 539 | - Use different firewalls security group: k8s, k8s-dmz, k8s-master, to be ready for production 540 | - Use persistent data for Elasticsearch and prometheus 541 | - Fix prometheus k8s_pod scraping both port 80 and 9102... 542 | -------------------------------------------------------------------------------- /ansible/LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "{}" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright {yyyy} {name of copyright owner} 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. 202 | -------------------------------------------------------------------------------- /ansible/README.md: -------------------------------------------------------------------------------- 1 | Ansible playbook to install a development Kubernetes (k8s) cluster 2 | ================================================================== 3 | 4 | Basic recipes using the ansible cloudstack module to create ssh keys, sec group etc and deploy [Kubernetes](http://kubernetes.io) on [CoreOS](http://coreos.com). 5 | This setup is to be used for development purposes only, as there are no HA features in place. 6 | 7 | Prerequisites 8 | ------------- 9 | 10 | You will need Ansible >= 2.0, sshpubkeys and [cs](https://github.com/exoscale/cs) :) 11 | 12 | $ sudo apt-get install -y python-pip 13 | $ sudo pip install ansible 14 | $ sudo pip install cs 15 | $ sudo pip install sshpubkeys 16 | 17 | Setup cloudstack 18 | ---------------- 19 | 20 | Create a `~/.cloudstack.ini` file with your creds and cloudstack endpoint, for example: 21 | 22 | [cloudstack] 23 | endpoint = https://api.exoscale.ch/compute 24 | key = 25 | secret = 26 | method = post 27 | 28 | We need to use the http POST method to pass the userdata to the coreOS instances. 29 | 30 | Create a Kubernetes cluster 31 | --------------------------- 32 | 33 | $ ansible-playbook k8s.yml 34 | 35 | Some variables can be edited in the `k8s.yml` file. 36 | This will start a Kubernetes master node and a number of compute nodes. 37 | 38 | Check the tasks and templates in `roles/k8s`. 39 | 40 | Test your cluster 41 | ----------------- 42 | 43 | You will need `kubectl` CLI installed, the playbook will configure, set and use a new context automatically. 44 | 45 | Then run 46 | 47 | $ kubectl get nodes 48 | 49 | 50 | 51 | 52 | -------------------------------------------------------------------------------- /ansible/ansible.cfg: -------------------------------------------------------------------------------- 1 | [defaults] 2 | hostfile = ./inventory 3 | -------------------------------------------------------------------------------- /ansible/inventory: -------------------------------------------------------------------------------- 1 | [default] 2 | # localhost ansible_python_interpreter=/opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin/python 3 | 4 | [nodes] 5 | 185.19.30.189 6 | 185.19.31.60 7 | 185.19.31.5 8 | 9 | [nodes:vars] 10 | ansible_ssh_user=core 11 | ansible_ssh_private_key_file=~/.ssh/id_rsa_foobar 12 | ansible_python_interpreter="PATH=/home/core/bin:$PATH python" 13 | -------------------------------------------------------------------------------- /ansible/k8s-remove.yml: -------------------------------------------------------------------------------- 1 | --- 2 | - hosts: localhost 3 | connection: local 4 | vars: 5 | ssh_key: k8s 6 | k8s_num_nodes: 2 7 | k8s_security_group_name: k8s 8 | k8s_node_prefix: k8s 9 | roles: 10 | - k8s-remove 11 | -------------------------------------------------------------------------------- /ansible/k8s.retry: -------------------------------------------------------------------------------- 1 | localhost 2 | -------------------------------------------------------------------------------- /ansible/k8s.yml: -------------------------------------------------------------------------------- 1 | --- 2 | - hosts: localhost 3 | connection: local 4 | vars: 5 | ssh_key: foobar 6 | k8s_version: 1.4.6 7 | k8s_num_nodes: 2 8 | k8s_security_group_name: k8s 9 | k8s_node_prefix: k8s 10 | k8s_username: foobar 11 | k8s_password: FdKPSuwQ 12 | # This template name is specific to http://exoscale.ch, replace it 13 | k8s_template: Linux CoreOS stable 1122 64-bit 50G Disk (2016-09-06-dcb493) 14 | k8s_instance_type: Medium 15 | 16 | k8s_binaries: https://storage.googleapis.com/kubernetes-release/release/v{{ k8s_version }}/bin 17 | 18 | roles: 19 | - common 20 | - k8s 21 | -------------------------------------------------------------------------------- /ansible/roles/common/tasks/create_sshkey.yml: -------------------------------------------------------------------------------- 1 | - name: Create SSH Key 2 | local_action: 3 | module: cs_sshkeypair 4 | name: "{{ ssh_key }}" 5 | register: key 6 | tags: sshkey 7 | 8 | - debug: msg='private key is {{ key.private_key }}' 9 | when: key.changed 10 | tags: sshkey 11 | 12 | - local_action: copy content="{{ key.private_key }}" dest="~/.ssh/id_rsa_{{ ssh_key }}" 13 | when: key.changed 14 | tags: sshkey 15 | 16 | - file: path="~/.ssh/id_rsa_{{ ssh_key }}" mode=0600 17 | when: key.changed 18 | tags: sshkey 19 | -------------------------------------------------------------------------------- /ansible/roles/common/tasks/main.yml: -------------------------------------------------------------------------------- 1 | --- 2 | - include: create_sshkey.yml 3 | -------------------------------------------------------------------------------- /ansible/roles/k8s-remove/tasks/delete_context.yml: -------------------------------------------------------------------------------- 1 | # Create k8s context 2 | 3 | - name: Unset cluster 4 | command: kubectl config unset clusters.exo 5 | tags: context 6 | 7 | - name: Unset context 8 | command: kubectl config unset contexts.exo 9 | tags: context 10 | 11 | - name: Unset current context 12 | command: kubectl config unset current-context 13 | tags: context 14 | -------------------------------------------------------------------------------- /ansible/roles/k8s-remove/tasks/delete_inv.yml: -------------------------------------------------------------------------------- 1 | # Removes inventory file 2 | 3 | - name: Remove old inventory 4 | file: path=./inventory state=absent 5 | -------------------------------------------------------------------------------- /ansible/roles/k8s-remove/tasks/delete_secgroup.yml: -------------------------------------------------------------------------------- 1 | # Create k8s security group 2 | 3 | - name: Remove k8s Security Group 4 | local_action: 5 | module: cs_securitygroup 6 | name: "{{ k8s_security_group_name }}" 7 | state: absent 8 | -------------------------------------------------------------------------------- /ansible/roles/k8s-remove/tasks/delete_secgroup_rules.yml: -------------------------------------------------------------------------------- 1 | # Add Rule to etcd security group 2 | 3 | # - name: etcd access for k8s 4 | # local_action: 5 | # module: cs_securitygroup_rule 6 | # name: "{{ etcd_security_group_name }}" 7 | # start_port: 4001 8 | # end_port: 4001 9 | # user_security_group: "{{ k8s_security_group_name }}" 10 | 11 | # Rules for the k8s security group 12 | 13 | - name: Remove SSH for k8s 14 | local_action: 15 | module: cs_securitygroup_rule 16 | security_group: "{{ k8s_security_group_name }}" 17 | start_port: 22 18 | end_port: 22 19 | state: absent 20 | 21 | 22 | #etcd 23 | - name: Remove etcd 4001 24 | local_action: 25 | module: cs_securitygroup_rule 26 | security_group: "{{ k8s_security_group_name }}" 27 | start_port: 4001 28 | end_port: 4001 29 | user_security_group: "{{ k8s_security_group_name }}" 30 | state: absent 31 | 32 | - name: Remove etcd 7001 33 | local_action: 34 | module: cs_securitygroup_rule 35 | security_group: "{{ k8s_security_group_name }}" 36 | start_port: 7001 37 | end_port: 7001 38 | user_security_group: "{{ k8s_security_group_name }}" 39 | state: absent 40 | 41 | - name: Remove etcd 2379 42 | local_action: 43 | module: cs_securitygroup_rule 44 | security_group: "{{ k8s_security_group_name }}" 45 | start_port: 7001 46 | end_port: 7001 47 | user_security_group: "{{ k8s_security_group_name }}" 48 | state: absent 49 | 50 | - name: Remove etcd 2380 51 | local_action: 52 | module: cs_securitygroup_rule 53 | security_group: "{{ k8s_security_group_name }}" 54 | start_port: 2380 55 | end_port: 2380 56 | user_security_group: "{{ k8s_security_group_name }}" 57 | state: absent 58 | 59 | - name: Remove flannel UDP 8472 60 | local_action: 61 | module: cs_securitygroup_rule 62 | security_group: "{{ k8s_security_group_name }}" 63 | start_port: 8472 64 | end_port: 8472 65 | protocol: udp 66 | user_security_group: "{{ k8s_security_group_name }}" 67 | state: absent 68 | 69 | - name: Remove secure 443 70 | local_action: 71 | module: cs_securitygroup_rule 72 | security_group: "{{ k8s_security_group_name }}" 73 | start_port: 443 74 | end_port: 443 75 | protocol: tcp 76 | user_security_group: "{{ k8s_security_group_name }}" 77 | state: absent 78 | 79 | #k8s api 80 | - name: Remove k8s 8080 81 | local_action: 82 | module: cs_securitygroup_rule 83 | security_group: "{{ k8s_security_group_name }}" 84 | start_port: 8080 85 | end_port: 8080 86 | user_security_group: "{{ k8s_security_group_name }}" 87 | state: absent 88 | 89 | - name: Remove k8s public 8080 90 | local_action: 91 | module: cs_securitygroup_rule 92 | security_group: "{{ k8s_security_group_name }}" 93 | start_port: 8080 94 | end_port: 8080 95 | state: absent 96 | -------------------------------------------------------------------------------- /ansible/roles/k8s-remove/tasks/delete_vm.yml: -------------------------------------------------------------------------------- 1 | - name: Remove k8s head node 2 | local_action: 3 | module: cs_instance 4 | name: "{{ k8s_node_prefix }}-head" 5 | state: absent 6 | 7 | - name: Remove k8s nodes 8 | local_action: 9 | module: cs_instance 10 | name: "{{ k8s_node_prefix }}-node-{{ item }}" 11 | state: absent 12 | with_sequence: count={{ k8s_num_nodes }} 13 | -------------------------------------------------------------------------------- /ansible/roles/k8s-remove/tasks/main.yml: -------------------------------------------------------------------------------- 1 | --- 2 | - include: delete_vm.yml 3 | - include: delete_inv.yml 4 | - include: delete_secgroup_rules.yml 5 | - include: delete_secgroup.yml 6 | - include: delete_context.yml 7 | -------------------------------------------------------------------------------- /ansible/roles/k8s/tasks/create_context.yml: -------------------------------------------------------------------------------- 1 | # Create k8s context 2 | 3 | - name: Set context cluster 4 | command: kubectl config set-cluster exo --server=https://{{ k8s_master.default_ip }}:443 --insecure-skip-tls-verify=true 5 | tags: context 6 | 7 | - name: Set context user 8 | command: kubectl config set-credentials exo --username={{ k8s_username }} --password={{ k8s_password}} 9 | 10 | - name: Create context 11 | command: kubectl config set-context exo --cluster=exo --user=exo 12 | 13 | - name: Use context 14 | command: kubectl config use-context exo 15 | -------------------------------------------------------------------------------- /ansible/roles/k8s/tasks/create_inv.yml: -------------------------------------------------------------------------------- 1 | # Create inventory file 2 | 3 | - name: Create inventory file 4 | template: src=inventory.j2 dest=./inventory 5 | tags: inventory 6 | -------------------------------------------------------------------------------- /ansible/roles/k8s/tasks/create_secgroup.yml: -------------------------------------------------------------------------------- 1 | # Create k8s security group 2 | 3 | - name: Create k8s Security Group 4 | local_action: 5 | module: cs_securitygroup 6 | name: "{{ k8s_security_group_name }}" 7 | description: k8s 8 | -------------------------------------------------------------------------------- /ansible/roles/k8s/tasks/create_secgroup_rules.yml: -------------------------------------------------------------------------------- 1 | # Add Rule to etcd security group 2 | 3 | # - name: etcd access for k8s 4 | # local_action: 5 | # module: cs_securitygroup_rule 6 | # name: "{{ etcd_security_group_name }}" 7 | # start_port: 4001 8 | # end_port: 4001 9 | # user_security_group: "{{ k8s_security_group_name }}" 10 | 11 | # Rules for the k8s security group 12 | 13 | # internal 14 | - name: flannel etcd 2379 15 | local_action: 16 | module: cs_securitygroup_rule 17 | security_group: "{{ k8s_security_group_name }}" 18 | start_port: 2379 19 | end_port: 2379 20 | user_security_group: "{{ k8s_security_group_name }}" 21 | 22 | - name: flannel etcd 2380 23 | local_action: 24 | module: cs_securitygroup_rule 25 | security_group: "{{ k8s_security_group_name }}" 26 | start_port: 2380 27 | end_port: 2380 28 | user_security_group: "{{ k8s_security_group_name }}" 29 | 30 | - name: k8s 8080 31 | local_action: 32 | module: cs_securitygroup_rule 33 | security_group: "{{ k8s_security_group_name }}" 34 | start_port: 8080 35 | end_port: 8080 36 | user_security_group: "{{ k8s_security_group_name }}" 37 | 38 | - name: flannel UDP 8472 39 | local_action: 40 | module: cs_securitygroup_rule 41 | security_group: "{{ k8s_security_group_name }}" 42 | start_port: 8472 43 | end_port: 8472 44 | protocol: udp 45 | user_security_group: "{{ k8s_security_group_name }}" 46 | 47 | - name: kubelet logs and exec TCP 10250 48 | local_action: 49 | module: cs_securitygroup_rule 50 | security_group: "{{ k8s_security_group_name }}" 51 | start_port: 10250 52 | end_port: 10250 53 | protocol: tcp 54 | user_security_group: "{{ k8s_security_group_name }}" 55 | 56 | - name: prometheus scrape node exporter TCP 9100 57 | local_action: 58 | module: cs_securitygroup_rule 59 | security_group: "{{ k8s_security_group_name }}" 60 | start_port: 9100 61 | end_port: 9100 62 | protocol: tcp 63 | user_security_group: "{{ k8s_security_group_name }}" 64 | 65 | - name: prometheus scrape kubernetes-nodes TCP 10255 66 | local_action: 67 | module: cs_securitygroup_rule 68 | security_group: "{{ k8s_security_group_name }}" 69 | start_port: 10255 70 | end_port: 10255 71 | protocol: tcp 72 | user_security_group: "{{ k8s_security_group_name }}" 73 | 74 | # external access 75 | 76 | - name: SSH for k8s 77 | local_action: 78 | module: cs_securitygroup_rule 79 | security_group: "{{ k8s_security_group_name }}" 80 | start_port: 22 81 | end_port: 22 82 | 83 | - name: k8s public secure 84 | local_action: 85 | module: cs_securitygroup_rule 86 | security_group: "{{ k8s_security_group_name }}" 87 | start_port: 443 88 | end_port: 443 89 | 90 | - name: kibana 91 | local_action: 92 | module: cs_securitygroup_rule 93 | security_group: "{{ k8s_security_group_name }}" 94 | start_port: 5601 95 | end_port: 5601 96 | 97 | - name: prometheus 98 | local_action: 99 | module: cs_securitygroup_rule 100 | security_group: "{{ k8s_security_group_name }}" 101 | start_port: 9090 102 | end_port: 9090 103 | 104 | - name: grafana (prometheus) 105 | local_action: 106 | module: cs_securitygroup_rule 107 | security_group: "{{ k8s_security_group_name }}" 108 | start_port: 3000 109 | end_port: 3000 110 | 111 | - name: grafana2 (heapster) 112 | local_action: 113 | module: cs_securitygroup_rule 114 | security_group: "{{ k8s_security_group_name }}" 115 | start_port: 3002 116 | end_port: 3002 117 | 118 | - name: k8s dashboard addon 119 | local_action: 120 | module: cs_securitygroup_rule 121 | security_group: "{{ k8s_security_group_name }}" 122 | start_port: 9999 123 | end_port: 9999 124 | 125 | - name: nodePort random service range 126 | local_action: 127 | module: cs_securitygroup_rule 128 | security_group: "{{ k8s_security_group_name }}" 129 | start_port: 30000 130 | end_port: 33000 131 | 132 | - name: traefik gui 133 | local_action: 134 | module: cs_securitygroup_rule 135 | security_group: "{{ k8s_security_group_name }}" 136 | start_port: 8080 137 | end_port: 8080 138 | 139 | - name: traefik 140 | local_action: 141 | module: cs_securitygroup_rule 142 | security_group: "{{ k8s_security_group_name }}" 143 | start_port: 80 144 | end_port: 80 145 | -------------------------------------------------------------------------------- /ansible/roles/k8s/tasks/create_vm.yml: -------------------------------------------------------------------------------- 1 | - name: Start k8s head node 2 | local_action: 3 | module: cs_instance 4 | name: "{{ k8s_node_prefix }}-head" 5 | template: "{{ k8s_template }}" 6 | service_offering: "{{ k8s_instance_type }}" 7 | ssh_key: "{{ ssh_key }}" 8 | security_groups: [ '{{ k8s_security_group_name }}' ] 9 | user_data: "{{ lookup('template', '../templates/k8s-master.j2') }}" 10 | register: k8s_master 11 | 12 | - debug: msg='k8s master IP is {{ k8s_master.default_ip }}' 13 | 14 | - name: Start k8s nodes 15 | local_action: 16 | module: cs_instance 17 | name: "{{ k8s_node_prefix }}-node-{{ item }}" 18 | template: "{{ k8s_template }}" 19 | service_offering: "{{ k8s_instance_type }}" 20 | ssh_key: "{{ ssh_key }}" 21 | security_groups: [ '{{ k8s_security_group_name }}' ] 22 | user_data: "{{ lookup('template', '../templates/k8s-node.j2') }}" 23 | with_sequence: count={{ k8s_num_nodes }} 24 | register: k8s_nodes 25 | -------------------------------------------------------------------------------- /ansible/roles/k8s/tasks/main.yml: -------------------------------------------------------------------------------- 1 | --- 2 | - include: create_secgroup.yml 3 | - include: create_secgroup_rules.yml 4 | - include: create_vm.yml 5 | - include: create_inv.yml 6 | - include: create_context.yml 7 | -------------------------------------------------------------------------------- /ansible/roles/k8s/templates/inventory.j2: -------------------------------------------------------------------------------- 1 | [default] 2 | # localhost ansible_python_interpreter=/opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin/python 3 | 4 | [nodes] 5 | {{ k8s_master.default_ip }} 6 | {% for item in k8s_nodes.results %} 7 | {{ item.default_ip }} 8 | {% endfor %} 9 | 10 | [nodes:vars] 11 | ansible_ssh_user=core 12 | ansible_ssh_private_key_file=~/.ssh/id_rsa_{{ ssh_key }} 13 | ansible_python_interpreter="PATH=/home/core/bin:$PATH python" 14 | -------------------------------------------------------------------------------- /ansible/roles/k8s/templates/k8s-master-rbac.j2: -------------------------------------------------------------------------------- 1 | #cloud-config 2 | 3 | --- 4 | write-files: 5 | - path: /etc/conf.d/nfs 6 | permissions: '0644' 7 | content: | 8 | OPTS_RPC_MOUNTD="" 9 | 10 | - path: /srv/kubernetes/basicauth.csv 11 | permissions: '0644' 12 | content: | 13 | {{ k8s_password }},{{ k8s_username }},1 14 | password,randomuser,2 15 | 16 | - path: /opt/bin/wupiao 17 | permissions: '0755' 18 | content: | 19 | #!/bin/bash 20 | # [w]ait [u]ntil [p]ort [i]s [a]ctually [o]pen 21 | [ -n "$1" ] && \ 22 | until curl -o /dev/null -sIf http://${1}; do \ 23 | sleep 1 && echo .; 24 | done; 25 | exit $? 26 | 27 | - path: /srv/kubernetes/manifests/kube-system.json 28 | content: | 29 | { 30 | "apiVersion": "v1", 31 | "kind": "Namespace", 32 | "metadata": { 33 | "name": "kube-system" 34 | } 35 | } 36 | 37 | - path: /opt/bin/install-kube-system 38 | permissions: 0700 39 | owner: root:root 40 | content: | 41 | #!/bin/bash -e 42 | /usr/bin/curl -L -o /srv/kubernetes/manifests/dashboard-controller.yaml https://raw.githubusercontent.com/kubernetes/dashboard/v1.5.0/src/deploy/kubernetes-dashboard.yaml 43 | 44 | /usr/bin/curl -L -o /srv/kubernetes/manifests/kube-dns-svc.json https://raw.githubusercontent.com/skippbox/configk8s/master/json/kube-dns-svc.json 45 | 46 | /usr/bin/curl -L -o /srv/kubernetes/manifests/kube-dns-rc.json https://raw.githubusercontent.com/skippbox/configk8s/master/json/kube-dns-rc.json 47 | 48 | /usr/bin/curl -H "Content-Type: application/json" -XPOST -d @"/srv/kubernetes/manifests/kube-system.json" "http://127.0.0.1:8080/api/v1/namespaces" 49 | 50 | /usr/bin/curl -H "Content-Type: application/json" -XPOST -d @"/srv/kubernetes/manifests/dashboard-controller.yaml" "http://127.0.0.1:8080/api/v1/namespaces/kube-system/replicationcontrollers" 51 | 52 | /usr/bin/curl -H "Content-Type: application/json" -XPOST -d @"/srv/kubernetes/manifests/kube-dns-svc.json" "http://127.0.0.1:8080/api/v1/namespaces/kube-system/services" 53 | 54 | /usr/bin/curl -H "Content-Type: application/json" -XPOST -d @"/srv/kubernetes/manifests/kube-dns-rc.json" "http://127.0.0.1:8080/api/v1/namespaces/kube-system/replicationcontrollers" 55 | 56 | hostname: master 57 | 58 | coreos: 59 | 60 | etcd2: 61 | name: master 62 | listen-client-urls: http://0.0.0.0:2379 63 | advertise-client-urls: http://$private_ipv4:2379 64 | initial-cluster-token: k8s_etcd 65 | listen-peer-urls: http://$private_ipv4:2380 66 | initial-advertise-peer-urls: http://$private_ipv4:2380 67 | initial-cluster: master=http://$private_ipv4:2380 68 | initial-cluster-state: new 69 | 70 | fleet: 71 | metadata: "role=master" 72 | 73 | units: 74 | - name: fleet.service 75 | command: start 76 | 77 | - name: flanneld.service 78 | command: start 79 | drop-ins: 80 | - name: 50-network-config.conf 81 | content: | 82 | [Unit] 83 | Requires=etcd2.service 84 | [Service] 85 | ExecStartPre=/usr/bin/etcdctl set /coreos.com/network/config '{"Network":"10.244.0.0/16", "Backend": {"Type": "vxlan"}}' 86 | 87 | - name: docker.service 88 | command: start 89 | drop-ins: 90 | - name: 40-flannel.conf 91 | content: | 92 | [Unit] 93 | Requires=flanneld.service 94 | After=flanneld.service 95 | 96 | - name: generate-serviceaccount-key.service 97 | command: start 98 | content: | 99 | [Unit] 100 | Description=Generate service-account key file 101 | [Service] 102 | ExecStartPre=-/usr/bin/mkdir -p /opt/bin 103 | ExecStart=/bin/openssl genrsa -out /opt/bin/kube-serviceaccount.key 2048 2>/dev/null 104 | RemainAfterExit=yes 105 | Type=oneshot 106 | 107 | - name: setup-network-environment.service 108 | command: start 109 | content: | 110 | [Unit] 111 | Description=Setup Network Environment 112 | Documentation=https://github.com/kelseyhightower/setup-network-environment 113 | Requires=network-online.target 114 | After=network-online.target 115 | [Service] 116 | ExecStartPre=-/usr/bin/mkdir -p /opt/bin 117 | ExecStartPre=/usr/bin/curl -L -o /opt/bin/setup-network-environment -z /opt/bin/setup-network-environment https://github.com/kelseyhightower/setup-network-environment/releases/download/v1.0.0/setup-network-environment 118 | ExecStartPre=/usr/bin/chmod +x /opt/bin/setup-network-environment 119 | ExecStart=/opt/bin/setup-network-environment 120 | RemainAfterExit=yes 121 | Type=oneshot 122 | 123 | - name: install-kube-system.service 124 | command: start 125 | content: | 126 | [Unit] 127 | Requires=kube-apiserver.service 128 | After=kube-apiserver.service 129 | 130 | [Service] 131 | Type=simple 132 | StartLimitInterval=0 133 | Restart=on-failure 134 | ExecStartPre=/usr/bin/curl http://127.0.0.1:8080/version 135 | ExecStart=/opt/bin/install-kube-system 136 | 137 | - name: kube-apiserver.service 138 | command: start 139 | content: | 140 | [Unit] 141 | Description=Kubernetes API Server 142 | Documentation=https://github.com/GoogleCloudPlatform/kubernetes 143 | Requires=setup-network-environment.service etcd2.service generate-serviceaccount-key.service 144 | After=setup-network-environment.service etcd2.service generate-serviceaccount-key.service 145 | [Service] 146 | EnvironmentFile=/etc/network-environment 147 | ExecStartPre=-/usr/bin/mkdir -p /opt/bin 148 | ExecStartPre=/usr/bin/curl -L -o /opt/bin/kube-apiserver -z /opt/bin/kube-apiserver {{ k8s_binaries }}/linux/amd64/kube-apiserver 149 | ExecStartPre=/usr/bin/chmod +x /opt/bin/kube-apiserver 150 | ExecStartPre=/opt/bin/wupiao 127.0.0.1:2379/v2/machines 151 | ExecStart=/opt/bin/kube-apiserver \ 152 | --service-account-key-file=/opt/bin/kube-serviceaccount.key \ 153 | --service-account-lookup=false \ 154 | --basic-auth-file=/srv/kubernetes/basicauth.csv \ 155 | --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \ 156 | --authorization-mode=RBAC \ 157 | --authorization-rbac-super-user={{ k8s_username }} \ 158 | --runtime-config=api/v1,rbac.authorization.k8s.io/v1alpha1 \ 159 | --allow-privileged=true \ 160 | --insecure-port=8080 \ 161 | --secure-port=443 \ 162 | --service-cluster-ip-range=10.0.0.0/16 \ 163 | --etcd-servers=http://127.0.0.1:2379 \ 164 | --bind-address=0.0.0.0 \ 165 | --insecure-bind-address=0.0.0.0 \ 166 | --logtostderr=true 167 | Restart=always 168 | RestartSec=10 169 | 170 | - name: kube-controller-manager.service 171 | command: start 172 | content: | 173 | [Unit] 174 | Description=Kubernetes Controller Manager 175 | Documentation=https://github.com/GoogleCloudPlatform/kubernetes 176 | Requires=kube-apiserver.service 177 | After=kube-apiserver.service 178 | [Service] 179 | ExecStartPre=/usr/bin/curl -L -o /opt/bin/kube-controller-manager -z /opt/bin/kube-controller-manager {{ k8s_binaries }}/linux/amd64/kube-controller-manager 180 | ExecStartPre=/usr/bin/chmod +x /opt/bin/kube-controller-manager 181 | ExecStart=/opt/bin/kube-controller-manager \ 182 | --service-account-private-key-file=/opt/bin/kube-serviceaccount.key \ 183 | --root-ca-file=/var/run/kubernetes/apiserver.crt \ 184 | --master=127.0.0.1:8080 \ 185 | --logtostderr=true 186 | Restart=always 187 | RestartSec=10 188 | 189 | - name: kube-scheduler.service 190 | command: start 191 | content: | 192 | [Unit] 193 | Description=Kubernetes Scheduler 194 | Documentation=https://github.com/GoogleCloudPlatform/kubernetes 195 | Requires=kube-apiserver.service 196 | After=kube-apiserver.service 197 | [Service] 198 | ExecStartPre=/usr/bin/curl -L -o /opt/bin/kube-scheduler -z /opt/bin/kube-scheduler {{ k8s_binaries }}/linux/amd64/kube-scheduler 199 | ExecStartPre=/usr/bin/chmod +x /opt/bin/kube-scheduler 200 | ExecStart=/opt/bin/kube-scheduler --master=127.0.0.1:8080 201 | Restart=always 202 | RestartSec=10 203 | 204 | update: 205 | group: stable 206 | reboot-strategy: off 207 | -------------------------------------------------------------------------------- /ansible/roles/k8s/templates/k8s-master.j2: -------------------------------------------------------------------------------- 1 | #cloud-config 2 | 3 | --- 4 | write-files: 5 | - path: /etc/conf.d/nfs 6 | permissions: '0644' 7 | content: | 8 | OPTS_RPC_MOUNTD="" 9 | 10 | - path: /srv/kubernetes/basicauth.csv 11 | permissions: '0644' 12 | content: | 13 | {{ k8s_password }},{{ k8s_username }},1 14 | password,randomuser,2 15 | 16 | - path: /opt/bin/wupiao 17 | permissions: '0755' 18 | content: | 19 | #!/bin/bash 20 | # [w]ait [u]ntil [p]ort [i]s [a]ctually [o]pen 21 | [ -n "$1" ] && \ 22 | until curl -o /dev/null -sIf http://${1}; do \ 23 | sleep 1 && echo .; 24 | done; 25 | exit $? 26 | 27 | - path: /srv/kubernetes/manifests/kube-system.json 28 | content: | 29 | { 30 | "apiVersion": "v1", 31 | "kind": "Namespace", 32 | "metadata": { 33 | "name": "kube-system" 34 | } 35 | } 36 | 37 | - path: /opt/bin/install-kube-system 38 | permissions: 0700 39 | owner: root:root 40 | content: | 41 | #!/bin/bash -e 42 | /usr/bin/curl -L -o /srv/kubernetes/manifests/dashboard-controller.json https://raw.githubusercontent.com/skippbox/configk8s/master/json/dashboard-controller.json 43 | 44 | /usr/bin/curl -L -o /srv/kubernetes/manifests/dashboard-service.json https://raw.githubusercontent.com/skippbox/configk8s/master/json/dashboard-service.json 45 | 46 | /usr/bin/curl -L -o /srv/kubernetes/manifests/kube-dns-svc.json https://raw.githubusercontent.com/skippbox/configk8s/master/json/kube-dns-svc.json 47 | 48 | /usr/bin/curl -L -o /srv/kubernetes/manifests/kube-dns-rc.json https://raw.githubusercontent.com/skippbox/configk8s/master/json/kube-dns-rc.json 49 | 50 | /usr/bin/curl -H "Content-Type: application/json" -XPOST -d @"/srv/kubernetes/manifests/kube-system.json" "http://127.0.0.1:8080/api/v1/namespaces" 51 | 52 | /usr/bin/curl -H "Content-Type: application/json" -XPOST -d @"/srv/kubernetes/manifests/dashboard-controller.yaml" "http://127.0.0.1:8080/api/v1/namespaces/kube-system/replicationcontrollers" 53 | 54 | /usr/bin/curl -H "Content-Type: application/json" -XPOST -d @"/srv/kubernetes/manifests/kube-dns-svc.json" "http://127.0.0.1:8080/api/v1/namespaces/kube-system/services" 55 | 56 | /usr/bin/curl -H "Content-Type: application/json" -XPOST -d @"/srv/kubernetes/manifests/kube-dns-rc.json" "http://127.0.0.1:8080/api/v1/namespaces/kube-system/replicationcontrollers" 57 | 58 | hostname: master 59 | 60 | coreos: 61 | 62 | etcd2: 63 | name: master 64 | listen-client-urls: http://0.0.0.0:2379 65 | advertise-client-urls: http://$private_ipv4:2379 66 | initial-cluster-token: k8s_etcd 67 | listen-peer-urls: http://$private_ipv4:2380 68 | initial-advertise-peer-urls: http://$private_ipv4:2380 69 | initial-cluster: master=http://$private_ipv4:2380 70 | initial-cluster-state: new 71 | 72 | fleet: 73 | metadata: "role=master" 74 | 75 | units: 76 | - name: fleet.service 77 | command: start 78 | 79 | - name: flanneld.service 80 | command: start 81 | drop-ins: 82 | - name: 50-network-config.conf 83 | content: | 84 | [Unit] 85 | Requires=etcd2.service 86 | [Service] 87 | ExecStartPre=/usr/bin/etcdctl set /coreos.com/network/config '{"Network":"10.244.0.0/16", "Backend": {"Type": "vxlan"}}' 88 | 89 | - name: docker.service 90 | command: start 91 | drop-ins: 92 | - name: 40-flannel.conf 93 | content: | 94 | [Unit] 95 | Requires=flanneld.service 96 | After=flanneld.service 97 | 98 | - name: generate-serviceaccount-key.service 99 | command: start 100 | content: | 101 | [Unit] 102 | Description=Generate service-account key file 103 | [Service] 104 | ExecStartPre=-/usr/bin/mkdir -p /opt/bin 105 | ExecStart=/bin/openssl genrsa -out /opt/bin/kube-serviceaccount.key 2048 2>/dev/null 106 | RemainAfterExit=yes 107 | Type=oneshot 108 | 109 | - name: setup-network-environment.service 110 | command: start 111 | content: | 112 | [Unit] 113 | Description=Setup Network Environment 114 | Documentation=https://github.com/kelseyhightower/setup-network-environment 115 | Requires=network-online.target 116 | After=network-online.target 117 | [Service] 118 | ExecStartPre=-/usr/bin/mkdir -p /opt/bin 119 | ExecStartPre=/usr/bin/curl -L -o /opt/bin/setup-network-environment -z /opt/bin/setup-network-environment https://github.com/kelseyhightower/setup-network-environment/releases/download/v1.0.0/setup-network-environment 120 | ExecStartPre=/usr/bin/chmod +x /opt/bin/setup-network-environment 121 | ExecStart=/opt/bin/setup-network-environment 122 | RemainAfterExit=yes 123 | Type=oneshot 124 | 125 | - name: install-kube-system.service 126 | command: start 127 | content: | 128 | [Unit] 129 | Requires=kube-apiserver.service 130 | After=kube-apiserver.service 131 | 132 | [Service] 133 | Type=simple 134 | StartLimitInterval=0 135 | Restart=on-failure 136 | ExecStartPre=/usr/bin/curl http://127.0.0.1:8080/version 137 | ExecStart=/opt/bin/install-kube-system 138 | 139 | - name: kube-apiserver.service 140 | command: start 141 | content: | 142 | [Unit] 143 | Description=Kubernetes API Server 144 | Documentation=https://github.com/GoogleCloudPlatform/kubernetes 145 | Requires=setup-network-environment.service etcd2.service generate-serviceaccount-key.service 146 | After=setup-network-environment.service etcd2.service generate-serviceaccount-key.service 147 | [Service] 148 | EnvironmentFile=/etc/network-environment 149 | ExecStartPre=-/usr/bin/mkdir -p /opt/bin 150 | ExecStartPre=/usr/bin/curl -L -o /opt/bin/kube-apiserver -z /opt/bin/kube-apiserver {{ k8s_binaries }}/linux/amd64/kube-apiserver 151 | ExecStartPre=/usr/bin/chmod +x /opt/bin/kube-apiserver 152 | ExecStartPre=/opt/bin/wupiao 127.0.0.1:2379/v2/machines 153 | ExecStart=/opt/bin/kube-apiserver \ 154 | --service-account-key-file=/opt/bin/kube-serviceaccount.key \ 155 | --service-account-lookup=false \ 156 | --basic-auth-file=/srv/kubernetes/basicauth.csv \ 157 | --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \ 158 | --allow-privileged=true \ 159 | --insecure-port=8080 \ 160 | --secure-port=443 \ 161 | --service-cluster-ip-range=10.0.0.0/16 \ 162 | --etcd-servers=http://127.0.0.1:2379 \ 163 | --bind-address=0.0.0.0 \ 164 | --insecure-bind-address=0.0.0.0 \ 165 | --logtostderr=true 166 | Restart=always 167 | RestartSec=10 168 | 169 | - name: kube-controller-manager.service 170 | command: start 171 | content: | 172 | [Unit] 173 | Description=Kubernetes Controller Manager 174 | Documentation=https://github.com/GoogleCloudPlatform/kubernetes 175 | Requires=kube-apiserver.service 176 | After=kube-apiserver.service 177 | [Service] 178 | ExecStartPre=/usr/bin/curl -L -o /opt/bin/kube-controller-manager -z /opt/bin/kube-controller-manager {{ k8s_binaries }}/linux/amd64/kube-controller-manager 179 | ExecStartPre=/usr/bin/chmod +x /opt/bin/kube-controller-manager 180 | ExecStart=/opt/bin/kube-controller-manager \ 181 | --service-account-private-key-file=/opt/bin/kube-serviceaccount.key \ 182 | --root-ca-file=/var/run/kubernetes/apiserver.crt \ 183 | --master=127.0.0.1:8080 \ 184 | --logtostderr=true 185 | Restart=always 186 | RestartSec=10 187 | 188 | - name: kube-scheduler.service 189 | command: start 190 | content: | 191 | [Unit] 192 | Description=Kubernetes Scheduler 193 | Documentation=https://github.com/GoogleCloudPlatform/kubernetes 194 | Requires=kube-apiserver.service 195 | After=kube-apiserver.service 196 | [Service] 197 | ExecStartPre=/usr/bin/curl -L -o /opt/bin/kube-scheduler -z /opt/bin/kube-scheduler {{ k8s_binaries }}/linux/amd64/kube-scheduler 198 | ExecStartPre=/usr/bin/chmod +x /opt/bin/kube-scheduler 199 | ExecStart=/opt/bin/kube-scheduler --master=127.0.0.1:8080 200 | Restart=always 201 | RestartSec=10 202 | 203 | update: 204 | group: stable 205 | reboot-strategy: off 206 | -------------------------------------------------------------------------------- /ansible/roles/k8s/templates/k8s-node.j2: -------------------------------------------------------------------------------- 1 | #cloud-config 2 | write-files: 3 | - path: /opt/bin/wupiao 4 | permissions: '0755' 5 | content: | 6 | #!/bin/bash 7 | # [w]ait [u]ntil [p]ort [i]s [a]ctually [o]pen 8 | [ -n "$1" ] && [ -n "$2" ] && while ! curl --output /dev/null \ 9 | --silent --head --fail \ 10 | http://${1}:${2}; do sleep 1 && echo -n .; done; 11 | exit $? 12 | - path: /etc/sysctl.d/elasticsearch.conf 13 | permissions: '0755' 14 | content: | 15 | vm.max_map_count=262144 16 | coreos: 17 | etcd2: 18 | listen-client-urls: http://0.0.0.0:2379 19 | advertise-client-urls: http://0.0.0.0:2379 20 | initial-cluster: master=http://{{ k8s_master.default_ip }}:2380 21 | proxy: on 22 | fleet: 23 | metadata: "role=node" 24 | units: 25 | - name: rpcbind.service 26 | command: start 27 | - name: etcd2.service 28 | command: start 29 | - name: fleet.service 30 | command: start 31 | - name: flanneld.service 32 | command: start 33 | - name: docker.service 34 | drop-ins: 35 | - name: 40-flannel.conf 36 | content: | 37 | [Unit] 38 | Requires=flanneld.service 39 | After=flanneld.service 40 | command: start 41 | - name: setup-network-environment.service 42 | command: start 43 | content: | 44 | [Unit] 45 | Description=Setup Network Environment 46 | Documentation=https://github.com/kelseyhightower/setup-network-environment 47 | Requires=network-online.target 48 | After=network-online.target 49 | [Service] 50 | ExecStartPre=-/usr/bin/mkdir -p /opt/bin 51 | ExecStartPre=/usr/bin/curl -L -o /opt/bin/setup-network-environment -z /opt/bin/setup-network-environment https://github.com/kelseyhightower/setup-network-environment/releases/download/v1.0.0/setup-network-environment 52 | ExecStartPre=/usr/bin/chmod +x /opt/bin/setup-network-environment 53 | ExecStart=/opt/bin/setup-network-environment 54 | RemainAfterExit=yes 55 | Type=oneshot 56 | - name: kube-proxy.service 57 | command: start 58 | content: | 59 | [Unit] 60 | Description=Kubernetes Proxy 61 | Documentation=https://github.com/GoogleCloudPlatform/kubernetes 62 | Requires=setup-network-environment.service 63 | After=setup-network-environment.service 64 | [Service] 65 | ExecStartPre=/usr/bin/curl -L -o /opt/bin/kube-proxy -z /opt/bin/kube-proxy {{ k8s_binaries }}/linux/amd64/kube-proxy 66 | ExecStartPre=/usr/bin/chmod +x /opt/bin/kube-proxy 67 | # wait for kubernetes master to be up and ready 68 | ExecStartPre=/opt/bin/wupiao {{ k8s_master.default_ip }} 8080 69 | ExecStart=/opt/bin/kube-proxy \ 70 | --master={{ k8s_master.default_ip }}:8080 \ 71 | --logtostderr=true 72 | Restart=always 73 | RestartSec=10 74 | - name: kube-kubelet.service 75 | command: start 76 | content: | 77 | [Unit] 78 | Description=Kubernetes Kubelet 79 | Documentation=https://github.com/GoogleCloudPlatform/kubernetes 80 | Requires=setup-network-environment.service 81 | After=setup-network-environment.service 82 | [Service] 83 | EnvironmentFile=/etc/network-environment 84 | Environment=KUBELET_VERSION=v{{ k8s_version }}_coreos.0 85 | # In order to have logs in /var/log/containers to be pickup by fluentd 86 | Environment="RKT_OPTS=--volume dns,kind=host,source=/etc/resolv.conf --mount volume=dns,target=/etc/resolv.conf --volume var-log,kind=host,source=/var/log --mount volume=var-log,target=/var/log" 87 | ExecStartPre=/usr/bin/mkdir -p /var/log/containers 88 | # wait for kubernetes master to be up and ready 89 | ExecStartPre=/opt/bin/wupiao {{ k8s_master.default_ip }} 8080 90 | ExecStart=/usr/lib/coreos/kubelet-wrapper \ 91 | --address=0.0.0.0 \ 92 | --port=10250 \ 93 | --cluster-dns=10.0.0.10 \ 94 | --cluster-domain=cluster.local \ 95 | --hostname-override=${DEFAULT_IPV4} \ 96 | --api-servers={{ k8s_master.default_ip }}:8080 \ 97 | --allow-privileged=true \ 98 | --logtostderr=true \ 99 | --cadvisor-port=4194 \ 100 | --healthz-bind-address=0.0.0.0 \ 101 | --healthz-port=10248 102 | Restart=always 103 | RestartSec=10 104 | update: 105 | group: stable 106 | reboot-strategy: off 107 | -------------------------------------------------------------------------------- /kubernetes-dashboard/kubernetes-dashboard-deployment.yaml: -------------------------------------------------------------------------------- 1 | kind: Deployment 2 | apiVersion: extensions/v1beta1 3 | metadata: 4 | labels: 5 | app: kubernetes-dashboard 6 | name: kubernetes-dashboard 7 | namespace: kube-system 8 | spec: 9 | replicas: 1 10 | selector: 11 | matchLabels: 12 | app: kubernetes-dashboard 13 | template: 14 | metadata: 15 | labels: 16 | app: kubernetes-dashboard 17 | # Comment the following annotation if Dashboard must not be deployed on master 18 | annotations: 19 | scheduler.alpha.kubernetes.io/tolerations: | 20 | [ 21 | { 22 | "key": "dedicated", 23 | "operator": "Equal", 24 | "value": "master", 25 | "effect": "NoSchedule" 26 | } 27 | ] 28 | spec: 29 | containers: 30 | - name: kubernetes-dashboard 31 | image: gcr.io/google_containers/kubernetes-dashboard-amd64:v1.5.0 32 | imagePullPolicy: Always 33 | ports: 34 | - containerPort: 9090 35 | protocol: TCP 36 | args: 37 | # Uncomment the following line to manually specify Kubernetes API server Host 38 | # If not specified, Dashboard will attempt to auto discover the API server and connect 39 | # to it. Uncomment only if the default does not work. 40 | # - --apiserver-host=http://my-address:port 41 | livenessProbe: 42 | httpGet: 43 | path: / 44 | port: 9090 45 | initialDelaySeconds: 30 46 | timeoutSeconds: 30 47 | 48 | -------------------------------------------------------------------------------- /kubernetes-dashboard/kubernetes-dashboard-service.yaml: -------------------------------------------------------------------------------- 1 | kind: Service 2 | apiVersion: v1 3 | metadata: 4 | labels: 5 | app: kubernetes-dashboard 6 | name: kubernetes-dashboard 7 | namespace: kube-system 8 | spec: 9 | type: NodePort 10 | ports: 11 | - port: 9999 12 | nodePort: 30999 13 | targetPort: 9090 14 | selector: 15 | app: kubernetes-dashboard 16 | -------------------------------------------------------------------------------- /logging/_logging-namespace.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: v1 3 | kind: Namespace 4 | metadata: 5 | name: logging 6 | 7 | -------------------------------------------------------------------------------- /logging/dashboards/elk-v1.json: -------------------------------------------------------------------------------- 1 | [ 2 | { 3 | "_id": "ELK", 4 | "_type": "dashboard", 5 | "_source": { 6 | "title": "ELK", 7 | "hits": 0, 8 | "description": "", 9 | "panelsJSON": "[{\"col\":1,\"id\":\"ERRORS\",\"panelIndex\":1,\"row\":4,\"size_x\":5,\"size_y\":3,\"type\":\"visualization\"},{\"col\":6,\"id\":\"ERROR-LOGS\",\"panelIndex\":2,\"row\":4,\"size_x\":6,\"size_y\":3,\"type\":\"visualization\"},{\"col\":6,\"id\":\"CONTAINERS\",\"panelIndex\":3,\"row\":1,\"size_x\":3,\"size_y\":3,\"type\":\"visualization\"},{\"col\":1,\"id\":\"LOGS\",\"panelIndex\":4,\"row\":1,\"size_x\":5,\"size_y\":3,\"type\":\"visualization\"},{\"col\":1,\"columns\":[\"kubernetes.container_name\",\"kubernetes.pod_name\",\"kubernetes.host\",\"log\"],\"id\":\"SEARCH\",\"panelIndex\":5,\"row\":7,\"size_x\":11,\"size_y\":7,\"sort\":[\"@timestamp\",\"desc\"],\"type\":\"search\"},{\"col\":9,\"columns\":[\"kubernetes.container_name\",\"kubernetes.pod_name\",\"kubernetes.host\",\"log\"],\"id\":\"PARSING-FAILED\",\"panelIndex\":6,\"row\":1,\"size_x\":4,\"size_y\":3,\"sort\":[\"@timestamp\",\"desc\"],\"type\":\"search\"}]", 10 | "optionsJSON": "{\"darkTheme\":true}", 11 | "uiStateJSON": "{\"P-2\":{\"vis\":{\"params\":{\"sort\":{\"columnIndex\":null,\"direction\":null}}}}}", 12 | "version": 1, 13 | "timeRestore": true, 14 | "timeTo": "now", 15 | "timeFrom": "now-1h", 16 | "refreshInterval": { 17 | "display": "Off", 18 | "pause": false, 19 | "value": 0 20 | }, 21 | "kibanaSavedObjectMeta": { 22 | "searchSourceJSON": "{\"filter\":[{\"query\":{\"query_string\":{\"analyze_wildcard\":true,\"query\":\"*\"}}}]}" 23 | } 24 | } 25 | }, 26 | { 27 | "_id": "PARSING-FAILED", 28 | "_type": "search", 29 | "_source": { 30 | "title": "PARSING FAILED", 31 | "description": "", 32 | "hits": 0, 33 | "columns": [ 34 | "_source" 35 | ], 36 | "sort": [ 37 | "@timestamp", 38 | "desc" 39 | ], 40 | "version": 1, 41 | "kibanaSavedObjectMeta": { 42 | "searchSourceJSON": "{\"index\":\"logstash-*\",\"filter\":[],\"highlight\":{\"pre_tags\":[\"@kibana-highlighted-field@\"],\"post_tags\":[\"@/kibana-highlighted-field@\"],\"fields\":{\"*\":{}},\"require_field_match\":false,\"fragment_size\":2147483647},\"query\":{\"query_string\":{\"query\":\"tags:_grokparsefailure\",\"analyze_wildcard\":true}}}" 43 | } 44 | } 45 | }, 46 | { 47 | "_id": "SEARCH", 48 | "_type": "search", 49 | "_source": { 50 | "title": "SEARCH", 51 | "description": "", 52 | "hits": 0, 53 | "columns": [ 54 | "kubernetes.container_name", 55 | "kubernetes.pod_name", 56 | "kubernetes.host", 57 | "log" 58 | ], 59 | "sort": [ 60 | "@timestamp", 61 | "desc" 62 | ], 63 | "version": 1, 64 | "kibanaSavedObjectMeta": { 65 | "searchSourceJSON": "{\"index\":\"logstash-*\",\"filter\":[],\"highlight\":{\"pre_tags\":[\"@kibana-highlighted-field@\"],\"post_tags\":[\"@/kibana-highlighted-field@\"],\"fields\":{\"*\":{}},\"require_field_match\":false,\"fragment_size\":2147483647}}" 66 | } 67 | } 68 | }, 69 | { 70 | "_id": "LOGS", 71 | "_type": "visualization", 72 | "_source": { 73 | "title": "LOGS", 74 | "visState": "{\"title\":\"LOGS\",\"type\":\"histogram\",\"params\":{\"shareYAxis\":true,\"addTooltip\":true,\"addLegend\":true,\"legendPosition\":\"right\",\"scale\":\"linear\",\"mode\":\"stacked\",\"times\":[],\"addTimeMarker\":false,\"defaultYExtents\":false,\"setYExtents\":false,\"yAxis\":{}},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{}},{\"id\":\"2\",\"enabled\":true,\"type\":\"date_histogram\",\"schema\":\"segment\",\"params\":{\"field\":\"@timestamp\",\"interval\":\"auto\",\"customInterval\":\"2h\",\"min_doc_count\":1,\"extended_bounds\":{}}},{\"id\":\"3\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"group\",\"params\":{\"field\":\"kubernetes.container_name.keyword\",\"size\":5,\"order\":\"desc\",\"orderBy\":\"1\"}}],\"listeners\":{}}", 75 | "uiStateJSON": "{}", 76 | "description": "", 77 | "version": 1, 78 | "kibanaSavedObjectMeta": { 79 | "searchSourceJSON": "{\"index\":\"logstash-*\",\"query\":{\"query_string\":{\"query\":\"*\",\"analyze_wildcard\":true}},\"filter\":[]}" 80 | } 81 | } 82 | }, 83 | { 84 | "_id": "CONTAINERS", 85 | "_type": "visualization", 86 | "_source": { 87 | "title": "CONTAINERS", 88 | "visState": "{\"title\":\"CONTAINERS\",\"type\":\"pie\",\"params\":{\"addLegend\":true,\"addTooltip\":true,\"isDonut\":false,\"legendPosition\":\"right\",\"shareYAxis\":true},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{}},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"segment\",\"params\":{\"field\":\"kubernetes.container_name.keyword\",\"size\":5,\"order\":\"desc\",\"orderBy\":\"1\"}}],\"listeners\":{}}", 89 | "uiStateJSON": "{}", 90 | "description": "", 91 | "version": 1, 92 | "kibanaSavedObjectMeta": { 93 | "searchSourceJSON": "{\"index\":\"logstash-*\",\"query\":{\"query_string\":{\"analyze_wildcard\":true,\"query\":\"*\"}},\"filter\":[]}" 94 | } 95 | } 96 | }, 97 | { 98 | "_id": "ERRORS", 99 | "_type": "visualization", 100 | "_source": { 101 | "title": "ERRORS", 102 | "visState": "{\"title\":\"ERRORS\",\"type\":\"histogram\",\"params\":{\"shareYAxis\":true,\"addTooltip\":true,\"addLegend\":true,\"legendPosition\":\"right\",\"scale\":\"linear\",\"mode\":\"stacked\",\"times\":[],\"addTimeMarker\":false,\"defaultYExtents\":false,\"setYExtents\":false,\"yAxis\":{}},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{}},{\"id\":\"2\",\"enabled\":true,\"type\":\"date_histogram\",\"schema\":\"segment\",\"params\":{\"field\":\"@timestamp\",\"interval\":\"auto\",\"customInterval\":\"2h\",\"min_doc_count\":1,\"extended_bounds\":{}}},{\"id\":\"3\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"group\",\"params\":{\"field\":\"kubernetes.container_name.keyword\",\"size\":5,\"order\":\"desc\",\"orderBy\":\"1\"}}],\"listeners\":{}}", 103 | "uiStateJSON": "{}", 104 | "description": "", 105 | "version": 1, 106 | "kibanaSavedObjectMeta": { 107 | "searchSourceJSON": "{\"index\":\"logstash-*\",\"query\":{\"query_string\":{\"query\":\"*error* OR *fatal* OR *fail* OR *timeout* OR *reset*\",\"analyze_wildcard\":true}},\"filter\":[]}" 108 | } 109 | } 110 | }, 111 | { 112 | "_id": "ERROR-LOGS", 113 | "_type": "visualization", 114 | "_source": { 115 | "title": "ERROR LOGS", 116 | "visState": "{\"title\":\"ERROR LOGS\",\"type\":\"table\",\"params\":{\"perPage\":10,\"showPartialRows\":false,\"showMeticsAtAllLevels\":false,\"sort\":{\"columnIndex\":null,\"direction\":null},\"showTotal\":false,\"totalFunc\":\"sum\"},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{}},{\"id\":\"3\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"bucket\",\"params\":{\"field\":\"log.keyword\",\"size\":5,\"order\":\"desc\",\"orderBy\":\"1\",\"customLabel\":\"log\"}},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"bucket\",\"params\":{\"field\":\"kubernetes.container_name.keyword\",\"size\":5,\"order\":\"desc\",\"orderBy\":\"1\",\"customLabel\":\"container\"}}],\"listeners\":{}}", 117 | "uiStateJSON": "{\"vis\":{\"params\":{\"sort\":{\"columnIndex\":null,\"direction\":null}}}}", 118 | "description": "", 119 | "version": 1, 120 | "kibanaSavedObjectMeta": { 121 | "searchSourceJSON": "{\"index\":\"logstash-*\",\"query\":{\"query_string\":{\"query\":\"*error* OR *fatal* OR *fail* OR *timeout* OR *reset*\",\"analyze_wildcard\":true}},\"filter\":[]}" 122 | } 123 | } 124 | } 125 | ] 126 | -------------------------------------------------------------------------------- /logging/elasticsearch-deployment.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: extensions/v1beta1 3 | kind: Deployment 4 | metadata: 5 | name: elasticsearch 6 | namespace: logging 7 | labels: 8 | app: elastcisearch 9 | spec: 10 | replicas: 1 11 | template: 12 | metadata: 13 | name: elasticsearch 14 | labels: 15 | app: elasticsearch 16 | spec: 17 | containers: 18 | - name: elasticsearch 19 | image: elasticsearch:5 20 | ports: 21 | - containerPort: 9200 22 | - containerPort: 9300 23 | env: 24 | - name: ES_JAVA_OPTS 25 | value: "-Xms1g -Xmx1g" 26 | 27 | -------------------------------------------------------------------------------- /logging/elasticsearch-service.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: v1 3 | kind: Service 4 | metadata: 5 | name: elasticsearch 6 | namespace: logging 7 | spec: 8 | type: NodePort 9 | selector: 10 | app: elasticsearch 11 | ports: 12 | - name: http 13 | port: 9200 14 | protocol: TCP 15 | nodePort: 30200 16 | - name: transport 17 | port: 9300 18 | protocol: TCP 19 | nodePort: 30300 20 | -------------------------------------------------------------------------------- /logging/fluentd-daemonset.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: extensions/v1beta1 3 | kind: DaemonSet 4 | metadata: 5 | name: fluentd 6 | namespace: logging 7 | spec: 8 | template: 9 | metadata: 10 | labels: 11 | app: fluentd 12 | spec: 13 | containers: 14 | - name: fluentd 15 | image: gcr.io/google_containers/fluentd-elasticsearch:1.20 16 | command: 17 | - '/bin/sh' 18 | - '-c' 19 | - '/usr/sbin/td-agent -vv 2>&1 >>/var/log/fluentd.log' 20 | resources: 21 | limits: 22 | memory: 200Mi 23 | requests: 24 | cpu: 100m 25 | memory: 200Mi 26 | volumeMounts: 27 | - name: varlog 28 | mountPath: /var/log 29 | - name: varlibdockercontainers 30 | mountPath: /var/lib/docker/containers 31 | readOnly: true 32 | - name: fluentd-conf 33 | mountPath: /etc/td-agent 34 | terminationGracePeriodSeconds: 30 35 | volumes: 36 | - name: varlog 37 | hostPath: 38 | path: /var/log 39 | - name: varlibdockercontainers 40 | hostPath: 41 | path: /var/lib/docker/containers 42 | - name: fluentd-conf 43 | configMap: 44 | name: fluentd-conf 45 | --- 46 | apiVersion: v1 47 | data: 48 | td-agent.conf: | 49 | { 50 | # This configuration file for Fluentd / td-agent is used 51 | # to watch changes to Docker log files. The kubelet creates symlinks that 52 | # capture the pod name, namespace, container name & Docker container ID 53 | # to the docker logs for pods in the /var/log/containers directory on the host. 54 | # If running this fluentd configuration in a Docker container, the /var/log 55 | # directory should be mounted in the container. 56 | # 57 | # These logs are then submitted to Elasticsearch which assumes the 58 | # installation of the fluent-plugin-elasticsearch & the 59 | # fluent-plugin-kubernetes_metadata_filter plugins. 60 | # See https://github.com/uken/fluent-plugin-elasticsearch & 61 | # https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter for 62 | # more information about the plugins. 63 | # Maintainer: Jimmi Dyson 64 | # 65 | # Example 66 | # ======= 67 | # A line in the Docker log file might look like this JSON: 68 | # 69 | # {"log":"2014/09/25 21:15:03 Got request with path wombat\n", 70 | # "stream":"stderr", 71 | # "time":"2014-09-25T21:15:03.499185026Z"} 72 | # 73 | # The time_format specification below makes sure we properly 74 | # parse the time format produced by Docker. This will be 75 | # submitted to Elasticsearch and should appear like: 76 | # $ curl 'http://elasticsearch-logging:9200/_search?pretty' 77 | # ... 78 | # { 79 | # "_index" : "logstash-2014.09.25", 80 | # "_type" : "fluentd", 81 | # "_id" : "VBrbor2QTuGpsQyTCdfzqA", 82 | # "_score" : 1.0, 83 | # "_source":{"log":"2014/09/25 22:45:50 Got request with path wombat\n", 84 | # "stream":"stderr","tag":"docker.container.all", 85 | # "@timestamp":"2014-09-25T22:45:50+00:00"} 86 | # }, 87 | # ... 88 | # 89 | # The Kubernetes fluentd plugin is used to write the Kubernetes metadata to the log 90 | # record & add labels to the log record if properly configured. This enables users 91 | # to filter & search logs on any metadata. 92 | # For example a Docker container's logs might be in the directory: 93 | # 94 | # /var/lib/docker/containers/997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b 95 | # 96 | # and in the file: 97 | # 98 | # 997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b-json.log 99 | # 100 | # where 997599971ee6... is the Docker ID of the running container. 101 | # The Kubernetes kubelet makes a symbolic link to this file on the host machine 102 | # in the /var/log/containers directory which includes the pod name and the Kubernetes 103 | # container name: 104 | # 105 | # synthetic-logger-0.25lps-pod_default_synth-lgr-997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b.log 106 | # -> 107 | # /var/lib/docker/containers/997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b/997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b-json.log 108 | # 109 | # The /var/log directory on the host is mapped to the /var/log directory in the container 110 | # running this instance of Fluentd and we end up collecting the file: 111 | # 112 | # /var/log/containers/synthetic-logger-0.25lps-pod_default_synth-lgr-997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b.log 113 | # 114 | # This results in the tag: 115 | # 116 | # var.log.containers.synthetic-logger-0.25lps-pod_default_synth-lgr-997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b.log 117 | # 118 | # The Kubernetes fluentd plugin is used to extract the namespace, pod name & container name 119 | # which are added to the log message as a kubernetes field object & the Docker container ID 120 | # is also added under the docker field object. 121 | # The final tag is: 122 | # 123 | # kubernetes.var.log.containers.synthetic-logger-0.25lps-pod_default_synth-lgr-997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b.log 124 | # 125 | # And the final log record look like: 126 | # 127 | # { 128 | # "log":"2014/09/25 21:15:03 Got request with path wombat\n", 129 | # "stream":"stderr", 130 | # "time":"2014-09-25T21:15:03.499185026Z", 131 | # "kubernetes": { 132 | # "namespace": "default", 133 | # "pod_name": "synthetic-logger-0.25lps-pod", 134 | # "container_name": "synth-lgr" 135 | # }, 136 | # "docker": { 137 | # "container_id": "997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b" 138 | # } 139 | # } 140 | # 141 | # This makes it easier for users to search for logs by pod name or by 142 | # the name of the Kubernetes container regardless of how many times the 143 | # Kubernetes pod has been restarted (resulting in a several Docker container IDs). 144 | # 145 | # TODO: Propagate the labels associated with a container along with its logs 146 | # so users can query logs using labels as well as or instead of the pod name 147 | # and container name. This is simply done via configuration of the Kubernetes 148 | # fluentd plugin but requires secrets to be enabled in the fluent pod. This is a 149 | # problem yet to be solved as secrets are not usable in static pods which the fluentd 150 | # pod must be until a per-node controller is available in Kubernetes. 151 | # Do not directly collect fluentd's own logs to avoid infinite loops. 152 | 153 | type null 154 | 155 | # Example: 156 | # {"log":"[info:2016-02-16T16:04:05.930-08:00] Some log text here\n","stream":"stdout","time":"2016-02-17T00:04:05.931087621Z"} 157 | 158 | type tail 159 | path /var/log/containers/*.log 160 | pos_file /var/log/es-containers.log.pos 161 | time_format %Y-%m-%dT%H:%M:%S.%NZ 162 | tag kubernetes.* 163 | format json 164 | read_from_head true 165 | 166 | # Example: 167 | # 2015-12-21 23:17:22,066 [salt.state ][INFO ] Completed state [net.ipv4.ip_forward] at time 23:17:22.066081 168 | 169 | type tail 170 | format /^(?