├── images ├── .keep └── kubectl-tools.jpg └── README.md /images/.keep: -------------------------------------------------------------------------------- 1 | 2 | -------------------------------------------------------------------------------- /images/kubectl-tools.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mfilotto/k8s-tips-n-tricks/HEAD/images/kubectl-tools.jpg -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Kubernetes tips and tricks 2 | 3 | ## Kubernetes new releases 4 | https://sysdig.com/?s=What%E2%80%99s+new+in+Kubernetes++ 5 | 6 | ## kubectl commands 7 | 8 | ### Official kubectl Cheat Sheet 9 | https://kubernetes.io/docs/reference/kubectl/cheatsheet/ 10 | 11 | ### View resource usage 12 | ``` 13 | kubectl top no 14 | kubectl top po 15 | ``` 16 | ### Display docker image tage and SHA 17 | `kubectl get pod -ojson | jq '.status.containerStatuses[] | { "image": .image, "imageID": .imageID }'` 18 | 19 | ### Display logs for previous started container to debug abnormal successive restarts 20 | `kubectl logs --previous` 21 | 22 | ### Display http requests made by kubectl to kube-api 23 | `kubectl get po -v=6` 24 | 25 | ### Extract a token from a secret 26 | `kubectl get secret -n -ojsonpath='{.data.token}' | base64 -d` 27 | 28 | ### Extract the content of a file from a secret (dot in file name must be escaped by \\) 29 | `kubectl get secret -ojsonpath='{.data.jmxremote\\.password}' | base64 -d` 30 | 31 | ### Copy an object from one namespace to another 32 | `kubectl get secrets -ojson -n | jq '.metadata.namespace = ""' | kubectl create -f -` 33 | 34 | ### Clean up an helm release manually 35 | `kubectl get deploy,sts,cm,secret,pvc,svc -oname -lrelease= | while read name; do kubectl delete $name; done` 36 | 37 | ### Wait for pod to be ready 38 | `kubectl wait po --for=condition=Ready` 39 | 40 | ### Watch pods 41 | `watch kubectl get po -lrelease=` 42 | 43 | ### Find pods by date with jq 44 | ``` 45 | DEPLOYMENT_STARTDATE=`jq -n 'now'` 46 | kubectl get po -lrelease= -ojson | jq -r --arg deployment_startdate $DEPLOYMENT_STARTDATE '.items[] | select(.metadata.creationTimestamp | fromdate | tostring > $deployment_startdate) | .metadata.name' 47 | ``` 48 | 49 | ### Find pods using a specific environment variable in secret 50 | ``` 51 | kubectl get po -ojson | jq -r '.items[] | select(.spec.containers[].env[]?.valueFrom.secretKeyRef.key=="") | .metadata.name' 52 | ``` 53 | 54 | ### Find deployments using a specific environment variable in secret 55 | ``` 56 | kubectl get deploy -ojson | jq -r '.items[] | select(.spec.template.spec.containers[].env[]?.valueFrom.secretKeyRef.key=="") | .metadata.name' 57 | ``` 58 | 59 | ### Inject an environment variable in a deployment 60 | `kubectl set env deployment/registry STORAGE_DIR=/local` 61 | 62 | ### Restart pods properly with a rollout (from 1.15) 63 | `kubectl rollout restart deploy ` 64 | 65 | ### Check if I'm allowed to do an action 66 | `kubectl auth can-i exec pod` 67 | 68 | ### Suspend all cronjobs at once 69 | `kubectl get cj -oname | while read name; do kubectl $name -p '{"spec":{"suspend":true}}'; done` 70 | 71 | ### Restart pod launched by job 72 | `kubectl get job -o json | jq 'del(.spec.selector)' | jq 'del(.spec.template.metadata.labels)' | kubectl replace --force -f -` 73 | 74 | ### Patch all Terminating PVC to remove the finalizer 75 | ``` 76 | NS=mynamespace 77 | kubectl -n $NS patch pvc $(kubectl -n $NS get pvc --no-headers | grep Terminating | awk '{print $1}') -p '{"metadata":{"finalizers":null}}' 78 | ``` 79 | 80 | ### List image in a deployment 81 | `kubectl get deploy -lrelease=si-labo -ojson | jq .items[].spec.template.spec.containers[0].image` 82 | 83 | ### List all image references in a namespace 84 | `kubectl get deploy -ojson | jq -r '.. | .image? // empty' | sort -u` 85 | 86 | ### List pods in status other than Running or Completed 87 | `kubectl get po -owide -A | grep -v 'Running\|Completed'` 88 | 89 | ### List evicted pods on all cluster 90 | - `kubectl get po --field-selector=status.phase=Failed -A -owide` 91 | - `kubectl get po -A -ojson | jq -r '.items[] | select(.status.reason=="Evicted") | .metadata.namespace + " " + .spec.nodeName + " " + (.spec.priority|tostring)+ " " + .metadata.name + " : " + .status.message' | sort -k2,2 -k3nr` 92 | 93 | ### List pods with anti affinity 94 | `kubectl get po -ojson | jq '.items[] | select(.spec.affinity.podAntiAffinity!=null) | .metadata.name'` 95 | 96 | ### List pods with a guaranteed qos 97 | `kubectl get po -ojson | jq '.items[] | select(.status.qosClass=="Guaranteed") | .metadata.name'` 98 | 99 | ### List prority classes sort by value 100 | `kubectl get pc -ojson | jq -r '.items[] | .metadata.name + " : " + (.value|tostring)' | sort -k3nr` 101 | 102 | ### List priority infos for all pods sort by value 103 | `kubectl get po -ojson | jq -r '.items[] | .metadata.namespace + " : " + .spec.nodeName + " : " + .metadata.name + " : " + .spec.priorityClassName+ " : " + (.spec.priority|tostring)' | sort -k9nr -k5` 104 | 105 | ### List pods by restart count 106 | `kubectl get po --sort-by='.status.containerStatuses[0].restartCount'` 107 | 108 | ### List pods by age 109 | `kubectl get po --sort-by=.status.startTime` 110 | 111 | ### List all OOMKilled pods 112 | `kubectl get po -A -ojson | jq -r '.items[] | select(.status.containerStatuses[0].lastState.terminated.reason=="OOMKilled") | .metadata.namespace + " " + (.status.containerStatuses[0].restartCount|tostring) + " " + .metadata.name' | sort -k1,1r -k2nr` 113 | 114 | ### List all pods with privileged mode 115 | `kubectl get po -ojson -A | jq '.items[] | select(.spec.containers[].securityContext.privileged==true) | .metadata.namespace + " : " + .metadata.name'` 116 | 117 | ### List all pods using host's IPC namespace 118 | `kubectl get po -ojson -A | jq '.items[] | select(.spec.hostIPC==true) | .metadata.namespace + " : " + .metadata.name'` 119 | 120 | ### List all pods using host's network namespace 121 | `kubectl get po -ojson -A | jq '.items[] | select(.spec.hostNetwork==true) | .metadata.namespace + " : " + .metadata.name'` 122 | 123 | ### List all pods using host's PID namespace 124 | `kubectl get po -ojson -A | jq '.items[] | select(.spec.hostPID==true) | .metadata.namespace + " : " + .metadata.name'` 125 | 126 | ### Test anonymous access to health endpoints 127 | ``` 128 | KUBE_API_URL= 129 | for ep in version healthz livez readyz; do curl -k $KUBE_API_URL/$ep?verbose; done 130 | ``` 131 | ### List anonymous access 132 | `kubectl get clusterrolebindings -o json | jq '.items[] | select(.subjects? // [] | any(.kind == "User" and .name == "system:anonymous" or .kind == "Group" and .name == "system:unauthenticated"))'` 133 | 134 | ### Force delete a pod stuck in terminating status 135 | `kubectl delete pod --grace-period=0 --force` 136 | 137 | ### Force delete all pods stuck in terminating status at once 138 | `kubectl get po -owide | grep 'Terminating' | awk -F ' ' '{print $1}' | while read name; do kubectl delete po $name --grace-period=0 --force; done` 139 | 140 | ### List pre hook jobs for a release 141 | `kubectl get jobs -ojson | jq -r '.items[] | select(.metadata.annotations["helm.sh/hook"] and (.metadata.annotations["helm.sh/hook"]|contains("pre")) and .metadata.labels.release=="") | .metadata.name'` 142 | 143 | ### List pod owned by a hook job 144 | `kubectl get po -ojson | jq -r '.items[] | select(.metadata.ownerReferences[].name == "") | .metadata.name'` 145 | 146 | ### Delete succeeded jobs 147 | `kubectl get jobs -ojson | jq -r '.items[] | select(.metadata.annotations["helm.sh/hook"] and .status.succeeded==1) | .metadata.name' | while read name; do kubectl delete jobs $name ; done` 148 | 149 | ### List nodes with memory or disk pressure Taint Based Evictions 150 | `kubectl get no -ojson | jq -r '.items[] | select(.spec.taints!=null and (.spec.taints[0].key|contains("pressure"))) | .metadata.name + " : " + .spec.taints[0].key'` 151 | 152 | ### List allocated ressources per node 153 | `kubectl get no --no-headers | awk '{print $1}' | xargs -I {} sh -c 'echo {}; kubectl describe node {} | grep Allocated -A 5 | grep -ve Event -ve Allocated -ve percent -ve -- ; echo'` 154 | `for i in {01..12}; do echo dbk-k8s-worker-dev-${i}v; kubectl describe node dbk-k8s-worker-dev-${i}v|grep -A6 'Allocated resources:'; done` 155 | 156 | ### Drain a node 157 | `kubectl drain --ignore-daemonsets --force --delete-local-data` 158 | 159 | ## ctr commands 160 | 161 | ### List images 162 | `ctr --namespace k8s.io i ls` 163 | 164 | ### Push an image to a private registry 165 | `ctr --namespace k8s.io i push -u : ` 166 | 167 | ### Official helm Cheat Sheet 168 | https://helm.sh/docs/intro/cheatsheet/ 169 | 170 | ## helm commands 171 | 172 | ### Render chart templates locally 173 | `helm template . --output-dir=output-dir` 174 | 175 | ### List resources of a release 176 | `helm status --show-resources` 177 | 178 | ### Get all values of a release 179 | `helm get values -a` 180 | 181 | ### Get manifests of a release 182 | `helm get manifest ` 183 | 184 | ### Official flux Commands list 185 | https://fluxcd.io/flux/cmd/ 186 | 187 | ## flux commands 188 | 189 | ### Display all flux objects to check cluster health 190 | `flux get all -A` 191 | 192 | ### Display all not ready flux objects 193 | `flux get all -n --status-selector="ready=false"` 194 | 195 | ### Display objects managed by a kustomization 196 | `flux tree ks -n` 197 | 198 | ### Force kustomization reconciliation 199 | `flux reconcile ks -n --with-source` 200 | 201 | ### Force helm release reconciliation 202 | `flux reconcile hr -n --with-source` 203 | 204 | ### Display diff between local and server kustomization 205 | `flux diff ks --path= -n` 206 | 207 | ### Search for drifts in reconciliation (helm revisions should increase periodically in case of an unwanted drift) 208 | `flux get hr -ndev --no-header | awk '{print $1}' | while read name; do helm history $name --max 1; done` 209 | 210 | ### Search for replicas specified in helm manifests (no defined replicas allow to scale without any detected drift) 211 | `flux get hr -ndev --no-header | awk '{print $1}' | while read name; do bash -c "echo $name && helm get manifest $name | grep replica"; done` 212 | 213 | ## Some recipes 214 | 215 | ### Browse google registry 216 | https://console.cloud.google.com/gcr/images/google-containers/GLOBAL 217 | 218 | ### Ease kubectl use 219 | ![Test](/images/kubectl-tools.jpg?raw=true) 220 | 221 | ### Add kubectl aliases 222 | https://github.com/ahmetb/kubectl-aliases 223 | 224 | ### Decode all secret content easily with ksd 225 | https://github.com/ashleyschuett/kubernetes-secret-decode 226 | 227 | `kubectl get secret my-secret -o yaml | ksd` 228 | 229 | ### Add fuzzy search to your command with fzf 230 | https://github.com/junegunn/fzf 231 | 232 | `kubectl get po | fzf` 233 | 234 | ### Read logs from all replicas at a time with stern 235 | https://github.com/wercker/stern 236 | 237 | ### Advanced Kubernetes Objects You Need to Know 238 | https://engineering.opsgenie.com/advanced-kubernetes-objects-53f5e9bc0c28 239 | 240 | ### Interact with kube-api like any other API 241 | https://thenewstack.io/taking-kubernetes-api-spin/ 242 | 243 | ### How to terminate a side-car container in Kubernetes Job 244 | https://medium.com/@cotton_ori/how-to-terminate-a-side-car-container-in-kubernetes-job-2468f435ca99 245 | 246 | ### Docker Awareness in Java 247 | - https://efekahraman.github.io/2018/04/docker-awareness-in-java 248 | - https://blog.csanchez.org/2017/05/31/running-a-jvm-in-a-container-without-getting-killed/ 249 | - https://blogs.oracle.com/java-platform-group/java-se-support-for-docker-cpu-and-memory-limits 250 | - https://banzaicloud.com/blog/java-resource-limits/ 251 | 252 | ### Docker-in-Docker on Kubernetes 253 | https://applatix.com/case-docker-docker-kubernetes-part-2/ 254 | 255 | ### How To Back Up and Restore a Kubernetes Cluster using Heptio Ark 256 | https://www.digitalocean.com/community/tutorials/how-to-back-up-and-restore-a-kubernetes-cluster-on-digitalocean-using-heptio-ark 257 | 258 | ### Configuring the Kubernetes CLI by using service account tokens 259 | https://www.ibm.com/developerworks/community/blogs/fe25b4ef-ea6a-4d86-a629-6f87ccf4649e/entry/Configuring_the_Kubernetes_CLI_by_using_service_account_tokens1?lang=en 260 | 261 | ### Treat your pods according to their needs - three QoS classes in Kubernetes 262 | https://cloudowski.com/articles/three-qos-classes-in-kubernetes/ 263 | 264 | ### Prometheus Operator 265 | https://github.com/helm/charts/tree/master/stable/prometheus-operator 266 | https://github.com/coreos/prometheus-operator/tree/master/Documentation 267 | https://sysdig.com/blog/kubernetes-monitoring-prometheus-operator-part3/ 268 | 269 | ### Kube eagle: prometheus exporter and grafana dashboard for a nice overview of a cluster 270 | https://github.com/cloudworkz/kube-eagle 271 | 272 | ### Checklist of production ready best practices for a kubernetes cluster 273 | https://learnk8s.io/production-best-practices 274 | 275 | ### Lens a great Kubernetes IDE 276 | https://k8slens.dev/ 277 | 278 | ## Resources 279 | - https://hackernoon.com/top-10-kubernetes-tips-and-tricks-27528c2d0222 280 | - https://github.com/mhausenblas/kubectl-in-action 281 | - https://discuss.kubernetes.io/t/kubectl-tips-and-tricks/ 282 | --------------------------------------------------------------------------------