├── EXAMPLE-k8s-utils-config.sh ├── README.md ├── k8s ├── k8s-describe-nodes ├── k8s-pod-errors ├── k8s-pod-grep ├── k8s-pods ├── k8s-port-forward ├── k8s-tail ├── k8s-top-nodes ├── k8s-top-pods └── load-config.sh /EXAMPLE-k8s-utils-config.sh: -------------------------------------------------------------------------------- 1 | # Sample config load 2 | # This demonstrates use of 2 k8s clusters - dev & prod - each with 2 namespaces. 3 | # In reality this can be as simple or complicated as your k8s setup requires. 4 | 5 | env=$1 6 | 7 | # sample dev cluster - custom namespaces 8 | if [[ $env == "dev" ]]; then 9 | cfg="config.dev" 10 | namespace="dev" 11 | elif [[ $env == "qa" ]]; then 12 | cfg="config.dev" 13 | namespace="qa" 14 | 15 | # sample dev cluster - default and system namespaces 16 | elif [[ $env == "dev-default" ]]; then 17 | cfg="config.dev" 18 | namespace="default" 19 | elif [[ $env == "dev-sys" ]]; then 20 | cfg="config.dev" 21 | namespace="kube-system" 22 | 23 | # sample prod cluster - custom namespaces 24 | elif [[ $env == "stage" ]]; then 25 | cfg="config.prod" 26 | namespace="stage" 27 | elif [[ $env == "prod" ]]; then 28 | cfg="config.prod" 29 | namespace="prod" 30 | 31 | # sample prod cluster - default and system namespaces 32 | elif [[ $env == "prod-default" ]]; then 33 | cfg="config.prod" 34 | namespace="default" 35 | elif [[ $env == "prod-sys" ]]; then 36 | cfg="config.prod" 37 | namespace="kube-system" 38 | fi 39 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # k8s-utils 2 | 3 | This repo is a collection of small bash scripts oriented primarily to assist with Kubernetes production support, although they can certainly be used in non-prod environments as well. 4 | 5 | ## Setup 6 | 7 | ### Add to PATH 8 | 9 | Add the cloned copy of this repo directly to your PATH via `.bash_profile`, `.profile`, `.bashrc`, or equivalent. 10 | 11 | export PATH="$PATH:/path/to/k8s-utils" 12 | 13 | > Note: currently these scripts have only been tested on OSX. 14 | 15 | ### Create script to define your k8s envs 16 | 17 | It's necessary to create a script at `~/.kube/k8s-utils-config.sh` which will define the `` param values used below. This will detail the k8s config file(s) and namespaces specific to your k8s environment. 18 | 19 | A sample script is provided which can be copied to your `~/.kube` dir: 20 | 21 | cp EXAMPLE-k8s-utils-config.sh ~/.kube/k8s-utils-config.sh 22 | 23 | It will then need to be modified to reflect your config file and namespace setup. 24 | 25 | ## General notes 26 | 27 | ### Grepping pod names 28 | 29 | A number of these scripts use a parameter named `pod_names_to_grep`. The convention here is that this will be passed to `kubectl get pods | grep $pod_names_to_grep`. 30 | 31 | As such, it's important to be specific enough to not pick up pods you weren't expecting. Alternatively, you can also provide an actual pod name to narrow the scope down to a single pod. 32 | 33 | Typically `pod_names_to_grep` will be the name of the service you're interested in. 34 | 35 | ### Each command is env-specific 36 | 37 | Beyond the general convenience aspect, a large intent of these scripts is to not have to set the kubectl context between commands. By just changing the `` param the same command can quickly be run against different envs. Additionally, I find it worthile to explicitly state what env each command is being executed against. 38 | 39 | ## Scripts 40 | 41 | ### k8s 42 | 43 | Usage: `k8s ` 44 | 45 | - Env-specific shortcut for using `kubectl` directly. 46 | 47 | - Example: `k8s dev get pods` 48 | 49 | --- 50 | 51 | ### k8s-pods 52 | 53 | Usage: `k8s-pods ` 54 | 55 | - This script is an alternative to using `kubectl get pods -w`. It uses the `watch` command and is useful to keep an eye on pods when doing a deploy, to see them move between Terminating, Pending, and Running statuses. 56 | 57 | - Note that `kubectl rollout status deployment/ -w` can also be used to monitor a deploy, but the command is a bit more verbose and doesn't show the same level of detail. For instance, if you have multiple init containers this will let you see that the deploy is running into problems on container 2/4, for example. 58 | 59 | - Example: `k8s-pods stage search-api` 60 | 61 | - Example result: 62 | ``` 63 | Every 2.0s: kubectl --namespace=stage get pods | grep search-api 64 | 65 | search-api-1568909879-476vm 1/1 Running 0 1d 66 | search-api-1568909879-f2pdb 1/1 Running 0 1d 67 | search-api-1568909879-gzd9n 1/1 Running 0 1d 68 | ``` 69 | 70 | - It can also be useful to quickly check the status of multiple pods across services, by using a less specific `pod_names_to_grep` value. 71 | 72 | --- 73 | 74 | ### k8s-tail 75 | 76 | Usage: `k8s-tail ` 77 | 78 | - Use this to tail the logs for a service in a given environment. If there are multiple pods running, it merges all of the logs together into a single output -- useful, but verbose. 79 | 80 | - Because all pod logs are merged into a single output stream, this is more useful for a quick look at a service as opposed to debugging specific issues. 81 | 82 | - Example: `k8s-tail qa catalog-api` 83 | 84 | - Example: `k8s-tail production recommender-api --since=5m` 85 | 86 | - It is also possible to pass a specific pod name for the grep value, which will be equivalent to doing `kubectl logs -f `. 87 | 88 | - For a more full-featured tail script, see [johanhaleby/kubetail](https://github.com/johanhaleby/kubetail). 89 | 90 | --- 91 | 92 | ### k8s-pod-errors 93 | 94 | Usage: `k8s-pod-errors ` 95 | 96 | - This script will grep for case-insentitive "error" lines in the logs of the pods in the given service, by default over the last 5 minutes. If a service is alerting, this is useful to help determine if one pod is acting as an outlier or if all pods are reporting similar error levels and any problem is service-wide. 97 | 98 | - Example: `k8s-pod-errors production recommender-api` 99 | 100 | - Example result: 101 | ``` 102 | pod = recommender-api-1966416721-dl4k3, error count = 178 103 | pod = recommender-api-1966416721-kdc3g, error count = 212 104 | pod = recommender-api-1966416721-wsjn4, error count = 153 105 | ``` 106 | 107 | - Example result of an outlier pod: 108 | ``` 109 | pod = recommender-api-1966416721-0dyv3, error count = 8 110 | pod = recommender-api-1966416721-7li0w, error count = 9843 111 | pod = recommender-api-1966416721-mmgna, error count = 9 112 | pod = recommender-api-1966416721-q6us9, error count = 9 113 | pod = recommender-api-1966416721-vqz9h, error count = 11 114 | pod = recommender-api-1966416721-yyd5s, error count = 4 115 | ``` 116 | 117 | - Example of changing the --since= window: `k8s-pod-errors dev recommender-api 1h` 118 | 119 | --- 120 | 121 | ### k8s-pod-grep 122 | 123 | Usage: `k8s-pod-grep ` 124 | 125 | - This script is similar to `k8s-pod-errors` but instead of doing a hard coded `grep -i error` this script allows the user to input the string param to grep. (Again, by default over the last 5 minutes.) This can be useful to drill into specific errors or other behaviors when there's a concern it may not be consistently happening across all pods for the given service. 126 | 127 | - Example: `k8s-pod-grep production customer-api "Read timed out"` 128 | ``` 129 | pod = customer-api-1966416721-dl4k3, grep count = 0 130 | pod = customer-api-1966416721-kdc3g, grep count = 4 131 | pod = customer-api-1966416721-wsjn4, grep count = 5 132 | ``` 133 | 134 | - Example of changing the --since= window: `k8s-pod-grep qa customer-api 30m` 135 | 136 | --- 137 | 138 | ### k8s-top-pods 139 | 140 | Usage: `k8s-top-pods ` 141 | 142 | - Use this to look at CPU and memory consumption at the pod/container level. It calls `kubectl top pod --containers` on each pod found. 143 | 144 | - Example: `k8s-top-pods production order-api` 145 | ``` 146 | POD NAME CPU(cores) MEMORY(bytes) 147 | order-api-2838787885-dnp5q order-api 2m 1496Mi 148 | POD NAME CPU(cores) MEMORY(bytes) 149 | order-api-2838787885-22n13 order-api 8m 1507Mi 150 | POD NAME CPU(cores) MEMORY(bytes) 151 | order-api-2838787885-7dcq8 order-api 5m 1554Mi 152 | ``` 153 | 154 | - Note: requires your cluster is running [Heapster](https://github.com/kubernetes/heapster). 155 | 156 | --- 157 | 158 | ### k8s-top-nodes 159 | 160 | Usage: `k8s-top-nodes ` 161 | 162 | - Use this to look at CPU and memory consumption at the node level. It calls `kubectl top node` on each node in the cluster related to the given environment. 163 | 164 | - Example: `k8s-top-nodes production` 165 | ``` 166 | NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% 167 | ip-172-00-000-000.us-west-2.compute.internal 417m 2% 23407Mi 36% 168 | NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% 169 | ip-172-11-111-111.us-west-2.compute.internal 781m 4% 25774Mi 40% 170 | NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% 171 | ip-172-22-222-222.us-west-2.compute.internal 533m 3% 34249Mi 53% 172 | ``` 173 | 174 | - Note: requires your cluster is running [Heapster](https://github.com/kubernetes/heapster). 175 | 176 | --- 177 | 178 | ### k8s-describe-nodes 179 | 180 | Usage: `k8s-describe-nodes ` 181 | 182 | - Use this to pull information on the nodes in a given environment. Note that this is not too different from simply running `kubectl describe node` on all nodes in the cluster, but it attempts to clean up and minimize the output in order to focus on finding nodes that are out of disk space and/or have over-provisioned CPU and/or RAM. 183 | 184 | - Example: `k8s-describe-nodes production` 185 | 186 | - Example result: 187 | ``` 188 | node = ip-172-11-11-1.us-west-2.compute.internal 189 | ------------------------------------------------ 190 | Conditions: 191 | Type Status LastHeartbeatTime LastTransitionTime Reason Message 192 | ---- ------ ----------------- ------------------ ------ ------- 193 | OutOfDisk False Wed, 11 Oct 2017 13:56:56 -0700 Mon, 02 Oct 2017 17:55:09 -0700 KubeletHasSufficientDisk kubelet has sufficient disk space available 194 | MemoryPressure False Wed, 11 Oct 2017 13:56:56 -0700 Mon, 02 Oct 2017 17:55:09 -0700 KubeletHasSufficientMemory kubelet has sufficient memory available 195 | DiskPressure False Wed, 11 Oct 2017 13:56:56 -0700 Mon, 02 Oct 2017 17:55:09 -0700 KubeletHasNoDiskPressure kubelet has no disk pressure 196 | Ready True Wed, 11 Oct 2017 13:56:56 -0700 Mon, 02 Oct 2017 17:55:20 -0700 KubeletReady kubelet is posting ready status 197 | -- 198 | Allocated resources: 199 | (Total limits may be over 100 percent, i.e., overcommitted.) 200 | CPU Requests CPU Limits Memory Requests Memory Limits 201 | ------------ ---------- --------------- ------------- 202 | 8510m (53%) 7566m (47%) 20477Mi (31%) 19325Mi (30%) 203 | Events: 204 | ``` 205 | 206 | --- 207 | 208 | ### k8s-port-forward 209 | 210 | Usage: `k8s-port-forward ` 211 | 212 | - This script is a minor convenience over using `kubectl port-forward` directly, in that it's not necessary to specify the specific pod to port forward from. Instead it forwards from the first pod matching the given grep string. 213 | 214 | - Example: `k8s-port-forward mk web 8080:8080` 215 | 216 | - Example result: 217 | ``` 218 | port-forwarding from pod concourse-minikube-web-5b66ffbc6-cjgd7 219 | Forwarding from 127.0.0.1:8080 -> 8080 220 | Forwarding from [::1]:8080 -> 8080 221 | Handling connection for 8080 222 | ... 223 | ``` 224 | -------------------------------------------------------------------------------- /k8s: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | usage="k8s " 4 | notes="shortcut for using 'kubectl' directly" 5 | 6 | # reads "env" param, loads "namespace" param 7 | . "$(dirname "${BASH_SOURCE[0]}")/load-config.sh" 8 | 9 | kubectl --namespace=$namespace "${@:2}" 10 | -------------------------------------------------------------------------------- /k8s-describe-nodes: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | usage="k8s-describe-nodes " 4 | notes="calls 'kubectl describe node' on all nodes in the cluster for the given env, and truncates the output to hopefully the most interesting bits" 5 | 6 | # reads "env" param, loads "namespace" param 7 | . "$(dirname "${BASH_SOURCE[0]}")/load-config.sh" 8 | 9 | for node in $(kubectl get nodes | grep 'node\|master' | cut -f 1 -d ' '); do 10 | echo "node = $node" 11 | echo "------------------------------------------------" 12 | kubectl describe node $node | egrep "Allocated resources|Conditions" -A 6 13 | echo "" 14 | done 15 | -------------------------------------------------------------------------------- /k8s-pod-errors: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | usage="k8s-pod-errors " 4 | notes="greps for case-insentitive 'error' string over the last 5 minutes. Can optionally add additional param like '10m' or '24h' to change the --since= value." 5 | 6 | pod_names=$2 7 | since="${3:-5m}" 8 | 9 | # reads "env" param, loads "namespace" param 10 | . "$(dirname "${BASH_SOURCE[0]}")/load-config.sh" 11 | 12 | for pod in $(kubectl --namespace=$namespace get pods | grep $pod_names | cut -f 1 -d ' '); do 13 | error_count=$(kubectl --namespace=$namespace logs $pod --since=$since | grep -ai 'error' | wc -l) 14 | echo "pod = $pod, error count = $error_count" 15 | done 16 | -------------------------------------------------------------------------------- /k8s-pod-grep: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | usage="k8s-pod-grep " 4 | notes="reports count of case-insensitive grep string per pod, over the last 5 minutes. Can optionally add additional param like '10m' or '24h' to change the --since= value." 5 | 6 | pod_names=$2 7 | grep_string=$3 8 | since="${4:-5m}" 9 | 10 | # reads "env" param, loads "namespace" param 11 | . "$(dirname "${BASH_SOURCE[0]}")/load-config.sh" 12 | 13 | for pod in $(kubectl --namespace=$namespace get pods | grep "$pod_names" | cut -f 1 -d ' '); do 14 | grep_count=$(kubectl --namespace=$namespace logs $pod --since=$since | grep -ai "$grep_string" | wc -l) 15 | echo "pod = $pod, grep count = $grep_count" 16 | done -------------------------------------------------------------------------------- /k8s-pods: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | usage="k8s-pods " 4 | notes="watch pods matching the grep string" 5 | 6 | pod_names=$2 7 | 8 | # reads "env" param, loads "namespace" param 9 | . "$(dirname "${BASH_SOURCE[0]}")/load-config.sh" 10 | 11 | watch "kubectl --namespace=$namespace get pods | grep $pod_names" 12 | -------------------------------------------------------------------------------- /k8s-port-forward: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | usage="k8s-port-forward " 4 | notes="this port-forwards the port mapping from the first pod matching the grep string" 5 | 6 | pod_names=$2 7 | ports=$3 8 | 9 | # reads "env" param, loads "namespace" param 10 | . "$(dirname "${BASH_SOURCE[0]}")/load-config.sh" 11 | 12 | first_matching_pod=$(kubectl --namespace=$namespace get pods | grep $pod_names | cut -f 1 -d ' ' | head -n 1) 13 | 14 | if [[ -z "$first_matching_pod" ]]; then 15 | echo "No pods found in env '$env' with namespace '$namespace' matching '$pod_names'" 16 | else 17 | echo "port-forwarding from pod $first_matching_pod" 18 | 19 | kubectl --namespace=$namespace port-forward $first_matching_pod $ports 20 | fi 21 | -------------------------------------------------------------------------------- /k8s-tail: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | usage="k8s-tail " 4 | notes='can optionally add additional params, such as "--since=5m"' 5 | 6 | pod_names=$2 7 | since=$3 8 | 9 | # reads "env" param, loads "namespace" param 10 | . "$(dirname "${BASH_SOURCE[0]}")/load-config.sh" 11 | 12 | kubectl --namespace=$namespace get pods | grep $pod_names | cut -f 1 -d ' ' | xargs -P 20 -I {} kubectl --namespace=$namespace logs -f {} $since 13 | -------------------------------------------------------------------------------- /k8s-top-nodes: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | usage="k8s-top-nodes " 4 | notes="calls 'kubectl top node' on all nodes in the cluster for the given env" 5 | 6 | # reads "env" param, loads "namespace" param 7 | . "$(dirname "${BASH_SOURCE[0]}")/load-config.sh" 8 | 9 | kubectl get nodes | grep 'node\|master' | awk '{print $1}' | xargs -I {} kubectl top node {} 10 | -------------------------------------------------------------------------------- /k8s-top-pods: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | usage="k8s-top-pods " 4 | notes="calls 'kubectl top pod' for the pods found that are in Running status" 5 | 6 | pod_names=$2 7 | 8 | # reads "env" param, loads "namespace" param 9 | . "$(dirname "${BASH_SOURCE[0]}")/load-config.sh" 10 | 11 | kubectl --namespace=$namespace get pods | grep $pod_names | grep Running | cut -f 1 -d ' ' | xargs -P 20 -I {} kubectl --namespace=$namespace top pod {} --containers 12 | -------------------------------------------------------------------------------- /load-config.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Central location to determine config files and k8s namespaces based on known params, so this isn't repeated among scripts. 4 | 5 | # Note that all scripts using this must have a "usage" var defined. 6 | 7 | 8 | # In an effort to not have real config in a Git repo, this script will first check for a script at ~/.kube/k8s-utils-config.sh. 9 | # If it exists it will delegate loading of the "cfg" and "namespace" vars to that script. 10 | # Remember that it needs to be executable: 11 | # chmod 755 ~/.kube/k8s-utils-config.sh 12 | 13 | config_script="${HOME}/.kube/k8s-utils-config.sh" 14 | 15 | env=$1 16 | 17 | # if params or config isn't right, remind user of required input 18 | fail() { 19 | echo -e "\nERROR: $1\n" 20 | echo "Usage: $usage" 21 | if [ -n "$notes" ]; then 22 | echo "Note: $notes" 23 | fi 24 | echo "Doc: https://github.com/davidkuster/k8s-utils#$(basename $0)" 25 | echo "" 26 | exit 1 27 | } 28 | 29 | # load "$cfg" & "$namespace" vars from config script 30 | if [ -f "$config_script" ]; then 31 | . "$config_script" "$env" 32 | else 33 | fail "$config_script does not exist, cannot run script" 34 | fi 35 | 36 | # verify env param 37 | if [ -z $cfg ] || [ -z $namespace ]; then 38 | echo "Configured env values are:" 39 | grep "^[^#;]" $config_script \ 40 | | grep '==' | \ 41 | cut -f 3 -d '=' \ 42 | | cut -f 1 -d ']' \ 43 | | sed 's/"//g' \ 44 | | sort 45 | fail "No configuration found for env: '$env'" 46 | fi 47 | 48 | # verify sufficient number of args have been passed to the script 49 | all_args_count=$(echo "$usage" | wc -w) 50 | if [ "$#" -lt $(($all_args_count - 1)) ]; then 51 | fail "Not enough arguments" 52 | fi 53 | 54 | # set the KUBECONFIG var here to DRY out the various scripts 55 | export KUBECONFIG=${HOME}/.kube/$cfg 56 | --------------------------------------------------------------------------------