├── LICENSE ├── README.md └── kubernetes_observer /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2021 Daniel Pilon 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Run Observer on a Kubernetes pod deployment 2 | 3 | This is an adaptation for Kubernetes of the amazing [dominicletz](https://github.com/dominicletz) script for SSH, [remote_observe](https://github.com/dominicletz/remote_observe). 4 | 5 | `kubernetes_observer` runs an Observer on a remote Kubernetes pod deployment, using `kubectl`. The script automates port discovery and forwarding. 6 | 7 | You only have to ensure that in your mix.exs you've added the runtime_tools in your deployment: 8 | 9 | ```elixir 10 | # Run "mix help compile.app" to learn about applications. 11 | def application do 12 | [ 13 | mod: {My.Application, []}, 14 | extra_applications: [:runtime_tools] 15 | ] 16 | end 17 | ``` 18 | 19 | ## SYNOPSIS 20 | 21 | ``` 22 | Usage: kubernetes_observer (-options) () 23 | 24 | target pod or type/name. 25 | is only required if there is more than one node 26 | running on the remote machine. 27 | 28 | Options: 29 | 30 | -e Start an erlang shell instead of an elixir shell 31 | -l Use long names instead of default short names 32 | -k Set the cookie value 33 | -h Looks for the cookie in /.erlang.cookie 34 | on the remote node 35 | -p Looks for the cookie in on the remote 36 | node 37 | -o Set the kubectl options 38 | -c Set the Kubernetes cluster namespace 39 | 40 | Example: 41 | kubernetes_observer -k secret my-k8s-pod 42 | kubernetes_observer -p /opt/app/releases/COOKIE my-k8s-deployment 43 | ``` 44 | 45 | ### Optional Environment Variables 46 | 47 | - `ERL_EPMD_PORT` - Define an alternative EPMD port 48 | 49 | ## Installation 50 | 51 | Mark the script `kubernetes_observer` as executable and move it somewhere in your \$PATH for convinience: 52 | 53 | ### Linux 54 | 55 | ``` 56 | wget https://raw.githubusercontent.com/danielpilon/kubernetes_observer/master/kubernetes_observer 57 | chmod +x ./kubernetes_observer 58 | sudo mv ./kubernetes_observer /usr/local/bin 59 | ``` 60 | 61 | ### MacOS (with curl) 62 | 63 | ``` 64 | curl https://raw.githubusercontent.com/danielpilon/kubernetes_observer/master/kubernetes_observer -o kubernetes_observer 65 | chmod +x ./kubernetes_observer 66 | sudo mv ./kubernetes_observer /usr/local/bin 67 | ``` 68 | 69 | ## Notes 70 | 71 | - [Releases with Runtime tools](#releases-with-runtime-tools) 72 | - [kubectl](#kubectl) 73 | - [Compatible VM](#compatible-vm) 74 | - [Access to the Cookie](#access-to-the-cookie) 75 | - [Short and long names](#short-and-long-names) 76 | - [EPMD Port](#epmd-port) 77 | 78 | 79 | ### Releases with Runtime tools 80 | In order to run the observer tool locally, your release needs to include the runtime tools. You can [read more about this here](https://tkowal.wordpress.com/2016/04/23/observer-in-erlangelixir-release/) and about [enabling distribution in releases here](https://elixirforum.com/t/remote-observer-connection-issues/26315/3) 81 | 82 | ### kubectl 83 | To interact with a Kubernetes cluster, `kubernetes_observer` uses [kubectl](https://kubernetes.io/docs/reference/kubectl/overview/). It's required to have it preinstalled and configured in order to use the script. 84 | 85 | Basically it uses the [port-foward](https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/) command to establish a connection between the local and the remote machine. 86 | 87 | `kubernetes_observer` accepts via the `-o` flag the same level of options as the `port-foward` command does. 88 | 89 | If multiple containers are running at the same pod, the name of the desired container can be specified via the `-c` option. 90 | 91 | ### Compatible VM 92 | 93 | You need to have a local installed Elixir VM that is compatible with the remote Elixir VM that you have 94 | 95 | ### Access to the Cookie 96 | 97 | To connect to a remote VM `kubernetes_observer` tries to determine the secret cookie. It assumes to be in the home directory of the pod. 98 | 99 | 1. If you are running a release the cookie can be in the releases/COOKIE subdirectory of your app. In that case provide the full cookie path using the -p option. 100 | 101 | 1. If the cookie is in a different home directory you can use the parameter -h to provide an alternative home directory. 102 | 103 | 1. If you know the cookie you can provide the cookie directly via the option -c 104 | 105 | ### Short and long names 106 | 107 | Currently the script is only using `-sname`. Even though there is an -l option this is currently not functional. **TODO** 108 | 109 | ### EPMD Port 110 | 111 | The default epmd port is 4369. Unfortunately the remote epmd port and locally mapped epmd port need to be identical. So in the default setup `kubernetes_observer` will kill any locally running epmd service before forwarding its port, so that the Elixir/Erlang remote shell is opened against the remote node and not locally. 112 | -------------------------------------------------------------------------------- /kubernetes_observer: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | function show_help { 3 | echo "Usage: kubernetes_observer (-options) ()" 4 | echo "" 5 | echo " target pod or type/name." 6 | echo " is only required if there is more than one node" 7 | echo " running on the remote machine." 8 | echo "" 9 | echo "Options:" 10 | echo "" 11 | echo " -e Start an erlang shell instead of an elixir shell" 12 | echo " -l Use long names instead of default short names" 13 | echo " -k Set the cookie value" 14 | echo " -h Looks for the cookie in /.erlang.cookie" 15 | echo " on the remote node" 16 | echo " -p Looks for the cookie in on the remote" 17 | echo " node" 18 | echo " -o Set the kubectl options" 19 | echo " -c Set the Kubernetes cluster namespace" 20 | echo "" 21 | echo "Example:" 22 | echo " kubernetes_observer -k secret my-k8s-pod" 23 | echo " kubernetes_observer -p /opt/app/releases/COOKIE my-k8s-deployment" 24 | echo "" 25 | exit 26 | } 27 | 28 | if [[ "$1" == "" ]]; then 29 | show_help 30 | fi 31 | 32 | OPTIND=1 33 | use_erlang=0 34 | name=sname 35 | 36 | while getopts "?elk:p:h:o:c:" opt; do 37 | case "$opt" in 38 | \?) 39 | show_help 40 | exit 0 41 | ;; 42 | e) 43 | use_erlang=1 44 | ;; 45 | l) 46 | name=name 47 | ;; 48 | k) 49 | COOKIE=$OPTARG 50 | ;; 51 | p) 52 | COOKIE_PATH=$OPTARG 53 | ;; 54 | h) 55 | COOKIE_HOME=$OPTARG 56 | ;; 57 | o) 58 | KUBECTL_OPTIONS=$OPTARG 59 | ;; 60 | c) 61 | CONTAINER_NAME="-c $OPTARG" 62 | ;; 63 | esac 64 | done 65 | 66 | shift $((OPTIND - 1)) 67 | 68 | target=$1 69 | remote_do="kubectl $KUBECTL_OPTIONS exec $CONTAINER_NAME $target --" 70 | 71 | echo "Starting integration with $target..." 72 | 73 | [ "${1:-}" = "--" ] && shift 74 | 75 | ## Piping EPMD 76 | if [[ "$ERL_EPMD_PORT" == "" ]]; then 77 | echo "Killing local epmd" 78 | killall epmd -w 2>/dev/null 79 | export ERL_EPMD_PORT="4369" 80 | fi 81 | echo "Epmd: $ERL_EPMD_PORT" 82 | 83 | echo "Connecting..." 84 | 85 | ## Tmp dir for socket and config 86 | tmp_dir=$(mktemp -d) 87 | echo $"Temp dir created at $tmp_dir" 88 | function cleanup { 89 | echo $1 90 | ps aux | grep -i 'port-forward $target' | grep -v grep | awk {'print $2'} | xargs kill 2>/dev/null 91 | rm $tmp_dir/erl_inetrc 2>/dev/null 92 | rm -rf $tmp_dir 93 | exit 94 | } 95 | 96 | function forward_port { 97 | output="$tmp_dir/$1" 98 | kubectl $KUBECTL_OPTIONS port-forward $target $1 &>$output & 99 | pid=$! 100 | echo "Forwarding $1 on PID $pid. Waiting for connection" 101 | until grep -q -i 'Forwarding from' $output; do 102 | if ! ps $pid >/dev/null; then 103 | cleanup "The connection could not be established" 104 | fi 105 | echo -n "." 106 | sleep 1 107 | done 108 | echo 109 | echo "Connection established!" 110 | } 111 | 112 | # Unfortunately we can't remap epmd to a different port or we get {error, address} 113 | # Check erl_epmd:do_get_names/1 implementation 114 | forward_port $ERL_EPMD_PORT 115 | 116 | epmd=$(epmd -names) || cleanup "Couldn't run epmd" 117 | 118 | # Ports 119 | ports=$(echo "$epmd" | tail -n +2 | cut -d' ' -f 5) 120 | 121 | # Finding the node name 122 | names=$(echo "$epmd" | tail -n +2 | cut -d' ' -f 2) 123 | # If there is more than one remote beam, we need to know which one to connect to 124 | len=$(echo "$names" | wc -w) 125 | if [[ "$len" == "0" ]]; then 126 | cleanup "Did not find any remote beams" 127 | elif [[ "$len" == "1" ]]; then 128 | node="$names" 129 | elif [[ "$2" != "" ]]; then 130 | node="$2" 131 | else 132 | echo "Identified multiple remote beams:" 133 | cleanup "$names" 134 | fi 135 | echo "Node: $node" 136 | 137 | host=$($remote_do hostname) 138 | echo "Host: $host" 139 | 140 | ## Detecting the secret distribution cookie 141 | if [[ "$COOKIE" == "" ]]; then 142 | if [[ "$COOKIE_PATH" == "" ]]; then 143 | if [[ "$COOKIE_HOME" == "" ]]; then 144 | COOKIE_HOME="~" 145 | fi 146 | COOKIE_PATH="$COOKIE_HOME/.erlang.cookie" 147 | fi 148 | echo "Trying to load erlang cookie from $1:$COOKIE_PATH" 149 | COOKIE=$($remote_do "cat $COOKIE_PATH") 150 | fi 151 | echo "Cookie: ${COOKIE:0:3}*****************" 152 | 153 | if [[ "$COOKIE" == "" ]]; then 154 | cleanup "Could not determine the cookie" 155 | fi 156 | 157 | ## Creating Port mappings for each remote beam 158 | for port in $ports; do 159 | echo "Port: $port" 160 | forward_port $port 161 | done 162 | 163 | ## Overriding hostname lookup to convince local beam that it's running remote 164 | export ERL_INETRC=$tmp_dir/erl_inetrc 165 | echo "{host, {127,0,0,1}, [\"$host\", \"$node@$host\"]}." >$ERL_INETRC 166 | echo "{edns,0}." >>$ERL_INETRC 167 | echo "{lookup, [file, dns]}." >>$ERL_INETRC 168 | 169 | ## Starting up beam 170 | if [[ $use_erlang == 1 ]]; then 171 | erl -remsh "$node@$host" -hidden -setcookie "$COOKIE" "-$name" "remsh_$$@$host" -eval "observer:start()." 172 | else 173 | iex --hidden --cookie "$COOKIE" "--$name" "remsh_$$@$host" --remsh "$node@$host" --eval ":observer.start()" 174 | fi 175 | 176 | cleanup "Done" 177 | --------------------------------------------------------------------------------