├── Dockerfile ├── LICENSE ├── README.md ├── docker-bench-custom-monitor ├── Dockerfile ├── README.md ├── monit.sh └── stats.sh ├── docker-bench-iperf3 ├── Dockerfile └── README.md ├── docker-entrypoint.sh ├── docker-knb ├── Dockerfile ├── README.md ├── entrypoint.sh ├── monit.sh └── stats.sh ├── knb └── plotly-templates ├── bandwidth.jq ├── cpu-usage.jq └── ram-usage.jq /Dockerfile: -------------------------------------------------------------------------------- 1 | FROM bitnami/kubectl:1.19 2 | LABEL mantainer "Oleg Basov " 3 | USER 0 4 | 5 | RUN apt-get update &&\ 6 | DEBIAN_FRONTEND=noninteractive apt-get install -y jq ncurses-bin && \ 7 | rm -r /var/lib/apt/lists /var/cache/apt/archives 8 | 9 | COPY knb / 10 | COPY docker-entrypoint.sh / 11 | COPY plotly-templates /plotly-templates 12 | 13 | ENTRYPOINT ["/docker-entrypoint.sh"] 14 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2020 infraBuilder 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # k8s-bench-suite 2 | Bash scripts collection to benchmark kubernetes cluster performance 3 | 4 | ## [knb](knb) : Kubernetes Network Benchmark 5 | 6 | [knb](knb) is a bash script that will start a networking benchmark on a target Kubernetes cluster. 7 | 8 | Here are some highlights: 9 | 10 | - **Plain bash script** with very few dependencies 11 | - Complete benchmark **takes only 2 minutes** 12 | - Ability to select only a subset of benchmark tests to run 13 | - Testing **both TCP and UDP** bandwidth 14 | - **Automatic detection of CNI MTU** 15 | - **Includes host cpu and ram monitoring** in benchmark report 16 | - Ability to create static graph images based on the result data using plotly/orca (see examples below) 17 | - No ssh access required, just an access to the target cluster through **standard kubectl** 18 | - **No need for high privileges**, the script will just launch very lightweight pods on two nodes. 19 | - Based on **very lights containers** images : 20 | - ![Docker Image Size (tag)](https://img.shields.io/docker/image-size/infrabuilder/bench-iperf3/latest) [infrabuilder/bench-iperf3](https://hub.docker.com/r/infrabuilder/bench-iperf3), is used to run benchmark tests 21 | - ![Docker Image Size (tag)](https://img.shields.io/docker/image-size/infrabuilder/bench-custom-monitor/latest) [infrabuilder/bench-custom-monitor](https://hub.docker.com/r/infrabuilder/bench-custom-monitor), is used to monitor nodes 22 | - **Ability to run the whole suite in a container** [olegeech/k8s-bench-suite](https://hub.docker.com/r/olegeech/k8s-bench-suite): 23 | - Image is based on the bitnami/kubectl 24 | - Nodes for testing can be auto-preselected 25 | 26 | ### Requirements 27 | 28 | This script needs a valid `kubectl` setup with an access to the target cluster. 29 | 30 | Binaries dependencies for the host that will execute [knb](knb) : 31 | 32 | - awk 33 | - grep 34 | - tail 35 | - date 36 | - kubectl 37 | - jq (for plotting) 38 | 39 | ### Quickstart 40 | 41 | Choose two nodes to act as server/client on your cluster (for example node1 and node2) . Then start the knb : 42 | 43 | ```bash 44 | ./knb --verbose --client-node node1 --server-node node2 45 | ``` 46 | 47 | If you omit the `--verbose` flag, it will also complete, but you will have no output until the end of the benchmark. 48 | 49 | ### Docker quickstart 50 | 51 | **Environment variables:** 52 | 53 | - `NODE_AUTOSELECT` Auto-selects a few nodes from cluster for running tests 54 | - `MASTER_ELIGIBLE` Master nodes can also be chosen 55 | 56 | You need to mount a valid kubeconfig inside the container and provide all other required flags to knb: 57 | 58 | ```bash 59 | docker run -e NODE_AUTOSELECT=1 -it --hostname knb --name knb --rm -v /home/user/my-graphs:/my-graphs -v /path/to/my/kubeconfig:/.kube/config olegeech/k8s-bench-suite --verbose --plot --plot-dir /my-graphs 60 | ``` 61 | 62 | ### Examples 63 | 64 | - Simple benchmark from "node1" to "node2" in **verbose** mode : 65 | 66 | ```bash 67 | knb -v -cn node1 -sn node2 68 | ``` 69 | 70 | - Benchmark from "nA" to "nB" and **save data** in file `mybench.knbdata` 71 | 72 | ```bash 73 | knb -cn nA -sn nB -o data -f mybench.knbdata 74 | ``` 75 | 76 | - Generate report **in json** from **previous benchmark** data file `mybench.knbdata` 77 | 78 | ```bash 79 | knb -fd mybench.knbdata -o json 80 | ``` 81 | 82 | - Plot graphs from **previous benchmark** data file `mybench.knbdata` 83 | 84 | ```bash 85 | knb -fd mybench.knbdata --plot --plot-args '--width 900 --height 600' 86 | ``` 87 | 88 | - To run benchmark from node A to node B, showing only result **in yaml** format : 89 | 90 | ```bash 91 | knb -cn A -sn B -o yaml 92 | ``` 93 | 94 | - To run benchmark from node Asterix to node Obelix, with the **most verbose output** and a result as **json** in a `res.json` file : 95 | 96 | ```bash 97 | knb --debug -cn Asterix -sn Obelix -o json -f res.json 98 | ``` 99 | 100 | - Running benchmark **in namespace** `myns` : 101 | 102 | ```bash 103 | knb -n myns -cn node1 -sn node2 104 | ``` 105 | 106 | - Run **only idle and tcp** benchmark : 107 | 108 | ```bash 109 | knb -cn clientnode -sn servernode -ot idle,tcp 110 | ``` 111 | 112 | ### Usage 113 | 114 | To display usage, use the `-h` flag : 115 | 116 | ```bash 117 | aducastel@infrabuilder:~/k8s-bench-suite$ ./knb -h 118 | 119 | knb is a network benchmark tool for Kubernetes CNI 120 | 121 | There are two modes : 122 | - benchmark mode : will actually run benchmark on a cluster 123 | - from data mode : read data generated by previous benchmark with "-o data" flag 124 | 125 | =====[ Benchmark mode ]==================================================== 126 | 127 | Mandatory flags : 128 | 129 | -cn 130 | --client-node : Define kubernetes node name that will host the client part 131 | 132 | -sn 133 | --server-node : Define kubernetes node name that will host the server part 134 | 135 | Optionnal flags : 136 | -d 137 | --duration : Set the benchmark duration for each test in seconds (Default 10) 138 | 139 | -k 140 | --keep : Keep data directory instead of cleaning it (tmp dir that contains raw benchmark data) 141 | 142 | -n 143 | --namespace : Set the target kubernetes namespace 144 | 145 | --name : Set the name of this benchmark run 146 | 147 | -ot 148 | --only-tests : Only run a subset of benchmark tests, comma separated (Ex: -ot tcp,idle) 149 | Possible values: all, tcp, udp, p2p, p2s , p2ptcp, p2pudp, p2stcp, p2sudp, idle 150 | 151 | -sbs 152 | --socket-buffer-size : Set the UDP socket buffer size with unit, or 'auto'. ex: '256K' (Default: auto) 153 | 154 | -t 155 | --timeout : Set the pod ready wait timeout in seconds (Default 30) 156 | 157 | =====[ From Data mode ]==================================================== 158 | 159 | Mandatory flags : 160 | -fd 161 | --from-data : Define the path to the data to read from 162 | Data file must be rendered with '--output data' 163 | 164 | =====[ Common optionnal flags ]============================================ 165 | 166 | --debug : Set the debug level to "debug" 167 | 168 | -dl 169 | --debug-level : Set the debug level 170 | Possible values: standard, warn, info, debug 171 | 172 | -f 173 | --file : Set the output file 174 | 175 | -h 176 | --help : Display this help message 177 | 178 | -p 179 | --plot : Plot data using plotly/orca 180 | 181 | -pd 182 | --plot-dir : Directory where to save graphs 183 | Defaults to the current directory 184 | 185 | -pa 186 | --plot-args : Arguments to the plotly's 'orca graph' function 187 | Defaults to '--width 900 --height 500' 188 | 189 | -o 190 | --output : Set the output format. Defaults to 'text' 191 | Possible values: text, yaml, json, data 192 | -v 193 | --verbose : Activate the verbose mode by setting debug-level to 'info' 194 | 195 | -V 196 | --version : Show current script version 197 | 198 | =====[ Examples ]========================================================== 199 | 200 | Simple benchmark from "node1" to "node2" in verbose mode 201 | ------------------------------------------------------------------------- 202 | | knb -v -cn node1 -sn node2 | 203 | ------------------------------------------------------------------------- 204 | 205 | Benchmark from "nA" to "nB" with data saved in file "mybench.knbdata" 206 | ------------------------------------------------------------------------- 207 | | knb -cn nA -sn nB -o data -f mybench.knbdata | 208 | ------------------------------------------------------------------------- 209 | 210 | Generate report in json from previous benchmark file "mybench.knbdata" 211 | ------------------------------------------------------------------------- 212 | | knb -fd mybench.knbdata -o json | 213 | ------------------------------------------------------------------------- 214 | 215 | Create graph images from previous benchmark file "mybench.knbdata" 216 | ------------------------------------------------------------------------- 217 | | knb -fd mybench.knbdata --plot --plot-dir ./my-graphs | 218 | ------------------------------------------------------------------------- 219 | 220 | Run only idle and tcp benchmark : 221 | ------------------------------------------------------------------------- 222 | | knb -cn clientnode -sn servernode -ot idle,tcp | 223 | ------------------------------------------------------------------------- 224 | 225 | ``` 226 | ### Graph examples 227 | 228 | ![bandwidth](https://user-images.githubusercontent.com/21361354/102022246-d6e1d080-3d85-11eb-8ca6-37064ac3918f.png) 229 | ![cpu-usage](https://user-images.githubusercontent.com/21361354/102022247-d812fd80-3d85-11eb-820f-f5108cf8b930.png) 230 | ![ram-usage](https://user-images.githubusercontent.com/21361354/102022250-d812fd80-3d85-11eb-9f1b-650571bb0054.png) 231 | 232 | -------------------------------------------------------------------------------- /docker-bench-custom-monitor/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM alpine:3 2 | RUN apk add --update sysstat && rm -rf /var/cache/apk/* && mkdir /data 3 | COPY *.sh / 4 | CMD ["sh","/monit.sh"] 5 | -------------------------------------------------------------------------------- /docker-bench-custom-monitor/README.md: -------------------------------------------------------------------------------- 1 | Custom benchmark monitor for https://github.com/InfraBuilder/k8s-bench-suite 2 | 3 | ![Docker Image Size (tag)](https://img.shields.io/docker/image-size/infrabuilder/bench-custom-monitor/latest) [Dockerfile](https://github.com/InfraBuilder/k8s-bench-suite/blob/master/docker-bench-custom-monitor/Dockerfile) 4 | 5 | ## How does it work 6 | 7 | Default CMD starts the /[monit.sh](monit.sh) script that will monitor the host using tools like **sar** and **free** , outputing a new line in `/data/metrics.log` each second with following information : 8 | 9 | ```sh 10 | %cpu-user %cpu-nice %cpu-system %cpu-iowait %cpu-steal memory-used-MB timestamp 11 | ``` 12 | 13 | Data produced per second : **40 bytes/s** 14 | 15 | ## Get metrics 16 | 17 | You can query a subset of data between two timestamp with the /[stats.sh](stats.sh) script : 18 | 19 | ```sh 20 | sh /script.sh 1598047515 1598047530 21 | ``` 22 | -------------------------------------------------------------------------------- /docker-bench-custom-monitor/monit.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # Free -m output sample : 3 | # total used free shared buff/cache available 4 | # Mem: 64316 6544 35949 143 21821 58631 5 | # Swap: 0 0 0 6 | # 7 | # Sar output sample: 8 | # Linux 4.15.0-101-generic (tokyo) 06/03/2020 _x86_64_ (16 CPU) 9 | # 10 | # 04:09:29 PM CPU %user %nice %system %iowait %steal %idle 11 | # 04:09:30 PM all 24.45 0.00 4.33 0.00 0.00 71.22 12 | # Average: all 24.45 0.00 4.33 0.00 0.00 71.22 13 | # 14 | while true 15 | do 16 | # Output will be like one line per second : 17 | # %cpu-user %cpu-nice %cpu-system %cpu-iowait %cpu-steal memory-used-MB timestamp 18 | echo "$(sar 1 1 | grep Average|awk '{print $3" "$4" "$5" "$6" "$7}') $(free -m|grep Mem:|awk '{print $3}') $(date "+%s")" >> /data/metrics.log 19 | done 20 | -------------------------------------------------------------------------------- /docker-bench-custom-monitor/stats.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | # Usage: $0 start-time stop-time 4 | START=$1 5 | STOP=$2 6 | 7 | METRICSFILE="/data/metrics.log" 8 | 9 | # Selecting the timestamp range we need 10 | echo "cpu-user cpu-nice cpu-system cpu-iowait cpu-steal memory-used timestamp" 11 | awk '$7 >= '$START' && $7 <= '$STOP "$METRICSFILE" 12 | 13 | 14 | # awk '$7 >= '$START' && $7 <= '$STOP "$METRICSFILE" \ 15 | # | awk ' 16 | # { 17 | # M+=$6;U+=$1;N+=$2;S+=$3;I+=$4;T+=$5; 18 | # print $0 19 | # } 20 | # END { 21 | # print "============================================================================" 22 | # print "cpu-user cpu-nice cpu-system cpu-iowait cpu-steal memory-used records-number"; 23 | # printf "%.2f %.2f %.2f %.2f %.2f %d %d\n",U/NR,N/NR,S/NR,I/NR,T/NR,M/NR,NR 24 | # } 25 | # ' 26 | -------------------------------------------------------------------------------- /docker-bench-iperf3/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM alpine:3 2 | RUN apk add --update iperf3 && rm -rf /var/cache/apk/* 3 | -------------------------------------------------------------------------------- /docker-bench-iperf3/README.md: -------------------------------------------------------------------------------- 1 | Simple container image based on **Alpine 3** with package **iperf3** 2 | 3 | ![Docker Image Size (tag)](https://img.shields.io/docker/image-size/infrabuilder/bench-iperf3/latest) [Dockerfile](https://github.com/InfraBuilder/k8s-bench-suite/blob/master/docker-bench-iperf3/Dockerfile) 4 | 5 | -------------------------------------------------------------------------------- /docker-entrypoint.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # NODE_AUTOSELECT=1 pre-selects a few worker nodes and 4 | # provides them to knb as --client/server-node arguments 5 | # 6 | # MASTER_ELIGIBLE=1 choose from manager nodes as well 7 | # 8 | 9 | node_args= 10 | master_eligible="!" 11 | 12 | if [[ -n $NODE_AUTOSELECT ]] 13 | then 14 | if [[ -n $MASTER_ELIGIBLE ]] 15 | then 16 | master_eligible= 17 | fi 18 | worker_nodes=$(kubectl get nodes -l "$master_eligible node-role.kubernetes.io/master" -o name|awk -F '/' '{print $2}') 19 | if [[ -z $worker_nodes ]] 20 | then 21 | echo "No available nodes found for scheduling" 22 | exit 1 23 | fi 24 | test_nodes=($worker_nodes) 25 | node_args="--client-node ${test_nodes[0]} --server-node ${test_nodes[1]}" 26 | echo "Pre-selected nodes for tests: $node_args" 27 | fi 28 | 29 | /knb "$@" $node_args 30 | -------------------------------------------------------------------------------- /docker-knb/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM alpine:3 2 | RUN apk add --no-cache iperf iperf3 nuttcp sysstat && mkdir /data 3 | COPY *.sh / 4 | ENTRYPOINT ["/entrypoint.sh"] 5 | -------------------------------------------------------------------------------- /docker-knb/README.md: -------------------------------------------------------------------------------- 1 | Single container image for benchmark and monitoring for [knb](https://github.com/InfraBuilder/k8s-bench-suite/) based on **Alpine 3** 2 | 3 | ![Docker Image Size (tag)](https://img.shields.io/docker/image-size/infrabuilder/knb/latest) [Dockerfile](https://github.com/InfraBuilder/k8s-bench-suite/blob/master/docker-knb/Dockerfile) 4 | 5 | -------------------------------------------------------------------------------- /docker-knb/entrypoint.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | case $1 in 4 | monitor) exec /monit.sh ;; 5 | *) exec $@;; 6 | esac 7 | -------------------------------------------------------------------------------- /docker-knb/monit.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # Free -m output sample : 3 | # total used free shared buff/cache available 4 | # Mem: 64316 6544 35949 143 21821 58631 5 | # Swap: 0 0 0 6 | # 7 | # Sar output sample: 8 | # Linux 4.15.0-101-generic (tokyo) 06/03/2020 _x86_64_ (16 CPU) 9 | # 10 | # 04:09:29 PM CPU %user %nice %system %iowait %steal %idle 11 | # 04:09:30 PM all 24.45 0.00 4.33 0.00 0.00 71.22 12 | # Average: all 24.45 0.00 4.33 0.00 0.00 71.22 13 | # 14 | while true 15 | do 16 | # Output will be like one line per second : 17 | # %cpu-user %cpu-nice %cpu-system %cpu-iowait %cpu-steal memory-used-MB timestamp 18 | echo "$(sar 1 1 | grep Average|awk '{print $3" "$4" "$5" "$6" "$7}') $(free -m|grep Mem:|awk '{print $3}') $(date "+%s")" >> /data/metrics.log 19 | done 20 | -------------------------------------------------------------------------------- /docker-knb/stats.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | # Usage: $0 start-time stop-time 4 | START=$1 5 | STOP=$2 6 | 7 | METRICSFILE="/data/metrics.log" 8 | 9 | # Selecting the timestamp range we need 10 | echo "cpu-user cpu-nice cpu-system cpu-iowait cpu-steal memory-used timestamp" 11 | awk '$7 >= '$START' && $7 <= '$STOP "$METRICSFILE" 12 | 13 | 14 | # awk '$7 >= '$START' && $7 <= '$STOP "$METRICSFILE" \ 15 | # | awk ' 16 | # { 17 | # M+=$6;U+=$1;N+=$2;S+=$3;I+=$4;T+=$5; 18 | # print $0 19 | # } 20 | # END { 21 | # print "============================================================================" 22 | # print "cpu-user cpu-nice cpu-system cpu-iowait cpu-steal memory-used records-number"; 23 | # printf "%.2f %.2f %.2f %.2f %.2f %d %d\n",U/NR,N/NR,S/NR,I/NR,T/NR,M/NR,NR 24 | # } 25 | # ' 26 | -------------------------------------------------------------------------------- /knb: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | ################################################################### 3 | # @Description : Kubernetes Network Benchmark 4 | # @Author : Alexis Ducastel 5 | # @License : MIT 6 | ################################################################### 7 | 8 | #============================================================================== 9 | # Utility functions 10 | #============================================================================== 11 | 12 | [ "$(tput colors)" -gt 0 ] && COLOR="true" || COLOR="false" 13 | function color { 14 | $COLOR || return 0 15 | color="0" 16 | case $1 in 17 | normal) color="0" ;; 18 | rst) color="0" ;; 19 | red) color="31" ;; 20 | green) color="32" ;; 21 | yellow) color="33" ;; 22 | blue) color="34" ;; 23 | magenta) color="35" ;; 24 | cyan) color="36" ;; 25 | lightred) color="91" ;; 26 | lightgreen) color="92" ;; 27 | lightyellow) color="93" ;; 28 | lightblue) color="94" ;; 29 | lightmagenta) color="95" ;; 30 | lightcyan) color="96" ;; 31 | white) color="97" ;; 32 | *) color="0" ;; 33 | esac 34 | echo -e "\033[0;${color}m" 35 | } 36 | function logdate { date "+%Y-%m-%d %H:%M:%S"; } 37 | function fatal { echo "$(logdate) $(color red)[FATAL]$(color normal) $@" >&2; cleanandexit 1; } 38 | function err { echo "$(logdate) $(color lightred)[ERROR]$(color normal) $@" >&2; } 39 | function warn { [$DEBUG_LEVEL -ge 1 ] && echo "$(logdate) $(color yellow)[WARNING]$(color normal) $@" >&2; } 40 | function info { [ $DEBUG_LEVEL -ge 2 ] && echo "$(logdate) $(color cyan)[INFO]$(color normal) $@" >&2; } 41 | function debug { [ $DEBUG_LEVEL -ge 3 ] && echo "$(logdate) $(color lightcyan)[DEBUG]$(color normal) $@" >&2; } 42 | 43 | function now { date +%s; } 44 | function cleanandexit { 45 | # Cleaning Kubernetes resources 46 | for i in $RESOURCE_TO_CLEAN_BEFORE_EXIT 47 | do 48 | $DEBUG && debug "Deleting resource $i" 49 | kubectl delete $NAMESPACEOPT $i --wait=false >/dev/null 50 | done 51 | 52 | # Cleaning temp directory 53 | if [ "$CLEANDIR" = "true" ] 54 | then 55 | [ ! -z "$DATADIR" ] && [ -d $DATADIR ] && touch $DATADIR/something && rm $DATADIR/* && rmdir $DATADIR 56 | else 57 | info "Keeping data directory, you will have to clean it manually." 58 | info "$DATADIR" 59 | fi 60 | 61 | exit $1 62 | } 63 | 64 | # Example : waitpod podname Running 60 65 | function waitpod { 66 | POD=$1 67 | PHASE=$2 68 | TIMEOUT=$3 69 | TMAX=$(( $(now) + $TIMEOUT )) 70 | $DEBUG && debug "Waiting for pod $POD to be $PHASE until $TMAX" 71 | while [ "$(now)" -lt "$TMAX" ] 72 | do 73 | CURRENTPHASE=$(kubectl get --request-timeout 2s $NAMESPACEOPT pod $POD -o jsonpath={.status.phase}) 74 | $DEBUG && debug "[$(now)/$TMAX] Pod $POD is in phase $CURRENTPHASE, waiting for $PHASE" 75 | [ "$CURRENTPHASE" = "$PHASE" ] && return 0 76 | sleep 1 77 | done 78 | return 1 79 | } 80 | 81 | #============================================================================== 82 | # Default config 83 | #============================================================================== 84 | 85 | VERSION=1.5.0 86 | GENERATOR="knb" 87 | BENCHMARK_DATE=$(date -u "+%Y-%m-%d %H:%M:%S") 88 | DEBUG=false 89 | DEBUG_LEVEL=0 90 | SERVER_NODE="" 91 | CLIENT_NODE="" 92 | EXECID="$$" 93 | BENCHMARK_RUN_NAME="knb-$EXECID" 94 | NAMESPACEOPT="" 95 | POD_WAIT_TIMEOUT="30" 96 | BENCHMARK_DURATION="10" 97 | SOCKET_BUFFER_SIZE="auto" 98 | RESOURCE_TO_CLEAN_BEFORE_EXIT="" 99 | OUTPUT_FORMAT="text" 100 | FILENAME="/dev/stdout" 101 | PLOT_DIR="$(pwd)" 102 | PLOT_DATA=false 103 | PLOT_POD_NAME="knb-plotly-orca-$EXECID" 104 | PLOT_GRAPH_ARGS="--width 900 --height 500" 105 | CLEANDIR="true" 106 | FROM_DATA="false" 107 | 108 | RUN_TEST_IDLE="true" 109 | RUN_TEST_P2P_TCP="true" 110 | RUN_TEST_P2P_UDP="true" 111 | RUN_TEST_P2S_TCP="true" 112 | RUN_TEST_P2S_UDP="true" 113 | 114 | export LC_ALL=C # Trick to no depend on locale 115 | BIN_AWK="awk" 116 | 117 | #============================================================================== 118 | # Core functions 119 | #============================================================================== 120 | 121 | function detect-pod-failure-reason-and-fatal { 122 | POD_NAME=$1 123 | shift 124 | 125 | CURRENTPHASE=$(kubectl get --request-timeout 2s $NAMESPACEOPT pod $POD_NAME -o jsonpath={.status.phase}) 126 | $DEBUG && debug "Trying to detect pod failure for pod '$POD_NAME' in phase '$CURRENTPHASE'" 127 | case $CURRENTPHASE in 128 | "Pending") 129 | err "Pod $POD_NAME is still in Pending state, please check node name" 130 | ;; 131 | "Failed") 132 | # Is it a window size problem in iperf ? 133 | kubectl logs $NAMESPACEOPT $POD_NAME \ 134 | | grep "iperf3: error - unable to write to stream socket: Message too large" > /dev/null \ 135 | && err "Iperf message was too large, you should try a lower value with flag '-sbs' (current value is : $SOCKET_BUFFER_SIZE)" && return 0 136 | 137 | ;; 138 | *) 139 | err "Unknown pod phase '$CURRENTPHASE' for pod '$POD_NAME'" 140 | ;; 141 | esac 142 | 143 | fatal $@ 144 | } 145 | 146 | function usage { 147 | cat <<-EOF 148 | 149 | knb is a network benchmark tool for Kubernetes CNI 150 | 151 | There are two modes : 152 | - benchmark mode : will actually run benchmark on a cluster 153 | - from data mode : read data generated by previous benchmark with "-o data" flag 154 | 155 | =====[ Benchmark mode ]==================================================== 156 | 157 | Mandatory flags : 158 | 159 | -cn 160 | --client-node : Define kubernetes node name that will host the client part 161 | 162 | -sn 163 | --server-node : Define kubernetes node name that will host the server part 164 | 165 | Optionnal flags : 166 | -d 167 | --duration : Set the benchmark duration for each test in seconds (Default ${BENCHMARK_DURATION}) 168 | 169 | -k 170 | --keep : Keep data directory instead of cleaning it (tmp dir that contains raw benchmark data) 171 | 172 | -n 173 | --namespace : Set the target kubernetes namespace 174 | 175 | --name : Set the name of this benchmark run 176 | 177 | -ot 178 | --only-tests : Only run a subset of benchmark tests, comma separated (Ex: -ot tcp,idle) 179 | Possible values: all, tcp, udp, p2p, p2s , p2ptcp, p2pudp, p2stcp, p2sudp, idle 180 | 181 | -sbs 182 | --socket-buffer-size : Set the UDP socket buffer size with unit, or 'auto'. ex: '256K' (Default: $SOCKET_BUFFER_SIZE) 183 | 184 | -t 185 | --timeout : Set the pod ready wait timeout in seconds (Default ${POD_WAIT_TIMEOUT}) 186 | 187 | =====[ From Data mode ]==================================================== 188 | 189 | Mandatory flags : 190 | -fd 191 | --from-data : Define the path to the data to read from 192 | Data file must be rendered with '--output data' 193 | 194 | =====[ Common optionnal flags ]============================================ 195 | 196 | --debug : Set the debug level to "debug" 197 | 198 | -dl 199 | --debug-level : Set the debug level 200 | Possible values: standard, warn, info, debug 201 | 202 | -f 203 | --file : Set the output file 204 | 205 | -h 206 | --help : Display this help message 207 | 208 | -p 209 | --plot : Plot data using plotly/orca 210 | 211 | -pd 212 | --plot-dir : Directory where to save graphs 213 | Defaults to the current directory 214 | 215 | -pa 216 | --plot-args : Arguments to the plotly's 'orca graph' function 217 | Defaults to '--width 900 --height 500' 218 | 219 | -o 220 | --output : Set the output format. Defaults to 'text' 221 | Possible values: text, yaml, json, data, ibdbench 222 | -v 223 | --verbose : Activate the verbose mode by setting debug-level to 'info' 224 | 225 | -V 226 | --version : Show current script version 227 | 228 | =====[ Examples ]========================================================== 229 | 230 | Simple benchmark from "node1" to "node2" in verbose mode 231 | ------------------------------------------------------------------------- 232 | | knb -v -cn node1 -sn node2 | 233 | ------------------------------------------------------------------------- 234 | 235 | Benchmark from "nA" to "nB" with data saved in file "mybench.knbdata" 236 | ------------------------------------------------------------------------- 237 | | knb -cn nA -sn nB -o data -f mybench.knbdata | 238 | ------------------------------------------------------------------------- 239 | 240 | Generate report in json from previous benchmark file "mybench.knbdata" 241 | ------------------------------------------------------------------------- 242 | | knb -fd mybench.knbdata -o json | 243 | ------------------------------------------------------------------------- 244 | 245 | Create graph images from previous benchmark file "mybench.knbdata" 246 | ------------------------------------------------------------------------- 247 | | knb -fd mybench.knbdata --plot --plot-dir ./my-graphs | 248 | ------------------------------------------------------------------------- 249 | 250 | Run only idle and tcp benchmark : 251 | ------------------------------------------------------------------------- 252 | | knb -cn clientnode -sn servernode -ot idle,tcp | 253 | ------------------------------------------------------------------------- 254 | 255 | EOF 256 | } 257 | 258 | function run-client { 259 | POD_NAME=$1 260 | TARGET=$2 261 | 262 | SOCKET_BUFFER_FLAG="" 263 | if [ "$SOCKET_BUFFER_SIZE" = "auto" ] 264 | then 265 | SOCKET_BUFFER_FLAG="" 266 | else 267 | SOCKET_BUFFER_FLAG="-w $SOCKET_BUFFER_SIZE" 268 | fi 269 | 270 | CMD="" 271 | case $3 in 272 | idle) CMD="sleep $BENCHMARK_DURATION; echo 0 0 0 0 0 0 0 receiver" ;; 273 | udp) CMD="iperf3 -u -b 0 -c $TARGET -O 1 $SOCKET_BUFFER_FLAG -f m -t $BENCHMARK_DURATION" ;; 274 | tcp) CMD="iperf3 -c $TARGET -O 1 -f m -t $BENCHMARK_DURATION" ;; 275 | *) fatal "Unknown benchmark type '$2'" ;; 276 | esac 277 | 278 | info "Starting pod $POD_NAME on node $CLIENT_NODE" 279 | cat <<-EOF | kubectl apply $NAMESPACEOPT -f - >/dev/null|| fatal "Cannot create pod $POD_NAME" 280 | apiVersion: v1 281 | kind: Pod 282 | metadata: 283 | labels: 284 | app: $POD_NAME 285 | name: $POD_NAME 286 | spec: 287 | containers: 288 | - name: iperf 289 | image: infrabuilder/bench-iperf3 290 | args: 291 | - /bin/sh 292 | - -c 293 | - $CMD 294 | nodeSelector: 295 | kubernetes.io/hostname: $CLIENT_NODE 296 | restartPolicy: Never 297 | EOF 298 | 299 | RESOURCE_TO_CLEAN_BEFORE_EXIT="$RESOURCE_TO_CLEAN_BEFORE_EXIT pod/$POD_NAME" 300 | 301 | # Waiting for Pod to be Running 302 | kubectl wait $NAMESPACEOPT --for=condition=Ready pod/$POD_NAME \ 303 | --timeout=${POD_WAIT_TIMEOUT}s >/dev/null 2>/dev/null \ 304 | || detect-pod-failure-reason-and-fatal $POD_NAME "Failed to start pod $POD_NAME until timeout" 305 | 306 | STARTTIME=$(now) 307 | ENDTIME=$(( $STARTTIME + $BENCHMARK_DURATION )) 308 | 309 | # Waiting test duration 310 | $DEBUG && debug "sleeping during the benchmark duration ($BENCHMARK_DURATION) to avoid useless api calls" 311 | sleep $BENCHMARK_DURATION 312 | 313 | info "Waiting for pod $POD_NAME to be completed" 314 | # Waiting for test to be succeeded 315 | waitpod $POD_NAME Succeeded $POD_WAIT_TIMEOUT \ 316 | || fatal "Failed to run pod $POD_NAME until timeout" 317 | 318 | # Extracting data 319 | $DEBUG && debug "Writing $POD_NAME logs to $DATADIR/$POD_NAME.log" 320 | kubectl logs $NAMESPACEOPT $POD_NAME > $DATADIR/$POD_NAME.log 321 | grep receiver $DATADIR/$POD_NAME.log | $BIN_AWK '{print $7}' > $DATADIR/$POD_NAME.bw 322 | $DEBUG && debug "$POD_NAME Bandwidth = $(cat $DATADIR/$POD_NAME.bw)" 323 | 324 | # Extracting monitoring 325 | $DEBUG && debug "Writing $POD_NAME server metrics to $DATADIR/$POD_NAME-server.metrics" 326 | kubectl exec -i $NAMESPACEOPT $MONITOR_SERVER_POD_NAME \ 327 | -- sh /stats.sh $STARTTIME $ENDTIME > $DATADIR/$POD_NAME-server.metrics 328 | 329 | $DEBUG && debug "Computing $POD_NAME server metrics average to $DATADIR/$POD_NAME-server.avg" 330 | 331 | tail -n +2 $DATADIR/$POD_NAME-server.metrics | LC_ALL=C $BIN_AWK '{M+=$6;U+=$1;N+=$2;S+=$3;I+=$4;T+=$5;} 332 | END { 333 | print "cpu-user cpu-nice cpu-system cpu-iowait cpu-steal memory-used records-number"; 334 | printf "%.2f %.2f %.2f %.2f %.2f %d %d\n",U/NR,N/NR,S/NR,I/NR,T/NR,M/NR,NR 335 | }' > $DATADIR/$POD_NAME-server.avg 336 | 337 | $DEBUG && debug "Writing $POD_NAME client metrics to $DATADIR/$POD_NAME-client.metrics" 338 | kubectl exec -i $NAMESPACEOPT $MONITOR_CLIENT_POD_NAME \ 339 | -- sh /stats.sh $STARTTIME $ENDTIME > $DATADIR/$POD_NAME-client.metrics 340 | 341 | $DEBUG && debug "Computing $POD_NAME client metrics average to $DATADIR/$POD_NAME-client.avg" 342 | tail -n +2 $DATADIR/$POD_NAME-client.metrics | $BIN_AWK '{M+=$6;U+=$1;N+=$2;S+=$3;I+=$4;T+=$5;} 343 | END { 344 | print "cpu-user cpu-nice cpu-system cpu-iowait cpu-steal memory-used records-number"; 345 | printf "%.2f %.2f %.2f %.2f %.2f %d %d\n",U/NR,N/NR,S/NR,I/NR,T/NR,M/NR,NR 346 | }' > $DATADIR/$POD_NAME-client.avg 347 | } 348 | 349 | function initialize-plotly-pod { 350 | POD_NAME=$1 351 | 352 | info "Starting plotly-orca pod $POD_NAME" 353 | cat <<-EOF | kubectl apply $NAMESPACEOPT -f - >/dev/null|| fatal "Cannot create pod $POD_NAME" 354 | apiVersion: v1 355 | kind: Pod 356 | metadata: 357 | labels: 358 | app: $POD_NAME 359 | name: $POD_NAME 360 | spec: 361 | containers: 362 | - name: plotly-orca 363 | image: quay.io/plotly/orca 364 | restartPolicy: Never 365 | EOF 366 | 367 | RESOURCE_TO_CLEAN_BEFORE_EXIT="$RESOURCE_TO_CLEAN_BEFORE_EXIT pod/$POD_NAME" 368 | 369 | # Waiting for Pod to be Running 370 | kubectl wait $NAMESPACEOPT --for=condition=Ready pod/$POD_NAME \ 371 | --timeout=$((POD_WAIT_TIMEOUT + 120))s >/dev/null 2>/dev/null \ 372 | || detect-pod-failure-reason-and-fatal $POD_NAME "Failed to start pod $POD_NAME until timeout" 373 | } 374 | 375 | function plot-data { 376 | [ -x "$(which jq)" ] || fatal "jq is required for rendering json data for plots" 377 | info "Plotting data ..." 378 | initialize-plotly-pod $PLOT_POD_NAME 379 | [ ! -d "$PLOT_DIR" ] && mkdir $PLOT_DIR 380 | for tmpl in $(dirname $0)/plotly-templates/* 381 | do 382 | $DEBUG && debug "Rendering graph from $tmpl" 383 | compute-json-result | jq -M -f $tmpl |\ 384 | kubectl exec -i $NAMESPACEOPT $PLOT_POD_NAME -- orca graph \ 385 | "$PLOT_GRAPH_ARGS" > ${PLOT_DIR}/"$(basename $tmpl .jq).png" 386 | done 387 | } 388 | 389 | function compute-data-result { 390 | (cd $DATADIR; tar -czf - .) 391 | } 392 | 393 | function compute-ibdbench-result { 394 | function compute-ibdbench-metrics { 395 | POD_NAME=$1 396 | 397 | AVGMETRICS=$($BIN_AWK 'NR==2' $DATADIR/$POD_NAME-server.avg) 398 | echo -en "$($BIN_AWK '{printf $6}' <<< $AVGMETRICS)\t" 399 | echo -en "$($BIN_AWK '{printf "=%.2f+%.2f+%.2f+%.2f+%.2f",$1,$2,$3,$4,$5}' <<< $AVGMETRICS)\t" 400 | 401 | AVGMETRICS=$($BIN_AWK 'NR==2' $DATADIR/$POD_NAME-client.avg) 402 | echo -en "$($BIN_AWK '{printf $6}' <<< $AVGMETRICS)\t" 403 | echo -en "$($BIN_AWK '{printf "=%.2f+%.2f+%.2f+%.2f+%.2f",$1,$2,$3,$4,$5}' <<< $AVGMETRICS)" 404 | } 405 | 406 | # MTU 407 | echo -en "$(cat $DATADIR/mtu)\t\t" 408 | 409 | # Idle 410 | compute-ibdbench-metrics $IDLE_POD_NAME 411 | echo -en "\t" 412 | 413 | # P2P TCP 414 | echo -en "$(cat $DATADIR/$CLIENT_TCP_P2P_POD_NAME.bw)\t" 415 | compute-ibdbench-metrics $CLIENT_TCP_P2P_POD_NAME 416 | echo -en "\t" 417 | 418 | # P2P UDP 419 | echo -en "$(cat $DATADIR/$CLIENT_UDP_P2P_POD_NAME.bw)\t" 420 | compute-ibdbench-metrics $CLIENT_UDP_P2P_POD_NAME 421 | echo -en "\t" 422 | 423 | # P2S TCP 424 | echo -en "$(cat $DATADIR/$CLIENT_TCP_P2S_POD_NAME.bw)\t" 425 | compute-ibdbench-metrics $CLIENT_TCP_P2S_POD_NAME 426 | echo -en "\t" 427 | 428 | # P2S UDP 429 | echo -en "$(cat $DATADIR/$CLIENT_UDP_P2S_POD_NAME.bw)\t" 430 | compute-ibdbench-metrics $CLIENT_UDP_P2S_POD_NAME 431 | } 432 | 433 | function compute-json-result { 434 | function compute-json-result-for-pod { 435 | POD_NAME=$1 436 | PADDING=$(printf "%${2}s") 437 | 438 | echo "${PADDING}\"bandwidth\": $(cat $DATADIR/$POD_NAME.bw)," 439 | 440 | AVGMETRICS=$($BIN_AWK 'NR==2' $DATADIR/$POD_NAME-client.avg) 441 | $DEBUG && debug "pod $POD_NAME client metrics avg : $AVGMETRICS " 442 | echo "${PADDING}\"client\": {" 443 | echo "${PADDING} \"cpu\": {" 444 | echo "${PADDING} \"total\": $($BIN_AWK '{printf "%.2f",$1+$2+$3+$4+$5}' <<< $AVGMETRICS)," 445 | echo "${PADDING} \"user\": $($BIN_AWK '{print $1}' <<< $AVGMETRICS)," 446 | echo "${PADDING} \"nice\": $($BIN_AWK '{print $2}' <<< $AVGMETRICS)," 447 | echo "${PADDING} \"system\": $($BIN_AWK '{print $3}' <<< $AVGMETRICS)," 448 | echo "${PADDING} \"iowait\": $($BIN_AWK '{print $4}' <<< $AVGMETRICS)," 449 | echo "${PADDING} \"steal\": $($BIN_AWK '{print $5}' <<< $AVGMETRICS)" 450 | echo "${PADDING} }," 451 | echo "${PADDING} \"ram\": $($BIN_AWK '{print $6}' <<< $AVGMETRICS)" 452 | echo "${PADDING}}," 453 | 454 | AVGMETRICS=$($BIN_AWK 'NR==2' $DATADIR/$POD_NAME-server.avg) 455 | $DEBUG && debug "pod $POD_NAME server metrics avg : $AVGMETRICS " 456 | echo "${PADDING}\"server\": {" 457 | echo "${PADDING} \"cpu\": {" 458 | echo "${PADDING} \"total\": $($BIN_AWK '{printf "%.2f",$1+$2+$3+$4+$5}' <<< $AVGMETRICS)," 459 | echo "${PADDING} \"user\": $($BIN_AWK '{print $1}' <<< $AVGMETRICS)," 460 | echo "${PADDING} \"nice\": $($BIN_AWK '{print $2}' <<< $AVGMETRICS)," 461 | echo "${PADDING} \"system\": $($BIN_AWK '{print $3}' <<< $AVGMETRICS)," 462 | echo "${PADDING} \"iowait\": $($BIN_AWK '{print $4}' <<< $AVGMETRICS)," 463 | echo "${PADDING} \"steal\": $($BIN_AWK '{print $5}' <<< $AVGMETRICS)" 464 | echo "${PADDING} }," 465 | echo "${PADDING} \"ram\": $($BIN_AWK '{print $6}' <<< $AVGMETRICS)" 466 | echo "${PADDING}}" 467 | } 468 | 469 | echo "{" 470 | echo " \"metadata\": {" 471 | echo " \"name\": \"$BENCHMARK_RUN_NAME\"," 472 | echo " \"generator\": \"$GENERATOR\"," 473 | echo " \"version\": \"$VERSION\"," 474 | echo " \"date\": \"$BENCHMARK_DATE\"," 475 | echo " \"server-node\": \"$SERVER_NODE\"," 476 | echo " \"client-node\": \"$CLIENT_NODE\"", 477 | echo " \"socket-buffer-size\": \"$SOCKET_BUFFER_SIZE\"" 478 | echo " }," 479 | echo " \"data\": {" 480 | echo " \"cpu\": \"$(cat $DATADIR/cpu)\"," 481 | echo " \"kernel\": \"$(cat $DATADIR/kernel)\"," 482 | echo " \"k8s-version\": \"$(cat $DATADIR/k8s-version)\"," 483 | echo " \"mtu\": \"$(cat $DATADIR/mtu)\"," 484 | if [ "${IDLE_POD_NAME}" != "" ] 485 | then 486 | echo " \"idle\": {" 487 | compute-json-result-for-pod $IDLE_POD_NAME 6 488 | echo " }," 489 | fi 490 | 491 | if [ "${CLIENT_TCP_P2P_POD_NAME}${CLIENT_UDP_P2P_POD_NAME}" != "" ] 492 | then 493 | echo " \"pod2pod\": {" 494 | if [ "${CLIENT_TCP_P2P_POD_NAME}" != "" ] 495 | then 496 | echo " \"tcp\": {" 497 | compute-json-result-for-pod $CLIENT_TCP_P2P_POD_NAME 8 498 | if [ "${CLIENT_UDP_P2P_POD_NAME}" != "" ] 499 | then 500 | echo " }," 501 | else 502 | echo " }" 503 | fi 504 | fi 505 | if [ "${CLIENT_UDP_P2P_POD_NAME}" != "" ] 506 | then 507 | echo " \"udp\": {" 508 | compute-json-result-for-pod $CLIENT_UDP_P2P_POD_NAME 8 509 | echo " }" 510 | fi 511 | if [ "${CLIENT_TCP_P2S_POD_NAME}" != "" ] || [ "${CLIENT_UDP_P2S_POD_NAME}" != "" ] 512 | then 513 | echo " }," 514 | else 515 | echo " }" 516 | fi 517 | fi 518 | 519 | if [ "${CLIENT_TCP_P2S_POD_NAME}${CLIENT_UDP_P2S_POD_NAME}" != "" ] 520 | then 521 | echo " \"pod2svc\": {" 522 | if [ "${CLIENT_TCP_P2S_POD_NAME}" != "" ] 523 | then 524 | echo " \"tcp\": {" 525 | compute-json-result-for-pod $CLIENT_TCP_P2S_POD_NAME 8 526 | if [ "${CLIENT_UDP_P2S_POD_NAME}" != "" ] 527 | then 528 | echo " }," 529 | else 530 | echo " }" 531 | fi 532 | fi 533 | if [ "${CLIENT_UDP_P2S_POD_NAME}" != "" ] 534 | then 535 | echo " \"udp\": {" 536 | compute-json-result-for-pod $CLIENT_UDP_P2S_POD_NAME 8 537 | echo " }" 538 | fi 539 | echo " }" 540 | fi 541 | echo " }" 542 | echo "}" 543 | } 544 | 545 | function compute-yaml-result { 546 | 547 | function compute-yaml-result-for-pod { 548 | POD_NAME=$1 549 | PADDING=$(printf "%${2}s") 550 | 551 | echo "${PADDING}bandwidth: $(cat $DATADIR/$POD_NAME.bw)" 552 | AVGMETRICS=$($BIN_AWK 'NR==2' $DATADIR/$POD_NAME-client.avg) 553 | $DEBUG && debug "pod $POD_NAME client metrics avg : $AVGMETRICS " 554 | echo "${PADDING}client:" 555 | echo "${PADDING} cpu:" 556 | echo "${PADDING} total: $($BIN_AWK '{printf "%.2f",$1+$2+$3+$4+$5}' <<< $AVGMETRICS)" 557 | echo "${PADDING} user: $($BIN_AWK '{print $1}' <<< $AVGMETRICS)" 558 | echo "${PADDING} nice: $($BIN_AWK '{print $2}' <<< $AVGMETRICS)" 559 | echo "${PADDING} system: $($BIN_AWK '{print $3}' <<< $AVGMETRICS)" 560 | echo "${PADDING} iowait: $($BIN_AWK '{print $4}' <<< $AVGMETRICS)" 561 | echo "${PADDING} steal: $($BIN_AWK '{print $5}' <<< $AVGMETRICS)" 562 | echo "${PADDING} ram: $($BIN_AWK '{print $6}' <<< $AVGMETRICS)" 563 | 564 | AVGMETRICS=$($BIN_AWK 'NR==2' $DATADIR/$POD_NAME-server.avg) 565 | $DEBUG && debug "pod $POD_NAME server metrics avg : $AVGMETRICS " 566 | echo "${PADDING}server:" 567 | echo "${PADDING} cpu:" 568 | echo "${PADDING} total: $($BIN_AWK '{printf "%.2f",$1+$2+$3+$4+$5}' <<< $AVGMETRICS)" 569 | echo "${PADDING} user: $($BIN_AWK '{print $1}' <<< $AVGMETRICS)" 570 | echo "${PADDING} nice: $($BIN_AWK '{print $2}' <<< $AVGMETRICS)" 571 | echo "${PADDING} system: $($BIN_AWK '{print $3}' <<< $AVGMETRICS)" 572 | echo "${PADDING} iowait: $($BIN_AWK '{print $4}' <<< $AVGMETRICS)" 573 | echo "${PADDING} steal: $($BIN_AWK '{print $5}' <<< $AVGMETRICS)" 574 | echo "${PADDING} ram: $($BIN_AWK '{print $6}' <<< $AVGMETRICS)" 575 | } 576 | 577 | echo "metadata:" 578 | echo " name: \"$BENCHMARK_RUN_NAME\"" 579 | echo " generator: \"$GENERATOR\"" 580 | echo " version: \"$VERSION\"" 581 | echo " date: \"$BENCHMARK_DATE\"" 582 | echo " server-node: \"$SERVER_NODE\"" 583 | echo " client-node: \"$CLIENT_NODE\"" 584 | echo " socket-buffer-size: \"$SOCKET_BUFFER_SIZE\"" 585 | echo "data:" 586 | echo " cpu: \"$(cat $DATADIR/cpu)\"" 587 | echo " kernel: \"$(cat $DATADIR/kernel)\"" 588 | echo " k8s-version: \"$(cat $DATADIR/k8s-version)\"" 589 | echo " mtu: \"$(cat $DATADIR/mtu)\"" 590 | if [ "${IDLE_POD_NAME}" != "" ] 591 | then 592 | echo " idle:" 593 | compute-yaml-result-for-pod $IDLE_POD_NAME 4 594 | fi 595 | if [ "${CLIENT_TCP_P2P_POD_NAME}${CLIENT_UDP_P2P_POD_NAME}" != "" ] 596 | then 597 | echo " pod2pod:" 598 | if [ "${CLIENT_TCP_P2P_POD_NAME}" != "" ] 599 | then 600 | echo " tcp:" 601 | compute-yaml-result-for-pod $CLIENT_TCP_P2P_POD_NAME 6 602 | fi 603 | if [ "${CLIENT_UDP_P2P_POD_NAME}" != "" ] 604 | then 605 | echo " udp:" 606 | compute-yaml-result-for-pod $CLIENT_UDP_P2P_POD_NAME 6 607 | fi 608 | fi 609 | if [ "${CLIENT_TCP_P2S_POD_NAME}${CLIENT_UDP_P2S_POD_NAME}" != "" ] 610 | then 611 | echo " pod2svc:" 612 | if [ "${CLIENT_TCP_P2S_POD_NAME}" != "" ] 613 | then 614 | echo " tcp:" 615 | compute-yaml-result-for-pod $CLIENT_TCP_P2S_POD_NAME 6 616 | fi 617 | if [ "${CLIENT_UDP_P2S_POD_NAME}" != "" ] 618 | then 619 | echo " udp:" 620 | compute-yaml-result-for-pod $CLIENT_UDP_P2S_POD_NAME 6 621 | fi 622 | fi 623 | } 624 | 625 | function compute-text-result { 626 | function compute-text-result-for-pod { 627 | POD_NAME=$1 628 | PADDING=$(printf "%${2}s") 629 | echo "${PADDING}bandwidth = $(cat $DATADIR/$POD_NAME.bw) Mbit/s" 630 | echo "${PADDING}client cpu = $($BIN_AWK 'NR==2 {printf "total %.2f%% (user %.2f%%, nice %.2f%%, system %.2f%%, iowait %.2f%%, steal %.2f%%)",$1+$2+$3+$4+$5,$1,$2,$3,$4,$5}' $DATADIR/$POD_NAME-client.avg)" 631 | echo "${PADDING}server cpu = $($BIN_AWK 'NR==2 {printf "total %.2f%% (user %.2f%%, nice %.2f%%, system %.2f%%, iowait %.2f%%, steal %.2f%%)",$1+$2+$3+$4+$5,$1,$2,$3,$4,$5}' $DATADIR/$POD_NAME-server.avg)" 632 | echo "${PADDING}client ram = $($BIN_AWK 'NR==2 {print $6}' $DATADIR/$POD_NAME-client.avg) MB" 633 | echo "${PADDING}server ram = $($BIN_AWK 'NR==2 {print $6}' $DATADIR/$POD_NAME-server.avg) MB" 634 | } 635 | 636 | echo "=========================================================" 637 | echo " Benchmark Results" 638 | echo "=========================================================" 639 | echo " Name : $BENCHMARK_RUN_NAME" 640 | echo " Date : $BENCHMARK_DATE UTC" 641 | echo " Generator : $GENERATOR" 642 | echo " Version : $VERSION" 643 | echo " Server : $SERVER_NODE" 644 | echo " Client : $CLIENT_NODE" 645 | echo " UDP Socket size : $SOCKET_BUFFER_SIZE" 646 | echo "=========================================================" 647 | echo " Discovered CPU : $(cat $DATADIR/cpu)" 648 | echo " Discovered Kernel : $(cat $DATADIR/kernel)" 649 | echo " Discovered k8s version : $(cat $DATADIR/k8s-version)" 650 | echo " Discovered MTU : $(cat $DATADIR/mtu)" 651 | if [ "${IDLE_POD_NAME}" != "" ] 652 | then 653 | echo " Idle :" 654 | compute-text-result-for-pod $IDLE_POD_NAME 4 655 | fi 656 | if [ "${CLIENT_TCP_P2P_POD_NAME}${CLIENT_UDP_P2P_POD_NAME}" != "" ] 657 | then 658 | echo " Pod to pod :" 659 | if [ "${CLIENT_TCP_P2P_POD_NAME}" != "" ] 660 | then 661 | echo " TCP :" 662 | compute-text-result-for-pod $CLIENT_TCP_P2P_POD_NAME 6 663 | fi 664 | if [ "${CLIENT_UDP_P2P_POD_NAME}" != "" ] 665 | then 666 | echo " UDP :" 667 | compute-text-result-for-pod $CLIENT_UDP_P2P_POD_NAME 6 668 | fi 669 | fi 670 | 671 | if [ "${CLIENT_TCP_P2S_POD_NAME}${CLIENT_UDP_P2S_POD_NAME}" != "" ] 672 | then 673 | echo " Pod to Service :" 674 | if [ "${CLIENT_TCP_P2S_POD_NAME}" != "" ] 675 | then 676 | echo " TCP :"; 677 | compute-text-result-for-pod $CLIENT_TCP_P2S_POD_NAME 6 678 | fi 679 | if [ "${CLIENT_UDP_P2S_POD_NAME}" != "" ] 680 | then 681 | echo " UDP :" 682 | compute-text-result-for-pod $CLIENT_UDP_P2S_POD_NAME 6 683 | fi 684 | echo "=========================================================" 685 | fi 686 | } 687 | 688 | function generate-report { 689 | case $OUTPUT_FORMAT in 690 | json) compute-json-result > $FILENAME;; 691 | yaml) compute-yaml-result > $FILENAME;; 692 | data) compute-data-result > $FILENAME;; 693 | ibdbench) compute-ibdbench-result > $FILENAME;; 694 | *) compute-text-result > $FILENAME;; 695 | esac 696 | } 697 | 698 | function source-data-from-dir { 699 | source "$DATADIR/env" 700 | $DEBUG && debug "GENERATOR=$GENERATOR" 701 | $DEBUG && debug "VERSION=$VERSION" 702 | $DEBUG && debug "BENCHMARK_DATE=$BENCHMARK_DATE" 703 | $DEBUG && debug "SERVER_NODE=$SERVER_NODE" 704 | $DEBUG && debug "CLIENT_NODE=$CLIENT_NODE" 705 | 706 | $DEBUG && debug "MONITOR_SERVER_POD_NAME=$MONITOR_SERVER_POD_NAME" 707 | $DEBUG && debug "MONITOR_CLIENT_POD_NAME=$MONITOR_CLIENT_POD_NAME" 708 | $DEBUG && debug "SERVER_POD_NAME=$SERVER_POD_NAME" 709 | $DEBUG && debug "IDLE_POD_NAME=$IDLE_POD_NAME" 710 | $DEBUG && debug "CLIENT_TCP_P2P_POD_NAME=$CLIENT_TCP_P2P_POD_NAME" 711 | $DEBUG && debug "CLIENT_UDP_P2P_POD_NAME=$CLIENT_UDP_P2P_POD_NAME" 712 | $DEBUG && debug "CLIENT_TCP_P2S_POD_NAME=$CLIENT_TCP_P2S_POD_NAME" 713 | $DEBUG && debug "CLIENT_UDP_P2S_POD_NAME=$CLIENT_UDP_P2S_POD_NAME" 714 | } 715 | 716 | #============================================================================== 717 | # Argument parsing 718 | #============================================================================== 719 | 720 | [ "$1" = "" ] && usage && exit 721 | 722 | UNKNOWN_ARGLIST="" 723 | while [ "$1" != "" ] 724 | do 725 | arg=$1 726 | case $arg in 727 | #--- Benchmark mode - Mandatory flags --------------------------------- 728 | 729 | # Define kubernetes node name that will host the client part 730 | --server-node|-sn) 731 | shift 732 | [ "$1" = "" ] && fatal "$arg flag must be followed by a value" 733 | SERVER_NODE=$1 734 | info "Server node will be '$SERVER_NODE'" 735 | ;; 736 | 737 | # Define kubernetes node name that will host the server part 738 | --client-node|-cn) 739 | shift 740 | [ "$1" = "" ] && fatal "$arg flag must be followed by a value" 741 | CLIENT_NODE=$1 742 | info "Client node will be '$CLIENT_NODE'" 743 | ;; 744 | 745 | #--- Benchmark mode - Optionnal flags --------------------------------- 746 | 747 | # Set the benchmark duration for each test in seconds 748 | --duration|-d) 749 | shift 750 | [ "$1" = "" ] && fatal "$arg flag must be followed by a value" 751 | BENCHMARK_DURATION="$1" 752 | info "Setting benchmark duration to ${1}s" 753 | ;; 754 | 755 | # Keep data directory instead of cleaning it (tmp dir that contains raw benchmark data) 756 | --keep|-k) 757 | CLEANDIR="false" 758 | info "Keeping datadir after benchmark run" 759 | ;; 760 | 761 | # Set the target kubernetes namespace 762 | --namespace|-n) 763 | shift 764 | [ "$1" = "" ] && fatal "$arg flag must be followed by a value" 765 | NAMESPACEOPT="--namespace $1" 766 | info "Setting target namespace to '$1'" 767 | ;; 768 | 769 | # Set the name of this benchmark run 770 | --name) 771 | shift 772 | [ "$1" = "" ] && fatal "$arg flag must be followed by a value" 773 | BENCHMARK_RUN_NAME="$1" 774 | info "Setting benchmark run nameto '$1'" 775 | ;; 776 | 777 | # Only run a subset of benchmark tests, comma separated (Ex: -ot tcp,idle) 778 | --only-tests|-ot) 779 | shift 780 | [ "$1" = "" ] && fatal "$arg flag must be followed by a value" 781 | 782 | RUN_TEST_IDLE="false" 783 | RUN_TEST_P2P_TCP="false" 784 | RUN_TEST_P2P_UDP="false" 785 | RUN_TEST_P2S_TCP="false" 786 | RUN_TEST_P2S_UDP="false" 787 | 788 | for i in $(awk '{gsub(/,/," ");print}' <<< "$1") 789 | do 790 | case $i in 791 | all) 792 | RUN_TEST_P2P_TCP="true" 793 | RUN_TEST_P2P_UDP="true" 794 | RUN_TEST_P2S_TCP="true" 795 | RUN_TEST_P2S_UDP="true" 796 | RUN_TEST_IDLE="true" 797 | ;; 798 | p2p) 799 | RUN_TEST_P2P_TCP="true" 800 | RUN_TEST_P2P_UDP="true" 801 | ;; 802 | p2s) 803 | RUN_TEST_P2S_TCP="true" 804 | RUN_TEST_P2S_UDP="true" 805 | ;; 806 | tcp) 807 | RUN_TEST_P2P_TCP="true" 808 | RUN_TEST_P2S_TCP="true" 809 | ;; 810 | udp) 811 | RUN_TEST_P2P_UDP="true" 812 | RUN_TEST_P2S_UDP="true" 813 | ;; 814 | p2ptcp) RUN_TEST_P2P_TCP="true" ;; 815 | p2pudp) RUN_TEST_P2P_UDP="true" ;; 816 | p2stcp) RUN_TEST_P2S_TCP="true" ;; 817 | p2sudp) RUN_TEST_P2S_UDP="true" ;; 818 | idle) RUN_TEST_IDLE="true" ;; 819 | *) fatal "Unknown test slug '${i}', " ;; 820 | esac 821 | done 822 | info "Benchmark will run only following tests: ${1}" 823 | ;; 824 | 825 | # Set the socket buffer size with unit, or 'auto'. ex: '256K' 826 | --socket-buffer-size|-sbs) 827 | shift 828 | [ "$1" = "" ] && fatal "$arg flag must be followed by a value" 829 | SOCKET_BUFFER_SIZE="$1" 830 | info "Setting UDP socket buffer size to ${1}" 831 | ;; 832 | 833 | # Set the pod ready wait timeout in seconds 834 | --timeout|-t) 835 | shift 836 | [ "$1" = "" ] && fatal "$arg flag must be followed by a value" 837 | POD_WAIT_TIMEOUT="$1" 838 | info "Setting pod wait timeout to ${1}s" 839 | ;; 840 | 841 | #--- From Data - Mandatory flags -------------------------------------- 842 | 843 | # Define the path to the data to read from 844 | --from-data|-fd) 845 | shift 846 | [ "$1" = "" ] && fatal "$arg flag must be followed by a value" 847 | FROM_DATA="true" 848 | CLEANDIR="false" 849 | DATADIR="$1" 850 | info "Benchmark will not run, report will be generated from ${1}" 851 | ;; 852 | 853 | #--- Common - Optionnal flags ----------------------------------------- 854 | 855 | # Set the debug level to "debug" 856 | --debug) 857 | DEBUG_LEVEL=3; 858 | DEBUG=true; 859 | debug "Mode debug ON" 860 | ;; 861 | 862 | # Set the debug level 863 | --debug-level|-dl) 864 | shift 865 | case $1 in 866 | standard|0) DEBUG_LEVEL=0; DEBUG=false;; 867 | warn|1) DEBUG_LEVEL=1; DEBUG=false;; 868 | info|2) DEBUG_LEVEL=2; DEBUG=false;; 869 | debug|3) DEBUG_LEVEL=3; DEBUG=true; debug "Mode debug ON";; 870 | *) fatal "Unknown debug level '$1'. Here is a list of acceptable debug levels : standard, warn, info, debug";; 871 | esac 872 | ;; 873 | 874 | # Set the output file 875 | --file|-f) 876 | shift 877 | [ "$1" = "" ] && fatal "$arg flag must be followed by a value" 878 | FILENAME="$1" 879 | [ "$FILENAME" = "-" ] && FILENAME="/dev/stdout" 880 | info "Output file set to ${FILENAME}" 881 | ;; 882 | 883 | # Display help message 884 | --help|-h) 885 | usage && exit 886 | ;; 887 | 888 | # Plot data with plotly/orca 889 | --plot|-p) 890 | PLOT_DATA=true 891 | ;; 892 | # Directory where to save graphs: 893 | --plot-dir|-pd) 894 | shift 895 | [ "$1" = "" ] && fatal "$arg flag must be followed by a value" 896 | PLOT_DIR=$1 897 | info "Plot directory set to ${PLOT_DIR}" 898 | ;; 899 | # Arguments for 'orca graph': 900 | --plot-args|-pa) 901 | shift 902 | [ "$1" = "" ] && fatal "$arg flag must be followed by a value" 903 | PLOT_GRAPH_ARGS=$1 904 | info "Plot orca graph arguments are set to ${PLOT_GRAPH_ARGS}" 905 | ;; 906 | # Set the output format 907 | --output|-o) 908 | shift 909 | [ "$1" = "" ] && fatal "$arg flag must be followed by a value" 910 | # Checking values 911 | case $1 in 912 | json) ;; 913 | yaml) ;; 914 | text) ;; 915 | data) ;; 916 | ibdbench) ;; 917 | *) fatal "Unknown output format '$1'. Here is a list of acceptable formats : text, json, yaml, data";; 918 | esac 919 | 920 | info "Setting output format to ${1}s" 921 | OUTPUT_FORMAT="$1" 922 | ;; 923 | 924 | # Activate the verbose mode by setting debug-level to 'info' 925 | --verbose|-v) 926 | DEBUG_LEVEL=2 927 | ;; 928 | 929 | # Show current script version 930 | --version|-V) 931 | echo "$VERSION" 932 | exit 933 | ;; 934 | 935 | # Unknown flag 936 | *) UNKNOWN_ARGLIST="$UNKNOWN_ARGLIST $arg" ;; 937 | esac 938 | shift 939 | done 940 | 941 | [ "$UNKNOWN_ARGLIST" != "" ] && fatal "Unknown arguments : $UNKNOWN_ARGLIST" 942 | $DEBUG && debug "Argument parsing done" 943 | 944 | #============================================================================== 945 | # Preflight checks 946 | #============================================================================== 947 | 948 | [ "$FILENAME" = "/dev/stdout" -a "$OUTPUT_FORMAT" = "data" ] && fatal "I cowardly refuse to output data on stdout, please use '-f' flag" 949 | 950 | #--- Mode detection --------------------------------------- 951 | 952 | # We do not run benchmark test, we source from existing data 953 | if $FROM_DATA 954 | then 955 | $DEBUG && debug "Running in 'from data' mode" 956 | 957 | # Data source is a directory, let's source data 958 | if [ -d "$DATADIR" -a -f "$DATADIR/env" ] 959 | then 960 | $DEBUG && debug "Data is a directory, sourcing env file" 961 | 962 | source-data-from-dir 963 | generate-report 964 | 965 | # In this case, the datasource is a file, a compressed tar 966 | elif [ -f "$DATADIR" ] 967 | then 968 | $DEBUG && debug "Data is a file" 969 | 970 | # Saving filename 971 | FROM_DATA_FILE="$DATADIR" 972 | 973 | # Generating a temp dir to work 974 | DATADIR="$(mktemp -d)" 975 | 976 | # Setting the cleandir flag, as datadit is now a temp dir 977 | CLEANDIR="true" 978 | 979 | # Extracting data in DATADIR tmp dir 980 | tar -xzf "$FROM_DATA_FILE" -C "$DATADIR" || fatal "Cannot extract data from file $FROM_DATA_FILE" 981 | 982 | source-data-from-dir 983 | generate-report 984 | $PLOT_DATA && plot-data 985 | 986 | else 987 | fatal "Cannot recognize data format $DATADIR" 988 | fi 989 | 990 | # In every case, we didn't want to run benchmark test so we exit 991 | cleanandexit 992 | fi 993 | 994 | $DEBUG && debug "Running in 'benchmark' mode" 995 | 996 | #--- Node name check -------------------------------------- 997 | 998 | [ "$SERVER_NODE" = "" -o "$CLIENT_NODE" = "" ] \ 999 | && fatal "Both client node and server node must be set (--server-node and --client-node)" 1000 | 1001 | #============================================================================== 1002 | # Preparation 1003 | #============================================================================== 1004 | 1005 | DATADIR="$(mktemp -d)" 1006 | 1007 | # Generating pod names 1008 | MONITOR_SERVER_POD_NAME="knb-monitor-server-$EXECID" 1009 | MONITOR_CLIENT_POD_NAME="knb-monitor-client-$EXECID" 1010 | SERVER_POD_NAME="knb-server-$EXECID" 1011 | SERVER_SERVICE_NAME=$SERVER_POD_NAME 1012 | 1013 | IDLE_POD_NAME="" 1014 | CLIENT_TCP_P2P_POD_NAME="" 1015 | CLIENT_UDP_P2P_POD_NAME="" 1016 | CLIENT_TCP_P2S_POD_NAME="" 1017 | CLIENT_UDP_P2S_POD_NAME="" 1018 | 1019 | $RUN_TEST_IDLE && IDLE_POD_NAME="knb-client-idle-$EXECID" 1020 | $RUN_TEST_P2P_TCP && CLIENT_TCP_P2P_POD_NAME="knb-client-tcp-p2p-$EXECID" 1021 | $RUN_TEST_P2P_UDP && CLIENT_UDP_P2P_POD_NAME="knb-client-udp-p2p-$EXECID" 1022 | $RUN_TEST_P2S_TCP && CLIENT_TCP_P2S_POD_NAME="knb-client-tcp-p2s-$EXECID" 1023 | $RUN_TEST_P2S_UDP && CLIENT_UDP_P2S_POD_NAME="knb-client-udp-p2s-$EXECID" 1024 | 1025 | # Creating env file with metadata 1026 | cat < $DATADIR/env 1027 | BENCHMARK_RUN_NAME="$BENCHMARK_RUN_NAME" 1028 | GENERATOR="$GENERATOR" 1029 | VERSION="$VERSION" 1030 | BENCHMARK_DATE="$BENCHMARK_DATE" 1031 | SERVER_NODE="$SERVER_NODE" 1032 | CLIENT_NODE="$CLIENT_NODE" 1033 | 1034 | SOCKET_BUFFER_SIZE="$SOCKET_BUFFER_SIZE" 1035 | 1036 | MONITOR_SERVER_POD_NAME="$MONITOR_SERVER_POD_NAME" 1037 | MONITOR_CLIENT_POD_NAME="$MONITOR_CLIENT_POD_NAME" 1038 | SERVER_POD_NAME="$SERVER_POD_NAME" 1039 | 1040 | IDLE_POD_NAME="$IDLE_POD_NAME" 1041 | CLIENT_TCP_P2P_POD_NAME="$CLIENT_TCP_P2P_POD_NAME" 1042 | CLIENT_UDP_P2P_POD_NAME="$CLIENT_UDP_P2P_POD_NAME" 1043 | CLIENT_TCP_P2S_POD_NAME="$CLIENT_TCP_P2S_POD_NAME" 1044 | CLIENT_UDP_P2S_POD_NAME="$CLIENT_UDP_P2S_POD_NAME" 1045 | EOF 1046 | 1047 | # Catching CTRL-C 1048 | trap ctrl_c INT 1049 | function ctrl_c { 1050 | err "Trapped CTRL-C, trying to exit gracefully ..." 1051 | cleanandexit 1 1052 | } 1053 | 1054 | #============================================================================== 1055 | # Starting monitors 1056 | #============================================================================== 1057 | 1058 | #--- Server monitor -------------------------------- 1059 | info "Deploying server monitor on node $SERVER_NODE" 1060 | cat </dev/null|| fatal "Cannot create server monitor pod" 1061 | apiVersion: v1 1062 | kind: Pod 1063 | metadata: 1064 | labels: 1065 | app: $MONITOR_SERVER_POD_NAME 1066 | name: $MONITOR_SERVER_POD_NAME 1067 | spec: 1068 | containers: 1069 | - name: monitor 1070 | image: infrabuilder/bench-custom-monitor 1071 | nodeSelector: 1072 | kubernetes.io/hostname: $SERVER_NODE 1073 | EOF 1074 | 1075 | RESOURCE_TO_CLEAN_BEFORE_EXIT="$RESOURCE_TO_CLEAN_BEFORE_EXIT pod/$MONITOR_SERVER_POD_NAME" 1076 | 1077 | info "Waiting for server monitor to be running" 1078 | waitpod $MONITOR_SERVER_POD_NAME Running $POD_WAIT_TIMEOUT \ 1079 | || detect-pod-failure-reason-and-fatal $MONITOR_SERVER_POD_NAME "Failed to start server monitor pod" 1080 | 1081 | #--- Client monitor -------------------------------- 1082 | info "Deploying client monitor on node $CLIENT_NODE" 1083 | cat </dev/null|| fatal "Cannot create client monitor pod" 1084 | apiVersion: v1 1085 | kind: Pod 1086 | metadata: 1087 | labels: 1088 | app: $MONITOR_CLIENT_POD_NAME 1089 | name: $MONITOR_CLIENT_POD_NAME 1090 | spec: 1091 | containers: 1092 | - name: monitor 1093 | image: infrabuilder/bench-custom-monitor 1094 | nodeSelector: 1095 | kubernetes.io/hostname: $CLIENT_NODE 1096 | EOF 1097 | 1098 | RESOURCE_TO_CLEAN_BEFORE_EXIT="$RESOURCE_TO_CLEAN_BEFORE_EXIT pod/$MONITOR_CLIENT_POD_NAME" 1099 | 1100 | info "Waiting for client monitor to be running" 1101 | waitpod $MONITOR_CLIENT_POD_NAME Running $POD_WAIT_TIMEOUT \ 1102 | || detect-pod-failure-reason-and-fatal $MONITOR_CLIENT_POD_NAME "Failed to start client monitor pod" 1103 | 1104 | #============================================================================== 1105 | # Starting server 1106 | #============================================================================== 1107 | 1108 | info "Deploying iperf server on node $SERVER_NODE" 1109 | cat </dev/null|| fatal "Cannot create server pod" 1110 | apiVersion: v1 1111 | kind: Pod 1112 | metadata: 1113 | labels: 1114 | app: $SERVER_POD_NAME 1115 | name: $SERVER_POD_NAME 1116 | spec: 1117 | containers: 1118 | - name: iperf 1119 | image: infrabuilder/netbench:server-iperf3 1120 | args: 1121 | - iperf3 1122 | - -s 1123 | nodeSelector: 1124 | kubernetes.io/hostname: $SERVER_NODE 1125 | --- 1126 | apiVersion: v1 1127 | kind: Service 1128 | metadata: 1129 | name: $SERVER_SERVICE_NAME 1130 | spec: 1131 | selector: 1132 | app: $SERVER_POD_NAME 1133 | ports: 1134 | - protocol: TCP 1135 | port: 5201 1136 | targetPort: 5201 1137 | name: tcp 1138 | - protocol: UDP 1139 | port: 5201 1140 | targetPort: 5201 1141 | name: udp 1142 | EOF 1143 | 1144 | RESOURCE_TO_CLEAN_BEFORE_EXIT="$RESOURCE_TO_CLEAN_BEFORE_EXIT pod/$SERVER_POD_NAME svc/$SERVER_SERVICE_NAME" 1145 | 1146 | info "Waiting for server to be running" 1147 | kubectl wait $NAMESPACEOPT --for=condition=Ready pod/$SERVER_POD_NAME --timeout=${POD_WAIT_TIMEOUT}s >/dev/null \ 1148 | || detect-pod-failure-reason-and-fatal $SERVER_POD_NAME "Failed to start server pod" 1149 | 1150 | SERVER_IP=$(kubectl get $NAMESPACEOPT pod $SERVER_POD_NAME -o jsonpath={.status.podIP}) 1151 | $DEBUG && debug "Server IP address is $SERVER_IP" 1152 | 1153 | #============================================================================== 1154 | # Data Detection 1155 | #============================================================================== 1156 | 1157 | #--- CPU Detection ---------------------------------------- 1158 | 1159 | info "Detecting CPU" 1160 | DISCOVERED_CPU=$(kubectl exec -i $NAMESPACEOPT $SERVER_POD_NAME -- grep "model name" /proc/cpuinfo | tail -n 1 | awk -F: '{print $2}') 1161 | $DEBUG && debug "Cpu is $DISCOVERED_CPU" 1162 | echo $DISCOVERED_CPU > $DATADIR/cpu 1163 | 1164 | #--- Kernel Detection ------------------------------------- 1165 | 1166 | info "Detecting Kernel" 1167 | DISCOVERED_KERNEL=$(kubectl exec -i $NAMESPACEOPT $SERVER_POD_NAME -- uname -r) 1168 | $DEBUG && debug "Kernel is $DISCOVERED_KERNEL" 1169 | echo $DISCOVERED_KERNEL > $DATADIR/kernel 1170 | 1171 | 1172 | #--- K8S Version Detection -------------------------------- 1173 | 1174 | info "Detecting Kubernetes version" 1175 | DISCOVERED_K8S_VERSION=$(kubectl version --short | awk '$1=="Server" {print $3}' ) 1176 | $DEBUG && debug "Discovered k8s version is $DISCOVERED_K8S_VERSION" 1177 | echo $DISCOVERED_K8S_VERSION > $DATADIR/k8s-version 1178 | 1179 | #--- MTU Detection ---------------------------------------- 1180 | 1181 | info "Detecting CNI MTU" 1182 | CNI_MTU=$(kubectl exec -i $NAMESPACEOPT $SERVER_POD_NAME -- ip link \ 1183 | | grep "UP,LOWER_UP" | grep -v LOOPBACK | grep -oE "mtu [0-9]*"| $BIN_AWK '{print $2}') 1184 | $DEBUG && debug "CNI MTU is $CNI_MTU" 1185 | echo $CNI_MTU > $DATADIR/mtu 1186 | 1187 | #============================================================================== 1188 | # Benchmark tests 1189 | #============================================================================== 1190 | 1191 | #--- Idle measurment ------------------------------------- 1192 | 1193 | if [ "$IDLE_POD_NAME" != "" ] 1194 | then 1195 | run-client $IDLE_POD_NAME $SERVER_IP idle 1196 | else 1197 | info "Skipping Idle test" 1198 | fi 1199 | 1200 | #--- Pod to pod ------------------------------------------- 1201 | 1202 | if [ "$CLIENT_TCP_P2P_POD_NAME" != "" ] 1203 | then 1204 | run-client $CLIENT_TCP_P2P_POD_NAME $SERVER_IP tcp 1205 | else 1206 | info "Skipping P2P TCP test" 1207 | fi 1208 | 1209 | if [ "$CLIENT_UDP_P2P_POD_NAME" != "" ] 1210 | then 1211 | run-client $CLIENT_UDP_P2P_POD_NAME $SERVER_IP udp 1212 | else 1213 | info "Skipping P2P UDP test" 1214 | fi 1215 | 1216 | #--- Pod to Service --------------------------------------- 1217 | 1218 | if [ "$CLIENT_TCP_P2S_POD_NAME" != "" ] 1219 | then 1220 | run-client $CLIENT_TCP_P2S_POD_NAME $SERVER_SERVICE_NAME tcp 1221 | else 1222 | info "Skipping P2S TCP test" 1223 | fi 1224 | 1225 | if [ "$CLIENT_UDP_P2S_POD_NAME" != "" ] 1226 | then 1227 | run-client $CLIENT_UDP_P2S_POD_NAME $SERVER_SERVICE_NAME udp 1228 | else 1229 | info "Skipping P2S UDP test" 1230 | fi 1231 | 1232 | #============================================================================== 1233 | # Output 1234 | #============================================================================== 1235 | 1236 | generate-report 1237 | $PLOT_DATA && plot-data 1238 | 1239 | #============================================================================== 1240 | # Cleaning 1241 | #============================================================================== 1242 | 1243 | info "Cleaning kubernetes resources ..." 1244 | cleanandexit 1245 | -------------------------------------------------------------------------------- /plotly-templates/bandwidth.jq: -------------------------------------------------------------------------------- 1 | { 2 | "data": [ 3 | { 4 | "meta": { 5 | "columnNames": { 6 | "x": "scenario", 7 | "y": "throughput" 8 | } 9 | }, 10 | "name": "throughput", 11 | "type": "bar", 12 | "x": [ 13 | if .data.pod2pod.tcp? then "pod2pod-tcp" else empty end, 14 | if .data.pod2pod.udp? then "pod2pod-udp" else empty end, 15 | if .data.pod2svc.tcp? then "pod2svc-tcp" else empty end, 16 | if .data.pod2svc.udp? then "pod2svc-udp" else empty end 17 | ], 18 | "y": [ 19 | if .data.pod2pod.tcp? then .data.pod2pod.tcp.bandwidth else empty end, 20 | if .data.pod2pod.udp? then .data.pod2pod.udp.bandwidth else empty end, 21 | if .data.pod2svc.tcp? then .data.pod2svc.tcp.bandwidth else empty end, 22 | if .data.pod2svc.udp? then .data.pod2svc.udp.bandwidth else empty end 23 | ], 24 | "marker": { 25 | "line": {}, 26 | "color": "rgb(49, 61, 172)" 27 | }, 28 | "text": [ 29 | if .data.pod2pod.tcp? then .data.pod2pod.tcp.bandwidth else empty end, 30 | if .data.pod2pod.udp? then .data.pod2pod.udp.bandwidth else empty end, 31 | if .data.pod2svc.tcp? then .data.pod2svc.tcp.bandwidth else empty end, 32 | if .data.pod2svc.udp? then .data.pod2svc.udp.bandwidth else empty end 33 | ], 34 | "showlegend": true, 35 | "legendgroup": 1, 36 | "textposition": "inside" 37 | } 38 | ], 39 | "layout": { 40 | "font": { 41 | "size": 12, 42 | "color": "rgb(33, 33, 33)", 43 | "family": "\"Droid Serif\", serif" 44 | }, 45 | "title": { 46 | "text": "K8S internal network bandwidth, Mbit/s" 47 | }, 48 | "xaxis": { 49 | "tickfont": { 50 | "size": 14, 51 | "family": "Droid Serif" 52 | }, 53 | "tickangle": -45 54 | }, 55 | "yaxis": { 56 | "tickfont": { 57 | "size": 14, 58 | "family": "Droid Serif" 59 | } 60 | } 61 | } 62 | } 63 | -------------------------------------------------------------------------------- /plotly-templates/cpu-usage.jq: -------------------------------------------------------------------------------- 1 | { 2 | "data": [ 3 | { 4 | "meta": { 5 | "columnNames": { 6 | "x": "scenario", 7 | "y": "client" 8 | } 9 | }, 10 | "name": "client", 11 | "type": "bar", 12 | "x": [ 13 | if .data.idle? then "Idle" else empty end, 14 | if .data.pod2pod.tcp? then "pod2pod-tcp" else empty end, 15 | if .data.pod2pod.udp? then "pod2pod-udp" else empty end, 16 | if .data.pod2svc.tcp? then "pod2svc-tcp" else empty end, 17 | if .data.pod2svc.udp? then "pod2svc-udp" else empty end 18 | ], 19 | "y": [ 20 | if .data.idle? then .data.idle.client.cpu.total else empty end, 21 | if .data.pod2pod.tcp? then .data.pod2pod.tcp.client.cpu.total else empty end, 22 | if .data.pod2pod.udp? then .data.pod2pod.udp.client.cpu.total else empty end, 23 | if .data.pod2svc.tcp? then .data.pod2svc.tcp.client.cpu.total else empty end, 24 | if .data.pod2svc.udp? then .data.pod2svc.udp.client.cpu.total else empty end 25 | ], 26 | "marker": { 27 | "color": "rgb(31, 119, 180)" 28 | }, 29 | "text": [ 30 | if .data.idle? then .data.idle.client.cpu.total else empty end, 31 | if .data.pod2pod.tcp? then .data.pod2pod.tcp.client.cpu.total else empty end, 32 | if .data.pod2pod.udp? then .data.pod2pod.udp.client.cpu.total else empty end, 33 | if .data.pod2svc.tcp? then .data.pod2svc.tcp.client.cpu.total else empty end, 34 | if .data.pod2svc.udp? then .data.pod2svc.udp.client.cpu.total else empty end 35 | ], 36 | "showlegend": true, 37 | "legendgroup": 1, 38 | "textposition": "inside" 39 | }, 40 | { 41 | "meta": { 42 | "columnNames": { 43 | "x": "scenario", 44 | "y": "server" 45 | } 46 | }, 47 | "name": "server", 48 | "type": "bar", 49 | "x": [ 50 | if .data.idle? then "Idle" else empty end, 51 | if .data.pod2pod.tcp? then "pod2pod-tcp" else empty end, 52 | if .data.pod2pod.udp? then "pod2pod-udp" else empty end, 53 | if .data.pod2svc.tcp? then "pod2svc-tcp" else empty end, 54 | if .data.pod2svc.udp? then "pod2svc-udp" else empty end 55 | ], 56 | "y": [ 57 | if .data.idle? then .data.idle.server.cpu.total else empty end, 58 | if .data.pod2pod.tcp? then .data.pod2pod.tcp.server.cpu.total else empty end, 59 | if .data.pod2pod.udp? then .data.pod2pod.udp.server.cpu.total else empty end, 60 | if .data.pod2svc.tcp? then .data.pod2svc.tcp.server.cpu.total else empty end, 61 | if .data.pod2svc.udp? then .data.pod2svc.udp.server.cpu.total else empty end 62 | ], 63 | "yaxis": "y", 64 | "marker": { 65 | "color": "rgb(214, 39, 40)" 66 | }, 67 | "text": [ 68 | if .data.idle? then .data.idle.server.cpu.total else empty end, 69 | if .data.pod2pod.tcp? then .data.pod2pod.tcp.server.cpu.total else empty end, 70 | if .data.pod2pod.udp? then .data.pod2pod.udp.server.cpu.total else empty end, 71 | if .data.pod2svc.tcp? then .data.pod2svc.tcp.server.cpu.total else empty end, 72 | if .data.pod2svc.udp? then .data.pod2svc.udp.server.cpu.total else empty end 73 | ], 74 | "showlegend": true, 75 | "legendgroup": 1, 76 | "textposition": "inside" 77 | } 78 | ], 79 | "layout": { 80 | "font": { 81 | "size": 12, 82 | "color": "rgb(33, 33, 33)", 83 | "family": "\"Droid Serif\", serif" 84 | }, 85 | "title": { 86 | "text": "CPU usage" 87 | }, 88 | "xaxis": { 89 | "tickfont": { 90 | "size": 14, 91 | "family": "Droid Serif" 92 | }, 93 | "tickangle": -45 94 | }, 95 | "yaxis": { 96 | "tickfont": { 97 | "size": 14, 98 | "family": "Droid Serif" 99 | } 100 | }, 101 | "barmode": "group", 102 | "boxmode": "overlay" 103 | } 104 | } 105 | -------------------------------------------------------------------------------- /plotly-templates/ram-usage.jq: -------------------------------------------------------------------------------- 1 | { 2 | "data": [ 3 | { 4 | "meta": { 5 | "columnNames": { 6 | "x": "scenario", 7 | "y": "client", 8 | "text": "client" 9 | } 10 | }, 11 | "name": "client", 12 | "type": "bar", 13 | "x": [ 14 | if .data.idle? then "Idle" else empty end, 15 | if .data.pod2pod.tcp? then "pod2pod-tcp" else empty end, 16 | if .data.pod2pod.udp? then "pod2pod-udp" else empty end, 17 | if .data.pod2svc.tcp? then "pod2svc-tcp" else empty end, 18 | if .data.pod2svc.udp? then "pod2svc-udp" else empty end 19 | ], 20 | "y": [ 21 | if .data.idle? then .data.idle.client.ram else empty end, 22 | if .data.pod2pod.tcp? then .data.pod2pod.tcp.client.ram else empty end, 23 | if .data.pod2pod.udp? then .data.pod2pod.udp.client.ram else empty end, 24 | if .data.pod2svc.tcp? then .data.pod2svc.tcp.client.ram else empty end, 25 | if .data.pod2svc.udp? then .data.pod2svc.udp.client.ram else empty end 26 | ], 27 | "marker": { 28 | "color": "rgb(147, 75, 232)" 29 | }, 30 | "text": [ 31 | if .data.idle? then .data.idle.client.ram else empty end, 32 | if .data.pod2pod.tcp? then .data.pod2pod.tcp.client.ram else empty end, 33 | if .data.pod2pod.udp? then .data.pod2pod.udp.client.ram else empty end, 34 | if .data.pod2svc.tcp? then .data.pod2svc.tcp.client.ram else empty end, 35 | if .data.pod2svc.udp? then .data.pod2svc.udp.client.ram else empty end 36 | ], 37 | "showlegend": true, 38 | "legendgroup": 1, 39 | "textposition": "inside" 40 | }, 41 | { 42 | "meta": { 43 | "columnNames": { 44 | "x": "scenario", 45 | "y": "server", 46 | "text": "server", 47 | "marker": { 48 | "color": "server" 49 | } 50 | } 51 | }, 52 | "name": "server", 53 | "type": "bar", 54 | "x": [ 55 | if .data.idle? then "Idle" else empty end, 56 | if .data.pod2pod.tcp? then "pod2pod-tcp" else empty end, 57 | if .data.pod2pod.udp? then "pod2pod-udp" else empty end, 58 | if .data.pod2svc.tcp? then "pod2svc-tcp" else empty end, 59 | if .data.pod2svc.udp? then "pod2svc-udp" else empty end 60 | ], 61 | "y": [ 62 | if .data.idle? then .data.idle.server.ram else empty end, 63 | if .data.pod2pod.tcp? then .data.pod2pod.tcp.server.ram else empty end, 64 | if .data.pod2pod.udp? then .data.pod2pod.udp.server.ram else empty end, 65 | if .data.pod2svc.tcp? then .data.pod2svc.tcp.server.ram else empty end, 66 | if .data.pod2svc.udp? then .data.pod2svc.udp.server.ram else empty end 67 | ], 68 | "yaxis": "y", 69 | "marker": { 70 | "meta": { 71 | "columnNames": { 72 | "color": "server" 73 | } 74 | }, 75 | "color": "rgb(252, 1, 149)" 76 | }, 77 | "text": [ 78 | if .data.idle? then .data.idle.server.ram else empty end, 79 | if .data.pod2pod.tcp? then .data.pod2pod.tcp.server.ram else empty end, 80 | if .data.pod2pod.udp? then .data.pod2pod.udp.server.ram else empty end, 81 | if .data.pod2svc.tcp? then .data.pod2svc.tcp.server.ram else empty end, 82 | if .data.pod2svc.udp? then .data.pod2svc.udp.server.ram else empty end 83 | ], 84 | "showlegend": true, 85 | "legendgroup": 1, 86 | "textposition": "inside" 87 | } 88 | ], 89 | "layout": { 90 | "font": { 91 | "size": 12, 92 | "color": "rgb(33, 33, 33)", 93 | "family": "\"Droid Serif\", serif" 94 | }, 95 | "title": { 96 | "text": "RAM usage, MB" 97 | }, 98 | "xaxis": { 99 | "tickfont": { 100 | "size": 14, 101 | "family": "Droid Serif" 102 | }, 103 | "tickangle": -45 104 | }, 105 | "yaxis": { 106 | "tickfont": { 107 | "size": 14, 108 | "family": "Droid Serif" 109 | } 110 | }, 111 | "barmode": "group", 112 | "boxmode": "overlay", 113 | "modebar": { 114 | "orientation": "h" 115 | } 116 | } 117 | } 118 | --------------------------------------------------------------------------------