├── .gitignore ├── Dockerfile ├── LICENSE ├── dbench.yaml ├── README.md └── docker-entrypoint.sh /.gitignore: -------------------------------------------------------------------------------- 1 | *.txt 2 | -------------------------------------------------------------------------------- /Dockerfile: -------------------------------------------------------------------------------- 1 | FROM dmonakhov/alpine-fio 2 | 3 | MAINTAINER Lee Liu 4 | 5 | VOLUME /tmp 6 | WORKDIR /tmp 7 | COPY ./docker-entrypoint.sh / 8 | ENTRYPOINT ["/docker-entrypoint.sh"] 9 | CMD ["fio"] 10 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2018 LogDNA 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /dbench.yaml: -------------------------------------------------------------------------------- 1 | kind: PersistentVolumeClaim 2 | apiVersion: v1 3 | metadata: 4 | name: dbench-pv-claim 5 | spec: 6 | storageClassName: ssd 7 | # storageClassName: gp2 8 | # storageClassName: local-storage 9 | # storageClassName: ibmc-block-bronze 10 | # storageClassName: ibmc-block-silver 11 | # storageClassName: ibmc-block-gold 12 | accessModes: 13 | - ReadWriteOnce 14 | resources: 15 | requests: 16 | storage: 1000Gi 17 | --- 18 | apiVersion: batch/v1 19 | kind: Job 20 | metadata: 21 | name: dbench 22 | spec: 23 | template: 24 | spec: 25 | containers: 26 | - name: dbench 27 | image: logdna/dbench:latest 28 | imagePullPolicy: Always 29 | env: 30 | - name: DBENCH_MOUNTPOINT 31 | value: /data 32 | # - name: DBENCH_QUICK 33 | # value: "yes" 34 | # - name: FIO_SIZE 35 | # value: 1G 36 | # - name: FIO_OFFSET_INCREMENT 37 | # value: 256M 38 | # - name: FIO_DIRECT 39 | # value: "0" 40 | volumeMounts: 41 | - name: dbench-pv 42 | mountPath: /data 43 | restartPolicy: Never 44 | volumes: 45 | - name: dbench-pv 46 | persistentVolumeClaim: 47 | claimName: dbench-pv-claim 48 | backoffLimit: 4 49 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # dbench 2 | Benchmark Kubernetes persistent disk volumes with `fio`: Read/write IOPS, bandwidth MB/s and latency. 3 | 4 | # Usage 5 | 6 | 1. Download [dbench.yaml](https://raw.githubusercontent.com/logdna/dbench/master/dbench.yaml) and edit the `storageClassName` to match your Kubernetes provider's Storage Class `kubectl get storageclasses` 7 | 2. Deploy Dbench using: `kubectl apply -f dbench.yaml` 8 | 3. Once deployed, the Dbench Job will: 9 | * provision a Persistent Volume of `1000Gi` (default) using `storageClassName: ssd` (default) 10 | * run a series of `fio` tests on the newly provisioned disk 11 | * currently there are 9 tests, 15s per test - total runtime is ~2.5 minutes 12 | 4. Follow benchmarking progress using: `kubectl logs -f job/dbench` (empty output means the Job not yet created, or `storageClassName` is invalid, see Troubleshooting below) 13 | 5. At the end of all tests, you'll see a summary that looks similar to this: 14 | ``` 15 | ================== 16 | = Dbench Summary = 17 | ================== 18 | Random Read/Write IOPS: 75.7k/59.7k. BW: 523MiB/s / 500MiB/s 19 | Average Latency (usec) Read/Write: 183.07/76.91 20 | Sequential Read/Write: 536MiB/s / 512MiB/s 21 | Mixed Random Read/Write IOPS: 43.1k/14.4k 22 | ``` 23 | 6. Once the tests are finished, clean up using: `kubectl delete -f dbench.yaml` and that should deprovision the persistent disk and delete it to minimize storage billing. 24 | 25 | ## Notes / Troubleshooting 26 | 27 | * If the Persistent Volume Claim is stuck on Pending, it's likely you didn't specify a valid Storage Class. Double check using `kubectl get storageclasses`. Also check that the volume size of `1000Gi` (default) is available for provisioning. 28 | * It can take some time for a Persistent Volume to be Bound and the Kubernetes Dashboard UI will show the Dbench Job as red until the volume is finished provisioning. 29 | * It's useful to test multiple disk sizes as most cloud providers price IOPS per GB provisioned. So a `4000Gi` volume will perform better than a `1000Gi` volume. Just edit the yaml, `kubectl delete -f dbench.yaml` and run `kubectl apply -f dbench.yaml` again after deprovision/delete completes. 30 | * A list of all `fio` tests are in [docker-entrypoint.sh](https://github.com/logdna/dbench/blob/master/docker-entrypoint.sh). 31 | 32 | ## Contributors 33 | 34 | * Lee Liu (LogDNA) 35 | * [Alexis Turpin](https://github.com/alexis-turpin) 36 | 37 | ## License 38 | 39 | * MIT 40 | -------------------------------------------------------------------------------- /docker-entrypoint.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env sh 2 | set -e 3 | 4 | if [ -z $DBENCH_MOUNTPOINT ]; then 5 | DBENCH_MOUNTPOINT=/tmp 6 | fi 7 | 8 | if [ -z $FIO_SIZE ]; then 9 | FIO_SIZE=2G 10 | fi 11 | 12 | if [ -z $FIO_OFFSET_INCREMENT ]; then 13 | FIO_OFFSET_INCREMENT=500M 14 | fi 15 | 16 | if [ -z $FIO_DIRECT ]; then 17 | FIO_DIRECT=1 18 | fi 19 | 20 | echo Working dir: $DBENCH_MOUNTPOINT 21 | echo 22 | 23 | if [ "$1" = 'fio' ]; then 24 | 25 | echo Testing Read IOPS... 26 | READ_IOPS=$(fio --randrepeat=0 --verify=0 --ioengine=libaio --direct=$FIO_DIRECT --gtod_reduce=1 --name=read_iops --filename=$DBENCH_MOUNTPOINT/fiotest --bs=4K --iodepth=64 --size=$FIO_SIZE --readwrite=randread --time_based --ramp_time=2s --runtime=15s) 27 | echo "$READ_IOPS" 28 | READ_IOPS_VAL=$(echo "$READ_IOPS"|grep -E 'read ?:'|grep -Eoi 'IOPS=[0-9k.]+'|cut -d'=' -f2) 29 | echo 30 | echo 31 | 32 | echo Testing Write IOPS... 33 | WRITE_IOPS=$(fio --randrepeat=0 --verify=0 --ioengine=libaio --direct=$FIO_DIRECT --gtod_reduce=1 --name=write_iops --filename=$DBENCH_MOUNTPOINT/fiotest --bs=4K --iodepth=64 --size=$FIO_SIZE --readwrite=randwrite --time_based --ramp_time=2s --runtime=15s) 34 | echo "$WRITE_IOPS" 35 | WRITE_IOPS_VAL=$(echo "$WRITE_IOPS"|grep -E 'write:'|grep -Eoi 'IOPS=[0-9k.]+'|cut -d'=' -f2) 36 | echo 37 | echo 38 | 39 | echo Testing Read Bandwidth... 40 | READ_BW=$(fio --randrepeat=0 --verify=0 --ioengine=libaio --direct=$FIO_DIRECT --gtod_reduce=1 --name=read_bw --filename=$DBENCH_MOUNTPOINT/fiotest --bs=128K --iodepth=64 --size=$FIO_SIZE --readwrite=randread --time_based --ramp_time=2s --runtime=15s) 41 | echo "$READ_BW" 42 | READ_BW_VAL=$(echo "$READ_BW"|grep -E 'read ?:'|grep -Eoi 'BW=[0-9GMKiBs/.]+'|cut -d'=' -f2) 43 | echo 44 | echo 45 | 46 | echo Testing Write Bandwidth... 47 | WRITE_BW=$(fio --randrepeat=0 --verify=0 --ioengine=libaio --direct=$FIO_DIRECT --gtod_reduce=1 --name=write_bw --filename=$DBENCH_MOUNTPOINT/fiotest --bs=128K --iodepth=64 --size=$FIO_SIZE --readwrite=randwrite --time_based --ramp_time=2s --runtime=15s) 48 | echo "$WRITE_BW" 49 | WRITE_BW_VAL=$(echo "$WRITE_BW"|grep -E 'write:'|grep -Eoi 'BW=[0-9GMKiBs/.]+'|cut -d'=' -f2) 50 | echo 51 | echo 52 | 53 | if [ "$DBENCH_QUICK" == "" ] || [ "$DBENCH_QUICK" == "no" ]; then 54 | echo Testing Read Latency... 55 | READ_LATENCY=$(fio --randrepeat=0 --verify=0 --ioengine=libaio --direct=$FIO_DIRECT --name=read_latency --filename=$DBENCH_MOUNTPOINT/fiotest --bs=4K --iodepth=4 --size=$FIO_SIZE --readwrite=randread --time_based --ramp_time=2s --runtime=15s) 56 | echo "$READ_LATENCY" 57 | READ_LATENCY_VAL=$(echo "$READ_LATENCY"|grep ' lat.*avg'|grep -Eoi 'avg=[0-9.]+'|cut -d'=' -f2) 58 | echo 59 | echo 60 | 61 | echo Testing Write Latency... 62 | WRITE_LATENCY=$(fio --randrepeat=0 --verify=0 --ioengine=libaio --direct=$FIO_DIRECT --name=write_latency --filename=$DBENCH_MOUNTPOINT/fiotest --bs=4K --iodepth=4 --size=$FIO_SIZE --readwrite=randwrite --time_based --ramp_time=2s --runtime=15s) 63 | echo "$WRITE_LATENCY" 64 | WRITE_LATENCY_VAL=$(echo "$WRITE_LATENCY"|grep ' lat.*avg'|grep -Eoi 'avg=[0-9.]+'|cut -d'=' -f2) 65 | echo 66 | echo 67 | 68 | echo Testing Read Sequential Speed... 69 | READ_SEQ=$(fio --randrepeat=0 --verify=0 --ioengine=libaio --direct=$FIO_DIRECT --gtod_reduce=1 --name=read_seq --filename=$DBENCH_MOUNTPOINT/fiotest --bs=1M --iodepth=16 --size=$FIO_SIZE --readwrite=read --time_based --ramp_time=2s --runtime=15s --thread --numjobs=4 --offset_increment=$FIO_OFFSET_INCREMENT) 70 | echo "$READ_SEQ" 71 | READ_SEQ_VAL=$(echo "$READ_SEQ"|grep -E 'READ:'|grep -Eoi '(aggrb|bw)=[0-9GMKiBs/.]+'|cut -d'=' -f2) 72 | echo 73 | echo 74 | 75 | echo Testing Write Sequential Speed... 76 | WRITE_SEQ=$(fio --randrepeat=0 --verify=0 --ioengine=libaio --direct=$FIO_DIRECT --gtod_reduce=1 --name=write_seq --filename=$DBENCH_MOUNTPOINT/fiotest --bs=1M --iodepth=16 --size=$FIO_SIZE --readwrite=write --time_based --ramp_time=2s --runtime=15s --thread --numjobs=4 --offset_increment=$FIO_OFFSET_INCREMENT) 77 | echo "$WRITE_SEQ" 78 | WRITE_SEQ_VAL=$(echo "$WRITE_SEQ"|grep -E 'WRITE:'|grep -Eoi '(aggrb|bw)=[0-9GMKiBs/.]+'|cut -d'=' -f2) 79 | echo 80 | echo 81 | 82 | echo Testing Read/Write Mixed... 83 | RW_MIX=$(fio --randrepeat=0 --verify=0 --ioengine=libaio --direct=$FIO_DIRECT --gtod_reduce=1 --name=rw_mix --filename=$DBENCH_MOUNTPOINT/fiotest --bs=4k --iodepth=64 --size=$FIO_SIZE --readwrite=randrw --rwmixread=75 --time_based --ramp_time=2s --runtime=15s) 84 | echo "$RW_MIX" 85 | RW_MIX_R_IOPS=$(echo "$RW_MIX"|grep -E 'read ?:'|grep -Eoi 'IOPS=[0-9k.]+'|cut -d'=' -f2) 86 | RW_MIX_W_IOPS=$(echo "$RW_MIX"|grep -E 'write:'|grep -Eoi 'IOPS=[0-9k.]+'|cut -d'=' -f2) 87 | echo 88 | echo 89 | fi 90 | 91 | echo All tests complete. 92 | echo 93 | echo ================== 94 | echo = Dbench Summary = 95 | echo ================== 96 | echo "Random Read/Write IOPS: $READ_IOPS_VAL/$WRITE_IOPS_VAL. BW: $READ_BW_VAL / $WRITE_BW_VAL" 97 | if [ -z $DBENCH_QUICK ] || [ "$DBENCH_QUICK" == "no" ]; then 98 | echo "Average Latency (usec) Read/Write: $READ_LATENCY_VAL/$WRITE_LATENCY_VAL" 99 | echo "Sequential Read/Write: $READ_SEQ_VAL / $WRITE_SEQ_VAL" 100 | echo "Mixed Random Read/Write IOPS: $RW_MIX_R_IOPS/$RW_MIX_W_IOPS" 101 | fi 102 | 103 | rm $DBENCH_MOUNTPOINT/fiotest 104 | exit 0 105 | fi 106 | 107 | exec "$@" 108 | --------------------------------------------------------------------------------