├── doc ├── docker-killer-cpubomb.gif ├── docker-killer-membomb.gif ├── docker-killer-netbomb.gif └── dockerize-all-the-things.jpg ├── docker-image-files ├── forkbomb.c ├── membomb.c └── test.sh ├── Dockerfile ├── LICENSE └── README.md /doc/docker-killer-cpubomb.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/monitoringartist/docker-killer/master/doc/docker-killer-cpubomb.gif -------------------------------------------------------------------------------- /doc/docker-killer-membomb.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/monitoringartist/docker-killer/master/doc/docker-killer-membomb.gif -------------------------------------------------------------------------------- /doc/docker-killer-netbomb.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/monitoringartist/docker-killer/master/doc/docker-killer-netbomb.gif -------------------------------------------------------------------------------- /doc/dockerize-all-the-things.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/monitoringartist/docker-killer/master/doc/dockerize-all-the-things.jpg -------------------------------------------------------------------------------- /docker-image-files/forkbomb.c: -------------------------------------------------------------------------------- 1 | #include 2 | 3 | int main(void) { 4 | for(;;) 5 | fork(); 6 | return 0; 7 | } 8 | -------------------------------------------------------------------------------- /Dockerfile: -------------------------------------------------------------------------------- 1 | FROM alpine:latest 2 | MAINTAINER Jan Garaj info@monitoringartist.com 3 | 4 | COPY docker-image-files / 5 | 6 | RUN \ 7 | chmod +x /test.sh && \ 8 | apk --update add bash iperf gcc musl-dev && \ 9 | gcc membomb.c -o /membomb && \ 10 | gcc forkbomb.c -o /forkbomb && \ 11 | apk del gcc musl-dev&& \ 12 | rm -rf /var/cache/apk/* 13 | 14 | EXPOSE 5001 15 | CMD ["help"] 16 | ENTRYPOINT ["/test.sh"] 17 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2017 Monitoring Artist 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /docker-image-files/membomb.c: -------------------------------------------------------------------------------- 1 | #define _GNU_SOURCE 2 | 3 | #include 4 | #include 5 | #include 6 | #include 7 | #include 8 | 9 | 10 | int main(int argc, char* argv[]) { 11 | 12 | /* 13 | set the real, effective and set user id to root, 14 | so that the process can adjust possible limits. 15 | if the process doesn't have the CAP_SETUID capability, terminate the process. 16 | */ 17 | if (setresuid(0, 0, 0) == -1) { 18 | printf("Are you root?!\n"); 19 | return 1; 20 | } 21 | 22 | /* 23 | block all signals except for kill and stop. 24 | this allows to terminate the parent process (most likely a terminal) 25 | that this process is running in and turn it into a daemon. 26 | additionally this makes it impossible to terminate the process 27 | in a normal way and therefore satisfies the requirement that closing 28 | it should still make it hog memory. 29 | */ 30 | sigset_t mask; 31 | sigfillset(&mask); 32 | sigprocmask(SIG_SETMASK, &mask, NULL); 33 | 34 | /* 35 | allow the process to acquire a virtually unlimited amount of memory 36 | and queue a virtually unlimited amount of signals. 37 | this is to prevent an out of memory error due to a virtual limit for the root user, 38 | which would prevent the process from leaking any more memory 39 | and to prevent the process from getting killed due to too many queued 40 | signals that the process is blocking. 41 | */ 42 | struct rlimit memory = { RLIM_INFINITY, RLIM_INFINITY }, 43 | signal = { RLIM_INFINITY, RLIM_INFINITY}; 44 | setrlimit(RLIMIT_AS, &memory); 45 | setrlimit(RLIMIT_SIGPENDING, &signal); 46 | 47 | /* 48 | allocate a buffer big enough to store a file name into it 49 | that is generated from the process' pid. 50 | if the file can be opened (which should always be the case unless /proc is not mounted) 51 | the file will be opened and the string -17 followed by a new line written to it. 52 | this will cause the oom killer to ignore our process and only kill other, 53 | innocent processes when running out of memory. 54 | */ 55 | char file_name[20]; 56 | sprintf(file_name, "/proc/%u/oom_adj", getpid()); 57 | 58 | FILE* oom_killer_file = fopen(file_name, "w"); 59 | if (oom_killer_file) { 60 | fprintf(oom_killer_file, "-17\n"); 61 | fclose(oom_killer_file); 62 | } 63 | 64 | /* 65 | get the size of virtual memory pages in bytes, 66 | so the process knows the size of chunks that have to be 67 | made dirty to force the kernel to map the virtual memory page into RAM. 68 | */ 69 | long page_size = sysconf(_SC_PAGESIZE); 70 | 71 | // allocate a virtually infinite amount of memory by chunks of a page size. 72 | while(1) { 73 | // will overwrite any previous stored address in tmp, leaking that memory. 74 | char* tmp = (char*) malloc(page_size); 75 | if (tmp) 76 | // make the memory page dirty to force the kernel to map it into RAM. 77 | tmp[0] = 0; 78 | } 79 | 80 | return 0; 81 | } -------------------------------------------------------------------------------- /docker-image-files/test.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # default timeout is 60sec 4 | export TIMEOUT=${TIMEOUT:-60} 5 | TIMEOUT2=$((TIMEOUT - 1)) 6 | export NETBOMB=${NETBOMB:-"iperf -c iperf.scottlinux.com -t ${TIMEOUT2} -i 1 -p 5201 -u"} 7 | 8 | export green='\e[0;32m' 9 | export yellow='\e[0;33m' 10 | export red='\e[0;31m' 11 | export blue='\e[0;34m' 12 | export endColor='\e[0m' 13 | 14 | help() { 15 | echo -e "${red}WARNING: IT IS NOT GUARANTEED THAT YOUR SYSTEM/CONTAINERS WILL SURVIVE THIS KILLER TESTING! DO NOT USE THIS IMAGE UNLESS YOU REALLY KNOW WHAT ARE YOU DOING!${endColor}" 16 | echo -e "${yellow}Tests included in this image: cpubomb, membomb, netbomb, forkbomb, die, kernelpanic ...${endColor}" 17 | echo -e "${yellow}Use 'all' or name of the particular test, for example:${endColor}" 18 | echo -e "${yellow}docker run --rm -ti --privileged -v /:/rootfs --oom-kill-disable monitoringartist/docker-killer cpubomb${endColor}" 19 | } 20 | export -f help 21 | 22 | forkbomb() { 23 | echo -e "${red}forkbomb - duration ${TIMEOUT}s${endColor}" 24 | echo -e "${yellow}Test: excessive number of forks${endColor}" 25 | # TODO display stat 26 | #while true; do { grep procs_running /proc/stat; sleep 1; } & done 27 | /forkbomb 28 | } 29 | export -f forkbomb 30 | 31 | cpubomb() { 32 | echo -e "${red}cpubomb - duration ${TIMEOUT}s${endColor}" 33 | echo -e "${yellow}Test: excessive CPU utilization - one proces per processor with empty cycles${endColor}" 34 | top -b -n${TIMEOUT} -d1 | grep "^CPU:" & 35 | #top -b -n${TIMEOUT} -d1 | grep "^Load average:" & 36 | ( 37 | pids="" 38 | cpus=$(getconf _NPROCESSORS_ONLN) 39 | trap 'for p in $pids; do kill $p; done' 0 40 | for ((i=0;i/proc/sysrq-trigger 87 | } 88 | export -f kernelpanic 89 | 90 | if [ "$1" == "all" ]; then 91 | timeout -t ${TIMEOUT} -s KILL bash -c cpubomb 92 | echo -e "${yellow}----------------------------${endColor}\n" 93 | timeout -t ${TIMEOUT} -s KILL bash -c netbomb 94 | echo -e "${yellow}----------------------------${endColor}\n" 95 | timeout -t ${TIMEOUT} -s KILL bash -c forkbomb 96 | echo -e "${yellow}----------------------------${endColor}\n" 97 | timeout -t ${TIMEOUT} -s KILL bash -c membomb 98 | echo -e "${yellow}----------------------------${endColor}" 99 | else 100 | timeout -t ${TIMEOUT} -s KILL bash -c $@ 101 | fi 102 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Docker Killer 2 | 3 | **WARNING: IT IS NOT GUARANTEED THAT YOUR SYSTEM/CONTAINERS WILL SURVIVE THIS KILLER TESTING! DO NOT USE THIS IMAGE UNLESS YOU REALLY KNOW WHAT ARE YOU DOING!** 4 | 5 | ![Dockerize ak the things](https://raw.githubusercontent.com/monitoringartist/docker-killer/master/doc/dockerize-all-the-things.jpg) 6 | 7 | There is a million of articles how Docker containers are awesome and only a few articles about Docker problems in enterprise production. Hopefully this image will help you to understand problems of Docker in production, which I have discovered from my 2 years commercial Docker experience. Feel free to add more usefull test cases. And again: 8 | 9 | **WARNING: IT IS NOT GUARANTEED THAT YOUR SYSTEM/CONTAINERS WILL SURVIVE THIS KILLER TESTING! DO NOT USE THIS IMAGE UNLESS YOU REALLY KNOW WHAT ARE YOU DOING!** 10 | 11 | All tests with default 60s timeout per test (except die/kernel panic test): 12 | 13 | ``` 14 | docker run \ 15 | --rm -ti \ 16 | --privileged \ 17 | -v /:/rootfs \ 18 | --oom-kill-disable \ 19 | monitoringartist/docker-killer \ 20 | all 21 | ``` 22 | 23 | # Particular tests 24 | 25 | ## cpubomb 26 | 27 | It will 100% utilize all CPUs. 28 | 29 | ``` 30 | docker run \ 31 | --rm -ti \ 32 | --privileged \ 33 | -v /:/rootfs \ 34 | --oom-kill-disable \ 35 | monitoringartist/docker-killer \ 36 | cpubomb 37 | ``` 38 | ![docker-kill cpubomb](https://raw.githubusercontent.com/monitoringartist/docker-killer/master/doc/docker-killer-cpubomb.gif) 39 | 40 | Solution: Use [cgroup CPU subsytem limitation](https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Resource_Management_Guide/sec-cpu.html). 41 | 42 | ## membomb 43 | 44 | It will exhaust host memory and swap space. 45 | 46 | ``` 47 | docker run \ 48 | --rm -ti \ 49 | --privileged \ 50 | -v /:/rootfs \ 51 | --oom-kill-disable \ 52 | monitoringartist/docker-killer \ 53 | membomb 54 | ``` 55 | 56 | ![docker-kill membomb](https://raw.githubusercontent.com/monitoringartist/docker-killer/master/doc/docker-killer-membomb.gif) 57 | 58 | Solution: Don't use `--oom-kill-disable` unless it's neccessary and use [cgroup memory subsystem](https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Resource_Management_Guide/sec-memory.html). 59 | 60 | ## netbomb 61 | 62 | It will utilize your internet connectivity by using UDP stream. 63 | 64 | ``` 65 | docker run \ 66 | --rm -ti \ 67 | --privileged \ 68 | -v /:/rootfs \ 69 | --oom-kill-disable \ 70 | monitoringartist/docker-killer \ 71 | netbomb 72 | ``` 73 | 74 | ![docker-kill netbomb](https://raw.githubusercontent.com/monitoringartist/docker-killer/master/doc/docker-killer-netbomb.gif) 75 | 76 | Solution: Use OS network limitations (shapping, ....) or [cgroup network priority (net_prio) subsystem](https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Resource_Management_Guide/net_prio.html). If you need the best performance, don't use default Docker userland proxy. Try to use host network or some custom network solution. 77 | 78 | 79 | ## forkbomb 80 | 81 | It will execute number of forks. 82 | 83 | ``` 84 | docker run \ 85 | --rm -ti \ 86 | --privileged \ 87 | -v /:/rootfs \ 88 | --oom-kill-disable \ 89 | monitoringartist/docker-killer \ 90 | forkbomb 91 | ``` 92 | 93 | Solution: Use your standard OS number of processes limitations: number of processes per user and use custom user with defined limit in the container. 94 | 95 | ## kernelpanic 96 | 97 | It will generate kernel panic. 98 | 99 | ``` 100 | docker run \ 101 | --rm -ti \ 102 | --privileged \ 103 | -v /:/rootfs \ 104 | --oom-kill-disable \ 105 | monitoringartist/docker-killer \ 106 | kernelpanic 107 | ``` 108 | 109 | Solution: Don't run any Docker images, which can cause kernel panic. 110 | 111 | ## die 112 | 113 | Exit container with exit code 1. 114 | 115 | ``` 116 | docker run \ 117 | --rm -ti \ 118 | --privileged \ 119 | -v /:/rootfs \ 120 | --oom-kill-disable \ 121 | monitoringartist/docker-killer \ 122 | die 123 | ``` 124 | 125 | Test your Docker orchestration of daemonized containers. Your solution should (re)start container automatically. Some people are using supervisor in the container, but it's adding another process into container. The best approach, which I have seen is systemd, which handle Docker container lifecycle. Your container is your system(d) service, which you are able to start/stop/restart by using standard systemd utilities. Recommened Puppet module - https://github.com/garethr/garethr-docker 126 | 127 | # Environment variables 128 | 129 | | Environment variable | Description | 130 | | -------------------- | ----------- | 131 | | TIMEOUT | Test duration - default 60 sec per test | 132 | | NETBOMB | Default command for netbomb test: iperf -c iperf.scottlinux.com -i 1 -p 5201 -u -t $TIMEOUT, you may to use another public iperf server https://iperf.fr/iperf-servers.php | 133 | 134 | # TODO 135 | 136 | * chaosmonkey - stop random container 137 | 138 | # General Docker recommendations 139 | 140 | * Trust only to your own well tested Docker images from your own repo. Public Docker hub images can be created by (choose here your favorite hacker flavour: Russian, American, Slovak, black/white hat, ...) hackers. 141 | * Use explicit tag, instead of implicit default tag. Otherwise your containers can be upgraded unexpectedly. 142 | * Monitoring - monitor high level of the service provided by container. Container should expose healthcheck (HTTP) endpoint, which will be monitored for any unexepected status codes/strings. 143 | 144 | # Recommended documentation 145 | 146 | - https://docs.docker.com/engine/reference/run/ 147 | - https://www.kernel.org/doc/Documentation/cgroups/memory.txt 148 | - https://www.kernel.org/doc/Documentation/cgroups/cpuacct.txt 149 | - https://www.kernel.org/doc/Documentation/cgroups/blkio-controller.txt 150 | - https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Resource_Management_Guide/index.html 151 | - https://goldmann.pl/blog/2014/09/11/resource-management-in-docker/ 152 | 153 | # Inspiration 154 | 155 | [Hitler uses Docker](https://www.youtube.com/watch?v=PivpCKEiQOQ) - overview of Docker in production problems. If you are familiar with Docker problems, then it's funny video. 156 | 157 | # Author 158 | 159 | [Devops Monitoring Expert](http://www.jangaraj.com 'DevOps / Docker / Kubernetes / AWS ECS / Google GCP / Zabbix / Zenoss / Terraform / Monitoring'), 160 | who loves monitoring systems and cutting/bleeding edge technologies: Docker, 161 | Kubernetes, ECS, AWS, Google GCP, Terraform, Lambda, Zabbix, Grafana, Elasticsearch, 162 | Kibana, Prometheus, Sysdig, ... 163 | 164 | Summary: 165 | * 1000+ [GitHub](https://github.com/monitoringartist/) stars 166 | * 6000+ [Grafana dashboard](https://grafana.net/monitoringartist) downloads 167 | * 800 000+ [Docker image](https://hub.docker.com/u/monitoringartist/) pulls 168 | 169 | Professional devops / monitoring / consulting services: 170 | 171 | [![Monitoring Artist](http://monitoringartist.com/img/github-monitoring-artist-logo.jpg)](http://www.monitoringartist.com 'DevOps / Docker / Kubernetes / AWS ECS / Google GCP / Zabbix / Zenoss / Terraform / Monitoring') 172 | --------------------------------------------------------------------------------