├── .gitignore ├── .travis.yml ├── README.md ├── Vagrantfile ├── images ├── htop-secauditengine-off.png └── htop-secauditengine-relevantonly.png ├── minion ├── pillars ├── top.sls └── versions.sls └── states ├── benchenv.sls ├── build.sls ├── debugenv.sls ├── files ├── Makefile.tmpl ├── batchbench.sh ├── batchperfrun.sh ├── etc │ └── nginx │ │ ├── conf.d │ │ └── grpc-proxy.conf │ │ ├── modsec │ │ └── main.conf │ │ └── nginx.conf ├── gencert.sh ├── grpc_data_transmission │ ├── client.py │ ├── demo.proto │ ├── demo_pb2.py │ ├── demo_pb2_grpc.py │ └── server.py ├── perfrun.sh └── upload-app │ ├── upload-app.ini │ ├── upload-app.py │ └── upload-app.service ├── grpc.sls ├── makeenv.sls ├── nginx.sls ├── owasp-crs.sls ├── system.sls ├── top.sls ├── upload-app.sls └── user.sls /.gitignore: -------------------------------------------------------------------------------- 1 | .vagrant 2 | *.log 3 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | language: python 2 | python: 3 | - '2.7' 4 | 5 | before_install: 6 | - sudo apt-get update 7 | - curl -L http://bootstrap.saltstack.org | sudo sh -s -- stable 8 | 9 | install: 10 | - sudo ln -s `pwd`/states /srv/salt 11 | - sudo ln -s `pwd`/pillars /srv/pillar 12 | - sudo cp minion /etc/salt/minion 13 | - sudo service salt-minion restart 14 | - sudo cat /var/log/salt/* 15 | - sudo salt-call grains.items --local 16 | 17 | script: 18 | - sudo salt-call state.show_highstate --local 19 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # About 2 | 3 | This repository contains a number of configurations (represented 4 | in SaltStack states and Vagrantfile descriptions) that can be used 5 | to test performance of 6 | [ModSecurity](https://github.com/SpiderLabs/ModSecurity) 7 | with various connectors, primarily 8 | [ModSecurity-nginx](https://github.com/SpiderLabs/ModSecurity-nginx). 9 | 10 | ## Prerequisites 11 | 12 | * [Vagrant](https://www.vagrantup.com/) 13 | * your favorite virtualization plug-in for Vagrant 14 | 15 | [VirtualBox](https://www.virtualbox.org/) 16 | is known to work on MacOS/Linux-based laptops, while 17 | [KVM/libvirt](https://github.com/vagrant-libvirt/vagrant-libvirt) 18 | is probably the best choice for servers. 19 | 20 | ## How to use 21 | 22 | 1. Adjust 23 | [pillars/versions.sls](https://github.com/defanator/modsecurity-performance/blob/master/pillars/versions.sls) 24 | if you want to build some custom versions/revisions/branches 25 | of either ModSecurity or ModSecurity-nginx. 26 | 27 | 2. Prepare VM (could take some time as this step includes 28 | compilation of all the prerequisites required for testing: 29 | libmodsecurity and ModSecurity-nginx connector module): 30 | 31 | ``` 32 | vagrant up 33 | ``` 34 | 35 | For the reference: 36 | 37 | * on libvirt-based 12-core VM (backed by bare-metal server with 24-core 38 | Xeon E5645 2.4Ghz) provisioning takes about 7 minutes 39 | * on VirtualBox-based 2-core VM (backed by early 2015 MBP A1502 2-core 40 | i5 2.9GHz) provisioning takes about 8.5 minutes 41 | 42 | 3. Log in into the VM: 43 | 44 | ``` 45 | vagrant ssh 46 | ``` 47 | 48 | 4. Run a set of performance tests and get a summary: 49 | 50 | ``` 51 | vagrant@vagrant:~$ sudo su -l test 52 | test@vagrant:~$ ./perfrun.sh run 53 | test@vagrant:~$ ./perfrun.sh stats 54 | ``` 55 | 56 | ## What is being tested 57 | 58 | Currently three locations are being benchmarked on locally configured 59 | nginx instance: 60 | 61 | * `/modsec-off/` - proxies all requests to local server with no additional 62 | processing 63 | 64 | * `/modsec-light/` - proxies all requests to local server with libmodsecurity 65 | turned on, but without any actual rules 66 | 67 | * `/modsec-full/` - proxies all requests to local server with libmodsecurity 68 | turned on and full OWASP CRS v3 loaded 69 | 70 | Please refer to the [nginx.conf](https://github.com/defanator/modsecurity-performance/blob/master/states/files/etc/nginx/nginx.conf) 71 | for the details. 72 | 73 | ## Batch benchmarking 74 | 75 | If you want to run benchmark for a particular subset of libmodsecurity 76 | changesets, this can be done in a following way: 77 | 78 | test@vagrant:~$ cat batchbench.revs 79 | 10c4f9b1b2476f71159fa5569d9238001760404c 80 | 9e9db08b874fe7c1200aafd95fe6bccd41148ae5 81 | fa7973a4ef99b0d91122d16ffee51744288d037f 82 | 2988c5bb07c4a5ad434855413f20fec11008c818 83 | 63bef3d142b2ae25ed42d344c40729fb5f3d552e 84 | d9d702f401c870bf399d8cd5bc4ae212c7d52195 85 | 86 | test@vagrant:~$ ./batchbench.sh run 87 | [..] 88 | 89 | test@vagrant:~$ ./batchbench.sh stats 90 | ;rps_avg,latency_avg,workers_utime_avg,revision,date,commit_log 91 | 530.57,787.49,17869.33,10c4f9b1b2476f71159fa5569d9238001760404c,2017-08-19 10:21:57 +0300,add a test for macro expansion in @rx 92 | 533.27,719.25,17855.33,9e9db08b874fe7c1200aafd95fe6bccd41148ae5,2017-08-19 11:16:54 +0300,add @rx macro expansion test to list in Makefile 93 | 29.81,1562.69,17968.00,fa7973a4ef99b0d91122d16ffee51744288d037f,2017-10-06 20:32:40 +0000,Removes a regex optimization added at #1536 94 | 28.26,1528.49,17946.33,2988c5bb07c4a5ad434855413f20fec11008c818,2017-10-06 20:35:09 +0000,CHANGES: add info about #1536 95 | 28.64,1495.39,17951.00,63bef3d142b2ae25ed42d344c40729fb5f3d552e,2017-10-03 20:50:02 +0000,Support to JSON stuff on serial logging 96 | 633.89,680.80,17829.33,d9d702f401c870bf399d8cd5bc4ae212c7d52195,2018-01-03 09:49:20 -0300,Fix the debuglogs for the regression tests 97 | 98 | Build logs and raw wrk output will be in the `batch/` directory. 99 | Please note that `batchbench.sh` uses separate script for launching wrk - 100 | [batchperfrun.sh](https://github.com/defanator/modsecurity-performance/blob/master/states/files/batchperfrun.sh) 101 | (e.g. it uses extended request with additional headers, it only tests `/modsec-full` location, 102 | and finally it uses more threads/connections for wrk). 103 | 104 | ## Important notes 105 | 106 | Please adjust nginx configuration and wrk parameters according to your environment and available resources. 107 | Default values (like `worker_processes 1` in nginx.conf) most likely won't meet your expectations in some scenarios. 108 | 109 | ## Sample results 110 | 111 | Available on [wiki](https://github.com/defanator/modsecurity-performance/wiki). 112 | -------------------------------------------------------------------------------- /Vagrantfile: -------------------------------------------------------------------------------- 1 | # -*- mode: ruby -*- 2 | # vi: set ft=ruby : 3 | 4 | Vagrant.configure("2") do |config| 5 | host = RbConfig::CONFIG['host_os'] 6 | cpus = [2, `getconf _NPROCESSORS_ONLN`.to_i / 2].max 7 | 8 | # provide more resources to vm 9 | if host =~ /darwin/ 10 | mem = [4096, `sysctl -n hw.memsize`.to_i / 1024 / 1024 / 2].min 11 | elsif host =~ /linux/ 12 | mem = [cpus * 512, `awk '/MemTotal/ {print $2}' /proc/meminfo`.to_i / 1024 / 2].min 13 | end 14 | 15 | config.vm.box = "generic/ubuntu2004" 16 | config.vm.hostname = "vagrant" 17 | 18 | # use NFS for performant file sharing 19 | config.vm.network :private_network, type: "dhcp" 20 | 21 | # mount states and pillars to the appropriate locations 22 | config.vm.synced_folder "states/", "/srv/salt/" 23 | config.vm.synced_folder "pillars/", "/srv/pillar/" 24 | 25 | # provider specific settings 26 | config.vm.provider "virtualbox" do |vbox, override| 27 | vbox.cpus = cpus 28 | vbox.memory = mem 29 | end 30 | 31 | config.vm.provider :libvirt do |libvirt, override| 32 | libvirt.cpus = cpus 33 | libvirt.memory = mem 34 | libvirt.driver = "kvm" 35 | libvirt.cpu_mode = "host-passthrough" 36 | end 37 | 38 | config.vm.provision :salt do |salt| 39 | salt.minion_config = "minion" 40 | salt.install_type = "stable" 41 | salt.bootstrap_options = "-n -X" 42 | salt.masterless = true 43 | salt.run_highstate = true 44 | end 45 | 46 | config.vm.define "default", primary: true do |default| 47 | end 48 | 49 | config.vm.define "debian10", autostart: false do |debian10| 50 | debian10.vm.box = "debian/buster64" 51 | end 52 | end 53 | -------------------------------------------------------------------------------- /images/htop-secauditengine-off.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/defanator/modsecurity-performance/c066fad16444932fdb7dd8636aa298dd7704ac5d/images/htop-secauditengine-off.png -------------------------------------------------------------------------------- /images/htop-secauditengine-relevantonly.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/defanator/modsecurity-performance/c066fad16444932fdb7dd8636aa298dd7704ac5d/images/htop-secauditengine-relevantonly.png -------------------------------------------------------------------------------- /minion: -------------------------------------------------------------------------------- 1 | master_type: disable 2 | file_client: local 3 | state_output: changes 4 | state_verbose: False 5 | -------------------------------------------------------------------------------- /pillars/top.sls: -------------------------------------------------------------------------------- 1 | base: 2 | '*': 3 | - versions 4 | -------------------------------------------------------------------------------- /pillars/versions.sls: -------------------------------------------------------------------------------- 1 | versions: 2 | - nginx: 1.25.3 3 | - libmodsecurity: v3/master 4 | - connector: master 5 | - owasp-crs: v3.3.5 6 | - wrk: 4.0.2 7 | -------------------------------------------------------------------------------- /states/benchenv.sls: -------------------------------------------------------------------------------- 1 | {% set release = salt['grains.get']('lsb_distrib_codename', 'focal') %} 2 | 3 | include: 4 | - user 5 | - makeenv 6 | 7 | perfrun.sh: 8 | file.managed: 9 | - user: test 10 | - name: /home/test/perfrun.sh 11 | - mode: 755 12 | - source: salt://files/perfrun.sh 13 | - require: 14 | - Test user 15 | 16 | batchbench.sh: 17 | file.managed: 18 | - user: test 19 | - name: /home/test/batchbench.sh 20 | - mode: 755 21 | - source: salt://files/batchbench.sh 22 | - require: 23 | - Test user 24 | 25 | batchperfrun.sh: 26 | file.managed: 27 | - user: test 28 | - name: /home/test/batchperfrun.sh 29 | - mode: 755 30 | - source: salt://files/batchperfrun.sh 31 | - require: 32 | - Test user 33 | 34 | batchbench.revs: 35 | file.managed: 36 | - user: test 37 | - name: /home/test/batchbench.revs 38 | - contents: | 39 | 10c4f9b1b2476f71159fa5569d9238001760404c 40 | 9e9db08b874fe7c1200aafd95fe6bccd41148ae5 41 | fa7973a4ef99b0d91122d16ffee51744288d037f 42 | 2988c5bb07c4a5ad434855413f20fec11008c818 43 | 63bef3d142b2ae25ed42d344c40729fb5f3d552e 44 | - require: 45 | - Test user 46 | 47 | report.lua: 48 | file.managed: 49 | - user: test 50 | - name: /home/test/report.lua 51 | - contents: | 52 | done = function(summary, latency, requests) 53 | io.write(string.format("latency.avg: %.2f\n", latency.mean/1000)) 54 | io.write(string.format("rps: %.2f\n", summary.requests/(summary.duration/1000000))) 55 | end 56 | - require: 57 | - Test user 58 | 59 | {% if grains['os'] == 'Debian' %} 60 | Debian Non-Free Repository: 61 | pkgrepo.managed: 62 | - humanname: non-free 63 | - name: 'deb http://httpredir.debian.org/debian {{ release }} non-free' 64 | - file: /etc/apt/sources.list.d/non-free.list 65 | - enabled: True 66 | {% endif %} 67 | 68 | nikto package: 69 | pkg.latest: 70 | - name: nikto 71 | {% if grains['os'] == 'Debian' %} 72 | - require: 73 | - Debian Non-Free Repository 74 | {% endif %} 75 | -------------------------------------------------------------------------------- /states/build.sls: -------------------------------------------------------------------------------- 1 | include: 2 | - user 3 | 4 | Build all: 5 | cmd.run: 6 | - name: make all 7 | - cwd: /home/test 8 | - runas: test 9 | - unless: test -e /home/test/ngx_http_modsecurity_module.so 10 | - require: 11 | - Test user 12 | 13 | Symlink for wrk: 14 | file.symlink: 15 | - name: /usr/bin/wrk 16 | - target: /home/test/wrk/wrk 17 | - require: 18 | - Build all 19 | -------------------------------------------------------------------------------- /states/debugenv.sls: -------------------------------------------------------------------------------- 1 | {% set release = salt['grains.get']('lsb_distrib_codename', 'focal') %} 2 | {% set kernelrelease = salt['grains.get']('kernelrelease') %} 3 | {% set virtual = salt['grains.get']('virtual') %} 4 | {% set baseuser = 'vagrant' %} 5 | 6 | {% if grains['os'] == 'Ubuntu' %} 7 | {% for repo_path in ['', '-updates', '-proposed'] %} 8 | Ubuntu Debug Repository{{ repo_path }}: 9 | pkgrepo.managed: 10 | - humanname: repo{{ repo_path }} 11 | - name: 'deb http://ddebs.ubuntu.com/ {{ release }}{{ repo_path }} main restricted universe multiverse' 12 | - file: /etc/apt/sources.list.d/ddebs.list 13 | - enabled: True 14 | - gpgcheck: 1 15 | - keyid: '0xC8CAB6595FDFF622' 16 | - keyserver: keyserver.ubuntu.com 17 | {% endfor %} 18 | {% endif %} 19 | 20 | Debug packages: 21 | pkg.latest: 22 | - pkgs: 23 | - gdb 24 | - libc6-dbg 25 | {% if grains['os'] == 'Ubuntu' %} 26 | - keyutils-dbgsym 27 | - libasn1-8-heimdal-dbgsym 28 | - libbrotli1-dbgsym 29 | - libcom-err2-dbgsym 30 | - libcrypt1-dbgsym 31 | # - libcurl4-dbgsym 32 | - libffi7-dbgsym 33 | - libgcc-s1-dbgsym 34 | - libgeoip1-dbgsym 35 | - libgmp10-dbgsym 36 | - libgssapi3-heimdal-dbgsym 37 | - libhcrypto4-heimdal-dbgsym 38 | - libheimbase1-heimdal-dbgsym 39 | - libheimntlm0-heimdal-dbgsym 40 | - libhogweed5-dbgsym 41 | - libidn2-0-dbgsym 42 | - libicu66-dbgsym 43 | - libkeyutils1-dbgsym 44 | - libkrb5-dbg 45 | - libldap-2.4-2-dbgsym 46 | - libnettle7-dbgsym 47 | - libnghttp2-14-dbgsym 48 | # - libnss-systemd-dbgsym 49 | - libssl1.1-dbgsym 50 | - libsqlite3-0-dbgsym 51 | - libidn11-dbgsym 52 | - liblzma5-dbgsym 53 | - libp11-kit0-dbgsym 54 | - libpcre2-8-0-dbgsym 55 | - libpcre2-16-0-dbgsym 56 | - libpcre2-32-0-dbgsym 57 | - libpcre3-dbg 58 | - libpsl5-dbgsym 59 | - libroken18-heimdal-dbgsym 60 | - librtmp1-dbgsym 61 | - libsasl2-2-dbgsym 62 | - libsasl2-modules-dbgsym 63 | - libsasl2-modules-db-dbgsym 64 | - libssh-4-dbgsym 65 | - libstdc++6-dbgsym 66 | - libtasn1-6-dbgsym 67 | - libunistring2-dbgsym 68 | - libwind0-heimdal-dbgsym 69 | - libxml2-dbgsym 70 | - libyajl2-dbgsym 71 | - zlib1g-dbgsym 72 | - linux-image-{{ kernelrelease }}-dbgsym 73 | - linux-tools-{{ kernelrelease }} 74 | {% endif %} 75 | - systemtap 76 | - valgrind 77 | {% if grains['os'] == 'Ubuntu' %} 78 | - require: 79 | - Ubuntu Debug Repository 80 | - Ubuntu Debug Repository-proposed 81 | - Ubuntu Debug Repository-updates 82 | {% endif %} 83 | 84 | {% for group in ['stapusr', 'stapsys', 'stapdev'] %} 85 | {{ group }} membership: 86 | group.present: 87 | - name: {{ group }} 88 | - addusers: 89 | - {{ baseuser }} 90 | - test 91 | - require: 92 | - Debug packages 93 | {% endfor %} 94 | 95 | Fix permissions for systemtap: 96 | cmd.run: 97 | - name: chmod 644 /boot/System.map-{{ kernelrelease }} 98 | - unless: test `stat -c '%a' /boot/System.map-{{ kernelrelease }}` = 644 99 | - require: 100 | - Debug packages 101 | -------------------------------------------------------------------------------- /states/files/Makefile.tmpl: -------------------------------------------------------------------------------- 1 | #!/usr/bin/make 2 | 3 | NGINXVER={{ nginxver }} 4 | LMSREV={{ lmsrev }} 5 | CONNECTORREV={{ connectorrev }} 6 | WRKREV={{ wrkrev }} 7 | 8 | NPROC=$(shell getconf _NPROCESSORS_ONLN) 9 | 10 | MAKEJFLAG= 11 | ifneq ($(NPROC),1) 12 | MAKEJFLAG=-j$(NPROC) 13 | endif 14 | 15 | CC_OPT_ADD= 16 | LD_OPT_ADD= 17 | ifdef ASAN 18 | CC_OPT_ADD= -fsanitize=address 19 | LD_OPT_ADD= -fsanitize=address 20 | LMS_ASAN_ENV= CFLAGS=-fsanitize=address CXXFLAGS=-fsanitize=address LDFLAGS=-fsanitize=address 21 | endif 22 | 23 | NGINX_CC_OPT=-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC 24 | # enabling MODSECURITY_SANITY_CHECKS leads to test failures 25 | #NGINX_CC_OPT_DEBUG=$(NGINX_CC_OPT) -DMODSECURITY_DDEBUG=1 -DMODSECURITY_SANITY_CHECKS=1 26 | NGINX_CC_OPT_DEBUG=$(NGINX_CC_OPT) -DMODSECURITY_DDEBUG=1 27 | NGINX_LD_OPT=-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie 28 | NGINX_LD_OPT_DEBUG=$(NGINX_LD_OPT) 29 | 30 | NGINX_CONFIGURE_ARGS=\ 31 | --prefix=/etc/nginx \ 32 | --sbin-path=/usr/sbin/nginx \ 33 | --modules-path=/usr/lib/nginx/modules \ 34 | --conf-path=/etc/nginx/nginx.conf \ 35 | --error-log-path=/var/log/nginx/error.log \ 36 | --http-log-path=/var/log/nginx/access.log \ 37 | --pid-path=/var/run/nginx.pid \ 38 | --lock-path=/var/run/nginx.lock \ 39 | --http-client-body-temp-path=/var/cache/nginx/client_temp \ 40 | --http-proxy-temp-path=/var/cache/nginx/proxy_temp \ 41 | --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp \ 42 | --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp \ 43 | --http-scgi-temp-path=/var/cache/nginx/scgi_temp \ 44 | --user=nginx \ 45 | --group=nginx \ 46 | --with-compat \ 47 | --with-file-aio \ 48 | --with-threads \ 49 | --with-http_addition_module \ 50 | --with-http_auth_request_module \ 51 | --with-http_dav_module \ 52 | --with-http_flv_module \ 53 | --with-http_gunzip_module \ 54 | --with-http_gzip_static_module \ 55 | --with-http_mp4_module \ 56 | --with-http_random_index_module \ 57 | --with-http_realip_module \ 58 | --with-http_secure_link_module \ 59 | --with-http_slice_module \ 60 | --with-http_ssl_module \ 61 | --with-http_stub_status_module \ 62 | --with-http_sub_module \ 63 | --with-http_v2_module \ 64 | --with-mail \ 65 | --with-mail_ssl_module \ 66 | --with-stream \ 67 | --with-stream_realip_module \ 68 | --with-stream_ssl_module \ 69 | --with-stream_ssl_preread_module \ 70 | --add-module=../ModSecurity-nginx 71 | 72 | default: 73 | @echo "valid targets: all, lms, ngxmod, nginx-static" 74 | 75 | all: lms ngxmod wrk 76 | 77 | ModSecurity: 78 | @git clone https://github.com/SpiderLabs/ModSecurity.git && \ 79 | cd ModSecurity && \ 80 | git checkout $(LMSREV) && \ 81 | git submodule init && \ 82 | git submodule update 83 | 84 | ModSecurity-nginx: 85 | @git clone https://github.com/SpiderLabs/ModSecurity-nginx.git && \ 86 | cd ModSecurity-nginx && \ 87 | git checkout $(CONNECTORREV) 88 | 89 | lms: ModSecurity 90 | @cd ModSecurity && \ 91 | ./build.sh && \ 92 | $(LMS_ASAN_ENV) ./configure --prefix=/usr --without-lmdb && \ 93 | make $(MAKEJFLAG) && \ 94 | sudo make install 95 | @touch $@ 96 | 97 | nginx-$(NGINXVER).tar.gz: 98 | @wget http://nginx.org/download/nginx-$(NGINXVER).tar.gz 99 | 100 | nginx-$(NGINXVER): nginx-$(NGINXVER).tar.gz 101 | @tar mzvxf nginx-$(NGINXVER).tar.gz 102 | 103 | ngxmod: ModSecurity ModSecurity-nginx nginx-$(NGINXVER) 104 | @cd nginx-$(NGINXVER) && \ 105 | export MODSECURITY_INC=/usr/include && \ 106 | export MODSECURITY_LIB=/usr/lib && \ 107 | export NGX_IGNORE_RPATH=YES && \ 108 | ./configure --with-compat --add-dynamic-module=../ModSecurity-nginx \ 109 | --with-cc-opt='$(NGINX_CC_OPT_DEBUG) $(CC_OPT_ADD)' \ 110 | --with-ld-opt='$(NGINX_LD_OPT_DEBUG) $(LD_OPT_ADD)' \ 111 | --with-debug && \ 112 | make modules && \ 113 | cp -p objs/ngx_http_modsecurity_module.so ngx_http_modsecurity_module-debug.so && \ 114 | make clean && \ 115 | ./configure --with-compat --add-dynamic-module=../ModSecurity-nginx \ 116 | --with-cc-opt='$(NGINX_CC_OPT) $(CC_OPT_ADD)' \ 117 | --with-ld-opt='$(NGINX_LD_OPT) $(LD_OPT_ADD)' && \ 118 | make modules && \ 119 | cp -p objs/ngx_http_modsecurity_module.so ./ 120 | @cp -p nginx-$(NGINXVER)/*.so ./ 121 | @touch $@ 122 | 123 | nginx-static: ModSecurity ModSecurity-nginx nginx-$(NGINXVER) 124 | @cd nginx-$(NGINXVER) && \ 125 | export MODSECURITY_INC=/usr/include && \ 126 | export MODSECURITY_LIB=/usr/lib && \ 127 | export NGX_IGNORE_RPATH=YES && \ 128 | ./configure $(NGINX_CONFIGURE_ARGS) --with-cc-opt='$(NGINX_CC_OPT_DEBUG) $(CC_OPT_ADD)' --with-ld-opt='$(NGINX_LD_OPT_DEBUG) $(LD_OPT_ADD)' --with-debug && \ 129 | make $(MAKEJFLAG) && \ 130 | cp -p objs/nginx nginx-static.debug && \ 131 | make clean && \ 132 | ./configure $(NGINX_CONFIGURE_ARGS) --with-cc-opt='$(NGINX_CC_OPT) $(CC_OPT_ADD)' --with-ld-opt='$(NGINX_LD_OPT) $(LD_OPT_ADD)' && \ 133 | make $(MAKEJFLAG) && \ 134 | cp -p objs/nginx nginx-static 135 | @cp -p nginx-$(NGINXVER)/nginx-static* ./ 136 | 137 | wrk: 138 | @git clone https://github.com/wg/wrk.git && \ 139 | cd wrk && \ 140 | git checkout $(WRKREV) && \ 141 | make 142 | 143 | nginx-tests: 144 | @hg clone http://hg.nginx.org/nginx-tests 145 | 146 | nginx-tests-modsec-on: 147 | @hg clone http://hg.nginx.org/nginx-tests nginx-tests-modsec-on && \ 148 | cd nginx-tests-modsec-on && \ 149 | for i in *.t; do cp -n $$i $$i.orig; perl ../ModSecurity-nginx/tests/nginx-tests-cvt.pl < $$i.orig > $$i; rm $$i.orig; done 150 | 151 | test: lms ngxmod nginx-tests 152 | @cd nginx-tests && \ 153 | cp ../ModSecurity-nginx/tests/*.t ./ && \ 154 | TEST_NGINX_BINARY=/usr/sbin/nginx \ 155 | TEST_NGINX_GLOBALS="load_module /home/test/nginx-$(NGINXVER)/ngx_http_modsecurity_module.so;" \ 156 | prove modsecurity*.t 157 | 158 | test-debug: lms ngxmod nginx-tests 159 | @cd nginx-tests && \ 160 | cp ../ModSecurity-nginx/tests/*.t ./ && \ 161 | TEST_NGINX_BINARY=/usr/sbin/nginx-debug \ 162 | TEST_NGINX_GLOBALS="load_module /home/test/nginx-$(NGINXVER)/ngx_http_modsecurity_module-debug.so;" \ 163 | prove modsecurity*.t 164 | 165 | test-all: lms ngxmod nginx-tests 166 | @cd nginx-tests && \ 167 | cp ../ModSecurity-nginx/tests/*.t ./ && \ 168 | TEST_NGINX_BINARY=/usr/sbin/nginx \ 169 | TEST_NGINX_GLOBALS="load_module /home/test/nginx-$(NGINXVER)/ngx_http_modsecurity_module.so;" \ 170 | prove . 171 | 172 | test-all-debug: lms ngxmod nginx-tests 173 | @cd nginx-tests && \ 174 | cp ../ModSecurity-nginx/tests/*.t ./ && \ 175 | TEST_NGINX_BINARY=/usr/sbin/nginx-debug \ 176 | TEST_NGINX_GLOBALS="load_module /home/test/nginx-$(NGINXVER)/ngx_http_modsecurity_module-debug.so;" \ 177 | prove . 178 | 179 | test-all-modsec-on: lms ngxmod nginx-tests-modsec-on 180 | @cd nginx-tests-modsec-on && \ 181 | cp ../ModSecurity-nginx/tests/*.t ./ && \ 182 | TEST_NGINX_BINARY=/usr/sbin/nginx \ 183 | TEST_NGINX_GLOBALS="load_module /home/test/nginx-$(NGINXVER)/ngx_http_modsecurity_module.so;" \ 184 | prove . 185 | 186 | test-all-modsec-on-debug: lms ngxmod nginx-tests-modsec-on 187 | @cd nginx-tests-modsec-on && \ 188 | cp ../ModSecurity-nginx/tests/*.t ./ && \ 189 | TEST_NGINX_BINARY=/usr/sbin/nginx-debug \ 190 | TEST_NGINX_GLOBALS="load_module /home/test/nginx-$(NGINXVER)/ngx_http_modsecurity_module-debug.so;" \ 191 | prove . 192 | 193 | clean: 194 | @rm -rf ModSecurity/ ModSecurity-nginx/ nginx-$(NGINXVER).tar.gz nginx-$(NGINXVER)/ lms ngxmod nginx-tests* ngx_http_modsecurity_module*.so nginx-static* 195 | -------------------------------------------------------------------------------- /states/files/batchbench.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # vim: tabstop=4 expandtab 3 | 4 | function run { 5 | mkdir -p ${HOME}/batch/raw ${HOME}/batch/res 6 | 7 | if [ ! -f Makefile.in ]; then 8 | sed -e "s#^LMSREV=.*#LMSREV=%%REV%%#g" < Makefile > Makefile.in 9 | fi 10 | 11 | while read rev ; do 12 | if [ -f "${HOME}/batch/res/${rev}" ]; then 13 | continue 14 | fi 15 | 16 | echo "===> Working with ${rev}" 17 | rm -rf ModSecurity lms 18 | sed -e "s#%%REV%%#${rev}#g" < Makefile.in > Makefile 19 | 20 | echo "---> Building ${rev}" 21 | make lms > ${HOME}/batch/raw/${rev}.build 2>&1 22 | (cd ModSecurity && git log -1 --pretty=format:"%H,%ai,%s" ${rev}) > ${HOME}/batch/res/${rev}.log 23 | 24 | echo "---> Benchmarking ${rev}" 25 | ./batchperfrun.sh run ${rev} 26 | ./batchperfrun.sh stats ${rev} > ${HOME}/batch/res/${rev} 27 | 28 | cat ${HOME}/batch/res/${rev} 29 | done < batchbench.revs 30 | } 31 | 32 | function stats { 33 | echo ";rps_avg,latency_avg,workers_utime_avg,revision,date,commit_log" 34 | 35 | while read rev ; do 36 | if [ ! -f ${HOME}/batch/res/${rev} ]; then 37 | continue 38 | fi 39 | 40 | str=`cat ${HOME}/batch/res/${rev}.log` 41 | avgs=`cat ${HOME}/batch/res/${rev} | grep "^x" | awk '{print $6}' | tr '\n' ' '` 42 | rps=`echo ${avgs} | cut -d ' ' -f 1` 43 | lat=`echo ${avgs} | cut -d ' ' -f 2` 44 | utime=`echo ${avgs} | cut -d ' ' -f 3` 45 | 46 | if [ -z "${rps}" -o -z "${lat}" ]; then 47 | continue 48 | fi 49 | 50 | printf "%.2f,%.2f,%.2f,%s\n" "${rps}" "${lat}" "${utime}" "${str}" 51 | done < batchbench.revs 52 | } 53 | 54 | case $1 in 55 | run) run ;; 56 | stats) stats ;; 57 | *) echo "usage: `basename $0` run|stats" ;; 58 | esac 59 | -------------------------------------------------------------------------------- /states/files/batchperfrun.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # vim: tabstop=4 expandtab 3 | 4 | set -e 5 | 6 | locations="/modsec-full" 7 | iterations=10 8 | duration=10s 9 | threads=1 10 | connections=50 11 | 12 | function usage { 13 | echo "usage: `basename $0` run|stats rev" >&2 14 | exit 1 15 | } 16 | 17 | function run { 18 | sudo service nginx stop 19 | 20 | echo "Running wrk, please be patient" 21 | 22 | for location in ${locations}; do 23 | rm -f ${PREFIX}${location}.out 24 | i=1 25 | while [ $i -le ${iterations} ]; do 26 | echo "[${i}/${iterations}] $location ..." 27 | sudo service nginx start 28 | sleep 1 29 | wrk \ 30 | -t${threads} -c${connections} -d${duration} -s ${HOME}/report.lua \ 31 | -H "Host: www.example.com" \ 32 | -H "User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5 (.NET CLR 3.5.30729)" \ 33 | -H "Accept: text/html,application/xhtml+xml,application/xml; q=0.9,*/*;q=0.8" \ 34 | -H "Accept-Language: en-us,en;q=0.5" \ 35 | -H "Accept-Encoding: gzip,deflate" \ 36 | -H "Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7" \ 37 | -H "Keep-Alive: 300" \ 38 | -H "Connection: keep-alive" \ 39 | -H "Cookie: PHPSESSID=r2t5uvjq435r4q7ib3vtdjq120" \ 40 | -H "Pragma: no-cache" \ 41 | -H "Cache-Control: no-cache" \ 42 | "http://localhost${location}/test.pl?param1=test¶2=test2" 2>&1 >>${PREFIX}${location}.out 43 | 44 | ppid=`cat /var/run/nginx.pid` 45 | ut=0 46 | for pid in `pgrep -P ${ppid}`; do 47 | utime=`cat /proc/${pid}/stat | cut -d ' ' -f 14` 48 | ut=$((ut + $utime)) 49 | done 50 | echo "workers_utime: $ut" >>${PREFIX}${location}.out 51 | 52 | sudo service nginx stop 53 | sleep 1 54 | i=$((i+1)) 55 | done 56 | done 57 | } 58 | 59 | function stats { 60 | for location in ${locations}; do 61 | echo "Summary for ${location}, RPS (count):" 62 | awk '{if ($1 == "rps:") {print $2}}' ${PREFIX}${location}.out | ministat -n -w 80 | fgrep -v stdin 63 | echo " latency (ms)" 64 | awk '{if ($1 == "latency.avg:") {print $2}}' ${PREFIX}${location}.out | ministat -n -w 80 | tail -1 65 | echo " workers_utime" 66 | awk '{if ($1 == "workers_utime:") {print $2}}' ${PREFIX}${location}.out | ministat -n -w 80 | tail -1 67 | echo 68 | done 69 | } 70 | 71 | if [ $# -lt 2 ]; then 72 | usage 73 | fi 74 | 75 | PREFIX="${HOME}/batch/raw/rev-$2" 76 | mkdir -p ${PREFIX} 77 | 78 | case $1 in 79 | run) run ;; 80 | stats) stats ;; 81 | *) usage ;; 82 | esac 83 | -------------------------------------------------------------------------------- /states/files/etc/nginx/conf.d/grpc-proxy.conf: -------------------------------------------------------------------------------- 1 | upstream grpcupstream { 2 | zone grpcupstream 64k; 3 | server 127.0.0.1:23333; 4 | keepalive 3; 5 | } 6 | 7 | server { 8 | listen 9999 http2; 9 | location / { 10 | grpc_pass grpc://grpcupstream; 11 | } 12 | } 13 | 14 | server { 15 | listen 6666 http2; 16 | location / { 17 | modsecurity on; 18 | modsecurity_rules_file /etc/nginx/modsec/main.conf; 19 | grpc_pass grpc://grpcupstream; 20 | } 21 | } 22 | -------------------------------------------------------------------------------- /states/files/etc/nginx/modsec/main.conf: -------------------------------------------------------------------------------- 1 | include /etc/nginx/modsec/modsecurity.conf 2 | 3 | # OWASP CRS rules 4 | include /etc/nginx/modsec/owasp-crs/crs-setup.conf 5 | include /etc/nginx/modsec/owasp-crs/rules/*.conf 6 | -------------------------------------------------------------------------------- /states/files/etc/nginx/nginx.conf: -------------------------------------------------------------------------------- 1 | load_module /home/test/ngx_http_modsecurity_module.so; 2 | 3 | user nginx; 4 | worker_processes 1; 5 | 6 | #master_process off; 7 | worker_rlimit_core 1000M; 8 | working_directory /tmp/; 9 | 10 | error_log /var/log/nginx/error.log debug; 11 | pid /var/run/nginx.pid; 12 | 13 | events { 14 | worker_connections 65536; 15 | } 16 | 17 | http { 18 | log_format main '$remote_addr - $remote_user [$time_local] "$request" ' 19 | '$status $body_bytes_sent "$http_referer" ' 20 | '"$http_user_agent" "$http_x_forwarded_for" "$ssl_protocol" "$http2"'; 21 | 22 | #access_log /var/log/nginx/access.log main; 23 | access_log off; 24 | 25 | keepalive_requests 100000; 26 | 27 | upstream local { 28 | server 127.0.0.1:8080; 29 | keepalive 128; 30 | } 31 | 32 | upstream upload-app { 33 | server 127.0.0.1:3131; 34 | } 35 | 36 | server { 37 | listen 8080 default_server backlog=32000; 38 | location / { 39 | default_type text/plain; 40 | return 200 "Thank you for requesting ${request_uri}\n"; 41 | } 42 | } 43 | 44 | server { 45 | listen 80 default_server backlog=32000; 46 | listen 443 ssl http2 default_server backlog=32000; 47 | 48 | ssl_certificate localhost.pem; 49 | ssl_certificate_key localhost.key; 50 | 51 | proxy_http_version 1.1; 52 | proxy_set_header Connection ""; 53 | 54 | location = /status { 55 | stub_status on; 56 | } 57 | 58 | location = /204 { 59 | return 204; 60 | } 61 | 62 | location / { 63 | proxy_pass http://local; 64 | } 65 | 66 | location /upload/ { 67 | include uwsgi_params; 68 | uwsgi_pass upload-app; 69 | client_max_body_size 256m; 70 | } 71 | 72 | location /modsec-light/ { 73 | modsecurity on; 74 | proxy_pass http://local; 75 | } 76 | 77 | location /modsec-light/upload/ { 78 | modsecurity on; 79 | client_max_body_size 256m; 80 | include uwsgi_params; 81 | uwsgi_pass upload-app; 82 | } 83 | 84 | location /modsec-full/ { 85 | modsecurity on; 86 | modsecurity_rules_file /etc/nginx/modsec/main.conf; 87 | proxy_pass http://local; 88 | } 89 | 90 | location /modsec-full/upload/ { 91 | modsecurity on; 92 | modsecurity_rules_file /etc/nginx/modsec/main.conf; 93 | client_max_body_size 256m; 94 | include uwsgi_params; 95 | uwsgi_pass upload-app; 96 | } 97 | } 98 | 99 | include /etc/nginx/conf.d/*.conf; 100 | } 101 | -------------------------------------------------------------------------------- /states/files/gencert.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | if [ $# -lt 1 ]; then 4 | echo "usage: `basename $0` domain" >&2 5 | exit 1 6 | fi 7 | 8 | DOMAIN=$1 9 | 10 | openssl genrsa -out $DOMAIN.key 2048 && \ 11 | openssl req -new -key $DOMAIN.key -batch -subj /CN=$DOMAIN/ -out $DOMAIN.csr && \ 12 | openssl x509 -req -days 365 -in $DOMAIN.csr -signkey $DOMAIN.key -out $DOMAIN.pem && \ 13 | rm $DOMAIN.csr 14 | -------------------------------------------------------------------------------- /states/files/grpc_data_transmission/client.py: -------------------------------------------------------------------------------- 1 | # Copyright 2019 gRPC authors. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """The example of four ways of data transmission using gRPC in Python.""" 15 | 16 | import time 17 | import grpc 18 | 19 | import demo_pb2_grpc 20 | import demo_pb2 21 | 22 | SERVER_ADDRESS = "localhost:23333" 23 | CLIENT_ID = 1 24 | 25 | # 中文注释和英文翻译 26 | # Note that this example was contributed by an external user using Chinese comments. 27 | # In all cases, the Chinese comment text is translated to English just below it. 28 | 29 | 30 | # 一元模式(在一次调用中, 客户端只能向服务器传输一次请求数据, 服务器也只能返回一次响应) 31 | # unary-unary(In a single call, the client can only send request once, and the server can 32 | # only respond once.) 33 | def simple_method(stub): 34 | print("--------------Call SimpleMethod Begin--------------") 35 | request = demo_pb2.Request(client_id=CLIENT_ID, 36 | request_data="called by Python client") 37 | response = stub.SimpleMethod(request) 38 | print("resp from server(%d), the message=%s" % 39 | (response.server_id, response.response_data)) 40 | print("--------------Call SimpleMethod Over---------------") 41 | 42 | 43 | # 客户端流模式（在一次调用中, 客户端可以多次向服务器传输数据, 但是服务器只能返回一次响应） 44 | # stream-unary (In a single call, the client can transfer data to the server several times, 45 | # but the server can only return a response once.) 46 | def client_streaming_method(stub): 47 | print("--------------Call ClientStreamingMethod Begin--------------") 48 | 49 | # 创建一个生成器 50 | # create a generator 51 | def request_messages(): 52 | for i in range(5): 53 | request = demo_pb2.Request( 54 | client_id=CLIENT_ID, 55 | request_data=("called by Python client, message:%d" % i)) 56 | yield request 57 | 58 | response = stub.ClientStreamingMethod(request_messages()) 59 | print("resp from server(%d), the message=%s" % 60 | (response.server_id, response.response_data)) 61 | print("--------------Call ClientStreamingMethod Over---------------") 62 | 63 | 64 | # 服务端流模式（在一次调用中, 客户端只能一次向服务器传输数据, 但是服务器可以多次返回响应） 65 | # unary-stream (In a single call, the client can only transmit data to the server at one time, 66 | # but the server can return the response many times.) 67 | def server_streaming_method(stub): 68 | print("--------------Call ServerStreamingMethod Begin--------------") 69 | request = demo_pb2.Request(client_id=CLIENT_ID, 70 | request_data="called by Python client") 71 | response_iterator = stub.ServerStreamingMethod(request) 72 | for response in response_iterator: 73 | print("recv from server(%d), message=%s" % 74 | (response.server_id, response.response_data)) 75 | 76 | print("--------------Call ServerStreamingMethod Over---------------") 77 | 78 | 79 | # 双向流模式 (在一次调用中, 客户端和服务器都可以向对方多次收发数据) 80 | # stream-stream (In a single call, both client and server can send and receive data 81 | # to each other multiple times.) 82 | def bidirectional_streaming_method(stub): 83 | print( 84 | "--------------Call BidirectionalStreamingMethod Begin---------------") 85 | 86 | # 创建一个生成器 87 | # create a generator 88 | def request_messages(): 89 | for i in range(5): 90 | request = demo_pb2.Request( 91 | client_id=CLIENT_ID, 92 | request_data=("called by Python client, message: %d" % i)) 93 | yield request 94 | time.sleep(1) 95 | 96 | response_iterator = stub.BidirectionalStreamingMethod(request_messages()) 97 | for response in response_iterator: 98 | print("recv from server(%d), message=%s" % 99 | (response.server_id, response.response_data)) 100 | 101 | print("--------------Call BidirectionalStreamingMethod Over---------------") 102 | 103 | 104 | def main(): 105 | with grpc.insecure_channel(SERVER_ADDRESS) as channel: 106 | stub = demo_pb2_grpc.GRPCDemoStub(channel) 107 | 108 | simple_method(stub) 109 | 110 | client_streaming_method(stub) 111 | 112 | server_streaming_method(stub) 113 | 114 | bidirectional_streaming_method(stub) 115 | 116 | 117 | if __name__ == '__main__': 118 | main() 119 | -------------------------------------------------------------------------------- /states/files/grpc_data_transmission/demo.proto: -------------------------------------------------------------------------------- 1 | // Copyright 2019 gRPC authors. 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | // 语法版本声明，必须放在非注释的第一行 16 | // Syntax version declaration. Must be placed on the first line of non-commentary. 17 | 18 | syntax = "proto3"; 19 | // The document of proto3: https://developers.google.com/protocol-buffers/docs/proto3 20 | 21 | // 包名定义, Python中使用时可以省略不写 22 | // Package name definition, which can be omitted in Python. 23 | package demo; 24 | 25 | /* 26 | `message`是用来定义传输的数据的格式的, 等号后面的是字段编号 27 | 消息定义中的每个字段都有唯一的编号 28 | 总体格式类似于Python中定义一个类或者Golang中定义一个结构体 29 | */ 30 | /* 31 | `message` is used to define the structure of the data to be transmitted, after the equal sign 32 | is the field number. Each field in the message definition has a unique number. 33 | The overall format is similar to defining a class in Python or a structure in Golang. 34 | */ 35 | message Request { 36 | int64 client_id = 1; 37 | string request_data = 2; 38 | } 39 | 40 | message Response { 41 | int64 server_id = 1; 42 | string response_data = 2; 43 | } 44 | 45 | // `service` 是用来给gRPC服务定义方法的, 格式固定, 类似于Golang中定义一个接口 46 | // `service` is used to define methods for gRPC services in a fixed format, similar to defining 47 | //an interface in Golang 48 | service GRPCDemo { 49 | // 一元模式(在一次调用中, 客户端只能向服务器传输一次请求数据, 服务器也只能返回一次响应) 50 | // unary-unary(In a single call, the client can only send request once, and the server can 51 | // only respond once.) 52 | rpc SimpleMethod (Request) returns (Response); 53 | 54 | // 客户端流模式（在一次调用中, 客户端可以多次向服务器传输数据, 但是服务器只能返回一次响应） 55 | // stream-unary (In a single call, the client can transfer data to the server several times, 56 | // but the server can only return a response once.) 57 | rpc ClientStreamingMethod (stream Request) returns (Response); 58 | 59 | // 服务端流模式（在一次调用中, 客户端只能一次向服务器传输数据, 但是服务器可以多次返回响应） 60 | // unary-stream (In a single call, the client can only transmit data to the server at one time, 61 | // but the server can return the response many times.) 62 | rpc ServerStreamingMethod (Request) returns (stream Response); 63 | 64 | // 双向流模式 (在一次调用中, 客户端和服务器都可以向对方多次收发数据) 65 | // stream-stream (In a single call, both client and server can send and receive data 66 | // to each other multiple times.) 67 | rpc BidirectionalStreamingMethod (stream Request) returns (stream Response); 68 | } 69 | 70 | -------------------------------------------------------------------------------- /states/files/grpc_data_transmission/demo_pb2.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # Generated by the protocol buffer compiler. DO NOT EDIT! 3 | # source: demo.proto 4 | 5 | import sys 6 | _b=sys.version_info[0]<3 and (lambda x:x) or (lambda x:x.encode('latin1')) 7 | from google.protobuf import descriptor as _descriptor 8 | from google.protobuf import message as _message 9 | from google.protobuf import reflection as _reflection 10 | from google.protobuf import symbol_database as _symbol_database 11 | # @@protoc_insertion_point(imports) 12 | 13 | _sym_db = _symbol_database.Default() 14 | 15 | 16 | 17 | 18 | DESCRIPTOR = _descriptor.FileDescriptor( 19 | name='demo.proto', 20 | package='demo', 21 | syntax='proto3', 22 | serialized_options=None, 23 | serialized_pb=_b('\n\ndemo.proto\x12\x04\x64\x65mo\"2\n\x07Request\x12\x11\n\tclient_id\x18\x01 \x01(\x03\x12\x14\n\x0crequest_data\x18\x02 \x01(\t\"4\n\x08Response\x12\x11\n\tserver_id\x18\x01 \x01(\x03\x12\x15\n\rresponse_data\x18\x02 \x01(\t2\xf0\x01\n\x08GRPCDemo\x12-\n\x0cSimpleMethod\x12\r.demo.Request\x1a\x0e.demo.Response\x12\x38\n\x15\x43lientStreamingMethod\x12\r.demo.Request\x1a\x0e.demo.Response(\x01\x12\x38\n\x15ServerStreamingMethod\x12\r.demo.Request\x1a\x0e.demo.Response0\x01\x12\x41\n\x1c\x42idirectionalStreamingMethod\x12\r.demo.Request\x1a\x0e.demo.Response(\x01\x30\x01\x62\x06proto3') 24 | ) 25 | 26 | 27 | 28 | 29 | _REQUEST = _descriptor.Descriptor( 30 | name='Request', 31 | full_name='demo.Request', 32 | filename=None, 33 | file=DESCRIPTOR, 34 | containing_type=None, 35 | fields=[ 36 | _descriptor.FieldDescriptor( 37 | name='client_id', full_name='demo.Request.client_id', index=0, 38 | number=1, type=3, cpp_type=2, label=1, 39 | has_default_value=False, default_value=0, 40 | message_type=None, enum_type=None, containing_type=None, 41 | is_extension=False, extension_scope=None, 42 | serialized_options=None, file=DESCRIPTOR), 43 | _descriptor.FieldDescriptor( 44 | name='request_data', full_name='demo.Request.request_data', index=1, 45 | number=2, type=9, cpp_type=9, label=1, 46 | has_default_value=False, default_value=_b("").decode('utf-8'), 47 | message_type=None, enum_type=None, containing_type=None, 48 | is_extension=False, extension_scope=None, 49 | serialized_options=None, file=DESCRIPTOR), 50 | ], 51 | extensions=[ 52 | ], 53 | nested_types=[], 54 | enum_types=[ 55 | ], 56 | serialized_options=None, 57 | is_extendable=False, 58 | syntax='proto3', 59 | extension_ranges=[], 60 | oneofs=[ 61 | ], 62 | serialized_start=20, 63 | serialized_end=70, 64 | ) 65 | 66 | 67 | _RESPONSE = _descriptor.Descriptor( 68 | name='Response', 69 | full_name='demo.Response', 70 | filename=None, 71 | file=DESCRIPTOR, 72 | containing_type=None, 73 | fields=[ 74 | _descriptor.FieldDescriptor( 75 | name='server_id', full_name='demo.Response.server_id', index=0, 76 | number=1, type=3, cpp_type=2, label=1, 77 | has_default_value=False, default_value=0, 78 | message_type=None, enum_type=None, containing_type=None, 79 | is_extension=False, extension_scope=None, 80 | serialized_options=None, file=DESCRIPTOR), 81 | _descriptor.FieldDescriptor( 82 | name='response_data', full_name='demo.Response.response_data', index=1, 83 | number=2, type=9, cpp_type=9, label=1, 84 | has_default_value=False, default_value=_b("").decode('utf-8'), 85 | message_type=None, enum_type=None, containing_type=None, 86 | is_extension=False, extension_scope=None, 87 | serialized_options=None, file=DESCRIPTOR), 88 | ], 89 | extensions=[ 90 | ], 91 | nested_types=[], 92 | enum_types=[ 93 | ], 94 | serialized_options=None, 95 | is_extendable=False, 96 | syntax='proto3', 97 | extension_ranges=[], 98 | oneofs=[ 99 | ], 100 | serialized_start=72, 101 | serialized_end=124, 102 | ) 103 | 104 | DESCRIPTOR.message_types_by_name['Request'] = _REQUEST 105 | DESCRIPTOR.message_types_by_name['Response'] = _RESPONSE 106 | _sym_db.RegisterFileDescriptor(DESCRIPTOR) 107 | 108 | Request = _reflection.GeneratedProtocolMessageType('Request', (_message.Message,), { 109 | 'DESCRIPTOR' : _REQUEST, 110 | '__module__' : 'demo_pb2' 111 | # @@protoc_insertion_point(class_scope:demo.Request) 112 | }) 113 | _sym_db.RegisterMessage(Request) 114 | 115 | Response = _reflection.GeneratedProtocolMessageType('Response', (_message.Message,), { 116 | 'DESCRIPTOR' : _RESPONSE, 117 | '__module__' : 'demo_pb2' 118 | # @@protoc_insertion_point(class_scope:demo.Response) 119 | }) 120 | _sym_db.RegisterMessage(Response) 121 | 122 | 123 | 124 | _GRPCDEMO = _descriptor.ServiceDescriptor( 125 | name='GRPCDemo', 126 | full_name='demo.GRPCDemo', 127 | file=DESCRIPTOR, 128 | index=0, 129 | serialized_options=None, 130 | serialized_start=127, 131 | serialized_end=367, 132 | methods=[ 133 | _descriptor.MethodDescriptor( 134 | name='SimpleMethod', 135 | full_name='demo.GRPCDemo.SimpleMethod', 136 | index=0, 137 | containing_service=None, 138 | input_type=_REQUEST, 139 | output_type=_RESPONSE, 140 | serialized_options=None, 141 | ), 142 | _descriptor.MethodDescriptor( 143 | name='ClientStreamingMethod', 144 | full_name='demo.GRPCDemo.ClientStreamingMethod', 145 | index=1, 146 | containing_service=None, 147 | input_type=_REQUEST, 148 | output_type=_RESPONSE, 149 | serialized_options=None, 150 | ), 151 | _descriptor.MethodDescriptor( 152 | name='ServerStreamingMethod', 153 | full_name='demo.GRPCDemo.ServerStreamingMethod', 154 | index=2, 155 | containing_service=None, 156 | input_type=_REQUEST, 157 | output_type=_RESPONSE, 158 | serialized_options=None, 159 | ), 160 | _descriptor.MethodDescriptor( 161 | name='BidirectionalStreamingMethod', 162 | full_name='demo.GRPCDemo.BidirectionalStreamingMethod', 163 | index=3, 164 | containing_service=None, 165 | input_type=_REQUEST, 166 | output_type=_RESPONSE, 167 | serialized_options=None, 168 | ), 169 | ]) 170 | _sym_db.RegisterServiceDescriptor(_GRPCDEMO) 171 | 172 | DESCRIPTOR.services_by_name['GRPCDemo'] = _GRPCDEMO 173 | 174 | # @@protoc_insertion_point(module_scope) 175 | -------------------------------------------------------------------------------- /states/files/grpc_data_transmission/demo_pb2_grpc.py: -------------------------------------------------------------------------------- 1 | # Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT! 2 | import grpc 3 | 4 | import demo_pb2 as demo__pb2 5 | 6 | 7 | class GRPCDemoStub(object): 8 | """service是用来给GRPC服务定义方法的, 格式固定, 类似于Golang中定义一个接口 9 | `service` is used to define methods for GRPC services in a fixed format, similar to defining an interface in Golang 10 | """ 11 | 12 | def __init__(self, channel): 13 | """Constructor. 14 | 15 | Args: 16 | channel: A grpc.Channel. 17 | """ 18 | self.SimpleMethod = channel.unary_unary( 19 | '/demo.GRPCDemo/SimpleMethod', 20 | request_serializer=demo__pb2.Request.SerializeToString, 21 | response_deserializer=demo__pb2.Response.FromString, 22 | ) 23 | self.ClientStreamingMethod = channel.stream_unary( 24 | '/demo.GRPCDemo/ClientStreamingMethod', 25 | request_serializer=demo__pb2.Request.SerializeToString, 26 | response_deserializer=demo__pb2.Response.FromString, 27 | ) 28 | self.ServerStreamingMethod = channel.unary_stream( 29 | '/demo.GRPCDemo/ServerStreamingMethod', 30 | request_serializer=demo__pb2.Request.SerializeToString, 31 | response_deserializer=demo__pb2.Response.FromString, 32 | ) 33 | self.BidirectionalStreamingMethod = channel.stream_stream( 34 | '/demo.GRPCDemo/BidirectionalStreamingMethod', 35 | request_serializer=demo__pb2.Request.SerializeToString, 36 | response_deserializer=demo__pb2.Response.FromString, 37 | ) 38 | 39 | 40 | class GRPCDemoServicer(object): 41 | """service是用来给GRPC服务定义方法的, 格式固定, 类似于Golang中定义一个接口 42 | `service` is used to define methods for GRPC services in a fixed format, similar to defining an interface in Golang 43 | """ 44 | 45 | def SimpleMethod(self, request, context): 46 | """简单模式 47 | unary-unary 48 | """ 49 | context.set_code(grpc.StatusCode.UNIMPLEMENTED) 50 | context.set_details('Method not implemented!') 51 | raise NotImplementedError('Method not implemented!') 52 | 53 | def ClientStreamingMethod(self, request_iterator, context): 54 | """客户端流模式（在一次调用中, 客户端可以多次向服务器传输数据, 但是服务器只能返回一次响应） 55 | stream-unary (In a single call, the client can transfer data to the server several times, 56 | but the server can only return a response once.) 57 | """ 58 | context.set_code(grpc.StatusCode.UNIMPLEMENTED) 59 | context.set_details('Method not implemented!') 60 | raise NotImplementedError('Method not implemented!') 61 | 62 | def ServerStreamingMethod(self, request, context): 63 | """服务端流模式（在一次调用中, 客户端只能一次向服务器传输数据, 但是服务器可以多次返回响应） 64 | unary-stream (In a single call, the client can only transmit data to the server at one time, 65 | but the server can return the response many times.) 66 | """ 67 | context.set_code(grpc.StatusCode.UNIMPLEMENTED) 68 | context.set_details('Method not implemented!') 69 | raise NotImplementedError('Method not implemented!') 70 | 71 | def BidirectionalStreamingMethod(self, request_iterator, context): 72 | """双向流模式 (在一次调用中, 客户端和服务器都可以向对方多次收发数据) 73 | stream-stream (In a single call, both client and server can send and receive data 74 | to each other multiple times.) 75 | """ 76 | context.set_code(grpc.StatusCode.UNIMPLEMENTED) 77 | context.set_details('Method not implemented!') 78 | raise NotImplementedError('Method not implemented!') 79 | 80 | 81 | def add_GRPCDemoServicer_to_server(servicer, server): 82 | rpc_method_handlers = { 83 | 'SimpleMethod': grpc.unary_unary_rpc_method_handler( 84 | servicer.SimpleMethod, 85 | request_deserializer=demo__pb2.Request.FromString, 86 | response_serializer=demo__pb2.Response.SerializeToString, 87 | ), 88 | 'ClientStreamingMethod': grpc.stream_unary_rpc_method_handler( 89 | servicer.ClientStreamingMethod, 90 | request_deserializer=demo__pb2.Request.FromString, 91 | response_serializer=demo__pb2.Response.SerializeToString, 92 | ), 93 | 'ServerStreamingMethod': grpc.unary_stream_rpc_method_handler( 94 | servicer.ServerStreamingMethod, 95 | request_deserializer=demo__pb2.Request.FromString, 96 | response_serializer=demo__pb2.Response.SerializeToString, 97 | ), 98 | 'BidirectionalStreamingMethod': grpc.stream_stream_rpc_method_handler( 99 | servicer.BidirectionalStreamingMethod, 100 | request_deserializer=demo__pb2.Request.FromString, 101 | response_serializer=demo__pb2.Response.SerializeToString, 102 | ), 103 | } 104 | generic_handler = grpc.method_handlers_generic_handler( 105 | 'demo.GRPCDemo', rpc_method_handlers) 106 | server.add_generic_rpc_handlers((generic_handler,)) 107 | -------------------------------------------------------------------------------- /states/files/grpc_data_transmission/server.py: -------------------------------------------------------------------------------- 1 | # Copyright 2019 gRPC authors. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """The example of four ways of data transmission using gRPC in Python.""" 15 | 16 | from threading import Thread 17 | from concurrent import futures 18 | 19 | import grpc 20 | import demo_pb2_grpc 21 | import demo_pb2 22 | 23 | SERVER_ADDRESS = 'localhost:23333' 24 | SERVER_ID = 1 25 | 26 | 27 | class DemoServer(demo_pb2_grpc.GRPCDemoServicer): 28 | 29 | # 一元模式(在一次调用中, 客户端只能向服务器传输一次请求数据, 服务器也只能返回一次响应) 30 | # unary-unary(In a single call, the client can only send request once, and the server can 31 | # only respond once.) 32 | def SimpleMethod(self, request, context): 33 | print("SimpleMethod called by client(%d) the message: %s" % 34 | (request.client_id, request.request_data)) 35 | response = demo_pb2.Response( 36 | server_id=SERVER_ID, 37 | response_data="Python server SimpleMethod Ok!!!!") 38 | return response 39 | 40 | # 客户端流模式（在一次调用中, 客户端可以多次向服务器传输数据, 但是服务器只能返回一次响应） 41 | # stream-unary (In a single call, the client can transfer data to the server several times, 42 | # but the server can only return a response once.) 43 | def ClientStreamingMethod(self, request_iterator, context): 44 | print("ClientStreamingMethod called by client...") 45 | for request in request_iterator: 46 | print("recv from client(%d), message= %s" % 47 | (request.client_id, request.request_data)) 48 | response = demo_pb2.Response( 49 | server_id=SERVER_ID, 50 | response_data="Python server ClientStreamingMethod ok") 51 | return response 52 | 53 | # 服务端流模式（在一次调用中, 客户端只能一次向服务器传输数据, 但是服务器可以多次返回响应） 54 | # unary-stream (In a single call, the client can only transmit data to the server at one time, 55 | # but the server can return the response many times.) 56 | def ServerStreamingMethod(self, request, context): 57 | print("ServerStreamingMethod called by client(%d), message= %s" % 58 | (request.client_id, request.request_data)) 59 | 60 | # 创建一个生成器 61 | # create a generator 62 | def response_messages(): 63 | for i in range(5): 64 | response = demo_pb2.Response( 65 | server_id=SERVER_ID, 66 | response_data=("send by Python server, message=%d" % i)) 67 | yield response 68 | 69 | return response_messages() 70 | 71 | # 双向流模式 (在一次调用中, 客户端和服务器都可以向对方多次收发数据) 72 | # stream-stream (In a single call, both client and server can send and receive data 73 | # to each other multiple times.) 74 | def BidirectionalStreamingMethod(self, request_iterator, context): 75 | print("BidirectionalStreamingMethod called by client...") 76 | 77 | # 开启一个子线程去接收数据 78 | # Open a sub thread to receive data 79 | def parse_request(): 80 | for request in request_iterator: 81 | print("recv from client(%d), message= %s" % 82 | (request.client_id, request.request_data)) 83 | 84 | t = Thread(target=parse_request) 85 | t.start() 86 | 87 | for i in range(5): 88 | yield demo_pb2.Response( 89 | server_id=SERVER_ID, 90 | response_data=("send by Python server, message= %d" % i)) 91 | 92 | t.join() 93 | 94 | 95 | def main(): 96 | server = grpc.server(futures.ThreadPoolExecutor()) 97 | 98 | demo_pb2_grpc.add_GRPCDemoServicer_to_server(DemoServer(), server) 99 | 100 | server.add_insecure_port(SERVER_ADDRESS) 101 | print("------------------start Python GRPC server") 102 | server.start() 103 | server.wait_for_termination() 104 | 105 | # If raise Error: 106 | # AttributeError: '_Server' object has no attribute 'wait_for_termination' 107 | # You can use the following code instead: 108 | # import time 109 | # while 1: 110 | # time.sleep(10) 111 | 112 | 113 | if __name__ == '__main__': 114 | main() 115 | -------------------------------------------------------------------------------- /states/files/perfrun.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # vim: tabstop=4 expandtab 3 | 4 | set -e 5 | 6 | locations="/modsec-off /modsec-light /modsec-full" 7 | iterations=10 8 | duration=10s 9 | threads=1 10 | connections=50 11 | 12 | function run { 13 | sudo service nginx stop 14 | 15 | echo "Running wrk, please be patient" 16 | 17 | for location in ${locations}; do 18 | rm -f /tmp${location}.out 19 | i=1 20 | while [ $i -le ${iterations} ]; do 21 | echo "[${i}/${iterations}] $location ..." 22 | sudo service nginx start 23 | sleep 1 24 | wrk -t${threads} -c${connections} -d${duration} -s ${HOME}/report.lua \ 25 | -H "Host: www.example.com" \ 26 | -H "User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5 (.NET CLR 3.5.30729)" \ 27 | -H "Accept: text/html,application/xhtml+xml,application/xml; q=0.9,*/*;q=0.8" \ 28 | -H "Accept-Language: en-us,en;q=0.5" \ 29 | -H "Accept-Encoding: gzip,deflate" \ 30 | -H "Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7" \ 31 | -H "Keep-Alive: 300" \ 32 | -H "Connection: keep-alive" \ 33 | -H "Cookie: PHPSESSID=r2t5uvjq435r4q7ib3vtdjq120" \ 34 | -H "Pragma: no-cache" \ 35 | -H "Cache-Control: no-cache" \ 36 | "http://localhost${location}/test.pl?param1=test¶2=test2" 2>&1 >>/tmp${location}.out 37 | sudo service nginx stop 38 | sleep 1 39 | i=$((i+1)) 40 | done 41 | done 42 | } 43 | 44 | function stats { 45 | for location in ${locations}; do 46 | echo "Summary for ${location}, RPS (count):" 47 | awk '{if ($1 == "rps:") {print $2}}' /tmp${location}.out | ministat -n -w 80 | fgrep -v stdin 48 | echo " latency (ms)" 49 | awk '{if ($1 == "latency.avg:") {print $2}}' /tmp${location}.out | ministat -n -w 80 | tail -1 50 | echo 51 | done 52 | } 53 | 54 | case $1 in 55 | run) run ;; 56 | stats) stats ;; 57 | *) echo "usage: `basename $0` run|stats" ;; 58 | esac 59 | 60 | 61 | -------------------------------------------------------------------------------- /states/files/upload-app/upload-app.ini: -------------------------------------------------------------------------------- 1 | [uwsgi] 2 | uid = nginx 3 | gid = nginx 4 | socket = 0.0.0.0:3131 5 | max-requests = 4000 6 | buffer-size = 16384 7 | master = true 8 | log-master = true 9 | die-on-term = true 10 | vacuum = true 11 | file = /data/upload-app/upload-app.py 12 | pidfile = /data/upload-app/upload-app.pid 13 | master-fifo = /data/upload-app/upload-app-fifo 14 | logto = /data/upload-app/upload-app-uwsgi.log 15 | workers = 2 16 | -------------------------------------------------------------------------------- /states/files/upload-app/upload-app.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import hashlib 4 | 5 | def application(environ, start_response): 6 | environ_summary = "ENVIRON:\n\n" + "\n".join(["{0}: {1}".format(k, environ[k]) for k in sorted(environ.keys())]) + "\n" 7 | log = environ['wsgi.errors'] 8 | 9 | body = '' 10 | try: 11 | length = int(environ.get('CONTENT_LENGTH', '0')) 12 | except ValueError: 13 | length = 0 14 | 15 | if length != 0: 16 | body = environ['wsgi.input'].read(length) 17 | m = hashlib.md5() 18 | m.update(body) 19 | response_body = "%d:%s\n" % (len(body), m.hexdigest()) 20 | log.write("content-type: %s, body.len=%d, body.md5=%s\n" % (environ['CONTENT_TYPE'], len(body), m.hexdigest())) 21 | else: 22 | response_body = environ_summary 23 | 24 | status = '200 OK' 25 | response_headers = [('Content-Type', 'text/plain'), 26 | ('Content-Length', str(len(response_body)))] 27 | 28 | start_response(status, response_headers) 29 | 30 | return [response_body.encode()] 31 | 32 | if __name__ == '__main__': 33 | try: 34 | from wsgiref.simple_server import make_server 35 | httpd = make_server('', 8080, application) 36 | print('Serving on port 8080...') 37 | httpd.serve_forever() 38 | except KeyboardInterrupt: 39 | print('Goodbye.') 40 | -------------------------------------------------------------------------------- /states/files/upload-app/upload-app.service: -------------------------------------------------------------------------------- 1 | [Unit] 2 | Description=upload app 3 | After=syslog.target 4 | 5 | [Service] 6 | ExecStart=/usr/bin/uwsgi_python3 --ini /data/upload-app/upload-app.ini 7 | ExecReload=/bin/kill -HUP $MAINPID 8 | RuntimeDirectory=uwsgi 9 | Restart=always 10 | KillSignal=SIGTERM 11 | Type=notify 12 | StandardError=syslog 13 | NotifyAccess=all 14 | 15 | [Install] 16 | WantedBy=multi-user.target 17 | -------------------------------------------------------------------------------- /states/grpc.sls: -------------------------------------------------------------------------------- 1 | include: 2 | - nginx 3 | 4 | python3-pip package: 5 | pkg.latest: 6 | - name: python3-pip 7 | 8 | {%- for module in ('grpcio', 'protobuf') %} 9 | {{ module }} for python3: 10 | cmd.run: 11 | - name: pip3 install {{ module }} 12 | - unless: pip3 list | grep {{ module }} 13 | - require: 14 | - python3-pip package 15 | {%- endfor %} 16 | 17 | {%- for f in ('client.py', 'server.py', 'demo.proto', 'demo_pb2_grpc.py', 'demo_pb2.py') %} 18 | /home/test/grpc_data_transmission/{{ f }}: 19 | file.managed: 20 | - name: /home/test/grpc_data_transmission/{{ f }} 21 | - mode: 0644 22 | - makedirs: True 23 | - user: test 24 | - group: test 25 | - source: salt://files/grpc_data_transmission/{{ f }} 26 | {%- endfor %} 27 | 28 | /etc/nginx/conf.d/grpc-proxy.conf: 29 | file.managed: 30 | - source: salt://files/etc/nginx/conf.d/grpc-proxy.conf 31 | - watch_in: 32 | - service: NGINX service 33 | -------------------------------------------------------------------------------- /states/makeenv.sls: -------------------------------------------------------------------------------- 1 | include: 2 | - user 3 | 4 | Dependency packages: 5 | pkg.latest: 6 | - pkgs: 7 | - automake 8 | - autoconf 9 | - bison 10 | - curl 11 | - flex 12 | - g++ 13 | - gcc 14 | - git 15 | - htop 16 | - libcurl4-openssl-dev 17 | - libgeoip-dev 18 | - libpcre2-dev 19 | - libpcre3-dev 20 | - libssl-dev 21 | - libtool 22 | - libxml2-dev 23 | - libyajl-dev 24 | - make 25 | - man-db 26 | - mercurial 27 | - ministat 28 | - net-tools 29 | - pkg-config 30 | - uwsgi-plugin-python3 31 | - zlib1g-dev 32 | 33 | Makefile: 34 | file.managed: 35 | - user: test 36 | - name: /home/test/Makefile 37 | - source: salt://files/Makefile.tmpl 38 | - template: jinja 39 | - context: 40 | nginxver: {{ salt['pillar.get']('versions:nginx') }} 41 | lmsrev: {{ salt['pillar.get']('versions:libmodsecurity') }} 42 | connectorrev: {{ salt['pillar.get']('versions:connector') }} 43 | wrkrev: {{ salt['pillar.get']('versions:wrk') }} 44 | - require: 45 | - Test user 46 | 47 | prove configuration for test: 48 | file.managed: 49 | - user: test 50 | - name: /home/test/.proverc 51 | - contents: | 52 | -j16 53 | 54 | /etc/motd: 55 | file.managed: 56 | - contents: | 57 | In order to extend this environment with a set of debug utilities, 58 | such as gdb, valgrind, systemtap, perf and others, please run: 59 | 60 | sudo salt-call state.apply debugenv 61 | 62 | Please note that this action will also install debug symbols 63 | for the current running kernel, it may take some time. 64 | 65 | Nikto web server security scanner tool is available for testing, e.g.: 66 | 67 | nikto -host localhost -root /modsec-off/ 68 | nikto -host localhost -root /modsec-full/ 69 | 70 | -------------------------------------------------------------------------------- /states/nginx.sls: -------------------------------------------------------------------------------- 1 | {% set os = salt['grains.get']('os').lower() %} 2 | {% set release = salt['grains.get']('lsb_distrib_codename', 'focal') %} 3 | {% set nginxver = salt['pillar.get']('versions:nginx') %} 4 | 5 | include: 6 | - build 7 | 8 | NGINX Package Repository: 9 | pkgrepo.managed: 10 | - humanname: NGINX Package Repository 11 | - name: deb http://nginx.org/packages/mainline/{{ os }} {{ release }} nginx 12 | - dist: {{ release }} 13 | - file: /etc/apt/sources.list.d/nginx-oss.list 14 | - clean_file: True 15 | - gpgcheck: 1 16 | - keyid: '0xABF5BD827BD9BF62' 17 | - keyserver: keyserver.ubuntu.com 18 | 19 | NGINX Package: 20 | pkg.installed: 21 | - name: nginx 22 | - version: {{ nginxver }}-1~{{ release }} 23 | - require: 24 | - pkgrepo: NGINX Package Repository 25 | 26 | NGINX empty default.conf: 27 | file.managed: 28 | - name: /etc/nginx/conf.d/default.conf 29 | - contents: '' 30 | - contents_newline: False 31 | - require: 32 | - NGINX Package 33 | 34 | NGINX Debug Symbols: 35 | pkg.installed: 36 | - name: nginx-dbg 37 | - version: {{ nginxver }}-1~{{ release }} 38 | - require: 39 | - pkgrepo: NGINX Package Repository 40 | 41 | /etc/nginx/nginx.conf: 42 | file.managed: 43 | - source: salt://files/etc/nginx/nginx.conf 44 | 45 | /etc/nginx/modsec/main.conf: 46 | file.managed: 47 | - source: salt://files/etc/nginx/modsec/main.conf 48 | 49 | /etc/nginx/modsec/modsecurity.conf: 50 | cmd.run: 51 | - name: cat /home/test/ModSecurity/modsecurity.conf-recommended | grep -v "^#" | strings | sed -e 's#^SecRuleEngine.*#SecRuleEngine On#' > /etc/nginx/modsec/modsecurity.conf 52 | - unless: test -e /etc/nginx/modsec/modsecurity.conf 53 | - require: 54 | - Build all 55 | 56 | /etc/nginx/modsec/unicode.mapping: 57 | file.copy: 58 | - name: /etc/nginx/modsec/unicode.mapping 59 | - source: /home/test/ModSecurity/unicode.mapping 60 | - require: 61 | - Build all 62 | 63 | /root/gencert.sh: 64 | file.managed: 65 | - source: salt://files/gencert.sh 66 | - mode: 0755 67 | 68 | Create Self-Signed Certificate: 69 | cmd.run: 70 | - name: /root/gencert.sh localhost 71 | - cwd: /etc/nginx 72 | - unless: test -e /etc/nginx/localhost.pem 73 | - require: 74 | - /root/gencert.sh 75 | 76 | NGINX service: 77 | service.running: 78 | - name: nginx 79 | - enable: True 80 | - reload: True 81 | - watch: 82 | - /etc/nginx/nginx.conf 83 | - /etc/nginx/modsec/main.conf 84 | - /etc/nginx/modsec/modsecurity.conf 85 | - /etc/nginx/modsec/unicode.mapping 86 | - require: 87 | - Create Self-Signed Certificate 88 | -------------------------------------------------------------------------------- /states/owasp-crs.sls: -------------------------------------------------------------------------------- 1 | OWASP CRS: 2 | git.latest: 3 | - name: https://github.com/coreruleset/coreruleset.git 4 | - target: /etc/nginx/modsec/owasp-crs 5 | - rev: {{ salt['pillar.get']('versions:owasp-crs') }} 6 | 7 | Default crs-setup.conf: 8 | file.symlink: 9 | - name: /etc/nginx/modsec/owasp-crs/crs-setup.conf 10 | - target: /etc/nginx/modsec/owasp-crs/crs-setup.conf.example 11 | - require: 12 | - OWASP CRS 13 | -------------------------------------------------------------------------------- /states/system.sls: -------------------------------------------------------------------------------- 1 | /etc/security/limits.d/global-overrides.conf: 2 | file.managed: 3 | - mode: 0644 4 | - makedirs: True 5 | - contents: | 6 | * soft core unlimited 7 | * hard core unlimited 8 | * soft nofile 131072 9 | * hard nofile 131072 10 | 11 | global systemd overrides: 12 | file.managed: 13 | - name: /etc/systemd/system.conf.d/global-overrides.conf 14 | - mode: 0644 15 | - makedirs: True 16 | - contents: | 17 | [Manager] 18 | DefaultLimitCORE=infinity 19 | DefaultLimitNOFILE=131072 20 | module.run: 21 | - name: service.systemctl_reload 22 | - onchanges: 23 | - file: global systemd overrides 24 | 25 | kernel.core_pattern: 26 | sysctl.present: 27 | - value: /tmp/%e.%p.core 28 | 29 | net.ipv4.ip_local_port_range: 30 | sysctl.present: 31 | - value: 1024 64999 32 | 33 | net.ipv4.tcp_tw_reuse: 34 | sysctl.present: 35 | - value: 1 36 | 37 | net.core.somaxconn: 38 | sysctl.present: 39 | - value: 32768 40 | -------------------------------------------------------------------------------- /states/top.sls: -------------------------------------------------------------------------------- 1 | base: 2 | '*': 3 | - system 4 | - user 5 | - makeenv 6 | - build 7 | - owasp-crs 8 | - nginx 9 | - upload-app 10 | - benchenv 11 | -------------------------------------------------------------------------------- /states/upload-app.sls: -------------------------------------------------------------------------------- 1 | include: 2 | - nginx 3 | 4 | /data/upload-app: 5 | file.directory: 6 | - user: nginx 7 | - group: nginx 8 | - mode: 755 9 | - makedirs: True 10 | 11 | /data/upload-app/upload-app.ini: 12 | file.managed: 13 | - source: salt://files/upload-app/upload-app.ini 14 | - user: nginx 15 | - group: nginx 16 | - mode: 644 17 | 18 | /data/upload-app/upload-app.py: 19 | file.managed: 20 | - source: salt://files/upload-app/upload-app.py 21 | - user: nginx 22 | - group: nginx 23 | - mode: 755 24 | 25 | upload-app systemd service: 26 | file.managed: 27 | - name: /lib/systemd/system/upload-app.service 28 | - mode: 0644 29 | - makedirs: True 30 | - source: salt://files/upload-app/upload-app.service 31 | module.run: 32 | - name: service.systemctl_reload 33 | - onchanges: 34 | - file: upload-app systemd service 35 | 36 | upload-app: 37 | service.running: 38 | - enable: True 39 | - reload: True 40 | - watch: 41 | - file: /data/upload-app/upload-app.ini 42 | -------------------------------------------------------------------------------- /states/user.sls: -------------------------------------------------------------------------------- 1 | Test user: 2 | user.present: 3 | - name: test 4 | - empty_password: True 5 | - shell: /bin/bash 6 | 7 | sudo for test user: 8 | file.managed: 9 | - name: /etc/sudoers.d/01-sudo-for-test 10 | - mode: 0440 11 | - contents: test ALL=(ALL) NOPASSWD:ALL 12 | --------------------------------------------------------------------------------