├── demo.png ├── AutoRL.png ├── demo-cf.png ├── README.md └── autorl.py /demo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/n0vad3v/AutoRL/HEAD/demo.png -------------------------------------------------------------------------------- /AutoRL.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/n0vad3v/AutoRL/HEAD/AutoRL.png -------------------------------------------------------------------------------- /demo-cf.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/n0vad3v/AutoRL/HEAD/demo-cf.png -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # AutoRL 2 | 3 | This is a PoC of automatically block traffic on Cloudflare's side based on Nginx Log parsing. 4 | 5 | It will evaluate Nginx `access.log` and find potential CC pattern, and block them on Cloudflare's side and send a message(alert) to Telegram Group. 6 | 7 | ## Topology 8 | 9 | With Cloudflare Argo Tunnel, we can set security group to allow inbound traffic for SSH only, this can guarantee the Host's IP will not be exposed to the Internet (ref: [Use Cloudflare Argo Tunnel (cloudflared) to accelerate and protect your website.](https://nova.moe/accelerate-and-secure-with-cloudflared-en/)), however, attackers can still CC your website by sending enormous requests cocurrently, AutoRL is here trying to mitigate this problem. 10 | 11 | ![](./AutoRL.png) 12 | 13 | ## Prerequisites 14 | 15 | Since this is only a PoC, the following condition must be met to use AutoRL. 16 | 17 | * Python 3 installed on Host 18 | * Nginx used for Reverse proxy and all the logs are logged into one `access.log` file. 19 | * Nginx has the following log format (in `/etc/nginx/nginx.conf`) 20 | 21 | ``` 22 | log_format main '$remote_addr $time_iso8601 "$request" $server_name ' 23 | '$status $body_bytes_sent "$http_referer" ' 24 | '"$http_user_agent" "$http_x_forwarded_for"'; 25 | ``` 26 | On this condition, the raw log should look like this: 27 | ``` 28 | 108.162.245.152 2022-05-05T10:14:19+08:00 "GET /grafana/api/live/ws HTTP/1.1" xxxx.yyyy.tld 400 12 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko)" "145.xx.xxx.xxx" 29 | ``` 30 | Where, `108.162.245.152` is Cloudflare's IP,`xxxx.yyyy.tld` is the requested domain , `2022-05-02T10:44:16+08:00` stands for request datetime and `"145.xx.xx.xxx"` is the real visitor IP. 31 | 32 | 33 | ## Usage 34 | 35 | 1. Download the `autorl.py` to your host 36 | 2. Edit the following variable in the `autorl.py` 37 | 38 | * CF_EMAIL (Your Cloudflare login email) 39 | * CF_AUTH_KEY (Your Cloudflare Global API Key) 40 | * ACCESS_LOG_PATH (Default is `/var/log/nginx/access.log`) 41 | * INTERVAL_MIN (Default is 1, then this script will evaluate for 1min's traffic) 42 | * RATE_PER_MINUTE (How many requests are allowed for single IP, e,g, when this is set to 600 and `INTERVAL_MIN` is 1, then one IP can send at most 600 requests, after that, this IP will be blocked.) 43 | * TG_CHAT_ID (Your Telegram Chat Group ID, optional) 44 | * TG_BOT_TOKEN (You should invite a bot to your group, and fillin the bot token here, optional) 45 | * IP_WHITE_LIST (If you'd like to whitelist some IP, fillin here) 46 | 3. Create a crontab for this script, example: 47 | ``` 48 | * * * * * for i in {1..6}; do /usr/bin/python3 /path/to/autorl.py & sleep 10; done 49 | ``` 50 | 51 | ## Demo 52 | 53 | On Telegram side: 54 | 55 | ![](./demo.png) 56 | 57 | On Cloudflare side: 58 | 59 | ![](./demo-cf.png) 60 | 61 | ## Notes 62 | 63 | * Blocked IP address will never gets unblocked. 64 | * If logrotate is not setup correctly, then parsing the whole `access.log` might consume a lot of system resources. 65 | * The attack pattern/sample is not stored so we have no idea how the attack is conducted. -------------------------------------------------------------------------------- /autorl.py: -------------------------------------------------------------------------------- 1 | import requests 2 | import json 3 | import datetime 4 | import socket 5 | 6 | prefix = "https://api.cloudflare.com/client/v4/" 7 | 8 | ## BEGIN CONFIGURATION 9 | CF_EMAIL = "" 10 | CF_AUTH_KEY = "" 11 | DEFAULT_MODE = "block" 12 | 13 | ACCESS_LOG_PATH = "/var/log/nginx/access.log" 14 | INTERVAL_MIN = 1 15 | RATE_PER_MINUTE = 600 16 | 17 | TG_CHAT_ID = "" 18 | TG_BOT_TOKEN = "" 19 | 20 | IP_WHITE_LIST = ["1.1.1.1"] 21 | ## END CONFIGURATION 22 | 23 | def get_existing_rules(): 24 | path = "user/firewall/access_rules/rules?per_page=10" 25 | header = { 26 | "X-Auth-Email": CF_EMAIL, 27 | "X-Auth-Key": CF_AUTH_KEY, 28 | "Content-Type": "application/json" 29 | } 30 | r = requests.get(prefix + path, headers=header) 31 | total_pages = r.json()['result_info']['total_pages'] 32 | rules = [] 33 | for page in range(1, total_pages + 1): 34 | path = "user/firewall/access_rules/rules?per_page=1000&page=" + str(page) 35 | r = requests.get(prefix + path, headers=header) 36 | rules += r.json()['result'] 37 | return rules 38 | 39 | def add_ip_to_block_rule(ip_addr, domain): 40 | path = "user/firewall/access_rules/rules" 41 | header = { 42 | "X-Auth-Email": CF_EMAIL, 43 | "X-Auth-Key": CF_AUTH_KEY, 44 | "Content-Type": "application/json" 45 | } 46 | data = { 47 | "mode": DEFAULT_MODE, 48 | "configuration": { 49 | "target": "ip", 50 | "value": ip_addr 51 | }, 52 | "notes": "Auto blocked by AutoRL on " + datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S") + " for " + domain 53 | } 54 | r = requests.post(prefix + path, headers=header, data=json.dumps(data)) 55 | return r.json() 56 | 57 | def remove_ip_from_block_rule(ip_addr): 58 | existing_rules = get_existing_rules() 59 | for rule in existing_rules['result']: 60 | if rule['configuration']['value'] == ip_addr: 61 | path = "user/firewall/access_rules/rules/" + rule['id'] 62 | header = { 63 | "X-Auth-Email": CF_EMAIL, 64 | "X-Auth-Key": CF_AUTH_KEY, 65 | "Content-Type": "application/json" 66 | } 67 | r = requests.delete(prefix + path, headers=header) 68 | return r.json() 69 | 70 | def parse_nginx_log(log_path): 71 | with open(log_path, 'r') as f: 72 | lines = f.readlines() 73 | 74 | # NGINX format is 2022-04-30T20:15:18+08:00 75 | datetime_format = "%Y-%m-%dT%H:%M:%S+08:00" 76 | 77 | ip_addr_counter = {} 78 | ip_domain_counter = {} 79 | for line in lines: 80 | ip_addr = line.rsplit(' ', 1)[1].strip().replace('"', '') 81 | log_datetime = datetime.datetime.strptime(line.split(' ')[1], datetime_format) 82 | requested_domain = line.split(' ')[5].strip() 83 | if (datetime.datetime.now() - log_datetime).total_seconds() / 60 < INTERVAL_MIN: 84 | if "-" in ip_addr or ip_addr in IP_WHITE_LIST: 85 | continue 86 | else: 87 | if ip_addr not in ip_addr_counter: 88 | ip_addr_counter[ip_addr] = 1 89 | else: 90 | ip_addr_counter[ip_addr] += 1 91 | 92 | if ip_addr not in ip_domain_counter: 93 | ip_domain_counter[ip_addr] = {} 94 | if requested_domain not in ip_domain_counter[ip_addr]: 95 | ip_domain_counter[ip_addr][requested_domain] = 1 96 | else: 97 | ip_domain_counter[ip_addr][requested_domain] += 1 98 | return ip_addr_counter, ip_domain_counter 99 | 100 | def get_bad_ips(ip_addr_counter,ip_domain_counter): 101 | bad_ip_list = [] 102 | bad_ip_visit_count = [] 103 | bad_ip_visited_top_domain = [] 104 | for ip_addr, count in ip_addr_counter.items(): 105 | if count > RATE_PER_MINUTE: 106 | bad_ip_list.append(ip_addr) 107 | bad_ip_visit_count.append(count) 108 | 109 | bad_ip_domain_list = ip_domain_counter[ip_addr] 110 | bad_ip_visited_top_domain.append(max(bad_ip_domain_list, key=bad_ip_domain_list.get)) 111 | 112 | return bad_ip_list, bad_ip_visit_count, bad_ip_visited_top_domain 113 | 114 | def send_message_to_telegram(chat_id, text): 115 | url = "https://api.telegram.org/bot" + TG_BOT_TOKEN + "/sendMessage" 116 | data = { 117 | "chat_id": chat_id, 118 | "text": text 119 | } 120 | r = requests.post(url, data=data) 121 | return r.json() 122 | 123 | if __name__ == '__main__': 124 | ip_addr_counter, ip_domain_counter = parse_nginx_log(ACCESS_LOG_PATH) 125 | bad_ip_list, bad_ip_visit_count, bad_ip_visited_top_domain = get_bad_ips(ip_addr_counter,ip_domain_counter) 126 | for i in range(len(bad_ip_list)): 127 | msg = "On Host: " + socket.gethostname() + ", with IP " + bad_ip_list[i] + " has accessed domain " + str(bad_ip_visited_top_domain[i]) + " for " + str(bad_ip_visit_count[i]) + " times in the last " + str(INTERVAL_MIN) + " minutes, now blocked." 128 | add_ip_to_block_rule(bad_ip_list[i], bad_ip_visited_top_domain[i]) 129 | if TG_CHAT_ID != "": 130 | send_message_to_telegram(TG_CHAT_ID, msg) --------------------------------------------------------------------------------