├── README.md ├── config.ini ├── ddos_detect.py └── ip-white-list.txt /README.md: -------------------------------------------------------------------------------- 1 | # DDoS-detect 2 | NetFlow data based DDoS detection tool. 3 | It uses nfdump package to analyse NetFlow data and detects possible DDoS attack. 4 | If DDoS attack is occurred it sends an e-mail with victim's ip-address. 5 | 6 | ### Prerequisites 7 | - Linux or FreeBSD, 8 | - Python > 3.6, 9 | - NFDump 10 | 11 | 1. Configure NetFlow (v5/v9/ipfix) on a network device with active timeout 60 sec (see the v5 version JunOS config example below). 12 | ``` 13 | forwarding-options { 14 | sampling { 15 | input { 16 | rate 2000; 17 | run-length 0; 18 | } 19 | family inet { 20 | output { 21 | flow-inactive-timeout 15; 22 | flow-active-timeout 60; 23 | flow-server 10.0.0.10 { 24 | port 9999; 25 | autonomous-system-type origin; 26 | source-address 10.0.0.1; 27 | version 5; 28 | } 29 | } 30 | } 31 | } 32 | } 33 | ``` 34 | 2. Install nfdump on a server and configure it to rotate files every minute (see options example below) 35 | ``` 36 | options='-z -t 60 -w -D -T all -l /var/db/flows/ -I any -S 1 -P /var/run/nfcapd.allflows.pid -p 9999 -b 10.0.0.10 -e 10G' 37 | ``` 38 | 39 | ### Installation 40 | Configure some settings in the 'config.ini' file. 41 | 1. Check the log options 42 | ``` 43 | [SYSTEM] 44 | LogDir = /var/log/ 45 | # log file size in bytes 46 | LogFileSize = 50000 47 | ``` 48 | Create the log file 'LogDir'/ddos-detect.log and be sure it's writable by the user that’s running the DDoS-detect. 49 | 50 | 2. Specify the location of the binary and NetFlow statistics files. 51 | ``` 52 | [FILES] 53 | NfdumpBinDir = /usr/local/bin/ 54 | SysBinDir = /usr/bin/ 55 | FlowsDir = /var/db/flows/ 56 | ``` 57 | 3. Configure email settings. 58 | ``` 59 | [EMAIL] 60 | SMTPServer = localhost 61 | MailFrom = mail@example.com 62 | MailTo = mail@example.com 63 | ``` 64 | 4. DDoS-detect uses four report profiles to detect abnormal traffic. Configure 'threshold's for this profiles based on your traffic activity and network device sampling options or left them default. You can change set of the current working profiles and configure options for them in the 'REPORTS' section config.ini file. 65 | ``` 66 | [REPORTS] 67 | sdport_flows 68 | dport_packets 69 | flows 70 | packets 71 | 72 | [sdport_flows] 73 | threshold = 300 74 | key_field = 4 75 | 76 | [dport_packets] 77 | threshold = 3000 78 | key_field = 3 79 | 80 | [flows] 81 | threshold = 1500 82 | key_field = 2 83 | 84 | [packets] 85 | threshold = 5000 86 | key_field = 2 87 | ``` 88 | - 'key_field' is a report field index number to which the threshold applies (you don't have to change it). 89 | 5. Check the IP WhiteList ACL filename 90 | ``` 91 | [FILES] 92 | # White List ACL 93 | IpWhiteListFile = ip-white-list.txt 94 | ``` 95 | And add to the file IP addresses you want to exclude from DDoS-detection (or leave file blank) 96 | ``` 97 | # Add local ip/net to exclude it from checking on DDoS 98 | 127.0.0.1 99 | 127.0.0.0/8 100 | ``` 101 | 102 | 6. After that put script to a cron to execute it every minute. 103 | 104 | 105 | ## Authors 106 | 107 | * **Evgeniy Kolosov** - [owenear](https://github.com/owenear) 108 | -------------------------------------------------------------------------------- /config.ini: -------------------------------------------------------------------------------- 1 | [SYSTEM] 2 | Code = latin-1 3 | Whois = True 4 | LogDir = /var/log/ 5 | # log file size in bytes 6 | LogFileSize = 50000 7 | 8 | 9 | [FILES] 10 | # Dir with nfdump binary files 11 | NfdumpBinDir = /usr/local/bin/ 12 | # Dir with Whois, AWK binary files 13 | SysBinDir = /usr/bin/ 14 | FlowsDir = /backup/nfcapd/ 15 | # White List ACL 16 | IpWhiteListFile = ip-white-list.txt 17 | 18 | 19 | [EMAIL] 20 | SMTPServer = localhost 21 | # 0-default port 22 | SMTPPort = 0 23 | MailFrom = mail@example.com 24 | MailTo = mail@example.com 25 | Subject = [DDoS Detect] 26 | # Amount of flows records in an email 27 | FlowRecAmount = 50 28 | # Use TLS (if True, Auth must be the same) 29 | Secure = False 30 | # Use Login and Password 31 | Auth = False 32 | # Notification Frequency about current DDoS attack - every value minutes 33 | # if 0 - don't send email, print victims ip to stdout 34 | NotifyFreq = 10 35 | 36 | 37 | # Act if Auth = True 38 | [EMAIL-AUTH] 39 | Login = ... 40 | Password = ... 41 | 42 | # DDoS Reports(rules) 43 | # Reports 44 | [REPORTS] 45 | sdport_flows 46 | dport_packets 47 | flows 48 | packets 49 | 50 | # Reports(rules) options 51 | # threshold - threshold value 52 | # key_field - report field index number to which the threshold applies (you don't have to change it) 53 | [sdport_flows] 54 | threshold = 250 55 | key_field = 4 56 | 57 | [dport_packets] 58 | threshold = 3000 59 | key_field = 3 60 | 61 | [flows] 62 | threshold = 3000 63 | key_field = 2 64 | 65 | [packets] 66 | threshold = 3000 67 | key_field = 2 68 | 69 | -------------------------------------------------------------------------------- /ddos_detect.py: -------------------------------------------------------------------------------- 1 | import os 2 | import subprocess 3 | from datetime import datetime, timedelta 4 | from smtplib import SMTP_SSL, SMTP 5 | from configparser import ConfigParser 6 | import re 7 | import logging.handlers 8 | from pathlib import Path 9 | 10 | 11 | # Load and Check Config 12 | if os.path.exists(os.path.join(os.path.abspath(os.path.dirname(__file__)), 'config.ini')): 13 | config = ConfigParser(allow_no_value=True) 14 | config.read(os.path.join(os.path.abspath(os.path.dirname(__file__)), 'config.ini')) 15 | else: 16 | print("ERROR: Configuration file 'config.ini' not found!") 17 | exit(1) 18 | 19 | try: 20 | LOG_FILE = os.path.join(config['SYSTEM']['LogDir'], 'ddos-detect.log') 21 | LOG_FILE_SIZE = int(config['SYSTEM']['LogFileSize']) 22 | CODE = config['SYSTEM']['code'] 23 | # AWK 24 | AWK = os.path.join(config['FILES']['SysBinDir'], 'awk') 25 | # Reports options 26 | REPORTS = {} 27 | for key in config['REPORTS']: 28 | REPORTS.update({key:{}}) 29 | REPORTS[key].update({'threshold': int(config[key]['threshold'])}) 30 | REPORTS[key].update({'key_field': int(config[key]['key_field'])}) 31 | 32 | NFDUMP = os.path.join(config['FILES']['NfdumpBinDir'], 'nfdump') 33 | FLOWS_DIR = config['FILES']['FlowsDir'] 34 | FLOW_REC_AMOUNT = config['EMAIL']['FlowRecAmount'] 35 | IP_WHITE_LIST_FILE = os.path.join(os.path.abspath(os.path.dirname(__file__)), config['FILES']['IpWhiteListFile']) 36 | 37 | # E-mail notify freq 38 | NOTIFY_FREQ = int(config['EMAIL']['NotifyFreq']) 39 | except KeyError as e: 40 | print(f"ERROR: Wrong configuration in 'config.ini'!: {e} Not Found") 41 | exit(1) 42 | 43 | # Logging configuration 44 | logger = logging.getLogger("mainlog") 45 | logger.setLevel(logging.INFO) 46 | fh = logging.handlers.RotatingFileHandler(LOG_FILE, maxBytes=LOG_FILE_SIZE, backupCount=5) 47 | fh.setFormatter(logging.Formatter('%(asctime)s %(message)s')) 48 | logger.addHandler(fh) 49 | 50 | 51 | def log(msg = '', notify_counter = 0, level = 6): 52 | if os.path.getsize(LOG_FILE) == 0: 53 | logger.info("LogFile created; NotificationCounter: 0") 54 | # Return Notification minute counter if msg is empty 55 | if not msg: 56 | with open(LOG_FILE, 'r', encoding=CODE) as f: 57 | match = re.findall(r'NotificationCounter: (\w*)', f.read()) 58 | if match: 59 | return int(match[-1]) 60 | else: 61 | return 0 62 | if level == 3: 63 | logger.error(f"ERROR: {msg}") 64 | else: 65 | logger.info(f"{msg}; NotificationCounter: {notify_counter}") 66 | 67 | 68 | def whois_net(ip): 69 | try: 70 | whois = subprocess.run([os.path.join(config['FILES']['SysBinDir'], 'whois'), ip], stdout=subprocess.PIPE) 71 | except Exception as e: 72 | log(f"While runing shell 'whois': {e}", level=3) 73 | else: 74 | netname = re.findall(r'(?:[nN]et|[Oo]rg)-?[Nn]ame: +(.+)', whois.stdout.decode(CODE)) 75 | if netname: 76 | return '; '.join(netname) 77 | return '---' 78 | 79 | 80 | def send_mail(msg, ip_set): 81 | try: 82 | if config['EMAIL']['Secure'].lower() == 'true': 83 | server = SMTP_SSL(config['EMAIL']['SMTPServer'], int(config['EMAIL']['SMTPPort'])) 84 | else: 85 | server = SMTP(config['EMAIL']['SMTPServer'], int(config['EMAIL']['SMTPPort'])) 86 | except Exception as e: 87 | log(f"While connecting to SMTP server: {e}", level=3) 88 | else: 89 | server.set_debuglevel(0) 90 | if config['EMAIL']['Auth'].lower() == 'true': 91 | try: 92 | server.login(config['EMAIL-AUTH']['Login'], config['EMAIL-AUTH']['Password']) 93 | except Exception as e: 94 | log(f"While login to SMTP Server: {e}", level=3) 95 | headers = (f"From: {config['EMAIL']['MailFrom']}\n" 96 | f"To: {config['EMAIL']['MailTo']}\n" 97 | f"Subject:{config['EMAIL']['Subject']} dIP: {', '.join(ip_set)}\n" 98 | "MIME-Version: 1.0\n" 99 | f"Content-Type: text/plain; charset=\"{CODE}\"\n") 100 | try: 101 | server.sendmail(f"{config['EMAIL']['MailFrom']}", [ f"{config['EMAIL']['MailTo']}" ], headers + msg) 102 | except Exception as e: 103 | log(f"While sending e-mail: {e}", level=3) 104 | else: 105 | log(f"DDoS E-mail sent to {config['EMAIL']['MailTo']}") 106 | server.quit() 107 | 108 | 109 | def format_msg(reports_output, ip_set, flow_print): 110 | email_msg = '' 111 | for report, output in reports_output.items(): 112 | email_msg += (f"\nTRIGGERED RULE: '{report}' with a THRESHOLD: " 113 | f"{REPORTS[report]['threshold']} {output['head'].split(',')[-1]}:\n\n") 114 | email_msg += ''.join(f"{s.replace('*', '').lower():<25}" for s in output['head'].split(',')) 115 | if config['SYSTEM']['Whois'].lower() == 'true': 116 | email_msg += f"{'netname, orgname':<25}" 117 | email_msg += "\n" + "-"*120 + "\n" 118 | for ip, values in output['list']: 119 | email_msg += f"{ip.strip():<25}" + ''.join(f"{s.strip():<25}" for s in values.split(',') if s) 120 | if config['SYSTEM']['Whois'].lower() == 'true': 121 | email_msg += f"{whois_net(ip):<25}" 122 | email_msg += "\n" 123 | email_msg += "\n" 124 | email_msg += f"\nFlow report (last {FLOW_REC_AMOUNT} flows to dIP: {', '.join(ip_set)}):\n\n" 125 | email_msg += ( f"{'Start':<24}{'End':<27}{'Sif':<6}{'SrcIPaddress':<16}{'SrcP':<9}" 126 | f"{'DIf':<7}{'DstIPaddress':<16}{'DstP':<7}{'Prot':<7}{'Flows':<7}{'Pkts':<7}{'Octets':<7}" ) 127 | email_msg += "\n" + "-"*120 + "\n" 128 | email_msg += flow_print 129 | return email_msg 130 | 131 | 132 | def create_white_filter(): 133 | with open(IP_WHITE_LIST_FILE, 'r', encoding=CODE) as f: 134 | ip_list = re.findall(r'^\s*(\d+\.\d+\.\d+\.\d+)\/?(\d+)?', f.read(), flags=re.MULTILINE) 135 | filter_string = '' 136 | if ip_list: 137 | for ip, prefix in ip_list: 138 | filter_string += f"not net {ip}/{prefix} and " if prefix else f"not host {ip} and " 139 | filter_string = filter_string[:-4] 140 | return filter_string 141 | 142 | 143 | def ddos_check(flows_file): 144 | NFDUMP_REPORTS = { 145 | 'sdport_flows' : { 'aggr' : 'dstip,srcport,dstport' , 'format' : 'fmt:%da,%sp,%dp,%fl', 'order' : 'flows' }, 146 | 'dport_packets' : { 'aggr' : 'dstip,dstport' , 'format' : 'fmt:%da,%dp,%pkt', 'order' : 'packets'}, 147 | 'flows' : { 'aggr' : 'dstip' , 'format' : 'fmt:%da,%fl', 'order' : 'flows'}, 148 | 'packets' : { 'aggr' : 'dstip' , 'format' : 'fmt:%da,%pkt', 'order' : 'packets' } 149 | } 150 | ip_set = set() 151 | reports_output = {} 152 | for report, options in REPORTS.items(): 153 | command = ( f"{NFDUMP} -r {flows_file} -A {NFDUMP_REPORTS[report]['aggr']} " 154 | f"-O {NFDUMP_REPORTS[report]['order']} -N -q -o '{NFDUMP_REPORTS[report]['format']}' '{create_white_filter()}'| " 155 | f"{AWK} -F, '${options['key_field']} > int({options['threshold']})'") 156 | result = subprocess.run([command], stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True) 157 | if result.stderr: 158 | log(f"Return: {result.stderr} Command: '{command}'", level=3) 159 | exit(1) 160 | report_head = f"{NFDUMP_REPORTS[report]['aggr']},{NFDUMP_REPORTS[report]['order']}" 161 | report_list = re.findall(r'\s*(\d+\.\d+\.\d+\.\d+)(.*)', result.stdout.decode(CODE)) 162 | if report_list: 163 | reports_output[report] = {} 164 | reports_output[report]['head'] = report_head 165 | reports_output[report]['list'] = report_list 166 | for ip, _ in report_list: 167 | ip_set.add(ip) 168 | return (reports_output, ip_set) 169 | 170 | 171 | def flows_print(ip_set, flows_file): 172 | filter_ip = '' 173 | for ip in ip_set: 174 | filter_ip += f"host {ip} or " 175 | command = ( f"{NFDUMP} -r {flows_file} -N -q -o 'fmt:%ts,%te,%in,%sa,%sp,%out,%da,%dp,%pr,%fl,%pkt,%byt' '{filter_ip[:-3]}' | " 176 | f" tail -n {FLOW_REC_AMOUNT}") 177 | result = subprocess.run([command], stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True) 178 | if result.stderr: 179 | log(f"Return:{result.stderr} Command: '{command}'", level = 3) 180 | exit(1) 181 | else: 182 | return result.stdout.decode(CODE) 183 | 184 | 185 | def main(): 186 | notify_counter = log() 187 | date = (datetime.now() - timedelta(minutes=2)).strftime('%Y%m%d%H%M') 188 | try: 189 | flows_file = next(Path(FLOWS_DIR).rglob(f'*{date}*')) 190 | except StopIteration: 191 | log(f"Can not find flow-capture file for the date: {date}", level=3) 192 | exit(1) 193 | reports_output, ip_set = ddos_check(flows_file) 194 | if reports_output: 195 | if NOTIFY_FREQ > 0: 196 | if notify_counter == 0: 197 | email_msg = format_msg(reports_output, ip_set, flows_print(ip_set, flows_file)) 198 | send_mail(email_msg, ip_set) 199 | notify_counter += 1 200 | log(f"DDoS dIP:{', '.join(ip_set)}", notify_counter) 201 | log("Notification counter changed", notify_counter=0) if notify_counter >= NOTIFY_FREQ else True 202 | else: 203 | log(f"DDoS dIP:{', '.join(ip_set)}; Printed to STDOUT", notify_counter=0) 204 | print('\n'.join(ip_set)) 205 | else: 206 | log("Notification counter changed", notify_counter=0) if log() != 0 else True 207 | 208 | 209 | if __name__ == '__main__': 210 | main() 211 | -------------------------------------------------------------------------------- /ip-white-list.txt: -------------------------------------------------------------------------------- 1 | # Add local ip/net to exclude it from checking on DDoS 2 | #127.0.0.1 3 | #127.0.0.0/8 --------------------------------------------------------------------------------