├── .gitignore ├── README.md ├── tool1 ├── html-output.py └── parse.py ├── tool2 ├── counters.py ├── dmarc-EmailTest-client.py ├── dmarc-EmailTest-server.py ├── dns-record-tracker.py ├── domain-status.py ├── graph.py ├── sendTestMail.py ├── spf-ip-extract.py └── trusted-list.txt └── visualizations ├── asn-mapping.dat ├── bubble-chart-AS.py ├── bubble-chart.py ├── heatMap-in.py └── heatMap-out.py /.gitignore: -------------------------------------------------------------------------------- 1 | *~ 2 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | Description 2 | ====== 3 | 4 | A DMARC deployment/monitoring tool written in python. 5 | 6 | Requirements 7 | ======= 8 | 9 | * Bootstrap 10 | * Rickshaw JS 11 | * w3data.js. Can be found at: http://www.w3schools.com/lib/w3data.js 12 | * python 2.7 (not tested with python 3) 13 | * dnspython 14 | * pythone MySQLdb 15 | * python netaddr 16 | * For DMARC email test: 17 | * pydkim 3 18 | * pyspf 2.0.12 19 | * python email 20 | * ssl 21 | * Techsneeze's dmarc parser. Can be found at: https://github.com/techsneeze/dmarcts-report-parser/blob/master/dmarcts-report-parser.pl 22 | * For Visualizations: 23 | * Matplotlib 24 | * numpy 25 | * python Tkinter 26 | * pyasn 27 | * urllib 28 | * json 29 | * MySQL database 30 | 31 | Make sure you place the database user credentials in the following files: 32 | 33 | * parse.py 34 | * counters.py 35 | * graph.py 36 | * dns-record-tracker.py 37 | * bubble-chart.py 38 | * heatMap-in.py 39 | * heatMap-out.py 40 | 41 | For use of `dns-record-tracker.py`, insert the following table into the datbase named `dmarc` which is created by the dmarc parser from Techsneeze: 42 | 43 | | Column name | Type | 44 | |---|---| 45 | | dmarc | varchar(255) | 46 | | spf | varchar(255) | 47 | | dkim | varchar(255) | 48 | | dateStamp | timestamp | 49 | 50 | 51 | Tool 1 52 | ============ 53 | 54 | This tool is meant to give the domain owner an overview of the possible sources 55 | that can deliver DMARC reports. 56 | 57 | Phase 1 consist of two files: 58 | 59 | * `parse.py` 60 | * `html-output.py` 61 | 62 | Use `parse.py` to generate a CSV file that is accepted by `html-output.py`. 63 | `parse.py` should be supplied with file containing the mail log. An example: 64 | 65 | python parse.py /var/log/mail.log 66 | 67 | This will create a CSV file called `output.csv` which must be supplied to `html-output.py`: 68 | 69 | python html-output.py output.CSV 70 | 71 | After running `html-output.py`, a HTML file called `dm-phase1.html` is created that holds the result. 72 | 73 | 74 | 75 | Tool 2 76 | =========== 77 | 78 | This tool is meant to monitor the domain during the deployment/operation of DMARC. 79 | It provides various tools which include: current DMARC status, DMARC tester, authentication 80 | results and a DNS history tool. 81 | 82 | Phase 2 consist of the following files with their corresponding output: 83 | 84 | * `domain-status.py` -> `domainstatus.html` 85 | * `dmarc-EmailTest-client.py` 86 | * `dmarc-EmailTest-server.py` -> `dmarcCheck.html` 87 | * `sendTestMail.py` 88 | * `counters.py` -> `counterTrust.html`, `counterForeign.html` 89 | * `graph.py` -> `graphTrust.js`, `graphForeign.js` 90 | * `dns-record-tracker.py` 91 | * `spf-ip-extract.py` 92 | * `trusted-list.txt` 93 | * `dm-ph2.html` 94 | 95 | `domain-status.py` checks the presence of several important DMARC parameters and warns the user if any of these are not configured. Additionally the current DMARC record is displayed. 96 | 97 | `dmarc-EmailTest-client.py` is the client application that should have access to a remote mailbox. It polls the mailbox for any new message. If a message is found, it sends the message including the headers to the server over a secure channel. `dmarc-EmailTest-server.py` implements the server. The server listens for a client connection. If the client sends a message, its evaluated on SPF and DKIM allignment. The results are written to `dmarcCheck.html`. For the secure connection, the user needs to create a certificate and a key. OpenSSL can do this: 98 | 99 | 100 | openssl genrsa 2048 > key 101 | openssl req -new -x509 -nodes -sha1 -days 365 -key key > cert 102 | 103 | 104 | `sendTestMail.py` can automatically send an test mail to the reserved mailbox that 105 | is used by `dmarc-EmailTest-client.py` 106 | 107 | `counters.py` generates statistics about authentication results. The results are dived into 108 | two sections: Trusted and Unknown sources. Each section contains an IP list that displays 109 | the IP addresses that fall within this category and the number of messages that they have sent. 110 | Additionally, a set of counters shows the aggregate authentication results. These include SPF, DKIM and DMARC results. This script relies on the trusted host which must be defined in `trusted-list.txt`. This can be done manually or automatically using `spf-ip-extract.py`. 111 | 112 | 113 | `graph.py` generates the graphs which show DMARC authentication results over the last 30 days (by default). Like `counters.py`, this done for both trusted and unknown sources. Additionally it also generates the DNS history time line. This script relies on the trusted host which must be defined in `trusted-list.txt`. This can be done manually or automatically using `spf-ip-extract.py`. 114 | 115 | `spf-ip-extract.py` can automatically extract IP addresses from a SPF record. Network addresses are also supported. All IP addresses are written to `trusted-list.txt` 116 | 117 | `dns-record-tracker.py` tracks the SPF, DKIM and DMARC records of a domain. Any record change is saved 118 | in the MySQL database. These records are used by `graph.py` to generate the DNS history time line. 119 | It advised to run this script frequently when one is changing one of the 3 records (SPF, DKIM, DMARC) frequently (for example during the deployment). 120 | 121 | The individual generated files for each widget are combined into one web interface in `dm-ph2.html`. 122 | 123 | 124 | Visualizations 125 | ============= 126 | 127 | The visualizations consist of the following files: 128 | 129 | * bubble-chart-ASN.py 130 | * heatMap-in.py 131 | * heatMap-out.py 132 | * bubble-chart.py 133 | 134 | 135 | `bubble-chart-ASN.py` generates a bubble chart to review where emails come from, in which quantities and the ratio of successful DMARC authentication results. The categorization is based on ASN numbers obtained using `pyasn`. This libary requires a BGP/MRT dump file as input. These dumps can be found at http://archive.routeviews.org/. The ASN <-> IP mapping for IPv4 and IPv6 is found in seppreate files which are not autmatically merged by `pyasn`. An mergeged file of this mapping can be found under `asn-mapping.dat` (version of 12-08-15). Additionally, a text file with the results of each AS is generated. This file is named `asn-mapping.dat`. This file also contains the ip addresses found in each AS togheter with authentication results. The user can optionally call this script with the `--asn-lookup` argument which will lookup the corrosponding (orginizational) name of the AS. 136 | 137 | `heatMap-in.py` generates a heat map that displays the authentication results of different domains based on incoming reports. Each tile is awarded a color based on the ratio of successful authentication results against the total amount of emails. Each tile contains text fields that indicate the total number of emails, volume of emails that passed DMARC and volume of email that failed DMARC. 138 | 139 | `heatMap-out.py` has the same functionality as `heatMap-in.py` but than for outgoing reports. Based on OpenDmarc's import functionality. 140 | 141 | `bubble-chart.py` similar to `bubble-chart-ASN.py` but works on IP chunks rather than ASs. 142 | -------------------------------------------------------------------------------- /tool1/html-output.py: -------------------------------------------------------------------------------- 1 | ################################################# 2 | # Author: Yadvir Singh # 3 | # Date: 16-07-2016 # 4 | # Description: # 5 | # # 6 | # Script that parses a csv file created with # 7 | # parse.py. The contents of the csv file are # 8 | # converted to an webpage. # 9 | # # 10 | # To be set by user: # 11 | # # 12 | # None # 13 | # # 14 | ################################################# 15 | 16 | import sys 17 | import csv 18 | 19 | 20 | 21 | def init(): 22 | 23 | filename = sys.argv[1] 24 | return filename 25 | 26 | #Function that parses the input csv file and 27 | #generates a HTML page. 28 | def parse(input_csv): 29 | 30 | totalCount = 0 31 | domainCount = 0 32 | dmarcPubCount = 0 33 | dmarcNoPubCount = 0 34 | domain_list = [] 35 | 36 | f = open(input_csv, 'r') 37 | 38 | 39 | 40 | #Read input csv line by line 41 | try: 42 | reader = csv.reader(f) 43 | for row in reader: 44 | print row 45 | domain_list.append(row) 46 | totalCount += int(row[1]) 47 | if row[2] != "None": 48 | dmarcPubCount += 1 49 | domainCount = len(domain_list) 50 | dmarcNoPubCount = domainCount - dmarcPubCount 51 | 52 | finally: 53 | f.close() 54 | 55 | #The corrsonding HTML page is written to dm-ph1.html 56 | output = open('dm-ph1.html', 'w') 57 | 58 | 59 | header = """ 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 72 | 73 | 74 |

DMARC Pre-check

75 | 76 | 77 | 78 | 79 |
80 |
81 | 82 |
""" 83 | emailCountHtml = """
84 | 85 |

""" + str(totalCount) + """

86 |

Emails

87 | 88 |
""" 89 | domainCountHtml = """
90 | 91 |

""" + str(domainCount) + """

92 |

Domains

93 | 94 |
""" 95 | dmarcPubHtml = """
96 | 97 |

""" + str(dmarcPubCount) +"""

98 |

DMARC record published

99 | 100 |
""" 101 | dmarcNoPubHtml = """
102 | 103 |

""" + str(dmarcNoPubCount) +"""

104 |

No DMARC record published

105 | 106 |
""" 107 | tailCountersHtml = """
108 | 109 |
110 | 111 |
""" 112 | startTableHtml = """
113 | 114 |

Domains which have published a DMARC record may provide you with reports on you mail systems's operation.

115 |
116 | 117 | 118 | 119 | 120 | 121 | 122 | 123 | 124 | """ 125 | pageEndHtml = """ 126 |
DomainEmail VolumeDMARC record
127 |
128 | 129 | 130 | 131 |
132 | 133 | 134 | 135 | 136 | """ 137 | 138 | 139 | #Write all parts of webpage to the HTML file 140 | output.write(header) 141 | output.write(emailCountHtml) 142 | output.write(domainCountHtml) 143 | output.write(dmarcPubHtml) 144 | output.write(dmarcNoPubHtml) 145 | output.write(tailCountersHtml) 146 | output.write(startTableHtml) 147 | 148 | #Genergate the table overview 149 | for domain in domain_list: 150 | output.write(""" 151 | """ + str(domain[0]) + """ 152 | """ + str(domain[1]) + """ 153 | """ + str(domain[2]) + """ 154 | """) 155 | 156 | 157 | output.write(pageEndHtml) 158 | output.close() 159 | 160 | 161 | parse(init()) 162 | -------------------------------------------------------------------------------- /tool1/parse.py: -------------------------------------------------------------------------------- 1 | ################################################# 2 | # Author: Yadvir Singh # 3 | # Date: 16-07-2016 # 4 | # Description: # 5 | # # 6 | # Script that parses emails log and outputs # 7 | # a csv containing all outgoing # 8 | # email domains and their DMARC records # 9 | # # 10 | # To be set by user: # 11 | # # 12 | # domain variable # 13 | # # 14 | ################################################# 15 | 16 | 17 | 18 | import sys 19 | import re 20 | from collections import Counter 21 | import dns.resolver 22 | import csv 23 | 24 | #Setup your domain here: 25 | domain = "dmarc-research.nl" 26 | 27 | 28 | 29 | #Function that creates a filehandle for the mail log 30 | def init(): 31 | 32 | filename = sys.argv[1] 33 | try: 34 | mail_log = open(filename, 'r') 35 | except: 36 | print "File error" 37 | sys.exit() 38 | 39 | return mail_log 40 | 41 | #Funtion that parses the email log 42 | def parse(input_file, domain): 43 | 44 | tokens = input_file.readlines(); 45 | pattern = re.compile(r'@.*?>') 46 | domains = [] 47 | 48 | 49 | #Read mail log line by line 50 | for token in tokens: 51 | token = token.lower() 52 | fields = token.split(" ") 53 | result = re.findall(pattern, token) 54 | #We dont want emails that are sent to the domain, only outgoing emails. 55 | #Ugly but works for now. We need a propper domain selection tool 56 | if result and ("@" + domain +">," not in result): 57 | print result 58 | domains.append(result[0].strip('@').strip('>,')) 59 | 60 | counts = Counter(domains) 61 | return counts 62 | 63 | 64 | #Function to retrieve a DMARC record 65 | def dmarc_lookup(domain): 66 | print domain 67 | 68 | answer="" 69 | 70 | try: 71 | answer = dns.resolver.query("_dmarc." + domain + "." , "TXT") 72 | for data in answer: 73 | answer = data 74 | except: 75 | answer = "None" 76 | 77 | return answer 78 | 79 | 80 | #Function for writing the results to output.csv 81 | def write_csv(values): 82 | 83 | f = open('output.csv', 'wt') 84 | 85 | try: 86 | writer = csv.writer(f) 87 | for domain in values: 88 | dmarc = dmarc_lookup(domain) #Lookup DMARC record of domain 89 | writer.writerow( (domain, values[domain], dmarc)) 90 | 91 | finally: 92 | f.close() 93 | 94 | 95 | domain_stats = parse(init(), domain) 96 | write_csv(domain_stats) 97 | -------------------------------------------------------------------------------- /tool2/counters.py: -------------------------------------------------------------------------------- 1 | ################################################# 2 | # Author: Yadvir Singh # 3 | # Date: 16-07-2016 # 4 | # Description: # 5 | # # 6 | # Script that generates statistics about # 7 | # authentication results. This includes # 8 | # an IP list and counters of SPF, DKIM, # 9 | # and DMARC authentication results. # 10 | # The statics are generated for both # 11 | # trusted and unknown sources. # 12 | # # 13 | # Database variables to be set by user: # 14 | # # 15 | # - dbAddress # 16 | # - dbName # 17 | # - dbUserName # 18 | # - dbPassword # 19 | # # 20 | # # 21 | ################################################# 22 | 23 | import sys 24 | import MySQLdb 25 | import socket 26 | import struct 27 | from collections import Counter 28 | 29 | 30 | dbAddress = "127.0.0.1" 31 | dbName = "dmarc" 32 | dbUserName = "dmarc" 33 | dbPassword = "dmarcrp2" 34 | 35 | 36 | def calculateRatio(x1, x2): 37 | 38 | result = "" 39 | 40 | try: 41 | result = str(round((float(x1)/x2) * 100)) 42 | except: 43 | result = "0" 44 | 45 | return result 46 | 47 | 48 | #Function that connects to the SQL database. 49 | def connectToDB(dbAddress, dbUserName, dbPassword, dbName): 50 | 51 | try: 52 | db = MySQLdb.connect(dbAddress,dbUserName, dbPassword, dbName) 53 | return db 54 | except: 55 | return None 56 | 57 | 58 | #Funtion that prepares the HTML ouput files and reads the arguments list to 59 | #obtain the list of trusted sources. 60 | def openFile(): 61 | 62 | domains = [] 63 | argumentLen = len(sys.argv) 64 | fileHandle = None 65 | 66 | trusted_list = open("trusted-list.txt", 'r') 67 | 68 | 69 | #Collect all trusted sources given by the the trusted list 70 | for line in trusted_list: 71 | domains.append(line[0:-1]) 72 | 73 | 74 | if len(domains) > 0: 75 | 76 | #Collect all trusted sources given by the argument list 77 | #for x in range(1, argumentLen): 78 | # domains.append(sys.argv[x]) 79 | 80 | #Prepare output files 81 | try: 82 | fileHandleTrusted = open('counterTrust.html', 'w') 83 | fileHandleForeign = open('counterForeign.html', 'w') 84 | return fileHandleTrusted, fileHandleForeign, domains 85 | except: 86 | print "file error" 87 | sys.exit() 88 | else: 89 | print "No IP address! Closing" 90 | sys.exit(0) 91 | 92 | 93 | #The main function that retrieves data from the MySQL database and generate 94 | #statics that are written to HTML files. 95 | def retrieveData(db, fileHandleTrusted, fileHanldeForeign, domains): 96 | 97 | ipBoxString = "" 98 | fileHandle = None 99 | trustedString = "(" 100 | 101 | #Generate sql query to filter out only trusted domains 102 | for domain in domains: 103 | trustedString += str(struct.unpack("!I", socket.inet_aton(domain))[0]) + "," 104 | 105 | trustedString = trustedString[:-1] 106 | trustedString += ")" 107 | 108 | 109 | # prepare a cursor object using cursor() method 110 | cursor = db.cursor() 111 | 112 | #Generate statistics for both trusted and foreign hosts. 113 | for domain in ["trusted", "foreign"]: 114 | 115 | if domain == "trusted": 116 | cursor.execute("select * from report inner join rptrecord on report.serial=rptrecord.serial where ip IN " + trustedString) 117 | fileHandle = fileHandleTrusted 118 | else: 119 | cursor.execute("select * from report inner join rptrecord on report.serial=rptrecord.serial where ip NOT IN " + trustedString) 120 | fileHandle = fileHanldeForeign 121 | 122 | data = cursor.fetchall() 123 | resultCount = len(data) 124 | 125 | totalCount = 0 126 | dkimPass = 0 127 | dkimFail = 0 128 | spfPass = 0 129 | spfFail = 0 130 | dmarcCompliant = 0 131 | dmarcUnCompliant = 0 132 | 133 | ipaddr = [] 134 | 135 | 136 | #Acumulate authentication results 137 | for entry in data: 138 | 139 | count = entry[17] 140 | totalCount += count 141 | dkimResult = entry[21] 142 | spfResult = entry[24] 143 | 144 | for y in range (0, count): 145 | ipaddr.append(socket.inet_ntoa(struct.pack('!L', entry[15]))) 146 | 147 | #Populate the authentication counters from the result of the SQL query 148 | if (dkimResult == 'pass'): 149 | dkimPass += count 150 | else: 151 | dkimFail += count 152 | 153 | if spfResult == 'pass': 154 | spfPass += count 155 | else: 156 | spfFail += count 157 | 158 | #Configure strictness of DMARC check here. 159 | if ((dkimResult == 'pass') or (spfResult == 'pass')): 160 | dmarcCompliant += count 161 | else: 162 | dmarcUnCompliant += count 163 | 164 | print domain + ":\n" 165 | print "Totalcount:\t" + str(totalCount) 166 | print "dkim pass:\t" + str(dkimPass) 167 | print "dkim fail:\t" + str(dkimFail) 168 | print "spf pass:\t" + str(spfPass) 169 | print "spf fail:\t" + str(spfFail) 170 | 171 | # Count the number of message for each IP 172 | ipCount = Counter(ipaddr) 173 | 174 | # Generate string for Ip address box: 175 | for entry in ipCount: 176 | ipBoxString += entry + "\t|\t" + str(ipCount[entry]) + "\n------------------------------------\n" 177 | 178 | 179 | print ipBoxString 180 | 181 | 182 | 183 | 184 | 185 | 186 | result = """ 187 | 188 |
189 |
190 |
191 | 192 | 193 |
194 |
195 |
196 | 197 | 198 | 199 | 200 | 201 |
202 | 203 | 204 | 208 | 209 | 210 | 214 | 215 |
205 |

""" + str(dkimPass) + """

""" + calculateRatio(dkimPass, totalCount) + """ %
206 |

DKIM pass

207 |
211 |

""" + str(dkimFail) + """

""" + calculateRatio(dkimFail, totalCount) + """ %
212 |

DKIM fail

213 |
216 |
217 | 218 |
219 | 220 | 221 | 225 | 226 | 227 | 231 | 232 |
222 |

""" + str(spfPass) + """

""" + calculateRatio(spfPass, totalCount) + """ %
223 |

SPF pass

224 |
228 |

""" + str(spfFail) + """

""" + calculateRatio(spfFail, totalCount) + """ %
229 |

SPF fail

230 |
233 |
234 | 235 |
236 | 237 | 238 | 242 | 243 | 244 | 249 | 250 |
239 |

""" + str(totalCount) + """

tekst
240 |

Total count

241 |
245 |

""" + str(dmarcCompliant) + """

""" + calculateRatio(dmarcCompliant, totalCount) + """ %
246 |

DMARC compliant

247 | 248 |
251 |
252 | 253 | 254 | """ 255 | 256 | ipBoxString = "" 257 | 258 | # Write HTML results 259 | fileHandle.write(result) 260 | fileHandle.close() 261 | 262 | 263 | 264 | database = connectToDB(dbAddress,dbUserName, dbPassword, dbName) 265 | fileHandleTrusted,fileHandleForeign, domains = openFile() 266 | retrieveData(database, fileHandleTrusted, fileHandleForeign, domains) 267 | 268 | -------------------------------------------------------------------------------- /tool2/dmarc-EmailTest-client.py: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | 3 | ################################################# 4 | # Author: Yadvir Singh # 5 | # Date: 14-08-2016 # 6 | # Description: # 7 | # # 8 | # Client script that reads email from a # 9 | # reserved mailbox. The complete email # 10 | # (including headers) is sent to the server # 11 | # over a secure channel # 12 | # # 13 | # SSL connection based on: # 14 | # # 15 | # https://carlo-hamalainen.net/blog # 16 | # /2013/1/24/python-ssl-socket-echo- # 17 | # test-with-self-signed-certificate # 18 | # # 19 | # To be set by user: # 20 | # # 21 | # mailbox variable # 22 | # certificate variable # 23 | # # 24 | ################################################# 25 | 26 | 27 | import socket, ssl 28 | import os 29 | import time 30 | 31 | mailbox = 'domain.txt' 32 | certificate = "cert" 33 | 34 | #Check if any mail is present in the reserved mailbox. 35 | #If so, sent the mail including all the headers over the secure channel. 36 | def poll(): 37 | 38 | while True: 39 | mailBoxFileHandle = open(mailbox,'r+') 40 | message = mailBoxFileHandle.read() 41 | 42 | #Check if message is not empty 43 | if len(message) > 10: 44 | send(message) 45 | mailBoxFileHandle.close() 46 | 47 | 48 | # Empty the mail box for the the next test, remove this line if 49 | # you whish to keep the email. 50 | # (Hackish way of emtying the file contents) 51 | open(mailbox, 'w').close() 52 | 53 | time.sleep(5) 54 | 55 | #Function that builds a secure connection between the client and server. 56 | def send(data): 57 | s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 58 | 59 | # Require a certificate from the server. We used a self-signed certificate 60 | # so here ca_certs must be the server certificate itself. 61 | ssl_sock = ssl.wrap_socket(s, 62 | ca_certs=certificate, 63 | cert_reqs=ssl.CERT_REQUIRED) 64 | 65 | 66 | ssl_sock.connect(('localhost', 12345)) 67 | ssl_sock.write(data) 68 | ssl_sock.close() 69 | 70 | 71 | 72 | 73 | poll() 74 | 75 | -------------------------------------------------------------------------------- /tool2/dmarc-EmailTest-server.py: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | 3 | ################################################# 4 | # Author: Yadvir Singh # 5 | # Date: 14-08-2016 # 6 | # Description: # 7 | # # 8 | # Server script that listen for incomming # 9 | # connections from clients. The email passed # 10 | # by client is checked on SPF and DKIM # 11 | # allignment. The result is written to an # 12 | # HTML file # 13 | # # 14 | # SSL connection based on: # 15 | # # 16 | # https://carlo-hamalainen.net/blog # 17 | # /2013/1/24/python-ssl-socket-echo- # 18 | # test-with-self-signed-certificate # 19 | # # 20 | # To be set by user: # 21 | # # 22 | # key variable # 23 | # certificate variable # 24 | # # 25 | ################################################# 26 | 27 | import socket, ssl 28 | import dkim 29 | import spf 30 | import sys 31 | import re 32 | from email.parser import Parser 33 | import email 34 | 35 | 36 | certificate = "cert" 37 | key = "key" 38 | 39 | 40 | #Function that checks the mail obtained from the client on SPF and DKIM allignment. 41 | def checkMail (message): 42 | 43 | outputFileHandle = None 44 | 45 | try: 46 | outputFileHandle = open('dmarcCheck.html', 'w') 47 | except: 48 | print "file error" 49 | sys.exit() 50 | 51 | 52 | htmlDMARCBox = "" 53 | htmlDKIMBox = "" 54 | htmlSPFBox = "" 55 | receivedHeader = "" 56 | fromHeader = "" 57 | 58 | #Read the message from the inbox 59 | headers = email.parser.Parser().parsestr(message) 60 | 61 | 62 | for field in headers.items(): 63 | if field[0] == "Received" and "[" in field[1] and "]" in field[1]: 64 | receivedHeader = field[1] 65 | 66 | pattern = re.compile(r'\[.*]') 67 | result = re.search(pattern, receivedHeader).group(0) 68 | 69 | pattern = re.compile(r'from .*? ') 70 | result2 = re.search(pattern, receivedHeader).group(0) 71 | 72 | subject = headers['subject'] 73 | 74 | #Variables needed for spf check 75 | 76 | #We need only the email address 77 | if "<" in headers['from'] and ">" in headers['from']: 78 | pattern = re.compile(r'\<.*>') 79 | fromHeader = re.search(pattern, headers['from']).group(0) 80 | fromHeader = fromHeader[1:-1] 81 | else: 82 | fromHeader = headers['from'] 83 | 84 | 85 | ipaddr = result[1:-1] 86 | host = result2[5:-1] 87 | 88 | 89 | # Perfom SPF and DKIM checks 90 | spfResult = spf.check(i=ipaddr,s=fromHeader,h=host) 91 | dkimResult = dkim.verify(message ,None) 92 | 93 | 94 | #Create HTML conentent according to the results of the test 95 | if (spfResult[0] == 'pass'): 96 | htmlSPFBox = """ 97 | SPF check passed
↳""" + spfResult[2] + """ 98 | """ 99 | else: 100 | htmlSPFBox = """ 101 | SPF check failed
↳""" + spfResult[2] + """ 102 | """ 103 | 104 | if (dkimResult == True): 105 | htmlDKIMBox = """ 106 | DKIM check passed 107 | """ 108 | else: 109 | htmlDKIMBox = """ 110 | DKIM check failed 111 | """ 112 | 113 | if (spfResult[0] == 'pass' or dkimResult == True): 114 | htmlDMARCBox = """ 115 | DMARC check passed 116 | """ 117 | else: 118 | htmlDMARCBox = """ 119 | DMARC check failed 120 | """ 121 | 122 | 123 | html = """ 124 |
125 |

DMARC test

126 | 127 | 128 | 129 | 130 | 131 | 132 | 133 | 134 | 135 | 136 | 137 | 138 | """ + htmlDKIMBox + htmlSPFBox + """""" + htmlDMARCBox + """ 139 | 140 | 141 |
Result
Subject: """ + subject + """
142 |
143 | """ 144 | outputFileHandle.write(html) 145 | outputFileHandle.close() 146 | 147 | # Empty the mail box for the the next test, remove this line if 148 | # you whish to keep the email. 149 | # (Hackish way of emtying the file contents) 150 | #open(mailbox, 'w').close() 151 | 152 | 153 | def writeMailTofile(data): 154 | try: 155 | fileHandle = open('dmarcTestEmail', 'w') 156 | fileHandle.write(data) 157 | fileHandle.close() 158 | except: 159 | print "file error" 160 | sys.exit() 161 | 162 | 163 | def read_data(connstream, data): 164 | print "read_data:", data 165 | 166 | if data[-1] == "\n" and data[-2] == "\n": 167 | return False 168 | 169 | 170 | def deal_with_client(connstream): 171 | data = connstream.recv(8192) 172 | while data: 173 | print "in loop" 174 | if not read_data(connstream, data): 175 | break 176 | data = connstream.recv(8192) 177 | return data 178 | 179 | 180 | def startSocket(): 181 | bindsocket = socket.socket() 182 | bindsocket.bind(('localhost', 12345)) 183 | bindsocket.listen(1) 184 | 185 | while True: 186 | newsocket, fromaddr = bindsocket.accept() 187 | connstream = ssl.wrap_socket(newsocket, 188 | server_side=True, 189 | certfile=certificate, 190 | keyfile=key, ssl_version = ssl.PROTOCOL_TLSv1_2) 191 | 192 | try: 193 | data = deal_with_client(connstream) 194 | finally: 195 | connstream.shutdown(socket.SHUT_RDWR) 196 | connstream.close() 197 | newsocket.close() 198 | break 199 | writeMailTofile(data) 200 | checkMail(data) 201 | 202 | 203 | startSocket() 204 | 205 | -------------------------------------------------------------------------------- /tool2/dns-record-tracker.py: -------------------------------------------------------------------------------- 1 | ################################################# 2 | # Author: Yadvir Singh # 3 | # Date: 17-07-2016 # 4 | # Description: # 5 | # # 6 | # Script retrieves the current SPF, DKIM and # 7 | # DMARC records. These records are saved in # 8 | # the database. If DNS records have been # 9 | # changed, a new entry is inserted into # 10 | # the database # 11 | # # 12 | # Database variables to be set by user: # 13 | # # 14 | # - dbAddress # 15 | # - dbName # 16 | # - dbUserName # 17 | # - dbPassword # 18 | # - domain # 19 | # - dkim_id (dkim identifier) # 20 | # # 21 | # # 22 | ################################################# 23 | 24 | 25 | 26 | import dns.resolver 27 | import MySQLdb 28 | import time 29 | 30 | 31 | dbAddress = "127.0.0.1" 32 | dbName = "dmarc" 33 | dbUserName = "dmarc" 34 | dbPassword = "dmarcrp2" 35 | domain = "dmarc-research.nl" 36 | dkim_id = "mail" 37 | 38 | 39 | #Create the database connection 40 | def DBconnect(dbAddress, dbUserName, dbPassword, dbName): 41 | 42 | try: 43 | 44 | db = MySQLdb.connect(dbAddress, dbUserName, dbPassword, dbName) 45 | return db 46 | 47 | except: 48 | return None 49 | 50 | 51 | def dnsLookup(domain): 52 | record = "" 53 | 54 | try: 55 | answer = dns.resolver.query(domain , "TXT") 56 | for data in answer: 57 | record = str(data).replace("\"", "") 58 | return record 59 | except: 60 | return "None" 61 | 62 | #Function that does a DNS lookup of SPF, DKIM and DMARC and checks if 63 | #any of the 3 records have changed. 64 | def recordLookup(domain, dkim_id, db): 65 | 66 | dmarcRecord = "" 67 | dkimRecord = "" 68 | spfRecord = "" 69 | 70 | #Prepare a cursor object 71 | cursor = db.cursor() 72 | 73 | 74 | #Fetch all previous records. 75 | cursor.execute("select * from dns_records") 76 | 77 | data = cursor.fetchall(); 78 | resultCount = len(data) 79 | 80 | #Fetch current records 81 | dmarcRecord = str(dnsLookup("_dmarc." + domain)) 82 | dkimRecord = str(dnsLookup(dkim_id + "._domainkey." + domain)) 83 | spfRecord = str(dnsLookup (domain)) 84 | currentTime = str(time.strftime("%y-%m-%d")) 85 | 86 | 87 | #If no records are found in database, insert the first one. 88 | if resultCount == 0: 89 | 90 | addUpdate = ("insert into dns_records (dmarc ,dkim, spf, dateStamp) values (%s, %s, %s, %s)" ) 91 | cursor.execute(addUpdate, (dmarcRecord, dkimRecord, spfRecord, currentTime)) 92 | db.commit() 93 | 94 | else: 95 | 96 | print "Not the first" 97 | 98 | #Check if the record is changed, if so insert the current record. 99 | if (dmarcRecord != data[-1][1]) or (dkimRecord != data[-1][2]) or (spfRecord != data[-1][3]): 100 | print "difference spotted" 101 | addUpdate = ("insert into dns_records (dmarc ,dkim, spf, dateStamp) values (%s, %s, %s, %s)" ) 102 | cursor.execute(addUpdate, (dmarcRecord, dkimRecord, spfRecord, currentTime)) 103 | db.commit() 104 | 105 | else: 106 | print "No change detected" 107 | 108 | 109 | 110 | 111 | db = DBconnect(dbAddress, dbUserName, dbPassword, dbName) 112 | recordLookup(domain, dkim_id, db) 113 | -------------------------------------------------------------------------------- /tool2/domain-status.py: -------------------------------------------------------------------------------- 1 | ################################################# 2 | # Author: Yadvir Singh # 3 | # Date: 16-07-2016 # 4 | # Description: # 5 | # # 6 | # Script that retrieves the DMARC record # 7 | # and checks several parameters for their # 8 | # presences. Generates a HTML widget with # 9 | # the results # 10 | # # 11 | # To be set by user: # 12 | # # 13 | # variable domain # 14 | # # 15 | ################################################# 16 | 17 | 18 | 19 | import dns.resolver 20 | import sys 21 | 22 | domain = "dmarc-research.nl" 23 | 24 | 25 | #Funtion that creates the filehandle for writing the results 26 | def openFile(): 27 | 28 | try: 29 | fileHandle = open('domainstatus.html', 'w') 30 | 31 | except: 32 | print "file error" 33 | sys.exit() 34 | 35 | return fileHandle 36 | 37 | 38 | #Funtion for lookup of DMARC DNS record. 39 | def dnsLookup(domain): 40 | record = "" 41 | 42 | try: 43 | answer = dns.resolver.query('_dmarc.' + domain , "TXT") 44 | for data in answer: 45 | record = str(data).replace("\"", "") 46 | return record 47 | except: 48 | return "None" 49 | 50 | 51 | #Helper funtion to strip DNS results 52 | def getSubString(string, char): 53 | 54 | try: 55 | IndexStart = string.index(char) 56 | except: 57 | return None 58 | IndexEqual = string.index ('=', IndexStart) + 1 59 | IndexEnd = string.index(';',IndexStart) 60 | 61 | subString = string[IndexEqual:IndexEnd] 62 | 63 | return subString 64 | 65 | 66 | #Funtion that writes the results to an HTML file. 67 | def generateHtml(fileHandle, recordData): 68 | 69 | print recordData 70 | 71 | rowString = "" 72 | 73 | dmarcRecordDisection = [] 74 | 75 | htmlStart = """
76 |

Status

77 | 78 | 79 | 80 | 81 | 82 | 83 | """ 84 | 85 | htmlEnd = """ 86 | 87 |
DMARC status
88 |
89 | 90 | 91 |
92 | 93 |
94 | """ 95 | 96 | 97 | #Extract the parameters from the DMARC record 98 | policy = getSubString(recordData, "p=") 99 | subPolicy = getSubString(recordData, "sp=") 100 | rua = getSubString(recordData, "rua=") 101 | ruf = getSubString(recordData, "ruf=") 102 | pct = getSubString(recordData, "pct=") 103 | 104 | 105 | dmarcRecordDisection.append(("Policy",policy)) 106 | dmarcRecordDisection.append(("Sub-policy",subPolicy)) 107 | dmarcRecordDisection.append(("RUA",rua)) 108 | dmarcRecordDisection.append(("RUF",ruf)) 109 | dmarcRecordDisection.append(("PCT",pct)) 110 | 111 | 112 | 113 | #Wrap results into HTML. 114 | for data in dmarcRecordDisection: 115 | 116 | if (data[1] == None) and ((data[0] == "RUA") or (data[0] == "RUF")): 117 | rowString += """No email specified for """ + data [0] +""" reports \n""" 118 | 119 | elif (data[1] == None): 120 | rowString += """No value found for """ + data [0] +""" \n""" 121 | 122 | else: 123 | rowString += """""" + data [0] +""" configured \n""" 124 | 125 | 126 | 127 | fileHandle.write(htmlStart) 128 | fileHandle.write(rowString) 129 | fileHandle.write(htmlEnd) 130 | fileHandle.close() 131 | 132 | 133 | fileHandle = openFile() 134 | record = dnsLookup(domain) 135 | generateHtml(fileHandle, record) 136 | 137 | 138 | 139 | 140 | 141 | 142 | 143 | 144 | -------------------------------------------------------------------------------- /tool2/graph.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | 3 | ################################################# 4 | # Author: Yadvir Singh # 5 | # Date: 16-07-2016 # 6 | # Description: # 7 | # # 8 | # Script that generates DMARC authentication # 9 | # graphs. Additionally, a time line with DNS # 10 | # Record changes is also created. # 11 | # # 12 | # Database variables to be set by user: # 13 | # # 14 | # - dbAddress # 15 | # - dbName # 16 | # - dbUserName # 17 | # - dbPassword # 18 | # # 19 | # # 20 | ################################################# 21 | 22 | 23 | 24 | import sys 25 | import time 26 | from datetime import date, timedelta 27 | import MySQLdb 28 | import socket 29 | import struct 30 | 31 | 32 | dbAddress = "127.0.0.1" 33 | dbName = "dmarc" 34 | dbUserName = "dmarc" 35 | dbPassword = "dmarcrp2" 36 | 37 | 38 | #Function to generate date points. By default this function will generate 39 | #a set of dates that contains the 30 past days from date argument. 40 | #based on #http://stackoverflow.com/questions/1060279/iterating-through-a-range-of-dates-in-python 41 | def dateItterator(date): 42 | 43 | dates = [] 44 | day = timedelta(days=1) 45 | 46 | date1 = date 47 | date2 = date - timedelta(days=30) # Adjust date range here 48 | 49 | while date2 < date1: 50 | dates.append(date2) 51 | date2 += day 52 | 53 | return dates 54 | 55 | 56 | #Funtion that prepares the HTML ouput files and reads the arguments list to 57 | #obtain the list of trusted sources. 58 | def openFile(): 59 | 60 | domains = [] 61 | argumentLen = len(sys.argv) 62 | fileHandle = None 63 | 64 | trusted_list = open("trusted-list.txt", 'r') 65 | 66 | 67 | #Collect all trusted sources given by the the trusted list 68 | for line in trusted_list: 69 | domains.append(line[0:-1]) 70 | 71 | 72 | if len(domains) > 0: 73 | 74 | #Collect all trusted sources given by the argument list 75 | #for x in range(1, argumentLen): 76 | # domains.append(sys.argv[x]) 77 | 78 | try: 79 | fileHandleTrusted = open('graphTrust.js', 'w') 80 | fileHandleForeign = open('graphForeign.js', 'w') 81 | return fileHandleTrusted, fileHandleForeign, domains 82 | except: 83 | print "file error" 84 | sys.exit() 85 | 86 | else: 87 | print "No IP address! Closing" 88 | sys.exit(0) 89 | 90 | #Function to obtain all reports of trusted and unknown sources from the 91 | #database. 92 | def sqlFilterQuery(domainList): 93 | 94 | trustedString = "(" 95 | 96 | for domain in domains: 97 | trustedString += str(struct.unpack("!I", socket.inet_aton(domain))[0]) + "," 98 | 99 | trustedString = trustedString[:-1] 100 | trustedString += ")" 101 | 102 | queryTrusted = "select * from report inner join rptrecord on report.serial=rptrecord.serial where ip IN " + trustedString 103 | queryForeign = "select * from report inner join rptrecord on report.serial=rptrecord.serial where ip NOT IN " + trustedString 104 | 105 | return queryTrusted, queryForeign 106 | 107 | #Main function that collects information for generating the graphs 108 | #and the DNS history time line. 109 | def generateGraph(dates, query, dbAddress, dbUserName, dbPassword, dbName): 110 | 111 | dkimResult = "" 112 | spfResult = "" 113 | count = 0 114 | dmarcCompliant = 0 115 | dmarcUnCompliant = 0 116 | 117 | reportAggergate = [] 118 | annotations = [] 119 | 120 | 121 | # Open database connection 122 | db = MySQLdb.connect(dbAddress, dbUserName, dbPassword, dbName) 123 | 124 | # prepare a cursor object using cursor() method 125 | cursor = db.cursor() 126 | 127 | for date in dates: 128 | 129 | lowerbound = date - timedelta(days= 1) 130 | #print str(date), str(lowerbound) 131 | 132 | cursor.execute(query + " and maxdate <= '"+ str(date) +" 23:59:59' and mindate >= '" + str(lowerbound) + " 00:00:00'") 133 | 134 | data = cursor.fetchall() 135 | resultCount = len(data) 136 | 137 | 138 | # Loop through the resulting rows and populate the counters 139 | if not (len(data) == 0): 140 | 141 | for entry in data: 142 | 143 | dkimResult = entry[21] 144 | spfResult = entry[23] 145 | count = entry[17] 146 | 147 | #Configure strictness of DMARC check here. 148 | if ((dkimResult == 'pass') or (spfResult == 'pass')): 149 | dmarcCompliant += count 150 | else: 151 | dmarcUnCompliant += count 152 | 153 | 154 | reportAggergate.append((date, int(dmarcCompliant),int(dmarcUnCompliant))) 155 | 156 | dmarcCompliant = 0 157 | dmarcUnCompliant = 0 158 | 159 | 160 | #Retrieve the DNS record changes from the database. 161 | cursor.execute("select * from dns_records") 162 | 163 | data = cursor.fetchall() 164 | resultCount = len(data) 165 | 166 | for x in data: 167 | annotations.append(x) 168 | 169 | db.close() 170 | 171 | return reportAggergate, annotations 172 | 173 | 174 | #Function that transform the restults of generateGraph into javascript file 175 | def writeToFile (filehandle, data, annotations): 176 | 177 | prepend = "data: [ " 178 | 179 | datapointCompliant = "" 180 | datapointUnCompliant = "" 181 | annotationString = "" 182 | 183 | #Generate graph points 184 | for x in data: 185 | datapointCompliant += "{ x:" + str((x[0]-date(1970,1,1)).total_seconds()) + ", y:" + str(x[1]) +" }," 186 | datapointUnCompliant += ("{ x:" + str((x[0]-date(1970,1,1)).total_seconds()) + ", y:" + str(x[2]) +" },") 187 | 188 | 189 | graph = """ 190 | var palette = new Rickshaw.Color.Palette(); 191 | 192 | var graph = new Rickshaw.Graph( { 193 | element: document.querySelector("#chart"), 194 | width: 550, 195 | height: 250, 196 | series: [ 197 | { 198 | name: "Dmarc UnCompliant", 199 | data: [""" + datapointUnCompliant + """], 200 | color: palette.color() 201 | }, 202 | { 203 | name: "Dmarc Compliant", 204 | data: [""" + datapointCompliant + """], 205 | color: palette.color() 206 | }, 207 | 208 | ] 209 | } ); 210 | 211 | var x_axis = new Rickshaw.Graph.Axis.Time( { graph: graph } ); 212 | 213 | var y_axis = new Rickshaw.Graph.Axis.Y( { 214 | graph: graph, 215 | orientation: 'left', 216 | tickFormat: Rickshaw.Fixtures.Number.formatKMBT, 217 | element: document.getElementById('y_axis'), 218 | } ); 219 | 220 | var legend = new Rickshaw.Graph.Legend( { 221 | element: document.querySelector('#legend'), 222 | graph: graph 223 | } ); 224 | 225 | var annotator = new Rickshaw.Graph.Annotate({ 226 | graph: graph, 227 | element: document.getElementById('timeline') 228 | }); 229 | 230 | """ 231 | unixTime = str((x[0]-date(1970,1,1)).total_seconds()) 232 | 233 | #Add annotations to the time line with buttons 234 | for data in annotations: 235 | annotationString += ("annotator.add(" + unixTime + ", \"\" );\n") 236 | 237 | 238 | graphEnd = """ 239 | annotator.update(); 240 | 241 | 242 | graph.render(); 243 | 244 | """ 245 | 246 | 247 | #Write javascript files to appropriate files. 248 | filehandle.write(graph) 249 | filehandle.write(annotationString) 250 | filehandle.write(graphEnd) 251 | filehandle.close() 252 | 253 | 254 | fileHandleTrusted, fileHandleForeign, domains = openFile() 255 | 256 | trusted, foreign = sqlFilterQuery(domains) 257 | dates = dateItterator(date(2016, 6, 30)) 258 | 259 | data, annotations = generateGraph(dates, trusted, dbAddress, dbUserName, dbPassword, dbName) 260 | writeToFile(fileHandleTrusted, data, annotations) 261 | 262 | data, annotations = generateGraph(dates, foreign, dbAddress, dbUserName, dbPassword, dbName) 263 | writeToFile(fileHandleForeign, data, annotations) 264 | 265 | -------------------------------------------------------------------------------- /tool2/sendTestMail.py: -------------------------------------------------------------------------------- 1 | ################################################################################# 2 | # Author: Yadvir Singh # 3 | # Date: 16-07-2016 # 4 | # Description: # 5 | # # 6 | # Script that automatically sends an email # 7 | # to the reserved mailbox that is read by # 8 | # email-test.py # 9 | # # 10 | # Based on example from: https://docs.python.org/2/library/email-examples.html # 11 | # # 12 | # To be set by user: # 13 | # # 14 | # - smptAddress # 15 | # - mailAddress # 16 | ################################################################################# 17 | 18 | import smtplib 19 | import time 20 | from email.mime.text import MIMEText 21 | 22 | smtpAddress = '145.100.104.165' 23 | mailAddress = 'test@dmarc-research.nl' 24 | 25 | #Email message content 26 | msg = MIMEText('Test email for DMARC') 27 | 28 | #Use the current time as an identifier in the subject line 29 | msg['Subject'] = str(time.time()) 30 | msg['From'] = mailAddress 31 | msg['To'] = mailAddress 32 | 33 | # Send the message via our own SMTP server. 34 | s = smtplib.SMTP(smtpAddress) 35 | s.sendmail(mailAddress, [mailAddress], msg.as_string()) 36 | s.quit() 37 | 38 | -------------------------------------------------------------------------------- /tool2/spf-ip-extract.py: -------------------------------------------------------------------------------- 1 | ################################################# 2 | # Author: Yadvir Singh # 3 | # Date: 15-08-2016 # 4 | # Description: # 5 | # # 6 | # Script that extracts IP addresses from # 7 | # spf records. Networks addresses are also # 8 | # supported. All addresses are written to a # 9 | # file. # 10 | # # 11 | # Variables to be set by user: # 12 | # # 13 | # - domain # 14 | # # 15 | ################################################# 16 | 17 | 18 | 19 | 20 | import re 21 | import dns.resolver 22 | import netaddr as ne 23 | 24 | domain = "dmarc-research.nl" 25 | ip_list = [] 26 | 27 | def writeToFile(data): 28 | 29 | fileHandle = open('trusted-list.txt', 'w') 30 | 31 | for line in data: 32 | fileHandle.write(line + "\n") 33 | 34 | fileHandle.close() 35 | 36 | def dnsLookup(domain): 37 | record = "" 38 | 39 | try: 40 | answer = dns.resolver.query(domain , "TXT") 41 | return str(answer[0]) 42 | except: 43 | return "None" 44 | 45 | 46 | #Start 47 | def init(): 48 | # Pattern based on https://gist.github.com/0x9900/4471462 49 | 50 | pattern = re.compile(r'ip4[:=](.*?\s)', re.IGNORECASE) 51 | spfRecord = dnsLookup(domain) 52 | 53 | result = re.findall(pattern, spfRecord) 54 | 55 | 56 | #Parse IP addresses 57 | for x in result: 58 | ip = x[0:-1] 59 | 60 | print ip 61 | 62 | if "/" in ip: 63 | print "Dealing with complete network" 64 | 65 | ip_network = ne.IPNetwork(ip) 66 | 67 | for address in list(ip_network): 68 | ip_list.append(str(address)) 69 | 70 | else: 71 | ip_list.append(ip) 72 | 73 | writeToFile(ip_list) 74 | 75 | 76 | init() 77 | 78 | 79 | -------------------------------------------------------------------------------- /tool2/trusted-list.txt: -------------------------------------------------------------------------------- 1 | 192.168.0.1 2 | -------------------------------------------------------------------------------- /visualizations/bubble-chart-AS.py: -------------------------------------------------------------------------------- 1 | 2 | ################################################# 3 | # Author: Yadvir Singh # 4 | # Date: 17-07-2016 # 5 | # Description: # 6 | # # 7 | # Script that generates a bubble chart to # 8 | # review where emails come from, in which # 9 | # quantities and the ratio of successful # 10 | # DMARC authentication results # 11 | # # 12 | # Database variables to be set by user: # 13 | # # 14 | # - dbAddress # 15 | # - dbName # 16 | # - dbUserName # 17 | # - dbPassword # 18 | # # 19 | # Other variables to be set by the user: # 20 | # - asnDatabase that contains IP <-> AN # 21 | # mapping # 22 | # # 23 | # # 24 | ################################################# 25 | 26 | 27 | 28 | from __future__ import division # needed for float division in python < 3 29 | import MySQLdb 30 | import matplotlib.pyplot as plt 31 | import collections 32 | import numpy as np 33 | import random 34 | import pyasn 35 | import struct 36 | import netaddr 37 | import urllib 38 | import json 39 | import sys 40 | 41 | 42 | dbAddress = "127.0.0.1" 43 | dbName = "dmarc" 44 | dbUserName = "dmarc" 45 | dbPassword = "dmarcrp2" 46 | asnDatabase = 'asn-mapping.dat' 47 | 48 | ASvolume = {} 49 | ASipMapping = {} 50 | asndb = pyasn.pyasn(asnDatabase) 51 | 52 | lookup = False 53 | output = open('asn-list.txt', 'w') 54 | 55 | 56 | #Convert MySQL binary blob data to IPv6 string 57 | def convertIPv6(ipv6): 58 | 59 | ipv6String = "" 60 | octetCount = 0 61 | 62 | for a in ipv6: 63 | hexValue = hex(int(struct.unpack('!B', a)[0]))[2:] 64 | 65 | if len(hexValue) < 2: 66 | hexValue = "0" + hexValue 67 | 68 | ipv6String = ipv6String + hexValue 69 | octetCount += 1 70 | 71 | if octetCount == 2: 72 | ipv6String += ":" 73 | octetCount = 0 74 | 75 | return ipv6String[:-1] 76 | 77 | #Store IP addresses found from each AS and their authentication result. 78 | def insertIP(AS ,ipv4, ipv6, dkimResult, spfResult): 79 | 80 | insertString = None 81 | 82 | if ipv6 == None: 83 | insertString = ipv4 84 | else: 85 | insertString = ipv6 86 | 87 | if dkimResult is not "pass" and spfResult is not "pass": 88 | ASipMapping[AS].append("F " + insertString) 89 | 90 | else: 91 | ASipMapping[AS].append("P " + insertString) 92 | 93 | #Write IP address authentication results to a file. 94 | def writeIP(filehandle, array): 95 | 96 | for address in array: 97 | filehandle.write(address + "\n") 98 | 99 | 100 | #Get name linked to the AS 101 | def getASNdata(asn): 102 | url = "http://rdap.arin.net/bootstrap/autnum/" + str(asn) 103 | response = urllib.urlopen(url) 104 | data = json.loads(response.read()) 105 | return data['name'] 106 | 107 | 108 | #Check if the user wants to lookup ASN names 109 | if len(sys.argv) > 1: 110 | argument = sys.argv[1] 111 | if argument == '--asn-lookup': 112 | lookup = True 113 | print "ASN lookup enabled" 114 | 115 | try: 116 | db = MySQLdb.connect(dbAddress, dbUserName, dbPassword, dbName) 117 | except: 118 | print "Error while accessing the database" 119 | 120 | cursor = db.cursor() 121 | 122 | #Fetch all reports that are neede for the bubble chart 123 | cursor.execute("select * from report inner join rptrecord on report.serial=rptrecord.serial") 124 | queryResult = cursor.fetchall() 125 | 126 | #Comment in this for loop for creating real bubble chart 127 | #Parse result and count total volume and successful authentication results. 128 | for x in queryResult: 129 | ip = x[15] 130 | ipString = None 131 | ipv6 = x[16] 132 | ipv6String = None 133 | dkimResult = x[21] 134 | spfResult = x[23] 135 | count = x[17] 136 | 137 | 138 | #Check if the address is IPv4 or IPv6 and retrieve the ASN 139 | if ipv6 == None: 140 | ipString = str(netaddr.IPAddress(ip)) 141 | ASnumber = asndb.lookup(ipString)[0] 142 | else: 143 | #The binary blob from the MySQL database needs to be converted to a 144 | #IPv6 string first. 145 | ipv6String = convertIPv6(ipv6) 146 | ASnumber = asndb.lookup(str(netaddr.IPAddress(ipv6String)))[0] 147 | 148 | try: 149 | volume = ASvolume[ASnumber] 150 | ASipList = ASvolume[ASnumber][3] 151 | 152 | print ASipList 153 | 154 | 155 | dmarcPassCount = volume[0] 156 | volumeCount = volume[1] 157 | 158 | if dkimResult == "pass" or spfResult == "pass": 159 | dmarcPassCount += count 160 | volumeCount += count 161 | else: 162 | volumeCount += count 163 | 164 | 165 | ASvolume[ASnumber] = (dmarcPassCount, volumeCount) 166 | 167 | insertIP(ASnumber, ip, ipv6, dkimResult, spfResult) 168 | 169 | except: 170 | #Apperently we are the first to insert something for this IP block 171 | if dkimResult == "pass" or spfResult == "pass": 172 | ASvolume[ASnumber] = (count, count,[]) 173 | else: 174 | ASvolume[ASnumber] = (0, count, []) 175 | 176 | ASipMapping[ASnumber] = [] 177 | 178 | insertIP(ASnumber, ipString, ipv6String, dkimResult, spfResult) 179 | 180 | 181 | #Comment in this for loop for creating real bubble chart 182 | #Calculate the ratio of each IP chunk 183 | for key, value in ASvolume.iteritems(): 184 | dmarcPass = value[0] 185 | totalCount = value[1] 186 | 187 | ratio = dmarcPass / totalCount 188 | ASvolume[key] = (dmarcPass, totalCount , ratio) 189 | 190 | 191 | 192 | 193 | fig, ax = plt.subplots() 194 | xAxis = [] 195 | yAxis = [] 196 | volume = [] 197 | ip = [] 198 | ratioList = [] 199 | chunkIP = None 200 | chunkRatio = None 201 | chunkVolume = None 202 | 203 | 204 | #Comment in this section for creating real bubble chart 205 | for key, value in ASvolume.iteritems(): 206 | 207 | ASnumber = key 208 | chunkVolume = value[1] 209 | chunkRatio = value[2] 210 | 211 | ip.append(chunkIP) 212 | volume.append(chunkVolume) 213 | ratioList.append(chunkRatio) 214 | 215 | xAxis.append(int(ASnumber)) 216 | yAxis.append(chunkRatio) 217 | if(chunkVolume < 10000): 218 | ax.annotate(str(ASnumber), (xAxis[-1]+(chunkVolume/3), yAxis[-1]), size=20) 219 | else: 220 | ax.annotate(str(ASnumber), (xAxis[-1]+(chunkVolume/10), yAxis[-1]), size=20) 221 | 222 | 223 | #For demo purposes only. Comment out and comment in for loop above if 224 | #you want to generate the bubble chart for a real domain 225 | #############Demo start 226 | '''or x in range (0,4): 227 | 228 | ASNrandom = random.randrange(0, 30000) 229 | 230 | while ASNrandom in ip: 231 | ASNrandom = random.randrange(0, 30000) 232 | 233 | ip.append(ASNrandom) 234 | total = random.randrange(0, 1000) 235 | 236 | 237 | ASipMapping[ASNrandom] = [] 238 | 239 | 240 | for x in range(0, total): 241 | ASipMapping[ASNrandom].append("demo address") 242 | 243 | 244 | ratio = random.random() 245 | 246 | ratioList.append(ratio) 247 | 248 | xAxis.append(ASNrandom) 249 | yAxis.append(ratio) 250 | 251 | 252 | ASvolume[ASNrandom] = (0, total , ratio) 253 | 254 | if(total < 10000): 255 | ax.annotate(str(ASNrandom), (xAxis[-1]+int(total/3), yAxis[-1]), size=20) 256 | else: 257 | ax.annotate(str(1103), (xAxis[-1]+int(total/10), yAxis[-1]), size=20) 258 | 259 | volume.append(total) 260 | 261 | ''' 262 | #############Demo end 263 | 264 | ax.scatter(xAxis, yAxis, s=volume, marker='o', c=ratioList, cmap=plt.cm.RdYlGn, vmin=0, vmax=1) 265 | ax.set_ylim(bottom=-0.1) 266 | ax.set_xlim(left=0, right=max(xAxis)+5000) 267 | ax.set_xlabel('AS Number') 268 | ax.set_ylabel('Ratio') 269 | 270 | plt.show() 271 | 272 | 273 | #Generate asn list which holds volume and ratio per AS 274 | if lookup: 275 | for asn, value in ASvolume.iteritems(): 276 | output.write("----- " + str(asn) + " -----\n") 277 | output.write("Name: \t\t" + getASNdata(asn) + "\n") 278 | output.write("Ratio: \t\t" + str(value[2]) + "\n") 279 | output.write("Volume: \t" + str(value[1]) + "\n") 280 | writeIP(output, ASipMapping[asn]) 281 | output.write("\n\n") 282 | else: 283 | for asn, value in ASvolume.iteritems(): 284 | output.write("----- " + str(asn) + " -----\n") 285 | output.write("Ratio: \t\t" + str(value[2]) + "\n") 286 | output.write("Volume: \t" + str(value[1]) + "\n") 287 | writeIP(output, ASipMapping[asn]) 288 | output.write("\n\n") 289 | 290 | output.close() 291 | 292 | 293 | 294 | -------------------------------------------------------------------------------- /visualizations/bubble-chart.py: -------------------------------------------------------------------------------- 1 | 2 | ################################################# 3 | # Author: Yadvir Singh # 4 | # Date: 17-07-2016 # 5 | # Description: # 6 | # # 7 | # Script that generates a bubble chart to # 8 | # review where emails come from, in which # 9 | # quantities and the ratio of successful # 10 | # DMARC authentication results # 11 | # # 12 | # Database variables to be set by user: # 13 | # # 14 | # - dbAddress # 15 | # - dbName # 16 | # - dbUserName # 17 | # - dbPassword # 18 | # # 19 | # # 20 | ################################################# 21 | 22 | 23 | 24 | from __future__ import division # needed for float division in python < 3 25 | import netaddr as ne 26 | import MySQLdb 27 | import matplotlib.pyplot as plt 28 | import collections 29 | import numpy as np 30 | import random 31 | 32 | 33 | dbAddress = "127.0.0.1" 34 | dbName = "dmarc" 35 | dbUserName = "dmarc" 36 | dbPassword = "dmarcrp2" 37 | 38 | 39 | ipVolume = {} 40 | 41 | 42 | #The complete IPv4 Space divided into 256 chunks. Used for categorization. 43 | ipBlocks = ['0.0.0.0/8', '1.0.0.0/8', '2.0.0.0/8', '3.0.0.0/8', '4.0.0.0/8', '5.0.0.0/8', '6.0.0.0/8', '7.0.0.0/8', '8.0.0.0/8', '9.0.0.0/8', '10.0.0.0/8', '11.0.0.0/8', '12.0.0.0/8', '13.0.0.0/8', '14.0.0.0/8', '15.0.0.0/8', '16.0.0.0/8', '17.0.0.0/8', '18.0.0.0/8', '19.0.0.0/8', '20.0.0.0/8', '21.0.0.0/8', '22.0.0.0/8', '23.0.0.0/8', '24.0.0.0/8', '25.0.0.0/8', '26.0.0.0/8', '27.0.0.0/8', '28.0.0.0/8', '29.0.0.0/8', '30.0.0.0/8', '31.0.0.0/8', '32.0.0.0/8', '33.0.0.0/8', '34.0.0.0/8', '35.0.0.0/8', '36.0.0.0/8', '37.0.0.0/8', '38.0.0.0/8', '39.0.0.0/8', '40.0.0.0/8', '41.0.0.0/8', '42.0.0.0/8', '43.0.0.0/8', '44.0.0.0/8', '45.0.0.0/8', '46.0.0.0/8', '47.0.0.0/8', '48.0.0.0/8', '49.0.0.0/8', '50.0.0.0/8', '51.0.0.0/8', '52.0.0.0/8', '53.0.0.0/8', '54.0.0.0/8', '55.0.0.0/8', '56.0.0.0/8', '57.0.0.0/8', '58.0.0.0/8', '59.0.0.0/8', '60.0.0.0/8', '61.0.0.0/8', '62.0.0.0/8', '63.0.0.0/8', '64.0.0.0/8', '65.0.0.0/8', '66.0.0.0/8', '67.0.0.0/8', '68.0.0.0/8', '69.0.0.0/8', '70.0.0.0/8', '71.0.0.0/8', '72.0.0.0/8', '73.0.0.0/8', '74.0.0.0/8', '75.0.0.0/8', '76.0.0.0/8', '77.0.0.0/8', '78.0.0.0/8', '79.0.0.0/8', '80.0.0.0/8', '81.0.0.0/8', '82.0.0.0/8', '83.0.0.0/8', '84.0.0.0/8', '85.0.0.0/8', '86.0.0.0/8', '87.0.0.0/8', '88.0.0.0/8', '89.0.0.0/8', '90.0.0.0/8', '91.0.0.0/8', '92.0.0.0/8', '93.0.0.0/8', '94.0.0.0/8', '95.0.0.0/8', '96.0.0.0/8', '97.0.0.0/8', '98.0.0.0/8', '99.0.0.0/8', '100.0.0.0/8', '101.0.0.0/8', '102.0.0.0/8', '103.0.0.0/8', '104.0.0.0/8', '105.0.0.0/8', '106.0.0.0/8', '107.0.0.0/8', '108.0.0.0/8', '109.0.0.0/8', '110.0.0.0/8', '111.0.0.0/8', '112.0.0.0/8', '113.0.0.0/8', '114.0.0.0/8', '115.0.0.0/8', '116.0.0.0/8', '117.0.0.0/8', '118.0.0.0/8', '119.0.0.0/8', '120.0.0.0/8', '121.0.0.0/8', '122.0.0.0/8', '123.0.0.0/8', '124.0.0.0/8', '125.0.0.0/8', '126.0.0.0/8', '127.0.0.0/8', '128.0.0.0/8', '129.0.0.0/8', '130.0.0.0/8', '131.0.0.0/8', '132.0.0.0/8', '133.0.0.0/8', '134.0.0.0/8', '135.0.0.0/8', '136.0.0.0/8', '137.0.0.0/8', '138.0.0.0/8', '139.0.0.0/8', '140.0.0.0/8', '141.0.0.0/8', '142.0.0.0/8', '143.0.0.0/8', '144.0.0.0/8', '145.0.0.0/8', '146.0.0.0/8', '147.0.0.0/8', '148.0.0.0/8', '149.0.0.0/8', '150.0.0.0/8', '151.0.0.0/8', '152.0.0.0/8', '153.0.0.0/8', '154.0.0.0/8', '155.0.0.0/8', '156.0.0.0/8', '157.0.0.0/8', '158.0.0.0/8', '159.0.0.0/8', '160.0.0.0/8', '161.0.0.0/8', '162.0.0.0/8', '163.0.0.0/8', '164.0.0.0/8', '165.0.0.0/8', '166.0.0.0/8', '167.0.0.0/8', '168.0.0.0/8', '169.0.0.0/8', '170.0.0.0/8', '171.0.0.0/8', '172.0.0.0/8', '173.0.0.0/8', '174.0.0.0/8', '175.0.0.0/8', '176.0.0.0/8', '177.0.0.0/8', '178.0.0.0/8', '179.0.0.0/8', '180.0.0.0/8', '181.0.0.0/8', '182.0.0.0/8', '183.0.0.0/8', '184.0.0.0/8', '185.0.0.0/8', '186.0.0.0/8', '187.0.0.0/8', '188.0.0.0/8', '189.0.0.0/8', '190.0.0.0/8', '191.0.0.0/8', '192.0.0.0/8', '193.0.0.0/8', '194.0.0.0/8', '195.0.0.0/8', '196.0.0.0/8', '197.0.0.0/8', '198.0.0.0/8', '199.0.0.0/8', '200.0.0.0/8', '201.0.0.0/8', '202.0.0.0/8', '203.0.0.0/8', '204.0.0.0/8', '205.0.0.0/8', '206.0.0.0/8', '207.0.0.0/8', '208.0.0.0/8', '209.0.0.0/8', '210.0.0.0/8', '211.0.0.0/8', '212.0.0.0/8', '213.0.0.0/8', '214.0.0.0/8', '215.0.0.0/8', '216.0.0.0/8', '217.0.0.0/8', '218.0.0.0/8', '219.0.0.0/8', '220.0.0.0/8', '221.0.0.0/8', '222.0.0.0/8', '223.0.0.0/8', '224.0.0.0/8', '225.0.0.0/8', '226.0.0.0/8', '227.0.0.0/8', '228.0.0.0/8', '229.0.0.0/8', '230.0.0.0/8', '231.0.0.0/8', '232.0.0.0/8', '233.0.0.0/8', '234.0.0.0/8', '235.0.0.0/8', '236.0.0.0/8', '237.0.0.0/8', '238.0.0.0/8', '239.0.0.0/8', '240.0.0.0/8', '241.0.0.0/8', '242.0.0.0/8', '243.0.0.0/8', '244.0.0.0/8', '245.0.0.0/8', '246.0.0.0/8', '247.0.0.0/8', '248.0.0.0/8', '249.0.0.0/8', '250.0.0.0/8', '251.0.0.0/8', '252.0.0.0/8', '253.0.0.0/8', '254.0.0.0/8', '255.0.0.0/8'] 44 | 45 | 46 | try: 47 | db = MySQLdb.connect(dbAddress, dbUserName, dbPassword, dbName) 48 | except: 49 | print "Error while accessing the database" 50 | 51 | cursor = db.cursor() 52 | 53 | #Fetch all reports that are neede for the bubble chart 54 | cursor.execute("select * from report inner join rptrecord on report.serial=rptrecord.serial") 55 | queryResult = cursor.fetchall() 56 | 57 | 58 | #Parse result and count total volume and successful authentication results. 59 | for x in queryResult: 60 | ip = x[15] 61 | dkimResult = x[21] 62 | spfResult = x[23] 63 | count = x[17] 64 | 65 | 66 | # Categorize the IP address from the report into one of the 256 IP chunks. 67 | ipSpace = str(ne.all_matching_cidrs(ip, ipBlocks)[0]) 68 | 69 | try: 70 | volume = ipVolume[ipSpace] 71 | dmarcPassCount = volume[0] 72 | volumeCount = volume[1] 73 | 74 | if dkimResult == "pass" or spfResult == "pass": 75 | dmarcPassCount += count 76 | volumeCount += count 77 | else: 78 | volumeCount += count 79 | 80 | ipVolume[ipSpace] = (dmarcPassCount, volumeCount) 81 | 82 | except: 83 | #Apperently we are the first to insert something for this IP block 84 | if dkimResult == "pass" or spfResult == "pass": 85 | ipVolume[ipSpace] = (count, count) 86 | else: 87 | ipVolume[ipSpace] = (0, count) 88 | 89 | 90 | 91 | #Calculate the ratio of each IP chunk 92 | for key, value in ipVolume.iteritems(): 93 | dmarcPass = value[0] 94 | totalCount = value[1] 95 | 96 | ratio = dmarcPass / totalCount 97 | ipVolume[key] = (dmarcPass, totalCount , ratio) 98 | 99 | 100 | fig, ax = plt.subplots() 101 | 102 | 103 | xAxis = [] 104 | yAxis = [] 105 | volume = [] 106 | ip = [] 107 | ratioList = [] 108 | chunkIP = None 109 | chunkRatio = None 110 | chunkVolume = None 111 | 112 | 113 | #Comment in this section for creating real bubble chart 114 | """for key, value in ipVolume.iteritems(): 115 | 116 | chunkIP = key.partition(".")[0] 117 | chunkVolume = value[1] 118 | chunkRatio = value[2] 119 | 120 | 121 | print int(chunkIP) 122 | 123 | ip.append(chunkIP) 124 | volume.append(chunkVolume) 125 | ratioList.append(chunkRatio) 126 | 127 | xAxis.append(int(chunkIP)) 128 | yAxis.append(chunkRatio) 129 | 130 | ax.annotate(str(chunkIP), (xAxis[-1]+(chunkVolume/600), yAxis[-1]), size=20) 131 | 132 | """ 133 | 134 | #For demo purposes only. Comment out and comment in for loop above if 135 | #you want to generate the bubble chart for a real domain 136 | for x in range (0,12): 137 | 138 | ipRandom = random.randrange(0, 256) 139 | 140 | while ipRandom in ip: 141 | ipRandom = random.randrange(0, 256) 142 | 143 | ip.append(ipRandom) 144 | total = random.randrange(0, 10000) 145 | 146 | ratio = random.random() 147 | 148 | ratioList.append(ratio) 149 | 150 | xAxis.append(ipRandom) 151 | yAxis.append(ratio) 152 | 153 | ax.annotate(str(ipRandom), (xAxis[-1]+(total/600), yAxis[-1]), size=20) 154 | 155 | volume.append(total) 156 | 157 | ax.scatter(xAxis, yAxis, s=volume, marker='o', c=ratioList, cmap=plt.cm.RdYlGn) 158 | ax.set_ylim(bottom=0) 159 | ax.set_xlabel('IP chunk') 160 | ax.set_ylabel('Ratio') 161 | 162 | plt.show() 163 | 164 | -------------------------------------------------------------------------------- /visualizations/heatMap-in.py: -------------------------------------------------------------------------------- 1 | ################################################# 2 | # Author: Yadvir Singh # 3 | # Date: 17-07-2016 # 4 | # Description: # 5 | # # 6 | # Script that generates a heat map of # 7 | # incoming reports by name. Each tile # 8 | # contains the domain, the total number of # 9 | # email, volume of DMARC passes and DMARC # 10 | # failures. Each tile is color coded with # 11 | # a ratio. # 12 | # This ratio is calculated as: total DMARC # 13 | # passes divided by total email volume. # 14 | # # 15 | # Database variables to be set by user: # 16 | # # 17 | # - dbAddress # 18 | # - dbName # 19 | # - dbUserName # 20 | # - dbPassword # 21 | # # 22 | # # 23 | ################################################# 24 | 25 | from __future__ import division 26 | import matplotlib.pyplot as plt 27 | import MySQLdb 28 | import numpy as np 29 | from mpl_toolkits.mplot3d import axes3d, Axes3D 30 | import sys 31 | import random 32 | import string 33 | 34 | 35 | dbAddress = "127.0.0.1" 36 | dbName = "dmarc" 37 | dbUserName = "dmarc" 38 | dbPassword = "dmarcrp2" 39 | 40 | 41 | #Main function that retrieves data from the database and populates all counters. 42 | def getData(dbAddress, dbName, dbUserName, dbPassword): 43 | 44 | domainsOfIntrest = [] 45 | domains = [] 46 | domainRatioList = [] 47 | 48 | try: 49 | db = MySQLdb.connect(dbAddress, dbUserName, dbPassword, dbName) 50 | 51 | except: 52 | print "Error while accessing the database" 53 | 54 | 55 | #Fetch all organization from which reports have been received 56 | cursor = db.cursor() 57 | cursor.execute("select distinct org from report") 58 | queryResult = cursor.fetchall() 59 | 60 | dmarcPass = 0 61 | dmarcFail = 0 62 | totalCount = 0 63 | ratio = 0 64 | 65 | 66 | # Fetch authentication results per organization 67 | for x in queryResult: 68 | domain = x[0] 69 | cursor.execute("select * from report inner join rptrecord on report.serial=rptrecord.serial where org = \"" + domain + "\"") 70 | queryResult = cursor.fetchall() 71 | 72 | for row in queryResult: 73 | dkimResult = row[21] 74 | spfResult = row[23] 75 | count = row[17] 76 | 77 | if dkimResult == "pass" or spfResult == "pass": 78 | dmarcPass += count 79 | else: 80 | dmarcFail += count 81 | 82 | totalCount += count 83 | 84 | ratio = dmarcPass/totalCount 85 | domainRatioList.append((domain, ratio, totalCount, dmarcPass, dmarcFail)) 86 | 87 | dmarcPass = 0 88 | dmarcFail = 0 89 | totalCount = 0 90 | ratio = 0 91 | 92 | return domainRatioList 93 | 94 | 95 | 96 | #Function that inserts text values in each tile. Based on: 97 | #http://stackoverflow.com/questions/25071968/heatmap-with-text-in-each-cell-with-matplotlibs-pyplot 98 | def show_values(pc, domainList, fmt="%.2f", **kw): 99 | from itertools import izip 100 | pc.update_scalarmappable() 101 | ax = pc.get_axes() 102 | for p, color, value, ratio in izip(pc.get_paths(), pc.get_facecolors(), pc.get_array(), domainList): 103 | x, y = p.vertices[:-2, :].mean(0) 104 | 105 | #Choose a lite color if we the tile is dark and vice versa. 106 | if np.all(color[:3] > 0.5): 107 | color = (0.0, 0.0, 0.0) 108 | else: 109 | color = (1.0, 1.0, 1.0) 110 | ax.text(x, y+0.15, ratio[0], ha="center", va="center", color=color, size='large', **kw) 111 | ax.text(x, y-0.15, "Total " + str(ratio[2]) + "\n DMARC-P " + str(ratio[3]) + "\n DMARC-F " + str(ratio[4]), ha="center", va="center", color=color, size='x-small', **kw) 112 | 113 | 114 | #Funtion that generates the heatmap from a list of domains and their 115 | #authentication results. Based on: 116 | #http://stackoverflow.com/questions/14391959/heatmap-in-matplotlib-with-pcolor 117 | def generateMap(ratioList): 118 | 119 | #If there are not enough results, populate missing tiles manually. We need 120 | #more elegant solution for this. 121 | while len(ratioList) < 36: 122 | ratioList.append(("",1.0,0,0,0)) 123 | 124 | values = [] 125 | 126 | 127 | #Uncomment the following for a demo heatmap 128 | """ 129 | demo = [] 130 | 131 | for x in range (0,36): 132 | 133 | domain = ''.join(random.choice(string.lowercase) for i in range(8)) + ".com" 134 | 135 | total = random.randrange(0, 1000) 136 | 137 | dmarcPass = random.randrange(0,total) 138 | 139 | dmarcFail = total - dmarcPass 140 | 141 | ratio = dmarcPass/total 142 | 143 | demo.append((domain, ratio, total, dmarcPass, dmarcFail )) 144 | 145 | demo = sorted(demo, key=lambda x: x[1]) 146 | 147 | ratioList = demo 148 | 149 | """ 150 | 151 | ratioList = sorted(ratioList, key=lambda x: x[1]) 152 | 153 | for data in ratioList: 154 | values.append(data[1]) 155 | 156 | 157 | #Create the tile map 158 | values = np.array(values).reshape((6, 6)) 159 | 160 | column_labels = list('ABCD') 161 | row_labels = list('WXYZ') 162 | 163 | data = values 164 | fig, ax = plt.subplots() 165 | 166 | heatmap = ax.pcolor(data, cmap=plt.cm.RdYlGn, vmin = 0.0, vmax = 1.0) 167 | heatmap.update_scalarmappable() 168 | 169 | col = heatmap.get_facecolors() 170 | 171 | show_values(heatmap, ratioList) 172 | plt.colorbar(heatmap) 173 | plt.show() 174 | 175 | ratioList = getData(dbAddress, dbName, dbUserName, dbPassword) 176 | generateMap(ratioList) 177 | 178 | -------------------------------------------------------------------------------- /visualizations/heatMap-out.py: -------------------------------------------------------------------------------- 1 | ################################################# 2 | # Author: Yadvir Singh # 3 | # Date: 17-07-2016 # 4 | # Description: # 5 | # # 6 | # Script that generates a heat map of # 7 | # outgoing reports by name. Each tile # 8 | # contains the domain, # 9 | # the total number of email, volume # 10 | # of DMARC passes and DMARC failures. # 11 | # Each tile is color coded with a ratio. # 12 | # This ratio is calulated as: total DMARC # 13 | # passes divided by total email volume. # 14 | # This script reads a MySQL table that # 15 | # is generated by OpenDMARC # 16 | # # 17 | # Database variables to be set by user: # 18 | # # 19 | # - dbAddress # 20 | # - dbName # 21 | # - dbUserName # 22 | # - dbPassword # 23 | # # 24 | # # 25 | ################################################# 26 | from __future__ import division # needed for float division in python < 3 27 | import matplotlib.pyplot as plt 28 | import MySQLdb 29 | import numpy as np 30 | from mpl_toolkits.mplot3d import axes3d, Axes3D 31 | import sys 32 | import random 33 | import string 34 | 35 | 36 | dbAddress = "127.0.0.1" 37 | dbName = "opendmarc" 38 | dbUserName = "dmarc" 39 | dbPassword = "dmarcrp2" 40 | 41 | #Main function that retrieves data from the database and populates all counters. 42 | def getData(dbAddress, dbUserName, dbPassword, dbName): 43 | 44 | domainsOfIntrest = [] 45 | domains = [] 46 | domainRatioList = [] 47 | 48 | #Prepare database connection 49 | try: 50 | db = MySQLdb.connect(dbAddress, dbUserName, dbPassword, dbName) 51 | 52 | except: 53 | print "Error while accessing the database" 54 | 55 | 56 | #Fetch all organization to which reports are sent 57 | cursor = db.cursor() 58 | cursor.execute("select distinct repuri from requests") 59 | queryResult = cursor.fetchall() 60 | 61 | dmarcPass = 0 62 | dmarcFail = 0 63 | totalCount = 0 64 | ratio = 0 65 | 66 | # Fetch authentication results per organization 67 | for x in queryResult: 68 | 69 | domain = x[0] 70 | domains.append(domain) 71 | cursor.execute("SELECT * FROM messages INNER JOIN requests ON messages.env_domain=requests.id where repuri = '" + x[0] + "'") 72 | queryResult = cursor.fetchall() 73 | 74 | 75 | for y in queryResult: 76 | 77 | dkimResult = y[11] 78 | spfResult = y[12] 79 | 80 | if dkimResult == 5 or spfResult == 5: 81 | dmarcPass += 1 82 | else: 83 | dmarcFails += 1 84 | 85 | total = dmarcPass + dmarcFail 86 | 87 | ratio = dmarcPass / total 88 | 89 | domainRatioList.append((domain, ratio, total, dmarcPass, dmarcFail)) 90 | 91 | dmarcPass = 0 92 | dmarcFail = 0 93 | totalCount = 0 94 | ratio = 0 95 | 96 | return domainRatioList 97 | 98 | 99 | #Function that inserts text values in each tile. Based on: 100 | #http://stackoverflow.com/questions/25071968/heatmap-with-text-in-each-cell-with-matplotlibs-pyplot 101 | def show_values(pc, domainList, fmt="%.2f", **kw): 102 | from itertools import izip 103 | pc.update_scalarmappable() 104 | ax = pc.get_axes() 105 | for p, color, value, ratio in izip(pc.get_paths(), pc.get_facecolors(), pc.get_array(), domainList): 106 | x, y = p.vertices[:-2, :].mean(0) 107 | 108 | #Choose a lite color if we the tile is dark and vice versa. 109 | if np.all(color[:3] > 0.5): 110 | color = (0.0, 0.0, 0.0) 111 | else: 112 | color = (1.0, 1.0, 1.0) 113 | ax.text(x, y+0.15, ratio[0], ha="center", va="center", color=color, size='large', **kw) 114 | ax.text(x, y-0.15, "Total " + str(ratio[2]) + "\n DMARC-P " + str(ratio[3]) + "\n DMARC-F " + str(ratio[4]), ha="center", va="center", color=color, size='x-small', **kw) 115 | 116 | 117 | #Funtion that generates the heatmap from a list of domains and their 118 | #authentication results. Based on: 119 | #http://stackoverflow.com/questions/14391959/heatmap-in-matplotlib-with-pcolor 120 | def generateMap(ratioList): 121 | 122 | #If there are not enough results, populate missing tiles manually. We need 123 | #more elegant solution for this. 124 | while len(ratioList) < 36: 125 | ratioList.append(("",1.0,0,0,0)) 126 | 127 | values = [] 128 | 129 | ratioList = sorted(ratioList, key=lambda x: x[1]) 130 | 131 | for data in ratioList: 132 | values.append(data[1]) 133 | 134 | 135 | #Create the tile map 136 | values = np.array(values).reshape((6, 6)) 137 | 138 | column_labels = list('ABCD') 139 | row_labels = list('WXYZ') 140 | 141 | data = values 142 | fig, ax = plt.subplots() 143 | 144 | heatmap = ax.pcolor(data, cmap=plt.cm.RdYlGn, vmin = 0.0, vmax = 1.0) 145 | heatmap.update_scalarmappable() 146 | 147 | col = heatmap.get_facecolors() 148 | 149 | show_values(heatmap, ratioList) 150 | plt.colorbar(heatmap) 151 | plt.show() 152 | 153 | ratioList = getData(dbAddress, dbUserName, dbPassword, dbName) 154 | generateMap(ratioList) 155 | 156 | 157 | --------------------------------------------------------------------------------