├── .gitignore ├── dmarcReportProcessor.timer ├── dmarcReportProcessor.service ├── LICENSE.md ├── README.md ├── dmarc-convertor.sh └── bin ├── dmarc-parser.py └── imap-client.py /.gitignore: -------------------------------------------------------------------------------- 1 | var/ 2 | pwd 3 | -------------------------------------------------------------------------------- /dmarcReportProcessor.timer: -------------------------------------------------------------------------------- 1 | ### DMARC Report Processor timer ### 2 | # 3 | # DMARC Report Processor take RUA Report 4 | # and manage them for you! 5 | 6 | [Unit] 7 | Description=DMARC Report Processor timer 8 | After=syslog.target network.target 9 | 10 | [Timer] 11 | OnCalendar=daily 12 | 13 | [Install] 14 | WantedBy=rsyslog.service 15 | -------------------------------------------------------------------------------- /dmarcReportProcessor.service: -------------------------------------------------------------------------------- 1 | ### DMARC Report Processor ### 2 | # 3 | 4 | [Unit] 5 | Description=DMARC Report Processor Service 6 | After=syslog.target network.target 7 | 8 | [Service] 9 | User=root 10 | Environment=RUAEMAIL=dmarc@example.com 11 | Environment=IMAPSERVER=msa.example.com 12 | Environment=RUAFOLDER=INBOX/rua 13 | Environment=PWDFILE=/usr/local/dmarc-report-processor/pwd 14 | ExecStart=/usr/local/dmarc-report-processor/dmarc-convertor.sh -u ${RUAEMAIL} -s ${IMAPSERVER} -P ${PWDFILE} -c /etc/pki/tls/certs/ca-bundle.crt 15 | -------------------------------------------------------------------------------- /LICENSE.md: -------------------------------------------------------------------------------- 1 | Software License Agreement (BSD License) 2 | ======================================== 3 | 4 | Copyright (c) 2014, Yahoo! Inc. All rights reserved. 5 | ---------------------------------------------------- 6 | 7 | Redistribution and use of this software in source and binary forms, 8 | with or without modification, are permitted provided that the following 9 | conditions are met: 10 | 11 | * Redistributions of source code must retain the above 12 | copyright notice, this list of conditions and the 13 | following disclaimer. 14 | * Redistributions in binary form must reproduce the above 15 | copyright notice, this list of conditions and the 16 | following disclaimer in the documentation and/or other 17 | materials provided with the distribution. 18 | * Neither the name of Yahoo! Inc. nor the names of its 19 | contributors may be used to endorse or promote products 20 | derived from this software without specific prior 21 | written permission of Yahoo! Inc. 22 | 23 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS 24 | IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED 25 | TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A 26 | PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 27 | OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 28 | SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 29 | LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 30 | DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 31 | THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 32 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 33 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 34 | 35 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | Script to pull DMARC records, process and pass it to splunk. 2 | 3 | **imap-client.py** - Pull attachments from mail imap server and store 4 | it in the given directory. This is a generic program that can be used to 5 | fetch emails and/or attachments using IMAP protocol. 6 | 7 | **dmarc-parser.py** - Convert the xml files to comma-seperated key=value 8 | pair (line oriented output for splunk). This script can handle large xml files 9 | 10 | **dmarc-convertor.sh** - An uber script to manage the workflow end-to-end: 11 | 1. Download attachments from mail server 12 | 2. Unzip the attachments 13 | 3. Parse unzipped xml files and convert it line oriented format for splunk 14 | 15 | ### Usage 16 | 17 | #### imap-client.py 18 | 19 | ``` 20 | imap-client.py [-h] [-v] [--attachmentsonly] [--disablereadonly] 21 | [--quiet] -s HOST [-p PORT] -c CACERTS -u USER -f FOLDER 22 | -o OUTDIR [-S SEARCH] [-P PWDFILE] 23 | 24 | optional arguments: 25 | -h, --help show this help message and exit 26 | -v, --verbose increase output verbosity 27 | --attachmentsonly download attachments only 28 | --disablereadonly enable state changes on server; Default readonly 29 | --quiet supress all comments (stdout) 30 | -s HOST, --host HOST imap server; eg. imap.mail.yahoo.com 31 | -p PORT, --port PORT imap server port; Default is 993 32 | -c CACERTS, --cacerts CACERTS 33 | CA certificates, which are used to validate 34 | certificates passed from imap server 35 | -u USER, --user USER user's email id 36 | -f FOLDER, --folder FOLDER 37 | mail folder from which the mail to retrieve 38 | -o OUTDIR, --outdir OUTDIR 39 | directory to output 40 | -S SEARCH, --search SEARCH 41 | search criteria, defined in IMAP RFC 3501; eg. "SINCE 42 | \"8-Sep-2014\"" 43 | -P PWDFILE, --pwdfile PWDFILE 44 | A file that stores IMAP user password. If not set, the 45 | user is prompted to provide a passwd 46 | 47 | Example: 48 | % imap-client.py -s imap.example.com -c ./cacert.pem -u dmarc@example.com -f inbox -o ./mymail -S "SINCE \"8-Sep-2014\"" -P 49 | ./paswdfile 50 | ``` 51 | 52 | #### dmarc-parser.py 53 | 54 | ``` 55 | dmarc-parser.py [-h] dmarcfile 56 | 57 | positional arguments: 58 | dmarcfile dmarc file in XML format 59 | 60 | optional arguments: 61 | -h, --help show this help message and exit 62 | 63 | Example: 64 | % dmarc-parser.py dmarc-xml-file 1> outfile.csv 65 | ``` 66 | 67 | #### dmarc-convertor.sh 68 | 69 | ``` 70 | dmarc-convertor.sh -u user_emailid -s imapserver -c cacertfile [-p port] [-P pwdfile] [-h] 71 | Options: 72 | -u User email id 73 | -P File that contains user password. Default: The user will be 74 | prompted to provide password if you leave this option. 75 | WARNING: The file should be with permission 76 | 0400 or 0440 (ie should NOT be world readable) 77 | -s IMAP server name 78 | -p IMAP port number. Default: 993 79 | -c CA certificate file (eg. cacert.pem), used to validate certificates 80 | passed from IMAP server 81 | -h Help 82 | 83 | Example: 84 | % dmarc-convertor.sh -u dmarc@example.com -P ./pwd -s imap.example.com -p 993 -c ./cacert.pem 85 | ``` 86 | 87 | The 88 | 89 | ``` 90 | dmarcReportProcessor.service 91 | dmarcReportProcessor.timer 92 | ``` 93 | 94 | shows a possible systemd call to execute the report collection. The env RUAFOLDER defines the IMAP folder where the reports are. 95 | 96 | 97 | *NOTE* The above script expects `imap-client.py` and `dmarc-parser.py` available in $ROOT/bin. You may change the path by modifiying `dmarc-convertor.sh`. 98 | 99 | Tested on python 2.7 100 | -------------------------------------------------------------------------------- /dmarc-convertor.sh: -------------------------------------------------------------------------------- 1 | #!/bin/env bash 2 | # Copyright (c) 2014, Yahoo! Inc. 3 | # Copyrights licensed under the New BSD License. See the 4 | # accompanying LICENSE.txt file for terms. 5 | # 6 | # Author Binu P. Ramakrishnan 7 | # Created 09/18/2014 8 | # 9 | # This script does 3 things: 10 | # 1. Pull dmarc report attachments from yahoo mail 11 | # 2. Unzip these report files to a separate folder 12 | # 3. Convert dmarc xml files to line oriented format for splunk 13 | # 14 | 15 | ROOT='/usr/local/dmarc-report-processor' 16 | DMARC_ROOT="${ROOT}/var" 17 | ATTACH="${DMARC_ROOT}/attach_raw" 18 | XML="${DMARC_ROOT}/dmarc_xml" 19 | DMARC_SPLUNK="${DMARC_ROOT}/dmarc_splunk" 20 | 21 | os=`uname` 22 | ydate=`date -d "yesterday 00:00 " '+%d-%h-%Y'` 23 | #ydate=`date -d "1 week ago 13:00 " '+%d-%h-%Y'` 24 | if [ "$os" == "Darwin" ] 25 | then 26 | ydate=`date -v-1d +%d-%h-%Y` 27 | fi 28 | 29 | tdate=`date '+%d-%h-%Y'` 30 | 31 | d_host="" 32 | d_port=993 33 | d_user="" 34 | d_pwd="" 35 | d_cert="" 36 | 37 | dmarc_help() { 38 | cat << EOF 39 | Usage: ${0##*/} -u user_emailid -s imapserver -c cacertfile [-p port] [-P pwdfile] [-h] 40 | Options: 41 | -u User email id 42 | -P File that contains user password. Default: The user will be 43 | prompted to provide password if you leave this option. 44 | WARNING: The file should be with permission 45 | 0400 or 0440 (ie should NOT be world readable) 46 | -s IMAP server name 47 | -p IMAP port number. Default: 993 48 | -c CA certificate file (eg. cacert.pem), used to validate certificates 49 | passed from IMAP server 50 | -h Help 51 | 52 | Example: 53 | ${0##*/} -u dmarc@example.com -P ./pwd -s imap.example.com -p 993 -c ./cacert.pem 54 | 55 | EOF 56 | } 57 | 58 | OPTIND=1 59 | while getopts "hs:p:u:P:c:" opt; do 60 | case "$opt" in 61 | h) 62 | dmarc_help 63 | exit 0 64 | ;; 65 | s) d_host=$OPTARG 66 | ;; 67 | p) d_port=$OPTARG 68 | ;; 69 | u) d_user=$OPTARG 70 | ;; 71 | P) d_pwd=$OPTARG 72 | ;; 73 | c) d_cert=$OPTARG 74 | ;; 75 | '?') 76 | dmarc_help >&2 77 | exit 1 78 | ;; 79 | esac 80 | done 81 | 82 | shift "$((OPTIND-1))" # Shift off the options and optional --. 83 | 84 | if [ -z "${d_user}" ] 85 | then 86 | echo "Error: User name not provided (-u)" >&2 87 | exit 1 88 | fi 89 | 90 | # create directory if it doesn't exists. Ignore error if exists 91 | mkdir -m 0755 ${DMARC_ROOT} 2> /dev/null 92 | mkdir -m 0755 ${ATTACH} 2> /dev/null 93 | mkdir -m 0755 ${XML} 2> /dev/null 94 | mkdir -m 0755 ${DMARC_SPLUNK} 2> /dev/null 95 | mkdir -m 0755 ${ATTACH}/${ydate} 96 | if [ "$?" -ne "0" ] 97 | then 98 | rm -rf "${ATTACH}/${ydate}.old" 2> /dev/null 99 | mv "${ATTACH}/${ydate}" "${ATTACH}/${ydate}.old" 100 | mkdir -m 0755 "${ATTACH}/${ydate}" 101 | fi 102 | 103 | 104 | # imap search criteria. Defined here: http://tools.ietf.org/html/rfc3501.html#page-49 105 | d_search="SINCE \"${ydate}\" BEFORE \"${tdate}\"" 106 | #d_search="SINCE \"11-Sep-2014\" BEFORE \"12-Sep-2014\"" 107 | 108 | #1 109 | echo "Step 1: Fetch dmarc reports from mailbox" 110 | echo "----------------------------------------" 111 | ${ROOT}/bin/imap-client.py --attachmentsonly -s "${d_host}" -c "${d_cert}" --port "${d_port}" -u "${d_user}" -o ${ATTACH}/${ydate} -f ${RUAFOLDER} --pwdfile "${d_pwd}" -S "${d_search}" 112 | if [ "$?" -ne "0" ] 113 | then 114 | echo "Error: imap-client mail attachment fetch failed; exiting ..." 115 | exit 1 116 | fi 117 | 118 | #2 119 | shopt -s nullglob 120 | files=( "${ATTACH}/${ydate}"/* ) 121 | if [ "${#files[@]}" -eq "0" ] 122 | then 123 | echo "No new reports found. Exiting ..." 124 | exit 0 125 | fi 126 | 127 | echo "Step 2: Unzipping files" 128 | echo "-----------------------" 129 | mkdir "${XML}/${ydate}" 130 | rm -rf "${XML}/${ydate}/*" 2> /dev/null 131 | for f in "${files[@]}"; do 132 | echo "$f" 133 | extn="${f##*.}" 134 | if [ "$extn" == "zip" ] 135 | then 136 | # remove just in case 137 | rm -f ${XML}/${ydate}/${f} 2>/dev/null 138 | unzip $f -d "${XML}/${ydate}" 139 | elif [ "$extn" == "gz" ] 140 | then 141 | fname=$(basename $f) 142 | gunzip -c $f > "${XML}/${ydate}/${fname%.*}" 143 | else 144 | echo "File extension not supported: ${f}; skipping ..." 145 | fi 146 | done 147 | 148 | #3 149 | echo "Step 3: Converting xml files" 150 | echo "----------------------------" 151 | mkdir "${DMARC_SPLUNK}/${ydate}" 152 | rm -rf "${DMARC_SPLUNK}/${ydate}/*" 2> /dev/null 153 | for f in "${XML}/${ydate}"/*; do 154 | fname=$(basename $f) 155 | fname2=$(printf '%q' "${f}") 156 | ${ROOT}/bin/dmarc-parser.py ${f} > "${DMARC_SPLUNK}/${ydate}/${fname%.*}.log" 157 | if [ "$?" -ne "0" ] 158 | then 159 | echo "Error: Splunk conversion failed; File: ${f}" 160 | fi 161 | done 162 | 163 | rm "${DMARC_SPLUNK}/latest" 2> /dev/null 164 | ln -s "${DMARC_SPLUNK}/${ydate}" "${DMARC_SPLUNK}/latest" 165 | 166 | #remove folders that are more than 5 days old 167 | find "${ATTACH}/${ydate}" -type d -ctime +5 2>/dev/null | xargs rm -rf 168 | find "${XML}/${ydate}" -type d -ctime +5 2>/dev/null | xargs rm -rf 169 | find "${DMARC_SPLUNK}/${ydate}" -type d -ctime +5 2>/dev/null | xargs rm -rf 170 | find "${ROOT}/logs/dmarc-report-processor" -type d -ctime +5 2>/dev/null | xargs rm -rf 171 | 172 | 173 | -------------------------------------------------------------------------------- /bin/dmarc-parser.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | # 3 | # Copyright (c) 2014, Yahoo! Inc. 4 | # Copyrights licensed under the New BSD License. See the 5 | # accompanying LICENSE.txt file for terms. 6 | # 7 | # Author Binu P. Ramakrishnan 8 | # Created 09/12/2014 9 | # 10 | # Program that accepts a (LARGE) xml file and convert it to 11 | # easy-to-process comma separated key=value pair format 12 | # (line oriented splunk friendly record format) 13 | # 14 | # Usage: dmarc-parser.py 1> outfile 15 | # Returns 0 for success and 1 for errors. 16 | # Error messages are directed to stderr 17 | # 18 | import sys 19 | import xml.etree.cElementTree as etree 20 | import argparse 21 | import socket 22 | 23 | # returns meta fields 24 | def get_meta(context): 25 | report_meta = "" 26 | feedback_pub = "" 27 | 28 | pp = 0 29 | rm = 0 30 | 31 | # get the root element 32 | event, root = context.next() 33 | for event, elem in context: 34 | if event == "end" and elem.tag == "report_metadata": 35 | # process record elements 36 | org_name = (elem.findtext("org_name", 'NULL')).translate(None, ',') 37 | email = (elem.findtext("email", 'NULL')).translate(None, ',') 38 | extra_contact_info = (elem.findtext("extra_contact_info", 'NULL')).translate(None, ',') 39 | report_id = (elem.findtext("report_id", 'NULL')).translate(None, ',') 40 | date_range_begin = (elem.findtext("date_range/begin", 'NULL')).translate(None, ',') 41 | date_range_end = (elem.findtext("date_range/end", 'NULL')).translate(None, ',') 42 | 43 | report_meta = "org_name=" + org_name + ", email=" + email + ", extra_contact_info=" + extra_contact_info \ 44 | + ", date_range_begin=" + date_range_begin + ", date_range_end=" + date_range_end 45 | rm = 1 46 | root.clear(); 47 | continue 48 | 49 | if event == "end" and elem.tag == "policy_published": 50 | domain = elem.findtext("domain", 'NULL') 51 | adkim = elem.findtext("adkim", 'NULL') 52 | aspf = elem.findtext("aspf", 'NULL') 53 | p = elem.findtext("p", 'NULL') 54 | pct = elem.findtext("pct", 'NULL') 55 | 56 | feedback_pub = "domain=" + domain + ", adkim=" + adkim + ", aspf=" + aspf + ", p=" + p + ", pct=" + pct 57 | pp = 1 58 | root.clear(); 59 | continue 60 | 61 | if pp == 1 and rm == 1: 62 | meta = report_meta + ", " + feedback_pub 63 | #print meta 64 | return meta 65 | 66 | return 67 | 68 | def print_record(context, meta, args): 69 | 70 | # get the root element 71 | event, root = context.next(); 72 | 73 | for event, elem in context: 74 | if event == "end" and elem.tag == "record": 75 | 76 | # process record elements 77 | # NOTE: This may require additional input validation 78 | source_ip = (elem.findtext("row/source_ip", 'NULL')).translate(None, ',') 79 | count = (elem.findtext("row/count", 'NULL')).translate(None, ',') 80 | disposition = (elem.findtext("row/policy_evaluated/disposition", 'NULL')).translate(None, ',') 81 | dkim = (elem.findtext("row/policy_evaluated/dkim", 'NULL')).translate(None, ',') 82 | spf = (elem.findtext("row/policy_evaluated/spf", 'NULL')).translate(None, ',') 83 | reason_type = (elem.findtext("row/policy_evaluated/reason/type", 'NULL')).translate(None, ',') 84 | comment = (elem.findtext("row/policy_evaluated/reason/comment", 'NULL')).translate(None, ',') 85 | envelope_to = (elem.findtext("identifiers/envelope_to", 'NULL')).translate(None, ',') 86 | header_from = (elem.findtext("identifiers/header_from", 'NULL')).translate(None, ',') 87 | dkim_domain = (elem.findtext("auth_results/dkim/domain", 'NULL')).translate(None, ',') 88 | dkim_result = (elem.findtext("auth_results/dkim/result", 'NULL')).translate(None, ',') 89 | dkim_hresult = (elem.findtext("auth_results/dkim/human_result", 'NULL')).translate(None, ',') 90 | spf_domain = (elem.findtext("auth_results/spf/domain", 'NULL')).translate(None, ',') 91 | spf_result = (elem.findtext("auth_results/spf/result", 'NULL')).translate(None, ',') 92 | 93 | # If you can identify internal IP 94 | x_host_name = "NULL" 95 | #try: 96 | # if IS_INTERNAL_IP(source_ip): 97 | # x_host_name = socket.getfqdn(source_ip) 98 | #except: 99 | # x_host_name = "NULL" 100 | 101 | print meta + ", source_ip=" + source_ip + ", count=" + count + ", disposition=" + disposition + ", dkim=" + dkim \ 102 | + ", spf=" + spf + ", reason_type=" + reason_type + ", comment=" + comment + ", envelope_to=" + envelope_to \ 103 | + ", header_from=" + header_from + ", dkim_domain=" + dkim_domain + ", dkim_result=" + dkim_result \ 104 | + ", dkim_hresult=" + dkim_hresult + ", spf_domain=" + spf_domain + ", spf_result=" + spf_result \ 105 | + ", x-host_name=" + x_host_name 106 | 107 | root.clear(); 108 | continue 109 | 110 | return; 111 | 112 | 113 | def main(): 114 | global args 115 | options = argparse.ArgumentParser(epilog="Example: \ 116 | %(prog)s dmarc-xml-file 1> outfile.log") 117 | options.add_argument("dmarcfile", help="dmarc file in XML format") 118 | args = options.parse_args() 119 | 120 | # get an iterable and turn it into an iterator 121 | meta_fields = get_meta(iter(etree.iterparse(args.dmarcfile, events=("start", "end")))); 122 | if not meta_fields: 123 | print >> sys.stderr, "Error: No valid 'policy_published' and 'report_metadata' xml tags found; File: " + args.dmarcfile 124 | sys.exit(1) 125 | 126 | print_record(iter(etree.iterparse(args.dmarcfile, events=("start", "end"))), meta_fields, args) 127 | 128 | if __name__ == "__main__": 129 | main() 130 | 131 | -------------------------------------------------------------------------------- /bin/imap-client.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | # 3 | # Copyright (c) 2014, Yahoo! Inc. 4 | # Copyrights licensed under the New BSD License. See the 5 | # accompanying LICENSE.txt file for terms. 6 | # 7 | # Author Binu P. Ramakrishnan 8 | # Created 09/12/2014 9 | # 10 | # An easy to use python script that 11 | # 1. Dumps emails from a given IMAP folder to a local folder. 12 | # 2. A option to dump mail attachments only (not contents) 13 | # 3. Support search criteria. Eg. all mail SINCE '10-Sep-2014' 14 | # 15 | 16 | import sys 17 | import os 18 | import socket 19 | import ssl 20 | import imaplib, email 21 | import argparse 22 | import getpass 23 | import datetime 24 | 25 | # Override IMAP4_SSL to validate server identity (CA cert validation) 26 | class IMAP4_SSL_Ex(imaplib.IMAP4_SSL): 27 | def __init__(self, host = '', port = imaplib.IMAP4_SSL_PORT, 28 | ca_certs = None, cert_reqs = ssl.CERT_REQUIRED, 29 | ssl_version = ssl.PROTOCOL_TLSv1): 30 | self.cert_reqs = cert_reqs 31 | self.ca_certs = ca_certs 32 | self.ssl_version = ssl_version 33 | imaplib.IMAP4_SSL.__init__(self, host, port, keyfile = None, certfile = None) 34 | 35 | def open(self, host = '', port = imaplib.IMAP4_SSL_PORT): 36 | self.host = host 37 | self.port = port 38 | self.sock = socket.create_connection((host, port)) 39 | self.sslobj = ssl.wrap_socket(self.sock, self.keyfile, 40 | self.certfile, 41 | cert_reqs=self.cert_reqs, 42 | ssl_version=self.ssl_version, 43 | ca_certs=self.ca_certs) 44 | self.file = self.sslobj.makefile('rb') 45 | 46 | # global object 47 | args = "" 48 | 49 | def vprint(msg): 50 | global args 51 | if args.quiet: return 52 | if args.verbose: print msg; 53 | 54 | def process_mailbox(mail): 55 | # dumps emails/attachments in the folder to output directory. 56 | global args 57 | count=0 58 | vprint(args.search) 59 | 60 | ret, data = mail.search(None, '(' + args.search + ')') 61 | if ret != 'OK': 62 | print >> sys.stderr, "ERROR: No messages found" 63 | return 1 64 | 65 | if not os.path.exists(args.outdir): 66 | os.makedirs(args.outdir) 67 | 68 | for num in data[0].split(): 69 | ret, data = mail.fetch(num, '(RFC822)') 70 | if ret != 'OK': 71 | print >> sys.stderr, "ERROR getting message from IMAP server", num 72 | return 1 73 | 74 | if not args.attachmentsonly: 75 | vprint ("Writing message "+ num) 76 | fp = open('%s/%s.eml' %(args.outdir, num), 'wb') 77 | fp.write(data[0][1]) 78 | fp.close() 79 | count = count + 1 80 | print args.outdir+"/"+num+".eml" 81 | 82 | else: 83 | m = email.message_from_string(data[0][1]) 84 | if m.get_content_maintype() == 'multipart' or \ 85 | m.get_content_type() == 'application/zip' or \ 86 | m.get_content_type() == 'application/gzip': 87 | for part in m.walk(): 88 | 89 | #find the attachment part 90 | if part.get_content_maintype() == 'multipart': continue 91 | if part.get('Content-Disposition') is None: continue 92 | 93 | #save the attachment in the given directory 94 | filename = part.get_filename() 95 | if not filename: continue 96 | filename = args.outdir+"/"+filename 97 | fp = open(filename, 'wb') 98 | fp.write(part.get_payload(decode=True)) 99 | fp.close() 100 | print filename 101 | count = count + 1 102 | 103 | if args.attachmentsonly: 104 | print "\nTotal attachments downloaded: ", count 105 | else: 106 | print "\nTotal mails downloaded: ", count 107 | 108 | def main(): 109 | global args 110 | options = argparse.ArgumentParser(epilog='Example: \ 111 | %(prog)s -s imap.example.com -c ./cacert.pem -u dmarc@example.com -f inbox -o ./mymail -S \"SINCE \\\"8-Sep-2014\\\"\" -P ./paswdfile') 112 | options.add_argument("-v", "--verbose", help="increase output verbosity", action="store_true") 113 | options.add_argument("--attachmentsonly", help="download attachments only", action="store_true") 114 | options.add_argument("--disablereadonly", help="enable state changes on server; Default readonly", action="store_true") 115 | options.add_argument("--quiet", help="supress all comments (stdout)", action="store_true") 116 | options.add_argument("-s", "--host", help="imap server; eg. imap.mail.yahoo.com", required=True) 117 | options.add_argument("-p", "--port", help="imap server port; Default is 993", default=993) 118 | options.add_argument("-c", "--cacerts", help="CA certificates, which are used to validate certificates passed from imap server", required=True) 119 | options.add_argument("-u", "--user", help="user's email id", required=True) 120 | options.add_argument("-f", "--folder", help="mail folder from which the mail to retrieve", required=True) 121 | options.add_argument("-o", "--outdir", help="directory to output", required=True) 122 | options.add_argument("-S", "--search", help="search criteria, defined in IMAP RFC 3501; eg. \"SINCE \\\"8-Sep-2014\\\"\"", default="ALL") 123 | options.add_argument("-P", "--pwdfile", help="A file that stores IMAP user password. If not set, the user is prompted to provide a passwd") 124 | args = options.parse_args() 125 | 126 | # redirect stdout to /dev/null 127 | if args.quiet: 128 | f = open(os.devnull, 'w') 129 | sys.stdout = f 130 | 131 | if args.pwdfile: 132 | infile = open(args.pwdfile, 'r') 133 | firstline = infile.readline().strip() 134 | args.pwd = firstline 135 | else: 136 | args.pwd = getpass.getpass() 137 | 138 | mail = IMAP4_SSL_Ex(args.host, args.port, args.cacerts) 139 | mail.login(args.user, args.pwd) 140 | ret, data = mail.select(args.folder, True) 141 | if ret == 'OK': 142 | vprint("Processing mailbox: " + args.folder) 143 | if process_mailbox(mail): 144 | mail.close() 145 | mail.logout() 146 | sys.exit(1) 147 | 148 | mail.close() 149 | else: 150 | print >> sys.stderr, "ERROR: Unable to open mailbox ", rv 151 | mail.logout() 152 | sys.exit(1) 153 | 154 | mail.logout() 155 | 156 | # entry point 157 | if __name__ == "__main__": 158 | main() 159 | 160 | --------------------------------------------------------------------------------