├── .gitignore ├── README.md ├── elf.py └── elf2.py /.gitignore: -------------------------------------------------------------------------------- 1 | .csv 2 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ## A Simple Python Script for Downloading EventLogFiles 2 | 3 | ### Pre-requisites 4 | 5 | Elf.py is built on the Python Standard Library. Your script will automatically import the following modules: 6 | 7 | * import urllib.request 8 | * import json 9 | * import getpass 10 | * import os 11 | * import sys 12 | * import gzip 13 | * import time 14 | * from io import BytesIO 15 | 16 | As a result, there shouldn't be a need to install any other modules. However, you should verify that you have python. Open a terminal (or cmd in Windows) and type: 17 | 18 | $ python --version 19 | 20 | You should see 21 | 22 | Python 3.x or 2.x. If you have 2.x use elf2.py or upgrade. 23 | 24 | If you do not, you may need to install it: https://www.python.org/downloads/ 25 | 26 | If you have multiple versions of python installed, you can declare which version you want on the command line: 27 | 28 | $ python --version 29 | 30 | For more information, read the help topic (https://docs.python.org/3/installing/#work-with-multiple-versions-of-python-installed-in-parallel). 31 | 32 | ### Download 33 | 34 | To use this script, just click the Download ZIP button in GitHub and store locally on your laptop or desktop. 35 | 36 | ### Run Script 37 | 38 | From the terminal, navigate to the directory where you downloaded elf.py. For instance 39 | 40 | $ cd ~/Users//Desktop 41 | 42 | To run the script, type: 43 | 44 | $ python elf.py (or python elf2.py for old python version) 45 | 46 | You will be prompted for several inputs: 47 | 1. Username 48 | 2. Password (hidden) 49 | 3. Date range (recommend: *Last_n_Days:2* instead of Yesterday) 50 | 4. Output directory 51 | 52 | It's possible to change the defaults for these four prompts within the code. I recommend this if you want to test or automate the script. 53 | 54 | The output will be an Event Log File CSV (Comma Separated Values) file for each day in the date range with the following file name convention: yyyy-mm-dd-eventtype.csv. For instance: 55 | 56 | 2019-10-08-Login.csv 57 | 58 | ### Troubleshooting 59 | 60 | I tested this script on Ubuntu Linux, Mac OSX, and Microsoft Windows platform. However, that doesn't mean you won't run into problems. 61 | 62 | One issue I did encounter was the use of a security token with your password if you are in a non-whitelisted organization or profile. Many administrators won't encounter this but if you try to log into the API without the token, you'll be blocked. 63 | 64 | To reset your token, go to My Settings | Personal | Reset My Security Token. 65 | 66 | If you don't see this menu item, then a security token is *not* required. 67 | 68 | When entering your password with a security token, the token follows your password: 69 | 70 | If your password = "mypassword" 71 | And your security token = "XXXXXXXXXX" 72 | You must enter "mypasswordXXXXXXXXXX" in place of your password 73 | 74 | You can also read more about this in the help topic (https://help.salesforce.com/htviewhelpdoc?id=user_security_token.htm&siteLang=en_US). 75 | 76 | You may also encounter a login error due to the ClientId and ClientSecret. If you do, make sure to change the constants in elf.py to reflect your own ClientId and ClientSecret. You can read more about how to create your own ClientId and ClientSecret in the help topic (https://help.salesforce.com/HTViewHelpDoc?id=connected_app_create.htm&language=en_US). 77 | 78 | ### Supported platforms 79 | 80 | Supported platforms: MacOS, Linux, and Windows. 81 | 82 | ## Getting help 83 | 84 | tweet to @atorman: 85 | 86 | * http://twitter.com/atorman 87 | 88 | or post a comment to Salesforcehacker.com 89 | 90 | * http://www.salesforcehacker.com 91 | -------------------------------------------------------------------------------- /elf.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | ''' 3 | # Python 3.x script to download EventLogFiles 4 | # Pre-requisite: standard library functionality = e.g urllib, json 5 | 6 | #/** 7 | #* Copyright (c) 2019, Salesforce.com, Inc. All rights reserved. 8 | #* 9 | #* Redistribution and use in source and binary forms, with or without 10 | #* modification, are permitted provided that the following conditions are 11 | #* met: 12 | #* 13 | #* * Redistributions of source code must retain the above copyright 14 | #* notice, this list of conditions and the following disclaimer. 15 | #* 16 | #* * Redistributions in binary form must reproduce the above copyright 17 | #* notice, this list of conditions and the following disclaimer in 18 | #* the documentation and/or other materials provided with the 19 | #* distribution. 20 | #* 21 | #* * Neither the name of Salesforce.com nor the names of its 22 | #* contributors may be used to endorse or promote products derived 23 | #* from this software without specific prior written permission. 24 | #* 25 | #* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 26 | #* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 27 | #* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 28 | #* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 29 | #* HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 30 | #* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 31 | #* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 32 | #* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 33 | #* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 34 | #* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 35 | #* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 36 | #*/ 37 | ''' 38 | ########################################################################################## 39 | # Connected App information: fill it in by creating a connected app 40 | # https://help.salesforce.com/articleView?id=connected_app_create.htm&language=en_US&type=0 41 | CLIENT_ID = 'FILL_ME_IN' 42 | CLIENT_SECRET = 'FILL_ME_IN' 43 | ########################################################################################## 44 | 45 | API_VERSION = 'v47.0'; 46 | 47 | #Imports 48 | import urllib.request 49 | import json 50 | import getpass 51 | import os 52 | import sys 53 | import gzip 54 | import time 55 | from io import BytesIO 56 | 57 | # login function 58 | def login(): 59 | ''' Login to salesforce service using OAuth2 ''' 60 | # prompt for username and password 61 | username = input('Username: \n') 62 | password = getpass.getpass('Password: \n') 63 | 64 | # check to see if anything was entered and if not, default values 65 | # change default values for username and password to your own 66 | if len(username) < 1: 67 | username = 'user@company.com' 68 | password = 'Passw0rd' 69 | print('Using default username: ' + username) 70 | else: 71 | print('Using user inputed username: ' + username) 72 | 73 | # create a new salesforce REST API OAuth request 74 | url = 'https://login.salesforce.com/services/oauth2/token' 75 | data = { 'grant_type': 'password', 'client_id': CLIENT_ID, 'client_secret': CLIENT_SECRET, 'username': username, 'password': password} 76 | headers = {'X-PrettyPrint' : '1'} 77 | 78 | # These lines are for when you have a proxy server 79 | # uncomment the next line to work with a local proxy server. replace the URL with the URL of your proxy 80 | # proxy = urllib.request.ProxyHandler({'https': 'http://127.0.0.1:8888/'}) 81 | # uncomment the next two lines if your proxy needs authentication. replace 'realm' through 'password' with appropriate values. 'realm' is often null 82 | # proxy_auth_handler = urllib.request.HTTPBasicAuthHandler() 83 | # proxy_auth_handler.add_password('realm', 'host', 'username', 'password') 84 | # pick one of the next two lines based on whether you need proxy authentication. The first is authenticated, the second is unauthenticated 85 | # opener = urllib.request.build_opener(proxy, proxy_auth_handler) 86 | # opener = urllib.request.build_opener(proxy) 87 | # uncomment the final line to enable the proxy for any calls 88 | # urllib.request.install_opener(opener) 89 | 90 | # call salesforce REST API and pass in OAuth credentials 91 | encoded_data = urllib.parse.urlencode(data).encode("utf-8") 92 | req = urllib.request.Request(url, encoded_data, headers) 93 | res = urllib.request.urlopen(req) 94 | 95 | # load results to dictionary 96 | res_dict = json.load(res) 97 | 98 | # close connection 99 | res.close() 100 | 101 | # return OAuth access token necessary for additional REST API calls 102 | access_token = res_dict['access_token'] 103 | instance_url = res_dict['instance_url'] 104 | 105 | return access_token, instance_url 106 | 107 | # download function 108 | def download_elf(): 109 | ''' Query salesforce service using REST API ''' 110 | # login and retrieve access_token and day 111 | access_token, instance_url = login() 112 | 113 | day = input('\nDate range (e.g. Last_n_Days:2, Today, Tomorrow):\n') 114 | 115 | # check to see if anything was entered and if not, default values 116 | if len(day) < 1: 117 | day = 'Last_n_Days:2' 118 | print('Using default date range: ' + day + '\n') 119 | else: 120 | print('Using user inputed date range: ' + day + '\n') 121 | 122 | # query Ids from Event Log File 123 | url = instance_url+'/services/data/' + API_VERSION + '/query?q=SELECT+Id+,+EventType+,+Logdate+From+EventLogFile+Where+LogDate+=+'+day 124 | headers = {'Authorization' : 'Bearer ' + access_token, 'X-PrettyPrint' : '1'} 125 | req = urllib.request.Request(url, None, headers) 126 | res = urllib.request.urlopen(req) 127 | res_dict = json.load(res) 128 | 129 | # capture record result size to loop over 130 | total_size = res_dict['totalSize'] 131 | 132 | # provide feedback if no records are returned 133 | if total_size < 1: 134 | print('No records were returned for ' + day) 135 | sys.exit() 136 | 137 | # create a directory for the output 138 | dir = input("Output directory: ") 139 | 140 | # check to see if anything 141 | if len(dir) < 1: 142 | dir = 'elf' 143 | print('\ndefault directory name used: ' + dir) 144 | else: 145 | print('\ndirectory name used: ' + dir) 146 | 147 | # If directory doesn't exist, create one 148 | if not os.path.exists(dir): 149 | os.makedirs(dir) 150 | 151 | # close connection 152 | res.close 153 | 154 | # check to see if the user wants to download it compressed 155 | compress = input('\nUse compression (y/n)\n').lower() 156 | print(compress) 157 | 158 | # check to see if anything 159 | if len(compress) < 1: 160 | compress = 'yes' 161 | print('\ndefault compression being used: ' + compress) 162 | else: 163 | print('\ncompression being used: ' + compress) 164 | 165 | # loop over json elements in result and download each file locally 166 | for i in range(total_size): 167 | # pull attributes out of JSON for file naming 168 | ids = res_dict['records'][i]['Id'] 169 | types = res_dict['records'][i]['EventType'] 170 | dates = res_dict['records'][i]['LogDate'] 171 | 172 | # create REST API request 173 | url = instance_url+'/services/data/' + API_VERSION + '/sobjects/EventLogFile/'+ids+'/LogFile' 174 | 175 | # provide correct compression header 176 | if (compress == 'y') or (compress == 'yes'): 177 | headers = {'Authorization' : 'Bearer ' + access_token, 'X-PrettyPrint' : '1', 'Accept-encoding' : 'gzip'} 178 | print('Using gzip compression\n') 179 | else: 180 | headers = {'Authorization' : 'Bearer ' + access_token, 'X-PrettyPrint' : '1'} 181 | print('Not using gzip compression\n') 182 | 183 | # begin profiling 184 | start = time.time() 185 | 186 | # open connection 187 | req = urllib.request.Request(url, None, headers) 188 | res = urllib.request.urlopen(req) 189 | 190 | print('********************************') 191 | 192 | # provide feedback to user 193 | print('Downloading: ' + dates[:10] + '-' + types + '.csv to ' + os.getcwd() + '/' + dir + '\n') 194 | 195 | # print the response to see the content type 196 | # print(res.info()) 197 | 198 | # if the response is gzip-encoded as expected 199 | # compression code from http://bit.ly/pyCompression 200 | if res.info().get('Content-Encoding') == 'gzip': 201 | # buffer results 202 | buf = BytesIO(res.read()) 203 | # gzip decode the response 204 | f = gzip.GzipFile(fileobj=buf) 205 | # print data 206 | data = f.read() 207 | # close buffer 208 | buf.close() 209 | else: 210 | # buffer results 211 | buf = BytesIO(res.read()) 212 | # get the value from the buffer 213 | data = buf.getvalue() 214 | #print data 215 | buf.close() 216 | 217 | # write buffer to CSV with following naming convention yyyy-mm-dd-eventtype.csv 218 | file = open(dir + '/' +dates[:10]+'-'+types+'.csv', 'wb') 219 | file.write(data) 220 | 221 | # end profiling 222 | end = time.time() 223 | secs = end - start 224 | 225 | #msecs = secs * 1000 # millisecs 226 | #print 'elapsed time: %f ms' % msecs 227 | print('Total download time: %f seconds\n' % secs) 228 | 229 | file.close 230 | i = i + 1 231 | 232 | # close connection 233 | res.close 234 | 235 | download_elf() 236 | -------------------------------------------------------------------------------- /elf2.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | ''' 3 | # Python 2.7.9 script to download EventLogFiles 4 | # Pre-requisite: standard library functionality = e.g urrlib2, json, StringIO 5 | 6 | #/** 7 | #* Copyright (c) 2012, Salesforce.com, Inc. All rights reserved. 8 | #* 9 | #* Redistribution and use in source and binary forms, with or without 10 | #* modification, are permitted provided that the following conditions are 11 | #* met: 12 | #* 13 | #* * Redistributions of source code must retain the above copyright 14 | #* notice, this list of conditions and the following disclaimer. 15 | #* 16 | #* * Redistributions in binary form must reproduce the above copyright 17 | #* notice, this list of conditions and the following disclaimer in 18 | #* the documentation and/or other materials provided with the 19 | #* distribution. 20 | #* 21 | #* * Neither the name of Salesforce.com nor the names of its 22 | #* contributors may be used to endorse or promote products derived 23 | #* from this software without specific prior written permission. 24 | #* 25 | #* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 26 | #* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 27 | #* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 28 | #* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 29 | #* HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 30 | #* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 31 | #* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 32 | #* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 33 | #* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 34 | #* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 35 | #* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 36 | #*/ 37 | ''' 38 | ########################################################################################## 39 | # Connected App information: fill it in by creating a connected app 40 | # https://help.salesforce.com/articleView?id=connected_app_create.htm&language=en_US&type=0 41 | CLIENT_ID = 'FILL_ME_IN' 42 | CLIENT_SECRET = 'FILL_ME_IN' 43 | ########################################################################################## 44 | 45 | API_VERSION = 'v47.0'; 46 | 47 | #Imports 48 | import urllib2 49 | import json 50 | #import ssl 51 | import getpass 52 | import os 53 | import sys 54 | import gzip 55 | import time 56 | from StringIO import StringIO 57 | import base64 58 | 59 | # login function 60 | def login(): 61 | ''' Login to salesforce service using OAuth2 ''' 62 | # prompt for username and password 63 | username = raw_input('Username: \n') 64 | password = getpass.getpass('Password: \n') 65 | 66 | # check to see if anything was entered and if not, default values 67 | # change default values for username and password to your own 68 | if len(username) < 1: 69 | username = 'user@company.com' 70 | password = 'Passw0rd' 71 | print 'Using default username: ' + username 72 | else: 73 | print 'Using user inputed username: ' + username 74 | 75 | print 'check point' 76 | # create a new salesforce REST API OAuth request 77 | url = 'https://login.salesforce.com/services/oauth2/token' 78 | data = '&grant_type=password&client_id='+CLIENT_ID+'&client_secret='+CLIENT_SECRET+'&username='+username+'&password='+password 79 | headers = {'X-PrettyPrint' : '1'} 80 | 81 | # workaround to ssl issue introduced before version 2.7.9 82 | #if hasattr(ssl, '_create_unverified_context'): 83 | #ssl._create_default_https_context = ssl._create_unverified_context 84 | 85 | # These lines are for when you have a proxy server 86 | # uncomment the next line to work with a local proxy server. replace the URL with the URL of your proxy 87 | # proxy = urllib2.ProxyHandler({'https': 'http://127.0.0.1:8888/'}) 88 | # uncomment the next two lines if your proxy needs authentication. replace 'realm' through 'password' with appropriate values. 'realm' is often null 89 | # proxy_auth_handler = urllib2.HTTPBasicAuthHandler() 90 | # proxy_auth_handler.add_password('realm', 'host', 'username', 'password') 91 | # pick one of the next two lines based on whether you need proxy authentication. The first is authenticated, the second is unauthenticated 92 | # opener = urllib2.build_opener(proxy, proxy_auth_handler) 93 | # opener = urllib2.build_opener(proxy) 94 | # uncomment the final line to enable the proxy for any calls 95 | # urllib2.install_opener(opener) 96 | 97 | # call salesforce REST API and pass in OAuth credentials 98 | req = urllib2.Request(url, data, headers) 99 | res = urllib2.urlopen(req) 100 | 101 | # load results to dictionary 102 | res_dict = json.load(res) 103 | 104 | # close connection 105 | res.close() 106 | 107 | # return OAuth access token necessary for additional REST API calls 108 | access_token = res_dict['access_token'] 109 | instance_url = res_dict['instance_url'] 110 | 111 | return access_token, instance_url 112 | 113 | # download function 114 | def download_elf(): 115 | ''' Query salesforce service using REST API ''' 116 | # login and retrieve access_token and day 117 | access_token, instance_url = login() 118 | 119 | day = raw_input('\nDate range (e.g. Last_n_Days:2, Today, Tomorrow):\n') 120 | 121 | # check to see if anything was entered and if not, default values 122 | if len(day) < 1: 123 | day = 'Last_n_Days:2' 124 | print 'Using default date range: ' + day + '\n' 125 | else: 126 | print 'Using user inputed date range: ' + day + '\n' 127 | 128 | # query Ids from Event Log File 129 | url = instance_url+'/services/data/' + API_VERSION + '/query?q=SELECT+Id+,+EventType+,+Logdate+From+EventLogFile+Where+LogDate+=+'+day 130 | headers = {'Authorization' : 'Bearer ' + access_token, 'X-PrettyPrint' : '1'} 131 | req = urllib2.Request(url, None, headers) 132 | res = urllib2.urlopen(req) 133 | res_dict = json.load(res) 134 | 135 | # capture record result size to loop over 136 | total_size = res_dict['totalSize'] 137 | 138 | # provide feedback if no records are returned 139 | if total_size < 1: 140 | print 'No records were returned for ' + day 141 | sys.exit() 142 | 143 | # create a directory for the output 144 | dir = raw_input("Output directory: ") 145 | 146 | # check to see if anything 147 | if len(dir) < 1: 148 | dir = 'elf' 149 | print '\ndefault directory name used: ' + dir 150 | else: 151 | print '\ndirectory name used: ' + dir 152 | 153 | # If directory doesn't exist, create one 154 | if not os.path.exists(dir): 155 | os.makedirs(dir) 156 | 157 | # close connection 158 | res.close 159 | 160 | # check to see if the user wants to download it compressed 161 | compress = raw_input('\nUse compression (y/n)\n').lower() 162 | print compress 163 | 164 | # check to see if anything 165 | if len(compress) < 1: 166 | compress = 'yes' 167 | print '\ndefault compression being used: ' + compress 168 | else: 169 | print '\ncompression being used: ' + compress 170 | 171 | # loop over json elements in result and download each file locally 172 | for i in range(total_size): 173 | # pull attributes out of JSON for file naming 174 | ids = res_dict['records'][i]['Id'] 175 | types = res_dict['records'][i]['EventType'] 176 | dates = res_dict['records'][i]['LogDate'] 177 | 178 | # create REST API request 179 | url = instance_url+'/services/data/' + API_VERSION + '/sobjects/EventLogFile/'+ids+'/LogFile' 180 | 181 | # provide correct compression header 182 | if (compress == 'y') or (compress == 'yes'): 183 | headers = {'Authorization' : 'Bearer ' + access_token, 'X-PrettyPrint' : '1', 'Accept-encoding' : 'gzip'} 184 | print 'Using gzip compression\n' 185 | else: 186 | headers = {'Authorization' : 'Bearer ' + access_token, 'X-PrettyPrint' : '1'} 187 | print 'Not using gzip compression\n' 188 | 189 | # begin profiling 190 | start = time.time() 191 | 192 | # open connection 193 | req = urllib2.Request(url, None, headers) 194 | res = urllib2.urlopen(req) 195 | 196 | print '********************************' 197 | 198 | # provide feedback to user 199 | print 'Downloading: ' + dates[:10] + '-' + types + '.csv to ' + os.getcwd() + '/' + dir + '\n' 200 | 201 | # print the response to see the content type 202 | # print res.info() 203 | 204 | # if the response is gzip-encoded as expected 205 | # compression code from http://bit.ly/pyCompression 206 | if res.info().get('Content-Encoding') == 'gzip': 207 | # buffer results 208 | buf = StringIO(res.read()) 209 | # gzip decode the response 210 | f = gzip.GzipFile(fileobj=buf) 211 | # print data 212 | data = f.read() 213 | # close buffer 214 | buf.close() 215 | else: 216 | # buffer results 217 | buf = StringIO(res.read()) 218 | # get the value from the buffer 219 | data = buf.getvalue() 220 | #print data 221 | buf.close() 222 | 223 | # write buffer to CSV with following naming convention yyyy-mm-dd-eventtype.csv 224 | file = open(dir + '/' +dates[:10]+'-'+types+'.csv', 'w') 225 | file.write(data) 226 | 227 | # end profiling 228 | end = time.time() 229 | secs = end - start 230 | 231 | #msecs = secs * 1000 # millisecs 232 | #print 'elapsed time: %f ms' % msecs 233 | print 'Total download time: %f seconds\n' % secs 234 | 235 | file.close 236 | i = i + 1 237 | 238 | # close connection 239 | res.close 240 | 241 | download_elf() --------------------------------------------------------------------------------