├── README.md ├── ese2csv.exe ├── ese2csv.py ├── spartan_plugin.py └── srudb_plugin.py /README.md: -------------------------------------------------------------------------------- 1 | # ese-analyst 2 | 3 | The tool ese2csv is intended to make analyzing and dumping any ese file simple. Its options are 4 | 5 | ``` 6 | C:\>ese2csv.exe --help 7 | usage: ese2csv.exe [-h] [--make-plugin] [--pluggin CONFIG] [--outpath OUTPATH] 8 | [--acquire-live] [--verbose] [--list-tables] [--recurse] 9 | [--dump-tables [DUMPTABLES [DUMPTABLES ...]]] 10 | [--plugin-args [PLUGINARGS [PLUGINARGS ...]]] 11 | ese_file 12 | 13 | Find and dump ESE databases. 14 | 15 | positional arguments: 16 | ese_file This required file name is an ese database 17 | 18 | optional arguments: 19 | -h, --help show this help message and exit 20 | --make-plugin, -m Produce an editable shell of a plugin for the 21 | specified ese. 22 | --pluggin CONFIG, -p CONFIG 23 | Use a plugin that defines fields in the ese database. 24 | --outpath OUTPATH, -o OUTPATH 25 | The directory to which the CSV(s) will be written. 26 | --acquire-live, -a Use FGET to extract locked file for processing. 27 | --verbose, -v Generate Verbose output for debugging. 28 | --list-tables, -l List all tables in the ese. 29 | --recurse, -r Recurse subdirectories to find ese in path. 30 | --dump-tables [DUMPTABLES [DUMPTABLES ...]], -d [DUMPTABLES [DUMPTABLES ...]] 31 | Only dump tables listed here separated by spaces. End 32 | this list with double dash if this is not followed by 33 | additional optional arguments. --. 34 | --plugin-args [PLUGINARGS [PLUGINARGS ...]] 35 | Arguments passed to plugin. End arguments with double 36 | dash if this is not followed by additional optional 37 | arguments. --. 38 | 39 | ``` 40 | 41 | The idea of ese2csv is to allow you to dump the data from any ESE database that the libesedb engine can read. However, you can also create a "plugin" for the ese file with the -m option that allows you to use YAML to define the formats for fields, gives them friendly names and provides functions for processing the database. The tool is distributed with two plugin. The srudb_plugin.py can be used to dump the srum database. spartan_plugin.py is a beta of dumping Edge spartan files that took me 30 minutes to put together with the -m option! The tool supports wildcards and directory recursion so you can search your drive and let the tool extract what ever it can find. The tool assumes the plugins are in the same directory as the executable. 42 | 43 | Two slightly confusing argument are -d and -p. Both of those can be followed by multiple parameters. For example, -d can be followed by multiple table names separated by spaces. ese2csv will dump each of those table names. -p can be followed by multiple plugin arguments. All of them will be passed on to the plugin for processing. So how does the ese2csv.exe command line know when your done with your list of table names or plugin arguments? With Python programs you end these list by providing another optional argument. If you use -p or -d as the last argument before the name of your ese file then you end the list with double dash (--). 44 | 45 | ## Dump all the tables in the the specified srudb.dat file to csv_files (default). Acquire a copy of the locked srudb.dat file before use (-a). Ignored tables specified by the YAML and use Table, Field names specified in in the srudb_plugin.py (-p srudb_plugin). Also tell the plugin to live acquire the software registry hive (--plugin-args LIVE) . Output all of the CSV files to the root of the file system (-o c:\) 46 | 47 | ``` 48 | C:>ese2csv.exe -p srudb_plugin -o c:\ -a --plugin-args LIVE -- C:\Windows\System32\sru\srudb.dat 49 | ``` 50 | 51 | 52 | ## Same as above but provide the name of the SOFTWARE registry hive to the plugin and write to the current directory. 53 | 54 | ``` 55 | C:>ese2csv.exe -p srudb_plugin -a --plugin-args C:\SOFTWARE -- C:\Windows\System32\sru\srudb.dat 56 | ``` 57 | 58 | 59 | ## List the tables (-l) in an ese database. File must not be locked by the OS. If it is use -a (acquire). 60 | 61 | ``` 62 | C:>ese2csv.exe -l C:\Windows\SoftwareDistribution\DataStore\DataStore.edb 63 | C:\Windows\SoftwareDistribution\DataStore\DataStore.edb True 64 | Table MSysObjects has 479 records 65 | Table MSysObjectsShadow has 479 records 66 | Table MSysObjids has 59 records 67 | Table MSysLocales has 8 records 68 | Table tbAUState has 0 records 69 | Table tbCcrDownloadData has 0 records 70 | Table tbComputerInfo has 0 records 71 | Table tbDownloadJob has 0 records 72 | Table tbEula has 0 records 73 | Table tbFiles has 704 records 74 | Table tbHistory has 71 records 75 | Table tbServerConfig has 2 records 76 | Table tbServerCookies has 27 records 77 | Table tbServiceData has 3 records 78 | Table tbScanTransInfo has 2 records 79 | Table tbStoreVersion has 1 records 80 | Table tbUpdateLocalizedProps has 950 records 81 | Table tbUpdates has 2707 records 82 | Table tbHiddenUpdates has 0 records 83 | Table tbLocalUserIds has 1 records 84 | Table tbTimers has 1 records 85 | Table tbSLSData has 4 records 86 | Table tbPerSrvUpdate9116a23d9de3a64d8a4bb43877bcb1b7 has 0 records 87 | Table tbPerSrvUserUpdateData9116a23d9de3a64d8a4bb43877bcb1b7 has 0 records 88 | Table tbPerSrvUpdateb4f4829443e3b643b1709a65bc822c77 has 1964 records 89 | Table tbPerSrvUserUpdateDatab4f4829443e3b643b1709a65bc822c77 has 0 records 90 | Table tbPerSrvUpdate7c8a5e85b4eca34cb0451dfa50104289 has 713 records 91 | Table tbPerSrvUserUpdateData7c8a5e85b4eca34cb0451dfa50104289 has 0 records 92 | ``` 93 | 94 | ## Find all *.edb files in c:\ and list their table contents. Recurse (-r) all subdirectories and acquire live files (-a) with FGET before you list tables (-l). 95 | 96 | ``` 97 | C:\> ese2csv.exe -ral "C:\*.edb" 98 | TABLE LISTING FOR ESE FILE C:\ProgramData\Microsoft\Search\Data\Applications\Windows\Windows.edb 99 | 100 | Table MSysObjects has 858 records 101 | Table MSysObjectsShadow has 858 records 102 | Table MSysObjids has 55 records 103 | Table MSysLocales has 8 records 104 | Table CatalogManager_Properties has 0 records 105 | Table CatalogStorageManager has 1 records 106 | Table SystemIndex_Gthr has 3170 records 107 | Table SystemIndex_GthrPth has 292 records 108 | Table SystemIndex_1_Properties has 225 records 109 | Table SystemIndex_1 has 18 records 110 | 111 | ``` 112 | 113 | ## Recursivly (-r) go through every *.edb file in c:\. Acquire (-a) a copy of the locked file. Find the table named tbTimers in one of them and dump (-d) it. 114 | 115 | ``` 116 | C:\Users\Win10Lab\Desktop>ese2csv.exe -rad tbTimers -- "C:\*.edb" 117 | Processing tbTimers 118 | 119 | C:\Users\Win10Lab\Desktop>type tbTimers.csv 120 | TimerId,ExpirationTime,IdleOnly,NetworkOnly 121 | b'e763a82909861e4db7cd5668f857f1db',b'31354874915ed501',0,0 122 | ``` 123 | 124 | ## Dump the dbTimers table without a plugin specifying the full path. Since no -a is provided the file must not be locked by the OS. 125 | 126 | ``` 127 | C:>ese2csv.exe -d tbTimers -- C:\Windows\SoftwareDistribution\DataStore\DataStore.edb 128 | C:\Windows\SoftwareDistribution\DataStore\DataStore.edb True 129 | Processing tbTimers 130 | 131 | C:>type tbTimers.csv 132 | TimerId,ExpirationTime,IdleOnly,NetworkOnly 133 | b'e763a82909861e4db7cd5668f857f1db',b'eaaadd49c85dd501',0,0 134 | ``` 135 | 136 | 137 | ## Make (-m) a srum plugin template (which requires editing) 138 | 139 | ``` 140 | C:>ese2csv.exe -ma c:\windows\system32\sru\srudb.dat > srudb_plugin.py 141 | ``` 142 | 143 | ## After editing the YAML and functions in the new srum plugin use it (-p) to list new friendly table names 144 | 145 | ``` 146 | C:>ese2csv.exe -p srudb_plugin -l srudb.dat 147 | srudb.dat True 148 | Table MSysObjects aka MSysObjects has 262 records 149 | Table MSysObjectsShadow aka MSysObjectsShadow has 262 records 150 | Table MSysObjids aka MSysObjids has 41 records 151 | Table MSysLocales aka MSysLocales has 7 records 152 | Table SruDbIdMapTable aka SruDbIdMapTable has 33946 records 153 | Table SruDbCheckpointTable aka SruDbCheckpointTable has 0 records 154 | Table {D10CA2FE-6FCF-4F6D-848E-B2E99266FA89} aka Application Resources has 122856 records 155 | Table {DD6636C4-8929-4683-974E-22C046A43763} aka Network Connections has 2599 records 156 | Table {FEE4E14F-02A9-4550-B5CE-5FA2DA202E37} aka Energy Usage has 1736 records 157 | Table {973F5D5C-1D90-4944-BE8E-24B94231A174} aka Network Usage has 14445 records 158 | Table {FEE4E14F-02A9-4550-B5CE-5FA2DA202E37}LT aka Energy Usage (Long-Term) has 126 records 159 | Table {D10CA2FE-6FCF-4F6D-848E-B2E99266FA86} aka Application Resource Usage has 3719 records 160 | Table {DA73FB89-2BEA-4DDC-86B8-6E048C6DA477} aka Unknown4 has 13778 records 161 | Table {5C8CF1C7-7257-4F13-B223-970EF5939312} aka Unknown1 has 16982 records 162 | Table {7ACBBAA3-D029-4BE4-9A7A-0885927F1D8F} aka Unknown2 has 1544 records 163 | Table {B6D82AF1-F780-4E17-8077-6CB9AD8A6FC4} aka Unknown3 has 98 records 164 | ``` 165 | 166 | ## And dump a table based on its friendly name 167 | 168 | ``` 169 | C:>ese2csv.exe -p srudb_config -d "Network Usage" -- srudb.dat 170 | srudb.dat True 171 | Processing Network Usage 172 | ``` 173 | 174 | 175 | ## Creating a plugin. 176 | 177 | 178 | I know this needs to be flushed out more but here is some basic documentation. 179 | 180 | Start out your plugins with the -m option and redirecting it to a file. Then edit the plugin and customize the output. This example creates a plugin for the srudb.dat file. 181 | 182 | ``` 183 | C:>ese2csv.exe -ma c:\windows\system32\sru\srudb.dat > srudb_plugin.py 184 | ``` 185 | 186 | You can place four "call back" functions in your plugin that are executed automatically by ese2csv. 187 | 188 | - **plugin_init(ese_database)** - This function will receive one argument. The ese_database. It is called when the program first loads and can be used to setup data structures that other functions depend upon. 189 | - **plugin_modify_header(list_of_headers, table_name)** - This function is called after the headers are read from the ESE database before they are written to the CSV. This is your chance to add or change the headers. 190 | - **plugin_modify_row(list_of_row_values, table_name)** - This function is called for every row in the ESE before itis written to the CSV. This is your chance to add to or modify rows. This is also where you can accumulate values or do additional processing or row data. 191 | - **plugin_end_of_file(csv_writer_object, table_name)** - This function is called before the csv file is closed. Write your accumulated values or clean up data structures. 192 | 193 | There are also built in functions that are available inside the plugin for you to call 194 | 195 | - **lookup("yaml table name", value)** - Will lookup the value in the the specified YAML lookup table (defined in the yaml or by 196 | table_reference entries). 197 | - **extract_live_file(live_path)** - Takes in a path to a file locked by 198 | the OS and returns a path to an unlocked copy that was extracted with FGET.EXE 199 | - **smart_retrieve(ese_db, rownum, colnum)** - Retrieves the specified row and column from the ese database 200 | - **blob_to_string(bytes)** - Attempts to convert the bytes into a 201 | string 202 | - **ole_timestamp(bytes)** - Takes in bytes and interprets it as an OLE timestamp 203 | - **file_timestamp(bytes)** - Takes in bytes and interprets it as a File Timestamp 204 | - **args** - The variable args is a list containing the arguments passwd to --plugin-args on the CLI 205 | 206 | You can also create your own functions. These functions are called by setting the format fields in the YAML. See srudb_plugin for example. 207 | 208 | YAML format field can be in the form of: 209 | 210 | - **None** - Do Nothing to data. Put in CSV as is. 211 | 212 | - **function:function_name** - Call function_name and pass it the data. Put what is returned in the CSV 213 | - **lookup:YAML_LOOKUP_TABLE** - Call lookup("YAML_LOOKUP_TABLE", current_value) and put what is returned in the CSV. 214 | -------------------------------------------------------------------------------- /ese2csv.exe: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/MarkBaggett/ese-analyst/d9111cde1792d197843cf33ab03d3c4833bb9a54/ese2csv.exe -------------------------------------------------------------------------------- /ese2csv.py: -------------------------------------------------------------------------------- 1 | import pyesedb 2 | import yaml 3 | import csv 4 | import codecs 5 | import struct 6 | import uuid 7 | import argparse 8 | import pathlib 9 | import importlib 10 | import re 11 | import sys 12 | import os 13 | import tempfile 14 | import urllib.request 15 | import subprocess 16 | from Registry.Registry import Registry 17 | from datetime import datetime,timedelta 18 | 19 | def blob_to_string(chrblob): 20 | """Takes in a binary blob hex characters and does its best to convert it to a readable string. 21 | Works great for UTF-16 LE, UTF-16 BE, ASCII like data. Otherwise return it as hex. 22 | """ 23 | try: 24 | chrblob = codecs.decode(chrblob,"hex") 25 | except: 26 | pass 27 | try: 28 | if re.match(b'^(?:[^\x00]\x00)+\x00\x00$', chrblob): 29 | binblob = chrblob.decode("utf-16-le").strip("\x00") 30 | elif re.match(b'^(?:\x00[^\x00])+\x00\x00$', chrblob): 31 | binblob = chrblob.decode("utf-16-be").strip("\x00") 32 | else: 33 | binblob = chrblob.decode("latin1").strip("\x00") 34 | except Exception as e: 35 | binblob = "" if not chrblob else codecs.decode(chrblob,"latin-1") 36 | return binblob 37 | 38 | def ole_timestamp(binblob): 39 | """converts a hex encoded OLE time stamp to a time string""" 40 | try: 41 | td,ts = str(struct.unpack("= 8: 180 | subauthority_count = sid[1] 181 | identifier_authority = struct.unpack(">H", sid[2:4])[0] 182 | identifier_authority <<= 32 183 | identifier_authority |= struct.unpack(">L", sid[4:8])[0] 184 | str_sid_components.append(identifier_authority) 185 | start = 8 186 | for i in range(subauthority_count): 187 | authority = sid[start:start + 4] 188 | if not authority: 189 | break 190 | if len(authority) < 4: 191 | raise ValueError("In binary SID '%s', component %d has been truncated. " 192 | "Expected 4 bytes, found %d: (%s)", 193 | ",".join([str(ord(c)) for c in sid]), i, 194 | len(authority), authority) 195 | str_sid_components.append(struct.unpack("= 8: 1216 | subauthority_count = sid[1] 1217 | identifier_authority = struct.unpack(">H", sid[2:4])[0] 1218 | identifier_authority <<= 32 1219 | identifier_authority |= struct.unpack(">L", sid[4:8])[0] 1220 | str_sid_components.append(identifier_authority) 1221 | start = 8 1222 | for i in range(subauthority_count): 1223 | authority = sid[start:start + 4] 1224 | if not authority: 1225 | break 1226 | if len(authority) < 4: 1227 | raise ValueError("In binary SID '%s', component %d has been truncated. " 1228 | "Expected 4 bytes, found %d: (%s)", 1229 | ",".join([str(ord(c)) for c in sid]), i, 1230 | len(authority), authority) 1231 | str_sid_components.append(struct.unpack("H6B", codecs.decode(format(luidval,'016x'),'hex'))[0] 1253 | return lookup("LUID_interface_types",inttype) 1254 | --------------------------------------------------------------------------------