├── .gitignore ├── LICENSE ├── README.md ├── binarly_query.py └── images ├── classification.jpg ├── hunt.jpg ├── search.jpg ├── search_exact.jpg └── signer_smartinstaller_detection.jpg /.gitignore: -------------------------------------------------------------------------------- 1 | apikey.txt 2 | .vscode/ -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2016 binarlyhq 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | #Binarly Query 2 | Command-line script to interact with the Binarly API. 3 | #Dependencies 4 | 1. [Binarly-SDK](https://github.com/binarlyhq/binarly-sdk) 5 | 2. Colorama `pip install colorama` 6 | 7 | #Configuration 8 | To use this script you need an API key from [Binarly](https://binar.ly). Just register for free, activate your account and you will receive an API key in your inbox. 9 | 10 | The API Key can either be passed to the script using `--key` option or can be saved in a file named `apikey.txt` in the same directory. 11 | 12 | #Usage 13 | __Binarly__ is binary search engine with __extremely fast query times__ (measured in milliseconds) for arbitrary sequences of bytes over millions of clean and malicious binaries. 14 | 15 | In order to take full advantage of Binarly, a few concepts have to be explained first. 16 | 17 | Our implementation is based on indexing files with a [4-gram](https://en.wikipedia.org/wiki/N-gram) granularity. When searching for a pattern such as `01 02 03 04 05 06`, the query is split into 4-grams that are looked up independently: `01 02 03 04`, `02 03 04 05` and `03 04 05 06`. Since the offsets of the 4-grams are not stored in the index, Binarly can potentially return files that contain all the 4-grams from the search pattern but not the pattern itself. We call such results "false positives". 18 | 19 | Users can specify the `--exact` flag to enable actual validation of search results and filter out "false positives". 20 | 21 | The `binarly_query` script provides an interface to the following services: 22 | 23 | * [Search](#Search) 24 | * [IOC Generation](#ioc-generation) 25 | * [Sample Hunting With YARA Signatures](#sample-hunting-with-yara-signatures) 26 | * [File Classification](#file-classification) 27 | 28 | # Search 29 | The most basic operation that we expose is searching for binary and string patterns using the following `modifiers`: 30 | * Arbitrary hex patterns i.e: `"00 01 02 03 04"` or `"01020304"` 31 | * ASCII strings i.e `-a "Microsoft"` 32 | * WIDE strings i.e `-w "Microsoft"` 33 | 34 | __Search options:__ 35 | * `--exact` Filter out false positives. When this option is used, the statistics returned apply only to the current result limit. 36 | * `--limit=N` Limit search results to the first N. By default, this value is 20. Specifying a value of 0 means that only statistics (the number of files matching this pattern) will be returned. If `--exact` is not specified the statistics returned apply to all results, not only the ones displayed. 37 | 38 | Below is an example of searching for files that contain a combination of patterns, with default limit 20: 39 | * Hex sequence `"01 02 FF 03 04 FF"` 40 | * ASCII string `"Microsoft"` 41 | * Unicode string `"Kernel32"` 42 | 43 | __Command:__ 44 | 45 | __`binarly_query.py search "0102FF0304FF" -a "Microsoft" -w "Kernel32"`__ 46 | 47 | __Output:__ 48 | 49 | ![Basic Search](./images/search.jpg) 50 | 51 | Below is an example of an __exact__ match with the same default limit of 20: 52 | 53 | __Command:__ 54 | 55 | __`binarly_query.py search "eb 04 03 c0 85 d2 52 eb 05 8d 1a 8d 4a fb 52 8b" --exact`__ 56 | 57 | __Output:__ 58 | 59 | ![Exact Search](./images/search_exact.jpg) 60 | 61 | __Best practice__: try to use normal searches to reduce the number of matching files and, optionally,at the end use `exact` search to validate the results 62 | 63 | # IOC Generation 64 | __Binarly__ provides an API to automatically generate [YARA](https://plusvic.github.io/yara/) rules that cover a specified set of files. The algorithm for selecting signature patterns has access to the entire universe of data indexed by the search engine, enabling it to identify the best signature candidate for a given file. 65 | 66 | The following are true for all Binarly generated YARA rules: 67 | * Each code pattern that make up a rule is _guaranteed not to hit on any of the millions of clean files_ indexed 68 | * For performance reasons, the first 4-gram of every pattern is selected such that the probability of the entire pattern to match is extremely high 69 | * The generated YARA rules are generic, i.e: they cover hundreds and even thousands of malicious files. 70 | 71 | The command `sign` accepts as input one of the following: 72 | * file names 73 | * directory paths 74 | * file hashes: MD5, SHA-1 and SHA-256 75 | 76 | `sign` accepts the following switches: 77 | * `--cluster` - this informs the signing process that the submitted files should be treated as being part of the same cluster i.e one signature should cover multiple files. This option is set to `False` by default. 78 | * `--patternCount` - sets the number of patterns that will make up the rule. Default value is 3. 79 | * `--strategy` - tells the signer which regions of the file to use, in order to generate rules. The default value is equivalent to `full file`. Another valid option is `strict`: signer will select file regions which will most probably be available to a resource-limited file scanner (such as an AntiVirus). 80 | * `--upload` - tells the script whether it should upload samples that are not in the Binarly collection when generating IOCs or doing file classification. By default this is set to `True`. 81 | 82 | Let's assume you want to sign a sample which has a certain SHA-1: 83 | 84 | `binarly_query.py sign ab9d4a010bb8d15c448a486f89ba2471b0e17d62` 85 | 86 | If successful, the signing process will generate a number of files: 87 | * A YARA file which contains all of the rules generated for your request. A typical rule would look as follows: 88 | 89 | ```ini 90 | rule auto_a50d5d0f_8fd1_4c61_a73c_8681bf05981f 91 | { 92 | meta: 93 | license = "Non-commercial use only" 94 | description = "Generated by Binarly (https://www.binar.ly)" 95 | label = "malware" 96 | allSampleCount = "6" 97 | 98 | strings: 99 | // 0x4032e3 03 4d 8c add ecx, dword ptr [ebp - 0x74] 100 | // 0x4032e6 85 c9 test ecx, ecx 101 | // 0x4032e8 74 09 je 0x4032f3 102 | // 0x4032ea 8b 55 ?? mov edx, dword ptr [ebp - 0x74] 103 | $a = { 03 4d 8c 85 c9 74 09 8b 55 } 104 | // 0x401611 8b 4d 8c mov ecx, dword ptr [ebp - 0x74] 105 | // 0x401614 03 4d d0 add ecx, dword ptr [ebp - 0x30] 106 | // 0x401617 85 c9 test ecx, ecx 107 | // 0x401619 74 ?? je 0x40162c 108 | $b = { 8b 4d 8c 03 4d d0 85 c9 74 } 109 | // 0x402fa7 8b 55 8c mov edx, dword ptr [ebp - 0x74] 110 | // 0x402faa 03 55 8c add edx, dword ptr [ebp - 0x74] 111 | // 0x402fad 85 d2 test edx, edx 112 | // 0x402faf 74 09 je 0x402fba 113 | // 0x402fb1 8b 45 ?? mov eax, dword ptr [ebp - 0x74] 114 | $c = { 8b 55 8c 03 55 8c 85 d2 74 09 8b 45 } 115 | 116 | condition: 117 | all of them 118 | } 119 | ``` 120 | * One or more JSON files which contain signature information for each of the signatures generated: 121 | - signature `patterns` 122 | - hashes of the `files that are detected` by the signature 123 | - `label of the detected files` (malware/unwanted software) 124 | - `family name` for the malware files detected by this rule 125 | 126 | The signer has many safeguards in place in order to avoid generation of rules on known clean containers such as packers/protectors/installers. 127 | 128 | Example of sign attempt on a [SmartInstaller](http://www.sminstall.com) installer file (referenced by SHA256): 129 | 130 | ![SmartInstaller Request](./images/signer_smartinstaller_detection.jpg) 131 | 132 | #Sample Hunting With YARA Signatures 133 | While `search` functionality allows one to search for files that contain a certain combination of query terms, you can use the `hunt` command for more complex queries. The sample hunting functionality works straight out of the box with most YARA rules already written. 134 | 135 | `hunt` returns a list of files that match a YARA rule specified as input. 136 | 137 | Let's assume that you want to search for samples detected by the following rule: 138 | 139 | ```ini 140 | rule SHIFU_Banking_Trojan { 141 | meta: 142 | description = "Detects SHIFU Banking Trojan" 143 | author = "Florian Roth" 144 | reference = "http://goo.gl/52n8WE" 145 | date = "2015-10-31" 146 | score = 70 147 | strings: 148 | $x1 = "\\Gather\\Dividerail.pdb" ascii 149 | 150 | $s0 = "\\payload\\payload.x86.pdb" ascii 151 | $s1 = "USER_PRIV_GUEST" fullword wide 152 | $s2 = "USER_PRIV_ADMIN" fullword wide 153 | $s3 = "USER_PRIV_USER" fullword wide 154 | $s4 = "PPSWVPP" fullword ascii 155 | $s5 = "WinSCard.dll" fullword ascii /* Goodware String - occured 83 times */ 156 | condition: 157 | uint16(0) == 0x5a4d and ($x1 or 5 of ($s*)) 158 | } 159 | ``` 160 | 161 | __Command:__ 162 | 163 | If the rule is contained in a file named `shifu_banking_trojan.yar`: 164 | 165 | `binarly_query.py hunt shifu_banking_trojan.yar` 166 | 167 | __Output:__ 168 | 169 | ![Results](./images/hunt.jpg) 170 | 171 | Constraints: 172 | * Only one YARA rule per request is allowed 173 | 174 | Current limitations: 175 | * YARA rules without strings is not supported 176 | 177 | #File classification 178 | One of the most useful and powerful features of Binarly is an API that classifies files as either clean or malicious based on an advanced __Machine Learning__ model. The model is trained using millions of samples with over 500,000 Machine Learning features per file. It is frequently updated in order to take into account the latest threats. 179 | 180 | The `classify` command accepts one of: 181 | * file names 182 | * directory paths 183 | * file hashes: MD5, SHA-1 and SHA-256 184 | 185 | File classification is __extremely fast__ because it relies on static file features. 186 | 187 | Below is an example of classification done on all of the file in a directory: 188 | 189 | ![Classification](./images/classification.jpg) 190 | -------------------------------------------------------------------------------- /binarly_query.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | import argparse 3 | import datetime 4 | import glob 5 | import json 6 | import os 7 | import sys 8 | 9 | try: 10 | from tabulate import tabulate 11 | 12 | HAS_TABULATE = True 13 | except ImportError: 14 | HAS_TABULATE = False 15 | 16 | try: 17 | from colorama import init, Fore, Style 18 | except ImportError: 19 | print( 20 | "Error importing colorama. Please make sure you have it (pip install colorama)" 21 | ) 22 | sys.exit(-1) 23 | 24 | try: 25 | from BinarlyAPIv1 import BinarlyAPI, hex_pattern, ascii_pattern, wide_pattern 26 | except ImportError: 27 | print( 28 | "Error importing BinarlyAPI. You can find it here https://github.com/binarlyhq/binarly-sdk" 29 | ) 30 | sys.exit(-1) 31 | 32 | BINOBJ = None 33 | APIKEYFILENAME = 'apikey.txt' 34 | APIKEYPATH = os.path.join(os.path.dirname(__file__), APIKEYFILENAME) 35 | 36 | APIKEY = '' 37 | 38 | ARGPARSER = argparse.ArgumentParser( 39 | description='Binarly API Query', fromfile_prefix_chars="@") 40 | ARGPARSER.add_argument("--key", "-k", help="Binarly APIKey", default='') 41 | ARGPARSER.add_argument( 42 | "--server", "-s", help="Set Binarly API endpoint", default='www.binar.ly') 43 | 44 | ARGPARSER.add_argument( 45 | "--usehttp", 46 | "-u", 47 | help="Use HTTP instead of HTTPS when communicating. By default HTTPS is used.", 48 | action="store_true") 49 | 50 | ARGPARSER.add_argument( 51 | "--pretty-print", 52 | "-p", 53 | help="Display results in a nicely formated table (Requires tabulate python module)", 54 | action="store_true" 55 | ) 56 | 57 | ARG_SUBPARSERS = ARGPARSER.add_subparsers(help='commands', dest='commands') 58 | 59 | SEARCH_PARSER = ARG_SUBPARSERS.add_parser( 60 | 'search', help="Search arbitrary hex patterns") 61 | SEARCH_PARSER.add_argument("hex", type=str, nargs='*', default=[]) 62 | SEARCH_PARSER.add_argument( 63 | "-a", nargs="*", help="ASCII string to search", default=[], action="append") 64 | SEARCH_PARSER.add_argument( 65 | "-w", nargs="*", help="WIDE string to search", default=[], action="append") 66 | SEARCH_PARSER.add_argument( 67 | "--limit", 68 | type=int, 69 | default=20, 70 | help="Limit the number of results returned. If 0 only statistics are returned") 71 | SEARCH_PARSER.add_argument( 72 | "--exact", action='store_true', help="Validate search results") 73 | SEARCH_PARSER.add_argument( 74 | "--test", action='store_true', help="Run in test environment") 75 | 76 | HUNT_PARSER = ARG_SUBPARSERS.add_parser( 77 | 'hunt', help='Hunt for files using YARA rules') 78 | HUNT_PARSER.add_argument('yarafile', type=str) 79 | HUNT_PARSER.add_argument( 80 | "--test", action='store_true', help="Run in test environment") 81 | 82 | SIGN_PARSER = ARG_SUBPARSERS.add_parser('sign', help="Generate IOC on samples") 83 | SIGN_PARSER.add_argument( 84 | "files", 85 | type=str, 86 | nargs='+', 87 | help="Files/Hashes (md5/sha1/sha256) to send to signer") 88 | 89 | SIGN_PARSER.add_argument( 90 | "--patternCount", 91 | "-c", 92 | type=int, 93 | default=3, 94 | help="Specify the number of fragments in a generated rule", 95 | dest='fragcount') 96 | SIGN_PARSER.add_argument( 97 | "--strategy", 98 | "-s", 99 | type=str, 100 | choices=['none', 'strict'], 101 | help="Specify if the signature should be extracted from full file (none) or a subset (strict)", 102 | default='none') 103 | SIGN_PARSER.add_argument( 104 | "--cluster", 105 | help="Treat files as a cluster in order to minimize the number of generated signatures", 106 | action='store_true') 107 | SIGN_PARSER.add_argument( 108 | "--other", 109 | nargs='*', 110 | help="Specify additional options to send, in the form of a tuple (key, value)", 111 | default=[], 112 | action='store') 113 | 114 | SIGN_PARSER.add_argument( 115 | "--u", type=bool, help='Upload file(s) if missing', default=True) 116 | SIGN_PARSER.add_argument( 117 | "--yara", 118 | help='Dump generated YARA signatures to screen', 119 | default=False, 120 | action="store_true") 121 | 122 | CLASSIFY_PARSER = ARG_SUBPARSERS.add_parser( 123 | 'classify', help="Classify samples using Machine Learning") 124 | CLASSIFY_PARSER.add_argument("files", type=str, nargs='+') 125 | CLASSIFY_PARSER.add_argument( 126 | "-u", type=bool, help='Upload file(s) if missing', default=True) 127 | 128 | FILEINFO_PARSER = ARG_SUBPARSERS.add_parser( 129 | 'metadata', help="Retrieve file metadata") 130 | FILEINFO_PARSER.add_argument( 131 | "filehash", 132 | type=str, 133 | help="File hash (md5/sha1/sha256) to retrieve metadata") 134 | 135 | USAGE_PARSER = ARG_SUBPARSERS.add_parser('demo', help="Show usage examples") 136 | 137 | LABEL_COLOR = { 138 | 'clean': Style.BRIGHT + Fore.GREEN, 139 | 'malware': Style.BRIGHT + Fore.RED, 140 | 'pua': Style.BRIGHT + Fore.YELLOW, 141 | 'unknown': Style.BRIGHT + Fore.CYAN, 142 | 'suspicious': Style.BRIGHT + Fore.MAGENTA 143 | } 144 | 145 | 146 | def dump(obj, nested_level=0, output=sys.stdout): 147 | spacing = ' ' 148 | if isinstance(obj, dict): 149 | print >> output, '%s{' % (nested_level * spacing) 150 | for key, value in obj.items(): 151 | if hasattr(value, '__iter__'): 152 | print >> output, '%s%s:' % ((nested_level + 1) * spacing, key) 153 | dump(value, nested_level + 1, output) 154 | else: 155 | print >> output, '%s%s: %s' % ((nested_level + 1) * spacing, 156 | key, value) 157 | print >> output, '%s}' % (nested_level * spacing) 158 | elif isinstance(obj, list): 159 | print >> output, '%s[' % (nested_level * spacing) 160 | for value in obj: 161 | if hasattr(value, '__iter__'): 162 | dump(value, nested_level + 1, output) 163 | else: 164 | print >> output, '%s%s' % ((nested_level + 1) * spacing, value) 165 | print >> output, '%s]' % (nested_level * spacing) 166 | else: 167 | print >> output, '%s%s' % (nested_level * spacing, obj) 168 | 169 | 170 | def smart_size(size): 171 | if not isinstance(size, int): 172 | try: 173 | size = int(size) 174 | except ValueError: 175 | return size 176 | 177 | if size >= 1024 * 1024 * 1024: 178 | return "{0:>7.2f}GB".format(float(size) / (1024 * 1024 * 1024)) 179 | elif size >= 1024 * 1024: 180 | return "{0:>7.2f}MB".format(float(size) / (1024 * 1024)) 181 | elif size > 1024: 182 | return "{0:>7.2f}KB".format(float(size) / 1024) 183 | else: 184 | return "{0:>8d}B".format(int(size)) 185 | 186 | 187 | def get_filelist(dirname): 188 | return [x for x in glob.glob(os.path.join(dirname, '*')) 189 | if os.path.isfile(x)] 190 | 191 | 192 | def color_row(row): 193 | color = Fore.WHITE 194 | label = "." 195 | if u'label' in row: 196 | color = LABEL_COLOR.get(row[u'label'], Fore.WHITE) 197 | label = row[u'label'] 198 | row[u'label'] = "%s%s%s" % (color, label, Style.RESET_ALL) 199 | 200 | row['family'] = "%s%s%s" % (color, row.get('family', "."), Style.RESET_ALL) 201 | row['size'] = smart_size(row.get(u'size', ".")) 202 | return row 203 | 204 | 205 | def show_row(row): 206 | row = color_row(row) 207 | print(" ".join(["%s%s%s:%s" % (Style.NORMAL, x.capitalize(), Style.BRIGHT, y) for (x, y) in row.items()])) 208 | 209 | 210 | def show_results(results, pretty_print): 211 | if pretty_print: 212 | [color_row(x) for x in results] 213 | print tabulate(results, headers="keys", tablefmt="grid", stralign="right") 214 | else: 215 | print("-" * 100) 216 | for val in results: 217 | show_row(val) 218 | 219 | 220 | def show_stats(stats): 221 | print( 222 | "Found {0} results : {1}{2} clean {3}{4} malware {5}{6} PUA {7}{8} unknown {9}{10} suspicious".format( 223 | stats['total_count'], LABEL_COLOR['clean'], 224 | stats['clean_count'], LABEL_COLOR['malware'], 225 | stats['malware_count'], LABEL_COLOR['pua'], stats['pua_count'], 226 | LABEL_COLOR['unknown'], stats['unknown_count'], 227 | LABEL_COLOR['suspicious'], stats['suspicious_count']) 228 | ) 229 | 230 | 231 | def show_stats_new(stats, limit): 232 | if stats['total_count'] > limit: 233 | print( 234 | "Results [{0}/{1}] : {2}{3} clean {4}{5} malware {6}{7} PUA {8}{9} unknown {10}{11} suspicious".format( 235 | limit, stats['total_count'], LABEL_COLOR['clean'], 236 | stats['clean_count'], LABEL_COLOR['malware'], 237 | stats['malware_count'], LABEL_COLOR['pua'], stats['pua_count'], 238 | LABEL_COLOR['unknown'], stats['unknown_count'], 239 | LABEL_COLOR['suspicious'], stats['suspicious_count'])) 240 | else: 241 | print( 242 | "Results [{0}/{0}] : {1}{2} clean {3}{4} malware {5}{6} PUA {7}{8} unknown {9}{10} suspicious".format( 243 | stats['total_count'], LABEL_COLOR['clean'], 244 | stats['clean_count'], LABEL_COLOR['malware'], 245 | stats['malware_count'], LABEL_COLOR['pua'], stats['pua_count'], 246 | LABEL_COLOR['unknown'], stats['unknown_count'], 247 | LABEL_COLOR['suspicious'], stats['suspicious_count'])) 248 | 249 | 250 | def process_search(options): 251 | search_query = [] 252 | search_query.extend([hex_pattern(val.replace(' ', '')) for val in options.hex]) 253 | search_query.extend([ascii_pattern(val) for lst in options.a for val in lst]) 254 | search_query.extend([wide_pattern(val) for lst in options.w for val in lst]) 255 | 256 | result = BINOBJ.search( 257 | search_query, limit=options.limit, exact=options.exact, test=options.test) 258 | if 'error' in result: 259 | print(Style.BRIGHT + Fore.RED + result['error']['message']) 260 | return 261 | 262 | if 'stats' in result: 263 | show_stats_new(result['stats'], options.limit) 264 | 265 | if len(result['results']) == 0: 266 | return 267 | 268 | # if len(result['results']) >= options.limit: 269 | # print("Showing top {0} results:".format(options.limit)) 270 | # else: 271 | # print("Results:") 272 | 273 | show_results(result['results'], pretty_print=options.pretty_print) 274 | 275 | 276 | def process_classify(options): 277 | if os.path.exists(options.files[0]): 278 | filelist = options.files 279 | if os.path.isdir(options.files[0]): 280 | filelist = get_filelist(filelist[0]) 281 | 282 | result = BINOBJ.classify_files( 283 | filelist, upload_missing=options.u, status_callback=my_callback) 284 | else: 285 | result = BINOBJ.classify_hashes(options.files) 286 | 287 | if 'error' in result or result['status'] != 'done': 288 | print(Style.BRIGHT + Fore.RED + "Request failed") 289 | else: 290 | print("Classification Results:") 291 | 292 | reqid = result.get('results', None) 293 | if reqid is None: 294 | # the request failed before any files could be analyzed 295 | print(Style.BRIGHT + Fore.RED + 296 | "Fail reason: {0} (error code={1})".format( 297 | result['error']['message'], result['error']['code'])) 298 | return 299 | 300 | classify_data = [] 301 | for key, value in result['results'].iteritems(): 302 | status = Style.RESET_ALL + Fore.GREEN + "OK" + Style.RESET_ALL 303 | if 'error' in value: 304 | status = Fore.RED + value['error']['message'] + Style.RESET_ALL 305 | row = {'SHA1': key, 'label': value.get('label', '.'), 'family': value.get('family', '.'), 'Status': status} 306 | 307 | classify_data.append(row) 308 | 309 | if options.pretty_print: 310 | show_results(classify_data, pretty_print=options.pretty_print) 311 | else: 312 | print("-" * 100) 313 | for row in classify_data: 314 | show_row(row) 315 | return 316 | 317 | 318 | def process_hunt(options): 319 | result = BINOBJ.yara_hunt(options.yarafile, options.test, my_callback) 320 | if 'error' in result or result['status'] != 'done': 321 | print(Style.BRIGHT + Fore.RED + "Request failed.") 322 | print(Style.BRIGHT + Fore.RED + 323 | "Fail reason: {0} (error code={1})".format( 324 | result['error']['message'], result['error']['code'])) 325 | return 326 | 327 | if 'stats' in result: 328 | show_stats(result['stats']) 329 | 330 | if len(result['results']) > 0: 331 | show_results(result['results'], pretty_print=options.pretty_print) 332 | 333 | 334 | def my_callback(response): 335 | print("{0} : Request status = {1:<10}".format( 336 | datetime.datetime.now(), response.get('status', None))) 337 | 338 | 339 | def process_sign(options): 340 | sign_options = {'strategy': options.strategy, 341 | 'frag_count': options.fragcount, 342 | 'cluster': options.cluster} 343 | 344 | if os.path.exists(options.files[0]): 345 | filelist = options.files 346 | if os.path.isdir(options.files[0]): 347 | filelist = get_filelist(filelist[0]) 348 | 349 | result = BINOBJ.gen_ioc_files( 350 | filelist, 351 | options=sign_options, 352 | upload_missing=options.u, 353 | status_callback=my_callback) 354 | else: 355 | result = BINOBJ.gen_ioc_hashes( 356 | options.files, status_callback=my_callback) 357 | 358 | if 'error' in result or result['status'] != 'done': 359 | print(Style.BRIGHT + Fore.RED + "Request failed.") 360 | else: 361 | print("Generated {0} signature(s) in {1:d}s".format( 362 | len(result.get('signatures', [])), 363 | result['stats']['time_ms'] / 1000)) 364 | 365 | reqid = result.get('reqid', None) 366 | if reqid is None: 367 | # the request failed before any files could be analyzed 368 | print(Style.BRIGHT + Fore.RED + 369 | "Fail reason: {0} (error code={1})".format( 370 | result['error']['message'], result['error']['code'])) 371 | return 372 | 373 | yara_signatures = [] 374 | for idx, signature in enumerate(result.get('signatures', [])): 375 | sig_info = BINOBJ.get_request(signature['info']) 376 | with open("auto_{0}_{1}.json".format(reqid, idx), mode="w") as sigfile: 377 | sigfile.write(json.dumps(sig_info)) 378 | 379 | yarasig = BINOBJ.get_request(signature['yarasig']) 380 | yara_signatures.append(yarasig) 381 | 382 | with open("auto_{0}.yar".format(reqid), mode="a") as sigfile: 383 | sigfile.write(yarasig) 384 | 385 | print( 386 | "Sig #{0} - detects {1} indexed files from family: {2}{3}".format( 387 | idx, len(sig_info.get('samples', [])), 388 | LABEL_COLOR[sig_info.get('label', "malware")], 389 | sig_info.get('family', "N/A"))) 390 | 391 | print("Signing results:") 392 | for filehash, info in result['results'].iteritems(): 393 | status = Fore.GREEN + 'Signed' 394 | if info['status'] != 'signed': 395 | status = Fore.RED + "Failed ({0})".format(info['error']['message']) 396 | 397 | print("Hash:{0}{1}{2} Status:{3}".format(Style.BRIGHT, filehash, 398 | Style.RESET_ALL, status)) 399 | 400 | if len(yara_signatures) > 0: 401 | print("\nPlease check {0} file for generated signature(s).".format( 402 | "auto_{0}.yar".format(reqid))) 403 | 404 | if options.yara: 405 | print "YARA Rules:" 406 | for rule in yara_signatures: 407 | print rule 408 | return 409 | 410 | 411 | def process_metadata(options): 412 | result = BINOBJ.get_metadata(options.filehash) 413 | if 'error' in result: 414 | print(Style.BRIGHT + Fore.RED + result['error']['message']) 415 | return 416 | 417 | dump(result[options.filehash]) 418 | 419 | 420 | def process_demo(options): 421 | return 422 | 423 | 424 | def read_apikey(filepath=APIKEYPATH): 425 | global APIKEY 426 | 427 | if not os.path.exists(filepath): 428 | return False 429 | 430 | with open(filepath, 'r') as fhandle: 431 | APIKEY = fhandle.readline() 432 | 433 | APIKEY = APIKEY.strip() 434 | return True 435 | 436 | 437 | def init_api(options): 438 | global BINOBJ, APIKEY 439 | 440 | APIKEY = options.key 441 | if len(APIKEY) == 0 and read_apikey() is False: 442 | raise RuntimeError( 443 | "You need to provide an API access key. Register at https://binar.ly in order to receive one") 444 | 445 | BINOBJ = BinarlyAPI( 446 | server=options.server, 447 | api_key=APIKEY, 448 | use_http=options.usehttp, 449 | project="BinarlyPyQuery") 450 | return 451 | 452 | 453 | def main(options): 454 | if options.pretty_print and not HAS_TABULATE: 455 | print(Style.BRIGHT + Fore.RED + "Pretty printing requires tabulate python module. (pip install tabulate)") 456 | return 457 | 458 | init_api(options) 459 | cmd = options.commands 460 | 461 | switcher = { 462 | 'search': process_search, 463 | 'hunt': process_hunt, 464 | 'sign': process_sign, 465 | 'classify': process_classify, 466 | 'metadata': process_metadata, 467 | 'demo': process_demo 468 | } 469 | 470 | # Get the function from switcher dictionary 471 | process_fn = switcher.get(cmd) 472 | # Execute the function 473 | return process_fn(options) 474 | 475 | 476 | if __name__ == "__main__": 477 | init(autoreset=True) 478 | main(ARGPARSER.parse_args()) 479 | -------------------------------------------------------------------------------- /images/classification.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/binarlyhq/binarly-query/5d47e73b83d55e5bb7a6f31711243f764b48b27d/images/classification.jpg -------------------------------------------------------------------------------- /images/hunt.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/binarlyhq/binarly-query/5d47e73b83d55e5bb7a6f31711243f764b48b27d/images/hunt.jpg -------------------------------------------------------------------------------- /images/search.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/binarlyhq/binarly-query/5d47e73b83d55e5bb7a6f31711243f764b48b27d/images/search.jpg -------------------------------------------------------------------------------- /images/search_exact.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/binarlyhq/binarly-query/5d47e73b83d55e5bb7a6f31711243f764b48b27d/images/search_exact.jpg -------------------------------------------------------------------------------- /images/signer_smartinstaller_detection.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/binarlyhq/binarly-query/5d47e73b83d55e5bb7a6f31711243f764b48b27d/images/signer_smartinstaller_detection.jpg --------------------------------------------------------------------------------