├── LICENSE ├── README.md ├── intel_linter.py └── tests ├── t.intel ├── t.intel-addr ├── t.intel-bad-addr ├── t.intel-bad-blank-field ├── t.intel-bad-character ├── t.intel-bad-field-sep ├── t.intel-bad-indicator-type ├── t.intel-bad-tab ├── t.intel-default-handler ├── t.intel-email ├── t.intel-missing-header ├── t.intel-too-few-fields └── t.intel-too-many-fields /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "[]" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright 2019 MixMode.ai (dba, Packetsled, Inc.) 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. 202 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | 2 | # Zeek (Bro) Intel Feed Linter 3 | The nsm_intel_linter was built to verify all the appropriate header delineation and mandatory field verification, tab separation, correlation of indicator and indicator_type. 4 | 5 | Currently supports Zeek (Bro) intelligence feeds. 6 | 7 | ## Usage 8 | intel_linter.py -f 9 | 10 | ## Example 11 | 12 | ### Example File 13 | 14 | Test File: 15 | ~~~ 16 | #fields indicator indicator_type meta.source meta.desc meta.url meta.do_notice meta.if_in meta.whitelist meta.severity 17 | 192.168.1.1 Intel::ADDR my imagination ADDR - F - - 6 18 | 192.168.1.2 Intel::ADDR my imagination ADDR - F - - 6 19 | 192.168.1.300 Intel::ADDR my imagination ADDR - F - - 6 20 | ~~~ 21 | 22 | Result: 23 | ~~~ 24 | WARNING: Line 4 - Indicator type "Intel::ADDR" does not correlate with indicator: "192.168.1.300" 25 | ~~~ 26 | 27 | A clean execution means the intelligence file supplied passed all lint testing. 28 | -------------------------------------------------------------------------------- /intel_linter.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | # 3 | # MixMode.ai - Bro Intel Linter 4 | # 5 | # WHEN WHAT WHO 6 | # 03-04-2015 Initial development Aaron Eppert 7 | # 08-24-2015 Explicitly verify single character fields Aaron Eppert 8 | # 08-24-2015 GPL and pushed to GitHub Aaron Eppert 9 | # 08-25-2015 Small cleanups and proper exit codes for using 10 | # as a git pre-commit hook Aaron Eppert 11 | # 09-01-2015 Added column-based type verifications Aaron Eppert 12 | # 09-25-2015 Verify printable characters and escape in error Aaron Eppert 13 | # 10-07-2015 Added --psled and --warn-only options Aaron Eppert 14 | # 10-08-2015 Additional details - WARNING vs ERROR Aaron Eppert 15 | # 03-03-2016 Minor bugfix Peter McKay 16 | # 04-08-2016 Added Intel::NET support Aaron Eppert 17 | # 06-02-2017 Fixed line ending issue Aaron Eppert 18 | # 09-15-2017 Changed Intel::NET to Intel::SUBNET Kory Kyzar 19 | # 03-28-2018 Fixed IPv6 validation Aaron Eppert 20 | # 03-27-2019 Add Intel::PUBKEY_HASH and Intel::JA3 Aaron Eppert 21 | # 07-13-2019 Add CERT HASH validaion for using regex Juan Jaramillo 22 | # MD5, SHA1, SHA256, SHA512 hashes. 23 | 24 | import sys 25 | import re 26 | import string 27 | from optparse import OptionParser 28 | 29 | 30 | def write_stderr(msg): 31 | sys.stderr.write(msg + '\n') 32 | 33 | 34 | def warning_line(line, *objs): 35 | out = 'WARNING: Line %d - ' % (int(line)+1) 36 | for o in objs: 37 | out += o 38 | write_stderr(out) 39 | 40 | 41 | def error_line(line, *objs): 42 | out = 'ERROR: Line %d - ' % (int(line)+1) 43 | for o in objs: 44 | out += o 45 | write_stderr(out) 46 | 47 | 48 | def escape(c): 49 | if ord(c) > 31 and ord(c) < 127: 50 | return c 51 | c = ord(c) 52 | if c <= 0xff: 53 | return r'\x{0:02x}'.format(c) 54 | elif c <= '\uffff': 55 | return r'\u{0:04x}'.format(c) 56 | else: 57 | return r'\U{0:08x}'.format(c) 58 | 59 | 60 | def hex_escape(s): 61 | return ''.join(escape(c) for c in s) 62 | 63 | 64 | class bro_intel_indicator_return: 65 | OKAY = 0 66 | WARNING = 1 67 | ERROR = 2 68 | 69 | 70 | ############################################################################### 71 | # class bro_intel_indicator_type 72 | # 73 | # This class is for handling the "indicator_type" fields within a Bro Intel 74 | # file. Note, each type of field has a specific handler. 75 | # 76 | class bro_intel_indicator_type: 77 | def __init__(self): 78 | self.__INDICATOR_TYPE_handler = {'Intel::ADDR': self.__handle_intel_addr, 79 | 'Intel::SUBNET': self.__handle_intel_subnet, 80 | 'Intel::URL': self.__handle_intel_url, 81 | 'Intel::SOFTWARE': self.__handle_intel_software, 82 | 'Intel::EMAIL': self.__handle_intel_email, 83 | 'Intel::DOMAIN': self.__handle_intel_domain, 84 | 'Intel::USER_NAME': self.__handle_intel_user_name, 85 | 'Intel::FILE_HASH': self.__handle_intel_file_hash, 86 | 'Intel::FILE_NAME': self.__handle_intel_file_name, 87 | 'Intel::CERT_HASH': self.__handle_intel_cert_hash, 88 | 'Intel::PUBKEY_HASH': self.__handle_intel_pubkey_hash, 89 | 'Intel::JA3': self.__handle_intel_ja3_hash} 90 | 91 | # Source: https://stackoverflow.com/questions/319279/how-to-validate-ip-address-in-python 92 | def __is_valid_ipv4_address(self, address): 93 | import socket 94 | 95 | try: 96 | socket.inet_pton(socket.AF_INET, address) 97 | except AttributeError: # no inet_pton here, sorry 98 | try: 99 | socket.inet_aton(address) 100 | except socket.error: 101 | return False 102 | return address.count('.') == 3 103 | except socket.error: # not a valid address 104 | return False 105 | 106 | return True 107 | 108 | # Source: https://stackoverflow.com/questions/319279/how-to-validate-ip-address-in-python 109 | def __is_valid_ipv6_address(self, address): 110 | import socket 111 | try: 112 | socket.inet_pton(socket.AF_INET6, address) 113 | except socket.error: # not a valid address 114 | return False 115 | return True 116 | 117 | def __handle_intel_addr(self, indicator): 118 | ret = (bro_intel_indicator_return.OKAY, None) 119 | 120 | if self.__is_valid_ipv4_address(indicator) or self.__is_valid_ipv6_address(indicator): 121 | return ret 122 | return (bro_intel_indicator_return.ERROR, 'Invalid IP address') 123 | 124 | # In an effort to keep this script minimal and without requiring external 125 | # libraries, we will verify an Intel::SUBNET simply as: 126 | # 127 | # 0 <= octet < 255 128 | # 0 <= netmask <= 32 129 | # 130 | def __handle_intel_subnet(self, indicator): 131 | ret = (bro_intel_indicator_return.OKAY, None) 132 | if '/' in indicator: 133 | addr, net = indicator.split('/') 134 | if all([(int(x) >= 0 and int(x) < 255) for x in addr.split('.')]): 135 | if not (int(net) >= 0 and int(x) <= 32): 136 | ret = (bro_intel_indicator_return.ERROR, 'Invalid network block designation') 137 | else: 138 | ret = (bro_intel_indicator_return.ERROR, 'Invalid network address') 139 | else: 140 | ret = (bro_intel_indicator_return.ERROR, 'Invalid network designation') 141 | return ret 142 | 143 | # We will call this minimalist, but effective. 144 | def __handle_intel_url(self, indicator): 145 | ret = (bro_intel_indicator_return.OKAY, None) 146 | 147 | t_uri_present = re.findall(r'^https?://', indicator) 148 | if t_uri_present is not None and len(t_uri_present) > 0: 149 | ret = (bro_intel_indicator_return.WARNING, 'URI present (e.g. http(s)://)') 150 | else: 151 | rx = re.compile(r'^[https?://]?' # http:// or https:// 152 | r'(?:(?:[A-Z0-9](?:[A-Z0-9-]{0,61}[A-Z0-9])?\.)+[A-Z]{2,6}\.?|' # domain... 153 | r'localhost|' # localhost... 154 | r'\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})' # ...or ip 155 | r'(?::\d+)?' # optional port 156 | r'(?:/?|[/?]\S+)$', re.IGNORECASE) 157 | t = rx.search(indicator) 158 | if t: 159 | ret = (bro_intel_indicator_return.OKAY, None) 160 | return ret 161 | 162 | def __handle_intel_email(self, indicator): 163 | ret = (bro_intel_indicator_return.WARNING, 'Invalid email address') 164 | rx = r"(^[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+$)" 165 | t_email = re.findall(rx, indicator) 166 | if len(t_email) > 0: 167 | ret = (bro_intel_indicator_return.OKAY, None) 168 | return ret 169 | 170 | def __handle_intel_software(self, indicator): 171 | ret = (bro_intel_indicator_return.WARNING, 'Invalid software string') 172 | if len(indicator) > 0: 173 | ret = (bro_intel_indicator_return.OKAY, None) 174 | return ret 175 | 176 | def __handle_intel_domain(self, indicator): 177 | ret = (bro_intel_indicator_return.WARNING, 'Invalid domain name') 178 | rx = r'(?=^.{4,253}$)(^((?!-)[a-zA-Z0-9-]{1,63}(? 0: 181 | if indicator in t_domain[0]: 182 | ret = (bro_intel_indicator_return.OKAY, None) 183 | return ret 184 | 185 | def __handle_intel_user_name(self, indicator): 186 | ret = (bro_intel_indicator_return.WARNING, 'Invalid username - %s' % (indicator)) 187 | if len(indicator) > 0: 188 | ret = (bro_intel_indicator_return.OKAY, None) 189 | return ret 190 | 191 | def __handle_intel_file_name(self, indicator): 192 | ret = (bro_intel_indicator_return.WARNING, 'Invalid username length') 193 | if len(indicator) > 0: 194 | ret = (bro_intel_indicator_return.OKAY, None) 195 | return ret 196 | 197 | # Pretty weak, but should suffice for now. 198 | def __handle_intel_file_hash(self, indicator): 199 | ret = (bro_intel_indicator_return.WARNING, 'Invalid hash length') 200 | VALID_HASH_LEN = {32: 'md5', 201 | 40: 'sha1', 202 | 64: 'sha256'} 203 | if VALID_HASH_LEN.get(len(indicator), None): 204 | ret = (bro_intel_indicator_return.OKAY, None) 205 | return ret 206 | 207 | def __handle_intel_cert_hash(self, indicator): 208 | ret = (bro_intel_indicator_return.WARNING, 'Invalid Intel::CERT_HASH - ISSUES %s' % (indicator)) 209 | hash_present = re.compile( 210 | r'^[0-9A-F]{32}$|' # MD5 211 | r'^[0-9A-F]{40}$|' # SHA1 212 | r'^[0-9A-F]{64}$|' # SHA256 213 | r'^[0-9A-F]{128}$', re.IGNORECASE) # SHA512 214 | t = hash_present.search(indicator) 215 | if t: 216 | ret = (bro_intel_indicator_return.OKAY, None) 217 | return ret 218 | 219 | def __handle_intel_pubkey_hash(self, indicator): 220 | return (bro_intel_indicator_return.WARNING, 'Intel::PUBKEY_HASH - Needs additional validation') 221 | 222 | def __handle_intel_ja3_hash(self, indicator): 223 | ret = (bro_intel_indicator_return.WARNING, 'Intel::JA3 - Needs additional validation') 224 | if len(indicator) == 32: 225 | ret = (bro_intel_indicator_return.OKAY, None) 226 | return ret 227 | 228 | def verify_indicator_type(self, indicator_type): 229 | ret = (bro_intel_indicator_return.ERROR, 'Invalid indicator - %s' % (indicator_type)) 230 | it = self.__INDICATOR_TYPE_handler.get(indicator_type, None) 231 | if it is not None: 232 | ret = (bro_intel_indicator_return.OKAY, None) 233 | return ret 234 | 235 | def correlate(self, indicator, indicator_type): 236 | ret = (bro_intel_indicator_return.WARNING, 'Could not correlate - %s with %s' % (indicator, indicator_type)) 237 | if len(indicator) > 1 and len(indicator_type) > 1: 238 | h = self.__INDICATOR_TYPE_handler.get(indicator_type, None) 239 | if h: 240 | ret = h(indicator) 241 | else: 242 | ret = (bro_intel_indicator_return.OKAY, None) 243 | return ret 244 | 245 | 246 | ############################################################################### 247 | # class bro_data_intel_field_values 248 | # 249 | # This class is for processing the individual Bro Intel fields and verifying 250 | # their validity. 251 | # 252 | # Note, it may be easily expanded via adding entries to self.__VERIFY within 253 | # the class constructor. 254 | # 255 | class bro_data_intel_field_values: 256 | EMPTY_FIELD_CHAR = '-' 257 | META_DO_NOTICE = ['T', 'F'] 258 | 259 | META_IF_IN = ['-', 260 | 'Conn::IN_ORIG', 261 | 'Conn::IN_RESP', 262 | 'Files::IN_HASH', 263 | 'Files::IN_NAME', 264 | 'DNS::IN_REQUEST', 265 | 'DNS::IN_RESPONSE', 266 | 'HTTP::IN_HOST_HEADER', 267 | 'HTTP::IN_REFERRER_HEADER', 268 | 'HTTP::IN_USER_AGENT_HEADER', 269 | 'HTTP::IN_X_FORWARDED_FOR_HEADER', 270 | 'HTTP::IN_URL', 271 | 'SMTP::IN_MAIL_FROM', 272 | 'SMTP::IN_RCPT_TO', 273 | 'SMTP::IN_FROM', 274 | 'SMTP::IN_TO', 275 | 'SMTP::IN_RECEIVED_HEADER', 276 | 'SMTP::IN_REPLY_TO', 277 | 'SMTP::IN_X_ORIGINATING_IP_HEADER', 278 | 'SMTP::IN_MESSAGE', 279 | 'SSL::IN_SERVER_CERT', 280 | 'SSL::IN_CLIENT_CERT', 281 | 'SSL::IN_SERVER_NAME', 282 | 'SMTP::IN_HEADER'] 283 | 284 | def __init__(self): 285 | self.__VERIFY = {'indicator': self.verify_indicator, 286 | 'indicator_type': self.verify_indicator_type, 287 | 'meta.do_notice': self.verify_meta_do_notice, 288 | 'meta.if_in': self.verify_meta_if_in, 289 | 'meta.desc': self.verify_meta_desc, 290 | 'meta.source': self.verify_meta_source, 291 | 'meta.cif_confidence': self.verify_meta_cif_confidence, 292 | 'meta.url': self.verify_meta_url, 293 | 'meta.whitelist': self.verify_meta_whitelist, 294 | 'meta.severity': self.verify_meta_severity, 295 | 'meta.cif_severity': self.verify_meta_cif_severity, 296 | 'meta.cif_impact': self.verify_meta_cif_impact} 297 | 298 | self.biit = bro_intel_indicator_type() 299 | 300 | def get_verifier(self, v): 301 | return self.__VERIFY.get(v, self.default) 302 | 303 | def __verify_chars(self, t): 304 | return all(ord(l) > 31 and ord(l) < 127 and l in string.printable for l in t) 305 | 306 | def __is_ignore_field(self, t): 307 | return self.EMPTY_FIELD_CHAR in t 308 | 309 | def verify_indicator(self, t): 310 | ret = (bro_intel_indicator_return.ERROR, 'Invalid indicator - %s' % (t)) 311 | if len(t) > 1 and self.__verify_chars(t): 312 | ret = (bro_intel_indicator_return.OKAY, None) 313 | return ret 314 | 315 | def verify_indicator_type(self, t): 316 | return self.biit.verify_indicator_type(t) 317 | 318 | def correlate_indictor_and_indicator_type(self, i, it): 319 | return self.biit.correlate(i, it) 320 | 321 | def verify_meta_do_notice(self, t): 322 | ret = (bro_intel_indicator_return.OKAY, None) 323 | t_ret = t in bro_data_intel_field_values.META_DO_NOTICE 324 | if not t_ret: 325 | ret = (bro_intel_indicator_return.ERROR, 'Invalid do_notice - %s' % (str(t))) 326 | return ret 327 | 328 | def verify_meta_if_in(self, t): 329 | ret = (bro_intel_indicator_return.OKAY, None) 330 | t_ret = t in bro_data_intel_field_values.META_IF_IN 331 | if not t_ret: 332 | ret = (bro_intel_indicator_return.ERROR, 'Invalid if_in - %s' % (str(t))) 333 | return ret 334 | 335 | def verify_meta_cif_confidence(self, t): 336 | ret = (bro_intel_indicator_return.ERROR, 'Invalid confidence - %s - Needs to be 1-100' % (str(t))) 337 | try: 338 | t_int = int(t) 339 | if isinstance(t_int, (int, long)) and (t_int > 0 and t_int < 100): 340 | ret = (bro_intel_indicator_return.OKAY, None) 341 | except ValueError: 342 | ret = (bro_intel_indicator_return.ERROR, 'Invalid confidence - %s - Needs to be 1-100' % (str(t))) 343 | return ret 344 | 345 | def verify_meta_desc(self, t): 346 | ret = (bro_intel_indicator_return.WARNING, 'Invalid desc - %s' % (t)) 347 | if self.__is_ignore_field(t): 348 | ret = (bro_intel_indicator_return.OKAY, None) 349 | elif len(t) > 1 and self.__verify_chars(t): 350 | ret = (bro_intel_indicator_return.OKAY, None) 351 | return ret 352 | 353 | def verify_meta_source(self, t): 354 | ret = (bro_intel_indicator_return.WARNING, 'Invalid source - %s' % (t)) 355 | if self.__is_ignore_field(t): 356 | ret = (bro_intel_indicator_return.OKAY, None) 357 | elif len(t) > 1 and self.__verify_chars(t): 358 | ret = (bro_intel_indicator_return.OKAY, None) 359 | return ret 360 | 361 | def verify_meta_url(self, t): 362 | ret = (bro_intel_indicator_return.WARNING, 'Invalid url - %s' % (t)) 363 | if self.__is_ignore_field(t): 364 | ret = (bro_intel_indicator_return.OKAY, None) 365 | elif len(t) > 1 and self.__verify_chars(t): 366 | ret = (bro_intel_indicator_return.OKAY, None) 367 | return ret 368 | 369 | def verify_meta_whitelist(self, t): 370 | ret = (bro_intel_indicator_return.OKAY, 'Invalid whitelist - %s' % (t)) 371 | if self.__is_ignore_field(t): 372 | ret = (bro_intel_indicator_return.OKAY, None) 373 | elif len(t) > 1 and self.__verify_chars(t): 374 | ret = (bro_intel_indicator_return.OKAY, None) 375 | return ret 376 | 377 | def verify_meta_severity(self, t): 378 | ret = (bro_intel_indicator_return.ERROR, 'Invalid severity - %s (valid: 1-10)' % (t)) 379 | try: 380 | t_int = int(t) 381 | if isinstance(t_int, (int, long)) and (t_int > 0 and t_int < 10): 382 | ret = (bro_intel_indicator_return.OKAY, None) 383 | except ValueError: 384 | ret = (bro_intel_indicator_return.ERROR, 'Invalid severity - %s (valid: 1-10)' % (t)) 385 | return ret 386 | 387 | def verify_meta_cif_severity(self, t): 388 | VALID_SEVERITY = ['-', 'low', 'medium', 'med', 'high'] 389 | ret = (bro_intel_indicator_return.ERROR, 'Invalid cif_severity - %s (valid: %s)' % (t, ','.join(VALID_SEVERITY))) 390 | if t in VALID_SEVERITY: 391 | ret = (bro_intel_indicator_return.OKAY, None) 392 | return ret 393 | 394 | def verify_meta_cif_impact(self, t): 395 | ret = (bro_intel_indicator_return.WARNING, 'Invalid cif_impact - %s' % (t)) 396 | if self.__is_ignore_field(t): 397 | ret = (bro_intel_indicator_return.OKAY, None) 398 | elif len(t) > 1 and self.__verify_chars(t): 399 | ret = (bro_intel_indicator_return.OKAY, None) 400 | return ret 401 | 402 | def default(self, t): 403 | ret = (bro_intel_indicator_return.WARNING, 'Invalid - %s' % (t)) 404 | write_stderr("Running default handler for: %s" % (t)) 405 | if self.__is_ignore_field(t): 406 | ret = (bro_intel_indicator_return.OKAY, None) 407 | elif len(t) > 1 and self.__verify_chars(t): 408 | ret = (bro_intel_indicator_return.OKAY, None) 409 | return ret 410 | 411 | 412 | ############################################################################### 413 | # class bro_intel_feed_verifier 414 | # 415 | # This is the control class for Bro Intel Feed verification 416 | # 417 | class bro_intel_feed_verifier: 418 | stock_required_fields = ['indicator', 419 | 'indicator_type', 420 | 'meta.source'] 421 | psled_required_fields = ['indicator', 422 | 'indicator_type', 423 | 'meta.source', 424 | 'meta.desc'] 425 | field_header_designator = '#fields' 426 | feed_rx = r'([\S]+)' 427 | feed_sep_rx = r'(\t)+' 428 | 429 | header_fields = [] 430 | 431 | def __init__(self, options): 432 | self.feed_file = options.feed_file 433 | self.psled = options.psled 434 | self.__feed_header_found = False 435 | self.__num_of_fields = 0 436 | self.required_fields = bro_intel_feed_verifier.stock_required_fields 437 | self.warn_only = options.warn_only 438 | 439 | if self.psled is not None: 440 | self.required_fields = bro_intel_feed_verifier.psled_required_fields 441 | 442 | def __make_one_indexed(self, l): 443 | return map(lambda x: x+1, l) 444 | 445 | def __is_start_of_feed(self, l): 446 | ret = False 447 | if len(l) >= 2: 448 | if l[0] == self.field_header_designator: 449 | ret = True 450 | return ret 451 | 452 | def __are_header_fields_valid(self, l): 453 | ret = False 454 | _fields_found = [] 455 | if l[0] == self.field_header_designator: 456 | for index, item in enumerate(l): 457 | if index == 0: 458 | continue 459 | if item in self.required_fields: 460 | _fields_found.append(item) 461 | self.header_fields.append(item) 462 | 463 | t_list_diff = list(set(self.required_fields) - set(_fields_found)) 464 | if len(t_list_diff) == 0: 465 | ret = True 466 | else: 467 | warning_line(0, 'Fields missing: %s' % (','.join(t_list_diff))) 468 | return ret 469 | 470 | def __count_fields(self, l): 471 | return (len(l) - 1) 472 | 473 | ## 474 | # <0 - Too few fields 475 | # 0 - Proper field count 476 | # >0 - Too many fields 477 | ## 478 | def __verify_field_count(self, l): 479 | return (len(l) - self.__num_of_fields) 480 | 481 | def __verify_non_space(self, offset, l): 482 | ret = True 483 | 484 | r = [i for i, x in enumerate(l) if x == ' '] 485 | if len(r) > 0: 486 | warning_line(offset, 'Invalid empty field, offset %s' % (self.__make_one_indexed(r))) 487 | ret = False 488 | return ret 489 | 490 | def __get_field_contents(self, l): 491 | return l.split('\t') 492 | 493 | def __verify_field_sep(self, offset, l, is_header=False): 494 | ret = True 495 | field_seps = re.findall(self.feed_sep_rx, l, re.IGNORECASE) 496 | __field_total = self.__num_of_fields 497 | 498 | if is_header: 499 | __field_total += 1 500 | 501 | if len(field_seps) >= __field_total: 502 | warning_line(offset, 'Excess field separators found') 503 | ret = False 504 | 505 | for index, item in enumerate(field_seps): 506 | for s in item: 507 | if s != '\t': 508 | warning_line(offset, 'Field separator incorrect in field offset %d' % (self.__make_one_indexed(index))) 509 | ret = False 510 | return ret 511 | 512 | def __verify_header(self, index, l): 513 | ret = False 514 | contents = self.__get_field_contents(l) 515 | if self.__is_start_of_feed(contents) and self.__are_header_fields_valid(contents): 516 | if not self.__feed_header_found: 517 | self.__num_of_fields = self.__count_fields(contents) 518 | if self.__verify_field_sep(index, l, is_header=True): 519 | ret = True 520 | self.__feed_header_found = True 521 | else: 522 | write_stderr("Invalid field separator found in header. Must be a tab.") 523 | else: 524 | warning_line(index, "Duplicate header found") 525 | return ret 526 | 527 | def __verify_fields(self, index, content): 528 | ret = (bro_intel_indicator_return.OKAY, None) 529 | reason = '' 530 | _fields_to_process = {} 531 | validator = bro_data_intel_field_values() 532 | 533 | # 534 | # Not thrilled about this, but we need it to pull out correlatable fields 535 | # since, order of the actual feed fields aren't guaranteed. Ugly for now, 536 | # but workable and can likely be optimized shortly. 537 | # 538 | for content_index, t in enumerate(content): 539 | _fields_to_process[self.header_fields[content_index]] = t 540 | 541 | for k in _fields_to_process: 542 | ret = validator.get_verifier(k)(_fields_to_process[k]) 543 | 544 | if len(ret) > 0 and ret[0] != bro_intel_indicator_return.OKAY: 545 | if all(ord(l) > 31 and ord(l) < 127 and l in string.printable for l in k): 546 | t_line = str(_fields_to_process[k]) 547 | t_line = hex_escape(t_line) 548 | warning_line(index, 'Invalid entry \"%s\" for column \"%s\"' % (str(t_line), str(k))) 549 | else: 550 | warning_line(index, 'Unprintable character found for column \"%s\"' % (str(k))) 551 | break 552 | 553 | if ret: 554 | # Special case to verify indicator with indicator_type 555 | c = validator.correlate_indictor_and_indicator_type(_fields_to_process['indicator'], 556 | _fields_to_process['indicator_type']) 557 | 558 | if c is not None: 559 | if c[0] == bro_intel_indicator_return.WARNING: 560 | warning_line(index, 'Indicator type \"%s\" possible issue with indicator: \"%s\"' % (_fields_to_process['indicator_type'], _fields_to_process['indicator'])) 561 | elif c[0] == bro_intel_indicator_return.ERROR: 562 | error_line(index, 'Indicator type \"%s\" possible issue with indicator: \"%s\"' % (_fields_to_process['indicator_type'], _fields_to_process['indicator'])) 563 | ret = c 564 | return ret 565 | 566 | def __verify_entry(self, index, l): 567 | ret = (bro_intel_indicator_return.ERROR, '') 568 | contents = self.__get_field_contents(l) 569 | _content_field_count = self.__verify_field_count(contents) 570 | _warn_str = None 571 | 572 | if _content_field_count == 0: 573 | if self.__verify_field_sep(index, l) and self.__verify_non_space(index, contents): 574 | ret = self.__verify_fields(index, contents) 575 | elif _content_field_count > 0: 576 | ret = (bro_intel_indicator_return.ERROR, 'Invalid number of fields - Found: %d, Header Fields: %d - Look for: EXTRA fields or tab seperators' % (len(contents), self.__num_of_fields)) 577 | elif _content_field_count < 0: 578 | ret = (bro_intel_indicator_return.ERROR, 'Invalid number of fields - Found: %d, Header Fields: %d - Look for: EMPTY fields' % (len(contents), self.__num_of_fields)) 579 | return ret 580 | 581 | def __load_feed(self, feed): 582 | with open(feed) as f: 583 | for line in f: 584 | t_line = line.rstrip('\r\n') 585 | if len(t_line): 586 | yield t_line 587 | 588 | def __handle_reporting(self, index, c): 589 | if c is not None: 590 | if c[0] == bro_intel_indicator_return.ERROR: 591 | error_line(index, 'Details - %s' % (c[1])) 592 | 593 | elif c[0] == bro_intel_indicator_return.WARNING: 594 | warning_line(index, c[1]) 595 | 596 | def verify(self): 597 | for index, l in enumerate(self.__load_feed(self.feed_file)): 598 | # Check the header 599 | if index == 0: 600 | if not self.__verify_header(index, l): 601 | error_line(index, "Invalid header") 602 | sys.exit(2) 603 | else: 604 | t_ret = self.__verify_entry(index, l) 605 | if t_ret[0] != bro_intel_indicator_return.OKAY: 606 | self.__handle_reporting(index, t_ret) 607 | 608 | if t_ret[0] == bro_intel_indicator_return.ERROR and self.warn_only is None: 609 | sys.exit(3) 610 | 611 | 612 | ############################################################################### 613 | # main() 614 | ############################################################################### 615 | def main(): 616 | parser = OptionParser() 617 | parser.add_option('-f', '--file', dest='feed_file', help='Bro Intel Feed to Verify') 618 | parser.add_option('--psled', action='store_true', dest='psled', help='Verify Intel meets PacketSled requirements') 619 | parser.add_option('--warn-only', action='store_true', dest='warn_only', help='Warn ONLY on errors, continue processing and report') 620 | (options, args) = parser.parse_args() 621 | 622 | if len(sys.argv) < 2: 623 | parser.print_help() 624 | sys.exit(1) 625 | 626 | bifv = bro_intel_feed_verifier(options) 627 | bifv.verify() 628 | 629 | 630 | ############################################################################### 631 | # __name__ checking 632 | ############################################################################### 633 | if __name__ == '__main__': 634 | main() 635 | -------------------------------------------------------------------------------- /tests/t.intel: -------------------------------------------------------------------------------- 1 | #fields indicator indicator_type meta.source meta.desc meta.url meta.do_notice meta.if_in meta.whitelist meta.severity 2 | my.domain.com Intel::DOMAIN domain DOMAIN - F - - 6 3 | your.com Intel::DOMAIN domain DOMAIN - F - - 6 4 | other.domain.com Intel::DOMAIN domain DOMAIN - F - - 6 5 | 6 | -------------------------------------------------------------------------------- /tests/t.intel-addr: -------------------------------------------------------------------------------- 1 | #fields indicator indicator_type meta.source meta.desc meta.url meta.do_notice meta.if_in meta.whitelist meta.severity 2 | 192.168.1.1 Intel::ADDR my imagination ADDR - F - - 6 3 | 192.168.1.2 Intel::ADDR my imagination ADDR - F - - 6 4 | 192.168.1.3 Intel::ADDR my imagination ADDR - F - - 6 5 | 6 | -------------------------------------------------------------------------------- /tests/t.intel-bad-addr: -------------------------------------------------------------------------------- 1 | #fields indicator indicator_type meta.source meta.desc meta.url meta.do_notice meta.if_in meta.whitelist meta.severity 2 | 192.168.1.1 Intel::ADDR my imagination ADDR - F - - 6 3 | 192.168.1.2 Intel::ADDR my imagination ADDR - F - - 6 4 | 192.168.1.300 Intel::ADDR my imagination ADDR - F - - 6 5 | 6 | -------------------------------------------------------------------------------- /tests/t.intel-bad-blank-field: -------------------------------------------------------------------------------- 1 | #fields indicator indicator_type meta.source meta.desc meta.url meta.do_notice meta.if_in meta.whitelist meta.severity 2 | my.domain.com Intel::DOMAIN domain DOMAIN - F - - 6 3 | your.com Intel::DOMAIN domainn DOMAIN - F - 6 4 | other.domain.com Intel::DOMAIN domain DOMAIN - F - - 6 5 | 6 | -------------------------------------------------------------------------------- /tests/t.intel-bad-character: -------------------------------------------------------------------------------- 1 | #fields indicator indicator_type meta.source meta.url meta.do_notice meta.if_in meta.whitelist meta.desc 2 | 192.168.1.1 Intel::ADDR Some kind of intelligent system - F - - Router 3 | 192.168.1.2 Intel::ADDR Some kind of intelligent system - F _ - Firewall 4 | 192.168.1.3 Intel::ADDR Some kind of intelligent system - F - - You don't want to know 5 | 6 | -------------------------------------------------------------------------------- /tests/t.intel-bad-field-sep: -------------------------------------------------------------------------------- 1 | #fields indicator indicator_type meta.source meta.desc meta.url meta.do_notice meta.if_in meta.whitelist meta.severity 2 | my.domain.com Intel::DOMAIN domain DOMAIN - F - - 6 3 | your.com Intel::DOMAIN domainn DOMAIN - F - - 6 4 | other.domain.com Intel::DOMAIN domain DOMAIN - F - - 6 5 | 6 | -------------------------------------------------------------------------------- /tests/t.intel-bad-indicator-type: -------------------------------------------------------------------------------- 1 | #fields indicator indicator_type meta.source meta.desc meta.url meta.do_notice meta.if_in meta.whitelist meta.severity 2 | my.domain.com Intel::DOMAIN domain DOMAIN - F - - 6 3 | your.com Intel::DONAIN domainn DOMAIN - F - - 6 4 | other.domain.com Intel::DOMAIN domain DOMAIN - F - - 6 5 | 6 | -------------------------------------------------------------------------------- /tests/t.intel-bad-tab: -------------------------------------------------------------------------------- 1 | #fields indicator indicator_type meta.source meta.desc meta.url meta.do_notice meta.if_in meta.whitelist meta.severity 2 | my.domain.com Intel::DOMAIN domain DOMAIN - F - - 6 3 | your.com Intel::DOMAIN domain DOMAIN - F - - 6 4 | other.domain.com Intel::DOMAIN domain DOMAIN - F - - 6 5 | 6 | -------------------------------------------------------------------------------- /tests/t.intel-default-handler: -------------------------------------------------------------------------------- 1 | #fields indicator indicator_type meta.source meta.desc meta.url meta.do_notice meta.if_in meta.whitelist meta.severity meta.bob 2 | my.domain.com Intel::DOMAIN domain DOMAIN - F - - 6 bob 3 | your.com Intel::DOMAIN domain DOMAIN - F - - 6 bob 4 | other.domain.com Intel::DOMAIN domain DOMAIN - F - - 6 bob 5 | 6 | -------------------------------------------------------------------------------- /tests/t.intel-email: -------------------------------------------------------------------------------- 1 | #fields indicator indicator_type meta.source meta.desc meta.url meta.do_notice meta.if_in meta.whitelist meta.severity 2 | my.domain.com Intel::DOMAIN domain DOMAIN - F - - 6 3 | your.com Intel::DOMAIN domain DOMAIN - F - - 6 4 | other.domain.com Intel::DOMAIN domain DOMAIN - F - - 6 5 | bob@hi.org Intel::EMAIL email EMAIL - F - - 6 6 | broken.org Intel::EMAIL email EMAIL - F - - 6 7 | 8 | -------------------------------------------------------------------------------- /tests/t.intel-missing-header: -------------------------------------------------------------------------------- 1 | #fields indicator indicator_type meta.source meta.desc meta.url meta.do_notice meta.if_in meta.whitelist meta.severity 2 | my.domain.com Intel::DOMAIN domain DOMAIN - F - - 6 3 | your.com Intel::DOMAIN domain DOMAIN - F - - 6 4 | other.domain.com Intel::DOMAIN domain DOMAIN - F - - 6 5 | 6 | -------------------------------------------------------------------------------- /tests/t.intel-too-few-fields: -------------------------------------------------------------------------------- 1 | #fields indicator indicator_type meta.source meta.desc meta.url meta.do_notice meta.if_in meta.whitelist meta.severity 2 | my.domain.com Intel::DOMAIN domain DOMAIN - F - - 6 3 | your.com Intel::DOMAIN domain DOMAIN F - - 6 4 | other.domain.com Intel::DOMAIN domain DOMAIN - F - - 6 5 | 6 | -------------------------------------------------------------------------------- /tests/t.intel-too-many-fields: -------------------------------------------------------------------------------- 1 | #fields indicator indicator_type meta.source meta.desc meta.url meta.do_notice meta.if_in meta.whitelist meta.severity 2 | my.domain.com Intel::DOMAIN domain DOMAIN - F - - 6 3 | your.com Intel::DOMAIN domain DOMAIN - F - - 6 4 | other.domain.com Intel::DOMAIN domain DOMAIN - F - - 6 5 | 6 | --------------------------------------------------------------------------------