├── .github ├── dependabot.yml └── workflows │ └── make-release.yaml ├── .gitignore ├── .travis.yml ├── LICENSE ├── README.md ├── forklift.py ├── requirements.txt ├── shiploader.py ├── vulndb.py ├── vulnpryer.conf.sample └── vulnpryer.py /.github/dependabot.yml: -------------------------------------------------------------------------------- 1 | # To get started with Dependabot version updates, you'll need to specify which 2 | # package ecosystems to update and where the package manifests are located. 3 | # Please see the documentation for all configuration options: 4 | # https://help.github.com/github/administering-a-repository/configuration-options-for-dependency-updates 5 | 6 | version: 2 7 | updates: 8 | - package-ecosystem: "pip" # See documentation for possible values 9 | directory: "/" # Location of package manifests 10 | schedule: 11 | interval: "daily" 12 | -------------------------------------------------------------------------------- /.github/workflows/make-release.yaml: -------------------------------------------------------------------------------- 1 | on: 2 | push: 3 | # Sequence of patterns matched against refs/tags 4 | tags: 5 | - 'v*' # Push events to matching v*, i.e. v1.0, v20.15.10 6 | 7 | name: Create Release 8 | 9 | jobs: 10 | build: 11 | name: Create Release 12 | runs-on: ubuntu-latest 13 | steps: 14 | - name: Checkout code 15 | uses: actions/checkout@v2 16 | - name: Create Release 17 | id: create_release 18 | uses: softprops/action-gh-release@v0.1.14 19 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | 5 | # C extensions 6 | *.so 7 | 8 | # Distribution / packaging 9 | .Python 10 | env/ 11 | bin/ 12 | build/ 13 | develop-eggs/ 14 | dist/ 15 | eggs/ 16 | lib/ 17 | lib64/ 18 | parts/ 19 | sdist/ 20 | var/ 21 | *.egg-info/ 22 | .installed.cfg 23 | *.egg 24 | 25 | # Installer logs 26 | pip-log.txt 27 | pip-delete-this-directory.txt 28 | 29 | # Unit test / coverage reports 30 | htmlcov/ 31 | .tox/ 32 | .coverage 33 | .cache 34 | nosetests.xml 35 | coverage.xml 36 | 37 | # Translations 38 | *.mo 39 | 40 | # Mr Developer 41 | .mr.developer.cfg 42 | .project 43 | .pydevproject 44 | 45 | # Rope 46 | .ropeproject 47 | 48 | # Django stuff: 49 | *.log 50 | *.pot 51 | 52 | # Sphinx documentation 53 | docs/_build/ 54 | 55 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | language: python 2 | python: 3 | - '2.6' 4 | - '2.7' 5 | install: 6 | - pip install flake8 --use-mirrors 7 | - pip install -r requirements.txt 8 | before_script: 9 | - flake8 . 10 | script: nosetests 11 | notifications: 12 | slack: 13 | secure: VpvXEX0d/vzCPfMYVGezTZ++eKjSzP06QnDHwMt0ppuk+2sO0ctiEjtNtTLCL4dSCl/TaIN2i1vdFDOSgtWnGh3bE2UQEm+3NO/UKtme9o+cnG6wpkgGlKvhpTbEpjWIF4/dNn36cyN5+EiNtEnOxzkuvDM0A7fQlXawie36wPM= 14 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2015 David F. Severski 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | [![Build Status](https://secure.travis-ci.org/davvidski/VulnPryer.png)](http://travis-ci.org/davidski/VulnPryer) 2 | 3 | VulnPryer 4 | ========= 5 | 6 | Vulnerability Pryer - Pries more context into your vulnerability data. 7 | 8 | # Description 9 | 10 | VulnPryer is the code behind a [vulnerability reprioritization project](https://blog.severski.net/2014/08/27/2014-08-27-introducing-vulnpryer/). 11 | Using a vulnerability data feed (VulnPryer uses the VulnDB commercial project by default), VulnPryer will 12 | download that feed on an incremental basis, load the feed into MongoDB for storage, extract a 13 | mapping of features, and provide a remapping of vulnerabilities to custom severities for importing 14 | into your analysis product of choice (VulnPryer targets the [RedSeal](https://www.redseal.net/) platform by default). 15 | 16 | # Installation 17 | 18 | VulnPryer may be set up the hard (manual) way and the easy (automated) way. 19 | 20 | ## Manual Installation 21 | 1. Setup an instance of MongoDB (authentication not currently supported) 22 | 2. git clone https://github.com/davidski/VulnPryer vulnpryer 23 | 3. cd ./vulnpryer 24 | 4. pip install -r requirements 25 | 5. cp vulnpryer.conf{.sample,} 26 | 6. vi vulnpryer.conf #modify with your settings and credentials. 27 | 28 | # Usage 29 | 30 | VulnPryer targets running daily extracts out of VulnDB and generating updated RedSeal Threat 31 | Reference Library files with modified CVSS ratings on an Amazon S3 bucket. This is accomplished 32 | via the `vulndb` module for working with the VulnDB API, the `shiploader` module for loading that 33 | data into MongoDB and creating feature extracts, and the `forklift` module for taking the feature file and 34 | applying a custom formula for creating vulnerability severities and generating TRL files. 35 | 36 | The simplest means is to run the `vulnpryer.py` wrapper script. If you want to replace indlvidual 37 | modules (e.g. to use a different prioritization scheme, import a different vulnerability data feed), 38 | you can run the individual compoents manually: 39 | 40 | 1. vulndb.py 41 | 2. shiploader.py 42 | 3. forklift.py 43 | 44 | # Dependencies 45 | VulnPryer relies on the following third-party libraries. Note that newer versions of these libraries may be available, but have not been tested. 46 | 47 | ``` 48 | argparse >= 1.2.1 [http://code.google.com/p/argparse/ - Now part of Python, version 2.7, 3.2, or higher] 49 | boto >= 2.32.1 [https://github.com/boto/boto] 50 | filechunkio >= 1.5 [https://bitbucket.org/fabian/filechunkio] 51 | lxml >= 3.3.5 [http://lxml.de/] 52 | oauth2 >= 1.5.211 [http://oauth.net/2/] 53 | pandas >= 0.13.1 [http://pandas.pydata.org/] 54 | pymongo >= 2.7.2 [http://api.mongodb.org/python/current/] 55 | restkit >= 4.2.2 [http://restkit.org/] 56 | simplejson >= 3.6.2 [https://pypi.python.org/pypi/simplejson/] 57 | ``` 58 | 59 | # Acknowledgements 60 | VulnPryer would not exist without the inspiration and assistance of the following individuals 61 | and organizations: 62 | 63 | - [@alexcpsec](https://twitter.com/alexcpsec) and 64 | [@kylemaxwell](https://twitter.com/alexcpsec) for the 65 | [combine](https://github.com/mlsecproject/combine) project. VulnPryer has cribbed heavily from 66 | that design pattern, including a crude aping of naming metaphors. :grin: 67 | - [Risk Based Security](https://vulndb.cyberriskanalytics.com/) (RBS) 68 | for providing the VulnDB product and for the support in getting this project 69 | off the ground. 70 | - [Kenna Security](https://www.kennasecurity.com/) for providing the inspiration 71 | on this project and their continued support of the community. 72 | - [RedSeal](https://www.redseal.net) for providing the analysis platform for network 73 | security posture review and analysis. 74 | -------------------------------------------------------------------------------- /forklift.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | import ConfigParser 4 | from lxml import objectify 5 | import gzip 6 | import urllib2 7 | import base64 8 | import os 9 | from lxml import etree 10 | import pandas as pd 11 | import logging 12 | import tempfile 13 | import re 14 | 15 | logger = logging.getLogger('vulnpryer.forklift') 16 | 17 | config = ConfigParser.ConfigParser() 18 | config.read('vulnpryer.conf') 19 | 20 | trl_source_url = config.get('RedSeal', 'trl_url') 21 | username = config.get('RedSeal', 'username') 22 | password = config.get('RedSeal', 'password') 23 | temp_directory = config.get('VulnDB', 'working_dir') 24 | s3_bucket = config.get('S3', 'bucket_name') 25 | s3_region = config.get('S3', 'region') 26 | s3_key = config.get('S3', 'key') 27 | 28 | 29 | class HeadRequest(urllib2.Request): 30 | def get_method(self): 31 | return "HEAD" 32 | 33 | 34 | def _read_trl(trl_location): 35 | """Read and Import TRL""" 36 | 37 | parsed = objectify.parse(gzip.open(trl_location)) 38 | root = parsed.getroot() 39 | logger.info('Finished reading TRL') 40 | 41 | return root 42 | 43 | 44 | def get_trl(trl_path): 45 | """Getch the TRL from RedSeal""" 46 | 47 | req = urllib2.Request(trl_source_url) 48 | base64str = base64.encodestring('%s:%s' % (username, 49 | password)).replace('\n', '') 50 | req.add_header("Authorization", "Basic %s" % base64str) 51 | result = urllib2.urlopen(req) 52 | 53 | with open(trl_path, "wb") as local_file: 54 | local_file.write(result.read()) 55 | local_file.close() 56 | 57 | logger.info('Downloaded TRL from RedSeal') 58 | 59 | 60 | def _read_vulndb_extract(): 61 | """read in the extracted VulnDB data""" 62 | vulndb = pd.read_csv(temp_directory + 'vulndb_export.csv') 63 | return vulndb 64 | 65 | 66 | def _remap_trl(trl_data, vulndb): 67 | """Rectify CVSS Values""" 68 | 69 | avg_cvss_score = 6.2 70 | msp_factor = 2.5 71 | edb_factor = 1.5 72 | private_exploit_factor = .5 73 | network_vector_factor = 2 74 | impact_factor = 3 75 | 76 | for vulnerability in trl_data.vulnerabilities.vulnerability: 77 | 78 | logger.debug('Adjusting priority of {}'.format( 79 | vulnerability.get('cveID'))) 80 | 81 | # start off with the NVD definition 82 | modified_score = float(vulnerability.get('CVSSTemporalScore')) 83 | # add deviation from mean 84 | modified_score = modified_score + (modified_score - 85 | avg_cvss_score) / avg_cvss_score 86 | # adjust up if metasploit module exists 87 | if vulndb[vulndb['CVE_ID'] == 88 | vulnerability.get('cveID')].msp.any >= 1: 89 | modified_score = modified_score + msp_factor 90 | # adjust up if exploit DB entry exists 91 | if vulndb[vulndb['CVE_ID'] == 92 | vulnerability.get('cveID')].edb.any >= 1: 93 | modified_score = modified_score + edb_factor 94 | # adjust up if a private exploit is known 95 | if vulndb[vulndb['CVE_ID'] == 96 | vulnerability.get('cveID')].private_exploit.any >= 1: 97 | modified_score = modified_score + private_exploit_factor 98 | else: 99 | modified_score = modified_score - private_exploit_factor 100 | # adjust down for impacts that aren't relevant to our loss scenario 101 | if (vulndb[vulndb['CVE_ID'] == 102 | vulnerability.get('cveID')].impact_integrity.any < 1 and 103 | vulndb[vulndb['CVE_ID'] == 104 | vulnerability.get('cveID')].impact_confidentiality.any < 1): 105 | modified_score = modified_score - impact_factor 106 | # adjust down for attack vectors that aren't in our loss scenario 107 | if vulndb[vulndb['CVE_ID'] == 108 | vulnerability.get('cveID')].network_vector.any < 1: 109 | modified_score = modified_score - network_vector_factor 110 | # confirm that our modified score is within max/min limits 111 | if modified_score > 10: 112 | modified_score = 10 113 | if modified_score < 0: 114 | modified_score = 0 115 | # set the modified score 116 | vulnerability.set('CVSSTemporalScore', str(modified_score)) 117 | logger.debug('Completed adjustments to TRL.') 118 | return trl_data 119 | 120 | 121 | def _write_trl(trl_data, modified_trl_path): 122 | """Write the modified trl out to disk""" 123 | # etree.cleanup_namespaces(trl) 124 | obj_xml = etree.tostring(trl_data, xml_declaration=True, 125 | pretty_print=True, encoding='UTF-8') 126 | with gzip.open(modified_trl_path, "wb") as f: 127 | f.write(obj_xml) 128 | 129 | 130 | def _fixup_trl(modified_trl_path): 131 | """Fix attribute order for trl node which RS 7.x is particular about""" 132 | temp_file = tempfile.NamedTemporaryFile(delete=False) 133 | output_file = gzip.open(temp_file.name, "wb") 134 | reg_expression = '^$' 135 | reg_expression = re.compile(reg_expression) 136 | fh = gzip.open(modified_trl_path, "rb") 137 | for line in fh: 138 | line = re.sub(reg_expression, r'', line) 139 | output_file.write(line) 140 | fh.close() 141 | output_file.close() 142 | os.rename(temp_file.name, modified_trl_path) 143 | 144 | 145 | def modify_trl(original_trl): 146 | """public full trl modification script""" 147 | vulndb = _read_vulndb_extract() 148 | trl_data = _read_trl(original_trl) 149 | modified_trl_data = _remap_trl(trl_data, vulndb) 150 | 151 | new_trl_path = os.path.dirname(original_trl) + '/modified_trl.gz' 152 | _write_trl(modified_trl_data, new_trl_path) 153 | _fixup_trl(new_trl_path) 154 | return new_trl_path 155 | 156 | 157 | def post_trl(file_path): 158 | """store the TRL to S3""" 159 | 160 | from filechunkio import FileChunkIO 161 | import math 162 | import os 163 | import boto.s3 164 | conn = boto.s3.connect_to_region(s3_region) 165 | 166 | bucket = conn.get_bucket(s3_bucket, validate=False) 167 | 168 | logger.info('Uploading {} to Amazon S3 bucket {}'.format( 169 | file_path, s3_bucket)) 170 | 171 | import sys 172 | 173 | def percent_cb(complete, total): 174 | sys.stdout.write('.') 175 | sys.stdout.flush() 176 | 177 | source_size = os.stat(file_path).st_size 178 | chunk_size = 10000000 179 | chunk_count = int(math.ceil(source_size / chunk_size)) 180 | mp = bucket.initiate_multipart_upload(s3_key, encrypt_key=True, 181 | policy='public-read') 182 | for i in range(chunk_count + 1): 183 | offset = chunk_size * i 184 | bytes = min(chunk_size, source_size - offset) 185 | with FileChunkIO(file_path, 'r', offset=offset, bytes=bytes) as fp: 186 | mp.upload_part_from_file(fp, part_num=i + 1) 187 | mp.complete_upload() 188 | 189 | # old single part upload not used due to bug in boto with continuation 190 | # headers 191 | # from boto.s3.key import Key 192 | # k = Key(bucket) 193 | # k.key = key_name 194 | # k.set_contents_from_filename(file_path, cb=percent_cb, num_cb=10, 195 | # encrypt_key=True, policy='public-read') 196 | 197 | return 198 | 199 | if __name__ == "__main__": 200 | modify_trl('/tmp/trl.gz') 201 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | argparse>=1.2.1 2 | boto>=2.32.1 3 | filechunkio>=1.5 4 | lxml>=3.3.5 5 | oauth2>=1.5.211 6 | pandas>=0.13.1 7 | pymongo>=2.7.2 8 | restkit>=4.2.2 9 | simplejson>=3.6.2 -------------------------------------------------------------------------------- /shiploader.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | import ConfigParser 4 | from pymongo import MongoClient 5 | import csv 6 | import simplejson as json 7 | import sys 8 | import glob 9 | import logging 10 | import os 11 | 12 | logger = logging.getLogger('vulnpryer.shiploader') 13 | 14 | config = ConfigParser.ConfigParser() 15 | config.read('vulnpryer.conf') 16 | 17 | mongo_host = config.get('Mongo', 'hostname') 18 | temp_directory = config.get('VulnDB', 'working_dir') 19 | json_directory = config.get('VulnDB', 'json_dir') 20 | 21 | # connect to our MongoDB instance 22 | client = MongoClient(host=mongo_host) 23 | db = client.vulndb 24 | collection = db.osvdb 25 | 26 | 27 | def _decode_list(data): 28 | rv = [] 29 | for item in data: 30 | if isinstance(item, unicode): 31 | item = item.encode('utf-8') 32 | elif isinstance(item, list): 33 | item = _decode_list(item) 34 | elif isinstance(item, dict): 35 | item = _decode_dict(item) 36 | rv.append(item) 37 | return rv 38 | 39 | 40 | def _decode_dict(data): 41 | rv = {} 42 | for key, value in data.iteritems(): 43 | if isinstance(key, unicode): 44 | key = key.encode('utf-8') 45 | if isinstance(value, unicode): 46 | value = value.encode('utf-8') 47 | elif isinstance(value, list): 48 | value = _decode_list(value) 49 | elif isinstance(value, dict): 50 | value = _decode_dict(value) 51 | rv[key] = value 52 | return rv 53 | 54 | 55 | def load_mongo(json_glob_pattern): 56 | """Load a pattern of JSON files to Mongo""" 57 | path_to_json = json_directory + json_glob_pattern 58 | for filename in sorted(glob.glob(path_to_json), key=os.path.getmtime): 59 | logger.info("Working on: {}".format(filename)) 60 | json_data = open(filename).read() 61 | try: 62 | # auto-handling unicode object hook derived from 63 | # http://stackoverflow.com/questions/956867/how-to-get-string- 64 | # objects-instead-unicode-ones-from-json-in-python 65 | data = json.loads(json_data, object_hook=_decode_dict) 66 | except: 67 | print sys.argv[0], " Unexpected error:", sys.exc_info()[1] 68 | if data is None: 69 | continue 70 | for vulndb in data['results']: 71 | logger.debug(json.dumps(vulndb, sort_keys=True, indent=4 * ' ')) 72 | vulndb['_id'] = vulndb['osvdb_id'] 73 | osvdb_id = collection.save(vulndb) 74 | # osvdb_id = collection.insert(vulndb) 75 | logger.debug("Saved: {} with MongoDB id: {}".format( 76 | filename, osvdb_id)) 77 | logger.info("Mapping OSVDB entries to CVE IDs") 78 | _map_osvdb_to_cve() 79 | logger.info("Marking deprecated VUulnDB entries") 80 | _mark_deprecated_entries() 81 | 82 | 83 | def _map_osvdb_to_cve(): 84 | """Add CVE_ID field to all OSVDB entries""" 85 | results = db.osvdb.aggregate([ 86 | {"$unwind": "$ext_references"}, 87 | {"$match": {"ext_references.type": "CVE ID"}}, 88 | {"$project": {"CVE_ID": "$ext_references.value"}}, 89 | {"$group": {"_id": "$_id", "CVE_ID": {"$addToSet": "$CVE_ID"}}} 90 | ]) 91 | for entry in results['result']: 92 | db.osvdb.update({"_id": entry['_id']}, {"$set": 93 | {"CVE_ID": entry['CVE_ID']}}) 94 | logger.info("Adding CVEs to {}".format(entry['_id'])) 95 | 96 | 97 | def _mark_deprecated_entries(): 98 | """Mark deprecated entries as such""" 99 | logger.info("Marking deprecated entries based on title.") 100 | db.osvdb.update( 101 | {'title': {'$regex': '^DEPRECA'}}, 102 | {'$set': {'deprecated': True}}, 103 | upsert=False, multi=True 104 | ) 105 | 106 | 107 | def _run_aggregation(): 108 | """Set the classifications to a bogus array value if it's empty (size 0) 109 | this keeps the unwind from dropping empty classification documents 110 | alternate query based upon ext.references.type == 'CVE ID.' 111 | """ 112 | results = [] 113 | result_cursor = db.osvdb.aggregate([ 114 | {"$unwind": "$CVE_ID"}, 115 | {"$unwind": "$cvss_metrics"}, 116 | {"$project": {"CVE_ID": 1, "ext_references": 1, 117 | "cvss_score": 118 | "$cvss_metrics.calculated_cvss_base_score", 119 | "classifications": {"$cond": { 120 | "if": {"$eq": [{"$size": 121 | "$classifications"}, 0]}, 122 | "then": ["bogus"], 123 | "else": "$classifications"}}}}, 124 | {"$unwind": "$classifications"}, 125 | {"$unwind": "$ext_references"}, 126 | {"$group": { 127 | "_id": {"_id": "$_id", "CVE_ID": {"$concat": 128 | ["CVE-", "$CVE_ID"]}}, 129 | "public_exploit": {"$sum": {"$cond": [ 130 | {"$eq": ["$classifications.name", "exploit_public"]}, 1, 0]}}, 131 | "private_exploit": {"$sum": {"$cond": [ 132 | {"$eq": ["$classifications.name", "exploit_private"]}, 1, 0]}}, 133 | "cvss_score": {"$max": "$cvss_score"}, 134 | "msp": {"$sum": {"$cond": [{"$eq": ["$ext_references.type", 135 | "Metasploit URL"]}, 1, 0]}}, 136 | "edb": {"$sum": {"$cond": [{"$eq": ["$ext_references.type", 137 | "Exploit Database"]}, 1, 0]}}, 138 | "network_vector": {"$sum": {"$cond": [{"$eq": [ 139 | "$classifications.name", "location_remote"]}, 1, 0]}}, 140 | "impact_integrity": {"$sum": {"$cond": [ 141 | {"$eq": ["$classifications.name", 142 | "impact_integrity"]}, 1, 0]}}, 143 | "impact_confidential": {"$sum": {"$cond": [ 144 | {"$eq": ["$classifications.name", 145 | "impact_confidential"]}, 1, 0]}}}}, 146 | {"$project": {"_id": 0, "OSVDB": "$_id._id", 147 | "CVE_ID": "$_id.CVE_ID", 148 | "public_exploit": 1, "private_exploit": 1, 149 | "cvss_score": 1, "msp": 1, "edb": 1, 150 | "network_vector": 1, "impact_integrity": 1, 151 | "impact_confidentiality": "$impact_confidential"}} 152 | ], cursor={} 153 | ) 154 | # comment out {"$match": {"network_vector": {"$gt": 0}}} 155 | 156 | for doc in result_cursor: 157 | results.append(doc) 158 | 159 | logger.info("There are {} entries in this aggregation.".format( 160 | len(results))) 161 | logger.debug("The headers are: {}".format(results[0].keys())) 162 | return results 163 | 164 | 165 | def _calculate_mean_cvss(): 166 | """Calcuate the mean CVSS score across all known vulnerabilities""" 167 | results = db.osvdb.aggregate([ 168 | {"$unwind": "$cvss_metrics"}, 169 | {"$group": { 170 | "_id": "null", 171 | "avgCVSS": {"$avg": "$cvss_metrics.calculated_cvss_base_score"} 172 | }} 173 | ]) 174 | logger.info("There are {} entries in this aggregation.".format( 175 | len(results['result']))) 176 | logger.debug("The headers are: {}".format(results['result'][0].keys())) 177 | try: 178 | avgCVSS = results['result'][0]['avgCVSS'] 179 | except: 180 | avgCVSS = None 181 | return avgCVSS 182 | 183 | 184 | class _DictUnicodeProxy(object): 185 | """Create helper function for writing unicode to CSV""" 186 | def __init__(self, d): 187 | self.d = d 188 | 189 | def __iter__(self): 190 | return self.d.__iter__() 191 | 192 | def get(self, item, default=None): 193 | i = self.d.get(item, default) 194 | if isinstance(i, list): 195 | i = i[0] 196 | if isinstance(i, unicode): 197 | return i.encode('utf-8') 198 | return i 199 | 200 | 201 | def _write_vulndb(results, filename): 202 | """Dump output to CSV""" 203 | csvfile = open(filename, 'wb') 204 | 205 | # headers = ['CVE_ID', 'OSVDB', 'public_exploit', 'private_exploit', 206 | # 'cvss_score', 'msp', 'edb', 'network_vector', 'impact_integrity', 207 | # 'impact_confidentialit', 'network_vector'] 208 | headers = results[0].keys() 209 | csvwriter = csv.DictWriter(csvfile, fieldnames=headers) 210 | csvwriter.writeheader() 211 | for result in results: 212 | csvwriter.writerow(_DictUnicodeProxy(result)) 213 | 214 | csvfile.close() 215 | 216 | 217 | def get_extract(extract_file): 218 | results = _run_aggregation() 219 | _write_vulndb(results, extract_file) 220 | 221 | if __name__ == "__main__": 222 | """Read in all the json files""" 223 | load_mongo("data_*.json") 224 | get_extract(temp_directory + 'vulndb_export.csv') 225 | -------------------------------------------------------------------------------- /vulndb.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | from restkit import OAuthFilter, request 4 | import simplejson as json 5 | import oauth2 6 | from datetime import date, timedelta 7 | import logging 8 | import ConfigParser 9 | 10 | logger = logging.getLogger('vulnpryer.vulndb') 11 | 12 | config = ConfigParser.ConfigParser() 13 | config.read('vulnpryer.conf') 14 | 15 | consumer_key = config.get('VulnDB', 'consumer_key') 16 | consumer_secret = config.get('VulnDB', 'consumer_secret') 17 | request_token_url = config.get('VulnDB', 'request_token_url') 18 | temp_directory = config.get('VulnDB', 'working_dir') 19 | json_directory = config.get('VulnDB', 'json_dir') 20 | page_size = int(config.get('VulnDB', 'page_size')) 21 | 22 | 23 | def _fetch_data(from_date, to_date, page_size=20, first_page=1): 24 | """Fetch a chunk of vulndb""" 25 | 26 | from_date = from_date.strftime("%Y-%m-%d") 27 | to_date = to_date.strftime("%Y-%m-%d") 28 | 29 | logger.info("Working on date range: {} - {}".format(from_date, to_date)) 30 | 31 | consumer = oauth2.Consumer(key=consumer_key, secret=consumer_secret) 32 | # client = oauth2.Client(consumer) 33 | 34 | # now get our request token 35 | auth = OAuthFilter('*', consumer) 36 | 37 | # initialize the page counter either at the first page or whatever page 38 | # was requested 39 | page_counter = first_page 40 | 41 | finished = False 42 | reply = dict() 43 | reply['results'] = [] 44 | 45 | while not finished: 46 | url = 'https://vulndb.cyberriskanalytics.com' + \ 47 | '/api/v1/vulnerabilities/find_by_date?' + \ 48 | 'start_date=' + from_date + '&end_date=' + to_date + '&page=' + \ 49 | str(page_counter) + '&size=' + str(page_size) + \ 50 | '&date_type=updated_on' + \ 51 | '&nested=true' 52 | logger.debug("Working on url: {} ".format(url)) 53 | 54 | resp = request(url, filters=[auth]) 55 | if resp.status_int == 404: 56 | logger.warning("Could not find anything for the week " + 57 | "begining: {}".format(from_date)) 58 | return 59 | if resp.status_int != 200: 60 | raise Exception("Invalid response {}.".format(resp['status'])) 61 | 62 | logger.debug("\tHTTP Response code: " + str(resp.status_int)) 63 | 64 | """parse response and append to working set""" 65 | page_reply = json.loads(resp.body_string()) 66 | logger.debug("Retrieving page {} of {}.".format(page_counter, 67 | -(-page_reply['total_entries'] // page_size))) 68 | 69 | if len(page_reply['results']) < page_size: 70 | finished = True 71 | reply['results'].extend(page_reply['results']) 72 | reply['total_entries'] = page_reply['total_entries'] 73 | else: 74 | page_counter += 1 75 | reply['results'].extend(page_reply['results']) 76 | 77 | logger.info("Returning {} out of {} results".format(str(len( 78 | reply['results'])), str(reply['total_entries']))) 79 | return reply 80 | 81 | 82 | def query_vulndb(from_date, to_date, day_interval=1): 83 | """Query RBS's VulnDB for a chunk of data""" 84 | 85 | from dateutil.parser import parse 86 | import io 87 | 88 | if not isinstance(from_date, date): 89 | from_date = parse(from_date) 90 | 91 | if not isinstance(to_date, date): 92 | to_date = parse(to_date) 93 | 94 | current_date = from_date 95 | 96 | while (current_date < to_date): 97 | window_start = current_date 98 | current_date = current_date + timedelta(days=day_interval) 99 | window_end = current_date 100 | 101 | reply = _fetch_data(window_start, window_end, page_size) 102 | 103 | with io.open(json_directory + 'data_' + window_start.strftime( 104 | "%Y-%m-%d") + '.json', 'w', encoding='utf-8') as f: 105 | f.write(unicode(json.dumps(reply, ensure_ascii=False))) 106 | f.close 107 | 108 | if __name__ == "__main__": 109 | """Pull in the previous day's events by default""" 110 | 111 | to_date = date.today() 112 | from_date = to_date + timedelta(days=-1) 113 | 114 | query_vulndb(from_date, to_date) 115 | -------------------------------------------------------------------------------- /vulnpryer.conf.sample: -------------------------------------------------------------------------------- 1 | [VulnDB] 2 | consumer_key= YOUR_API_CONSUMER_KEY 3 | consumer_secret= YOUR_API_CONSUMER_SECRET 4 | json_dir= /data/vulndb_json/ 5 | page_size= 100 6 | request_token_url= https://vulndb.cyberriskanalytics.com/oauth/request_token 7 | working_dir= /tmp/ 8 | 9 | [RedSeal] 10 | username= YOUR_SUPPORT_EMAIL 11 | password= YOUR_SUPPORT_PASSWORD 12 | trl_url= https://www.redsealnetworks.com/login/trl/RedSeal_TRL_7-0-latest.gz 13 | 14 | [Mongo] 15 | hostname= localhost 16 | 17 | [S3] 18 | bucket_name= YOUR_S3_BUCKET 19 | region= YOUR_S3_BUCKET_REGION 20 | key= KEY_TO_YOUR_REDSEAL_TRL 21 | -------------------------------------------------------------------------------- /vulnpryer.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | import argparse 4 | from datetime import date, timedelta 5 | from dateutil.parser import parse 6 | # from time import ctime 7 | import logging 8 | import sys 9 | 10 | # VulnDB components 11 | from vulndb import query_vulndb 12 | from shiploader import load_mongo, get_extract 13 | from forklift import get_trl, modify_trl, post_trl 14 | 15 | # set default dates 16 | to_date = date.today() 17 | from_date = to_date + timedelta(days=-1) 18 | 19 | 20 | def mkdate(datestr): 21 | """Coerce arguments into date type""" 22 | if not isinstance(datestr, date): 23 | return parse(datestr) 24 | else: 25 | return datestr 26 | 27 | parser = argparse.ArgumentParser() 28 | parser.add_argument('-e', '--enddate', type=mkdate, default=to_date, 29 | help="Start date.") 30 | parser.add_argument('-s', '--startdate', type=mkdate, default=from_date, 31 | help="End date.") 32 | parser.add_argument('-l', '--loglevel', default="info", help="Log level.") 33 | args = parser.parse_args() 34 | 35 | start_string = args.startdate 36 | start_string = start_string.strftime("%Y-%m-%d") 37 | end_string = args.enddate 38 | end_string = end_string.strftime("%Y-%m-%d") 39 | 40 | # set logging level 41 | numeric_level = getattr(logging, args.loglevel.upper(), None) 42 | if not isinstance(numeric_level, int): 43 | raise ValueError('Invalid log level; %s' % args.loglevel) 44 | logging.basicConfig(stream=sys.stdout, 45 | level=numeric_level, 46 | format='%(asctime)s %(name)s %(levelname)s %(message)s') 47 | 48 | logger = logging.getLogger('vulnpryer') 49 | logger.info("Range requested {} - {}".format(start_string, end_string)) 50 | print("Range requested {} - {}".format(start_string, end_string)) 51 | query_vulndb(args.startdate, args.enddate) 52 | 53 | logger.info("Loading data into Mongo.") 54 | print("Loading data into Mongo.") 55 | load_mongo('data_*.json') 56 | 57 | logger.info("Generating extract.") 58 | print("Generating extract.") 59 | get_extract('/tmp/vulndb_export.csv') 60 | 61 | logger.info("Fetching RedSeal TRL.") 62 | print("Fetching RedSeal TRL.") 63 | get_trl('/tmp/trl.gz') 64 | 65 | logger.info("Generating modified TRL.") 66 | print("Generating modified TRL.") 67 | new_trl_path = modify_trl('/tmp/trl.gz') 68 | 69 | logger.info("Posting modified TRL to S3.") 70 | print("Posting modified TRL to S3.") 71 | post_trl(new_trl_path) 72 | 73 | logger.info("VulnPryer run complete.") 74 | print("VulnPryer run complete.") 75 | --------------------------------------------------------------------------------