├── CVE Manager.pdf ├── Changelog ├── LICENCE ├── README.md └── cve_manager.py /CVE Manager.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aatlasis/cve_manager/95afa6f3f0c2d4a686e41bbb551ce31088d78b33/CVE Manager.pdf -------------------------------------------------------------------------------- /Changelog: -------------------------------------------------------------------------------- 1 | Changelog 2 | --------- 3 | 4 | 2021 Feb 21 5 | 6 | Bug fixes 7 | UTF encodign to avoid parsing errors 8 | CWE processing 9 | Prompts for a password (instead of askign it in the command line) 10 | 11 | 2020 Aug 05 12 | 13 | A bug in importing data was corrected. 14 | Manual was updated to reflect the correct process of importing the data. 15 | 16 | 2020 Aug 02 17 | 18 | Most significant changes: 19 | You can query (offline) the database and get: 20 | - details about specific CVE including affected products (as CPEs) and CWEs (ie related weaknesses). 21 | - the affected CWE and get info about consequences, potential mitigations, etc. 22 | - list all CVEs greater than a score, published after a specific date, applicable to specific products (based on CPEs), or any combination of the above, and extract them (optionally) to CSV files for further processing. 23 | 24 | -------------------------------------------------------------------------------- /LICENCE: -------------------------------------------------------------------------------- 1 | __copyright__ = """Copyright (C) 2016 Antonios Atlasis 2 | 3 | This program is free software; you can redistribute it and/or modify it 4 | under the terms of the GNU General Public License as published by the 5 | Free Software Foundation; either version 2 of the License, or (at your 6 | option) any later version. 7 | 8 | This program is distributed in the hope that it will be useful, but 9 | WITHOUT ANY WARRANTY; without even the implied warranty of 10 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU 11 | General Public License for more details. 12 | 13 | You should have received a copy of the GNU General Public License along 14 | with this program; if not, write to the Free Software Foundation, Inc., 15 | 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA 16 | 17 | This tool may be used for legal purposes only. Users take full responsibility 18 | for any actions performed using this tool. If these terms are not acceptable to 19 | you, then do not use this tool. 20 | 21 | You are encouraged to send comments, improvements or suggestions to 22 | aatlasis@secfu.net 23 | """ 24 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # cve_manager 2 | A python script that: 3 | 4 | a) parses NIST NVD CVEs, 5 | b) prcoesses and exports them to CSV files, 6 | c) creates a postgres database and imports all the data in it, and 7 | d) provides (basic) query capabilities for this CVEs database. 8 | 9 | It requires Python 3 ("psycopg2" and "requests" python libraries) 10 | 11 | Usage examples: 12 | 13 | - Download, parse and save in CSV files all CVEs from NIST NVD: 14 | ./cve_manager.py -d -p -csv 15 | 16 | - Create a postgresql database to host the downloaded CVEs: 17 | ./cve_manager.py -u host -db -ow -cd 18 | 19 | - Create the tables and views at the database: 20 | ./cve_manager.py -u -host -db -ct 21 | 22 | - Import all data into the created database (requires the download, parse and sdtore as CSV files first, as explained above): 23 | ./cve_manager.py -u -host -db -idb -p 24 | 25 | - Query for a specific CVE: 26 | ./cve_manager.py -u -host -db -cve 2019-2434 27 | 28 | - Truncate the contents of all tables (required if you want to repeat the import process so as to update the data): 29 | ./cve_manager.py -u -host -db -tr 30 | 31 | - Delete the database (remove it completely): 32 | ./cve_manager.py -u -host -db -dd 33 | 34 | Complete list of supported arguments: 35 | 36 | -h, --help show this help message and exit 37 | 38 | -v, --version show program's version number and exit 39 | 40 | -p, --parse Process downloaded CVEs. 41 | 42 | -d, --download Download CVEs. 43 | 44 | -y YEAR, --year YEAR The year for which CVEs shall be downloaded (e.g. 2019) 45 | 46 | -csv, --cvs_files Create CSVs files. 47 | 48 | -idb, --import_to_db Import CVEs into a database. 49 | 50 | -i INPUT, --input INPUT 51 | The directory where NVD json files will been downloaded, and the one from where they will be parsed 52 | (default: nvd/) 53 | 54 | -o RESULTS, --output RESULTS 55 | The directory where the csv files will be stored (default: results/) 56 | 57 | -u USER, --user USER The user to connect to the database. 58 | 59 | -ow OWNER, --owner OWNER 60 | The owner of the database (if different from the connected user). 61 | 62 | -host HOST, --host HOST 63 | The host or IP of the database server. 64 | 65 | -db DATABASE, --database DATABASE 66 | The name of the database. 67 | 68 | -cd, --create_database 69 | Create the database 70 | 71 | -dd, --drop_database Drop the database 72 | 73 | -ct, --create_tables Create the tables of the database 74 | 75 | -tr, --truncate_cves_tables 76 | Truncate the CVEs-related tables 77 | 78 | -cve CVE, --cvs_number CVE 79 | Print info for a CVE (CVSS score and other) 80 | 81 | Please check the CVE Manager pdf file for more capabilities, information, and screenshots. 82 | -------------------------------------------------------------------------------- /cve_manager.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import psycopg2 4 | from psycopg2 import Error 5 | from psycopg2 import connect 6 | import sys 7 | from psycopg2.extensions import ISOLATION_LEVEL_AUTOCOMMIT 8 | import argparse 9 | import os 10 | from os import listdir 11 | from os.path import isfile, join, exists 12 | from os import makedirs 13 | import zipfile 14 | import json 15 | import requests 16 | import re 17 | import io 18 | import csv 19 | import getpass 20 | 21 | ##This is the Postgresql Database Schema## 22 | query = ''' 23 | CREATE TABLE public.cvss ( 24 | cve character(20) NOT NULL, 25 | attack_complexity_3 character(5), 26 | attack_vector_3 character(20), 27 | availability_impact_3 character(5), 28 | confidentiality_impact_3 character(5), 29 | integrity_impact_3 character(5), 30 | privileges_required_3 character(5), 31 | scope_3 character(10), 32 | user_interaction_3 character(10), 33 | vector_string_3 character(50), 34 | exploitability_score_3 real, 35 | impact_score_3 real, 36 | base_score_3 real, 37 | base_severity_3 character(10), 38 | access_complexity character(10), 39 | access_vector character(20), 40 | authentication character(10), 41 | availability_impact character(10), 42 | confidentiality_impact character(10), 43 | integrity_impact character(10), 44 | obtain_all_privileges boolean, 45 | obtain_other_privileges boolean, 46 | obtain_user_privileges boolean, 47 | user_interaction_required boolean, 48 | vector_string character(50), 49 | exploitability_score real, 50 | impact_score real, 51 | base_score real, 52 | severity character(10), 53 | description text, 54 | published_date date, 55 | last_modified_date date 56 | ); 57 | 58 | CREATE TABLE public.cpe ( 59 | cve character(20) NOT NULL, 60 | cpe23uri text, 61 | vulnerable character(5) 62 | ); 63 | 64 | CREATE TABLE public.cve_problem ( 65 | cve character(20) NOT NULL, 66 | problem text 67 | ); 68 | 69 | CREATE TABLE public.cwe ( 70 | cwe_id integer NOT NULL, 71 | name text, 72 | description text, 73 | extended_description text, 74 | modes_of_introduction text, 75 | common_consequences text, 76 | potential_mitigations text 77 | ); 78 | 79 | CREATE VIEW public.cvss_vs_cpes AS 80 | SELECT cvss.cve, 81 | cvss.base_score_3, 82 | cvss.base_severity_3, 83 | cvss.base_score, 84 | cvss.severity, 85 | cpe.cpe23uri, 86 | cvss.description, 87 | cvss.published_date 88 | FROM public.cpe, 89 | public.cvss 90 | WHERE (cpe.cve = cvss.cve) AND cpe.vulnerable = 'True'::bpchar; 91 | ''' 92 | 93 | ## functions to manage the database (optional) 94 | #def create_database(myuser,mypassword,myhost,database, owner): 95 | def create_database(myuser,myhost,database, owner): 96 | con = None 97 | try: 98 | con = connect(dbname=database, user=myuser, host = myhost) 99 | except (Exception, psycopg2.DatabaseError) as error : 100 | mypassword = getpass.getpass('Password:') 101 | finally: 102 | try: 103 | con = connect(dbname='postgres', user=myuser, host = myhost, password=mypassword) 104 | dbname = database 105 | con.set_isolation_level(ISOLATION_LEVEL_AUTOCOMMIT) 106 | cur = con.cursor() 107 | cur.execute('CREATE DATABASE ' + dbname) 108 | print("Database",database,"was created.") 109 | cur = con.cursor() 110 | query = '''ALTER DATABASE '''+ database + ''' OWNER TO ''' + owner+''';''' 111 | print("Owner of the database changed to:",owner) 112 | cur.execute(query) 113 | con.commit() 114 | except (Exception, psycopg2.DatabaseError) as error : 115 | print("Error while creating PostgreSQL Database", error) 116 | finally: 117 | #closing database connection. 118 | if(con): 119 | cur.close() 120 | con.close() 121 | print("PostgreSQL connection is closed") 122 | 123 | #def drop_database(myuser,mypassword,myhost,database): 124 | def drop_database(myuser,myhost,database): 125 | con = None 126 | try: 127 | con = connect(dbname=database, user=myuser, host = myhost) 128 | except (Exception, psycopg2.DatabaseError) as error : 129 | mypassword = getpass.getpass('Password:') 130 | finally: 131 | try: 132 | con = connect(dbname='postgres', user=myuser, host = myhost, password=mypassword) 133 | con.set_isolation_level(ISOLATION_LEVEL_AUTOCOMMIT) 134 | cur = con.cursor() 135 | cur.execute('DROP DATABASE ' + database) 136 | print("Database",database,"was dropped.") 137 | except (Exception, psycopg2.DatabaseError) as error : 138 | print ("Error while dropping PostgreSQL Database", error) 139 | finally: 140 | #closing database connection. 141 | if(con): 142 | cur.close() 143 | con.close() 144 | print("PostgreSQL connection is closed") 145 | 146 | #def create_tables(myuser,mypassword,myhost,database): 147 | def create_tables(myuser,myhost,database): 148 | con = None 149 | try: 150 | con = connect(dbname=database, user=myuser, host = myhost) 151 | except (Exception, psycopg2.DatabaseError) as error : 152 | mypassword = getpass.getpass('Password:') 153 | finally: 154 | try: 155 | con = connect(dbname=database, user=myuser, host = myhost, password=mypassword) 156 | cursor = con.cursor() 157 | create_tables_query = query 158 | cursor.execute(create_tables_query) 159 | con.commit() 160 | print("Tables and Views created successfully for database: "+database) 161 | except (Exception, psycopg2.DatabaseError) as error : 162 | print ("Error while creating PostgreSQL tables", error) 163 | finally: 164 | #closing database connection. 165 | if(con): 166 | cursor.close() 167 | con.close() 168 | print("PostgreSQL connection is closed") 169 | 170 | 171 | #Download CVEs 172 | def download_cves(directory,year): 173 | if not os.path.exists(directory): 174 | try: 175 | os.makedirs(directory) 176 | except OSError: 177 | print ('Error: Creating directory. ' + directory) 178 | exit(0) 179 | else: 180 | print ("Successfully created the directory %s" % directory) 181 | else: 182 | print ("Directory %s already exists" % directory) 183 | try: 184 | r = requests.get('https://nvd.nist.gov/vuln/data-feeds#JSON_FEED') 185 | except Exception as e: 186 | print(e) 187 | if year: 188 | print("downloading ",year," only") 189 | filename = "nvdcve-1.1-"+year+".json.zip" 190 | print(filename) 191 | r_file = requests.get("https://nvd.nist.gov/feeds/json/cve/1.1/" + filename, stream=True) 192 | with open(directory +"/" + filename, 'wb') as f: 193 | for chunk in r_file: 194 | f.write(chunk) 195 | else: 196 | for filename in re.findall("nvdcve-1.1-[0-9]*\.json\.zip",r.text): 197 | print(filename) 198 | r_file = requests.get("https://nvd.nist.gov/feeds/json/cve/1.1/" + filename, stream=True) 199 | with open(directory +"/" + filename, 'wb') as f: 200 | for chunk in r_file: 201 | f.write(chunk) 202 | 203 | #processes the already downloaded in json format CVEs 204 | #def process_cves(directory, results, csv_file, import_db,myuser,mypassword,myhost,database): 205 | def process_cves(directory, results, csv_file, import_db,myuser,myhost,database): 206 | if csv_file: 207 | if not os.path.exists(results): 208 | try: 209 | os.makedirs(results) 210 | except OSError: 211 | print ('Error: Creating directory. ' + results) 212 | exit(0) 213 | else: 214 | print ("Successfully created the directory %s" % results) 215 | else: 216 | print ("Directory %s already exists" % results) 217 | 218 | file_cve_related_problems = open(results+"cve_related_problems.csv","w",encoding='utf8') 219 | writer_cwe=csv.writer(file_cve_related_problems,delimiter="\t") 220 | 221 | file_cvss_score = open(results+"cve_cvss_scores.csv","w",encoding='utf8') 222 | writer_cvss=csv.writer(file_cvss_score ,delimiter="\t") 223 | 224 | file_cpes = open(results+"cve_cpes.csv","w",encoding='utf8') 225 | writer_cpe=csv.writer(file_cpes,delimiter="\t") 226 | 227 | writer_cpe.writerow(["CVE","cpe23Uri","Vulnerable"]) 228 | writer_cwe.writerow(["CVE","Problem"]) 229 | writer_cvss.writerow(["CVE","Attack Complexity","Attack Vector","Availability Impact","Confidentiality Impact","Integrity Impact","Privileges Required","Scope","UserInteraction","Vector String","Exploitability Score","Impact Score","base Score","base Severity","Access Complexity","Access Vector","Authentication","Availability Impact","Confidentiality Impact","Integrity Impact","Obtain All Privilege","Obtain Other Privilege","Obtain User Privilege","User Interaction Required","Vector String","Exploitability Score","impact Score","baseScore","severity","Description","Published Date","Last Modified Date"]) 230 | ######################################################################################## 231 | all_cves = [] 232 | directory = directory + "/" 233 | files = [f for f in listdir(directory) if isfile(join(directory, f))] 234 | files.sort(reverse=True) 235 | for file in files: 236 | print("\nProcessing", file) 237 | archive = zipfile.ZipFile(join(directory, file), 'r') 238 | jsonfile = archive.open(archive.namelist()[0]) 239 | cve_dict = json.loads(jsonfile.read()) 240 | print("CVE_data_timestamp: " + str(cve_dict['CVE_data_timestamp'])) 241 | print("CVE_data_version: " + str(cve_dict['CVE_data_version'])) 242 | print("CVE_data_format: " + str(cve_dict['CVE_data_format'])) 243 | print("CVE_data_number of CVEs: " + str(cve_dict['CVE_data_numberOfCVEs'])) 244 | print("CVE_data_type: " + str(cve_dict['CVE_data_type'])) 245 | all_cves = all_cves + cve_dict['CVE_Items'] 246 | #print(json.dumps(cve_dict['CVE_Items'][0], sort_keys=True, indent=4, separators=(',', ': '))) 247 | jsonfile.close() 248 | cvssv_score=[] 249 | for cves in all_cves: 250 | cve = cves['cve']['CVE_data_meta']['ID'] 251 | description = "" 252 | for descriptions in cves['cve']['description']['description_data']: 253 | description = description + descriptions['value'] 254 | description = description.replace("\r"," ") 255 | description = description.replace("\n"," ") 256 | description = description.replace("\t"," ") 257 | try: 258 | writer_cvss.writerow([cve,cves['impact']['baseMetricV3']['cvssV3']['attackComplexity'],cves['impact']['baseMetricV3']['cvssV3']['attackVector'],cves['impact']['baseMetricV3']['cvssV3']['availabilityImpact'],cves['impact']['baseMetricV3']['cvssV3']['confidentialityImpact'],cves['impact']['baseMetricV3']['cvssV3']['integrityImpact'],cves['impact']['baseMetricV3']['cvssV3']['privilegesRequired'],cves['impact']['baseMetricV3']['cvssV3']['scope'],cves['impact']['baseMetricV3']['cvssV3']['userInteraction'],cves['impact']['baseMetricV3']['cvssV3']['vectorString'],str(cves['impact']['baseMetricV3']['exploitabilityScore']),str(cves['impact']['baseMetricV3']['impactScore']),str(cves['impact']['baseMetricV3']['cvssV3']['baseScore']),str(cves['impact']['baseMetricV3']['cvssV3']['baseSeverity']),cves['impact']['baseMetricV2']['cvssV2']['accessComplexity'],cves['impact']['baseMetricV2']['cvssV2']['accessVector'],cves['impact']['baseMetricV2']['cvssV2']['authentication'],cves['impact']['baseMetricV2']['cvssV2']['availabilityImpact'],cves['impact']['baseMetricV2']['cvssV2']['confidentialityImpact'],cves['impact']['baseMetricV2']['cvssV2']['integrityImpact'],str(cves['impact']['baseMetricV2']['obtainAllPrivilege']),str(cves['impact']['baseMetricV2']['obtainOtherPrivilege']),str(cves['impact']['baseMetricV2']['obtainUserPrivilege']),str(cves['impact']['baseMetricV2']['userInteractionRequired']),cves['impact']['baseMetricV2']['cvssV2']['vectorString'],str(cves['impact']['baseMetricV2']['exploitabilityScore']),str(cves['impact']['baseMetricV2']['impactScore']),str(cves['impact']['baseMetricV2']['cvssV2']['baseScore']),str(cves['impact']['baseMetricV2']['severity']),description,cves['publishedDate'],cves['lastModifiedDate']]) 259 | except Exception as e: 260 | if str(e) == "'baseMetricV3'": 261 | try: 262 | writer_cvss.writerow([cve,None,None,None,None,None,None,None,None,None,None,None,None,None,cves['impact']['baseMetricV2']['cvssV2']['accessComplexity'],cves['impact']['baseMetricV2']['cvssV2']['accessVector'],cves['impact']['baseMetricV2']['cvssV2']['authentication'],cves['impact']['baseMetricV2']['cvssV2']['availabilityImpact'],cves['impact']['baseMetricV2']['cvssV2']['confidentialityImpact'],cves['impact']['baseMetricV2']['cvssV2']['integrityImpact'],str(cves['impact']['baseMetricV2']['obtainAllPrivilege']),str(cves['impact']['baseMetricV2']['obtainOtherPrivilege']),str(cves['impact']['baseMetricV2']['obtainUserPrivilege']),str(cves['impact']['baseMetricV2']['userInteractionRequired']),cves['impact']['baseMetricV2']['cvssV2']['vectorString'],str(cves['impact']['baseMetricV2']['exploitabilityScore']),str(cves['impact']['baseMetricV2']['impactScore']),str(cves['impact']['baseMetricV2']['cvssV2']['baseScore']),str(cves['impact']['baseMetricV2']['severity']),description,cves['publishedDate'],cves['lastModifiedDate']]) 263 | except Exception as e2: 264 | if str(e2) == "'baseMetricV2'": 265 | try: 266 | writer_cvss.writerow([cve,None,None,None,None,None,None,None,None,None,None,None,None,None,None,None,None,None,None,None,None,None,None,None,None,None,None,None,None,description,cves['publishedDate'],cves['lastModifiedDate']]) 267 | except Exception as e3: 268 | print("Error e3",e3) 269 | elif str(e2) == "'userInteractionRequired'": 270 | writer_cvss.writerow([cve,None,None,None,None,None,None,None,None,None,None,None,None,None,cves['impact']['baseMetricV2']['cvssV2']['accessComplexity'],cves['impact']['baseMetricV2']['cvssV2']['accessVector'],cves['impact']['baseMetricV2']['cvssV2']['authentication'],cves['impact']['baseMetricV2']['cvssV2']['availabilityImpact'],cves['impact']['baseMetricV2']['cvssV2']['confidentialityImpact'],cves['impact']['baseMetricV2']['cvssV2']['integrityImpact'],str(cves['impact']['baseMetricV2']['obtainAllPrivilege']),str(cves['impact']['baseMetricV2']['obtainOtherPrivilege']),str(cves['impact']['baseMetricV2']['obtainUserPrivilege']),None,cves['impact']['baseMetricV2']['cvssV2']['vectorString'],str(cves['impact']['baseMetricV2']['exploitabilityScore']),str(cves['impact']['baseMetricV2']['impactScore']),str(cves['impact']['baseMetricV2']['cvssV2']['baseScore']),str(cves['impact']['baseMetricV2']['severity']),description,cves['publishedDate'],cves['lastModifiedDate']]) 271 | else: 272 | print("Error e2",e2) 273 | elif str(e) == "'baseMetricV2'": 274 | writer_cvss.writerow([cve,cves['impact']['baseMetricV3']['cvssV3']['attackComplexity'],cves['impact']['baseMetricV3']['cvssV3']['attackVector'],cves['impact']['baseMetricV3']['cvssV3']['availabilityImpact'],cves['impact']['baseMetricV3']['cvssV3']['confidentialityImpact'],cves['impact']['baseMetricV3']['cvssV3']['integrityImpact'],cves['impact']['baseMetricV3']['cvssV3']['privilegesRequired'],cves['impact']['baseMetricV3']['cvssV3']['scope'],cves['impact']['baseMetricV3']['cvssV3']['userInteraction'],cves['impact']['baseMetricV3']['cvssV3']['vectorString'],str(cves['impact']['baseMetricV3']['exploitabilityScore']),str(cves['impact']['baseMetricV3']['impactScore']),str(cves['impact']['baseMetricV3']['cvssV3']['baseScore']),str(cves['impact']['baseMetricV3']['cvssV3']['baseSeverity']),None,None,None,None,None,None,None,None,None,None,None,None,None,None,None,description,cves['publishedDate'],cves['lastModifiedDate']]) 275 | elif str(e) == "'userInteractionRequired'": 276 | writer_cvss.writerow([cve,cves['impact']['baseMetricV3']['cvssV3']['attackComplexity'],cves['impact']['baseMetricV3']['cvssV3']['attackVector'],cves['impact']['baseMetricV3']['cvssV3']['availabilityImpact'],cves['impact']['baseMetricV3']['cvssV3']['confidentialityImpact'],cves['impact']['baseMetricV3']['cvssV3']['integrityImpact'],cves['impact']['baseMetricV3']['cvssV3']['privilegesRequired'],cves['impact']['baseMetricV3']['cvssV3']['scope'],cves['impact']['baseMetricV3']['cvssV3']['userInteraction'],cves['impact']['baseMetricV3']['cvssV3']['vectorString'],str(cves['impact']['baseMetricV3']['exploitabilityScore']),str(cves['impact']['baseMetricV3']['impactScore']),str(cves['impact']['baseMetricV3']['cvssV3']['baseScore']),str(cves['impact']['baseMetricV3']['cvssV3']['baseSeverity']),cves['impact']['baseMetricV2']['cvssV2']['accessComplexity'],cves['impact']['baseMetricV2']['cvssV2']['accessVector'],cves['impact']['baseMetricV2']['cvssV2']['authentication'],cves['impact']['baseMetricV2']['cvssV2']['availabilityImpact'],cves['impact']['baseMetricV2']['cvssV2']['confidentialityImpact'],cves['impact']['baseMetricV2']['cvssV2']['integrityImpact'],str(cves['impact']['baseMetricV2']['obtainAllPrivilege']),str(cves['impact']['baseMetricV2']['obtainOtherPrivilege']),str(cves['impact']['baseMetricV2']['obtainUserPrivilege']),None,cves['impact']['baseMetricV2']['cvssV2']['vectorString'],str(cves['impact']['baseMetricV2']['exploitabilityScore']),str(cves['impact']['baseMetricV2']['impactScore']),str(cves['impact']['baseMetricV2']['cvssV2']['baseScore']),str(cves['impact']['baseMetricV2']['severity']),description,cves['publishedDate'],cves['lastModifiedDate']]) 277 | else: 278 | print("Error e",e) 279 | 280 | for problem_type in cves['cve']['problemtype']['problemtype_data']: 281 | for descr in problem_type['description']: 282 | problem = descr['value'] 283 | if csv_file: 284 | writer_cwe.writerow([cve,problem]) 285 | try: 286 | cpe_list_length=len(cves['configurations']['nodes']) 287 | if (cpe_list_length !=0): 288 | for i in range(0,cpe_list_length): 289 | if 'children' in cves['configurations']['nodes'][i]: 290 | cpe_child_list_length=len(cves['configurations']['nodes'][i]['children']) 291 | if (cpe_child_list_length !=0): 292 | for j in range(0,cpe_child_list_length): 293 | if('cpe_match' in cves['configurations']['nodes'][i]['children'][j]): 294 | cpes = cves['configurations']['nodes'][i]['children'][j]['cpe_match'] 295 | for cpe in cpes: 296 | if csv_file: 297 | if 'cpe23Uri' in cpe: 298 | writer_cpe.writerow([cve,cpe['cpe23Uri'],str(cpe['vulnerable'])]) 299 | else: 300 | if('cpe_match' in cves['configurations']['nodes'][i]): 301 | cpes = cves['configurations']['nodes'][i]['cpe_match'] 302 | for cpe in cpes: 303 | if csv_file: 304 | if 'cpe23Uri' in cpe: 305 | writer_cpe.writerow([cve,cpe['cpe23Uri'],str(cpe['vulnerable'])]) 306 | else: 307 | cpe_inner_list_length=len(cves['configurations']['nodes']) 308 | if (cpe_inner_list_length!=0): 309 | for k in range(0,cpe_inner_list_length): 310 | if('cpe_match' in cves['configurations']['nodes'][i]): 311 | cpes = cves['configurations']['nodes'][i]['cpe_match'] 312 | for cpe in cpes: 313 | if csv_file: 314 | if 'cpe23Uri' in cpe: 315 | writer_cpe.writerow([cve,cpe['cpe23Uri'],str(cpe['vulnerable'])]) 316 | except Exception as e: 317 | print(str(e),cves['configurations']) #check it 318 | file_cve_related_problems.close() 319 | file_cvss_score.close() 320 | file_cpes.close() 321 | if import_db: 322 | print('Connecting to the PostgreSQL database...') 323 | try: 324 | con = connect(dbname=database, user=myuser, host = myhost) 325 | except (Exception, psycopg2.DatabaseError) as error : 326 | mypassword = getpass.getpass('Password:') 327 | finally: 328 | try: 329 | conn = psycopg2.connect("dbname='"+database+"' user='"+myuser+"' host='"+myhost+"' password='"+mypassword+"'") 330 | except psycopg2.Error as e: 331 | print( "I am unable to connect to the database. Error:",e) 332 | print( "Exiting") 333 | sys.exit(1) 334 | cur = conn.cursor() 335 | filename = results+"cve_cvss_scores.csv" 336 | with open(filename, 'r',encoding='utf8') as f: 337 | print("importing CVSS") 338 | filedata = f.read() 339 | filedata = filedata.replace("\\","\\\\") 340 | output = io.StringIO() 341 | output.write(filedata) 342 | output.seek(0) 343 | output.readline() 344 | cur.copy_from(output, 'cvss', sep='\t', null="") 345 | conn.commit() 346 | f.close() 347 | filename = results+"cve_related_problems.csv" 348 | with open(filename, 'r') as f: 349 | print("importing CVE-related problems") 350 | f.readline() 351 | cur.copy_from(f, 'cve_problem', sep='\t',columns=('cve', 'problem')) 352 | conn.commit() 353 | f.close() 354 | filename = results+"cve_cpes.csv" 355 | with open(filename, 'r') as f: 356 | print("importing CVEs vs CPEs") 357 | f.readline() 358 | cur.copy_from(f, 'cpe', sep='\t',columns=('cve','cpe23uri','vulnerable')) 359 | conn.commit() 360 | f.close() 361 | 362 | #def truncate_database(myuser,mypassword,myhost,database): 363 | def truncate_database(myuser,myhost,database): 364 | con = None 365 | mypassword = None 366 | try: 367 | con = connect(dbname=database, user=myuser, host = myhost) 368 | except (Exception, psycopg2.DatabaseError) as error : 369 | mypassword = getpass.getpass('Password:') 370 | finally: 371 | try: 372 | con = connect(dbname=database, user=myuser, host = myhost, password=mypassword) 373 | con.set_isolation_level(ISOLATION_LEVEL_AUTOCOMMIT) 374 | cur = con.cursor() 375 | print("Truncating CVEs tables") 376 | cur.execute("Truncate cpe, cve_problem, cvss;") 377 | con.commit() 378 | except (Exception, psycopg2.DatabaseError) as error : 379 | print("Error while Truncating PostgreSQL Database", error) 380 | finally: 381 | if(con): 382 | cur.close() 383 | con.close() 384 | print("PostgreSQL connection is closed") 385 | 386 | #def execute_query(myuser,mypassword,myhost,database,cve,score,date,csv_on,output_folder): 387 | def execute_query(myuser,myhost,database,cve,score,date,csv_on,output_folder): 388 | con = None 389 | mypassword = None 390 | try: 391 | con = connect(dbname=database, user=myuser, host = myhost) 392 | except (Exception, psycopg2.DatabaseError) as error : 393 | mypassword = getpass.getpass('Password:') 394 | finally: 395 | try: 396 | con = connect(dbname=database, user=myuser, host = myhost, password=mypassword) 397 | cur = con.cursor() 398 | print("Executing query\r\n") 399 | if cve: 400 | cur.execute("SELECT cve, vector_string_3, base_score_3, base_severity_3, vector_string, base_score, severity, description, published_date, last_modified_date FROM cvss WHERE cve LIKE '%"+cve+"%'") 401 | selected_cve = cur.fetchone() 402 | print("CVE:\t\t\t",selected_cve[0]) 403 | print("CVSSv3.x Attack vector: ",selected_cve[1]) 404 | print("CVSSv3.x Base Score:\t",selected_cve[2],selected_cve[3]) 405 | print("CVSSv2.x Attack vector: ",selected_cve[4]) 406 | print("CVSSv2.x Base Score:\t",selected_cve[5],selected_cve[6]) 407 | print("Description:") 408 | print(selected_cve[7]) 409 | print("\r\nPubished Date:\t\t",selected_cve[8]) 410 | print("Last Modified Date:\t",selected_cve[9]) 411 | cur.execute("SELECT problem FROM cve_problem WHERE cve LIKE '%"+cve+"%'") 412 | selected_cve = cur.fetchall() 413 | print("\r\nRelated Common Weakness Enumerations (CWE)") 414 | print("-------------------------------------------") 415 | for i in selected_cve: 416 | cwe=i[0].lstrip('CWE-') 417 | if cwe.isdigit(): 418 | cur.execute("SELECT name FROM cwe WHERE cwe_id = "+cwe) 419 | selected_cve2 = cur.fetchone() 420 | if (selected_cve2): 421 | print(i[0], selected_cve2[0]) 422 | else: 423 | print(i[0]) 424 | cur.execute("SELECT cpe23uri FROM cpe WHERE cve LIKE '%"+cve+"%' AND vulnerable='True'") 425 | selected_cve = cur.fetchall() 426 | print("\r\nRelated Common Platform Enumerations (CPE)") 427 | print("-------------------------------------------") 428 | for i in selected_cve: 429 | print(i[0]) 430 | elif score or date: 431 | if csv_on: 432 | cves=[] 433 | if date: 434 | cur.execute("SELECT cve, base_score_3, vector_string_3, base_score, vector_string, published_date FROM cvss WHERE (base_score_3 >= "+score+"OR base_score >= "+ score+") AND (published_date >= '"+date+"'::date)") 435 | selected_cves = cur.fetchall() 436 | print("CVE \t\tCVSSv3.x Score CVSSv3.x Vector String \t\t\tCVSSv2 Score CVSSv2 Vector String\t\t\t Published Date") 437 | for r in selected_cves: 438 | print(r[0],r[1],r[2],r[3],r[4],r[5]) 439 | if csv_on: 440 | cves.append(r) 441 | else: 442 | cur.execute("SELECT cve, base_score_3, vector_string_3, base_score, vector_string, published_date FROM cvss WHERE base_score_3 >= "+score+"OR base_score >= "+ score) 443 | selected_cves = cur.fetchall() 444 | print("CVE \t\tCVSSv3.x Score CVSSv3.x Vector String \t\t\tCVSSv2 Score CVSSv2 Vector String") 445 | for r in selected_cves: 446 | print(r[0],r[1],r[2],r[3],r[4]) 447 | if csv_on: 448 | cves.append(r) 449 | except (Exception, psycopg2.DatabaseError) as error : 450 | print ("Error while Querying Database", error) 451 | finally: 452 | if(con): 453 | cur.close() 454 | con.close() 455 | print("\r\nPostgreSQL connection is closed") 456 | if csv_on: 457 | if not os.path.exists(output_folder): 458 | try: 459 | os.makedirs(output_folder) 460 | except OSError: 461 | print ('Error: Creating directory. ' + output_folder) 462 | exit(0) 463 | else: 464 | print ("Successfully created the directory %s" % output_folder) 465 | else: 466 | print ("Directory %s already exists" % output_folder) 467 | file_cve = open(output_folder+"CVEs_score"+score+"_"+str(date)+".csv","w") 468 | writer_cve=csv.writer(file_cve,delimiter=",") 469 | writer_cve.writerow(["CVE","CVSSv3 Score","CVSSv3 Vector String","CVSSv2 Score","CVSSv2 Vector String","Published Date"]) 470 | for r in cves: 471 | writer_cve.writerow([r[0],r[1],r[2],r[3],r[4],r[5]]) 472 | file_cve.close() 473 | 474 | #def execute_query_cpe(myuser,mypassword,myhost,database,cpe,score,date,csv_on,output_folder): 475 | def execute_query_cpe(myuser,myhost,database,cpe,score,date,csv_on,output_folder): 476 | con = None 477 | mypassword = None 478 | if csv_on: 479 | cpes=[] 480 | try: 481 | con = connect(dbname=database, user=myuser, host = myhost) 482 | print("OK") 483 | exit(0) 484 | except (Exception, psycopg2.DatabaseError) as error : 485 | mypassword = getpass.getpass('Password:') 486 | finally: 487 | try: 488 | con = connect(dbname=database, user=myuser, host = myhost, password=mypassword) 489 | cur = con.cursor() 490 | print("Executing query\r\n") 491 | if date: 492 | cur.execute("SELECT cpe23uri, cve, base_score_3, base_score, published_date FROM cvss_vs_cpes WHERE cpe23uri LIKE '%"+cpe+"%' AND (base_score_3 >= "+score+"OR base_score >= "+ score+") AND (published_date >= '"+date+"'::date)") 493 | selected_cpe = cur.fetchall() 494 | print("CPE\t\t\t\t\t\t\tCVE\t\tCVSSv3.x CVSSv2\t Published Date") 495 | for r in selected_cpe: 496 | print(r[0],r[1],r[2],"\t",r[3],"\t",r[4]) 497 | if csv_on: 498 | cpes.append(r) 499 | else: 500 | cur.execute("SELECT cpe23uri, cve, base_score_3, base_score,published_date FROM cvss_vs_cpes WHERE cpe23uri LIKE '%"+cpe+"%' AND (base_score_3 >= "+score+"OR base_score >= "+ score+")") 501 | selected_cpe = cur.fetchall() 502 | print("CPE\t\t\t\t\t\t\tCVE\t\tCVSSv3.x CVSSv2") 503 | for r in selected_cpe: 504 | print(r[0],r[1],r[2],"\t",r[3]) 505 | if csv_on: 506 | cpes.append(r) 507 | except (Exception, psycopg2.DatabaseError) as error : 508 | print ("Error while Querying Database", error) 509 | finally: 510 | if(con): 511 | cur.close() 512 | con.close() 513 | print("\r\nPostgreSQL connection is closed") 514 | if csv_on: 515 | if not os.path.exists(output_folder): 516 | try: 517 | os.makedirs(output_folder) 518 | except OSError: 519 | print ('Error: Creating directory. ' + output_folder) 520 | exit(0) 521 | else: 522 | print ("Successfully created the directory %s" % output_folder) 523 | else: 524 | print ("Directory %s already exists" % output_folder) 525 | file_cpe = open(output_folder+cpe+"_"+score+"_"+str(date)+".csv","w") 526 | writer_cpe=csv.writer(file_cpe,delimiter=",") 527 | writer_cpe.writerow(["CPE","CVE","CVSSv3 Score","CVSSv2 Score","Published Date"]) 528 | for r in cpes: 529 | writer_cpe.writerow([r[0],r[1],r[2],r[3],r[4]]) 530 | file_cpe.close() 531 | 532 | #def execute_query_cwe(myuser,mypassword,myhost,database,cwe): 533 | def execute_query_cwe(myuser,myhost,database,cwe): 534 | con = None 535 | mypassword = None 536 | try: 537 | con = connect(dbname=database, user=myuser, host = myhost) 538 | except (Exception, psycopg2.DatabaseError) as error : 539 | mypassword = getpass.getpass('Password:') 540 | finally: 541 | try: 542 | con = connect(dbname=database, user=myuser, host = myhost, password=mypassword) 543 | cur = con.cursor() 544 | print("Executing query\r\n") 545 | cur.execute("SELECT * FROM cwe WHERE cwe_id = "+cwe) 546 | selected_cwe = cur.fetchone() 547 | if selected_cwe: 548 | print("CWE-"+str(selected_cwe[0])) 549 | print("========") 550 | print(selected_cwe[1]) 551 | if(selected_cwe[2]): 552 | print(selected_cwe[2]) 553 | if(selected_cwe[3]): 554 | print(selected_cwe[3]) 555 | if(selected_cwe[4]): 556 | print("\r\nModes of Introduction") 557 | print("--------------------") 558 | print(selected_cwe[4]) 559 | if(selected_cwe[5]): 560 | print("\r\nCommon Consequences") 561 | print("--------------------") 562 | print(selected_cwe[5]) 563 | if(selected_cwe[6]): 564 | print("\r\nPotential Mitigations") 565 | print("--------------------") 566 | print(selected_cwe[6]) 567 | else: 568 | print("CWE-"+cwe,"not found") 569 | except (Exception, psycopg2.DatabaseError) as error : 570 | print ("Error while Querying Database") 571 | #print ("Error while Querying Database", error) 572 | print("Hint: Use just the number of teh CWE you are looking for, e.g.: 169") 573 | finally: 574 | if(con): 575 | cur.close() 576 | con.close() 577 | print("\r\nPostgreSQL connection is closed") 578 | 579 | #def cwe(myuser,mypassword,myhost,database,filename): 580 | def cwe(myuser,myhost,database,filename): 581 | con = None 582 | mypassword = None 583 | try: 584 | con = connect(dbname=database, user=myuser, host = myhost) 585 | except (Exception, psycopg2.DatabaseError) as error : 586 | mypassword = getpass.getpass('Password:') 587 | finally: 588 | try: 589 | con = connect(dbname=database, user=myuser, host = myhost, password=mypassword) 590 | cur = con.cursor() 591 | with open(filename, 'r') as f: 592 | print("importing cwe") 593 | reader = csv.reader(f) 594 | header_row = next(reader) 595 | cwe = io.StringIO() 596 | writer_cwe=csv.writer(cwe) 597 | writer_cwe.writerow(["cwe_id","name","description","extended_decription","modes_of_introduction","common_consequences","potential_mitigations"]) 598 | for row in reader: 599 | writer_cwe.writerow([row[0],row[1],row[4],row[5],row[11],row[14],row[16]]) 600 | cwe.seek(0) 601 | cwe.readline() 602 | cur.copy_expert("""COPY cwe(cwe_id, name, description, extended_description, modes_of_introduction, common_consequences, potential_mitigations) FROM STDIN WITH (FORMAT CSV)""", cwe) 603 | con.commit() 604 | except (Exception, psycopg2.DatabaseError) as error : 605 | print ("Error while Querying Database", error) 606 | finally: 607 | if(con): 608 | cur.close() 609 | con.close() 610 | print("PostgreSQL connection is closed") 611 | 612 | if __name__ == '__main__': 613 | parser = argparse.ArgumentParser(description='CVEs Manager.') 614 | parser.add_argument('-p', '--parse', action="store_true", dest="process", default=False, help="Process downloaded CVEs.") 615 | parser.add_argument('-d', '--download', action="store_true", dest="download", default=False, help="Download CVEs.") 616 | parser.add_argument('-y', '--year', action="store", dest="year", default=False, help="The year for which CVEs shall be downloaded (e.g. 2019)") 617 | parser.add_argument('-csv', '--csv_files', action="store_true", dest="csv_file", default=False, help="Create CSVs files.") 618 | parser.add_argument('-icwe', '--import_cwe', action="store", dest="icwe", default=None, help="Import CWE from the provided filename into the database.") 619 | parser.add_argument('-idb', '--import_to_db', action="store_true", dest="idb", default=False, help="Import CVEs into a database.") 620 | parser.add_argument('-i', '--input', action="store", default = 'nvd/', dest="input", help="The directory where NVD json files will been downloaded, and the one from where they will be parsed (default: nvd/") 621 | parser.add_argument('-o', '--output', action="store", default = 'results/', dest="results", help="The directory where the csv files will be stored (default: results/") 622 | parser.add_argument('-u', '--user', action="store", dest="user", default="postgres", help="The user to connect to the database.") 623 | parser.add_argument('-ow', '--owner', action="store", dest="owner", default=None, help="The owner of the database (if different from the connected user).") 624 | #parser.add_argument('-ps', '--password', action="store", dest="password", default="", help="The password to connect to the database.") 625 | parser.add_argument('-host', '--host', action="store", dest="host", default=None, help="The hostname or IP for which you want to list the applicable vulnerabilities") 626 | parser.add_argument('-server', '--server', action="store", dest="server", default="localhost", help="The hostname or IP of the database server.") 627 | parser.add_argument('-db', '--database', action="store", dest="database", default="postgres", help="The name of the database.") 628 | parser.add_argument('-cd', '--create_database', action="store_true", dest="cd", default=False, help="Create the database") 629 | parser.add_argument('-dd', '--drop_database', action="store_true", dest="dd", default=False, help="Drop the database") 630 | parser.add_argument('-ct', '--create_tables', action="store_true", dest="ct", default=False, help="Create the tables of the database") 631 | parser.add_argument('-tr', '--truncate_cves_tables', action="store_true", dest="tr", default=False, help="Truncate the CVEs-related tables") 632 | parser.add_argument('-cve', '--cve_number', action="store", dest="cve", default=None, help="Print info for a CVE (CVSS score and other)") 633 | parser.add_argument('-cpe', '--cpe', action="store", dest="cpe", default=None, help="List all the CVEs for the selected CPE(s)") 634 | parser.add_argument('-cwe', '--cwe', action="store", dest="cwe", default=None, help="Provide info for the requested CWE)") 635 | parser.add_argument('-sc', '--score', action="store", dest="score", default=0.0, help="Use base score of a CVE as a selection criterion") 636 | parser.add_argument('-dt', '--date', action="store", dest="date", default=False, help="Use publication date of a CVE as a selection criterion") 637 | values = parser.parse_args() 638 | 639 | if not values.owner: 640 | values.owner=values.user 641 | if values.dd: 642 | print("Dropping the database") 643 | #drop_database(values.user,values.password,values.server,values.database) 644 | drop_database(values.user,values.server,values.database) 645 | if values.cd: 646 | print("Creating the database") 647 | #create_database(values.user,values.password,values.server,values.database,values.owner) 648 | create_database(values.user,values.server,values.database,values.owner) 649 | if values.ct: 650 | print("Creating the necessary schema of the database") 651 | #create_tables(values.user,values.password,values.server,values.database) 652 | create_tables(values.user,values.server,values.database) 653 | if values.download: 654 | print("Downloading NIST NVD") 655 | download_cves(values.input,values.year) 656 | if values.tr: 657 | print("Truncating NIST NVD imported data") 658 | #truncate_database(values.user,values.password,values.server,values.database) 659 | truncate_database(values.user,values.server,values.database) 660 | if values.process: 661 | print("Processing downloaded data") 662 | #process_cves(values.input, values.results, values.csv_file, values.idb,values.user,values.password,values.server,values.database) 663 | process_cves(values.input, values.results, values.csv_file, values.idb,values.user,values.server,values.database) 664 | if values.icwe: 665 | print("Importing CWE data") 666 | #cwe(values.user,values.password,values.host,values.database,values.icwe) 667 | cwe(values.user,values.host,values.database,values.icwe) 668 | if values.cpe: 669 | print("CPE queries") 670 | #execute_query_cpe(values.user,values.password,values.host,values.database,values.cpe,str(values.score),values.date,values.csv_file,values.results) 671 | execute_query_cpe(values.user,values.host,values.database,values.cpe,str(values.score),values.date,values.csv_file,values.results) 672 | elif values.cwe: 673 | print("CWE queries") 674 | #execute_query_cwe(values.user,values.password,values.host,values.database,values.cwe) 675 | execute_query_cwe(values.user,values.host,values.database,values.cwe) 676 | elif values.cve or float(values.score) > 0.0: 677 | print("CVE queries") 678 | #execute_query(values.user,values.password,values.host,values.database,values.cve,values.score,values.date,values.csv_file,values.results) 679 | execute_query(values.user,values.host,values.database,values.cve,values.score,values.date,values.csv_file,values.results) 680 | elif not values.download and not values.process and not values.cd and not values.ct and not values.dd and not values.tr and not values.icwe: 681 | print("Choose an option (check --help)") 682 | --------------------------------------------------------------------------------