Multithreaded parallel network command execution, simplified, for network engineers of the world!
4 | 5 |  6 | 7 | ## Table of Contents 8 | 9 | - [Prerequisites](#prerequisites) 10 | - [Installation](#installation) 11 | - [Usage](#usage) 12 | - [Contributing](#contributing) 13 | 14 | ## Prerequisites 15 | 16 | For everything: 17 | 18 | ```pip install netmiko docopt getpass``` 19 | 20 | ## Installation 21 | 22 | Clone the repo, and run the scripts using Python 3. This is NOT a library or module, it is meant to be templates and examples that you can use, copy, and make better. 23 | 24 | ## Usage 25 | 26 | Create your 'customer' file and 'credentials' file based on exampleCSV.md. 27 | 28 | #### python netspark.py -h 29 | This script is the master controller script. It will make the necessary 30 | commands to run whatever is needed against whatever is needed. 31 | 32 | Usage: 33 | netspark.py -h | --help 34 | netspark.py (--info COMMAND | --config CONFIGFILE) (--csv FILENAME | --db QUERYNAME | --ip IPADDR) [-c CREDFILE] [--debug] 35 | 36 | Options: 37 | -h, --help Shows this menu. 38 | -d, --debug Print debug information. This is the most verbose 39 | option. 40 | --info COMMAND This will run regular show commands 41 | --config CONFIGFILE This will run configuration commands 42 | --csv FILENAME Input file if using CSV [default: test.csv] 43 | --db QUERYNAME SQL field: 'groupname' [default: test] 44 | --ip IPADDR Single IP address of test switch 45 | -c CREDFILE Crednetials file [default: credentials.csv] 46 | 47 | ### Examples: 48 | To run write mem on every host in your csv file switches.csv (assuming you only have one credentials.csv): 49 | 50 | `python netspark.py --info "wr mem" --csv switches.csv` 51 | 52 | To make a config change with many lines, such as changing a local account, you'd make a text file with the lines to change it which will then be run against the hosts. For example, lets make a changeuser.txt: 53 | 54 | ``` 55 | exit 56 | conf t revert time 5 57 | username secretbackdoor privilege 15 password Thiscantbereal! 58 | ``` 59 | 60 | --config runs in the context of config mode, which is why I use an exit above. 61 | 62 | Then you'd run it like so: `python netspark.py --config changeuser.txt --csv switches.csv` 63 | 64 | This command supports all of the netmiko classes of devices, but I designed it to be used with Cisco gear since that's what most of us run. You just need to specify a different device object in your switches.csv file to work with new stuff. 65 | 66 | Final example, lets say you have a rogue sysadmin at one branch who refuses to use newer credentials/radius/whatever. So you have a different nonstandard set of creds and routers. 67 | 68 | `python netspark.py --info "configure confirm" --csv crappyrouters.csv -c crappycreds.csv` 69 | 70 | ### Caveats 71 | 72 | This is multithreaded and I haven't added error handling. I force all of my changes to use config revision (the 'conf t revert time 5' above) as a workaround until I have good error handling in place. It's laziness and a lack of time, it would be decently easy to implement. 73 | 74 | The multiprocess code I wrote is messy because of context issues. I plan on making that part into a library later on, but something something time and laziness. 75 | 76 | The default number of simultaneous executions is 8, I have run this successfully at 50x simultaneous on a network of 400+ devices but keep in mind that they will return output at a rate of 50x, and it spits it out to STDOUT, so you might want to just leave it. 77 | 78 | I will add support for --db and --ip later on, probably --db first because I want this to dive into the rest of my open-source stack I'm slowly writing about on https://teamignition.us so that this all ties in beautifully. 79 | 80 | ## The Example_Scripts folder 81 | This is old code and alternative projects. It is useful reference material so I'm leaving it up, but it will not be updated. 82 | 83 | ## Contributing 84 | 85 | All patches welcome! Please read [CONTRIBUTING.md](https://github.com/admiralspark/netspark-scripts/CONTRIBUTING.md) for furthers details...whenever I make it. For now, submit PR's and we'll chat about them. 86 | 87 | ## License 88 | 89 | GNU GPL Version 3 - see the [LICENSE](https://github.com/admiralspark/NetSpark-Scripts/blob/master/LICENSE) file for details 90 | -------------------------------------------------------------------------------- /credentials.py: -------------------------------------------------------------------------------- 1 | ''' 2 | ------------------------------------------------------------------------------- 3 | This function loads the credentials from a CSV file for connecting to your 4 | devices. Will hopefully become an *option* in the future instead of the only 5 | one supported. 6 | ------------------------------------------------------------------------------- 7 | ''' 8 | 9 | import csv 10 | import os.path 11 | import getpass 12 | 13 | #Read in the credentials from file 14 | def cred_csv(filename="credentials.csv"): 15 | """Check for existence of credentials file, read data if True, else ask for it manually""" 16 | if os.path.isfile(filename) is True: 17 | with open(filename, mode='r') as credfile: 18 | reader = csv.DictReader(credfile) 19 | for row in reader: 20 | username = row['username'] 21 | password = row['password'] 22 | secret = row['secret'] 23 | return (username, password, secret) 24 | else: 25 | username = input("Username? ") 26 | password = getpass.getpass("Password? ") 27 | secret = getpass.getpass("Enable Password? ") 28 | return (username, password, secret) 29 | -------------------------------------------------------------------------------- /exampleCSV.md: -------------------------------------------------------------------------------- 1 | #How to make a 'company.csv' file 2 | For your **'company.csv'** you'll want something with this format: 3 | 4 | IP_Address,SysName,device_type,authentication 5 | 10.0.0.1,HOST-SW-1,cisco_ios,LOCAL 6 | 7 | For your **'credentials.csv'** you'll want something like this: 8 | 9 | username,password,secret 10 | cisco,cisco123,enable123 11 | -------------------------------------------------------------------------------- /netspark.py: -------------------------------------------------------------------------------- 1 | ''' 2 | This script is the master controller script. It will make the necessary 3 | commands to run whatever is needed against whatever is needed. 4 | 5 | Usage: 6 | netspark.py -h | --help 7 | netspark.py (--info COMMAND | --config CONFIGFILE) (--csv FILENAME | --db QUERYNAME | --ip IPADDR) [-c CREDFILE] [--debug] 8 | 9 | Options: 10 | -h, --help Shows this menu. 11 | -d, --debug Print debug information. This is the most verbose 12 | option. 13 | --info COMMAND This will run regular show commands 14 | --config CONFIGFILE This will run configuration commands 15 | --csv FILENAME Input file if using CSV [default: test.csv] 16 | --db QUERYNAME SQL field: 'groupname' [default: test] 17 | --ip IPADDR Single IP address of test switch 18 | -c CREDFILE Crednetials file [default: credentials.csv] 19 | ''' 20 | 21 | import os.path 22 | import logging 23 | from docopt import docopt 24 | import spark_threaded as st 25 | 26 | 27 | arguments = docopt(__doc__) 28 | 29 | # Set logging level https://www.digitalocean.com/community/tutorials/how-to-use-logging-in-python-3 30 | # This is for debugging. 31 | if arguments['--debug'] == True: 32 | logging.basicConfig(level=logging.DEBUG) 33 | print("Arguments: \n" + str(arguments)) 34 | else: 35 | logging.basicConfig(level=logging.WARNING) 36 | 37 | # Global variable MODE stores whether we're running config or not. This is here instead of defined 38 | # in the functions below because I'm going to use it for testing. 39 | MODE = st.check_config_mode(arguments['--config']) 40 | logging.debug("Value for MODE evaluated to: " + str(MODE)) 41 | 42 | if MODE is False: 43 | st.info_command(arguments['--info'], arguments['--csv'], arguments['--db'], arguments['--ip'], arguments['-c']) 44 | 45 | elif MODE is True: 46 | COMMANDLIST = [] 47 | if os.path.exists(arguments['--config']): 48 | with open(arguments['--config']) as conffile: 49 | for row in conffile: 50 | COMMANDLIST.append(row) 51 | st.COMMANDLIST = COMMANDLIST 52 | st.config_command(arguments['--config'], arguments['--csv'], arguments['--db'], arguments['--ip'], arguments['-c']) 53 | 54 | else: 55 | logging.info("You somehow broke the required info/config arguments!") -------------------------------------------------------------------------------- /spark_single.py: -------------------------------------------------------------------------------- 1 | ''' 2 | This module is a base for single-thread scripts. 3 | ''' 4 | 5 | #Global imports 6 | from datetime import datetime 7 | import csv 8 | from netmiko import ConnectHandler 9 | #Local imports 10 | import credentials 11 | 12 | # Begin timing the script 13 | STARTTIME = datetime.now() 14 | 15 | # Define the primary function (to be moved to a separate module some day...) 16 | def netcon(username, password, secret, CUSTOMER, COMMANDLIST): 17 | ''' 18 | This is the core function. Iterates through a CSV, forms a dict, runs the command 19 | and logics it. 20 | ''' 21 | with open(CUSTOMER, mode='r') as csvfile: 22 | reader = csv.DictReader(csvfile) 23 | # Now iterate through every row in the CSVfile and make dictionaries 24 | for row in reader: 25 | hostname = row['SysName'] 26 | device_type = row['device_type'] 27 | ipaddr = row['IP_Address'] 28 | switch = { 29 | 'device_type': device_type, 30 | 'ip': ipaddr, 31 | 'username': username, 32 | 'password': password, 33 | 'secret': secret, 34 | 'verbose': False, 35 | } 36 | # This is your connection handler for commands from here on out 37 | net_connect = ConnectHandler(**switch) 38 | # Insert your commands here 39 | net_connect.enable() 40 | # or maybe send configuration stuff with 41 | # net_connect.send_config_set(username cisco priv 15 pass cisco) 42 | connect_return = net_connect.send_config_set(COMMANDLIST) 43 | # Now make it pretty 44 | print("\n\n>>>>>>>>> Device {0} {1} <<<<<<<<<".format(hostname, ipaddr)) 45 | print(connect_return) 46 | print("\n>>>>>>>>> End <<<<<<<<<") 47 | # Disconnect from this session 48 | net_connect.disconnect() 49 | 50 | # Grab the Customer name to search 51 | CUSTOMER = input('Customer name: ') + ".csv" 52 | # Flesh out these variables using the credentials.cred_csv module 53 | username, password, secret = credentials.cred_csv() 54 | # Just for testing 55 | #COMMANDSTRING = input('Command string to run: ') 56 | COMMANDLIST = [] 57 | command = input("Input one command per line, end with an extra newline: ") 58 | while command is not "": 59 | COMMANDLIST.append(command) 60 | command = input("Input one command per line, end with an extra newline: ") 61 | 62 | # Run the primary function in this program 63 | netcon(username, password, secret, CUSTOMER, COMMANDLIST) 64 | 65 | 66 | ENDTIME = datetime.now() 67 | # How long did it run? 68 | TOTALTIME = ENDTIME - STARTTIME 69 | print("\nTotal time for script: \n" + str(TOTALTIME)) 70 | -------------------------------------------------------------------------------- /spark_threaded.py: -------------------------------------------------------------------------------- 1 | ''' 2 | ------------------------------------------------------------------------------- 3 | This script is a baseline for multithreaded conversion of all of the scripts. 4 | I'll look into converting the repo to use multithread, some day. 5 | ------------------------------------------------------------------------------- 6 | ''' 7 | 8 | from datetime import datetime 9 | import csv 10 | import logging 11 | from multiprocessing.dummy import Pool as ThreadPool 12 | #from multiprocessing.dummy import cpu_count # Broken as of March 2017 in 3.x 13 | #from netmiko import ConnectHandler 14 | import netmiko 15 | import credentials # Local import of credentials.py 16 | 17 | 18 | STARTTIME = datetime.now() # Begin timing the script 19 | 20 | COMMANDLIST = [] 21 | command = "" 22 | #POOL = ThreadPool(cpu_count() - 1) # Missing from lib as of 03/2017 23 | POOL = ThreadPool() 24 | 25 | def check_config_mode(config): 26 | '''Verifies if script is running config changes or not. Redundant.''' 27 | if config is not None: 28 | logging.debug("Config Mode Enabled") 29 | return True 30 | else: 31 | logging.debug("Config Mode Disabled") 32 | return False 33 | 34 | def generate_ip_list(custdictionary): 35 | '''Return a list of IP's from the dictionary''' 36 | ip_list = [d['IP_Address'] for d in custdictionary if 'IP_Address' in d] 37 | logging.debug("IP List generated:") 38 | logging.debug(str(ip_list)) 39 | return ip_list 40 | 41 | 42 | def generate_cust_dict(customer): 43 | '''Generates a dictionary from the customer data csv file''' 44 | with open(customer, mode='r') as csvfile: 45 | reader = csv.DictReader(csvfile) 46 | data = [] 47 | for line in reader: 48 | data.append(line) 49 | logging.debug("Customer Dictionary:") 50 | logging.debug(str(data)) 51 | return data 52 | 53 | 54 | def find_by_ip(lst, value): 55 | '''Returns the row that a specific IP is in (search for row by IP)''' 56 | for row in lst: 57 | if str(row['IP_Address']) == str(value): 58 | return row 59 | 60 | 61 | def generate_switch_dict(username, password, secret, matchrow, command): 62 | '''Makes the switch dictionary for Netmiko's connection handler''' 63 | swlist = [username, password, secret, matchrow['device_type'], matchrow['IP_Address'], matchrow['SysName'], command] 64 | logging.debug("Switchlist:") 65 | logging.debug(str(swlist)) 66 | return swlist 67 | 68 | 69 | def generate_listof_lists(custdictionary, command, creds): 70 | '''Returns a list of lists from the input dictionary''' 71 | swlist = [] 72 | if creds is not None: 73 | username, password, secret = credentials.cred_csv(creds) 74 | else: 75 | username, password, secret = credentials.cred_csv() 76 | for row in custdictionary: 77 | swlist.append(generate_switch_dict(username, password, secret, row, command)) 78 | logging.debug("List of Lists:") 79 | logging.debug(str(swlist)) 80 | return swlist 81 | 82 | 83 | def switch_run_command(username, password, secret, devicetype, ipaddr, hostname, clicomm): 84 | '''All the logic happens here. Take the data, process it, print results''' 85 | sessiondict = { 86 | 'device_type': devicetype, 87 | 'ip': ipaddr, 88 | 'username': username, 89 | 'password': password, 90 | 'secret': secret, 91 | 'verbose': False 92 | } 93 | try: 94 | # Start the session, enable, send the commands, capture terminal output and remove the connections 95 | session = netmiko.ConnectHandler(**sessiondict) 96 | session.enable() 97 | session_return = session.send_command(clicomm) 98 | session.disconnect() 99 | except (netmiko.ssh_exception.NetMikoTimeoutException): 100 | session_return = "----------DEVICE CONNECTION FAILED----------" 101 | # Fancy formatting here for results 102 | print("\n\n>>>>>>>>> {0} {1} <<<<<<<<<\n".format(hostname, ipaddr) 103 | + session_return 104 | + "\n>>>>>>>>> End <<<<<<<<<\n") 105 | 106 | 107 | def switch_run_config(username, password, secret, devicetype, ipaddr, hostname, clicomm): 108 | '''All the logic happens here. Take the data, process it, print results''' 109 | sessiondict = { 110 | 'device_type': devicetype, 111 | 'ip': ipaddr, 112 | 'username': username, 113 | 'password': password, 114 | 'secret': secret, 115 | 'verbose': False 116 | } 117 | try: 118 | # Start the session, enable, send the commands, capture terminal output and remove the connections 119 | session = netmiko.ConnectHandler(**sessiondict) 120 | session.enable() 121 | session_return = session.send_config_set(COMMANDLIST) 122 | session.disconnect() 123 | except (netmiko.ssh_exception.NetMikoTimeoutException): 124 | session_return = "----------DEVICE CONNECTION FAILED----------" 125 | # Fancy formatting here for results 126 | print("\n\n>>>>>>>>> {0} {1} <<<<<<<<<\n".format(hostname, ipaddr) 127 | + session_return 128 | + "\n>>>>>>>>> End <<<<<<<<<\n") 129 | 130 | def info_command(command, csv, db, ip, creds): 131 | '''This runs a single command against all devices''' 132 | if csv is not None: 133 | switchdata = generate_cust_dict(csv) #dictionary of all switch data 134 | switchlists = generate_listof_lists(switchdata, command, creds) 135 | results = POOL.starmap(switch_run_command, switchlists) 136 | POOL.close() 137 | POOL.join() 138 | elif db is not None: 139 | # TODO 140 | print("SQL functionality is not supported at this time.") 141 | elif ip is not None: 142 | # TODO 143 | print("IP-specific functionality is not supported at this time") 144 | 145 | 146 | def config_command(config, csv, db, ip, creds): 147 | '''A variable here can be made available to a subprocess...maybe?''' 148 | if csv is not None: 149 | switchdata = generate_cust_dict(csv) #dictionary of all switch data 150 | switchlists = generate_listof_lists(switchdata, command, creds) 151 | results = POOL.starmap(switch_run_config, switchlists) 152 | POOL.close() 153 | POOL.join() 154 | elif db is not None: 155 | print("SQL functionality is not supported at this time.") 156 | elif ip is not None: 157 | print("IP-specific functionality is not supported at this time") 158 | 159 | 160 | #RESULTS = POOL.map(switch_run_command, IP_LIST) 161 | 162 | #POOL.close() 163 | #POOL.join() 164 | 165 | ENDTIME = datetime.now() 166 | TOTALTIME = ENDTIME - STARTTIME 167 | --------------------------------------------------------------------------------