├── AUTHORS ├── LICENSE ├── README.md ├── __init__.py ├── base-query.swql.example └── netmonkey.py /AUTHORS: -------------------------------------------------------------------------------- 1 | Austin de Coup-Crank 2 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2017 Austin de Coup-Crank 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # netmonkey 2 | 3 | A simple Python framework for managing network devices, leveraging `netmiko` and the SolarWinds Orion API. 4 | 5 | **Full disclosure**: This is not production-ready code. It's a toy project I started to teach myself Python, netmiko, and git, and to make my day job a bit less cumbersome. This is probably most immediately useful to someone also seeking to learn Python and network automation. If you need stable, production-ready code, have a look at `NAPALM`. 6 | 7 | ## Features 8 | 9 | * Detect ssh/telnet support on-the-fly (yes, a few of our devices still use telnet...welcome to K12) 10 | * Request credentials at run-time, storing only in memory (nothing stored to disk) 11 | * Leverages `multiprocessing.Pool()` to run everything in parallel, massively speeding up operations on multiple devices, with a neat little progress bar, thanks to `tqdm` 12 | * Provide several ways of supplying hostnames/IPs: 13 | * Ad-hoc single host (as a `string`, e.g. `'172.16.35.240'`) 14 | * Ad-hoc multiple hosts (as a `list`, e.g. `['rtr1', '10.250.2.4', 'sw02']`) 15 | * Hosts from a plaintext file, one hostname/IP per line 16 | * Retrieved directly from our NPM (SolarWinds Orion) 17 | * Run arbitrary show commands against any number of hosts, returning the output in an object format suitable for futher parsing 18 | * Run arbitrary one-line configuration commands against any number of hosts, returning the output 19 | * Run custom functions that do whatever I want, parallelized against any number of hosts, with custom return codes and messages 20 | * Abstract away all the junk of requesting credentials, creating the `netmiko` session objects, handling exceptions, etc. 21 | 22 | ## Requires 23 | 24 | * Python 2.7 on unix-like OS (Linux, MacOS). Likely wouldn't take much work to add Windows support. 25 | * [netmiko](https://github.com/ktbyers/netmiko) - Multi-vendor network device SSH library 26 | * [tqdm](https://github.com/tqdm/tqdm) - Simple and extensible progress bars 27 | * [orionsdk](https://github.com/solarwinds/OrionSDK) - Query SolarWinds Orion for devices using SWQL 28 | 29 | ## Usage 30 | 31 | ### Run a `show` command 32 | 33 | **Worth mentioning**: SNMP is generally a faster way to get info. This is just for fun. 34 | 35 | `netmonkey.show()` accepts two paramters: the `show` command to run, and device(s) to run against. The method automatically prepends "show " to the command, so `show ip int br` would be called as `netmonkey.show('ip int br', 'rtr1')` 36 | ```py 37 | >>> from netmonkey import netmonkey 38 | >>> print netmonkey.show('snmp location', ['rtr01', 'sw02']) 39 | Network username [austind]: 40 | Network password: 41 | Telnet password: # Ignore this...don't ask. 42 | Enable secret: 43 | Progress: 100%|################################| 2/2 [00:06<00:00, 2.00Device/s] 44 | [{'rtr01': {'status': 0, 'message': u'MDF', 'port': 22}}, {'sw02': {'status': 0, 'message': u'Room 615a', 'port': 22}}] 45 | >>> 46 | ``` 47 | 48 | Or use a text file to supply a list of hosts. 49 | ``` 50 | cat hosts.txt 51 | rtr01 52 | 10.250.2.3 53 | sw02 54 | 55 | python2.7 56 | >>> from netmonkey import netmonkey 57 | >>> print netmonkey.show('snmp location', 'hosts.txt') 58 | # output omitted 59 | ``` 60 | 61 | ### Retrieve devices from SolarWinds Orion 62 | 63 | #### Setup 64 | 65 | 1. **Install API TLS cert**. The API uses its own self-signed certificate, even if you have a valid certificate for Orion (not sure why?) See [this guide](https://github.com/solarwinds/orionsdk-python#ssl-certificate-verification) for how to set it up. 66 | 1. **Edit base-query.swql**. There is a `base-query.swql.example`, use that as a starting point. See the [official docs](https://support.solarwinds.com/Success_Center/Network_Performance_Monitor_%28NPM%29/How_to_use_SolarWinds_Query_Language_%28SWQL%29) for more info. 67 | 1. Adjust `get_devices()` as necessary. I have it set to parse specific custom attributes easily. You will need to adapt it to your environment to be useful. 68 | 69 | #### Basic usage 70 | 71 | ```py 72 | netmonkey.get_devices(name="sw*") # Retrieves all devices with hostnames beginning with 'sw' 73 | ``` 74 | 75 | ### Combine the two 76 | 77 | ``` 78 | >>> print netmonkey.show('snmp location', netmonkey.get_devices(name="sw*")) 79 | Orion username [austind]: 80 | Orion password: 81 | Network username [austind]: 82 | Network password: 83 | Telnet password: 84 | Enable secret: 85 | Progress: 100%|#########################################################| 7/7 [00:06<00:00, 13.97Device/s] 86 | [{u'sw01': {'status': 0, 'message': u'Room 9 (IDF3)', 'port': 22}}, {u'sw02': {'status': 0, 'message': u'MDF1A', 'port': 22}}, {u'sw03': {'status': 0, 'message': u'MDF', 'port': 22}}, {u'sw04': {'status': 0, 'message': u'MDF 1B', 'port': 22}}, {u'sw05': {'status': 0, 'message': u'Room 16 (IDF2)', 'port': 22}}, {u'sw06': {'status': 0, 'message': u'Gym Closet West (IDF1)', 'port': 22}}, {u'sw07': {'status': 0, 'message': u'MDF 1C', 'port': 22}}] 87 | ``` 88 | 89 | I just pulled data from 7 devices in one line of Python in 6 seconds. Not too shabby! 90 | 91 | Output follows a list/dictionary format to make it easier for further programmatic parsing: 92 | 93 | ```py 94 | [ 95 | { 96 | 'host1' = { 97 | 'status': 0 # Status codes. See source of netmonkey.command() for all code meanings 98 | 'message': "Sample output" # Output from successful command, or error message from exception 99 | 'port': 22 # Port connected to (None if no connection made) 100 | } 101 | ] 102 | ``` 103 | 104 | ### Human-readable output 105 | 106 | Maybe you don't want to further parse the output, and you just want to see it in a human-readable format. 107 | 108 | ``` 109 | >>> results = netmonkey.show('snmp location', netmonkey.get_devices(name="sw*")) 110 | # password prompts and progress bar omitted 111 | >>> netmonkey.print_results(results) 112 | sw01:22 - [0] Room 9 (IDF3) 113 | sw02:22 - [0] MDF1A 114 | sw03:22 - [0] MDF 115 | sw04:22 - [0] MDF 1B 116 | sw05:22 - [0] Room 16 (IDF2) 117 | sw06:22 - [0] Gym Closet West (IDF1) 118 | sw07:22 - [0] MDF 1C 119 | ``` 120 | 121 | ### One-line config changes 122 | 123 | Follow the same pattern as above for show commands, except with `netmonkey.config()` instead. This will: 124 | * Connect to the host 125 | * Enter enable mode 126 | * Enter config mode 127 | * Send config change 128 | * Write config 129 | * Backup config via TFTP (using in-house backup alias) 130 | * Disconnect from the host 131 | 132 | **WARNING:** Please review the source for `netmonkey.config()` before using this in production. I call a custom backup alias that you will need to change or add before using. 133 | 134 | ```py 135 | >>> netmonkey.config('clock timezone PST -8 0', netmonkey.get_devices(name="sw*")) # Output omitted 136 | ``` 137 | 138 | ### Custom functions 139 | 140 | Running single show/config commands are cool and all, but what about doing some real custom work? What if you need to include some logic, applying certain values only if other values exist? 141 | 142 | You can define your own function as if it were being run against a single `netmiko` session object, then call the function with `netmonkey.run()`, optionally pairing it with `netmonkey.get_devices()`, to run it in parallel against dozens or hundreds of hosts. 143 | 144 | ```py 145 | from netmonkey import netmonkey 146 | 147 | # For now, the only arguments custom methods accept is the session object. 148 | # Adding custom arguments is coming in a future release. 149 | def new_user(session): 150 | 151 | # Enter enable mode 152 | session.enable() 153 | 154 | # All of `netmiko`'s methods are available (see https://pynet.twb-tech.com/blog/automation/netmiko.html) 155 | users = session.send_command('show running-config | include username') 156 | 157 | # Only move forward if 'newuser' isn't already in the running-config 158 | if 'newuser' not in users: 159 | 160 | # Enter config mode 161 | session.config_mode() 162 | 163 | output = session.send_command('username newuser secret 5 $1$Z30o$.......') 164 | 165 | # Write changes to flash 166 | session.send_command('copy running-config startup-config') 167 | 168 | # Call backup alias (custom to my environment) 169 | session.send_command('backup') 170 | 171 | # Return values are a tuple of ([int]statuscode, [string]message) 172 | # Status codes 0-6 are reserved, use anything from 7+ 173 | return (7, "Added user 'newuser' successfully.") 174 | else: 175 | return (0, "User 'newuser' already present in config.") 176 | 177 | # Make a list of devices you want 'newuser' added to 178 | devices = ['sw01', 'rtr01'] 179 | 180 | # Or like in my case, I want 'newuser' on all devices 181 | devices = netmonkey.get_devices(name='*') 182 | 183 | # Call our custom new_user() method, parallelizing across all hosts 184 | results = netmonkey.run(new_user, devices) 185 | 186 | # Print results in human-readable format, skipping devices that already have 'newuser' 187 | netmonkey.print_results(results, 1) # 1 is the minimum status code that will be displayed (ignores 0) 188 | ``` 189 | 190 | The output will be a human-readable list of only devices that either successfully added 'newuser', or had a problem (e.g., not network-reachable, no open SSH/telnet ports, bad credentials) 191 | 192 | ### Background 193 | 194 | I'm a junior network engineer in the K12 space, managing ~450 Cisco devices, spanning several disparate networks that we oversee. I found out very quickly that I would hate my job if I didn't find a way to somehow automate changes. You might say I wanted to take the monkey work out of my job ;) 195 | 196 | I signed up for @ktbyers Python training course after finding it on /r/networking. I set up a dev environment on our Linux jumphost and tinkered with netmiko. Its power was clear to me, but all by itself, it's a bit unwieldy. I didn't want to create device objects by hand, or store credentials in plaintext, among other things. 197 | 198 | 199 | By the time I had a decent start, I looked more closely at [NAPALM](https://github.com/napalm-automation/napalm) and discovered I was basically re-implementing a very mature and robust framework, poorly. NAPALM provides a wrapper around netmiko (for IOS at least) providing some really awesome parsing methods. A NAPALM connection object gives you easy access to all kinds of info like device facts, port statistics, port info, NTP status, and more. 200 | 201 | As it stands, the `napalm-ios` module doesn't support transport selection (for dynamically selecting telnet over ssh). Once 0.08 drops, that will become an option, and I will very likely refactor netmonkey to use NAPALM objects instead of straight netmiko objects. 202 | 203 | ### Roadmap 204 | 205 | High on my priorities are: 206 | * Loosely coupling everything to my environment as much as possible, hopefully making it more directly useful to others 207 | * Better SWQL syntax options 208 | * Running arbitrary batches of commands from a text file (instead of only one-line config commands) 209 | * Better testing and exception handling 210 | 211 | ### Limitations 212 | 213 | **Disclaimer:** This is a pet project in its infancy. Its main purpose is to give me a learning experience to hack away with, while also making my day job easier. Feel free to contribute or make suggestions, but don't expect miracles. You are responsible for testing, etc. 214 | 215 | * **Tightly coupled to Cisco IOS.** We are a 99% Cisco shop, so I didn't abstract the code on my first run. Since netmiko supports many vendors, this will be easy to change. 216 | * **Not extensively tested.** Don't expect miracles, use at your own risk. 217 | * **Assumes SolarWinds Orion.** Uses SWQL to query Orion for devices based on custom properties for my environment. Have a look at the `get_devices()` method for info on how to set it up, and how you might adapt it to your environment. 218 | 219 | ### Known Issues 220 | 221 | Most common exception types are handled gracefully, but recently I saw a random NetMikoTimeoutException when processing a large batch of hosts. The error was transient, so I couldn't dive deeper. 222 | 223 | ### Conventions 224 | 225 | * Wherever the keyword or variable `host` is used in the source, this refers to any routable identifier for a device: IP address, hostname, or FQDN. You can use all 3 options interchangeably. Of course, hostnames and FQDNs need to resolve properly. 226 | * All functions accept single hostnames (strings), lists, text files with one host per line, or custom SWQL queries 227 | 228 | 229 | -------------------------------------------------------------------------------- /__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/austind/netmonkey/7a21add801ad6db4e87f3242d4b6a1e9db7d6e7d/__init__.py -------------------------------------------------------------------------------- /base-query.swql.example: -------------------------------------------------------------------------------- 1 | SELECT Caption, IPAddress, Location 2 | FROM Orion.Nodes 3 | WHERE Vendor = 'Cisco' 4 | -------------------------------------------------------------------------------- /netmonkey.py: -------------------------------------------------------------------------------- 1 | import os 2 | import netmiko 3 | from paramiko.ssh_exception import SSHException 4 | import orionsdk 5 | from getpass import (getpass, getuser) 6 | from distutils.util import strtobool 7 | import urllib3 8 | import multiprocessing 9 | from tqdm import tqdm 10 | import re 11 | from time import sleep 12 | import socket 13 | 14 | # Usage of globals is bad, I know. I use them so I don't have to keep 15 | # re-entering my creds with every function call, and without needing 16 | # to store plaintext creds. I can't use keyring because I haven't 17 | # figured out how to use it on my headless jumphost. 18 | network_username = '' 19 | network_password = '' 20 | telnet_password = '' 21 | secret = '' 22 | orion_username = '' 23 | orion_password = '' 24 | 25 | # Number of threads for concurrent sessions. 26 | # Typically, multiprocessing pools are set to use the number of cores 27 | # available on the server. However, in our case, the threads are not 28 | # CPU-intensive, so we can get away with a lot more. Experimentation has shown 29 | # that 50 strikes a good balance. Fewer means slower performance, more means 30 | # maxing out system resources. 31 | THREADS = 50 32 | 33 | 34 | class NetmonkeyError(Exception): 35 | """ Base class for exceptions in this module. """ 36 | pass 37 | 38 | class HostOfflineError(NetmonkeyError): 39 | """ Raised when an attempt is made to connect to a host that is not 40 | network reachable. 41 | 42 | Attributes: 43 | host -- the hostname or IP that was attempted 44 | msg -- explanation of the error 45 | """ 46 | 47 | def __init__(self, host, msg): 48 | self.host = host 49 | self.msg = msg 50 | 51 | class NoOpenPortError(NetmonkeyError): 52 | """ Raised when an attempt is made to connect to a host that does 53 | not have either SSH or telnet available (ports 22 or 23, respectively). 54 | 55 | Attributes: 56 | host -- the hostname or IP that was attempted 57 | msg -- explanation of the error 58 | """ 59 | 60 | def __init__(self, host, msg): 61 | self.host = host 62 | self.msg = msg 63 | 64 | class InvalidCommandTypeError(NetmonkeyError): 65 | """ Raised when a command type is given that is not either 'show' or 'config' 66 | """ 67 | 68 | def __init__(self, msg): 69 | self.msg = msg 70 | 71 | # http://mattoc.com/python-yes-no-prompt-cli.html 72 | def prompt(query): 73 | """ Returns boolean for y/n input """ 74 | print '%s [y/n]: ' % query 75 | val = raw_input() 76 | try: 77 | ret = strtobool(val) 78 | except ValueError: 79 | print 'Reply with y/n' 80 | return prompt(query) 81 | return ret 82 | 83 | def get_creds(): 84 | """ Prompts for credentials that are stored in global variables for reuse 85 | """ 86 | global default_username 87 | global network_username 88 | global network_password 89 | global telnet_password 90 | global secret 91 | default_username = getuser() 92 | if not network_username or not network_password or not telnet_password or not secret: 93 | network_username = raw_input('Network username [' + default_username + ']: ') or default_username 94 | network_password = getpass('Network password: ') 95 | telnet_password = getpass('Telnet password: ') or None 96 | secret = getpass('Enable secret: ') 97 | 98 | def orion_init(): 99 | """ Prompts for Orion credentials and returns a SwisClient object 100 | """ 101 | global orion_server 102 | global orion_username 103 | global orion_password 104 | if not orion_username: 105 | default_username = getuser() 106 | orion_username = raw_input('Orion username [' + default_username + ']: ') or default_username 107 | if not orion_password: 108 | orion_password = getpass('Orion password: ') 109 | # SolarWinds-Orion is a special hostname in /etc/hosts 110 | # This was necessary to implement SSL checking 111 | # https://github.com/solarwinds/orionsdk-python#ssl-certificate-verification 112 | # this disables the SubjectAltNameWarning 113 | urllib3.disable_warnings() 114 | # TODO: Need a better/more resilient way of referencing server cert. 115 | return orionsdk.SwisClient('SolarWinds-Orion', orion_username, orion_password, verify='server.pem') 116 | 117 | def get_devices(*args, **kwargs): 118 | """ Retrieve a list of hosts for later use. 119 | 120 | TODO: 121 | Ideally this will be able to handle: 122 | - An SWQL query directly from Orion 123 | - A CSV 124 | - A flat text file of hostnames/IPs 125 | """ 126 | 127 | if args: 128 | # Hosts file 129 | if os.path.isfile(str(args[0])): 130 | host_file = open(args[0], 'rb') 131 | host_list = host_file.read().splitlines() 132 | host_file.close() 133 | return host_list 134 | 135 | # Single host 136 | elif type(args[0]) is str: 137 | # Returns as a single-item list 138 | # This is important since most methods below iterate, 139 | # and a string is not iterable. 140 | return args[0].split() 141 | 142 | # If hosts given as list, simply return that list again 143 | elif type(args[0]) is list or type(args[0]) is dict: 144 | return args[0] 145 | 146 | # Otherwise assume the source is a SWQL query 147 | elif kwargs: 148 | # Initialize Orion connection 149 | swis = orion_init() 150 | 151 | # Read base query from file 152 | # TODO: This seems wrong. Not sure how to abstract the base SWQL query 153 | # properly. Should it be a file or somehow hardcoded? 154 | module_dir = os.path.split(os.path.abspath(__file__))[0] 155 | base_query_file = open(module_dir + '/base-query.swql') 156 | query = base_query_file.read() 157 | base_query_file.close() 158 | 159 | # Can't append this to the query until we have all filters in place 160 | query_order = "ORDER BY Caption\n" 161 | 162 | query += 'AND ' 163 | query_filter = [] 164 | district = kwargs.get('district') 165 | site = kwargs.get('site') 166 | name = kwargs.get('name') 167 | if district: 168 | query_filter.append("Nodes.CustomProperties.School_District = '%s'\n" % district) 169 | if site: 170 | query_filter.append("Nodes.CustomProperties.School_Site = '%s'\n" % site) 171 | if name: 172 | name = name.replace('*', '%') 173 | query_filter.append("Caption LIKE '%s'\n" % name) 174 | query += ' AND '.join(query_filter) 175 | query += query_order 176 | return (swis.query(query))['results'] 177 | 178 | def is_online(host): 179 | """ Pings hostname once and returns true if response received. """ 180 | # TODO: Investigate refactoring this using socket instead of os.system 181 | # '> /dev/null 2>&1' redirects stderr to stdout, and stdout to null. 182 | # In other words, print nothing at all. 183 | response = os.system('ping -q -c 1 ' + host + ' > /dev/null 2>&1') 184 | if response == 0: 185 | return True 186 | else: 187 | raise HostOfflineError(host, 'Host is not network-reachable.') 188 | return False 189 | 190 | def check_proto(host): 191 | """ Checks if port 22 or 23 is open, and returns open port number. """ 192 | check = socket.socket() 193 | response = check.connect_ex((host, 22)) 194 | if response == 0: 195 | return {'port': 22, 'name': 'ssh'} 196 | else: 197 | response = check.connect_ex((host, 23)) 198 | if response == 0: 199 | return {'port': 23, 'name': 'telnet'} 200 | else: 201 | raise NoOpenPortError(host, 'Neither port 22 nor 23 is open.') 202 | 203 | def sanitize_host(host): 204 | """ Removes extraneous characters from a hostname, leaves an IP untouched. """ 205 | if re.match(r'[a-zA-Z]', host): 206 | return host.split()[0].strip().replace('_', '-').replace('.', '') 207 | else: 208 | return host 209 | 210 | def connect(host): 211 | """ Opens an SSH/telnet session with an online host. 212 | 213 | Returns session object, or exception if none could be created. 214 | """ 215 | global network_username 216 | global network_password 217 | global telnet_password 218 | global secret 219 | get_creds() 220 | if is_online(host): 221 | open_proto = check_proto(host) 222 | if open_proto: 223 | device = { 224 | 'device_type': 'cisco_ios_' + open_proto['name'], 225 | 'ip': host, 226 | 'username': network_username, 227 | 'password': network_password, 228 | 'secret': secret, 229 | 'port': open_proto['port'], 230 | 'verbose': False 231 | } 232 | try: 233 | session = netmiko.ConnectHandler(**device) 234 | if session: 235 | session.enable() 236 | return session 237 | except netmiko.ssh_exception.NetMikoAuthenticationException: 238 | # If my creds are rejected, try the generic telnet password 239 | device['password'] = telnet_password 240 | try: 241 | return netmiko.ConnectHandler(**device) 242 | except netmiko.ssh_exception.NetMikoAuthenticationException: 243 | # If we still can't log in, nothing more to try 244 | raise 245 | except: 246 | raise 247 | 248 | def write_config(session): 249 | if session.check_config_mode(): 250 | session.exit_config_mode() 251 | return session.send_command_expect('copy running-config startup-config') 252 | 253 | def backup_config(session): 254 | """ Runs custom alias to back up config via TFTP 255 | 256 | Reference: https://github.com/ktbyers/netmiko/issues/330 257 | """ 258 | output = session.send_command_timing('backup') 259 | 260 | # Our backup alias expects two [enter] keystrokes 261 | if '?' in output: 262 | output += session.send_command_timing('\n') 263 | output += session.send_command_timing('\n') 264 | return output 265 | 266 | def show(cmd, target): 267 | """ Returns aggregate output for one or more show commands. """ 268 | return batch(target, command, ['show', cmd]) 269 | 270 | def config(cmd, target): 271 | """ Writes one or more configuration commands and returns aggregate output. """ 272 | return batch(target, command, ['config', cmd]) 273 | 274 | def run(function, target): 275 | """ Runs a custom function in parallel and returns aggregate output. """ 276 | return batch(target, command, ['fn', function]) 277 | 278 | def print_results(results, errlvl=0): 279 | """ Prints results from command() in human-readable format. 280 | Mostly useful for debugging. 281 | 282 | """ 283 | for result in results: 284 | for hostname, output in result.iteritems(): 285 | if output['status'] >= errlvl: 286 | print "%s:%s - [%s] %s" % (hostname, output['port'], output['status'], output['message']) 287 | 288 | def save_results(results, filename, errlvl=0): 289 | """ Saves results to a file 290 | 291 | Resulting filename can be supplied to get_devices() to make future 292 | operations quicker. 293 | 294 | The behavior of this method is likely to change in the future. 295 | 296 | For now, it takes a results list and saves it as a plaintext file with 297 | one host per line. 298 | 299 | WARNING: Overwrites 'filename' without prompt or warning 300 | """ 301 | f = open(filename, 'w') 302 | 303 | for result in results: 304 | for hostname, output in result.iteritems(): 305 | if output['status'] >= errlvl: 306 | f.write(hostname + '\n') 307 | 308 | f.close() 309 | 310 | def command(target, cmd_type, cmd, result_list=None): 311 | """ Runs arbitrary commands or functions against a single target device. """ 312 | 313 | # TODO 314 | # If command() is called by batch(), batch() absolutely needs command() 315 | # to append *something* to result_list, or else it will stay stuck waiting 316 | # for results. 317 | 318 | # However, if command() is called independently of batch(), we don't care 319 | # and just want the output returned. I tried several methods of accommodating 320 | # both cases, but ended up with this. If result_list hasn't been passed, 321 | # create an empty list and append output to it. 322 | if result_list == None: 323 | result_list = [] 324 | 325 | # Return data structure: 326 | # hostname = {port, status, message} 327 | # Port 328 | # - 22 or 329 | # - 23 330 | # Status: 331 | # - 0: Success 332 | # - 1: Host offline 333 | # - 2: Ports 22 and 23 are both closed 334 | # - 3: Invalid credentials 335 | # - 4: SSH exception 336 | # - 5: ValueError from Netmiko, usually bad enable secret 337 | # - 6+: Available to custom functions 338 | # Message: 339 | # - if status == 0, the result of command(s) given 340 | # - if status != 0, description of error 341 | 342 | return_data = {} 343 | 344 | if cmd_type not in ['show', 'config', 'fn']: 345 | raise InvalidCommandTypeError('cmd_type must be either "show", "config", or "fn"') 346 | 347 | # TODO: If cmd is a file, send batch file 348 | 349 | # If we received the target via SWQL query, we need to parse the dictionary 350 | # to get the actual hostname or IP. 351 | # The other fields of the query such as Location are not used here, but could 352 | # be used if they were ever helpful. 353 | if type(target) is dict: 354 | host = target['Caption'] 355 | ipaddress = target['IPAddress'] 356 | location = target['Location'] 357 | else: 358 | host = target 359 | 360 | session = None 361 | 362 | # Orion sometimes contains invalid characters in hostnames. 363 | host = sanitize_host(host) 364 | 365 | try: 366 | session = connect(host) 367 | 368 | except HostOfflineError as e: 369 | # Error 1: Host is not network-reachable. 370 | return_data[host] = { 371 | 'port': None, 372 | 'status': 1, 373 | 'message': e.msg 374 | } 375 | pass 376 | 377 | except NoOpenPortError as e: 378 | # Error 2: Neither port 22 nor 23 is open. 379 | return_data[host] = { 380 | 'port': None, 381 | 'status': 2, 382 | 'message': e.msg 383 | } 384 | pass 385 | 386 | except netmiko.ssh_exception.NetMikoAuthenticationException as e: 387 | # Error 3: Authentication failure - provided credentials rejected. 388 | 389 | # I can get the port from the session object if it's created, 390 | # but if there is no session object, I have to infer the port that 391 | # was attempted from the authentication exception. 392 | # This seems ugly to me, it seems even uglier to me to do something 393 | # with connect(), which, I feel, behaves as it should. 394 | # Otherwise, check to see if we are parsing by district/site 395 | 396 | error = str(e).replace('\n', '-') 397 | if ':22' in error: 398 | port = 22 399 | if 'Telnet' in error: 400 | port = 23 401 | return_data[host] = { 402 | 'port': port, 403 | 'status': 3, 404 | 'message': 'Authentication failure - provided credentials rejected.' 405 | } 406 | pass 407 | 408 | except netmiko.ssh_exception.NetMikoTimeoutException as e: 409 | # Error 3: Timeout 410 | return_data[host] = { 411 | 'port': None, 412 | 'status': 4, 413 | 'message': str(e) 414 | } 415 | 416 | except SSHException as e: 417 | # Error 4: SSH exception 418 | return_data[host] = { 419 | 'port': 22, 420 | 'status': 5, 421 | 'message': str(e) 422 | } 423 | pass 424 | 425 | except ValueError as e: 426 | # Error 5: Bad enable secret 427 | return_data[host] = { 428 | 'port': None, 429 | 'status': 6, 430 | 'message': str(e) 431 | } 432 | pass 433 | 434 | if session: 435 | # If we have been passed an actual function as cmd_type, call that 436 | # function with the session variable passed. 437 | # TODO: It might be nice to allow custom functions to be passed 438 | # arbitrary arguments, not just the session object. 439 | if cmd_type == 'fn': 440 | output = cmd(session) 441 | status, message = output 442 | # Otherwise, proceed to run the commands literally as show or config 443 | # commands. 444 | else: 445 | session.enable() 446 | if cmd_type == 'config': 447 | session.config_mode() 448 | if cmd_type == 'show': 449 | cmd = 'show ' + cmd 450 | output = session.send_command(cmd) 451 | if cmd_type == 'config': 452 | write_config(session) 453 | backup_config(session) 454 | session.disconnect() 455 | status = 0 456 | message = output 457 | 458 | return_data[host] = { 459 | # Status 0: Success or no changes made. 460 | 'port': session.port, 461 | 'status': status, 462 | 'message': message 463 | } 464 | 465 | result_list.append(return_data) 466 | return return_data 467 | 468 | def batch(targets, worker, argument_list=None, threads=THREADS): 469 | """ Parallelizes an arbitrary function against arbitrary target devices. 470 | 471 | When running commands against hundreds or thousands of devices, 472 | creating simultaneous sessions would exhaust system resources. 473 | This approach ensures that resources are used maximally, neither 474 | under-using or over-using them. 475 | 476 | """ 477 | # Get credentials and devices 478 | get_creds() 479 | targets = get_devices(targets) 480 | 481 | # Initialize work pool 482 | pool = multiprocessing.Pool(threads) 483 | manager = multiprocessing.Manager() 484 | result_list = manager.list() 485 | 486 | # For all targets, add a job to the pool 487 | jobs = [] 488 | for target in targets: 489 | if not argument_list: 490 | worker_args = [target, result_list] 491 | else: 492 | # If we have been passed an argument list, we need to insert 493 | # that into the arguments we pass to the worker function 494 | worker_args = argument_list[:] 495 | worker_args.insert(0, target) 496 | worker_args.append(result_list) 497 | jobs.append(tuple(worker_args)) 498 | 499 | results = [pool.apply_async(worker, args = j) for j in jobs] 500 | 501 | # Closing the pool means no other jobs will be submitted 502 | pool.close() 503 | 504 | for result in tqdm(results, desc='Progress', unit='Device', ascii=True): 505 | result.get(timeout=120) 506 | 507 | 508 | # Progress bar 509 | 510 | # tqdm() is much simpler if you can call an iterable with it. 511 | # As far as I can tell, nothing about the pool semantics above 512 | # use an iterable that actually represents the progress of work done. 513 | # The one loop above only queues jobs, it doesn't acutally track them. 514 | # This construct below makes our own progress bar. 515 | 516 | # TODO: This just isn't acceptable. There needs to be some kind of timeout 517 | # so a single failed thread doesn't hang the whole job. I know there is 518 | # a timeout argument in the get() method of queues, look into that. 519 | #pbar = tqdm(total=len(targets), desc='Progress', unit='Device', ascii=True) 520 | #progress = 0 521 | #while len(result_list) <= len(targets): 522 | # pbar.update(len(result_list) - progress) 523 | # progress = len(result_list) 524 | # if progress == len(targets): 525 | # break 526 | # # Check for progress updates every half second 527 | # sleep(0.5) 528 | #pbar.close() 529 | 530 | return result_list 531 | --------------------------------------------------------------------------------