├── LICENSE ├── README.md └── ping_to_netbox.py /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2017 Sean Hunter 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | 23 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | This software is designed to be used in conjunction with Netbox as a system to maintain state data for Netbox's IPAM functionality. Netbox is an excellent DCIM and "source of truth" with an REST API. 2 | 3 | # Purpose 4 | 5 | The goal of this software is to load Netbox with basic state data regarding IP addresses. Specifically, it currently pings out and also performs a reverse DNS lookup on the supplied IP addresses (either range via Netbox prefix, or existing Netbox IP address objects). 6 | 7 | The problem the software was designed to solve is an environment currently without an IPAM or using spreadsheets as an IPAM that would prefer to use a FOSS solution with state data to assist in maintaining/cleaning up their IP environment. For example, if someone brings online a new printer and neglects to inform Netbox, this tool is capable of helping to flag that up. Feel free to request features via the issue tracker here on Github. 8 | 9 | # Install 10 | 11 | Perform a pull from this Github and also make sure you have a copy of Samuel's "ping" module for python. You'll need to grab a fresh copy from here on Github as the pypi module is either outdated or another branch; it will *not* work with this code, nor will other forks, such as ping3. Trust me, I've tried. You can currently find this code here, and note that it was pulled as of 12/31/17: https://github.com/samuel/python-ping 12 | 13 | # Configuration 14 | 15 | Open ping_to_netbox.py in your favorite editor (e.g. nano) and READ THE FREAKING HEADERS! They've got big, bright "#" symbols to indicate this is a distinct, important section that you need to check out. 16 | 17 | ### Note on performance and multiprocessing 18 | The original design of this application was single-threaded. It was split into a multiprocessing model to speed scanning due to needing to hit large prefixes without waiting for timeouts on every individual IP address serially. Due to this design, scaling up the number of processes requires substantial resources, both in CPU and RAM. Testing at 1000 processes in the pool on an Ubuntu VM with 2GB of RAM yielded a completely non-responsive system that had to be cold booted to restore functionality. 19 | 20 | *DO NOT TOUCH THE THREAD POOL UNLESS YOU ARE PREPARED TO DEAL WITH A NON-RESPONSIVE SYSTEM* 21 | 22 | Performance is still not great with the current design and it may be improved in the future by moving to a newer library, such as multi-ping. 23 | 24 | ### Defaults 25 | 26 | By default, the application runs in a mode where it pulls IP address objects only from Netbox, then polls them and replaces them in Netbox. Be aware, in the current version of the application, this will destroy any existing description attached to an IP address. There are probably better ways to accomplish this. These may be adopted in the future. 27 | 28 | ### Initial load 29 | 30 | This application was designed for an environment currently operating without an IPAM. Therefore, there is a mode for reading "prefix" objects from Netbox, pinging through the entire address space, and *only saving IP address objects that respond or have a DNS entry*. Why behave this way? Netbox has convenience methods for providing new, available IP addresses upon request and provisioning them. If we create IP address objects and fill the prefix space, how will Netbox know that these addresses are available? It will not know and will think the entire prefix space has been exhausted. That's a problem, so we intentionally do not populate the entire prefix space. 31 | 32 | Also worth noting here is that there is an option for the lazy that is currently not fleshed out, which is to automatically perform a similar scan on all private IP address space. This does not currently work and trying it will fail. My advice is currently to populate Netbox with the RFC1918 space manually prior to running Netbox Status Loader in prefix mode, if that is your desired goal. Notably, prefixes can overlap in Netbox, so this should not be too big a concern. 33 | 34 | Finally, the initial load is intended to be just that. Once Netbox has your current state IP space, it is expected that it will become the "source of truth" for your network forever after that. Therefore, it is recommended to only use the default mode to pull IP addresses only from Netbox. Certainly, running in the prefix mode is possible, to continue scanning for new rogue devices, etc. However, performance of this application is already currently very poor, and performing the initial load with any reasonable frequency is likely to result in only frustration due to the time required to process it. See the above notes about performance for more detail. 35 | 36 | # Runtime 37 | 38 | Also worth noting here that this is a python2 application. I've been testing it in Ubuntu with Python 2.7.14. Testing with python3 was unsuccessful and requires changes to the ping library in addition to a few tweaks to the application itself that have not themselves been tested as adjusting the ping library was never completed. 39 | 40 | More details on how to install and use this software will be added here in the future. 41 | -------------------------------------------------------------------------------- /ping_to_netbox.py: -------------------------------------------------------------------------------- 1 | #Copyright Sean Hunter 2 | #Using MIT License (see license file for detail) 3 | 4 | import json, requests, ipaddress, dns.resolver, datetime 5 | #Python3 way (currently broken) 6 | #import ping3 7 | #Python2 way (currently the only non-broken way) 8 | import ping 9 | from multiprocessing import Pool 10 | 11 | ############################################################################# 12 | #DEAR USER - YOU MUST DEFINE EVERYTHING FROM HERE ########################### 13 | ############################################################################# 14 | 15 | #How long to wait for ping responses 16 | timeout = 2 17 | #Currently configured for a local run against a Docker Netbox install 18 | #Adjust to fit your environment (in prod, probably on port 80 or 443) 19 | ip_addresses_url = "http://localhost:32768/api/ipam/ip-addresses/" 20 | ip_prefixes_url = "http://localhost:32768/api/ipam/prefixes/" 21 | 22 | #Authorization header 23 | #This must be generated from the Netbox web UI at http://netbox/user/api-tokens/ 24 | header={"Authorization": "Token 0123456789abcdef0123456789abcdef01234567"} 25 | 26 | #Set to: 27 | #1 - Scan RFC1918 space (192.168.0.0/16, 172.16.0.0/12, 10.0.0.0/8) 28 | #2 - Scan IP addresses defined in Netbox 29 | #3 - Scan prefixes defined in Netbox 30 | load_scanner_from_rfc1918_or_netbox = 3 31 | 32 | #OK, don't touch this line 33 | myResolver = dns.resolver.Resolver() 34 | #But MAKE CERTAIN that this one fits your environment; there will be reverse 35 | #DNS requests made to this(these) server(s) 36 | myResolver.nameservers = ['192.168.0.1','172.16.0.1','10.0.0.1'] 37 | 38 | #Determines the number of simultaneous pings to execute during timeout 39 | #This can have a *very* high impact on performance, and can impact system 40 | #stability so it is advised to only mess with it if you really know what you 41 | #are doing 42 | numProcesses = 250 43 | 44 | ############################################################################# 45 | #THROUGH HERE ############################################################### 46 | ############################################################################# 47 | 48 | #NOTE: Current possible status for IP addresses in Netbox include: 49 | #1 - Active 50 | #2 - Reserved 51 | #3 - Deprecated 52 | #5 - DHCP 53 | #Don't ask what happened to 4. I assume Jeremy Stretch murdered him, the poor fellow. 54 | 55 | def threadedPingReverseSave(addr): 56 | #Drop trailing /xx 57 | ip = (addr['address'].split("/"))[0] 58 | #Run ping twice - first to ensure ARP completes before second ping goes through 59 | #this is often visible in labs where the first ping to a newly online device will fail 60 | ping.do_one(ip, timeout) 61 | rtt = ping.do_one(ip, timeout) 62 | #Create the appropriate search string for a PTR record by reversing the IP space and 63 | #adding in-addr.arpa to the query 64 | dnsreq = '.'.join(reversed(ip.split("."))) + ".in-addr.arpa" 65 | #If the ping failed... 66 | if rtt == None: 67 | print(ip + ": NO RESPONSE BEFORE ICMP TIMEOUT EXPIRY") 68 | #There's not a great answer here, so we're going to assume the IP is merely reserved 69 | #Feel free to adjust this to your needs 70 | addr['status']=2 71 | try: 72 | #Do a reverse DNS (aka pointer/PTR) query 73 | myResolver.query(dnsreq, "PTR") 74 | dnsreply = myResolver.query(dnsreq, "PTR").response.answer 75 | for i in dnsreply: 76 | for j in i.items: 77 | #Save the reply to the description 78 | addr['description'] = j.to_text().rstrip('.') 79 | #Always save IP addresses with a reverse. 80 | saveAddr(addr) 81 | except: 82 | #If there is no reply, set the description variable to... 83 | addr['description'] = "No reverse." 84 | #No ping, but it is an IP-address scan, so we should save the result 85 | #Otherwise, it's from a prefix or RFC1918 space, so do NOT save it 86 | #this allows you to programmatically pull free IP addresses as needed 87 | if load_scanner_from_rfc1918_or_netbox == 2: 88 | saveAddr(addr) 89 | #raise 90 | 91 | #If the ping succeeded... 92 | else: 93 | addr['status'] = 1 94 | try: 95 | #Do a reverse DNS (aka pointer/PTR) query 96 | myResolver.query(dnsreq, "PTR") 97 | dnsreply = myResolver.query(dnsreq, "PTR").response.answer 98 | for i in dnsreply: 99 | for j in i.items: 100 | #Save the reply to a description variable 101 | desc = j.to_text() 102 | except: 103 | #If there is no reply, set the description variable to... 104 | desc = "No reverse.." 105 | #raise 106 | print(ip + ": " + str(rtt) + "s and reverse: " + desc) 107 | #Remove the right-most period (DNS resolver returns one, e.g. example.com.) 108 | addr['description'] = desc.rstrip('.') 109 | #Always save successful pings 110 | saveAddr(addr) 111 | result = addr 112 | return result 113 | 114 | def saveAddr(addr): 115 | print("entered saveAddr for " + addr['address']) 116 | if addr['isNew'] == "new": 117 | addr.pop('isNew', None) 118 | post = requests.post(ip_addresses_url, headers=header, json=addr) 119 | print(post.status_code) 120 | print(post.json()) 121 | elif addr['isNew'] == "old": 122 | addr.pop('isNew', None) 123 | try: 124 | role = addr['role']['value'] 125 | addr.pop('role', None) 126 | addr['role'] = role 127 | except: 128 | #Do nothing 129 | print("Role exception.") 130 | post = requests.put(ip_addresses_url + str(addr['id']) + "/", headers=header, json=addr) 131 | print(post.status_code) 132 | print(post.json()) 133 | return addr 134 | 135 | def mergeWithExisting(addr, listOfIps): 136 | for ip_addr in listOfIps: 137 | print("IP to match: " + addr['address'] + " and checking against IP: " + ip_addr['address']) 138 | if ip_addr['address'] == addr['address']: 139 | print("MATCH: " + json.dumps(ip_addr, indent=4)) 140 | #Get 'id' from existing and stuff it in addr 141 | print(ip_addr['id']) 142 | addr['id'] = ip_addr['id'] 143 | addr['isNew'] = "old" 144 | return addr 145 | 146 | if __name__ == '__main__': 147 | start = datetime.datetime.now() 148 | if load_scanner_from_rfc1918_or_netbox == 1: 149 | #Load from RFC1918 150 | print("Not currently loading addresses from RFC1918, even though I was told to!!") 151 | if load_scanner_from_rfc1918_or_netbox == 2: 152 | #GET IP addresses from Netbox 153 | response = requests.get(ip_addresses_url, headers = header) 154 | listOfIpsWithMask = response.json()['results'] 155 | while response.json()['next'] is not None: 156 | response = requests.get(response.json()['next'], headers = header) 157 | for ip in response.json()['results']: 158 | listOfIpsWithMask.append(ip) 159 | for ipaddr in listOfIpsWithMask: 160 | ipaddr['isNew'] = "old" 161 | print("Populated from list of existing IP addresses. No new ones will be scanned.") 162 | if load_scanner_from_rfc1918_or_netbox == 3: 163 | #GET IP prefixes from Netbox 164 | response = requests.get(ip_prefixes_url, headers = header) 165 | listOfPrefixes = response.json()['results'] 166 | while response.json()['next'] is not None: 167 | response = requests.get(ip_prefixes_url, headers = header) 168 | for prefix in response.json()['results']: 169 | listOfPrefixes.append(prefix) 170 | print(json.dumps(listOfPrefixes, indent=4)) 171 | listOfIps = [] 172 | for prefixObj in listOfPrefixes: 173 | print(prefixObj['prefix']) 174 | #print(ipaddress.ip_network(prefixObj['prefix']).hosts()) 175 | mask = prefixObj['prefix'].split("/")[1] 176 | for host in ipaddress.ip_network(prefixObj['prefix']).hosts(): 177 | listOfIps.append({"address": str(host) + "/" + mask}) 178 | listOfIpsWithMask = listOfIps 179 | print("List of IP's with mask: ") 180 | print(listOfIpsWithMask) 181 | print("List of existing IP addresses: ") 182 | ip_result = requests.get(ip_addresses_url, headers = header) 183 | listOfIps = ip_result.json()['results'] 184 | while ip_result.json()['next'] is not None: 185 | ip_result = requests.get(ip_result.json()['next'], headers = header) 186 | for ip in ip_result.json()['results']: 187 | listOfIps.append(ip) 188 | print(json.dumps(listOfIps, indent=4)) 189 | for ipaddr in listOfIpsWithMask: 190 | ipaddr['isNew'] = "new" 191 | ipaddr = mergeWithExisting(ipaddr, listOfIps) 192 | print(json.dumps(ipaddr, indent=4)) 193 | # print "Stopping..." + None 194 | #Pretty print new copy for debugging 195 | print(json.dumps(listOfIpsWithMask, indent=4)) 196 | #Multithread the process. This isn't to speed up "processing" but eliminate the 197 | #wait on timeout expiry; adjust to speed up/slow down system. 198 | pool = Pool(processes=numProcesses) 199 | #Here, we call isPingable for every entity in listOfIpsWithMask 200 | result = pool.map(threadedPingReverseSave, listOfIpsWithMask) 201 | #Presently looking at solutions that can sort this mess by IP address... currently results are unsorted. 202 | print(result) 203 | print(json.dumps(result, indent=4)) 204 | # r = requests.post(ip_addresses_url, headers=header, json={"address": "192.168.3.2", "status": "1"}) 205 | # print r.status_code 206 | # print r.json() 207 | finish = datetime.datetime.now() 208 | print("Completed in: " + str(finish - start)) 209 | --------------------------------------------------------------------------------