4 |
5 |
6 | 
7 |
8 |
9 |
netbox-sync-physical-hosts
10 | 11 |
12 | Scan your network and populate info to Netbox, fast and reliable
13 |
14 | Report Bug
15 | ·
16 | Request Feature
17 |
21 |
22 |
48 |
49 |
50 | ## About The Project
51 | The project is meant to be as stable as robust as possible.
52 | There's a reason behind Netbox project not wanting to create a scanner, if you use Netbox it is recommended
53 | to be your `source of truth`. In order to make that statement true we need to make sure Netbox doesn't contain
54 | outdated, not acurate or not useful info.
55 | This script aims to keep things as simple as possible and pre-populate info into Netbox to make your life easier.
56 |
57 | It's recommended to first run the script pointing at a dev instance of Netbox first. This project tries to populate
58 | info in a safe way but there's never 100% certainty and things can get ugly if you run this script directly in prod
59 | and something goes wrong.
60 | Because automated source of truth can be handy sometimes ;-)
61 |
62 | ### Built With
63 |
64 | * [Python](https://www.python.org/)
65 | * [PyNetbox](https://github.com/digitalocean/pynetbox)
66 | * [python3-nmap](https://pypi.org/project/python3-nmap/)
67 |
68 |
69 |
70 | ## Getting Started
71 |
72 | To get a local copy up and running follow these simple steps.
73 |
74 | ### Prerequisites
75 |
76 | This script works with Netbox >= 2.9 and python >=3.6
77 | In order to run the software you just need to install the requirement.
78 | * python >= 3.6
79 | ```sh
80 | pip install -r requirements.txt
81 | ```
82 |
83 | ### Installation
84 |
85 | 1. Clone the repo
86 | ```sh
87 | git clone https://github.com/guanana/netbox-sync-physical-hosts.git
88 | ```
89 | 2. Install python packages
90 | ```sh
91 | pip install -r requirements.txt
92 | ```
93 |
94 |
95 |
96 | ## Usage
97 |
98 | The script can be run with multiple configuration options.
99 | Most of the configuration options can be overwrite using environment variables
100 | ```buildoutcfg
101 | [GENERAL]
102 | cleanup: false
103 | tag: nmap-sync
104 |
105 | [NETBOX]
106 | nb_url: http://your-server-here:your-port-here
107 | nb_token: your-token-here
108 | nb_ignore-tls-errors: false
109 |
110 | [NMAP]
111 | get_mac: true
112 | get_services: false
113 | networks: your-networks-separated-by-comma-here ie: (192.168.4.0/24,192.168.3.0/24)
114 | ```
115 | ```shell
116 | export NETBOX_URL=http://your-server-here:your-port-here
117 | export NETBOX_TOKEN=your-token-here
118 | export NETWORKS=your-networks-separated-by-comma-here
119 | ```
120 |
121 |
122 |
123 | ### Get Services
124 |
125 | Be aware that if you activate get service option `it will take between 15sec and 30sec per host` (so it can be slow)
126 |
127 |
128 |
129 | ### Get Mac address
130 |
131 | This service is pretty fast but will only work if the scan is performed from the same subnet
132 | ie: scanning subnet `192.168.1.0/24` from `192.168.1.2`
133 |
134 |
135 |
136 | ### Help
137 |
138 | ```sh
139 | python netbox-sync.py --help
140 | usage: netbox-sync.py [-h] [-c CONFIG] -u NB_URL [-l L] -p NB_TOKEN [-x] [-f] [-t TAG] -n NETWORKS [-o] [-s]
141 |
142 | Args that start with '--' (eg. -u) can also be set in a config file (./Netbox-sync-physical-hosts/netbox-sync-physical-hosts/netbox-
143 | sync.conf or specified via -c). Config file syntax allows: key=value, flag=true, stuff=[a,b,c] (for details, see syntax at https://goo.gl/R74nmi). If an arg is specified
144 | in more than one place, then commandline values override environment variables which override config file values which override defaults.
145 |
146 | optional arguments:
147 | -h, --help show this help message and exit
148 | -c CONFIG, --config CONFIG
149 | Config file path
150 | -u NB_URL, --nb_url NB_URL
151 | Netbox URL [env var: NETBOX_URL]
152 | -l L log level [env var: LOG_LEVEL]
153 | -p NB_TOKEN, --nb_token NB_TOKEN
154 | Token for Netbox connection [env var: NETBOX_TOKEN]
155 | -x, --nb_ignore-tls-errors
156 | Ignore TLS conection errors
157 | -f, --cleanup Cleanup orphans
158 | -t TAG, --tag TAG Tag to use for device identification [env var: TAG]
159 | -n NETWORKS, --networks NETWORKS
160 | Networks/Hosts to scan [env var: NETWORKS]
161 | -o, --get_mac Enable if you want the script to try to collect MAC addresses/vendor [env var: MAC_DISCOVER]
162 | -s, --get_services Enable if you want the script to discover host services [env var: SERVICE_DISCOVER]
163 | ```
164 |
165 |
166 |
167 | ## Contributing
168 |
169 | Contributions are what make the open source community such an amazing place to be learn, inspire, and create.
170 | Any contributions you make are **greatly appreciated**.
171 |
172 | 1. Fork the Project
173 | 2. Create your Feature Branch (`git checkout -b feature/AmazingFeature`)
174 | 3. Commit your Changes (`git commit -m 'Add some AmazingFeature'`)
175 | 4. Push to the Branch (`git push origin feature/AmazingFeature`)
176 | 5. Open a Pull Request
177 |
178 |
179 |
180 | ## License
181 |
182 | Distributed under the GNU General Public License v3.0.
183 | See LICENSE for more information.
184 |
185 |
186 |
--------------------------------------------------------------------------------
/images/logo.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/guanana/netbox-sync-physical-hosts/47a53c598616d5dc1b52445bd6eb299dc6030f60/images/logo.png
--------------------------------------------------------------------------------
/netbox_sync_physical_hosts/NetboxSync.py:
--------------------------------------------------------------------------------
1 | import sys
2 | from config.config import parse_config
3 | from netboxhandler.NetBoxHandler import NetBoxHandler
4 | from modules.NmapHandler import NmapServiceScan, NmapMacScan, NmapBasic
5 |
6 |
7 | def main(conf):
8 | nb = NetBoxHandler(conf.nb_url, conf.nb_token,
9 | conf.nb_ignore_tls_errors, conf.tag, conf.cleanup)
10 |
11 | if conf.get_mac:
12 | nmap = NmapMacScan(conf.networks)
13 | hosts = nmap.run()
14 | nb.run(hosts)
15 |
16 | if conf.get_services:
17 | nmap = NmapServiceScan(conf.networks)
18 | hosts = nmap.run()
19 | nb.run(hosts)
20 |
21 | if not conf.get_mac and not conf.get_services:
22 | nmap = NmapBasic(conf.networks)
23 | hosts = nmap.run()
24 | nb.run(hosts)
25 |
26 |
27 | if __name__ == '__main__':
28 | conf = parse_config()
29 | sys.exit(main(conf))
30 |
31 |
--------------------------------------------------------------------------------
/netbox_sync_physical_hosts/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/guanana/netbox-sync-physical-hosts/47a53c598616d5dc1b52445bd6eb299dc6030f60/netbox_sync_physical_hosts/__init__.py
--------------------------------------------------------------------------------
/netbox_sync_physical_hosts/config/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/guanana/netbox-sync-physical-hosts/47a53c598616d5dc1b52445bd6eb299dc6030f60/netbox_sync_physical_hosts/config/__init__.py
--------------------------------------------------------------------------------
/netbox_sync_physical_hosts/config/config.py:
--------------------------------------------------------------------------------
1 | import logging
2 | import os
3 | import configargparse
4 |
5 |
6 | def parse_config():
7 | current_dir = os.path.dirname(os.path.abspath(__file__))
8 | p = configargparse.ArgParser(default_config_files=[os.path.join(current_dir,
9 | 'netbox-sync.conf')])
10 |
11 | p.add('-c', '--config', default=os.path.join(current_dir, 'netbox-sync.conf'),
12 | is_config_file=True,
13 | help="Config file path")
14 |
15 | p.add('-u', '--nb_url', required=True, env_var='NETBOX_URL', help="Netbox URL")
16 |
17 | p.add('-l', help='log level', default=logging.INFO,
18 | env_var='LOG_LEVEL')
19 |
20 | p.add('-p', '--nb_token', required=True, help="Token for Netbox connection",
21 | env_var='NETBOX_TOKEN')
22 |
23 | p.add('-x', '--nb_ignore-tls-errors', action='store_true',
24 | help="Ignore TLS conection errors")
25 |
26 | p.add('-f', '--cleanup', action='store_true', help="Cleanup orphans")
27 |
28 | p.add('-t', '--tag', help="Tag to use for device identification", env_var="TAG")
29 |
30 | p.add('-n', '--networks', required=True, help="Networks/Hosts to scan",
31 | env_var="NETWORKS")
32 |
33 | p.add('-o', '--get_mac', action='store_true', default=False,
34 | help="Enable if you want the script to try to collect MAC addresses/vendor",
35 | env_var="MAC_DISCOVER")
36 |
37 | p.add('-s', '--get_services', action='store_true', default=False,
38 | help="Enable if you want the script to discover host services",
39 | env_var="SERVICE_DISCOVER")
40 |
41 | conf = p.parse_args()
42 | logging.basicConfig(level=conf.l)
43 |
44 | return conf
45 |
--------------------------------------------------------------------------------
/netbox_sync_physical_hosts/config/netbox-sync.conf:
--------------------------------------------------------------------------------
1 | [GENERAL]
2 | cleanup: false
3 | tag: auto-sync
4 |
5 | [NETBOX]
6 | nb_url: http://localhost:8000
7 | nb_token: 0123456789abcdef0123456789abcdef01234567
8 | nb_ignore-tls-errors: false
9 |
10 | [NMAP]
11 | get_mac: true
12 | get_services: false
13 | networks: 192.168.4.0/24,192.168.3.0/24
14 |
--------------------------------------------------------------------------------
/netbox_sync_physical_hosts/config/tests/test_config.py:
--------------------------------------------------------------------------------
1 | from unittest.mock import MagicMock
2 | import pytest
3 |
4 | from netbox_sync_physical_hosts.config.config import parse_config
5 |
6 | def test_get_conf_call(monkeypatch):
7 | testargs = ["prog", "-u", "http://test", "-p", "1234", "-n", "127.0.0.1"]
8 | monkeypatch.setattr('sys.argv', testargs)
9 | testconf = parse_config()
10 | assert testconf.nb_url == "http://test"
11 | assert testconf.nb_token == "1234"
12 | assert testconf.networks == "127.0.0.1"
13 |
14 | def test_failed_conf_call(monkeypatch):
15 | with pytest.raises(SystemExit) as pytest_wrapped_e:
16 | testargs = ["prog", "-c", "test"]
17 | monkeypatch.setattr('sys.argv', testargs)
18 | parse_config()
19 | assert pytest_wrapped_e.value.code == 2
20 |
21 |
--------------------------------------------------------------------------------
/netbox_sync_physical_hosts/modules/NmapHandler.py:
--------------------------------------------------------------------------------
1 | import logging
2 | import getmac
3 | import nmap3
4 | from mac_vendor_lookup import MacLookup, VendorNotFoundError
5 |
6 |
7 | class NmapBasic(object):
8 | def __init__(self, networks):
9 | self.hosts = dict()
10 | self.nmap = nmap3.NmapHostDiscovery()
11 | self.networks = self.sanitaise_networks(networks)
12 | self.scan_results = self.basic_scan()
13 |
14 | @staticmethod
15 | def sanitaise_networks(networks):
16 | networks = networks.split(',')
17 | for index, item in enumerate(networks):
18 | networks[index] = item.replace('\n', '')
19 | return networks
20 |
21 | def basic_scan(self):
22 | logging.info(f"Start NMAP scan for {self.networks}")
23 | for item in self.networks:
24 | self.scan_results = self.nmap.nmap_no_portscan(item,
25 | args="-R --system-dns")
26 | self.scan_results.pop("stats")
27 | self.scan_results.pop("runtime")
28 | self.scan_results.pop("task_results")
29 | for host, v in self.scan_results.items():
30 | self.scan_results[host]["subnet"] = item
31 | self.sanitaise_dict(host)
32 | return self.scan_results
33 |
34 | def sanitaise_dict(self, host):
35 | """
36 | Remove unused dictionary entries
37 | :return: None
38 | """
39 | self.scan_results[host].pop("state")
40 | self.scan_results[host].pop("ports")
41 | self.scan_results[host].pop("osmatch")
42 | if self.scan_results[host]["hostname"]:
43 | self.scan_results[host]["dns_name"] = self.scan_results[host]["hostname"][0]["name"]
44 | self.scan_results[host].pop("hostname")
45 | else:
46 | self.scan_results[host].pop("hostname")
47 |
48 | def run(self):
49 | return self.scan_results
50 |
51 |
52 | class NmapMacScan(NmapBasic):
53 | def __init__(self, networks, unknown="unknown"):
54 | super().__init__(networks)
55 | self.unknown = unknown
56 | self.mac_search = MacLookup()
57 |
58 | def update_mac(self, ip):
59 | """
60 | Update Mac info
61 | :param ip: IP address (ie: 192.168.1.1)
62 | :return: True if MAC is found, False otherwise
63 | """
64 | mac = getmac.get_mac_address(ip=ip)
65 | if mac is None or mac == 'ff:ff:ff:ff:ff:ff':
66 | return False
67 | else:
68 | self.scan_results[ip]["macaddress"] = mac
69 | return True
70 |
71 | def update_vendor(self, ip):
72 | """
73 | Update MAC vendor if Mac is found
74 | :param ip: IP address (ie: 192.168.1.1)
75 | :return: None
76 | """
77 | try:
78 | vendor_fetch = self.mac_search.lookup(self.scan_results[ip]["macaddress"])
79 | except VendorNotFoundError:
80 | vendor_fetch = "NotFound"
81 | self.scan_results[ip]["vendor"] = vendor_fetch
82 |
83 | def correct_missing_mac(self, host):
84 | """
85 | Correct description if macaddress is not found
86 | :param host: host key in scan_results
87 | :return: None
88 | """
89 | if not self.scan_results[host]["macaddress"]:
90 | self.scan_results[host]["description"] = self.unknown
91 | self.scan_results[host].pop("macaddress")
92 |
93 | def scan(self):
94 | """
95 | Scan defined networks and conditionally check for mac vendor
96 | :return: scan_results = list()
97 | """
98 | logging.debug("Updating MAC table")
99 | self.mac_search.update_vendors()
100 | for host, v in self.scan_results.items():
101 | if v.get("macaddress") or self.update_mac(host):
102 | self.update_vendor(ip=host)
103 | self.correct_missing_mac(host)
104 | return self.scan_results
105 |
106 | def run(self):
107 | return self.scan()
108 |
109 |
110 | class NmapServiceScan(NmapBasic):
111 | def __init__(self, networks):
112 | super().__init__(networks)
113 | self.nmap = nmap3.Nmap()
114 | self.services = dict()
115 |
116 | def scan_service(self, host):
117 | # TODO: Investigate more if this can be parallelize
118 | logging.debug(f"Scan started for host: {host}")
119 | self.services[host] = self.nmap.nmap_version_detection(host, args="-F -T4")
120 |
121 | def scan(self):
122 | logging.info(f"Starting Service scan for hosts in {self.networks}")
123 | for host in self.scan_results:
124 | self.scan_service(host)
125 | self.append_service_results()
126 | return self.scan_results
127 |
128 | def append_service_results(self):
129 | self.sanitaise_services()
130 | for host, value in self.services.items():
131 | self.scan_results[host]["services"] = {}
132 | for service in value:
133 | try:
134 | self.scan_results[host]["services"][service['portid']] = service
135 | except TypeError:
136 | pass
137 |
138 | def sanitaise_services(self):
139 | for host, value in self.services.items():
140 | try:
141 | self.services[host] = value[host]["ports"]
142 | except KeyError:
143 | logging.debug(f"No services detected for {host}")
144 | continue
145 | for service in self.services[host]:
146 | try:
147 | service.pop("reason")
148 | service.pop("reason_ttl")
149 | service.pop("cpe")
150 | service.pop("scripts")
151 | service["service"].pop("method")
152 | service["service"].pop("conf")
153 | except KeyError:
154 | pass
155 |
156 | def run(self):
157 | return self.scan()
158 |
--------------------------------------------------------------------------------
/netbox_sync_physical_hosts/modules/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/guanana/netbox-sync-physical-hosts/47a53c598616d5dc1b52445bd6eb299dc6030f60/netbox_sync_physical_hosts/modules/__init__.py
--------------------------------------------------------------------------------
/netbox_sync_physical_hosts/modules/tests/test_NmapHandler.py:
--------------------------------------------------------------------------------
1 | from unittest.mock import MagicMock
2 | from netbox_sync_physical_hosts.modules.NmapHandler import NmapBasic, NmapMacScan, NmapServiceScan
3 | import pytest
4 |
5 | basic_result = {
6 | 'stats': {'scanner': 'nmap', 'args': 'test', 'start': '1609858372',
7 | 'startstr': 'Tue Jan 1 14:52:52 1988', 'version': '7.40',
8 | 'xmloutputversion': '1.04'
9 | },
10 | 'runtime': {'time': '1609858372', 'timestr': 'Tue Jan 1 14:52:52 1988', 'elapsed': '0.01',
11 | 'summary': 'Nmap done at Tue Jan 5 14:52:52 2021; 1 IP address (1 host up) scanned in 0.01 seconds',
12 | 'exit': 'success'
13 | },
14 | 'task_results': [{'task': 'Ping Scan', 'time': '1688348552', 'extrainfo': '128 total hosts'},
15 | {'task': 'System DNS resolution of 128 hosts.', 'time': '1688348553'}]
16 | }
17 | service_result = {'127.0.0.1': {'osmatch': {}, 'ports':
18 | [{'protocol': 'tcp', 'portid': '22', 'state': 'open', 'reason': 'syn-ack', 'reason_ttl': '0',
19 | 'service':
20 | {'name': 'ssh', 'product': 'OpenSSH', 'version': '7.4p1 Debian 10+deb9u7',
21 | 'extrainfo': 'protocol 2.0', 'ostype': 'Linux', 'method': 'probed', 'conf': '10'},
22 | 'scripts': []},
23 | {'protocol': 'tcp', 'portid': '80', 'state': 'open', 'reason': 'syn-ack', 'reason_ttl': '0',
24 | 'service': {'name': 'http', 'product': 'lighttpd', 'method': 'probed', 'conf': '10'},
25 | 'cpe': [{'cpe': 'cpe:/a:lighttpd:lighttpd'}], 'scripts': []},
26 | {'protocol': 'tcp', 'portid': '443', 'state': 'open', 'reason': 'syn-ack', 'reason_ttl': '0',
27 | 'service': {'name': 'http', 'product': 'lighttpd', 'tunnel': 'ssl', 'method': 'probed',
28 | 'conf': '10'},
29 | 'cpe': [{'cpe': 'cpe:/a:lighttpd:lighttpd'}], 'scripts': []}], 'hostname': [],
30 | 'macaddress': None,
31 | 'state': {'state': 'up', 'reason': 'syn-ack', 'reason_ttl': '0'}},
32 | 'stats': {'scanner': 'nmap', 'args': '/usr/local/bin/nmap -oX - -sV -F -T4 192.168.4.1',
33 | 'start': '1609901261', 'startstr': 'Wed Jan 6 02:47:41 2021', 'version': '7.40',
34 | 'xmloutputversion': '1.04'},
35 | 'runtime': {'time': '1609901291', 'timestr': 'Wed Jan 6 02:48:11 2021', 'elapsed': '30.27',
36 | 'summary': 'Nmap done at Wed Jan 6 02:48:11 2021; 1 IP address (1 host up) scanned in 30.27 seconds',
37 | 'exit': 'success'}}
38 |
39 | service_result_no_ports = {'127.0.0.1': {'osmatch': {}},
40 | 'stats': {'scanner': 'nmap', 'args': '/usr/local/bin/nmap -oX - -sV -F -T4 192.168.4.1',
41 | 'start': '1609901261', 'startstr': 'Wed Jan 6 02:47:41 2021', 'version': '7.40',
42 | 'xmloutputversion': '1.04'},
43 | 'runtime': {'time': '1609901291', 'timestr': 'Wed Jan 6 02:48:11 2021', 'elapsed': '30.27',
44 | 'summary': 'Nmap done at Wed Jan 6 02:48:11 2021; 1 IP address (1 host up) scanned in 30.27 seconds',
45 | 'exit': 'success'}}
46 |
47 |
48 | def create_result_dicts(add_dict: str):
49 | options = {
50 | 'simple_one_host': {
51 | '127.0.0.1': {'osmatch': {}, 'ports': [],
52 | 'hostname': [{'name': 'localhost', 'type': 'PTR'}],
53 | 'macaddress': None,
54 | 'state': {'state': 'up', 'reason': 'mock-test', 'reason_ttl': '0'}
55 | }
56 | },
57 | 'simple_one_host_noDNS': {
58 | '127.0.0.1': {'osmatch': {}, 'ports': [],
59 | 'hostname': [],
60 | 'macaddress': None,
61 | 'state': {'state': 'up', 'reason': 'mock-test', 'reason_ttl': '0'}
62 | }
63 | },
64 | 'simple_one_host_mac': {
65 | '1.1.1.1': {'osmatch': {}, 'ports': [],
66 | 'hostname': [{'name': 'localhost', 'type': 'PTR'}],
67 | 'macaddress': "00:00:00:00:00",
68 | 'state': {'state': 'up', 'reason': 'mock-test', 'reason_ttl': '0'}
69 | }
70 | },
71 | 'simple_one_host_no_mac': {
72 | '1.1.1.1': {'osmatch': {}, 'ports': [], 'macaddress': None,
73 | 'hostname': [{'name': 'localhost', 'type': 'PTR'}],
74 | 'state': {'state': 'up', 'reason': 'mock-test', 'reason_ttl': '0'}
75 | }
76 | },
77 | 'simple_one_host_mac_ff': {
78 | '1.1.1.1': {'osmatch': {}, 'ports': [], 'macaddress': 'ff:ff:ff:ff:ff:ff',
79 | 'hostname': [{'name': 'localhost', 'type': 'PTR'}],
80 | 'state': {'state': 'up', 'reason': 'mock-test', 'reason_ttl': '0'}
81 | }
82 | },
83 | 'simple_one_host_service': {
84 | '127.0.0.1': {'osmatch': {}, 'ports': [],
85 | 'hostname': [{'name': 'localhost', 'type': 'PTR'}],
86 | 'macaddress': None,
87 | 'state': {'state': 'up', 'reason': 'mock-test', 'reason_ttl': '0'}
88 | }
89 | },
90 | }
91 | fake_result = dict()
92 | fake_result.update(basic_result)
93 | fake_result.update(options[add_dict])
94 | return fake_result
95 |
96 |
97 | def aux_mockportscan(monkeypatch, dict_mock):
98 | mock_result = MagicMock(return_value=create_result_dicts(dict_mock))
99 | monkeypatch.setattr('nmap3.NmapHostDiscovery.nmap_no_portscan', mock_result)
100 | return mock_result
101 |
102 |
103 | def test_no_hostname(monkeypatch):
104 | aux_mockportscan(monkeypatch, "simple_one_host_noDNS")
105 | nmap = NmapBasic("test")
106 | nmap.run()
107 | assert nmap.scan_results == {'127.0.0.1': {'macaddress': None, 'subnet': 'test'}}
108 |
109 |
110 | @pytest.fixture()
111 | def mock_mac_vendor(monkeypatch):
112 | aux_mockportscan(monkeypatch, "simple_one_host_mac")
113 | mock_mac_vendor_update = MagicMock()
114 | monkeypatch.setattr("mac_vendor_lookup.MacLookup.update_vendors", mock_mac_vendor_update)
115 | mock_mac_vendor_lookup = MagicMock()
116 | monkeypatch.setattr("mac_vendor_lookup.MacLookup.lookup", mock_mac_vendor_lookup)
117 | return mock_mac_vendor_lookup
118 |
119 |
120 | def test_nmap_mac_scan_run(mock_mac_vendor):
121 | mock_mac_vendor.return_value = 'testVendor'
122 | nmap = NmapMacScan("testMac")
123 | nmap.run()
124 | assert nmap.scan_results['1.1.1.1']["vendor"] == 'testVendor'
125 |
126 |
127 | def test_nmap_mac_scan_run_no_mac(monkeypatch, mock_mac_vendor):
128 | aux_mockportscan(monkeypatch, "simple_one_host_no_mac")
129 | mock_mac_vendor.return_value = None
130 | nmap = NmapMacScan("test_no_mac")
131 | nmap.run()
132 | assert not nmap.scan_results['1.1.1.1'].get("vendor")
133 |
134 | def test_nmap_mac_scan_run_mac_ff(monkeypatch, mock_mac_vendor):
135 | aux_mockportscan(monkeypatch, "simple_one_host_mac_ff")
136 | mock_mac_vendor.return_value = None
137 | nmap = NmapMacScan("test_mac_ff")
138 | nmap.run()
139 | assert not nmap.scan_results['1.1.1.1'].get("vendor")
140 |
141 | def test_nmap_mac_scan_get_mac_from_network(monkeypatch, mock_mac_vendor):
142 | aux_mockportscan(monkeypatch, "simple_one_host_no_mac")
143 | mock_mac_vendor.return_value = None
144 | nmap = NmapMacScan("test_get_mac_from_network")
145 | mock_get_mac_address = MagicMock(return_value="00:11:22:33:44:55")
146 | monkeypatch.setattr("getmac.get_mac_address", mock_get_mac_address)
147 | nmap.run()
148 | assert not nmap.scan_results['1.1.1.1'].get("vendor")
149 |
150 |
151 | @pytest.fixture()
152 | def mock_nmap_version_detection(monkeypatch):
153 | mock_result = MagicMock()
154 | monkeypatch.setattr('nmap3.Nmap.nmap_version_detection', mock_result)
155 | return mock_result
156 |
157 |
158 | def test_nmap_service_scan_run(monkeypatch, mock_nmap_version_detection):
159 | aux_mockportscan(monkeypatch, "simple_one_host_service")
160 | mock_nmap_version_detection.return_value = service_result
161 | nmap = NmapServiceScan("test")
162 | nmap.run()
163 | assert nmap.services['127.0.0.1'][0]['service']['name'] == "ssh"
164 |
165 |
166 | def test_nmap_service_scan_run_no_services(monkeypatch, mock_nmap_version_detection):
167 | aux_mockportscan(monkeypatch, "simple_one_host")
168 | mock_nmap_version_detection.return_value = service_result_no_ports
169 | nmap = NmapServiceScan("test")
170 | nmap.run()
171 | with pytest.raises(KeyError):
172 | assert not nmap.services['127.0.0.1'][0]
173 |
--------------------------------------------------------------------------------
/netbox_sync_physical_hosts/netboxhandler/NetBoxHandler.py:
--------------------------------------------------------------------------------
1 | import logging
2 | from distutils.version import StrictVersion
3 | from pynetbox.core.query import RequestError as pynetbox_RequestError
4 | import pynetbox
5 | import requests
6 | from django.utils.text import slugify
7 |
8 |
9 | def get_host_by_ip(nb_ip):
10 | try:
11 | if nb_ip and hasattr(nb_ip.assigned_object, "device"):
12 | logging.info(f"{nb_ip}: Host found => "
13 | f"{nb_ip.assigned_object.device.name}")
14 | return nb_ip.assigned_object.device, "device"
15 | elif nb_ip and hasattr(nb_ip.assigned_object, "virtual_machine"):
16 | logging.info(f"{nb_ip}: Virtual Host found => "
17 | f"{nb_ip.assigned_object.virtual_machine.name}")
18 | return nb_ip.assigned_object.virtual_machine, "virtual_machine"
19 | else:
20 | return None, None
21 | except AttributeError:
22 | logging.critical("You can only get host from a NB ip object")
23 | exit(1)
24 |
25 |
26 | class NetBoxHandler:
27 | def __init__(self, url, token, tls_verify, tag, cleanup_allowed):
28 | self.url = url
29 | self.token = token
30 | self.tls_verify = not tls_verify
31 | self.scripttag = tag
32 | self.cleanup_allowed = cleanup_allowed
33 | self.nb_con = self.nb_con()
34 | self.nb_ver = self.nb_ver()
35 | # Netbox objects
36 | logging.info("Caching all Netbox data")
37 | try:
38 | self.all_ips = list(self.nb_con.ipam.ip_addresses.all())
39 | self.all_interfaces = list(self.nb_con.dcim.interfaces.all())
40 | self.all_devices = list(self.nb_con.dcim.devices.all())
41 | self.all_sites = list(self.nb_con.dcim.sites.all())
42 | self.all_services = list(self.nb_con.ipam.services.all())
43 | except pynetbox_RequestError:
44 | logging.critical("Invalid token")
45 | exit(1)
46 | # Netbox pre-reqs
47 | self.pre_reqs()
48 |
49 | def nb_con(self):
50 | session = requests.Session()
51 | session.verify = self.tls_verify
52 | nb_con = pynetbox.api(self.url, self.token, threading=True)
53 | nb_con.http_session = session
54 | return nb_con
55 |
56 | def nb_ver(self):
57 | try:
58 | return StrictVersion(self.nb_con.version)
59 | except ConnectionRefusedError:
60 | logging.critical("Wrong URL or TOKEN, please check your config")
61 | exit(1)
62 | except requests.exceptions.MissingSchema:
63 | logging.critical(f"{self.url}: URL format should contain http or https")
64 | exit(1)
65 | except requests.exceptions.ConnectionError:
66 | logging.critical(f"{self.url}: Impossible to contact Netbox")
67 | exit(1)
68 |
69 | def pre_reqs(self):
70 | if self.nb_ver >= StrictVersion("2.9"):
71 | self.scripttag = self.create_tag(self.scripttag, scripttag=True)
72 | else:
73 | raise Exception("This script only works with Netbox > 2.9")
74 |
75 | def create_tag(self, tag, scripttag=False):
76 | nb_tag = self.nb_con.extras.tags.get(name=tag)
77 | if not nb_tag:
78 | if scripttag:
79 | logging.info("First run on Netbox instance, creating tag")
80 | nb_tag = self.nb_con.extras.tags.create(
81 | {"name": tag,
82 | "slug": slugify(tag),
83 | "description": f"Created by {__file__.split('/')[-3]}",
84 | "color": '2196f3'}
85 | )
86 | logging.debug(f"Tag {tag} created!")
87 |
88 | return nb_tag
89 |
90 | def set_ip_attribute(self, ip, ip_attr):
91 | pre_mask = ip_attr.get("subnet").split('/')
92 | if len(pre_mask) == 2:
93 | mask = pre_mask[-1]
94 | else:
95 | logging.error(f"Problem with IP {ip}")
96 | return None
97 | nb_attr = {
98 | "address": f"{ip}/{mask}",
99 | "tags": [self.scripttag.id],
100 | "dns_name": ip_attr.get("dns_name", ""),
101 | "description": ip_attr.get("description", "")
102 | }
103 | return nb_attr
104 |
105 | def set_service_attribute(self, host, service, device_type, ip):
106 | nb_attr = {
107 | device_type: host.id,
108 | "name": service["service"]["name"],
109 | "description": f"{service['service'].get('product')}: "
110 | f"{service['service'].get('version','version_unknown')}",
111 | "tags": [self.scripttag.id],
112 | "protocol": service["protocol"],
113 | "port": service["portid"],
114 | "ipaddresses": [ip.id]
115 | }
116 | return nb_attr
117 |
118 | def lookup_ip_address(self, ip):
119 | # nb_ip = [nb_ip for nb_ip in self.nb_con.ipam.ip_addresses.filter(address=ip)]
120 | nb_ip = [nb_ip for nb_ip in self.all_ips if nb_ip.address.startswith(f"{ip}/")]
121 | if not nb_ip:
122 | return None, True
123 | if len(nb_ip) == 1:
124 | return nb_ip[0], True
125 | else:
126 | return nb_ip, False
127 |
128 | def lookup_service(self, host, service, device_type, ip):
129 | try:
130 | if device_type == "device":
131 | nb_service = [nb_service for nb_service in self.all_services
132 | if nb_service.device == host and
133 | nb_service.port == int(service["portid"]) and
134 | [True for nb_ip in nb_service.ipaddresses
135 | if nb_ip.id == ip["id"]]][0]
136 | else:
137 | nb_service = [nb_service for nb_service in self.all_services
138 | if nb_service.virtual_machine == host and
139 | nb_service.port == int(service["portid"]) and
140 | [True for nb_ip in nb_service.ipaddresses
141 | if nb_ip.id == ip["id"]]][0]
142 | except IndexError:
143 | return
144 | return nb_service
145 |
146 | def nb_create_ip(self, ip_attr):
147 | logging.debug(f"{ip_attr.get('address')}: Not found in Netbox, creating record")
148 | nb_ip = self.nb_con.ipam.ip_addresses.create(ip_attr)
149 | logging.info(f"Record {ip_attr.get('address')} created")
150 | return nb_ip
151 |
152 | def nb_create_service(self, service_attr):
153 | logging.debug(f"{service_attr.get('name')}: Creating service")
154 | nb_service = self.nb_con.ipam.services.create(service_attr)
155 | logging.info(f"Service {service_attr.get('name')} created")
156 | return nb_service
157 |
158 | def create_service(self, host, service, device_type, nb_ip):
159 | logging.info(f"Creating service {service['portid']}")
160 | service_attr = self.set_service_attribute(host, service, device_type, nb_ip)
161 | nb_service = self.lookup_service(host, service, device_type, nb_ip)
162 | if not nb_service:
163 | nb_service = self.nb_create_service(service_attr)
164 | else:
165 | for tag in nb_service.tags:
166 | if self.scripttag.id == tag.id:
167 | nb_service.update(service_attr)
168 | return nb_service
169 | logging.info(f"Service {service['portid']} "
170 | f"found but scripttags is not present, "
171 | f"skipping update")
172 | return nb_service
173 |
174 | def run(self, scanned_hosts):
175 | logging.debug(f"Netbox version: {self.nb_ver}")
176 | for ip, attr in scanned_hosts.items():
177 | nb_ip, single = self.lookup_ip_address(ip)
178 | if not single:
179 | logging.warning(f"Found {ip} duplicated, skipping")
180 | continue
181 | if nb_ip:
182 | nb_host, device_type = get_host_by_ip(nb_ip)
183 | if not nb_host:
184 | logging.debug(f"Not host found for {ip}")
185 | continue
186 | else:
187 | if attr.get("services"):
188 | for port, service in attr["services"].items():
189 | self.create_service(nb_host, service, device_type, nb_ip)
190 | logging.debug(f"Found ports: {nb_host} with ip {ip}")
191 | # TODO: Check what to do
192 | logging.debug(f"Found host: {nb_host} with ip {ip}")
193 | else:
194 | ip_attr = self.set_ip_attribute(ip, attr)
195 | if ip_attr:
196 | self.nb_create_ip(ip_attr)
197 | else:
198 | logging.error(f"Problem found, IP not created")
199 |
--------------------------------------------------------------------------------
/netbox_sync_physical_hosts/netboxhandler/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/guanana/netbox-sync-physical-hosts/47a53c598616d5dc1b52445bd6eb299dc6030f60/netbox_sync_physical_hosts/netboxhandler/__init__.py
--------------------------------------------------------------------------------
/netbox_sync_physical_hosts/netboxhandler/tests/test_NetBoxHandler.py:
--------------------------------------------------------------------------------
1 | import logging
2 |
3 | import pytest
4 | from unittest.mock import MagicMock, PropertyMock
5 | from netbox_sync_physical_hosts.netboxhandler.NetBoxHandler import get_host_by_ip, NetBoxHandler
6 | # NB Model classes
7 |
8 | class Device:
9 | name = "test"
10 | id = 1
11 |
12 |
13 | class Dcim:
14 | device = Device()
15 |
16 |
17 | class Ip:
18 | def __init__(self, address, assign_host=True, virtual=False):
19 | self.address = address
20 | if assign_host:
21 | if virtual:
22 | self.assigned_object = VirtualMachine()
23 | else:
24 | self.assigned_object = Dcim()
25 | else:
26 | self.assigned_object = None
27 | self.id = 1
28 |
29 | def __getitem__(self, item):
30 | if item == "id":
31 | return 1
32 |
33 |
34 | class VirtualMachine:
35 | virtual_machine = Device()
36 |
37 |
38 | class WrongObject:
39 | assigned_object = "test"
40 |
41 |
42 | class Tag:
43 | def __init__(self, id, name):
44 | self.id = id
45 | self.name = name
46 |
47 |
48 | class Service:
49 | def __init__(self, device, portid, ip, tag):
50 | self.virtual_machine = device
51 | self.device = device
52 | self.port = int(portid)
53 | self.ipaddresses = ip
54 | self.tags = tag
55 |
56 | def update(self, test):
57 | return
58 |
59 | scan_host_service = {'127.0.0.1':
60 | {'macaddress': None, 'subnet': 'test', 'dns_name': 'localhost',
61 | 'services':
62 | {'22':
63 | {'protocol': 'tcp', 'portid': '22', 'state': 'open',
64 | 'service': {'name': 'ssh', 'product': 'OpenSSH',
65 | 'version': '7.4p1 Debian 10+deb9u7',
66 | 'extrainfo': 'protocol 2.0', 'ostype': 'Linux',
67 | 'method': 'probed', 'conf': '10'}, 'scripts': []},
68 | '80': {'protocol': 'tcp', 'portid': '80', 'state': 'open',
69 | 'service': {'name': 'http', 'product': 'lighttpd'}},
70 | '443': {'protocol': 'tcp', 'portid': '443', 'state': 'open',
71 | 'service': {'name': 'http', 'product': 'lighttpd', 'tunnel': 'ssl'}}}}}
72 |
73 |
74 | def test_invalid_get_host_by_ip():
75 | with pytest.raises(SystemExit) as pytest_wrapped_e:
76 | get_host_by_ip("wrong")
77 | assert pytest_wrapped_e.value.code == 1
78 |
79 |
80 | def test_invalid_object_get_host_by_ip():
81 | test, device_type = get_host_by_ip(WrongObject())
82 | assert not test
83 | assert not device_type
84 |
85 |
86 | @pytest.fixture()
87 | def mock_pynetbox_con(monkeypatch):
88 | mock_pynetbox_con = MagicMock()
89 | monkeypatch.setattr('pynetbox.api', mock_pynetbox_con)
90 | mock_pynetbox_con.return_value.extras.tags.get.return_value = Tag(1, "test")
91 | type(mock_pynetbox_con.return_value).version = PropertyMock(return_value="2.9")
92 | return mock_pynetbox_con
93 |
94 |
95 | def test_nb_host_unreachable():
96 | with pytest.raises(SystemExit):
97 | NetBoxHandler("http://unresolvable:8000", "1234", False, "test", False)
98 |
99 |
100 | def test_nb_wrong_schema():
101 | with pytest.raises(SystemExit):
102 | NetBoxHandler("test", "1234", False, "test", False)
103 |
104 |
105 | # TODO: PENDING TO IMPLEMENT
106 | # def test_nb_invalid_token(mock_pynetbox_session):
107 | # with pytest.raises(SystemExit) as pytest_wrapped_e:
108 | # NetBoxHandler("http://test:8000", "1234",
109 | # False, "test", False)
110 | # assert pytest_wrapped_e.value.code == 1
111 |
112 |
113 | def test_nb_wrong_version(mock_pynetbox_con):
114 | type(mock_pynetbox_con.return_value).version = PropertyMock(return_value="2.8")
115 | with pytest.raises(Exception):
116 | NetBoxHandler("http://test:8000", "1234", False, "test", False)
117 |
118 |
119 | @pytest.fixture()
120 | def nb(mock_pynetbox_con):
121 | nb = NetBoxHandler("http://test:8000", "1234", False, "test", False)
122 | return nb
123 |
124 |
125 | def test_netboxhandler_run_ip_no_host(caplog, nb):
126 | ip = Ip("127.0.0.1/32", assign_host=False)
127 | nb.all_ips = [ip]
128 | with caplog.at_level(logging.DEBUG):
129 | nb.run({"127.0.0.1": {}})
130 | assert [True for record in caplog.records if record.message == 'Not host found for 127.0.0.1']
131 |
132 |
133 |
134 | def test_create_ip(nb):
135 | nb.all_ips = []
136 | nb.run({'192.168.4.1': {'macaddress': "00:11:22:33:44:55", 'subnet': '192.168.4.0/24'}})
137 | nb.run({'192.168.4.2': {'macaddress': None, 'subnet': '192.168.4.0/24', 'dns_name': 'test.test.local'}})
138 |
139 |
140 | def test_create_ip_with_no_mask(caplog, nb):
141 | nb.all_ips = []
142 | with caplog.at_level(logging.DEBUG):
143 | nb.run({'192.168.4.1': {'macaddress': "00:11:22:33:44:55", 'subnet': '192.168.4.0'}})
144 | assert [True for record in caplog.records if record.message == 'Problem with IP 192.168.4.1']
145 |
146 |
147 | def test_netboxhandler_creation_scripttag(mock_pynetbox_con):
148 | mock_pynetbox_con.return_value.extras.tags.get.return_value = None
149 | mock_pynetbox_con.return_value.extras.tags.create.return_value = True
150 | NetBoxHandler("http://test:8000", "1234", False, "test_tag", False)
151 | mock_pynetbox_con.return_value.extras.tags.get.assert_called_with(name="test_tag")
152 |
153 |
154 | @pytest.fixture()
155 | def aux_create_service(mock_pynetbox_con):
156 | nb = NetBoxHandler("http://test:8000", "1234", False, "test_tag", False)
157 | mock_pynetbox_con.return_value.ipam.services.update.return_value = True
158 | return nb
159 |
160 |
161 | def aux_add_ips_and_services(nb, ips:list, services:list):
162 | nb.all_ips = ips
163 | nb.all_services = services
164 |
165 |
166 | def test_netboxhandler_create_service(aux_create_service, mock_pynetbox_con):
167 | nb = aux_create_service
168 | ip = Ip("127.0.0.1/32")
169 | device = ip.assigned_object.device
170 | nb_service = Service(device, 22, [ip], [Tag(1, "test_tag")])
171 | aux_add_ips_and_services(nb, [ip],[nb_service])
172 | nb.run(scan_host_service)
173 | assert mock_pynetbox_con.return_value.ipam.services.create.call_count == 2
174 | assert mock_pynetbox_con.return_value.ipam.services.update.call_count == 0
175 |
176 |
177 | def test_netboxhandler_create_service_virtual_machine(aux_create_service, mock_pynetbox_con):
178 | nb = aux_create_service
179 | virtual_ip = Ip("127.0.0.1/32", virtual=True)
180 | device = virtual_ip.assigned_object.virtual_machine
181 | nb_service = Service(device, 22, [virtual_ip], [Tag(1, "test_tag")])
182 | aux_add_ips_and_services(nb, [virtual_ip],[nb_service])
183 | nb.run(scan_host_service)
184 | print(nb)
185 | assert mock_pynetbox_con.return_value.ipam.services.create.call_count == 2
186 | assert mock_pynetbox_con.return_value.ipam.services.update.call_count == 0
187 |
188 |
189 | def test_netboxhandler_try_update_service_no_tag(caplog, mock_pynetbox_con):
190 | nb = NetBoxHandler("http://test:8000", "1234", False, "test_tag", False)
191 | ip = Ip("127.0.0.1/32")
192 | device = ip.assigned_object.device
193 | service_tag = Tag(2, "no_matching_tag_id")
194 | nb_service22 = Service(device, 22, [ip], [service_tag])
195 | nb_service80 = Service(device, 80, [ip], [])
196 | nb_service443 = Service(device, 443, [ip], [service_tag])
197 | nb.all_ips = [ip]
198 | nb.all_services = [nb_service22, nb_service80, nb_service443]
199 | nb.run(scan_host_service)
200 | assert mock_pynetbox_con.return_value.ipam.services.create.call_count == 0
201 | assert mock_pynetbox_con.return_value.ipam.services.update.call_count == 0
202 |
203 |
204 |
205 | def test_netboxhandler_duplicated_ip(caplog, mock_pynetbox_con):
206 | ip = Ip("127.0.0.1/32")
207 | ip2 = Ip("127.0.0.1/32")
208 | with caplog.at_level(logging.WARNING):
209 | nb = NetBoxHandler("http://test:8000", "1234", False, "test_tag", False)
210 | nb.all_ips = [ip, ip2]
211 | nb.run({"127.0.0.1": {}})
212 | assert [True for record in caplog.records if record.message == 'Found 127.0.0.1 duplicated, skipping']
213 |
--------------------------------------------------------------------------------
/requirements.txt:
--------------------------------------------------------------------------------
1 | pynetbox~=7.0.1
2 | requests~=2.31.0
3 | ConfigArgParse~=1.2.3
4 | python3-nmap
5 | mac-vendor-lookup
6 | getmac~=0.8.2
7 | chardet==3.0.4
8 | Django~=3.2.22
9 | aiohttp>=3.9.0 # not directly required, pinned by Snyk to avoid a vulnerability
10 | sqlparse>=0.5.0 # not directly required, pinned by Snyk to avoid a vulnerability
11 |
--------------------------------------------------------------------------------
Table of Contents
22 | -
23 |
-
24 | About The Project
25 |
-
26 |
- Built With 27 |
29 | -
30 | Getting Started
31 |
-
32 |
- Prerequisites 33 |
- Installation 34 |
36 | -
37 | Usage
38 |
-
39 |
- Get Services 40 |
- Get MAC address info 41 |
- Help 42 |
44 | - Contributing 45 |
- License 46 |