├── .gitignore
├── .pylintrc
├── LICENSE.txt
├── README.md
├── apisrv
├── __init__.py
├── errors.py
├── netdb.py
├── user.py
└── views.py
├── ctlsrv.py
├── cypher
├── buildfw.cyp
├── constraints.cyp
└── sample-queries.cyp
├── datasources
├── allvlans.py
├── asaparse.py
├── ciscoparse.py
├── csnip.py
├── netdb-devfile.py
├── netdb-nd.py
├── update_csv_from_netdb.sh
└── update_parse.sh
├── docs
├── API.md
├── About.md
├── CHANGES.md
├── CLI.md
├── CONTRIBUTING.md
├── INSTALL.md
├── Napalm.md
├── PathSample.md
├── Tutorials.md
├── VERSION
├── images
│ ├── network-graph.svg
│ ├── security-path2.svg
│ └── svipath2.svg
├── index.md
├── netgrph-logrotate
├── netgrph.ini
├── playbooks
│ ├── README.md
│ ├── netgrph.yml
│ ├── netgrph
│ │ ├── netgrph.yml
│ │ └── tasks
│ │ │ ├── apt-packages.yml
│ │ │ ├── apt-update.yml
│ │ │ ├── clonerepo.yml
│ │ │ ├── java8.yml
│ │ │ ├── neo4j.yml
│ │ │ └── ngpip.yml
│ ├── tasks
│ │ ├── apt-packages.yml
│ │ ├── apt-update.yml
│ │ ├── clonerepo.yml
│ │ ├── java8.yml
│ │ ├── neo4j.yml
│ │ ├── netgrph-user.yml
│ │ ├── ngpip.yml
│ │ └── setup.yml
│ └── test.yml
└── workflow.txt
├── extra
├── apisrv-nginx
├── nsj-sample.ini
├── nsj.py
├── svipath2.svg
└── wsgi-api.ini
├── mkdocs.yml
├── netgrph.py
├── nglib
├── __init__.py
├── alerts.py
├── cache_update.py
├── dev_update.py
├── exceptions.py
├── fw_update.py
├── net_update.py
├── netdb
│ ├── __init__.py
│ ├── ip.py
│ └── switch.py
├── ngtree
│ ├── __init__.py
│ ├── export.py
│ └── upgrade.py
├── query
│ ├── __init__.py
│ ├── dev.py
│ ├── nNode.py
│ ├── net.py
│ ├── path.py
│ └── vlan.py
├── report
│ └── __init__.py
└── vlan_update.py
├── ngreport.py
├── ngtest.py
├── ngupdate.py
├── requirements-client.txt
├── requirements.txt
├── test
├── __pycache__
│ └── testng.cpython-34-PYTEST.pyc
├── apisrv
├── bitwise.py
├── csv
│ ├── allnets.csv
│ ├── allvlans.csv
│ ├── devices.csv
│ ├── devinfo.csv
│ ├── firewalls.csv
│ ├── links.csv
│ ├── nd.csv
│ ├── supernets.csv
│ └── vrfs.csv
├── first_import.sh
├── ngclient.py
├── nglib
├── pre-push.sh
├── set_neo4j_password.sh
├── test_full.sh
├── test_queries.sh
├── testapi.py
├── testrand.py
└── uwsgi-api.sh
└── wsgi-api.py
/.gitignore:
--------------------------------------------------------------------------------
1 | __pycache__
2 | *~
3 | *.pyc
4 | log/*.log
5 | netgrphdev.ini
6 | netgrphtest.ini
7 | neo4j-queries.txt
8 | build-commands
9 | .vscode/
10 | ngtest.py.bak
11 | .cache/
12 | .DS_Store
13 | venv/
14 | api.log
15 | api.log.*
16 | api.db
17 | newhal.crt
18 | newhal.key
19 | prod.ini
20 | test.ini
21 | nsj.ini
22 | nsj.log
23 | job.log
24 | *.cmd
25 | *.json
26 | *.log
27 |
--------------------------------------------------------------------------------
/.pylintrc:
--------------------------------------------------------------------------------
1 | [MASTER]
2 | disable=invalid-name
3 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | ## Synopsis
2 |
3 | NetGrph is an abstract network model for automation, providing a unified network
4 | view across diverse network components in order to manage them as a single
5 | system via the [Neo4j Graph Database](http://neo4j.com). This enables you to
6 | navigate your traditional LAN/WAN and/or mixed SDN networks as interconnected
7 | nodes and relationships in software, all modeled via the network configurations
8 | as they exist rather than secondary sources.
9 |
10 | NetGrph can perform universal L2/L3/L4 path traversals, providing context for
11 | each layer along the path. It also serves as a VLAN and CIDR database, showing
12 | how everything is related. It scales well on even the largest networks,
13 | allowing sub-second queries across thousands of network devices. This enables
14 | the mapping of complex network relationships for discovery and automation.
15 |
16 | Data from queries can be returned as CSV, JSON, YAML, or Ascii tree-art. Network
17 | Visualizations can be created by querying the Neo4j webapp as shown below. The
18 | data model should translate for use with tools such as D3.js, vis.js or Graphwiz
19 | via both the native Neo4j API as well as NetGrph's tree data structure.
20 |
21 | All data is accessible via an API, and the lightweight netgrph client can be
22 | distributed to multiple machines.
23 |
24 | ## Graph Data Model Example: Vlan 110 -> 200 Traversal
25 | 
26 |
27 |
28 | [L3 SVIs: Yellow] [L2 VLANs: Green] [Switches/Routers: Blue]
29 |
30 |
31 |
32 | ## Features
33 |
34 | * Universal Layer2 - Layer4 pathfinding between any two network devices (Full L2 path completion requires NetDB)
35 | * Path Queries can return a single path, or all ECMPs
36 | * L3 Network Database of all networks (Automated, VRF aware, and searchable)
37 | * Search for networks via CIDR or VRF/Role based filters (eg. perim:printers|thinclient, all printers and thin clients in the perim VRF)
38 | * VLAN Inventory of all VLAN instances across the network, segmented by switch domain
39 | * Maps L2 VLAN bridges across switch domains, and calculates local/global STP roots
40 | * Maps L2 paths between devices (regexs supported, eg. dc.* -> dc.* for all datacenter links)
41 | * Reports the configured VLANs and actual VLANs existing on each link for all L2 paths
42 | * Secure REST API Server and Client
43 | * High performance, low latency queries (All queries are sub-second)
44 | * Easily extendable to support mixed-vendor environnments
45 | * Ansible playbooks for a five minute install on Ubuntu 14.04/16.04
46 |
47 | ## Planned Features
48 | * Rewrite API to be more consistent
49 | * Develop a Web frontend
50 | * Integrate with Napalm for datasources
51 |
52 | ## Requirements
53 | * Python 3.4+ (recommend running via virtualenv)
54 | * Ubuntu or MacOS (should run on any Python compatible platform, but I only support these)
55 | * [Neo4j Graph Database](https://neo4j.com) and Java8
56 | * For Cisco devices, must provide stored configurations (See [Rancid](http://www.shrubbery.net/rancid/) / [Oxidized](https://github.com/ytti/oxidized))
57 | * Requires CDP/LLDP Discovery Data. [ndcrawl](https://github.com/yantisj/ndcrawl) will output this in the proper format for NetGrph with the -ng_file option.
58 | * Third-party network devices need to be parsed into the [NetGrph CSV format](test/csv/)
59 | * Please send me any parsers you create
60 |
61 | ## Documentation
62 |
63 | * [NetGrph Read The Docs](http://netgrph.readthedocs.io/)
64 |
65 | ## Path Traversal Example
66 | ```
67 | $ ./netgrph.py -p 10.26.72.142 10.34.72.24
68 |
69 | ┌─[ PATHs L2-L4 ]
70 | │
71 | ├── L2 Path : abc7t1sw1 (Gi2/42) -> abc7t1sw1 (Gi1/38)
72 | ├── L3 Path : 10.26.72.0/22 -> 10.34.72.0/22
73 | ├── L4 Path : VRF:default -> FwutilFW -> VRF:utility
74 | ├── Lx Path : 10.26.72.142 -> 10.34.72.24
75 | ├── Traversal Type : All Paths
76 | │
77 | ├─────[ SRC 10.26.72.142 04bd.88cb.xxxx abc7t1sw1(Gi2/42) [vid:260] ]
78 | │
79 | ├───┬─[ L2-PATH abc7t1sw1 -> abcmdf1|abcmdf2 ]
80 | │ │
81 | │ ├─────[ L2-HOP #1 abc7t1sw1(Te5/1) -> abcmdf1(Eth1/8) [pc:1->108] ]
82 | │ │
83 | │ └─────[ L2-HOP #1 abc7t1sw1(Te6/1) -> abcmdf2(Eth1/8) [pc:1->108] ]
84 | │
85 | ├─────[ L3GW 10.26.72.0/22 abcmdf1|abcmdf2 ]
86 | │
87 | ├───┬─[ L3-PATH 10.26.72.0/22 -> 10.25.11.0/24 ]
88 | │ │
89 | │ ├───┬─[ L3-HOP #1 abcmdf1(10.23.74.11) -> core1(10.23.74.10) [vid:2074] ]
90 | │ │ │
91 | │ │ └─────[ L2-HOP #1 abcmdf1(Eth2/26) -> core1(Eth7/27) ]
92 | │ │
93 | │ ├───┬─[ L3-HOP #1 abcmdf1(10.23.74.21) -> core2(10.23.74.20) [vid:3074] ]
94 | │ │ │
95 | │ │ └─────[ L2-HOP #1 abcmdf1(Eth3/8) -> core2(Eth4/25) ]
96 | │ │
97 | │ ├───┬─[ L3-HOP #1 abcmdf2(10.23.78.11) -> core1(10.23.78.10) [vid:2078] ]
98 | │ │ │
99 | │ │ └─────[ L2-HOP #1 abcmdf2(Eth2/26) -> core1(Eth8/25) ]
100 | │ │
101 | │ └───┬─[ L3-HOP #1 abcmdf2(10.23.78.21) -> core2(10.23.78.20) [vid:3078] ]
102 | │ │
103 | │ └─────[ L2-HOP #1 abcmdf2(Eth3/8) -> core2(Eth8/25) ]
104 | │
105 | ├─────[ L4GW 10.25.11.0/24 [rtr: vid:1601 vrf:default] ]
106 | │
107 | ├─────[ L4FW FwutilFW ]
108 | │
109 | ├─────[ L4GW 10.25.12.0/24 [rtr: vid:1602 vrf:utility] ]
110 | │
111 | ├───┬─[ L3-PATH 10.25.12.0/24 -> 10.34.72.0/22 ]
112 | │ │
113 | │ ├───┬─[ L3-HOP #1 core1(10.23.74.10) -> abcmdf1(10.23.74.11) [vid:2461] ]
114 | │ │ │
115 | │ │ └─────[ L2-HOP #1 core1(Eth7/27) -> abcmdf1(Eth2/26) ]
116 | │ │
117 | │ ├───┬─[ L3-HOP #1 core1(10.23.78.10) -> abcmdf2(10.23.78.11) [vid:2462] ]
118 | │ │ │
119 | │ │ └─────[ L2-HOP #1 core1(Eth8/25) -> abcmdf2(Eth2/26) ]
120 | │ │
121 | │ ├───┬─[ L3-HOP #1 core2(10.23.74.20) -> abcmdf1(10.23.74.21) [vid:3461] ]
122 | │ │ │
123 | │ │ └─────[ L2-HOP #1 core2(Eth4/25) -> abcmdf1(Eth3/8) ]
124 | │ │
125 | │ └───┬─[ L3-HOP #1 core2(10.23.78.20) -> abcmdf2(10.23.78.21) [vid:3462] ]
126 | │ │
127 | │ └─────[ L2-HOP #1 core2(Eth8/25) -> abcmdf2(Eth3/8) ]
128 | │
129 | ├─────[ L3GW 10.34.72.0/22 abcmdf1|abcmdf2 ]
130 | │
131 | ├───┬─[ L2-PATH abcmdf1|abcmdf2 -> abc7t1sw1 ]
132 | │ │
133 | │ ├─────[ L2-HOP #1 abcmdf1(Eth1/8) -> abc7t1sw1(Te5/1) [pc:108->1] ]
134 | │ │
135 | │ └─────[ L2-HOP #1 abcmdf2(Eth1/8) -> abc7t1sw1(Te6/1) [pc:108->1] ]
136 | │
137 | └─────[ DST 10.34.72.24 000a.b004.xxxx abc7t1sw1(Gi1/38) [vid:340] ]
138 | ```
139 |
140 | ## Installation
141 |
142 | See the [Install Instructions](docs/INSTALL.md)
143 |
144 | ## Support
145 |
146 | See refer to the documentation first: [NetGrph Read The
147 | Docs](http://netgrph.readthedocs.io/)
148 |
149 | You can open an issue via GitHub, or if you would like to speak with me
150 | directly, I monitor the #netgrph channel the [networktocode slack
151 | group](https://networktocode.herokuapp.com/). Please try and contact me there
152 | for any interactive support.
153 |
154 | ## Contributions
155 |
156 | Please see the [Contributions](docs/CONTRIBUTING.md) document in docs for
157 | information about how you can contribute back to NetGrph.
158 |
159 | ## Contributors
160 | * Jonathan Yantis ([yantisj](https://github.com/yantisj))
161 |
162 | ## License
163 | NetGrph is licensed under the GNU AGPLv3 License.
164 |
--------------------------------------------------------------------------------
/apisrv/__init__.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python
2 | # Copyright (c) 2016 "Jonathan Yantis"
3 | #
4 | # This program is free software: you can redistribute it and/or modify
5 | # it under the terms of the GNU Affero General Public License, version 3,
6 | # as published by the Free Software Foundation.
7 | #
8 | # This program is distributed in the hope that it will be useful,
9 | # but WITHOUT ANY WARRANTY; without even the implied warranty of
10 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
11 | # GNU Affero General Public License for more details.
12 | #
13 | # You should have received a copy of the GNU Affero General Public License
14 | # along with this program. If not, see .
15 | #
16 | # As a special exception, the copyright holders give permission to link the
17 | # code of portions of this program with the OpenSSL library under certain
18 | # conditions as described in each individual source file and distribute
19 | # linked combinations including the program with the OpenSSL library. You
20 | # must comply with the GNU Affero General Public License in all respects
21 | # for all of the code used other than as permitted herein. If you modify
22 | # file(s) with this exception, you may extend this exception to your
23 | # version of the file(s), but you are not obligated to do so. If you do not
24 | # wish to do so, delete this exception statement from your version. If you
25 | # delete this exception statement from all source files in the program,
26 | # then also delete it in the license file.
27 | #
28 | #
29 | """
30 | Flask API Server
31 | """
32 | import builtins
33 | import sys
34 | import os
35 | import re
36 | import logging
37 | from logging.handlers import RotatingFileHandler
38 | import configparser
39 | from flask import Flask, jsonify, request, g, make_response
40 | from flask_httpauth import HTTPBasicAuth
41 | from flask_limiter import Limiter
42 | from flask_limiter.util import get_remote_address
43 | from flask_sqlalchemy import SQLAlchemy
44 | import nglib
45 |
46 | logger = logging.getLogger(__name__)
47 |
48 | # Populated from config file
49 | debug = 0
50 |
51 | # Flask Limits for Safety
52 | flask_limits = ["100000 per day", "5000 per hour", "600 per minute"]
53 |
54 | # Initialize Configuration
55 | config_file = None
56 | if 'NGLIB_config_file' in os.environ:
57 | config_file = os.environ['NGLIB_config_file']
58 | else:
59 | config_file = builtins.apisrv_CONFIG
60 | config = configparser.ConfigParser()
61 | config.read(config_file)
62 |
63 | # Check for sane config file
64 | if 'apisrv' not in config:
65 | print("Could not parse config file: " + config_file)
66 | sys.exit(1)
67 |
68 | # Logging Configuration, default level INFO
69 | logger = logging.getLogger('')
70 | logger.setLevel(logging.INFO)
71 | lformat = logging.Formatter('%(asctime)s %(name)s:%(levelname)s: %(message)s')
72 |
73 | # Debug mode
74 | if 'debug' in config['apisrv'] and int(config['apisrv']['debug']) != 0:
75 | debug = int(config['apisrv']['debug'])
76 | logger.setLevel(logging.DEBUG)
77 | logging.debug('Enabled Debug mode')
78 |
79 | # Enable logging to file if configured
80 | if 'logfile' in config['apisrv']:
81 | lfh = RotatingFileHandler(config['apisrv']['logfile'], maxBytes=(1048576*5), backupCount=3)
82 | lfh.setFormatter(lformat)
83 | logger.addHandler(lfh)
84 |
85 | # STDOUT Logging defaults to Warning
86 | if not debug:
87 | lsh = logging.StreamHandler(sys.stdout)
88 | lsh.setFormatter(lformat)
89 | lsh.setLevel(logging.WARNING)
90 | logger.addHandler(lsh)
91 |
92 | # Create Flask APP
93 | app = Flask(__name__)
94 | app.config.from_object(__name__)
95 | app.config.update(dict(
96 | DATABASE=os.path.join(app.root_path, config['apisrv']['database']),
97 | SQLALCHEMY_DATABASE_URI='sqlite:///' + os.path.join(app.root_path, config['apisrv']['database']),
98 | SQLALCHEMY_TRACK_MODIFICATIONS=False,
99 | #SECRET_KEY='development key',
100 | #USERNAME='dbuser',
101 | #PASSWORD='dbpass'
102 | ))
103 |
104 | # Auth module
105 | auth = HTTPBasicAuth()
106 |
107 | # Database module
108 | db = SQLAlchemy(app)
109 |
110 | # Apply Rate limiting
111 | limiter = Limiter(
112 | app,
113 | key_func=get_remote_address,
114 | global_limits=flask_limits
115 | )
116 |
117 | def get_bolt_db():
118 | """Store nglib database handle under thread"""
119 | bdb = getattr(g, '_boltdb', None)
120 | if bdb is None:
121 | bdb = g._boltdb = nglib.get_bolt_db()
122 | return bdb
123 |
124 | def get_py2neo_db():
125 | """Store nglib database handle under thread"""
126 | pydb = getattr(g, '_py2neodb', None)
127 | if pydb is None:
128 | pydb = g._py2neodb = nglib.get_py2neo_db()
129 | return pydb
130 |
131 | @app.before_request
132 | def init_db():
133 | """Initialize Library on each request"""
134 | logging.debug('Getting Neo4j Database Connections')
135 | nglib.verbose = debug
136 | nglib.init_nglib(config_file, initdb=False)
137 | nglib.bolt_ses = get_bolt_db()
138 | g.neo4j_db_bolt = nglib.bolt_ses
139 | nglib.py2neo_ses = get_py2neo_db()
140 | g.neo4j_db_py2neo = nglib.py2neo_ses
141 |
142 | @app.teardown_appcontext
143 | def close_db(error):
144 | """Closes the database again at the end of the request."""
145 | if hasattr(g, 'sqlite_db'):
146 | g.sqlite_db.close()
147 |
148 | bolt_ses = getattr(g, 'neo4j_db_bolt', None)
149 | if bolt_ses is not None:
150 | logger.debug('Closing Neo4j Database Connection')
151 | g.neo4j_db = None
152 | nglib.bolt_ses = None
153 | bolt_ses.last_result = None
154 | bolt_ses.close()
155 |
156 | py2neo_ses = getattr(g, 'neo4j_db_py2neo', None)
157 | if py2neo_ses is not None:
158 | logger.debug('Closing Neo4j Database Connection')
159 | g.neo4j_db = None
160 | nglib.py2neo_ses = None
161 | py2neo_ses.last_result = None
162 | py2neo_ses = None
163 |
164 |
165 | def upgrade_api(ngt, version):
166 | 'Upgrade the output of NGT API data'
167 |
168 | nt = ngt
169 |
170 | if version == 'v2':
171 | nt = nglib.ngtree.upgrade.upgrade_ngt_v2(ngt)
172 | return nt
173 |
174 | def version_chk(version, versions=['v1.1', 'v2']):
175 | 'Make sure version in versions'
176 |
177 | if version not in versions:
178 | return jsonify(errors.json_error('API Version Error', 'Version ' \
179 | + version + ' not supported on this endpoint'))
180 |
181 | # Safe circular imports per Flask guide
182 | import apisrv.errors
183 | import apisrv.views
184 | import apisrv.user
185 | import apisrv.netdb
186 |
187 |
188 |
--------------------------------------------------------------------------------
/apisrv/errors.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python
2 | # Copyright (c) 2016 "Jonathan Yantis"
3 | #
4 | # This program is free software: you can redistribute it and/or modify
5 | # it under the terms of the GNU Affero General Public License, version 3,
6 | # as published by the Free Software Foundation.
7 | #
8 | # This program is distributed in the hope that it will be useful,
9 | # but WITHOUT ANY WARRANTY; without even the implied warranty of
10 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
11 | # GNU Affero General Public License for more details.
12 | #
13 | # You should have received a copy of the GNU Affero General Public License
14 | # along with this program. If not, see .
15 | #
16 | # As a special exception, the copyright holders give permission to link the
17 | # code of portions of this program with the OpenSSL library under certain
18 | # conditions as described in each individual source file and distribute
19 | # linked combinations including the program with the OpenSSL library. You
20 | # must comply with the GNU Affero General Public License in all respects
21 | # for all of the code used other than as permitted herein. If you modify
22 | # file(s) with this exception, you may extend this exception to your
23 | # version of the file(s), but you are not obligated to do so. If you do not
24 | # wish to do so, delete this exception statement from your version. If you
25 | # delete this exception statement from all source files in the program,
26 | # then also delete it in the license file.
27 | #
28 | """
29 | API Errors Jsonified
30 | """
31 | import logging
32 | from flask import Flask, jsonify, request, g, make_response
33 | from apisrv import app, auth
34 |
35 | logger = logging.getLogger(__name__)
36 |
37 | @app.errorhandler(404)
38 | def not_found(error=None):
39 | message = {
40 | 'status': 404,
41 | 'message': 'Not Found: ' + request.url,
42 | }
43 | resp = jsonify(message)
44 | resp.status_code = 404
45 | return resp
46 |
47 | @app.errorhandler(429)
48 | def ratelimit_handler(e):
49 | return make_response(
50 | jsonify(error="ratelimit exceeded %s" % e.description)
51 | , 429
52 | )
53 |
54 | @auth.error_handler
55 | def auth_failed(error=None):
56 | message = {
57 | 'status': 401,
58 | 'message': 'Authentication Failed: ' + request.url
59 | }
60 | resp = jsonify(message)
61 | resp.status_code = 401
62 |
63 | return resp
64 |
65 | @app.errorhandler(400)
66 | def bad_request(error):
67 | print("Bad Request")
68 | message = {
69 | 'status': 400,
70 | 'message': 'Bad Request: ' + request.url,
71 | }
72 | resp = jsonify(message)
73 | resp.status_code = 400
74 | return resp
75 |
76 | def json_error(error, message, code=404):
77 |
78 | ngerror = dict()
79 | ngerror['Name'] = error
80 | ngerror['_type'] = 'Error'
81 | ngerror['message'] = error + ': ' + message
82 | ngerror['status'] = code
83 |
84 | return ngerror
85 |
--------------------------------------------------------------------------------
/apisrv/netdb.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python
2 | # Copyright (c) 2016 "Jonathan Yantis"
3 | #
4 | # This program is free software: you can redistribute it and/or modify
5 | # it under the terms of the GNU Affero General Public License, version 3,
6 | # as published by the Free Software Foundation.
7 | #
8 | # This program is distributed in the hope that it will be useful,
9 | # but WITHOUT ANY WARRANTY; without even the implied warranty of
10 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
11 | # GNU Affero General Public License for more details.
12 | #
13 | # You should have received a copy of the GNU Affero General Public License
14 | # along with this program. If not, see .
15 | #
16 | # As a special exception, the copyright holders give permission to link the
17 | # code of portions of this program with the OpenSSL library under certain
18 | # conditions as described in each individual source file and distribute
19 | # linked combinations including the program with the OpenSSL library. You
20 | # must comply with the GNU Affero General Public License in all respects
21 | # for all of the code used other than as permitted herein. If you modify
22 | # file(s) with this exception, you may extend this exception to your
23 | # version of the file(s), but you are not obligated to do so. If you do not
24 | # wish to do so, delete this exception statement from your version. If you
25 | # delete this exception statement from all source files in the program,
26 | # then also delete it in the license file.
27 | #
28 | """
29 | NetDB API Views
30 | """
31 | import logging
32 | import nglib
33 | import nglib.query
34 | import nglib.report
35 | import nglib.netdb.switch
36 | from nglib.exceptions import ResultError
37 | from flask import jsonify, request
38 | from apisrv import app, auth, config, errors, upgrade_api, version_chk
39 |
40 | # Setup
41 | logger = logging.getLogger(__name__)
42 | app_name = config['apisrv']['app_name']
43 |
44 | # Device Queries
45 | @app.route('/netdb/api//table/arp', methods=['GET'])
46 | @app.route('/api//table/arp', methods=['GET'])
47 | @auth.login_required
48 | def get_arptable(ver):
49 | """ Get ARP Tables from NetDB
50 | """
51 |
52 | error = version_chk(ver, ['v1.0', 'v2'])
53 | if error:
54 | return error
55 |
56 | router = None
57 | hours = 1
58 |
59 | if 'router' in request.args:
60 | router = request.args['router']
61 | router = router.replace('*', '%')
62 | else:
63 | return jsonify(errors.json_error('InputError', 'router= is required'))
64 |
65 | if 'hours' in request.args:
66 | hours = int(request.args['hours'])
67 |
68 |
69 | try:
70 | return jsonify(upgrade_api(nglib.netdb.ip.arp(router=router, hours=hours), ver))
71 | except ResultError as e:
72 | return jsonify(errors.json_error(e.expression, e.message))
73 |
74 | @app.route('/netdb/api//table/mac', methods=['GET'])
75 | @app.route('/api//table/mac', methods=['GET'])
76 | @auth.login_required
77 | def get_mactable(ver):
78 | """ Get the MAC Table from NetDB
79 | """
80 |
81 | error = version_chk(ver, ['v1.0', 'v2'])
82 | if error:
83 | return error
84 |
85 | switch = None
86 | port = '%'
87 | hours = 1
88 |
89 | if 'switch' in request.args:
90 | switch = request.args['switch']
91 | switch = switch.replace('*', '%')
92 | else:
93 | return jsonify(errors.json_error('InputError', 'switch= is required'))
94 | if 'port' in request.args:
95 | port = request.args['port']
96 | port = port.replace('*', '%')
97 | print("port", port)
98 |
99 | if 'hours' in request.args:
100 | hours = int(request.args['hours'])
101 |
102 | try:
103 | return jsonify(upgrade_api(nglib.netdb.switch.mac(switch=switch, port=port, hours=hours), ver))
104 | except ResultError as e:
105 | return jsonify(errors.json_error(e.expression, e.message))
106 |
107 |
108 | @app.route('/netdb/api//table/mac//count', methods=['GET'])
109 | @app.route('/api//table/mac//count', methods=['GET'])
110 | @auth.login_required
111 | def get_mac_count(ver, switch):
112 | """ Get the MAC Table count from NetDB
113 | """
114 |
115 | error = version_chk(ver, ['v1.0', 'v2'])
116 | if error:
117 | return error
118 |
119 | hours = 1
120 | switch = switch.replace('*', '%')
121 |
122 | if 'hours' in request.args:
123 | hours = int(request.args['hours'])
124 |
125 | try:
126 | return jsonify(upgrade_api(nglib.netdb.switch.count(switch=switch, hours=hours), ver))
127 | except ResultError as e:
128 | return jsonify(errors.json_error(e.expression, e.message))
129 |
--------------------------------------------------------------------------------
/apisrv/user.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python
2 | # Copyright (c) 2016 "Jonathan Yantis"
3 | #
4 | # This program is free software: you can redistribute it and/or modify
5 | # it under the terms of the GNU Affero General Public License, version 3,
6 | # as published by the Free Software Foundation.
7 | #
8 | # This program is distributed in the hope that it will be useful,
9 | # but WITHOUT ANY WARRANTY; without even the implied warranty of
10 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
11 | # GNU Affero General Public License for more details.
12 | #
13 | # You should have received a copy of the GNU Affero General Public License
14 | # along with this program. If not, see .
15 | #
16 | # As a special exception, the copyright holders give permission to link the
17 | # code of portions of this program with the OpenSSL library under certain
18 | # conditions as described in each individual source file and distribute
19 | # linked combinations including the program with the OpenSSL library. You
20 | # must comply with the GNU Affero General Public License in all respects
21 | # for all of the code used other than as permitted herein. If you modify
22 | # file(s) with this exception, you may extend this exception to your
23 | # version of the file(s), but you are not obligated to do so. If you do not
24 | # wish to do so, delete this exception statement from your version. If you
25 | # delete this exception statement from all source files in the program,
26 | # then also delete it in the license file.
27 | #
28 | """
29 | API User Functions
30 | """
31 | import sys
32 | import logging
33 | import time
34 | from passlib.hash import sha256_crypt
35 |
36 | import sqlite3
37 | from flask_sqlalchemy import SQLAlchemy
38 | from apisrv import auth, config, db
39 | from sqlalchemy.exc import OperationalError
40 |
41 | logger = logging.getLogger(__name__)
42 |
43 | class User(db.Model):
44 | """ SQL User Model"""
45 | id = db.Column(db.Integer, primary_key=True)
46 | username = db.Column(db.String(80), unique=True)
47 | password = db.Column(db.String(120), unique=True)
48 | role = db.Column(db.String(40))
49 |
50 | def __init__(self, username, password, role='default'):
51 | self.username = username
52 | self.password = password
53 | self.role = role
54 |
55 | def __repr__(self):
56 | return ''.format(self.username)
57 |
58 | @auth.verify_password
59 | def verify_password(username, password):
60 | """API Password Verification"""
61 |
62 | if authenticate_user(username, password):
63 | return True
64 | return False
65 |
66 | def get_hash(password):
67 | """ Returns a new SHA256 Hash for a password (lower rounds for speed)"""
68 |
69 | phash = sha256_crypt.encrypt(password, rounds=100000)
70 | return phash
71 |
72 | def authenticate_user(username, password):
73 | """ Authenticate a user """
74 |
75 | try:
76 | user = User.query.filter_by(username=username).first()
77 | except OperationalError:
78 | db.create_all()
79 | user = User.query.filter_by(username=username).first()
80 |
81 |
82 | authenticated = False
83 |
84 | if user:
85 | authenticated = sha256_crypt.verify(password, user.password)
86 | else:
87 | time.sleep(1)
88 | logger.info("Authentication Error: User not found in DB: %s", username)
89 | return False
90 |
91 | if authenticated:
92 | logger.debug("Successfully Authenticated user: %s", username)
93 | else:
94 | logger.info("Authentication Failed: %s", username)
95 |
96 | return authenticated
97 |
98 |
99 | def add_user(username, password):
100 | """
101 | Add a new user to the database
102 | """
103 |
104 | user = User.query.filter_by(username=username).first()
105 |
106 | if user:
107 | #print("Error: User already exists in DB", file=sys.stderr)
108 | raise Exception("Error: User already exists in DB")
109 | elif len(password) < 6:
110 | print("Error: Password must be 6 or more characters", file=sys.stderr)
111 | exit(1)
112 | else:
113 | logger.info("Adding new user to the database: %s", username)
114 |
115 | phash = get_hash(password)
116 |
117 | newuser = User(username, phash)
118 | db.session.add(newuser)
119 | db.session.commit()
120 |
121 | return phash
122 |
123 | def update_password(username, password):
124 | """ Update password for user """
125 |
126 | user = User.query.filter_by(username=username).first()
127 |
128 | if len(password) < 6:
129 | print("Error: Password must be 6 or more characters", file=sys.stderr)
130 | exit(1)
131 | elif user:
132 | logger.info("Updating password for user: %s", username)
133 |
134 | phash = get_hash(password)
135 |
136 | user.password = phash
137 | db.session.commit()
138 |
139 | return phash
140 | else:
141 |
142 | print("Error: User does not exists in DB", file=sys.stderr)
143 | exit(1)
144 |
145 | def del_user(username):
146 | """ Delete a user from the database """
147 |
148 | user = User.query.filter_by(username=username).first()
149 |
150 | if user:
151 | logger.info("Deleting user: %s", username)
152 |
153 | db.session.delete(user)
154 | db.session.commit()
155 |
156 | return True
157 | else:
158 |
159 | print("Error: User does not exists in DB", file=sys.stderr)
160 | exit(1)
161 |
162 |
--------------------------------------------------------------------------------
/ctlsrv.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python3
2 | # Copyright (c) 2016 "Jonathan Yantis"
3 | #
4 | # This program is free software: you can redistribute it and/or modify
5 | # it under the terms of the GNU Affero General Public License, version 3,
6 | # as published by the Free Software Foundation.
7 | #
8 | # This program is distributed in the hope that it will be useful,
9 | # but WITHOUT ANY WARRANTY; without even the implied warranty of
10 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
11 | # GNU Affero General Public License for more details.
12 | #
13 | # You should have received a copy of the GNU Affero General Public License
14 | # along with this program. If not, see .
15 | #
16 | # As a special exception, the copyright holders give permission to link the
17 | # code of portions of this program with the OpenSSL library under certain
18 | # conditions as described in each individual source file and distribute
19 | # linked combinations including the program with the OpenSSL library. You
20 | # must comply with the GNU Affero General Public License in all respects
21 | # for all of the code used other than as permitted herein. If you modify
22 | # file(s) with this exception, you may extend this exception to your
23 | # version of the file(s), but you are not obligated to do so. If you do not
24 | # wish to do so, delete this exception statement from your version. If you
25 | # delete this exception statement from all source files in the program,
26 | # then also delete it in the license file.
27 | #
28 | """
29 | Manage the API Server
30 | """
31 | import builtins
32 | import os
33 | import re
34 | import getpass
35 | import argparse
36 | import logging
37 |
38 | logger = logging.getLogger('ctlsrv')
39 |
40 | parser = argparse.ArgumentParser()
41 |
42 | parser = argparse.ArgumentParser(prog='ctlsrv.py',
43 | description='Manage the API Server')
44 |
45 | parser.add_argument('--run', help='Run the API Server', action="store_true")
46 | parser.add_argument("--adduser", metavar="user", help="Add API User to DB", type=str)
47 | parser.add_argument("--newpass", metavar="user", help="Update Password", type=str)
48 | parser.add_argument("--testuser", metavar="user", help="Test Authentication", type=str)
49 | parser.add_argument("--deluser", metavar="user", help="Delete API User", type=str)
50 | parser.add_argument("--initdb", help="Initialize the Database", action="store_true")
51 | parser.add_argument("--debug", help="Set debugging level", type=int)
52 | parser.add_argument("-v", help="Verbose Output", action="store_true")
53 |
54 | args = parser.parse_args()
55 |
56 |
57 | # Default Config File Location
58 | config_file = '/etc/netgrph.ini'
59 | alt_config = './docs/netgrph.ini'
60 |
61 | # Test/Dev Config
62 | dirname = os.path.dirname(os.path.realpath(__file__))
63 | if re.search(r'\/dev$', dirname):
64 | config_file = 'netgrphdev.ini'
65 | elif re.search(r'\/test$', dirname):
66 | config_file = "netgrphdev.ini"
67 |
68 | # Test configuration exists
69 | if not os.path.exists(config_file):
70 | if not os.path.exists(alt_config):
71 | raise Exception("Configuration File not found", config_file)
72 | else:
73 | config_file = alt_config
74 |
75 | # Import API
76 | builtins.apisrv_CONFIG = config_file
77 | import apisrv
78 |
79 | verbose = 0
80 | if args.v:
81 | verbose = 1
82 | if args.debug:
83 | verbose = args.debug
84 |
85 | # Initialize the Database
86 | if args.initdb:
87 | apisrv.db.create_all()
88 | apisrv.db.session.commit()
89 |
90 | # Run the API Server
91 | elif args.run:
92 |
93 | builtins.apisrv_CONFIG = config_file
94 | from apisrv import app, debug, config
95 |
96 | # Production server in HTTPS Mode
97 | if config['apisrv']['https'] != '0':
98 | context = (config['apisrv']['ssl_crt'], config['apisrv']['ssl_key'])
99 | app.run(host='0.0.0.0', port=int(config['apisrv']['port']), \
100 | ssl_context=context, threaded=True, debug=debug)
101 |
102 | # Localhost development server
103 | else:
104 | logger.warning("HTTPS is not configured, defaulting to localhost only")
105 | app.run(debug=debug, port=int(config['apisrv']['port']))
106 |
107 | # Add user to DB
108 | elif args.adduser:
109 |
110 | passwd = getpass.getpass('Password:')
111 | verify = getpass.getpass('Verify Password:')
112 |
113 | if passwd == verify:
114 | phash = apisrv.user.add_user(args.adduser, passwd)
115 | if phash:
116 | print("Successfully Added User to Database")
117 | else:
118 | print("Error: Could not Add User to Database")
119 | else:
120 | print("Error: Passwords do not match")
121 |
122 | # Update User Password
123 | elif args.newpass:
124 | passwd = getpass.getpass('New Password:')
125 | verify = getpass.getpass('Verify Password:')
126 |
127 | if passwd == verify:
128 | phash = apisrv.user.update_password(args.newpass, passwd)
129 | if phash:
130 | print("Successfully Updated Password")
131 | else:
132 | print("Error: Could not Update Password")
133 | else:
134 | print("Error: Passwords do not match")
135 |
136 | # Delete a User
137 | elif args.deluser:
138 | ucheck = apisrv.user.del_user(args.deluser)
139 |
140 | if ucheck:
141 | print("Successfully Deleted User")
142 | else:
143 | print("Username not found in DB")
144 |
145 | # Test Authentication
146 | elif args.testuser:
147 | passwd = getpass.getpass('Password:')
148 | phash = apisrv.user.authenticate_user(args.testuser, passwd)
149 | if phash:
150 | print("Successfully Authenticated")
151 | else:
152 | print("Authentication Failed")
153 | else:
154 | parser.print_help()
155 | print()
--------------------------------------------------------------------------------
/cypher/buildfw.cyp:
--------------------------------------------------------------------------------
1 | --
2 | -- Manual Cypher Topology Commands
3 | -- Will not be purged by cache since they're missing a timestamp
4 | --
5 | -- Create a Default network and link it to the outside VRF
6 | MERGE (n:Network {vrfcidr:"outside-0.0.0.0/0"}) SET n += {name:"outside", cidr:"0.0.0.0/0", vid:'0', desc:"Default Route", vrf:"outside", gateway:"0.0.0.0"}
7 | MATCH (n:Network {cidr:"0.0.0.0/0"}), (v:VRF {name:"outside"}) MERGE (n)-[e:VRF_IN]->(v)
8 | MATCH (s:Switch {name:"ExternalFW"}), (n:Network {vrfcidr:"outside-0.0.0.0/0"}) MERGE (n)-[e:ROUTED_BY {vrf:"OUTSIDE"}]->(s)
9 |
10 | -- Create a SWITCHED_FW and insert it between two networks on different VRFs
11 | MERGE (fw:Switch:FW {name:'ExternalFW', type:"PaloAlto", hostname:"pa-fw-l2fw.domain.com", logIndex:"fwlogs"});
12 | MATCH (fw:Switch:FW {name:'ExternalFW'}), (n:Network {vid:"1095"}) MERGE (n)-[e:SWITCHED_FW]->(fw)
13 | MATCH (fw:Switch:FW {name:'ExternalFW'}), (n:Network {vid:"1099"}) MERGE (n)-[e:SWITCHED_FW]->(fw)
14 |
--------------------------------------------------------------------------------
/cypher/constraints.cyp:
--------------------------------------------------------------------------------
1 | -- Create constraints on DB for consistency and performance
2 | --CREATE Garbage
3 | CREATE CONSTRAINT ON (n:Network) ASSERT n.vrfcidr IS UNIQUE
4 | CREATE CONSTRAINT ON (s:Switch) ASSERT s.name IS UNIQUE
5 | CREATE CONSTRAINT ON (v:VRF) ASSERT v.name IS UNIQUE
6 | CREATE CONSTRAINT ON (v:VLAN) ASSERT v.name IS UNIQUE
7 | CREATE CONSTRAINT ON (e:Entity) ASSERT e.name IS UNIQUE
8 | CREATE INDEX ON :Network(cidr)
9 | CREATE INDEX ON :VLAN(vid)
10 | CREATE INDEX ON :VLAN(mgmt)
11 |
--------------------------------------------------------------------------------
/cypher/sample-queries.cyp:
--------------------------------------------------------------------------------
1 | --- Switched path between two switches
2 | MATCH (ss:Switch {name:"abc2sw1"}), (ds:Switch {name:"xyz2sw1"}),
3 | sp = allShortestPaths((ss)-[:NEI*0..5]-(ds)) RETURN sp
4 |
5 | -- All downstream neighbors from here
6 | MATCH(s:Switch {name:"core1"})-[e:NEI*0..5]->(rs), (s)-[p:NEI]-(parent) RETURN e,p
7 |
8 | -- All networks in the default table
9 | match (v:VRF {name:"default"})<-[e:VRF_IN]-(n:Network) return e
10 |
11 | -- Checkout the nglib/query modules
12 |
--------------------------------------------------------------------------------
/datasources/asaparse.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python3
2 | #
3 | #
4 | # Copyright (c) 2016 "Jonathan Yantis"
5 | #
6 | # This file is a part of NetGrph.
7 | #
8 | # This program is free software: you can redistribute it and/or modify
9 | # it under the terms of the GNU Affero General Public License, version 3,
10 | # as published by the Free Software Foundation.
11 | #
12 | # This program is distributed in the hope that it will be useful,
13 | # but WITHOUT ANY WARRANTY; without even the implied warranty of
14 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
15 | # GNU Affero General Public License for more details.
16 | #
17 | # You should have received a copy of the GNU Affero General Public License
18 | # along with this program. If not, see .
19 | #
20 | # As a special exception, the copyright holders give permission to link the
21 | # code of portions of this program with the OpenSSL library under certain
22 | # conditions as described in each individual source file and distribute
23 | # linked combinations including the program with the OpenSSL library. You
24 | # must comply with the GNU Affero General Public License in all respects
25 | # for all of the code used other than as permitted herein. If you modify
26 | # file(s) with this exception, you may extend this exception to your
27 | # version of the file(s), but you are not obligated to do so. If you do not
28 | # wish to do so, delete this exception statement from your version. If you
29 | # delete this exception statement from all source files in the program,
30 | # then also delete it in the license file.
31 | #
32 | #
33 | """Parse ASA Config Files for NetGrph Data"""
34 |
35 | import sys
36 | import os
37 | import re
38 | import argparse
39 | import socket
40 | import csv
41 | import ipaddress
42 | import logging
43 | from ciscoconfparse import CiscoConfParse
44 |
45 | logger = logging.getLogger(__name__)
46 |
47 |
48 | # Config Options
49 |
50 | fwdir = "/tftpboot/asafw/"
51 | extension = ".cfg"
52 |
53 | # Argument Parser
54 | parser = argparse.ArgumentParser()
55 | parser = argparse.ArgumentParser(description='Generate Cisco ASA Configuration Details')
56 |
57 | parser.add_argument("-fd", metavar='asafwfile', help="ASA FW Device file", type=str)
58 | parser.add_argument("-ffile", metavar='outfile', help="NetGrph Firewall File", type=str)
59 | parser.add_argument("-debug", help="Set debugging level", type=int)
60 |
61 | # Argument Reference
62 | args = parser.parse_args()
63 |
64 | DEBUG = 0
65 | if args.debug: DEBUG = args.debug
66 |
67 |
68 | def load_fwfile(fw_file):
69 | """Import ASA CSV Configuration File"""
70 |
71 | logger.debug("Importing Firewalls from " + fw_file)
72 |
73 | f = open(fw_file, 'r')
74 | fwdb = csv.DictReader(f)
75 |
76 | fwimport = []
77 |
78 | for en in fwdb:
79 | asafile = fwdir + en['FW'] + extension
80 |
81 | if os.path.exists(asafile):
82 | logger.debug("Found ASA File: " + asafile)
83 |
84 | fwints = process_asa_file(asafile, en)
85 | fwimport.append(fwints)
86 |
87 | else:
88 | logger.error("Could not find ASA file: " + asafile)
89 |
90 | return fwimport
91 |
92 |
93 | def process_asa_file(asafile, en):
94 | """Go through individual ASA Configuration Looking for Interfaces to import"""
95 |
96 | f = open(asafile)
97 |
98 | fwints = []
99 | instance = en['FW']
100 |
101 | instance = instance.title()
102 | instance = instance + "FW"
103 |
104 | inside = None
105 | intVlan = None
106 | desc = None
107 | seclevel = None
108 | ip = None
109 |
110 | for line in f:
111 | line = line.strip()
112 | ivlan = re.search(r'interface (Vlan\d+)', line)
113 | rdesc = re.search(r'^description ', line)
114 | rsec = re.search(r'^security-level (\d+)', line)
115 |
116 | if inside and re.search('^!', line):
117 | logger.debug("FW Found Ints: %s , %s , %s , %s , %s , %s , %s",
118 | instance, intVlan, desc, seclevel, ip, en['hostname'],
119 | en['logIndex'])
120 |
121 | fwints.append("{0},{1},{2},{3},{4},{5},{6}".format(
122 | instance, intVlan, desc, seclevel, ip, en['hostname'], en['logIndex']))
123 |
124 | inside = None
125 | intVlan = None
126 | desc = None
127 | seclevel = None
128 | ip = None
129 |
130 | if ivlan:
131 | inside = 1
132 | intVlan = ivlan.group(1)
133 | elif inside and rdesc:
134 | desc = line
135 | desc = desc.replace('description ', '')
136 | elif inside and rsec:
137 | seclevel = rsec.group(1)
138 |
139 | # Return found ints
140 | return fwints
141 |
142 |
143 | # Write results to file
144 | def save_fw_file(data, out_file):
145 |
146 | save = open(out_file, "w")
147 | print("Name,Interface,Description,Security-Level,IP,Hostname,Log-Index", file=save)
148 |
149 | for fw in data:
150 | for entry in fw:
151 | print(entry, file=save)
152 | save.close()
153 |
154 |
155 | ## Process Arguments to generate output
156 | # Got VLAN Range and Switch Name
157 | if args.fd and args.ffile:
158 |
159 | loglevel = logging.INFO
160 |
161 | if DEBUG:
162 | loglevel = logging.DEBUG
163 |
164 | logging.basicConfig(format='%(asctime)s %(name)s:%(levelname)s: %(message)s',
165 | stream=sys.stdout, level=loglevel)
166 |
167 | # Load FW Data
168 | fwimport = load_fwfile(args.fd)
169 |
170 | # Save FW Data
171 | save_fw_file(fwimport, args.ffile)
172 |
173 |
174 | # Print Help
175 | else:
176 | parser.print_help()
177 | print()
178 |
--------------------------------------------------------------------------------
/datasources/csnip.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python3
2 | #
3 | # Generate Config Snippets from Cisco Devices
4 | #
5 | # Copyright (c) 2016 "Jonathan Yantis"
6 | #
7 | # This file is a part of NetGrph.
8 | #
9 | # This program is free software: you can redistribute it and/or modify
10 | # it under the terms of the GNU Affero General Public License, version 3,
11 | # as published by the Free Software Foundation.
12 | #
13 | # This program is distributed in the hope that it will be useful,
14 | # but WITHOUT ANY WARRANTY; without even the implied warranty of
15 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
16 | # GNU Affero General Public License for more details.
17 | #
18 | # You should have received a copy of the GNU Affero General Public License
19 | # along with this program. If not, see .
20 | #
21 | # As a special exception, the copyright holders give permission to link the
22 | # code of portions of this program with the OpenSSL library under certain
23 | # conditions as described in each individual source file and distribute
24 | # linked combinations including the program with the OpenSSL library. You
25 | # must comply with the GNU Affero General Public License in all respects
26 | # for all of the code used other than as permitted herein. If you modify
27 | # file(s) with this exception, you may extend this exception to your
28 | # version of the file(s), but you are not obligated to do so. If you do not
29 | # wish to do so, delete this exception statement from your version. If you
30 | # delete this exception statement from all source files in the program,
31 | # then also delete it in the license file.
32 | #
33 | #
34 | import sys
35 | import re
36 | import argparse
37 | import csv
38 | from ciscoconfparse import CiscoConfParse
39 |
40 | # Config Options
41 | conf_dir = "/tftpboot/"
42 | extension = "-confg"
43 |
44 | # Argument Parser
45 | parser = argparse.ArgumentParser()
46 | parser = argparse.ArgumentParser(description='Generate Cisco Snippets')
47 | parser.add_argument("switch", help="Switch to get snippet from",
48 | type=str)
49 | parser.add_argument("-int", metavar="interface",
50 | help="Interface (switchport or VLAN) to pull config from",
51 | type=str)
52 | parser.add_argument('-match', metavar='string',
53 | help='Match a string on an interface',
54 | type=str)
55 | parser.add_argument("-debug", help="Set debugging level", type=int)
56 |
57 | # Argument Reference
58 | args = parser.parse_args()
59 |
60 |
61 | DEBUG = 0
62 | if args.debug: DEBUG = args.debug
63 |
64 |
65 | def get_int(switch, interface):
66 | """ Get a config snippet from a device """
67 |
68 | conf_file = conf_dir + switch + extension
69 |
70 | interface = expand_port(interface)
71 |
72 | try:
73 | parse = CiscoConfParse(conf_file)
74 | except:
75 | print("Error: could not load config for ", conf_file)
76 | sys.exit(1)
77 |
78 | search_int = "^interface " + interface + "$"
79 |
80 |
81 |
82 | if args.match:
83 | m = parse.find_objects_w_child(parentspec=search_int,
84 | childspec=args.match)
85 | if m:
86 | print(args.switch + ',' + args.int + ',' + args.match)
87 |
88 | else:
89 | iface = parse.find_all_children(search_int)
90 | if iface:
91 | print('!' + switch + ' ' + interface)
92 | for line in iface:
93 | print(line)
94 | if iface:
95 | print('!')
96 |
97 |
98 | def normalize_port(port):
99 | """Normalize Ports (GigabitEthernet1/1 -> Gi1/1)"""
100 |
101 |
102 | port = port.replace('interface ', '')
103 | port = port.replace('TenGigabitEthernet', 'Te')
104 | port = port.replace('GigabitEthernet', 'Gi')
105 | port = port.replace('FastEthernet', 'Fa')
106 | port = port.replace('Ethernet', 'Eth')
107 | port = port.replace('v', 'Vlan')
108 | port = port.replace('vlan', 'Vlan')
109 |
110 | return port
111 |
112 | def expand_port(port):
113 |
114 | port = port.title()
115 |
116 | port = re.sub(r'^Gi(\d+)', r'GigabitEthernet\1', port)
117 | port = re.sub(r'^Te(\d+)', r'TenGigabitEthernet\1', port)
118 | port = re.sub(r'^Fa(\d+)', r'FastEthernet\1', port)
119 | port = re.sub(r'^Eth(\d+)', r'Ethernet\1', port)
120 | port = re.sub(r'^E(\d+)', r'Ethernet\1', port)
121 | port = re.sub(r'^Po(\d+)', r'Port-channel\1', port)
122 |
123 | return port
124 |
125 | ## Process Arguments to generate output
126 | # Got VLAN Range and Switch Name
127 | if args.switch and args.int:
128 |
129 | get_int(args.switch, args.int)
130 |
131 | # Print Help
132 | else:
133 | parser.print_help()
134 | print()
135 |
--------------------------------------------------------------------------------
/datasources/netdb-devfile.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python3
2 | #
3 | # Generate Device File from NetDB
4 | #
5 | # Copyright (c) 2016 "Jonathan Yantis"
6 | #
7 | # This file is a part of NetGrph.
8 | #
9 | # This program is free software: you can redistribute it and/or modify
10 | # it under the terms of the GNU Affero General Public License, version 3,
11 | # as published by the Free Software Foundation.
12 | #
13 | # This program is distributed in the hope that it will be useful,
14 | # but WITHOUT ANY WARRANTY; without even the implied warranty of
15 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
16 | # GNU Affero General Public License for more details.
17 | #
18 | # You should have received a copy of the GNU Affero General Public License
19 | # along with this program. If not, see .
20 | #
21 | # As a special exception, the copyright holders give permission to link the
22 | # code of portions of this program with the OpenSSL library under certain
23 | # conditions as described in each individual source file and distribute
24 | # linked combinations including the program with the OpenSSL library. You
25 | # must comply with the GNU Affero General Public License in all respects
26 | # for all of the code used other than as permitted herein. If you modify
27 | # file(s) with this exception, you may extend this exception to your
28 | # version of the file(s), but you are not obligated to do so. If you do not
29 | # wish to do so, delete this exception statement from your version. If you
30 | # delete this exception statement from all source files in the program,
31 | # then also delete it in the license file.
32 | #
33 | #
34 | import sys
35 | import re
36 | import argparse
37 | import socket
38 |
39 | # Config Options
40 | configDir = "/opt/netdb/data/"
41 |
42 | # Argument Parser
43 | parser = argparse.ArgumentParser()
44 | parser = argparse.ArgumentParser(description='Generate Devicefile Database')
45 | parser.add_argument("-df", metavar='neighbor_file', help="devicefile.csv from NetDB", type=str)
46 | parser.add_argument("-of", metavar='outfile', help="Output File for CSV", type=str)
47 | parser.add_argument("-debug", help="Set debugging level", type=int)
48 |
49 | # Argument Reference
50 | args = parser.parse_args()
51 |
52 | DEBUG = 0
53 | if args.debug: DEBUG = args.debug
54 |
55 | def load_nd_file(fileName):
56 |
57 | df = open(fileName, 'r')
58 |
59 | devdata = []
60 |
61 | for line in df:
62 | if re.search('^\#', line ):
63 | continue
64 |
65 | line = line.strip()
66 | en = line.split(',')
67 |
68 | device = get_entry(en[0].split('.'))
69 | fqdn = en[0]
70 | mgmt = None
71 | dtype = "Switch"
72 | platform = 'cisco_ios'
73 |
74 | for e in en:
75 | # Device Flags
76 | mg = re.search('mgmtgroup\=(\w+)', e)
77 | arp = re.search('^(arp|netdbarp)$', e)
78 | standby = re.search('^(standby)$', e)
79 | p = re.search('devtype\=(\w+)', e)
80 | if mg:
81 | mgmt = mg.group(1)
82 | elif arp:
83 | dtype = "Primary"
84 | elif standby:
85 | dtype = "Standby"
86 | elif p:
87 | #print(p.group(1))
88 | platform = tr_devtype(p.group(1))
89 |
90 | #print(router)
91 | #print(en)
92 |
93 | dev = ("{0},{1},{2},{3},{4}").format(device,fqdn,mgmt,dtype,platform)
94 | devdata.append(dev)
95 |
96 | return devdata
97 |
98 | def tr_devtype(dt):
99 | """Translation NetDB devtype to Netmiko"""
100 | if dt == 'nxos':
101 | dt = 'cisco_nxos'
102 | elif dt == 'asa':
103 | dt = 'cisco_asa'
104 |
105 | return dt
106 |
107 |
108 | def save_nd_data(ndata):
109 |
110 | save = open(args.of, "w")
111 | print("Device,FQDN,MgmtGroup,Type,Platform", file=save)
112 |
113 | print(*ndata, sep='\n', file=save)
114 | save.close()
115 |
116 | def get_entry(l,pos=0):
117 | return l[pos]
118 |
119 | ## Process Arguments to generate output
120 | if args.df and args.of:
121 |
122 | devdata = load_nd_file(args.df)
123 | save_nd_data(devdata)
124 |
125 | # Print Help
126 | else:
127 | parser.print_help()
128 | print()
129 |
--------------------------------------------------------------------------------
/datasources/netdb-nd.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python3
2 | #
3 | # Generate Neighbor Data from NetDB
4 | #
5 | # Copyright (c) 2016 "Jonathan Yantis"
6 | #
7 | # This file is a part of NetGrph.
8 | #
9 | # This program is free software: you can redistribute it and/or modify
10 | # it under the terms of the GNU Affero General Public License, version 3,
11 | # as published by the Free Software Foundation.
12 | #
13 | # This program is distributed in the hope that it will be useful,
14 | # but WITHOUT ANY WARRANTY; without even the implied warranty of
15 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
16 | # GNU Affero General Public License for more details.
17 | #
18 | # You should have received a copy of the GNU Affero General Public License
19 | # along with this program. If not, see .
20 | #
21 | # As a special exception, the copyright holders give permission to link the
22 | # code of portions of this program with the OpenSSL library under certain
23 | # conditions as described in each individual source file and distribute
24 | # linked combinations including the program with the OpenSSL library. You
25 | # must comply with the GNU Affero General Public License in all respects
26 | # for all of the code used other than as permitted herein. If you modify
27 | # file(s) with this exception, you may extend this exception to your
28 | # version of the file(s), but you are not obligated to do so. If you do not
29 | # wish to do so, delete this exception statement from your version. If you
30 | # delete this exception statement from all source files in the program,
31 | # then also delete it in the license file.
32 | #
33 | #
34 | import sys
35 | import re
36 | import argparse
37 | import socket
38 |
39 | # Config Options
40 | configDir = "/opt/netdb/data/"
41 |
42 | # Argument Parser
43 | parser = argparse.ArgumentParser()
44 | parser = argparse.ArgumentParser(description='Generate ND Database')
45 | parser.add_argument("-nf", metavar='neighbor_file', help="Neighbor Data from NetDB", type=str)
46 | parser.add_argument("-of", metavar='outfile', help="Output File for CSV", type=str)
47 | parser.add_argument("-debug", help="Set debugging level", type=int)
48 |
49 | # Argument Reference
50 | args = parser.parse_args()
51 |
52 | DEBUG = 0
53 | if args.debug: DEBUG = args.debug
54 |
55 | def load_nd_file(fileName):
56 |
57 | nd = open(fileName, 'r')
58 |
59 | ndata = []
60 |
61 | for line in nd:
62 | if re.search('^\#', line ):
63 | continue
64 |
65 | line = line.strip()
66 | en = line.split(',')
67 | localName = en[0]
68 | localPort = en[1]
69 | remoteName = get_entry(en[2].split('.'))
70 | remotePort = en[6]
71 |
72 | # Save data to list
73 | ndline = "{0},{1},{2},{3}".format(localName,localPort,remoteName,remotePort)
74 | ndata.append(ndline)
75 |
76 | return ndata
77 |
78 | def save_nd_data(ndata):
79 |
80 | save = open(args.of, "w")
81 | print("LocalName,LocalPort,RemoteName,RemotePort", file=save)
82 |
83 | print(*ndata, sep='\n', file=save)
84 | save.close()
85 |
86 | def get_entry(l,pos=0):
87 | return l[pos]
88 |
89 | ## Process Arguments to generate output
90 | if args.nf and args.of:
91 |
92 | ndata = load_nd_file(args.nf)
93 | save_nd_data(ndata)
94 |
95 | # Print Help
96 | else:
97 | parser.print_help()
98 | print()
99 |
--------------------------------------------------------------------------------
/datasources/update_csv_from_netdb.sh:
--------------------------------------------------------------------------------
1 | #!/bin/sh
2 | # Change to the current netgrph folder
3 | export NGHOME=/home/netgrph/netgrph
4 | export NGCSV=/home/netgrph/csv
5 | export NETDBDATA=/opt/netdb/data
6 |
7 | cd $NGHOME/datasources/
8 | ./netdb-nd.py -nf $NETDBDATA/nd.csv -of $NGCSV/nd.csv
9 | ./netdb-devfile.py -df $NETDBDATA/devicelist.csv -of $NGCSV/devices.csv
10 |
--------------------------------------------------------------------------------
/datasources/update_parse.sh:
--------------------------------------------------------------------------------
1 | #/bin/sh
2 | export NGHOME=/home/netgrph/netgrph
3 | export NGCSV=/home/netgrph/csv
4 | export NETDBDATA=/opt/netdb/data
5 |
6 | $NGHOME/datasources/ciscoparse.py -vr 1-4096 -ivr 0-4096 -df $NGCSV/devices.csv -dfile $NGCSV/devinfo.csv -mf /scripts/versions.txt -lfile $NGCSV/links.csv -ifile $NGCSV/allnets.csv -vfile $NGCSV/allvlans.csv
7 | $NGHOME/datasources/asaparse.py -fd $NGCSV/asafirewalls.csv -debug 0 -ffile $NGCSV/firewalls.csv
8 |
--------------------------------------------------------------------------------
/docs/API.md:
--------------------------------------------------------------------------------
1 | # REST API Reference
2 |
3 | * Version 1.1 of the API allows granular querying of the data model via REST Calls
4 | * Most query options support Neo4j/JAVA style regexes
5 | * **Base URL**
6 | * __`/netgrph/api/v1.1`__
7 |
8 | ## Setup
9 | * Configure the [api] section in netgrph.ini
10 | * Enable HTTPS using valid certs for serving API calls beyond localhost
11 | * Setup Process
12 | * Initialize the SQL Lite database: ```./ctlsrv.py --initdb```
13 | * Add an API User: ```./ctlsrv --adduser testuser```
14 | * Run the Server: ```./ctlsrv.py --run```
15 | * Test a Query: ```curl -u testuser:testpass http://localhost:4096/netgrph/api/v1.1/devs```
16 |
17 | # Examples
18 |
19 | ### Switch path
20 |
21 | ```
22 | $ curl -u testuser:testpass 'http://localhost:4096/netgrph/api/v1.0/spath?src=mdcmdf&dst=core1'
23 | {
24 | "Distance": 1,
25 | "Links": 1,
26 | "Name": "mdcmdf -> core1",
27 | "Search Depth": "20",
28 | "Traversal Type": "All Paths",
29 | "_ccount": 1,
30 | "_type": "L2-PATH",
31 | "data": [
32 | {
33 | "From Channel": "0",
34 | "From Model": "WS-C3850-48U",
35 | "From Port": "Te1/1/4",
36 | "From Switch": "mdcmdf",
37 | "Link VLANs": "1246,2200,2314,2320,2324,2365,2376",
38 | "Link rVLANs": "1246,2200,2314,2320,2324,2365,2376",
39 | "Name": "#1 mdcmdf(Te1/1/4) -> core1(Eth10/21)",
40 | "Native VLAN": "2200",
41 | "To Channel": "0",
42 | "To Model": "Nexus7000 C7010",
43 | "To Port": "Eth10/21",
44 | "To Switch": "core1",
45 | "_ccount": 0,
46 | "_cversion": "03.06.04.E RELEASE SOFTWARE (fc2)",
47 | "_pversion": "Version 6.2(16)",
48 | "_reverse": 1,
49 | "_rvlans": "1246,2200,2314,2320,2324,2365,2376",
50 | "_type": "L2-HOP",
51 | "data": [],
52 | "distance": 1
53 | }
54 | ]
55 | }
56 | ```
57 |
58 | ### Networks on a filter (guest VRF with guest access role)
59 |
60 | ```
61 | curl -u testuser:testpass 'http://localhost:4096/netgrph/api/v1.1/nets?filter=guest:nac-guest'
62 | {
63 | "Count": 23,
64 | "Filter": "guest:nac-guest",
65 | "Name": "Networks",
66 | "_ccount": 23,
67 | "_type": "NET",
68 | "data": [
69 | {
70 | "CIDR": "10.29.2.0/23",
71 | "Description": "None",
72 | "Gateway": "10.29.2.1",
73 | "Location": null,
74 | "Name": "10.29.2.0/23",
75 | "NetRole": "nac-guest",
76 | "Router": "mdcmdf",
77 | "SecurityLevel": "10",
78 | "StandbyRouter": null,
79 | "VLAN": "270",
80 | "VRF": "guest",
81 | "_type": "CIDR",
82 | "vrfcidr": "guest-10.29.2.0/23"
83 | },
84 | ```
85 | ___
86 |
87 | # Device Calls
88 |
89 |
90 | Device Queries allow you to query the API for all devices, regexes and gather
91 | specific information from individual devices.
92 |
93 | * __URLs__
94 | * __`/devs`__: List all Devices
95 | * __`/devs/{device}`__: Specific Device
96 | * __`/devs/{device}/vlans`__: Device VLANs
97 | * __`/devs/{device}/neighbors`__: Device Neighbors
98 | * __`/devs/{device}/nets`__: Device Networks
99 |
100 | * **Method:**
101 | * `GET`
102 |
103 | * __URL Parameters__
104 | * __`group`__: Restrict on device group regex
105 | * __`search`__: Restrict on device name regex
106 | * __`full`__: Return full device reports (non-truncated and slower)
107 |
108 | * **Success Response:**
109 | * **Code:** 200
110 | **Content:** `{ Name : Report, _type : "VIDs" }`
111 |
112 | * **Error Response:**
113 | * **Code:** 401
114 | **Content:** `{ message : "Request Error" }`
115 |
116 | ___
117 |
118 | # Network Calls
119 |
120 | Network Queries allow you to query for networks based on CIDRs, VRF, Role and
121 | Group filters.
122 |
123 | * __URLs__
124 | * __`/nets`__: List of All Networks
125 | * __`/nets?group=X&cidr=X`__: Networks filtered on group and cidr
126 | * __`/nets?ip=X`__: Most Specific CIDR from IP
127 |
128 | * **Method:**
129 | * `GET`
130 |
131 | * __URL Parameters__
132 | * __`group`__: Restrict on device group regex
133 | * __`cidr`__: Restrict on device name regex (optionally replace /24 mask with -24)
134 | * __`filter`__: Filter via NetGrph vrf:role syntax
135 | * __`ip`__: Find most specific CIDR for IP
136 |
137 | * **Success Response:**
138 | * **Code:** 200
139 | **Content:** `{ Name : Report, _type : "Networks" }`
140 |
141 | * **Error Response:**
142 | * **Code:** 401
143 | **Content:** `{ message : "Request Error" }`
144 |
145 | ___
146 |
147 | # VLAN Calls
148 |
149 | VLAN Queries return lists of all VLANs, specific VLAN trees and VLANs for groups
150 |
151 | * __URLs__
152 | * __`/vlans`__: Retrieve All VLANs
153 | * __`/vlans/`__: Retrieve Specific VLAN in VID or VNAME Format
154 |
155 | * __URL Parameters__
156 | * __`group`__: VLANs in a Management Group
157 | * __`range`__: VLAN Range eg. 1-1005
158 |
159 | * **Success Response:**
160 | * **Code:** 200
161 | **Content:** `{ Name : Report, _type : "VLANs" }`
162 |
163 | * **Error Response:**
164 | * **Code:** 401
165 | **Content:** `{ message : "Request Error" }`
166 |
167 | ___
168 |
169 | # Path Calls
170 |
171 | Path queries allow you to do L2-L4 traversals between any two points on the network.
172 |
173 | * __URLs__
174 | * __`/path?src=[ip]&dst=[ip]`__: Universal L2-L4 Path Query
175 | * __`/spath?src=[switch]&dst[switch.*]`__: Switch paths using a regex
176 | * __`/rpath?src=[ip]&dst=[ip]&vrf=pci`__: L3 Path inside a VRF
177 | * __`/fpath?src=[ip]&dst=[ip]`__: L4 Path Query
178 |
179 | * __URL Parameters__
180 | * __`src`__(required): Source of Path Query (regex support on switch paths)
181 | * __`dst`__(required): Destination of Path Query
182 | * __`onepath`__(set True): Only show one path, no ECMP
183 | * __`depth`__: Depth of Graph Search (default 20)
184 | * __`vrf`__: VRF for Routed Queries
185 | * __Note__: src and dst support regexes on switch paths
186 |
187 | * **Method:**
188 | * `GET`
189 |
190 | * **Success Response:**
191 | * **Code:** 200
192 | **Content:** `{ Name : Report, _type : "PATH" }`
193 |
194 | * **Error Response:**
195 | * **Code:** 401
196 | **Content:** `{ message : "Request Error" }`
197 |
198 |
--------------------------------------------------------------------------------
/docs/About.md:
--------------------------------------------------------------------------------
1 | ## About
2 |
3 | NetGrph was created to serve as a singular network model for automation to
4 | replace the patchwork inherent to our current automation solutions. The API is
5 | developing to provide strong automation capabilities for our network team,
6 | as well as our server, security, and desktop teams.
7 |
8 | ## Data Model
9 | ### Routed SVI Paths from Vlan 110 to 200
10 | 
11 |
12 |
13 | [L3 SVIs: Yellow] [L2 VLANs: Green] [Switches/Routers: Blue]
14 |
15 |
16 |
17 | ### Security Path from Vlan 696 --> 641 across multiple L2/L3 Firewalls
18 | 
19 |
20 |
21 | [Networks: Yellow] [VRFs: Green] [Firewalls: Blue]
22 |
23 |
24 |
25 | ### Neighbor Tree from the Core out to a distance of 3
26 |
27 | 
28 |
29 |
30 |
31 |
32 |
33 | ## Performance
34 |
35 | On our network containing 700+ routers/switches, 2000+ SVIs and 10,000+ VLAN to
36 | switch mappings, performance on almost all netgrph queries is below 150ms. Full
37 | ngreport queries can take longer, but it's for creating network-wide reports.
38 |
39 | The database update time on our localhost server with SSDs is 150sec for a
40 | complete network refresh. When importing to an unoptimized VM on a remote
41 | server, the import time is 5min. The network import speed could be dramatically
42 | increased if rewritten using the Bolt driver, which is planned.
43 |
44 | ## Dependencies
45 |
46 | NetGrph is a Python3.4+ application, but could be back-ported to Python2.7
47 | fairly easily. It relies on both the py2neo and neo4j bolt driver (install via
48 | pip3). The plan is to eventually convert all driver code to the bolt driver as
49 | it matures.
50 |
51 | NetGrph requires [CSV files](test/csv/) with all of your Switches/Routers, Networks,
52 | VLANs, and CDP/LLDP Neighbors in order to be multi-vendor compatible. I provide
53 | IOS and NXOS configuration parsers, as well as a sample network topology to play
54 | with. For Cisco-based networks, a generic CDP/LLDP mapper is all that's missing.
55 | NetGrph is compatible with NetDB Neighbor Data:
56 | (https://sourceforge.net/projects/netdbtracking/). Once stable, I'll package a
57 | bundled NetDB OVA with NetGrph included.
58 |
59 | If anyone has a dependable LLDP/CDP walker they recommend (I'm sure there are
60 | many), please contact me. This only needs src_switch,src_port,dst_switch,dst_port.
61 | I have tested a few different open-source packages, but have yet to find a suitable
62 | tool that is reliable enough.
63 |
64 | NetGrph was developed on Ubuntu 14.04 LTS, tested on 16.04, and should be
65 | compatible with other versions of linux. I highly recommend using Ubuntu at this
66 | early stage for better support. I also plan to create an ansible build script in
67 | the next few months. It also works just find on MacOS but there are no Ansible or
68 | install instructions for that OS.
69 |
70 | ## Planned Features
71 |
72 | * Add configuration snippets for each hop on traverals
73 | * Import all Network ACL's for L4 analysis
74 | * Improve NetDB integration with universal search
75 | * Implement Dijkstra's Algorithm for cost-based path traversals (database plugin)
76 | * Simple Web Interface for Path Traversals and report generation
77 | * Statseeker integration for including graphs/errors in reports
78 |
79 | ## Future
80 |
81 | I am open to expanding NetGrph for the needs of MPLS networks and other
82 | network/security domains where appropriate. The application was written to be
83 | generic and approachable for use with both SDN and existing networks.
84 |
85 | I have also added some lightweight integration with my existing [NetDB
86 | application](http://netdbtracking.sourceforge.net), but that will be both
87 | focused and optional. If you manage to create any new parsers or integrate with
88 | other vendor APIs, please contribute your code back.
89 |
90 | I would like to add a GUI, but at this time I'm focussed on using the
91 | application to automate tasks. In theory, it should be easy to create [D3
92 | visualizations](https://github.com/d3/d3/wiki/Gallery) from the NGTree
93 | data-structures. If anyone manages to create a simple GUI or use this
94 | application to create some interesting visualizations, I'd be happy to help and
95 | would love to see the results.
96 |
97 | I will not be expanding the application to be a full-blown NMS, since there are
98 | many tools that accomplish this. If you manage to expand the core modeling
99 | functionality and think it should be included in the main codebase, I'd like to
100 | consider including it here.
101 |
102 | See the [CONTRIBUTING](CONTRIBUTING.md) document for more information.
103 |
104 | ## Support
105 |
106 | You can open an issue via GitHub, or if you would like to speak with me
107 | directly, I monitor the #netgrph channel the [networktocode slack
108 | group](https://networktocode.herokuapp.com/). Please try and contact me there
109 | for any interactive support.
110 |
111 | ## Contributions
112 |
113 | Please see the [Contributions](CONTRIBUTING.md) document in docs for
114 | information about how you can contribute back to NetGrph.
115 |
116 | ## Contributors
117 | * Jonathan Yantis ([yantisj](https://github.com/yantisj))
118 |
119 | ## License
120 | NetGrph is licensed under the GNU AGPLv3 License.
121 |
--------------------------------------------------------------------------------
/docs/CHANGES.md:
--------------------------------------------------------------------------------
1 |
2 | ### v0.7.3
3 | - Added vrf and dev to ngreport
4 | - Added nfilter to netgrph to pass in custom filters at runtime
5 | - Cleaned up debug netgrph.ini handling
6 | - VLAN Group printing follows terminal size
7 |
8 | ### v0.7.0
9 | - Created public repo
10 | - Added lots of documentation
11 | - Cleaned up config files
12 |
--------------------------------------------------------------------------------
/docs/CLI.md:
--------------------------------------------------------------------------------
1 | # CLI Documentation
2 |
3 | NetGrph has two CLI query programs, netgrph.py and ngreport.py. The first is for
4 | queries such as networks, paths etc. The second is for network-wide reporting on
5 | devices, VRFs etc.
6 |
7 | Data can be returned in several outputs via the --output switch
8 |
9 | ## Query Options
10 | ```
11 |
12 | usage: netgrph [-h] [-ip] [-net] [-nlist] [-dev] [-fpath src] [-rpath src]
13 | [-spath src] [-group] [-vrange 1[-4096]] [-vid] [-vtree]
14 | [-output TREE] [--conf file] [--debug DEBUG] [--verbose]
15 | search
16 |
17 | Query the NetGrph Database
18 |
19 | positional arguments:
20 | search Search the NetGrph Database (Wildcard Default)
21 |
22 | optional arguments:
23 | -h, --help show this help message and exit
24 | -ip Network Details for an IP
25 | -net All networks within a CIDR (eg. 10.0.0.0/8)
26 | -nlist Get all networks in an alert group
27 | -dev Get the Details for a Device (Switch/Router/FW)
28 | -fpath src Security Path between -fp src dst
29 | -rpath src Routed Path between -rp IP/CIDR1 IP/CIDR2
30 | -spath src Switched Path between -sp sw1 sw2 (Neo4j Regex)
31 | -group Get VLANs for a Management Group
32 | -vrange 1[-4096] VLAN Range (default 1-1999)
33 | -vid VLAN ID Search
34 | -vtree Get the VLAN Tree for a VNAME
35 | -output TREE Return Format: TREE, TABLE, CSV, JSON, YAML
36 | --conf file Alternate Config File
37 | --debug DEBUG Set debugging level
38 | --verbose Verbose Output
39 |
40 | Examples: netgrph 10.1.1.1 (Free Search for IP), netgrph -net 10.1.1.0/24
41 | (Search for CIDR), netgrph -group MDC (VLAN Database Search), netgrph -fp
42 | 10.1.1.1 10.2.2.1 (Firewall Path Search)
43 |
44 | ```
45 |
46 |
47 | ## Report Options
48 | ```
49 | $ ngreport -h
50 | usage: ngreport [-h] [-vrf name] [-vrfs] [-vlans] [-vrange 1[-4096]] [-dev .*]
51 | [-output TREE] [-empty] [--conf file] [--debug DEBUG]
52 | [--verbose]
53 |
54 | Generate Reports from NetGrph
55 |
56 | optional arguments:
57 | -h, --help show this help message and exit
58 | -vrf name Generate a Report on a VRF
59 | -vrfs VRF Report on all VRFs
60 | -vlans VLAN ID Report (combine with -vra and -empty)
61 | -vrange 1[-4096] VLAN Range (default 1-1999)
62 | -dev .* Report on Network Devices in regex
63 | -output TREE Return Format: TREE, TABLE, CSV, JSON, YAML
64 | -empty Only Return Empty VLANs (requires NetDB)
65 | --conf file Alternate Config File
66 | --debug DEBUG Set debugging level
67 | --verbose Verbose Output
68 | ```
69 |
--------------------------------------------------------------------------------
/docs/CONTRIBUTING.md:
--------------------------------------------------------------------------------
1 | ## Contributions
2 |
3 | These are the early days of NetGrph, but I fully plan to support this program
4 | going forward and am looking for contributors. I am also the primary maintainer
5 | of the [Network Tracking Database
6 | (NetDB)](http://netdbtracking.sourceforge.net/), which is in production on some
7 | extremely large networks. NetGrph is currently serving as our internal model for
8 | network automation tasks, and we hope others find it useful as well. It's
9 | primary purpose is to do work at this time, from automation to troubleshooting
10 | tasks.
11 |
12 | ### Data Sources
13 |
14 | NetGrph is short on data sources at this time, so if you manage to generate the
15 | necessary [CSV files](csv/) for your network, I will most likely accept them in
16 | to the repository without much trouble. All source data is CSV based to make it
17 | easy for others to contribute to the project. Please explore those files for
18 | understanding how to get your network data in to the program.
19 |
20 | I am considering using [Napalm](https://github.com/napalm-automation/napalm) in
21 | the future for a more robust data gathering source, but at the moment I will be
22 | busy integrating this application in to our automation and troubleshooting test
23 | suite. If you have some ideas or would like to work on this, please contact me
24 | on Slack. I am open to any and all solutions at the moment, as long as they are
25 | simple, maintainable and based off of other open-source network tools.
26 |
27 | ### Core Library Changes
28 |
29 | [nglib](nglib/) is the core library of the application. Currently, the only code
30 | using the library are the CLI scripts netgrph, ngupdate and ngreport. This
31 | application was written with APIs in mind though, so the next phase after
32 | finishing up all the reporting and user functionality via the CLI will be to
33 | create an API via Flask. I will also be creating some user facing
34 | troubleshooting GUI tools, but at this time, the GUI plans are quite simple.
35 | That said, there is no reason via nglib and a framework like Flask, that this
36 | could not be developed in to a full fledged GUI app.
37 |
38 | If you submit a pull request on the core library, please make sure you put your
39 | code through the pylint process and follow the existing code style. If you have
40 | any concerns about whether your code will be accepted, please contact me on
41 | Slack. I'd like to see contributions beyond simple data sources as long as they
42 | don't break the data model or complicate the maintainability of the code. This
43 | is very much an experimental program at this time, so I'm open to new directions
44 | from here.
45 |
--------------------------------------------------------------------------------
/docs/INSTALL.md:
--------------------------------------------------------------------------------
1 | ## Install Notes
2 |
3 | Netgrph was built on Ubuntu 14.04 LTS but should be portable to any other Python
4 | 3.4+ system. I have done testing on Ubuntu 16.04 and MacOS, and I highly
5 | recommend Ubuntu trusty or xenial for support purposes. I provide ansible
6 | scripts for an easy install on a base Ubuntu system, and will be creating a
7 | Docker file before long.
8 |
9 | ### Ansible Install
10 | - See the README.md in docs/playbooks/
11 | - Install ansible on your machine, and point the playbook towards localhost as
12 | documented, or a remote host if you are familiar with Ansible.
13 |
14 | ### Manual Install Instructions
15 |
16 | - For testing, you can install everything but the database under your user on any system
17 | - If you do not have root access to your system, use virtualenv to satisfy the pip requirements
18 |
19 | - Clone the github repository to ~/
20 | - Install [Neo4j Community Edition](http://neo4j.com/download/) (available as a package for Ubuntu but requires java8+)
21 | ```
22 | ## Python system requirements includes testing libraries
23 | sudo apt-get install python3-pip python3-pytest python3-logilab-common
24 |
25 | # Java and Neo4j
26 | sudo add-apt-repository ppa:webupd8team/java
27 |
28 | sudo apt-get update
29 | echo oracle-java8-installer shared/accepted-oracle-license-v1-1 select true | sudo /usr/bin/debconf-set-selections
30 | sudo apt-get install oracle-java8-installer
31 | wget -O - https://debian.neo4j.org/neotechnology.gpg.key | sudo apt-key add -
32 | echo 'deb http://debian.neo4j.org/repo stable/' >/tmp/neo4j.list
33 | sudo mv /tmp/neo4j.list /etc/apt/sources.list.d
34 | sudo apt-get update
35 | sudo apt-get install neo4j
36 | ```
37 |
38 | - To allow remote connections to neo4j, open this file and uncomment this line:
39 | ```
40 | sudo vim /etc/neo4j/neo4j.conf
41 | dbms.connector.http.address=0.0.0.0:7474
42 | ```
43 |
44 | - To install the software on a server separate from the database, uncomment this:
45 | ```
46 | dbms.connector.bolt.address=0.0.0.0:7687
47 | ```
48 |
49 | - Browse to the database http://localhost:7474 and set a password (takes a few minutes to startup).
50 | - Configure ~/netgrph/docs/netgraph.ini with your DB password.
51 | - Install json, yaml, pymysql, py2neo and neo4j-bolt packages:
52 | ```
53 | sudo pip3 install -r requirements.txt
54 | ```
55 | - Import the sample network: `./test/first_import.sh`
56 | - Run `./ngupdate.py -v -full` a couple more times to seed the network topology
57 | - Browse to the database web interface and run `MATCH (s:Switch) RETURN s`
58 | - Try out some sample commands
59 |
60 | ## Sample commands for use with test data
61 | ```
62 | ./netgrph.py -nf all
63 | ./netgrph.py -nf all -o tree
64 | ./ngreport.py -dev ".*"
65 | ./ngreport.py -vrf "perim|default"
66 | ./netgrph.py abc4mdf
67 | ./netgrph.py abc4mdf -o json
68 | ./netgrph.py abc4mdf -o yaml
69 | ./netgrph.py -sp abc2sw1 xyz2sw1
70 | ./netgrph.py -sp abc.* xyz.* -o csv
71 | ./netgrph.py -rp 10.1.120.50 10.7.206.0/23
72 | ./netgrph.py 120
73 | ./netgrph.py 1246
74 | ./netgrph.py -fp 10.1.120.50 8.8.8.8
75 | ./netgrph.py -nlist test_group
76 | ./netgrph.py -nlist test_group -o tree
77 | ./netgrph.py -group ABC
78 | ./ngreport.py -vrf "perim|default"
79 | ./ngreport.py -vlans
80 | ```
81 |
82 | ## Sample Reporting (to be expanded)
83 | ./ngreport.py -vlans
84 |
85 |
86 | ### Production Install
87 |
88 | - I would add a netgrph user to your system and clone the repo under that user
89 | - If you have a Cisco network, see the datasources/ directory file for parsing information
90 | - At the top of the files, you can configure your config locations and extensions.
91 | - See the update_parse.py and update_csv_from_netdb.sh scripts to see how I gather my data from netdb and stored configs.
92 | - Consider installing NetDB for access to the Neighbor Discovery Data (I hope to find an alternative soon)
93 | - Consider using something like Rancid or [Oxidized](https://github.com/ytti/oxidized) for gathering configurations from your devices.
94 | - Create symlinks to netgrph.py, ngupdate.py and ngreport.py under /usr/local/bin/
95 | - Make sure all users can read your config file and consider installing it under /etc/netgrph.ini (preferred)
96 | - Consider changing your log location to /var/log/nglib.log and make sure all users have access to write to this file
97 | - Schedule cron jobs under the netgrph user to update regularly (Full NetDB integration example)
98 | ```
99 | 01,16,31,46 * * * * /home/yantisj/netgrph/prod/datasources/update_csv_from_netdb.sh
100 | 22 * * * * /home/yantisj/netgrph/prod/datasources/update_parse.sh
101 | 02,17,32,48 * * * * /home/yantisj/netgrph/prod/ngupdate.py -full
102 | 12 * * * * /home/yantisj/netgrph/prod/ngupdate.py -unetdb
103 | ```
104 | - Periodically clear node caches as they expire (adding -v will only show you what's expired)
105 | ```
106 | ngupdate --clearEdges --hours 12
107 | ngupdate --clearNodes --hours 12
108 | ```
109 |
110 | ## Adding firewalls and third-party devices
111 | - Examine the csv files in test/csv/ to understand the required datasources for importing third-party data
112 | - Examine the cyp/buildfw.cyp for understanding how to insert a L2 firewall between VRFs
113 | - Examine the cyp/sample-queries.cyp to start querying the Neo4j database directly for data
114 |
--------------------------------------------------------------------------------
/docs/Napalm.md:
--------------------------------------------------------------------------------
1 | # Napalm Support
2 |
3 | ## NetGrph
4 |
5 | ### Supported data
6 | * Routed Interfaces
7 | * LLDP Neighbors
8 | * Model, Version, Serial, Uptime
9 |
10 | ### Missing Data
11 |
12 | #### L3
13 | * VRF
14 | * Virtual gateways (HSRP/VRRP)
15 |
16 | #### L2
17 | * L2 VLAN List
18 | * VLAN Names
19 | * STP priority configuration
20 |
21 | #### Interfaces
22 | * Port-channel membership (match mac address?)
23 | * Trunked VLAN configuration (1-4096 etc)
24 | * Native VLAN ID
25 |
26 | ## NetDB Support
27 | * MAC Table
28 | * ARP Table
29 | * Interface Table
30 |
--------------------------------------------------------------------------------
/docs/VERSION:
--------------------------------------------------------------------------------
1 | v0.9.1
2 |
--------------------------------------------------------------------------------
/docs/index.md:
--------------------------------------------------------------------------------
1 | # Welcome to the NetGrph Documentation
2 |
3 | NetGrph is an abstract network model for automation, providing a unified network
4 | view across diverse network components in order to manage them as a single
5 | system via the [Neo4j Graph Database](http://neo4j.com). This enables you to
6 | navigate your traditional LAN/WAN and/or mixed SDN networks as interconnected
7 | nodes and relationships in software, all modeled via the network configurations
8 | as they exist rather than secondary sources.
9 |
10 | NetGrph can perform universal L2/L3/L4 path traversals, providing context for
11 | each layer along the path. It also serves as a VLAN and CIDR database, showing
12 | how everything is related. It scales well on even the largest networks,
13 | allowing sub-second queries across thousands of network devices. This enables
14 | the mapping of complex network relationships for discovery and automation.
15 |
16 | Data from queries can be returned as CSV, JSON, YAML, or Ascii tree-art. Network
17 | Visualizations can be created by querying the Neo4j webapp as shown below. The
18 | data model should translate for use with tools such as D3.js, vis.js or Graphwiz
19 | via both the native Neo4j API as well as NetGrph's tree data structure.
20 |
21 | All data is accessible via an API, and the lightweight netgrph client can be
22 | distributed to multiple machines.
23 |
24 | ## Graph Data Model Example: Vlan 110 -> 200 Traversal
25 | 
26 |
27 |
28 | [L3 SVIs: Yellow] [L2 VLANs: Green] [Switches/Routers: Blue]
29 |
30 |
31 |
32 | ## Features
33 |
34 | * Universal Layer2 - Layer4 pathfinding between any two network devices (Full L2 path completion requires NetDB)
35 | * Path Queries can return a single path, or all ECMPs
36 | * L3 Network Database of all networks (Automated, VRF aware, and searchable)
37 | * Search for networks via CIDR or VRF/Role based filters (eg. perim:printers|thinclient, all printers and thin clients in the perim VRF)
38 | * VLAN Inventory of all VLAN instances across the network, segmented by switch domain
39 | * Maps L2 VLAN bridges across switch domains, and calculates local/global STP roots
40 | * Maps L2 paths between devices (regexs supported, eg. dc.* -> dc.* for all datacenter links)
41 | * Reports the configured VLANs and actual VLANs existing on each link for all L2 paths
42 | * Secure REST API Server and Client
43 | * High performance, low latency queries (All queries are sub-second)
44 | * Easily extendable to support mixed-vendor environnments
45 | * Ansible playbooks for a five minute install on Ubuntu 14.04/16.04
46 |
47 | ## Planned Features
48 | * Rewrite API to be more consistent
49 | * Develop a Web frontend
50 | * Integrate with Napalm for datasources
51 |
52 | ## Requirements
53 |
54 | * Python 3.4+ (recommend running via virtualenv)
55 | * Ubuntu or MacOS (should run on any Python compatible platform, but I only support these)
56 | * [Neo4j Graph Database](https://neo4j.com) and Java8
57 | * For Cisco devices, must provide stored configurations (See [Rancid](http://www.shrubbery.net/rancid/) / [Oxidized](https://github.com/ytti/oxidized))
58 | * Requires CDP/LLDP Discovery Data via [NetDB](http://netdbtracking.sourceforge.net) or [NetCrawl](https://github.com/ytti/netcrawl)
59 | * Third-party network devices need to be parsed into the [NetGrph CSV format](test/csv/)
60 | * Please send me any parsers you create
61 |
62 | ## Supported Devices
63 |
64 | NetGrph is designed to support all network vendors, but currently only has
65 | scrapers available for Cisco IOS and NXOS devices. Adding support for other
66 | devices should be relatively trivial.
67 |
68 | ## About NetGrph
69 |
70 | NetGrph can perform universal L2/L3/L4 path traversals, providing context for
71 | each layer along the path. It also serves as a VLAN and CIDR database, showing
72 | how everything is related. It scales well on even the largest networks,
73 | allowing sub-second queries across thousands of network devices. This enables
74 | the mapping of complex network relationships for discovery and automation.
75 |
76 | Data from queries can be returned as CSV, JSON, YAML, or Ascii tree-art. Network
77 | Visualizations can be created by querying the Neo4j webapp as shown below. The
78 | data model should translate for use with tools such as D3.js, vis.js or Graphwiz
79 | via both the native Neo4j API as well as NetGrph's tree data structure.
80 |
81 | All data is accessible via an API, and the lightweight netgrph client can be
82 | distributed to multiple machines.
83 |
84 | Find out more [About NetGrph](About.md)
85 |
86 | # Documentation Guide
87 |
88 | ### Installation
89 |
90 | See the [Installation Guide](INSTALL.md)
91 |
92 | Ansible Instructions are located [Here](playbooks/README.md)
93 |
94 | ### Tutorials
95 |
96 | See the [Tutorials Documentation](Tutorials.md)
97 |
98 | ### Command Line Interface
99 |
100 | See the [Command Line Interface Documentation](CLI.md)
101 |
102 | ### REST API Documentation
103 |
104 | See the [REST API Documentation](API.md)
105 |
106 |
--------------------------------------------------------------------------------
/docs/netgrph-logrotate:
--------------------------------------------------------------------------------
1 | /var/log/nglib.log {
2 | daily
3 | missingok
4 | rotate 7
5 | notifempty
6 | create 666 netdb www-data
7 | }
8 |
--------------------------------------------------------------------------------
/docs/netgrph.ini:
--------------------------------------------------------------------------------
1 | # NetGraph INI File
2 |
3 | [Global]
4 | enabled = 1
5 |
6 | [nglib]
7 | dbuser = neo4j
8 | dbpass = your_passwd
9 | dbhost = localhost
10 |
11 | # Make sure this is writable by your user
12 | logfile = nglib.log
13 |
14 | # Default VLAN Range
15 | vrange = 1-1999
16 |
17 | # debuglib, infolib, info, warning, critical
18 | loglevel = info
19 | #loglevel = debuglib
20 |
21 | # Command to run to search FW Logs
22 | logcmd = splunksearch -m 15 -fs -cs
23 |
24 | # URL to provide for log search (primarily splunk for now)
25 | logurl = https://splunk.yourdomain.com/en-US/app/search/search?q=search%%20
26 |
27 | # Optional NetDB Credentials
28 | [netdb]
29 | #host = localhost
30 | #user = netdbadmin
31 | #pass = netdbpass
32 |
33 | [topology]
34 | max_distance = 100
35 | seeds = core1,core2
36 |
37 | # Prioritize certain switches in NEI_EQ decisions, name usually takes precedence
38 | nei_priority = switch2,switch1
39 |
40 | # Exclude certain switches from distance calculations (py regex)
41 | #dist_exclude = (voip|bldg10\-mwavesw1)
42 | dist_exclude = (noexclusion)
43 |
44 | # Rewrite the default VRF on these devices
45 | [default_vrf]
46 | #dc-perim = perim
47 |
48 | # API Client Config
49 | #[api]
50 | #user = testuser
51 | #pass = testpass
52 | #url = http://localhost:4096/netgrph/api/v1.0/
53 | #verify = 1
54 |
55 | # API Server Configuration
56 | [apisrv]
57 | app_name = netgrph
58 | port = 4096
59 |
60 | # Debug mode logs to stdout and enable Flask debugging
61 | # Set to 0 for production!
62 | debug = 0
63 | logfile = api.log
64 | database = ../api.db
65 |
66 | # SSL Certificate and Key, required for enabling HTTPS
67 | # HTTPS must not be enabled to listen on 0.0.0.0
68 | https = 0
69 | ssl_crt = ssl.crt
70 | ssl_key = ssl.key
71 |
72 | [ngfiles]
73 | vrfs = ./test/csv/vrfs.csv
74 | devices = ./test/csv/devices.csv
75 | device_info = ./test/csv/devinfo.csv
76 | neighbors = ./test/csv/nd.csv
77 | networks = ./test/csv/allnets.csv
78 | vlans = ./test/csv/allvlans.csv
79 | supernets = ./test/csv/supernets.csv
80 | firewalls = ./test/csv/firewalls.csv
81 | links = ./test/csv/links.csv
82 |
83 | # ASA FW Directory
84 | [ngfw]
85 | fwdir = /tftpboot/asafw/
86 |
87 |
88 | ## NetAlert Section
89 |
90 | [NetAlert]
91 | from = from@testnet.com
92 | subject = New Network Alert
93 | mailServer = yourmailhost.com
94 |
95 | [NetAlertGroups]
96 | test_group = emailfor@group.com
97 |
98 |
99 | # Filters Networks based on vrf:role (from supernets)
100 | # Examples:
101 | # net = all (all networks)
102 | # internal_access = default:none|access-private|access-wan (default VRF only with no supernet role or roles in the new access layer supernets 10.177 etc)
103 | # security = default:none|access-private|access-wan pci:all (security wants all access layer networks plus all PCI networks)
104 | # fwutil = fwutil:all (Wants all fwutil networks)
105 | # com = default:printer (COM wants only printer networks)
106 |
107 | [NetAlertFilter]
108 | test_group = all
109 |
--------------------------------------------------------------------------------
/docs/playbooks/README.md:
--------------------------------------------------------------------------------
1 | ## Ansible NetGrph Playbooks
2 |
3 | These playbooks will install NetGrph via Ansible on Ubuntu 14.04 or 16.04 for
4 | you. It will not set your database password or configure your netgrph.ini file
5 | for now, so you need to do that manually. Once the scripts run, browse to
6 | http://machine:7474 and setup your password. Set that same password in
7 | /home/netgrph/docs/netgrph.ini, and run the /home/netgrph/test/first_import.sh.
8 |
9 | #### Setting up Ansible to run via localhost
10 |
11 | ```
12 | sudo su -
13 | apt-get install ansible
14 | echo '[netgrph]' >> /etc/ansible/hosts
15 | echo localhost ansible_connection=local >> /etc/ansible/hosts
16 | exit
17 | ```
18 |
19 | #### Run playbooks against localhost (installs under netgrph user)
20 |
21 | ```
22 | git clone https://github.com/yantisj/netgrph.git /tmp/netgrph/
23 | cd /tmp/netgrph/docs/playbooks/
24 | ansible-playbook netgrph.yml --ask-sudo-pass
25 | ```
26 |
27 | #### Test the install
28 |
29 | ```
30 | sudo su - netgrph
31 | cd netgrph
32 | ```
33 |
34 | ##### Use an insecure DB password for testing (not recommended)
35 | ```
36 | ./test/set_neo4j_password.sh
37 | ```
38 |
39 | ##### Test a database import
40 | ```
41 | ./test/first_import.sh
42 | ```
43 |
44 | - See the INSTALL.md file for test queries and production install information
45 |
--------------------------------------------------------------------------------
/docs/playbooks/netgrph.yml:
--------------------------------------------------------------------------------
1 | # Configure your nguser account under vars: below
2 | # Add your server to [netgrph] in Ansible Hosts
3 | # Add your sudo user to [netgrp:vars]
4 | # ansible_ssh_user = yantisj
5 |
6 | - hosts: netgrph
7 | become: yes
8 | become_user: root
9 | vars:
10 | nguser: netgrph
11 | option_install_java: false
12 | option_allow_remote_connections: true
13 | neo4j_edition: neo4j
14 |
15 | neo4j_properties:
16 | - {regexp: "^allow_store_upgrade.*", line: "allow_store_upgrade=true"}
17 |
18 | tasks:
19 | - name: Update Apt Packages
20 | include: tasks/apt-update.yml
21 |
22 | - name: Apt Dependencies
23 | include: tasks/apt-packages.yml
24 |
25 | - name: Install Java8
26 | include: tasks/java8.yml
27 |
28 | - name: Install Neo4j
29 | include: tasks/neo4j.yml
30 |
31 | - name: Add netgrph user
32 | include: tasks/netgrph-user.yml
33 |
34 | - name: Clone NetGrph Repo
35 | include: tasks/clonerepo.yml
36 |
37 | - name: Install PIP Requirements
38 | include: tasks/ngpip.yml
39 |
40 | - name: NetGrph Setup
41 | include: tasks/setup.yml
42 |
43 |
--------------------------------------------------------------------------------
/docs/playbooks/netgrph/netgrph.yml:
--------------------------------------------------------------------------------
1 | # Configure your nguser account under vars: below
2 | # Add your server to [netgrph] in Ansible Hosts
3 | # Add your sudo user to [netgrp:vars]
4 | # ansible_ssh_user = yantisj
5 |
6 | - hosts: netgrph
7 | become: yes
8 | become_user: root
9 | vars:
10 | nguser: yantisj
11 | option_install_java: false
12 | option_allow_remote_connections: true
13 | neo4j_edition: neo4j
14 |
15 | neo4j_properties:
16 | - {regexp: "^allow_store_upgrade.*", line: "allow_store_upgrade=true"}
17 |
18 | tasks:
19 | - name: Update Apt Packages
20 | include: tasks/apt-update.yml
21 |
22 | - name: Apt Dependencies
23 | include: tasks/apt-packages.yml
24 |
25 | - name: Install Java8
26 | include: tasks/java8.yml
27 |
28 | - name: Install Neo4j
29 | include: tasks/neo4j.yml
30 |
31 | - name: Clone NetGrph Repo
32 | become: yes
33 | become_user: "{{nguser}}"
34 | include: tasks/clonerepo.yml
35 |
36 | - name: Install PIP Requirements
37 | include: tasks/ngpip.yml
38 |
39 |
--------------------------------------------------------------------------------
/docs/playbooks/netgrph/tasks/apt-packages.yml:
--------------------------------------------------------------------------------
1 | # file: update.yml
2 | - name: Install Apt Packages
3 | apt: name={{item}} state=present
4 | with_items:
5 | - python3-pip
6 | - python3-pytest
7 | - python3-logilab-common
8 | - emacs24-nox
9 |
--------------------------------------------------------------------------------
/docs/playbooks/netgrph/tasks/apt-update.yml:
--------------------------------------------------------------------------------
1 | ---
2 | - name: Update Packages
3 | apt: update_cache=yes upgrade=dist dpkg_options='force-confold,force-confdef'
4 |
--------------------------------------------------------------------------------
/docs/playbooks/netgrph/tasks/clonerepo.yml:
--------------------------------------------------------------------------------
1 | ---
2 | - name: Clone NetGrph Repo
3 | git: repo=https://github.com/yantisj/netgrph dest=/home/{{nguser}}/netgrph
4 |
5 |
6 |
--------------------------------------------------------------------------------
/docs/playbooks/netgrph/tasks/java8.yml:
--------------------------------------------------------------------------------
1 | ---
2 | - name: add repo for java 8
3 | apt_repository: repo='ppa:webupd8team/java' state=present
4 |
5 | - name: set licence selected
6 | shell: /bin/echo debconf shared/accepted-oracle-license-v1-1 select true | /usr/bin/debconf-set-selections
7 | sudo: yes
8 |
9 | - name: set licence seen
10 | shell: /bin/echo debconf shared/accepted-oracle-license-v1-1 seen true | /usr/bin/debconf-set-selections
11 | sudo: yes
12 |
13 | - name: install java 8
14 | apt: name=oracle-java8-installer state=latest update-cache=yes force=yes
15 | sudo: yes
16 |
--------------------------------------------------------------------------------
/docs/playbooks/netgrph/tasks/neo4j.yml:
--------------------------------------------------------------------------------
1 | # According to http://debian.neo4j.org/
2 | ---
3 | - name: NEO4J | Add key for Neo4j repo
4 | apt_key: url=http://debian.neo4j.org/neotechnology.gpg.key state=present
5 | become: yes
6 |
7 | - name: NEO4J | Add Neo4j repo to sources list
8 | apt_repository: repo='deb http://debian.neo4j.org/repo stable/' state=present
9 | become: yes
10 |
11 | - name: NEO4J | Install Neo4j packages
12 | apt: pkg={{ item }} state=installed update_cache=yes force=yes
13 | become: yes
14 | with_items:
15 | - "{{neo4j_edition}}"
16 |
17 | # http://www.delimited.io/blog/2014/1/15/getting-started-with-neo4j-on-ubuntu-server
18 |
19 | - name: NEO4J | Update /etc/security/limits.conf file (1/2)
20 | lineinfile: dest=/etc/security/limits.conf
21 | insertbefore='# End of file'
22 | line='neo4j soft nofile 40000'
23 | state=present
24 | become: yes
25 |
26 | - name: NEO4J | Update /etc/security/limits.conf file (2/2)
27 | lineinfile: dest=/etc/security/limits.conf
28 | insertbefore='# End of file'
29 | line='neo4j hard nofile 40000'
30 | state=present
31 | become: yes
32 |
33 | - name: NEO4J | Update /etc/pam.d/su file
34 | lineinfile: dest=/etc/pam.d/su
35 | regexp="^session required pam_limits.so"
36 | insertafter='^# session required pam_limits.so'
37 | line="session required pam_limits.so"
38 | state=present
39 | become: yes
40 |
41 | # - name: NEO4J | Update /etc/neo4j/neo4j-server.properties to enable remote users to login to neo4j
42 | # lineinfile: dest=/etc/neo4j/neo4j-server.properties
43 | # regexp="^org.neo4j.server.webserver.address=0.0.0.0"
44 | # insertafter='^#org.neo4j.server.webserver.address=0.0.0.0'
45 | # line="org.neo4j.server.webserver.address=0.0.0.0"
46 | # state=present
47 |
48 | - name: NEO4J | Update /etc/neo4j/neo4j.conf to enable remote users to login to neo4j
49 | lineinfile: dest=/etc/neo4j/neo4j.conf
50 | regexp="^dbms.connector.http.address=0.0.0.0:7474"
51 | insertafter='^#dbms.connector.http.address=0.0.0.0:7474'
52 | line="dbms.connector.http.address=0.0.0.0:7474"
53 | state=present
54 | when: option_allow_remote_connections
55 | become: yes
56 |
57 | - name: NEO4J | Restart Neo4j
58 | service: name=neo4j state=restarted
59 | become: yes
60 |
--------------------------------------------------------------------------------
/docs/playbooks/netgrph/tasks/ngpip.yml:
--------------------------------------------------------------------------------
1 | ---
2 | - name: Install PIP Requirements via PIP3
3 | pip: requirements=/home/{{nguser}}/netgrph/requirements.txt executable=pip3
4 |
5 |
--------------------------------------------------------------------------------
/docs/playbooks/tasks/apt-packages.yml:
--------------------------------------------------------------------------------
1 | # file: update.yml
2 | - name: APT | Install Apt Packages
3 | apt: name={{item}} state=present
4 | with_items:
5 | - python3-pip
6 | - python3-pytest
7 | - python3-logilab-common
8 | - emacs24-nox
9 |
--------------------------------------------------------------------------------
/docs/playbooks/tasks/apt-update.yml:
--------------------------------------------------------------------------------
1 | ---
2 | - name: APT | Update Packages
3 | apt: update_cache=yes upgrade=dist dpkg_options='force-confold,force-confdef'
4 |
--------------------------------------------------------------------------------
/docs/playbooks/tasks/clonerepo.yml:
--------------------------------------------------------------------------------
1 | ---
2 | - name: GIT | Clone NetGrph Repo
3 | git: repo=https://github.com/yantisj/netgrph dest=/home/{{nguser}}/netgrph
4 | become: yes
5 | become_user: "{{nguser}}"
6 |
7 |
--------------------------------------------------------------------------------
/docs/playbooks/tasks/java8.yml:
--------------------------------------------------------------------------------
1 | ---
2 | - name: JAVA | Add repo for java 8
3 | apt_repository: repo='ppa:webupd8team/java' mode=664 state=present
4 |
5 | - name: JAVA | Set licence selected
6 | shell: /bin/echo debconf shared/accepted-oracle-license-v1-1 select true | /usr/bin/debconf-set-selections
7 |
8 | - name: JAVA | Set licence seen
9 | shell: /bin/echo debconf shared/accepted-oracle-license-v1-1 seen true | /usr/bin/debconf-set-selections
10 |
11 | - name: JAVA | Install java 8
12 | apt: name=oracle-java8-installer state=latest update-cache=yes force=yes
13 |
--------------------------------------------------------------------------------
/docs/playbooks/tasks/neo4j.yml:
--------------------------------------------------------------------------------
1 | # According to http://debian.neo4j.org/
2 | ---
3 | - name: NEO4J | Add key for Neo4j repo
4 | apt_key: url=http://debian.neo4j.org/neotechnology.gpg.key state=present
5 | become: yes
6 |
7 | - name: NEO4J | Add Neo4j repo to sources list
8 | apt_repository: repo='deb http://debian.neo4j.org/repo stable/' mode=664 state=present
9 | become: yes
10 |
11 | - name: NEO4J | Install Neo4j packages
12 | apt: pkg={{ item }} state=installed update_cache=yes force=yes
13 | become: yes
14 | with_items:
15 | - "{{neo4j_edition}}"
16 |
17 | # http://www.delimited.io/blog/2014/1/15/getting-started-with-neo4j-on-ubuntu-server
18 |
19 | - name: NEO4J | Update /etc/security/limits.conf file (1/2)
20 | lineinfile: dest=/etc/security/limits.conf
21 | insertbefore='# End of file'
22 | line='neo4j soft nofile 40000'
23 | state=present
24 | become: yes
25 |
26 | - name: NEO4J | Update /etc/security/limits.conf file (2/2)
27 | lineinfile: dest=/etc/security/limits.conf
28 | insertbefore='# End of file'
29 | line='neo4j hard nofile 40000'
30 | state=present
31 | become: yes
32 |
33 | - name: NEO4J | Update /etc/pam.d/su file
34 | lineinfile: dest=/etc/pam.d/su
35 | regexp="^session required pam_limits.so"
36 | insertafter='^# session required pam_limits.so'
37 | line="session required pam_limits.so"
38 | state=present
39 | become: yes
40 |
41 | # - name: NEO4J | Update /etc/neo4j/neo4j-server.properties to enable remote users to login to neo4j
42 | # lineinfile: dest=/etc/neo4j/neo4j-server.properties
43 | # regexp="^org.neo4j.server.webserver.address=0.0.0.0"
44 | # insertafter='^#org.neo4j.server.webserver.address=0.0.0.0'
45 | # line="org.neo4j.server.webserver.address=0.0.0.0"
46 | # state=present
47 |
48 | - name: NEO4J | Update /etc/neo4j/neo4j.conf to enable remote users to login to neo4j
49 | lineinfile: dest=/etc/neo4j/neo4j.conf
50 | regexp="^dbms.connector.http.address=0.0.0.0:7474"
51 | insertafter='^#dbms.connector.http.address=0.0.0.0:7474'
52 | line="dbms.connector.http.address=0.0.0.0:7474"
53 | state=present
54 | when: option_allow_remote_connections
55 | become: yes
56 |
57 | - name: NEO4J | Restart Neo4j
58 | service: name=neo4j state=restarted
59 | become: yes
60 |
--------------------------------------------------------------------------------
/docs/playbooks/tasks/netgrph-user.yml:
--------------------------------------------------------------------------------
1 | ---
2 | - group: name=netgrph state=present
3 | - user: name=netgrph comment="NetGrph User" group=netgrph generate_ssh_key=yes ssh_key_bits=2048 ssh_key_file=.ssh/id_rsa
4 |
--------------------------------------------------------------------------------
/docs/playbooks/tasks/ngpip.yml:
--------------------------------------------------------------------------------
1 | ---
2 | - name: PIP | Install NetGrph Requirements via PIP3
3 | pip: requirements=/home/{{nguser}}/netgrph/requirements.txt executable=pip3
4 |
5 |
--------------------------------------------------------------------------------
/docs/playbooks/tasks/setup.yml:
--------------------------------------------------------------------------------
1 | ---
2 |
3 | - name: NetGrph | Symlinking executables
4 | file: src=/home/{{ nguser }}/netgrph/netgrph.py dest=/usr/local/bin/netgrph state=link
5 | - file: src=/home/{{ nguser }}/netgrph/ngupdate.py dest=/usr/local/bin/ngupdate state=link
6 | - file: src=/home/{{ nguser }}/netgrph/ngreport.py dest=/usr/local/bin/ngreport state=link
7 |
8 |
--------------------------------------------------------------------------------
/docs/playbooks/test.yml:
--------------------------------------------------------------------------------
1 | # Configure your nguser account under vars: below
2 | # Add your server to [netgrph] in Ansible Hosts
3 | # Add your sudo user to [netgrp:vars]
4 | # ansible_ssh_user = yantisj
5 |
6 | - hosts: netgrph
7 | become: yes
8 | become_user: root
9 | vars:
10 | nguser: netgrph
11 | option_install_java: false
12 | option_allow_remote_connections: true
13 | neo4j_edition: neo4j
14 |
15 | neo4j_properties:
16 | - {regexp: "^allow_store_upgrade.*", line: "allow_store_upgrade=true"}
17 |
18 | tasks:
19 | # - name: Update Apt Packages
20 | # include: tasks/apt-update.yml
21 |
22 | # - name: Apt Dependencies
23 | # include: tasks/apt-packages.yml
24 |
25 | # - name: Install Java8
26 | # include: tasks/java8.yml
27 |
28 | # - name: Install Neo4j
29 | # include: tasks/neo4j.yml
30 |
31 | # - name: Add netgrph user
32 | # include: tasks/netgrph-user.yml
33 |
34 | # - name: Clone NetGrph Repo
35 | # become: yes
36 | # become_user: "{{nguser}}"
37 | # include: tasks/clonerepo.yml
38 |
39 | # - name: Install PIP Requirements
40 | # include: tasks/ngpip.yml
41 |
42 | - name: NetGrph Setup
43 | include: tasks/setup.yml
44 |
--------------------------------------------------------------------------------
/docs/workflow.txt:
--------------------------------------------------------------------------------
1 |
2 | edit docs/VERSION
3 | git tag v0.7.x
4 | git merge --no-ff --log origin/dev
5 | git push origin master
6 |
7 |
--------------------------------------------------------------------------------
/extra/apisrv-nginx:
--------------------------------------------------------------------------------
1 | server {
2 | listen 9000 ssl;
3 | server_name 0.0.0.0;
4 | ssl_certificate /home/netgrph/netgrph/newhal.crt;
5 | ssl_certificate_key /home/netgrph/netgrph/newhal.key;
6 |
7 | location / {
8 | include uwsgi_params;
9 | uwsgi_pass unix:/home/netgrph/netgrph/apisrv.sock;
10 | }
11 | }
12 |
--------------------------------------------------------------------------------
/extra/nsj-sample.ini:
--------------------------------------------------------------------------------
1 | # NSJ Config
2 | [nsj]
3 | username = switch-user
4 | password = password
5 | runlog = nsj.log
6 | outlog = job.log
7 | loglevel = warning
8 | threads = 50
9 | debug = 0
10 |
11 | # API Client Config
12 | [api]
13 | user = testuser
14 | pass = testpass
15 | url = http://localhost:4096/netgrph/api/v1.1/
16 | verify = 0
17 |
--------------------------------------------------------------------------------
/extra/wsgi-api.ini:
--------------------------------------------------------------------------------
1 | [uwsgi]
2 | module = wsgi-api
3 |
4 | master = true
5 | processes = 5
6 |
7 | socket = apisrv.sock
8 | chmod-socket = 666
9 | vacuum = true
10 |
11 | die-on-term = true
12 |
--------------------------------------------------------------------------------
/mkdocs.yml:
--------------------------------------------------------------------------------
1 | site_name: NetGrph
2 | pages:
3 | - Home: index.md
4 | - About NetGrph: About.md
5 | - Installing NetGrph:
6 | - Installation: INSTALL.md
7 | - Ansible Install: playbooks/README.md
8 | - Using NetGrph:
9 | - Usage Overview: Tutorials.md
10 | - Command Line Interface: CLI.md
11 | - Topology and Pathfinding: PathSample.md
12 | - REST API Reference: API.md
13 |
--------------------------------------------------------------------------------
/nglib/__init__.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python
2 | #
3 | #
4 | # Copyright (c) 2016 "Jonathan Yantis"
5 | #
6 | # This file is a part of NetGrph.
7 | #
8 | # This program is free software: you can redistribute it and/or modify
9 | # it under the terms of the GNU Affero General Public License, version 3,
10 | # as published by the Free Software Foundation.
11 | #
12 | # This program is distributed in the hope that it will be useful,
13 | # but WITHOUT ANY WARRANTY; without even the implied warranty of
14 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
15 | # GNU Affero General Public License for more details.
16 | #
17 | # You should have received a copy of the GNU Affero General Public License
18 | # along with this program. If not, see .
19 | #
20 | # As a special exception, the copyright holders give permission to link the
21 | # code of portions of this program with the OpenSSL library under certain
22 | # conditions as described in each individual source file and distribute
23 | # linked combinations including the program with the OpenSSL library. You
24 | # must comply with the GNU Affero General Public License in all respects
25 | # for all of the code used other than as permitted herein. If you modify
26 | # file(s) with this exception, you may extend this exception to your
27 | # version of the file(s), but you are not obligated to do so. If you do not
28 | # wish to do so, delete this exception statement from your version. If you
29 | # delete this exception statement from all source files in the program,
30 | # then also delete it in the license file.
31 | #
32 | #
33 | """
34 | NGDB Main Library
35 | - Main Library functions and variables used by modules
36 | - Call module.init_nglib(config) to initialize
37 | - Sets logging levels, loads config file
38 | """
39 | import csv
40 | import re
41 | import sys
42 | import os
43 | import pwd
44 | import datetime
45 | import configparser
46 | import logging
47 |
48 | try:
49 | from neo4j.v1 import TRUST_ON_FIRST_USE, TRUST_SIGNED_CERTIFICATES, SSL_AVAILABLE
50 | from neo4j.v1.exceptions import CypherError, ProtocolError
51 | from neo4j.v1 import GraphDatabase, basic_auth
52 | from py2neo import Node, Relationship, Graph
53 | except ImportError:
54 | pass
55 |
56 | logger = logging.getLogger(__name__)
57 |
58 | # Global variables (all library global variables go here)
59 | verbose = 0
60 | config = None
61 |
62 | # Save user for library
63 | user = pwd.getpwuid(os.getuid())[0]
64 |
65 | # DB Sessions accessed globally
66 | bolt_ses = None
67 | py2neo_ses = None
68 |
69 | # Topology Variables
70 | max_distance = 100
71 | dev_seeds = None
72 |
73 | # NetDB Enabled
74 | use_netdb = False
75 |
76 | def get_bolt_db():
77 | """Return Bolt Session"""
78 |
79 | # DB Credentials
80 | dbuser = config['nglib']['dbuser']
81 | dbpass = config['nglib']['dbpass']
82 | dbhost = config['nglib']['dbhost']
83 |
84 | return get_db_client(dbhost, dbuser, dbpass, bolt=True)
85 |
86 | def get_py2neo_db():
87 | """Return Bolt Session"""
88 |
89 | # DB Credentials
90 | dbuser = config['nglib']['dbuser']
91 | dbpass = config['nglib']['dbpass']
92 | dbhost = config['nglib']['dbhost']
93 |
94 | return get_db_client(dbhost, dbuser, dbpass, bolt=False)
95 |
96 | def get_db_client(dbhost, dbuser, dbpass, bolt=False):
97 | """Return a Neo4j DB session. bolt=True uses bolt driver"""
98 |
99 | if verbose > 4:
100 | print("DB Creds", dbhost, dbuser, dbpass)
101 |
102 | if bolt:
103 | bolt_url = "bolt://" + dbhost
104 | auth_token = basic_auth(dbuser, dbpass)
105 | try:
106 | driver = GraphDatabase.driver(bolt_url, auth=auth_token, max_pool_size=5)
107 | bolt_session = driver.session()
108 | return bolt_session
109 | except Exception as e:
110 | print("Database connection/authentication error:", e)
111 | sys.exit(1)
112 | else:
113 | login = "http://{0}:{1}@{2}:7474/db/data/".format(dbuser, dbpass, dbhost)
114 | py2neo_session = Graph(login)
115 | return py2neo_session
116 |
117 | def drop_database():
118 | """Drop all database data (requires a full import to restore)"""
119 |
120 | logger.warning('Dropping Database')
121 |
122 | bolt_ses.run("MATCH ()-[e]-() DELETE e")
123 | bolt_ses.run("MATCH (n) DELETE n")
124 |
125 |
126 | def import_cypher(fileName):
127 | """Import MATCH/CREATE/MERGE Cypher Commands Directly"""
128 | f = open(fileName)
129 |
130 | for line in f:
131 | if re.search(r'^(MATCH|CREATE|MERGE)', line):
132 | line = line.strip()
133 | logger.debug(line)
134 | results = bolt_ses.run(line)
135 |
136 | test = results.consume()
137 | if test:
138 | logger.info('Executed ' + test.statement)
139 |
140 | def get_time(hours=None):
141 | """Get current time, optionally time shifted by hours"""
142 |
143 | if hours:
144 | timeShifted = datetime.datetime.now() - datetime.timedelta(hours=hours)
145 | return str(timeShifted)
146 |
147 | time = str(datetime.datetime.now())
148 | return time
149 |
150 |
151 | def getEntry(l, pos=0):
152 | """Returns first entry in a list, or at position pos=x"""
153 |
154 | return l[pos]
155 |
156 |
157 | def importCSVasDict(fileName):
158 | """Imports a file with DictReader"""
159 |
160 | f = open(fileName)
161 | db = csv.DictReader(f)
162 | return db
163 |
164 |
165 | def importCSVasList(fileName):
166 | """Imports CSV File as a list"""
167 |
168 | f = open(fileName)
169 | flist = csv.DictReader(f)
170 |
171 | return flist
172 |
173 |
174 | # Initialize Configuration
175 | def init_nglib(configFile, initdb=True):
176 | """Initializes Library based on config file
177 |
178 | - Sets global variables in library for use with other modules
179 | - Configures debugging and sets up logging
180 | - Modifies py2neo and bolt library levels
181 |
182 | """
183 |
184 | # Global variables for use throughout library
185 | global config
186 | global max_distance
187 | global dev_seeds
188 | global bolt_ses
189 | global py2neo_ses
190 | global use_netdb
191 |
192 | if verbose > 1:
193 | print("Config File", configFile)
194 |
195 | config = configparser.ConfigParser()
196 | config.read(configFile)
197 |
198 | # Initialize Logging
199 | init_logging()
200 |
201 | # Tries to Loads NetDB Variables
202 | try:
203 | use_netdb = config['netdb']['host']
204 | except KeyError:
205 | pass
206 | if use_netdb:
207 | use_netdb = True
208 |
209 | if initdb:
210 | # DB Credentials
211 | dbuser = config['nglib']['dbuser']
212 | dbpass = config['nglib']['dbpass']
213 | dbhost = config['nglib']['dbhost']
214 |
215 | # Login to DB for parent Variables
216 | if verbose > 1:
217 | logger.info("Connecting to Neo4j: %s %s", dbhost, dbuser)
218 | bolt_ses = get_db_client(dbhost, dbuser, dbpass, bolt=True)
219 | py2neo_ses = get_db_client(dbhost, dbuser, dbpass)
220 |
221 | # Topology
222 | max_distance = int(config['topology']['max_distance'])
223 | dev_seeds = config['topology']['seeds']
224 |
225 | logger.debug("Initialized Configuration Successfully")
226 |
227 |
228 | def init_logging():
229 | '''Initialize Logging'''
230 | # Default loglevel
231 | level = logging.INFO
232 |
233 | logfile = config['nglib']['logfile']
234 | loglevel = config['nglib']['loglevel']
235 |
236 | if loglevel == 'info':
237 | level = logging.INFO
238 | if loglevel == 'debug':
239 | level = logging.DEBUG
240 | if loglevel == 'debuglib':
241 | level = logging.DEBUG
242 | if loglevel == 'warning':
243 | level = logging.WARNING
244 | if loglevel == 'critical':
245 | level = logging.CRITICAL
246 |
247 | # Debug logging
248 | if verbose > 2:
249 | loglevel = 'debug'
250 | level = logging.DEBUG
251 | elif verbose > 1:
252 | loglevel = 'debuglib'
253 | level = logging.DEBUG
254 |
255 | # Verbose Logs to stdout
256 | if verbose:
257 | logging.basicConfig(format='%(asctime)s %(name)s:%(levelname)s: %(message)s',
258 | stream=sys.stdout, level=level)
259 | logger.info("Enabled STDOUT Logging")
260 |
261 | # Log to File
262 | else:
263 | logging.basicConfig(format='%(asctime)s %(name)s:%(levelname)s: %(message)s',
264 | filename=logfile, level=level)
265 |
266 | # Silence Cypher Logging
267 | if loglevel == 'info' or loglevel == 'debuglib':
268 | logging.getLogger("py2neo.cypher").setLevel(logging.WARNING)
269 | logging.getLogger("py2neo.batch").setLevel(logging.WARNING)
270 | logging.getLogger("httpstream").setLevel(logging.WARNING)
271 | logging.getLogger("neo4j.bolt").setLevel(logging.WARNING)
272 |
--------------------------------------------------------------------------------
/nglib/alerts.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python
2 | #
3 | # Copyright (c) 2016 "Jonathan Yantis"
4 | #
5 | # This file is a part of NetGrph.
6 | #
7 | # This program is free software: you can redistribute it and/or modify
8 | # it under the terms of the GNU Affero General Public License, version 3,
9 | # as published by the Free Software Foundation.
10 | #
11 | # This program is distributed in the hope that it will be useful,
12 | # but WITHOUT ANY WARRANTY; without even the implied warranty of
13 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
14 | # GNU Affero General Public License for more details.
15 | #
16 | # You should have received a copy of the GNU Affero General Public License
17 | # along with this program. If not, see .
18 | #
19 | # As a special exception, the copyright holders give permission to link the
20 | # code of portions of this program with the OpenSSL library under certain
21 | # conditions as described in each individual source file and distribute
22 | # linked combinations including the program with the OpenSSL library. You
23 | # must comply with the GNU Affero General Public License in all respects
24 | # for all of the code used other than as permitted herein. If you modify
25 | # file(s) with this exception, you may extend this exception to your
26 | # version of the file(s), but you are not obligated to do so. If you do not
27 | # wish to do so, delete this exception statement from your version. If you
28 | # delete this exception statement from all source files in the program,
29 | # then also delete it in the license file.
30 |
31 | """Process Alerts for different network events"""
32 |
33 | import configparser
34 | import smtplib
35 | import logging
36 | from email.mime.text import MIMEText
37 | #from email.mime.multipart import MIMEMultipart
38 | import nglib
39 |
40 | verbose = 0
41 | logger = logging.getLogger(__name__)
42 |
43 | # Generate Alerts on new networks
44 | def gen_new_network_alerts():
45 | """Generate New Network Alerts"""
46 |
47 | gAlert = dict()
48 | newNets = []
49 |
50 | # Find any new networks
51 | results = nglib.py2neo_ses.cypher.execute(
52 | 'MATCH(n:NewNetwork) return n.vrfcidr AS vrfcidr')
53 |
54 | ncount = len(results)
55 |
56 | if ncount > 0:
57 | logger.info("Found %s New Networks to Alert On", ncount)
58 |
59 | # Load groups in to dictionary to add list of networks to
60 | loadGroups(gAlert)
61 |
62 | # Append all networks to a list
63 | for network in results:
64 | #print(network.vrfcidr)
65 | newNets.append(network.vrfcidr)
66 |
67 | # Filter Nets for each group and add to their list to alert
68 | loadNetAlerts(gAlert, newNets)
69 |
70 | # Send Alerts to each group
71 | for g in gAlert.keys():
72 | if verbose > 1:
73 | print("\n\nALERTS for {0}:\n".format(g))
74 | acount = len(gAlert[g])
75 | logger.info("Sending %s New Network Alerts to %s", acount, g)
76 |
77 | if not verbose:
78 | sendEmailAlert(g, gAlert[g])
79 |
80 | # Deleting NewNetwork Objects
81 | if not verbose:
82 | results = nglib.py2neo_ses.cypher.execute(
83 | 'MATCH(n:NewNetwork) delete n')
84 |
85 | def gen_new_vlan_alerts():
86 | """ Send new VLAN Alerts to all groups that receive 'all' alerts """
87 |
88 | newVlans = []
89 | groups = []
90 |
91 | # Find any new networks
92 | results = nglib.py2neo_ses.cypher.execute(
93 | 'MATCH(v:NewVLAN) return v.name AS name')
94 |
95 | ncount = len(results)
96 |
97 | if ncount > 0:
98 | logger.info("Found %s New Vlans to Alert On", ncount)
99 |
100 | #
101 | for group in nglib.config['NetAlertGroups']:
102 | if nglib.config['NetAlertFilter'][group] == 'all':
103 | groups.append(group)
104 |
105 | # Append all vlans to a list
106 | for vlan in results:
107 | newVlans.append(vlan.name)
108 |
109 | if nglib.verbose:
110 | print(groups)
111 | print(newVlans)
112 |
113 | for group in groups:
114 | sendEmailAlert(group, newVlans, vlan=True)
115 |
116 | # Deleting NewVLAN Objects
117 | if not verbose:
118 | results = nglib.py2neo_ses.cypher.execute(
119 | 'MATCH(n:NewVLAN) delete n')
120 |
121 | def loadGroups(gAlert):
122 | """Load all groups from config file into a dictionary of lists"""
123 |
124 | for group in nglib.config['NetAlertGroups']:
125 | gAlert[group] = []
126 |
127 |
128 | def loadNetAlerts(gAlert, newNets):
129 | """Add networks to each group after filtering attributes"""
130 |
131 | for net in newNets:
132 |
133 | netDict = nglib.query.net.get_net_props(net)
134 |
135 | if len(netDict.keys()):
136 | for group in gAlert.keys():
137 | if nglib.query.check_net_filter(netDict, group):
138 | logger.debug("Adding " + netDict['CIDR'] + " to alerts for " + group)
139 | gAlert[group].append(netDict)
140 |
141 |
142 | def sendEmailAlert(group, nList, vlan=False):
143 | """Send group Network Alert from matched list"""
144 |
145 | if nglib.verbose:
146 | print("Emailing {0} New Network List Alert:\n {1}".format(group, str(nList)))
147 |
148 | emailAddress = nglib.config['NetAlertGroups'][group]
149 | fromAddress = nglib.config['NetAlert']['from']
150 | mailServer = nglib.config['NetAlert']['mailServer']
151 | subject = nglib.config['NetAlert']['subject']
152 |
153 | if vlan:
154 | subject = nglib.config['NetAlert']['vlansubject']
155 |
156 | nTable = ""
157 |
158 | for n in nList:
159 | nTable = nTable + "\n" + str(n)
160 |
161 | nTable = "
".join(nTable.split("\n"))
162 |
163 | contents = '' + nTable + ''
164 |
165 | msg = MIMEText(contents, 'html')
166 | msg['Subject'] = subject
167 | msg['From'] = fromAddress
168 | msg['To'] = emailAddress
169 |
170 | s = smtplib.SMTP(mailServer)
171 | s.sendmail(fromAddress, [emailAddress], msg.as_string())
172 | s.quit()
173 |
174 | if nglib.verbose:
175 | print(emailAddress, fromAddress, subject, contents)
176 |
177 | # END
178 |
--------------------------------------------------------------------------------
/nglib/cache_update.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python
2 | #
3 | # Copyright (c) 2016 "Jonathan Yantis"
4 | #
5 | # This file is a part of NetGrph.
6 | #
7 | # This program is free software: you can redistribute it and/or modify
8 | # it under the terms of the GNU Affero General Public License, version 3,
9 | # as published by the Free Software Foundation.
10 | #
11 | # This program is distributed in the hope that it will be useful,
12 | # but WITHOUT ANY WARRANTY; without even the implied warranty of
13 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
14 | # GNU Affero General Public License for more details.
15 | #
16 | # You should have received a copy of the GNU Affero General Public License
17 | # along with this program. If not, see .
18 | #
19 | # As a special exception, the copyright holders give permission to link the
20 | # code of portions of this program with the OpenSSL library under certain
21 | # conditions as described in each individual source file and distribute
22 | # linked combinations including the program with the OpenSSL library. You
23 | # must comply with the GNU Affero General Public License in all respects
24 | # for all of the code used other than as permitted herein. If you modify
25 | # file(s) with this exception, you may extend this exception to your
26 | # version of the file(s), but you are not obligated to do so. If you do not
27 | # wish to do so, delete this exception statement from your version. If you
28 | # delete this exception statement from all source files in the program,
29 | # then also delete it in the license file.
30 |
31 | """NetGrph Cache Management"""
32 |
33 | import logging
34 | from nglib.query.nNode import getRelationship, getLabel, getJSONProperties
35 | import nglib
36 |
37 | logger = logging.getLogger(__name__)
38 |
39 |
40 | def clear_edges(hours):
41 | """
42 | Clear Expired Edges
43 |
44 | Notes: nglib.verbose returns edges to delete but does not delete
45 | """
46 |
47 | logger.info("Clearing Edges older than " + str(hours) + " hours")
48 |
49 | # Time shifted datetime
50 | age = nglib.get_time(hours=hours)
51 |
52 | edges = nglib.py2neo_ses.cypher.execute(
53 | 'MATCH ()-[e]->() WHERE e.time < {age} RETURN e',
54 | age=age)
55 |
56 | if len(edges) > 0:
57 | for e in edges:
58 | neighbors = getRelationship(e.e)
59 | logger.info("Expired Edge: " + neighbors)
60 |
61 | count = nglib.py2neo_ses.cypher.execute(
62 | 'MATCH ()-[e]->() WHERE e.time < {age} RETURN count(e) as count',
63 | age=age)
64 |
65 | if nglib.verbose:
66 | logger.info("Expired Edges: " + str(count[0].count))
67 | else:
68 | logger.info("Deleting Edges: " + str(count[0].count))
69 | nglib.py2neo_ses.cypher.execute(
70 | 'MATCH ()-[e]->() WHERE e.time < {age} DELETE e',
71 | age=age)
72 |
73 |
74 | def clear_nodes(hours):
75 | """
76 | Clear Expired Nodes
77 |
78 | Notes: verbose returns nodes to delete but does not delete
79 | """
80 |
81 | logger.info("Finding Nodes to Clear older than " + str(hours) + " hours")
82 |
83 | # Time shifted datetime
84 | age = nglib.get_time(hours=hours)
85 |
86 | nodes = nglib.py2neo_ses.cypher.execute(
87 | 'MATCH (n) WHERE n.time < {age} RETURN n',
88 | age=age)
89 |
90 | if len(nodes) > 0:
91 |
92 | for r in nodes:
93 | label = getLabel(r.n)
94 | pj = getJSONProperties(r.n)
95 | logger.info("Expired Node: " + label + pj['name'])
96 |
97 | count = nglib.py2neo_ses.cypher.execute(
98 | 'MATCH (n) WHERE n.time < {age} RETURN count(n) as count',
99 | age=age)
100 |
101 | logger.info("Expired Nodes: " + str(count[0].count))
102 |
103 | if not nglib.verbose:
104 | logger.info("Deleting Nodes: " + str(count[0].count))
105 |
106 | nglib.py2neo_ses.cypher.execute(
107 | 'MATCH (n)-[e]-() WHERE n.time < {age} DELETE e',
108 | age=age)
109 |
110 | nglib.py2neo_ses.cypher.execute(
111 | 'MATCH (n) WHERE n.time < {age} DELETE n',
112 | age=age)
113 |
114 |
115 | def swap_quotes(myString):
116 | """Swap Quote Types for JSON from Neo4j"""
117 | myString = myString.replace("'", '"')
118 | return myString
119 |
--------------------------------------------------------------------------------
/nglib/exceptions.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python
2 | #
3 | #
4 | # Copyright (c) 2016 "Jonathan Yantis"
5 | #
6 | # This file is a part of NetGrph.
7 | #
8 | # This program is free software: you can redistribute it and/or modify
9 | # it under the terms of the GNU Affero General Public License, version 3,
10 | # as published by the Free Software Foundation.
11 | #
12 | # This program is distributed in the hope that it will be useful,
13 | # but WITHOUT ANY WARRANTY; without even the implied warranty of
14 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
15 | # GNU Affero General Public License for more details.
16 | #
17 | # You should have received a copy of the GNU Affero General Public License
18 | # along with this program. If not, see .
19 | #
20 | # As a special exception, the copyright holders give permission to link the
21 | # code of portions of this program with the OpenSSL library under certain
22 | # conditions as described in each individual source file and distribute
23 | # linked combinations including the program with the OpenSSL library. You
24 | # must comply with the GNU Affero General Public License in all respects
25 | # for all of the code used other than as permitted herein. If you modify
26 | # file(s) with this exception, you may extend this exception to your
27 | # version of the file(s), but you are not obligated to do so. If you do not
28 | # wish to do so, delete this exception statement from your version. If you
29 | # delete this exception statement from all source files in the program,
30 | # then also delete it in the license file.
31 | """
32 | NetGrph Exceptions
33 | """
34 |
35 | class Error(Exception):
36 | """Base class for exceptions in this module."""
37 | pass
38 |
39 | class OutputError(Error):
40 | """Exception raised for errors in the output."""
41 |
42 | def __init__(self, expression, message):
43 | self.expression = expression
44 | self.message = message
45 |
46 | class ResultError(Error):
47 | """Exception raised for Result Errors"""
48 |
49 | def __init__(self, expression, message):
50 | self.expression = expression
51 | self.message = message
52 |
--------------------------------------------------------------------------------
/nglib/fw_update.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python
2 | #
3 | # Copyright (c) 2016 "Jonathan Yantis"
4 | #
5 | # This file is a part of NetGrph.
6 | #
7 | # This program is free software: you can redistribute it and/or modify
8 | # it under the terms of the GNU Affero General Public License, version 3,
9 | # as published by the Free Software Foundation.
10 | #
11 | # This program is distributed in the hope that it will be useful,
12 | # but WITHOUT ANY WARRANTY; without even the implied warranty of
13 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
14 | # GNU Affero General Public License for more details.
15 | #
16 | # You should have received a copy of the GNU Affero General Public License
17 | # along with this program. If not, see .
18 | #
19 | # As a special exception, the copyright holders give permission to link the
20 | # code of portions of this program with the OpenSSL library under certain
21 | # conditions as described in each individual source file and distribute
22 | # linked combinations including the program with the OpenSSL library. You
23 | # must comply with the GNU Affero General Public License in all respects
24 | # for all of the code used other than as permitted herein. If you modify
25 | # file(s) with this exception, you may extend this exception to your
26 | # version of the file(s), but you are not obligated to do so. If you do not
27 | # wish to do so, delete this exception statement from your version. If you
28 | # delete this exception statement from all source files in the program,
29 | # then also delete it in the license file.
30 | #
31 | """
32 | NetGrph Firewall Import Routines
33 | """
34 | import logging
35 | import nglib
36 |
37 | logger = logging.getLogger(__name__)
38 |
39 | def import_fw(fileName):
40 | """Import FW CSV File"""
41 |
42 | logger.info("Importing Firewalls from " + fileName)
43 |
44 | fwdb = nglib.importCSVasDict(fileName)
45 |
46 | # Import FW Ints to DB
47 | import_fw_ints(fwdb)
48 |
49 |
50 | def import_fw_ints(fwdb):
51 | """Import Firewall Interfaces from FW File"""
52 |
53 | time = nglib.get_time()
54 |
55 | # Iterate through all firewall ints
56 | for fwint in fwdb:
57 |
58 | # DB Values
59 | name = fwint['Name']
60 | hostname = fwint['Hostname']
61 | #IP = fwint['IP']
62 | vlanInt = fwint['Interface']
63 | desc = fwint['Description']
64 | seclevel = fwint['Security-Level']
65 | logIndex = fwint['Log-Index']
66 |
67 | #print(name, vlanInt, desc, seclevel, IP, hostname, logIndex)
68 |
69 | vlan = vlanInt.replace('Vlan', '')
70 |
71 | # Search for existing Firewall
72 | results = nglib.py2neo_ses.cypher.execute(
73 | 'MATCH (fw:Switch:Router:FW {name:{name}}) RETURN fw',
74 | name=name)
75 |
76 | # Insert new Firewall
77 | if len(results) == 0:
78 | logger.info("Creating New Firewall: " + name)
79 |
80 | results = nglib.py2neo_ses.cypher.execute(
81 | 'CREATE (fw:Switch:Router:FW {name:{name}, hostname:{hostname}, '
82 | + 'logIndex:{logIndex}, time:{time}}) RETURN fw',
83 | name=name, hostname=hostname, logIndex=logIndex, time=time)
84 |
85 | # Update FW
86 | else:
87 | logger.debug("Updating Firewall: " + name)
88 |
89 | nglib.py2neo_ses.cypher.execute(
90 | 'MATCH (fw:Switch:Router:FW {name:{name}}) SET fw += '
91 | + '{hostname:{hostname}, logIndex:{logIndex}, time:{time}} RETURN fw',
92 | name=name, hostname=hostname, logIndex=logIndex, time=time)
93 |
94 | # Search for existing Vlan
95 | results = nglib.py2neo_ses.cypher.execute(
96 | 'MATCH (n:Network {vid:{vlan}})-[e:ROUTED_FW]->'
97 | + '(fw:Switch:Router:FW {name:{name}}) '
98 | + 'RETURN e',
99 | vlan=vlan, name=name)
100 |
101 | if len(results) == 0:
102 | logger.info("Creating New ROUTED_FW Link: %s --> %s", vlan, name)
103 |
104 | results = nglib.py2neo_ses.cypher.execute(
105 | 'MATCH (n:Network {vid:{vlan}}), (fw:Switch:Router:FW {name:{name}})'
106 | + 'CREATE (n)-[e:ROUTED_FW '
107 | + '{desc:{desc}, seclevel:{seclevel}, time:{time}}]->(fw)',
108 | vlan=vlan, name=name, desc=desc, seclevel=seclevel, time=time)
109 |
110 | # Update Firewall Interface
111 | else:
112 | logger.debug("Updating ROUTED_FW: %s --> %s", vlan, name)
113 |
114 | results = nglib.py2neo_ses.cypher.execute(
115 | 'MATCH (n:Network {vid:{vlan}})-[e:ROUTED_FW]->'
116 | + '(fw:Switch:Router:FW {name:{name}})'
117 | + 'SET e += {desc:{desc}, seclevel:{seclevel}, time:{time}} '
118 | + 'RETURN e',
119 | vlan=vlan, name=name, desc=desc, seclevel=seclevel, time=time)
120 |
--------------------------------------------------------------------------------
/nglib/netdb/__init__.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python3
2 | #
3 | # Copyright (c) 2016 "Jonathan Yantis"
4 | #
5 | # This file is a part of NetGrph.
6 | #
7 | # This program is free software: you can redistribute it and/or modify
8 | # it under the terms of the GNU Affero General Public License, version 3,
9 | # as published by the Free Software Foundation.
10 | #
11 | # This program is distributed in the hope that it will be useful,
12 | # but WITHOUT ANY WARRANTY; without even the implied warranty of
13 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
14 | # GNU Affero General Public License for more details.
15 | #
16 | # You should have received a copy of the GNU Affero General Public License
17 | # along with this program. If not, see .
18 | #
19 | # As a special exception, the copyright holders give permission to link the
20 | # code of portions of this program with the OpenSSL library under certain
21 | # conditions as described in each individual source file and distribute
22 | # linked combinations including the program with the OpenSSL library. You
23 | # must comply with the GNU Affero General Public License in all respects
24 | # for all of the code used other than as permitted herein. If you modify
25 | # file(s) with this exception, you may extend this exception to your
26 | # version of the file(s), but you are not obligated to do so. If you do not
27 | # wish to do so, delete this exception statement from your version. If you
28 | # delete this exception statement from all source files in the program,
29 | # then also delete it in the license file.
30 | #
31 | """
32 | NetGrph NetDB Interface
33 | """
34 |
35 | import datetime
36 | import locale
37 | import csv
38 | import re
39 | import os
40 | import sys
41 | import logging
42 | import pymysql
43 | import nglib
44 |
45 |
46 | logger = logging.getLogger(__name__)
47 |
48 | # DB connection is global
49 | netdb_ses = None
50 |
51 | def connect_netdb():
52 | """Update the netdb_ses handle"""
53 |
54 | global netdb_ses
55 | netdbhost = None
56 | netdbuser = None
57 | netdbpasswd = None
58 |
59 | try:
60 | netdbhost = nglib.config['netdb']['host']
61 | netdbuser = nglib.config['netdb']['user']
62 | netdbpasswd = nglib.config['netdb']['pass']
63 |
64 | except KeyError:
65 | print("Error: NetDB Database Credentials no Configured", file=sys.stderr)
66 | raise
67 |
68 | if not netdbuser:
69 | raise Exception("NetDB Credentials not Configured")
70 |
71 | netdb_ses = pymysql.connect(user=netdbuser, password=netdbpasswd,
72 | host=netdbhost,
73 | database="netdb")
74 |
75 | return netdb_ses
76 |
77 |
78 | def get_lastseen(hours=168):
79 | """Get lastseen string for MySQL"""
80 |
81 | lastseen = datetime.datetime.now() - datetime.timedelta(hours=hours)
82 | lastseen = lastseen.isoformat()
83 | return lastseen
84 |
85 |
86 | def get_mac_and_port_counts(switch, vlan):
87 | """Get the number of mac addresses and ports on a VLAN for a switch"""
88 |
89 | connect_netdb()
90 |
91 | lastseen = get_lastseen()
92 |
93 | cursor = netdb_ses.cursor(pymysql.cursors.DictCursor)
94 |
95 | cursor.execute("SELECT count(vlan) AS pcount FROM switchstatus "
96 | + "WHERE switch = '{}' and vlan='{}'".format(switch, vlan))
97 |
98 | pc = cursor.fetchall()
99 | pcount = pc[0]['pcount']
100 |
101 | # MACs on switch with lastseen
102 | mac_query = "SELECT count(mac) AS mcount FROM switchports "
103 | mac_query += "WHERE switch = '{}' AND s_vlan='{}' ".format(switch, vlan)
104 | mac_query += "AND lastseen > '{}'".format(lastseen)
105 | cursor.execute(mac_query)
106 | mc = cursor.fetchall()
107 | mcount = mc[0]['mcount']
108 |
109 | return(pcount, mcount)
110 |
--------------------------------------------------------------------------------
/nglib/netdb/ip.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python3
2 | #
3 | # Copyright (c) 2016 "Jonathan Yantis"
4 | #
5 | # This file is a part of NetGrph.
6 | #
7 | # This program is free software: you can redistribute it and/or modify
8 | # it under the terms of the GNU Affero General Public License, version 3,
9 | # as published by the Free Software Foundation.
10 | #
11 | # This program is distributed in the hope that it will be useful,
12 | # but WITHOUT ANY WARRANTY; without even the implied warranty of
13 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
14 | # GNU Affero General Public License for more details.
15 | #
16 | # You should have received a copy of the GNU Affero General Public License
17 | # along with this program. If not, see .
18 | #
19 | # As a special exception, the copyright holders give permission to link the
20 | # code of portions of this program with the OpenSSL library under certain
21 | # conditions as described in each individual source file and distribute
22 | # linked combinations including the program with the OpenSSL library. You
23 | # must comply with the GNU Affero General Public License in all respects
24 | # for all of the code used other than as permitted herein. If you modify
25 | # file(s) with this exception, you may extend this exception to your
26 | # version of the file(s), but you are not obligated to do so. If you do not
27 | # wish to do so, delete this exception statement from your version. If you
28 | # delete this exception statement from all source files in the program,
29 | # then also delete it in the license file.
30 | #
31 | """
32 | NetGrph NetDB IP Interface
33 | """
34 | import re
35 | import socket
36 | import logging
37 | import pymysql
38 | import functools
39 | import nglib.netdb
40 | import nglib.ngtree
41 |
42 |
43 | logger = logging.getLogger(__name__)
44 |
45 |
46 | @functools.lru_cache(maxsize=2)
47 | def get_netdb_ip(ip, hours=720):
48 | """Get All Details for an IP"""
49 |
50 | # Check for non-ip, try DNS
51 | if not re.search(r'^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$', ip):
52 | try:
53 | ip = socket.gethostbyname(ip)
54 | except socket.gaierror:
55 | raise Exception("Hostname Lookup Failure on: " + ip)
56 |
57 | netdb_ses = nglib.netdb.connect_netdb()
58 |
59 | lastseen = nglib.netdb.get_lastseen(hours)
60 |
61 | cursor = netdb_ses.cursor(pymysql.cursors.DictCursor)
62 | # cursor = netdb_ses.cursor(dictionary=True)
63 |
64 | cursor.execute("SELECT * FROM superarp "
65 | + "WHERE ip = '{}' AND lastseen > '{}'".format(ip, lastseen))
66 |
67 | pc = cursor.fetchall()
68 |
69 | multi_entry = False
70 | pngtree = nglib.ngtree.get_ngtree("IPs", tree_type="NetDB")
71 |
72 | if not len(pc):
73 | return None
74 |
75 | # If multiple entries, primary returns under ngtree, the rest return
76 | # as historical nested data
77 | if len(pc) > 1:
78 | multi_entry = True
79 |
80 | latest = None
81 |
82 | # Gather details from DB in ngtree structure
83 | for en in pc:
84 | ngtree = nglib.ngtree.get_ngtree("IP", tree_type="NetDB")
85 | ngtree['Name'] = ip
86 |
87 | ngtree['firstSeen'] = str(en['firstseen'])
88 | ngtree['lastSeen'] = str(en['lastseen'])
89 | ngtree['MAC'] = en['mac']
90 | ngtree['FQDN'] = en['name']
91 | ngtree['vendor'] = en['vendor']
92 | ngtree['Switch'] = en['lastswitch']
93 | ngtree['SwitchPort'] = en['lastport']
94 | ngtree['UserID'] = en['userID']
95 | ngtree['VLAN'] = en['vlan']
96 |
97 |
98 | if not latest:
99 | latest = ngtree
100 | elif latest['lastSeen'] < ngtree['lastSeen']:
101 | latest = ngtree
102 |
103 | nglib.ngtree.add_child_ngtree(pngtree, ngtree)
104 |
105 | # Return as parent if single entry
106 | if not multi_entry:
107 | return ngtree
108 |
109 | # Return latest IP as root and nest others under a new tree
110 | multitree = nglib.ngtree.get_ngtree("Historical IPs", tree_type="NetDB")
111 | for en in pngtree:
112 | if '_child' in en and pngtree[en] != latest:
113 | nglib.ngtree.add_child_ngtree(multitree, pngtree[en])
114 | nglib.ngtree.add_child_ngtree(latest, multitree)
115 | return latest
116 |
117 | def arp(router, hours=1):
118 | """Pull the ARP Table on a router from NetDB, use router='%' for everything"""
119 |
120 | netdb_ses = nglib.netdb.connect_netdb()
121 |
122 | lastseen = nglib.netdb.get_lastseen(hours)
123 |
124 | cursor = netdb_ses.cursor(pymysql.cursors.DictCursor)
125 | # cursor = netdb_ses.cursor(dictionary=True)
126 |
127 | cursor.execute("SELECT * FROM superarp "
128 | + "WHERE router LIKE '{}' AND lastseen > '{}'".format(router, lastseen))
129 |
130 | pc = cursor.fetchall()
131 |
132 | pngtree = nglib.ngtree.get_ngtree("ARP-Table", tree_type="NetDB")
133 |
134 | for en in pc:
135 | ngtree = nglib.ngtree.get_ngtree("ARP Entry", tree_type="NetDB")
136 |
137 | ngtree['firstSeen'] = str(en['firstseen'])
138 | ngtree['lastSeen'] = str(en['lastseen'])
139 | ngtree['ip'] = en['ip']
140 | ngtree['vrf'] = en['vrf']
141 | ngtree['mac'] = en['mac']
142 | ngtree['fqdn'] = en['name']
143 | ngtree['vendor'] = en['vendor']
144 | ngtree['switch'] = en['lastswitch']
145 | ngtree['port'] = en['lastport']
146 | ngtree['userid'] = en['userID']
147 | ngtree['vlan'] = en['vlan']
148 |
149 | nglib.ngtree.add_child_ngtree(pngtree, ngtree)
150 |
151 | return pngtree
152 |
--------------------------------------------------------------------------------
/nglib/netdb/switch.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python3
2 | #
3 | # Copyright (c) 2016 "Jonathan Yantis"
4 | #
5 | # This file is a part of NetGrph.
6 | #
7 | # This program is free software: you can redistribute it and/or modify
8 | # it under the terms of the GNU Affero General Public License, version 3,
9 | # as published by the Free Software Foundation.
10 | #
11 | # This program is distributed in the hope that it will be useful,
12 | # but WITHOUT ANY WARRANTY; without even the implied warranty of
13 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
14 | # GNU Affero General Public License for more details.
15 | #
16 | # You should have received a copy of the GNU Affero General Public License
17 | # along with this program. If not, see .
18 | #
19 | # As a special exception, the copyright holders give permission to link the
20 | # code of portions of this program with the OpenSSL library under certain
21 | # conditions as described in each individual source file and distribute
22 | # linked combinations including the program with the OpenSSL library. You
23 | # must comply with the GNU Affero General Public License in all respects
24 | # for all of the code used other than as permitted herein. If you modify
25 | # file(s) with this exception, you may extend this exception to your
26 | # version of the file(s), but you are not obligated to do so. If you do not
27 | # wish to do so, delete this exception statement from your version. If you
28 | # delete this exception statement from all source files in the program,
29 | # then also delete it in the license file.
30 | #
31 | """
32 | NetGrph NetDB Switch Interface
33 | """
34 | import re
35 | import socket
36 | import logging
37 | import pymysql
38 | import nglib.netdb
39 | import nglib.ngtree
40 |
41 | logger = logging.getLogger(__name__)
42 |
43 |
44 | def get_switch(switch, port='%', hours=720, trunc=True):
45 | """Get All Interface Details for Switch"""
46 |
47 | # Truncated Values to retrieve
48 | tr = ['switch', 'port', 'status', 'description', 'vlan', 'speed', 'duplex']
49 |
50 | netdb_ses = nglib.netdb.connect_netdb()
51 | lastseen = nglib.netdb.get_lastseen(hours)
52 |
53 | cursor = netdb_ses.cursor(pymysql.cursors.DictCursor)
54 |
55 | # cursor.execute("SELECT * FROM superswitch "
56 | # + "WHERE ip = '{}' AND lastseen > '{}'".format(switch, lastseen))
57 |
58 | cursor.execute(
59 | "SELECT switchstatus.switch,switchstatus.port,switchstatus.vlan,switchstatus.status,"
60 | + "switchstatus.speed,switchstatus.duplex,switchstatus.description,"
61 | + "switchstatus.p_uptime,switchstatus.p_minutes,switchstatus.lastup,"
62 | + "superswitch.mac,superswitch.ip,superswitch.s_ip,superswitch.s_name,"
63 | + "superswitch.name,superswitch.static,superswitch.mac_nd,"
64 | + "superswitch.vendor,superswitch.vrf,superswitch.router,"
65 | + "superswitch.uptime,superswitch.minutes,superswitch.firstseen,"
66 | + "superswitch.lastseen,nacreg.userID,nacreg.firstName,nacreg.lastName,"
67 | + "nacreg.role,nd.n_host,nd.n_ip,nd.n_desc,nd.n_model,nd.n_port,"
68 | + "nd.n_protocol,nd.n_lastseen,superswitch.s_speed,superswitch.s_ip,"
69 | + "superswitch.s_vlan "
70 | + "FROM switchstatus LEFT OUTER JOIN superswitch "
71 | + "ON switchstatus.switch = superswitch.switch "
72 | + "AND switchstatus.port = superswitch.port "
73 | + "AND superswitch.lastseen > '{}' ".format(lastseen)
74 | + "LEFT OUTER JOIN neighbor as nd "
75 | + "ON ( switchstatus.switch = nd.switch AND switchstatus.port = nd.port ) "
76 | + "LEFT OUTER JOIN nacreg ON nacreg.mac = superswitch.mac "
77 | + "WHERE (switchstatus.switch like '{}' AND switchstatus.port like '{}') ".format(switch, port)
78 | + "ORDER BY switchstatus.port"
79 | )
80 |
81 | pc = cursor.fetchall()
82 |
83 | pngtree = nglib.ngtree.get_ngtree(switch, tree_type="INTs")
84 |
85 | pseen = dict()
86 |
87 | # Gather details from DB in ngtree structure
88 | for en in pc:
89 | ngtree = nglib.ngtree.get_ngtree("INT", tree_type="INT")
90 |
91 | for e in en:
92 | if e in tr and en['port'] not in pseen:
93 | ngtree[e] = en[e]
94 |
95 | if en['port'] not in pseen:
96 | pseen[en['port']] = True
97 | ngtree['Name'] = ngtree['port']
98 | nglib.ngtree.add_child_ngtree(pngtree, ngtree)
99 |
100 | return pngtree
101 |
102 | def mac(switch, port='%', hours=1):
103 | """Pull the MAC Table on a switch from NetDB, use switch='%' for everything"""
104 |
105 | netdb_ses = nglib.netdb.connect_netdb()
106 |
107 | lastseen = nglib.netdb.get_lastseen(hours)
108 |
109 | cursor = netdb_ses.cursor(pymysql.cursors.DictCursor)
110 | # cursor = netdb_ses.cursor(dictionary=True)
111 |
112 | cursor.execute("SELECT * FROM superswitch "
113 | + "WHERE switch LIKE '{}' AND port LIKE '{}' ".format(switch, port)
114 | + "AND lastseen > '{}'".format(lastseen))
115 |
116 | pc = cursor.fetchall()
117 |
118 | pngtree = nglib.ngtree.get_ngtree("MAC-Table", tree_type="NetDB")
119 |
120 | for en in pc:
121 | ngtree = nglib.ngtree.get_ngtree("MAC", tree_type="NetDB")
122 |
123 | for f in en:
124 | ngtree[f] = str(en[f])
125 |
126 | nglib.ngtree.add_child_ngtree(pngtree, ngtree)
127 |
128 | return pngtree
129 |
130 | def count(switch, hours=1):
131 | """ Get the mac address count from a switch """
132 |
133 | netdb_ses = nglib.netdb.connect_netdb()
134 |
135 | lastseen = nglib.netdb.get_lastseen(hours)
136 |
137 | cursor = netdb_ses.cursor(pymysql.cursors.DictCursor)
138 |
139 | cursor.execute("SELECT count(mac) FROM superswitch "
140 | + "WHERE switch LIKE '{}' ".format(switch)
141 | + "AND lastseen > '{}'".format(lastseen))
142 |
143 | pc = cursor.fetchall()
144 |
145 | pngtree = nglib.ngtree.get_ngtree("MAC-Count", tree_type="NetDB")
146 | pngtree['switch'] = switch
147 |
148 | for en in pc:
149 | print(en)
150 | pngtree['mac_count'] = en['count(mac)']
151 |
152 | return pngtree
153 |
--------------------------------------------------------------------------------
/nglib/ngtree/__init__.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python
2 | #
3 | # Work with netgrph tree structures
4 | #
5 | # Copyright (c) 2016 "Jonathan Yantis"
6 | #
7 | # This file is a part of NetGrph.
8 | #
9 | # This program is free software: you can redistribute it and/or modify
10 | # it under the terms of the GNU Affero General Public License, version 3,
11 | # as published by the Free Software Foundation.
12 | #
13 | # This program is distributed in the hope that it will be useful,
14 | # but WITHOUT ANY WARRANTY; without even the implied warranty of
15 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
16 | # GNU Affero General Public License for more details.
17 | #
18 | # You should have received a copy of the GNU Affero General Public License
19 | # along with this program. If not, see .
20 | #
21 | # As a special exception, the copyright holders give permission to link the
22 | # code of portions of this program with the OpenSSL library under certain
23 | # conditions as described in each individual source file and distribute
24 | # linked combinations including the program with the OpenSSL library. You
25 | # must comply with the GNU Affero General Public License in all respects
26 | # for all of the code used other than as permitted herein. If you modify
27 | # file(s) with this exception, you may extend this exception to your
28 | # version of the file(s), but you are not obligated to do so. If you do not
29 | # wish to do so, delete this exception statement from your version. If you
30 | # delete this exception statement from all source files in the program,
31 | # then also delete it in the license file.
32 | #
33 | #
34 | """
35 | Work with netgrph tree structures
36 |
37 | ngtrees are nested dicts that convert to JSON/YAML
38 |
39 | - Getting an ngtree will create a new unnested ngtree to populate with properties
40 | - Adding a child ngtree will nest an ngtree under a parent
41 | - Adding a parent ngtree will add a special parent ngtree for when you want
42 | the perspective of a certain tree level, but want to add a parent object
43 |
44 | """
45 | import re
46 | import datetime
47 | import logging
48 | import nglib
49 | from . import export
50 | from . import upgrade
51 |
52 | logger = logging.getLogger(__name__)
53 |
54 | verbose = 0
55 |
56 |
57 | def get_ngtree(name, tree_type="VLAN"):
58 | """Initialize an NGTree"""
59 |
60 | ngtree = dict()
61 | ngtree['Name'] = name
62 | ngtree['_type'] = tree_type
63 | ngtree['_ccount'] = 0
64 | ngtree['data'] = []
65 |
66 | return ngtree
67 |
68 | def add_child_ngtree(ngtree, cngtree):
69 | """
70 | Nest a child ngtree under data list in ngtree
71 | """
72 |
73 | ngtree['_ccount'] = ngtree['_ccount'] + 1
74 | ngtree['data'].append(cngtree)
75 |
76 | def print_ngtree(ngtree, dtree, parent=False, depth=0, lasttree=False):
77 | """
78 | Recrusively print NGTrees using UTF-8 line drawing characters. If this
79 | causes terminal problems for you and you would prefer an ASCII only mode,
80 | contact me.
81 |
82 | Notes: This code is complicated, even to me. It should probably be rewritten,
83 | but basically, it nests multiple levels of ngtrees and their children in a
84 | pretty output format for use on the CLI.
85 |
86 | get_space_indent() gets output to prepend to lines to form the tree
87 | structure during output. It keeps track of where output is relative to the
88 | rest of the tree structure.
89 |
90 | The dtree dict keeps track of positions on tree to print continuations.
91 |
92 | The close out section needs to be better understood, but when closing out a
93 | child tree, you need to keep pipes from connecting sections below.
94 |
95 | """
96 |
97 | dtree = dtree.copy()
98 |
99 | # Get indentation spaces variable based on depth
100 | spaces, indent = get_space_indent(depth, dtree)
101 |
102 | # Get indentation spaces variable based on depth
103 | spaces, indent = get_space_indent(depth, dtree)
104 |
105 | # Print Header
106 |
107 | if depth == 0:
108 | indent = ""
109 |
110 | # Last tree terminates with └
111 | if lasttree:
112 | indent = indent.replace('├', '└')
113 |
114 | # Abbreviate certain types for shorter headers
115 | ngtype = ngtree['_type']
116 | if ngtype == "VLAN":
117 | ngtype = ""
118 | elif ngtree['_type'] == "Neighbor":
119 | ngtype = ""
120 | else:
121 | ngtype = " " + ngtype
122 | header = " "
123 | header = header.join([ngtype, ngtree['Name']])
124 |
125 | # Print section header -[header]
126 | if depth == 0:
127 | print("{:}┌─[{:} ]".format(indent, header))
128 | print("│")
129 | else:
130 | # Headonly for QPATH Results
131 | headonly = True
132 | for en in ngtree:
133 | #if not re.search(r'Name|_type|_ccount', en):
134 | if not re.search(r'Name|_type|_ccount|data', en) \
135 | or len(ngtree['data']):
136 |
137 | headonly = False
138 | break
139 | if headonly:
140 | print("{:}──[{:} ]".format(indent, header))
141 | else:
142 | print("{:}┬─[{:} ]".format(indent, header))
143 |
144 | # Store Children list from data
145 | clist = ngtree['data']
146 |
147 | # If there are no children, do not indent tree structure
148 | if len(clist) == 0:
149 | dtree.pop(depth, None)
150 |
151 | # Get indentation spaces variable based on current depth
152 | spaces, indent = get_space_indent(depth, dtree)
153 |
154 | # Filter tree of structural data (_ccount etc)
155 | ftree = filter_tree(ngtree)
156 | if nglib.verbose > 1:
157 | ftree = ngtree
158 |
159 | # Print all keys at current depth
160 | # Last one prints special if terminating section
161 | lastcount = len(ftree.keys())
162 | for key in sorted(ftree.keys()):
163 | lastcount = lastcount - 1
164 |
165 | # If there are children of current tree, then continue tree.
166 | # Otherwise terminate tree with └
167 | if lastcount or len(clist) > 0:
168 | print("{:}├── {:} : {:}".format(spaces, key, ftree[key]))
169 | else:
170 | print("{:}└── {:} : {:}".format(spaces, key, ftree[key]))
171 |
172 |
173 | # Close out a section with empty line for visual separation
174 | if len(clist) > 0:
175 | spaces = spaces + "│"
176 | print(spaces)
177 |
178 | # Print child trees with recursive call to this function
179 | while len(clist) > 0:
180 |
181 | ctree = clist.pop(0)
182 |
183 | # Continue printing with depth
184 | if len(clist) != 0:
185 | dtree[depth] = 1
186 |
187 | lasttree = False
188 | # End of indentation, un-indent
189 | if len(clist) == 0:
190 | dtree.pop(depth, None)
191 | lasttree = True
192 |
193 | spaces, indent = get_space_indent(depth, dtree)
194 |
195 | # Indent and print child tree recursively
196 | cdepth = depth + 4
197 | print_ngtree(ctree, dtree, depth=cdepth, lasttree=lasttree)
198 |
199 | # Ending section, un-indent
200 | if len(clist) == 0:
201 | dtree.pop(depth, None)
202 |
203 |
204 | def get_space_indent(depth, dtree):
205 | """Returns indentation and spacing strings for building ngtree output"""
206 |
207 | spaces = ""
208 | indent = ""
209 | count = 0
210 |
211 | while count < depth:
212 | count = count + 1
213 |
214 | if count - 1 in dtree.keys():
215 | #print("Found Dtree at " + str(count-1))
216 | spaces = spaces + "│"
217 | else:
218 | spaces = spaces + " "
219 | #iline = "-" + iline
220 | if count < depth - 4:
221 | indent = spaces + " "
222 |
223 | indent = indent + "├───"
224 |
225 | return spaces, indent
226 |
227 |
228 | def filter_tree(ngtree):
229 | '''Filter structural data'''
230 |
231 | keys = ngtree.keys()
232 | newtree = dict()
233 |
234 | for key in keys:
235 | if not re.search('(^_)|(^Name$)|(^data$)', key):
236 | newtree[key] = ngtree[key]
237 | #ngtree.pop(key)
238 |
239 | return newtree
240 |
--------------------------------------------------------------------------------
/nglib/ngtree/export.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python
2 | #
3 | # NetGrph Export Routines
4 | #
5 | # Copyright (c) 2016 "Jonathan Yantis"
6 | #
7 | # This file is a part of NetGrph.
8 | #
9 | # This program is free software: you can redistribute it and/or modify
10 | # it under the terms of the GNU Affero General Public License, version 3,
11 | # as published by the Free Software Foundation.
12 | #
13 | # This program is distributed in the hope that it will be useful,
14 | # but WITHOUT ANY WARRANTY; without even the implied warranty of
15 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
16 | # GNU Affero General Public License for more details.
17 | #
18 | # You should have received a copy of the GNU Affero General Public License
19 | # along with this program. If not, see .
20 | #
21 | # As a special exception, the copyright holders give permission to link the
22 | # code of portions of this program with the OpenSSL library under certain
23 | # conditions as described in each individual source file and distribute
24 | # linked combinations including the program with the OpenSSL library. You
25 | # must comply with the GNU Affero General Public License in all respects
26 | # for all of the code used other than as permitted herein. If you modify
27 | # file(s) with this exception, you may extend this exception to your
28 | # version of the file(s), but you are not obligated to do so. If you do not
29 | # wish to do so, delete this exception statement from your version. If you
30 | # delete this exception statement from all source files in the program,
31 | # then also delete it in the license file.
32 | #
33 | #
34 | """
35 | Helper functions to export ngtrees in the right format
36 | """
37 | import re
38 | import logging
39 | import json
40 | import yaml
41 | import csv
42 | import sys
43 | import nglib.ngtree
44 |
45 | verbose = 0
46 | logger = logging.getLogger(__name__)
47 |
48 | def exp_ngtree(ngtree, rtype):
49 | """Prints or Returns NGTree in Requested format"""
50 |
51 | if rtype == "TREE":
52 | nglib.ngtree.print_ngtree(ngtree, dtree=dict())
53 | elif rtype == 'QTREE':
54 | exp_qtree(ngtree)
55 | elif rtype == "CSV":
56 | exp_CSV(ngtree)
57 | elif rtype == "CSV2":
58 | exp_CSV(ngtree, level=2)
59 | elif rtype == "JSON":
60 | exp_JSON(ngtree)
61 | elif rtype == "YAML":
62 | exp_YAML(ngtree)
63 | else:
64 | return ngtree
65 |
66 | def exp_JSON(ngtree):
67 | """Prints an ngtree as JSON"""
68 |
69 | print(get_JSON(ngtree))
70 |
71 | def get_JSON(ngtree):
72 | """Returns an ngtree as JSON Object"""
73 |
74 | jtree = json.dumps(ngtree, indent=2, sort_keys=True)
75 | return jtree
76 |
77 | # Export as YAML
78 | def exp_YAML(ngtree):
79 | """Prints an ngtree as YAML"""
80 | print(get_YAML(ngtree))
81 |
82 | def get_YAML(ngtree):
83 | """Returns an ngtree as YAML Object"""
84 | ytree = yaml.dump(ngtree, Dumper=yaml.Dumper, default_flow_style=False)
85 | return ytree
86 |
87 | def exp_qtree(ngtree):
88 | """Prints an ngtree with headers only"""
89 |
90 | stree = strip_ngtree(ngtree)
91 | nglib.ngtree.print_ngtree(stree, dtree=dict())
92 |
93 | def cleanNGTree(ngtree):
94 | """Removes counts from output"""
95 |
96 | cleanND = ngtree.copy()
97 | cleanND.pop('_ccount', None)
98 | return cleanND
99 |
100 | def exp_CSV(ngtree, level=1):
101 | """Flatten NGTREE and dump as CSV, optional two levels deep"""
102 |
103 | fieldnames = []
104 | if level == 2:
105 | fieldnames.append('_ctype')
106 | fieldnames.append('CName')
107 |
108 | for child in ngtree['data']:
109 | for en in sorted(child.keys()):
110 | if isinstance(en, (int, str)) and en != 'data' \
111 | and en not in fieldnames:
112 | fieldnames.append(en)
113 |
114 | # Two levels deep
115 | if level == 2:
116 | for gchild in child['data']:
117 | for en in sorted(gchild.keys()):
118 | if isinstance(en, (int, str)) and en != 'data' \
119 | and en not in fieldnames:
120 | fieldnames.append(en)
121 |
122 | excsv = csv.DictWriter(sys.stdout, fieldnames=fieldnames)
123 | excsv.writeheader()
124 |
125 | for child in ngtree['data']:
126 | entry = dict()
127 | for en in sorted(child.keys()):
128 | if isinstance(en, (int, str)) and not re.search('data|Switches', en):
129 | entry[en] = child[en]
130 | if level == 2:
131 | for gchild in child['data']:
132 | centry = entry.copy()
133 | for en in sorted(gchild.keys()):
134 | if isinstance(en, (int, str)) and not re.search('data|Switches', en):
135 | if en == 'Name':
136 | centry['CName'] = gchild[en]
137 | elif en == '_type':
138 | centry['_ctype'] = gchild[en]
139 | else:
140 | centry[en] = gchild[en]
141 | excsv.writerow(centry)
142 |
143 | else:
144 | excsv.writerow(entry)
145 |
146 |
147 | def strip_ngtree(ngtree, top=True):
148 | """Strips everything but headers from ngtree"""
149 |
150 | newtree = nglib.ngtree.get_ngtree(ngtree['Name'], tree_type=ngtree['_type'])
151 |
152 | for en in ngtree['data']:
153 | newtree['data'].append(strip_ngtree(en, top=False))
154 | if top:
155 | for en in ngtree:
156 | if en != 'data':
157 | newtree[en] = ngtree[en]
158 |
159 | return newtree
160 |
--------------------------------------------------------------------------------
/nglib/ngtree/upgrade.py:
--------------------------------------------------------------------------------
1 | 'Upgrade older ngtrees to newer version'
2 | import logging
3 |
4 | logger = logging.getLogger(__name__)
5 |
6 | def upgrade_ngt_v2(ngt):
7 | 'Upgrade ngt structures to version 2 for the API'
8 |
9 | stack = list()
10 |
11 | # Add dictionary to stack
12 | stack.append(ngt)
13 |
14 | # upgrade keys on all dictionaries
15 | for tree in stack:
16 | # Copy NGT and traverse
17 | nt = tree.copy()
18 | for f in nt:
19 | # Upgrade dictionary key
20 | tree.pop(f)
21 | tree[_new_name(f)] = nt[f]
22 |
23 | # Found a nested dict, add to stack
24 | if isinstance(nt[f], dict):
25 | stack.append(nt[f])
26 |
27 | # Found a nested list
28 | elif isinstance(nt[f], list):
29 | for en in nt[f]:
30 | # nested dict in list, add to stack
31 | if isinstance(en, dict):
32 | stack.append(en)
33 |
34 | return ngt
35 |
36 |
37 | def _new_name(old):
38 | 'Get new name for fields (lowercase, replace spaces with _)'
39 |
40 | nmap = {
41 | 'StandbyRouter': 'standby_router',
42 | 'SecurityLevel': 'security_level',
43 | 'mgmtgroup': 'mgmt_group'
44 | }
45 |
46 | if old in nmap:
47 | return nmap[old]
48 |
49 | old = old.replace(' ', '_')
50 | old = old.lower()
51 |
52 | if old == 'data':
53 | old = 'xdata'
54 |
55 | return old
56 |
--------------------------------------------------------------------------------
/nglib/query/__init__.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python3
2 | #
3 | # Copyright (c) 2016 "Jonathan Yantis"
4 | #
5 | # This file is a part of NetGrph.
6 | #
7 | # This program is free software: you can redistribute it and/or modify
8 | # it under the terms of the GNU Affero General Public License, version 3,
9 | # as published by the Free Software Foundation.
10 | #
11 | # This program is distributed in the hope that it will be useful,
12 | # but WITHOUT ANY WARRANTY; without even the implied warranty of
13 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
14 | # GNU Affero General Public License for more details.
15 | #
16 | # You should have received a copy of the GNU Affero General Public License
17 | # along with this program. If not, see .
18 | #
19 | # As a special exception, the copyright holders give permission to link the
20 | # code of portions of this program with the OpenSSL library under certain
21 | # conditions as described in each individual source file and distribute
22 | # linked combinations including the program with the OpenSSL library. You
23 | # must comply with the GNU Affero General Public License in all respects
24 | # for all of the code used other than as permitted herein. If you modify
25 | # file(s) with this exception, you may extend this exception to your
26 | # version of the file(s), but you are not obligated to do so. If you do not
27 | # wish to do so, delete this exception statement from your version. If you
28 | # delete this exception statement from all source files in the program,
29 | # then also delete it in the license file.
30 | #
31 | """NetGrph Queries Library (Parent Module)"""
32 |
33 | import csv
34 | import re
35 | import os
36 | import sys
37 | import logging
38 | import functools
39 | import configparser
40 | import ipaddress
41 | import nglib
42 | import nglib.ngtree
43 | import nglib.query.vlan
44 | import nglib.query.dev
45 | import nglib.query.net
46 | import nglib.query.nNode
47 | import nglib.query.path
48 |
49 |
50 | logger = logging.getLogger(__name__)
51 |
52 | def exp_ngtree(ngtree, rtype):
53 | """Prints or Returns NGTree in Requested format"""
54 |
55 | if rtype == "TREE":
56 | nglib.ngtree.print_ngtree(ngtree, dtree=dict())
57 | elif rtype == 'QTREE':
58 | nglib.ngtree.export.exp_qtree(ngtree)
59 | elif rtype == "CSV":
60 | nglib.ngtree.export.exp_CSV(ngtree)
61 | elif rtype == "JSON":
62 | nglib.ngtree.export.exp_JSON(ngtree)
63 | elif rtype == "YAML":
64 | nglib.ngtree.export.exp_YAML(ngtree)
65 | else:
66 | return ngtree
67 |
68 | def display_mgmt_groups():
69 | """Print all Management Groups to the Screen"""
70 |
71 | mgmt = nglib.py2neo_ses.cypher.execute(
72 | 'MATCH (s:Switch) RETURN DISTINCT(s.mgmt) as name ORDER BY name')
73 |
74 | if len(mgmt) > 0:
75 | print("Available Groups:")
76 | for s in mgmt.records:
77 | print("> " + str(s.name))
78 | else:
79 | print("No management groups found in DB")
80 |
81 |
82 | def print_dict_csv(netList):
83 | """Print out List of Dictionary Objects as CSV"""
84 |
85 | netKeys = []
86 |
87 | # Get Dict keys for CSV Out
88 | for key in sorted(netList[0].keys()):
89 | if key != "__values__":
90 | netKeys.append(key)
91 |
92 | netWriter = csv.writer(sys.stdout)
93 | netWriter.writerow(netKeys)
94 |
95 | # Go through all entries and dump them to CSV
96 | for en in netList:
97 | netValues = []
98 | for key in sorted(en.keys()):
99 | if key != "__values__":
100 | netValues.append(en[key])
101 | # Write Values to CSV
102 | netWriter.writerow(netValues)
103 |
104 |
105 | def get_net_filter(group):
106 | """Try to get a filter from the config"""
107 |
108 | try:
109 | gFilter = nglib.config['NetAlertFilter'][group]
110 | return gFilter
111 | except:
112 | raise Exception("No Group Filter Found", group)
113 |
114 |
115 | def check_net_filter(netDict, group=None, nFilter=None):
116 | """
117 | Filters Networks based on vrf:role (from supernets)
118 |
119 | Pass in a group filter to read from config, or a custom nFilter
120 |
121 | Examples:
122 | nst = all (all networks)
123 |
124 | # default VRF only with null supernet or new access layer supernets 10.177
125 | radonc = default:none|access-private|access-wan
126 |
127 | # EST wants all access layer networks plus all PCI networks
128 | est = default:none|access-private|access-wan pci:all
129 |
130 | # JCI Wants all fwutil networks
131 | jci = fwutil:all
132 |
133 | # COM wants only printer networks
134 | com = default:printer
135 | """
136 |
137 | if group:
138 | vDict = get_filter_dict(group=group)
139 | elif nFilter:
140 | vDict = get_filter_dict(nFilter=nFilter)
141 |
142 | # Check filter against vDict
143 | for vrf in vDict.keys():
144 | if netDict['VRF'] == vrf or vrf == "all":
145 | #print("VRF MATCH", vrf, netDict['VRF'])
146 | for role in vDict[vrf]:
147 | if netDict['NetRole'] == role or role == "all":
148 | if nglib.verbose > 1:
149 | logger.debug("VRF and Role MATCH: %s, %s, %s, %s, %s",
150 | group, vrf, netDict['VRF'], netDict['NetRole'], netDict['CIDR'])
151 | return True
152 |
153 | # No supernet role associated (none=None)
154 | elif role == "none" and not netDict['NetRole']:
155 | if nglib.verbose > 1:
156 | logger.debug("VRF and Role MATCH on Null Role: %s, %s, %s, %s, %s",
157 | group, vrf, netDict['VRF'], netDict['NetRole'], netDict['CIDR'])
158 | return True
159 |
160 | # No Match
161 | elif nglib.verbose > 3:
162 | logger.debug("NOMATCH VRF and Role: %s, %s, %s, %s, %s",
163 | group, vrf, netDict['VRF'], netDict['NetRole'], netDict['CIDR'])
164 |
165 | if nglib.verbose > 2:
166 | logger.debug("No Network Match for %s on %s", netDict['CIDR'], group)
167 | return False
168 |
169 |
170 | @functools.lru_cache(maxsize=1)
171 | def get_filter_dict(group=None, nFilter=None):
172 | """Process config for group or custom filter and cache it for each call"""
173 |
174 | vDict = dict()
175 | gFilter = None
176 |
177 | if group:
178 | gFilter = nglib.config['NetAlertFilter'][group]
179 | elif nFilter:
180 | gFilter = nFilter
181 | else:
182 | raise Exception("Must pass in group or filter")
183 |
184 | # Split all VRFs by spaces
185 | vrfFilters = gFilter.rsplit()
186 |
187 | # Process Filters
188 | for f in vrfFilters:
189 |
190 | # Multiple roles specified
191 | if re.search(':', f):
192 | (vrf, roles) = f.split(':')
193 | vDict[vrf] = [] # Empty vDict List
194 | if re.search('|', roles):
195 | rList = roles.split('|') # Split up multiple roles if exists
196 | for r in rList:
197 | vDict[vrf].append(r) # Append multiple roles to List
198 | else:
199 | vDict[vrf].append(roles) # Append single role to list
200 |
201 | # All roles for VRF
202 | else:
203 | vDict[f] = []
204 | vDict[f].append("all")
205 |
206 | if nglib.verbose > 1:
207 | print("vDict Contents", str(vDict))
208 |
209 | return vDict
210 |
211 | def universal_text_search(text, vrange, rtype="TREE"):
212 | """
213 | Try to find what someone is looking for based on a text string
214 | Uses a lot of try/except code, best to debug non-universal before
215 | debugging here.
216 | """
217 |
218 | if nglib.verbose:
219 | print("Universal Search for", text)
220 |
221 | found = False
222 |
223 | #Look for group filter first
224 | try:
225 | get_net_filter(text)
226 | nglib.query.net.get_networks_on_filter(text, rtype=rtype)
227 | found = True
228 | except:
229 | pass
230 |
231 | if not found:
232 |
233 | # Look for MGMT Group
234 | mgmt = nglib.bolt_ses.run(
235 | 'MATCH (s:Switch {mgmt:{mgmt}}) RETURN DISTINCT(s.mgmt) as name',
236 | {"mgmt": text})
237 |
238 | for m in mgmt:
239 | nglib.query.vlan.get_vlans_on_group(text, vrange)
240 | found = True
241 |
242 | if not found:
243 | # Look for Device Name
244 | devices = nglib.bolt_ses.run(
245 | 'MATCH (s:Switch {name:{switch}}) RETURN s.name as name',
246 | {"switch": text})
247 |
248 | for d in devices:
249 | nglib.query.dev.get_device(text, rtype=rtype, vrange=vrange)
250 | found = True
251 |
252 | if not found:
253 | print("Nothing found for Universal Search", text)
254 |
--------------------------------------------------------------------------------
/nglib/query/nNode.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python
2 | #
3 | # Neo4j py2neo Node and Edge Methods
4 | #
5 | # Copyright (c) 2016 "Jonathan Yantis"
6 | #
7 | # This file is a part of NetGrph.
8 | #
9 | # This program is free software: you can redistribute it and/or modify
10 | # it under the terms of the GNU Affero General Public License, version 3,
11 | # as published by the Free Software Foundation.
12 | #
13 | # This program is distributed in the hope that it will be useful,
14 | # but WITHOUT ANY WARRANTY; without even the implied warranty of
15 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
16 | # GNU Affero General Public License for more details.
17 | #
18 | # You should have received a copy of the GNU Affero General Public License
19 | # along with this program. If not, see .
20 | #
21 | # As a special exception, the copyright holders give permission to link the
22 | # code of portions of this program with the OpenSSL library under certain
23 | # conditions as described in each individual source file and distribute
24 | # linked combinations including the program with the OpenSSL library. You
25 | # must comply with the GNU Affero General Public License in all respects
26 | # for all of the code used other than as permitted herein. If you modify
27 | # file(s) with this exception, you may extend this exception to your
28 | # version of the file(s), but you are not obligated to do so. If you do not
29 | # wish to do so, delete this exception statement from your version. If you
30 | # delete this exception statement from all source files in the program,
31 | # then also delete it in the license file.
32 | """
33 | Py2Neo Node Properties (Deprecated)
34 |
35 | """
36 | import logging
37 | import json
38 | import nglib
39 |
40 | logger = logging.getLogger(__name__)
41 |
42 |
43 | def getJSONProperties(node):
44 | """Returns Properties for a node from JSON as a nested dictionary"""
45 |
46 | json_string = swapQuotes(str(node.properties))
47 | if nglib.verbose > 1:
48 | print("JSON:", json_string)
49 | parsed_json = json.loads(json_string)
50 | return parsed_json
51 |
52 |
53 | def getLabel(node):
54 | """Returns the label in plaintext for a node"""
55 |
56 | label = node.labels.copy()
57 | label = label.__str__()
58 | label = label.replace('{', '(')
59 | label = label.replace('}', ')')
60 |
61 | return label
62 |
63 | def getEdge(edge):
64 | """Return Edge Type"""
65 | return str(edge.type)
66 |
67 | def getRelationship(edge):
68 | """Get Relationship on edge for printing"""
69 |
70 | snode = edge.start_node
71 | snodeProp = getJSONProperties(snode)
72 | enode = edge.end_node
73 | enodeProp = getJSONProperties(enode)
74 | relation = getLabel(snode) + "{name:" + snodeProp['name'] + "}-[" + getEdge(edge)
75 | relation += "]->" + getLabel(enode) + "{name:" + enodeProp['name'] + "}"
76 |
77 | return relation
78 |
79 |
80 | def swapQuotes(myString):
81 | """Swaps single quote with double quotes"""
82 |
83 | myString = myString.replace("'", '"')
84 | return myString
85 |
--------------------------------------------------------------------------------
/nglib/report/__init__.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python3
2 | #
3 | # Copyright (c) 2016 "Jonathan Yantis"
4 | #
5 | # This file is a part of NetGrph.
6 | #
7 | # This program is free software: you can redistribute it and/or modify
8 | # it under the terms of the GNU Affero General Public License, version 3,
9 | # as published by the Free Software Foundation.
10 | #
11 | # This program is distributed in the hope that it will be useful,
12 | # but WITHOUT ANY WARRANTY; without even the implied warranty of
13 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
14 | # GNU Affero General Public License for more details.
15 | #
16 | # You should have received a copy of the GNU Affero General Public License
17 | # along with this program. If not, see .
18 | #
19 | # As a special exception, the copyright holders give permission to link the
20 | # code of portions of this program with the OpenSSL library under certain
21 | # conditions as described in each individual source file and distribute
22 | # linked combinations including the program with the OpenSSL library. You
23 | # must comply with the GNU Affero General Public License in all respects
24 | # for all of the code used other than as permitted herein. If you modify
25 | # file(s) with this exception, you may extend this exception to your
26 | # version of the file(s), but you are not obligated to do so. If you do not
27 | # wish to do so, delete this exception statement from your version. If you
28 | # delete this exception statement from all source files in the program,
29 | # then also delete it in the license file.
30 | #
31 | """Generates Reports on NetGrph Data"""
32 |
33 | import csv
34 | import re
35 | import os
36 | import sys
37 | import logging
38 | import functools
39 | import configparser
40 | import ipaddress
41 | import nglib
42 | import nglib.query
43 |
44 | verbose = 0
45 | logger = logging.getLogger(__name__)
46 |
47 |
48 | def get_vlan_report(vrange, group='.*', report="full", rtype="NGTREE"):
49 | """Generate VLAN Reports"""
50 |
51 | rtypes = ('TREE', 'JSON', 'YAML', 'NGTREE')
52 |
53 | if rtype in rtypes:
54 |
55 | # Full VLAN Report
56 | if report == "full":
57 | logger.info("Query: Generating Full VLAN Report (%s) for %s", vrange, nglib.user)
58 |
59 | # Get all VLANs as NGTree
60 | ngtree = None
61 | if group != '.*':
62 | ngtree = nglib.query.vlan.get_vlans_on_group(group, vrange, rtype="NGTREE")
63 | else:
64 | ngtree = get_vlan_data(vrange, rtype)
65 | if '_ccount' in ngtree.keys():
66 | nglib.query.exp_ngtree(ngtree, rtype)
67 | return ngtree
68 | else:
69 | print("No Vlans in Range", vrange)
70 |
71 | # Empty VLAN Report
72 | elif report == "empty":
73 | logger.info("Query: Generating Empty VLAN Report (%s) for %s", vrange, nglib.user)
74 |
75 | # Get all VLANs as NGTree
76 | ngtree = get_vlan_data(vrange, rtype)
77 |
78 | # Found some VLANs in Range
79 | if '_ccount' in ngtree:
80 | etree = nglib.ngtree.get_ngtree("Empty VLAN Report", tree_type="VIDs")
81 | clist = ngtree['data']
82 |
83 | # Find Empty VLANs in NGTree
84 | for cv in clist:
85 | get_empty_vlans(cv, etree)
86 |
87 | # Found Empty VLANs, count and print
88 | if '_ccount' in etree.keys():
89 | etree['Empty VLAN Count'] = etree['_ccount']
90 | nglib.query.exp_ngtree(etree, rtype)
91 | return etree
92 | else:
93 | print("No Empty Vlans in Range", vrange)
94 |
95 | else:
96 | print("No Vlans in Range", vrange)
97 | else:
98 | raise Exception("RType Not Supported, use:" + str(rtypes))
99 |
100 |
101 | def get_empty_vlans(ngtree, etree):
102 | """
103 | Find all empty VLANs on a VID tree instance
104 |
105 | Notes: Called recursively on any bridge trees and builds etree
106 | """
107 |
108 | vidlist = ngtree['data']
109 |
110 | # VID in Bridge Group
111 | for vid in vidlist:
112 |
113 | # No MACs in VLAN
114 | if vid['MAC Count'] == 0:
115 | childlist = vid['data']
116 | if not childlist:
117 | nglib.ngtree.add_child_ngtree(etree, vid)
118 | else:
119 | get_empty_vlans(vid, etree)
120 |
121 | return etree
122 |
123 | def get_vlan_data(vrange, rtype, group='.*'):
124 | """Get all vlans in a range for reports"""
125 |
126 | allSwitches = True
127 | if rtype == "TREE":
128 | allSwitches = False
129 |
130 | (vlow, vhigh) = nglib.query.vlan.get_vlan_range(vrange)
131 |
132 | vlans = nglib.bolt_ses.run(
133 | 'MATCH(v:VLAN) WHERE toInt(v.vid) >= {vlow} AND toInt(v.vid) <= {vhigh} '
134 | + 'AND v.mgmt =~ {group} '
135 | + 'RETURN DISTINCT v.vid AS vid ORDER BY toInt(vid)',
136 | {"vlow": vlow, "vhigh": vhigh, "group": group})
137 |
138 | pngtree = nglib.ngtree.get_ngtree("Report", tree_type="VIDs")
139 |
140 | for v in vlans:
141 | vtree = nglib.query.vlan.search_vlan_id(v['vid'], allSwitches=allSwitches)
142 | nglib.ngtree.add_child_ngtree(pngtree, vtree)
143 | return pngtree
144 |
145 |
146 | def get_vrf_report(vrf, rtype="NGTREE"):
147 | """
148 | Get a report on vrfs that match regex
149 | """
150 | rtypes = ('TREE', 'JSON', 'YAML', 'NGTREE')
151 |
152 | if rtype in rtypes:
153 | logger.info("Query: Generating VRF Report (%s) for %s", vrf, nglib.user)
154 |
155 | vrfs = nglib.bolt_ses.run(
156 | 'MATCH(v:VRF) WHERE v.name =~ {vrf} '
157 | + 'RETURN v.name AS name ORDER BY name',
158 | {"vrf": vrf})
159 |
160 | ngtree = nglib.ngtree.get_ngtree("Report", tree_type="VRFs")
161 | ngtree['VRF Regex'] = vrf
162 |
163 | # Process VRFs
164 | tree_count = 0
165 | vrflist = []
166 | for v in vrfs:
167 | tree_count += 1
168 | cngtree = nglib.query.net.get_networks_on_filter(nFilter=v["name"], rtype="NGTREE")
169 | if cngtree:
170 | vrflist.append(v["name"])
171 | devlist = nglib.query.dev.get_devlist_vrf(v["name"])
172 | cngtree["Routers"] = devlist
173 | tree_count += 1
174 | nglib.ngtree.add_child_ngtree(ngtree, cngtree)
175 |
176 | # Return output
177 | if not tree_count:
178 | print("No VRFs found on regex:", vrf, file=sys.stderr)
179 | elif tree_count == 1:
180 | nglib.query.exp_ngtree(cngtree, rtype)
181 | return ngtree
182 | else:
183 | ngtree["VRFs"] = vrflist
184 | nglib.query.exp_ngtree(ngtree, rtype)
185 | return ngtree
186 | else:
187 | raise Exception("RType Not Supported, use:" + str(rtypes))
188 |
189 |
190 | def get_dev_report(dev, group=".*", trunc=False, rtype="NGTREE"):
191 | """
192 | Get all devices on a regex
193 |
194 | Options: trunc==True returns truncated list
195 | """
196 |
197 | rtypes = ('TREE', 'JSON', 'YAML', 'NGTREE')
198 |
199 | if rtype in rtypes:
200 | logger.info("Query: Generating Device Report (%s) for %s", dev, nglib.user)
201 |
202 | devices = nglib.bolt_ses.run(
203 | 'MATCH(s:Switch) WHERE s.mgmt =~ {mgmt} AND s.name =~ {dev} '
204 | + 'RETURN s.name AS name, s.mgmt AS mgmt, '
205 | + 's.location AS location, s.model AS model, s.version AS version, '
206 | + 's.distance AS distance, s.Platform AS platform, s.FQDN as FQDN '
207 | + 'ORDER BY name',
208 | {'dev': dev, 'mgmt': group})
209 |
210 | ngtree = nglib.ngtree.get_ngtree("Report", tree_type="DEVS")
211 | ngtree['Device Regex'] = dev
212 |
213 | for d in devices:
214 | if d["mgmt"]:
215 | if trunc:
216 | ct = nglib.ngtree.get_ngtree(d['name'], tree_type="DEV")
217 | ct['Distance'] = d['distance']
218 | ct['Location'] = d['location']
219 | ct['MGMT Group'] = d['mgmt']
220 | ct['Model'] = d['model']
221 | ct['Version'] = d['version']
222 | ct['Platform'] = d['platform']
223 | ct['FQDN'] = d['FQDN']
224 | nglib.ngtree.add_child_ngtree(ngtree, ct)
225 | else:
226 | cngtree = nglib.query.dev.get_device(d["name"])
227 | if cngtree:
228 | nglib.ngtree.add_child_ngtree(ngtree, cngtree)
229 |
230 | # Found Devices, count and print
231 | if '_ccount' in ngtree.keys():
232 | ngtree['Device Count'] = ngtree['_ccount']
233 | nglib.query.exp_ngtree(ngtree, rtype)
234 | return ngtree
235 | else:
236 | print("No Devices found on regex:", dev)
237 |
238 | else:
239 | raise Exception("RType Not Supported, use:" + str(rtypes))
240 |
241 |
242 |
243 |
--------------------------------------------------------------------------------
/ngreport.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python3
2 | #
3 | #
4 | # Copyright (c) 2016 "Jonathan Yantis"
5 | #
6 | # This file is a part of NetGrph.
7 | #
8 | # This program is free software: you can redistribute it and/or modify
9 | # it under the terms of the GNU Affero General Public License, version 3,
10 | # as published by the Free Software Foundation.
11 | #
12 | # This program is distributed in the hope that it will be useful,
13 | # but WITHOUT ANY WARRANTY; without even the implied warranty of
14 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
15 | # GNU Affero General Public License for more details.
16 | #
17 | # You should have received a copy of the GNU Affero General Public License
18 | # along with this program. If not, see .
19 | #
20 | # As a special exception, the copyright holders give permission to link the
21 | # code of portions of this program with the OpenSSL library under certain
22 | # conditions as described in each individual source file and distribute
23 | # linked combinations including the program with the OpenSSL library. You
24 | # must comply with the GNU Affero General Public License in all respects
25 | # for all of the code used other than as permitted herein. If you modify
26 | # file(s) with this exception, you may extend this exception to your
27 | # version of the file(s), but you are not obligated to do so. If you do not
28 | # wish to do so, delete this exception statement from your version. If you
29 | # delete this exception statement from all source files in the program,
30 | # then also delete it in the license file.
31 | #
32 | #
33 | """ngreport Generates Reports from the NetGrph Database"""
34 | import os
35 | import re
36 | import argparse
37 | import nglib
38 | import nglib.report
39 |
40 | # Default Config File Location
41 | config_file = '/etc/netgrph.ini'
42 | alt_config = './docs/netgrph.ini'
43 |
44 | # Test/Dev Config
45 | dirname = os.path.dirname(os.path.realpath(__file__))
46 | if re.search(r'\/dev$', dirname):
47 | config_file = 'netgrphdev.ini'
48 | elif re.search(r'\/test$', dirname):
49 | config_file = "netgrphdev.ini"
50 |
51 | parser = argparse.ArgumentParser()
52 |
53 | parser = argparse.ArgumentParser(prog='ngreport',
54 | description='Generate Reports from NetGrph')
55 |
56 | parser.add_argument("-vrf", metavar='name',
57 | help="Generate a Report on a VRF (.* for all)",
58 | type=str)
59 | parser.add_argument("-vlans", help="VLAN ID Report (combine with -vra and -e)",
60 | action="store_true")
61 |
62 | parser.add_argument("-vrange", metavar='1[-4096]', help="VLAN Range (default 1-1999)",
63 | type=str)
64 | parser.add_argument("-dev", metavar=".*", help="Report on Network Devices in regex",
65 | type=str)
66 | parser.add_argument("-output", metavar='TREE',
67 | help="Return Format: TREE, TABLE, CSV, JSON, YAML", type=str)
68 | parser.add_argument("-empty",
69 | help="Only Return Empty VLANs (requires NetDB)",
70 | action="store_true")
71 | parser.add_argument("--conf", metavar='file', help="Alternate Config File", type=str)
72 | parser.add_argument("--debug", help="Set debugging level", type=int)
73 | parser.add_argument("-v", help="Verbose Output", action="store_true")
74 |
75 | args = parser.parse_args()
76 |
77 | # Alternate Config File
78 | if args.conf:
79 | config_file = args.conf
80 |
81 | # Test configuration exists
82 | if not os.path.exists(config_file):
83 | if not os.path.exists(alt_config):
84 | raise Exception("Configuration File not found", config_file)
85 | else:
86 | config_file = alt_config
87 |
88 | verbose = 0
89 | if args.v:
90 | verbose = 1
91 | if args.debug:
92 | verbose = args.debug
93 |
94 | # Default VLAN Range
95 | if not args.vrange:
96 | args.vrange = "1-1999"
97 | if args.output:
98 | args.output = args.output.upper()
99 |
100 | # Setup Globals
101 | nglib.verbose = verbose
102 |
103 | # Initialize Library
104 | nglib.init_nglib(config_file)
105 |
106 | if args.vlans:
107 | report = "full"
108 | if args.empty:
109 | report = "empty"
110 |
111 | rtype = "TREE"
112 | if args.output:
113 | rtype = args.output
114 |
115 | nglib.report.get_vlan_report(args.vrange, report=report, rtype=rtype)
116 |
117 | elif args.vrf:
118 | rtype = "TREE"
119 | if args.output:
120 | rtype = args.output
121 |
122 | nglib.report.get_vrf_report(args.vrf, rtype=rtype)
123 |
124 | elif args.dev:
125 | rtype = "TREE"
126 | if args.output:
127 | rtype = args.output
128 |
129 | nglib.report.get_dev_report(args.dev, rtype=rtype)
130 |
131 | else:
132 | parser.print_help()
133 | print()
134 |
--------------------------------------------------------------------------------
/ngtest.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python3
2 | #
3 | # Copyright (c) 2016 "Jonathan Yantis"
4 | #
5 | # This file is a part of NetGrph.
6 | #
7 | # This program is free software: you can redistribute it and/or modify
8 | # it under the terms of the GNU Affero General Public License, version 3,
9 | # as published by the Free Software Foundation.
10 | #
11 | # This program is distributed in the hope that it will be useful,
12 | # but WITHOUT ANY WARRANTY; without even the implied warranty of
13 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
14 | # GNU Affero General Public License for more details.
15 | #
16 | # You should have received a copy of the GNU Affero General Public License
17 | # along with this program. If not, see .
18 | #
19 | # As a special exception, the copyright holders give permission to link the
20 | # code of portions of this program with the OpenSSL library under certain
21 | # conditions as described in each individual source file and distribute
22 | # linked combinations including the program with the OpenSSL library. You
23 | # must comply with the GNU Affero General Public License in all respects
24 | # for all of the code used other than as permitted herein. If you modify
25 | # file(s) with this exception, you may extend this exception to your
26 | # version of the file(s), but you are not obligated to do so. If you do not
27 | # wish to do so, delete this exception statement from your version. If you
28 | # delete this exception statement from all source files in the program,
29 | # then also delete it in the license file.
30 | #
31 | #
32 | """
33 | Test Database functions for errors
34 | """
35 | import re
36 | import sys
37 | import subprocess
38 |
39 | ngcmd = './netgrph.py' + ' '
40 | upngcmd = './ngupdate.py' + ' '
41 | ngrepcmd = './ngreport.py' + ' '
42 |
43 | # Query Tests (cmd, result)
44 | qtest = dict()
45 | qtest['-dev core1'] = 'Child Neighbors'
46 | qtest['-ip 10.9.46.1'] = 'Gateway'
47 | qtest['-fp 10.1.120.50 8.8.8.8'] = 'ExternalFW'
48 | qtest['-net 10.9.46.0/23 -o YAML'] = 'Router: xyz2mdf'
49 | qtest['-nlist test_group'] = '10.9.46.0/23'
50 | qtest['-group Core'] = 'core1'
51 | qtest['-vid 120'] = '10.1.120.0/23'
52 | qtest['-vtree ABC-120'] = '10.1.120.0/23'
53 | qtest['-ip 10.9.136.1 -o JSON'] = '"IP": "10.9.136.1"'
54 | qtest['-ip 10.9.136.1 -o YAML'] = 'IP: 10.9.136.1'
55 | qtest['-sp xyz2mdf abc4mdf'] = 'core1'
56 | qtest['-rp 10.1.108.50 10.1.20.50'] = 'core1'
57 | qtest['-p 10.1.120.50 8.8.8.8'] = 'Description : Default Route'
58 |
59 | # Production tests
60 | ptest = dict()
61 | ptest['-dev core1'] = 'Child Neighbors'
62 | ptest['-ip 10.32.1.1'] = 'Gateway'
63 | ptest['-fp 10.33.1.1 8.8.8.8'] = 'ExternalFW'
64 | ptest['-net 10.32.0.0/17'] = 'wireless-GUEST'
65 | ptest['-nlist security'] = '10.32.0.0/17'
66 | ptest['-group Core'] = 'core1'
67 | ptest['-vid 641'] = '10.32.0.0/17'
68 | ptest['-vtree Core-641'] = '10.32.0.0/17'
69 | ptest['-ip 10.32.1.1 -o JSON'] = '"IP": "10.32.1.1"'
70 | ptest['-sp mdcmdf hvt404mdf'] = 'core1'
71 | ptest['-rp 10.33.1.100 10.26.76.1'] = 'core1'
72 | ptest['-p 10.26.72.142 10.34.72.24'] = 'L4-FW FwutilFW'
73 | ptest['10.26.72.142 10.34.72.24'] = 'L4-FW FwutilFW'
74 |
75 |
76 | # NetDB
77 | #ptest['-p 10.26.72.142 10.28.6.27'] = 'Link rVLANs'
78 |
79 | rtest = dict()
80 | #rtest['-vlans -empty -vrange 200'] = 'v6test'
81 | rtest['-vlans -vrange 120 -o yaml'] = 'Root: abc4mdf'
82 | rtest['-v -dev "xyz.*" -o json'] = '"Parent Switch": "core1"'
83 |
84 | prtest = dict()
85 | prtest['-vlans -empty -vrange 200'] = 'v6test'
86 |
87 | # Update Tests (check for errors)
88 | utest = ('-ivrf', '-id', '-ind', '-inet', '-isnet', '-iasa', '-ivlan', '-uvlan', '-ild')
89 |
90 | def run_query(cmd, option, result):
91 | """Run Queries for testing"""
92 |
93 | runcmd = cmd + option
94 | proc = subprocess.Popen([runcmd], stdout=subprocess.PIPE, stderr=subprocess.PIPE,
95 | shell=True, universal_newlines=True)
96 | (out, err) = proc.communicate()
97 |
98 | if re.search(r'Traceback', err):
99 | print(err, file=sys.stderr)
100 | return False
101 | elif result and re.search(result, out):
102 | return True
103 | elif not result:
104 | return True
105 | else:
106 | return False
107 |
108 | def test_dev_updates():
109 | print("Testing Update Routines")
110 |
111 | for ut in utest:
112 | print(ut)
113 | assert(run_query(upngcmd, ut, None)) == True
114 |
115 | def test_dev_queries():
116 | """netgrph query testing"""
117 |
118 | print("Testing Query Routines")
119 |
120 | for key in qtest.keys():
121 | print(key)
122 | assert(run_query(ngcmd, key, qtest[key])) == True
123 |
124 | for key in rtest.keys():
125 | print(key)
126 | assert(run_query(ngrepcmd, key, rtest[key])) == True
127 |
128 | def test_prod_updates():
129 | test_dev_updates()
130 |
131 | def test_prod_queries():
132 | global qtest
133 | global rtest
134 | qtest = ptest
135 | rtest = prtest
136 |
137 | test_dev_queries()
138 |
139 |
140 |
--------------------------------------------------------------------------------
/requirements-client.txt:
--------------------------------------------------------------------------------
1 | requests
2 | pyyaml
3 |
--------------------------------------------------------------------------------
/requirements.txt:
--------------------------------------------------------------------------------
1 | # Python Requirements
2 | ciscoconfparse
3 | pylint
4 | pyyaml
5 | pymysql
6 | py2neo==2.0.8
7 | neo4j-driver==1.0.2
8 | Flask
9 | Flask-HTTPAuth
10 | flask_limiter
11 | passlib
12 | sqlalchemy
13 | flask_sqlalchemy
14 | requests
15 | uwsgi
16 |
--------------------------------------------------------------------------------
/test/__pycache__/testng.cpython-34-PYTEST.pyc:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/yantisj/netgrph/0ebc72efbd970e0ce76d44d91a27d167103633b5/test/__pycache__/testng.cpython-34-PYTEST.pyc
--------------------------------------------------------------------------------
/test/apisrv:
--------------------------------------------------------------------------------
1 | ../apisrv/
--------------------------------------------------------------------------------
/test/bitwise.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python
2 | import sys
3 | import ipaddress
4 |
5 |
6 | def int_to_bin(num, bits=32):
7 | r = ''
8 | while bits:
9 | r = ('1' if num&1 else '0') + r
10 | bits = bits - 1
11 | num = num >> 1
12 | print(r)
13 |
14 |
15 | myip = sys.argv[1]
16 |
17 |
18 | ip = ipaddress.IPv4Address(myip)
19 | i = int.from_bytes(ip.packed, byteorder='big')
20 |
21 | print(ip.packed)
22 | print(i)
23 | int_to_bin(i)
24 |
25 |
--------------------------------------------------------------------------------
/test/csv/allnets.csv:
--------------------------------------------------------------------------------
1 | Subnet,VLAN,VRF,Router,Gateway,MGMT Group,Description,P2P,Standby
2 | 10.3.108.10/31,2108,default,core1,10.3.108.10,Core,abc4mdf,True,False
3 | 10.3.108.10/31,2323,guest,core1,10.3.108.10,Core,abc4mdf-core1 vrf guest,True,False
4 | 10.3.108.10/31,2360,perim,core1,10.3.108.10,Core,core1-abc4mdf vrf perim,True,False
5 | 10.3.108.10/31,2460,utility,core1,10.3.108.10,Core,core1-abc4mdf vrf utility,True,False
6 | 10.3.108.20/31,3108,default,core2,10.3.108.20,Core,abc4mdf,True,True
7 | 10.3.108.20/31,3323,guest,core2,10.3.108.20,Core,abc4mdf-core2 vrf guest,True,True
8 | 10.3.108.20/31,3360,perim,core2,10.3.108.20,Core,core2-abc4mdf vrf perim,True,True
9 | 10.3.108.20/31,3460,utility,core2,10.3.108.20,Core,core2-abc4mdf vrf utility,True,True
10 | 10.4.108.0/24,1,default,abc4mdf,10.4.108.1,ABC,None,False,False
11 | 10.1.108.0/23,108,default,abc4mdf,10.1.108.50,ABC,None,False,False
12 | 10.1.120.0/23,120,default,abc4mdf,10.1.120.50,ABC,None,False,False
13 | 10.6.64.0/22,260,default,abc4mdf,10.6.64.50,ABC,ABC-Utility,False,False
14 | 10.9.8.0/23,270,guest,abc4mdf,10.9.8.1,ABC,guest,False,False
15 | 10.9.136.0/23,271,perim,abc4mdf,10.9.136.1,ABC,reg,False,False
16 | 10.9.200.0/23,272,perim,abc4mdf,10.9.200.1,ABC,deadend,False,False
17 | 10.8.64.0/22,280,default,abc4mdf,10.8.64.50,ABC,None,False,False
18 | 10.14.64.0/22,340,utility,abc4mdf,10.14.64.50,ABC,ABC-SECURE-UTILITY,False,False
19 | 10.7.108.0/23,408,default,abc4mdf,10.7.108.50,ABC,None,False,False
20 | 10.7.120.0/23,420,default,abc4mdf,10.7.120.50,ABC,None,False,False
21 | 10.3.108.10/31,2108,default,abc4mdf,10.3.108.11,ABC,None,True,False
22 | 10.3.108.10/31,2323,guest,abc4mdf,10.3.108.11,ABC,abc4mdf-core1 vrf guest,True,False
23 | 10.3.108.10/31,2360,perim,abc4mdf,10.3.108.11,ABC,abc4mdf-core1 vrf perim,True,False
24 | 10.3.108.10/31,2460,utility,abc4mdf,10.3.108.11,ABC,abc4mdf-core1 vrf utility,True,False
25 | 10.3.108.20/31,3108,default,abc4mdf,10.3.108.21,ABC,None,True,False
26 | 10.3.108.20/31,3323,guest,abc4mdf,10.3.108.21,ABC,abc4mdf-core1 vrf guest,True,False
27 | 10.3.108.20/31,3360,perim,abc4mdf,10.3.108.21,ABC,abc4mdf-core2 vrf perim,True,False
28 | 10.3.108.20/31,3460,utility,abc4mdf,10.3.108.21,ABC,abc4mdf-core2 vrf utlility,True,False
29 | 10.3.20.10/31,2020,default,core1,10.3.20.10,Core,XYZ2mdf EALERT,True,False
30 | 10.3.20.10/31,2372,guest,core1,10.3.20.10,Core,xyz2mdf vrf guest,True,False
31 | 10.3.20.10/31,2373,perim,core1,10.3.20.10,Core,xyz2mdf-core1-vrf-perim,True,False
32 | 10.3.20.10/31,2474,utility,core1,10.3.20.10,Core,core1-xyz vrf utility,True,False
33 | 10.3.20.20/31,3020,default,core2,10.3.20.20,Core,XYZ2MDF P2P,True,True
34 | 10.3.20.20/31,3372,guest,core2,10.3.20.20,Core,XYZ2MDF-core2-vrf-perim,True,True
35 | 10.3.20.20/31,3474,utility,core2,10.3.20.20,Core,core2-xyz vrf utility,True,True
36 | 10.4.20.0/24,1,default,xyz2mdf,10.4.20.1,XYZ,None,False,False
37 | 10.1.20.0/23,20,default,xyz2mdf,10.1.20.50,XYZ,None,False,False
38 | 10.1.206.0/23,206,default,xyz2mdf,10.1.206.50,XYZ,None,False,False
39 | 10.6.188.0/22,260,default,xyz2mdf,10.6.188.1,XYZ,None,False,False
40 | 10.9.46.0/23,270,guest,xyz2mdf,10.9.46.1,XYZ,guest,False,False
41 | 10.8.96.0/22,280,default,xyz2mdf,10.8.96.1,XYZ,None,False,False
42 | 10.14.188.0/22,340,utility,xyz2mdf,10.14.188.50,XYZ,None,False,False
43 | 10.7.206.0/23,406,default,xyz2mdf,10.7.206.50,XYZ,None,False,False
44 | 10.7.20.0/23,420,default,xyz2mdf,10.7.20.50,XYZ,None,False,False
45 | 10.3.20.10/31,2020,default,xyz2mdf,10.3.20.11,XYZ,None,True,False
46 | 10.3.20.10/31,2372,guest,xyz2mdf,10.3.20.11,XYZ,XYZ2mdf-core1 vrf guest,True,False
47 | 10.3.20.10/31,2373,perim,xyz2mdf,10.3.20.11,XYZ,XYZ2mdf-core1 vrf perim,True,False
48 | 10.3.20.10/31,2474,utility,xyz2mdf,10.3.20.11,XYZ,XYZ-core1 vrf utility,True,False
49 | 10.3.20.20/31,3020,default,xyz2mdf,10.3.20.21,XYZ,None,True,False
50 | 10.3.20.20/31,3372,guest,xyz2mdf,10.3.20.21,XYZ,XYZ2mdf-core1 vrf guest,True,False
51 | 10.3.20.20/31,3373,perim,xyz2mdf,10.3.20.21,XYZ,None,True,False
52 | 10.3.20.20/31,3474,utility,xyz2mdf,10.3.20.21,XYZ,XYZ-core2 vrf utility,True,False
53 | 10.5.20.0/24,630,perim,core1,10.5.20.50,Core,FW,False,False
54 | 10.5.10.0/24,631,default,core1,10.5.10.50,Core,None,False,False
55 | 10.5.20.0/24,630,perim,core2,10.5.20.50,Core,FW,False,True
56 | 10.5.10.0/24,631,default,core2,10.5.10.50,Core,None,False,True
57 | 10.1.0.224/29,1095,perim,core1,10.1.0.225,Core,External-FW,False,False
58 | 10.1.0.224/29,1099,outside,core1,10.1.0.226,Core,None,False,False
59 |
--------------------------------------------------------------------------------
/test/csv/allvlans.csv:
--------------------------------------------------------------------------------
1 | MGMT,VID,VName,Switch,STP
2 | Core,2108,core-abc4mdf,core1,0
3 | Core,2323,core1-abcmdf-guest-vrf,core1,0
4 | Core,2360,core1-abc4mdf-vrf-perim,core1,0
5 | Core,2460,core1-abc4mdf-utility,core1,0
6 | Core,3108,core2-abc4mdf,core2,0
7 | Core,3323,core2-abcmdf-guest-vrf,core2,0
8 | Core,3360,core2-abc4mdf-vrf-perim,core2,0
9 | Core,3460,core1-abc4mdf-utility,core2,0
10 | ABC,108,Abc,abc4mdf,24576
11 | ABC,108,Abc-TestNet,abc2e1sw1,0
12 | ABC,108,Abc-TestNet,abc2sw1,0
13 | ABC,108,Abc-TestNet,abc437sw1,0
14 | ABC,108,Abc-TestNet,abc437sw2,0
15 | ABC,108,Abc-TestNet,abc4sw1,0
16 | ABC,120,Abc2,abc4mdf,24576
17 | ABC,120,Abc2-TestNet,abc2e1sw1,0
18 | ABC,120,Abc2-TestNet,abc2sw1,0
19 | ABC,120,Abc2-TestNet,abc437sw1,0
20 | ABC,120,Abc2-TestNet,abc437sw2,0
21 | ABC,120,Abc2-TestNet,abc4sw1,0
22 | ABC,1246,vendor-span,abc2e1sw1,0
23 | ABC,1246,vendor-span,abc4mdf,0
24 | ABC,2108,core1-abc4mdf-default,abc4mdf,0
25 | ABC,2323,abcmdf-core1-guest-vrf,abc4mdf,0
26 | ABC,2360,abc4mdf-core1-vrf-perim,abc4mdf,0
27 | ABC,2460,core1-abc4mdf-utility,abc4mdf,0
28 | ABC,260,Utility-Devices,abc2e1sw1,0
29 | ABC,260,Utility-Devices,abc2sw1,0
30 | ABC,260,Utility-Devices,abc437sw1,0
31 | ABC,260,Utility-Devices,abc437sw2,0
32 | ABC,260,Utility-Devices,abc4sw1,0
33 | ABC,260,utility,abc4mdf,24576
34 | ABC,280,Printers,abc2e1sw1,0
35 | ABC,280,Printers,abc2sw1,0
36 | ABC,280,Printers,abc437sw1,0
37 | ABC,280,Printers,abc437sw2,0
38 | ABC,280,Printers,abc4mdf,24576
39 | ABC,280,Printers,abc4sw1,0
40 | ABC,3108,core2-abc4mdf-default,abc4mdf,0
41 | ABC,3323,abcmdf-core2-guest-vrf,abc4mdf,0
42 | ABC,3360,abc4mdf-core2-vrf-perim,abc4mdf,0
43 | ABC,34,PerimeterNet,abc2e1sw1,0
44 | ABC,34,PerimeterNet,abc2sw1,0
45 | ABC,34,PerimeterNet,abc437sw1,0
46 | ABC,34,PerimeterNet,abc437sw2,0
47 | ABC,34,PerimeterNet,abc4sw1,0
48 | ABC,340,SECURE-UTILITY,abc2e1sw1,0
49 | ABC,340,SECURE-UTILITY,abc2sw1,0
50 | ABC,340,SECURE-UTILITY,abc437sw1,0
51 | ABC,340,SECURE-UTILITY,abc437sw2,0
52 | ABC,340,SECURE-UTILITY,abc4mdf,0
53 | ABC,340,SECURE-UTILITY,abc4sw1,0
54 | ABC,3460,core2-abc4mdf-utility,abc4mdf,0
55 | ABC,408,abc-v108-voice,abc2e1sw1,0
56 | ABC,408,abc-v108-voice,abc2sw1,0
57 | ABC,408,abc-v108-voice,abc437sw1,0
58 | ABC,408,abc-v108-voice,abc437sw2,0
59 | ABC,408,abc-v108-voice,abc4mdf,24576
60 | ABC,408,abc-v108-voice,abc4sw1,0
61 | ABC,420,abc-v120-voice,abc2e1sw1,0
62 | ABC,420,abc-v120-voice,abc2sw1,0
63 | ABC,420,abc-v120-voice,abc437sw1,0
64 | ABC,420,abc-v120-voice,abc437sw2,0
65 | ABC,420,abc-v120-voice,abc4mdf,24576
66 | ABC,420,abc-v120-voice,abc4sw1,0
67 | ABC,499,abc-utility-voice,abc2e1sw1,0
68 | ABC,499,abc-utility-voice,abc2sw1,0
69 | ABC,499,abc-utility-voice,abc437sw1,0
70 | ABC,499,abc-utility-voice,abc437sw2,0
71 | ABC,499,abc-utility-voice,abc4mdf,24576
72 | ABC,499,abc-utility-voice,abc4sw1,0
73 | XYZ,20,XYZ-Date,xyz1sw1,0
74 | XYZ,20,XYZ-Date,xyz1sw2,0
75 | XYZ,20,XYZ-Date,xyz2mdf,24576
76 | XYZ,20,XYZ-Date,xyz2sw1,0
77 | XYZ,20,XYZ-Date,xyz3sw1,0
78 | XYZ,20,XYZ-Date,xyz4sw1,0
79 | XYZ,20,XYZ-Date,xyz4sw2,0
80 | XYZ,2000,NONAME,xyz2mdf,0
81 | XYZ,2020,XYZ2mdf-core1,xyz2mdf,0
82 | XYZ,206,XYZ-date2,xyz1sw1,0
83 | XYZ,206,XYZ-date2,xyz1sw2,0
84 | XYZ,206,XYZ-date2,xyz2mdf,24576
85 | XYZ,206,XYZ-date2,xyz2sw1,0
86 | XYZ,206,XYZ-date2,xyz3sw1,0
87 | XYZ,206,XYZ-date2,xyz4sw1,0
88 | XYZ,206,XYZ-date2,xyz4sw2,0
89 | XYZ,2372,XYZ-core1-vrf-guest,xyz2mdf,0
90 | XYZ,2373,XYZ-core1-vrf-perim,xyz2mdf,0
91 | XYZ,2474,core1-XYZ-utility,xyz2mdf,0
92 | XYZ,260,[XYZ-utility],xyz1sw1,0
93 | XYZ,260,[XYZ-utility],xyz1sw2,0
94 | XYZ,260,[XYZ-utility],xyz2mdf,24576
95 | XYZ,260,[XYZ-utility],xyz2sw1,0
96 | XYZ,260,[XYZ-utility],xyz3sw1,0
97 | XYZ,260,[XYZ-utility],xyz4sw1,0
98 | XYZ,260,[XYZ-utility],xyz4sw2,0
99 | XYZ,280,[XYZ_printers],xyz1sw1,0
100 | XYZ,280,[XYZ_printers],xyz1sw2,0
101 | XYZ,280,[XYZ_printers],xyz2mdf,24576
102 | XYZ,280,[XYZ_printers],xyz2sw1,0
103 | XYZ,280,[XYZ_printers],xyz3sw1,0
104 | XYZ,280,[XYZ_printers],xyz4sw1,0
105 | XYZ,280,[XYZ_printers],xyz4sw2,0
106 | XYZ,3020,XYZ2mdf-core2,xyz2mdf,0
107 | XYZ,3372,XYZ-core2-vrf-guest,xyz2mdf,0
108 | XYZ,3373,XYZ-core2-vrf-perim,xyz2mdf,0
109 | XYZ,340,Utility-SEC,xyz1sw1,0
110 | XYZ,340,Utility-SEC,xyz1sw2,0
111 | XYZ,340,Utility-SEC,xyz2mdf,0
112 | XYZ,340,Utility-SEC,xyz2sw1,0
113 | XYZ,340,Utility-SEC,xyz3sw1,0
114 | XYZ,340,Utility-SEC,xyz4sw1,0
115 | XYZ,340,Utility-SEC,xyz4sw2,0
116 | XYZ,3474,core2-XYZ-utility,xyz2mdf,0
117 | XYZ,406,XYZ-406-voice,xyz1sw1,0
118 | XYZ,406,XYZ-406-voice,xyz1sw2,0
119 | XYZ,406,XYZ-406-voice,xyz2mdf,24576
120 | XYZ,406,XYZ-406-voice,xyz2sw1,0
121 | XYZ,406,XYZ-406-voice,xyz3sw1,0
122 | XYZ,406,XYZ-406-voice,xyz4sw1,0
123 | XYZ,406,XYZ-406-voice,xyz4sw2,0
124 | XYZ,420,XYZ-420-voice,xyz1sw1,0
125 | XYZ,420,XYZ-420-voice,xyz1sw2,0
126 | XYZ,420,XYZ-420-voice,xyz2mdf,24576
127 | XYZ,420,XYZ-420-voice,xyz2sw1,0
128 | XYZ,420,XYZ-420-voice,xyz3sw1,0
129 | XYZ,420,XYZ-420-voice,xyz4sw1,0
130 | XYZ,420,XYZ-420-voice,xyz4sw2,0
131 | Core,2020,core1-XYZ2mdf,core1,0
132 | Core,2372,core1-XYZ2mdf-vrf-guest,core1,0
133 | Core,2373,core1-XYZ2mdf-vrf-perim,core1,0
134 | Core,2474,core1-xyz-utility,core1,0
135 | Core,3020,core2-XYZ2mdf,core2,0
136 | Core,3372,core2-XYZ-vrf-guest,core2,0
137 | Core,3373,core2-XYZ-vrf-perim,core2,0
138 | Core,3474,core2-xyz-utility,core2,0
139 | Core,1246,SPAN-Net,core1,20480
140 | Core,1246,SPAN-Net,core2,20480
141 | XYZ,1246,vendor-span,xyz2mdf,0
142 | XYZ,1246,vendor-span,xyz4sw1,0
143 |
--------------------------------------------------------------------------------
/test/csv/devices.csv:
--------------------------------------------------------------------------------
1 | Device,FQDN,MgmtGroup,Type
2 | core1,core1.testnetwork.com,Core,Primary
3 | core2,core2.testnetwork.com,Core,Standby
4 | abc2e1sw1,abc2e1sw1.testnetwork.com,ABC,Switch
5 | abc2sw1,abc2sw1.testnetwork.com,ABC,Switch
6 | abc437sw1,abc437sw1.testnetwork.com,ABC,Switch
7 | abc437sw2,abc437sw2.testnetwork.com,ABC,Switch
8 | abc4mdf,abc4mdf.testnetwork.com,ABC,Primary
9 | abc4sw1,abc4sw1.testnetwork.com,ABC,Switch
10 | xyz2mdf,xyz2mdf.testnetwork.com,XYZ,Primary
11 | xyz1sw1,xyz1sw1.testnetwork.com,XYZ,Switch
12 | xyz1sw2,xyz1sw2.testnetwork.com,XYZ,Switch
13 | xyz2sw1,xyz2sw1.testnetwork.com,XYZ,Switch
14 | xyz3sw1,xyz3sw1.testnetwork.com,XYZ,Switch
15 | xyz4sw1,xyz4sw1.testnetwork.com,XYZ,Switch
16 | xyz4sw2,xyz4sw2.testnetwork.com,XYZ,Switch
17 |
--------------------------------------------------------------------------------
/test/csv/devinfo.csv:
--------------------------------------------------------------------------------
1 | Device,Location
2 |
--------------------------------------------------------------------------------
/test/csv/firewalls.csv:
--------------------------------------------------------------------------------
1 | Name,Interface,Description,Security-Level,IP,Hostname,Log-Index
2 | PerimeterFW,Vlan630,VRF Perim side of FW,0,None,fsm,firewalls
3 | PerimeterFW,Vlan631,Internal Network side of FW,100,None,fsm,firewalls
4 |
--------------------------------------------------------------------------------
/test/csv/links.csv:
--------------------------------------------------------------------------------
1 | Port,Switch,channel,desc,native,vlans
2 | Eth10/16,core1,0,abc4mdf EALERT,2108,"1246,2108,2323,2360,2460"
3 | Eth10/16,core2,0,abc4mdf EALERT,3108,"1246,3108,3323,3360,3460"
4 | Gi1/0/49,abc2e1sw1,0,,1,"1-1005,1246"
5 | Gi1/0/50,abc2e1sw1,0,,1,1-4096
6 | Gi1/0/51,abc2e1sw1,0,,1,1-4096
7 | Gi1/0/52,abc2e1sw1,0,,1,1-4096
8 | Gi1/0/49,abc2sw1,0,,1,1-1005
9 | Gi1/0/50,abc2sw1,0,,1,1-1005
10 | Gi1/0/51,abc2sw1,0,,1,1-1005
11 | Gi1/0/52,abc2sw1,0,,1,1-1005
12 | Gi1/0/49,abc437sw1,0,,1,1-1005
13 | Gi1/0/50,abc437sw1,0,,1,1-4096
14 | Gi1/0/51,abc437sw1,0,,1,1-1005
15 | Gi1/0/52,abc437sw1,0,,1,1-4096
16 | Gi2/0/49,abc437sw1,0,,1,1-4096
17 | Gi2/0/50,abc437sw1,0,,1,1-4096
18 | Gi2/0/51,abc437sw1,0,,1,1-4096
19 | Gi2/0/52,abc437sw1,0,,1,1-4096
20 | Gi3/0/49,abc437sw1,0,,1,1-4096
21 | Gi3/0/50,abc437sw1,0,,1,1-4096
22 | Gi3/0/51,abc437sw1,0,,1,1-4096
23 | Gi3/0/52,abc437sw1,0,,1,1-4096
24 | Gi4/0/49,abc437sw1,0,,1,1-4096
25 | Gi4/0/50,abc437sw1,0,,1,1-4096
26 | Gi4/0/51,abc437sw1,0,,1,1-4096
27 | Gi4/0/52,abc437sw1,0,,1,1-4096
28 | Gi1/0/49,abc437sw2,0,,1,1-1005
29 | Gi1/0/50,abc437sw2,0,,1,1-4096
30 | Gi1/0/51,abc437sw2,0,,1,1-4096
31 | Gi1/0/52,abc437sw2,0,,1,1-4096
32 | Gi2/0/49,abc437sw2,0,,1,1-4096
33 | Gi2/0/50,abc437sw2,0,,1,1-4096
34 | Gi2/0/51,abc437sw2,0,,1,1-4096
35 | Gi2/0/52,abc437sw2,0,,1,1-4096
36 | Gi1/0/25,abc4mdf,0,abc4sw1,1,1-1005
37 | Gi1/0/26,abc4mdf,0,abc2e1sw1,1,"1-1005,1246"
38 | Gi1/0/27,abc4mdf,0,,1,1-1005
39 | Gi1/0/28,abc4mdf,0,core1 EALERT,2108,"1246,2108,2323,2360,2460"
40 | Gi2/0/25,abc4mdf,0,abc437sw1,1,1-1005
41 | Gi2/0/26,abc4mdf,0,abc2sw1,1,1-1005
42 | Gi2/0/27,abc4mdf,0,,1,1-1005
43 | Gi2/0/28,abc4mdf,0,core2 EALERT,3108,"1246,3108,3323,3360,3460"
44 | Gi3/0/25,abc4mdf,0,abc437sw2,1,1-1005
45 | Gi3/0/26,abc4mdf,0,,1,1-1005
46 | Gi3/0/27,abc4mdf,0,,1,1-1005
47 | Gi1/0/49,abc4sw1,0,,1,1-1005
48 | Gi1/0/50,abc4sw1,0,,1,1-4096
49 | Gi1/0/51,abc4sw1,0,,1,1-4096
50 | Gi1/0/52,abc4sw1,0,,1,1-4096
51 | Gi2/0/49,abc4sw1,0,,1,1-4096
52 | Gi2/0/50,abc4sw1,0,,1,1-4096
53 | Gi2/0/51,abc4sw1,0,,1,1-4096
54 | Gi2/0/52,abc4sw1,0,,1,1-4096
55 | Eth10/5,core1,0,xyz2mdf EALERT,2020,"2020,2372-2373,2474"
56 | Eth10/5,core2,0,xyz2mdf EALERT,3020,"3020,3372-3373,3474"
57 | Gi1/0/1,xyz2mdf,10,,1,1-1005
58 | Gi1/0/2,xyz2mdf,11,,1,1-1005
59 | Gi1/0/3,xyz2mdf,20,,1,1-1005
60 | Gi1/0/4,xyz2mdf,31,,1,1-1005
61 | Gi1/0/5,xyz2mdf,41,,1,1-1005
62 | Gi1/0/6,xyz2mdf,42,,1,1-1005
63 | Gi1/0/7,xyz2mdf,0,,1,1-4000
64 | Gi1/0/8,xyz2mdf,0,,1,1-4000
65 | Gi1/0/9,xyz2mdf,0,,1,1-4000
66 | Gi1/0/10,xyz2mdf,0,,1,1-4000
67 | Gi1/0/11,xyz2mdf,0,,1,1-4000
68 | Gi1/0/12,xyz2mdf,0,core1 EALERT,2020,"2020,2372,2373,2474"
69 | Gi2/0/1,xyz2mdf,10,,1,1-1005
70 | Gi2/0/2,xyz2mdf,11,,1,1-1005
71 | Gi2/0/3,xyz2mdf,20,,1,1-1005
72 | Gi2/0/4,xyz2mdf,31,,1,1-1005
73 | Gi2/0/5,xyz2mdf,41,,1,1-1005
74 | Gi2/0/6,xyz2mdf,42,,1,1-1005
75 | Gi2/0/7,xyz2mdf,0,,1,1-4000
76 | Gi2/0/8,xyz2mdf,0,,1,1-4000
77 | Gi2/0/9,xyz2mdf,0,,1,1-4000
78 | Gi2/0/10,xyz2mdf,0,,1,1-4000
79 | Gi2/0/11,xyz2mdf,0,,1,1-4000
80 | Gi2/0/12,xyz2mdf,0,core2 EALERT,3020,"3020,3372,3373,3474"
81 | Te5/1,xyz1sw1,1,,1,1-1005
82 | Te5/2,xyz1sw1,1,,1,1-1005
83 | Te5/3,xyz1sw1,0,,1,1-1005
84 | Te5/4,xyz1sw1,0,,1,1-1005
85 | Gi1/1/1,xyz1sw2,1,,1,1-1005
86 | Gi1/1/2,xyz1sw2,0,,1,1-1005
87 | Gi2/1/1,xyz1sw2,1,,1,1-1005
88 | Te5/1,xyz2sw1,1,,1,1-1005
89 | Te5/2,xyz2sw1,1,,1,1-1005
90 | Te5/3,xyz2sw1,0,,1,1-1005
91 | Te5/4,xyz2sw1,0,,1,1-1005
92 | Te5/1,xyz3sw1,1,,1,1-1005
93 | Te5/2,xyz3sw1,1,,1,1-1005
94 | Te5/3,xyz3sw1,0,,1,1-1005
95 | Te5/4,xyz3sw1,0,,1,1-1005
96 | Te5/1,xyz4sw1,1,,1,1-1005
97 | Te5/2,xyz4sw1,1,,1,1-1005
98 | Te5/3,xyz4sw1,0,,1,1-1005
99 | Te5/4,xyz4sw1,0,,1,1-1005
100 | Gi1/1/1,xyz4sw2,1,,1,1-1005
101 | Gi1/1/2,xyz4sw2,1,,1,1-1005
102 | Gi1/1/3,xyz4sw2,0,,1,1-1005
103 | Gi1/1/4,xyz4sw2,0,,1,1-1005
104 | Eth7/26,core1,100,core2 EALERT,2001,"16,105,249,464,630-637,642-643,650-652,706,710,805,990,1246,1601-1610,1701,1803-1804,2001,2301,2315,2396,2404,2421-2422,2495,2519,3001,3238,3442"
105 | Eth8/26,core1,100,core2 EALERT,2001,"16,105,249,464,630-637,642-643,650-652,706,710,805,990,1246,1601-1610,1701,1803-1804,2001,2301,2315,2396,2404,2421-2422,2495,2519,3001,3238,3442"
106 | Eth7/8,core2,100,core1 EALERT,2001,"16,105,249,464,630-637,642-643,650-652,701,706,710,805,990,1246,1601-1610,1701,1803-1804,2001,2301,2315,2396,2404,2421-2422,2495,2519,3001,3238,3442"
107 | Eth8/26,core2,100,core1 EALERT,2001,"16,105,249,464,630-637,642-643,650-652,701,706,710,805,990,1246,1601-1610,1701,1803-1804,2001,2301,2315,2396,2404,2421-2422,2495,2519,3001,3238,3442"
108 |
--------------------------------------------------------------------------------
/test/csv/nd.csv:
--------------------------------------------------------------------------------
1 | LocalName,LocalPort,RemoteName,RemotePort
2 | core2,Eth10/16,abc4mdf,Gi2/0/28
3 | core1,Eth10/16,abc4mdf,Gi1/0/28
4 | abc2e1sw1,Gi1/0/49,abc4mdf,Gi1/0/26
5 | abc437sw2,Gi1/0/49,abc4mdf,Gi3/0/25
6 | abc437sw1,Gi1/0/51,abc4mdf,Gi2/0/25
7 | abc4sw1,Gi1/0/49,abc4mdf,Gi1/0/25
8 | abc2sw1,Gi1/0/49,abc4mdf,Gi2/0/26
9 | abc4mdf,Gi1/0/26,abc2e1sw1,Gi1/0/49
10 | abc4mdf,Gi1/0/28,core1,Eth10/16
11 | abc4mdf,Gi2/0/28,core2,Eth10/16
12 | abc4mdf,Gi1/0/25,abc4sw1,Gi1/0/49
13 | abc4mdf,Gi2/0/25,abc437sw1,Gi1/0/51
14 | abc4mdf,Gi2/0/26,abc2sw1,Gi1/0/49
15 | abc4mdf,Gi3/0/25,abc437sw2,Gi1/0/49
16 | core2,Eth7/8,core1,Eth8/26
17 | core2,Eth8/26,core1,Eth7/26
18 | core1,Eth7/26,core2,Eth8/26
19 | core1,Eth8/26,core2,Eth7/8
20 | core2,Eth10/5,xyz2mdf,Gi2/0/12
21 | core1,Eth10/5,xyz2mdf,Gi1/0/12
22 | xyz1sw1,Te5/2,xyz2mdf,Gi2/0/1
23 | xyz1sw1,Te5/1,xyz2mdf,Gi1/0/1
24 | xyz2sw1,Te5/2,xyz2mdf,Gi2/0/3
25 | xyz2sw1,Te5/1,xyz2mdf,Gi1/0/3
26 | xyz4sw1,Te5/2,xyz2mdf,Gi2/0/5
27 | xyz4sw1,Te5/1,xyz2mdf,Gi1/0/5
28 | xyz3sw1,Te5/2,xyz2mdf,Gi2/0/4
29 | xyz3sw1,Te5/1,xyz2mdf,Gi1/0/4
30 | xyz4sw2,Gi1/1/2,xyz2mdf,Gi2/0/6
31 | xyz4sw2,Gi1/1/1,xyz2mdf,Gi1/0/6
32 | xyz1sw2,Gi2/1/1,xyz2mdf,Gi1/0/2
33 | xyz1sw2,Gi1/1/1,xyz2mdf,Gi2/0/2
34 | xyz2mdf,Gi1/0/12,core1,Eth10/5
35 | xyz2mdf,Gi2/0/12,core2,Eth10/5
36 | xyz2mdf,Gi2/0/4,xyz3sw1,Te5/2
37 | xyz2mdf,Gi1/0/4,xyz3sw1,Te5/1
38 | xyz2mdf,Gi1/0/2,xyz1sw2,Gi2/1/1
39 | xyz2mdf,Gi2/0/3,xyz2sw1,Te5/2
40 | xyz2mdf,Gi1/0/3,xyz2sw1,Te5/1
41 | xyz2mdf,Gi2/0/2,xyz1sw2,Gi1/1/1
42 | xyz2mdf,Gi2/0/1,xyz1sw1,Te5/2
43 | xyz2mdf,Gi1/0/1,xyz1sw1,Te5/1
44 | xyz2mdf,Gi2/0/6,xyz4sw2,Gi1/1/2
45 | xyz2mdf,Gi1/0/6,xyz4sw2,Gi1/1/1
46 | xyz2mdf,Gi2/0/5,xyz4sw1,Te5/2
47 | xyz2mdf,Gi1/0/5,xyz4sw1,Te5/1
48 |
--------------------------------------------------------------------------------
/test/csv/supernets.csv:
--------------------------------------------------------------------------------
1 | cidr,role,description,secure
2 | 10.9.0.0/16,nac,NAC Registration,50
3 | 10.14.0.0/16,pci,Secure Utility Network,150
4 |
--------------------------------------------------------------------------------
/test/csv/vrfs.csv:
--------------------------------------------------------------------------------
1 | default,100,Default Routing Table
2 | guest,10,Guest Network
3 | outside,1,External Networks
4 | perim,25,Perimeter DMZ Network
5 | utility,150,Utility Network
6 | test,1,Test Empty VRF
7 |
--------------------------------------------------------------------------------
/test/first_import.sh:
--------------------------------------------------------------------------------
1 | #!/bin/sh
2 | ./ngupdate.py -v --dropDatabase
3 | sleep 2
4 | ./ngupdate.py -ivrf -v
5 | ./ngupdate.py -id -v
6 | ./ngupdate.py -ind -v
7 | ./ngupdate.py -v -inet --ignoreNew
8 | ./ngupdate.py -isnet -v
9 | ./ngupdate.py -ivlan -v --ignoreNew
10 | ./ngupdate.py -uvlan -v
11 | ./ngupdate.py -ifw -v
12 | ./ngupdate.py -v -ifile ./cypher/buildfw.cyp
13 | ./ngupdate.py -v -ifile ./cypher/constraints.cyp
14 | ./ngupdate.py -full -v
15 | ./ngupdate.py -v -ifile ./cypher/buildfw.cyp
16 | ./test/test_full.sh
17 |
--------------------------------------------------------------------------------
/test/ngclient.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python3
2 | #
3 | # Copyright (c) 2016 "Jonathan Yantis"
4 | #
5 | # This file is a part of NetGrph.
6 | #
7 | # This program is free software: you can redistribute it and/or modify
8 | # it under the terms of the GNU Affero General Public License, version 3,
9 | # as published by the Free Software Foundation.
10 | #
11 | # This program is distributed in the hope that it will be useful,
12 | # but WITHOUT ANY WARRANTY; without even the implied warranty of
13 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
14 | # GNU Affero General Public License for more details.
15 | #
16 | # You should have received a copy of the GNU Affero General Public License
17 | # along with this program. If not, see .
18 | #
19 | # As a special exception, the copyright holders give permission to link the
20 | # code of portions of this program with the OpenSSL library under certain
21 | # conditions as described in each individual source file and distribute
22 | # linked combinations including the program with the OpenSSL library. You
23 | # must comply with the GNU Affero General Public License in all respects
24 | # for all of the code used other than as permitted herein. If you modify
25 | # file(s) with this exception, you may extend this exception to your
26 | # version of the file(s), but you are not obligated to do so. If you do not
27 | # wish to do so, delete this exception statement from your version. If you
28 | # delete this exception statement from all source files in the program,
29 | # then also delete it in the license file.
30 | #
31 | #
32 | """
33 | NetGrph API Client
34 | """
35 | import sys
36 | import requests
37 | import json
38 | import nglib.ngtree.export
39 |
40 |
41 | user = 'yantisj'
42 | passwd = 'testapi'
43 | url = 'http://localhost:4096'
44 |
45 | output = 'JSON'
46 |
47 | if len(sys.argv) > 1:
48 | output = sys.argv[1]
49 |
50 | #qlist = ['/netgrph/api/v1.1/devs?group=MDC', '/netgrph/api/v1.1/devs?group=CON&full=1', \
51 | # '/netgrph/api/v1.1/devs/waringsw1']
52 |
53 | qlist = ['/netgrph/api/v1.1/vlans?vrange=200-205', '/netgrph/api/v1.1/devs?full=1&search=cr3.*']
54 |
55 |
56 | for q in qlist:
57 | r = requests.get(url + q, auth=(user, passwd), verify=False)
58 |
59 | if r.status_code == 200:
60 | response = r.json()
61 | nglib.ngtree.export.exp_ngtree(response, output)
62 | else:
63 | print("Request Error:", r.status_code, r.text)
64 |
--------------------------------------------------------------------------------
/test/nglib:
--------------------------------------------------------------------------------
1 | ../nglib/
--------------------------------------------------------------------------------
/test/pre-push.sh:
--------------------------------------------------------------------------------
1 | #!/bin/sh
2 | pylint3 ./nglib/ netgrph.py ngupdate.py > ./log/pylint.log
3 | less ./log/pylint.log
4 |
--------------------------------------------------------------------------------
/test/set_neo4j_password.sh:
--------------------------------------------------------------------------------
1 | #!/bin/sh
2 | #
3 | # Sets password to "your_passwd" for automated testing
4 | #
5 | curl -H "Content-Type: application/json" -X POST -d '{"password":"your_passwd"}' -u neo4j:neo4j http://localhost:7474/user/neo4j/password
6 |
--------------------------------------------------------------------------------
/test/test_full.sh:
--------------------------------------------------------------------------------
1 | #/bin/sh
2 |
3 | # Check for Ubuntu Python3
4 | export pytestcmd="py.test-3"
5 |
6 | # Revert to default pytest
7 | if ! type "$pytestcmd" 2> /dev/null; then
8 | export pytestcmd="py.test"
9 | fi
10 |
11 | if [ "$1" = "prod" ]; then
12 | $pytestcmd --resultlog=/tmp/pytest.log -k test_prod ngtest.py
13 | else
14 | $pytestcmd --resultlog=/tmp/pytest.log -k test_dev ngtest.py
15 | fi
16 |
17 |
--------------------------------------------------------------------------------
/test/test_queries.sh:
--------------------------------------------------------------------------------
1 | #/bin/sh
2 |
3 | export pytestcmd="py.test-3"
4 |
5 | if ! type "$pytestcmd" 2> /dev/null; then
6 | export pytestcmd="py.test"
7 | fi
8 |
9 | if [ "$1" = "prod" ]; then
10 | $pytestcmd --resultlog=/tmp/pytest.log ngtest.py::test_prod_queries
11 | else
12 | $pytestcmd --resultlog=/tmp/pytest.log ngtest.py::test_dev_queries
13 | fi
14 |
--------------------------------------------------------------------------------
/test/testapi.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python3
2 | # Copyright (c) 2016 Jonathan Yantis
3 | #
4 | # Permission is hereby granted, free of charge, to any person obtaining a copy of
5 | # this software and associated documentation files (the "Software"), to deal in
6 | # the Software without restriction, including without limitation the rights to
7 | # use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
8 | # the Software, and to permit persons to whom the Software is furnished to do so,
9 | # subject to the following conditions:
10 | #
11 | # The above copyright notice and this permission notice shall be included in all
12 | # copies or substantial portions of the Software.
13 | #
14 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
15 | # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
16 | # FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
17 | # COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
18 | # IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
19 | # CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
20 | #
21 | """
22 | Test the API Application
23 | """
24 | import os
25 | import builtins
26 | import re
27 | from base64 import b64encode
28 | import unittest
29 | import tempfile
30 | import time
31 |
32 | headers = {
33 | 'Authorization': 'Basic %s' % b64encode(b"testuser:testpass").decode("ascii")
34 | }
35 |
36 | # Default Config File Location
37 | config_file = '/etc/netgrph.ini'
38 | alt_config = './docs/netgrph.ini'
39 |
40 | # Test/Dev Config
41 | dirname = os.path.dirname(os.path.realpath(__file__))
42 | if re.search(r'\/dev/test$', dirname):
43 | config_file = './netgrphdev.ini'
44 |
45 | # Test configuration exists
46 | if not os.path.exists(config_file):
47 | if not os.path.exists(alt_config):
48 | raise Exception("Configuration File not found", config_file)
49 | else:
50 | config_file = alt_config
51 |
52 | # Import API
53 | builtins.apisrv_CONFIG = config_file
54 | import apisrv
55 |
56 | class APITestCase(unittest.TestCase):
57 | """ API Testing Cases"""
58 | def setUp(self):
59 | apisrv.app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///' + tempfile.mkstemp()[1]
60 | apisrv.app.config['TESTING'] = True
61 | self.app = apisrv.app.test_client()
62 |
63 | with apisrv.app.app_context():
64 | apisrv.db.create_all()
65 | try:
66 | apisrv.user.add_user("testuser", "testpass")
67 | except Exception:
68 | pass
69 |
70 | def tearDown(self):
71 | apisrv.db.drop_all()
72 |
73 | def test_index(self):
74 | print('Testing Connectivity..')
75 | rv = self.app.get('/')
76 | assert b'Authentication required' in rv.data
77 |
78 | def test_info(self):
79 | print('Testing Authentication..')
80 | rv = self.app.get('/netgrph/api/v1.0/info?test=GETTEST', headers=headers)
81 | assert b'GET test' in rv.data
82 |
83 | def test_paths(self):
84 | print("Testing Paths..")
85 | rv = self.app.get('/netgrph/api/v1.1/path?src=10.26.10.10&dst=10.34.10.10', headers=headers)
86 | assert b'L4-GW' in rv.data
87 | assert b'L3-PATH' in rv.data
88 | assert b'L3-GW' in rv.data
89 |
90 | rv = self.app.get('/netgrph/api/v1.1/spath?dst=mdc.*&src=hvt.*', headers=headers)
91 | assert b'"distance": 4' in rv.data
92 |
93 | def test_vlans(self):
94 | print("Testing VLANs..")
95 | rv = self.app.get('/netgrph/api/v1.1/vlans?vrange=250-300', headers=headers)
96 | assert b'"VLAN ID"' in rv.data
97 | rv = self.app.get('/netgrph/api/v1.1/vlans/200', headers=headers)
98 | assert b'"VLAN ID"' in rv.data
99 |
100 | def test_nets(self):
101 | print("Testing Nets..")
102 | rv = self.app.get('/netgrph/api/v1.1/nets?cidr=10.0.0.0-11', headers=headers)
103 | assert b'"VLAN"' in rv.data
104 |
105 |
106 | if __name__ == '__main__':
107 | unittest.main()
108 | time.sleep(0.25)
109 |
--------------------------------------------------------------------------------
/test/testrand.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python3
2 | """ Test Random IP's for errors"""
3 | import subprocess
4 |
5 | ip1 = open('/tmp/ip1.txt', 'r')
6 | ip2 = open('/tmp/ip2.txt', 'r')
7 |
8 | verbose = False
9 |
10 | for e1, e2 in zip(ip1, ip2):
11 | e1 = e1.strip()
12 | e2 = e2.strip()
13 | #print(e1, e2)
14 |
15 | cmd = './netgrph.py ' + e1 + ' ' + e2
16 |
17 | proc = subprocess.Popen(
18 | [cmd],
19 | stdout=subprocess.PIPE,
20 | shell=True,
21 | universal_newlines=True)
22 |
23 | (out, err) = proc.communicate()
24 |
25 | if err:
26 | print(cmd, err)
27 | elif out and verbose:
28 | print(out)
29 |
--------------------------------------------------------------------------------
/test/uwsgi-api.sh:
--------------------------------------------------------------------------------
1 | #!/bin/sh
2 | export NGLIB_config_file="./netgrphdev.ini"
3 | uwsgi --socket 0.0.0.0:9000 --protocol=http -w wsgi-api -H ./venv/
4 |
--------------------------------------------------------------------------------
/wsgi-api.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python3
2 | """ Run API Server as UWSGI App"""
3 |
4 | from apisrv import app as application
5 |
6 | if __name__ == "__main__":
7 | application.run()
8 |
--------------------------------------------------------------------------------