├── .gitignore ├── README.md ├── __init__.py ├── auditcreeper.py ├── auditdiff.py ├── confirm.py ├── exec_cmd.py ├── get_property.py ├── gifs ├── superloop_audit_diff_demo.gif ├── superloop_auditcreeper.gif ├── superloop_auditcreeper_demo.gif ├── superloop_gitops_operational_framework.png ├── superloop_host_exec_after_push_cfgs_demo.gif ├── superloop_host_exec_demo.gif ├── superloop_push_cfgs_demo.gif ├── superloop_push_local_demo.gif └── superloop_push_onscreen_demo.gif ├── initialize.py ├── lib ├── __init__.py ├── mediators │ ├── __init__.py │ ├── generic.py │ └── juniper.py └── objects │ ├── __init__.py │ └── basenode.py ├── mediator.py ├── modifydb.py ├── multithread.py ├── node_create.py ├── node_list.py ├── parse_cmd.py ├── processdb.py ├── pull_cfgs.py ├── push_acl.py ├── push_cfgs.py ├── push_local.py ├── push_regex.py ├── push_render.py ├── render.py ├── requirements.apt ├── requirements.txt ├── search.py ├── snmp.py ├── snmp_helper.py ├── ssh_connect.py └── superloop.py /.gitignore: -------------------------------------------------------------------------------- 1 | *.yaml 2 | *.jinja2 3 | *.pyc 4 | *.conf 5 | *.bak 6 | *.txt 7 | __pycache__ 8 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # superloop 2 | Inspired by the world's leading social media tech company (Facebook) for network automation, I have created my own version of the framework. 3 | 4 | ## Prerequisites 5 | 1. Python 3.6 or higher. 6 | 2. netmiko - A HUGE thanks and shout out to Kirk Byers for developing the library! 7 | 3. snmp_helper.py - module written by Kirk Byers (https://github.com/ktbyers/pynet/blob/master/snmp/snmp_helper.py). 8 | 4. ciscoconfparse - A library to help parse out Cisco (or similiar) CLI configs (https://pypi.org/project/ciscoconfparse/). 9 | 5. yaml - YAML is a human-readable data-serialization language (https://en.wikipedia.org/wiki/YAML). 10 | 6. libyaml-cpp-dev - C parser for yaml (xargs apt-get install < requirements.apt) 11 | 7. jinja2 - template engine for Python (https://jinja.palletsprojects.com/en/2.11.x/) 12 | 8. hvac - Python client for hashicorp vault (https://pypi.org/project/hvac/). 13 | 14 | ## Support 15 | 16 | |__Platform__|__audit diff__|__push cfgs__|__host exec__|__ssh__ |__node list__|__host add__|__host remove__|__host discover__|__push acl__|__pull cfgs__| 17 | |------------|:------------:|:-----------:|:-----------:|:------:|:-----------:|:----------:|:-------------:|:---------------:|:----------:|:------------| 18 | | Cisco IOS | x | x | x | x | x | x | x | x | - | x | 19 | | Cisco NXOS | x | x | x | x | x | x | x | x | - | x | 20 | | Cisco ASA | x | x | x | x | x | x | x | x | | x | 21 | | Juniper OS | x | x | x | x | x | x | x | x | | x | 22 | |F5 BigIP LTM| x | x | x | x | x | x | x | x | - | x | 23 | | Netscaler | x | x | x | x | x | x | x | x | - | x | 24 | 25 | ## Overview 26 | 27 | ![superloop gitops_operational_framework](https://github.com/superloopnetwork/superloop/blob/master/gifs/superloop_gitops_operational_framework.png) 28 | 29 | 30 | ## Install 31 | 32 | There are a few methods to install superloop but the easiest is the following: 33 | 34 | An appropriate install location would be in ```/usr/local/``` 35 | 36 | ``` 37 | $ cd /usr/local/ 38 | $ git clone https://github.com/superloopnetwork/superloop 39 | $ cd superloop/ 40 | $ pip3 install -r requirements.txt 41 | $ xargs apt-get install < requirements.apt 42 | ``` 43 | 44 | This will install superloop along with all required dependencies to the directory. 45 | 46 | IMPORTANT: To simplify the execution of superloop application, please do the following after installation. 47 | 48 | Create a symbolic link of 'superloop.py' and place it in '/usr/local/bin/'. Set the permission to 755. (replace python3.x with your correct python version) 49 | ``` 50 | $ ln -s /usr/local/superloop/superloop.py /usr/local/bin/superloop 51 | $ chmod 755 /usr/local/bin/superloop 52 | ``` 53 | Now uncomment the following code within ```/usr/local/bin/superloop``` near the top: 54 | ``` 55 | #import sys 56 | #sys.path.append('/usr/local/superloop') 57 | ``` 58 | So it looks like this . . . . 59 | ``` 60 | #!/usr/bin/python3 61 | import sys 62 | sys.path.append('/usr/local/superloop') 63 | from auditdiff import auditdiff 64 | from push_cfgs import push_cfgs 65 | ... 66 | .. 67 | . 68 | 69 | ``` 70 | This will set the system path of superloop to '/usr/local/superloop'. If you have superloop installed in another directory, change the path accordingly (replace python3.x with appropriate version). 71 | 72 | In Netmiko version 3.x by default is going to expect the configuration command to be echoed to the screen. This ensures Netmiko doesn't get out of sync with the underlying device (ex. keep sending configuration commands even though the remote device might be too slow and buffering them). 73 | 74 | We will need to turn off command verification in netmiko base_connection.py file: 75 | ``` 76 | vi /usr/local/lib/python3.7/dist-packages/netmiko/base_connection.py 77 | ``` 78 | Search for the function 'send_config_set' and change 'cmd_verify=True' to 'cmd_verify=False' like this: 79 | ``` 80 | def send_config_set( 81 | self, 82 | config_commands=None, 83 | exit_config_mode=True, 84 | delay_factor=1, 85 | max_loops=150, 86 | strip_prompt=False, 87 | strip_command=False, 88 | config_mode_command=None, 89 | cmd_verify=False, 90 | enter_config_mode=True, 91 | ): 92 | ``` 93 | We also need to disable `enqueue` in ciscoconfparse package (dependancy of loguru package) as this stalls/hangs/locks threading in superloop. 94 | ``` 95 | vi /usr/local/lib/python3.7/dist-packages/ciscoconfparse/ccp_util.py 96 | ``` 97 | Search for `enqueue=True,` and change it to `enqueue=False,' 98 | 99 | Before we begin, I've constructed this application for easy database management by utilizing the power of YAML files. There are a combination of two YAML files that require management (default path is ~/database/): 100 | 101 | 1. nodes.yaml 102 | 2. templates.yaml 103 | 104 | The nodes.yaml holds all inventory of devices in the network. The templates.yaml file are mappings of templates so superloop knows which template is provisioned. The state of each template is appended at the end of the mapping to either 'disabled' or 'enabled' pushing of template. This acts as a safety feature. Zero to low confidence templates should never be pushed in a production environment as it may cause impact. However, all templates can be audited or rendered as there is no impact involved; this does not require the state of the template(s) to be flipped to 'enabled' 105 | ``` 106 | root@devvm:~# cat superloop_code/database/templates.yaml 107 | --- 108 | - hardware_vendor: cisco 109 | type: firewall 110 | opersys: asa 111 | templates: 112 | - ~/superloop_code/templates/hardware_vendors/cisco/asa/firewall/base.jinja2: disabled 113 | - ~/superloop_code/templates/hardware_vendors/cisco/asa/firewall/object-groups.jinja2: disabled 114 | - ~/superloop_code/templates/hardware_vendors/cisco/asa/firewall/logging.jinja2: enabled 115 | - hardware_vendor: cisco 116 | type: router 117 | opersys: ios 118 | templates: 119 | - ~/superloop_code/templates/hardware_vendors/cisco/ios/router/base.jinja2: enabled 120 | - hardware_vendor: cisco 121 | type: switch 122 | opersys: ios 123 | templates: 124 | - ~/superloop_code/templates/hardware_vendors/cisco/ios/switch/base.jinja2: disabled 125 | - ~/superloop_code/templates/hardware_vendors/cisco/ios/switch/logging.jinja2: enabled 126 | - ~/superloop_code/templates/hardware_vendors/cisco/ios/switch/service.jinja2: disabled 127 | - ~/superloop_code/templates/hardware_vendors/cisco/ios/switch/dhcp.jinja2: enabled 128 | - ~/superloop_code/templates/hardware_vendors/cisco/ios/switch/snmp.jinja2: enabled 129 | - ~/superloop_code/templates/hardware_vendors/cisco/ios/switch/interfaces.jinja2: disabled 130 | - hardware_vendor: juniper 131 | type: vfirewall 132 | opersys: junos 133 | templates: 134 | - ~/superloop_code/templates/hardware_vendors/juniper/junos/vfirewall/routing-instances.jinja2: disabled 135 | - ~/superloop_code/templates/hardware_vendors/juniper/junos/vfirewall/routing-options.jinja2: disabled 136 | - ~/superloop_code/templates/hardware_vendors/juniper/junos/vfirewall/system.jinja2: enabled 137 | - ~/superloop_code/templates/hardware_vendors/juniper/junos/vfirewall/interfaces.jinja2: disabled 138 | - ~/superloop_code/templates/hardware_vendors/juniper/junos/vfirewall/protocols.jinja2: disabled 139 | - hardware_vendor: synology 140 | type: nas 141 | opersys: busybox 142 | templates: 143 | - ~/superloop_code/templates/hardware_vendors/synology/busybox/base.jinja2: enabled 144 | ``` 145 | 146 | ## Credentials 147 | 148 | Credentials used to connect to nodes are via the OS environment varilable, $USER. It will prompt you for your password 149 | ``` 150 | export USERNAME=username 151 | ``` 152 | 153 | ## Hierarchy 154 | 155 | You'll noticed the superloo_code/ and superloop/ source code are completely segregated by different repositories. superloop_code/ repo can be found here and should be cloned to the home directory of the user as that is where superloop references to. Hourly backups should be stored in the superloop_code/backup-configs directory via CI/CD. superloop_code/database are where the inventory of the devices, templates (reference) files are stored. superloop_code/templates are where all the templates are stored. The hierarchy is structured based on vendor, OS and device type when it comes to templates. That's because different vendors and OS have different syntaxes. ex. Cisco IOS have different syntaxes than Cisco NXOS. 156 | ``` 157 | root@devvm:~# tree superloop_code/ 158 | superloop_code/ 159 | ├── database 160 | │   ├── nodes.yaml 161 | │   ├── policies.yaml 162 | │   ├── templates.yaml 163 | │   └── templates.yaml.bak 164 | ├── policy 165 | │   ├── APPLICATIONS.net 166 | │   ├── cisco 167 | │   │   └── ios 168 | │   │   └── firewall 169 | │   │   └── base_policy.json 170 | │   ├── juniper 171 | │   │   └── junos 172 | │   │   └── vfirewall 173 | │   │   └── policy.json 174 | │   ├── NETWORKS.net 175 | │   ├── SERVICES.net 176 | │   ├── SOURCE_DEVICE.net 177 | │   ├── SOURCE_USER.net 178 | │   ├── :w 179 | │   └── ZONES.net 180 | └── templates 181 | ├── hardware_vendors 182 | │   ├── cisco 183 | │   │   ├── asa 184 | │   │   │   └── firewall 185 | │   │   │   ├── base.jinja2 186 | │   │   │   ├── logging.jinja2 187 | │   │   │   ├── object-groups.jinja2 188 | │   │   │   └── snmp.jinja2 189 | │   │   ├── ios 190 | │   │   │   ├── router 191 | │   │   │   │   └── base.jinja2 192 | │   │   │   └── switch 193 | │   │   │   ├── aaa.jinja2 194 | │   │   │   ├── base.jinja2 195 | │   │   │   ├── base.jinja2.bak 196 | │   │   │   ├── crypto.jinja2 197 | │   │   │   ├── cs_vserver.jinja2 198 | │   │   │   ├── dhcp.jinja2 199 | │   │   │   ├── interfaces.jinja2 200 | │   │   │   ├── logging.jinja2 201 | │   │   │   ├── service.jinja2 202 | │   │   │   └── snmp.jinja2 203 | │   │   └── nxos 204 | │   ├── f5 205 | │   │   └── bigip 206 | │   └── juniper 207 | │   └── junos 208 | │   └── vfirewall 209 | │   ├── interfaces.jinja2 210 | │   ├── protocols.jinja2 211 | │   ├── routing-instances.jinja2 212 | │   ├── routing-options.jinja2 213 | │   └── system.jinja2 214 | └── standards 215 | └── common.jinja2 216 | 217 | ``` 218 | Let's look at a simple Cisco platform_name jinja2 template as an example. 219 | 220 | ``` 221 | {# audit_filter = ['snmp-server (?!user).*'] #} 222 | {%- import 'global.jinja2' as global -%} 223 | {%- import 'datacenter.jinja2' as dc -%} 224 | {%- import 'environment.jinja2' as env -%} 225 | {# %- import node.name ~ '.jinja2' as device -% #} 226 | snmp-server community {{ secrets['community_1'] }} group network-operator 227 | snmp-server community {{ secrets['community_2'] }} group network-operator 228 | snmp-server community {{ secrets['community_3'] }} group network-operator 229 | snmp-server location {{ dc.snmp.location }} 230 | ``` 231 | Notice there is a section called 'audit_filter' at the top of file. This audit filter should be included in all templates of Cisco and Citrix Netscaler. It accepts a regular expression. This tells superloop which lines to look for and compare against when rendering the configs. In other words, superloop will look for only lines that begin with 'snmp-server' and anything else trailing but exclude 'user' as the second piece of string. If you have additional lines that you want superloop to look at, simply append strings seperated by a comma like so... 232 | ``` 233 | ['snmp-server (?!user).*','hello','world'] 234 | ``` 235 | There are a few import statements that you may need to include depending on the variables you need to use. The files are organized based on the logic and reference to their geographic location. 236 | 237 | global.jinja2 maps to ~/superloop_code/templates/standards/global.jinja2 # all global variables will stored in this file. 238 | 239 | datacenter.jinja2 maps to ~/superloop_code/templates/datacenter//datacenter.jinja2 # all variables pertaining to datacenter/region specific will be stored in this file. 240 | 241 | environment.jinja2 maps to ~/superloop_code/templates/datacenter//prod/environment.jinja2 # all variables pertaining to the different region and environment will be stored in this file. 242 | 243 | NEVER include any secrets (static) within any templates as they will be exposed in clear text and visible in version control. Instead we want to mask our secrets by storing them in hashicorp and calling them in this fashion: 244 | ``` 245 | {{ secrets['community_1'] }} 246 | ``` 247 | Mappings can be found here: Networking > prod > secrets in vault. Every time when a template is being rendered for output, superloop will authenticate with vault. If successful, it then queries the requested secret. The secret is returned to superloop and is then pushed to jinja for output. With this method, no secrets are exposed in any files. 248 | 249 | You may also have a template that consist of one or several levels deep like so... 250 | ``` 251 | {# audit_filter = ['ip dhcp .*'] #} 252 | ip dhcp excluded-address 10.50.80.1 253 | ip dhcp ping packets 5 254 | ! 255 | ip dhcp pool DATA 256 | network 10.10.20.0 255.255.255.0 257 | default-router 10.10.20.1 258 | dns-server 8.8.8.8 259 | ``` 260 | Look at 'ip dhcp pool DATA'. The next line of config has an indentation. The parent is considered 'ip dhcp pool DATA' and the child are anything below that section. superloop is intelligent enough to parse the remaining 3 lines of configs without having to include it into the audit_filter. 261 | 262 | Now that I have explained the basic operations, onto the fun stuff! 263 | 264 | ## superloop host add 265 | When you add a device, every attribute of the node will be discovered automatically so there is no need to populate it manually. 266 | ``` 267 | root@devvm:~# superloop host add 10.202.1.7 268 | + SNMP discovery successful. 269 | + New node appended to database. 270 | ``` 271 | 272 | ## superloop host remove 273 | To remove a node, simply execute a 'superloop host remove ': 274 | ``` 275 | wailit.loi@pc-netauto-001:~$ superloop host remove 10.202.1.7 276 | - Node successfully removed from database. 277 | ``` 278 | 279 | ## superloop node list 280 | To verify the device attributes: 281 | 282 | ``` 283 | root@devvm:~# superloop node list pt-switch-001 284 | [ 285 | { 286 | "created_at": "2022-09-12 14:02:10" 287 | "created_by": "wailit.loi" 288 | "data": { 289 | "managed_configs": { 290 | "logging.jinja2" 291 | "ntp.jinja2" 292 | "snmp.jinja2" 293 | } 294 | } 295 | "domain_name": "null" 296 | "environment": "prod" 297 | "hardware_vendor": "cisco" 298 | "lifecycle_status": "null" 299 | "location_name": "toronto" 300 | "mgmt_con_ip4": "null" 301 | "mgmt_ip4": "10.202.1.7" 302 | "mgmt_oob_ip4": "null" 303 | "mgmt_snmp_community4": "null" 304 | "name": "core1.leaf.yyz.demo.domain.name" 305 | "opersys": "ios" 306 | "platform_name": "WS-C3750X-48" 307 | "role_name": "datacenter-switch" 308 | "serial_num": "FDO1629R0JL" 309 | "software_image": "null" 310 | "software_version": "null" 311 | "status": "online" 312 | "type": "switch" 313 | "updated_at": "null" 314 | "updated_by": "null" 315 | } 316 | ] 317 | ``` 318 | 319 | ## superloop host update 320 | 321 | Notice the 'name' or hostname of the device has the domain appended because the 'ip domain-name domain.name' is configured. If the domain name is not required, superloop has the ability to modify the database attribute from cli: 322 | ``` 323 | root@devvm:~# superloop host update core1.leaf.yyz.domain.name --help 324 | usage: superloop host update [-h] [-a ATTRIBUTE] [-am AMEND] node 325 | positional arguments: 326 | node 327 | optional arguments: 328 | -h, --help show this help message and exit 329 | -a ATTRIBUTE, --attribute ATTRIBUTE 330 | Specify the attribute that requires updating 331 | -am AMEND, --amend AMEND 332 | The value that is being amended 333 | 334 | root@devvm:~# superloop host update core1.leaf.demo.domain.name -a name -am core1.leaf.yyz.demo 335 | Please confirm you would like to change the value from core1.leaf.yyz.demo.domain.name : name : core1.leaf.yyz.demo.domain.name to core1.leaf.yyz.demo.domain.name : name : core1.leaf.yyz.demo. [y/N]: y 336 | + Amendment to database was successful. 337 | ``` 338 | We can take a look at the 'node list' feature to verify the 'name' attribute has changed: 339 | ``` 340 | root@devvm:~# superloop node list core1.leaf.yyz.demo 341 | [ 342 | { 343 | "created_at": "2022-09-12 14:02:10" 344 | "created_by": "wailit.loi" 345 | "data": { 346 | "managed_configs": { 347 | "logging.jinja2" 348 | "ntp.jinja2" 349 | "snmp.jinja2" 350 | } 351 | } 352 | "domain_name": "null" 353 | "environment": "prod" 354 | "hardware_vendor": "cisco" 355 | "lifecycle_status": "null" 356 | "location_name": "telecity" 357 | "mgmt_con_ip4": "null" 358 | "mgmt_ip4": "10.202.1.7" 359 | "mgmt_oob_ip4": "null" 360 | "mgmt_snmp_community4": "null" 361 | "name": "core1.leaf.yyz.demo" 362 | "opersys": "ios" 363 | "platform_name": "WS-C3750X-48" 364 | "role_name": "datacenter-switch" 365 | "serial_num": "FDO1629R0JL" 366 | "software_image": "null" 367 | "software_version": "null" 368 | "status": "online" 369 | "type": "switch" 370 | "updated_at": "2022-09-12 14:13:36" 371 | "updated_by": "wailit.loi" 372 | } 373 | ] 374 | ``` 375 | When it comes to templating, we are able to call these attributes directly and make logical decisions based on the value. We'll discuss more later on in this article... 376 | 377 | ## superloop push render 378 | The 'push render' function, simply renders a template created in jinja2. Ensure the template is provisioned in the ~/superloop_code/database/templates.yaml file so superloop understands which template(s) is/are loaded. If we want to render a template, we simply execute 'superloop push render --node --file '. --node accepts a regular expression to match (multiple) node(s) and it can be as granular as you wish. Ex. matching an entire datacenter and/or device type. If there is no '–file' flag supplied, ALL templates for the device specific type will be rendered. 379 | ``` 380 | root@devvm:~# superloop push render --node core.*sw.*.yyz.*demo --file logging 381 | core1.sw.yyz.demo 382 | /root/superloop_code/templates/hardware_vendors/cisco/nxos/switch/logging.jinja2 383 | logging message interface type ethernet description 384 | logging logfile messages 6 size 32768 385 | logging server 10.100.10.53 386 | logging server 10.100.2.40 387 | logging timestamp milliseconds 388 | logging monitor 3 389 | no logging rate-limit 390 | 391 | core2.sw.yyz.demo 392 | /root/superloop_code/templates/hardware_vendors/cisco/nxos/switch/logging.jinja2 393 | logging message interface type ethernet description 394 | logging logfile messages 6 size 32768 395 | logging server 10.100.10.53 396 | logging server 10.100.2.40 397 | logging timestamp milliseconds 398 | logging monitor 3 399 | no logging rate-limit 400 | ``` 401 | 402 | ## superloop audit diff 403 | This function was designed to compare against the jinja2 templates with your running-configurations/candidate-configurations to see if they are according to standards. You could imagine if you had hundreds, if not thousands of devices to maintain, standardization would be a nightmare without some form of auditing/automation tool. To paint you an example, say one day, an employee decides to make an unauthorized manual configuration change on a switch. No one knows about it or what they did. 'superloop' is able to dive into all devices and see if there were any discrepancies against the template as that is considered the trusted source. 'superloop' is then able to determine what was exactly modified or changed. Whatever was configured would essentially be negated automatically. This works the other way around as well. If configuration(s) on a device(s) does not have the standard rendered configs from the template (configs removed), superloop will determine they are missing and you may proceed to remediate by pushing the rendered configs. 'audit diff' will audit against ONE or ALL templates belonging to the matched device(s) from the query. If you want to audit against ONE template, simply include the option '--file ' (exclude extension .jinja2). If you want to audit against ALL templates belonging to the matched device(s) query, do not include the '--file' option. 404 | 405 | ``` 406 | root@devvm:~# superloop audit diff -n pc.*test -f snmp 407 | Password: 408 | [>] complete [0:00:10.911552] 409 | 410 | Only in the device: - 411 | Only in the generated config: + 412 | pc-n9ktest-001 413 | /root/superloop_code/templates/hardware_vendors/cisco/nxos/switch/snmp.jinja2 414 | - snmp-server community helloworld group network-operator 415 | + snmp-server location coresite 416 | ``` 417 | 418 | ## superloop push cfgs 419 | The 'push cfgs' function simply pushes the template(s) to the specified node(s). For Cisco, Citrix, F5 and Palo Alto devices, a debug output will be shown with a list of commands (if any) of what will be sent first before user commits to push. From the below example, you can see which templates are enabled for pushing, represented by [>] vs. which templates are disabled, represented by [x]. The state of the template can be controlled in the ~/superloop_code/database/templates.yaml file. As a safety, we disable any templates that we are not confident in pushing. If enabled, superloop will auto remediate those template(s). Please use with caution as it can cause severe impact. Two phases happens when pushing templates of Cisco, Citrix, F5 and Palo Alto devices. First, superloop performs an audit diff. It will check to see what configs are missing or removed. Second, it will encapsulate the necessary configs and prepare it for pushing. If the device has no diffs, then no configs will be pushed to the device. The output of session when pushing will be displayed so users can see what happens behind the scenes. 420 | 421 | ``` 422 | root@devvm:~# superloop push cfgs --node p.*nxs 423 | [x] core1.sw.yyz.demo ; base.jinja2 424 | [x] core1.sw.yyz.demo ; base.jinja2 425 | [x] core2.sw.yyz.demo ; base.jinja2 426 | [x] core2.sw.yyz.demo ; base.jinja2 427 | [x] core1.leaf.yyz.demo ; base.jinja2 428 | [x] core2.leaf.yyz.demo ; base.jinja2 429 | [x] core3.leaf.yyz.demo ; base.jinja2 430 | [>] core1.sw.yyz.demo ; logging.jinja2 431 | [>] core1.sw.yyz.demo ; ntp.jinja2 432 | [>] core1.sw.yyz.demo ; snmp.jinja2 433 | [>] core1.sw.yyz.demo ; logging.jinja2 434 | [>] core1.sw.yyz.demo ; ntp.jinja2 435 | [>] core1.sw.yyz.demo ; snmp.jinja2 436 | [>] core2.sw.yyz.demo ; logging.jinja2 437 | [>] core2.sw.yyz.demo ; ntp.jinja2 438 | [>] core2.sw.yyz.demo ; snmp.jinja2 439 | [>] core2.sw.yyz.demo ; logging.jinja2 440 | [>] core2.sw.yyz.demo ; ntp.jinja2 441 | [>] core2.sw.yyz.demo ; snmp.jinja2 442 | [>] core1.leaf.yyz.demo ; logging.jinja2 443 | [>] core1.leaf.yyz.demo ; ntp.jinja2 444 | [>] core1.leaf.yyz.demo ; snmp.jinja2 445 | [>] core2.leaf.yyz.demo ; logging.jinja2 446 | [>] core2.leaf.yyz.demo ; ntp.jinja2 447 | [>] core2.leaf.yyz.demo ; snmp.jinja2 448 | [>] core3.leaf.yyz.demo ; logging.jinja2 449 | [>] core3.leaf.yyz.demo ; ntp.jinja2 450 | [>] core3.leaf.yyz.demo ; snmp.jinja2 451 | + complete [0:00:11.403772] 452 | 453 | [DEBUG] 454 | { 455 | "core2.leaf.yyz.demo": [ 456 | [ 457 | "ntp logging" 458 | ] 459 | ], 460 | "core3.leaf.yyz.demo": [ 461 | [ 462 | "ntp logging" 463 | ] 464 | ] 465 | } 466 | config term 467 | Enter configuration commands, one per line. End with CNTL/Z. 468 | 469 | core2.leaf.yyz.demo(config)# ntp logging 470 | 471 | core2.leaf.yyz.demo(config)# end 472 | 473 | core2.leaf.yyz.demo# 474 | config term 475 | Enter configuration commands, one per line. End with CNTL/Z. 476 | 477 | core3.leaf.yyz.demo(config)# ntp logging 478 | 479 | core3.leaf.yyz.demo(config)# end 480 | 481 | core3.leaf.yyz.demo# 482 | + complete [0:00:14.664805] 483 | ``` 484 | 485 | ## superloop push local 486 | 487 | The 'push local' command allows you to push configs that are stored in a text file (home directory). This feature is useful when performing migrations. For example, if we wanted to drain/undrain traffic from one node, we could pre-configure the set of commands in the text file. At the time of migration, we can push the configs to the selected nodes. This method would eliminate any human error in the process. 488 | 489 | ## superloop ssh 490 | SSH feature allows us to quickly search up node(s) via regular expression and establish a SSH session with the device. This is useful when you have thousands of nodes in the network and memorizing IP addresses is simply not an option. 491 | ``` 492 | root@devvm:~# superloop ssh p.*lb.*act 493 | id name address platform 494 | 1 core1.lb.yyz.demo.active 10.10.10.1 citrix 495 | 2 core1.lb.sfo.demo.active 10.10.10.2 citrix 496 | 3 core1.lb.sin.demo.active 10.10.10.3 citrix 497 | ``` 498 | ## superloop host exec 499 | The 'host exec' features allow you to execute a command on the device(s) without requiring you to log in manually one by one. This feature is extremely useful at times when you want to check status across multi devices. 500 | ``` 501 | root@devvm:~# superloop host exec "show ip interface brief" --node core.*sw.*yyz.*demo 502 | Password: 503 | core1.sw.yyz.demo: IP Interface Status for VRF "default"(1) 504 | core1.sw.yyz.demo: Interface IP Address Interface Status 505 | core1.sw.yyz.demo: Vlan201 10.201.1.22 protocol-up/link-up/admin-up 506 | core1.sw.yyz.demo: 507 | 508 | core2.sw.yyz.demo: IP Interface Status for VRF "default"(1) 509 | core2.sw.yyz.demo: Interface IP Address Interface Status 510 | core2.sw.yyz.demo: Vlan201 10.201.1.23 protocol-up/link-up/admin-up 511 | core2.sw.yyz.demo: 512 | 513 | [>] Complete [0:00:08.375958] 514 | ``` 515 | -------------------------------------------------------------------------------- /__init__.py: -------------------------------------------------------------------------------- 1 | from . import auditcreeper 2 | from . import auditdiff 3 | from . import basenode 4 | from . import confirm_push 5 | from . import directory 6 | from . import exec_cmd 7 | from . import get_property 8 | from . import initialize 9 | from . import mediator 10 | from . import modifydb 11 | from . import multithread 12 | from . import node_create 13 | from . import node_list 14 | from . import parse_commands 15 | from . import port 16 | from . import processdb 17 | from . import pull_acl 18 | from . import pull_cfgs 19 | from . import push_cfgs 20 | from . import push_config 21 | from . import push_local 22 | from . import push_render 23 | from . import remediate 24 | from . import render 25 | from . import search 26 | from . import snmp 27 | from . import snmp_helper 28 | from . import ssh_connect 29 | from . import superloop 30 | -------------------------------------------------------------------------------- /auditcreeper.py: -------------------------------------------------------------------------------- 1 | """ 2 | This module executes auditcreeper, a continous auditing and remediating cycle. 3 | """ 4 | from processdb import process_nodes 5 | from processdb import process_templates 6 | from search import search_node 7 | from search import search_template 8 | from mediator import mediator 9 | from node_create import node_create 10 | from multithread import multithread_engine 11 | import threading 12 | import os 13 | import initialize 14 | 15 | def auditcreeper(): 16 | argument_node = '.+' 17 | auditcreeper = True 18 | commands = initialize.configuration 19 | output = True 20 | redirect = [] 21 | template_list = [] 22 | with_remediation = True 23 | """ 24 | :param argument_node: Argument accepted as regular expression. 25 | :type augument_node: str 26 | 27 | :param auditcreeper: When auditcreeper is active/non-active. 28 | :type auditcreeper: bool 29 | 30 | :param commands: Referenced to global variable commands which keeps track of all commands per node. 31 | :type commands: list 32 | 33 | :param output: Flag to output to stdout. 34 | :type ext: bool 35 | 36 | :param redirect: A list of which method superloop will access. This variable is sent to the multithread_engine. Each element is a redirect per node. 37 | :type alt_key_file: list 38 | 39 | :param template_list_original: Take a duplicate copy of template_list 40 | :type template_list_original: list 41 | 42 | :param with_remediation: Current function to remediate or not remediate. 43 | :type ext: bool 44 | """ 45 | initialize.variables() 46 | redirect.append('push_cfgs') 47 | os.system('clear') 48 | node_object = process_nodes() 49 | node_template = process_templates() 50 | match_node = search_node(argument_node,node_object) 51 | match_template = search_template(template_list,match_node,node_template,node_object,auditcreeper) 52 | """ 53 | :param node_object: All node(s) in the database with all attributes. 54 | :type node_object: list 55 | 56 | :param node_template: All templates based on hardware_vendor and device type. 57 | :type node_template: list 58 | 59 | :param match_node: Nodes that matches the arguements passed in by user. 60 | :type match_node: list 61 | 62 | :param match_template: Return a list of 'match' and/or 'no match'. 63 | :type match_template: list 64 | """ 65 | node_create(match_node,node_object) 66 | mediator(template_list,node_object,auditcreeper,output,with_remediation) 67 | for index in initialize.element: 68 | redirect.append('push_cfgs') 69 | multithread_engine(initialize.ntw_device,redirect,commands) 70 | threading.Timer(5.0, auditcreeper).start() 71 | 72 | if __name__ == "__main__": 73 | auditcreeper() 74 | -------------------------------------------------------------------------------- /auditdiff.py: -------------------------------------------------------------------------------- 1 | """ 2 | This module controls the pushing of the templates. node_object is a list of dictionary compiled by the processdb module. 3 | It processes the information from the nodes.yaml file and stores it into a list of dictionary. 4 | """ 5 | import initialize 6 | from lib.objects.basenode import BaseNode 7 | from processdb import process_nodes 8 | from processdb import process_policies 9 | from processdb import process_templates 10 | from search import search_node 11 | from search import search_policy 12 | from search import search_template 13 | from mediator import mediator 14 | from node_create import node_create 15 | from confirm import confirm 16 | 17 | def auditdiff(args): 18 | argument_node = args.node 19 | argument_file = args.file 20 | argument_policy = args.policy 21 | auditcreeper = False 22 | commands = initialize.configuration 23 | ext = { 24 | 'jinja':'.jinja2', 25 | 'json':'.json' 26 | } 27 | output = True 28 | push_cfgs = False 29 | redirect = [] 30 | safe_push_list = [] 31 | with_remediation = False 32 | """ 33 | :param argument_node: Argument accepted as regular expression. 34 | :type augument_node: str 35 | 36 | :param auditcreeper: When auditcreeper is active/non-active. 37 | :type auditcreeper: bool 38 | 39 | :param commands: Referenced to global variable commands which keeps track of all commands per node. 40 | :type commands: list 41 | 42 | :param ext: File extention 43 | :type ext: str 44 | 45 | :param output: Flag to output to stdout. 46 | :type ext: bool 47 | 48 | :param push_cfgs: This flag is to determine if a push is required for Cisco like platforms. Juniper will continue to push configs no matter if there are no diffs. 49 | :type ext: bool 50 | 51 | :param redirect: A list of which method superloop will access. This variable is sent to the multithread_engine. Each element is a redirect per node. 52 | :type alt_key_file: list 53 | 54 | :param safe_push_list: A list of enable/disabled strings. This corresponds to templates that are safe to push (enable) vs. templates that are not safe to push (disabled). 55 | :type ext: list 56 | 57 | :param with_remediation: Current function to remediate or not remediate. 58 | :type ext: bool 59 | 60 | :param node_object: All node(s) in the database with all attributes. 61 | :type node_object: list 62 | 63 | :param node_template: All templates based on hardware_vendor and device type. 64 | :type node_template: list 65 | 66 | :param match_node: Nodes that matches the arguements passed in by user. 67 | :type match_node: list 68 | 69 | :param match_template: Return a list of 'match' and/or 'no match'. 70 | :type match_template: list 71 | """ 72 | redirect.append('exec_cmd') 73 | node_object = process_nodes() 74 | match_node = search_node(argument_node,node_object) 75 | if argument_policy is not None: 76 | policy = argument_policy + ext['json'] 77 | policy_list = [] 78 | policy_list.append(policy) 79 | node_policy = process_policies() 80 | match_policy = search_policy(policy_list,safe_push_list,match_node,node_policy,node_object,auditcreeper,push_cfgs) 81 | policy_list_original = policy_list[:] 82 | policy_list_copy = policy_list 83 | if len(match_policy) != 0: 84 | policy_list = policy_list_copy 85 | if len(match_node) == 0: 86 | print('+ No matching node(s) found in database.') 87 | exit() 88 | elif 'NO MATCH' in match_policy: 89 | print('+ No matching policy(ies) found in database.') 90 | exit() 91 | else: 92 | node_create(match_node,node_object) 93 | mediator(args,policy_list,node_object,auditcreeper,output,with_remediation) 94 | exit() 95 | if argument_file is None and argument_policy is None: 96 | template_list = [] 97 | auditcreeper = True 98 | else: 99 | template = args.file + ext['jinja'] 100 | template_list = [] 101 | template_list.append(template) 102 | node_template = process_templates() 103 | match_template = search_template(template_list,safe_push_list,match_node,node_template,node_object,auditcreeper,push_cfgs) 104 | if len(match_node) == 0: 105 | print('[x] No matching nodes found in database.') 106 | print('') 107 | exit() 108 | elif 'NO MATCH' in match_template or len(match_template) == 0: 109 | print('[x] No matching templates found in database.') 110 | print('') 111 | exit() 112 | else: 113 | node_create(match_node,node_object) 114 | mediator(args,template_list,node_object,auditcreeper,output,with_remediation) 115 | # if len(initialize.configuration) == 0: 116 | # pass 117 | # else: 118 | # if remediation: 119 | # confirm(redirect,commands) 120 | 121 | return None 122 | -------------------------------------------------------------------------------- /confirm.py: -------------------------------------------------------------------------------- 1 | """ 2 | This module controls the confirmation of pushing and pulling function 3 | from the user. 4 | """ 5 | import initialize 6 | from multithread import multithread_engine 7 | 8 | def confirm(redirect,commands,authentication): 9 | index = 0 10 | if(redirect[index] == 'push_cfgs'): 11 | try: 12 | check = str(input("Push configs? [y/N]: ")).strip() 13 | except KeyboardInterrupt as error: 14 | print('[>] Terminating...') 15 | exit() 16 | else: 17 | check = str(input("Pull configs? [y/N]: ")).strip() 18 | try: 19 | if check[0] == 'y': 20 | multithread_engine(initialize.ntw_device,redirect,commands,authentication) 21 | elif check[0] == 'N': 22 | return False 23 | else: 24 | print("RuntimeError: aborted at user request") 25 | except Exception as error: 26 | print("ExceptionError: an exception occured") 27 | print(error) 28 | 29 | return None 30 | -------------------------------------------------------------------------------- /exec_cmd.py: -------------------------------------------------------------------------------- 1 | """ 2 | This module executes user requested commands to devices. 3 | """ 4 | import initialize 5 | from lib.objects.basenode import BaseNode 6 | from processdb import process_nodes 7 | from search import search_node 8 | from search import node_element 9 | from node_create import node_create 10 | from multithread import multithread_engine 11 | 12 | def exec_cmd(args): 13 | argument_command = args.command 14 | argument_node = args.node 15 | redirect = [] 16 | authentication = True 17 | """ 18 | :param argument_command: Referenced to global variable commands which keeps track of all commands per node. 19 | :type commands: list 20 | 21 | :param argument_node: Argument accepted as regular expression. 22 | :type augument_node: str 23 | 24 | :param redirect: A list of which method superloop will access. This variable is sent to the multithread_engine. Each element is a redirect per node. 25 | :type alt_key_file: list 26 | """ 27 | redirect.append('exec_cmd') 28 | try: 29 | if argument_command == 'reload' or argument_command == 'reboot': 30 | print("superloopError: command not supported") 31 | return False 32 | else: 33 | node_object = process_nodes() 34 | match_node = search_node(argument_node,node_object) 35 | """ 36 | :param node_object: All node(s) in the database with all attributes. 37 | :type node_object: list 38 | 39 | :param match_node: Nodes that matches the arguements passed in by user. 40 | :type match_node: list 41 | """ 42 | if len(match_node) == 0: 43 | print('+ No matching node(s) found in database.') 44 | print() 45 | else: 46 | node_element(match_node,node_object) 47 | node_create(match_node,node_object) 48 | multithread_engine(initialize.ntw_device,redirect,argument_command,authentication) 49 | except Exception as error: 50 | print("ExceptionError: an exception occured") 51 | print(error) 52 | 53 | return None 54 | -------------------------------------------------------------------------------- /get_property.py: -------------------------------------------------------------------------------- 1 | """ 2 | This module holds properties that superloop is required to retrieve. 3 | """ 4 | import datetime 5 | import hvac 6 | import os 7 | import socket 8 | import time 9 | 10 | def get_home_directory(): 11 | home_directory = os.getenv('HOME') 12 | 13 | return home_directory 14 | 15 | def get_real_path(): 16 | real_path = os.path.dirname(os.path.realpath(__file__)) 17 | 18 | return real_path 19 | 20 | def get_port(node_object,element,ssh_id): 21 | if node_object[element[ssh_id]]['type'] == 'switch': 22 | port = '22' 23 | elif node_object[element[ssh_id]]['type'] == 'nas': 24 | port = '2222' 25 | else: 26 | port = '22' 27 | 28 | return port 29 | 30 | def get_type(name): 31 | if('fw' in name): 32 | device_type = 'firewall' 33 | elif('rt' in name): 34 | device_type = 'router' 35 | elif('sw' in name): 36 | device_type = 'switch' 37 | 38 | return device_type 39 | 40 | def get_template_directory(hardware_vendor,opersys,device_type): 41 | """ 42 | This will return the appropreiate directory based on the device 43 | hardware_vendor, operating system and type 44 | """ 45 | directory = '' 46 | if hardware_vendor == 'cisco' and opersys == 'asa' and device_type == 'firewall': 47 | directory = '{}/superloop_code/templates/hardware_vendors/cisco/asa/firewall/'.format(get_home_directory()) 48 | elif hardware_vendor == 'cisco' and opersys == 'ios'and device_type == 'router': 49 | directory = '{}/superloop_code/templates/hardware_vendors/cisco/ios/router'.format(get_home_directory()) 50 | elif hardware_vendor == 'cisco' and opersys == 'ios'and device_type == 'switch': 51 | directory = '{}/superloop_code/templates/hardware_vendors/cisco/ios/switch/'.format(get_home_directory()) 52 | elif hardware_vendor == 'cisco' and opersys == 'nxos'and device_type == 'switch': 53 | directory = '{}/superloop_code/templates/hardware_vendors/cisco/nxos/switch/'.format(get_home_directory()) 54 | elif hardware_vendor == 'cisco' and opersys == 'nxos'and device_type == 'router': 55 | directory = '{}/superloop_code/templates/hardware_vendors/cisco/nxos/router/'.format(get_home_directory()) 56 | elif hardware_vendor == 'juniper' and opersys == 'junos' and device_type == 'vfirewall': 57 | directory = '{}/superloop_code/templates/hardware_vendors/juniper/junos/vfirewall/'.format(get_home_directory()) 58 | elif hardware_vendor == 'juniper' and opersys == 'junos' and device_type == 'router': 59 | directory = '{}/superloop_code/templates/hardware_vendors/juniper/junos/router/'.format(get_home_directory()) 60 | elif hardware_vendor == 'citrix' and opersys == 'netscaler' and device_type == 'loadbalancer': 61 | directory = '{}/superloop_code/templates/hardware_vendors/citrix/netscaler/vpx/'.format(get_home_directory()) 62 | elif hardware_vendor == 'f5' and opersys == 'tmsh' and device_type == 'loadbalancer': 63 | directory = '{}/superloop_code/templates/hardware_vendors/f5/tmsh/ltm/'.format(get_home_directory()) 64 | 65 | return directory 66 | 67 | def get_policy_directory(hardware_vendor,opersys,device_type): 68 | directory = '' 69 | if hardware_vendor == 'cisco' and opersys == 'asa' and device_type == 'firewall': 70 | directory = '{}/superloop_code/policy/cisco/ios/firewall/'.format(get_home_directory()) 71 | elif hardware_vendor == 'juniper' and opersys == 'junos' and device_type == 'vfirewall': 72 | directory = '{}/superloop_code/policy/juniper/junos/vfirewall/'.format(get_home_directory()) 73 | 74 | return directory 75 | 76 | def get_updated_list(list_copy): 77 | """ 78 | This will get the current template list from the list. 79 | Example: ['base.jinja'],['snmp.jinja','tacacs.jinja']]. 80 | It will continue to pop off the 1st element until there 81 | are only one element left. 82 | """ 83 | updated_list = [] 84 | if len(list_copy) != 1: 85 | list_copy.pop(0) 86 | updated_list = list_copy[0] 87 | 88 | return updated_list 89 | 90 | def get_syntax(node_object,index): 91 | """ 92 | This will return the correct syntax used for CiscoConfParse 93 | based on device hardware vendor. 94 | """ 95 | syntax = '' 96 | if node_object[index]['hardware_vendor'] == 'cisco' and node_object[index]['type'] == 'firewall': 97 | syntax = 'asa' 98 | elif node_object[index]['hardware_vendor'] == 'cisco' and node_object[index]['type'] == 'switch': 99 | syntax = 'ios' 100 | elif node_object[index]['hardware_vendor'] == 'juniper' and node_object[index]['type'] == 'switch': 101 | syntax = 'junos' 102 | elif node_object[index]['hardware_vendor'] == 'f5' and node_object[index]['type'] == 'loadbalancer': 103 | syntax = 'ios' 104 | 105 | return syntax 106 | 107 | def get_sorted_juniper_template_list(template_list): 108 | 109 | """ This will sort the Juniper template list from top configuration 110 | in the order they appear in a 'show configuration' juniper output. 111 | For example: groups, systems, chassis, security, snmp etc... 112 | """ 113 | sorted_juniper_template_list = [] 114 | sorted_juniper_template_list_index = [] 115 | config_order = { 116 | 'groups.jinja2': 0 , 117 | 'system.jinja2': 1 , 118 | 'interfaces.jinja2': 2, 119 | 'chassis.jinja2': 3 , 120 | 'snmp.jinja2': 4 , 121 | 'routing-options.jinja2': 5 , 122 | 'protocols.jinja2': 6 , 123 | 'policy-options.jinja2': 7 , 124 | 'security.jinja2': 8 , 125 | 'routing-instances.jinja2': 9 126 | } 127 | for template in template_list: 128 | if template in config_order.keys(): 129 | sorted_juniper_template_list_index.append(config_order[template]) 130 | 131 | """ 132 | Sorting the order of how the templates should be in comparison with 133 | with the Juniper 'show configuration' output. 134 | """ 135 | sorted_juniper_template_list_index.sort() 136 | """ 137 | Building the sorted template list and returning 138 | """ 139 | for element in sorted_juniper_template_list_index: 140 | template = list(config_order.keys())[list(config_order.values()).index(element)] 141 | sorted_juniper_template_list.append(template) 142 | 143 | return sorted_juniper_template_list 144 | 145 | def get_standards_directory(name,hardware_vendor,type): 146 | directory = '{}/superloop_code/templates/standards/'.format(get_home_directory()) 147 | 148 | return directory 149 | 150 | def timestamp(): 151 | time_stamp =time.time() 152 | date_time = datetime.datetime.fromtimestamp(time_stamp).strftime('%Y-%m-%d %H:%M:%S') 153 | 154 | return date_time 155 | 156 | def get_resolve_hostname(fqdn): 157 | try: 158 | mgmt_ip4 = socket.gethostbyname(fqdn) 159 | 160 | return mgmt_ip4 161 | 162 | except socket.error: 163 | mgmt_ip4 = 'null' 164 | 165 | return mgmt_ip4 166 | 167 | 168 | def get_serial_oid(snmp_platform_name): 169 | platform_name = snmp_platform_name.lower() 170 | device_serial = '' 171 | SERIAL_OID = { 172 | 'firefly-perimeter':'1.3.6.1.4.1.2636.3.1.3.0', 173 | 'c3750':'1.3.6.1.4.1.9.5.1.2.19.0', 174 | 'adaptive security appliance':'1.3.6.1.2.1.47.1.1.1.1.11.1', 175 | 'cisco nx-os':'1.3.6.1.2.1.47.1.1.1.1.11.22', 176 | 'netscaler':'1.3.6.1.4.1.5951.4.1.1.14.0' 177 | } 178 | for model in SERIAL_OID: 179 | if model in platform_name: 180 | device_serial_oid = SERIAL_OID[model] 181 | break 182 | else: 183 | device_serial_oid = 'null' 184 | 185 | return device_serial_oid 186 | 187 | def get_secrets(): 188 | client = hvac.Client() 189 | data = client.auth.approle.login( 190 | role_id = os.environ.get('VAULT_ROLE_ID'), 191 | secret_id = os.environ.get('VAULT_SECRET_ID') 192 | ) 193 | VAULT_TOKEN = data['auth']['client_token'] 194 | secret_data = client.read('{}'.format(os.environ.get('VAULT_PATH'))) 195 | secrets = secret_data['data']['data'] 196 | 197 | return secrets 198 | 199 | """ 200 | get_no_negate() function stores a list of strings that do not require a negation when it comes to the Cisco. However, the command by default begins with a 'no'. 201 | For example. 'no logging rate-limit', 'no service-pad'. The keywords must be stored in this list so superloop checks against it to know which commands to not 202 | negate. 203 | """ 204 | def get_no_negate(): 205 | no_negate = [ 206 | 'logging', 207 | 'service', 208 | ] 209 | 210 | return no_negate 211 | -------------------------------------------------------------------------------- /gifs/superloop_audit_diff_demo.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/superloopnetwork/superloop/996fd08a4381b5d6b95e47d75bdfbadfc3bdae6c/gifs/superloop_audit_diff_demo.gif -------------------------------------------------------------------------------- /gifs/superloop_auditcreeper.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/superloopnetwork/superloop/996fd08a4381b5d6b95e47d75bdfbadfc3bdae6c/gifs/superloop_auditcreeper.gif -------------------------------------------------------------------------------- /gifs/superloop_auditcreeper_demo.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/superloopnetwork/superloop/996fd08a4381b5d6b95e47d75bdfbadfc3bdae6c/gifs/superloop_auditcreeper_demo.gif -------------------------------------------------------------------------------- /gifs/superloop_gitops_operational_framework.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/superloopnetwork/superloop/996fd08a4381b5d6b95e47d75bdfbadfc3bdae6c/gifs/superloop_gitops_operational_framework.png -------------------------------------------------------------------------------- /gifs/superloop_host_exec_after_push_cfgs_demo.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/superloopnetwork/superloop/996fd08a4381b5d6b95e47d75bdfbadfc3bdae6c/gifs/superloop_host_exec_after_push_cfgs_demo.gif -------------------------------------------------------------------------------- /gifs/superloop_host_exec_demo.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/superloopnetwork/superloop/996fd08a4381b5d6b95e47d75bdfbadfc3bdae6c/gifs/superloop_host_exec_demo.gif -------------------------------------------------------------------------------- /gifs/superloop_push_cfgs_demo.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/superloopnetwork/superloop/996fd08a4381b5d6b95e47d75bdfbadfc3bdae6c/gifs/superloop_push_cfgs_demo.gif -------------------------------------------------------------------------------- /gifs/superloop_push_local_demo.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/superloopnetwork/superloop/996fd08a4381b5d6b95e47d75bdfbadfc3bdae6c/gifs/superloop_push_local_demo.gif -------------------------------------------------------------------------------- /gifs/superloop_push_onscreen_demo.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/superloopnetwork/superloop/996fd08a4381b5d6b95e47d75bdfbadfc3bdae6c/gifs/superloop_push_onscreen_demo.gif -------------------------------------------------------------------------------- /initialize.py: -------------------------------------------------------------------------------- 1 | """ 2 | This module define the global variables. 3 | """ 4 | def variables(): 5 | global backup_config 6 | global compliance_percentage 7 | global configuration 8 | global debug_element 9 | global element 10 | global element_policy 11 | global netscaler_ha 12 | global ntw_device 13 | global password 14 | global rendered_config 15 | 16 | backup_config = [] 17 | compliance_percentage = 0 18 | configuration = [] 19 | debug_element = [] 20 | element = [] 21 | element_policy = [] 22 | netscaler_ha = [] 23 | ntw_device = [] 24 | password = '' 25 | rendered_config = [] 26 | 27 | """ 28 | :param backup_config: Backup configuration from taken from devices. 29 | :type backup_config: list 30 | :param compliance_percentage: This keeps track of the percentage of code compliance from our templates with respect to our running configurations. 31 | :type configuration: int 32 | :param configuration: Referenced to global variable configuration which keeps track of all commands per node. 33 | :type configuration: list 34 | :param debug_element: Element number of device for when there are no diffs, then device gets popped off. 35 | :type debug_element: list 36 | :param element_policy: All elements of matched policies. 37 | :type element_policy: int 38 | :param netscaler_ha: All matched nodes. 39 | :type netscaler_ha: list 40 | :param element: The element number of the match device(s) from the list of nodes. 41 | :type element: list 42 | :param element_policy: The element number of the match policy from the list of polcies. 43 | :type element_policy: list 44 | :param ntw_device: All matched nodes. 45 | :type ntw_device: list 46 | :param password: Only a single time use, per session of push_cfgs.py for Cisco like devices so users do not require to authenticate twice. 47 | :type password: str 48 | :param rendered_config: Rendered configurations from templates. 49 | :type rendered_config: list 50 | """ 51 | -------------------------------------------------------------------------------- /lib/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/superloopnetwork/superloop/996fd08a4381b5d6b95e47d75bdfbadfc3bdae6c/lib/__init__.py -------------------------------------------------------------------------------- /lib/mediators/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/superloopnetwork/superloop/996fd08a4381b5d6b95e47d75bdfbadfc3bdae6c/lib/mediators/__init__.py -------------------------------------------------------------------------------- /lib/mediators/generic.py: -------------------------------------------------------------------------------- 1 | """ 2 | This mediator is a generic audit diff for various hardware vendors including Cisco, Citrix and F5. 3 | """ 4 | import diffios 5 | import re 6 | import initialize 7 | import itertools 8 | import os 9 | from ciscoconfparse import CiscoConfParse 10 | from ciscoconfparse import HDiff 11 | from get_property import get_no_negate 12 | from get_property import get_policy_directory 13 | from get_property import get_template_directory 14 | from get_property import get_syntax 15 | from get_property import get_sorted_juniper_template_list 16 | from parse_cmd import citrix_parse_negation_commands 17 | from parse_cmd import cisco_parse_negation_commands 18 | 19 | home_directory = os.environ.get('HOME') 20 | 21 | def generic_audit_diff(args,node_configs,node_object,index,template,input_list,AUDIT_FILTER_RE,output,with_remediation): 22 | length_rendered_config = 0 23 | length_backup_config = 0 24 | delta_diff_counter = 0 25 | delta_length_rendered_config = 0 26 | for template in input_list: 27 | filtered_backup_config = [] 28 | rendered_config = [] 29 | backup_config = [] 30 | """ 31 | :param filtered_backup_config: Audit filters that matches the lines in backup_config. Entries include depths/deep configs. 32 | :type filtered_backup_config: list 33 | 34 | :param rendered_config: Rendered configs from the templates. 35 | :type rendered_config: list 36 | 37 | :param backup_config: Backup configs from the 'show run', 'list ltm' etc... 38 | :type backup_config: list 39 | 40 | :param commands: ... Configurations generated from the diff'ed output. 41 | :type commands: list 42 | """ 43 | with open("{}/rendered-configs/{}.{}".format(home_directory,node_object[index]['name'],template.split('.')[0]) + ".conf", "r") as file: 44 | init_config = file.readlines() 45 | for config_line in init_config: 46 | strip_config = config_line.strip('\n') 47 | strip_config = strip_config.rstrip() 48 | if strip_config == '' or strip_config == ' ' or strip_config == '!': 49 | continue 50 | else: 51 | rendered_config.append(strip_config) 52 | length_rendered_config = len(rendered_config) 53 | with open("{}/backup-configs/{}".format(home_directory,node_object[index]['name']) + ".conf", "r") as file: 54 | init_config = file.readlines() 55 | for config_line in init_config: 56 | strip_config = config_line.strip('\n') 57 | strip_config = strip_config.rstrip() 58 | if strip_config == '' or strip_config == ' ' or strip_config == '!' or strip_config == '! ': 59 | continue 60 | else: 61 | backup_config.append(strip_config) 62 | length_backup_config = len(backup_config) 63 | if args.policy is not None: 64 | directory = get_policy_directory(node_object[index]['hardware_vendor'],node_object[index]['opersys'],node_object[index]['type']) 65 | with open("{}".format(directory) + template, "r") as file: 66 | parse_audit = file.readline() 67 | else: 68 | directory = get_template_directory(node_object[index]['hardware_vendor'],node_object[index]['opersys'],node_object[index]['type']) 69 | with open("{}".format(directory) + template, "r") as file: 70 | parse_audit = file.readline() 71 | """ 72 | This will take each element from the audit_filter list and search for the matched lines in backup_config. 73 | """ 74 | audit_filter = eval(re.findall(AUDIT_FILTER_RE, parse_audit)[0]) 75 | parse_backup_configs = CiscoConfParse(backup_config, syntax=get_syntax(node_object,index)) 76 | """ 77 | Matched entries are then appended to the filter_backup_config variable. parse_audit_filter() call will find all parent/child. 78 | """ 79 | filtered_backup_config = parse_audit_filter( 80 | node_object, 81 | index, 82 | parse_backup_configs, 83 | audit_filter 84 | ) 85 | diff = diffios.Compare(filtered_backup_config,rendered_config) 86 | push_configs = diff.missing() + diff.additional() 87 | push_configs = list(itertools.chain.from_iterable(push_configs)) 88 | addition = list(itertools.chain.from_iterable(diff.additional())) 89 | """ 90 | Calculating the percentage of config lines from the template that matches the running-configuration of the device. 91 | This willprovide an accurate reading config standardization. 92 | """ 93 | total_template_lines = len(rendered_config) 94 | total_filtered_backup_config_lines = len(filtered_backup_config) 95 | total_backup_config_lines = len(backup_config) 96 | total_push_config_lines = len(push_configs) 97 | """ 98 | :param total_template_lines: Total number of lines in the template. 99 | :type total_template_lines: int 100 | 101 | :param total_filtered_backup_config_lines: Total number of lines in the filtered backup config. 102 | :type total_filtered_backup_config_lines: int 103 | 104 | :param total_backup_config_lines: Total number of lines in the backup config. 105 | :type total_backup_config_lines: int 106 | """ 107 | 108 | """ 109 | If there are no diffs and only and audit diff is executed, (none) will be printed to show users the result. However, if there are no diffs but a push cfgs 110 | is executed, no configs would be pushed as an empty list is appended. 111 | """ 112 | if len(push_configs) == 0 and output: 113 | if output: 114 | print('{}{} (none)'.format(directory,template)) 115 | print('') 116 | template_percentage = round(length_rendered_config / length_backup_config * 100,2) 117 | initialize.compliance_percentage = round(initialize.compliance_percentage + template_percentage,2) 118 | else: 119 | initialize.configuration.append([]) 120 | print('There are no diffs to be pushed for template {} on {}'.format(template,node_object[index]['name'])) 121 | if len(initialize.element) == 0: 122 | node_configs.append('') 123 | break 124 | else: 125 | """ 126 | If an audit diff is executed, the diff is outputed to user. If a push cfgs is executed against Cisco like platforms, the commands from the diff are executed 127 | with the negation (no). This is to maintain the sequence of the the commands in order to match the jinja2 templates. What you see on the template is what the 128 | users want exactly as the running-configurations. The with_remediation flag is no longer required on the template itself as it may cause disruptions to 129 | services. For example, blowing out an entire logging configs (no logging) and re-adding all the logging host back on. 130 | """ 131 | if output: 132 | print("{}{}".format(directory,template)) 133 | print('\n'.join(HDiff(filtered_backup_config,rendered_config,syntax="ios",ordered_diff=True).unified_diffs()[3:])) 134 | delta_length_rendered_config = length_rendered_config - delta_diff_counter 135 | template_percentage = round(delta_length_rendered_config / length_backup_config * 100,2) 136 | initialize.compliance_percentage = round(initialize.compliance_percentage + template_percentage,2) 137 | else: 138 | if node_object[index]['hardware_vendor'] == 'cisco' and len(push_configs) != 0: 139 | negate_configs = cisco_parse_negation_commands(diff.missing()) 140 | for config in negate_configs: 141 | node_configs.append(config) 142 | for config in addition: 143 | node_configs.append(config) 144 | elif node_object[index]['hardware_vendor'] == 'citrix': 145 | negate_configs = citrix_parse_negation_commands(diff.missing()) 146 | for config in negate_configs: 147 | node_configs.append(config) 148 | for config in addition: 149 | node_configs.append(config) 150 | return None 151 | 152 | def parse_audit_filter(node_object,index,parse_backup_configs,audit_filter): 153 | filtered_backup_config = [] 154 | for audit in audit_filter: 155 | current_template = parse_backup_configs.find_objects(r"^{}".format(audit)) 156 | for audit_string in current_template: 157 | filtered_backup_config.append(audit_string.text) 158 | if audit_string.is_parent: 159 | for child in audit_string.all_children: 160 | filtered_backup_config.append(child.text) 161 | 162 | return filtered_backup_config 163 | -------------------------------------------------------------------------------- /lib/mediators/juniper.py: -------------------------------------------------------------------------------- 1 | """ 2 | This mediator is a audit diff supporting the Juniper hardware vendor. 3 | """ 4 | import re 5 | import initialize 6 | import os 7 | from get_property import get_sorted_juniper_template_list 8 | 9 | home_directory = os.environ.get('HOME') 10 | 11 | def juniper_mediator(node_object,template_list,diff_config,edit_list,index): 12 | 13 | for template in template_list: 14 | ### THIS SECTION IS FOR JUNIPER NETWORKS PLATFORM ### 15 | 16 | ###UN-COMMENT THE BELOW PRINT STATEMENT FOR DEBUGING PURPOSES 17 | # print ("DIFF CONFIG: {}".format(diff_config)) 18 | 19 | ### THIS FIRST SECTION WILL FIND ALL THE INDEXES WITH THE '[edit