├── .gitignore ├── LICENSE ├── README.md ├── ansible_hacking.json └── ansible_hacking.py /.gitignore: -------------------------------------------------------------------------------- 1 | *.pyc -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2017 Joel W. King 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # ansible-hacking 2 | Run Python code (without modifications) inside or outside (for debugging) the Ansible framework 3 | 4 | The goal of this effort is to create a efficient and easy to use, and ephemeral remote development environment. I wanted to be more efficient and productive in developing code. Some considerations: 5 | 6 | * Desire to create a development host independent of the laptop OS, using an ephemeral environment (Vagrant) 7 | * Ability to run current Ansible production code (rather that installing from source for testing) 8 | * Use of effective debugging tools, (PyCharm Pro, pydevd) 9 | 10 | Additionally, Other than adding the necessary pydevd commands, I wanted an executable which would run without modifications in either test mode or production mode. 11 | 12 | The underlying theme is to create environments where a network engineer who is learning NetDevOps concepts can learn without the distractions of system administration. 13 | 14 | App development in Phantom Cyber implements the approach of running Python modules from the shell (uid phantom) in test mode which reads a JSON file to pass parameters. Once the app is functional in that environment, you use their compiler to check the app and app description file, and then optionally install into the Phantom GUI framework. Co-winners of the [Phantom app contest](https://blog.phantom.us/2016/06/20/winners-announced-10000-phantom-app-playbook-contest/), Mauricio Velazco and Nelson Santos, highlighted how useful the remote debugging feature of PyCharm Pro was in their development efforts. I wanted to leverage that concept for my Ansible environment. 15 | 16 | ## The Setup 17 | 18 | We have created a Ubuntu Virtual Machine with Vagrant. Ansible is installed using the [Latest Releases Via Apt](http://docs.ansible.com/ansible/intro_installation.html#latest-releases-via-apt-ubuntu) using this [Vagrantfile](https://github.com/joelwking/devnet-create-meraki-api/blob/master/netops/Vagrantfile). 19 | 20 | By default Ansible will look for user modules in `./library` from the playbook directory, and `/usr/share/ansible`. I've specified these locations in my `ansible.cfg` file as a reminder. 21 | 22 | ``` 23 | $ more ansible.cfg 24 | [defaults] 25 | inventory = /home/ubuntu/ansible/playbooks/hosts 26 | library = /home/ubuntu/ansible/playbooks/library:/usr/share/ansible 27 | ``` 28 | My hosts file is empty, we are only using local host. Ansible will complain, ignore it. 29 | 30 | ``` 31 | $ ansible --version 32 | ansible 2.3.0.0 33 | config file = /home/ubuntu/ansible/playbooks/ansible.cfg 34 | configured module search path = [u'/home/ubuntu/ansible/playbooks/library', u'/usr/share/ansible'] 35 | python version = 2.7.12 (default, Nov 19 2016, 06:48:10) [GCC 5.4.0 20160609] 36 | ``` 37 | ## Modules 38 | In this example, the main module we want to test is `meraki_vlan.py` which imports `Meraki_Connector`. The `ansible_hacking.py` module is also present in this directory along with the associated `ansible_hacking.json` file containing arguments for execution. The upload feature of PyCharm can be used to upload the module(s) we are developing. Download the `ansible_hacking` Python and JSON with cURL or wget, dropping it in the library directory. 39 | ``` 40 | ubuntu@ubuntu-xenial:~/ansible/playbooks/library$ ls -lt 41 | total 44 42 | -rw-rw-r-- 1 ubuntu ubuntu 282 May 12 15:34 ansible_hacking.json 43 | -rwxr-xr-x 1 ubuntu ubuntu 2055 May 12 15:31 ansible_hacking.py 44 | -rw-rw-r-- 1 ubuntu ubuntu 4528 May 12 12:24 meraki_vlan.py 45 | -rw-rw-r-- 1 ubuntu ubuntu 11779 May 3 12:53 Meraki_Connector.py 46 | ``` 47 | The `meraki_vlan.py` module will attempt to import the AnsibleModule class from `ansible_hacking`, if not found, defaults to the normal ansible import. The code snippet is as: 48 | ```python 49 | try: 50 | from ansible_hacking import AnsibleModule # Test 51 | except ImportError: 52 | from ansible.module_utils.basic import * # Production 53 | main() 54 | ``` 55 | If we were to run this module using an Ansible playbook, the task would look similar to the following: 56 | ```yaml 57 | - name: manage vlans 58 | meraki_vlan: 59 | dashboard: "{{inventory_hostname}}" 60 | organization: "{{meraki.organization}}" 61 | api_key: "{{meraki_params.apikey}}" 62 | action: add # add, delete update 63 | network: "{{meraki.network}}" # Name of the network 64 | id: "1492" # VLAN number 65 | name: VLAN1492 # VLAN name 66 | applianceIp: "192.0.2.1" # Default Gateway IP address 67 | subnet: "192.0.2.0/24" # Layer 3 network address of the VLAN 68 | ``` 69 | ## Input file 70 | For our testing, create a file `ansible_hacking` in JSON format to present the arguments. 71 | 72 | ```json 73 | { 74 | "subnet": "203.0.113.0/24", 75 | "network": "KINGJOE", 76 | "applianceIp": "203.0.113.1", 77 | "dashboard": "dashboard.meraki.com", 78 | "action": "add", 79 | "organization": "WWTINC", 80 | "api_key": "bf89redactedfac313c87a1", 81 | "id": "1492", 82 | "name": "NET3" 83 | } 84 | ``` 85 | ## Test execution 86 | Execute the module by invoking python and specify `meraki_vlan.py`. For debugging, we output the value of `argument_spec` used for production execution, but we don't process it. 87 | ``` 88 | ~/ansible/playbooks/library$ python meraki_vlan.py 89 | Entered ansible_hacking, AnsibleModule 90 | { 91 | "argument_spec": { 92 | "subnet": { 93 | "required": true 94 | }, 95 | "network": { 96 | "required": true 97 | }, 98 | "applianceIp": { 99 | "required": true 100 | }, 101 | "dashboard": { 102 | "required": true 103 | }, 104 | "action": { 105 | "default": "add", 106 | "required": false, 107 | "choices": [ 108 | "add", 109 | "delete", 110 | "update" 111 | ] 112 | }, 113 | "organization": { 114 | "required": true 115 | }, 116 | "api_key": { 117 | "required": true 118 | }, 119 | "id": { 120 | "required": true 121 | }, 122 | "name": { 123 | "required": true 124 | } 125 | } 126 | } 127 | loading params from ansible_hacking.json 128 | params: 129 | { 130 | "subnet": "203.0.113.0/24", 131 | "network": "KINGJOE", 132 | "applianceIp": "203.0.113.1", 133 | "dashboard": "dashboard.meraki.com", 134 | "action": "add", 135 | "organization": "WWTINC", 136 | "api_key": "bf89redactedfac313c87a1", 137 | "id": "1492", 138 | "name": "NET3" 139 | } 140 | Exiting AnsibleModule __init__ 141 | { 142 | "status_code": 201, 143 | "changed": true, 144 | "result": { 145 | "networkId": "L_62redacted25030308", 146 | "subnet": "203.0.113.0/24", 147 | "fixedIpAssignments": {}, 148 | "name": "NET3", 149 | "applianceIp": "203.0.113.1", 150 | "reservedIpRanges": [], 151 | "dnsNameservers": "upstream_dns", 152 | "id": 1492 153 | } 154 | } 155 | ``` 156 | The last bit of JSON above is what the module would normally output when run in the Ansible framework. 157 | 158 | ## Production 159 | From the Meraki dashboard, delete the VLAN created, as we will now execute the same code in the Ansible framework. We also move the `ansible_hacking` files out of the library directory. 160 | ``` 161 | $ mv ansible_hacking* /tmp 162 | ``` 163 | What remains are the modules under development. 164 | ``` 165 | ~/ansible/playbooks/library$ ls -salt 166 | total 44 167 | 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 12 18:27 . 168 | 4 drwxr-xr-x 3 ubuntu ubuntu 4096 May 12 18:26 .. 169 | 8 -rw-rw-r-- 1 ubuntu ubuntu 4528 May 12 12:24 meraki_vlan.py 170 | 16 -rw-rw-r-- 1 ubuntu ubuntu 12583 May 12 01:33 Meraki_Connector.pyc 171 | 12 -rw-rw-r-- 1 ubuntu ubuntu 11779 May 3 12:53 Meraki_Connector.py 172 | ``` 173 | Go up to the playbook directory. I normally run and store playbooks in this directory. 174 | ``` 175 | $ cd ~/ansible/playbooks 176 | ``` 177 | Run the same code, this time using Ansible. 178 | ``` 179 | ~/ansible/playbooks$ ansible localhost -m meraki_vlan -a "network=KINGJOE id=1492 name=NET3 organization=WWTINC applianceIp=203.0.113.1 subnet=203.0.113.0/24 api_key=bf89redactedfac313c87a1 dashboard=dashboard.meraki.com" 180 | [WARNING]: Host file not found: /home/ubuntu/ansible/playbooks/hosts 181 | 182 | [WARNING]: provided hosts list is empty, only localhost is available 183 | 184 | localhost | SUCCESS => { 185 | "changed": true, 186 | "result": { 187 | "applianceIp": "203.0.113.1", 188 | "dnsNameservers": "upstream_dns", 189 | "fixedIpAssignments": {}, 190 | "id": 1492, 191 | "name": "NET3", 192 | "networkId": "L_62redacted25030308", 193 | "reservedIpRanges": [], 194 | "subnet": "203.0.113.0/24" 195 | }, 196 | "status_code": 201 197 | } 198 | ``` 199 | From the above, you can see that we have successfully executed the same module inside the Ansible production environment without modifications. 200 | 201 | ## Module Documentation 202 | Additionally, we can successfully run `ansible-doc`. 203 | ``` 204 | ~/ansible/playbooks$ ansible-doc meraki_vlan 205 | > MERAKI_VLAN (/home/ubuntu/ansible/playbooks/library/meraki_vlan.py) 206 | 207 | Manage VLANs on Meraki Networks 208 | 209 | [lines removed for breviety] 210 | 211 | MAINTAINERS: Joel W. King, (@joelwking) World Wide Technology 212 | 213 | METADATA: 214 | Status: ['preview'] 215 | Supported_by: community 216 | ``` 217 | 218 | ## Invoking with PyCharm Professional 219 | At this point, the only use of PyCharm was to upload the module(s) under development. Move the `ansible_hacking` module back to the library directory, and run the `meraki_vlan` using the remote Python interpreter feature. More on that configuration in a separate post. 220 | 221 | ``` 222 | ~/ansible/playbooks/library$ mv /tmp/ansible_hacking.* ./ 223 | ubuntu@ubuntu-xenial:~/ansible/playbooks/library$ ls 224 | ansible_hacking.json ansible_hacking.py ansible_hacking.pyc Meraki_Connector.py Meraki_Connector.pyc meraki_vlan.py 225 | ``` 226 | The output from the first little bit of the PyCharm output window looks as follows: 227 | 228 | ``` 229 | ssh://ubuntu@192.168.56.200:22/usr/bin/python -u /home/ubuntu/ansible/playbooks/library/meraki_vlan.py 230 | Entered ansible_hacking, AnsibleModule 231 | { 232 | "argument_spec": { 233 | "subnet": { 234 | "required": true 235 | }, 236 | ``` 237 | 238 | ## Remote Debugging 239 | Remote [debugging](https://www.jetbrains.com/help/pycharm/2017.1/remote-debugging.html) can be enabled in PyCharm Pro and has been tested with the `ansible-hacking` module. 240 | 241 | 242 | 243 | ## References: 244 | 245 | * [Debug Ansible Modules remotely in PyCharm on windows](https://github.com/mmumshad/debug-ansible-modules-pycharm) 246 | * [Work remotely with PyCharm, TensorFlow and SSH](https://medium.com/@erikhallstrm/work-remotely-with-pycharm-tensorflow-and-ssh-c60564be862d) 247 | * [Phantom Developer Resources](https://www.phantom.us/community/) 248 | * [Getting Ansible](http://docs.ansible.com/ansible/intro_installation.html) 249 | * [Building A Simple Module](http://docs.ansible.com/ansible/dev_guide/developing_modules_general.html) 250 | -------------------------------------------------------------------------------- /ansible_hacking.json: -------------------------------------------------------------------------------- 1 | {"dashboard": "dashboard.meraki.com", 2 | "organization": "WWTINC", 3 | "api_key": "bf81478redactedc313c87a1", 4 | "action": "add", 5 | "network": "TELEWORKER", 6 | "id": "1492", 7 | "name": "NET3", 8 | "applianceIp": "203.0.113.1", 9 | "subnet": "203.0.113.0/24"} -------------------------------------------------------------------------------- /ansible_hacking.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | """ 3 | Copyright (c) 2017 World Wide Technology, Inc. 4 | All rights reserved. 5 | 6 | module: ansible_hacking.py 7 | 8 | author: Joel W. King, (@joelwking) World Wide Technology 9 | 10 | short_description: A mock object to simulate the Ansible environment for remote debugging 11 | reference: http://docs.ansible.com/ansible/dev_guide/developing_modules_general.html 12 | description: This code provides an alternative to using the Ansible hacking/test-module 13 | """ 14 | import sys 15 | import json 16 | 17 | class AnsibleModule(object): 18 | """ Mock class for testing Ansible modules outside of the Ansible framework. 19 | """ 20 | INPUT_JSON = "ansible_hacking.json" 21 | NO = ("n","N", "no", "No", "NO", "False", "FALSE", "false", "off", "Off", "OFF") 22 | YES = ("y", "Y", "yes", "Yes", "YES", "True", "TRUE", "true", "on", "On", "ON") 23 | 24 | 25 | def __init__(self, **kwargs): 26 | """ Ansible internally saves arguments to an arguments file, 27 | we read from our own JSON file. 28 | """ 29 | print "Entered ansible_hacking, AnsibleModule \n%s" % (json.dumps(kwargs, indent=4)) 30 | self.params = self.read_params(AnsibleModule.INPUT_JSON) 31 | print "params:\n%s\nExiting AnsibleModule __init__" % (json.dumps(self.params, indent=4)) 32 | 33 | def read_params(self, fname): 34 | """ Arguments for testing are read from JSON format file. 35 | """ 36 | 37 | try: 38 | jsonfile = open(fname, 'r') 39 | except IOError: 40 | print "input file: %s not found!" % fname 41 | sys.exit(1) 42 | 43 | infile = jsonfile.read() 44 | jsonfile.close() 45 | 46 | # BOOLEAN logic 47 | params = json.loads(infile) 48 | for key, value in params.items(): 49 | if value in AnsibleModule.NO: 50 | params[key] = False 51 | if value in AnsibleModule.YES: 52 | params[key] = True 53 | 54 | return params 55 | 56 | def exit_json(self, **kwargs): 57 | """ Modules return information to Ansible by printing a JSON string 58 | to stdout before exiting. 59 | """ 60 | print "%s" % (json.dumps(kwargs, indent=4)) 61 | sys.exit(0) 62 | 63 | def fail_json(self, msg): 64 | """Fail with a message formatted in JSON. 65 | """ 66 | print json.dumps({'msg': msg}, indent=4) 67 | sys.exit(1) 68 | --------------------------------------------------------------------------------