├── LICENSE ├── README.md ├── yaml_inventory.conf └── yaml_inventory.py /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) Jiri Tyr 2019 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy of 6 | this software and associated documentation files (the "Software"), to deal in 7 | the Software without restriction, including without limitation the rights to 8 | use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of 9 | the Software, and to permit persons to whom the Software is furnished to do so, 10 | subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS 17 | FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 18 | COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER 19 | IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN 20 | CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 21 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | yaml_inventory 2 | ============== 3 | 4 | Ansible dynamic inventory script which reads the inventory from a specially 5 | formatted YAML file. 6 | 7 | 8 | Description 9 | ----------- 10 | 11 | Standard Ansible inventory suffers of several issues: 12 | 13 | - It must be a single file or multiple files in a directory. 14 | - It has a flat structure where all groups are on the same level and relate to 15 | each other via `:children` definition leading to long group names when trying 16 | to capture more complex relationships. That also affects the names of the 17 | `group_vars`. 18 | - When using Vault file, it requires the files the be either named like the 19 | group or to be placed inside a directory of the group name or to be 20 | explicitely included in the play. 21 | - Hosts that belong to multiple groups must be defined in multiple places. 22 | - Sharing the same `group_vars` across multiple groups is a challenging problem. 23 | 24 | This Ansible dynamic inventory script is trying to address these issues 25 | by allowing the following features: 26 | 27 | - Possibility to split single inventory into multiple files. 28 | - Self-generating group names based on the YAML structure. 29 | - Truly hierarchical vars files. 30 | - Automatic `.vault` files loading. 31 | - Possibility to add hosts to an other group. 32 | - Possibility to include hosts from an other groups via regexp. 33 | - Using vars files as a common template. 34 | 35 | See the usage bellow for more details. 36 | 37 | 38 | Installation 39 | ------------ 40 | 41 | ``` 42 | $ git clone https://github.com/jtyr/ansible-yaml_inventory yaml_inventory 43 | $ ln -s yaml_inventory/yaml_inventory.py hosts 44 | $ ansible-playbook -i hosts site.yaml 45 | ``` 46 | 47 | 48 | Usage 49 | ----- 50 | 51 | ### Inventory YAML file 52 | 53 | Here is an example of the standard Ansible inventory file: 54 | 55 | ``` 56 | [aws:children] 57 | aws-dev 58 | aws-qa 59 | aws-stg 60 | aws-prd 61 | 62 | [aws-dev] 63 | aws-dev-host01 ansible_host=192.168.1.15 64 | 65 | [aws-dev:children] 66 | aws-dev-jenkins 67 | 68 | [aws-dev-jenkins] 69 | aws-dev-jenkins01 ansible_host=192.168.1.16 70 | 71 | [aws-qa] 72 | aws-qa-host01 ansible_host=192.168.2.15 73 | 74 | [aws-stg] 75 | aws-stg-host01 ansible_host=192.168.3.15 76 | 77 | [aws-prd] 78 | aws-prd-host01 ansible_host=192.168.4.15 79 | 80 | 81 | [azure:children] 82 | azure-dev 83 | azure-qa 84 | azure-stg 85 | azure-prd 86 | 87 | [azure-dev] 88 | azure-dev-host01 ansible_host=10.0.1.15 89 | 90 | [azure-dev:children] 91 | azure-dev-jenkins 92 | 93 | [azure-dev-jenkins] 94 | azure-dev-jenkins01 ansible_host=10.0.1.16 95 | 96 | [azure-qa] 97 | azure-qa-host01 ansible_host=10.0.2.15 98 | 99 | [azure-stg] 100 | azure-stg-host01 ansible_host=10.0.3.15 101 | 102 | [azure-prd] 103 | azure-prd-host01 ansible_host=10.0.4.15 104 | ``` 105 | 106 | And here is the same but in the YAML format: 107 | 108 | ``` 109 | --- 110 | 111 | aws: 112 | dev: 113 | :hosts: 114 | - aws-dev-host01: { ansible_host: 192.168.1.15 } 115 | jenkins: 116 | :hosts: 117 | - aws-dev-jenkins01: { ansible_host: 192.168.1.16 } 118 | qa: 119 | :hosts: 120 | - aws-qa-host01: { ansible_host: 192.168.2.15 } 121 | stg: 122 | :hosts: 123 | - aws-stg-host01: { ansible_host: 192.168.3.15 } 124 | prd: 125 | :hosts: 126 | - aws-prd-host01: { ansible_host: 192.168.4.15 } 127 | 128 | azure: 129 | dev: 130 | :hosts: 131 | - azure-dev-host01: { ansible_host: 10.0.1.15 } 132 | jenkins: 133 | :hosts: 134 | - azure-dev-jenkins01: { ansible_host: 10.0.1.16 } 135 | qa: 136 | :hosts: 137 | - azure-qa-host01: { ansible_host: 10.0.2.15 } 138 | stg: 139 | :hosts: 140 | - azure-stg-host01: { ansible_host: 10.0.3.15 } 141 | prd: 142 | :hosts: 143 | - azure-prd-host01: { ansible_host: 10.0.4.15 } 144 | ``` 145 | 146 | The main YAML inventory should be stored in the `main.yaml` file located 147 | by default in the `inventory` directory. The location can be changed in 148 | the config file (`inventory_path` - see the `yaml_inventory.conf` file) 149 | or via environment variable (`YAML_INVENTORY_PATH`). 150 | 151 | This is an example of a monolithic inventory YAML file: 152 | 153 | ``` 154 | --- 155 | 156 | aws: 157 | dev: 158 | elk: 159 | elasticsearch: 160 | # Hosts of the aws-dev-elk-elasticsearch group 161 | :hosts: 162 | # Hosts with variables 163 | - elk01: { ansible_host: 192.168.1.11 } 164 | - elk02: { ansible_host: 192.168.1.12 } 165 | - elk03: { ansible_host: 192.168.1.13 } 166 | kibana: 167 | # Hosts of the aws-dev-elk-kibana group 168 | :hosts: 169 | # Host with no variables 170 | - elk04 171 | ``` 172 | 173 | The same like above but with YAML reference: 174 | 175 | ``` 176 | --- 177 | 178 | # This is a subset of the main data structure referenced bellow 179 | aws-dev: &aws-dev 180 | elk: 181 | elasticsearch: 182 | :hosts: 183 | - elk01: { ansible_host: 192.168.1.11 } 184 | - elk02: { ansible_host: 192.168.1.12 } 185 | - elk03: { ansible_host: 192.168.1.13 } 186 | kibana: 187 | :hosts: 188 | - elk04 189 | 190 | # This is the main data structure 191 | aws: 192 | dev: 193 | # Reference to the above data structure 194 | <<: *aws-dev 195 | ``` 196 | 197 | This is the same like above but with the referenced content in a separate 198 | file: 199 | 200 | Content of the `aws-dev.yaml` file: 201 | 202 | ``` 203 | --- 204 | 205 | # This can be still referenced from the main YAML file 206 | aws-dev: &aws-dev 207 | elk: 208 | elasticsearch: 209 | :hosts: 210 | - elk01: { ansible_host: 192.168.1.11 } 211 | - elk02: { ansible_host: 192.168.1.12 } 212 | - elk03: { ansible_host: 192.168.1.13 } 213 | kibana: 214 | :hosts: 215 | - elk04 216 | ``` 217 | 218 | Content of the `main.yaml` file: 219 | 220 | ``` 221 | --- 222 | 223 | # This is the main data structure 224 | aws: 225 | dev: 226 | # Refference the above data structure 227 | <<: *aws-dev 228 | ``` 229 | 230 | The inventory script reads all YAML files from the `inventory` directory 231 | and merges them all together. The `main.yaml` portion is always inserted 232 | at the end so that the YAML references can still be resolved. Group names 233 | are composed from elements of the tree separated by `-` sign. 234 | 235 | As visible above, the YAML structure contains special keyword `:hosts`. 236 | That keyword indicates that content of that section is a list of hosts. 237 | Other keywords are `:vars`, `:template`, `:groups` and `:add_hosts`. 238 | The following is an example of usage of all the keywords: 239 | 240 | ``` 241 | --- 242 | 243 | g1: 244 | :hosts: 245 | - ahost1 246 | - ahost2 247 | :vars: 248 | # Variable for the group g1 249 | my_var: 123 250 | g2: 251 | :hosts: 252 | - bhost1 253 | - bhost2 254 | :groups: 255 | # Add all hosts from group g2 into the group g1 256 | - g1 257 | g3: 258 | :hosts: 259 | - chost1 260 | - chost2 261 | :templates: 262 | # This creates g3@template-g3 group which has the g3 group as its 263 | # child. That will allow to load the inventory vars from the file in 264 | # inventory/vars/template/g3. As the g3 group is a child of the 265 | # template group, variables from the g3 inventory vars file can 266 | # still override the vars in the template file. 267 | - template-g3 268 | g4: 269 | :add_hosts: 270 | # Regular expression to add hosts ahost2 and bhost2 into this group 271 | - ^[ab]host2 272 | ``` 273 | 274 | 275 | ### Inventory vars 276 | 277 | Classical `group_vars` files can be structured in two levels only - files 278 | in the `group_vars` directory and file in the directories located in the 279 | `group_vars` directory. This is quite restrictive and forces the user to 280 | capture the inventory groups structure in the `group_vars` file name. 281 | 282 | This Ansible dynamic inventory is trying to address this issue by 283 | introducing the inventory vars. Inventory vars are variation of the 284 | `group_vars` with the difference that it copies the hierarchical 285 | structure of YAML inventory file. The inventory vars directory is by 286 | default in the `inventory/vars` directory but can be changed in the 287 | config file (`inventory_vars_path`) or via environment variable 288 | (`YAML_INVENTORY_VARS_PATH`). This feature is enabled by default but can 289 | be disabled by setting the `create_symlinks` config option or the 290 | `YAML_INVENTORY_CREATE_SYMLINKS` environment variable to value `no`. 291 | 292 | The inventory vars file for a specific group can be called either like 293 | the last element of the group name or like `all` inside a directory 294 | called like the last element of the group name. If the group contains 295 | another groups, only the second option is available because there cannot 296 | coexist file and directory of the same name. 297 | 298 | If this is the inventory file: 299 | 300 | ``` 301 | $ cat inventory/main.yaml 302 | --- 303 | 304 | # This is a group containing another group (vars in `all` file) 305 | aws: 306 | dev: 307 | # This is the leaf group (vars in `jenkins` file) 308 | jenkins: 309 | :hosts: 310 | - aws-dev-jenkins01 311 | ``` 312 | 313 | then the corresponding file structure of the inventory vars can look like 314 | this: 315 | 316 | ``` 317 | $ tree -p inventory/vars 318 | inventory/vars 319 | └── [drwxr-xr-x] aws 320 | ├── [-rw-r--r--] all 321 | └── [drwxr-xr-x] dev 322 | └── [-rw-r--r--] jenkins 323 | ``` 324 | 325 | If enabled, the inventory vars are symlinked into the `group_vars` 326 | directory during the execution of the inventory script. The `group_vars` 327 | file names are based on the structure of the inventory vars directory. 328 | From the example above, the path `invenotory/aws/all` is symlinked like 329 | `group_vars/aws` and the path `invenotory/aws/dev/jenkins` is symlinked 330 | like `group_vars/aws-dev-jenkins`. 331 | 332 | ``` 333 | $ ls -la ./group_vars 334 | total 8 335 | drwxr-xr-x 2 jtyr users 4096 Mar 28 17:20 . 336 | drwxr-xr-x 9 jtyr users 4096 Mar 27 10:10 .. 337 | lrwxrwxrwx 1 jtyr users 21 Mar 28 17:20 aws -> ../inventory/vars/aws/all 338 | lrwxrwxrwx 1 jtyr users 29 Mar 28 17:20 aws-dev-jenkins -> ../inventory/vars/aws/dev/jenkins 339 | ``` 340 | 341 | The script also simplifies the use of Vault files by automatically 342 | creating relationship between the group (e.g. `mygroup`) and the secured 343 | content of that group (`mygroup.vault`). This convention makes sure that 344 | the Vault file is always loaded if it exists. 345 | 346 | 347 | ### Inventory script 348 | 349 | The inventory script can be used as any other Ansible dynamic inventory. 350 | With the default settings (the `main.yaml` inventory file in the 351 | `./inventory` directory and the inventory vars in the `./inventory/vars` 352 | directory) the command can be as follows: 353 | 354 | ``` 355 | $ ansible-playbook -i ./yaml_inventory.py site.yaml 356 | ``` 357 | 358 | The inventory script implements the standard `--list` and `--host` 359 | command line options and can be influenced by a config file (see 360 | `yaml_inventory.conf` file) or environment variables: 361 | 362 | ``` 363 | usage: yaml_inventory.py [-h] [--list] [--host HOST] 364 | 365 | Ansible dynamic inventory reading YAML file. 366 | 367 | optional arguments: 368 | -h, --help show this help message and exit 369 | --list list all groups and hosts 370 | --host HOST get vars for a specific host 371 | 372 | environment variables: 373 | YAML_INVENTORY_CONFIG_PATH 374 | location of the config file (default locations: 375 | ./yaml_inventory.conf 376 | ~/.ansible/yaml_inventory.conf 377 | /etc/ansible/yaml_inventory.conf) 378 | YAML_INVENTORY_PATH 379 | location of the inventory directory (./inventory by default) 380 | YAML_INVENTORY_VARS_PATH 381 | location of the inventory vars directory (YAML_INVENTORY_PATH/vars by default) 382 | YAML_INVENTORY_GROUP_VARS_PATH 383 | location of the vars directory (./group_vars by default) 384 | YAML_INVENTORY_SUPPORT_VAULTS\n' 385 | flag to take in account .vault files (yes by default) 386 | YAML_INVENTORY_CREATE_SYMLINKS 387 | flag to create group_vars symlinks (yes by default) 388 | ``` 389 | 390 | ### Combining it with other inventory scripts 391 | 392 | Put the YAML dynamic inventory script together with the other (e.g. 393 | [AWS EC2](http://docs.ansible.com/ansible/latest/intro_dynamic_inventory.html#example-aws-ec2-external-inventory-script)) 394 | dynamic inventory script into the `inventory_scripts` directory and name them 395 | to be in alphabetical order that the YAML inventory is first and the other 396 | inventory is second: 397 | 398 | ``` 399 | $ ls inventory_scripts 400 | 01_yaml_inventory 401 | 02_aws_inventory 402 | ``` 403 | 404 | Use the YAML inventory to create the desired inventory structure and the 405 | other inventory to fill in the hosts into the groups: 406 | 407 | ``` 408 | $ cat inventory/main.yaml 409 | --- 410 | aws: 411 | dev: 412 | jenkins: 413 | :hosts: [] 414 | ``` 415 | 416 | Then run Ansible like this: 417 | 418 | ``` 419 | $ ansible-playbook -i inventory_scripts site.yaml 420 | ``` 421 | 422 | 423 | Issues 424 | ------ 425 | 426 | - When using `create_symlinks = yes` and the inventory structure has changed, 427 | the YAML inventory must be run manually before the Ansible runs because the 428 | `group_vars` symlinks get created after the `group_vars` files are normally 429 | read by Ansible. 430 | - No Vault support for `all` group due to the circular relationship between 431 | the `all.vault` and the `all` group. 432 | - No support for encrypted variables available in Ansible v2.3+. 433 | 434 | 435 | TODO 436 | ---- 437 | 438 | - Implement hosts enumeration. 439 | - Refactor it into an inventory plugin to facilitate support for encrypted 440 | variables. 441 | 442 | 443 | License 444 | ------- 445 | 446 | MIT 447 | 448 | 449 | Author 450 | ------ 451 | 452 | Jiri Tyr 453 | -------------------------------------------------------------------------------- /yaml_inventory.conf: -------------------------------------------------------------------------------- 1 | ### 2 | # Config file for the YAML inventory script 3 | ### 4 | 5 | 6 | [paths] 7 | # Location of the inventory directory 8 | inventory_path = inventory 9 | 10 | # Location of the vars directory 11 | inventory_vars_path = inventory/vars 12 | 13 | # Location of the group_vars directory 14 | group_vars_path = group_vars 15 | 16 | 17 | [features] 18 | # If set to "yes", symbolic links for files from the inventory/vars will be 19 | # created in the group_vars directory. This is required when using fully 20 | # encrypted .vault files. If set to "no", the variables from the files in the 21 | # inventory/vars will be added directly into the dynamic inventory output. This 22 | # should only be used if there are no .vault files. 23 | create_symlinks = yes 24 | -------------------------------------------------------------------------------- /yaml_inventory.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | try: 4 | import ConfigParser as configparser 5 | except: 6 | import configparser 7 | 8 | import argparse 9 | import glob 10 | import json 11 | import logging 12 | import os 13 | import re 14 | import sys 15 | import yaml 16 | 17 | 18 | # Get logger 19 | log = logging.getLogger(__name__) 20 | 21 | 22 | def create_symlinks(cfg, inv): 23 | for root, dirs, files in os.walk(cfg['vars_path']): 24 | for f in files: 25 | src = "%s/%s" % (root, f) 26 | src_list = src[len(cfg['vars_path'])+1:].split('/') 27 | 28 | # Ignore dotted (e.g. ".git") 29 | if src_list[0].startswith('.'): 30 | continue 31 | 32 | # Strip out the YAML file extension 33 | if src_list[-1].endswith('.yaml'): 34 | src_list[-1] = src_list[-1][:-5] 35 | elif src_list[-1].endswith('.yml'): 36 | src_list[-1] = src_list[-1][:-4] 37 | elif src_list[-1].endswith('.yaml.vault'): 38 | src_list[-1] = "%s.vault" % src_list[-1][:-11] 39 | elif src_list[-1].endswith('.yml.vault'): 40 | src_list[-1] = "%s.vault" % src_list[-1][:-10] 41 | 42 | # Keep only the top-level "all" file 43 | if src_list[-1] in ['all', 'all.vault'] and len(src_list) > 1: 44 | # Keep the .vault extension 45 | if src_list[-1] == 'all.vault': 46 | src_list[-2] += '.vault' 47 | 48 | del src_list[-1] 49 | 50 | src_list_s = '-'.join(src_list) 51 | dst = [] 52 | 53 | # Ignore files which are not groups 54 | if src_list[0] in ['all', 'all.vault'] or src_list_s in inv.keys(): 55 | dst.append("%s/%s" % (cfg['group_vars_path'], src_list_s)) 56 | 57 | # Add templates into the dst list 58 | for ig in inv.keys(): 59 | if '@' in ig: 60 | g, t = ig.split('@') 61 | 62 | if t == src_list_s: 63 | dst.append("%s/%s" % (cfg['group_vars_path'], ig)) 64 | 65 | # Create all destination symlinks 66 | for d in dst: 67 | # Make the source relative to the destination 68 | s = os.path.relpath(src, os.path.dirname(d)) 69 | 70 | # Clear files and dirs of the same name 71 | try: 72 | if os.path.isdir(d): 73 | os.rmdir(d) 74 | elif os.path.exists(d) or os.path.lexists(d): 75 | os.remove(d) 76 | except Exception as e: 77 | log.error("E: Cannot delete %s.\n%s" % (d, e)) 78 | sys.exit(1) 79 | 80 | # Create new symlink 81 | try: 82 | os.symlink(s, d) 83 | except Exception as e: 84 | log.error("E: Cannot create symlink.\n%s" % e) 85 | sys.exit(1) 86 | 87 | 88 | def read_vars_file(inv, group, cfg, vars_always=False): 89 | g = group 90 | 91 | # Get template name 92 | if '@' in group: 93 | _, g = group.split('@') 94 | 95 | # Do not try to load vault files 96 | if g.endswith('.vault'): 97 | return 98 | 99 | path = "%s/%s" % (cfg['vars_path'], g.replace('-', '/')) 100 | data = None 101 | 102 | # Check if vars file exists 103 | if os.path.isfile(path): 104 | pass 105 | elif os.path.isfile("%s/all" % path): 106 | path += '/all' 107 | else: 108 | path = None 109 | 110 | # Read the group file or the "all" file from the group dir if exists 111 | if path is not None: 112 | try: 113 | data = yaml.safe_load(read_yaml_file(path, False)) 114 | except yaml.YAMLError as e: 115 | log.error("E: Cannot load YAML inventory vars file.\n%s" % e) 116 | sys.exit(1) 117 | 118 | # Create empty group if needed 119 | if group not in inv: 120 | inv[group] = { 121 | 'hosts': [] 122 | } 123 | 124 | # Create empty vars if required 125 | if ( 126 | ( 127 | vars_always or 128 | ( 129 | data is not None and 130 | not cfg['symlinks'])) and 131 | 'vars' not in inv[group]): 132 | inv[group]['vars'] = {} 133 | 134 | # Update the vars with the file data if any 135 | if data is not None and not cfg['symlinks']: 136 | inv[group]['vars'].update(data) 137 | 138 | 139 | def add_param(inv, path, param, val, cfg): 140 | if param.startswith(':'): 141 | param = param[1:] 142 | 143 | _path = list(path) 144 | 145 | if cfg['symlinks'] and cfg['vaults']: 146 | # Create link g1.vault -> g1 147 | _path[-1] += '.vault' 148 | cfg_tmp = dict(cfg) 149 | cfg_tmp['symlinks'] = None 150 | add_param(inv, _path, 'children', ['-'.join(path)], cfg_tmp) 151 | 152 | if isinstance(val, list) and len(val) and param == 'children': 153 | val[0] += '.vault' 154 | 155 | group = '-'.join(path) 156 | 157 | # Add empty group 158 | if group not in inv: 159 | inv[group] = {} 160 | 161 | # Add empty parameter 162 | if param not in inv[group]: 163 | if param == 'vars': 164 | inv[group][param] = {} 165 | else: 166 | inv[group][param] = [] 167 | 168 | # Add parameter value 169 | if isinstance(inv[group][param], dict) and isinstance(val, dict): 170 | inv[group][param].update(val) 171 | elif isinstance(inv[group][param], list) and isinstance(val, list): 172 | # Add individual items if they don't exist 173 | for v in val: 174 | if v not in inv[group][param]: 175 | inv[group][param] += val 176 | 177 | # Read inventory vars file 178 | if not cfg['symlinks']: 179 | read_vars_file(inv, group, cfg) 180 | 181 | 182 | def walk_yaml(inv, data, cfg, parent=None, path=[]): 183 | if data is None: 184 | return 185 | 186 | params = list(k for k in data.keys() if k[0] == ':') 187 | groups = list(k for k in data.keys() if k[0] != ':') 188 | 189 | for p in params: 190 | if parent is None: 191 | _path = ['all'] 192 | else: 193 | _path = list(path) 194 | 195 | if p == ':templates' and parent is not None: 196 | for t in data[p]: 197 | _pth = list(_path) 198 | _pth[-1] += "@%s" % t 199 | 200 | add_param( 201 | inv, _pth, 'children', ['-'.join(_path)], cfg) 202 | 203 | elif p == ':hosts': 204 | for h in data[p]: 205 | # Add host with vars into the _meta hostvars 206 | if isinstance(h, dict): 207 | host_name = list(h.keys())[0] 208 | host_vars = list(h.values())[0] 209 | 210 | # Add host vars 211 | if host_name not in inv['_meta']['hostvars']: 212 | inv['_meta']['hostvars'].update(h) 213 | else: 214 | inv['_meta']['hostvars'][host_name].update(host_vars) 215 | 216 | # Add host 217 | add_param( 218 | inv, _path, p, [host_name], cfg) 219 | else: 220 | add_param(inv, _path, p, [h], cfg) 221 | else: 222 | # Create empty hosts list if :hosts exists but it's empty 223 | add_param(inv, _path, p, [], cfg) 224 | 225 | elif p == ':vars': 226 | add_param(inv, _path, p, data[p], cfg) 227 | 228 | elif p == ':groups' and ':hosts' in data: 229 | for g in data[p]: 230 | g_path = g.split('-') 231 | 232 | # Add hosts in the same way like above 233 | for h in data[':hosts']: 234 | if isinstance(h, dict): 235 | add_param( 236 | inv, g_path, 'hosts', [list(h.keys())[0]], cfg) 237 | else: 238 | add_param( 239 | inv, g_path, 'hosts', [h], cfg) 240 | 241 | elif p == ':add_hosts': 242 | key = '__YAML_INVENTORY' 243 | 244 | if key not in inv: 245 | inv[key] = [] 246 | 247 | record = { 248 | 'path': path, 249 | 'patterns': data[p] 250 | } 251 | 252 | # Make a list of groups which want to add hosts by regexps 253 | inv[key].append(record) 254 | 255 | for g in groups: 256 | if parent is not None: 257 | if ':templates' in data[g]: 258 | if data[g] is not None: 259 | for t in data[g][':templates']: 260 | _path = list(path + [g]) 261 | _path[-1] += "@%s" % t 262 | 263 | add_param( 264 | inv, path, 'children', ['-'.join(_path)], cfg) 265 | else: 266 | add_param( 267 | inv, path, 'children', ['-'.join(path + [g])], cfg) 268 | 269 | walk_yaml(inv, data[g], cfg, g, path + [g]) 270 | 271 | 272 | def read_yaml_file(f_path, strip_hyphens=True): 273 | content = '' 274 | 275 | try: 276 | f = open(f_path, 'r') 277 | except IOError as e: 278 | log.error("E: Cannot open file %s.\n%s" % (f_path, e)) 279 | sys.exit(1) 280 | 281 | for line in f.readlines(): 282 | if not strip_hyphens or strip_hyphens and not line.startswith('---'): 283 | content += line 284 | 285 | try: 286 | f.close() 287 | except IOError as e: 288 | log.error("E: Cannot close file %s.\n%s" % (f_path, e)) 289 | sys.exit(1) 290 | 291 | return content 292 | 293 | 294 | def read_inventory(inventory_path): 295 | # Check if the path is a directory 296 | if not os.path.isdir(inventory_path): 297 | log.error( 298 | "E: No inventory directory %s.\n" 299 | "Use YAML_INVENTORY_PATH environment variable to specify the " 300 | "custom directory." % inventory_path) 301 | sys.exit(1) 302 | 303 | if not ( 304 | os.path.isfile("%s/main.yaml" % inventory_path) or 305 | os.path.isfile("%s/main.yml" % inventory_path)): 306 | log.error( 307 | "E: Cannot find %s/main.yaml." % inventory_path) 308 | sys.exit(1) 309 | 310 | # Get names of all YAML files 311 | yaml_files = glob.glob("%s/*.yaml" % inventory_path) 312 | yaml_files += glob.glob("%s/*.yml" % inventory_path) 313 | 314 | yaml_main = '' 315 | yaml_content = '' 316 | 317 | # Read content of all the files 318 | for f_path in sorted(yaml_files): 319 | file_name = os.path.basename(f_path) 320 | 321 | # Keep content of the main.yaml file in a separate variable 322 | if file_name == 'main.yaml' or file_name == 'main.yml': 323 | yaml_main += read_yaml_file(f_path) 324 | else: 325 | yaml_content += read_yaml_file(f_path) 326 | 327 | # Convert YAML string to data structure 328 | try: 329 | data = yaml.safe_load(yaml_content + yaml_main) 330 | # Remove all YAML references 331 | yaml_main = re.sub(r':\s+\*', ': ', yaml_main).replace('<<:', 'k:') 332 | data_main = yaml.safe_load(yaml_main) 333 | except yaml.YAMLError as e: 334 | log.error("E: Cannot load YAML inventory.\n%s" % e) 335 | sys.exit(1) 336 | 337 | if data is not None: 338 | # Delete all non-main variables 339 | for key in list(data.keys()): 340 | if key not in data_main: 341 | data.pop(key, None) 342 | 343 | return data 344 | 345 | 346 | def my_construct_mapping(self, node, deep=False): 347 | data = self.construct_mapping_org(node, deep) 348 | 349 | return { 350 | (str(key) if isinstance(key, int) else key): data[key] for key in data 351 | } 352 | 353 | 354 | def get_vars(config): 355 | cwd = os.getcwd() 356 | inventory_path = "%s/inventory" % cwd 357 | TRUE = ('1', 'yes', 'y', 'true') 358 | 359 | # Check if there is the config var specifying the inventory dir 360 | if config.has_option('paths', 'inventory_path'): 361 | inventory_path = config.get('paths', 'inventory_path') 362 | 363 | # Check if there is the env var specifying the inventory dir 364 | if 'YAML_INVENTORY_PATH' in os.environ: 365 | inventory_path = os.environ['YAML_INVENTORY_PATH'] 366 | 367 | vars_path = "%s/vars" % inventory_path 368 | 369 | # Check if there is the config var specifying the inventory/vars dir 370 | if config.has_option('paths', 'inventory_vars_path'): 371 | vars_path = config.get('paths', 'inventory_vars_path') 372 | 373 | # Check if there is the env var specifying the inventory/vars dir 374 | if 'YAML_INVENTORY_VARS_PATH' in os.environ: 375 | vars_path = os.environ['YAML_INVENTORY_VARS_PATH'] 376 | 377 | group_vars_path = "%s/group_vars" % cwd 378 | 379 | # Check if there is the config var specifying the group_vars dir 380 | if config.has_option('paths', 'group_vars_path'): 381 | group_vars_path = config.get('paths', 'group_vars_path') 382 | 383 | # Check if there is the env var specifying the group_vars dir 384 | if 'YAML_INVENTORY_GROUP_VARS_PATH' in os.environ: 385 | group_vars_path = os.environ['YAML_INVENTORY_GROUP_VARS_PATH'] 386 | 387 | vaults = True 388 | 389 | # Check if there is the config var specifying the support_vaults flag 390 | if config.has_option('features', 'support_vaults'): 391 | try: 392 | vaults = config.getboolean('features', 'support_vaults') 393 | except ValueError as e: 394 | log.error("E: Wrong value of the support_vaults option.\n%s" % e) 395 | 396 | # Check if there is the env var specifying the support_vaults flag 397 | if ( 398 | 'YAML_INVENTORY_SUPPORT_VAULTS' in os.environ and 399 | os.environ['YAML_INVENTORY_SUPPORT_VAULTS'].lower() not in TRUE): 400 | vaults = False 401 | 402 | symlinks = True 403 | 404 | # Check if there is the config var specifying the create_symlinks flag 405 | if config.has_option('features', 'create_symlinks'): 406 | try: 407 | symlinks = config.getboolean('features', 'create_symlinks') 408 | except ValueError as e: 409 | log.error("E: Wrong value of the create_symlinks option.\n%s" % e) 410 | 411 | # Check if there is the env var specifying the create_symlinks flag 412 | if ( 413 | 'YAML_INVENTORY_CREATE_SYMLINKS' in os.environ and 414 | os.environ['YAML_INVENTORY_CREATE_SYMLINKS'].lower() not in TRUE): 415 | symlinks = False 416 | 417 | cfg = { 418 | 'inventory_path': inventory_path, 419 | 'vars_path': vars_path, 420 | 'group_vars_path': group_vars_path, 421 | 'symlinks': symlinks, 422 | 'vaults': vaults, 423 | } 424 | 425 | return cfg 426 | 427 | 428 | def read_config(): 429 | # Possible config file locations 430 | config_locations = [ 431 | 'yaml_inventory.conf', 432 | os.path.expanduser('~/.ansible/yaml_inventory.conf'), 433 | '/etc/ansible/yaml_inventory.conf' 434 | ] 435 | 436 | # Add the env var into the list if defined 437 | if 'YAML_INVENTORY_CONFIG_PATH' in os.environ: 438 | config_locations = ( 439 | [os.environ['YAML_INVENTORY_CONFIG_PATH']] + config_locations) 440 | 441 | config = configparser.ConfigParser() 442 | 443 | # Search for the config file and read it if found 444 | for cl in config_locations: 445 | if os.path.isfile(cl): 446 | try: 447 | config.read(cl) 448 | except Exception as e: 449 | log.error('E: Cannot read config file %s.\n%s', (cl, e)) 450 | sys.exit(1) 451 | 452 | break 453 | 454 | return config 455 | 456 | 457 | def parse_arguments(): 458 | description = 'Ansible dynamic inventory reading YAML file.' 459 | epilog = ( 460 | 'environment variables:\n' 461 | ' YAML_INVENTORY_CONFIG_PATH\n' 462 | ' location of the config file (default locations:\n' 463 | ' ./yaml_inventory.conf\n' 464 | ' ~/.ansible/yaml_inventory.conf\n' 465 | ' /etc/ansible/yaml_inventory.conf)\n' 466 | ' YAML_INVENTORY_PATH\n' 467 | ' location of the inventory directory (./inventory by default)\n' 468 | ' YAML_INVENTORY_VARS_PATH\n' 469 | ' location of the inventory vars directory ' 470 | '(YAML_INVENTORY_PATH/vars by default)\n' 471 | ' YAML_INVENTORY_GROUP_VARS_PATH\n' 472 | ' location of the vars directory (./group_vars by default)\n' 473 | ' YAML_INVENTORY_SUPPORT_VAULTS\n' 474 | ' flag to take in account .vault files (yes by default)\n' 475 | ' YAML_INVENTORY_CREATE_SYMLINKS\n' 476 | ' flag to create group_vars symlinks (yes by default)') 477 | 478 | parser = argparse.ArgumentParser( 479 | description=description, 480 | epilog=epilog, 481 | formatter_class=argparse.RawDescriptionHelpFormatter) 482 | parser.add_argument( 483 | '--list', 484 | action='store_true', 485 | help='list all groups and hosts') 486 | parser.add_argument( 487 | '--host', 488 | metavar='HOST', 489 | help='get vars for a specific host') 490 | 491 | return (parser.parse_args(), parser) 492 | 493 | 494 | def main(): 495 | # Configure the logger (required for Python v2.6) 496 | logging.basicConfig() 497 | 498 | # Parse command line arguments 499 | (args, parser) = parse_arguments() 500 | 501 | if not args.list and args.host is None: 502 | log.error('No action specified.') 503 | parser.print_help() 504 | sys.exit(1) 505 | 506 | # Read config file 507 | config = read_config() 508 | 509 | # Get config vars 510 | cfg = get_vars(config) 511 | 512 | # Configure YAML parser 513 | yaml.SafeLoader.construct_mapping_org = yaml.SafeLoader.construct_mapping 514 | yaml.SafeLoader.construct_mapping = my_construct_mapping 515 | 516 | # Read the inventory 517 | data = read_inventory(cfg['inventory_path']) 518 | 519 | # Initiate the dynamic inventory 520 | dyn_inv = { 521 | '_meta': { 522 | 'hostvars': {} 523 | } 524 | } 525 | 526 | # Walk through the data structure (start with groups only) 527 | if data is not None: 528 | walk_yaml( 529 | dyn_inv, 530 | dict((k, v) for k, v in data.items() if ( 531 | k[0] != ':' or 532 | k[0] != ':vars')), 533 | cfg) 534 | 535 | # Add hosts by regexp 536 | if '__YAML_INVENTORY' in dyn_inv: 537 | tmp_inv = dict(dyn_inv) 538 | 539 | for inv_group, inv_group_content in tmp_inv.items(): 540 | if ( 541 | not inv_group.endswith('.vault') and 542 | '@' not in inv_group and 543 | 'hosts' in inv_group_content): 544 | for re_data in tmp_inv['__YAML_INVENTORY']: 545 | for pattern in re_data['patterns']: 546 | for host in inv_group_content['hosts']: 547 | if re.match(pattern, host): 548 | add_param( 549 | dyn_inv, re_data['path'], 'hosts', [host], 550 | cfg) 551 | 552 | # Clear the regexp data 553 | tmp_inv = None 554 | dyn_inv.pop('__YAML_INVENTORY', None) 555 | 556 | # Create group_vars symlinks if enabled 557 | if cfg['symlinks']: 558 | create_symlinks(cfg, dyn_inv) 559 | 560 | # Get the host's vars if requested 561 | if args.host is not None: 562 | if args.host in dyn_inv['_meta']['hostvars']: 563 | dyn_inv = dyn_inv['_meta']['hostvars'][args.host] 564 | else: 565 | dyn_inv = {} 566 | 567 | # Print the final dynamic inventory in JSON format 568 | print(json.dumps(dyn_inv, sort_keys=True, indent=2)) 569 | 570 | 571 | if __name__ == '__main__': 572 | main() 573 | --------------------------------------------------------------------------------