├── doc ├── .gitignore ├── source │ ├── substs.inc │ ├── extrefs.inc │ ├── puppet.rst │ ├── refs.rst │ ├── intro.inc │ ├── configfile.rst │ ├── manpage.rst │ ├── usage.rst │ ├── hacking.rst │ ├── index.rst │ ├── install.rst │ ├── changelog.rst │ ├── concepts.rst │ └── todo.rst └── Makefile ├── reclass ├── values │ ├── tests │ │ ├── __init__.py │ │ ├── test_listitem.py │ │ ├── test_scaitem.py │ │ ├── test_item.py │ │ ├── test_refitem.py │ │ ├── test_compitem.py │ │ ├── test_parser_functions.py │ │ └── test_value.py │ ├── dictitem.py │ ├── __init__.py │ ├── listitem.py │ ├── scaitem.py │ ├── compitem.py │ ├── refitem.py │ ├── item.py │ ├── value.py │ └── parser.py ├── tests │ ├── data │ │ ├── 04 │ │ │ ├── classes │ │ │ │ ├── one.yml │ │ │ │ ├── two.yml │ │ │ │ └── three.yml │ │ │ └── nodes │ │ │ │ └── alpha │ │ │ │ └── node1.yml │ │ ├── 01 │ │ │ ├── nodes │ │ │ │ ├── data_types.yml │ │ │ │ └── class_notfound.yml │ │ │ └── classes │ │ │ │ └── standard.yml │ │ ├── 02 │ │ │ ├── classes │ │ │ │ ├── init.yml │ │ │ │ ├── three.yml │ │ │ │ ├── four.yml │ │ │ │ ├── one │ │ │ │ │ ├── beta.yml │ │ │ │ │ └── alpha.yml │ │ │ │ └── two │ │ │ │ │ ├── beta.yml │ │ │ │ │ └── gamma.yml │ │ │ └── nodes │ │ │ │ ├── relative.yml │ │ │ │ └── top_relative.yml │ │ └── 03 │ │ │ ├── nodes │ │ │ ├── alpha │ │ │ │ ├── one.yml │ │ │ │ └── two.yml │ │ │ └── beta │ │ │ │ ├── one.yml │ │ │ │ └── two.yml │ │ │ └── classes │ │ │ ├── a.yml │ │ │ ├── b.yml │ │ │ ├── c.yml │ │ │ └── d.yml │ ├── __init__.py │ └── test_core.py ├── utils │ ├── __init__.py │ ├── tests │ │ ├── __init__.py │ │ └── test_dictpath.py │ ├── parameterdict.py │ ├── parameterlist.py │ └── dictpath.py ├── storage │ ├── tests │ │ ├── __init__.py │ │ ├── test_loader.py │ │ ├── test_yamldata.py │ │ └── test_memcache_proxy.py │ ├── loader.py │ ├── common.py │ ├── __init__.py │ ├── yaml_fs │ │ ├── directory.py │ │ └── __init__.py │ ├── memcache_proxy.py │ ├── mixed │ │ └── __init__.py │ └── yamldata.py ├── datatypes │ ├── tests │ │ ├── __init__.py │ │ ├── test_applications.py │ │ └── test_classes.py │ ├── __init__.py │ ├── applications.py │ ├── classes.py │ ├── exports.py │ └── entity.py ├── adapters │ ├── __init__.py │ └── ansible.py ├── constants.py ├── output │ ├── json_outputter.py │ ├── yaml_outputter.py │ └── __init__.py ├── version.py ├── __init__.py ├── defaults.py ├── cli.py └── settings.py ├── ChangeLog.rst ├── .pylintrc ├── examples ├── ansible │ ├── reclass-config.yml │ ├── hosts │ └── test.yml ├── classes │ ├── webserver.yml │ ├── mail │ │ ├── init.yml │ │ ├── server.yml │ │ ├── relay.yml │ │ └── satellite.yml │ ├── debian │ │ ├── release │ │ │ ├── sid.yml │ │ │ ├── jessie.yml │ │ │ ├── lenny.yml │ │ │ ├── wheezy.yml │ │ │ └── squeeze.yml │ │ ├── suite │ │ │ ├── stable.yml │ │ │ ├── oldstable.yml │ │ │ ├── testing.yml │ │ │ ├── unstable.yml │ │ │ ├── include_multimedia.yml │ │ │ ├── include_backports.yml │ │ │ ├── include_experimental.yml │ │ │ ├── archived.yml │ │ │ └── include_volatile.yml │ │ └── init.yml │ ├── hosted@munich.yml │ ├── hosted@zurich.yml │ ├── salt.minion.yml │ ├── sudo.yml │ └── example.org.yml ├── salt │ ├── reclass-config.yml │ └── reclass └── nodes │ ├── munich │ ├── yellow.example.org.yml │ └── black.example.org.yml │ └── zurich │ ├── white.example.org.yml │ └── blue.example.org.yml ├── test └── model │ ├── default │ ├── reclass-config.yml │ ├── nodes │ │ └── reclass.yml │ └── classes │ │ ├── lab │ │ └── env │ │ │ └── dev.yml │ │ ├── second.yml │ │ ├── third.yml │ │ └── first.yml │ └── extensions │ ├── nodes │ └── reclass.yml │ ├── classes │ ├── relative │ │ ├── init.yml │ │ └── nested │ │ │ ├── deep │ │ │ ├── common.yml │ │ │ └── init.yml │ │ │ ├── dive │ │ │ └── session.yml │ │ │ ├── common.yml │ │ │ └── init.yml │ ├── lab │ │ └── env │ │ │ └── dev.yml │ ├── defaults.yml │ ├── first.yml │ ├── second.yml │ └── third.yml │ └── reclass-config.yml ├── requirements.txt ├── .gitignore ├── releasenotes ├── notes │ └── escaping-references-e76699d8ca010013.yaml └── config.yaml ├── setup.cfg ├── Pipfile ├── MANIFEST.in ├── reclass.py ├── run_tests.py ├── .kitchen-verify.sh ├── .kitchen.yml ├── tox.ini ├── README.rst ├── Makefile ├── setup.py ├── .travis.yml └── contrib └── modules ├── pillar └── reclass_adapter.py └── tops └── reclass_adapter.py /doc/.gitignore: -------------------------------------------------------------------------------- 1 | /build 2 | -------------------------------------------------------------------------------- /reclass/values/tests/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /ChangeLog.rst: -------------------------------------------------------------------------------- 1 | doc/source/changelog.rst -------------------------------------------------------------------------------- /.pylintrc: -------------------------------------------------------------------------------- 1 | [MASTER] 2 | 3 | [REPORTS] 4 | reports=no 5 | -------------------------------------------------------------------------------- /doc/source/substs.inc: -------------------------------------------------------------------------------- 1 | .. |reclass| replace:: **reclass** 2 | -------------------------------------------------------------------------------- /examples/ansible/reclass-config.yml: -------------------------------------------------------------------------------- 1 | inventory_base_uri: .. 2 | -------------------------------------------------------------------------------- /examples/classes/webserver.yml: -------------------------------------------------------------------------------- 1 | applications: 2 | - apache 3 | -------------------------------------------------------------------------------- /examples/salt/reclass-config.yml: -------------------------------------------------------------------------------- 1 | inventory_base_uri: .. 2 | -------------------------------------------------------------------------------- /test/model/default/reclass-config.yml: -------------------------------------------------------------------------------- 1 | storage_type: yaml_fs 2 | -------------------------------------------------------------------------------- /examples/classes/mail/init.yml: -------------------------------------------------------------------------------- 1 | applications: 2 | - postfix 3 | -------------------------------------------------------------------------------- /reclass/tests/data/04/classes/one.yml: -------------------------------------------------------------------------------- 1 | parameters: 2 | test1: 1 3 | -------------------------------------------------------------------------------- /reclass/tests/data/04/classes/two.yml: -------------------------------------------------------------------------------- 1 | parameters: 2 | test2: 2 3 | -------------------------------------------------------------------------------- /reclass/tests/data/04/nodes/alpha/node1.yml: -------------------------------------------------------------------------------- 1 | classes: 2 | - one 3 | -------------------------------------------------------------------------------- /test/model/default/nodes/reclass.yml: -------------------------------------------------------------------------------- 1 | 2 | classes: 3 | - third 4 | -------------------------------------------------------------------------------- /reclass/tests/data/01/nodes/data_types.yml: -------------------------------------------------------------------------------- 1 | classes: 2 | - standard 3 | -------------------------------------------------------------------------------- /reclass/tests/data/02/classes/init.yml: -------------------------------------------------------------------------------- 1 | parameters: 2 | alpha_init: 5 -------------------------------------------------------------------------------- /reclass/tests/data/02/classes/three.yml: -------------------------------------------------------------------------------- 1 | classes: 2 | - .one.alpha 3 | -------------------------------------------------------------------------------- /reclass/tests/data/02/nodes/relative.yml: -------------------------------------------------------------------------------- 1 | classes: 2 | - one.alpha 3 | -------------------------------------------------------------------------------- /reclass/tests/data/02/nodes/top_relative.yml: -------------------------------------------------------------------------------- 1 | classes: 2 | - three 3 | -------------------------------------------------------------------------------- /reclass/tests/data/03/nodes/alpha/one.yml: -------------------------------------------------------------------------------- 1 | classes: 2 | - a 3 | - b 4 | -------------------------------------------------------------------------------- /reclass/tests/data/03/nodes/alpha/two.yml: -------------------------------------------------------------------------------- 1 | classes: 2 | - a 3 | - c 4 | -------------------------------------------------------------------------------- /reclass/tests/data/03/nodes/beta/one.yml: -------------------------------------------------------------------------------- 1 | classes: 2 | - b 3 | - c 4 | -------------------------------------------------------------------------------- /reclass/tests/data/03/nodes/beta/two.yml: -------------------------------------------------------------------------------- 1 | classes: 2 | - c 3 | - d 4 | -------------------------------------------------------------------------------- /reclass/tests/data/04/classes/three.yml: -------------------------------------------------------------------------------- 1 | parameters: 2 | test3: 3 3 | -------------------------------------------------------------------------------- /test/model/extensions/nodes/reclass.yml: -------------------------------------------------------------------------------- 1 | 2 | classes: 3 | - .third 4 | -------------------------------------------------------------------------------- /reclass/tests/data/02/classes/four.yml: -------------------------------------------------------------------------------- 1 | parameters: 2 | four_alpha: 3 3 | -------------------------------------------------------------------------------- /reclass/tests/data/02/classes/one/beta.yml: -------------------------------------------------------------------------------- 1 | parameters: 2 | one_beta: 1 3 | -------------------------------------------------------------------------------- /reclass/tests/data/02/classes/two/beta.yml: -------------------------------------------------------------------------------- 1 | parameters: 2 | two_beta: 2 3 | -------------------------------------------------------------------------------- /reclass/tests/data/02/classes/two/gamma.yml: -------------------------------------------------------------------------------- 1 | parameters: 2 | two_gamma: 4 3 | -------------------------------------------------------------------------------- /test/model/extensions/classes/relative/init.yml: -------------------------------------------------------------------------------- 1 | 2 | classes: 3 | - .nested 4 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | pyparsing 2 | pyyaml 3 | six 4 | enum34 ; python_version<'3.4' 5 | ddt 6 | -------------------------------------------------------------------------------- /test/model/default/classes/lab/env/dev.yml: -------------------------------------------------------------------------------- 1 | 2 | parameters: 3 | lab: 4 | name: dev 5 | -------------------------------------------------------------------------------- /test/model/extensions/classes/lab/env/dev.yml: -------------------------------------------------------------------------------- 1 | 2 | parameters: 3 | lab: 4 | name: dev 5 | -------------------------------------------------------------------------------- /test/model/extensions/classes/defaults.yml: -------------------------------------------------------------------------------- 1 | 2 | parameters: 3 | config: 4 | defaults: True 5 | -------------------------------------------------------------------------------- /examples/classes/mail/server.yml: -------------------------------------------------------------------------------- 1 | classes: 2 | - mail 3 | parameters: 4 | mail: 5 | role: server 6 | -------------------------------------------------------------------------------- /reclass/tests/data/01/classes/standard.yml: -------------------------------------------------------------------------------- 1 | parameters: 2 | int: 1 3 | string: '1' 4 | bool: True 5 | -------------------------------------------------------------------------------- /reclass/tests/data/03/classes/a.yml: -------------------------------------------------------------------------------- 1 | parameters: 2 | a: 1 3 | alpha: 4 | - ${a} 5 | beta: 6 | a: ${a} 7 | -------------------------------------------------------------------------------- /reclass/tests/data/03/classes/b.yml: -------------------------------------------------------------------------------- 1 | parameters: 2 | b: 2 3 | alpha: 4 | - ${b} 5 | beta: 6 | b: ${b} 7 | -------------------------------------------------------------------------------- /reclass/tests/data/03/classes/c.yml: -------------------------------------------------------------------------------- 1 | parameters: 2 | c: 3 3 | alpha: 4 | - ${c} 5 | beta: 6 | c: ${c} 7 | -------------------------------------------------------------------------------- /reclass/tests/data/03/classes/d.yml: -------------------------------------------------------------------------------- 1 | parameters: 2 | d: 4 3 | alpha: 4 | - ${d} 5 | beta: 6 | d: ${d} 7 | -------------------------------------------------------------------------------- /examples/classes/mail/relay.yml: -------------------------------------------------------------------------------- 1 | classes: 2 | - mail 3 | parameters: 4 | mail: 5 | role: relay 6 | port: 587 7 | -------------------------------------------------------------------------------- /examples/classes/debian/release/sid.yml: -------------------------------------------------------------------------------- 1 | classes: 2 | - debian.suite.unstable 3 | parameters: 4 | debian_codename: sid 5 | -------------------------------------------------------------------------------- /examples/salt/reclass: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | cd ../../ 3 | PYTHONPATH="`pwd`:$PYTHONPATH" exec python reclass/adapters/salt.py "$@" 4 | -------------------------------------------------------------------------------- /examples/ansible/hosts: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | cd ../../ 3 | PYTHONPATH="`pwd`:$PYTHONPATH" exec python reclass/adapters/ansible.py "$@" 4 | -------------------------------------------------------------------------------- /examples/classes/debian/release/jessie.yml: -------------------------------------------------------------------------------- 1 | classes: 2 | - debian.suite.testing 3 | parameters: 4 | debian_codename: jessie 5 | -------------------------------------------------------------------------------- /examples/classes/debian/release/lenny.yml: -------------------------------------------------------------------------------- 1 | classes: 2 | - debian.suite.archived 3 | parameters: 4 | debian_codename: lenny 5 | -------------------------------------------------------------------------------- /examples/classes/debian/release/wheezy.yml: -------------------------------------------------------------------------------- 1 | classes: 2 | - debian.suite.stable 3 | parameters: 4 | debian_codename: wheezy 5 | -------------------------------------------------------------------------------- /reclass/tests/data/01/nodes/class_notfound.yml: -------------------------------------------------------------------------------- 1 | classes: 2 | - missing 3 | 4 | parameters: 5 | node_test: class not found 6 | -------------------------------------------------------------------------------- /test/model/extensions/reclass-config.yml: -------------------------------------------------------------------------------- 1 | storage_type: yaml_fs 2 | ignore_class_notfound: True 3 | ignore_class_regexp: ['.*'] 4 | -------------------------------------------------------------------------------- /examples/classes/debian/release/squeeze.yml: -------------------------------------------------------------------------------- 1 | classes: 2 | - debian.suite.oldstable 3 | parameters: 4 | debian_codename: squeeze 5 | -------------------------------------------------------------------------------- /test/model/extensions/classes/relative/nested/deep/common.yml: -------------------------------------------------------------------------------- 1 | 2 | parameters: 3 | nested: 4 | deep: 5 | common: False 6 | -------------------------------------------------------------------------------- /test/model/extensions/classes/relative/nested/dive/session.yml: -------------------------------------------------------------------------------- 1 | 2 | parameters: 3 | nested: 4 | deep: 5 | session: True 6 | -------------------------------------------------------------------------------- /test/model/extensions/classes/first.yml: -------------------------------------------------------------------------------- 1 | parameters: 2 | _param: 3 | some: param 4 | lab: 5 | name: test 6 | label: first 7 | -------------------------------------------------------------------------------- /test/model/extensions/classes/relative/nested/common.yml: -------------------------------------------------------------------------------- 1 | 2 | parameters: 3 | nested: 4 | deep: 5 | common: to be overriden 6 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | *.py[co] 2 | .*.sw? 3 | .DS_Store 4 | /reclass-config.yml 5 | /reclass.egg-info 6 | /build 7 | /dist 8 | /.coverage 9 | .kitchen 10 | -------------------------------------------------------------------------------- /examples/ansible/test.yml: -------------------------------------------------------------------------------- 1 | - name: Test playbook against all test hosts 2 | hosts: test_hosts 3 | tasks: 4 | - name: Greet the world 5 | debug: msg='$greeting' 6 | -------------------------------------------------------------------------------- /releasenotes/notes/escaping-references-e76699d8ca010013.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | others: 3 | - The escaping of references changes how the constructs '\${xxx}' and '\\${xxx}' are rendered. 4 | -------------------------------------------------------------------------------- /test/model/default/classes/second.yml: -------------------------------------------------------------------------------- 1 | classes: 2 | - first 3 | 4 | parameters: 5 | will: 6 | warn: 7 | at: 8 | second: ${_param:notfound} 9 | three: ${one} 10 | -------------------------------------------------------------------------------- /test/model/extensions/classes/second.yml: -------------------------------------------------------------------------------- 1 | classes: 2 | - first 3 | - relative 4 | 5 | parameters: 6 | will: 7 | warn: 8 | at: 9 | second: ${_param:notfound} 10 | -------------------------------------------------------------------------------- /test/model/extensions/classes/relative/nested/deep/init.yml: -------------------------------------------------------------------------------- 1 | 2 | classes: 3 | - .common 4 | 5 | parameters: 6 | nested: 7 | deep: 8 | init: True 9 | common: True 10 | -------------------------------------------------------------------------------- /examples/classes/hosted@munich.yml: -------------------------------------------------------------------------------- 1 | parameters: 2 | location: 3 | address: Briennerstrasse 32, 80333 Munich 4 | rack: 2.64 5 | local_admin: 6 | email: local-admins@munich.example.org 7 | -------------------------------------------------------------------------------- /examples/classes/hosted@zurich.yml: -------------------------------------------------------------------------------- 1 | parameters: 2 | location: 3 | address: Letzigraben 4, 8004 Zurich 4 | rack: C/IV 6.43 5 | local_admin: 6 | email: local-admins@zurich.example.org 7 | -------------------------------------------------------------------------------- /test/model/extensions/classes/relative/nested/init.yml: -------------------------------------------------------------------------------- 1 | 2 | classes: 3 | - .common 4 | - .deep 5 | - .dive.session 6 | 7 | parameters: 8 | nested: 9 | deep: 10 | init: True 11 | -------------------------------------------------------------------------------- /examples/classes/debian/suite/stable.yml: -------------------------------------------------------------------------------- 1 | classes: 2 | - debian 3 | parameters: 4 | debian_suite: stable 5 | apt: 6 | include_security: yes 7 | include_updates: yes 8 | include_proposed_updates: no 9 | -------------------------------------------------------------------------------- /reclass/tests/__init__.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 2 | from __future__ import absolute_import 3 | from __future__ import division 4 | from __future__ import print_function 5 | from __future__ import unicode_literals 6 | -------------------------------------------------------------------------------- /reclass/utils/__init__.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 2 | from __future__ import absolute_import 3 | from __future__ import division 4 | from __future__ import print_function 5 | from __future__ import unicode_literals 6 | -------------------------------------------------------------------------------- /examples/classes/debian/suite/oldstable.yml: -------------------------------------------------------------------------------- 1 | classes: 2 | - debian 3 | parameters: 4 | debian_suite: oldstable 5 | apt: 6 | include_security: yes 7 | include_updates: no 8 | include_proposed_updates: no 9 | -------------------------------------------------------------------------------- /examples/classes/debian/suite/testing.yml: -------------------------------------------------------------------------------- 1 | classes: 2 | - debian 3 | parameters: 4 | debian_suite: testing 5 | apt: 6 | include_security: yes 7 | include_updates: no 8 | include_proposed_updates: no 9 | -------------------------------------------------------------------------------- /examples/classes/debian/suite/unstable.yml: -------------------------------------------------------------------------------- 1 | classes: 2 | - debian 3 | parameters: 4 | debian_suite: unstable 5 | apt: 6 | include_security: no 7 | include_updates: no 8 | include_proposed_updates: no 9 | -------------------------------------------------------------------------------- /reclass/storage/tests/__init__.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 2 | from __future__ import absolute_import 3 | from __future__ import division 4 | from __future__ import print_function 5 | from __future__ import unicode_literals 6 | -------------------------------------------------------------------------------- /reclass/utils/tests/__init__.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 2 | from __future__ import absolute_import 3 | from __future__ import division 4 | from __future__ import print_function 5 | from __future__ import unicode_literals 6 | -------------------------------------------------------------------------------- /reclass/datatypes/tests/__init__.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 2 | from __future__ import absolute_import 3 | from __future__ import division 4 | from __future__ import print_function 5 | from __future__ import unicode_literals 6 | -------------------------------------------------------------------------------- /examples/classes/salt.minion.yml: -------------------------------------------------------------------------------- 1 | applications: 2 | - salt_minion 3 | parameters: 4 | salt_minion: 5 | master: salt-master.example.org 6 | master_fingerprint: ed:38:43:88:4b:2d:22:04:76:60:95:18:2e:cd:cf:bf:cc:63:20:c9 7 | -------------------------------------------------------------------------------- /examples/nodes/munich/yellow.example.org.yml: -------------------------------------------------------------------------------- 1 | classes: 2 | - example.org 3 | - debian.release.wheezy 4 | - hosted@munich 5 | - salt.minion 6 | - mail.relay 7 | environment: dev 8 | parameters: 9 | rgb_colour_code: "00ffff" 10 | -------------------------------------------------------------------------------- /examples/nodes/zurich/white.example.org.yml: -------------------------------------------------------------------------------- 1 | classes: 2 | - example.org 3 | - debian.release.jessie 4 | - hosted@zurich 5 | - salt.minion 6 | - mail.server 7 | environment: prod 8 | parameters: 9 | rgb_colour_code: "ffffff" 10 | -------------------------------------------------------------------------------- /reclass/values/dictitem.py: -------------------------------------------------------------------------------- 1 | # 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of reclass 5 | # 6 | 7 | from reclass.values import item 8 | 9 | 10 | class DictItem(item.ContainerItem): 11 | 12 | type = item.ItemTypes.DICTIONARY 13 | -------------------------------------------------------------------------------- /examples/classes/mail/satellite.yml: -------------------------------------------------------------------------------- 1 | classes: 2 | - mail 3 | parameters: 4 | mail: 5 | role: satellite 6 | smtp_relay: smtp.example.org:587 7 | smtp_relay_fingerprint: 45:88:ff:11:b0:be:39:c8:30:2a:84:bd:fc:6c:52:ff:76:d4:c5:41 8 | tls: enforce 9 | -------------------------------------------------------------------------------- /examples/nodes/munich/black.example.org.yml: -------------------------------------------------------------------------------- 1 | classes: 2 | - example.org 3 | - debian.release.jessie 4 | - hosted@munich 5 | - salt.minion 6 | - webserver 7 | - mail.satellite 8 | environment: dev 9 | parameters: 10 | rgb_colour_code: "000000" 11 | -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | [bdist_wheel] 2 | # This flag says that the code is written to work on both Python 2 and Python 3 | # 3. If at all possible, it is good practice to do this. If you cannot, you 4 | # will need to generate wheels for each Python version that you support. 5 | universal=0 6 | 7 | -------------------------------------------------------------------------------- /reclass/tests/data/02/classes/one/alpha.yml: -------------------------------------------------------------------------------- 1 | classes: 2 | - .beta 3 | - two.beta 4 | - ..four 5 | - ..two.gamma 6 | - ..init 7 | 8 | parameters: 9 | test1: ${one_beta} 10 | test2: ${two_beta} 11 | test3: ${four_alpha} 12 | test4: ${two_gamma} 13 | test5: ${alpha_init} 14 | -------------------------------------------------------------------------------- /test/model/extensions/classes/third.yml: -------------------------------------------------------------------------------- 1 | classes: 2 | - missing.class 3 | - second 4 | - .defaults 5 | 6 | parameters: 7 | _param: 8 | notfound: exist 9 | myparam: ${_param:some} 10 | will: 11 | not: 12 | fail: 13 | at: 14 | tree: ${_param:notfound} 15 | -------------------------------------------------------------------------------- /examples/nodes/zurich/blue.example.org.yml: -------------------------------------------------------------------------------- 1 | classes: 2 | - example.org 3 | - debian.release.wheezy 4 | - debian.suite.include_backports 5 | - hosted@zurich 6 | - salt.minion 7 | - webserver 8 | - mail.satellite 9 | environment: prod 10 | parameters: 11 | rgb_colour_code: "0000ff" 12 | -------------------------------------------------------------------------------- /doc/source/extrefs.inc: -------------------------------------------------------------------------------- 1 | .. _Puppet: http://puppetlabs.com/puppet/puppet-open-source 2 | .. _Salt: http://saltstack.com/community 3 | .. _Ansible: http://www.ansibleworks.com 4 | .. _Hiera: http://projects.puppetlabs.com/projects/hiera 5 | .. _Artistic Licence 2.0: http://opensource.org/licenses/Artistic-2.0 6 | .. _Jinja2: http://jinja.pocoo.org 7 | -------------------------------------------------------------------------------- /Pipfile: -------------------------------------------------------------------------------- 1 | [[source]] 2 | url = "https://pypi.python.org/simple" 3 | verify_ssl = true 4 | name = "pypi" 5 | 6 | [dev-packages] 7 | 8 | [packages] 9 | pyparsing = "*" 10 | PyYAML = "*" 11 | six = "*" 12 | pyyaml = "*" 13 | enum34 = "*" 14 | # FIXME, issues with compile phase 15 | #"pygit2" = "*" 16 | 17 | [requires] 18 | python_version = "2.7" 19 | -------------------------------------------------------------------------------- /reclass/values/__init__.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | from __future__ import absolute_import 3 | from __future__ import division 4 | from __future__ import print_function 5 | from __future__ import unicode_literals 6 | 7 | import collections 8 | 9 | NodeInventory = collections.namedtuple('NodeInventory', ['items', 'env_matches'], rename=False) 10 | -------------------------------------------------------------------------------- /reclass/utils/parameterdict.py: -------------------------------------------------------------------------------- 1 | class ParameterDict(dict): 2 | def __init__(self, *args, **kwargs): 3 | self._uri = kwargs.pop('uri', None) 4 | dict.__init__(self, *args, **kwargs) 5 | 6 | @property 7 | def uri(self): 8 | return self._uri 9 | 10 | @uri.setter 11 | def uri(self, uri): 12 | self._uri = uri 13 | -------------------------------------------------------------------------------- /reclass/utils/parameterlist.py: -------------------------------------------------------------------------------- 1 | class ParameterList(list): 2 | def __init__(self, *args, **kwargs): 3 | self._uri = kwargs.pop('uri', None) 4 | list.__init__(self, *args, **kwargs) 5 | 6 | @property 7 | def uri(self): 8 | return self._uri 9 | 10 | @uri.setter 11 | def uri(self, uri): 12 | self._uri = uri 13 | -------------------------------------------------------------------------------- /examples/classes/sudo.yml: -------------------------------------------------------------------------------- 1 | applications: 2 | - sudo 3 | parameters: 4 | sudo: 5 | opt_lecture: false 6 | opt_ignore_dot: true 7 | opt_listpw: true 8 | opt_insults: true 9 | opt_requiretty: true 10 | opt_tty_tickets: true 11 | opt_passwd_tries: 1 12 | opt_secure_path: "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" 13 | no_passwd_group: wheel 14 | -------------------------------------------------------------------------------- /examples/classes/debian/suite/include_multimedia.yml: -------------------------------------------------------------------------------- 1 | classes: 2 | - debian 3 | parameters: 4 | apt: 5 | repo_uri_multimedia: http://deb-multimedia.org 6 | include_multimedia: yes 7 | apt_repos: 8 | - id: debian-multimedia 9 | enabled: ${apt:include_multimedia} 10 | uri: ${apt:repo_uri_multimedia} 11 | components: ${apt:default_components} 12 | sources: ${apt:include_sources} 13 | -------------------------------------------------------------------------------- /reclass/adapters/__init__.py: -------------------------------------------------------------------------------- 1 | # 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of reclass (http://github.com/madduck/reclass) 5 | # 6 | # Copyright © 2007–14 martin f. krafft 7 | # Released under the terms of the Artistic Licence 2.0 8 | 9 | from __future__ import absolute_import 10 | from __future__ import division 11 | from __future__ import print_function 12 | from __future__ import unicode_literals 13 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | # Include the license and changelog 2 | include LICENSE ChangeLog.rst 3 | # Exclude development tooling 4 | exclude Makefile requirements.txt .pylintrc reclass.py 5 | # Exclude testing infra 6 | exclude run_tests.py 7 | prune reclass/tests 8 | prune reclass/datatypes/tests 9 | prune reclass/storage/tests 10 | prune reclass/utils/tests 11 | prune reclass/values/tests 12 | # Exclude "source only" content 13 | prune doc 14 | prune examples 15 | -------------------------------------------------------------------------------- /examples/classes/debian/suite/include_backports.yml: -------------------------------------------------------------------------------- 1 | classes: 2 | - debian 3 | parameters: 4 | apt: 5 | repo_uri_backports: http://http.debian.net/debian 6 | include_backports: yes 7 | apt_repos: 8 | - id: debian-backports 9 | enabled: ${apt:include_backports} 10 | uri: ${apt:repo_uri_backports} 11 | suite_postfix: -backports 12 | components: ${apt:default_components} 13 | sources: ${apt:include_sources} 14 | -------------------------------------------------------------------------------- /examples/classes/debian/suite/include_experimental.yml: -------------------------------------------------------------------------------- 1 | classes: 2 | - debian 3 | parameters: 4 | apt: 5 | repo_uri_experimental: ${apt:repo_uri} 6 | include_experimental: yes 7 | apt_repos: 8 | - id: debian-experimental 9 | enabled: ${apt:include_experimental} 10 | uri: ${apt:repo_uri_experimental} 11 | suite: experimental 12 | components: ${apt:default_components} 13 | sources: ${apt:include_sources} 14 | -------------------------------------------------------------------------------- /reclass.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of reclass (http://github.com/madduck/reclass) 5 | # 6 | # Copyright © 2007–13 martin f. krafft 7 | # Released under the terms of the Artistic Licence 2.0 8 | # 9 | from __future__ import absolute_import 10 | from __future__ import division 11 | from __future__ import print_function 12 | from __future__ import unicode_literals 13 | 14 | import reclass.cli 15 | reclass.cli.main() 16 | -------------------------------------------------------------------------------- /reclass/values/listitem.py: -------------------------------------------------------------------------------- 1 | # 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of reclass 5 | # 6 | 7 | from reclass.values import item 8 | 9 | 10 | class ListItem(item.ContainerItem): 11 | 12 | type = item.ItemTypes.LIST 13 | 14 | def merge_over(self, other): 15 | if other.type == item.ItemTypes.LIST: 16 | other.contents.extend(self.contents) 17 | return other 18 | raise RuntimeError('Failed to merge %s over %s' % (self, other)) 19 | -------------------------------------------------------------------------------- /test/model/default/classes/third.yml: -------------------------------------------------------------------------------- 1 | classes: 2 | - second 3 | 4 | parameters: 5 | _param: 6 | notfound: exist 7 | myparam: ${_param:some} 8 | will: 9 | not: 10 | fail: 11 | at: 12 | tree: ${_param:notfound} 13 | 1: 14 | an_numeric_key: true 15 | as_a_dict: 1 16 | 2: 17 | - as_a_list 18 | 3: value 19 | three: ${two} 20 | empty: 21 | list: [] 22 | dict: {} 23 | ~list_to_override: ${empty:list} 24 | ~dict_to_override: ${empty:dict} 25 | -------------------------------------------------------------------------------- /run_tests.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of reclass (http://github.com/madduck/reclass) 5 | # 6 | # Copyright © 2007–13 martin f. krafft 7 | # Released under the terms of the Artistic Licence 2.0 8 | # 9 | from __future__ import absolute_import 10 | from __future__ import division 11 | from __future__ import print_function 12 | from __future__ import unicode_literals 13 | 14 | import unittest 15 | tests = unittest.TestLoader().discover('reclass') 16 | unittest.TextTestRunner(verbosity=1).run(tests) 17 | -------------------------------------------------------------------------------- /examples/classes/debian/suite/archived.yml: -------------------------------------------------------------------------------- 1 | classes: 2 | - debian 3 | parameters: 4 | debian_suite: archived 5 | apt: 6 | repo_uri: http://archive.debian.org/debian 7 | repo_uri_security: http://archive.debian.org/debian-security 8 | repo_uri_backports: http://archive.debian.org/debian-backports 9 | repo_uri_volatile: http://archive.debian.org/debian-volatile 10 | include_security: no 11 | include_updates: no 12 | include_proposed_updates: no 13 | motd: 14 | newsitems: 15 | - This host is no longer kept up-to-date and will be decomissioned soon. 16 | -------------------------------------------------------------------------------- /test/model/default/classes/first.yml: -------------------------------------------------------------------------------- 1 | parameters: 2 | _param: 3 | some: param 4 | colour: red 5 | lab: 6 | name: test 7 | label: first 8 | colour: 9 | escaped: \${_param:colour} 10 | doubleescaped: \\${_param:colour} 11 | unescaped: ${_param:colour} 12 | colours: 13 | red: 14 | name: red 15 | blue: 16 | name: blue 17 | one: 18 | a: 1 19 | b: 2 20 | two: 21 | c: 3 22 | d: 4 23 | three: 24 | e: 5 25 | list_to_override: 26 | - one 27 | - two 28 | dict_to_override: 29 | one: 1 30 | two: 2 31 | 32 | 33 | -------------------------------------------------------------------------------- /doc/source/puppet.rst: -------------------------------------------------------------------------------- 1 | ========================= 2 | Using reclass with Puppet 3 | ========================= 4 | 5 | .. todo:: 6 | 7 | The adapter between |reclass| and `Puppet`_ has not actually been written, 8 | since I rage-quit using Puppet before the rewrite of |reclass|. 9 | 10 | It should be trivial to do, and if you need it or are interested in working 11 | on it, and you require assistance, please get in touch with me `on the 12 | mailing list `_. Else just send the 13 | patch! 14 | 15 | .. include:: extrefs.inc 16 | .. include:: substs.inc 17 | -------------------------------------------------------------------------------- /reclass/datatypes/__init__.py: -------------------------------------------------------------------------------- 1 | # 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of reclass (http://github.com/madduck/reclass) 5 | # 6 | # Copyright © 2007–14 martin f. krafft 7 | # Released under the terms of the Artistic Licence 2.0 8 | # 9 | from __future__ import absolute_import 10 | from __future__ import division 11 | from __future__ import print_function 12 | from __future__ import unicode_literals 13 | 14 | from .applications import Applications 15 | from .classes import Classes 16 | from .entity import Entity 17 | from .exports import Exports 18 | from .parameters import Parameters 19 | -------------------------------------------------------------------------------- /.kitchen-verify.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | #set -x 3 | 4 | # setup 5 | source /*.env 6 | INVENTORY_BASE_URI=/tmp/kitchen/test/model/$MODEL 7 | RECLASS=/tmp/kitchen 8 | 9 | # prereq 10 | python -m ensurepip --default-pip 11 | pip install pipenv 12 | 13 | # env 14 | cd $RECLASS 15 | pipenv --venv || pipenv install --python ${PYVER} 16 | test -e /etc/reclsss || mkdir /etc/reclass 17 | cp -avf $INVENTORY_BASE_URI/reclass-config* /etc/reclass 18 | 19 | # verify 20 | for n in $(ls $INVENTORY_BASE_URI/nodes/*|sort); do 21 | pipenv run python${PYVER} ./reclass.py --inventory-base-uri=$INVENTORY_BASE_URI --nodeinfo $(basename $n .yml) 22 | done 23 | -------------------------------------------------------------------------------- /examples/classes/example.org.yml: -------------------------------------------------------------------------------- 1 | classes: 2 | - sudo # all nodes in the example.org domain provide sudo 3 | applications: 4 | - motd # all nodes in the example.org domain are expected to provide /etc/motd 5 | parameters: 6 | motd: 7 | legalese: This system is for authorized users only. All traffic on this 8 | device is monitored and will be used as evidence if necessary. Use your 9 | brain. 10 | support: "Please write a message to <${local_admin:email}> in case of problems." 11 | location: "Rack ${location:rack}, ${location:address}" 12 | tagline: "My hostname's RGB colour code is ${rgb_colour_code}." 13 | -------------------------------------------------------------------------------- /examples/classes/debian/suite/include_volatile.yml: -------------------------------------------------------------------------------- 1 | classes: 2 | - debian 3 | parameters: 4 | apt: 5 | repo_uri_volatile: ${repo_uri}-volatile 6 | include_volatile: True 7 | include_volatile_sloppy: False 8 | apt_repos: 9 | - id: debian-volatile 10 | enabled: ${apt:include_volatile} 11 | uri: ${apt:repo_uri_volatile} 12 | components: ${apt:default_components} 13 | sources: ${apt:include_sources} 14 | - id: debian-volatile-sloppy 15 | enabled: ${apt:include_volatile_sloppy} 16 | uri: ${apt:repo_uri_volatile}-sloppy 17 | components: ${apt:default_components} 18 | sources: ${apt:include_sources} 19 | -------------------------------------------------------------------------------- /reclass/constants.py: -------------------------------------------------------------------------------- 1 | # 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of reclass (http://github.com/madduck/reclass) 5 | # 6 | # Copyright © 2007–14 martin f. krafft 7 | # Released under the terms of the Artistic Licence 2.0 8 | # 9 | from __future__ import absolute_import 10 | from __future__ import division 11 | from __future__ import print_function 12 | from __future__ import unicode_literals 13 | 14 | 15 | class _Constant(object): 16 | 17 | def __init__(self, displayname): 18 | self._repr = displayname 19 | 20 | __str__ = __repr__ = lambda self: self._repr 21 | 22 | MODE_NODEINFO = _Constant('NODEINFO') 23 | MODE_NODEAPPS = _Constant('NODEAPPS') 24 | MODE_INVENTORY = _Constant('INVENTORY') 25 | -------------------------------------------------------------------------------- /reclass/output/json_outputter.py: -------------------------------------------------------------------------------- 1 | # 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of reclass (http://github.com/madduck/reclass) 5 | # 6 | # Copyright © 2007–14 martin f. krafft 7 | # Released under the terms of the Artistic Licence 2.0 8 | # 9 | from __future__ import absolute_import 10 | from __future__ import division 11 | from __future__ import print_function 12 | from __future__ import unicode_literals 13 | 14 | from reclass.output import OutputterBase 15 | import json 16 | 17 | 18 | class Outputter(OutputterBase): 19 | 20 | def dump(self, data, pretty_print=False, no_refs=False): 21 | separators = (',', ': ') if pretty_print else (',', ':') 22 | indent = 2 if pretty_print else None 23 | return json.dumps(data, indent=indent, separators=separators) 24 | -------------------------------------------------------------------------------- /reclass/storage/tests/test_loader.py: -------------------------------------------------------------------------------- 1 | # 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of reclass (http://github.com/madduck/reclass) 5 | # 6 | # Copyright © 2007–14 martin f. krafft 7 | # Released under the terms of the Artistic Licence 2.0 8 | # 9 | from __future__ import absolute_import 10 | from __future__ import division 11 | from __future__ import print_function 12 | from __future__ import unicode_literals 13 | 14 | from reclass.storage.loader import StorageBackendLoader 15 | 16 | import unittest 17 | 18 | 19 | class TestLoader(unittest.TestCase): 20 | 21 | def test_load(self): 22 | loader = StorageBackendLoader('yaml_fs') 23 | from reclass.storage.yaml_fs import ExternalNodeStorage as YamlFs 24 | self.assertEqual(loader.load(), YamlFs) 25 | 26 | if __name__ == '__main__': 27 | unittest.main() 28 | -------------------------------------------------------------------------------- /reclass/values/scaitem.py: -------------------------------------------------------------------------------- 1 | # 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of reclass 5 | # 6 | from __future__ import absolute_import 7 | from __future__ import division 8 | from __future__ import print_function 9 | from __future__ import unicode_literals 10 | 11 | from reclass.settings import Settings 12 | from reclass.values import item 13 | 14 | 15 | class ScaItem(item.Item): 16 | 17 | type = item.ItemTypes.SCALAR 18 | 19 | def __init__(self, value, settings): 20 | super(ScaItem, self).__init__(value, settings) 21 | 22 | def merge_over(self, other): 23 | if other.type in [item.ItemTypes.SCALAR, item.ItemTypes.COMPOSITE]: 24 | return self 25 | raise RuntimeError('Failed to merge %s over %s' % (self, other)) 26 | 27 | def render(self, context, inventory): 28 | return self.contents 29 | 30 | def __str__(self): 31 | return str(self.contents) 32 | -------------------------------------------------------------------------------- /reclass/values/compitem.py: -------------------------------------------------------------------------------- 1 | # 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of reclass 5 | # 6 | 7 | from reclass.settings import Settings 8 | from reclass.values import item 9 | 10 | 11 | class CompItem(item.ItemWithReferences): 12 | 13 | type = item.ItemTypes.COMPOSITE 14 | 15 | def merge_over(self, other): 16 | if (other.type == item.ItemTypes.SCALAR or 17 | other.type == item.ItemTypes.COMPOSITE): 18 | return self 19 | raise RuntimeError('Failed to merge %s over %s' % (self, other)) 20 | 21 | def render(self, context, inventory): 22 | # Preserve type if only one item 23 | if len(self.contents) == 1: 24 | return self.contents[0].render(context, inventory) 25 | # Multiple items 26 | strings = [str(i.render(context, inventory)) for i in self.contents] 27 | return "".join(strings) 28 | 29 | def __str__(self): 30 | return ''.join([str(i) for i in self.contents]) 31 | -------------------------------------------------------------------------------- /.kitchen.yml: -------------------------------------------------------------------------------- 1 | --- 2 | driver: 3 | name: docker 4 | priviledged: false 5 | use_sudo: false 6 | volume: 7 | - <%= ENV['PWD'] %>:/tmp/kitchen 8 | 9 | 10 | provisioner: 11 | name: shell 12 | script: .kitchen-verify.sh 13 | 14 | 15 | verifier: 16 | name: inspec 17 | 18 | <%- pyver = ENV['PYTHON_VERSION'] || '2.7' %> 19 | 20 | platforms: 21 | <% `find test/model -maxdepth 1 -mindepth 1 -type d |sort -u`.split().each do |model| %> 22 | <% model=model.split('/')[2] %> 23 | - name: <%= model %> 24 | driver_config: 25 | image: python:<%= pyver %> 26 | platform: ubuntu 27 | hostname: reclass 28 | provision_command: 29 | #FIXME, setup reclass env (prereq, configs, upload models) 30 | #- apt-get install -y rsync 31 | - echo " 32 | export LC_ALL=C.UTF-8;\n 33 | export LANG=C.UTF-8;\n 34 | export PYVER=<%= pyver %>;\n 35 | export MODEL=<%= model %>;\n 36 | " > /kitchen.env 37 | <% end %> 38 | 39 | suites: 40 | - name: model 41 | 42 | -------------------------------------------------------------------------------- /reclass/version.py: -------------------------------------------------------------------------------- 1 | # 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of reclass (http://github.com/madduck/reclass) 5 | # 6 | # Copyright © 2007–14 martin f. krafft 7 | # Released under the terms of the Artistic Licence 2.0 8 | # 9 | from __future__ import absolute_import 10 | from __future__ import division 11 | from __future__ import print_function 12 | from __future__ import unicode_literals 13 | 14 | RECLASS_NAME = 'reclass' 15 | DESCRIPTION = ('merge data by recursive descent down an ancestry hierarchy ' 16 | '(forked extended version)') 17 | VERSION = '1.7.0' 18 | AUTHOR = 'martin f. krafft / Andrew Pickford / salt-formulas community' 19 | AUTHOR_EMAIL = 'salt-formulas@freelists.org' 20 | MAINTAINER = 'salt-formulas community' 21 | MAINTAINER_EMAIL = 'salt-formulas@freelists.org' 22 | COPYRIGHT = ('Copyright © 2007–14 martin f. krafft, extensions © 2017 Andrew' 23 | ' Pickford, extensions © salt-formulas community') 24 | LICENCE = 'Artistic Licence 2.0' 25 | URL = 'https://github.com/salt-formulas/reclass' 26 | -------------------------------------------------------------------------------- /reclass/values/tests/test_listitem.py: -------------------------------------------------------------------------------- 1 | from reclass.settings import Settings 2 | from reclass.values.value import Value 3 | from reclass.values.compitem import CompItem 4 | from reclass.values.scaitem import ScaItem 5 | from reclass.values.valuelist import ValueList 6 | from reclass.values.listitem import ListItem 7 | from reclass.values.dictitem import DictItem 8 | import unittest 9 | 10 | SETTINGS = Settings() 11 | 12 | class TestListItem(unittest.TestCase): 13 | 14 | def test_merge_over_merge_list(self): 15 | listitem1 = ListItem([1], SETTINGS) 16 | listitem2 = ListItem([2], SETTINGS) 17 | expected = ListItem([1, 2], SETTINGS) 18 | 19 | result = listitem2.merge_over(listitem1) 20 | 21 | self.assertEquals(result.contents, expected.contents) 22 | 23 | def test_merge_other_types_not_allowed(self): 24 | other = type('Other', (object,), {'type': 34}) 25 | val1 = Value(None, SETTINGS, '') 26 | listitem = ListItem(val1, SETTINGS) 27 | 28 | self.assertRaises(RuntimeError, listitem.merge_over, other) 29 | 30 | if __name__ == '__main__': 31 | unittest.main() 32 | -------------------------------------------------------------------------------- /reclass/output/yaml_outputter.py: -------------------------------------------------------------------------------- 1 | # 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of reclass (http://github.com/madduck/reclass) 5 | # 6 | # Copyright © 2007–14 martin f. krafft 7 | # Released under the terms of the Artistic Licence 2.0 8 | # 9 | from __future__ import absolute_import 10 | from __future__ import division 11 | from __future__ import print_function 12 | from __future__ import unicode_literals 13 | 14 | from reclass.output import OutputterBase 15 | import yaml 16 | 17 | _SafeDumper = yaml.CSafeDumper if yaml.__with_libyaml__ else yaml.SafeDumper 18 | 19 | 20 | class Outputter(OutputterBase): 21 | 22 | def dump(self, data, pretty_print=False, no_refs=False): 23 | if (no_refs): 24 | return yaml.dump(data, default_flow_style=not pretty_print, Dumper=ExplicitDumper) 25 | else: 26 | return yaml.dump(data, default_flow_style=not pretty_print, Dumper=_SafeDumper) 27 | 28 | 29 | class ExplicitDumper(_SafeDumper): 30 | """ 31 | A dumper that will never emit aliases. 32 | """ 33 | 34 | def ignore_aliases(self, data): 35 | return True 36 | -------------------------------------------------------------------------------- /tox.ini: -------------------------------------------------------------------------------- 1 | # this file is *not* meant to cover or endorse the use of tox or pytest or 2 | # testing in general, 3 | # 4 | # It's meant to show the use of: 5 | # 6 | # - check-manifest 7 | # confirm items checked into vcs are in your sdist 8 | # - python setup.py check (using the readme_renderer extension) 9 | # confirms your long_description will render correctly on pypi 10 | # 11 | # and also to help confirm pull requests to this project. 12 | 13 | [tox] 14 | envlist = py{27} 15 | 16 | [testenv] 17 | basepython = 18 | py27: python2.7 19 | whitelist_externals= 20 | make 21 | deps = 22 | check-manifest 23 | {py27}: readme_renderer 24 | # flake8 out of the picture right now 25 | pytest 26 | mock 27 | pylint 28 | nose 29 | commands = 30 | check-manifest --ignore tox.ini,tests* 31 | {py27}: python setup.py check -m -r -s 32 | # flake8 . # FIXME: This code smell check goes poorly for us at present 33 | make tests 34 | # make lint-errors # FIXME: Cause these to operate properly inside tox 35 | # make coverage 36 | [flake8] 37 | exclude = .tox,*.egg,build,data 38 | select = E,W,F 39 | -------------------------------------------------------------------------------- /reclass/__init__.py: -------------------------------------------------------------------------------- 1 | # 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of reclass (http://github.com/madduck/reclass) 5 | # 6 | # Copyright © 2007–14 martin f. krafft 7 | # Released under the terms of the Artistic Licence 2.0 8 | # 9 | from __future__ import absolute_import 10 | from __future__ import division 11 | from __future__ import print_function 12 | from __future__ import unicode_literals 13 | 14 | from .output import OutputLoader 15 | from .storage.loader import StorageBackendLoader 16 | from .storage.memcache_proxy import MemcacheProxy 17 | 18 | def get_storage(storage_type, nodes_uri, classes_uri, compose_node_name, **kwargs): 19 | storage_class = StorageBackendLoader(storage_type).load() 20 | return MemcacheProxy(storage_class(nodes_uri, classes_uri, compose_node_name, **kwargs)) 21 | 22 | def get_path_mangler(storage_type, **kwargs): 23 | return StorageBackendLoader(storage_type).path_mangler() 24 | 25 | def output(data, fmt, pretty_print=False, no_refs=False): 26 | output_class = OutputLoader(fmt).load() 27 | outputter = output_class() 28 | return outputter.dump(data, pretty_print=pretty_print, no_refs=no_refs) 29 | -------------------------------------------------------------------------------- /reclass/output/__init__.py: -------------------------------------------------------------------------------- 1 | # 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of reclass (http://github.com/madduck/reclass) 5 | # 6 | # Copyright © 2007–14 martin f. krafft 7 | # Released under the terms of the Artistic Licence 2.0 8 | # 9 | from __future__ import absolute_import 10 | from __future__ import division 11 | from __future__ import print_function 12 | from __future__ import unicode_literals 13 | 14 | class OutputterBase(object): 15 | 16 | def __init__(self): 17 | pass 18 | 19 | def dump(self, data, pretty_print=False): 20 | raise NotImplementedError("dump() method not implemented.") 21 | 22 | 23 | class OutputLoader(object): 24 | 25 | def __init__(self, outputter): 26 | self._name = 'reclass.output.' + outputter + '_outputter' 27 | try: 28 | self._module = __import__(self._name, globals(), locals(), self._name) 29 | except ImportError: 30 | raise NotImplementedError() 31 | 32 | def load(self, attr='Outputter'): 33 | klass = getattr(self._module, attr, None) 34 | if klass is None: 35 | raise AttributeError('Outputter class {0} does not export "{1}"'.format(self._name, klass)) 36 | return klass 37 | -------------------------------------------------------------------------------- /reclass/values/tests/test_scaitem.py: -------------------------------------------------------------------------------- 1 | from reclass.settings import Settings 2 | from reclass.values.value import Value 3 | from reclass.values.compitem import CompItem 4 | from reclass.values.scaitem import ScaItem 5 | from reclass.values.valuelist import ValueList 6 | from reclass.values.listitem import ListItem 7 | from reclass.values.dictitem import DictItem 8 | import unittest 9 | 10 | SETTINGS = Settings() 11 | 12 | class TestScaItem(unittest.TestCase): 13 | 14 | def test_merge_over_merge_scalar(self): 15 | scalar1 = ScaItem([1], SETTINGS) 16 | scalar2 = ScaItem([2], SETTINGS) 17 | 18 | result = scalar2.merge_over(scalar1) 19 | 20 | self.assertEquals(result.contents, scalar2.contents) 21 | 22 | def test_merge_over_merge_composite(self): 23 | scalar1 = CompItem(Value(1, SETTINGS, ''), SETTINGS) 24 | scalar2 = ScaItem([2], SETTINGS) 25 | 26 | result = scalar2.merge_over(scalar1) 27 | 28 | self.assertEquals(result.contents, scalar2.contents) 29 | 30 | def test_merge_other_types_not_allowed(self): 31 | other = type('Other', (object,), {'type': 34}) 32 | val1 = Value(None, SETTINGS, '') 33 | scalar = ScaItem(val1, SETTINGS) 34 | 35 | self.assertRaises(RuntimeError, scalar.merge_over, other) 36 | 37 | if __name__ == '__main__': 38 | unittest.main() 39 | -------------------------------------------------------------------------------- /doc/source/refs.rst: -------------------------------------------------------------------------------- 1 | =================== 2 | External references 3 | =================== 4 | 5 | * I `presented reclass`__ at `LCA 2014`_, which as been recorded: 6 | 7 | * (Slides forthcoming) 8 | * `Video recording`__ 9 | 10 | __ http://linux.conf.au/schedule/30203/view_talk?day=wednesday 11 | __ http://mirror.linux.org.au/pub/linux.conf.au/2014/Wednesday/59-Hierarchical_infrastructure_description_for_your_system_management_needs_-_Martin_Krafft.mp4 12 | 13 | .. _LCA 2014: https://lca2014.linux.org.au 14 | 15 | * I gave `a talk about reclass`__ at `DebConf13`_, which has been recorded: 16 | 17 | * `Slides`__ 18 | * Video recording: `high quality (ogv)`__ | `high quality (webm)`__ | `low(er) quality (ogv)`__ 19 | 20 | __ http://penta.debconf.org/dc13_schedule/events/1048.en.html 21 | __ http://annex.debconf.org/debconf-share/debconf13/slides/reclass.pdf 22 | __ http://meetings-archive.debian.net/pub/debian-meetings/2013/debconf13/high/1048_Recursive_node_classification_for_system_automation.ogv 23 | __ http://meetings-archive.debian.net/pub/debian-meetings/2013/debconf13/webm-high/1048_Recursive_node_classification_for_system_automation.webm 24 | __ http://meetings-archive.debian.net/pub/debian-meetings/2013/debconf13/low/1048_Recursive_node_classification_for_system_automation.ogv 25 | 26 | .. _DebConf13: http://debconf13.debconf.org 27 | 28 | .. include:: substs.inc 29 | -------------------------------------------------------------------------------- /reclass/storage/loader.py: -------------------------------------------------------------------------------- 1 | # 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of reclass (http://github.com/madduck/reclass) 5 | # 6 | # Copyright © 2007–14 martin f. krafft 7 | # Released under the terms of the Artistic Licence 2.0 8 | # 9 | from __future__ import absolute_import 10 | from __future__ import division 11 | from __future__ import print_function 12 | from __future__ import unicode_literals 13 | 14 | from importlib import import_module 15 | 16 | class StorageBackendLoader(object): 17 | 18 | def __init__(self, storage_name): 19 | self._name = str('reclass.storage.' + storage_name) 20 | try: 21 | self._module = import_module(self._name) 22 | except ImportError as e: 23 | raise NotImplementedError 24 | 25 | def load(self, klassname='ExternalNodeStorage'): 26 | klass = getattr(self._module, klassname, None) 27 | if klass is None: 28 | raise AttributeError('Storage backend class {0} does not export ' 29 | '"{1}"'.format(self._name, klassname)) 30 | return klass 31 | 32 | def path_mangler(self, name='path_mangler'): 33 | function = getattr(self._module, name, None) 34 | if function is None: 35 | raise AttributeError('Storage backend class {0} does not export ' 36 | '"{1}"'.format(self._name, name)) 37 | return function 38 | -------------------------------------------------------------------------------- /reclass/storage/tests/test_yamldata.py: -------------------------------------------------------------------------------- 1 | # 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of reclass (http://github.com/madduck/reclass) 5 | # 6 | from __future__ import absolute_import 7 | from __future__ import division 8 | from __future__ import print_function 9 | from __future__ import unicode_literals 10 | 11 | from reclass.storage.yamldata import YamlData 12 | 13 | import unittest 14 | 15 | class TestYamlData(unittest.TestCase): 16 | 17 | def setUp(self): 18 | lines = [ 'classes:', 19 | ' - testdir.test1', 20 | ' - testdir.test2', 21 | ' - test3', 22 | '', 23 | 'environment: base', 24 | '', 25 | 'parameters:', 26 | ' _TEST_:', 27 | ' alpha: 1', 28 | ' beta: two' ] 29 | self.data = '\n'.join(lines) 30 | self.yamldict = { 'classes': [ 'testdir.test1', 'testdir.test2', 'test3' ], 31 | 'environment': 'base', 32 | 'parameters': { '_TEST_': { 'alpha': 1, 'beta': 'two' } } 33 | } 34 | 35 | def test_yaml_from_string(self): 36 | res = YamlData.from_string(self.data, 'testpath') 37 | self.assertEqual(res.uri, 'testpath') 38 | self.assertEqual(res.get_data(), self.yamldict) 39 | 40 | if __name__ == '__main__': 41 | unittest.main() 42 | -------------------------------------------------------------------------------- /reclass/storage/common.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 2 | from __future__ import absolute_import 3 | from __future__ import division 4 | from __future__ import print_function 5 | from __future__ import unicode_literals 6 | 7 | import os 8 | 9 | class NameMangler: 10 | @staticmethod 11 | def nodes(relpath, name): 12 | # nodes are identified just by their basename, so 13 | # no mangling required 14 | return relpath, name 15 | 16 | @staticmethod 17 | def composed_nodes(relpath, name): 18 | if relpath == '.' or relpath == '': 19 | # './' is converted to None 20 | return None, name 21 | parts = relpath.split(os.path.sep) 22 | if parts[0].startswith("_"): 23 | return relpath, name 24 | parts.append(name) 25 | return relpath, '.'.join(parts) 26 | 27 | @staticmethod 28 | def classes(relpath, name): 29 | if relpath == '.' or relpath == '': 30 | # './' is converted to None 31 | return None, name 32 | parts = relpath.split(os.path.sep) 33 | if name != 'init': 34 | # "init" is the directory index, so only append the basename 35 | # to the path parts for all other filenames. This has the 36 | # effect that data in file "foo/init.yml" will be registered 37 | # as data for class "foo", not "foo.init" 38 | parts.append(name) 39 | return relpath, '.'.join(parts) 40 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | Reclass README 2 | ========================= 3 | 4 | This is the fork of original **reclass** that is available at: 5 | https://github.com/madduck/reclass 6 | 7 | Extentions 8 | ========== 9 | 10 | List of the core features: 11 | 12 | * Escaping of References and Inventory Queries 13 | * Merging Referenced Lists and Dictionaries 14 | * Nested References 15 | * Inventory Queries 16 | * Ignore class notfound/regexp option 17 | 18 | 19 | Documentation 20 | ============= 21 | 22 | .. _README-extensions: README-extensions.rst 23 | 24 | Documentation covering the original version is in the doc directory. 25 | See the `README-extensions`_ file for documentation on the extentions. 26 | 27 | 28 | .. include:: ./README-extensions.rst 29 | 30 | 31 | Reclass related projects/tools 32 | ============================== 33 | 34 | Queries: 35 | 36 | * yg, yaml grep with 'jq' syntax - https://gist.github.com/epcim/f1c5b748fa7c942de50677aef04f29f8, (https://asciinema.org/a/84173) 37 | * reclass-graph - https://github.com/tomkukral/reclass-graph 38 | 39 | Introspection, manupulation: 40 | 41 | * reclass-tools, for manipulating reclass models - https://github.com/dis-xcom/reclass_tools 42 | 43 | YAML merge tools: 44 | 45 | * spruce, general purpose YAML & JSON merging tool - https://github.com/geofffranks/spruce 46 | 47 | Other: 48 | 49 | * saltclass, new pillar/master_tops module for salt with the behaviour of reclass - https://github.com/saltstack/salt/pull/42349 50 | 51 | -------------------------------------------------------------------------------- /doc/source/intro.inc: -------------------------------------------------------------------------------- 1 | |reclass| is an "external node classifier" (ENC) as can be used with 2 | automation tools, such as `Puppet`_, `Salt`_, and `Ansible`_. It is also 3 | a stand-alone tool for merging data sources recursively. 4 | 5 | The purpose of an ENC is to allow a system administrator to maintain an 6 | inventory of nodes to be managed, completely separately from the configuration 7 | of the automation tool. Usually, the external node classifier completely 8 | replaces the tool-specific inventory (such as ``site.pp`` for Puppet, 9 | ``ext_pillar``/``master_tops`` for Salt, or ``/etc/ansible/hosts``). 10 | 11 | With respect to the configuration management tool, the ENC then fulfills two 12 | jobs: 13 | 14 | - it provides information about groups of nodes and group memberships 15 | - it gives access to node-specific information, such as variables 16 | 17 | |reclass| allows you to define your nodes through class inheritance, while 18 | always able to override details further up the tree (i.e. in more specific 19 | nodes). Think of classes as feature sets, as commonalities between nodes, or 20 | as tags. Add to that the ability to nest classes (multiple inheritance is 21 | allowed, well-defined, and encouraged), and you can assemble your 22 | infrastructure from smaller bits, eliminating duplication and exposing all 23 | important parameters to a single location, logically organised. And if that 24 | isn't enough, |reclass| lets you reference other parameters in the very 25 | hierarchy you are currently assembling. 26 | -------------------------------------------------------------------------------- /doc/source/configfile.rst: -------------------------------------------------------------------------------- 1 | ========================== 2 | reclass configuration file 3 | ========================== 4 | |reclass| can read some of its configuration from a file. The file is 5 | a YAML-file and simply defines key-value pairs. 6 | 7 | The configuration file can be used to set defaults for all the options that 8 | are otherwise configurable via the command-line interface, so please use the 9 | ``--help`` output of |reclass| (or the :doc:`manual page `) for 10 | reference. The command-line option ``--nodes-uri`` corresponds to the key 11 | ``nodes_uri`` in the configuration file. For example:: 12 | 13 | storage_type: yaml_fs 14 | pretty_print: True 15 | output: json 16 | inventory_base_uri: /etc/reclass 17 | nodes_uri: ../nodes 18 | 19 | |reclass| first looks in the current directory for the file called 20 | ``reclass-config.yml`` (see ``reclass/defaults.py``) and if no such file is 21 | found, it looks in ``$HOME``, then in ``/etc/reclass``, and then "next to" the 22 | ``reclass`` script itself, i.e. if the script is symlinked to 23 | ``/srv/provisioning/reclass``, then the the script will try to access 24 | ``/srv/provisioning/reclass-config.yml``. 25 | 26 | Note that ``yaml_fs`` is currently the only supported ``storage_type``, and 27 | it's the default if you don't set it. 28 | 29 | Adapters may implement their own lookup logic, of course, so make sure to read 30 | their documentation (for :doc:`Salt `, for :doc:`Ansible `, and 31 | for :doc:`Puppet `). 32 | 33 | .. include:: substs.inc 34 | -------------------------------------------------------------------------------- /reclass/values/refitem.py: -------------------------------------------------------------------------------- 1 | # 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of reclass 5 | # 6 | 7 | from reclass.values import item 8 | from reclass.utils.dictpath import DictPath 9 | from reclass.errors import ResolveError 10 | 11 | 12 | class RefItem(item.ItemWithReferences): 13 | 14 | type = item.ItemTypes.REFERENCE 15 | 16 | def assembleRefs(self, context={}): 17 | super(RefItem, self).assembleRefs(context) 18 | try: 19 | self._refs.append(self._flatten_contents(context)) 20 | except ResolveError as e: 21 | self.allRefs = False 22 | 23 | def _flatten_contents(self, context, inventory=None): 24 | result = [str(i.render(context, inventory)) for i in self.contents] 25 | return "".join(result) 26 | 27 | def _resolve(self, ref, context): 28 | path = DictPath(self._settings.delimiter, ref) 29 | try: 30 | return path.get_value(context) 31 | except (KeyError, TypeError) as e: 32 | raise ResolveError(ref) 33 | 34 | def render(self, context, inventory): 35 | #strings = [str(i.render(context, inventory)) for i in self.contents] 36 | #return self._resolve("".join(strings), context) 37 | return self._resolve(self._flatten_contents(context, inventory), 38 | context) 39 | 40 | def __str__(self): 41 | strings = [str(i) for i in self.contents] 42 | rs = self._settings.reference_sentinels 43 | return '{0}{1}{2}'.format(rs[0], ''.join(strings), rs[1]) 44 | -------------------------------------------------------------------------------- /examples/classes/debian/init.yml: -------------------------------------------------------------------------------- 1 | applications: 2 | - apt 3 | - locales 4 | parameters: 5 | debian_stable_suite: wheezy 6 | apt: 7 | repo_uri: http://http.debian.net/debian 8 | repo_uri_security: http://security.debian.org/debian-security 9 | default_components: main # TODO: pass as a list! 10 | include_sources: no 11 | include_security: yes 12 | include_updates: yes 13 | include_proposed_updates: no 14 | disable_sources_dir: no 15 | disable_preferences_dir: no 16 | acquire_pdiffs: no 17 | install_recommends: no 18 | cache_limit: 67108864 19 | apt_repos: 20 | - id: debian 21 | enabled: yes 22 | uri: ${apt:repo_uri} 23 | components: ${apt:default_components} 24 | sources: ${apt:include_sources} 25 | - id: debian-security 26 | enabled: ${apt:include_security} 27 | uri: ${apt:repo_uri_security} 28 | suite_postfix: /updates 29 | components: ${apt:default_components} 30 | sources: ${apt:include_sources} 31 | - id: debian-updates 32 | enabled: ${apt:include_updates} 33 | suite_postfix: -updates 34 | uri: ${apt:repo_uri} 35 | components: ${apt:default_components} 36 | sources: ${apt:include_sources} 37 | - id: debian-proposed-updates 38 | enabled: ${apt:include_proposed_updates} 39 | uri: ${apt:repo_uri} 40 | suite_postfix: -proposed-updates 41 | components: ${apt:default_components} 42 | sources: ${apt:include_sources} 43 | locales: 44 | list: 45 | - en_NZ.UTF-8 UTF-8 46 | - de_CH.UTF-8 UTF-8 47 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | # 2 | # This file is part of reclass (http://github.com/madduck/reclass) 3 | # 4 | # Copyright © 2007–13 martin f. krafft 5 | # Released under the terms of the Artistic Licence 2.0 6 | # 7 | PYFILES = $(shell find -name .git -o -name dist -o -name build -prune -o -name '*.py' -print) 8 | 9 | tests: 10 | python ./run_tests.py 11 | .PHONY: tests 12 | 13 | lint: 14 | @echo pylint --rcfile=.pylintrc $(ARGS) … 15 | @pylint --rcfile=.pylintrc $(ARGS) $(PYFILES) 16 | .PHONY: lint 17 | 18 | lint-errors: ARGS=--errors-only 19 | lint-errors: lint 20 | .PHONY: lint-errors 21 | 22 | lint-report: ARGS=--report=y 23 | lint-report: lint 24 | .PHONY: lint-report 25 | 26 | coverage: .coverage 27 | python-coverage -r -m 28 | .PHONY: coverage 29 | .coverage: $(PYFILES) 30 | python-coverage -x setup.py nosetests 31 | 32 | docs: 33 | $(MAKE) -C doc man html 34 | 35 | GH_BRANCH=gh-pages 36 | HTMLDIR=doc/build/html 37 | docspub: 38 | ifeq ($(shell git branch --list $(GH_BRANCH)-base),) 39 | @echo "Please fetch the $(GH_BRANCH)-base branch from Github to be able to publish documentation:" >&2 40 | @echo " git branch gh-pages-base origin/gh-pages-base" >&2 41 | @false 42 | else 43 | $(MAKE) docs 44 | git checkout $(GH_BRANCH) || git checkout -b $(GH_BRANCH) $(GH_BRANCH)-base 45 | git reset --hard $(GH_BRANCH)-base 46 | git add $(HTMLDIR) 47 | git mv $(HTMLDIR)/* . 48 | git commit -m'Webpage update' 49 | git push -f $(shell git config --get branch.$(GH_BRANCH)-base.remote) $(GH_BRANCH) 50 | git checkout '@{-1}' 51 | endif 52 | 53 | docsclean: 54 | $(MAKE) -C doc clean 55 | -------------------------------------------------------------------------------- /reclass/values/tests/test_item.py: -------------------------------------------------------------------------------- 1 | from reclass.settings import Settings 2 | from reclass.values.value import Value 3 | from reclass.values.compitem import CompItem 4 | from reclass.values.scaitem import ScaItem 5 | from reclass.values.valuelist import ValueList 6 | from reclass.values.listitem import ListItem 7 | from reclass.values.dictitem import DictItem 8 | from reclass.values.item import ContainerItem 9 | from reclass.values.item import ItemWithReferences 10 | import unittest 11 | try: 12 | import unittest.mock as mock 13 | except ImportError: 14 | import mock 15 | 16 | SETTINGS = Settings() 17 | 18 | 19 | class TestItemWithReferences(unittest.TestCase): 20 | 21 | def test_assembleRef_allrefs(self): 22 | phonyitem = mock.MagicMock() 23 | phonyitem.has_references = True 24 | phonyitem.get_references = lambda *x: [1] 25 | 26 | iwr = ItemWithReferences([phonyitem], {}) 27 | 28 | self.assertEquals(iwr.get_references(), [1]) 29 | self.assertTrue(iwr.allRefs) 30 | 31 | def test_assembleRef_partial(self): 32 | phonyitem = mock.MagicMock() 33 | phonyitem.has_references = True 34 | phonyitem.allRefs = False 35 | phonyitem.get_references = lambda *x: [1] 36 | 37 | iwr = ItemWithReferences([phonyitem], {}) 38 | 39 | self.assertEquals(iwr.get_references(), [1]) 40 | self.assertFalse(iwr.allRefs) 41 | 42 | 43 | class TestContainerItem(unittest.TestCase): 44 | 45 | def test_render(self): 46 | container = ContainerItem('foo', SETTINGS) 47 | 48 | self.assertEquals(container.render(None, None), 'foo') 49 | 50 | if __name__ == '__main__': 51 | unittest.main() 52 | -------------------------------------------------------------------------------- /doc/source/manpage.rst: -------------------------------------------------------------------------------- 1 | =============== 2 | reclass manpage 3 | =============== 4 | 5 | Synopsis 6 | -------- 7 | | |reclass| --help 8 | | |reclass| *[options]* --inventory 9 | | |reclass| *[options]* --nodeinfo=NODENAME 10 | 11 | Description 12 | ----------- 13 | .. include:: intro.inc 14 | 15 | |reclass| will be used indirectly through adapters most of the time. However, 16 | there exists a command-line interface that allows querying the database. This 17 | manual page describes this interface. 18 | 19 | Options 20 | ------- 21 | Please see the output of ``reclass --help`` for the default values of these 22 | options: 23 | 24 | Database options 25 | '''''''''''''''' 26 | -s, --storage-type The type of storage backend to use 27 | -b, --inventory-base-uri The base URI to prepend to nodes and classes 28 | -u, --nodes-uri The URI to the nodes storage 29 | -c, --classes-uri The URI to the classes storage 30 | 31 | Output options 32 | '''''''''''''' 33 | -o, --output The output format to use (yaml or json) 34 | -y, --pretty-print Try to make the output prettier 35 | 36 | Modes 37 | ''''' 38 | -i, --inventory Output the entire inventory 39 | -n, --nodeinfo Output information for a specific node 40 | 41 | Information 42 | ''''''''''' 43 | -h, --help Help output 44 | --version Display version number 45 | 46 | See also 47 | -------- 48 | Please visit http://reclass.pantsfullofunix.net/ for more information about 49 | |reclass|. 50 | 51 | The documentation is also available from the ``./doc`` subtree in the source 52 | checkout, or from ``/usr/share/doc/reclass-doc``. 53 | 54 | .. include:: substs.inc 55 | .. include:: extrefs.inc 56 | -------------------------------------------------------------------------------- /reclass/storage/__init__.py: -------------------------------------------------------------------------------- 1 | # 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of reclass (http://github.com/madduck/reclass) 5 | # 6 | # Copyright © 2007–14 martin f. krafft 7 | # Released under the terms of the Artistic Licence 2.0 8 | # 9 | from __future__ import absolute_import 10 | from __future__ import division 11 | from __future__ import print_function 12 | from __future__ import unicode_literals 13 | 14 | from reclass.storage.common import NameMangler 15 | 16 | class NodeStorageBase(object): 17 | 18 | def __init__(self, name): 19 | self._name = name 20 | 21 | name = property(lambda self: self._name) 22 | 23 | def get_node(self, name, settings): 24 | msg = "Storage class '{0}' does not implement node entity retrieval." 25 | raise NotImplementedError(msg.format(self.name)) 26 | 27 | def get_class(self, name, environment, settings): 28 | msg = "Storage class '{0}' does not implement class entity retrieval." 29 | raise NotImplementedError(msg.format(self.name)) 30 | 31 | def enumerate_nodes(self): 32 | msg = "Storage class '{0}' does not implement node enumeration." 33 | raise NotImplementedError(msg.format(self.name)) 34 | 35 | def path_mangler(self): 36 | msg = "Storage class '{0}' does not implement path_mangler." 37 | raise NotImplementedError(msg.format(self.name)) 38 | 39 | 40 | class ExternalNodeStorageBase(NodeStorageBase): 41 | 42 | def __init__(self, name, compose_node_name): 43 | super(ExternalNodeStorageBase, self).__init__(name) 44 | self.class_name_mangler = NameMangler.classes 45 | if compose_node_name: 46 | self.node_name_mangler = NameMangler.composed_nodes 47 | else: 48 | self.node_name_mangler = NameMangler.nodes 49 | -------------------------------------------------------------------------------- /reclass/values/tests/test_refitem.py: -------------------------------------------------------------------------------- 1 | from reclass import errors 2 | 3 | from reclass.settings import Settings 4 | from reclass.values.value import Value 5 | from reclass.values.compitem import CompItem 6 | from reclass.values.scaitem import ScaItem 7 | from reclass.values.valuelist import ValueList 8 | from reclass.values.listitem import ListItem 9 | from reclass.values.dictitem import DictItem 10 | from reclass.values.refitem import RefItem 11 | import unittest 12 | try: 13 | import unittest.mock as mock 14 | except ImportError: 15 | import mock 16 | 17 | SETTINGS = Settings() 18 | 19 | class TestRefItem(unittest.TestCase): 20 | 21 | def test_assembleRefs_ok(self): 22 | phonyitem = mock.MagicMock() 23 | phonyitem.render = lambda x, k: 'bar' 24 | phonyitem.has_references = True 25 | phonyitem.get_references = lambda *x: ['foo'] 26 | 27 | iwr = RefItem([phonyitem], {}) 28 | 29 | self.assertEquals(iwr.get_references(), ['foo', 'bar']) 30 | self.assertTrue(iwr.allRefs) 31 | 32 | def test_assembleRefs_failedrefs(self): 33 | phonyitem = mock.MagicMock() 34 | phonyitem.render.side_effect = errors.ResolveError('foo') 35 | phonyitem.has_references = True 36 | phonyitem.get_references = lambda *x: ['foo'] 37 | 38 | iwr = RefItem([phonyitem], {}) 39 | 40 | self.assertEquals(iwr.get_references(), ['foo']) 41 | self.assertFalse(iwr.allRefs) 42 | 43 | def test__resolve_ok(self): 44 | reference = RefItem('', Settings({'delimiter': ':'})) 45 | 46 | result = reference._resolve('foo:bar', {'foo':{'bar': 1}}) 47 | 48 | self.assertEquals(result, 1) 49 | 50 | def test__resolve_fails(self): 51 | refitem = RefItem('', Settings({'delimiter': ':'})) 52 | context = {'foo':{'bar': 1}} 53 | reference = 'foo:baz' 54 | 55 | self.assertRaises(errors.ResolveError, refitem._resolve, reference, 56 | context) 57 | 58 | 59 | if __name__ == '__main__': 60 | unittest.main() 61 | -------------------------------------------------------------------------------- /doc/source/usage.rst: -------------------------------------------------------------------------------- 1 | ============= 2 | Using reclass 3 | ============= 4 | .. todo:: 5 | 6 | With |reclass| now in Debian, as well as installable from source, the 7 | following should be checked for path consistency… 8 | 9 | For information on how to use |reclass| directly, call ``reclass --help`` 10 | and study the output, or have a look at its :doc:`manual page `. 11 | 12 | The three options, ``--inventory-base-uri``, ``--nodes-uri``, and 13 | ``--classes-uri`` together specify the location of the inventory. If the base 14 | URI is specified, then it is prepended to the other two URIs, unless they are 15 | absolute URIs. If these two URIs are not specified, they default to ``nodes`` 16 | and ``classes``. Therefore, if your inventory is in ``/etc/reclass/nodes`` and 17 | ``/etc/reclass/classes``, all you need to specify is the base URI as 18 | ``/etc/reclass`` — which is actually the default (specified in 19 | ``reclass/defaults.py``). 20 | 21 | If you've installed |reclass| from source as per the :doc:`installation 22 | instructions `, try to run it from the source directory like this:: 23 | 24 | $ reclass -b examples/ --inventory 25 | $ reclass -b examples/ --nodeinfo localhost 26 | 27 | This will make it use the data from ``examples/nodes`` and 28 | ``examples/classes``, and you can surely make your own way from here. 29 | 30 | On Debian-systems, use the following:: 31 | 32 | $ reclass -b /usr/share/doc/reclass/examples/ --inventory 33 | $ reclass -b /usr/share/doc/reclass/examples/ --nodeinfo localhost 34 | 35 | More commonly, however, use of |reclass| will happen indirectly, and through 36 | so-called adapters. The job of an adapter is to translate between different 37 | invocation paradigms, provide a sane set of default options, and massage the 38 | data from |reclass| into the format expected by the automation tool in use. 39 | Please have a look at the respective README files for these adapters, i.e. 40 | for :doc:`Salt `, for :doc:`Ansible `, and for :doc:`Puppet 41 | `. 42 | 43 | .. include:: substs.inc 44 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | # 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of reclass (http://github.com/madduck/reclass) 5 | # 6 | # Copyright © 2007–13 martin f. krafft 7 | # Released under the terms of the Artistic Licence 2.0 8 | # 9 | from __future__ import absolute_import 10 | from __future__ import division 11 | from __future__ import print_function 12 | from __future__ import unicode_literals 13 | 14 | from reclass.version import * 15 | from setuptools import setup, find_packages 16 | 17 | # use consistent encoding of readme for pypi 18 | from codecs import open 19 | from os import path 20 | 21 | here = path.abspath(path.dirname(__file__)) 22 | 23 | # Get the long description from the README file 24 | with open(path.join(here, 'README.rst'), encoding='utf-8') as f: 25 | long_description = f.read() 26 | 27 | ADAPTERS = ['salt', 'ansible'] 28 | console_scripts = ['reclass = reclass.cli:main'] 29 | console_scripts.extend('reclass-{0} = reclass.adapters.{0}:cli'.format(i) 30 | for i in ADAPTERS) 31 | 32 | setup( 33 | name = RECLASS_NAME, 34 | description = DESCRIPTION, 35 | long_description=long_description, 36 | version = VERSION, 37 | author = AUTHOR, 38 | author_email = AUTHOR_EMAIL, 39 | maintainer = MAINTAINER, 40 | maintainer_email = MAINTAINER_EMAIL, 41 | license = LICENCE, 42 | url = URL, 43 | packages = find_packages(exclude=['*tests']), #FIXME validate this 44 | entry_points = { 'console_scripts': console_scripts }, 45 | install_requires = ['pyparsing', 'pyyaml', 'six', 'ddt'], #FIXME pygit2 (require libffi-dev, libgit2-dev 0.26.x ) 46 | extras_require = { 47 | ":python_version<'3.4'": ['enum34'], 48 | }, 49 | 50 | classifiers=[ 51 | 'Development Status :: 4 - Beta', 52 | 'Intended Audience :: System Administrators', 53 | 'Topic :: System :: Systems Administration', 54 | 'License :: OSI Approved :: Artistic License', 55 | 'Programming Language :: Python :: 2.7', 56 | ], 57 | 58 | keywords='enc ansible salt' 59 | ) 60 | -------------------------------------------------------------------------------- /reclass/defaults.py: -------------------------------------------------------------------------------- 1 | # 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of reclass (http://github.com/madduck/reclass) 5 | # 6 | # Copyright © 2007–14 martin f. krafft 7 | # Released under the terms of the Artistic Licence 2.0 8 | # 9 | from __future__ import absolute_import 10 | from __future__ import division 11 | from __future__ import print_function 12 | from __future__ import unicode_literals 13 | 14 | import os, sys 15 | from .version import RECLASS_NAME 16 | 17 | # defaults for the command-line options 18 | OPT_STORAGE_TYPE = 'yaml_fs' 19 | OPT_INVENTORY_BASE_URI = os.path.join('/etc', RECLASS_NAME) 20 | OPT_NODES_URI = 'nodes' 21 | OPT_CLASSES_URI = 'classes' 22 | OPT_PRETTY_PRINT = True 23 | OPT_GROUP_ERRORS = True 24 | OPT_COMPOSE_NODE_NAME = False 25 | OPT_NO_REFS = False 26 | OPT_OUTPUT = 'yaml' 27 | 28 | OPT_IGNORE_CLASS_NOTFOUND = False 29 | OPT_IGNORE_CLASS_NOTFOUND_REGEXP = ['.*'] 30 | OPT_IGNORE_CLASS_NOTFOUND_WARNING = True 31 | 32 | OPT_IGNORE_OVERWRITTEN_MISSING_REFERENCES = True 33 | OPT_STRICT_CONSTANT_PARAMETERS = True 34 | 35 | OPT_ALLOW_ADAPTER_ENV_OVERRIDE = False 36 | 37 | OPT_ALLOW_SCALAR_OVER_DICT = False 38 | OPT_ALLOW_SCALAR_OVER_LIST = False 39 | OPT_ALLOW_LIST_OVER_SCALAR = False 40 | OPT_ALLOW_DICT_OVER_SCALAR = False 41 | OPT_ALLOW_NONE_OVERRIDE = False 42 | 43 | OPT_INVENTORY_IGNORE_FAILED_NODE = False 44 | OPT_INVENTORY_IGNORE_FAILED_RENDER = False 45 | 46 | CONFIG_FILE_SEARCH_PATH = [os.getcwd(), 47 | os.path.expanduser('~'), 48 | OPT_INVENTORY_BASE_URI, 49 | os.path.dirname(sys.argv[0]) 50 | ] 51 | CONFIG_FILE_NAME = RECLASS_NAME + '-config.yml' 52 | 53 | REFERENCE_SENTINELS = ('${', '}') 54 | EXPORT_SENTINELS = ('$[', ']') 55 | PARAMETER_INTERPOLATION_DELIMITER = ':' 56 | PARAMETER_DICT_KEY_OVERRIDE_PREFIX = '~' 57 | PARAMETER_DICT_KEY_CONSTANT_PREFIX = '=' 58 | ESCAPE_CHARACTER = '\\' 59 | 60 | AUTOMATIC_RECLASS_PARAMETERS = True 61 | SCALAR_RECLASS_PARAMETERS = False 62 | DEFAULT_ENVIRONMENT = 'base' 63 | 64 | CLASS_MAPPINGS_MATCH_PATH = False 65 | -------------------------------------------------------------------------------- /releasenotes/config.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | # Usage: 3 | # 4 | # reno -qd .releasenotes list 5 | # reno -qd .releasenotes new slug-title --edit 6 | # reno -qd .releasenotes report --no-show-source 7 | 8 | # Change prelude_section_name to 'summary' from default value prelude 9 | prelude_section_name: summary 10 | show_source: False 11 | sections: 12 | - [summary, Summary] 13 | - [features, New features] 14 | - [fixes, Bug fixes] 15 | - [others, Other notes] 16 | template: | 17 | --- 18 | # Author the following sections or remove the section if it is not related. 19 | # Use one release note per a feature. 20 | # 21 | # If you miss a section from the list below, please first submit a review 22 | # adding it to .releasenotes/config.yaml. 23 | # 24 | # Format content with reStructuredText (RST). 25 | # **Formatting examples:** 26 | # - | 27 | # This is a brief description of the feature. It may include a 28 | # number of components: 29 | # 30 | # * List item 1 31 | # * List item 2. 32 | # This code block below will appear as part of the list item 2: 33 | # 34 | # .. code-block:: yaml 35 | # 36 | # classes: 37 | # - system.class.to.load 38 | # 39 | # The code block below will appear on the same level as the feature 40 | # description: 41 | # 42 | # .. code-block:: text 43 | # 44 | # provide model/formula pillar snippets 45 | 46 | 47 | summary: > 48 | This section is not mandatory. Use it to highlight the change. 49 | 50 | features: 51 | - Use the list to record summary of **NEW** features 52 | - Provide detailed description of the feature indicating the use cases 53 | when users benefit from using it 54 | - Provide steps to deploy the feature (if the procedure is complicated 55 | indicate during what stage of the deployment workflow it should be 56 | deployed). 57 | - Provide troubleshooting information, if any. 58 | 59 | fixes: 60 | - Use the list to record summary of a bug fix for blocker, critical. 61 | - Provide a brief summary of what has been fixed. 62 | 63 | others: 64 | - Author any additional notes. Use this section if note is not related to 65 | any of the common sections above. 66 | 67 | -------------------------------------------------------------------------------- /reclass/cli.py: -------------------------------------------------------------------------------- 1 | # 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of reclass (http://github.com/madduck/reclass) 5 | # 6 | # Copyright © 2007–14 martin f. krafft 7 | # Released under the terms of the Artistic Licence 2.0 8 | # 9 | from __future__ import absolute_import 10 | from __future__ import division 11 | from __future__ import print_function 12 | from __future__ import unicode_literals 13 | 14 | import sys, os, posix 15 | 16 | from reclass import get_storage, output 17 | from reclass.core import Core 18 | from reclass.settings import Settings 19 | from reclass.config import find_and_read_configfile, get_options 20 | from reclass.defaults import * 21 | from reclass.errors import ReclassException 22 | from reclass.constants import MODE_NODEINFO, MODE_NODEAPPS 23 | from reclass.version import * 24 | 25 | def main(): 26 | try: 27 | defaults = {'no_refs' : OPT_NO_REFS, 28 | 'pretty_print' : OPT_PRETTY_PRINT, 29 | 'output' : OPT_OUTPUT 30 | } 31 | defaults.update(find_and_read_configfile()) 32 | 33 | options = get_options(RECLASS_NAME, VERSION, DESCRIPTION, defaults=defaults) 34 | storage = get_storage(options.storage_type, 35 | options.nodes_uri, 36 | options.classes_uri, 37 | options.compose_node_name) 38 | class_mappings = defaults.get('class_mappings') 39 | defaults.update(vars(options)) 40 | settings = Settings(defaults) 41 | reclass = Core(storage, class_mappings, settings) 42 | 43 | if options.mode == MODE_NODEINFO: 44 | data = reclass.nodeinfo(options.nodename, override_environment=options.environment) 45 | elif options.mode == MODE_NODEAPPS: 46 | nodeinfo = reclass.nodeinfo(options.nodename, override_environment=options.environment) 47 | data = { 'applications': nodeinfo['applications'] } 48 | else: 49 | data = reclass.inventory() 50 | 51 | print(output(data, options.output, options.pretty_print, options.no_refs)) 52 | 53 | except ReclassException as e: 54 | e.exit_with_message(sys.stderr) 55 | 56 | sys.exit(posix.EX_OK) 57 | 58 | if __name__ == '__main__': 59 | main() 60 | -------------------------------------------------------------------------------- /doc/source/hacking.rst: -------------------------------------------------------------------------------- 1 | ================== 2 | Hacking on reclass 3 | ================== 4 | 5 | Installation 6 | ------------ 7 | If you just want to run |reclass| from source, e.g. because you are going to be 8 | making and testing changes, install it in "development mode":: 9 | 10 | python setup.py develop 11 | 12 | Now the ``reclass`` script, as well as the adapters, will be available in 13 | ``/usr/local/bin``, and you can also invoke them directly from the source 14 | tree. 15 | 16 | To uninstall:: 17 | 18 | python setup.py develop --uninstall 19 | 20 | Discussing reclass 21 | ------------------ 22 | If you want to talk about |reclass|, use the `mailing list`_ or to find me on 23 | IRC, in ``#reclass`` on ``irc.oftc.net``. 24 | 25 | .. _mailing list: http://lists.pantsfullofunix.net/listinfo/reclass 26 | 27 | Contributing to reclass 28 | ----------------------- 29 | |reclass| is currently maintained `on Github 30 | `_. 31 | 32 | Conttributions to |reclass| are very welcome. Since I prefer to keep a somewhat 33 | clean history, I will not just merge pull request. 34 | 35 | You can submit pull requests, of course, and I'll rebase them onto ``HEAD`` 36 | before merging. Or send your patches using ``git-format-patch`` and 37 | ``git-send-e-mail`` to `the mailing list 38 | `_. 39 | 40 | I have added rudimentary unit tests, and it would be nice if you could submit 41 | your changes with appropriate changes to the tests. To run tests, invoke 42 | 43 | :: 44 | 45 | $ make tests 46 | 47 | in the top-level checkout directory. The tests are rather inconsistent, some 48 | using mock objects, and only the datatypes-related code is covered. If you are 49 | a testing expert, I could certainly use some help here to improve the 50 | consistency of the existing tests, as well as their coverage. 51 | 52 | Also, there is a Makefile giving access to PyLint and ``coverage.py`` (running 53 | tests). If you run that, you can see there is a lot of work to be done 54 | cleaning up the code. If this is the sort of stuff you want to do — by all 55 | means — be my guest! ;) 56 | 57 | There are a number of items on the :doc:`to-do list `, so if you are 58 | bored… 59 | 60 | If you have larger ideas, I'll be looking forward to discuss them with you. 61 | 62 | .. include:: substs.inc 63 | -------------------------------------------------------------------------------- /reclass/storage/yaml_fs/directory.py: -------------------------------------------------------------------------------- 1 | # 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of reclass (http://github.com/madduck/reclass) 5 | # 6 | # Copyright © 2007–14 martin f. krafft 7 | # Released under the terms of the Artistic Licence 2.0 8 | # 9 | from __future__ import absolute_import 10 | from __future__ import division 11 | from __future__ import print_function 12 | from __future__ import unicode_literals 13 | 14 | import os 15 | from reclass.errors import NotFoundError 16 | 17 | SKIPDIRS = ('CVS', 'SCCS') 18 | FILE_EXTENSION = ('.yml', '.yaml') 19 | 20 | def vvv(msg): 21 | #print(msg, file=sys.stderr) 22 | pass 23 | 24 | 25 | class Directory(object): 26 | 27 | def __init__(self, path, fileclass=None): 28 | ''' Initialise a directory object ''' 29 | if not os.path.isdir(path): 30 | raise NotFoundError('No such directory: %s' % path) 31 | if not os.access(path, os.R_OK|os.X_OK): 32 | raise NotFoundError('Cannot change to or read directory: %s' % path) 33 | self._path = path 34 | self._fileclass = fileclass 35 | self._files = {} 36 | 37 | def _register_files(self, dirpath, filenames): 38 | for f in filter(lambda f: f.endswith(FILE_EXTENSION), filenames): 39 | vvv('REGISTER {0}'.format(f)) 40 | f = os.path.join(dirpath, f) 41 | ptr = None if not self._fileclass else self._fileclass(f) 42 | self._files[f] = ptr 43 | 44 | files = property(lambda self: self._files) 45 | 46 | def walk(self, register_fn=None): 47 | if not callable(register_fn): 48 | register_fn = self._register_files 49 | 50 | def _error(exc): 51 | raise(exc) 52 | 53 | for dirpath, dirnames, filenames in os.walk(self._path, 54 | topdown=True, 55 | onerror=_error, 56 | followlinks=True): 57 | vvv('RECURSE {0}, {1} files, {2} subdirectories'.format( 58 | dirpath.replace(os.getcwd(), '.'), len(filenames), len(dirnames))) 59 | for d in dirnames: 60 | if d.startswith('.') or d in SKIPDIRS: 61 | vvv(' SKIP subdirectory {0}'.format(d)) 62 | dirnames.remove(d) 63 | register_fn(dirpath, filenames) 64 | 65 | def __repr__(self): 66 | return '<{0} {1}>'.format(self.__class__.__name__, self._path) 67 | -------------------------------------------------------------------------------- /reclass/storage/memcache_proxy.py: -------------------------------------------------------------------------------- 1 | # 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of reclass (http://github.com/madduck/reclass) 5 | # 6 | # Copyright © 2007–14 martin f. krafft 7 | # Released under the terms of the Artistic Licence 2.0 8 | # 9 | from __future__ import absolute_import 10 | from __future__ import division 11 | from __future__ import print_function 12 | from __future__ import unicode_literals 13 | 14 | from reclass.storage import NodeStorageBase 15 | 16 | STORAGE_NAME = 'memcache_proxy' 17 | 18 | class MemcacheProxy(NodeStorageBase): 19 | 20 | def __init__(self, real_storage, cache_classes=True, cache_nodes=True, 21 | cache_nodelist=True): 22 | name = '{0}({1})'.format(STORAGE_NAME, real_storage.name) 23 | super(MemcacheProxy, self).__init__(name) 24 | self._real_storage = real_storage 25 | self._cache_classes = cache_classes 26 | if cache_classes: 27 | self._classes_cache = {} 28 | self._cache_nodes = cache_nodes 29 | if cache_nodes: 30 | self._nodes_cache = {} 31 | self._cache_nodelist = cache_nodelist 32 | if cache_nodelist: 33 | self._nodelist_cache = None 34 | 35 | name = property(lambda self: self._real_storage.name) 36 | 37 | def get_node(self, name, settings): 38 | if not self._cache_nodes: 39 | return self._real_storage.get_node(name, settings) 40 | try: 41 | return self._nodes_cache[name] 42 | except KeyError as e: 43 | ret = self._real_storage.get_node(name, settings) 44 | self._nodes_cache[name] = ret 45 | return ret 46 | 47 | def get_class(self, name, environment, settings): 48 | if not self._cache_classes: 49 | return self._real_storage.get_class(name, environment, settings) 50 | try: 51 | return self._classes_cache[environment][name] 52 | except KeyError as e: 53 | if environment not in self._classes_cache: 54 | self._classes_cache[environment] = dict() 55 | ret = self._real_storage.get_class(name, environment, settings) 56 | self._classes_cache[environment][name] = ret 57 | return ret 58 | 59 | def enumerate_nodes(self): 60 | if not self._cache_nodelist: 61 | return self._real_storage.enumerate_nodes() 62 | 63 | elif self._nodelist_cache is None: 64 | self._nodelist_cache = self._real_storage.enumerate_nodes() 65 | 66 | return self._nodelist_cache 67 | -------------------------------------------------------------------------------- /doc/source/index.rst: -------------------------------------------------------------------------------- 1 | ================================================ 2 | reclass — Recursive external node classification 3 | ================================================ 4 | .. include:: intro.inc 5 | 6 | Releases and source code 7 | ------------------------ 8 | The latest released |reclass| version is |release|. Please have a look at the 9 | :doc:`change log ` for information about recent changes. 10 | 11 | For now, |reclass| is hosted `on Github`_, and you may clone it with the 12 | following command:: 13 | 14 | git clone https://github.com/madduck/reclass.git 15 | 16 | Please see the :doc:`install instructions ` for information about 17 | distribution packages and tarballs. 18 | 19 | .. _on Github: https://github.com/madduck/reclass 20 | 21 | Community 22 | --------- 23 | There is a `mailing list`_, where you can bring up anything related to 24 | |reclass|. 25 | 26 | .. _mailing list: http://lists.pantsfullofunix.net/listinfo/reclass 27 | 28 | For real-time communication, please join the ``#reclass`` IRC channel on 29 | ``irc.oftc.net``. 30 | 31 | If you're using `Salt`_, you can also ask your |reclass|-and-Salt-related 32 | questions on the mailing list, ideally specifying "reclass" in the subject of 33 | your message. 34 | 35 | Licence 36 | ------- 37 | |reclass| is © 2007–2014 by martin f. krafft and released under the terms of 38 | the `Artistic Licence 2.0`_. 39 | 40 | Contents 41 | -------- 42 | These documents aim to get you started with |reclass|: 43 | 44 | .. toctree:: 45 | :maxdepth: 2 46 | 47 | install 48 | concepts 49 | operations 50 | usage 51 | refs 52 | manpage 53 | configfile 54 | salt 55 | ansible 56 | puppet 57 | hacking 58 | todo 59 | changelog 60 | 61 | About the name 62 | -------------- 63 | "reclass" stands for **r**\ ecursive **e**\ xternal node **class**\ ifier, 64 | which is somewhat of a misnomer. I chose the name very early on, based on the 65 | recursive nature of the data merging. However, to the user, a better paradigm 66 | would be "hierarchical", as s/he does not and should not care too much about 67 | the implementation internals. By the time that I realised this, unfortunately, 68 | `Hiera`_ (Puppet-specific) had already occupied this prefix. Oh well. Once you 69 | start using |reclass|, you'll think recursively as well as hierarchically at 70 | the same time. It's really quite simple. 71 | 72 | .. 73 | Indices and tables 74 | ================== 75 | 76 | * :ref:`genindex` 77 | * :ref:`modindex` 78 | * :ref:`search` 79 | 80 | .. include:: extrefs.inc 81 | .. include:: substs.inc 82 | -------------------------------------------------------------------------------- /reclass/datatypes/applications.py: -------------------------------------------------------------------------------- 1 | # 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of reclass (http://github.com/madduck/reclass) 5 | # 6 | # Copyright © 2007–14 martin f. krafft 7 | # Released under the terms of the Artistic Licence 2.0 8 | # 9 | from __future__ import absolute_import 10 | from __future__ import division 11 | from __future__ import print_function 12 | from __future__ import unicode_literals 13 | 14 | 15 | from .classes import Classes 16 | 17 | class Applications(Classes): 18 | ''' 19 | Extends Classes with the possibility to let specially formatted items 20 | remove earlier occurences of the item. For instance, if the "negater" is 21 | '~', then "adding" an element "~foo" to a list causes a previous element 22 | "foo" to be removed. If no such element exists, nothing happens, but 23 | a reference of the negation is kept, in case the instance is later used to 24 | extend another instance, in which case the negations should apply to the 25 | instance to be extended. 26 | ''' 27 | DEFAULT_NEGATION_PREFIX = '~' 28 | 29 | def __init__(self, iterable=None, 30 | negation_prefix=DEFAULT_NEGATION_PREFIX): 31 | self.negation_prefix = negation_prefix 32 | self._offset = len(negation_prefix) 33 | self._negations = [] 34 | super(Applications, self).__init__(iterable) 35 | 36 | def append_if_new(self, item): 37 | self._assert_is_string(item) 38 | if item.startswith(self.negation_prefix): 39 | item = item[self._offset:] 40 | self._negations.append(item) 41 | try: 42 | self._items.remove(item) 43 | except ValueError: 44 | pass 45 | else: 46 | super(Applications, self)._append_if_new(item) 47 | 48 | def merge_unique(self, iterable): 49 | if isinstance(iterable, self.__class__): 50 | # we might be extending ourselves to include negated applications, 51 | # in which case we need to remove our own content accordingly: 52 | for negation in iterable._negations: 53 | try: 54 | self._items.remove(negation) 55 | except ValueError: 56 | pass 57 | iterable = iterable.as_list() 58 | for i in iterable: 59 | self.append_if_new(i) 60 | 61 | def __repr__(self): 62 | contents = self._items + \ 63 | ['%s%s' % (self.negation_prefix, i) for i in self._negations] 64 | return "%s(%r, %r)" % (self.__class__.__name__, contents, 65 | str(self.negation_prefix)) 66 | -------------------------------------------------------------------------------- /reclass/datatypes/classes.py: -------------------------------------------------------------------------------- 1 | # 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of reclass (http://github.com/madduck/reclass) 5 | # 6 | # Copyright © 2007–14 martin f. krafft 7 | # Released under the terms of the Artistic Licence 2.0 8 | # 9 | from __future__ import absolute_import 10 | from __future__ import division 11 | from __future__ import print_function 12 | from __future__ import unicode_literals 13 | 14 | import six 15 | import os 16 | from reclass.errors import InvalidClassnameError 17 | 18 | INVALID_CHARACTERS_FOR_CLASSNAMES = ' ' + os.sep 19 | 20 | 21 | class Classes(object): 22 | ''' 23 | A very limited ordered set of strings with O(n) uniqueness constraints. It 24 | is neither a proper list or a proper set, on purpose, to keep things 25 | simple. 26 | ''' 27 | def __init__(self, iterable=None): 28 | self._items = [] 29 | if iterable is not None: 30 | self.merge_unique(iterable) 31 | 32 | def __len__(self): 33 | return len(self._items) 34 | 35 | def __eq__(self, rhs): 36 | if isinstance(rhs, list): 37 | return self._items == rhs 38 | else: 39 | try: 40 | return self._items == rhs._items 41 | except AttributeError as e: 42 | return False 43 | 44 | def __ne__(self, rhs): 45 | return not self.__eq__(rhs) 46 | 47 | def as_list(self): 48 | return self._items[:] 49 | 50 | def merge_unique(self, iterable): 51 | if isinstance(iterable, self.__class__): 52 | iterable = iterable.as_list() 53 | # Cannot just call list.extend here, as iterable's items might not 54 | # be unique by themselves, or in the context of self. 55 | for i in iterable: 56 | self.append_if_new(i) 57 | 58 | def _assert_is_string(self, item): 59 | if not isinstance(item, six.string_types): 60 | raise TypeError('%s instances can only contain strings, ' 61 | 'not %s' % (self.__class__.__name__, type(item))) 62 | 63 | def _assert_valid_characters(self, item): 64 | for c in INVALID_CHARACTERS_FOR_CLASSNAMES: 65 | if c in item: 66 | raise InvalidClassnameError(c, item) 67 | 68 | def _append_if_new(self, item): 69 | if item not in self._items: 70 | self._items.append(item) 71 | 72 | def append_if_new(self, item): 73 | self._assert_is_string(item) 74 | self._assert_valid_characters(item) 75 | self._append_if_new(item) 76 | 77 | def __repr__(self): 78 | return '%s(%r)' % (self.__class__.__name__, self._items) 79 | -------------------------------------------------------------------------------- /reclass/storage/mixed/__init__.py: -------------------------------------------------------------------------------- 1 | # 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of reclass 5 | from __future__ import absolute_import 6 | from __future__ import division 7 | from __future__ import print_function 8 | from __future__ import unicode_literals 9 | 10 | import collections 11 | import copy 12 | 13 | from six import iteritems 14 | 15 | import reclass.errors 16 | from reclass import get_storage 17 | from reclass.storage import ExternalNodeStorageBase 18 | 19 | def path_mangler(inventory_base_uri, nodes_uri, classes_uri): 20 | if nodes_uri == classes_uri: 21 | raise errors.DuplicateUriError(nodes_uri, classes_uri) 22 | return nodes_uri, classes_uri 23 | 24 | STORAGE_NAME = 'mixed' 25 | 26 | class ExternalNodeStorage(ExternalNodeStorageBase): 27 | 28 | MixedUri = collections.namedtuple('MixedURI', 'storage_type options') 29 | 30 | def __init__(self, nodes_uri, classes_uri, compose_node_name): 31 | super(ExternalNodeStorage, self).__init__(STORAGE_NAME, compose_node_name) 32 | 33 | self._nodes_uri = self._uri(nodes_uri) 34 | self._nodes_storage = get_storage(self._nodes_uri.storage_type, self._nodes_uri.options, None, compose_node_name) 35 | self._classes_default_uri = self._uri(classes_uri) 36 | self._classes_default_storage = get_storage(self._classes_default_uri.storage_type, None, self._classes_default_uri.options, compose_node_name) 37 | 38 | self._classes_storage = dict() 39 | if 'env_overrides' in classes_uri: 40 | for override in classes_uri['env_overrides']: 41 | for (env, options) in iteritems(override): 42 | uri = copy.deepcopy(classes_uri) 43 | uri.update(options) 44 | uri = self._uri(uri) 45 | self._classes_storage[env] = get_storage(uri.storage_type, None, uri.options, compose_node_name) 46 | 47 | def _uri(self, uri): 48 | ret = copy.deepcopy(uri) 49 | ret['storage_type'] = uri['storage_type'] 50 | if 'env_overrides' in ret: 51 | del ret['env_overrides'] 52 | if uri['storage_type'] == 'yaml_fs': 53 | ret = ret['uri'] 54 | return self.MixedUri(uri['storage_type'], ret) 55 | 56 | def get_node(self, name, settings): 57 | return self._nodes_storage.get_node(name, settings) 58 | 59 | def get_class(self, name, environment, settings): 60 | storage = self._classes_storage.get(environment, self._classes_default_storage) 61 | return storage.get_class(name, environment, settings) 62 | 63 | def enumerate_nodes(self): 64 | return self._nodes_storage.enumerate_nodes() 65 | -------------------------------------------------------------------------------- /reclass/values/item.py: -------------------------------------------------------------------------------- 1 | # 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of reclass 5 | # 6 | from __future__ import absolute_import 7 | from __future__ import division 8 | from __future__ import print_function 9 | from __future__ import unicode_literals 10 | 11 | from enum import Enum 12 | 13 | from reclass.utils.dictpath import DictPath 14 | 15 | ItemTypes = Enum('ItemTypes', 16 | ['COMPOSITE', 'DICTIONARY', 'INV_QUERY', 'LIST', 17 | 'REFERENCE', 'SCALAR']) 18 | 19 | 20 | class Item(object): 21 | 22 | def __init__(self, item, settings): 23 | self._settings = settings 24 | self.contents = item 25 | self.has_inv_query = False 26 | 27 | def allRefs(self): 28 | return True 29 | 30 | @property 31 | def has_references(self): 32 | return False 33 | 34 | def is_container(self): 35 | return False 36 | 37 | @property 38 | def is_complex(self): 39 | return (self.has_references | self.has_inv_query) 40 | 41 | def merge_over(self, item): 42 | msg = "Item class {0} does not implement merge_over()" 43 | raise NotImplementedError(msg.format(self.__class__.__name__)) 44 | 45 | def render(self, context, exports): 46 | msg = "Item class {0} does not implement render()" 47 | raise NotImplementedError(msg.format(self.__class__.__name__)) 48 | 49 | def type_str(self): 50 | return self.type.name.lower() 51 | 52 | def __repr__(self): 53 | return '%s(%r)' % (self.__class__.__name__, self.contents) 54 | 55 | 56 | class ItemWithReferences(Item): 57 | 58 | def __init__(self, items, settings): 59 | super(ItemWithReferences, self).__init__(items, settings) 60 | try: 61 | iter(self.contents) 62 | except TypeError: 63 | self.contents = [self.contents] 64 | self.assembleRefs() 65 | 66 | @property 67 | def has_references(self): 68 | return len(self._refs) > 0 69 | 70 | def get_references(self): 71 | return self._refs 72 | 73 | # NOTE: possibility of confusion. Looks like 'assemble' should be either 74 | # 'gather' or 'extract'. 75 | def assembleRefs(self, context={}): 76 | self._refs = [] 77 | self.allRefs = True 78 | for item in self.contents: 79 | if item.has_references: 80 | item.assembleRefs(context) 81 | self._refs.extend(item.get_references()) 82 | if item.allRefs is False: 83 | self.allRefs = False 84 | 85 | 86 | class ContainerItem(Item): 87 | 88 | def is_container(self): 89 | return True 90 | 91 | def render(self, context, inventory): 92 | return self.contents 93 | -------------------------------------------------------------------------------- /reclass/datatypes/tests/test_applications.py: -------------------------------------------------------------------------------- 1 | # 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of reclass (http://github.com/madduck/reclass) 5 | # 6 | # Copyright © 2007–14 martin f. krafft 7 | # Released under the terms of the Artistic Licence 2.0 8 | # 9 | from __future__ import absolute_import 10 | from __future__ import division 11 | from __future__ import print_function 12 | from __future__ import unicode_literals 13 | 14 | from reclass.datatypes import Applications, Classes 15 | import unittest 16 | 17 | try: 18 | import unittest.mock as mock 19 | except ImportError: 20 | import mock 21 | 22 | TESTLIST1 = ['one', 'two', 'three'] 23 | TESTLIST2 = ['red', 'green', '~two', '~three'] 24 | GOALLIST = ['one', 'red', 'green'] 25 | 26 | #TODO: mock out the underlying list 27 | 28 | class TestApplications(unittest.TestCase): 29 | 30 | def test_inheritance(self): 31 | a = Applications() 32 | self.assertIsInstance(a, Classes) 33 | 34 | def test_constructor_negate(self): 35 | a = Applications(TESTLIST1 + TESTLIST2) 36 | self.assertSequenceEqual(a, GOALLIST) 37 | 38 | def test_merge_unique_negate_list(self): 39 | a = Applications(TESTLIST1) 40 | a.merge_unique(TESTLIST2) 41 | self.assertSequenceEqual(a, GOALLIST) 42 | 43 | def test_merge_unique_negate_instance(self): 44 | a = Applications(TESTLIST1) 45 | a.merge_unique(Applications(TESTLIST2)) 46 | self.assertSequenceEqual(a, GOALLIST) 47 | 48 | def test_append_if_new_negate(self): 49 | a = Applications(TESTLIST1) 50 | a.append_if_new(TESTLIST2[2]) 51 | self.assertSequenceEqual(a, TESTLIST1[::2]) 52 | 53 | def test_repr_empty(self): 54 | negater = '%%' 55 | a = Applications(negation_prefix=negater) 56 | self.assertEqual('%r' % a, "%s(%r, '%s')" % (a.__class__.__name__, [], negater)) 57 | 58 | def test_repr_contents(self): 59 | negater = '%%' 60 | a = Applications(TESTLIST1, negation_prefix=negater) 61 | self.assertEqual('%r' % a, "%s(%r, '%s')" % (a.__class__.__name__, TESTLIST1, negater)) 62 | 63 | def test_repr_negations(self): 64 | negater = '~' 65 | a = Applications(TESTLIST2, negation_prefix=negater) 66 | self.assertEqual('%r' % a, "%s(%r, '%s')" % (a.__class__.__name__, TESTLIST2, negater)) 67 | 68 | def test_repr_negations_interspersed(self): 69 | l = ['a', '~b', 'a', '~d'] 70 | a = Applications(l) 71 | is_negation = lambda x: x.startswith(a.negation_prefix) 72 | GOAL = list(filter(lambda x: not is_negation(x), set(l))) + list(filter(is_negation, l)) 73 | self.assertEqual('%r' % a, "%s(%r, '~')" % (a.__class__.__name__, GOAL)) 74 | 75 | if __name__ == '__main__': 76 | unittest.main() 77 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | sudo: required 2 | language: python 3 | dist: trusty 4 | cache: pip 5 | python: 6 | - '2.7' 7 | - '3.6' 8 | service: 9 | - docker 10 | 11 | #apt: 12 | #update: true 13 | 14 | #stages: 15 | #- name: test 16 | #- name: coverage 17 | #- name: models 18 | #- name: build 19 | # if: fork = false 20 | #- name: publish 21 | # if: tag =~ ^v.* and fork = false and branch = 'master' 22 | 23 | env: 24 | global: 25 | - PACKAGENAME="reclass" 26 | 27 | install: &pyinst 28 | - pip install -r requirements.txt 29 | #- pip install pyparsing 30 | #- pip install PyYAML 31 | # To test example models with kitchen: 32 | - | 33 | test -e Gemfile || cat < Gemfile 34 | source 'https://rubygems.org' 35 | gem 'rake' 36 | gem 'test-kitchen' 37 | gem 'kitchen-docker' 38 | gem 'kitchen-inspec' 39 | gem 'inspec' 40 | - bundle install 41 | 42 | script: 43 | - python setup.py install 44 | - find . reclass -name 'test_*.py' | sort | xargs -n1 -i% bash -c "echo %; python %" 45 | # To test example models with kitchen: 46 | - export PYTHON_VERSION=$TRAVIS_PYTHON_VERSION 47 | - kitchen list 48 | - kitchen test 49 | 50 | # NOTE: travis stage builds, below saved for future reference 51 | #jobs: 52 | # include: 53 | # - stage: test 54 | # script: &unittest 55 | # - python setup.py install 56 | # - find . reclass -name 'test_*.py' | sort | xargs -n1 -i% bash -c "echo %; python %" 57 | # 58 | # - stage: coverage 59 | # install: *pyinst 60 | # script: 61 | # - python3 -m pytest --cov=. --cov-report=term-missing:skip-covered 62 | # - coverage xml 63 | # #- coveralls 64 | # #- | 65 | # #[ ! -z "${CODACY_PROJECT_TOKEN}" ] && python-codacy-coverage -r coverage.xml || echo "Codacy coverage NOT exported" 66 | # 67 | # - stage: lint 68 | # script: 69 | # - python3 -m flake8 70 | # 71 | # - stage: models 72 | # install: &kitchen 73 | # - pip install PyYAML 74 | # - pip install virtualenv 75 | # - | 76 | # test -e Gemfile || cat < Gemfile 77 | # source 'https://rubygems.org' 78 | # gem 'rake' 79 | # gem 'test-kitchen' 80 | # gem 'kitchen-docker' 81 | # gem 'kitchen-inspec' 82 | # gem 'inspec' 83 | # - bundle install 84 | # script: 85 | # - export PYTHON_VERSION=$TRAVIS_PYTHON_VERSION 86 | # - kitchen list 87 | # #FIXME- kitchen test 88 | # 89 | # - stage: build 90 | # install: *pyinst 91 | # script: [] 92 | # 93 | # - stage: publish 94 | # install: 95 | # - "/bin/true" 96 | # script: 97 | # - "/bin/true" 98 | # deploy: 99 | # provider: pypi 100 | # user: epcim 101 | # password: 102 | # secure: TBD 103 | # on: 104 | # tags: true 105 | # repo: salt-formulas/reclass 106 | # branch: master 107 | # #FIXME, $TRAVIS_PYTHON_VERSION == '2.7' 108 | 109 | notifications: 110 | webhooks: 111 | on_success: change # options: [always|never|change] default: always 112 | on_failure: never 113 | on_start: never 114 | on_cancel: never 115 | on_error: never 116 | email: true 117 | 118 | -------------------------------------------------------------------------------- /reclass/values/value.py: -------------------------------------------------------------------------------- 1 | # 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of reclass 5 | # 6 | from __future__ import absolute_import 7 | from __future__ import division 8 | from __future__ import print_function 9 | from __future__ import unicode_literals 10 | 11 | from .parser import Parser 12 | from .dictitem import DictItem 13 | from .listitem import ListItem 14 | from .scaitem import ScaItem 15 | from reclass.errors import InterpolationError 16 | 17 | from six import string_types 18 | 19 | class Value(object): 20 | 21 | _parser = Parser() 22 | 23 | def __init__(self, value, settings, uri, parse_string=True): 24 | self._settings = settings 25 | self.uri = uri 26 | self.overwrite = False 27 | self.constant = False 28 | if isinstance(value, string_types): 29 | if parse_string: 30 | try: 31 | self._item = self._parser.parse(value, self._settings) 32 | except InterpolationError as e: 33 | e.uri = self.uri 34 | raise 35 | else: 36 | self._item = ScaItem(value, self._settings) 37 | elif isinstance(value, list): 38 | self._item = ListItem(value, self._settings) 39 | elif isinstance(value, dict): 40 | self._item = DictItem(value, self._settings) 41 | else: 42 | self._item = ScaItem(value, self._settings) 43 | 44 | def item_type(self): 45 | return self._item.type 46 | 47 | def item_type_str(self): 48 | return self._item.type_str() 49 | 50 | def is_container(self): 51 | return self._item.is_container() 52 | 53 | @property 54 | def allRefs(self): 55 | return self._item.allRefs 56 | 57 | @property 58 | def has_references(self): 59 | return self._item.has_references 60 | 61 | @property 62 | def has_inv_query(self): 63 | return self._item.has_inv_query 64 | 65 | @property 66 | def needs_all_envs(self): 67 | if self._item.has_inv_query: 68 | return self._item.needs_all_envs 69 | return False 70 | 71 | def ignore_failed_render(self): 72 | return self._item.ignore_failed_render 73 | 74 | @property 75 | def is_complex(self): 76 | return self._item.is_complex 77 | 78 | def get_references(self): 79 | return self._item.get_references() 80 | 81 | def get_inv_references(self): 82 | return self._item.get_inv_references() 83 | 84 | def assembleRefs(self, context): 85 | if self._item.has_references: 86 | self._item.assembleRefs(context) 87 | 88 | def render(self, context, inventory): 89 | try: 90 | return self._item.render(context, inventory) 91 | except InterpolationError as e: 92 | e.uri = self.uri 93 | raise 94 | 95 | @property 96 | def contents(self): 97 | return self._item.contents 98 | 99 | def merge_over(self, value): 100 | self._item = self._item.merge_over(value._item) 101 | return self 102 | 103 | def __repr__(self): 104 | return 'Value(%r)' % self._item 105 | 106 | def __str__(self): 107 | return str(self._item) 108 | -------------------------------------------------------------------------------- /reclass/settings.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 2 | 3 | from __future__ import absolute_import 4 | from __future__ import division 5 | from __future__ import print_function 6 | from __future__ import unicode_literals 7 | 8 | import reclass.defaults as defaults 9 | 10 | from six import string_types, iteritems 11 | 12 | 13 | class Settings(object): 14 | 15 | known_opts = { 16 | 'allow_adapter_env_override': defaults.OPT_ALLOW_ADAPTER_ENV_OVERRIDE, 17 | 'allow_scalar_over_dict': defaults.OPT_ALLOW_SCALAR_OVER_DICT, 18 | 'allow_scalar_over_list': defaults.OPT_ALLOW_SCALAR_OVER_LIST, 19 | 'allow_list_over_scalar': defaults.OPT_ALLOW_LIST_OVER_SCALAR, 20 | 'allow_dict_over_scalar': defaults.OPT_ALLOW_DICT_OVER_SCALAR, 21 | 'allow_none_override': defaults.OPT_ALLOW_NONE_OVERRIDE, 22 | 'automatic_parameters': defaults.AUTOMATIC_RECLASS_PARAMETERS, 23 | 'class_mappings_match_path': defaults.CLASS_MAPPINGS_MATCH_PATH, 24 | 'scalar_parameters': defaults.SCALAR_RECLASS_PARAMETERS, 25 | 'default_environment': defaults.DEFAULT_ENVIRONMENT, 26 | 'delimiter': defaults.PARAMETER_INTERPOLATION_DELIMITER, 27 | 'dict_key_override_prefix': 28 | defaults.PARAMETER_DICT_KEY_OVERRIDE_PREFIX, 29 | 'dict_key_constant_prefix': 30 | defaults.PARAMETER_DICT_KEY_CONSTANT_PREFIX, 31 | 'escape_character': defaults.ESCAPE_CHARACTER, 32 | 'export_sentinels': defaults.EXPORT_SENTINELS, 33 | 'inventory_ignore_failed_node': 34 | defaults.OPT_INVENTORY_IGNORE_FAILED_NODE, 35 | 'inventory_ignore_failed_render': 36 | defaults.OPT_INVENTORY_IGNORE_FAILED_RENDER, 37 | 'reference_sentinels': defaults.REFERENCE_SENTINELS, 38 | 'ignore_class_notfound': defaults.OPT_IGNORE_CLASS_NOTFOUND, 39 | 'strict_constant_parameters': 40 | defaults.OPT_STRICT_CONSTANT_PARAMETERS, 41 | 'ignore_class_notfound_regexp': 42 | defaults.OPT_IGNORE_CLASS_NOTFOUND_REGEXP, 43 | 'ignore_class_notfound_warning': 44 | defaults.OPT_IGNORE_CLASS_NOTFOUND_WARNING, 45 | 'ignore_overwritten_missing_references': 46 | defaults.OPT_IGNORE_OVERWRITTEN_MISSING_REFERENCES, 47 | 'group_errors': defaults.OPT_GROUP_ERRORS, 48 | 'compose_node_name': defaults.OPT_COMPOSE_NODE_NAME, 49 | } 50 | 51 | def __init__(self, options={}): 52 | for opt_name, opt_value in iteritems(self.known_opts): 53 | setattr(self, opt_name, options.get(opt_name, opt_value)) 54 | 55 | self.dict_key_prefixes = [str(self.dict_key_override_prefix), 56 | str(self.dict_key_constant_prefix)] 57 | if isinstance(self.ignore_class_notfound_regexp, string_types): 58 | self.ignore_class_notfound_regexp = [ 59 | self.ignore_class_notfound_regexp] 60 | 61 | def __eq__(self, other): 62 | if isinstance(other, type(self)): 63 | return all(getattr(self, opt) == getattr(other, opt) 64 | for opt in self.known_opts) 65 | return False 66 | 67 | def __copy__(self): 68 | cls = self.__class__ 69 | result = cls.__new__(cls) 70 | result.__dict__.update(self.__dict__) 71 | return result 72 | 73 | def __deepcopy__(self, memo): 74 | return self.__copy__() 75 | -------------------------------------------------------------------------------- /reclass/values/parser.py: -------------------------------------------------------------------------------- 1 | # 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of reclass 5 | # 6 | from __future__ import absolute_import 7 | from __future__ import division 8 | from __future__ import print_function 9 | from __future__ import unicode_literals 10 | 11 | import pyparsing as pp 12 | 13 | from .compitem import CompItem 14 | from .invitem import InvItem 15 | from .refitem import RefItem 16 | from .scaitem import ScaItem 17 | 18 | from reclass.errors import ParseError 19 | from reclass.values.parser_funcs import tags 20 | import reclass.values.parser_funcs as parsers 21 | 22 | import collections 23 | import six 24 | 25 | 26 | class Parser(object): 27 | 28 | def __init__(self): 29 | self._ref_parser = None 30 | self._simple_parser = None 31 | self._old_settings = None 32 | 33 | @property 34 | def ref_parser(self): 35 | if self._ref_parser is None or self._settings != self._old_settings: 36 | self._ref_parser = parsers.get_ref_parser(self._settings) 37 | self._old_settings = self._settings 38 | return self._ref_parser 39 | 40 | @property 41 | def simple_ref_parser(self): 42 | if self._simple_parser is None or self._settings != self._old_settings: 43 | self._simple_parser = parsers.get_simple_ref_parser(self._settings) 44 | self._old_settings = self._settings 45 | return self._simple_parser 46 | 47 | def parse(self, value, settings): 48 | def full_parse(): 49 | try: 50 | return self.ref_parser.parseString(value) 51 | except pp.ParseException as e: 52 | raise ParseError(e.msg, e.line, e.col, e.lineno) 53 | 54 | self._settings = settings 55 | sentinel_count = (value.count(settings.reference_sentinels[0]) + 56 | value.count(settings.export_sentinels[0])) 57 | if sentinel_count == 0: 58 | # speed up: only use pyparsing if there are sentinels in the value 59 | return ScaItem(value, self._settings) 60 | elif sentinel_count == 1: # speed up: try a simple reference 61 | try: 62 | tokens = self.simple_ref_parser.parseString(value) 63 | except pp.ParseException: 64 | tokens = full_parse() # fall back on the full parser 65 | else: 66 | tokens = full_parse() # use the full parser 67 | 68 | tokens = parsers.listify(tokens) 69 | items = self._create_items(tokens) 70 | if len(items) == 1: 71 | return items[0] 72 | return CompItem(items, self._settings) 73 | 74 | _item_builders = {tags.STR: (lambda s, v: ScaItem(v, s._settings)), 75 | tags.REF: (lambda s, v: s._create_ref(v)), 76 | tags.INV: (lambda s, v: s._create_inv(v)) } 77 | 78 | def _create_items(self, tokens): 79 | return [self._item_builders[t](self, v) for t, v in tokens ] 80 | 81 | def _create_ref(self, tokens): 82 | items = [ self._item_builders[t](self, v) for t, v in tokens ] 83 | return RefItem(items, self._settings) 84 | 85 | def _create_inv(self, tokens): 86 | items = [ScaItem(v, self._settings) for t, v in tokens] 87 | if len(items) == 1: 88 | return InvItem(items[0], self._settings) 89 | return InvItem(CompItem(items), self._settings) 90 | -------------------------------------------------------------------------------- /doc/source/install.rst: -------------------------------------------------------------------------------- 1 | ============ 2 | Installation 3 | ============ 4 | 5 | For Debian users (including Ubuntu) 6 | ----------------------------------- 7 | |reclass| has been `packaged for Debian`_. To use it, just install it with 8 | APT:: 9 | 10 | $ apt-get install reclass [reclass-doc] 11 | 12 | .. _packaged for Debian: http://packages.debian.org/search?keywords=reclass 13 | 14 | For ArchLinux users 15 | ------------------- 16 | |reclass| is `available for ArchLinux`_, thanks to Niels Abspoel. 17 | Dowload the tarball_ from ``aur`` or ``yaourt``:: 18 | 19 | $ yaourt -S reclass 20 | 21 | or:: 22 | 23 | $ tar xvzf reclass-git.tar.gz 24 | $ cd reclass-git; makepkg 25 | $ sudo pacman -U reclass-git-.tar.gz 26 | 27 | .. _available for ArchLinux: https://aur.archlinux.org/packages/reclass-git/ 28 | .. _tarball: https://aur.archlinux.org/packages/re/reclass-git/reclass-git.tar.gz 29 | 30 | Other distributions 31 | ------------------- 32 | Developers of other distributions are cordially invited to package |reclass| 33 | themselves and `write to the mailing list 34 | `_ to have details included here. Or send 35 | a patch! 36 | 37 | From source 38 | ----------- 39 | |reclass| is currently maintained `on Github 40 | `_, so to obtain the source, run:: 41 | 42 | $ git clone https://github.com/madduck/reclass.git 43 | 44 | or:: 45 | 46 | $ git clone ssh://git@github.com:madduck/reclass.git 47 | 48 | If you want a tarball, please `obtain it from the Debian archive`_. 49 | 50 | .. _obtain it from the Debian archive: http://http.debian.net/debian/pool/main/r/reclass/ 51 | 52 | Before you can use |reclass|, you need to install it into a place where Python 53 | can find it. The following step should install the package to ``/usr/local``:: 54 | 55 | $ python setup.py install 56 | 57 | If you want to install to a different location, use --prefix like so:: 58 | 59 | $ python setup.py install --prefix=/opt/local 60 | 61 | .. todo:: 62 | 63 | These will install the ``reclass-salt`` and ``reclass-ansible`` adapters to 64 | ``$prefix/bin``, but they should go to ``$prefix/share/reclass``. How can 65 | setup.py be told to do so? It would be better for consistency if this was 66 | done "upstream", rather than fixed by the distros. 67 | 68 | Just make sure that the destination is in the Python module search path, which 69 | you can check like this:: 70 | 71 | $ python -c 'import sys; print sys.path' 72 | 73 | More options can be found in the output of 74 | 75 | :: 76 | 77 | $ python setup.py install --help 78 | $ python setup.py --help 79 | $ python setup.py --help-commands 80 | $ python setup.py --help [cmd] 81 | 82 | If you just want to run |reclass| from source, e.g. because you are going to be 83 | making and testing changes, install it in "development mode":: 84 | 85 | $ python setup.py develop 86 | 87 | To uninstall (the rm call is necessary due to `a bug in setuptools`_):: 88 | 89 | $ python setup.py develop --uninstall 90 | $ rm /usr/local/bin/reclass* 91 | 92 | `Uninstallation currently isn't possible`_ for packages installed to 93 | /usr/local as per the above method, unfortunately. The following should do:: 94 | 95 | $ rm -r /usr/local/lib/python*/dist-packages/reclass* /usr/local/bin/reclass* 96 | 97 | .. _a bug in setuptools: http://bugs.debian.org/714960 98 | .. _Uninstallation currently isn't possible: http://bugs.python.org/issue4673 99 | 100 | .. include:: substs.inc 101 | -------------------------------------------------------------------------------- /reclass/datatypes/exports.py: -------------------------------------------------------------------------------- 1 | # 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of reclass (http://github.com/madduck/reclass) 5 | # 6 | from __future__ import absolute_import 7 | from __future__ import division 8 | from __future__ import print_function 9 | from __future__ import unicode_literals 10 | 11 | import copy 12 | 13 | from six import iteritems, next 14 | 15 | from .parameters import Parameters 16 | from reclass.errors import ResolveError 17 | from reclass.values.value import Value 18 | from reclass.values.valuelist import ValueList 19 | from reclass.utils.dictpath import DictPath 20 | 21 | class Exports(Parameters): 22 | 23 | def __init__(self, mapping, settings, uri): 24 | super(Exports, self).__init__(mapping, settings, uri) 25 | 26 | def delete_key(self, key): 27 | self._base.pop(key, None) 28 | self._unrendered.pop(key, None) 29 | 30 | def overwrite(self, other): 31 | overdict = {'~' + key: value for (key, value) in iteritems(other)} 32 | self.merge(overdict) 33 | 34 | def interpolate_from_external(self, external): 35 | while len(self._unrendered) > 0: 36 | path, v = next(iteritems(self._unrendered)) 37 | value = path.get_value(self._base) 38 | if isinstance(value, (Value, ValueList)): 39 | external._interpolate_references(path, value, None) 40 | new = self._interpolate_render_from_external(external._base, path, value) 41 | path.set_value(self._base, new) 42 | del self._unrendered[path] 43 | else: 44 | # references to lists and dicts are only deepcopied when merged 45 | # together so it's possible a value with references in a referenced 46 | # list or dict has already been rendered 47 | del self._unrendered[path] 48 | 49 | def interpolate_single_from_external(self, external, query): 50 | for r in query.get_inv_references(): 51 | self._interpolate_single_path_from_external(r, external, query) 52 | 53 | def _interpolate_single_path_from_external(self, mainpath, external, query): 54 | required = self._get_required_paths(mainpath) 55 | while len(required) > 0: 56 | while len(required) > 0: 57 | path, v = next(iteritems(required)) 58 | value = path.get_value(self._base) 59 | if isinstance(value, (Value, ValueList)): 60 | try: 61 | external._interpolate_references(path, value, None) 62 | new = self._interpolate_render_from_external(external._base, path, value) 63 | path.set_value(self._base, new) 64 | except ResolveError as e: 65 | if query.ignore_failed_render(): 66 | path.delete(self._base) 67 | else: 68 | raise 69 | del required[path] 70 | del self._unrendered[path] 71 | required = self._get_required_paths(mainpath) 72 | 73 | def _get_required_paths(self, mainpath): 74 | paths = {} 75 | path = DictPath(self._settings.delimiter) 76 | for i in mainpath.key_parts(): 77 | path.add_subpath(i) 78 | if path in self._unrendered: 79 | paths[path] = True 80 | for i in self._unrendered: 81 | if mainpath.is_ancestor_of(i) or mainpath == i: 82 | paths[i] = True 83 | return paths 84 | 85 | def _interpolate_render_from_external(self, context, path, value): 86 | try: 87 | new = value.render(context, None) 88 | except ResolveError as e: 89 | e.context = path 90 | raise 91 | if isinstance(new, dict): 92 | new = self._render_simple_dict(new, path) 93 | elif isinstance(new, list): 94 | new = self._render_simple_list(new, path) 95 | return new 96 | -------------------------------------------------------------------------------- /doc/source/changelog.rst: -------------------------------------------------------------------------------- 1 | ========= 2 | ChangeLog 3 | ========= 4 | 5 | ========= ========== ======================================================== 6 | Version Date Changes 7 | ========= ========== ======================================================== 8 | 1.7.0 2020-10-02 Fixes and few new features: 9 | * Allow class mappings to wildcard match against either the node name and class 10 | * Support for .yaml along with .yml 11 | * Support to use current node parameters as references in class name 12 | 1.6.0 2018-11-06 * Python code and parser refactoring by a-ovchinnikov 13 | * Improvements in yaml_git and mixed setup by Andrew Pickford 14 | * Relative paths in class names by Petr Michalec, Martin Polreich and Andrew Pickford 15 | * Bug Fixes for recently added features 16 | 1.5.6 2018-07-30 * Fix, usage of integers as pillar keys 17 | * Refactoring python codebase by @a-ovchinkonv 18 | * New feature, "compose node name" from node subdirectory structure (by @gburiola) 19 | 1.5.5 2018-07 * Add immutable (constant) parameters 20 | * Fixes 21 | 1.5.4 2018-05 * Add support for salt 2018.3 22 | * Add support for python 2.7/3.x 23 | * Extend tests coverage 24 | 1.5.3 2018 * Add new features + fixes 25 | - last 'known' full compatible release with original reclass 26 | - release shipped as well as .deb package at mirror.mirantis.com 27 | 1.5.x 2017 * Project forked under salt-formulas/reclass 28 | - based on @andrewpickford fork and community fixes 29 | - features against original are in README-extensions.rst 30 | 1.4.1 2014-10-28 * Revert debug logging, which wasn't fault-free and so 31 | it needs more time to mature. 32 | 1.4 2014-10-25 * Add rudimentary debug logging 33 | * Prevent interpolate() from overwriting merged values 34 | * Look for "init" instead of "index" when being fed 35 | a directory. 36 | * Fix error reporting on node name collision across 37 | subdirectories. 38 | 1.3 2014-03-01 * Salt: pillar data from previous pillars are now 39 | available to reclass parameter interpolation 40 | * yaml_fs: classes may be defined in subdirectories 41 | (closes: #12, #19, #20) 42 | * Migrate Salt adapter to new core API (closes: #18) 43 | * Fix --nodeinfo invocation in docs (closes: #21) 44 | 1.2.2 2013-12-27 * Recurse classes obtained from class mappings 45 | (closes: #16) 46 | * Fix class mapping regexp rendering in docs 47 | (closes: #15) 48 | 1.2.1 2013-12-26 * Fix Salt adapter wrt. class mappings 49 | (closes: #14) 50 | 1.2 2013-12-10 * Introduce class mappings (see :doc:`operations`) 51 | (closes: #5) 52 | * Fix parameter interpolation across merged lists 53 | (closes: #13). 54 | * Caching of classes for performance reasons, especially 55 | during the inventory runs 56 | * yaml_fs: nodes may be defined in subdirectories 57 | (closes: #10). 58 | * Classes and nodes URI must not overlap anymore 59 | * Class names must not contain spaces 60 | 1.1 2013-08-28 Salt adapter: fix interface to include minion_id, filter 61 | output accordingly; fixes master_tops 62 | 1.0.2 2013-08-27 Fix incorrect versioning in setuptools 63 | 1.0.1 2013-08-27 Documentation updates, new homepage 64 | 1.0 2013-08-26 Initial release 65 | ========= ========== ======================================================== 66 | -------------------------------------------------------------------------------- /reclass/storage/yamldata.py: -------------------------------------------------------------------------------- 1 | # 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of reclass (http://github.com/madduck/reclass) 5 | # 6 | # Copyright © 2007–14 martin f. krafft 7 | # Released under the terms of the Artistic Licence 2.0 8 | # 9 | from __future__ import absolute_import 10 | from __future__ import division 11 | from __future__ import print_function 12 | from __future__ import unicode_literals 13 | 14 | from reclass import datatypes 15 | import yaml 16 | import os 17 | from reclass.errors import NotFoundError 18 | 19 | _SafeLoader = yaml.CSafeLoader if yaml.__with_libyaml__ else yaml.SafeLoader 20 | 21 | class YamlData(object): 22 | 23 | @classmethod 24 | def from_file(cls, path): 25 | ''' Initialise yaml data from a local file ''' 26 | abs_path = os.path.abspath(path) 27 | if not os.path.isfile(abs_path): 28 | raise NotFoundError('No such file: %s' % abs_path) 29 | if not os.access(abs_path, os.R_OK): 30 | raise NotFoundError('Cannot open: %s' % abs_path) 31 | y = cls('yaml_fs://{0}'.format(abs_path)) 32 | with open(abs_path) as fp: 33 | data = yaml.load(fp, Loader=_SafeLoader) 34 | if data is not None: 35 | y._data = data 36 | return y 37 | 38 | @classmethod 39 | def from_string(cls, string, uri): 40 | ''' Initialise yaml data from a string ''' 41 | y = cls(uri) 42 | data = yaml.load(string, Loader=_SafeLoader) 43 | if data is not None: 44 | y._data = data 45 | return y 46 | 47 | def __init__(self, uri): 48 | self._uri = uri 49 | self._data = dict() 50 | 51 | uri = property(lambda self: self._uri) 52 | 53 | def get_data(self): 54 | return self._data 55 | 56 | def set_absolute_names(self, name, names): 57 | new_names = [] 58 | for n in names: 59 | if n[0] == '.': 60 | dots = self.count_dots(n) 61 | levels_up = (dots * (-1)) 62 | parent = '.'.join(name.split('.')[0:levels_up]) 63 | if parent == '': 64 | n = n[dots:] 65 | else: 66 | n = parent + n[dots - 1:] 67 | new_names.append(n) 68 | return new_names 69 | 70 | def yield_dots(self, value): 71 | try: 72 | idx = value.index('.') 73 | except ValueError: 74 | return 75 | if idx == 0: 76 | yield '.' 77 | for dot in self.yield_dots(value[1:]): 78 | yield dot 79 | 80 | def count_dots(self, value): 81 | return len(list(self.yield_dots(value))) 82 | 83 | def get_entity(self, name, pathname, settings): 84 | classes = self._data.get('classes') 85 | if classes is None: 86 | classes = [] 87 | classes = self.set_absolute_names(name, classes) 88 | classes = datatypes.Classes(classes) 89 | 90 | applications = self._data.get('applications') 91 | if applications is None: 92 | applications = [] 93 | applications = datatypes.Applications(applications) 94 | 95 | parameters = self._data.get('parameters') 96 | if parameters is None: 97 | parameters = {} 98 | parameters = datatypes.Parameters(parameters, settings, self._uri) 99 | 100 | exports = self._data.get('exports') 101 | if exports is None: 102 | exports = {} 103 | exports = datatypes.Exports(exports, settings, self._uri) 104 | 105 | env = self._data.get('environment', None) 106 | 107 | return datatypes.Entity(settings, classes=classes, applications=applications, parameters=parameters, 108 | exports=exports, name=name, pathname=pathname, environment=env, uri=self.uri) 109 | 110 | def __str__(self): 111 | return '<{0} {1}, {2}>'.format(self.__class__.__name__, self._uri, 112 | self._data) 113 | 114 | def __repr__(self): 115 | return '<{0} {1}, {2}>'.format(self.__class__.__name__, self._uri, 116 | self._data.keys()) 117 | -------------------------------------------------------------------------------- /reclass/storage/tests/test_memcache_proxy.py: -------------------------------------------------------------------------------- 1 | # 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of reclass (http://github.com/madduck/reclass) 5 | # 6 | # Copyright © 2007–14 martin f. krafft 7 | # Released under the terms of the Artistic Licence 2.0 8 | # 9 | from __future__ import absolute_import 10 | from __future__ import division 11 | from __future__ import print_function 12 | from __future__ import unicode_literals 13 | 14 | from reclass.settings import Settings 15 | from reclass.storage.memcache_proxy import MemcacheProxy 16 | from reclass.storage import NodeStorageBase 17 | 18 | import unittest 19 | 20 | try: 21 | import unittest.mock as mock 22 | except ImportError: 23 | import mock 24 | 25 | 26 | class TestMemcacheProxy(unittest.TestCase): 27 | 28 | def setUp(self): 29 | self._storage = mock.MagicMock(spec_set=NodeStorageBase) 30 | 31 | def test_no_nodes_caching(self): 32 | p = MemcacheProxy(self._storage, cache_nodes=False) 33 | NAME = 'foo'; NAME2 = 'bar'; RET = 'baz'; SETTINGS = Settings() 34 | self._storage.get_node.return_value = RET 35 | self.assertEqual(p.get_node(NAME, SETTINGS), RET) 36 | self.assertEqual(p.get_node(NAME, SETTINGS), RET) 37 | self.assertEqual(p.get_node(NAME2, SETTINGS), RET) 38 | self.assertEqual(p.get_node(NAME2, SETTINGS), RET) 39 | expected = [mock.call(NAME, SETTINGS), mock.call(NAME, SETTINGS), 40 | mock.call(NAME2, SETTINGS), mock.call(NAME2, SETTINGS)] 41 | self.assertListEqual(self._storage.get_node.call_args_list, expected) 42 | 43 | def test_nodes_caching(self): 44 | p = MemcacheProxy(self._storage, cache_nodes=True) 45 | NAME = 'foo'; NAME2 = 'bar'; RET = 'baz'; SETTINGS = Settings() 46 | self._storage.get_node.return_value = RET 47 | self.assertEqual(p.get_node(NAME, SETTINGS), RET) 48 | self.assertEqual(p.get_node(NAME, SETTINGS), RET) 49 | self.assertEqual(p.get_node(NAME2, SETTINGS), RET) 50 | self.assertEqual(p.get_node(NAME2, SETTINGS), RET) 51 | expected = [mock.call(NAME, SETTINGS), mock.call(NAME2, SETTINGS)] # called once each 52 | self.assertListEqual(self._storage.get_node.call_args_list, expected) 53 | 54 | def test_no_classes_caching(self): 55 | p = MemcacheProxy(self._storage, cache_classes=False) 56 | NAME = 'foo'; NAME2 = 'bar'; RET = 'baz'; SETTINGS = Settings() 57 | self._storage.get_class.return_value = RET 58 | self.assertEqual(p.get_class(NAME, None, SETTINGS), RET) 59 | self.assertEqual(p.get_class(NAME, None, SETTINGS), RET) 60 | self.assertEqual(p.get_class(NAME2, None, SETTINGS), RET) 61 | self.assertEqual(p.get_class(NAME2, None, SETTINGS), RET) 62 | expected = [mock.call(NAME, None, SETTINGS), mock.call(NAME, None, SETTINGS), 63 | mock.call(NAME2, None, SETTINGS), mock.call(NAME2, None, SETTINGS)] 64 | self.assertListEqual(self._storage.get_class.call_args_list, expected) 65 | 66 | def test_classes_caching(self): 67 | p = MemcacheProxy(self._storage, cache_classes=True) 68 | NAME = 'foo'; NAME2 = 'bar'; RET = 'baz'; SETTINGS = Settings() 69 | self._storage.get_class.return_value = RET 70 | self.assertEqual(p.get_class(NAME, None, SETTINGS), RET) 71 | self.assertEqual(p.get_class(NAME, None, SETTINGS), RET) 72 | self.assertEqual(p.get_class(NAME2, None, SETTINGS), RET) 73 | self.assertEqual(p.get_class(NAME2, None, SETTINGS), RET) 74 | expected = [mock.call(NAME, None, SETTINGS), mock.call(NAME2, None, SETTINGS)] # called once each 75 | self.assertListEqual(self._storage.get_class.call_args_list, expected) 76 | 77 | def test_nodelist_no_caching(self): 78 | p = MemcacheProxy(self._storage, cache_nodelist=False) 79 | p.enumerate_nodes() 80 | p.enumerate_nodes() 81 | expected = [mock.call(), mock.call()] 82 | self.assertListEqual(self._storage.enumerate_nodes.call_args_list, expected) 83 | 84 | def test_nodelist_caching(self): 85 | p = MemcacheProxy(self._storage, cache_nodelist=True) 86 | p.enumerate_nodes() 87 | p.enumerate_nodes() 88 | expected = [mock.call()] # once only 89 | self.assertListEqual(self._storage.enumerate_nodes.call_args_list, expected) 90 | 91 | 92 | if __name__ == '__main__': 93 | unittest.main() 94 | -------------------------------------------------------------------------------- /reclass/storage/yaml_fs/__init__.py: -------------------------------------------------------------------------------- 1 | # 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of reclass (http://github.com/madduck/reclass) 5 | # 6 | # Copyright © 2007–14 martin f. krafft 7 | # Released under the terms of the Artistic Licence 2.0 8 | # 9 | from __future__ import absolute_import 10 | from __future__ import division 11 | from __future__ import print_function 12 | from __future__ import unicode_literals 13 | 14 | import os, sys 15 | import yaml 16 | from reclass.output.yaml_outputter import ExplicitDumper 17 | from reclass.storage import ExternalNodeStorageBase 18 | from reclass.storage.yamldata import YamlData 19 | from .directory import Directory 20 | from reclass.datatypes import Entity 21 | import reclass.errors 22 | 23 | FILE_EXTENSION = ('.yml', '.yaml') 24 | STORAGE_NAME = 'yaml_fs' 25 | 26 | def vvv(msg): 27 | #print(msg, file=sys.stderr) 28 | pass 29 | 30 | def path_mangler(inventory_base_uri, nodes_uri, classes_uri): 31 | 32 | if inventory_base_uri is None: 33 | # if inventory_base is not given, default to current directory 34 | inventory_base_uri = os.getcwd() 35 | 36 | nodes_uri = nodes_uri or 'nodes' 37 | classes_uri = classes_uri or 'classes' 38 | 39 | def _path_mangler_inner(path): 40 | ret = os.path.join(inventory_base_uri, path) 41 | ret = os.path.expanduser(ret) 42 | return os.path.abspath(ret) 43 | 44 | n, c = map(_path_mangler_inner, (nodes_uri, classes_uri)) 45 | if n == c: 46 | raise errors.DuplicateUriError(n, c) 47 | common = os.path.commonprefix((n, c)) 48 | if common == n or common == c: 49 | raise errors.UriOverlapError(n, c) 50 | 51 | return n, c 52 | 53 | 54 | class ExternalNodeStorage(ExternalNodeStorageBase): 55 | 56 | def __init__(self, nodes_uri, classes_uri, compose_node_name): 57 | super(ExternalNodeStorage, self).__init__(STORAGE_NAME, compose_node_name) 58 | 59 | if nodes_uri is not None: 60 | self._nodes_uri = nodes_uri 61 | self._nodes = self._enumerate_inventory(nodes_uri, self.node_name_mangler) 62 | 63 | if classes_uri is not None: 64 | self._classes_uri = classes_uri 65 | self._classes = self._enumerate_inventory(classes_uri, self.class_name_mangler) 66 | 67 | nodes_uri = property(lambda self: self._nodes_uri) 68 | classes_uri = property(lambda self: self._classes_uri) 69 | 70 | def _enumerate_inventory(self, basedir, name_mangler): 71 | ret = {} 72 | def register_fn(dirpath, filenames): 73 | filenames = [f for f in filenames if f.endswith(FILE_EXTENSION)] 74 | vvv('REGISTER {0} in path {1}'.format(filenames, dirpath)) 75 | for f in filenames: 76 | name = os.path.splitext(f)[0] 77 | relpath = os.path.relpath(dirpath, basedir) 78 | if callable(name_mangler): 79 | relpath, name = name_mangler(relpath, name) 80 | uri = os.path.join(dirpath, f) 81 | if name in ret: 82 | E = reclass.errors.DuplicateNodeNameError 83 | raise E(self.name, name, 84 | os.path.join(basedir, ret[name]), uri) 85 | if relpath: 86 | f = os.path.join(relpath, f) 87 | ret[name] = f 88 | 89 | d = Directory(basedir) 90 | d.walk(register_fn) 91 | return ret 92 | 93 | def get_node(self, name, settings): 94 | vvv('GET NODE {0}'.format(name)) 95 | try: 96 | relpath = self._nodes[name] 97 | path = os.path.join(self.nodes_uri, relpath) 98 | pathname = os.path.splitext(relpath)[0] 99 | except KeyError as e: 100 | raise reclass.errors.NodeNotFound(self.name, name, self.nodes_uri) 101 | entity = YamlData.from_file(path).get_entity(name, pathname, settings) 102 | return entity 103 | 104 | def get_class(self, name, environment, settings): 105 | vvv('GET CLASS {0}'.format(name)) 106 | try: 107 | path = os.path.join(self.classes_uri, self._classes[name]) 108 | pathname = os.path.splitext(self._classes[name])[0] 109 | except KeyError as e: 110 | raise reclass.errors.ClassNotFound(self.name, name, self.classes_uri) 111 | entity = YamlData.from_file(path).get_entity(name, pathname, settings) 112 | return entity 113 | 114 | def enumerate_nodes(self): 115 | return self._nodes.keys() 116 | -------------------------------------------------------------------------------- /reclass/values/tests/test_compitem.py: -------------------------------------------------------------------------------- 1 | from reclass.settings import Settings 2 | from reclass.values.value import Value 3 | from reclass.values.compitem import CompItem 4 | from reclass.values.scaitem import ScaItem 5 | from reclass.values.valuelist import ValueList 6 | from reclass.values.listitem import ListItem 7 | from reclass.values.dictitem import DictItem 8 | import unittest 9 | 10 | SETTINGS = Settings() 11 | 12 | class TestCompItem(unittest.TestCase): 13 | 14 | def test_assembleRefs_no_items(self): 15 | composite = CompItem([], SETTINGS) 16 | 17 | self.assertFalse(composite.has_references) 18 | 19 | def test_assembleRefs_one_item_without_refs(self): 20 | val1 = Value('foo', SETTINGS, '') 21 | 22 | composite = CompItem([val1], SETTINGS) 23 | 24 | self.assertFalse(composite.has_references) 25 | 26 | def test_assembleRefs_one_item_with_one_ref(self): 27 | val1 = Value('${foo}', SETTINGS, '') 28 | expected_refs = ['foo'] 29 | 30 | composite = CompItem([val1], SETTINGS) 31 | 32 | self.assertTrue(composite.has_references) 33 | self.assertEquals(composite.get_references(), expected_refs) 34 | 35 | def test_assembleRefs_one_item_with_two_refs(self): 36 | val1 = Value('${foo}${bar}', SETTINGS, '') 37 | expected_refs = ['foo', 'bar'] 38 | 39 | composite = CompItem([val1], SETTINGS) 40 | 41 | self.assertTrue(composite.has_references) 42 | self.assertEquals(composite.get_references(), expected_refs) 43 | 44 | def test_assembleRefs_two_items_one_with_one_ref_one_without(self): 45 | val1 = Value('${foo}bar', SETTINGS, '') 46 | val2 = Value('baz', SETTINGS, '') 47 | expected_refs = ['foo'] 48 | 49 | composite = CompItem([val1, val2], SETTINGS) 50 | 51 | self.assertTrue(composite.has_references) 52 | self.assertEquals(composite.get_references(), expected_refs) 53 | 54 | def test_assembleRefs_two_items_both_with_one_ref(self): 55 | val1 = Value('${foo}', SETTINGS, '') 56 | val2 = Value('${bar}', SETTINGS, '') 57 | expected_refs = ['foo', 'bar'] 58 | 59 | composite = CompItem([val1, val2], SETTINGS) 60 | 61 | self.assertTrue(composite.has_references) 62 | self.assertEquals(composite.get_references(), expected_refs) 63 | 64 | def test_assembleRefs_two_items_with_two_refs(self): 65 | val1 = Value('${foo}${baz}', SETTINGS, '') 66 | val2 = Value('${bar}${meep}', SETTINGS, '') 67 | expected_refs = ['foo', 'baz', 'bar', 'meep'] 68 | 69 | composite = CompItem([val1, val2], SETTINGS) 70 | 71 | self.assertTrue(composite.has_references) 72 | self.assertEquals(composite.get_references(), expected_refs) 73 | 74 | def test_string_representation(self): 75 | composite = CompItem(Value(1, SETTINGS, ''), SETTINGS) 76 | expected = '1' 77 | 78 | result = str(composite) 79 | 80 | self.assertEquals(result, expected) 81 | 82 | def test_render_single_item(self): 83 | val1 = Value('${foo}', SETTINGS, '') 84 | 85 | composite = CompItem([val1], SETTINGS) 86 | 87 | self.assertEquals(1, composite.render({'foo': 1}, None)) 88 | 89 | 90 | def test_render_multiple_items(self): 91 | val1 = Value('${foo}', SETTINGS, '') 92 | val2 = Value('${bar}', SETTINGS, '') 93 | 94 | composite = CompItem([val1, val2], SETTINGS) 95 | 96 | self.assertEquals('12', composite.render({'foo': 1, 'bar': 2}, None)) 97 | 98 | def test_merge_over_merge_scalar(self): 99 | val1 = Value(None, SETTINGS, '') 100 | scalar = ScaItem(1, SETTINGS) 101 | composite = CompItem([val1], SETTINGS) 102 | 103 | result = composite.merge_over(scalar) 104 | 105 | self.assertEquals(result, composite) 106 | 107 | def test_merge_over_merge_composite(self): 108 | val1 = Value(None, SETTINGS, '') 109 | val2 = Value(None, SETTINGS, '') 110 | composite1 = CompItem([val1], SETTINGS) 111 | composite2 = CompItem([val2], SETTINGS) 112 | 113 | result = composite2.merge_over(composite1) 114 | 115 | self.assertEquals(result, composite2) 116 | 117 | def test_merge_other_types_not_allowed(self): 118 | other = type('Other', (object,), {'type': 34}) 119 | val1 = Value(None, SETTINGS, '') 120 | composite = CompItem([val1], SETTINGS) 121 | 122 | self.assertRaises(RuntimeError, composite.merge_over, other) 123 | 124 | 125 | if __name__ == '__main__': 126 | unittest.main() 127 | -------------------------------------------------------------------------------- /reclass/datatypes/tests/test_classes.py: -------------------------------------------------------------------------------- 1 | # 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of reclass (http://github.com/madduck/reclass) 5 | # 6 | # Copyright © 2007–14 martin f. krafft 7 | # Released under the terms of the Artistic Licence 2.0 8 | # 9 | from __future__ import absolute_import 10 | from __future__ import division 11 | from __future__ import print_function 12 | from __future__ import unicode_literals 13 | 14 | from reclass.datatypes import Classes 15 | from reclass.datatypes.classes import INVALID_CHARACTERS_FOR_CLASSNAMES 16 | import unittest 17 | 18 | try: 19 | import unittest.mock as mock 20 | except ImportError: 21 | import mock 22 | from reclass.errors import InvalidClassnameError 23 | 24 | TESTLIST1 = ['one', 'two', 'three'] 25 | TESTLIST2 = ['red', 'green', 'blue'] 26 | 27 | #TODO: mock out the underlying list 28 | 29 | class TestClasses(unittest.TestCase): 30 | 31 | def test_len_empty(self): 32 | with mock.patch.object(Classes, 'merge_unique') as m: 33 | c = Classes() 34 | self.assertEqual(len(c), 0) 35 | self.assertFalse(m.called) 36 | 37 | def test_constructor(self): 38 | with mock.patch.object(Classes, 'merge_unique') as m: 39 | c = Classes(TESTLIST1) 40 | m.assert_called_once_with(TESTLIST1) 41 | 42 | def test_equality_list_empty(self): 43 | self.assertEqual(Classes(), []) 44 | 45 | def test_equality_list(self): 46 | self.assertEqual(Classes(TESTLIST1), TESTLIST1) 47 | 48 | def test_equality_instance_empty(self): 49 | self.assertEqual(Classes(), Classes()) 50 | 51 | def test_equality_instance(self): 52 | self.assertEqual(Classes(TESTLIST1), Classes(TESTLIST1)) 53 | 54 | def test_inequality(self): 55 | self.assertNotEqual(Classes(TESTLIST1), Classes(TESTLIST2)) 56 | 57 | def test_construct_duplicates(self): 58 | c = Classes(TESTLIST1 + TESTLIST1) 59 | self.assertSequenceEqual(c, TESTLIST1) 60 | 61 | def test_append_if_new(self): 62 | c = Classes() 63 | c.append_if_new(TESTLIST1[0]) 64 | self.assertEqual(len(c), 1) 65 | self.assertSequenceEqual(c, TESTLIST1[:1]) 66 | 67 | def test_append_if_new_duplicate(self): 68 | c = Classes(TESTLIST1) 69 | c.append_if_new(TESTLIST1[0]) 70 | self.assertEqual(len(c), len(TESTLIST1)) 71 | self.assertSequenceEqual(c, TESTLIST1) 72 | 73 | def test_append_if_new_nonstring(self): 74 | c = Classes() 75 | with self.assertRaises(TypeError): 76 | c.append_if_new(0) 77 | 78 | def test_append_invalid_characters(self): 79 | c = Classes() 80 | invalid_name = ' '.join(('foo', 'bar')) 81 | with self.assertRaises(InvalidClassnameError) as e: 82 | c.append_if_new(invalid_name) 83 | self.assertEqual(e.exception.message, "Invalid character ' ' in class name 'foo bar'.") 84 | 85 | def test_merge_unique(self): 86 | c = Classes(TESTLIST1) 87 | c.merge_unique(TESTLIST2) 88 | self.assertSequenceEqual(c, TESTLIST1 + TESTLIST2) 89 | 90 | def test_merge_unique_duplicate1_list(self): 91 | c = Classes(TESTLIST1) 92 | c.merge_unique(TESTLIST1) 93 | self.assertSequenceEqual(c, TESTLIST1) 94 | 95 | def test_merge_unique_duplicate1_instance(self): 96 | c = Classes(TESTLIST1) 97 | c.merge_unique(Classes(TESTLIST1)) 98 | self.assertSequenceEqual(c, TESTLIST1) 99 | 100 | def test_merge_unique_duplicate2_list(self): 101 | c = Classes(TESTLIST1) 102 | c.merge_unique(TESTLIST2 + TESTLIST2) 103 | self.assertSequenceEqual(c, TESTLIST1 + TESTLIST2) 104 | 105 | def test_merge_unique_duplicate2_instance(self): 106 | c = Classes(TESTLIST1) 107 | c.merge_unique(Classes(TESTLIST2 + TESTLIST2)) 108 | self.assertSequenceEqual(c, TESTLIST1 + TESTLIST2) 109 | 110 | def test_merge_unique_nonstring(self): 111 | c = Classes() 112 | with self.assertRaises(TypeError): 113 | c.merge_unique([0,1,2]) 114 | 115 | def test_repr_empty(self): 116 | c = Classes() 117 | self.assertEqual('%r' % c, '%s(%r)' % (c.__class__.__name__, [])) 118 | 119 | def test_repr_contents(self): 120 | c = Classes(TESTLIST1) 121 | self.assertEqual('%r' % c, '%s(%r)' % (c.__class__.__name__, TESTLIST1)) 122 | 123 | def test_as_list(self): 124 | c = Classes(TESTLIST1) 125 | self.assertListEqual(c.as_list(), TESTLIST1) 126 | 127 | if __name__ == '__main__': 128 | unittest.main() 129 | -------------------------------------------------------------------------------- /reclass/adapters/ansible.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of reclass (http://github.com/madduck/reclass) 5 | # 6 | # IMPORTANT NOTICE: I was kicked out of the Ansible community, and therefore 7 | # I have no interest in developing this adapter anymore. If you use it and 8 | # have changes, I will take your patch. 9 | # 10 | # Copyright © 2007–14 martin f. krafft 11 | # Released under the terms of the Artistic Licence 2.0 12 | # 13 | # 2017.08.08 Andew Pickford 14 | # The ansible adapter has received little testing and may not work at all now. 15 | 16 | 17 | from __future__ import absolute_import 18 | from __future__ import division 19 | from __future__ import print_function 20 | from __future__ import unicode_literals 21 | 22 | import os, sys, posix, optparse 23 | 24 | from six import iteritems 25 | 26 | from reclass import get_storage, output 27 | from reclass.core import Core 28 | from reclass.errors import ReclassException 29 | from reclass.config import find_and_read_configfile, get_options 30 | from reclass.version import * 31 | from reclass.constants import MODE_NODEINFO 32 | from reclass.settings import Settings 33 | 34 | def cli(): 35 | try: 36 | # this adapter has to be symlinked to ansible_dir, so we can use this 37 | # information to initialise the inventory_base_uri to ansible_dir: 38 | ansible_dir = os.path.abspath(os.path.dirname(sys.argv[0])) 39 | 40 | defaults = {'inventory_base_uri': ansible_dir, 41 | 'no_refs' : False, 42 | 'pretty_print' : True, 43 | 'output' : 'json', 44 | 'applications_postfix': '_hosts' 45 | } 46 | defaults.update(find_and_read_configfile()) 47 | 48 | def add_ansible_options_group(parser, defaults): 49 | group = optparse.OptionGroup(parser, 'Ansible options', 50 | 'Ansible-specific options') 51 | group.add_option('--applications-postfix', 52 | dest='applications_postfix', 53 | default=defaults.get('applications_postfix'), 54 | help='postfix to append to applications to '\ 55 | 'turn them into groups') 56 | parser.add_option_group(group) 57 | 58 | options = get_options(RECLASS_NAME, VERSION, DESCRIPTION, 59 | inventory_shortopt='-l', 60 | inventory_longopt='--list', 61 | inventory_help='output the inventory', 62 | nodeinfo_shortopt='-t', 63 | nodeinfo_longopt='--host', 64 | nodeinfo_dest='hostname', 65 | nodeinfo_help='output host_vars for the given host', 66 | add_options_cb=add_ansible_options_group, 67 | defaults=defaults) 68 | 69 | storage = get_storage(options.storage_type, 70 | options.nodes_uri, 71 | options.classes_uri, 72 | options.compose_node_name) 73 | class_mappings = defaults.get('class_mappings') 74 | defaults.update(vars(options)) 75 | settings = Settings(defaults) 76 | reclass = Core(storage, class_mappings, settings) 77 | 78 | if options.mode == MODE_NODEINFO: 79 | data = reclass.nodeinfo(options.hostname) 80 | # Massage and shift the data like Ansible wants it 81 | data['parameters']['__reclass__'] = data['__reclass__'] 82 | for i in ('classes', 'applications'): 83 | data['parameters']['__reclass__'][i] = data[i] 84 | data = data['parameters'] 85 | 86 | else: 87 | data = reclass.inventory() 88 | # Ansible inventory is only the list of groups. Groups are the set 89 | # of classes plus the set of applications with the postfix added: 90 | groups = data['classes'] 91 | apps = data['applications'] 92 | if options.applications_postfix: 93 | postfix = options.applications_postfix 94 | groups.update([(k + postfix, v) for (k, v) in iteritems(apps)]) 95 | else: 96 | groups.update(apps) 97 | 98 | data = groups 99 | 100 | print(output(data, options.output, options.pretty_print, options.no_refs)) 101 | 102 | except ReclassException as e: 103 | e.exit_with_message(sys.stderr) 104 | 105 | sys.exit(posix.EX_OK) 106 | 107 | if __name__ == '__main__': 108 | cli() 109 | -------------------------------------------------------------------------------- /reclass/values/tests/test_parser_functions.py: -------------------------------------------------------------------------------- 1 | from reclass import settings 2 | from reclass.values import parser_funcs as pf 3 | import unittest 4 | import ddt 5 | 6 | 7 | SETTINGS = settings.Settings() 8 | 9 | # Test cases for parsers. Each test case is a two-tuple of input string and 10 | # expected output. NOTE: default values for sentinels are used here to avoid 11 | # cluttering up the code. 12 | test_pairs_simple = ( 13 | # Basic test cases. 14 | ('${foo}', [(pf.tags.REF, [(pf.tags.STR, 'foo')])]), 15 | # Basic combinations. 16 | ('bar${foo}', [(pf.tags.STR, 'bar'), 17 | (pf.tags.REF, [(pf.tags.STR, 'foo')])]), 18 | ('bar${foo}baz', [(pf.tags.STR, 'bar'), 19 | (pf.tags.REF, [(pf.tags.STR, 'foo')]), 20 | (pf.tags.STR, 'baz')]), 21 | ('${foo}baz', [(pf.tags.REF, [(pf.tags.STR, 'foo')]), 22 | (pf.tags.STR, 'baz')]), 23 | # Whitespace preservation cases. 24 | ('bar ${foo}', [(pf.tags.STR, 'bar '), 25 | (pf.tags.REF, [(pf.tags.STR, 'foo')])]), 26 | ('bar ${foo baz}', [(pf.tags.STR, 'bar '), 27 | (pf.tags.REF, [(pf.tags.STR, 'foo baz')])]), 28 | ('bar${foo} baz', [(pf.tags.STR, 'bar'), 29 | (pf.tags.REF, [(pf.tags.STR, 'foo')]), 30 | (pf.tags.STR, ' baz')]), 31 | (' bar${foo} baz ', [(pf.tags.STR, ' bar'), 32 | (pf.tags.REF, [(pf.tags.STR, 'foo')]), 33 | (pf.tags.STR, ' baz ')]), 34 | ) 35 | 36 | # Simple parser test cases are also included in this test grouop. 37 | test_pairs_full = ( 38 | # Single elements sanity. 39 | ('foo', [(pf.tags.STR, 'foo')]), 40 | ('$foo', [(pf.tags.STR, '$foo')]), 41 | ('{foo}', [(pf.tags.STR, '{foo}')]), 42 | ('[foo]', [(pf.tags.STR, '[foo]')]), 43 | ('$(foo)', [(pf.tags.STR, '$(foo)')]), 44 | ('$[foo]', [(pf.tags.INV, [(pf.tags.STR, 'foo')])]), 45 | 46 | # Escape sequences. 47 | # NOTE: these sequences apparently are not working as expected. 48 | #(r'\\\\${foo}', [(pf.tags.REF, [(pf.tags.STR, 'foo')])]), 49 | #(r'\\${foo}', [(pf.tags.REF, [(pf.tags.STR, 'foo')])]), 50 | #(r'\${foo}', [(pf.tags.REF, [(pf.tags.STR, 'foo')])]), 51 | 52 | # Basic combinations. 53 | ('bar$[foo]', [(pf.tags.STR, 'bar'), 54 | (pf.tags.INV, [(pf.tags.STR, 'foo')])]), 55 | ('bar$[foo]baz', [(pf.tags.STR, 'bar'), 56 | (pf.tags.INV, [(pf.tags.STR, 'foo')]), 57 | (pf.tags.STR, 'baz')]), 58 | ('$[foo]baz', [(pf.tags.INV, [(pf.tags.STR, 'foo')]), 59 | (pf.tags.STR, 'baz')]), 60 | 61 | # Whitespace preservation in various positions. 62 | (' foo ', [(pf.tags.STR, ' foo ')]), 63 | ('foo bar', [(pf.tags.STR, 'foo bar')]), 64 | ('bar $[foo baz]', [(pf.tags.STR, 'bar '), 65 | (pf.tags.INV, [(pf.tags.STR, 'foo baz')])]), 66 | ('bar$[foo] baz ', [(pf.tags.STR, 'bar'), 67 | (pf.tags.INV, [(pf.tags.STR, 'foo')]), 68 | (pf.tags.STR, ' baz ')]), 69 | 70 | # Nested references and inventory items. 71 | ('${foo}${bar}',[(pf.tags.REF, [(pf.tags.STR, 'foo')]), 72 | (pf.tags.REF, [(pf.tags.STR, 'bar')])]), 73 | ('${foo${bar}}',[(pf.tags.REF, [(pf.tags.STR, 'foo'), 74 | (pf.tags.REF, [(pf.tags.STR, 'bar')])])]), 75 | ('$[foo]$[bar]',[(pf.tags.INV, [(pf.tags.STR, 'foo')]), 76 | (pf.tags.INV, [(pf.tags.STR, 'bar')])]), 77 | # NOTE: the cases below do not work as expected, which is probably a bug. 78 | # Any nesting in INV creates a string. 79 | #('${$[foo]}', [(pf.tags.REF, [(pf.tags.INV, [(pf.tags.STR, 'foo')])])]), 80 | #('$[${foo}]', [(pf.tags.INV, [(pf.tags.REF, [(pf.tags.STR, 'foo')])])]), 81 | #('$[foo$[bar]]',[(pf.tags.INV, [(pf.tags.STR, 'foo'), 82 | # (pf.tags.INV, [(pf.tags.STR, 'bar')])])]), 83 | 84 | ) + test_pairs_simple 85 | 86 | 87 | @ddt.ddt 88 | class TestRefParser(unittest.TestCase): 89 | 90 | @ddt.data(*test_pairs_full) 91 | def test_standard_reference_parser(self, data): 92 | instring, expected = data 93 | parser = pf.get_ref_parser(SETTINGS) 94 | 95 | result = pf.listify(parser.parseString(instring).asList()) 96 | 97 | self.assertEquals(expected, result) 98 | 99 | 100 | @ddt.ddt 101 | class TestSimpleRefParser(unittest.TestCase): 102 | 103 | @ddt.data(*test_pairs_simple) 104 | def test_standard_reference_parser(self, data): 105 | # NOTE: simple reference parser can parse references only. It fails 106 | # on inventory items. 107 | instring, expected = data 108 | parser = pf.get_simple_ref_parser(SETTINGS) 109 | 110 | result = pf.listify(parser.parseString(instring).asList()) 111 | 112 | self.assertEquals(expected, result) 113 | 114 | 115 | if __name__ == '__main__': 116 | unittest.main() 117 | -------------------------------------------------------------------------------- /reclass/values/tests/test_value.py: -------------------------------------------------------------------------------- 1 | # 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of reclass (http://github.com/madduck/reclass) 5 | # 6 | # Copyright © 2007–14 martin f. krafft 7 | # Released under the terms of the Artistic Licence 2.0 8 | # 9 | from __future__ import absolute_import 10 | from __future__ import division 11 | from __future__ import print_function 12 | from __future__ import unicode_literals 13 | 14 | from reclass.settings import Settings 15 | from reclass.values.value import Value 16 | from reclass.errors import ResolveError, ParseError 17 | import unittest 18 | 19 | SETTINGS = Settings() 20 | 21 | def _var(s): 22 | return '%s%s%s' % (SETTINGS.reference_sentinels[0], s, 23 | SETTINGS.reference_sentinels[1]) 24 | 25 | CONTEXT = {'favcolour':'yellow', 26 | 'motd':{'greeting':'Servus!', 27 | 'colour':'${favcolour}' 28 | }, 29 | 'int':1, 30 | 'list':[1,2,3], 31 | 'dict':{1:2,3:4}, 32 | 'bool':True 33 | } 34 | 35 | def _poor_mans_template(s, var, value): 36 | return s.replace(_var(var), value) 37 | 38 | class TestValue(unittest.TestCase): 39 | 40 | def test_simple_string(self): 41 | s = 'my cat likes to hide in boxes' 42 | tv = Value(s, SETTINGS, '') 43 | self.assertFalse(tv.has_references) 44 | self.assertEquals(tv.render(CONTEXT, None), s) 45 | 46 | def _test_solo_ref(self, key): 47 | s = _var(key) 48 | tv = Value(s, SETTINGS, '') 49 | res = tv.render(CONTEXT, None) 50 | self.assertTrue(tv.has_references) 51 | self.assertEqual(res, CONTEXT[key]) 52 | 53 | def test_solo_ref_string(self): 54 | self._test_solo_ref('favcolour') 55 | 56 | def test_solo_ref_int(self): 57 | self._test_solo_ref('int') 58 | 59 | def test_solo_ref_list(self): 60 | self._test_solo_ref('list') 61 | 62 | def test_solo_ref_dict(self): 63 | self._test_solo_ref('dict') 64 | 65 | def test_solo_ref_bool(self): 66 | self._test_solo_ref('bool') 67 | 68 | def test_single_subst_bothends(self): 69 | s = 'I like ' + _var('favcolour') + ' and I like it' 70 | tv = Value(s, SETTINGS, '') 71 | self.assertTrue(tv.has_references) 72 | self.assertEqual(tv.render(CONTEXT, None), 73 | _poor_mans_template(s, 'favcolour', 74 | CONTEXT['favcolour'])) 75 | 76 | def test_single_subst_start(self): 77 | s = _var('favcolour') + ' is my favourite colour' 78 | tv = Value(s, SETTINGS, '') 79 | self.assertTrue(tv.has_references) 80 | self.assertEqual(tv.render(CONTEXT, None), 81 | _poor_mans_template(s, 'favcolour', 82 | CONTEXT['favcolour'])) 83 | 84 | def test_single_subst_end(self): 85 | s = 'I like ' + _var('favcolour') 86 | tv = Value(s, SETTINGS, '') 87 | self.assertTrue(tv.has_references) 88 | self.assertEqual(tv.render(CONTEXT, None), 89 | _poor_mans_template(s, 'favcolour', 90 | CONTEXT['favcolour'])) 91 | 92 | def test_deep_subst_solo(self): 93 | motd = SETTINGS.delimiter.join(('motd', 'greeting')) 94 | s = _var(motd) 95 | tv = Value(s, SETTINGS, '') 96 | self.assertTrue(tv.has_references) 97 | self.assertEqual(tv.render(CONTEXT, None), 98 | _poor_mans_template(s, motd, 99 | CONTEXT['motd']['greeting'])) 100 | 101 | def test_multiple_subst(self): 102 | greet = SETTINGS.delimiter.join(('motd', 'greeting')) 103 | s = _var(greet) + ' I like ' + _var('favcolour') + '!' 104 | tv = Value(s, SETTINGS, '') 105 | self.assertTrue(tv.has_references) 106 | want = _poor_mans_template(s, greet, CONTEXT['motd']['greeting']) 107 | want = _poor_mans_template(want, 'favcolour', CONTEXT['favcolour']) 108 | self.assertEqual(tv.render(CONTEXT, None), want) 109 | 110 | def test_multiple_subst_flush(self): 111 | greet = SETTINGS.delimiter.join(('motd', 'greeting')) 112 | s = _var(greet) + ' I like ' + _var('favcolour') 113 | tv = Value(s, SETTINGS, '') 114 | self.assertTrue(tv.has_references) 115 | want = _poor_mans_template(s, greet, CONTEXT['motd']['greeting']) 116 | want = _poor_mans_template(want, 'favcolour', CONTEXT['favcolour']) 117 | self.assertEqual(tv.render(CONTEXT, None), want) 118 | 119 | def test_undefined_variable(self): 120 | s = _var('no_such_variable') 121 | tv = Value(s, SETTINGS, '') 122 | with self.assertRaises(ResolveError): 123 | tv.render(CONTEXT, None) 124 | 125 | def test_incomplete_variable(self): 126 | s = SETTINGS.reference_sentinels[0] + 'incomplete' 127 | with self.assertRaises(ParseError): 128 | tv = Value(s, SETTINGS, '') 129 | 130 | if __name__ == '__main__': 131 | unittest.main() 132 | -------------------------------------------------------------------------------- /reclass/datatypes/entity.py: -------------------------------------------------------------------------------- 1 | # 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of reclass (http://github.com/madduck/reclass) 5 | # 6 | # Copyright © 2007–14 martin f. krafft 7 | # Released under the terms of the Artistic Licence 2.0 8 | # 9 | from __future__ import absolute_import 10 | from __future__ import division 11 | from __future__ import print_function 12 | from __future__ import unicode_literals 13 | 14 | from .classes import Classes 15 | from .applications import Applications 16 | from .exports import Exports 17 | from .parameters import Parameters 18 | 19 | class Entity(object): 20 | ''' 21 | A collection of Classes, Parameters, and Applications, mainly as a wrapper 22 | for merging. The name and uri of an Entity will be updated to the name and 23 | uri of the Entity that is being merged. 24 | ''' 25 | def __init__(self, settings, classes=None, applications=None, 26 | parameters=None, exports=None, uri=None, name=None, 27 | pathname=None, environment=None): 28 | self._uri = '' if uri is None else uri 29 | self._name = '' if name is None else name 30 | self._pathname = '' if pathname is None else pathname 31 | self._classes = self._set_field(classes, Classes) 32 | self._applications = self._set_field(applications, Applications) 33 | pars = [None, settings, uri] 34 | self._parameters = self._set_field(parameters, Parameters, pars) 35 | self._exports = self._set_field(exports, Exports, pars) 36 | self._environment = environment 37 | self._original_environment = environment 38 | 39 | name = property(lambda s: s._name) 40 | uri = property(lambda s: s._uri) 41 | pathname = property(lambda s: s._pathname) 42 | classes = property(lambda s: s._classes) 43 | applications = property(lambda s: s._applications) 44 | parameters = property(lambda s: s._parameters) 45 | exports = property(lambda s: s._exports) 46 | original_environment = property(lambda s: s._original_environment) 47 | 48 | @property 49 | def environment(self): 50 | return self._environment 51 | 52 | @environment.setter 53 | def environment(self, value): 54 | self._environment = value 55 | 56 | def _set_field(self, received_value, expected_type, parameters=None): 57 | if parameters is None: 58 | parameters = [] 59 | if received_value is None: 60 | return expected_type(*parameters) 61 | if not isinstance(received_value, expected_type): 62 | raise TypeError('Entity.%s cannot be set to instance of type %s' % 63 | (type(expected_type), type(received_value))) 64 | return received_value 65 | 66 | def merge(self, other): 67 | self._classes.merge_unique(other.classes) 68 | self._applications.merge_unique(other.applications) 69 | self._parameters.merge(other.parameters) 70 | self._exports.merge(other.exports) 71 | self._name = other.name 72 | self._uri = other.uri 73 | self._parameters._uri = other.uri 74 | if other.environment != None: 75 | self._environment = other.environment 76 | self._original_environment = other.original_environment 77 | 78 | def merge_parameters(self, params): 79 | self._parameters.merge(params) 80 | 81 | def interpolate(self, inventory): 82 | self._parameters.interpolate(inventory) 83 | self.interpolate_exports() 84 | 85 | def initialise_interpolation(self): 86 | self._parameters.initialise_interpolation() 87 | self._exports.initialise_interpolation() 88 | 89 | def interpolate_exports(self): 90 | self.initialise_interpolation() 91 | self._exports.interpolate_from_external(self._parameters) 92 | 93 | def interpolate_single_export(self, references): 94 | self._exports.interpolate_single_from_external(self._parameters, references) 95 | 96 | def __eq__(self, other): 97 | return isinstance(other, type(self)) \ 98 | and self._applications == other.applications \ 99 | and self._classes == other.classes \ 100 | and self._parameters == other.parameters \ 101 | and self._exports == other.exports \ 102 | and self._name == other.name \ 103 | and self._uri == other.uri 104 | 105 | def __ne__(self, other): 106 | return not self.__eq__(other) 107 | 108 | def __repr__(self): 109 | return "%s(%r, %r, %r, %r, uri=%r, name=%r, pathname=%r, environment=%r)" % ( 110 | self.__class__.__name__, self.classes, self.applications, 111 | self.parameters, self.exports, self.uri, self.name, 112 | self.pathname, self.environment) 113 | 114 | def as_dict(self): 115 | return {'classes': self._classes.as_list(), 116 | 'applications': self._applications.as_list(), 117 | 'parameters': self._parameters.as_dict(), 118 | 'exports': self._exports.as_dict(), 119 | 'environment': self._environment 120 | } 121 | -------------------------------------------------------------------------------- /reclass/utils/tests/test_dictpath.py: -------------------------------------------------------------------------------- 1 | # 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of reclass (http://github.com/madduck/reclass) 5 | # 6 | # Copyright © 2007–14 martin f. krafft 7 | # Released under the terms of the Artistic Licence 2.0 8 | # 9 | from __future__ import absolute_import 10 | from __future__ import division 11 | from __future__ import print_function 12 | from __future__ import unicode_literals 13 | 14 | from reclass.utils.dictpath import DictPath 15 | import unittest 16 | 17 | class TestDictPath(unittest.TestCase): 18 | 19 | def test_constructor0(self): 20 | p = DictPath(':') 21 | self.assertListEqual(p._parts, []) 22 | 23 | def test_constructor_list(self): 24 | l = ['a', 'b', 'c'] 25 | p = DictPath(':', l) 26 | self.assertListEqual(p._parts, l) 27 | 28 | def test_constructor_str(self): 29 | delim = ':' 30 | s = 'a{0}b{0}c'.format(delim) 31 | l = ['a', 'b', 'c'] 32 | p = DictPath(delim, s) 33 | self.assertListEqual(p._parts, l) 34 | 35 | def test_constructor_str_escaped(self): 36 | delim = ':' 37 | s = 'a{0}b\{0}b{0}c'.format(delim) 38 | l = ['a', 'b\\{0}b'.format(delim), 'c'] 39 | p = DictPath(delim, s) 40 | self.assertListEqual(p._parts, l) 41 | 42 | def test_constructor_invalid_type(self): 43 | with self.assertRaises(TypeError): 44 | p = DictPath(':', 5) 45 | 46 | def test_equality(self): 47 | delim = ':' 48 | s = 'a{0}b{0}c'.format(delim) 49 | l = ['a', 'b', 'c'] 50 | p1 = DictPath(delim, s) 51 | p2 = DictPath(delim, l) 52 | self.assertEqual(p1, p2) 53 | 54 | def test_inequality_content(self): 55 | delim = ':' 56 | s = 'a{0}b{0}c'.format(delim) 57 | l = ['d', 'e', 'f'] 58 | p1 = DictPath(delim, s) 59 | p2 = DictPath(delim, l) 60 | self.assertNotEqual(p1, p2) 61 | 62 | def test_inequality_delimiter(self): 63 | l = ['a', 'b', 'c'] 64 | p1 = DictPath(':', l) 65 | p2 = DictPath('%', l) 66 | self.assertNotEqual(p1, p2) 67 | 68 | def test_repr(self): 69 | delim = '%' 70 | s = 'a:b\:b:c' 71 | p = DictPath(delim, s) 72 | self.assertEqual('%r' % p, "DictPath(%r, %r)" % (delim, str(s))) 73 | 74 | def test_str(self): 75 | s = 'a:b\:b:c' 76 | p = DictPath(':', s) 77 | self.assertEqual(str(p), s) 78 | 79 | def test_path_accessor(self): 80 | l = ['a', 'b', 'c'] 81 | p = DictPath(':', l) 82 | self.assertListEqual(p.path, l) 83 | 84 | def test_new_subpath(self): 85 | l = ['a', 'b', 'c'] 86 | p = DictPath(':', l[:-1]) 87 | p = p.new_subpath(l[-1]) 88 | self.assertListEqual(p.path, l) 89 | 90 | def test_get_value(self): 91 | v = 42 92 | l = ['a', 'b', 'c'] 93 | d = {'a':{'b':{'c':v}}} 94 | p = DictPath(':', l) 95 | self.assertEqual(p.get_value(d), v) 96 | 97 | def test_get_value_escaped(self): 98 | v = 42 99 | l = ['a', 'b:b', 'c'] 100 | d = {'a':{'b:b':{'c':v}}} 101 | p = DictPath(':', l) 102 | self.assertEqual(p.get_value(d), v) 103 | 104 | def test_get_value_listindex_list(self): 105 | v = 42 106 | l = ['a', 1, 'c'] 107 | d = {'a':[None, {'c':v}, None]} 108 | p = DictPath(':', l) 109 | self.assertEqual(p.get_value(d), v) 110 | 111 | def test_get_value_listindex_str(self): 112 | v = 42 113 | s = 'a:1:c' 114 | d = {'a':[None, {'c':v}, None]} 115 | p = DictPath(':', s) 116 | self.assertEqual(p.get_value(d), v) 117 | 118 | def test_set_value(self): 119 | v = 42 120 | l = ['a', 'b', 'c'] 121 | d = {'a':{'b':{'c':v}}} 122 | p = DictPath(':', l) 123 | p.set_value(d, v+1) 124 | self.assertEqual(d['a']['b']['c'], v+1) 125 | 126 | def test_set_value_escaped(self): 127 | v = 42 128 | l = ['a', 'b:b', 'c'] 129 | d = {'a':{'b:b':{'c':v}}} 130 | p = DictPath(':', l) 131 | p.set_value(d, v+1) 132 | self.assertEqual(d['a']['b:b']['c'], v+1) 133 | 134 | def test_set_value_escaped_listindex_list(self): 135 | v = 42 136 | l = ['a', 1, 'c'] 137 | d = {'a':[None, {'c':v}, None]} 138 | p = DictPath(':', l) 139 | p.set_value(d, v+1) 140 | self.assertEqual(d['a'][1]['c'], v+1) 141 | 142 | def test_set_value_escaped_listindex_str(self): 143 | v = 42 144 | s = 'a:1:c' 145 | d = {'a':[None, {'c':v}, None]} 146 | p = DictPath(':', s) 147 | p.set_value(d, v+1) 148 | self.assertEqual(d['a'][1]['c'], v+1) 149 | 150 | def test_get_nonexistent_value(self): 151 | l = ['a', 'd'] 152 | p = DictPath(':', l) 153 | with self.assertRaises(KeyError): 154 | p.get_value(dict()) 155 | 156 | def test_set_nonexistent_value(self): 157 | l = ['a', 'd'] 158 | p = DictPath(':', l) 159 | with self.assertRaises(KeyError): 160 | p.set_value(dict(), 42) 161 | 162 | if __name__ == '__main__': 163 | unittest.main() 164 | -------------------------------------------------------------------------------- /contrib/modules/pillar/reclass_adapter.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | ''' 3 | *** 2020-06-19 saltenv environment override: 4 | This reclass adapter is only required to enable salt commands setting saltenv or 5 | pillarenv to pass that value to reclass. If the configuration setting 6 | allow_adapter_env_override is False (the default) the value of saltenv or pillarenv 7 | is ignored and the environment of the node is taken from the node file as normal. 8 | If allow_adapter_env_override is True and saltenv or pillarenv is set (depending 9 | on which salt command is used) the value of saltenv/pillarenv will be used as the 10 | environment of the node. 11 | 12 | This file should be placed in the pillar directory of the extension_modules 13 | directory on the salt master. 14 | *** 15 | 16 | Use the "reclass" database as a Pillar source 17 | 18 | .. |reclass| replace:: **reclass** 19 | 20 | This ``ext_pillar`` plugin provides access to the |reclass| database, such 21 | that Pillar data for a specific minion are fetched using |reclass|. 22 | 23 | You can find more information about |reclass| at 24 | http://reclass.pantsfullofunix.net. 25 | 26 | To use the plugin, add it to the ``ext_pillar`` list in the Salt master config 27 | and tell |reclass| by way of a few options how and where to find the 28 | inventory: 29 | 30 | .. code-block:: yaml 31 | 32 | ext_pillar: 33 | - reclass: 34 | storage_type: yaml_fs 35 | inventory_base_uri: /srv/salt 36 | 37 | This would cause |reclass| to read the inventory from YAML files in 38 | ``/srv/salt/nodes`` and ``/srv/salt/classes``. 39 | 40 | If you are also using |reclass| as ``master_tops`` plugin, and you want to 41 | avoid having to specify the same information for both, use YAML anchors (take 42 | note of the differing data types for ``ext_pillar`` and ``master_tops``): 43 | 44 | .. code-block:: yaml 45 | 46 | reclass: &reclass 47 | storage_type: yaml_fs 48 | inventory_base_uri: /srv/salt 49 | reclass_source_path: ~/code/reclass 50 | 51 | ext_pillar: 52 | - reclass: *reclass 53 | 54 | master_tops: 55 | reclass: *reclass 56 | 57 | If you want to run reclass from source, rather than installing it, you can 58 | either let the master know via the ``PYTHONPATH`` environment variable, or by 59 | setting the configuration option, like in the example above. 60 | ''' 61 | 62 | 63 | # This file cannot be called reclass.py, because then the module import would 64 | # not work. Thanks to the __virtual__ function, however, the plugin still 65 | # responds to the name 'reclass'. 66 | 67 | # Import python libs 68 | from __future__ import absolute_import, print_function, unicode_literals 69 | 70 | # Import salt libs 71 | from salt.exceptions import SaltInvocationError 72 | from salt.utils.reclass import ( 73 | prepend_reclass_source_path, 74 | filter_out_source_path_option, 75 | set_inventory_base_uri_default 76 | ) 77 | 78 | # Import 3rd-party libs 79 | from salt.ext import six 80 | 81 | # Define the module's virtual name 82 | __virtualname__ = 'reclass' 83 | 84 | 85 | def __virtual__(retry=False): 86 | try: 87 | import reclass 88 | return __virtualname__ 89 | 90 | except ImportError as e: 91 | if retry: 92 | return False 93 | 94 | for pillar in __opts__.get('ext_pillar', []): 95 | if 'reclass' not in pillar: 96 | continue 97 | 98 | # each pillar entry is a single-key hash of name -> options 99 | opts = next(six.itervalues(pillar)) 100 | prepend_reclass_source_path(opts) 101 | break 102 | 103 | return __virtual__(retry=True) 104 | 105 | 106 | def ext_pillar(minion_id, pillar, **kwargs): 107 | ''' 108 | Obtain the Pillar data from **reclass** for the given ``minion_id``. 109 | ''' 110 | 111 | # If reclass is installed, __virtual__ put it onto the search path, so we 112 | # don't need to protect against ImportError: 113 | # pylint: disable=3rd-party-module-not-gated 114 | from reclass.adapters.salt import ext_pillar as reclass_ext_pillar 115 | from reclass.errors import ReclassException 116 | # pylint: enable=3rd-party-module-not-gated 117 | 118 | try: 119 | # the source path we used above isn't something reclass needs to care 120 | # about, so filter it: 121 | filter_out_source_path_option(kwargs) 122 | 123 | # if no inventory_base_uri was specified, initialize it to the first 124 | # file_roots of class 'base' (if that exists): 125 | set_inventory_base_uri_default(__opts__, kwargs) 126 | 127 | # if saltenv or pillarenv has been set add it to the kwargs, this allows 128 | # reclass to override a nodes environment 129 | env_override = None 130 | if __opts__.get('saltenv', None): 131 | env_override = __opts__['saltenv'] 132 | if __opts__.get('pillarenv', None): 133 | env_override = __opts__['pillarenv'] 134 | 135 | # I purposely do not pass any of __opts__ or __salt__ or __grains__ 136 | # to reclass, as I consider those to be Salt-internal and reclass 137 | # should not make any assumptions about it. 138 | return reclass_ext_pillar(minion_id, pillar, pillarenv=env_override, **kwargs) 139 | 140 | except TypeError as e: 141 | if 'unexpected keyword argument' in six.text_type(e): 142 | arg = six.text_type(e).split()[-1] 143 | raise SaltInvocationError('ext_pillar.reclass: unexpected option: ' 144 | + arg) 145 | else: 146 | raise 147 | 148 | except KeyError as e: 149 | if 'id' in six.text_type(e): 150 | raise SaltInvocationError('ext_pillar.reclass: __opts__ does not ' 151 | 'define minion ID') 152 | else: 153 | raise 154 | 155 | except ReclassException as e: 156 | raise SaltInvocationError('ext_pillar.reclass: {0}'.format(e)) 157 | -------------------------------------------------------------------------------- /contrib/modules/tops/reclass_adapter.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | ''' 3 | Read tops data from a reclass database 4 | 5 | .. |reclass| replace:: **reclass** 6 | 7 | This :ref:`master_tops ` plugin provides access to 8 | the |reclass| database, such that state information (top data) are retrieved 9 | from |reclass|. 10 | 11 | You can find more information about |reclass| at 12 | http://reclass.pantsfullofunix.net. 13 | 14 | To use the plugin, add it to the ``master_tops`` list in the Salt master config 15 | and tell |reclass| by way of a few options how and where to find the 16 | inventory: 17 | 18 | .. code-block:: yaml 19 | 20 | master_tops: 21 | reclass: 22 | storage_type: yaml_fs 23 | inventory_base_uri: /srv/salt 24 | 25 | This would cause |reclass| to read the inventory from YAML files in 26 | ``/srv/salt/nodes`` and ``/srv/salt/classes``. 27 | 28 | If you are also using |reclass| as ``ext_pillar`` plugin, and you want to 29 | avoid having to specify the same information for both, use YAML anchors (take 30 | note of the differing data types for ``ext_pillar`` and ``master_tops``): 31 | 32 | .. code-block:: yaml 33 | 34 | reclass: &reclass 35 | storage_type: yaml_fs 36 | inventory_base_uri: /srv/salt 37 | reclass_source_path: ~/code/reclass 38 | 39 | ext_pillar: 40 | - reclass: *reclass 41 | 42 | master_tops: 43 | reclass: *reclass 44 | 45 | If you want to run reclass from source, rather than installing it, you can 46 | either let the master know via the ``PYTHONPATH`` environment variable, or by 47 | setting the configuration option, like in the example above. 48 | ''' 49 | from __future__ import absolute_import, print_function, unicode_literals 50 | 51 | # This file cannot be called reclass.py, because then the module import would 52 | # not work. Thanks to the __virtual__ function, however, the plugin still 53 | # responds to the name 'reclass'. 54 | 55 | import sys 56 | from salt.utils.reclass import ( 57 | prepend_reclass_source_path, 58 | filter_out_source_path_option, 59 | set_inventory_base_uri_default 60 | ) 61 | 62 | from salt.exceptions import SaltInvocationError 63 | from salt.ext import six 64 | 65 | # Define the module's virtual name 66 | __virtualname__ = 'reclass' 67 | 68 | import logging 69 | log = logging.getLogger(__name__) 70 | 71 | def __virtual__(retry=False): 72 | try: 73 | import reclass 74 | return __virtualname__ 75 | except ImportError: 76 | if retry: 77 | return False 78 | 79 | opts = __opts__.get('master_tops', {}).get('reclass', {}) 80 | prepend_reclass_source_path(opts) 81 | return __virtual__(retry=True) 82 | 83 | 84 | def top(**kwargs): 85 | ''' 86 | Query |reclass| for the top data (states of the minions). 87 | ''' 88 | 89 | # If reclass is installed, __virtual__ put it onto the search path, so we 90 | # don't need to protect against ImportError: 91 | # pylint: disable=3rd-party-module-not-gated 92 | from reclass.adapters.salt import top as reclass_top 93 | from reclass.errors import ReclassException 94 | # pylint: enable=3rd-party-module-not-gated 95 | 96 | try: 97 | # Salt's top interface is inconsistent with ext_pillar (see #5786) and 98 | # one is expected to extract the arguments to the master_tops plugin 99 | # by parsing the configuration file data. I therefore use this adapter 100 | # to hide this internality. 101 | reclass_opts = __opts__['master_tops']['reclass'] 102 | 103 | # the source path we used above isn't something reclass needs to care 104 | # about, so filter it: 105 | filter_out_source_path_option(reclass_opts) 106 | 107 | # if no inventory_base_uri was specified, initialise it to the first 108 | # file_roots of class 'base' (if that exists): 109 | set_inventory_base_uri_default(__opts__, kwargs) 110 | 111 | # Salt expects the top data to be filtered by minion_id, so we better 112 | # let it know which minion it is dealing with. Unfortunately, we must 113 | # extract these data (see #6930): 114 | minion_id = kwargs['opts']['id'] 115 | 116 | # if saltenv or pillarenv has been set add it to the kwargs, this allows 117 | # reclass to override a nodes environment 118 | env_override = None 119 | if kwargs['opts'].get('saltenv', None): 120 | env_override = kwargs['opts']['saltenv'] 121 | if kwargs['opts'].get('pillarenv', None): 122 | env_override = kwargs['opts']['pillarenv'] 123 | 124 | # I purposely do not pass any of __opts__ or __salt__ or __grains__ 125 | # to reclass, as I consider those to be Salt-internal and reclass 126 | # should not make any assumptions about it. Reclass only needs to know 127 | # how it's configured, so: 128 | return reclass_top(minion_id, pillarenv=env_override, **reclass_opts) 129 | 130 | except ImportError as e: 131 | if 'reclass' in six.text_type(e): 132 | raise SaltInvocationError( 133 | 'master_tops.reclass: cannot find reclass module ' 134 | 'in {0}'.format(sys.path) 135 | ) 136 | else: 137 | raise 138 | 139 | except TypeError as e: 140 | if 'unexpected keyword argument' in six.text_type(e): 141 | arg = six.text_type(e).split()[-1] 142 | raise SaltInvocationError( 143 | 'master_tops.reclass: unexpected option: {0}'.format(arg) 144 | ) 145 | else: 146 | raise 147 | 148 | except KeyError as e: 149 | if 'reclass' in six.text_type(e): 150 | raise SaltInvocationError('master_tops.reclass: no configuration ' 151 | 'found in master config') 152 | else: 153 | raise 154 | 155 | except ReclassException as e: 156 | raise SaltInvocationError('master_tops.reclass: {0}'.format(six.text_type(e))) 157 | -------------------------------------------------------------------------------- /reclass/utils/dictpath.py: -------------------------------------------------------------------------------- 1 | # 2 | # -*- coding: utf-8 -*- 3 | # 4 | # This file is part of reclass (http://github.com/madduck/reclass) 5 | # 6 | # Copyright © 2007–14 martin f. krafft 7 | # Released under the terms of the Artistic Licence 2.0 8 | # 9 | from __future__ import absolute_import 10 | from __future__ import division 11 | from __future__ import print_function 12 | from __future__ import unicode_literals 13 | 14 | import six 15 | import re 16 | 17 | class DictPath(object): 18 | ''' 19 | Represents a path into a nested dictionary. 20 | 21 | Given a dictionary like 22 | 23 | d['foo']['bar'] = 42 24 | 25 | it can be desirable to obtain a reference to the value stored in the 26 | sub-levels, allowing that value to be accessed and changed. Unfortunately, 27 | Python provides no easy way to do this, since 28 | 29 | ref = d['foo']['bar'] 30 | 31 | does become a reference to the integer 42, but that reference is 32 | overwritten when one assigns to it. Hence, DictPath represents the path 33 | into a nested dictionary, and can be "applied to" a dictionary to obtain 34 | and set values, using a list of keys, or a string representation using 35 | a delimiter (which can be escaped): 36 | 37 | p = DictPath(':', 'foo:bar') 38 | p.get_value(d) 39 | p.set_value(d, 43) 40 | 41 | This is a bit backwards, but the right way around would require support by 42 | the dict() type. 43 | 44 | The primary purpose of this class within reclass is to cater for parameter 45 | interpolation, so that a reference such as ${foo:bar} in a parameter value 46 | may be resolved in the context of the Parameter collections (a nested 47 | dict). 48 | 49 | If the value is a list, then the "key" is assumed to be and interpreted as 50 | an integer index: 51 | 52 | d = {'list': [{'one':1},{'two':2}]} 53 | p = DictPath(':', 'list:1:two') 54 | p.get_value(d) → 2 55 | 56 | This heuristic is okay within reclass, because dictionary keys (parameter 57 | names) will always be strings. Therefore it is okay to interpret each 58 | component of the path as a string, unless one finds a list at the current 59 | level down the nested dictionary. 60 | ''' 61 | 62 | def __init__(self, delim, contents=None): 63 | self._delim = delim 64 | 65 | if contents is None: 66 | self._parts = [] 67 | elif isinstance(contents, list): 68 | self._parts = contents 69 | elif isinstance(contents, six.string_types): 70 | self._parts = self._split_string(contents) 71 | elif isinstance(contents, tuple): 72 | self._parts = list(contents) 73 | else: 74 | raise TypeError('DictPath() takes string or list, '\ 75 | 'not %s' % type(contents)) 76 | 77 | def __repr__(self): 78 | return "DictPath(%r, %r)" % (self._delim, str(self)) 79 | 80 | def __str__(self): 81 | return self._delim.join(str(i) for i in self._parts) 82 | 83 | def __eq__(self, other): 84 | if not (isinstance(other, six.string_types) or 85 | isinstance(other, self.__class__)): 86 | return False 87 | if isinstance(other, six.string_types): 88 | other = DictPath(self._delim, other) 89 | return self._parts == other._parts and self._delim == other._delim 90 | 91 | def __ne__(self, other): 92 | return not self.__eq__(other) 93 | 94 | def __hash__(self): 95 | return hash(str(self)) 96 | 97 | @property 98 | def path(self): 99 | return self._parts 100 | 101 | def _get_key(self): 102 | if len(self._parts) == 0: 103 | return None 104 | return self._parts[-1] 105 | 106 | def _get_innermost_container(self, base): 107 | container = base 108 | for i in self.path[:-1]: 109 | if isinstance(container, (list, tuple)): 110 | container = container[int(i)] 111 | else: 112 | container = container[i] 113 | return container 114 | 115 | def _split_string(self, string): 116 | return re.split(r'(?' where is one of" 21 | @echo " html to make standalone HTML files" 22 | @echo " dirhtml to make HTML files named index.html in directories" 23 | @echo " singlehtml to make a single large HTML file" 24 | @echo " pickle to make pickle files" 25 | @echo " json to make JSON files" 26 | @echo " htmlhelp to make HTML files and a HTML help project" 27 | @echo " qthelp to make HTML files and a qthelp project" 28 | @echo " devhelp to make HTML files and a Devhelp project" 29 | @echo " epub to make an epub" 30 | @echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter" 31 | @echo " latexpdf to make LaTeX files and run them through pdflatex" 32 | @echo " text to make text files" 33 | @echo " man to make manual pages" 34 | @echo " texinfo to make Texinfo files" 35 | @echo " info to make Texinfo files and run them through makeinfo" 36 | @echo " gettext to make PO message catalogs" 37 | @echo " changes to make an overview of all changed/added/deprecated items" 38 | @echo " linkcheck to check all external links for integrity" 39 | @echo " doctest to run all doctests embedded in the documentation (if enabled)" 40 | 41 | clean: 42 | -rm -rf $(BUILDDIR)/* 43 | 44 | html: 45 | $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html 46 | @echo 47 | @echo "Build finished. The HTML pages are in $(BUILDDIR)/html." 48 | 49 | dirhtml: 50 | $(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml 51 | @echo 52 | @echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml." 53 | 54 | singlehtml: 55 | $(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml 56 | @echo 57 | @echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml." 58 | 59 | pickle: 60 | $(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle 61 | @echo 62 | @echo "Build finished; now you can process the pickle files." 63 | 64 | json: 65 | $(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json 66 | @echo 67 | @echo "Build finished; now you can process the JSON files." 68 | 69 | htmlhelp: 70 | $(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp 71 | @echo 72 | @echo "Build finished; now you can run HTML Help Workshop with the" \ 73 | ".hhp project file in $(BUILDDIR)/htmlhelp." 74 | 75 | qthelp: 76 | $(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp 77 | @echo 78 | @echo "Build finished; now you can run "qcollectiongenerator" with the" \ 79 | ".qhcp project file in $(BUILDDIR)/qthelp, like this:" 80 | @echo "# qcollectiongenerator $(BUILDDIR)/qthelp/reclass.qhcp" 81 | @echo "To view the help file:" 82 | @echo "# assistant -collectionFile $(BUILDDIR)/qthelp/reclass.qhc" 83 | 84 | devhelp: 85 | $(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp 86 | @echo 87 | @echo "Build finished." 88 | @echo "To view the help file:" 89 | @echo "# mkdir -p $$HOME/.local/share/devhelp/reclass" 90 | @echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/reclass" 91 | @echo "# devhelp" 92 | 93 | epub: 94 | $(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub 95 | @echo 96 | @echo "Build finished. The epub file is in $(BUILDDIR)/epub." 97 | 98 | latex: 99 | $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex 100 | @echo 101 | @echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex." 102 | @echo "Run \`make' in that directory to run these through (pdf)latex" \ 103 | "(use \`make latexpdf' here to do that automatically)." 104 | 105 | latexpdf: 106 | $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex 107 | @echo "Running LaTeX files through pdflatex..." 108 | $(MAKE) -C $(BUILDDIR)/latex all-pdf 109 | @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." 110 | 111 | text: 112 | $(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text 113 | @echo 114 | @echo "Build finished. The text files are in $(BUILDDIR)/text." 115 | 116 | man: 117 | $(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man 118 | @echo 119 | @echo "Build finished. The manual pages are in $(BUILDDIR)/man." 120 | 121 | texinfo: 122 | $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo 123 | @echo 124 | @echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo." 125 | @echo "Run \`make' in that directory to run these through makeinfo" \ 126 | "(use \`make info' here to do that automatically)." 127 | 128 | info: 129 | $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo 130 | @echo "Running Texinfo files through makeinfo..." 131 | make -C $(BUILDDIR)/texinfo info 132 | @echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo." 133 | 134 | gettext: 135 | $(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale 136 | @echo 137 | @echo "Build finished. The message catalogs are in $(BUILDDIR)/locale." 138 | 139 | changes: 140 | $(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes 141 | @echo 142 | @echo "The overview file is in $(BUILDDIR)/changes." 143 | 144 | linkcheck: 145 | $(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck 146 | @echo 147 | @echo "Link check complete; look for any errors in the above output " \ 148 | "or in $(BUILDDIR)/linkcheck/output.txt." 149 | 150 | doctest: 151 | $(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest 152 | @echo "Testing of doctests in the sources finished, look at the " \ 153 | "results in $(BUILDDIR)/doctest/output.txt." 154 | -------------------------------------------------------------------------------- /doc/source/concepts.rst: -------------------------------------------------------------------------------- 1 | ================ 2 | reclass concepts 3 | ================ 4 | |reclass| assumes a node-centric perspective into your inventory. This is 5 | obvious when you query |reclass| for node-specific information, but it might not 6 | be clear when you ask |reclass| to provide you with a list of groups. In that 7 | case, |reclass| loops over all nodes it can find in its database, reads all 8 | information it can find about the nodes, and finally reorders the result to 9 | provide a list of groups with the nodes they contain. 10 | 11 | Since the term "groups" is somewhat ambiguous, it helps to start off with 12 | a short glossary of |reclass|-specific terminology: 13 | 14 | ============ ============================================================== 15 | Concept Description 16 | ============ ============================================================== 17 | node A node, usually a computer in your infrastructure 18 | class A category, tag, feature, or role that applies to a node 19 | Classes may be nested, i.e. there can be a class hierarchy 20 | application A specific set of behaviour to apply 21 | parameter Node-specific variables, with inheritance throughout the class 22 | hierarchy. 23 | ============ ============================================================== 24 | 25 | A class consists of zero or more parent classes, zero or more applications, 26 | and any number of parameters. 27 | 28 | A class name must not contain spaces. 29 | 30 | A node is almost equivalent to a class, except that it usually does not (but 31 | can) specify applications. 32 | 33 | When |reclass| parses a node (or class) definition and encounters a parent 34 | class, it recurses to this parent class first before reading any data of the 35 | node (or class). When |reclass| returns from the recursive, depth first walk, it 36 | then merges all information of the current node (or class) into the 37 | information it obtained during the recursion. 38 | 39 | Furthermore, a node (or class) may define a list of classes it derives from, 40 | in which case classes defined further down the list will be able to override 41 | classes further up the list. 42 | 43 | Information in this context is essentially one of a list of applications or 44 | a list of parameters. 45 | 46 | The interaction between the depth-first walk and the delayed merging of data 47 | means that the node (and any class) may override any of the data defined by 48 | any of the parent classes (ancestors). This is in line with the assumption 49 | that more specific definitions ("this specific host") should have a higher 50 | precedence than more general definitions ("all webservers", which includes all 51 | webservers in Munich, which includes "this specific host", for example). 52 | 53 | Here's a quick example, showing how parameters accumulate and can get 54 | replaced. 55 | 56 | All "unixnodes" (i.e. nodes who have the ``unixnode`` class in their 57 | ancestry) have ``/etc/motd`` centrally-managed (through the ``motd`` 58 | application), and the `unixnode` class definition provides a generic 59 | message-of-the-day to be put into this file. 60 | 61 | All descendants of the class ``debiannode``, a descendant of ``unixnode``, 62 | should include the Debian codename in this message, so the 63 | message-of-the-day is overwritten in the ``debiannodes`` class. 64 | 65 | The node ``quantum.example.org`` (a `debiannode`) will have a scheduled 66 | downtime this weekend, so until Monday, an appropriate message-of-the-day is 67 | added to the node definition. 68 | 69 | When the ``motd`` application runs, it receives the appropriate 70 | message-of-the-day (from ``quantum.example.org`` when run on that node) and 71 | writes it into ``/etc/motd``. 72 | 73 | At this point it should be noted that parameters whose values are lists or 74 | key-value pairs don't get overwritten by children classes or node definitions, 75 | but the information gets merged (recursively) instead. 76 | 77 | Similarly to parameters, applications also accumulate during the recursive 78 | walk through the class ancestry. It is possible for a node or child class to 79 | *remove* an application added by a parent class, by prefixing the application 80 | with `~`. 81 | 82 | Finally, |reclass| happily lets you use multiple inheritance, and ensures that 83 | the resolution of parameters is still well-defined. Here's another example 84 | building upon the one about ``/etc/motd`` above: 85 | 86 | ``quantum.example.org`` (which is back up and therefore its node definition 87 | no longer contains a message-of-the-day) is at a site in Munich. Therefore, 88 | it is a child of the class ``hosted@munich``. This class is independent of 89 | the ``unixnode`` hierarchy, ``quantum.example.org`` derives from both. 90 | 91 | In this example infrastructure, ``hosted@munich`` is more specific than 92 | ``debiannode`` because there are plenty of Debian nodes at other sites (and 93 | some non-Debian nodes in Munich). Therefore, ``quantum.example.org`` derives 94 | from ``hosted@munich`` _after_ ``debiannodes``. 95 | 96 | When an electricity outage is expected over the weekend in Munich, the admin 97 | can change the message-of-the-day in the ``hosted@munich`` class, and it 98 | will apply to all hosts in Munich. 99 | 100 | However, not all hosts in Munich have ``/etc/motd``, because some of them 101 | are of class ``windowsnode``. Since the ``windowsnode`` ancestry does not 102 | specify the ``motd`` application, those hosts have access to the 103 | message-of-the-day in the node variables, but the message won't get used… 104 | 105 | … unless, of course, ``windowsnode`` specified a Windows-specific 106 | application to bring such notices to the attention of the user. 107 | 108 | It's also trivial to ensure a certain order of class evaluation. Here's 109 | another example: 110 | 111 | The ``ssh.server`` class defines the ``permit_root_login`` parameter to ``no``. 112 | 113 | The ``backuppc.client`` class defines the parameter to ``without-password``, 114 | because the BackupPC server might need to log in to the host as root. 115 | 116 | Now, what happens if the admin accidentally provides the following two 117 | classes? 118 | 119 | - ``backuppc.client`` 120 | - ``ssh.server`` 121 | 122 | Theoretically, this would mean ``permit_root_login`` gets set to ``no``. 123 | 124 | However, since all ``backuppc.client`` nodes need ``ssh.server`` (at least 125 | in most setups), the class ``backuppc.client`` itself derives from 126 | ``ssh.server``, ensuring that it gets parsed before ``backuppc.client``. 127 | 128 | When |reclass| returns to the node and encounters the ``ssh.server`` class 129 | defined there, it simply skips it, as it's already been processed. 130 | 131 | Now read about :doc:`operations`! 132 | 133 | .. include:: substs.inc 134 | -------------------------------------------------------------------------------- /doc/source/todo.rst: -------------------------------------------------------------------------------- 1 | ================== 2 | reclass to-do list 3 | ================== 4 | 5 | Common set of classes 6 | --------------------- 7 | A lot of the classes I have set up during the various stages of development of 8 | |reclass| are generic. It would probably be sensible to make them available as 9 | part of |reclass|, to give people a common baseline to work from, and to 10 | ensure a certain level of consistency between users. 11 | 12 | This could also provide a more realistic example to users on how to use 13 | |reclass|. 14 | 15 | Testing framework 16 | ----------------- 17 | There is rudimentary testing in place, but it's inconsistent. I got 18 | side-tracked into discussions about the philosphy of mocking objects. This 19 | could all be fixed and unified. 20 | 21 | Also, storage, outputters, CLI and adapters have absolutely no tests yet… 22 | 23 | The testing framework should also incorporate the example classes mentioned 24 | above. 25 | 26 | Configurable file extension 27 | --------------------------- 28 | Right now, ``.yml`` is hard-coded. This could be exported to the 29 | configuration file, or even given as a list, so that ``.yml`` and ``.yaml`` 30 | can both be used. 31 | 32 | Actually, I don't think this is such a good idea. If we create too many 33 | options right now, it'll be harder to unify later. Please also see `issue #17 34 | ` in a fixed set of locations. On of those derives from 104 | ``OPT_INVENTORY_BASE_URI``, the default inventory base URI (``/etc/reclass``). 105 | This should probably be updated in case the user changes the URI. 106 | 107 | Furthermore, ``$CWD`` and ``~`` might not make a lot of sense in all 108 | use-cases. 109 | 110 | However, this might be better addressed by the following point: 111 | 112 | Adapter class hierarchy 113 | ----------------------- 114 | At the moment, adapters are just imperative code. It might make more sense to 115 | wrap them in classes, which customise things like command-line and config file 116 | parsing. 117 | 118 | One nice way would be to generalise configuration file reading, integrate it 119 | with command-line parsing, and then allow the consumers (the adapters) to 120 | configure them, for instance, in the Salt adapter:: 121 | 122 | config_proxy = ConfigProxy() 123 | config_proxy.set_configfile_search_path(['/etc/reclass', '/etc/salt']) 124 | config_proxy.lock_config_option('output', 'yaml') 125 | 126 | The last call would effectively remove the ``--output`` config option from the 127 | CLI, and yield an error (or warning) if the option was encountered while 128 | parsing the configuration file. 129 | 130 | Furthermore, the class instances could become long-lived and keep a reference 131 | to a storage proxy, e.g. to prevent having to reload storage on every request. 132 | 133 | Node lists 134 | ---------- 135 | Class mappings are still experimental, and one of the reasons I am not too 136 | happy with them right now is that one would still need to provide node files 137 | for all nodes for ``inventory`` invocations. This is because class mappings 138 | can assign classes based on patterns or regular expressions, but it is not 139 | possible to turn a pattern or regular expression into a list of valid nodes. 140 | 141 | `Issue #9 `_ contains a lengthy 142 | discussion on this. At the moment, I am unsure what the best way forward is. 143 | 144 | Inventory filters 145 | ----------------- 146 | As described in `issue #11 `_: 147 | provide a means to limit the enumeration of the inventory, according to node 148 | name patterns, or using classes white-/blacklists. 149 | 150 | .. include:: substs.inc 151 | --------------------------------------------------------------------------------