├── poni ├── __init__.py ├── errors.py ├── rcontrol_all.py ├── cloud.py ├── cloud_image.py ├── colors.py ├── cloud_eucalyptus.py ├── work.py ├── vc.py ├── rcontrol_openssh.py ├── times.py ├── importer.py ├── newconfig.py ├── cloudbase.py ├── cloud_docker.py ├── template.py ├── util.py ├── recode.py ├── orddict.py └── listout.py ├── debian ├── compat ├── pyversions ├── changelog.in ├── rules ├── copyright └── control ├── doc ├── .gitignore ├── definitions.rst ├── index.rst ├── vc.rst ├── remote.rst ├── plugins.rst ├── Makefile ├── make.bat ├── overview.rst ├── template-variables.rst ├── conf.py ├── properties.rst ├── changes.rst ├── ec2.rst └── getting-started.rst ├── examples ├── hello │ └── hello │ │ ├── hello.txt │ │ └── plugin.py ├── db-cluster │ ├── report.txt │ ├── README.rst │ ├── tables2.sql │ ├── plugin.py │ ├── tables.sql │ ├── report_plugin.py │ ├── network.dot │ └── inst-db-cluster.sh └── puppet │ ├── ec2-deb6 │ ├── plugin.py │ └── deb6-upgrade.sh │ ├── puppet-master │ ├── site.pp │ ├── inst-puppet-master.sh │ └── plugin.py │ ├── puppet-agent │ ├── inst-puppet-agent.sh │ └── plugin.py │ └── inst-puppet.sh ├── setup.cfg ├── .gitignore ├── MANIFEST.in ├── tests ├── test_util.py ├── test_cloud_aws_comparison.py ├── test_settings.py ├── test_control.py ├── test_vc.py ├── helper.py ├── test_times.py ├── test_templates.py ├── test_cloud_libvirt.py └── test_cmd_basic.py ├── .travis.yml ├── package.mk ├── pylintrc ├── version.py ├── Makefile ├── setup.py ├── poni.spec └── README.rst /poni/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /debian/compat: -------------------------------------------------------------------------------- 1 | 8 2 | -------------------------------------------------------------------------------- /debian/pyversions: -------------------------------------------------------------------------------- 1 | 2.6- 2 | -------------------------------------------------------------------------------- /doc/.gitignore: -------------------------------------------------------------------------------- 1 | _build 2 | -------------------------------------------------------------------------------- /examples/hello/hello/hello.txt: -------------------------------------------------------------------------------- 1 | Hello from node $node.name, config $config.name! 2 | -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | [egg_info] 2 | tag_build = 3 | tag_date = 0 4 | tag_svn_revision = 0 5 | 6 | -------------------------------------------------------------------------------- /examples/db-cluster/report.txt: -------------------------------------------------------------------------------- 1 | blah 2 | 3 | #for $item in $dynconf 4 | $item.source => $item.dest ($item.protocol) 5 | #end for 6 | -------------------------------------------------------------------------------- /debian/changelog.in: -------------------------------------------------------------------------------- 1 | poni (0.1-1) unstable; urgency=low 2 | 3 | * Initial release 4 | 5 | -- Mika Eloranta Mon, 29 Nov 2010 09:37:54 +0200 6 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | *~ 2 | dist 3 | build 4 | *.pyc 5 | poni.egg-info/ 6 | cover 7 | .coverage 8 | poni/version.py 9 | 10 | README.html 11 | README.txt 12 | debian/ 13 | .pc/ 14 | -------------------------------------------------------------------------------- /debian/rules: -------------------------------------------------------------------------------- 1 | #!/usr/bin/make -f 2 | 3 | 4 | DEB_PYTHON_SYSTEM=pysupport 5 | 6 | include /usr/share/cdbs/1/rules/debhelper.mk 7 | include /usr/share/cdbs/1/class/python-distutils.mk 8 | 9 | 10 | -------------------------------------------------------------------------------- /examples/db-cluster/README.rst: -------------------------------------------------------------------------------- 1 | ================== 2 | db-cluster example 3 | ================== 4 | 5 | This is a dummy example demonstrating dynamic configuration of X db nodes 6 | with Y shards. 7 | 8 | -------------------------------------------------------------------------------- /examples/hello/hello/plugin.py: -------------------------------------------------------------------------------- 1 | from poni import config 2 | 3 | class PlugIn(config.PlugIn): 4 | def add_actions(self): 5 | self.add_file("hello.txt", mode=0644, dest_path="/tmp/") 6 | 7 | @config.control() 8 | def loadavg(self, arg): 9 | print "foo!" 10 | -------------------------------------------------------------------------------- /examples/puppet/ec2-deb6/plugin.py: -------------------------------------------------------------------------------- 1 | from poni import config 2 | 3 | class PlugIn(config.PlugIn): 4 | def add_actions(self): 5 | self.add_file("deb6-upgrade.sh", mode=0500, 6 | dest_path="/root/deb6-upgrade.sh", 7 | render=self.render_cheetah) 8 | 9 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | include poni/*.py 2 | include tests/*.py 3 | include README.html 4 | include README.rst 5 | include README.txt 6 | include version.py 7 | include setup.py 8 | include setup.cfg 9 | include LICENSE 10 | include MANIFEST.in 11 | 12 | recursive-include examples * 13 | recursive-exclude examples *~ *.pyc \.* 14 | -------------------------------------------------------------------------------- /examples/db-cluster/tables2.sql: -------------------------------------------------------------------------------- 1 | -- Hello from $node.name (parent system: $node.system.name) 2 | 3 | -- I'm a DB node, but I know that there are $get_system("frontend$").sub_count frontends: 4 | #for $fe in $find("frontend") 5 | -- * frontend $fe.name at address $fe.host $edge($node.name, $fe.name, protocol="http") 6 | #end for 7 | -------------------------------------------------------------------------------- /tests/test_util.py: -------------------------------------------------------------------------------- 1 | from poni import util 2 | import os 3 | 4 | 5 | def test_dir_stats(): 6 | poni_src_dir = os.path.dirname(os.path.dirname(__file__)) 7 | stats = util.dir_stats(poni_src_dir) 8 | assert stats['file_count'] > 30 9 | assert stats['total_bytes'] > 100000 10 | assert stats['path'] == poni_src_dir 11 | -------------------------------------------------------------------------------- /debian/copyright: -------------------------------------------------------------------------------- 1 | Format: http://dep.debian.net/deps/dep5 2 | Upstream-Name: poni 3 | Source: https://github.com/melor/poni/ 4 | 5 | Files: * 6 | Copyright: 2010-2012 Mika Eloranta 7 | License: Apache-2.0 8 | 9 | License: Apache-2.0 10 | On Debian GNU/Linux system you can find the complete text of the 11 | Apache 2.0 license in '/usr/share/common-licenses/Apache-2.0'. 12 | -------------------------------------------------------------------------------- /examples/db-cluster/plugin.py: -------------------------------------------------------------------------------- 1 | from poni import config 2 | 3 | class PlugIn(config.PlugIn): 4 | def add_actions(self): 5 | self.add_file("tables.sql", dest_path="/tmp/tables${node.index}.sql", 6 | render=self.render_cheetah) 7 | self.add_file("tables2.sql", dest_path="/tmp/blah${node.index}.sql", 8 | render=self.render_cheetah) 9 | 10 | -------------------------------------------------------------------------------- /doc/definitions.rst: -------------------------------------------------------------------------------- 1 | .. _`Amazon EC2`: http://aws.amazon.com/ec2/ 2 | .. _Paramiko: http://pypi.python.org/pypi/paramiko 3 | .. _Boto: http://pypi.python.org/pypi/boto 4 | .. _Argh: http://pypi.python.org/pypi/argh 5 | .. _GitPython: http://pypi.python.org/pypi/GitPython 6 | .. _Cheetah: http://pypi.python.org/pypi/Cheetah 7 | .. _Git: http://git-scm.com/ 8 | .. _JSON: http://json.org/ 9 | .. _UUID: http://en.wikipedia.org/wiki/Universally_unique_identifier 10 | -------------------------------------------------------------------------------- /examples/db-cluster/tables.sql: -------------------------------------------------------------------------------- 1 | -- id: $node.name 2 | -- this is db node $node.index and there are $system.sub_count nodes. 3 | -- the system has $system.shards shards, so... 4 | #set $start = $node.index * $system.shards / $system.sub_count 5 | #set $end = ($node.index + 1) * $system.shards / $system.sub_count 6 | -- I should be handling shards from $start to $end-1. 7 | 8 | #for $shard in $range($start, $end) 9 | CREATE TABLE data_${str(shard).zfill(3)} (a INT, b TEXT); 10 | #end for 11 | 12 | -------------------------------------------------------------------------------- /examples/puppet/puppet-master/site.pp: -------------------------------------------------------------------------------- 1 | node 'default' { 2 | notice 'no specific rules for node' 3 | } 4 | 5 | class nginx { 6 | package { nginx: 7 | ensure => latest 8 | } 9 | 10 | # service { nginx: 11 | # running => true 12 | # } 13 | } 14 | 15 | #for $agent in $find("demo") 16 | node '$agent.private.dns.lower()' { 17 | # fscm node: $agent.name 18 | file { "/etc/sudoers": 19 | owner => root, group => root, mode => 440 20 | } 21 | 22 | include nginx 23 | } 24 | #end for 25 | -------------------------------------------------------------------------------- /examples/puppet/ec2-deb6/deb6-upgrade.sh: -------------------------------------------------------------------------------- 1 | #! /bin/sh 2 | 3 | set -e 4 | 5 | # make upgrades quiet 6 | export DEBIAN_FRONTEND=noninteractive 7 | echo force-confold >> /etc/dpkg/dpkg.cfg 8 | echo force-confdef >> /etc/dpkg/dpkg.cfg 9 | 10 | # upgrade 11 | apt-get update 12 | 13 | set +e 14 | # this will fail... 15 | apt-get --yes dist-upgrade 16 | set -e 17 | 18 | # ...fix (hack) the failed upgrade and retry 19 | mv /etc/init.d/ec2* /root/ 20 | apt-get --yes -f install 21 | apt-get --yes dist-upgrade 22 | 23 | # add software 24 | apt-get --yes install ack-grep less 25 | -------------------------------------------------------------------------------- /examples/puppet/puppet-master/inst-puppet-master.sh: -------------------------------------------------------------------------------- 1 | #! /bin/bash -ue 2 | 3 | # bootstrap puppet master in a node 4 | 5 | # make apt-get upgrades quiet 6 | export DEBIAN_FRONTEND=noninteractive 7 | echo force-confold >> /etc/dpkg/dpkg.cfg 8 | echo force-confdef >> /etc/dpkg/dpkg.cfg 9 | 10 | apt-get -q --yes update 11 | 12 | # install 13 | perl -pi -e 's/(127.0.0.1.*)/\1 puppet/' /etc/hosts 14 | apt-get install -q --yes --force-yes puppetmaster puppet 15 | 16 | # wait until csr arrives... 17 | # puppetca --list 18 | # puppetca --sign ip-10-212-235-164.ec2.internal 19 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | language: python 2 | 3 | python: 4 | - "2.6" 5 | - "2.7" 6 | - "3.4" 7 | 8 | install: 9 | - pip install pytest pylint==1.3.1 argh boto docutils genshi GitPython paramiko setuptools sphinx 10 | - "(python -V 2>&1 | grep -qF 'Python 3') || pip install cheetah" 11 | 12 | script: 13 | - # Travis does a shallow clone and won't find tags with git describe 14 | - echo "__version__ = '0.8-travis'" > poni/version.py 15 | - make all rst2html=rst2html.py 16 | - "(python -V 2>&1 | grep -qF 'Python 2.6') || make pylint" 17 | - make tests 18 | 19 | notifications: 20 | slack: ohmu:rihgYxa0c4dFiPvdAPAgP7R4 21 | -------------------------------------------------------------------------------- /debian/control: -------------------------------------------------------------------------------- 1 | Source: poni 2 | Section: misc 3 | Priority: extra 4 | Maintainer: Mika Eloranta 5 | Build-Depends: cdbs (>= 0.4.49), debhelper (>= 7.0.50~), python-support, python (>=2.6) 6 | XS-Python-Version: >= 2.6 7 | Standards-Version: 3.9.1 8 | Homepage: http://github.com/melor/poni 9 | #Vcs-Git: git://git.debian.org/collab-maint/poni.git 10 | #Vcs-Browser: http://git.debian.org/?p=collab-maint/poni.git;a=summary 11 | 12 | Package: poni 13 | Architecture: all 14 | Depends: ${shlibs:Depends}, ${misc:Depends}, ${python:Depends} 15 | Description: systems configuration management tool 16 | Poni is TODO 17 | -------------------------------------------------------------------------------- /examples/db-cluster/report_plugin.py: -------------------------------------------------------------------------------- 1 | import os 2 | from poni import config 3 | import subprocess 4 | 5 | def dot(file_path): 6 | subprocess.call(["dot", "-Tpng", "-o", 7 | "%s.png" % os.path.splitext(file_path)[0], 8 | file_path]) 9 | 10 | 11 | class PlugIn(config.PlugIn): 12 | def add_actions(self): 13 | self.add_file("report.txt", dest_path="/tmp/report.txt", 14 | render=self.render_cheetah, report=True) 15 | self.add_file("network.dot", dest_path="/tmp/network.dot", 16 | render=self.render_cheetah, report=True, 17 | post_process=dot) 18 | 19 | -------------------------------------------------------------------------------- /examples/db-cluster/network.dot: -------------------------------------------------------------------------------- 1 | digraph G { 2 | 3 | #def foo($item) 4 | 5 | #if not $item.get("verify", True) or $item.name == "report" 6 | #foo 7 | #else if $item.type == "node" 8 | #set $nodelabel = $str(item).replace(", ", "\\n") 9 | "$item.name" [ label = "${item.name.rsplit("/")[-1]}\n$nodelabel" ]; 10 | #else 11 | subgraph "cluster_$item.name" { 12 | label = "$item.name\n$item"; 13 | 14 | #for $n in $find("^" + $item.name, nodes=True, systems=True, depth=[$item.depth + 1]) 15 | $foo($n) 16 | #end for 17 | } 18 | #end if 19 | 20 | #end def 21 | 22 | #for $n in $find(".", nodes=True, systems=True, depth=[1]) 23 | $foo($n) 24 | #end for 25 | 26 | #for $item in $dynconf 27 | "$item.source" -> "$item.dest" [ label = "$item.protocol" ]; 28 | #end for 29 | 30 | } 31 | -------------------------------------------------------------------------------- /tests/test_cloud_aws_comparison.py: -------------------------------------------------------------------------------- 1 | from pytest import skip 2 | from poni import cloud, cloud_aws 3 | 4 | def test_hash(): 5 | """validate aws provider hash and comparison implementation""" 6 | if not cloud_aws.boto_is_current: 7 | skip("boto is not installed or is too old") 8 | sky = cloud.Sky() 9 | east_prop = dict(provider="aws-ec2", region="us-east-1") 10 | east1 = sky.get_provider(east_prop) 11 | east2 = sky.get_provider(east_prop) 12 | assert hash(east1) == hash(east2) 13 | assert hash(east1) != hash(east1.get_provider_key(east_prop)) 14 | assert east1 == east2 15 | assert not east1 != east2 16 | 17 | west = sky.get_provider(dict(provider="aws-ec2", region="us-west-1")) 18 | assert hash(east1) != hash(west) 19 | assert not east1 == west 20 | assert east1 != west 21 | 22 | -------------------------------------------------------------------------------- /package.mk: -------------------------------------------------------------------------------- 1 | version = $(shell git describe --long) 2 | major_version = $(shell git describe --abbrev=0) 3 | 4 | export DEBFULLNAME := Mika Eloranta 5 | export DEBEMAIL := mika.eloranta@gmail.com 6 | 7 | rpm: poni/version.py 8 | git archive -o rpm-src-poni.tar --prefix=poni/ HEAD 9 | # add poni/version.py to the tar, it's not in git repository 10 | tar -r -f rpm-src-poni.tar --transform=s-poni-poni/poni- poni/version.py 11 | rpmbuild -ta rpm-src-poni.tar \ 12 | --define 'major_version $(major_version)' \ 13 | --define 'minor_version $(subst -,.,$(subst $(major_version)-,,$(version)))' 14 | $(RM) rpm-src-poni.tar 15 | 16 | deb: poni/version.py 17 | cp debian/changelog.in debian/changelog 18 | dch -v $(version) -D unstable "Automatic change log entry generated by make" 19 | dpkg-buildpackage -A -us -uc 20 | 21 | .PHONY: debian 22 | -------------------------------------------------------------------------------- /pylintrc: -------------------------------------------------------------------------------- 1 | [MESSAGES CONTROL] 2 | # C0302: 1,0: Too many lines in module (1247) 3 | # E0012:217,0: Bad option value 'W0123' 4 | # F0401: 28,4: Unable to import 'DNS' 5 | # W0511:192,0: TODO: separate stdout/stderr? 6 | # W0212: 62,23:InstanceStarter.__init__: Access to a protected member _get_instances of a client class 7 | # W0142:448,30:AwsProvider._run_instance: Used * or ** magic 8 | # W0603:491,8:ConfigMan.reset_cache: Using the global statement 9 | disable=I,R,C0111,C0103,F0401,R0201,W0612,W0613,C0301,C0302,W0511,W0212,W0142,W0603,E0012 10 | # In a newer pylint version we'd also like to disable the following errors, 11 | # but old pylint versions choke on message ids they don't recognize 12 | # C0325: Unnecessary parens after %r keyword 13 | # C0330: Wrong hanging indentation. 14 | 15 | [REPORTS] 16 | reports=no 17 | msg-template={msg_id}:{line:3d},{column}: {obj}: {msg} 18 | -------------------------------------------------------------------------------- /examples/puppet/puppet-agent/inst-puppet-agent.sh: -------------------------------------------------------------------------------- 1 | #! /bin/bash -ue 2 | 3 | # bootstrap puppet agent in a node 4 | 5 | # make upgrades quiet 6 | export DEBIAN_FRONTEND=noninteractive 7 | echo force-confold >> /etc/dpkg/dpkg.cfg 8 | echo force-confdef >> /etc/dpkg/dpkg.cfg 9 | 10 | apt-get -q update 11 | 12 | # add puppet master to /etc/hosts 13 | #set $master = $get_node("example/master") 14 | echo >> /etc/hosts 15 | echo "# puppet master" >> /etc/hosts 16 | echo "$master.private.ip $master.private.dns.lower() puppet" >> /etc/hosts 17 | 18 | # install puppet agent 19 | apt-get -q --yes --force-yes install puppet 20 | perl -pi -e 's/START=no/START=yes/' /etc/default/puppet 21 | /etc/init.d/puppet start 22 | 23 | # debug command if you get "Could not retrieve catalog from remote server: hostname was not match with the server certificate": 24 | # openssl s_client -connect $master.private.dns.lower():8140 -showcerts -showcerts > debug 25 | -------------------------------------------------------------------------------- /version.py: -------------------------------------------------------------------------------- 1 | """ 2 | automatically maintains the latest git tag + revision info in a python file 3 | 4 | """ 5 | 6 | import imp 7 | import subprocess 8 | 9 | def get_project_version(version_file): 10 | try: 11 | module = imp.load_source("verfile", version_file) 12 | file_ver = module.__version__ 13 | except: 14 | file_ver = None 15 | 16 | try: 17 | proc = subprocess.Popen(["git", "describe", "--long"], 18 | stdout=subprocess.PIPE, 19 | stderr=subprocess.PIPE) 20 | proc.stderr.close() 21 | git_ver = proc.stdout.readline().strip().decode("utf-8") 22 | if git_ver and ((git_ver != file_ver) or not file_ver): 23 | open(version_file, "w").write("__version__ = '%s'\n" % git_ver) 24 | return git_ver 25 | except: 26 | pass 27 | 28 | if not file_ver: 29 | raise Exception("version not available from git or from file %r" 30 | % version_file) 31 | 32 | return file_ver 33 | 34 | if __name__ == "__main__": 35 | import sys 36 | get_project_version(sys.argv[1]) 37 | -------------------------------------------------------------------------------- /poni/errors.py: -------------------------------------------------------------------------------- 1 | """ 2 | error types 3 | 4 | Copyright (c) 2010-2012 Mika Eloranta 5 | See LICENSE for details. 6 | 7 | """ 8 | 9 | class Error(Exception): 10 | """error""" 11 | 12 | class InvalidProperty(Error): 13 | """invalid property""" 14 | 15 | class MissingProperty(Error): 16 | """missing property""" 17 | 18 | class UserError(Error): 19 | """user error""" 20 | 21 | class InvalidRange(Error): 22 | """invalid range""" 23 | 24 | class SettingsError(Error): 25 | """settings error""" 26 | 27 | class VerifyError(Error): 28 | """verify error""" 29 | 30 | class TemplateError(Error): 31 | """template rendering error""" 32 | 33 | class CloudError(Error): 34 | """cloud error""" 35 | 36 | class RemoteError(Error): 37 | """remote error""" 38 | 39 | class RemoteFileDoesNotExist(RemoteError): 40 | """remote file does not exist""" 41 | 42 | class RepoError(Error): 43 | """repository error""" 44 | 45 | class ImporterError(Error): 46 | """importer error""" 47 | 48 | class MissingLibraryError(Error): 49 | """missing library error""" 50 | 51 | class RequirementError(Error): 52 | """requirement error""" 53 | 54 | class ControlError(Error): 55 | """control error""" 56 | 57 | class OperationError(Error): 58 | """operation error""" 59 | -------------------------------------------------------------------------------- /tests/test_settings.py: -------------------------------------------------------------------------------- 1 | import json 2 | from poni import tool 3 | from helper import * 4 | 5 | plugin_text = """ 6 | from poni import config 7 | 8 | class PlugIn(config.PlugIn): 9 | pass 10 | 11 | """ 12 | 13 | class TestSettings(Helper): 14 | def test_settings_list_empty(self): 15 | poni = self.repo_and_config("node", "conf", plugin_text) 16 | assert not poni.run(["settings", "list"]) 17 | 18 | def test_settings_set(self): 19 | input_dir = self.temp_file() 20 | settings_dir = os.path.join(input_dir, "settings") 21 | os.makedirs(settings_dir) 22 | defaults_file = os.path.join(settings_dir, "00-defaults.json") 23 | defaults = {"foo":"bar"} 24 | with open(defaults_file, "w") as f: 25 | json.dump(defaults, f) 26 | 27 | poni = self.repo_and_config("node", "conf", plugin_text, 28 | copy=input_dir) 29 | 30 | assert not poni.run(["settings", "list"]) 31 | # TODO: verify output 32 | 33 | assert not poni.run(["settings", "set", "node/conf", "foo=baz"]) 34 | assert not poni.run(["settings", "list"]) 35 | # TODO: verify list output 36 | 37 | # TODO: test for invalid 'set' value types 38 | # TODO: test for inherited scenarios 39 | -------------------------------------------------------------------------------- /examples/puppet/puppet-master/plugin.py: -------------------------------------------------------------------------------- 1 | import argh 2 | from poni import config 3 | from poni import errors 4 | 5 | INST_MASTER_SH = "/root/inst-puppet-master.sh" 6 | 7 | class PlugIn(config.PlugIn): 8 | @argh.arg("-x", "--extra", help="extra info") 9 | def install(self, arg): 10 | self.log.info("%s/%s install: verbose=%r, extra=%r", 11 | self.node.name, self.config.name, arg.verbose, 12 | arg.extra) 13 | if arg.extra: 14 | print "extra info: %r" % arg.extra 15 | 16 | remote = self.node.get_remote(override=arg.method) 17 | exit_code = remote.execute(INST_MASTER_SH, verbose=arg.verbose) 18 | if exit_code: 19 | raise errors.ControlError("%r failed with exit code %r" % ( 20 | INST_MASTER_SH, exit_code)) 21 | 22 | self.log.info("%s/%s install - done", 23 | self.node.name, self.config.name) 24 | 25 | def add_controls(self): 26 | self.add_argh_control(self.install, provides=["puppet-master"]) 27 | 28 | def add_actions(self): 29 | self.add_file("site.pp", dest_path="/etc/puppet/manifests/", 30 | report=True) 31 | self.add_file("inst-puppet-master.sh", mode=0500, 32 | dest_path=INST_MASTER_SH) 33 | -------------------------------------------------------------------------------- /tests/test_control.py: -------------------------------------------------------------------------------- 1 | import json 2 | from poni import tool 3 | from helper import * 4 | 5 | plugin_text = """ 6 | import argh 7 | from poni import config 8 | 9 | class PlugIn(config.PlugIn): 10 | @config.control(provides=["foo"]) 11 | @argh.arg("output") 12 | def foo(self, arg): 13 | open(arg.output, "a").write("foo") 14 | 15 | @config.control(requires=["foo"]) 16 | @argh.arg("output") 17 | def bar(self, arg): 18 | open(arg.output, "a").write("bar") 19 | 20 | @config.control(optional_requires=["foo"]) 21 | @argh.arg("output") 22 | def baz(self, arg): 23 | open(arg.output, "a").write("baz") 24 | 25 | @config.control(optional_requires=["xxx"]) 26 | @argh.arg("output") 27 | def bax(self, arg): 28 | open(arg.output, "a").write("bax") 29 | 30 | """ 31 | 32 | class TestControls(Helper): 33 | def test_basic_controls(self): 34 | poni = self.repo_and_config("node", "conf", plugin_text) 35 | temp = self.temp_file() 36 | 37 | def cmd_output(cmd, output): 38 | assert not poni.run(["control", ".", cmd, "--", temp]) 39 | assert open(temp).read() == output 40 | os.unlink(temp) 41 | 42 | cmd_output("foo", "foo") 43 | cmd_output("bar", "foobar") 44 | cmd_output("baz", "foobaz") 45 | cmd_output("bax", "bax") 46 | -------------------------------------------------------------------------------- /examples/db-cluster/inst-db-cluster.sh: -------------------------------------------------------------------------------- 1 | #! /bin/sh 2 | 3 | set -e 4 | 5 | REPO="/tmp/example-db-cluster" 6 | CONF_FILES="plugin.py tables.sql tables2.sql" 7 | SETTINGS="" 8 | REPORTS="report.txt network.dot" 9 | 10 | rm -rf $REPO 11 | 12 | echo create system 13 | 14 | poni -d $REPO script < $@ 21 | 22 | README.html: README.rst LICENSE 23 | $(rst2html) $< $@ 24 | 25 | clean: 26 | $(RM) -r dist/ build/ poni.egg-info/ cover/ 27 | $(RM) poni/version.py poni/*.pyc tests/*.pyc *.pyc README.html README.txt \ 28 | examples/puppet/README.html examples/db-cluster/README.html 29 | $(RM) ../poni?$(shell git describe)* \ 30 | ../poni?$(shell git describe --abbrev=0)-*.tar.gz 31 | $(MAKE) -C doc clean 32 | 33 | build-dep: 34 | apt-get --yes install python-setuptools python-docutils python-sphinx lynx 35 | 36 | test-dep: 37 | apt-get --yes install pep8 pylint python-pytest \ 38 | python-argh python-boto python-cheetah python-genshi python-git 39 | 40 | pep8: 41 | pep8 --ignore=E501 poni/*.py 42 | 43 | pylint: 44 | if $(PYTHON) -m pylint.lint --help-msg C0330 | grep -qF bad-continuation; \ 45 | then $(PYTHON) -m pylint.lint --rcfile pylintrc --disable=C0325,C0330 poni; \ 46 | else $(PYTHON) -m pylint.lint --rcfile pylintrc poni; \ 47 | fi 48 | 49 | tests: 50 | PYTHONPATH=. $(PYTHON) -m pytest -vv tests 51 | 52 | .PHONY: readme 53 | .PHONY: coverage 54 | .PHONY: tests 55 | .PHONY: dist 56 | .PHONY: doc 57 | -------------------------------------------------------------------------------- /poni/rcontrol_all.py: -------------------------------------------------------------------------------- 1 | """ 2 | Node remote control switchboard 3 | 4 | Copyright (c) 2010-2012 Mika Eloranta 5 | See LICENSE for details. 6 | 7 | """ 8 | 9 | from . import rcontrol 10 | from . import rcontrol_paramiko 11 | #from . import rcontrol_openssh 12 | from . import errors 13 | 14 | METHODS = { 15 | "ssh": rcontrol_paramiko.ParamikoRemoteControl, 16 | "local": rcontrol.LocalControl, 17 | "tar": rcontrol.LocalTarControl, 18 | } 19 | 20 | 21 | class RemoteManager(object): 22 | def __init__(self): 23 | self.remotes = {} 24 | 25 | def cleanup(self): 26 | for remote in self.remotes.values(): 27 | remote.close() 28 | 29 | def get_remote(self, node, method): 30 | method = method or "ssh" 31 | key = (node.name, method) 32 | remote = self.remotes.get(key) 33 | if not remote: 34 | parts = method.split(":", 1) 35 | if len(parts) == 2: 36 | method, args = parts 37 | args = [args] 38 | else: 39 | args = [] 40 | try: 41 | control_class = METHODS[method] 42 | except KeyError: 43 | raise errors.RemoteError( 44 | "unknown remote control method %r" % method) 45 | 46 | remote = control_class(node, *args) 47 | self.remotes[key] = remote 48 | 49 | return remote 50 | 51 | manager = RemoteManager() 52 | 53 | get_remote = manager.get_remote 54 | -------------------------------------------------------------------------------- /examples/puppet/inst-puppet.sh: -------------------------------------------------------------------------------- 1 | #! /bin/bash -ue 2 | # Usage: Requires defining AWS_KEYPAIR 3 | 4 | REPO="$HOME/tmp/puppet" 5 | TIMELOG="/tmp/puppet-timelog" 6 | MAX_SPOT_PRICE=0.090 7 | 8 | # remove old repo 9 | rm -rf $REPO 10 | 11 | # create new puppet repo with the example system 12 | poni -L "$TIMELOG" -d $REPO script - -v <=0.13"] 16 | 17 | try: 18 | import json 19 | except ImportError: 20 | depends.append("simplejson") 21 | 22 | try: 23 | import argparse 24 | except ImportError: 25 | depends.append("argparse") 26 | 27 | 28 | setup( 29 | name = 'poni', 30 | version = version.get_project_version("poni/version.py"), 31 | description = 'system configuration software', 32 | long_description = long_desc, 33 | author = "Mika Eloranta", 34 | author_email = "mika.eloranta@gmail.com", 35 | url = "http://github.com/melor/poni", 36 | classifiers = [ 37 | "Programming Language :: Python", 38 | "Development Status :: 4 - Beta", 39 | "License :: OSI Approved :: Apache Software License", 40 | "Intended Audience :: Developers", 41 | "Intended Audience :: System Administrators", 42 | "Operating System :: OS Independent", 43 | "Topic :: System :: Installation/Setup", 44 | "Topic :: System :: Software Distribution", 45 | "Topic :: Software Development :: Libraries :: Python Modules", 46 | ], 47 | packages = find_packages(), 48 | zip_safe = False, 49 | install_requires = depends, 50 | entry_points = { 51 | 'console_scripts': [ 52 | 'poni = poni.tool:Tool.run_exit', 53 | ] 54 | } 55 | ) 56 | 57 | os.chdir(old_dir) 58 | -------------------------------------------------------------------------------- /doc/index.rst: -------------------------------------------------------------------------------- 1 | .. Poni documentation master file, created by 2 | sphinx-quickstart on Sat Dec 4 17:47:18 2010. 3 | You can adapt this file completely to your liking, but it should at least 4 | contain the root `toctree` directive. 5 | 6 | Welcome to Poni's documentation! 7 | ================================ 8 | 9 | Poni is a systems management tool for defining, deploying and verifying complex 10 | multi-node computer systems. 11 | 12 | Overview 13 | -------- 14 | Poni helps managing systems in many ways: 15 | 16 | * Systems, nodes, installed software and settings are stored in a central Poni 17 | repository, so there is a single location where the system is defined and documented 18 | * Changes to the Poni repository can be version-controlled with Git_ allowing rollback 19 | and access to history of configuration changes 20 | * Applications are configured by rendering configuration templates, using a powerful 21 | template language (such as Cheetah_) reducing the need to write application-specific 22 | installation or configuration scripts 23 | * The entire system configuration is available when creating node-specific configuration 24 | files, making it easy to wire multiple nodes together 25 | * Virtual machines can be easily provisioned from a cloud provider (currently 26 | `Amazon EC2`_ is supported) 27 | 28 | Contents 29 | -------- 30 | 31 | .. toctree:: 32 | :maxdepth: 2 33 | 34 | overview 35 | changes 36 | getting-started 37 | properties 38 | modify 39 | template-variables 40 | plugins 41 | ec2 42 | vsphere 43 | remote 44 | vc 45 | examples/puppet 46 | 47 | .. 48 | Indices and tables 49 | ================== 50 | 51 | * :ref:`genindex` 52 | * :ref:`modindex` 53 | * :ref:`search` 54 | 55 | .. include:: definitions.rst 56 | -------------------------------------------------------------------------------- /poni/cloud.py: -------------------------------------------------------------------------------- 1 | """ 2 | Cloud VM instance operations: creating, querying status, terminating 3 | 4 | Copyright (c) 2010-2012 Mika Eloranta 5 | See LICENSE for details. 6 | 7 | """ 8 | from . import cloud_aws 9 | from . import cloud_docker 10 | from . import cloud_eucalyptus 11 | from . import cloud_image 12 | from . import cloud_libvirt 13 | from . import cloud_vsphere 14 | from . import errors 15 | from .cloudbase import Provider # provides backward compatibility with older extensions # pylint: disable=W0611 16 | 17 | 18 | PROVIDERS = { 19 | "aws-ec2": cloud_aws.AwsProvider, 20 | "docker": cloud_docker.DockerProvider, 21 | "eucalyptus": cloud_eucalyptus.EucalyptusProvider, 22 | "image": cloud_image.ImageProvider, 23 | "libvirt": cloud_libvirt.LibvirtProvider, 24 | "vsphere": cloud_vsphere.VSphereProvider, 25 | } 26 | 27 | 28 | class Sky(object): 29 | """Super-cloud provider""" 30 | def __init__(self): 31 | self.providers = {} 32 | 33 | def get_provider(self, cloud_prop): 34 | """ 35 | Return a suitable cloud Provider object for the given cloud properties 36 | input and specifically the 'provider' type attribute. 37 | """ 38 | provider_id = cloud_prop.get("provider") 39 | if not provider_id: 40 | raise errors.CloudError("cloud 'provider' property not set") 41 | 42 | try: 43 | provider_class = PROVIDERS[provider_id] 44 | except KeyError: 45 | raise errors.CloudError("unknown cloud provider %r" % (provider_id,)) 46 | 47 | key = provider_class.get_provider_key(cloud_prop) 48 | cached = self.providers.get(key) 49 | if not cached: 50 | cached = provider_class(cloud_prop) 51 | self.providers[key] = cached 52 | return cached 53 | 54 | return cached 55 | -------------------------------------------------------------------------------- /tests/test_vc.py: -------------------------------------------------------------------------------- 1 | from __future__ import print_function 2 | from pytest import skip 3 | from poni import tool, vc 4 | from helper import * 5 | import os 6 | import subprocess 7 | 8 | 9 | class TestVersionControl(Helper): 10 | @classmethod 11 | def setup_class(cls): 12 | if not vc.git: 13 | skip("GitPython not installed or too old") 14 | 15 | def git(self, repo, cmd): 16 | full_cmd = ["git", 17 | "--git-dir=%s/.git" % repo, 18 | "--work-tree=%s" % repo] + cmd 19 | print("git cmd: %r" % full_cmd) 20 | git = subprocess.Popen(full_cmd, stdout=subprocess.PIPE, 21 | stderr=subprocess.PIPE) 22 | git.wait() 23 | stdout = git.stdout.read() 24 | stderr = git.stderr.read() 25 | print("git stdout: %r" % stdout) 26 | print("git stderr: %r" % stderr) 27 | assert git.returncode == 0 28 | return stdout 29 | 30 | def vc_init(self): 31 | poni, repo = self.init_repo() 32 | 33 | assert not poni.run(["add-node", "foo/bar"]) 34 | assert not poni.run(["add-config", "foo/bar", "baz"]) 35 | 36 | assert not poni.run(["vc", "init"]) 37 | 38 | assert os.path.exists(os.path.join(repo, ".git")) 39 | assert self.git(repo, ["status", "-s"]) == b"" 40 | 41 | return poni, repo 42 | 43 | def test_checkpoint(self): 44 | poni, repo = self.vc_init() 45 | assert not poni.run(["add-node", "foo/bar2"]) 46 | assert not poni.run(["add-config", "foo/bar2", "baz2"]) 47 | assert b"foo/bar2" in self.git(repo, ["status", "-s"]) 48 | assert not poni.run(["vc", "checkpoint", "checkpoint changes"]) 49 | assert self.git(repo, ["status", "-s"]) == b"" 50 | assert b"checkpoint changes" in self.git(repo, ["log"]) 51 | -------------------------------------------------------------------------------- /poni/cloud_image.py: -------------------------------------------------------------------------------- 1 | """ 2 | Cloud-provider implementation: Image builder 3 | 4 | Copyright (c) 2014 F-Secure Corporation 5 | See LICENSE for details. 6 | 7 | """ 8 | from . import cloudbase 9 | import copy 10 | import logging 11 | 12 | 13 | class ImageProvider(cloudbase.Provider): 14 | def __init__(self, cloud_prop): 15 | cloudbase.Provider.__init__(self, 'image', cloud_prop) 16 | self.log = logging.getLogger('poni.image') 17 | 18 | @classmethod 19 | def get_provider_key(cls, cloud_prop): 20 | """ 21 | Return a cloud Provider object for the given cloud properties. 22 | """ 23 | return ("image",) 24 | 25 | def init_instance(self, cloud_prop): 26 | """ 27 | Create a new instance with the given properties. 28 | 29 | Returns node properties that are changed. 30 | """ 31 | return self._updated_prop(cloud_prop) 32 | 33 | def _updated_prop(self, cloud_prop): 34 | ip = cloud_prop.get("dummy_ip", "127.255.255.255") 35 | vm_name = cloud_prop["vm_name"] 36 | out_prop = dict(cloud=copy.deepcopy(cloud_prop), 37 | host=vm_name, private=dict(ip=ip, dns=ip)) 38 | out_prop["cloud"]["instance"] = vm_name 39 | return out_prop 40 | 41 | def get_instance_status(self, prop): 42 | """ 43 | Return instance status string for the instance specified in the given 44 | cloud properties dict. 45 | """ 46 | return "stopped" 47 | 48 | def terminate_instances(self, props): 49 | """ 50 | Terminate instances specified in the given sequence of cloud 51 | properties dicts. 52 | """ 53 | return 54 | 55 | def wait_instances(self, props, wait_state="running"): 56 | """ 57 | Wait for all the given instances to reach status specified by 58 | the 'wait_state' argument. 59 | 60 | Returns a dict {instance_id: dict()} 61 | """ 62 | return dict((prop["instance"], self._updated_prop(prop)) for prop in props) 63 | -------------------------------------------------------------------------------- /tests/helper.py: -------------------------------------------------------------------------------- 1 | import json 2 | import itertools 3 | import shutil 4 | import os 5 | import tempfile 6 | import unittest 7 | from poni import tool 8 | 9 | 10 | class Helper(unittest.TestCase): 11 | def __init__(self, *args, **kwa): 12 | super(Helper, self).__init__(*args, **kwa) 13 | self.temp_files = [] 14 | 15 | def temp_file(self, prefix="test_poni"): 16 | f = tempfile.mktemp(prefix=prefix) 17 | self.temp_files.append(f) 18 | return f 19 | 20 | def temp_dir(self, prefix="test_poni_dir"): 21 | d = tempfile.mkdtemp(prefix=prefix) 22 | self.temp_files.append(d) 23 | return d 24 | 25 | def setup(self): 26 | pass 27 | 28 | def teardown(self): 29 | for temp_file in self.temp_files: 30 | if os.path.isfile(temp_file): 31 | os.unlink(temp_file) 32 | elif os.path.isdir(temp_file): 33 | shutil.rmtree(temp_file) 34 | 35 | def init_repo(self): 36 | repo = self.temp_file() 37 | poni = tool.Tool(default_repo_path=repo) 38 | assert not poni.run(["init"]) 39 | config = json.load(open(os.path.join(repo, "repo.json"))) 40 | assert isinstance(config, dict) 41 | return poni, repo 42 | 43 | def repo_and_config(self, node, conf, plugin_text, copy=None): 44 | poni, repo = self.init_repo() 45 | assert not poni.run(["add-node", node]) 46 | add_conf = ["add-config", node, conf] 47 | if copy: 48 | add_conf.extend(["-d", copy]) 49 | 50 | assert not poni.run(add_conf) 51 | conf_path = os.path.join(node, conf) 52 | output_dir = self.temp_file() 53 | os.makedirs(output_dir) 54 | plugin_py = os.path.join(output_dir, "plugin.py") 55 | open(plugin_py, "w").write(plugin_text) 56 | assert not poni.run(["update-config", conf_path, plugin_py]) 57 | return poni 58 | 59 | 60 | def combos(seq, max_len=None): 61 | seq = list(seq) 62 | for length in range(0, max_len or (len(seq) + 1)): 63 | for combo in itertools.combinations(seq, length): 64 | yield combo 65 | -------------------------------------------------------------------------------- /tests/test_times.py: -------------------------------------------------------------------------------- 1 | from __future__ import print_function 2 | import random 3 | import json 4 | from poni import tool 5 | from helper import * 6 | from poni import times 7 | 8 | def test_times(): 9 | tasks = times.Times() 10 | N = 10 11 | for i in range(0, N): 12 | start = random.uniform(1000, 2000) 13 | stop = start + random.uniform(1, 1000) 14 | tasks.add_task(i, "task%d" % i, start, stop) 15 | 16 | report = "".join(tasks.iter_report()) 17 | print(report) 18 | for i in range(0, N): 19 | name = "task%d" % i 20 | assert name in report, "%r missing from report" % name 21 | 22 | 23 | 24 | plugin_text = """ 25 | from poni import config 26 | import time 27 | 28 | class PlugIn(config.PlugIn): 29 | @config.control() 30 | def dummy(self, arg): 31 | print("dummy") 32 | 33 | """ 34 | 35 | class TestClockedOps(Helper): 36 | def setup_method(self, method): 37 | Helper.setup(self) 38 | self.poni = self.repo_and_config("node", "conf", plugin_text) 39 | self.time_log = self.temp_file() 40 | self.verify = [] 41 | 42 | def op(self, args, verify=None): 43 | full_args = ["-L", self.time_log] + args 44 | assert not self.poni.run(full_args) 45 | if verify: 46 | self.verify.append(verify) 47 | self.verify_log(self.verify) 48 | 49 | def test_simple(self): 50 | self.op(["-T", "one", "version"], 51 | verify=dict(name="one", task_id="C")) 52 | self.op(["-T", "-", "version"], 53 | verify=dict(name="-L %s -T - version" % self.time_log, task_id="C")) 54 | self.op(["-T", "two", "version"], 55 | verify=dict(name="two", task_id="C")) 56 | self.op(["control", "-t", "node/conf", "dummy"], 57 | verify=dict(task_id=0, name="node/conf")) 58 | self.op(["report"]) 59 | # TODO: verify output 60 | 61 | def verify_log(self, ops): 62 | with open(self.time_log, "r") as f: 63 | data = json.load(f) 64 | def has_all(a, b): 65 | assert a["stop"] >= a["start"] 66 | for k, v in b.items(): 67 | if a.get(k) != v: 68 | return False 69 | 70 | return True 71 | 72 | for op in ops: 73 | assert any(has_all(d, op) for d in data) 74 | -------------------------------------------------------------------------------- /poni/colors.py: -------------------------------------------------------------------------------- 1 | """ 2 | ANSI color code escapes for output 3 | 4 | Copyright (c) 2010-2012 Mika Eloranta 5 | See LICENSE for details. 6 | 7 | """ 8 | from __future__ import print_function 9 | 10 | CODES = { 11 | 'reset': '\033[0;m', 12 | 'gray' : '\033[1;30m', 13 | 'red' : '\033[1;31m', 14 | 'green' : '\033[1;32m', 15 | 'yellow' : '\033[1;33m', 16 | 'blue' : '\033[1;34m', 17 | 'magenta' : '\033[1;35m', 18 | 'cyan' : '\033[1;36m', 19 | 'white' : '\033[1;37m', 20 | 'crimson' : '\033[1;38m', 21 | 'hred' : '\033[1;41m', 22 | 'hgreen' : '\033[1;42m', 23 | 'hbrown' : '\033[1;43m', 24 | 'hblue' : '\033[1;44m', 25 | 'hmagenta' : '\033[1;45m', 26 | 'hcyan' : '\033[1;46m', 27 | 'hgray' : '\033[1;47m', 28 | 'lgreen' : '\033[0;32m', 29 | 'lred' : '\033[0;31m', 30 | 'lcyan' : '\033[0;36m', 31 | 'lyellow' : '\033[0;33m', 32 | 'bold': '\033[1m', 33 | } 34 | 35 | CODES.update({ 36 | 'key' : '\033[0;36m', 37 | 'cloudkey' : '\033[0;35m', 38 | 'str' : '\033[0;32m', 39 | 'bool' : CODES['yellow'], 40 | 'int' : CODES['bold'], 41 | 'status': CODES['red'], 42 | 'system': CODES['cyan'], 43 | 'node': CODES['green'], 44 | 'nodetype': CODES['lgreen'], 45 | 'systemtype': CODES['lcyan'], 46 | 'configtype': CODES['lyellow'], 47 | 'config': CODES['yellow'], 48 | 'configparent' : '\033[0;33m', 49 | 'nodeparent' : '\033[0;32m', 50 | 'header': CODES['cyan'], 51 | 'path': CODES['lyellow'], 52 | 'host': CODES['lyellow'], 53 | 'command': CODES['bold'], 54 | 'op_error': CODES['hred'], 55 | 'op_ok': CODES['green'], 56 | 'setting': CODES['reset'], 57 | 'layer': CODES['reset'], 58 | 'controls': CODES['lred'], 59 | 'controlstype': CODES['lred'], 60 | None: CODES["reset"], 61 | }) 62 | 63 | 64 | class Output(object): 65 | def __init__(self, out_file, color="auto"): 66 | self.out = out_file 67 | if ((color == "on") or (color == "auto" 68 | and (hasattr(out_file, 'isatty') 69 | and out_file.isatty()))): 70 | self.color = lambda text, code: "%s%s%s" % (CODES[code], 71 | text, 72 | CODES["reset"]) 73 | else: 74 | self.color = lambda text, code: text 75 | 76 | 77 | if __name__ == "__main__": 78 | for name, code in sorted(CODES.items()): 79 | print("%s%s%s" % (code, name, CODES["reset"])) 80 | -------------------------------------------------------------------------------- /poni/cloud_eucalyptus.py: -------------------------------------------------------------------------------- 1 | """ 2 | Cloud-provider implementation: Eucalyptus 3 | 4 | Implement Eucalyptus support by overriding AWS provider connection method. 5 | 6 | Copyright (c) 2010-2013 Mika Eloranta 7 | Copyright (c) 2012-2014 F-Secure 8 | See LICENSE for details. 9 | 10 | """ 11 | import logging 12 | import os 13 | try: 14 | from urllib.parse import urlsplit # pylint: disable=E0611 15 | except ImportError: 16 | from urlparse import urlsplit 17 | 18 | from . import errors 19 | 20 | from . import cloud_aws 21 | 22 | EUCALYPTUS = "eucalyptus" 23 | 24 | try: 25 | import boto 26 | except ImportError: 27 | boto = None 28 | 29 | 30 | class EucalyptusProvider(cloud_aws.AwsProvider): 31 | @classmethod 32 | def get_provider_key(cls, cloud_prop): 33 | endpoint_url = os.environ.get('EC2_URL') 34 | if not endpoint_url: 35 | raise errors.CloudError( 36 | "EC2_URL must be set for Eucalyptus provider") 37 | 38 | # ("eucalyptus", endpoint) uniquely identifies the DC we are talking to 39 | return (EUCALYPTUS, endpoint_url) 40 | 41 | def __init__(self, cloud_prop): 42 | # the chain will end up with provider_id set by the AWS module, so we reset it here 43 | cloud_aws.AwsProvider.__init__(self, cloud_prop) 44 | self.provider_id = EUCALYPTUS 45 | self.log = logging.getLogger(EUCALYPTUS) 46 | 47 | def _get_conn(self): 48 | if self._conn: 49 | return self._conn 50 | 51 | required_env = ["AWS_ACCESS_KEY_ID", "AWS_SECRET_ACCESS_KEY", "EC2_URL"] 52 | for env_key in required_env: 53 | if not os.environ.get(env_key): 54 | raise errors.CloudError("%r environment variable must be set" 55 | % env_key) 56 | 57 | try: 58 | parsed_url = urlsplit(os.environ.get("EC2_URL")) 59 | host, port = parsed_url.netloc.split(':', 1) # pylint: disable=E1103 60 | port = int(port) 61 | except (ValueError, AttributeError): 62 | raise errors.CloudError("Failed to parse EC2_URL environmental variable") 63 | 64 | self._conn = boto.connect_euca(aws_access_key_id=os.environ['AWS_ACCESS_KEY_ID'], aws_secret_access_key=os.environ['AWS_SECRET_ACCESS_KEY'], 65 | host=host, 66 | port=port, 67 | path=parsed_url.path) # pylint: disable=E1103 68 | return self._conn 69 | 70 | def _get_vpc_conn(self): 71 | raise errors.CloudError("Eucalyptus doesn't support VPCs yet") 72 | -------------------------------------------------------------------------------- /poni/work.py: -------------------------------------------------------------------------------- 1 | """ 2 | parallel task management 3 | 4 | Copyright (c) 2010-2012 Mika Eloranta 5 | See LICENSE for details. 6 | 7 | """ 8 | 9 | import logging 10 | import threading 11 | import time 12 | try: 13 | import Queue as queue 14 | except ImportError: 15 | import queue 16 | 17 | 18 | class Task(threading.Thread): 19 | def __init__(self, target=None): 20 | threading.Thread.__init__(self, target=target) 21 | self.log = logging.getLogger("task") 22 | self.daemon = True 23 | self.runner = None 24 | self.start_time = None 25 | self.stop_time = None 26 | 27 | def can_start(self): 28 | return True 29 | 30 | def execute(self): 31 | assert False, "override in sub-class" 32 | 33 | def run(self): 34 | try: 35 | self.start_time = time.time() 36 | self.execute() 37 | finally: 38 | self.stop_time = time.time() 39 | self.runner.task_finished(self) 40 | 41 | 42 | class Runner(object): 43 | def __init__(self, max_jobs=None): 44 | self.log = logging.getLogger("runner") 45 | self.not_started = set() 46 | self.started = set() 47 | self.stopped = set() 48 | self.finished_queue = queue.Queue() 49 | self.max_jobs = max_jobs 50 | 51 | def add_task(self, task): 52 | task.runner = self 53 | self.not_started.add(task) 54 | 55 | def task_finished(self, task): 56 | self.finished_queue.put(task) 57 | 58 | def check(self): 59 | for task in list(self.not_started): 60 | if self.max_jobs and (len(self.started) >= self.max_jobs): 61 | continue 62 | 63 | if not task.can_start(): 64 | continue 65 | 66 | self.started.add(task) 67 | self.not_started.remove(task) 68 | task.start() 69 | 70 | def wait_task_to_finish(self): 71 | try: 72 | task = self.finished_queue.get(timeout=60.0) 73 | except queue.Empty: 74 | self.log.warning( 75 | "tasks taking long to finish: %s and %r tasks waiting to be started", 76 | ", ".join(str(task) for task in self.started), len(self.not_started)) 77 | return 78 | 79 | self.log.debug("task %s finished, took %.2f seconds", task, 80 | (task.stop_time - task.start_time)) 81 | self.started.remove(task) 82 | self.stopped.add(task) 83 | self.finished_queue.task_done() 84 | 85 | def run_all(self): 86 | while self.not_started or self.started: 87 | self.check() 88 | self.wait_task_to_finish() 89 | -------------------------------------------------------------------------------- /poni/vc.py: -------------------------------------------------------------------------------- 1 | """ 2 | version control with Git 3 | 4 | Copyright (c) 2010-2012 Mika Eloranta 5 | See LICENSE for details. 6 | 7 | """ 8 | 9 | # pylint doesn't like git.Repo 10 | # pylint: disable=E1101 11 | # pylint: disable=E1103 12 | 13 | 14 | import os 15 | import re 16 | 17 | try: 18 | import git 19 | if git.__version__ < '0.3': 20 | git = None 21 | except ImportError: 22 | git = None 23 | 24 | GIT_IGNORE = """\ 25 | *~ 26 | *.pyc 27 | """ 28 | 29 | class VersionControl(object): 30 | def __init__(self, repo_dir): 31 | self.repo_dir = repo_dir 32 | 33 | 34 | class GitVersionControl(VersionControl): 35 | def __init__(self, repo_dir, init=False): 36 | assert git, "GitPython not installed or too old." 37 | VersionControl.__init__(self, repo_dir) 38 | if init: 39 | self.init_repo(repo_dir) 40 | else: 41 | self.git = git.Repo(repo_dir) 42 | 43 | self.add = self.git.index.add 44 | self.commit = self.git.index.commit 45 | 46 | def init_repo(self, repo_dir): 47 | self.git = git.Repo.init(repo_dir) 48 | open(os.path.join(repo_dir, ".gitignore"), "w").write(GIT_IGNORE) 49 | self.git.index.add([".gitignore"]) 50 | self.commit_all("initial commit") 51 | 52 | def get_deleted_files(self): 53 | """get a list deleted files that are not yet staged for commit""" 54 | status = self.git.git.status() 55 | not_staged = "\n# Changes not staged for commit" 56 | idx = status.find(not_staged) 57 | if not idx: 58 | return [] 59 | return re.findall(r"^#\s+deleted:\s+(.+)$", status[idx:], re.MULTILINE) 60 | 61 | def commit_all(self, message): 62 | self.git.index.add(["*"]) 63 | deleted = self.get_deleted_files() 64 | if deleted: 65 | # must delete in batches in order to avoid "OSError: [Errno 7] Argument list too long" 66 | BATCH = 1000 67 | deleted = list(deleted) 68 | for i in range(0, len(deleted), BATCH): 69 | self.git.index.remove(deleted[i:i + BATCH]) 70 | self.git.index.commit(message) 71 | 72 | def status(self): 73 | diff = self.git.git.diff() 74 | if diff: 75 | yield "Changes:\n" 76 | yield diff 77 | 78 | untracked = self.git.untracked_files 79 | if untracked: 80 | yield "\n\nUntracked files:\n" 81 | for file_path in untracked: 82 | yield " %s\n" % file_path 83 | 84 | 85 | def create_vc(repo_dir): 86 | if os.path.exists(os.path.join(repo_dir, ".git")): 87 | return GitVersionControl(repo_dir) 88 | else: 89 | return None 90 | 91 | -------------------------------------------------------------------------------- /poni.spec: -------------------------------------------------------------------------------- 1 | Name: poni 2 | Version: %{major_version} 3 | Release: %{minor_version}%{?dist} 4 | Summary: simple system configuration management tool 5 | 6 | Group: Development/Languages 7 | License: ASL 2.0 8 | Source0: rpm-src-poni.tar 9 | 10 | Requires: python-argh, python-boto, python-dns, python-lxml, libvirt-python 11 | BuildRequires: pylint, pytest 12 | BuildArch: noarch 13 | 14 | %description 15 | Poni is a simple system configuration management tool. 16 | 17 | This is the Python 2 package of Poni. 18 | 19 | %if %{?python3_sitelib:1}0 20 | %package -n python3-poni 21 | Summary: simple system configuration management tool (python 3) 22 | Requires: python3-argh, python3-boto, python3-dns, python3-lxml, libvirt-python3 23 | BuildRequires: python3-pylint, python3-pytest, %{requires} 24 | BuildArch: noarch 25 | 26 | %description -n python3-poni 27 | Poni is a simple system configuration management tool. 28 | 29 | This is the Python 3 package of Poni. 30 | %endif 31 | 32 | %prep 33 | %setup -q -n %{name} 34 | 35 | 36 | %build 37 | python2 setup.py build 38 | %if %{?python3_sitelib:1}0 39 | python3 setup.py build 40 | %endif 41 | 42 | 43 | %install 44 | python2 setup.py install --skip-build --root %{buildroot} 45 | mv %{buildroot}%{_bindir}/poni %{buildroot}%{_bindir}/poni-py2 46 | sed -e "s@#!/bin/python@#!%{_bindir}/python@" -i %{buildroot}%{_bindir}/poni-py2 47 | %if %{?python3_sitelib:1}0 48 | python3 setup.py install --skip-build --root %{buildroot} 49 | mv %{buildroot}%{_bindir}/poni %{buildroot}%{_bindir}/poni-py3 50 | sed -e "s@#!/bin/python@#!%{_bindir}/python@" -i %{buildroot}%{_bindir}/poni-py3 51 | %endif 52 | ln -sf poni-py2 %{buildroot}%{_bindir}/poni 53 | 54 | 55 | %check 56 | make PYTHON=python2 pylint tests 57 | %if %{?python3_sitelib:1}0 58 | make PYTHON=python3 pylint tests 59 | %endif 60 | 61 | 62 | %files 63 | %defattr(-,root,root,-) 64 | %doc README.rst LICENSE doc 65 | %{_bindir}/poni 66 | %{_bindir}/poni-py2 67 | %{python_sitelib}/* 68 | 69 | %if %{?python3_sitelib:1}0 70 | %files -n python3-poni 71 | %defattr(-,root,root,-) 72 | %doc README.rst LICENSE doc 73 | %{_bindir}/poni-py3 74 | %{python3_sitelib}/* 75 | %endif 76 | 77 | 78 | %changelog 79 | * Thu Feb 26 2015 Oskari Saarenmaa - 0.7-160 80 | - Build and package python 3 version. 81 | 82 | * Thu Feb 19 2015 Oskari Saarenmaa - 0.7-150 83 | - Refactored packaging, run tests, etc. 84 | 85 | * Mon Jan 10 2011 Oskari Saarenmaa - 0.4-0 86 | - Update to 0.4; bundle into poni proper. 87 | 88 | * Mon Dec 27 2010 Oskari Saarenmaa - 0.3.1-0 89 | - Update to 0.3.1. 90 | 91 | * Thu Dec 23 2010 Oskari Saarenmaa - 0.2-0 92 | - Initial. 93 | -------------------------------------------------------------------------------- /poni/rcontrol_openssh.py: -------------------------------------------------------------------------------- 1 | """ 2 | Node remote control using OpenSSH command-line apps 3 | 4 | Copyright (c) 2010-2012 Mika Eloranta 5 | See LICENSE for details. 6 | 7 | """ 8 | 9 | import os 10 | import subprocess 11 | from . import rcontrol 12 | 13 | 14 | class OpenSshRemoteControl(rcontrol.SshRemoteControl): 15 | """ 16 | OpenSSH remote control connection 17 | 18 | Maintains a shell connection that can be piggy-backed by further ssh or 19 | scp operations if OpenSSH is configured for shared connections, 20 | for example: 21 | 22 | .ssh/config: 23 | ---clip--- 24 | Host * 25 | ControlPath ~/.ssh/master-%l-%r@%h:%p 26 | ControlMaster auto 27 | ---clip--- 28 | 29 | """ 30 | 31 | def __init__(self, node): 32 | rcontrol.SshRemoteControl.__init__(self, node) 33 | self.node = node 34 | self._shared_conn = None 35 | 36 | def close(self): 37 | if self._shared_conn: 38 | self._shared_conn.stdin.close() 39 | self._shared_conn.stderr.close() 40 | self._shared_conn.stdout.close() 41 | 42 | def open_shared_connection(self): 43 | if not self._shared_conn: 44 | cmd = self.cmd([]) 45 | self._shared_conn = subprocess.Popen( 46 | cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE, 47 | stderr=subprocess.PIPE) 48 | 49 | def cmd(self, args): 50 | assert isinstance(args, list) 51 | full_key_path = "%s/.ssh/%s" % (os.environ["HOME"], self.key_filename) 52 | command = [ 53 | "ssh", 54 | "-i", full_key_path, 55 | "-l", self.node["user"], 56 | self.node["host"] 57 | ] 58 | command.extend(args) 59 | return command 60 | 61 | def stat(self, file_path): 62 | # TODO: implementation 63 | return None 64 | 65 | def read_file(self, file_path): 66 | self.open_shared_connection() 67 | process = subprocess.Popen(self.cmd(["cat", file_path]), 68 | stdout=subprocess.PIPE) 69 | return process.stdout.read() 70 | 71 | 72 | def write_file(self, file_path, contents, mode=None, owner=None, 73 | group=None): 74 | self.open_shared_connection() 75 | process = subprocess.Popen(self.cmd(["cat", ">", file_path]), 76 | stdin=subprocess.PIPE) 77 | process.stdin.write(contents) 78 | 79 | def execute_command(self, command, pseudo_tty=False): 80 | self.open_shared_connection() 81 | return subprocess.call(self.cmd([command])) 82 | 83 | def execute_shell(self): 84 | self.open_shared_connection() 85 | return subprocess.call(self.cmd([])) 86 | 87 | 88 | -------------------------------------------------------------------------------- /poni/times.py: -------------------------------------------------------------------------------- 1 | """ 2 | timing reports 3 | 4 | Copyright (c) 2010-2012 Mika Eloranta 5 | See LICENSE for details. 6 | 7 | """ 8 | 9 | import sys 10 | import json 11 | import datetime 12 | from . import util 13 | 14 | timediff = lambda a, b: str(datetime.timedelta(seconds=int(a - b))) 15 | 16 | class Times(object): 17 | def __init__(self): 18 | self.entry = [] 19 | 20 | def load(self, file_path): 21 | self.entry = json.load(open(file_path)) 22 | 23 | def save(self, file_path): 24 | util.json_dump(self.entry, file_path) 25 | 26 | def add_task(self, task_id, name, start, stop, args=None): 27 | self.entry.append(dict(task_id=task_id, name=name, start=start, 28 | stop=stop, args=args)) 29 | 30 | def positions(self, prop, start, stop): 31 | span = stop - start 32 | if not span: 33 | return "n/a" 34 | 35 | start_rel = (prop["start"] - start) / span 36 | stop_rel = (prop["stop"] - start) / span 37 | 38 | line_len = 70 39 | a = int(start_rel * line_len) 40 | b = int((stop_rel - start_rel) * line_len) or 1 41 | c = line_len - a - b 42 | return a, b, c 43 | 44 | def time_line(self, prop, start, stop): 45 | a, b, c = self.positions(prop, start, stop) 46 | pr = 10 47 | return "%s%s%s%s (%s)" % ( 48 | " "*pr, "-" * a, "#" * b, "-" * c, 49 | timediff(prop["stop"], prop["start"])) 50 | 51 | def pointer_line(self, prop, start, stop): 52 | a, b, c = self.positions(prop, start, stop) 53 | if b == 1: 54 | t = "" 55 | b = 0 56 | else: 57 | t = "^" 58 | b -= 2 59 | 60 | t0 = timediff(prop["start"], start) 61 | t1 = timediff(prop["stop"], start) 62 | begin = a + 10 - len(t0) - 1 63 | return "%s%s ^%s%s %s" % (" " * begin, t0, " " * b, t, t1) 64 | 65 | def print_report(self): 66 | for chunk in self.iter_report(): 67 | sys.stdout.write(chunk) 68 | 69 | sys.stdout.flush() 70 | 71 | def iter_report(self): 72 | if not self.entry: 73 | return 74 | 75 | first_start = min(e["start"] for e in self.entry) 76 | last_stop = max(e["stop"] for e in self.entry) 77 | out = [] 78 | for prop in self.entry: 79 | out.append((prop["start"], 80 | "%s: %s" % (prop["task_id"], prop["name"]), 81 | self.time_line(prop, first_start, last_stop), 82 | self.pointer_line(prop, first_start, last_stop))) 83 | 84 | out.sort() 85 | longest = max(len(e[1]) for e in out) 86 | f = "%%-%ds %%s\n%%%ds %%s\n" % (longest, longest) 87 | for start, title, tl, pl in out: 88 | yield f % (title, tl, "", pl) 89 | 90 | -------------------------------------------------------------------------------- /poni/importer.py: -------------------------------------------------------------------------------- 1 | """ 2 | debian .deb package importer 3 | 4 | Copyright (c) 2010-2012 Mika Eloranta 5 | See LICENSE for details. 6 | 7 | """ 8 | 9 | import logging 10 | import os 11 | from . import errors 12 | 13 | try: 14 | import apt_inst 15 | except ImportError: 16 | apt_inst = None 17 | 18 | 19 | class Importer(object): 20 | def __init__(self, source, verbose=False): 21 | self.log = logging.getLogger("importer") 22 | self.source = source 23 | self.verbose = verbose 24 | 25 | def import_to(self, confman): 26 | assert 0, "override in sub-class" 27 | 28 | 29 | class DebImporter(Importer): 30 | def __init__(self, source, verbose=False): 31 | Importer.__init__(self, source, verbose=verbose) 32 | if not apt_inst: 33 | raise errors.MissingLibraryError( 34 | "this feature requires the 'python-apt' library") 35 | 36 | def import_to(self, confman): 37 | try: 38 | return self.__import_to(confman) 39 | except (OSError, IOError) as error: 40 | raise errors.ImporterError("importing from '%s' failed: %s: %s" % ( 41 | self.source, error.__class__.__name__, error)) 42 | 43 | def __import_to(self, confman): 44 | prefix = "usr/lib/poni-config/" 45 | 46 | def callback(member, contents): 47 | if member.name.endswith("/") or not member.name.startswith(prefix): 48 | # not a poni-config file, skip 49 | return 50 | 51 | dest_sub = member.name[len(prefix):] 52 | dest_path = os.path.join(confman.system_root, dest_sub) 53 | dest_dir = os.path.dirname(dest_path) 54 | if not os.path.exists(dest_dir): 55 | os.makedirs(dest_dir) 56 | 57 | write = not os.path.exists(dest_path) 58 | if (not write) and os.path.exists(dest_path): 59 | old = open(dest_path).read() 60 | write = (old != contents) 61 | 62 | logger = self.log.info if self.verbose else self.log.debug 63 | pretty_path = os.path.relpath(dest_path, start=confman.root_dir) 64 | if write: 65 | open(dest_path, "wb").write(contents) 66 | logger("imported: %s", pretty_path) 67 | else: 68 | logger("unchanged: %s", pretty_path) 69 | 70 | data_tar = apt_inst.DebFile(open(self.source)).data 71 | # reads each file into memory and calls the callback, but there's no file-object based option... 72 | data_tar.go(callback) # pylint: disable=E1101 73 | 74 | 75 | def get_importer(source_path, **kwargs): 76 | if os.path.isdir(source_path): 77 | assert 0, "unimplemented" 78 | elif os.path.isfile(source_path) and source_path.endswith(".deb"): 79 | return DebImporter(source_path, **kwargs) 80 | else: 81 | raise errors.ImporterError( 82 | "don't know how to import '%s'" % source_path) 83 | -------------------------------------------------------------------------------- /doc/vc.rst: -------------------------------------------------------------------------------- 1 | Repository Version Control 2 | ========================== 3 | Poni supports basic version control operations for its repository using the Git_ version 4 | control system. 5 | 6 | Initializing Version Control 7 | ---------------------------- 8 | After you have created your Poni repository, you can start version controlling it using 9 | the ``vc init`` command:: 10 | 11 | $ poni init 12 | $ poni add-node "web/frontend{id}" -n 4 13 | $ poni vc init 14 | 15 | ``vc init`` automatically commits everything currently in the repository as the first 16 | revision:: 17 | 18 | $ git log 19 | commit ab99567c71d26c787aea5f8ceedae5ade4e89205 20 | Author: user 21 | Date: Sun Dec 5 01:41:23 2010 +0300 22 | 23 | initial commit 24 | 25 | Committing Changes 26 | ------------------ 27 | Now let's make some changes:: 28 | 29 | $ poni set frontend2 host=fe2.company.com 30 | $ poni add-node web/database 31 | 32 | The working set can be compared with the last commit using the ``vc diff`` command:: 33 | 34 | $ poni vc diff 35 | Changes: 36 | diff --git a/system/web/frontend2/node.json b/system/web/frontend2/node.json 37 | index 4a921a3..27f7588 100644 38 | --- a/system/web/frontend2/node.json 39 | +++ b/system/web/frontend2/node.json 40 | @@ -1,3 +1,3 @@ 41 | { 42 | - "host": "" 43 | + "host": "fe2.company.com" 44 | } 45 | \ No newline at end of file 46 | 47 | Untracked files: 48 | system/web/database/node.json 49 | 50 | Changes can be committed using the ``vc checkpoint `` command that automatically 51 | adds all added files and commits changed files:: 52 | 53 | $ poni vc checkpoint "added a db node and adjusted things" 54 | 55 | Now we have two revisions:: 56 | 57 | $ git log 58 | commit 6a750b460c2d13d35b83fa24e3e81060e409fe57 59 | Author: user 60 | Date: Sun Dec 5 01:52:43 2010 +0300 61 | 62 | added a db node and adjusted things 63 | 64 | commit ab99567c71d26c787aea5f8ceedae5ade4e89205 65 | Author: user 66 | Date: Sun Dec 5 01:41:23 2010 +0300 67 | 68 | initial commit 69 | 70 | The last commits contains the changes we made:: 71 | 72 | $ git show 73 | commit 6a750b460c2d13d35b83fa24e3e81060e409fe57 74 | Author: user 75 | Date: Sun Dec 5 01:52:43 2010 +0300 76 | 77 | added a db node and adjusted things 78 | 79 | diff --git a/system/web/database/node.json b/system/web/database/node.json 80 | new file mode 100644 81 | index 0000000..4a921a3 82 | --- /dev/null 83 | +++ b/system/web/database/node.json 84 | @@ -0,0 +1,3 @@ 85 | +{ 86 | + "host": "" 87 | +} 88 | \ No newline at end of file 89 | diff --git a/system/web/frontend2/node.json b/system/web/frontend2/node.json 90 | index 4a921a3..27f7588 100644 91 | --- a/system/web/frontend2/node.json 92 | +++ b/system/web/frontend2/node.json 93 | @@ -1,3 +1,3 @@ 94 | { 95 | - "host": "" 96 | + "host": "fe2.company.com" 97 | } 98 | \ No newline at end of file 99 | 100 | .. include:: definitions.rst 101 | -------------------------------------------------------------------------------- /doc/remote.rst: -------------------------------------------------------------------------------- 1 | Node Remote Control 2 | =================== 3 | Commands can be executed on remote nodes using Poni's ``remote exec`` and 4 | ``remote shell`` commands. 5 | 6 | Commands are executed over an SSH connection unless the node ``deploy`` property has been 7 | set to ``local``. In that case, the commands are simply run locally in the current host. 8 | 9 | Remote Execution of Shell Commands 10 | ---------------------------------- 11 | Having already setup our system:: 12 | 13 | $ poni list 14 | node web/database 15 | node web/frontend1 16 | node web/frontend2 17 | node web/frontend3 18 | node web/frontend4 19 | 20 | Let's run the ``last`` command on all of the frontend nodes:: 21 | 22 | $ poni remote exec frontend last 23 | reboot system boot 2.6.21.7-2.fc8xe Sat Dec 4 23:10 - 23:20 (00:09) 24 | 25 | wtmp begins Sat Dec 4 23:10:36 2010 26 | reboot system boot 2.6.21.7-2.fc8xe Sat Dec 4 23:10 - 23:20 (00:09) 27 | 28 | wtmp begins Sat Dec 4 23:10:31 2010 29 | reboot system boot 2.6.21.7-2.fc8xe Sat Dec 4 23:10 - 23:20 (00:09) 30 | 31 | wtmp begins Sat Dec 4 23:10:40 2010 32 | reboot system boot 2.6.21.7-2.fc8xe Sat Dec 4 23:10 - 23:20 (00:09) 33 | 34 | wtmp begins Sat Dec 4 23:10:45 2010 35 | 36 | All good, except we don't know which output comes from which node. The ``-v`` option 37 | helps with that:: 38 | 39 | $ poni remote exec frontend last -v 40 | --- BEGIN web/frontend1 (ec2-184-72-68-108.compute-1.amazonaws.com): exec: 'last' --- 41 | reboot system boot 2.6.21.7-2.fc8xe Sat Dec 4 23:10 - 23:23 (00:12) 42 | 43 | wtmp begins Sat Dec 4 23:10:36 2010 44 | --- END web/frontend1 (ec2-184-72-68-108.compute-1.amazonaws.com): exec: 'last' --- 45 | 46 | --- BEGIN web/frontend2 (ec2-184-72-72-65.compute-1.amazonaws.com): exec: 'last' --- 47 | reboot system boot 2.6.21.7-2.fc8xe Sat Dec 4 23:10 - 23:23 (00:12) 48 | 49 | wtmp begins Sat Dec 4 23:10:31 2010 50 | --- END web/frontend2 (ec2-184-72-72-65.compute-1.amazonaws.com): exec: 'last' --- 51 | 52 | --- BEGIN web/frontend3 (ec2-67-202-44-0.compute-1.amazonaws.com): exec: 'last' --- 53 | reboot system boot 2.6.21.7-2.fc8xe Sat Dec 4 23:10 - 23:23 (00:12) 54 | 55 | wtmp begins Sat Dec 4 23:10:40 2010 56 | --- END web/frontend3 (ec2-67-202-44-0.compute-1.amazonaws.com): exec: 'last' --- 57 | 58 | --- BEGIN web/frontend4 (ec2-50-16-50-77.compute-1.amazonaws.com): exec: 'last' --- 59 | reboot system boot 2.6.21.7-2.fc8xe Sat Dec 4 23:10 - 23:23 (00:12) 60 | 61 | wtmp begins Sat Dec 4 23:10:45 2010 62 | --- END web/frontend4 (ec2-50-16-50-77.compute-1.amazonaws.com): exec: 'last' --- 63 | 64 | The command is executed in shell, so multiple commands, piping, etc. is ok:: 65 | 66 | $ poni remote exec frontend1 "last | head -1" 67 | reboot system boot 2.6.21.7-2.fc8xe Sat Dec 4 23:10 - 23:25 (00:14) 68 | $ poni remote exec frontend1 "id; whoami; pwd" 69 | uid=0(root) gid=0(root) groups=0(root) 70 | root 71 | /root 72 | 73 | Remote Interactive Shell 74 | ------------------------ 75 | ``remote shell`` opens an interactive shell connection the the remote node:: 76 | 77 | $ poni remote shell frontend1 -v 78 | --- BEGIN web/frontend1 (ec2-184-72-68-108.compute-1.amazonaws.com): shell --- 79 | Linux ip-10-122-179-29 2.6.21.7-2.fc8xen-ec2-v1.0 #2 SMP Tue Sep 1 10:04:29 EDT 2009 i686 80 | 81 | ip-10-122-179-29:~# echo "hello, world" 82 | hello, world 83 | ip-10-122-179-29:~# exit 84 | logout 85 | --- END web/frontend1 (ec2-184-72-68-108.compute-1.amazonaws.com): shell --- 86 | 87 | 88 | .. include:: definitions.rst 89 | -------------------------------------------------------------------------------- /poni/newconfig.py: -------------------------------------------------------------------------------- 1 | """ 2 | Multi-layer settings management 3 | 4 | Copyright (c) 2010-2012 Mika Eloranta 5 | See LICENSE for details. 6 | 7 | TODO: this simple draft is VERY likely to change a lot 8 | 9 | """ 10 | 11 | import logging 12 | import os 13 | from glob import glob 14 | from . import errors 15 | from .util import json 16 | 17 | 18 | class Config(dict): 19 | def __init__(self, config_dirs): 20 | dict.__init__(self) 21 | self.log = logging.getLogger("config") 22 | self.config_dirs = list(config_dirs) 23 | self.layers = [] 24 | self.reload() 25 | 26 | def reload(self): 27 | self.clear() 28 | self.layers = [] 29 | # when combinining settings from multiple files, they are primarily 30 | # sorted by the filename prefix (first two letters) and secondarily 31 | # by the inheritance order (parent before child) 32 | for i, (layer_name, config_dir) in enumerate(self.config_dirs): 33 | if os.path.exists(config_dir): 34 | for file_path in glob(os.path.join(config_dir, "*.json")): 35 | self.layers.append(((os.path.basename(file_path)[:2], i), 36 | layer_name, file_path)) 37 | 38 | self.layers.sort() 39 | 40 | for sort_key, layer_name, file_path in self.layers: 41 | try: 42 | config_dict = json.load(open(file_path, "r")) 43 | except ValueError as error: 44 | raise errors.SettingsError("%s: %s: %s" % ( 45 | file_path, error.__class__.__name__, error)) 46 | 47 | self.log.debug("loaded %r: %r", file_path, config_dict) 48 | if not self: 49 | # base config (defaults) 50 | self.update(config_dict) 51 | else: 52 | self.apply_update(config_dict, self, file_path) 53 | 54 | def apply_update(self, update, target, file_path): 55 | self.log.debug("apply update: %r -> %r", update, target) 56 | if not isinstance(update, dict): 57 | raise errors.SettingsError("%s: expected dict, got %s (%r)" % ( 58 | file_path, type(update), update)) 59 | 60 | for key, value in update.items(): 61 | first = key[:1] 62 | if first in ["!", "+", "-"]: 63 | try: 64 | target_value = target[key[1:]] 65 | except KeyError: 66 | raise errors.SettingsError( 67 | "%s: cannot override missing setting %r" % ( 68 | file_path, key)) 69 | 70 | if first == "!": 71 | target[key[1:]] = value 72 | elif first == "+": 73 | target_value.extend(value) 74 | else: # "-" 75 | for remove_key in value: 76 | if remove_key in target_value: 77 | target_value.remove(remove_key) 78 | else: 79 | if key not in target: 80 | raise errors.SettingsError( 81 | "%s: unknown setting %r (not in default settings)" % ( 82 | file_path, key)) 83 | 84 | self.apply_update(value, target[key], file_path) 85 | 86 | 87 | class Proxy(object): 88 | def __init__(self, target): 89 | self.target = target 90 | 91 | def __getattr__(self, item): 92 | target = self.target 93 | for part in item.split("."): 94 | target = target[part] 95 | 96 | return target 97 | -------------------------------------------------------------------------------- /tests/test_templates.py: -------------------------------------------------------------------------------- 1 | from __future__ import print_function 2 | from pytest import skip 3 | from poni import template 4 | from poni import tool 5 | from helper import * 6 | import os 7 | 8 | 9 | single_xml_file_plugin_text = """ 10 | from poni import config 11 | 12 | class PlugIn(config.PlugIn): 13 | def add_actions(self): 14 | self.add_file("%(source)s", dest_path="%(dest)s", 15 | render=self.render_genshi_xml) 16 | """ 17 | 18 | genshi_xml_template = """\ 19 | 20 | 21 | 22 | """ 23 | 24 | class objectvar(object): 25 | val1 = "object variable 1" 26 | val2 = {"key": "dict in object variable 2"} 27 | 28 | g_vars = { 29 | "text": "dummy text variable", 30 | "object": objectvar, 31 | "dict": {"key": "dummy value in a dict"}, 32 | } 33 | 34 | class TestTemplates(Helper): 35 | def test_xml_template(self): 36 | if not template.genshi: 37 | skip("Genshi not available") 38 | 39 | poni, repo = self.init_repo() 40 | 41 | # add template config 42 | template_node = "tnode" 43 | template_conf = "tconf" 44 | assert not poni.run(["add-node", template_node]) 45 | assert not poni.run(["set", template_node, "verify:bool=off"]) 46 | assert not poni.run(["add-config", template_node, template_conf]) 47 | 48 | # write template config plugin.py 49 | tconf_path = "%s/%s" % (template_node, template_conf) 50 | conf_dir = os.path.join(repo, "system", template_node, "config", template_conf) 51 | tplugin_path = os.path.join(conf_dir, "plugin.py") 52 | output_file = self.temp_file() 53 | tfile = self.temp_file() 54 | with open(tfile, "w") as f: 55 | f.write(genshi_xml_template) 56 | args = dict(source=tfile, dest=output_file) 57 | with open(tplugin_path, "w") as f: 58 | f.write(single_xml_file_plugin_text % args) 59 | 60 | # add inherited config 61 | instance_node = "inode" 62 | instance_conf = "iconf" 63 | assert not poni.run(["add-node", instance_node]) 64 | assert not poni.run(["set", instance_node, "deploy=local", "host=baz"]) 65 | assert not poni.run(["add-config", instance_node, instance_conf, 66 | "--inherit", tconf_path]) 67 | 68 | # deploy and verify 69 | assert not poni.run(["deploy"]) 70 | with open(output_file, "r") as f: 71 | output = f.read() 72 | print(output) 73 | assert "baz" in output 74 | 75 | name_tests = { 76 | "foo": "foo", 77 | "foo [$text]": "foo [dummy text variable]", 78 | "foo ${dict.key}bar": "foo dummy value in a dictbar", 79 | "foo \\${dict.key}bar": "foo ${dict.key}bar", 80 | "foo \\$1": "foo $1", 81 | "foo \\\\$object.val1": "foo \\$object.val1", 82 | "foo \\$object.val1": "foo $object.val1", 83 | "foo $object.val1": "foo object variable 1", 84 | "$object.val2.key: x": "dict in object variable 2: x", 85 | } 86 | 87 | def test_render_name(self): 88 | for tmpl, exp in self.name_tests.items(): 89 | res = template.render_name(tmpl, None, g_vars) 90 | assert exp == res, \ 91 | "template {0!r}, expected {1!r}, got {2!r}".format(tmpl, exp, res) 92 | 93 | def test_render_cheetah(self): 94 | if not template.CheetahTemplate: 95 | skip("CheetahTemplate not available") 96 | for tmpl, exp in self.name_tests.items(): 97 | res = template.render_cheetah(tmpl, None, g_vars) 98 | assert exp == res, \ 99 | "template {0!r}, expected {1!r}, got {2!r}".format(tmpl, exp, res) 100 | -------------------------------------------------------------------------------- /tests/test_cloud_libvirt.py: -------------------------------------------------------------------------------- 1 | """ 2 | tests for cloud_libvirt 3 | 4 | XXX: just testing a couple of utility functions for now, implement a more 5 | complete set of tests. 6 | """ 7 | 8 | from poni import cloud_libvirt 9 | 10 | def test_parse_ip_addr(): 11 | a = list(cloud_libvirt.parse_ip_addr("")) 12 | assert a == [] 13 | 14 | a = list(cloud_libvirt.parse_ip_addr("1: foo\n2: bar")) 15 | assert a == [ 16 | {'hardware-address': None, 'ip-addresses': [], 'name': 'foo'}, 17 | {'hardware-address': None, 'ip-addresses': [], 'name': 'bar'}, 18 | ] 19 | 20 | s = """ 21 | 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default 22 | link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 23 | inet 127.0.0.1/8 scope host lo 24 | valid_lft forever preferred_lft forever 25 | inet6 ::1/128 scope host 26 | valid_lft forever preferred_lft forever 27 | 2: wwp0s20u4i6: mtu 1500 qdisc noop state DOWN group default qlen 1000 28 | link/ether 72:11:80:21:27:fb brd ff:ff:ff:ff:ff:ff 29 | 3: em1: mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000 30 | link/ether 3c:97:0e:9d:7c:f5 brd ff:ff:ff:ff:ff:ff 31 | 4: wlp3s0: mtu 1500 qdisc mq state UP group default qlen 1000 32 | link/ether 6c:88:14:65:80:80 brd ff:ff:ff:ff:ff:ff 33 | inet 192.168.50.164/24 brd 192.168.50.255 scope global dynamic wlp3s0 34 | valid_lft 83510sec preferred_lft 83510sec 35 | inet6 fe80::6e88:14ff:fe65:8080/64 scope link 36 | valid_lft forever preferred_lft forever 37 | 5: virbr0: mtu 1500 qdisc noqueue state DOWN group default 38 | link/ether 52:54:00:fb:be:ef brd ff:ff:ff:ff:ff:ff 39 | inet 192.168.232.1/24 brd 192.168.232.255 scope global virbr0 40 | valid_lft forever preferred_lft forever 41 | inet6 fe80::5054:ff:fefb:beef/64 scope link 42 | valid_lft forever preferred_lft forever 43 | 6: virbr0-nic: mtu 1500 qdisc pfifo_fast master virbr0 state DOWN group default qlen 500 44 | link/ether 52:54:00:fb:be:ef brd ff:ff:ff:ff:ff:ff 45 | """ 46 | a = list(cloud_libvirt.parse_ip_addr(s)) 47 | assert len(a) == 6 48 | assert a[0] == { 49 | "name": "lo", 50 | "hardware-address": "00:00:00:00:00:00", 51 | "ip-addresses": [ 52 | {"ip-address-type": "ipv4", "ip-address": "127.0.0.1", "prefix": 8}, 53 | {"ip-address-type": "ipv6", "ip-address": "::1", "prefix": 128}, 54 | ], 55 | } 56 | assert a[1] == { 57 | "name": "wwp0s20u4i6", 58 | "hardware-address": "72:11:80:21:27:fb", 59 | "ip-addresses": [], 60 | } 61 | assert a[2] == { 62 | "name": "em1", 63 | "hardware-address": "3c:97:0e:9d:7c:f5", 64 | "ip-addresses": [], 65 | } 66 | assert a[3] == { 67 | "name": "wlp3s0", 68 | "hardware-address": "6c:88:14:65:80:80", 69 | "ip-addresses": [ 70 | {"ip-address-type": "ipv4", "ip-address": "192.168.50.164", "prefix": 24}, 71 | {"ip-address-type": "ipv6", "ip-address": "fe80::6e88:14ff:fe65:8080", "prefix": 64}, 72 | ], 73 | } 74 | assert a[4] == { 75 | "name": "virbr0", 76 | "hardware-address": "52:54:00:fb:be:ef", 77 | "ip-addresses": [ 78 | {"ip-address-type": "ipv4", "ip-address": "192.168.232.1", "prefix": 24}, 79 | {"ip-address-type": "ipv6", "ip-address": "fe80::5054:ff:fefb:beef", "prefix": 64}, 80 | ], 81 | } 82 | assert a[5] == { 83 | "name": "virbr0-nic", 84 | "hardware-address": "52:54:00:fb:be:ef", 85 | "ip-addresses": [], 86 | } 87 | 88 | def test_mac_to_ipv6(): 89 | a = cloud_libvirt.mac_to_ipv6("fe80::", "52:54:00:fb:be:ef") 90 | assert a == "fe80::5054:ff:fefb:beef" 91 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | =========== 2 | Poni readme 3 | =========== 4 | 5 | Overview 6 | ======== 7 | Poni is a simple system configuration management tool implemented in Python_. 8 | 9 | General Information 10 | =================== 11 | :documentation: http://melor.github.com/poni/ 12 | :source repo: https://github.com/melor/poni 13 | :pypi: http://pypi.python.org/pypi/poni 14 | :email: mika dot eloranta at gmail dot com 15 | :bug tracker: https://github.com/melor/poni/issues 16 | :build status: |BuildStatus|_ 17 | 18 | .. |BuildStatus| image:: https://travis-ci.org/melor/poni.png?branch=master 19 | .. _BuildStatus: https://travis-ci.org/melor/poni 20 | 21 | Pre-requisites 22 | ============== 23 | 24 | Installing and operating Poni requires: 25 | 26 | * Python_ 2.6, 2.7 or 3.4 or greater 27 | * setuptools_ installed 28 | * Internet connection for downloading the dependency Python packages from PyPI_ 29 | 30 | .. _Python: http://www.python.org/ 31 | .. _setuptools: http://http://pypi.python.org/pypi/setuptools 32 | .. _PyPI: http://pypi.python.org/ 33 | 34 | Using Amazon EC2 requires setting the following environment variables:: 35 | 36 | export AWS_ACCESS_KEY_ID= 37 | export AWS_SECRET_ACCESS_KEY= 38 | 39 | Additionally, running the included automated tests requires: 40 | 41 | * pytest_ 42 | 43 | .. _pytest: http://pytest.org/ 44 | 45 | Building HTML files from the included ReST_ documentation requires: 46 | 47 | * docutils_ 48 | * Sphinx_ 49 | 50 | .. _ReST: http://docutils.sourceforge.net/rst.html 51 | .. _docutils: http://pypi.python.org/pypi/docutils 52 | 53 | Installation 54 | ============ 55 | NOTE: during installation the following package and its dependencies are 56 | automatically installed from PyPI_: 57 | 58 | * Argh_ (command-line argument parsing) 59 | 60 | Installing the following Python libraries will add optional functionality: 61 | 62 | * Cheetah_ (text-based templating language) 63 | * Genshi_ (XML-based templating language) 64 | * Mako_ (text-based templating language) 65 | * Paramiko_ (Remote node control using SSH) 66 | * GitPython_ (Version controlling the repository with Git) 67 | * Boto_ (`Amazon EC2`_ virtual machine provisioning) 68 | * pyvsphere_ (VMWare virtual machine provisioning) 69 | * libvirt-python_ (libvirt virtual machine provisioning) 70 | * lxml_ (libvirt provisioning dependency) 71 | * dnspython_ (libvirt provisioning dependency) 72 | * PyDNS_ (used if dnspython_ isn't available) 73 | 74 | .. _`Amazon EC2`: http://aws.amazon.com/ec2/ 75 | .. _Paramiko: http://pypi.python.org/pypi/paramiko 76 | .. _Boto: http://pypi.python.org/pypi/boto 77 | .. _Argh: http://pypi.python.org/pypi/argh 78 | .. _GitPython: http://pypi.python.org/pypi/GitPython 79 | .. _Cheetah: http://pypi.python.org/pypi/Cheetah 80 | .. _Mako: http://www.makotemplates.org/ 81 | .. _Genshi: http://pypi.python.org/pypi/Genshi 82 | .. _Sphinx: http://sphinx.pocoo.org/ 83 | .. _pyvsphere: https://github.com/F-Secure/pyvsphere 84 | .. _libvirt-python: http://libvirt.org/python.html 85 | .. _lxml: http://lxml.de/ 86 | .. _dnspython: http://www.dnspython.org/ 87 | .. _PyDNS: http://pydns.sourceforge.net/ 88 | 89 | Installation using pip or easy_install 90 | -------------------------------------- 91 | Poni can be installed from Python Package Index (PyPI) by running ``pip install poni`` or 92 | ``easy_install poni``. 93 | 94 | Manual Installation steps 95 | ------------------------- 96 | 1. Unpack the ``poni-v.vv.tar.gz`` package 97 | 2. ``cd poni-v.vv/`` 98 | 3. ``python setup.py install`` 99 | 100 | Verifying the installation 101 | -------------------------- 102 | * You should be able to ``import poni`` from Python 103 | * The ``poni`` command-line tool is installed (to a platform-specific location), 104 | try running ``poni -h`` for help 105 | * Running automated tests: ``make tests`` 106 | 107 | Usage 108 | ===== 109 | Please refer to the documentation under the ``doc/`` directory 110 | (published at http://melor.github.com/poni/) and to the example systems under the 111 | ``examples/`` directory. 112 | 113 | Credits 114 | ======= 115 | Thanks for the contributions! 116 | 117 | * Oskari Saarenmaa (features) 118 | * Santeri Paavolainen (fixes) 119 | * Lakshmi Vyas (new features for AWS-EC2 support) 120 | * Lauri Heiskanen (enabling pseudo-tty) 121 | * F-Secure Corporation (major improvements, VMWare vSphere and libvirt support) 122 | 123 | License (Apache 2.0) 124 | ==================== 125 | This package is licensed under the open-source "Apache License, Version 2.0". 126 | 127 | The full license text is available in the file ``LICENSE`` and at 128 | http://www.apache.org/licenses/LICENSE-2.0.txt 129 | 130 | **Note:** poni versions older than 0.6 were licensed under the MIT license. 131 | -------------------------------------------------------------------------------- /doc/plugins.rst: -------------------------------------------------------------------------------- 1 | ========================= 2 | Developing Config Plugins 3 | ========================= 4 | 5 | Overview 6 | ======== 7 | Each config needs to have Python plug-in file that defines the behavior of the 8 | config in different phases of the config's life-cycle. 9 | 10 | Plugins can be use to define: 11 | 12 | * Individual template configuration files, how they are rendered and where 13 | are the resulting configuration files deployed 14 | * Deployment-time file copy operations (individual files or recursively 15 | copying full directories) 16 | * Custom ``poni control`` operations that provide a convenient (relatively!) 17 | command-line interface to manipulating the config (e.g. install, start, 18 | stop, restart, verify) 19 | 20 | Hello World! 21 | ============ 22 | Here we'll create a simple plugin that illustrates using some of the basic 23 | plugin functionality. The full example can be found in the ``examples/hello`` 24 | -directory in the Poni source code package. 25 | 26 | First, we'll create a directory for the ``hello`` config and edit 27 | ``hello/plugin.py`` with your favourite file, inserting the following 28 | contents:: 29 | 30 | from poni import config 31 | 32 | class PlugIn(config.PlugIn): 33 | def add_actions(self): 34 | self.add_file("hello.txt", mode=0644, dest_path="/tmp/") 35 | 36 | Next, edit a new new file ``hello/hello.txt``, to be used as the source 37 | template, and insert the following contents:: 38 | 39 | Hello from node $node.name, config $config.name! 40 | 41 | Now we have an *amazing* config that can be added to a node and deployed. 42 | Let's create a simple system for testing it out:: 43 | 44 | $ poni -d /tmp/hello init 45 | $ poni -d /tmp/hello add-node batman 46 | $ poni -d /tmp/hello add-config -d hello/ batman hello-config 47 | $ poni -d /tmp/hello list -snc 48 | node batman 49 | config batman/hello-config 50 | $ poni -d /tmp/hello show -v 51 | --- BEGIN batman: path=/tmp/hello.txt --- 52 | Hello from node batman, config hello-config! 53 | 54 | --- END batman: path=/tmp/hello.txt --- 55 | 56 | As can be seen from the above output, the ``$node.name`` and ``$config.name`` 57 | template directives were rendered as expected. The difference between the 58 | original template and the fully-rendered one can be shown with the 59 | ``poni show --diff`` -command:: 60 | 61 | $ poni -d /tmp/hello show -v --diff 62 | --- BEGIN batman: path=/tmp/hello.txt --- 63 | --- template 64 | +++ rendered 65 | @@ -1,1 +1,1 @@ 66 | -Hello from node $node.name, config $config.name! 67 | +Hello from node batman, config hello-config! 68 | --- END batman: path=/tmp/hello.txt --- 69 | 70 | 71 | The ``add_actions()`` method is called by Poni to collect deployable files for 72 | each config. 73 | 74 | **TODO:** 75 | 76 | * ``add_file()`` arguments 77 | * ``add_dir()`` method 78 | 79 | Control Commands 80 | ================ 81 | Custom commands for controlling deployed configs can be easily added. These 82 | commands are executed by running ``poni control CONFIG-PATTERN COMMAND``, 83 | which executes the ``COMMAND`` for each config that matches the 84 | ``CONFIG-PATTERN``. 85 | 86 | Let's add a command for observing the load-averages of the node. Edit the 87 | ``hello/plugin.py``, and add the ``loadavg()`` method with just a dummy 88 | print statement for now:: 89 | 90 | from poni import config 91 | 92 | class PlugIn(config.PlugIn): 93 | def add_actions(self): 94 | self.add_file("hello.txt", mode=0644, dest_path="/tmp/") 95 | 96 | @config.control() 97 | def loadavg(self, arg): 98 | print "foo!" 99 | 100 | The ``@config.control()`` decorator defines a method as a control command. 101 | These methods are automatically collected by Poni and made available using 102 | the ``poni control`` command. 103 | 104 | Now we can update the changes file to our repository with the 105 | ``update-config`` command and view the available controls with the 106 | ``list -C`` (note: *capital* "C") command:: 107 | 108 | $ poni -d /tmp/hello update-config batman/hello-config hello/ 109 | $ poni -d /tmp/hello/ list -C 110 | config batman/hello-config 111 | controls loadavg 112 | 113 | Our loadavg command appears under the ``hello-config``, let's try running it:: 114 | 115 | $ poni -d /tmp/hello/ control batman/hello-config loadavg 116 | foo! 117 | poni INFO all [1] control tasks finished successfully 118 | 119 | **TODO:** 120 | 121 | * config match patterns (system/node/config, system//config, system//, 122 | /config) 123 | * remote execution 124 | * --verbose mode 125 | * ``poni control`` argh parsers, arguments 126 | * control command dependencies 127 | * idempotency 128 | * parallel execution (dependencies, max one concurrent job per host, --jobs=N) 129 | 130 | 131 | -------------------------------------------------------------------------------- /doc/Makefile: -------------------------------------------------------------------------------- 1 | # Makefile for Sphinx documentation 2 | # 3 | 4 | # You can set these variables from the command line. 5 | SPHINXOPTS = 6 | SPHINXBUILD = sphinx-build 7 | PAPER = 8 | BUILDDIR = _build 9 | 10 | # Internal variables. 11 | PAPEROPT_a4 = -D latex_paper_size=a4 12 | PAPEROPT_letter = -D latex_paper_size=letter 13 | ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . 14 | 15 | .PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest 16 | 17 | help: 18 | @echo "Please use \`make ' where is one of" 19 | @echo " html to make standalone HTML files" 20 | @echo " dirhtml to make HTML files named index.html in directories" 21 | @echo " singlehtml to make a single large HTML file" 22 | @echo " pickle to make pickle files" 23 | @echo " json to make JSON files" 24 | @echo " htmlhelp to make HTML files and a HTML help project" 25 | @echo " qthelp to make HTML files and a qthelp project" 26 | @echo " devhelp to make HTML files and a Devhelp project" 27 | @echo " epub to make an epub" 28 | @echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter" 29 | @echo " latexpdf to make LaTeX files and run them through pdflatex" 30 | @echo " text to make text files" 31 | @echo " man to make manual pages" 32 | @echo " changes to make an overview of all changed/added/deprecated items" 33 | @echo " linkcheck to check all external links for integrity" 34 | @echo " doctest to run all doctests embedded in the documentation (if enabled)" 35 | 36 | clean: 37 | -$(RM) -r $(BUILDDIR)/ 38 | 39 | html: 40 | $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html 41 | @echo 42 | @echo "Build finished. The HTML pages are in $(BUILDDIR)/html." 43 | 44 | dirhtml: 45 | $(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml 46 | @echo 47 | @echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml." 48 | 49 | singlehtml: 50 | $(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml 51 | @echo 52 | @echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml." 53 | 54 | pickle: 55 | $(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle 56 | @echo 57 | @echo "Build finished; now you can process the pickle files." 58 | 59 | json: 60 | $(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json 61 | @echo 62 | @echo "Build finished; now you can process the JSON files." 63 | 64 | htmlhelp: 65 | $(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp 66 | @echo 67 | @echo "Build finished; now you can run HTML Help Workshop with the" \ 68 | ".hhp project file in $(BUILDDIR)/htmlhelp." 69 | 70 | qthelp: 71 | $(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp 72 | @echo 73 | @echo "Build finished; now you can run "qcollectiongenerator" with the" \ 74 | ".qhcp project file in $(BUILDDIR)/qthelp, like this:" 75 | @echo "# qcollectiongenerator $(BUILDDIR)/qthelp/Poni.qhcp" 76 | @echo "To view the help file:" 77 | @echo "# assistant -collectionFile $(BUILDDIR)/qthelp/Poni.qhc" 78 | 79 | devhelp: 80 | $(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp 81 | @echo 82 | @echo "Build finished." 83 | @echo "To view the help file:" 84 | @echo "# mkdir -p $$HOME/.local/share/devhelp/Poni" 85 | @echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/Poni" 86 | @echo "# devhelp" 87 | 88 | epub: 89 | $(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub 90 | @echo 91 | @echo "Build finished. The epub file is in $(BUILDDIR)/epub." 92 | 93 | latex: 94 | $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex 95 | @echo 96 | @echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex." 97 | @echo "Run \`make' in that directory to run these through (pdf)latex" \ 98 | "(use \`make latexpdf' here to do that automatically)." 99 | 100 | latexpdf: 101 | $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex 102 | @echo "Running LaTeX files through pdflatex..." 103 | make -C $(BUILDDIR)/latex all-pdf 104 | @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." 105 | 106 | text: 107 | $(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text 108 | @echo 109 | @echo "Build finished. The text files are in $(BUILDDIR)/text." 110 | 111 | man: 112 | $(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man 113 | @echo 114 | @echo "Build finished. The manual pages are in $(BUILDDIR)/man." 115 | 116 | changes: 117 | $(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes 118 | @echo 119 | @echo "The overview file is in $(BUILDDIR)/changes." 120 | 121 | linkcheck: 122 | $(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck 123 | @echo 124 | @echo "Link check complete; look for any errors in the above output " \ 125 | "or in $(BUILDDIR)/linkcheck/output.txt." 126 | 127 | doctest: 128 | $(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest 129 | @echo "Testing of doctests in the sources finished, look at the " \ 130 | "results in $(BUILDDIR)/doctest/output.txt." 131 | -------------------------------------------------------------------------------- /doc/make.bat: -------------------------------------------------------------------------------- 1 | @ECHO OFF 2 | 3 | REM Command file for Sphinx documentation 4 | 5 | if "%SPHINXBUILD%" == "" ( 6 | set SPHINXBUILD=sphinx-build 7 | ) 8 | set BUILDDIR=_build 9 | set ALLSPHINXOPTS=-d %BUILDDIR%/doctrees %SPHINXOPTS% . 10 | if NOT "%PAPER%" == "" ( 11 | set ALLSPHINXOPTS=-D latex_paper_size=%PAPER% %ALLSPHINXOPTS% 12 | ) 13 | 14 | if "%1" == "" goto help 15 | 16 | if "%1" == "help" ( 17 | :help 18 | echo.Please use `make ^` where ^ is one of 19 | echo. html to make standalone HTML files 20 | echo. dirhtml to make HTML files named index.html in directories 21 | echo. singlehtml to make a single large HTML file 22 | echo. pickle to make pickle files 23 | echo. json to make JSON files 24 | echo. htmlhelp to make HTML files and a HTML help project 25 | echo. qthelp to make HTML files and a qthelp project 26 | echo. devhelp to make HTML files and a Devhelp project 27 | echo. epub to make an epub 28 | echo. latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter 29 | echo. text to make text files 30 | echo. man to make manual pages 31 | echo. changes to make an overview over all changed/added/deprecated items 32 | echo. linkcheck to check all external links for integrity 33 | echo. doctest to run all doctests embedded in the documentation if enabled 34 | goto end 35 | ) 36 | 37 | if "%1" == "clean" ( 38 | for /d %%i in (%BUILDDIR%\*) do rmdir /q /s %%i 39 | del /q /s %BUILDDIR%\* 40 | goto end 41 | ) 42 | 43 | if "%1" == "html" ( 44 | %SPHINXBUILD% -b html %ALLSPHINXOPTS% %BUILDDIR%/html 45 | if errorlevel 1 exit /b 1 46 | echo. 47 | echo.Build finished. The HTML pages are in %BUILDDIR%/html. 48 | goto end 49 | ) 50 | 51 | if "%1" == "dirhtml" ( 52 | %SPHINXBUILD% -b dirhtml %ALLSPHINXOPTS% %BUILDDIR%/dirhtml 53 | if errorlevel 1 exit /b 1 54 | echo. 55 | echo.Build finished. The HTML pages are in %BUILDDIR%/dirhtml. 56 | goto end 57 | ) 58 | 59 | if "%1" == "singlehtml" ( 60 | %SPHINXBUILD% -b singlehtml %ALLSPHINXOPTS% %BUILDDIR%/singlehtml 61 | if errorlevel 1 exit /b 1 62 | echo. 63 | echo.Build finished. The HTML pages are in %BUILDDIR%/singlehtml. 64 | goto end 65 | ) 66 | 67 | if "%1" == "pickle" ( 68 | %SPHINXBUILD% -b pickle %ALLSPHINXOPTS% %BUILDDIR%/pickle 69 | if errorlevel 1 exit /b 1 70 | echo. 71 | echo.Build finished; now you can process the pickle files. 72 | goto end 73 | ) 74 | 75 | if "%1" == "json" ( 76 | %SPHINXBUILD% -b json %ALLSPHINXOPTS% %BUILDDIR%/json 77 | if errorlevel 1 exit /b 1 78 | echo. 79 | echo.Build finished; now you can process the JSON files. 80 | goto end 81 | ) 82 | 83 | if "%1" == "htmlhelp" ( 84 | %SPHINXBUILD% -b htmlhelp %ALLSPHINXOPTS% %BUILDDIR%/htmlhelp 85 | if errorlevel 1 exit /b 1 86 | echo. 87 | echo.Build finished; now you can run HTML Help Workshop with the ^ 88 | .hhp project file in %BUILDDIR%/htmlhelp. 89 | goto end 90 | ) 91 | 92 | if "%1" == "qthelp" ( 93 | %SPHINXBUILD% -b qthelp %ALLSPHINXOPTS% %BUILDDIR%/qthelp 94 | if errorlevel 1 exit /b 1 95 | echo. 96 | echo.Build finished; now you can run "qcollectiongenerator" with the ^ 97 | .qhcp project file in %BUILDDIR%/qthelp, like this: 98 | echo.^> qcollectiongenerator %BUILDDIR%\qthelp\Poni.qhcp 99 | echo.To view the help file: 100 | echo.^> assistant -collectionFile %BUILDDIR%\qthelp\Poni.ghc 101 | goto end 102 | ) 103 | 104 | if "%1" == "devhelp" ( 105 | %SPHINXBUILD% -b devhelp %ALLSPHINXOPTS% %BUILDDIR%/devhelp 106 | if errorlevel 1 exit /b 1 107 | echo. 108 | echo.Build finished. 109 | goto end 110 | ) 111 | 112 | if "%1" == "epub" ( 113 | %SPHINXBUILD% -b epub %ALLSPHINXOPTS% %BUILDDIR%/epub 114 | if errorlevel 1 exit /b 1 115 | echo. 116 | echo.Build finished. The epub file is in %BUILDDIR%/epub. 117 | goto end 118 | ) 119 | 120 | if "%1" == "latex" ( 121 | %SPHINXBUILD% -b latex %ALLSPHINXOPTS% %BUILDDIR%/latex 122 | if errorlevel 1 exit /b 1 123 | echo. 124 | echo.Build finished; the LaTeX files are in %BUILDDIR%/latex. 125 | goto end 126 | ) 127 | 128 | if "%1" == "text" ( 129 | %SPHINXBUILD% -b text %ALLSPHINXOPTS% %BUILDDIR%/text 130 | if errorlevel 1 exit /b 1 131 | echo. 132 | echo.Build finished. The text files are in %BUILDDIR%/text. 133 | goto end 134 | ) 135 | 136 | if "%1" == "man" ( 137 | %SPHINXBUILD% -b man %ALLSPHINXOPTS% %BUILDDIR%/man 138 | if errorlevel 1 exit /b 1 139 | echo. 140 | echo.Build finished. The manual pages are in %BUILDDIR%/man. 141 | goto end 142 | ) 143 | 144 | if "%1" == "changes" ( 145 | %SPHINXBUILD% -b changes %ALLSPHINXOPTS% %BUILDDIR%/changes 146 | if errorlevel 1 exit /b 1 147 | echo. 148 | echo.The overview file is in %BUILDDIR%/changes. 149 | goto end 150 | ) 151 | 152 | if "%1" == "linkcheck" ( 153 | %SPHINXBUILD% -b linkcheck %ALLSPHINXOPTS% %BUILDDIR%/linkcheck 154 | if errorlevel 1 exit /b 1 155 | echo. 156 | echo.Link check complete; look for any errors in the above output ^ 157 | or in %BUILDDIR%/linkcheck/output.txt. 158 | goto end 159 | ) 160 | 161 | if "%1" == "doctest" ( 162 | %SPHINXBUILD% -b doctest %ALLSPHINXOPTS% %BUILDDIR%/doctest 163 | if errorlevel 1 exit /b 1 164 | echo. 165 | echo.Testing of doctests in the sources finished, look at the ^ 166 | results in %BUILDDIR%/doctest/output.txt. 167 | goto end 168 | ) 169 | 170 | :end 171 | -------------------------------------------------------------------------------- /poni/cloudbase.py: -------------------------------------------------------------------------------- 1 | """ 2 | Cloud-provider: base-classes 3 | 4 | Copyright (c) 2010-2012 Mika Eloranta 5 | See LICENSE for details. 6 | 7 | """ 8 | from . import errors 9 | 10 | 11 | class NoProviderMethod(NotImplementedError): 12 | def __init__(self, obj, func): 13 | name = (obj if isinstance(obj, type) else obj.__class__).__name__ 14 | NotImplementedError.__init__(self, "{0} does not implement {1}".format(name, func)) 15 | 16 | 17 | class Provider(object): 18 | """Abstract base-class for cloud-specific cloud provider logic""" 19 | def __init__(self, provider_id, cloud_prop): 20 | self.provider_id = provider_id 21 | self._provider_key = self.get_provider_key(cloud_prop) 22 | 23 | def __eq__(self, other): 24 | if not other or not isinstance(other, Provider): 25 | return False 26 | return self._provider_key == other._provider_key 27 | 28 | def __ne__(self, other): 29 | if not other or not isinstance(other, Provider): 30 | return True 31 | return self._provider_key != other._provider_key 32 | 33 | def __hash__(self): 34 | # shuffle the hash a bit to create unique hashes for Provider objects 35 | # and their provider_keys 36 | return hash(("cloudbase.Provider", self._provider_key)) 37 | 38 | def required_prop(self, cloud_prop, prop_name): 39 | value = cloud_prop.get(prop_name) 40 | if value is None: 41 | raise errors.CloudError("'cloud.{0}' property required by {1} not defined".format( 42 | prop_name, self.provider_id)) 43 | return value 44 | 45 | @classmethod 46 | def get_provider_key(cls, cloud_prop): 47 | """ 48 | Return the cloud provider key for the given cloud properties. 49 | 50 | A unique provider key can be returned based on, for example, the 51 | region of the specified data-center. 52 | 53 | Returns a minimum unique key value needed to uniquely describe the 54 | cloud Provider. Can be e.g. (provider_type, data_center_id), like 55 | with AWS-EC2. The return value also needs to be hash()able. 56 | """ 57 | raise NoProviderMethod(cls, "get_provider_key") 58 | 59 | def init_instance(self, cloud_prop): 60 | """ 61 | Create a new instance with the given properties. 62 | 63 | Returns node properties that are changed. 64 | """ 65 | raise NoProviderMethod(self, "init_instance") 66 | 67 | def assign_ip(self, props): 68 | """ 69 | Assign the ip's to the instances based on the given properties. 70 | """ 71 | raise NoProviderMethod(self, "assign_ip") 72 | 73 | def get_instance_status(self, prop): 74 | """ 75 | Return instance status string for the instance specified in the given 76 | cloud properties dict. 77 | """ 78 | raise NoProviderMethod(self, "get_instance_status") 79 | 80 | def terminate_instances(self, props): 81 | """ 82 | Terminate instances specified in the given sequence of cloud 83 | properties dicts. 84 | """ 85 | raise NoProviderMethod(self, "terminate_instances") 86 | 87 | def wait_instances(self, props, wait_state="running"): 88 | """ 89 | Wait for all the given instances to reach status specified by 90 | the 'wait_state' argument. 91 | 92 | Returns a dict {instance_id: dict()} 93 | """ 94 | raise NoProviderMethod(self, "wait_instances") 95 | 96 | def create_snapshot(self, props, name=None, description=None, memory=False): 97 | """ 98 | Create a new snapshot for the given instances with the specified props. 99 | 100 | Returns a dict {instance_id: dict()} 101 | """ 102 | raise NoProviderMethod(self, "create_snapshot") 103 | 104 | def revert_to_snapshot(self, props, name=None): 105 | """ 106 | Revert the given instances to the specified snapshot. 107 | 108 | Returns a dict {instance_id: dict()} 109 | """ 110 | raise NoProviderMethod(self, "revert_to_snapshot") 111 | 112 | def remove_snapshot(self, props, name): 113 | """ 114 | Remove the specified snapshot on the given instances. 115 | 116 | Returns a dict {instance_id: dict()} 117 | """ 118 | raise NoProviderMethod(self, "remove_snapshot") 119 | 120 | def power_off_instances(self, props): 121 | """ 122 | Power off the given instances. 123 | 124 | Returns a dict {instance_id: dict()} 125 | """ 126 | raise NoProviderMethod(self, "power_off_instances") 127 | 128 | def power_on_instances(self, props): 129 | """ 130 | Power on the given instances. 131 | 132 | Returns a dict {instance_id: dict()} 133 | """ 134 | raise NoProviderMethod(self, "power_on_instances") 135 | 136 | def find_instances(self, match_function): 137 | """ 138 | Look up instances which have a name matching match_function. 139 | 140 | Returns a list [{vm_name: "vm_name", ...}, ...] 141 | """ 142 | raise NoProviderMethod(self, "find_instances") 143 | -------------------------------------------------------------------------------- /poni/cloud_docker.py: -------------------------------------------------------------------------------- 1 | """ 2 | Cloud-provider implementation: Docker 3 | 4 | Copyright (c) 2014 F-Secure Corporation 5 | See LICENSE for details. 6 | 7 | """ 8 | from . import cloudbase 9 | from . import errors 10 | import copy 11 | import logging 12 | import os 13 | import time 14 | 15 | try: 16 | import docker 17 | import docker.errors 18 | except ImportError: 19 | docker = None 20 | 21 | 22 | def convert_docker_errors(method): 23 | """Convert docker errors to errors.CloudError""" 24 | def wrapper(self, *args, **kw): 25 | try: 26 | return method(self, *args, **kw) 27 | except docker.errors.APIError as error: 28 | raise errors.CloudError("%s: %s" % (error.__class__.__name__, error)) 29 | 30 | wrapper.__doc__ = method.__doc__ 31 | wrapper.__name__ = method.__name__ 32 | 33 | return wrapper 34 | 35 | 36 | class DockerProvider(cloudbase.Provider): 37 | def __init__(self, cloud_prop): 38 | assert docker, "docker-py is not installed, cannot access docker" 39 | cloudbase.Provider.__init__(self, 'docker', cloud_prop) 40 | self.log = logging.getLogger('poni.docker') 41 | self.base_url = cloud_prop.get("base_url") or os.environ.get('DOCKER_BASE_URL', 'unix://var/run/docker.sock') 42 | self._conn = None 43 | 44 | @classmethod 45 | def get_provider_key(cls, cloud_prop): 46 | """ 47 | Return a cloud Provider object for the given cloud properties. 48 | """ 49 | return ("docker", cloud_prop.get("base_url")) 50 | 51 | def _get_conn(self, cloud_prop): 52 | if not self._conn: 53 | # NOTE: version 1.10 is required for 'dns' setting to work 54 | self._conn = docker.Client(base_url=self.base_url, version="1.10", timeout=10) 55 | 56 | return self._conn 57 | 58 | def _find_container(self, cloud_prop): 59 | conn = self._get_conn(cloud_prop) 60 | c_name = "/" + cloud_prop["vm_name"] 61 | for prop in conn.containers(all=True): 62 | if c_name in (prop.get("Names") or []): 63 | return prop["Id"] 64 | return None 65 | 66 | @convert_docker_errors 67 | def init_instance(self, cloud_prop): 68 | """ 69 | Create a new instance with the given properties. 70 | 71 | Returns node properties that are changed. 72 | """ 73 | vm_name = self.required_prop(cloud_prop, "vm_name") 74 | image = self.required_prop(cloud_prop, "image") 75 | conn = self._get_conn(cloud_prop) 76 | container_id = self._find_container(cloud_prop) 77 | binds = cloud_prop.get("binds") 78 | dns = cloud_prop.get("dns") 79 | if not container_id: 80 | prop = conn.create_container( 81 | image, hostname=vm_name, name=vm_name, 82 | volumes=binds.keys() if binds else None, 83 | environment=dict(container="docker")) 84 | container_id = prop['Id'] 85 | 86 | conn.start(container_id, privileged=cloud_prop.get("privileged", False), 87 | binds=binds, dns=dns) 88 | return self._updated_prop(cloud_prop, container_id) 89 | 90 | def _updated_prop(self, cloud_prop, container_id=None): 91 | conn = self._get_conn(cloud_prop) 92 | container_id = container_id or cloud_prop["instance"] 93 | cont_prop = conn.inspect_container(container_id) 94 | ip = cont_prop["NetworkSettings"]["IPAddress"] 95 | out_prop = dict(cloud=copy.deepcopy(cloud_prop), 96 | host=ip, private=dict(ip=ip, dns=ip)) 97 | out_prop["cloud"]["instance"] = container_id 98 | return out_prop 99 | 100 | @convert_docker_errors 101 | def get_instance_status(self, prop): 102 | """ 103 | Return instance status string for the instance specified in the given 104 | cloud properties dict. 105 | """ 106 | conn = self._get_conn(prop) 107 | cont_prop = conn.inspect_container(prop["instance"]) 108 | if cont_prop["State"]["Paused"]: 109 | return "paused" 110 | elif cont_prop["State"]["Running"]: 111 | return "running" 112 | else: 113 | return "stopped" 114 | 115 | @convert_docker_errors 116 | def terminate_instances(self, props): 117 | """ 118 | Terminate instances specified in the given sequence of cloud 119 | properties dicts. 120 | """ 121 | for prop in props: 122 | conn = self._get_conn(prop) 123 | conn.kill(prop["instance"]) 124 | conn.remove_container(prop["instance"]) 125 | 126 | @convert_docker_errors 127 | def wait_instances(self, props, wait_state="running"): 128 | """ 129 | Wait for all the given instances to reach status specified by 130 | the 'wait_state' argument. 131 | 132 | Returns a dict {instance_id: dict()} 133 | """ 134 | out = {} 135 | for prop in props: 136 | conn = self._get_conn(prop) 137 | cont_prop = conn.inspect_container(prop["instance"]) 138 | if wait_state == "running" and cont_prop["State"]["Running"]: 139 | out[prop["instance"]] = self._updated_prop(prop) 140 | continue 141 | 142 | time.sleep(1.0) 143 | 144 | return out 145 | -------------------------------------------------------------------------------- /doc/overview.rst: -------------------------------------------------------------------------------- 1 | Overview 2 | ======== 3 | Poni is a systems management tool for defining, deploying and verifying complex 4 | multi-node computer systems. 5 | 6 | Poni helps managing systems in many ways: 7 | 8 | * Systems, nodes, installed software and settings are stored in a central Poni 9 | repository, so there is a single location where the system is defined and 10 | documented. 11 | * Changes to the Poni repository can be version-controlled with Git_ allowing 12 | rollback and access to history of configuration changes. 13 | * Applications are configured by rendering configuration templates, using a 14 | powerful template language (such as Cheetah_) reducing the need to write 15 | application-specific installation or configuration scripts. 16 | * The entire system configuration is available when creating node-specific 17 | configuration files, making it easy to wire multiple nodes together. 18 | * Virtual machines can be easily provisioned from a cloud provider (currently 19 | `Amazon EC2`_ is supported). 20 | 21 | Challenges 22 | ---------- 23 | These are some of the challenges that Poni attemps to address: 24 | 25 | * Full automation (read: scriptability) support. 26 | * Deployment so easy and repeatable that all developers, testers, admins and 27 | support engineers are able to rollout their own playground environments. 28 | * Repeatable multi-tier software deployment. 29 | * Configuring N frontend nodes to talk to M backend nodes. What happens when 30 | you add or remove a few nodes? 31 | * Auditable configuration management over as much as possible of the deployed 32 | software stack. 33 | * Integrating with multiple hypervisor (or "cloud") providers. 34 | * Optimizing the total time spent deploying a build from scratch for 35 | development and testing purposes. 36 | * Traceability for configuration changes. 37 | * Complexity of deploying different management systems. 38 | * License cost and poor feature match of proprietary hypervisor 39 | implementations for fast-paced development activities. 40 | * Managing configurations of any existing 3rd party and open-source software 41 | included in the solution stack. 42 | * Common interface for configuring settings for all of the components in the 43 | stack. 44 | * Producing reliable, up-to-date information regarding network addresses and 45 | the different network connections between nodes. 46 | * Deep integration with the software stack that is deployed. 47 | * Zero-downtime upgrades for a multi-tier, redundant system. 48 | * Dynamic deployment for nodes and different features: must be able to leave 49 | out sub-systems and large features and still be able to deploy a whole, 50 | functional environment. 51 | * Post-deployment administrative operations: starting/stopping components, 52 | online/offline nodes, checking node/component status, etc. 53 | * Deployment-time dependencies (e.g. DB backends are deployed before DB 54 | frontends, package repository is deployed before nodes that require packages 55 | from it). 56 | 57 | Solutions 58 | --------- 59 | * Built for automation: all commands can be run from scripts, parameterized 60 | and with proper exit codes. 61 | * Provides a holistic view to the entire system, all the nodes and their 62 | settings for configration file templates. 63 | * Provides a ``poni audit`` command for verifying all deployed files. Can also 64 | diff the "unauthorized" changes. 65 | * Possible to support multiple hypervisors. 66 | * Deployment is fast enough to be done from scratch for most purposes. Base 67 | VM images do not contain any software, which helps reducing manual CM effort. 68 | * The entire system configuration is stored in a single directory tree that 69 | is version controlled using Git. Full history of changes is visible as 70 | commits. 71 | * Dynamic information collection from templates: pieces of information can be 72 | collected and reports produced out of them. Allows drawing network diagrams, 73 | defining firewall rules, etc. 74 | * Provides multiple ways of controlling the deployed software post-deployment: 75 | custom "control commands" which are written in Python or simply by running 76 | shell commands over a specific set of nodes. 77 | * Custom control commands can have dependencies: useful for installation 78 | commands that need to be executed in a certain order. 79 | 80 | How Poni is Used 81 | ---------------- 82 | #. Bundle software-specific configuration file templates and installation 83 | scripts into "Poni configs". Typically one config represent one software 84 | component (for example DB, monitoring agent, HTTP server, etc.) Custom 85 | remote commands (e.g. for executing the installation script) are also 86 | defined in this step. 87 | #. Create node templates for the different node types. Configure each node 88 | type to include one or more Poni configs. 89 | #. Instantiate the node templates into node instances. Multiple nodes can 90 | be created from a single node template. (``poni create-node``) 91 | #. (Optional) Automatically provision the nodes from a private or public 92 | cloud provider. Can also use pre-created HW-based nodes or pre-deployed 93 | VMs. (``poni cloud init``) 94 | #. Deploy the templates, software packages and installation scripts to the 95 | hosts. (``poni deploy``) 96 | #. Run the installation scripts. (``poni control NODES COMMAND``) 97 | #. DONE, system is up and running. 98 | 99 | The node creation, VM provisioning and software deployment steps are typically 100 | executed from a single script in order to provide an easy method of deploying 101 | from scratch. 102 | 103 | -------------------------------------------------------------------------------- /poni/template.py: -------------------------------------------------------------------------------- 1 | """ 2 | template rendering 3 | 4 | Copyright (c) 2010-2012 Mika Eloranta 5 | Copyright (c) 2013 Oskari Saarenmaa 6 | See LICENSE for details. 7 | 8 | """ 9 | 10 | from . import errors 11 | from io import StringIO 12 | import re 13 | import sys 14 | 15 | 16 | if sys.version_info[0] == 2: 17 | string_types = basestring # pylint: disable=E0602 18 | else: 19 | string_types = str 20 | 21 | 22 | try: 23 | # https://github.com/cheetahtemplate/cheetah/commit/bbca0d9e1db4710b523271399b3fae89d9993eb7 24 | from os.path import splitdrive 25 | import Cheetah.convertTmplPathToModuleName 26 | _unitrans = unicode(Cheetah.convertTmplPathToModuleName._pathNameTransChars) # pylint: disable=E0602 27 | def _patched_convertTmplPathToModuleName(tmplPath): 28 | try: 29 | return splitdrive(tmplPath)[1].translate(_unitrans) 30 | except (UnicodeError, TypeError): 31 | return unicode(splitdrive(tmplPath)[1]).translate(_unitrans) # pylint: disable=E0602 32 | Cheetah.convertTmplPathToModuleName.convertTmplPathToModuleName = _patched_convertTmplPathToModuleName 33 | 34 | import Cheetah.Template 35 | from Cheetah.Template import Template as CheetahTemplate 36 | 37 | import random 38 | 39 | # https://github.com/cheetahtemplate/cheetah/pull/2 40 | def _patched_genUniqueModuleName(baseModuleName): 41 | """ 42 | Workaround the problem that Cheetah creates conflicting module names due to 43 | a poor module generator function. Monkey-patch the module with a workaround. 44 | 45 | Fixes failures that look like this: 46 | 47 | File "cheetah_DynamicallyCompiledCheetahTemplate_1336479589_95_84044.py", line 58, in _init_ 48 | TypeError: super() argument 1 must be type, not None 49 | """ 50 | if baseModuleName not in sys.modules: 51 | return baseModuleName 52 | else: 53 | return 'cheetah_%s_%x' % (baseModuleName, random.getrandbits(128)) 54 | Cheetah.Template._genUniqueModuleName = _patched_genUniqueModuleName 55 | except ImportError: 56 | CheetahTemplate = None 57 | 58 | try: 59 | from mako.template import Template as MakoTemplate 60 | from mako.exceptions import MakoException 61 | except ImportError: 62 | MakoTemplate = None 63 | 64 | try: 65 | import genshi 66 | import genshi.template 67 | except ImportError: 68 | genshi = None 69 | 70 | 71 | _name_re = re.compile(r"(\\?\$(?:\{.+?\}|[._a-zA-Z0-9]+))") 72 | 73 | 74 | def render_name(source_text, source_path, variables): 75 | """simplified filename rendering with dollar-variable substitution only""" 76 | if source_path: 77 | source_text = open(source_path).read() 78 | 79 | def sub(match): 80 | token = match.group(1) 81 | if token[0] == '\\': 82 | return token[1:] # strip escape 83 | if token[1] == '{' and token[-1] == '}': 84 | token = token[2:-1] # strip ${} 85 | else: 86 | token = token[1:] # strip $ 87 | node = variables 88 | tpath, _, targs = token.partition("(") 89 | for part in tpath.split("."): 90 | if isinstance(node, dict) and part in node: 91 | node = node[part] 92 | else: 93 | node = getattr(node, part) 94 | if callable(node): 95 | node = node(*eval("(" + targs)) if targs else node() # pylint: disable=W0123 96 | if not isinstance(node, string_types): 97 | node = str(node) 98 | return node 99 | return _name_re.sub(sub, source_text) 100 | 101 | 102 | def render_cheetah(source_text, source_path, variables): 103 | assert CheetahTemplate, "Cheetah is not installed" 104 | try: 105 | return str(CheetahTemplate(source=source_text, file=source_path, searchList=[variables])) 106 | except (Cheetah.Template.Error, SyntaxError, Cheetah.NameMapper.NotFound) as error: 107 | raise errors.TemplateError("{0}: {1}: {2}".format(source_path, error.__class__.__name__, error)) 108 | 109 | 110 | def render_mako(source_text, source_path, variables): 111 | assert MakoTemplate, "Mako is not installed" 112 | try: 113 | return MakoTemplate(text=source_text, filename=source_path).render(**variables) 114 | except MakoException as error: 115 | raise errors.TemplateError("{0}: {1}: {2}".format(source_path, error.__class__.__name__, error)) 116 | 117 | 118 | def render_genshi(source_text, source_path, variables): 119 | assert genshi, "Genshi is not installed" 120 | if source_path: 121 | source = open(source_path) 122 | else: 123 | source = StringIO(source_text) 124 | try: 125 | tmpl = genshi.template.MarkupTemplate(source, filepath=source_path) 126 | stream = tmpl.generate(**variables) 127 | return stream.render('xml') 128 | except (genshi.template.TemplateError, IOError) as error: 129 | raise errors.TemplateError("{0}: {1}: {2}".format(source_path, error.__class__.__name__, error)) 130 | 131 | 132 | def render(engine=None, source_text=None, source_path=None, variables=None): 133 | if engine in ("name", "poni"): 134 | return render_name(source_text, source_path, variables) 135 | elif engine == "cheetah": 136 | return render_cheetah(source_text, source_path, variables) 137 | elif engine in ("genshi", "xml"): 138 | return render_genshi(source_text, source_path, variables) 139 | elif engine == "mako": 140 | return render_mako(source_text, source_path, variables) 141 | else: 142 | raise errors.TemplateError("unknown rendering engine {0!r}".format(engine)) 143 | -------------------------------------------------------------------------------- /doc/template-variables.rst: -------------------------------------------------------------------------------- 1 | ================================ 2 | Template Variables and Functions 3 | ================================ 4 | 5 | Variables 6 | --------- 7 | The following variables are accessible from templates: 8 | 9 | .. list-table:: 10 | :widths: 20 30 20 30 11 | :header-rows: 1 12 | 13 | * - Variable 14 | - Description 15 | - Data Type 16 | - Example Usage 17 | * - ``node`` 18 | - Node properties 19 | - ``poni.core.Node`` 20 | - ``$node.private.dns`` 21 | * - ``settings`` 22 | - Config settings (with all settings layers applied), shortcut to ``$config.settings`` 23 | - dict 24 | - ``$settings.my_server.http_port`` 25 | * - ``s`` 26 | - **deprecated** Use ``settings`` instead 27 | - 28 | - 29 | * - ``system`` 30 | - System properties of the current node's system 31 | - ``poni.core.System`` 32 | - ``$system.sub_count`` 33 | * - ``config`` 34 | - Config properties 35 | - ``poni.core.Config`` 36 | - ``$config.name``, ``$config.settings`` 37 | * - ``plugin`` 38 | - Current config's plug-in object. Allows e.g. calling custom methods 39 | defined in your ``plugin.py`` file. 40 | - ``poni.core.PlugIn`` 41 | - ``$plugin.my_custom_method($node, "hello")`` 42 | 43 | Functions 44 | --------- 45 | The following functions are accessible from templates: 46 | 47 | .. function:: find(pattern, nodes=True, systems=False, full_match=False) 48 | 49 | Find nodes and/or systems matching the given search pattern. 50 | 51 | :param pattern: Regular expression node path pattern 52 | :param nodes: Returns nodes if True 53 | :param systems: Returns systems if True 54 | :param full_match: Require full regexp match instead of default sub-string 55 | match 56 | :rtype: generator object returning Node objects 57 | 58 | Example usage:: 59 | 60 | #for $node in $find("webshop/frontend") 61 | $node.name has address $node.host 62 | #end for 63 | 64 | .. function:: find_config(pattern, all_configs=False, full_match=False) 65 | 66 | Find configs matching given pattern. 67 | 68 | A double-backslash combination in the pattern signifies "anything", i.e. 69 | ``//foo`` will find all ``foo`` configs in the entire system and 70 | ``bar//baz`` will find all ``baz`` configs under the ``bar`` system. 71 | 72 | :param pattern: Regular expression config path pattern. 73 | :param all_configs: Returns also inherited configs if True 74 | :param full_match: Require full regexp match instead of default sub-string 75 | match 76 | :rtype: generator object returning ``(node, config)`` pairs 77 | 78 | Example usage:: 79 | 80 | # find all "http-server" configs under the entire "webshop" system: 81 | #for $node, $conf in $find_config("webshop//http-server", all_configs=True) 82 | $node.name has $conf.name at port $conf.settings.server_port 83 | #end for 84 | 85 | .. function:: get_node(pattern) 86 | 87 | Return exactly one node that matches the pattern. An error is raise if 88 | zero or more than one nodes match the pattern. 89 | 90 | :param pattern: Regular expression node path pattern. 91 | :rtype: Node object 92 | 93 | .. function:: get_system(pattern) 94 | 95 | Return exactly one system that matches the pattern. An error is raise if 96 | zero or more than one systems match the pattern. 97 | 98 | :param pattern: Regular expression system path pattern. 99 | :rtype: System object 100 | 101 | .. function:: get_config(pattern) 102 | 103 | Return exactly one node and one config that matches the pattern. An error 104 | is raise if zero or more than one configs match the pattern. 105 | 106 | :param pattern: Regular expression system/node/config path pattern. 107 | :rtype: a single tuple of (Node, Config) 108 | 109 | .. function:: edge(bucket_name, dest_node, dest_config, **kwargs) 110 | 111 | Add a directed graph edge as a ``dict`` object into a bucket. This can be 112 | used to, for example, automatically collect information about network 113 | connections between nodes. 114 | 115 | :param bucket_name: Bucket name 116 | :type bucket_name: string 117 | :param dest_node: Edge destination node 118 | :type dest_node: ``poni.core.Node`` 119 | :param dest_config: Edge destination config 120 | :type dest_config: ``poni.core.Config`` 121 | :param kwargs: Extra information to store in the ``dict`` object 122 | 123 | Example usage:: 124 | 125 | #for $db_node, $db_config in $find_config("webshop//pg84") 126 | $edge("tcp", $db_node, $db_config, protocol="sql", port=$db_config.settings.db_server_port)#slurp 127 | #end for 128 | 129 | .. function:: bucket(bucket_name) 130 | 131 | Return a bucket object for accessing dynamically collected data during 132 | the template rendering process. 133 | 134 | :param bucket_name: Bucket name 135 | :type bucket_name: string 136 | :rtype: ``list`` object 137 | 138 | **NOTE:** Accessing buckets from templates should be done only after all 139 | other templates are rendered so that all dynamic data is collected. This 140 | can be achieved by giving the extra ``report=True`` argument to the 141 | ``poni.core.PlugIn`` ``add_file()`` call. 142 | 143 | Example usage:: 144 | 145 | #for $item in $bucket("tcp") 146 | Node $item.source_node.name config $item.source_config.name connects to: 147 | $item.dest_node:$item.port for some $item.protocol action... 148 | #end for 149 | 150 | Registering the template to be processed after all regular templates:: 151 | 152 | class PlugIn(config.PlugIn): 153 | def add_actions(self): 154 | self.add_file("node-report.txt", dest_path="/tmp/", report=True) 155 | -------------------------------------------------------------------------------- /poni/util.py: -------------------------------------------------------------------------------- 1 | """ 2 | Generic utility functions and classes 3 | 4 | Copyright (c) 2010-2012 Mika Eloranta 5 | See LICENSE for details. 6 | 7 | """ 8 | 9 | import logging 10 | import os 11 | from multiprocessing.pool import ThreadPool 12 | from . import errors 13 | from . import recode 14 | 15 | try: 16 | import json 17 | except ImportError: 18 | import simplejson as json 19 | 20 | 21 | DEF_VALUE = object() # used as default value where None cannot be used 22 | 23 | 24 | # TODO: refactor and write tests for get_dict_prop/set_dict_prop 25 | def get_dict_prop(item, address, verify=False): 26 | error = False 27 | for part in address[:-1]: 28 | if not isinstance(item, dict): 29 | error = True 30 | break 31 | 32 | old = item.get(part, DEF_VALUE) 33 | if old is DEF_VALUE: 34 | if verify: 35 | error = True 36 | break 37 | 38 | item = item.setdefault(part, {}) 39 | else: 40 | item = old 41 | 42 | if error or (not isinstance(item, dict)): 43 | raise errors.InvalidProperty( 44 | "%r does not exist" % (".".join(address))) 45 | 46 | old = item.get(address[-1], DEF_VALUE) 47 | 48 | return item, old 49 | 50 | 51 | def set_dict_prop(item, address, value, verify=False, schema=None): 52 | item, old = get_dict_prop(item, address, verify=verify) 53 | if verify and (old is DEF_VALUE) and (schema is not None): 54 | schema_item, old = get_dict_prop(schema, address, verify=verify) 55 | return set_dict_prop(item, address, value, verify=False) 56 | 57 | if verify: 58 | if old is DEF_VALUE: 59 | raise errors.InvalidProperty( 60 | "%r does not exist" % (".".join(address))) 61 | elif type(value) != type(old): # pylint: disable=W1504 62 | raise errors.InvalidProperty("%r type is %r, got %r: %r" % ( 63 | ".".join(address), type(old).__name__, 64 | type(value).__name__, value)) 65 | else: 66 | if old is DEF_VALUE: 67 | old = None 68 | 69 | item[address[-1]] = value 70 | 71 | return old 72 | 73 | 74 | def json_dump(data, file_path): 75 | """safe json dump to file, writes to temp file first""" 76 | temp_path = "%s.json_dump.tmp" % file_path 77 | with open(temp_path, "w") as out: 78 | json.dump(data, out, indent=4, sort_keys=True) 79 | 80 | os.rename(temp_path, file_path) 81 | 82 | 83 | def parse_prop(prop_str, converters=None): 84 | """ 85 | parse and return (keyname, value) from input 'prop_str' 86 | 87 | 'prop_str' may contain converters, for example: 88 | 89 | 'foo=hello' => ('foo', 'hello') 90 | 'bar:int=123' => ('bar', 123) 91 | """ 92 | val_parts = prop_str.split("=", 1) 93 | if len(val_parts) == 1: 94 | # no value specified 95 | name = prop_str 96 | value = None 97 | else: 98 | name, value = val_parts 99 | 100 | parts = name.split(":", 1) 101 | try: 102 | if len(parts) > 1: 103 | name, enc_str = parts 104 | codec = recode.Codec(enc_str, default=recode.ENCODE, 105 | converters=converters) 106 | else: 107 | codec = recode.Codec("-ascii") 108 | 109 | out = name, codec.process(value) 110 | except (ValueError, recode.Error) as error: 111 | raise errors.InvalidProperty("%s: %s" % (error.__class__.__name__, 112 | error)) 113 | 114 | return out 115 | 116 | 117 | def parse_count(count_str): 118 | """parse and return integers (start, end) from input 'N' or 'N..M'""" 119 | ranges = count_str.split("..") 120 | try: 121 | if len(ranges) == 1: 122 | return 1, (int(count_str) + 1) 123 | elif len(ranges) == 2: 124 | return int(ranges[0]), (int(ranges[1]) + 1) 125 | except ValueError: 126 | pass 127 | 128 | raise errors.InvalidRange("invalid range: %r" % (count_str,)) 129 | 130 | 131 | def format_error(error): 132 | return "ERROR: %s: %s" % (error.__class__.__name__, error) 133 | 134 | 135 | def dir_stats(dir_path): 136 | """return a statistics dict about a directory and its contents""" 137 | out = {"path": dir_path, "file_count": 0, "total_bytes": 0} 138 | for dir_name, dir_entries, file_entries in os.walk(dir_path): 139 | for file_name in file_entries: 140 | out["file_count"] += 1 141 | out["total_bytes"] += os.path.getsize(os.path.join(dir_name, file_name)) 142 | 143 | return out 144 | 145 | 146 | def path_iter_dict(dict_obj, prefix=None): 147 | """ 148 | yield (path, value) for each item in a dict possibly containing other dicts 149 | 150 | 'path' is in format 'key1.key2.valuename' 151 | """ 152 | prefix = prefix or [] 153 | for key, value in sorted(dict_obj.items()): 154 | location = prefix + [key] 155 | if isinstance(value, dict): 156 | for item in path_iter_dict(value, prefix=location): 157 | yield item 158 | else: 159 | yield ".".join(location), value 160 | 161 | 162 | def hash_any(ob): 163 | if isinstance(ob, (list, set, tuple)): 164 | ob = tuple(hash_any(v) for v in ob) 165 | elif isinstance(ob, dict): 166 | ob = tuple((hash_any(k), hash_any(v)) for k, v in ob.items()) 167 | return hash(ob) 168 | 169 | 170 | class hashed_dict(dict): 171 | def __hash__(self): 172 | return hash_any(self) 173 | 174 | 175 | class PropDict(dict): 176 | def __getattr__(self, name): 177 | return self.get(name, None) 178 | 179 | 180 | class TaskPool(ThreadPool): 181 | def __init__(self, task_count=10): 182 | ThreadPool.__init__(self, task_count) 183 | self.log = logging.getLogger("taskpool") 184 | self.applied = 0 185 | 186 | def __reduce__(self): 187 | pass 188 | 189 | def _call_wrapper(self, method, method_args, method_kwargs=None): 190 | try: 191 | return method(*method_args, **(method_kwargs or {})) 192 | except errors.Error as error: 193 | self.log.fatal("task error: {0.__class__.__name__}: {0}".format(error)) # pylint: disable=W1202,E1306 194 | raise 195 | except Exception as error: 196 | self.log.exception("task error: {0.__class__.__name__}: {0}".format(error)) # pylint: disable=W1202,E1306 197 | raise 198 | 199 | def apply_async(self, method, args=(), kwds=None, callback=None): # pylint: disable=W0221 200 | kwds = kwds or {} 201 | ThreadPool.apply_async(self, self._call_wrapper, [method, args], kwds, callback) 202 | self.applied += 1 203 | 204 | def wait_all(self): 205 | self.close() 206 | self.join() 207 | -------------------------------------------------------------------------------- /poni/recode.py: -------------------------------------------------------------------------------- 1 | """ 2 | Data type conversions 3 | 4 | Copyright (c) 2010-2012 Mika Eloranta 5 | See LICENSE for details. 6 | 7 | """ 8 | 9 | # TODO: pg_size_pretty() style output formatting, e.g. 1024 -> 1k 10 | 11 | import os 12 | import re 13 | import codecs 14 | import json 15 | import socket 16 | import sys 17 | import uuid 18 | 19 | 20 | if sys.version_info[0] >= 3: 21 | unicode = str # pylint: disable=W0622 22 | 23 | 24 | class Error(Exception): 25 | """recode error""" 26 | 27 | class EncodeError(Error): 28 | """encode error""" 29 | 30 | class InvalidCodecDefinition(Error): 31 | """invalid codec definition""" 32 | 33 | 34 | ENCODE = "+" 35 | DECODE = "-" 36 | 37 | BOOL_MAP = {"true": True, "1": True, "on": True, 38 | "false": False, "0": False, "off": False} 39 | 40 | RE_CODER = re.compile("([-+]?)([a-z0-9_-]+)", re.I) 41 | 42 | MULTIPLES = { 43 | # SI 44 | "k": 10 ** 3, 45 | "M": 10 ** 6, 46 | "G": 10 ** 9, 47 | "T": 10 ** 12, 48 | "P": 10 ** 15, 49 | "E": 10 ** 18, 50 | "Z": 10 ** 21, 51 | "Y": 10 ** 24, 52 | 53 | # IEEE 1541 54 | 'Ki': 2 ** 10, 55 | 'Mi': 2 ** 20, 56 | 'Gi': 2 ** 30, 57 | 'Ti': 2 ** 40, 58 | 'Pi': 2 ** 50, 59 | 'Ei': 2 ** 60, 60 | } 61 | 62 | RE_MULT = re.compile("(.*)(%s)$" % ("|".join(MULTIPLES))) 63 | 64 | 65 | def to_int(value): 66 | if value is None: 67 | return 0 68 | else: 69 | return int(value, 0) # supports binary, octal, decimal and hex formats 70 | 71 | 72 | def to_float(value): 73 | if value is None: 74 | return 0.0 75 | else: 76 | return float(value) 77 | 78 | 79 | def to_str(value): 80 | if value is None: 81 | return u"" 82 | elif isinstance(value, unicode): 83 | return value 84 | else: 85 | return unicode(value) 86 | 87 | 88 | def from_env(value): 89 | """This function supports MUST or OPTIONAL environment variables 90 | 91 | If you wish to enforce that environment variable must exist use 'VAR' 92 | If you wish to have that variable as optional use 'VAR|' 93 | If you wish to supply default value in case enviromental variable is not set use 'VAR|default_value' 94 | """ 95 | try: 96 | (env_key, default_value) = value.split("|", 1) 97 | except ValueError: 98 | (env_key, default_value) = (value, None) 99 | 100 | try: 101 | return unicode(os.environ[env_key]) 102 | except KeyError: 103 | if (default_value != None): 104 | return unicode(default_value) 105 | 106 | raise ValueError("environment variable %r is not set" % env_key) 107 | 108 | 109 | def resolve_ip(name, family): 110 | try: 111 | addresses = socket.getaddrinfo(name, None, family) 112 | except (socket.error, socket.gaierror) as error: 113 | raise EncodeError("resolving %r failed: %s: %s" % ( 114 | name, error.__class__.__name__, error)) 115 | 116 | if not addresses: 117 | raise EncodeError("name %r does not resolve to any addresses" % name) 118 | 119 | return unicode(addresses[0][-1][0]) 120 | 121 | def convert_num(cls, value): 122 | if value is None: 123 | return cls(value) 124 | 125 | match = RE_MULT.match(value) 126 | if match: 127 | num_val = cls(match.group(1)) 128 | return num_val * MULTIPLES[match.group(2)] 129 | else: 130 | return cls(value) 131 | 132 | 133 | def to_bool(value): 134 | if value is None: 135 | return False 136 | 137 | try: 138 | return BOOL_MAP[value] 139 | except KeyError: 140 | raise ValueError("invalid boolean value: %r, expected one of: %s" % ( 141 | value, ", ".join(repr(x) for x in BOOL_MAP))) 142 | 143 | 144 | def to_uuid(value): 145 | return unicode(uuid.UUID(bytes=value)) 146 | 147 | 148 | def to_uuid4(value): 149 | return unicode(uuid.uuid4()) 150 | 151 | 152 | type_conversions = { 153 | "str": (to_str, None), 154 | "int": (lambda x: convert_num(to_int, x), None), 155 | "float": (lambda x: convert_num(to_float, x), None), 156 | "bool": (to_bool, None), 157 | "json": (json.dumps, json.loads), 158 | "null": (lambda x: None, None), 159 | "eval": (eval, None), 160 | "env": (from_env, None), 161 | "ipv4": (lambda name: resolve_ip(name, socket.AF_INET), None), 162 | "ipv6": (lambda name: resolve_ip(name, socket.AF_INET6), None), 163 | "uuid": (to_uuid, None), 164 | "uuid4": (to_uuid4, None), 165 | } 166 | 167 | 168 | class Codec(object): 169 | def __init__(self, chain_str, converters=None, default=None): 170 | self.default = default 171 | self.chain = [] 172 | self.converters = converters or {} 173 | self.parse_chain(chain_str) 174 | 175 | def parse_chain(self, chain_str): 176 | parts = chain_str.split(":") 177 | for part in parts: 178 | match = RE_CODER.match(part) 179 | if not match: 180 | raise InvalidCodecDefinition( 181 | "invalid codec definition: %r, at %r" % (chain_str, part)) 182 | 183 | direction, codec_name = match.groups() 184 | self.add_to_chain(codec_name, direction) 185 | 186 | def get_coder(self, codec_name, direction): 187 | converters = self.converters.get(codec_name) 188 | if not converters: 189 | converters = type_conversions.get(codec_name) 190 | 191 | if converters: 192 | if direction == DECODE: 193 | converter = converters[1] 194 | else: 195 | converter = converters[0] 196 | 197 | if not converter: 198 | raise InvalidCodecDefinition( 199 | "cannot convert with %r to this direction" % codec_name) 200 | 201 | return converter 202 | 203 | if direction == ENCODE: 204 | get_func = codecs.getencoder 205 | else: 206 | get_func = codecs.getdecoder 207 | 208 | try: 209 | codec_func = get_func(codec_name) 210 | return lambda value: codec_func(value)[0] 211 | except LookupError as error: 212 | raise InvalidCodecDefinition(str(error)) 213 | 214 | def add_to_chain(self, codec_name, direction): 215 | if codec_name == "pass": 216 | # "no-operation" codec 217 | return 218 | 219 | direction = direction or self.default 220 | if not direction: 221 | raise InvalidCodecDefinition( 222 | "coding direction must be defined when no default direction " 223 | "is specified") 224 | 225 | coder = self.get_coder(codec_name, direction) 226 | self.chain.append((direction, codec_name, coder)) 227 | 228 | def process(self, input_str): 229 | result = input_str 230 | for direction, codec_name, coder in self.chain: 231 | # short-circuit decodes from ascii if we're already unicode 232 | if codec_name == "ascii" and direction == DECODE and isinstance(result, unicode): 233 | continue 234 | try: 235 | result = coder(result) 236 | except Exception as error: 237 | if direction == ENCODE: 238 | format_str = "converting value %r to %r failed: %s" 239 | else: 240 | format_str = "converting value %r from %r failed: %s" 241 | 242 | raise ValueError(format_str % (result, codec_name, error)) 243 | 244 | return result 245 | 246 | -------------------------------------------------------------------------------- /tests/test_cmd_basic.py: -------------------------------------------------------------------------------- 1 | from __future__ import print_function 2 | import json 3 | import os 4 | from poni import tool 5 | from helper import * 6 | 7 | single_file_plugin_text = """ 8 | from poni import config 9 | 10 | class PlugIn(config.PlugIn): 11 | def __init__(self, *a, **kwa): 12 | super(PlugIn, self).__init__(*a, **kwa) 13 | self.render = self.render_name_template 14 | 15 | def add_actions(self): 16 | self.add_file("%(source)s", dest_path="%(dest)s", auto_override=%(override)s) 17 | """ 18 | 19 | class TestCommands(Helper): 20 | def test_add_node(self): 21 | poni, repo = self.init_repo() 22 | nodes = ["foo", "bar/foo/baz"] 23 | for node in nodes: 24 | poni.run(["add-node", node]) 25 | node_config = os.path.join(repo, "system", node, "node.json") 26 | with open(node_config, "r") as f: 27 | config = json.load(f) 28 | print(node, config) 29 | assert isinstance(config, dict) 30 | assert config["host"] == "" 31 | 32 | def test_create(self): 33 | poni, repo = self.init_repo() 34 | nodes = ["foo", "bar/foo/baz"] 35 | for node in nodes: 36 | poni.run(["add-system", node]) 37 | node_config = os.path.join(repo, "system", node, "system.json") 38 | with open(node_config, "r") as f: 39 | config = json.load(f) 40 | print(node, config) 41 | assert isinstance(config, dict) 42 | assert config == {} 43 | 44 | def test_set(self): 45 | poni, repo = self.init_repo() 46 | node = "test" 47 | assert not poni.run(["add-node", node]) 48 | vals = { 49 | "foo": ("=bar", "bar"), 50 | "a": ("=b=c", "b=c"), 51 | "int": (":int=1", 1), 52 | "float": (":float=2", 2.0), 53 | "str": (":str=123", "123"), 54 | "b0": (":bool=0", False), 55 | "b1": (":bool=1", True), 56 | "bt": (":bool=true", True), 57 | "bf": (":bool=false", False), 58 | "bon": (":bool=on", True), 59 | "boff": (":bool=off", False), 60 | } 61 | 62 | node_config = os.path.join(repo, "system", node, "node.json") 63 | for key, (inval, outval) in vals.items(): 64 | set_str = "%s%s" % (key, inval) 65 | assert not poni.run(["set", node, set_str]) 66 | 67 | with open(node_config, "r") as f: 68 | config = json.load(f) 69 | assert config[key] == outval, "got %r, expected %r" % ( 70 | config[key], outval) 71 | 72 | assert not poni.run(["set", node, "one.two.three.four=five", "-v"]) 73 | with open(node_config, "r") as f: 74 | config = json.load(f) 75 | assert config["one"]["two"]["three"]["four"] == "five" 76 | 77 | def test_list(self): 78 | poni, repo = self.init_repo() 79 | node = "test" 80 | conf = "conf" 81 | assert not poni.run(["add-node", node]) 82 | assert not poni.run(["add-config", node, conf]) 83 | assert not poni.run(["set", node, "cloud.blah=foo"]) 84 | flags = ["--systems", "--config", "--tree", "--full-match", 85 | "--controls", "--nodes", "--systems", 86 | "--node-prop", "--cloud", "--query-status", "--config-prop", 87 | "--inherits", "--line-per-prop"] 88 | for combo in combos(flags, max_len=4): 89 | cmd = ["list"] 90 | cmd.extend(combo) 91 | print(cmd) 92 | assert not poni.run(cmd) 93 | 94 | def test_script(self): 95 | for combo in combos(["--verbose"]): 96 | poni, repo = self.init_repo() 97 | node = "test" 98 | script_file = self.temp_file() 99 | with open(script_file, "w") as f: 100 | f.write("# poni.template: name\nadd-node %s\nset %s foo=bar" % (node, node)) 101 | assert not poni.run(["script", script_file]) 102 | node_config = os.path.join(repo, "system", node, "node.json") 103 | with open(node_config, "r") as f: 104 | config = json.load(f) 105 | assert config["foo"] == "bar" 106 | 107 | # TODO: stdin version of "script" command 108 | 109 | def test_verify_no_content(self): 110 | poni, repo = self.init_repo() 111 | node = "test" 112 | conf = "conf" 113 | assert not poni.run(["add-node", node]) 114 | assert not poni.run(["add-config", node, conf]) 115 | assert not poni.run(["verify"]) 116 | assert not poni.run(["deploy"]) 117 | assert not poni.run(["audit"]) 118 | 119 | def _make_inherited_config(self, template_node, template_conf, 120 | instance_node, instance_conf, 121 | source_file, file_contents, output_file, 122 | auto_override=False): 123 | # add template config 124 | tconf_path = "%s/%s" % (template_node, template_conf) 125 | source_file = "test.txt" 126 | args = dict(source=source_file, dest=output_file, override=auto_override) 127 | plugin_text = single_file_plugin_text % args 128 | 129 | poni = self.repo_and_config(template_node, template_conf, plugin_text) 130 | assert not poni.run(["set", template_node, "verify:bool=off"]) 131 | 132 | # write template config template file 133 | tfile_path = os.path.join(poni.default_repo_path, "system", template_node, "config", template_conf, source_file) 134 | with open(tfile_path, "w") as f: 135 | f.write(file_contents) 136 | 137 | # add inherited config 138 | assert not poni.run(["add-node", instance_node]) 139 | assert not poni.run(["set", instance_node, "deploy=local"]) 140 | assert not poni.run(["add-config", instance_node, instance_conf, 141 | "--inherit", tconf_path]) 142 | # deploy and verify 143 | assert not poni.run(["deploy"]) 144 | return poni 145 | 146 | def test_config_inherit(self): 147 | template_text = "hello" 148 | output_file = self.temp_file() 149 | poni = self._make_inherited_config("tnode", "tconf", "inode", "iconf", 150 | "test.txt", template_text, output_file) 151 | with open(output_file, "r") as f: 152 | assert f.read() == template_text 153 | 154 | def test_auto_override_config(self): 155 | template_text = "hello" 156 | output_file = self.temp_file() 157 | poni = self._make_inherited_config("tnode", "tconf", "inode", "iconf", 158 | "test.txt", template_text, output_file, 159 | auto_override=True) 160 | with open(output_file, "r") as f: 161 | assert f.read() == template_text 162 | # update inherited node with new content in "test.txt" 163 | new_template_text = "world" 164 | tmpfile = os.path.join(self.temp_dir(), "test.txt") 165 | with open(tmpfile, "w") as f: 166 | f.write(new_template_text) 167 | poni.run(["update-config", "-v", "inode/iconf", tmpfile]) 168 | # see if the file is deployed with changed content 169 | poni.run(["deploy"]) 170 | with open(output_file, "r") as f: 171 | assert f.read() == new_template_text 172 | 173 | def test_require(self): 174 | poni, repo = self.init_repo() 175 | assert not poni.run(["require", "poni_version>='0.1'"]) 176 | assert not poni.run(["require", "-v", "poni_version<'100'"]) 177 | assert poni.run(["require", "poni_version=='0.0'"]) == -1 178 | assert poni.run(["require", "xxx"]) == -1 179 | 180 | def test_version(self): 181 | poni, repo = self.init_repo() 182 | assert not poni.run(["version"]) 183 | -------------------------------------------------------------------------------- /doc/conf.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # 3 | # Poni documentation build configuration file, created by 4 | # sphinx-quickstart on Sat Dec 4 17:47:18 2010. 5 | # 6 | # This file is execfile()d with the current directory set to its containing dir. 7 | # 8 | # Note that not all possible configuration values are present in this 9 | # autogenerated file. 10 | # 11 | # All configuration values have a default; values that are commented out 12 | # serve to show the default. 13 | 14 | import sys, os 15 | import subprocess 16 | 17 | # If extensions (or modules to document with autodoc) are in another directory, 18 | # add these directories to sys.path here. If the directory is relative to the 19 | # documentation root, use os.path.abspath to make it absolute, like shown here. 20 | #sys.path.insert(0, os.path.abspath('.')) 21 | sys.path.insert(0, os.path.abspath('..')) 22 | 23 | # -- General configuration ----------------------------------------------------- 24 | 25 | # If your documentation needs a minimal Sphinx version, state it here. 26 | #needs_sphinx = '1.0' 27 | 28 | # Add any Sphinx extension module names here, as strings. They can be extensions 29 | # coming with Sphinx (named 'sphinx.ext.*') or your custom ones. 30 | extensions = ['sphinx.ext.autodoc'] 31 | try: 32 | import sphinxtogithub 33 | extensions.append('sphinxtogithub') 34 | except ImportError: 35 | pass 36 | 37 | # Add any paths that contain templates here, relative to this directory. 38 | templates_path = ['_templates'] 39 | 40 | # The suffix of source filenames. 41 | source_suffix = '.rst' 42 | 43 | # The encoding of source files. 44 | #source_encoding = 'utf-8-sig' 45 | 46 | # The master toctree document. 47 | master_doc = 'index' 48 | 49 | # General information about the project. 50 | project = u'Poni' 51 | copyright = u'2010-2012, Mika Eloranta' 52 | 53 | # The version info for the project you're documenting, acts as replacement for 54 | # |version| and |release|, also used in various other places throughout the 55 | # built documents. 56 | # 57 | git_version = subprocess.Popen(["git", "describe", "--long"], stdout=subprocess.PIPE).communicate()[0] 58 | # The short X.Y version. 59 | version = git_version.decode("utf-8").split("-")[0] 60 | # The full version, including alpha/beta/rc tags. 61 | release = git_version.strip() 62 | 63 | # The language for content autogenerated by Sphinx. Refer to documentation 64 | # for a list of supported languages. 65 | #language = None 66 | 67 | # There are two options for replacing |today|: either, you set today to some 68 | # non-false value, then it is used: 69 | #today = '' 70 | # Else, today_fmt is used as the format for a strftime call. 71 | #today_fmt = '%B %d, %Y' 72 | 73 | # List of patterns, relative to source directory, that match files and 74 | # directories to ignore when looking for source files. 75 | exclude_patterns = ['_build', 'definitions.rst'] 76 | 77 | # The reST default role (used for this markup: `text`) to use for all documents. 78 | #default_role = None 79 | 80 | # If true, '()' will be appended to :func: etc. cross-reference text. 81 | #add_function_parentheses = True 82 | 83 | # If true, the current module name will be prepended to all description 84 | # unit titles (such as .. function::). 85 | #add_module_names = True 86 | 87 | # If true, sectionauthor and moduleauthor directives will be shown in the 88 | # output. They are ignored by default. 89 | #show_authors = False 90 | 91 | # The name of the Pygments (syntax highlighting) style to use. 92 | pygments_style = 'sphinx' 93 | 94 | # A list of ignored prefixes for module index sorting. 95 | #modindex_common_prefix = [] 96 | 97 | 98 | # -- Options for HTML output --------------------------------------------------- 99 | 100 | # The theme to use for HTML and HTML Help pages. See the documentation for 101 | # a list of builtin themes. 102 | html_theme = 'haiku' 103 | 104 | # Theme options are theme-specific and customize the look and feel of a theme 105 | # further. For a list of options available for each theme, see the 106 | # documentation. 107 | #html_theme_options = {} 108 | 109 | # Add any paths that contain custom themes here, relative to this directory. 110 | #html_theme_path = [] 111 | 112 | # The name for this set of Sphinx documents. If None, it defaults to 113 | # " v documentation". 114 | #html_title = None 115 | 116 | # A shorter title for the navigation bar. Default is the same as html_title. 117 | #html_short_title = None 118 | 119 | # The name of an image file (relative to this directory) to place at the top 120 | # of the sidebar. 121 | #html_logo = None 122 | 123 | # The name of an image file (within the static path) to use as favicon of the 124 | # docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 125 | # pixels large. 126 | #html_favicon = None 127 | 128 | # Add any paths that contain custom static files (such as style sheets) here, 129 | # relative to this directory. They are copied after the builtin static files, 130 | # so a file named "default.css" will overwrite the builtin "default.css". 131 | html_static_path = ['_static'] 132 | 133 | # If not '', a 'Last updated on:' timestamp is inserted at every page bottom, 134 | # using the given strftime format. 135 | #html_last_updated_fmt = '%b %d, %Y' 136 | 137 | # If true, SmartyPants will be used to convert quotes and dashes to 138 | # typographically correct entities. 139 | #html_use_smartypants = True 140 | 141 | # Custom sidebar templates, maps document names to template names. 142 | #html_sidebars = {} 143 | 144 | # Additional templates that should be rendered to pages, maps page names to 145 | # template names. 146 | #html_additional_pages = {} 147 | 148 | # If false, no module index is generated. 149 | #html_domain_indices = True 150 | 151 | # If false, no index is generated. 152 | #html_use_index = True 153 | 154 | # If true, the index is split into individual pages for each letter. 155 | #html_split_index = False 156 | 157 | # If true, links to the reST sources are added to the pages. 158 | #html_show_sourcelink = True 159 | 160 | # If true, "Created using Sphinx" is shown in the HTML footer. Default is True. 161 | #html_show_sphinx = True 162 | 163 | # If true, "(C) Copyright ..." is shown in the HTML footer. Default is True. 164 | #html_show_copyright = True 165 | 166 | # If true, an OpenSearch description file will be output, and all pages will 167 | # contain a tag referring to it. The value of this option must be the 168 | # base URL from which the finished HTML is served. 169 | #html_use_opensearch = '' 170 | 171 | # This is the file name suffix for HTML files (e.g. ".xhtml"). 172 | #html_file_suffix = None 173 | 174 | # Output file base name for HTML help builder. 175 | htmlhelp_basename = 'Ponidoc' 176 | 177 | 178 | # -- Options for LaTeX output -------------------------------------------------- 179 | 180 | # The paper size ('letter' or 'a4'). 181 | #latex_paper_size = 'letter' 182 | 183 | # The font size ('10pt', '11pt' or '12pt'). 184 | #latex_font_size = '10pt' 185 | 186 | # Grouping the document tree into LaTeX files. List of tuples 187 | # (source start file, target name, title, author, documentclass [howto/manual]). 188 | latex_documents = [ 189 | ('index', 'Poni.tex', u'Poni Documentation', 190 | u'Mika Eloranta', 'manual'), 191 | ] 192 | 193 | # The name of an image file (relative to this directory) to place at the top of 194 | # the title page. 195 | #latex_logo = None 196 | 197 | # For "manual" documents, if this is true, then toplevel headings are parts, 198 | # not chapters. 199 | #latex_use_parts = False 200 | 201 | # If true, show page references after internal links. 202 | #latex_show_pagerefs = False 203 | 204 | # If true, show URL addresses after external links. 205 | #latex_show_urls = False 206 | 207 | # Additional stuff for the LaTeX preamble. 208 | #latex_preamble = '' 209 | 210 | # Documents to append as an appendix to all manuals. 211 | #latex_appendices = [] 212 | 213 | # If false, no module index is generated. 214 | #latex_domain_indices = True 215 | 216 | 217 | # -- Options for manual page output -------------------------------------------- 218 | 219 | # One entry per manual page. List of tuples 220 | # (source start file, name, description, authors, manual section). 221 | man_pages = [ 222 | ('index', 'poni', u'Poni Documentation', 223 | [u'Mika Eloranta'], 1) 224 | ] 225 | -------------------------------------------------------------------------------- /doc/properties.rst: -------------------------------------------------------------------------------- 1 | .. _propref: 2 | 3 | System and Node Property Reference 4 | ================================== 5 | 6 | Properties Controlled by Poni 7 | ----------------------------- 8 | The following properties are automatically maintained by Poni: 9 | 10 | .. list-table:: 11 | :widths: 10 30 8 30 12 | :header-rows: 1 13 | 14 | * - Property 15 | - Description 16 | - Data Type 17 | - Example 18 | * - ``depth`` 19 | - How many levels deep the node (or system) is in the hierarchy 20 | - integer 21 | - ``1`` (means: root-level) 22 | * - ``index`` 23 | - Location of the node/system in relation to its siblings in the same 24 | system 25 | - integer 26 | - ``0`` (means: first node) 27 | 28 | Generic Properties 29 | ------------------ 30 | 31 | .. list-table:: 32 | :widths: 10 30 8 30 33 | :header-rows: 1 34 | 35 | * - Property 36 | - Description 37 | - Data Type 38 | - Example 39 | * - ``host`` 40 | - Host network address (used with SSH-based access) 41 | - string 42 | - ``server01.company.com`` 43 | * - ``user`` 44 | - Username for accessing the host (used with SSH-based access) 45 | - string 46 | - ``root`` 47 | * - ``ssh-key`` 48 | - Filename of the SSH key used to access the host as ``user`` 49 | - string 50 | - ``id_rsa`` 51 | * - ``ssh-port`` 52 | - SSH server port number. If not set, a default value from the 53 | environment variable ``PONI_SSH_PORT`` is used. If that is not set, 54 | then the standard port ``22`` is used. 55 | - string 56 | - ``8022`` 57 | * - ``parent`` 58 | - Full name of the parent node (if defined), set automatically when node 59 | is created with ``poni add-node CHILD -i PARENT`` 60 | - string 61 | - ``some/other/node`` 62 | * - ``verify`` 63 | - Is verification enabled for this node/system and all of its 64 | sub-systems? Set to ``false`` for e.g. template nodes. **NOTE:** 65 | Affects all sub-systems and their nodes, too. 66 | - boolean 67 | - ``true`` (verification is enabled) or ``false`` (verification is 68 | disabled) 69 | * - ``template`` 70 | - Indicates a system or node only containing templates. Control commands 71 | are not run under template nodes and implies ``verify=false``. 72 | - boolean 73 | - ``true`` (system/node contains only templates) or ``false`` (regular 74 | system/node, default) 75 | * - ``deploy`` 76 | - Node access method. Default is ``ssh`` if not defined with this 77 | property. **NOTE:** Affects all sub-systems and their nodes, too. 78 | - string 79 | - ``ssh`` or ``local`` 80 | 81 | Amazon EC2 Properties 82 | --------------------- 83 | .. list-table:: 84 | :widths: 15 30 3 8 30 85 | :header-rows: 1 86 | 87 | * - Property 88 | - Description 89 | - Required 90 | - Data Type 91 | - Example 92 | * - ``cloud.provider`` 93 | - Selects the cloud provider, must be ``aws-ec2`` 94 | - **YES** 95 | - string 96 | - ``aws-ec2`` 97 | * - ``cloud.image`` 98 | - The "AMI" code of the VM image 99 | - **YES** 100 | - string 101 | - ``ami-daf615b3`` 102 | * - ``cloud.kernel`` 103 | - The "AKI" code of the kernel 104 | - NO 105 | - string 106 | - ``aki-6eaa4907`` 107 | * - ``cloud.ramdisk`` 108 | - The "ARI" code of the ramdisk 109 | - NO 110 | - string 111 | - ``ari-42b95a2b`` 112 | * - ``cloud.region`` 113 | - AWS EC2 data-center 114 | - **YES** 115 | - string 116 | - one of ``us-west-1``, ``us-east-1``, ``eu-west-1`` or ``ap-southeast-1`` 117 | * - ``cloud.type`` 118 | - Instance type 119 | - NO 120 | - string 121 | - ``m1.small``, ``m2.xlarge``, etc. 122 | * - ``cloud.key_pair`` 123 | - Name of the EC2 data-center specific key-pair to use **without** the 124 | ``.pem`` suffix 125 | - **YES** 126 | - string 127 | - ``my-key-pair`` (in case the file you have is ``my-key-pair.pem``) 128 | * - ``cloud.instance`` 129 | - Poni updates the id of the instance here once it has been started 130 | - n/a 131 | - string 132 | - ``i-4692cd2b`` 133 | * - ``cloud.vm_name`` 134 | - User-friendly name for the VM, visible in the EC2 console. 135 | - **YES** 136 | - string 137 | - ``cloud-server-01`` 138 | * - ``cloud.placement`` 139 | - The availability zone in which to launch the instance. 140 | - NO 141 | - string 142 | - ``us-east-1c`` 143 | * - ``cloud.placement_group`` 144 | - Name of the placement group in which the instance will be launched. 145 | - NO 146 | - string 147 | - ``my-group`` 148 | * - ``cloud.billing`` 149 | - Instance billing type 150 | - NO 151 | - string 152 | - ``on-demand`` (default) or ``spot`` 153 | * - ``cloud.spot.max_price`` 154 | - Spot instance maximum price, required if ``cloud.billing`` is set to ``spot``. 155 | - NO 156 | - float 157 | - ``0.123`` 158 | * - ``cloud.hardware`` 159 | - Extra "hardware" to attach to the instance. (See description below) 160 | - NO 161 | - dict 162 | - ``{"disk0": {"size": 2048, "device": "/dev/sdh"}`` 163 | * - ``disable_api_termination`` 164 | - If True, the instances will be locked and will not be able to be terminated via the API. 165 | - NO 166 | - bool 167 | - ``False`` (default) or ``True`` 168 | * - ``monitoring_enabled`` 169 | - Enable CloudWatch monitoring on the instance. 170 | - NO 171 | - bool 172 | - ``False`` (default) or ``True`` 173 | * - ``subnet`` 174 | - The subnet ID or name within which to launch the instances for VPC. The name is in subnet 175 | object's tags with the key 'Name'. 176 | - NO 177 | - str 178 | - ````, ``My subnet X`` 179 | * - ``private_ip_address`` 180 | - If you’re using VPC, you can optionally use this parameter to assign the 181 | instance a specific available IP address from the subnet. 182 | - NO 183 | - str 184 | - ``10.0.0.25`` 185 | * - ``tenancy`` 186 | - The tenancy of the instance you want to launch. An instance with a 187 | tenancy of ‘dedicated’ runs on single-tenant hardware and can only be 188 | launched into a VPC. Valid values are: “default” or “dedicated”. 189 | NOTE: To use dedicated tenancy you MUST specify a VPC subnet-ID as well. 190 | - NO 191 | - str 192 | - ``default``, ``dedicated`` 193 | * - ``instance_profile_name`` 194 | - IAM instance profile name. 195 | - NO 196 | - str 197 | - ```` 198 | * - ``extra_tags`` 199 | - Extra tag names and corresponding values used to tag the created VM instances. 200 | Can be used to maintain extra book-keeping of e.g. owners of the VMs in a 201 | shared environment. Note that each key and value are required to be strings. 202 | - NO 203 | - dict 204 | - ``'cloud.extra_tags:-json={"cost_centre": "12345", "owner": "John Doe"}'`` 205 | * - ``init_timeout`` 206 | - Maximum time to wait until instance reaches healthy running status after 207 | creation. If not specified the value from the environment variable 208 | ``PONI_AWS_INIT_TIMEOUT`` is used. If the environment variable is not 209 | defined, then a default of ``300.0`` seconds is used. 210 | - NO 211 | - float or int 212 | - ``600.0`` 213 | * - ``check_health`` 214 | - Control usage of "instance" and "system" health checks during ``cloud init``. 215 | If ``True`` (default), wait until both health checks return "ok". Otherwise 216 | proceed immediately without checking instance health. Leaving this enabled 217 | will result in a somewhat slower instance creation as the health check 218 | results are not immediately available after the instance is "running". 219 | - NO 220 | - float 221 | - ``True`` or ``False`` 222 | 223 | .. note:: 224 | Many EC2 instance properties cannot be controlled yet, for example: user data, 225 | addressing types or monitoring. 226 | 227 | 228 | Extra Hardware 229 | ~~~~~~~~~~~~~~ 230 | The ``cloud.hardware`` property can be used to define additional EBS volumes to 231 | be created and automatically attached to the instance. The value needs to be a 232 | ``dict`` and can be set as follows:: 233 | 234 | poni set some/server 'cloud.hardware:-json={"disk0": {"size": 2048, "device": "/dev/sdh"}' 235 | 236 | The keys in the dict (or JSON object...) define the type of the hardware 237 | resource, currently ``disk0..disk9`` are supported. Each disk definition 238 | corresponds to one EBS volume and one device path within the instance. 239 | 240 | The value of each ``diskN`` is another dict/JSON object, definiting the 241 | properties of the disk: 242 | 243 | .. list-table:: 244 | :widths: 15 30 3 8 30 245 | :header-rows: 1 246 | 247 | * - Property 248 | - Description 249 | - Required 250 | - Data Type 251 | - Example 252 | * - ``size`` 253 | - Size in megabytes, must be at least 1024 MB. 254 | - **YES** 255 | - int 256 | - ``8192`` (8 GB) 257 | * - ``device`` 258 | - Device path within the instance where the volume will be available. 259 | - **YES** 260 | - string 261 | - ``/dev/sdh`` 262 | * - ``delete_on_termination`` 263 | - If set to false, the EBS volume will remain after the instance gets terminated. 264 | - NO 265 | - bool 266 | - ``true`` (default), ``false`` 267 | -------------------------------------------------------------------------------- /poni/orddict.py: -------------------------------------------------------------------------------- 1 | # Backport of OrderedDict() class that runs on Python 2.4, 2.5, 2.6, 2.7 and pypy. 2 | # Passes Python2.7's test suite and incorporates all the latest updates. 3 | # http://code.activestate.com/recipes/576693/ 4 | 5 | try: 6 | from thread import get_ident as _get_ident 7 | except ImportError: 8 | from dummy_thread import get_ident as _get_ident 9 | 10 | try: 11 | from _abcoll import KeysView, ValuesView, ItemsView 12 | except ImportError: 13 | pass 14 | 15 | 16 | class OrderedDict(dict): 17 | 'Dictionary that remembers insertion order' 18 | # An inherited dict maps keys to values. 19 | # The inherited dict provides __getitem__, __len__, __contains__, and get. 20 | # The remaining methods are order-aware. 21 | # Big-O running times for all methods are the same as for regular dictionaries. 22 | 23 | # The internal self.__map dictionary maps keys to links in a doubly linked list. 24 | # The circular doubly linked list starts and ends with a sentinel element. 25 | # The sentinel element never gets deleted (this simplifies the algorithm). 26 | # Each link is stored as a list of length three: [PREV, NEXT, KEY]. 27 | 28 | def __init__(self, *args, **kwds): 29 | '''Initialize an ordered dictionary. Signature is the same as for 30 | regular dictionaries, but keyword arguments are not recommended 31 | because their insertion order is arbitrary. 32 | 33 | ''' 34 | dict.__init__(self) 35 | if len(args) > 1: 36 | raise TypeError('expected at most 1 arguments, got %d' % len(args)) 37 | try: 38 | self.__root 39 | except AttributeError: 40 | self.__root = root = [] # sentinel node 41 | root[:] = [root, root, None] 42 | self.__map = {} 43 | self.__update(*args, **kwds) 44 | 45 | def __setitem__(self, key, value, dict_setitem=dict.__setitem__): 46 | 'od.__setitem__(i, y) <==> od[i]=y' 47 | # Setting a new item creates a new link which goes at the end of the linked 48 | # list, and the inherited dictionary is updated with the new key/value pair. 49 | if key not in self: 50 | root = self.__root 51 | last = root[0] 52 | last[1] = root[0] = self.__map[key] = [last, root, key] 53 | dict_setitem(self, key, value) 54 | 55 | def __delitem__(self, key, dict_delitem=dict.__delitem__): 56 | 'od.__delitem__(y) <==> del od[y]' 57 | # Deleting an existing item uses self.__map to find the link which is 58 | # then removed by updating the links in the predecessor and successor nodes. 59 | dict_delitem(self, key) 60 | link_prev, link_next, key = self.__map.pop(key) 61 | link_prev[1] = link_next 62 | link_next[0] = link_prev 63 | 64 | def __iter__(self): 65 | 'od.__iter__() <==> iter(od)' 66 | root = self.__root 67 | curr = root[1] 68 | while curr is not root: 69 | yield curr[2] 70 | curr = curr[1] 71 | 72 | def __reversed__(self): 73 | 'od.__reversed__() <==> reversed(od)' 74 | root = self.__root 75 | curr = root[0] 76 | while curr is not root: 77 | yield curr[2] 78 | curr = curr[0] 79 | 80 | def clear(self): 81 | 'od.clear() -> None. Remove all items from od.' 82 | try: 83 | for node in self.__map.itervalues(): # pylint: disable=E1101 84 | del node[:] 85 | root = self.__root 86 | root[:] = [root, root, None] 87 | self.__map.clear() 88 | except AttributeError: 89 | pass 90 | dict.clear(self) 91 | 92 | def popitem(self, last=True): 93 | '''od.popitem() -> (k, v), return and remove a (key, value) pair. 94 | Pairs are returned in LIFO order if last is true or FIFO order if false. 95 | 96 | ''' 97 | if not self: 98 | raise KeyError('dictionary is empty') 99 | root = self.__root 100 | if last: 101 | link = root[0] 102 | link_prev = link[0] 103 | link_prev[1] = root 104 | root[0] = link_prev 105 | else: 106 | link = root[1] 107 | link_next = link[1] 108 | root[1] = link_next 109 | link_next[0] = root 110 | key = link[2] 111 | del self.__map[key] 112 | value = dict.pop(self, key) 113 | return key, value 114 | 115 | # -- the following methods do not depend on the internal structure -- 116 | 117 | def keys(self): 118 | 'od.keys() -> list of keys in od' 119 | return list(self) 120 | 121 | def values(self): 122 | 'od.values() -> list of values in od' 123 | return [self[key] for key in self] 124 | 125 | def items(self): 126 | 'od.items() -> list of (key, value) pairs in od' 127 | return [(key, self[key]) for key in self] 128 | 129 | def iterkeys(self): 130 | 'od.iterkeys() -> an iterator over the keys in od' 131 | return iter(self) 132 | 133 | def itervalues(self): 134 | 'od.itervalues -> an iterator over the values in od' 135 | for k in self: 136 | yield self[k] 137 | 138 | def iteritems(self): 139 | 'od.iteritems -> an iterator over the (key, value) items in od' 140 | for k in self: 141 | yield (k, self[k]) 142 | 143 | def update(*args, **kwds): # pylint: disable=E0211 144 | '''od.update(E, **F) -> None. Update od from dict/iterable E and F. 145 | 146 | If E is a dict instance, does: for k in E: od[k] = E[k] 147 | If E has a .keys() method, does: for k in E.keys(): od[k] = E[k] 148 | Or if E is an iterable of items, does: for k, v in E: od[k] = v 149 | In either case, this is followed by: for k, v in F.items(): od[k] = v 150 | 151 | ''' 152 | if len(args) > 2: 153 | raise TypeError('update() takes at most 2 positional ' 154 | 'arguments (%d given)' % (len(args),)) 155 | elif not args: 156 | raise TypeError('update() takes at least 1 argument (0 given)') 157 | self = args[0] 158 | # Make progressively weaker assumptions about "other" 159 | other = () 160 | if len(args) == 2: 161 | other = args[1] 162 | if isinstance(other, dict): 163 | for key in other: 164 | self[key] = other[key] 165 | elif hasattr(other, 'keys'): 166 | for key in other.keys(): # pylint: disable=E1103 167 | self[key] = other[key] 168 | else: 169 | for key, value in other: 170 | self[key] = value 171 | for key, value in kwds.items(): 172 | self[key] = value 173 | 174 | __update = update # let subclasses override update without breaking __init__ 175 | 176 | __marker = object() 177 | 178 | def pop(self, key, default=__marker): 179 | '''od.pop(k[,d]) -> v, remove specified key and return the corresponding value. 180 | If key is not found, d is returned if given, otherwise KeyError is raised. 181 | 182 | ''' 183 | if key in self: 184 | result = self[key] 185 | del self[key] 186 | return result 187 | if default is self.__marker: 188 | raise KeyError(key) 189 | return default 190 | 191 | def setdefault(self, key, default=None): 192 | 'od.setdefault(k[,d]) -> od.get(k,d), also set od[k]=d if k not in od' 193 | if key in self: 194 | return self[key] 195 | self[key] = default 196 | return default 197 | 198 | def __repr__(self, _repr_running=None): 199 | 'od.__repr__() <==> repr(od)' 200 | _repr_running = {} if _repr_running is None else _repr_running 201 | call_key = id(self), _get_ident() 202 | if call_key in _repr_running: 203 | return '...' 204 | _repr_running[call_key] = 1 205 | try: 206 | if not self: 207 | return '%s()' % (self.__class__.__name__,) 208 | return '%s(%r)' % (self.__class__.__name__, self.items()) 209 | finally: 210 | del _repr_running[call_key] 211 | 212 | def __reduce__(self): 213 | 'Return state information for pickling' 214 | items = [[k, self[k]] for k in self] 215 | inst_dict = vars(self).copy() 216 | for k in vars(OrderedDict()): 217 | inst_dict.pop(k, None) 218 | if inst_dict: 219 | return (self.__class__, (items,), inst_dict) 220 | return self.__class__, (items,) 221 | 222 | def copy(self): 223 | 'od.copy() -> a shallow copy of od' 224 | return self.__class__(self) 225 | 226 | @classmethod 227 | def fromkeys(cls, iterable, value=None): 228 | '''OD.fromkeys(S[, v]) -> New ordered dictionary with keys from S 229 | and values equal to v (which defaults to None). 230 | 231 | ''' 232 | d = cls() 233 | for key in iterable: 234 | d[key] = value 235 | return d 236 | 237 | def __eq__(self, other): 238 | '''od.__eq__(y) <==> od==y. Comparison to another OD is order-sensitive 239 | while comparison to a regular mapping is order-insensitive. 240 | 241 | ''' 242 | if isinstance(other, OrderedDict): 243 | return len(self) == len(other) and self.items() == other.items() 244 | return dict.__eq__(self, other) 245 | 246 | def __ne__(self, other): 247 | return not self == other 248 | 249 | # -- the following methods are only used in Python 2.7 -- 250 | 251 | def viewkeys(self): 252 | "od.viewkeys() -> a set-like object providing a view on od's keys" 253 | return KeysView(self) 254 | 255 | def viewvalues(self): 256 | "od.viewvalues() -> an object providing a view on od's values" 257 | return ValuesView(self) 258 | 259 | def viewitems(self): 260 | "od.viewitems() -> a set-like object providing a view on od's items" 261 | return ItemsView(self) 262 | -------------------------------------------------------------------------------- /poni/listout.py: -------------------------------------------------------------------------------- 1 | """ 2 | List output formatting 3 | 4 | TODO: messy, rewrite 5 | 6 | Copyright (c) 2010-2012 Mika Eloranta 7 | See LICENSE for details. 8 | 9 | """ 10 | 11 | import os 12 | import sys 13 | from . import colors 14 | from . import core 15 | from . import util 16 | 17 | if sys.version_info[0] == 2: 18 | int_types = (int, long) # pylint: disable=E0602 19 | unicode_types = unicode # pylint: disable=E0602 20 | else: 21 | int_types = int 22 | unicode_types = None 23 | 24 | 25 | class ListOutput(colors.Output): 26 | def __init__(self, tool, confman, show_nodes=False, show_systems=False, 27 | show_config=False, show_tree=False, show_inherits=False, 28 | pattern=False, full_match=False, show_node_prop=False, 29 | show_cloud_prop=False, show_config_prop=False, 30 | list_props=False, show_layers=False, color="auto", 31 | show_controls=False, exclude=None, 32 | query_status=False, show_settings=False, **kwargs): 33 | colors.Output.__init__(self, sys.stdout, color=color) 34 | self.exclude = exclude or [] 35 | self.show_nodes = show_nodes 36 | self.show_systems = show_systems 37 | self.show_config = show_config 38 | self.show_tree = show_tree 39 | self.show_inherits = show_inherits 40 | self.show_node_prop = show_node_prop 41 | self.show_cloud_prop = show_cloud_prop 42 | self.show_config_prop = show_config_prop 43 | self.show_settings = show_settings 44 | self.show_layers = show_layers 45 | self.show_controls = show_controls 46 | self.list_props = list_props 47 | self.query_status = query_status 48 | self.pattern = pattern 49 | self.full_match = full_match 50 | self.tool = tool 51 | self.confman = confman 52 | self.hindent = 4 * " " 53 | self.indent = int(show_tree) * self.hindent 54 | self.formatters = { 55 | "node": self.format_node, 56 | "system": self.format_system, 57 | "config": self.format_config, 58 | "prop": self.format_prop, 59 | "cloud": self.format_prop, 60 | "confprop": self.format_prop, 61 | "controls": self.format_controls, 62 | "status": self.format_status, 63 | "setting": self.format_setting, 64 | "layer": self.format_layer, 65 | } 66 | 67 | def value_repr(self, value, top_level=False): 68 | if unicode_types and isinstance(value, unicode_types): 69 | try: 70 | value = repr(value.encode("ascii")) 71 | except UnicodeEncodeError: 72 | pass 73 | 74 | if isinstance(value, dict): 75 | if not value: 76 | yield "none", "gray" 77 | raise StopIteration() 78 | 79 | if not top_level: 80 | yield "{", None 81 | 82 | for i, (key, value) in enumerate(sorted(value.items())): 83 | if i > 0: 84 | yield ", ", None 85 | 86 | yield key, "key" 87 | yield ":", None 88 | 89 | for output in self.value_repr(value): 90 | yield output 91 | 92 | if not top_level: 93 | yield "}", None 94 | elif isinstance(value, str): 95 | yield value, "str" 96 | elif isinstance(value, bool): 97 | yield str(value), "bool" 98 | elif isinstance(value, int_types): 99 | yield str(value), "int" 100 | else: 101 | yield repr(value), "red" 102 | 103 | def format_setting(self, entry): 104 | yield entry["setting"], "setting" 105 | yield " = ", None 106 | for output in self.value_repr(entry["value"]): 107 | yield output 108 | 109 | def format_status(self, entry): 110 | yield entry["status"], "status" 111 | 112 | def format_layer(self, entry): 113 | yield "#%d: %s: %s" % (entry["index"], entry["layer"], 114 | os.path.basename(entry["file_path"])), "layer" 115 | 116 | def format_prop(self, entry): 117 | return self.value_repr(entry["prop"], top_level=True) 118 | 119 | def format_controls(self, entry): 120 | yield ", ".join(sorted(entry["controls"])), "controls" 121 | 122 | def format_system(self, entry): 123 | name = entry["item"].name 124 | if self.show_tree: 125 | name = name.rsplit("/", 1)[-1] 126 | 127 | yield name, "system" 128 | 129 | def format_node(self, entry): 130 | system = entry["item"].system 131 | if not self.show_tree and system.name: 132 | for output in self.format_system(dict(type="system", item=system)): 133 | yield output 134 | 135 | yield "/", None 136 | 137 | node_name = entry["item"].name.rsplit("/", 1)[-1] 138 | yield node_name, "node" 139 | 140 | parent_name = entry["item"].get("parent") 141 | if parent_name and self.show_inherits: 142 | yield "(", None 143 | yield parent_name, "nodeparent" 144 | yield ")", None 145 | 146 | def format_config(self, entry): 147 | if not self.show_tree: 148 | node = False 149 | for output in self.format_node(entry): 150 | node = True 151 | yield output 152 | 153 | if node: 154 | yield "/", None 155 | 156 | yield entry["config"].name, "config" 157 | 158 | parent_name = entry["config"].get("parent") 159 | if parent_name and self.show_inherits: 160 | yield "(", None 161 | yield parent_name, "configparent" 162 | yield ")", None 163 | 164 | if entry["inherited"]: 165 | yield " [inherited]", None 166 | 167 | def format_unknown(self, entry): 168 | yield "UNKNOWN: %r" % entry["type"], "red" 169 | 170 | def output(self): 171 | """Yields formatted strings ready for writing to the output file""" 172 | for text, color_code in self.output_pairs(): 173 | yield self.color(text, color_code) 174 | 175 | def output_pairs(self): 176 | for entry in self.iter_tree(): 177 | indent = (entry["item"].get("depth", 0) - 1) * self.indent 178 | if entry["type"] not in ["system", "node", "config"]: 179 | indent += self.hindent 180 | 181 | if entry["type"] == "config": 182 | indent += self.indent 183 | 184 | 185 | type_code = "%stype" % entry["type"] 186 | if type_code not in colors.CODES: 187 | type_code = "reset" 188 | 189 | yield "%8s" % entry["type"], type_code 190 | yield " ", None 191 | yield indent, None 192 | for output in self.formatters.get(entry["type"], 193 | self.format_unknown)(entry): 194 | yield output 195 | 196 | yield "\n", None 197 | 198 | 199 | def iter_tree(self): 200 | """ 201 | Yields every system, node, config, etc. that needs to be produced to 202 | the output. 203 | """ 204 | for item in self.confman.find(self.pattern, systems=self.show_systems, 205 | full_match=self.full_match, 206 | exclude=self.exclude): 207 | if isinstance(item, core.Node): 208 | if self.show_nodes: 209 | yield dict(type="node", item=item) 210 | elif self.show_systems: 211 | yield dict(type="system", item=item) 212 | 213 | if self.show_node_prop: 214 | items = dict(item.showable()) 215 | if self.list_props: 216 | for key_path, value in util.path_iter_dict(items): 217 | yield dict(type="prop", item=item, 218 | prop={key_path: value}) 219 | else: 220 | yield dict(type="prop", item=item, prop=items) 221 | 222 | if isinstance(item, core.Node): 223 | for conf in sorted(item.iter_all_configs(), 224 | key=lambda x: x.name): 225 | if self.show_config: 226 | yield dict(type="config", item=item, config=conf, inherited=(conf.node != item)) 227 | 228 | if self.show_layers: 229 | for i, (sort_key, layer_name, file_path) \ 230 | in enumerate(conf.settings.layers): 231 | yield dict(type="layer", item=item, config=conf, 232 | layer=layer_name, file_path=file_path, 233 | index=i) 234 | 235 | if self.show_config_prop: 236 | yield dict(type="confprop", item=item, config=conf, 237 | prop=conf) 238 | 239 | plugin = conf.get_plugin() 240 | if plugin and self.show_controls and plugin.controls: 241 | yield dict(type="controls", item=item, config=conf, 242 | controls=plugin.controls) 243 | 244 | if self.show_settings: 245 | for key, value in util.path_iter_dict(conf.settings): 246 | yield dict(type="setting", item=item, config=conf, 247 | setting=key, value=value) 248 | 249 | cloud_prop = item.get("cloud", {}) 250 | if self.show_cloud_prop and cloud_prop: 251 | if self.list_props: 252 | for key_path, value in util.path_iter_dict(cloud_prop): 253 | yield dict(type="cloud", item=item, 254 | prop={key_path: value}) 255 | else: 256 | yield dict(type="cloud", cloud=cloud_prop, item=item, 257 | prop=cloud_prop) 258 | 259 | if self.query_status and cloud_prop.get("instance"): 260 | provider = self.tool.sky.get_provider(cloud_prop) 261 | status = provider.get_instance_status(cloud_prop) 262 | yield dict(type="status", item=item, status=status) 263 | -------------------------------------------------------------------------------- /doc/changes.rst: -------------------------------------------------------------------------------- 1 | ============== 2 | Poni Changelog 3 | ============== 4 | 5 | Version 0.8 6 | =========== 7 | :release date: 2015-03-11 8 | 9 | * Python 3 compatibility 10 | * ``docker``, ``eucalyptus`` and ``image`` cloud deployment targets 11 | * Major improvements in ``libvirt`` cloud deployments 12 | * Support for ``mako`` and basic ``name`` templates 13 | * Tests use ``py.test`` instead of ``nose`` 14 | * Continuous integration on travis-ci.org 15 | * Cheetah is now an optional dependency 16 | * Removed ``path.py`` dependency 17 | * Various bug fixes and minor improvements 18 | 19 | Version 0.7 20 | =========== 21 | :release date: 2013-01-08 22 | 23 | * support for ``libvirt`` cloud provider for deploying to a set of hosts 24 | running libvirt 25 | 26 | Version 0.6 27 | =========== 28 | :release date: 2012-12-05 29 | 30 | * changed license from MIT to Apache 2.0 31 | * support for ``vsphere`` cloud provider for deploying to vmware clusters 32 | * various performance improvements 33 | * added ``--exclude PATTERN`` support for many commands: allows skipping nodes 34 | that match a pattern when e.g. running a remote command over multiple nodes 35 | * optimization: internal cache for loaded plugin modules 36 | * ``add_file()`` supports ``owner=uid`` and ``group=gid`` optional args 37 | * AWS security groups can be set via ``cloud.security_group`` 38 | **(thanks, Lakshmi!)** 39 | * ``cloud ip`` command for assigning an AWS Elastic IP for a node 40 | **(thanks, Lakshmi!)** 41 | * remote operations support ``pseudo_tty=True`` for allocating a ptty for 42 | the operation **(thanks, Lauri!)** 43 | * control command node logs include the ``BEGIN`` and ``END`` tags 44 | * render also ``source_path`` as a template 45 | * bugfix: ``poni script`` handles files with multi-line commands with comments 46 | in the middle 47 | * bugfix: fixed listing settings from a root-level node 48 | * bugfix: fixed "poni control" dependency ops being added multiple times (fixes 49 | huge memory usage when running complex operations over tens of servers) 50 | 51 | Version 0.5.0 52 | ============= 53 | :release date: 2011-07-12 54 | 55 | * added ``add_file()`` support for ``dest_bucket`` argument: allows rendering 56 | templates into buckets instead of deploying them to target file paths 57 | * bugfix: add extra lib paths to sys.path only once 58 | * bugfix: full stdout/stderr is now read from remote ssh commands 59 | * bugfix: hierarchical settings overrides 60 | 61 | Version 0.4.9 62 | ============= 63 | :release date: 2011-04-05 64 | 65 | * minor fixes 66 | 67 | Version 0.4.8 68 | ============= 69 | :release date: 2011-04-03 70 | 71 | * made "control" operation errors stand out better by color highlighting 72 | * print remote operation tag lines in one piece to make output cleaner in 73 | multi-threaded ops 74 | * bugfix: load plugins to separate modules 75 | * added explicit deployment handling for ENOENT files 76 | * ``deploy`` command reports status at the end 77 | * added ``$record()`` method for templates 78 | * added configrable ssh connect timeout property: ``ssh-timeout:int=N`` 79 | * bugfix: skip trying file deployment if checking existing file fails 80 | * tuned down unnecessary paramiko ERROR level logging 81 | * added full_path property to ``Item``; renamed ``Config.full_name`` to 82 | ``full_path`` 83 | * bugfix: verify/audit final file count 84 | 85 | Version 0.4.7 86 | ============= 87 | :release date: 2011-03-20 88 | 89 | * bugfix: audit reports number of errors properly 90 | * added Node.addr(network) method for getting the node address for a given 91 | network 92 | * added ``--quiet`` and ``--output-dir`` args for remote ops 93 | * bugfix: logging bug in update-config when used with --verbose flag 94 | 95 | Version 0.4.6 96 | ============= 97 | :release date: 2011-03-09 98 | 99 | * added summary output line for ``audit`` command 100 | * prefix remote command output lines with [nodename] 101 | * added ``Config.full_name`` property 102 | * hashable Node, System and Config 103 | * bugfix: paramiko no output warning timeout fix 104 | 105 | Version 0.4.5 106 | ============= 107 | :release date: 2011-02-22 108 | 109 | * bugfix: added set_combine_stderr(True) for paramiko, tuned rx loop 110 | 111 | Version 0.4.4 112 | ============= 113 | :release date: 2011-02-20 114 | 115 | * operations can be timed (``poni --clock`` or ``control --clock-tasks``) and 116 | results stored in a log file (``--time-log FILE``), reports printed out with 117 | ``poni --time-log FILE report`` 118 | * added warning messages for ``control`` tasks that do not send output in a 119 | long while, kill jobs after five minutes of inactivity 120 | * added support for ``optional_requires`` task dependencies, such tasks are not 121 | required to exist but are guaranteed to be run before the dependent task 122 | * ``deploy`` post-process actions are run even if file is unchanged 123 | * bugfix: paramiko ssh connection error is reported neatly 124 | * bugfix: various small fixes 125 | 126 | Version 0.4.3 127 | ============= 128 | :release date: 2011-01-25 129 | 130 | * control tasks can be run without dependent tasks with ``--no-deps`` 131 | * bugfix: control tasks with in ``script`` files 132 | * bugfix: Tool.execute() exit code check 133 | 134 | Version 0.4.2 135 | ============= 136 | :release date: 2011-01-23 137 | 138 | * added ``poni add-library`` command for specifying directories (that can be 139 | within Poni configs) which are added to the Python ``sys.path`` accessible 140 | by Poni plugins 141 | * requires GitPython>=0.3.1 and Cheetah>=2.4.2 142 | 143 | Version 0.4.1 144 | ============= 145 | :release date: 2011-01-11 146 | 147 | * bugfix: ``remote exec`` process exit code is now properly checked 148 | * better error messages for failed ``poni control`` commands 149 | 150 | Version 0.4 151 | =========== 152 | :release date: 2011-01-10 153 | 154 | * ``poni control`` command dependencies using ``provides=["foo"]`` and 155 | ``requires=["foo"]`` 156 | * parallel execution of "control" commands, runs max one concurrent task per 157 | host and obeys control command dependencies 158 | * ``remote.execute()`` and ``remote.shell()`` support ``verbose=True/False`` 159 | keyword arg 160 | * updated puppet example to install everything using control commands 161 | * new ``template:bool`` system/node property for disabling control commands 162 | and config template verification for template nodes 163 | * limiting concurrent ``poni control`` tasks with ``--jobs=N`` 164 | 165 | Version 0.3.1 166 | ============= 167 | :release date: 2010-12-26 168 | 169 | * added ``poni require`` command that can be used to specify minimum poni 170 | version required by a script, e.g. ``poni require 'poni_version >= "0.3.1"'`` 171 | 172 | Version 0.3 173 | =========== 174 | :release date: 2010-12-25 175 | 176 | * Poni is now in Python Package Index: http://pypi.python.org/pypi/poni and 177 | easy_installable 178 | * syntax change for setting properties, new syntax: 179 | ``poni set NODE PROPERTY[:MOD1[:MODN[...]]]=VALUE`` 180 | * allow multiple conversions using the ``set`` command, e.g. 181 | ``poni set linux/ private.ip:prop:ipv4=node.host`` will get the ``node.host`` 182 | value, resolve it to an ipv4 address and store it to ``private.ip`` 183 | (see http://melor.github.com/poni/modify.html#chaining-conversions) 184 | * setting properties supports UUIDs, resolving ipv4 and ipv6 addresses, 185 | decoding/encoding using Python codecs, JSON encoding/decoding, SI and IEEE 186 | multiplier suffies (e.g. ``10M`` or ``100Kib``) for numbers 187 | * basic support for custom ``poni control`` commands defined in config 188 | plugins (see e.g. ``examples/puppet/puppet-agent/plugin.py``) 189 | * documented functions and variables that are available in templates: 190 | http://melor.github.com/poni/template-variables.html 191 | * ``poni deploy/audit --path-prefix=/foo/bar`` now creates sub-directories for 192 | each node to prevent conflicts when deploying files from multiple nodes to 193 | the same directory 194 | * Genshi XML-based template support using ``self.render_genshi_xml`` in 195 | plugins 196 | * ``find_config(PATTERN)`` is available in templates, yields matching configs 197 | and their nodes 198 | * added ``poni version`` command, displays the poni version number 199 | 200 | Version 0.2 201 | =========== 202 | :release date: 2010-12-09 203 | 204 | * added heaps of docs: http://melor.github.com/poni/ 205 | * colored output 206 | * ``poni show --diff`` displays differences between unrendered templates and 207 | fully rendered templates in diff format 208 | * plugin-objects are visible to templates as ``$plugin`` 209 | * config settings are now visible to templates as ``$settings``, deprecated 210 | ``$s`` 211 | * ``poni show --raw`` displays raw, unrendered templates 212 | * ``poni --color=on/off/auto`` controls colored output 213 | * ``poni settings list`` lists config settings and their values 214 | * ``poni settings set`` sets config settings 215 | * ``poni list --line-per-prop`` displays each property on a separate line 216 | * ``poni verify -v`` shows status for each file 217 | * ``poni list`` arguments ``-n`` (show nodes, default), ``-s`` (show systems) 218 | and ``-c`` (show configs) 219 | * the top-level config is available to templates as ``$config`` 220 | * renamed version-control commands: ``commit`` is now ``checkpoint`` and 221 | ``status`` is now ``diff`` 222 | * added ``--full-match``, ``-M`` to many commands, requires full pattern 223 | match (e.g. with node names) instead of partial match (which is default) 224 | * ssh connections are retried on failure 225 | * ``poni import DEBFILE`` pulls poni configs from a Debian DEB package 226 | * ``poni cloud wait --state=STATE`` waits until node reaches specified running 227 | state 228 | * deployment: specifying ``dest_path`` ending in a backslash will use the 229 | source filename as the deployed filename 230 | * ``poni deploy`` creates all directory levels when deploying a file 231 | * config settings are inherited/loaded from parent configs 232 | * ``poni add-node`` supports ``--copy-props`` (used with ``--inherit NODE``), 233 | copies all node properties from the source node 234 | * parent node's inherited configs are properly collected and used in deployment 235 | * basic repository version-control support with Git using ``poni vc init``, 236 | ``poni vc checkpoint MSG`` and ``poni vc diff`` commands 237 | * ``poni add-config --copy-dir=DIR`` copies config templates, plugins, etc. 238 | from the given directory 239 | 240 | Version 0.1 241 | =========== 242 | :release date: 2010-11-28 243 | 244 | * Initial version with basic deployment support 245 | -------------------------------------------------------------------------------- /doc/ec2.rst: -------------------------------------------------------------------------------- 1 | Cloud Provisioning with Amazon EC2 2 | ================================== 3 | Poni can manage provisioning your VMs from Amazon's EC2 for you. There is a little 4 | setup involved, which is covered in the next section. 5 | 6 | Pre-requisites 7 | -------------- 8 | * Basic EC2 knowledge (AMIs, key-pairs, security groups, etc.) 9 | * `Amazon EC2`_ account (amazingly) 10 | * Key-pair for each of the data-centers (e.g. ``us-west-1``, ``eu-west-1``, etc.) you 11 | will be using 12 | * EC2 access key and secret key 13 | 14 | Your EC2 credentials need to be stored in the following environment variables:: 15 | 16 | export AWS_ACCESS_KEY_ID= 17 | export AWS_SECRET_ACCESS_KEY= 18 | 19 | The key-pairs need to be copied to:: 20 | 21 | $HOME/.ssh/.pem 22 | 23 | Limitations: 24 | 25 | * Currently the "default" security group is applied to all VMs, this group should be 26 | configured to allow SSH (port 22) access for remote deployment/control 27 | * Many instance properties are not configurable yet 28 | 29 | Configuring Nodes 30 | ----------------- 31 | At minimum, the following node properties need to be set: 32 | 33 | ``cloud.provider`` 34 | Must be ``aws-ec2``. 35 | 36 | ``cloud.region`` 37 | Data-center region code, one of ``us-west-1``, ``us-east-1``, ``eu-west-1`` or 38 | ``ap-southeast-1`` 39 | 40 | ``cloud.image`` 41 | The AMI image id of the image you want to instantiate. 42 | 43 | ``cloud.key-pair`` 44 | Key-pair name **without** the ``.pem`` -suffix. **NOTE:** key-pairs are 45 | region-specific and will not work cross the data-centers. 46 | 47 | ``cloud.vm_name`` 48 | Unique name for the VM. Provides the VM with a friendly name visible in 49 | the EC2 console. 50 | 51 | ``user`` 52 | Username used to login to the system. 53 | 54 | Creating a repository and a node from scratch:: 55 | 56 | $ poni init 57 | $ poni add-node drumbo1 58 | $ poni set drumbo cloud.provider=aws-ec2 cloud.region=us-east-1 cloud.image=ami-daf615b3 cloud.key-pair=my-keypair user=root 59 | $ poni set drumbo1 cloud.vm_name=drumbo1 60 | $ poni set drumbo2 cloud.vm_name=drumbo2 61 | $ poni set drumbo3 cloud.vm_name=drumbo3 62 | $ poni set drumbo4 cloud.vm_name=drumbo4 63 | 64 | In order to see the cloud properties you can use ``list -o``:: 65 | 66 | $ poni list -o 67 | node drumbo1 68 | cloud image:'ami-daf615b3' key-pair:'my-keypair' provider:'aws-ec2' region:'us-east-1' 69 | 70 | While at it, why not create a few more instances:: 71 | 72 | $ poni add-node "drumbo{id}" -n 2..4 -c -i drumbo1 73 | $ poni list -o 74 | node drumbo1 75 | cloud image:'ami-daf615b3' key-pair:'my-keypair' provider:'aws-ec2' region:'us-east-1' vm_name:'drumbo1' 76 | node drumbo2 <= drumbo1 77 | cloud image:'ami-daf615b3' key-pair:'my-keypair' provider:'aws-ec2' region:'us-east-1' vm_name:'drumbo2' 78 | node drumbo3 <= drumbo1 79 | cloud image:'ami-daf615b3' key-pair:'my-keypair' provider:'aws-ec2' region:'us-east-1' vm_name:'drumbo3' 80 | node drumbo4 <= drumbo1 81 | cloud image:'ami-daf615b3' key-pair:'my-keypair' provider:'aws-ec2' region:'us-east-1' vm_name:'drumbo4' 82 | 83 | Option ``-i drumbo1`` means that the new nodes should be inherited from ``drumbo1`` and 84 | ``-c`` copies all node properties from the parent node while at it. 85 | 86 | The ``<= drumbo1`` part in the output above tells us that the node is inherited from 87 | ``drumbo1``. 88 | 89 | Now the nodes are ready to be started. 90 | 91 | Starting Nodes 92 | -------------- 93 | Creating node instances is done with the ``init`` -command:: 94 | 95 | $ poni cloud init drumbo --wait 96 | poni INFO drumbo1: initialized: key-pair=u'my-keypair', region=u'us-east-1', instance=u'i-4c92cd21', image=u'ami-daf615b3', provider=u'aws-ec2', vm_name=u'drumbo1' 97 | poni INFO drumbo2: initialized: key-pair=u'my-keypair', region=u'us-east-1', instance=u'i-4e92cd23', image=u'ami-daf615b3', provider=u'aws-ec2', vm_name=u'drumbo2' 98 | poni INFO drumbo3: initialized: key-pair=u'my-keypair', region=u'us-east-1', instance=u'i-4692cd2b', image=u'ami-daf615b3', provider=u'aws-ec2', vm_name=u'drumbo3' 99 | poni INFO drumbo4: initialized: key-pair=u'my-keypair', region=u'us-east-1', instance=u'i-5c92cd31', image=u'ami-daf615b3', provider=u'aws-ec2', vm_name=u'drumbo4' 100 | aws-ec2 INFO [0/4] instances 'running', waiting... 101 | aws-ec2 INFO [0/4] instances 'running', waiting... 102 | aws-ec2 INFO [0/4] instances 'running', waiting... 103 | aws-ec2 INFO [0/4] instances 'running', waiting... 104 | aws-ec2 INFO [0/4] instances 'running', waiting... 105 | aws-ec2 INFO [1/4] instances 'running', waiting... 106 | poni INFO drumbo1: updated: host=u'ec2-50-16-65-176.compute-1.amazonaws.com' (from u''), private={'ip': u'10.253.202.50', 'dns': u'domU-12-31-38-01-C5-C4.compute-1.internal'} (from None) 107 | poni INFO drumbo2: updated: host=u'ec2-184-72-214-101.compute-1.amazonaws.com' (from u''), private={'ip': u'10.206.237.206', 'dns': u'domU-12-31-39-14-EE-24.compute-1.internal'} (from None) 108 | poni INFO drumbo3: updated: host=u'ec2-184-73-110-99.compute-1.amazonaws.com' (from u''), private={'ip': u'10.122.251.180', 'dns': u'ip-10-122-251-180.ec2.internal'} (from None) 109 | poni INFO drumbo4: updated: host=u'ec2-184-72-156-215.compute-1.amazonaws.com' (from u''), private={'ip': u'10.206.239.167', 'dns': u'domU-12-31-39-14-EC-59.compute-1.internal'} (from None) 110 | 111 | First, each node is instantiated and they get their unique ``cloud.instance`` id, e.g. 112 | ``i-4c92cd21`` above. 113 | 114 | Then Poni polls each instance's status until they are running. This behavior is 115 | requested with the ``--wait`` -option. 116 | 117 | Finally, when every instance is running, Poni updates the nodes' properties into the Poni 118 | repository. 119 | 120 | Now the cloud properties include the ``instance`` value:: 121 | 122 | $ poni list -o 123 | node drumbo1 124 | cloud image:'ami-daf615b3' instance:'i-4c92cd21' key-pair:'my-keypair' provider:'aws-ec2' region:'us-east-1' vm_name:'drumbo1' 125 | node drumbo2 126 | cloud image:'ami-daf615b3' instance:'i-4e92cd23' key-pair:'my-keypair' provider:'aws-ec2' region:'us-east-1' vm_name:'drumbo2' 127 | node drumbo3 128 | cloud image:'ami-daf615b3' instance:'i-4692cd2b' key-pair:'my-keypair' provider:'aws-ec2' region:'us-east-1' vm_name:'drumbo3' 129 | node drumbo4 130 | cloud image:'ami-daf615b3' instance:'i-5c92cd31' key-pair:'my-keypair' provider:'aws-ec2' region:'us-east-1' vm_name:'drumbo4' 131 | 132 | Also the node address information is updated to the node properties:: 133 | 134 | $ poni list -p 135 | node drumbo1 136 | prop depth:1 host:'ec2-50-16-65-176.compute-1.amazonaws.com' index:0 private:{dns:'domU-12-31-38-01-C5-C4.compute-1.internal' ip:'10.253.202.50'} 137 | node drumbo2 138 | prop depth:1 host:'ec2-184-72-214-101.compute-1.amazonaws.com' index:1 parent:'drumbo1' private:{dns:'domU-12-31-39-14-EE-24.compute-1.internal' ip:'10.206.237.206'} 139 | node drumbo3 140 | prop depth:1 host:'ec2-184-73-110-99.compute-1.amazonaws.com' index:2 parent:'drumbo1' private:{dns:'ip-10-122-251-180.ec2.internal' ip:'10.122.251.180'} 141 | node drumbo4 142 | prop depth:1 host:'ec2-184-72-156-215.compute-1.amazonaws.com' index:3 parent:'drumbo1' private:{dns:'domU-12-31-39-14-EC-59.compute-1.internal' ip:'10.206.239.167'} 143 | 144 | The following properties are updated: 145 | 146 | ``host`` 147 | Full public internet DNS name 148 | ``private.dns`` 149 | Full internal EC2 network hostname 150 | ``private.ip`` 151 | Internal EC2 network IP-address 152 | 153 | If the instance properties need to be updated later, the ``cloud update`` command can be 154 | used. This can be done for example if instances have been initialized without the 155 | ``--wait`` -option, which does not update node address properties. 156 | 157 | 158 | Assigning an elastic ip to an instance 159 | --------------------------------------- 160 | 161 | You can assign elastic ip's to a running instance using the ``cloud ip`` command. This command 162 | uses the ``cloud.eip`` property value and assigns it to the instance. 163 | 164 | $ poni set drumbo1 cloud.eip=xxx.xxx.xxx.xxx 165 | $ poni cloud ip drumbo1 166 | 167 | Checking Instance Status 168 | ------------------------ 169 | The ``list -q`` queries each cloud instances' status and shows it in the output:: 170 | 171 | $ poni list -q 172 | node drumbo1 173 | status running 174 | node drumbo2 175 | status running 176 | node drumbo3 177 | status running 178 | node drumbo4 179 | status running 180 | 181 | Terminating Instances 182 | --------------------- 183 | To get rid of instances use the ``cloud terminate`` command:: 184 | 185 | $ poni cloud terminate drumbo 186 | poni INFO terminated: drumbo1 187 | poni INFO terminated: drumbo2 188 | poni INFO terminated: drumbo3 189 | poni INFO terminated: drumbo4 190 | poni INFO 4 instances terminated 191 | 192 | The nodes are not actually terminated yet, but are 'shutting-down', which gives us a nice 193 | excuse to try the ``cloud wait`` command:: 194 | 195 | $ poni cloud wait drumbo --state=terminated 196 | aws-ec2 INFO [0/4] instances 'terminated', waiting... 197 | aws-ec2 INFO [0/4] instances 'terminated', waiting... 198 | aws-ec2 INFO [3/4] instances 'terminated', waiting... 199 | aws-ec2 INFO [3/4] instances 'terminated', waiting... 200 | poni INFO drumbo1: updated: host='' (from u'ec2-50-16-65-176.compute-1.amazonaws.com'), private={'ip': None, 'dns': ''} (from {u'ip': u'10.253.202.50', u'dns': u'domU-12-31-38-01-C5-C4.compute-1.internal'}) 201 | poni INFO drumbo2: updated: host='' (from u'ec2-184-72-214-101.compute-1.amazonaws.com'), private={'ip': None, 'dns': ''} (from {u'ip': u'10.206.237.206', u'dns': u'domU-12-31-39-14-EE-24.compute-1.internal'}) 202 | poni INFO drumbo3: updated: host='' (from u'ec2-184-73-110-99.compute-1.amazonaws.com'), private={'ip': None, 'dns': ''} (from {u'ip': u'10.122.251.180', u'dns': u'ip-10-122-251-180.ec2.internal'}) 203 | poni INFO drumbo4: updated: host='' (from u'ec2-184-72-156-215.compute-1.amazonaws.com'), private={'ip': None, 'dns': ''} (from {u'ip': u'10.206.239.167', u'dns': u'domU-12-31-39-14-EC-59.compute-1.internal'}) 204 | 205 | ``cloud wait`` polls each target nodes' status until all of them reach the given 206 | ``terminated`` status. It also empties node address properties once finished waiting. 207 | 208 | Verifying that the instances are dead:: 209 | 210 | $ poni list -q 211 | node drumbo1 212 | status terminated 213 | node drumbo2 214 | status terminated 215 | node drumbo3 216 | status terminated 217 | node drumbo4 218 | status terminated 219 | 220 | .. include:: definitions.rst 221 | -------------------------------------------------------------------------------- /doc/getting-started.rst: -------------------------------------------------------------------------------- 1 | Getting Started with Poni 2 | ========================= 3 | Poni is a command-line tool with many different sub-commands for manipulating the 4 | Poni repository and the system nodes. 5 | 6 | Let's see the command-line help:: 7 | 8 | $ poni -h 9 | usage: poni [-h] [-D] [-d DIR] 10 | 11 | {audit,set,remote,script,verify,add-system,list,init,vc,add-config,show,import,add-node,deploy,cloud} 12 | ... 13 | 14 | positional arguments: 15 | {audit,set,remote,script,verify,add-system,list,init,vc,add-config,show,import,add-node,deploy,cloud} 16 | list list systems and nodes 17 | add-system add a sub-system 18 | init init repository 19 | import import nodes/configs 20 | script run commands from a script file 21 | add-config add a config to node(s) 22 | set set system/node properties 23 | show render and show node config files 24 | deploy deploy node configs 25 | audit audit active node configs 26 | verify verify local node configs 27 | add-node add a new node 28 | cloud cloud operations 29 | remote remote operations 30 | vc version-control operations 31 | 32 | optional arguments: 33 | -h, --help show this help message and exit 34 | -D, --debug enable debug output 35 | -d DIR, --root-dir DIR 36 | repository root directory (default: $HOME/.poni/default) 37 | 38 | 39 | The default Poni repository directory, unless specified by the ``PONI_ROOT`` environment 40 | variable or by the ``--root-dir DIR`` switch, is ``$HOME/.poni/default``. 41 | 42 | Command-specific help can be viewed by executing ``poni -h``, for example:: 43 | 44 | $ poni add-system -h 45 | usage: poni add-system [-h] system 46 | 47 | positional arguments: 48 | system system name 49 | 50 | optional arguments: 51 | -h, --help show this help message and exit 52 | 53 | Creating the Poni Repository 54 | ---------------------------- 55 | First, we will need to initialize the repository using the ``init`` command:: 56 | 57 | $ poni init 58 | 59 | There is no output if the command is successful. Most Poni commands will not output much 60 | if there are no errors. 61 | 62 | Adding Nodes 63 | ------------ 64 | A "system" in the Poni context refers to a collection of nodes and/or sub-systems. 65 | 66 | Let's say we are defining a system ``webshop`` with four HTTP frontend servers and a 67 | single backend SQL database server. We will divided the ``webshop`` system into two 68 | sub-systems ``frontend`` and ``backend``. 69 | 70 | First, we will create the systems:: 71 | 72 | $ poni add-system webshop 73 | $ poni add-system webshop/frontend 74 | $ poni add-system webshop/backend 75 | 76 | Again, no output since everything went ok. 77 | 78 | Next, we will add the backend SQL database ``postgres1`` server node into the 79 | ``backend`` system:: 80 | 81 | $ poni add-node webshop/backend/postgres1 82 | 83 | Let's see how the system looks now:: 84 | 85 | $ poni list 86 | node webshop/backend/postgres1 87 | 88 | The ``list`` command shows only nodes by default. Let's also view the systems:: 89 | 90 | $ poni list -sn 91 | system webshop 92 | system webshop/backend 93 | node webshop/backend/postgres1 94 | system webshop/frontend 95 | 96 | The left column tells the type of the item shown on the right. 97 | 98 | The four HTTP frontend nodes can be added with a single command using the ``-n COUNT`` 99 | option:: 100 | 101 | 102 | $ poni add-node "webshop/frontend/http{id}" -n 4 103 | $ poni list 104 | node webshop/backend/postgres1 105 | node webshop/frontend/http1 106 | node webshop/frontend/http2 107 | node webshop/frontend/http3 108 | node webshop/frontend/http4 109 | 110 | ``-n 4`` tells Poni that four nodes are to be added. Value ranges can also be given, for 111 | example ``-n 5..8`` will create nodes 5, 6, 7 and 8. 112 | 113 | The ``{id}`` in the node name gets replaced with the node number. Any normal Python 114 | ``string.format()`` formatting codes can be used, too. For example, if you wanted two 115 | digits then ``http{id:02}`` would do the job. 116 | 117 | Adding Configs 118 | -------------- 119 | A Poni "config" is a configurable item, often a piece of software, than can be added to 120 | a node. A config often contains multiple configuration file templates and a bunch of 121 | settings that will be used in the final configuration files deployed to the nodes. Each 122 | node can have multiple configs applied to them. 123 | 124 | Our example DB backend uses PostgreSQL 8.4 as the database so we will call it ``pg84``. 125 | We can create the config and view it using the ``-c`` option:: 126 | 127 | $ poni add-config postg pg84 128 | $ poni list -nc 129 | node webshop/backend/postgres1 130 | config webshop/backend/postgres1/pg84 131 | node webshop/frontend/http1 132 | node webshop/frontend/http2 133 | node webshop/frontend/http3 134 | node webshop/frontend/http4 135 | 136 | Above we were a bit lazy and only wrote ``postg`` above as the target node. 137 | 138 | .. note:: 139 | Poni system/node/config arguments are evaluated as regular expressions and will match 140 | as long as the given pattern appears somewhere in the full name of the target. If there 141 | are multiple hits, the command will be executed for each of them. Stricter full name 142 | matching can be enabled by adding the ``-M`` option. 143 | 144 | We want to deploy a file describing the DB access permissions named ``pg_hba.conf`` to 145 | the backend node. Use an editor to create a file named ``pg_hba.conf`` with the following contents:: 146 | 147 | # This the pg_hba.conf for $node.name 148 | # 149 | # TYPE DATABASE USER ADDRESS METHOD 150 | local all all trust 151 | 152 | Every Poni config needs a ``plugin.py`` file that tells Poni what files need to be 153 | installed and where. Use an editor to create the file with the following contents:: 154 | 155 | from poni import config 156 | 157 | class PlugIn(config.PlugIn): 158 | def add_actions(self): 159 | self.add_file("pg_hba.conf", dest_path="/etc/postgres/8.4/") 160 | 161 | The above plugin will install a single file ``pg_hba.conf`` into the directory 162 | ``/etc/postgres/8.4/``. 163 | 164 | Now the files can be added into the existing ``pg84`` config:: 165 | 166 | $ poni update-config pg84 plugin.py pg_hba.conf -v 167 | poni INFO webshop/backend/postgres1/pg84: added 'plugin.py' 168 | poni INFO webshop/backend/postgres1/pg84: added 'pg_hba.conf' 169 | 170 | Now the database node is setup and we can move on to verifying and deployment... 171 | 172 | Verifying Configs 173 | ----------------- 174 | Checking that there are no problems rendering any of the configs can be done with the 175 | ``verify`` command:: 176 | 177 | $ poni verify 178 | poni INFO all [1] files ok 179 | 180 | No errors reported, good. Let's see how our ``pg_hba.conf`` looks like:: 181 | 182 | $ poni show 183 | --- BEGIN webshop/backend/postgres1: dest=/etc/postgres/8.4/pg_hba.conf --- 184 | # This is the pg_hba.conf for webshop/backend/postgres1 185 | # 186 | # TYPE DATABASE USER ADDRESS METHOD 187 | local all all trust 188 | 189 | --- END webshop/backend/postgres1: dest=/etc/postgres/8.4/pg_hba.conf --- 190 | 191 | Note that the ``$node.name`` template directive got replaced with the name (full path) 192 | of the node. 193 | 194 | Deploying 195 | --------- 196 | In order to be able to deploy, Poni needs to know the hostnames of each nodes involved. 197 | For this exercise we'll deploy the files locally instead of copying them over the 198 | network. By default Poni attempts an SSH based deployment:: 199 | 200 | $ poni deploy postgres1 201 | poni ERROR RemoteError: webshop/backend/postgres1: 'host' property not set poni ERROR VerifyError: failed: there were 1 errors 202 | 203 | 204 | Node and system properties can be adjusted with the ``set`` command. We'll set a special 205 | property ``deploy`` to the value ``local`` that tells Poni to install the files to the 206 | local file-system:: 207 | 208 | $ poni set postgres1 deploy=local 209 | $ poni list postgres1 -p 210 | node webshop/backend/postgres1 211 | prop deploy:'local' depth:3 host:'' index:0 212 | 213 | The ``list`` option ``-p`` shows node and system properties. In addition to ``host`` there 214 | are a couple of automatically set properties ``depth`` (how deep is the node in the 215 | system hierarchy) and ``index`` (tells the location of the node within its sub-system). 216 | 217 | Now deployment can be completed and we'll override the target directory for this exercise 218 | using the ``--path-prefix`` argument, which makes it possible to install all the template files from multiple nodes under a single directory. Sub-directories are added automatically for each system and node level to prevent files from different nodes colliding:: 219 | 220 | $ poni deploy postgres1 --path-prefix=/tmp 221 | manager INFO WROTE webshop/backend/postgres1: /tmp/webshop/backend/postgres1/etc/postgres/8.4/pg_hba.conf 222 | 223 | $ cat /tmp/webshop/backend/postgres1/etc/postgres/8.4/pg_hba.conf 224 | # This the pg_hba.conf for webshop/backend/postgres1 225 | # TYPE DATABASE USER ADDRESS METHOD 226 | local all all trust 227 | 228 | Auditing 229 | -------- 230 | Checking that the deployed configuration is still up-to-date and intact is simple:: 231 | 232 | $ poni audit -v --path-prefix=/tmp 233 | manager INFO OK webshop/backend/postgres1: /tmp/webshop/backend/postgres1/etc/postgres/8.4/pg_hba.conf 234 | 235 | Let's see what happens if the file is changed:: 236 | 237 | $ echo hello >> /tmp/webshop/backend/postgres1/etc/postgres/8.4/pg_hba.conf 238 | $ poni audit -v --path-prefix=/tmp 239 | manager WARNING DIFFERS webshop/backend/postgres1: /tmp/webshop/backend/postgres1/etc/postgres/8.4/pg_hba.conf 240 | 241 | The difference to the proper contents can be viewed by adding the ``--diff`` argument:: 242 | 243 | $ poni audit -v --path-prefix=/tmp --diff 244 | manager WARNING DIFFERS webshop/backend/postgres1: /tmp/webshop/backend/postgres1/etc/postgres/8.4/pg_hba.conf 245 | --- config 246 | +++ active 2011-03-02 19:38:41 247 | @@ -2,3 +2,4 @@ 248 | # TYPE DATABASE USER ADDRESS METHOD 249 | local all all trust 250 | 251 | +hello 252 | 253 | To repair the file, simply run the ``deploy`` command again:: 254 | 255 | $ poni deploy postgres1 --path-prefix=/tmp 256 | manager INFO WROTE webshop/backend/postgres1: /tmp/webshop/backend/postgres1/etc/postgres/8.4/pg_hba.conf 257 | $ poni audit -v --path-prefix=/tmp --diff 258 | manager INFO OK webshop/backend/postgres1: /tmp/webshop/backend/postgres1/etc/postgres/8.4/pg_hba.conf 259 | --------------------------------------------------------------------------------