├── .github └── workflows │ └── ci.yaml ├── .gitignore ├── .tsuruignore ├── AUTHORS ├── LICENSE ├── Makefile ├── Procfile ├── Procfile.local ├── README.rst ├── etc ├── consul.conf ├── location_template.conf └── userdata.sh ├── manifest.yaml ├── requirements.apt ├── requirements.txt ├── rpaas ├── __init__.py ├── acl.py ├── admin_api.py ├── admin_plugin.py ├── api.py ├── auth.py ├── celery_sentinel.py ├── consul_manager.py ├── flavor.py ├── flower_uimodules.py ├── hc.py ├── healing.py ├── lock.py ├── manager.py ├── misc.py ├── nginx.py ├── plan.py ├── plugin.py ├── router_api.py ├── scheduler.py ├── session_resumption.py ├── ssl_plugins │ ├── __init__.py │ ├── default.py │ ├── le.py │ ├── le_authenticator.py │ └── le_renewer.py ├── sslutils.py ├── storage.py └── tasks.py ├── runner.sh ├── setup.py └── tests ├── __init__.py ├── managers.py ├── test_acl.py ├── test_admin_api.py ├── test_admin_plugin.py ├── test_api.py ├── test_auth.py ├── test_consul_manager.py ├── test_hc.py ├── test_healing.py ├── test_le_renewer.py ├── test_lock.py ├── test_manager.py ├── test_nginx.py ├── test_plugin.py ├── test_router_api.py ├── test_session_resumption.py ├── test_ssl_le.py ├── test_storage.py ├── test_tasks.py └── testdata ├── block_http ├── location ├── lua_module └── sentinel_conf ├── redis_sentinel2_test.conf ├── redis_sentinel_test.conf ├── redis_test.conf ├── redis_test2.conf └── redis_test3.conf /.github/workflows/ci.yaml: -------------------------------------------------------------------------------- 1 | name: ci 2 | on: [push, pull_request] 3 | jobs: 4 | test: 5 | runs-on: ubuntu-18.04 6 | services: 7 | mongodb: 8 | image: mongo:4 9 | ports: 10 | - 27017:27017 11 | steps: 12 | - uses: actions/checkout@v2 13 | - name: Set up Python 2.7 14 | uses: actions/setup-python@v2 15 | with: 16 | python-version: 2.7 17 | - uses: shogo82148/actions-setup-redis@v1 18 | with: 19 | redis-version: '4.x' 20 | - name: Install dependencies 21 | run: | 22 | sudo apt-get update && sudo -E apt-get install -y unzip 23 | pip install -U pip 24 | make deps 25 | curl -k -LO https://releases.hashicorp.com/consul/0.6.4/consul_0.6.4_linux_amd64.zip 26 | unzip consul_0.6.4_linux_amd64.zip 27 | export GOMAXPROCS=8 PATH=$PATH:$PWD 28 | make start-consul 29 | - name: Run test 30 | run: make ci-test 31 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | build/ 2 | dist/ 3 | *.pyc 4 | *.egg-info/ 5 | *.env 6 | .coverage 7 | dump.rdb 8 | .python-version 9 | -------------------------------------------------------------------------------- /.tsuruignore: -------------------------------------------------------------------------------- 1 | .git 2 | build/ 3 | dist/ 4 | *.pyc 5 | *.egg-info/ 6 | *.env 7 | .coverage 8 | dump.rdb 9 | tests 10 | -------------------------------------------------------------------------------- /AUTHORS: -------------------------------------------------------------------------------- 1 | # This is the official list of rpaas authors for copyright purposes. 2 | 3 | Andrews Medina 4 | Cezar Sá Espinola 5 | Diogo Munaro Vieira 6 | Flavia Missi 7 | Francisco Souza 8 | Paulo Sousa 9 | Vicente Fiebig 10 | Wagner Souza 11 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2015, rpaas authors. All rights reserved. 2 | 3 | Redistribution and use in source and binary forms, with or without 4 | modification, are permitted provided that the following conditions are met: 5 | 6 | * Redistributions of source code must retain the above copyright 7 | notice, this list of conditions and the following disclaimer. 8 | * Redistributions in binary form must reproduce the above copyright 9 | notice, this list of conditions and the following disclaimer in the 10 | documentation and/or other materials provided with the distribution. 11 | * Neither the name of the Globo.com nor the names of its contributors 12 | may be used to endorse or promote products derived from this software without 13 | specific prior written permission. 14 | 15 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 16 | "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 17 | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 18 | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 19 | OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 20 | SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 21 | LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 22 | DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 23 | THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 24 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 25 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 26 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | # Copyright 2016 rpaas authors. All rights reserved. 2 | # Use of this source code is governed by a BSD-style 3 | # license that can be found in the LICENSE file. 4 | 5 | .PHONY: test deps 6 | 7 | clean_pycs: 8 | find . -name \*.pyc -delete 9 | 10 | run: deps 11 | python ./rpaas/api.py 12 | 13 | worker: deps 14 | celery -A rpaas.tasks worker 15 | 16 | flower: deps 17 | celery flower -A rpaas.tasks 18 | 19 | start-consul: stop-consul 20 | consul agent -ui -server -bind 127.0.0.1 -bootstrap-expect 1 -data-dir /tmp/consul -config-file etc/consul.conf -node=rpaas-test & 21 | while ! consul info; do sleep 1; done 22 | 23 | stop-consul: 24 | -consul leave 25 | rm -rf /tmp/consul 26 | 27 | test: start-consul ci-test 28 | 29 | ci-test: clean_pycs deps redis-sentinel-test 30 | @python -m unittest discover 31 | @flake8 --max-line-length=120 . 32 | 33 | deps: 34 | pip install -e .[tests] 35 | 36 | coverage: deps 37 | rm -f .coverage 38 | coverage run --source=. -m unittest discover 39 | coverage report -m --omit=test\*,run\*.py 40 | 41 | kill-redis-sentinel-test: 42 | -redis-cli -a mypass -p 51115 shutdown 43 | -redis-cli -a mypass -p 51114 shutdown 44 | -redis-cli -a mypass -p 51113 shutdown 45 | -redis-cli -p 51112 shutdown 46 | -redis-cli -p 51111 shutdown 47 | 48 | redis-sentinel-test: kill-redis-sentinel-test copy-redis-conf 49 | redis-sentinel /tmp/redis_sentinel_test.conf --daemonize yes || redis-server /tmp/redis_sentinel_test.conf --sentinel --daemonize yes; sleep 1 50 | redis-sentinel /tmp/redis_sentinel2_test.conf --daemonize yes || redis-server /tmp/redis_sentinel2_test.conf --sentinel --daemonize yes; sleep 1 51 | redis-server /tmp/redis_test.conf --daemonize yes; sleep 1 52 | redis-server /tmp/redis_test2.conf --daemonize yes; sleep 1 53 | redis-server /tmp/redis_test3.conf --daemonize yes; sleep 1 54 | redis-cli -p 51111 info | grep sentinel 55 | 56 | copy-redis-conf: 57 | @cp tests/testdata/sentinel_conf/*.conf /tmp 58 | -------------------------------------------------------------------------------- /Procfile: -------------------------------------------------------------------------------- 1 | web: sh runner.sh 2 | -------------------------------------------------------------------------------- /Procfile.local: -------------------------------------------------------------------------------- 1 | web: python ./rpaas/api.py 2 | celery: celery -A rpaas.tasks worker 3 | flower: celery flower -A rpaas.tasks 4 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | Reverse proxy service for tsuru PaaS 2 | ==================================== 3 | 4 | Deprecated in favor of https://github.com/tsuru/rpaas-operator 5 | 6 | .. image:: https://travis-ci.org/tsuru/rpaas.png?branch=master 7 | :target: https://travis-ci.org/tsuru/rpaas 8 | 9 | Deploying the API 10 | ----------------- 11 | 12 | First, let's create an app in tsuru, from the project root, execute the following: 13 | 14 | .. highlight: bash 15 | 16 | :: 17 | 18 | % tsuru app-create rpaas python 19 | % git remote add tsuru git@remote.sbrubles.com 20 | % git push tsuru master 21 | 22 | The push will return an error telling you that you can't push code before the 23 | app unit is up, wait until your unit is in service, you can check with: 24 | 25 | 26 | .. highlight: bash 27 | 28 | :: 29 | 30 | % tsuru app-list 31 | 32 | When you get an output like this you can proceed to push. 33 | 34 | .. highlight: bash 35 | 36 | :: 37 | 38 | +-------------+-------------------------+--------------------------------------+ 39 | | Application | Units State Summary | Address | 40 | +-------------+-------------------------+--------------------------------------+ 41 | | your-app | 1 of 1 units in-service | your-app.somewhere.com | 42 | +-------------+-------------------------+--------------------------------------+ 43 | 44 | Now if you access our app endpoint at "/" (you can check with `tsuru app-info` 45 | cmd) you should get a 404, which is right, since the API does not respond 46 | through this url. 47 | -------------------------------------------------------------------------------- /etc/consul.conf: -------------------------------------------------------------------------------- 1 | { 2 | "acl_master_token": "rpaas-test", 3 | "acl_default_policy": "deny", 4 | "acl_down_policy": "deny", 5 | "acl_datacenter": "dc1" 6 | } 7 | -------------------------------------------------------------------------------- /etc/location_template.conf: -------------------------------------------------------------------------------- 1 | location {path} {{ 2 | proxy_set_header Host {host}; 3 | proxy_set_header X-Real-IP $remote_addr; 4 | proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; 5 | proxy_set_header X-Forwarded-Proto $scheme; 6 | proxy_set_header X-Forwarded-Host $host; 7 | proxy_pass http://{host}:80/; 8 | proxy_redirect ~^http://{host}(:\d+)?/(.*)$ {path}$2; 9 | }} -------------------------------------------------------------------------------- /etc/userdata.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | NGINX_CONF=$(cat < /dev/null 66 | echo "$NGINX_CONF" | sudo tee /etc/nginx/nginx.conf > /dev/null 67 | sudo /usr/sbin/service nginx restart 68 | -------------------------------------------------------------------------------- /manifest.yaml: -------------------------------------------------------------------------------- 1 | id: rpaas 2 | password: abc123 3 | endpoint: 4 | production: rpaas.192.168.50.4.nip.io 5 | -------------------------------------------------------------------------------- /requirements.apt: -------------------------------------------------------------------------------- 1 | python-dev 2 | libevent-dev 3 | libffi-dev 4 | libssl-dev 5 | dialog -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | -e /home/application/current 2 | -------------------------------------------------------------------------------- /rpaas/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright 2016 rpaas authors. All rights reserved. 2 | # Use of this source code is governed by a BSD-style 3 | # license that can be found in the LICENSE file. 4 | 5 | import os 6 | 7 | from rpaas import manager 8 | 9 | _manager = None 10 | 11 | 12 | def get_manager(): 13 | global _manager 14 | if _manager is None: 15 | _manager = manager.Manager(dict(os.environ)) 16 | return _manager 17 | -------------------------------------------------------------------------------- /rpaas/acl.py: -------------------------------------------------------------------------------- 1 | # Copyright 2017 rpaas authors. All rights reserved. 2 | # Use of this source code is governed by a BSD-style 3 | # license that can be found in the LICENSE file. 4 | 5 | import ipaddress 6 | import requests 7 | from networkapiclient import (Ip, Network) 8 | from requests.auth import HTTPBasicAuth 9 | 10 | 11 | class Dumb(object): 12 | 13 | def __init__(self, storage): 14 | self.storage = storage 15 | 16 | def add_acl(self, name, src, dst): 17 | src = str(ipaddress.ip_network(unicode(src))) 18 | self.storage.store_acl_network(name, src, dst) 19 | 20 | def remove_acl(self, name, src): 21 | src = str(ipaddress.ip_network(unicode(src))) 22 | self.storage.remove_acl_network(name, src) 23 | 24 | 25 | class AclApiError(Exception): 26 | pass 27 | 28 | 29 | class AclNotFound(Exception): 30 | pass 31 | 32 | 33 | class AclManager(object): 34 | 35 | def __init__(self, config, storage, lock_manager): 36 | self.storage = storage 37 | self.service_name = config.get("RPAAS_SERVICE_NAME", "rpaas") 38 | self.acl_api_host = config.get("ACL_API_HOST") 39 | self.acl_api_user = config.get("ACL_API_USER") 40 | self.acl_api_password = config.get("ACL_API_PASSWORD") 41 | self.acl_api_timeout = int(config.get("ACL_API_TIMEOUT", 30)) 42 | self.acl_port_range_start = config.get("ACL_PORT_RANGE_START", "30000") 43 | self.acl_port_range_end = config.get("ACL_PORT_RANGE_END", "61000") 44 | self.network_api_url = config.get("NETWORK_API_URL", None) 45 | self.acl_auth_basic = HTTPBasicAuth(self.acl_api_user, self.acl_api_password) 46 | self.lock_manager = lock_manager 47 | self.lock_name = "acl_manager:{}".format(self.service_name) 48 | if self.network_api_url: 49 | self.network_api_username = config.get("NETWORK_API_USERNAME") 50 | self.network_api_password = config.get("NETWORK_API_PASSWORD") 51 | self.ip_client = Ip.Ip(self.network_api_url, self.network_api_username, self.network_api_password) 52 | self.network_client = Network.Network(self.network_api_url, self.network_api_username, 53 | self.network_api_password) 54 | 55 | def add_acl(self, name, src, dst): 56 | src_network = self._get_network_from_ip(src) 57 | if src_network == src: 58 | src_network = str(ipaddress.ip_network(unicode(src_network))) 59 | src = str(ipaddress.ip_network(unicode(src))) 60 | if ipaddress.ip_network(unicode(dst)).prefixlen == 32: 61 | dst = self._get_network_from_ip(dst) 62 | if self._check_acl_exists(name, src, dst): 63 | return 64 | request_data = self._request_data("permit", name, src, dst) 65 | instance_lock = "{}:{}".format(self.lock_name, name) 66 | if self.lock_manager.lock(instance_lock, timeout=(self.acl_api_timeout * 2)): 67 | try: 68 | response = self._make_request("PUT", "api/ipv4/acl/{}".format(src_network), request_data) 69 | self._check_acl_response(response) 70 | self.storage.store_acl_network(name, src, dst) 71 | finally: 72 | self.lock_manager.unlock(instance_lock) 73 | else: 74 | raise AclApiError("could not get lock for {} instance".format(name)) 75 | 76 | def remove_acl(self, name, src): 77 | src = str(ipaddress.ip_network(unicode(src))) 78 | acls = self.storage.find_acl_network(name, src) 79 | if not acls: 80 | return 81 | destinations = [dst for acl in acls if src == acl['source'] for dst in acl['destination']] 82 | for dst in destinations: 83 | request_data = self._request_data("permit", name, src, dst, True) 84 | for env, vlan, acl_id in self._iter_on_acl_query_results(request_data): 85 | instance_lock = "{}:{}".format(self.lock_name, name) 86 | if self.lock_manager.lock(instance_lock, timeout=(self.acl_api_timeout * 2)): 87 | try: 88 | response = self._make_request("DELETE", "api/ipv4/acl/{}/{}/{}".format(env, vlan, acl_id), None) 89 | self._check_acl_response(response) 90 | self.storage.remove_acl_network(name, src) 91 | except AclNotFound: 92 | self.storage.remove_acl_network(name, src) 93 | finally: 94 | self.lock_manager.unlock(instance_lock) 95 | else: 96 | raise AclApiError("could not get lock for {} instance".format(name)) 97 | 98 | def _check_acl_response(self, response): 99 | try: 100 | response.encoding = 'utf-8' 101 | if response.status_code in [400, 404] and response.json().get('message') == "Acesso nao existe!": 102 | raise AclNotFound() 103 | if response.status_code not in [200, 201]: 104 | raise AclApiError( 105 | "Error applying ACL: {}: {}".format(response.url, response.text.encode('utf-8'))) 106 | if response.json().get('result') and response.json()['result'] != "success": 107 | raise AclApiError("invalid response: {}".format(response.json()['result'])) 108 | return response.json() 109 | except ValueError: 110 | raise AclApiError("no valid json returned") 111 | 112 | def _check_acl_exists(self, name, src, dst): 113 | acl_data = self.storage.find_acl_network(name, src) 114 | if not acl_data: 115 | return False 116 | if dst in acl_data[0]['destination']: 117 | return True 118 | return False 119 | 120 | def _iter_on_acl_query_results(self, request_data): 121 | response = self._make_request("POST", "api/ipv4/acl/search", request_data) 122 | query_results = self._check_acl_response(response) 123 | for environment in query_results.get('envs', []): 124 | for vlan in environment.get('vlans', []): 125 | environment_id = vlan['environment'] 126 | vlan_id = vlan['num_vlan'] 127 | for rule in vlan.get('rules', []): 128 | rule_id = rule['id'] 129 | yield environment_id, vlan_id, rule_id 130 | 131 | def _request_data(self, action, name, src, dst, rule_only=False): 132 | description = "{} {} rpaas access for {} instance {}".format( 133 | action, src, self.service_name, name 134 | ) 135 | data = {"kind": "object#acl", "rules": []} 136 | rule = {"protocol": "tcp", 137 | "source": src, 138 | "destination": dst, 139 | "description": description, 140 | "action": action, 141 | "l4-options": {"dest-port-start": self.acl_port_range_start, 142 | "dest-port-end": self.acl_port_range_end, 143 | "dest-port-op": "range"} 144 | } 145 | if rule_only: 146 | return rule 147 | data['rules'].append(rule) 148 | return data 149 | 150 | def _make_request(self, method, path, data): 151 | params = {} 152 | url = "{}/{}".format(self.acl_api_host, path) 153 | if data: 154 | params['json'] = data 155 | return requests.request(method.lower(), url, timeout=self.acl_api_timeout, 156 | auth=self.acl_auth_basic, **params) 157 | 158 | def _get_network_from_ip(self, ip): 159 | if not self.network_api_url: 160 | return ip 161 | ips = self.ip_client.get_ipv4_or_ipv6(ip) 162 | ips = ips['ips'] 163 | if not isinstance(ips, list): 164 | ips = [ips] 165 | net_ip = ips[0] 166 | network = self.network_client.get_network_ipv4(net_ip['networkipv4']) 167 | network = network['network'] 168 | return str(ipaddress.ip_network(unicode("{}/{}".format(ip, network['block'])), strict=False)) 169 | -------------------------------------------------------------------------------- /rpaas/admin_api.py: -------------------------------------------------------------------------------- 1 | # Copyright 2016 rpaas authors. All rights reserved. 2 | # Use of this source code is governed by a BSD-style 3 | # license that can be found in the LICENSE file. 4 | 5 | import json 6 | from bson import json_util 7 | 8 | from flask import request, Response 9 | 10 | from rpaas import auth, get_manager, storage, plan, flavor 11 | 12 | 13 | @auth.required 14 | def healings(): 15 | manager = get_manager() 16 | quantity = request.args.get("quantity", type=int) 17 | if quantity is None or quantity <= 0: 18 | quantity = 20 19 | healing_list = manager.storage.list_healings(quantity) 20 | return json.dumps(healing_list, default=json_util.default) 21 | 22 | 23 | @auth.required 24 | def create_plan(): 25 | name = request.form.get("name") 26 | description = request.form.get("description") 27 | config = json.loads(request.form.get("config", "null")) 28 | manager = get_manager() 29 | p = plan.Plan(name=name, description=description, config=config) 30 | try: 31 | manager.storage.store_plan(p) 32 | except storage.DuplicateError: 33 | return "plan already exists", 409 34 | except plan.InvalidPlanError as e: 35 | return unicode(e), 400 36 | return "", 201 37 | 38 | 39 | @auth.required 40 | def retrieve_plan(name): 41 | manager = get_manager() 42 | try: 43 | plan = manager.storage.find_plan(name) 44 | except storage.PlanNotFoundError: 45 | return "plan not found", 404 46 | return json.dumps(plan.to_dict()) 47 | 48 | 49 | @auth.required 50 | def update_plan(name): 51 | description = request.form.get("description") 52 | config = json.loads(request.form.get("config", "null")) 53 | manager = get_manager() 54 | try: 55 | manager.storage.update_plan(name, description, config) 56 | except storage.PlanNotFoundError: 57 | return "plan not found", 404 58 | return "" 59 | 60 | 61 | @auth.required 62 | def delete_plan(name): 63 | manager = get_manager() 64 | try: 65 | manager.storage.delete_plan(name) 66 | except storage.PlanNotFoundError: 67 | return "plan not found", 404 68 | return "" 69 | 70 | 71 | @auth.required 72 | def create_flavor(): 73 | name = request.form.get("name") 74 | description = request.form.get("description") 75 | config = json.loads(request.form.get("config", "null")) 76 | manager = get_manager() 77 | f = flavor.Flavor(name=name, description=description, config=config) 78 | try: 79 | manager.storage.store_flavor(f) 80 | except storage.DuplicateError: 81 | return "flavor already exists", 409 82 | except flavor.InvalidFlavorError as e: 83 | return unicode(e), 400 84 | return "", 201 85 | 86 | 87 | @auth.required 88 | def retrieve_flavor(name): 89 | manager = get_manager() 90 | try: 91 | flavor = manager.storage.find_flavor(name) 92 | except storage.FlavorNotFoundError: 93 | return "flavor not found", 404 94 | return json.dumps(flavor.to_dict()) 95 | 96 | 97 | @auth.required 98 | def update_flavor(name): 99 | description = request.form.get("description") 100 | config = json.loads(request.form.get("config", "null")) 101 | manager = get_manager() 102 | try: 103 | manager.storage.update_flavor(name, description, config) 104 | except storage.FlavorNotFoundError: 105 | return "flavor not found", 404 106 | return "" 107 | 108 | 109 | @auth.required 110 | def delete_flavor(name): 111 | manager = get_manager() 112 | try: 113 | manager.storage.delete_flavor(name) 114 | except storage.FlavorNotFoundError: 115 | return "flavor not found", 404 116 | return "" 117 | 118 | 119 | @auth.required 120 | def view_team_quota(team_name): 121 | manager = get_manager() 122 | used, quota = manager.storage.find_team_quota(team_name) 123 | return json.dumps({"used": used, "quota": quota}) 124 | 125 | 126 | @auth.required 127 | def set_team_quota(team_name): 128 | quota = request.form.get("quota", "") 129 | try: 130 | quota = int(quota) 131 | if quota < 1: 132 | raise ValueError() 133 | except ValueError: 134 | return "quota must be an integer value greather than 0", 400 135 | manager = get_manager() 136 | manager.storage.set_team_quota(team_name, quota) 137 | return "" 138 | 139 | 140 | @auth.required 141 | def restore_instance(): 142 | instance_name = request.form.get("instance_name") 143 | if not instance_name: 144 | return "instance name required", 400 145 | manager = get_manager() 146 | return Response(manager.restore_instance(instance_name), content_type='event/stream') 147 | 148 | 149 | def register_views(app, list_plans, list_flavors): 150 | app.add_url_rule("/admin/healings", methods=["GET"], 151 | view_func=healings) 152 | app.add_url_rule("/admin/plans", methods=["GET"], 153 | view_func=list_plans) 154 | app.add_url_rule("/admin/plans", methods=["POST"], 155 | view_func=create_plan) 156 | app.add_url_rule("/admin/plans/", methods=["GET"], 157 | view_func=retrieve_plan) 158 | app.add_url_rule("/admin/plans/", methods=["PUT"], 159 | view_func=update_plan) 160 | app.add_url_rule("/admin/plans/", methods=["DELETE"], 161 | view_func=delete_plan) 162 | app.add_url_rule("/admin/flavors", methods=["GET"], 163 | view_func=list_flavors) 164 | app.add_url_rule("/admin/flavors", methods=["POST"], 165 | view_func=create_flavor) 166 | app.add_url_rule("/admin/flavors/", methods=["GET"], 167 | view_func=retrieve_flavor) 168 | app.add_url_rule("/admin/flavors/", methods=["PUT"], 169 | view_func=update_flavor) 170 | app.add_url_rule("/admin/flavors/", methods=["DELETE"], 171 | view_func=delete_flavor) 172 | app.add_url_rule("/admin/quota/", methods=["GET"], 173 | view_func=view_team_quota) 174 | app.add_url_rule("/admin/quota/", methods=["POST"], 175 | view_func=set_team_quota) 176 | app.add_url_rule("/admin/restore", methods=["POST"], 177 | view_func=restore_instance) 178 | -------------------------------------------------------------------------------- /rpaas/admin_plugin.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | # Copyright 2016 rpaas authors. All rights reserved. 4 | # Use of this source code is governed by a BSD-style 5 | # license that can be found in the LICENSE file. 6 | 7 | 8 | import argparse 9 | import datetime 10 | import copy 11 | import json 12 | import os 13 | import sys 14 | import time 15 | import urllib 16 | import urllib2 17 | import shlex 18 | from functools import partial 19 | 20 | try: 21 | from bson import json_util 22 | except: 23 | sys.stderr.write("This plugin requires json_util module\n") 24 | sys.stderr.write("You can install it using: pip install pymongo\n") 25 | sys.exit(1) 26 | 27 | 28 | class CommandNotFoundError(Exception): 29 | 30 | def __init__(self, name): 31 | super(Exception, self).__init__(name) 32 | self.name = name 33 | 34 | def __str__(self): 35 | return """command "{}" not found""".format(self.name) 36 | 37 | def __unicode__(self): 38 | return unicode(str(self)) 39 | 40 | 41 | class DisplayTable: 42 | 43 | def __init__(self, fields, max_field_width=30): 44 | self.fields_names = fields 45 | self.rows = [] 46 | self.fields_widths = [] 47 | self.max_field_width = max_field_width 48 | 49 | def _compute_widths(self): 50 | widths = [len(field) for field in self.fields_names] 51 | for row in self.rows: 52 | for index, value in enumerate(row): 53 | field_width = max(widths[index], len(value)) 54 | if field_width > self.max_field_width: 55 | widths[index] = self.max_field_width 56 | else: 57 | widths[index] = field_width 58 | self.fields_widths = widths 59 | 60 | def _add_hrule(self): 61 | bits = [] 62 | bits.append("\n+") 63 | for field, width in zip(self.fields_names, self.fields_widths): 64 | bits.append((width + 2) * "-") 65 | bits.append("+") 66 | return "".join(bits) 67 | 68 | def _align_left(self, fieldname, width): 69 | padding_width = width - len(fieldname) 70 | return fieldname + " " * padding_width 71 | 72 | def _write_row(self, row): 73 | bits = [] 74 | bits.append("\n|") 75 | extra_size_row = [] 76 | for field, width in zip(row, self.fields_widths): 77 | if len(field) > self.max_field_width: 78 | bits.append(" " + self._align_left(field[:self.max_field_width], width) + " |") 79 | extra_size_row.append(field[self.max_field_width:]) 80 | else: 81 | bits.append(" " + self._align_left(field, width) + " |") 82 | extra_size_row.append("") 83 | if extra_size_row != ["" for x in self.fields_widths]: 84 | return "".join(bits) + self._write_row(extra_size_row) 85 | return "".join(bits) 86 | 87 | def add_row(self, *args): 88 | row = [] 89 | for value in args: 90 | if value is None: 91 | row.append("") 92 | continue 93 | row.append(str(value)) 94 | self.rows.append(row) 95 | 96 | def display(self): 97 | self._compute_widths() 98 | sys.stdout.write(self._add_hrule()) 99 | sys.stdout.write(self._write_row(self.fields_names)) 100 | sys.stdout.write(self._add_hrule()) 101 | for row in self.rows: 102 | sys.stdout.write(self._write_row(row)) 103 | sys.stdout.write(self._add_hrule()) 104 | sys.stdout.write("\n") 105 | 106 | 107 | def handle_plan_flavor(option, args): 108 | parser = argparse.ArgumentParser(option) 109 | subparsers = parser.add_subparsers(help="Action to {} option".format(option)) 110 | parser_choice = {} 111 | for choice in ["list", "create", "update", "delete", "show"]: 112 | parser_choice[choice] = subparsers.add_parser(choice) 113 | parser_choice[choice] = _base_args(None, parser_choice[choice]) 114 | if args and args[0] in ["list", "create", "update", "delete", "show"]: 115 | globals().get("{}_plan_flavor".format(args[0]))(option, args, parser_choice[args[0]], parser) 116 | else: 117 | parser.parse_args(args) 118 | 119 | 120 | def list_plan_flavor(option, args, parser_choice, parser): 121 | parsed_args = parser.parse_args(args) 122 | service_name = parsed_args.service 123 | result = proxy_request(service_name, "/admin/{}s".format(option), method="GET") 124 | body = result.read().rstrip("\n") 125 | if result.getcode() != 200: 126 | sys.stderr.write("ERROR: " + body + "\n") 127 | sys.exit(1) 128 | data = json.loads(body) 129 | sys.stdout.write("List of available {0}s (use {0} show for details):\n\n".format(option)) 130 | for d in data: 131 | sys.stdout.write("{name}\t\t{description}\n".format(**d)) 132 | 133 | 134 | def create_plan_flavor(option, args, parser_choice, parser): 135 | service_name, name, description, config = _change_plan_flavor_args(args, parser_choice, parser) 136 | params = { 137 | "name": name, 138 | "description": description, 139 | "config": json.dumps(config), 140 | } 141 | result = proxy_request(service_name, "/admin/{}s".format(option), body=urllib.urlencode(params), 142 | headers={"Content-Type": "application/x-www-form-urlencoded"}) 143 | if result.getcode() != 201: 144 | sys.stderr.write("ERROR: " + result.read().strip("\n") + "\n") 145 | sys.exit(1) 146 | sys.stdout.write("{} successfully created\n".format(option.capitalize())) 147 | 148 | 149 | def update_plan_flavor(option, args, parser_choice, parser): 150 | service_name, name, description, config = _change_plan_flavor_args(args, parser_choice, parser) 151 | data = _retrieve_plan_flavor(option, service_name, name) 152 | config = _merge_config(data["config"], config) 153 | params = { 154 | "description": description, 155 | "config": json.dumps(config), 156 | } 157 | result = proxy_request(service_name, "/admin/{}s/".format(option)+name, body=urllib.urlencode(params), 158 | headers={"Content-Type": "application/x-www-form-urlencoded"}, 159 | method="PUT") 160 | if result.getcode() != 200: 161 | sys.stderr.write("ERROR: " + result.read().strip("\n") + "\n") 162 | sys.exit(1) 163 | sys.stdout.write("{} successfully updated\n".format(option.capitalize())) 164 | 165 | 166 | def delete_plan_flavor(option, args, parser_choice, parser): 167 | service_name, name = _plan_flavor_arg(option, args, parser_choice, parser) 168 | result = proxy_request(service_name, "/admin/{}s/".format(option)+name, method="DELETE") 169 | if result.getcode() != 200: 170 | sys.stderr.write("ERROR: " + result.read().strip("\n") + "\n") 171 | sys.exit(1) 172 | sys.stdout.write("{} successfully deleted\n".format(option.capitalize())) 173 | 174 | 175 | def show_plan_flavor(option, args, parser_choice, parser): 176 | service_name, name = _plan_flavor_arg(option, args, parser_choice, parser) 177 | data = _retrieve_plan_flavor(option, service_name, name) 178 | _render_plan_flavor(data) 179 | 180 | 181 | def _merge_config(current, changes): 182 | current_copy = copy.deepcopy(current) 183 | current_copy.update(changes) 184 | return {k: v for k, v in current_copy.iteritems() if v} 185 | 186 | 187 | def _retrieve_plan_flavor(option, service_name, name): 188 | result = proxy_request(service_name, "/admin/{}s/".format(option)+name, method="GET") 189 | data = result.read().strip("\n") 190 | if result.getcode() != 200: 191 | sys.stderr.write("ERROR: " + data + "\n") 192 | sys.exit(1) 193 | return json.loads(data) 194 | 195 | 196 | def _render_plan_flavor(option): 197 | sys.stdout.write("Name: {name}\nDescription: {description}\n".format(**option)) 198 | sys.stdout.write("Config:\n\n") 199 | vars = [] 200 | for name, value in option["config"].iteritems(): 201 | vars.append("{}={}".format(name, value)) 202 | for var in sorted(vars): 203 | sys.stdout.write(" {}\n".format(var)) 204 | 205 | 206 | def _change_plan_flavor_args(args, parser_choice, parser): 207 | parser_choice.add_argument("-n", "--name", required=True) 208 | parser_choice.add_argument("-d", "--description", required=True) 209 | parser_choice.add_argument("-c", "--config", required=True) 210 | parsed_args = parser.parse_args(args) 211 | config_parts = shlex.split(parsed_args.config) 212 | if len(config_parts) < 1: 213 | sys.stderr.write("ERROR: Invalid config format, supported format is KEY=VALUE\n") 214 | sys.exit(2) 215 | config = {} 216 | for value in config_parts: 217 | env_parts = value.split("=") 218 | if len(env_parts) < 2: 219 | sys.stderr.write("ERROR: Invalid config format, supported format is KEY=VALUE\n") 220 | sys.exit(2) 221 | config[env_parts[0]] = "=".join(env_parts[1:]) 222 | return parsed_args.service, parsed_args.name, parsed_args.description, config 223 | 224 | 225 | def _plan_flavor_arg(option, args, parser_choice, parser): 226 | parser_choice.add_argument("{}_name".format(option)) 227 | parsed_args = parser.parse_args(args) 228 | return parsed_args.service, getattr(parsed_args, "{}_name".format(option)) 229 | 230 | 231 | def show_quota(args): 232 | parser = _base_args("show-quota") 233 | parser.add_argument("-t", "--team", required=True) 234 | parsed_args = parser.parse_args(args) 235 | result = proxy_request(parsed_args.service, "/admin/quota/"+parsed_args.team, 236 | method="GET") 237 | body = result.read().rstrip("\n") 238 | if result.getcode() != 200: 239 | sys.stderr.write("ERROR: " + body + "\n") 240 | sys.exit(1) 241 | quota = json.loads(body) 242 | sys.stdout.write("Quota usage: {usage}/{total_available}.\n".format( 243 | usage=len(quota["used"]), 244 | total_available=quota["quota"], 245 | )) 246 | 247 | 248 | def set_quota(args): 249 | parser = _base_args("set-quota") 250 | parser.add_argument("-t", "--team", required=True) 251 | parser.add_argument("-q", "--quota", required=True, type=int) 252 | parsed_args = parser.parse_args(args) 253 | result = proxy_request(parsed_args.service, "/admin/quota/"+parsed_args.team, 254 | method="POST", body=urllib.urlencode({"quota": parsed_args.quota})) 255 | body = result.read().rstrip("\n") 256 | if result.getcode() != 200: 257 | sys.stderr.write("ERROR: " + body + "\n") 258 | sys.exit(1) 259 | sys.stdout.write("Quota successfully updated.\n") 260 | 261 | 262 | def list_healings(args): 263 | parser = _base_args("list-healings") 264 | parser.add_argument("-n", "--quantity", default=20, required=False, type=int) 265 | parsed_args = parser.parse_args(args) 266 | result = proxy_request(parsed_args.service, "/admin/healings?quantity=" + str(parsed_args.quantity), 267 | method="GET") 268 | body = result.read().rstrip("\n") 269 | if result.getcode() != 200: 270 | sys.stderr.write("ERROR: " + body + "\n") 271 | sys.exit(1) 272 | try: 273 | healings_list = [] 274 | healings_list = json.loads(body, object_hook=json_util.object_hook) 275 | except Exception as e: 276 | sys.stderr.write("ERROR: invalid json response - {}\n".format(e.message)) 277 | sys.exit(1) 278 | healings_table = DisplayTable(['Instance', 'Machine', 'Start Time', 'Duration', 'Status']) 279 | _render_healings_list(healings_table, healings_list) 280 | 281 | 282 | def restore_instance(args): 283 | parser = _base_args("restore-instance") 284 | parser.add_argument("-i", "--instance", required=True) 285 | parsed_args = parser.parse_args(args) 286 | result = proxy_request(parsed_args.service, "/admin/restore", method="POST", 287 | body=urllib.urlencode({"instance_name": parsed_args.instance}), 288 | headers={"Content-Type": "application/x-www-form-urlencoded"}) 289 | if result.getcode() == 200: 290 | for msg in parser_result(result): 291 | sys.stdout.write(msg) 292 | sys.stdout.flush() 293 | else: 294 | sys.stderr.write("ERROR: " + result.content + "\n") 295 | sys.exit(1) 296 | 297 | 298 | def parser_result(fileobj, buffersize=1): 299 | for chunk in iter(partial(fileobj.read, buffersize), ''): 300 | yield chunk 301 | 302 | 303 | def _render_healings_list(healings_table, healings_list): 304 | for healing in healings_list: 305 | elapsed_time = None 306 | if 'end_time' in healing and healing['end_time'] is not None: 307 | seconds = int((healing['end_time'] - healing['start_time']).total_seconds()) 308 | elapsed_time = '{:02}:{:02}:{:02}'.format(seconds // 3600, seconds % 3600 // 60, seconds % 60) 309 | start_time = (healing['start_time'] - datetime.timedelta(seconds=time.timezone)).strftime('%b %d %X') 310 | healings_table.add_row(healing['instance'], healing['machine'], start_time, 311 | elapsed_time, healing.get('status')) 312 | healings_table.display() 313 | 314 | 315 | def _base_args(cmd_name, parser=None): 316 | if not parser: 317 | parser = argparse.ArgumentParser(cmd_name) 318 | parser.add_argument("-s", "--service", required=True) 319 | return parser 320 | 321 | 322 | def available_commands(): 323 | return { 324 | "plan": [handle_plan_flavor, "plan"], 325 | "flavor": [handle_plan_flavor, "flavor"], 326 | "show-quota": show_quota, 327 | "set-quota": set_quota, 328 | "list-healings": list_healings, 329 | "restore-instance": restore_instance 330 | } 331 | 332 | 333 | def get_command(name): 334 | command = available_commands().get(name) 335 | if not command: 336 | raise CommandNotFoundError(name) 337 | return command 338 | 339 | 340 | def get_env(name): 341 | env = os.environ.get(name) 342 | if not env: 343 | sys.stderr.write("ERROR: missing {}\n".format(name)) 344 | sys.exit(2) 345 | return env 346 | 347 | 348 | def proxy_request(service_name, path, body=None, headers=None, method='POST'): 349 | target = get_env("TSURU_TARGET").rstrip("/") 350 | token = get_env("TSURU_TOKEN") 351 | url = "{}/services/proxy/service/{}?callback={}".format(target, service_name, 352 | path) 353 | request = urllib2.Request(url) 354 | request.add_header("Authorization", "bearer " + token) 355 | request.get_method = lambda: method 356 | if body: 357 | request.add_data(body) 358 | if headers: 359 | for key, value in headers.items(): 360 | request.add_header(key, value) 361 | try: 362 | return urllib2.urlopen(request) 363 | except urllib2.HTTPError as e: 364 | sys.stderr.write("ERROR: {} - {}\n".format(e.code, e.reason)) 365 | sys.stderr.write(" {}\n".format(e.read())) 366 | sys.exit(1) 367 | 368 | 369 | def help_commands(): 370 | sys.stderr.write('Available commands:\n') 371 | for key in available_commands().keys(): 372 | sys.stderr.write(' {}\n'.format(key)) 373 | 374 | 375 | def main(args=None): 376 | if args is None: 377 | args = sys.argv[1:] 378 | if len(args) == 0: 379 | help_commands() 380 | return 381 | cmd, args = args[0], args[1:] 382 | try: 383 | command = get_command(cmd) 384 | if isinstance(command, list): 385 | command[0](command[1], args) 386 | else: 387 | command(args) 388 | except CommandNotFoundError as e: 389 | help_commands() 390 | sys.stderr.write(unicode(e) + u"\n") 391 | sys.exit(2) 392 | 393 | if __name__ == "__main__": 394 | main() 395 | -------------------------------------------------------------------------------- /rpaas/auth.py: -------------------------------------------------------------------------------- 1 | # Copyright 2016 rpaas authors. All rights reserved. 2 | # Use of this source code is governed by a BSD-style 3 | # license that can be found in the LICENSE file. 4 | 5 | import functools 6 | import os 7 | 8 | import flask 9 | 10 | 11 | def check_auth(auth): 12 | username = os.environ.get("API_USERNAME") 13 | password = os.environ.get("API_PASSWORD") 14 | if not username or not password: 15 | return True 16 | return auth and auth.username == username and auth.password == password 17 | 18 | 19 | def required(fn): 20 | @functools.wraps(fn) 21 | def decorated(*args, **kwargs): 22 | auth = flask.request.authorization 23 | if not check_auth(auth): 24 | return "you do not have access to this resource", 401 25 | return fn(*args, **kwargs) 26 | return decorated 27 | -------------------------------------------------------------------------------- /rpaas/celery_sentinel.py: -------------------------------------------------------------------------------- 1 | """ 2 | This module adds Redis Sentinel transport support to Celery. 3 | 4 | Migrate to https://github.com/celery/kombu/pull/559 when available on a 5 | release. 6 | 7 | To use it:: 8 | 9 | import register_celery_alias 10 | register_celery_alias("redis-sentinel") 11 | 12 | celery = Celery(..., broker="redis-sentinel://...", backend="redis-sentinel://...") 13 | """ 14 | import logging 15 | 16 | from celery.backends import BACKEND_ALIASES 17 | from kombu.transport import TRANSPORT_ALIASES 18 | from celery.backends.redis import RedisBackend 19 | from kombu.transport.redis import Transport, Channel 20 | from redis import Redis 21 | from redis.sentinel import Sentinel, SentinelManagedConnection 22 | 23 | 24 | class RedisSentinelBackend(RedisBackend): 25 | 26 | _redis_shared_connection = [] 27 | 28 | def __new__(cls, *args, **kwargs): 29 | obj = super(RedisSentinelBackend, cls).__new__(cls, *args, **kwargs) 30 | obj._redis_connection = cls._redis_shared_connection 31 | return obj 32 | 33 | def __init__(self, sentinels=None, sentinel_timeout=None, socket_timeout=None, 34 | min_other_sentinels=0, service_name=None, **kwargs): 35 | super(RedisSentinelBackend, self).__init__(**kwargs) 36 | self.sentinel_conf = self.app.conf['CELERY_SENTINEL_BACKEND_SETTINGS'] or {} 37 | 38 | @property 39 | def client(self): 40 | if not self._redis_connection: 41 | sentinel = Sentinel( 42 | self.sentinel_conf.get('sentinels'), 43 | min_other_sentinels=self.sentinel_conf.get("min_other_sentinels", 0), 44 | password=self.sentinel_conf.get("password"), 45 | socket_timeout=self.sentinel_conf.get("sentinel_timeout", None) 46 | ) 47 | redis_connection = sentinel.master_for(self.sentinel_conf.get("service_name"), Redis, 48 | socket_timeout=self.sentinel_conf.get("socket_timeout")) 49 | self._redis_connection.append(redis_connection) 50 | return self._redis_connection[0] 51 | 52 | 53 | class SentinelChannel(Channel): 54 | 55 | from_transport_options = Channel.from_transport_options + ( 56 | "service_name", 57 | "sentinels", 58 | "password", 59 | "min_other_sentinels", 60 | "sentinel_timeout", 61 | "max_connections" 62 | ) 63 | 64 | def _sentinel_managed_pool(self, connection_class, async=False): 65 | sentinel = Sentinel( 66 | self.sentinels, 67 | min_other_sentinels=getattr(self, "min_other_sentinels", 0), 68 | password=getattr(self, "password", None), 69 | socket_timeout=getattr(self, "sentinel_timeout", None) 70 | ) 71 | return sentinel.master_for(self.service_name, self.Client, 72 | socket_timeout=self.socket_timeout, 73 | connection_class=connection_class).connection_pool 74 | 75 | def _on_connection_disconnect(self, connection): 76 | if self._closing: 77 | self._in_poll = False 78 | self._in_listen = False 79 | if self.connection and self.connection.cycle: 80 | self.connection.cycle._on_connection_disconnect(connection) 81 | self._disconnect_pools() 82 | 83 | def _get_pool(self, async=False): 84 | channel = self 85 | 86 | class Connection(SentinelManagedConnection): 87 | def disconnect(self): 88 | super(Connection, self).disconnect() 89 | channel._on_connection_disconnect(self) 90 | connection_class = Connection 91 | self.keyprefix_fanout = self.keyprefix_fanout.format(db=0) 92 | return self._sentinel_managed_pool(connection_class, async) 93 | 94 | 95 | class RedisSentinelTransport(Transport): 96 | Channel = SentinelChannel 97 | 98 | 99 | def patch_flower_broker(): 100 | import tornado.web # NOQA 101 | from flower.views.broker import Broker 102 | from flower.utils.broker import Redis as RedisBroker 103 | from urlparse import urlparse 104 | 105 | old_new = Broker.__new__ 106 | 107 | def new_new(_, cls, broker_url, *args, **kwargs): 108 | scheme = urlparse(broker_url).scheme 109 | if scheme == 'redis-sentinel': 110 | from rpaas.tasks import app 111 | opts = app.conf.BROKER_TRANSPORT_OPTIONS 112 | s = Sentinel( 113 | opts['sentinels'], 114 | password=opts['password'], 115 | ) 116 | host, port = s.discover_master(opts['service_name']) 117 | return RedisBroker('redis://:{}@{}:{}'.format(opts['password'], host, port)) 118 | else: 119 | old_new(cls, broker_url, *args, **kwargs) 120 | Broker.__new__ = classmethod(new_new) 121 | 122 | from flower.command import settings 123 | from flower.views.tasks import TasksView 124 | from rpaas import flower_uimodules 125 | settings['ui_modules'] = flower_uimodules 126 | 127 | def new_render(self, *args, **kwargs): 128 | self._ui_module('FixTasks', self.application.ui_modules['FixTasks'])(self) 129 | super(TasksView, self).render(*args, **kwargs) 130 | 131 | TasksView.render = new_render 132 | 133 | 134 | def register_celery_alias(alias="redis-sentinel"): 135 | BACKEND_ALIASES[alias] = "rpaas.celery_sentinel.RedisSentinelBackend" 136 | TRANSPORT_ALIASES[alias] = "rpaas.celery_sentinel.RedisSentinelTransport" 137 | try: 138 | patch_flower_broker() 139 | except: 140 | logging.exception('ignored error patching flower') 141 | -------------------------------------------------------------------------------- /rpaas/consul_manager.py: -------------------------------------------------------------------------------- 1 | # Copyright 2016 rpaas authors. All rights reserved. 2 | # Use of this source code is governed by a BSD-style 3 | # license that can be found in the LICENSE file. 4 | 5 | import consul 6 | import os 7 | 8 | from . import nginx 9 | from misc import host_from_destination 10 | 11 | ACL_TEMPLATE = """key "{service_name}/{instance_name}" {{ 12 | policy = "read" 13 | }} 14 | 15 | key "{service_name}/{instance_name}/status" {{ 16 | policy = "write" 17 | }} 18 | 19 | service "nginx" {{ 20 | policy = "write" 21 | }} 22 | """ 23 | 24 | 25 | class InstanceAlreadySwappedError(Exception): 26 | pass 27 | 28 | 29 | class CertificateNotFoundError(Exception): 30 | pass 31 | 32 | 33 | class ConsulManager(object): 34 | 35 | def __init__(self, config): 36 | host = config.get("CONSUL_HOST") 37 | port = int(config.get("CONSUL_PORT", "8500")) 38 | token = config.get("CONSUL_TOKEN") 39 | self.client = consul.Consul(host=host, port=port, token=token) 40 | self.config_manager = nginx.ConfigManager(config) 41 | self.service_name = config.get("RPAAS_SERVICE_NAME", "rpaas") 42 | 43 | def generate_token(self, instance_name): 44 | rules = ACL_TEMPLATE.format(service_name=self.service_name, 45 | instance_name=instance_name) 46 | acl_name = "{}/{}/token".format(self.service_name, instance_name) 47 | return self.client.acl.create(name=acl_name, rules=rules) 48 | 49 | def destroy_token(self, acl_id): 50 | self.client.acl.destroy(acl_id) 51 | 52 | def destroy_instance(self, instance_name): 53 | self.client.kv.delete(self._key("{}/".format(instance_name)), recurse=True) 54 | 55 | def write_healthcheck(self, instance_name): 56 | self.client.kv.put(self._key(instance_name, "healthcheck"), "true") 57 | 58 | def remove_healthcheck(self, instance_name): 59 | self.client.kv.delete(self._key(instance_name, "healthcheck")) 60 | 61 | def service_healthcheck(self): 62 | _, instances = self.client.health.service("nginx", tag=self.service_name) 63 | return instances 64 | 65 | def list_node(self): 66 | _, nodes = self.client.catalog.nodes() 67 | return nodes 68 | 69 | def remove_node(self, instance_name, server_name, host_id): 70 | self.client.kv.delete(self._server_status_key(instance_name, server_name)) 71 | self.client.kv.delete(self._ssl_cert_path(instance_name, "", host_id), recurse=True) 72 | self.client.agent.force_leave(server_name) 73 | 74 | def node_hostname(self, host): 75 | for node in self.list_node(): 76 | if node['Address'] == host: 77 | return node['Node'] 78 | return None 79 | 80 | def node_status(self, instance_name): 81 | node_status = self.client.kv.get(self._server_status_key(instance_name), recurse=True) 82 | node_status_list = {} 83 | if node_status is not None: 84 | for node in node_status[1]: 85 | node_server_name = node['Key'].split('/')[-1] 86 | node_status_list[node_server_name] = node['Value'] 87 | return node_status_list 88 | 89 | def write_location(self, instance_name, path, destination=None, content=None, router_mode=False, 90 | bind_mode=False, https_only=False): 91 | if content: 92 | content = content.strip() 93 | else: 94 | upstream, _ = host_from_destination(destination) 95 | upstream_server = upstream 96 | if bind_mode: 97 | upstream = "rpaas_default_upstream" 98 | content = self.config_manager.generate_host_config(path, destination, upstream, router_mode, https_only) 99 | if router_mode: 100 | upstream_server = None 101 | self.add_server_upstream(instance_name, upstream, upstream_server) 102 | self.client.kv.put(self._location_key(instance_name, path), content) 103 | 104 | def remove_location(self, instance_name, path): 105 | self.client.kv.delete(self._location_key(instance_name, path)) 106 | 107 | def write_block(self, instance_name, block_name, content): 108 | content = self._set_header_footer(content, block_name) 109 | self.client.kv.put(self._block_key(instance_name, block_name), content) 110 | 111 | def remove_block(self, instance_name, block_name): 112 | self.write_block(instance_name, block_name, None) 113 | 114 | def list_blocks(self, instance_name, block_name=None): 115 | blocks = self.client.kv.get(self._block_key(instance_name, block_name), 116 | recurse=True) 117 | block_list = [] 118 | if blocks[1]: 119 | for block in blocks[1]: 120 | block_name = block['Key'].split('/')[-2] 121 | block_value = self._set_header_footer(block['Value'], block_name, True) 122 | if not block_value: 123 | continue 124 | block_list.append({'block_name': block_name, 'content': block_value}) 125 | return block_list 126 | 127 | def _set_header_footer(self, content, block_name, remove=False): 128 | begin_block = "## Begin custom RpaaS {} block ##\n".format(block_name) 129 | end_block = "## End custom RpaaS {} block ##".format(block_name) 130 | if remove: 131 | content = content.replace(begin_block, "") 132 | content = content.replace(end_block, "") 133 | return content.strip() 134 | if content: 135 | content = begin_block + content.strip() + '\n' + end_block 136 | else: 137 | content = begin_block + end_block 138 | return content 139 | 140 | def write_lua(self, instance_name, lua_module_name, lua_module_type, content): 141 | content_block = self._lua_module_escope(lua_module_name, content) 142 | key = self._lua_key(instance_name, lua_module_name, lua_module_type) 143 | return self.client.kv.put(key, content_block) 144 | 145 | def _lua_module_escope(self, lua_module_name, content=""): 146 | begin_escope = "-- Begin custom RpaaS {} lua module --".format(lua_module_name) 147 | end_escope = "-- End custom RpaaS {} lua module --".format(lua_module_name) 148 | content_stripped = "" 149 | if content: 150 | content_stripped = content.strip() 151 | escope = "{0}\n{1}\n{2}".format(begin_escope, content_stripped, end_escope) 152 | return escope 153 | 154 | def list_lua_modules(self, instance_name): 155 | modules = self.client.kv.get(self._lua_key(instance_name), recurse=True) 156 | module_list = [] 157 | if modules[1]: 158 | for module in modules[1]: 159 | module_name = module['Key'].split('/')[-2] 160 | module_value = module['Value'] 161 | module_list.append({'module_name': module_name, 'content': module_value}) 162 | return module_list 163 | 164 | def remove_lua(self, instance_name, lua_module_name, lua_module_type): 165 | self.write_lua(instance_name, lua_module_name, lua_module_type, None) 166 | 167 | def add_server_upstream(self, instance_name, upstream_name, server): 168 | if not server: 169 | return 170 | servers = self.list_upstream(instance_name, upstream_name) 171 | if isinstance(server, list): 172 | for idx, _ in enumerate(server): 173 | server[idx] = ":".join(map(str, filter(None, host_from_destination(server[idx])))) 174 | servers |= set(server) 175 | else: 176 | server = ":".join(map(str, filter(None, host_from_destination(server)))) 177 | servers.add(server) 178 | self._save_upstream(instance_name, upstream_name, servers) 179 | 180 | def remove_server_upstream(self, instance_name, upstream_name, server): 181 | servers = self.list_upstream(instance_name, upstream_name) 182 | if isinstance(server, list): 183 | for idx, _ in enumerate(server): 184 | server[idx] = ":".join(map(str, filter(None, host_from_destination(server[idx])))) 185 | servers -= set(server) 186 | else: 187 | server = ":".join(map(str, filter(None, host_from_destination(server)))) 188 | if server in servers: 189 | servers.remove(server) 190 | if len(servers) < 1: 191 | self._remove_upstream(instance_name, upstream_name) 192 | else: 193 | self._save_upstream(instance_name, upstream_name, servers) 194 | 195 | def _remove_upstream(self, instance_name, upstream_name): 196 | content = self._set_header_footer(None, "upstream") 197 | self.client.kv.put(self._upstream_key(instance_name, upstream_name), content) 198 | 199 | def list_upstream(self, instance_name, upstream_name): 200 | servers = self.client.kv.get(self._upstream_key(instance_name, upstream_name))[1] 201 | if servers: 202 | servers = self._set_header_footer(servers["Value"], "upstream", True) 203 | if servers == "": 204 | return set() 205 | return set(servers.split(",")) 206 | return set() 207 | 208 | def _save_upstream(self, instance_name, upstream_name, servers): 209 | content = self._set_header_footer(",".join(servers), "upstream") 210 | self.client.kv.put(self._upstream_key(instance_name, upstream_name), content) 211 | 212 | def swap_instances(self, src_instance, dst_instance): 213 | if not self.check_swap_state(src_instance, dst_instance): 214 | raise InstanceAlreadySwappedError() 215 | src_instance_value = self.client.kv.get(self._key(src_instance, "swap"))[1] 216 | if not src_instance_value: 217 | self.client.kv.put(self._key(src_instance, "swap"), dst_instance) 218 | self.client.kv.put(self._key(dst_instance, "swap"), src_instance) 219 | return 220 | if src_instance_value['Value'] == dst_instance: 221 | self.client.kv.delete(self._key(src_instance, "swap")) 222 | self.client.kv.delete(self._key(dst_instance, "swap")) 223 | return 224 | self.client.kv.put(self._key(src_instance, "swap"), dst_instance) 225 | self.client.kv.put(self._key(dst_instance, "swap"), src_instance) 226 | 227 | def check_swap_state(self, src_instance, dst_instance): 228 | src_instance_status = self.client.kv.get(self._key(src_instance, "swap"))[1] 229 | dst_instance_status = self.client.kv.get(self._key(dst_instance, "swap"))[1] 230 | if not src_instance_status and not dst_instance_status: 231 | return True 232 | if not src_instance_status or not dst_instance_status: 233 | return False 234 | if sorted([src_instance_status['Value'], dst_instance_status['Value']]) != sorted([src_instance, dst_instance]): 235 | return False 236 | return True 237 | 238 | def find_acl_network(self, instance_name, src=None): 239 | src = self._normalize_acl_src(src) 240 | acls = self.client.kv.get(self._acl_key(instance_name, src), recurse=True)[1] 241 | if not acls: 242 | return [] 243 | acls_list = [] 244 | for acl in acls: 245 | src = acl['Key'].split('/')[-1] 246 | acls_list.append({"source": self._normalize_acl_src(src), 247 | "destination": acl["Value"].split(",")}) 248 | return acls_list 249 | 250 | def store_acl_network(self, instance_name, src, dst): 251 | acls = self.find_acl_network(instance_name, src) 252 | if acls: 253 | acls = set(acls[0]['destination']) 254 | acls |= set([dst]) 255 | else: 256 | acls.append(dst) 257 | src = self._normalize_acl_src(src) 258 | self.client.kv.put(self._acl_key(instance_name, src), ",".join(acls)) 259 | 260 | def remove_acl_network(self, instance_name, src): 261 | src = self._normalize_acl_src(src) 262 | self.client.kv.delete(self._acl_key(instance_name, src)) 263 | 264 | def _normalize_acl_src(self, src): 265 | if not src: 266 | return 267 | if "_" in src: 268 | return src.replace("_", "/") 269 | return src.replace("/", "_") 270 | 271 | def get_certificate(self, instance_name, host_id=None): 272 | cert = self.client.kv.get(self._ssl_cert_path(instance_name, "cert", host_id))[1] 273 | key = self.client.kv.get(self._ssl_cert_path(instance_name, "key", host_id))[1] 274 | if not cert or not key: 275 | raise CertificateNotFoundError() 276 | return cert["Value"], key["Value"] 277 | 278 | def set_certificate(self, instance_name, cert_data, key_data, host_id=None): 279 | self.client.kv.put(self._ssl_cert_path(instance_name, "cert", host_id), 280 | cert_data.replace("\r\n", "\n")) 281 | self.client.kv.put(self._ssl_cert_path(instance_name, "key", host_id), 282 | key_data.replace("\r\n", "\n")) 283 | 284 | def delete_certificate(self, instance_name): 285 | self.client.kv.delete(self._ssl_cert_path(instance_name, "cert")) 286 | self.client.kv.delete(self._ssl_cert_path(instance_name, "key")) 287 | 288 | def _ssl_cert_path(self, instance_name, key_type, host_id=None): 289 | if host_id: 290 | return os.path.join(self._key(instance_name, "ssl/{}".format(host_id)), key_type) 291 | return os.path.join(self._key(instance_name, "ssl"), key_type) 292 | 293 | def _location_key(self, instance_name, path): 294 | location_key = "ROOT" 295 | if path != "/": 296 | location_key = path.replace("/", "___") 297 | return self._key(instance_name, "locations/" + location_key) 298 | 299 | def _block_key(self, instance_name, block_name=None): 300 | block_key = "ROOT" 301 | if block_name: 302 | block_path_key = self._key(instance_name, 303 | "blocks/%s/%s" % (block_name, 304 | block_key)) 305 | else: 306 | block_path_key = self._key(instance_name, "blocks") 307 | return block_path_key 308 | 309 | def _server_status_key(self, instance_name, server_name=None): 310 | if server_name: 311 | return self._key(instance_name, "status/%s" % server_name) 312 | return self._key(instance_name, "status") 313 | 314 | def _lua_key(self, instance_name, lua_module_name="", lua_module_type=""): 315 | base_key = "lua_module" 316 | if lua_module_name and lua_module_type: 317 | base_key = "lua_module/{0}/{1}".format(lua_module_type, lua_module_name) 318 | return self._key(instance_name, base_key) 319 | 320 | def _upstream_key(self, instance_name, upstream_name): 321 | base_key = "upstream/{}".format(upstream_name) 322 | return self._key(instance_name, base_key) 323 | 324 | def _acl_key(self, instance_name, src=None): 325 | base_key = "acl" 326 | if src: 327 | base_key = "acl/{}".format(src) 328 | return self._key(instance_name, base_key) 329 | 330 | def _key(self, instance_name, suffix=None): 331 | key = "{}/{}".format(self.service_name, instance_name) 332 | if suffix: 333 | key += "/" + suffix 334 | return key 335 | -------------------------------------------------------------------------------- /rpaas/flavor.py: -------------------------------------------------------------------------------- 1 | # Copyright 2018 rpaas authors. All rights reserved. 2 | # Use of this source code is governed by a BSD-style 3 | # license that can be found in the LICENSE file. 4 | 5 | 6 | class InvalidFlavorError(Exception): 7 | 8 | def __init__(self, field): 9 | self.field = field 10 | 11 | def __unicode__(self): 12 | return u"invalid rpaas flavor - {} is required".format(self.field) 13 | 14 | 15 | class Flavor(object): 16 | def __init__(self, name, description, config): 17 | self.name = name 18 | self.description = description 19 | self.config = config 20 | 21 | def validate(self): 22 | if not self.name: 23 | raise InvalidFlavorError("name") 24 | if not self.description: 25 | raise InvalidFlavorError("description") 26 | if not self.config: 27 | raise InvalidFlavorError("config") 28 | 29 | def to_dict(self): 30 | return {"name": self.name, "description": self.description, 31 | "config": self.config} 32 | -------------------------------------------------------------------------------- /rpaas/flower_uimodules.py: -------------------------------------------------------------------------------- 1 | import tornado.web 2 | 3 | 4 | class FixTasks(tornado.web.UIModule): 5 | def embedded_css(self): 6 | return """ 7 | #tasks-table, .dataTable { 8 | table-layout: fixed; 9 | } 10 | #tasks-table th:nth-child(3), .dataTable th:nth-child(3) { 11 | width: 50px !important; 12 | } 13 | #tasks-table th:nth-child(4), .dataTable th:nth-child(4) { 14 | width: 180px !important; 15 | } 16 | #tasks-table th:nth-child(5), .dataTable th:nth-child(5) { 17 | width: 50px !important; 18 | } 19 | #tasks-table td:nth-child(4) { 20 | overflow: hidden; 21 | text-overflow: ellipsis; 22 | white-space: nowrap; 23 | } 24 | """ 25 | 26 | def render(*args, **kwargs): 27 | pass 28 | -------------------------------------------------------------------------------- /rpaas/hc.py: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | 3 | # Copyright 2016 rpaas authors. All rights reserved. 4 | # Use of this source code is governed by a BSD-style 5 | # license that can be found in the LICENSE file. 6 | 7 | import json 8 | import uuid 9 | 10 | import requests 11 | from requests import auth 12 | 13 | 14 | class Dumb(object): 15 | 16 | def __init__(self, *args, **kwargs): 17 | self.hcs = {} 18 | 19 | def create(self, name): 20 | self.hcs[name] = [] 21 | 22 | def destroy(self, name): 23 | if name in self.hcs: 24 | del self.hcs[name] 25 | 26 | def add_url(self, name, url): 27 | if name in self.hcs: 28 | self.hcs[name].append(url) 29 | 30 | def remove_url(self, name, url): 31 | if name in self.hcs: 32 | self.hcs[name].remove(url) 33 | 34 | 35 | class HCAPI(object): 36 | 37 | def __init__(self, storage, url, user=None, password=None, 38 | hc_format=None, expected_string="WORKING"): 39 | self.storage = storage 40 | self.url = url 41 | self.user = user 42 | self.password = password 43 | self.hc_format = hc_format 44 | self.expected_string = expected_string 45 | 46 | def _issue_request(self, method, path, data=None): 47 | url = "/".join((self.url.rstrip("/"), path.lstrip("/"))) 48 | kwargs = {"data": data} 49 | if self.user and self.password: 50 | kwargs["auth"] = auth.HTTPBasicAuth(self.user, self.password) 51 | return requests.request(method, url, **kwargs) 52 | 53 | def create(self, name): 54 | resource_name = "rpaas_%s_%s" % (name, uuid.uuid4().hex) 55 | resp = self._issue_request("POST", "/resources", data={"name": resource_name}) 56 | if resp.status_code > 299: 57 | raise HCCreationError(resp.text) 58 | self.storage.store_hc({"_id": name, "resource_name": resource_name}) 59 | 60 | def destroy(self, name): 61 | hc = self.storage.retrieve_hc(name) 62 | if hc is None: 63 | return 64 | self._issue_request("DELETE", "/resources/" + hc["resource_name"]) 65 | self.storage.remove_hc(hc["_id"]) 66 | 67 | def add_url(self, name, url): 68 | hc = self.storage.retrieve_hc(name) 69 | if self.hc_format: 70 | url = self.hc_format.format(url) 71 | data = {"name": hc["resource_name"], "url": url, 72 | "expected_string": self.expected_string} 73 | resp = self._issue_request("POST", "/url", data=json.dumps(data)) 74 | if resp.status_code > 399: 75 | raise URLCreationError(resp.text) 76 | if "urls" not in hc: 77 | hc["urls"] = [] 78 | hc["urls"].append(url) 79 | self.storage.store_hc(hc) 80 | 81 | def remove_url(self, name, url): 82 | hc = self.storage.retrieve_hc(name) 83 | if self.hc_format: 84 | url = self.hc_format.format(url) 85 | data = {"name": hc["resource_name"], "url": url} 86 | self._issue_request("DELETE", "/url", data=json.dumps(data)) 87 | hc["urls"].remove(url) 88 | self.storage.store_hc(hc) 89 | 90 | 91 | class HCCreationError(Exception): 92 | pass 93 | 94 | 95 | class URLCreationError(Exception): 96 | pass 97 | -------------------------------------------------------------------------------- /rpaas/healing.py: -------------------------------------------------------------------------------- 1 | # Copyright 2016 rpaas authors. All rights reserved. 2 | # Use of this source code is governed by a BSD-style 3 | # license that can be found in the LICENSE file. 4 | 5 | import os 6 | import time 7 | from rpaas import scheduler, tasks 8 | 9 | 10 | class RestoreMachine(scheduler.JobScheduler): 11 | """ 12 | RestoreMachine is a thread for execute restore machine jobs. 13 | 14 | """ 15 | 16 | def __init__(self, config=None, *args, **kwargs): 17 | super(RestoreMachine, self).__init__(config, *args, **kwargs) 18 | self.config = config or dict(os.environ) 19 | self.interval = int(self.config.get("RESTORE_MACHINE_RUN_INTERVAL", 30)) 20 | self.last_run_key = self.get_last_run_key("RESTORE_MACHINE") 21 | 22 | def run(self): 23 | self.running = True 24 | while self.running: 25 | if self.try_lock(): 26 | tasks.RestoreMachineTask().delay(self.config) 27 | time.sleep(self.interval / 2) 28 | 29 | 30 | class CheckMachine(scheduler.JobScheduler): 31 | """ 32 | CheckMachine detects machines where checks as marked 'critical' on 33 | Consul and creates tasks to be consumed by RestoreMachine. 34 | 35 | """ 36 | 37 | def __init__(self, config=None, *args, **kwargs): 38 | super(CheckMachine, self).__init__(config, *args, **kwargs) 39 | self.config = config or dict(os.environ) 40 | self.interval = int(self.config.get("CHECK_MACHINE_RUN_INTERVAL", 30)) 41 | self.last_run_key = self.get_last_run_key("CHECK_MACHINE") 42 | 43 | def run(self): 44 | self.running = True 45 | while self.running: 46 | if self.try_lock(): 47 | tasks.CheckMachineTask().delay(self.config) 48 | time.sleep(self.interval / 2) 49 | -------------------------------------------------------------------------------- /rpaas/lock.py: -------------------------------------------------------------------------------- 1 | # Copyright 2017 rpaas authors. All rights reserved. 2 | # Use of this source code is governed by a BSD-style 3 | # license that can be found in the LICENSE file. 4 | 5 | 6 | class Lock(object): 7 | 8 | def __init__(self, redis_conn): 9 | self.redis_conn = redis_conn 10 | self.redis_locks = [] 11 | 12 | def lock(self, lock_name, timeout): 13 | position = self._find_lock_pos(lock_name) 14 | if position is not None: 15 | return self.redis_locks[position].acquire(blocking=False) 16 | self.redis_locks.append(self.redis_conn.lock(name=lock_name, timeout=timeout, blocking_timeout=1)) 17 | position = self._find_lock_pos(lock_name) 18 | return self.redis_locks[position].acquire(blocking=False) 19 | 20 | def unlock(self, lock_name): 21 | position = self._find_lock_pos(lock_name) 22 | if position is not None: 23 | self.redis_locks[position].release() 24 | del self.redis_locks[position] 25 | 26 | def extend_lock(self, lock_name, extra_time): 27 | position = self._find_lock_pos(lock_name) 28 | if position is not None: 29 | self.redis_locks[position].extend(extra_time) 30 | 31 | def _find_lock_pos(self, lock_name): 32 | if not self.redis_locks: 33 | return None 34 | position = [i for i, x in enumerate(self.redis_locks) if x.name == lock_name] 35 | if position: 36 | return position.pop() 37 | return None 38 | -------------------------------------------------------------------------------- /rpaas/misc.py: -------------------------------------------------------------------------------- 1 | # Copyright 2017 rpaas authors. All rights reserved. 2 | # Use of this source code is governed by a BSD-style 3 | # license that can be found in the LICENSE file. 4 | 5 | import os 6 | import re 7 | import urlparse 8 | import json 9 | 10 | 11 | class ValidationError(Exception): 12 | pass 13 | 14 | 15 | def check_option_enable(option): 16 | if option is not None and str(option) in ('True', 'true', '1'): 17 | return True 18 | return False 19 | 20 | 21 | def validate_content(content): 22 | if not content: 23 | return 24 | deny_patterns = os.environ.get("CONFIG_DENY_PATTERNS") 25 | if not deny_patterns: 26 | return 27 | patterns = json.loads(deny_patterns) 28 | for pattern in patterns: 29 | if re.search(pattern, content): 30 | raise ValidationError("content contains the forbidden pattern {}".format(pattern)) 31 | 32 | 33 | def validate_name(name): 34 | instance_length = None 35 | if os.environ.get("INSTANCE_LENGTH"): 36 | instance_length = int(os.environ.get("INSTANCE_LENGTH")) 37 | if not name or re.search("^[0-9a-z-]+$", name) is None or (instance_length and len(name) > instance_length): 38 | validation_error_msg = "instance name must match [0-9a-z-]" 39 | if instance_length: 40 | validation_error_msg = "{} and length up to {} chars".format(validation_error_msg, instance_length) 41 | raise ValidationError(validation_error_msg) 42 | 43 | 44 | def require_plan(): 45 | return "RPAAS_REQUIRE_PLAN" in os.environ 46 | 47 | 48 | def host_from_destination(destination): 49 | if '//' not in destination: 50 | destination = '%s%s' % ('http://', destination) 51 | return urlparse.urlparse(destination).hostname, urlparse.urlparse(destination).port 52 | -------------------------------------------------------------------------------- /rpaas/nginx.py: -------------------------------------------------------------------------------- 1 | # Copyright 2016 rpaas authors. All rights reserved. 2 | # Use of this source code is governed by a BSD-style 3 | # license that can be found in the LICENSE file. 4 | 5 | import time 6 | import datetime 7 | import os 8 | 9 | import requests 10 | 11 | from hm import config 12 | 13 | NGINX_LOCATION_INSTANCE_NOT_BOUND = ''' 14 | location / { 15 | return 404 "Instance not bound"; 16 | } 17 | ''' 18 | 19 | NGINX_LOCATION_TEMPLATE_DEFAULT = ''' 20 | location {path} {{ 21 | {https_only} 22 | proxy_set_header Host {host}; 23 | proxy_set_header X-Real-IP $remote_addr; 24 | proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; 25 | proxy_set_header X-Forwarded-Proto $scheme; 26 | proxy_set_header X-Forwarded-Host $host; 27 | proxy_set_header Connection ""; 28 | proxy_http_version 1.1; 29 | proxy_pass http://{upstream}/; 30 | proxy_redirect ~^http://{host}(:\d+)?/(.*)$ {path}$2; 31 | }} 32 | ''' 33 | 34 | NGINX_LOCATION_TEMPLATE_ROUTER = ''' 35 | location {path} {{ 36 | {https_only} 37 | proxy_set_header X-Real-IP $remote_addr; 38 | proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; 39 | proxy_set_header X-Forwarded-Proto $scheme; 40 | proxy_set_header X-Forwarded-Host $host; 41 | proxy_set_header Connection ""; 42 | proxy_http_version 1.1; 43 | proxy_pass http://{upstream}; 44 | }} 45 | ''' 46 | 47 | NGINX_HTTPS_ONLY = ''' 48 | if ($scheme = 'http') { 49 | return 301 https://$http_host$request_uri; 50 | } 51 | ''' 52 | 53 | 54 | class NginxError(Exception): 55 | pass 56 | 57 | 58 | def retry_request(f): 59 | def f_retry(self, *args, **kwargs): 60 | timeout = kwargs.get("timeout") 61 | if not timeout: 62 | timeout = 30 63 | t0 = datetime.datetime.now() 64 | timeout = datetime.timedelta(seconds=timeout) 65 | while True: 66 | try: 67 | f(self, *args, **kwargs) 68 | break 69 | except: 70 | now = datetime.datetime.now() 71 | if now > t0 + timeout: 72 | raise 73 | time.sleep(1) 74 | return f_retry 75 | 76 | 77 | class ConfigManager(object): 78 | 79 | def __init__(self, conf=None): 80 | self.location_template_default = self._load_location_template(conf, "default") 81 | self.location_template_router = self._load_location_template(conf, "router") 82 | 83 | def generate_host_config(self, path, destination, upstream, router_mode=False, https_only=False): 84 | https_only_template = '' 85 | if https_only: 86 | https_only_template = NGINX_HTTPS_ONLY 87 | if router_mode: 88 | return self.location_template_router.format( 89 | path=path.rstrip('/') + '/', 90 | host=destination, 91 | upstream=upstream, 92 | https_only=https_only_template 93 | ) 94 | return self.location_template_default.format( 95 | path=path.rstrip('/') + '/', 96 | host=destination, 97 | upstream=upstream, 98 | https_only=https_only_template 99 | ) 100 | 101 | def _load_location_template(self, conf, mode): 102 | mode = mode.upper() 103 | template_txt = config.get_config('NGINX_LOCATION_TEMPLATE_{}_TXT'.format(mode), None, conf) 104 | if template_txt: 105 | return template_txt 106 | template_url = config.get_config('NGINX_LOCATION_TEMPLATE_{}_URL'.format(mode), None, conf) 107 | if template_url: 108 | rsp = requests.get(template_url) 109 | if rsp.status_code > 299: 110 | raise NginxError("Error trying to load location template: {} - {}". 111 | format(rsp.status_code, rsp.text)) 112 | return rsp.text 113 | if mode == "DEFAULT": 114 | return NGINX_LOCATION_TEMPLATE_DEFAULT 115 | return NGINX_LOCATION_TEMPLATE_ROUTER 116 | 117 | 118 | class Nginx(object): 119 | 120 | def __init__(self, conf=None): 121 | self.nginx_manage_port = config.get_config('NGINX_MANAGE_PORT', '8089', conf) 122 | self.nginx_manage_port_tls = config.get_config('NGINX_MANAGE_PORT_TLS', '8090', conf) 123 | self.nginx_purge_path = config.get_config('NGINX_PURGE_PATH', '/purge', conf) 124 | self.nginx_expected_healthcheck = config.get_config('NGINX_HEALTHECK_EXPECTED', 125 | 'WORKING', conf) 126 | self.nginx_healthcheck_path = config.get_config('NGINX_HEALTHCHECK_PATH', 127 | '/healthcheck', conf) 128 | self.nginx_healthcheck_app_path = config.get_config('NGINX_HEALTHCHECK_APP_PATH', 129 | '/_nginx_healthcheck/', conf) 130 | self.nginx_app_port = config.get_config('NGINX_APP_PORT', '8080', conf) 131 | self.nginx_app_expected_healthcheck = config.get_config('NGINX_HEALTHECK_APP_EXPECTED', 132 | 'WORKING', conf) 133 | self.ca_cert = config.get_config('CA_CERT', None, conf) 134 | self.ca_path = "/tmp/rpaas_ca.pem" 135 | self.config_manager = ConfigManager(conf) 136 | 137 | def purge_location(self, host, path, preserve_path=False): 138 | purge_path = self.nginx_purge_path.lstrip('/') 139 | purged = False 140 | if preserve_path: 141 | for encoding in ['gzip', 'identity']: 142 | try: 143 | self._nginx_request(host, "{}/{}".format(purge_path, path), 144 | {'Accept-Encoding': encoding}) 145 | purged = True 146 | except: 147 | pass 148 | return purged 149 | for scheme in ['http', 'https']: 150 | for encoding in ['gzip', 'identity']: 151 | try: 152 | self._nginx_request(host, "{}/{}{}".format(purge_path, scheme, path), 153 | {'Accept-Encoding': encoding}) 154 | purged = True 155 | except: 156 | pass 157 | return purged 158 | 159 | @retry_request 160 | def wait_healthcheck(self, host, timeout=30, manage_healthcheck=True): 161 | if manage_healthcheck: 162 | healthcheck_path = self.nginx_healthcheck_path.lstrip('/') 163 | expected_response = self.nginx_expected_healthcheck 164 | port = self.nginx_manage_port 165 | else: 166 | healthcheck_path = self.nginx_healthcheck_app_path.lstrip('/') 167 | expected_response = self.nginx_app_expected_healthcheck 168 | port = self.nginx_app_port 169 | self._nginx_request(host, healthcheck_path, port=port, expected_response=expected_response) 170 | 171 | @retry_request 172 | def add_session_ticket(self, host, data, timeout=30): 173 | self._nginx_request(host, 'session_ticket', data=data, method='POST', secure=True, 174 | expected_response='ticket was succsessfully added') 175 | 176 | def _nginx_request(self, host, path, headers=None, port=None, 177 | expected_response=None, secure=False, method='GET', data=None): 178 | params = {} 179 | if secure: 180 | if not port: 181 | port = self.nginx_manage_port_tls 182 | protocol = 'https' 183 | self._ensure_ca_cert_file() 184 | params['verify'] = self.ca_path 185 | else: 186 | if not port: 187 | port = self.nginx_manage_port 188 | protocol = 'http' 189 | url = "{}://{}:{}/{}".format(protocol, host, port, path) 190 | if method not in ['POST', 'PUT', 'GET']: 191 | raise NginxError("Unsupported method {}".format(method)) 192 | if headers: 193 | params['headers'] = headers 194 | if data: 195 | params['data'] = data 196 | rsp = requests.request(method.lower(), url, timeout=2, **params) 197 | if rsp.status_code != 200 or (expected_response and expected_response not in rsp.text): 198 | raise NginxError( 199 | "Error trying to access admin path in nginx: {}: {}".format(url, rsp.text)) 200 | 201 | def _ensure_ca_cert_file(self): 202 | if not self.ca_cert: 203 | raise NginxError("CA_CERT should be set for nginx https internal requests") 204 | if not os.path.exists(self.ca_path): 205 | with open(self.ca_path, "w") as ca_file: 206 | ca_file.write(self.ca_cert) 207 | -------------------------------------------------------------------------------- /rpaas/plan.py: -------------------------------------------------------------------------------- 1 | # Copyright 2016 rpaas authors. All rights reserved. 2 | # Use of this source code is governed by a BSD-style 3 | # license that can be found in the LICENSE file. 4 | 5 | 6 | class InvalidPlanError(Exception): 7 | 8 | def __init__(self, field): 9 | self.field = field 10 | 11 | def __unicode__(self): 12 | return u"invalid plan - {} is required".format(self.field) 13 | 14 | 15 | class Plan(object): 16 | def __init__(self, name, description, config): 17 | self.name = name 18 | self.description = description 19 | self.config = config 20 | 21 | def validate(self): 22 | if not self.name: 23 | raise InvalidPlanError("name") 24 | if not self.description: 25 | raise InvalidPlanError("description") 26 | if not self.config: 27 | raise InvalidPlanError("config") 28 | 29 | def to_dict(self): 30 | return {"name": self.name, "description": self.description, 31 | "config": self.config} 32 | -------------------------------------------------------------------------------- /rpaas/router_api.py: -------------------------------------------------------------------------------- 1 | # Copyright 2017 rpaas authors. All rights reserved. 2 | # Use of this source code is governed by a BSD-style 3 | # license that can be found in the LICENSE file. 4 | 5 | import json 6 | 7 | from flask import request, Response, Blueprint 8 | 9 | from rpaas import (auth, get_manager, storage, manager, tasks, consul_manager) 10 | from rpaas.misc import (validate_name, require_plan, ValidationError) 11 | 12 | router = Blueprint('router', __name__, url_prefix='/router') 13 | supported_extra_features = ['tls', 'status', 'info'] # possible values: "cname", "tls", "healthcheck" 14 | 15 | 16 | @router.url_value_preprocessor 17 | def add_name_prefix(endpoint, values): 18 | if 'name' in values: 19 | values['name'] = 'router-{}'.format(values['name']) 20 | 21 | 22 | @router.route("/backend/", methods=["GET"]) 23 | @auth.required 24 | def get_backend(name): 25 | try: 26 | addr = get_manager().status(name) 27 | except storage.InstanceNotFoundError: 28 | return "Backend not found", 404 29 | if addr == manager.FAILURE: 30 | return addr, 500 31 | if addr == manager.PENDING: 32 | addr = "" 33 | return Response(response=json.dumps({"address": addr}), status=200, 34 | mimetype="application/json") 35 | 36 | 37 | @router.route("/backend/", methods=["POST"]) 38 | @auth.required 39 | def add_backend(name): 40 | try: 41 | validate_name(name) 42 | except ValidationError as e: 43 | return str(e), 400 44 | data = request.get_json() 45 | if not data: 46 | return "could not decode body json", 400 47 | team = data.get('team') or data.get('tsuru.io/app-teamowner') 48 | plan = data.get('plan') 49 | flavor = data.get('flavor') 50 | if not team: 51 | return "team name is required", 400 52 | if require_plan() and not plan: 53 | return "plan is required", 400 54 | try: 55 | if flavor: 56 | get_manager().new_instance(name, team=team, 57 | plan_name=plan, flavor_name=flavor) 58 | else: 59 | get_manager().new_instance(name, team=team, plan_name=plan) 60 | except storage.PlanNotFoundError: 61 | return "Plan not found", 404 62 | except storage.FlavorNotFoundError: 63 | return "Flavor not found", 404 64 | except storage.DuplicateError: 65 | return "{} backend already exists".format(name), 409 66 | except manager.QuotaExceededError as e: 67 | return str(e), 403 68 | return "", 201 69 | 70 | 71 | @router.route("/backend/", methods=["PUT"]) 72 | @auth.required 73 | def update_backend(name): 74 | data = request.get_json() 75 | if not data: 76 | return "could not decode body json", 400 77 | plan = data.get('plan') 78 | flavor = data.get('flavor') 79 | scale = data.get('scale') 80 | if not plan and not flavor and not scale: 81 | return "Invalid option. Valid update options are: scale, flavor and plan", 400 82 | try: 83 | if scale and int(scale) <= 0: 84 | raise ValueError 85 | get_manager().update_instance(name, plan, flavor) 86 | if scale: 87 | get_manager().scale_instance(name, scale) 88 | except tasks.NotReadyError as e: 89 | return "Backend not ready: {}".format(e), 412 90 | except storage.InstanceNotFoundError: 91 | return "Backend not found", 404 92 | except storage.PlanNotFoundError: 93 | return "Plan not found", 404 94 | except storage.FlavorNotFoundError: 95 | return "Flavor not found", 404 96 | except ValueError: 97 | return "Scale option should be integer and >0", 400 98 | return "", 204 99 | 100 | 101 | @router.route("/backend/", methods=["DELETE"]) 102 | @auth.required 103 | def delete_backend(name): 104 | try: 105 | get_manager().remove_instance(name) 106 | except storage.InstanceNotFoundError: 107 | return "Backend not found", 404 108 | except consul_manager.InstanceAlreadySwappedError: 109 | return "Instance with swap enabled", 412 110 | return "", 200 111 | 112 | 113 | @router.route("/backend//routes", methods=["GET"]) 114 | @auth.required 115 | def list_routes(name): 116 | try: 117 | routes = get_manager().list_upstreams(name, name) 118 | routes = ["http://{}".format(route) for route in routes] 119 | except tasks.NotReadyError as e: 120 | return "Backend not ready: {}".format(e), 412 121 | except storage.InstanceNotFoundError: 122 | return "Backend not found", 404 123 | return Response(response=json.dumps({"addresses": list(routes)}), status=200, 124 | mimetype="application/json") 125 | 126 | 127 | @router.route("/backend//routes", methods=["POST"]) 128 | @auth.required 129 | def add_routes(name): 130 | data = request.get_json() 131 | if not data: 132 | return "could not decode body json", 400 133 | addresses = data.get('addresses') 134 | if not addresses: 135 | return "", 200 136 | m = get_manager() 137 | try: 138 | m.bind(name, name, router_mode=True) 139 | m.add_upstream(name, name, addresses, True) 140 | except tasks.NotReadyError as e: 141 | return "Backend not ready: {}".format(e), 412 142 | except storage.InstanceNotFoundError: 143 | return "Backend not found", 404 144 | return "", 200 145 | # TODO: wait nginx reload and report status? 146 | 147 | 148 | @router.route("/backend//status", methods=["GET"]) 149 | @auth.required 150 | def status(name): 151 | node_status = get_manager().node_status(name) 152 | status = [] 153 | for node in node_status: 154 | status.append("{} - {}: {}".format(node, node_status[node]['address'], node_status[node]['status'])) 155 | node_status = {} 156 | node_status['status'] = "\n".join(status) 157 | return Response(response=json.dumps(node_status), status=200, 158 | mimetype="application/json") 159 | 160 | 161 | @router.route("/info", methods=["GET"]) 162 | @auth.required 163 | def info(): 164 | plans = get_manager().storage.list_plans() 165 | flavors = get_manager().storage.list_flavors() 166 | options_plans = ["{} - {}".format(p.name, p.description) for p in plans] 167 | options_flavors = ["{} - {}".format(f.name, f.description) for f in flavors] 168 | options = """ 169 | scale - number of instance vms 170 | plan - set instance to plan 171 | flavor - set instance to flavor 172 | """ 173 | if options_plans: 174 | options = options + "\nAvailable plans: \n" + "\n".join(options_plans) 175 | if options_plans: 176 | options = options + "\n" 177 | if options_flavors: 178 | options = options + "\nAvailable flavors: \n" + "\n".join(options_flavors) 179 | 180 | return Response(response=json.dumps({'Router options': options}), status=200, 181 | mimetype="application/json") 182 | 183 | 184 | @router.route("/backend//routes/remove", methods=["POST"]) 185 | @auth.required 186 | def delete_routes(name): 187 | data = request.get_json() 188 | if not data: 189 | return "could not decode body json", 400 190 | addresses = data.get('addresses') 191 | if not addresses: 192 | return "", 200 193 | m = get_manager() 194 | try: 195 | m.remove_upstream(name, name, addresses) 196 | routes = m.list_upstreams(name, name) 197 | if len(routes) < 1: 198 | m.unbind(name) 199 | except tasks.NotReadyError as e: 200 | return "Backend not ready: {}".format(e), 412 201 | except storage.InstanceNotFoundError: 202 | return "Backend not found", 404 203 | return "", 200 204 | # TODO: wait nginx reload and report status? 205 | 206 | 207 | @router.route("/backend//swap", methods=["POST"]) 208 | @auth.required 209 | def swap(name): 210 | data = request.get_json() 211 | if not data: 212 | return "Could not decode body json", 400 213 | if data.get('cnameOnly'): 214 | return "Swap cname only not supported", 400 215 | target_instance = data.get('target') 216 | if not target_instance: 217 | return "Target instance cannot be empty", 400 218 | m = get_manager() 219 | try: 220 | m.swap(name, "router-{}".format(target_instance)) 221 | except tasks.NotReadyError as e: 222 | return "Backend not ready: {}".format(e), 412 223 | except storage.InstanceNotFoundError: 224 | return "Backend not found", 404 225 | except consul_manager.InstanceAlreadySwappedError: 226 | return "Instance already swapped", 412 227 | return "", 200 228 | 229 | 230 | @router.route("/backend//certificate/", methods=["GET"]) 231 | @auth.required 232 | def get_certificate(name, cname): 233 | m = get_manager() 234 | try: 235 | certificate, _ = m.get_certificate(name) 236 | except storage.InstanceNotFoundError: 237 | return "Backend not found", 404 238 | except consul_manager.CertificateNotFoundError: 239 | return "Certificate not found", 404 240 | return Response(response=json.dumps({'certificate': certificate}), 241 | status=200, mimetype="application/json") 242 | 243 | 244 | @router.route("/backend//certificate/", methods=["PUT"]) 245 | @auth.required 246 | def update_certificate(name, cname): 247 | data = request.get_json() 248 | if not data: 249 | return "Could not decode body json", 400 250 | certificate = data.get('certificate') 251 | key = data.get('key') 252 | if not key or not certificate: 253 | return "Certificate or key is missing", 400 254 | m = get_manager() 255 | try: 256 | m.update_certificate(name, certificate, key) 257 | except storage.InstanceNotFoundError: 258 | return "Backend not found", 404 259 | return "", 200 260 | 261 | 262 | @router.route("/backend//certificate/", methods=["DELETE"]) 263 | @auth.required 264 | def delete_certificate(name, cname): 265 | m = get_manager() 266 | try: 267 | m.delete_certificate(name) 268 | except storage.InstanceNotFoundError: 269 | return "Backend not found", 404 270 | return "", 200 271 | 272 | 273 | @router.route("/support/", methods=["GET"]) 274 | @auth.required 275 | def supports(feature): 276 | if feature in supported_extra_features: 277 | return "", 200 278 | return "", 404 279 | -------------------------------------------------------------------------------- /rpaas/scheduler.py: -------------------------------------------------------------------------------- 1 | # Copyright 2016 rpaas authors. All rights reserved. 2 | # Use of this source code is governed by a BSD-style 3 | # license that can be found in the LICENSE file. 4 | 5 | import datetime 6 | import os 7 | import threading 8 | 9 | import redis 10 | 11 | from rpaas import tasks 12 | 13 | DATETIME_FORMAT = "%Y-%m-%d %H:%M:%S" 14 | 15 | 16 | class JobScheduler(threading.Thread): 17 | """ 18 | Generic Job Scheduler. 19 | 20 | It should run on the API role, as it depends on environment variables for 21 | working. 22 | """ 23 | 24 | def __init__(self, config=None, *args, **kwargs): 25 | super(JobScheduler, self).__init__(*args, **kwargs) 26 | self.daemon = True 27 | self.config = config or dict(os.environ) 28 | self.service_name = self.config.get("RPAAS_SERVICE_NAME", "rpaas") 29 | self.interval = int(self.config.get("JOB_SCHEDULER_RUN_INTERVAL", 30)) 30 | self.last_run_key = self.get_last_run_key("JOB_SCHEDULER") 31 | self.conn = tasks.app.broker_connection().channel().client 32 | 33 | def get_last_run_key(self, key): 34 | last_run_key = "{}_LAST_RUN_KEY".format(key) 35 | return self.config.get(last_run_key, "{}:{}:last_run".format(key.lower(), self.service_name)) 36 | 37 | def try_lock(self): 38 | interval_delta = datetime.timedelta(seconds=self.interval) 39 | with self.conn.pipeline() as pipe: 40 | try: 41 | now = datetime.datetime.utcnow() 42 | pipe.watch(self.last_run_key) 43 | last_run = pipe.get(self.last_run_key) 44 | if last_run: 45 | last_run_date = datetime.datetime.strptime(last_run, DATETIME_FORMAT) 46 | if now - last_run_date < interval_delta: 47 | pipe.unwatch() 48 | return False 49 | pipe.multi() 50 | pipe.set(self.last_run_key, now.strftime(DATETIME_FORMAT)) 51 | pipe.execute() 52 | return True 53 | except redis.WatchError: 54 | return False 55 | 56 | def run(self): 57 | raise NotImplementedError() 58 | 59 | def stop(self): 60 | self.running = False 61 | self.join() 62 | -------------------------------------------------------------------------------- /rpaas/session_resumption.py: -------------------------------------------------------------------------------- 1 | # Copyright 2017 rpaas authors. All rights reserved. 2 | # Use of this source code is governed by a BSD-style 3 | # license that can be found in the LICENSE file. 4 | 5 | import time 6 | import os 7 | from rpaas import scheduler, tasks 8 | 9 | 10 | class SessionResumption(scheduler.JobScheduler): 11 | """ 12 | SessionResumption is a thread to renew session keys on host instances. 13 | 14 | """ 15 | 16 | def __init__(self, config=None, *args, **kwargs): 17 | super(SessionResumption, self).__init__(config, *args, **kwargs) 18 | self.config = config or dict(os.environ) 19 | self.interval = int(self.config.get("SESSION_RESUMPTION_RUN_INTERVAL", 300)) 20 | self.last_run_key = self.get_last_run_key("SESSION_RESUMPTION") 21 | 22 | def run(self): 23 | self.running = True 24 | while self.running: 25 | if self.try_lock(): 26 | tasks.SessionResumptionTask().delay(self.config) 27 | time.sleep(self.interval / 2) 28 | -------------------------------------------------------------------------------- /rpaas/ssl_plugins/__init__.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | 3 | # Copyright 2016 rpaas authors. All rights reserved. 4 | # Use of this source code is governed by a BSD-style 5 | # license that can be found in the LICENSE file. 6 | 7 | from abc import ABCMeta, abstractmethod 8 | 9 | 10 | class BaseSSLPlugin(object): 11 | __metaclass__ = ABCMeta 12 | 13 | @abstractmethod 14 | def __init__(self, domain, *args, **kwargs): 15 | pass 16 | 17 | @abstractmethod 18 | def upload_csr(self, *args, **kwargs): 19 | raise NotImplementedError() 20 | 21 | @abstractmethod 22 | def download_crt(self, *args, **kwargs): 23 | raise NotImplementedError() 24 | 25 | @abstractmethod 26 | def revoke(self): 27 | raise NotImplementedError() 28 | 29 | _plugins = {} 30 | 31 | 32 | def register_plugins(): 33 | from . import default, le 34 | _plugins["le"] = le.LE 35 | _plugins["default"] = default.Default 36 | 37 | 38 | def get(name): 39 | return _plugins.get(name) 40 | -------------------------------------------------------------------------------- /rpaas/ssl_plugins/default.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | 3 | # Copyright 2016 rpaas authors. All rights reserved. 4 | # Use of this source code is governed by a BSD-style 5 | # license that can be found in the LICENSE file. 6 | 7 | import datetime 8 | import uuid 9 | 10 | from cryptography import x509 11 | from cryptography.hazmat.backends import default_backend 12 | from cryptography.x509.oid import NameOID 13 | from cryptography.hazmat.primitives import serialization 14 | from cryptography.hazmat.primitives import hashes 15 | 16 | from rpaas.ssl_plugins import BaseSSLPlugin 17 | 18 | 19 | class Default(BaseSSLPlugin): 20 | ''' Generate self-signed certificate 21 | ''' 22 | 23 | def __init__(self, domain): 24 | self.domain = domain 25 | 26 | def upload_csr(self, csr): 27 | pass 28 | 29 | def download_crt(self, key=None): 30 | one_day = datetime.timedelta(1, 0, 0) 31 | 32 | private_key = serialization.load_pem_private_key( 33 | key, 34 | password=None, 35 | backend=default_backend() 36 | ) 37 | 38 | public_key = private_key.public_key() 39 | 40 | builder = x509.CertificateBuilder() 41 | builder = builder.subject_name(x509.Name([ 42 | x509.NameAttribute(NameOID.COMMON_NAME, self.domain), 43 | ])) 44 | builder = builder.issuer_name(x509.Name([ 45 | x509.NameAttribute(NameOID.COMMON_NAME, self.domain), 46 | ])) 47 | builder = builder.not_valid_before(datetime.datetime.today() - one_day) 48 | builder = builder.not_valid_after(datetime.datetime(2018, 8, 2)) 49 | builder = builder.serial_number(int(uuid.uuid4())) 50 | builder = builder.public_key(public_key) 51 | builder = builder.add_extension( 52 | x509.BasicConstraints(ca=False, path_length=None), critical=True, 53 | ) 54 | certificate = builder.sign( 55 | private_key=private_key, algorithm=hashes.SHA256(), 56 | backend=default_backend() 57 | ) 58 | 59 | return certificate.public_bytes(serialization.Encoding.PEM) 60 | 61 | def revoke(self): 62 | pass 63 | -------------------------------------------------------------------------------- /rpaas/ssl_plugins/le.py: -------------------------------------------------------------------------------- 1 | # Copyright 2016 rpaas authors. All rights reserved. 2 | # Use of this source code is governed by a BSD-style 3 | # license that can be found in the LICENSE file. 4 | 5 | import json 6 | import logging 7 | import OpenSSL 8 | import os 9 | 10 | import acme.client as acme_client 11 | 12 | from certbot.client import Client, register 13 | from certbot.configuration import NamespaceConfig 14 | from certbot.account import AccountMemoryStorage 15 | from certbot import crypto_util 16 | from acme import jose 17 | from acme.jose.jwk import JWKRSA 18 | from cryptography.hazmat.primitives import serialization 19 | from cryptography.hazmat.backends import default_backend 20 | 21 | import zope.component 22 | 23 | from rpaas.ssl_plugins import BaseSSLPlugin 24 | from le_authenticator import RpaasLeAuthenticator 25 | 26 | logger = logging.getLogger(__name__) 27 | 28 | 29 | class LE(BaseSSLPlugin): 30 | 31 | def __init__(self, domain, email, instance_name, consul_manager=None): 32 | self.domain = str(domain) 33 | self.email = str(email) 34 | self.instance_name = str(instance_name) 35 | self.consul_manager = consul_manager 36 | 37 | def upload_csr(self, csr=None): 38 | return None 39 | 40 | def download_crt(self, id=None): 41 | try: 42 | crt, chain, key = _main([self.domain], self.email, self.instance_name, 43 | consul_manager=self.consul_manager) 44 | return json.dumps({'crt': crt, 'chain': chain, 'key': key}) 45 | finally: 46 | self.consul_manager.remove_location(self.instance_name, "/acme-validate") 47 | 48 | def revoke(self): 49 | cert, key = self.consul_manager.get_certificate(self.instance_name) 50 | return _revoke(key, cert) 51 | 52 | 53 | class ConfigNamespace(object): 54 | def __init__(self, email, domains): 55 | self.server = os.environ.get("RPAAS_PLUGIN_LE_URL", 56 | "https://acme-v01.api.letsencrypt.org/directory") 57 | self.config_dir = './le/conf' 58 | self.work_dir = './le/work' 59 | self.http01_port = None 60 | self.tls_sni_01_port = 5001 61 | self.email = email 62 | self.domains = domains 63 | self.rsa_key_size = 2048 64 | self.no_verify_ssl = False 65 | self.key_dir = './le/key' 66 | self.accounts_dir = './le/account' 67 | self.backup_dir = './le/bkp' 68 | self.csr_dir = './le/csr' 69 | self.in_progress_dir = './le/progress' 70 | self.temp_checkpoint_dir = './le/tmp' 71 | self.renewer_config_file = './le/renew' 72 | self.strict_permissions = False 73 | self.logs_dir = './le/logs' 74 | self.user_agent = 'rpaas' 75 | self.pref_challs = [] 76 | self.allow_subset_of_names = False 77 | self.must_staple = False 78 | 79 | 80 | def _main(domains=[], email=None, instance_name="", consul_manager=None): 81 | ns = ConfigNamespace(email, domains) 82 | config = NamespaceConfig(ns) 83 | zope.component.provideUtility(config) 84 | 85 | ams = AccountMemoryStorage() 86 | acc, acme = register(config, ams) 87 | 88 | authenticator = RpaasLeAuthenticator(instance_name, config=config, name='', 89 | consul_manager=consul_manager) 90 | installer = None 91 | lec = Client(config, acc, authenticator, installer, acme) 92 | certr, chain, key, _ = lec.obtain_certificate(domains) 93 | return ( 94 | OpenSSL.crypto.dump_certificate(OpenSSL.crypto.FILETYPE_PEM, certr.body), 95 | crypto_util.dump_pyopenssl_chain(chain), 96 | key.pem, 97 | ) 98 | 99 | 100 | def _revoke(rawkey, rawcert): 101 | ns = ConfigNamespace(None) 102 | acme = acme_client.Client(ns.server, key=JWKRSA( 103 | key=serialization.load_pem_private_key( 104 | rawkey, password=None, backend=default_backend()))) 105 | acme.revoke(jose.ComparableX509(OpenSSL.crypto.load_certificate( 106 | OpenSSL.crypto.FILETYPE_PEM, rawcert))) 107 | -------------------------------------------------------------------------------- /rpaas/ssl_plugins/le_authenticator.py: -------------------------------------------------------------------------------- 1 | # Copyright 2016 rpaas authors. All rights reserved. 2 | # Use of this source code is governed by a BSD-style 3 | # license that can be found in the LICENSE file. 4 | 5 | import logging 6 | import pipes 7 | import time 8 | 9 | import zope.interface 10 | 11 | from acme import challenges 12 | 13 | from certbot import interfaces 14 | from certbot.plugins import common 15 | 16 | 17 | logger = logging.getLogger(__name__) 18 | 19 | 20 | class RpaasLeAuthenticator(common.Plugin): 21 | """RPAAS Authenticator. 22 | 23 | This plugin create a authentticator for Tsuru RPAAS. 24 | """ 25 | zope.interface.implements(interfaces.IAuthenticator) 26 | zope.interface.classProvides(interfaces.IPluginFactory) 27 | hidden = True 28 | 29 | description = "Configure RPAAS HTTP server" 30 | 31 | CMD_TEMPLATE = """\ 32 | location /{achall.URI_ROOT_PATH}/{encoded_token} {{ 33 | default_type text/plain; 34 | echo -n '{validation}'; 35 | }} 36 | """ 37 | """Command template.""" 38 | 39 | def __init__(self, instance_name, consul_manager, *args, **kwargs): 40 | super(RpaasLeAuthenticator, self).__init__(*args, **kwargs) 41 | self._root = './le' 42 | self._httpd = None 43 | self.instance_name = instance_name 44 | self.consul_manager = consul_manager 45 | 46 | def get_chall_pref(self, domain): 47 | return [challenges.HTTP01] 48 | 49 | def perform(self, achalls): 50 | responses = [] 51 | for achall in achalls: 52 | responses.append(self._perform_single(achall)) 53 | return responses 54 | 55 | def _perform_single(self, achall): 56 | response, validation = achall.response_and_validation() 57 | 58 | self._notify_and_wait(self.CMD_TEMPLATE.format( 59 | achall=achall, validation=pipes.quote(validation), 60 | encoded_token=achall.chall.encode("token"))) 61 | 62 | if response.simple_verify( 63 | achall.chall, achall.domain, 64 | achall.account_key.public_key(), self.config.http01_port): 65 | return response 66 | else: 67 | logger.error( 68 | "Self-verify of challenge failed, authorization abandoned.") 69 | return None 70 | 71 | def _notify_and_wait(self, message): 72 | self.consul_manager.write_location(self.instance_name, "/acme-validate", 73 | content=message) 74 | time.sleep(6) 75 | 76 | def cleanup(self, achalls): 77 | pass 78 | -------------------------------------------------------------------------------- /rpaas/ssl_plugins/le_renewer.py: -------------------------------------------------------------------------------- 1 | # Copyright 2016 rpaas authors. All rights reserved. 2 | # Use of this source code is governed by a BSD-style 3 | # license that can be found in the LICENSE file. 4 | 5 | import time 6 | import os 7 | 8 | from rpaas import tasks, scheduler 9 | 10 | 11 | class LeRenewer(scheduler.JobScheduler): 12 | """ 13 | LeRenewer is a thread that prevents certificate LE expiration. It just adds 14 | a task to the queue, so workers can properly do the job. 15 | 16 | It should run on the API role, as it depends on environment variables for 17 | working. 18 | """ 19 | 20 | def __init__(self, config=None, *args, **kwargs): 21 | super(LeRenewer, self).__init__(config, *args, **kwargs) 22 | self.config = config or dict(os.environ) 23 | self.interval = int(self.config.get("LE_RENEWER_RUN_INTERVAL", 86400)) 24 | self.last_run_key = self.get_last_run_key("LE_RENEWER") 25 | 26 | def run(self): 27 | self.running = True 28 | while self.running: 29 | if self.try_lock(): 30 | tasks.RenewCertsTask().delay(self.config) 31 | time.sleep(self.interval / 2) 32 | -------------------------------------------------------------------------------- /rpaas/sslutils.py: -------------------------------------------------------------------------------- 1 | # Copyright 2016 rpaas authors. All rights reserved. 2 | # Use of this source code is governed by a BSD-style 3 | # license that can be found in the LICENSE file. 4 | 5 | import json 6 | import os 7 | import datetime 8 | import ipaddress 9 | import base64 10 | 11 | from cryptography import x509 12 | from cryptography.hazmat.backends import default_backend 13 | from cryptography.hazmat.primitives import hashes, serialization 14 | from cryptography.hazmat.primitives.asymmetric import rsa 15 | from cryptography.x509.oid import NameOID 16 | from hm.model.load_balancer import LoadBalancer 17 | 18 | 19 | from rpaas import consul_manager, ssl_plugins, storage 20 | 21 | 22 | def generate_session_ticket(length=48): 23 | return base64.b64encode(os.urandom(length)) 24 | 25 | 26 | def generate_key(serialized=False): 27 | key = rsa.generate_private_key( 28 | public_exponent=65537, 29 | key_size=2048, 30 | backend=default_backend() 31 | ) 32 | if serialized: 33 | return key.private_bytes( 34 | encoding=serialization.Encoding.PEM, 35 | format=serialization.PrivateFormat.TraditionalOpenSSL, 36 | encryption_algorithm=serialization.NoEncryption(), 37 | ) 38 | return key 39 | 40 | 41 | def generate_csr(key, domainname): 42 | private_key = serialization.load_pem_private_key(key, password=None, 43 | backend=default_backend()) 44 | csr = x509.CertificateSigningRequestBuilder().subject_name(x509.Name([ 45 | # Provide various details about who we are. 46 | x509.NameAttribute(NameOID.COUNTRY_NAME, u"BR"), 47 | x509.NameAttribute(NameOID.STATE_OR_PROVINCE_NAME, u"RJ"), 48 | x509.NameAttribute(NameOID.LOCALITY_NAME, u"Rio de Janeiro"), 49 | x509.NameAttribute(NameOID.ORGANIZATION_NAME, u"globo.com"), 50 | x509.NameAttribute(NameOID.COMMON_NAME, domainname), 51 | ])).add_extension( 52 | x509.SubjectAlternativeName([x509.DNSName(domainname)]), 53 | critical=False, 54 | ).sign(private_key, hashes.SHA256(), default_backend()) 55 | 56 | return csr.public_bytes(serialization.Encoding.PEM) 57 | 58 | 59 | def generate_crt(config, name, plugin, csr, key, domain): 60 | lb = LoadBalancer.find(name, config) 61 | if lb is None: 62 | raise storage.InstanceNotFoundError() 63 | strg = storage.MongoDBStorage(config) 64 | consul_mngr = consul_manager.ConsulManager(config) 65 | 66 | crt = None 67 | 68 | plugin_class = ssl_plugins.get(plugin) 69 | if not plugin_class: 70 | raise Exception("Invalid plugin {}".format(plugin)) 71 | plugin_obj = plugin_class(domain, os.environ.get('RPAAS_PLUGIN_LE_EMAIL', 'admin@'+domain), 72 | name, consul_manager=consul_mngr) 73 | 74 | # Upload csr and get an Id 75 | plugin_id = plugin_obj.upload_csr(csr) 76 | crt = plugin_obj.download_crt(id=str(plugin_id)) 77 | 78 | # Download the certificate and update nginx with it 79 | if crt: 80 | try: 81 | js_crt = json.loads(crt) 82 | cert = js_crt['crt'] 83 | cert = cert+js_crt['chain'] if 'chain' in js_crt else cert 84 | key = js_crt['key'] if 'key' in js_crt else key 85 | except: 86 | cert = crt 87 | 88 | consul_mngr.set_certificate(name, cert, key) 89 | strg.store_le_certificate(name, domain) 90 | else: 91 | raise Exception('Could not download certificate') 92 | 93 | 94 | def generate_admin_crt(config, host): 95 | private_key = generate_key() 96 | public_key = private_key.public_key() 97 | one_day = datetime.timedelta(1, 0, 0) 98 | ca_cert = config.get("CA_CERT", None) 99 | ca_key = config.get("CA_KEY", None) 100 | cert_expiration = config.get("CERT_ADMIN_EXPIRE", 1825) 101 | if not ca_cert or not ca_key: 102 | raise Exception('CA_CERT or CA_KEY not defined') 103 | ca_key = serialization.load_pem_private_key(str(ca_key), password=None, backend=default_backend()) 104 | ca_cert = x509.load_pem_x509_certificate(str(ca_cert), backend=default_backend()) 105 | builder = x509.CertificateBuilder() 106 | builder = builder.subject_name(x509.Name([ 107 | x509.NameAttribute(NameOID.COMMON_NAME, host), 108 | ])) 109 | builder = builder.issuer_name(ca_cert.subject) 110 | builder = builder.not_valid_before(datetime.datetime.today() - one_day) 111 | builder = builder.not_valid_after(datetime.datetime.today() + datetime.timedelta(days=cert_expiration)) 112 | builder = builder.serial_number(x509.random_serial_number()) 113 | builder = builder.public_key(public_key) 114 | builder = builder.add_extension( 115 | x509.SubjectAlternativeName( 116 | [x509.IPAddress(ipaddress.IPv4Address(host))] 117 | ), 118 | critical=False 119 | ) 120 | builder = builder.add_extension( 121 | x509.BasicConstraints(ca=False, path_length=None), critical=True, 122 | ) 123 | certificate = builder.sign( 124 | private_key=ca_key, algorithm=hashes.SHA256(), 125 | backend=default_backend() 126 | ) 127 | private_key = private_key.private_bytes( 128 | encoding=serialization.Encoding.PEM, 129 | format=serialization.PrivateFormat.TraditionalOpenSSL, 130 | encryption_algorithm=serialization.NoEncryption(), 131 | ) 132 | certificate = certificate.public_bytes(serialization.Encoding.PEM) 133 | return private_key, certificate 134 | -------------------------------------------------------------------------------- /rpaas/storage.py: -------------------------------------------------------------------------------- 1 | # Copyright 2016 rpaas authors. All rights reserved. 2 | # Use of this source code is governed by a BSD-style 3 | # license that can be found in the LICENSE file. 4 | 5 | import datetime 6 | 7 | import pymongo.errors 8 | 9 | from hm import storage 10 | 11 | from rpaas import plan, flavor 12 | 13 | 14 | class InstanceNotFoundError(Exception): 15 | pass 16 | 17 | 18 | class PlanNotFoundError(Exception): 19 | pass 20 | 21 | 22 | class FlavorNotFoundError(Exception): 23 | pass 24 | 25 | 26 | class DuplicateError(Exception): 27 | pass 28 | 29 | 30 | class MongoDBStorage(storage.MongoDBStorage): 31 | hcs_collections = "hcs" 32 | tasks_collection = "tasks" 33 | bindings_collection = "bindings" 34 | plans_collection = "plans" 35 | flavors_collection = "flavors" 36 | instance_metadata_collection = "instance_metadata" 37 | quota_collection = "quota" 38 | le_certificates_collection = "le_certificates" 39 | healing_collection = "healing" 40 | 41 | def store_hc(self, hc): 42 | self.db[self.hcs_collections].update({"_id": hc["_id"]}, hc, upsert=True) 43 | 44 | def retrieve_hc(self, name): 45 | return self.db[self.hcs_collections].find_one({"_id": name}) 46 | 47 | def remove_hc(self, name): 48 | self.db[self.hcs_collections].remove({"_id": name}) 49 | 50 | def store_healing(self, instance, machine): 51 | return self.db[self.healing_collection].insert({"instance": instance, "machine": machine, 52 | "start_time": datetime.datetime.utcnow()}) 53 | 54 | def update_healing(self, id, status): 55 | self.db[self.healing_collection].update({"_id": id}, 56 | {"$set": {"status": status, 57 | "end_time": datetime.datetime.utcnow()}}) 58 | 59 | def list_healings(self, quantity): 60 | coll = self.healing_collection 61 | healings = self.db[coll].find({}, {'_id': 0}).sort("start_time", -1).limit(quantity) 62 | return [healing for healing in healings] 63 | 64 | def store_task(self, name): 65 | try: 66 | if isinstance(name, dict): 67 | self.db[self.tasks_collection].insert(name) 68 | else: 69 | self.db[self.tasks_collection].insert({'_id': name}) 70 | except pymongo.errors.DuplicateKeyError: 71 | raise DuplicateError(name) 72 | 73 | def remove_task(self, query): 74 | self.db[self.tasks_collection].remove(query) 75 | 76 | def update_task(self, name, task_id_or_spec): 77 | if isinstance(task_id_or_spec, dict): 78 | self.db[self.tasks_collection].update({'_id': name}, {'$set': task_id_or_spec}) 79 | else: 80 | self.db[self.tasks_collection].update({'_id': name}, {'$set': {'task_id': task_id_or_spec}}) 81 | 82 | def find_task(self, query): 83 | if isinstance(query, dict): 84 | return self.db[self.tasks_collection].find(query) 85 | else: 86 | return self.db[self.tasks_collection].find({"_id": query}) 87 | 88 | def store_instance_metadata(self, instance_name, **data): 89 | data['_id'] = instance_name 90 | self.db[self.instance_metadata_collection].update({'_id': instance_name}, 91 | data, upsert=True) 92 | 93 | def find_instance_metadata(self, instance_name): 94 | return self.db[self.instance_metadata_collection].find_one({'_id': instance_name}) 95 | 96 | def find_host_id(self, name): 97 | return self.db[self.hosts_collection].find_one({'dns_name': name}) 98 | 99 | def remove_instance_metadata(self, instance_name): 100 | self.db[self.instance_metadata_collection].remove({'_id': instance_name}) 101 | 102 | def store_plan(self, plan): 103 | plan.validate() 104 | d = plan.to_dict() 105 | d["_id"] = d["name"] 106 | del d["name"] 107 | try: 108 | self.db[self.plans_collection].insert(d) 109 | except pymongo.errors.DuplicateKeyError: 110 | raise DuplicateError(plan.name) 111 | 112 | def update_plan(self, name, description=None, config=None): 113 | update = {} 114 | if description: 115 | update["description"] = description 116 | if config: 117 | update["config"] = config 118 | if update: 119 | result = self.db[self.plans_collection].update({"_id": name}, 120 | {"$set": update}) 121 | if not result.get("updatedExisting"): 122 | raise PlanNotFoundError() 123 | 124 | def delete_plan(self, name): 125 | result = self.db[self.plans_collection].remove({"_id": name}) 126 | if result.get("n", 0) < 1: 127 | raise PlanNotFoundError() 128 | 129 | def find_plan(self, name): 130 | plan_dict = self.db[self.plans_collection].find_one({'_id': name}) 131 | if not plan_dict: 132 | raise PlanNotFoundError() 133 | return self._plan_from_dict(plan_dict) 134 | 135 | def list_plans(self): 136 | plan_list = self.db[self.plans_collection].find() 137 | return [self._plan_from_dict(p) for p in plan_list] 138 | 139 | def _plan_from_dict(self, dict): 140 | dict["name"] = dict["_id"] 141 | del dict["_id"] 142 | return plan.Plan(**dict) 143 | 144 | def store_flavor(self, flavor): 145 | flavor.validate() 146 | d = flavor.to_dict() 147 | d["_id"] = d["name"] 148 | del d["name"] 149 | try: 150 | self.db[self.flavors_collection].insert(d) 151 | except pymongo.errors.DuplicateKeyError: 152 | raise DuplicateError(flavor.name) 153 | 154 | def update_flavor(self, name, description=None, config=None): 155 | update = {} 156 | if description: 157 | update["description"] = description 158 | if config: 159 | update["config"] = config 160 | if update: 161 | result = self.db[self.flavors_collection].update({"_id": name}, 162 | {"$set": update}) 163 | if not result.get("updatedExisting"): 164 | raise FlavorNotFoundError() 165 | 166 | def delete_flavor(self, name): 167 | result = self.db[self.flavors_collection].remove({"_id": name}) 168 | if result.get("n", 0) < 1: 169 | raise FlavorNotFoundError() 170 | 171 | def find_flavor(self, name): 172 | flavor_dict = self.db[self.flavors_collection].find_one({'_id': name}) 173 | if not flavor_dict: 174 | raise FlavorNotFoundError() 175 | return self._flavor_from_dict(flavor_dict) 176 | 177 | def list_flavors(self): 178 | flavor_list = self.db[self.flavors_collection].find() 179 | return [self._flavor_from_dict(p) for p in flavor_list] 180 | 181 | def _flavor_from_dict(self, dict): 182 | dict["name"] = dict["_id"] 183 | del dict["_id"] 184 | return flavor.Flavor(**dict) 185 | 186 | def store_binding(self, name, app_host, app_host_only=False): 187 | if app_host_only: 188 | self.db[self.bindings_collection].update({'_id': name}, { 189 | '$set': {'app_host': app_host} 190 | }, upsert=True) 191 | return 192 | try: 193 | self.delete_binding_path(name, '/') 194 | except: 195 | pass 196 | self.db[self.bindings_collection].update({'_id': name}, { 197 | '$set': {'app_host': app_host}, 198 | '$push': {'paths': { 199 | 'path': '/', 200 | 'destination': app_host 201 | }} 202 | }, upsert=True) 203 | 204 | def remove_binding(self, name): 205 | self.db[self.bindings_collection].remove({'_id': name}) 206 | 207 | def remove_root_binding(self, name, remove_root_binding_path): 208 | if remove_root_binding_path: 209 | self.delete_binding_path(name, '/') 210 | self.db[self.bindings_collection].update({'_id': name}, { 211 | '$unset': {'app_host': '1'} 212 | }) 213 | 214 | def find_binding(self, name): 215 | return self.db[self.bindings_collection].find_one({'_id': name}) 216 | 217 | def replace_binding_path(self, name, path, destination=None, content=None, https_only=False): 218 | try: 219 | self.delete_binding_path(name, path) 220 | except: 221 | pass 222 | self.db[self.bindings_collection].update({'_id': name}, {'$push': {'paths': { 223 | 'path': path, 224 | 'destination': destination, 225 | 'content': content, 226 | 'https_only': https_only 227 | }}}, upsert=True) 228 | 229 | def delete_binding_path(self, name, path): 230 | result = self.db[self.bindings_collection].update({ 231 | '_id': name, 232 | 'paths.path': path, 233 | }, { 234 | '$pull': { 235 | 'paths': { 236 | 'path': path 237 | } 238 | } 239 | }) 240 | if result['n'] == 0: 241 | raise InstanceNotFoundError() 242 | 243 | def set_team_quota(self, teamname, quota): 244 | q = self._find_team_quota(teamname) 245 | q['quota'] = quota 246 | self.db[self.quota_collection].update({'_id': teamname}, {'$set': {'quota': quota}}) 247 | return q 248 | 249 | def find_team_quota(self, teamname): 250 | quota = self._find_team_quota(teamname) 251 | return quota['used'], quota['quota'] 252 | 253 | def _find_team_quota(self, teamname): 254 | quota = self.db[self.quota_collection].find_one({'_id': teamname}) 255 | if quota is None: 256 | quota = {'_id': teamname, 'used': [], 'quota': 5} 257 | self.db[self.quota_collection].insert(quota) 258 | return quota 259 | 260 | def increment_quota(self, teamname, prev_used, servicename): 261 | result = self.db[self.quota_collection].update( 262 | {'_id': teamname, 'used': prev_used}, 263 | {'$addToSet': {'used': servicename}}) 264 | return result['n'] == 1 265 | 266 | def decrement_quota(self, servicename): 267 | self.db[self.quota_collection].update({}, {'$pull': {'used': servicename}}, multi=True) 268 | 269 | def store_le_certificate(self, name, domain): 270 | doc = {"_id": name, "domain": domain, 271 | "created": datetime.datetime.utcnow()} 272 | self.db[self.le_certificates_collection].update({"_id": name}, doc, 273 | upsert=True) 274 | 275 | def remove_le_certificate(self, name, domain): 276 | self.db[self.le_certificates_collection].remove({"_id": name, "domain": domain}) 277 | 278 | def find_le_certificates(self, query): 279 | if "name" in query: 280 | query["_id"] = query["name"] 281 | del query["name"] 282 | certificates = self.db[self.le_certificates_collection].find(query) 283 | for certificate in certificates: 284 | certificate["name"] = certificate["_id"] 285 | del certificate["_id"] 286 | yield certificate 287 | -------------------------------------------------------------------------------- /runner.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | case $RPAAS_ROLE in 4 | "worker") 5 | celery -A rpaas.tasks worker 6 | ;; 7 | "flower") 8 | celery flower -A rpaas.tasks --address=0.0.0.0 --port=$PORT --basic_auth=$FLOWER_USER:$FLOWER_PASSWORD 9 | ;; 10 | *) 11 | gunicorn rpaas.api:api -b 0.0.0.0:$PORT --access-logfile - -w ${WORKERS:=1} -k gevent 12 | ;; 13 | esac 14 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | 3 | # Copyright 2016 rpaas authors. All rights reserved. 4 | # Use of this source code is governed by a BSD-style 5 | # license that can be found in the LICENSE file. 6 | import codecs 7 | 8 | from setuptools import setup, find_packages 9 | 10 | README = codecs.open('README.rst', encoding='utf-8').read() 11 | 12 | setup( 13 | name="tsuru-rpaas", 14 | version="0.5.1", 15 | description="Reverse proxy as-a-service API for Tsuru PaaS", 16 | long_description=README, 17 | author="Tsuru", 18 | author_email="tsuru@corp.globo.com", 19 | classifiers=[ 20 | "Programming Language :: Python :: 2.7", 21 | ], 22 | packages=find_packages(exclude=["docs", "tests"]), 23 | include_package_data=True, 24 | install_requires=[ 25 | "acme==0.9.3", 26 | "amqp==1.4.9", 27 | "anyjson==0.3.3", 28 | "asn1crypto==1.4.0", 29 | "Babel==2.3.4", 30 | "backports.ssl-match-hostname==3.7.0.1", 31 | "billiard==3.3.0.23", 32 | "blinker==1.4", 33 | "boto==2.25.0", 34 | "celery[redis]==3.1.23", 35 | "certbot==0.9.3", 36 | "certifi==2017.4.17", 37 | "cffi==1.8.3", 38 | "chardet==3.0.4", 39 | "click==7.1.2", 40 | "ConfigArgParse==1.2.3", 41 | "configobj==5.0.6", 42 | "cryptography==2.1.4", 43 | "enum34==1.1.10", 44 | "Flask==0.12.4", 45 | "flower==1.2.0", 46 | "futures==3.3.0", 47 | "gevent==1.1b6", 48 | "GloboNetworkAPI==0.8.1", 49 | "greenlet==0.4.17", 50 | "gunicorn==19.5.0", 51 | "idna==2.6", 52 | "ipaddress==1.0.23", 53 | "itsdangerous==1.1.0", 54 | "Jinja2==2.11.2", 55 | "kombu==3.0.37", 56 | "letsencrypt==0.7.0", 57 | "MarkupSafe==1.1.1", 58 | "ndg-httpsclient==0.5.1", 59 | "parsedatetime==2.1", 60 | "pbr==3.1.1", 61 | "pyasn1==0.4.8", 62 | "pycparser==2.20", 63 | "pymongo==3.3.0", 64 | "pyOpenSSL==17.5.0", 65 | "pyRFC3339==1.1", 66 | "python-consul==0.6.1", 67 | "python2-pythondialog==3.5.1", 68 | "pytz==2020.4", 69 | "raven==4.2.3", 70 | "redis==2.10.6", 71 | "requests==2.18.4", 72 | "six==1.15.0", 73 | "tornado==4.2", 74 | "tsuru-hm==0.6.18", 75 | "tsuru-rpaas==0.4.1", 76 | "urllib3==1.22", 77 | "Werkzeug==0.11.15", 78 | "zope.component==4.6.2", 79 | "zope.deferredimport==4.3.1", 80 | "zope.deprecation==4.4.0", 81 | "zope.event==4.5.0", 82 | "zope.hookable==5.0.1", 83 | "zope.interface==4.3.3", 84 | "zope.proxy==4.3.5", 85 | ], 86 | extras_require={ 87 | 'tests': [ 88 | "mock==2.0.0", 89 | "flake8==2.1.0", 90 | "coverage==3.7.1", 91 | "freezegun==0.3.7", 92 | ] 93 | }, 94 | ) 95 | -------------------------------------------------------------------------------- /tests/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tsuru/rpaas/97cfd8eea1b76a6178c5b1b5a698110dba6321d8/tests/__init__.py -------------------------------------------------------------------------------- /tests/managers.py: -------------------------------------------------------------------------------- 1 | # Copyright 2014 rpaas authors. All rights reserved. 2 | # Use of this source code is governed by a BSD-style 3 | # license that can be found in the LICENSE file. 4 | 5 | from collections import defaultdict 6 | 7 | from rpaas import storage, manager, consul_manager 8 | 9 | 10 | class FakeInstance(object): 11 | 12 | def __init__(self, name, state, plan, flavor): 13 | self.name = name 14 | self.state = state 15 | self.units = 1 16 | self.plan = plan 17 | self.flavor = flavor 18 | self.bound = False 19 | self.routes = {} 20 | self.blocks = {} 21 | self.lua_modules = {} 22 | self.node_status = {} 23 | self.upstreams = defaultdict(set) 24 | self.cert = None 25 | self.key = None 26 | 27 | 28 | class FakeManager(object): 29 | 30 | def __init__(self, storage=None): 31 | self.instances = [] 32 | self.storage = storage 33 | 34 | def new_instance(self, name, state="running", team=None, plan_name=None, flavor_name=None): 35 | if plan_name: 36 | self.storage.find_plan(plan_name) 37 | if flavor_name: 38 | self.storage.find_flavor(flavor_name) 39 | instance = FakeInstance(name, state, plan_name, flavor_name) 40 | self.instances.append(instance) 41 | return instance 42 | 43 | def bind(self, name, app_host, router_mode=False): 44 | index, instance = self.find_instance(name) 45 | if index < 0: 46 | raise storage.InstanceNotFoundError() 47 | instance.bound = True 48 | 49 | def unbind(self, name): 50 | index, instance = self.find_instance(name) 51 | if index < 0: 52 | raise storage.InstanceNotFoundError() 53 | instance.bound = False 54 | 55 | def check_bound(self, name): 56 | index, instance = self.find_instance(name) 57 | if index < 0: 58 | raise storage.InstanceNotFoundError() 59 | return instance.bound 60 | 61 | def remove_instance(self, name): 62 | if name == 'router-swap_error': 63 | raise consul_manager.InstanceAlreadySwappedError() 64 | index, _ = self.find_instance(name) 65 | if index == -1: 66 | raise storage.InstanceNotFoundError() 67 | del self.instances[index] 68 | 69 | def update_instance(self, name, plan_name=None, flavor_name=None): 70 | index, _ = self.find_instance(name) 71 | if index == -1: 72 | raise storage.InstanceNotFoundError() 73 | if plan_name: 74 | self.storage.find_plan(plan_name) 75 | if flavor_name: 76 | self.storage.find_flavor(flavor_name) 77 | self.instances[index].flavor = flavor_name 78 | if plan_name: 79 | self.instances[index].plan = plan_name 80 | 81 | def info(self, name): 82 | index, instance = self.find_instance(name) 83 | if index < 0: 84 | raise storage.InstanceNotFoundError() 85 | return {"name": instance.name, "plan": instance.plan} 86 | 87 | def node_status(self, name): 88 | index, instance = self.find_instance(name) 89 | if index < 0: 90 | raise storage.InstanceNotFoundError() 91 | return instance.node_status 92 | 93 | def status(self, name): 94 | index, instance = self.find_instance(name) 95 | if index < 0: 96 | raise storage.InstanceNotFoundError() 97 | return instance.state 98 | 99 | def scale_instance(self, name, quantity): 100 | if quantity < 1: 101 | raise ValueError("invalid quantity: %d" % quantity) 102 | index, instance = self.find_instance(name) 103 | if index < 0: 104 | raise storage.InstanceNotFoundError() 105 | difference = quantity - instance.units 106 | instance.units += difference 107 | self.instances[index] = instance 108 | 109 | def update_certificate(self, name, cert, key): 110 | index, instance = self.find_instance(name) 111 | if index < 0: 112 | raise storage.InstanceNotFoundError() 113 | instance.cert = cert 114 | instance.key = key 115 | 116 | def get_certificate(self, name): 117 | index, instance = self.find_instance(name) 118 | if index < 0: 119 | raise storage.InstanceNotFoundError() 120 | if not instance.cert or not instance.key: 121 | raise consul_manager.CertificateNotFoundError() 122 | return instance.cert, instance.key 123 | 124 | def delete_certificate(self, name): 125 | index, instance = self.find_instance(name) 126 | if index < 0: 127 | raise storage.InstanceNotFoundError() 128 | instance.cert = None 129 | instance.key = None 130 | 131 | def find_instance(self, name): 132 | for i, instance in enumerate(self.instances): 133 | if instance.name == name: 134 | return i, instance 135 | return -1, None 136 | 137 | def add_route(self, name, path, destination, content, https_only): 138 | _, instance = self.find_instance(name) 139 | instance.routes[path] = {'destination': destination, 'content': content, 'https_only': https_only} 140 | 141 | def delete_route(self, name, path): 142 | _, instance = self.find_instance(name) 143 | del instance.routes[path] 144 | 145 | def list_routes(self, name): 146 | _, instance = self.find_instance(name) 147 | return instance.routes 148 | 149 | def add_block(self, name, block_name, content): 150 | _, instance = self.find_instance(name) 151 | instance.blocks[block_name] = {'content': content} 152 | 153 | def delete_block(self, name, block_name): 154 | _, instance = self.find_instance(name) 155 | del instance.blocks[block_name] 156 | 157 | def list_blocks(self, name): 158 | _, instance = self.find_instance(name) 159 | return instance.blocks 160 | 161 | def purge_location(self, name, path, preserve_path): 162 | _, instance = self.find_instance(name) 163 | if preserve_path: 164 | return 3 165 | return 4 166 | 167 | def reset(self): 168 | self.instances = [] 169 | 170 | def restore_machine_instance(self, name, machine, cancel_task=False): 171 | index, instance = self.find_instance(name) 172 | if index < 0: 173 | raise storage.InstanceNotFoundError() 174 | if machine != 'foo': 175 | raise manager.InstanceMachineNotFoundError() 176 | 177 | def restore_instance(self, name): 178 | if name in "invalid": 179 | yield "instance {} not found".format(name) 180 | return 181 | for machine in ["a", "b"]: 182 | yield "host {} restored".format(machine) 183 | if name in "error": 184 | yield "host c failed to restore" 185 | 186 | def add_lua(self, name, lua_module_name, lua_module_type, content): 187 | _, instance = self.find_instance(name) 188 | instance.lua_modules[lua_module_name] = {lua_module_type: {'content': content}} 189 | 190 | def list_lua(self, name): 191 | _, instance = self.find_instance(name) 192 | return instance.lua_modules 193 | 194 | def delete_lua(self, name, lua_module_name, lua_module_type): 195 | _, instance = self.find_instance(name) 196 | del instance.lua_modules[lua_module_type][lua_module_name] 197 | 198 | def add_upstream(self, name, upstream_name, server, acl=False): 199 | _, instance = self.find_instance(name) 200 | if isinstance(server, list): 201 | instance.upstreams[upstream_name] |= set(server) 202 | else: 203 | instance.upstreams[upstream_name].add(server) 204 | 205 | def remove_upstream(self, name, upstream_name, server): 206 | _, instance = self.find_instance(name) 207 | servers = instance.upstreams[upstream_name] 208 | if isinstance(server, list): 209 | servers -= set(server) 210 | else: 211 | if server in servers: 212 | servers.remove(server) 213 | instance.upstreams[upstream_name] = servers 214 | 215 | def list_upstreams(self, name, upstream_name): 216 | _, instance = self.find_instance(name) 217 | return instance.upstreams[upstream_name] 218 | 219 | def swap(self, instance_a, instance_b): 220 | _, instance_a = self.find_instance(instance_a) 221 | _, instance_b = self.find_instance(instance_b) 222 | if not instance_a: 223 | raise storage.InstanceNotFoundError() 224 | if not instance_b: 225 | raise consul_manager.InstanceAlreadySwappedError() 226 | -------------------------------------------------------------------------------- /tests/test_admin_api.py: -------------------------------------------------------------------------------- 1 | # Copyright 2016 rpaas authors. All rights reserved. 2 | # Use of this source code is governed by a BSD-style 3 | # license that can be found in the LICENSE file. 4 | 5 | import datetime 6 | import json 7 | import unittest 8 | import os 9 | 10 | from bson import json_util 11 | from rpaas import api, storage, admin_api 12 | from . import managers 13 | 14 | 15 | class AdminAPITestCase(unittest.TestCase): 16 | 17 | @classmethod 18 | def setUpClass(cls): 19 | os.environ["MONGO_DATABASE"] = "api_admin_test" 20 | cls.storage = storage.MongoDBStorage() 21 | cls.manager = managers.FakeManager(storage=cls.storage) 22 | api.get_manager = lambda: cls.manager 23 | admin_api.get_manager = lambda: cls.manager 24 | cls.api = api.api.test_client() 25 | 26 | def setUp(self): 27 | self.manager.reset() 28 | colls = self.storage.db.collection_names(False) 29 | for coll in colls: 30 | self.storage.db.drop_collection(coll) 31 | 32 | def test_list_healings(self): 33 | resp = self.api.get("/admin/healings") 34 | self.assertEqual(200, resp.status_code) 35 | self.assertEqual("[]", resp.data) 36 | loop_time = datetime.datetime(2016, 8, 2, 10, 53, 0) 37 | healing_list = [] 38 | for x in range(1, 30): 39 | data = {"instance": "myinstance", "machine": "10.10.1.{}".format(x), 40 | "start_time": loop_time, "end_time": loop_time, "status": "success"} 41 | healing_list.append(json.loads(json.dumps(data, default=json_util.default))) 42 | self.storage.db[self.storage.healing_collection].insert(data) 43 | loop_time = loop_time + datetime.timedelta(minutes=5) 44 | healing_list.reverse() 45 | resp = self.api.get("/admin/healings") 46 | self.assertEqual(200, resp.status_code) 47 | self.assertListEqual(healing_list[:20], json.loads(resp.data)) 48 | resp = self.api.get("/admin/healings?quantity=10") 49 | self.assertEqual(200, resp.status_code) 50 | self.assertListEqual(healing_list[:10], json.loads(resp.data)) 51 | resp = self.api.get("/admin/healings?quantity=aaaa") 52 | self.assertEqual(200, resp.status_code) 53 | self.assertListEqual(healing_list[:20], json.loads(resp.data)) 54 | 55 | def test_list_plans(self): 56 | resp = self.api.get("/admin/plans") 57 | self.assertEqual(200, resp.status_code) 58 | self.assertEqual("[]", resp.data) 59 | self.storage.db[self.storage.plans_collection].insert( 60 | {"_id": "small", 61 | "description": "some cool plan", 62 | "config": {"serviceofferingid": "abcdef123456"}} 63 | ) 64 | self.storage.db[self.storage.plans_collection].insert( 65 | {"_id": "huge", 66 | "description": "some cool huge plan", 67 | "config": {"serviceofferingid": "abcdef123459"}} 68 | ) 69 | resp = self.api.get("/resources/plans") 70 | self.assertEqual(200, resp.status_code) 71 | expected = [ 72 | {"name": "small", "description": "some cool plan", 73 | "config": {"serviceofferingid": "abcdef123456"}}, 74 | {"name": "huge", "description": "some cool huge plan", 75 | "config": {"serviceofferingid": "abcdef123459"}}, 76 | ] 77 | self.assertEqual(expected, json.loads(resp.data)) 78 | 79 | def test_create_plan(self): 80 | config = json.dumps({ 81 | "serviceofferingid": "abcdef1234", 82 | "NAME": "super", 83 | }) 84 | resp = self.api.post("/admin/plans", data={"name": "small", 85 | "description": "small instance", 86 | "config": config}) 87 | self.assertEqual(201, resp.status_code) 88 | plan = self.storage.find_plan("small") 89 | self.assertEqual("small", plan.name) 90 | self.assertEqual("small instance", plan.description) 91 | self.assertEqual(json.loads(config), plan.config) 92 | 93 | def test_create_plan_duplicate(self): 94 | self.storage.db[self.storage.plans_collection].insert( 95 | {"_id": "small", 96 | "description": "some cool plan", 97 | "config": {"serviceofferingid": "abcdef123456"}} 98 | ) 99 | config = json.dumps({ 100 | "serviceofferingid": "abcdef1234", 101 | "NAME": "super", 102 | }) 103 | resp = self.api.post("/admin/plans", data={"name": "small", 104 | "description": "small instance", 105 | "config": config}) 106 | self.assertEqual(409, resp.status_code) 107 | 108 | def test_create_plan_invalid(self): 109 | config = json.dumps({ 110 | "serviceofferingid": "abcdef1234", 111 | "NAME": "super", 112 | }) 113 | resp = self.api.post("/admin/plans", data={"description": "small instance", 114 | "config": config}) 115 | self.assertEqual(400, resp.status_code) 116 | self.assertEqual("invalid plan - name is required", resp.data) 117 | resp = self.api.post("/admin/plans", data={"name": "small", 118 | "config": config}) 119 | self.assertEqual(400, resp.status_code) 120 | self.assertEqual("invalid plan - description is required", resp.data) 121 | resp = self.api.post("/admin/plans", data={"name": "small", 122 | "description": "something small"}) 123 | self.assertEqual(400, resp.status_code) 124 | self.assertEqual("invalid plan - config is required", resp.data) 125 | 126 | def test_retrieve_plan(self): 127 | self.storage.db[self.storage.plans_collection].insert( 128 | {"_id": "small", 129 | "description": "some cool plan", 130 | "config": {"serviceofferingid": "abcdef123456"}} 131 | ) 132 | plan = self.storage.find_plan("small") 133 | resp = self.api.get("/admin/plans/small") 134 | self.assertEqual(200, resp.status_code) 135 | self.assertEqual(plan.to_dict(), json.loads(resp.data)) 136 | 137 | def test_retrieve_plan_not_found(self): 138 | resp = self.api.get("/admin/plans/small") 139 | self.assertEqual(404, resp.status_code) 140 | self.assertEqual("plan not found", resp.data) 141 | 142 | def test_update_plan(self): 143 | self.storage.db[self.storage.plans_collection].insert( 144 | {"_id": "small", 145 | "description": "some cool plan", 146 | "config": {"serviceofferingid": "abcdef123456"}} 147 | ) 148 | config = json.dumps({ 149 | "serviceofferingid": "abcdef1234", 150 | "NAME": "super", 151 | }) 152 | resp = self.api.put("/admin/plans/small", data={"description": "small instance", 153 | "config": config}) 154 | self.assertEqual(200, resp.status_code) 155 | plan = self.storage.find_plan("small") 156 | self.assertEqual("small", plan.name) 157 | self.assertEqual("small instance", plan.description) 158 | self.assertEqual(json.loads(config), plan.config) 159 | 160 | def test_update_plan_partial(self): 161 | self.storage.db[self.storage.plans_collection].insert( 162 | {"_id": "small", 163 | "description": "some cool plan", 164 | "config": {"serviceofferingid": "abcdef123456"}} 165 | ) 166 | config = json.dumps({ 167 | "serviceofferingid": "abcdef1234", 168 | "NAME": "super", 169 | }) 170 | resp = self.api.put("/admin/plans/small", data={"config": config}) 171 | self.assertEqual(200, resp.status_code) 172 | plan = self.storage.find_plan("small") 173 | self.assertEqual("small", plan.name) 174 | self.assertEqual("some cool plan", plan.description) 175 | self.assertEqual(json.loads(config), plan.config) 176 | 177 | def test_update_plan_not_found(self): 178 | config = json.dumps({ 179 | "serviceofferingid": "abcdef1234", 180 | "NAME": "super", 181 | }) 182 | resp = self.api.put("/admin/plans/small", data={"description": "small instance", 183 | "config": config}) 184 | self.assertEqual(404, resp.status_code) 185 | self.assertEqual("plan not found", resp.data) 186 | 187 | def test_delete_plan(self): 188 | self.storage.db[self.storage.plans_collection].insert( 189 | {"_id": "small", 190 | "description": "some cool plan", 191 | "config": {"serviceofferingid": "abcdef123456"}} 192 | ) 193 | resp = self.api.delete("/admin/plans/small") 194 | self.assertEqual(200, resp.status_code) 195 | with self.assertRaises(storage.PlanNotFoundError): 196 | self.storage.find_plan("small") 197 | 198 | def test_delete_plan_not_found(self): 199 | resp = self.api.delete("/admin/plans/small") 200 | self.assertEqual(404, resp.status_code) 201 | self.assertEqual("plan not found", resp.data) 202 | 203 | def test_list_flavors(self): 204 | resp = self.api.get("/admin/flavors") 205 | self.assertEqual(200, resp.status_code) 206 | self.assertEqual("[]", resp.data) 207 | self.storage.db[self.storage.flavors_collection].insert( 208 | {"_id": "orange", 209 | "description": "nginx 1.12", 210 | "config": {"nginx_version": "1.12"}} 211 | ) 212 | self.storage.db[self.storage.flavors_collection].insert( 213 | {"_id": "vanilla", 214 | "description": "nginx 1.10", 215 | "config": {"nginx_version": "1.10"}} 216 | ) 217 | resp = self.api.get("/resources/flavors") 218 | self.assertEqual(200, resp.status_code) 219 | expected = [ 220 | {"name": "orange", "description": "nginx 1.12", 221 | "config": {"nginx_version": "1.12"}}, 222 | {"name": "vanilla", "description": "nginx 1.10", 223 | "config": {"nginx_version": "1.10"}}, 224 | ] 225 | self.assertEqual(expected, json.loads(resp.data)) 226 | 227 | def test_create_flavor(self): 228 | config = json.dumps({ 229 | "nginx_version": "1.12", 230 | "extra_config": "dsr", 231 | }) 232 | resp = self.api.post("/admin/flavors", data={"name": "nginx_dsr", 233 | "description": "nginx 1.12 + dsr", 234 | "config": config}) 235 | self.assertEqual(201, resp.status_code) 236 | flavor = self.storage.find_flavor("nginx_dsr") 237 | self.assertEqual("nginx_dsr", flavor.name) 238 | self.assertEqual("nginx 1.12 + dsr", flavor.description) 239 | self.assertEqual(json.loads(config), flavor.config) 240 | 241 | def test_create_flavor_duplicate(self): 242 | self.storage.db[self.storage.flavors_collection].insert( 243 | {"_id": "orange", 244 | "description": "nginx 1.12", 245 | "config": {"nginx_version": "1.12"}} 246 | ) 247 | config = json.dumps({ 248 | "nginx_version": "1.10" 249 | }) 250 | resp = self.api.post("/admin/flavors", data={"name": "orange", 251 | "description": "nginx 1.10", 252 | "config": config}) 253 | self.assertEqual(409, resp.status_code) 254 | 255 | def test_create_flavor_invalid(self): 256 | config = json.dumps({ 257 | "nginx_version": "1.10" 258 | }) 259 | resp = self.api.post("/admin/flavors", data={"description": "nginx 1.10", 260 | "config": config}) 261 | self.assertEqual(400, resp.status_code) 262 | self.assertEqual("invalid rpaas flavor - name is required", resp.data) 263 | resp = self.api.post("/admin/flavors", data={"name": "orange", 264 | "config": config}) 265 | self.assertEqual(400, resp.status_code) 266 | self.assertEqual("invalid rpaas flavor - description is required", resp.data) 267 | resp = self.api.post("/admin/flavors", data={"name": "orange", 268 | "description": "nginx 1.10"}) 269 | self.assertEqual(400, resp.status_code) 270 | self.assertEqual("invalid rpaas flavor - config is required", resp.data) 271 | 272 | def test_retrieve_flavor(self): 273 | self.storage.db[self.storage.flavors_collection].insert( 274 | {"_id": "orange", 275 | "description": "nginx 1.10", 276 | "config": {"nginx_version": "1.10"}} 277 | ) 278 | flavor = self.storage.find_flavor("orange") 279 | resp = self.api.get("/admin/flavors/orange") 280 | self.assertEqual(200, resp.status_code) 281 | self.assertEqual(flavor.to_dict(), json.loads(resp.data)) 282 | 283 | def test_retrieve_flavor_not_found(self): 284 | resp = self.api.get("/admin/flavors/vanilla") 285 | self.assertEqual(404, resp.status_code) 286 | self.assertEqual("flavor not found", resp.data) 287 | 288 | def test_update_flavor(self): 289 | self.storage.db[self.storage.flavors_collection].insert( 290 | {"_id": "orange", 291 | "description": "nginx 1.10", 292 | "config": {"nginx_version": "1.10"}} 293 | ) 294 | config = json.dumps({ 295 | "nginx_version": "1.10.1", 296 | "dsr": "true", 297 | }) 298 | resp = self.api.put("/admin/flavors/orange", data={"description": "nginx 1.10", 299 | "config": config}) 300 | self.assertEqual(200, resp.status_code) 301 | flavor = self.storage.find_flavor("orange") 302 | self.assertEqual("orange", flavor.name) 303 | self.assertEqual("nginx 1.10", flavor.description) 304 | self.assertEqual(json.loads(config), flavor.config) 305 | 306 | def test_update_flavor_not_found(self): 307 | config = json.dumps({ 308 | "nginx_version": "1.10" 309 | }) 310 | resp = self.api.put("/admin/flavors/vanilla", data={"description": "nginx 1.10", 311 | "config": config}) 312 | self.assertEqual(404, resp.status_code) 313 | self.assertEqual("flavor not found", resp.data) 314 | 315 | def test_delete_flavor(self): 316 | self.storage.db[self.storage.flavors_collection].insert( 317 | {"_id": "vanilla", 318 | "description": "nginx version 1.10", 319 | "config": {"nginx_version": "1.10"}} 320 | ) 321 | resp = self.api.delete("/admin/flavors/vanilla") 322 | self.assertEqual(200, resp.status_code) 323 | with self.assertRaises(storage.FlavorNotFoundError): 324 | self.storage.find_flavor("vanilla") 325 | 326 | def test_delete_flavor_not_found(self): 327 | resp = self.api.delete("/admin/flavors/vanilla") 328 | self.assertEqual(404, resp.status_code) 329 | self.assertEqual("flavor not found", resp.data) 330 | 331 | def test_view_team_quota(self): 332 | self.storage.db[self.storage.quota_collection].insert( 333 | {"_id": "myteam", 334 | "used": ["inst1", "inst2"], 335 | "quota": 10} 336 | ) 337 | resp = self.api.get("/admin/quota/myteam") 338 | self.assertEqual(200, resp.status_code) 339 | self.assertEqual({"used": ["inst1", "inst2"], "quota": 10}, 340 | json.loads(resp.data)) 341 | resp = self.api.get("/admin/quota/yourteam") 342 | self.assertEqual(200, resp.status_code) 343 | self.assertEqual({"used": [], "quota": 5}, json.loads(resp.data)) 344 | 345 | def test_set_team_quota(self): 346 | self.storage.db[self.storage.quota_collection].insert( 347 | {"_id": "myteam", 348 | "used": ["inst1", "inst2"], 349 | "quota": 10} 350 | ) 351 | resp = self.api.post("/admin/quota/myteam", data={"quota": 12}) 352 | self.assertEqual(200, resp.status_code) 353 | used, quota = self.storage.find_team_quota("myteam") 354 | self.assertEqual(["inst1", "inst2"], used) 355 | self.assertEqual(12, quota) 356 | resp = self.api.post("/admin/quota/yourteam", data={"quota": 3}) 357 | self.assertEqual(200, resp.status_code) 358 | used, quota = self.storage.find_team_quota("yourteam") 359 | self.assertEqual([], used) 360 | self.assertEqual(3, quota) 361 | 362 | def test_set_team_quota_invalid_value(self): 363 | resp = self.api.post("/admin/quota/myteam", data={}) 364 | self.assertEqual(400, resp.status_code) 365 | self.assertEqual("quota must be an integer value greather than 0", resp.data) 366 | resp = self.api.post("/admin/quota/myteam", data={"quota": "abc"}) 367 | self.assertEqual(400, resp.status_code) 368 | self.assertEqual("quota must be an integer value greather than 0", resp.data) 369 | resp = self.api.post("/admin/quota/myteam", data={"quota": "0"}) 370 | self.assertEqual(400, resp.status_code) 371 | self.assertEqual("quota must be an integer value greather than 0", resp.data) 372 | resp = self.api.post("/admin/quota/myteam", data={"quota": "-3"}) 373 | self.assertEqual(400, resp.status_code) 374 | self.assertEqual("quota must be an integer value greather than 0", resp.data) 375 | 376 | def test_restore_instance_successfully(self): 377 | resp = self.api.post("/admin/restore", data={"instance_name": "blah"}) 378 | self.assertEqual(200, resp.status_code) 379 | response = ["host a restored", "host b restored"] 380 | self.assertEqual("".join(response), resp.data) 381 | 382 | def test_restore_invalid_instance_name(self): 383 | resp = self.api.post("/admin/restore", data={"instance_name": "invalid"}) 384 | self.assertEqual(200, resp.status_code) 385 | self.assertEqual("instance invalid not found", resp.data) 386 | 387 | def test_restore_instance_error_on_restore(self): 388 | resp = self.api.post("/admin/restore", data={"instance_name": "error"}) 389 | self.assertEqual(200, resp.status_code) 390 | response = ["host a restored", "host b restored", "host c failed to restore"] 391 | self.assertEqual("".join(response), resp.data) 392 | -------------------------------------------------------------------------------- /tests/test_auth.py: -------------------------------------------------------------------------------- 1 | # Copyright 2014 rpaas authors. All rights reserved. 2 | # Use of this source code is governed by a BSD-style 3 | # license that can be found in the LICENSE file. 4 | 5 | import base64 6 | import os 7 | import unittest 8 | 9 | import flask 10 | 11 | from rpaas import auth 12 | 13 | 14 | class AuthTestCase(unittest.TestCase): 15 | 16 | def setUp(self): 17 | self.called = False 18 | self.app = flask.Flask(__name__) 19 | self.client = self.app.test_client() 20 | 21 | @self.app.route("/") 22 | @auth.required 23 | def myfn(): 24 | self.called = True 25 | return "hello world" 26 | 27 | def set_envs(self): 28 | os.environ["API_USERNAME"] = self.username = "rpaas" 29 | os.environ["API_PASSWORD"] = self.password = "rpaas123" 30 | 31 | def delete_envs(self): 32 | del os.environ["API_USERNAME"], os.environ["API_PASSWORD"] 33 | 34 | def get(self, url, user, password): 35 | encoded = base64.b64encode(user + ":" + password) 36 | return self.client.open(url, method="GET", 37 | headers={"Authorization": "Basic " + encoded}) 38 | 39 | def test_authentication_required_no_auth_in_environment(self): 40 | resp = self.get("/", "", "") 41 | self.assertEqual(200, resp.status_code) 42 | self.assertEqual("hello world", resp.data) 43 | 44 | def test_authentication_required_no_auth_provided(self): 45 | self.set_envs() 46 | self.addCleanup(self.delete_envs) 47 | resp = self.client.get("/") 48 | self.assertEqual(401, resp.status_code) 49 | self.assertEqual("you do not have access to this resource", resp.data) 50 | 51 | def test_authentication_required_wrong_data(self): 52 | pairs = [("joao", "joao123"), ("joao", "rpaas123"), 53 | ("rpaas", "joao123")] 54 | self.set_envs() 55 | self.addCleanup(self.delete_envs) 56 | for user, password in pairs: 57 | resp = self.get("/", user, password) 58 | self.assertEqual(401, resp.status_code) 59 | self.assertEqual("you do not have access to this resource", resp.data) 60 | 61 | def test_authentication_required_right_data(self): 62 | self.set_envs() 63 | self.addCleanup(self.delete_envs) 64 | resp = self.get("/", self.username, self.password) 65 | self.assertEqual(200, resp.status_code) 66 | self.assertEqual("hello world", resp.data) 67 | -------------------------------------------------------------------------------- /tests/test_hc.py: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | 3 | # Copyright 2016 rpaas authors. All rights reserved. 4 | # Use of this source code is governed by a BSD-style 5 | # license that can be found in the LICENSE file. 6 | 7 | import json 8 | import unittest 9 | import os 10 | 11 | import mock 12 | from requests import auth 13 | 14 | from rpaas import hc, storage 15 | 16 | 17 | class DumbTestCase(unittest.TestCase): 18 | 19 | def setUp(self): 20 | self.hc = hc.Dumb(None) 21 | 22 | def test_create(self): 23 | self.hc.create("hello") 24 | self.assertEqual([], self.hc.hcs["hello"]) 25 | 26 | def test_destroy(self): 27 | self.hc.create("hello") 28 | self.hc.destroy("hello") 29 | self.assertNotIn("hello", self.hc.hcs) 30 | 31 | def test_add_url(self): 32 | self.hc.create("hello") 33 | self.hc.add_url("hello", "myunit.tsuru.io") 34 | self.assertEqual(["myunit.tsuru.io"], self.hc.hcs["hello"]) 35 | 36 | def test_remove_url(self): 37 | self.hc.create("hello") 38 | self.hc.add_url("hello", "myunit.tsuru.io") 39 | self.hc.remove_url("hello", "myunit.tsuru.io") 40 | self.assertEqual([], self.hc.hcs["hello"]) 41 | 42 | 43 | class HCAPITestCase(unittest.TestCase): 44 | 45 | def setUp(self): 46 | os.environ['MONGO_DATABASE'] = 'host_manager_test' 47 | self.storage = storage.MongoDBStorage() 48 | colls = self.storage.db.collection_names(False) 49 | for coll in colls: 50 | self.storage.db.drop_collection(coll) 51 | self.hc = hc.HCAPI(self.storage, "http://localhost", hc_format="http://{}:8080/") 52 | self.hc._issue_request = self.issue_request = mock.Mock() 53 | 54 | @mock.patch("requests.request") 55 | def test_issue_request(self, request): 56 | request.return_value = resp = mock.Mock() 57 | hc_api = hc.HCAPI(self.storage, "http://localhost/") 58 | got_resp = hc_api._issue_request("GET", "/url", data={"abc": 1}) 59 | self.assertEqual(resp, got_resp) 60 | request.assert_called_with("GET", "http://localhost/url", 61 | data={"abc": 1}) 62 | 63 | @mock.patch("requests.request") 64 | def test_issue_authenticated_request(self, request): 65 | request.return_value = resp = mock.Mock() 66 | hc_api = hc.HCAPI(self.storage, "http://localhost/", 67 | user="zabbix", password="zabbix123") 68 | got_resp = hc_api._issue_request("GET", "/url", data={"abc": 1}) 69 | self.assertEqual(resp, got_resp) 70 | call = request.call_args_list[0] 71 | self.assertEqual(("GET", "http://localhost/url"), call[0]) 72 | self.assertEqual({"abc": 1}, call[1]["data"]) 73 | req_auth = call[1]["auth"] 74 | self.assertIsInstance(req_auth, auth.HTTPBasicAuth) 75 | self.assertEqual("zabbix", req_auth.username) 76 | self.assertEqual("zabbix123", req_auth.password) 77 | 78 | @mock.patch("uuid.uuid4") 79 | def test_create(self, uuid4): 80 | self.issue_request.return_value = mock.Mock(status_code=201) 81 | uuid = mock.Mock(hex="abc123") 82 | uuid4.return_value = uuid 83 | self.hc.create("myinstance") 84 | self.issue_request.assert_called_with("POST", "/resources", 85 | data={"name": 86 | "rpaas_myinstance_abc123"}) 87 | hc = self.storage.retrieve_hc("myinstance") 88 | self.assertDictEqual(hc, { 89 | "_id": "myinstance", "resource_name": "rpaas_myinstance_abc123" 90 | }) 91 | 92 | def test_create_response_error(self): 93 | self.issue_request.return_value = mock.Mock(status_code=409, 94 | text="something went wrong") 95 | with self.assertRaises(hc.HCCreationError) as cm: 96 | self.hc.create("myinstance") 97 | exc = cm.exception 98 | self.assertEqual(("something went wrong",), exc.args) 99 | 100 | def test_destroy(self): 101 | self.storage.store_hc({"_id": "myinstance", "resource_name": "my_resource"}) 102 | self.hc.destroy("myinstance") 103 | self.issue_request.assert_called_with("DELETE", "/resources/my_resource") 104 | 105 | def test_add_url(self): 106 | self.storage.store_hc({"_id": "myinstance", "resource_name": "my_resource"}) 107 | self.issue_request.return_value = mock.Mock(status_code=200) 108 | self.hc.add_url("myinstance", "something.tsuru.io") 109 | hcheck_url = "http://something.tsuru.io:8080/" 110 | data = {"name": "my_resource", "url": hcheck_url, 111 | "expected_string": "WORKING"} 112 | self.issue_request.assert_called_with("POST", "/url", 113 | data=json.dumps(data)) 114 | hc = self.storage.retrieve_hc("myinstance") 115 | self.assertDictEqual(hc, { 116 | "_id": "myinstance", 117 | "resource_name": "my_resource", 118 | "urls": ["http://something.tsuru.io:8080/"] 119 | }) 120 | 121 | def test_add_second_url(self): 122 | self.storage.store_hc({ 123 | "_id": "myinstance", "resource_name": "my_resource", 124 | "urls": ["http://a.com:8080/health"], 125 | }) 126 | self.issue_request.return_value = mock.Mock(status_code=200) 127 | self.hc.add_url("myinstance", "something.tsuru.io") 128 | hcheck_url = "http://something.tsuru.io:8080/" 129 | data = {"name": "my_resource", "url": hcheck_url, 130 | "expected_string": "WORKING"} 131 | self.issue_request.assert_called_with("POST", "/url", 132 | data=json.dumps(data)) 133 | hc = self.storage.retrieve_hc("myinstance") 134 | self.assertDictEqual(hc, { 135 | "_id": "myinstance", 136 | "resource_name": "my_resource", 137 | "urls": ["http://a.com:8080/health", "http://something.tsuru.io:8080/"] 138 | }) 139 | 140 | def test_add_url_request_error(self): 141 | self.storage.store_hc({ 142 | "_id": "myinstance", 143 | "resource_name": "my_resource", 144 | }) 145 | self.issue_request.return_value = mock.Mock(status_code=500, 146 | text="failed to add url") 147 | with self.assertRaises(hc.URLCreationError) as cm: 148 | self.hc.add_url("myinstance", "wat") 149 | exc = cm.exception 150 | self.assertEqual(("failed to add url",), exc.args) 151 | 152 | def test_remove_url(self): 153 | hcheck_url = "http://something.tsuru.io:8080/" 154 | self.storage.store_hc({ 155 | "_id": "myinstance", "resource_name": "my_resource", 156 | "urls": [hcheck_url] 157 | }) 158 | self.hc.remove_url("myinstance", "something.tsuru.io") 159 | data = {"name": "my_resource", "url": hcheck_url} 160 | self.issue_request.assert_called_with("DELETE", "/url", 161 | data=json.dumps(data)) 162 | hc = self.storage.retrieve_hc("myinstance") 163 | self.assertDictEqual(hc, { 164 | "_id": "myinstance", 165 | "resource_name": "my_resource", 166 | "urls": [] 167 | }) 168 | -------------------------------------------------------------------------------- /tests/test_le_renewer.py: -------------------------------------------------------------------------------- 1 | # Copyright 2016 rpaas authors. All rights reserved. 2 | # Use of this source code is governed by a BSD-style 3 | # license that can be found in the LICENSE file. 4 | 5 | import datetime 6 | import mock 7 | import time 8 | import unittest 9 | 10 | import redis 11 | 12 | from rpaas import storage, tasks 13 | from rpaas.ssl_plugins import le_renewer 14 | 15 | tasks.app.conf.CELERY_ALWAYS_EAGER = True 16 | 17 | 18 | class LeRenewerTestCase(unittest.TestCase): 19 | 20 | def setUp(self): 21 | self.config = { 22 | "MONGO_DATABASE": "le_renewer_test", 23 | "RPAAS_SERVICE_NAME": "test_rpaas_renewer", 24 | "LE_RENEWER_RUN_INTERVAL": 2, 25 | } 26 | self.storage = storage.MongoDBStorage(self.config) 27 | colls = self.storage.db.collection_names(False) 28 | for coll in colls: 29 | self.storage.db.drop_collection(coll) 30 | 31 | now = datetime.datetime.utcnow() 32 | certs = [ 33 | {"_id": "instance0", "domain": "i0.tsuru.io", 34 | "created": now - datetime.timedelta(days=88)}, 35 | {"_id": "instance1", "domain": "i1.tsuru.io", 36 | "created": now - datetime.timedelta(days=89)}, 37 | {"_id": "instance2", "domain": "i2.tsuru.io", 38 | "created": now}, 39 | {"_id": "instance3", "domain": "i3.tsuru.io", 40 | "created": now - datetime.timedelta(days=30)}, 41 | {"_id": "instance4", "domain": "i4.tsuru.io", 42 | "created": now - datetime.timedelta(days=90)}, 43 | {"_id": "instance5", "domain": "i5.tsuru.io", 44 | "created": now - datetime.timedelta(days=365)}, 45 | {"_id": "instance6", "domain": "i6.tsuru.io", 46 | "created": now - datetime.timedelta(days=87)}, 47 | {"_id": "instance7", "domain": "i7.tsuru.io", 48 | "created": now - datetime.timedelta(days=86)}, 49 | ] 50 | for cert in certs: 51 | self.storage.db[self.storage.le_certificates_collection].insert(cert) 52 | redis.StrictRedis().delete("le_renewer:last_run") 53 | 54 | def tearDown(self): 55 | self.storage.db[self.storage.le_certificates_collection].remove() 56 | 57 | @mock.patch("rpaas.sslutils.generate_crt") 58 | @mock.patch("rpaas.sslutils.generate_csr") 59 | @mock.patch("rpaas.sslutils.generate_key") 60 | def test_renew_certificates(self, generate_key, generate_csr, generate_crt): 61 | generate_key.return_value = "secret-key" 62 | generate_csr.return_value = "domain-csr" 63 | renewer = le_renewer.LeRenewer(self.config) 64 | renewer.start() 65 | time.sleep(1) 66 | renewer.stop() 67 | self.assertEqual([mock.call(True)] * 5, generate_key.mock_calls) 68 | expected_csr_calls = [mock.call("secret-key", "i0.tsuru.io"), 69 | mock.call("secret-key", "i1.tsuru.io"), 70 | mock.call("secret-key", "i4.tsuru.io"), 71 | mock.call("secret-key", "i5.tsuru.io"), 72 | mock.call("secret-key", "i6.tsuru.io")] 73 | self.assertEqual(expected_csr_calls, generate_csr.mock_calls) 74 | expected_crt_calls = [mock.call(self.config, "instance0", "le", "domain-csr", 75 | "secret-key", "i0.tsuru.io"), 76 | mock.call(self.config, "instance1", "le", "domain-csr", 77 | "secret-key", "i1.tsuru.io"), 78 | mock.call(self.config, "instance4", "le", "domain-csr", 79 | "secret-key", "i4.tsuru.io"), 80 | mock.call(self.config, "instance5", "le", "domain-csr", 81 | "secret-key", "i5.tsuru.io"), 82 | mock.call(self.config, "instance6", "le", "domain-csr", 83 | "secret-key", "i6.tsuru.io")] 84 | self.assertEqual(expected_crt_calls, generate_crt.mock_calls) 85 | -------------------------------------------------------------------------------- /tests/test_lock.py: -------------------------------------------------------------------------------- 1 | # Copyright 2017 rpaas authors. All rights reserved. 2 | # Use of this source code is governed by a BSD-style 3 | # license that can be found in the LICENSE file. 4 | 5 | import redis 6 | import unittest 7 | import time 8 | from rpaas import lock 9 | 10 | 11 | class LockManagerTestCase(unittest.TestCase): 12 | 13 | def setUp(self): 14 | self.redis_conn = redis.StrictRedis() 15 | self.redis_conn.flushall() 16 | 17 | def test_create_lock_when_empty_locks(self): 18 | lock_manager = lock.Lock(self.redis_conn) 19 | self.assertEqual(len(lock_manager.redis_locks), 0) 20 | lock_acquire = lock_manager.lock("lock1", 60) 21 | self.assertEqual(len(lock_manager.redis_locks), 1) 22 | self.assertTrue(lock_acquire) 23 | self.assertEqual(lock_manager.redis_locks[0].name, "lock1") 24 | 25 | def test_create_lock_try_to_acquire_lock_in_use(self): 26 | lock_manager = lock.Lock(self.redis_conn) 27 | lock_acquire = lock_manager.lock("lock1", 60) 28 | lock_acquire = lock_manager.lock("lock1", 60) 29 | self.assertEqual(len(lock_manager.redis_locks), 1) 30 | self.assertFalse(lock_acquire) 31 | 32 | def test_create_multiple_locks(self): 33 | lock_manager = lock.Lock(self.redis_conn) 34 | lock_acquire_1 = lock_manager.lock("lock1", 60) 35 | lock_acquire_2 = lock_manager.lock("lock2", 60) 36 | lock_acquire_3 = lock_manager.lock("lock3", 60) 37 | self.assertEqual(len(lock_manager.redis_locks), 3) 38 | self.assertTrue(lock_acquire_1) 39 | self.assertTrue(lock_acquire_2) 40 | self.assertTrue(lock_acquire_3) 41 | 42 | def test_unlock_and_release_lock(self): 43 | lock_manager = lock.Lock(self.redis_conn) 44 | lock_manager.lock("lock1", 60) 45 | lock1 = lock_manager.redis_locks[0] 46 | lock_manager.unlock("lock1") 47 | self.assertEqual(len(lock_manager.redis_locks), 0) 48 | with self.assertRaises(redis.exceptions.LockError) as cm: 49 | lock1.release() 50 | self.assertEqual(cm.exception.message, "Cannot release an unlocked lock") 51 | 52 | def test_unlock_and_release_lock_with_multiple_locks(self): 53 | lock_manager = lock.Lock(self.redis_conn) 54 | lock_acquire_1 = lock_manager.lock("lock1", 60) 55 | lock_acquire_2 = lock_manager.lock("lock2", 60) 56 | lock_acquire_3 = lock_manager.lock("lock3", 60) 57 | self.assertTrue(lock_acquire_1) 58 | self.assertTrue(lock_acquire_2) 59 | self.assertTrue(lock_acquire_3) 60 | lock2 = lock_manager.redis_locks[1] 61 | self.assertEqual(lock2.name, "lock2") 62 | lock_manager.unlock("lock2") 63 | self.assertEqual(len(lock_manager.redis_locks), 2) 64 | lock_acquire_1 = lock_manager.lock("lock1", 60) 65 | lock_acquire_2 = lock_manager.lock("lock2", 60) 66 | lock_acquire_3 = lock_manager.lock("lock3", 60) 67 | self.assertFalse(lock_acquire_1) 68 | self.assertTrue(lock_acquire_2) 69 | self.assertFalse(lock_acquire_3) 70 | 71 | def test_extend_lock_extra_time(self): 72 | lock_manager = lock.Lock(self.redis_conn) 73 | lock_acquire = lock_manager.lock("lock1", 1) 74 | time.sleep(2) 75 | with self.assertRaises(redis.exceptions.LockError) as cm: 76 | lock_manager.extend_lock("lock1", 30) 77 | self.assertEqual(cm.exception.message, "Cannot extend a lock that's no longer owned") 78 | self.assertTrue(lock_acquire) 79 | lock_1 = lock_manager.redis_locks[0] 80 | lock_1.acquire(blocking=False) 81 | lock_manager.extend_lock("lock1", 30) 82 | time.sleep(3) 83 | self.assertFalse(lock_1.acquire(blocking=False)) 84 | -------------------------------------------------------------------------------- /tests/test_nginx.py: -------------------------------------------------------------------------------- 1 | # Copyright 2016 rpaas authors. All rights reserved. 2 | # Use of this source code is governed by a BSD-style 3 | # license that can be found in the LICENSE file. 4 | 5 | import unittest 6 | 7 | import mock 8 | 9 | from rpaas.nginx import Nginx, NginxError 10 | 11 | 12 | class NginxTestCase(unittest.TestCase): 13 | 14 | def setUp(self): 15 | self.cache_headers = [{'Accept-Encoding': 'gzip'}, {'Accept-Encoding': 'identity'}] 16 | 17 | def test_init_default(self): 18 | nginx = Nginx() 19 | self.assertEqual(nginx.nginx_manage_port, '8089') 20 | self.assertEqual(nginx.nginx_purge_path, '/purge') 21 | self.assertEqual(nginx.nginx_healthcheck_path, '/healthcheck') 22 | 23 | def test_init_config(self): 24 | nginx = Nginx({ 25 | 'NGINX_PURGE_PATH': '/2', 26 | 'NGINX_MANAGE_PORT': '4', 27 | 'NGINX_LOCATION_TEMPLATE_DEFAULT_TXT': '5', 28 | 'NGINX_LOCATION_TEMPLATE_ROUTER_TXT': '6', 29 | 'NGINX_HEALTHCHECK_PATH': '7', 30 | }) 31 | self.assertEqual(nginx.nginx_purge_path, '/2') 32 | self.assertEqual(nginx.nginx_manage_port, '4') 33 | self.assertEqual(nginx.config_manager.location_template_default, '5') 34 | self.assertEqual(nginx.config_manager.location_template_router, '6') 35 | self.assertEqual(nginx.nginx_healthcheck_path, '7') 36 | 37 | @mock.patch('rpaas.nginx.requests') 38 | def test_init_config_location_url(self, requests): 39 | def mocked_requests_get(*args, **kwargs): 40 | class MockResponse: 41 | def __init__(self, text, status_code): 42 | self.text = text 43 | self.status_code = status_code 44 | if args[0] == 'http://my.com/default': 45 | return MockResponse("my result default", 200) 46 | elif args[0] == 'http://my.com/router': 47 | return MockResponse("my result router", 200) 48 | 49 | with mock.patch('rpaas.nginx.requests.get', side_effect=mocked_requests_get) as requests_get: 50 | nginx = Nginx({ 51 | 'NGINX_LOCATION_TEMPLATE_DEFAULT_URL': 'http://my.com/default', 52 | 'NGINX_LOCATION_TEMPLATE_ROUTER_URL': 'http://my.com/router', 53 | }) 54 | self.assertEqual(nginx.config_manager.location_template_default, 'my result default') 55 | self.assertEqual(nginx.config_manager.location_template_router, 'my result router') 56 | expected_calls = [mock.call('http://my.com/default'), 57 | mock.call('http://my.com/router')] 58 | requests_get.assert_has_calls(expected_calls) 59 | 60 | @mock.patch('rpaas.nginx.requests') 61 | def test_purge_location_successfully(self, requests): 62 | nginx = Nginx() 63 | 64 | response = mock.Mock() 65 | response.status_code = 200 66 | response.text = 'purged' 67 | 68 | side_effect = mock.Mock() 69 | side_effect.status_code = 404 70 | side_effect.text = "Not Found" 71 | 72 | requests.request.side_effect = [response, side_effect, response, side_effect] 73 | purged = nginx.purge_location('myhost', '/foo/bar') 74 | self.assertTrue(purged) 75 | self.assertEqual(requests.request.call_count, 4) 76 | expec_responses = [] 77 | for scheme in ['http', 'https']: 78 | for header in self.cache_headers: 79 | expec_responses.append(mock.call('get', 'http://myhost:8089/purge/{}/foo/bar'.format(scheme), 80 | headers=header, timeout=2)) 81 | requests.request.assert_has_calls(expec_responses) 82 | 83 | @mock.patch('rpaas.nginx.requests') 84 | def test_purge_location_preserve_path_successfully(self, requests): 85 | nginx = Nginx() 86 | 87 | response = mock.Mock() 88 | response.status_code = 200 89 | response.text = 'purged' 90 | 91 | requests.request.side_effect = [response] 92 | purged = nginx.purge_location('myhost', 'http://example.com/foo/bar', True) 93 | self.assertTrue(purged) 94 | self.assertEqual(requests.request.call_count, 2) 95 | expected_responses = [] 96 | for header in self.cache_headers: 97 | expected_responses.append(mock.call('get', 'http://myhost:8089/purge/http://example.com/foo/bar', 98 | headers=header, timeout=2)) 99 | requests.request.assert_has_calls(expected_responses) 100 | 101 | @mock.patch('rpaas.nginx.requests') 102 | def test_purge_location_not_found(self, requests): 103 | nginx = Nginx() 104 | 105 | response = mock.Mock() 106 | response.status_code = 404 107 | response.text = 'Not Found' 108 | 109 | requests.request.side_effect = [response, response, response, response] 110 | purged = nginx.purge_location('myhost', '/foo/bar') 111 | self.assertFalse(purged) 112 | self.assertEqual(requests.request.call_count, 4) 113 | expec_responses = [] 114 | for scheme in ['http', 'https']: 115 | for header in self.cache_headers: 116 | expec_responses.append(mock.call('get', 'http://myhost:8089/purge/{}/foo/bar'.format(scheme), 117 | headers=header, timeout=2)) 118 | requests.request.assert_has_calls(expec_responses) 119 | 120 | @mock.patch('rpaas.nginx.requests') 121 | def test_wait_healthcheck(self, requests): 122 | nginx = Nginx() 123 | count = [0] 124 | response = mock.Mock() 125 | response.status_code = 200 126 | response.text = 'WORKING' 127 | 128 | def side_effect(method, url, timeout, **params): 129 | count[0] += 1 130 | if count[0] < 2: 131 | raise Exception('some error') 132 | return response 133 | 134 | requests.request.side_effect = side_effect 135 | nginx.wait_healthcheck('myhost.com', timeout=5) 136 | self.assertEqual(requests.request.call_count, 2) 137 | requests.request.assert_called_with('get', 'http://myhost.com:8089/healthcheck', timeout=2) 138 | 139 | @mock.patch('rpaas.nginx.requests') 140 | def test_wait_app_healthcheck(self, requests): 141 | nginx = Nginx() 142 | count = [0] 143 | response = mock.Mock() 144 | response.status_code = 200 145 | response.text = '\n\nWORKING' 146 | 147 | def side_effect(method, url, timeout, **params): 148 | count[0] += 1 149 | if count[0] < 2: 150 | raise Exception('some error') 151 | return response 152 | 153 | requests.request.side_effect = side_effect 154 | nginx.wait_healthcheck('myhost.com', timeout=5, manage_healthcheck=False) 155 | self.assertEqual(requests.request.call_count, 2) 156 | requests.request.assert_called_with('get', 'http://myhost.com:8080/_nginx_healthcheck/', timeout=2) 157 | 158 | @mock.patch('rpaas.nginx.requests') 159 | def test_wait_app_healthcheck_invalid_response(self, requests): 160 | nginx = Nginx() 161 | count = [0] 162 | response = mock.Mock() 163 | response.status_code = 200 164 | response.text = '\nFAIL\n' 165 | 166 | def side_effect(method, url, timeout, **params): 167 | count[0] += 1 168 | if count[0] < 2: 169 | raise Exception('some error') 170 | return response 171 | 172 | requests.request.side_effect = side_effect 173 | with self.assertRaises(NginxError): 174 | nginx.wait_healthcheck('myhost.com', timeout=5, manage_healthcheck=False) 175 | self.assertEqual(requests.request.call_count, 6) 176 | requests.request.assert_called_with('get', 'http://myhost.com:8080/_nginx_healthcheck/', timeout=2) 177 | 178 | @mock.patch('rpaas.nginx.requests') 179 | def test_wait_healthcheck_timeout(self, requests): 180 | nginx = Nginx() 181 | 182 | def side_effect(method, url, timeout, **params): 183 | raise Exception('some error') 184 | 185 | requests.request.side_effect = side_effect 186 | with self.assertRaises(Exception): 187 | nginx.wait_healthcheck('myhost.com', timeout=2) 188 | self.assertGreaterEqual(requests.request.call_count, 2) 189 | requests.request.assert_called_with('get', 'http://myhost.com:8089/healthcheck', timeout=2) 190 | 191 | @mock.patch('os.path') 192 | @mock.patch('rpaas.nginx.requests') 193 | def test_add_session_ticket_success(self, requests, os_path): 194 | nginx = Nginx({'CA_CERT': 'cert data'}) 195 | os_path.exists.return_value = True 196 | response = mock.Mock() 197 | response.status_code = 200 198 | response.text = '\n\nticket was succsessfully added' 199 | requests.request.return_value = response 200 | nginx.add_session_ticket('host-1', 'random data', timeout=2) 201 | requests.request.assert_called_once_with('post', 'https://host-1:8090/session_ticket', timeout=2, 202 | data='random data', verify='/tmp/rpaas_ca.pem') 203 | 204 | @mock.patch('rpaas.nginx.requests') 205 | def test_missing_ca_cert(self, requests): 206 | nginx = Nginx() 207 | with self.assertRaises(NginxError): 208 | nginx.add_session_ticket('host-1', 'random data', timeout=2) 209 | -------------------------------------------------------------------------------- /tests/test_session_resumption.py: -------------------------------------------------------------------------------- 1 | # Copyright 2017 rpaas authors. All rights reserved. 2 | # Use of this source code is governed by a BSD-style 3 | # license that can be found in the LICENSE file. 4 | 5 | import datetime 6 | import time 7 | import unittest 8 | import redis 9 | import consul 10 | 11 | from freezegun import freeze_time 12 | from mock import patch, call 13 | from rpaas import storage, tasks 14 | from rpaas import session_resumption, consul_manager 15 | from cryptography import x509 16 | from cryptography.hazmat.backends import default_backend 17 | from cryptography.hazmat.primitives import hashes, serialization 18 | from cryptography.hazmat.primitives.asymmetric import rsa 19 | from cryptography.x509.oid import NameOID 20 | 21 | tasks.app.conf.CELERY_ALWAYS_EAGER = True 22 | 23 | 24 | class LoadBalancerFake(object): 25 | 26 | def __init__(self, name): 27 | self.name = name 28 | self.hosts = [] 29 | 30 | 31 | class HostFake(object): 32 | 33 | def __init__(self, id, group, dns_name): 34 | self.id = id 35 | self.group = group 36 | self.dns_name = dns_name 37 | self.fail_property = None 38 | 39 | def set_fail(self, name): 40 | self.fail_property = name 41 | 42 | def unset_fail(self, name): 43 | self.fail_property = None 44 | 45 | def __getattribute__(self, name): 46 | fail_property = object.__getattribute__(self, "fail_property") 47 | if fail_property and fail_property == name: 48 | raise AttributeError("{} not defined".format(name)) 49 | return object.__getattribute__(self, name) 50 | 51 | 52 | @freeze_time("2016-02-03 12:00:00") 53 | class SessionResumptionTestCase(unittest.TestCase): 54 | 55 | @classmethod 56 | def setUpClass(cls): 57 | cls.ca_key, cls.ca_cert = cls.generate_ca() 58 | 59 | @classmethod 60 | def generate_ca(cls): 61 | key = rsa.generate_private_key( 62 | public_exponent=65537, 63 | key_size=2048, 64 | backend=default_backend() 65 | ) 66 | subject = issuer = x509.Name([ 67 | x509.NameAttribute(NameOID.COUNTRY_NAME, u"BR"), 68 | x509.NameAttribute(NameOID.STATE_OR_PROVINCE_NAME, u"RJ"), 69 | x509.NameAttribute(NameOID.LOCALITY_NAME, u"Rio de Janeiro"), 70 | x509.NameAttribute(NameOID.ORGANIZATION_NAME, u"Tsuru Inc"), 71 | x509.NameAttribute(NameOID.COMMON_NAME, u"tsuru.io"), 72 | ]) 73 | cert = x509.CertificateBuilder().subject_name( 74 | subject 75 | ).issuer_name( 76 | issuer 77 | ).public_key( 78 | key.public_key() 79 | ).serial_number( 80 | x509.random_serial_number() 81 | ).not_valid_before( 82 | datetime.datetime.utcnow() 83 | ).not_valid_after( 84 | datetime.datetime.utcnow() + datetime.timedelta(days=10) 85 | ).add_extension( 86 | x509.SubjectAlternativeName([x509.DNSName(u"tsuru.io")]), 87 | critical=False, 88 | ).sign(key, hashes.SHA256(), default_backend()) 89 | 90 | key = key.private_bytes( 91 | encoding=serialization.Encoding.PEM, 92 | format=serialization.PrivateFormat.TraditionalOpenSSL, 93 | encryption_algorithm=serialization.NoEncryption(), 94 | ) 95 | cert = cert.public_bytes(serialization.Encoding.PEM) 96 | return key, cert 97 | 98 | def setUp(self): 99 | self.master_token = "rpaas-test" 100 | self.config = { 101 | "CONSUL_HOST": "127.0.0.1", 102 | "CONSUL_TOKEN": self.master_token, 103 | "MONGO_DATABASE": "session_resumption_test", 104 | "RPAAS_SERVICE_NAME": "test_rpaas_session_resumption", 105 | "HOST_MANAGER": "fake", 106 | "SESSION_RESUMPTION_RUN_INTERVAL": 2, 107 | u"CA_CERT": unicode(self.ca_cert), 108 | u"CA_KEY": unicode(self.ca_key) 109 | } 110 | self.consul = consul.Consul(token=self.master_token) 111 | self.consul.kv.delete("test_rpaas_session_resumption", recurse=True) 112 | self.storage = storage.MongoDBStorage(self.config) 113 | self.consul_manager = consul_manager.ConsulManager(self.config) 114 | colls = self.storage.db.collection_names(False) 115 | for coll in colls: 116 | self.storage.db.drop_collection(coll) 117 | redis.StrictRedis().flushall() 118 | 119 | @patch("rpaas.tasks.sslutils.generate_session_ticket") 120 | @patch("rpaas.tasks.LoadBalancer") 121 | @patch("rpaas.tasks.nginx") 122 | def test_renew_session_tickets(self, nginx, load_balancer, ticket): 123 | nginx_manager = nginx.Nginx.return_value 124 | lb1 = LoadBalancerFake("instance-a") 125 | lb2 = LoadBalancerFake("instance-b") 126 | lb1.hosts = [HostFake("xxx", "instance-a", "10.1.1.1"), HostFake("yyy", "instance-a", "10.1.1.2")] 127 | lb2.hosts = [HostFake("aaa", "instance-b", "10.2.2.2"), HostFake("bbb", "instance-b", "10.2.2.3")] 128 | load_balancer.list.return_value = [lb1, lb2] 129 | ticket.side_effect = ["ticket1", "ticket2", "ticket3", "ticket4"] 130 | session = session_resumption.SessionResumption(self.config) 131 | session.start() 132 | time.sleep(1) 133 | session.stop() 134 | nginx_expected_calls = [call('10.1.1.1', 'ticket1', 30), call('10.1.1.2', 'ticket1', 30), 135 | call('10.2.2.2', 'ticket2', 30), call('10.2.2.3', 'ticket2', 30)] 136 | self.assertEqual(nginx_expected_calls, nginx_manager.add_session_ticket.call_args_list) 137 | cert_a, key_a = self.consul_manager.get_certificate("instance-a", "xxx") 138 | cert_b, key_b = self.consul_manager.get_certificate("instance-b", "bbb") 139 | redis.StrictRedis().delete("session_resumption:test_rpaas_session_resumption:last_run") 140 | nginx_manager.reset_mock() 141 | session = session_resumption.SessionResumption(self.config) 142 | session.start() 143 | time.sleep(1) 144 | session.stop() 145 | nginx_expected_calls = [call('10.1.1.1', 'ticket3', 30), call('10.1.1.2', 'ticket3', 30), 146 | call('10.2.2.2', 'ticket4', 30), call('10.2.2.3', 'ticket4', 30)] 147 | self.assertEqual(nginx_expected_calls, nginx_manager.add_session_ticket.call_args_list) 148 | self.assertTupleEqual((cert_a, key_a), self.consul_manager.get_certificate("instance-a", "xxx")) 149 | self.assertTupleEqual((cert_b, key_b), self.consul_manager.get_certificate("instance-b", "bbb")) 150 | 151 | @patch.object(tasks.SessionResumptionTask, "rotate_session_ticket", return_value=None) 152 | @patch("rpaas.tasks.LoadBalancer") 153 | def test_renew_session_tickets_only_on_selected_instances(self, load_balancer, rotate_session): 154 | self.config["SESSION_RESUMPTION_INSTANCES"] = "instance-a,instance-c" 155 | lb1 = LoadBalancerFake("instance-a") 156 | lb2 = LoadBalancerFake("instance-b") 157 | lb3 = LoadBalancerFake("instance-c") 158 | lb4 = LoadBalancerFake("instance-d") 159 | lb1.hosts = [HostFake("xxx", "instance-a", "10.1.1.1")] 160 | lb2.hosts = [HostFake("yyy", "instance-b", "10.2.1.1")] 161 | lb3.hosts = [HostFake("aaa", "instance-c", "10.3.2.2")] 162 | lb4.hosts = [HostFake("bbb", "instance-d", "10.4.2.2")] 163 | load_balancer.list.return_value = [lb1, lb2, lb3, lb4] 164 | session = session_resumption.SessionResumption(self.config) 165 | session.start() 166 | time.sleep(1) 167 | session.stop() 168 | self.assertEqual(rotate_session.call_args_list, [call(lb1.hosts), call(lb3.hosts)]) 169 | 170 | @patch("rpaas.tasks.logging") 171 | @patch("rpaas.tasks.sslutils.generate_session_ticket") 172 | @patch("rpaas.tasks.LoadBalancer") 173 | @patch("rpaas.tasks.nginx") 174 | def test_renew_session_tickets_fail_and_unlock(self, nginx, load_balancer, ticket, logging): 175 | nginx_manager = nginx.Nginx.return_value 176 | lb1_host2 = HostFake("yyy", "instance-a", "10.1.1.2") 177 | lb1_host2.set_fail("dns_name") 178 | lb1 = LoadBalancerFake("instance-a") 179 | lb2 = LoadBalancerFake("instance-b") 180 | lb1.hosts = [HostFake("xxx", "instance-a", "10.1.1.1"), lb1_host2] 181 | lb2.hosts = [HostFake("aaa", "instance-b", "10.2.2.2"), HostFake("bbb", "instance-b", "10.2.2.3")] 182 | load_balancer.list.return_value = [lb1, lb2] 183 | ticket.side_effect = ["ticket1", "ticket2", "ticket3", "ticket4"] 184 | session = session_resumption.SessionResumption(self.config) 185 | session.start() 186 | time.sleep(1) 187 | session.stop() 188 | nginx_expected_calls = [call('10.1.1.1', 'ticket1', 30), call('10.2.2.2', 'ticket2', 30), 189 | call('10.2.2.3', 'ticket2', 30)] 190 | self.assertEqual(nginx_expected_calls, nginx_manager.add_session_ticket.call_args_list) 191 | redis.StrictRedis().delete("session_resumption:test_rpaas_session_resumption:last_run") 192 | lb1_host2.unset_fail("dns_name") 193 | nginx_manager.reset_mock() 194 | session = session_resumption.SessionResumption(self.config) 195 | session.start() 196 | time.sleep(1) 197 | session.stop() 198 | nginx_expected_calls = [call('10.1.1.1', 'ticket3', 30), call('10.1.1.2', 'ticket3', 30), 199 | call('10.2.2.2', 'ticket4', 30), call('10.2.2.3', 'ticket4', 30)] 200 | self.assertEqual(nginx_expected_calls, nginx_manager.add_session_ticket.call_args_list) 201 | error_msg = "Error renewing session ticket for instance-a: AttributeError('dns_name not defined',)" 202 | logging.error.assert_called_with(error_msg) 203 | 204 | @patch("rpaas.tasks.logging") 205 | @patch("rpaas.tasks.sslutils.generate_admin_crt") 206 | @patch("rpaas.tasks.LoadBalancer") 207 | @patch("rpaas.tasks.nginx") 208 | def test_renew_session_tickets_return_first_error(self, nginx, load_balancer, generate_cert, logging): 209 | nginx_manager = nginx.Nginx.return_value 210 | lb1 = LoadBalancerFake("instance-a") 211 | lb1.hosts = [HostFake("xxx", "instance-a", "10.1.1.1")] 212 | load_balancer.list.return_value = [lb1] 213 | generate_cert.side_effect = Exception("could not generate certificate") 214 | nginx_manager.add_session_ticket.side_effect = Exception("nginx error connecting to host") 215 | session = session_resumption.SessionResumption(self.config) 216 | session.start() 217 | time.sleep(1) 218 | session.stop() 219 | error_msg = "Error renewing session ticket for instance-a: " \ 220 | "Exception('could not generate certificate',)" 221 | logging.error.assert_called_with(error_msg) 222 | nginx_manager.add_session_ticket.assert_not_called() 223 | -------------------------------------------------------------------------------- /tests/test_ssl_le.py: -------------------------------------------------------------------------------- 1 | # Copyright 2016 rpaas authors. All rights reserved. 2 | # Use of this source code is governed by a BSD-style 3 | # license that can be found in the LICENSE file. 4 | 5 | import unittest 6 | import mock 7 | 8 | from rpaas.ssl_plugins import le 9 | 10 | 11 | def patch_main(args): 12 | return "crt", "chain", "key" 13 | 14 | 15 | class LETest(unittest.TestCase): 16 | 17 | def setUp(self): 18 | self.patcher = mock.patch('rpaas.ssl_plugins.le._main', patch_main) 19 | self.patcher.start() 20 | self.instance = le.LE(['domain'], 'email@corp', ['host1']) 21 | 22 | def tearDown(self): 23 | self.patcher.stop() 24 | 25 | def test_upload_csr(self): 26 | self.assertEqual(self.instance.upload_csr('asdasdasdasdadasd'), None) 27 | 28 | def test_download_crt(self): 29 | with mock.patch.object(le.LE, 'download_crt', return_value=None) as mock_method: 30 | instance = self.instance 31 | instance.download_crt(None) 32 | mock_method.assert_called_once_with(None) 33 | -------------------------------------------------------------------------------- /tests/test_storage.py: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | 3 | # Copyright 2016 rpaas authors. All rights reserved. 4 | # Use of this source code is governed by a BSD-style 5 | # license that can be found in the LICENSE file. 6 | 7 | import datetime 8 | import unittest 9 | import os 10 | 11 | import freezegun 12 | 13 | from rpaas import plan, storage, flavor 14 | 15 | 16 | class MongoDBStorageTestCase(unittest.TestCase): 17 | 18 | def setUp(self): 19 | os.environ["MONGO_DATABASE"] = "storage_test" 20 | self.storage = storage.MongoDBStorage() 21 | colls = self.storage.db.collection_names(False) 22 | for coll in colls: 23 | self.storage.db.drop_collection(coll) 24 | self.storage.db[self.storage.plans_collection].insert( 25 | {"_id": "small", 26 | "description": "some cool plan", 27 | "config": {"serviceofferingid": "abcdef123456"}} 28 | ) 29 | self.storage.db[self.storage.plans_collection].insert( 30 | {"_id": "huge", 31 | "description": "some cool huge plan", 32 | "config": {"serviceofferingid": "abcdef123459"}} 33 | ) 34 | self.storage.db[self.storage.flavors_collection].insert( 35 | {"_id": "vanilla", 36 | "description": "nginx 1.10", 37 | "config": {"nginx_version": "1.10"}} 38 | ) 39 | self.storage.db[self.storage.flavors_collection].insert( 40 | {"_id": "orange", 41 | "description": "nginx 1.12", 42 | "config": {"nginx_version": "1.12"}} 43 | ) 44 | 45 | def test_set_team_quota(self): 46 | q = self.storage.set_team_quota("myteam", 8) 47 | used, quota = self.storage.find_team_quota("myteam") 48 | self.assertEqual([], used) 49 | self.assertEqual(8, quota) 50 | self.assertEqual(used, q["used"]) 51 | self.assertEqual(quota, q["quota"]) 52 | 53 | def test_list_plans(self): 54 | plans = self.storage.list_plans() 55 | expected = [ 56 | {"name": "small", "description": "some cool plan", 57 | "config": {"serviceofferingid": "abcdef123456"}}, 58 | {"name": "huge", "description": "some cool huge plan", 59 | "config": {"serviceofferingid": "abcdef123459"}}, 60 | ] 61 | self.assertEqual(expected, [p.to_dict() for p in plans]) 62 | 63 | def test_find_plan(self): 64 | plan = self.storage.find_plan("small") 65 | expected = {"name": "small", "description": "some cool plan", 66 | "config": {"serviceofferingid": "abcdef123456"}} 67 | self.assertEqual(expected, plan.to_dict()) 68 | with self.assertRaises(storage.PlanNotFoundError): 69 | self.storage.find_plan("something that doesn't exist") 70 | 71 | def test_store_plan(self): 72 | p = plan.Plan(name="super_huge", description="very huge thing", 73 | config={"serviceofferingid": "abcdef123"}) 74 | self.storage.store_plan(p) 75 | got_plan = self.storage.find_plan(p.name) 76 | self.assertEqual(p.to_dict(), got_plan.to_dict()) 77 | 78 | def test_store_plan_duplicate(self): 79 | p = plan.Plan(name="small", description="small thing", 80 | config={"serviceofferingid": "abcdef123"}) 81 | with self.assertRaises(storage.DuplicateError): 82 | self.storage.store_plan(p) 83 | 84 | def test_update_plan(self): 85 | p = plan.Plan(name="super_huge", description="very huge thing", 86 | config={"serviceofferingid": "abcdef123"}) 87 | self.storage.store_plan(p) 88 | self.storage.update_plan(p.name, description="wat?", 89 | config={"serviceofferingid": "abcdef123459"}) 90 | p = self.storage.find_plan(p.name) 91 | self.assertEqual("super_huge", p.name) 92 | self.assertEqual("wat?", p.description) 93 | self.assertEqual({"serviceofferingid": "abcdef123459"}, p.config) 94 | 95 | def test_update_plan_partial(self): 96 | p = plan.Plan(name="super_huge", description="very huge thing", 97 | config={"serviceofferingid": "abcdef123"}) 98 | self.storage.store_plan(p) 99 | self.storage.update_plan(p.name, config={"serviceofferingid": "abcdef123459"}) 100 | p = self.storage.find_plan(p.name) 101 | self.assertEqual("super_huge", p.name) 102 | self.assertEqual("very huge thing", p.description) 103 | self.assertEqual({"serviceofferingid": "abcdef123459"}, p.config) 104 | 105 | def test_update_plan_not_found(self): 106 | with self.assertRaises(storage.PlanNotFoundError): 107 | self.storage.update_plan("my_plan", description="woot") 108 | 109 | def test_delete_plan(self): 110 | p = plan.Plan(name="super_huge", description="very huge thing", 111 | config={"serviceofferingid": "abcdef123"}) 112 | self.storage.store_plan(p) 113 | self.storage.delete_plan(p.name) 114 | with self.assertRaises(storage.PlanNotFoundError): 115 | self.storage.find_plan(p.name) 116 | 117 | def test_delete_plan_not_found(self): 118 | with self.assertRaises(storage.PlanNotFoundError): 119 | self.storage.delete_plan("super_huge") 120 | 121 | def test_list_flavors(self): 122 | flavors = self.storage.list_flavors() 123 | expected = [ 124 | {"name": "vanilla", "description": "nginx 1.10", 125 | "config": {"nginx_version": "1.10"}}, 126 | {"name": "orange", "description": "nginx 1.12", 127 | "config": {"nginx_version": "1.12"}}, 128 | ] 129 | self.assertEqual(expected, [f.to_dict() for f in flavors]) 130 | 131 | def test_find_flavor(self): 132 | flavor = self.storage.find_flavor("vanilla") 133 | expected = {"name": "vanilla", "description": "nginx 1.10", 134 | "config": {"nginx_version": "1.10"}} 135 | self.assertEqual(expected, flavor.to_dict()) 136 | with self.assertRaises(storage.FlavorNotFoundError): 137 | self.storage.find_flavor("something that doesn't exist") 138 | 139 | def test_store_flavor(self): 140 | f = flavor.Flavor(name="lemon", description="nginx 1.13", 141 | config={"nginx_version": "1.13"}) 142 | self.storage.store_flavor(f) 143 | got_flavor = self.storage.find_flavor(f.name) 144 | self.assertEqual(f.to_dict(), got_flavor.to_dict()) 145 | 146 | def test_store_flavor_duplicate(self): 147 | f = flavor.Flavor(name="vanilla", description="nginx 1.10", 148 | config={"nginx_version": "1.10"}) 149 | with self.assertRaises(storage.DuplicateError): 150 | self.storage.store_flavor(f) 151 | 152 | def test_update_flavor(self): 153 | f = flavor.Flavor(name="lemon", description="nginx 1.13", 154 | config={"nginx_version": "1.13"}) 155 | self.storage.store_flavor(f) 156 | self.storage.update_flavor(f.name, description="nginx 1.13.1", 157 | config={"nginx_version": "1.13.1"}) 158 | f = self.storage.find_flavor(f.name) 159 | self.assertEqual("lemon", f.name) 160 | self.assertEqual("nginx 1.13.1", f.description) 161 | self.assertEqual({"nginx_version": "1.13.1"}, f.config) 162 | 163 | def test_update_flavor_partial(self): 164 | f = flavor.Flavor(name="lemon", description="nginx 1.13", 165 | config={"nginx_version": "1.13"}) 166 | self.storage.store_flavor(f) 167 | self.storage.update_flavor(f.name, config={"nginx_version": "1.13.1"}) 168 | f = self.storage.find_flavor(f.name) 169 | self.assertEqual("lemon", f.name) 170 | self.assertEqual("nginx 1.13", f.description) 171 | self.assertEqual({"nginx_version": "1.13.1"}, f.config) 172 | 173 | def test_update_flavor_not_found(self): 174 | with self.assertRaises(storage.FlavorNotFoundError): 175 | self.storage.update_flavor("lemon", description="nginx 1.13") 176 | 177 | def test_delete_flavor(self): 178 | f = flavor.Flavor(name="lemon", description="nginx 1.13", 179 | config={"nginx_version": "1.13"}) 180 | self.storage.store_flavor(f) 181 | self.storage.delete_flavor(f.name) 182 | with self.assertRaises(storage.FlavorNotFoundError): 183 | self.storage.find_flavor(f.name) 184 | 185 | def test_delete_flavor_not_found(self): 186 | with self.assertRaises(storage.FlavorNotFoundError): 187 | self.storage.delete_flavor("lemon") 188 | 189 | def test_instance_metadata_storage(self): 190 | self.storage.store_instance_metadata("myinstance", plan="small") 191 | inst_metadata = self.storage.find_instance_metadata("myinstance") 192 | self.assertEqual({"_id": "myinstance", 193 | "plan": "small"}, inst_metadata) 194 | self.storage.store_instance_metadata("myinstance", plan="medium") 195 | inst_metadata = self.storage.find_instance_metadata("myinstance") 196 | self.assertEqual({"_id": "myinstance", "plan": "medium"}, inst_metadata) 197 | self.storage.remove_instance_metadata("myinstance") 198 | inst_metadata = self.storage.find_instance_metadata("myinstance") 199 | self.assertIsNone(inst_metadata) 200 | 201 | @freezegun.freeze_time("2014-12-23 10:53:00", tz_offset=2) 202 | def test_store_le_certificate(self): 203 | self.storage.store_le_certificate("myinstance", "docs.tsuru.io") 204 | coll = self.storage.db[self.storage.le_certificates_collection] 205 | item = coll.find_one({"_id": "myinstance"}) 206 | expected = {"_id": "myinstance", "domain": "docs.tsuru.io", 207 | "created": datetime.datetime(2014, 12, 23, 10, 53, 0)} 208 | self.assertEqual(expected, item) 209 | 210 | @freezegun.freeze_time("2014-12-23 10:53:00", tz_offset=2) 211 | def test_store_le_certificate_overwrite(self): 212 | self.storage.store_le_certificate("myinstance", "docs.tsuru.io") 213 | self.storage.store_le_certificate("myinstance", "docs.tsuru.com") 214 | coll = self.storage.db[self.storage.le_certificates_collection] 215 | item = coll.find_one({"_id": "myinstance"}) 216 | expected = {"_id": "myinstance", "domain": "docs.tsuru.com", 217 | "created": datetime.datetime(2014, 12, 23, 10, 53, 0)} 218 | self.assertEqual(expected, item) 219 | 220 | def test_remove_le_certificate(self): 221 | self.storage.store_le_certificate("myinstance", "docs.tsuru.io") 222 | self.storage.remove_le_certificate("myinstance", "docs.tsuru.io") 223 | coll = self.storage.db[self.storage.le_certificates_collection] 224 | item = coll.find_one({"_id": "myinstance"}) 225 | self.assertIsNone(item) 226 | 227 | def test_remove_le_certificate_wrong_domain(self): 228 | self.storage.store_le_certificate("myinstance", "docs.tsuru.io") 229 | self.storage.remove_le_certificate("myinstance", "docs.tsuru.com") 230 | coll = self.storage.db[self.storage.le_certificates_collection] 231 | item = coll.find_one({"_id": "myinstance"}) 232 | self.assertIsNotNone(item) 233 | 234 | def test_find_le_certificates(self): 235 | self.storage.store_le_certificate("myinstance", "docs.tsuru.io") 236 | self.storage.store_le_certificate("myinstance", "docs.tsuru.com") 237 | certs_domain = list(self.storage.find_le_certificates({"domain": "docs.tsuru.com"})) 238 | certs_name = list(self.storage.find_le_certificates({"name": "myinstance"})) 239 | self.assertEqual(certs_name, certs_domain) 240 | self.assertEqual("myinstance", certs_name[0]["name"]) 241 | 242 | @freezegun.freeze_time("2016-08-02 10:53:00", tz_offset=2) 243 | def test_store_update_retrieve_healing(self): 244 | healing_id = self.storage.store_healing("myinstance", "10.10.1.1") 245 | loop_time = datetime.datetime.utcnow() 246 | for x in range(2, 5): 247 | loop_time = loop_time + datetime.timedelta(minutes=5) 248 | with freezegun.freeze_time(loop_time, tz_offset=2): 249 | healing_tmp = self.storage.store_healing("myinstance", "10.10.1.{}".format(x)) 250 | self.storage.update_healing(healing_tmp, "success") 251 | coll = self.storage.db[self.storage.healing_collection] 252 | item = coll.find_one({"_id": healing_id}) 253 | expected = {"_id": healing_id, "instance": "myinstance", "machine": "10.10.1.1", 254 | "start_time": datetime.datetime(2016, 8, 2, 10, 53, 0)} 255 | self.assertDictEqual(item, expected) 256 | with freezegun.freeze_time("2016-08-02 10:55:00", tz_offset=2): 257 | self.storage.update_healing(healing_id, "some random reason") 258 | item = coll.find_one({"_id": healing_id}) 259 | expected = {"_id": healing_id, "instance": "myinstance", "machine": "10.10.1.1", 260 | "start_time": datetime.datetime(2016, 8, 2, 10, 53, 0), 261 | "end_time": datetime.datetime(2016, 8, 2, 10, 55, 0), 262 | "status": "some random reason"} 263 | self.assertDictEqual(item, expected) 264 | loop_time = datetime.datetime.utcnow() + datetime.timedelta(minutes=5) 265 | expected = [] 266 | for x in range(2, 5): 267 | expected.append({"instance": "myinstance", "machine": "10.10.1.{}".format(x), 268 | "start_time": loop_time, "end_time": loop_time, "status": "success"}) 269 | loop_time = loop_time + datetime.timedelta(minutes=5) 270 | expected.reverse() 271 | healing_list = self.storage.list_healings(3) 272 | self.assertListEqual(healing_list, expected) 273 | -------------------------------------------------------------------------------- /tests/test_tasks.py: -------------------------------------------------------------------------------- 1 | # Copyright 2016 rpaas authors. All rights reserved. 2 | # Use of this source code is governed by a BSD-style 3 | # license that can be found in the LICENSE file. 4 | 5 | import unittest 6 | import os 7 | import redis 8 | import time 9 | 10 | from rpaas import tasks 11 | 12 | tasks.app.conf.CELERY_ALWAYS_EAGER = True 13 | 14 | 15 | class TasksTestCase(unittest.TestCase): 16 | 17 | def setUp(self): 18 | os.environ['REDIS_HOST'] = '' 19 | os.environ['REDIS_PORT'] = '' 20 | os.environ['DBAAS_SENTINEL_ENDPOINT'] = '' 21 | os.environ['SENTINEL_ENDPOINT'] = '' 22 | os.environ['REDIS_ENDPOINT'] = '' 23 | sentinel_conn = redis.StrictRedis().from_url("redis://127.0.0.1:51111") 24 | _, master_port = sentinel_conn.execute_command("sentinel get-master-addr-by-name mymaster") 25 | self.master_port = int(master_port) 26 | 27 | def tearDown(self): 28 | self.setUp() 29 | 30 | def with_env_var(self, env_var, env_value, redis_port): 31 | os.environ[env_var] = env_value 32 | app = tasks.initialize_celery() 33 | ch = app.broker_connection().channel() 34 | ch.client.set('mykey', env_var) 35 | self.assertEqual(ch.client.get('mykey'), env_var) 36 | self.assertEqual(ch.client.info()['tcp_port'], redis_port) 37 | self.assertEqual(app.backend.client.info()['tcp_port'], redis_port) 38 | 39 | def test_default_redis_connection(self): 40 | app = tasks.initialize_celery() 41 | ch = app.broker_connection().channel() 42 | ch.client.set('mykey', '1') 43 | self.assertEqual(ch.client.get('mykey'), '1') 44 | self.assertEqual(ch.client.info()['tcp_port'], 6379) 45 | self.assertEqual(app.backend.client.info()['tcp_port'], 6379) 46 | 47 | def test_sentinel_with_many_envs(self): 48 | self.with_env_var('RANDOM_VAR', 'random_value', 6379) 49 | self.setUp() 50 | sentinel_endpoint = 'sentinel://:mypass@127.0.0.1:51111,127.0.0.1:51112/service_name:mymaster' 51 | self.with_env_var('SENTINEL_ENDPOINT', sentinel_endpoint, self.master_port) 52 | self.setUp() 53 | dbaas_endpoint = sentinel_endpoint 54 | self.with_env_var('DBAAS_SENTINEL_ENDPOINT', dbaas_endpoint, self.master_port) 55 | self.setUp() 56 | redis_endpoint = 'redis://127.0.0.1:6379/0' 57 | self.with_env_var('REDIS_ENDPOINT', redis_endpoint, 6379) 58 | 59 | def test_simple_redis_string(self): 60 | os.environ['REDIS_ENDPOINT'] = "redis://:mypass@127.0.0.1:{}/0".format(self.master_port) 61 | app = tasks.initialize_celery() 62 | ch = app.broker_connection().channel() 63 | ch.client.set('mykey_simple_redis', '1') 64 | self.assertEqual(ch.client.get('mykey_simple_redis'), '1') 65 | self.assertEqual(ch.client.info()['tcp_port'], self.master_port) 66 | self.assertEqual(app.backend.client.info()['tcp_port'], self.master_port) 67 | 68 | def test_sentinel_master_failover(self): 69 | os.environ['SENTINEL_ENDPOINT'] = "sentinel://:mypass@127.0.0.1:51111/service_name:mymaster" 70 | app = tasks.initialize_celery() 71 | ch = app.broker_connection().channel() 72 | redis_conn = redis.StrictRedis().from_url("redis://:mypass@127.0.0.1:{}".format(self.master_port)) 73 | self.assertEqual(redis_conn.info()['role'], 'master') 74 | redis_conn = redis.StrictRedis().from_url("redis://127.0.0.1:51111") 75 | redis_conn.execute_command("sentinel failover mymaster") 76 | redis_conn = redis.StrictRedis().from_url("redis://:mypass@127.0.0.1:{}".format(self.master_port)) 77 | timeout_failover = 0 78 | while redis_conn.info()['role'] == 'slave' and timeout_failover <= 30: 79 | time.sleep(1) 80 | timeout_failover += 1 81 | self.assertEqual(ch.client.info()['role'], 'master') 82 | ch.client.set('mykey_failover', 'sentinel_failover_key') 83 | self.assertEqual(ch.client.get('mykey_failover'), 'sentinel_failover_key') 84 | self.assertEqual(app.backend.client.get('mykey_failover'), 'sentinel_failover_key') 85 | 86 | def redis_clients_manager(self, kill=False): 87 | redis_conn = redis.StrictRedis().from_url("redis://:mypass@127.0.0.1:{}".format(self.master_port)) 88 | client_conn_list = 0 89 | for client in redis_conn.client_list(): 90 | if 'sentinel' in client['name'] or client['cmd'] in ['replconf', 'client']: 91 | continue 92 | client_conn_list += 1 93 | if kill: 94 | redis_conn.execute_command("client kill addr {} skipme yes \ 95 | type normal".format(client['addr'])) 96 | return client_conn_list 97 | 98 | def test_sentinel_connection_pool_reconnect(self): 99 | os.environ['SENTINEL_ENDPOINT'] = "sentinel://:mypass@127.0.0.1:51111/service_name:mymaster" 100 | app = tasks.initialize_celery() 101 | app_client = app.backend.client 102 | self.redis_clients_manager(kill=True) 103 | app_client.set('mykey_connection_pool_client', 'sentinel_connection_pool') 104 | self.assertEqual(app_client.get('mykey_connection_pool_client'), 'sentinel_connection_pool') 105 | self.assertEqual(self.redis_clients_manager(), 1) 106 | 107 | def test_sentinel_connection_pool_shared(self): 108 | os.environ['SENTINEL_ENDPOINT'] = "sentinel://:mypass@127.0.0.1:51111/service_name:mymaster" 109 | app = tasks.initialize_celery() 110 | app_client = [] 111 | for x in range(10): 112 | app_client.append(app.backend.client) 113 | app_client[x].set('mykey_app_client_shared', 'sentinel_connection_shared_{}'.format(x)) 114 | self.assertEqual(app_client[x].get('mykey_app_client_shared'), 115 | 'sentinel_connection_shared_{}'.format(x)) 116 | self.assertEqual(id(app_client[0].connection_pool), id(app_client[9].connection_pool)) 117 | self.assertEqual(self.redis_clients_manager(), 1) 118 | -------------------------------------------------------------------------------- /tests/testdata/block_http: -------------------------------------------------------------------------------- 1 | content 2 | -------------------------------------------------------------------------------- /tests/testdata/location: -------------------------------------------------------------------------------- 1 | content 2 | -------------------------------------------------------------------------------- /tests/testdata/lua_module: -------------------------------------------------------------------------------- 1 | 'init globo_ab' -------------------------------------------------------------------------------- /tests/testdata/sentinel_conf/redis_sentinel2_test.conf: -------------------------------------------------------------------------------- 1 | sentinel monitor mymaster 127.0.0.1 51113 1 2 | sentinel auth-pass mymaster mypass 3 | sentinel down-after-milliseconds mymaster 5000 4 | sentinel failover-timeout mymaster 60000 5 | sentinel parallel-syncs mymaster 1 6 | 7 | port 51112 8 | daemonize yes 9 | dir "/tmp" 10 | bind 0.0.0.0 11 | -------------------------------------------------------------------------------- /tests/testdata/sentinel_conf/redis_sentinel_test.conf: -------------------------------------------------------------------------------- 1 | sentinel monitor mymaster 127.0.0.1 51113 1 2 | sentinel auth-pass mymaster mypass 3 | sentinel down-after-milliseconds mymaster 5000 4 | sentinel failover-timeout mymaster 60000 5 | sentinel parallel-syncs mymaster 1 6 | 7 | port 51111 8 | daemonize yes 9 | dir "/tmp" 10 | bind 0.0.0.0 11 | -------------------------------------------------------------------------------- /tests/testdata/sentinel_conf/redis_test.conf: -------------------------------------------------------------------------------- 1 | daemonize yes 2 | pidfile /tmp/redis_test-trs1.pid 3 | port 51113 4 | dbfilename redis_test1.rdb 5 | dir /tmp 6 | requirepass mypass 7 | masterauth mypass 8 | bind 0.0.0.0 9 | -------------------------------------------------------------------------------- /tests/testdata/sentinel_conf/redis_test2.conf: -------------------------------------------------------------------------------- 1 | daemonize yes 2 | pidfile /tmp/redis_test-trs2.pid 3 | port 51114 4 | slaveof 127.0.0.1 51113 5 | dbfilename redis_test2.rdb 6 | dir /tmp 7 | requirepass mypass 8 | masterauth mypass 9 | bind 0.0.0.0 10 | -------------------------------------------------------------------------------- /tests/testdata/sentinel_conf/redis_test3.conf: -------------------------------------------------------------------------------- 1 | daemonize yes 2 | pidfile /tmp/redis_test-trs3.pid 3 | port 51115 4 | slaveof 127.0.0.1 51113 5 | dbfilename redis_test3.rdb 6 | dir /tmp 7 | requirepass mypass 8 | masterauth mypass 9 | bind 0.0.0.0 10 | --------------------------------------------------------------------------------