├── MANIFEST.in ├── README.md ├── nesoi ├── __init__.py ├── api.py ├── keystore.py ├── model.py ├── rest.py └── service.py ├── setup.py └── twisted └── plugins └── nesoi_plugin.py /MANIFEST.in: -------------------------------------------------------------------------------- 1 | include *.md 2 | recursive-include nesoi *.py 3 | recursive-include twisted *.py 4 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Nesoi # 2 | 3 | _Nesoi_ is a coordination and configuration manager for distributed 4 | applications. It allows clients to find services to communicate with, 5 | and services to find its configuration. It also allows clients to 6 | register webhooks that will be notified when something changes. 7 | 8 | _Nesoi_ has three concepts: 9 | 10 | * _Applications_ stores configuration for a service. 11 | * _Service_ holds information about instances of a service. 12 | * _Host_ communicate endpoints for an instance of a service. 13 | 14 | The normal use-case is this: When an application is bootstrapped it 15 | reads out its configuration from an app-resource. When done, it 16 | registers itself as a host for a service that it provides. Clients 17 | find endpoints for a service by inspecting `/srv/NAME`. 18 | 19 | Applications are located under the `/app` tree. A simple example 20 | of creating an application configuration: 21 | 22 | $ curl -X PUT -d '{"name":"dm", "config":{}}' http://localhost:6553/app/dm 23 | 24 | Each app object needs a `name` and `config` attribute. 25 | 26 | To list all available applications, issue a GET to `/app`: 27 | 28 | $ curl http://localhost:6553/app 29 | { 30 | "apps": ["dm"] 31 | } 32 | 33 | When your service has been bootstrapped, register the host instance 34 | with the service using a simple command like this: 35 | 36 | $ curl -X PUT -d '{"endpoints":{"http":"http://localhost:5432/"}}' http://localhost:6553/srv/dm/host1 37 | { 38 | "endpoints": { 39 | "http": "http://localhost:5432/" 40 | }, 41 | "updated_at": 1319523542 42 | } 43 | 44 | `endpoints` property is required. Hosts, and apps can of course be 45 | deleted using `DELETE`. 46 | 47 | To make sure that your application instance gets notified about 48 | changes to the configuration it can register itself with as a 49 | web-hook. 50 | 51 | First find the subscriptions resource: 52 | 53 | $ curl --verbose -X HEAD http://localhost:6553/app/dm 54 | ... 55 | < HTTP/1.1 200 OK 56 | < Date: Mon, 29 Aug 2011 10:00:50 GMT 57 | < Link: http://localhost:6553/app/dm/web-hooks, rel="Subscriptions" 58 | 59 | POST your endpoint information to the Subscriptions resource: 60 | 61 | $ curl -X POST -d '{"name":"dm-host1", "endpoint":"http://localhost:4322/web-hooks"}' http://localhost:6553/app/dm/web-hooks 62 | { 63 | "endpoint": "http://localhost:4322/web-hooks", 64 | "name": "dm-host1" 65 | } 66 | 67 | When the configuration for `dm` changes, a `HTTP POST` will be made 68 | with a `JSON` object to the specified endpoint. Endpoints have to be 69 | explicitly removed using `DELETE`. Make a note of the `Location` 70 | header when registering the web-hook. 71 | 72 | # Running Nesoi # 73 | 74 | Requirements: 75 | 76 | - Twisted (core and web) 77 | - txgossip 78 | 79 | _Nesoi_ is installed as a twistd plugin, so you'll have to start it 80 | using the `twistd` command-line tool. See `twistd --help` for more 81 | information on generic options and such. 82 | 83 | The `nesoi` service accepts the following options: 84 | 85 | - `--listen-address IP` listen address (*required*) 86 | - `--listen-port PORT` listen port (*required*) 87 | - `--data-file FILE` where to store config data (*required*) 88 | - `--seed IP:PORT` another nesoi instance to comminicate with 89 | 90 | Example: 91 | 92 | twistd nesoi --listen-address 10.2.2.2 --listen-port 6553 --seed 10.2.2.1:6553 93 | 94 | # Implementation # 95 | 96 | _Nesoi_ is in its foundation a distributed key-value store. Each 97 | resource pretty much maps to a key-value pair (except for the 98 | collection resources that maps to many key-value pairs of course). 99 | 100 | _Nesoi_ is in itself distributed. All _Nesoi_ instances communicate 101 | using a gossip protocol (using `txgossip`). Instances gossip with 102 | each other about state changes to their local key-value stores. 103 | Eventually all data has propagated to all nodes in the system. 104 | 105 | Each value in the key-value store is annotated with a timestamp. This 106 | timestamp is used to resolve conflicts. A newer value always wins. 107 | As an effect of this, _Nesoi_ assumes that all nodes running _Nesoi_ 108 | instances have synchronized clocks. 109 | 110 | Each _Nesoi_ cluster has a leader. This leader is responsible for 111 | sending out the watcher notifications. 112 | 113 | # API # 114 | 115 | The API is quite simple. 116 | 117 | ## Application Configurations 118 | 119 | Application configurations live under `/app`. You can retreive a 120 | configuration using `GET /app/`. To update a create or 121 | update a configuration use `PUT /app/`. 122 | 123 | The data pushed to a `/app/` must be a JSON object holding a 124 | `config` property. 125 | 126 | To get a list of all application configurations do a `GET /app`. 127 | 128 | ## Services and Hosts 129 | 130 | Instances of _applications_ register themselves as a service running 131 | on a host. They do this but issuing a `PUT` to `/srv//`. 132 | 133 | The data pushed to a `/srv//` must be a JSON object 134 | holding a `endpoints` property. 135 | 136 | Services can update their state by issuing further `PUT`s. Also, when 137 | an instance shuts down it **SHOULD** delete it itself from the 138 | registry using a `DELETE` on `/srv//`. 139 | 140 | ## Webhooks (change notifications) ## 141 | 142 | _Nesoi_ implements webhooks [1] to allow clients to monitor changes to 143 | a resource. See [2] for more information. The `Notification-Type` is 144 | currently ignored. Hooks will be informed about all changes. 145 | 146 | When registering a webhook, `POST` a `json` object with the following 147 | attributes to the subscription resource: 148 | 149 | * A client name (`name`). Used to identify the endpoint from a 150 | service point of view. Normally constructed from hostname and 151 | service name. 152 | * An endpoint (`endpoint`). URI where notification should be posted. 153 | 154 | When something happens in _Nesoi_ that triggers a notification, a 155 | `HTTP` `POST` will be sent to the registered endpoint. 156 | 157 | The payload of the body is a `json` object with the following 158 | attributes: 159 | 160 | * Name of webhook (`name`) 161 | * URI to the resource that was changed (`uri`) 162 | 163 | Web-hooks can be attached to application configurations 164 | (`/app//web-hooks`) and service (`/srv//web-hooks`). 165 | 166 | An example: 167 | 168 | { 169 | "name": "node1-test", 170 | "uri": "/app/test" 171 | } 172 | 173 | [1] http://wiki.webhooks.org 174 | [2] http://wiki.webhooks.org/w/page/13385128/RESTful%20WebHooks 175 | 176 | # Use Cases # 177 | 178 | ## Service Configuration ## 179 | 180 | To ease configuration management, Nesoi can help with distributing 181 | configuration data to all instances of a service. 182 | 183 | The configuration manager (a person, or a piece of software) creates a 184 | `/app/NAME` resource using the REST interface. When the configuration 185 | is changed, he or she simply updates the resource with the new config. 186 | 187 | Service instances are configured with information about where to reach 188 | Nesoi. When starting up, the instance fetches the `/app/NAME` 189 | resource. It also installes a _watch_ on `/app/NAME` allowing the 190 | service to be informed when the configuration is changed. 191 | 192 | ## Service Announcement ## 193 | 194 | When a service instance has found its configuratio and initialized 195 | itself it should announce its presence to the rest of the distributed 196 | system. It does this by create (or updating) a service information 197 | resource at '/srv/APP/HOST'. The resource holds information about 198 | where the service endpoints (communication endspoints) are. Other 199 | information may also be included. 200 | 201 | The service instance resource contains metadata about when the data 202 | was last updated. This information can be used to communicate some 203 | kind of _status_ about the service instance. For example, a protocol 204 | can be put in place that the service instance should update its 205 | resource once every minute. If the resource has not been updated for 206 | two minutes, it should be considered dead. 207 | 208 | If a leader among all service instances has to communicated to users 209 | of the service, this can be done by setting a `master` field to `true` 210 | in the resource. When a instance looses an election, it should right 211 | away update its resource with `master` set to `false`. If a client 212 | finds several service instances with that states that they are master, 213 | the one with the latest "updated_at" time should be trusted. 214 | 215 | When a service instance is removed from the system, the service 216 | instance resource should be removed with it. 217 | 218 | ## Service Discovery ## 219 | 220 | When a client wants to talk to a service it fetches a representation 221 | of the `/srv/NAME` resource. The resource will enumerate all known 222 | service instances and their communication endpoints. 223 | 224 | A client may also register a _watch_ on `/srv/NAME` so that it will 225 | get notified about changes to service instances. 226 | 227 | -------------------------------------------------------------------------------- /nesoi/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jrydberg/nesoi/e1d83de750719c0b3abd96f02efc2fbd8e64d68f/nesoi/__init__.py -------------------------------------------------------------------------------- /nesoi/api.py: -------------------------------------------------------------------------------- 1 | # Copyright 2011 Johan Rydberg 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | from twisted.web import http 16 | 17 | from nesoi import rest 18 | 19 | 20 | class WebhookResourceMixin: 21 | """Mixin for resource controllers that want to provide webhooks 22 | subscriptions on their resource. 23 | 24 | This mixin only implements the discovery part of the webhooks. A 25 | webhook resource also has to be present in the routing table. 26 | """ 27 | 28 | def head(self, router, request, url, **args): 29 | """Response with a link to the subscriptions resource""" 30 | request.setHeader('Link', '%s, rel="Subscriptions"' % ( 31 | str(url.child('web-hooks')))) 32 | return http.OK 33 | 34 | 35 | class WebhookResource(object): 36 | """Resource for web-hooks.""" 37 | 38 | def __init__(self, model, kwname, method_name): 39 | self.model = model 40 | self.kwname = kwname 41 | self.method_name = method_name 42 | 43 | def put(self, router, request, url, config, hookname=None, **kwargs): 44 | """Update an existing web-hook watcher.""" 45 | try: 46 | getattr(self.model, 'watch_%s' % (self.method_name))( 47 | kwargs[self.kwname], config, hookname) 48 | except ValueError: 49 | raise rest.ControllerError(400) 50 | else: 51 | return http.CREATED 52 | 53 | def delete(self, router, request, url, hookname=None, **kwargs): 54 | """Delete a web-hook watcher.""" 55 | try: 56 | getattr(self.model, 'unwatch_%s' % (self.method_name))( 57 | hookname, kwargs[self.kwname]) 58 | except ValueError, ve: 59 | return http.BAD_REQUEST, str(ve) 60 | else: 61 | return http.NO_CONTENT 62 | 63 | 64 | class WebhookCollectionResource(object): 65 | """Resource for working with web-hooks.""" 66 | 67 | def __init__(self, model, kwname, method_name): 68 | self.model = model 69 | self.kwname = kwname 70 | self.method_name = method_name 71 | 72 | def post(self, router, request, url, config, **kwargs): 73 | """Create a web-hook.""" 74 | try: 75 | getattr(self.model, 'watch_%s' % (self.method_name))( 76 | kwargs[self.kwname], config) 77 | except ValueError, ve: 78 | return http.BAD_REQUEST, str(ve) 79 | else: 80 | return http.CREATED 81 | 82 | def get(self, router, request, url, **kwargs): 83 | """List all registered web-hooks.""" 84 | watchers = {} 85 | for watcher in getattr(self.model, '%s_watchers' % ( 86 | self.method_name))(kwargs[self.kwname]): 87 | watchers[watcher['name']] = watcher 88 | return watchers 89 | 90 | 91 | class ApplicationResource(WebhookResourceMixin): 92 | """Application config resource.""" 93 | 94 | def __init__(self, model): 95 | self.model = model 96 | 97 | def put(self, router, request, url, config, appname=None): 98 | """Update or create application config.""" 99 | try: 100 | self.model.set_app(appname, config) 101 | except ValueError, ve: 102 | return http.BAD_REQUEST, str(ve) 103 | else: 104 | return http.NO_CONTENT 105 | 106 | def get(self, router, request, url, appname=None): 107 | """Read out application configuration.""" 108 | try: 109 | return self.model.app(appname) 110 | except ValueError: 111 | raise rest.NoSuchResourceError() 112 | 113 | 114 | class ApplicationCollectionResource(object): 115 | """Resource for listing all applications.""" 116 | 117 | def __init__(self, model): 118 | self.model = model 119 | 120 | def get(self, router, request, url): 121 | """Read out applications.""" 122 | return {'apps': list(self.model.apps())} 123 | 124 | 125 | class ServiceHostResource(object): 126 | """Configuration resource for a service host pair.""" 127 | 128 | def __init__(self, model): 129 | self.model = model 130 | 131 | def get(self, router, request, url, srvname=None, hostname=None): 132 | """Return host configuration.""" 133 | try: 134 | return self.model.host(srvname, hostname) 135 | except ValueError: 136 | raise rest.NoSuchResourceError() 137 | 138 | def delete(self, router, request, url, srvname=None, hostname=None): 139 | """Delete a host configuration.""" 140 | try: 141 | self.model.del_host(srvname, hostname) 142 | except ValueError: 143 | raise rest.NoSuchResourceError() 144 | else: 145 | return http.NO_CONTENT 146 | 147 | def put(self, router, request, url, config, srvname=None, 148 | hostname=None): 149 | """Update or create a host configuration.""" 150 | try: 151 | self.model.set_host(srvname, hostname, config) 152 | except ValueError, ve: 153 | return http.BAD_REQUEST, str(ve) 154 | else: 155 | return http.NO_CONTENT 156 | 157 | 158 | class ServiceHostCollectionResource(WebhookResourceMixin): 159 | """Collection that will list all hosts for a particular service. 160 | 161 | Will also include the whole config for the host. 162 | """ 163 | 164 | def __init__(self, model): 165 | self.model = model 166 | 167 | def get(self, router, request, url, srvname=None): 168 | """Return a mapping of all known hosts.""" 169 | hosts = {} 170 | for hostname in self.model.hosts(srvname): 171 | hosts[hostname] = self.model.host(srvname, hostname) 172 | return hosts 173 | 174 | 175 | class ServiceCollectionResource(object): 176 | """Collection that will list all services and their hosts.""" 177 | 178 | def __init__(self, model): 179 | self.model = model 180 | 181 | def get(self, router, request, url): 182 | """Return a mapping of all known services.""" 183 | services = {} 184 | for srvname in self.model.services(): 185 | services[srvname] = { 186 | 'hosts': list(self.model.hosts(srvname))} 187 | return services 188 | -------------------------------------------------------------------------------- /nesoi/keystore.py: -------------------------------------------------------------------------------- 1 | # Copyright 2011 Johan Rydberg 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | import json 16 | 17 | from twisted.application import service 18 | from twisted.web import client 19 | from twisted.python import log 20 | from txgossip.recipies import KeyStoreMixin, LeaderElectionMixin 21 | 22 | 23 | class _LeaderElectionProtocol(LeaderElectionMixin): 24 | """Private version of the leader election protocol that informs 25 | the application logic about election results. 26 | """ 27 | 28 | def __init__(self, clock, app): 29 | LeaderElectionMixin.__init__(self, clock, vote_delay=2) 30 | self._app = app 31 | 32 | def leader_elected(self, is_leader, leader): 33 | LeaderElectionMixin.leader_elected(self, is_leader, leader) 34 | self._app.leader_elected(is_leader) 35 | 36 | 37 | class ClusterNode(service.Service, KeyStoreMixin, LeaderElectionMixin): 38 | """Gossip participant that both implements our replicated 39 | key-value store and a leader-election mechanism. 40 | """ 41 | 42 | def __init__(self, clock, storage, client=client): 43 | self.election = _LeaderElectionProtocol(clock, self) 44 | self.keystore = KeyStoreMixin(clock, storage, 45 | [self.election.LEADER_KEY, self.election.VOTE_KEY, 46 | self.election.PRIO_KEY]) 47 | self.client = client 48 | self.storage = storage 49 | 50 | def startService(self): 51 | self.keystore.load_from(self.storage) 52 | service.Service.startService(self) 53 | 54 | def value_changed(self, peer, key, value): 55 | """A peer changed one of its values.""" 56 | if key == '__heartbeat__': 57 | return 58 | 59 | if self.election.value_changed(peer, key, value): 60 | # This value change was handled by the leader election 61 | # protocol. 62 | return 63 | self.keystore.value_changed(peer, key, value) 64 | 65 | if self.election.is_leader and peer.name == self.gossiper.name: 66 | # This peer is the leader of the cluster, which means that 67 | # we're responsible for firing notifications. 68 | if not key.startswith('watcher:'): 69 | self._check_notify(key) 70 | 71 | def make_connection(self, gossiper): 72 | """Make connection to gossip instance.""" 73 | self.gossiper = gossiper 74 | self.election.make_connection(gossiper) 75 | self.keystore.make_connection(gossiper) 76 | self.gossiper.set(self.election.PRIO_KEY, 0) 77 | 78 | def peer_alive(self, peer): 79 | """The gossiper reports that C{peer} is alive.""" 80 | self.election.peer_alive(peer) 81 | 82 | def peer_dead(self, peer): 83 | """The gossiper reports that C{peer} is dead.""" 84 | self.election.peer_alive(peer) 85 | 86 | def leader_elected(self, is_leader): 87 | """Leader elected.""" 88 | print "is leader?", is_leader 89 | if is_leader: 90 | # Go through and possible trigger all notifications. 91 | for key in self.keystore.keys('app:*'): 92 | self._check_notify(key) 93 | for key in self.keystore.keys('srv:*'): 94 | self._check_notify(key) 95 | 96 | def _notify(self, wkey, watcher): 97 | """Notification watcher about change.""" 98 | def done(result): 99 | watcher['last-hit'] = self.clock.seconds() 100 | # Verify that the watcher has not been deleted. 101 | if wkey in self and self.keystore[wkey] is not None: 102 | self.keystore.set(wkey, watcher) 103 | d = self.client.getPage(str(watcher['endpoint']), method='POST', 104 | postdata=json.dumps({'name': watcher['name'], 105 | 'uri': watcher['uri']}), 106 | timeout=3) 107 | return d.addCallback(done).addErrback(log.err) 108 | 109 | def _check_notify(self, key): 110 | """Possible notify listener that something has changed.""" 111 | for wkey in self.keystore.keys('watcher:*'): 112 | watcher = self.keystore.get(wkey) 113 | if watcher is None: 114 | continue 115 | timestamp = self.keystore.timestamp_for_key(key) 116 | if (key.startswith(watcher['pattern']) 117 | and watcher['last-hit'] < timestamp): 118 | self._notify(wkey, watcher) 119 | -------------------------------------------------------------------------------- /nesoi/model.py: -------------------------------------------------------------------------------- 1 | # Copyright 2011 Johan Rydberg 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | class ResourceModel(object): 16 | """Data model for the resources.""" 17 | 18 | def __init__(self, clock, keystore): 19 | self.clock = clock 20 | self.keystore = keystore 21 | 22 | def apps(self): 23 | """Return a list of all applicaitons in the model.""" 24 | return [key.split(':', 1)[1] 25 | for key in self.keystore.keys('app:*') 26 | if self.keystore.get(key) is not None] 27 | 28 | def app(self, appname): 29 | """Return configuration for app C{appname}.""" 30 | key = 'app:%s' % (appname,) 31 | if not key in self.keystore or self.keystore.get(key) is None: 32 | raise ValueError('no such app: %s' % (appname,)) 33 | return self.keystore.get(key) 34 | 35 | def set_app(self, appname, config): 36 | """Update an application.""" 37 | key = 'app:%s' % (appname,) 38 | for required in ('config',): 39 | if not required in config: 40 | raise ValueError('missing field "%s" in config' % ( 41 | required,)) 42 | config['updated_at'] = self.clock.seconds() 43 | self.keystore.set(key, config) 44 | 45 | def del_app(self, appname): 46 | """Delete application.""" 47 | key = 'app:%s' % (appname,) 48 | if self.keystore.get(key) is None: 49 | raise ValueError('no such app: %s' % (appname,)) 50 | self.keystore.set(key, None) 51 | 52 | def hosts(self, srvname): 53 | """Return names of all available hosts for service C{srvname}.""" 54 | keypattern = 'srv:%s:*' % (srvname,) 55 | for key in self.keystore.keys(keypattern): 56 | if self.keystore.get(key) is not None: 57 | yield key.split(':', 2)[2] 58 | 59 | def host(self, srvname, hostname): 60 | """Return config for a service and hostname pair.""" 61 | key = 'srv:%s:%s' % (srvname, hostname) 62 | if self.keystore.get(key) is None: 63 | raise ValueError('no such host: %s/%s' % (srvname, hostname)) 64 | return self.keystore.get(key) 65 | 66 | def set_host(self, srvname, hostname, config): 67 | """Set config for a service and hostname pair.""" 68 | key = 'srv:%s:%s' % (srvname, hostname) 69 | for required in ('endpoints',): 70 | if not required in config: 71 | raise ValueError('missing field "%s" in config' % ( 72 | required,)) 73 | config['updated_at'] = self.clock.seconds() 74 | self.keystore.set(key, config) 75 | 76 | def del_host(self, srvname, hostname): 77 | """Delete a service and hostname pair.""" 78 | key = 'srv:%s:%s' % (srvname, hostname) 79 | if self.keystore.get(key) is None: 80 | raise ValueError('no such host: %s/%s' % (srvname, hostname)) 81 | self.keystore.set(key, None) 82 | 83 | def services(self): 84 | """Return an iterable that will yield the name of all 85 | available services. 86 | """ 87 | services = set() 88 | for key in self.keystore.keys('srv:*:*'): 89 | if self.keystore.get(key) is None: 90 | continue 91 | srvname, hostname = key.split(':', 2)[1:] 92 | services.add(srvname) 93 | return services 94 | 95 | def _validate_watcher(self, config, hookname=None): 96 | for required in ('name', 'endpoint',): 97 | if not required in config: 98 | raise ValueError('required field "%s" is missing' % ( 99 | required,)) 100 | if hookname is not None: 101 | if config['name'] != hookname: 102 | raise ValueError('name do not match') 103 | 104 | def _watch(self, keypattern, uri, config, hookname): 105 | self._validate_watcher(config, hookname) 106 | watcher = { 107 | 'name': config['name'], 108 | 'endpoint': config['endpoint'], 109 | 'uri': uri, 110 | 'pattern': keypattern, 111 | 'last-hit': self.clock.seconds() 112 | } 113 | wkey = str('watcher:%s:%s' % (keypattern, watcher['name'])) 114 | if hookname is None and self.keystore.get(wkey) is not None: 115 | raise ValueError("already exists") 116 | self.keystore.set(wkey, watcher) 117 | 118 | def _unwatch(self, keypattern, hookname): 119 | wkey = str('watcher:%s:%s' % (keypattern, hookname)) 120 | if self.keystore.get(wkey) is None: 121 | raise Value("no such watcher") 122 | self.keystore.set(wkey, None) 123 | 124 | def watch_service(self, srvname, config, hookname=None): 125 | """Watch service C{srvname}.""" 126 | self._watch('srv:%s' % (srvname), '/srv/%s' % (srvname), config, 127 | hookname=hookname) 128 | 129 | def unwatch_service(self, hookname, srvname): 130 | """Stop watching service C{srvname}.""" 131 | self._unwatch('srv:%s' % (srvname), hookname) 132 | 133 | def watch_app(self, appname, config, hookname=None): 134 | """Watch app config C{appname}.""" 135 | self._watch('app:%s' % (appname), '/app/%s' % (appname), config, 136 | hookname=hookname) 137 | 138 | def unwatch_app(self, hookname, appname): 139 | """Stop watching app config C{appname}.""" 140 | self._unwatch('app:%s' % (appname), hookname) 141 | 142 | def service_watcher(self, srvname, hookname): 143 | """Return service watcher called C{hookname}.""" 144 | wkey = str('watcher:srv:%s:%s' % (srvname, hookname)) 145 | if self.keystore.get(wkey) is None: 146 | raise ValueError("no such hook") 147 | return self.keystore.get(wkey) 148 | 149 | def service_watchers(self, srvname): 150 | """Return all watcher for service C{srvname}.""" 151 | for key in self.keystore.keys('watcher:srv:%s:*' % ( 152 | srvname,)): 153 | if self.keystore.get(key) is None: 154 | continue 155 | yield self.keystore.get(key) 156 | 157 | def app_watcher(self, srvname, hookname): 158 | """Return app watcher called C{hookname}.""" 159 | wkey = str('watcher:app:%s:%s' % (appname, hookname)) 160 | if self.keystore.get(wkey) is None: 161 | raise ValueError("no such hook") 162 | return self.keystore.get(wkey) 163 | 164 | def app_watchers(self, appname): 165 | """Return all watcher for app config C{appname}.""" 166 | for key in self.keystore.keys('watcher:app:%s:*' % ( 167 | appname,)): 168 | if self.keystore.get(key) is None: 169 | continue 170 | yield self.keystore.get(key) 171 | -------------------------------------------------------------------------------- /nesoi/rest.py: -------------------------------------------------------------------------------- 1 | from twisted.web.resource import Resource 2 | from twisted.web import server, http, client, error 3 | from twisted.internet import defer 4 | from twisted.python import log 5 | from zope.interface import Interface, implements 6 | import re 7 | try: 8 | import json 9 | except ImportError: 10 | import simplejson as json 11 | 12 | 13 | class ControllerError(Exception): 14 | 15 | def __init__(self, responseCode): 16 | self.responseCode = responseCode 17 | self.message = None 18 | 19 | 20 | class UnsupportedRepresentationError(ControllerError): 21 | """ 22 | The given representation was not supported by the controller. 23 | """ 24 | 25 | def __init__(self): 26 | ControllerError.__init__(self, http.UNSUPPORTED_MEDIA_TYPE) 27 | 28 | 29 | class NoSuchResourceError(ControllerError): 30 | 31 | def __init__(self): 32 | ControllerError.__init__(self, http.NOT_FOUND) 33 | 34 | 35 | def read_json(request): 36 | return json.loads(request.content.read()) 37 | 38 | 39 | def write_json(request, data, ct='application/json', rc=200): 40 | """ 41 | Write JSON reponse to request. 42 | """ 43 | sdata = json.dumps(data, indent=2).encode('utf-8') 44 | request.setHeader('content-type', ct) 45 | request.setHeader('content-length', len(sdata)) 46 | request.setResponseCode(rc) 47 | request.write(sdata) 48 | 49 | 50 | def compile_regexp(url_def): 51 | """ 52 | Compile url defintion to a regular expression. 53 | """ 54 | elements = url_def.split('/') 55 | 56 | l = list() 57 | for element in elements: 58 | try: 59 | front, rest = element.split('{', 1) 60 | middle, end = rest.split('}', 1) 61 | 62 | expr = '(?P<%s>[0-9a-zA-Z\.\-_]+)' % middle.replace('-', '_') 63 | l.append(''.join([front, expr, end])) 64 | except ValueError: 65 | l.append(element) 66 | 67 | return '/'.join(l) + '$' 68 | 69 | 70 | class Router(Resource): 71 | isLeaf = True 72 | 73 | def __init__(self): 74 | self.controllers = list() 75 | 76 | def addController(self, controllerPath, controller): 77 | """ 78 | Add router. 79 | """ 80 | regexp = re.compile(compile_regexp(controllerPath)) 81 | self.controllers.append((regexp, controller)) 82 | 83 | def getController(self, request): 84 | """ 85 | Return an initialized controller based on the given request. 86 | """ 87 | controllerUrl = request.URLPath() 88 | postpath = list(request.postpath) 89 | if postpath: 90 | if not postpath[-1]: 91 | del postpath[-1] 92 | p = '/'.join(postpath) 93 | for regexp, controller in self.controllers: 94 | m = regexp.match(p) 95 | if m is not None: 96 | return controller, controllerUrl.click(p), m.groupdict() 97 | print "no matching controller", p 98 | 99 | def ebControl(self, reason, request): 100 | reason.printTraceback() 101 | reason.trap(ControllerError) 102 | request.setResponseCode(reason.value.responseCode) 103 | if reason.value.message is not None: 104 | request.setHeader('content-length', str(len(reason.value.message))) 105 | request.write(reason.value.message) 106 | else: 107 | request.setHeader('content-length', '0') 108 | request.finish() 109 | 110 | def ebInternal(self, reason, request): 111 | request.setResponseCode(http.INTERNAL_SERVER_ERROR) 112 | request.setHeader('content-length', '0') 113 | request.finish() 114 | return reason 115 | 116 | def cbControl(self, result, request): 117 | """ 118 | Callback from controller method. 119 | 120 | C{repr} is a provider of L{IRepresentation} that should be 121 | rendered to the client. 122 | """ 123 | rc = 200 124 | if type(result) == tuple: 125 | rc, result = result 126 | elif type(result) == int: 127 | rc = result 128 | 129 | if type(result) == dict: 130 | write_json(request, result, rc=rc) 131 | elif type(result) == str: 132 | request.setResponseCode(rc) 133 | request.write(result) 134 | else: 135 | request.setHeader('content-length', 0) 136 | 137 | request.finish() 138 | 139 | def render(self, request): 140 | """ 141 | Render request. 142 | """ 143 | controller, url, params = self.getController(request) 144 | if controller is None: 145 | request.setResponseCode(http.INTERNAL_SERVER_ERROR) 146 | return 'No controller found for URL' 147 | 148 | method = getattr(controller, request.method.lower(), None) 149 | if method is None: 150 | request.setResponseCode(http.NOT_ALLOWED) 151 | return '' 152 | input = [] 153 | if request.method.lower() in ('post', 'put'): 154 | input.append(read_json(request)) 155 | 156 | doneDeferred = defer.maybeDeferred(method, self, request, url, 157 | *input, **params) 158 | doneDeferred.addCallback(self.cbControl, request) 159 | doneDeferred.addErrback(self.ebControl, request) 160 | doneDeferred.addErrback(self.ebInternal, request) 161 | doneDeferred.addErrback(log.deferr) 162 | return server.NOT_DONE_YET 163 | -------------------------------------------------------------------------------- /nesoi/service.py: -------------------------------------------------------------------------------- 1 | # Copyright 2011 Johan Rydberg 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | import shelve 16 | 17 | from twisted.application.service import MultiService 18 | from twisted.application.internet import TCPServer, UDPServer 19 | from twisted.web.server import Site 20 | from txgossip.gossip import Gossiper 21 | 22 | from nesoi.model import ResourceModel 23 | from nesoi.keystore import ClusterNode 24 | from nesoi import api, rest 25 | 26 | 27 | def create_service(reactor, options): 28 | """Based on options provided by the user create a service that 29 | will provide whatever it is that Nesoi do. 30 | """ 31 | service = MultiService() 32 | 33 | listen_address = options['listen-address'] 34 | 35 | storage = shelve.open(options['data-file'], writeback=True) 36 | cluster_node = ClusterNode(reactor, storage) 37 | service.addService(cluster_node) 38 | 39 | model = ResourceModel(reactor, cluster_node.keystore) 40 | 41 | gossiper = Gossiper(reactor, cluster_node, listen_address) 42 | if options['seed']: 43 | gossiper.seed([options['seed']]) 44 | 45 | service.addService(UDPServer(int(options['listen-port']), gossiper, 46 | interface=listen_address)) 47 | 48 | router = rest.Router() 49 | router.addController('app', api.ApplicationCollectionResource(model)) 50 | router.addController('app/{appname}/web-hooks', api.WebhookCollectionResource(model, 'appname', 'app')) 51 | router.addController('app/{appname}/web-hooks/{hookname}', api.WebhookResource(model, 'appname', 'app')) 52 | router.addController('app/{appname}', api.ApplicationResource(model)) 53 | router.addController('srv', api.ServiceCollectionResource(model)) 54 | router.addController('srv/{srvname}', api.ServiceHostCollectionResource(model)) 55 | router.addController('srv/{srvname}/web-hooks', api.WebhookCollectionResource(model, 'srvname', 'service')) 56 | router.addController('srv/{srvname}/web-hooks/{hookname}', api.WebhookResource(model, 'srvname', 'service')) 57 | router.addController('srv/{srvname}/{hostname}', api.ServiceHostResource(model)) 58 | 59 | service.addService(TCPServer(int(options['listen-port']), Site(router), 60 | interface=listen_address)) 61 | 62 | #gossiper.set(cluster_node.election.PRIO_KEY, 0) 63 | #cluster_node.keystore.load_from(storage) 64 | 65 | return service 66 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | from setuptools import setup, find_packages 2 | setup(name='nesoi', 3 | version='0.0.2', 4 | description='a coordination and configuration manager', 5 | author='Johan Rydberg', 6 | author_email='johan.rydberg@gmail.com', 7 | url='http://github.com/jrydberg/nesoi', 8 | packages=find_packages() + ['twisted.plugins'] 9 | ) 10 | -------------------------------------------------------------------------------- /twisted/plugins/nesoi_plugin.py: -------------------------------------------------------------------------------- 1 | # Copyright 2011 Johan Rydberg 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | from zope.interface import implements 16 | 17 | from twisted.python import usage 18 | from twisted.plugin import IPlugin 19 | from twisted.application.service import IServiceMaker 20 | from twisted.internet import reactor 21 | 22 | from nesoi import service 23 | 24 | 25 | class Options(usage.Options): 26 | 27 | optParameters = ( 28 | ("listen-port", "p", 6553, "The port number to listen on."), 29 | ("listen-address", "a", None, "The listen address."), 30 | ("data-file", "d", "nesoi.data", "File to store data in."), 31 | ("seed", "s", None, "Address to running Nesoi instance.") 32 | ) 33 | 34 | 35 | class MyServiceMaker(object): 36 | implements(IServiceMaker, IPlugin) 37 | 38 | tapname = "nesoi" 39 | description = "coordination and configuration manager" 40 | options = Options 41 | 42 | def makeService(self, options): 43 | """.""" 44 | if not options['listen-address']: 45 | raise usage.UsageError("listen address must be specified") 46 | return service.create_service(reactor, options) 47 | 48 | serviceMaker = MyServiceMaker() 49 | --------------------------------------------------------------------------------