├── .gitignore ├── LICENSE ├── MANIFEST.in ├── Makefile ├── README.md ├── setup.py └── skypipe ├── __init__.py ├── cli.py ├── client.py ├── cloud.py └── satellite ├── builder ├── dotcloud.yml ├── requirements.txt └── server.py /.gitignore: -------------------------------------------------------------------------------- 1 | *.egg-info 2 | build 3 | TODO 4 | *.pyc 5 | **/*.pyc 6 | dist 7 | .DS_Store 8 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2012 Jeff Lindsay 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining a 4 | copy of this software and associated documentation files (the 5 | “Software”), to deal in the Software without restriction, including 6 | without limitation the rights to use, copy, modify, merge, publish, 7 | distribute, sublicense, and/or sell copies of the Software, and to 8 | permit persons to whom the Software is furnished to do so, subject to 9 | the following conditions: 10 | 11 | The above copyright notice and this permission notice shall be included 12 | in all copies or substantial portions of the Software. 13 | 14 | THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS 15 | OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 16 | MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. 17 | IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY 18 | CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, 19 | TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE 20 | SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 21 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | include README.md 2 | include skypipe/satellite/* 3 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | release: 2 | python setup.py sdist register upload 3 | python2.6 setup.py bdist_egg register upload 4 | python2.7 setup.py bdist_egg register upload 5 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | Skypipe (alpha) 2 | =============== 3 | 4 | Skypipe is a magical command line tool that lets you easily pipe data across terminal sessions, regardless of whether the sessions are on the same machine, across thousands of machines, or even behind a firewall. It gives you named pipes in the sky and lets you magically pipe data *anywhere*. 5 | 6 | Installing 7 | ---------- 8 | 9 | Currently you need Python 2.6, ZeroMQ 2.0+ and the ability to compile extensions. 10 | Then install with pip: 11 | 12 | $ pip install skypipe 13 | 14 | Setting up 15 | ---------- 16 | 17 | The magic of skypipe requires a free dotcloud account. If you don't have 18 | one, you can easily create one for free. The first time you use skypipe, 19 | you will be asked for credentials. 20 | 21 | Using Skypipe 22 | ------------- 23 | 24 | Skypipe combines named pipes and netcat and gives you even more power in a simpler tool. Here is a simple example using skypipe like you would a named pipe in order to gzip a file across shells: 25 | 26 | $ skypipe | gzip -9 -c > out.gz 27 | 28 | Your skypipe is ready to receive some data from another shell process: 29 | 30 | $ cat file | skypipe 31 | 32 | Unliked named pipes, however, *this will work across any machines connected to the Internet*. And you didn't have to specify a host address or set up "listen mode" like you would with netcat. In fact, unlike netcat, which is point to point, you could use skypipe for log aggregation. Here we'll used named skypipes. Run this on several hosts: 33 | 34 | $ tail -f /var/log/somefile | skypipe logs 35 | 36 | And then run this on a single machine: 37 | 38 | $ skypipe logs > /var/log/aggregate 39 | 40 | Or alternatively you can broadcast to multiple hosts. With the above, you can "listen in" by running this from your laptop, even while behind a NAT: 41 | 42 | $ skypipe logs 43 | 44 | Also unlike netcat, you can temporarily store data "in the pipe" until you pull it out. In fact, you can keep several "files" in the pipe. On one machine load some files into it a named skypipe: 45 | 46 | $ cat file_a | skypipe files 47 | $ cat file_b | skypipe files 48 | 49 | Now from somewhere else pull them out, in order: 50 | 51 | $ skypipe files > new_file_a 52 | $ skypipe files > new_file_b 53 | 54 | Lastly, you can use skypipe like the channel primitive in Go for coordinating between shell scripts. As a simple example, here we use skypipe to wait for an event triggered by another script: 55 | 56 | #!/bin/bash 57 | echo "I'm going to wait until triggered" 58 | skypipe trigger 59 | echo "I was triggered!" 60 | 61 | Triggering is just sending an EOF over the pipe, causing the listening skypipe to terminate. We can do this with a simple idiom: 62 | 63 | $ echo | skypipe trigger 64 | 65 | Software with a service 66 | ----------------------- 67 | 68 | The trick to this private pipe in the sky is that when you first use skypipe, behind the scenes it will deploy a very simple messaging server to dotcloud. Skypipe will use your account to transparently find and use this server, no matter where you are. 69 | 70 | This all works without you ever thinking about it because this server is managed automatically and runs on dotcloud for free. It might as well not exist! 71 | 72 | This represents a new paradigm of creating tools that transparently leverage the cloud to create magical experiences. It's not quite software as a service, it's software *with* a service. Nobody is using a shared, central service. The software deploys its own service on your behalf for you to use. 73 | 74 | Thanks to platforms like dotcloud (and Heroku), we can now build software leveraging features of software as a service that is *packaged and distributed like normal open source software*. 75 | 76 | Contributing 77 | ------------ 78 | 79 | There aren't any tests yet, but it's pretty well documented and the code 80 | is written to be read. Fork and send pull requests. Check out the issues 81 | to see how you can be most helpful. 82 | 83 | Contributors 84 | ------------ 85 | 86 | * Jeff Lindsay 87 | 88 | License 89 | ------- 90 | 91 | MIT 92 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | import os 3 | from setuptools import setup, find_packages 4 | 5 | import skypipe 6 | 7 | setup( 8 | name='skypipe', 9 | version=skypipe.VERSION, 10 | author='Jeff Lindsay', 11 | author_email='progrium@gmail.com', 12 | description='Magic pipe in the sky', 13 | long_description=open(os.path.join(os.path.dirname(__file__), 14 | "README.md")).read().replace(':', '::'), 15 | license='MIT', 16 | classifiers=[ 17 | "Development Status :: 3 - Alpha", 18 | "Topic :: Utilities", 19 | "License :: OSI Approved :: MIT License", 20 | ], 21 | url="http://github.com/progrium/skypipe", 22 | packages=find_packages(), 23 | install_requires=['pyzmq', 'dotcloud>=0.7', 'argparse'], 24 | zip_safe=False, 25 | package_data={ 26 | 'skypipe': ['satellite/*']}, 27 | entry_points={ 28 | 'console_scripts': [ 29 | 'skypipe = skypipe.cli:run',]} 30 | ) 31 | -------------------------------------------------------------------------------- /skypipe/__init__.py: -------------------------------------------------------------------------------- 1 | VERSION = '0.1.4' 2 | -------------------------------------------------------------------------------- /skypipe/cli.py: -------------------------------------------------------------------------------- 1 | """Command line interface / frontend for skypipe 2 | 3 | This module contains an argparse configuration and handles endpoint 4 | loading for now. Ultimately it just runs the client. 5 | """ 6 | import sys 7 | import os 8 | import argparse 9 | import atexit 10 | import runpy 11 | 12 | from dotcloud.ui.cli import CLI as DotCloudCLI 13 | 14 | import skypipe 15 | from skypipe import client 16 | from skypipe import cloud 17 | 18 | def fix_zmq_exit(): 19 | """ 20 | Temporary fix until master of pyzmq is released 21 | See: https://github.com/zeromq/pyzmq/pull/265 22 | """ 23 | import zmq 24 | ctx = zmq.Context.instance() 25 | ctx.term() 26 | atexit.register(fix_zmq_exit) 27 | 28 | if sys.platform == 'win32': 29 | appdata = os.path.join(os.environ.get('APPDATA'), "skypipe") 30 | else: 31 | appdata = os.path.expanduser(os.path.join("~",".{0}".format("skypipe"))) 32 | appconfig = os.path.join(appdata, "config") 33 | 34 | def load_satellite_endpoint(): 35 | """loads any cached endpoint data""" 36 | pass 37 | 38 | def save_satellite_endpoint(endpoint): 39 | """caches endpoint data in config""" 40 | pass 41 | 42 | def get_parser(): 43 | parser = argparse.ArgumentParser(prog='skypipe', epilog=""" 44 | Use --setup to find or deploy a satellite in the sky. You can configure 45 | skypipe to use a custom satellite with the environment variable SATELLITE. 46 | Example: SATELLITE=tcp://12.0.0.1:1234 47 | """.strip()) 48 | parser.add_argument('name', metavar='NAME', type=str, nargs='?', 49 | help='use a named skypipe', default='') 50 | parser.add_argument('--version', action='version', 51 | version='%(prog)s {0}'.format(skypipe.VERSION)) 52 | parser.add_argument('--setup', action='store_const', 53 | const=True, default=False, 54 | help='setup account and satellite') 55 | parser.add_argument('--check', action='store_const', 56 | const=True, default=False, 57 | help='check if satellite is online') 58 | parser.add_argument('--reset', action='store_const', 59 | const=True, default=False, 60 | help='destroy any existing satellite') 61 | parser.add_argument('--satellite', action='store', 62 | default=None, metavar='PORT', 63 | help='manually run a satellite on PORT') 64 | return parser 65 | 66 | def run(): 67 | parser = get_parser() 68 | args = parser.parse_args() 69 | 70 | dotcloud_endpoint = os.environ.get('DOTCLOUD_API_ENDPOINT', 71 | 'https://rest.dotcloud.com/v1') 72 | cli = DotCloudCLI(endpoint=dotcloud_endpoint) 73 | 74 | if args.setup: 75 | cloud.setup(cli) 76 | elif args.reset: 77 | cloud.destroy_satellite(cli) 78 | cli.success("Skypipe system reset. Now run `skypipe --setup`") 79 | elif args.satellite: 80 | os.environ['PORT_ZMQ'] = args.satellite 81 | runpy.run_path('/'.join([os.path.dirname(__file__), 'satellite', 'server.py'])) 82 | else: 83 | skypipe_endpoint = os.environ.get("SATELLITE", load_satellite_endpoint()) 84 | skypipe_endpoint = skypipe_endpoint or cloud.discover_satellite(cli, deploy=False) 85 | if not skypipe_endpoint: 86 | cli.die("Unable to locate satellite. Please run `skypipe --setup`") 87 | save_satellite_endpoint(skypipe_endpoint) 88 | 89 | if args.check: 90 | cli.success("Skypipe is ready for action") 91 | else: 92 | client.run(skypipe_endpoint, args.name) 93 | -------------------------------------------------------------------------------- /skypipe/client.py: -------------------------------------------------------------------------------- 1 | """Skypipe client 2 | 3 | This contains the client implementation for skypipe, which operates in 4 | two modes: 5 | 6 | 1. Input mode: STDIN -> Skypipe satellite 7 | 2. Output mode: Skypipe satellite -> STDOUT 8 | 9 | The satellite is a server managed by the cloud module. They use ZeroMQ 10 | to message with each other. They use a simple protocol on top of ZeroMQ 11 | using multipart messages. The first part is the header, which identifies 12 | the name and version of the protocol being used. The second part is 13 | always a command. Depending on the command there may be more parts. 14 | There are only four commands as of 0.1: 15 | 16 | 1. HELLO: Used to ping the server. Server should HELLO back. 17 | 2. DATA : Send/recv one piece of data (usually a line) for pipe 18 | 3. LISTEN : Start listening for data on a pipe 19 | 4. UNLISTEN : Stop listening for data on a pipe 20 | 21 | The pipe parameter is the name of the pipe. It can by an empty string to 22 | represent the default pipe. 23 | 24 | EOF is an important concept for skypipe. We represent it with a DATA 25 | command using an empty string for the data. 26 | """ 27 | import os 28 | import sys 29 | import time 30 | 31 | import zmq 32 | 33 | ctx = zmq.Context.instance() 34 | 35 | SP_HEADER = "SKYPIPE/0.1" 36 | SP_CMD_HELLO = "HELLO" 37 | SP_CMD_DATA = "DATA" 38 | SP_CMD_LISTEN = "LISTEN" 39 | SP_CMD_UNLISTEN = "UNLISTEN" 40 | SP_DATA_EOF = "" 41 | 42 | def sp_msg(cmd, pipe=None, data=None): 43 | """Produces skypipe protocol multipart message""" 44 | msg = [SP_HEADER, cmd] 45 | if pipe is not None: 46 | msg.append(pipe) 47 | if data is not None: 48 | msg.append(data) 49 | return msg 50 | 51 | def check_skypipe_endpoint(endpoint, timeout=10): 52 | """Skypipe endpoint checker -- pings endpoint 53 | 54 | Returns True if endpoint replies with valid header, 55 | Returns False if endpoint replies with invalid header, 56 | Returns None if endpoint does not reply within timeout 57 | """ 58 | socket = ctx.socket(zmq.DEALER) 59 | socket.linger = 0 60 | socket.connect(endpoint) 61 | socket.send_multipart(sp_msg(SP_CMD_HELLO)) 62 | timeout_time = time.time() + timeout 63 | while time.time() < timeout_time: 64 | reply = None 65 | try: 66 | reply = socket.recv_multipart(zmq.NOBLOCK) 67 | break 68 | except zmq.ZMQError: 69 | time.sleep(0.1) 70 | socket.close() 71 | if reply: 72 | return str(reply.pop(0)) == SP_HEADER 73 | 74 | 75 | def stream_skypipe_output(endpoint, name=None): 76 | """Generator for reading skypipe data""" 77 | name = name or '' 78 | socket = ctx.socket(zmq.DEALER) 79 | socket.connect(endpoint) 80 | try: 81 | socket.send_multipart(sp_msg(SP_CMD_LISTEN, name)) 82 | 83 | while True: 84 | msg = socket.recv_multipart() 85 | try: 86 | data = parse_skypipe_data_stream(msg, name) 87 | if data: 88 | yield data 89 | except EOFError: 90 | raise StopIteration() 91 | 92 | finally: 93 | socket.send_multipart(sp_msg(SP_CMD_UNLISTEN, name)) 94 | socket.close() 95 | 96 | def parse_skypipe_data_stream(msg, for_pipe): 97 | """May return data from skypipe message or raises EOFError""" 98 | header = str(msg.pop(0)) 99 | command = str(msg.pop(0)) 100 | pipe_name = str(msg.pop(0)) 101 | data = str(msg.pop(0)) 102 | if header != SP_HEADER: return 103 | if pipe_name != for_pipe: return 104 | if command != SP_CMD_DATA: return 105 | if data == SP_DATA_EOF: 106 | raise EOFError() 107 | else: 108 | return data 109 | 110 | def skypipe_input_stream(endpoint, name=None): 111 | """Returns a context manager for streaming data into skypipe""" 112 | name = name or '' 113 | class context_manager(object): 114 | def __enter__(self): 115 | self.socket = ctx.socket(zmq.DEALER) 116 | self.socket.connect(endpoint) 117 | return self 118 | 119 | def send(self, data): 120 | data_msg = sp_msg(SP_CMD_DATA, name, data) 121 | self.socket.send_multipart(data_msg) 122 | 123 | def __exit__(self, *args, **kwargs): 124 | eof_msg = sp_msg(SP_CMD_DATA, name, SP_DATA_EOF) 125 | self.socket.send_multipart(eof_msg) 126 | self.socket.close() 127 | 128 | return context_manager() 129 | 130 | def stream_stdin_lines(): 131 | """Generator for unbuffered line reading from STDIN""" 132 | stdin = os.fdopen(sys.stdin.fileno(), 'r', 0) 133 | while True: 134 | line = stdin.readline() 135 | if line: 136 | yield line 137 | else: 138 | break 139 | 140 | def run(endpoint, name=None): 141 | """Runs the skypipe client""" 142 | try: 143 | if os.isatty(0): 144 | # output mode 145 | for data in stream_skypipe_output(endpoint, name): 146 | sys.stdout.write(data) 147 | sys.stdout.flush() 148 | 149 | else: 150 | # input mode 151 | with skypipe_input_stream(endpoint, name) as stream: 152 | for line in stream_stdin_lines(): 153 | stream.send(line) 154 | 155 | except KeyboardInterrupt: 156 | pass 157 | -------------------------------------------------------------------------------- /skypipe/cloud.py: -------------------------------------------------------------------------------- 1 | """Cloud satellite manager 2 | 3 | Here we use dotcloud to lookup or deploy the satellite server. This also 4 | means we need dotcloud credentials, so we get those if we need them. 5 | Most of this functionality is pulled from the dotcloud client, but is 6 | modified and organized to meet our needs. This is why we pass around and 7 | work with a cli object. This is the CLI object from the dotcloud client. 8 | """ 9 | import time 10 | import os 11 | import os.path 12 | import socket 13 | import sys 14 | import subprocess 15 | import threading 16 | from StringIO import StringIO 17 | 18 | import dotcloud.ui.cli 19 | from dotcloud.ui.config import GlobalConfig, CLIENT_KEY, CLIENT_SECRET 20 | from dotcloud.client import RESTClient 21 | from dotcloud.client.auth import NullAuth 22 | from dotcloud.client.errors import RESTAPIError 23 | 24 | from skypipe import client 25 | 26 | APPNAME = "skypipe0" 27 | satellite_path = os.path.join(os.path.abspath(os.path.dirname(__file__)), 'satellite') 28 | 29 | # This is a monkey patch to silence rsync output 30 | class FakeSubprocess(object): 31 | @staticmethod 32 | def call(*args, **kwargs): 33 | kwargs['stdout'] = subprocess.PIPE 34 | return subprocess.call(*args, **kwargs) 35 | dotcloud.ui.cli.subprocess = FakeSubprocess 36 | 37 | def wait_for(text, finish=None, io=None): 38 | """Displays dots until returned event is set""" 39 | if finish: 40 | finish.set() 41 | time.sleep(0.1) # threads, sigh 42 | if not io: 43 | io = sys.stdout 44 | finish = threading.Event() 45 | io.write(text) 46 | def _wait(): 47 | while not finish.is_set(): 48 | io.write('.') 49 | io.flush() 50 | finish.wait(timeout=1) 51 | io.write('\n') 52 | threading.Thread(target=_wait).start() 53 | return finish 54 | 55 | 56 | def lookup_endpoint(cli): 57 | """Looks up the application endpoint from dotcloud""" 58 | url = '/applications/{0}/environment'.format(APPNAME) 59 | environ = cli.user.get(url).item 60 | port = environ['DOTCLOUD_SATELLITE_ZMQ_PORT'] 61 | host = socket.gethostbyname(environ['DOTCLOUD_SATELLITE_ZMQ_HOST']) 62 | return "tcp://{0}:{1}".format(host, port) 63 | 64 | 65 | def setup_dotcloud_account(cli): 66 | """Gets user/pass for dotcloud, performs auth, and stores keys""" 67 | client = RESTClient(endpoint=cli.client.endpoint) 68 | client.authenticator = NullAuth() 69 | urlmap = client.get('/auth/discovery').item 70 | username = cli.prompt('dotCloud email') 71 | password = cli.prompt('Password', noecho=True) 72 | credential = {'token_url': urlmap.get('token'), 73 | 'key': CLIENT_KEY, 'secret': CLIENT_SECRET} 74 | try: 75 | token = cli.authorize_client(urlmap.get('token'), credential, username, password) 76 | except Exception as e: 77 | cli.die('Username and password do not match. Try again.') 78 | token['url'] = credential['token_url'] 79 | config = GlobalConfig() 80 | config.data = {'token': token} 81 | config.save() 82 | cli.global_config = GlobalConfig() # reload 83 | cli.setup_auth() 84 | cli.get_keys() 85 | 86 | def setup(cli): 87 | """Everything to make skypipe ready to use""" 88 | if not cli.global_config.loaded: 89 | setup_dotcloud_account(cli) 90 | discover_satellite(cli) 91 | cli.success("Skypipe is ready for action") 92 | 93 | 94 | def discover_satellite(cli, deploy=True, timeout=5): 95 | """Looks to make sure a satellite exists, returns endpoint 96 | 97 | First makes sure we have dotcloud account credentials. Then it looks 98 | up the environment for the satellite app. This will contain host and 99 | port to construct an endpoint. However, if app doesn't exist, or 100 | endpoint does not check out, we call `launch_satellite` to deploy, 101 | which calls `discover_satellite` again when finished. Ultimately we 102 | return a working endpoint. If deploy is False it will not try to 103 | deploy. 104 | """ 105 | if not cli.global_config.loaded: 106 | cli.die("Please setup skypipe by running `skypipe --setup`") 107 | 108 | try: 109 | endpoint = lookup_endpoint(cli) 110 | ok = client.check_skypipe_endpoint(endpoint, timeout) 111 | if ok: 112 | return endpoint 113 | else: 114 | return launch_satellite(cli) if deploy else None 115 | except (RESTAPIError, KeyError): 116 | return launch_satellite(cli) if deploy else None 117 | 118 | def destroy_satellite(cli): 119 | url = '/applications/{0}'.format(APPNAME) 120 | try: 121 | res = cli.user.delete(url) 122 | except RESTAPIError: 123 | pass 124 | 125 | def launch_satellite(cli): 126 | """Deploys a new satellite app over any existing app""" 127 | 128 | cli.info("Launching skypipe satellite:") 129 | 130 | finish = wait_for(" Pushing to dotCloud") 131 | 132 | # destroy any existing satellite 133 | destroy_satellite(cli) 134 | 135 | # create new satellite app 136 | url = '/applications' 137 | try: 138 | cli.user.post(url, { 139 | 'name': APPNAME, 140 | 'flavor': 'sandbox' 141 | }) 142 | except RESTAPIError as e: 143 | if e.code == 409: 144 | cli.die('Application "{0}" already exists.'.format(APPNAME)) 145 | else: 146 | cli.die('Creating application "{0}" failed: {1}'.format(APPNAME, e)) 147 | class args: application = APPNAME 148 | #cli._connect(args) 149 | 150 | # push satellite code 151 | protocol = 'rsync' 152 | url = '/applications/{0}/push-endpoints{1}'.format(APPNAME, '') 153 | endpoint = cli._select_endpoint(cli.user.get(url).items, protocol) 154 | class args: path = satellite_path 155 | cli.push_with_rsync(args, endpoint) 156 | 157 | # tell dotcloud to deploy, then wait for it to finish 158 | revision = None 159 | clean = False 160 | url = '/applications/{0}/deployments'.format(APPNAME) 161 | response = cli.user.post(url, {'revision': revision, 'clean': clean}) 162 | deploy_trace_id = response.trace_id 163 | deploy_id = response.item['deploy_id'] 164 | 165 | 166 | original_stdout = sys.stdout 167 | 168 | finish = wait_for(" Waiting for deployment", finish, original_stdout) 169 | 170 | try: 171 | sys.stdout = StringIO() 172 | res = cli._stream_deploy_logs(APPNAME, deploy_id, 173 | deploy_trace_id=deploy_trace_id, follow=True) 174 | if res != 0: 175 | return res 176 | except KeyboardInterrupt: 177 | cli.error('You\'ve closed your log stream with Ctrl-C, ' \ 178 | 'but the deployment is still running in the background.') 179 | cli.error('If you aborted because of an error ' \ 180 | '(e.g. the deployment got stuck), please e-mail\n' \ 181 | 'support@dotcloud.com and mention this trace ID: {0}' 182 | .format(deploy_trace_id)) 183 | cli.error('If you want to continue following your deployment, ' \ 184 | 'try:\n{0}'.format( 185 | cli._fmt_deploy_logs_command(deploy_id))) 186 | cli.die() 187 | except RuntimeError: 188 | # workaround for a bug in the current dotcloud client code 189 | pass 190 | finally: 191 | sys.stdout = original_stdout 192 | 193 | finish = wait_for(" Satellite coming online", finish) 194 | 195 | endpoint = lookup_endpoint(cli) 196 | ok = client.check_skypipe_endpoint(endpoint, 120) 197 | 198 | finish.set() 199 | time.sleep(0.1) # sigh, threads 200 | 201 | if ok: 202 | return endpoint 203 | else: 204 | cli.die("Satellite failed to come online") 205 | 206 | -------------------------------------------------------------------------------- /skypipe/satellite/builder: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | cp -a . ~ 3 | cd 4 | (cat <<-EOF 5 | virtualenv --python=python2.7 env 6 | . env/bin/activate 7 | pip install -r requirements.txt 8 | python server.py 9 | EOF 10 | ) > run 11 | chmod a+x run 12 | -------------------------------------------------------------------------------- /skypipe/satellite/dotcloud.yml: -------------------------------------------------------------------------------- 1 | satellite: 2 | type: custom 3 | buildscript: builder 4 | ports: 5 | zmq: tcp 6 | -------------------------------------------------------------------------------- /skypipe/satellite/requirements.txt: -------------------------------------------------------------------------------- 1 | pyzmq 2 | -------------------------------------------------------------------------------- /skypipe/satellite/server.py: -------------------------------------------------------------------------------- 1 | import collections 2 | import sys 3 | import os 4 | import zmq 5 | 6 | SP_HEADER = "SKYPIPE/0.1" 7 | SP_CMD_HELLO = "HELLO" 8 | SP_CMD_DATA = "DATA" 9 | SP_CMD_LISTEN = "LISTEN" 10 | SP_CMD_UNLISTEN = "UNLISTEN" 11 | SP_DATA_EOF = "" 12 | 13 | context = zmq.Context() 14 | port = os.environ.get("PORT_ZMQ", 9000) 15 | 16 | router = context.socket(zmq.ROUTER) 17 | router.bind("tcp://0.0.0.0:{}".format(port)) 18 | 19 | pipe_clients = collections.defaultdict(list) # connected skypipe clients 20 | pipe_buffers = collections.defaultdict(list) # any buffered data for pipes 21 | 22 | print "Skypipe satellite serving on {}...".format(port) 23 | 24 | def cmd_listen(): 25 | pipe_clients[pipe_name].append(client) 26 | if len(pipe_clients[pipe_name]) == 1: 27 | # if only client after adding, then previously there were 28 | # no clients and it was buffering, so spit out buffered data 29 | while len(pipe_buffers[pipe_name]) > 0: 30 | data = pipe_buffers[pipe_name].pop(0) 31 | router.send_multipart([client, 32 | SP_HEADER, SP_CMD_DATA, pipe_name, data]) 33 | if data == SP_DATA_EOF: 34 | # remember this kicks the client, so stop 35 | # sending data until the next one listens 36 | break 37 | 38 | def cmd_unlisten(): 39 | if client in pipe_clients[pipe_name]: 40 | pipe_clients[pipe_name].remove(client) 41 | 42 | def cmd_data(): 43 | data = msg.pop(0) 44 | if not pipe_clients[pipe_name]: 45 | pipe_buffers[pipe_name].append(data) 46 | else: 47 | for listener in pipe_clients[pipe_name]: 48 | router.send_multipart([listener, 49 | SP_HEADER, SP_CMD_DATA, pipe_name, data]) 50 | 51 | while True: 52 | sys.stdout.flush() 53 | 54 | msg = router.recv_multipart() 55 | client = msg.pop(0) 56 | header = str(msg.pop(0)) 57 | command = str(msg.pop(0)) 58 | 59 | # Human-friendlier version of client UUID 60 | client_display = hex(abs(hash(client)))[-6:] 61 | 62 | if SP_CMD_HELLO == command: 63 | router.send_multipart([client, SP_HEADER, SP_CMD_HELLO]) 64 | else: 65 | pipe_name = msg.pop(0) 66 | try: 67 | { 68 | SP_CMD_LISTEN: cmd_listen, 69 | SP_CMD_UNLISTEN: cmd_unlisten, 70 | SP_CMD_DATA: cmd_data, 71 | }[command]() 72 | except KeyError: 73 | print client_display, "Unknown command:", command 74 | 75 | 76 | print client_display, command 77 | --------------------------------------------------------------------------------