├── .gitignore ├── 01-simple-python-app ├── LICENSE.txt ├── hello.py ├── manifest.yml └── requirements.txt ├── 02-pydata-spyre-app ├── LICENSE.txt ├── environment.yml ├── manifest.yml └── sine_wave.py ├── 03-services-redis ├── LICENSE.txt ├── main.py ├── manifest.yml └── requirements.txt ├── 04-learning-api ├── LICENSE.txt ├── README.md ├── environment.yml ├── main.py ├── manifest.yml ├── models │ ├── ModelFactory.py │ ├── StandardModels.py │ └── __init__.py └── templates │ └── help.html ├── README.md ├── Slides-Ian_Huston-Getting_Started_With_Cloud_Foundry.pdf └── helper-notes.md /.gitignore: -------------------------------------------------------------------------------- 1 | *.pptx 2 | plan.txt 3 | -------------------------------------------------------------------------------- /01-simple-python-app/LICENSE.txt: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2015 Ian Huston 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /01-simple-python-app/hello.py: -------------------------------------------------------------------------------- 1 | """ 2 | A first simple Cloud Foundry Flask app 3 | 4 | Author: Ian Huston 5 | License: See LICENSE.txt 6 | 7 | """ 8 | from flask import Flask 9 | import os 10 | 11 | app = Flask(__name__) 12 | 13 | # Get port from environment variable or choose 9099 as local default 14 | port = int(os.getenv("PORT", 9099)) 15 | 16 | @app.route('/') 17 | def hello_world(): 18 | return 'Hello World! I am instance ' + str(os.getenv("CF_INSTANCE_INDEX", 0)) 19 | 20 | if __name__ == '__main__': 21 | # Run the app, listening on all IPs with our chosen port number 22 | app.run(host='0.0.0.0', port=port) 23 | -------------------------------------------------------------------------------- /01-simple-python-app/manifest.yml: -------------------------------------------------------------------------------- 1 | --- 2 | applications: 3 | - name: myapp 4 | memory: 128MB 5 | disk_quota: 256MB 6 | random-route: true 7 | buildpack: python_buildpack 8 | command: python hello.py 9 | -------------------------------------------------------------------------------- /01-simple-python-app/requirements.txt: -------------------------------------------------------------------------------- 1 | Flask 2 | gunicorn 3 | -------------------------------------------------------------------------------- /02-pydata-spyre-app/LICENSE.txt: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Original example: Copyright (c) 2014 Adam Hajari 4 | Additional contributions: Copyright (c) 2015 Ian Huston 5 | 6 | Permission is hereby granted, free of charge, to any person obtaining a copy 7 | of this software and associated documentation files (the "Software"), to deal 8 | in the Software without restriction, including without limitation the rights 9 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 10 | copies of the Software, and to permit persons to whom the Software is 11 | furnished to do so, subject to the following conditions: 12 | 13 | The above copyright notice and this permission notice shall be included in all 14 | copies or substantial portions of the Software. 15 | 16 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 17 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 18 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 19 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 20 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 21 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 22 | SOFTWARE. 23 | -------------------------------------------------------------------------------- /02-pydata-spyre-app/environment.yml: -------------------------------------------------------------------------------- 1 | name: spyre 2 | dependencies: 3 | - libgfortran=1.0=0 4 | - openblas=0.2.14=1 5 | - system=5.8=2 6 | - freetype=2.5.2=2 7 | - libpng=1.6.17=0 8 | - matplotlib=1.4.3=np19py27_2 9 | - numpy=1.9.2=py27_0 10 | - openssl=1.0.1k=1 11 | - pandas=0.16.1=np19py27_0 12 | - pip=7.0.3=py27_0 13 | - pyparsing=2.0.3=py27_0 14 | - python=2.7.10=0 15 | - python-dateutil=2.4.2=py27_0 16 | - pytz=2015.4=py27_0 17 | - readline=6.2=2 18 | - setuptools=17.0=py27_0 19 | - six=1.9.0=py27_0 20 | - sqlite=3.8.4.1=1 21 | - tk=8.5.18=0 22 | - zlib=1.2.8=0 23 | - pip: 24 | - cherrypy==3.7.0 25 | - dataspyre==0.1.9 26 | - jinja2==2.7.3 27 | - markupsafe==0.23 28 | - mock==1.0.1 29 | - nose==1.3.7 30 | -------------------------------------------------------------------------------- /02-pydata-spyre-app/manifest.yml: -------------------------------------------------------------------------------- 1 | --- 2 | applications: 3 | - name: sinewave 4 | memory: 512MB 5 | disk_quota: 2GB 6 | random-route: true 7 | command: python sine_wave.py 8 | buildpack: python_buildpack 9 | -------------------------------------------------------------------------------- /02-pydata-spyre-app/sine_wave.py: -------------------------------------------------------------------------------- 1 | """Example Spyre app from https://github.com/adamhajari/spyre""" 2 | import matplotlib 3 | # Needed to use headless 4 | matplotlib.use('Agg') 5 | 6 | from spyre import server 7 | 8 | import os 9 | 10 | 11 | import matplotlib.pyplot as plt 12 | import numpy as np 13 | 14 | class SimpleSineApp(server.App): 15 | title = "Simple Sine App" 16 | inputs = [{ "input_type":"text", 17 | "variable_name":"freq", 18 | "value":5, 19 | "action_id":"sine_wave_plot"}] 20 | 21 | outputs = [{"output_type":"plot", 22 | "output_id":"sine_wave_plot", 23 | "on_page_load":True }] 24 | 25 | def getPlot(self, params): 26 | f = float(params['freq']) 27 | print f 28 | x = np.arange(0,2*np.pi,np.pi/150) 29 | y = np.sin(f*x) 30 | fig = plt.figure() 31 | splt1 = fig.add_subplot(1,1,1) 32 | splt1.plot(x,y) 33 | return fig 34 | 35 | # Need to specify port for Cloud Foundry 36 | port = int(os.getenv("PORT", 9099)) 37 | 38 | app = SimpleSineApp() 39 | app.launch(port=port, host='0.0.0.0') 40 | 41 | -------------------------------------------------------------------------------- /03-services-redis/LICENSE.txt: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2015 Ian Huston 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /03-services-redis/main.py: -------------------------------------------------------------------------------- 1 | """ 2 | A simple app to show redis service integration. 3 | 4 | Author: Ian Huston 5 | License: See LICENSE.txt 6 | 7 | """ 8 | from flask import Flask 9 | import os 10 | import redis 11 | import json 12 | 13 | app = Flask(__name__) 14 | 15 | # Get port from environment variable or choose 9099 as local default 16 | port = int(os.getenv("PORT", 9099)) 17 | 18 | # Get Redis credentials 19 | if 'VCAP_SERVICES' in os.environ: 20 | services = json.loads(os.getenv('VCAP_SERVICES')) 21 | redis_env = services['rediscloud'][0]['credentials'] 22 | else: 23 | redis_env = dict(hostname='localhost', port=6379, password='') 24 | redis_env['host'] = redis_env['hostname'] 25 | del redis_env['hostname'] 26 | redis_env['port'] = int(redis_env['port']) 27 | 28 | # Connect to redis 29 | try: 30 | r = redis.StrictRedis(**redis_env) 31 | r.info() 32 | except redis.ConnectionError: 33 | r = None 34 | 35 | @app.route('/') 36 | def keys(): 37 | if r: 38 | current_hits = r.incr('hits') 39 | return 'Hits: {}\n'.format(current_hits) + 'Available Keys: ' + str(r.keys()) 40 | else: 41 | return 'No Redis connection available!' 42 | 43 | @app.route('/') 44 | def get_current_values(key): 45 | if r: 46 | current_values = r.lrange(key, 0, -1) 47 | return str(current_values) 48 | else: 49 | abort(503) 50 | 51 | @app.route('//') 52 | def add_value(key, s): 53 | if r: 54 | r.rpush(key, s) 55 | return 'Added {} to {}.'.format(s, key) 56 | else: 57 | abort(503) 58 | 59 | 60 | if __name__ == '__main__': 61 | # Run the app, listening on all IPs with our chosen port number 62 | app.run(host='0.0.0.0', port=port) 63 | -------------------------------------------------------------------------------- /03-services-redis/manifest.yml: -------------------------------------------------------------------------------- 1 | --- 2 | applications: 3 | - name: testredis 4 | memory: 128MB 5 | disk_quota: 256MB 6 | random-route: true 7 | buildpack: python_buildpack 8 | command: python main.py 9 | services: 10 | - myredis 11 | -------------------------------------------------------------------------------- /03-services-redis/requirements.txt: -------------------------------------------------------------------------------- 1 | Flask 2 | gunicorn 3 | redis 4 | -------------------------------------------------------------------------------- /04-learning-api/LICENSE.txt: -------------------------------------------------------------------------------- 1 | Copyright (c) 2015, Alexander Kagoshima, Pivotal Software Inc. 2 | All rights reserved. 3 | 4 | Redistribution and use in source and binary forms, with or without 5 | modification, are permitted provided that the following conditions are met: 6 | 7 | * Redistributions of source code must retain the above copyright notice, this 8 | list of conditions and the following disclaimer. 9 | 10 | * Redistributions in binary form must reproduce the above copyright notice, 11 | this list of conditions and the following disclaimer in the documentation 12 | and/or other materials provided with the distribution. 13 | 14 | * Neither the name of ds-cfpylearning nor the names of its 15 | contributors may be used to endorse or promote products derived from 16 | this software without specific prior written permission. 17 | 18 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 19 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 20 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 21 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 22 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 23 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 24 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 25 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 26 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 27 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 28 | -------------------------------------------------------------------------------- /04-learning-api/README.md: -------------------------------------------------------------------------------- 1 | # Simple Cloud Foundry based machine learning API 2 | 3 | Modified from code originally written by Alexander Kagoshima 4 | See the full version at https://github.com/alexkago/ds-cfpylearning 5 | 6 | This app demonstrates a very simple API that can be used to create model instances, feed data to them and let these models retrain periodically. Currently, it uses redis to store model instances, model state and data as well - for scalability and distributed processing of data this should be replaced by a distributed data storage. 7 | 8 | For all the examples below replace ```http://``` with your Cloud Foundry app domain. 9 | 10 | 11 | Create a model 12 | -- 13 | 14 | ``` 15 | curl -i -X POST -H "Content-Type: application/json" -d '{"model_name": "model1", "model_type": "LinearRegression", "retrain_counter": 10}' http:///createModel 16 | ``` 17 | 18 | 19 | Add in some data 20 | -- 21 | 22 | This example shows how to send data into the model created before, s.t. the linear regression model becomes y = x. Since we set the retrain_counter to 10 previously, the model will retrain after it received the 10th data instance. 23 | 24 | ``` 25 | curl -i -X POST -H "Content-Type: application/json" -d '{"model_name": "model1", "input": 1, "label": 1}' http:///ingest 26 | curl -i -X POST -H "Content-Type: application/json" -d '{"model_name": "model1", "input": 2, "label": 2}' http:///ingest 27 | curl -i -X POST -H "Content-Type: application/json" -d '{"model_name": "model1", "input": 3, "label": 3}' http:///ingest 28 | curl -i -X POST -H "Content-Type: application/json" -d '{"model_name": "model1", "input": 4, "label": 4}' http:///ingest 29 | curl -i -X POST -H "Content-Type: application/json" -d '{"model_name": "model1", "input": 5, "label": 5}' http:///ingest 30 | curl -i -X POST -H "Content-Type: application/json" -d '{"model_name": "model1", "input": 6, "label": 6}' http:///ingest 31 | curl -i -X POST -H "Content-Type: application/json" -d '{"model_name": "model1", "input": 7, "label": 7}' http:///ingest 32 | curl -i -X POST -H "Content-Type: application/json" -d '{"model_name": "model1", "input": 8, "label": 8}' http:///ingest 33 | curl -i -X POST -H "Content-Type: application/json" -d '{"model_name": "model1", "input": 9, "label": 9}' http:///ingest 34 | curl -i -X POST -H "Content-Type: application/json" -d '{"model_name": "model1", "input": 10, "label": 10}' http:///ingest 35 | ``` 36 | 37 | 38 | Look at all created models 39 | -- 40 | 41 | There's a very rudimentary view on the redis set of all models that have been created: 42 | 43 | ``` 44 | http:///models/ 45 | ``` 46 | 47 | 48 | Look at model details 49 | -- 50 | 51 | This lets you check out the status of the previously created model as well as its trained parameters: 52 | 53 | ``` 54 | http:///models/model1 55 | ``` 56 | 57 | License 58 | -- 59 | 60 | This application is released under the Modified BSD license. Please see the LICENSE.txt file for details. 61 | -------------------------------------------------------------------------------- /04-learning-api/environment.yml: -------------------------------------------------------------------------------- 1 | name: cfpylearning 2 | dependencies: 3 | - libgfortran=1.0=0 4 | - openblas=0.2.14=1 5 | - system=5.8=2 6 | - flask=0.10.1=py27_1 7 | - itsdangerous=0.24=py27_0 8 | - jinja2=2.7.3=py27_1 9 | - markdown=2.6.2=py27_0 10 | - markupsafe=0.23=py27_0 11 | - nose=1.3.7=py27_0 12 | - numpy=1.9.2=py27_0 13 | - openssl=1.0.1k=1 14 | - pip=7.0.3=py27_0 15 | - python=2.7.10=0 16 | - readline=6.2=2 17 | - scikit-learn=0.16.1=np19py27_0 18 | - scipy=0.15.1=np19py27_0 19 | - setuptools=17.1.1=py27_0 20 | - sqlite=3.8.4.1=1 21 | - tk=8.5.18=0 22 | - werkzeug=0.10.4=py27_0 23 | - zlib=1.2.8=0 24 | - pip: 25 | - redis==2.10.3 26 | -------------------------------------------------------------------------------- /04-learning-api/main.py: -------------------------------------------------------------------------------- 1 | import os 2 | import json 3 | import redis 4 | import pickle 5 | from markdown import markdown 6 | from flask import Flask, request, jsonify, abort, make_response, Markup, render_template, g 7 | from models.StandardModels import LinearRegression 8 | from models import ModelFactory 9 | 10 | app = Flask(__name__) 11 | 12 | # Get hostname 13 | cf_app_env = os.getenv('VCAP_APPLICATION') 14 | if cf_app_env is not None: 15 | host = json.loads(cf_app_env)['application_uris'][0] 16 | else: 17 | host = 'localhost' 18 | 19 | # initialize redis connection for local and CF deployment 20 | def connect_db(): 21 | if os.environ.get('VCAP_SERVICES') is None: # running locally 22 | DB_HOST = 'localhost' 23 | DB_PORT = 6379 24 | DB_PW = '' 25 | REDIS_DB = 1 if app.config["TESTING"] else 0 # use other db for testing 26 | 27 | else: # running on CF 28 | env_vars = os.environ['VCAP_SERVICES'] 29 | rediscloud_service = json.loads(env_vars)['rediscloud'][0] 30 | credentials = rediscloud_service['credentials'] 31 | DB_HOST = credentials['hostname'] 32 | DB_PORT = credentials['port'] 33 | DB_PW = password=credentials['password'] 34 | REDIS_DB = 0 35 | 36 | 37 | app.r = redis.StrictRedis(host=DB_HOST, 38 | port=DB_PORT, 39 | password=DB_PW, 40 | db=REDIS_DB) 41 | 42 | 43 | # define routes 44 | @app.route('/') 45 | def hello(): 46 | 47 | return render_template('help.html', host=host) 48 | 49 | 50 | @app.route('/flushDB') 51 | def flushDB(): 52 | app.r.flushdb() 53 | return 'db flushed', 200 54 | 55 | 56 | @app.route('/createModel', methods=['POST']) 57 | def createModel(): 58 | json_data = request.get_json(force=True) 59 | 60 | # check if all fields are there 61 | if json_data.get('model_name') is None: 62 | abort(make_response("model_name field is missing.\n", 422)) 63 | 64 | if json_data.get('model_type') is None: 65 | abort(make_response("model_type field is missing.\n", 422)) 66 | 67 | if json_data.get('retrain_counter') is None: 68 | abort(make_response("no retrain information set.\n", 422)) 69 | 70 | # add model to list of models 71 | app.r.sadd('models', json_data.get('model_name')) 72 | 73 | # save model definition 74 | mdl = ModelFactory.createModel(json_data.get('model_type'), 75 | json_data.get('model_name'), 76 | json_data.get('retrain_counter')) 77 | 78 | if mdl is None: 79 | return abort(make_response("No model available of type " + 80 | json_data.get('model_type') + "\n", 81 | 422)) 82 | 83 | app.r.set(json_data.get('model_name') + '_object', pickle.dumps(mdl)) 84 | 85 | return "created model: " + str(mdl) + "\n", 201 86 | 87 | 88 | @app.route('/models') 89 | def modelOverview(): 90 | return str(app.r.smembers('models')), 200 91 | 92 | 93 | @app.route('/models/') 94 | def modelInfo(model_name): 95 | return str(pickle.loads(app.r.get(model_name + '_object'))), 200 96 | 97 | 98 | @app.route('/ingest', methods=['POST']) 99 | def ingest(): 100 | json_data = request.get_json(force=True) 101 | 102 | if json_data.get('model_name') is None: 103 | abort(make_response("model_name field is missing.\n", 422)) 104 | 105 | # prepare db keys 106 | mdl_key = json_data.get('model_name') + '_object' 107 | data_key = json_data.get('model_name') + '_data' 108 | 109 | # get the model from the db 110 | pickled_mdl = app.r.get(mdl_key) 111 | mdl = pickle.loads(pickled_mdl) 112 | 113 | # pre-process data 114 | del json_data['model_name'] 115 | col_names = json_data.keys() 116 | 117 | # update the model 118 | if mdl.available_data == 0: 119 | mdl.set_data_format(col_names) 120 | else: 121 | if mdl.col_names != col_names: 122 | return abort(make_response("Data format changed!\n", 422)) 123 | 124 | mdl.avail_data_incr() 125 | 126 | # save data to redis 127 | app.r.rpush(data_key, json.dumps(json_data)) 128 | 129 | # kick off re-training 130 | if (mdl.available_data % mdl.retrain_counter) == 0: 131 | data = app.r.lrange(data_key, 0, mdl.available_data) 132 | mdl.train(data) 133 | 134 | # save model file 135 | app.r.set(mdl_key, pickle.dumps(mdl)) 136 | 137 | return json.dumps(json_data) + " added at " + data_key + "\n", 201 138 | 139 | @app.route('/score', methods=['POST']) 140 | def score(): 141 | json_data = request.get_json(force=True) 142 | 143 | if json_data.get('model_name') is None: 144 | abort(make_response("model_name field is missing.\n", 422)) 145 | 146 | # prepare db keys 147 | mdl_key = json_data.get('model_name') + '_object' 148 | pickled_mdl = app.r.get(mdl_key) 149 | mdl = pickle.loads(pickled_mdl) 150 | 151 | if not mdl.trained: 152 | return abort(make_response("Model has not been trained yet!\n", 404)) 153 | 154 | train_data = dict(json_data) 155 | del train_data['model_name'] 156 | input_keys = mdl.col_names 157 | input_keys.remove('label') 158 | 159 | if input_keys != train_data.keys(): 160 | return abort(make_response("Data format for training is different!\n", 422)) 161 | 162 | pred_val = mdl.score([train_data[key] for key in input_keys]) 163 | 164 | prediction = {'predicted_label': pred_val[0], 'request': json_data} 165 | 166 | return json.dumps(prediction), 201 167 | 168 | # run app 169 | if __name__ == "__main__": 170 | if os.environ.get('VCAP_SERVICES') is None: # running locally 171 | PORT = 8080 172 | DEBUG = True 173 | else: # running on CF 174 | PORT = int(os.getenv("PORT")) 175 | DEBUG = False 176 | 177 | connect_db() 178 | app.run(host='0.0.0.0', port=PORT, debug=DEBUG) 179 | -------------------------------------------------------------------------------- /04-learning-api/manifest.yml: -------------------------------------------------------------------------------- 1 | --- 2 | applications: 3 | - name: learning-api 4 | memory: 512M 5 | instances: 1 6 | domain: cfapps.io 7 | random-route: true 8 | path: . 9 | buildpack: python_buildpack 10 | command: python main.py 11 | services: 12 | - myredis 13 | -------------------------------------------------------------------------------- /04-learning-api/models/ModelFactory.py: -------------------------------------------------------------------------------- 1 | import json 2 | import abc 3 | 4 | class ModelInterface: 5 | __metaclass__ = abc.ABCMeta 6 | def __init__(self, model_name, retrain_counter, model_type): 7 | self.model_name = model_name 8 | self.model_type = model_type 9 | self.trained = False 10 | self.available_data = 0 11 | self.used_training_data = 0 12 | self.retrain_counter = retrain_counter 13 | 14 | def avail_data_incr(self): 15 | self.available_data += 1 16 | 17 | def set_data_format(self, col_names): 18 | self.col_names = col_names 19 | 20 | def update_mdl_state(self): 21 | self.used_training_data = self.available_data 22 | self.trained = True 23 | 24 | @abc.abstractmethod 25 | def get_parameters(self): 26 | """This method needs to be implemented""" 27 | 28 | @abc.abstractmethod 29 | def train(self, train_data): 30 | """This method needs to be implemented""" 31 | 32 | @abc.abstractmethod 33 | def score(self, score_data): 34 | """This method needs to be implemented""" 35 | 36 | def __eq__(self, other): 37 | return (isinstance(other, self.__class__) 38 | and self.__dict__ == other.__dict__) 39 | 40 | def __str__(self): 41 | obj_dict = self.__dict__ 42 | if self.trained: 43 | obj_dict['parameters'] = self.get_parameters() 44 | return str(obj_dict) 45 | 46 | 47 | def train_wrapper(func): 48 | def wrapper(self, data): 49 | # pre-process data 50 | dict_data = [json.loads(el) for el in data] 51 | col_names = dict_data[0].keys() 52 | 53 | # # run some update functions on the object 54 | # if not self.trained: 55 | # self.set_data_format(col_names) 56 | # else: 57 | # if self.col_names != col_names: 58 | # raise InputError('Data format is not the same as used before.') 59 | 60 | # run the actual training function 61 | val = func(self, dict_data, col_names) 62 | 63 | # update the model state 64 | self.update_mdl_state() 65 | 66 | return val 67 | 68 | return wrapper 69 | 70 | 71 | def createModel(model_type, model_name, retrain_counter): 72 | try: 73 | import StandardModels 74 | return getattr(StandardModels, model_type)(model_name, retrain_counter) 75 | except: 76 | try: 77 | import CustomModels 78 | return getattr(CustomModels, model_type)(model_name, retrain_counter) 79 | except: 80 | return None 81 | -------------------------------------------------------------------------------- /04-learning-api/models/StandardModels.py: -------------------------------------------------------------------------------- 1 | from ModelFactory import ModelInterface, train_wrapper 2 | from sklearn import linear_model 3 | 4 | class LinearRegression(ModelInterface): 5 | def __init__(self, name, rt_counter): 6 | ModelInterface.__init__(self, name, rt_counter, 'LinearRegression') 7 | 8 | @train_wrapper 9 | def train(self, data, col_names): 10 | col_names.remove('label') 11 | 12 | x = [[el[key] for key in col_names] for el in data] 13 | y = [el['label'] for el in data] 14 | 15 | self.mdl = linear_model.LinearRegression() 16 | self.mdl.fit(x, y, 1) 17 | 18 | return self.get_parameters() 19 | 20 | def score(self, data): 21 | return self.mdl.predict(data) 22 | 23 | def get_parameters(self): 24 | coefficients = self.mdl.coef_.tolist() 25 | coefficients.append(self.mdl.intercept_) 26 | 27 | col_names = self.col_names[:] 28 | col_names.remove('label') 29 | col_names.append('constant') 30 | 31 | return dict(zip(col_names, coefficients)) 32 | -------------------------------------------------------------------------------- /04-learning-api/models/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ihuston/python-cf-examples/ff74918e4af0540c07c43ce32d683b233d09cf83/04-learning-api/models/__init__.py -------------------------------------------------------------------------------- /04-learning-api/templates/help.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | CF based learning API 4 | 5 | 6 |

Simple Cloud Foundry based machine learning API

7 |

8 | Modified from code originally written by Alexander Kagoshima 9 | See the full version at https://github.com/alexkago/ds-cfpylearning 10 |

11 |

12 | This app demonstrates a very simple API that can be used to create model instances, feed data to them and let these models retrain periodically. Currently, it uses redis to store model instances, model state and data as well - for scalability and distributed processing of data this should be replaced by distributed data storage. 13 | 14 | 15 |

Create a model

16 | 17 | 
18 | curl -i -X POST -H "Content-Type: application/json" -d '{"model_name": "model1", "model_type": "LinearRegression", "retrain_counter": 10}' http://{{ host }}/createModel
19 |
20 | 21 | 22 |

Add in some data

23 | 24 | This example shows how to send data into the model created before, s.t. the linear regression model becomes y = x. Since we set the retrain_counter to 10 previously, the model will retrain after it received the 10th data instance. 25 | 26 | 
27 | curl -i -X POST -H "Content-Type: application/json" -d '{"model_name": "model1", "input": 1, "label": 1}' http://{{ host }}/ingest
28 | curl -i -X POST -H "Content-Type: application/json" -d '{"model_name": "model1", "input": 2, "label": 2}' http://{{ host }}/ingest
29 | curl -i -X POST -H "Content-Type: application/json" -d '{"model_name": "model1", "input": 3, "label": 3}' http://{{ host }}/ingest
30 | curl -i -X POST -H "Content-Type: application/json" -d '{"model_name": "model1", "input": 4, "label": 4}' http://{{ host }}/ingest
31 | curl -i -X POST -H "Content-Type: application/json" -d '{"model_name": "model1", "input": 5, "label": 5}' http://{{ host }}/ingest
32 | curl -i -X POST -H "Content-Type: application/json" -d '{"model_name": "model1", "input": 6, "label": 6}' http://{{ host }}/ingest
33 | curl -i -X POST -H "Content-Type: application/json" -d '{"model_name": "model1", "input": 7, "label": 7}' http://{{ host }}/ingest
34 | curl -i -X POST -H "Content-Type: application/json" -d '{"model_name": "model1", "input": 8, "label": 8}' http://{{ host }}/ingest
35 | curl -i -X POST -H "Content-Type: application/json" -d '{"model_name": "model1", "input": 9, "label": 9}' http://{{ host }}/ingest
36 | curl -i -X POST -H "Content-Type: application/json" -d '{"model_name": "model1", "input": 10, "label": 10}' http://{{ host }}/ingest
37 |
38 | 39 |

Score a new datapoint

40 | Now we can score a new datapoint by using the /score endpoint: 41 | 
42 | curl -i -X POST -H "Content-Type: application/json" -d '{"model_name": "model1", "input": 3.5}' http://{{ host }}/score
43 |
44 | 45 |

Look at all created models

46 | 47 | There's a very rudimentary view on the redis set of all models that have been created: 48 | http://{{ host }}/models. 49 | 50 |

Look at model details

51 | 52 | This lets you check out the status of the previously created model as well as its trained parameters: 53 | http://{{ host }}/models/model1 54 | 55 |

56 | 57 | 58 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | Python Cloud Foundry Examples 2 | ============================= 3 | 4 | Author: [Ian Huston](http://ianhuston.net) [@ianhuston](http://twitter.com/ianhuston) 5 | 6 | License: See LICENSE.txt in each example code folder 7 | 8 | First presented at [PyData London 2015](http://london.pydata.org). 9 | 10 | This tutorial is an introduction to deploying Python applications 11 | to [Cloud Foundry](http://cloudfoundry.org) and covers: 12 | 13 | * Pushing your first application to Cloud Foundry 14 | * Using a custom buildpack to use PyData packages in your app 15 | * Adding data services like Redis to your app 16 | 17 | Each example app is self-contained and can be deployed to a Cloud Foundry 18 | instance with 19 | 20 | $ cf push 21 | 22 | The slides are available [on Speakerdeck](https://speakerdeck.com/ihuston/pydata-london-2015-getting-started-with-cloud-foundry-for-data-science). There is also a [full length video](https://www.youtube.com/watch?v=lp2c05yDJko) of this tutorial from PyData London 2015. 23 | 24 | A [shorter version of these notes](./helper-notes.md), which can be used by assistants in the tutorial 25 | is also provided. 26 | 27 | ### Changes: 28 | 29 | *Update 03/10/2016*: The official CF Python buildpack now includes support for the conda package manager. 30 | This means there is no need to use a separate community buildpack to run apps using PyData packages. 31 | The tutorial has been updated to reflect this. 32 | 33 | *Update 23/11/2015*: These sample apps have been updated to use the [new Diego runtime](http://support.run.pivotal.io/entries/105844873-Migrating-Applications-from-DEAs-to-Diego). 34 | The main difference is that app instances all run on the same port inside different containers. 35 | The first sample app has been changed to show the instance number rather than the port so that multiple instances can be recognised. 36 | 37 | ### Getting Started 38 | 39 | If you do not have an account on a Cloud Foundry installation you can 40 | register for a free trial at [Pivotal Web Services (PWS)](http://run.pivotal.io). 41 | 42 | Download the Cloud Foundry Command Line Interface from the CF management console 43 | or [the CF Github repo](https://github.com/cloudfoundry/cli). 44 | This provides the `cf` command which you will use to interact with a CF installation. 45 | 46 | Target the CF installation API with `cf api`. For example for PWS: 47 | 48 | $ cf api https://api.run.pivotal.io 49 | 50 | ### Pushing your first app 51 | 52 | The first app is in the folder `01-simple-python-app`. 53 | Have a look at the code and then deploy using `cf push`. 54 | 55 | $ cd 01-simple-python-app 56 | $ cf push 57 | 58 | You can inspect the app's streaming logs using `cf logs myapp`. 59 | 60 | You can scale your app by changing the number of instances and the available RAM: 61 | 62 | $ cf scale myapp -i 5 63 | $ cf scale myapp -m 256M 64 | 65 | ### Pushing an app with PyData dependencies 66 | 67 | The second app is in the folder `02-pydata-spyre-app`. 68 | This app is built using [Spyre](https://github.com/adamhajari/spyre) by Adam Hajari. 69 | 70 | This application uses a new feature of the official [Python buildpack](https://github.com/cloudfoundry/python-buildpack/) 71 | to provision dependencies using `conda`. 72 | 73 | The `environment.yml` file contains the `conda` environment specification. 74 | The buildpack detects this file and uses `conda` instead of `pip` to install the dependencies. 75 | 76 | Push the app using `cf push` and visit the app URL to see a simple interactive visualisation. 77 | 78 | $ cd ../02-pydata-spyre-app 79 | $ cf push 80 | 81 | ### Using data services 82 | 83 | The third app is in the folder `03-services-redis`. 84 | 85 | Services provide persistent storage and other useful resources for your applications. 86 | 87 | First, create a free Redis service provided by Rediscloud: 88 | 89 | $ cf create-service rediscloud 30mb myredis 90 | 91 | You can bind this to your apps making it available via the `VCAP_SERVICES` 92 | environmental variable. 93 | 94 | $ cf bind-service myapp myredis 95 | 96 | Look at the environmental variables your app has with `cf env myapp`. 97 | 98 | Now you can push the third app which uses Redis as its backing store. 99 | 100 | $ cd ../03-services-redis 101 | $ cf push 102 | 103 | The manifest specifies that this app should be bound to the myredis service instance that you just created. 104 | 105 | ### Putting it all together 106 | 107 | The fourth app is in the folder `04-learning-api`. 108 | 109 | We can put everything we've learned into practice by building our own simple 110 | machine learning API. 111 | 112 | $ cd ../04-learning-api 113 | $ cf push 114 | 115 | This is a simplified version of [a project](https://github.com/alexkago/ds-cfpylearning) 116 | by [Alex Kagoshima](http://twitter.com/akagoshima). 117 | 118 | **Don't forget to stop all your apps when you are finished by doing the following:** 119 | 120 | $ cf stop myapp 121 | $ cf stop sinewave 122 | $ cf stop testredis 123 | $ cf stop learning-api 124 | 125 | ### Resources 126 | 127 | * [Cloud Foundry documentation](http://docs.cloudfoundry.org) 128 | * [Pivotal Web Services](http://run.pivotal.io) 129 | * [CF Summit Videos](https://www.youtube.com/playlist?list=PLhuMOCWn4P9g-UMN5nzDiw78zgf5rJ4gR) 130 | * [CF Meetups](http://cloud-foundry.meetup.com) 131 | * [Slides for this tutorial](https://speakerdeck.com/ihuston/pydata-london-2015-getting-started-with-cloud-foundry-for-data-science) 132 | * [Video of this tutorial](https://www.youtube.com/watch?v=lp2c05yDJko) 133 | -------------------------------------------------------------------------------- /Slides-Ian_Huston-Getting_Started_With_Cloud_Foundry.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ihuston/python-cf-examples/ff74918e4af0540c07c43ce32d683b233d09cf83/Slides-Ian_Huston-Getting_Started_With_Cloud_Foundry.pdf -------------------------------------------------------------------------------- /helper-notes.md: -------------------------------------------------------------------------------- 1 | ### Helper Instructions 2 | 3 | Each example app is self-contained and can be deployed to a Cloud Foundry 4 | instance with `$ cf push`. 5 | 6 | The slides are available at https://speakerdeck.com/ihuston/pydata-london-2015-getting-started-with-cloud-foundry-for-data-science. 7 | 8 | What participants need to do: 9 | 10 | 1. Register on http://run.pivotal.io 11 | 2. Download CF CLI from https://github.com/cloudfoundry/cli 12 | 3. Clone the tutorial repo from http://github.com/ihuston/python-cf-examples 13 | OR download file at http://tinyurl.com/cf-pydata 14 | 15 | ### Getting Started 16 | 17 | If you do not have an account on a Cloud Foundry installation you can 18 | register for a free trial at Pivotal Web Services (PWS) http://run.pivotal.io. 19 | 20 | Download the Cloud Foundry Command Line Interface from the CF management console 21 | or the CF Github repo: https://github.com/cloudfoundry/cli. 22 | This provides the `cf` command which you will use to interact with a CF installation. 23 | 24 | Target the CF installation API with `cf api`. For example for PWS: 25 | 26 | $ cf api https://api.run.pivotal.io 27 | 28 | ### Pushing your first app 29 | 30 | The first app is in the folder `01-simple-python-app`. 31 | Have a look at the code and then deploy using `cf push`. 32 | 33 | You can inspect the app's streaming logs using `cf logs myapp`. 34 | 35 | You can scale your app by changing the number of instances and the available RAM: 36 | 37 | $ cf scale myapp -i 5 38 | $ cf scale myapp -m 256M 39 | 40 | ### Pushing an app with PyData dependencies 41 | 42 | The second app is in the folder `02-pydata-spyre-app`. 43 | This app is built using [Spyre](https://github.com/adamhajari/spyre) by Adam Hajari. 44 | 45 | This application uses a new feature of the official [Python buildpack](https://github.com/cloudfoundry/python-buildpack/) 46 | to provision dependencies using `conda`. 47 | 48 | The `environment.yml` file contains the `conda` environment specification. 49 | The buildpack detects this file and uses `conda` instead of `pip` to install the dependencies. 50 | 51 | Push the app using `cf push` and visit the app URL to see a simple interactive visualisation. 52 | 53 | ### Using data services 54 | 55 | The third app is in the folder `03-services-redis`. 56 | 57 | Services provide persistent storage and other useful resources for your applications. 58 | 59 | First, create a free Redis service provided by Rediscloud: 60 | 61 | $ cf create-service rediscloud 30mb myredis 62 | 63 | You can bind this to your apps making it available via the `VCAP_SERVICES` 64 | environmental variable. 65 | 66 | $ cf bind-service myapp myredis 67 | 68 | Look at the environmental variables your app has with `cf env myapp`. 69 | 70 | Now you can push the third app which uses Redis as its backing store. 71 | 72 | The manifest specifies that this app should be bound to the myredis service instance that you just created. 73 | 74 | ### Putting it all together 75 | 76 | The fourth app is in the folder `04-learning-api`. 77 | 78 | We can put everything we've learned into practice by building our own simple 79 | machine learning API. 80 | 81 | **Don't forget to stop all your apps when you are finished.** 82 | --------------------------------------------------------------------------------