├── .gitignore ├── LICENSE ├── README.md ├── TensorFlow_World_Presentation_on_Advanced_Model_Deployments_with_TensorFlow_Serving_Hannes_Hapke.pdf ├── example_model ├── TensorFlow Lite Example.ipynb ├── Train_example_model.ipynb ├── client_example.ipynb ├── convert_model_to_tflite.py └── train.py ├── example_tf_serving_configurations ├── model_config_list.txt ├── model_config_list_with_labels.txt ├── monitoring_config_file.txt └── prometheus.yml └── requirements.txt /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | *$py.class 5 | 6 | # C extensions 7 | *.so 8 | 9 | # Distribution / packaging 10 | .Python 11 | build/ 12 | develop-eggs/ 13 | dist/ 14 | downloads/ 15 | eggs/ 16 | .eggs/ 17 | lib/ 18 | lib64/ 19 | parts/ 20 | sdist/ 21 | var/ 22 | wheels/ 23 | *.egg-info/ 24 | .installed.cfg 25 | *.egg 26 | MANIFEST 27 | 28 | # PyInstaller 29 | # Usually these files are written by a python script from a template 30 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 31 | *.manifest 32 | *.spec 33 | 34 | # Installer logs 35 | pip-log.txt 36 | pip-delete-this-directory.txt 37 | 38 | # Unit test / coverage reports 39 | htmlcov/ 40 | .tox/ 41 | .coverage 42 | .coverage.* 43 | .cache 44 | nosetests.xml 45 | coverage.xml 46 | *.cover 47 | .hypothesis/ 48 | .pytest_cache/ 49 | 50 | # Translations 51 | *.mo 52 | *.pot 53 | 54 | # Django stuff: 55 | *.log 56 | local_settings.py 57 | db.sqlite3 58 | 59 | # Flask stuff: 60 | instance/ 61 | .webassets-cache 62 | 63 | # Scrapy stuff: 64 | .scrapy 65 | 66 | # Sphinx documentation 67 | docs/_build/ 68 | 69 | # PyBuilder 70 | target/ 71 | 72 | # Jupyter Notebook 73 | .ipynb_checkpoints 74 | 75 | # pyenv 76 | .python-version 77 | 78 | # celery beat schedule file 79 | celerybeat-schedule 80 | 81 | # SageMath parsed files 82 | *.sage.py 83 | 84 | # Environments 85 | .env 86 | .venv 87 | env/ 88 | venv/ 89 | ENV/ 90 | env.bak/ 91 | venv.bak/ 92 | 93 | # Spyder project settings 94 | .spyderproject 95 | .spyproject 96 | 97 | # Rope project settings 98 | .ropeproject 99 | 100 | # mkdocs documentation 101 | /site 102 | 103 | # mypy 104 | .mypy_cache/ 105 | 106 | # project 107 | example_model/dataset.csv 108 | tf-world-tf-serving/ -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2019 Hannes Hapke 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Code Examples for the TensorFlow World talk on "Advanced model deployments with TensorFlow Serving" 2 | 3 | This repository contains all code examples for my TensorFlow World talk about "Advanced model deployments with TensorFlow Serving" 4 | 5 | If you have questions, please reach out via twitter @hanneshapke. 6 | You can also find further information on [buildingmlpipelines.com](http://buildingmlpipelines.com). 7 | 8 | 9 | ### Model Data Set 10 | 11 | The example model was trained with the dataset for "IMDB movie reviews for Sentiment Analysis" which can be found [here](https://www.kaggle.com/oumaimahourrane/imdb-reviews/data). In order to retrain the model, log in to Kaggle and obtain the dataset. It is assumed that the uncompressed file is saved in the same folder as `train.py` file. 12 | 13 | For copyright reasons, the data set isn't included in this repository. 14 | 15 | 16 | ### Getting started 17 | 18 | ``` 19 | $ virtualenv -p python3 tf-world-tf-serving 20 | $ source tf-world-tf-serving/bin/activate 21 | $ pip3 install -r requirements.txt 22 | ``` 23 | 24 | 25 | 26 | ### Starting up TensorFlow Serving 27 | 28 | 29 | #### 90 sec Model Deployment 30 | 31 | With the example below you can start your TensorFlow Serving instance on your host machine. 32 | 33 | ``` 34 | $ docker run -p 8500:8500 \ 35 | -p 8501:8501 \ 36 | --mount type=bind, source=saved_models/, target=/models/my_model \ 37 | -e MODEL_NAME=my_model 38 | -t tensorflow/serving 39 | ``` 40 | 41 | #### Loading two Versions of the same Model 42 | 43 | The example below lets you server multiple models. You'll need to update your configuration file before starting the server. 44 | 45 | ``` 46 | docker run -p 8501:8501 \ 47 | --mount type=bind,source=`pwd`,target=/models/my_model \ 48 | --mount type=bind,source=`pwd`/../../example_tf_serving_configurations,target=/models/model_config \ 49 | -t tensorflow/serving \ 50 | --model_config_file=/models/model_config/model_config_list.txt 51 | ``` 52 | 53 | #### Loading two Versions of the same Model with Version Labels 54 | 55 | The example below lets you server multiple model version. You'll need to update your configuration file before starting the server. 56 | 57 | ``` 58 | docker run -p 8501:8501 \ 59 | --mount type=bind,source=`pwd`,target=/models/my_model \ 60 | --mount type=bind,source=`pwd`/../../example_tf_serving_configurations,target=/models/model_config \ 61 | -t tensorflow/serving \ 62 | --model_config_file=/models/model_config/model_config_list_with_labels.txt \ 63 | --allow_version_labels_for_unavailable_models=true 64 | ``` 65 | 66 | 67 | 68 | #### Run Prometheus Container 69 | 70 | With the Prometheus config file below, you can run a basic instance of Prometheus. 71 | 72 | ``` 73 | global: 74 | scrape_interval: 15s 75 | evaluation_interval: 15s 76 | external_labels: 77 | monitor: 'tf-serving-monitor' 78 | scrape_configs: 79 | - job_name: 'prometheus' 80 | scrape_interval: 5s 81 | metrics_path: /monitoring/prometheus/metrics 82 | static_configs: 83 | - targets: ['host.docker.internal:8501'] 84 | ``` 85 | 86 | ``` 87 | docker run -p 9090:9090 -v /tmp/prometheus.yml:/etc/prometheus/prometheus.yml \ 88 | prom/prometheus 89 | ``` 90 | 91 | Once Prometheus is running, you can config TFServing to provide an monitoring endpoint for Prometheus. 92 | 93 | ``` 94 | docker run -p 8501:8501 \ 95 | --mount type=bind,source=`pwd`,target=/models/my_model \ 96 | --mount type=bind,source=`pwd`/../../example_tf_serving_configurations,target=/models/model_config \ 97 | -t tensorflow/serving \ 98 | --model_config_file=/models/model_config/model_config_list_with_labels.txt \ 99 | --monitoring_config_file=/models/model_config/monitoring_config_file.txt 100 | ``` 101 | 102 | #### Loading TF Lite Models 103 | 104 | If you want to host TFLite models, use the command below to run your TFServing instance. 105 | 106 | ``` 107 | docker run -p 8501:8501 \ 108 | --mount type=bind,source=`pwd`/tflite,target=/models/my_model \ 109 | -e MODEL_BASE_PATH=/models \ 110 | -e MODEL_NAME=my_model \ 111 | -t tensorflow/serving:latest \ 112 | --use_tflite_model=true 113 | ``` 114 | -------------------------------------------------------------------------------- /TensorFlow_World_Presentation_on_Advanced_Model_Deployments_with_TensorFlow_Serving_Hannes_Hapke.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hanneshapke/TensorFlow-World-Adv-Introduction-TF-Serving/b08d9f8f6ebca5ace806787c723725879be4c847/TensorFlow_World_Presentation_on_Advanced_Model_Deployments_with_TensorFlow_Serving_Hannes_Hapke.pdf -------------------------------------------------------------------------------- /example_model/TensorFlow Lite Example.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [] 9 | } 10 | ], 11 | "metadata": { 12 | "kernelspec": { 13 | "display_name": "Python 3", 14 | "language": "python", 15 | "name": "python3" 16 | }, 17 | "language_info": { 18 | "codemirror_mode": { 19 | "name": "ipython", 20 | "version": 3 21 | }, 22 | "file_extension": ".py", 23 | "mimetype": "text/x-python", 24 | "name": "python", 25 | "nbconvert_exporter": "python", 26 | "pygments_lexer": "ipython3", 27 | "version": "3.7.1" 28 | } 29 | }, 30 | "nbformat": 4, 31 | "nbformat_minor": 2 32 | } 33 | -------------------------------------------------------------------------------- /example_model/Train_example_model.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 7, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import os\n", 10 | "import time\n", 11 | "\n", 12 | "import numpy as np\n", 13 | "import pandas as pd\n", 14 | "import tensorflow as tf\n", 15 | "import tensorflow_hub as hub\n", 16 | "from tensorflow.keras import backend as K\n", 17 | "\n", 18 | "\n", 19 | "def load_training_data(fpath, num_val_samples=250):\n", 20 | " df = pd.read_csv(fpath, usecols=['SentimentText', 'Sentiment'])\n", 21 | " df = df.sample(frac=1).reset_index(drop=True)\n", 22 | "\n", 23 | " text = df['SentimentText'].tolist()\n", 24 | " text = [str(t).encode('ascii', 'replace') for t in text]\n", 25 | " text = np.array(text, dtype=object)[:]\n", 26 | " # text = np.array(text, dtype=object)[:, np.newaxis]\n", 27 | " # labels = np.asarray(pd.get_dummies(df.label), dtype=np.int8)\n", 28 | " labels = df['Sentiment'].tolist()\n", 29 | " labels = np.array(pd.get_dummies(labels), dtype=int)[:] \n", 30 | "\n", 31 | " train_text = text[num_val_samples:]\n", 32 | " train_labels = labels[num_val_samples:]\n", 33 | " val_text = text[:num_val_samples]\n", 34 | " val_labels = labels[:num_val_samples]\n", 35 | "\n", 36 | " return (train_text, train_labels), (val_text, val_labels)" 37 | ] 38 | }, 39 | { 40 | "cell_type": "code", 41 | "execution_count": 8, 42 | "metadata": {}, 43 | "outputs": [], 44 | "source": [ 45 | "fpath = \"dataset.csv\"\n", 46 | "training_data, val_data = load_training_data(fpath)" 47 | ] 48 | }, 49 | { 50 | "cell_type": "code", 51 | "execution_count": 25, 52 | "metadata": {}, 53 | "outputs": [], 54 | "source": [ 55 | "def get_model(num_categories=4):\n", 56 | " hub_layer = hub.KerasLayer(\"https://tfhub.dev/google/tf2-preview/nnlm-en-dim50/1\", output_shape=[50], \n", 57 | " input_shape=[], dtype=tf.string)\n", 58 | "\n", 59 | " # hub_layer = hub.KerasLayer(\"https://tfhub.dev/google/tf2-preview/gnews-swivel-20dim-with-oov/1\", output_shape=[20],\n", 60 | " # input_shape=[], dtype=tf.string)\n", 61 | "\n", 62 | " model = tf.keras.Sequential()\n", 63 | " model.add(hub_layer)\n", 64 | " model.add(tf.keras.layers.Dense(16, activation='relu'))\n", 65 | " model.add(tf.keras.layers.Dense(2, activation='softmax'))\n", 66 | "\n", 67 | " model.summary()\n", 68 | " model.compile(loss='categorical_crossentropy',\n", 69 | " optimizer='RMSProp', metrics=['acc'])\n", 70 | " return model\n", 71 | "\n", 72 | "\n", 73 | "def train(fpath, epochs=2, batch_size=32):\n", 74 | " training_data, val_data = load_training_data(fpath)\n", 75 | "\n", 76 | " model = get_model()\n", 77 | " model.fit(training_data[0],\n", 78 | " training_data[1],\n", 79 | " validation_data=val_data,\n", 80 | " epochs=epochs,\n", 81 | " batch_size=batch_size)\n", 82 | " return model" 83 | ] 84 | }, 85 | { 86 | "cell_type": "code", 87 | "execution_count": 26, 88 | "metadata": { 89 | "scrolled": true 90 | }, 91 | "outputs": [ 92 | { 93 | "name": "stdout", 94 | "output_type": "stream", 95 | "text": [ 96 | "Model: \"sequential_3\"\n", 97 | "_________________________________________________________________\n", 98 | "Layer (type) Output Shape Param # \n", 99 | "=================================================================\n", 100 | "keras_layer_3 (KerasLayer) (None, 50) 48190600 \n", 101 | "_________________________________________________________________\n", 102 | "dense_6 (Dense) (None, 16) 816 \n", 103 | "_________________________________________________________________\n", 104 | "dense_7 (Dense) (None, 2) 34 \n", 105 | "=================================================================\n", 106 | "Total params: 48,191,450\n", 107 | "Trainable params: 850\n", 108 | "Non-trainable params: 48,190,600\n", 109 | "_________________________________________________________________\n", 110 | "Train on 24750 samples, validate on 250 samples\n", 111 | "Epoch 1/2\n", 112 | "24750/24750 [==============================] - 133s 5ms/sample - loss: 0.5762 - acc: 0.7004 - val_loss: 0.5094 - val_acc: 0.7640\n", 113 | "Epoch 2/2\n", 114 | "24750/24750 [==============================] - 138s 6ms/sample - loss: 0.5301 - acc: 0.7347 - val_loss: 0.4992 - val_acc: 0.7600\n" 115 | ] 116 | } 117 | ], 118 | "source": [ 119 | " fpath = \"dataset.csv\"\n", 120 | " model = train(fpath)" 121 | ] 122 | }, 123 | { 124 | "cell_type": "code", 125 | "execution_count": 30, 126 | "metadata": {}, 127 | "outputs": [ 128 | { 129 | "data": { 130 | "text/plain": [ 131 | "array([[0.46619886, 0.53380114]], dtype=float32)" 132 | ] 133 | }, 134 | "execution_count": 30, 135 | "metadata": {}, 136 | "output_type": "execute_result" 137 | } 138 | ], 139 | "source": [ 140 | "model.predict([\"Terrible movie.\"])" 141 | ] 142 | }, 143 | { 144 | "cell_type": "code", 145 | "execution_count": 29, 146 | "metadata": {}, 147 | "outputs": [ 148 | { 149 | "data": { 150 | "text/plain": [ 151 | "array([[0.4473396 , 0.55266035]], dtype=float32)" 152 | ] 153 | }, 154 | "execution_count": 29, 155 | "metadata": {}, 156 | "output_type": "execute_result" 157 | } 158 | ], 159 | "source": [ 160 | "model.predict([\"Very best movie ever.\"])" 161 | ] 162 | }, 163 | { 164 | "cell_type": "code", 165 | "execution_count": 31, 166 | "metadata": {}, 167 | "outputs": [], 168 | "source": [ 169 | "def export_model(model, base_path=\"./exported_models/\"):\n", 170 | " path = os.path.join(base_path, str(int(time.time())))\n", 171 | " tf.saved_model.save(model, path)" 172 | ] 173 | }, 174 | { 175 | "cell_type": "code", 176 | "execution_count": 33, 177 | "metadata": {}, 178 | "outputs": [ 179 | { 180 | "name": "stdout", 181 | "output_type": "stream", 182 | "text": [ 183 | "INFO:tensorflow:Assets written to: ./exported_models/1571698198/assets\n" 184 | ] 185 | }, 186 | { 187 | "name": "stderr", 188 | "output_type": "stream", 189 | "text": [ 190 | "INFO:tensorflow:Assets written to: ./exported_models/1571698198/assets\n" 191 | ] 192 | } 193 | ], 194 | "source": [ 195 | "export_model(model)" 196 | ] 197 | }, 198 | { 199 | "cell_type": "code", 200 | "execution_count": null, 201 | "metadata": {}, 202 | "outputs": [], 203 | "source": [] 204 | } 205 | ], 206 | "metadata": { 207 | "kernelspec": { 208 | "display_name": "Python 3", 209 | "language": "python", 210 | "name": "python3" 211 | }, 212 | "language_info": { 213 | "codemirror_mode": { 214 | "name": "ipython", 215 | "version": 3 216 | }, 217 | "file_extension": ".py", 218 | "mimetype": "text/x-python", 219 | "name": "python", 220 | "nbconvert_exporter": "python", 221 | "pygments_lexer": "ipython3", 222 | "version": "3.7.1" 223 | } 224 | }, 225 | "nbformat": 4, 226 | "nbformat_minor": 2 227 | } 228 | -------------------------------------------------------------------------------- /example_model/client_example.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "#!pip install requests" 10 | ] 11 | }, 12 | { 13 | "cell_type": "code", 14 | "execution_count": 8, 15 | "metadata": {}, 16 | "outputs": [], 17 | "source": [ 18 | "HOST = '127.0.0.1'\n", 19 | "PORT = '8501'\n", 20 | "MODEL_NAME = 'my_model'\n", 21 | "url = f'http://{HOST}:{PORT}/v1/models/{MODEL_NAME}'" 22 | ] 23 | }, 24 | { 25 | "cell_type": "code", 26 | "execution_count": 28, 27 | "metadata": {}, 28 | "outputs": [], 29 | "source": [ 30 | "import json\n", 31 | "import requests\n", 32 | "\n", 33 | "def rest_request(text, url=None):\n", 34 | " \"\"\"Example inference of a text classification\"\"\"\n", 35 | " if url is None:\n", 36 | " url = 'http://localhost:8501/v1/models/my_model/versions/1571698198:predict'\n", 37 | " payload = json.dumps({\"instances\": [text]})\n", 38 | " response = requests.post(url, payload)\n", 39 | " return response" 40 | ] 41 | }, 42 | { 43 | "cell_type": "code", 44 | "execution_count": 43, 45 | "metadata": {}, 46 | "outputs": [ 47 | { 48 | "name": "stdout", 49 | "output_type": "stream", 50 | "text": [ 51 | "{'predictions': [[0.0526611395, 0.947338879]]}\n" 52 | ] 53 | } 54 | ], 55 | "source": [ 56 | "rs = rest_request(text=\"This is a really really really really really really really really really really really really really really really really really really really really really really really really really long sentence\")\n", 57 | "print(rs.json())\n" 58 | ] 59 | }, 60 | { 61 | "cell_type": "code", 62 | "execution_count": 57, 63 | "metadata": {}, 64 | "outputs": [], 65 | "source": [ 66 | "import json\n", 67 | "import requests\n", 68 | "\n", 69 | "def get_model_status(model_name, host='localhost', port='8501', version=None):\n", 70 | " url = f\"http://{host}:{port}/v1/models/{model_name}/\"\n", 71 | " if version:\n", 72 | " url += f\"versions/{version}\"\n", 73 | " response = requests.get(url)\n", 74 | " return response" 75 | ] 76 | }, 77 | { 78 | "cell_type": "code", 79 | "execution_count": 63, 80 | "metadata": {}, 81 | "outputs": [ 82 | { 83 | "name": "stdout", 84 | "output_type": "stream", 85 | "text": [ 86 | "{'model_version_status': [{'version': '1571698198', 'state': 'AVAILABLE', 'status': {'error_code': 'OK', 'error_message': ''}}]}\n" 87 | ] 88 | } 89 | ], 90 | "source": [ 91 | "rs = get_model_status(model_name='my_model', version='1571698198')\n", 92 | "print(rs.json())" 93 | ] 94 | }, 95 | { 96 | "cell_type": "code", 97 | "execution_count": 64, 98 | "metadata": {}, 99 | "outputs": [], 100 | "source": [ 101 | "import json\n", 102 | "import requests\n", 103 | "\n", 104 | "def get_model_metadata(model_name, host='localhost', port='8501', version=None):\n", 105 | " url = f\"http://{host}:{port}/v1/models/{model_name}/\"\n", 106 | " if version:\n", 107 | " url += f\"versions/{version}\"\n", 108 | " url += f\"/metadata\"\n", 109 | " response = requests.get(url)\n", 110 | " return response" 111 | ] 112 | }, 113 | { 114 | "cell_type": "code", 115 | "execution_count": 65, 116 | "metadata": {}, 117 | "outputs": [ 118 | { 119 | "data": { 120 | "text/plain": [ 121 | "{'model_spec': {'name': 'my_model',\n", 122 | " 'signature_name': '',\n", 123 | " 'version': '1571698198'},\n", 124 | " 'metadata': {'signature_def': {'signature_def': {'serving_default': {'inputs': {'keras_layer_3_input': {'dtype': 'DT_STRING',\n", 125 | " 'tensor_shape': {'dim': [{'size': '-1', 'name': ''}],\n", 126 | " 'unknown_rank': False},\n", 127 | " 'name': 'serving_default_keras_layer_3_input:0'}},\n", 128 | " 'outputs': {'dense_7': {'dtype': 'DT_FLOAT',\n", 129 | " 'tensor_shape': {'dim': [{'size': '-1', 'name': ''},\n", 130 | " {'size': '2', 'name': ''}],\n", 131 | " 'unknown_rank': False},\n", 132 | " 'name': 'StatefulPartitionedCall_2:0'}},\n", 133 | " 'method_name': 'tensorflow/serving/predict'},\n", 134 | " '__saved_model_init_op': {'inputs': {},\n", 135 | " 'outputs': {'__saved_model_init_op': {'dtype': 'DT_INVALID',\n", 136 | " 'tensor_shape': {'dim': [], 'unknown_rank': True},\n", 137 | " 'name': 'NoOp'}},\n", 138 | " 'method_name': ''}}}}}" 139 | ] 140 | }, 141 | "execution_count": 65, 142 | "metadata": {}, 143 | "output_type": "execute_result" 144 | } 145 | ], 146 | "source": [ 147 | "rs = get_model_metadata(model_name='my_model', version='1571698198')\n", 148 | "rs.json()" 149 | ] 150 | }, 151 | { 152 | "cell_type": "code", 153 | "execution_count": null, 154 | "metadata": {}, 155 | "outputs": [], 156 | "source": [] 157 | } 158 | ], 159 | "metadata": { 160 | "kernelspec": { 161 | "display_name": "Python 3", 162 | "language": "python", 163 | "name": "python3" 164 | }, 165 | "language_info": { 166 | "codemirror_mode": { 167 | "name": "ipython", 168 | "version": 3 169 | }, 170 | "file_extension": ".py", 171 | "mimetype": "text/x-python", 172 | "name": "python", 173 | "nbconvert_exporter": "python", 174 | "pygments_lexer": "ipython3", 175 | "version": "3.7.1" 176 | } 177 | }, 178 | "nbformat": 4, 179 | "nbformat_minor": 2 180 | } 181 | -------------------------------------------------------------------------------- /example_model/convert_model_to_tflite.py: -------------------------------------------------------------------------------- 1 | import tensorflow as tf 2 | 3 | saved_model_dir = "path_to_saved_model" 4 | converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir) 5 | 6 | converter.optimizations = [tf.lite.Optimize.DEFAULT] 7 | tflite_model = converter.convert() 8 | open("/tmp/model.tflite", "wb").write(tflite_model) 9 | -------------------------------------------------------------------------------- /example_model/train.py: -------------------------------------------------------------------------------- 1 | 2 | import os 3 | import time 4 | 5 | import numpy as np 6 | import pandas as pd 7 | import tensorflow as tf 8 | import tensorflow_hub as hub 9 | from tensorflow.keras import backend as K 10 | 11 | 12 | def load_training_data(fpath, num_val_samples=250): 13 | df = pd.read_csv(fpath, usecols=['SentimentText', 'Sentiment']) 14 | df = df.sample(frac=1).reset_index(drop=True) 15 | 16 | text = df['SentimentText'].tolist() 17 | text = [str(t).encode('ascii', 'replace') for t in text] 18 | text = np.array(text, dtype=object)[:] 19 | # text = np.array(text, dtype=object)[:, np.newaxis] 20 | # labels = np.asarray(pd.get_dummies(df.label), dtype=np.int8) 21 | labels = df['Sentiment'].tolist() 22 | labels = np.array(labels, dtype=int)[:] 23 | 24 | train_text = text[num_val_samples:] 25 | train_labels = labels[num_val_samples:] 26 | val_text = text[:num_val_samples] 27 | val_labels = labels[:num_val_samples] 28 | 29 | return (train_text, train_labels), (val_text, val_labels) 30 | 31 | 32 | def get_model(num_categories=4): 33 | hub_layer = hub.KerasLayer("https://tfhub.dev/google/tf2-preview/nnlm-en-dim50/1", output_shape=[50], 34 | input_shape=[], dtype=tf.string) 35 | 36 | # hub_layer = hub.KerasLayer("https://tfhub.dev/google/tf2-preview/gnews-swivel-20dim-with-oov/1", output_shape=[20], 37 | # input_shape=[], dtype=tf.string) 38 | 39 | model = tf.keras.Sequential() 40 | model.add(hub_layer) 41 | model.add(tf.keras.layers.Dense(16, activation='relu')) 42 | model.add(tf.keras.layers.Dense(1, activation='sigmoid')) 43 | 44 | model.summary() 45 | model.compile(loss='binary_crossentropy', 46 | optimizer='RMSProp', metrics=['acc']) 47 | return model 48 | 49 | 50 | def train(fpath, epochs=4, batch_size=32): 51 | training_data, val_data = load_training_data(fpath) 52 | 53 | model = get_model() 54 | model.fit(training_data[0], 55 | training_data[1], 56 | validation_data=val_data, 57 | epochs=epochs, 58 | batch_size=batch_size) 59 | return model 60 | 61 | 62 | def export_model(model, base_path="./exported_models/"): 63 | path = os.path.join(base_path, str(int(time.time()))) 64 | tf.saved_model.save(model, path) 65 | 66 | 67 | if __name__ == '__main__': 68 | fpath = "dataset.csv" 69 | model = train(fpath) 70 | export_model(model) -------------------------------------------------------------------------------- /example_tf_serving_configurations/model_config_list.txt: -------------------------------------------------------------------------------- 1 | model_config_list { 2 | config { 3 | name: 'my_model' 4 | base_path: '/models/my_model/' 5 | model_platform: 'tensorflow' 6 | model_version_policy { 7 | specific { 8 | versions: 1571697725 9 | versions: 1571698198 10 | } 11 | } 12 | } 13 | } 14 | -------------------------------------------------------------------------------- /example_tf_serving_configurations/model_config_list_with_labels.txt: -------------------------------------------------------------------------------- 1 | model_config_list { 2 | config { 3 | name: 'my_model' 4 | base_path: '/models/my_model/' 5 | model_platform: 'tensorflow' 6 | model_version_policy { 7 | specific { 8 | versions: 1571698198 9 | versions: 1571697725 10 | } 11 | } 12 | version_labels { 13 | key: 'stable' 14 | value: 1571698198 15 | } 16 | version_labels { 17 | key: 'canary' 18 | value: 1571697725 19 | } 20 | } 21 | } -------------------------------------------------------------------------------- /example_tf_serving_configurations/monitoring_config_file.txt: -------------------------------------------------------------------------------- 1 | prometheus_config { 2 | enable: true, 3 | path: "/monitoring/prometheus/metrics" 4 | } -------------------------------------------------------------------------------- /example_tf_serving_configurations/prometheus.yml: -------------------------------------------------------------------------------- 1 | global: 2 | scrape_interval: 15s 3 | evaluation_interval: 15s 4 | external_labels: 5 | monitor: 'tf-serving-monitor' 6 | scrape_configs: 7 | - job_name: 'prometheus' 8 | scrape_interval: 5s 9 | metrics_path: /monitoring/prometheus/metrics 10 | static_configs: 11 | - targets: ['host.docker.internal:8501'] -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | absl-py==0.8.1 2 | appnope==0.1.0 3 | astor==0.8.0 4 | attrs==19.3.0 5 | backcall==0.1.0 6 | bleach==3.1.0 7 | certifi==2019.9.11 8 | chardet==3.0.4 9 | decorator==4.4.0 10 | defusedxml==0.6.0 11 | entrypoints==0.3 12 | gast==0.2.2 13 | google-pasta==0.1.7 14 | grpcio==1.24.1 15 | h5py==2.10.0 16 | idna==2.8 17 | importlib-metadata==0.23 18 | ipykernel==5.1.3 19 | ipython==7.8.0 20 | ipython-genutils==0.2.0 21 | ipywidgets==7.5.1 22 | jedi==0.15.1 23 | Jinja2==2.10.3 24 | jsonschema==3.1.1 25 | jupyter==1.0.0 26 | jupyter-client==5.3.4 27 | jupyter-console==6.0.0 28 | jupyter-core==4.6.0 29 | Keras-Applications==1.0.8 30 | Keras-Preprocessing==1.1.0 31 | Markdown==3.1.1 32 | MarkupSafe==1.1.1 33 | mistune==0.8.4 34 | more-itertools==7.2.0 35 | nbconvert==5.6.0 36 | nbformat==4.4.0 37 | notebook==6.0.1 38 | numpy==1.17.3 39 | opt-einsum==3.1.0 40 | pandas==0.25.2 41 | pandocfilters==1.4.2 42 | parso==0.5.1 43 | pexpect==4.7.0 44 | pickleshare==0.7.5 45 | prometheus-client==0.7.1 46 | prompt-toolkit==2.0.10 47 | protobuf==3.10.0 48 | ptyprocess==0.6.0 49 | Pygments==2.4.2 50 | pyrsistent==0.15.4 51 | python-dateutil==2.8.0 52 | pytz==2019.3 53 | pyzmq==18.1.0 54 | qtconsole==4.5.5 55 | requests==2.22.0 56 | Send2Trash==1.5.0 57 | six==1.12.0 58 | tensorboard==2.0.0 59 | tensorflow==2.0.0 60 | tensorflow-estimator==2.0.1 61 | tensorflow-hub==0.6.0 62 | termcolor==1.1.0 63 | terminado==0.8.2 64 | testpath==0.4.2 65 | tornado==6.0.3 66 | traitlets==4.3.3 67 | urllib3==1.25.6 68 | wcwidth==0.1.7 69 | webencodings==0.5.1 70 | Werkzeug==0.16.0 71 | widgetsnbextension==3.5.1 72 | wrapt==1.11.2 73 | zipp==0.6.0 74 | --------------------------------------------------------------------------------