├── images ├── dock.jpg ├── gold.jpg ├── onnx.png ├── pfa.png ├── sas.png ├── uber.png ├── docker.png ├── mlflow.png ├── PMML_Logo.png ├── kubeflow.png ├── pfa-doc-1.png ├── pfa-doc-2.png ├── pfa-line.png ├── pmml_example.png └── squeezenet.png ├── binder ├── postBuild ├── start ├── environment.yml └── jupyterlab-workspace.json ├── README.md ├── .gitignore ├── 02-OSS-Amalgamation-and-Single-Product.ipynb ├── 03-PMML.ipynb ├── 04-PFA.ipynb ├── 05-ONNX.ipynb ├── 06-Platforms-Wrapup.ipynb ├── 01-Intro.ipynb └── LICENSE /images/dock.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/adbreind/open-standard-models-2019/master/images/dock.jpg -------------------------------------------------------------------------------- /images/gold.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/adbreind/open-standard-models-2019/master/images/gold.jpg -------------------------------------------------------------------------------- /images/onnx.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/adbreind/open-standard-models-2019/master/images/onnx.png -------------------------------------------------------------------------------- /images/pfa.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/adbreind/open-standard-models-2019/master/images/pfa.png -------------------------------------------------------------------------------- /images/sas.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/adbreind/open-standard-models-2019/master/images/sas.png -------------------------------------------------------------------------------- /images/uber.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/adbreind/open-standard-models-2019/master/images/uber.png -------------------------------------------------------------------------------- /images/docker.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/adbreind/open-standard-models-2019/master/images/docker.png -------------------------------------------------------------------------------- /images/mlflow.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/adbreind/open-standard-models-2019/master/images/mlflow.png -------------------------------------------------------------------------------- /images/PMML_Logo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/adbreind/open-standard-models-2019/master/images/PMML_Logo.png -------------------------------------------------------------------------------- /images/kubeflow.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/adbreind/open-standard-models-2019/master/images/kubeflow.png -------------------------------------------------------------------------------- /images/pfa-doc-1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/adbreind/open-standard-models-2019/master/images/pfa-doc-1.png -------------------------------------------------------------------------------- /images/pfa-doc-2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/adbreind/open-standard-models-2019/master/images/pfa-doc-2.png -------------------------------------------------------------------------------- /images/pfa-line.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/adbreind/open-standard-models-2019/master/images/pfa-line.png -------------------------------------------------------------------------------- /images/pmml_example.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/adbreind/open-standard-models-2019/master/images/pmml_example.png -------------------------------------------------------------------------------- /images/squeezenet.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/adbreind/open-standard-models-2019/master/images/squeezenet.png -------------------------------------------------------------------------------- /binder/postBuild: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Install the JupyterLab extenstions, e.g.: 4 | #jupyter labextension install dask-labextension 5 | -------------------------------------------------------------------------------- /binder/start: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Import the workspace 4 | jupyter lab workspaces import binder/jupyterlab-workspace.json 5 | 6 | exec "$@" 7 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # open-standard-models-2019 2 | 3 | [![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/adbreind/open-standard-models-2019/master) 4 | -------------------------------------------------------------------------------- /binder/environment.yml: -------------------------------------------------------------------------------- 1 | name: open-standard-models 2 | channels: 3 | - conda-forge 4 | dependencies: 5 | - python=3.7 6 | - jupyterlab 7 | - nomkl 8 | - numpy 9 | - pip 10 | - pandas 11 | - pyarrow 12 | - python-graphviz 13 | - seaborn 14 | - scikit-learn 15 | - matplotlib 16 | - h5py 17 | - requests 18 | - pytables 19 | - pip: 20 | - SKompiler[full] 21 | - nyoka 22 | - pyyaml 23 | - onnxmltools 24 | - onnxruntime 25 | -------------------------------------------------------------------------------- /binder/jupyterlab-workspace.json: -------------------------------------------------------------------------------- 1 | {"data":{"layout-restorer:data":{"main":{"dock":{"type":"tab-area","currentIndex":0,"widgets":["notebook:01-Intro.ipynb"]},"mode":"multiple-document","current":"notebook:01-Intro.ipynb"},"left":{"collapsed":false,"current":"filebrowser","widgets":["filebrowser","running-sessions","command-palette","tab-manager"]},"right":{"collapsed":true,"widgets":[]}},"notebook:01-Intro.ipynb":{"data":{"path":"01-Intro.ipynb","factory":"Notebook"}},"file-browser-filebrowser:cwd":{"path":""}},"metadata":{"id":"/lab"}} -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | *$py.class 5 | 6 | # C extensions 7 | *.so 8 | 9 | # Distribution / packaging 10 | .Python 11 | build/ 12 | develop-eggs/ 13 | dist/ 14 | downloads/ 15 | eggs/ 16 | .eggs/ 17 | lib/ 18 | lib64/ 19 | parts/ 20 | sdist/ 21 | var/ 22 | wheels/ 23 | *.egg-info/ 24 | .installed.cfg 25 | *.egg 26 | MANIFEST 27 | 28 | # PyInstaller 29 | # Usually these files are written by a python script from a template 30 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 31 | *.manifest 32 | *.spec 33 | 34 | # Installer logs 35 | pip-log.txt 36 | pip-delete-this-directory.txt 37 | 38 | # Unit test / coverage reports 39 | htmlcov/ 40 | .tox/ 41 | .coverage 42 | .coverage.* 43 | .cache 44 | nosetests.xml 45 | coverage.xml 46 | *.cover 47 | .hypothesis/ 48 | .pytest_cache/ 49 | 50 | # Translations 51 | *.mo 52 | *.pot 53 | 54 | # Django stuff: 55 | *.log 56 | local_settings.py 57 | db.sqlite3 58 | 59 | # Flask stuff: 60 | instance/ 61 | .webassets-cache 62 | 63 | # Scrapy stuff: 64 | .scrapy 65 | 66 | # Sphinx documentation 67 | docs/_build/ 68 | 69 | # PyBuilder 70 | target/ 71 | 72 | # Jupyter Notebook 73 | .ipynb_checkpoints 74 | 75 | # pyenv 76 | .python-version 77 | 78 | # celery beat schedule file 79 | celerybeat-schedule 80 | 81 | # SageMath parsed files 82 | *.sage.py 83 | 84 | # Environments 85 | .env 86 | .venv 87 | env/ 88 | venv/ 89 | ENV/ 90 | env.bak/ 91 | venv.bak/ 92 | 93 | # Spyder project settings 94 | .spyderproject 95 | .spyproject 96 | 97 | # Rope project settings 98 | .ropeproject 99 | 100 | # mkdocs documentation 101 | /site 102 | 103 | # mypy 104 | .mypy_cache/ 105 | -------------------------------------------------------------------------------- /02-OSS-Amalgamation-and-Single-Product.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Open-Source Amalgamation and Single-Product Formats\n", 8 | "\n", 9 | "## Amalgamation\n", 10 | "\n", 11 | "The simplest form of export is \"amalgamation\" ... in which case the model and all necessary code to run are emitted as one big chunk.\n", 12 | "\n", 13 | "In some cases, it's a single source code file that can be compiled on nearly any platform as a standalone program.\n", 14 | " * Classic amalgamation: MXNet + model code https://mxnet.apache.org/api/faq/smart_device\n", 15 | "\n", 16 | "In other cases, it's a chunk of consumable IR code that can be consumed in a common runtime:\n", 17 | " * H2O POJO export https://github.com/h2oai/h2o-3/blob/master/h2o-docs/src/product/productionizing.rst#pojo-quick-start\n", 18 | " \n", 19 | "And sometimes ... it's a coder implementing a model by hand and compiling it! (For simple, popular models, like linear/logistic regression, it's pretty easy once you have the model params.)" 20 | ] 21 | }, 22 | { 23 | "cell_type": "markdown", 24 | "metadata": {}, 25 | "source": [ 26 | "### What Does this Look Like? Let's Try It...\n", 27 | "\n", 28 | "First, we need a model. So we'll train a quick linear regression on R's Diamonds dataset:" 29 | ] 30 | }, 31 | { 32 | "cell_type": "code", 33 | "execution_count": null, 34 | "metadata": {}, 35 | "outputs": [], 36 | "source": [ 37 | "import pandas as pd\n", 38 | "\n", 39 | "data = pd.read_csv('data/diamonds.csv')\n", 40 | "\n", 41 | "X = data.carat\n", 42 | "y = data.price" 43 | ] 44 | }, 45 | { 46 | "cell_type": "code", 47 | "execution_count": null, 48 | "metadata": {}, 49 | "outputs": [], 50 | "source": [ 51 | "from sklearn.linear_model import LinearRegression\n", 52 | "\n", 53 | "model = LinearRegression().fit(X.values.reshape(-1,1), y)" 54 | ] 55 | }, 56 | { 57 | "cell_type": "markdown", 58 | "metadata": {}, 59 | "source": [ 60 | "This is a basic model for predicting price (in dollars) from weight (in carats).\n", 61 | "\n", 62 | "The params are:" 63 | ] 64 | }, 65 | { 66 | "cell_type": "code", 67 | "execution_count": null, 68 | "metadata": {}, 69 | "outputs": [], 70 | "source": [ 71 | "model.coef_, model.intercept_" 72 | ] 73 | }, 74 | { 75 | "cell_type": "markdown", 76 | "metadata": {}, 77 | "source": [ 78 | "We can create a standalone (\"amalgamated\") version of this model using [SKompiler](https://pypi.org/project/SKompiler/)" 79 | ] 80 | }, 81 | { 82 | "cell_type": "code", 83 | "execution_count": null, 84 | "metadata": {}, 85 | "outputs": [], 86 | "source": [ 87 | "from skompiler import skompile\n", 88 | "\n", 89 | "expr = skompile(model.predict)\n", 90 | "expr.to('sqlalchemy/postgresql')" 91 | ] 92 | }, 93 | { 94 | "cell_type": "code", 95 | "execution_count": null, 96 | "metadata": {}, 97 | "outputs": [], 98 | "source": [ 99 | "expr.to('sympy/c')" 100 | ] 101 | }, 102 | { 103 | "cell_type": "markdown", 104 | "metadata": {}, 105 | "source": [ 106 | "Here's a quick example with a decision tree. First, we'll create the model." 107 | ] 108 | }, 109 | { 110 | "cell_type": "code", 111 | "execution_count": null, 112 | "metadata": {}, 113 | "outputs": [], 114 | "source": [ 115 | "from sklearn.tree import DecisionTreeRegressor\n", 116 | "\n", 117 | "model_tree = DecisionTreeRegressor(max_depth=4).fit(X.values.reshape(-1,1), y)" 118 | ] 119 | }, 120 | { 121 | "cell_type": "code", 122 | "execution_count": null, 123 | "metadata": {}, 124 | "outputs": [], 125 | "source": [ 126 | "for line in skompile(model_tree.predict).to('sympy/c').split('\\n'):\n", 127 | " print(line)" 128 | ] 129 | }, 130 | { 131 | "cell_type": "markdown", 132 | "metadata": {}, 133 | "source": [ 134 | "#### Pros and Cons: Amalgamation\n", 135 | "\n", 136 | "Pros:\n", 137 | "* Easy-to-understand concept\n", 138 | "* Fairly portable\n", 139 | "* Can be compact and performant\n", 140 | " * May be a good choice for extremely constrained embedded environments\n", 141 | "\n", 142 | "Cons:\n", 143 | "* Not interoperable with other high-level environments\n", 144 | "* Violates separation of code (logic) from data (parameters)\n", 145 | "* May not fit in well with enterprise manageability and operations needs" 146 | ] 147 | }, 148 | { 149 | "cell_type": "markdown", 150 | "metadata": {}, 151 | "source": [ 152 | "## Open-Source Single-Product Format(s)\n", 153 | "\n", 154 | "I.e., a format which serves a specific product ecosystem, but is not intended to interoperate with other systems nor serve as a \"standard\"\n", 155 | "\n", 156 | "*Examples:*\n", 157 | "\n", 158 | "__SparkML + MLeap__\n", 159 | " * MLeap supports Spark, some scikit-learn models and some TensorFlow models\n", 160 | " * Represents models in a \"MLeap Bundle\"\n", 161 | " * MLeap runtime is a JAR that can run in any Java application (or by with a lightweight scoring wrapper provided by MLeap)\n", 162 | "\n", 163 | "__TensorFlow + TensorFlow Serving__\n", 164 | " * TensorFlow models (created directly with TensorFlow or with Keras) serialize to a TF-specific protocol buffer representation\n", 165 | " * TensorFlow Serving loads the latest version of a model\n", 166 | " * TF Serving exposes a gRPC service and a REST endpoint\n", 167 | " \n", 168 | "Within the intended ecosystem, we can easily export and use a model, but we don't get portability across tools/ecosystems." 169 | ] 170 | }, 171 | { 172 | "cell_type": "code", 173 | "execution_count": null, 174 | "metadata": {}, 175 | "outputs": [], 176 | "source": [] 177 | } 178 | ], 179 | "metadata": { 180 | "kernelspec": { 181 | "display_name": "Python 3", 182 | "language": "python", 183 | "name": "python3" 184 | }, 185 | "language_info": { 186 | "codemirror_mode": { 187 | "name": "ipython", 188 | "version": 3 189 | }, 190 | "file_extension": ".py", 191 | "mimetype": "text/x-python", 192 | "name": "python", 193 | "nbconvert_exporter": "python", 194 | "pygments_lexer": "ipython3", 195 | "version": "3.7.5" 196 | } 197 | }, 198 | "nbformat": 4, 199 | "nbformat_minor": 4 200 | } 201 | -------------------------------------------------------------------------------- /03-PMML.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Existing Standard Format: PMML\n", 8 | "\n", 9 | "" 10 | ] 11 | }, 12 | { 13 | "cell_type": "markdown", 14 | "metadata": {}, 15 | "source": [ 16 | "## What is PMML?\n", 17 | "\n", 18 | "PMML is a Data Mining Group (http://dmg.org/dmg-members.html) standard that has existed and evolved for over 20 years, and is used widely throughout the world. \n", 19 | "\n", 20 | "Formally, it's a XML dialect that describes a model and/or pipeline." 21 | ] 22 | }, 23 | { 24 | "cell_type": "markdown", 25 | "metadata": {}, 26 | "source": [ 27 | "## Example\n", 28 | "\n", 29 | "Here is an example of a logistic regression classifier trained using R on the Iris dataset:\n", 30 | "\n", 31 | "(http://dmg.org/pmml/pmml_examples/rattle_pmml_examples/IrisMultinomReg.xml)\n", 32 | "\n", 33 | "" 34 | ] 35 | }, 36 | { 37 | "cell_type": "markdown", 38 | "metadata": {}, 39 | "source": [ 40 | "## Where do we get a PMML model?\n", 41 | "\n", 42 | "A partial list of products supporting PMML is at http://dmg.org/pmml/products.html\n", 43 | "\n", 44 | "Focusing on the *producing PMML* side, we can see there are a lot of products that can create PMML, even if most of them are commercial or have effectively commercial licensing schemes (e.g. JPMML).\n", 45 | "\n", 46 | "In the open-source world (again, excluding AGPL code like JPMML), we have\n", 47 | "* R -- strongest open-source export support\n", 48 | "* Spark -- very limited support: the listed models are only supported under the *old/deprecated* RDD MLlib API\n", 49 | " * There is work in progress to add PMML export to the new API but it has just begun and may not make progress\n", 50 | "* Python \n", 51 | " * Historically, aside from the wrapper around the above-mentioned JPMML, there is little support\n", 52 | " * e.g., https://pypi.org/project/scikit2pmml/ from https://github.com/vaclavcadek/scikit2pmml\n", 53 | " * Recently, SoftwareAG has created Nyoka, a permissively licensed Python-to-PMML export tool\n", 54 | " * Source https://github.com/nyoka-pmml/nyoka\n", 55 | " * Docs at https://nyoka-pmml.github.io/nyoka/index.html\n", 56 | " \n", 57 | "It is important to note that\n", 58 | "* although there are plenty of commercial products with at least some PMML support\n", 59 | "* and although large enterprises can (and for support/legal reasons prefer to) pay for a product\n", 60 | "* the lack of openness and community is leaving commercial-only ML tooling far behind\n", 61 | " * e.g., all of the top deep learning tools are FOSS\n", 62 | " * this means most of the performance-focused work is tied to the FOSS tools\n", 63 | " * scaling is owned by FOSS (kubeflow, Horovod, etc.)\n", 64 | " \n", 65 | "### Let's create a PMML model" 66 | ] 67 | }, 68 | { 69 | "cell_type": "code", 70 | "execution_count": null, 71 | "metadata": {}, 72 | "outputs": [], 73 | "source": [ 74 | "import pandas as pd\n", 75 | "from sklearn.pipeline import Pipeline\n", 76 | "from sklearn.linear_model import LinearRegression\n", 77 | "\n", 78 | "data = pd.read_csv('data/diamonds.csv')\n", 79 | "X = data.carat\n", 80 | "y = data.price\n", 81 | "pipeline_model = Pipeline([('lin_reg', LinearRegression())]).fit(X.values.reshape(-1,1), y)" 82 | ] 83 | }, 84 | { 85 | "cell_type": "code", 86 | "execution_count": null, 87 | "metadata": {}, 88 | "outputs": [], 89 | "source": [ 90 | "from nyoka import skl_to_pmml\n", 91 | "\n", 92 | "skl_to_pmml(pipeline_model, ['carat'], 'price', 'diamonds.pmml')" 93 | ] 94 | }, 95 | { 96 | "cell_type": "code", 97 | "execution_count": null, 98 | "metadata": {}, 99 | "outputs": [], 100 | "source": [ 101 | "! cat diamonds.pmml" 102 | ] 103 | }, 104 | { 105 | "cell_type": "markdown", 106 | "metadata": {}, 107 | "source": [ 108 | "### How do we run a PMML model?\n", 109 | "\n", 110 | "Enterprise-grade permissive OSS support for running PMML models is effectively nonexistent, so we need to architect in tandem with business decisions around a vendor's analytics server product. These business decisions will go beyond the licensing and support, because they will affect all of our enterprise architectures: hardware, network, software, management/monitoring/operations, reliability/continuity, compliance etc.\n", 111 | "\n", 112 | "The most comprehensive set of tools around PMML is almost certainly JPMML\n", 113 | "\n", 114 | "#### JPMML\n", 115 | "\n", 116 | "JPMML (https://github.com/jpmml) is a set of AGPL OSS projects that \n", 117 | "* form the de facto Java implementation of PMML (but also has tools for Python, etc.)\n", 118 | "* offer interop with key FOSS tools like Apache Spark, R, Scikit-learn, XGBoost, TensorFlow, etc.\n", 119 | "* provide easy scoring in your own apps, or using a \"scoring wrapper\" or hosted in the cloud\n", 120 | "* is maintained and licensed in connection with https://openscoring.io/ \n", 121 | "* *note: there is an older, abandoned, version of JPMML under a more friendly Apache 2.0 license*\n", 122 | " * this older version has many features and might be suitable for some organizations with a higher risk/ownership appetite\n", 123 | " * https://github.com/jpmml/jpmml" 124 | ] 125 | }, 126 | { 127 | "cell_type": "markdown", 128 | "metadata": {}, 129 | "source": [ 130 | "#### Pros and Cons: PMML\n", 131 | "\n", 132 | "Pros:\n", 133 | "* In wide use / well-accepted / large community\n", 134 | "* Core XML dialect can be human readable\n", 135 | "* Models can be processed/managed by text-based tools (VCS/CMS/etc.)\n", 136 | "* Covers the majority of modeling cases companies use today\n", 137 | "* *Formally* interoperable (reading/writing the container file format)\n", 138 | "\n", 139 | "Cons:\n", 140 | "* Support for production of models in the open-source world is spotty\n", 141 | "* Support for consuming models in the OSS is sparse/minimal\n", 142 | "* Importance of modern open-source tooling has been dragging PMML down\n", 143 | "* Some modern model types and pipelines are not supported, or not supported efficiently/compactly\n", 144 | "* *Semantic* interop is limited\n", 145 | "\n", 146 | "In practice, PMML -- even with commercial/enterprise, supported products -- is more like USB C than USB 3. \n", 147 | "\n", 148 | "I.e., like USB C, it's very versatile in theory, and the plug always fits, but that tells you little or nothing about whether the two devices connected can have any conversation, let alone the specific conversation you need them to have.\n", 149 | "\n", 150 | "Despite its imperfections, it has many advantages over single-product formats, so we often use it even if it cannot fulfill a promise of being the \"universal\" tool." 151 | ] 152 | }, 153 | { 154 | "cell_type": "code", 155 | "execution_count": null, 156 | "metadata": {}, 157 | "outputs": [], 158 | "source": [] 159 | } 160 | ], 161 | "metadata": { 162 | "kernelspec": { 163 | "display_name": "Python 3", 164 | "language": "python", 165 | "name": "python3" 166 | }, 167 | "language_info": { 168 | "codemirror_mode": { 169 | "name": "ipython", 170 | "version": 3 171 | }, 172 | "file_extension": ".py", 173 | "mimetype": "text/x-python", 174 | "name": "python", 175 | "nbconvert_exporter": "python", 176 | "pygments_lexer": "ipython3", 177 | "version": "3.7.5" 178 | } 179 | }, 180 | "nbformat": 4, 181 | "nbformat_minor": 4 182 | } 183 | -------------------------------------------------------------------------------- /04-PFA.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Next-Gen Standard Format: PFA\n", 8 | "\n", 9 | "" 10 | ] 11 | }, 12 | { 13 | "cell_type": "markdown", 14 | "metadata": {}, 15 | "source": [ 16 | "## PFA (Portable Format for Analytics) is a Modern Replacement for PMML\n", 17 | "\n", 18 | "__\"As data analyses mature, they must be hardened — they must have fewer dependencies, a more maintainable structure, and they must be robust against errors.\" - DMG__\n", 19 | "\n", 20 | "PFA, created in 2015, is intended to improve upon PMML.\n", 21 | "\n", 22 | "From http://dmg.org/pfa/docs/motivation/:\n", 23 | "\n", 24 | "*Tools such as Hadoop and Storm provide automated data pipelines, separating the data flow from the functions that are performed on data (mappers and reducers in Hadoop, spouts and bolts in Storm). Ordinarily, these functions are written in code that has access to the pipeline internals, the host operating system, the remote filesystem, the network, etc. However, all they should do is math.*\n", 25 | "\n", 26 | "*PFA completes the abstraction by encapsulating these functions as PFA documents. From the point of view of the pipeline system, the documents are configuration files that may be loaded or replaced independently of the pipeline code.*\n", 27 | "\n", 28 | "*This separation of concerns allows the data analysis to evolve independently of the pipeline. Since scoring engines written in PFA are not capable of accessing or manipulating their environment, they cannot jeopardize the production system. Data analysts can focus on the mathematical correctness of their algorithms and security reviews are only needed when the pipeline itself changes.*\n", 29 | "\n", 30 | "*This decoupling is important because statistical models usually change more quickly than pipeline frameworks. Model details are often tweaked in response to discoveries about the data and models frequently need to be refreshed with new training samples.*" 31 | ] 32 | }, 33 | { 34 | "cell_type": "markdown", 35 | "metadata": {}, 36 | "source": [ 37 | "\n", 38 | "\n", 39 | "(summarized from DMG)\n", 40 | "\n", 41 | "### Overview of PFA capabilities\n", 42 | "\n", 43 | "PFA flexibility:\n", 44 | "* Control structures, such as conditionals, loops, and user-defined functions\n", 45 | "* Entirely expressed within JSON, and can therefore be easily generated and manipulated by other programs\n", 46 | "* Fine-grained function library supporting extensibility callbacks\n", 47 | "\n", 48 | "The following contribute to PFA’s safety:\n", 49 | "* Strict numerical compatibility: the same PFA document and the same input results in the same output, regardless of platform.\n", 50 | "* Spec only defines functions that transform data. I/O is all controlled by the host system.\n", 51 | "* Type system that can be statically checked. ... This system has a type-safe null and PFA only performs type-safe casting, which ensure that missing data never cause run-time errors.\n", 52 | "* The callbacks that generalize PFA’s statistical models are not first-class functions\n", 53 | " * The set of functions that a PFA document might call can be predicted before it runs\n", 54 | " * A PFA host may choose to only allow certain functions." 55 | ] 56 | }, 57 | { 58 | "cell_type": "markdown", 59 | "metadata": {}, 60 | "source": [ 61 | "__Example__\n", 62 | "\n", 63 | "Here are some data records:\n", 64 | "\n", 65 | "\n", 66 | "\n", 67 | "And a PFA document which returns the square-root of the sum of the squares of a record's x, y, and z values:\n", 68 | "\n", 69 | "" 70 | ] 71 | }, 72 | { 73 | "cell_type": "markdown", 74 | "metadata": {}, 75 | "source": [ 76 | "The above example -- along with numerous other tutorials -- can be viewed, *modified*, and run live online at http://dmg.org/pfa/docs/tutorial2/ and other dmg.org pages.\n", 77 | "\n", 78 | "Although it may not be obvious from this small example, PFA is effectively a programming language, albeit a restricted one, and as such can express complex transformations and aggregations of data. A compliant PFA scoring system must implement the full spec properly: http://dmg.org/pfa/docs/library/\n", 79 | "\n", 80 | "The PFA document is a serialized representation or description of a scoring engine, of which one or more instances can be created by a runtime.\n", 81 | "\n", 82 | "The Avro, JSON, and YAML representations are interchangeable, with the JSON and YAML working better for humans and text tools, while the Avro is better suited to performance, type checking, etc. That said, it is still intended to be a machine-generated and machine-consumed document." 83 | ] 84 | }, 85 | { 86 | "cell_type": "markdown", 87 | "metadata": {}, 88 | "source": [ 89 | "### Let's make a PFA version of our Diamonds model" 90 | ] 91 | }, 92 | { 93 | "cell_type": "code", 94 | "execution_count": null, 95 | "metadata": {}, 96 | "outputs": [], 97 | "source": [ 98 | "import pandas as pd\n", 99 | "from sklearn.linear_model import LinearRegression\n", 100 | "\n", 101 | "data = pd.read_csv('data/diamonds.csv')\n", 102 | "X = data.carat\n", 103 | "y = data.price\n", 104 | "model = LinearRegression().fit(X.values.reshape(-1,1), y)" 105 | ] 106 | }, 107 | { 108 | "cell_type": "code", 109 | "execution_count": null, 110 | "metadata": {}, 111 | "outputs": [], 112 | "source": [ 113 | "from skompiler import skompile\n", 114 | "\n", 115 | "expr = skompile(model.predict)\n", 116 | "\n", 117 | "for line in expr.to('pfa/yaml').split('\\n'):\n", 118 | " print(line)" 119 | ] 120 | }, 121 | { 122 | "cell_type": "markdown", 123 | "metadata": {}, 124 | "source": [ 125 | "#### Try Scoring Some Records Online\n", 126 | "\n", 127 | "1. Go to: http://dmg.org/pfa/docs/tutorial1/ (in another browser tab)\n", 128 | "2. Copy the PFA engine document above, and paste it in the block marked \"PFA Document (YAML)\"\n", 129 | "3. Try scoring with the \"Run\" button\n", 130 | "4. You'll notice that you get an Avro-related type error\n", 131 | "5. Format your scoring records, one per line, in JSON, using the input name (\"x\") as the name for the carat weight.\n", 132 | " * For example, `{ \"x\" : [1.0] }` to represent a 1-carat diamond" 133 | ] 134 | }, 135 | { 136 | "cell_type": "markdown", 137 | "metadata": {}, 138 | "source": [ 139 | "Hadrian (https://github.com/opendatagroup/hadrian) is a permissive OSS implementation of a compliant PFA runtime. You can use the source, or use the pre-built `.war` file to create a JVM-based scoring server.\n", 140 | "\n", 141 | "If you want to build your own Python scoring server for PFA, see the Titus installation instructions here: https://github.com/opendatagroup/hadrian/wiki/Installation#case-4-you-want-to-install-titus-in-python" 142 | ] 143 | }, 144 | { 145 | "cell_type": "markdown", 146 | "metadata": {}, 147 | "source": [ 148 | "#### Pros and Cons: PFA\n", 149 | "\n", 150 | "Pros:\n", 151 | "* Flexible, extensible\n", 152 | "* Permissive open-source scoring engine and some OSS export support (SKompile, Aardpfark)\n", 153 | "* Addresses many of the shortcomings of PMML\n", 154 | "\n", 155 | "Cons:\n", 156 | "* Limited OSS export support\n", 157 | "* Timing appears to have been \"unlucky\"\n", 158 | " * Perhaps the project is too new?\n", 159 | " * Or other, more \"modern\" open initiatives have overtaken the DMG and swept this approach away\n", 160 | "* In any case, community has not embraced PFA as of late 2019" 161 | ] 162 | }, 163 | { 164 | "cell_type": "code", 165 | "execution_count": null, 166 | "metadata": {}, 167 | "outputs": [], 168 | "source": [] 169 | } 170 | ], 171 | "metadata": { 172 | "kernelspec": { 173 | "display_name": "Python 3", 174 | "language": "python", 175 | "name": "python3" 176 | }, 177 | "language_info": { 178 | "codemirror_mode": { 179 | "name": "ipython", 180 | "version": 3 181 | }, 182 | "file_extension": ".py", 183 | "mimetype": "text/x-python", 184 | "name": "python", 185 | "nbconvert_exporter": "python", 186 | "pygments_lexer": "ipython3", 187 | "version": "3.7.4" 188 | } 189 | }, 190 | "nbformat": 4, 191 | "nbformat_minor": 4 192 | } 193 | -------------------------------------------------------------------------------- /05-ONNX.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "# ONNX (Open Neural Network eXchange)\n", 15 | "\n", 16 | "Originally created by Facebook and Microsoft as an industry collaboration for import/export of neural networks\n", 17 | "* ONNX has grown to include support for \"traditional\" ML models\n", 18 | "* interop with many software libraries\n", 19 | "* has both software (CPU, optional GPU accelerated) and hardware (Intel, Qualcomm, etc.) runtimes.\n", 20 | "\n", 21 | "https://onnx.ai/\n", 22 | "\n", 23 | "* DAG-based model\n", 24 | "* Built-in operators, data types\n", 25 | "* Extensible -- e.g., ONNX-ML\n", 26 | "* Goal is to allow tools to share a single model format\n", 27 | "\n", 28 | "*Of the \"standard/open\" formats, ONNX clearly has the most momentum in the past year or two.*\n", 29 | "\n", 30 | "## Viewing a Model\n", 31 | "\n", 32 | "ONNX models are not directly (as raw data) human-readable, but, as they represent a graph, can easily be converted into textual or graphical representations.\n", 33 | "\n", 34 | "Here is a snippet of the [SqueezeNet](https://arxiv.org/abs/1602.07360) image-recognition model, as rendered in the ONNX visualization tutorial at https://github.com/onnx/tutorials/blob/master/tutorials/VisualizingAModel.md. \n", 35 | "\n", 36 | "> The ONNX codebase comes with the visualization converter used in this example -- it's a simple script currently located at https://github.com/onnx/onnx/blob/master/onnx/tools/net_drawer.py\n", 37 | "\n", 38 | "\n", 39 | "\n", 40 | "### Let's Build a Model and Convert it to ONNX" 41 | ] 42 | }, 43 | { 44 | "cell_type": "code", 45 | "execution_count": null, 46 | "metadata": {}, 47 | "outputs": [], 48 | "source": [ 49 | "import pandas as pd\n", 50 | "from sklearn.linear_model import LinearRegression\n", 51 | "\n", 52 | "data = pd.read_csv('data/diamonds.csv')\n", 53 | "X = data.carat\n", 54 | "y = data.price\n", 55 | "model = LinearRegression().fit(X.values.reshape(-1,1), y)" 56 | ] 57 | }, 58 | { 59 | "cell_type": "markdown", 60 | "metadata": {}, 61 | "source": [ 62 | "ONNX can be generated from many modeling tools. A partial list is here: https://github.com/onnx/tutorials#converting-to-onnx-format\n", 63 | "\n", 64 | "Microsoft has contributed a lot of resources toward open-source ONNX capabilities, including, in early 2019, support for Apache Spark ML Pipelines: https://github.com/onnx/onnxmltools/blob/master/onnxmltools/convert/sparkml/README.md\n", 65 | "\n", 66 | "__Convert to ONNX__\n", 67 | "\n", 68 | "Note that we can print a string representation of the converted graph." 69 | ] 70 | }, 71 | { 72 | "cell_type": "code", 73 | "execution_count": null, 74 | "metadata": {}, 75 | "outputs": [], 76 | "source": [ 77 | "from skl2onnx import convert_sklearn\n", 78 | "from skl2onnx.common.data_types import FloatTensorType\n", 79 | "\n", 80 | "initial_type = [('float_input', FloatTensorType([1, 1]))]\n", 81 | "onx = convert_sklearn(model, initial_types=initial_type)\n", 82 | "print(onx)" 83 | ] 84 | }, 85 | { 86 | "cell_type": "markdown", 87 | "metadata": {}, 88 | "source": [ 89 | "Let's save it as a file:" 90 | ] 91 | }, 92 | { 93 | "cell_type": "code", 94 | "execution_count": null, 95 | "metadata": {}, 96 | "outputs": [], 97 | "source": [ 98 | "with open(\"diamonds.onnx\", \"wb\") as f:\n", 99 | " f.write(onx.SerializeToString())" 100 | ] 101 | }, 102 | { 103 | "cell_type": "markdown", 104 | "metadata": {}, 105 | "source": [ 106 | "The file itself is binary:" 107 | ] 108 | }, 109 | { 110 | "cell_type": "code", 111 | "execution_count": null, 112 | "metadata": {}, 113 | "outputs": [], 114 | "source": [ 115 | "! head diamonds.onnx" 116 | ] 117 | }, 118 | { 119 | "cell_type": "markdown", 120 | "metadata": {}, 121 | "source": [ 122 | "## How Do We Consume ONNX and Make Predictions?\n", 123 | "\n", 124 | "One of the things that makes ONNX a compelling solution in 2019 is the wide industry support not just for model creation, but also for performant model inference.\n", 125 | "\n", 126 | "Here is a partial list of tools that consume ONNX: https://github.com/onnx/tutorials#scoring-onnx-models\n", 127 | "\n", 128 | "Of particular interest for productionizing models are\n", 129 | "* Apple CoreML\n", 130 | "* Microsoft `onnxruntime` and `onnxruntime-gpu` for CPU & GPU-accelerated inference\n", 131 | "* TensorRT for NVIDIA GPU\n", 132 | "* Conversion for Qualcomm Snapdragon hardware: https://developer.qualcomm.com/docs/snpe/model_conv_onnx.html\n", 133 | "\n", 134 | "Today, we'll look at \"regular\" server-based inference with a sample REST server, using `onnxruntime`\n", 135 | "\n", 136 | "#### We'll start by loading the `onnxruntime` library, and seeing how we make predictions" 137 | ] 138 | }, 139 | { 140 | "cell_type": "code", 141 | "execution_count": null, 142 | "metadata": {}, 143 | "outputs": [], 144 | "source": [ 145 | "import onnxruntime as rt\n", 146 | "\n", 147 | "sess = rt.InferenceSession(\"diamonds.onnx\")\n", 148 | "\n", 149 | "print(\"In\", [(i.name, i.type, i.shape) for i in sess.get_inputs()])\n", 150 | " \n", 151 | "print(\"Out\", [(i.name, i.type, i.shape) for i in sess.get_outputs()])" 152 | ] 153 | }, 154 | { 155 | "cell_type": "markdown", 156 | "metadata": {}, 157 | "source": [ 158 | "We've skipped some metadata annotation in the model creation for this quick example -- that's why our input field name is \"float_input\" and the output is called \"variable\"" 159 | ] 160 | }, 161 | { 162 | "cell_type": "code", 163 | "execution_count": null, 164 | "metadata": {}, 165 | "outputs": [], 166 | "source": [ 167 | "import numpy as np\n", 168 | "\n", 169 | "sample_to_score = np.array([[1.0]], dtype=np.float32)" 170 | ] 171 | }, 172 | { 173 | "cell_type": "code", 174 | "execution_count": null, 175 | "metadata": {}, 176 | "outputs": [], 177 | "source": [ 178 | "output = sess.run(['variable'], {'float_input': sample_to_score})\n", 179 | "\n", 180 | "output" 181 | ] 182 | }, 183 | { 184 | "cell_type": "code", 185 | "execution_count": null, 186 | "metadata": {}, 187 | "outputs": [], 188 | "source": [ 189 | "output[0][0][0]" 190 | ] 191 | }, 192 | { 193 | "cell_type": "markdown", 194 | "metadata": {}, 195 | "source": [ 196 | "#### At this point, we can build our service...\n", 197 | "\n", 198 | "Now we are free to choose our service infrastructure of choice.\n", 199 | "\n", 200 | "Moreover, we can containerize that service, so we get back all of the benefits of Docker, Kubernetes, etc.\n", 201 | "\n", 202 | "But this time, we have a minimal serving infrastructure that knows only about the model itself, and loads models in a single, open, industry-standard format." 203 | ] 204 | }, 205 | { 206 | "cell_type": "markdown", 207 | "metadata": {}, 208 | "source": [ 209 | "#### Pros and Cons: ONNX\n", 210 | "* Most major deep learning tools have ONNX support\n", 211 | "* MIT license makes it both OSS and business friendly\n", 212 | "* Seems to achieve its first-order goal of allowing tools interop for neural nets\n", 213 | "* As of 2019, is the closest thing we have to an open, versatile, next-gen format *with wide support*\n", 214 | "* Protobuf format is compact and typesafe\n", 215 | "* Biggest weakness was \"classical\" ML and feature engineering support -- this has now been fixed\n", 216 | "* Microsoft open-sourced (Dec 2018) a high-perf runtime (GPU, CPU, language bindings, etc.) https://azure.microsoft.com/en-us/blog/onnx-runtime-is-now-open-source/\n", 217 | " * Being used as part of Windows ML / Azure ML\n", 218 | " * https://github.com/Microsoft/onnxruntime\n", 219 | "* In Q1-Q2 of 2019, Microsoft added a Spark ML Pipeline exporter to the `onnxmltools` project\n", 220 | " * https://github.com/onnx/onnxmltools\n", 221 | "\n", 222 | "Cons:\n", 223 | "* Wasn't originally intended as a deployment format *per se*\n", 224 | " * Doesn't have a standard or reference runtime\n", 225 | " * Doesn't provide certification or standard around correctness\n", 226 | " * No opinion on security, etc.\n", 227 | "* Protobuf format is not human readable or manageable via text-oriented tooling\n", 228 | " * Though the graph itself can be (e.g., PyTorch export output)" 229 | ] 230 | }, 231 | { 232 | "cell_type": "code", 233 | "execution_count": null, 234 | "metadata": {}, 235 | "outputs": [], 236 | "source": [] 237 | } 238 | ], 239 | "metadata": { 240 | "kernelspec": { 241 | "display_name": "Python 3", 242 | "language": "python", 243 | "name": "python3" 244 | }, 245 | "language_info": { 246 | "codemirror_mode": { 247 | "name": "ipython", 248 | "version": 3 249 | }, 250 | "file_extension": ".py", 251 | "mimetype": "text/x-python", 252 | "name": "python", 253 | "nbconvert_exporter": "python", 254 | "pygments_lexer": "ipython3", 255 | "version": "3.7.5" 256 | }, 257 | "name": "02-ONNX-Models", 258 | "notebookId": 2375086480049026 259 | }, 260 | "nbformat": 4, 261 | "nbformat_minor": 4 262 | } 263 | -------------------------------------------------------------------------------- /06-Platforms-Wrapup.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Enterprise Model Deployment Challenges\n", 8 | "\n", 9 | "Once we have a model and a runtime, we can start to look at the bigger picture.\n", 10 | "\n", 11 | "__First, we have core enterprise deployment concerns:__\n", 12 | "* Easily deploying the model as a service\n", 13 | "* Scaling / performance / SLA\n", 14 | "* Basic management, monitoring, network ops\n", 15 | "\n", 16 | "__Next, we have have higher-level concerns to address:__\n", 17 | "* Model versioning\n", 18 | "* Updating and \"discrete\" online learning\n", 19 | "* Rollout / traffic routing / model performance assessment\n", 20 | "* Rollback\n", 21 | "* A/B testing, multi-armed bandit testing\n", 22 | "* Compliance / decision auditing, illegal biases\n", 23 | "* Model drift\n", 24 | "* Adversarial examples / security / robustness\n", 25 | " * https://ai.googleblog.com/2018/09/introducing-unrestricted-adversarial.html\n", 26 | "* Information leakage\n", 27 | " * https://www.theregister.co.uk/AMP/2018/03/02/secrets_fed_into_ai_models_as_training_data_can_be_stolen\n", 28 | " * https://arxiv.org/abs/1802.08232" 29 | ] 30 | }, 31 | { 32 | "cell_type": "markdown", 33 | "metadata": {}, 34 | "source": [ 35 | "## Deployment Patterns\n", 36 | "\n", 37 | "__The Bad News__\n", 38 | "\n", 39 | "There is no simple, standard, open solution that addresses all of these concerns\n", 40 | "\n", 41 | "__The Good News__\n", 42 | "\n", 43 | "There are lot more choices today than just two years ago, and the \"rise\" of ML/AI has meant a lot more attention across the industry recently\n", 44 | "\n", 45 | "We'll tackle the first, easier list of concerns, and then consider options that help us with the second list." 46 | ] 47 | }, 48 | { 49 | "cell_type": "markdown", 50 | "metadata": {}, 51 | "source": [ 52 | "### Containerization...\n", 53 | "\n", 54 | "May not be the answer to everything, but it helps quite a bit with the core deployment task.\n", 55 | "\n", 56 | "Why? ML model inference tends to be \n", 57 | "* stateless\n", 58 | "* inexpensive (there are some exceptions)\n", 59 | "* easy on the network (simple protocols, light traffic, layer 7 routable, etc.)\n", 60 | "* idempotent\n", 61 | "\n", 62 | "In other words, a container hosting a ML inference app is close to the text book \"tennis-without-a-net\" case for containerized deployment.\n", 63 | "\n", 64 | "Commercial clouds provide de facto containerization\n", 65 | "* AWS Lambda is not an uncommon way to deploy a model (for better or worse)\n", 66 | " * e.g., https://github.com/alecrubin/pytorch-serverless\n", 67 | "* Google App Engine servlet host is effectively containerized\n", 68 | " * DMG PFA interactive demos are served via GAE\n", 69 | "* etc." 70 | ] 71 | }, 72 | { 73 | "cell_type": "markdown", 74 | "metadata": {}, 75 | "source": [ 76 | "" 77 | ] 78 | }, 79 | { 80 | "cell_type": "markdown", 81 | "metadata": {}, 82 | "source": [ 83 | "*For on-prem deployments, hybrid on-prem-and-cloud, or cloud deployments that need more standard and less lock-in, __Docker__ is the de facto standard containerization tech.*\n", 84 | "\n", 85 | "So ... all of the runtimes should have a vetted, documented, ready-to-run Dockerfile and/or image ... right? Sadly, no.\n", 86 | "\n", 87 | "Although we don't always have a Dockerfile to start with, there's a bright side or two:\n", 88 | "* Most of the components are there, so assembling a Docker image is straightforward\n", 89 | "* Enterprises may prefer to build and maintain their own images anyway\n", 90 | " * Security / auditing\n", 91 | " * Customization\n", 92 | " * Recovery / BCP, etc.\n", 93 | " \n", 94 | "## Model Deployment Platforms - Recap\n", 95 | "\n", 96 | "### Past\n", 97 | "\n", 98 | "In the commercial world, vendors that offer modeling tooling typically offer a scoring platform as well.\n", 99 | "\n", 100 | "The vendors include SAS, FICO, Minitab, KNIME, Open Data Group, and many others. \n", 101 | "\n", 102 | "As with traditional enterprise software, choosing and using one is a complex decision that takes into account...\n", 103 | "* functionality\n", 104 | "* technology requirements\n", 105 | "* cost/value\n", 106 | "* support/maintainability\n", 107 | "* staff skillset\n", 108 | "* business unit strategies\n", 109 | "* etc.\n", 110 | "\n", 111 | "" 112 | ] 113 | }, 114 | { 115 | "cell_type": "markdown", 116 | "metadata": {}, 117 | "source": [ 118 | "### Present\n", 119 | "\n", 120 | "Today there is a lot of pressure on the traditional ecosystem from\n", 121 | "\n", 122 | "* Open source tooling like Python, R, SparkML, XGBoost, etc.\n", 123 | " * which can handle bigger volumes of data\n", 124 | " * more varieties models\n", 125 | " * innovative worklows\n", 126 | "\n", 127 | "\n", 128 | "* A generation of data scientists \n", 129 | " * who know and use those tools, data, and workflows\n", 130 | " * are less familiar with traditional tooling\n", 131 | " * want to integrate data science processes with visualization tools, mobile applications, etc.\n", 132 | "\n", 133 | "#### Case Study: Michelangelo\n", 134 | "\n", 135 | "__Some firms have built their own ML deployment ecosystems from the ground up to meet these needs.__\n", 136 | "\n", 137 | "That approach is not for everyone, but looking at an example case study can help us get perspective on when we may want in a modern but more off-the-shelf solution.\n", 138 | "\n", 139 | "https://eng.uber.com/michelangelo/\n", 140 | "\n", 141 | "" 142 | ] 143 | }, 144 | { 145 | "cell_type": "markdown", 146 | "metadata": {}, 147 | "source": [ 148 | "*Michelangelo enables internal teams to seamlessly build, deploy, and operate machine learning solutions at Uber’s scale. It is designed to cover the end-to-end ML workflow: manage data, train, evaluate, and deploy models, make predictions, and monitor predictions. The system also supports traditional ML models, time series forecasting, and deep learning.*\n", 149 | "\n", 150 | "##### More to look at \n", 151 | "\n", 152 | "* Airbnb's \"Bighead\" platform: https://databricks.com/session/bighead-airbnbs-end-to-end-machine-learning-platform\n", 153 | "* Facebook FBLearner https://code.fb.com/core-data/introducing-fblearner-flow-facebook-s-ai-backbone/\n", 154 | "\n", 155 | "### (Near) Future\n", 156 | "\n", 157 | "__The traditional single-vendor analytics platform + server is not flexible enough. But most companies have neither the desire nor the ability to build an end-to-end ML platform like Uber and Airbnb have done.__\n", 158 | "\n", 159 | "The solutions slowly gaining traction are end-to-end free/open-source platforms built on open technologies, and cloud-based solutions which may not be open but provide ease of use.\n", 160 | "\n", 161 | "#### Cloud-Native vs. Cloud-or-Local\n", 162 | "\n", 163 | "__Cloud, Open-Source Compatible__\n", 164 | "* Sagemaker, Model Server for MXNet\n", 165 | "* Azure, Azure ML + Azure ML Server\n", 166 | "* Google Cloud ML\n", 167 | "* more...\n", 168 | "\n", 169 | "" 170 | ] 171 | }, 172 | { 173 | "cell_type": "markdown", 174 | "metadata": {}, 175 | "source": [ 176 | "__On-Prem or Cloud, Open-Source ... Partial Solutions__\n", 177 | "* Kubeflow\n", 178 | "* Seldon Core\n", 179 | "* MLflow\n", 180 | "* Google/TFX\n", 181 | "* GraphPipe (Oracle OSS)\n", 182 | "* Pachyderm\n", 183 | "\n", 184 | "Components:\n", 185 | "* MIT/CSAIL ModelDB https://mitdbg.github.io/modeldb/\n", 186 | "* Hopsworks Feature Store\n", 187 | "* Google Cloud / GO-JEK Feast (feature store) https://github.com/gojek/feast\n", 188 | "* Polyaxon\n", 189 | "\n", 190 | "" 191 | ] 192 | }, 193 | { 194 | "cell_type": "markdown", 195 | "metadata": {}, 196 | "source": [ 197 | "__And dozens of commercial products__ some with free tiers, others sold in the traditional enterprise software manner. Commercial tools range from industry giants like Anaconda Enterprise and Domino Data Lab to literally dozens of startups.\n", 198 | "\n", 199 | "Since we're focusing on open source, we won't go into detail on the proprietary tools.\n", 200 | "\n", 201 | "__But the takeaway for OSS and proprietary alike is that no extant tool is perfect.__\n", 202 | "\n", 203 | "As the industry coalesces around a common set of goals, there is reason to hope that both the open and commercial tools with distill to a smaller set of options. And, in the open-source world, that will allow more focused contributions and a faster path to maturity for a few great tools moving forward." 204 | ] 205 | }, 206 | { 207 | "cell_type": "code", 208 | "execution_count": null, 209 | "metadata": {}, 210 | "outputs": [], 211 | "source": [] 212 | } 213 | ], 214 | "metadata": { 215 | "kernelspec": { 216 | "display_name": "Python 3", 217 | "language": "python", 218 | "name": "python3" 219 | }, 220 | "language_info": { 221 | "codemirror_mode": { 222 | "name": "ipython", 223 | "version": 3 224 | }, 225 | "file_extension": ".py", 226 | "mimetype": "text/x-python", 227 | "name": "python", 228 | "nbconvert_exporter": "python", 229 | "pygments_lexer": "ipython3", 230 | "version": "3.7.5" 231 | } 232 | }, 233 | "nbformat": 4, 234 | "nbformat_minor": 4 235 | } 236 | -------------------------------------------------------------------------------- /01-Intro.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Deploy Machine Learning Projects in Production with Open Standard Models" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "## Setting the Scene: Core Problem" 15 | ] 16 | }, 17 | { 18 | "cell_type": "markdown", 19 | "metadata": {}, 20 | "source": [ 21 | "In the context of machine learning...\n", 22 | "\n", 23 | "* __Training__ is the process of obtaining a (hopefully useful) machine learning model from data\n", 24 | "* __Inference__ or __Prediction__ or __Scoring__ is the process of using a model to obtain a __score__ for some new data encountered in your business or research\n", 25 | "* Sometimes (e.g., in __online learning__ use cases) the same system is performing training and inference at approximately the same time\n", 26 | "\n", 27 | "In this session, we will __not__ focus on training (or data preparation, cleaning, model tuning, selection, etc.) \n", 28 | "\n", 29 | "There are lots of great resources focusing on that phase of work, and we're going to assume that you already have a model you're happy with (or at least a process for creating those models).\n", 30 | "\n", 31 | "So you've trained and tuned a model, done some validation tests to ensure it generates appropriate predictions on new data, and you want to deploy this model into an enterprise-scale inference service, where it can deliver predictions for the business. \n", 32 | "\n", 33 | "> For example, you've trained a great recommender system, and *now it needs to be exposed as a scalable service* consumed by your company's online-store app, which will send carts of products to that recommender, and receive back recommended products to offer customers." 34 | ] 35 | }, 36 | { 37 | "cell_type": "markdown", 38 | "metadata": {}, 39 | "source": [ 40 | "## Open Source Pre-History: Proprietary Inference Servers\n", 41 | "\n", 42 | "Why is this a challenge today?\n", 43 | "\n", 44 | "For a long time, businesses using machine learning employed proprietary tools like SAS, SPSS, and FICO to perform modeling.\n", 45 | "\n", 46 | "Many of these products and vendors licensed proprietary \"model servers\" or \"inference servers\" which were created specifically to take models and expose them elsewhere in the IT infrastructure as a service.\n", 47 | "\n", 48 | "If your company was a customer of these products, the enterprise \"solution\" included both the data mining tools (modeling) and the serving tools (inference)." 49 | ] 50 | }, 51 | { 52 | "cell_type": "markdown", 53 | "metadata": {}, 54 | "source": [ 55 | "## The Rise of Open Source: Stone Age\n", 56 | "\n", 57 | "As open-source data science tools rose in prominence over the last decade, more data scientists, statisticians, researchers, and analysts began relying on\n", 58 | "* Python\n", 59 | " * SciPy stack\n", 60 | " * scitkit-learn\n", 61 | " * TensorFlow\n", 62 | " * etc.\n", 63 | "* R\n", 64 | " * dplyr\n", 65 | " * ggplot2\n", 66 | " * etc.\n", 67 | "* Spark, H2O, others...\n", 68 | "\n", 69 | "As we've all seen, the cycle of research, development, publication, and open-source tooling has led to a huge explosion of data-driven uses throughout the world.\n", 70 | "\n", 71 | "__But__ none of those tools had a clear, complete story for how to deploy a model once it was trained.\n", 72 | "\n", 73 | "So engineers carved out the *Stone-Age Solution* ... namely, attempt to wrap the data science stack in a lightweight web service framework, and put it into production.\n", 74 | "\n", 75 | "The classic example is a Python [Flask](https://en.wikipedia.org/wiki/Flask_(web_framework)) web endpoint that wraps a call to scikit-learn's `model.predict(...)`\n", 76 | "\n", 77 | "Before discussing the many drawbacks of this approach, let's quickly review..." 78 | ] 79 | }, 80 | { 81 | "cell_type": "markdown", 82 | "metadata": {}, 83 | "source": [ 84 | "## Open Source: Bronze Age\n", 85 | "\n", 86 | "Since model inference is typically lightweight, stateless, and idempotent, it is an ideal candidate for a scale-out containerized service using a container scaling framework like Kubernetes.\n", 87 | "\n", 88 | "The \"Bronze Age\" of open-source model deployment containerized the Stone Age approach, making it easy to scale, robust, etc.\n", 89 | "\n", 90 | "Containerization was definitely an improvement ..." 91 | ] 92 | }, 93 | { 94 | "cell_type": "markdown", 95 | "metadata": {}, 96 | "source": [ 97 | "## Open Source: Platform Gold Rush\n", 98 | "\n", 99 | "\n", 100 | "\n", 101 | "Businesses realized that they wanted enterprise manageability over these ML inference services ... \n", 102 | "\n", 103 | "And a lot of entrepreneurs realized that making money in novel ML training was hard (after all, thousands of Ph.D. researchers were working on the same problems, and giving the results away for free) ... but making a \"platform\" that\n", 104 | "* Dockerized open-source ML stacks\n", 105 | "* Deployed them on-prem or in the cloud via Kubernetes\n", 106 | "* and provided some manageability (\"ML Ops\") \n", 107 | "was both easy and lucrative.\n", 108 | "\n", 109 | "### 2018-2019 will go down as the ML Ops Gold Rush\n", 110 | "\n", 111 | "And, as in the California Gold Rush, it has been easier selling tools and services than finding actual gold.\n", 112 | "\n", 113 | "__ML deployment platforms *do* have value to offer__ and we'll come back to that part. But first we need to focus on the Achilles heel, namely Dockerized data science stacks." 114 | ] 115 | }, 116 | { 117 | "cell_type": "markdown", 118 | "metadata": {}, 119 | "source": [ 120 | "## You Wouldn't Deploy an Enterprise Service by Putting Your Dev Machine in the Datacenter\n", 121 | "\n", 122 | "\n", 123 | "\n", 124 | "## So Why Would You Deploy a ML Service by Putting the ML Stack in a Container?\n", 125 | "\n", 126 | "\"It works (for now)\" is about the best thing you can say about such a deployment.\n", 127 | "\n", 128 | "Meanwhile, how do we address...\n", 129 | "\n", 130 | "* model inspection\n", 131 | "* versioning\n", 132 | "* diffing model versions\n", 133 | "* porting to other environments (e.g., ARM vs. Intel or mobile vs. client vs. server)\n", 134 | "* using the model in an alternate runtime (e.g., a scikit-learn model in a Spark job)\n", 135 | "* using models *from* an alternate runtime (e.g., Spark cannot natively export a ML pipeline to a containerizable service)\n", 136 | "* updating dependencies (e.g., patching a security vulnerability in underlying components https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-6446)\n", 137 | "* not to mention lots of design issues like...\n", 138 | " * Why should a ML model (which is typically a limited set of math operations) be deployed as a full computing stack and environment?\n", 139 | " * Why should we use an enormous container and billions of compute cycles to perform arithmetic and a bit of trigonometry?\n", 140 | " \n", 141 | "### Fundamentally, Containerizing a ML Stack (with Model) Violates Separation of Concerns\n", 142 | "\n", 143 | "Consider: we send each other plain-text emails, which can be written and read according to standard text encodings, rather than, say, the absurd idea of sending executable VM Images with an OS and a word processor, along with document in the word processor's proprietary format\n", 144 | "\n", 145 | "The ML model, once trained, can be viewed as data. \n", 146 | "\n", 147 | "It should be possible to \n", 148 | "* manage this data using standard, well known data-management tools and practices\n", 149 | "* create this data using any compliant tool\n", 150 | "* consume this data using any compliant tool\n", 151 | "* validate that this data has a single universal interpretation\n", 152 | " * why is this important? consider the impact of tiny difference in implementation of, say, *ln(x)* on inference at scale" 153 | ] 154 | }, 155 | { 156 | "cell_type": "markdown", 157 | "metadata": {}, 158 | "source": [ 159 | "## How Can We Address this Separation-of-Concerns Problem?\n", 160 | "\n", 161 | "__Get the model *out* of the model-creation environment (both logically and physically)__\n", 162 | "\n", 163 | "Physically: create a separate entity like a file\n", 164 | "\n", 165 | "Logically: ensure that entity is independent -- so saving a scikit-learn model as a pickle file (which will later need scikit-learn after being unpickled) does not count as a solution\n" 166 | ] 167 | }, 168 | { 169 | "cell_type": "markdown", 170 | "metadata": {}, 171 | "source": [ 172 | "## What Kind of Separate File Do We Want?\n", 173 | "\n", 174 | "Ideally, we'd like a format that is ...\n", 175 | "* an open, cross-industry standard\n", 176 | "* not owned or controlled by any one organization\n", 177 | "* not encumbered by intellectual property restrictions (licensing rules)\n", 178 | "* works with many ML tools, on many platforms\n", 179 | "* time/space efficient\n", 180 | "* robust (can support many kinds of ML models, including future types)\n", 181 | "* consistent (produces the same output for the same model, no matter the deployment OS, architecture, etc.)\n", 182 | "* simple (does not support unnecessary operations)\n", 183 | "* secure (minimizes attack surface by design, offers verifiability, etc)\n", 184 | "* is human readable (or can be made human readable)\n", 185 | "* can be managed in any database, content-management system, source control, etc.\n", 186 | "\n", 187 | "*As in most engineering scenarios, there is no single, magical solution that hits every bullet-point*\n", 188 | "\n", 189 | "But there are number of approaches which offer many of these attributes and which are worthy of consideration.\n", 190 | "\n", 191 | "This session looks at several of these tools." 192 | ] 193 | }, 194 | { 195 | "cell_type": "code", 196 | "execution_count": null, 197 | "metadata": {}, 198 | "outputs": [], 199 | "source": [] 200 | } 201 | ], 202 | "metadata": { 203 | "kernelspec": { 204 | "display_name": "Python 3", 205 | "language": "python", 206 | "name": "python3" 207 | }, 208 | "language_info": { 209 | "codemirror_mode": { 210 | "name": "ipython", 211 | "version": 3 212 | }, 213 | "file_extension": ".py", 214 | "mimetype": "text/x-python", 215 | "name": "python", 216 | "nbconvert_exporter": "python", 217 | "pygments_lexer": "ipython3", 218 | "version": "3.7.4" 219 | } 220 | }, 221 | "nbformat": 4, 222 | "nbformat_minor": 4 223 | } 224 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "[]" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright 2018-2019 by Adam Breindel 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. 202 | --------------------------------------------------------------------------------