├── model_store └── .gitkeep ├── sample ├── fastsource.png ├── sample_pred_mask.png ├── street_view_of_a_small_neighborhood.png └── sample_output.txt ├── deployment ├── config.properties ├── dockerd-entrypoint.sh ├── model.py └── handler.py ├── CODE_OF_CONDUCT.md ├── LICENSE ├── Dockerfile ├── .gitignore ├── CONTRIBUTING.md ├── notebook ├── 01_U-net_Modelling.ipynb ├── 02_Inference_in_pytorch.ipynb ├── 04_SageMaker.ipynb └── 03_TorchServe.ipynb └── README.md /model_store/.gitkeep: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /sample/fastsource.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-sagemaker-endpoint-deployment-of-fastai-model-with-torchserve/HEAD/sample/fastsource.png -------------------------------------------------------------------------------- /sample/sample_pred_mask.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-sagemaker-endpoint-deployment-of-fastai-model-with-torchserve/HEAD/sample/sample_pred_mask.png -------------------------------------------------------------------------------- /sample/street_view_of_a_small_neighborhood.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-sagemaker-endpoint-deployment-of-fastai-model-with-torchserve/HEAD/sample/street_view_of_a_small_neighborhood.png -------------------------------------------------------------------------------- /deployment/config.properties: -------------------------------------------------------------------------------- 1 | inference_address=http://0.0.0.0:8080 2 | management_address=http://0.0.0.0:8081 3 | number_of_netty_threads=32 4 | job_queue_size=1000 5 | model_store=/opt/ml/model 6 | load_models=all -------------------------------------------------------------------------------- /deployment/dockerd-entrypoint.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | set -e 3 | 4 | if [[ "$1" = "serve" ]]; then 5 | shift 1 6 | printenv 7 | ls /opt 8 | torchserve --start --ts-config /home/model-server/config.properties 9 | else 10 | eval "$@" 11 | fi 12 | 13 | # prevent docker exit 14 | tail -f /dev/null -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | ## Code of Conduct 2 | 3 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 4 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 5 | opensource-codeofconduct@amazon.com with any additional questions or comments. 6 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining a copy of this 4 | software and associated documentation files (the "Software"), to deal in the Software 5 | without restriction, including without limitation the rights to use, copy, modify, 6 | merge, publish, distribute, sublicense, and/or sell copies of the Software, and to 7 | permit persons to whom the Software is furnished to do so. 8 | 9 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, 10 | INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A 11 | PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT 12 | HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION 13 | OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE 14 | SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 15 | -------------------------------------------------------------------------------- /Dockerfile: -------------------------------------------------------------------------------- 1 | FROM pytorch/pytorch:1.6.0-cuda10.1-cudnn7-runtime 2 | 3 | ENV PYTHONUNBUFFERED TRUE 4 | 5 | # FASTAI 6 | RUN apt-get update && apt-get install -y software-properties-common rsync 7 | RUN add-apt-repository -y ppa:git-core/ppa && apt-get update && apt-get install -y git libglib2.0-dev graphviz && apt-get update 8 | RUN pip install albumentations \ 9 | catalyst \ 10 | captum \ 11 | "fastprogress>=0.1.22" \ 12 | graphviz \ 13 | kornia \ 14 | matplotlib \ 15 | "nbconvert<6"\ 16 | neptune-cli \ 17 | opencv-python \ 18 | pandas \ 19 | pillow \ 20 | pyarrow \ 21 | pydicom \ 22 | pyyaml \ 23 | scikit-learn \ 24 | scikit-image \ 25 | scipy \ 26 | "sentencepiece<0.1.90" \ 27 | spacy \ 28 | tensorboard \ 29 | wandb 30 | 31 | # TORCHSERVE 32 | RUN DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends -y \ 33 | fakeroot \ 34 | ca-certificates \ 35 | dpkg-dev \ 36 | g++ \ 37 | openjdk-11-jdk \ 38 | curl \ 39 | vim \ 40 | && rm -rf /var/lib/apt/lists/* 41 | 42 | # FASTAI 43 | RUN git clone https://github.com/fastai/fastai.git && git clone https://github.com/fastai/fastcore.git 44 | RUN /bin/bash -c "cd fastai && git checkout 2.0.18 && pip install . && cd ../fastcore && git checkout 1.1.0 && pip install ." 45 | 46 | # TORCHSERVE 47 | RUN git clone https://github.com/pytorch/serve.git 48 | RUN pip install ./serve/ 49 | RUN pip install captum 50 | 51 | COPY ./deployment/dockerd-entrypoint.sh /usr/local/bin/dockerd-entrypoint.sh 52 | RUN chmod +x /usr/local/bin/dockerd-entrypoint.sh 53 | 54 | RUN mkdir -p /home/model-server/ && mkdir -p /home/model-server/tmp 55 | COPY ./deployment/config.properties /home/model-server/config.properties 56 | 57 | WORKDIR /home/model-server 58 | ENV TEMP=/home/model-server/tmp 59 | ENTRYPOINT ["/usr/local/bin/dockerd-entrypoint.sh"] 60 | CMD ["serve"] 61 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Created by .ignore support plugin (hsz.mobi) 2 | ### Example user template template 3 | ### Example user template 4 | 5 | # IntelliJ project files 6 | .idea 7 | *.iml 8 | out 9 | gen 10 | ### Python template 11 | # Byte-compiled / optimized / DLL files 12 | __pycache__/ 13 | *.py[cod] 14 | *$py.class 15 | 16 | # C extensions 17 | *.so 18 | 19 | # Distribution / packaging 20 | .Python 21 | build/ 22 | develop-eggs/ 23 | dist/ 24 | downloads/ 25 | eggs/ 26 | .eggs/ 27 | lib/ 28 | lib64/ 29 | parts/ 30 | sdist/ 31 | var/ 32 | wheels/ 33 | pip-wheel-metadata/ 34 | share/python-wheels/ 35 | *.egg-info/ 36 | .installed.cfg 37 | *.egg 38 | MANIFEST 39 | 40 | # PyInstaller 41 | # Usually these files are written by a python script from a template 42 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 43 | *.manifest 44 | *.spec 45 | 46 | # Installer logs 47 | pip-log.txt 48 | pip-delete-this-directory.txt 49 | 50 | # Unit test / coverage reports 51 | htmlcov/ 52 | .tox/ 53 | .nox/ 54 | .coverage 55 | .coverage.* 56 | .cache 57 | nosetests.xml 58 | coverage.xml 59 | *.cover 60 | *.py,cover 61 | .hypothesis/ 62 | .pytest_cache/ 63 | 64 | # Translations 65 | *.mo 66 | *.pot 67 | 68 | # Django stuff: 69 | *.log 70 | local_settings.py 71 | db.sqlite3 72 | db.sqlite3-journal 73 | 74 | # Flask stuff: 75 | instance/ 76 | .webassets-cache 77 | 78 | # Scrapy stuff: 79 | .scrapy 80 | 81 | # Sphinx documentation 82 | docs/_build/ 83 | 84 | # PyBuilder 85 | target/ 86 | 87 | # Jupyter Notebook 88 | .ipynb_checkpoints 89 | 90 | # IPython 91 | profile_default/ 92 | ipython_config.py 93 | 94 | # pyenv 95 | .python-version 96 | 97 | # pipenv 98 | # According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. 99 | # However, in case of collaboration, if having platform-specific dependencies or dependencies 100 | # having no cross-platform support, pipenv may install dependencies that don't work, or not 101 | # install all needed dependencies. 102 | #Pipfile.lock 103 | 104 | # PEP 582; used by e.g. github.com/David-OConnor/pyflow 105 | __pypackages__/ 106 | 107 | # Celery stuff 108 | celerybeat-schedule 109 | celerybeat.pid 110 | 111 | # SageMath parsed files 112 | *.sage.py 113 | 114 | # Environments 115 | .env 116 | .venv 117 | env/ 118 | venv/ 119 | ENV/ 120 | env.bak/ 121 | venv.bak/ 122 | 123 | # Spyder project settings 124 | .spyderproject 125 | .spyproject 126 | 127 | # Rope project settings 128 | .ropeproject 129 | 130 | # mkdocs documentation 131 | /site 132 | 133 | # mypy 134 | .mypy_cache/ 135 | .dmypy.json 136 | dmypy.json 137 | 138 | # Pyre type checker 139 | .pyre/ 140 | 141 | .DS_Store 142 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing Guidelines 2 | 3 | Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional 4 | documentation, we greatly value feedback and contributions from our community. 5 | 6 | Please read through this document before submitting any issues or pull requests to ensure we have all the necessary 7 | information to effectively respond to your bug report or contribution. 8 | 9 | ## Reporting Bugs/Feature Requests 10 | 11 | We welcome you to use the GitHub issue tracker to report bugs or suggest features. 12 | 13 | When filing an issue, please check existing open, or recently closed, issues to make sure somebody else hasn't already 14 | reported the issue. Please try to include as much information as you can. Details like these are incredibly useful: 15 | 16 | * A reproducible test case or series of steps 17 | * The version of our code being used 18 | * Any modifications you've made relevant to the bug 19 | * Anything unusual about your environment or deployment 20 | 21 | ## Contributing via Pull Requests 22 | 23 | Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that: 24 | 25 | 1. You are working against the latest source on the *master* branch. 26 | 2. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already. 27 | 3. You open an issue to discuss any significant work - we would hate for your time to be wasted. 28 | 29 | To send us a pull request, please: 30 | 31 | 1. Fork the repository. 32 | 2. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change. 33 | 3. Ensure local tests pass. 34 | 4. Commit to your fork using clear commit messages. 35 | 5. Send us a pull request, answering any default questions in the pull request interface. 36 | 6. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation. 37 | 38 | GitHub provides additional document on [forking a repository](https://help.github.com/articles/fork-a-repo/) and 39 | [creating a pull request](https://help.github.com/articles/creating-a-pull-request/). 40 | 41 | ## Finding contributions to work on 42 | 43 | Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels (enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any 'help wanted' issues is a great place to start. 44 | 45 | ## Code of Conduct 46 | 47 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 48 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 49 | opensource-codeofconduct@amazon.com with any additional questions or comments. 50 | 51 | ## Security issue notifications 52 | 53 | If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github issue. 54 | 55 | ## Licensing 56 | 57 | See the [LICENSE](LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution. 58 | 59 | We may ask you to sign a [Contributor License Agreement (CLA)](http://en.wikipedia.org/wiki/Contributor_License_Agreement) for larger changes. 60 | -------------------------------------------------------------------------------- /deployment/model.py: -------------------------------------------------------------------------------- 1 | from fastai.vision.all import * 2 | from fastai.vision.learner import _default_meta 3 | from fastai.vision.models.unet import _get_sz_change_idxs, UnetBlock, ResizeToOrig 4 | 5 | 6 | class DynamicUnetDIY(SequentialEx): 7 | "Create a U-Net from a given architecture." 8 | 9 | def __init__( 10 | self, 11 | arch=resnet50, 12 | n_classes=32, 13 | img_size=(96, 128), 14 | blur=False, 15 | blur_final=True, 16 | y_range=None, 17 | last_cross=True, 18 | bottle=False, 19 | init=nn.init.kaiming_normal_, 20 | norm_type=None, 21 | self_attention=None, 22 | act_cls=defaults.activation, 23 | n_in=3, 24 | cut=None, 25 | **kwargs 26 | ): 27 | meta = model_meta.get(arch, _default_meta) 28 | encoder = create_body( 29 | arch, n_in, pretrained=False, cut=ifnone(cut, meta["cut"]) 30 | ) 31 | imsize = img_size 32 | 33 | sizes = model_sizes(encoder, size=imsize) 34 | sz_chg_idxs = list(reversed(_get_sz_change_idxs(sizes))) 35 | self.sfs = hook_outputs([encoder[i] for i in sz_chg_idxs], detach=False) 36 | x = dummy_eval(encoder, imsize).detach() 37 | 38 | ni = sizes[-1][1] 39 | middle_conv = nn.Sequential( 40 | ConvLayer(ni, ni * 2, act_cls=act_cls, norm_type=norm_type, **kwargs), 41 | ConvLayer(ni * 2, ni, act_cls=act_cls, norm_type=norm_type, **kwargs), 42 | ).eval() 43 | x = middle_conv(x) 44 | layers = [encoder, BatchNorm(ni), nn.ReLU(), middle_conv] 45 | 46 | for i, idx in enumerate(sz_chg_idxs): 47 | not_final = i != len(sz_chg_idxs) - 1 48 | up_in_c, x_in_c = int(x.shape[1]), int(sizes[idx][1]) 49 | do_blur = blur and (not_final or blur_final) 50 | sa = self_attention and (i == len(sz_chg_idxs) - 3) 51 | unet_block = UnetBlock( 52 | up_in_c, 53 | x_in_c, 54 | self.sfs[i], 55 | final_div=not_final, 56 | blur=do_blur, 57 | self_attention=sa, 58 | act_cls=act_cls, 59 | init=init, 60 | norm_type=norm_type, 61 | **kwargs 62 | ).eval() 63 | layers.append(unet_block) 64 | x = unet_block(x) 65 | 66 | ni = x.shape[1] 67 | if imsize != sizes[0][-2:]: 68 | layers.append(PixelShuffle_ICNR(ni, act_cls=act_cls, norm_type=norm_type)) 69 | layers.append(ResizeToOrig()) 70 | if last_cross: 71 | layers.append(MergeLayer(dense=True)) 72 | ni += in_channels(encoder) 73 | layers.append( 74 | ResBlock( 75 | 1, 76 | ni, 77 | ni // 2 if bottle else ni, 78 | act_cls=act_cls, 79 | norm_type=norm_type, 80 | **kwargs 81 | ) 82 | ) 83 | layers += [ 84 | ConvLayer(ni, n_classes, ks=1, act_cls=None, norm_type=norm_type, **kwargs) 85 | ] 86 | apply_init(nn.Sequential(layers[3], layers[-2]), init) 87 | # apply_init(nn.Sequential(layers[2]), init) 88 | if y_range is not None: 89 | layers.append(SigmoidRange(*y_range)) 90 | super().__init__(*layers) 91 | 92 | def __del__(self): 93 | if hasattr(self, "sfs"): 94 | self.sfs.remove() 95 | -------------------------------------------------------------------------------- /deployment/handler.py: -------------------------------------------------------------------------------- 1 | import base64 2 | import io 3 | import logging 4 | import os 5 | 6 | import numpy as np 7 | import torch 8 | from PIL import Image 9 | from torch.autograd import Variable 10 | from torchvision import transforms 11 | 12 | logger = logging.getLogger(__name__) 13 | 14 | 15 | class DIYSegmentation: 16 | """ 17 | DIYSegmentation handler class. 18 | """ 19 | 20 | def __init__(self): 21 | self.model = None 22 | self.mapping = None 23 | self.device = None 24 | self.initialized = False 25 | 26 | def initialize(self, ctx): 27 | """ 28 | load eager mode state_dict based model 29 | """ 30 | properties = ctx.system_properties 31 | self.device = torch.device( 32 | "cuda:" + str(properties.get("gpu_id")) 33 | if torch.cuda.is_available() 34 | else "cpu" 35 | ) 36 | 37 | logger.info(f"Device on initialization is: {self.device}") 38 | model_dir = properties.get("model_dir") 39 | 40 | manifest = ctx.manifest 41 | logger.error(manifest) 42 | serialized_file = manifest["model"]["serializedFile"] 43 | model_pt_path = os.path.join(model_dir, serialized_file) 44 | if not os.path.isfile(model_pt_path): 45 | raise RuntimeError("Missing the model definition file") 46 | 47 | logger.debug(model_pt_path) 48 | 49 | from model import DynamicUnetDIY 50 | 51 | state_dict = torch.load(model_pt_path, map_location=self.device) 52 | self.model = DynamicUnetDIY() 53 | self.model.load_state_dict(state_dict) 54 | self.model.to(self.device) 55 | self.model.eval() 56 | 57 | logger.debug("Model file {0} loaded successfully".format(model_pt_path)) 58 | self.initialized = True 59 | 60 | def preprocess(self, data): 61 | """ 62 | Scales and normalizes a PIL image for an U-net model 63 | """ 64 | image = data[0].get("data") 65 | if image is None: 66 | image = data[0].get("body") 67 | 68 | image_transform = transforms.Compose( 69 | [ 70 | # must be consistent with model training 71 | transforms.Resize((96, 128)), 72 | transforms.ToTensor(), 73 | # default statistics from imagenet 74 | transforms.Normalize( 75 | mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225] 76 | ), 77 | ] 78 | ) 79 | image = Image.open(io.BytesIO(image)).convert( 80 | "RGB" 81 | ) # in case of an alpha channel 82 | image = image_transform(image).unsqueeze_(0) 83 | return image 84 | 85 | def inference(self, img): 86 | """ 87 | Predict the chip stack mask of an image using a trained deep learning model. 88 | """ 89 | logger.info(f"Device on inference is: {self.device}") 90 | self.model.eval() 91 | inputs = Variable(img).to(self.device) 92 | outputs = self.model.forward(inputs) 93 | logging.debug(outputs.shape) 94 | return outputs 95 | 96 | def postprocess(self, inference_output): 97 | 98 | if torch.cuda.is_available(): 99 | inference_output = inference_output[0].argmax(dim=0).cpu() 100 | else: 101 | inference_output = inference_output[0].argmax(dim=0) 102 | 103 | return [ 104 | { 105 | "base64_prediction": base64.b64encode( 106 | inference_output.numpy().astype(np.uint8) 107 | ).decode("utf-8") 108 | } 109 | ] 110 | 111 | 112 | _service = DIYSegmentation() 113 | 114 | 115 | def handle(data, context): 116 | if not _service.initialized: 117 | _service.initialize(context) 118 | 119 | if data is None: 120 | return None 121 | 122 | data = _service.preprocess(data) 123 | data = _service.inference(data) 124 | data = _service.postprocess(data) 125 | 126 | return data 127 | -------------------------------------------------------------------------------- /notebook/01_U-net_Modelling.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "%reload_ext autoreload\n", 10 | "%autoreload 2\n", 11 | "\n", 12 | "\n", 13 | "%matplotlib inline" 14 | ] 15 | }, 16 | { 17 | "cell_type": "markdown", 18 | "metadata": {}, 19 | "source": [ 20 | "## Install FastAI\n", 21 | "\n", 22 | "https://github.com/fastai/fastai\n", 23 | "\n", 24 | "In short, `conda activate` the python environement, and run: \n", 25 | "```bash\n", 26 | "conda install -c fastai -c pytorch -c anaconda fastai gh anaconda\n", 27 | "```\n", 28 | "\n", 29 | "Note: For more details about the modelling approach, please refer to: \n", 30 | "https://github.com/fastai/fastbook/blob/master/01_intro.ipynb" 31 | ] 32 | }, 33 | { 34 | "cell_type": "code", 35 | "execution_count": null, 36 | "metadata": {}, 37 | "outputs": [], 38 | "source": [ 39 | "from fastai.vision.all import *" 40 | ] 41 | }, 42 | { 43 | "cell_type": "code", 44 | "execution_count": null, 45 | "metadata": {}, 46 | "outputs": [], 47 | "source": [ 48 | "path = untar_data(URLs.CAMVID_TINY)" 49 | ] 50 | }, 51 | { 52 | "cell_type": "code", 53 | "execution_count": null, 54 | "metadata": {}, 55 | "outputs": [], 56 | "source": [ 57 | "def acc_camvid(inp, targ, void_code=0):\n", 58 | " targ = targ.squeeze(1)\n", 59 | " mask = targ != void_code\n", 60 | " return (inp.argmax(dim=1)[mask] == targ[mask]).float().mean()" 61 | ] 62 | }, 63 | { 64 | "cell_type": "code", 65 | "execution_count": null, 66 | "metadata": {}, 67 | "outputs": [], 68 | "source": [ 69 | "def get_y(o, path=path):\n", 70 | " return path / \"labels\" / f\"{o.stem}_P{o.suffix}\"" 71 | ] 72 | }, 73 | { 74 | "cell_type": "code", 75 | "execution_count": null, 76 | "metadata": {}, 77 | "outputs": [], 78 | "source": [ 79 | "dls = SegmentationDataLoaders.from_label_func(\n", 80 | " path,\n", 81 | " bs=8,\n", 82 | " fnames=get_image_files(path / \"images\"),\n", 83 | " label_func=get_y,\n", 84 | " codes=np.loadtxt(path / \"codes.txt\", dtype=str),\n", 85 | ")\n", 86 | "dls.one_batch()[0].shape[-2:], get_c(dls)" 87 | ] 88 | }, 89 | { 90 | "cell_type": "code", 91 | "execution_count": null, 92 | "metadata": {}, 93 | "outputs": [], 94 | "source": [ 95 | "learn = unet_learner(dls, resnet50, metrics=acc_camvid)\n", 96 | "learn.fine_tune(20)" 97 | ] 98 | }, 99 | { 100 | "cell_type": "code", 101 | "execution_count": null, 102 | "metadata": {}, 103 | "outputs": [], 104 | "source": [ 105 | "learn.show_results(max_n=6, figsize=(7,8))" 106 | ] 107 | }, 108 | { 109 | "cell_type": "code", 110 | "execution_count": null, 111 | "metadata": {}, 112 | "outputs": [], 113 | "source": [ 114 | "learn.export(\"./fastai_unet.pkl\")" 115 | ] 116 | } 117 | ], 118 | "metadata": { 119 | "kernelspec": { 120 | "display_name": "Python 3", 121 | "language": "python", 122 | "name": "python3" 123 | }, 124 | "language_info": { 125 | "codemirror_mode": { 126 | "name": "ipython", 127 | "version": 3 128 | }, 129 | "file_extension": ".py", 130 | "mimetype": "text/x-python", 131 | "name": "python", 132 | "nbconvert_exporter": "python", 133 | "pygments_lexer": "ipython3", 134 | "version": "3.8.5" 135 | }, 136 | "toc": { 137 | "base_numbering": 1, 138 | "nav_menu": {}, 139 | "number_sections": true, 140 | "sideBar": true, 141 | "skip_h1_title": false, 142 | "title_cell": "Table of Contents", 143 | "title_sidebar": "Contents", 144 | "toc_cell": false, 145 | "toc_position": {}, 146 | "toc_section_display": true, 147 | "toc_window_display": false 148 | }, 149 | "varInspector": { 150 | "cols": { 151 | "lenName": 16, 152 | "lenType": 16, 153 | "lenVar": 40 154 | }, 155 | "kernels_config": { 156 | "python": { 157 | "delete_cmd_postfix": "", 158 | "delete_cmd_prefix": "del ", 159 | "library": "var_list.py", 160 | "varRefreshCmd": "print(var_dic_list())" 161 | }, 162 | "r": { 163 | "delete_cmd_postfix": ") ", 164 | "delete_cmd_prefix": "rm(", 165 | "library": "var_list.r", 166 | "varRefreshCmd": "cat(var_dic_list()) " 167 | } 168 | }, 169 | "types_to_exclude": [ 170 | "module", 171 | "function", 172 | "builtin_function_or_method", 173 | "instance", 174 | "_Feature" 175 | ], 176 | "window_display": false 177 | } 178 | }, 179 | "nbformat": 4, 180 | "nbformat_minor": 4 181 | } 182 | -------------------------------------------------------------------------------- /notebook/02_Inference_in_pytorch.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "%reload_ext autoreload\n", 10 | "%autoreload 2\n", 11 | "\n", 12 | "\n", 13 | "%matplotlib inline" 14 | ] 15 | }, 16 | { 17 | "cell_type": "code", 18 | "execution_count": null, 19 | "metadata": {}, 20 | "outputs": [], 21 | "source": [ 22 | "from fastai.vision.all import *\n", 23 | "import matplotlib.pyplot as plt\n", 24 | "from PIL import Image" 25 | ] 26 | }, 27 | { 28 | "cell_type": "markdown", 29 | "metadata": {}, 30 | "source": [ 31 | "## Load FastAI Model and Save Its Well Trained Weights" 32 | ] 33 | }, 34 | { 35 | "cell_type": "markdown", 36 | "metadata": {}, 37 | "source": [ 38 | "### Load Model" 39 | ] 40 | }, 41 | { 42 | "cell_type": "code", 43 | "execution_count": null, 44 | "metadata": {}, 45 | "outputs": [], 46 | "source": [ 47 | "def acc_camvid(*_): pass\n", 48 | "\n", 49 | "def get_y(*_): pass\n", 50 | "\n", 51 | "learn = load_learner(\"/home/ubuntu/.fastai/data/camvid_tiny/fastai_unet.pkl\")" 52 | ] 53 | }, 54 | { 55 | "cell_type": "markdown", 56 | "metadata": {}, 57 | "source": [ 58 | "### Load Data" 59 | ] 60 | }, 61 | { 62 | "cell_type": "code", 63 | "execution_count": null, 64 | "metadata": {}, 65 | "outputs": [], 66 | "source": [ 67 | "image_path = \"../sample/street_view_of_a_small_neighborhood.png\"\n", 68 | "Image.open(image_path)" 69 | ] 70 | }, 71 | { 72 | "cell_type": "markdown", 73 | "metadata": {}, 74 | "source": [ 75 | "### Inference" 76 | ] 77 | }, 78 | { 79 | "cell_type": "code", 80 | "execution_count": null, 81 | "metadata": {}, 82 | "outputs": [], 83 | "source": [ 84 | "%%time\n", 85 | "pred_fastai = learn.predict(image_path)\n", 86 | "plt.imshow(pred_fastai[0].numpy());" 87 | ] 88 | }, 89 | { 90 | "cell_type": "markdown", 91 | "metadata": {}, 92 | "source": [ 93 | "### Save Torch Weights" 94 | ] 95 | }, 96 | { 97 | "cell_type": "code", 98 | "execution_count": null, 99 | "metadata": {}, 100 | "outputs": [], 101 | "source": [ 102 | "torch.save(learn.model.state_dict(), \"../model_store/fasti_unet_weights.pth\")\n", 103 | "learn.model" 104 | ] 105 | }, 106 | { 107 | "cell_type": "markdown", 108 | "metadata": {}, 109 | "source": [ 110 | "## Extract FastAI Model in PyTorch" 111 | ] 112 | }, 113 | { 114 | "cell_type": "code", 115 | "execution_count": null, 116 | "metadata": {}, 117 | "outputs": [], 118 | "source": [ 119 | "??unet_learner" 120 | ] 121 | }, 122 | { 123 | "cell_type": "code", 124 | "execution_count": null, 125 | "metadata": {}, 126 | "outputs": [], 127 | "source": [ 128 | "??DynamicUnet" 129 | ] 130 | }, 131 | { 132 | "cell_type": "code", 133 | "execution_count": null, 134 | "metadata": {}, 135 | "outputs": [], 136 | "source": [ 137 | "from fastai.vision.all import *\n", 138 | "from fastai.vision.learner import _default_meta\n", 139 | "from fastai.vision.models.unet import _get_sz_change_idxs, UnetBlock, ResizeToOrig\n", 140 | "\n", 141 | "\n", 142 | "class DynamicUnetDIY(SequentialEx):\n", 143 | " \"Create a U-Net from a given architecture.\"\n", 144 | "\n", 145 | " def __init__(\n", 146 | " self,\n", 147 | " arch=resnet50,\n", 148 | " n_classes=32,\n", 149 | " img_size=(96, 128),\n", 150 | " blur=False,\n", 151 | " blur_final=True,\n", 152 | " y_range=None,\n", 153 | " last_cross=True,\n", 154 | " bottle=False,\n", 155 | " init=nn.init.kaiming_normal_,\n", 156 | " norm_type=None,\n", 157 | " self_attention=None,\n", 158 | " act_cls=defaults.activation,\n", 159 | " n_in=3,\n", 160 | " cut=None,\n", 161 | " **kwargs\n", 162 | " ):\n", 163 | " meta = model_meta.get(arch, _default_meta)\n", 164 | " encoder = create_body(\n", 165 | " arch, n_in, pretrained=False, cut=ifnone(cut, meta[\"cut\"])\n", 166 | " )\n", 167 | " imsize = img_size\n", 168 | "\n", 169 | " sizes = model_sizes(encoder, size=imsize)\n", 170 | " sz_chg_idxs = list(reversed(_get_sz_change_idxs(sizes)))\n", 171 | " self.sfs = hook_outputs([encoder[i] for i in sz_chg_idxs], detach=False)\n", 172 | " x = dummy_eval(encoder, imsize).detach()\n", 173 | "\n", 174 | " ni = sizes[-1][1]\n", 175 | " middle_conv = nn.Sequential(\n", 176 | " ConvLayer(ni, ni * 2, act_cls=act_cls, norm_type=norm_type, **kwargs),\n", 177 | " ConvLayer(ni * 2, ni, act_cls=act_cls, norm_type=norm_type, **kwargs),\n", 178 | " ).eval()\n", 179 | " x = middle_conv(x)\n", 180 | " layers = [encoder, BatchNorm(ni), nn.ReLU(), middle_conv]\n", 181 | "\n", 182 | " for i, idx in enumerate(sz_chg_idxs):\n", 183 | " not_final = i != len(sz_chg_idxs) - 1\n", 184 | " up_in_c, x_in_c = int(x.shape[1]), int(sizes[idx][1])\n", 185 | " do_blur = blur and (not_final or blur_final)\n", 186 | " sa = self_attention and (i == len(sz_chg_idxs) - 3)\n", 187 | " unet_block = UnetBlock(\n", 188 | " up_in_c,\n", 189 | " x_in_c,\n", 190 | " self.sfs[i],\n", 191 | " final_div=not_final,\n", 192 | " blur=do_blur,\n", 193 | " self_attention=sa,\n", 194 | " act_cls=act_cls,\n", 195 | " init=init,\n", 196 | " norm_type=norm_type,\n", 197 | " **kwargs\n", 198 | " ).eval()\n", 199 | " layers.append(unet_block)\n", 200 | " x = unet_block(x)\n", 201 | "\n", 202 | " ni = x.shape[1]\n", 203 | " if imsize != sizes[0][-2:]:\n", 204 | " layers.append(PixelShuffle_ICNR(ni, act_cls=act_cls, norm_type=norm_type))\n", 205 | " layers.append(ResizeToOrig())\n", 206 | " if last_cross:\n", 207 | " layers.append(MergeLayer(dense=True))\n", 208 | " ni += in_channels(encoder)\n", 209 | " layers.append(\n", 210 | " ResBlock(\n", 211 | " 1,\n", 212 | " ni,\n", 213 | " ni // 2 if bottle else ni,\n", 214 | " act_cls=act_cls,\n", 215 | " norm_type=norm_type,\n", 216 | " **kwargs\n", 217 | " )\n", 218 | " )\n", 219 | " layers += [\n", 220 | " ConvLayer(ni, n_classes, ks=1, act_cls=None, norm_type=norm_type, **kwargs)\n", 221 | " ]\n", 222 | " apply_init(nn.Sequential(layers[3], layers[-2]), init)\n", 223 | " # apply_init(nn.Sequential(layers[2]), init)\n", 224 | " if y_range is not None:\n", 225 | " layers.append(SigmoidRange(*y_range))\n", 226 | " super().__init__(*layers)\n", 227 | "\n", 228 | " def __del__(self):\n", 229 | " if hasattr(self, \"sfs\"):\n", 230 | " self.sfs.remove()" 231 | ] 232 | }, 233 | { 234 | "cell_type": "code", 235 | "execution_count": null, 236 | "metadata": {}, 237 | "outputs": [], 238 | "source": [ 239 | "model_torch_rep = DynamicUnetDIY()\n", 240 | "model_torch_rep" 241 | ] 242 | }, 243 | { 244 | "cell_type": "code", 245 | "execution_count": null, 246 | "metadata": {}, 247 | "outputs": [], 248 | "source": [ 249 | "state = torch.load(\"../model_store/fasti_unet_weights.pth\")\n", 250 | "model_torch_rep.load_state_dict(state)\n", 251 | "model_torch_rep.eval();" 252 | ] 253 | }, 254 | { 255 | "cell_type": "markdown", 256 | "metadata": {}, 257 | "source": [ 258 | "### Testing" 259 | ] 260 | }, 261 | { 262 | "cell_type": "code", 263 | "execution_count": null, 264 | "metadata": {}, 265 | "outputs": [], 266 | "source": [ 267 | "image = Image.open(image_path).convert(\"RGB\")\n", 268 | "image" 269 | ] 270 | }, 271 | { 272 | "cell_type": "code", 273 | "execution_count": null, 274 | "metadata": {}, 275 | "outputs": [], 276 | "source": [ 277 | "from torchvision import transforms" 278 | ] 279 | }, 280 | { 281 | "cell_type": "code", 282 | "execution_count": null, 283 | "metadata": {}, 284 | "outputs": [], 285 | "source": [ 286 | "image_tfm = transforms.Compose(\n", 287 | " [\n", 288 | " # must be consistent with model training\n", 289 | " transforms.Resize((96, 128)),\n", 290 | " transforms.ToTensor(),\n", 291 | " # default statistics from imagenet\n", 292 | " transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),\n", 293 | " ]\n", 294 | ")" 295 | ] 296 | }, 297 | { 298 | "cell_type": "code", 299 | "execution_count": null, 300 | "metadata": {}, 301 | "outputs": [], 302 | "source": [ 303 | "x = image_tfm(image).unsqueeze_(0)" 304 | ] 305 | }, 306 | { 307 | "cell_type": "code", 308 | "execution_count": null, 309 | "metadata": {}, 310 | "outputs": [], 311 | "source": [ 312 | "%%time\n", 313 | "# inference on CPU\n", 314 | "raw_out = model_torch_rep(x)\n", 315 | "raw_out.shape" 316 | ] 317 | }, 318 | { 319 | "cell_type": "code", 320 | "execution_count": null, 321 | "metadata": {}, 322 | "outputs": [], 323 | "source": [ 324 | "pred_res = raw_out[0].argmax(dim=0).numpy().astype(np.uint8)\n", 325 | "pred_res" 326 | ] 327 | }, 328 | { 329 | "cell_type": "code", 330 | "execution_count": null, 331 | "metadata": {}, 332 | "outputs": [], 333 | "source": [ 334 | "import base64\n", 335 | "import numpy as np\n", 336 | "\n", 337 | "pred_encoded = base64.b64encode(pred_res).decode(\"utf-8\")\n", 338 | "pred_decoded_byte = base64.decodebytes(bytes(pred_encoded, encoding=\"utf-8\"))\n", 339 | "pred_decoded = np.reshape(\n", 340 | " np.frombuffer(pred_decoded_byte, dtype=np.uint8), pred_res.shape\n", 341 | ")\n", 342 | "\n", 343 | "assert np.allclose(pred_decoded, pred_res)" 344 | ] 345 | }, 346 | { 347 | "cell_type": "code", 348 | "execution_count": null, 349 | "metadata": {}, 350 | "outputs": [], 351 | "source": [ 352 | "plt.imshow(pred_decoded);" 353 | ] 354 | }, 355 | { 356 | "cell_type": "code", 357 | "execution_count": null, 358 | "metadata": {}, 359 | "outputs": [], 360 | "source": [ 361 | "np.all(pred_fastai[0].numpy() == pred_res)" 362 | ] 363 | } 364 | ], 365 | "metadata": { 366 | "kernelspec": { 367 | "display_name": "Python 3", 368 | "language": "python", 369 | "name": "python3" 370 | }, 371 | "language_info": { 372 | "codemirror_mode": { 373 | "name": "ipython", 374 | "version": 3 375 | }, 376 | "file_extension": ".py", 377 | "mimetype": "text/x-python", 378 | "name": "python", 379 | "nbconvert_exporter": "python", 380 | "pygments_lexer": "ipython3", 381 | "version": "3.8.5" 382 | } 383 | }, 384 | "nbformat": 4, 385 | "nbformat_minor": 4 386 | } 387 | -------------------------------------------------------------------------------- /sample/sample_output.txt: -------------------------------------------------------------------------------- 1 | { 2 | "base64_prediction": "GhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoVFRUVFRUVFQQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhUVFRUVFRUVBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBBoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhUVFRUVFRUEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhUVFRUVBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhUVFRUEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBBoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhUVBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaFQQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBBoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoVFQQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhUVBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBBoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaFRUVBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoVFRUVBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhUVFRUVBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBBoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoVFRUVFRUEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoVFRUVFRUVFQQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaFRUVFRUVFRUVFRUVFRUVBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBBoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaFRUVFRUVFRUVFRUVFRUaBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhUVFRUVFRUVFRUVFRUVFRoEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhUVFRUVFRUVFRUVFRUVGgQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBBoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaFRUVFRUVFRUVFRUVFRUVBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaFRUVFRUVFRUVFRUVFRUEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoVFRUVFRUVFRUVFRUVFQQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBBoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoVFRUVFRUVFRUVFRUEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoVFRUVFRUVFQQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaFRUVFRUVBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBBoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaFRUVFRUEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoEGhoaGhoaGhoaGgQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaBAQEGhoaGhoaGhoaBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBBoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoEBAQEGhoaGhoaGhoEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoEBAQaGhoaGhoaBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGgQEBBoaGhoaGgQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBBoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBBoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGgQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBBoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGgQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBBoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGgQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBBoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGgQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBBoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaBAQEBAQEBBoaGhoaGhoaGhoaBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQeGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGgQEBAQEBAQEBAQaGhoaGhoaGhoEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQeHh4aGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoEBAQEBAQEBAQEBAQaGhoaGhoaBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBB4eHh4EGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGgQEBAQEBAQEBAQEBAQaGhoaGhoEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEHh4eHh4EGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQeHh4eHh4EBBoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGgQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBB4eHh4eHh4EGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoEGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBB4eHh4eHh4eBAQEBAQEBAQEBAQEBAQEBAQEHh4eHh4eHgQEGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGgQEGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEHh4eHh4eHh4eHh4eBAQEBAQEBAQEBAQEBAQeHh4eHh4eHgQaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoEBAQaGhoEBAQaGhoaBAQEBAQEGhoaGhoEGgQaBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQeHh4eHh4eHh4eHh4EBAQEBAQEBAQEBAQEHh4eHh4eHh4eBAQaGhoaGhoaGhoaGhoaGhoaGhoaGhoaHgQEBAQeHh4EBAQaGgQEBAQEBAQEGhoaBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEHh4eHh4eHh4eHh4eHgQEBAQEBAQEBAQEBAQeHh4eHh4eHh4eBBoaGhoaGhoaGhoaGhoaGhoaGhoaHh4eBAQEGB4eHgQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBB4eHh4eHh4eHh4eHh4eBAQEBAQEBAQEBAQEHh4eHh4eHh4eHh4EGhoaGhoaGhoaGhoaGhoaGhoEGh4eHh4eBB4YHh4eBAQEHh4EBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEHh4eHh4eHh4eHh4eHh4EBAQEBAQEBAQEBAQeHh4eHh4eHh4eHh4EGhoaGhoaGhoaGhoaGhoaBAQeHh4eHh4eHh4eHh4EBAQEHgQEBB4EBAQEHh4EBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQeHh4eHh4eHgQEBAQEBAQEBAQEBAQEBAQEBB4eHh4eHh4eHh4eHgQEGhoaGhoaGhoaGhoaGgQEBB4eHh4eHh4eHh4eHgQEBB4eHh4eHh4eHh4eHh4EBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBB4eHh4eHh4eBAQEBAQEBAQEBAQEBAQEBAQEHh4eHh4eHh4eHh4eHgQEBBoaGhoaGhoaGhoEBAQEHh4eHh4eHh4eHh4eBAQEHh4eHh4eHh4eHh4eHgQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEHh4eHh4eBAQEBAQEBAQEBAQEBAQEBAQEBAQeHh4eHh4eHh4eHh4eHgQEBAQaGhoaGhoEBAQEBAQeHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBB4eHh4eHh4EBAQEBAQEBAQEBAQEBAQEBAQEBB4eHh4eHh4eHh4eHh4eBAQEBAQEBAQEBAQEBAQEBB4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEHh4eHh4eHh4EBAQEBAQEBAQEBAQEBAQEBAQEHh4eHh4eHh4eHh4eHh4eBAQEBAQEBAQEBAQEBAQeHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eBAQEBAQEBAQEBQUFBQUFBQQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBB4eHh4eHh4eBAQEBAQEBAQEBAQEBAQEBAQEBAQeHh4eHh4eHh4eHh4eHh4EBAQEBAQEBAQEBAQEBB4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eBAQEHh4FBQUFBQUFBQUFBQUFBQQEBAQEBAQEBAQEBAUFBQUFBQUFHh4eHh4eHh4eBAQEBAQEBAQEBAQEBAQEBAQEBB4eHh4eHh4eHh4eHh4eHh4EBAQEBAQEBAQEBAQEHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eBQUFBQUFBQUFBQUFBQUFBQUFBAQFBAQFBQUFBQUFBQUFBQUeHh4eHh4eHh4eBAQEBAQEBAQEBAQEBAQEBAQEHh4eHh4eHh4eHh4eHh4eHgUFBQUFBQUFBQUFBQUFBR4eHh4eHgUeHh4eBQUFBR4eHh4eHh4eHh4eHh4eHh4eHgUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFFhYWFhYWEBAeHh4eHh4EBAQEBAQEBAQEBAQEBAQEBAQeHh4eHh4eHh4eHh4eHh4FBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBR4eHh4eHh4eHh4eHh4eHh4eBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBRYWFhYWFhYQEBAeHh4eHh4EBAQEBAQEBAQEBAQEBAQEBB4eHh4eHh4eHh4eHh4FBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBR4eHh4eHh4eHh4eHh4eHh4FBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBRYWFhYWFhAQEB4eHh4eBAQEBAQEBAQEBAQEBAQEBAQEHh4eHh4eHh4eHh4eBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBR4eHh4eHh4eExMTExMTExMFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFFhYWFhYWEBAQHh4eHh4EBAQEBAQEBAQEBAQEBAQEBAQeHh4eHh4eHh4FBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBR4TExMTExMTExMTExEREREREQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBRYWFhYWFhYWFgIQEBAeHh4eHgQEBAQEBAQEBAQEBAQEBAQEBB4eHh4eHh4eBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFExMTExMTEREREREREREREREREREFBQUFBQUFBQUFBQUFBQUFBQUFBQUFFhYWFhYWFhYCAhAQHh4eHh4eBAQEBAQEBAQEBAQEBAQEBAQEHh4eHh4FBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQURERERERERERERERERERERERERERERBQUFBQUFBQUFBQUFBQUFBQUFBQUWFhYWFhYWFgIQEBAeHh4eHh4EBAQEBAQEBAQEBAQEBAQEBAQeBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFERERERERERERERERERERERERERERERERBQUFBQUFBQUFBQUFBQUFBQUFBRYWFhYWFhYCAh4eHh4eHh4eHgQEBAQEBAQEBAQEBAQEBAQEBAUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBREREREREREREREREREREREREREREREREREREREFBQUFBQUFBQUFBQUFBQUFBQUWBRYWHh4CHh4eHh4eHh4eHgQEBAQEBAQEBAQEBAQEBAQEBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFEREREREREREREREREREREREREREREREREREREREREQUFBQUFBQUFBQUFBQUFBQUFBR4eHh4eHh4eBQUFBQUeHh4EBAQEBAQEBAQEBAQEBAQFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFERERERERERERERERERERERERERERERERERERERERERERBQUFBQUFBQUFBQUFBQUFBQUFBQUCAgUFBQUFBQUFBQUFHh4eBAQEBAQEBAQEBAQEBAUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBREREREFBQUFBQUFBQUFBREREREREREREREREREREREREREREREREREREREREREREREFBQUFBQUFBQUFBQUFBQUFBQUFBQUWBQUFBQUFBQUFBQUFBQUFBAQEBAQEBAQEBBAEBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREQUFBQUFBQUFBQUFBQUFBQUFBQUFBRYFBQUFBQUFBQUFBQUFBQUFBQUFBQUeHh4eBAQFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERBQUFBQUFBQUFBQUFBQUFBQUFFhYWBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREQUFBQUFBQUFBQUFBQUFBQUWFgUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFEREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREQUFBQUFBQUFBQUFBQUFBRYWFgUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREQUFBQUFBQUFBQUFBQUFFhYWFgUWFhYFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFEREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREFBQUFBQUFBQUFBQUWFhYWFgICFgUFBQUFBQUFFhYWFhYWBQUFBQUFBQUREREREREREQUFBQUFBQUFBQUFBQUFBRERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERBQUFBQUFBQUFBRECAgICAgICBQUFBQUFFhYWFhYWFhYWBQUFBQUFBREREREREREREREREREREREREQUFBRERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERECAgICAgIFBQUFBRYWFhYWFhYWFhYWFhYFBQUFEREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREQICAgICAgUFBQUFBRYWFhYWFhYCAhYWFgUFBQURERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERAgICAgIFBQUFBQUWFhYWFhYWAgICAhYWBQUFBREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREQIFBQUFBRYWFh4eHh4eHh4eBAQEBQUEEREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREQUFBQUFBR4eHh4eHh4eHh4eHgQEBAQREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREQUFBR4eHh4eHh4eHh4eHh4eHh4eBBEREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREeHh4eHh4eHh4eHh4eHh4eHh4EERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERER4eHh4eHh4eHh4eHh4eHh4eHh4RERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERHh4eHh4eHh4eHh4eHh4eHhERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERER4eHh4eHh4eHh4eHh4eHh4e" 3 | } -------------------------------------------------------------------------------- /notebook/04_SageMaker.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "%reload_ext autoreload\n", 10 | "%autoreload 2\n", 11 | "\n", 12 | "\n", 13 | "%matplotlib inline" 14 | ] 15 | }, 16 | { 17 | "cell_type": "markdown", 18 | "metadata": {}, 19 | "source": [ 20 | "# Amazon SageMaker" 21 | ] 22 | }, 23 | { 24 | "cell_type": "code", 25 | "execution_count": null, 26 | "metadata": {}, 27 | "outputs": [], 28 | "source": [ 29 | "import base64\n", 30 | "import json\n", 31 | "import numpy as np\n", 32 | "import matplotlib.pyplot as plt\n", 33 | "from PIL import Image" 34 | ] 35 | }, 36 | { 37 | "cell_type": "markdown", 38 | "metadata": {}, 39 | "source": [ 40 | "## Boilerplate" 41 | ] 42 | }, 43 | { 44 | "cell_type": "markdown", 45 | "metadata": {}, 46 | "source": [ 47 | "### Session" 48 | ] 49 | }, 50 | { 51 | "cell_type": "code", 52 | "execution_count": null, 53 | "metadata": {}, 54 | "outputs": [], 55 | "source": [ 56 | "import boto3, time, json\n", 57 | "\n", 58 | "sess = boto3.Session()\n", 59 | "sm = sess.client(\"sagemaker\")\n", 60 | "region = sess.region_name\n", 61 | "account = boto3.client(\"sts\").get_caller_identity().get(\"Account\")" 62 | ] 63 | }, 64 | { 65 | "cell_type": "markdown", 66 | "metadata": {}, 67 | "source": [ 68 | "### IAM Role\n", 69 | "\n", 70 | "**Note**: make sure the IAM role has: \n", 71 | "- `AmazonS3FullAccess` \n", 72 | "- `AmazonEC2ContainerRegistryFullAccess` \n", 73 | "- `AmazonSageMakerFullAccess` " 74 | ] 75 | }, 76 | { 77 | "cell_type": "code", 78 | "execution_count": null, 79 | "metadata": {}, 80 | "outputs": [], 81 | "source": [ 82 | "import sagemaker\n", 83 | "\n", 84 | "role = sagemaker.get_execution_role()\n", 85 | "role" 86 | ] 87 | }, 88 | { 89 | "cell_type": "markdown", 90 | "metadata": {}, 91 | "source": [ 92 | "### Amazon Elastic Container Registry (ECR)\n", 93 | "\n", 94 | "**Note**: create ECR if it doesn't exist" 95 | ] 96 | }, 97 | { 98 | "cell_type": "code", 99 | "execution_count": null, 100 | "metadata": {}, 101 | "outputs": [], 102 | "source": [ 103 | "registry_name = \"fastai-torchserve-sagemaker\"\n", 104 | "# !aws ecr create-repository --repository-name {registry_name}" 105 | ] 106 | }, 107 | { 108 | "cell_type": "code", 109 | "execution_count": null, 110 | "metadata": {}, 111 | "outputs": [], 112 | "source": [ 113 | "image = f\"{account}.dkr.ecr.{region}.amazonaws.com/{registry_name}:latest\"\n", 114 | "image" 115 | ] 116 | }, 117 | { 118 | "cell_type": "markdown", 119 | "metadata": {}, 120 | "source": [ 121 | "### Pytorch Model Artifact\n", 122 | "\n", 123 | "Create a compressed `*.tar.gz` file from the `*.mar` file per requirement of Amazon SageMaker and upload the model to your Amazon S3 bucket." 124 | ] 125 | }, 126 | { 127 | "cell_type": "code", 128 | "execution_count": null, 129 | "metadata": {}, 130 | "outputs": [], 131 | "source": [ 132 | "model_file_name = \"fastunet\"\n", 133 | "s3_bucket_name = \"\"\n", 134 | "# !tar cvzf {model_file_name}.tar.gz fastunet.mar\n", 135 | "# !aws s3 cp {model_file_name}.tar.gz s3://{s3_bucket_name}/" 136 | ] 137 | }, 138 | { 139 | "cell_type": "markdown", 140 | "metadata": {}, 141 | "source": [ 142 | "### Build a FastAI+TorchServe Docker container and push it to Amazon ECR" 143 | ] 144 | }, 145 | { 146 | "cell_type": "code", 147 | "execution_count": null, 148 | "metadata": {}, 149 | "outputs": [], 150 | "source": [ 151 | "!aws ecr get-login-password --region {region} | docker login --username AWS --password-stdin {account}.dkr.ecr.{region}.amazonaws.com\n", 152 | "!docker build -t {registry_name} ../\n", 153 | "!docker tag {registry_name}:latest {image}\n", 154 | "!docker push {image}" 155 | ] 156 | }, 157 | { 158 | "cell_type": "markdown", 159 | "metadata": {}, 160 | "source": [ 161 | "### Model" 162 | ] 163 | }, 164 | { 165 | "cell_type": "code", 166 | "execution_count": null, 167 | "metadata": {}, 168 | "outputs": [], 169 | "source": [ 170 | "model_data = f\"s3://{s3_bucket_name}/{model_file_name}.tar.gz\"\n", 171 | "sm_model_name = \"fastai-unet-torchserve-sagemaker\"\n", 172 | "\n", 173 | "container = {\"Image\": image, \"ModelDataUrl\": model_data}\n", 174 | "\n", 175 | "create_model_response = sm.create_model(\n", 176 | " ModelName=sm_model_name, ExecutionRoleArn=role, PrimaryContainer=container\n", 177 | ")\n", 178 | "\n", 179 | "print(create_model_response[\"ModelArn\"])" 180 | ] 181 | }, 182 | { 183 | "cell_type": "markdown", 184 | "metadata": {}, 185 | "source": [ 186 | "## Batch Transform" 187 | ] 188 | }, 189 | { 190 | "cell_type": "markdown", 191 | "metadata": {}, 192 | "source": [ 193 | "### S3 Input and Output" 194 | ] 195 | }, 196 | { 197 | "cell_type": "code", 198 | "execution_count": null, 199 | "metadata": {}, 200 | "outputs": [], 201 | "source": [ 202 | "batch_input = f\"s3://{s3_bucket_name}/batch_transform_fastai_torchserve_sagemaker/\"\n", 203 | "batch_output = f\"s3://{s3_bucket_name}/batch_transform_fastai_torchserve_sagemaker_output/\"" 204 | ] 205 | }, 206 | { 207 | "cell_type": "code", 208 | "execution_count": null, 209 | "metadata": {}, 210 | "outputs": [], 211 | "source": [ 212 | "!aws s3 ls {batch_input}" 213 | ] 214 | }, 215 | { 216 | "cell_type": "code", 217 | "execution_count": null, 218 | "metadata": {}, 219 | "outputs": [], 220 | "source": [ 221 | "import time\n", 222 | "\n", 223 | "batch_job_name = 'fastunet-batch' + time.strftime(\"%Y-%m-%d-%H-%M-%S\", time.gmtime())\n", 224 | "batch_job_name" 225 | ] 226 | }, 227 | { 228 | "cell_type": "markdown", 229 | "metadata": {}, 230 | "source": [ 231 | "### Batch transform jobs" 232 | ] 233 | }, 234 | { 235 | "cell_type": "code", 236 | "execution_count": null, 237 | "metadata": {}, 238 | "outputs": [], 239 | "source": [ 240 | "request = {\n", 241 | " \"ModelClientConfig\": {\n", 242 | " \"InvocationsTimeoutInSeconds\": 3600,\n", 243 | " \"InvocationsMaxRetries\": 1,\n", 244 | " },\n", 245 | " \"TransformJobName\": batch_job_name,\n", 246 | " \"ModelName\": sm_model_name,\n", 247 | " \"BatchStrategy\": \"MultiRecord\",\n", 248 | " \"TransformOutput\": {\"S3OutputPath\": batch_output, \"AssembleWith\": \"Line\"},\n", 249 | " \"TransformInput\": {\n", 250 | " \"DataSource\": {\n", 251 | " \"S3DataSource\": {\"S3DataType\": \"S3Prefix\", \"S3Uri\": batch_input}\n", 252 | " },\n", 253 | " \"CompressionType\": \"None\",\n", 254 | " },\n", 255 | " \"TransformResources\": {\"InstanceType\": \"ml.p2.xlarge\", \"InstanceCount\": 1},\n", 256 | "}" 257 | ] 258 | }, 259 | { 260 | "cell_type": "code", 261 | "execution_count": null, 262 | "metadata": {}, 263 | "outputs": [], 264 | "source": [ 265 | "%%time\n", 266 | "sm.create_transform_job(**request)\n", 267 | "\n", 268 | "while True:\n", 269 | " response = sm.describe_transform_job(TransformJobName=batch_job_name)\n", 270 | " status = response[\"TransformJobStatus\"]\n", 271 | " if status == \"Completed\":\n", 272 | " print(\"Transform job ended with status: \" + status)\n", 273 | " break\n", 274 | " if status == \"Failed\":\n", 275 | " message = response[\"FailureReason\"]\n", 276 | " print(\"Transform failed with the following error: {}\".format(message))\n", 277 | " raise Exception(\"Transform job failed\")\n", 278 | " print(\"Transform job is still in status: \" + status)\n", 279 | " time.sleep(30)" 280 | ] 281 | }, 282 | { 283 | "cell_type": "markdown", 284 | "metadata": {}, 285 | "source": [ 286 | "### Testing" 287 | ] 288 | }, 289 | { 290 | "cell_type": "code", 291 | "execution_count": null, 292 | "metadata": {}, 293 | "outputs": [], 294 | "source": [ 295 | "s3 = boto3.resource(\"s3\")\n", 296 | "s3.Bucket(f\"{s3_bucket_name}\").download_file(\n", 297 | " \"batch_transform_fastai_torchserve_sagemaker_output/street_view_of_a_small_neighborhood.png.out\",\n", 298 | " \"street_view_of_a_small_neighborhood.txt\",\n", 299 | ")\n", 300 | "s3.Bucket(f\"{s3_bucket_name}\").download_file(\n", 301 | " \"batch_transform_fastai_torchserve_sagemaker/street_view_of_a_small_neighborhood.png\",\n", 302 | " \"street_view_of_a_small_neighborhood.png\",\n", 303 | ")" 304 | ] 305 | }, 306 | { 307 | "cell_type": "code", 308 | "execution_count": null, 309 | "metadata": {}, 310 | "outputs": [], 311 | "source": [ 312 | "with open(\"street_view_of_a_small_neighborhood.txt\") as f:\n", 313 | " results = f.read()\n", 314 | "\n", 315 | "response = json.loads(results)" 316 | ] 317 | }, 318 | { 319 | "cell_type": "code", 320 | "execution_count": null, 321 | "metadata": {}, 322 | "outputs": [], 323 | "source": [ 324 | "pred_decoded_byte = base64.decodebytes(bytes(response[\"base64_prediction\"], encoding=\"utf-8\"))\n", 325 | "pred_decoded = np.reshape(\n", 326 | " np.frombuffer(pred_decoded_byte, dtype=np.uint8), (96, 128)\n", 327 | ")\n", 328 | "plt.imshow(pred_decoded);" 329 | ] 330 | }, 331 | { 332 | "cell_type": "markdown", 333 | "metadata": {}, 334 | "source": [ 335 | "## Inference Endpoint" 336 | ] 337 | }, 338 | { 339 | "cell_type": "markdown", 340 | "metadata": {}, 341 | "source": [ 342 | "### Endpoint configuration\n", 343 | "\n", 344 | "**Note**: choose your preferred `InstanceType`: https://aws.amazon.com/sagemaker/pricing/" 345 | ] 346 | }, 347 | { 348 | "cell_type": "code", 349 | "execution_count": null, 350 | "metadata": {}, 351 | "outputs": [], 352 | "source": [ 353 | "import time\n", 354 | "\n", 355 | "endpoint_config_name = \"torchserve-endpoint-config-\" + time.strftime(\n", 356 | " \"%Y-%m-%d-%H-%M-%S\", time.gmtime()\n", 357 | ")\n", 358 | "print(endpoint_config_name)\n", 359 | "\n", 360 | "create_endpoint_config_response = sm.create_endpoint_config(\n", 361 | " EndpointConfigName=endpoint_config_name,\n", 362 | " ProductionVariants=[\n", 363 | " {\n", 364 | " \"InstanceType\": \"ml.g4dn.xlarge\",\n", 365 | " \"InitialVariantWeight\": 1,\n", 366 | " \"InitialInstanceCount\": 1,\n", 367 | " \"ModelName\": sm_model_name,\n", 368 | " \"VariantName\": \"AllTraffic\",\n", 369 | " }\n", 370 | " ],\n", 371 | ")\n", 372 | "\n", 373 | "print(\"Endpoint Config Arn: \" + create_endpoint_config_response[\"EndpointConfigArn\"])" 374 | ] 375 | }, 376 | { 377 | "cell_type": "markdown", 378 | "metadata": {}, 379 | "source": [ 380 | "### Endpoint" 381 | ] 382 | }, 383 | { 384 | "cell_type": "code", 385 | "execution_count": null, 386 | "metadata": {}, 387 | "outputs": [], 388 | "source": [ 389 | "endpoint_name = \"fastunet-torchserve-endpoint-\" + time.strftime(\n", 390 | " \"%Y-%m-%d-%H-%M-%S\", time.gmtime()\n", 391 | ")\n", 392 | "print(endpoint_name)\n", 393 | "\n", 394 | "create_endpoint_response = sm.create_endpoint(\n", 395 | " EndpointName=endpoint_name, EndpointConfigName=endpoint_config_name\n", 396 | ")\n", 397 | "print(create_endpoint_response[\"EndpointArn\"])" 398 | ] 399 | }, 400 | { 401 | "cell_type": "code", 402 | "execution_count": null, 403 | "metadata": {}, 404 | "outputs": [], 405 | "source": [ 406 | "%%time\n", 407 | "resp = sm.describe_endpoint(EndpointName=endpoint_name)\n", 408 | "status = resp[\"EndpointStatus\"]\n", 409 | "print(\"Status: \" + status)\n", 410 | "\n", 411 | "while status == \"Creating\":\n", 412 | " time.sleep(60)\n", 413 | " resp = sm.describe_endpoint(EndpointName=endpoint_name)\n", 414 | " status = resp[\"EndpointStatus\"]\n", 415 | " print(\"Status: \" + status)\n", 416 | "\n", 417 | "print(\"Arn: \" + resp[\"EndpointArn\"])\n", 418 | "print(\"Status: \" + status)" 419 | ] 420 | }, 421 | { 422 | "cell_type": "markdown", 423 | "metadata": {}, 424 | "source": [ 425 | "### Testing" 426 | ] 427 | }, 428 | { 429 | "cell_type": "code", 430 | "execution_count": null, 431 | "metadata": {}, 432 | "outputs": [], 433 | "source": [ 434 | "file_name = \"../sample/street_view_of_a_small_neighborhood.png\"\n", 435 | "\n", 436 | "with open(file_name, 'rb') as f:\n", 437 | " payload = f.read()\n", 438 | " \n", 439 | "Image.open(file_name)" 440 | ] 441 | }, 442 | { 443 | "cell_type": "code", 444 | "execution_count": null, 445 | "metadata": {}, 446 | "outputs": [], 447 | "source": [ 448 | "%%time\n", 449 | "client = boto3.client(\"runtime.sagemaker\")\n", 450 | "response = client.invoke_endpoint(\n", 451 | " EndpointName=endpoint_name, ContentType=\"application/x-image\", Body=payload\n", 452 | ")\n", 453 | "response = json.loads(response[\"Body\"].read())" 454 | ] 455 | }, 456 | { 457 | "cell_type": "code", 458 | "execution_count": null, 459 | "metadata": {}, 460 | "outputs": [], 461 | "source": [ 462 | "pred_decoded_byte = base64.decodebytes(bytes(response[\"base64_prediction\"], encoding=\"utf-8\"))\n", 463 | "pred_decoded = np.reshape(\n", 464 | " np.frombuffer(pred_decoded_byte, dtype=np.uint8), (96, 128)\n", 465 | ")\n", 466 | "plt.imshow(pred_decoded);" 467 | ] 468 | }, 469 | { 470 | "cell_type": "markdown", 471 | "metadata": {}, 472 | "source": [ 473 | "### Cleanup" 474 | ] 475 | }, 476 | { 477 | "cell_type": "code", 478 | "execution_count": null, 479 | "metadata": {}, 480 | "outputs": [], 481 | "source": [ 482 | "client = boto3.client(\"sagemaker\")\n", 483 | "client.delete_model(ModelName=sm_model_name)\n", 484 | "client.delete_endpoint(EndpointName=endpoint_name)\n", 485 | "client.delete_endpoint_config(EndpointConfigName=endpoint_config_name)" 486 | ] 487 | } 488 | ], 489 | "metadata": { 490 | "kernelspec": { 491 | "display_name": "Python 3", 492 | "language": "python", 493 | "name": "python3" 494 | }, 495 | "language_info": { 496 | "codemirror_mode": { 497 | "name": "ipython", 498 | "version": 3 499 | }, 500 | "file_extension": ".py", 501 | "mimetype": "text/x-python", 502 | "name": "python", 503 | "nbconvert_exporter": "python", 504 | "pygments_lexer": "ipython3", 505 | "version": "3.8.5" 506 | }, 507 | "toc": { 508 | "base_numbering": 1, 509 | "nav_menu": {}, 510 | "number_sections": true, 511 | "sideBar": true, 512 | "skip_h1_title": false, 513 | "title_cell": "Table of Contents", 514 | "title_sidebar": "Contents", 515 | "toc_cell": false, 516 | "toc_position": {}, 517 | "toc_section_display": true, 518 | "toc_window_display": false 519 | }, 520 | "varInspector": { 521 | "cols": { 522 | "lenName": 16, 523 | "lenType": 16, 524 | "lenVar": 40 525 | }, 526 | "kernels_config": { 527 | "python": { 528 | "delete_cmd_postfix": "", 529 | "delete_cmd_prefix": "del ", 530 | "library": "var_list.py", 531 | "varRefreshCmd": "print(var_dic_list())" 532 | }, 533 | "r": { 534 | "delete_cmd_postfix": ") ", 535 | "delete_cmd_prefix": "rm(", 536 | "library": "var_list.r", 537 | "varRefreshCmd": "cat(var_dic_list()) " 538 | } 539 | }, 540 | "types_to_exclude": [ 541 | "module", 542 | "function", 543 | "builtin_function_or_method", 544 | "instance", 545 | "_Feature" 546 | ], 547 | "window_display": false 548 | } 549 | }, 550 | "nbformat": 4, 551 | "nbformat_minor": 4 552 | } 553 | -------------------------------------------------------------------------------- /notebook/03_TorchServe.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "%reload_ext autoreload\n", 10 | "%autoreload 2\n", 11 | "\n", 12 | "\n", 13 | "%matplotlib inline" 14 | ] 15 | }, 16 | { 17 | "cell_type": "markdown", 18 | "metadata": {}, 19 | "source": [ 20 | "# TorchServe" 21 | ] 22 | }, 23 | { 24 | "cell_type": "markdown", 25 | "metadata": {}, 26 | "source": [ 27 | "For installation, please refer to: \n", 28 | "https://github.com/pytorch/serve" 29 | ] 30 | }, 31 | { 32 | "cell_type": "markdown", 33 | "metadata": {}, 34 | "source": [ 35 | "### model.py" 36 | ] 37 | }, 38 | { 39 | "cell_type": "code", 40 | "execution_count": null, 41 | "metadata": {}, 42 | "outputs": [], 43 | "source": [ 44 | "!pygmentize ../deployment/model.py" 45 | ] 46 | }, 47 | { 48 | "cell_type": "markdown", 49 | "metadata": {}, 50 | "source": [ 51 | "### handler.py\n", 52 | "\n", 53 | "Reference: \n", 54 | "- https://github.com/pytorch/serve/tree/master/ts/torch_handler" 55 | ] 56 | }, 57 | { 58 | "cell_type": "code", 59 | "execution_count": null, 60 | "metadata": {}, 61 | "outputs": [], 62 | "source": [ 63 | "!pygmentize ../deployment/handler.py" 64 | ] 65 | }, 66 | { 67 | "cell_type": "markdown", 68 | "metadata": {}, 69 | "source": [ 70 | "### Archive the Model\n", 71 | "\n", 72 | "```bash\n", 73 | "torch-model-archiver --model-name fastunet --version 1.0 --model-file deployment/model.py --serialized-file model_store/fasti_unet_weights.pth --export-path model_store --handler deployment/handler.py -f\n", 74 | "```" 75 | ] 76 | }, 77 | { 78 | "cell_type": "markdown", 79 | "metadata": {}, 80 | "source": [ 81 | "### Serve the Model\n", 82 | "\n", 83 | "```bash\n", 84 | "torchserve --start --ncs --model-store model_store --models fastunet.mar\n", 85 | "```" 86 | ] 87 | }, 88 | { 89 | "cell_type": "markdown", 90 | "metadata": {}, 91 | "source": [ 92 | "### Call API\n", 93 | "\n", 94 | "```bash\n", 95 | "time http POST http://127.0.0.1:8080/predictions/fastunet/ @sample/street_view_of_a_small_neighborhood.png\n", 96 | "```" 97 | ] 98 | }, 99 | { 100 | "cell_type": "markdown", 101 | "metadata": {}, 102 | "source": [ 103 | "### Sample Response\n", 104 | "\n", 105 | "```bash\n", 106 | "HTTP/1.1 200\n", 107 | "Cache-Control: no-cache; no-store, must-revalidate, private\n", 108 | "Expires: Thu, 01 Jan 1970 00:00:00 UTC\n", 109 | "Pragma: no-cache\n", 110 | "connection: keep-alive\n", 111 | "content-length: 16413\n", 112 | "x-request-id: 7491c48f-4b12-483b-a5a9-92f0fbbdaee8\n", 113 | "\n", 114 | "{\n", 115 | " \"base64_prediction\": \"GhoaGhoaGhoaGhoaGhoaGhoaGh...RERERERER4eHh4eHh4eHh4eHh4eHh4e\"\n", 116 | "}\n", 117 | "\n", 118 | "real 0m0.374s\n", 119 | "user 0m0.298s\n", 120 | "sys 0m0.030s\n", 121 | "```" 122 | ] 123 | }, 124 | { 125 | "cell_type": "markdown", 126 | "metadata": {}, 127 | "source": [ 128 | "### Decode and Visualise Returned Prediction from TorchServe" 129 | ] 130 | }, 131 | { 132 | "cell_type": "code", 133 | "execution_count": null, 134 | "metadata": {}, 135 | "outputs": [], 136 | "source": [ 137 | "response = {\n", 138 | " \"base64_prediction\": \"GhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoVFRUVFRUVFQQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhUVFRUVFRUVBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBBoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhUVFRUVFRUEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhUVFRUVBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhUVFRUEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBBoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhUVBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaFQQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBBoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoVFQQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhUVBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBBoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaFRUVBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoVFRUVBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhUVFRUVBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBBoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoVFRUVFRUEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoVFRUVFRUVFQQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaFRUVFRUVFRUVFRUVFRUVBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBBoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaFRUVFRUVFRUVFRUVFRUaBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhUVFRUVFRUVFRUVFRUVFRoEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhUVFRUVFRUVFRUVFRUVGgQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBBoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaFRUVFRUVFRUVFRUVFRUVBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaFRUVFRUVFRUVFRUVFRUEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoVFRUVFRUVFRUVFRUVFQQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBBoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoVFRUVFRUVFRUVFRUEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoVFRUVFRUVFQQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaFRUVFRUVBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBBoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaFRUVFRUEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoEGhoaGhoaGhoaGgQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaBAQEGhoaGhoaGhoaBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBBoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoEBAQEGhoaGhoaGhoEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoEBAQaGhoaGhoaBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGgQEBBoaGhoaGgQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBBoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBBoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGgQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBBoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGgQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBBoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGgQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBBoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGgQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBBoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaBAQEBAQEBBoaGhoaGhoaGhoaBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQeGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGgQEBAQEBAQEBAQaGhoaGhoaGhoEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQeHh4aGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoEBAQEBAQEBAQEBAQaGhoaGhoaBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBB4eHh4EGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGgQEBAQEBAQEBAQEBAQaGhoaGhoEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEHh4eHh4EGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQeHh4eHh4EBBoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGgQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBB4eHh4eHh4EGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoEGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBB4eHh4eHh4eBAQEBAQEBAQEBAQEBAQEBAQEHh4eHh4eHgQEGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGgQEGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEHh4eHh4eHh4eHh4eBAQEBAQEBAQEBAQEBAQeHh4eHh4eHgQaGhoaGhoaGhoaGhoaGhoaGhoaGhoaGhoEBAQaGhoEBAQaGhoaBAQEBAQEGhoaGhoEGgQaBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQeHh4eHh4eHh4eHh4EBAQEBAQEBAQEBAQEHh4eHh4eHh4eBAQaGhoaGhoaGhoaGhoaGhoaGhoaGhoaHgQEBAQeHh4EBAQaGgQEBAQEBAQEGhoaBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEHh4eHh4eHh4eHh4eHgQEBAQEBAQEBAQEBAQeHh4eHh4eHh4eBBoaGhoaGhoaGhoaGhoaGhoaGhoaHh4eBAQEGB4eHgQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBB4eHh4eHh4eHh4eHh4eBAQEBAQEBAQEBAQEHh4eHh4eHh4eHh4EGhoaGhoaGhoaGhoaGhoaGhoEGh4eHh4eBB4YHh4eBAQEHh4EBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEHh4eHh4eHh4eHh4eHh4EBAQEBAQEBAQEBAQeHh4eHh4eHh4eHh4EGhoaGhoaGhoaGhoaGhoaBAQeHh4eHh4eHh4eHh4EBAQEHgQEBB4EBAQEHh4EBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQeHh4eHh4eHgQEBAQEBAQEBAQEBAQEBAQEBB4eHh4eHh4eHh4eHgQEGhoaGhoaGhoaGhoaGgQEBB4eHh4eHh4eHh4eHgQEBB4eHh4eHh4eHh4eHh4EBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBB4eHh4eHh4eBAQEBAQEBAQEBAQEBAQEBAQEHh4eHh4eHh4eHh4eHgQEBBoaGhoaGhoaGhoEBAQEHh4eHh4eHh4eHh4eBAQEHh4eHh4eHh4eHh4eHgQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEHh4eHh4eBAQEBAQEBAQEBAQEBAQEBAQEBAQeHh4eHh4eHh4eHh4eHgQEBAQaGhoaGhoEBAQEBAQeHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBB4eHh4eHh4EBAQEBAQEBAQEBAQEBAQEBAQEBB4eHh4eHh4eHh4eHh4eBAQEBAQEBAQEBAQEBAQEBB4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEHh4eHh4eHh4EBAQEBAQEBAQEBAQEBAQEBAQEHh4eHh4eHh4eHh4eHh4eBAQEBAQEBAQEBAQEBAQeHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eBAQEBAQEBAQEBQUFBQUFBQQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBB4eHh4eHh4eBAQEBAQEBAQEBAQEBAQEBAQEBAQeHh4eHh4eHh4eHh4eHh4EBAQEBAQEBAQEBAQEBB4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eBAQEHh4FBQUFBQUFBQUFBQUFBQQEBAQEBAQEBAQEBAUFBQUFBQUFHh4eHh4eHh4eBAQEBAQEBAQEBAQEBAQEBAQEBB4eHh4eHh4eHh4eHh4eHh4EBAQEBAQEBAQEBAQEHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eBQUFBQUFBQUFBQUFBQUFBQUFBAQFBAQFBQUFBQUFBQUFBQUeHh4eHh4eHh4eBAQEBAQEBAQEBAQEBAQEBAQEHh4eHh4eHh4eHh4eHh4eHgUFBQUFBQUFBQUFBQUFBR4eHh4eHgUeHh4eBQUFBR4eHh4eHh4eHh4eHh4eHh4eHgUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFFhYWFhYWEBAeHh4eHh4EBAQEBAQEBAQEBAQEBAQEBAQeHh4eHh4eHh4eHh4eHh4FBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBR4eHh4eHh4eHh4eHh4eHh4eBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBRYWFhYWFhYQEBAeHh4eHh4EBAQEBAQEBAQEBAQEBAQEBB4eHh4eHh4eHh4eHh4FBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBR4eHh4eHh4eHh4eHh4eHh4FBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBRYWFhYWFhAQEB4eHh4eBAQEBAQEBAQEBAQEBAQEBAQEHh4eHh4eHh4eHh4eBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBR4eHh4eHh4eExMTExMTExMFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFFhYWFhYWEBAQHh4eHh4EBAQEBAQEBAQEBAQEBAQEBAQeHh4eHh4eHh4FBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBR4TExMTExMTExMTExEREREREQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBRYWFhYWFhYWFgIQEBAeHh4eHgQEBAQEBAQEBAQEBAQEBAQEBB4eHh4eHh4eBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFExMTExMTEREREREREREREREREREFBQUFBQUFBQUFBQUFBQUFBQUFBQUFFhYWFhYWFhYCAhAQHh4eHh4eBAQEBAQEBAQEBAQEBAQEBAQEHh4eHh4FBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQURERERERERERERERERERERERERERERBQUFBQUFBQUFBQUFBQUFBQUFBQUWFhYWFhYWFgIQEBAeHh4eHh4EBAQEBAQEBAQEBAQEBAQEBAQeBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFERERERERERERERERERERERERERERERERBQUFBQUFBQUFBQUFBQUFBQUFBRYWFhYWFhYCAh4eHh4eHh4eHgQEBAQEBAQEBAQEBAQEBAQEBAUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBREREREREREREREREREREREREREREREREREREREFBQUFBQUFBQUFBQUFBQUFBQUWBRYWHh4CHh4eHh4eHh4eHgQEBAQEBAQEBAQEBAQEBAQEBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFEREREREREREREREREREREREREREREREREREREREREQUFBQUFBQUFBQUFBQUFBQUFBR4eHh4eHh4eBQUFBQUeHh4EBAQEBAQEBAQEBAQEBAQFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFERERERERERERERERERERERERERERERERERERERERERERBQUFBQUFBQUFBQUFBQUFBQUFBQUCAgUFBQUFBQUFBQUFHh4eBAQEBAQEBAQEBAQEBAUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBREREREFBQUFBQUFBQUFBREREREREREREREREREREREREREREREREREREREREREREREFBQUFBQUFBQUFBQUFBQUFBQUFBQUWBQUFBQUFBQUFBQUFBQUFBAQEBAQEBAQEBBAEBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREQUFBQUFBQUFBQUFBQUFBQUFBQUFBRYFBQUFBQUFBQUFBQUFBQUFBQUFBQUeHh4eBAQFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERBQUFBQUFBQUFBQUFBQUFBQUFFhYWBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREQUFBQUFBQUFBQUFBQUFBQUWFgUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFEREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREQUFBQUFBQUFBQUFBQUFBRYWFgUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREQUFBQUFBQUFBQUFBQUFFhYWFgUWFhYFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFEREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREFBQUFBQUFBQUFBQUWFhYWFgICFgUFBQUFBQUFFhYWFhYWBQUFBQUFBQUREREREREREQUFBQUFBQUFBQUFBQUFBRERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERBQUFBQUFBQUFBRECAgICAgICBQUFBQUFFhYWFhYWFhYWBQUFBQUFBREREREREREREREREREREREREQUFBRERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERECAgICAgIFBQUFBRYWFhYWFhYWFhYWFhYFBQUFEREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREQICAgICAgUFBQUFBRYWFhYWFhYCAhYWFgUFBQURERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERAgICAgIFBQUFBQUWFhYWFhYWAgICAhYWBQUFBREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREQIFBQUFBRYWFh4eHh4eHh4eBAQEBQUEEREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREQUFBQUFBR4eHh4eHh4eHh4eHgQEBAQREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREQUFBR4eHh4eHh4eHh4eHh4eHh4eBBEREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREREeHh4eHh4eHh4eHh4eHh4eHh4EERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERER4eHh4eHh4eHh4eHh4eHh4eHh4RERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERHh4eHh4eHh4eHh4eHh4eHhERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERER4eHh4eHh4eHh4eHh4eHh4e\"\n", 139 | "}" 140 | ] 141 | }, 142 | { 143 | "cell_type": "code", 144 | "execution_count": null, 145 | "metadata": {}, 146 | "outputs": [], 147 | "source": [ 148 | "import base64\n", 149 | "import numpy as np\n", 150 | "import matplotlib.pyplot as plt\n", 151 | "\n", 152 | "pred_decoded_byte = base64.decodebytes(bytes(response[\"base64_prediction\"], encoding=\"utf-8\"))\n", 153 | "pred_decoded = np.reshape(\n", 154 | " np.frombuffer(pred_decoded_byte, dtype=np.uint8), (96, 128)\n", 155 | ")\n", 156 | "plt.imshow(pred_decoded);" 157 | ] 158 | } 159 | ], 160 | "metadata": { 161 | "kernelspec": { 162 | "display_name": "Python 3", 163 | "language": "python", 164 | "name": "python3" 165 | }, 166 | "language_info": { 167 | "codemirror_mode": { 168 | "name": "ipython", 169 | "version": 3 170 | }, 171 | "file_extension": ".py", 172 | "mimetype": "text/x-python", 173 | "name": "python", 174 | "nbconvert_exporter": "python", 175 | "pygments_lexer": "ipython3", 176 | "version": "3.8.5" 177 | }, 178 | "toc": { 179 | "base_numbering": 1, 180 | "nav_menu": {}, 181 | "number_sections": true, 182 | "sideBar": true, 183 | "skip_h1_title": false, 184 | "title_cell": "Table of Contents", 185 | "title_sidebar": "Contents", 186 | "toc_cell": false, 187 | "toc_position": {}, 188 | "toc_section_display": true, 189 | "toc_window_display": false 190 | }, 191 | "varInspector": { 192 | "cols": { 193 | "lenName": 16, 194 | "lenType": 16, 195 | "lenVar": 40 196 | }, 197 | "kernels_config": { 198 | "python": { 199 | "delete_cmd_postfix": "", 200 | "delete_cmd_prefix": "del ", 201 | "library": "var_list.py", 202 | "varRefreshCmd": "print(var_dic_list())" 203 | }, 204 | "r": { 205 | "delete_cmd_postfix": ") ", 206 | "delete_cmd_prefix": "rm(", 207 | "library": "var_list.r", 208 | "varRefreshCmd": "cat(var_dic_list()) " 209 | } 210 | }, 211 | "types_to_exclude": [ 212 | "module", 213 | "function", 214 | "builtin_function_or_method", 215 | "instance", 216 | "_Feature" 217 | ], 218 | "window_display": false 219 | } 220 | }, 221 | "nbformat": 4, 222 | "nbformat_minor": 4 223 | } 224 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Deploy FastAI Trained PyTorch Model in TorchServe and Host in Amazon SageMaker Inference Endpoint 2 | 3 | - [Deploy FastAI Trained PyTorch Model in TorchServe and Host in Amazon SageMaker Inference Endpoint](#deploy-fastai-trained-pytorch-model-in-torchserve-and-host-in-amazon-sagemaker-inference-endpoint) 4 | - [Introduction](#introduction) 5 | - [Getting Started with A FastAI Model](#getting-started-with-a-fastai-model) 6 | - [Installation](#installation) 7 | - [Modelling](#modelling) 8 | - [PyTorch Transfer Modeling from FastAI](#pytorch-transfer-modeling-from-fastai) 9 | - [Export Model Weights from FastAI](#export-model-weights-from-fastai) 10 | - [PyTorch Model from FastAI Source Code](#pytorch-model-from-fastai-source-code) 11 | - [Weights Transfer](#weights-transfer) 12 | - [Deployment to TorchServe](#deployment-to-torchserve) 13 | - [Custom Handler](#custom-handler) 14 | - [`initialize`](#initialize) 15 | - [`preprocess`](#preprocess) 16 | - [`inference`](#inference) 17 | - [`postprocess`](#postprocess) 18 | - [TorchServe in Action](#torchserve-in-action) 19 | - [Deployment to Amazon SageMaker Inference Endpoint](#deployment-to-amazon-sagemaker-inference-endpoint) 20 | - [Getting Started with Amazon SageMaker Endpoint](#getting-started-with-amazon-sagemaker-endpoint) 21 | - [Real-time Inference with Python SDK](#real-time-inference-with-python-sdk) 22 | - [What's Next](#whats-next) 23 | - [Clean Up](#clean-up) 24 | - [Conclusion](#conclusion) 25 | - [Reference](#reference) 26 | 27 | ## Introduction 28 | 29 | Over the past few years, [FastAI](https://www.fast.ai/) has become one of the most cutting-edge open-source deep learning framework and the go-to choice for many machine learning use cases based on [PyTorch](https://pytorch.org/). It not only democratized deep learning and made it approachable to the general audiences, but also set as a role model on how scientific software shall be engineered, especially in Python programming. Currently, however, to deploy a FastAI model to production environment often involves setting up and self-maintaining a customized inference solution, e.g. with [Flask](https://flask.palletsprojects.com/en/1.1.x/), which is time-consuming and distracting to manage and maintain issues like security, load balancing, services orchestration, etc. 30 | 31 | Recently, AWS developed *[TorchServe](https://github.com/pytorch/serve)* in partnership with Facebook, which is a flexible and easy-to-use open-source tool for serving PyTorch models. It removes the heavy lifting of deploying and serving PyTorch models with Kubernetes, and AWS and Facebook will maintain and continue contributing to TorchServe along with the broader PyTorch community. With TorchServe, many features are out-of-the-box and they provide full flexibility of deploying trained PyTorch models at scale so that a trained model can go to production deployment with few extra lines of code. 32 | 33 | Meanwhile, Amazon SageMaker endpoint has been a fully managed service that allows users to make real-time inferences via a REST API, and save Data Scientists and Machine Learning Engineers from managing their own server instances, load balancing, fault-tolerance, auto-scaling and model monitoring, etc. Amazon SageMaker endpoint provides different type of instances suitable for different tasks, including ones with GPU(s), which supports industry level machine learning inference and graphics-intensive applications while being [cost-effective](https://aws.amazon.com/sagemaker/pricing/). 34 | 35 | In this repository we demonstrate how to deploy a FastAI trained PyTorch model in TorchServe eager mode and host it in Amazon SageMaker Inference endpoint. 36 | 37 | ## Getting Started with A FastAI Model 38 | 39 | In this section we train a FastAI model that can solve a real-world problem with performance meeting the use-case specification. As an example, we focus on a **Scene Segmentation** use case from self-driving car. 40 | 41 | ### Installation 42 | 43 | The first step is to install FastAI package, which is covered in its [Github](https://github.com/fastai/fastai) repository. 44 | 45 | > If you're using Anaconda then run: 46 | > ```python 47 | > conda install -c fastai -c pytorch -c anaconda fastai gh anaconda 48 | > ``` 49 | > ...or if you're using miniconda) then run: 50 | > ```python 51 | > conda install -c fastai -c pytorch fastai 52 | > ``` 53 | 54 | For other installation options, please refer to the FastAI documentation. 55 | 56 | ### Modelling 57 | 58 | The following materials are based on the FastAI course: "[Practical Deep Learning for Coders](https://course.fast.ai/)". 59 | 60 | First, import `fastai.vision` modules and download the sample data `CAMVID_TINY`, by: 61 | 62 | ```python 63 | from fastai.vision.all import * 64 | path = untar_data(URLs.CAMVID_TINY) 65 | ``` 66 | 67 | Secondly, define helper functions to calculate segmentation performance and read in segmentation mask for each training image. 68 | 69 | **Note**: it's tempting to define one-line python `lambda` functions to pass to fastai, however, this will introduce issue on serialization when we want to export a FastAI model. Therefore we avoid using anonymous python functions during FastAI modeling steps. 70 | 71 | ```python 72 | def acc_camvid(inp, targ, void_code=0): 73 | targ = targ.squeeze(1) 74 | mask = targ != void_code 75 | return (inp.argmax(dim=1)[mask] == targ[mask]).float().mean() 76 | 77 | def get_y(o, path=path): 78 | return path / "labels" / f"{o.stem}_P{o.suffix}" 79 | ``` 80 | 81 | Thirdly, we setup the `DataLoader` which defines modelling path, training image path, batch size, mask path, mask code, etc. In this example we also record the image size and number of classes from the data. In real-world problem their values may be known in priori and shall be defined when constructing the dataset. 82 | 83 | ```python 84 | dls = SegmentationDataLoaders.from_label_func( 85 | path, 86 | bs=8, 87 | fnames=get_image_files(path / "images"), 88 | label_func=get_y, 89 | codes=np.loadtxt(path / "codes.txt", dtype=str), 90 | ) 91 | dls.one_batch()[0].shape[-2:], get_c(dls) 92 | >>> (torch.Size([96, 128]), 32) 93 | ``` 94 | 95 | Next, setup an [U-Net](https://arxiv.org/abs/1505.04597) learner with a Residual Neural Network (ResNet) backbone, then trigger the FastAI training process. 96 | 97 | ```python 98 | learn = unet_learner(dls, resnet50, metrics=acc_camvid) 99 | learn.fine_tune(20) 100 | >>> 101 | epoch train_loss valid_loss acc_camvid time 102 | 0 3.901105 2.671725 0.419333 00:04 103 | epoch train_loss valid_loss acc_camvid time 104 | 0 1.732219 1.766196 0.589736 00:03 105 | 1 1.536345 1.550913 0.612496 00:02 106 | 2 1.416585 1.170476 0.650690 00:02 107 | 3 1.300092 1.087747 0.665566 00:02 108 | 4 1.334166 1.228493 0.649878 00:03 109 | 5 1.269190 1.047625 0.711870 00:02 110 | 6 1.243131 0.969567 0.719976 00:03 111 | 7 1.164861 0.988767 0.700076 00:03 112 | 8 1.103572 0.791861 0.787799 00:02 113 | 9 1.026181 0.721673 0.806758 00:02 114 | 10 0.949283 0.650206 0.815247 00:03 115 | 11 0.882919 0.696920 0.812805 00:03 116 | 12 0.823694 0.635109 0.824582 00:03 117 | 13 0.766428 0.631013 0.832627 00:02 118 | 14 0.715637 0.591066 0.839386 00:03 119 | 15 0.669535 0.601648 0.836554 00:03 120 | 16 0.628947 0.598065 0.840095 00:03 121 | 17 0.593876 0.578633 0.841116 00:02 122 | 18 0.563728 0.582522 0.841409 00:03 123 | 19 0.539064 0.580864 0.842272 00:02 124 | ``` 125 | 126 | Finally, we export the fastai model to use for following sections of this tutorial. 127 | 128 | ```python 129 | learn.export("./fastai_unet.pkl") 130 | ``` 131 | 132 | For more details about the modeling process, refer to `notebook/01_U-net_Modelling.ipynb` [[link](notebook/01_U-net_Modelling.ipynb)]. 133 | 134 | ## PyTorch Transfer Modeling from FastAI 135 | 136 | In this section we build a pure PyTorch model and transfer the model weights from FastAI. The following materials are inspired by "[Practical-Deep-Learning-for-Coders-2.0](https://github.com/muellerzr/Practical-Deep-Learning-for-Coders-2.0/blob/master/Computer%20Vision/06_Hybridizing_Models.ipynb)" by Zachary Mueller *et al*. 137 | 138 | ### Export Model Weights from FastAI 139 | 140 | First, restore the FastAI learner from the export pickle at the last Section, and save its model weights with PyTorch. 141 | 142 | ```python 143 | from fastai.vision.all import * 144 | import torch 145 | 146 | def acc_camvid(*_): pass 147 | def get_y(*_): pass 148 | 149 | learn = load_learner("/home/ubuntu/.fastai/data/camvid_tiny/fastai_unet.pkl") 150 | torch.save(learn.model.state_dict(), "fasti_unet_weights.pth") 151 | ``` 152 | 153 | It's also straightforward to obtain the FastAI prediction on a sample image. 154 | 155 | > "2013.04 - 'Streetview of a small neighborhood', with residential buildings, Amsterdam city photo by Fons Heijnsbroek, The Netherlands" by Amsterdam free photos & pictures of the Dutch city is marked under CC0 1.0. To view the terms, visit https://creativecommons.org/licenses/cc0/1.0/ 156 | 157 | ![sample_image](sample/street_view_of_a_small_neighborhood.png) 158 | 159 | ```python 160 | image_path = "street_view_of_a_small_neighborhood.png" 161 | pred_fastai = learn.predict(image_path) 162 | pred_fastai[0].numpy() 163 | >>> 164 | array([[26, 26, 26, ..., 4, 4, 4], 165 | [26, 26, 26, ..., 4, 4, 4], 166 | [26, 26, 26, ..., 4, 4, 4], 167 | ..., 168 | [17, 17, 17, ..., 30, 30, 30], 169 | [17, 17, 17, ..., 30, 30, 30], 170 | [17, 17, 17, ..., 30, 30, 30]]) 171 | ``` 172 | 173 | ### PyTorch Model from FastAI Source Code 174 | 175 | Next, we need to define the model in pure PyTorch. In [Jupyter](https://jupyter.org/) notebook, one can investigate the FastAI source code by adding `??` in front of a function name. Here we look into `unet_learner` and `DynamicUnet`, by: 176 | 177 | ```python 178 | >> ??unet_learner 179 | >> ??DynamicUnet 180 | ``` 181 | 182 | Each of these command will pop up a window at bottom of the browser: 183 | 184 | ![fastai_source_code](sample/fastsource.png) 185 | 186 | After investigating, the PyTorch model can be defined as: 187 | 188 | ```python 189 | from fastai.vision.all import * 190 | from fastai.vision.learner import _default_meta 191 | from fastai.vision.models.unet import _get_sz_change_idxs, UnetBlock, ResizeToOrig 192 | 193 | 194 | class DynamicUnetDIY(SequentialEx): 195 | "Create a U-Net from a given architecture." 196 | 197 | def __init__( 198 | self, 199 | arch=resnet50, 200 | n_classes=32, 201 | img_size=(96, 128), 202 | blur=False, 203 | blur_final=True, 204 | y_range=None, 205 | last_cross=True, 206 | bottle=False, 207 | init=nn.init.kaiming_normal_, 208 | norm_type=None, 209 | self_attention=None, 210 | act_cls=defaults.activation, 211 | n_in=3, 212 | cut=None, 213 | **kwargs 214 | ): 215 | meta = model_meta.get(arch, _default_meta) 216 | encoder = create_body( 217 | arch, n_in, pretrained=False, cut=ifnone(cut, meta["cut"]) 218 | ) 219 | imsize = img_size 220 | 221 | sizes = model_sizes(encoder, size=imsize) 222 | sz_chg_idxs = list(reversed(_get_sz_change_idxs(sizes))) 223 | self.sfs = hook_outputs([encoder[i] for i in sz_chg_idxs], detach=False) 224 | x = dummy_eval(encoder, imsize).detach() 225 | 226 | ni = sizes[-1][1] 227 | middle_conv = nn.Sequential( 228 | ConvLayer(ni, ni * 2, act_cls=act_cls, norm_type=norm_type, **kwargs), 229 | ConvLayer(ni * 2, ni, act_cls=act_cls, norm_type=norm_type, **kwargs), 230 | ).eval() 231 | x = middle_conv(x) 232 | layers = [encoder, BatchNorm(ni), nn.ReLU(), middle_conv] 233 | 234 | for i, idx in enumerate(sz_chg_idxs): 235 | not_final = i != len(sz_chg_idxs) - 1 236 | up_in_c, x_in_c = int(x.shape[1]), int(sizes[idx][1]) 237 | do_blur = blur and (not_final or blur_final) 238 | sa = self_attention and (i == len(sz_chg_idxs) - 3) 239 | unet_block = UnetBlock( 240 | up_in_c, 241 | x_in_c, 242 | self.sfs[i], 243 | final_div=not_final, 244 | blur=do_blur, 245 | self_attention=sa, 246 | act_cls=act_cls, 247 | init=init, 248 | norm_type=norm_type, 249 | **kwargs 250 | ).eval() 251 | layers.append(unet_block) 252 | x = unet_block(x) 253 | 254 | ni = x.shape[1] 255 | if imsize != sizes[0][-2:]: 256 | layers.append(PixelShuffle_ICNR(ni, act_cls=act_cls, norm_type=norm_type)) 257 | layers.append(ResizeToOrig()) 258 | if last_cross: 259 | layers.append(MergeLayer(dense=True)) 260 | ni += in_channels(encoder) 261 | layers.append( 262 | ResBlock( 263 | 1, 264 | ni, 265 | ni // 2 if bottle else ni, 266 | act_cls=act_cls, 267 | norm_type=norm_type, 268 | **kwargs 269 | ) 270 | ) 271 | layers += [ 272 | ConvLayer(ni, n_classes, ks=1, act_cls=None, norm_type=norm_type, **kwargs) 273 | ] 274 | apply_init(nn.Sequential(layers[3], layers[-2]), init) 275 | # apply_init(nn.Sequential(layers[2]), init) 276 | if y_range is not None: 277 | layers.append(SigmoidRange(*y_range)) 278 | super().__init__(*layers) 279 | 280 | def __del__(self): 281 | if hasattr(self, "sfs"): 282 | self.sfs.remove() 283 | ``` 284 | 285 | Also check the inheritance hierarchy of the FastAI defined class `SequentialEx` by: 286 | 287 | ```python 288 | SequentialEx.mro() 289 | >>> [fastai.layers.SequentialEx, 290 | fastai.torch_core.Module, 291 | torch.nn.modules.module.Module, 292 | object] 293 | ``` 294 | 295 | Here we can see `SequentialEx` stems from the PyTorch `torch.nn.modules`, therefore `DynamicUnetDIY` is a PyTorch Model. 296 | 297 | **Note**: parameters of `arch`, `n_classes`, `img_size`, etc., must be consistent with the training process. If other parameters are customized during training, they must be reflected here as well. Also in the `create_body` we set `pretrained=False` as we are transferring the weights from FastAI so there is no need to download weights from PyTorch again. 298 | 299 | ### Weights Transfer 300 | 301 | Now initialize the PyTorch model, load the saved model weights, and transfer that weights to the PyTorch model. 302 | 303 | ```python 304 | model_torch_rep = DynamicUnetDIY() 305 | state = torch.load("fasti_unet_weights.pth") 306 | model_torch_rep.load_state_dict(state) 307 | model_torch_rep.eval(); 308 | ``` 309 | 310 | If take one sample image, transform it, and pass it to the `model_torch_rep`, we shall get an identical prediction result as FastAI's. 311 | 312 | ```python 313 | from torchvision import transforms 314 | from PIL import Image 315 | import numpy as np 316 | 317 | image_path = "street_view_of_a_small_neighborhood.png" 318 | 319 | image = Image.open(image_path).convert("RGB") 320 | image_tfm = transforms.Compose( 321 | [ 322 | transforms.Resize((96, 128)), 323 | transforms.ToTensor(), 324 | transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]), 325 | ] 326 | ) 327 | 328 | x = image_tfm(image).unsqueeze_(0) 329 | 330 | # inference on CPU 331 | raw_out = model_torch_rep(x) 332 | raw_out.shape 333 | >>> torch.Size([1, 32, 96, 128]) 334 | 335 | pred_res = raw_out[0].argmax(dim=0).numpy().astype(np.uint8) 336 | pred_res 337 | >>> 338 | array([[26, 26, 26, ..., 4, 4, 4], 339 | [26, 26, 26, ..., 4, 4, 4], 340 | [26, 26, 26, ..., 4, 4, 4], 341 | ..., 342 | [17, 17, 17, ..., 30, 30, 30], 343 | [17, 17, 17, ..., 30, 30, 30], 344 | [17, 17, 17, ..., 30, 30, 30]], dtype=uint8) 345 | 346 | np.all(pred_fastai[0].numpy() == pred_res) 347 | >>> True 348 | ``` 349 | 350 | Here we can see the difference: in FastAI model `fastai_unet.pkl`, it packages all the steps including the data transformation, image dimension alignment, etc.; but in `fasti_unet_weights.pth` it has only the pure weights and we have to manually re-define the data transformation procedures among others and make sure they are consistent with the training step. 351 | 352 | **Note**: in `image_tfm` make sure the image size and normalization statistics are consistent with the training step. In our example here, the size is `96x128` and normalization is by default from [ImageNet](http://www.image-net.org/) as used in FastAI. If other transformations were applied during training, they may need to be added here as well. 353 | 354 | For more details about the PyTorch weights transferring process, please refer to `notebook/02_Inference_in_pytorch.ipynb` [[link](notebook/02_Inference_in_pytorch.ipynb)]. 355 | 356 | ## Deployment to TorchServe 357 | 358 | In this section we deploy the PyTorch model to TorchServe. For installation, please refer to TorchServe [Github](https://github.com/pytorch/serve) Repository. 359 | 360 | Overall, there are mainly 3 steps to use TorchServe: 361 | 362 | 1. Archive the model into `*.mar`. 363 | 2. Start the `torchserve`. 364 | 3. Call the API and get the response. 365 | 366 | In order to archive the model, at least 3 files are needed in our case: 367 | 368 | 1. PyTorch model weights `fasti_unet_weights.pth`. 369 | 2. PyTorch model definition `model.py`, which is identical to `DynamicUnetDIY` definition described in the last section. 370 | 3. TorchServe custom handler. 371 | 372 | ### Custom Handler 373 | 374 | As shown in `/deployment/handler.py`, the TorchServe handler accept `data` and `context`. In our example, we define another helper Python class with 4 instance methods to implement: `initialize`, `preprocess`, `inference` and `postprocess`. 375 | 376 | #### `initialize` 377 | 378 | Here we workout if GPU is available, then identify the serialized model weights file path and finally instantiate the PyTorch model and put it to evaluation mode. 379 | 380 | ```python 381 | def initialize(self, ctx): 382 | """ 383 | load eager mode state_dict based model 384 | """ 385 | properties = ctx.system_properties 386 | self.device = torch.device( 387 | "cuda:" + str(properties.get("gpu_id")) 388 | if torch.cuda.is_available() 389 | else "cpu" 390 | ) 391 | model_dir = properties.get("model_dir") 392 | 393 | manifest = ctx.manifest 394 | logger.error(manifest) 395 | serialized_file = manifest["model"]["serializedFile"] 396 | model_pt_path = os.path.join(model_dir, serialized_file) 397 | if not os.path.isfile(model_pt_path): 398 | raise RuntimeError("Missing the model definition file") 399 | 400 | logger.debug(model_pt_path) 401 | 402 | from model import DynamicUnetDIY 403 | 404 | state_dict = torch.load(model_pt_path, map_location=self.device) 405 | self.model = DynamicUnetDIY() 406 | self.model.load_state_dict(state_dict) 407 | self.model.to(self.device) 408 | self.model.eval() 409 | 410 | logger.debug("Model file {0} loaded successfully".format(model_pt_path)) 411 | self.initialized = True 412 | ``` 413 | 414 | #### `preprocess` 415 | 416 | As described in the previous section, we re-define the image transform steps and apply them to the inference data. 417 | 418 | ```python 419 | def preprocess(self, data): 420 | """ 421 | Scales and normalizes a PIL image for an U-net model 422 | """ 423 | image = data[0].get("data") 424 | if image is None: 425 | image = data[0].get("body") 426 | 427 | image_transform = transforms.Compose( 428 | [ 429 | transforms.Resize((96, 128)), 430 | transforms.ToTensor(), 431 | transforms.Normalize( 432 | mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225] 433 | ), 434 | ] 435 | ) 436 | image = Image.open(io.BytesIO(image)).convert( 437 | "RGB" 438 | ) 439 | image = image_transform(image).unsqueeze_(0) 440 | return image 441 | ``` 442 | 443 | #### `inference` 444 | 445 | Now convert image into PyTorch Tensor, load it into GPU if available, and pass it through the model. 446 | 447 | ```python 448 | def inference(self, img): 449 | """ 450 | Predict the chip stack mask of an image using a trained deep learning model. 451 | """ 452 | self.model.eval() 453 | inputs = Variable(img).to(self.device) 454 | outputs = self.model.forward(inputs) 455 | logging.debug(outputs.shape) 456 | return outputs 457 | ``` 458 | 459 | #### `postprocess` 460 | 461 | Here the inference raw output is unloaded from GPU if available, and encoded with Base64 to be returned back to the API trigger. 462 | 463 | ```python 464 | def postprocess(self, inference_output): 465 | 466 | if torch.cuda.is_available(): 467 | inference_output = inference_output[0].argmax(dim=0).cpu() 468 | else: 469 | inference_output = inference_output[0].argmax(dim=0) 470 | 471 | return [ 472 | { 473 | "base64_prediction": base64.b64encode( 474 | inference_output.numpy().astype(np.uint8) 475 | ).decode("utf-8") 476 | } 477 | ] 478 | ``` 479 | 480 | Now it's ready to setup and launch TorchServe. 481 | 482 | ### TorchServe in Action 483 | 484 | Step 1: Archive the model PyTorch 485 | 486 | ```bash 487 | >>> torch-model-archiver --model-name fastunet --version 1.0 --model-file deployment/model.py --serialized-file model_store/fasti_unet_weights.pth --export-path model_store --handler deployment/handler.py -f 488 | ``` 489 | 490 | Step 2: Serve the Model 491 | 492 | ```bash 493 | >>> torchserve --start --ncs --model-store model_store --models fastunet.mar 494 | ``` 495 | 496 | Step 3: Call API and Get the Response (here we use [httpie](https://httpie.org/)). For a complete response see `sample/sample_output.txt` at [here](sample/sample_output.txt). 497 | 498 | ```bash 499 | >>> time http POST http://127.0.0.1:8080/predictions/fastunet/ @sample/street_view_of_a_small_neighborhood.png 500 | 501 | HTTP/1.1 200 502 | Cache-Control: no-cache; no-store, must-revalidate, private 503 | Expires: Thu, 01 Jan 1970 00:00:00 UTC 504 | Pragma: no-cache 505 | connection: keep-alive 506 | content-length: 131101 507 | x-request-id: 96c25cb1-99c2-459e-9165-aa5ef9e3a439 508 | 509 | { 510 | "base64_prediction": "GhoaGhoaGhoaGhoaGhoaGhoaGh...ERERERERERERERERERERER" 511 | } 512 | 513 | real 0m0.979s 514 | user 0m0.280s 515 | sys 0m0.039s 516 | ``` 517 | 518 | The first call would have longer latency due to model weights loading defined in `initialize`, but this will be mitigated from the second call onward. For more details about TorchServe setup and usage, please refer to `notebook/03_TorchServe.ipynb` [[link](notebook/03_TorchServe.ipynb)]. 519 | 520 | ## Deployment to Amazon SageMaker Inference Endpoint 521 | 522 | In this section we deploy the FastAI trained Scene Segmentation PyTorch model with TorchServe in Amazon SageMaker Endpoint using customized Docker image, and we will be using a `ml.g4dn.xlarge` instance. For more details about Amazon G4 Instances, please refer to [here](https://aws.amazon.com/ec2/instance-types/g4/). 523 | 524 | ### Getting Started with Amazon SageMaker Endpoint 525 | 526 | There are 4 steps to setup a SageMaker Endpoint with TorchServe: 527 | 528 | 1. Build customized Docker Image and push to Amazon Elastic Container Registry (ECR). The dockerfile is provided in root of this code repository, which helps setup FastAI and TorchServe dependencies. 529 | 2. Compress `*.mar` into `*.tar.gz` and upload to Amazon Simple Storage Service (S3). 530 | 3. Create SageMaker model using the docker image from step 1 and the compressed model weights from step 2. 531 | 4. Create the SageMaker endpoint using the model from step 3. 532 | 533 | The details of these steps are described in `notebook/04_SageMaker.ipynb` [[link](notebook/04_SageMaker.ipynb)]. Once ready, we can invoke the SageMaker endpoint with image in real-time. 534 | 535 | ### Real-time Inference with Python SDK 536 | 537 | Read a sample image. 538 | 539 | ```python 540 | file_name = "street_view_of_a_small_neighborhood.png" 541 | 542 | with open(file_name, 'rb') as f: 543 | payload = f.read() 544 | ``` 545 | 546 | Invoke the SageMaker endpoint with the image and obtain the response from the API. 547 | 548 | ```python 549 | client = boto3.client("runtime.sagemaker") 550 | response = client.invoke_endpoint( 551 | EndpointName=endpoint_name, ContentType="application/x-image", Body=payload 552 | ) 553 | response = json.loads(response["Body"].read()) 554 | ``` 555 | 556 | Decode the response and visualize the predicted Scene Segmentation mask. 557 | 558 | ```python 559 | pred_decoded_byte = base64.decodebytes(bytes(response["base64_prediction"], encoding="utf-8")) 560 | pred_decoded = np.reshape( 561 | np.frombuffer(pred_decoded_byte, dtype=np.uint8), (96, 128) 562 | ) 563 | plt.imshow(pred_decoded) 564 | plt.axis("off") 565 | plt.show() 566 | ``` 567 | 568 | ![sample_prediction_response](sample/sample_pred_mask.png) 569 | 570 | ### What's Next 571 | 572 | With an inference endpoint up and running, one could levearge its full power by exploring other features that are important for a Machine Learning product, including [AutoScaling](https://docs.aws.amazon.com/sagemaker/latest/dg/endpoint-auto-scaling.html), Model monitoring with [Human-in-the-loop (HITL)](https://docs.aws.amazon.com/sagemaker/latest/dg/a2i-use-augmented-ai-a2i-human-review-loops.html) using Amazon Augmented AI ([A2I](https://aws.amazon.com/augmented-ai/)), and incremental modeling iteration. 573 | 574 | ### Clean Up 575 | 576 | Make sure that you delete the following resources to prevent any additional charges: 577 | 578 | 1. Amazon SageMaker endpoint. 579 | 2. Amazon SageMaker endpoint configuration. 580 | 3. Amazon SageMaker model. 581 | 4. Amazon Elastic Container Registry (ECR). 582 | 5. Amazon Simple Storage Service (S3) Buckets. 583 | 584 | ## Conclusion 585 | 586 | This repository presented an end-to-end demonstration of deploying FastAI trained PyTorch models on TorchServe eager mode and host in Amazon SageMaker Endpoint. You can use this repository as a template to deploy your own FastAI models. This approach eliminates the self-maintaining effort to build and manage a customized inference server, which helps you to speed up the process from training a cutting-edge deep learning model to its online application in real-world at scale. 587 | 588 | If you have questions please create an issue or submit Pull Request on the [GitHub](https://github.com/aws-samples/amazon-sagemaker-endpoint-deployment-of-fastai-model-with-torchserve) repository. 589 | 590 | ## Reference 591 | 592 | - [fast.ai · Making neural nets uncool again](https://www.fast.ai/) 593 | - [TORCHSERVE](https://pytorch.org/serve/) 594 | - [Deploying PyTorch models for inference at scale using TorchServe](https://aws.amazon.com/blogs/machine-learning/deploying-pytorch-models-for-inference-at-scale-using-torchserve/) 595 | - [Serving PyTorch models in production with the Amazon SageMaker native TorchServe integration](https://aws.amazon.com/blogs/machine-learning/serving-pytorch-models-in-production-with-the-amazon-sagemaker-native-torchserve-integration/) 596 | - [Building, training, and deploying fastai models with Amazon SageMaker](https://aws.amazon.com/blogs/machine-learning/building-training-and-deploying-fastai-models-with-amazon-sagemaker/) 597 | - [Running TorchServe on Amazon Elastic Kubernetes Service](https://aws.amazon.com/blogs/opensource/running-torchserve-on-amazon-elastic-kubernetes-service/) 598 | --------------------------------------------------------------------------------