├── .github ├── CODE_OF_CONDUCT.md ├── ISSUE_TEMPLATE.md └── PULL_REQUEST_TEMPLATE.md ├── .gitignore ├── CHANGELOG.md ├── CONTRIBUTING.md ├── LICENSE.md ├── README.md └── start ├── .funcignore ├── .vscode └── extensions.json ├── classify ├── __init__.py ├── function.json ├── labels.json └── predictonnx.py ├── host.json └── requirements.txt /.github/CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | # Microsoft Open Source Code of Conduct 2 | 3 | This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). 4 | 5 | Resources: 6 | 7 | - [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/) 8 | - [Microsoft Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) 9 | - Contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with questions or concerns 10 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE.md: -------------------------------------------------------------------------------- 1 | 4 | > Please provide us with the following information: 5 | > --------------------------------------------------------------- 6 | 7 | ### This issue is for a: (mark with an `x`) 8 | ``` 9 | - [ ] bug report -> please search issues before submitting 10 | - [ ] feature request 11 | - [ ] documentation issue or request 12 | - [ ] regression (a behavior that used to work and stopped in a new release) 13 | ``` 14 | 15 | ### Minimal steps to reproduce 16 | > 17 | 18 | ### Any log messages given by the failure 19 | > 20 | 21 | ### Expected/desired behavior 22 | > 23 | 24 | ### OS and Version? 25 | > Windows 7, 8 or 10. Linux (which distribution). macOS (Yosemite? El Capitan? Sierra?) 26 | 27 | ### Versions 28 | > 29 | 30 | ### Mention any other details that might be useful 31 | 32 | > --------------------------------------------------------------- 33 | > Thanks! We'll be in touch soon. 34 | -------------------------------------------------------------------------------- /.github/PULL_REQUEST_TEMPLATE.md: -------------------------------------------------------------------------------- 1 | ## Purpose 2 | 3 | * ... 4 | 5 | ## Does this introduce a breaking change? 6 | 7 | ``` 8 | [ ] Yes 9 | [ ] No 10 | ``` 11 | 12 | ## Pull Request Type 13 | What kind of change does this Pull Request introduce? 14 | 15 | 16 | ``` 17 | [ ] Bugfix 18 | [ ] Feature 19 | [ ] Code style update (formatting, local variables) 20 | [ ] Refactoring (no functional changes, no api changes) 21 | [ ] Documentation content changes 22 | [ ] Other... Please describe: 23 | ``` 24 | 25 | ## How to Test 26 | * Get the code 27 | 28 | ``` 29 | git clone [repo-address] 30 | cd [repo-name] 31 | git checkout [branch-name] 32 | npm install 33 | ``` 34 | 35 | * Test the code 36 | 37 | ``` 38 | ``` 39 | 40 | ## What to Check 41 | Verify that the following are valid 42 | * ... 43 | 44 | ## Other Information 45 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | *.onnx 2 | bin 3 | obj 4 | csx 5 | .vs 6 | edge 7 | Publish 8 | 9 | *.user 10 | *.suo 11 | *.cscfg 12 | *.Cache 13 | project.lock.json 14 | 15 | /packages 16 | /TestResults 17 | 18 | /tools/NuGet.exe 19 | /App_Data 20 | /secrets 21 | /data 22 | .secrets 23 | appsettings.json 24 | local.settings.json 25 | 26 | node_modules 27 | dist 28 | 29 | # Local python packages 30 | .python_packages/ 31 | 32 | # Python Environments 33 | .env 34 | .venv 35 | env/ 36 | venv/ 37 | ENV/ 38 | env.bak/ 39 | venv.bak/ 40 | 41 | # Byte-compiled / optimized / DLL files 42 | __pycache__/ 43 | *.py[cod] 44 | *$py.class 45 | -------------------------------------------------------------------------------- /CHANGELOG.md: -------------------------------------------------------------------------------- 1 | ## [project-title] Changelog 2 | 3 | 4 | # x.y.z (yyyy-mm-dd) 5 | 6 | *Features* 7 | * ... 8 | 9 | *Bug Fixes* 10 | * ... 11 | 12 | *Breaking Changes* 13 | * ... 14 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing to [project-title] 2 | 3 | This project welcomes contributions and suggestions. Most contributions require you to agree to a 4 | Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us 5 | the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com. 6 | 7 | When you submit a pull request, a CLA bot will automatically determine whether you need to provide 8 | a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions 9 | provided by the bot. You will only need to do this once across all repos using our CLA. 10 | 11 | This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). 12 | For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or 13 | contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments. 14 | 15 | - [Code of Conduct](#coc) 16 | - [Issues and Bugs](#issue) 17 | - [Feature Requests](#feature) 18 | - [Submission Guidelines](#submit) 19 | 20 | ## Code of Conduct 21 | Help us keep this project open and inclusive. Please read and follow our [Code of Conduct](https://opensource.microsoft.com/codeofconduct/). 22 | 23 | ## Found an Issue? 24 | If you find a bug in the source code or a mistake in the documentation, you can help us by 25 | [submitting an issue](#submit-issue) to the GitHub Repository. Even better, you can 26 | [submit a Pull Request](#submit-pr) with a fix. 27 | 28 | ## Want a Feature? 29 | You can *request* a new feature by [submitting an issue](#submit-issue) to the GitHub 30 | Repository. If you would like to *implement* a new feature, please submit an issue with 31 | a proposal for your work first, to be sure that we can use it. 32 | 33 | * **Small Features** can be crafted and directly [submitted as a Pull Request](#submit-pr). 34 | 35 | ## Submission Guidelines 36 | 37 | ### Submitting an Issue 38 | Before you submit an issue, search the archive, maybe your question was already answered. 39 | 40 | If your issue appears to be a bug, and hasn't been reported, open a new issue. 41 | Help us to maximize the effort we can spend fixing issues and adding new 42 | features, by not reporting duplicate issues. Providing the following information will increase the 43 | chances of your issue being dealt with quickly: 44 | 45 | * **Overview of the Issue** - if an error is being thrown a non-minified stack trace helps 46 | * **Version** - what version is affected (e.g. 0.1.2) 47 | * **Motivation for or Use Case** - explain what are you trying to do and why the current behavior is a bug for you 48 | * **Browsers and Operating System** - is this a problem with all browsers? 49 | * **Reproduce the Error** - provide a live example or a unambiguous set of steps 50 | * **Related Issues** - has a similar issue been reported before? 51 | * **Suggest a Fix** - if you can't fix the bug yourself, perhaps you can point to what might be 52 | causing the problem (line of code or commit) 53 | 54 | You can file new issues by providing the above information at the corresponding repository's issues link: https://github.com/[organization-name]/[repository-name]/issues/new]. 55 | 56 | ### Submitting a Pull Request (PR) 57 | Before you submit your Pull Request (PR) consider the following guidelines: 58 | 59 | * Search the repository (https://github.com/[organization-name]/[repository-name]/pulls) for an open or closed PR 60 | that relates to your submission. You don't want to duplicate effort. 61 | 62 | * Make your changes in a new git fork: 63 | 64 | * Commit your changes using a descriptive commit message 65 | * Push your fork to GitHub: 66 | * In GitHub, create a pull request 67 | * If we suggest changes then: 68 | * Make the required updates. 69 | * Rebase your fork and force push to your GitHub repository (this will update your Pull Request): 70 | 71 | ```shell 72 | git rebase master -i 73 | git push -f 74 | ``` 75 | 76 | That's it! Thank you for your contribution! 77 | -------------------------------------------------------------------------------- /LICENSE.md: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) Microsoft Corporation. 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Deploy ONNX Models to Azure Functions 2 | 3 | Samples for serverless deployment of ONNX models to Azure Functions. 4 | 5 | ## Pre-requisites 6 | 7 | Before you begin, you must have the following: 8 | 9 | 1. An Azure account with an active subscription. [Create an account for free](https://azure.microsoft.com/free). 10 | 11 | On a Linux system or Windows (WSL or WSL2) ensure you have the following installed: 12 | 13 | 2. The [Azure Functions Core Tools](https://docs.microsoft.com/en-us/azure/azure-functions/functions-run-local#v2) 14 | 3. The [Azure CLI](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli) 15 | 4. Python 3.7 16 | 17 | Note: You can also use Azure Cloud Shell which comes preinstalled with Azure CLI and Functions Core Tools. 18 | 19 | ## Develop and Test Locally 20 | 21 | You can develop for serverless deployment on Azure functions on your Linux machine locally or a development VM (like the Azure Data Science VM) on the cloud or Azure Cloud Shell. Run the following commands to setup your Azure Functions project locally. 22 | 23 | 1. Create a Function App project directory and start directory 24 | 25 | ``` 26 | mkdir << Your projectname >> 27 | cd << Your projectname>> 28 | mkdir start 29 | cd start 30 | ``` 31 | 32 | 2. Initialize Function App 33 | 34 | ``` 35 | func init --worker-runtime python 36 | func new --name classify --template "HTTP trigger" 37 | ``` 38 | 39 | 3. Copy the deployment code 40 | ``` 41 | git clone https://github.com/Azure-Samples/functions-deploy-pytorch-onnx.git ~/functions-deploy-pytorch-onnx 42 | 43 | # Copy the deployment sample to function app 44 | cp -r ~/functions-deploy-pytorch-onnx/start .. 45 | 46 | ``` 47 | The main files are **[__init__.py](https://github.com/Azure-Samples/functions-deploy-pytorch-onnx/blob/main/start/classify/__init__.py)** and **[predictonnx.py](https://github.com/Azure-Samples/functions-deploy-pytorch-onnx/blob/main/start/classify/predictonnx.py)** in ```start/classify``` dirtectory. The one in the repo works for the Bear detector example in fast.ai. It takes input from the HTTP GET request in "img" parameter which is a URL to an image which will be run through the model for prediction of the type of bear. You can adapt the same easily for deploying other models. 48 | 49 | 4. Create and activate Python virtualenv to setup ONNX runtime along with dependencies 50 | 51 | ``` 52 | python -m venv .venv 53 | source .venv/bin/activate 54 | 55 | pip install --no-cache-dir -r requirements.txt 56 | ``` 57 | 58 | 5. Copy your ONNX model file (which should have a name model.onnx) built from training the Pytorch model and converting to ONNX into the "start/classify" directory within your Function App project. This has been tested with the Bear detector model from fast.ai course. Here is code to generate the model.onnx file for this fast.ai (or Pytorch) model. 59 | 60 | ``` 61 | dummy_input = torch.randn(1, 3, 224, 224, device='cuda') 62 | onnx_path = "./model.onnx" 63 | torch.onnx.export(learn.model, dummy_input, onnx_path, verbose=False) 64 | ``` 65 | 66 | 67 | You also need to create a ```labels.json``` in ```start/classify``` directory. For the fast.ai 3 class Bear detector example it looks like this ```["black","grizzly","teddy"]``` matching the class label and index during training. 68 | 69 | 6. Run the test locally 70 | 71 | ``` 72 | func start 73 | ``` 74 | In a browser on your machine you can test the local Azure Function by visiting: 75 | 76 | "http://localhost:7071/api/classify?img=http://3.bp.blogspot.com/-S1scRCkI3vY/UHzV2kucsPI/AAAAAAAAA-k/YQ5UzHEm9Ss/s1600/Grizzly%2BBear%2BWildlife188.jpg" 77 | 78 | 79 | ## Create Resources in Azure and Publish 80 | 81 | 1. Create Azure Function App using Azure CLI 82 | 83 | ``` 84 | # If you have not logged into Azure CLI. you must first run "az login" and follow instructions 85 | 86 | az group create --name [[YOUR Function App name]] --location westus2 87 | az storage account create --name [[Your Storage Account Name]] -l westus2 --sku Standard_LRS -g [[YOUR Function App name]] 88 | az functionapp create --name [[YOUR Function App name]] -g [[YOUR Function App name]] --consumption-plan-location westus2 --storage-account [[Your Storage Account Name]] --runtime python --runtime-version 3.7 --functions-version 3 --disable-app-insights --os-type Linux 89 | ``` 90 | 2. Publish to Azure 91 | 92 | ``` 93 | # Install a local copy of ONNX runtime and dependencies to push to Azure Functions Runtime 94 | pip install --target="./.python_packages/lib/site-packages" -r requirements.txt 95 | 96 | # Publish Azure function to the 97 | func azure functionapp publish [[YOUR Function App name] --no-build 98 | 99 | 100 | ``` 101 | 102 | It will take a few minutes to publish and bring up the Azure functions with your ONNX model deployed and exposed as a http endpoint. Then you can find the URL by running the following command: ```func azure functionapp list-functions [[YOUR Function App name] --show-keys``` . Append ```&img=[[Your Image URL to run thru model]]``` to the URL on a browser to get predictions from the model running in the Azure Functions. 103 | 104 | ## Deleting Resources 105 | To delete all the resources (and avoiding any charges) after you are done, run the following Azure CLI command: 106 | ``` 107 | az group delete --name [[YOUR Function App name]] --yes 108 | 109 | ``` 110 | Azure Functions provides other options like auto scaling, larger instances, monitoring with Application Insights etc. 111 | 112 | ## Notes 113 | 114 | 1. If you are using Windows WSL or WSL2 and face authentication issues while deploying Function App to Azure, one of the main reason is your clock in WSL (Linux) may be out of sync with the underlying Windows host. You can synch it by running ```sudo hwclock -s``` in WSL. 115 | 116 | -------------------------------------------------------------------------------- /start/.funcignore: -------------------------------------------------------------------------------- 1 | .venv 2 | __pycache__/ 3 | 4 | -------------------------------------------------------------------------------- /start/.vscode/extensions.json: -------------------------------------------------------------------------------- 1 | { 2 | "recommendations": [ 3 | "ms-azuretools.vscode-azurefunctions" 4 | ] 5 | } -------------------------------------------------------------------------------- /start/classify/__init__.py: -------------------------------------------------------------------------------- 1 | import logging 2 | 3 | import azure.functions as func 4 | import json 5 | 6 | # Import helper script 7 | from .predictonnx import predict_image_from_url 8 | 9 | def main(req: func.HttpRequest) -> func.HttpResponse: 10 | image_url = req.params.get('img') 11 | logging.info('Image URL received: ' + image_url) 12 | 13 | results = predict_image_from_url(image_url) 14 | 15 | headers = { 16 | "Content-type": "application/json", 17 | "Access-Control-Allow-Origin": "*" 18 | } 19 | 20 | return func.HttpResponse(json.dumps(results, indent=4), headers = headers) 21 | 22 | -------------------------------------------------------------------------------- /start/classify/function.json: -------------------------------------------------------------------------------- 1 | { 2 | "scriptFile": "__init__.py", 3 | "bindings": [ 4 | { 5 | "authLevel": "function", 6 | "type": "httpTrigger", 7 | "direction": "in", 8 | "name": "req", 9 | "methods": [ 10 | "get", 11 | "post" 12 | ] 13 | }, 14 | { 15 | "type": "http", 16 | "direction": "out", 17 | "name": "$return" 18 | } 19 | ] 20 | } -------------------------------------------------------------------------------- /start/classify/labels.json: -------------------------------------------------------------------------------- 1 | ["black","grizzly","teddy"] 2 | -------------------------------------------------------------------------------- /start/classify/predictonnx.py: -------------------------------------------------------------------------------- 1 | import numpy as np # we're going to use numpy to process input and output data 2 | import onnxruntime # to inference ONNX models, we use the ONNX Runtime 3 | import onnx 4 | from onnx import numpy_helper 5 | from urllib.request import urlopen 6 | import json 7 | import time 8 | 9 | import logging 10 | import os 11 | import sys 12 | from datetime import datetime 13 | 14 | # display images in notebook 15 | from PIL import Image, ImageDraw, ImageFont, ImageOps 16 | 17 | def load_labels(path): 18 | with open(path) as f: 19 | data = json.load(f) 20 | return np.asarray(data) 21 | 22 | # Run the model on the backend 23 | d=os.path.dirname(os.path.abspath(__file__)) 24 | modelfile=os.path.join(d , 'model.onnx') 25 | labelfile=os.path.join(d , 'labels.json') 26 | 27 | session = onnxruntime.InferenceSession(modelfile, None) 28 | 29 | # get the name of the first input of the model 30 | input_name = session.get_inputs()[0].name 31 | 32 | labels = load_labels(labelfile) 33 | 34 | def preprocess(input_data): 35 | # convert the input data into the float32 input 36 | img_data = input_data.astype('float32') 37 | 38 | #normalize 39 | mean_vec = np.array([0.485, 0.456, 0.406]) 40 | stddev_vec = np.array([0.229, 0.224, 0.225]) 41 | norm_img_data = np.zeros(img_data.shape).astype('float32') 42 | for i in range(img_data.shape[0]): 43 | norm_img_data[i,:,:] = (img_data[i,:,:]/255 - mean_vec[i]) / stddev_vec[i] 44 | 45 | #add batch channel 46 | norm_img_data = norm_img_data.reshape(1,3,224, 224).astype('float32') 47 | return norm_img_data 48 | 49 | def softmax(x): 50 | x = x.reshape(-1) 51 | e_x = np.exp(x - np.max(x)) 52 | return e_x / e_x.sum(axis=0) 53 | 54 | def postprocess(result): 55 | return softmax(np.array(result)).tolist() 56 | 57 | def predict_image_from_url(image_url): 58 | with urlopen(image_url) as testImage: 59 | image = Image.open(testImage) 60 | 61 | imnew=ImageOps.fit(image, (224,224)) 62 | 63 | image_data = np.array(imnew).transpose(2, 0, 1) 64 | input_data = preprocess(image_data) 65 | 66 | start = time.time() 67 | raw_result = session.run([], {input_name: input_data}) 68 | end = time.time() 69 | res = postprocess(raw_result) 70 | 71 | inference_time = np.round((end - start) * 1000, 2) 72 | idx = np.argmax(res) 73 | 74 | response = { 75 | 'created': datetime.utcnow().isoformat(), 76 | 'prediction': labels[idx], 77 | 'latency': inference_time, 78 | 'confidence': res[idx] 79 | } 80 | logging.info(f'returning {response}') 81 | return response 82 | 83 | if __name__ == '__main__': 84 | print(predict_image_from_url(sys.argv[1])) 85 | -------------------------------------------------------------------------------- /start/host.json: -------------------------------------------------------------------------------- 1 | { 2 | "version": "2.0", 3 | "extensionBundle": { 4 | "id": "Microsoft.Azure.Functions.ExtensionBundle", 5 | "version": "[1.*, 2.0.0)" 6 | } 7 | } -------------------------------------------------------------------------------- /start/requirements.txt: -------------------------------------------------------------------------------- 1 | azure-functions 2 | requests 3 | pillow 4 | onnxruntime 5 | --------------------------------------------------------------------------------