├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── LICENSE ├── README.md ├── code ├── .ipynb_checkpoints │ └── inference-checkpoint.py ├── README.md ├── inference.py └── requirements.txt ├── images ├── Async_Diagram.png ├── backlog_size_metrics.png ├── birds.jpg ├── instance_count_0.png ├── invocations_metrics.png └── model_latency_metrics.png ├── mask-rcnn-async-inference.ipynb ├── videos └── ducks.mp4 └── visualization ├── annotated_frames └── birds.gif ├── generate_gif.ipynb └── output.json /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | ## Code of Conduct 2 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 3 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 4 | opensource-codeofconduct@amazon.com with any additional questions or comments. 5 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing Guidelines 2 | 3 | Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional 4 | documentation, we greatly value feedback and contributions from our community. 5 | 6 | Please read through this document before submitting any issues or pull requests to ensure we have all the necessary 7 | information to effectively respond to your bug report or contribution. 8 | 9 | 10 | ## Reporting Bugs/Feature Requests 11 | 12 | We welcome you to use the GitHub issue tracker to report bugs or suggest features. 13 | 14 | When filing an issue, please check existing open, or recently closed, issues to make sure somebody else hasn't already 15 | reported the issue. Please try to include as much information as you can. Details like these are incredibly useful: 16 | 17 | * A reproducible test case or series of steps 18 | * The version of our code being used 19 | * Any modifications you've made relevant to the bug 20 | * Anything unusual about your environment or deployment 21 | 22 | 23 | ## Contributing via Pull Requests 24 | Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that: 25 | 26 | 1. You are working against the latest source on the *main* branch. 27 | 2. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already. 28 | 3. You open an issue to discuss any significant work - we would hate for your time to be wasted. 29 | 30 | To send us a pull request, please: 31 | 32 | 1. Fork the repository. 33 | 2. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change. 34 | 3. Ensure local tests pass. 35 | 4. Commit to your fork using clear commit messages. 36 | 5. Send us a pull request, answering any default questions in the pull request interface. 37 | 6. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation. 38 | 39 | GitHub provides additional document on [forking a repository](https://help.github.com/articles/fork-a-repo/) and 40 | [creating a pull request](https://help.github.com/articles/creating-a-pull-request/). 41 | 42 | 43 | ## Finding contributions to work on 44 | Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels (enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any 'help wanted' issues is a great place to start. 45 | 46 | 47 | ## Code of Conduct 48 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 49 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 50 | opensource-codeofconduct@amazon.com with any additional questions or comments. 51 | 52 | 53 | ## Security issue notifications 54 | If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github issue. 55 | 56 | 57 | ## Licensing 58 | 59 | See the [LICENSE](LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution. 60 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining a copy of 4 | this software and associated documentation files (the "Software"), to deal in 5 | the Software without restriction, including without limitation the rights to 6 | use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of 7 | the Software, and to permit persons to whom the Software is furnished to do so. 8 | 9 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 10 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS 11 | FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 12 | COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER 13 | IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN 14 | CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 15 | 16 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ## Run computer vision inference on large videos with Amazon SageMaker asynchronous endpoints 2 | 3 | Associated blog: https://aws.amazon.com/blogs/machine-learning/run-computer-vision-inference-on-large-videos-with-amazon-sagemaker-asynchronous-endpoints/ 4 | 5 | In this sample, we serve a PyTorch Computer Vision model with SageMaker asynchronous inference endpoints to process a burst of traffic of large input payload videos. We demonstrate the new capabilities of an internal queue with user defined concurrency and completion notifications. We configure autoscaling of instances including scaling down to 0 when traffic subsides and scale back up as the request queue fills up. We use SageMaker’s pre-built TorchServe container with a custom inference script for preprocessing the videos before model invocation. 6 | 1. Large payload input of a high resolution video segment of 70 MB 7 | 2. Large payload output from a PyTorch pre-trained mask-rcnn model 8 | 3. Large response time from the model of 30 seconds on a gpu instance 9 | 4. Auto-queuing of inference requests with asyncrhonous inference 10 | 5. Notifications of completed requests via SNS 11 | 6. Auto-scaling of endpoints based on queue length metric with minimum value set to 0 instances 12 | 13 | ![Async_Diagram](https://user-images.githubusercontent.com/8871432/133468905-c0600795-b69a-4112-9777-3a4cd79c494c.png) 14 | 15 | 16 | ## Security 17 | 18 | See [CONTRIBUTING](CONTRIBUTING.md#security-issue-notifications) for more information. 19 | 20 | ## License 21 | 22 | This library is licensed under the MIT-0 License. See the LICENSE file. 23 | 24 | -------------------------------------------------------------------------------- /code/.ipynb_checkpoints/inference-checkpoint.py: -------------------------------------------------------------------------------- 1 | # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | # SPDX-License-Identifier: MIT-0 3 | 4 | import logging 5 | import json 6 | import torch 7 | import numpy as np 8 | import torchvision.transforms as transforms 9 | from six import BytesIO 10 | import io 11 | from PIL import Image 12 | import cv2 13 | import tempfile 14 | 15 | 16 | 17 | logger = logging.getLogger(__name__) 18 | 19 | 20 | def model_fn(model_dir): 21 | device = get_device() 22 | print('device is') 23 | print(device) 24 | model = torch.load(model_dir + '/model.pth', map_location=torch.device(device)) 25 | print(type(model)) 26 | return model 27 | 28 | 29 | def input_fn(request_body, request_content_type): 30 | frame_width = 1024 31 | frame_height = 1024 32 | interval = 30 33 | f = io.BytesIO(request_body) 34 | tfile = tempfile.NamedTemporaryFile(delete=False) 35 | tfile.write(f.read()) 36 | print(tfile.name) 37 | video_frames = video2frame(tfile,frame_width, frame_height, interval) 38 | #convert to tensor of float32 type 39 | transform = transforms.Compose([ 40 | transforms.Lambda(lambda video_frames: torch.stack([transforms.ToTensor()(frame) for frame in video_frames])) # returns a 4D tensor 41 | ]) 42 | image_tensors = transform(video_frames) 43 | 44 | return image_tensors 45 | 46 | def predict_fn(data, model): 47 | print('in custom predict function') 48 | with torch.no_grad(): 49 | device = get_device() 50 | model = model.to(device) 51 | input_data = data.to(device) 52 | model.eval() 53 | output = model(input_data) 54 | 55 | return output 56 | 57 | 58 | def output_fn(output_batch, accept='application/json'): 59 | res = [] 60 | print('output list length') 61 | print(len(output_batch)) 62 | for output in output_batch: 63 | res.append({'boxes':output['boxes'].detach().numpy().tolist(),'labels':output['labels'].detach().numpy().tolist(),'scores':output['scores'].detach().numpy().tolist()}) 64 | 65 | return json.dumps(res) 66 | 67 | def get_device(): 68 | device = 'cuda:0' if torch.cuda.is_available() else 'cpu' 69 | return device 70 | 71 | 72 | def video2frame( 73 | tfile,frame_width, frame_height, interval): 74 | """ 75 | Extract frame from video by interval 76 | :param video_src_path: video src path 77 | :param video:　video file name 78 | :param frame_width:　frame width 79 | :param frame_height:　frame height 80 | :param interval:　interval for frame to extract 81 | :return:　list of numpy.ndarray 82 | """ 83 | video_frames = [] 84 | cap = cv2.VideoCapture(tfile.name) 85 | frame_index = 0 86 | frame_count = 0 87 | if cap.isOpened(): 88 | success = True 89 | else: 90 | success = False 91 | print("Read failed!") 92 | 93 | while success: 94 | success, frame = cap.read() 95 | 96 | if frame_index % interval == 0: 97 | print("---> Reading the %d frame:" % frame_index, success) 98 | resize_frame = cv2.resize( 99 | frame, (frame_width, frame_height), interpolation=cv2.INTER_AREA 100 | ) 101 | video_frames.append(resize_frame) 102 | frame_count += 1 103 | 104 | frame_index += 1 105 | 106 | cap.release() 107 | print('Number of frames') 108 | print(frame_count) 109 | return video_frames 110 | -------------------------------------------------------------------------------- /code/README.md: -------------------------------------------------------------------------------- 1 | SageMaker asynchronous inference - video processing with a pytorch mask-rcnn model 2 | input payload is a 70 mb video, 3 | output payload is bounding boxes, labels and scores for every sampled frame in the video, 4 | model response time is approx 40 seconds for 1 FPS sampling rate 5 | 6 | 7 | 1. Demonstrate a burst of incoming requests beyond the acceptable threshold and automatic queuing of the jobs and completion status notification (In the performance section of the blog, we identify 300 requests per minute as the threshold for a single endpoint ) 8 | 2. Demonstrate scale down of resources when they are no incoming requests for ‘x’ minutes 9 | 10 | 1. S3 input and output payload 11 | 2. Autoqueueing 12 | 3. Concurrency 13 | 4. Autoscaling 14 | 5. Scale resources to 0 15 | 6. Notifications 16 | 7. Cloud watch metrics 17 | 8. Results - invocations, queue size, throughput 18 | 19 | 20 | 21 | 22 | -------------------------------------------------------------------------------- /code/inference.py: -------------------------------------------------------------------------------- 1 | # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | # SPDX-License-Identifier: MIT-0 3 | 4 | import io 5 | import json 6 | import logging 7 | import os 8 | import tempfile 9 | 10 | import cv2 11 | import torch 12 | import torchvision.transforms as transforms 13 | 14 | # This code will be loaded on each worker separately.. 15 | logger = logging.getLogger(__name__) 16 | logger.setLevel(logging.DEBUG) 17 | 18 | 19 | def model_fn(model_dir): 20 | device = get_device() 21 | logger.info(">>> Device is '%s'.." % device) 22 | model = torch.load(model_dir + '/model.pth', map_location=torch.device(device)) 23 | print(type(model)) 24 | logger.info(">>> Model loaded!..") 25 | return model 26 | 27 | def transform_fn(model, request_body, content_type, accept): 28 | interval = int(os.environ.get('FRAME_INTERVAL', 30)) 29 | frame_width = int(os.environ.get('FRAME_WIDTH', 1024)) 30 | frame_height = int(os.environ.get('FRAME_HEIGHT', 1024)) 31 | batch_size = int(os.environ.get('BATCH_SIZE', 24)) 32 | 33 | f = io.BytesIO(request_body) 34 | tfile = tempfile.NamedTemporaryFile(delete=False) 35 | tfile.write(f.read()) 36 | 37 | all_predictions = [] 38 | 39 | for batch_frames in batch_generator(tfile, frame_width, frame_height, interval, batch_size): 40 | batch_inputs = preprocess(batch_frames) # returns 4D tensor 41 | batch_outputs = predict(batch_inputs, model) 42 | logger.info(">>> Length of batch predictions: %d" % len(batch_outputs)) 43 | batch_predictions = postprocess(batch_outputs) 44 | all_predictions.extend(batch_predictions) 45 | 46 | logger.info(">>> Length of final predictions: %d" % len(all_predictions)) 47 | return json.dumps(all_predictions) 48 | 49 | def preprocess(inputs, preprocessor=transforms.ToTensor()): 50 | outputs = torch.stack([preprocessor(frame) for frame in inputs]) 51 | return outputs 52 | 53 | def predict(inputs, model): 54 | logger.info(">>> Invoking model!..") 55 | 56 | with torch.no_grad(): 57 | device = get_device() 58 | model = model.to(device) 59 | input_data = inputs.to(device) 60 | model.eval() 61 | outputs = model(input_data) 62 | 63 | return outputs 64 | 65 | def postprocess(inputs): 66 | outputs = [] 67 | for inp in inputs: 68 | outputs.append({ 69 | 'boxes': inp['boxes'].detach().cpu().numpy().tolist(), 70 | 'labels': inp['labels'].detach().cpu().numpy().tolist(), 71 | 'scores': inp['scores'].detach().cpu().numpy().tolist() 72 | }) 73 | return outputs 74 | 75 | def get_device(): 76 | device = 'cuda:0' if torch.cuda.is_available() else 'cpu' 77 | return device 78 | 79 | def batch_generator(tfile, frame_width, frame_height, interval, batch_size): 80 | cap = cv2.VideoCapture(tfile.name) 81 | frame_index = 0 82 | frame_buffer = [] 83 | 84 | while cap.isOpened(): 85 | 86 | success, frame = cap.read() 87 | 88 | if not success: 89 | cap.release() 90 | if frame_buffer: 91 | yield frame_buffer 92 | return 93 | 94 | if frame_index % interval == 0: 95 | frame_resized = cv2.resize(frame, (frame_width, frame_height), interpolation=cv2.INTER_AREA) 96 | frame_buffer.append(frame_resized) 97 | 98 | if len(frame_buffer) == batch_size: 99 | yield frame_buffer 100 | frame_buffer.clear() 101 | 102 | frame_index += 1 103 | else: 104 | raise Exception("Failed to open video '%s'!.." % tfile.name) 105 | 106 | 107 | -------------------------------------------------------------------------------- /code/requirements.txt: -------------------------------------------------------------------------------- 1 | opencv-python -------------------------------------------------------------------------------- /images/Async_Diagram.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-sagemaker-asynchronous-inference-computer-vision/d2f1251712f9e9e56559b3326b4011aa43fb1a1d/images/Async_Diagram.png -------------------------------------------------------------------------------- /images/backlog_size_metrics.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-sagemaker-asynchronous-inference-computer-vision/d2f1251712f9e9e56559b3326b4011aa43fb1a1d/images/backlog_size_metrics.png -------------------------------------------------------------------------------- /images/birds.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-sagemaker-asynchronous-inference-computer-vision/d2f1251712f9e9e56559b3326b4011aa43fb1a1d/images/birds.jpg -------------------------------------------------------------------------------- /images/instance_count_0.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-sagemaker-asynchronous-inference-computer-vision/d2f1251712f9e9e56559b3326b4011aa43fb1a1d/images/instance_count_0.png -------------------------------------------------------------------------------- /images/invocations_metrics.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-sagemaker-asynchronous-inference-computer-vision/d2f1251712f9e9e56559b3326b4011aa43fb1a1d/images/invocations_metrics.png -------------------------------------------------------------------------------- /images/model_latency_metrics.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-sagemaker-asynchronous-inference-computer-vision/d2f1251712f9e9e56559b3326b4011aa43fb1a1d/images/model_latency_metrics.png -------------------------------------------------------------------------------- /mask-rcnn-async-inference.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "\n", 8 | "## Execute computer vision inference on large videos using managed queues, notifications, and automatic scaling with Amazon SageMaker Asynchronous Endpoints" 9 | ] 10 | }, 11 | { 12 | "cell_type": "markdown", 13 | "metadata": {}, 14 | "source": [ 15 | "### Table of Contents" 16 | ] 17 | }, 18 | { 19 | "cell_type": "markdown", 20 | "metadata": {}, 21 | "source": [ 22 | "* Background\n", 23 | "* Setup\n", 24 | "* Download and trigger pre-trained maskrcnn model on a sample image\n", 25 | "* Create model archive and upload to S3\n", 26 | "* Create SageMaker model with PyTorch inference container\n", 27 | "* Real time hosted endpoint deployment and inference\n", 28 | "* Create Asynchronous inference endpoints\n", 29 | "* Invoke asynchronous endpoint\n", 30 | "* Enable autoscaling\n", 31 | "* Cleanup" 32 | ] 33 | }, 34 | { 35 | "cell_type": "markdown", 36 | "metadata": {}, 37 | "source": [ 38 | "### Background" 39 | ] 40 | }, 41 | { 42 | "cell_type": "markdown", 43 | "metadata": {}, 44 | "source": [ 45 | "AWS customers are increasingly using computer vision (CV) models on large input payloads that can take a few minutes of processing time. For example, space technology companies work with a stream of high resolution satellite imagery to detect particular objects of interest. Similarly, healthcare companies process high resolution biomedical images or videos like echocardiograms to detect anomalies. Also, media companies scan images and videos uploaded by their customers to ensure they are compliant and without copyright violations. These applications receive a burst of incoming traffic at different times in the day and require near real time processing with completion notifications at a low cost. \n", 46 | "\n", 47 | "In this notebook, we serve a PyTorch Computer Vision model with SageMaker asynchronous inference endpoints to process a burst of traffic of large input payload videos. We demonstrate the new capabilities of an internal queue with user defined concurrency and completion notifications. We configure autoscaling of instances including scaling down to 0 when traffic subsides and scale back up as the request queue fills up. We use [SageMaker’s pre-built TorchServe container](https://sagemaker.readthedocs.io/en/stable/frameworks/pytorch/using_pytorch.html) with a custom inference script for preprocessing the videos before model invocation. \n", 48 | "\n", 49 | " 1. Large payload input of a high resolution video segment of 70 MB\n", 50 | " 2. Large payload output from a PyTorch pre-trained mask-rcnn model \n", 51 | " 3. Large response time from the model of 30 seconds on a gpu instance\n", 52 | " 4. Auto-queuing of inference requests with asynchronous inference\n", 53 | " 5. Notifications of completed requests via SNS \n", 54 | " 6. Auto-scaling of endpoints based on queue length metric with minimum value set to 0" 55 | ] 56 | }, 57 | { 58 | "cell_type": "markdown", 59 | "metadata": {}, 60 | "source": [ 61 | "![Workflow](images/Async_Diagram.png)" 62 | ] 63 | }, 64 | { 65 | "cell_type": "markdown", 66 | "metadata": {}, 67 | "source": [ 68 | "### Setup" 69 | ] 70 | }, 71 | { 72 | "cell_type": "markdown", 73 | "metadata": {}, 74 | "source": [ 75 | "\n", 76 | "If you run this notebook in SageMaker Studio, you need to make sure ipywidgets is installed and restart the kernel, so please uncomment the code in the next cell, and run it." 77 | ] 78 | }, 79 | { 80 | "cell_type": "code", 81 | "execution_count": null, 82 | "metadata": {}, 83 | "outputs": [], 84 | "source": [ 85 | "# %%capture\n", 86 | "# import IPython\n", 87 | "# import sys\n", 88 | "\n", 89 | "# !{sys.executable} -m pip install ipywidgets\n", 90 | "# IPython.Application.instance().kernel.do_shutdown(True) # has to restart kernel so changes are used" 91 | ] 92 | }, 93 | { 94 | "cell_type": "code", 95 | "execution_count": null, 96 | "metadata": {}, 97 | "outputs": [], 98 | "source": [ 99 | "!python -m pip install --upgrade pip --quiet\n", 100 | "!pip install -U awscli --quiet\n", 101 | "!pip install torch==1.8.0 --quiet \n", 102 | "!pip install torchvision==0.9.0 --quiet \n", 103 | "!pip install -U sagemaker --quiet" 104 | ] 105 | }, 106 | { 107 | "cell_type": "code", 108 | "execution_count": null, 109 | "metadata": {}, 110 | "outputs": [], 111 | "source": [ 112 | "import torch\n", 113 | "import torchvision\n", 114 | "import torchvision.models as models\n", 115 | "import sagemaker\n", 116 | "from sagemaker import get_execution_role\n", 117 | "from sagemaker.utils import name_from_base\n", 118 | "from sagemaker.pytorch import PyTorchModel\n", 119 | "import boto3\n", 120 | "import datetime\n", 121 | "import time\n", 122 | "from time import strftime,gmtime\n", 123 | "import json\n", 124 | "import os\n", 125 | "import urllib\n", 126 | "import sys\n", 127 | "import io\n", 128 | "\n", 129 | "role = get_execution_role()\n", 130 | "boto_session = boto3.session.Session()\n", 131 | "sm_session = sagemaker.session.Session()\n", 132 | "sm_client = boto_session.client(\"sagemaker\")\n", 133 | "sm_runtime = boto_session.client(\"sagemaker-runtime\")\n", 134 | "sns_client = boto3.client('sns')\n", 135 | "region = boto_session.region_name\n", 136 | "bucket = sm_session.default_bucket()\n", 137 | "prefix = 'async-inference-maskrcnn'\n", 138 | "\n", 139 | "print(region)\n", 140 | "print(role)\n", 141 | "print(bucket)\n", 142 | "print(prefix)" 143 | ] 144 | }, 145 | { 146 | "cell_type": "code", 147 | "execution_count": null, 148 | "metadata": {}, 149 | "outputs": [], 150 | "source": [ 151 | "print(torch.__version__)\n", 152 | "print(torchvision.__version__)" 153 | ] 154 | }, 155 | { 156 | "cell_type": "markdown", 157 | "metadata": {}, 158 | "source": [ 159 | "Specify your IAM role. Go the AWS IAM console (https://console.aws.amazon.com/iam/home) and add the following policies to your IAM Role:\n", 160 | "* SageMakerFullAccessPolicy\n", 161 | "* Amazon S3 access: Apply this to get and put objects in your Amazon S3 bucket. Replace `bucket_name` with the name of your Amazon S3 bucket: \n", 162 | "\n", 163 | "```json\n", 164 | "{\n", 165 | " \"Version\": \"2012-10-17\",\n", 166 | " \"Statement\": [\n", 167 | " {\n", 168 | " \"Action\": [\n", 169 | " \"s3:GetObject\",\n", 170 | " \"s3:PutObject\",\n", 171 | " \"s3:AbortMultipartUpload\",\n", 172 | " \"s3:ListBucket\"\n", 173 | " ],\n", 174 | " \"Effect\": \"Allow\",\n", 175 | " \"Resource\": \"arn:aws:s3:::bucket_name/*\"\n", 176 | " }\n", 177 | " ]\n", 178 | "}\n", 179 | "```\n", 180 | "\n", 181 | "* (Optional) Amazon SNS access: Add `sns:Publish` on the topics you define. Apply this if you plan to use Amazon SNS to receive notifications.\n", 182 | "\n", 183 | "```json\n", 184 | "{\n", 185 | " \"Version\": \"2012-10-17\",\n", 186 | " \"Statement\": [\n", 187 | " {\n", 188 | " \"Action\": [\n", 189 | " \"sns:Publish\"\n", 190 | " ],\n", 191 | " \"Effect\": \"Allow\",\n", 192 | " \"Resource\": \"arn:aws:sns:us-east-2:123456789012:MyTopic\"\n", 193 | " }\n", 194 | " ]\n", 195 | "}\n", 196 | "```\n", 197 | "\n", 198 | "* (Optional) KMS decrypt, encrypt if your Amazon S3 bucket is encrypte." 199 | ] 200 | }, 201 | { 202 | "cell_type": "markdown", 203 | "metadata": {}, 204 | "source": [ 205 | "### Download and trigger maskrcnn model on a sample image" 206 | ] 207 | }, 208 | { 209 | "cell_type": "markdown", 210 | "metadata": {}, 211 | "source": [ 212 | "Using pre-trained maskrcnn resnet 50 model" 213 | ] 214 | }, 215 | { 216 | "cell_type": "code", 217 | "execution_count": null, 218 | "metadata": {}, 219 | "outputs": [], 220 | "source": [ 221 | "model = models.detection.maskrcnn_resnet50_fpn(pretrained=True)" 222 | ] 223 | }, 224 | { 225 | "cell_type": "code", 226 | "execution_count": null, 227 | "metadata": {}, 228 | "outputs": [], 229 | "source": [ 230 | "import matplotlib.pyplot as plt\n", 231 | "import numpy as np\n", 232 | "import torchvision.transforms.functional as F\n", 233 | "\n", 234 | "plt.rcParams[\"savefig.bbox\"] = 'tight'\n", 235 | "\n", 236 | "def show(imgs):\n", 237 | " if not isinstance(imgs, list):\n", 238 | " imgs = [imgs]\n", 239 | " fix, axs = plt.subplots(ncols=len(imgs), squeeze=False)\n", 240 | " for i, img in enumerate(imgs):\n", 241 | " img = img.detach()\n", 242 | " img = F.to_pil_image(img)\n", 243 | " \n", 244 | " axs[0, i].imshow(np.asarray(img))\n", 245 | " axs[0, i].set(xticklabels=[], yticklabels=[], xticks=[], yticks=[])" 246 | ] 247 | }, 248 | { 249 | "cell_type": "code", 250 | "execution_count": null, 251 | "metadata": {}, 252 | "outputs": [], 253 | "source": [ 254 | "from PIL import Image\n", 255 | "import torchvision.transforms as transforms\n", 256 | "\n", 257 | "img = io.BytesIO(open('images/birds.jpg', 'rb').read())\n", 258 | "birds_image = Image.open(img).convert('RGB')\n", 259 | "birds_image = transforms.ToTensor()(birds_image)\n", 260 | "print(type(birds_image))" 261 | ] 262 | }, 263 | { 264 | "cell_type": "code", 265 | "execution_count": null, 266 | "metadata": {}, 267 | "outputs": [], 268 | "source": [ 269 | "model.eval()\n", 270 | "outputs = model([birds_image])\n", 271 | "output = outputs[0]" 272 | ] 273 | }, 274 | { 275 | "cell_type": "code", 276 | "execution_count": null, 277 | "metadata": {}, 278 | "outputs": [], 279 | "source": [ 280 | "#Print output\n", 281 | "res = []\n", 282 | "for output in outputs:\n", 283 | " res.append({'boxes':output['boxes'].detach().numpy().tolist(),'labels':output['labels'].detach().numpy().tolist(),'scores':output['scores'].detach().numpy().tolist()})\n", 284 | "print(json.dumps(res))" 285 | ] 286 | }, 287 | { 288 | "cell_type": "code", 289 | "execution_count": null, 290 | "metadata": {}, 291 | "outputs": [], 292 | "source": [ 293 | "#Print original image\n", 294 | "from torchvision.utils import make_grid\n", 295 | "grid = make_grid([birds_image])\n", 296 | "show(grid)" 297 | ] 298 | }, 299 | { 300 | "cell_type": "markdown", 301 | "metadata": {}, 302 | "source": [ 303 | "Print original image with model output boxes" 304 | ] 305 | }, 306 | { 307 | "cell_type": "code", 308 | "execution_count": null, 309 | "metadata": {}, 310 | "outputs": [], 311 | "source": [ 312 | "from torchvision.utils import draw_bounding_boxes\n", 313 | "from torchvision.io import read_image\n", 314 | "score_threshold = .9\n", 315 | "birds_with_boxes = [\n", 316 | " draw_bounding_boxes(read_image('images/birds.jpg'), boxes=output['boxes'][output['scores'] > score_threshold], width=10)\n", 317 | "]\n", 318 | "show(birds_with_boxes)" 319 | ] 320 | }, 321 | { 322 | "cell_type": "markdown", 323 | "metadata": {}, 324 | "source": [ 325 | "### Create model archive and upload to S3" 326 | ] 327 | }, 328 | { 329 | "cell_type": "code", 330 | "execution_count": null, 331 | "metadata": {}, 332 | "outputs": [], 333 | "source": [ 334 | "!mkdir model_and_code\n", 335 | "torch.save(model, 'model_and_code/model.pth')\n", 336 | "!mkdir model_and_code/code\n", 337 | "!cp ./code/* model_and_code/code\n", 338 | "!tar cvzf model.tar.gz -C model_and_code/ . " 339 | ] 340 | }, 341 | { 342 | "cell_type": "code", 343 | "execution_count": null, 344 | "metadata": {}, 345 | "outputs": [], 346 | "source": [ 347 | "from sagemaker.s3 import S3Uploader\n", 348 | "file_key = 'model.tar.gz'\n", 349 | "model_artifact = S3Uploader.upload(file_key,'s3://{}/{}/model'.format(bucket, prefix))\n", 350 | "print(model_artifact)" 351 | ] 352 | }, 353 | { 354 | "cell_type": "markdown", 355 | "metadata": {}, 356 | "source": [ 357 | "### Create SageMaker model with PyTorch inference container" 358 | ] 359 | }, 360 | { 361 | "cell_type": "code", 362 | "execution_count": null, 363 | "metadata": {}, 364 | "outputs": [], 365 | "source": [ 366 | "from sagemaker.image_uris import retrieve\n", 367 | "\n", 368 | "deploy_instance_type = 'ml.g4dn.xlarge'\n", 369 | "pytorch_inference_image_uri = retrieve('pytorch',\n", 370 | " region,\n", 371 | " version='1.7.1',\n", 372 | " py_version='py3',\n", 373 | " instance_type = deploy_instance_type,\n", 374 | " accelerator_type=None,\n", 375 | " image_scope='inference')\n", 376 | "print(pytorch_inference_image_uri)" 377 | ] 378 | }, 379 | { 380 | "cell_type": "code", 381 | "execution_count": null, 382 | "metadata": {}, 383 | "outputs": [], 384 | "source": [ 385 | "container = pytorch_inference_image_uri\n", 386 | "model_name = 'sagemaker-maskrcnn-{0}'.format(str(int(time.time())))\n", 387 | "print(container)\n", 388 | "print(model_name)\n", 389 | "\n", 390 | "create_model_response = sm_client.create_model(\n", 391 | " ModelName = model_name,\n", 392 | " ExecutionRoleArn = role,\n", 393 | " PrimaryContainer = {\n", 394 | " 'Image': container,\n", 395 | " 'ModelDataUrl': model_artifact,\n", 396 | " 'Environment': {\n", 397 | " 'TS_MAX_REQUEST_SIZE': '100000000', #default max request size is 6 Mb for torchserve, need to update it to support the 70 mb input payload\n", 398 | " 'TS_MAX_RESPONSE_SIZE': '100000000',\n", 399 | " 'TS_DEFAULT_RESPONSE_TIMEOUT': '1000'\n", 400 | " }\n", 401 | " }, \n", 402 | ")" 403 | ] 404 | }, 405 | { 406 | "cell_type": "markdown", 407 | "metadata": {}, 408 | "source": [ 409 | "### Real time hosted endpoint deployment and inference" 410 | ] 411 | }, 412 | { 413 | "cell_type": "markdown", 414 | "metadata": {}, 415 | "source": [ 416 | "Create an endpoint config name. Here we create one based on the date so it we can search endpoints based on creation time." 417 | ] 418 | }, 419 | { 420 | "cell_type": "code", 421 | "execution_count": null, 422 | "metadata": {}, 423 | "outputs": [], 424 | "source": [ 425 | "print(model_name)\n", 426 | "endpoint_config_name = f\"maskrcnnEndpointConfig-{strftime('%Y-%m-%d-%H-%M-%S', gmtime())}\"\n", 427 | "create_endpoint_config_response = sm_client.create_endpoint_config(\n", 428 | " EndpointConfigName=endpoint_config_name,\n", 429 | " ProductionVariants=[\n", 430 | " {\n", 431 | " \"VariantName\": \"variant1\",\n", 432 | " \"ModelName\": model_name,\n", 433 | " \"InstanceType\": \"ml.g4dn.xlarge\",\n", 434 | " \"InitialInstanceCount\": 1\n", 435 | " }\n", 436 | " ]\n", 437 | ")\n", 438 | "print(f\"Created EndpointConfig: {create_endpoint_config_response['EndpointConfigArn']}\")" 439 | ] 440 | }, 441 | { 442 | "cell_type": "code", 443 | "execution_count": null, 444 | "metadata": {}, 445 | "outputs": [], 446 | "source": [ 447 | "endpoint_name = f\"sm-{strftime('%Y-%m-%d-%H-%M-%S', gmtime())}\"\n", 448 | "response = sm_client.create_endpoint(\n", 449 | " EndpointName=endpoint_name,\n", 450 | " EndpointConfigName=endpoint_config_name\n", 451 | ")" 452 | ] 453 | }, 454 | { 455 | "cell_type": "code", 456 | "execution_count": null, 457 | "metadata": {}, 458 | "outputs": [], 459 | "source": [ 460 | "waiter = boto3.client('sagemaker').get_waiter('endpoint_in_service')\n", 461 | "print(\"Waiting for endpoint to create...\")\n", 462 | "waiter.wait(EndpointName=endpoint_name)\n", 463 | "resp = sm_client.describe_endpoint(EndpointName=endpoint_name)\n", 464 | "print(f\"Endpoint Status: {resp['EndpointStatus']}\")" 465 | ] 466 | }, 467 | { 468 | "cell_type": "code", 469 | "execution_count": null, 470 | "metadata": {}, 471 | "outputs": [], 472 | "source": [ 473 | "ep= endpoint_name\n", 474 | "feed_data = open('images/birds.jpg', 'rb')\n", 475 | "sm_runtime = boto3.Session().client('sagemaker-runtime')\n", 476 | "r = sm_runtime.invoke_endpoint(EndpointName=ep, Body=feed_data)" 477 | ] 478 | }, 479 | { 480 | "cell_type": "code", 481 | "execution_count": null, 482 | "metadata": {}, 483 | "outputs": [], 484 | "source": [ 485 | "print(r['Body'].read())" 486 | ] 487 | }, 488 | { 489 | "cell_type": "markdown", 490 | "metadata": {}, 491 | "source": [ 492 | "### Create Asynchronous inference endpoints" 493 | ] 494 | }, 495 | { 496 | "cell_type": "markdown", 497 | "metadata": {}, 498 | "source": [ 499 | "Unlike real time hosted endpoints, asynchronous endpoints support scaling\n", 500 | "down instances to 0 by setting the minimum capacity to 0. With this feature, we can scale\n", 501 | "down to 0 instances when there is no traffic and pay only when the payloads arrive. Let's create an asynchronous endpoint to see it in action below -" 502 | ] 503 | }, 504 | { 505 | "cell_type": "code", 506 | "execution_count": null, 507 | "metadata": {}, 508 | "outputs": [], 509 | "source": [ 510 | "bucket_prefix = \"async-inference-blog\"\n", 511 | "resource_name = \"AsyncInferenceDemo-SNS\"" 512 | ] 513 | }, 514 | { 515 | "cell_type": "markdown", 516 | "metadata": {}, 517 | "source": [ 518 | "Create Error and Success SNS topics" 519 | ] 520 | }, 521 | { 522 | "cell_type": "code", 523 | "execution_count": null, 524 | "metadata": {}, 525 | "outputs": [], 526 | "source": [ 527 | "response = sns_client.create_topic(Name=\"Async-Demo-ErrorTopic\")\n", 528 | "error_topic= response['TopicArn']\n", 529 | "print(error_topic)" 530 | ] 531 | }, 532 | { 533 | "cell_type": "code", 534 | "execution_count": null, 535 | "metadata": {}, 536 | "outputs": [], 537 | "source": [ 538 | "response = sns_client.create_topic(Name=\"Async-Demo-SuccessTopic\")\n", 539 | "success_topic = response['TopicArn']\n", 540 | "print(success_topic)" 541 | ] 542 | }, 543 | { 544 | "cell_type": "markdown", 545 | "metadata": {}, 546 | "source": [ 547 | "List SNS topics" 548 | ] 549 | }, 550 | { 551 | "cell_type": "code", 552 | "execution_count": null, 553 | "metadata": {}, 554 | "outputs": [], 555 | "source": [ 556 | "response = sns_client.list_topics()\n", 557 | "topics = response[\"Topics\"]\n", 558 | "print(topics)" 559 | ] 560 | }, 561 | { 562 | "cell_type": "markdown", 563 | "metadata": {}, 564 | "source": [ 565 | "Optionally Subscribe to an SNS topic" 566 | ] 567 | }, 568 | { 569 | "cell_type": "code", 570 | "execution_count": null, 571 | "metadata": {}, 572 | "outputs": [], 573 | "source": [ 574 | "#Note: Replace with your email id\n", 575 | "\n", 576 | "# email_id = 'your-email@domain-name.com'\n", 577 | "# email_sub_1 = sns_client.subscribe(\n", 578 | "# TopicArn=success_topic,\n", 579 | "# Protocol='email',\n", 580 | "# Endpoint=email_id)\n", 581 | "\n", 582 | "# email_sub_2 = sns_client.subscribe(\n", 583 | "# TopicArn=error_topic,\n", 584 | "# Protocol='email',\n", 585 | "# Endpoint=email_id)\n", 586 | "\n", 587 | "#Note: You will need to confirm by clicking on the email you recieve to complete the subscription" 588 | ] 589 | }, 590 | { 591 | "cell_type": "markdown", 592 | "metadata": {}, 593 | "source": [ 594 | "Create an endpoint config name. Here we create one based on the date so it we can search endpoints based on creation time." 595 | ] 596 | }, 597 | { 598 | "cell_type": "code", 599 | "execution_count": null, 600 | "metadata": {}, 601 | "outputs": [], 602 | "source": [ 603 | "print(model_name)\n", 604 | "endpoint_config_name = f\"PyTorchAsyncEndpointConfig-{strftime('%Y-%m-%d-%H-%M-%S', gmtime())}\"\n", 605 | "create_endpoint_config_response = sm_client.create_endpoint_config(\n", 606 | " EndpointConfigName=endpoint_config_name,\n", 607 | " ProductionVariants=[\n", 608 | " {\n", 609 | " \"VariantName\": \"variant1\",\n", 610 | " \"ModelName\": model_name,\n", 611 | " \"InstanceType\": \"ml.g4dn.xlarge\",\n", 612 | " \"InitialInstanceCount\": 1\n", 613 | " }\n", 614 | " ],\n", 615 | " AsyncInferenceConfig={\n", 616 | " \"OutputConfig\": {\n", 617 | " \"S3OutputPath\": f\"s3://{bucket}/{bucket_prefix}/output\",\n", 618 | " # Optionally specify Amazon SNS topics\n", 619 | " \"NotificationConfig\": {\n", 620 | " \"SuccessTopic\": success_topic,\n", 621 | " \"ErrorTopic\": error_topic,\n", 622 | " }\n", 623 | " },\n", 624 | " \"ClientConfig\": {\n", 625 | " \"MaxConcurrentInvocationsPerInstance\": 2\n", 626 | " }\n", 627 | " }\n", 628 | ")\n", 629 | "print(f\"Created EndpointConfig: {create_endpoint_config_response['EndpointConfigArn']}\")" 630 | ] 631 | }, 632 | { 633 | "cell_type": "code", 634 | "execution_count": null, 635 | "metadata": {}, 636 | "outputs": [], 637 | "source": [ 638 | "endpoint_name = f\"sm-{strftime('%Y-%m-%d-%H-%M-%S', gmtime())}\"\n", 639 | "create_endpoint_response = sm_client.create_endpoint(EndpointName=endpoint_name, EndpointConfigName=endpoint_config_name)\n", 640 | "print(f\"Creating Endpoint: {create_endpoint_response['EndpointArn']}\")" 641 | ] 642 | }, 643 | { 644 | "cell_type": "code", 645 | "execution_count": null, 646 | "metadata": {}, 647 | "outputs": [], 648 | "source": [ 649 | "waiter = boto3.client('sagemaker').get_waiter('endpoint_in_service')\n", 650 | "print(\"Waiting for endpoint to create...\")\n", 651 | "waiter.wait(EndpointName=endpoint_name)\n", 652 | "resp = sm_client.describe_endpoint(EndpointName=endpoint_name)\n", 653 | "print(f\"Endpoint Status: {resp['EndpointStatus']}\")" 654 | ] 655 | }, 656 | { 657 | "cell_type": "markdown", 658 | "metadata": {}, 659 | "source": [ 660 | "### Upload input video file" 661 | ] 662 | }, 663 | { 664 | "cell_type": "code", 665 | "execution_count": null, 666 | "metadata": {}, 667 | "outputs": [], 668 | "source": [ 669 | "def upload_file(input_location):\n", 670 | " prefix = f\"{bucket_prefix}/input\"\n", 671 | " return sm_session.upload_data(\n", 672 | " input_location, \n", 673 | " bucket=sm_session.default_bucket(),\n", 674 | " key_prefix=prefix, \n", 675 | " extra_args={\"ContentType\": \"video/mp4\"})" 676 | ] 677 | }, 678 | { 679 | "cell_type": "code", 680 | "execution_count": null, 681 | "metadata": {}, 682 | "outputs": [], 683 | "source": [ 684 | "input_1_location = \"videos/ducks.mp4\"\n", 685 | "input_1_s3_location = upload_file(input_1_location)" 686 | ] 687 | }, 688 | { 689 | "cell_type": "code", 690 | "execution_count": null, 691 | "metadata": {}, 692 | "outputs": [], 693 | "source": [ 694 | "print(input_1_s3_location)" 695 | ] 696 | }, 697 | { 698 | "cell_type": "code", 699 | "execution_count": null, 700 | "metadata": {}, 701 | "outputs": [], 702 | "source": [ 703 | "print(endpoint_name)" 704 | ] 705 | }, 706 | { 707 | "cell_type": "markdown", 708 | "metadata": {}, 709 | "source": [ 710 | "### Invoke asynchronous endpoint" 711 | ] 712 | }, 713 | { 714 | "cell_type": "code", 715 | "execution_count": null, 716 | "metadata": {}, 717 | "outputs": [], 718 | "source": [ 719 | "response = sm_runtime.invoke_endpoint_async(\n", 720 | " EndpointName=endpoint_name, \n", 721 | " InputLocation=input_1_s3_location)\n", 722 | "output_location = response['OutputLocation']\n", 723 | "print(f\"OutputLocation: {output_location}\")" 724 | ] 725 | }, 726 | { 727 | "cell_type": "code", 728 | "execution_count": null, 729 | "metadata": {}, 730 | "outputs": [], 731 | "source": [ 732 | "from botocore.exceptions import ClientError\n", 733 | "\n", 734 | "def get_output(output_location):\n", 735 | " output_url = urllib.parse.urlparse(output_location)\n", 736 | " bucket = output_url.netloc\n", 737 | " key = output_url.path[1:]\n", 738 | " while True:\n", 739 | " try:\n", 740 | " return sm_session.read_s3_file(bucket=output_url.netloc, key_prefix=output_url.path[1:])\n", 741 | " except ClientError as e:\n", 742 | " if e.response['Error']['Code'] == 'NoSuchKey':\n", 743 | " print(\"waiting for output...\")\n", 744 | " time.sleep(2)\n", 745 | " continue\n", 746 | " raise\n" 747 | ] 748 | }, 749 | { 750 | "cell_type": "code", 751 | "execution_count": null, 752 | "metadata": {}, 753 | "outputs": [], 754 | "source": [ 755 | "output = get_output(output_location)\n", 756 | "print(f\"Output size in bytes: {((sys.getsizeof(output)))}\")" 757 | ] 758 | }, 759 | { 760 | "cell_type": "markdown", 761 | "metadata": {}, 762 | "source": [ 763 | "### Trigger 10 asynchronous requests on a single instance " 764 | ] 765 | }, 766 | { 767 | "cell_type": "code", 768 | "execution_count": null, 769 | "metadata": {}, 770 | "outputs": [], 771 | "source": [ 772 | "inferences = []\n", 773 | "for i in range(1,10):\n", 774 | " start = time.time()\n", 775 | " response = sm_runtime.invoke_endpoint_async(\n", 776 | " EndpointName=endpoint_name, \n", 777 | " InputLocation=input_1_s3_location)\n", 778 | " output_location = response[\"OutputLocation\"]\n", 779 | " inferences += [(input_1_s3_location, output_location)]\n", 780 | " time.sleep(0.5)\n", 781 | "print(\"\\Async invocations for Pytorch serving default: \\n\")\n", 782 | "\n", 783 | "for input_file, output_location in inferences:\n", 784 | " output = get_output(output_location)\n", 785 | " print(f\"Input File: {input_file}, Output location: {output_location}\")" 786 | ] 787 | }, 788 | { 789 | "cell_type": "markdown", 790 | "metadata": {}, 791 | "source": [ 792 | "### Enable autoscaling" 793 | ] 794 | }, 795 | { 796 | "cell_type": "code", 797 | "execution_count": null, 798 | "metadata": {}, 799 | "outputs": [], 800 | "source": [ 801 | "client = boto3.client('application-autoscaling') # Common class representing Application Auto Scaling for SageMaker amongst other services\n", 802 | "\n", 803 | "resource_id='endpoint/' + endpoint_name + '/variant/' + 'variant1' # This is the format in which application autoscaling references the endpoint\n", 804 | "\n", 805 | "response = client.register_scalable_target(\n", 806 | " ServiceNamespace='sagemaker', \n", 807 | " ResourceId=resource_id,\n", 808 | " ScalableDimension='sagemaker:variant:DesiredInstanceCount',\n", 809 | " MinCapacity=0, \n", 810 | " MaxCapacity=5\n", 811 | ")\n", 812 | "\n", 813 | "response = client.put_scaling_policy(\n", 814 | " PolicyName='Invocations-ScalingPolicy',\n", 815 | " ServiceNamespace='sagemaker', # The namespace of the AWS service that provides the resource. \n", 816 | " ResourceId=resource_id, # Endpoint name \n", 817 | " ScalableDimension='sagemaker:variant:DesiredInstanceCount', # SageMaker supports only Instance Count\n", 818 | " PolicyType='TargetTrackingScaling', # 'StepScaling'|'TargetTrackingScaling'\n", 819 | " TargetTrackingScalingPolicyConfiguration={\n", 820 | " 'TargetValue': 5.0, # The target value for the metric. \n", 821 | " 'CustomizedMetricSpecification': {\n", 822 | " 'MetricName': 'ApproximateBacklogSizePerInstance',\n", 823 | " 'Namespace': 'AWS/SageMaker',\n", 824 | " 'Dimensions': [\n", 825 | " {'Name': 'EndpointName', 'Value': endpoint_name }\n", 826 | " ],\n", 827 | " 'Statistic': 'Average',\n", 828 | " },\n", 829 | " 'ScaleInCooldown': 120, # The cooldown period helps you prevent your Auto Scaling group from launching or terminating \n", 830 | " # additional instances before the effects of previous activities are visible. \n", 831 | " # You can configure the length of time based on your instance startup time or other application needs.\n", 832 | " # ScaleInCooldown - The amount of time, in seconds, after a scale in activity completes before another scale in activity can start. \n", 833 | " 'ScaleOutCooldown': 120 # ScaleOutCooldown - The amount of time, in seconds, after a scale out activity completes before another scale out activity can start.\n", 834 | " \n", 835 | " # 'DisableScaleIn': True|False - ndicates whether scale in by the target tracking policy is disabled. \n", 836 | " # If the value is true , scale in is disabled and the target tracking policy won't remove capacity from the scalable resource.\n", 837 | " }\n", 838 | ")" 839 | ] 840 | }, 841 | { 842 | "cell_type": "code", 843 | "execution_count": null, 844 | "metadata": {}, 845 | "outputs": [], 846 | "source": [ 847 | "print(response)" 848 | ] 849 | }, 850 | { 851 | "cell_type": "markdown", 852 | "metadata": {}, 853 | "source": [ 854 | "### Trigger 1000 asynchronous invocations with autoscaling from 1 to 5 and then scale down to 0 on completion" 855 | ] 856 | }, 857 | { 858 | "cell_type": "markdown", 859 | "metadata": {}, 860 | "source": [ 861 | "Optionally [delete the SNS topic](https://boto3.amazonaws.com/v1/documentation/api/1.9.42/reference/services/sns.html#SNS.Client.delete_topic) to avoid flooding of notifications on 1000 invocations below " 862 | ] 863 | }, 864 | { 865 | "cell_type": "code", 866 | "execution_count": null, 867 | "metadata": {}, 868 | "outputs": [], 869 | "source": [ 870 | "print(endpoint_name)\n", 871 | "for i in range(1,1000):\n", 872 | " response = sm_runtime.invoke_endpoint_async(\n", 873 | " EndpointName=endpoint_name, \n", 874 | " InputLocation=input_1_s3_location)\n", 875 | "print(\"\\Async invocations for Pytorch serving with auotscaling \\n\")" 876 | ] 877 | }, 878 | { 879 | "cell_type": "markdown", 880 | "metadata": {}, 881 | "source": [ 882 | "Plot graphs from CloudWatch Metrics" 883 | ] 884 | }, 885 | { 886 | "cell_type": "code", 887 | "execution_count": null, 888 | "metadata": {}, 889 | "outputs": [], 890 | "source": [ 891 | "import pandas as pd\n", 892 | "cw = boto3.Session().client(\"cloudwatch\")" 893 | ] 894 | }, 895 | { 896 | "cell_type": "code", 897 | "execution_count": null, 898 | "metadata": {}, 899 | "outputs": [], 900 | "source": [ 901 | "import datetime\n", 902 | "from datetime import datetime,timedelta\n", 903 | "def get_sagemaker_metrics(endpoint_name,\n", 904 | " endpoint_config_name,\n", 905 | " variant_name,\n", 906 | " metric_name,\n", 907 | " statistic,\n", 908 | " start_time,\n", 909 | " end_time):\n", 910 | " dimensions = [\n", 911 | " {\n", 912 | " \"Name\": \"EndpointName\",\n", 913 | " \"Value\": endpoint_name\n", 914 | " },\n", 915 | " {\n", 916 | " \"Name\": \"VariantName\",\n", 917 | " \"Value\": variant_name\n", 918 | " }\n", 919 | " ]\n", 920 | " if endpoint_config_name is not None:\n", 921 | " dimensions.append({\n", 922 | " \"Name\": \"EndpointConfigName\",\n", 923 | " \"Value\": endpoint_config_name\n", 924 | " })\n", 925 | " metrics = cw.get_metric_statistics(\n", 926 | " Namespace=\"AWS/SageMaker\",\n", 927 | " MetricName=metric_name,\n", 928 | " StartTime=start_time,\n", 929 | " EndTime=end_time,\n", 930 | " Period=60,\n", 931 | " Statistics=[statistic],\n", 932 | " Dimensions=dimensions\n", 933 | " )\n", 934 | " rename = endpoint_config_name if endpoint_config_name is not None else 'ALL'\n", 935 | " return pd.DataFrame(metrics[\"Datapoints\"])\\\n", 936 | " .sort_values(\"Timestamp\")\\\n", 937 | " .set_index(\"Timestamp\")\\\n", 938 | " .drop([\"Unit\"], axis=1)\\\n", 939 | " .rename(columns={statistic: rename})\n", 940 | "\n", 941 | "def plot_endpoint_model_latency_metrics(endpoint_name, endpoint_config_name, variant_name, start_time=None):\n", 942 | " start_time = start_time or datetime.now() - timedelta(minutes=60)\n", 943 | " end_time = datetime.now()\n", 944 | " metric_name = \"ModelLatency\"\n", 945 | " statistic = \"Average\"\n", 946 | " metrics_variants = get_sagemaker_metrics(\n", 947 | " endpoint_name,\n", 948 | " endpoint_config_name,\n", 949 | " variant_name,\n", 950 | " metric_name, \n", 951 | " statistic,\n", 952 | " start_time,\n", 953 | " end_time)\n", 954 | " metrics_variants.plot(title=f\"{metric_name}-{statistic}\")\n", 955 | " return metrics_variants" 956 | ] 957 | }, 958 | { 959 | "cell_type": "code", 960 | "execution_count": null, 961 | "metadata": {}, 962 | "outputs": [], 963 | "source": [ 964 | "model_latency_metrics = plot_endpoint_model_latency_metrics(endpoint_name, None, \"variant1\")" 965 | ] 966 | }, 967 | { 968 | "cell_type": "markdown", 969 | "metadata": {}, 970 | "source": [ 971 | "Similarly, we plot other Cloud Watch Metrics associated with the Endpoint as shown below" 972 | ] 973 | }, 974 | { 975 | "cell_type": "markdown", 976 | "metadata": {}, 977 | "source": [ 978 | "### Cloud watch metrics - Approximate Backlog Size and Approximate Backlog Size per instance \n", 979 | "The backlog grows from 0 to 1000 when the burst of traffic is invoked. Then, the endpoint autoscales every 120 seconds upto the max number of instances = 5. The Backlog size per instance changes rapidly during autoscaling. At max number of instances, the queue backlog reduces at about 18 invocation per minute and finally reaches 0. " 980 | ] 981 | }, 982 | { 983 | "cell_type": "markdown", 984 | "metadata": {}, 985 | "source": [ 986 | "![title](images/backlog_size_metrics.png)" 987 | ] 988 | }, 989 | { 990 | "cell_type": "markdown", 991 | "metadata": {}, 992 | "source": [ 993 | "The number of invocations successfully processed are about 18 invocations per minute" 994 | ] 995 | }, 996 | { 997 | "cell_type": "markdown", 998 | "metadata": {}, 999 | "source": [ 1000 | "![title](images/invocations_metrics.png)" 1001 | ] 1002 | }, 1003 | { 1004 | "cell_type": "markdown", 1005 | "metadata": {}, 1006 | "source": [ 1007 | "The model latency for 2 concurrent invocations is approximately 30 seconds " 1008 | ] 1009 | }, 1010 | { 1011 | "cell_type": "markdown", 1012 | "metadata": {}, 1013 | "source": [ 1014 | "![title](images/model_latency_metrics.png)" 1015 | ] 1016 | }, 1017 | { 1018 | "cell_type": "markdown", 1019 | "metadata": {}, 1020 | "source": [ 1021 | "The instances autoscale down to 0 once the queue size goes down to 0" 1022 | ] 1023 | }, 1024 | { 1025 | "cell_type": "markdown", 1026 | "metadata": {}, 1027 | "source": [ 1028 | "![title](images/instance_count_0.png)" 1029 | ] 1030 | }, 1031 | { 1032 | "cell_type": "markdown", 1033 | "metadata": {}, 1034 | "source": [ 1035 | "### Cleanup" 1036 | ] 1037 | }, 1038 | { 1039 | "cell_type": "markdown", 1040 | "metadata": {}, 1041 | "source": [ 1042 | "If you enabled auto-scaling for your endpoint, ensure you deregister the endpoint as a scalable target before deleting the endpoint. To do this, please uncomment the cell below and run :" 1043 | ] 1044 | }, 1045 | { 1046 | "cell_type": "code", 1047 | "execution_count": null, 1048 | "metadata": {}, 1049 | "outputs": [], 1050 | "source": [ 1051 | "# response = client.deregister_scalable_target(\n", 1052 | "# ServiceNamespace='sagemaker',\n", 1053 | "# ResourceId='resource_id',\n", 1054 | "# ScalableDimension='sagemaker:variant:DesiredInstanceCount'\n", 1055 | "# )" 1056 | ] 1057 | }, 1058 | { 1059 | "cell_type": "markdown", 1060 | "metadata": {}, 1061 | "source": [ 1062 | "Endpoints should be deleted when no longer in use, since (per the [SageMaker pricing page](https://aws.amazon.com/sagemaker/pricing/)) they're billed by time deployed" 1063 | ] 1064 | }, 1065 | { 1066 | "cell_type": "code", 1067 | "execution_count": null, 1068 | "metadata": {}, 1069 | "outputs": [], 1070 | "source": [ 1071 | "sm_client.delete_endpoint(EndpointName=endpoint_name)" 1072 | ] 1073 | }, 1074 | { 1075 | "cell_type": "markdown", 1076 | "metadata": {}, 1077 | "source": [ 1078 | "You may also want to delete any other resources you might have created such as SNS topics, S3 objects, etc." 1079 | ] 1080 | } 1081 | ], 1082 | "metadata": { 1083 | "instance_type": "ml.t3.medium", 1084 | "kernelspec": { 1085 | "display_name": "Python 3 (Data Science)", 1086 | "language": "python", 1087 | "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/datascience-1.0" 1088 | }, 1089 | "language_info": { 1090 | "codemirror_mode": { 1091 | "name": "ipython", 1092 | "version": 3 1093 | }, 1094 | "file_extension": ".py", 1095 | "mimetype": "text/x-python", 1096 | "name": "python", 1097 | "nbconvert_exporter": "python", 1098 | "pygments_lexer": "ipython3", 1099 | "version": "3.7.10" 1100 | } 1101 | }, 1102 | "nbformat": 4, 1103 | "nbformat_minor": 4 1104 | } 1105 | -------------------------------------------------------------------------------- /videos/ducks.mp4: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-sagemaker-asynchronous-inference-computer-vision/d2f1251712f9e9e56559b3326b4011aa43fb1a1d/videos/ducks.mp4 -------------------------------------------------------------------------------- /visualization/annotated_frames/birds.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-sagemaker-asynchronous-inference-computer-vision/d2f1251712f9e9e56559b3326b4011aa43fb1a1d/visualization/annotated_frames/birds.gif -------------------------------------------------------------------------------- /visualization/generate_gif.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "id": "0e9f6a44", 7 | "metadata": {}, 8 | "outputs": [], 9 | "source": [ 10 | "import os\n", 11 | "import json\n", 12 | "\n", 13 | "import cv2\n", 14 | "import torch\n", 15 | "import numpy as np\n", 16 | "import matplotlib.pyplot as plt\n", 17 | "import torchvision.transforms.functional as F\n", 18 | "from torchvision.utils import draw_bounding_boxes\n", 19 | "from torchvision.io import read_image" 20 | ] 21 | }, 22 | { 23 | "cell_type": "code", 24 | "execution_count": null, 25 | "id": "e169a705", 26 | "metadata": {}, 27 | "outputs": [], 28 | "source": [ 29 | "os.makedirs(\"visualization/frames/\", exist_ok=True)\n", 30 | "os.makedirs(\"visualization/annotated_frames/\", exist_ok=True)" 31 | ] 32 | }, 33 | { 34 | "cell_type": "code", 35 | "execution_count": null, 36 | "id": "66c1e603", 37 | "metadata": {}, 38 | "outputs": [], 39 | "source": [ 40 | "def video2frame(\n", 41 | " tfile,frame_width, frame_height, interval):\n", 42 | " \"\"\"\n", 43 | " Extract frame from video by interval\n", 44 | " :param video_src_path: video src path\n", 45 | " :param video:　video file name\n", 46 | " :param frame_width:　frame width\n", 47 | " :param frame_height:　frame height\n", 48 | " :param interval:　interval for frame to extract\n", 49 | " :return:　list of numpy.ndarray \n", 50 | " \"\"\"\n", 51 | " video_frames = []\n", 52 | " cap = cv2.VideoCapture(tfile)\n", 53 | " frame_index = 0\n", 54 | " frame_count = 0\n", 55 | " if cap.isOpened():\n", 56 | " success = True\n", 57 | " else:\n", 58 | " success = False\n", 59 | " print(\"Read failed!\")\n", 60 | "\n", 61 | " while success:\n", 62 | " success, frame = cap.read()\n", 63 | "\n", 64 | " if frame_index % interval == 0:\n", 65 | " print(\"---> Reading the %d frame:\" % frame_index, success)\n", 66 | " resize_frame = cv2.resize(\n", 67 | " frame, (frame_width, frame_height), interpolation=cv2.INTER_AREA\n", 68 | " )\n", 69 | " video_frames.append(resize_frame)\n", 70 | " frame_count += 1\n", 71 | "\n", 72 | " frame_index += 1\n", 73 | "\n", 74 | " cap.release()\n", 75 | " \n", 76 | " print('Number of frames')\n", 77 | " print(frame_count)\n", 78 | " return video_frames" 79 | ] 80 | }, 81 | { 82 | "cell_type": "code", 83 | "execution_count": null, 84 | "id": "cfb14262", 85 | "metadata": {}, 86 | "outputs": [], 87 | "source": [ 88 | "video_frames = video2frame('videos/ducks.mp4', 1024, 1024, 30)\n", 89 | "for i in range(len(video_frames)):\n", 90 | " cv2.imwrite(f\"visualization/frames/image-{i}.jpg\", video_frames[i])" 91 | ] 92 | }, 93 | { 94 | "cell_type": "code", 95 | "execution_count": null, 96 | "id": "692fe844", 97 | "metadata": {}, 98 | "outputs": [], 99 | "source": [ 100 | "plt.rcParams[\"savefig.bbox\"] = 'tight'\n", 101 | "\n", 102 | "\n", 103 | "def save(imgs, img_num):\n", 104 | " if not isinstance(imgs, list):\n", 105 | " imgs = [imgs]\n", 106 | " fix, axs = plt.subplots(ncols=len(imgs), squeeze=False)\n", 107 | " for i, img in enumerate(imgs):\n", 108 | " img = img.detach()\n", 109 | " img = F.to_pil_image(img)\n", 110 | " \n", 111 | " axs[0, i].imshow(np.asarray(img))\n", 112 | " axs[0, i].set(xticklabels=[], yticklabels=[], xticks=[], yticks=[])\n", 113 | " plt.savefig(f\"visualization/annotated_frames/{str(img_num).zfill(3)}.jpg\")" 114 | ] 115 | }, 116 | { 117 | "cell_type": "code", 118 | "execution_count": null, 119 | "id": "1b49141d", 120 | "metadata": {}, 121 | "outputs": [], 122 | "source": [ 123 | "with open(\"visualization/output.json\", \"r\") as read_file:\n", 124 | " data = json.load(read_file)\n" 125 | ] 126 | }, 127 | { 128 | "cell_type": "code", 129 | "execution_count": null, 130 | "id": "c1878aa8", 131 | "metadata": {}, 132 | "outputs": [], 133 | "source": [ 134 | "for i in range(len(data)):\n", 135 | " score_threshold = .9\n", 136 | " scores = torch.from_numpy(np.array(data[i]['scores']))\n", 137 | " boxes = torch.from_numpy(np.array(data[i]['boxes']))\n", 138 | "\n", 139 | " birds_with_boxes = [\n", 140 | " draw_bounding_boxes(read_image(f'visualization/frames/image-{i}.jpg'), boxes=boxes[scores > score_threshold], width=10)\n", 141 | " ]\n", 142 | " save(birds_with_boxes, i)" 143 | ] 144 | }, 145 | { 146 | "cell_type": "code", 147 | "execution_count": null, 148 | "id": "ff5a0748", 149 | "metadata": {}, 150 | "outputs": [], 151 | "source": [ 152 | "import glob\n", 153 | "from PIL import Image\n", 154 | "\n", 155 | "# filepaths\n", 156 | "fp_in = \"visualization/annotated_frames/*.jpg\"\n", 157 | "fp_out = \"visualization/annotated_frames/birds.gif\"\n", 158 | "\n", 159 | "\n", 160 | "img, *imgs = [Image.open(f) for f in sorted(glob.glob(fp_in))]\n", 161 | "img.save(fp=fp_out, format='GIF', append_images=imgs,\n", 162 | " save_all=True, duration=200, loop=0)\n" 163 | ] 164 | }, 165 | { 166 | "cell_type": "code", 167 | "execution_count": null, 168 | "id": "e4ba837f", 169 | "metadata": {}, 170 | "outputs": [], 171 | "source": [] 172 | } 173 | ], 174 | "metadata": { 175 | "kernelspec": { 176 | "display_name": "conda_python3", 177 | "language": "python", 178 | "name": "conda_python3" 179 | }, 180 | "language_info": { 181 | "codemirror_mode": { 182 | "name": "ipython", 183 | "version": 3 184 | }, 185 | "file_extension": ".py", 186 | "mimetype": "text/x-python", 187 | "name": "python", 188 | "nbconvert_exporter": "python", 189 | "pygments_lexer": "ipython3", 190 | "version": "3.6.13" 191 | } 192 | }, 193 | "nbformat": 4, 194 | "nbformat_minor": 5 195 | } 196 | -------------------------------------------------------------------------------- /visualization/output.json: -------------------------------------------------------------------------------- 1 | [{"boxes": [[792.7402954101562, 827.6157836914062, 1022.9835815429688, 959.7604370117188], [199.03294372558594, 665.279052734375, 353.19525146484375, 814.4165649414062], [793.8594360351562, 330.6965637207031, 1018.514892578125, 706.9755249023438], [110.65870666503906, 321.8924255371094, 322.4138488769531, 572.6350708007812], [789.2503662109375, 315.8791809082031, 860.031005859375, 604.1093139648438], [1.8557031154632568, 544.9660034179688, 44.16393280029297, 604.3548583984375], [297.14990234375, 364.7087097167969, 562.2606811523438, 620.4047241210938], [361.85052490234375, 625.9448852539062, 640.3538818359375, 872.0062866210938], [530.3519287109375, 443.250732421875, 774.6878662109375, 723.021728515625], [404.8971862792969, 423.143310546875, 479.25390625, 476.9449462890625], [237.5638885498047, 565.2955932617188, 478.5809326171875, 673.04833984375], [563.7816162109375, 477.2519836425781, 737.6375732421875, 665.4427490234375], [350.0330810546875, 408.5719299316406, 547.4647827148438, 568.5377807617188], [300.75787353515625, 321.63726806640625, 397.9046936035156, 634.6998291015625], [238.06019592285156, 346.9185791015625, 462.8298645019531, 675.4202270507812], [121.505859375, 324.2644958496094, 575.0540771484375, 639.235595703125], [229.01466369628906, 346.89654541015625, 364.28546142578125, 582.77294921875], [261.2519226074219, 457.30267333984375, 538.3507080078125, 605.8827514648438], [324.68035888671875, 338.74853515625, 499.25897216796875, 571.2876586914062], [352.76751708984375, 482.1844787597656, 526.8155517578125, 583.3304443359375], [227.3241424560547, 416.5498352050781, 664.2718505859375, 742.8758544921875], [168.8102264404297, 330.3757629394531, 851.0071411132812, 771.6767578125], [799.4031982421875, 309.4621276855469, 931.014892578125, 616.5193481445312], [329.23541259765625, 313.5924072265625, 377.0837097167969, 585.2036743164062], [515.4857177734375, 420.47265625, 573.2276000976562, 671.7810668945312], [909.8567504882812, 440.0411376953125, 1015.5478515625, 738.0122680664062], [105.32122802734375, 309.9825744628906, 164.00424194335938, 577.6022338867188], [358.3017578125, 371.1335754394531, 503.96087646484375, 514.9586791992188], [310.7434387207031, 486.27435302734375, 615.238525390625, 836.7685546875], [112.57113647460938, 308.50897216796875, 209.20172119140625, 573.8779296875], [399.60784912109375, 413.8138122558594, 512.88671875, 488.94036865234375]], "labels": [16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16], "scores": [0.9982324242591858, 0.9935640692710876, 0.9856426119804382, 0.9698808193206787, 0.947115957736969, 0.9465072751045227, 0.9287243485450745, 0.9282578825950623, 0.9114907383918762, 0.9102818965911865, 0.6092159152030945, 0.4452628493309021, 0.42471304535865784, 0.31800419092178345, 0.3038536608219147, 0.30076292157173157, 0.27549946308135986, 0.2555689811706543, 0.2321557104587555, 0.1588582843542099, 0.12457413226366043, 0.12302789837121964, 0.0964767336845398, 0.09269554167985916, 0.08634419739246368, 0.0815337672829628, 0.08067897707223892, 0.07855504006147385, 0.06909957528114319, 0.06807995587587357, 0.05322429537773132]}, {"boxes": [[779.7919921875, 773.5408325195312, 991.040283203125, 916.5162963867188], [70.60564422607422, 283.0534973144531, 310.22528076171875, 549.0886840820312], [148.767333984375, 622.8328857421875, 298.3548278808594, 772.9346923828125], [371.5378723144531, 569.9741821289062, 638.7564086914062, 844.4873046875], [322.90142822265625, 340.96673583984375, 572.9640502929688, 594.1474609375], [486.908935546875, 423.6332702636719, 753.3279418945312, 660.6983642578125], [133.50953674316406, 327.60015869140625, 436.3482666015625, 665.521484375], [159.45240783691406, 515.2927856445312, 411.13433837890625, 651.579833984375], [325.2946472167969, 400.7059326171875, 628.6806640625, 808.2080078125], [321.930419921875, 317.9510192871094, 397.8635559082031, 591.878173828125], [809.156494140625, 298.07879638671875, 1011.6653442382812, 675.94482421875], [937.6802368164062, 460.2894287109375, 1023.1011352539062, 684.845458984375], [366.72979736328125, 339.82672119140625, 710.2713623046875, 675.4898071289062], [122.42698669433594, 309.06561279296875, 708.3379516601562, 636.774658203125], [76.64246368408203, 273.4060363769531, 170.2110137939453, 538.2824096679688], [234.44558715820312, 318.0899963378906, 394.0874938964844, 632.6419677734375], [784.1260986328125, 270.6763000488281, 834.4719848632812, 386.37109375], [121.5959243774414, 399.6400451660156, 348.8852233886719, 756.3601684570312], [785.266845703125, 292.3840026855469, 869.1702880859375, 589.7300415039062], [327.5863037109375, 317.78826904296875, 468.6667175292969, 639.1455078125], [153.80921936035156, 551.871826171875, 377.1701354980469, 723.796875], [798.106689453125, 290.276611328125, 1019.7487182617188, 658.4453735351562]], "labels": [16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 9], "scores": [0.9981316924095154, 0.9961701035499573, 0.9918718338012695, 0.9619076251983643, 0.9615800380706787, 0.9156687259674072, 0.8816136717796326, 0.8035715818405151, 0.6322587728500366, 0.5236784219741821, 0.4311314821243286, 0.4235643744468689, 0.2900961637496948, 0.24294091761112213, 0.2352154701948166, 0.18230341374874115, 0.13495321571826935, 0.10600730031728745, 0.08956040441989899, 0.08889094740152359, 0.07884270697832108, 0.05461465194821358]}, {"boxes": [[748.9327392578125, 734.9246215820312, 942.8089599609375, 884.4647216796875], [754.6668701171875, 287.3499450683594, 1010.357177734375, 588.2575073242188], [352.2267150878906, 498.5948181152344, 645.8988037109375, 824.4857788085938], [96.43374633789062, 602.4713745117188, 246.17062377929688, 754.2102661132812], [131.1888427734375, 516.4224243164062, 355.4106140136719, 635.5618286132812], [316.70147705078125, 332.2317199707031, 531.1807861328125, 569.161376953125], [505.3309326171875, 412.5145568847656, 734.4844360351562, 650.9404907226562], [603.5242919921875, 391.4824523925781, 672.796875, 466.8576965332031], [914.2075805664062, 497.2920227050781, 1018.3778686523438, 694.2372436523438], [120.4867935180664, 327.937255859375, 530.8488159179688, 608.8429565429688], [488.9606018066406, 346.598388671875, 555.3394165039062, 618.9868774414062], [316.8141784667969, 378.9973449707031, 621.3648681640625, 755.6156005859375], [344.3543701171875, 458.8138427734375, 508.9504699707031, 562.162109375], [102.35366821289062, 410.5265808105469, 369.6558532714844, 717.6915283203125], [153.5806427001953, 316.30926513671875, 705.5845336914062, 779.7180786132812], [491.3796691894531, 352.970703125, 609.4530029296875, 639.2251586914062], [98.75492095947266, 521.2321166992188, 452.7526550292969, 746.7368774414062], [389.0975646972656, 404.78643798828125, 728.8543090820312, 723.3977661132812], [71.38521575927734, 323.89404296875, 257.3454895019531, 738.2828979492188], [513.5889892578125, 434.2183837890625, 738.2078857421875, 670.5250854492188], [53.3193244934082, 300.43890380859375, 265.96392822265625, 531.0022583007812], [774.501953125, 396.1156921386719, 1006.6066284179688, 667.2739868164062], [759.7009887695312, 280.6639404296875, 898.6539916992188, 556.6341552734375], [94.9552230834961, 587.623779296875, 350.59075927734375, 735.235595703125], [556.768798828125, 419.3410949707031, 712.3505249023438, 574.5089721679688], [99.90026092529297, 392.6251220703125, 286.4956970214844, 549.980712890625]], "labels": [16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 9, 16, 16, 16, 16, 16, 16], "scores": [0.9990124702453613, 0.9867799878120422, 0.9811990261077881, 0.9805037975311279, 0.9681330919265747, 0.9533295631408691, 0.9143602848052979, 0.8934637904167175, 0.888221800327301, 0.7933657169342041, 0.39446374773979187, 0.3409476578235626, 0.26893213391304016, 0.24988976120948792, 0.2153112143278122, 0.1778298169374466, 0.16143910586833954, 0.14959928393363953, 0.07098961621522903, 0.06762989610433578, 0.06710357964038849, 0.06549021601676941, 0.0638924092054367, 0.0565246120095253, 0.05256647244095802, 0.0509001761674881]}, {"boxes": [[24.4058780670166, 580.7151489257812, 176.42526245117188, 731.6448364257812], [737.4765625, 287.4773254394531, 1013.5427856445312, 561.6415405273438], [899.2631225585938, 521.3482055664062, 1021.1388549804688, 663.9879760742188], [692.378173828125, 767.4085693359375, 901.7985229492188, 864.0435791015625], [317.8898620605469, 294.12261962890625, 376.45550537109375, 583.8785400390625], [64.81964874267578, 386.4314270019531, 298.1700744628906, 594.2677001953125], [683.7227172851562, 290.3352355957031, 777.490234375, 461.5067138671875], [353.29949951171875, 379.3611755371094, 638.0147094726562, 757.968505859375], [421.41754150390625, 353.36578369140625, 679.3868408203125, 640.528564453125], [23.786657333374023, 278.9138488769531, 76.59910583496094, 545.371826171875], [397.8188171386719, 361.92724609375, 534.767333984375, 607.2890625], [361.8531494140625, 636.67626953125, 615.8970336914062, 800.407470703125], [684.1209716796875, 389.0447692871094, 754.4114379882812, 446.5943603515625], [692.345947265625, 765.302490234375, 770.9912109375, 827.5209350585938], [444.43096923828125, 365.0987548828125, 503.3482360839844, 601.3815307617188], [876.5115966796875, 295.89007568359375, 1016.6149291992188, 546.81787109375], [22.967432022094727, 322.0617980957031, 188.04632568359375, 711.9046630859375], [941.7802734375, 430.2693786621094, 1020.0335693359375, 547.9009399414062], [829.2756958007812, 800.0531616210938, 900.5044555664062, 859.6131591796875], [800.0850830078125, 777.0767211914062, 905.9581909179688, 863.0026245117188], [733.4561767578125, 283.6423034667969, 769.5803833007812, 353.16259765625], [494.1592102050781, 478.34808349609375, 541.5821533203125, 555.4725341796875], [472.46527099609375, 417.9749450683594, 673.8698120117188, 579.9884643554688]], "labels": [16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16], "scores": [0.9970123767852783, 0.9937612414360046, 0.9750725030899048, 0.9682154655456543, 0.9482119083404541, 0.9408236145973206, 0.925314724445343, 0.8123009204864502, 0.7721654176712036, 0.6822930574417114, 0.6061674952507019, 0.5787786245346069, 0.5331127047538757, 0.5101737380027771, 0.44939371943473816, 0.3232860267162323, 0.15391093492507935, 0.14998047053813934, 0.14817775785923004, 0.1025175154209137, 0.06885498017072678, 0.0678330734372139, 0.06418805569410324]}, {"boxes": [[770.3121948242188, 402.6117248535156, 840.4567260742188, 459.6390380859375], [14.69735336303711, 314.4058532714844, 250.3174591064453, 626.340087890625], [645.419677734375, 770.5217895507812, 872.501708984375, 866.502685546875], [328.2623291015625, 453.529052734375, 625.6625366210938, 787.5701904296875], [368.5635986328125, 355.3805847167969, 496.1609191894531, 550.8571166992188], [838.3030395507812, 309.6589660644531, 1018.2970581054688, 675.2263793945312], [375.183349609375, 423.8713073730469, 626.3094482421875, 591.8656616210938], [415.7804870605469, 345.13427734375, 482.9043884277344, 569.8114624023438], [701.4609985351562, 467.4983825683594, 924.705078125, 572.0454711914062], [708.0299072265625, 298.8078308105469, 1001.5673828125, 618.547607421875], [662.8613891601562, 769.7575073242188, 856.2418823242188, 862.9716796875], [29.458749771118164, 474.63983154296875, 250.92898559570312, 629.9785766601562], [694.945068359375, 424.854248046875, 1018.3989868164062, 574.8710327148438], [788.5006713867188, 782.59130859375, 877.1669311523438, 858.50927734375], [360.1553039550781, 343.00640869140625, 644.353759765625, 655.2247314453125], [879.1867065429688, 463.6365661621094, 1022.8831176757812, 665.2471923828125], [830.0438842773438, 408.0291442871094, 856.2957763671875, 431.0523986816406], [308.1266784667969, 294.35076904296875, 369.3521423339844, 659.0260009765625], [690.3325805664062, 302.2069396972656, 728.2628784179688, 562.6427001953125], [3.95068359375, 283.3277587890625, 62.6636962890625, 552.2865600585938], [0.3639453053474426, 654.7247314453125, 121.75743103027344, 736.7047729492188], [433.45111083984375, 559.4571533203125, 653.719482421875, 660.3737182617188], [690.934814453125, 314.55560302734375, 728.2363891601562, 556.4573364257812], [897.4200439453125, 371.3204650878906, 1020.0338134765625, 578.4398193359375], [322.80279541015625, 514.3710327148438, 631.2456665039062, 789.3203125], [364.9381103515625, 449.0716247558594, 514.0585327148438, 552.8157958984375], [687.718505859375, 308.2250061035156, 734.8475341796875, 557.0075073242188], [322.7994384765625, 313.3398132324219, 870.834716796875, 800.7107543945312], [428.7285461425781, 522.2763671875, 655.373046875, 662.3072509765625], [691.302001953125, 301.41644287109375, 726.0232543945312, 570.7266845703125], [10.50748062133789, 282.998291015625, 245.9135284423828, 630.2049560546875], [1.4446460008621216, 282.7845458984375, 65.24847412109375, 563.8827514648438]], "labels": [16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 1, 16, 16, 16, 16, 16, 16, 16, 48, 42, 16, 51, 50, 16, 86, 16, 16, 16, 16, 28, 9, 9], "scores": [0.9820892810821533, 0.951194167137146, 0.8996574878692627, 0.8449898362159729, 0.7955359220504761, 0.7249425053596497, 0.6495713591575623, 0.6260160803794861, 0.608586847782135, 0.533688485622406, 0.4551076889038086, 0.38986027240753174, 0.388689249753952, 0.31935277581214905, 0.2925892472267151, 0.2871403694152832, 0.2199622392654419, 0.19603724777698517, 0.18744370341300964, 0.14141234755516052, 0.12185519933700562, 0.12109863013029099, 0.10296360403299332, 0.10035163909196854, 0.09252923727035522, 0.08679311722517014, 0.08559302985668182, 0.0719108134508133, 0.05907979980111122, 0.0529068261384964, 0.05268867313861847, 0.05133615434169769]}, {"boxes": [[628.6643676757812, 770.4349975585938, 824.7827758789062, 906.6129760742188], [288.8988342285156, 392.4006042480469, 621.1470336914062, 812.5045166015625], [833.9884033203125, 340.4768981933594, 1010.4783935546875, 700.3427734375], [5.303114891052246, 515.7572631835938, 206.3047637939453, 649.2296142578125], [866.3338623046875, 331.4053955078125, 989.1796875, 574.0218505859375], [655.6333618164062, 416.7301330566406, 854.2328491210938, 608.2319946289062], [662.5590209960938, 533.3837280273438, 842.64013671875, 612.3460083007812], [360.55133056640625, 409.8932189941406, 465.3282775878906, 603.0164184570312], [907.7974243164062, 329.0469970703125, 966.3911743164062, 535.0640258789062], [383.3594970703125, 489.9049072265625, 618.658447265625, 674.9691772460938], [834.6109008789062, 360.862060546875, 915.7704467773438, 579.5487670898438], [849.9287109375, 463.6383972167969, 1011.278564453125, 599.9931640625], [830.6069946289062, 528.8031616210938, 1003.6365356445312, 715.8099975585938], [880.606201171875, 452.3055725097656, 920.0865478515625, 509.6233825683594], [294.6688232421875, 371.64654541015625, 449.3182067871094, 628.5084228515625], [641.4341430664062, 346.47210693359375, 949.7994384765625, 605.9942016601562], [295.16900634765625, 648.6310424804688, 588.838623046875, 827.4613647460938], [280.8341979980469, 339.0287780761719, 339.08062744140625, 640.1782836914062], [872.4189453125, 513.9586791992188, 1003.3477172851562, 605.6770629882812], [446.1150207519531, 493.4956970214844, 599.4149780273438, 632.66015625], [364.37591552734375, 411.2027893066406, 565.951416015625, 635.6375122070312], [251.5388641357422, 322.3338317871094, 959.9105834960938, 774.5430297851562], [309.56854248046875, 481.1065979003906, 588.36376953125, 605.4454956054688], [1.2446117401123047, 391.9697570800781, 17.708843231201172, 586.7633666992188], [0.0965893566608429, 702.661376953125, 42.58867263793945, 753.2710571289062], [305.5238952636719, 565.9843139648438, 626.99169921875, 723.8080444335938], [632.7234497070312, 769.9915161132812, 819.1077880859375, 909.5888671875], [973.5432739257812, 536.1190795898438, 1019.453125, 794.0003051757812], [387.87646484375, 423.958740234375, 485.0726318359375, 659.0516967773438], [63.57654571533203, 504.8063659667969, 189.68675231933594, 578.7691650390625]], "labels": [16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 62, 16, 16, 1, 16, 16, 16], "scores": [0.9947420358657837, 0.9762651324272156, 0.8802237510681152, 0.7932839393615723, 0.7322701811790466, 0.6321080923080444, 0.6302846074104309, 0.5158110857009888, 0.47000521421432495, 0.45285436511039734, 0.37043482065200806, 0.29092612862586975, 0.24555140733718872, 0.22395586967468262, 0.2115117758512497, 0.20631301403045654, 0.20180194079875946, 0.17641040682792664, 0.16957516968250275, 0.1612282693386078, 0.11164876073598862, 0.0956370010972023, 0.09345745295286179, 0.08634720742702484, 0.07229092717170715, 0.06350290775299072, 0.062391478568315506, 0.05375930294394493, 0.05084926262497902, 0.05012155696749687]}, {"boxes": [[579.754150390625, 766.0466918945312, 773.2847290039062, 886.5422973632812], [798.4563598632812, 331.5716247558594, 1011.857177734375, 719.2095947265625], [550.0499267578125, 392.6270751953125, 806.4660034179688, 616.6182861328125], [236.28857421875, 435.0663146972656, 595.6832885742188, 805.2581787109375], [297.2967224121094, 407.01898193359375, 431.3189392089844, 595.7312622070312], [1.6268163919448853, 490.0785827636719, 162.87457275390625, 651.8802490234375], [340.2229919433594, 408.6321105957031, 621.8773193359375, 691.4219360351562], [590.5065307617188, 538.810546875, 807.8541870117188, 619.5828247070312], [239.9829864501953, 374.6640930175781, 413.75787353515625, 613.9055786132812], [417.68975830078125, 497.08123779296875, 598.331787109375, 687.4701538085938], [678.0887451171875, 555.6763916015625, 807.1215209960938, 618.2661743164062], [922.1428833007812, 332.21783447265625, 1020.6642456054688, 588.7445068359375], [260.2497863769531, 664.2015991210938, 576.8113403320312, 825.06005859375], [223.10641479492188, 340.15594482421875, 325.0106506347656, 653.9307861328125], [335.16265869140625, 374.0409240722656, 423.40045166015625, 676.2263793945312], [253.94454956054688, 379.2149963378906, 831.3008422851562, 638.5601196289062], [761.2661743164062, 387.1662292480469, 873.5328979492188, 655.4174194335938], [269.83447265625, 400.71710205078125, 497.8887634277344, 665.2860717773438], [425.9171142578125, 506.9583740234375, 541.9757080078125, 654.6207275390625], [576.6734008789062, 377.77386474609375, 689.1331176757812, 640.5693969726562], [914.1677856445312, 332.0924987792969, 979.8074951171875, 566.1903076171875], [726.2910766601562, 545.7147216796875, 812.3310546875, 612.4822387695312], [3.994145393371582, 492.3790588378906, 157.83895874023438, 650.8444213867188], [583.5221557617188, 369.4892883300781, 618.668212890625, 639.1763916015625], [570.23291015625, 363.8630065917969, 621.9168090820312, 652.2059326171875], [836.7034301757812, 525.874267578125, 956.3543701171875, 619.450927734375], [673.6259155273438, 499.57421875, 818.6743774414062, 617.6177978515625], [733.4125366210938, 460.8424072265625, 829.1569213867188, 623.8680419921875], [481.9681396484375, 532.154541015625, 793.34765625, 635.8978271484375]], "labels": [16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 21, 50, 16, 16, 16, 16, 16], "scores": [0.9967159032821655, 0.9825887680053711, 0.9320153594017029, 0.9104293584823608, 0.9048930406570435, 0.8171426057815552, 0.8115562200546265, 0.7545878887176514, 0.6293613910675049, 0.5758712887763977, 0.5379073619842529, 0.524858295917511, 0.4981594681739807, 0.3920019268989563, 0.35687223076820374, 0.34347259998321533, 0.27934327721595764, 0.20417240262031555, 0.19874094426631927, 0.1368868499994278, 0.11902621388435364, 0.11493197083473206, 0.1063586100935936, 0.06588461995124817, 0.06557053327560425, 0.06547854840755463, 0.06162464618682861, 0.055963169783353806, 0.050887104123830795]}, {"boxes": [[555.259521484375, 732.4757080078125, 743.9810791015625, 891.9644165039062], [764.5358276367188, 388.28118896484375, 1020.4002075195312, 692.5543823242188], [196.9892578125, 529.2047119140625, 492.57080078125, 803.5647583007812], [288.3777160644531, 395.3693542480469, 401.5353698730469, 612.8932495117188], [531.2017822265625, 358.0252685546875, 784.9669189453125, 606.0352783203125], [306.8182067871094, 387.5350341796875, 624.830322265625, 671.3787231445312], [3.2503514289855957, 474.68353271484375, 151.73841857910156, 635.8770751953125], [583.8807983398438, 513.08740234375, 785.65185546875, 608.1494140625], [196.9828643798828, 365.5129699707031, 784.7531127929688, 700.8565673828125], [326.1521301269531, 422.0442199707031, 520.2738037109375, 636.46484375], [208.08758544921875, 407.7174987792969, 579.4102783203125, 782.5625610351562], [973.8768310546875, 331.4168701171875, 1021.90576171875, 588.0148315429688], [756.1047973632812, 380.1691589355469, 903.8731079101562, 661.8887329101562], [277.0299377441406, 483.61553955078125, 419.8179931640625, 593.8635864257812], [190.05081176757812, 336.388427734375, 280.5032043457031, 747.366943359375], [433.5602111816406, 360.4298400878906, 644.2530517578125, 651.115478515625], [232.39089965820312, 399.1449279785156, 441.1207275390625, 737.221923828125], [866.7025756835938, 606.2614135742188, 1021.267578125, 768.278564453125], [211.5452117919922, 326.8675842285156, 277.0885925292969, 606.6616821289062], [540.0263061523438, 344.21240234375, 632.9588623046875, 645.83349609375], [792.8141479492188, 530.4526977539062, 1009.2313232421875, 720.1309814453125], [845.7863159179688, 512.5462036132812, 920.2819213867188, 610.336669921875], [680.9190673828125, 818.6387329101562, 739.4718627929688, 870.0288696289062], [197.0067901611328, 512.8721313476562, 282.4319763183594, 787.5972290039062], [822.6329345703125, 505.0350646972656, 1020.314208984375, 617.8776245117188], [861.5244140625, 342.52288818359375, 1016.206298828125, 585.2813720703125], [399.64111328125, 420.6037292480469, 582.0657348632812, 666.6322631835938]], "labels": [16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16], "scores": [0.9973053932189941, 0.9856605529785156, 0.9764061570167542, 0.9761471152305603, 0.9652791619300842, 0.9529644846916199, 0.9529134035110474, 0.8097999095916748, 0.7495962977409363, 0.4257911145687103, 0.3574798107147217, 0.3567490577697754, 0.34295454621315, 0.2849823236465454, 0.2520112693309784, 0.19650322198867798, 0.17969341576099396, 0.16725771129131317, 0.1267595738172531, 0.11468248069286346, 0.10476071387529373, 0.09366245567798615, 0.08201323449611664, 0.07256545126438141, 0.06998688727617264, 0.06072422116994858, 0.05446590483188629]}, {"boxes": [[573.1244506835938, 704.3446044921875, 750.0005493164062, 843.0198974609375], [489.57354736328125, 353.518310546875, 746.6195068359375, 656.631103515625], [190.61343383789062, 476.0819396972656, 486.7513122558594, 787.0510864257812], [237.70545959472656, 341.7289733886719, 550.6099853515625, 619.8512573242188], [825.2503662109375, 362.4986877441406, 1018.5000610351562, 731.4071655273438], [5.237627029418945, 498.9786071777344, 179.0528106689453, 620.4097290039062], [267.0072021484375, 353.6795959472656, 782.8321533203125, 655.4144897460938], [612.2738037109375, 518.572265625, 782.6965942382812, 591.4588012695312], [841.8067626953125, 368.43499755859375, 953.8208618164062, 677.940673828125], [248.21292114257812, 342.78411865234375, 424.91644287109375, 582.9969482421875], [317.9825134277344, 413.2830810546875, 530.6503295898438, 625.4646606445312], [779.1082153320312, 410.3800964355469, 826.0346069335938, 487.77362060546875], [783.4635009765625, 403.9045104980469, 871.6834716796875, 652.3700561523438], [532.2328491210938, 490.2133483886719, 787.0945434570312, 598.6802368164062], [235.53634643554688, 323.3953552246094, 332.1471862792969, 575.8788452148438], [306.8216247558594, 394.6713562011719, 439.1520080566406, 610.0194091796875], [32.29353332519531, 545.47607421875, 174.71165466308594, 622.7388916015625], [821.0317993164062, 443.4178466796875, 1020.9944458007812, 591.6318359375], [209.2649688720703, 328.6627197265625, 606.5748291015625, 793.7631225585938], [857.021728515625, 363.5264892578125, 913.8189086914062, 615.6615600585938], [229.8379669189453, 325.700927734375, 289.9475402832031, 562.099609375], [559.304443359375, 408.1955261230469, 796.4053955078125, 580.5729370117188]], "labels": [16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16], "scores": [0.9973456263542175, 0.9831937551498413, 0.9787824749946594, 0.9626886248588562, 0.94977205991745, 0.9045999050140381, 0.7891806960105896, 0.6104362607002258, 0.5839995741844177, 0.5836149454116821, 0.49531641602516174, 0.42480069398880005, 0.21358752250671387, 0.15047793090343475, 0.13873259723186493, 0.12458842247724533, 0.1082872524857521, 0.10668810456991196, 0.10450994968414307, 0.09770438075065613, 0.07308664172887802, 0.06073043867945671]}, {"boxes": [[557.3906860351562, 670.1119384765625, 727.8485717773438, 816.3233642578125], [761.1582641601562, 348.0139465332031, 1023.0735473632812, 716.915771484375], [188.9369659423828, 487.17828369140625, 450.86663818359375, 745.5422973632812], [229.81410217285156, 318.0658874511719, 421.40740966796875, 553.0616455078125], [541.4164428710938, 465.4037780761719, 754.91796875, 577.2230224609375], [499.0819091796875, 442.79827880859375, 761.2019653320312, 637.898193359375], [5.162885665893555, 450.55078125, 156.86935424804688, 596.3333740234375], [214.9951171875, 342.31268310546875, 778.1759033203125, 589.9326171875], [754.0513305664062, 399.0919494628906, 832.6744384765625, 647.8193359375], [841.9254760742188, 456.86224365234375, 1024.0, 572.0239868164062], [197.12306213378906, 332.8949279785156, 457.47662353515625, 657.0452880859375], [747.0813598632812, 400.5164794921875, 799.1812133789062, 473.16546630859375], [846.0859375, 353.43780517578125, 906.1138916015625, 573.1058349609375], [499.82611083984375, 527.5206298828125, 736.5435791015625, 647.1687622070312], [675.5940551757812, 490.0146789550781, 762.3175048828125, 568.9544677734375], [184.77099609375, 247.11830139160156, 777.9334106445312, 733.0217895507812], [823.0364379882812, 345.9517822265625, 958.5823974609375, 680.15576171875], [835.3914794921875, 353.1982421875, 1022.9205322265625, 580.4019775390625], [25.02252960205078, 520.8584594726562, 155.6440887451172, 598.1318359375], [378.3881530761719, 436.275146484375, 524.5048217773438, 606.04638671875], [823.6764526367188, 507.8637390136719, 1019.7012939453125, 698.2723999023438], [648.7457275390625, 383.3164367675781, 928.61669921875, 662.6414184570312], [645.0347290039062, 531.3280639648438, 692.8142700195312, 565.9243774414062], [249.4257354736328, 373.11810302734375, 572.5320434570312, 604.18701171875], [435.88861083984375, 379.21148681640625, 800.5852661132812, 602.3602905273438]], "labels": [16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16], "scores": [0.9987841248512268, 0.9789993762969971, 0.9599658846855164, 0.9334321022033691, 0.9115551114082336, 0.8204507827758789, 0.7332499027252197, 0.695830225944519, 0.4046187996864319, 0.3943004608154297, 0.3640921711921692, 0.3403904139995575, 0.329879492521286, 0.281869500875473, 0.2309332638978958, 0.14024615287780762, 0.1313314586877823, 0.10576146841049194, 0.08434849232435226, 0.07342402637004852, 0.07068312913179398, 0.06959065049886703, 0.0642528235912323, 0.058693259954452515, 0.0507233589887619]}, {"boxes": [[489.2799377441406, 647.7625732421875, 678.5608520507812, 789.314208984375], [165.7351531982422, 300.9657287597656, 404.0728454589844, 531.7705688476562], [106.09197998046875, 459.3362731933594, 392.3799133300781, 715.9226684570312], [520.5031127929688, 413.1332702636719, 755.3717651367188, 581.2701416015625], [745.0820922851562, 460.79193115234375, 1017.4462890625, 711.5062255859375], [1.8925243616104126, 421.2629089355469, 96.54302215576172, 571.55126953125], [463.52593994140625, 316.8602600097656, 756.6021118164062, 616.0193481445312], [537.7015380859375, 459.02581787109375, 740.8783569335938, 552.2317504882812], [168.87921142578125, 309.2414245605469, 712.7838134765625, 579.0067749023438], [774.848876953125, 591.9627075195312, 1014.16357421875, 723.009033203125], [729.9581298828125, 466.17437744140625, 830.2485961914062, 638.9014892578125], [505.423828125, 519.5430908203125, 712.174072265625, 625.8550415039062], [1.7919139862060547, 497.7210693359375, 103.14727020263672, 578.6828002929688], [729.6275634765625, 473.0250549316406, 922.4405517578125, 648.40576171875], [702.6861572265625, 405.4810485839844, 806.0662841796875, 624.961669921875], [833.3424682617188, 330.39617919921875, 882.7297973632812, 532.26904296875], [815.2392578125, 535.7802124023438, 956.8377075195312, 640.1669921875], [820.4522705078125, 352.1056823730469, 965.458984375, 682.3560791015625], [812.4205932617188, 337.1923828125, 1015.3228149414062, 544.2024536132812], [693.7748413085938, 360.5227355957031, 1006.5685424804688, 651.9310302734375], [681.8673095703125, 388.945068359375, 770.603271484375, 615.3325805664062], [863.079345703125, 460.3338317871094, 1022.9428100585938, 544.1884155273438], [803.7904663085938, 470.4093933105469, 1015.3073120117188, 592.7808837890625], [782.3045043945312, 335.48583984375, 886.7476196289062, 544.3597412109375], [830.168701171875, 334.98687744140625, 922.6658935546875, 586.7009887695312], [773.7860717773438, 473.4351501464844, 825.4629516601562, 631.0501708984375], [135.49253845214844, 253.54627990722656, 519.6690673828125, 696.2971801757812], [833.9431762695312, 329.2942810058594, 876.6297607421875, 400.7994384765625], [164.04306030273438, 302.04730224609375, 238.7193145751953, 521.6680908203125], [10.034098625183105, 450.8603515625, 88.5123062133789, 536.1495361328125], [2.900038957595825, 419.87060546875, 69.97014617919922, 478.9752197265625]], "labels": [16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16], "scores": [0.9983615279197693, 0.9929789900779724, 0.9853934645652771, 0.9634279012680054, 0.9485980272293091, 0.9432633519172668, 0.7473427653312683, 0.7306299805641174, 0.7158108353614807, 0.6963245272636414, 0.6867573857307434, 0.6682141423225403, 0.6429504752159119, 0.6231773495674133, 0.39067596197128296, 0.38758423924446106, 0.3669605851173401, 0.28010526299476624, 0.25723934173583984, 0.24051886796951294, 0.215842604637146, 0.2150534838438034, 0.20999287068843842, 0.08852969110012054, 0.08796457946300507, 0.0833660438656807, 0.07205602526664734, 0.06662242114543915, 0.06427748501300812, 0.0611691027879715, 0.051695067435503006]}, {"boxes": [[455.26904296875, 662.2774047851562, 631.9468383789062, 809.8120727539062], [875.7749633789062, 363.5458984375, 1022.2671508789062, 563.2008666992188], [59.737186431884766, 449.7547607421875, 365.9010009765625, 731.9725341796875], [201.349365234375, 329.51483154296875, 412.83154296875, 547.3077392578125], [552.0723876953125, 452.7734680175781, 772.9451904296875, 580.241455078125], [777.3668823242188, 559.5980834960938, 1014.7883911132812, 667.2257690429688], [316.40191650390625, 379.7013244628906, 526.9957885742188, 595.3301391601562], [517.1588745117188, 349.22186279296875, 768.3739013671875, 612.060302734375], [770.282958984375, 634.7698974609375, 1008.770263671875, 736.5717163085938], [780.2398071289062, 384.9892883300781, 1019.727294921875, 714.6760864257812], [348.251708984375, 471.4416198730469, 534.41357421875, 598.10888671875], [623.947265625, 482.62933349609375, 764.2255859375, 580.9044799804688], [726.3661499023438, 417.3584289550781, 803.6007080078125, 676.8828125], [640.7713012695312, 425.7308349609375, 781.6600341796875, 613.422119140625], [520.9595336914062, 511.397216796875, 730.775634765625, 648.1763305664062], [2.9795849323272705, 453.29632568359375, 111.15472412109375, 608.0199584960938], [0.172607421875, 453.1529846191406, 124.7943115234375, 617.597900390625], [564.3268432617188, 488.8879089355469, 731.2008666992188, 568.5936279296875], [523.38671875, 559.8478393554688, 688.1923217773438, 649.5800170898438], [518.5284423828125, 561.4761352539062, 686.442626953125, 650.0748291015625], [511.4267883300781, 338.25189208984375, 579.443115234375, 617.1117553710938], [866.9012451171875, 361.0774841308594, 929.173095703125, 560.2029418945312], [761.0316162109375, 456.0528869628906, 807.20068359375, 687.4634399414062], [517.9429321289062, 339.2769775390625, 579.6069946289062, 575.7372436523438], [2.3185107707977295, 444.96044921875, 102.9188461303711, 614.8635864257812], [809.674072265625, 367.30596923828125, 954.0787353515625, 622.7689208984375], [748.589111328125, 430.42120361328125, 825.1848754882812, 717.2401123046875], [205.98878479003906, 340.26202392578125, 732.7804565429688, 621.4966430664062]], "labels": [16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 9, 16, 16, 51, 16, 16, 16, 16, 28, 1, 16, 16, 16], "scores": [0.9989462494850159, 0.989563524723053, 0.9883620142936707, 0.985988974571228, 0.9444198608398438, 0.9214620590209961, 0.8984035849571228, 0.8455591797828674, 0.815325915813446, 0.7958259582519531, 0.6339020729064941, 0.5287792086601257, 0.5160776972770691, 0.363972932100296, 0.32305487990379333, 0.27988648414611816, 0.27410611510276794, 0.2604106664657593, 0.20369626581668854, 0.2008877545595169, 0.19764623045921326, 0.19753098487854004, 0.12822464108467102, 0.11828834563493729, 0.08362885564565659, 0.07457533478736877, 0.05683385953307152, 0.055223800241947174]}, {"boxes": [[879.59326171875, 364.7037353515625, 1022.9464111328125, 565.8064575195312], [401.4454650878906, 644.325439453125, 574.908203125, 778.1243286132812], [500.5787658691406, 337.3341064453125, 732.5054321289062, 624.2203979492188], [187.2967071533203, 322.1775817871094, 397.990966796875, 539.0809936523438], [18.833173751831055, 540.8221435546875, 279.50140380859375, 715.8399658203125], [749.2969360351562, 570.0476684570312, 946.2462768554688, 652.2548217773438], [712.7711791992188, 393.7078857421875, 956.228515625, 715.2933349609375], [317.75616455078125, 450.827880859375, 509.18060302734375, 596.88037109375], [206.55625915527344, 330.5555114746094, 666.9917602539062, 574.8226928710938], [74.77397918701172, 495.3077697753906, 135.08050537109375, 551.547607421875], [2.3067944049835205, 457.4920959472656, 73.09548950195312, 589.08740234375], [714.9593505859375, 611.6063232421875, 954.7638549804688, 735.69091796875], [468.9900207519531, 456.2926025390625, 751.1074829101562, 609.4195556640625], [694.8485717773438, 402.1354675292969, 762.6746826171875, 695.1986083984375], [276.373779296875, 458.7635192871094, 417.04791259765625, 587.5656127929688], [181.101318359375, 327.5669860839844, 243.48287963867188, 520.5997924804688], [8.640585899353027, 385.6676940917969, 261.2518005371094, 705.6742553710938], [877.2325439453125, 367.72186279296875, 935.1011352539062, 557.2374267578125], [502.2032165527344, 339.23968505859375, 574.8612670898438, 601.6586303710938], [304.2056884765625, 358.3963317871094, 563.9306640625, 606.3623046875], [327.35009765625, 358.0328063964844, 746.7196655273438, 651.1676635742188], [73.62519073486328, 478.710693359375, 92.39966583251953, 513.8228149414062], [544.5816040039062, 481.7384948730469, 714.3702392578125, 577.4174194335938], [4.4386887550354, 454.8232727050781, 65.29728698730469, 534.200927734375], [12.220077514648438, 467.205810546875, 81.9378662109375, 661.7112426757812], [12.912821769714355, 464.5663146972656, 138.22972106933594, 569.337158203125], [0.4209838807582855, 524.2501220703125, 42.399452209472656, 597.5609130859375], [214.8882598876953, 438.95550537109375, 382.61773681640625, 533.742919921875], [718.7579345703125, 503.7693786621094, 977.481201171875, 659.207275390625], [10.232123374938965, 504.94488525390625, 63.51220703125, 560.5891723632812], [311.0561828613281, 464.4343566894531, 377.6087951660156, 595.1024780273438], [519.1939697265625, 526.2258911132812, 698.269775390625, 640.9451293945312], [53.558162689208984, 274.96575927734375, 774.2762451171875, 707.4791870117188], [582.5457763671875, 412.2161865234375, 753.1568603515625, 629.8571166992188], [680.8338623046875, 430.554443359375, 752.2084350585938, 602.2747802734375]], "labels": [16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16], "scores": [0.9979666471481323, 0.9962523579597473, 0.9917431473731995, 0.9913517236709595, 0.9868313074111938, 0.9609776139259338, 0.917740523815155, 0.8773130774497986, 0.6367952227592468, 0.6303530931472778, 0.6267766952514648, 0.5567722320556641, 0.3282249867916107, 0.2913374900817871, 0.2557571828365326, 0.23935294151306152, 0.20784686505794525, 0.2046433985233307, 0.16322647035121918, 0.15962029993534088, 0.14703959226608276, 0.14556573331356049, 0.13695816695690155, 0.13127656280994415, 0.1254778951406479, 0.12417495250701904, 0.10390525311231613, 0.09832413494586945, 0.09527844190597534, 0.08288732916116714, 0.08252240717411041, 0.07536447793245316, 0.0692606121301651, 0.056442610919475555, 0.05153866112232208]}, {"boxes": [[371.25421142578125, 593.5260009765625, 534.3397827148438, 758.486328125], [468.0837707519531, 349.3611755371094, 706.21875, 582.8925170898438], [188.7080841064453, 299.5316467285156, 442.3016052246094, 544.1403198242188], [7.125946998596191, 488.65240478515625, 254.81077575683594, 695.1061401367188], [664.2550048828125, 367.1923828125, 953.9389038085938, 690.7052001953125], [900.85107421875, 336.070068359375, 1018.111083984375, 544.4246215820312], [1.573793888092041, 423.3262023925781, 86.83599090576172, 595.7509765625], [102.10192108154297, 438.929443359375, 150.9300994873047, 489.080078125], [7.504982948303223, 504.22564697265625, 84.701416015625, 585.0355224609375], [463.499755859375, 340.8567199707031, 561.2811889648438, 554.1431274414062], [181.3642120361328, 299.2696533203125, 251.486083984375, 494.8213195800781], [2.435502767562866, 420.4549255371094, 88.43755340576172, 509.7264099121094], [542.0902709960938, 467.3445129394531, 688.655517578125, 571.4343872070312], [3.541640520095825, 444.73394775390625, 124.203857421875, 635.8673095703125], [215.16807556152344, 403.8377380371094, 431.1226501464844, 570.6947631835938], [3.116220712661743, 346.0671081542969, 280.3709411621094, 679.6040649414062], [92.28594207763672, 428.6913146972656, 140.1705322265625, 480.2239685058594], [518.2828979492188, 351.0798034667969, 865.3848876953125, 680.024658203125], [83.12765502929688, 427.7547912597656, 119.77818298339844, 467.38134765625], [326.4046630859375, 493.14642333984375, 417.4972839355469, 574.9002685546875], [188.66085815429688, 293.64727783203125, 298.59326171875, 518.7291870117188], [672.89599609375, 588.0091552734375, 936.476318359375, 705.9090576171875]], "labels": [16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16], "scores": [0.9988355040550232, 0.9983476400375366, 0.9961743354797363, 0.9909794330596924, 0.9882213473320007, 0.9785693883895874, 0.8565244078636169, 0.8457401394844055, 0.7449917793273926, 0.7396731972694397, 0.43029913306236267, 0.2713790833950043, 0.20418184995651245, 0.1833495944738388, 0.13323812186717987, 0.13159681856632233, 0.09921778738498688, 0.09560877084732056, 0.07852627336978912, 0.07574252784252167, 0.0516761876642704, 0.05016455799341202]}, {"boxes": [[342.498046875, 616.71826171875, 514.8462524414062, 758.111572265625], [649.5892944335938, 388.76971435546875, 966.4177856445312, 696.6918334960938], [554.6572875976562, 360.4411315917969, 756.6868286132812, 580.46533203125], [5.301455020904541, 558.3023071289062, 250.1757049560547, 692.335205078125], [229.61077880859375, 368.7528381347656, 467.3729553222656, 574.8141479492188], [942.6798706054688, 361.2974548339844, 1015.2073974609375, 566.8654174804688], [0.5184765458106995, 411.14764404296875, 258.7507629394531, 677.9086303710938], [3.7602880001068115, 433.9362487792969, 85.7298812866211, 591.4129638671875], [184.18292236328125, 310.0528869628906, 234.46905517578125, 511.29998779296875], [3.810190439224243, 435.2681579589844, 91.14181518554688, 600.2443237304688], [941.4236450195312, 359.4342346191406, 991.226318359375, 572.2672729492188], [549.8515014648438, 345.6383056640625, 591.9762573242188, 423.0289306640625], [180.00411987304688, 304.0492248535156, 222.77200317382812, 375.837646484375], [550.7015380859375, 352.0437316894531, 623.8200073242188, 555.1806640625], [129.22769165039062, 446.6728515625, 194.0877227783203, 479.0776062011719], [373.1268005371094, 364.6621398925781, 470.6564636230469, 567.819091796875], [632.199951171875, 398.1086730957031, 765.7463989257812, 607.257080078125], [215.04689025878906, 431.39556884765625, 409.9205322265625, 567.7239379882812], [1.9843212366104126, 443.6042785644531, 86.07666015625, 588.1830444335938], [190.01226806640625, 308.20965576171875, 304.2745361328125, 513.6390380859375], [5.75137186050415, 434.1777648925781, 142.16075134277344, 647.5203857421875], [191.54751586914062, 318.4021911621094, 387.9504699707031, 540.1309814453125], [939.1954345703125, 366.2105712890625, 1011.39794921875, 558.6458129882812], [937.6864624023438, 366.7132873535156, 1015.632568359375, 566.87890625], [970.7095947265625, 499.0382995605469, 1022.316162109375, 564.0579833984375], [127.71358489990234, 306.0067138671875, 241.7041778564453, 495.70343017578125], [452.69256591796875, 697.3359375, 508.9558410644531, 749.5365600585938], [22.22483253479004, 518.399658203125, 86.39146423339844, 587.6796264648438], [571.6973876953125, 395.7137451171875, 694.8463134765625, 550.507080078125], [635.7479858398438, 459.7409362792969, 689.0523071289062, 536.8761596679688], [638.4267578125, 479.67730712890625, 665.7496948242188, 534.7861938476562], [930.8880004882812, 373.7706604003906, 979.2017822265625, 562.0269165039062], [670.4613647460938, 416.2799377441406, 843.1134033203125, 669.6737060546875], [26.940570831298828, 521.1444702148438, 86.29302215576172, 583.2565307617188], [389.01104736328125, 536.984130859375, 410.6325988769531, 574.3470458984375]], "labels": [16, 16, 16, 16, 16, 62, 16, 16, 16, 9, 28, 16, 16, 16, 16, 16, 16, 16, 62, 16, 16, 16, 16, 9, 62, 16, 16, 16, 16, 16, 16, 62, 16, 62, 16], "scores": [0.9978488683700562, 0.9966157078742981, 0.995097815990448, 0.9807868599891663, 0.9562608599662781, 0.7100315093994141, 0.7070847153663635, 0.49664080142974854, 0.4873140752315521, 0.43955036997795105, 0.39590340852737427, 0.3746374845504761, 0.2808833122253418, 0.27776816487312317, 0.27394360303878784, 0.22571542859077454, 0.22153852880001068, 0.21281002461910248, 0.19915592670440674, 0.18318647146224976, 0.17936043441295624, 0.16803281009197235, 0.1105712428689003, 0.11040769517421722, 0.10946394503116608, 0.10176584124565125, 0.08768459409475327, 0.08245328813791275, 0.08162234723567963, 0.06777756661176682, 0.06679610908031464, 0.05909170210361481, 0.05791599676012993, 0.05647309869527817, 0.050920240581035614]}, {"boxes": [[293.6831970214844, 636.5557250976562, 455.156005859375, 775.3026123046875], [122.62349700927734, 332.42333984375, 340.7217102050781, 533.0828857421875], [563.1065673828125, 365.2417907714844, 947.9428100585938, 702.2432861328125], [9.144560813903809, 546.6015625, 246.1130828857422, 714.1799926757812], [334.2181091308594, 375.8309020996094, 487.4795837402344, 599.3905029296875], [602.1611328125, 411.6259765625, 807.0725708007812, 632.44921875], [722.6646728515625, 591.994140625, 936.9273071289062, 688.12646484375], [559.9312744140625, 364.0758361816406, 638.0507202148438, 700.07958984375], [145.75064086914062, 440.6051330566406, 334.30584716796875, 536.9507446289062], [679.9489135742188, 495.30279541015625, 925.96044921875, 678.3563842773438], [2.73419189453125, 474.7082824707031, 55.86537170410156, 695.7583618164062], [437.7333984375, 376.0325012207031, 498.8217468261719, 570.7802734375], [123.39171600341797, 343.0415954589844, 515.9680786132812, 568.4686889648438], [608.9365844726562, 587.2784423828125, 871.2337036132812, 716.4067993164062], [925.6060180664062, 368.7951965332031, 968.9022216796875, 593.6101684570312], [919.677001953125, 373.3276062011719, 1015.9165649414062, 589.3746948242188], [555.2041015625, 368.36578369140625, 615.8251953125, 607.8796997070312], [92.93572235107422, 460.4086608886719, 148.76316833496094, 503.89093017578125], [576.3712158203125, 474.1988220214844, 816.6463623046875, 725.855712890625], [2.931940793991089, 476.5429992675781, 58.36779022216797, 687.9835815429688], [2.6193530559539795, 471.97662353515625, 63.22354507446289, 699.6180419921875], [10.230927467346191, 471.3961181640625, 128.92498779296875, 710.4535522460938], [557.0462646484375, 373.4068298339844, 714.083740234375, 654.434326171875], [3.097353458404541, 401.9346923828125, 265.2677307128906, 696.4900512695312], [629.5924682617188, 521.941162109375, 800.9718627929688, 633.1822509765625], [339.89154052734375, 438.3121032714844, 447.5616455078125, 593.3734741210938]], "labels": [16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 62, 16, 16, 16, 28, 9, 16, 16, 16, 16, 9, 16, 16, 16, 16, 16], "scores": [0.9940063953399658, 0.9812319874763489, 0.9646727442741394, 0.9497853517532349, 0.9483473896980286, 0.8424098491668701, 0.7947978377342224, 0.5515727400779724, 0.5133421421051025, 0.48732873797416687, 0.4474613666534424, 0.29226478934288025, 0.27693143486976624, 0.2515963017940521, 0.2513117790222168, 0.19130493700504303, 0.1770009845495224, 0.15943650901317596, 0.14475436508655548, 0.1411658525466919, 0.12718746066093445, 0.0930141806602478, 0.08968447893857956, 0.07682447135448456, 0.06667805463075638, 0.05629400163888931]}, {"boxes": [[268.0600280761719, 594.2029418945312, 425.38995361328125, 752.4029541015625], [615.6871337890625, 397.9661560058594, 770.0264892578125, 591.4425659179688], [542.4386596679688, 397.1883850097656, 780.3744506835938, 681.6895141601562], [685.92724609375, 524.75439453125, 884.9722290039062, 654.4999389648438], [143.11721801757812, 445.6438293457031, 327.271240234375, 530.48388671875], [551.4730834960938, 486.19781494140625, 868.8938598632812, 706.2105102539062], [18.60881805419922, 440.5019836425781, 276.7157897949219, 710.7686767578125], [126.42774200439453, 352.6555480957031, 329.3980407714844, 522.5650024414062], [22.700624465942383, 569.8156127929688, 268.1149597167969, 706.5614624023438], [936.3712768554688, 364.7507019042969, 1019.4064331054688, 571.4749145507812], [591.716552734375, 469.00286865234375, 767.8480834960938, 568.43896484375], [377.5086364746094, 390.5163879394531, 539.937744140625, 581.4363403320312], [942.6663818359375, 361.9186096191406, 1016.7266845703125, 565.9219970703125], [56.89851379394531, 312.162109375, 320.9718933105469, 572.2705078125], [52.18137741088867, 418.4733581542969, 132.8673095703125, 480.0685119628906], [539.5590209960938, 542.9583740234375, 777.4437255859375, 721.0494384765625], [823.2632446289062, 551.522705078125, 859.4652709960938, 594.52099609375], [111.6112289428711, 306.9171142578125, 351.4344482421875, 537.7896728515625], [581.5911254882812, 401.2626953125, 702.192626953125, 575.5610961914062], [710.0009155273438, 555.478759765625, 905.8218383789062, 692.4478759765625], [7.504345417022705, 430.0939025878906, 284.5434265136719, 706.2643432617188], [943.4780883789062, 360.23260498046875, 993.587646484375, 576.0248413085938], [33.574607849121094, 307.7044982910156, 341.5545349121094, 656.5415649414062], [76.89635467529297, 447.2720947265625, 134.16387939453125, 480.5230407714844], [938.7197265625, 366.9431457519531, 1016.1528930664062, 569.1664428710938], [693.0984497070312, 530.7481689453125, 872.1923217773438, 654.6948852539062]], "labels": [16, 16, 16, 16, 16, 16, 16, 16, 16, 9, 16, 16, 62, 16, 1, 16, 16, 9, 16, 16, 9, 28, 9, 1, 16, 21], "scores": [0.997703492641449, 0.9877215027809143, 0.8895960450172424, 0.8842537999153137, 0.8819469213485718, 0.8021725416183472, 0.647629976272583, 0.5587246417999268, 0.5574273467063904, 0.46138516068458557, 0.39727911353111267, 0.3881590664386749, 0.22630047798156738, 0.19711314141750336, 0.18544071912765503, 0.14940345287322998, 0.09701558947563171, 0.08707911521196365, 0.08521128445863724, 0.07973869889974594, 0.07731860131025314, 0.07655353844165802, 0.06576845794916153, 0.06016689911484718, 0.05175219848752022, 0.05060570687055588]}, {"boxes": [[263.0692443847656, 560.22021484375, 424.99114990234375, 709.7656860351562], [690.8350830078125, 528.0577392578125, 898.5397338867188, 624.5297241210938], [495.1639709472656, 351.8625793457031, 773.68359375, 656.659423828125], [24.270429611206055, 458.77197265625, 260.34814453125, 659.9403076171875], [112.63066864013672, 279.31597900390625, 319.47662353515625, 492.5322265625], [480.5464782714844, 362.60650634765625, 706.7124633789062, 558.3511962890625], [824.2434692382812, 494.40936279296875, 939.706787109375, 580.320068359375], [137.21829223632812, 391.6219482421875, 323.0503234863281, 503.0491943359375], [452.6824645996094, 331.49603271484375, 571.5894775390625, 555.9472045898438], [573.778564453125, 388.05010986328125, 711.197265625, 551.7985229492188], [582.5823364257812, 452.91339111328125, 720.4215698242188, 540.0739135742188], [58.26438522338867, 386.04388427734375, 121.11649322509766, 437.7047424316406], [817.60888671875, 344.8545227050781, 938.2109375, 572.6759033203125], [562.2333984375, 362.0822448730469, 912.4744873046875, 624.0882568359375], [974.5532836914062, 312.2263488769531, 1018.5899658203125, 551.7011108398438], [751.5159912109375, 500.3498229980469, 941.7081909179688, 609.163330078125], [973.6968383789062, 321.6734619140625, 1018.2271728515625, 542.9968872070312], [25.567724227905273, 312.3664855957031, 273.8661804199219, 640.5983276367188], [224.2498779296875, 405.00531005859375, 251.7381591796875, 463.4021911621094], [619.18408203125, 387.734130859375, 726.35498046875, 610.3992309570312], [49.815731048583984, 274.46197509765625, 153.9752960205078, 470.4300231933594], [23.555322647094727, 417.0887756347656, 292.2305908203125, 571.8982543945312], [10.383319854736328, 429.03997802734375, 255.44882202148438, 662.1237182617188], [220.01437377929688, 428.3116760253906, 328.1896057128906, 500.5908203125], [556.7855834960938, 502.1676330566406, 898.7138671875, 663.689208984375], [975.0003662109375, 310.24560546875, 1017.2863159179688, 543.490234375], [460.99609375, 378.6252136230469, 550.0560913085938, 528.1303100585938], [816.5828247070312, 338.43603515625, 949.7393798828125, 591.0054931640625], [190.00460815429688, 386.2022705078125, 280.47198486328125, 490.88336181640625], [58.10138702392578, 385.07672119140625, 76.00296020507812, 419.25921630859375], [975.1248168945312, 324.2934875488281, 1018.8331298828125, 537.5548095703125]], "labels": [16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 9, 16, 62, 16, 16, 16, 16, 16, 9, 16, 16, 86, 16, 9, 16, 16, 16], "scores": [0.9959748387336731, 0.9662884473800659, 0.9468867778778076, 0.9012860059738159, 0.8116380572319031, 0.8029219508171082, 0.7754765152931213, 0.6982285380363464, 0.6950058341026306, 0.5501882433891296, 0.5014765858650208, 0.4762125313282013, 0.38233062624931335, 0.3588114380836487, 0.31153592467308044, 0.31089678406715393, 0.2889587879180908, 0.2508528530597687, 0.14590781927108765, 0.134654238820076, 0.12695468962192535, 0.12277735769748688, 0.12031322717666626, 0.11930085718631744, 0.11418244242668152, 0.11356015503406525, 0.07360801845788956, 0.059812020510435104, 0.0576489083468914, 0.05617543309926987, 0.055970221757888794]}, {"boxes": [[521.366943359375, 352.35516357421875, 862.1326904296875, 672.0140991210938], [239.7632293701172, 404.4142150878906, 405.1598205566406, 712.5680541992188], [446.54522705078125, 328.2021484375, 625.0435180664062, 544.1801147460938], [1.7718359231948853, 406.21624755859375, 121.39753723144531, 553.8434448242188], [22.111074447631836, 527.2021484375, 258.99957275390625, 663.0960083007812], [22.93181610107422, 308.6789855957031, 361.956787109375, 642.4185180664062], [703.6611328125, 539.2216186523438, 880.080078125, 629.238525390625], [111.431640625, 281.60565185546875, 312.748779296875, 506.89300537109375], [548.5543823242188, 401.7587890625, 744.8442993164062, 641.3017578125], [862.2832641601562, 468.46673583984375, 1011.3546752929688, 551.2216186523438], [434.26202392578125, 320.01953125, 497.5529479980469, 542.5452880859375], [424.6571350097656, 338.869873046875, 720.90576171875, 602.6569213867188], [854.1563110351562, 373.54107666015625, 1006.0125732421875, 550.1716918945312], [543.5879516601562, 347.70257568359375, 674.861083984375, 557.3649291992188], [38.95980453491211, 377.4406433105469, 92.82644653320312, 422.4919128417969], [553.9835205078125, 485.6164855957031, 816.1971435546875, 679.4160766601562], [243.51864624023438, 558.74072265625, 409.025390625, 716.4395751953125], [137.50677490234375, 382.69476318359375, 380.3227233886719, 694.5613403320312], [6.490341663360596, 386.5841064453125, 178.8524627685547, 600.2326049804688], [609.177978515625, 436.816162109375, 676.6406860351562, 525.8944702148438], [108.99332427978516, 399.2375793457031, 336.88995361328125, 545.2173461914062], [3.6606101989746094, 302.13311767578125, 153.8372802734375, 549.3624877929688], [850.5454711914062, 344.18426513671875, 1010.555419921875, 559.2879028320312], [106.53793334960938, 271.27545166015625, 161.6510009765625, 522.5968017578125], [613.028564453125, 520.57958984375, 895.962646484375, 622.0994873046875], [615.16650390625, 371.61273193359375, 700.574462890625, 593.7899780273438], [34.09622573852539, 364.4704895019531, 52.77350616455078, 392.9521484375], [594.5552978515625, 386.1797180175781, 676.4244384765625, 541.0731811523438], [618.3447265625, 404.5936279296875, 877.7166137695312, 617.6603393554688], [26.581424713134766, 289.61395263671875, 245.1150360107422, 558.435546875], [20.168867111206055, 409.1775817871094, 311.2741394042969, 582.650146484375], [89.71167755126953, 288.960693359375, 203.53790283203125, 587.3070678710938], [507.3363952636719, 343.2095947265625, 598.2318725585938, 529.1009521484375]], "labels": [16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 9, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16], "scores": [0.9839367866516113, 0.9439980983734131, 0.9174496531486511, 0.9072659015655518, 0.849727988243103, 0.8494443893432617, 0.763895571231842, 0.7617740631103516, 0.5313340425491333, 0.5156661868095398, 0.47015616297721863, 0.46304604411125183, 0.45955827832221985, 0.3593994379043579, 0.34603753685951233, 0.3182562291622162, 0.2867828905582428, 0.2823019027709961, 0.2645604908466339, 0.22618520259857178, 0.2186415195465088, 0.21757304668426514, 0.1694456785917282, 0.14943748712539673, 0.14503274857997894, 0.11115000396966934, 0.10167347639799118, 0.07605578005313873, 0.07397910207509995, 0.0700848177075386, 0.06706711649894714, 0.061569586396217346, 0.05290675908327103]}] --------------------------------------------------------------------------------