├── functions ├── __init__.py ├── sqs_poller │ ├── __init__.py │ ├── requirements.txt │ └── app.py ├── start_model │ ├── __init__.py │ ├── requirements.txt │ └── app.py ├── stop_model │ ├── __init__.py │ ├── requirements.txt │ └── app.py ├── analyse_image │ ├── __init__.py │ ├── requirements.txt │ ├── app.py │ └── messageformats.txt └── toggle_trigger │ ├── __init__.py │ ├── requirements.txt │ └── app.py ├── requirements.txt ├── docs ├── deploy-to-aws.png ├── SA-Amazon Rekognition Custom Labels Batch Image Processing.png └── Solution Architecture - Serverless Computer Vision Label Detection.png ├── .gitignore ├── cfn-publish.config ├── CODE_OF_CONDUCT.md ├── Makefile ├── LICENSE ├── .github └── workflows │ ├── publish.yaml │ └── release.yaml ├── CONTRIBUTING.md ├── template.yaml └── README.md /functions/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /functions/sqs_poller/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /functions/start_model/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /functions/stop_model/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /functions/analyse_image/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /functions/sqs_poller/requirements.txt: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /functions/start_model/requirements.txt: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /functions/stop_model/requirements.txt: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /functions/toggle_trigger/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /functions/analyse_image/requirements.txt: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /functions/toggle_trigger/requirements.txt: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | awscli==1.16.292 2 | cfn-flip==1.2.3 3 | -------------------------------------------------------------------------------- /docs/deploy-to-aws.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-rekognition-custom-labels-batch-processing/HEAD/docs/deploy-to-aws.png -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | .aws-sam 2 | .DS_Store 3 | packaged.zip 4 | samconfig.toml 5 | docs/SA-Amazon Rekognition Custom Labels Batch Image Processing_old.png 6 | -------------------------------------------------------------------------------- /cfn-publish.config: -------------------------------------------------------------------------------- 1 | template=template.yaml 2 | acl="public-read" 3 | bucket_name_prefix="solution-builders" 4 | regions="eu-west-1 us-east-1 us-east-2 us-west-2 ap-southeast-2 eu-west-2 eu-central-1" 5 | -------------------------------------------------------------------------------- /docs/SA-Amazon Rekognition Custom Labels Batch Image Processing.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-rekognition-custom-labels-batch-processing/HEAD/docs/SA-Amazon Rekognition Custom Labels Batch Image Processing.png -------------------------------------------------------------------------------- /docs/Solution Architecture - Serverless Computer Vision Label Detection.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-rekognition-custom-labels-batch-processing/HEAD/docs/Solution Architecture - Serverless Computer Vision Label Detection.png -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | ## Code of Conduct 2 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 3 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 4 | opensource-codeofconduct@amazon.com with any additional questions or comments. 5 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | SHELL := /bin/bash 2 | 3 | package: 4 | zip -r packaged.zip \ 5 | functions \ 6 | cfn-publish.config \ 7 | template.yaml \ 8 | -x '**/__pycache*' @ 9 | 10 | version: 11 | @echo $(shell cfn-flip template.yaml | python -c 'import sys, json; print(json.load(sys.stdin)["Mappings"]["Solution"]["Constants"]["Version"])') 12 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining a copy of 4 | this software and associated documentation files (the "Software"), to deal in 5 | the Software without restriction, including without limitation the rights to 6 | use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of 7 | the Software, and to permit persons to whom the Software is furnished to do so. 8 | 9 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 10 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS 11 | FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 12 | COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER 13 | IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN 14 | CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 15 | 16 | -------------------------------------------------------------------------------- /.github/workflows/publish.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | 3 | name: Publish Version 4 | on: 5 | release: 6 | types: [created, edited] 7 | permissions: 8 | contents: read 9 | 10 | jobs: 11 | publish: 12 | name: Publish Version 13 | runs-on: ubuntu-latest 14 | steps: 15 | - uses: actions/checkout@v2 16 | - name: Fetch Tags 17 | run: git fetch --depth=1 origin +refs/tags/*:refs/tags/* || true 18 | - name: Configure AWS credentials 19 | uses: aws-actions/configure-aws-credentials@v1 20 | with: 21 | aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} 22 | aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} 23 | aws-session-token: ${{ secrets.AWS_SESSION_TOKEN }} 24 | aws-region: ${{ secrets.REGION }} 25 | - name: Set version 26 | id: version 27 | run: echo "VERSION=${GITHUB_REF/refs\/tags\//}" >> $GITHUB_ENV 28 | # Cache 29 | - uses: actions/cache@v1 30 | with: 31 | path: ~/.cache/pip 32 | key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }} 33 | restore-keys: | 34 | ${{ runner.os }}-pip- 35 | # Setup 36 | - name: Set up Python 3.8 37 | uses: actions/setup-python@v1 38 | with: 39 | python-version: 3.8 40 | - name: Install python dependencies 41 | run: pip3 install -r requirements.txt 42 | # Package and Upload Archive 43 | - name: Build Release 44 | run: make package 45 | - name: Upload artefact 46 | run: aws s3 cp packaged.zip s3://$CFN_BUCKET/amazon-rekognition-custom-labels-batch-processing/$VERSION/amazon-rekognition-custom-labels-batch-processing.zip 47 | env: 48 | CFN_BUCKET: ${{ secrets.CFN_BUCKET }} 49 | -------------------------------------------------------------------------------- /functions/sqs_poller/app.py: -------------------------------------------------------------------------------- 1 | # /* 2 | # * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | # * SPDX-License-Identifier: MIT-0 4 | # * 5 | # * Permission is hereby granted, free of charge, to any person obtaining a copy of this 6 | # * software and associated documentation files (the "Software"), to deal in the Software 7 | # * without restriction, including without limitation the rights to use, copy, modify, 8 | # * merge, publish, distribute, sublicense, and/or sell copies of the Software, and to 9 | # * permit persons to whom the Software is furnished to do so. 10 | # * 11 | # * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, 12 | # * INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A 13 | # * PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT 14 | # * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION 15 | # * OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE 16 | # * SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 17 | # */ 18 | 19 | import boto3 20 | import os 21 | 22 | 23 | def lambda_handler(event, context): 24 | 25 | # Create SQS client 26 | sqs = boto3.client('sqs') 27 | src_queue_url = os.environ['SQS_Queue_URL'] 28 | # Check message available in Incoming Queue 29 | response = sqs.get_queue_attributes( 30 | QueueUrl=src_queue_url, 31 | AttributeNames=[ 32 | 'ApproximateNumberOfMessages' 33 | ] 34 | ) 35 | count = response['Attributes']['ApproximateNumberOfMessages'][0] 36 | print('Message Count in Incoming Queue: %s' % count) 37 | if (int(count) > 0): 38 | return 'incoming' 39 | else: 40 | return 'stop' 41 | -------------------------------------------------------------------------------- /.github/workflows/release.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | 3 | name: Release Version 4 | on: 5 | push: 6 | branches: 7 | - main 8 | permissions: 9 | contents: write 10 | 11 | jobs: 12 | release: 13 | name: Release Version 14 | runs-on: ubuntu-latest 15 | steps: 16 | - uses: actions/checkout@v2 17 | - run: git fetch --depth=1 origin +refs/tags/*:refs/tags/* || true 18 | # Cache 19 | - uses: actions/cache@v1 20 | with: 21 | path: ~/.cache/pip 22 | key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }} 23 | restore-keys: | 24 | ${{ runner.os }}-pip- 25 | # Setup 26 | - name: Set up Python 3.8 27 | uses: actions/setup-python@v1 28 | with: 29 | python-version: 3.8 30 | - name: Install python dependencies 31 | run: pip3 install -r requirements.txt 32 | # Release if required 33 | - name: Set version 34 | id: version 35 | run: | 36 | function version { echo "$@" | awk -F. '{ printf("%d%03d%03d%03d\n", $1,$2,$3,$4); }'; } 37 | echo "THIS_VERSION=$(make version | sed s/^v//)" >> $GITHUB_ENV 38 | echo "THIS_VERSION_COMPARABLE=$(version $(make version | sed s/^v//))" >> $GITHUB_ENV 39 | echo "LATEST_VERSION_COMPARABLE=$(version $(git describe --tags $(git rev-list --tags --max-count=1) | sed s/^v// 2> /dev/null || echo '0'))" >> $GITHUB_ENV 40 | - name: Create Release 41 | id: create_release 42 | uses: actions/create-release@latest 43 | if: env.THIS_VERSION_COMPARABLE > env.LATEST_VERSION_COMPARABLE 44 | env: 45 | GITHUB_TOKEN: ${{ secrets.RELEASE_TOKEN }} 46 | with: 47 | tag_name: v${{ env.THIS_VERSION }} 48 | release_name: Release v${{ env.THIS_VERSION }} 49 | body: | 50 | See the commits for a list of features included in this release 51 | draft: false 52 | prerelease: false 53 | -------------------------------------------------------------------------------- /functions/stop_model/app.py: -------------------------------------------------------------------------------- 1 | # /* 2 | # * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | # * SPDX-License-Identifier: MIT-0 4 | # * 5 | # * Permission is hereby granted, free of charge, to any person obtaining a copy of this 6 | # * software and associated documentation files (the "Software"), to deal in the Software 7 | # * without restriction, including without limitation the rights to use, copy, modify, 8 | # * merge, publish, distribute, sublicense, and/or sell copies of the Software, and to 9 | # * permit persons to whom the Software is furnished to do so. 10 | # * 11 | # * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, 12 | # * INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A 13 | # * PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT 14 | # * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION 15 | # * OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE 16 | # * SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 17 | # */ 18 | 19 | import json 20 | import boto3 21 | import os 22 | 23 | 24 | def lambda_handler(event, context): 25 | 26 | rekog_client = boto3.client('rekognition') 27 | projectversionarn = os.environ['rekog_model_project_version_arn'] 28 | projectarn = os.environ['rekog_model_project_arn'] 29 | running_states = ['STARTING', 'RUNNING'] 30 | projectversionname = projectversionarn.split("/")[3] 31 | # Check if already running 32 | # Call Custom Rekog 33 | try: 34 | isrunning_response = rekog_client.describe_project_versions( 35 | ProjectArn=projectarn, 36 | VersionNames=[projectversionname] 37 | ) 38 | except Exception as e: 39 | print(e) 40 | running_status = isrunning_response['ProjectVersionDescriptions'][0]['Status'] 41 | if running_status in running_states: 42 | # Stop Model 43 | try: 44 | running_status = rekog_client.stop_project_version( 45 | ProjectVersionArn=projectversionarn 46 | ) 47 | except Exception as e: 48 | print(e) 49 | print('Model Start Status: %s' % running_status) 50 | else: 51 | # If not running - Do Nothing 52 | print('Model Start Status: %s' % running_status) 53 | return running_status 54 | -------------------------------------------------------------------------------- /functions/start_model/app.py: -------------------------------------------------------------------------------- 1 | # /* 2 | # * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | # * SPDX-License-Identifier: MIT-0 4 | # * 5 | # * Permission is hereby granted, free of charge, to any person obtaining a copy of this 6 | # * software and associated documentation files (the "Software"), to deal in the Software 7 | # * without restriction, including without limitation the rights to use, copy, modify, 8 | # * merge, publish, distribute, sublicense, and/or sell copies of the Software, and to 9 | # * permit persons to whom the Software is furnished to do so. 10 | # * 11 | # * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, 12 | # * INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A 13 | # * PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT 14 | # * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION 15 | # * OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE 16 | # * SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 17 | # */ 18 | 19 | import boto3 20 | import os 21 | 22 | 23 | def lambda_handler(event, context): 24 | 25 | rekog_client = boto3.client('rekognition') 26 | projectversionarn = os.environ['rekog_model_project_version_arn'] 27 | projectarn = os.environ['rekog_model_project_arn'] 28 | # running_states = ['STARTING', 'RUNNING'] 29 | projectversionname = projectversionarn.split("/")[3] 30 | # Check if already running 31 | # Call Custom Rekognition project version 32 | try: 33 | isrunning_response = rekog_client.describe_project_versions( 34 | ProjectArn=projectarn, 35 | VersionNames=[projectversionname] 36 | ) 37 | except Exception as e: 38 | print(e) 39 | 40 | running_status = isrunning_response['ProjectVersionDescriptions'][0]['Status'] 41 | if running_status == 'RUNNING': 42 | # Do nothing 43 | print('Model Start Status: %s' % running_status) 44 | return 'RUNNING' 45 | if running_status == 'STARTING': 46 | # Do nothing 47 | print('Model Start Status: %s' % running_status) 48 | return 'STARTING' 49 | else: 50 | # If not running - Start 51 | try: 52 | running_status = rekog_client.start_project_version( 53 | ProjectVersionArn = projectversionarn, 54 | MinInferenceUnits = 1 #Can be increased upto 5 for running multiple inference units 55 | ) 56 | except Exception as e: 57 | print(e) 58 | return running_status 59 | 60 | -------------------------------------------------------------------------------- /functions/toggle_trigger/app.py: -------------------------------------------------------------------------------- 1 | # /* 2 | # * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | # * SPDX-License-Identifier: MIT-0 4 | # * 5 | # * Permission is hereby granted, free of charge, to any person obtaining a copy of this 6 | # * software and associated documentation files (the "Software"), to deal in the Software 7 | # * without restriction, including without limitation the rights to use, copy, modify, 8 | # * merge, publish, distribute, sublicense, and/or sell copies of the Software, and to 9 | # * permit persons to whom the Software is furnished to do so. 10 | # * 11 | # * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, 12 | # * INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A 13 | # * PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT 14 | # * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION 15 | # * OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE 16 | # * SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 17 | # */ 18 | 19 | import json 20 | import boto3 21 | import os 22 | 23 | 24 | def lambda_handler(event, context): 25 | 26 | lambda_client = boto3.client('lambda') 27 | try: 28 | uuid_response = lambda_client.list_event_source_mappings( 29 | FunctionName=os.environ['analyze_lambda_arn'] 30 | ) 31 | except Exception as e: 32 | print(e) 33 | 34 | mylist = uuid_response['EventSourceMappings'] 35 | uuiddata = mylist[0]['UUID'] 36 | analyse_lambda_uuid = uuiddata 37 | 38 | try: 39 | response = lambda_client.get_event_source_mapping( 40 | UUID=analyse_lambda_uuid 41 | ) 42 | except Exception as e: 43 | print(e) 44 | 45 | # State (string) -- The state of the event source mapping. It can be one of the following: Creating , Enabling , Enabled , Disabling , Disabled , Updating , or Deleting . 46 | disabled_states = ["Disabled", "Disabling"] 47 | enabled_states = ["Enabling", "Enabled"] 48 | 49 | # Disable 50 | if (event[0]['Action'] == 'disable'): 51 | if (response['State'] in disabled_states): 52 | # Do Nothing 53 | return 'Already disabled' 54 | else: 55 | try: 56 | response = lambda_client.update_event_source_mapping( 57 | UUID=analyse_lambda_uuid, 58 | Enabled=False 59 | ) 60 | except Exception as e: 61 | print(e) 62 | else: 63 | # Enable 64 | if (event[0]['Action'] == 'enable'): 65 | if (response['State'] in enabled_states): 66 | # Do Nothing 67 | return 'Already_Running' 68 | else: 69 | try: 70 | response = lambda_client.update_event_source_mapping( 71 | UUID=analyse_lambda_uuid, 72 | Enabled=True 73 | ) 74 | except Exception as e: 75 | print(e) 76 | 77 | print("Current state is:", response['State']) 78 | return response['State'] 79 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing Guidelines 2 | 3 | Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional 4 | documentation, we greatly value feedback and contributions from our community. 5 | 6 | Please read through this document before submitting any issues or pull requests to ensure we have all the necessary 7 | information to effectively respond to your bug report or contribution. 8 | 9 | 10 | ## Reporting Bugs/Feature Requests 11 | 12 | We welcome you to use the GitHub issue tracker to report bugs or suggest features. 13 | 14 | When filing an issue, please check existing open, or recently closed, issues to make sure somebody else hasn't already 15 | reported the issue. Please try to include as much information as you can. Details like these are incredibly useful: 16 | 17 | * A reproducible test case or series of steps 18 | * The version of our code being used 19 | * Any modifications you've made relevant to the bug 20 | * Anything unusual about your environment or deployment 21 | 22 | 23 | ## Contributing via Pull Requests 24 | Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that: 25 | 26 | 1. You are working against the latest source on the *main* branch. 27 | 2. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already. 28 | 3. You open an issue to discuss any significant work - we would hate for your time to be wasted. 29 | 30 | To send us a pull request, please: 31 | 32 | 1. Fork the repository. 33 | 2. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change. 34 | 3. Ensure local tests pass. 35 | 4. Commit to your fork using clear commit messages. 36 | 5. Send us a pull request, answering any default questions in the pull request interface. 37 | 6. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation. 38 | 39 | GitHub provides additional document on [forking a repository](https://help.github.com/articles/fork-a-repo/) and 40 | [creating a pull request](https://help.github.com/articles/creating-a-pull-request/). 41 | 42 | 43 | ## Finding contributions to work on 44 | Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels (enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any 'help wanted' issues is a great place to start. 45 | 46 | 47 | ## Code of Conduct 48 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 49 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 50 | opensource-codeofconduct@amazon.com with any additional questions or comments. 51 | 52 | 53 | ## Security issue notifications 54 | If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github issue. 55 | 56 | 57 | ## Licensing 58 | 59 | See the [LICENSE](LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution. 60 | -------------------------------------------------------------------------------- /functions/analyse_image/app.py: -------------------------------------------------------------------------------- 1 | # /* 2 | # * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | # * SPDX-License-Identifier: MIT-0 4 | # * 5 | # * Permission is hereby granted, free of charge, to any person obtaining a copy of this 6 | # * software and associated documentation files (the "Software"), to deal in the Software 7 | # * without restriction, including without limitation the rights to use, copy, modify, 8 | # * merge, publish, distribute, sublicense, and/or sell copies of the Software, and to 9 | # * permit persons to whom the Software is furnished to do so. 10 | # * 11 | # * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, 12 | # * INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A 13 | # * PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT 14 | # * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION 15 | # * OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE 16 | # * SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 17 | # */ 18 | 19 | import json 20 | import os 21 | import boto3 22 | import logging 23 | import string 24 | import random 25 | from botocore.exceptions import ClientError 26 | 27 | 28 | def lambda_handler(event, context): 29 | # Create s3 client 30 | s3_client = boto3.client('s3') 31 | # Create Rekognition Client 32 | client = boto3.client('rekognition') 33 | model_arn = os.environ['rekognition_model_project_version_arn'] 34 | 35 | for msg in event["Records"]: 36 | msg_payload = json.loads(msg["body"]) 37 | print("msg_payload: ", msg_payload) 38 | if "Records" in msg_payload: 39 | bucket = msg_payload["Records"][0]["s3"]["bucket"]["name"] 40 | image = msg_payload["Records"][0]["s3"]["object"]["key"].replace("+", " ") 41 | response = client.detect_custom_labels( 42 | ProjectVersionArn = model_arn, 43 | Image={ 44 | 'S3Object': { 45 | 'Bucket': bucket, 46 | 'Name': image} 47 | }, 48 | MinConfidence = 70 49 | ) 50 | # Get the custom labels 51 | labels = response['CustomLabels'] 52 | # write image to final bucket and delete from incoming bucket 53 | s3 = boto3.resource('s3') 54 | finalbucket = os.environ['Final_S3_Bucket_Name'] 55 | copy_source = { 56 | 'Bucket': bucket, 57 | 'Key': image 58 | } 59 | random_letters = ''.join(random.choice(string.ascii_letters) for i in range(10)) 60 | put_image_name = random_letters+'-'+image 61 | s3.meta.client.copy(copy_source, finalbucket, put_image_name) 62 | 63 | # Dump json file with label data in final bucket 64 | json_object = json.dumps(labels) 65 | s3_client.put_object( 66 | Body = str(json_object), 67 | Bucket = finalbucket, 68 | Key = put_image_name+'.json' 69 | ) 70 | 71 | # Delete file from incoming s3 72 | s3_client.delete_object( 73 | Bucket = bucket, 74 | Key = image, 75 | ) 76 | 77 | else: 78 | # Invalid Message - To Be removed from Queue 79 | print("Invalid msg: ", msg) 80 | 81 | return {'status': '200'} 82 | -------------------------------------------------------------------------------- /functions/analyse_image/messageformats.txt: -------------------------------------------------------------------------------- 1 | 2 | msg_payload: {'Records': [{'eventVersion': '2.1', 'eventSource': 'aws:s3', 'awsRegion': 'us-east-1', 'eventTime': '2020-12-24T01:38:09.177Z', 'eventName': 'ObjectCreated:Put', 'userIdentity': {'principalId': 'AWS:AROA4VCA4ALVF4U6VUISX:rrahsri-Isengard'}, 'requestParameters': {'sourceIPAddress': '86.3.161.154'}, 'responseElements': {'x-amz-request-id': 'B566A449E7E0C55C', 'x-amz-id-2': 'hAzxb0FlQvN6DUv3M53XaeSIZC2nFnmFePD4BDc6pTYHaxkOuR920l8PahKLIyInLq0pbdVpOyws8xjCTqr22yc6qDfj/eqz'}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'c11f7ec2-27d3-4e24-a47e-d23a268e464b', 'bucket': {'name': 'serverless-cv-custom-label-detecti-sources3bucket-831ongtw0at0', 'ownerIdentity': {'principalId': 'A15PPX6WRZQDJU'}, 'arn': 'arn:aws:s3:::serverless-cv-custom-label-detecti-sources3bucket-831ongtw0at0'}, 'object': {'key': 'Screenshot+2020-11-17+at+22.27.01.png', 'size': 241069, 'eTag': 'ea66b44b23d0a49068a48d09a276ca00', 'sequencer': '005FE3F1017ABEE7D9'}}}]} 3 | 4 | 5 | msg_payload: {'Service': 'Amazon S3', 'Event': 's3:TestEvent', 'Time': '2020-12-24T01:34:03.091Z', 'Bucket': 'serverless-cv-custom-label-detecti-sources3bucket-831ongtw0at0', 'RequestId': 'A9EE9D353383BB4F', 'HostId': 'XbCl4x66Sl0HsvF2niKw0wYnJM0h3Dr3cCDJymGgA+47FgrDdiVPAbmVTArrkgPlsxkcRQQ4J0k='} 6 | Invalid msg: {'messageId': 'ad10228d-6845-4f1c-88b5-b20d4c585486', 'receiptHandle': 'AQEByeVAnChC2DFc3WO8Vb6lNWMtqzYSQHCS23Sw7z9aOAn8dH/1xp6LjVb3hOGAKY3Jj883SKRX3I/lEWXWzgdLCFiLT6a0gCgxlZBA0EjHxzFcbR7meRKaOPA/HFJGTF7c5V+C/UAUts1LnUIb7jEO0ueEvFY7VUsylGqIqImx0xGgRH4RQPpGBw74lR3l0odsylxxUCDZ6PRSbNRtsy7afTj46Yr3qOz1jtSye1Tc5KbAsSuKVEQ+CvysoSMH/tGf8jrQ6hbIxHR+lLU98t97OJAJwYqAx7f+vOWcwbIcAHW7r+hmukJoYvW4P4IQV7U0N3xQnzu+yBHn+jWoPWsQ19lOY+IoQ0qxFoiaKAqO7HVoOECe1NmzHnVxnBrWzYH5R9PMrYKsIofFjmxPgad5WRJYzPOaaApUseCPepQiXHlFf/Hb5HG0MjvK2OEM0Gev', 'body': '{"Service":"Amazon S3","Event":"s3:TestEvent","Time":"2020-12-24T01:34:03.091Z","Bucket":"serverless-cv-custom-label-detecti-sources3bucket-831ongtw0at0","RequestId":"A9EE9D353383BB4F","HostId":"XbCl4x66Sl0HsvF2niKw0wYnJM0h3Dr3cCDJymGgA+47FgrDdiVPAbmVTArrkgPlsxkcRQQ4J0k="}', 'attributes': {'ApproximateReceiveCount': '1', 'SentTimestamp': '1608773643232', 'SenderId': 'AIDAJHIPRHEMV73VRJEBU', 'ApproximateFirstReceiveTimestamp': '1608773928805'}, 'messageAttributes': {}, 'md5OfBody': 'bd9f6cb5f52402842920f5cc4a37d49a', 'eventSource': 'aws:sqs', 'eventSourceARN': 'arn:aws:sqs:us-east-1:869866930922:serverless-cv-custom-label-detection-SQSQueue-Q6PMWXD8G3TS', 'awsRegion': 'us-east-1'} 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | ================================ 27 | Invalid msg: %s {'messageId': 'f9e5b3a5-1adc-446d-ac9d-b2821853841d', 28 | 'receiptHandle': 'AQEB02SbhgUpnWLlxvrYZ/zURrDHcjFLVhFa3a+KmyyQRMWgdlJqNqa7zmLLUmAupLfQKyPE8lwaEgbw+QvAELpm6NR7stnNP5jBgcb9HEs+4VWDkbfSIV71pfwBY6T8FwQ6194OHz7CsgvOGcrCFc76XH9gIbVHq6qJIhTh2CjL6KCFVhNQXOPh0O51+hhlX3ZCK4J1GsDonvAwlIDYZh1zI+N1HdzbltAjTTjZcK/JwjqoLp3nu7vvXdrACMuHD7CfcXd4jPQbieRrbf1rGe6teYtQ0EVntnMxXqijZ8uXuUNye7rxGebbtyHBvlQUp62HeQcM2zI/O5e8smBY5i3q0n5OWXQFUn3XkwzMsDAAcQjgkcr2xpKQ1OXe1xr1S0ASyvbqSFJOH8Dm+7jLCkVg3KPN0k+FQawpHYQ5q2VCAlJK+tdL5aPGxDB4gxvQ0wC/', 29 | 'body': '{ 30 | "Service":"Amazon S3", 31 | "Event":"s3:TestEvent", 32 | "Time":"2020-12-23T12:20:08.265Z", 33 | "Bucket":"serverless-cv-custom-label-detecti-sources3bucket-xp4y8o6608q", 34 | "RequestId":"64F279B99DCAC18F", 35 | "HostId":"r+aRevq/Y9iVi8syij7/gD7aONVW1N7j5EQSGfkXg/UCK9/IqRKkiBetaIUuA4hDMX+BkILRrUE="}', 36 | 'attributes': { 37 | 'ApproximateReceiveCount': '1', 38 | 'SentTimestamp': '1608726008297', 39 | 'SenderId': 'AIDAJHIPRHEMV73VRJEBU', 40 | 'ApproximateFirstReceiveTimestamp': '1608727778667' 41 | }, 42 | 'messageAttributes': {}, 43 | 'md5OfBody': 'c0c91d6ae46c96d8736cd78f9fe63468', 44 | 'eventSource': 'aws:sqs', 45 | 'eventSourceARN': 'arn:aws:sqs:us-east-1:869866930922:serverless-cv-custom-label-detection-SQSQueue-4BEXLXR1EUYT', 46 | 'awsRegion': 'us-east-1'} 47 | 48 | 49 | Invalid msg: %s {'messageId': 'd93e569c-d685-4aa3-a07b-2321bca69096', 50 | 'receiptHandle': 'AQEBFABH1dxVoxj9NWv2oHRwMZh5b9QcjCNGj4/VVGIn+HMt2h7MtVmbbFRuwzo0h3nt4HtwgB8kpb0sl2GiyEyyR23k152o6KuNr3R0DZUTtVxu1+LbjEDZgbSiAMm69WrnluCqHGtae1c70PhPIeEIJ15qHjDwpSiyYOKdAkpJ9BgE5Cu9Xb+9lTcjrZItcicft0nWJNZt4vofwJyvFpQWqelWH0veJn6c68eLOf2/S3BBMb9DYLrMXy2C8LQZMm60zrAAUQM10guXFc76FEMOewS2eJtZ652+UQZ1+6kCAUMz3XzrtcK55Gfjzsj0VsRxI0C4KlDTFKAKsaUbNZ7JDmYt6NkuP5xXwMm/8Llcpl5EB+6SPAnt5qHhx1QFPXCc4PFIyaVkrc+V555sm8TRlblIxQ+yBnx9xBjjC8Y7VvFBU86Tu/d/X2/KSUUzLN7g', 51 | 'body': '{ 52 | "Records":[ 53 | {"eventVersion":"2.1", 54 | "eventSource":"aws:s3", 55 | "awsRegion":"us-east-1", 56 | "eventTime":"2020-12-23T12:40:10.695Z", 57 | "eventName":"ObjectCreated:Put", 58 | "userIdentity":{"principalId":"AWS:AROA4VCA4ALVF4U6VUISX:rrahsri-Isengard"}, 59 | "requestParameters":{"sourceIPAddress":"86.3.161.154"}, 60 | "responseElements":{"x-amz-request-id":"87516D517F42EF4D","x-amz-id-2":"1+p+sKONjMO1Fz6ssXCLps0Kt3OMcaDuUrK3urJZe8B6n5i8DZOpijZ8LAbcFHTvbdUci4aHzvmd0oIZ4ZEU4jzztxyDpL6e"}, 61 | "s3":{ 62 | "s3SchemaVersion":"1.0", 63 | "configurationId":"6cd66d32-2917-4a2f-8d9f-d8e004c09576", 64 | "bucket":{ 65 | "name":"serverless-cv-custom-label-detecti-sources3bucket-xp4y8o6608q", 66 | "ownerIdentity":{"principalId":"A15PPX6WRZQDJU"}, 67 | "arn":"arn:aws:s3:::serverless-cv-custom-label-detecti-sources3bucket-xp4y8o6608q"}, 68 | "object":{ 69 | "key":"Screenshot+2020-11-17+at+22.27.35.png","size":140517,"eTag":"94cb6c1a8ea14e7e89a83a31e389ab8e","sequencer":"005FE33AB244FCB3BE"}}} 70 | ] 71 | }', 72 | 'attributes': {'ApproximateReceiveCount': '1', 'SentTimestamp': '1608727219582', 'SenderId': 'AIDAJHIPRHEMV73VRJEBU', 'ApproximateFirstReceiveTimestamp': '1608727778733'}, 73 | 'messageAttributes': {}, 74 | 'md5OfBody': '2aa32e28afb02a89abeac032ca8bab3d', 75 | 'eventSource': 'aws:sqs', 76 | 'eventSourceARN': 'arn:aws:sqs:us-east-1:869866930922:serverless-cv-custom-label-detection-SQSQueue-4BEXLXR1EUYT', 77 | 'awsRegion': 'us-east-1'} 78 | 79 | 80 | 81 | 82 | 83 | 84 | Invalid data2: 85 | 86 | {'Records': [ 87 | { 88 | 'eventVersion': '2.1', 89 | 'eventSource': 'aws:s3', 90 | 'awsRegion': 'us-east-1', 91 | 'eventTime': '2020-12-23T14:08:38.898Z', 92 | 'eventName': 'ObjectCreated:Put', 93 | 'userIdentity': {'principalId': 'AWS:AROA4VCA4ALVF4U6VUISX:rrahsri-Isengard'}, 94 | 'requestParameters': {'sourceIPAddress': '86.3.161.154'}, 95 | 'responseElements': { 96 | 'x-amz-request-id': '9674C4609FBB2ADB', 97 | 'x-amz-id-2': 'I+TzooXAtIL8x04V3OQehielgn7QxK9JejxY05yOPoxoe9B1wa750PAbass3i2l8JBwiAOWhd+wIHAP5Cf6I2YhTk9/BMJCb'}, 98 | 's3': { 99 | 's3SchemaVersion': '1.0', 100 | 'configurationId': '3622f400-27cc-4296-bd41-f7af661e1a0f', 101 | 'bucket': { 102 | 'name': 'serverless-cv-custom-label-detecti-sources3bucket-mnfom9araj8z', 103 | 'ownerIdentity': {'principalId': 'A15PPX6WRZQDJU'}, 104 | 'arn': 'arn:aws:s3:::serverless-cv-custom-label-detecti-sources3bucket-mnfom9araj8z'}, 105 | 'object': { 106 | 'key': 'Screenshot+2020-11-17+at+22.28.13.png', 107 | 'size': 194551, 'eTag': '445fc230a1ae36e1d0f5d318ac310f43', 108 | 'sequencer': '005FE34F690D1607F1'}}}]} 109 | 110 | -------------------------------------------------------------------------------- /template.yaml: -------------------------------------------------------------------------------- 1 | AWSTemplateFormatVersion: "2010-09-09" 2 | Transform: AWS::Serverless-2016-10-31 3 | Description: Serverless Computer Vision Label Detection (uksb-1reh3a0p6) 4 | Parameters: 5 | RekognitionModelProjectARN: 6 | Type: String 7 | Description: Amazon Rekognition Model Project ARN 8 | RekognitionModelProjectVersionARN: 9 | Type: String 10 | Description: Amazon Rekognition Model Project Version ARN 11 | 12 | Mappings: 13 | Solution: 14 | Constants: 15 | Version: "v0.13" 16 | 17 | Resources: 18 | SourceS3Bucket: 19 | Type: AWS::S3::Bucket 20 | DeletionPolicy: Retain 21 | UpdateReplacePolicy: Retain 22 | Properties: 23 | BucketEncryption: 24 | ServerSideEncryptionConfiguration: 25 | - ServerSideEncryptionByDefault: 26 | SSEAlgorithm: AES256 27 | NotificationConfiguration: 28 | QueueConfigurations: 29 | - Event: s3:ObjectCreated:* 30 | Queue: !GetAtt SQSQueue.Arn 31 | Filter: 32 | S3Key: 33 | Rules: 34 | - Name: suffix 35 | Value: .png 36 | - Event: s3:ObjectCreated:* 37 | Queue: !GetAtt SQSQueue.Arn 38 | Filter: 39 | S3Key: 40 | Rules: 41 | - Name: suffix 42 | Value: .jpg 43 | 44 | SQSQueue: 45 | Type: AWS::SQS::Queue 46 | Properties: 47 | DelaySeconds: 0 48 | MaximumMessageSize: 262144 49 | MessageRetentionPeriod: 345600 50 | ReceiveMessageWaitTimeSeconds: 0 51 | VisibilityTimeout: 60 52 | 53 | SQSQueuePolicy: 54 | Type: AWS::SQS::QueuePolicy 55 | Properties: 56 | PolicyDocument: 57 | Version: "2012-10-17" 58 | Statement: 59 | - Sid: "TestSID" 60 | Effect: "Allow" 61 | Principal: 62 | Service: 63 | - "lambda.amazonaws.com" 64 | - "s3.amazonaws.com" 65 | Action: 66 | - "sqs:DeleteMessage" 67 | - "sqs:ReceiveMessage" 68 | - "sqs:SendMessage" 69 | Resource: !Sub "arn:aws:sqs:${AWS::Region}:${AWS::AccountId}:${SQSQueue.QueueName}" 70 | Condition: 71 | StringEquals: 72 | "aws:SourceAccount": !Ref AWS::AccountId 73 | Queues: 74 | - !Sub "https://sqs.${AWS::Region}.amazonaws.com/${AWS::AccountId}/${SQSQueue.QueueName}" 75 | 76 | CustomCVStateMachine: 77 | Type: AWS::Serverless::StateMachine 78 | Properties: 79 | Definition: 80 | StartAt: Check SQS Queue 81 | States: 82 | Check SQS Queue: 83 | Type: Task 84 | Resource: !GetAtt SQSPollerFunction.Arn 85 | ResultPath: $.messageinqueue 86 | Next: Are there images to process? 87 | Are there images to process?: 88 | Type: Choice 89 | Choices: 90 | - Variable: $.messageinqueue 91 | StringEquals: incoming 92 | Next: Start Model 93 | Default: Finish 94 | Start Model: 95 | Type: Task 96 | Resource: !GetAtt StartModelFunction.Arn 97 | ResultPath: "$.runningstatus" 98 | Next: Start States 99 | Start States: 100 | Type: Choice 101 | Choices: 102 | - Variable: "$.runningstatus" 103 | StringEquals: RUNNING 104 | Next: Enable SQS Trigger 105 | Default: Wait for the model to start 106 | Wait for the model to start: 107 | Type: Wait 108 | Seconds: 900 109 | Next: Start Model 110 | Keep Model running for 1 hr: 111 | Type: Wait 112 | Seconds: 3540 113 | Next: Check Queue Again 114 | Check Queue Again: 115 | Type: Task 116 | Resource: !GetAtt SQSPollerFunction.Arn 117 | ResultPath: $.moremessagesinqueue 118 | Next: Are there more images? 119 | Are there more images?: 120 | Type: Choice 121 | Choices: 122 | - Variable: $.moremessagesinqueue 123 | StringEquals: stop 124 | Next: Disable SQS Trigger 125 | Default: Keep Model running for 1 hr 126 | Enable SQS Trigger: 127 | Type: Task 128 | ResultPath: $.alreadyrunning 129 | Parameters: 130 | - Action: enable 131 | Resource: !GetAtt ToggleTriggerFunction.Arn 132 | Next: Is another machine already running? 133 | Is another machine already running?: 134 | Type: Choice 135 | Choices: 136 | - Variable: $.alreadyrunning 137 | StringEquals: Already_Running 138 | Next: Finish 139 | Default: Keep Model running for 1 hr 140 | Disable SQS Trigger: 141 | Type: Task 142 | Parameters: 143 | - Action: disable 144 | Resource: !GetAtt ToggleTriggerFunction.Arn 145 | Next: Stop Model 146 | Stop Model: 147 | Type: Task 148 | Resource: !GetAtt StopModelFunction.Arn 149 | Next: Finish 150 | Finish: 151 | Type: Succeed 152 | Events: 153 | HourlyPollingSchedule: 154 | Type: Schedule # More info about Schedule Event Source: https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-property-statemachine-schedule.html 155 | Properties: 156 | Description: Schedule to run the state machine every 1 hour 157 | Enabled: True # This schedule can be disabled based on the use case to avoid incurring charges. 158 | Schedule: "rate(1 hour)" 159 | Policies: # Find out more about SAM policy templates: https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-policy-templates.html 160 | - LambdaInvokePolicy: 161 | FunctionName: !Ref SQSPollerFunction 162 | - LambdaInvokePolicy: 163 | FunctionName: !Ref StartModelFunction 164 | - LambdaInvokePolicy: 165 | FunctionName: !Ref StopModelFunction 166 | - LambdaInvokePolicy: 167 | FunctionName: !Ref ToggleTriggerFunction 168 | 169 | SQSPollerFunction: 170 | Type: AWS::Serverless::Function # More info about Function Resource: https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-resource-function.html 171 | Properties: 172 | Description: "Lambda function to Poll SQS queue for incoming messages from S3" 173 | CodeUri: functions/sqs_poller/ 174 | Handler: app.lambda_handler 175 | Environment: 176 | Variables: 177 | SQS_Queue_URL: !Ref SQSQueue 178 | Runtime: python3.8 179 | MemorySize: 128 180 | Policies: 181 | - SQSPollerPolicy: 182 | QueueName: !GetAtt SQSQueue.QueueName 183 | - LambdaInvokePolicy: 184 | FunctionName: !Ref StartModelFunction 185 | - LambdaInvokePolicy: 186 | FunctionName: !Ref StopModelFunction 187 | 188 | StartModelFunction: 189 | Type: AWS::Serverless::Function # More info about Function Resource: https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-resource-function.html 190 | Properties: 191 | Description: "Lambda function to Start the Rekognition model" 192 | CodeUri: functions/start_model/ 193 | Handler: app.lambda_handler 194 | Environment: 195 | Variables: 196 | rekog_model_project_version_arn: !Ref RekognitionModelProjectVersionARN 197 | rekog_model_project_arn: !Ref RekognitionModelProjectARN 198 | Runtime: python3.8 199 | MemorySize: 128 200 | Timeout: 3 201 | Policies: 202 | - Statement: 203 | - Action: 204 | - "rekognition:DescribeProjectVersions" 205 | - "rekognition:StartProjectVersion" 206 | Effect: "Allow" 207 | Resource: 208 | - !Ref RekognitionModelProjectARN 209 | - !Ref RekognitionModelProjectVersionARN 210 | - LambdaInvokePolicy: 211 | FunctionName: !Ref ToggleTriggerFunction 212 | 213 | StopModelFunction: 214 | Type: AWS::Serverless::Function # More info about Function Resource: https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-resource-function.html 215 | Properties: 216 | Description: "Lambda function to Stop the Rekognition model" 217 | CodeUri: functions/stop_model/ 218 | Handler: app.lambda_handler 219 | Environment: 220 | Variables: 221 | rekog_model_project_version_arn: !Ref RekognitionModelProjectVersionARN 222 | rekog_model_project_arn: !Ref RekognitionModelProjectARN 223 | Runtime: python3.8 224 | MemorySize: 128 225 | Timeout: 3 226 | Policies: 227 | - Statement: 228 | - Action: 229 | - "rekognition:DescribeProjectVersions" 230 | - "rekognition:StopProjectVersion" 231 | Effect: "Allow" 232 | Resource: 233 | - !Ref RekognitionModelProjectARN 234 | - !Ref RekognitionModelProjectVersionARN 235 | - LambdaInvokePolicy: 236 | FunctionName: !Ref ToggleTriggerFunction 237 | 238 | ToggleTriggerFunction: 239 | Type: AWS::Serverless::Function # More info about Function Resource: https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-resource-function.html 240 | Properties: 241 | Description: "Lambda function to toggle the SQS trigger for analysis" 242 | CodeUri: functions/toggle_trigger/ 243 | Handler: app.lambda_handler 244 | Environment: 245 | Variables: 246 | analyze_lambda_arn: !GetAtt AnalyseImageFunction.Arn 247 | Runtime: python3.8 248 | MemorySize: 128 249 | Timeout: 3 250 | Policies: 251 | - Statement: 252 | - Action: 253 | - "lambda:ListEventSourceMappings" 254 | - "lambda:GetEventSourceMapping" 255 | - "lambda:UpdateEventSourceMapping" 256 | Effect: "Allow" 257 | Resource: "*" 258 | 259 | FinalS3Bucket: 260 | Type: "AWS::S3::Bucket" 261 | DeletionPolicy: Retain 262 | UpdateReplacePolicy: Retain 263 | Properties: 264 | BucketEncryption: 265 | ServerSideEncryptionConfiguration: 266 | - ServerSideEncryptionByDefault: 267 | SSEAlgorithm: AES256 268 | 269 | AnalyseImageFunction: 270 | Type: AWS::Serverless::Function # More info about Function Resource: https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-resource-function.html 271 | Properties: 272 | Description: "Lambda function to analyze Images using the Rekognition Model" 273 | CodeUri: functions/analyse_image/ 274 | Handler: app.lambda_handler 275 | Environment: 276 | Variables: 277 | rekognition_model_project_version_arn: !Ref RekognitionModelProjectVersionARN 278 | Final_S3_Bucket_Name: !Ref FinalS3Bucket 279 | Events: 280 | SQSEvent: 281 | Type: SQS 282 | Properties: 283 | Queue: !GetAtt SQSQueue.Arn 284 | BatchSize: 1 285 | Enabled: false 286 | MemorySize: 128 287 | Timeout: 60 288 | Runtime: python3.8 289 | Policies: 290 | - Statement: 291 | - Action: 292 | - "rekognition:DetectCustomLabels" 293 | Effect: "Allow" 294 | Resource: 295 | - !Ref RekognitionModelProjectARN 296 | - !Ref RekognitionModelProjectVersionARN 297 | - S3CrudPolicy: 298 | BucketName: !Ref FinalS3Bucket 299 | - S3CrudPolicy: 300 | BucketName: !Ref SourceS3Bucket 301 | - SQSPollerPolicy: 302 | QueueName: !GetAtt SQSQueue.QueueName 303 | 304 | Outputs: 305 | # CustomCVStateMachineHourlySchedule is an implicit Schedule event rule created out of Events key under Serverless::StateMachine 306 | # Find out more about other implicit resources you can reference within SAM 307 | # https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-specification-generated-resources.html 308 | SourceS3BucketName: 309 | Description: "Name of the S3 bucket to hold the incoming images" 310 | Value: !Ref SourceS3Bucket 311 | FinalS3BucketName: 312 | Description: "Name of the final S3 bucket to hold the image and the inference json" 313 | Value: !Ref FinalS3Bucket 314 | CustomCVStateMachineARN: 315 | Description: "ARN of the Step Function" 316 | Value: !Ref CustomCVStateMachine 317 | CustomCVStateMachineHourlyPollingScheduleARN: 318 | Description: "ARN of the AWS EventBridge Rule" 319 | Value: !Ref CustomCVStateMachineHourlyPollingSchedule 320 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Batch Image Processing with Amazon Rekognition Custom Labels 2 | 3 | Amazon Rekognition is a computer vision service that makes it easy to add image and video analysis to your applications using proven, highly scalable, deep learning technology that requires no machine expertise to use. With Amazon Rekognition, you can identify objects, people, text, scenes, and activities in images and videos, as well as detect any inappropriate content. Amazon Rekognition also provides highly accurate facial analysis and facial search capabilities that you can use to detect, analyze, and compare faces for a wide variety of use cases. 4 | 5 | Amazon Rekognition Custom Labels allows you to identify the objects and scenes in images that are specific to your business needs. For example, you can find your logo in social media posts, identify your products on store shelves, classify machine parts in an assembly line, distinguish healthy and infected plants etc. Amazon Rekognition Custom Labels provides a very simple end-to-end experience where you start by labeling a dataset. Custom Labels then build a custom machine learning model for you by inspecting the data and selecting the right machine learning algorithm. Once your model is trained you can start using your model immediately for image analysis. If you expect to process images in batches (e.g. once a day or week, or at scheduled times during the day), you can provision your custom model at scheduled times. 6 | 7 | In this post, we show how you can build cost-optimal batch solution with Amazon Rekognition Custom Labels which provision your custom model at scheduled times, process all your images, and then deprovision your resources to avoid incurring extra cost. 8 | 9 | This project contains source code and supporting files for a serverless application that you can deploy with the SAM CLI or using the Cloudformation links below. It includes the following files and folders: 10 | 11 | - \functions - Code for the application's Lambda functions to check the presence of messages in a Queue, start or stop a Amazon Rekognition Custom Label Model, Analyse Images using a Custom Label Model. 12 | - template.yaml - A template that defines the application's AWS resources. 13 | 14 | This application creates a serverless Amazon Rekognition Custom Label Detection workflow which runs on a pre-defined schedule (note that the schedule is enabled by default at deployment). It demonstrates the power of Step Functions to orchestrate Lambda functions and other AWS resources to form complex and robust workflows, coupled with event-driven development using Amazon EventBridge. 15 | 16 | Solution Architecture Diagram: 17 | The following architecture diagram shows how you can design a serverless workflow to process images in batches with Amazon Rekognition Custom Labels. 18 | 19 | Architecture Diagram 20 | 21 | 1. As an image is stored in Amazon S3 bucket, it triggers a message which gets stored in an Amazon SQS queue. 22 | 2. Amazon EventBridge is configured to trigger an AWS Step Function workflow at certain frequency (1 hour by default). 23 | 3. As the workflow runs it checks the number of items in the Amazon SQS queue. If there are no items to process in the queue, workflow ends. If there are items to process in the queue, workflow starts the Amazon Rekognition Custom Labels model and enables Amazon SQS integration with a Lambda function to process those images. 24 | 4. As integration between Amazon SQS queue and Lambda is enabled, Lambda start processing images using Amazon Rekognition Custom Labels. 25 | 5. Once all the images are processed, workflow stops the Amazon Rekognition Custom Labels model and disables integration between Amazon SQS queue and Lambda function. 26 | 27 | The application uses several AWS resources, including Amazon Simple Storage Service, Amazon Simple Queue Service, Step Functions state machines, Lambda functions and an EventBridge rule trigger. These resources are defined in the `template.yaml` file in this project. You can update the template to add AWS resources through the same deployment process that updates your application code. 28 | 29 | Please note that this code is provided as a working sample. However, if you intend to use it in production, it is recommended that you implement production best practices including but not limited to error handling, message visibility settings, timeouts, storage lifecycle rules etc. 30 | 31 | ### Usage 32 | 33 | #### Prerequisites 34 | 35 | 1. To deploy the sample application, you will require an AWS account. If you don’t already have an AWS account, create one at by following the on-screen instructions. Your access to the AWS account must have IAM permissions to launch AWS CloudFormation templates that create IAM roles. 36 | 37 | 2. Please refer [here](https://docs.aws.amazon.com/rekognition/latest/customlabels-dg/gs-introduction.html) for instructions on getting started with Amazon Rekognition Custom Labels. When deploying this application you will need to provide the following two parameters for your Custom Label Project. 38 | - Amazon Rekognition Model Project ARN: The Amazon Resource Name (ARN) of the Amazon Rekognition Custom Labels project that contains the models you want to use. 39 | - Amazon Rekognition Model Project Version ARN: The Amazon Resource Name(ARN) of the model version that you want to use. 40 | 41 | 42 | #### Deployment 43 | 44 | The demo application is deployed as an [AWS CloudFormation](https://aws.amazon.com/cloudformation) template. 45 | 46 | > **Note** 47 | > You are responsible for the cost of the AWS services used while running this sample deployment. There is no additional cost for using this sample. For full details, see the following pricing pages for each AWS service you will be using in this sample. Prices are subject to change. 48 | > 49 | > - [Amazon Rekognition Pricing](https://aws.amazon.com/rekognition/pricing/) 50 | > - [Amazon Lookout for Vision Pricing](https://aws.amazon.com/lookout-for-vision/pricing/) 51 | > - [Amazon S3 Pricing](https://aws.amazon.com/s3/pricing/) 52 | > - [Amazon SQS Pricing](https://aws.amazon.com/sqs/pricing/) 53 | > - [AWS Lambda Pricing](https://aws.amazon.com/lambda/pricing/) 54 | > - [AWS Step Functions Pricing](https://aws.amazon.com/step-functions/pricing/) 55 | 56 | 1. Deploy the latest CloudFormation template by following the link below for your preferred AWS region: 57 | 58 | | Region | Launch Template | 59 | | ------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | 60 | | **US East (N. Virginia)** (us-east-1) | [![Launch the LabelDetection Stack with CloudFormation](docs/deploy-to-aws.png)](https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?stackName=LabelDetection&templateURL=https://solution-builders-us-east-1.s3.us-east-1.amazonaws.com/amazon-rekognition-custom-labels-batch-processing/latest/template.yaml) | 61 | | **US East (Ohio)** (us-east-2) | [![Launch the LabelDetection Stack with CloudFormation](docs/deploy-to-aws.png)](https://console.aws.amazon.com/cloudformation/home?region=us-east-2#/stacks/new?stackName=LabelDetection&templateURL=https://solution-builders-us-east-2.s3.us-east-2.amazonaws.com/amazon-rekognition-custom-labels-batch-processing/latest/template.yaml) | 62 | | **US West (Oregon)** (us-west-2) | [![Launch the LabelDetection Stack with CloudFormation](docs/deploy-to-aws.png)](https://console.aws.amazon.com/cloudformation/home?region=us-west-2#/stacks/new?stackName=LabelDetection&templateURL=https://solution-builders-us-west-2.s3.us-west-2.amazonaws.com/amazon-rekognition-custom-labels-batch-processing/latest/template.yaml) | 63 | | **Europe (Ireland)** (eu-west-1) | [![Launch the LabelDetection Stack with CloudFormation](docs/deploy-to-aws.png)](https://console.aws.amazon.com/cloudformation/home?region=eu-west-1#/stacks/new?stackName=LabelDetection&templateURL=https://solution-builders-eu-west-1.s3.eu-west-1.amazonaws.com/amazon-rekognition-custom-labels-batch-processing/latest/template.yaml) | 64 | | **Europe (London)** (eu-west-2) | [![Launch the LabelDetection Stack with CloudFormation](docs/deploy-to-aws.png)](https://console.aws.amazon.com/cloudformation/home?region=eu-west-2#/stacks/new?stackName=LabelDetection&templateURL=https://solution-builders-eu-west-2.s3.eu-west-2.amazonaws.com/amazon-rekognition-custom-labels-batch-processing/latest/template.yaml) | 65 | | **Asia Pacific (Sydney)** (ap-southeast-2) | [![Launch the LabelDetection Stack with CloudFormation](docs/deploy-to-aws.png)](https://console.aws.amazon.com/cloudformation/home?region=ap-southeast-2#/stacks/new?stackName=LabelDetection&templateURL=https://solution-builders-ap-southeast-2.s3.ap-southeast-2.amazonaws.com/amazon-rekognition-custom-labels-batch-processing/latest/template.yaml) | 66 | | **Europe (Frankfurt)** (eu-central-1) | [![Launch the LabelDetection Stack with CloudFormation](docs/deploy-to-aws.png)](https://console.aws.amazon.com/cloudformation/home?region=eu-central-1#/stacks/new?stackName=LabelDetection&templateURL=https://solution-builders-eu-central-1.s3.eu-central-1.amazonaws.com/amazon-rekognition-custom-labels-batch-processing/latest/template.yaml) | 67 | 68 | 2. If prompted, login using your AWS account credentials. 69 | 3. You should see a screen titled "_Create Stack_" at the "_Specify template_" step. The fields specifying the CloudFormation template are pre-populated. Click the _Next_ button at the bottom of the page. 70 | 4. On the "_Specify stack details_" screen you may customize the following parameters of the CloudFormation stack: 71 | 72 | - **Stack Name:** (Default: LabelDetection) This is the name that is used to refer to this stack in CloudFormation once deployed. 73 | - **RekognitionModelProjectARN:** The Amazon Rekognition Model Project Arn 74 | - **RekognitionModelProjectVersionARN:** The Amazon Rekognition Model Project Version Arn 75 | 76 | When completed, click _Next_ 77 | 78 | 5. [Configure stack options](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-console-add-tags.html) if desired, then click _Next_. 79 | 6. On the review you screen, you must check the boxes for: 80 | 81 | - "_I acknowledge that AWS CloudFormation might create IAM resources_" 82 | - "_I acknowledge that AWS CloudFormation might create IAM resources with custom names_" 83 | - "_I acknowledge that AWS CloudFormation might require the following capability: CAPABILITY_AUTO_EXPAND_" 84 | 85 | These are required to allow CloudFormation to create a Role to allow access to resources needed by the stack and name the resources in a dynamic way. 86 | 87 | 7. Click _Create Change Set_ 88 | 8. On the _Change Set_ screen, click _Execute_ to launch your stack. 89 | - You may need to wait for the _Execution status_ of the change set to become "_AVAILABLE_" before the "_Execute_" button becomes available. 90 | 9. Wait for the CloudFormation stack to launch. Completion is indicated when the "Stack status" is "_CREATE_COMPLETE_". 91 | - You can monitor the stack creation progress in the "Events" tab. 92 | 10. Note the _url_ displayed in the _Outputs_ tab for the stack. This is used to access the application. 93 | 94 | #### Testing the workflow 95 | 96 | To test your workflow, complete the following steps: 97 | 1. Upload sample images to the input S3 bucket that was created by the solution (Example: xxxx-sources3bucket-xxxx). 98 | 2. Go to AWS Step Function console and select the state machine created by the solution (Example: CustomCVStateMachine-xxxx). You will see an execution triggered by the EventBridge at every hour. 99 | 3. To test the solution, you can also manually start the workflow by clicking on the “Start execution” button. 100 | 4. As images are processed you can go to the output S3 bucket (Example: xxxx-finals3bucket-xxxx) to see the JSON output for each image. The Final S3 bucket holds the images that have been processed along with the inferenced custom label json. As the images get processed, they will be deleted from the source bucket. 101 | 102 | 103 | ### Removing the application 104 | 105 | To remove the application open the AWS CloudFormation Console, click on the name of the project, right-click and select "_Delete Stack_". Your stack will take some time to be deleted. You can track its progress in the "Events" tab. When it is done, the status will change from "_DELETE_IN_PROGRESS_" to "_DELETE_COMPLETE_". It will then disappear from the list. 106 | 107 | **Note:** Please note that the provided configuration will ensure that the Amazon S3 buckets and their contents are retained when removing the application via the AWS Cloudformation console. This is to ensure that no data is accidently lost while removing the application. The buckets can be deleted from the S3 console. 108 | 109 | 110 | ## License 111 | 112 | This library is licensed under the MIT-0 License. See the LICENSE file. 113 | --------------------------------------------------------------------------------