├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── LICENSE ├── README.md ├── deploy.env ├── formation_cf_script ├── deploy.sh ├── form-greengrass_modules.yml ├── install_greengrass.sh └── lambda_deepstream_app │ └── run_deepstream.py └── provisioning_cf_script ├── deploy.sh ├── device_policy.json ├── greengrass_creation_cust_resource_lambda ├── cfnresponse.py └── greengrass_creation_cust_resource_lambda.py └── provision-greengrass.yml /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | ## Code of Conduct 2 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 3 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 4 | opensource-codeofconduct@amazon.com with any additional questions or comments. 5 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing Guidelines 2 | 3 | Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional 4 | documentation, we greatly value feedback and contributions from our community. 5 | 6 | Please read through this document before submitting any issues or pull requests to ensure we have all the necessary 7 | information to effectively respond to your bug report or contribution. 8 | 9 | 10 | ## Reporting Bugs/Feature Requests 11 | 12 | We welcome you to use the GitHub issue tracker to report bugs or suggest features. 13 | 14 | When filing an issue, please check existing open, or recently closed, issues to make sure somebody else hasn't already 15 | reported the issue. Please try to include as much information as you can. Details like these are incredibly useful: 16 | 17 | * A reproducible test case or series of steps 18 | * The version of our code being used 19 | * Any modifications you've made relevant to the bug 20 | * Anything unusual about your environment or deployment 21 | 22 | 23 | ## Contributing via Pull Requests 24 | Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that: 25 | 26 | 1. You are working against the latest source on the *master* branch. 27 | 2. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already. 28 | 3. You open an issue to discuss any significant work - we would hate for your time to be wasted. 29 | 30 | To send us a pull request, please: 31 | 32 | 1. Fork the repository. 33 | 2. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change. 34 | 3. Ensure local tests pass. 35 | 4. Commit to your fork using clear commit messages. 36 | 5. Send us a pull request, answering any default questions in the pull request interface. 37 | 6. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation. 38 | 39 | GitHub provides additional document on [forking a repository](https://help.github.com/articles/fork-a-repo/) and 40 | [creating a pull request](https://help.github.com/articles/creating-a-pull-request/). 41 | 42 | 43 | ## Finding contributions to work on 44 | Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels (enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any 'help wanted' issues is a great place to start. 45 | 46 | 47 | ## Code of Conduct 48 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 49 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 50 | opensource-codeofconduct@amazon.com with any additional questions or comments. 51 | 52 | 53 | ## Security issue notifications 54 | If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github issue. 55 | 56 | 57 | ## Licensing 58 | 59 | See the [LICENSE](LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution. 60 | 61 | We may ask you to sign a [Contributor License Agreement (CLA)](http://en.wikipedia.org/wiki/Contributor_License_Agreement) for larger changes. 62 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining a copy of 4 | this software and associated documentation files (the "Software"), to deal in 5 | the Software without restriction, including without limitation the rights to 6 | use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of 7 | the Software, and to permit persons to whom the Software is furnished to do so. 8 | 9 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 10 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS 11 | FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 12 | COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER 13 | IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN 14 | CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 15 | 16 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ## AWS IoT Greengrass Deploying NVIDIA DeepStream on Edge 2 | This is the cloudformation package to demonstrate Machine Learning model deployment at scale with AWS Greengrass on NVIDIA Jetson Devices. This demonstrates consists four parts: 3 | 1. Create IoT Things and attach certificates for Greengrass core thing and DeepStream app thing, which will later be attached to this Greengrass group 4 | 2. Simulate a DeepStream package to be deployed by Greengrass 5 | 3. Create a Greengrass core with modules attached on AWS Cloud 6 | 4. Deploy Greengrass on edge device and run DeepStream App 7 | Please follow the steps below to go through the Cloudformation Example: 8 | 9 | ## Pre-requisites 10 | 1. Be an AWS account admin user 11 | 2. Jetson device (this should work for all of the devices in Jetson family) and its sudo access 12 | 3. AWS CLI version2 and AWS SAM CLI installed on your computer 13 | - To install AWS CLI: https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html 14 | - To install AWS SAM CLI: https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install.html 15 | 16 | 17 | ## Part 1: Create things on AWS IoT (Your PC) 18 | ### Step 1: Clone repo and configure environment 19 | If you have not done so, configure your AWS CLI with a profile linked to your account. 20 | ``` 21 | aws configure --profile ANY_NAME_YOU_LIKE 22 | ``` 23 | 24 | Next, please clone this repo onto any PATH onto your machine. 25 | ``` 26 | git clone https://github.com/rvanderwerf/aws-iot-greengrass-deploy-nvidia-deepstream-on-edge.git 27 | cd aws-iot-greengrass-deploy-nvidia-deepstream-on-edge 28 | ``` 29 | 30 | Once you successfully cloned this repo, please edit deploy.env file to replace to the profile name you just created above. You can also edit AWS_REGION and ENVIRONMENT. Run the following command after you have modified deploy.env with your aws profile name: 31 | ``` 32 | source deploy.env 33 | ``` 34 | 35 | Then, we are going to use the contents in provisioning_cf_script folder. 36 | ``` 37 | cd provisioning_cf_script 38 | ``` 39 | 40 | ### Step 2: Understand deploy.sh script 41 | Examine the deploy.sh file. This is the starting point of the cloud formation. 42 | This bash script first creates a S3 bucket to store all of the relevant assets created in this repo. Then, it 43 | calls provision-greengrass-cfn.yml CloudFormation script with AWS SAM CLI toolset. 44 | 45 | After CloudFormation runs successfully, we are going to generate several commands with S3 pre-signed links the provisioned certificates to be downloaded to Jetson devices (which only last for one day) into install_greengrass.sh file, which we are going to use later. The pre-signed links last for a week. If those links expire, you will have to manually run the last 5 lines in deploy.sh again and generate new links. 46 | 47 | ### Step 3: Understand provision-greengrass.yml script 48 | This is a CloudFormation script run by deploy.sh. This script internally calls a one-time lambda function to create two things: 49 | 1. Greengrass Core 50 | 2. DeepStream App 51 | And provision them with IoT certificates. Then download certificates to the designated S3 bucket with name similar to "greengrass-deepstream-$AWS_ACCOUNT_ID-$ENVIRONMENT-assets". 52 | 53 | ### Step 4: 54 | Run the provisioning script: 55 | ``` 56 | . deploy.sh 57 | ``` 58 | After this, an output.txt file should be generated in this folder. Just leave it there. 59 | You can also navigate to AWS console, and see things just created in Manage -> Things. 60 | 61 | 62 | ## Part 2: Simulate a deployment package (Still on your PC) 63 | ### Step 5: Upload trained ML model to an S3 bucket 64 | We are going to simulate a package to be deployed from AWS cloud to Jetson device by using example applications and models from DeepStream SDK. We can first download DeepStream SDK for Jetson onto this machine from https://developer.nvidia.com/deepstream-download, and download the .tar version. After successful download, you can untar the package: 65 | 66 | ``` 67 | mkdir $GG_DEPLOYMENT_HOME/deepstream-source 68 | tar -xpvf /deepstream_sdk__jetson.tbz2 -C $GG_DEPLOYMENT_HOME/deepstream-source 69 | ``` 70 | 71 | Then, we are going to upload this model into the S3 bucket we created in Part 1. 72 | ``` 73 | cd $GG_DEPLOYMENT_HOME/deepstream-source/opt/nvidia/deepstream/deepstream-5.0/samples/models/Primary_Detector 74 | zip model.zip * 75 | aws s3 cp ./model.zip s3://$S3_BUCKET/model/model.zip --profile $AWS_PROFILE 76 | ``` 77 | 78 | ### Step 5.5: Install DeepStream onto Jetson (If not installed) 79 | If you do not have DeepStream installed, such as running a Jetson Nano SD JetPack 4.4 Image, copy the DeepStream SDK to your device: 80 | ``` 81 | scp /deepstream_sdk__jetson.tbz2 @:/home/ 82 | 83 | ``` 84 | Now ssh do your Jetson device and install the SDK 85 | ``` 86 | cd / 87 | sudo tar xpvf /deepstream_sdk__jetson.tbz2 88 | ``` 89 | ### Step 6: Prepare DeepStream Application to be deployed 90 | Once we upload the model, we also need to prepare a DeepStream package to be deployed with Greengrass. We are going to use DeepStream sample app on your Jetson device for this demonstration: 91 | ``` 92 | scp @:/sources/apps/sample_apps/deepstream-app/* $GG_DEPLOYMENT_HOME/formation_cf_script/lambda_deepstream_app/ 93 | ``` 94 | 95 | ### Step 7: Prepare corresponding configuration files for DeepStream Application 96 | Then we need to copy the configuration files to this sample application, and modify it with PLACEHOLDERS so we can locate the ML model correctly when the DeepStream app starts to run in Greengrass: 97 | ``` 98 | cp $GG_DEPLOYMENT_HOME/deepstream-source/opt/nvidia/deepstream/deepstream-5.0/samples/configs/deepstream-app/config_infer_primary_nano.txt $GG_DEPLOYMENT_HOME/formation_cf_script/lambda_deepstream_app/ 99 | cp $GG_DEPLOYMENT_HOME/deepstream-source/opt/nvidia/deepstream/deepstream-5.0/samples/configs/deepstream-app/source1_usb_dec_infer_resnet_int8.txt $GG_DEPLOYMENT_HOME/formation_cf_script/lambda_deepstream_app/ 100 | cd $GG_DEPLOYMENT_HOME/formation_cf_script/lambda_deepstream_app 101 | ### On MacOS/BSD 102 | sed -i '.tmp' -e 's|model-engine-file|#model-engine-file|g' config_infer_primary_nano.txt 103 | sed -i '.tmp' -e 's|../../models/Primary_Detector_Nano|/resnet_10_model|g' config_infer_primary_nano.txt 104 | sed -i '.tmp' -e 's|model-engine-file|#model-engine-file|g' source1_usb_dec_infer_resnet_int8.txt 105 | ### On Ubuntu/Debian 106 | sed -i -e 's|model-engine-file|#model-engine-file|g' config_infer_primary_nano.txt 107 | sed -i -e 's|../../models/Primary_Detector_Nano|/resnet_10_model|g' config_infer_primary_nano.txt 108 | sed -i -e 's|model-engine-file|#model-engine-file|g' source1_usb_dec_infer_resnet_int8.txt 109 | 110 | mv config_infer_primary_nano.txt config_infer_primary.txt 111 | ## Only needed on MacOS/BSD 112 | rm *.tmp 113 | ``` 114 | 115 | ### Step 8: Prepare files/streams to be run by this DeepStream Application 116 | Open the "source1_usb_dec_infer_resnet_int8.txt" you just created in any text editor you prefer. We need to modify the source and sink to the known source. In this demo, we are going to use a RTSP camera as the source. You may also choose to use other sources. So we modify the source to be: 117 | ``` 118 | [source0] 119 | enable=1 120 | #Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP 121 | type=4 122 | uri= 123 | num-sources=1 124 | #drop-frame-interval=2 125 | gpu-id=0 126 | # (0): memtype_device - Memory type Device 127 | # (1): memtype_pinned - Memory type Host Pinned 128 | # (2): memtype_unified - Memory type Unified 129 | cudadec-memtype=0 130 | ``` 131 | 132 | Then we disable any sink except the RTSP stream sink: 133 | ``` 134 | [sink2] 135 | enable=1 136 | #Type - 1=FakeSink 2=EglSink 3=File 4=RTSPStreaming 5=Overlay 137 | type=4 138 | #1=h264 2=h265 139 | codec=1 140 | sync=0 141 | bitrate=4000000 142 | # set below properties in case of RTSPStreaming 143 | rtsp-port=8554 144 | udp-port=5400 145 | ``` 146 | 147 | If you have other source preference, feel free to use those as well. As long as you verify that your Jetson device have access to these sources or sinks. 148 | 149 | 150 | ### Part 2 Troubleshooting 151 | If you are experiencing error, it is likely that you have switched your terminal and lost environmental variables. In this case, please run locate output.txt in this folder, and copy the last line, which looks similar to: greengrass-deepstream-\-test-assets. And re-export these environmental variables. 152 | 153 | ``` 154 | source ./deploy.env 155 | ``` 156 | And then pick up from where you experienced error in the previous commands and try again. 157 | 158 | ## Part 3: Create Greengrass Group on AWS Cloud 159 | ### Step 9: Understand YAML script for cloudformation 160 | The major function for this script is to create a greengrass group. And to this greengrass group, we attach 6 modules: 161 | - Greengrass Core: a thing with certificates that helps this Greengrass group to connect to cloud MQTT server securely 162 | - Local Lambda function: we use this lambda function to run Deepstream within as a subprocess. We make this function long-live, and run Greengrass container mode. 163 | - Greengrass Device: this is a separated thing with a different set of certs other than Greengrass Core. This set of certs is directly granted to Deepsteam app, so that Deepstream app can make secure connection with Greengrass Core, which is a local MQTT server. Please refer to "Extra Note" section for more information on this. 164 | - Greengrass Machine Learning Resource: in this case, this resource is the resnet-10 model we uploaded in S3 manually. It could also be coming directly from Sagemaker, or uploaded to S3 from other sources. 165 | - Other Greengrass Resource: we want to make sure the Greengrass container has all the hardware access within /dev/ repo on Jetson device to run this DeepStream app. The required resource might be different for applications, we need to make sure all of them are attached to the container. Otherwise, it would lead to inference failures. 166 | - Greengrass Subscriptions: we configure command messages to be published by cloud and subscribed by Deepstream app. And we configure performance update messages to be published from Deepstream app and subsribed by cloud. We can theoretically configure any publisher or subscribers based on our need, as long as they have the right credentials to communicate with MQTT server. 167 | 168 | This link posts more detailed information on this set-up: https://aws.amazon.com/blogs/iot/automating-aws-iot-greengrass-setup-with-aws-cloudformation/ 169 | 170 | ### Step 10: Install Greengrass Python SDK in local lambda function 171 | In order to use Greengrass lambda function successfully, you also need to git clone the Greengrass SDK in your local Lambda function. 172 | ``` 173 | cd $GG_DEPLOYMENT_HOME 174 | git clone https://github.com/aws/aws-greengrass-core-sdk-python.git 175 | cp -r aws-greengrass-core-sdk-python/greengrasssdk $GG_DEPLOYMENT_HOME/formation_cf_script/lambda_deepstream_app 176 | ``` 177 | 178 | ### Step 11: Form Greengrass group with Cloudformation template 179 | Run the scripts: 180 | ``` 181 | cd $GG_DEPLOYMENT_HOME/formation_cf_script 182 | . deploy.sh 183 | ``` 184 | After script runs successfully, you should be able to observe on AWS console a Greengrass group ready to be deployed. And you could click from AWS console to do a deployment on the right upper corner, but don't click deploy yet until Greengrass is set up. 185 | 186 | ### Part 3 Troubleshooting 187 | If the deployment status shows and yellow label and says "pending", that means the Greengrass group has been successfully created. Otherwise, please click on the deployment failure entry and read the error log. 188 | 189 | 190 | ## Part 4: Install and Start Greengrass on local Jetson device 191 | ### Step 12: Set up device 192 | Take out your Jetson device, and install Greengrass by running on your Jetson device install_greengrass.sh script we have prepared for you. In order to copy this script to your Jetson device first, you can either use scp to copy install_greengrass.sh to your Jetson device or upload it somewhere to be downloaded from your Jetson device. And then run 193 | ``` 194 | scp $GG_DEPLOYMENT_HOME/formation_cf_script/install_greengrass.sh @:~/Downloads 195 | ssh @ 196 | cd ~/Downloads 197 | sudo sh install_greengrass.sh 198 | ``` 199 | This script will automatically download and install Greengrass from the following page: 200 | https://docs.aws.amazon.com/greengrass/latest/developerguide/what-is-gg.html#gg-core-download-tab 201 | It will also automatically retrieve the certificates and Greengrass configurations you have created in Part 1 from S3 bucket, and download them on your Jetson device. 202 | 203 | 204 | You should now have all the resources needed to run DeepStream and Greengrass, so we can start Greengrass. You can now run the following script, and observe the inference result on the sink we just configured. 205 | ``` 206 | cd /greengrass/ggc/core 207 | sudo ./greengrassd start 208 | ``` 209 | ### Part 4 troubleshooting 210 | 211 | If your bucket is not public, you will have noticed some errors during the GG setup. If that's the case, from your PC, copy the bucket contents to your local machine: 212 | 213 | cd $GG_DEPLOYMENT_HOME 214 | mkdir s3bucket 215 | aws s3 cp --recursive s3://greengrass-deepstream-XXXX-test-assets . 216 | Now copy the items in the install_greengrass.sh script from your local folder to the correct places on your Jetson device. 217 | 218 | If your inference results are not outputted correctly, you can read Greengrass runtime logs to figure out why. Please navigate, on your Jetson device, to the following directory: 219 | ``` 220 | sudo su 221 | cd /greengrass/ggc/var/log/ 222 | ``` 223 | For Greengrass system logs, you can read log fils in the system folder. For example, "/greengrass/ggc/var/log/system/GGConnManager.log" contains all the messsages Greengrass had received or sent as MQTT broker. "/greengrass/ggc/var/log/system/runtime.log" shows generic Greengrass operation logs. 224 | 225 | For Lambda function specific logs, you can locate them in 226 | ``` 227 | /greengrass/ggc/var/log/user/us-west-2/xxxxxxxxxx 228 | ``` 229 | where xxxxxxxxxx is you AWS account number. All of the DeepStream operations logs should also be in this folder. 230 | 231 | ### Extra Note: Run DeepStream IoT Test Applications (test 4 or test 5) 232 | In this application, we also created a separate set of certificates and a corresponding thing for DeepStream app to authenticate with both AWS IoT Core and this Greengrass group. In order to use AWS specific msg-broker in DeepStream, you need to follow this GitHub (but skip provisioning process, because certificates have already been created for you): 233 | https://github.com/awslabs/aws-iot-core-integration-with-nvidia-deepstream 234 | 235 | Please use the links in output.txt in this folder to download the certificates to the right folders. 236 | 237 | 238 | ## License 239 | 240 | This library is licensed under the MIT-0 License. See the LICENSE file. 241 | -------------------------------------------------------------------------------- /deploy.env: -------------------------------------------------------------------------------- 1 | ENVIRONMENT="test" 2 | AWS_PROFILE="" 3 | GG_THING_GROUP_NAME="greengrass-core" 4 | DS_THING_GROUP_NAME="greengrass-deepstream-app" 5 | ML_RESOURCE_NAME="model.zip" 6 | GG_DEPLOYMENT_HOME=$(pwd) 7 | -------------------------------------------------------------------------------- /formation_cf_script/deploy.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # ################################################## 4 | # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 5 | 6 | # Permission is hereby granted, free of charge, to any person obtaining a copy of 7 | # this software and associated documentation files (the "Software"), to deal in 8 | # the Software without restriction, including without limitation the rights to 9 | # use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of 10 | # the Software, and to permit persons to whom the Software is furnished to do so. 11 | # ################################################## 12 | 13 | # define environment label [test/dev/prod...] 14 | if [[ -z "$ENVIRONMENT" ]]; then 15 | ENVIRONMENT=test 16 | echo "Environment variable not specified. Using default: test" 17 | fi 18 | echo "Using Environment $ENVIRONMENT" 19 | 20 | # define aws profile used as --profle YOUR_ACCOUNT 21 | if [ -z "$AWS_PROFILE" ]; then 22 | echo "Environment variable not specified. Using aws configure without a profile." 23 | fi 24 | echo "Using Region $AWS_PROFILE" 25 | AWS_ARGS="--profile $AWS_PROFILE" 26 | 27 | # get the region account is using 28 | AWS_REGION=$(aws configure get region ${AWS_ARGS}) 29 | echo "Using Region $AWS_REGION" 30 | 31 | # get the account ID to name the S3 asset bucket we are about to create 32 | AWS_ACCOUNT_ID=$(aws sts get-caller-identity --output text --query 'Account' $AWS_ARGS) 33 | echo account id: $AWS_ACCOUNT_ID 34 | 35 | #create S3 bucket 36 | S3_BUCKET="greengrass-deepstream-$AWS_ACCOUNT_ID-${ENVIRONMENT}-assets" 37 | echo "Bucket name : $S3_BUCKET"; 38 | #check if bucket exists 39 | bucketstatus=$(aws s3api head-bucket --bucket "${S3_BUCKET}" $AWS_ARGS 2>&1) 40 | if [ -z "$bucketstatus" ] 41 | then 42 | echo "Bucket exists: $S3_BUCKET, using existing bucket." 43 | else 44 | echo "Bucket does not exist, creating new bucket."; 45 | aws s3api create-bucket --bucket "$S3_BUCKET" --create-bucket-configuration LocationConstraint="$AWS_REGION" $AWS_ARGS; 46 | fi 47 | 48 | # Decide whether to create new thing for this deployment 49 | GG_THING_EXISTS=$(aws iot describe-thing --thing-name $GG_THING_GROUP_NAME $AWS_ARGS --query 'thingArn') 50 | DS_THING_EXISTS=$(aws iot describe-thing --thing-name $DS_THING_GROUP_NAME $AWS_ARGS --query 'thingArn') 51 | 52 | #check if thing exists, and create if they do not currently exist 53 | if [ -z "$GG_THING_EXISTS" ] || [ -z "$DS_THING_EXISTS" ] 54 | then 55 | echo "ERROR: The thing name to associate this Greengrass Core we are about to create does not exist." 56 | echo "Please run provision script first." 57 | else 58 | # Deploy or update 59 | GGDP_STACK_NAME="greengrass-deepstream-$AWS_ACCOUNT_ID-$AWS_REGION-${ENVIRONMENT}-stack" 60 | echo "Deploying Greengrass Stack env: $ENVIRONMENT, s3: $S3_BUCKET, greengrass-deepstream-stack: $GGDP_STACK_NAME with $AWS_ARGS" 61 | 62 | sam package \ 63 | --template-file ./form-greengrass_modules.yml \ 64 | --s3-bucket $S3_BUCKET \ 65 | --output-template-file ./form-greengrass_modules-packaged.yaml $AWS_ARGS 66 | sam deploy \ 67 | --template-file ./form-greengrass_modules-packaged.yaml \ 68 | --stack-name $GGDP_STACK_NAME \ 69 | --parameter-overrides \ 70 | CoreName=$GG_THING_GROUP_NAME \ 71 | S3BucketName=$S3_BUCKET \ 72 | MLResourceName=$ML_RESOURCE_NAME \ 73 | --capabilities CAPABILITY_IAM CAPABILITY_AUTO_EXPAND $AWS_ARGS 74 | fi 75 | -------------------------------------------------------------------------------- /formation_cf_script/form-greengrass_modules.yml: -------------------------------------------------------------------------------- 1 | # ################################################## 2 | # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | 4 | # Permission is hereby granted, free of charge, to any person obtaining a copy of 5 | # this software and associated documentation files (the "Software"), to deal in 6 | # the Software without restriction, including without limitation the rights to 7 | # use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of 8 | # the Software, and to permit persons to whom the Software is furnished to do so. 9 | # ################################################## 10 | 11 | AWSTemplateFormatVersion: "2010-09-09" 12 | Description: "Greengrass Deepstream Integration Template" 13 | Transform: AWS::Serverless-2016-10-31 14 | 15 | Description: Create Greengrass resources for a AWS Secure Transport tariler 16 | 17 | Parameters: 18 | CoreName: 19 | Description: Greengrass Core name to be created. A "Thing" with be created with _core appended to the name 20 | Type: String 21 | Default: DeepstreamGreengrassGroup 22 | S3BucketName: 23 | Description: S3 bucket name to put generated certificates 24 | Type: String 25 | MLResourceName: 26 | Description: ML resource zip file name 27 | Type: String 28 | 29 | Resources: 30 | GreengrassGroup: 31 | Type: AWS::Greengrass::Group 32 | DependsOn: 33 | - GreengrassResourceRole 34 | Properties: 35 | Name: !Ref CoreName 36 | RoleArn: !GetAtt GreengrassResourceRole.Arn 37 | InitialVersion: 38 | CoreDefinitionVersionArn: !Ref GreengrassCoreDefinitionVersion 39 | FunctionDefinitionVersionArn: !GetAtt FunctionDefinition.LatestVersionArn 40 | SubscriptionDefinitionVersionArn: !GetAtt SubscriptionDefinition.LatestVersionArn 41 | LoggerDefinitionVersionArn: !GetAtt LoggerDefinition.LatestVersionArn 42 | DeviceDefinitionVersionArn: !GetAtt DeviceDefinition.LatestVersionArn 43 | ResourceDefinitionVersionArn: !GetAtt ResourceDefinition.LatestVersionArn 44 | GreengrassCoreDefinition: 45 | Type: AWS::Greengrass::CoreDefinition 46 | Properties: 47 | # GG Core = CoreName + "_core" as the "thingName" 48 | Name: !Ref CoreName 49 | GreengrassCoreDefinitionVersion: 50 | Type: AWS::Greengrass::CoreDefinitionVersion 51 | Properties: 52 | CoreDefinitionId: !Ref GreengrassCoreDefinition 53 | Cores: 54 | - Id: !Ref CoreName 55 | ThingArn: !Join 56 | - ":" 57 | - - "arn:aws:iot" 58 | - !Ref AWS::Region 59 | - !Ref AWS::AccountId 60 | - !Join 61 | - "/" 62 | - - "thing" 63 | - !Ref CoreName 64 | CertificateArn: !Join 65 | - ":" 66 | - - "arn:aws:iot" 67 | - !Ref AWS::Region 68 | - !Ref AWS::AccountId 69 | - !Join 70 | - "/" 71 | - - "cert" 72 | - Fn::ImportValue: "GGCoreCertificateId" 73 | SyncShadow: "true" 74 | DeepstreamAppLambdaRole: 75 | Type: AWS::IAM::Role 76 | Properties: 77 | AssumeRolePolicyDocument: 78 | Version: '2012-10-17' 79 | Statement: 80 | - Effect: Allow 81 | Principal: 82 | Service: 83 | - lambda.amazonaws.com 84 | Action: 85 | - sts:AssumeRole 86 | Path: "/" 87 | Policies: 88 | - PolicyName: AggregationLambdaLogging 89 | PolicyDocument: 90 | Version: '2012-10-17' 91 | Statement: 92 | - Effect: Allow 93 | Action: 94 | - logs:CreateLogGroup 95 | - logs:CreateLogStream 96 | - logs:PutDestination 97 | - logs:PutLogEvents 98 | Resource: 99 | !Join 100 | - '' 101 | - - 'arn:aws:logs:' 102 | - !Ref AWS::Region 103 | - ':' 104 | - !Ref AWS::AccountId 105 | - ':log-group:*' 106 | 107 | DeepstreamAppLambda: 108 | Type: 'AWS::Serverless::Function' # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction 109 | Properties: 110 | CodeUri: lambda_deepstream_app/ 111 | AutoPublishAlias: test 112 | Handler: run_deepstream.function_handler 113 | Runtime: python2.7 114 | Description: Lambda that runs Deepstream App within as a subprocess 115 | MemorySize: 2048 116 | Role: !GetAtt DeepstreamAppLambdaRole.Arn 117 | Timeout: 900 118 | 119 | 120 | FunctionDefinition: 121 | Type: AWS::Greengrass::FunctionDefinition 122 | Properties: 123 | Name: FunctionDefinition 124 | InitialVersion: 125 | DefaultConfig: 126 | Execution: 127 | IsolationMode: GreengrassContainer 128 | Functions: 129 | - Id: !Join ["_", [!Ref CoreName, "DeepstreamAppLambda"] ] 130 | FunctionArn: !Join [":", [!GetAtt DeepstreamAppLambda.Arn, "test"] ] 131 | FunctionConfiguration: 132 | Pinned: 'true' 133 | Timeout: '300' 134 | MemorySize: '10000000' 135 | EncodingType: 'json' 136 | Environment: 137 | AccessSysfs: 'true' 138 | Execution: 139 | IsolationMode: GreengrassContainer 140 | RunAs: 141 | Uid: '0' 142 | Gid: '0' 143 | ResourceAccessPolicies: 144 | - ResourceId: MLModel 145 | Permission: rw 146 | - ResourceId: ResourceId2 147 | Permission: rw 148 | - ResourceId: ResourceId3 149 | Permission: rw 150 | - ResourceId: ResourceId4 151 | Permission: rw 152 | - ResourceId: ResourceId5 153 | Permission: rw 154 | - ResourceId: ResourceId6 155 | Permission: rw 156 | - ResourceId: ResourceId7 157 | Permission: rw 158 | - ResourceId: ResourceId8 159 | Permission: rw 160 | - ResourceId: ResourceId9 161 | Permission: rw 162 | - ResourceId: ResourceId10 163 | Permission: rw 164 | - ResourceId: ResourceId11 165 | Permission: rw 166 | - ResourceId: ResourceId12 167 | Permission: rw 168 | - ResourceId: ResourceId13 169 | Permission: rw 170 | 171 | DeviceDefinition: 172 | Type: AWS::Greengrass::DeviceDefinition 173 | Properties: 174 | InitialVersion: 175 | Devices: 176 | - CertificateArn: 177 | Fn::ImportValue: "DeepstreamAppThingCertificateArn" 178 | SyncShadow: "true" 179 | Id: !Join ["_", [!Ref CoreName, "DeepstreamApp"] ] 180 | ThingArn: 181 | Fn::ImportValue: "DeepstreamAppThingThingArn" 182 | Name: deepstream-app-device-definition 183 | 184 | 185 | ResourceDefinition: 186 | Type: 'AWS::Greengrass::ResourceDefinition' 187 | Properties: 188 | Name: deepstream-model-resource-definition 189 | InitialVersion: 190 | Resources: 191 | - Name: "resnet-10-model" 192 | Id: MLModel 193 | ResourceDataContainer: 194 | S3MachineLearningModelResourceData: 195 | DestinationPath: "/resnet_10_model/" 196 | S3Uri: !Sub 's3://${S3BucketName}/model/${MLResourceName}' 197 | - Id: ResourceId2 198 | Name: nvhost-ctrl 199 | ResourceDataContainer: 200 | LocalDeviceResourceData: 201 | SourcePath: /dev/nvhost-ctrl 202 | - Id: ResourceId3 203 | Name: nvhost-gpu 204 | ResourceDataContainer: 205 | LocalDeviceResourceData: 206 | SourcePath: /dev/nvhost-gpu 207 | - Id: ResourceId4 208 | Name: nvhost-ctrl-gpu 209 | ResourceDataContainer: 210 | LocalDeviceResourceData: 211 | SourcePath: /dev/nvhost-ctrl-gpu 212 | - Id: ResourceId5 213 | Name: nvhost-dbg-gpu 214 | ResourceDataContainer: 215 | LocalDeviceResourceData: 216 | SourcePath: /dev/nvhost-dbg-gpu 217 | - Id: ResourceId6 218 | Name: nvhost-prof-gpu 219 | ResourceDataContainer: 220 | LocalDeviceResourceData: 221 | SourcePath: /dev/nvhost-prof-gpu 222 | - Id: ResourceId7 223 | Name: nvmap 224 | ResourceDataContainer: 225 | LocalDeviceResourceData: 226 | SourcePath: /dev/nvmap 227 | - Id: ResourceId8 228 | Name: nvhost-vic 229 | ResourceDataContainer: 230 | LocalDeviceResourceData: 231 | SourcePath: /dev/nvhost-vic 232 | - Id: ResourceId9 233 | Name: nvhost-as-gpu 234 | ResourceDataContainer: 235 | LocalDeviceResourceData: 236 | SourcePath: /dev/nvhost-as-gpu 237 | - Id: ResourceId10 238 | Name: mem 239 | ResourceDataContainer: 240 | LocalDeviceResourceData: 241 | SourcePath: /dev/mem 242 | - Id: ResourceId11 243 | Name: tegra_dc_ctrl 244 | ResourceDataContainer: 245 | LocalDeviceResourceData: 246 | SourcePath: /dev/tegra_dc_ctrl 247 | - Id: ResourceId12 248 | Name: nvhost-msenc 249 | ResourceDataContainer: 250 | LocalDeviceResourceData: 251 | SourcePath: /dev/nvhost-msenc 252 | - Id: ResourceId13 253 | Name: nvhost-nvdec 254 | ResourceDataContainer: 255 | LocalDeviceResourceData: 256 | SourcePath: /dev/nvhost-nvdec 257 | 258 | LoggerDefinition: 259 | Type: 'AWS::Greengrass::LoggerDefinition' 260 | Properties: 261 | Name: LoggerDefinition 262 | InitialVersion: 263 | Loggers: 264 | - Component: GreengrassSystem 265 | Id: !Join 266 | - "-" 267 | - - !Ref CoreName 268 | - "1" 269 | Level: DEBUG 270 | Space: 25600 271 | Type: FileSystem 272 | - Component: GreengrassSystem 273 | Id: !Join 274 | - "-" 275 | - - !Ref CoreName 276 | - "2" 277 | Level: DEBUG 278 | Type: AWSCloudWatch 279 | - Component: Lambda 280 | Id: !Join 281 | - "-" 282 | - - !Ref CoreName 283 | - "3" 284 | Level: DEBUG 285 | Space: 25600 286 | Type: FileSystem 287 | - Component: Lambda 288 | Id: !Join 289 | - "-" 290 | - - !Ref CoreName 291 | - "4" 292 | Level: DEBUG 293 | Type: AWSCloudWatch 294 | 295 | SubscriptionDefinition: 296 | Type: 'AWS::Greengrass::SubscriptionDefinition' 297 | Properties: 298 | Name: SubscriptionDefinition 299 | InitialVersion: 300 | Subscriptions: 301 | - Id: DeepstreamNotifyGreengrass 302 | Source: 'cloud' 303 | Subject: "#" 304 | Target: 305 | Fn::ImportValue: "DeepstreamAppThingThingArn" 306 | - Id: DeepstreamNotifyCloud 307 | Source: 308 | Fn::ImportValue: "DeepstreamAppThingThingArn" 309 | Subject: "#" 310 | Target: 'cloud' 311 | - Id: DeepstreamLambdaNotifyCloud 312 | Source: 313 | !Join [":", [!GetAtt DeepstreamAppLambda.Arn, "test"] ] 314 | Subject: "#" 315 | Target: 'cloud' 316 | GreengrassResourceRole: 317 | # Role for deployed Lambda functions to a Greengrass core to call other AWS services directly 318 | Type: AWS::IAM::Role 319 | Properties: 320 | AssumeRolePolicyDocument: 321 | Version: 2012-10-17 322 | Statement: 323 | - Effect: Allow 324 | Principal: 325 | Service: greengrass.amazonaws.com 326 | Action: sts:AssumeRole 327 | Policies: 328 | - PolicyName: root 329 | PolicyDocument: 330 | Version: '2012-10-17' 331 | Statement: 332 | - Effect: Allow 333 | Action: 334 | - logs:CreateLogGroup 335 | - logs:CreateLogStream 336 | - logs:PutLogEvents 337 | Resource: arn:aws:logs:*:*:* 338 | - Effect: Allow 339 | Action: 340 | - iot:* 341 | Resource: "*" 342 | -------------------------------------------------------------------------------- /formation_cf_script/install_greengrass.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Greengrass installation script 4 | # General Updates 5 | # sudo apt-get -y update 6 | # sudo apt-get -y upgrade 7 | 8 | # greengrass dependencies 9 | sudo adduser --system ggc_user 10 | sudo addgroup --system ggc_group || sudo groupadd --system ggc_group 11 | 12 | if test -f "/boot/cmdline.txt"; then 13 | sudo bash -c 'echo " cgroup_enable=memory cgroup_memory=1" >> /boot/cmdline.txt' 14 | sudo sed -i '$ s/$/ cgroup_enable=memory cgroup_memory=1/' /boot/cmdline.txt 15 | fi 16 | 17 | 18 | # Install greengrass 19 | ggVersion="https://d1onfpft10uf5o.cloudfront.net/greengrass-core/downloads/1.10.0/greengrass-linux-armv7l-1.10.0.tar.gz" 20 | myUser="pi" 21 | if hostnamectl | grep "arm64"; then 22 | ggVersion="https://d1onfpft10uf5o.cloudfront.net/greengrass-core/downloads/1.10.0/greengrass-linux-aarch64-1.10.0.tar.gz" 23 | myUser="jetson" 24 | fi 25 | 26 | sudo wget -O /greengrass.tar.gz $ggVersion 27 | sudo tar -C / -xvf /greengrass.tar.gz 28 | sudo rm /greengrass.tar.gz 29 | sudo wget -O /greengrass/certs/root.ca.pem https://www.amazontrust.com/repository/AmazonRootCA1.pem 30 | -------------------------------------------------------------------------------- /formation_cf_script/lambda_deepstream_app/run_deepstream.py: -------------------------------------------------------------------------------- 1 | # ################################################## 2 | # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | 4 | # Permission is hereby granted, free of charge, to any person obtaining a copy of 5 | # this software and associated documentation files (the "Software"), to deal in 6 | # the Software without restriction, including without limitation the rights to 7 | # use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of 8 | # the Software, and to permit persons to whom the Software is furnished to do so. 9 | # ################################################## 10 | 11 | # When deployed to a Greengrass core, this code will be executed immediately 12 | # as a long-lived lambda function. The code will enter the infinite while 13 | # loop below. 14 | # If you execute a 'test' on the Lambda Console, this test will fail by 15 | # hitting the execution timeout of three seconds. This is expected as 16 | # this function never returns a result. 17 | import greengrasssdk 18 | import platform 19 | from threading import Timer 20 | import subprocess 21 | import os 22 | import sys 23 | import logging 24 | logger = logging.getLogger(__name__) 25 | logging.basicConfig(stream=sys.stdout, level=logging.DEBUG) 26 | # Creating a greengrass core sdk client 27 | client = greengrasssdk.client('iot-data') 28 | 29 | # Retrieving platform information to send from Greengrass Core 30 | my_platform = platform.platform() 31 | def send_heart_beat(): 32 | client.publish( 33 | topic='greengrass-deepstream-app/heartbeat', 34 | payload='Heart beat signal sent from ' 35 | 'Greengrass Core running on platform: {}' 36 | .format(my_platform)) 37 | Timer(60, send_heart_beat).start() 38 | return 39 | 40 | 41 | def jetson_deepstream_run(): 42 | send_heart_beat() 43 | bash_command = "./deepstream-app -c ./source1_usb_dec_infer_resnet_int8.txt" 44 | process = subprocess.Popen(bash_command.split(), stdout=subprocess.PIPE) 45 | output, error = process.communicate() 46 | print(output, error) 47 | 48 | # Start executing the function above 49 | jetson_deepstream_run() 50 | 51 | # This is a dummy handler and will not be invoked 52 | # Instead the code above will be executed in an infinite loop for our example 53 | def function_handler(event, context): 54 | return 55 | -------------------------------------------------------------------------------- /provisioning_cf_script/deploy.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # ################################################## 4 | # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 5 | 6 | # Permission is hereby granted, free of charge, to any person obtaining a copy of 7 | # this software and associated documentation files (the "Software"), to deal in 8 | # the Software without restriction, including without limitation the rights to 9 | # use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of 10 | # the Software, and to permit persons to whom the Software is furnished to do so. 11 | # ################################################## 12 | 13 | # define environment label [test/dev/prod...] 14 | if [[ -z "$ENVIRONMENT" ]]; then 15 | ENVIRONMENT=test 16 | echo "Environment variable not specified. Using default: test" 17 | fi 18 | echo "Using Environment $ENVIRONMENT" 19 | 20 | 21 | # define aws profile used as --profle YOUR_ACCOUNT 22 | if [ -z "$AWS_PROFILE" ]; then 23 | echo "Environment variable not specified. Using aws configure without a profile." 24 | fi 25 | echo "Using Region $AWS_PROFILE" 26 | AWS_ARGS="--profile $AWS_PROFILE" 27 | 28 | # get the region account is using 29 | AWS_REGION=$(aws configure get region ${AWS_ARGS}) 30 | echo "Using Region $AWS_REGION" 31 | 32 | # get the account ID to name the S3 asset bucket we are about to create 33 | AWS_ACCOUNT_ID=$(aws sts get-caller-identity --output text --query 'Account' $AWS_ARGS) 34 | echo account id: $AWS_ACCOUNT_ID 35 | 36 | #create S3 bucket 37 | S3_BUCKET="greengrass-deepstream-$AWS_ACCOUNT_ID-${ENVIRONMENT}-assets" 38 | echo "Bucket name : $S3_BUCKET"; 39 | #check if bucket exists 40 | bucketstatus=$(aws s3api head-bucket --bucket "${S3_BUCKET}" $AWS_ARGS 2>&1) 41 | if [ -z "$bucketstatus" ] 42 | then 43 | echo "Bucket exists: $S3_BUCKET, using existing bucket." 44 | else 45 | echo "Bucket does not exist, creating new bucket." 46 | if [ $AWS_REGION == "us-east-1" ]; 47 | then 48 | echo "using us-east-1" 49 | aws s3api create-bucket --bucket "$S3_BUCKET" $AWS_ARGS; 50 | else 51 | aws s3api create-bucket --bucket "$S3_BUCKET" --create-bucket-configuration LocationConstraint="$AWS_REGION" $AWS_ARGS; 52 | fi 53 | fi 54 | 55 | # Decide whether to create new thing for this deployment 56 | GG_THING_EXISTS=$(aws iot describe-thing --thing-name $GG_THING_GROUP_NAME $AWS_ARGS --query 'thingArn') 57 | DS_THING_EXISTS=$(aws iot describe-thing --thing-name $DS_THING_GROUP_NAME $AWS_ARGS --query 'thingArn') 58 | 59 | #check if thing exists, and create if they do not currently exist 60 | if [ -z "$GG_THING_EXISTS" ] || [ -z "$DS_THING_EXISTS" ] 61 | then 62 | echo "Thing does not exist, creating GG and GGDS things" 63 | GGDP_PROV_STACK_NAME="greengrass-deepstream-provisioning-$AWS_ACCOUNT_ID-$AWS_REGION-$ENVIRONMENT-stack" 64 | echo "Deploying Greengrass-Deepstream Provisioning Stack environment: $ENVIRONMENT, S3 bucket: $S3_BUCKET, Cloudformation Stack: $GGDP_PROV_STACK_NAME" 65 | sam package \ 66 | --template-file ./provision-greengrass.yml \ 67 | --s3-bucket $S3_BUCKET \ 68 | --output-template-file ./provision-packaged.yaml $AWS_ARGS 69 | sam deploy \ 70 | --template-file ./provision-packaged.yaml \ 71 | --stack-name $GGDP_PROV_STACK_NAME \ 72 | --parameter-overrides \ 73 | ParameterKey=CoreName,ParameterValue=$GG_THING_GROUP_NAME \ 74 | ParameterKey=DeepstreamAppThingName,ParameterValue=$DS_THING_GROUP_NAME \ 75 | S3BucketName=$S3_BUCKET \ 76 | --capabilities CAPABILITY_IAM CAPABILITY_AUTO_EXPAND $AWS_ARGS 77 | else 78 | echo "The thing name already exist, please modify deploy.env, source it, and try this script again." 79 | fi 80 | 81 | # generate pre-signed S3 links to download certificates for Greengrass 82 | echo sudo wget -O /greengrass/certs/certificatePem.cert.pem \"$(aws s3 presign s3://$S3_BUCKET/$GG_THING_GROUP_NAME/certs/certificatePem.cert.pem --expires-in 604800 $AWS_ARGS)\" >> ../formation_cf_script/install_greengrass.sh 83 | echo sudo wget -O /greengrass/certs/privateKey.private.key \"$(aws s3 presign s3://$S3_BUCKET/$GG_THING_GROUP_NAME/certs/privateKey.private.key --expires-in 604800 $AWS_ARGS)\" >> ../formation_cf_script/install_greengrass.sh 84 | echo sudo wget -O /greengrass/config/config.json \"$(aws s3 presign s3://$S3_BUCKET/$GG_THING_GROUP_NAME/config/config.json --expires-in 604800 $AWS_ARGS)\" >> ../formation_cf_script/install_greengrass.sh 85 | 86 | # Certs generated for DeepStream app to connect to IoT or Greengrass 87 | echo sudo wget -O /opt/nvidia/deepstream/deepstream-5.0/sources/libs/aws_protocol_adaptor/device_client/certs/certificatePem.cert.pem \"$(aws s3 presign s3://$S3_BUCKET/$DS_THING_GROUP_NAME/certs/certificatePem.cert.pem --expires-in 604800 $AWS_ARGS)\" > ../output.txt 88 | echo sudo wget -O /opt/nvidia/deepstream/deepstream-5.0/sources/libs/aws_protocol_adaptor/device_client/certs/privateKey.private.key \"$(aws s3 presign s3://$S3_BUCKET/$DS_THING_GROUP_NAME/certs/privateKey.private.key --expires-in 604800 $AWS_ARGS)\" >> ../output.txt 89 | echo sudo wget -O /opt/nvidia/deepstream/deepstream-5.0/sources/libs/aws_protocol_adaptor/device_client/certs/root.ca.pem https://www.amazontrust.com/repository/AmazonRootCA1.pem >> ../output.txt 90 | 91 | # Environmental variable 92 | echo S3_BUCKET=\"$S3_BUCKET\" >> ../deploy.env 93 | -------------------------------------------------------------------------------- /provisioning_cf_script/device_policy.json: -------------------------------------------------------------------------------- 1 | // ////////////////////////////////////////////////// 2 | // Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | 4 | // Permission is hereby granted, free of charge, to any person obtaining a copy of 5 | // this software and associated documentation files (the "Software"), to deal in 6 | // the Software without restriction, including without limitation the rights to 7 | // use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of 8 | // the Software, and to permit persons to whom the Software is furnished to do so. 9 | // ////////////////////////////////////////////////// 10 | 11 | { 12 | "Version": "2012-10-17", 13 | "Statement": [ 14 | { 15 | "Effect": "Allow", 16 | "Action":["iot:*"], 17 | "Resource": ["*"] 18 | }, 19 | { 20 | "Effect": "Allow", 21 | "Action":["greengrass:*"], 22 | "Resource": ["*"] 23 | } 24 | ] 25 | } 26 | -------------------------------------------------------------------------------- /provisioning_cf_script/greengrass_creation_cust_resource_lambda/cfnresponse.py: -------------------------------------------------------------------------------- 1 | # ################################################## 2 | # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | 4 | # Permission is hereby granted, free of charge, to any person obtaining a copy of 5 | # this software and associated documentation files (the "Software"), to deal in 6 | # the Software without restriction, including without limitation the rights to 7 | # use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of 8 | # the Software, and to permit persons to whom the Software is furnished to do so. 9 | # ################################################## 10 | 11 | from botocore.vendored import requests 12 | import json 13 | 14 | SUCCESS = "SUCCESS" 15 | FAILED = "FAILED" 16 | 17 | def send(event, context, responseStatus, responseData, physicalResourceId=None, noEcho=False): 18 | responseUrl = event['ResponseURL'] 19 | 20 | print(responseUrl) 21 | 22 | responseBody = {} 23 | responseBody['Status'] = responseStatus 24 | responseBody['Reason'] = 'See the details in CloudWatch Log Stream: ' + context.log_stream_name 25 | responseBody['PhysicalResourceId'] = physicalResourceId or context.log_stream_name 26 | responseBody['StackId'] = event['StackId'] 27 | responseBody['RequestId'] = event['RequestId'] 28 | responseBody['LogicalResourceId'] = event['LogicalResourceId'] 29 | responseBody['NoEcho'] = noEcho 30 | responseBody['Data'] = responseData 31 | 32 | json_responseBody = json.dumps(responseBody) 33 | 34 | print("Response body:\n" + json_responseBody) 35 | 36 | headers = { 37 | 'content-type' : '', 38 | 'content-length' : str(len(json_responseBody)) 39 | } 40 | 41 | try: 42 | response = requests.put(responseUrl, 43 | data=json_responseBody, 44 | headers=headers) 45 | print("Status code: " + response.reason) 46 | except Exception as e: 47 | print("send(..) failed executing requests.put(..): " + str(e)) -------------------------------------------------------------------------------- /provisioning_cf_script/greengrass_creation_cust_resource_lambda/greengrass_creation_cust_resource_lambda.py: -------------------------------------------------------------------------------- 1 | # ################################################## 2 | # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | 4 | # Permission is hereby granted, free of charge, to any person obtaining a copy of 5 | # this software and associated documentation files (the "Software"), to deal in 6 | # the Software without restriction, including without limitation the rights to 7 | # use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of 8 | # the Software, and to permit persons to whom the Software is furnished to do so. 9 | # ################################################## 10 | 11 | import sys 12 | import cfnresponse 13 | import boto3 14 | from botocore.exceptions import ClientError 15 | import json 16 | import logging 17 | import os 18 | logger = logging.getLogger() 19 | logger.setLevel(logging.INFO) 20 | S3BUCKETNAME = os.environ['S3BUCKETNAME'] 21 | policyDocument = { 22 | 'Version': '2012-10-17', 23 | 'Statement': [ 24 | { 25 | 'Effect': 'Allow', 26 | 'Action': 'iot:*', 27 | 'Resource': '*' 28 | }, 29 | { 30 | 'Effect': 'Allow', 31 | 'Action': 'greengrass:*', 32 | 'Resource': '*' 33 | } 34 | ] 35 | } 36 | 37 | 38 | def handler(event, context): 39 | responseData = {} 40 | try: 41 | logger.info('Received event: {}'.format(json.dumps(event))) 42 | result = cfnresponse.FAILED 43 | client = boto3.client('iot') 44 | thingName = event['ResourceProperties']['ThingName'] 45 | if event['RequestType'] == 'Create': 46 | thing = client.create_thing( 47 | thingName=thingName 48 | ) 49 | response = client.create_keys_and_certificate( 50 | setAsActive=True 51 | ) 52 | certId = response['certificateId'] 53 | certArn = response['certificateArn'] 54 | certPem = response['certificatePem'] 55 | privateKey = response['keyPair']['PrivateKey'] 56 | publicKey = response['keyPair']['PublicKey'] 57 | client.create_policy( 58 | policyName='{}-full-access'.format(thingName), 59 | policyDocument=json.dumps(policyDocument) 60 | ) 61 | response = client.attach_policy( 62 | policyName='{}-full-access'.format(thingName), 63 | target=certArn 64 | ) 65 | response = client.attach_thing_principal( 66 | thingName=thingName, 67 | principal=certArn, 68 | ) 69 | 70 | logger.info('Created thing: %s, cert: %s and policy: %s' % 71 | (thingName, certId, '{}-full-access'.format(thingName))) 72 | result = cfnresponse.SUCCESS 73 | s3 = boto3.resource('s3') 74 | responseData['certificateArn'] = certArn 75 | responseData['certificateId'] = certId 76 | responseData['certificatePem'] = certPem 77 | s3_obj = s3.Object(S3BUCKETNAME, thingName + 78 | '/certs/certificatePem.cert.pem') 79 | s3_obj.put(Body=certPem) 80 | responseData['privateKey'] = privateKey 81 | s3_obj = s3.Object(S3BUCKETNAME, thingName + 82 | '/certs/privateKey.private.key') 83 | s3_obj.put(Body=privateKey) 84 | s3_obj = s3.Object(S3BUCKETNAME, thingName + 85 | '/certs/publicKey.public.key') 86 | s3_obj.put(Body=publicKey) 87 | responseData['iotEndpoint'] = client.describe_endpoint( 88 | endpointType='iot:Data-ATS')['endpointAddress'] 89 | responseData['thingArn'] = thing["thingArn"] 90 | if "Region" in event['ResourceProperties']: 91 | config_string = """{{ 92 | "coreThing" : {{ 93 | "caPath" : "root.ca.pem", 94 | "certPath" : "certificatePem.cert.pem", 95 | "keyPath" : "privateKey.private.key", 96 | "thingArn" : "{}", 97 | "iotHost" : "{}", 98 | "ggHost" : "greengrass-ats.iot.{}.amazonaws.com", 99 | "keepAlive" : 600 100 | }}, 101 | "runtime" : {{ 102 | "cgroup" : {{ 103 | "useSystemd" : "yes" 104 | }}, 105 | "allowFunctionsToRunAsRoot" : "yes" 106 | }}, 107 | "managedRespawn" : false, 108 | "crypto" : {{ 109 | "principals" : {{ 110 | "SecretsManager" : {{ 111 | "privateKeyPath" : "file:///greengrass/certs/privateKey.private.key" 112 | }}, 113 | "IoTCertificate" : {{ 114 | "privateKeyPath" : "file:///greengrass/certs/privateKey.private.key", 115 | "certificatePath" : "file:///greengrass/certs/certificatePem.cert.pem" 116 | }} 117 | }}, 118 | "caPath" : "file:///greengrass/certs/root.ca.pem" 119 | }} 120 | }} 121 | """.format(responseData["thingArn"], responseData['iotEndpoint'], event['ResourceProperties']['Region']) 122 | s3_obj = s3.Object(S3BUCKETNAME, thingName + 123 | '/config/config.json') 124 | s3_obj.put(Body=config_string) 125 | elif event['RequestType'] == 'Update': 126 | logger.info('Updating thing: %s' % thingName) 127 | result = cfnresponse.SUCCESS 128 | elif event['RequestType'] == 'Delete': 129 | logger.info('Deleting thing: %s and cert/policy' % thingName) 130 | response = client.list_thing_principals( 131 | thingName=thingName 132 | ) 133 | for i in response['principals']: 134 | response = client.detach_thing_principal( 135 | thingName=thingName, 136 | principal=i 137 | ) 138 | response = client.detach_policy( 139 | policyName='{}-full-access'.format(thingName), 140 | target=i 141 | ) 142 | response = client.update_certificate( 143 | certificateId=i.split('/')[-1], 144 | newStatus='INACTIVE' 145 | ) 146 | response = client.delete_certificate( 147 | certificateId=i.split('/')[-1], 148 | forceDelete=True 149 | ) 150 | response = client.delete_policy( 151 | policyName='{}-full-access'.format(thingName), 152 | ) 153 | response = client.delete_thing( 154 | thingName=thingName 155 | ) 156 | result = cfnresponse.SUCCESS 157 | except ClientError as e: 158 | logger.error('Error: {}'.format(e)) 159 | result = cfnresponse.FAILED 160 | logger.info('Returning response of: {}, with result of: {}'.format( 161 | result, responseData)) 162 | sys.stdout.flush() 163 | cfnresponse.send(event, context, result, responseData) 164 | -------------------------------------------------------------------------------- /provisioning_cf_script/provision-greengrass.yml: -------------------------------------------------------------------------------- 1 | # ################################################## 2 | # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | 4 | # Permission is hereby granted, free of charge, to any person obtaining a copy of 5 | # this software and associated documentation files (the "Software"), to deal in 6 | # the Software without restriction, including without limitation the rights to 7 | # use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of 8 | # the Software, and to permit persons to whom the Software is furnished to do so. 9 | # ################################################## 10 | 11 | AWSTemplateFormatVersion: "2010-09-09" 12 | Description: "Greengrass Deepstream Integration Template" 13 | Transform: AWS::Serverless-2016-10-31 14 | 15 | Description: Create Greengrass resources for a AWS Secure Transport tariler 16 | 17 | Parameters: 18 | CoreName: 19 | Description: Green Core name to be created. A "Thing" with be created with _core appended to the name 20 | Type: String 21 | Default: greengrass-core 22 | DeepstreamAppThingName: 23 | Description: DeepstreamApp name to be created. A "Thing" with be created under greengrass group 24 | Type: String 25 | Default: greengrass-deepstream-app 26 | S3BucketName: 27 | Description: S3 bucket name to put generated certificates 28 | Type: String 29 | 30 | Resources: 31 | DeepstreamAppThing: 32 | # Resource creates thing, certificate key pair, and IoT policy 33 | Type: Custom::DeepstreamAppThing 34 | Properties: 35 | ServiceToken: !GetAtt CreateThingFunction.Arn 36 | ThingName: !Ref DeepstreamAppThingName 37 | GGCore: 38 | # Resource creates thing, certificate key pair, and IoT policy 39 | Type: Custom::GGCore 40 | Properties: 41 | ServiceToken: !GetAtt CreateThingFunction.Arn 42 | ThingName: !Ref CoreName 43 | Region: !Ref "AWS::Region" 44 | CreateThingFunction: 45 | Type: AWS::Serverless::Function 46 | Properties: 47 | Description: Create thing, certificate, and policy, return cert and private key 48 | Handler: greengrass_creation_cust_resource_lambda.handler 49 | Runtime: python3.6 50 | Environment: 51 | Variables: 52 | S3BUCKETNAME: !Ref S3BucketName 53 | Role: !GetAtt CFNHelperLambdaExecutionRole.Arn 54 | Timeout: 60 55 | CodeUri: greengrass_creation_cust_resource_lambda/ 56 | CFNHelperLambdaExecutionRole: 57 | # Role used by CloudFormation created Lambda functions, used by the custom 58 | # resource functions to perform their objectives. 59 | # Overly permissive for iot:* and greengrass:* to reduce Statement complexity 60 | Type: AWS::IAM::Role 61 | Properties: 62 | AssumeRolePolicyDocument: 63 | Version: 2012-10-17 64 | Statement: 65 | - Effect: Allow 66 | Principal: 67 | Service: lambda.amazonaws.com 68 | Action: sts:AssumeRole 69 | Policies: 70 | - PolicyName: root 71 | PolicyDocument: 72 | Version: '2012-10-17' 73 | Statement: 74 | - Effect: Allow 75 | Action: 76 | - logs:CreateLogGroup 77 | - logs:CreateLogStream 78 | - logs:PutLogEvents 79 | Resource: arn:aws:logs:*:*:* 80 | - Effect: Allow 81 | Action: 82 | - iot:* 83 | Resource: "*" 84 | - Effect: Allow 85 | Action: 86 | - s3:* 87 | Resource: "*" #!Sub arn:aws:s3:::${S3BucketName} 88 | - Effect: Allow 89 | Action: 90 | - greengrass:* 91 | Resource: "*" 92 | - Effect: Allow 93 | Action: 94 | - iam:CreateRole 95 | - iam:AttachRolePolicy 96 | - iam:GetRole 97 | - iam:DeleteRole 98 | - iam:PassRole 99 | Resource: !Join ["", ["arn:aws:iam::", !Ref "AWS::AccountId", ":role/greengrass_cfn_", !Ref "AWS::StackName", "_ServiceRole"] ] 100 | Outputs: 101 | DeepstreamAppThingCertificateArn: 102 | Description: DeepstreamAppThing-certificateArn 103 | Value: !GetAtt DeepstreamAppThing.certificateArn 104 | Export: 105 | Name: DeepstreamAppThingCertificateArn 106 | 107 | DeepstreamAppThingThingArn: 108 | Description: DeepstreamAppThing-thingArn 109 | Value: !GetAtt DeepstreamAppThing.thingArn 110 | Export: 111 | Name: DeepstreamAppThingThingArn 112 | 113 | GGCoreCertificateId: 114 | Description: GGCore-certificateId 115 | Value: !GetAtt GGCore.certificateId 116 | Export: 117 | Name: GGCoreCertificateId 118 | 119 | GGCoreThingArn: 120 | Description: GGCore-thingArn 121 | Value: !GetAtt GGCore.thingArn 122 | Export: 123 | Name: GGCoreThingArn --------------------------------------------------------------------------------