├── .github └── PULL_REQUEST_TEMPLATE.md ├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── LICENSE ├── README.md └── images ├── arch.png ├── payload.png └── quicksight.png /.github/PULL_REQUEST_TEMPLATE.md: -------------------------------------------------------------------------------- 1 | *Issue #, if available:* 2 | 3 | *Description of changes:* 4 | 5 | 6 | By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice. 7 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | ## Code of Conduct 2 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 3 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 4 | opensource-codeofconduct@amazon.com with any additional questions or comments. 5 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing Guidelines 2 | 3 | Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional 4 | documentation, we greatly value feedback and contributions from our community. 5 | 6 | Please read through this document before submitting any issues or pull requests to ensure we have all the necessary 7 | information to effectively respond to your bug report or contribution. 8 | 9 | 10 | ## Reporting Bugs/Feature Requests 11 | 12 | We welcome you to use the GitHub issue tracker to report bugs or suggest features. 13 | 14 | When filing an issue, please check [existing open](https://github.com/aws-samples/aws-iot-analytics-workshop/issues), or [recently closed](https://github.com/aws-samples/aws-iot-analytics-workshop/issues?utf8=%E2%9C%93&q=is%3Aissue%20is%3Aclosed%20), issues to make sure somebody else hasn't already 15 | reported the issue. Please try to include as much information as you can. Details like these are incredibly useful: 16 | 17 | * A reproducible test case or series of steps 18 | * The version of our code being used 19 | * Any modifications you've made relevant to the bug 20 | * Anything unusual about your environment or deployment 21 | 22 | 23 | ## Contributing via Pull Requests 24 | Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that: 25 | 26 | 1. You are working against the latest source on the *master* branch. 27 | 2. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already. 28 | 3. You open an issue to discuss any significant work - we would hate for your time to be wasted. 29 | 30 | To send us a pull request, please: 31 | 32 | 1. Fork the repository. 33 | 2. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change. 34 | 3. Ensure local tests pass. 35 | 4. Commit to your fork using clear commit messages. 36 | 5. Send us a pull request, answering any default questions in the pull request interface. 37 | 6. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation. 38 | 39 | GitHub provides additional document on [forking a repository](https://help.github.com/articles/fork-a-repo/) and 40 | [creating a pull request](https://help.github.com/articles/creating-a-pull-request/). 41 | 42 | 43 | ## Finding contributions to work on 44 | Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels (enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any ['help wanted'](https://github.com/aws-samples/aws-iot-analytics-workshop/labels/help%20wanted) issues is a great place to start. 45 | 46 | 47 | ## Code of Conduct 48 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 49 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 50 | opensource-codeofconduct@amazon.com with any additional questions or comments. 51 | 52 | 53 | ## Security issue notifications 54 | If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github issue. 55 | 56 | 57 | ## Licensing 58 | 59 | See the [LICENSE](https://github.com/aws-samples/aws-iot-analytics-workshop/blob/master/LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution. 60 | 61 | We may ask you to sign a [Contributor License Agreement (CLA)](http://en.wikipedia.org/wiki/Contributor_License_Agreement) for larger changes. 62 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining a copy of 4 | this software and associated documentation files (the "Software"), to deal in 5 | the Software without restriction, including without limitation the rights to 6 | use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of 7 | the Software, and to permit persons to whom the Software is furnished to do so. 8 | 9 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 10 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS 11 | FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 12 | COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER 13 | IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN 14 | CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 15 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # AWS IoT Analytics Workshop 2 | 3 | In this workshop, you will learn about the different components of AWS IoT Analytics. You will configure AWS IoT Core to ingest stream data from the AWS Device Simulator, process batch data using Amazon ECS, build an analytics pipeline using AWS IoT Analytics, visualize the data using Amazon QuickSight, and perform machine learning using Jupyter Notebooks. Join us, and build a solution that helps you perform analytics on appliance energy usage in a smart building and forecast energy utilization to optimize consumption. 4 | 5 | ## Table of Contents 6 | 7 | * [Prerequisites](#prerequisites) 8 | * [Solution Architecture Overview](#solution-architecture-overview) 9 | * [Step 1a: Build the Streaming data workflow](#step-1a-build-the-streaming-data-workflow) 10 | * [Step 1b: Create Stream Analytics Pipeline](#step-1b-create-stream-analytics-pipeline) 11 | * [Step 1c: Analyse the data](#step-1c-analyse-the-data) 12 | * [Step 2a: Build the Batch analytics workflow](#step-2a-build-the-batch-analytics-workflow) 13 | * [Step 2b: Create Batch Analytics Pipeline](#step-2b-create-batch-analytics-pipeline) 14 | * [Step 2c: Analyse Stream and Batch data](#step-2c-analyse-stream-and-batch-data) 15 | * [Recap and Review - So far](#recap-and-review-what-have-we-done-so-far) 16 | * [Step 3: Visualize your data using Quicksight](#step-3-visualize-your-data-using-quicksight) 17 | * [Step 4: Machine Learning and Forecasting with Jupyter Notebooks](#step-4-machine-learning-and-forecasting-with-jupyter-notebooks) 18 | * [Recap and Review - What did we learn in this workshop?](#recap-and-review-what-did-we-learn-in-this-workshop) 19 | * [Clean up resources in AWS](#clean-up-resources-in-aws) 20 | * [Troubleshooting](#troubleshooting) 21 | 22 | ## Prerequisites 23 | 24 | To conduct the workshop you will need the following tools/setup/knowledge: 25 | 26 | * AWS Account 27 | * Laptop 28 | * Secure shell (ssh) to login into your Docker instance (EC2) 29 | * Mac OS/Linux: command lines tools are installed by default 30 | * Windows 31 | * Putty: ssh client: [http://www.putty.org/](http://www.putty.org/) 32 | * Manual connect (ssh) to an EC2 instance from Windows with Putty: [http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/putty.html](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/putty.html) 33 | * You need to have an ssh key-pair - 34 | * A ssh key-pair can be generated or imported in the AWS console under EC2 -> Key Pairs 35 | * Download the .pem file locally to log into the Ec2 docker instance later in the workshop 36 | * For MAC / UNIX, change permissions - chmod 400 "paste-your-keypair-filename" 37 | * You are in one of the following regions: 38 | * us-east-1 (N Virgina) 39 | * us-east-2 (Ohio) 40 | * us-west-2 (Oregon) 41 | * eu-west-1 (Ireland) 42 | * You do not have more than 3 VPCs already deployed in the active region. In the workshop you will create 2 VPCs, and the limit for VPCs per region is 5. 43 | * You do not have more than 3 Elastic IP addresses already deloyed in the active region. In the workshop you will create 2 Elastic IP addresses for the Device Simulator, and the limit of Elastic IPs per region is 5. 44 | 45 | ## Solution Architecture Overview: 46 | 47 | ![alt text](https://github.com/aws-samples/aws-iot-analytics-workshop/blob/master/images/arch.png "Architecture") 48 | 49 | ## Step 1a: Build the Streaming data workflow 50 | 51 | ### Launch AWS IoT Device Simulator with CloudFormation 52 | 53 | The IoT Device Simulator allows you to simulate real world devices by creating device types and data schemas via a web interface and allowing them to connect to the AWS IoT message broker. 54 | 55 | By choosing one of the links below you will be automatically redirected to the CloudFormation section of the AWS Console where your IoT Device Simulator stack will be launched: 56 | 57 | * [Launch CloudFormation stack in us-east-1](https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/create/review?stackName=IoTDeviceSimulator&templateURL=https://s3.amazonaws.com/solutions-reference/iot-device-simulator/latest/iot-device-simulator.template) (N. Virginia) 58 | * [Launch CloudFormation stack in us-west-2](https://console.aws.amazon.com/cloudformation/home?region=us-west-2#/stacks/create/review?stackName=IoTDeviceSimulator&templateURL=https://s3.amazonaws.com/solutions-reference/iot-device-simulator/latest/iot-device-simulator.template) (Oregon) 59 | * [Launch CloudFormation stack in us-east-2](https://console.aws.amazon.com/cloudformation/home?region=us-east-2#/stacks/create/review?stackName=IoTDeviceSimulator&templateURL=https://s3.amazonaws.com/solutions-reference/iot-device-simulator/latest/iot-device-simulator.template) (Ohio) 60 | * [Launch CloudFormation stack in eu-west-1](https://console.aws.amazon.com/cloudformation/home?region=eu-west-1#/stacks/create/review?stackName=IoTDeviceSimulator&templateURL=https://s3.amazonaws.com/solutions-reference/iot-device-simulator/latest/iot-device-simulator.template) (Ireland) 61 | 62 | After you have been redirected to the AWS CloudFormation console, take the following steps to launch your stack: 63 | 64 | 1. **Parameters** - Input Administrator Name & Email (An ID and password will be emailed to you for the IoT Device Simulator) 65 | 2. **Capabilities** - Check "I acknowledge that AWS CloudFormation might create IAM resources." at the bottom of the page 66 | 3. **Create Stack** 67 | 4. Wait until the stack creation is complete 68 | 69 | The CloudFormation creation may take between **10-25 mins to complete**. In the **Outputs** section of your CloudFormation stack, you will find the Management console URL for the IoT simulator. Please **copy the url** to use in the next section. 70 | 71 | \[[Top](#Top)\] 72 | 73 | ### Connect your Smart Home to AWS IoT Core 74 | 75 | You will provision the smart home endpoint to publish telemetric data points to AWS IoT. 76 | 77 | ![alt text](https://github.com/aws-samples/aws-iot-analytics-workshop/blob/master/images/arch.png "Architecture") 78 | 79 | Please login to the IoT Device Simulator Management console (link copied from the earlier step) with the provided credentials. 80 | 81 | Credentials for the Device Simulator will be mailed to the email address provided during CloudFormation stack creation. 82 | 83 | 84 | #### Create the Simulated Device 85 | 1. Navigate to **Modules** -> **Device Types** -> Click **Add Device Type** 86 | * **Device Type Name:** smart-home 87 | * **Data Topic:** smarthome/house1/energy/appliances 88 | * **Data Transmission Duration:** 7200000 89 | * **Data Transmission Interval:** 3000 90 | * **Message Payload:** Click **Add Attribute** and add the following attributes: 91 | 92 | | Attribute Name | Data Type | Float Precision | Integer Minimum Value | Integer Maximum Value | 93 | |:---------------------:|:------------------------------:|:---------------:|:---------------------:|:---------------------:| 94 | | sub_metering_1 | float | 2 | 10 | 100 | 95 | | sub_metering_2 | float | 2 | 10 | 100 | 96 | | sub_metering_3 | float | 2 | 10 | 25 | 97 | | global_active_power | float | 2 | 1 | 8 | 98 | | global_reactive_power | float | 2 | 5 | 35 | 99 | | voltage | float | 2 | 10 | 250 | 100 | | timestamp | UTC Timestamp (Choose Default) | | | | 101 | 102 | 3. Once the sample message payload shows all the attributes above, click **Save** 103 | 4. Navigate to **Modules** -> **Widgets** -> **Add Widget** 104 | * Select 'smart-home' as the **Device Type** 105 | * **Number of widgets:** 1 -> **Submit** 106 | 107 | We have now created a simulated smart home device which is collecting power usage data and publishing that data to AWS IoT Core on the 'smarthome/house1/energy/appliances' topic. 108 | 109 | #### Verify that the data is being published to AWS IoT 110 | 111 | **Note**: *You will use the AWS console for the remainder of the workshop. Sign-in to the [AWS console](https://aws.amazon.com/console).* 112 | 113 | We will verify that the smart home device is configured and publishing data to the correct topic. 114 | 1. From the AWS console, choose the **IoT Core** service 115 | 2. Navigate to **Test** (On the left pane) 116 | 3. Under **Subscription** input the following: 117 | * **Subscription topic:** 'smarthome/house1/energy/appliances' 118 | * Click **Subscribe to topic** 119 | 120 | After a few seconds, you should see your simulated devices's data that is published on the 'smarthome/house1/energy/appliances' MQTT topic. 121 | 122 | \[[Top](#Top)\] 123 | 124 | ## Step 1b: Create Stream Analytics Pipeline 125 | 126 | In this section we will create the IoT Analytics components, analyze data and define different pipeline activities. 127 | 128 | ![alt text](https://github.com/aws-samples/aws-iot-analytics-workshop/blob/master/images/arch.png "Architecture") 129 | 130 | ### Create your IoT Analytics S3 Storage Buckets 131 | 132 | First you will need to create 3 S3 buckets, one for your IoT Analytics channel, one for the data store that holds your transformed data, and one for the data set that is resulted from an IoT Analytics SQL query. 133 | 134 | 1. Navigate to the **S3 Management Console** 135 | 2. Choose **Create Bucket** 136 | * **Bucket name:** Give your bucket a unique name (must be globally unique) and append it with '-channel'. For example: 'my-iot-analytics-channel'. 137 | * **Region:** The region should be the same as where you launched the Device Simulator Cloud Formation template. 138 | 3. Click **Next** and keep all options default. Click on **Create bucket** to finish the creation. 139 | 4. Repeat steps 1-3 twice more to finish creating the required buckets. Use the appendices '-datastore' and '-dataset' to differentiate the buckets. 140 | 141 | You will also need to give appropriate permissions to IoT Analytics to access your Data Store bucket. 142 | 1. Navigate to the **S3 Management Console** 143 | 2. Click on your data store bucket ending with '-datastore'. 144 | 3. Navigate to the **Permissions** tab 145 | 4. Click on **Bucket Policy** and enter the following JSON policy (be sure to edit to include your S3 bucket name): 146 | ``` 147 | { 148 | "Version": "2012-10-17", 149 | "Id": "IoTADataStorePolicy", 150 | "Statement": [ 151 | { 152 | "Sid": "IoTADataStorePolicyID", 153 | "Effect": "Allow", 154 | "Principal": { 155 | "Service": "iotanalytics.amazonaws.com" 156 | }, 157 | "Action": [ 158 | "s3:GetBucketLocation", 159 | "s3:GetObject", 160 | "s3:ListBucket", 161 | "s3:ListBucketMultipartUploads", 162 | "s3:ListMultipartUploadParts", 163 | "s3:AbortMultipartUpload", 164 | "s3:PutObject", 165 | "s3:DeleteObject" 166 | ], 167 | "Resource": [ 168 | "arn:aws:s3:::", 169 | "arn:aws:s3:::/*" 170 | ] 171 | } 172 | ] 173 | } 174 | ``` 175 | 5. Click **Save** 176 | 177 | ### Create the IoT Analytics Channel 178 | 179 | Next we will create the IoT Analytics channel that will consume data from the IoT Core broker and store the data into your S3 bucket. 180 | 181 | 1. Navigate to the **AWS IoT Analytics** console. 182 | 2. In the left navigation pane, navigate to **Channels** 183 | 3. **Create** a new channel 184 | * **ID:** streamchannel 185 | * **Choose the Storage Type:** Customer Managed S3 Bucket, and choose your Channel S3 bucket created in the previous step. 186 | * **IAM Role:** Create New, and give your new IAM Role a name. This will give IoT Analytics the correct IAM policies to access your S3 bucket. 187 | 7. Click '**Next**' and input the following. This step will create an IoT Rule that consumes data on the specified topic. 188 | * **IoT Core topic filter:** 'smarthome/house1/energy/appliances' 189 | * **IAM Role:** Create New, and give your new IAM Role a name. This will give IoT Analytics the correct IAM policies to access your AWS IoT Core topic. 190 | * Click on **See messages** to see the messages from your smartbuilding device arriving on the topic. Ensure your device is still running in Device Simulator if you do not see any messages. 191 | 8. Click **Create Channel** 192 | 193 | ### Create the IoT Analytics Data Store for your pipeline 194 | 195 | 1. Navigate to the **AWS IoT Analytics** console. 196 | 2. In the left navigation pane, navigate to **Data stores** 197 | 3. **Create** a new data store 198 | * **ID:** iotastore 199 | * **Choose the Storage Type:** Customer Managed S3 Bucket, and choose your Data Store S3 bucket created in the previous step. 200 | * **IAM Role:** Create New, and give your new IAM Role a name. This will give IoT Analytics the correct IAM policies to access your S3 bucket. 201 | 4. Click 'Next' and then **Create data store** 202 | 203 | ### Create the IoT Analytics Pipeline 204 | 205 | 1. Navigate to the **AWS IoT Analytics** console. 206 | 2. In the left navigation pane, navigate to **Pipelines** 207 | 3. **Create** a new Pipeline: 208 | * **ID:** streampipeline 209 | * **Pipeline source**: streamchannel 210 | 6. Click **Next** 211 | 7. IoT Analytics will automatically parse the data coming from your channel and list the attributes from your simulated device. By default, all messages are selected. 212 | 8. Click **Next** 213 | 9. Under 'Pipeline activites' you can trasform the data in your pipeline, add, or remove attributes 214 | 10. Click **Add Activity** and choose **Calculate a message attribute** as the type. 215 | * **Attribute Name:** cost 216 | * **Formula:** ``(sub_metering_1 + sub_metering_2 + sub_metering_3) * 1.5`` 217 | 13. Test your formula by clicking **Update preview** and the cost attribute will appear in the message payload below. 218 | 14. Add a second activity by clicking **Add activity** and **Remove attributes from a message** 219 | * **Attribute Name:** _id_ and click 'Next'. The _id_ attribute is a unique identifier coming from the Device Simulator, but adds noise to the data set. 220 | 16. Click **Update preview** and the _id_ attribute will disappear from the message payload. 221 | 17. Click 'Next' 222 | 18. **Pipeline output:** Click 'Edit' and choose 'iotastore' 223 | 19. Click **Create Pipeline** 224 | 225 | Your IoT Analytics pipeline is now set up. 226 | 227 | \[[Top](#Top)\] 228 | 229 | ## Step 1c: Analyse the data 230 | 231 | In this section, you will learn how to use IoT Analytics to extract insights from your data set using SQL over a specified time period. 232 | 233 | ![alt text](https://github.com/aws-samples/aws-iot-analytics-workshop/blob/master/images/arch.png "Architecture") 234 | 235 | 236 | ### Create a data set 237 | 238 | 1. Navigate to the **AWS IoT Analytics** console. 239 | 2. In the left navigation pane, navigate to **Data sets** 240 | 3. Choose **Create a data set** 241 | 4. Select **Create SQL** 242 | * **ID:** streamdataset 243 | * **Select data store source:** iotastore - this is the S3 bucket containing the transformed data created in step 1b. 244 | 7. Click **Next** 245 | 8. Keep the default SQL statement, which should read ``SELECT * FROM iotastore`` and click **Next** 246 | 9. Input the folllowing: 247 | * **Data selection window:** Delta time 248 | * **Offset:** -5 Seconds - the 5 second offset is to ensure all 'in-flight' data is included into the data set at time of execution. 249 | * **Timestamp expression:** ``from_iso8601_timestamp(timestamp)`` - we need to convert the ISO8601 timestamp coming from the streaming data to a standard timestamp. 250 | 12. Keep all other options as default and click **Next** until you reach 'Configure the delivery rules of your analytics results' 251 | 13. Click **Add rule** 252 | 14. Choose **Deliver result to S3** 253 | * **S3 bucket:** select the S3 bucket that ends with '-dataset' 254 | 13. Click **Create data set** to finalise the creation of the data set. 255 | 256 | ### Execute and save the dataset 257 | 258 | 1. Navigate to **Data sets** on the lefthand navigation pane of the AWS IoT Analytics console. 259 | 2. Click on 'streamdataset' 260 | 3. Click on **Actions** and in the dropdown menu choose **Run now** 261 | 4. On the left navigation menu, choose **Content** and monitor the status of your data set creation. 262 | 5. The results will be shown in the preview pane and saved as a .csv in the '-dataset' S3 bucket. 263 | 264 | 265 | \[[Top](#Top)\] 266 | 267 | ## Step 2a: Build the Batch analytics workflow 268 | 269 | In this section we will create an EC2 instance and docker image to batch a public dataset from S3 to an IoT Analytics data store using containers. 270 | 271 | ![alt text](https://github.com/aws-samples/aws-iot-analytics-workshop/blob/master/images/arch.png "Architecture") 272 | 273 | ### Launch Docker EC2 Instance with CloudFormation 274 | 275 | By choosing one of the links below you will be automatically redirected to the CloudFormation section of the AWS Console where your stack will be launched. 276 | 277 | Before launching the CloudFormation, you will need an SSH key pair to log into the EC2 instance. If you don't have an SSH key pair you can create one by: 278 | 1. Navigate to the **EC2 console** 279 | 2. Click on **Key Pairs** 280 | 3. Click on **Create Key Pair** and input a name. 281 | 4. Save the .pem file in a directory accessible on your computer. 282 | 5. If you are running Mac or Linux, set the appropriate permissions on the keypair: 283 | * ``chmod 400 myec2keypair.pem`` 284 | 285 | 286 | To launch the CloudFormation stack, choose one of the following links for your region, and follow the steps below: 287 | 288 | * [Launch CloudFormation stack in us-east-1](https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/create/review?stackName=IoTAnalyticsStack&templateURL=https://s3.amazonaws.com/iotareinvent18/iota-reinvent-cfn-east.json) (N. Virginia) 289 | * [Launch CloudFormation stack in us-west-2](https://console.aws.amazon.com/cloudformation/home?region=us-west-2#/stacks/create/review?stackName=IoTAnalyticsStack&templateURL=https://s3.amazonaws.com/iotareinvent18/iota-reinvent-cfn-west.json) (Oregon) 290 | * [Launch CloudFormation stack in us-east-2](https://console.aws.amazon.com/cloudformation/home?region=us-east-2#/stacks/create/review?stackName=IoTAnalyticsStack&templateURL=https://s3.amazonaws.com/iotareinvent18/iota-reinvent-cfn-east-2.json) (Ohio) 291 | * [Launch CloudFormation stack in eu-west-2](https://console.aws.amazon.com/cloudformation/home?region=eu-west-1#/stacks/create/review?stackName=IoTAnalyticsStack&templateURL=https://s3.amazonaws.com/iotareinvent18/iota-reinvent-cfn-west-2.json) (Ireland) 292 | 293 | 1. Navigate to **Parameters** 294 | * **SSHKeyName** - select the SSH key pair you will use to login to the EC2 instance. 295 | 2. Check the box "I acknowledge that AWS CloudFormation might create IAM resources." 296 | 3. Click **Create stack** 297 | 298 | The CloudFormation stack will take approximately 5-7 minutes to complete launching all the necessary resources. 299 | 300 | Once the CloudFormation has completed, navigate to the **Outputs** tab, and see the **SSHLogin** parameter. Copy this string to use when SSHing to the EC2 instance. 301 | 302 | ### Setup the docker image on EC2 303 | 304 | 1. SSH to the EC2 instance using the SSHLogin string copied from the above step. 305 | * Example: ``ssh -i Iotaworkshopkeypair.pem ec2-user@ec2-my-ec2-instance.eu-west-1.compute.amazonaws.com`` 306 | 2. Move to the docker-setup folder 307 | * ``cd /home/ec2-user/docker-setup`` 308 | 3. Update your EC2 instance: 309 | * ``sudo yum update`` 310 | 4. Build the docker image: 311 | * ``docker build -t container-app-ia .`` 312 | 5. Veryify the image is running: 313 | * ``docker image ls | grep container-app-ia`` 314 | * You should see an output similar to: ``container-app-ia latest ad81fed784f1 2 minutes ago 534MB`` 315 | 6. Create a new repository in Amazon Elastic Container Registry (ECR) using the AWS CLI (pre-built on your EC2 instance): 316 | * ``aws ecr create-repository --repository-name container-app-ia`` 317 | * The output should include a JSON object which includes the item 'repositoryURI'. Copy this value into a text editor for later use. 318 | 7. Login to your Docker environment _(you need the \` for the command to work)_: 319 | * `` `aws ecr get-login --no-include-email` `` 320 | 8. Tag the Docker image with the ECR Repository URI: 321 | * `docker tag container-app-ia:latest :latest` 322 | 9. Push the image to ECR 323 | * `docker push ` 324 | 325 | \[[Top](#Top)\] 326 | 327 | ## Step 2b: Create Batch Analytics Pipeline 328 | 329 | ![alt text](https://github.com/aws-samples/aws-iot-analytics-workshop/blob/master/images/arch.png "Architecture") 330 | 331 | In this section we will create the IoT Analytics pipeline for your public data-set, analyze data and define different pipeline activities. 332 | 333 | ### Create the IoT Analytics Channel 334 | 335 | Next we will create the IoT Analytics channel that will consume data from the IoT Core broker and store the data into your S3 bucket. 336 | 337 | 1. Navigate to the **AWS IoT Analytics** console. 338 | 2. In the left navigation pane, navigate to **Channels** 339 | 3. **Create** a new channel 340 | * **ID:** batchchannel 341 | * **Choose the Storage Type:** Service-managed store - in this step we will use an IoT Analytics managed S3 bucket, but you may specify a customer-managed bucket as in Step 1b if you wish. 342 | 4. **IoT Core topic filter:** Leave this blank, as the data source for this channel will not be from AWS IoT Core. 343 | 5. Leave all other options as default and click **Next**. 344 | 6. Click **Create Channel** 345 | 346 | ### Create the IoT Analytics Pipeline 347 | 348 | 1. Navigate to the **AWS IoT Analytics** console. 349 | 2. In the left navigation pane, navigate to **Pipelines** 350 | 3. **Create** a new Pipeline: 351 | * **ID:** batchpipeline 352 | * **Pipeline source**: batchchannel 353 | 4. Click **Next** 354 | 5. **See attributes of messages:** You will not see any data on this screen, as the data source has not been fully configured yet. 355 | 6. Click **Next** 356 | 7. Under 'Pipeline activites' you can trasform the data in your pipeline, add, or remove attributes 357 | 8. Click **Add Activity** and choose **Calculate a message attribute** as the type. 358 | * **Attribute Name:** cost 359 | * **Formula:** ``(sub_metering_1 + sub_metering_2 + sub_metering_3) * 1.5`` 360 | 9. Click 'Next' 361 | 10. **Pipeline output:** Click 'Edit' and choose 'iotastore' 362 | 11. Click **Create Pipeline** 363 | 364 | Now we have created the IoT Analytics Pipeline, we can load the batch data. 365 | 366 | ### Create Container Data Set 367 | 368 | A container data set allows you to automatically run your analysis tools and generate results. It brings together a SQL data set as input, a Docker container with your analysis tools and needed library files, input and output variables, and an optional schedule trigger. The input and output variables tell the executable image where to get the data and store the results. 369 | 370 | 1. Navigate to the **IoT Analytics Console** 371 | 2. Click on **Data sets** on the left-hand navigation pane 372 | 3. **Create** a new data set. 373 | 4. Choose **Create Container** next to **Container data sets**. 374 | * **ID**: container_dataset 375 | 5. Click **Next** 376 | 6. Click **Create** next to **Create an analysis without a data set** 377 | * **Frequency**: Not scheduled 378 | * Click **Next** 379 | 7. **Select analysis container and map variables** 380 | * Choose **Select from your Elastic Container Registry repository** 381 | * **Select your custom analysis container from Elastic Container Registry:** container-app-ia 382 | * **Select your image:** latest 383 | 8. Under **Configure the input variables of your container** add the following variables which will be passed to your Docker container and the Python script running in the instance: 384 | 385 | | Name | Type | Value | 386 | |-----------------------|--------|----------------| 387 | | inputDataS3BucketName | String | iotareinvent18 | 388 | | inputDataS3Key | String | inputdata.csv | 389 | | iotchannel | String | batchchannel | 390 | 391 | 9. Click **Next** 392 | 10. Under **IAM Role**, click **Edit** and add the role iotAContainerRole. This role was created as part of the CloudFormation stack. 393 | 11. Configure the resources for container execution: 394 | * **Compute Resource**: 4 vCPUs and 16 GiB Memory 395 | * **Volume Size (GB)**: 2 396 | 12. Click **Next** 397 | 13. Under **Configure the results of your analytics**, keep all options as default. 398 | 14. Click **Next** 399 | 15. Leave **Configure delivery rules for analysis results** as default and click **Next**. 400 | * On this menu you can configure the data set to be delivered to an S3 bucket if you wish. 401 | 16. Finalize the creation of the data-set by clicking **Create data set** 402 | 403 | ### Execute the data set 404 | 1. Navigate to the **AWS IoT Analytics console** 405 | 2. In the left navigation pane, navigate to **Data sets** 406 | 3. Click on **container_dataset** 407 | 4. Choose **Actions** and **Run Now** 408 | 5. The container dataset can take up to 10 minutes to run. If there are no errors, you will see a **SUCCEEDED** message. If it fails with an error, please see the Troubleshooting section below to enable logging. 409 | 410 | \[[Top](#Top)\] 411 | 412 | ## Step 2c: Analyse Stream and Batch data 413 | 414 | ![alt text](https://github.com/aws-samples/aws-iot-analytics-workshop/blob/master/images/arch.png "Architecture") 415 | 416 | Now that we have two data sets into the same data store, we can analyse the result of combining both the container data and simulated device data. 417 | 418 | ### Create and execute the combined data set 419 | 420 | 1. Navigate to the **AWS IoT Analytics console** 421 | 2. In the left navigation pane, navigate to **Data sets** 422 | * **ID**: batchdataset 423 | * **Select data store source**: iotastore 424 | 3. Click **Next** 425 | 4. **SQL Query**: ``SELECT * FROM iotastore limit 5000`` 426 | 5. Click **Next** 427 | 6. Leave the rest of the options as default and click **Next** 428 | 7. Optionally, you can choose to have your dataset be placed into an S3 bucket on the **Configure delivery rules for analysis results** pane. 429 | 8. Click **Create data set** 430 | 9. Click on your newly created **batchdataset** 431 | 10. Click on **Actions** and then **Run now** 432 | 11. The data set will take a few minutes to execute. You should see the results of the executed data set in the result preview, as well as an outputted .csv file which includes the complete data set. 433 | 434 | \[[Top](#Top)\] 435 | 436 | ## Recap and Review: What have we done so far? 437 | 438 | In the workshop so far you have accomplished the following: 439 | * Launched an IoT Device Simulator using CloudFormation 440 | * Used the IoT Device Simulator to simulate data coming from a home energy monitoring device. 441 | * Created an IoT Analytics pipeline that consumes, cleans and transforms that data. 442 | * Created a custom data set with SQL that can be used for further analytics. 443 | * Used a second CloudFormation template to launch an EC2 instance with a script to simulate a public data set. 444 | * Used Amazon Elastic Container Registry to host your docker image and IoT Analytics to launch that docker container to simulate a custom data anlytics workload. 445 | * Combined that simulated public data analytics workload with the simulated device data to make a meaningful data set. 446 | 447 | The purpose of the workshop has been to show you how you can use IoT Analytics for your various data analytics workloads, whether that be from external data sets, a custom analytics workload using external tools, or consuming data directly from IoT devices in real time. 448 | 449 | \[[Top](#Top)\] 450 | 451 | ## Step 3: Visualize your data using Quicksight 452 | 453 | In this section we will visualize the time series data captured from your smart home. 454 | 455 | ![alt text](https://github.com/aws-samples/aws-iot-analytics-workshop/blob/master/images/arch.png "Architecture") 456 | 457 | ### Sign up for AWS Quicksight 458 | 459 | If you have used AWS Quicksight in the past, you can skip these steps. 460 | 461 | 1. Navigate to the **AWS Quicksight** console 462 | 2. If you have never used AWS Quicksight before, you will be asked to sign up. Be sure to choose the 'Standard' tier, and choose the correct region for your locale. 463 | 3. During the sign up phase, give Quicksight access to your Amazon S3 buckets and AWS IoT Analytics via the sign up page. 464 | 465 | ### Visualise your data set with AWS Quicksight 466 | 467 | 1. Navige to the **AWS Quicksight** console. 468 | 2. Click on **New analysis** 469 | 3. Click on **New data set** 470 | 4. Choose **AWS IoT Analytics** under FROM NEW DATA SOURCES. 471 | 5. Configure your AWS IoT Analytics data source. 472 | * **Data source name**: smarthome-dashboard 473 | * **Select an AWS IoT Analytics data set to import**: batchdataset 474 | 6. Click on **Create data source** to finalise the Quicksight dashboard data source configuration. You should see the data has also been imported into SPICE, which is the analytics engine driving AWS Quicksight. 475 | 7. Click on **Visualize** and wait a few moments until you see 'Import complete' in the upper right hand of the console. You should see all 5000 rows have been imported into SPICE. 476 | 8. Under **Fields list** choose 'timestamp' to set the X axis for the graph. 477 | 9. Click on 'sub_metering_1', 'sub_metering_2' and 'sub_metering_3' to add them to the **Value** column. 478 | 10. For each sub_metering_value, choose the drop-down menu and set **Aggregate** to **Average**. 479 | 480 | The graphs will look similar to below. 481 | 482 | ![alt text](https://github.com/aws-samples/aws-iot-analytics-workshop/blob/master/images/quicksight.png "Quicksight") 483 | 484 | You can setwith different fields or visual types for visualizing other smart home related information. 485 | 486 | \[[Top](#Top)\] 487 | 488 | ## Step 4: Machine Learning and Forecasting with Jupyter Notebooks. 489 | 490 | ![alt text](https://github.com/aws-samples/aws-iot-analytics-workshop/blob/master/images/arch.png "Architecture") 491 | 492 | In this section we will configure an Amazon Sagemaker instance to forecast energy utilisation in the home. 493 | 494 | 1. Navigate to the **AWS IoT Analytics** console. 495 | 2. Select **Notebooks** from the left-hand navigation pane. 496 | 3. Click **Create** to begin configuring a new Notebook. 497 | 4. Click **Blank Notebook** and input the following: 498 | * **Name**: smarthome_notebook 499 | * **Select data set sources**: batchdataset 500 | * **Select a Notebook instance**: IoTAWorkshopSagemaker - this is the name of the notebook created with CloudFormation 501 | 5. **Create Notebook** 502 | 6. Click on **IoTAWorkshopSagemaker** 503 | * Choose **IoT Analytics** in the drop down menu 504 | * Next to smarthome_notebook.ipynb, choose **Open in Jupyter** 505 | 7. A new Amazon Sagemaker window should pop up and a message that says "Kernel not found". Click **Continue Without Kernel** 506 | 8. Download the following Jupyter notebook: https://s3.amazonaws.com/iotareinvent18/SmartHomeNotebook.ipynb 507 | 9. In the Jupyter Notebook console, click **File** -> **Open** 508 | 10. Choose **Upload** and select the SmartHomeNotebook.ipynb notebook downloaded in step 8. 509 | 11. Click on the SmartHomeNotebook.ipynb to be taken back to the Jupyter Notebook. 510 | 12. Select **conda_mxnet_p36** as the kernel and click **Set kernel**. 511 | 13. You should see some pre-filled Python code steps in the Jupyter notebook. 512 | 14. Click **Run** for each step. Follow the documentation presented in the Notebook. The Notebook includes information about how the machine learning process works for each step. 513 | 15. Run through all the steps in the SmartHomeNotebook to understand how the machine learning training process works with Jupyter Notebooks. _**Note**: Wait until the asterisk * disappears after running each cell. If you click 'Run' before the previous step is completed, this could cause some code to fail to complete and the algorithm will fail._ 514 | 515 | \[[Top](#Top)\] 516 | 517 | ## Recap and Review: What did we learn in this workshop? 518 | 519 | In Steps 3 and 4, we used AWS Quicksight, and Jupyter Notebooks to visualize and use machhine learning to gather additional insights into our data. 520 | 521 | During the workshop, we saw how you can combine raw IoT data coming in real time from devices and public data sets analysed by third party tools. Using Pipelines, you can clean and standardise this data, so you can then perform advanced visualisations and analytics on the data. IoT Analytics is designed to solve the challenge of cleaning, organising, and making usable data out of hundreds, thousands, or even millions of data points. 522 | 523 | 524 | ## Clean up resources in AWS 525 | 526 | In order to prevent incurring additonal charges, please clean up the resources created in the workshop. 527 | 528 | 1. SSH to the EC2 docker instance from step 2c and execute clean-up.sh. This script will delete the IoT Analytics channels, pipelines, and datasets. 529 | * Example: ``ssh -i Iotaworkshopkeypair.pem ec2-user@ec2-my-ec2-instance.eu-west-1.compute.amazonaws.com`` 530 | * `cd /home/ec2-user/clean-up` 531 | * `sh clean-up.sh` 532 | 2. Navigate to the **AWS CloudFormation** console to delete the CloudFormation stacks and their associated resources. 533 | * Click on **IoTAnalyticsStack** and click **Delete** 534 | * Click on **IoTDeviceSimulator** and click **Delete** 535 | * _Note: Deleting the CloudFormation stacks can take several minutes._ 536 | 3. Navigate to the **AWS Quicksight** console 537 | * Click on **Manage Data** 538 | * Click on **batchdataset** and then **Delete data set** then **Delete** 539 | 4. Navigate to the **Amazon ECS** console 540 | * Click on **Repositories** under **Amazon ECR** 541 | * Select **container-app-ia** and click **Delete** 542 | 5. Navigate to the **AWS EC2** console 543 | * Click on **Key Pairs** in the left navigation pane. 544 | * Choose the EC2 keypair you created to SSH to your docker instance and click **Delete**. 545 | 6. Navigate to the **S3 Management Console** and delete the following: 546 | * The 3 buckets you created in Step 1b ending with '-channel', '-dataset', and '-datastore' 547 | * Each bucket with the prefix 'iotdevicesimulator'. 548 | * Each bucket with the prefix 'sagemaker-' 549 | 7. Navigate to the **AWS IoT Analytics** console 550 | * Check that all of your pipelines, data sets, and channels have been deleted. If not, you can manually delete them from the console. 551 | 552 | \[[Top](#Top)\] 553 | 554 | ## Troubleshooting 555 | 556 | To aide in troubleshooting, you can enable logs for IoT Core and IoT Analytics that can be viewed in **AWS CloudWatch**. 557 | 558 | ### AWS IoT Core 559 | 560 | 1. Navigate to the **AWS IoT Core** console 561 | 2. Click on **Settings** in the left-hand pane 562 | 3. Under **Logs**, click on **Edit** 563 | * **Level of verbosity**: Debug (most verbose) 564 | * **Set role**: Click **Create New** 565 | * **Name:** iotcoreloggingrole 566 | 567 | The log files from AWS IoT are send to **Amazon CloudWatch**. The AWS console can be used to look at these logs. 568 | 569 | For additional troubleshooting, refer to here [IoT Core Troubleshooting](https://docs.aws.amazon.com/iot/latest/developerguide/iot_troubleshooting.html) 570 | 571 | \[[Top](#Top)\] 572 | 573 | ### AWS IoT Analytics 574 | 575 | 1. Navigate to the **AWS IoT Analytics** console 576 | 2. Click on **Settings** in the left-hand pane 577 | 3. Under **Logs**, click on **Edit** 578 | * **Level of verbosity**: Debug (most verbose) 579 | * **Set role**: Click **Create New** 580 | * **Name:** iotanalyticsloggingrole 581 | * **Create role** 582 | 4. Click on **Update** 583 | 584 | The log files from AWS IoT Analytics will be send to **Amazon CloudWatch**. The AWS console can be used to look at these logs. 585 | 586 | For additional troubleshooting, refer to here [IoT Analytics Troubleshooting](https://docs.aws.amazon.com/iotanalytics/latest/userguide/troubleshoot.html) 587 | 588 | \[[Top](#Top)\] 589 | -------------------------------------------------------------------------------- /images/arch.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-iot-analytics-workshop/d9026b89407ca488e124921cf70ac2d5e8589618/images/arch.png -------------------------------------------------------------------------------- /images/payload.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-iot-analytics-workshop/d9026b89407ca488e124921cf70ac2d5e8589618/images/payload.png -------------------------------------------------------------------------------- /images/quicksight.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-iot-analytics-workshop/d9026b89407ca488e124921cf70ac2d5e8589618/images/quicksight.png --------------------------------------------------------------------------------