├── manifest.json
├── random.json
├── LICENSE
├── SQL_Destination_stream.sql
├── README.md
└── sbs.py


/manifest.json:
--------------------------------------------------------------------------------
 1 | { 
 2 |     "fileLocations": [                                                    
 3 |               {"URIPrefixes": ["https://s3.amazonaws.com/<YOUR_BUCKET_NAME>/data/<YEAR>/<MONTH>/<DATE>/<HOUR>/"]}
 4 |      ],
 5 |      "globalUploadSettings": { 
 6 |      "format": "CSV",  
 7 |      "delimiter": ","
 8 |     }
 9 | }
10 | 
11 | 
12 | 


--------------------------------------------------------------------------------
/random.json:
--------------------------------------------------------------------------------
1 | {"deviceParameter": "Temperature", "deviceValue": 33, "deviceId": "SBS01", "dateTime": "2017-02-03 11:29:37"}
2 | {"deviceParameter": "Sound", "deviceValue": 140, "deviceId": "SBS03", "dateTime": "2017-02-03 11:29:38"}
3 | {"deviceParameter": "Humidity", "deviceValue": 63, "deviceId": "SBS01", "dateTime": "2017-02-03 11:29:39"}
4 | {"deviceParameter": "Flow", "deviceValue": 80, "deviceId": "SBS04", "dateTime": "2017-02-03 11:29:41"}
5 | 
6 | 
7 | 


--------------------------------------------------------------------------------
/LICENSE:
--------------------------------------------------------------------------------
 1 | MIT License
 2 | 
 3 | Copyright (c) 2020 Adit Modi
 4 | Permission is hereby granted, free of charge, to any person obtaining a copy
 5 | of this software and associated documentation files (the "Software"), to deal
 6 | in the Software without restriction, including without limitation the rights
 7 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
 8 | copies of the Software, and to permit persons to whom the Software is
 9 | furnished to do so, subject to the following conditions:
10 | 
11 | The above copyright notice and this permission notice shall be included in all
12 | copies or substantial portions of the Software.
13 | 
14 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
15 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
16 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
17 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
18 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
19 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
20 | SOFTWARE.
21 | 


--------------------------------------------------------------------------------
/SQL_Destination_stream.sql:
--------------------------------------------------------------------------------
 1 | CREATE OR REPLACE STREAM "DESTINATION_SQL_BASIC_STREAM" (dateTime TIMESTAMP, deviceId VARCHAR(8), deviceParameter VARCHAR(16), deviceValue INTEGER);
 2 | 
 3 | 
 4 | 
 5 | -- Create a pump that continuously selects from the source stream and inserts it into the output data stream
 6 | CREATE OR REPLACE PUMP "STREAM_PUMP_1" AS INSERT INTO "DESTINATION_SQL_BASIC_STREAM"
 7 | 
 8 | -- Filter specific columns from the source stream
 9 | SELECT STREAM "dateTime", "deviceId", "deviceParameter", "deviceValue" FROM "SOURCE_SQL_STREAM_001";
10 | 
11 | -- Create a second output stream with three columns, which is used to send aggregated min/max data to the destination
12 | CREATE OR REPLACE STREAM "DESTINATION_SQL_AGGREGATE_STREAM" (dateTime TIMESTAMP, highestTemp SMALLINT, lowestTemp SMALLINT);
13 | 
14 | -- Create a pump that continuously selects from a source stream 
15 | CREATE OR REPLACE PUMP "STREAM_PUMP_2" AS INSERT INTO "DESTINATION_SQL_AGGREGATE_STREAM"
16 | 
17 | 
18 | -- Extract time in minutes, plus the highest and lowest value of device temperature in that minute, into the destination aggregate stream, aggregated per minute
19 | SELECT STREAM FLOOR("SOURCE_SQL_STREAM_001".ROWTIME TO MINUTE) AS "dateTime", MAX("deviceValue") AS "highestTemp", MIN("deviceValue") AS "lowestTemp" FROM "SOURCE_SQL_STREAM_001" WHERE "deviceParameter"='Temperature' GROUP BY FLOOR("SOURCE_SQL_STREAM_001".ROWTIME TO MINUTE);
20 | 


--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
 1 | # IoT-Data-with-Amazon-Kinesis
 2 | 
 3 | 
 4 | Build a Visualization and Monitoring Dashboard for IoT Data with Amazon Kinesis Analytics and Amazon QuickSight
 5 | 
 6 | 
 7 | ![image](https://user-images.githubusercontent.com/48589838/77183351-357a6680-6af4-11ea-92a6-8f86db978554.png)
 8 | 
 9 | Implementation
10 | AWS IoT, Amazon Kinesis, and Amazon QuickSight are all fully managed services, which means you can complete the entire setup in just a few steps using the AWS Management Console.
11 | 
12 | 
13 | ### Step 1. Set up your AWS IoT data source
14 | 
15 | 
16 | Generate sample Data
17 | Running the sbs.py Python script generates fictitious AWS IoT messages from multiple SBS devices. The IoT rule sends the message to Firehose for further processing.
18 | 
19 | 
20 | ### Step 2. Create three Firehose delivery streams
21 | 
22 | you require three Firehose delivery streams:  one to batch raw data from AWS IoT, and two to batch output device data and aggregated data from Analytics.
23 | 
24 | 
25 | ### Step 3. Set up AWS IoT to receive and forward incoming data
26 | 
27 | 
28 | ### Step 4: Create an Analytics application to process data
29 | 
30 | 
31 | sample SQL query that generates two output streams:
32 | 
33 | DESTINATION_SQL_BASIC_STREAM contains the device ID, device parameter, its value, and the time stamp from the incoming stream.
34 | DESTINATION_SQL_AGGREGATE_STREAM aggregates the maximum and minimum values of temperatures from the sensors over a one-minute period from the incoming data.
35 | 
36 | 
37 | ### Step 5: Connect the Analytics application to output Firehose delivery streams
38 | 
39 | 
40 | ### Step 6: Set up Amazon QuickSight to analyze the data
41 | 
42 | 
43 | To build the visualization dashboard, ingest the processed CSV files from the S3 bucket into Amazon QuickSight.
44 | 
45 | ![image](https://user-images.githubusercontent.com/48589838/77183723-cf421380-6af4-11ea-963a-f090a624b3b9.png)
46 | ![image](https://user-images.githubusercontent.com/48589838/77183746-d701b800-6af4-11ea-9884-18174f9d6803.png)
47 | 
48 | 
49 | 
50 | ###### reference material:https://aws.amazon.com/blogs/big-data/build-a-visualization-and-monitoring-dashboard-for-iot-data-with-amazon-kinesis-analytics-and-amazon-quicksight/
51 | 


--------------------------------------------------------------------------------
/sbs.py:
--------------------------------------------------------------------------------
 1 | # Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
 2 | # SPDX-License-Identifier: MIT-0
 3 |     
 4 |   
 5 | # Script to generate simulated IoT device parameters data
 6 | import json
 7 | import random
 8 | import datetime
 9 | import boto3
10 | import time
11 | 
12 | deviceNames = ['SBS01', 'SBS02', 'SBS03', 'SBS04', 'SBS05']
13 | 
14 | iot = boto3.client('iot-data');
15 | 
16 | # generate Flow values
17 | def getFlowValues():
18 |     data = {}
19 |     data['deviceValue'] = random.randint(60, 100)
20 |     data['deviceParameter'] = 'Flow'
21 |     data['deviceId'] = random.choice(deviceNames)
22 |     data['dateTime'] = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")
23 |     return data
24 | 
25 | # generate Temperature values
26 | def getTemperatureValues():
27 |     data = {}
28 |     data['deviceValue'] = random.randint(15, 35)
29 |     data['deviceParameter'] = 'Temperature'
30 |     data['deviceId'] = random.choice(deviceNames)
31 |     data['dateTime'] = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")
32 |     return data
33 | 
34 | # generate Humidity values
35 | def getHumidityValues():
36 |     data = {}
37 |     data['deviceValue'] = random.randint(50, 90)
38 |     data['deviceParameter'] = 'Humidity'
39 |     data['deviceId'] = random.choice(deviceNames)
40 |     data['dateTime'] = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")
41 |     return data
42 | 
43 | # generate Sound values
44 | def getSoundValues():
45 |     data = {}
46 |     data['deviceValue'] = random.randint(100, 140)
47 |     data['deviceParameter'] = 'Sound'
48 |     data['deviceId'] = random.choice(deviceNames)
49 |     data['dateTime'] = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")
50 |     return data
51 | 
52 | # Generate each parameter's data input in varying proportions
53 | while True:
54 |     time.sleep(1)
55 |     rnd = random.random()
56 |     if (0 <= rnd < 0.20):
57 |         data = json.dumps(getFlowValues())
58 |         print data
59 |         response = iot.publish(
60 |              topic='/sbs/devicedata/flow',
61 |              payload=data
62 |         ) 
63 |     elif (0.20<= rnd < 0.55):
64 |         data = json.dumps(getTemperatureValues())
65 |         print data
66 |         response = iot.publish(
67 |              topic='/sbs/devicedata/temperature',
68 |              payload=data
69 |         )
70 |     elif (0.55<= rnd < 0.70):
71 |         data = json.dumps(getHumidityValues())
72 |         print data
73 |         response = iot.publish(
74 |              topic='/sbs/devicedata/humidity',
75 |              payload=data
76 |         )
77 |     else:
78 |         data = json.dumps(getSoundValues())
79 |         print data
80 |         response = iot.publish(
81 |              topic='/sbs/devicedata/sound',
82 |              payload=data     
83 | )
84 | 


--------------------------------------------------------------------------------