├── LICENSE.txt ├── NOTICE.txt ├── README.md ├── cform ├── add-output.template ├── streaming-analytics-pipeline-config.yaml └── streaming-analytics-pipeline.template └── code ├── add-output-code.zip └── streaming-analytics-code.zip /LICENSE.txt: -------------------------------------------------------------------------------- 1 | Amazon Software License 2 | This Amazon Software License (“License”) governs your use, reproduction, and distribution of the accompanying software as specified below. 3 | 1. Definitions 4 | “Licensor” means any person or entity that distributes its Work. 5 | 6 | “Software” means the original work of authorship made available under this License. 7 | 8 | “Work” means the Software and any additions to or derivative works of the Software that are made available under this License. 9 | 10 | The terms “reproduce,” “reproduction,” “derivative works,” and “distribution” have the meaning as provided under U.S. copyright law; provided, however, that for the purposes of this License, derivative works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work. 11 | 12 | Works, including the Software, are “made available” under this License by including in or with the Work either (a) a copyright notice referencing the applicability of this License to the Work, or (b) a copy of this License. 13 | 2. License Grants 14 | 2.1 Copyright Grant. Subject to the terms and conditions of this License, each Licensor grants to you a perpetual, worldwide, non-exclusive, royalty-free, copyright license to reproduce, prepare derivative works of, publicly display, publicly perform, sublicense and distribute its Work and any resulting derivative works in any form. 15 | 2.2 Patent Grant. Subject to the terms and conditions of this License, each Licensor grants to you a perpetual, worldwide, non-exclusive, royalty-free patent license to make, have made, use, sell, offer for sale, import, and otherwise transfer its Work, in whole or in part. The foregoing license applies only to the patent claims licensable by Licensor that would be infringed by Licensor’s Work (or portion thereof) individually and excluding any combinations with any other materials or technology. 16 | 3. Limitations 17 | 3.1 Redistribution. You may reproduce or distribute the Work only if (a) you do so under this License, (b) you include a complete copy of this License with your distribution, and (c) you retain without modification any copyright, patent, trademark, or attribution notices that are present in the Work. 18 | 3.2 Derivative Works. You may specify that additional or different terms apply to the use, reproduction, and distribution of your derivative works of the Work (“Your Terms”) only if (a) Your Terms provide that the use limitation in Section 3.3 applies to your derivative works, and (b) you identify the specific derivative works that are subject to Your Terms. Notwithstanding Your Terms, this License (including the redistribution requirements in Section 3.1) will continue to apply to the Work itself. 19 | 3.3 Use Limitation. The Work and any derivative works thereof only may be used or intended for use with the web services, computing platforms or applications provided by Amazon.com, Inc. or its affiliates, including Amazon Web Services, Inc. 20 | 3.4 Patent Claims. If you bring or threaten to bring a patent claim against any Licensor (including any claim, cross-claim or counterclaim in a lawsuit) to enforce any patents that you allege are infringed by any Work, then your rights under this License from such Licensor (including the grants in Sections 2.1 and 2.2) will terminate immediately. 21 | 3.5 Trademarks. This License does not grant any rights to use any Licensor’s or its affiliates’ names, logos, or trademarks, except as necessary to reproduce the notices described in this License. 22 | 3.6 Termination. If you violate any term of this License, then your rights under this License (including the grants in Sections 2.1 and 2.2) will terminate immediately. 23 | 4. Disclaimer of Warranty. 24 | THE WORK IS PROVIDED “AS IS” WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WARRANTIES OR CONDITIONS OF M ERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE OR NON-INFRINGEMENT. YOU BEAR THE RISK OF UNDERTAKING ANY ACTIVITIES UNDER THIS LICENSE. SOME STATES’ CONSUMER LAWS DO NOT ALLOW EXCLUSION OF AN IMPLIED WARRANTY, SO THIS DISCLAIMER MAY NOT APPLY TO YOU. 25 | 5. Limitation of Liability. 26 | EXCEPT AS PROHIBITED BY APPLICABLE LAW, IN NO EVENT AND UNDER NO LEGAL THEORY, WHETHER IN TORT (INCLUDING NEGLIGENCE), CONTRACT, OR OTHERWISE SHALL ANY LICENSOR BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES ARISING OUT OF OR RELATED TO THIS LICENSE, THE USE OR INABILITY TO USE THE WORK (INCLUDING BUT NOT LIMITED TO LOSS OF GOODWILL, BUSINESS INTERRUPTION, LOST PROFITS OR DATA, COMPUTER FAILURE OR MALFUNCTION, OR ANY OTHER COMM ERCIAL DAMAGES OR LOSSES), EVEN IF THE LICENSOR HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. 27 | Effective Date – April 18, 2008 © 2008 Amazon.com, Inc. or its affiliates. All rights reserved. 28 | 29 | Note: Other license terms may apply to certain, identified software files contained within or 30 | distributed with the accompanying software if such terms are included in the directory containing 31 | the accompanying software. Such other license terms will then apply in lieu of the terms of the 32 | software license above. -------------------------------------------------------------------------------- /NOTICE.txt: -------------------------------------------------------------------------------- 1 | Streaming Analytics Pipeline 2 | 3 | Copyright 2016 Amazon.com, Inc. or its affiliates. All Rights Reserved. 4 | Licensed under the Amazon Software License (the "License"). You may not use this file except 5 | in compliance with the License. A copy of the License is located at http://aws.amazon.com/asl/ 6 | or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, 7 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, express or implied. See the License for the 8 | specific language governing permissions and limitations under the License. 9 | 10 | ********************** 11 | THIRD PARTY COMPONENTS 12 | ********************** 13 | This software includes third party software subject to the following copyrights: 14 | 15 | simplejson under the Massachusetts Institute of Technology (MIT) license 16 | PyYAML under the Massachusetts Institute of Technology (MIT) license 17 | requests under the Apache License Version 2.0 -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Streaming Analytics Pipeline 2 | Many Amazon Web Services (AWS) customers use streaming data to gain real-time insight into customer activity and business trends. Streaming data is generated continuously from thousands of data sources, and this information can help companies make well-informed decisions and proactively respond to changing conditions. Amazon Kinesis is a platform for streaming data on AWS, offering powerful services that make it easy to build data processing applications, load massive volumes of data from hundreds of thousands of sources, and analyze streaming data in real time. 3 | 4 | To help customers easily configure a streaming data architecture, AWS offers the Streaming Analytics Pipeline. This solution automatically provisions and configures the AWS services necessary to start consuming and analyzing streaming data in minutes. This solution uses Amazon Kinesis Streams to load streaming data, Amazon Kinesis Analytics to filter and process that data, and Amazon Kinesis Firehose to deliver analyzed data to various data stores for search, storage, or further analytics. 5 | 6 | For the full solution overview visit [Streaming Analytics Pipeline](https://aws.amazon.com/answers/big-data/streaming-analytics-pipeline). 7 | 8 | ## Cloudformation templates 9 | - cform/streaming-analytics-pipeline.template 10 | - cform/add-output.template 11 | - cform/streaming-analytics-pipeline-config.yaml 12 | 13 | ## Solution code 14 | - code/streaming-analytics-code.zip 15 | - code/add-output-code.zip 16 | 17 | *** 18 | 19 | Copyright 2016 Amazon.com, Inc. or its affiliates. All Rights Reserved. 20 | 21 | Licensed under the Amazon Software License (the "License"). You may not use this file except in compliance with the License. A copy of the License is located at 22 | 23 | http://aws.amazon.com/asl/ 24 | 25 | or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions and limitations under the License. -------------------------------------------------------------------------------- /cform/add-output.template: -------------------------------------------------------------------------------- 1 | { 2 | "AWSTemplateFormatVersion": "2010-09-09", 3 | "Description": "(SO0014)- Streaming Analytics Pipeline: this template adds an additional external destination to an existing Amazon Kinesis Analytics application.", 4 | "Metadata": { 5 | "AWS::CloudFormation::Interface": { 6 | "ParameterGroups": [ 7 | { 8 | "Label": { 9 | "default": "Amazon Kinesis Analytics Configuration" 10 | }, 11 | "Parameters": [ 12 | "ApplicationName", 13 | "InAppDestStream", 14 | "ExternalDestination", 15 | "EncryptData" 16 | ] 17 | }, 18 | { 19 | "Label": { 20 | "default": "Amazon Redshift Destination Configuration" 21 | }, 22 | "Parameters": [ 23 | "MasterUser", 24 | "MasterUserPassword", 25 | "RedshiftJDBCURL", 26 | "RedshiftTableName", 27 | "RedshiftColumns", 28 | "RedshiftBufferInterval", 29 | "RedshiftBufferSize" 30 | ] 31 | }, 32 | { 33 | "Label": { 34 | "default": "Amazon Elasticsearch Destination Configuration" 35 | }, 36 | "Parameters": [ 37 | "ESDomain", 38 | "ESIndex", 39 | "ESType", 40 | "ESIndexRotation", 41 | "ESBufferInterval", 42 | "ESBufferSize" 43 | ] 44 | }, 45 | { 46 | "Label": { 47 | "default": "Amazon S3 Destination Configuration" 48 | }, 49 | "Parameters": [ 50 | "S3BufferInterval", 51 | "S3BufferSize", 52 | "S3DestinationPrefix" 53 | ] 54 | }, 55 | { 56 | "Label": { 57 | "default": "Amazon Kinesis Stream Destination Configuration" 58 | }, 59 | "Parameters": [ 60 | "DestinationStream" 61 | ] 62 | } 63 | ], 64 | "ParameterLabels": { 65 | "ApplicationName": { 66 | "default": "Application Name" 67 | }, 68 | "InAppDestStream": { 69 | "default": "In-application Output Stream Name" 70 | }, 71 | "ExternalDestination": { 72 | "default": "External Destination" 73 | }, 74 | "EncryptData": { 75 | "default": "Encrypt Data at Rest?" 76 | }, 77 | "MasterUser": { 78 | "default": "Master User Name" 79 | }, 80 | "MasterUserPassword": { 81 | "default": "Master User Password" 82 | }, 83 | "RedshiftJDBCURL": { 84 | "default": "JDBC URL" 85 | }, 86 | "RedshiftTableName": { 87 | "default": "Table Name" 88 | }, 89 | "RedshiftColumns": { 90 | "default": "Column Pattern" 91 | }, 92 | "RedshiftBufferInterval": { 93 | "default": "Buffer Interval" 94 | }, 95 | "RedshiftBufferSize": { 96 | "default": "Buffer Size" 97 | }, 98 | "ESDomain": { 99 | "default": "Domain Name" 100 | }, 101 | "ESIndex": { 102 | "default": "Index Name" 103 | }, 104 | "ESType": { 105 | "default": "Type Name" 106 | }, 107 | "ESIndexRotation": { 108 | "default": "Index Rotation" 109 | }, 110 | "ESBufferInterval": { 111 | "default": "Buffer Interval" 112 | }, 113 | "ESBufferSize": { 114 | "default": "Buffer Size" 115 | }, 116 | "S3BufferInterval": { 117 | "default": "Buffer Interval" 118 | }, 119 | "S3BufferSize": { 120 | "default": "Buffer Size" 121 | }, 122 | "S3DestinationPrefix": { 123 | "default": "Destination Prefix" 124 | }, 125 | "DestinationStream": { 126 | "default": "Destination Stream Name" 127 | } 128 | } 129 | } 130 | }, 131 | "Parameters": { 132 | "ApplicationName": { 133 | "Description": "Name of your existing Amazon Kinesis Analytics application.", 134 | "Type": "String" 135 | }, 136 | "InAppDestStream": { 137 | "Description": "Name of in-application output stream.", 138 | "Type": "String", 139 | "Default": "DESTINATION_SQL_STREAM_002" 140 | }, 141 | "S3DestinationPrefix": { 142 | "Description": "Name of prefix where the aggregated data will be stored.", 143 | "Type": "String", 144 | "Default": "AggregatedData" 145 | }, 146 | "ExternalDestination": { 147 | "Description": "Select a destination for your aggregated data.", 148 | "Type": "String", 149 | "AllowedValues": [ 150 | "Amazon Redshift", 151 | "Amazon ElasticSearch Service", 152 | "Amazon S3", 153 | "Amazon Kinesis Stream" 154 | ] 155 | }, 156 | "S3BufferInterval": { 157 | "Description": "Number of seconds to buffer data before delivering to S3 (60 to 900).", 158 | "Type": "Number", 159 | "Default": 300, 160 | "MinValue": 60, 161 | "MaxValue": 900 162 | }, 163 | "S3BufferSize": { 164 | "Description": "Number of MB of data to buffer before delivering to S3 (1 to 128).", 165 | "Type": "Number", 166 | "Default": 5, 167 | "MinValue": 1, 168 | "MaxValue": 128 169 | }, 170 | "EncryptData": { 171 | "Description": "Would you like to encrypt the data as it leaves your Kinesis Analytics application.", 172 | "Type": "String", 173 | "AllowedValues": [ 174 | "yes", 175 | "no" 176 | ] 177 | }, 178 | "ESDomain": { 179 | "Description": "Name of the destination Amazon Elasticsearch Domain.", 180 | "Type": "String" 181 | }, 182 | "ESIndex": { 183 | "Description": "Index name of the Amazon ElasticSearch Domain.", 184 | "Type": "String" 185 | }, 186 | "ESType": { 187 | "Description": "Name of the Amazon ElasticSearch type.", 188 | "Type": "String" 189 | }, 190 | "ESIndexRotation": { 191 | "Description": "Frequency at which the Amazon ElasticSearch index will be rotated.", 192 | "Type": "String", 193 | "AllowedValues": [ 194 | "NoRotation", 195 | "OneHour", 196 | "OneDay", 197 | "OneWeek", 198 | "OneMonth" 199 | ], 200 | "Default": "NoRotation" 201 | }, 202 | "ESBufferInterval": { 203 | "Description": "Number of seconds to buffer data before delivering to S3 to be copied to ElasticSearch (60 to 900).", 204 | "Type": "Number", 205 | "Default": 300, 206 | "MinValue": 60, 207 | "MaxValue": 900 208 | }, 209 | "ESBufferSize": { 210 | "Description": "Number of MB of data to buffer before delivering to S3 to be copied to ElasticSearch (1 to 100).", 211 | "Type": "Number", 212 | "Default": 5, 213 | "MinValue": 1, 214 | "MaxValue": 100 215 | }, 216 | "MasterUserPassword": { 217 | "Description": "Master user password of your preexisting Redshift cluster", 218 | "NoEcho": "true", 219 | "Type": "String" 220 | }, 221 | "MasterUser": { 222 | "Description": "Name of the master user of the preexisting Redshift Cluster", 223 | "Type": "String" 224 | }, 225 | "RedshiftBufferInterval": { 226 | "Description": "Number of seconds to buffer data before delivering to S3 (60 to 900).", 227 | "Type": "Number", 228 | "Default": 300, 229 | "MinValue": 60, 230 | "MaxValue": 900 231 | }, 232 | "RedshiftBufferSize": { 233 | "Description": "Number of MB of data to buffer before delivering to S3 (1 to 128).", 234 | "Type": "Number", 235 | "Default": 5, 236 | "MinValue": 1, 237 | "MaxValue": 128 238 | }, 239 | "RedshiftJDBCURL": { 240 | "Description": "URL to connect to the cluster database with the JDBC client", 241 | "Type": "String" 242 | }, 243 | "RedshiftTableName": { 244 | "Description": "Name of table in preexisting Redshift Cluster", 245 | "Type": "String" 246 | }, 247 | "RedshiftColumns": { 248 | "Description": "Comma-separated list of the columns in the destination Redshift table.", 249 | "Type": "String" 250 | }, 251 | "DestinationStream": { 252 | "Description": "Name of the existing destination Kinesis Stream", 253 | "Type": "String" 254 | } 255 | }, 256 | "Mappings": {}, 257 | "Conditions": { 258 | "NeedS3Bucket": { 259 | "Fn::Or": [ 260 | { 261 | "Fn::Equals": [ 262 | { 263 | "Ref": "ExternalDestination" 264 | }, 265 | "Amazon Redshift" 266 | ] 267 | }, 268 | { 269 | "Fn::Equals": [ 270 | { 271 | "Ref": "ExternalDestination" 272 | }, 273 | "Amazon ElasticSearch Service" 274 | ] 275 | }, 276 | { 277 | "Fn::Equals": [ 278 | { 279 | "Ref": "ExternalDestination" 280 | }, 281 | "Amazon S3" 282 | ] 283 | } 284 | ] 285 | }, 286 | "EncryptData": { 287 | "Fn::Equals": [ 288 | { 289 | "Ref": "EncryptData" 290 | }, 291 | "yes" 292 | ] 293 | }, 294 | "NoEncryption": { 295 | "Fn::Equals": [ 296 | { 297 | "Ref": "EncryptData" 298 | }, 299 | "no" 300 | ] 301 | }, 302 | "RedshiftDestination": { 303 | "Fn::Equals": [ 304 | { 305 | "Ref": "ExternalDestination" 306 | }, 307 | "Amazon Redshift" 308 | ] 309 | }, 310 | "ElasticsearchDestination": { 311 | "Fn::Equals": [ 312 | { 313 | "Ref": "ExternalDestination" 314 | }, 315 | "Amazon ElasticSearch Service" 316 | ] 317 | }, 318 | "S3Destination": { 319 | "Fn::Equals": [ 320 | { 321 | "Ref": "ExternalDestination" 322 | }, 323 | "Amazon S3" 324 | ] 325 | }, 326 | "StreamDestination": { 327 | "Fn::Equals": [ 328 | { 329 | "Ref": "ExternalDestination" 330 | }, 331 | "Amazon Kinesis Stream" 332 | ] 333 | } 334 | }, 335 | "Resources": { 336 | "EncryptionKey": { 337 | "Type": "AWS::KMS::Key", 338 | "Condition": "EncryptData", 339 | "Properties": { 340 | "Description": "KMS key generated to encrypt aggregated Kinesis Analytics data.", 341 | "EnableKeyRotation": "true", 342 | "KeyPolicy": { 343 | "Id": "KMS key policy", 344 | "Version": "2012-10-17", 345 | "Statement": [ 346 | { 347 | "Sid": "Enable IAM User Permissions", 348 | "Effect": "Allow", 349 | "Principal": { 350 | "AWS": [ 351 | { 352 | "Fn::Join": [ 353 | "", 354 | [ 355 | "arn:aws:iam::", 356 | { 357 | "Ref": "AWS::AccountId" 358 | }, 359 | ":root" 360 | ] 361 | ] 362 | } 363 | ] 364 | }, 365 | "Action": "kms:*", 366 | "Resource": "*" 367 | }, 368 | { 369 | "Sid": "Allow access for Key Administrators", 370 | "Effect": "Allow", 371 | "Principal": { 372 | "AWS": [] 373 | }, 374 | "Action": [ 375 | "kms:Create*", 376 | "kms:Describe*", 377 | "kms:Enable*", 378 | "kms:List*", 379 | "kms:Put*", 380 | "kms:Update*", 381 | "kms:Revoke*", 382 | "kms:Disable*", 383 | "kms:Get*", 384 | "kms:Delete*", 385 | "kms:ScheduleKeyDeletion", 386 | "kms:CancelKeyDeletion" 387 | ], 388 | "Resource": "*" 389 | }, 390 | { 391 | "Sid": "Allow use of the key", 392 | "Effect": "Allow", 393 | "Principal": { 394 | "AWS": [] 395 | }, 396 | "Action": [ 397 | "kms:Encrypt", 398 | "kms:Decrypt", 399 | "kms:ReEncrypt*", 400 | "kms:GenerateDataKey*", 401 | "kms:DescribeKey" 402 | ], 403 | "Resource": "*" 404 | }, 405 | { 406 | "Sid": "Allow attachment of persistent resources", 407 | "Effect": "Allow", 408 | "Principal": { 409 | "AWS": [] 410 | }, 411 | "Action": [ 412 | "kms:CreateGrant", 413 | "kms:ListGrants", 414 | "kms:RevokeGrant" 415 | ], 416 | "Resource": "*", 417 | "Condition": { 418 | "Bool": { 419 | "kms:GrantIsForAWSResource": true 420 | } 421 | } 422 | } 423 | ] 424 | } 425 | } 426 | }, 427 | "KMSAlias": { 428 | "Type": "AWS::KMS::Alias", 429 | "Condition": "EncryptData", 430 | "Properties": { 431 | "AliasName": { 432 | "Fn::Join": [ 433 | "", 434 | [ 435 | "alias/key-", 436 | { 437 | "Ref": "AWS::StackName" 438 | } 439 | ] 440 | ] 441 | }, 442 | "TargetKeyId": { 443 | "Fn::GetAtt": [ 444 | "EncryptionKey", 445 | "Arn" 446 | ] 447 | } 448 | } 449 | }, 450 | "StreamingAnalyticsBucket": { 451 | "Type": "AWS::S3::Bucket", 452 | "Condition": "NeedS3Bucket" 453 | }, 454 | "FIrehoseToS3Role": { 455 | "Type": "AWS::IAM::Role", 456 | "Condition": "S3Destination", 457 | "Properties": { 458 | "AssumeRolePolicyDocument": { 459 | "Version": "2012-10-17", 460 | "Statement": [ 461 | { 462 | "Effect": "Allow", 463 | "Principal": { 464 | "Service": [ 465 | "firehose.amazonaws.com" 466 | ] 467 | }, 468 | "Action": [ 469 | "sts:AssumeRole" 470 | ], 471 | "Condition": { 472 | "StringEquals": { 473 | "sts:ExternalId": { 474 | "Ref": "AWS::AccountId" 475 | } 476 | } 477 | } 478 | } 479 | ] 480 | }, 481 | "Policies": [ 482 | { 483 | "PolicyName": { 484 | "Fn::Join": [ 485 | "", 486 | [ 487 | "Firehose-to-S3-Delivery-", 488 | { 489 | "Ref": "AWS::StackName" 490 | } 491 | ] 492 | ] 493 | }, 494 | "PolicyDocument": { 495 | "Version": "2012-10-17", 496 | "Statement": [ 497 | { 498 | "Sid": "", 499 | "Effect": "Allow", 500 | "Action": [ 501 | "s3:AbortMultipartUpload", 502 | "s3:GetBucketLocation", 503 | "s3:GetObject", 504 | "s3:ListBucket", 505 | "s3:ListBucketMultipartUploads", 506 | "s3:PutObject" 507 | ], 508 | "Resource": [ 509 | { 510 | "Fn::Join": [ 511 | "", 512 | [ 513 | "arn:aws:s3:::", 514 | { 515 | "Ref": "StreamingAnalyticsBucket" 516 | } 517 | ] 518 | ] 519 | }, 520 | { 521 | "Fn::Join": [ 522 | "", 523 | [ 524 | "arn:aws:s3:::", 525 | { 526 | "Ref": "StreamingAnalyticsBucket" 527 | }, 528 | "/*" 529 | ] 530 | ] 531 | } 532 | ] 533 | }, 534 | { 535 | "Effect": "Allow", 536 | "Action": [ 537 | "kms:Decrypt", 538 | "kms:GenerateDataKey" 539 | ], 540 | "Resource": [ 541 | { 542 | "Fn::If": [ 543 | "NoEncryption", 544 | { 545 | "Fn::Join": [ 546 | "", 547 | [ 548 | "arn:aws:kms:", 549 | { 550 | "Ref": "AWS::Region" 551 | }, 552 | ":", 553 | { 554 | "Ref": "AWS::AccountId" 555 | }, 556 | ":key/placeholder-kms-id" 557 | ] 558 | ] 559 | }, 560 | { 561 | "Fn::GetAtt": [ 562 | "EncryptionKey", 563 | "Arn" 564 | ] 565 | } 566 | ] 567 | } 568 | ], 569 | "Condition": { 570 | "StringEquals": { 571 | "kms:ViaService": { 572 | "Fn::Join": [ 573 | "", 574 | [ 575 | "s3.", 576 | { 577 | "Ref": "AWS::Region" 578 | }, 579 | ".amazonaws.com" 580 | ] 581 | ] 582 | } 583 | }, 584 | "StringLike": { 585 | "kms:EncryptionContext:aws:s3:arn": { 586 | "Fn::Join": [ 587 | "", 588 | [ 589 | "arn:aws:s3:::", 590 | { 591 | "Ref": "StreamingAnalyticsBucket" 592 | }, 593 | "/", 594 | { 595 | "Ref": "S3DestinationPrefix" 596 | }, 597 | "/*" 598 | ] 599 | ] 600 | } 601 | } 602 | } 603 | }, 604 | { 605 | "Sid": "", 606 | "Effect": "Allow", 607 | "Action": [ 608 | "logs:PutLogEvents" 609 | ], 610 | "Resource": [ 611 | "*" 612 | ] 613 | } 614 | ] 615 | } 616 | } 617 | ] 618 | } 619 | }, 620 | "FIrehoseToS3": { 621 | "Type": "AWS::KinesisFirehose::DeliveryStream", 622 | "Condition": "S3Destination", 623 | "Properties": { 624 | "S3DestinationConfiguration": { 625 | "BucketARN": { 626 | "Fn::Join": [ 627 | "", 628 | [ 629 | "arn:aws:s3:::", 630 | { 631 | "Ref": "StreamingAnalyticsBucket" 632 | } 633 | ] 634 | ] 635 | }, 636 | "BufferingHints": { 637 | "IntervalInSeconds": { 638 | "Ref": "S3BufferInterval" 639 | }, 640 | "SizeInMBs": { 641 | "Ref": "S3BufferSize" 642 | } 643 | }, 644 | "CompressionFormat": "UNCOMPRESSED", 645 | "EncryptionConfiguration": { 646 | "KMSEncryptionConfig": { 647 | "Fn::If": [ 648 | "NoEncryption", 649 | { 650 | "Ref": "AWS::NoValue" 651 | }, 652 | { 653 | "AWSKMSKeyARN": { 654 | "Fn::GetAtt": [ 655 | "EncryptionKey", 656 | "Arn" 657 | ] 658 | } 659 | } 660 | ] 661 | }, 662 | "NoEncryptionConfig": { 663 | "Fn::If": [ 664 | "NoEncryption", 665 | "NoEncryption", 666 | { 667 | "Ref": "AWS::NoValue" 668 | } 669 | ] 670 | } 671 | }, 672 | "Prefix": { 673 | "Fn::Join": [ 674 | "", 675 | [ 676 | { 677 | "Ref": "S3DestinationPrefix" 678 | }, 679 | "/" 680 | ] 681 | ] 682 | }, 683 | "RoleARN": { 684 | "Fn::GetAtt": [ 685 | "FIrehoseToS3Role", 686 | "Arn" 687 | ] 688 | } 689 | } 690 | } 691 | }, 692 | "ElasticSearchDeliveryRole": { 693 | "Type": "AWS::IAM::Role", 694 | "Condition": "ElasticsearchDestination", 695 | "Properties": { 696 | "AssumeRolePolicyDocument": { 697 | "Version": "2012-10-17", 698 | "Statement": [ 699 | { 700 | "Effect": "Allow", 701 | "Principal": { 702 | "Service": [ 703 | "firehose.amazonaws.com" 704 | ] 705 | }, 706 | "Action": [ 707 | "sts:AssumeRole" 708 | ], 709 | "Condition": { 710 | "StringEquals": { 711 | "sts:ExternalId": { 712 | "Ref": "AWS::AccountId" 713 | } 714 | } 715 | } 716 | } 717 | ] 718 | }, 719 | "Policies": [ 720 | { 721 | "PolicyName": { 722 | "Fn::Join": [ 723 | "", 724 | [ 725 | "Firehose-to-ElasticSearch-Delivery-", 726 | { 727 | "Ref": "AWS::StackName" 728 | } 729 | ] 730 | ] 731 | }, 732 | "PolicyDocument": { 733 | "Version": "2012-10-17", 734 | "Statement": [ 735 | { 736 | "Sid": "", 737 | "Effect": "Allow", 738 | "Action": [ 739 | "s3:AbortMultipartUpload", 740 | "s3:GetBucketLocation", 741 | "s3:GetObject", 742 | "s3:ListBucket", 743 | "s3:ListBucketMultipartUploads", 744 | "s3:PutObject" 745 | ], 746 | "Resource": [ 747 | { 748 | "Fn::Join": [ 749 | "", 750 | [ 751 | "arn:aws:s3:::", 752 | { 753 | "Ref": "StreamingAnalyticsBucket" 754 | } 755 | ] 756 | ] 757 | }, 758 | { 759 | "Fn::Join": [ 760 | "", 761 | [ 762 | "arn:aws:s3:::", 763 | { 764 | "Ref": "StreamingAnalyticsBucket" 765 | }, 766 | "/*" 767 | ] 768 | ] 769 | } 770 | ] 771 | }, 772 | { 773 | "Effect": "Allow", 774 | "Action": [ 775 | "kms:Decrypt", 776 | "kms:GenerateDataKey" 777 | ], 778 | "Resource": [ 779 | { 780 | "Fn::If": [ 781 | "NoEncryption", 782 | { 783 | "Fn::Join": [ 784 | "", 785 | [ 786 | "arn:aws:kms:", 787 | { 788 | "Ref": "AWS::Region" 789 | }, 790 | ":", 791 | { 792 | "Ref": "AWS::AccountId" 793 | }, 794 | ":key/placeholder-kms-id" 795 | ] 796 | ] 797 | }, 798 | { 799 | "Fn::GetAtt": [ 800 | "EncryptionKey", 801 | "Arn" 802 | ] 803 | } 804 | ] 805 | } 806 | ], 807 | "Condition": { 808 | "StringEquals": { 809 | "kms:ViaService": { 810 | "Fn::Join": [ 811 | "", 812 | [ 813 | "s3.", 814 | { 815 | "Ref": "AWS::Region" 816 | }, 817 | ".amazonaws.com" 818 | ] 819 | ] 820 | } 821 | }, 822 | "StringLike": { 823 | "kms:EncryptionContext:aws:s3:arn": { 824 | "Fn::Join": [ 825 | "", 826 | [ 827 | "arn:aws:s3:::", 828 | { 829 | "Ref": "StreamingAnalyticsBucket" 830 | }, 831 | "/ESDeliveryBackup/*" 832 | ] 833 | ] 834 | } 835 | } 836 | } 837 | }, 838 | { 839 | "Sid": "", 840 | "Effect": "Allow", 841 | "Action": [ 842 | "es:DescribeElasticsearchDomain", 843 | "es:DescribeElasticsearchDomains", 844 | "es:DescribeElasticsearchDomainConfig", 845 | "es:ESHttpPost", 846 | "es:ESHttpPut" 847 | ], 848 | "Resource": [ 849 | { 850 | "Fn::Join": [ 851 | "", 852 | [ 853 | "arn:aws:es:", 854 | { 855 | "Ref": "AWS::Region" 856 | }, 857 | ":", 858 | { 859 | "Ref": "AWS::AccountId" 860 | }, 861 | ":domain/", 862 | { 863 | "Ref": "ESDomain" 864 | } 865 | ] 866 | ] 867 | }, 868 | { 869 | "Fn::Join": [ 870 | "", 871 | [ 872 | "arn:aws:es:", 873 | { 874 | "Ref": "AWS::Region" 875 | }, 876 | ":", 877 | { 878 | "Ref": "AWS::AccountId" 879 | }, 880 | ":domain/", 881 | { 882 | "Ref": "ESDomain" 883 | }, 884 | "/*" 885 | ] 886 | ] 887 | } 888 | ] 889 | }, 890 | { 891 | "Sid": "", 892 | "Effect": "Allow", 893 | "Action": [ 894 | "logs:PutLogEvents" 895 | ], 896 | "Resource": [ 897 | "*" 898 | ] 899 | } 900 | ] 901 | } 902 | } 903 | ] 904 | } 905 | }, 906 | "FIrehoseToElasticsearch": { 907 | "Type": "AWS::KinesisFirehose::DeliveryStream", 908 | "Condition": "ElasticsearchDestination", 909 | "Properties": { 910 | "ElasticsearchDestinationConfiguration": { 911 | "BufferingHints": { 912 | "IntervalInSeconds": { 913 | "Ref": "ESBufferInterval" 914 | }, 915 | "SizeInMBs": { 916 | "Ref": "ESBufferSize" 917 | } 918 | }, 919 | "DomainARN": { 920 | "Fn::Join": [ 921 | "", 922 | [ 923 | "arn:aws:es:", 924 | { 925 | "Ref": "AWS::Region" 926 | }, 927 | ":", 928 | { 929 | "Ref": "AWS::AccountId" 930 | }, 931 | ":domain/", 932 | { 933 | "Ref": "ESDomain" 934 | } 935 | ] 936 | ] 937 | }, 938 | "IndexName": { 939 | "Ref": "ESIndex" 940 | }, 941 | "IndexRotationPeriod": { 942 | "Ref": "ESIndexRotation" 943 | }, 944 | "RetryOptions": { 945 | "DurationInSeconds": 300 946 | }, 947 | "RoleARN": { 948 | "Fn::GetAtt": [ 949 | "ElasticSearchDeliveryRole", 950 | "Arn" 951 | ] 952 | }, 953 | "S3BackupMode": "AllDocuments", 954 | "S3Configuration": { 955 | "BucketARN": { 956 | "Fn::Join": [ 957 | "", 958 | [ 959 | "arn:aws:s3:::", 960 | { 961 | "Ref": "StreamingAnalyticsBucket" 962 | } 963 | ] 964 | ] 965 | }, 966 | "BufferingHints": { 967 | "IntervalInSeconds": { 968 | "Ref": "ESBufferInterval" 969 | }, 970 | "SizeInMBs": { 971 | "Ref": "ESBufferSize" 972 | } 973 | }, 974 | "CompressionFormat": "UNCOMPRESSED", 975 | "EncryptionConfiguration": { 976 | "KMSEncryptionConfig": { 977 | "Fn::If": [ 978 | "NoEncryption", 979 | { 980 | "Ref": "AWS::NoValue" 981 | }, 982 | { 983 | "AWSKMSKeyARN": { 984 | "Fn::GetAtt": [ 985 | "EncryptionKey", 986 | "Arn" 987 | ] 988 | } 989 | } 990 | ] 991 | }, 992 | "NoEncryptionConfig": { 993 | "Fn::If": [ 994 | "NoEncryption", 995 | "NoEncryption", 996 | { 997 | "Ref": "AWS::NoValue" 998 | } 999 | ] 1000 | } 1001 | }, 1002 | "Prefix": "ESDeliveryBackup/", 1003 | "RoleARN": { 1004 | "Fn::GetAtt": [ 1005 | "ElasticSearchDeliveryRole", 1006 | "Arn" 1007 | ] 1008 | } 1009 | }, 1010 | "TypeName": { 1011 | "Ref": "ESType" 1012 | } 1013 | } 1014 | } 1015 | }, 1016 | "RedshiftDeliveryRole": { 1017 | "Type": "AWS::IAM::Role", 1018 | "Condition": "RedshiftDestination", 1019 | "Properties": { 1020 | "AssumeRolePolicyDocument": { 1021 | "Version": "2012-10-17", 1022 | "Statement": [ 1023 | { 1024 | "Effect": "Allow", 1025 | "Principal": { 1026 | "Service": [ 1027 | "firehose.amazonaws.com" 1028 | ] 1029 | }, 1030 | "Action": [ 1031 | "sts:AssumeRole" 1032 | ], 1033 | "Condition": { 1034 | "StringEquals": { 1035 | "sts:ExternalId": { 1036 | "Ref": "AWS::AccountId" 1037 | } 1038 | } 1039 | } 1040 | } 1041 | ] 1042 | }, 1043 | "Policies": [ 1044 | { 1045 | "PolicyName": { 1046 | "Fn::Join": [ 1047 | "", 1048 | [ 1049 | "Firehose-to-Redshift-Delivery-", 1050 | { 1051 | "Ref": "AWS::StackName" 1052 | } 1053 | ] 1054 | ] 1055 | }, 1056 | "PolicyDocument": { 1057 | "Version": "2012-10-17", 1058 | "Statement": [ 1059 | { 1060 | "Sid": "", 1061 | "Effect": "Allow", 1062 | "Action": [ 1063 | "s3:AbortMultipartUpload", 1064 | "s3:GetBucketLocation", 1065 | "s3:GetObject", 1066 | "s3:ListBucket", 1067 | "s3:ListBucketMultipartUploads", 1068 | "s3:PutObject" 1069 | ], 1070 | "Resource": [ 1071 | { 1072 | "Fn::Join": [ 1073 | "", 1074 | [ 1075 | "arn:aws:s3:::", 1076 | { 1077 | "Ref": "StreamingAnalyticsBucket" 1078 | } 1079 | ] 1080 | ] 1081 | }, 1082 | { 1083 | "Fn::Join": [ 1084 | "", 1085 | [ 1086 | "arn:aws:s3:::", 1087 | { 1088 | "Ref": "StreamingAnalyticsBucket" 1089 | }, 1090 | "/*" 1091 | ] 1092 | ] 1093 | } 1094 | ] 1095 | }, 1096 | { 1097 | "Effect": "Allow", 1098 | "Action": [ 1099 | "kms:Decrypt", 1100 | "kms:GenerateDataKey" 1101 | ], 1102 | "Resource": [ 1103 | { 1104 | "Fn::If": [ 1105 | "NoEncryption", 1106 | { 1107 | "Fn::Join": [ 1108 | "", 1109 | [ 1110 | "arn:aws:kms:", 1111 | { 1112 | "Ref": "AWS::Region" 1113 | }, 1114 | ":", 1115 | { 1116 | "Ref": "AWS::AccountId" 1117 | }, 1118 | ":key/placeholder-kms-id" 1119 | ] 1120 | ] 1121 | }, 1122 | { 1123 | "Fn::GetAtt": [ 1124 | "EncryptionKey", 1125 | "Arn" 1126 | ] 1127 | } 1128 | ] 1129 | } 1130 | ], 1131 | "Condition": { 1132 | "StringEquals": { 1133 | "kms:ViaService": { 1134 | "Fn::Join": [ 1135 | "", 1136 | [ 1137 | "s3.", 1138 | { 1139 | "Ref": "AWS::Region" 1140 | }, 1141 | ".amazonaws.com" 1142 | ] 1143 | ] 1144 | } 1145 | }, 1146 | "StringLike": { 1147 | "kms:EncryptionContext:aws:s3:arn": { 1148 | "Fn::Join": [ 1149 | "", 1150 | [ 1151 | "arn:aws:s3:::", 1152 | { 1153 | "Ref": "StreamingAnalyticsBucket" 1154 | }, 1155 | "/RedshiftDelivery/*" 1156 | ] 1157 | ] 1158 | } 1159 | } 1160 | } 1161 | }, 1162 | { 1163 | "Sid": "", 1164 | "Effect": "Allow", 1165 | "Action": [ 1166 | "logs:PutLogEvents" 1167 | ], 1168 | "Resource": [ 1169 | "*" 1170 | ] 1171 | } 1172 | ] 1173 | } 1174 | } 1175 | ] 1176 | } 1177 | }, 1178 | "FIrehoseToRedshift": { 1179 | "Type": "AWS::KinesisFirehose::DeliveryStream", 1180 | "Condition": "RedshiftDestination", 1181 | "Properties": { 1182 | "RedshiftDestinationConfiguration": { 1183 | "ClusterJDBCURL": { 1184 | "Ref": "RedshiftJDBCURL" 1185 | }, 1186 | "CopyCommand": { 1187 | "CopyOptions": "delimiter ','", 1188 | "DataTableColumns": { 1189 | "Ref": "RedshiftColumns" 1190 | }, 1191 | "DataTableName": { 1192 | "Ref": "RedshiftTableName" 1193 | } 1194 | }, 1195 | "Password": { 1196 | "Ref": "MasterUserPassword" 1197 | }, 1198 | "RoleARN": { 1199 | "Fn::GetAtt": [ 1200 | "RedshiftDeliveryRole", 1201 | "Arn" 1202 | ] 1203 | }, 1204 | "S3Configuration": { 1205 | "BucketARN": { 1206 | "Fn::Join": [ 1207 | "", 1208 | [ 1209 | "arn:aws:s3:::", 1210 | { 1211 | "Ref": "StreamingAnalyticsBucket" 1212 | } 1213 | ] 1214 | ] 1215 | }, 1216 | "RoleARN": { 1217 | "Fn::GetAtt": [ 1218 | "RedshiftDeliveryRole", 1219 | "Arn" 1220 | ] 1221 | }, 1222 | "BufferingHints": { 1223 | "IntervalInSeconds": { 1224 | "Ref": "RedshiftBufferInterval" 1225 | }, 1226 | "SizeInMBs": { 1227 | "Ref": "RedshiftBufferSize" 1228 | } 1229 | }, 1230 | "CompressionFormat": "UNCOMPRESSED", 1231 | "EncryptionConfiguration": { 1232 | "KMSEncryptionConfig": { 1233 | "Fn::If": [ 1234 | "NoEncryption", 1235 | { 1236 | "Ref": "AWS::NoValue" 1237 | }, 1238 | { 1239 | "AWSKMSKeyARN": { 1240 | "Fn::GetAtt": [ 1241 | "EncryptionKey", 1242 | "Arn" 1243 | ] 1244 | } 1245 | } 1246 | ] 1247 | }, 1248 | "NoEncryptionConfig": { 1249 | "Fn::If": [ 1250 | "NoEncryption", 1251 | "NoEncryption", 1252 | { 1253 | "Ref": "AWS::NoValue" 1254 | } 1255 | ] 1256 | } 1257 | }, 1258 | "Prefix": "RedshiftDelivery/" 1259 | }, 1260 | "Username": { 1261 | "Ref": "MasterUser" 1262 | } 1263 | } 1264 | } 1265 | }, 1266 | "NewOutputRole": { 1267 | "Type": "AWS::IAM::Role", 1268 | "Properties": { 1269 | "AssumeRolePolicyDocument": { 1270 | "Version": "2012-10-17", 1271 | "Statement": [ 1272 | { 1273 | "Effect": "Allow", 1274 | "Principal": { 1275 | "Service": "kinesisanalytics.amazonaws.com" 1276 | }, 1277 | "Action": "sts:AssumeRole" 1278 | } 1279 | ] 1280 | }, 1281 | "Policies": [ 1282 | { 1283 | "PolicyName": { 1284 | "Fn::Join": [ 1285 | "", 1286 | [ 1287 | "OutputRole-", 1288 | { 1289 | "Ref": "AWS::StackName" 1290 | } 1291 | ] 1292 | ] 1293 | }, 1294 | "PolicyDocument": { 1295 | "Version": "2012-10-17", 1296 | "Statement": [ 1297 | { 1298 | "Sid": "WriteOutputFirehose", 1299 | "Effect": "Allow", 1300 | "Action": [ 1301 | "firehose:DescribeDeliveryStream", 1302 | "firehose:PutRecord", 1303 | "firehose:PutRecordBatch" 1304 | ], 1305 | "Resource": [ 1306 | { 1307 | "Fn::If": [ 1308 | "RedshiftDestination", 1309 | { 1310 | "Fn::Join": [ 1311 | "", 1312 | [ 1313 | "arn:aws:firehose:", 1314 | { 1315 | "Ref": "AWS::Region" 1316 | }, 1317 | ":", 1318 | { 1319 | "Ref": "AWS::AccountId" 1320 | }, 1321 | ":deliverystream/", 1322 | { 1323 | "Ref": "FIrehoseToRedshift" 1324 | } 1325 | ] 1326 | ] 1327 | }, 1328 | { 1329 | "Ref": "AWS::NoValue" 1330 | } 1331 | ] 1332 | }, 1333 | { 1334 | "Fn::If": [ 1335 | "S3Destination", 1336 | { 1337 | "Fn::Join": [ 1338 | "", 1339 | [ 1340 | "arn:aws:firehose:", 1341 | { 1342 | "Ref": "AWS::Region" 1343 | }, 1344 | ":", 1345 | { 1346 | "Ref": "AWS::AccountId" 1347 | }, 1348 | ":deliverystream/", 1349 | { 1350 | "Ref": "FIrehoseToS3" 1351 | } 1352 | ] 1353 | ] 1354 | }, 1355 | { 1356 | "Ref": "AWS::NoValue" 1357 | } 1358 | ] 1359 | }, 1360 | { 1361 | "Fn::If": [ 1362 | "ElasticsearchDestination", 1363 | { 1364 | "Fn::Join": [ 1365 | "", 1366 | [ 1367 | "arn:aws:firehose:", 1368 | { 1369 | "Ref": "AWS::Region" 1370 | }, 1371 | ":", 1372 | { 1373 | "Ref": "AWS::AccountId" 1374 | }, 1375 | ":deliverystream/", 1376 | { 1377 | "Ref": "FIrehoseToElasticsearch" 1378 | } 1379 | ] 1380 | ] 1381 | }, 1382 | { 1383 | "Ref": "AWS::NoValue" 1384 | } 1385 | ] 1386 | }, 1387 | { 1388 | "Fn::If": [ 1389 | "StreamDestination", 1390 | { 1391 | "Fn::Join": [ 1392 | "", 1393 | [ 1394 | "arn:aws:kinesis:", 1395 | { 1396 | "Ref": "AWS::Region" 1397 | }, 1398 | ":", 1399 | { 1400 | "Ref": "AWS::AccountId" 1401 | }, 1402 | ":stream/kinesis-analytics-placeholder-stream-destination" 1403 | ] 1404 | ] 1405 | }, 1406 | { 1407 | "Ref": "AWS::NoValue" 1408 | } 1409 | ] 1410 | } 1411 | ] 1412 | }, 1413 | { 1414 | "Sid": "WriteOutputKinesis", 1415 | "Effect": "Allow", 1416 | "Action": [ 1417 | "kinesis:DescribeStream", 1418 | "kinesis:PutRecord", 1419 | "kinesis:PutRecords" 1420 | ], 1421 | "Resource": [ 1422 | { 1423 | "Fn::If": [ 1424 | "StreamDestination", 1425 | { 1426 | "Fn::Join": [ 1427 | "", 1428 | [ 1429 | "arn:aws:kinesis:", 1430 | { 1431 | "Ref": "AWS::Region" 1432 | }, 1433 | ":", 1434 | { 1435 | "Ref": "AWS::AccountId" 1436 | }, 1437 | ":stream/", 1438 | { 1439 | "Ref": "DestinationStream" 1440 | } 1441 | ] 1442 | ] 1443 | }, 1444 | { 1445 | "Fn::Join": [ 1446 | "", 1447 | [ 1448 | "arn:aws:kinesis:", 1449 | { 1450 | "Ref": "AWS::Region" 1451 | }, 1452 | ":", 1453 | { 1454 | "Ref": "AWS::AccountId" 1455 | }, 1456 | ":stream/kinesis-analytics-placeholder-stream-destination" 1457 | ] 1458 | ] 1459 | } 1460 | ] 1461 | } 1462 | ] 1463 | } 1464 | ] 1465 | } 1466 | } 1467 | ] 1468 | } 1469 | }, 1470 | "AddKAOutputFunctionRole": { 1471 | "Type": "AWS::IAM::Role", 1472 | "Properties": { 1473 | "AssumeRolePolicyDocument": { 1474 | "Version": "2012-10-17", 1475 | "Statement": [ 1476 | { 1477 | "Effect": "Allow", 1478 | "Principal": { 1479 | "Service": [ 1480 | "lambda.amazonaws.com" 1481 | ] 1482 | }, 1483 | "Action": [ 1484 | "sts:AssumeRole" 1485 | ] 1486 | }, 1487 | { 1488 | "Effect": "Allow", 1489 | "Principal": { 1490 | "AWS": { 1491 | "Ref": "AWS::AccountId" 1492 | } 1493 | }, 1494 | "Action": "sts:AssumeRole" 1495 | } 1496 | ] 1497 | }, 1498 | "Policies": [ 1499 | { 1500 | "PolicyName": { 1501 | "Fn::Join": [ 1502 | "", 1503 | [ 1504 | "Lambda-addKAOutput-", 1505 | { 1506 | "Ref": "AWS::StackName" 1507 | } 1508 | ] 1509 | ] 1510 | }, 1511 | "PolicyDocument": { 1512 | "Version": "2012-10-17", 1513 | "Statement": [ 1514 | { 1515 | "Effect": "Allow", 1516 | "Action": [ 1517 | "logs:CreateLogGroup", 1518 | "logs:CreateLogStream", 1519 | "logs:PutLogEvents" 1520 | ], 1521 | "Resource": "*" 1522 | }, 1523 | { 1524 | "Effect": "Allow", 1525 | "Action": [ 1526 | "kinesisanalytics:AddApplicationOutput", 1527 | "kinesisanalytics:DeleteApplicationOutput", 1528 | "kinesisanalytics:DescribeApplication" 1529 | ], 1530 | "Resource": { 1531 | "Fn::Join": [ 1532 | "", 1533 | [ 1534 | "arn:aws:kinesisanalytics:", 1535 | { 1536 | "Ref": "AWS::Region" 1537 | }, 1538 | ":", 1539 | { 1540 | "Ref": "AWS::AccountId" 1541 | }, 1542 | ":application/", 1543 | {"Ref": "ApplicationName"} 1544 | ] 1545 | ] 1546 | } 1547 | }, 1548 | { 1549 | "Effect": "Allow", 1550 | "Action": [ 1551 | "iam:PassRole" 1552 | ], 1553 | "Resource": { 1554 | "Fn::GetAtt": [ 1555 | "NewOutputRole", 1556 | "Arn" 1557 | ] 1558 | } 1559 | } 1560 | ] 1561 | } 1562 | } 1563 | ] 1564 | } 1565 | }, 1566 | "AddKAOutputFunction": { 1567 | "Type": "AWS::Lambda::Function", 1568 | "Properties": { 1569 | "Code": { 1570 | "S3Bucket": {"Fn::Join": ["", [ 1571 | "solutions-", 1572 | {"Ref": "AWS::Region"} 1573 | ]]}, 1574 | "S3Key": "streaming-analytics/v1/add-output-code.zip" 1575 | }, 1576 | "Description": "This function adds an external destination to an existing Kinesis Analytics application", 1577 | "Handler": "add_output.lambda_handler", 1578 | "MemorySize": "128", 1579 | "Role": { 1580 | "Fn::GetAtt": [ 1581 | "AddKAOutputFunctionRole", 1582 | "Arn" 1583 | ] 1584 | }, 1585 | "Runtime": "python2.7", 1586 | "Timeout": "30" 1587 | } 1588 | }, 1589 | "AddOutput": { 1590 | "Type": "Custom::AddApplicationOutput", 1591 | "Properties": { 1592 | "ServiceToken": { 1593 | "Fn::GetAtt": [ 1594 | "AddKAOutputFunction", 1595 | "Arn" 1596 | ] 1597 | }, 1598 | "FunctionName": {"Ref": "AddKAOutputFunction"}, 1599 | "Region": {"Ref": "AWS::Region"}, 1600 | "AppName": {"Ref": "ApplicationName"}, 1601 | "OutputRole": {"Fn::GetAtt": ["NewOutputRole","Arn"]}, 1602 | "Destination": {"Ref": "ExternalDestination"}, 1603 | "DestinationStreamName": {"Ref": "InAppDestStream"}, 1604 | "RedshiftDest": { 1605 | "Fn::If": [ 1606 | "RedshiftDestination", 1607 | { 1608 | "Fn::Join": [ 1609 | "", 1610 | [ 1611 | "arn:aws:firehose:", 1612 | { 1613 | "Ref": "AWS::Region" 1614 | }, 1615 | ":", 1616 | { 1617 | "Ref": "AWS::AccountId" 1618 | }, 1619 | ":deliverystream/", 1620 | { 1621 | "Ref": "FIrehoseToRedshift" 1622 | } 1623 | ] 1624 | ] 1625 | }, 1626 | { 1627 | "Ref": "AWS::NoValue" 1628 | } 1629 | ] 1630 | }, 1631 | "S3Dest": { 1632 | "Fn::If": [ 1633 | "S3Destination", 1634 | { 1635 | "Fn::Join": [ 1636 | "", 1637 | [ 1638 | "arn:aws:firehose:", 1639 | { 1640 | "Ref": "AWS::Region" 1641 | }, 1642 | ":", 1643 | { 1644 | "Ref": "AWS::AccountId" 1645 | }, 1646 | ":deliverystream/", 1647 | { 1648 | "Ref": "FIrehoseToS3" 1649 | } 1650 | ] 1651 | ] 1652 | }, 1653 | { 1654 | "Ref": "AWS::NoValue" 1655 | } 1656 | ] 1657 | }, 1658 | "ElasticsearchDest": { 1659 | "Fn::If": [ 1660 | "ElasticsearchDestination", 1661 | { 1662 | "Fn::Join": [ 1663 | "", 1664 | [ 1665 | "arn:aws:firehose:", 1666 | { 1667 | "Ref": "AWS::Region" 1668 | }, 1669 | ":", 1670 | { 1671 | "Ref": "AWS::AccountId" 1672 | }, 1673 | ":deliverystream/", 1674 | { 1675 | "Ref": "FIrehoseToElasticsearch" 1676 | } 1677 | ] 1678 | ] 1679 | }, 1680 | { 1681 | "Ref": "AWS::NoValue" 1682 | } 1683 | ] 1684 | }, 1685 | "StreamDest": { 1686 | "Fn::If": [ 1687 | "StreamDestination", 1688 | { 1689 | "Fn::Join": [ 1690 | "", 1691 | [ 1692 | "arn:aws:kinesis:", 1693 | { 1694 | "Ref": "AWS::Region" 1695 | }, 1696 | ":", 1697 | { 1698 | "Ref": "AWS::AccountId" 1699 | }, 1700 | ":stream/", 1701 | { 1702 | "Ref": "DestinationStream" 1703 | } 1704 | ] 1705 | ] 1706 | }, 1707 | { 1708 | "Ref": "AWS::NoValue" 1709 | } 1710 | ] 1711 | } 1712 | } 1713 | } 1714 | }, 1715 | "Outputs": { 1716 | "OutputBucket": { 1717 | "Description": "Data will be stored in this Amazon S3 bucket", 1718 | "Condition": "NeedS3Bucket", 1719 | "Value": { 1720 | "Ref": "StreamingAnalyticsBucket" 1721 | } 1722 | }, 1723 | "KMSKey": { 1724 | "Description": "Data will be encrypted with this Amazon KMS encryption key", 1725 | "Value": { 1726 | "Ref": "KMSAlias" 1727 | }, 1728 | "Condition": "EncryptData" 1729 | }, 1730 | "DestStream": { 1731 | "Description": "Data will flow from this in-application stream to your new external destination", 1732 | "Value": { 1733 | "Ref": "InAppDestStream" 1734 | } 1735 | } 1736 | } 1737 | } -------------------------------------------------------------------------------- /cform/streaming-analytics-pipeline-config.yaml: -------------------------------------------------------------------------------- 1 | # Update this file according to your Input Schema and Kinesis Analytics application code 2 | # Note: pay attention to indentation - it matters 3 | 4 | # REQUIRED 5 | format: 6 | # Enter the input streaming data record format as it is entering your source Amazon Kinesis Stream, either 'JSON' or 'CSV' 7 | InputFormatType: JSON 8 | # Iff InputFormatType is 'CSV,' enter the column delimiter of the streaming data (i.e. ',' '|') 9 | RecordColumnDelimiter: "," 10 | # Iff InputFormatType is 'CSV,' enter the row delimiter of the streaming data (i.e. '\n') 11 | RecordRowDelimiter: "\n" 12 | # Iff InputFormatType is 'JSON,' enter a record row path which points to the specific key value pair of the data you'd like to extract 13 | RecordRowPath: "$" 14 | # Enter the output streaming data record format as it will be sent to your destination Amazon Kinesis Stream or Firehose Delivery Stream, either 'JSON' or 'CSV' 15 | # If your external destination is Amazon Redshift, this value will automatically be set to CSV 16 | # If your external destination is Amazon Elasticsearch, this value will automatically be set to JSON 17 | OutputFormatType: JSON 18 | 19 | # Iff RecordFormatType is 'CSV,' enter list items using the following format: '{Name: columnName, SqlType: dataType}' 20 | # Iff RecordFormatType is 'JSON,' enter list items using the following format: '{Name: columnName, SqlType: dataType, Mapping: $.rowPath}' 21 | # (Substitute '$.' with the RecordRowPath specified above) 22 | # See example below 23 | columns: 24 | - {Name: car, SqlType: VARCHAR(8), Mapping: $.car} 25 | - {Name: carid, SqlType: VARCHAR(20), Mapping: $.carid} 26 | - {Name: engine_temp, SqlType: REAL, Mapping: $.engine_temp} 27 | - {Name: speed, SqlType: REAL, Mapping: $.speed} 28 | - {Name: longitude, SqlType: DOUBLE, Mapping: $.long} 29 | - {Name: latitude, SqlType: DOUBLE, Mapping: $.lat} 30 | 31 | 32 | # Indent, and paste your SQL code verbatim below the comment. Please refer to implementation guide for detailed instructions. 33 | # Name your destination stream 'DESTINATION_SQL_STREAM' and name your source stream 'SOURCE_SQL_STREAM_001' 34 | # See example below 35 | sql_code: | 36 | -- Paste your SQL code here 37 | CREATE OR REPLACE STREAM "DESTINATION_SQL_STREAM" (car VARCHAR(8), vehicle_count INTEGER, avg_temp REAL, low_temp REAL, high_temp REAL); 38 | CREATE OR REPLACE PUMP "STREAM_PUMP" AS INSERT INTO "DESTINATION_SQL_STREAM" 39 | SELECT STREAM "car", COUNT(*) as vehicle_count, AVG("engine_temp") as avg_temp, MIN("engine_temp") as low_temp, MAX("engine_temp") as high_temp 40 | FROM "SOURCE_SQL_STREAM_001" 41 | GROUP BY "car", FLOOR(("SOURCE_SQL_STREAM_001".ROWTIME - TIMESTAMP '1970-01-01 00:00:00') SECOND / 30 TO SECOND); -------------------------------------------------------------------------------- /cform/streaming-analytics-pipeline.template: -------------------------------------------------------------------------------- 1 | { 2 | "AWSTemplateFormatVersion": "2010-09-09", 3 | "Description": "(SO0014)- Streaming Analytics Pipeline: this template creates an Amazon Kinesis Analytics application that can be used to analyze data from a new or existing Amazon Kinesis stream, and deliver the results to an external destination.", 4 | "Metadata": { 5 | "AWS::CloudFormation::Interface": { 6 | "ParameterGroups": [ 7 | { 8 | "Label": { 9 | "default": "Amazon Kinesis Analytics Configuration" 10 | }, 11 | "Parameters": [ 12 | "NewOrExistingStream", 13 | "SourceStreamSize", 14 | "ExistingStreamName", 15 | "ExternalDestination", 16 | "ConfigFileLocation", 17 | "EncryptData", 18 | "PersistRawDataParam" 19 | ] 20 | }, 21 | { 22 | "Label": { 23 | "default": "Amazon Redshift Destination Configuration" 24 | }, 25 | "Parameters": [ 26 | "MasterUser", 27 | "MasterUserPassword", 28 | "RedshiftJDBCURL", 29 | "RedshiftTableName", 30 | "RedshiftColumns", 31 | "RedshiftBufferInterval", 32 | "RedshiftBufferSize" 33 | ] 34 | }, 35 | { 36 | "Label": { 37 | "default": "Amazon Elasticsearch Destination Configuration" 38 | }, 39 | "Parameters": [ 40 | "ESDomain", 41 | "ESIndex", 42 | "ESType", 43 | "ESIndexRotation", 44 | "ESBufferInterval", 45 | "ESBufferSize" 46 | ] 47 | }, 48 | { 49 | "Label": { 50 | "default": "Amazon S3 Destination Configuration" 51 | }, 52 | "Parameters": [ 53 | "S3BufferInterval", 54 | "S3BufferSize", 55 | "S3DestinationPrefix" 56 | ] 57 | }, 58 | { 59 | "Label": { 60 | "default": "Amazon Kinesis Stream Destination Configuration" 61 | }, 62 | "Parameters": [ 63 | "DestinationStream" 64 | ] 65 | }, 66 | { 67 | "Label": { 68 | "default": "Anonymous Metrics Request" 69 | }, 70 | "Parameters": [ 71 | "CollectAnonymousData" 72 | ] 73 | } 74 | ], 75 | "ParameterLabels": { 76 | "SourceStreamSize": { 77 | "default": "New Stream Shard Count" 78 | }, 79 | "NewOrExistingStream": { 80 | "default": "New or Existing Stream" 81 | }, 82 | "ExistingStreamName": { 83 | "default": "Existing Stream Name" 84 | }, 85 | "ExternalDestination": { 86 | "default": "External Destination" 87 | }, 88 | "ConfigFileLocation": { 89 | "default": "Configuration File Location" 90 | }, 91 | "EncryptData": { 92 | "default": "Encrypt Data at Rest?" 93 | }, 94 | "PersistRawDataParam": { 95 | "default": "Persist Raw Source Data?" 96 | }, 97 | "MasterUser": { 98 | "default": "Master User Name" 99 | }, 100 | "MasterUserPassword": { 101 | "default": "Master User Password" 102 | }, 103 | "RedshiftJDBCURL": { 104 | "default": "JDBC URL" 105 | }, 106 | "RedshiftTableName": { 107 | "default": "Table Name" 108 | }, 109 | "RedshiftColumns": { 110 | "default": "Column Pattern" 111 | }, 112 | "RedshiftBufferInterval": { 113 | "default": "Buffer Interval" 114 | }, 115 | "RedshiftBufferSize": { 116 | "default": "Buffer Size" 117 | }, 118 | "ESDomain": { 119 | "default": "Domain Name" 120 | }, 121 | "ESIndex": { 122 | "default": "Index Name" 123 | }, 124 | "ESType": { 125 | "default": "Type Name" 126 | }, 127 | "ESIndexRotation": { 128 | "default": "Index Rotation" 129 | }, 130 | "ESBufferInterval": { 131 | "default": "Buffer Interval" 132 | }, 133 | "ESBufferSize": { 134 | "default": "Buffer Size" 135 | }, 136 | "S3BufferInterval": { 137 | "default": "Buffer Interval" 138 | }, 139 | "S3BufferSize": { 140 | "default": "Buffer Size" 141 | }, 142 | "S3DestinationPrefix": { 143 | "default": "Destination Prefix" 144 | }, 145 | "DestinationStream": { 146 | "default": "Destination Stream Name" 147 | }, 148 | "CollectAnonymousData": { 149 | "default": "Collect Anonymous Data" 150 | } 151 | } 152 | } 153 | }, 154 | "Parameters": { 155 | "CollectAnonymousData": { 156 | "Description": "Send anonymous data to AWS", 157 | "Type": "String", 158 | "Default": "Yes", 159 | "AllowedValues": [ 160 | "Yes", 161 | "No" 162 | ] 163 | }, 164 | "NewOrExistingStream": { 165 | "Description": "Use a new or existing Amazon Kinesis Stream?", 166 | "Type": "String", 167 | "AllowedValues": [ 168 | "New Kinesis Stream", 169 | "Existing Kinesis Stream" 170 | ], 171 | "Default": "New Kinesis Stream" 172 | }, 173 | "ExistingStreamName": { 174 | "Description": "If you selected 'Existing Kinesis Stream'", 175 | "Type": "String" 176 | }, 177 | "SourceStreamSize": { 178 | "Description": "If you selected 'New Kinesis Stream'", 179 | "Type": "String", 180 | "AllowedPattern": "[0-9]*", 181 | "Default": "2" 182 | }, 183 | "S3DestinationPrefix": { 184 | "Description": "Name of prefix where the aggregated data will be stored.", 185 | "Type": "String", 186 | "Default": "AggregatedData" 187 | }, 188 | "ExternalDestination": { 189 | "Description": "Select a destination for your aggregated data.", 190 | "Type": "String", 191 | "AllowedValues": [ 192 | "Amazon Redshift", 193 | "Amazon ElasticSearch Service", 194 | "Amazon S3", 195 | "Amazon Kinesis Stream" 196 | ], 197 | "Default": "Amazon S3" 198 | }, 199 | "ConfigFileLocation": { 200 | "Description": "/ Enter the S3 location of the yaml file.", 201 | "Type": "String" 202 | }, 203 | "S3BufferInterval": { 204 | "Description": "Number of seconds to buffer data before delivering to S3 (60 to 900).", 205 | "Type": "Number", 206 | "Default": 300, 207 | "MinValue": 60, 208 | "MaxValue": 900 209 | }, 210 | "S3BufferSize": { 211 | "Description": "Number of MB of data to buffer before delivering to S3 (1 to 128).", 212 | "Type": "Number", 213 | "Default": 5, 214 | "MinValue": 1, 215 | "MaxValue": 128 216 | }, 217 | "EncryptData": { 218 | "Description": "Would you like to encrypt the data as it leaves your Kinesis Analytics application.", 219 | "Type": "String", 220 | "AllowedValues": [ 221 | "yes", 222 | "no" 223 | ], 224 | "Default": "yes" 225 | }, 226 | "ESDomain": { 227 | "Description": "Name of the destination Amazon Elasticsearch Domain.", 228 | "Type": "String" 229 | }, 230 | "ESIndex": { 231 | "Description": "Index name of the Amazon ElasticSearch Domain.", 232 | "Type": "String" 233 | }, 234 | "ESType": { 235 | "Description": "Name of the Amazon ElasticSearch type.", 236 | "Type": "String" 237 | }, 238 | "ESIndexRotation": { 239 | "Description": "Frequency at which the Amazon ElasticSearch index will be rotated.", 240 | "Type": "String", 241 | "AllowedValues": [ 242 | "NoRotation", 243 | "OneHour", 244 | "OneDay", 245 | "OneWeek", 246 | "OneMonth" 247 | ], 248 | "Default": "NoRotation" 249 | }, 250 | "ESBufferInterval": { 251 | "Description": "Number of seconds to buffer data before delivering to S3 to be copied to ElasticSearch (60 to 900).", 252 | "Type": "Number", 253 | "Default": 300, 254 | "MinValue": 60, 255 | "MaxValue": 900 256 | }, 257 | "ESBufferSize": { 258 | "Description": "Number of MB of data to buffer before delivering to S3 to be copied to ElasticSearch (1 to 100).", 259 | "Type": "Number", 260 | "Default": 5, 261 | "MinValue": 1, 262 | "MaxValue": 100 263 | }, 264 | "MasterUserPassword": { 265 | "Description": "Master user password of your preexisting Redshift cluster", 266 | "NoEcho": "true", 267 | "Type": "String" 268 | }, 269 | "MasterUser": { 270 | "Description": "Name of the master user of the preexisting Redshift Cluster", 271 | "Type": "String" 272 | }, 273 | "RedshiftBufferInterval": { 274 | "Description": "Number of seconds to buffer data before delivering to S3 (60 to 900).", 275 | "Type": "Number", 276 | "Default": 300, 277 | "MinValue": 60, 278 | "MaxValue": 900 279 | }, 280 | "RedshiftBufferSize": { 281 | "Description": "Number of MB of data to buffer before delivering to S3 (1 to 128).", 282 | "Type": "Number", 283 | "Default": 5, 284 | "MinValue": 1, 285 | "MaxValue": 128 286 | }, 287 | "RedshiftJDBCURL": { 288 | "Description": "URL to connect to the cluster database with the JDBC client", 289 | "Type": "String" 290 | }, 291 | "RedshiftTableName": { 292 | "Description": "Name of table in preexisting Redshift Cluster", 293 | "Type": "String" 294 | }, 295 | "RedshiftColumns": { 296 | "Description": "Comma-separated list of the columns in the destination Redshift table.", 297 | "Type": "String" 298 | }, 299 | "PersistRawDataParam": { 300 | "Description": "Store raw incoming data in an S3 bucket?", 301 | "Type": "String", 302 | "AllowedValues": [ 303 | "yes", 304 | "no" 305 | ], 306 | "Default": "yes" 307 | }, 308 | "DestinationStream": { 309 | "Description": "Name of the existing destination Kinesis Stream", 310 | "Type": "String" 311 | } 312 | }, 313 | "Mappings": {}, 314 | "Conditions": { 315 | "ConfigFileNotProvidedCondition": { 316 | "Fn::Equals": [ 317 | { 318 | "Ref": "ConfigFileLocation" 319 | }, 320 | "" 321 | ] 322 | }, 323 | "AnonymousDataCondition": { 324 | "Fn::Equals": [ 325 | { 326 | "Ref": "CollectAnonymousData" 327 | }, 328 | "Yes" 329 | ] 330 | }, 331 | "SourceIsNewStream": { 332 | "Fn::Equals": [ 333 | { 334 | "Ref": "NewOrExistingStream" 335 | }, 336 | "New Kinesis Stream" 337 | ] 338 | }, 339 | "RawDataCondition": { 340 | "Fn::Equals": [ 341 | { 342 | "Ref": "PersistRawDataParam" 343 | }, 344 | "yes" 345 | ] 346 | }, 347 | "EncryptData": { 348 | "Fn::Equals": [ 349 | { 350 | "Ref": "EncryptData" 351 | }, 352 | "yes" 353 | ] 354 | }, 355 | "NoEncryption": { 356 | "Fn::Equals": [ 357 | { 358 | "Ref": "EncryptData" 359 | }, 360 | "no" 361 | ] 362 | }, 363 | "RedshiftDestination": { 364 | "Fn::Equals": [ 365 | { 366 | "Ref": "ExternalDestination" 367 | }, 368 | "Amazon Redshift" 369 | ] 370 | }, 371 | "ElasticsearchDestination": { 372 | "Fn::Equals": [ 373 | { 374 | "Ref": "ExternalDestination" 375 | }, 376 | "Amazon ElasticSearch Service" 377 | ] 378 | }, 379 | "S3Destination": { 380 | "Fn::Equals": [ 381 | { 382 | "Ref": "ExternalDestination" 383 | }, 384 | "Amazon S3" 385 | ] 386 | }, 387 | "StreamDestination": { 388 | "Fn::Equals": [ 389 | { 390 | "Ref": "ExternalDestination" 391 | }, 392 | "Amazon Kinesis Stream" 393 | ] 394 | } 395 | }, 396 | "Resources": { 397 | "EncryptionKey": { 398 | "Type": "AWS::KMS::Key", 399 | "Condition": "EncryptData", 400 | "Properties": { 401 | "Description": "KMS key generated to encrypt aggregated Kinesis Analytics data.", 402 | "EnableKeyRotation": "true", 403 | "KeyPolicy": { 404 | "Id": "KMS key policy", 405 | "Version": "2012-10-17", 406 | "Statement": [ 407 | { 408 | "Sid": "Enable IAM User Permissions", 409 | "Effect": "Allow", 410 | "Principal": { 411 | "AWS": [ 412 | { 413 | "Fn::Join": [ 414 | "", 415 | [ 416 | "arn:aws:iam::", 417 | { 418 | "Ref": "AWS::AccountId" 419 | }, 420 | ":root" 421 | ] 422 | ] 423 | } 424 | ] 425 | }, 426 | "Action": "kms:*", 427 | "Resource": "*" 428 | }, 429 | { 430 | "Sid": "Allow access for Key Administrators", 431 | "Effect": "Allow", 432 | "Principal": { 433 | "AWS": [] 434 | }, 435 | "Action": [ 436 | "kms:Create*", 437 | "kms:Describe*", 438 | "kms:Enable*", 439 | "kms:List*", 440 | "kms:Put*", 441 | "kms:Update*", 442 | "kms:Revoke*", 443 | "kms:Disable*", 444 | "kms:Get*", 445 | "kms:Delete*", 446 | "kms:ScheduleKeyDeletion", 447 | "kms:CancelKeyDeletion" 448 | ], 449 | "Resource": "*" 450 | }, 451 | { 452 | "Sid": "Allow use of the key", 453 | "Effect": "Allow", 454 | "Principal": { 455 | "AWS": [] 456 | }, 457 | "Action": [ 458 | "kms:Encrypt", 459 | "kms:Decrypt", 460 | "kms:ReEncrypt*", 461 | "kms:GenerateDataKey*", 462 | "kms:DescribeKey" 463 | ], 464 | "Resource": "*" 465 | }, 466 | { 467 | "Sid": "Allow attachment of persistent resources", 468 | "Effect": "Allow", 469 | "Principal": { 470 | "AWS": [] 471 | }, 472 | "Action": [ 473 | "kms:CreateGrant", 474 | "kms:ListGrants", 475 | "kms:RevokeGrant" 476 | ], 477 | "Resource": "*", 478 | "Condition": { 479 | "Bool": { 480 | "kms:GrantIsForAWSResource": true 481 | } 482 | } 483 | } 484 | ] 485 | } 486 | } 487 | }, 488 | "KMSAlias": { 489 | "Type": "AWS::KMS::Alias", 490 | "Condition": "EncryptData", 491 | "Properties": { 492 | "AliasName": { 493 | "Fn::Join": [ 494 | "", 495 | [ 496 | "alias/key-", 497 | { 498 | "Ref": "AWS::StackName" 499 | } 500 | ] 501 | ] 502 | }, 503 | "TargetKeyId": { 504 | "Fn::GetAtt": [ 505 | "EncryptionKey", 506 | "Arn" 507 | ] 508 | } 509 | } 510 | }, 511 | "SourceStream": { 512 | "Type": "AWS::Kinesis::Stream", 513 | "Condition": "SourceIsNewStream", 514 | "Properties": { 515 | "ShardCount": { 516 | "Ref": "SourceStreamSize" 517 | } 518 | } 519 | }, 520 | "StreamingAnalyticsBucket": { 521 | "Type": "AWS::S3::Bucket" 522 | }, 523 | "RawDatatoS3Role": { 524 | "Type": "AWS::IAM::Role", 525 | "Condition": "RawDataCondition", 526 | "Properties": { 527 | "AssumeRolePolicyDocument": { 528 | "Version": "2012-10-17", 529 | "Statement": [ 530 | { 531 | "Effect": "Allow", 532 | "Principal": { 533 | "Service": [ 534 | "firehose.amazonaws.com" 535 | ] 536 | }, 537 | "Action": [ 538 | "sts:AssumeRole" 539 | ], 540 | "Condition": { 541 | "StringEquals": { 542 | "sts:ExternalId": { 543 | "Ref": "AWS::AccountId" 544 | } 545 | } 546 | } 547 | } 548 | ] 549 | }, 550 | "Policies": [ 551 | { 552 | "PolicyName": { 553 | "Fn::Join": [ 554 | "", 555 | [ 556 | "Raw-Data-to-S3-Delivery-", 557 | { 558 | "Ref": "AWS::StackName" 559 | } 560 | ] 561 | ] 562 | }, 563 | "PolicyDocument": { 564 | "Version": "2012-10-17", 565 | "Statement": [ 566 | { 567 | "Sid": "", 568 | "Effect": "Allow", 569 | "Action": [ 570 | "s3:AbortMultipartUpload", 571 | "s3:GetBucketLocation", 572 | "s3:GetObject", 573 | "s3:ListBucket", 574 | "s3:ListBucketMultipartUploads", 575 | "s3:PutObject" 576 | ], 577 | "Resource": [ 578 | { 579 | "Fn::Join": [ 580 | "", 581 | [ 582 | "arn:aws:s3:::", 583 | { 584 | "Ref": "StreamingAnalyticsBucket" 585 | } 586 | ] 587 | ] 588 | }, 589 | { 590 | "Fn::Join": [ 591 | "", 592 | [ 593 | "arn:aws:s3:::", 594 | { 595 | "Ref": "StreamingAnalyticsBucket" 596 | }, 597 | "/*" 598 | ] 599 | ] 600 | } 601 | ] 602 | }, 603 | { 604 | "Effect": "Allow", 605 | "Action": [ 606 | "kms:Decrypt", 607 | "kms:GenerateDataKey" 608 | ], 609 | "Resource": [ 610 | { 611 | "Fn::If": [ 612 | "NoEncryption", 613 | { 614 | "Fn::Join": [ 615 | "", 616 | [ 617 | "arn:aws:kms:", 618 | { 619 | "Ref": "AWS::Region" 620 | }, 621 | ":", 622 | { 623 | "Ref": "AWS::AccountId" 624 | }, 625 | ":key/placeholder-kms-id" 626 | ] 627 | ] 628 | }, 629 | { 630 | "Fn::GetAtt": [ 631 | "EncryptionKey", 632 | "Arn" 633 | ] 634 | } 635 | ] 636 | } 637 | ], 638 | "Condition": { 639 | "StringEquals": { 640 | "kms:ViaService": { 641 | "Fn::Join": [ 642 | "", 643 | [ 644 | "s3.", 645 | { 646 | "Ref": "AWS::Region" 647 | }, 648 | ".amazonaws.com" 649 | ] 650 | ] 651 | } 652 | }, 653 | "StringLike": { 654 | "kms:EncryptionContext:aws:s3:arn": { 655 | "Fn::Join": [ 656 | "", 657 | [ 658 | "arn:aws:s3:::", 659 | { 660 | "Ref": "StreamingAnalyticsBucket" 661 | }, 662 | "/rawStreamData/*" 663 | ] 664 | ] 665 | } 666 | } 667 | } 668 | }, 669 | { 670 | "Sid": "", 671 | "Effect": "Allow", 672 | "Action": [ 673 | "logs:PutLogEvents" 674 | ], 675 | "Resource": [ 676 | "*" 677 | ] 678 | } 679 | ] 680 | } 681 | } 682 | ] 683 | } 684 | }, 685 | "RawDatatoS3": { 686 | "Type": "AWS::KinesisFirehose::DeliveryStream", 687 | "Condition": "RawDataCondition", 688 | "Properties": { 689 | "S3DestinationConfiguration": { 690 | "BucketARN": { 691 | "Fn::Join": [ 692 | "", 693 | [ 694 | "arn:aws:s3:::", 695 | { 696 | "Ref": "StreamingAnalyticsBucket" 697 | } 698 | ] 699 | ] 700 | }, 701 | "BufferingHints": { 702 | "IntervalInSeconds": 300, 703 | "SizeInMBs": 10 704 | }, 705 | "CompressionFormat": "UNCOMPRESSED", 706 | "EncryptionConfiguration": { 707 | "KMSEncryptionConfig": { 708 | "Fn::If": [ 709 | "NoEncryption", 710 | { 711 | "Ref": "AWS::NoValue" 712 | }, 713 | { 714 | "AWSKMSKeyARN": { 715 | "Fn::GetAtt": [ 716 | "EncryptionKey", 717 | "Arn" 718 | ] 719 | } 720 | } 721 | ] 722 | }, 723 | "NoEncryptionConfig": { 724 | "Fn::If": [ 725 | "NoEncryption", 726 | "NoEncryption", 727 | { 728 | "Ref": "AWS::NoValue" 729 | } 730 | ] 731 | } 732 | }, 733 | "Prefix": "rawStreamData/", 734 | "RoleARN": { 735 | "Fn::GetAtt": [ 736 | "RawDatatoS3Role", 737 | "Arn" 738 | ] 739 | } 740 | } 741 | } 742 | }, 743 | "PersistRawDataLambdaRole": { 744 | "Type": "AWS::IAM::Role", 745 | "Condition": "RawDataCondition", 746 | "Properties": { 747 | "AssumeRolePolicyDocument": { 748 | "Version": "2012-10-17", 749 | "Statement": [ 750 | { 751 | "Effect": "Allow", 752 | "Principal": { 753 | "Service": [ 754 | "lambda.amazonaws.com" 755 | ] 756 | }, 757 | "Action": [ 758 | "sts:AssumeRole" 759 | ] 760 | } 761 | ] 762 | }, 763 | "Policies": [ 764 | { 765 | "PolicyName": { 766 | "Fn::Join": [ 767 | "", 768 | [ 769 | "Lambda-PersistRawData-Role-", 770 | { 771 | "Ref": "AWS::StackName" 772 | } 773 | ] 774 | ] 775 | }, 776 | "PolicyDocument": { 777 | "Version": "2012-10-17", 778 | "Statement": [ 779 | { 780 | "Effect": "Allow", 781 | "Action": [ 782 | "logs:CreateLogGroup", 783 | "logs:CreateLogStream", 784 | "logs:PutLogEvents" 785 | ], 786 | "Resource": "*" 787 | }, 788 | { 789 | "Effect": "Allow", 790 | "Action": [ 791 | "kinesis:DescribeStream", 792 | "kinesis:GetShardIterator", 793 | "kinesis:GetRecords" 794 | ], 795 | "Resource": [ 796 | { 797 | "Fn::If": [ 798 | "SourceIsNewStream", 799 | { 800 | "Fn::GetAtt": [ 801 | "SourceStream", 802 | "Arn" 803 | ] 804 | }, 805 | { 806 | "Fn::Join": [ 807 | "", 808 | [ 809 | "arn:aws:kinesis:", 810 | { 811 | "Ref": "AWS::Region" 812 | }, 813 | ":", 814 | { 815 | "Ref": "AWS::AccountId" 816 | }, 817 | ":stream/", 818 | { 819 | "Ref": "ExistingStreamName" 820 | } 821 | ] 822 | ] 823 | } 824 | ] 825 | } 826 | ] 827 | }, 828 | { 829 | "Effect": "Allow", 830 | "Action": [ 831 | "firehose:PutRecordBatch" 832 | ], 833 | "Resource": [ 834 | { 835 | "Fn::Join": [ 836 | "", 837 | [ 838 | "arn:aws:firehose:", 839 | { 840 | "Ref": "AWS::Region" 841 | }, 842 | ":", 843 | { 844 | "Ref": "AWS::AccountId" 845 | }, 846 | ":deliverystream/", 847 | { 848 | "Ref": "RawDatatoS3" 849 | } 850 | ] 851 | ] 852 | } 853 | ] 854 | } 855 | ] 856 | } 857 | } 858 | ] 859 | } 860 | }, 861 | "PersistRawDataFunction": { 862 | "Type": "AWS::Lambda::Function", 863 | "Condition": "RawDataCondition", 864 | "Properties": { 865 | "Code": { 866 | "ZipFile": { 867 | "Fn::Join": [ 868 | "\n", 869 | [ 870 | "from __future__ import print_function", 871 | "import boto3", 872 | "import base64", 873 | "", 874 | "fh = boto3.client('firehose')", 875 | "", 876 | "def lambda_handler(event, context):", 877 | " rawDataChunks=[event['Records'][x:x+500] for x in xrange(0, len(event['Records']), 500)]", 878 | " for chunk in rawDataChunks:", 879 | " count = 0", 880 | " batch=[]", 881 | " for record in chunk:", 882 | " batchDict={}", 883 | " payload = base64.b64decode(record['kinesis']['data'])+'\\n'", 884 | " batchDict['Data']=payload", 885 | " batch.append(batchDict)", 886 | " count+=1", 887 | " try:", 888 | { 889 | "Fn::Join": [ 890 | "", 891 | [ 892 | " fh.put_record_batch(DeliveryStreamName='", 893 | { 894 | "Ref": "RawDatatoS3" 895 | }, 896 | "',Records=batch)" 897 | ] 898 | ] 899 | }, 900 | " print('Put '+str(count)+' records to Amazon Kinesis Firehose Delivery Stream')", 901 | " except Exception as e:", 902 | " print(e)" 903 | ] 904 | ] 905 | } 906 | }, 907 | "Description": "This function GETs and decodes raw data records from the source Kinesis stream, and batch PUTs them to a Firehose Delivery Stream", 908 | "Handler": "index.lambda_handler", 909 | "MemorySize": "128", 910 | "Role": { 911 | "Fn::GetAtt": [ 912 | "PersistRawDataLambdaRole", 913 | "Arn" 914 | ] 915 | }, 916 | "Runtime": "python2.7", 917 | "Timeout": "30" 918 | } 919 | }, 920 | "PersistRawDataTrigger": { 921 | "Type": "AWS::Lambda::EventSourceMapping", 922 | "Condition": "RawDataCondition", 923 | "Properties": { 924 | "BatchSize": 1000, 925 | "Enabled": "True", 926 | "EventSourceArn": { 927 | "Fn::If": [ 928 | "SourceIsNewStream", 929 | { 930 | "Fn::GetAtt": [ 931 | "SourceStream", 932 | "Arn" 933 | ] 934 | }, 935 | { 936 | "Fn::Join": [ 937 | "", 938 | [ 939 | "arn:aws:kinesis:", 940 | { 941 | "Ref": "AWS::Region" 942 | }, 943 | ":", 944 | { 945 | "Ref": "AWS::AccountId" 946 | }, 947 | ":stream/", 948 | { 949 | "Ref": "ExistingStreamName" 950 | } 951 | ] 952 | ] 953 | } 954 | ] 955 | }, 956 | "FunctionName": { 957 | "Fn::GetAtt": [ 958 | "PersistRawDataFunction", 959 | "Arn" 960 | ] 961 | }, 962 | "StartingPosition": "LATEST" 963 | } 964 | }, 965 | "FIrehoseToS3Role": { 966 | "Type": "AWS::IAM::Role", 967 | "Condition": "S3Destination", 968 | "Properties": { 969 | "AssumeRolePolicyDocument": { 970 | "Version": "2012-10-17", 971 | "Statement": [ 972 | { 973 | "Effect": "Allow", 974 | "Principal": { 975 | "Service": [ 976 | "firehose.amazonaws.com" 977 | ] 978 | }, 979 | "Action": [ 980 | "sts:AssumeRole" 981 | ], 982 | "Condition": { 983 | "StringEquals": { 984 | "sts:ExternalId": { 985 | "Ref": "AWS::AccountId" 986 | } 987 | } 988 | } 989 | } 990 | ] 991 | }, 992 | "Policies": [ 993 | { 994 | "PolicyName": { 995 | "Fn::Join": [ 996 | "", 997 | [ 998 | "Firehose-to-S3-Delivery-", 999 | { 1000 | "Ref": "AWS::StackName" 1001 | } 1002 | ] 1003 | ] 1004 | }, 1005 | "PolicyDocument": { 1006 | "Version": "2012-10-17", 1007 | "Statement": [ 1008 | { 1009 | "Sid": "", 1010 | "Effect": "Allow", 1011 | "Action": [ 1012 | "s3:AbortMultipartUpload", 1013 | "s3:GetBucketLocation", 1014 | "s3:GetObject", 1015 | "s3:ListBucket", 1016 | "s3:ListBucketMultipartUploads", 1017 | "s3:PutObject" 1018 | ], 1019 | "Resource": [ 1020 | { 1021 | "Fn::Join": [ 1022 | "", 1023 | [ 1024 | "arn:aws:s3:::", 1025 | { 1026 | "Ref": "StreamingAnalyticsBucket" 1027 | } 1028 | ] 1029 | ] 1030 | }, 1031 | { 1032 | "Fn::Join": [ 1033 | "", 1034 | [ 1035 | "arn:aws:s3:::", 1036 | { 1037 | "Ref": "StreamingAnalyticsBucket" 1038 | }, 1039 | "/*" 1040 | ] 1041 | ] 1042 | } 1043 | ] 1044 | }, 1045 | { 1046 | "Effect": "Allow", 1047 | "Action": [ 1048 | "kms:Decrypt", 1049 | "kms:GenerateDataKey" 1050 | ], 1051 | "Resource": [ 1052 | { 1053 | "Fn::If": [ 1054 | "NoEncryption", 1055 | { 1056 | "Fn::Join": [ 1057 | "", 1058 | [ 1059 | "arn:aws:kms:", 1060 | { 1061 | "Ref": "AWS::Region" 1062 | }, 1063 | ":", 1064 | { 1065 | "Ref": "AWS::AccountId" 1066 | }, 1067 | ":key/placeholder-kms-id" 1068 | ] 1069 | ] 1070 | }, 1071 | { 1072 | "Fn::GetAtt": [ 1073 | "EncryptionKey", 1074 | "Arn" 1075 | ] 1076 | } 1077 | ] 1078 | } 1079 | ], 1080 | "Condition": { 1081 | "StringEquals": { 1082 | "kms:ViaService": { 1083 | "Fn::Join": [ 1084 | "", 1085 | [ 1086 | "s3.", 1087 | { 1088 | "Ref": "AWS::Region" 1089 | }, 1090 | ".amazonaws.com" 1091 | ] 1092 | ] 1093 | } 1094 | }, 1095 | "StringLike": { 1096 | "kms:EncryptionContext:aws:s3:arn": { 1097 | "Fn::Join": [ 1098 | "", 1099 | [ 1100 | "arn:aws:s3:::", 1101 | { 1102 | "Ref": "StreamingAnalyticsBucket" 1103 | }, 1104 | "/", 1105 | { 1106 | "Ref": "S3DestinationPrefix" 1107 | }, 1108 | "/*" 1109 | ] 1110 | ] 1111 | } 1112 | } 1113 | } 1114 | }, 1115 | { 1116 | "Sid": "", 1117 | "Effect": "Allow", 1118 | "Action": [ 1119 | "logs:PutLogEvents" 1120 | ], 1121 | "Resource": [ 1122 | "*" 1123 | ] 1124 | } 1125 | ] 1126 | } 1127 | } 1128 | ] 1129 | } 1130 | }, 1131 | "FIrehoseToS3": { 1132 | "Type": "AWS::KinesisFirehose::DeliveryStream", 1133 | "Condition": "S3Destination", 1134 | "Properties": { 1135 | "S3DestinationConfiguration": { 1136 | "BucketARN": { 1137 | "Fn::Join": [ 1138 | "", 1139 | [ 1140 | "arn:aws:s3:::", 1141 | { 1142 | "Ref": "StreamingAnalyticsBucket" 1143 | } 1144 | ] 1145 | ] 1146 | }, 1147 | "BufferingHints": { 1148 | "IntervalInSeconds": { 1149 | "Ref": "S3BufferInterval" 1150 | }, 1151 | "SizeInMBs": { 1152 | "Ref": "S3BufferSize" 1153 | } 1154 | }, 1155 | "CompressionFormat": "UNCOMPRESSED", 1156 | "EncryptionConfiguration": { 1157 | "KMSEncryptionConfig": { 1158 | "Fn::If": [ 1159 | "NoEncryption", 1160 | { 1161 | "Ref": "AWS::NoValue" 1162 | }, 1163 | { 1164 | "AWSKMSKeyARN": { 1165 | "Fn::GetAtt": [ 1166 | "EncryptionKey", 1167 | "Arn" 1168 | ] 1169 | } 1170 | } 1171 | ] 1172 | }, 1173 | "NoEncryptionConfig": { 1174 | "Fn::If": [ 1175 | "NoEncryption", 1176 | "NoEncryption", 1177 | { 1178 | "Ref": "AWS::NoValue" 1179 | } 1180 | ] 1181 | } 1182 | }, 1183 | "Prefix": { 1184 | "Fn::Join": [ 1185 | "", 1186 | [ 1187 | { 1188 | "Ref": "S3DestinationPrefix" 1189 | }, 1190 | "/" 1191 | ] 1192 | ] 1193 | }, 1194 | "RoleARN": { 1195 | "Fn::GetAtt": [ 1196 | "FIrehoseToS3Role", 1197 | "Arn" 1198 | ] 1199 | } 1200 | } 1201 | } 1202 | }, 1203 | "ElasticSearchDeliveryRole": { 1204 | "Type": "AWS::IAM::Role", 1205 | "Condition": "ElasticsearchDestination", 1206 | "Properties": { 1207 | "AssumeRolePolicyDocument": { 1208 | "Version": "2012-10-17", 1209 | "Statement": [ 1210 | { 1211 | "Effect": "Allow", 1212 | "Principal": { 1213 | "Service": [ 1214 | "firehose.amazonaws.com" 1215 | ] 1216 | }, 1217 | "Action": [ 1218 | "sts:AssumeRole" 1219 | ], 1220 | "Condition": { 1221 | "StringEquals": { 1222 | "sts:ExternalId": { 1223 | "Ref": "AWS::AccountId" 1224 | } 1225 | } 1226 | } 1227 | } 1228 | ] 1229 | }, 1230 | "Policies": [ 1231 | { 1232 | "PolicyName": { 1233 | "Fn::Join": [ 1234 | "", 1235 | [ 1236 | "Firehose-to-ElasticSearch-Delivery-", 1237 | { 1238 | "Ref": "AWS::StackName" 1239 | } 1240 | ] 1241 | ] 1242 | }, 1243 | "PolicyDocument": { 1244 | "Version": "2012-10-17", 1245 | "Statement": [ 1246 | { 1247 | "Sid": "", 1248 | "Effect": "Allow", 1249 | "Action": [ 1250 | "s3:AbortMultipartUpload", 1251 | "s3:GetBucketLocation", 1252 | "s3:GetObject", 1253 | "s3:ListBucket", 1254 | "s3:ListBucketMultipartUploads", 1255 | "s3:PutObject" 1256 | ], 1257 | "Resource": [ 1258 | { 1259 | "Fn::Join": [ 1260 | "", 1261 | [ 1262 | "arn:aws:s3:::", 1263 | { 1264 | "Ref": "StreamingAnalyticsBucket" 1265 | } 1266 | ] 1267 | ] 1268 | }, 1269 | { 1270 | "Fn::Join": [ 1271 | "", 1272 | [ 1273 | "arn:aws:s3:::", 1274 | { 1275 | "Ref": "StreamingAnalyticsBucket" 1276 | }, 1277 | "/*" 1278 | ] 1279 | ] 1280 | } 1281 | ] 1282 | }, 1283 | { 1284 | "Effect": "Allow", 1285 | "Action": [ 1286 | "kms:Decrypt", 1287 | "kms:GenerateDataKey" 1288 | ], 1289 | "Resource": [ 1290 | { 1291 | "Fn::If": [ 1292 | "NoEncryption", 1293 | { 1294 | "Fn::Join": [ 1295 | "", 1296 | [ 1297 | "arn:aws:kms:", 1298 | { 1299 | "Ref": "AWS::Region" 1300 | }, 1301 | ":", 1302 | { 1303 | "Ref": "AWS::AccountId" 1304 | }, 1305 | ":key/placeholder-kms-id" 1306 | ] 1307 | ] 1308 | }, 1309 | { 1310 | "Fn::GetAtt": [ 1311 | "EncryptionKey", 1312 | "Arn" 1313 | ] 1314 | } 1315 | ] 1316 | } 1317 | ], 1318 | "Condition": { 1319 | "StringEquals": { 1320 | "kms:ViaService": { 1321 | "Fn::Join": [ 1322 | "", 1323 | [ 1324 | "s3.", 1325 | { 1326 | "Ref": "AWS::Region" 1327 | }, 1328 | ".amazonaws.com" 1329 | ] 1330 | ] 1331 | } 1332 | }, 1333 | "StringLike": { 1334 | "kms:EncryptionContext:aws:s3:arn": { 1335 | "Fn::Join": [ 1336 | "", 1337 | [ 1338 | "arn:aws:s3:::", 1339 | { 1340 | "Ref": "StreamingAnalyticsBucket" 1341 | }, 1342 | "/ESDeliveryBackup/*" 1343 | ] 1344 | ] 1345 | } 1346 | } 1347 | } 1348 | }, 1349 | { 1350 | "Sid": "", 1351 | "Effect": "Allow", 1352 | "Action": [ 1353 | "es:DescribeElasticsearchDomain", 1354 | "es:DescribeElasticsearchDomains", 1355 | "es:DescribeElasticsearchDomainConfig", 1356 | "es:ESHttpPost", 1357 | "es:ESHttpPut" 1358 | ], 1359 | "Resource": [ 1360 | { 1361 | "Fn::Join": [ 1362 | "", 1363 | [ 1364 | "arn:aws:es:", 1365 | { 1366 | "Ref": "AWS::Region" 1367 | }, 1368 | ":", 1369 | { 1370 | "Ref": "AWS::AccountId" 1371 | }, 1372 | ":domain/", 1373 | { 1374 | "Ref": "ESDomain" 1375 | } 1376 | ] 1377 | ] 1378 | }, 1379 | { 1380 | "Fn::Join": [ 1381 | "", 1382 | [ 1383 | "arn:aws:es:", 1384 | { 1385 | "Ref": "AWS::Region" 1386 | }, 1387 | ":", 1388 | { 1389 | "Ref": "AWS::AccountId" 1390 | }, 1391 | ":domain/", 1392 | { 1393 | "Ref": "ESDomain" 1394 | }, 1395 | "/*" 1396 | ] 1397 | ] 1398 | } 1399 | ] 1400 | }, 1401 | { 1402 | "Sid": "", 1403 | "Effect": "Allow", 1404 | "Action": [ 1405 | "logs:PutLogEvents" 1406 | ], 1407 | "Resource": [ 1408 | "*" 1409 | ] 1410 | } 1411 | ] 1412 | } 1413 | } 1414 | ] 1415 | } 1416 | }, 1417 | "FIrehoseToElasticsearch": { 1418 | "Type": "AWS::KinesisFirehose::DeliveryStream", 1419 | "Condition": "ElasticsearchDestination", 1420 | "Properties": { 1421 | "ElasticsearchDestinationConfiguration": { 1422 | "BufferingHints": { 1423 | "IntervalInSeconds": { 1424 | "Ref": "ESBufferInterval" 1425 | }, 1426 | "SizeInMBs": { 1427 | "Ref": "ESBufferSize" 1428 | } 1429 | }, 1430 | "DomainARN": { 1431 | "Fn::Join": [ 1432 | "", 1433 | [ 1434 | "arn:aws:es:", 1435 | { 1436 | "Ref": "AWS::Region" 1437 | }, 1438 | ":", 1439 | { 1440 | "Ref": "AWS::AccountId" 1441 | }, 1442 | ":domain/", 1443 | { 1444 | "Ref": "ESDomain" 1445 | } 1446 | ] 1447 | ] 1448 | }, 1449 | "IndexName": { 1450 | "Ref": "ESIndex" 1451 | }, 1452 | "IndexRotationPeriod": { 1453 | "Ref": "ESIndexRotation" 1454 | }, 1455 | "RetryOptions": { 1456 | "DurationInSeconds": 300 1457 | }, 1458 | "RoleARN": { 1459 | "Fn::GetAtt": [ 1460 | "ElasticSearchDeliveryRole", 1461 | "Arn" 1462 | ] 1463 | }, 1464 | "S3BackupMode": "AllDocuments", 1465 | "S3Configuration": { 1466 | "BucketARN": { 1467 | "Fn::Join": [ 1468 | "", 1469 | [ 1470 | "arn:aws:s3:::", 1471 | { 1472 | "Ref": "StreamingAnalyticsBucket" 1473 | } 1474 | ] 1475 | ] 1476 | }, 1477 | "BufferingHints": { 1478 | "IntervalInSeconds": { 1479 | "Ref": "ESBufferInterval" 1480 | }, 1481 | "SizeInMBs": { 1482 | "Ref": "ESBufferSize" 1483 | } 1484 | }, 1485 | "CompressionFormat": "UNCOMPRESSED", 1486 | "EncryptionConfiguration": { 1487 | "KMSEncryptionConfig": { 1488 | "Fn::If": [ 1489 | "NoEncryption", 1490 | { 1491 | "Ref": "AWS::NoValue" 1492 | }, 1493 | { 1494 | "AWSKMSKeyARN": { 1495 | "Fn::GetAtt": [ 1496 | "EncryptionKey", 1497 | "Arn" 1498 | ] 1499 | } 1500 | } 1501 | ] 1502 | }, 1503 | "NoEncryptionConfig": { 1504 | "Fn::If": [ 1505 | "NoEncryption", 1506 | "NoEncryption", 1507 | { 1508 | "Ref": "AWS::NoValue" 1509 | } 1510 | ] 1511 | } 1512 | }, 1513 | "Prefix": "ESDeliveryBackup/", 1514 | "RoleARN": { 1515 | "Fn::GetAtt": [ 1516 | "ElasticSearchDeliveryRole", 1517 | "Arn" 1518 | ] 1519 | } 1520 | }, 1521 | "TypeName": { 1522 | "Ref": "ESType" 1523 | } 1524 | } 1525 | } 1526 | }, 1527 | "RedshiftDeliveryRole": { 1528 | "Type": "AWS::IAM::Role", 1529 | "Condition": "RedshiftDestination", 1530 | "Properties": { 1531 | "AssumeRolePolicyDocument": { 1532 | "Version": "2012-10-17", 1533 | "Statement": [ 1534 | { 1535 | "Effect": "Allow", 1536 | "Principal": { 1537 | "Service": [ 1538 | "firehose.amazonaws.com" 1539 | ] 1540 | }, 1541 | "Action": [ 1542 | "sts:AssumeRole" 1543 | ], 1544 | "Condition": { 1545 | "StringEquals": { 1546 | "sts:ExternalId": { 1547 | "Ref": "AWS::AccountId" 1548 | } 1549 | } 1550 | } 1551 | } 1552 | ] 1553 | }, 1554 | "Policies": [ 1555 | { 1556 | "PolicyName": { 1557 | "Fn::Join": [ 1558 | "", 1559 | [ 1560 | "Firehose-to-Redshift-Delivery-", 1561 | { 1562 | "Ref": "AWS::StackName" 1563 | } 1564 | ] 1565 | ] 1566 | }, 1567 | "PolicyDocument": { 1568 | "Version": "2012-10-17", 1569 | "Statement": [ 1570 | { 1571 | "Sid": "", 1572 | "Effect": "Allow", 1573 | "Action": [ 1574 | "s3:AbortMultipartUpload", 1575 | "s3:GetBucketLocation", 1576 | "s3:GetObject", 1577 | "s3:ListBucket", 1578 | "s3:ListBucketMultipartUploads", 1579 | "s3:PutObject" 1580 | ], 1581 | "Resource": [ 1582 | { 1583 | "Fn::Join": [ 1584 | "", 1585 | [ 1586 | "arn:aws:s3:::", 1587 | { 1588 | "Ref": "StreamingAnalyticsBucket" 1589 | } 1590 | ] 1591 | ] 1592 | }, 1593 | { 1594 | "Fn::Join": [ 1595 | "", 1596 | [ 1597 | "arn:aws:s3:::", 1598 | { 1599 | "Ref": "StreamingAnalyticsBucket" 1600 | }, 1601 | "/*" 1602 | ] 1603 | ] 1604 | } 1605 | ] 1606 | }, 1607 | { 1608 | "Effect": "Allow", 1609 | "Action": [ 1610 | "kms:Decrypt", 1611 | "kms:GenerateDataKey" 1612 | ], 1613 | "Resource": [ 1614 | { 1615 | "Fn::If": [ 1616 | "NoEncryption", 1617 | { 1618 | "Fn::Join": [ 1619 | "", 1620 | [ 1621 | "arn:aws:kms:", 1622 | { 1623 | "Ref": "AWS::Region" 1624 | }, 1625 | ":", 1626 | { 1627 | "Ref": "AWS::AccountId" 1628 | }, 1629 | ":key/placeholder-kms-id" 1630 | ] 1631 | ] 1632 | }, 1633 | { 1634 | "Fn::GetAtt": [ 1635 | "EncryptionKey", 1636 | "Arn" 1637 | ] 1638 | } 1639 | ] 1640 | } 1641 | ], 1642 | "Condition": { 1643 | "StringEquals": { 1644 | "kms:ViaService": { 1645 | "Fn::Join": [ 1646 | "", 1647 | [ 1648 | "s3.", 1649 | { 1650 | "Ref": "AWS::Region" 1651 | }, 1652 | ".amazonaws.com" 1653 | ] 1654 | ] 1655 | } 1656 | }, 1657 | "StringLike": { 1658 | "kms:EncryptionContext:aws:s3:arn": { 1659 | "Fn::Join": [ 1660 | "", 1661 | [ 1662 | "arn:aws:s3:::", 1663 | { 1664 | "Ref": "StreamingAnalyticsBucket" 1665 | }, 1666 | "/FirehoseDelivery/*" 1667 | ] 1668 | ] 1669 | } 1670 | } 1671 | } 1672 | }, 1673 | { 1674 | "Sid": "", 1675 | "Effect": "Allow", 1676 | "Action": [ 1677 | "logs:PutLogEvents" 1678 | ], 1679 | "Resource": [ 1680 | "*" 1681 | ] 1682 | } 1683 | ] 1684 | } 1685 | } 1686 | ] 1687 | } 1688 | }, 1689 | "FIrehoseToRedshift": { 1690 | "Type": "AWS::KinesisFirehose::DeliveryStream", 1691 | "Condition": "RedshiftDestination", 1692 | "Properties": { 1693 | "RedshiftDestinationConfiguration": { 1694 | "ClusterJDBCURL": { 1695 | "Ref": "RedshiftJDBCURL" 1696 | }, 1697 | "CopyCommand": { 1698 | "CopyOptions": "delimiter ','", 1699 | "DataTableColumns": { 1700 | "Ref": "RedshiftColumns" 1701 | }, 1702 | "DataTableName": { 1703 | "Ref": "RedshiftTableName" 1704 | } 1705 | }, 1706 | "Password": { 1707 | "Ref": "MasterUserPassword" 1708 | }, 1709 | "RoleARN": { 1710 | "Fn::GetAtt": [ 1711 | "RedshiftDeliveryRole", 1712 | "Arn" 1713 | ] 1714 | }, 1715 | "S3Configuration": { 1716 | "BucketARN": { 1717 | "Fn::Join": [ 1718 | "", 1719 | [ 1720 | "arn:aws:s3:::", 1721 | { 1722 | "Ref": "StreamingAnalyticsBucket" 1723 | } 1724 | ] 1725 | ] 1726 | }, 1727 | "RoleARN": { 1728 | "Fn::GetAtt": [ 1729 | "RedshiftDeliveryRole", 1730 | "Arn" 1731 | ] 1732 | }, 1733 | "BufferingHints": { 1734 | "IntervalInSeconds": { 1735 | "Ref": "RedshiftBufferInterval" 1736 | }, 1737 | "SizeInMBs": { 1738 | "Ref": "RedshiftBufferSize" 1739 | } 1740 | }, 1741 | "CompressionFormat": "UNCOMPRESSED", 1742 | "EncryptionConfiguration": { 1743 | "KMSEncryptionConfig": { 1744 | "Fn::If": [ 1745 | "NoEncryption", 1746 | { 1747 | "Ref": "AWS::NoValue" 1748 | }, 1749 | { 1750 | "AWSKMSKeyARN": { 1751 | "Fn::GetAtt": [ 1752 | "EncryptionKey", 1753 | "Arn" 1754 | ] 1755 | } 1756 | } 1757 | ] 1758 | }, 1759 | "NoEncryptionConfig": { 1760 | "Fn::If": [ 1761 | "NoEncryption", 1762 | "NoEncryption", 1763 | { 1764 | "Ref": "AWS::NoValue" 1765 | } 1766 | ] 1767 | } 1768 | }, 1769 | "Prefix": "FirehoseDelivery/" 1770 | }, 1771 | "Username": { 1772 | "Ref": "MasterUser" 1773 | } 1774 | } 1775 | } 1776 | }, 1777 | "LambdaRoleCreateKAApp": { 1778 | "Type": "AWS::IAM::Role", 1779 | "Properties": { 1780 | "AssumeRolePolicyDocument": { 1781 | "Version": "2012-10-17", 1782 | "Statement": [ 1783 | { 1784 | "Effect": "Allow", 1785 | "Principal": { 1786 | "Service": [ 1787 | "lambda.amazonaws.com" 1788 | ] 1789 | }, 1790 | "Action": [ 1791 | "sts:AssumeRole" 1792 | ] 1793 | }, 1794 | { 1795 | "Effect": "Allow", 1796 | "Principal": { 1797 | "AWS": { 1798 | "Ref": "AWS::AccountId" 1799 | } 1800 | }, 1801 | "Action": "sts:AssumeRole" 1802 | } 1803 | ] 1804 | }, 1805 | "Policies": [ 1806 | { 1807 | "PolicyName": { 1808 | "Fn::Join": [ 1809 | "", 1810 | [ 1811 | "Lambda-Execution-", 1812 | { 1813 | "Ref": "AWS::StackName" 1814 | } 1815 | ] 1816 | ] 1817 | }, 1818 | "PolicyDocument": { 1819 | "Version": "2012-10-17", 1820 | "Statement": [ 1821 | { 1822 | "Effect": "Allow", 1823 | "Action": [ 1824 | "logs:CreateLogGroup", 1825 | "logs:CreateLogStream", 1826 | "logs:PutLogEvents" 1827 | ], 1828 | "Resource": "*" 1829 | }, 1830 | { 1831 | "Effect": "Allow", 1832 | "Action": [ 1833 | "kinesisanalytics:CreateApplication", 1834 | "kinesisanalytics:DeleteApplication", 1835 | "kinesisanalytics:DescribeApplication" 1836 | ], 1837 | "Resource": { 1838 | "Fn::Join": [ 1839 | "", 1840 | [ 1841 | "arn:aws:kinesisanalytics:", 1842 | { 1843 | "Ref": "AWS::Region" 1844 | }, 1845 | ":", 1846 | { 1847 | "Ref": "AWS::AccountId" 1848 | }, 1849 | ":application/*" 1850 | ] 1851 | ] 1852 | } 1853 | }, 1854 | { 1855 | "Effect": "Allow", 1856 | "Action": [ 1857 | "kinesisanalytics:ListApplications" 1858 | ], 1859 | "Resource": "*" 1860 | }, 1861 | { 1862 | "Effect": "Allow", 1863 | "Action": [ 1864 | "cloudwatch:GetMetricStatistics" 1865 | ], 1866 | "Resource": "*" 1867 | }, 1868 | { 1869 | "Effect": "Allow", 1870 | "Action": [ 1871 | "s3:GetObject" 1872 | ], 1873 | "Resource": { 1874 | "Fn::Join": [ 1875 | "", 1876 | [ 1877 | "arn:aws:s3:::", 1878 | { 1879 | "Ref": "ConfigFileLocation" 1880 | } 1881 | ] 1882 | ] 1883 | } 1884 | }, 1885 | { 1886 | "Effect": "Allow", 1887 | "Action": [ 1888 | "iam:PassRole" 1889 | ], 1890 | "Resource": { 1891 | "Fn::GetAtt": [ 1892 | "KinesisAnalyticsRole", 1893 | "Arn" 1894 | ] 1895 | } 1896 | }, 1897 | { 1898 | "Effect": "Allow", 1899 | "Action": [ 1900 | "cloudformation:DescribeStacks" 1901 | ], 1902 | "Resource": { 1903 | "Ref": "AWS::StackId" 1904 | } 1905 | } 1906 | ] 1907 | } 1908 | } 1909 | ] 1910 | } 1911 | }, 1912 | "KinesisAnalyticsRole": { 1913 | "Type": "AWS::IAM::Role", 1914 | "Properties": { 1915 | "AssumeRolePolicyDocument": { 1916 | "Version": "2012-10-17", 1917 | "Statement": [ 1918 | { 1919 | "Effect": "Allow", 1920 | "Principal": { 1921 | "Service": "kinesisanalytics.amazonaws.com" 1922 | }, 1923 | "Action": "sts:AssumeRole" 1924 | } 1925 | ] 1926 | }, 1927 | "Policies": [ 1928 | { 1929 | "PolicyName": { 1930 | "Fn::Join": [ 1931 | "", 1932 | [ 1933 | "KinesisAnalyticsRole-", 1934 | { 1935 | "Ref": "AWS::StackName" 1936 | } 1937 | ] 1938 | ] 1939 | }, 1940 | "PolicyDocument": { 1941 | "Version": "2012-10-17", 1942 | "Statement": [ 1943 | { 1944 | "Sid": "ReadInputKinesis", 1945 | "Effect": "Allow", 1946 | "Action": [ 1947 | "kinesis:DescribeStream", 1948 | "kinesis:GetShardIterator", 1949 | "kinesis:GetRecords" 1950 | ], 1951 | "Resource": [ 1952 | { 1953 | "Fn::If": [ 1954 | "SourceIsNewStream", 1955 | { 1956 | "Fn::GetAtt": [ 1957 | "SourceStream", 1958 | "Arn" 1959 | ] 1960 | }, 1961 | { 1962 | "Fn::Join": [ 1963 | "", 1964 | [ 1965 | "arn:aws:kinesis:", 1966 | { 1967 | "Ref": "AWS::Region" 1968 | }, 1969 | ":", 1970 | { 1971 | "Ref": "AWS::AccountId" 1972 | }, 1973 | ":stream/", 1974 | { 1975 | "Ref": "ExistingStreamName" 1976 | } 1977 | ] 1978 | ] 1979 | } 1980 | ] 1981 | } 1982 | ] 1983 | }, 1984 | { 1985 | "Sid": "WriteOutputFirehose", 1986 | "Effect": "Allow", 1987 | "Action": [ 1988 | "firehose:DescribeDeliveryStream", 1989 | "firehose:PutRecord", 1990 | "firehose:PutRecordBatch" 1991 | ], 1992 | "Resource": [ 1993 | { 1994 | "Fn::If": [ 1995 | "RedshiftDestination", 1996 | { 1997 | "Fn::Join": [ 1998 | "", 1999 | [ 2000 | "arn:aws:firehose:", 2001 | { 2002 | "Ref": "AWS::Region" 2003 | }, 2004 | ":", 2005 | { 2006 | "Ref": "AWS::AccountId" 2007 | }, 2008 | ":deliverystream/", 2009 | { 2010 | "Ref": "FIrehoseToRedshift" 2011 | } 2012 | ] 2013 | ] 2014 | }, 2015 | { 2016 | "Ref": "AWS::NoValue" 2017 | } 2018 | ] 2019 | }, 2020 | { 2021 | "Fn::If": [ 2022 | "S3Destination", 2023 | { 2024 | "Fn::Join": [ 2025 | "", 2026 | [ 2027 | "arn:aws:firehose:", 2028 | { 2029 | "Ref": "AWS::Region" 2030 | }, 2031 | ":", 2032 | { 2033 | "Ref": "AWS::AccountId" 2034 | }, 2035 | ":deliverystream/", 2036 | { 2037 | "Ref": "FIrehoseToS3" 2038 | } 2039 | ] 2040 | ] 2041 | }, 2042 | { 2043 | "Ref": "AWS::NoValue" 2044 | } 2045 | ] 2046 | }, 2047 | { 2048 | "Fn::If": [ 2049 | "ElasticsearchDestination", 2050 | { 2051 | "Fn::Join": [ 2052 | "", 2053 | [ 2054 | "arn:aws:firehose:", 2055 | { 2056 | "Ref": "AWS::Region" 2057 | }, 2058 | ":", 2059 | { 2060 | "Ref": "AWS::AccountId" 2061 | }, 2062 | ":deliverystream/", 2063 | { 2064 | "Ref": "FIrehoseToElasticsearch" 2065 | } 2066 | ] 2067 | ] 2068 | }, 2069 | { 2070 | "Ref": "AWS::NoValue" 2071 | } 2072 | ] 2073 | }, 2074 | { 2075 | "Fn::If": [ 2076 | "StreamDestination", 2077 | { 2078 | "Fn::Join": [ 2079 | "", 2080 | [ 2081 | "arn:aws:kinesis:", 2082 | { 2083 | "Ref": "AWS::Region" 2084 | }, 2085 | ":", 2086 | { 2087 | "Ref": "AWS::AccountId" 2088 | }, 2089 | ":stream/kinesis-analytics-placeholder-stream-destination" 2090 | ] 2091 | ] 2092 | }, 2093 | { 2094 | "Ref": "AWS::NoValue" 2095 | } 2096 | ] 2097 | } 2098 | ] 2099 | }, 2100 | { 2101 | "Sid": "WriteOutputKinesis", 2102 | "Effect": "Allow", 2103 | "Action": [ 2104 | "kinesis:DescribeStream", 2105 | "kinesis:PutRecord", 2106 | "kinesis:PutRecords" 2107 | ], 2108 | "Resource": [ 2109 | { 2110 | "Fn::If": [ 2111 | "StreamDestination", 2112 | { 2113 | "Fn::Join": [ 2114 | "", 2115 | [ 2116 | "arn:aws:kinesis:", 2117 | { 2118 | "Ref": "AWS::Region" 2119 | }, 2120 | ":", 2121 | { 2122 | "Ref": "AWS::AccountId" 2123 | }, 2124 | ":stream/", 2125 | { 2126 | "Ref": "DestinationStream" 2127 | } 2128 | ] 2129 | ] 2130 | }, 2131 | { 2132 | "Fn::Join": [ 2133 | "", 2134 | [ 2135 | "arn:aws:kinesis:", 2136 | { 2137 | "Ref": "AWS::Region" 2138 | }, 2139 | ":", 2140 | { 2141 | "Ref": "AWS::AccountId" 2142 | }, 2143 | ":stream/kinesis-analytics-placeholder-stream-destination" 2144 | ] 2145 | ] 2146 | } 2147 | ] 2148 | } 2149 | ] 2150 | } 2151 | ] 2152 | } 2153 | } 2154 | ] 2155 | } 2156 | }, 2157 | "KALambdaFunction": { 2158 | "Type": "AWS::Lambda::Function", 2159 | "DependsOn": [ 2160 | "LambdaRoleCreateKAApp" 2161 | ], 2162 | "Properties": { 2163 | "Code": { 2164 | "S3Bucket": {"Fn::Join": ["", [ 2165 | "solutions-", 2166 | {"Ref": "AWS::Region"} 2167 | ]]}, 2168 | "S3Key": "streaming-analytics/v1/streaming-analytics-code.zip" 2169 | }, 2170 | "Description": "This function configures and lunches a Kinesis Analytics Application", 2171 | "Handler": "lambda_function.lambda_handler", 2172 | "MemorySize": "128", 2173 | "Role": { 2174 | "Fn::GetAtt": [ 2175 | "LambdaRoleCreateKAApp", 2176 | "Arn" 2177 | ] 2178 | }, 2179 | "Runtime": "python2.7", 2180 | "Timeout": "30" 2181 | } 2182 | }, 2183 | "CreateKAApplication": { 2184 | "Type": "Custom::KinesisAnalyticsApp", 2185 | "Properties": { 2186 | "ServiceToken": { 2187 | "Fn::GetAtt": [ 2188 | "KALambdaFunction", 2189 | "Arn" 2190 | ] 2191 | }, 2192 | "FunctionName": {"Ref": "KALambdaFunction"}, 2193 | "Region": {"Ref": "AWS::Region"}, 2194 | "ConfigFile": { 2195 | "Fn::If": [ 2196 | "ConfigFileNotProvidedCondition", 2197 | { 2198 | "Ref": "AWS::NoValue" 2199 | }, 2200 | { 2201 | "Ref": "ConfigFileLocation" 2202 | } 2203 | ] 2204 | }, 2205 | "KAAppName": "KinesisAnalyticsApp-", 2206 | "SourceStream": { 2207 | "Fn::If": [ 2208 | "SourceIsNewStream", 2209 | { 2210 | "Fn::GetAtt": [ 2211 | "SourceStream", 2212 | "Arn" 2213 | ] 2214 | }, 2215 | { 2216 | "Fn::Join": [ 2217 | "", 2218 | [ 2219 | "arn:aws:kinesis:", 2220 | { 2221 | "Ref": "AWS::Region" 2222 | }, 2223 | ":", 2224 | { 2225 | "Ref": "AWS::AccountId" 2226 | }, 2227 | ":stream/", 2228 | { 2229 | "Ref": "ExistingStreamName" 2230 | } 2231 | ] 2232 | ] 2233 | } 2234 | ] 2235 | }, 2236 | "KARole": { 2237 | "Fn::GetAtt": [ 2238 | "KinesisAnalyticsRole", 2239 | "Arn" 2240 | ] 2241 | }, 2242 | "Destination": { 2243 | "Ref": "ExternalDestination" 2244 | }, 2245 | "RedshiftDest": { 2246 | "Fn::If": [ 2247 | "RedshiftDestination", 2248 | { 2249 | "Fn::Join": [ 2250 | "", 2251 | [ 2252 | "arn:aws:firehose:", 2253 | { 2254 | "Ref": "AWS::Region" 2255 | }, 2256 | ":", 2257 | { 2258 | "Ref": "AWS::AccountId" 2259 | }, 2260 | ":deliverystream/", 2261 | { 2262 | "Ref": "FIrehoseToRedshift" 2263 | } 2264 | ] 2265 | ] 2266 | }, 2267 | { 2268 | "Ref": "AWS::NoValue" 2269 | } 2270 | ] 2271 | }, 2272 | "S3Dest": { 2273 | "Fn::If": [ 2274 | "S3Destination", 2275 | { 2276 | "Fn::Join": [ 2277 | "", 2278 | [ 2279 | "arn:aws:firehose:", 2280 | { 2281 | "Ref": "AWS::Region" 2282 | }, 2283 | ":", 2284 | { 2285 | "Ref": "AWS::AccountId" 2286 | }, 2287 | ":deliverystream/", 2288 | { 2289 | "Ref": "FIrehoseToS3" 2290 | } 2291 | ] 2292 | ] 2293 | }, 2294 | { 2295 | "Ref": "AWS::NoValue" 2296 | } 2297 | ] 2298 | }, 2299 | "ElasticsearchDest": { 2300 | "Fn::If": [ 2301 | "ElasticsearchDestination", 2302 | { 2303 | "Fn::Join": [ 2304 | "", 2305 | [ 2306 | "arn:aws:firehose:", 2307 | { 2308 | "Ref": "AWS::Region" 2309 | }, 2310 | ":", 2311 | { 2312 | "Ref": "AWS::AccountId" 2313 | }, 2314 | ":deliverystream/", 2315 | { 2316 | "Ref": "FIrehoseToElasticsearch" 2317 | } 2318 | ] 2319 | ] 2320 | }, 2321 | { 2322 | "Ref": "AWS::NoValue" 2323 | } 2324 | ] 2325 | }, 2326 | "StreamDest": { 2327 | "Fn::If": [ 2328 | "StreamDestination", 2329 | { 2330 | "Fn::Join": [ 2331 | "", 2332 | [ 2333 | "arn:aws:kinesis:", 2334 | { 2335 | "Ref": "AWS::Region" 2336 | }, 2337 | ":", 2338 | { 2339 | "Ref": "AWS::AccountId" 2340 | }, 2341 | ":stream/", 2342 | { 2343 | "Ref": "DestinationStream" 2344 | } 2345 | ] 2346 | ] 2347 | }, 2348 | { 2349 | "Ref": "AWS::NoValue" 2350 | } 2351 | ] 2352 | }, 2353 | "PersistRawData": { 2354 | "Ref": "PersistRawDataParam" 2355 | }, 2356 | "AnonymousData": { 2357 | "Ref": "CollectAnonymousData" 2358 | } 2359 | } 2360 | }, 2361 | "SendAnonymousDataCloudWatchRole": { 2362 | "Type": "AWS::IAM::Role", 2363 | "Condition": "AnonymousDataCondition", 2364 | "Properties": { 2365 | "AssumeRolePolicyDocument": { 2366 | "Version": "2012-10-17", 2367 | "Statement": [ 2368 | { 2369 | "Effect": "Allow", 2370 | "Principal": { 2371 | "Service": [ 2372 | "events.amazonaws.com" 2373 | ] 2374 | }, 2375 | "Action": [ 2376 | "sts:AssumeRole" 2377 | ] 2378 | } 2379 | ] 2380 | }, 2381 | "Policies": [ 2382 | { 2383 | "PolicyName": { 2384 | "Fn::Join": [ 2385 | "", 2386 | [ 2387 | "SendAnonymousData-CloudWatchEvent-Role-", 2388 | { 2389 | "Ref": "AWS::StackName" 2390 | } 2391 | ] 2392 | ] 2393 | }, 2394 | "PolicyDocument": { 2395 | "Version": "2012-10-17", 2396 | "Statement": [ 2397 | { 2398 | "Sid": "", 2399 | "Effect": "Allow", 2400 | "Action": [ 2401 | "lambda:InvokeFunction" 2402 | ], 2403 | "Resource": [ 2404 | { 2405 | "Fn::GetAtt": [ 2406 | "KALambdaFunction", 2407 | "Arn" 2408 | ] 2409 | } 2410 | ] 2411 | }, 2412 | { 2413 | "Sid": "", 2414 | "Effect": "Allow", 2415 | "Action": [ 2416 | "logs:PutLogEvents" 2417 | ], 2418 | "Resource": [ 2419 | "*" 2420 | ] 2421 | } 2422 | ] 2423 | } 2424 | } 2425 | ] 2426 | } 2427 | }, 2428 | "SendAnonymousMetricsTrigger": { 2429 | "Type": "AWS::Events::Rule", 2430 | "Condition": "AnonymousDataCondition", 2431 | "Properties": { 2432 | "Description": "Invokes the KALambdaFunction every 15 minutes to collect and send anonymous metrics on solution usage to AWS.", 2433 | "Name": { 2434 | "Fn::Join": [ 2435 | "",[ 2436 | {"Fn::GetAtt": ["CreateKAApplication","AppName"]}, 2437 | "-metrics"]]}, 2438 | "ScheduleExpression": "rate(15 minutes)", 2439 | "RoleArn": { 2440 | "Fn::GetAtt": [ 2441 | "SendAnonymousDataCloudWatchRole", 2442 | "Arn" 2443 | ] 2444 | }, 2445 | "State": "ENABLED", 2446 | "Targets": [ 2447 | { 2448 | "Arn": { 2449 | "Fn::GetAtt": [ 2450 | "KALambdaFunction", 2451 | "Arn" 2452 | ] 2453 | }, 2454 | "Id": { 2455 | "Ref": "KALambdaFunction" 2456 | }, 2457 | "Input": { 2458 | "Fn::Join": [ 2459 | "", 2460 | [ 2461 | "{\"RequestType\" :\"SendAnonymousMetrics\",\"AppName\":\"", 2462 | { 2463 | "Fn::GetAtt": [ 2464 | "CreateKAApplication", 2465 | "AppName" 2466 | ] 2467 | }, 2468 | "\",\"uuid\":\"", 2469 | { 2470 | "Fn::GetAtt": [ 2471 | "CreateKAApplication", 2472 | "UUID" 2473 | ] 2474 | }, 2475 | "\"}" 2476 | ] 2477 | ] 2478 | } 2479 | } 2480 | ] 2481 | } 2482 | } 2483 | }, 2484 | "Outputs": { 2485 | "SourceStream": { 2486 | "Description": "Stream your data to this Amazon Kinesis Stream", 2487 | "Value": { 2488 | "Fn::If": [ 2489 | "SourceIsNewStream", 2490 | { 2491 | "Ref": "SourceStream" 2492 | }, 2493 | { 2494 | "Ref": "ExistingStreamName" 2495 | } 2496 | ] 2497 | } 2498 | }, 2499 | "Application": { 2500 | "Description": "Validate and start this Amazon Kinesis Analytics Application", 2501 | "Value": { 2502 | "Fn::GetAtt": [ 2503 | "CreateKAApplication", 2504 | "AppName" 2505 | ] 2506 | } 2507 | }, 2508 | "SolutionBucket": { 2509 | "Description": "Data will be stored in this Amazon S3 bucket", 2510 | "Value": { 2511 | "Ref": "StreamingAnalyticsBucket" 2512 | } 2513 | }, 2514 | "KMSKey": { 2515 | "Description": "Data will be encrypted with this Amazon KMS encryption key", 2516 | "Value": { 2517 | "Ref": "KMSAlias" 2518 | }, 2519 | "Condition": "EncryptData" 2520 | }, 2521 | "UUID": { 2522 | "Description": "UUID of the soln", 2523 | "Value": { 2524 | "Fn::GetAtt": [ 2525 | "CreateKAApplication", 2526 | "UUID" 2527 | ] 2528 | } 2529 | } 2530 | } 2531 | } -------------------------------------------------------------------------------- /code/add-output-code.zip: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/amazon-archives/streaming-analytics-pipeline/149c68276ad46baa6613be3de14fb603531f574d/code/add-output-code.zip -------------------------------------------------------------------------------- /code/streaming-analytics-code.zip: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/amazon-archives/streaming-analytics-pipeline/149c68276ad46baa6613be3de14fb603531f574d/code/streaming-analytics-code.zip --------------------------------------------------------------------------------