├── .github └── PULL_REQUEST_TEMPLATE.md ├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── LICENSE ├── README.md ├── cfn ├── AddSubscriptionFilter.yml └── centralLogging.yml └── python ├── CentralLogging.py ├── init_account_central_logging.py └── lambda.py /.github/PULL_REQUEST_TEMPLATE.md: -------------------------------------------------------------------------------- 1 | *Issue #, if available:* 2 | 3 | *Description of changes:* 4 | 5 | 6 | By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice. 7 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | ## Code of Conduct 2 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 3 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 4 | opensource-codeofconduct@amazon.com with any additional questions or comments. 5 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing Guidelines 2 | 3 | Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional 4 | documentation, we greatly value feedback and contributions from our community. 5 | 6 | Please read through this document before submitting any issues or pull requests to ensure we have all the necessary 7 | information to effectively respond to your bug report or contribution. 8 | 9 | 10 | ## Reporting Bugs/Feature Requests 11 | 12 | We welcome you to use the GitHub issue tracker to report bugs or suggest features. 13 | 14 | When filing an issue, please check [existing open](https://github.com/aws-samples/amazon-cloudwatch-log-centralizer/issues), or [recently closed](https://github.com/aws-samples/amazon-cloudwatch-log-centralizer/issues?utf8=%E2%9C%93&q=is%3Aissue%20is%3Aclosed%20), issues to make sure somebody else hasn't already 15 | reported the issue. Please try to include as much information as you can. Details like these are incredibly useful: 16 | 17 | * A reproducible test case or series of steps 18 | * The version of our code being used 19 | * Any modifications you've made relevant to the bug 20 | * Anything unusual about your environment or deployment 21 | 22 | 23 | ## Contributing via Pull Requests 24 | Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that: 25 | 26 | 1. You are working against the latest source on the *master* branch. 27 | 2. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already. 28 | 3. You open an issue to discuss any significant work - we would hate for your time to be wasted. 29 | 30 | To send us a pull request, please: 31 | 32 | 1. Fork the repository. 33 | 2. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change. 34 | 3. Ensure local tests pass. 35 | 4. Commit to your fork using clear commit messages. 36 | 5. Send us a pull request, answering any default questions in the pull request interface. 37 | 6. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation. 38 | 39 | GitHub provides additional document on [forking a repository](https://help.github.com/articles/fork-a-repo/) and 40 | [creating a pull request](https://help.github.com/articles/creating-a-pull-request/). 41 | 42 | 43 | ## Finding contributions to work on 44 | Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels (enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any ['help wanted'](https://github.com/aws-samples/amazon-cloudwatch-log-centralizer/labels/help%20wanted) issues is a great place to start. 45 | 46 | 47 | ## Code of Conduct 48 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 49 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 50 | opensource-codeofconduct@amazon.com with any additional questions or comments. 51 | 52 | 53 | ## Security issue notifications 54 | If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github issue. 55 | 56 | 57 | ## Licensing 58 | 59 | See the [LICENSE](https://github.com/aws-samples/amazon-cloudwatch-log-centralizer/blob/master/LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution. 60 | 61 | We may ask you to sign a [Contributor License Agreement (CLA)](http://en.wikipedia.org/wiki/Contributor_License_Agreement) for larger changes. 62 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining a copy of this 4 | software and associated documentation files (the "Software"), to deal in the Software 5 | without restriction, including without limitation the rights to use, copy, modify, 6 | merge, publish, distribute, sublicense, and/or sell copies of the Software, and to 7 | permit persons to whom the Software is furnished to do so. 8 | 9 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, 10 | INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A 11 | PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT 12 | HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION 13 | OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE 14 | SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 15 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ## Amazon CloudWatch Log Centralizer 2 | 3 | Centralized logging infrastructure for multiple AWS accounts using CloudFormation and Python 4 | 5 | ## License Summary 6 | 7 | This sample code is made available under a modified MIT license. See the LICENSE file. 8 | 9 | ## Overview 10 | Please see the AWS Architecture Blog (https://aws.amazon.com/blogs/architecture/) article _Stream Amazon CloudWatch Logs to a Centralized Account for Audit and Analysis_ for instructions and more context. 11 | 12 | While some AWS customers use the built-in ability to push Amazon CloudWatch Logs directly into Amazon Elasticsearch Service for analysis, others would prefer to move all logs into a centralized Amazon Simple Storage Service (Amazon S3) bucket location for access by custom and third-party tools. Setting up this solution assumes some knowledge of CloudFormation, Python3 and the boto3 AWS SDK. 13 | 14 | You will need to have or configure an AWS working account and logging account, an IAM access and secret key for those accounts, and a working environment containing Python and the boto3 SDK. For assistance, see the Getting Started Resource Center at https://aws.amazon.com/getting-started/ and Start Building with SDKs and Tools at https://aws.amazon.com/getting-started/tools-sdks/. 15 | -------------------------------------------------------------------------------- /cfn/AddSubscriptionFilter.yml: -------------------------------------------------------------------------------- 1 | # Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | # SPDX-License-Identifier: MIT-0 3 | 4 | AWSTemplateFormatVersion: 2010-09-09 5 | Description: Set up role and lambda to automatically add a subscription filter to all new log groups that are created within the account. 6 | 7 | Parameters: 8 | InfrastructureS3Bucket: 9 | Description: Name (not ARN) of the S3 Infrastructure bucket containing the lambda code in a zip file (e.g., infrastructure-bucket). 10 | MaxLength: 250 11 | MinLength: 1 12 | Type: String 13 | LambdaFolderAndFile: 14 | Description: S3 folder location and the file name of the lambda code in a zip file (e.g., lambdas/AddSubscriptionFilter.zip). 15 | MaxLength: 250 16 | MinLength: 1 17 | Type: String 18 | 19 | Resources: 20 | # IAM Roles and Policies 21 | AddSubscriptionFilterRole: 22 | Type: AWS::IAM::Role 23 | Properties: 24 | AssumeRolePolicyDocument: 25 | Statement: 26 | Effect: Allow 27 | Principal: 28 | Service: 29 | - lambda.amazonaws.com 30 | Action: 'sts:AssumeRole' 31 | Path: / 32 | ManagedPolicyArns: 33 | - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole 34 | Policies: 35 | - PolicyName: AddSubcriptionPolicy 36 | PolicyDocument: 37 | Statement: 38 | - Effect: Allow 39 | Action: 40 | - logs:* 41 | - ssm:GetParameter 42 | Resource: '*' 43 | - Action: 44 | - s3:GetObject 45 | - s3:GetObjectVersion 46 | - s3:GetBucketLocation 47 | - s3:ListBucket 48 | - s3:PutObject 49 | - s3:PutObjectAcl 50 | - s3:GetBucketVersioning 51 | Effect: Allow 52 | Resource: 53 | - arn:aws:s3::: 54 | - arn:aws:s3:::/* 55 | # Lambda 56 | AddSubscriptionLambda: 57 | Type: AWS::Lambda::Function 58 | Properties: 59 | Handler: lambda.lambda_handler 60 | Runtime: python3.6 61 | Code: 62 | S3Bucket: !Ref InfrastructureS3Bucket 63 | S3Key: !Ref LambdaFolderAndFile 64 | Role: !GetAtt 'AddSubscriptionFilterRole.Arn' 65 | AddSubscriptionLambdaPermission: # Grants the event rule permission to invoke the lambda 66 | Type: AWS::Lambda::Permission 67 | Properties: 68 | Action: lambda:InvokeFunction 69 | FunctionName: !GetAtt 'AddSubscriptionLambda.Arn' 70 | Principal: events.amazonaws.com 71 | SourceArn: !GetAtt 'CreateLogGroupEvent.Arn' 72 | # CloudWatch event rule 73 | CreateLogGroupEvent: 74 | Type: AWS::Events::Rule 75 | Properties: 76 | Description: Event rule for identifying new CloudWatch Logs log groups and calling Lambda to add new subscription filters 77 | EventPattern: '{"source":["aws.logs"],"detail-type":["AWS API Call via CloudTrail"],"detail":{"eventSource":["logs.amazonaws.com"],"eventName":["CreateLogGroup"]}}' 78 | Name: CreateLogGroupEvent 79 | State: ENABLED 80 | Targets: 81 | - 82 | Arn: !GetAtt 'AddSubscriptionLambda.Arn' 83 | Id: add_new_subscription 84 | -------------------------------------------------------------------------------- /cfn/centralLogging.yml: -------------------------------------------------------------------------------- 1 | # Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | # SPDX-License-Identifier: MIT-0 3 | 4 | AWSTemplateFormatVersion: 2010-09-09 5 | Description: Set up roles, logs destination, kinesis and delivery stream for centralizing logs to S3 in centralized logging account 6 | 7 | Parameters: 8 | LoggingS3Bucket: 9 | Description: ARN of the S3 Logging bucket to write to. 10 | MaxLength: 250 11 | MinLength: 1 12 | Type: String 13 | 14 | Resources: 15 | # IAM Roles 16 | CWLtoFirehoseRole: 17 | Type: 'AWS::IAM::Role' 18 | Properties: 19 | AssumeRolePolicyDocument: 20 | Version: 2012-10-17 21 | Statement: 22 | - Sid: '' 23 | Effect: Allow 24 | Principal: 25 | Service: 26 | - logs.eu-north-1.amazonaws.com 27 | - logs.ap-south-1.amazonaws.com 28 | - logs.eu-west-3.amazonaws.com 29 | - logs.eu-west-2.amazonaws.com 30 | - logs.eu-west-1.amazonaws.com 31 | - logs.ap-northeast-3.amazonaws.com 32 | - logs.ap-northeast-2.amazonaws.com 33 | - logs.ap-northeast-1.amazonaws.com 34 | - logs.sa-east-1.amazonaws.com 35 | - logs.ca-central-1.amazonaws.com 36 | - logs.ap-southeast-1.amazonaws.com 37 | - logs.ap-southeast-2.amazonaws.com 38 | - logs.eu-central-1.amazonaws.com 39 | - logs.us-east-1.amazonaws.com 40 | - logs.us-east-2.amazonaws.com 41 | - logs.us-west-1.amazonaws.com 42 | - logs.us-west-2.amazonaws.com 43 | Action: 'sts:AssumeRole' 44 | Path: '/' 45 | CWLtoFirehosePolicy: 46 | Type: 'AWS::IAM::Policy' 47 | Properties: 48 | PolicyName: CWL_to_Kinesis_Policy 49 | PolicyDocument: 50 | Version: 2012-10-17 51 | Statement: 52 | - Effect: Allow 53 | Action: 54 | - 'firehose:PutRecord' 55 | Resource: 56 | - !GetAtt 'FirehoseLoggingDeliveryStream.Arn' 57 | - Effect: Allow 58 | Action: 59 | - 'iam:PassRole' 60 | Resource: 61 | - !GetAtt 'CWLtoFirehoseRole.Arn' 62 | Roles: 63 | - !Ref CWLtoFirehoseRole 64 | FirehoseDeliveryRole: 65 | Type: 'AWS::IAM::Role' 66 | Properties: 67 | AssumeRolePolicyDocument: 68 | Version: 2012-10-17 69 | Statement: 70 | - Sid: '' 71 | Effect: Allow 72 | Principal: 73 | Service: firehose.amazonaws.com 74 | Action: 'sts:AssumeRole' 75 | Condition: 76 | StringEquals: 77 | 'sts:ExternalId': !Ref 'AWS::AccountId' 78 | FirehoseDeliveryPolicy: 79 | Type: 'AWS::IAM::Policy' 80 | Properties: 81 | PolicyName: Firehose_Delivery_Policy 82 | PolicyDocument: 83 | Version: 2012-10-17 84 | Statement: 85 | - Effect: Allow 86 | Action: 87 | - 's3:AbortMultipartUpload' 88 | - 's3:GetBucketLocation' 89 | - 's3:GetObject' 90 | - 's3:ListBucket' 91 | - 's3:ListBucketMultipartUploads' 92 | - 's3:PutObject' 93 | Resource: 94 | - !Ref LoggingS3Bucket 95 | - !Join 96 | - '' 97 | - - !Ref LoggingS3Bucket 98 | - '*' 99 | Roles: 100 | - !Ref FirehoseDeliveryRole 101 | # Log Destination 102 | LogDestination: 103 | Type: AWS::Logs::Destination 104 | DependsOn: 105 | - FirehoseLoggingDeliveryStream 106 | - CWLtoFirehoseRole 107 | - CWLtoFirehosePolicy 108 | Properties: 109 | DestinationName: CentralLogDestination 110 | RoleArn: !GetAtt 'CWLtoFirehoseRole.Arn' 111 | TargetArn: !GetAtt 'FirehoseLoggingDeliveryStream.Arn' 112 | DestinationPolicy: "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":{\"AWS\":[\"\",\"\",\"\",\"\",\"\"]},\"Action\":\"logs:PutSubscriptionFilter\",\"Resource\":\"arn:aws:logs:::destination:CentralLogDestination\"}]}" 113 | # Firehose 114 | FirehoseLoggingDeliveryStream: 115 | Type: 'AWS::KinesisFirehose::DeliveryStream' 116 | DependsOn: 117 | - FirehoseDeliveryRole 118 | - FirehoseDeliveryPolicy 119 | Properties: 120 | DeliveryStreamName: 'Centralized-Logging-Delivery-Stream' 121 | DeliveryStreamType: DirectPut 122 | S3DestinationConfiguration: 123 | BucketARN: !Ref LoggingS3Bucket 124 | BufferingHints: 125 | IntervalInSeconds: '300' 126 | SizeInMBs: '50' 127 | CompressionFormat: UNCOMPRESSED 128 | Prefix: 'CentralizedAccountLogs/' 129 | RoleARN: !GetAtt 'FirehoseDeliveryRole.Arn' 130 | Outputs: 131 | DestinationArnExport: 132 | Description: ARN for the LogDestination 133 | Export: 134 | Name: LogDestinationArn 135 | Value: !GetAtt 'LogDestination.Arn' 136 | -------------------------------------------------------------------------------- /python/CentralLogging.py: -------------------------------------------------------------------------------- 1 | # Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | # SPDX-License-Identifier: MIT-0 3 | 4 | import boto3 5 | 6 | 7 | class CentralLogging: 8 | def __init__(self): 9 | self.log_client = boto3.client('logs') 10 | self.ssm_client = boto3.client('ssm') 11 | print('Central Logger Made') 12 | 13 | def add_subscriptions_to_existing_log_groups(self): 14 | 15 | # Retrieve all of the existing log groups 16 | log_group_response = self.log_client.describe_log_groups() 17 | 18 | # Loop over multiple calls to describe_log_groups() as necessary using the next token 19 | while True: 20 | # If there are log groups, iterate over each one, retrieve its name, and call code to add the subscription to it 21 | if log_group_response: 22 | 23 | for log_group in log_group_response['logGroups']: 24 | log_group_name = log_group['logGroupName'] 25 | self.add_subscription_filter(log_group_name) 26 | 27 | if 'nextToken' in log_group_response: 28 | log_groups_next_token = log_group_response['nextToken'] 29 | 30 | if log_groups_next_token: 31 | log_group_response = self.log_client.describe_log_groups(nextToken=log_groups_next_token) 32 | else: 33 | break 34 | 35 | else: 36 | break 37 | 38 | # Add subscription to centralized logging to the log group with log_group_name 39 | def add_subscription_filter(self, log_group_name): 40 | # Retrieve the destination for the subscription from the Parameter Store 41 | destination_response = self.ssm_client.get_parameter(Name='LogDestination') 42 | 43 | # Error if no destination, otherwise extract destination id from response 44 | if not destination_response: 45 | raise ValueError( 46 | 'Cannot locate central logging destination, put_subscription_filter failed') 47 | else: 48 | destination = destination_response['Parameter']['Value'] 49 | 50 | # Error to try to add subscription if one already exists, so delete any existing subscription from this log group 51 | self.delete_existing_subscription_filter(log_group_name) 52 | 53 | # Put the new subscription with the destination onto the log group 54 | self.log_client.put_subscription_filter( 55 | logGroupName=log_group_name, 56 | filterName='Destination', 57 | filterPattern='', 58 | destinationArn=destination 59 | ) 60 | 61 | # Delete an existing subscription from the log group 62 | def delete_existing_subscription_filter(self, log_group_name): 63 | # Retrieve any existing subscription filters (only can be one) 64 | subscription_filters = self.log_client.describe_subscription_filters( 65 | logGroupName=log_group_name) 66 | 67 | # Iterate over results if there are any (again, should not be multiple, but to follow the convention of the SDK) 68 | for subscription_filter in subscription_filters['subscriptionFilters']: 69 | # Retrieve the subscription filter name to use in the call to delete 70 | filter_name = subscription_filter['filterName'] 71 | 72 | # Delete any subscriptions that are found on the log group 73 | self.log_client.delete_subscription_filter( 74 | logGroupName=log_group_name, 75 | filterName=filter_name 76 | ) 77 | -------------------------------------------------------------------------------- /python/init_account_central_logging.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | # Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved. 4 | # SPDX-License-Identifier: MIT-0 5 | 6 | from __future__ import print_function, absolute_import 7 | import boto3 8 | from CentralLogging import CentralLogging 9 | 10 | 11 | def add_log(destination_arn): 12 | central_logger = CentralLogging() 13 | # Initializes the account for centralized logging by adding the logging destination to the Parameter Store and then adding a subscription to the destination for each existing log group 14 | 15 | # Create an SSM SDK client to handle Parameter Store actions 16 | ssm_client = boto3.client('ssm') 17 | 18 | # Put the destination (currently hard-coded and needs to be changed if the destination ever changes) into the Parameter Store 19 | destination_response = ssm_client.put_parameter( 20 | Name='LogDestination', 21 | Description='Centralized logging account destination ARN for subscription filters', 22 | Value=destination_arn, 23 | Type='String', 24 | Overwrite=True 25 | ) 26 | 27 | print(destination_response) 28 | # Call the code to add a subscription to all existing log groups 29 | central_logger.add_subscriptions_to_existing_log_groups() 30 | 31 | 32 | if __name__ == "__main__": 33 | import argparse 34 | 35 | parser = argparse.ArgumentParser() 36 | parser.add_argument("-d", "--destination", help="logs:destination ARN to send logs to") 37 | 38 | args = parser.parse_args() 39 | if args.destination: 40 | add_log(args.destination) 41 | else: 42 | parser.print_help() 43 | raise ValueError('Must provide LogDestination ARN. See help') 44 | -------------------------------------------------------------------------------- /python/lambda.py: -------------------------------------------------------------------------------- 1 | # Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | # SPDX-License-Identifier: MIT-0 3 | 4 | import boto3 5 | from CentralLogging import CentralLogging 6 | 7 | # Lambda handler triggered by a CloudWatch Event whenever a new log group is created. Calls the core code to add new subscription to the newly created log group 8 | 9 | 10 | def lambda_handler(event, context): 11 | central_logger = CentralLogging() 12 | print(event) 13 | print("In add_new_subscription Lambda - requestParameters: ") 14 | 15 | # Retrieves the request parameters from the event that was called to create the log group 16 | request_parameters = event['detail']['requestParameters'] 17 | print(request_parameters) 18 | 19 | # Extract the log group name from the request parameters to create the log group 20 | if request_parameters: 21 | print(" Inspecting request parameters for log_group_name:") 22 | log_group_name = request_parameters['logGroupName'] 23 | print(log_group_name) 24 | 25 | # Call code to add the subscription to the log group 26 | central_logger.add_subscription_filter(log_group_name) 27 | --------------------------------------------------------------------------------