├── diagram.png ├── README.md ├── firehose.yml ├── every-account.yml └── logs-account.yml /diagram.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aidansteele/centralized-logs/HEAD/diagram.png -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Centralized AWS CloudWatch Logs aggregation 2 | 3 | This repo provides a code-less solution to aggregating AWS CloudWatch log 4 | subscriptions across multiple AWS accounts and regions. It uses Kinesis Firehose 5 | (for aggregation and forwarding), EventBridge (for notifications about new log 6 | group creation) and Step Functions (for assuming roles cross-account and calling 7 | the AWS SDK). 8 | 9 | ![architecture diagram](/diagram.png) 10 | 11 | First, you can create a Kinesis Firehose delivery stream (an example is 12 | available in [`firehose.yml`](/firehose.yml)) or a Kinesis data stream - this 13 | will be the destination that all your logs are forwarded to. You only need to 14 | deploy one of these, there doesn't have to be one in each region. 15 | 16 | Next, you deploy resources into your logs collection AWS account. This is the 17 | [`logs-account.yml`](/logs-account.yml) template. You should deploy a stack from 18 | this template into every AWS region that you want to collect logs from. It has 19 | a few parameters: `OrganizationId` is the string you get by running `aws organizations describe-organization --query 'Organization.Id'` 20 | and `FirehoseArn` should be the ARN of the Firehose delivery stream deployed 21 | earlier. 22 | 23 | Finally, you deploy the [`every-account.yml`](/every-account.yml) template into 24 | all the AWS accounts and regions that you want to collect logs from. This stack 25 | has a parameter `CentralBusAccountId`, which is the account ID of the logs 26 | collection account hosting the stacks deployed in the previous step. 27 | -------------------------------------------------------------------------------- /firehose.yml: -------------------------------------------------------------------------------- 1 | Parameters: 2 | EndpointUrl: 3 | Type: String 4 | NoEcho: true 5 | APIKey: 6 | Type: String 7 | NoEcho: true 8 | 9 | Resources: 10 | Bucket: 11 | DeletionPolicy: Retain 12 | Type: AWS::S3::Bucket 13 | Properties: 14 | VersioningConfiguration: 15 | Status: Enabled 16 | 17 | Firehose: 18 | Type: AWS::KinesisFirehose::DeliveryStream 19 | Properties: 20 | DeliveryStreamName: !Ref AWS::StackName 21 | DeliveryStreamType: DirectPut 22 | HttpEndpointDestinationConfiguration: 23 | RoleARN: !GetAtt FirehoseRole.Arn 24 | BufferingHints: 25 | IntervalInSeconds: 60 26 | SizeInMBs: 4 27 | EndpointConfiguration: 28 | Name: logs-saas 29 | Url: !Ref EndpointUrl 30 | AccessKey: !Ref APIKey 31 | RequestConfiguration: 32 | ContentEncoding: GZIP 33 | RetryOptions: 34 | DurationInSeconds: 60 35 | S3BackupMode: AllData 36 | S3Configuration: 37 | BucketARN: !Sub arn:aws:s3:::${Bucket} 38 | RoleARN: !GetAtt FirehoseRole.Arn 39 | Prefix: logs/!{timestamp:yyyy/MM/dd}/ 40 | ErrorOutputPrefix: errors/!{firehose:error-output-type}/!{timestamp:yyyy/MM/dd}/ 41 | BufferingHints: 42 | IntervalInSeconds: 60 43 | SizeInMBs: 128 44 | 45 | FirehoseRole: 46 | Type: AWS::IAM::Role 47 | Properties: 48 | AssumeRolePolicyDocument: 49 | Version: "2012-10-17" 50 | Statement: 51 | - Effect: Allow 52 | Principal: 53 | Service: firehose.amazonaws.com 54 | Action: sts:AssumeRole 55 | Condition: 56 | StringEquals: 57 | sts:ExternalId: !Ref AWS::AccountId 58 | Policies: 59 | - PolicyName: Firehose 60 | PolicyDocument: 61 | Version: "2012-10-17" 62 | Statement: 63 | - Effect: Allow 64 | Action: 65 | - s3:AbortMultipartUpload 66 | - s3:GetBucketLocation 67 | - s3:GetObject 68 | - s3:ListBucket 69 | - s3:ListBucketMultipartUploads 70 | - s3:PutObject 71 | Resource: 72 | - !Sub arn:aws:s3:::${Bucket} 73 | - !Sub arn:aws:s3:::${Bucket}/* 74 | 75 | Outputs: 76 | FirehoseArn: 77 | Value: !GetAtt Firehose.Arn 78 | -------------------------------------------------------------------------------- /every-account.yml: -------------------------------------------------------------------------------- 1 | Parameters: 2 | CentralBusAccountId: 3 | Type: String 4 | SubscriberRoleNamePrefix: 5 | Type: String 6 | Default: CentralizedLogsSubscriber 7 | LogPusherRoleNamePrefix: 8 | Type: String 9 | Default: CentralizedLogsPusher 10 | 11 | Resources: 12 | NewLogGroupEventRule: 13 | Type: AWS::Events::Rule 14 | Properties: 15 | EventPattern: 16 | source: [aws.logs] 17 | detail-type: [AWS API Call via CloudTrail] 18 | detail: 19 | eventName: [CreateLogGroup] 20 | Targets: 21 | - Id: central-bus 22 | Arn: !Sub arn:aws:events:${AWS::Region}:${CentralBusAccountId}:event-bus/CentralizedLogs 23 | RoleArn: !GetAtt NewLogGroupEventRole.Arn 24 | 25 | NewLogGroupEventRole: 26 | Type: AWS::IAM::Role 27 | Properties: 28 | AssumeRolePolicyDocument: 29 | Version: "2012-10-17" 30 | Statement: 31 | - Effect: Allow 32 | Action: sts:AssumeRole 33 | Principal: 34 | Service: events.amazonaws.com 35 | Policies: 36 | - PolicyName: PutEventsOnCentralBus 37 | PolicyDocument: 38 | Version: "2012-10-17" 39 | Statement: 40 | - Effect: Allow 41 | Action: events:PutEvents 42 | Resource: !Sub arn:aws:events:${AWS::Region}:${CentralBusAccountId}:event-bus/CentralizedLogs 43 | 44 | LogPusherRole: 45 | Type: AWS::IAM::Role 46 | Properties: 47 | RoleName: !Sub ${LogPusherRoleNamePrefix}-${AWS::Region} 48 | AssumeRolePolicyDocument: 49 | Version: "2012-10-17" 50 | Statement: 51 | - Effect: Allow 52 | Action: sts:AssumeRole 53 | Principal: 54 | Service: logs.amazonaws.com 55 | Policies: 56 | - PolicyName: PutLogEvents 57 | PolicyDocument: 58 | Version: "2012-10-17" 59 | Statement: 60 | - Effect: Allow 61 | Action: logs:PutLogEvents 62 | Resource: "*" 63 | 64 | SubscriberRole: 65 | Type: AWS::IAM::Role 66 | Properties: 67 | RoleName: !Sub ${SubscriberRoleNamePrefix}-${AWS::Region} 68 | AssumeRolePolicyDocument: 69 | Version: "2012-10-17" 70 | Statement: 71 | - Effect: Allow 72 | Action: sts:AssumeRole 73 | Principal: 74 | AWS: !Ref CentralBusAccountId 75 | Condition: 76 | ArnLike: 77 | sts:ExternalId: !Sub arn:aws:states:*:${CentralBusAccountId}:stateMachine:* 78 | Policies: 79 | - PolicyName: Subscriber 80 | PolicyDocument: 81 | Version: "2012-10-17" 82 | Statement: 83 | - Effect: Allow 84 | Action: 85 | - logs:PutSubscriptionFilter 86 | - logs:DeleteSubscriptionFilter 87 | - logs:DescribeSubscriptionFilters 88 | - logs:DescribeLogGroups 89 | Resource: "*" 90 | - Effect: Allow 91 | Action: iam:PassRole 92 | Resource: !GetAtt LogPusherRole.Arn 93 | -------------------------------------------------------------------------------- /logs-account.yml: -------------------------------------------------------------------------------- 1 | Transform: 2 | - AWS::LanguageExtensions 3 | - AWS::Serverless-2016-10-31 4 | 5 | Parameters: 6 | OrganizationId: 7 | Type: String 8 | FirehoseArn: 9 | Type: String 10 | SubscriberRoleNamePrefix: 11 | Type: String 12 | Default: CentralizedLogsSubscriber 13 | LogPusherRoleNamePrefix: 14 | Type: String 15 | Default: CentralizedLogsPusher 16 | 17 | Resources: 18 | Bus: 19 | Type: AWS::Events::EventBus 20 | Properties: 21 | Name: CentralizedLogs 22 | 23 | BusPolicy: 24 | Type: AWS::Events::EventBusPolicy 25 | Properties: 26 | EventBusName: !Ref Bus 27 | StatementId: AllowOrg 28 | Statement: 29 | Effect: Allow 30 | Principal: "*" 31 | Action: events:PutEvents 32 | Resource: !GetAtt Bus.Arn 33 | Condition: 34 | StringEquals: 35 | aws:PrincipalOrgID: !Ref OrganizationId 36 | 37 | DestinationRole: 38 | Type: AWS::IAM::Role 39 | Properties: 40 | AssumeRolePolicyDocument: 41 | Version: "2012-10-17" 42 | Statement: 43 | - Effect: Allow 44 | Principal: 45 | Service: logs.amazonaws.com 46 | Action: sts:AssumeRole 47 | Policies: 48 | - PolicyName: Firehose 49 | PolicyDocument: 50 | Version: "2012-10-17" 51 | Statement: 52 | - Effect: Allow 53 | Action: firehose:PutRecord* 54 | Resource: !Ref FirehoseArn 55 | 56 | Destination: 57 | Type: AWS::Logs::Destination 58 | Properties: 59 | DestinationName: CentralizedLogs # if you change this name, you also need to change the ARN ~10 lines down 60 | RoleArn: !GetAtt DestinationRole.Arn 61 | TargetArn: !Ref FirehoseArn 62 | DestinationPolicy: 63 | Fn::ToJsonString: 64 | Version: "2012-10-17" 65 | Statement: 66 | - Sid: AllowOrg 67 | Effect: Allow 68 | Principal: "*" 69 | Action: logs:PutSubscriptionFilter 70 | Resource: !Sub arn:aws:logs:${AWS::Region}:${AWS::AccountId}:destination:CentralizedLogs 71 | Condition: 72 | StringEquals: 73 | aws:PrincipalOrgID: !Ref OrganizationId 74 | 75 | Subscriber: 76 | Type: AWS::Serverless::StateMachine 77 | Properties: 78 | Type: STANDARD 79 | Policies: 80 | - Statement: 81 | - Effect: Allow 82 | Action: sts:AssumeRole 83 | Resource: !Sub arn:aws:iam::*:role/${SubscriberRoleNamePrefix}-${AWS::Region} 84 | Events: 85 | NewLogGroup: 86 | Type: EventBridgeRule 87 | Properties: 88 | EventBusName: !Ref Bus 89 | Pattern: 90 | source: [aws.logs] 91 | detail-type: [AWS API Call via CloudTrail] 92 | detail: 93 | eventName: [CreateLogGroup] 94 | Definition: 95 | StartAt: Create ARNs 96 | States: 97 | Create ARNs: 98 | Type: Pass 99 | Next: Create subscription 100 | Parameters: 101 | Subscriber.$: !Sub States.Format('arn:aws:iam::{}:role/${SubscriberRoleNamePrefix}-${AWS::Region}', $.account) 102 | Pusher.$: !Sub States.Format('arn:aws:iam::{}:role/${LogPusherRoleNamePrefix}-${AWS::Region}', $.account) 103 | ResultPath: $.RoleArns 104 | Create subscription: 105 | Type: Task 106 | End: true 107 | Resource: arn:aws:states:::aws-sdk:cloudwatchlogs:putSubscriptionFilter 108 | Parameters: 109 | DestinationArn: !GetAtt Destination.Arn 110 | FilterName: CentralizedLogs 111 | FilterPattern: '' # blank string matches all logs 112 | LogGroupName.$: $.detail.requestParameters.logGroupName 113 | RoleArn.$: $.RoleArns.Pusher 114 | Credentials: 115 | RoleArn.$: $.RoleArns.Subscriber 116 | 117 | Outputs: 118 | BusArn: 119 | Value: !GetAtt Bus.Arn 120 | Subscriber: 121 | Value: !Ref Subscriber 122 | --------------------------------------------------------------------------------