├── LICENSE ├── README.md ├── cloudformation ├── elastic-search-single-server.json └── lambda-traildash2.json ├── dashboard └── kibana-dashboard-export.json ├── documentation ├── image1.png ├── image2.png ├── image3.png ├── image4.png ├── image5.png ├── image6.png ├── image7.png ├── image8.png └── image9.png └── lambda-code ├── .DS_Store ├── index.js └── index.js.zip /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2017 David 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # TrailDash2 2 | 3 | A Serverless Dashboard system to display and filter AWS CloudTrail Logs 4 | ![Dashboard](/documentation/image1.png) 5 | 6 | ## Table of Contents 7 | 8 | * [What is TrailDash2?](#what-is-traildash2?) 9 | * [Installation and Usage instructions](#installation-and-usage-instructions) 10 | * [Part 1: Deploying](#part-1:-Deploying) 11 | * [Part 2: Setting up the dashboard](#part-2:-setting-up-the-dashboard) 12 | * [Troubleshooting and Known problems](#troubleshooting-and-known-problems) 13 | 14 | ## What is TrailDash2? 15 | 16 | TrailDash2 comes from a tool called TrailDash developed by the team from AppliedTrust https://github.com/AppliedTrust/traildash 17 | 18 | Unfortunately they have EOL their tool and creating your own dashboard from scratch isn't that straight forward. 19 | 20 | Enter TrailDash2, much like the original, however now completely serverless. Traildash2 uses AWS Lambda to capture the AWS CloudTrail logs and send them into AWS ElasticSearch 5. 21 | 22 | The Dashboard shown is using the built in the Kibana that comes with the AWS ElasticSearch service. The Dashboard configuration can be imported from this Repo directly into your AWS ElasticSearch to obtain the same dashboard. 23 | 24 | The repo includes CloudFormation Templates that can be applied to your AWS account that will perform the deployment of AWS ElasticSearch and AWS Lambda functions. 25 | 26 | 27 | 28 | 29 | 30 | ## Installation and Usage instructions 31 | For those familiar with AWS. CloudFormation is in **/cloudformation/**, Lambda code is in **/lambda-code/** and you'll need to setup an event trigger on your CloudTrail bucket to call the Lambda function. Then visit the ElasticSearch Kibana interface and import the **/dashboard/kibana-dashboard-export.json** file which imports the dashboard and all the required searches/virtualizations 32 | 33 | For everyone else see below! :) 34 | 35 | 36 | ### Part 1: Deploying 37 | 1. Git clone the repo or download the whole thing from the release page 38 | 39 | 2. Upload **/lambda-code/index.js.zip** file to an S3 bucket 40 | 41 | 3. Deploy the CloudFormation template to create your ElasticSearch server: **cloudformation/elastic-search-single-server.json**. Most of the default configuration should be fine. Set the AllowedAccessIPs to an IP address that you would like to be able to access Kibana from. An example of what the configuration will look like in cloudformation is shown below: 42 | ![ESDeployment](/documentation/image2.png) 43 | 44 | 4. Wait for ElasticSearch stack to create. Once the stack has been create, copy down the **CNAME ES endpoint** URL provided in the stack output 45 | ![ESStackOutput](/documentation/image3.png) 46 | Â 47 | 5. Deploy the CloudFormation template to create your Lambda Function: **cloudformation/lambda-traildash2.json**. In the lambda template you will be required to fill in some parameters 48 | * Enter in the elasticserach address that was provided from the previous step. 49 | 50 | * Enter in the name of the S3 bucket that you uploaded the lambda index.js.zip file into. 51 | 52 | * enter in the keypath to the index.js.zip file. If you upload the file into the root of the bucket that will be **index.js.zip**. If it was a subfolder like code then it will be **code/index.js.zip** 53 | 54 | * Enter in the S3 bucket name where your CloudTrail logs are being saved to 55 | 56 | ![CFLambdaDeployment](/documentation/image4.png) 57 | 58 | 6. Once the stack has successfully deployed the last remaining step is to attach events from the S3 cloudtrail bucket to the Lambda function. 59 | * Browse to your bucket and select **Properties** and **events** 60 | ![S3config](/documentation/image5.png) 61 | 62 | * Create a new event filter on **put** operations and select your lambda function from the drop down! 63 | ![S3event](/documentation/image6.png) 64 | 65 | 7. Now cloudtrail logs written to S3 events should be starting to be processed by the traildash2 lambda function and will start pushing those into elasticsearch. You can check by looking at the ElasticSearch index's. You should see an index titled logstash-YYYY-MM-DD 66 | 67 | ### Part 2: Setting up the dashboard 68 | 1. Now you can visit your Kibana URL, which you will be able to find in the elastic search console. *Remember you will only be able to access Kibana from an IP in the range that you specified on deployment of the ES service* 69 | 70 | 2. On load Kibana will ask you to tell it about your index and data. Configure it as shown below: 71 | ![kibanaIndex](/documentation/image7.png) 72 | 73 | 3. While still in management, go to Saved Objects. Click import and select the **/dashboard/kibana-dashboard-export.json** file. 74 | ![kibanaImport](/documentation/image8.png) 75 | 76 | 4. If all goes successfully you should see the following saved objects post the import. You can now go and view your dashboard by going to **Dashboard** selecting **open** and selecting **Main-Dashboard** 77 | ![kibanaPostImport](/documentation/image9.png) 78 | 79 | 80 | ## Troubleshooting and Known problems 81 | * Kibana shows error bar **Courier Fetch: x of n shards failed**: This is normally due to having selected the incorrect Time-field name in Part 2 of the setup. To fix, go into the AWS elasticsearch console. *Delete* the **.kibana** index. Reload Kibana in your browser and set it up again. 82 | 83 | * No data in elasticsearch: Check that the lambda function is being exceuted. If it is being executed check the cloudwatch logs for the result of the call to elasticsearch. If the lambda function is not being called, check that your event trigger in S3 is setup correctly and that the bucket is in the same region as your lambda function 84 | 85 | --- 86 | -------------------------------------------------------------------------------- /cloudformation/elastic-search-single-server.json: -------------------------------------------------------------------------------- 1 | { 2 | 3 | "Description": "ElasticSearch Template For TrailDash2", 4 | 5 | "Parameters": { 6 | 7 | "AllowedAccessIPs": { 8 | "Description": "CSV list of IPs in CIDR format for access into ES.", 9 | "Type": "CommaDelimitedList", 10 | "ConstraintDescription": "IPs need to be in CIDR format" 11 | }, 12 | "InstanceCount": { 13 | "Description": "The number instances that actually run the DB with data. 2 or more for HA deployments", 14 | "Type": "String", 15 | "Default": "1" 16 | }, 17 | "InstanceSize": { 18 | "Description": "The size of the instances to use for the ES domain", 19 | "Default": "t2.small.elasticsearch", 20 | "Type": "String", 21 | "AllowedValues": ["t2.micro.elasticsearch", "t2.small.elasticsearch", "t2.medium.elasticsearch", "m3.medium.elasticsearch", "m3.large.elasticsearch", "m3.xlarge.elasticsearch", "r3.large.elasticsearch", "r3.xlarge.elasticsearch"] 22 | }, 23 | "DiskSize": { 24 | "Description": "The amount of EBS to assign to each cluster node", 25 | "Default": 10, 26 | "Type": "Number" 27 | } 28 | 29 | }, 30 | 31 | "Resources": { 32 | "ElasticsearchDomain": { 33 | "Type": "AWS::Elasticsearch::Domain", 34 | "Properties": { 35 | "ElasticsearchClusterConfig": { 36 | "InstanceType": { 37 | "Ref": "InstanceSize" 38 | }, 39 | "InstanceCount": { 40 | "Ref": "InstanceCount" 41 | }, 42 | }, 43 | "EBSOptions": { 44 | "EBSEnabled": true, 45 | "VolumeSize": { 46 | "Ref": "DiskSize" 47 | }, 48 | "VolumeType": "gp2" 49 | }, 50 | "SnapshotOptions": { 51 | "AutomatedSnapshotStartHour": "6" 52 | }, 53 | "AccessPolicies": { 54 | "Version": "2012-10-17", 55 | "Statement": [{ 56 | "Effect": "Allow", 57 | "Principal": { 58 | "AWS": "*" 59 | }, 60 | "Action": "es:*", 61 | "Resource": "*", 62 | "Condition": { 63 | "IpAddress": { 64 | "aws:SourceIp": { 65 | "Ref": "AllowedAccessIPs" 66 | } 67 | } 68 | } 69 | }, 70 | { 71 | "Effect": "Allow", 72 | "Principal": { 73 | "AWS": { "Fn::Join": ["",["arn:aws:iam::",{"Ref":"AWS::AccountId"}, ":root" ]] } 74 | }, 75 | "Action": "es:*", 76 | "Resource": "*" 77 | }] 78 | }, 79 | "AdvancedOptions": { 80 | "rest.action.multi.allow_explicit_index": "true" 81 | }, 82 | "ElasticsearchVersion": "5.1", 83 | "Tags": [{ 84 | "Key": "Name", 85 | "Value": { 86 | "Ref": "AWS::StackName" 87 | } 88 | } 89 | ] 90 | } 91 | } 92 | }, 93 | 94 | "Outputs": { 95 | "ESEndpoint": { 96 | "Description": "CNAME ES endpoint", 97 | "Value": { 98 | "Fn::Join": ["", [{ 99 | "Fn::GetAtt": ["ElasticsearchDomain", "DomainEndpoint"] 100 | }]] 101 | } 102 | } 103 | } 104 | 105 | } 106 | -------------------------------------------------------------------------------- /cloudformation/lambda-traildash2.json: -------------------------------------------------------------------------------- 1 | { 2 | 3 | "Description": "Lambda deployment of TrailDash2", 4 | "AWSTemplateFormatVersion": "2010-09-09", 5 | "Parameters": { 6 | "1elasticsearchClusterAddress": { 7 | "Type": "String", 8 | "Description": "The full https url to your elastic search instance/cluster" 9 | }, 10 | "2S3Bucket": { 11 | "Description": "Name of the bucket where the lambda code will be", 12 | "Type": "String" 13 | }, 14 | 15 | "3S3ItemKey": { 16 | "Description": "path inside the bucket to the code file. 'traildash2/index.zip' for example", 17 | "Type": "String" 18 | }, 19 | "4CloudTrailS3Bucket":{ 20 | "Description":"Name of the bucket where your cloudtrail logs are stored", 21 | "Type": "String" 22 | }, 23 | 24 | "LambdaMemorySize": { 25 | "Type": "Number", 26 | "Description": "Amount of memory (must be a multiple of 64)", 27 | "Default": 128 28 | }, 29 | 30 | "LambdaFuntionTimeOut": { 31 | "Type": "Number", 32 | "Description": "Set timeout for lambda function (seconds)", 33 | "Default": 20 34 | }, 35 | 36 | "LambdaRunType": { 37 | "Type": "String", 38 | "Description": "The lambda runtime to use", 39 | "Default": "nodejs4.3" 40 | }, 41 | 42 | "LambdaHandler": { 43 | "Type": "String", 44 | "Description": "The default lambda Handler to use", 45 | "Default": "index.handler" 46 | }, 47 | 48 | 49 | 50 | }, 51 | 52 | 53 | "Resources": { 54 | 55 | "AWSIAM": { 56 | "Type": "AWS::IAM::Role", 57 | "Properties": { 58 | "AssumeRolePolicyDocument": { 59 | "Version": "2012-10-17", 60 | "Statement": [{ 61 | "Effect": "Allow", 62 | "Principal": { 63 | "Service": ["lambda.amazonaws.com"] 64 | }, 65 | "Action": ["sts:AssumeRole"] 66 | }] 67 | }, 68 | "Path": "/", 69 | "Policies": [{ 70 | "PolicyName": "Default-Lambda-Access", 71 | "PolicyDocument": { 72 | "Statement": [{ 73 | "Action": [ 74 | "logs:CreateLogGroup", 75 | "logs:CreateLogStream", 76 | "logs:PutLogEvents" 77 | ], 78 | "Effect": "Allow", 79 | "Resource": [ 80 | "arn:aws:logs:*:*:*" 81 | ] 82 | }] 83 | } 84 | }, { 85 | "PolicyName": "Elastic-Search-Access", 86 | "PolicyDocument": { 87 | "Statement": [{ 88 | "Action": ["es:*"], 89 | "Effect": "Allow", 90 | "Resource": ["*"] 91 | }] 92 | } 93 | }, { 94 | "PolicyName": "S3-access", 95 | "PolicyDocument": { 96 | "Statement": [{ 97 | "Action": ["s3:GetObject", "s3:ListBucket"], 98 | "Effect": "Allow", 99 | "Resource": [{ 100 | "Fn::Join": ["", ["arn:aws:s3:::", { 101 | "Ref": "2S3Bucket" 102 | }, { 103 | "Ref": "3S3ItemKey" 104 | }]] 105 | }] 106 | }] 107 | } 108 | }, 109 | { 110 | "PolicyName": "cloudtrail-s3-access", 111 | "PolicyDocument": { 112 | "Statement": [{ 113 | "Action": ["s3:GetObject", "s3:ListBucket"], 114 | "Effect": "Allow", 115 | "Resource": [{ 116 | "Fn::Join": ["", ["arn:aws:s3:::", {"Ref": "4CloudTrailS3Bucket"}, "/*"]] 117 | }] 118 | }] 119 | } 120 | }] 121 | } 122 | }, 123 | 124 | "RootInstanceProfile": { 125 | "Type": "AWS::IAM::InstanceProfile", 126 | "Properties": { 127 | "Path": "/", 128 | "Roles": [{ 129 | "Ref": "AWSIAM" 130 | }] 131 | } 132 | }, 133 | 134 | "LambdaFunction": { 135 | "Type": "AWS::Lambda::Function", 136 | "Properties": { 137 | "Handler": { 138 | "Ref": "LambdaHandler" 139 | }, 140 | "Role": { 141 | "Fn::Join": ["", ["arn:aws:iam::", { 142 | "Ref": "AWS::AccountId" 143 | }, ":role/", { 144 | "Ref": "AWSIAM" 145 | }]] 146 | }, 147 | "Code": { 148 | "S3Bucket": { 149 | "Ref": "2S3Bucket" 150 | }, 151 | "S3Key": { 152 | "Ref": "3S3ItemKey" 153 | } 154 | }, 155 | "Runtime": { 156 | "Ref": "LambdaRunType" 157 | }, 158 | "Environment": { 159 | "Variables": { 160 | "elasticsearchurl": { 161 | "Ref": "1elasticsearchClusterAddress" 162 | } 163 | } 164 | }, 165 | "MemorySize": { 166 | "Ref": "LambdaMemorySize" 167 | }, 168 | "Timeout": { 169 | "Ref": "LambdaFuntionTimeOut" 170 | } 171 | } 172 | }, 173 | "LambdaInvokePermission": { 174 | "Type": "AWS::Lambda::Permission", 175 | "Properties": { 176 | "FunctionName": { 177 | "Fn::GetAtt": ["LambdaFunction", "Arn"] 178 | }, 179 | "Action": "lambda:InvokeFunction", 180 | "Principal": "s3.amazonaws.com", 181 | "SourceAccount": { 182 | "Ref": "AWS::AccountId" 183 | } 184 | } 185 | 186 | } 187 | }, 188 | "Outputs": { 189 | "LambdaFunctionOutput": { 190 | "Description": "Lambda ARN", 191 | "Value": { 192 | "Fn::GetAtt": ["LambdaFunction", "Arn"] 193 | } 194 | } 195 | } 196 | } 197 | -------------------------------------------------------------------------------- /dashboard/kibana-dashboard-export.json: -------------------------------------------------------------------------------- 1 | [ 2 | { 3 | "_id": "Main-Dashboard", 4 | "_type": "dashboard", 5 | "_source": { 6 | "title": "Main-Dashboard", 7 | "hits": 0, 8 | "description": "", 9 | "panelsJSON": "[{\"col\":1,\"id\":\"CloudTrail-Events\",\"panelIndex\":1,\"row\":1,\"size_x\":6,\"size_y\":2,\"type\":\"visualization\"},{\"col\":7,\"id\":\"Active-Users\",\"panelIndex\":2,\"row\":1,\"size_x\":6,\"size_y\":2,\"type\":\"visualization\"},{\"col\":1,\"id\":\"Top-10-Active-AWS-Services\",\"panelIndex\":4,\"row\":3,\"size_x\":4,\"size_y\":4,\"type\":\"visualization\"},{\"col\":5,\"id\":\"Regions-In-Use\",\"panelIndex\":5,\"row\":5,\"size_x\":4,\"size_y\":2,\"type\":\"visualization\"},{\"col\":1,\"columns\":[\"eventSource\",\"eventName\",\"userIdentity.userName\",\"userIdentity.principalId\"],\"id\":\"Events-List\",\"panelIndex\":6,\"row\":7,\"size_x\":12,\"size_y\":36,\"sort\":[\"eventTime\",\"desc\"],\"type\":\"search\"},{\"col\":9,\"id\":\"Event-Names\",\"panelIndex\":7,\"row\":3,\"size_x\":4,\"size_y\":4,\"type\":\"visualization\"},{\"col\":5,\"id\":\"Console-Login-Number\",\"panelIndex\":8,\"row\":3,\"size_x\":2,\"size_y\":2,\"type\":\"visualization\"},{\"id\":\"Account-Number\",\"type\":\"visualization\",\"panelIndex\":12,\"size_x\":2,\"size_y\":2,\"col\":7,\"row\":3}]", 10 | "optionsJSON": "{\"darkTheme\":true}", 11 | "uiStateJSON": "{\"P-1\":{\"vis\":{\"legendOpen\":false}},\"P-2\":{\"vis\":{\"legendOpen\":true}},\"P-7\":{\"spy\":{\"mode\":{\"fill\":false,\"name\":null}}}}", 12 | "version": 1, 13 | "timeRestore": true, 14 | "timeTo": "now/d", 15 | "timeFrom": "now/d", 16 | "refreshInterval": { 17 | "display": "Off", 18 | "pause": false, 19 | "section": 0, 20 | "value": 0 21 | }, 22 | "kibanaSavedObjectMeta": { 23 | "searchSourceJSON": "{\"filter\":[{\"$state\":{\"store\":\"appState\"},\"meta\":{\"alias\":null,\"disabled\":false,\"index\":\"logstash-*\",\"key\":\"query\",\"negate\":true,\"value\":\"Describe*\"},\"query\":{\"query_string\":{\"analyze_wildcard\":true,\"query\":\"Describe*\"}}},{\"$state\":{\"store\":\"appState\"},\"meta\":{\"alias\":null,\"disabled\":false,\"index\":\"logstash-*\",\"key\":\"query\",\"negate\":true,\"value\":\"List*\"},\"query\":{\"query_string\":{\"analyze_wildcard\":true,\"query\":\"List*\"}}},{\"$state\":{\"store\":\"appState\"},\"meta\":{\"alias\":null,\"disabled\":false,\"index\":\"logstash-*\",\"key\":\"query\",\"negate\":true,\"value\":\"Get*\"},\"query\":{\"query_string\":{\"analyze_wildcard\":true,\"query\":\"Get*\"}}},{\"$state\":{\"store\":\"appState\"},\"meta\":{\"alias\":null,\"disabled\":false,\"index\":\"logstash-*\",\"key\":\"query\",\"negate\":true,\"value\":\"Assume*\"},\"query\":{\"query_string\":{\"analyze_wildcard\":true,\"query\":\"Assume*\"}}},{\"query\":{\"query_string\":{\"analyze_wildcard\":true,\"query\":\"*\"}}}]}" 24 | } 25 | } 26 | }, 27 | { 28 | "_id": "Events-List", 29 | "_type": "search", 30 | "_source": { 31 | "title": "Events-List", 32 | "description": "", 33 | "hits": 0, 34 | "columns": [ 35 | "eventSource", 36 | "eventName", 37 | "userIdentity.userName", 38 | "userIdentity.principalId" 39 | ], 40 | "sort": [ 41 | "eventTime", 42 | "desc" 43 | ], 44 | "version": 1, 45 | "kibanaSavedObjectMeta": { 46 | "searchSourceJSON": "{\"index\":\"logstash-*\",\"query\":{\"query_string\":{\"query\":\"*\",\"analyze_wildcard\":true}},\"filter\":[],\"highlight\":{\"pre_tags\":[\"@kibana-highlighted-field@\"],\"post_tags\":[\"@/kibana-highlighted-field@\"],\"fields\":{\"*\":{}},\"require_field_match\":false,\"fragment_size\":2147483647}}" 47 | } 48 | } 49 | }, 50 | { 51 | "_id": "ConsoleLogins", 52 | "_type": "search", 53 | "_source": { 54 | "title": "ConsoleLogins", 55 | "description": "", 56 | "hits": 0, 57 | "columns": [ 58 | "eventSource", 59 | "eventName", 60 | "userIdentity.userName", 61 | "userIdentity.principalId" 62 | ], 63 | "sort": [ 64 | "eventTime", 65 | "desc" 66 | ], 67 | "version": 1, 68 | "kibanaSavedObjectMeta": { 69 | "searchSourceJSON": "{\"index\":\"logstash-*\",\"query\":{\"query_string\":{\"query\":\"ConsoleLogin\",\"analyze_wildcard\":true}},\"filter\":[{\"meta\":{\"disabled\":false,\"index\":\"logstash-*\",\"key\":\"query\",\"negate\":true,\"value\":\"Describe*\",\"alias\":null},\"query\":{\"query_string\":{\"query\":\"Describe*\",\"analyze_wildcard\":true}},\"$state\":{\"store\":\"globalState\"}},{\"meta\":{\"disabled\":false,\"index\":\"logstash-*\",\"key\":\"query\",\"negate\":true,\"value\":\"List*\",\"alias\":null},\"query\":{\"query_string\":{\"query\":\"List*\",\"analyze_wildcard\":true}},\"$state\":{\"store\":\"globalState\"}},{\"meta\":{\"disabled\":false,\"index\":\"logstash-*\",\"key\":\"query\",\"negate\":true,\"value\":\"Get*\",\"alias\":null},\"query\":{\"query_string\":{\"query\":\"Get*\",\"analyze_wildcard\":true}},\"$state\":{\"store\":\"globalState\"}},{\"meta\":{\"disabled\":false,\"index\":\"logstash-*\",\"key\":\"query\",\"negate\":true,\"value\":\"Assume*\",\"alias\":null},\"query\":{\"query_string\":{\"query\":\"Assume*\",\"analyze_wildcard\":true}},\"$state\":{\"store\":\"globalState\"}}],\"highlight\":{\"pre_tags\":[\"@kibana-highlighted-field@\"],\"post_tags\":[\"@/kibana-highlighted-field@\"],\"fields\":{\"*\":{}},\"require_field_match\":false,\"fragment_size\":2147483647}}" 70 | } 71 | } 72 | }, 73 | { 74 | "_id": "CloudTrail-Events", 75 | "_type": "visualization", 76 | "_source": { 77 | "title": "CloudTrail Events", 78 | "visState": "{\"title\":\"CloudTrail Events\",\"type\":\"area\",\"params\":{\"shareYAxis\":true,\"addTooltip\":false,\"addLegend\":true,\"legendPosition\":\"top\",\"smoothLines\":false,\"scale\":\"linear\",\"interpolate\":\"linear\",\"mode\":\"stacked\",\"times\":[],\"addTimeMarker\":false,\"defaultYExtents\":false,\"setYExtents\":false,\"yAxis\":{}},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{\"customLabel\":\"AWS Events\"}},{\"id\":\"2\",\"enabled\":true,\"type\":\"date_histogram\",\"schema\":\"segment\",\"params\":{\"field\":\"eventTime\",\"interval\":\"auto\",\"customInterval\":\"2h\",\"min_doc_count\":1,\"extended_bounds\":{},\"customLabel\":\"Time\"}}],\"listeners\":{}}", 79 | "uiStateJSON": "{}", 80 | "description": "", 81 | "version": 1, 82 | "kibanaSavedObjectMeta": { 83 | "searchSourceJSON": "{\"index\":\"logstash-*\",\"query\":{\"query_string\":{\"query\":\"*\",\"analyze_wildcard\":true}},\"filter\":[]}" 84 | } 85 | } 86 | }, 87 | { 88 | "_id": "Active-Users", 89 | "_type": "visualization", 90 | "_source": { 91 | "title": "Active Users", 92 | "visState": "{\"title\":\"Active Users\",\"type\":\"histogram\",\"params\":{\"shareYAxis\":true,\"addTooltip\":true,\"addLegend\":true,\"legendPosition\":\"top\",\"scale\":\"linear\",\"mode\":\"stacked\",\"times\":[],\"addTimeMarker\":false,\"defaultYExtents\":false,\"setYExtents\":false,\"yAxis\":{}},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{\"customLabel\":\"Calls\"}},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"segment\",\"params\":{\"field\":\"userIdentity.userName.keyword\",\"size\":5,\"order\":\"desc\",\"orderBy\":\"1\",\"customLabel\":\"Users\"}}],\"listeners\":{}}", 93 | "uiStateJSON": "{}", 94 | "description": "", 95 | "version": 1, 96 | "kibanaSavedObjectMeta": { 97 | "searchSourceJSON": "{\"index\":\"logstash-*\",\"query\":{\"query_string\":{\"query\":\"*\",\"analyze_wildcard\":true}},\"filter\":[]}" 98 | } 99 | } 100 | }, 101 | { 102 | "_id": "Regions-In-Use", 103 | "_type": "visualization", 104 | "_source": { 105 | "title": "Regions In Use", 106 | "visState": "{\"title\":\"Regions In Use\",\"type\":\"tagcloud\",\"params\":{\"scale\":\"linear\",\"orientation\":\"single\",\"minFontSize\":16,\"maxFontSize\":45},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{}},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"segment\",\"params\":{\"field\":\"awsRegion.keyword\",\"size\":10,\"order\":\"desc\",\"orderBy\":\"1\",\"customLabel\":\"Regions in use\"}}],\"listeners\":{}}", 107 | "uiStateJSON": "{}", 108 | "description": "", 109 | "version": 1, 110 | "kibanaSavedObjectMeta": { 111 | "searchSourceJSON": "{\"index\":\"logstash-*\",\"query\":{\"query_string\":{\"query\":\"*\",\"analyze_wildcard\":true}},\"filter\":[]}" 112 | } 113 | } 114 | }, 115 | { 116 | "_id": "Event-Names", 117 | "_type": "visualization", 118 | "_source": { 119 | "title": "Event Names", 120 | "visState": "{\"title\":\"Event Names\",\"type\":\"pie\",\"params\":{\"shareYAxis\":true,\"addTooltip\":true,\"addLegend\":true,\"legendPosition\":\"right\",\"isDonut\":false},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{}},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"segment\",\"params\":{\"field\":\"eventName.keyword\",\"size\":10,\"order\":\"desc\",\"orderBy\":\"1\",\"customLabel\":\"AWS Events\"}}],\"listeners\":{}}", 121 | "uiStateJSON": "{}", 122 | "description": "", 123 | "version": 1, 124 | "kibanaSavedObjectMeta": { 125 | "searchSourceJSON": "{\"index\":\"logstash-*\",\"query\":{\"query_string\":{\"query\":\"*\",\"analyze_wildcard\":true}},\"filter\":[]}" 126 | } 127 | } 128 | }, 129 | { 130 | "_id": "Top-10-Active-AWS-Services", 131 | "_type": "visualization", 132 | "_source": { 133 | "title": "Top 10 Active AWS Services", 134 | "visState": "{\"title\":\"Top 10 Active AWS Services\",\"type\":\"pie\",\"params\":{\"shareYAxis\":true,\"addTooltip\":true,\"addLegend\":true,\"legendPosition\":\"left\",\"isDonut\":false},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{\"customLabel\":\"AWS Services\"}},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"segment\",\"params\":{\"field\":\"eventSource.keyword\",\"size\":10,\"order\":\"desc\",\"orderBy\":\"1\",\"customLabel\":\"Event Sources\"}}],\"listeners\":{}}", 135 | "uiStateJSON": "{}", 136 | "description": "", 137 | "version": 1, 138 | "kibanaSavedObjectMeta": { 139 | "searchSourceJSON": "{\"index\":\"logstash-*\",\"query\":{\"query_string\":{\"query\":\"*\",\"analyze_wildcard\":true}},\"filter\":[{\"$state\":{\"store\":\"globalState\"},\"meta\":{\"alias\":null,\"disabled\":false,\"index\":\"logstash-*\",\"key\":\"query\",\"negate\":true,\"value\":\"Describe*\"},\"query\":{\"query_string\":{\"analyze_wildcard\":true,\"query\":\"Describe*\"}}},{\"$state\":{\"store\":\"globalState\"},\"meta\":{\"alias\":null,\"disabled\":false,\"index\":\"logstash-*\",\"key\":\"query\",\"negate\":true,\"value\":\"List*\"},\"query\":{\"query_string\":{\"analyze_wildcard\":true,\"query\":\"List*\"}}},{\"$state\":{\"store\":\"globalState\"},\"meta\":{\"alias\":null,\"disabled\":false,\"index\":\"logstash-*\",\"key\":\"query\",\"negate\":true,\"value\":\"Get*\"},\"query\":{\"query_string\":{\"analyze_wildcard\":true,\"query\":\"Get*\"}}},{\"$state\":{\"store\":\"globalState\"},\"meta\":{\"alias\":null,\"disabled\":false,\"index\":\"logstash-*\",\"key\":\"query\",\"negate\":true,\"value\":\"Assume*\"},\"query\":{\"query_string\":{\"analyze_wildcard\":true,\"query\":\"Assume*\"}}}]}" 140 | } 141 | } 142 | }, 143 | { 144 | "_id": "Console-Login-Number", 145 | "_type": "visualization", 146 | "_source": { 147 | "title": "Console Login Number", 148 | "visState": "{\"title\":\"Console Login Number\",\"type\":\"metric\",\"params\":{\"handleNoResults\":true,\"fontSize\":\"43\"},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{\"customLabel\":\"\"}}],\"listeners\":{}}", 149 | "uiStateJSON": "{}", 150 | "description": "", 151 | "savedSearchId": "ConsoleLogins", 152 | "version": 1, 153 | "kibanaSavedObjectMeta": { 154 | "searchSourceJSON": "{\"filter\":[{\"meta\":{\"disabled\":false,\"index\":\"logstash-*\",\"key\":\"query\",\"negate\":true,\"value\":\"Describe*\",\"alias\":null},\"query\":{\"query_string\":{\"query\":\"Describe*\",\"analyze_wildcard\":true}},\"$state\":{\"store\":\"globalState\"}},{\"meta\":{\"disabled\":false,\"index\":\"logstash-*\",\"key\":\"query\",\"negate\":true,\"value\":\"List*\",\"alias\":null},\"query\":{\"query_string\":{\"query\":\"List*\",\"analyze_wildcard\":true}},\"$state\":{\"store\":\"globalState\"}},{\"meta\":{\"disabled\":false,\"index\":\"logstash-*\",\"key\":\"query\",\"negate\":true,\"value\":\"Get*\",\"alias\":null},\"query\":{\"query_string\":{\"query\":\"Get*\",\"analyze_wildcard\":true}},\"$state\":{\"store\":\"globalState\"}},{\"meta\":{\"disabled\":false,\"index\":\"logstash-*\",\"key\":\"query\",\"negate\":true,\"value\":\"Assume*\",\"alias\":null},\"query\":{\"query_string\":{\"query\":\"Assume*\",\"analyze_wildcard\":true}},\"$state\":{\"store\":\"globalState\"}}]}" 155 | } 156 | } 157 | }, 158 | { 159 | "_id": "Account-Number", 160 | "_type": "visualization", 161 | "_source": { 162 | "title": "Account Number", 163 | "visState": "{\"title\":\"Account Number\",\"type\":\"tagcloud\",\"params\":{\"scale\":\"linear\",\"orientation\":\"single\",\"minFontSize\":12,\"maxFontSize\":32},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{}},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"segment\",\"params\":{\"field\":\"recipientAccountId.keyword\",\"size\":5,\"order\":\"desc\",\"orderBy\":\"1\",\"customLabel\":\"Account Number\"}}],\"listeners\":{}}", 164 | "uiStateJSON": "{}", 165 | "description": "", 166 | "version": 1, 167 | "kibanaSavedObjectMeta": { 168 | "searchSourceJSON": "{\"index\":\"logstash-*\",\"query\":{\"query_string\":{\"query\":\"*\",\"analyze_wildcard\":true}},\"filter\":[]}" 169 | } 170 | } 171 | } 172 | ] -------------------------------------------------------------------------------- /documentation/image1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/adcreare/traildash2/1dca40a665de2c06ac21922d4e4710555b159cf4/documentation/image1.png -------------------------------------------------------------------------------- /documentation/image2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/adcreare/traildash2/1dca40a665de2c06ac21922d4e4710555b159cf4/documentation/image2.png -------------------------------------------------------------------------------- /documentation/image3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/adcreare/traildash2/1dca40a665de2c06ac21922d4e4710555b159cf4/documentation/image3.png -------------------------------------------------------------------------------- /documentation/image4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/adcreare/traildash2/1dca40a665de2c06ac21922d4e4710555b159cf4/documentation/image4.png -------------------------------------------------------------------------------- /documentation/image5.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/adcreare/traildash2/1dca40a665de2c06ac21922d4e4710555b159cf4/documentation/image5.png -------------------------------------------------------------------------------- /documentation/image6.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/adcreare/traildash2/1dca40a665de2c06ac21922d4e4710555b159cf4/documentation/image6.png -------------------------------------------------------------------------------- /documentation/image7.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/adcreare/traildash2/1dca40a665de2c06ac21922d4e4710555b159cf4/documentation/image7.png -------------------------------------------------------------------------------- /documentation/image8.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/adcreare/traildash2/1dca40a665de2c06ac21922d4e4710555b159cf4/documentation/image8.png -------------------------------------------------------------------------------- /documentation/image9.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/adcreare/traildash2/1dca40a665de2c06ac21922d4e4710555b159cf4/documentation/image9.png -------------------------------------------------------------------------------- /lambda-code/.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/adcreare/traildash2/1dca40a665de2c06ac21922d4e4710555b159cf4/lambda-code/.DS_Store -------------------------------------------------------------------------------- /lambda-code/index.js: -------------------------------------------------------------------------------- 1 | /* 2 | Auther: Adcreare 3 | Purpose: A Lambda function to be attached to an S3 bucket as a Trigger to consume cloudtrail events 4 | and load them into elastic search. 5 | 6 | Function will process S3 put event object, will download the supplied file 7 | Gunzip extract the json payload and them load that into elastic search 8 | 9 | Designed specifically as part of the TrailDash2 project 10 | */ 11 | 12 | 13 | var AWS = require('aws-sdk') 14 | var zlib = require('zlib'); 15 | 16 | 17 | /* Const Globals */ 18 | var s3 = new AWS.S3(); 19 | var esDomain = { 20 | region: 'us-east-1', 21 | index: 'logstash-', 22 | doctype: 'cloudtrail' 23 | }; 24 | 25 | var s3 = new AWS.S3(); 26 | 27 | /* 28 | * The AWS credentials are picked up from the environment. 29 | * They belong to the IAM role assigned to the Lambda function. 30 | * Since the ES requests are signed using these credentials, 31 | * make sure to apply a policy that permits ES domain operations 32 | * to the role. 33 | */ 34 | var creds = new AWS.EnvironmentCredentials('AWS'); 35 | 36 | 37 | 38 | /* Lambda "main": Execution starts here */ 39 | exports.handler = function(event, context, callback) { 40 | //setup the callback and esurl so we can access that across functions without a prototype or bind 41 | global.lambdaCallback = callback; 42 | global.elasticsearchURL = process.env.elasticsearchurl 43 | 44 | 45 | console.log(JSON.stringify(event)); //log the recreived event 46 | 47 | //download item from S3 to disk 48 | var params = {} 49 | params.Bucket = event.Records[0].s3.bucket.name; 50 | params.Key = event.Records[0].s3.object.key; 51 | 52 | s3.getObject(params, function(err, data) { 53 | if (err) 54 | { 55 | console.log(err, err.stack); // an error occurred 56 | global.lambdaCallback(err,'ERROR downloading from S3') //exit funtion 57 | } 58 | else 59 | { 60 | //if call is all good then run the buffer through extractFile 61 | extractFile(data.Body); // successful response 62 | } 63 | }); 64 | } 65 | 66 | //Function to extract a gzip file 67 | function extractFile(buffer) 68 | { 69 | zlib.gunzip(buffer,extractFileCallBack) 70 | } 71 | 72 | //Callback for file extraction 73 | function extractFileCallBack(error,result) 74 | { 75 | if (error){ 76 | global.lambdaCallback(error,'unable to extract file!') //exit function 77 | } 78 | else 79 | { 80 | loadIntoES(JSON.parse(result.toString())); //now load the data into ES 81 | } 82 | } 83 | 84 | 85 | //Function process each jsonpayload into a ES request that is made async 86 | function loadIntoES(jsonPayload) 87 | { 88 | //jsonPayload.Records.forEach(signAndSendRequestToES(record)) 89 | 90 | for (var i = 0; i < jsonPayload.Records.length; i++) 91 | { 92 | signAndSendRequestToES(jsonPayload.Records[i]); 93 | } 94 | } 95 | 96 | /* 97 | Purpose: The main lifting of the code 98 | 1. Works out which ES index to load the json payload into based on the current date 99 | 2. Builds the request 100 | 3. Uses the AWS>Signers.V4 library to sign the request with the IAM credentials 101 | 4. Posts the request to ES using the aws nodehttpclient 102 | */ 103 | function signAndSendRequestToES(singlejsonPayload) 104 | { 105 | var endpoint = new AWS.Endpoint(global.elasticsearchURL); 106 | var req = new AWS.HttpRequest(endpoint); 107 | 108 | //console.log(singlejsonPayload); 109 | //return 0; 110 | var date = new Date(singlejsonPayload.eventTime) 111 | //date = date.getFullYear()+"."+date.getMonth()+"."+date.getDate() 112 | mytime = date.toISOString(); 113 | mytime = mytime.substring(0,mytime.indexOf('T')); 114 | date = mytime.replace(/-/g,'.'); 115 | 116 | //build request 117 | req.method = 'POST'; 118 | req.path = '/' +esDomain.index+date+"/"+esDomain.doctype; 119 | req.region = esDomain.region; 120 | req.body = JSON.stringify(singlejsonPayload); 121 | req.headers['presigned-expires'] = false; 122 | req.headers['Host'] = global.elasticsearchURL; 123 | 124 | // Sign the request (Sigv4) 125 | var signer = new AWS.Signers.V4(req, 'es'); 126 | signer.addAuthorization(creds, new Date()); 127 | 128 | 129 | // Post document to ES 130 | var send = new AWS.NodeHttpClient(); 131 | 132 | send.handleRequest(req, null, function(httpResp) { 133 | var body = ''; 134 | 135 | //console.log(httpResp) 136 | 137 | httpResp.on('data', function (chunk) { 138 | body += chunk; 139 | }); 140 | httpResp.on('error', function (chunk) { 141 | console.log("ERRRRROROR") 142 | console.log(chunk) 143 | console.log("error chunk above - end") 144 | }); 145 | httpResp.on('end', function (chunk) { 146 | console.log('DONE!'); // log done but don't callback - we may have other threds still running here 147 | console.log('Date is: '+date) 148 | console.log("--- Elastic Search Response---") 149 | console.log(body) 150 | console.log("--- END Elastic Search Response ---") 151 | }); 152 | }, function(err) { 153 | console.log('Error: ' + err); /* Log the error - this probably should go to retry at the moment it just fails silently 154 | For cloudtrail this is acceptable for other uses it may not be */ 155 | }); 156 | } 157 | -------------------------------------------------------------------------------- /lambda-code/index.js.zip: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/adcreare/traildash2/1dca40a665de2c06ac21922d4e4710555b159cf4/lambda-code/index.js.zip --------------------------------------------------------------------------------