├── .gitignore ├── ASLCustomSourceGlueRole.yaml ├── CODE_OF_CONDUCT ├── CONTRIBUTING.md ├── LICENSE ├── README.md ├── cdk ├── app.py ├── cdk.json ├── ocsf_transformation │ └── ocsf_transformation_stack.py ├── requirements.txt └── setup.py ├── images ├── asl_custom_source_create.png ├── custom-sources │ ├── .DS_Store │ ├── alb │ │ └── alb_arch.png │ ├── aws-network-firewall │ │ ├── S3_SQS.jpg │ │ ├── S3_create_event_notification.jpg │ │ ├── S3_obj_actions.jpg │ │ ├── anf_arch.png │ │ ├── anf_update_glue_table.jpg │ │ └── deployment_overview.png │ └── windows-sysmon │ │ ├── deployment_overview.png │ │ ├── edit_crawler.png │ │ ├── edit_crawler_data_source.png │ │ ├── edit_crawler_output_scheduling.png │ │ └── sysmon_arch.png ├── custom_sources_config.png ├── ocsf_fundamentals.png ├── ocsf_v1_1_0.jpg ├── solution_architecture.png └── sources_config.png ├── mapping_docs ├── application-load-balancer-mapping.md ├── aws-network-firewall-mapping.md └── windows-sysmon-mapping.md └── transformation_function ├── __init__.py ├── app.py ├── mappings ├── aws_alb.json ├── aws_nfw.json └── windows_sysmon.json ├── preprocessors ├── alb_preprocessor.py └── sysmon_preprocessor.py ├── requirements.txt └── sources_config.json /.gitignore: -------------------------------------------------------------------------------- 1 | **/.DS_Store 2 | .DS_Store 3 | **/__pycache__ 4 | .vscode/** 5 | .aws-sam/** 6 | **/.venv/** 7 | **/cdk.out 8 | .aws-sam 9 | .vscode 10 | git_excludes/* 11 | -------------------------------------------------------------------------------- /ASLCustomSourceGlueRole.yaml: -------------------------------------------------------------------------------- 1 | AWSTemplateFormatVersion: 2010-09-09 2 | Description: >- 3 | This template creates an IAM role for the AWS Glue configuration for the Security Lake custom source. 4 | Parameters: 5 | CustomSourceName: 6 | Type: String 7 | Description: Name of the custom source. 8 | 9 | Resources: 10 | CustomSourceGlueRole: 11 | Type: AWS::IAM::Role 12 | Properties: 13 | RoleName: !Join ['', [ !Ref CustomSourceName, '-glue-role'] ] 14 | Path: "/service-role/" 15 | Policies: 16 | - PolicyName: !Join ['', [ !Ref CustomSourceName, '-glue-policy'] ] 17 | PolicyDocument: 18 | Version: 2012-10-17 19 | Statement: 20 | - Sid: S3WriteRead 21 | Effect: Allow 22 | Action: 23 | - "s3:GetObject" 24 | - "s3:PutObject" 25 | Resource: arn:aws:s3:::aws-security-data-lake-*/ext/* 26 | - Sid: GlueServicePolicy 27 | Effect: Allow 28 | Action: 29 | - glue:* 30 | - s3:GetBucketLocation 31 | - s3:ListBucket 32 | - s3:ListAllMyBuckets 33 | - s3:GetBucketAcl 34 | - ec2:DescribeVpcEndpoints 35 | - ec2:DescribeRouteTables 36 | - ec2:CreateNetworkInterface 37 | - ec2:DeleteNetworkInterface 38 | - ec2:DescribeNetworkInterfaces 39 | - ec2:DescribeSecurityGroups 40 | - ec2:DescribeSubnets 41 | - ec2:DescribeVpcAttribute 42 | - iam:ListRolePolicies 43 | - iam:GetRole 44 | - iam:GetRolePolicy 45 | - cloudwatch:PutMetricData 46 | Resource: '*' 47 | - Sid: GlueS3Create 48 | Effect: Allow 49 | Action: 50 | - s3:CreateBucket 51 | Resource: arn:aws:s3:::aws-glue-* 52 | - Sid: GlueS3ReadWrite 53 | Effect: Allow 54 | Action: 55 | - s3:GetObject 56 | - s3:PutObject 57 | - s3:DeleteObject 58 | Resource: 59 | - arn:aws:s3:::aws-glue-*/* 60 | - arn:aws:s3:::*/*aws-glue-*/* 61 | - Sid: GlueCrawlerS3Read 62 | Effect: Allow 63 | Action: 64 | - s3:GetObject 65 | Resource: 66 | - arn:aws:s3:::crawler-public* 67 | - arn:aws:s3:::aws-glue-* 68 | - Sid: GlueLogging 69 | Effect: Allow 70 | Action: 71 | - logs:CreateLogGroup 72 | - logs:CreateLogStream 73 | - logs:PutLogEvents 74 | Resource: arn:aws:logs:*:*:*:/aws-glue/* 75 | - Sid: EC2CreateTags 76 | Effect: Allow 77 | Condition: 78 | "ForAllValues:StringEquals": 79 | "aws:TagKeys": "aws-glue-service-resource" 80 | Action: 81 | - ec2:CreateTags 82 | - ec2:DeleteTags 83 | Resource: 84 | - arn:aws:ec2:*:*:network-interface/* 85 | - arn:aws:ec2:*:*:security-group/* 86 | - arn:aws:ec2:*:*:instance/* 87 | 88 | AssumeRolePolicyDocument: 89 | Version: "2012-10-17" 90 | Statement: 91 | - Effect: Allow 92 | Principal: 93 | Service: 94 | - glue.amazonaws.com 95 | Action: 96 | - sts:AssumeRole 97 | Outputs: 98 | CustomSourceGlueRoleARN: 99 | Description: ARN of the IAM role created for Glue to use with custom sources. 100 | Value: !GetAtt CustomSourceGlueRole.Arn -------------------------------------------------------------------------------- /CODE_OF_CONDUCT: -------------------------------------------------------------------------------- 1 | ## Code of Conduct 2 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 3 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact opensource-codeofconduct@amazon.com with any additional questions or comments. 4 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing Guidelines 2 | 3 | Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional 4 | documentation, we greatly value feedback and contributions from our community. 5 | 6 | Please read through this document before submitting any issues or pull requests to ensure we have all the necessary 7 | information to effectively respond to your bug report or contribution. 8 | 9 | 10 | ## Reporting Bugs/Feature Requests 11 | 12 | We welcome you to use the GitHub issue tracker to report bugs or suggest features. 13 | 14 | When filing an issue, please check existing open, or recently closed, issues to make sure somebody else hasn't already 15 | reported the issue. Please try to include as much information as you can. Details like these are incredibly useful: 16 | 17 | * A reproducible test case or series of steps 18 | * The version of our code being used 19 | * Any modifications you've made relevant to the bug 20 | * Anything unusual about your environment or deployment 21 | 22 | 23 | ## Contributing via Pull Requests 24 | Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that: 25 | 26 | 1. You are working against the latest source on the *main* branch. 27 | 2. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already. 28 | 3. You open an issue to discuss any significant work - we would hate for your time to be wasted. 29 | 30 | To send us a pull request, please: 31 | 32 | 1. Fork the repository. 33 | 2. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change. 34 | 3. Ensure local tests pass. 35 | 4. Commit to your fork using clear commit messages. 36 | 5. Send us a pull request, answering any default questions in the pull request interface. 37 | 6. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation. 38 | 39 | GitHub provides additional document on [forking a repository](https://help.github.com/articles/fork-a-repo/) and 40 | [creating a pull request](https://help.github.com/articles/creating-a-pull-request/). 41 | 42 | 43 | ## Finding contributions to work on 44 | Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels (enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any 'help wanted' issues is a great place to start. 45 | 46 | 47 | ## Code of Conduct 48 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 49 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 50 | opensource-codeofconduct@amazon.com with any additional questions or comments. 51 | 52 | 53 | ## Security issue notifications 54 | If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github issue. 55 | 56 | 57 | ## Licensing 58 | 59 | See the [LICENSE](LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution. 60 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining a copy of this 4 | software and associated documentation files (the "Software"), to deal in the Software 5 | without restriction, including without limitation the rights to use, copy, modify, 6 | merge, publish, distribute, sublicense, and/or sell copies of the Software, and to 7 | permit persons to whom the Software is furnished to do so. 8 | 9 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, 10 | INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A 11 | PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT 12 | HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION 13 | OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE 14 | SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Amazon Security Lake Transformation Library (v3) 2 | 3 | This project provides a solution for transforming various log sources into the Open Cybersecurity Schema Framework (OCSF) format for use with Amazon Security Lake. It enables you to ingest, transform, and store security data from custom sources in a standardized format. 4 | 5 | Supported custom sources (**This project produces logs mapped to OCSF v1.1.0.**): 6 | 7 | - [Windows Sysmon](./mapping_docs/windows-sysmon-mapping.md) 8 | - [AWS Network Firewall](./mapping_docs/aws-network-firewall-mapping.md) 9 | - [Application Load Balancer](./mapping_docs/application-load-balancer-mapping.md) 10 | 11 | > **_NOTE:_** If you have used a previous version of this library to deploy custom sources using AWS Serverless Application Model (AWS SAM), please refer to the [FAQs](#frequently-asked-questions-faqs) on migration path. 12 | 13 | ## What is Amazon Security Lake? 14 | 15 | Amazon Security Lake is a fully managed security data lake service that makes it easier to store, analyze, and manage security data at scale. Security Lake helps security teams centralize security data from cloud, on-premises, and custom sources into a purpose-built data lake stored in your AWS account. Security Lake automates the collection and management of security data across accounts and AWS Regions. 16 | 17 | Key benefits of Amazon Security Lake: 18 | 19 | - Centralizes security data from AWS services, third-party sources, and custom sources. 20 | - Normalizes data using the OCSF standard format. 21 | - Optimizes storage and query performance with Apache Parquet format. 22 | - Provides full ownership and control of your security data. 23 | - Enables flexible analytics with your preferred tools. 24 | 25 | ## Understanding Open Cybersecurity Schema Framework (OCSF) 26 | 27 | OCSF is an open-source project that provides a standardized schema for security data. It addresses the complex and heterogeneous nature of security log collection and analysis by providing a vendor-agnostic framework. 28 | 29 | Key aspects of OCSF: 30 | 31 | - Vendor-agnostic schema for security events. 32 | - Organized into categories (System Activity, Findings, Identity and Access Management, etc.). 33 | - Each category contains classes based on activity type (File System Activity, Process Activity, etc.). 34 | - Each class has a unique identifier (class UID). 35 | - Enables consistent security data normalization across different sources. 36 | 37 | OCSF Fundamentals 38 | 39 | Logs generated by various sources can have significantly different schemas. Storing logs in different schemas can add considerable overhead around data normalization for security teams before they can derive any significant value from the log information. All log data stored with Security Lake must conform to OCSF. An essential part of bringing custom log data into Security Lake is to map the raw attributes to the OCSF schema. This operation should be non-destructive, which means if there are attributes in the raw log that don't map directly to the OCSF class attributes, you should map those to the `unmapped` attribute. 40 | 41 | ![OCSF v1.1.0](./images/ocsf_v1_1_0.jpg) 42 | 43 | ## Prerequisites 44 | 45 | Before deploying this solution, ensure the below: 46 | 47 | 1. **AWS Organizations configured** in your AWS environment. 48 | 2. **[Security Lake activated](https://docs.aws.amazon.com/organizations/latest/userguide/services-that-can-integrate-sl.html#integrate-enable-da-sl)** with a delegated administrator configured in the [Log Archive AWS account](https://docs.aws.amazon.com/prescriptive-guidance/latest/security-reference-architecture/log-archive.html). 49 | 3. **[AWS CDK installed](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html)** on your development machine. 50 | 4. **Python 3.8 or higher** installed. 51 | 5. **AWS CLI configured** with appropriate credentials. 52 | 53 | ## Project Structure 54 | 55 | ``` 56 | amazon-security-lake-transformation-library/ 57 | ├── cdk/ # CDK deployment code 58 | │ ├── app.py # CDK application entry point 59 | │ ├── ocsf_transformation/ # Main stack definition 60 | │ └── requirements.txt # CDK dependencies 61 | └── transformation_function/ # Lambda function code 62 | ├── app.py # Lambda handler 63 | ├── mappings/ # OCSF mapping configurations. **Put all custom source mappings here.** 64 | ├── preprocessors/ # Source-specific preprocessors. **Add preprocessors here for any new log sources that need it** 65 | └── sources_config.json # Source configuration. **Update any new custom source definition here.** 66 | ``` 67 | 68 | ## How It Works 69 | 70 | #### Solution Architecture 71 | 72 | Solution architecture 73 | 74 | The solution consists of three main components: 75 | 76 | 1. **Transformation Lambda Function**: Processes incoming logs and transforms them to OCSF format. 77 | 2. **Infrastructure (CDK)**: Deploys the necessary AWS resources based on your configuration. 78 | 3. **Custom source mappings**: Mapping files are stored in the [mappings](./transformation_function/mappings/) folder under [transformation_function](./transformation_function/) folder. 79 | 80 | The transformation process follows these steps: 81 | 82 | 1. Raw logs are ingested via Kinesis Data Streams or S3 buckets. 83 | 2. The Lambda function processes these logs using source-specific preprocessors. 84 | 3. The function maps the log fields to OCSF schema using mapping configurations. 85 | 4. Transformed data is stored in the Security Lake S3 bucket in Parquet format. 86 | 87 | > **_NOTE:_** If you want to add more sources other than the supported after you have deployed the solution, please ensure that you update the mapping in the [mappings](./transformation_function/mappings/) and [sources_config.json](./transformation_function/sources_config.json) folder under [transformation_function](./transformation_function/) folder. 88 | Custom sources config 89 | 90 | ## Deployment Steps 91 | 92 | #### STEP 1: Add custom source in Security Lake 93 | 94 | Navigate to the Security Lake console. Click **Custom sources** then click **Create custom source**. Update the details of the custom source and click **Create**. 95 | 96 | Create custom source 97 | 98 | The service will automatically create the Glue IAM role and Crawler configuration for the custom source. 99 | 100 | > **_NOTE:_** If you want to create the custom source programmatically, please refer to Q2 in the [FAQs](#frequently-asked-questions-faqs) section. 101 | 102 | #### STEP 2: Custom Source Mapping Configuration 103 | 104 | The solution uses mapping files, found in [transformation_function/mappings](./transformation_function/mappings/) to transform logs to OCSF format. Here's an example of the mapping structure: 105 | 106 | ```json 107 | { 108 | "custom_source_events": { 109 | "source_name": "windows-sysmon", 110 | "matched_field": "$.EventId", 111 | "timestamp": { 112 | "field": "$.Description.UtcTime", 113 | "format": "%Y-%m-%d %H:%M:%S.%f" 114 | }, 115 | "ocsf_mapping": { 116 | "1": { 117 | "schema": "process_activity", 118 | "schema_mapping": { 119 | "metadata": { 120 | "profiles": "host", 121 | "version": "v1.0.0", 122 | "product": { 123 | "name": "System Monitor (Sysmon)", 124 | "vendor_name": "Microsoft Sysinternals", 125 | "version": "v15.0" 126 | } 127 | }, 128 | "activity_id": { 129 | "enum": { 130 | "evaluate": "$.EventId", 131 | "values": { 132 | "1": 1, 133 | "5": 2 134 | }, 135 | "other": 99 136 | } 137 | } 138 | } 139 | } 140 | } 141 | } 142 | } 143 | ``` 144 | 145 | Key components of the mapping: 146 | 147 | - **Static values**: Directly specified in the configuration (e.g., metadata). 148 | - **Derived values**: Prefixed with `$.` to extract from log data (e.g., `$.event.src_ip`). 149 | - **Enum mappings**: Map source values to OCSF-defined values using the `enum` type. 150 | 151 | #### STEP 3: AWS Cloud Development Kit (AWS CDK) 152 | 153 | 1. [Configure AWS credentials](https://docs.aws.amazon.com/cdk/v2/guide/configure-access.html) for CDK to use. This includes using AWS CLI commands like `aws configure` to configure security credentials on your local machine. 154 | 2. Clone the repository. 155 | 3. Set up a Python virtual environment. 156 | 157 | ```bash 158 | cd amazon-security-lake-transformation-library/cdk 159 | python -m venv .venv 160 | source .venv/bin/activate # On Windows, use `.venv\Scripts\activate` 161 | pip install -r requirements.txt 162 | ``` 163 | 164 | 4. [Bootstrap your CDK environment](https://docs.aws.amazon.com/cdk/v2/guide/bootstrapping-env.html). Replace the and with the values of the environment where you are choosing to deploy the project. 165 | 166 | ```bash 167 | cdk bootstrap aws:/// 168 | ``` 169 | 170 | #### Deployment Options 171 | 172 | You can deploy the solution using CDK context parameters or environment variables: 173 | 174 | **Using CDK Context Parameters** 175 | 176 | ```bash 177 | cdk deploy \ 178 | --context log_event_source=All \ 179 | --context asl_bucket_location=my-security-lake-bucket \ 180 | --context raw_log_s3_bucket_name=my-existing-bucket \ 181 | --context add_s3_event_notification=true \ 182 | --context kinesis_user_arns="arn:aws:iam::123456789012:user/kinesis-user1" \ 183 | --context stack_name="my-ocsf-transformation" # Optional - used when upgrading from previous version 184 | ``` 185 | 186 | **Using Environment Variables** 187 | 188 | ```bash 189 | export LOG_EVENT_SOURCE=All 190 | export ASL_BUCKET_LOCATION=my-security-lake-bucket 191 | export RAW_LOG_S3_BUCKET_NAME=my-existing-bucket 192 | export ADD_S3_EVENT_NOTIFICATION=true 193 | export KINESIS_USER_ARNS="arn:aws:iam::123456789012:user/kinesis-user1" 194 | 195 | cdk deploy 196 | ``` 197 | 198 | #### Key Parameters 199 | 200 | | Parameter | Description | Allowed Values | Default | 201 | |-----------|-------------|----------------|---------| 202 | | `log_event_source` | Source of raw logs | `S3Bucket`, `KinesisDataStream`, `All` | `All` | 203 | | `asl_bucket_location` | Security Lake S3 location | String (required) | - | 204 | | `raw_log_s3_bucket_name` | Existing S3 bucket name | String | `""` (creates new) | 205 | | `add_s3_event_notification` | Add S3 event notification | `true`, `false` | `false` | 206 | | `kinesis_user_arns` | IAM identities for Kinesis | List of ARNs | `[]` | 207 | | `stack_name` | CloudFormation stack name | String | `OcsfTransformationStack` | 208 | 209 | ## Accessing Transformed Data 210 | 211 | After deployment, you'll need to configure access to the transformed data: 212 | 213 | 1. Configure access to the tables in AWS LakeFormation. 214 | 2. Select the Security Lake database (`amazon_security_lake_glue_db_*`). 215 | 3. Select the custom source tables (e.g., `amazon_security_lake_table_*_ext_windows_sysmon_*`). 216 | 4. Grant appropriate permissions (Select, Describe) to your users or roles. 217 | 218 | ## Conclusion 219 | 220 | The Amazon Security Lake Transformation Library provides a flexible and powerful way to integrate custom security data sources with Amazon Security Lake. By leveraging the OCSF schema, you can standardize your security data for more effective analysis and threat detection. 221 | 222 | 223 | ## Frequently Asked Questions (FAQs) 224 | 225 | **Q1: Are there any breaking changes in this update?** 226 | 227 | **A1:** **YES, there are breaking changes in this version**, these changes make operating with sources with multiple classes like *windows-sysmon* significantly easier by taking away the need to manually update the underlying AWS Glue Crawler. To upgrade from the previous version, perform the steps below: 228 | 229 | 1. [Delete the custom source definition in Security Lake](https://docs.aws.amazon.com/security-lake/latest/userguide/delete-custom-source.html). You can use the [service console](https://console.aws.amazon.com/securitylake/) or command line interface. Deleting a custom source does not delete the underlying Glue crawler, custom source IAM role for Glue and S3 location. 230 | 2. Navigate to the [AWS Glue service console](https://console.aws.amazon.com/glue). Search for **Crawlers** under **Data Catalog** and delete the crawler associated with the custom source. 231 | 3. Create a new custom source in Security Lake, ensure that the name of the custom source is the same as defined in the [sources_config.json](./transformation_function/sources_config.json) file. 232 | 233 | Sources config 234 | 235 | **Q2: I use automation with custom source creation. How do I create a custom source without doing console actions?** 236 | 237 | **A2:** Use the following steps to automate custom source creation with this project. 238 | 239 | 1. Set up the Glue IAM role for the custom resource by deploying the [ASLCustomSourceGlueRole.yaml](./ASLCustomSourceGlueRole.yaml) CloudFormation template. The template requires the following inputs: 240 | 241 | * `CustomSourceName`: This is the name of the custom source you want to add in Security Lake. 242 | * `ASLCustomLogSourceLocation`: Amazon Security Lake S3 bucket name with custom log location without the trailing slash (eg. my_bucket/ext/my_custom_source) 243 | 244 | The template produces one output, `CustomSourceGlueRoleARN`, which is the ARN of the IAM role created for Glue to use with custom sources. 245 | 246 | Capture this output for use in the next step. 247 | 248 | 2. Use AWS CloudShell, a browser based shell, in the Security Lake delegated administrator account to run the command in this step after you have replaced the placeholders. 249 | 250 | * `` with the region where Security Lake is configured. 251 | * `` with the value of the CloudFormation output named `CustomSourceGlueRoleARN` captured in the previous step. 252 | * `` is an alphanumeric value you can assign to configure fine grained access control. For the self managed custom sources, you can assign it any value you like. In some cases, where you are using an external product, the vendor will supply the [External ID](https://aws.amazon.com/blogs/security/how-to-use-external-id-when-granting-access-to-your-aws-resources/) to you. 253 | * `` with the Security Lake delegated administrator AWS Account ID. 254 | 255 | > **_NOTE:_** For records pertaining to accounts outside of AWS, we recommend using a string such as *_`external`_* or *_`external_`_*. You should take care to avoid ambiguity in naming external account IDs so that they do not conflict with AWS account IDs or external account IDs maintained by other identity management systems, as queries across Amazon Security Lake tables may simultaneously cover data across multiple identity management systems. 256 | 257 | ```bash 258 | aws securitylake create-custom-log-source \ 259 | --source-name windows-sysmon \ 260 | --configuration crawlerConfiguration={"roleArn="},providerIdentity={"externalId=,principal="} \ 261 | --event-classes FILE_ACTIVITY PROCESS_ACTIVITY DNS_ACTIVITY NETWORK_ACTIVITY \ 262 | --region 263 | ``` 264 | 265 | The output of the command will reveal the attributes of the associated Glue resources along with Security Lake S3 bucket location and the associated IAM role ARN. Verify with sample output below: 266 | 267 | ```json 268 | { 269 | "source": { 270 | "attributes": { 271 | "crawlerArn": "arn:aws:glue:region:XXX:crawler/windows-sysmon", 272 | "databaseArn": "arn:aws:glue:region:XXX:database/amazon_security_lake_glue_db_region", 273 | "tableArn": "arn:aws:glue:region:XXX:table/amazon_security_lake_table_region_ext_windows_sysmon" 274 | }, 275 | "provider": { 276 | "location": "s3://aws-security-data-lake-region-exampleid/ext/windows-sysmon/", 277 | "roleArn": "arn:aws:iam::XXX:role/AmazonSecurityLake-Provider-windowssysmon-region" 278 | }, 279 | "sourceName": "windows-sysmon" 280 | } 281 | } 282 | ``` 283 | 284 | **Q3: I've deployed the solution but my Athena queries are failing with a HIVE error?** 285 | 286 | **A3:** Re-run the Glue Crawler associated with the custom source to update the partitions. Upon successful completion, re-run the Athena queries. 287 | -------------------------------------------------------------------------------- /cdk/app.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | import os 3 | import aws_cdk as cdk 4 | from ocsf_transformation.ocsf_transformation_stack import OcsfTransformationStack 5 | 6 | app = cdk.App() 7 | 8 | # Get parameters from context or environment variables 9 | log_event_source = app.node.try_get_context("log_event_source") or os.environ.get("LOG_EVENT_SOURCE", "All") 10 | asl_bucket_location = app.node.try_get_context("asl_bucket_location") or os.environ.get("ASL_BUCKET_LOCATION") 11 | raw_log_s3_bucket_name = app.node.try_get_context("raw_log_s3_bucket_name") or os.environ.get("RAW_LOG_S3_BUCKET_NAME", "") 12 | add_s3_event_notification = app.node.try_get_context("add_s3_event_notification") or os.environ.get("ADD_S3_EVENT_NOTIFICATION", False) 13 | 14 | # Parse list parameters (comma-separated strings) 15 | kinesis_user_arns_str = app.node.try_get_context("kinesis_user_arns") or os.environ.get("KINESIS_USER_ARNS", "") 16 | kinesis_user_arns = [arn.strip() for arn in kinesis_user_arns_str.split(",")] if kinesis_user_arns_str else [] 17 | 18 | kinesis_encryption_key_admin_arns_str = app.node.try_get_context("kinesis_encryption_key_admin_arns") or os.environ.get("KINESIS_ENCRYPTION_KEY_ADMIN_ARNS", "") 19 | kinesis_encryption_key_admin_arns = [arn.strip() for arn in kinesis_encryption_key_admin_arns_str.split(",")] if kinesis_encryption_key_admin_arns_str else [] 20 | 21 | # Get the stack name from context or use default 22 | # This allows updating an existing stack by providing its name 23 | stack_name = ( 24 | app.node.try_get_context("stack-name") or 25 | app.node.try_get_context("stack_name") or 26 | "OcsfTransformationStack" 27 | ) 28 | 29 | # Create stack with parameters 30 | OcsfTransformationStack( 31 | app, 32 | "OcsfTransformationStack", # This is just the construct ID, not the actual stack name 33 | stack_name=stack_name, # This is the actual CloudFormation stack name 34 | log_event_source=log_event_source, 35 | asl_bucket_location=asl_bucket_location, 36 | raw_log_s3_bucket_name=raw_log_s3_bucket_name, 37 | kinesis_user_arns=kinesis_user_arns, 38 | kinesis_encryption_key_admin_arns=kinesis_encryption_key_admin_arns, 39 | add_s3_event_notification=add_s3_event_notification, 40 | env=cdk.Environment( 41 | account=os.environ.get("CDK_DEFAULT_ACCOUNT"), 42 | region=os.environ.get("CDK_DEFAULT_REGION") 43 | ), 44 | description="CDK deployment for OCSF transformation library" 45 | ) 46 | 47 | app.synth() 48 | -------------------------------------------------------------------------------- /cdk/cdk.json: -------------------------------------------------------------------------------- 1 | { 2 | "app": "python app.py" 3 | } -------------------------------------------------------------------------------- /cdk/ocsf_transformation/ocsf_transformation_stack.py: -------------------------------------------------------------------------------- 1 | from aws_cdk import ( 2 | Stack, 3 | aws_s3 as s3, 4 | aws_sqs as sqs, 5 | aws_lambda as lambda_, 6 | aws_iam as iam, 7 | aws_kinesis as kinesis, 8 | aws_kms as kms, 9 | aws_s3_notifications as s3n, 10 | aws_lambda_event_sources as lambda_event_sources, 11 | Duration, 12 | RemovalPolicy, 13 | CfnOutput, 14 | CfnResource, 15 | ) 16 | from constructs import Construct 17 | from aws_cdk import CfnDeletionPolicy 18 | 19 | 20 | class OcsfTransformationStack(Stack): 21 | def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None: 22 | # Extract our custom properties from kwargs 23 | log_event_source = kwargs.pop("log_event_source", "All") 24 | asl_bucket_location = kwargs.pop("asl_bucket_location") 25 | raw_log_s3_bucket_name = kwargs.pop("raw_log_s3_bucket_name", None) 26 | kinesis_user_arns = kwargs.pop("kinesis_user_arns", []) 27 | kinesis_encryption_key_admin_arns = kwargs.pop("kinesis_encryption_key_admin_arns", []) 28 | add_s3_event_notification = kwargs.pop("add_s3_event_notification", False) 29 | 30 | super().__init__(scope, construct_id, **kwargs) 31 | 32 | # Define conditions like in the SAM template 33 | is_s3_backed = log_event_source in ["S3Bucket", "All"] 34 | is_kinesis_backed = log_event_source in ["KinesisDataStream", "All"] 35 | create_kinesis_agent_role = is_kinesis_backed and len(kinesis_user_arns) > 0 36 | create_staging_s3_bucket = is_s3_backed and not raw_log_s3_bucket_name 37 | 38 | # Create a single transformation Lambda execution role with all permissions 39 | # Always use the unified role ID regardless of deployment type 40 | transformation_lambda_role = iam.Role( 41 | self, "TransformationLambdaExecutionRole", 42 | role_name=f"{self.stack_name.lower()}-LambdaExecutionRole", 43 | assumed_by=iam.ServicePrincipal("lambda.amazonaws.com") 44 | ) 45 | 46 | # Always use the unified role logical ID 47 | (transformation_lambda_role.node.default_child).override_logical_id("TransformationLambdaExecutionRole") 48 | 49 | # Add policies for both S3 and Kinesis permissions, regardless of deployment type 50 | # This ensures the role has all necessary permissions for any combination 51 | 52 | # Common policies 53 | transformation_lambda_role.add_to_policy( 54 | iam.PolicyStatement( 55 | sid="LogGroupCreate", 56 | effect=iam.Effect.ALLOW, 57 | actions=["logs:CreateLogGroup"], 58 | resources=[f"arn:aws:logs:{self.region}:{self.account}:*"] 59 | ) 60 | ) 61 | 62 | transformation_lambda_role.add_to_policy( 63 | iam.PolicyStatement( 64 | sid="LogsWrite", 65 | effect=iam.Effect.ALLOW, 66 | actions=["logs:CreateLogStream", "logs:PutLogEvents"], 67 | resources=[f"arn:aws:logs:{self.region}:{self.account}:log-group:/aws/lambda/*:*"] 68 | ) 69 | ) 70 | 71 | transformation_lambda_role.add_to_policy( 72 | iam.PolicyStatement( 73 | sid="S3Write", 74 | effect=iam.Effect.ALLOW, 75 | actions=["s3:PutObject", "s3:PutObjectAcl"], 76 | resources=[f"arn:aws:s3:::{asl_bucket_location}/*"] 77 | ) 78 | ) 79 | 80 | transformation_lambda_role.add_to_policy( 81 | iam.PolicyStatement( 82 | sid="TracingWithXRay", 83 | effect=iam.Effect.ALLOW, 84 | actions=[ 85 | "xray:PutTraceSegments", 86 | "xray:PutTelemetryRecords", 87 | "xray:GetSamplingRules", 88 | "xray:GetSamplingTargets", 89 | "xray:GetSamplingStatisticSummaries" 90 | ], 91 | resources=["*"] 92 | ) 93 | ) 94 | 95 | # Add Kinesis permissions always 96 | transformation_lambda_role.add_to_policy( 97 | iam.PolicyStatement( 98 | sid="KinesisReadWrite", 99 | effect=iam.Effect.ALLOW, 100 | actions=[ 101 | "kinesis:PutRecord", 102 | "kinesis:PutRecords", 103 | "kinesis:DescribeStream", 104 | "kinesis:DescribeStreamSummary", 105 | "kinesis:GetRecords", 106 | "kinesis:GetShardIterator", 107 | "kinesis:ListShards", 108 | "kinesis:ListStreams", 109 | "kinesis:SubscribeToShard" 110 | ], 111 | resources=["*"] 112 | ) 113 | ) 114 | 115 | transformation_lambda_role.add_to_policy( 116 | iam.PolicyStatement( 117 | sid="KMSAccess", 118 | effect=iam.Effect.ALLOW, 119 | actions=["kms:Decrypt"], 120 | resources=["*"] 121 | ) 122 | ) 123 | 124 | # S3-specific resources 125 | if is_s3_backed: 126 | # Create SQS queue 127 | sqs_queue = sqs.Queue( 128 | self, "SqsQueue", 129 | visibility_timeout=Duration.seconds(1200) 130 | ) 131 | 132 | # Ensure SQS queue has the same logical ID as in the SAM template 133 | (sqs_queue.node.default_child).override_logical_id("SqsQueue") 134 | 135 | # Create or use existing S3 bucket 136 | if create_staging_s3_bucket: 137 | # Create a new bucket 138 | staging_log_bucket = s3.Bucket( 139 | self, "StagingLogBucket", 140 | bucket_name=f"{self.stack_name.lower()}-staging-log-bucket", 141 | removal_policy=RemovalPolicy.RETAIN, 142 | enforce_ssl=True 143 | ) 144 | # Ensure S3 bucket has the same logical ID as in the SAM template 145 | (staging_log_bucket.node.default_child).override_logical_id("StagingLogBucket") 146 | 147 | log_bucket = staging_log_bucket 148 | 149 | # Add bucket policy for AWS Logs Delivery 150 | bucket_policy = staging_log_bucket.add_to_resource_policy( 151 | iam.PolicyStatement( 152 | sid="AWSLogDeliveryWrite", 153 | effect=iam.Effect.ALLOW, 154 | actions=["s3:PutObject"], 155 | resources=[staging_log_bucket.arn_for_objects("*")], 156 | principals=[iam.ServicePrincipal("delivery.logs.amazonaws.com")], 157 | conditions={"StringEquals": {"s3:x-amz-acl": "bucket-owner-full-control"}} 158 | ) 159 | ) 160 | 161 | staging_log_bucket.add_to_resource_policy( 162 | iam.PolicyStatement( 163 | sid="AWSLogDeliveryAclCheck", 164 | effect=iam.Effect.ALLOW, 165 | actions=["s3:GetBucketAcl"], 166 | resources=[staging_log_bucket.bucket_arn], 167 | principals=[iam.ServicePrincipal("delivery.logs.amazonaws.com")] 168 | ) 169 | ) 170 | 171 | # Get the CFN representation of the bucket policy 172 | bucket_policy_cfn = staging_log_bucket.policy.node.default_child 173 | # Override logical ID to match SAM template 174 | bucket_policy_cfn.override_logical_id("StagingLogBucketPolicy") 175 | 176 | # Add S3 notification for SQS 177 | staging_log_bucket.add_event_notification( 178 | s3.EventType.OBJECT_CREATED_PUT, 179 | s3n.SqsDestination(sqs_queue), 180 | s3.NotificationKeyFilter(suffix=".log.gz") 181 | ) 182 | else: 183 | # Use existing bucket 184 | log_bucket = s3.Bucket.from_bucket_name( 185 | self, "ExistingRawLogBucket", 186 | bucket_name=raw_log_s3_bucket_name 187 | ) 188 | if add_s3_event_notification: 189 | # Add S3 notification for SQS 190 | log_bucket.add_event_notification( 191 | s3.EventType.OBJECT_CREATED_PUT, 192 | s3n.SqsDestination(sqs_queue), 193 | s3.NotificationKeyFilter(suffix=".log.gz") 194 | ) 195 | 196 | # Add S3-specific policies to the Lambda role 197 | transformation_lambda_role.add_to_policy( 198 | iam.PolicyStatement( 199 | sid="S3Read", 200 | effect=iam.Effect.ALLOW, 201 | actions=["s3:GetObject*"], 202 | resources=[f"{log_bucket.bucket_arn}/*"] 203 | ) 204 | ) 205 | 206 | transformation_lambda_role.add_to_policy( 207 | iam.PolicyStatement( 208 | sid="SQSTrigger", 209 | effect=iam.Effect.ALLOW, 210 | actions=[ 211 | "sqs:DeleteMessage", 212 | "sqs:GetQueueAttributes", 213 | "sqs:ReceiveMessage" 214 | ], 215 | resources=[sqs_queue.queue_arn] 216 | ) 217 | ) 218 | 219 | # Kinesis-specific resources 220 | if is_kinesis_backed: 221 | # Create Kinesis Agent IAM role if needed 222 | if create_kinesis_agent_role: 223 | kinesis_agent_role = iam.Role( 224 | self, "KinesisAgentIAMRole", 225 | role_name=f"{self.stack_name.lower()}-KinesisAgentRole", 226 | assumed_by=iam.CompositePrincipal( 227 | *[iam.ArnPrincipal(arn) for arn in kinesis_user_arns] 228 | ) 229 | ) 230 | # Use the same logical ID as in the SAM template 231 | (kinesis_agent_role.node.default_child).override_logical_id("KinesisAgentIAMRole") 232 | 233 | kinesis_agent_role.add_to_policy( 234 | iam.PolicyStatement( 235 | sid="KinesisWrite", 236 | effect=iam.Effect.ALLOW, 237 | actions=["kinesis:PutRecord", "kinesis:PutRecords"], 238 | resources=["*"] 239 | ) 240 | ) 241 | 242 | kinesis_agent_role.add_to_policy( 243 | iam.PolicyStatement( 244 | sid="KMSAccess", 245 | effect=iam.Effect.ALLOW, 246 | actions=["kms:GenerateDataKey"], 247 | resources=["*"] 248 | ) 249 | ) 250 | 251 | # Determine which key to use for Kinesis Stream 252 | # If no admin ARNs provided, use AWS managed key (like in SAM template) 253 | use_aws_managed_key = len(kinesis_encryption_key_admin_arns) == 0 254 | 255 | if use_aws_managed_key: 256 | # Create Kinesis stream with AWS managed key 257 | log_collection_stream = kinesis.Stream( 258 | self, "LogCollectionStream", 259 | shard_count=1, 260 | encryption=kinesis.StreamEncryption.MANAGED 261 | ) 262 | # Use the same logical ID as in the SAM template 263 | (log_collection_stream.node.default_child).override_logical_id("LogCollectionStream") 264 | else: 265 | # Create initial policy statements 266 | policy_statements = [ 267 | iam.PolicyStatement( 268 | sid="Allow IAM access delegation", 269 | effect=iam.Effect.ALLOW, 270 | principals=[iam.ArnPrincipal(f"arn:aws:iam::{self.account}:root")], 271 | actions=["kms:*"], 272 | resources=["*"] 273 | ), 274 | iam.PolicyStatement( 275 | sid="Allow access to key administrators", 276 | effect=iam.Effect.ALLOW, 277 | principals=[iam.ArnPrincipal(arn) for arn in kinesis_encryption_key_admin_arns], 278 | actions=[ 279 | "kms:Create*", "kms:Describe*", "kms:Enable*", "kms:List*", 280 | "kms:Put*", "kms:Update*", "kms:Revoke*", "kms:Disable*", 281 | "kms:Get*", "kms:Delete*", "kms:TagResource", "kms:UntagResource", 282 | "kms:ScheduleKeyDeletion", "kms:CancelKeyDeletion" 283 | ], 284 | resources=["*"] 285 | ), 286 | ] 287 | 288 | # Create the KMS key to represent the existing resource in the stack 289 | kinesis_stream_key = kms.Key( 290 | self, "KinesisStreamKey", 291 | enable_key_rotation=True, 292 | policy=iam.PolicyDocument(statements=policy_statements) 293 | ) 294 | 295 | # Use the same logical ID as in the SAM template 296 | (kinesis_stream_key.node.default_child).override_logical_id("KinesisStreamKey") 297 | 298 | kinesis_stream_key.grant_decrypt(transformation_lambda_role) 299 | 300 | # Add Kinesis agent access to key policy if role was created 301 | if create_kinesis_agent_role: 302 | kinesis_stream_key.add_to_resource_policy( 303 | iam.PolicyStatement( 304 | sid="Allow access to generate data key", 305 | effect=iam.Effect.ALLOW, 306 | principals=[iam.ArnPrincipal(f"arn:aws:iam::{self.account}:role/{self.stack_name.lower()}-KinesisAgentRole")], 307 | actions=["kms:GenerateDataKey"], 308 | resources=["*"] 309 | ) 310 | ) 311 | 312 | # Create encrypted Kinesis stream with custom key 313 | log_collection_stream = kinesis.Stream( 314 | self, "CMKEncryptedLogCollectionStream", 315 | shard_count=1, 316 | encryption=kinesis.StreamEncryption.KMS, 317 | encryption_key=kinesis_stream_key 318 | ) 319 | # Use the same logical ID as in the SAM template 320 | (log_collection_stream.node.default_child).override_logical_id("CMKEncryptedLogCollectionStream") 321 | 322 | 323 | # Create the unified transformation Lambda function with a unified logical ID 324 | lambda_function = lambda_.Function( 325 | self, "TransformationLambdaFunction", 326 | runtime=lambda_.Runtime.PYTHON_3_10, 327 | handler="app.lambda_handler", 328 | code=lambda_.Code.from_asset("../transformation_function/"), 329 | role=transformation_lambda_role, 330 | tracing=lambda_.Tracing.ACTIVE, 331 | reserved_concurrent_executions=10, 332 | timeout=Duration.seconds(10), 333 | environment={ 334 | "SEC_LAKE_BUCKET": asl_bucket_location, 335 | "DEBUG": "false" 336 | }, 337 | layers=[ 338 | lambda_.LayerVersion.from_layer_version_arn( 339 | self, "AWSSDKPandasLayer", 340 | f"arn:aws:lambda:{self.region}:336392948345:layer:AWSSDKPandas-Python310:5" 341 | ) 342 | ] 343 | ) 344 | 345 | # Always use "TransformationLambdaFunction" as the logical ID for the Lambda function 346 | (lambda_function.node.default_child).override_logical_id("TransformationLambdaFunction") 347 | 348 | # Add event sources to Lambda based on selected approach 349 | if is_s3_backed: 350 | lambda_event_source = lambda_event_sources.SqsEventSource( 351 | sqs_queue, 352 | batch_size=10 353 | ) 354 | lambda_function.add_event_source(lambda_event_source) 355 | 356 | if is_kinesis_backed: 357 | lambda_event_source = lambda_event_sources.KinesisEventSource( 358 | log_collection_stream, 359 | batch_size=100, 360 | starting_position=lambda_.StartingPosition.LATEST 361 | ) 362 | lambda_function.add_event_source(lambda_event_source) 363 | 364 | # Define outputs - use a unified output name for the Lambda function 365 | CfnOutput( 366 | self, "TransformationLambdaFunctionARN", 367 | description="OCSF Transformation Lambda Function ARN", 368 | value=lambda_function.function_arn 369 | ) 370 | 371 | # Additional outputs for created resources 372 | if is_s3_backed and create_staging_s3_bucket: 373 | CfnOutput( 374 | self, "StagingLogBucketARN", 375 | description="Name of the bucket for temporary log storage", 376 | value=staging_log_bucket.bucket_arn 377 | ) 378 | 379 | if is_kinesis_backed: 380 | if use_aws_managed_key: 381 | CfnOutput( 382 | self, "LogCollectionStreamName", 383 | description="Name of the log collection Kinesis Stream", 384 | value=log_collection_stream.stream_name 385 | ) 386 | else: 387 | CfnOutput( 388 | self, "CMKEncryptedLogCollectionStreamName", 389 | description="Name of the log collection Kinesis Stream", 390 | value=log_collection_stream.stream_name 391 | ) 392 | 393 | if create_kinesis_agent_role: 394 | CfnOutput( 395 | self, "KinesisAgentIAMRoleARN", 396 | description="ARN of the IAM role created for Kinesis agent to assume for log streaming", 397 | value=kinesis_agent_role.role_arn 398 | ) -------------------------------------------------------------------------------- /cdk/requirements.txt: -------------------------------------------------------------------------------- 1 | aws-cdk-lib>=2.0.0 2 | constructs>=10.0.0 3 | pytest>=7.0.0 4 | pytest-mock>=3.10.0 5 | pandas>=1.5.0 6 | awswrangler>=3.0.0 7 | boto3>=1.26.0 8 | -------------------------------------------------------------------------------- /cdk/setup.py: -------------------------------------------------------------------------------- 1 | import setuptools 2 | 3 | with open("README.md") as fp: 4 | long_description = fp.read() 5 | 6 | setuptools.setup( 7 | name="ocsf_transformation", 8 | version="0.1.0", 9 | 10 | description="CDK app for OCSF transformation", 11 | long_description=long_description, 12 | long_description_content_type="text/markdown", 13 | 14 | author="Amazon", 15 | 16 | package_dir={"": "ocsf_transformation"}, 17 | packages=setuptools.find_packages(where="ocsf_transformation"), 18 | 19 | install_requires=[ 20 | "aws-cdk-lib>=2.0.0", 21 | "constructs>=10.0.0", 22 | ], 23 | 24 | python_requires=">=3.8", 25 | ) 26 | -------------------------------------------------------------------------------- /images/asl_custom_source_create.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-security-lake-transformation-library/af68fe6fb3430fb1aad2ff7b68109c62de384be6/images/asl_custom_source_create.png -------------------------------------------------------------------------------- /images/custom-sources/.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-security-lake-transformation-library/af68fe6fb3430fb1aad2ff7b68109c62de384be6/images/custom-sources/.DS_Store -------------------------------------------------------------------------------- /images/custom-sources/alb/alb_arch.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-security-lake-transformation-library/af68fe6fb3430fb1aad2ff7b68109c62de384be6/images/custom-sources/alb/alb_arch.png -------------------------------------------------------------------------------- /images/custom-sources/aws-network-firewall/S3_SQS.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-security-lake-transformation-library/af68fe6fb3430fb1aad2ff7b68109c62de384be6/images/custom-sources/aws-network-firewall/S3_SQS.jpg -------------------------------------------------------------------------------- /images/custom-sources/aws-network-firewall/S3_create_event_notification.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-security-lake-transformation-library/af68fe6fb3430fb1aad2ff7b68109c62de384be6/images/custom-sources/aws-network-firewall/S3_create_event_notification.jpg -------------------------------------------------------------------------------- /images/custom-sources/aws-network-firewall/S3_obj_actions.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-security-lake-transformation-library/af68fe6fb3430fb1aad2ff7b68109c62de384be6/images/custom-sources/aws-network-firewall/S3_obj_actions.jpg -------------------------------------------------------------------------------- /images/custom-sources/aws-network-firewall/anf_arch.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-security-lake-transformation-library/af68fe6fb3430fb1aad2ff7b68109c62de384be6/images/custom-sources/aws-network-firewall/anf_arch.png -------------------------------------------------------------------------------- /images/custom-sources/aws-network-firewall/anf_update_glue_table.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-security-lake-transformation-library/af68fe6fb3430fb1aad2ff7b68109c62de384be6/images/custom-sources/aws-network-firewall/anf_update_glue_table.jpg -------------------------------------------------------------------------------- /images/custom-sources/aws-network-firewall/deployment_overview.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-security-lake-transformation-library/af68fe6fb3430fb1aad2ff7b68109c62de384be6/images/custom-sources/aws-network-firewall/deployment_overview.png -------------------------------------------------------------------------------- /images/custom-sources/windows-sysmon/deployment_overview.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-security-lake-transformation-library/af68fe6fb3430fb1aad2ff7b68109c62de384be6/images/custom-sources/windows-sysmon/deployment_overview.png -------------------------------------------------------------------------------- /images/custom-sources/windows-sysmon/edit_crawler.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-security-lake-transformation-library/af68fe6fb3430fb1aad2ff7b68109c62de384be6/images/custom-sources/windows-sysmon/edit_crawler.png -------------------------------------------------------------------------------- /images/custom-sources/windows-sysmon/edit_crawler_data_source.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-security-lake-transformation-library/af68fe6fb3430fb1aad2ff7b68109c62de384be6/images/custom-sources/windows-sysmon/edit_crawler_data_source.png -------------------------------------------------------------------------------- /images/custom-sources/windows-sysmon/edit_crawler_output_scheduling.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-security-lake-transformation-library/af68fe6fb3430fb1aad2ff7b68109c62de384be6/images/custom-sources/windows-sysmon/edit_crawler_output_scheduling.png -------------------------------------------------------------------------------- /images/custom-sources/windows-sysmon/sysmon_arch.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-security-lake-transformation-library/af68fe6fb3430fb1aad2ff7b68109c62de384be6/images/custom-sources/windows-sysmon/sysmon_arch.png -------------------------------------------------------------------------------- /images/custom_sources_config.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-security-lake-transformation-library/af68fe6fb3430fb1aad2ff7b68109c62de384be6/images/custom_sources_config.png -------------------------------------------------------------------------------- /images/ocsf_fundamentals.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-security-lake-transformation-library/af68fe6fb3430fb1aad2ff7b68109c62de384be6/images/ocsf_fundamentals.png -------------------------------------------------------------------------------- /images/ocsf_v1_1_0.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-security-lake-transformation-library/af68fe6fb3430fb1aad2ff7b68109c62de384be6/images/ocsf_v1_1_0.jpg -------------------------------------------------------------------------------- /images/solution_architecture.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-security-lake-transformation-library/af68fe6fb3430fb1aad2ff7b68109c62de384be6/images/solution_architecture.png -------------------------------------------------------------------------------- /images/sources_config.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-security-lake-transformation-library/af68fe6fb3430fb1aad2ff7b68109c62de384be6/images/sources_config.png -------------------------------------------------------------------------------- /mapping_docs/application-load-balancer-mapping.md: -------------------------------------------------------------------------------- 1 | # Application Load Balancer 2 | 3 | **OCSF Classes**: HTTP_ACTIVITY 4 | 5 | ## Mapping 6 | 7 | **Sample HTTP Activity event mapping for http/https/h2 type logs** 8 | 9 | 1. `HTTPS` type log event. 10 | 11 | ``` 12 | 13 | https 2018-07-02T22:23:00.186641Z app/my-loadbalancer/50dc6c495c0c9188 192.168.131.39:2817 10.0.0.1:80 14 | 0.086 0.048 0.037 200 200 0 57 "GET https://www.example.com:443/ HTTP/1.1" "curl/7.46.0" 15 | ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 16 | arn:aws:elasticloadbalancing:us-east-2:XXXXXXXXXXXX:targetgroup/my-targets/73e2d6bc24d8a067 17 | "Root=1-58337281-1d84f3d73c47ec4e58577259" "www.example.com" 18 | "arn:aws:acm:us-east-2:XXXXXXXXXXXX:certificate/XXXXXXXX-1234-1234-1234-XXXXXXXXXXXX" 1 19 | 2018-07-02T22:22:48.364000Z "authenticate,forward" "-" "-" "10.0.0.1:80" "200" "-" "-" TID_123456 20 | ``` 21 | 22 | 2. Attribute mapping for HTTP Activity class 23 | 24 | |OCSF|Raw| 25 | |-|-| 26 | | app_name | `` | 27 | | cloud.provider | AWS | 28 | | metadata.profiles | [cloud, loadbalancing] | 29 | | metadata.product.name | AWS ELB | 30 | | metadata.product.feature.name | Application Load Balancer | 31 | | metadata.product.vendor_name | AWS | 32 | | metadata.product.uid | `` | 33 | | severity | Informational | 34 | | severity_id | 1 | 35 | | category_uid | 4 | 36 | | category_name | Network Activity | 37 | | class_uid | 4002 | 38 | | class_name | HTTP Activity | 39 | | activity_id | `enum` evaluated on `` | 40 | | activity_name | `` | 41 | | type_uid | `enum` evaluated on `` calculated as `class_uid * 100 + activity_id` | 42 | | http_request.http_method | `` | 43 | | http_request.user_agent | ``| 44 | | http_request.url.hostname | `` | 45 | | http_request.url.url_string | `` | 46 | | http_request.url.scheme | `` | 47 | | http_response.code | `` | 48 | | connection_info | {direction_id: 1, protocol_ver_id: 4, protocol_name: "tcp", protocol_num: 6} | 49 | | dst_endpoint | {hostname: ``, ip: ``, port: ``, type: "Virtual", type_id: 6} | 50 | | src_endpoint | {ip: ``, port: ``} | 51 | | duration | `` | 52 | | time | `