├── detect-labels ├── .npmignore ├── .eslintignore ├── .prettierrc.js ├── jest.config.ts ├── tsconfig.json ├── .eslintrc.js ├── package.json ├── app.ts └── tests │ └── unit │ └── test-handler.test.ts ├── extract-text ├── .npmignore ├── .eslintignore ├── .prettierrc.js ├── jest.config.ts ├── tsconfig.json ├── .eslintrc.js ├── package.json ├── app.ts └── tests │ └── unit │ └── test-handler.test.ts ├── samples └── skateboard.jpg ├── images ├── aggregated-log-group.png └── s3-event-lambda-detection.png ├── CODE_OF_CONDUCT.md ├── events └── event.json ├── LICENSE ├── samconfig.toml ├── CONTRIBUTING.md ├── .gitignore ├── template.yaml └── README.md /detect-labels/.npmignore: -------------------------------------------------------------------------------- 1 | tests/* 2 | -------------------------------------------------------------------------------- /extract-text/.npmignore: -------------------------------------------------------------------------------- 1 | tests/* 2 | -------------------------------------------------------------------------------- /detect-labels/.eslintignore: -------------------------------------------------------------------------------- 1 | node_modules 2 | .aws-sam -------------------------------------------------------------------------------- /extract-text/.eslintignore: -------------------------------------------------------------------------------- 1 | node_modules 2 | .aws-sam -------------------------------------------------------------------------------- /samples/skateboard.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/advanced-logging-controls-lambda/main/samples/skateboard.jpg -------------------------------------------------------------------------------- /images/aggregated-log-group.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/advanced-logging-controls-lambda/main/images/aggregated-log-group.png -------------------------------------------------------------------------------- /images/s3-event-lambda-detection.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/advanced-logging-controls-lambda/main/images/s3-event-lambda-detection.png -------------------------------------------------------------------------------- /detect-labels/.prettierrc.js: -------------------------------------------------------------------------------- 1 | module.exports = { 2 | semi: true, 3 | trailingComma: "all", 4 | singleQuote: true, 5 | printWidth: 120, 6 | tabWidth: 4 7 | }; -------------------------------------------------------------------------------- /extract-text/.prettierrc.js: -------------------------------------------------------------------------------- 1 | module.exports = { 2 | semi: true, 3 | trailingComma: "all", 4 | singleQuote: true, 5 | printWidth: 120, 6 | tabWidth: 4 7 | }; -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | ## Code of Conduct 2 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 3 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 4 | opensource-codeofconduct@amazon.com with any additional questions or comments. 5 | -------------------------------------------------------------------------------- /detect-labels/jest.config.ts: -------------------------------------------------------------------------------- 1 | /* 2 | * For a detailed explanation regarding each configuration property and type check, visit: 3 | * https://jestjs.io/docs/configuration 4 | */ 5 | 6 | export default { 7 | transform: { 8 | '^.+\\.ts?$': 'ts-jest', 9 | }, 10 | clearMocks: true, 11 | collectCoverage: true, 12 | coverageDirectory: 'coverage', 13 | coverageProvider: 'v8', 14 | testMatch: ['**/tests/unit/*.test.ts'], 15 | }; 16 | -------------------------------------------------------------------------------- /extract-text/jest.config.ts: -------------------------------------------------------------------------------- 1 | /* 2 | * For a detailed explanation regarding each configuration property and type check, visit: 3 | * https://jestjs.io/docs/configuration 4 | */ 5 | 6 | export default { 7 | transform: { 8 | '^.+\\.ts?$': 'ts-jest', 9 | }, 10 | clearMocks: true, 11 | collectCoverage: true, 12 | coverageDirectory: 'coverage', 13 | coverageProvider: 'v8', 14 | testMatch: ['**/tests/unit/*.test.ts'], 15 | }; 16 | -------------------------------------------------------------------------------- /detect-labels/tsconfig.json: -------------------------------------------------------------------------------- 1 | { 2 | "compilerOptions": { 3 | "target": "es2020", 4 | "strict": true, 5 | "preserveConstEnums": true, 6 | "noEmit": true, 7 | "sourceMap": false, 8 | "module":"es2015", 9 | "moduleResolution":"node", 10 | "esModuleInterop": true, 11 | "skipLibCheck": true, 12 | "forceConsistentCasingInFileNames": true, 13 | }, 14 | "exclude": ["node_modules", "**/*.test.ts"] 15 | } -------------------------------------------------------------------------------- /extract-text/tsconfig.json: -------------------------------------------------------------------------------- 1 | { 2 | "compilerOptions": { 3 | "target": "es2020", 4 | "strict": true, 5 | "preserveConstEnums": true, 6 | "noEmit": true, 7 | "sourceMap": false, 8 | "module":"es2015", 9 | "moduleResolution":"node", 10 | "esModuleInterop": true, 11 | "skipLibCheck": true, 12 | "forceConsistentCasingInFileNames": true, 13 | }, 14 | "exclude": ["node_modules", "**/*.test.ts"] 15 | } -------------------------------------------------------------------------------- /extract-text/.eslintrc.js: -------------------------------------------------------------------------------- 1 | module.exports = { 2 | parser: "@typescript-eslint/parser", 3 | parserOptions: { 4 | ecmaVersion: 2020, // Allows for the parsing of modern ECMAScript features 5 | sourceType: "module" 6 | }, 7 | extends: [ 8 | "plugin:@typescript-eslint/recommended", // recommended rules from the @typescript-eslint/eslint-plugin 9 | "plugin:prettier/recommended" // Enables eslint-plugin-prettier and eslint-config-prettier. This will display prettier errors as ESLint errors. Make sure this is always the last configuration in the extends array. 10 | ], 11 | rules: { 12 | // Place to specify ESLint rules. Can be used to overwrite rules specified from the extended configs 13 | // e.g. "@typescript-eslint/explicit-function-return-type": "off", 14 | } 15 | }; -------------------------------------------------------------------------------- /detect-labels/.eslintrc.js: -------------------------------------------------------------------------------- 1 | module.exports = { 2 | parser: "@typescript-eslint/parser", 3 | parserOptions: { 4 | ecmaVersion: 2020, // Allows for the parsing of modern ECMAScript features 5 | sourceType: "module" 6 | }, 7 | extends: [ 8 | "plugin:@typescript-eslint/recommended", // recommended rules from the @typescript-eslint/eslint-plugin 9 | "plugin:prettier/recommended" // Enables eslint-plugin-prettier and eslint-config-prettier. This will display prettier errors as ESLint errors. Make sure this is always the last configuration in the extends array. 10 | ], 11 | rules: { 12 | // Place to specify ESLint rules. Can be used to overwrite rules specified from the extended configs 13 | // e.g. "@typescript-eslint/explicit-function-return-type": "off", 14 | } 15 | }; -------------------------------------------------------------------------------- /events/event.json: -------------------------------------------------------------------------------- 1 | { 2 | "version": "0", 3 | "id": "0c8c1c41-0ba7-23ac-5678-30c3ff95cb8b", 4 | "detail-type": "Object Created", 5 | "source": "aws.s3", 6 | "account": "123456789012", 7 | "time": "2023-10-19T20:43:21Z", 8 | "region": "eu-west-1", 9 | "resources": [ 10 | "arn:aws:s3:::sample-uploads-bucket" 11 | ], 12 | "detail": { 13 | "version": "0", 14 | "bucket": { 15 | "name": "sample-uploads-bucket" 16 | }, 17 | "object": { 18 | "key": "skateboard.jpg", 19 | "size": 1048576, 20 | "etag": "7d376a104fc650a066b3604d334c50a3", 21 | "sequencer": "00653194E62FD2BA22" 22 | }, 23 | "request-id": "MPSJKN3JACGJW67W", 24 | "requester": "123456789012", 25 | "source-ip-address": "255.240.197.232", 26 | "reason": "PutObject" 27 | } 28 | } -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT No Attribution 2 | 3 | Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy of 6 | this software and associated documentation files (the "Software"), to deal in 7 | the Software without restriction, including without limitation the rights to 8 | use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of 9 | the Software, and to permit persons to whom the Software is furnished to do so. 10 | 11 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 12 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS 13 | FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 14 | COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER 15 | IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN 16 | CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 17 | 18 | -------------------------------------------------------------------------------- /samconfig.toml: -------------------------------------------------------------------------------- 1 | # More information about the configuration file can be found here: 2 | # https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-config.html 3 | version = 0.1 4 | 5 | [default] 6 | [default.global.parameters] 7 | stack_name = "advanced-logging-controls-lambda" 8 | 9 | [default.build.parameters] 10 | cached = true 11 | parallel = true 12 | 13 | [default.validate.parameters] 14 | lint = true 15 | 16 | [default.deploy.parameters] 17 | capabilities = "CAPABILITY_IAM" 18 | confirm_changeset = true 19 | resolve_s3 = true 20 | s3_prefix = "advanced-logging-controls-lambda" 21 | region = "eu-west-1" 22 | parameter_overrides = "UploadsBucketName=\"ng-uploads-bucket2\"" 23 | image_repositories = [] 24 | 25 | [default.package.parameters] 26 | resolve_s3 = true 27 | 28 | [default.sync.parameters] 29 | watch = true 30 | 31 | [default.local_start_api.parameters] 32 | warm_containers = "EAGER" 33 | 34 | [default.local_start_lambda.parameters] 35 | warm_containers = "EAGER" 36 | -------------------------------------------------------------------------------- /extract-text/package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "extract_text", 3 | "version": "1.0.0", 4 | "description": "Extract Text function in NodeJS", 5 | "main": "app.js", 6 | "repository": "https://github.com/awslabs/aws-sam-cli/tree/develop/samcli/local/init/templates/cookiecutter-aws-sam-hello-nodejs", 7 | "author": "SAM CLI", 8 | "license": "MIT", 9 | "scripts": { 10 | "unit": "jest", 11 | "lint": "eslint '*.ts' --quiet --fix", 12 | "compile": "tsc", 13 | "test": "npm run compile && npm run unit" 14 | }, 15 | "dependencies": { 16 | "@aws-sdk/client-textract": "^3.431.0", 17 | "aws-lambda": "^1.0.7", 18 | "esbuild": "^0.14.14" 19 | }, 20 | "devDependencies": { 21 | "@types/aws-lambda": "^8.10.92", 22 | "@types/jest": "^29.2.0", 23 | "@types/node": "^18.11.4", 24 | "@typescript-eslint/eslint-plugin": "^5.10.2", 25 | "@typescript-eslint/parser": "^5.10.2", 26 | "eslint": "^8.8.0", 27 | "eslint-config-prettier": "^8.3.0", 28 | "eslint-plugin-prettier": "^4.0.0", 29 | "jest": "^29.2.1", 30 | "prettier": "^2.5.1", 31 | "ts-jest": "^29.0.5", 32 | "ts-node": "^10.9.1", 33 | "typescript": "^4.8.4" 34 | } 35 | } 36 | -------------------------------------------------------------------------------- /detect-labels/package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "detect_labels", 3 | "version": "1.0.0", 4 | "description": "Detect Labels function in NodeJS", 5 | "main": "app.js", 6 | "repository": "https://github.com/awslabs/aws-sam-cli/tree/develop/samcli/local/init/templates/cookiecutter-aws-sam-hello-nodejs", 7 | "author": "SAM CLI", 8 | "license": "MIT", 9 | "scripts": { 10 | "unit": "jest", 11 | "lint": "eslint '*.ts' --quiet --fix", 12 | "compile": "tsc", 13 | "test": "npm run compile && npm run unit" 14 | }, 15 | "dependencies": { 16 | "@aws-sdk/client-rekognition": "^3.421.0", 17 | "aws-lambda": "^1.0.7", 18 | "esbuild": "^0.14.14" 19 | }, 20 | "devDependencies": { 21 | "@types/aws-lambda": "^8.10.92", 22 | "@types/jest": "^29.2.0", 23 | "@types/node": "^18.11.4", 24 | "@typescript-eslint/eslint-plugin": "^5.10.2", 25 | "@typescript-eslint/parser": "^5.10.2", 26 | "eslint": "^8.8.0", 27 | "eslint-config-prettier": "^8.3.0", 28 | "eslint-plugin-prettier": "^4.0.0", 29 | "jest": "^29.2.1", 30 | "prettier": "^2.5.1", 31 | "ts-jest": "^29.0.5", 32 | "ts-node": "^10.9.1", 33 | "typescript": "^4.8.4" 34 | } 35 | } 36 | -------------------------------------------------------------------------------- /detect-labels/app.ts: -------------------------------------------------------------------------------- 1 | import { S3ObjectCreatedNotificationEvent } from 'aws-lambda'; 2 | import { RekognitionClient, DetectLabelsCommand } from '@aws-sdk/client-rekognition'; 3 | 4 | const rekognitionClient = new RekognitionClient(); 5 | 6 | export const lambdaHandler = async (event: S3ObjectCreatedNotificationEvent) => { 7 | // Only printing out the event structure for Debug purposes 8 | console.debug(event); 9 | 10 | const bucket = event.detail.bucket.name; 11 | const key = decodeURIComponent(event.detail.object.key.replace(/\+/g, ' ')); 12 | 13 | console.debug(`Going to detect lables in bucket ${bucket} and key ${key}`); 14 | 15 | const labels: string[] = await detectLabels(bucket, key); 16 | 17 | const response = { 18 | 'S3Bucket' : bucket, 19 | 'S3Key' : key, 20 | 'labels' : labels 21 | }; 22 | 23 | // Results logged in a structured JSON format 24 | console.info(response); 25 | }; 26 | 27 | /* 28 | * Detects labels for the given image. 29 | */ 30 | const detectLabels = async (bucket: string, key: string) : Promise => { 31 | const labels: string[] = []; 32 | const imageParams = { 33 | Image: { 34 | S3Object: { 35 | Bucket: bucket, 36 | Name: key 37 | } 38 | } 39 | }; 40 | 41 | const labelData = await rekognitionClient.send( 42 | new DetectLabelsCommand(imageParams) 43 | ); 44 | 45 | console.debug('DetectLabelsCommand:', labelData); 46 | 47 | labelData.Labels?.forEach(label => { 48 | labels.push(label.Name!); 49 | }); 50 | 51 | return labels 52 | }; -------------------------------------------------------------------------------- /extract-text/app.ts: -------------------------------------------------------------------------------- 1 | import { S3ObjectCreatedNotificationEvent } from 'aws-lambda'; 2 | import { TextractClient, DetectDocumentTextCommand } from '@aws-sdk/client-textract'; 3 | 4 | const textractClient = new TextractClient() 5 | 6 | export const lambdaHandler = async (event: S3ObjectCreatedNotificationEvent) => { 7 | // Only printing out the event structure for Debug purposes 8 | console.debug(event); 9 | 10 | const bucket = event.detail.bucket.name; 11 | const key = decodeURIComponent(event.detail.object.key.replace(/\+/g, ' ')); 12 | 13 | console.debug(`Going to detect texts in bucket ${bucket} and key ${key}`); 14 | 15 | const words: string[] = await extractText(bucket, key); 16 | 17 | const response = { 18 | 'S3Bucket' : bucket, 19 | 'S3Key' : key, 20 | 'words' : words 21 | }; 22 | 23 | // Results logged in a structured JSON format 24 | console.info(response); 25 | }; 26 | 27 | // use Textract to extract text from image 28 | const extractText = async (bucket: string, key: string) : Promise => { 29 | const params = { 30 | Document: { 31 | S3Object: { 32 | Bucket: bucket, 33 | Name: key 34 | } 35 | } 36 | }; 37 | 38 | // extract text from S3 image synchronouslly 39 | const textData = await textractClient.send(new DetectDocumentTextCommand(params)); 40 | 41 | console.debug('DetectDocumentTextCommand:', textData); 42 | 43 | let words = new Array(); 44 | 45 | // loop over Blocks array in textData 46 | textData?.Blocks?.forEach(block => { 47 | if (block.BlockType === 'WORD') { 48 | words.push(block.Text!); 49 | } 50 | }); 51 | 52 | return words; 53 | }; 54 | -------------------------------------------------------------------------------- /detect-labels/tests/unit/test-handler.test.ts: -------------------------------------------------------------------------------- 1 | import { S3Event } from 'aws-lambda'; 2 | import { lambdaHandler } from '../../app'; 3 | import { expect, describe, it } from '@jest/globals'; 4 | 5 | describe('Unit test for app handler', function () { 6 | it('verifies successful response', async () => { 7 | const event: S3Event = { 8 | "Records": [ 9 | { 10 | "eventVersion": "2.1", 11 | "eventSource": "aws:s3", 12 | "awsRegion": "eu-south-1", 13 | "eventTime": "2023-10-03T19:37:27.192Z", 14 | "eventName": "ObjectCreated:Put", 15 | "userIdentity": { 16 | "principalId": "AWS:AIDAINPONIXQXHT3IKHL2" 17 | }, 18 | "requestParameters": { 19 | "sourceIPAddress": "205.255.255.255" 20 | }, 21 | "responseElements": { 22 | "x-amz-request-id": "D82B88E5F771F645", 23 | "x-amz-id-2": "vlR7PnpV2Ce81l0PRw6jlUpck7Jo5ZsQjryTjKlc5aLWGVHPZLj5NeC6qMa0emYBDXOo6QBU0Wo=" 24 | }, 25 | "s3": { 26 | "s3SchemaVersion": "1.0", 27 | "configurationId": "828aa6fc-f7b5-4305-8584-487c791949c1", 28 | "bucket": { 29 | "name": "DOC-EXAMPLE-BUCKET", 30 | "ownerIdentity": { 31 | "principalId": "A3I5XTEXAMAI3E" 32 | }, 33 | "arn": "arn:aws:s3:::lambda-artifacts-deafc19498e3f2df" 34 | }, 35 | "object": { 36 | "key": "image-name.jpg", 37 | "size": 1305107, 38 | "eTag": "b21b84d653bb07b05b1e6b33684dc11b", 39 | "sequencer": "0C0F6F405D6ED209E1" 40 | } 41 | } 42 | } 43 | ] 44 | }; 45 | const result = await lambdaHandler(event); 46 | }); 47 | }); 48 | -------------------------------------------------------------------------------- /extract-text/tests/unit/test-handler.test.ts: -------------------------------------------------------------------------------- 1 | import { S3Event } from 'aws-lambda'; 2 | import { lambdaHandler } from '../../app'; 3 | import { expect, describe, it } from '@jest/globals'; 4 | 5 | describe('Unit test for app handler', function () { 6 | it('verifies successful response', async () => { 7 | const event: S3Event = { 8 | "Records": [ 9 | { 10 | "eventVersion": "2.1", 11 | "eventSource": "aws:s3", 12 | "awsRegion": "eu-south-1", 13 | "eventTime": "2023-10-03T19:37:27.192Z", 14 | "eventName": "ObjectCreated:Put", 15 | "userIdentity": { 16 | "principalId": "AWS:AIDAINPONIXQXHT3IKHL2" 17 | }, 18 | "requestParameters": { 19 | "sourceIPAddress": "205.255.255.255" 20 | }, 21 | "responseElements": { 22 | "x-amz-request-id": "D82B88E5F771F645", 23 | "x-amz-id-2": "vlR7PnpV2Ce81l0PRw6jlUpck7Jo5ZsQjryTjKlc5aLWGVHPZLj5NeC6qMa0emYBDXOo6QBU0Wo=" 24 | }, 25 | "s3": { 26 | "s3SchemaVersion": "1.0", 27 | "configurationId": "828aa6fc-f7b5-4305-8584-487c791949c1", 28 | "bucket": { 29 | "name": "DOC-EXAMPLE-BUCKET", 30 | "ownerIdentity": { 31 | "principalId": "A3I5XTEXAMAI3E" 32 | }, 33 | "arn": "arn:aws:s3:::lambda-artifacts-deafc19498e3f2df" 34 | }, 35 | "object": { 36 | "key": "image-name.jpg", 37 | "size": 1305107, 38 | "eTag": "b21b84d653bb07b05b1e6b33684dc11b", 39 | "sequencer": "0C0F6F405D6ED209E1" 40 | } 41 | } 42 | } 43 | ] 44 | }; 45 | const result = await lambdaHandler(event); 46 | }); 47 | }); 48 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing Guidelines 2 | 3 | Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional 4 | documentation, we greatly value feedback and contributions from our community. 5 | 6 | Please read through this document before submitting any issues or pull requests to ensure we have all the necessary 7 | information to effectively respond to your bug report or contribution. 8 | 9 | 10 | ## Reporting Bugs/Feature Requests 11 | 12 | We welcome you to use the GitHub issue tracker to report bugs or suggest features. 13 | 14 | When filing an issue, please check existing open, or recently closed, issues to make sure somebody else hasn't already 15 | reported the issue. Please try to include as much information as you can. Details like these are incredibly useful: 16 | 17 | * A reproducible test case or series of steps 18 | * The version of our code being used 19 | * Any modifications you've made relevant to the bug 20 | * Anything unusual about your environment or deployment 21 | 22 | 23 | ## Contributing via Pull Requests 24 | Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that: 25 | 26 | 1. You are working against the latest source on the *main* branch. 27 | 2. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already. 28 | 3. You open an issue to discuss any significant work - we would hate for your time to be wasted. 29 | 30 | To send us a pull request, please: 31 | 32 | 1. Fork the repository. 33 | 2. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change. 34 | 3. Ensure local tests pass. 35 | 4. Commit to your fork using clear commit messages. 36 | 5. Send us a pull request, answering any default questions in the pull request interface. 37 | 6. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation. 38 | 39 | GitHub provides additional document on [forking a repository](https://help.github.com/articles/fork-a-repo/) and 40 | [creating a pull request](https://help.github.com/articles/creating-a-pull-request/). 41 | 42 | 43 | ## Finding contributions to work on 44 | Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels (enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any 'help wanted' issues is a great place to start. 45 | 46 | 47 | ## Code of Conduct 48 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 49 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 50 | opensource-codeofconduct@amazon.com with any additional questions or comments. 51 | 52 | 53 | ## Security issue notifications 54 | If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github issue. 55 | 56 | 57 | ## Licensing 58 | 59 | See the [LICENSE](LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution. 60 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | 2 | # Created by https://www.toptal.com/developers/gitignore/api/osx,node,linux,windows,sam 3 | # Edit at https://www.toptal.com/developers/gitignore?templates=osx,node,linux,windows,sam 4 | 5 | ### Linux ### 6 | *~ 7 | 8 | # temporary files which can be created if a process still has a handle open of a deleted file 9 | .fuse_hidden* 10 | 11 | # KDE directory preferences 12 | .directory 13 | 14 | # Linux trash folder which might appear on any partition or disk 15 | .Trash-* 16 | 17 | # .nfs files are created when an open file is removed but is still being accessed 18 | .nfs* 19 | 20 | ### Node ### 21 | # Logs 22 | logs 23 | *.log 24 | npm-debug.log* 25 | yarn-debug.log* 26 | yarn-error.log* 27 | lerna-debug.log* 28 | 29 | # Diagnostic reports (https://nodejs.org/api/report.html) 30 | report.[0-9]*.[0-9]*.[0-9]*.[0-9]*.json 31 | 32 | # Runtime data 33 | pids 34 | *.pid 35 | *.seed 36 | *.pid.lock 37 | 38 | # Directory for instrumented libs generated by jscoverage/JSCover 39 | lib-cov 40 | 41 | # Coverage directory used by tools like istanbul 42 | coverage 43 | *.lcov 44 | 45 | # nyc test coverage 46 | .nyc_output 47 | 48 | # Grunt intermediate storage (https://gruntjs.com/creating-plugins#storing-task-files) 49 | .grunt 50 | 51 | # Bower dependency directory (https://bower.io/) 52 | bower_components 53 | 54 | # node-waf configuration 55 | .lock-wscript 56 | 57 | # Compiled binary addons (https://nodejs.org/api/addons.html) 58 | build/Release 59 | 60 | # Dependency directories 61 | node_modules/ 62 | jspm_packages/ 63 | 64 | # TypeScript v1 declaration files 65 | typings/ 66 | 67 | # TypeScript cache 68 | *.tsbuildinfo 69 | 70 | # Optional npm cache directory 71 | .npm 72 | 73 | # Optional eslint cache 74 | .eslintcache 75 | 76 | # Optional stylelint cache 77 | .stylelintcache 78 | 79 | # Microbundle cache 80 | .rpt2_cache/ 81 | .rts2_cache_cjs/ 82 | .rts2_cache_es/ 83 | .rts2_cache_umd/ 84 | 85 | # Optional REPL history 86 | .node_repl_history 87 | 88 | # Output of 'npm pack' 89 | *.tgz 90 | 91 | # Yarn Integrity file 92 | .yarn-integrity 93 | 94 | # dotenv environment variables file 95 | .env 96 | .env.test 97 | .env*.local 98 | 99 | # parcel-bundler cache (https://parceljs.org/) 100 | .cache 101 | .parcel-cache 102 | 103 | # Next.js build output 104 | .next 105 | 106 | # Nuxt.js build / generate output 107 | .nuxt 108 | dist 109 | 110 | # Storybook build outputs 111 | .out 112 | .storybook-out 113 | storybook-static 114 | 115 | # rollup.js default build output 116 | dist/ 117 | 118 | # Gatsby files 119 | .cache/ 120 | # Comment in the public line in if your project uses Gatsby and not Next.js 121 | # https://nextjs.org/blog/next-9-1#public-directory-support 122 | # public 123 | 124 | # vuepress build output 125 | .vuepress/dist 126 | 127 | # Serverless directories 128 | .serverless/ 129 | 130 | # FuseBox cache 131 | .fusebox/ 132 | 133 | # DynamoDB Local files 134 | .dynamodb/ 135 | 136 | # TernJS port file 137 | .tern-port 138 | 139 | # Stores VSCode versions used for testing VSCode extensions 140 | .vscode-test 141 | 142 | # Temporary folders 143 | tmp/ 144 | temp/ 145 | 146 | ### OSX ### 147 | # General 148 | .DS_Store 149 | .AppleDouble 150 | .LSOverride 151 | 152 | # Icon must end with two \r 153 | Icon 154 | 155 | 156 | # Thumbnails 157 | ._* 158 | 159 | # Files that might appear in the root of a volume 160 | .DocumentRevisions-V100 161 | .fseventsd 162 | .Spotlight-V100 163 | .TemporaryItems 164 | .Trashes 165 | .VolumeIcon.icns 166 | .com.apple.timemachine.donotpresent 167 | 168 | # Directories potentially created on remote AFP share 169 | .AppleDB 170 | .AppleDesktop 171 | Network Trash Folder 172 | Temporary Items 173 | .apdisk 174 | 175 | ### SAM ### 176 | # Ignore build directories for the AWS Serverless Application Model (SAM) 177 | # Info: https://aws.amazon.com/serverless/sam/ 178 | # Docs: https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-reference.html 179 | 180 | **/.aws-sam 181 | samconfig.toml 182 | 183 | ### Windows ### 184 | # Windows thumbnail cache files 185 | Thumbs.db 186 | Thumbs.db:encryptable 187 | ehthumbs.db 188 | ehthumbs_vista.db 189 | 190 | # Dump file 191 | *.stackdump 192 | 193 | # Folder config file 194 | [Dd]esktop.ini 195 | 196 | # Recycle Bin used on file shares 197 | $RECYCLE.BIN/ 198 | 199 | # Windows Installer files 200 | *.cab 201 | *.msi 202 | *.msix 203 | *.msm 204 | *.msp 205 | 206 | # Windows shortcuts 207 | *.lnk 208 | 209 | # End of https://www.toptal.com/developers/gitignore/api/osx,node,linux,windows,sam 210 | 211 | # Drawio raw files 212 | *.drawio -------------------------------------------------------------------------------- /template.yaml: -------------------------------------------------------------------------------- 1 | AWSTemplateFormatVersion: '2010-09-09' 2 | Transform: AWS::Serverless-2016-10-31 3 | Description: > 4 | advanced-logging-controls-lambda 5 | 6 | SAM Template demonstrating Lambda function custom log group configuration 7 | 8 | Parameters: 9 | UploadsBucketName: 10 | Type: String 11 | 12 | Globals: 13 | Function: 14 | Timeout: 20 15 | 16 | Resources: 17 | # Add S3 bucket for uploads 18 | UploadsBucket: 19 | Type: AWS::S3::Bucket 20 | Properties: 21 | BucketName: !Ref UploadsBucketName 22 | NotificationConfiguration: 23 | EventBridgeConfiguration: 24 | EventBridgeEnabled: true 25 | 26 | # Configuring a custom log group to store logs from mutliple sources (optionally), with a customized retention period 27 | CloudWatchLogGroup: 28 | Type: AWS::Logs::LogGroup 29 | Properties: 30 | LogGroupName: AggregatedLabelsLogGroup 31 | RetentionInDays: 30 32 | 33 | # Detect Labels Lambda function written in NodeJS 18.x with TypeScript 34 | DetectLabelsFunction: 35 | Type: AWS::Serverless::Function 36 | Properties: 37 | CodeUri: detect-labels/ 38 | Handler: app.lambdaHandler 39 | Runtime: nodejs18.x 40 | Architectures: 41 | - x86_64 42 | Events: 43 | ObjectCreatedEvent: 44 | Type: EventBridgeRule 45 | Properties: 46 | Pattern: 47 | source: 48 | - "aws.s3" 49 | detail-type: 50 | - "Object Created" 51 | detail: 52 | bucket: 53 | name: 54 | - !Ref UploadsBucket 55 | object: 56 | key: 57 | - wildcard: "*.png" 58 | - wildcard: "*.jpg" 59 | # Set Lambda execution role with IAM policies to access the S3 bucket and execute Rekognition 60 | Policies: 61 | - S3ReadPolicy: 62 | BucketName: !Ref UploadsBucketName 63 | - RekognitionDetectOnlyPolicy: {} 64 | - Version: 2012-10-17 65 | Statement: 66 | - Sid: CloudWatchLogGroup 67 | Action: 68 | - logs:CreateLogStream 69 | - logs:PutLogEvents 70 | Resource: !GetAtt CloudWatchLogGroup.Arn 71 | Effect: Allow 72 | # Advanced logging controls for AWS Lambda 73 | LoggingConfig: 74 | LogFormat: JSON # Application logs format, defaults to JSON (TEXT is optional) 75 | ApplicationLogLevel: DEBUG # Application log level, defaults to INFO 76 | SystemLogLevel: INFO # System log level, defaults to INFO 77 | LogGroup: !Ref CloudWatchLogGroup # Customized log group to emit logs to 78 | 79 | Metadata: # Manage esbuild properties 80 | BuildMethod: esbuild 81 | BuildProperties: 82 | Minify: true 83 | Target: es2020 84 | Sourcemap: true 85 | EntryPoints: 86 | - app.ts 87 | 88 | # Extract Text Lambda function written in NodeJS 18.x with TypeScript 89 | ExtractTextFunction: 90 | Type: AWS::Serverless::Function 91 | Properties: 92 | CodeUri: extract-text/ 93 | Handler: app.lambdaHandler 94 | Runtime: nodejs18.x 95 | Architectures: 96 | - x86_64 97 | Events: 98 | ObjectCreatedEvent: 99 | Type: EventBridgeRule 100 | Properties: 101 | Pattern: 102 | source: 103 | - "aws.s3" 104 | detail-type: 105 | - "Object Created" 106 | detail: 107 | bucket: 108 | name: 109 | - !Ref UploadsBucket 110 | object: 111 | key: 112 | - wildcard: "*.png" 113 | - wildcard: "*.jpg" 114 | # Set Lambda execution role with IAM policies to access the S3 bucket and execute Textract 115 | Policies: 116 | - S3ReadPolicy: 117 | BucketName: !Ref UploadsBucketName 118 | - Version: 2012-10-17 119 | Statement: 120 | - Sid: DetectDocumentText 121 | Action: textract:DetectDocumentText 122 | Resource: "*" 123 | Effect: Allow 124 | - Version: 2012-10-17 125 | Statement: 126 | - Sid: CloudWatchLogGroup 127 | Action: 128 | - logs:CreateLogStream 129 | - logs:PutLogEvents 130 | Resource: !GetAtt CloudWatchLogGroup.Arn 131 | Effect: Allow 132 | 133 | # New custom log configration 134 | LoggingConfig: 135 | LogFormat: JSON 136 | ApplicationLogLevel: INFO 137 | SystemLogLevel: INFO 138 | LogGroup: !Ref CloudWatchLogGroup 139 | 140 | Metadata: # Manage esbuild properties 141 | BuildMethod: esbuild 142 | BuildProperties: 143 | Minify: true 144 | Target: es2020 145 | Sourcemap: true 146 | EntryPoints: 147 | - app.ts 148 | 149 | Outputs: 150 | UploadsBucket: 151 | Description: Uploads S3 Bucket 152 | Value: !Ref UploadsBucket 153 | LogGroupArn: 154 | Description: Custom aggregated log group ARN 155 | Value: !GetAtt CloudWatchLogGroup.Arn 156 | DetectLabelsFunction: 157 | Description: Detect Labels Function ARN 158 | Value: !GetAtt DetectLabelsFunction.Arn 159 | ExtractTextFunction: 160 | Description: Extract Text Function ARN 161 | Value: !GetAtt ExtractTextFunction.Arn 162 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Advanced Logging Controls Lambda Demo 2 | 3 | The purpose of this project is to demonstrate how to use the advanced logging controls for Lambda, using AWS SAM to build and deploy the resources in your AWS account. 4 | 5 | Using advanced logging controls you can capture logs in JSON structured format, allowing you to quickly search, filter, and analyze large volumes of log entries. You can also control the granularity of logs emitted by the Lambda function to debug and troubleshoot issues more effectively. Lastly, you can now choose the Amazon CloudWatch Log Group where Lambda sends logs to, making it easier to aggregate and manage logs at scale. 6 | 7 | ## Overview 8 | The following diagram shows how you can use a couple of Lambda functions to process newly created objects inside an Amazon S3 bucket, where both functions emit logs into the same CloudWatch Log Group: 9 |

10 | 11 |

12 | 13 | The architecture includes the following steps: 14 | 1. A new object is created inside an S3 bucket. 15 | 2. S3 publishes an event using [S3 Event Notifications](https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventNotifications.html) to [Amazon EventBridge](https://aws.amazon.com/eventbridge/). 16 | 3. EventBridge triggers two Lambda functions asynchronously. 17 | 4. Each function processes the object to extract labels and text, using [Amazon Rekogniton](https://aws.amazon.com/rekognition/) and [Amazon Textract](https://aws.amazon.com/textract/) (respectively). 18 | 5. Both functions then emit logs into the same CloudWatch Log Group. 19 | 20 | This project contains source code and supporting files for a serverless application that you can deploy with the SAM CLI. It includes the following files and folders. 21 | 22 | - detect-labels - AWS Lambda function code to detect labels inside an image using Amazon Rekognition, written in TypeScript (NodeJS 18.x). 23 | - extract-text - Lambda function's code to extract text inside an image using Amazon Textract, written in TypeScript (NodeJS 18.x). 24 | - samples - Sample image(s) you can use to test the application. 25 | - template.yaml - A template that defines the application's AWS resources, using AWS SAM. 26 | 27 | Each function definition (in the `template.yaml` file) contains a `LoggingConfig` configuration. For example: 28 | ```yaml 29 | LoggingConfig: 30 | LogFormat: JSON # Application logs format, defaults to JSON (TEXT is optional) 31 | ApplicationLogLevel: DEBUG # Application log level, defaults to INFO 32 | SystemLogLevel: INFO # System log level, defaults to INFO 33 | LogGroup: !Ref CloudWatchLogGroup # Customized log group to emit logs to 34 | ``` 35 | 36 | - `LogFormat` - allows you to choose between text and JSON format for your logs. 37 | - `ApplicationLogLevel` - for application logs (which are logs generated by your function code) you can choose between `TRACE`, `DEBUG`, `INFO` (default), `WARN`, `ERROR` and `FATAL`, where the `TRACE` log-level provides the most fine-grained information, and `FATAL` provides the least. 38 | - `SystemLogLevel` - for system logs (the logs generated by the Lambda service) you can choose between `DEBUG`, `INFO` (default) and `WARN`. 39 | - `LogGroup` - ARN of a custom log group, where the Lambda functions sends logs to. 40 | 41 | For your function to send logs to CloudWatch Logs, it must have the `logs:PutLogEvents` and `logs:CreateLogStream` permissions. When you configure your function's log group using the Lambda console, if your function doesn't have these permissions, Lambda adds them to the function's execution role by default. When Lambda adds this permission, it gives the function permission to send logs to any CloudWatch Logs log group. 42 | When you use SAM to configure your function, you must explicitly add these permissions: 43 | ```yaml 44 | Policies: 45 | - Version: 2012-10-17 46 | Statement: 47 | - Sid: CloudWatchLogGroup 48 | Action: 49 | - logs:CreateLogStream 50 | - logs:PutLogEvents 51 | Resource: !GetAtt CloudWatchLogGroup.Arn 52 | Effect: Allow 53 | 54 | ``` 55 | 56 | ## Deploying the application 57 | Clone the GitHub repository and explore the application. 58 | 59 | ```bash 60 | git clone https://github.com/aws-samples/advanced-logging-controls-lambda/ 61 | cd advanced-logging-controls-lambda 62 | ``` 63 | 64 | Use AWS SAM to build and deploy the resources to your AWS account: 65 | 1. Build the solution using AWS SAM. This will compile and build the application using npm, and then populate the template required to deploy the resources: 66 | ```bash 67 | sam build 68 | ``` 69 | 70 | 2. Deploy the solution to your AWS account with a guided deployment. Replace the bucket name with a unique name of your choosing: 71 | ```bash 72 | sam deploy --guided --parameter-overrides UploadsBucketName=example-s3-images-bucket 73 | ``` 74 | 75 | 3. Accept the initial defaults. 76 | 77 | 4. Use AWS CLI to copy an image into the Amazon S3 bucket you just created. You can use the sample image provided: 78 | ```bash 79 | aws s3 cp samples/skateboard.jpg s3://example-s3-images-bucket 80 | ``` 81 | 82 | ## Exploring the results 83 | Explore CloudWatch Logs to view the logs emitted into the log group created, `AggregatedLabelsLogGroup`: 84 |

85 | 86 |

87 | 88 | The DetectLabels Lambda function emits `DEBUG` log events in JSON format to the log stream. Log events with the same log level from the ExtractText Lambda function are omitted. This is a result of the different application log level settings for each function (`DEBUG` and `INFO`). 89 | 90 | Use [CloudWatch Logs Insights](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AnalyzingLogData.html) to search, filter, and analyze the logs in JSON format using this sample query. The query parse the JSON logs and filter only the logs events containing the S3 object key we copied earlier: 91 | ``` 92 | fields @timestamp, message.S3Bucket as bucket, message.S3Key as key, @message, @logStream, @log 93 | | filter key = 'skateboard.jpg' 94 | | sort @timestamp desc 95 | | limit 20 96 | ``` 97 | 98 | ## Cleanup 99 | 100 | To delete the sample application that you created, use the AWS CLI. Assuming you used your project name for the stack name, you can run the following: 101 | 102 | ```bash 103 | sam delete --stack-name advanced-logging-controls-lambda 104 | ``` 105 | 106 | ## Resources 107 | 108 | Learn more about [AWS Lambda Advanced Logging Controls](https://docs.aws.amazon.com/lambda/latest/dg/monitoring-cloudwatchlogs.html). 109 | 110 | See the [AWS SAM developer guide](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/what-is-sam.html) for an introduction to SAM specification, the SAM CLI, and serverless application concepts. 111 | 112 | ## License 113 | 114 | This library is licensed under the MIT-0 License. See the LICENSE file. --------------------------------------------------------------------------------