├── source ├── __init__.py ├── infrastructure │ ├── __init__.py │ ├── lib │ │ ├── __init__.py │ │ ├── blueprints │ │ │ ├── __init__.py │ │ │ ├── lambdas │ │ │ │ ├── __init__.py │ │ │ │ ├── inference │ │ │ │ │ ├── tests │ │ │ │ │ │ ├── __init__.py │ │ │ │ │ │ └── test_inference.py │ │ │ │ │ ├── requirements-test.txt │ │ │ │ │ ├── .coveragerc │ │ │ │ │ ├── setup.py │ │ │ │ │ └── main.py │ │ │ │ ├── batch_transform │ │ │ │ │ ├── tests │ │ │ │ │ │ ├── __init__.py │ │ │ │ │ │ └── test_batch_transform.py │ │ │ │ │ ├── requirements-test.txt │ │ │ │ │ ├── .coveragerc │ │ │ │ │ ├── setup.py │ │ │ │ │ └── main.py │ │ │ │ ├── create_baseline_job │ │ │ │ │ ├── tests │ │ │ │ │ │ └── __init__.py │ │ │ │ │ ├── requirements-test.txt │ │ │ │ │ ├── .coveragerc │ │ │ │ │ └── setup.py │ │ │ │ ├── create_model_training_job │ │ │ │ │ ├── __init__.py │ │ │ │ │ ├── tests │ │ │ │ │ │ ├── __init__.py │ │ │ │ │ │ ├── fixtures │ │ │ │ │ │ │ └── training_fixtures.py │ │ │ │ │ │ └── test_create_model_training.py │ │ │ │ │ ├── requirements-test.txt │ │ │ │ │ ├── .coveragerc │ │ │ │ │ ├── setup.py │ │ │ │ │ ├── model_training_helper.py │ │ │ │ │ └── main.py │ │ │ │ ├── create_sagemaker_autopilot_job │ │ │ │ │ ├── __init__.py │ │ │ │ │ ├── tests │ │ │ │ │ │ ├── __init__.py │ │ │ │ │ │ ├── test_create_autopilot_job.py │ │ │ │ │ │ └── fixtures │ │ │ │ │ │ │ └── autopilot_fixtures.py │ │ │ │ │ ├── requirements-test.txt │ │ │ │ │ ├── .coveragerc │ │ │ │ │ ├── setup.py │ │ │ │ │ └── main.py │ │ │ │ ├── create_update_cf_stackset │ │ │ │ │ ├── tests │ │ │ │ │ │ ├── __init__.py │ │ │ │ │ │ └── fixtures │ │ │ │ │ │ │ └── stackset_fixtures.py │ │ │ │ │ ├── requirements-test.txt │ │ │ │ │ ├── .coveragerc │ │ │ │ │ ├── setup.py │ │ │ │ │ └── main.py │ │ │ │ ├── invoke_lambda_custom_resource │ │ │ │ │ ├── tests │ │ │ │ │ │ ├── __init__.py │ │ │ │ │ │ └── test_invoke_lambda.py │ │ │ │ │ ├── requirements-test.txt │ │ │ │ │ ├── requirements.txt │ │ │ │ │ ├── .coveragerc │ │ │ │ │ ├── setup.py │ │ │ │ │ └── index.py │ │ │ │ └── sagemaker_layer │ │ │ │ │ └── requirements.txt │ │ │ ├── ml_pipelines │ │ │ │ ├── __init__.py │ │ │ │ ├── byom_custom_algorithm_image_builder.py │ │ │ │ └── single_account_codepipeline.py │ │ │ ├── pipeline_definitions │ │ │ │ ├── __init__.py │ │ │ │ ├── cdk_context_value.py │ │ │ │ ├── sagemaker_endpoint.py │ │ │ │ ├── approval_actions.py │ │ │ │ ├── sagemaker_model_registry.py │ │ │ │ ├── sagemaker_model.py │ │ │ │ ├── source_actions.py │ │ │ │ ├── sagemaker_endpoint_config.py │ │ │ │ ├── sagemaker_role.py │ │ │ │ ├── sagemaker_monitor_role.py │ │ │ │ └── build_actions.py │ │ │ ├── .gitignore │ │ │ └── aspects │ │ │ │ ├── protobuf_config_aspect.py │ │ │ │ ├── conditional_resource.py │ │ │ │ ├── aws_sdk_config_aspect.py │ │ │ │ └── app_registry_aspect.py │ │ └── utils │ │ │ └── cfnguard_helper.py │ ├── test │ │ ├── __init__.py │ │ └── context_helper.py │ ├── pytest.ini │ ├── .coveragerc │ └── cdk.json ├── lambdas │ ├── custom_resource │ │ ├── __init__.py │ │ ├── requirements-test.txt │ │ ├── requirements.txt │ │ ├── .coveragerc │ │ ├── setup.py │ │ ├── index.py │ │ └── tests │ │ │ └── test_custom_resource.py │ ├── pipeline_orchestration │ │ ├── __init__.py │ │ ├── shared │ │ │ ├── __init__.py │ │ │ ├── helper.py │ │ │ ├── logger.py │ │ │ └── wrappers.py │ │ ├── tests │ │ │ ├── __init__.py │ │ │ ├── test_helper.py │ │ │ └── test_logger.py │ │ ├── requirements-test.txt │ │ ├── .coveragerc │ │ └── setup.py │ └── solution_helper │ │ ├── requirements.txt │ │ ├── requirements-test.txt │ │ ├── .coveragerc │ │ ├── .gitignore │ │ └── lambda_function.py ├── pytest.ini ├── .coveragerc ├── architecture-option-1.png ├── architecture-option-2.png ├── requirements-test.txt ├── requirements.txt ├── .gitignore └── conftest.py ├── .github ├── PULL_REQUEST_TEMPLATE.md └── ISSUE_TEMPLATE │ ├── feature_request.md │ └── bug_report.md ├── CODE_OF_CONDUCT.md ├── deployment ├── cdk-solution-helper │ ├── package.json │ ├── package-lock.json │ ├── index.js │ └── README.md └── run-unit-tests.sh ├── SECURITY.md ├── .gitignore ├── CONTRIBUTING.md └── NOTICE.txt /source/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /source/infrastructure/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /source/infrastructure/lib/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /source/infrastructure/test/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /source/lambdas/custom_resource/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /source/lambdas/pipeline_orchestration/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/lambdas/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /source/lambdas/custom_resource/requirements-test.txt: -------------------------------------------------------------------------------- 1 | -e . -------------------------------------------------------------------------------- /source/lambdas/pipeline_orchestration/shared/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /source/lambdas/pipeline_orchestration/tests/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/ml_pipelines/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /source/lambdas/custom_resource/requirements.txt: -------------------------------------------------------------------------------- 1 | crhelper==2.0.10 -------------------------------------------------------------------------------- /source/lambdas/pipeline_orchestration/requirements-test.txt: -------------------------------------------------------------------------------- 1 | -e . -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/lambdas/inference/tests/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/pipeline_definitions/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/lambdas/batch_transform/tests/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /source/pytest.ini: -------------------------------------------------------------------------------- 1 | [pytest] 2 | norecursedirs = .venv-test .venv-prod cdk.out -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/lambdas/create_baseline_job/tests/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/lambdas/create_model_training_job/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/lambdas/inference/requirements-test.txt: -------------------------------------------------------------------------------- 1 | -e . -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/lambdas/batch_transform/requirements-test.txt: -------------------------------------------------------------------------------- 1 | -e . -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/lambdas/create_model_training_job/tests/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/lambdas/create_sagemaker_autopilot_job/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/lambdas/create_update_cf_stackset/tests/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/lambdas/create_baseline_job/requirements-test.txt: -------------------------------------------------------------------------------- 1 | -e . -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/lambdas/create_sagemaker_autopilot_job/tests/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/lambdas/invoke_lambda_custom_resource/tests/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/lambdas/create_model_training_job/requirements-test.txt: -------------------------------------------------------------------------------- 1 | -e . -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/lambdas/create_update_cf_stackset/requirements-test.txt: -------------------------------------------------------------------------------- 1 | -e . -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/lambdas/create_sagemaker_autopilot_job/requirements-test.txt: -------------------------------------------------------------------------------- 1 | -e . -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/lambdas/invoke_lambda_custom_resource/requirements-test.txt: -------------------------------------------------------------------------------- 1 | -e . -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/.gitignore: -------------------------------------------------------------------------------- 1 | # Generated cloudformation templates 2 | byom_*.yaml 3 | -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/lambdas/invoke_lambda_custom_resource/requirements.txt: -------------------------------------------------------------------------------- 1 | crhelper==2.0.10 -------------------------------------------------------------------------------- /source/infrastructure/pytest.ini: -------------------------------------------------------------------------------- 1 | [pytest] 2 | norecursedirs = lib/blueprints/lambdas source/lambdas cdk.out -------------------------------------------------------------------------------- /source/lambdas/solution_helper/requirements.txt: -------------------------------------------------------------------------------- 1 | crhelper==2.0.6 2 | urllib3==1.26.19 3 | requests==2.32.3 -------------------------------------------------------------------------------- /source/lambdas/solution_helper/requirements-test.txt: -------------------------------------------------------------------------------- 1 | crhelper==2.0.6 2 | urllib3==1.26.19 3 | requests==2.32.3 -------------------------------------------------------------------------------- /source/.coveragerc: -------------------------------------------------------------------------------- 1 | [run] 2 | omit = 3 | tests/* 4 | setup.py 5 | */.venv-test/* 6 | cdk.out/* 7 | source = 8 | . -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/lambdas/sagemaker_layer/requirements.txt: -------------------------------------------------------------------------------- 1 | botocore==1.34.98 2 | boto3==1.34.98 3 | sagemaker==2.218.0 -------------------------------------------------------------------------------- /source/architecture-option-1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-solutions/mlops-workload-orchestrator/main/source/architecture-option-1.png -------------------------------------------------------------------------------- /source/architecture-option-2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-solutions/mlops-workload-orchestrator/main/source/architecture-option-2.png -------------------------------------------------------------------------------- /source/requirements-test.txt: -------------------------------------------------------------------------------- 1 | sagemaker==2.218.0 2 | boto3==1.34.98 3 | crhelper==2.0.6 4 | pytest==7.2.0 5 | pytest-cov==4.1.0 6 | moto[all]==5.0.6 -------------------------------------------------------------------------------- /source/lambdas/custom_resource/.coveragerc: -------------------------------------------------------------------------------- 1 | [run] 2 | omit = 3 | tests/* 4 | setup.py 5 | */.venv-test/* 6 | cdk.out/* 7 | conftest.py 8 | test_*.py 9 | source = 10 | . -------------------------------------------------------------------------------- /source/lambdas/pipeline_orchestration/.coveragerc: -------------------------------------------------------------------------------- 1 | [run] 2 | omit = 3 | tests/* 4 | setup.py 5 | */.venv-test/* 6 | cdk.out/* 7 | conftest.py 8 | test_*.py 9 | source = 10 | . -------------------------------------------------------------------------------- /source/lambdas/solution_helper/.coveragerc: -------------------------------------------------------------------------------- 1 | [run] 2 | omit = 3 | tests/* 4 | setup.py 5 | charset_normalizer/** 6 | */.venv-test/* 7 | cdk.out/* 8 | conftest.py 9 | test_*.py 10 | source = 11 | . -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/lambdas/batch_transform/.coveragerc: -------------------------------------------------------------------------------- 1 | [run] 2 | omit = 3 | tests/* 4 | setup.py 5 | */.venv-test/* 6 | cdk.out/* 7 | conftest.py 8 | test_*.py 9 | source = 10 | . -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/lambdas/create_baseline_job/.coveragerc: -------------------------------------------------------------------------------- 1 | [run] 2 | omit = 3 | tests/* 4 | setup.py 5 | */.venv-test/* 6 | cdk.out/* 7 | conftest.py 8 | test_*.py 9 | source = 10 | . -------------------------------------------------------------------------------- /source/infrastructure/.coveragerc: -------------------------------------------------------------------------------- 1 | [run] 2 | omit = 3 | tests/* 4 | setup.py 5 | */.venv-test/* 6 | cdk.out/* 7 | conftest.py 8 | test_*.py 9 | app.py 10 | lib/blueprints/lambdas/* 11 | source = 12 | . -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/lambdas/create_model_training_job/.coveragerc: -------------------------------------------------------------------------------- 1 | [run] 2 | omit = 3 | tests/* 4 | setup.py 5 | */.venv-test/* 6 | cdk.out/* 7 | conftest.py 8 | test_*.py 9 | source = 10 | . -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/lambdas/create_update_cf_stackset/.coveragerc: -------------------------------------------------------------------------------- 1 | [run] 2 | omit = 3 | tests/* 4 | setup.py 5 | */.venv-test/* 6 | cdk.out/* 7 | conftest.py 8 | test_*.py 9 | source = 10 | . -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/lambdas/inference/.coveragerc: -------------------------------------------------------------------------------- 1 | [run] 2 | omit = 3 | tests/* 4 | setup.py 5 | */.venv-test/* 6 | cdk.out/* 7 | test_*.py 8 | conftest.py 9 | *wrappers.py 10 | source = 11 | . -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/lambdas/invoke_lambda_custom_resource/.coveragerc: -------------------------------------------------------------------------------- 1 | [run] 2 | omit = 3 | tests/* 4 | setup.py 5 | */.venv-test/* 6 | cdk.out/* 7 | conftest.py 8 | test_*.py 9 | source = 10 | . -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/lambdas/create_sagemaker_autopilot_job/.coveragerc: -------------------------------------------------------------------------------- 1 | [run] 2 | omit = 3 | tests/* 4 | setup.py 5 | */.venv-test/* 6 | cdk.out/* 7 | conftest.py 8 | test_*.py 9 | source = 10 | . -------------------------------------------------------------------------------- /.github/PULL_REQUEST_TEMPLATE.md: -------------------------------------------------------------------------------- 1 | *Issue #, if available:* 2 | 3 | *Description of changes:* 4 | 5 | By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice. 6 | -------------------------------------------------------------------------------- /source/lambdas/solution_helper/.gitignore: -------------------------------------------------------------------------------- 1 | # exclude python 3rd party modules 2 | *.dist-info/ 3 | bin 4 | certifi/ 5 | chardet/ 6 | crhelper/ 7 | idna/ 8 | requests/ 9 | charset_normalizer/ 10 | # crhelper tests directory 11 | tests/ 12 | urllib3/ 13 | -------------------------------------------------------------------------------- /source/requirements.txt: -------------------------------------------------------------------------------- 1 | aws-cdk-lib==2.167.0 2 | constructs==10.4.2 3 | aws-solutions-constructs.aws-apigateway-lambda==2.74.0 4 | aws-solutions-constructs.aws-lambda-sagemakerendpoint==2.74.0 5 | aws-solutions-constructs.core==2.74.0 6 | aws-cdk.aws-servicecatalogappregistry-alpha==2.167.0a0 -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | ## Code of Conduct 2 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 3 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 4 | opensource-codeofconduct@amazon.com with any additional questions or comments. 5 | -------------------------------------------------------------------------------- /source/.gitignore: -------------------------------------------------------------------------------- 1 | *.swp 2 | package-lock.json 3 | __pycache__ 4 | .pytest_cache 5 | .venv-test 6 | *.egg-info 7 | .coverage 8 | 9 | # CDK asset staging directory 10 | .cdk.staging 11 | cdk.out 12 | 13 | # Python virtual environment 14 | lib/**/lambdas/*/python 15 | 16 | # Environments 17 | .env 18 | env/ 19 | venv/ 20 | .venv* 21 | ENV/ 22 | env.bak/ 23 | venv.bak/ -------------------------------------------------------------------------------- /source/infrastructure/cdk.json: -------------------------------------------------------------------------------- 1 | { 2 | "app": "python3 app.py", 3 | "context": { 4 | "SolutionId": "SO0136", 5 | "SolutionName": "%%SOLUTION_NAME%%", 6 | "Version": "%%VERSION%%", 7 | "AppRegistryName": "mlops", 8 | "ApplicationType": "AWS-Solutions", 9 | "SourceBucket": "%%BUCKET_NAME%%", 10 | "BlueprintsFile": "blueprints.zip" 11 | 12 | } 13 | } -------------------------------------------------------------------------------- /deployment/cdk-solution-helper/package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "cdk-solution-helper", 3 | "version": "0.1.0", 4 | "description": "This script performs token replacement as part of the build pipeline", 5 | "dependencies": { 6 | "fs": "^0.0.2" 7 | }, 8 | "license": "Apache-2.0", 9 | "author": { 10 | "name": "Amazon Web Services", 11 | "url": "https://aws.amazon.com/solutions" 12 | } 13 | } 14 | -------------------------------------------------------------------------------- /SECURITY.md: -------------------------------------------------------------------------------- 1 | ## Reporting Security Issues 2 | 3 | We take all security reports seriously. When we receive such reports, 4 | we will investigate and subsequently address any potential vulnerabilities as 5 | quickly as possible. If you discover a potential security issue in this project, 6 | please notify AWS/Amazon Security via our [vulnerability reporting page] 7 | (http://aws.amazon.com/security/vulnerability-reporting/) or directly via email 8 | to [AWS Security](mailto:aws-security@amazon.com). 9 | Please do *not* create a public GitHub issue in this project. -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/feature_request.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Feature request 3 | about: Suggest an idea for this solution 4 | title: '' 5 | labels: enhancement 6 | assignees: '' 7 | 8 | --- 9 | 10 | **Is your feature request related to a problem? Please describe.** 11 | A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] 12 | 13 | **Describe the feature you'd like** 14 | A clear and concise description of what you want to happen. 15 | 16 | **Additional context** 17 | Add any other context or screenshots about the feature request here. 18 | -------------------------------------------------------------------------------- /deployment/cdk-solution-helper/package-lock.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "cdk-solution-helper", 3 | "version": "0.1.0", 4 | "lockfileVersion": 2, 5 | "requires": true, 6 | "packages": { 7 | "": { 8 | "name": "cdk-solution-helper", 9 | "version": "0.1.0", 10 | "license": "Apache-2.0", 11 | "dependencies": { 12 | "fs": "^0.0.2" 13 | } 14 | }, 15 | "node_modules/fs": { 16 | "version": "0.0.2", 17 | "resolved": "https://registry.npmjs.org/fs/-/fs-0.0.2.tgz", 18 | "integrity": "sha1-4fJE7zkzwbKmS9R5kTYGDQ9ZFPg=" 19 | } 20 | }, 21 | "dependencies": { 22 | "fs": { 23 | "version": "0.0.2", 24 | "resolved": "https://registry.npmjs.org/fs/-/fs-0.0.2.tgz", 25 | "integrity": "sha1-4fJE7zkzwbKmS9R5kTYGDQ9ZFPg=" 26 | } 27 | } 28 | } -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | *.egg 2 | *.egg-info 3 | __pycache__ 4 | *.js 5 | !jest.config.js 6 | *.d.ts 7 | node_modules 8 | !source/.typescript/lambda/**/*.js 9 | !source/lambda/**/*.js 10 | 11 | # CDK asset staging directory 12 | .cdk.staging 13 | cdk.out 14 | 15 | !**/cdk-solution-helper/index.js 16 | 17 | deployment/global-s3-assets 18 | deployment/regional-s3-assets 19 | open-source/ 20 | .vscode/settings.json 21 | 22 | source/lambda/layers/* 23 | 24 | # Coverage directory used by tools like istanbul 25 | coverage-reports/ 26 | coverage 27 | *.lcov 28 | 29 | # Unit test / coverage reports 30 | htmlcov/ 31 | .tox/ 32 | .nox/ 33 | .coverage 34 | .coverage.* 35 | .cache 36 | nosetests.xml 37 | coverage.xml 38 | *.cover 39 | *.py,cover 40 | .hypothesis/ 41 | .pytest_cache/ 42 | 43 | # linting, scanning configurations, sonarqube 44 | .scannerwork/ 45 | 46 | .DS_Store 47 | 48 | # Temporary folders 49 | tmp/ 50 | temp/ 51 | 52 | ### VisualStudioCode Path ### 53 | .vscode 54 | 55 | 56 | # Python Virtual env 57 | .venv 58 | 59 | source/infrastructure/lib/blueprints/lambdas/sagemaker_layer/python/** 60 | **crhelper** 61 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/bug_report.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Bug report 3 | about: Create a report to help us improve 4 | title: '' 5 | labels: bug 6 | assignees: '' 7 | 8 | --- 9 | 10 | **Describe the bug** 11 | A clear and concise description of what the bug is. 12 | 13 | **To Reproduce** 14 | Steps to reproduce the behavior. 15 | 16 | **Expected behavior** 17 | A clear and concise description of what you expected to happen. 18 | 19 | **Please complete the following information about the solution:** 20 | - [ ] Version: [e.g. v1.0.0] 21 | 22 | To get the version of the solution, you can look at the description of the created CloudFormation stack. For example, "(SO0136) - AWS MLOps Framework. Version v1.0.0". 23 | 24 | - [ ] Region: [e.g. us-east-1] 25 | - [ ] Was the solution modified from the version published on this repository? 26 | - [ ] If the answer to the previous question was yes, are the changes available on GitHub? 27 | - [ ] Have you checked your [service quotas](https://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html) for the services this solution uses? 28 | - [ ] Were there any errors in the CloudWatch Logs? 29 | 30 | **Screenshots** 31 | If applicable, add screenshots to help explain your problem (please **DO NOT include sensitive information**). 32 | 33 | **Additional context** 34 | Add any other context about the problem here. -------------------------------------------------------------------------------- /source/lambdas/custom_resource/setup.py: -------------------------------------------------------------------------------- 1 | ####################################################################################################################### 2 | # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. # 3 | # # 4 | # Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance # 5 | # with the License. A copy of the License is located at # 6 | # # 7 | # http://www.apache.org/licenses/LICENSE-2.0 # 8 | # # 9 | # or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES # 10 | # OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions # 11 | # and limitations under the License. # 12 | # ##################################################################################################################### 13 | from setuptools import setup, find_packages 14 | 15 | setup(name="custom_resource", packages=find_packages()) -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/lambdas/inference/setup.py: -------------------------------------------------------------------------------- 1 | ################################################################################################################## 2 | # Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. # 3 | # # 4 | # Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance # 5 | # with the License. A copy of the License is located at # 6 | # # 7 | # http://www.apache.org/licenses/LICENSE-2.0 # 8 | # # 9 | # or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES # 10 | # OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions # 11 | # and limitations under the License. # 12 | # ##################################################################################################################### 13 | from setuptools import setup, find_packages 14 | 15 | setup(name="inference", packages=find_packages()) -------------------------------------------------------------------------------- /source/lambdas/pipeline_orchestration/setup.py: -------------------------------------------------------------------------------- 1 | ####################################################################################################################### 2 | # Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. # 3 | # # 4 | # Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance # 5 | # with the License. A copy of the License is located at # 6 | # # 7 | # http://www.apache.org/licenses/LICENSE-2.0 # 8 | # # 9 | # or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES # 10 | # OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions # 11 | # and limitations under the License. # 12 | # ##################################################################################################################### 13 | from setuptools import setup, find_packages 14 | 15 | setup(name="pipeline_orchestration", packages=find_packages()) -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/lambdas/batch_transform/setup.py: -------------------------------------------------------------------------------- 1 | ####################################################################################################################### 2 | # Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. # 3 | # # 4 | # Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance # 5 | # with the License. A copy of the License is located at # 6 | # # 7 | # http://www.apache.org/licenses/LICENSE-2.0 # 8 | # # 9 | # or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES # 10 | # OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions # 11 | # and limitations under the License. # 12 | # ##################################################################################################################### 13 | from setuptools import setup, find_packages 14 | 15 | setup(name="batch_transform", packages=find_packages()) -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/lambdas/create_baseline_job/setup.py: -------------------------------------------------------------------------------- 1 | ####################################################################################################################### 2 | # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. # 3 | # # 4 | # Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance # 5 | # with the License. A copy of the License is located at # 6 | # # 7 | # http://www.apache.org/licenses/LICENSE-2.0 # 8 | # # 9 | # or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES # 10 | # OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions # 11 | # and limitations under the License. # 12 | # ##################################################################################################################### 13 | from setuptools import setup, find_packages 14 | 15 | setup(name="create_data_baseline_job", packages=find_packages()) -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/lambdas/create_model_training_job/setup.py: -------------------------------------------------------------------------------- 1 | ####################################################################################################################### 2 | # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. # 3 | # # 4 | # Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance # 5 | # with the License. A copy of the License is located at # 6 | # # 7 | # http://www.apache.org/licenses/LICENSE-2.0 # 8 | # # 9 | # or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES # 10 | # OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions # 11 | # and limitations under the License. # 12 | # ##################################################################################################################### 13 | from setuptools import setup, find_packages 14 | 15 | setup(name="create_model_training_job", packages=find_packages()) -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/lambdas/create_update_cf_stackset/setup.py: -------------------------------------------------------------------------------- 1 | ####################################################################################################################### 2 | # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. # 3 | # # 4 | # Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance # 5 | # with the License. A copy of the License is located at # 6 | # # 7 | # http://www.apache.org/licenses/LICENSE-2.0 # 8 | # # 9 | # or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES # 10 | # OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions # 11 | # and limitations under the License. # 12 | # ##################################################################################################################### 13 | from setuptools import setup, find_packages 14 | 15 | setup(name="create_update_cf_stackset", packages=find_packages()) -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/lambdas/invoke_lambda_custom_resource/setup.py: -------------------------------------------------------------------------------- 1 | ####################################################################################################################### 2 | # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. # 3 | # # 4 | # Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance # 5 | # with the License. A copy of the License is located at # 6 | # # 7 | # http://www.apache.org/licenses/LICENSE-2.0 # 8 | # # 9 | # or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES # 10 | # OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions # 11 | # and limitations under the License. # 12 | # ##################################################################################################################### 13 | from setuptools import setup, find_packages 14 | 15 | setup(name="invoke_lambda_custom_resource", packages=find_packages()) -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/lambdas/create_sagemaker_autopilot_job/setup.py: -------------------------------------------------------------------------------- 1 | ####################################################################################################################### 2 | # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. # 3 | # # 4 | # Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance # 5 | # with the License. A copy of the License is located at # 6 | # # 7 | # http://www.apache.org/licenses/LICENSE-2.0 # 8 | # # 9 | # or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES # 10 | # OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions # 11 | # and limitations under the License. # 12 | # ##################################################################################################################### 13 | from setuptools import setup, find_packages 14 | 15 | setup(name="create_sagemaker_autopilot_job", packages=find_packages()) -------------------------------------------------------------------------------- /source/infrastructure/test/context_helper.py: -------------------------------------------------------------------------------- 1 | # ##################################################################################################################### 2 | # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. # 3 | # # 4 | # Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance # 5 | # with the License. A copy of the License is located at # 6 | # # 7 | # http://www.apache.org/licenses/LICENSE-2.0 # 8 | # # 9 | # or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES # 10 | # OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions # 11 | # and limitations under the License. # 12 | # ##################################################################################################################### 13 | import json 14 | 15 | 16 | def get_cdk_context(file): 17 | with open(file) as json_file: 18 | raw_context = json.load(json_file) 19 | return raw_context 20 | -------------------------------------------------------------------------------- /source/conftest.py: -------------------------------------------------------------------------------- 1 | # ##################################################################################################################### 2 | # Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. # 3 | # # 4 | # Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance # 5 | # with the License. A copy of the License is located at # 6 | # # 7 | # http://www.apache.org/licenses/LICENSE-2.0 # 8 | # # 9 | # or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES # 10 | # OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions # 11 | # and limitations under the License. # 12 | # ##################################################################################################################### 13 | import os 14 | import boto3 15 | import pytest 16 | 17 | CONFIG_FILE = "config_and_overrides.yaml" 18 | 19 | 20 | @pytest.fixture(autouse=True) 21 | def aws_credentials(): 22 | """Mocked AWS Credentials""" 23 | os.environ["AWS_ACCESS_KEY_ID"] = "testing" 24 | os.environ["AWS_SECRET_ACCESS_KEY"] = "testing" 25 | os.environ["AWS_REGION"] = "us-east-1" # must be a valid region 26 | -------------------------------------------------------------------------------- /source/infrastructure/lib/utils/cfnguard_helper.py: -------------------------------------------------------------------------------- 1 | # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | # SPDX-License-Identifier: Apache-2.0 3 | 4 | from typing import List 5 | 6 | import jsii 7 | from aws_cdk import CfnResource, IAspect 8 | from constructs import IConstruct 9 | 10 | 11 | 12 | def add_cfn_guard_suppressions( 13 | resource: CfnResource, suppressions: List[str] 14 | ): 15 | if resource.node.default_child: 16 | resource.node.default_child.add_metadata( 17 | "guard", 18 | { 19 | "SuppressedRules": suppressions 20 | }, 21 | ) 22 | else: 23 | resource.add_metadata( 24 | "guard", 25 | { 26 | "SuppressedRules": suppressions 27 | }, 28 | ) 29 | 30 | @jsii.implements(IAspect) 31 | class CfnGuardSuppressResourceList: 32 | """Suppress certain cfn_guard warnings that can be ignored by this solution""" 33 | 34 | def __init__(self, resource_suppressions: dict): 35 | self.resource_suppressions = resource_suppressions 36 | 37 | def visit(self, node: IConstruct): 38 | if "is_cfn_element" in dir(node) and \ 39 | node.is_cfn_element(node) and \ 40 | getattr(node, "cfn_resource_type", None) is not None and \ 41 | node.cfn_resource_type in self.resource_suppressions: 42 | add_cfn_guard_suppressions(node, self.resource_suppressions[node.cfn_resource_type]) 43 | elif "is_cfn_element" in dir(node.node.default_child) and \ 44 | getattr(node.node.default_child, "cfn_resource_type", None) is not None and \ 45 | node.node.default_child.cfn_resource_type in self.resource_suppressions: 46 | add_cfn_guard_suppressions(node.node.default_child, self.resource_suppressions[node.node.default_child.cfn_resource_type]) 47 | -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/pipeline_definitions/cdk_context_value.py: -------------------------------------------------------------------------------- 1 | # ##################################################################################################################### 2 | # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. # 3 | # # 4 | # Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance # 5 | # with the License. A copy of the License is located at # 6 | # # 7 | # http://www.apache.org/licenses/LICENSE-2.0 # 8 | # # 9 | # or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES # 10 | # OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions # 11 | # and limitations under the License. # 12 | # ##################################################################################################################### 13 | def get_cdk_context_value(scope, key): 14 | """ 15 | get_cdk_context_value gets the cdk context value for a provided key 16 | 17 | :scope: CDK Construct scope 18 | 19 | :returns: context value 20 | :Raises: Exception: The context key: {key} is undefined. 21 | """ 22 | value = scope.node.try_get_context(key) 23 | if value is None: 24 | raise ValueError(f"The CDK context key: {key} is undefined.") 25 | else: 26 | return value 27 | -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/pipeline_definitions/sagemaker_endpoint.py: -------------------------------------------------------------------------------- 1 | # ##################################################################################################################### 2 | # Copyright 2021 Amazon.com, Inc. or its affiliates. All Rights Reserved. # 3 | # # 4 | # Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance # 5 | # with the License. A copy of the License is located at # 6 | # # 7 | # http://www.apache.org/licenses/LICENSE-2.0 # 8 | # # 9 | # or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES # 10 | # OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions # 11 | # and limitations under the License. # 12 | # ##################################################################################################################### 13 | from aws_cdk import ( 14 | aws_sagemaker as sagemaker, 15 | ) 16 | 17 | 18 | def create_sagemaker_endpoint(scope, id, endpoint_config_name, endpoint_name, model_name, **kwargs): 19 | # create Sagemaker endpoint 20 | sagemaker_endpoint = sagemaker.CfnEndpoint( 21 | scope, 22 | id, 23 | endpoint_config_name=endpoint_config_name, 24 | endpoint_name=endpoint_name, 25 | tags=[{"key": "endpoint-name", "value": f"{model_name}-endpoint"}], 26 | **kwargs, 27 | ) 28 | 29 | return sagemaker_endpoint 30 | -------------------------------------------------------------------------------- /source/lambdas/pipeline_orchestration/tests/test_helper.py: -------------------------------------------------------------------------------- 1 | # ##################################################################################################################### 2 | # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. # 3 | # # 4 | # Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance # 5 | # with the License. A copy of the License is located at # 6 | # # 7 | # http://www.apache.org/licenses/LICENSE-2.0 # 8 | # # 9 | # or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES # 10 | # OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions # 11 | # and limitations under the License. # 12 | # ##################################################################################################################### 13 | import pytest 14 | from shared.helper import get_client, reset_client 15 | 16 | 17 | _helpers_service_clients = dict() 18 | 19 | 20 | @pytest.mark.parametrize("service,enpoint_url", [("s3", "https://s3"), ("cloudformation", "https://cloudformation")]) 21 | def test_get_client(service, enpoint_url): 22 | client = get_client(service) 23 | assert enpoint_url in client.meta.endpoint_url 24 | 25 | 26 | @pytest.mark.parametrize("service", ["s3", "cloudformation"]) 27 | def test_reset_client(service): 28 | get_client(service) 29 | reset_client() 30 | assert _helpers_service_clients == dict() -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/aspects/protobuf_config_aspect.py: -------------------------------------------------------------------------------- 1 | # ##################################################################################################################### 2 | # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. # 3 | # # 4 | # Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance # 5 | # with the License. A copy of the License is located at # 6 | # # 7 | # http://www.apache.org/licenses/LICENSE-2.0 # 8 | # # 9 | # or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES # 10 | # OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions # 11 | # and limitations under the License. # 12 | # ##################################################################################################################### 13 | import jsii 14 | from constructs import IConstruct, Construct 15 | from aws_cdk import IAspect 16 | from aws_cdk.aws_lambda import Function 17 | 18 | 19 | @jsii.implements(IAspect) 20 | class ProtobufConfigAspect(Construct): 21 | def __init__(self, scope: Construct, id: str): 22 | super().__init__(scope, id) 23 | 24 | def visit(self, node: IConstruct): 25 | if isinstance(node, Function): 26 | # this is to handle the protobuf package breaking changes. 27 | node.add_environment( 28 | key="PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION", value="python" 29 | ) 30 | -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/lambdas/create_sagemaker_autopilot_job/tests/test_create_autopilot_job.py: -------------------------------------------------------------------------------- 1 | ####################################################################################################################### 2 | # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. # 3 | # # 4 | # Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance # 5 | # with the License. A copy of the License is located at # 6 | # # 7 | # http://www.apache.org/licenses/LICENSE-2.0 # 8 | # # 9 | # or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES # 10 | # OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions # 11 | # and limitations under the License. # 12 | # ##################################################################################################################### 13 | from unittest.mock import patch 14 | from unittest import TestCase 15 | from tests.fixtures.autopilot_fixtures import mocked_job_name, mocked_autopilot_env_vars, mocked_automl_config 16 | 17 | 18 | @patch("main.AutoML") 19 | @patch("main.Session") 20 | @patch("main.get_client") 21 | def test_handler( 22 | mocked_client, mocked_session, mocked_automl, mocked_job_name, mocked_autopilot_env_vars, mocked_automl_config 23 | ): 24 | mocked_client.boto_region_name = "us-east-1" 25 | from main import handler 26 | 27 | handler(None, None) 28 | mocked_automl.asseert_called_with(**mocked_automl_config) 29 | -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/aspects/conditional_resource.py: -------------------------------------------------------------------------------- 1 | # ##################################################################################################################### 2 | # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. # 3 | # # 4 | # Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance # 5 | # with the License. A copy of the License is located at # 6 | # # 7 | # http://www.apache.org/licenses/LICENSE-2.0 # 8 | # # 9 | # or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES # 10 | # OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions # 11 | # and limitations under the License. # 12 | # ##################################################################################################################### 13 | import jsii 14 | from aws_cdk import CfnCondition, CfnResource, IAspect 15 | from constructs import IConstruct 16 | 17 | # This code enables `apply_aspect()` to apply conditions to a resource. 18 | # This way we can provision some resources if a condition is true. 19 | # https://docs.aws.amazon.com/cdk/latest/guide/aspects.html 20 | 21 | 22 | @jsii.implements(IAspect) 23 | class ConditionalResources: 24 | def __init__(self, condition: CfnCondition): 25 | self.condition = condition 26 | 27 | def visit(self, node: IConstruct): 28 | child = node.node.default_child # type: CfnResource 29 | if child: 30 | child.cfn_options.condition = self.condition 31 | -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/aspects/aws_sdk_config_aspect.py: -------------------------------------------------------------------------------- 1 | # ##################################################################################################################### 2 | # Copyright 2021 Amazon.com, Inc. or its affiliates. All Rights Reserved. # 3 | # # 4 | # Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance # 5 | # with the License. A copy of the License is located at # 6 | # # 7 | # http://www.apache.org/licenses/LICENSE-2.0 # 8 | # # 9 | # or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES # 10 | # OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions # 11 | # and limitations under the License. # 12 | # ##################################################################################################################### 13 | import jsii 14 | import json 15 | from constructs import IConstruct, Construct 16 | from aws_cdk import IAspect 17 | from aws_cdk.aws_lambda import Function 18 | 19 | 20 | @jsii.implements(IAspect) 21 | class AwsSDKConfigAspect(Construct): 22 | def __init__(self, scope: Construct, id: str, solution_id: str, version: str): 23 | super().__init__(scope, id) 24 | self.solution_id = solution_id 25 | self.version = version 26 | 27 | def visit(self, node: IConstruct): 28 | if isinstance(node, Function): 29 | user_agent = json.dumps( 30 | {"user_agent_extra": f"AwsSolution/{self.solution_id}/{self.version}"} 31 | ) 32 | node.add_environment(key="AWS_SDK_USER_AGENT", value=user_agent) 33 | -------------------------------------------------------------------------------- /source/lambdas/pipeline_orchestration/tests/test_logger.py: -------------------------------------------------------------------------------- 1 | # ##################################################################################################################### 2 | # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. # 3 | # # 4 | # Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance # 5 | # with the License. A copy of the License is located at # 6 | # # 7 | # http://www.apache.org/licenses/LICENSE-2.0 # 8 | # # 9 | # or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES # 10 | # OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions # 11 | # and limitations under the License. # 12 | # ##################################################################################################################### 13 | import logging 14 | import os 15 | import pytest 16 | from shared.logger import get_level, get_logger 17 | 18 | 19 | @pytest.fixture(scope="function", autouse=True) 20 | def rest_logger(): 21 | if "LOG_LEVEL" in os.environ: 22 | os.environ.pop("LOG_LEVEL") 23 | 24 | 25 | @pytest.mark.parametrize("log_level", ["DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"]) 26 | def test_get_level(log_level): 27 | os.environ["LOG_LEVEL"] = log_level 28 | assert get_level() == log_level 29 | 30 | 31 | def test_default_level(): 32 | os.environ["LOG_LEVEL"] = "no_supported" 33 | assert get_level() == "WARNING" 34 | 35 | 36 | def test_get_level_locally(): 37 | logging.getLogger().handlers = [] 38 | logger = get_logger(__name__) 39 | assert logger.level == 0 40 | -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/pipeline_definitions/approval_actions.py: -------------------------------------------------------------------------------- 1 | # ##################################################################################################################### 2 | # Copyright 2021 Amazon.com, Inc. or its affiliates. All Rights Reserved. # 3 | # # 4 | # Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance # 5 | # with the License. A copy of the License is located at # 6 | # # 7 | # http://www.apache.org/licenses/LICENSE-2.0 # 8 | # # 9 | # or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES # 10 | # OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions # 11 | # and limitations under the License. # 12 | # ##################################################################################################################### 13 | from aws_cdk import ( 14 | aws_codepipeline_actions as codepipeline_actions, 15 | ) 16 | 17 | 18 | def approval_action(approval_name, sns_topic, description): 19 | """ 20 | approval_action configures a codepipeline manual approval 21 | 22 | :approval_name: name of the manual approval action 23 | :sns_topic: sns topic to use for notifications 24 | :description: description of the manual approval action 25 | :return: codepipeline action in a form of a CDK object that can be attached to a codepipeline stage 26 | """ 27 | return codepipeline_actions.ManualApprovalAction( 28 | action_name=approval_name, 29 | notification_topic=sns_topic, 30 | additional_information=description, 31 | run_order=2, 32 | ) 33 | -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/pipeline_definitions/sagemaker_model_registry.py: -------------------------------------------------------------------------------- 1 | # ##################################################################################################################### 2 | # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. # 3 | # # 4 | # Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance # 5 | # with the License. A copy of the License is located at # 6 | # # 7 | # http://www.apache.org/licenses/LICENSE-2.0 # 8 | # # 9 | # or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES # 10 | # OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions # 11 | # and limitations under the License. # 12 | # ##################################################################################################################### 13 | from aws_cdk import Aws, RemovalPolicy, aws_sagemaker as sagemaker 14 | 15 | 16 | def create_sagemaker_model_registry(scope, id, model_package_group_name): 17 | """ 18 | create_sagemaker_model_registry creates SageMaker model package group (i.e., model registry) 19 | 20 | :scope: CDK Construct scope that's needed to create CDK resources 21 | :model_package_group_name: the name of the model package group name to be created 22 | 23 | :return: SageMaker model package group CDK object 24 | """ 25 | # create model registry 26 | model_registry = sagemaker.CfnModelPackageGroup( 27 | scope, 28 | id, 29 | model_package_group_name=model_package_group_name, 30 | model_package_group_description="SageMaker model package group name (model registry) for mlops", 31 | tags=[{"key": "stack-name", "value": Aws.STACK_NAME}], 32 | ) 33 | 34 | # add update/deletion policy 35 | model_registry.apply_removal_policy(RemovalPolicy.RETAIN) 36 | 37 | return model_registry 38 | -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/lambdas/inference/main.py: -------------------------------------------------------------------------------- 1 | # ##################################################################################################################### 2 | # Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. # 3 | # # 4 | # Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance # 5 | # with the License. A copy of the License is located at # 6 | # # 7 | # http://www.apache.org/licenses/LICENSE-2.0 # 8 | # # 9 | # or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES # 10 | # OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions # 11 | # and limitations under the License. # 12 | # ##################################################################################################################### 13 | import os 14 | import json 15 | import boto3 16 | from shared.wrappers import api_exception_handler 17 | from shared.logger import get_logger 18 | from shared.helper import get_client 19 | 20 | logger = get_logger(__name__) 21 | sagemaker_client = get_client("sagemaker-runtime") 22 | 23 | 24 | @api_exception_handler 25 | def handler(event, context): 26 | event_body = json.loads(event["body"]) 27 | endpoint_name = os.environ["SAGEMAKER_ENDPOINT_NAME"] 28 | return invoke(event_body, endpoint_name) 29 | 30 | 31 | def invoke(event_body, endpoint_name, sm_client=sagemaker_client): 32 | # convert the payload to a string if it not a string (to support JSON input) 33 | payload = event_body["payload"] if isinstance(event_body["payload"], str) else json.dumps(event_body["payload"]) 34 | response = sm_client.invoke_endpoint( 35 | EndpointName=endpoint_name, Body=payload, ContentType=event_body["content_type"] 36 | ) 37 | logger.info(response) 38 | predictions = response["Body"].read().decode() 39 | logger.info(predictions) 40 | return { 41 | "statusCode": 200, 42 | "isBase64Encoded": False, 43 | "body": predictions, 44 | "headers": {"Content-Type": "plain/text"}, 45 | } 46 | -------------------------------------------------------------------------------- /source/lambdas/pipeline_orchestration/shared/helper.py: -------------------------------------------------------------------------------- 1 | # ##################################################################################################################### 2 | # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. # 3 | # # 4 | # Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance # 5 | # with the License. A copy of the License is located at # 6 | # # 7 | # http://www.apache.org/licenses/LICENSE-2.0 # 8 | # # 9 | # or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES # 10 | # OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions # 11 | # and limitations under the License. # 12 | # ##################################################################################################################### 13 | import boto3 14 | import json 15 | import os 16 | import datetime 17 | from json import JSONEncoder 18 | from botocore.config import Config 19 | from shared.logger import get_logger 20 | 21 | logger = get_logger(__name__) 22 | _helpers_service_clients = dict() 23 | 24 | 25 | # Set Boto3 configuration to track the solution's usage 26 | CLIENT_CONFIG = Config( 27 | retries={"max_attempts": 3, "mode": "standard"}, 28 | **json.loads(os.environ.get("AWS_SDK_USER_AGENT", '{"user_agent_extra": null}')), 29 | ) 30 | 31 | 32 | def get_client(service_name, config=CLIENT_CONFIG): 33 | global _helpers_service_clients 34 | if service_name not in _helpers_service_clients: 35 | logger.debug(f"Initializing global boto3 client for {service_name}") 36 | _helpers_service_clients[service_name] = boto3.client(service_name, config=config) 37 | return _helpers_service_clients[service_name] 38 | 39 | 40 | def reset_client(): 41 | global _helpers_service_clients 42 | _helpers_service_clients = dict() 43 | 44 | 45 | # subclass JSONEncoder to be able to convert pipeline status to json 46 | class DateTimeEncoder(JSONEncoder): 47 | # Override the default method 48 | def default(self, obj): 49 | if isinstance(obj, (datetime.date, datetime.datetime)): 50 | return obj.isoformat() -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/lambdas/inference/tests/test_inference.py: -------------------------------------------------------------------------------- 1 | ################################################################################################################## 2 | # Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. # 3 | # # 4 | # Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance # 5 | # with the License. A copy of the License is located at # 6 | # # 7 | # http://www.apache.org/licenses/LICENSE-2.0 # 8 | # # 9 | # or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES # 10 | # OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions # 11 | # and limitations under the License. # 12 | # ##################################################################################################################### 13 | import os 14 | import json 15 | from datetime import datetime 16 | import unittest 17 | from unittest.mock import MagicMock, patch 18 | import pytest 19 | import botocore.session 20 | from botocore.stub import Stubber, ANY 21 | import boto3 22 | from shared.logger import get_logger 23 | from main import handler, invoke 24 | 25 | mock_env_variables = {"ENDPOINT_URI": "test/test", "SAGEMAKER_ENDPOINT_NAME": "test-endpoint"} 26 | 27 | 28 | @pytest.fixture 29 | def event(): 30 | return {"body": '{"payload": "test", "content_type": "text/csv"}'} 31 | 32 | 33 | @pytest.fixture 34 | def expected_response(): 35 | return { 36 | "statusCode": 200, 37 | "isBase64Encoded": False, 38 | "body": [1, 0, 1, 0], 39 | "headers": {"Content-Type": "plain/text"}, 40 | } 41 | 42 | 43 | @patch.dict(os.environ, mock_env_variables) 44 | def test_invoke(event): 45 | with patch("boto3.client") as mock_client: 46 | invoke(json.loads(event["body"]), "test", sm_client=mock_client) 47 | mock_client.invoke_endpoint.assert_called_with(EndpointName="test", Body="test", ContentType="text/csv") 48 | 49 | 50 | @patch("main.invoke") 51 | @patch("boto3.client") 52 | @patch.dict(os.environ, mock_env_variables) 53 | def test_handler(mocked_client, mocked_invoke, event, expected_response): 54 | mocked_invoke.return_value = expected_response 55 | response = handler(event, {}) 56 | assert response == expected_response 57 | -------------------------------------------------------------------------------- /source/lambdas/pipeline_orchestration/shared/logger.py: -------------------------------------------------------------------------------- 1 | # ##################################################################################################################### 2 | # Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. # 3 | # # 4 | # Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance # 5 | # with the License. A copy of the License is located at # 6 | # # 7 | # http://www.apache.org/licenses/LICENSE-2.0 # 8 | # # 9 | # or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES # 10 | # OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions # 11 | # and limitations under the License. # 12 | # ##################################################################################################################### 13 | import logging 14 | import os 15 | 16 | DEFAULT_LEVEL = "WARNING" 17 | 18 | 19 | def get_level(): 20 | """ 21 | Get the logging level from the LOG_LEVEL environment variable if it is valid. Otherwise set to WARNING 22 | :return: The logging level to use 23 | """ 24 | valid_levels = ["DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"] 25 | requested_level = os.environ.get("LOG_LEVEL", DEFAULT_LEVEL) 26 | if requested_level and requested_level in valid_levels: 27 | return requested_level 28 | return DEFAULT_LEVEL 29 | 30 | 31 | def get_logger(name): 32 | """ 33 | Get a configured logger. Compatible with both the AWS Lambda runtime (root logger) and local execution 34 | :param name: The name of the logger (most often __name__ of the calling module) 35 | :return: The logger to use 36 | """ 37 | logger = None 38 | # first case: running as a lambda function or in pytest with conftest 39 | # second case: running a single test or locally under test 40 | if len(logging.getLogger().handlers) > 0: 41 | logger = logging.getLogger() 42 | logger.setLevel(get_level()) 43 | # overrides 44 | logging.getLogger("boto3").setLevel(logging.WARNING) 45 | logging.getLogger("botocore").setLevel(logging.WARNING) 46 | logging.getLogger("urllib3").setLevel(logging.WARNING) 47 | else: 48 | logging.basicConfig(level=get_level()) # NOSONAR (python:S4792) 49 | logger = logging.getLogger(name) 50 | return logger 51 | -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/lambdas/invoke_lambda_custom_resource/index.py: -------------------------------------------------------------------------------- 1 | # ##################################################################################################################### 2 | # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. # 3 | # # 4 | # Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance # 5 | # with the License. A copy of the License is located at # 6 | # # 7 | # http://www.apache.org/licenses/LICENSE-2.0 # 8 | # # 9 | # or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES # 10 | # OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions # 11 | # and limitations under the License. # 12 | # ##################################################################################################################### 13 | import logging 14 | import uuid 15 | import json 16 | from crhelper import CfnResource 17 | from shared.helper import get_client 18 | 19 | logger = logging.getLogger(__name__) 20 | 21 | lambda_client = get_client("lambda") 22 | helper = CfnResource(json_logging=True, log_level="INFO") 23 | 24 | 25 | def handler(event, context): 26 | helper(event, context) 27 | 28 | 29 | @helper.update 30 | @helper.create 31 | def invoke_lambda(event, _, lm_client=lambda_client): 32 | try: 33 | logger.info(f"Event received: {event}") 34 | resource_properties = event["ResourceProperties"] 35 | resource = resource_properties["Resource"] 36 | if resource == "InvokeLambda": 37 | logger.info("Invoking lambda function is initiated...") 38 | resource_id = str(uuid.uuid4()) 39 | lm_client.invoke( 40 | FunctionName=resource_properties["function_name"], 41 | InvocationType="Event", 42 | Payload=json.dumps({"message": resource_properties["message"]}), 43 | ) 44 | helper.Data.update({"ResourceId": resource_id}) 45 | 46 | return resource_id 47 | 48 | else: 49 | raise ValueError(f"The Resource {resource} is unsupported by the Invoke Lambda custom resource.") 50 | 51 | except Exception as e: 52 | logger.error(f"Custom resource failed: {str(e)}") 53 | raise e 54 | 55 | 56 | @helper.delete 57 | def no_op(_, __): 58 | pass # No action is required when stack is deleted 59 | -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/lambdas/create_sagemaker_autopilot_job/tests/fixtures/autopilot_fixtures.py: -------------------------------------------------------------------------------- 1 | ####################################################################################################################### 2 | # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. # 3 | # # 4 | # Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance # 5 | # with the License. A copy of the License is located at # 6 | # # 7 | # http://www.apache.org/licenses/LICENSE-2.0 # 8 | # # 9 | # or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES # 10 | # OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions # 11 | # and limitations under the License. # 12 | # ##################################################################################################################### 13 | import os 14 | import json 15 | import pytest 16 | from unittest.mock import Mock 17 | 18 | 19 | @pytest.fixture() 20 | def mocked_autopilot_env_vars(monkeypatch): 21 | autopilot_env_vars = { 22 | "ASSETS_BUCKET": "testbucket", 23 | "JOB_NAME": "test-training-job", 24 | "ROLE_ARN": "test-role", 25 | "JOB_OUTPUT_LOCATION": "job_output", 26 | "PROBLEM_TYPE": "BinaryClassification", 27 | "JOB_OBJECTIVE": "auc", 28 | "TRAINING_DATA_KEY": "data/train/training-dataset.csv", 29 | "TARGET_ATTRIBUTE_NAME": "COLUMN_NAME", 30 | "MAX_CANDIDATES": "20", 31 | "TAGS": json.dumps([{"Key": "job-type", "Value": "autopilot"}]), 32 | } 33 | 34 | monkeypatch.setattr(os, "environ", autopilot_env_vars) 35 | 36 | 37 | @pytest.fixture() 38 | def mocked_automl_config(mocked_autopilot_env_vars): 39 | return dict( 40 | role=os.environ["ROLE_ARN"], 41 | target_attribute_name=os.environ["TARGET_ATTRIBUTE_NAME"], 42 | output_path=f"s3://{os.environ['ASSETS_BUCKET']}/{os.environ['JOB_OUTPUT_LOCATION']}", 43 | problem_type=os.environ["PROBLEM_TYPE"], 44 | max_candidates=int(os.environ["MAX_CANDIDATES"]), 45 | encrypt_inter_container_traffic=True, 46 | max_runtime_per_training_job_in_seconds=None, 47 | total_job_runtime_in_seconds=None, 48 | job_objective=os.environ["JOB_OBJECTIVE"], 49 | generate_candidate_definitions_only=False, 50 | sagemaker_session=Mock(), 51 | tags=json.loads(os.environ["TAGS"]), 52 | ) 53 | 54 | 55 | @pytest.fixture() 56 | def mocked_job_name(): 57 | return "test-autopilot-job" 58 | -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/pipeline_definitions/sagemaker_model.py: -------------------------------------------------------------------------------- 1 | # ##################################################################################################################### 2 | # Copyright 2021 Amazon.com, Inc. or its affiliates. All Rights Reserved. # 3 | # # 4 | # Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance # 5 | # with the License. A copy of the License is located at # 6 | # # 7 | # http://www.apache.org/licenses/LICENSE-2.0 # 8 | # # 9 | # or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES # 10 | # OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions # 11 | # and limitations under the License. # 12 | # ##################################################################################################################### 13 | from aws_cdk import Aws, Fn, aws_sagemaker as sagemaker 14 | 15 | 16 | def create_sagemaker_model( 17 | scope, # NOSONAR:S107 this function is designed to take many arguments 18 | id, 19 | execution_role, 20 | model_registry_provided, 21 | algorithm_image_uri, 22 | assets_bucket_name, 23 | model_artifact_location, 24 | model_package_name, 25 | model_name, 26 | **kwargs, 27 | ): 28 | # Create the model 29 | model = sagemaker.CfnModel( 30 | scope, 31 | id, 32 | execution_role_arn=execution_role.role_arn, 33 | # the primary container is set based on whether the SageMaker model registry is used or not 34 | # if model registry is used, the "modelPackageName" must be provided 35 | # else "image" and "modelDataUrl" must be provided 36 | # "image" and "modelDataUrl" will be ignored if "modelPackageName" is provided 37 | primary_container={ 38 | "image": Fn.condition_if( 39 | model_registry_provided.logical_id, Aws.NO_VALUE, algorithm_image_uri 40 | ).to_string(), 41 | "modelDataUrl": Fn.condition_if( 42 | model_registry_provided.logical_id, 43 | Aws.NO_VALUE, 44 | f"s3://{assets_bucket_name}/{model_artifact_location}", 45 | ).to_string(), 46 | "modelPackageName": Fn.condition_if( 47 | model_registry_provided.logical_id, model_package_name, Aws.NO_VALUE 48 | ).to_string(), 49 | }, 50 | tags=[{"key": "model_name", "value": model_name}], 51 | **kwargs, 52 | ) 53 | 54 | # add dependency on the Sagemaker execution role 55 | model.node.add_dependency(execution_role) 56 | 57 | return model 58 | -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/pipeline_definitions/source_actions.py: -------------------------------------------------------------------------------- 1 | # ##################################################################################################################### 2 | # Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. # 3 | # # 4 | # Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance # 5 | # with the License. A copy of the License is located at # 6 | # # 7 | # http://www.apache.org/licenses/LICENSE-2.0 # 8 | # # 9 | # or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES # 10 | # OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions # 11 | # and limitations under the License. # 12 | # ##################################################################################################################### 13 | from aws_cdk import ( 14 | aws_codepipeline as codepipeline, 15 | aws_codepipeline_actions as codepipeline_actions, 16 | ) 17 | 18 | 19 | def source_action_custom(assets_bucket, custom_container): 20 | """ 21 | source_action configures a codepipeline action with S3 as source 22 | 23 | :model_artifact_location: path to the model artifact in the S3 bucket: assets_bucket 24 | :assets_bucket: the bucket cdk object where pipeline assets are stored 25 | :custom_container: point to a zip file containing dockerfile and assets for building a custom model 26 | :return: codepipeline action in a form of a CDK object that can be attached to a codepipeline stage 27 | """ 28 | source_output = codepipeline.Artifact() 29 | return source_output, codepipeline_actions.S3SourceAction( 30 | action_name="S3Source", 31 | bucket=assets_bucket, 32 | bucket_key=custom_container.value_as_string, 33 | output=source_output, 34 | ) 35 | 36 | 37 | def source_action_template(template_location, assets_bucket): 38 | """ 39 | source_action_model_monitor configures a codepipeline action with S3 as source 40 | 41 | :template_location: path to the zip file containg the CF template and stages configuration in the S3 bucket: assets_bucket 42 | :assets_bucket: the bucket cdk object where pipeline assets are stored 43 | :return: codepipeline action in a form of a CDK object that can be attached to a codepipeline stage 44 | """ 45 | source_output = codepipeline.Artifact() 46 | return source_output, codepipeline_actions.S3SourceAction( 47 | action_name="S3Source", 48 | bucket=assets_bucket, 49 | bucket_key=template_location.value_as_string, 50 | output=source_output, 51 | ) 52 | -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/lambdas/invoke_lambda_custom_resource/tests/test_invoke_lambda.py: -------------------------------------------------------------------------------- 1 | # ##################################################################################################################### 2 | # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. # 3 | # # 4 | # Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance # 5 | # with the License. A copy of the License is located at # 6 | # # 7 | # http://www.apache.org/licenses/LICENSE-2.0 # 8 | # # 9 | # or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES # 10 | # OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions # 11 | # and limitations under the License. # 12 | # ##################################################################################################################### 13 | import boto3 14 | import pytest 15 | from unittest.mock import patch 16 | from moto import mock_aws 17 | from index import invoke_lambda, no_op, handler 18 | 19 | 20 | @pytest.fixture() 21 | def invoke_event(): 22 | return { 23 | "RequestType": "Create", 24 | "ResourceProperties": { 25 | "Resource": "InvokeLambda", 26 | "function_name": "myfunction", 27 | "message": "Start batch transform job", 28 | }, 29 | } 30 | 31 | 32 | @pytest.fixture() 33 | def invoke_bad_event(): 34 | return { 35 | "RequestType": "Create", 36 | "ResourceProperties": { 37 | "Resource": "NotSupported", 38 | }, 39 | } 40 | 41 | 42 | @patch("boto3.client") 43 | def test_invoke_lambda(mocked_client, invoke_event, invoke_bad_event): 44 | response = invoke_lambda(invoke_event, None, mocked_client) 45 | assert response is not None 46 | # unsupported 47 | with pytest.raises(Exception) as error: 48 | invoke_lambda(invoke_bad_event, None, mocked_client) 49 | assert str(error.value) == ( 50 | f"The Resource {invoke_bad_event['ResourceProperties']['Resource']} " 51 | f"is unsupported by the Invoke Lambda custom resource." 52 | ) 53 | 54 | 55 | @mock_aws 56 | def test_invoke_lambda_error(invoke_event): 57 | mocked_client = boto3.client("lambda") 58 | with pytest.raises(Exception): 59 | invoke_lambda(invoke_event, None, mocked_client) 60 | 61 | 62 | @patch("index.invoke_lambda") 63 | def test_no_op(mocked_invoke, invoke_event): 64 | response = no_op(invoke_event, {}) 65 | assert response is None 66 | mocked_invoke.assert_not_called() 67 | 68 | 69 | @patch("index.helper") 70 | def test_handler(mocked_helper, invoke_event): 71 | handler(invoke_event, {}) 72 | mocked_helper.assert_called_with(invoke_event, {}) 73 | -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/lambdas/batch_transform/main.py: -------------------------------------------------------------------------------- 1 | # ##################################################################################################################### 2 | # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. # 3 | # # 4 | # Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance # 5 | # with the License. A copy of the License is located at # 6 | # # 7 | # http://www.apache.org/licenses/LICENSE-2.0 # 8 | # # 9 | # or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES # 10 | # OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions # 11 | # and limitations under the License. # 12 | # ##################################################################################################################### 13 | import os 14 | import uuid 15 | from shared.logger import get_logger 16 | from shared.helper import get_client 17 | 18 | logger = get_logger(__name__) 19 | sm_client = get_client("sagemaker") 20 | 21 | 22 | def handler(*_): 23 | try: 24 | model_name = os.environ.get("model_name").lower() 25 | batch_inference_data = os.environ.get("batch_inference_data") 26 | batch_job_output_location = os.environ.get("batch_job_output_location") 27 | inference_instance = os.environ.get("inference_instance") 28 | kms_key_arn = os.environ.get("kms_key_arn") 29 | batch_job_name = f"{model_name}-batch-transform-{str(uuid.uuid4())[:8]}" 30 | 31 | request = { 32 | "TransformJobName": batch_job_name, 33 | "ModelName": model_name, 34 | "TransformOutput": { 35 | "S3OutputPath": f"s3://{batch_job_output_location}", 36 | "Accept": "text/csv", 37 | "AssembleWith": "Line", 38 | }, 39 | "TransformInput": { 40 | "DataSource": {"S3DataSource": {"S3DataType": "S3Prefix", "S3Uri": f"s3://{batch_inference_data}"}}, 41 | "ContentType": "text/csv", 42 | "SplitType": "Line", 43 | "CompressionType": "None", 44 | }, 45 | "TransformResources": {"InstanceType": inference_instance, "InstanceCount": 1}, 46 | } 47 | # add KmsKey if provided by the customer 48 | if kms_key_arn: 49 | request["TransformOutput"].update({"KmsKeyId": kms_key_arn}) 50 | request["TransformResources"].update({"VolumeKmsKeyId": kms_key_arn}) 51 | 52 | response = sm_client.create_transform_job(**request) 53 | logger.info(f"Response from create transform job request. response: {response}") 54 | logger.info(f"Created Transform job with name: {batch_job_name}") 55 | 56 | except Exception as e: 57 | logger.error(f"Error creating the batch transform job {batch_job_name}: {str(e)}") 58 | raise e 59 | -------------------------------------------------------------------------------- /source/lambdas/custom_resource/index.py: -------------------------------------------------------------------------------- 1 | # ##################################################################################################################### 2 | # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. # 3 | # # 4 | # Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance # 5 | # with the License. A copy of the License is located at # 6 | # # 7 | # http://www.apache.org/licenses/LICENSE-2.0 # 8 | # # 9 | # or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES # 10 | # OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions # 11 | # and limitations under the License. # 12 | # ##################################################################################################################### 13 | import os 14 | import sys 15 | import shutil 16 | import tempfile 17 | import logging 18 | import traceback 19 | import boto3 20 | from crhelper import CfnResource 21 | 22 | 23 | logger = logging.getLogger(__name__) 24 | 25 | s3_client = boto3.client("s3") 26 | helper = CfnResource(json_logging=True, log_level="INFO") 27 | 28 | 29 | def copy_assets_to_s3(s3_client): 30 | # get the source/destination bukcets and file key 31 | s3_bucket_name = os.environ.get("SOURCE_BUCKET") 32 | bucket = os.environ.get("DESTINATION_BUCKET") 33 | file_key = os.environ.get("FILE_KEY") 34 | base_dir = "blueprints" 35 | 36 | # create a tmpdir for the zip file to downlaod 37 | zip_tmpdir = tempfile.mkdtemp() 38 | zip_file_path = os.path.join(zip_tmpdir, f"{base_dir}.zip") 39 | 40 | # download blueprints.zip 41 | s3_client.download_file(s3_bucket_name, file_key, zip_file_path) 42 | 43 | # unpack the zip file in another tmp directory 44 | unpack_tmpdir = tempfile.mkdtemp() 45 | shutil.unpack_archive(zip_file_path, unpack_tmpdir, "zip") 46 | 47 | # construct the path to the unpacked file 48 | local_directory = os.path.join(unpack_tmpdir, base_dir) 49 | 50 | # enumerate local files recursively 51 | for root, dirs, files in os.walk(local_directory): 52 | 53 | for filename in files: 54 | 55 | # construct the full local path 56 | local_path = os.path.join(root, filename) 57 | 58 | # construct the full s3 path 59 | relative_path = os.path.relpath(local_path, local_directory) 60 | s3_path = os.path.join(base_dir, relative_path) 61 | logger.info(f"Uploading {s3_path}...") 62 | s3_client.upload_file(local_path, bucket, s3_path) 63 | 64 | return "CopyAssets-" + bucket 65 | 66 | 67 | def on_event(event, context): 68 | helper(event, context) 69 | 70 | 71 | @helper.create 72 | @helper.update 73 | def custom_resource(event, _): 74 | 75 | try: 76 | resource_id = copy_assets_to_s3(s3_client) 77 | return resource_id 78 | 79 | except Exception as e: 80 | exc_type, exc_value, exc_tb = sys.exc_info() 81 | logger.error(traceback.format_exception(exc_type, exc_value, exc_tb)) 82 | raise e 83 | 84 | 85 | @helper.delete 86 | def no_op(_, __): 87 | pass # No action is required when stack is deleted 88 | -------------------------------------------------------------------------------- /source/lambdas/custom_resource/tests/test_custom_resource.py: -------------------------------------------------------------------------------- 1 | # ##################################################################################################################### 2 | # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. # 3 | # # 4 | # Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance # 5 | # with the License. A copy of the License is located at # 6 | # # 7 | # http://www.apache.org/licenses/LICENSE-2.0 # 8 | # # 9 | # or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES # 10 | # OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions # 11 | # and limitations under the License. # 12 | # ##################################################################################################################### 13 | import os 14 | import boto3 15 | import tempfile 16 | import pytest 17 | from unittest.mock import patch 18 | from moto import mock_aws 19 | 20 | from index import copy_assets_to_s3, on_event, custom_resource, no_op 21 | 22 | 23 | @pytest.fixture(autouse=True) 24 | def mock_env_variables(): 25 | os.environ["SOURCE_BUCKET"] = "solutions-bucket" 26 | os.environ["DESTINATION_BUCKET"] = "blueprints-bucket" 27 | os.environ["FILE_KEY"] = "blueprints.zip" 28 | 29 | 30 | @pytest.fixture 31 | def event(): 32 | return {"bucket": os.environ["SOURCE_BUCKET"]} 33 | 34 | 35 | @pytest.fixture 36 | def mocked_response(): 37 | return f"CopyAssets-{os.environ['DESTINATION_BUCKET']}" 38 | 39 | 40 | @mock_aws 41 | @patch("index.os.walk") 42 | @patch("index.shutil.unpack_archive") 43 | def test_copy_assets_to_s3(mocked_shutil, mocked_walk, mocked_response): 44 | s3_client = boto3.client("s3", region_name="us-east-1") 45 | testfile = tempfile.NamedTemporaryFile() 46 | s3_client.create_bucket(Bucket="solutions-bucket") 47 | s3_client.create_bucket(Bucket="blueprints-bucket") 48 | s3_client.upload_file(testfile.name, os.environ["SOURCE_BUCKET"], os.environ["FILE_KEY"]) 49 | local_file = tempfile.NamedTemporaryFile() 50 | s3_client.download_file(os.environ["SOURCE_BUCKET"], os.environ["FILE_KEY"], local_file.name) 51 | tmp = tempfile.mkdtemp() 52 | mocked_walk.return_value = [ 53 | (tmp, (local_file.name,), (local_file.name,)), 54 | ] 55 | 56 | assert copy_assets_to_s3(s3_client) == mocked_response 57 | 58 | 59 | @patch("index.custom_resource") 60 | def test_no_op(mocked_custom, event): 61 | response = no_op(event, {}) 62 | assert response is None 63 | mocked_custom.assert_not_called() 64 | 65 | 66 | @patch("index.helper") 67 | def test_on_event(mocked_helper, event): 68 | on_event(event, {}) 69 | mocked_helper.assert_called_with(event, {}) 70 | 71 | 72 | @patch("index.copy_assets_to_s3") 73 | def test_custom_resource(mocked_copy, event, mocked_response): 74 | # assert expected response 75 | mocked_copy.return_value = mocked_response 76 | respone = custom_resource(event, {}) 77 | assert respone == mocked_response 78 | # assert for error 79 | mocked_copy.side_effect = Exception("mocked error") 80 | with pytest.raises(Exception): 81 | custom_resource(event, {}) 82 | -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/pipeline_definitions/sagemaker_endpoint_config.py: -------------------------------------------------------------------------------- 1 | # ##################################################################################################################### 2 | # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. # 3 | # # 4 | # Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance # 5 | # with the License. A copy of the License is located at # 6 | # # 7 | # http://www.apache.org/licenses/LICENSE-2.0 # 8 | # # 9 | # or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES # 10 | # OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions # 11 | # and limitations under the License. # 12 | # ##################################################################################################################### 13 | from aws_cdk import ( 14 | aws_sagemaker as sagemaker, 15 | ) 16 | 17 | 18 | def create_sagemaker_endpoint_config( 19 | scope, # NOSONAR:S107 this function is designed to take many arguments 20 | id, 21 | sagemaker_model_name, 22 | model_name, 23 | inference_instance, 24 | data_capture_location, 25 | kms_key_arn, 26 | **kwargs, 27 | ): 28 | # Create the sagemaker endpoint config 29 | sagemaker_endpoint_config = sagemaker.CfnEndpointConfig( 30 | scope, 31 | id, 32 | production_variants=[ 33 | { 34 | "variantName": "AllTraffic", 35 | "modelName": sagemaker_model_name, 36 | "initialVariantWeight": 1, 37 | "initialInstanceCount": 1, 38 | "instanceType": inference_instance, 39 | } 40 | ], 41 | data_capture_config={ 42 | "enableCapture": True, 43 | "initialSamplingPercentage": 100, 44 | "destinationS3Uri": f"s3://{data_capture_location}", 45 | "captureOptions": [{"captureMode": "Output"}, {"captureMode": "Input"}], 46 | "captureContentTypeHeader": {"csvContentTypes": ["text/csv"]}, 47 | # The key specified here is used to encrypt data on S3 captured by the endpoint. If you don't provide 48 | # a KMS key ID, Amazon SageMaker uses the default KMS key for Amazon S3 for your role's account. 49 | # for more info see DataCaptureConfig 50 | # https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-sagemaker-endpointconfig.html 51 | "kmsKeyId": kms_key_arn, 52 | }, 53 | # The key specified here is used to encrypt data on the storage volume attached to the 54 | # ML compute instance that hosts the endpoint. Note: a key can not be specified here when 55 | # using an instance type with local storage (e.g. certain Nitro-based instances) 56 | # for more info see the KmsKeyId doc at 57 | # https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-sagemaker-endpointconfig.html 58 | kms_key_id=kms_key_arn, 59 | tags=[{"key": "endpoint-config-name", "value": f"{model_name}-endpoint-config"}], 60 | **kwargs, 61 | ) 62 | 63 | return sagemaker_endpoint_config 64 | -------------------------------------------------------------------------------- /source/lambdas/pipeline_orchestration/shared/wrappers.py: -------------------------------------------------------------------------------- 1 | # ##################################################################################################################### 2 | # Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. # 3 | # # 4 | # Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance # 5 | # with the License. A copy of the License is located at # 6 | # # 7 | # http://www.apache.org/licenses/LICENSE-2.0 # 8 | # # 9 | # or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES # 10 | # OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions # 11 | # and limitations under the License. # 12 | # ##################################################################################################################### 13 | import json 14 | import sys 15 | import os 16 | from typing import Any, Callable 17 | import traceback 18 | from functools import wraps 19 | import botocore 20 | from shared.logger import get_logger 21 | 22 | logger = get_logger(__name__) 23 | endable_detailed_error_message = os.getenv("ALLOW_DETAILED_ERROR_MESSAGE", "Yes") 24 | 25 | 26 | class BadRequest(Exception): 27 | pass 28 | 29 | 30 | def handle_exception(error_description, error_object, status_code): 31 | # log the error 32 | logger.error(f"{error_description}. Error: {str(error_object)}") 33 | exc_type, exc_value, exc_tb = sys.exc_info() 34 | logger.error(traceback.format_exception(exc_type, exc_value, exc_tb)) 35 | # update the response body 36 | body = {"message": error_description} 37 | if endable_detailed_error_message == "Yes": 38 | body.update({"detailedMessage": str(error_object)}) 39 | 40 | return { 41 | "statusCode": status_code, 42 | "isBase64Encoded": False, 43 | "body": json.dumps(body), 44 | "headers": {"Content-Type": "plain/text"}, 45 | } 46 | 47 | 48 | def api_exception_handler(f): 49 | @wraps(f) 50 | def wrapper(event, context): 51 | try: 52 | return f(event, context) 53 | 54 | except BadRequest as bad_request_error: 55 | return handle_exception("A BadRequest exception occurred", bad_request_error, 400) 56 | 57 | except botocore.exceptions.ClientError as client_error: 58 | status_code = client_error.response["ResponseMetadata"]["HTTPStatusCode"] 59 | return handle_exception("A boto3 ClientError occurred", client_error, status_code) 60 | 61 | except Exception as e: 62 | return handle_exception("An Unexpected Server side exception occurred", e, 500) 63 | 64 | return wrapper 65 | 66 | 67 | def exception_handler(func: Callable[..., Any]) -> Any: 68 | """ 69 | Docorator function to handle exceptions 70 | 71 | Args: 72 | func (object): function to be decorated 73 | 74 | Returns: 75 | func's return value 76 | 77 | Raises: 78 | Exception thrown by the decorated function 79 | """ 80 | 81 | def wrapper_function(*args, **kwargs): 82 | try: 83 | return func(*args, **kwargs) 84 | 85 | except Exception as e: 86 | logger.error(f"Error in {func.__name__}: {str(e)}") 87 | raise e 88 | 89 | return wrapper_function 90 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing Guidelines 2 | 3 | Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional 4 | documentation, we greatly value feedback and contributions from our community. 5 | 6 | Please read through this document before submitting any issues or pull requests to ensure we have all the necessary 7 | information to effectively respond to your bug report or contribution. 8 | 9 | ## Reporting Bugs/Feature Requests 10 | 11 | We welcome you to use the GitHub issue tracker to report bugs or suggest features. 12 | 13 | When filing an issue, please check [existing open](https://github.com/aws-solutions/mlops-workload-orchestrator/issues), or [recently closed](https://github.com/aws-solutions/mlops-workload-orchestrator/issues?utf8=%E2%9C%93&q=is%3Aissue%20is%3Aclosed%20), issues to make sure somebody else hasn't already 14 | reported the issue. Please try to include as much information as you can. Details like these are incredibly useful: 15 | 16 | - A reproducible test case or series of steps 17 | - The version of our code being used 18 | - Any modifications you've made relevant to the bug 19 | - Anything unusual about your environment or deployment 20 | 21 | ## Contributing via Pull Requests 22 | 23 | Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that: 24 | 25 | 1. You are working against the latest source on the _main_ branch. 26 | 2. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already. 27 | 3. You open an issue to discuss any significant work - we would hate for your time to be wasted. 28 | 29 | To send us a pull request, please: 30 | 31 | 1. Fork the repository. 32 | 2. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change. 33 | 3. Ensure all build processes execute successfully (see README.md for additional guidance). 34 | 4. Ensure all unit, integration, and/or snapshot tests pass, as applicable. 35 | 5. Commit to your fork using clear commit messages. 36 | 6. Send us a pull request, answering any default questions in the pull request interface. 37 | 7. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation. 38 | 39 | GitHub provides additional document on [forking a repository](https://help.github.com/articles/fork-a-repo/) and 40 | [creating a pull request](https://help.github.com/articles/creating-a-pull-request/). 41 | 42 | ## Finding contributions to work on 43 | 44 | Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels ((enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any ['help wanted'](https://github.com/aws-solutions/mlops-workload-orchestrator/labels/help%20wanted) issues is a great place to start. 45 | 46 | ## Code of Conduct 47 | 48 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 49 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 50 | opensource-codeofconduct@amazon.com with any additional questions or comments. 51 | 52 | ## Security issue notifications 53 | 54 | If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public GitHub issue. 55 | 56 | ## Licensing 57 | 58 | See the [LICENSE](https://github.com/aws-solutions/mlops-workload-orchestrator/blob/main/LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution. 59 | 60 | We may ask you to sign a [Contributor License Agreement (CLA)](http://en.wikipedia.org/wiki/Contributor_License_Agreement) for larger changes. 61 | -------------------------------------------------------------------------------- /deployment/cdk-solution-helper/index.js: -------------------------------------------------------------------------------- 1 | /** 2 | * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | * 4 | * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance 5 | * with the License. A copy of the License is located at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES 10 | * OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions 11 | * and limitations under the License. 12 | */ 13 | 14 | // Imports 15 | const fs = require("fs"); 16 | 17 | // Paths 18 | const global_s3_assets = '../global-s3-assets'; 19 | 20 | //this regular express also takes into account lambda functions defined in nested stacks 21 | const _regex = /[\w]*AssetParameters/g; // NOSONAR: this regex is used only to clean the CloudFormation tempaltes 22 | 23 | // For each template in global_s3_assets ... 24 | fs.readdirSync(global_s3_assets).forEach((file) => { // NOSONAR: acceptable Cognitive Complexity 25 | // Import and parse template file 26 | const raw_template = fs.readFileSync(`${global_s3_assets}/${file}`); 27 | let template = JSON.parse(raw_template); 28 | 29 | // Clean-up Lambda function code dependencies 30 | const resources = template.Resources ? template.Resources : {}; 31 | const lambdaFunctions = Object.keys(resources).filter(function (key) { 32 | return resources[key].Type === "AWS::Lambda::Function"; 33 | }); 34 | 35 | lambdaFunctions.forEach(function (f) { 36 | const fn = template.Resources[f]; 37 | let prop; 38 | if (fn.Properties.hasOwnProperty("Code")) { 39 | prop = fn.Properties.Code; 40 | } else if (fn.Properties.hasOwnProperty("Content")) { 41 | prop = fn.Properties.Content; 42 | } 43 | 44 | if (prop.hasOwnProperty("S3Bucket")) { 45 | // Set the S3 key reference 46 | let artifactHash = Object.assign(prop.S3Key); 47 | const assetPath = `asset${artifactHash}`; 48 | prop.S3Key = `%%SOLUTION_NAME%%/%%VERSION%%/${assetPath}`; 49 | 50 | // Set the S3 bucket reference 51 | prop.S3Bucket = { 52 | "Fn::Sub": "%%BUCKET_NAME%%-${AWS::Region}", 53 | }; 54 | } else { 55 | console.warn(`No S3Bucket Property found for ${JSON.stringify(prop)}`); 56 | } 57 | }); 58 | 59 | // Clean-up nested template stack dependencies 60 | const nestedStacks = Object.keys(resources).filter(function (key) { 61 | return resources[key].Type === "AWS::CloudFormation::Stack"; 62 | }); 63 | 64 | nestedStacks.forEach(function (f) { 65 | const fn = template.Resources[f]; 66 | if (!fn.Metadata.hasOwnProperty("aws:asset:path")) { 67 | throw new Error("Nested stack construct missing file name metadata"); 68 | } 69 | fn.Properties.TemplateURL = { 70 | "Fn::Join": [ 71 | "", 72 | [ 73 | "https://%%TEMPLATE_BUCKET_NAME%%.s3.", 74 | { 75 | Ref: "AWS::URLSuffix", 76 | }, 77 | "/", 78 | `%%SOLUTION_NAME%%/%%VERSION%%/${fn.Metadata["aws:asset:path"].slice(0, -".json".length)}`, 79 | ], 80 | ], 81 | }; 82 | 83 | const params = fn.Properties.Parameters ? fn.Properties.Parameters : {}; 84 | const nestedStackParameters = Object.keys(params).filter(function (key) { 85 | return key.search(_regex) > -1; 86 | }); 87 | 88 | nestedStackParameters.forEach(function (stkParam) { 89 | fn.Properties.Parameters[stkParam] = undefined; 90 | }); 91 | }); 92 | 93 | // Clean-up parameters section 94 | const parameters = template.Parameters ? template.Parameters : {}; 95 | const assetParameters = Object.keys(parameters).filter(function (key) { 96 | return key.search(_regex) > -1; 97 | }); 98 | assetParameters.forEach(function (a) { 99 | template.Parameters[a] = undefined; 100 | }); 101 | 102 | // Output modified template file 103 | const output_template = JSON.stringify(template, null, 2); 104 | fs.writeFileSync(`${global_s3_assets}/${file}`, output_template); 105 | }); -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/lambdas/batch_transform/tests/test_batch_transform.py: -------------------------------------------------------------------------------- 1 | ####################################################################################################################### 2 | # Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. # 3 | # # 4 | # Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance # 5 | # with the License. A copy of the License is located at # 6 | # # 7 | # http://www.apache.org/licenses/LICENSE-2.0 # 8 | # # 9 | # or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES # 10 | # OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions # 11 | # and limitations under the License. # 12 | # ##################################################################################################################### 13 | import os 14 | from unittest.mock import MagicMock, patch 15 | import pytest 16 | from moto import mock_aws 17 | import botocore.session 18 | from botocore.stub import Stubber, ANY 19 | from main import handler 20 | from shared.helper import get_client, reset_client 21 | 22 | 23 | @pytest.fixture(autouse=True) 24 | def mock_env_variables(): 25 | new_env = { 26 | "model_name": "test", 27 | "assets_bucket": "testbucket", 28 | "batch_inference_data": "test", 29 | "inference_instance": "ml.m5.4xlarge", 30 | "batch_job_output_location": "output-location", 31 | "kms_key_arn": "mykey", 32 | } 33 | os.environ = {**os.environ, **new_env} 34 | 35 | 36 | @pytest.fixture 37 | def sm_expected_params(): 38 | return { 39 | "TransformJobName": ANY, 40 | "ModelName": "test", 41 | "TransformOutput": {"S3OutputPath": ANY, "Accept": "text/csv", "AssembleWith": "Line", "KmsKeyId": "mykey"}, 42 | "TransformInput": { 43 | "DataSource": {"S3DataSource": {"S3DataType": "S3Prefix", "S3Uri": ANY}}, 44 | "ContentType": "text/csv", 45 | "SplitType": "Line", 46 | "CompressionType": "None", 47 | }, 48 | "TransformResources": {"InstanceType": ANY, "InstanceCount": 1, "VolumeKmsKeyId": "mykey"}, 49 | } 50 | 51 | 52 | @pytest.fixture 53 | def sm_response_200(): 54 | return { 55 | "ResponseMetadata": {"HTTPStatusCode": 200}, 56 | "TransformJobArn": "arn:aws:sagemaker:region:account:transform-job/name", 57 | } 58 | 59 | 60 | @pytest.fixture 61 | def sm_response_500(): 62 | return { 63 | "ResponseMetadata": {"HTTPStatusCode": 500}, 64 | "TransformJobArn": "arn:aws:sagemaker:region:account:transform-job/name", 65 | } 66 | 67 | 68 | @pytest.fixture() 69 | def event(): 70 | return { 71 | "CodePipeline.job": {"id": "test_job_id"}, 72 | } 73 | 74 | 75 | @mock_aws 76 | def test_handler_success(sm_expected_params, sm_response_200, event): 77 | sm_client = get_client("sagemaker") 78 | sm_stubber = Stubber(sm_client) 79 | 80 | # success path 81 | sm_stubber.add_response("create_transform_job", sm_response_200, sm_expected_params) 82 | 83 | with sm_stubber: 84 | handler(event, {}) 85 | reset_client() 86 | 87 | 88 | def test_handler_fail(sm_expected_params, sm_response_500, event): 89 | sm_client = get_client("sagemaker") 90 | sm_stubber = Stubber(sm_client) 91 | 92 | # fail path 93 | sm_stubber.add_response("create_transform_job", sm_response_500, sm_expected_params) 94 | 95 | with pytest.raises(Exception): 96 | handler(event, {}) 97 | 98 | reset_client() 99 | -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/lambdas/create_sagemaker_autopilot_job/main.py: -------------------------------------------------------------------------------- 1 | # ##################################################################################################################### 2 | # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. # 3 | # # 4 | # Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance # 5 | # with the License. A copy of the License is located at # 6 | # # 7 | # http://www.apache.org/licenses/LICENSE-2.0 # 8 | # # 9 | # or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES # 10 | # OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions # 11 | # and limitations under the License. # 12 | # ##################################################################################################################### 13 | import os 14 | import json 15 | from sagemaker import AutoML 16 | from sagemaker import Session 17 | from shared.wrappers import exception_handler 18 | from shared.logger import get_logger 19 | from shared.helper import get_client 20 | 21 | logger = get_logger(__name__) 22 | 23 | # get the environment variables 24 | assets_bucket = os.environ["ASSETS_BUCKET"] 25 | job_name = os.environ["JOB_NAME"] 26 | 27 | # get the estimator config 28 | role_arn = os.environ["ROLE_ARN"] 29 | job_output_location = os.environ["JOB_OUTPUT_LOCATION"] 30 | output_path = f"s3://{assets_bucket}/{job_output_location}" 31 | 32 | # if problem_type is provided, job_objective must be provided and vice versa 33 | problem_type = os.environ.get("PROBLEM_TYPE") 34 | job_objective = {"MetricName": os.environ.get("JOB_OBJECTIVE")} if os.environ.get("JOB_OBJECTIVE") else None 35 | 36 | # get the training data config 37 | training_dataset_key = os.environ["TRAINING_DATA_KEY"] 38 | training_data_s3_uri = f"s3://{assets_bucket}/{training_dataset_key}" 39 | compression_type = os.environ.get("COMPRESSION_TYPE") 40 | # target attribute name in the training dataset 41 | target_attribute_name = os.environ.get("TARGET_ATTRIBUTE_NAME") 42 | max_candidates = int(os.environ.get("MAX_CANDIDATES", "10")) 43 | kms_key_arn = os.environ.get("KMS_KEY_ARN") 44 | # encrypt inter container traffic 45 | encrypt_inter_container_traffic_str = os.environ.get("ENCRYPT_INTER_CONTAINER_TRAFFIC", "True") 46 | encrypt_inter_container_traffic = False if encrypt_inter_container_traffic_str == "False" else True 47 | max_runtime_per_training_job_in_seconds = ( 48 | int(os.environ.get("MAX_RUNTIME_PER_JOB")) if os.environ.get("MAX_RUNTIME_PER_JOB") else None 49 | ) 50 | total_job_runtime_in_seconds = int(os.environ.get("TOTAL_JOB_RUNTIME")) if os.environ.get("TOTAL_JOB_RUNTIME") else None 51 | generate_candidate_definitions_only_str = os.environ.get("GENERATE_CANDIDATE_DEFINITIONS_ONLY", "False") 52 | generate_candidate_definitions_only = True if generate_candidate_definitions_only_str == "True" else False 53 | 54 | 55 | tags = json.loads(os.environ.get("TAGS")) if os.environ.get("TAGS") else None 56 | 57 | 58 | @exception_handler 59 | def handler(event, context): 60 | # sm client 61 | sm_client = get_client("sagemaker") 62 | 63 | # create the SageMaker Autopilot job 64 | autopilot_job = AutoML( 65 | role=role_arn, 66 | target_attribute_name=target_attribute_name, 67 | output_path=output_path, 68 | problem_type=problem_type, 69 | max_candidates=max_candidates, 70 | encrypt_inter_container_traffic=encrypt_inter_container_traffic, 71 | max_runtime_per_training_job_in_seconds=max_runtime_per_training_job_in_seconds, 72 | total_job_runtime_in_seconds=total_job_runtime_in_seconds, 73 | job_objective=job_objective, 74 | generate_candidate_definitions_only=generate_candidate_definitions_only, 75 | sagemaker_session=Session(sagemaker_client=sm_client), 76 | tags=tags, 77 | ) 78 | 79 | # start the autopilot job 80 | autopilot_job.fit(job_name=job_name, inputs=training_data_s3_uri, wait=False, logs=False) 81 | logger.info(f"Autopilot job {job_name} started...") 82 | logger.info(autopilot_job.describe_auto_ml_job(job_name)) 83 | -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/lambdas/create_model_training_job/model_training_helper.py: -------------------------------------------------------------------------------- 1 | # ##################################################################################################################### 2 | # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. # 3 | # # 4 | # Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance # 5 | # with the License. A copy of the License is located at # 6 | # # 7 | # http://www.apache.org/licenses/LICENSE-2.0 # 8 | # # 9 | # or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES # 10 | # OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions # 11 | # and limitations under the License. # 12 | # ##################################################################################################################### 13 | from sagemaker.estimator import Estimator 14 | from sagemaker.tuner import HyperparameterTuner 15 | from sagemaker.inputs import TrainingInput 16 | from sagemaker.parameter import ParameterRange 17 | from sagemaker.tuner import ContinuousParameter, IntegerParameter, CategoricalParameter 18 | from shared.wrappers import exception_handler 19 | from typing import Dict, Any, List 20 | from enum import Enum 21 | 22 | 23 | class TrainingType(Enum): 24 | TrainingJob = 1 25 | HyperparameterTuningJob = 2 26 | 27 | 28 | class SolutionModelTraining: 29 | def __init__( 30 | self, 31 | job_name: str, 32 | estimator_config: Dict[str, Any], 33 | hyperparameters: Dict[str, Any], 34 | data_channels: Dict[str, TrainingInput], 35 | job_type: TrainingType = TrainingType.TrainingJob, 36 | hyperparameter_tuner_config: Dict[str, Any] = None, 37 | hyperparameter_ranges: Dict[str, ParameterRange] = None, 38 | ) -> None: 39 | self.job_name = job_name 40 | self.estimator_config = estimator_config 41 | self.hyperparameters = hyperparameters 42 | self.data_channels = data_channels 43 | self.job_type = job_type 44 | self.hyperparameter_tuner_config = hyperparameter_tuner_config 45 | self.hyperparameter_ranges = hyperparameter_ranges 46 | 47 | @exception_handler 48 | def create_training_job(self): 49 | # create Training Job type MonitoringType->function_name map 50 | type_function_map = dict( 51 | TrainingJob="_create_estimator", 52 | HyperparameterTuningJob="_create_hyperparameter_tuner", 53 | ) 54 | 55 | # call the right function to create the training/hyperparameter tunning Job 56 | job = getattr(self, type_function_map[self.job_type.name])() 57 | # start the training/hyperparameters tuning job 58 | job.fit(job_name=self.job_name, inputs=self.data_channels, wait=False) 59 | 60 | @exception_handler 61 | def _create_estimator(self): 62 | # create the estimator 63 | estimator = Estimator(**self.estimator_config) 64 | # set its hyperparameters 65 | estimator.set_hyperparameters(**self.hyperparameters) 66 | # return the estimator to the caller 67 | return estimator 68 | 69 | @exception_handler 70 | def _create_hyperparameter_tuner(self): 71 | # create the estimator 72 | estimator = self._create_estimator() 73 | # configure the hyperparameters tuning job 74 | hyperparameter_tuner = HyperparameterTuner( 75 | estimator=estimator, hyperparameter_ranges=self.hyperparameter_ranges, **self.hyperparameter_tuner_config 76 | ) 77 | # return the hyperparameter tuner 78 | return hyperparameter_tuner 79 | 80 | @staticmethod 81 | @exception_handler 82 | def format_search_grid(hyperparameter_ranges: Dict[str, List]) -> Dict[str, ParameterRange]: 83 | parameter_type_map = dict( 84 | integer=IntegerParameter, continuous=ContinuousParameter, categorical=CategoricalParameter 85 | ) 86 | search_grid = { 87 | k: (parameter_type_map[v[0]](*v[1]) if v[0] != "categorical" else parameter_type_map[v[0]](v[1])) 88 | for k, v in hyperparameter_ranges.items() 89 | } 90 | return search_grid 91 | -------------------------------------------------------------------------------- /source/lambdas/solution_helper/lambda_function.py: -------------------------------------------------------------------------------- 1 | ###################################################################################################################### 2 | # Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. # 3 | # # 4 | # Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance # 5 | # with the License. A copy of the License is located at # 6 | # # 7 | # http://www.apache.org/licenses/LICENSE-2.0 # 8 | # # 9 | # or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES # 10 | # OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions # 11 | # and limitations under the License. # 12 | ###################################################################################################################### 13 | 14 | import logging, uuid, requests 15 | from copy import copy 16 | from crhelper import CfnResource 17 | from datetime import datetime 18 | 19 | logger = logging.getLogger(__name__) 20 | helper = CfnResource(json_logging=True, log_level="INFO") 21 | 22 | # requests.post timeout in seconds 23 | REQUST_TIMEOUT = 60 24 | 25 | 26 | def _sanitize_data(resource_properties): 27 | # Define allowed keys. You need to update this list with new metrics 28 | main_keys = [ 29 | "bucketSelected", 30 | "configBucketProvided", 31 | "ecrProvided", 32 | "Region", 33 | "IsMultiAccount", 34 | "UseModelRegistry", 35 | "createModelPackageGroup", 36 | "allowDetailedErrorMessages", 37 | "Version", 38 | ] 39 | optional_keys = ["IsDelegatedAccount"] 40 | allowed_keys = main_keys + optional_keys 41 | 42 | # Remove ServiceToken (lambda arn) to avoid sending AccountId 43 | resource_properties.pop("ServiceToken", None) 44 | resource_properties.pop("Resource", None) 45 | 46 | # Solution ID and unique ID are sent separately 47 | resource_properties.pop("SolutionId", None) 48 | resource_properties.pop("UUID", None) 49 | 50 | # send only allowed metrics 51 | sanitized_data = { 52 | key: resource_properties[key] 53 | for key in allowed_keys 54 | if key in resource_properties 55 | } 56 | 57 | return sanitized_data 58 | 59 | 60 | def _send_anonymous_metrics(request_type, resource_properties): 61 | try: 62 | metrics_data = _sanitize_data(copy(resource_properties)) 63 | metrics_data["RequestType"] = request_type 64 | 65 | headers = {"Content-Type": "application/json"} 66 | 67 | # create the payload 68 | payload = { 69 | "Solution": resource_properties["SolutionId"], 70 | "UUID": resource_properties["UUID"], 71 | "TimeStamp": datetime.utcnow().isoformat(), 72 | "Data": metrics_data, 73 | } 74 | 75 | logger.info(f"Sending payload: {payload}") 76 | response = requests.post( 77 | "https://metrics.awssolutionsbuilder.com/generic", 78 | json=payload, 79 | headers=headers, 80 | timeout=REQUST_TIMEOUT, 81 | ) 82 | # log the response 83 | logger.info( 84 | f"Response from the metrics endpoint: {response.status_code} {response.reason}" 85 | ) 86 | # raise error if response is an 404, 503, 500, 403 etc. 87 | response.raise_for_status() 88 | return response 89 | except Exception as e: 90 | logger.exception(f"Error when trying to send anonymous_metrics: {str(e)}") 91 | return None 92 | 93 | 94 | @helper.create 95 | @helper.update 96 | @helper.delete 97 | def custom_resource(event, _): 98 | request_type = event["RequestType"] 99 | resource_properties = event["ResourceProperties"] 100 | resource = resource_properties["Resource"] 101 | 102 | if resource == "UUID" and (request_type == "Create" or request_type == "Update"): 103 | random_id = str(uuid.uuid4()) 104 | helper.Data.update({"UUID": random_id}) 105 | elif resource == "AnonymizedMetric": 106 | # send Anonymous Metrics to AWS 107 | _send_anonymous_metrics(request_type, resource_properties) 108 | 109 | 110 | def handler(event, context): 111 | helper(event, context) 112 | -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/pipeline_definitions/sagemaker_role.py: -------------------------------------------------------------------------------- 1 | # ##################################################################################################################### 2 | # Copyright 2021 Amazon.com, Inc. or its affiliates. All Rights Reserved. # 3 | # # 4 | # Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance # 5 | # with the License. A copy of the License is located at # 6 | # # 7 | # http://www.apache.org/licenses/LICENSE-2.0 # 8 | # # 9 | # or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES # 10 | # OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions # 11 | # and limitations under the License. # 12 | # ##################################################################################################################### 13 | from aws_cdk import Aspects, Aws, aws_iam as iam 14 | from lib.blueprints.aspects.conditional_resource import ConditionalResources 15 | 16 | from lib.blueprints.pipeline_definitions.iam_policies import ( 17 | ecr_policy_document, 18 | kms_policy_document, 19 | sagemaker_policy_statement, 20 | sagemaker_monitor_policy_statement, 21 | sagemaker_tags_policy_statement, 22 | sagemaker_logs_metrics_policy_document, 23 | s3_policy_read, 24 | s3_policy_write, 25 | pass_role_policy_statement, 26 | get_role_policy_statement, 27 | model_registry_policy_document, 28 | ) 29 | 30 | 31 | def create_sagemaker_role( 32 | scope, # NOSONAR:S107 this function is designed to take many arguments 33 | id, 34 | custom_algorithms_ecr_arn, 35 | kms_key_arn, 36 | model_package_group_name, 37 | assets_bucket_name, 38 | input_bucket_name, 39 | input_s3_location, 40 | output_s3_location, 41 | ecr_repo_arn_provided_condition, 42 | kms_key_arn_provided_condition, 43 | model_registry_provided_condition, 44 | is_realtime_pipeline=False, 45 | endpoint_name=None, 46 | endpoint_name_provided=None, 47 | ): 48 | # create optional policies 49 | ecr_policy = ecr_policy_document(scope, "MLOpsECRPolicy", custom_algorithms_ecr_arn) 50 | kms_policy = kms_policy_document(scope, "MLOpsKmsPolicy", kms_key_arn) 51 | model_registry = model_registry_policy_document( 52 | scope, "ModelRegistryPolicy", model_package_group_name 53 | ) 54 | 55 | # add conditions to KMS and ECR policies 56 | Aspects.of(kms_policy).add(ConditionalResources(kms_key_arn_provided_condition)) 57 | Aspects.of(ecr_policy).add(ConditionalResources(ecr_repo_arn_provided_condition)) 58 | Aspects.of(model_registry).add( 59 | ConditionalResources(model_registry_provided_condition) 60 | ) 61 | 62 | # create sagemaker role 63 | role = iam.Role( 64 | scope, id, assumed_by=iam.ServicePrincipal("sagemaker.amazonaws.com") 65 | ) 66 | 67 | # permissions to create sagemaker resources 68 | sagemaker_policy = sagemaker_policy_statement( 69 | is_realtime_pipeline, endpoint_name, endpoint_name_provided 70 | ) 71 | 72 | # sagemaker tags permissions 73 | sagemaker_tags_policy = sagemaker_tags_policy_statement() 74 | # logs permissions 75 | logs_policy = sagemaker_logs_metrics_policy_document(scope, "LogsMetricsPolicy") 76 | # S3 permissions 77 | s3_read = s3_policy_read( 78 | list( 79 | set( 80 | [ 81 | f"arn:{Aws.PARTITION}:s3:::{assets_bucket_name}", 82 | f"arn:{Aws.PARTITION}:s3:::{assets_bucket_name}/*", 83 | f"arn:{Aws.PARTITION}:s3:::{input_bucket_name}", 84 | f"arn:{Aws.PARTITION}:s3:::{input_s3_location}", 85 | ] 86 | ) 87 | ) 88 | ) 89 | s3_write = s3_policy_write( 90 | [ 91 | f"arn:{Aws.PARTITION}:s3:::{output_s3_location}/*", 92 | ] 93 | ) 94 | # IAM PassRole permission 95 | pass_role_policy = pass_role_policy_statement(role) 96 | # IAM GetRole permission 97 | get_role_policy = get_role_policy_statement(role) 98 | 99 | # add policy statements 100 | role.add_to_policy(sagemaker_policy) 101 | role.add_to_policy(sagemaker_tags_policy) 102 | logs_policy.attach_to_role(role) 103 | role.add_to_policy(s3_read) 104 | role.add_to_policy(s3_write) 105 | role.add_to_policy(pass_role_policy) 106 | role.add_to_policy(get_role_policy) 107 | 108 | # attach the conditional policies 109 | kms_policy.attach_to_role(role) 110 | ecr_policy.attach_to_role(role) 111 | model_registry.attach_to_role(role) 112 | 113 | return role 114 | -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/lambdas/create_update_cf_stackset/main.py: -------------------------------------------------------------------------------- 1 | # ##################################################################################################################### 2 | # Copyright 2021 Amazon.com, Inc. or its affiliates. All Rights Reserved. # 3 | # # 4 | # Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance # 5 | # with the License. A copy of the License is located at # 6 | # # 7 | # http://www.apache.org/licenses/LICENSE-2.0 # 8 | # # 9 | # or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES # 10 | # OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions # 11 | # and limitations under the License. # 12 | # ##################################################################################################################### 13 | import json 14 | import traceback 15 | from stackset_helpers import ( 16 | find_artifact, 17 | get_template, 18 | put_job_failure, 19 | start_stackset_update_or_create, 20 | check_stackset_update_status, 21 | get_user_params, 22 | setup_s3_client, 23 | ) 24 | from shared.logger import get_logger 25 | from shared.helper import get_client 26 | 27 | logger = get_logger(__name__) 28 | 29 | logger.info("Loading stackset helpers...") 30 | 31 | cf_client = get_client("cloudformation") 32 | cp_client = get_client("codepipeline") 33 | 34 | 35 | def lambda_handler(event, _): 36 | """The Lambda function handler 37 | 38 | If a continuing job then checks the CloudFormation stackset and its instances status 39 | and updates the job accordingly. 40 | 41 | If a new job then kick of an update or creation of the target 42 | CloudFormation stackset and its instances. 43 | 44 | Args: 45 | event: The event passed by Lambda 46 | context: The context passed by Lambda 47 | 48 | """ 49 | job_id = None 50 | try: 51 | # Extract the Job ID 52 | job_id = event["CodePipeline.job"]["id"] 53 | 54 | # Extract the Job Data 55 | job_data = event["CodePipeline.job"]["data"] 56 | 57 | # Get user paramameters 58 | # User data is expected to be passed to lambda , for example: 59 | # {"stackset_name": "model2", "artifact":"SourceArtifact", 60 | # "template_file":"realtime-inference-pipeline.yaml", 61 | # "stage_params_file":"staging-config.json", 62 | # "account_ids":[""], "org_ids":[""], 63 | # "regions":["us-east-1"]} 64 | params = get_user_params(job_data) 65 | 66 | # Get the list of artifacts passed to the function 67 | artifacts = job_data["inputArtifacts"] 68 | # Extract parameters 69 | stackset_name = params["stackset_name"] 70 | artifact = params["artifact"] 71 | template_file = params["template_file"] 72 | stage_params_file = params["stage_params_file"] 73 | account_ids = params["account_ids"] 74 | org_ids = params["org_ids"] 75 | regions = params["regions"] 76 | 77 | if "continuationToken" in job_data: 78 | logger.info(f"Checking the status of {stackset_name}") 79 | # If we're continuing then the create/update has already been triggered 80 | # we just need to check if it has finished. 81 | check_stackset_update_status(job_id, stackset_name, account_ids[0], regions[0], cf_client, cp_client) 82 | 83 | else: 84 | logger.info(f"Creating StackSet {stackset_name} and its instances") 85 | # Get the artifact details 86 | artifact_data = find_artifact(artifacts, artifact) 87 | # Get S3 client to access artifact with 88 | s3 = setup_s3_client(job_data) 89 | # Get the JSON template file out of the artifact 90 | template, stage_params = get_template(s3, artifact_data, template_file, stage_params_file) 91 | logger.info(stage_params) 92 | # Kick off a stackset update or create 93 | start_stackset_update_or_create( 94 | job_id, 95 | stackset_name, 96 | template, 97 | json.loads(stage_params), 98 | account_ids, 99 | org_ids, 100 | regions, 101 | cf_client, 102 | cp_client, 103 | ) 104 | 105 | except Exception as e: 106 | logger.error(f"Error in create_update_cf_stackset lambda functions: {str(e)}") 107 | traceback.print_exc() 108 | put_job_failure(job_id, "Function exception", cp_client) 109 | raise e 110 | -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/pipeline_definitions/sagemaker_monitor_role.py: -------------------------------------------------------------------------------- 1 | # ##################################################################################################################### 2 | # Copyright 2021 Amazon.com, Inc. or its affiliates. All Rights Reserved. # 3 | # # 4 | # Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance # 5 | # with the License. A copy of the License is located at # 6 | # # 7 | # http://www.apache.org/licenses/LICENSE-2.0 # 8 | # # 9 | # or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES # 10 | # OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions # 11 | # and limitations under the License. # 12 | # ##################################################################################################################### 13 | from aws_cdk import Aspects, Aws, aws_iam as iam 14 | from lib.blueprints.aspects.conditional_resource import ConditionalResources 15 | 16 | from lib.blueprints.pipeline_definitions.iam_policies import ( 17 | kms_policy_document, 18 | sagemaker_monitor_policy_statement, 19 | sagemaker_tags_policy_statement, 20 | sagemaker_logs_metrics_policy_document, 21 | s3_policy_read, 22 | s3_policy_write, 23 | pass_role_policy_statement, 24 | get_role_policy_statement, 25 | ) 26 | 27 | 28 | def create_sagemaker_monitor_role( 29 | scope, # NOSONAR:S107 this function is designed to take many arguments 30 | id, 31 | kms_key_arn, 32 | assets_bucket_name, 33 | data_capture_bucket, 34 | data_capture_s3_location, 35 | baseline_output_bucket, 36 | baseline_job_output_location, 37 | output_s3_location, 38 | kms_key_arn_provided_condition, 39 | baseline_job_name, 40 | monitoring_schedule_name, 41 | endpoint_name, 42 | model_monitor_ground_truth_bucket, 43 | model_monitor_ground_truth_input, 44 | monitoring_type, 45 | ): 46 | # create optional policies 47 | kms_policy = kms_policy_document(scope, "MLOpsKmsPolicy", kms_key_arn) 48 | 49 | # add conditions to KMS and ECR policies 50 | Aspects.of(kms_policy).add(ConditionalResources(kms_key_arn_provided_condition)) 51 | 52 | # create sagemaker role 53 | role = iam.Role( 54 | scope, id, assumed_by=iam.ServicePrincipal("sagemaker.amazonaws.com") 55 | ) 56 | 57 | # permissions to create sagemaker resources 58 | sagemaker_policy = sagemaker_monitor_policy_statement( 59 | baseline_job_name, monitoring_schedule_name, endpoint_name, monitoring_type 60 | ) 61 | 62 | # sagemaker tags permissions 63 | sagemaker_tags_policy = sagemaker_tags_policy_statement() 64 | # logs/metrics permissions 65 | logs_metrics_policy = sagemaker_logs_metrics_policy_document( 66 | scope, "SagemakerLogsMetricsPolicy" 67 | ) 68 | # S3 permissions 69 | s3_read_resources = list( 70 | set( # set is used since a same bucket can be used more than once 71 | [ 72 | f"arn:{Aws.PARTITION}:s3:::{assets_bucket_name}", 73 | f"arn:{Aws.PARTITION}:s3:::{assets_bucket_name}/*", 74 | f"arn:{Aws.PARTITION}:s3:::{data_capture_bucket}", 75 | f"arn:{Aws.PARTITION}:s3:::{data_capture_s3_location}/*", 76 | f"arn:{Aws.PARTITION}:s3:::{baseline_output_bucket}", 77 | f"arn:{Aws.PARTITION}:s3:::{baseline_job_output_location}/*", 78 | ] 79 | ) 80 | ) 81 | 82 | # add permissions to read ground truth data (only for ModelQuality monitor) 83 | if model_monitor_ground_truth_bucket: 84 | s3_read_resources.extend( 85 | [ 86 | f"arn:{Aws.PARTITION}:s3:::{model_monitor_ground_truth_bucket}", 87 | f"arn:{Aws.PARTITION}:s3:::{model_monitor_ground_truth_input}/*", 88 | ] 89 | ) 90 | s3_read = s3_policy_read(s3_read_resources) 91 | s3_write = s3_policy_write( 92 | [ 93 | f"arn:{Aws.PARTITION}:s3:::{output_s3_location}/*", 94 | ] 95 | ) 96 | # IAM PassRole permission 97 | pass_role_policy = pass_role_policy_statement(role) 98 | # IAM GetRole permission 99 | get_role_policy = get_role_policy_statement(role) 100 | 101 | # add policy statements 102 | role.add_to_policy(sagemaker_policy) 103 | role.add_to_policy(sagemaker_tags_policy) 104 | role.add_to_policy(s3_read) 105 | role.add_to_policy(s3_write) 106 | role.add_to_policy(pass_role_policy) 107 | role.add_to_policy(get_role_policy) 108 | 109 | # attach he logs/metrics policy document 110 | logs_metrics_policy.attach_to_role(role) 111 | # attach the conditional policies 112 | kms_policy.attach_to_role(role) 113 | 114 | return role 115 | -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/aspects/app_registry_aspect.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | # ##################################################################################################################### 4 | # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. # 5 | # # 6 | # Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance # 7 | # with the License. You may obtain a copy of the License at # 8 | # # 9 | # http://www.apache.org/licenses/LICENSE-2.0 # 10 | # # 11 | # Unless required by applicable law or agreed to in writing, software distributed under the License is distributed # 12 | # on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for # 13 | # the specific language governing permissions and limitations under the License. # 14 | # ##################################################################################################################### 15 | 16 | import jsii 17 | 18 | import aws_cdk as cdk 19 | from aws_cdk import aws_servicecatalogappregistry_alpha as appreg 20 | 21 | from constructs import Construct, IConstruct 22 | 23 | 24 | @jsii.implements(cdk.IAspect) 25 | class AppRegistry(Construct): 26 | """This construct creates the resources required for AppRegistry and injects them as Aspects""" 27 | 28 | def __init__( 29 | self, 30 | scope: Construct, 31 | id: str, 32 | solution_id: str, 33 | solution_name: str, 34 | solution_version: str, 35 | app_registry_name: str, 36 | application_type: str, 37 | ): 38 | super().__init__(scope, id) 39 | self.solution_id = solution_id 40 | self.solution_name = solution_name 41 | self.solution_version = solution_version 42 | self.app_registry_name = app_registry_name 43 | self.application_type = application_type 44 | self.application: appreg.Application = None 45 | 46 | def visit(self, node: IConstruct) -> None: 47 | """The visitor method invoked during cdk synthesis""" 48 | if isinstance(node, cdk.Stack): 49 | if not node.nested: 50 | # parent stack 51 | stack: cdk.Stack = node 52 | self.__create_app_for_app_registry() 53 | self.application.associate_stack(stack) 54 | self.__create_atttribute_group() 55 | self.__add_tags_for_application() 56 | else: 57 | # nested stack 58 | if not self.application: 59 | self.__create_app_for_app_registry() 60 | 61 | self.application.associate_stack(node) 62 | 63 | def __create_app_for_app_registry(self) -> None: 64 | """Method to create an AppRegistry Application""" 65 | self.application = appreg.Application( 66 | self, 67 | "RegistrySetup", 68 | application_name=cdk.Fn.join( 69 | "-", 70 | [ 71 | "App", 72 | cdk.Aws.STACK_NAME, 73 | self.app_registry_name, 74 | ], 75 | ), 76 | description=f"Service Catalog application to track and manage all your resources for the solution {self.solution_name}", 77 | ) 78 | 79 | def __add_tags_for_application(self) -> None: 80 | """Method to add tags to the AppRegistry's Application instance""" 81 | if not self.application: 82 | self.__create_app_for_app_registry() 83 | 84 | cdk.Tags.of(self.application).add("Solutions:SolutionID", self.solution_id) 85 | cdk.Tags.of(self.application).add("Solutions:SolutionName", self.solution_name) 86 | cdk.Tags.of(self.application).add( 87 | "Solutions:SolutionVersion", self.solution_version 88 | ) 89 | cdk.Tags.of(self.application).add( 90 | "Solutions:ApplicationType", self.application_type 91 | ) 92 | 93 | def __create_atttribute_group(self) -> None: 94 | """Method to add attributes to be as associated with the Application's instance in AppRegistry""" 95 | if not self.application: 96 | self.__create_app_for_app_registry() 97 | 98 | self.application.associate_attribute_group( 99 | appreg.AttributeGroup( 100 | self, 101 | "AppAttributes", 102 | attribute_group_name=f"AttrGrp-{cdk.Aws.STACK_NAME}", 103 | description="Attributes for Solutions Metadata", 104 | attributes={ 105 | "applicationType": self.application_type, 106 | "version": self.solution_version, 107 | "solutionID": self.solution_id, 108 | "solutionName": self.solution_name, 109 | }, 110 | ) 111 | ) 112 | -------------------------------------------------------------------------------- /NOTICE.txt: -------------------------------------------------------------------------------- 1 | mlops-workload-orchestrator 2 | Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | Licensed under the Apache License Version 2.0 (the "License"). You may not use this file except 4 | in compliance with the License. A copy of the License is located at http://www.apache.org/licenses/ 5 | or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, 6 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, express or implied. See the License for the 7 | specific language governing permissions and limitations under the License. 8 | 9 | ********************** 10 | THIRD PARTY COMPONENTS 11 | ********************** 12 | This software includes third party software subject to the following copyrights: 13 | 14 | Jinja2 under the BSD License 15 | MarkupSafe under the BSD License 16 | PyYAML under the MIT License 17 | Werkzeug under the BSD License 18 | annotated-types under the MIT License 19 | antlr4-python3-runtime under the BSD License 20 | attrs under the MIT License 21 | aws-cdk-lib under the Apache-2.0 license 22 | aws-cdk.asset-awscli-v1 under the Apache-2.0 license 23 | aws-cdk.asset-kubectl-v20 under the Apache-2.0 license 24 | aws-cdk.asset-node-proxy-agent-v5 under the Apache-2.0 license 25 | aws-cdk.aws-servicecatalogappregistry-alpha under the Apache-2.0 license 26 | aws-sam-translator under the Apache Software License 27 | aws-solutions-constructs.aws-apigateway-lambda under the Apache-2.0 license 28 | aws-solutions-constructs.aws-lambda-sagemakerendpoint under the Apache-2.0 license 29 | aws-solutions-constructs.core under the Apache-2.0 license 30 | aws-xray-sdk under the Apache Software License 31 | boto3 under the Apache-2.0 license 32 | botocore under the Apache-2.0 license 33 | cattrs under the MIT License 34 | certifi under the Mozilla Public License 2.0 (MPL 2.0) 35 | cffi under the MIT License 36 | cfn-lint under the MIT License 37 | charset-normalizer under the MIT License 38 | cloudpickle under the BSD License 39 | constructs under the Apache-2.0 license 40 | contextlib2 under the Apache Software License; Python Software Foundation License 41 | coverage under the Apache Software License 42 | crhelper under the Apache-2.0 license. 43 | cryptography under the Apache Software License; BSD License 44 | dill under the BSD License 45 | docker under the Apache Software License 46 | ecdsa under the MIT License 47 | exceptiongroup under the MIT License 48 | google-pasta under the Apache Software License 49 | graphql-core under the MIT License 50 | idna under the BSD License 51 | importlib-metadata under the Apache Software License 52 | importlib-resources under the Apache Software License 53 | importlib_resources under the Apache Software License 54 | iniconfig under the MIT License 55 | jmespath under the MIT License 56 | joserfc under the BSD License (BSD-3-Clause) 57 | jschema-to-python under the MIT License 58 | jsii under the Apache Software License 59 | jsondiff under the MIT License 60 | jsonpatch under the BSD License 61 | jsonpickle under the BSD License 62 | jsonpointer under the BSD License 63 | jsonpath-ng under the Apache Software License 64 | jsonschema under the MIT License 65 | jsonschema-path under the Apache Software License 66 | jsonschema-spec under the Apache Software License 67 | jsonschema-specifications under the MIT License 68 | junit-xml under the Freely Distributable; MIT License 69 | lazy-object-proxy under the BSD License 70 | moto under the Apache-2.0 license 71 | mpmath under the BSD License 72 | multipart under the Apache Software License 73 | multiprocess under the BSD License 74 | networkx under the BSD License 75 | numpy under the BSD License 76 | openapi-schema-validator under the BSD License 77 | openapi-spec-validator under the Apache Software License 78 | packaging under the Apache Software License; BSD License 79 | pandas under the BSD License 80 | pathable under the Other/Proprietary License 81 | pathos under the BSD License 82 | pbr under the Apache Software License 83 | platformdirs under the MIT License 84 | pluggy under the MIT License 85 | ply under the BSD License 86 | pox under the BSD License 87 | ppft under the BSD License 88 | protobuf under the BSD-3-Clause 89 | protobuf3-to-dict under the Public Domain 90 | psutil under the BSD-3-Clause 91 | publication under the MIT License 92 | py-partiql-parser under the MIT License 93 | pyasn1 under the BSD License 94 | pycparser under the BSD License 95 | pydantic under the MIT License 96 | pydantic_core under the MIT License 97 | pyparsing under the MIT License 98 | pytest under the MIT license 99 | pytest-cov under the MIT license 100 | python-dateutil under the Apache Software License; BSD License 101 | python-jose under the MIT License 102 | pytz under the MIT License 103 | referencing under the MIT License 104 | regex under the Apache Software License 105 | requests under the Apache-2.0 license 106 | responses under the Apache 2.0 107 | rfc3339-validator under the MIT License 108 | rpds-py under the MIT License 109 | rsa under the Apache Software License 110 | s3transfer under the Apache Software License 111 | sagemaker under the Apache-2.0 license 112 | sarif-om under the MIT License 113 | schema under the MIT License 114 | six under the MIT License 115 | smdebug-rulesconfig under the Apache Software License 116 | sshpubkeys under the BSD License 117 | sympy under the BSD License 118 | tblib under the BSD License 119 | tomli under the MIT License 120 | tqdm under the MIT License 121 | typeguard under the MIT License 122 | types-PyYAML under the Apache Software License 123 | typing_extensions under the Python Software Foundation License 124 | tzdata under the Apache Software License 125 | urllib3 under the MIT license 126 | websocket-client under the Apache Software License 127 | wrapt under the BSD License 128 | xmltodict under the MIT License 129 | zipp under the MIT License 130 | fs under the MIT license 131 | 132 | ******************** 133 | OPEN SOURCE LICENSES 134 | ******************** 135 | 136 | Apache-2.0 - https://spdx.org/licenses/Apache-2.0.html -------------------------------------------------------------------------------- /deployment/cdk-solution-helper/README.md: -------------------------------------------------------------------------------- 1 | # cdk-solution-helper 2 | 3 | A lightweight helper function that cleans-up synthesized templates from the AWS Cloud Development Kit (CDK) and prepares 4 | them for use with the AWS Solutions publishing pipeline. This function performs the following tasks: 5 | 6 | #### Lambda function preparation 7 | 8 | Replaces the AssetParameter-style properties that identify source code for Lambda functions with the common variables 9 | used by the AWS Solutions publishing pipeline. 10 | 11 | - `Code.S3Bucket` is assigned the `%%BUCKET_NAME%%` placeholder value. 12 | - `Code.S3Key` is assigned the `%%SOLUTION_NAME%%`/`%%VERSION%%` placeholder value. 13 | - `Handler` is given a prefix identical to the artifact hash, enabling the Lambda function to properly find the handler in the extracted source code package. 14 | 15 | These placeholders are then replaced with the appropriate values using the default find/replace operation run by the pipeline. 16 | 17 | Before: 18 | 19 | ``` 20 | "examplefunction67F55935": { 21 | "Type": "AWS::Lambda::Function", 22 | "Properties": { 23 | "Code": { 24 | "S3Bucket": { 25 | "Ref": "AssetParametersd513e93e266931de36e1c7e79c27b196f84ab928fce63d364d9152ca501551f7S3Bucket54E71A95" 26 | }, 27 | "S3Key": { 28 | "Fn::Join": [ 29 | "", 30 | [ 31 | { 32 | "Fn::Select": [ 33 | 0, 34 | { 35 | "Fn::Split": [ 36 | "||", 37 | { 38 | "Ref": "AssetParametersd513e93e266931de36e1c7e79c27b196f84ab928fce63d364d9152ca501551f7S3VersionKeyC789D8B1" 39 | } 40 | ] 41 | } 42 | ] 43 | }, 44 | { 45 | "Fn::Select": [ 46 | 1, 47 | { 48 | "Fn::Split": [ 49 | "||", 50 | { 51 | "Ref": "AssetParametersd513e93e266931de36e1c7e79c27b196f84ab928fce63d364d9152ca501551f7S3VersionKeyC789D8B1" 52 | } 53 | ] 54 | } 55 | ] 56 | } 57 | ] 58 | ] 59 | } 60 | }, ... 61 | Handler: "index.handler", ... 62 | ``` 63 | 64 | After helper function run: 65 | 66 | ``` 67 | "examplefunction67F55935": { 68 | "Type": "AWS::Lambda::Function", 69 | "Properties": { 70 | "Code": { 71 | "S3Bucket": "%%BUCKET_NAME%%", 72 | "S3Key": "%%SOLUTION_NAME%%/%%VERSION%%/assetd513e93e266931de36e1c7e79c27b196f84ab928fce63d364d9152ca501551f7.zip" 73 | }, ... 74 | "Handler": "assetd513e93e266931de36e1c7e79c27b196f84ab928fce63d364d9152ca501551f7/index.handler" 75 | ``` 76 | 77 | After build script run: 78 | 79 | ``` 80 | "examplefunction67F55935": { 81 | "Type": "AWS::Lambda::Function", 82 | "Properties": { 83 | "Code": { 84 | "S3Bucket": "solutions", 85 | "S3Key": "trademarked-solution-name/v1.0.0/asset.d513e93e266931de36e1c7e79c27b196f84ab928fce63d364d9152ca501551f7.zip" 86 | }, ... 87 | "Handler": "assetd513e93e266931de36e1c7e79c27b196f84ab928fce63d364d9152ca501551f7/index.handler" 88 | ``` 89 | 90 | After CloudFormation deployment: 91 | 92 | ``` 93 | "examplefunction67F55935": { 94 | "Type": "AWS::Lambda::Function", 95 | "Properties": { 96 | "Code": { 97 | "S3Bucket": "solutions-us-east-1", 98 | "S3Key": "trademarked-solution-name/v1.0.0/asset.d513e93e266931de36e1c7e79c27b196f84ab928fce63d364d9152ca501551f7.zip" 99 | }, ... 100 | "Handler": "assetd513e93e266931de36e1c7e79c27b196f84ab928fce63d364d9152ca501551f7/index.handler" 101 | ``` 102 | 103 | #### Template cleanup 104 | 105 | Cleans-up the parameters section and improves readability by removing the AssetParameter-style fields that would have 106 | been used to specify Lambda source code properties. This allows solution-specific parameters to be highlighted and 107 | removes unnecessary clutter. 108 | 109 | Before: 110 | 111 | ``` 112 | "Parameters": { 113 | "AssetParametersd513e93e266931de36e1c7e79c27b196f84ab928fce63d364d9152ca501551f7S3Bucket54E71A95": { 114 | "Type": "String", 115 | "Description": "S3 bucket for asset \"d513e93e266931de36e1c7e79c27b196f84ab928fce63d364d9152ca501551f7\"" 116 | }, 117 | "AssetParametersd513e93e266931de36e1c7e79c27b196f84ab928fce63d364d9152ca501551f7S3VersionKeyC789D8B1": { 118 | "Type": "String", 119 | "Description": "S3 key for asset version \"d513e93e266931de36e1c7e79c27b196f84ab928fce63d364d9152ca501551f7\"" 120 | }, 121 | "AssetParametersd513e93e266931de36e1c7e79c27b196f84ab928fce63d364d9152ca501551f7ArtifactHash7AA751FE": { 122 | "Type": "String", 123 | "Description": "Artifact hash for asset \"d513e93e266931de36e1c7e79c27b196f84ab928fce63d364d9152ca501551f7\"" 124 | }, 125 | "CorsEnabled" : { 126 | "Description" : "Would you like to enable Cross-Origin Resource Sharing (CORS) for the image handler API? Select 'Yes' if so.", 127 | "Default" : "No", 128 | "Type" : "String", 129 | "AllowedValues" : [ "Yes", "No" ] 130 | }, 131 | "CorsOrigin" : { 132 | "Description" : "If you selected 'Yes' above, please specify an origin value here. A wildcard (*) value will support any origin.", 133 | "Default" : "*", 134 | "Type" : "String" 135 | } 136 | } 137 | ``` 138 | 139 | After: 140 | 141 | ``` 142 | "Parameters": { 143 | "CorsEnabled" : { 144 | "Description" : "Would you like to enable Cross-Origin Resource Sharing (CORS) for the image handler API? Select 'Yes' if so.", 145 | "Default" : "No", 146 | "Type" : "String", 147 | "AllowedValues" : [ "Yes", "No" ] 148 | }, 149 | "CorsOrigin" : { 150 | "Description" : "If you selected 'Yes' above, please specify an origin value here. A wildcard (*) value will support any origin.", 151 | "Default" : "*", 152 | "Type" : "String" 153 | } 154 | } 155 | ``` 156 | 157 | --- 158 | 159 | © Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. 160 | -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/pipeline_definitions/build_actions.py: -------------------------------------------------------------------------------- 1 | # ##################################################################################################################### 2 | # Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. # 3 | # # 4 | # Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance # 5 | # with the License. A copy of the License is located at # 6 | # # 7 | # http://www.apache.org/licenses/LICENSE-2.0 # 8 | # # 9 | # or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES # 10 | # OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions # 11 | # and limitations under the License. # 12 | # ##################################################################################################################### 13 | from aws_cdk import ( 14 | Aws, 15 | aws_iam as iam, 16 | aws_codebuild as codebuild, 17 | aws_codepipeline as codepipeline, 18 | aws_codepipeline_actions as codepipeline_actions, 19 | ) 20 | from lib.blueprints.pipeline_definitions.helpers import suppress_pipeline_policy 21 | 22 | 23 | def build_action(scope, ecr_repository_name, image_tag, source_output): 24 | """ 25 | build_action configures a codepipeline action with repository name and tag 26 | 27 | :scope: CDK Construct scope that's needed to create CDK resources 28 | :ecr_repository_name: name of Amazon ECR repository where the image will be stored 29 | :image_tag: docker image tag to be assigned. 30 | :return: codepipeline action in a form of a CDK object that can be attached to a codepipeline stage 31 | """ 32 | 33 | codebuild_role = iam.Role( 34 | scope, 35 | "codebuildRole", 36 | assumed_by=iam.ServicePrincipal("codebuild.amazonaws.com"), 37 | ) 38 | 39 | codebuild_policy = iam.PolicyStatement( 40 | actions=[ 41 | "ecr:BatchCheckLayerAvailability", 42 | "ecr:CompleteLayerUpload", 43 | "ecr:InitiateLayerUpload", 44 | "ecr:PutImage", 45 | "ecr:UploadLayerPart", 46 | ], 47 | resources=[ 48 | f"arn:{Aws.PARTITION}:ecr:{Aws.REGION}:{Aws.ACCOUNT_ID}:repository/{ecr_repository_name}", 49 | ], 50 | ) 51 | codebuild_role.add_to_policy(codebuild_policy) 52 | codebuild_role.add_to_policy( 53 | iam.PolicyStatement(actions=["ecr:GetAuthorizationToken"], resources=["*"]) 54 | ) 55 | # add suppression 56 | codebuild_role.node.find_child( 57 | "DefaultPolicy" 58 | ).node.default_child.cfn_options.metadata = { "cfn_nag": suppress_pipeline_policy() } 59 | # codebuild setup for build stage 60 | container_factory_project = codebuild.PipelineProject( 61 | scope, 62 | "Container_Factory", 63 | build_spec=codebuild.BuildSpec.from_object( 64 | { 65 | "version": "0.2", 66 | "phases": { 67 | "pre_build": { 68 | "commands": [ 69 | "echo Logging in to Amazon ECR...", 70 | ( 71 | "aws ecr get-login-password --region $AWS_DEFAULT_REGION | docker login --username AWS " 72 | "--password-stdin $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" 73 | ), 74 | 'find . -iname "serve" -exec chmod 777 "{}" \\;', 75 | 'find . -iname "train" -exec chmod 777 "{}" \\;', 76 | ] 77 | }, 78 | "build": { 79 | "commands": [ 80 | "echo Build started on `date`", 81 | "echo Building the Docker image...", 82 | "docker build -t $IMAGE_REPO_NAME:$IMAGE_TAG .", 83 | ( 84 | "docker tag $IMAGE_REPO_NAME:$IMAGE_TAG $AWS_ACCOUNT_ID.dkr.ecr." 85 | "$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:$IMAGE_TAG" 86 | ), 87 | ] 88 | }, 89 | "post_build": { 90 | "commands": [ 91 | "echo Build completed on `date`", 92 | "echo Pushing the Docker image...", 93 | ( 94 | "docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/" 95 | "$IMAGE_REPO_NAME:$IMAGE_TAG" 96 | ), 97 | ] 98 | }, 99 | }, 100 | } 101 | ), 102 | environment=codebuild.BuildEnvironment( 103 | build_image=codebuild.LinuxBuildImage.STANDARD_6_0, 104 | compute_type=codebuild.ComputeType.SMALL, 105 | environment_variables={ 106 | "AWS_DEFAULT_REGION": {"value": Aws.REGION}, 107 | "AWS_ACCOUNT_ID": {"value": Aws.ACCOUNT_ID}, 108 | "IMAGE_REPO_NAME": {"value": ecr_repository_name}, 109 | "IMAGE_TAG": {"value": image_tag}, 110 | }, 111 | privileged=True, 112 | ), 113 | role=codebuild_role, 114 | ) 115 | build_action_definition = codepipeline_actions.CodeBuildAction( 116 | action_name="CodeBuild", 117 | project=container_factory_project, 118 | input=source_output, 119 | outputs=[codepipeline.Artifact()], 120 | ) 121 | container_uri = f"{Aws.ACCOUNT_ID}.dkr.ecr.{Aws.REGION}.amazonaws.com/{ecr_repository_name}:{image_tag}" 122 | return build_action_definition, container_uri 123 | -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/ml_pipelines/byom_custom_algorithm_image_builder.py: -------------------------------------------------------------------------------- 1 | # ##################################################################################################################### 2 | # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. # 3 | # # 4 | # Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance # 5 | # with the License. A copy of the License is located at # 6 | # # 7 | # http://www.apache.org/licenses/LICENSE-2.0 # 8 | # # 9 | # or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES # 10 | # OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions # 11 | # and limitations under the License. # 12 | # ##################################################################################################################### 13 | from constructs import Construct 14 | from aws_cdk import ( 15 | Stack, 16 | Aws, 17 | CfnOutput, 18 | aws_iam as iam, 19 | aws_s3 as s3, 20 | aws_sns as sns, 21 | aws_events_targets as targets, 22 | aws_events as events, 23 | aws_codepipeline as codepipeline, 24 | ) 25 | from lib.blueprints.pipeline_definitions.source_actions import source_action_custom 26 | from lib.blueprints.pipeline_definitions.build_actions import build_action 27 | from lib.blueprints.pipeline_definitions.helpers import ( 28 | pipeline_permissions, 29 | suppress_pipeline_bucket, 30 | suppress_iam_complex, 31 | ) 32 | from lib.blueprints.pipeline_definitions.templates_parameters import ( 33 | ParameteresFactory as pf, 34 | ) 35 | 36 | 37 | class BYOMCustomAlgorithmImageBuilderStack(Stack): 38 | def __init__(self, scope: Construct, id: str, **kwargs) -> None: 39 | super().__init__(scope, id, **kwargs) 40 | 41 | # Parameteres # 42 | assets_bucket_name = pf.create_assets_bucket_name_parameter(self) 43 | custom_container = pf.create_custom_container_parameter(self) 44 | ecr_repo_name = pf.create_ecr_repo_name_parameter(self) 45 | image_tag = pf.create_image_tag_parameter(self) 46 | mlops_sns_topic_arn = pf.create_sns_topic_arn_parameter(self) 47 | 48 | # Resources # 49 | assets_bucket = s3.Bucket.from_bucket_name( 50 | self, "ImportedAssetsBucket", assets_bucket_name.value_as_string 51 | ) 52 | 53 | # Defining pipeline stages 54 | # source stage 55 | source_output, source_action_definition = source_action_custom( 56 | assets_bucket, custom_container 57 | ) 58 | 59 | # build stage 60 | build_action_definition, container_uri = build_action( 61 | self, 62 | ecr_repo_name.value_as_string, 63 | image_tag.value_as_string, 64 | source_output, 65 | ) 66 | 67 | # import the sns Topic 68 | pipeline_notification_topic = sns.Topic.from_topic_arn( 69 | self, 70 | "ImageBuilderPipelineNotification", 71 | mlops_sns_topic_arn.value_as_string, 72 | ) 73 | 74 | # createing pipeline stages 75 | source_stage = codepipeline.StageProps( 76 | stage_name="Source", actions=[source_action_definition] 77 | ) 78 | build_stage = codepipeline.StageProps( 79 | stage_name="Build", actions=[build_action_definition] 80 | ) 81 | 82 | image_builder_pipeline = codepipeline.Pipeline( 83 | self, 84 | "BYOMPipelineRealtimeBuild", 85 | stages=[source_stage, build_stage], 86 | cross_account_keys=False, 87 | ) 88 | image_builder_pipeline.on_state_change( 89 | "NotifyUser", 90 | description="Notify user of the outcome of the pipeline", 91 | target=targets.SnsTopic( 92 | pipeline_notification_topic, 93 | message=events.RuleTargetInput.from_text( 94 | ( 95 | f"Pipeline {events.EventField.from_path('$.detail.pipeline')} finished executing. " 96 | f"Pipeline execution result is {events.EventField.from_path('$.detail.state')}" 97 | ) 98 | ), 99 | ), 100 | event_pattern=events.EventPattern( 101 | detail={"state": ["SUCCEEDED", "FAILED"]} 102 | ), 103 | ) 104 | 105 | image_builder_pipeline.add_to_role_policy( 106 | iam.PolicyStatement( 107 | actions=["events:PutEvents"], 108 | resources=[ 109 | f"arn:{Aws.PARTITION}:events:{Aws.REGION}:{Aws.ACCOUNT_ID}:event-bus/*", 110 | ], 111 | ) 112 | ) 113 | 114 | # add ArtifactBucket cfn supression (not needing a logging bucket) 115 | image_builder_pipeline.node.find_child( 116 | "ArtifactsBucket" 117 | ).node.default_child.cfn_options.metadata = { "cfn_nag": suppress_pipeline_bucket() } 118 | 119 | # add supression for complex policy 120 | image_builder_pipeline.node.find_child("Role").node.find_child( 121 | "DefaultPolicy" 122 | ).node.default_child.cfn_options.metadata = { "cfn_nag": suppress_iam_complex() } 123 | 124 | # attaching iam permissions to the pipelines 125 | pipeline_permissions(image_builder_pipeline, assets_bucket) 126 | 127 | # Outputs # 128 | CfnOutput( 129 | self, 130 | id="Pipelines", 131 | value=( 132 | f"https://console.aws.amazon.com/codesuite/codepipeline/pipelines/" 133 | f"{image_builder_pipeline.pipeline_name}/view?region={Aws.REGION}" 134 | ), 135 | ) 136 | CfnOutput( 137 | self, 138 | id="CustomAlgorithmImageURI", 139 | value=container_uri, 140 | ) 141 | -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/lambdas/create_update_cf_stackset/tests/fixtures/stackset_fixtures.py: -------------------------------------------------------------------------------- 1 | ####################################################################################################################### 2 | # Copyright 2021 Amazon.com, Inc. or its affiliates. All Rights Reserved. # 3 | # # 4 | # Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance # 5 | # with the License. A copy of the License is located at # 6 | # # 7 | # http://www.apache.org/licenses/LICENSE-2.0 # 8 | # # 9 | # or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES # 10 | # OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions # 11 | # and limitations under the License. # 12 | # ##################################################################################################################### 13 | import json 14 | import pytest 15 | 16 | 17 | @pytest.fixture() 18 | def stackset_name(): 19 | return "mlops-stackset" 20 | 21 | 22 | @pytest.fixture() 23 | def mocked_org_ids(): 24 | return ["ou-x9x7-xxx1xxx3"] 25 | 26 | 27 | @pytest.fixture() 28 | def mocked_account_ids(): 29 | return ["Test_Account_Id"] 30 | 31 | 32 | @pytest.fixture() 33 | def mocked_regions(): 34 | return ["us-east-1"] 35 | 36 | 37 | @pytest.fixture() 38 | def mocked_job_id(): 39 | return "mocked_job_id" 40 | 41 | 42 | @pytest.fixture() 43 | def mocked_cp_success_message(): 44 | return "StackSet Job SUCCEEDED" 45 | 46 | 47 | @pytest.fixture() 48 | def mocked_cp_failure_message(): 49 | return "StackSet Job Failed" 50 | 51 | @pytest.fixture() 52 | def mocked_describe_response(): 53 | return {'StackInstance':{"StackInstanceStatus": {"DetailedStatus": "SUCCEEDED"}}} 54 | 55 | @pytest.fixture() 56 | def mocked_cp_continuation_message(): 57 | return "StackSet Job is continued" 58 | 59 | 60 | @pytest.fixture() 61 | def required_user_params(): 62 | return [ 63 | "stackset_name", 64 | "artifact", 65 | "template_file", 66 | "stage_params_file", 67 | "account_ids", 68 | "org_ids", 69 | "regions", 70 | ] 71 | 72 | 73 | @pytest.fixture() 74 | def mocked_decoded_parameters(): 75 | return { 76 | "stackset_name": "model2", 77 | "artifact": "SourceArtifact", 78 | "template_file": "template.yaml", 79 | "stage_params_file": "staging-config-test.json", 80 | "account_ids": ["mocked_account_id"], 81 | "org_ids": ["mocked_org_unit_id"], 82 | "regions": ["us-east-1"], 83 | } 84 | 85 | 86 | @pytest.fixture() 87 | def mocked_codepipeline_event(mocked_decoded_parameters): 88 | return { 89 | "CodePipeline.job": { 90 | "id": "11111111-abcd-1111-abcd-111111abcdef", 91 | "accountId": "test-account-id", 92 | "data": { 93 | "actionConfiguration": { 94 | "configuration": { 95 | "FunctionName": "stacketset-lambda", 96 | "UserParameters": json.dumps(mocked_decoded_parameters), 97 | } 98 | }, 99 | "inputArtifacts": [ 100 | { 101 | "location": { 102 | "s3Location": { 103 | "bucketName": "test-bucket", 104 | "objectKey": "template.zip", 105 | }, 106 | "type": "S3", 107 | }, 108 | "revision": None, 109 | "name": "SourceArtifact", 110 | } 111 | ], 112 | "outputArtifacts": [], 113 | "artifactCredentials": { 114 | "secretAccessKey": "test-secretkey", 115 | "sessionToken": "test-tockedn", 116 | "accessKeyId": "test-accesskey", 117 | }, 118 | }, 119 | } 120 | } 121 | 122 | 123 | @pytest.fixture() 124 | def mocked_invalid_user_parms(mocked_decoded_parameters): 125 | return { 126 | "CodePipeline.job": { 127 | "id": "11111111-abcd-1111-abcd-111111abcdef", 128 | "accountId": "test-account-id", 129 | "data": { 130 | "actionConfiguration": { 131 | "configuration": { 132 | "FunctionName": "stacketset-lambda", 133 | "UserParameters": mocked_decoded_parameters, 134 | } 135 | } 136 | }, 137 | } 138 | } 139 | 140 | 141 | @pytest.fixture() 142 | def mocked_template_parameters(): 143 | return json.dumps( 144 | [ 145 | {"ParameterKey": "TagDescription", "ParameterValue": "StackSetValue"}, 146 | {"ParameterKey": "TagName", "ParameterValue": "StackSetValue2"}, 147 | ] 148 | ) 149 | 150 | 151 | @pytest.fixture() 152 | def mocked_template(): 153 | template = """--- 154 | AWSTemplateFormatVersion: 2010-09-09 155 | Description: Stack1 with yaml template 156 | Parameters: 157 | TagDescription: 158 | Type: String 159 | TagName: 160 | Type: String 161 | Resources: 162 | EC2Instance1: 163 | Type: AWS::EC2::Instance 164 | Properties: 165 | ImageId: ami-03cf127a 166 | KeyName: dummy 167 | InstanceType: t2.micro 168 | Tags: 169 | - Key: Description 170 | Value: 171 | Ref: TagDescription 172 | - Key: Name 173 | Value: !Ref TagName 174 | """ 175 | return template 176 | 177 | 178 | @pytest.fixture(scope="function") 179 | def mocked_stackset(cf_client, stackset_name, mocked_template_parameters): 180 | cf_client.create_stack_set( 181 | StackSetName=stackset_name, 182 | TemplateBody=stackset_name, 183 | Parameters=mocked_template_parameters, 184 | ) 185 | -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/lambdas/create_model_training_job/tests/fixtures/training_fixtures.py: -------------------------------------------------------------------------------- 1 | ####################################################################################################################### 2 | # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. # 3 | # # 4 | # Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance # 5 | # with the License. A copy of the License is located at # 6 | # # 7 | # http://www.apache.org/licenses/LICENSE-2.0 # 8 | # # 9 | # or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES # 10 | # OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions # 11 | # and limitations under the License. # 12 | # ##################################################################################################################### 13 | import os 14 | import json 15 | import pytest 16 | from unittest.mock import Mock 17 | from sagemaker.inputs import TrainingInput 18 | from sagemaker.tuner import ContinuousParameter, IntegerParameter, CategoricalParameter 19 | from model_training_helper import SolutionModelTraining 20 | 21 | 22 | @pytest.fixture() 23 | def mocked_common_env_vars(): 24 | common_env_vars = { 25 | "ASSETS_BUCKET": "testbucket", 26 | "JOB_NAME": "test-training-job", 27 | "ROLE_ARN": "test-role", 28 | "JOB_OUTPUT_LOCATION": "job_output", 29 | "IMAGE_URI": "test-image", 30 | "INSTANCE_TYPE": "ml.m4.xlarge", 31 | "INSTANCE_COUNT": "1", 32 | "INSTANCE_VOLUME_SIZE": "20", 33 | "TRAINING_DATA_KEY": "data/train/training-dataset.csv", 34 | "VALIDATION_DATA_KEY": "data/validation/validation-dataset.csv", 35 | "CONTENT_TYPE": "csv", 36 | "USE_SPOT_INSTANCES": "True", 37 | "HYPERPARAMETERS": json.dumps( 38 | dict( 39 | eval_metric="auc", 40 | objective="binary:logistic", 41 | num_round=400, 42 | rate_drop=0.3, 43 | ) 44 | ), 45 | "TAGS": json.dumps([{"pipeline": "training"}]), 46 | } 47 | 48 | return common_env_vars 49 | 50 | 51 | @pytest.fixture() 52 | def mocked_training_job_env_vars(monkeypatch, mocked_common_env_vars): 53 | training_job_env_vars = mocked_common_env_vars.copy() 54 | training_job_env_vars.update({"JOB_TYPE": "TrainingJob"}) 55 | monkeypatch.setattr(os, "environ", training_job_env_vars) 56 | 57 | 58 | @pytest.fixture() 59 | def mocked_tuning_job_env_vars(monkeypatch, mocked_common_env_vars): 60 | tuning_job_env_vars = mocked_common_env_vars.copy() 61 | tuning_job_env_vars.update( 62 | { 63 | "JOB_TYPE": "HyperparameterTuningJob", 64 | "TUNER_CONFIG": json.dumps( 65 | dict( 66 | early_stopping_type="Auto", 67 | objective_metric_name="validation:auc", 68 | strategy="Bayesian", 69 | objective_type="Maximize", 70 | max_jobs=10, 71 | max_parallel_jobs=2, 72 | ) 73 | ), 74 | "HYPERPARAMETER_RANGES": json.dumps( 75 | { 76 | "eta": ["continuous", [0.1, 0.5]], 77 | "gamma": ["continuous", [0, 5]], 78 | "min_child_weight": ["continuous", [0, 120]], 79 | "max_depth": ["integer", [1, 15]], 80 | "optimizer": ["categorical", ["sgd", "Adam"]], 81 | } 82 | ), 83 | } 84 | ) 85 | 86 | monkeypatch.setattr(os, "environ", tuning_job_env_vars) 87 | 88 | 89 | @pytest.fixture() 90 | def mocked_hyperparameters(mocked_training_job_env_vars): 91 | return json.loads(os.environ["HYPERPARAMETERS"]) 92 | 93 | 94 | @pytest.fixture() 95 | def mocked_sagemaker_session(): 96 | region = "us-east-1" 97 | boto_mock = Mock(name="boto_session", region_name=region) 98 | sms = Mock( 99 | name="sagemaker_session", 100 | boto_session=boto_mock, 101 | boto_region_name=region, 102 | config=None, 103 | local_mode=False, 104 | s3_resource=None, 105 | s3_client=None, 106 | ) 107 | sms.sagemaker_config = {} 108 | return sms 109 | 110 | 111 | @pytest.fixture() 112 | def mocked_estimator_config(mocked_training_job_env_vars, mocked_sagemaker_session): 113 | return dict( 114 | image_uri=os.environ["IMAGE_URI"], 115 | role=os.environ["ROLE_ARN"], 116 | instance_count=int(os.environ["INSTANCE_COUNT"]), 117 | instance_type=os.environ["INSTANCE_TYPE"], 118 | volume_size=int(os.environ["INSTANCE_VOLUME_SIZE"]), 119 | output_path=f"s3://{os.environ['ASSETS_BUCKET']}/{os.environ['JOB_OUTPUT_LOCATION']}", 120 | sagemaker_session=mocked_sagemaker_session, 121 | ) 122 | 123 | 124 | @pytest.fixture() 125 | def mocked_data_channels(mocked_training_job_env_vars): 126 | train_input = TrainingInput( 127 | f"s3://{os.environ['ASSETS_BUCKET']}/{os.environ['TRAINING_DATA_KEY']}", 128 | content_type=os.environ["CONTENT_TYPE"], 129 | ) 130 | validation_input = TrainingInput( 131 | f"s3://{os.environ['ASSETS_BUCKET']}/{os.environ['TRAINING_DATA_KEY']}", 132 | content_type=os.environ["CONTENT_TYPE"], 133 | ) 134 | 135 | data_channels = {"train": train_input, "validation": validation_input} 136 | return data_channels 137 | 138 | 139 | @pytest.fixture() 140 | def mocked_tuner_config(mocked_tuning_job_env_vars): 141 | return json.loads(os.environ["TUNER_CONFIG"]) 142 | 143 | 144 | @pytest.fixture() 145 | def mocked_job_name(): 146 | return "test-training-job" 147 | 148 | 149 | @pytest.fixture() 150 | def mocked_raw_search_grid(mocked_tuning_job_env_vars): 151 | return json.loads(os.environ["HYPERPARAMETER_RANGES"]) 152 | 153 | 154 | @pytest.fixture() 155 | def mocked_hyperparameter_ranges(mocked_raw_search_grid): 156 | return SolutionModelTraining.format_search_grid(mocked_raw_search_grid) 157 | -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/lambdas/create_model_training_job/tests/test_create_model_training.py: -------------------------------------------------------------------------------- 1 | ####################################################################################################################### 2 | # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. # 3 | # # 4 | # Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance # 5 | # with the License. A copy of the License is located at # 6 | # # 7 | # http://www.apache.org/licenses/LICENSE-2.0 # 8 | # # 9 | # or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES # 10 | # OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions # 11 | # and limitations under the License. # 12 | # ##################################################################################################################### 13 | from unittest.mock import patch 14 | from unittest import TestCase 15 | from model_training_helper import TrainingType, SolutionModelTraining 16 | from tests.fixtures.training_fixtures import ( 17 | mocked_common_env_vars, 18 | mocked_training_job_env_vars, 19 | mocked_tuning_job_env_vars, 20 | mocked_estimator_config, 21 | mocked_hyperparameters, 22 | mocked_data_channels, 23 | mocked_tuner_config, 24 | mocked_job_name, 25 | mocked_raw_search_grid, 26 | mocked_hyperparameter_ranges, 27 | mocked_sagemaker_session, 28 | ) 29 | 30 | 31 | def test_create_estimator( 32 | mocked_estimator_config, 33 | mocked_hyperparameters, 34 | mocked_data_channels, 35 | mocked_job_name, 36 | ): 37 | job = SolutionModelTraining( 38 | job_name=mocked_job_name, 39 | estimator_config=mocked_estimator_config, 40 | hyperparameters=mocked_hyperparameters, 41 | data_channels=mocked_data_channels, 42 | ) 43 | 44 | # create the estimator 45 | estimator = job._create_estimator() 46 | 47 | # assert some of the properties 48 | assert estimator.image_uri == mocked_estimator_config["image_uri"] 49 | assert estimator.role == mocked_estimator_config["role"] 50 | assert estimator.instance_type == mocked_estimator_config["instance_type"] 51 | assert estimator.output_path == mocked_estimator_config["output_path"] 52 | 53 | 54 | def test_create_hyperparameter_tuner( 55 | mocked_estimator_config, 56 | mocked_hyperparameters, 57 | mocked_data_channels, 58 | mocked_tuner_config, 59 | mocked_job_name, 60 | mocked_hyperparameter_ranges, 61 | ): 62 | job = SolutionModelTraining( 63 | job_name=mocked_job_name, 64 | estimator_config=mocked_estimator_config, 65 | hyperparameters=mocked_hyperparameters, 66 | data_channels=mocked_data_channels, 67 | job_type=TrainingType.HyperparameterTuningJob, 68 | hyperparameter_tuner_config=mocked_tuner_config, 69 | hyperparameter_ranges=mocked_hyperparameter_ranges, 70 | ) 71 | # create the Hyperparameters tuner 72 | tuner = job._create_hyperparameter_tuner() 73 | # assert some of the properties 74 | assert tuner.max_parallel_jobs == 2 75 | assert tuner.max_jobs == 10 76 | assert tuner.strategy == "Bayesian" 77 | assert tuner.objective_type == "Maximize" 78 | TestCase().assertDictEqual( 79 | tuner._hyperparameter_ranges, mocked_hyperparameter_ranges 80 | ) 81 | 82 | 83 | def test_format_search_grid(mocked_raw_search_grid): 84 | formeated_grid = SolutionModelTraining.format_search_grid(mocked_raw_search_grid) 85 | # assert a Continuous parameter 86 | TestCase().assertListEqual( 87 | mocked_raw_search_grid["eta"][1], 88 | [formeated_grid["eta"].min_value, formeated_grid["eta"].max_value], 89 | ) 90 | # assert an Integer parameter 91 | TestCase().assertListEqual( 92 | mocked_raw_search_grid["max_depth"][1], 93 | [formeated_grid["max_depth"].min_value, formeated_grid["max_depth"].max_value], 94 | ) 95 | # assert a Categorical parameter 96 | TestCase().assertListEqual( 97 | mocked_raw_search_grid["optimizer"][1], formeated_grid["optimizer"].values 98 | ) 99 | 100 | 101 | @patch("model_training_helper.SolutionModelTraining._create_hyperparameter_tuner") 102 | @patch("model_training_helper.SolutionModelTraining._create_estimator") 103 | def test_create_training_job( 104 | mocked_create_estimator, 105 | mocked_create_tuner, 106 | mocked_estimator_config, 107 | mocked_hyperparameters, 108 | mocked_data_channels, 109 | mocked_tuner_config, 110 | mocked_job_name, 111 | mocked_hyperparameter_ranges, 112 | ): 113 | # assert the SolutionModelTraining._create_estimator is called for a training job 114 | training_job = SolutionModelTraining( 115 | job_name=mocked_job_name, 116 | estimator_config=mocked_estimator_config, 117 | hyperparameters=mocked_hyperparameters, 118 | data_channels=mocked_data_channels, 119 | ) 120 | 121 | training_job.create_training_job() 122 | mocked_create_estimator.assert_called() 123 | 124 | # assert the SolutionModelTraining._create_hyperparameter_tuner is called for a tuning job 125 | tuner_job = SolutionModelTraining( 126 | job_name=mocked_job_name, 127 | estimator_config=mocked_estimator_config, 128 | hyperparameters=mocked_hyperparameters, 129 | data_channels=mocked_data_channels, 130 | job_type=TrainingType.HyperparameterTuningJob, 131 | hyperparameter_tuner_config=mocked_tuner_config, 132 | hyperparameter_ranges=mocked_hyperparameter_ranges, 133 | ) 134 | tuner_job.create_training_job() 135 | mocked_create_tuner.assert_called() 136 | 137 | 138 | @patch("model_training_helper.SolutionModelTraining._create_estimator") 139 | @patch("main.Session") 140 | @patch("main.get_client") 141 | def test_handler_training_job( 142 | mocked_client, mocked_session, mocked_create_estimator, mocked_training_job_env_vars 143 | ): 144 | mocked_client.boto_region_name = "us-east-1" 145 | from main import handler 146 | 147 | handler(None, None) 148 | mocked_create_estimator.assert_called() 149 | -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/ml_pipelines/single_account_codepipeline.py: -------------------------------------------------------------------------------- 1 | # ##################################################################################################################### 2 | # Copyright 2021 Amazon.com, Inc. or its affiliates. All Rights Reserved. # 3 | # # 4 | # Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance # 5 | # with the License. A copy of the License is located at # 6 | # # 7 | # http://www.apache.org/licenses/LICENSE-2.0 # 8 | # # 9 | # or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES # 10 | # OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions # 11 | # and limitations under the License. # 12 | # ##################################################################################################################### 13 | from constructs import Construct 14 | from aws_cdk import ( 15 | Stack, 16 | Aws, 17 | CfnOutput, 18 | aws_iam as iam, 19 | aws_s3 as s3, 20 | aws_sns as sns, 21 | aws_events_targets as targets, 22 | aws_events as events, 23 | aws_codepipeline as codepipeline, 24 | ) 25 | from lib.blueprints.pipeline_definitions.source_actions import ( 26 | source_action_template, 27 | ) 28 | from lib.blueprints.pipeline_definitions.deploy_actions import ( 29 | create_cloudformation_action, 30 | ) 31 | from lib.blueprints.pipeline_definitions.helpers import ( 32 | pipeline_permissions, 33 | suppress_pipeline_bucket, 34 | suppress_iam_complex, 35 | suppress_cloudformation_action, 36 | ) 37 | from lib.blueprints.pipeline_definitions.templates_parameters import ( 38 | ParameteresFactory as pf, 39 | ) 40 | 41 | 42 | class SingleAccountCodePipelineStack(Stack): 43 | def __init__(self, scope: Construct, id: str, **kwargs) -> None: 44 | super().__init__(scope, id, **kwargs) 45 | 46 | # Parameteres # 47 | template_zip_name = pf.create_template_zip_name_parameter(self) 48 | template_file_name = pf.create_template_file_name_parameter(self) 49 | template_params_file_name = pf.create_stage_params_file_name_parameter( 50 | self, "TemplateParamsName", "main" 51 | ) 52 | assets_bucket_name = pf.create_assets_bucket_name_parameter(self) 53 | stack_name = pf.create_stack_name_parameter(self) 54 | sns_topic_arn = pf.create_sns_topic_arn_parameter(self) 55 | 56 | # Resources # 57 | assets_bucket = s3.Bucket.from_bucket_name( 58 | self, "ImportedAssetsBucket", assets_bucket_name.value_as_string 59 | ) 60 | 61 | # import the sns Topic 62 | pipeline_notification_topic = sns.Topic.from_topic_arn( 63 | self, "SinglePipelineNotification", sns_topic_arn.value_as_string 64 | ) 65 | 66 | # Defining pipeline stages 67 | # source stage 68 | source_output, source_action_definition = source_action_template( 69 | template_zip_name, assets_bucket 70 | ) 71 | 72 | # create cloudformation action 73 | cloudformation_action = create_cloudformation_action( 74 | "deploy_stack", 75 | stack_name.value_as_string, 76 | source_output, 77 | template_file_name.value_as_string, 78 | template_params_file_name.value_as_string, 79 | ) 80 | 81 | source_stage = codepipeline.StageProps( 82 | stage_name="Source", actions=[source_action_definition] 83 | ) 84 | deploy = codepipeline.StageProps( 85 | stage_name="DeployCloudFormation", 86 | actions=[cloudformation_action], 87 | ) 88 | 89 | single_account_pipeline = codepipeline.Pipeline( 90 | self, 91 | "SingleAccountPipeline", 92 | stages=[source_stage, deploy], 93 | cross_account_keys=False, 94 | ) 95 | 96 | # Add CF suppressions to the action 97 | cloudformation_action.deployment_role.node.find_child( 98 | "DefaultPolicy" 99 | ).node.default_child.cfn_options.metadata = { "cfn_nag": suppress_cloudformation_action() } 100 | 101 | # add notification to the single-account pipeline 102 | single_account_pipeline.on_state_change( 103 | "NotifyUser", 104 | description="Notify user of the outcome of the pipeline", 105 | target=targets.SnsTopic( 106 | pipeline_notification_topic, 107 | message=events.RuleTargetInput.from_text( 108 | ( 109 | f"Pipeline {events.EventField.from_path('$.detail.pipeline')} finished executing. " 110 | f"Pipeline execution result is {events.EventField.from_path('$.detail.state')}" 111 | ) 112 | ), 113 | ), 114 | event_pattern=events.EventPattern( 115 | detail={"state": ["SUCCEEDED", "FAILED"]} 116 | ), 117 | ) 118 | single_account_pipeline.add_to_role_policy( 119 | iam.PolicyStatement( 120 | actions=["events:PutEvents"], 121 | resources=[ 122 | f"arn:{Aws.PARTITION}:events:{Aws.REGION}:{Aws.ACCOUNT_ID}:event-bus/*", 123 | ], 124 | ) 125 | ) 126 | 127 | # add ArtifactBucket cfn supression (not needing a logging bucket) 128 | single_account_pipeline.node.find_child( 129 | "ArtifactsBucket" 130 | ).node.default_child.cfn_options.metadata = { "cfn_nag": suppress_pipeline_bucket() } 131 | 132 | # add supression for complex policy 133 | single_account_pipeline.node.find_child("Role").node.find_child( 134 | "DefaultPolicy" 135 | ).node.default_child.cfn_options.metadata = { "cfn_nag": suppress_iam_complex() } 136 | 137 | # attaching iam permissions to the pipelines 138 | pipeline_permissions(single_account_pipeline, assets_bucket) 139 | 140 | # Outputs # 141 | CfnOutput( 142 | self, 143 | id="Pipelines", 144 | value=( 145 | f"https://console.aws.amazon.com/codesuite/codepipeline/pipelines/" 146 | f"{single_account_pipeline.pipeline_name}/view?region={Aws.REGION}" 147 | ), 148 | ) 149 | -------------------------------------------------------------------------------- /deployment/run-unit-tests.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | ###################################################################################################################### 3 | # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. # 4 | # # 5 | # Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance # 6 | # with the License. A copy of the License is located at # 7 | # # 8 | # http://www.apache.org/licenses/LICENSE-2.0 # 9 | # # 10 | # or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES # 11 | # OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions # 12 | # and limitations under the License. # 13 | ###################################################################################################################### 14 | # 15 | # This script runs all tests for the root CDK project, as well as any microservices, Lambda functions, or dependency 16 | # source code packages. These include unit tests, integration tests, and snapshot tests. 17 | # 18 | # It is important that this script be tested and validated to ensure that all available test fixtures are run. 19 | # 20 | 21 | [ "$DEBUG" == 'true' ] && set -x 22 | set -e 23 | 24 | setup_python_env() { 25 | if [ -d "./.venv-test" ]; then 26 | echo "Reusing already setup python venv in ./.venv-test. Delete ./.venv-test if you want a fresh one created." 27 | return 28 | fi 29 | echo "Setting up python venv" 30 | python3 -m venv .venv-test 31 | echo "Initiating virtual environment" 32 | source .venv-test/bin/activate 33 | echo "upgrading pip -> python3 -m pip install --upgrade pip" 34 | python3 -m pip install --upgrade pip 35 | echo "Installing python packages" 36 | pip3 install -r requirements-test.txt 37 | pip3 install -r requirements.txt 38 | echo "deactivate virtual environment" 39 | deactivate 40 | } 41 | 42 | 43 | run_python_lambda_test() { 44 | lambda_name=$1 45 | lambda_description=$2 46 | run_python_test $source_dir/lambda $lambda_name 47 | } 48 | 49 | setup_and_activate_python_env() { 50 | module_path=$1 51 | cd $module_path 52 | 53 | [ "${CLEAN:-true}" = "true" ] && rm -fr .venv-test 54 | 55 | setup_python_env 56 | 57 | echo "Initiating virtual environment" 58 | source .venv-test/bin/activate 59 | } 60 | 61 | 62 | run_python_test() { 63 | module_path=$(pwd) 64 | module_name=$1 65 | echo "------------------------------------------------------------------------------" 66 | echo "[Test] Python path=$module_path module=$module_name" 67 | echo "------------------------------------------------------------------------------" 68 | 69 | coverage_report_path=$coverage_dir/$module_name.coverage.xml 70 | echo "coverage report path set to $coverage_report_path" 71 | 72 | # Use -vv for debugging 73 | python3 -m pytest --cov --cov-fail-under=80 --cov-report=term-missing --cov-report "xml:$coverage_report_path" 74 | if [ "$?" = "1" ]; then 75 | echo "(deployment/run-unit-tests.sh) ERROR: there is likely output above." 1>&2 76 | exit 1 77 | fi 78 | sed -i -e "s,$source_dir,source,g" $coverage_report_path 79 | } 80 | 81 | run_javascript_lambda_test() { 82 | lambda_name=$1 83 | lambda_description=$2 84 | echo "------------------------------------------------------------------------------" 85 | echo "[Test] Javascript Lambda: $lambda_name, $lambda_description" 86 | echo "------------------------------------------------------------------------------" 87 | cd $source_dir/lambda/$lambda_name 88 | [ "${CLEAN:-true}" = "true" ] && npm run clean 89 | npm ci 90 | npm test 91 | if [ "$?" = "1" ]; then 92 | echo "(deployment/run-unit-tests.sh) ERROR: there is likely output above." 1>&2 93 | exit 1 94 | fi 95 | [ "${CLEAN:-true}" = "true" ] && rm -fr coverage 96 | } 97 | 98 | 99 | run_cdk_project_test() { 100 | echo "------------------------------------------------------------------------------" 101 | echo "[Test] Running CDK tests" 102 | echo "------------------------------------------------------------------------------" 103 | 104 | # Test the Lambda functions 105 | cd $source_dir/infrastructure 106 | 107 | coverage_report_path=$coverage_dir/cdk.coverage.xml 108 | echo "coverage report path set to $coverage_report_path" 109 | 110 | cd $source_dir/infrastructure 111 | # Use -vv for debugging 112 | python3 -m pytest --cov --cov-fail-under=80 --cov-report=term-missing --cov-report "xml:$coverage_report_path" 113 | rm -rf *.egg-info 114 | sed -i -e "s,$source_dir,source,g" $coverage_report_path 115 | 116 | 117 | } 118 | 119 | run_framework_lambda_test() { 120 | echo "------------------------------------------------------------------------------" 121 | echo "[Test] Run framework lambda unit tests" 122 | echo "------------------------------------------------------------------------------" 123 | 124 | # Test the Lambda functions 125 | cd $source_dir/lambdas 126 | for folder in */ ; do 127 | cd "$folder" 128 | function_name=${PWD##*/} 129 | 130 | pip install -r requirements-test.txt 131 | run_python_test $(basename $folder) 132 | rm -rf *.egg-info 133 | 134 | cd .. 135 | done 136 | } 137 | 138 | run_blueprint_lambda_test() { 139 | echo "------------------------------------------------------------------------------" 140 | echo "[Test] Run blueprint lambda unit tests" 141 | echo "------------------------------------------------------------------------------" 142 | 143 | cd $source_dir/infrastructure/lib/blueprints/lambdas 144 | for folder in */ ; do 145 | echo "$folder" 146 | cd "$folder" 147 | if [ "$folder" != "sagemaker_layer/" ]; then 148 | pip install -r requirements-test.txt 149 | run_python_test $(basename $folder) 150 | rm -rf *.egg-info 151 | cd .. 152 | fi 153 | done 154 | } 155 | 156 | # Save the current working directory and set source directory 157 | starting_dir=$PWD 158 | cd ../source 159 | source_dir=$PWD 160 | 161 | # setup coverage report directory 162 | coverage_dir=$source_dir/test/coverage-reports 163 | mkdir -p $coverage_dir 164 | 165 | # Clean the test environment before running tests and after finished running tests 166 | # The variable is option with default of 'true'. It can be overwritten by caller 167 | # setting the CLEAN environment variable. For example 168 | # $ CLEAN=true ./run-unit-tests.sh 169 | # or 170 | # $ CLEAN=false ./run-unit-tests.sh 171 | # 172 | CLEAN="${CLEAN:-true}" 173 | 174 | setup_and_activate_python_env $source_dir 175 | python --version 176 | run_framework_lambda_test 177 | run_blueprint_lambda_test 178 | run_cdk_project_test 179 | 180 | # deactive Python envn 181 | deactivate 182 | 183 | 184 | # Return to the folder where where we started 185 | cd $starting_dir -------------------------------------------------------------------------------- /source/infrastructure/lib/blueprints/lambdas/create_model_training_job/main.py: -------------------------------------------------------------------------------- 1 | # ##################################################################################################################### 2 | # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. # 3 | # # 4 | # Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance # 5 | # with the License. A copy of the License is located at # 6 | # # 7 | # http://www.apache.org/licenses/LICENSE-2.0 # 8 | # # 9 | # or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES # 10 | # OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions # 11 | # and limitations under the License. # 12 | # ##################################################################################################################### 13 | import os 14 | import json 15 | from sagemaker import Session 16 | from sagemaker.inputs import TrainingInput 17 | from model_training_helper import TrainingType, SolutionModelTraining 18 | from shared.wrappers import exception_handler 19 | from shared.logger import get_logger 20 | from shared.helper import get_client 21 | 22 | logger = get_logger(__name__) 23 | 24 | 25 | # get the environment variables 26 | assets_bucket = os.environ["ASSETS_BUCKET"] 27 | job_name = os.environ["JOB_NAME"] 28 | type_map = dict(TrainingJob=1, HyperparameterTuningJob=2) 29 | job_type_str = os.environ.get("JOB_TYPE", "TrainingJob") 30 | job_type = TrainingType(type_map[job_type_str]) 31 | # get the estimator config 32 | role_arn = os.environ["ROLE_ARN"] 33 | s3_output_location = os.environ["JOB_OUTPUT_LOCATION"] 34 | image_uri = os.environ["IMAGE_URI"] 35 | instance_type = os.environ["INSTANCE_TYPE"] 36 | instance_count = int(os.environ.get("INSTANCE_COUNT", "1")) 37 | instance_volume_size = int(os.environ.get("INSTANCE_VOLUME_SIZE", "20")) 38 | kms_key_arn = os.environ.get("KMS_KEY_ARN") 39 | # max time in seconds the training job is allowed to run 40 | max_run = int(os.environ.get("JOB_MAX_RUN_SECONDS", "7200")) 41 | # use spot instances for training 42 | use_spot_instances_str = os.environ.get("USE_SPOT_INSTANCES", "True") 43 | use_spot_instances = False if use_spot_instances_str == "False" else True 44 | # encrypt inter container traffic 45 | encrypt_inter_container_traffic_str = os.environ.get("ENCRYPT_INTER_CONTAINER_TRAFFIC", "True") 46 | encrypt_inter_container_traffic = False if encrypt_inter_container_traffic_str == "False" else True 47 | # max wait time for Spot instances (required if use_spot_instances = True). Must be greater than max_run 48 | max_wait = int(os.environ.get("MAX_WAIT_SECONDS", str(2 * max_run))) if use_spot_instances else None 49 | # define checkpoint_s3_uri if use_spot_instances = True 50 | checkpoint_s3_uri = f"s3://{assets_bucket}/{s3_output_location}/{job_name}/checkpoint" if use_spot_instances else None 51 | 52 | # get the training data config 53 | training_dataset_key = os.environ["TRAINING_DATA_KEY"] 54 | validation_dataset_key = os.environ.get("VALIDATION_DATA_KEY") 55 | # create data config 56 | data_config = dict( 57 | content_type=os.environ.get("CONTENT_TYPE", "csv"), # MIME type of the input data 58 | distribution=os.environ.get( 59 | "DATA_DISTRIBUTION", "FullyReplicated" 60 | ), # valid values ‘FullyReplicated’, ‘ShardedByS3Key’ 61 | compression=os.environ.get("COMPRESSION_TYPE"), # valid values: ‘Gzip’, None. This is used only in Pipe input mode. 62 | record_wrapping=os.environ.get("DATA_RECORD_WRAPPING"), # valid values: 'RecordIO', None 63 | s3_data_type=os.environ.get( 64 | "S3_DATA_TYPE", "S3Prefix" 65 | ), # valid values: ‘S3Prefix’, ‘ManifestFile’, ‘AugmentedManifestFile’ 66 | input_mode=os.environ.get("DATA_INPUT_MODE"), # valid values 'File', 'Pipe', 'FastFile', None 67 | ) 68 | 69 | # A list of one or more attribute names to use that are found in a 70 | # specified AugmentedManifestFile (if s3_data_type="AugmentedManifestFile") 71 | attribute_names = os.environ.get("ATTRIBUTE_NAMES") 72 | 73 | # add it to data_config 74 | data_config["attribute_names"] = json.loads(attribute_names) if attribute_names else attribute_names 75 | 76 | # get training algorithm's hyperparameters 77 | hyperparameters = json.loads(os.environ["HYPERPARAMETERS"]) 78 | 79 | # create training inputs 80 | training_input = TrainingInput(s3_data=f"s3://{assets_bucket}/{training_dataset_key}", **data_config) 81 | 82 | # create data channels 83 | data_channels = {"train": training_input} 84 | 85 | # add validation data if provided 86 | if validation_dataset_key: 87 | validation_input = TrainingInput(s3_data=f"s3://{assets_bucket}/{validation_dataset_key}", **data_config) 88 | data_channels["validation"] = validation_input 89 | 90 | 91 | # get hyperparameter tuner config (if job_type="HyperparameterTuningJob") 92 | tuner_config_str = os.environ.get("TUNER_CONFIG") 93 | hyperparameter_tuner_config = json.loads(tuner_config_str) if job_type_str == "HyperparameterTuningJob" else None 94 | 95 | # get hyperparameter ranges 96 | hyperparameter_ranges_dict = os.environ.get("HYPERPARAMETER_RANGES") 97 | hyperparameter_ranges = ( 98 | SolutionModelTraining.format_search_grid(json.loads(hyperparameter_ranges_dict)) 99 | if job_type_str == "HyperparameterTuningJob" 100 | else None 101 | ) 102 | 103 | tags = json.loads(os.environ.get("TAGS")) if os.environ.get("TAGS") else None 104 | 105 | 106 | @exception_handler 107 | def handler(event, context): 108 | # create sagemaker boto3 client, to be passed to SageMaker session 109 | sm_client = get_client("sagemaker") 110 | 111 | # create estimator config 112 | estimator_config = dict( 113 | image_uri=image_uri, 114 | role=role_arn, 115 | instance_count=instance_count, 116 | instance_type=instance_type, 117 | volume_size=instance_volume_size, 118 | output_path=f"s3://{assets_bucket}/{s3_output_location}", 119 | volume_kms_key=kms_key_arn, 120 | output_kms_key=kms_key_arn, 121 | use_spot_instances=use_spot_instances, 122 | max_run=max_run, 123 | max_wait=max_wait, 124 | checkpoint_s3_uri=checkpoint_s3_uri, 125 | encrypt_inter_container_traffic=encrypt_inter_container_traffic, 126 | sagemaker_session=Session(sagemaker_client=sm_client), 127 | tags=tags, 128 | ) 129 | 130 | # create the training job 131 | logger.info("Creating the training job") 132 | job = SolutionModelTraining( 133 | job_name=job_name, 134 | estimator_config=estimator_config, 135 | hyperparameters=hyperparameters, 136 | data_channels=data_channels, 137 | job_type=job_type, 138 | hyperparameter_tuner_config=hyperparameter_tuner_config, 139 | hyperparameter_ranges=hyperparameter_ranges, 140 | ) 141 | 142 | # start the training job 143 | job.create_training_job() 144 | logger.info(f"Training job {job_name} started") 145 | --------------------------------------------------------------------------------