├── .gitignore ├── README.md └── terraform-pr-pipeline ├── README.md ├── iam-codebuild.tf ├── iam-codepipeline.tf ├── kms.tf ├── main.tf ├── outputs.tf ├── pipeline-create.tf ├── pipeline-create ├── buildspec-terraform-fmt.yml ├── buildspec-terraform-plan.yml ├── buildspec-terrascan.yml ├── pipeline-create.py └── requirements.txt ├── pipeline-delete.tf ├── pipeline-delete ├── pipeline-delete.py └── requirements.txt ├── poller-create.tf ├── poller-create ├── poller-create.py └── requirements.txt ├── poller-delete.tf ├── poller-delete ├── poller-delete.py └── requirements.txt ├── setup.sh └── variables.tf /.gitignore: -------------------------------------------------------------------------------- 1 | # Compiled files 2 | *.tfstate 3 | *.tfstate.backup 4 | 5 | # Module directory 6 | .terraform/ 7 | 8 | # Other ignore 9 | terraform.tfvars 10 | *.ignore 11 | .lambda-zip/ 12 | 13 | # Byte-compiled / optimized / DLL files 14 | __pycache__/ 15 | *.py[cod] 16 | *$py.class 17 | 18 | # C extensions 19 | *.so 20 | 21 | # Distribution / packaging 22 | .Python 23 | build/ 24 | develop-eggs/ 25 | dist/ 26 | downloads/ 27 | eggs/ 28 | .eggs/ 29 | lib/ 30 | lib64/ 31 | parts/ 32 | sdist/ 33 | var/ 34 | wheels/ 35 | *.egg-info/ 36 | .installed.cfg 37 | *.egg 38 | MANIFEST 39 | 40 | # PyInstaller 41 | # Usually these files are written by a python script from a template 42 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 43 | *.manifest 44 | *.spec 45 | 46 | # Installer logs 47 | pip-log.txt 48 | pip-delete-this-directory.txt 49 | 50 | # Unit test / coverage reports 51 | htmlcov/ 52 | .tox/ 53 | .coverage 54 | .coverage.* 55 | .cache 56 | nosetests.xml 57 | coverage.xml 58 | *.cover 59 | .hypothesis/ 60 | 61 | # Translations 62 | *.mo 63 | *.pot 64 | 65 | # Django stuff: 66 | *.log 67 | .static_storage/ 68 | .media/ 69 | local_settings.py 70 | 71 | # Flask stuff: 72 | instance/ 73 | .webassets-cache 74 | 75 | # Scrapy stuff: 76 | .scrapy 77 | 78 | # Sphinx documentation 79 | docs/_build/ 80 | 81 | # PyBuilder 82 | target/ 83 | 84 | # Jupyter Notebook 85 | .ipynb_checkpoints 86 | 87 | # pyenv 88 | .python-version 89 | 90 | # celery beat schedule file 91 | celerybeat-schedule 92 | 93 | # SageMath parsed files 94 | *.sage.py 95 | 96 | # Environments 97 | .env 98 | .venv 99 | env/ 100 | venv/ 101 | ENV/ 102 | env.bak/ 103 | venv.bak/ 104 | 105 | # Spyder project settings 106 | .spyderproject 107 | .spyproject 108 | 109 | # Rope project settings 110 | .ropeproject 111 | 112 | # mkdocs documentation 113 | /site 114 | 115 | # mypy 116 | .mypy_cache/ 117 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # terraform-aws-pipelines 2 | Collection of pipelines for infrastructure provisioning using terraform using AWS CodePipeline and CodeBuild. 3 | 4 | - terraform-pr-pipeline = Provisions CI/CD pipeline for terraform pull request reviews. 5 | -------------------------------------------------------------------------------- /terraform-pr-pipeline/README.md: -------------------------------------------------------------------------------- 1 | # AWS terraform pull request pipeline 2 | Provisions CI/CD pipeline for terraform pull request reviews. A new pipeline (AWS CodePipeline) is created for each new pull request in the given GitHub repository. The solution uses AWS lambda to sync contents of PRs with S3 and to create the pipeline. CodeBuild is used to check for terraform fmt, runs terrascan for static code analysis, and comments results into the PR. 3 | 4 | The pipeline will only perform tests on directories that contain modified files only. Testing that terraform plan/apply exesutes without errors will be done on /tests directories assuming that such directory exists on each template directory. Here's an example directory tree: 5 | ``` 6 | . 7 | |-- main.tf 8 | |-- tests 9 | | `-- main.tf 10 | |-- tf_templates 11 | | |-- main.tf 12 | | `-- tests 13 | | `-- main.tf 14 | |-- tf_another_dir 15 | | |-- main.tf 16 | | `-- tests 17 | | `-- main.tf 18 | `-- tf_yet_another_dir 19 | |-- main.tf 20 | `-- tests 21 | `-- main.tf 22 | ``` 23 | 24 | Prior to running the terraform templates for the first time. Execute setup.sh to prepopulate required zip files. Before the pipeline gets executed place your GitHub Personal Access Token (PAT) in the SSM parameter store (using a KMS key to encrypt it) at ${PROJECT_NAME}-terraform-pr-pat. Here's an AWS CLI example for PROJECT_NAME=therasec: `aws ssm put-parameter --name therasec-terraform-pr-pat --value THE_TOKEN --type SecureString --key-id THE_KMS_KEY_ID`. 25 | 26 | ## poller-create lambda 27 | Funtion that is triggered by default every 5 minutes to poll the repository for open pull requets. A zip file of latest commit for each pull request is saved into an S3 bucket. 28 | 29 | ## poller-delete lambda 30 | Function that is triggered by default every 60 minutes to remove zip files from S3 corresponding to pull requests no longer open. 31 | 32 | ## pipeline-create lambda 33 | Triggered each time there's a zip file uploaded to S3. This function creates an AWS CodePipeline pipeline if one doesn't exists yet for that pull request. 34 | 35 | ## pipeline-delete lambda 36 | Triggered each time there's a zip file deleted from S3. This function deletes the pipeline for closed PRs. 37 | 38 | 39 | ## Inputs 40 | 41 | | Name | Description | Type | Default | Required | 42 | |------|-------------|:----:|:-----:|:-----:| 43 | | aws_profile | AWS credentials profile to use | string | - | yes | 44 | | aws_region | AWS region where resources are provisioned | string | - | yes | 45 | | code_build_image | Docker image to use for CodeBuild container - Use http://amzn.to/2mjCI91 for reference | string | `aws/codebuild/ubuntu-base:14.04` | no | 46 | | github_api_url | API URL for GitHub | string | `https://api.github.com` | no | 47 | | github_repo_name | Name of the repository to track pull requests in org/repo format (e.g. cesar-rodriguez/test-repo) | string | - | yes | 48 | | poller_create_rate | Rate in minutes for polling the GitHub repository for open pull requests | string | `5` | no | 49 | | poller_delete_rate | Rate in minutes for polling the GitHub repository to check if PRs are still open | string | `60` | no | 50 | | project_name | All resources will be prepended with this name | string | - | yes | 51 | | terraform_download_url | URL for terraform version to be used for builds | string | `https://releases.hashicorp.com/terraform/0.11.1/terraform_0.11.1_linux_amd64.zip` | no | 52 | 53 | ## Outputs 54 | 55 | | Name | Description | 56 | |------|-------------| 57 | | pipeline_create_lambda | ARN for pipeline-create lambda function | 58 | | poller_create_lambda | ARN for poller-create lambda function | 59 | | poller_delete_lambda | ARN for poller-delete lambda function | 60 | | s3_bucket_name | Name of the s3 bucket used for storage of PRs | 61 | 62 | -------------------------------------------------------------------------------- /terraform-pr-pipeline/iam-codebuild.tf: -------------------------------------------------------------------------------- 1 | // Allows IAM roles to be assumed by lambda 2 | data "aws_iam_policy_document" "codebuild_assume_role" { 3 | statement { 4 | actions = ["sts:AssumeRole"] 5 | 6 | principals { 7 | type = "Service" 8 | 9 | identifiers = [ 10 | "codebuild.amazonaws.com", 11 | ] 12 | } 13 | } 14 | } 15 | 16 | // AWS IAM role for codebuild 17 | resource "aws_iam_role" "codebuild" { 18 | name = "${var.project_name}-codebuild" 19 | assume_role_policy = "${data.aws_iam_policy_document.codebuild_assume_role.json}" 20 | } 21 | 22 | data "aws_iam_policy_document" "codebuild_service" { 23 | statement { 24 | sid = "CloudWatchLogsPolicy" 25 | 26 | actions = [ 27 | "logs:CreateLogGroup", 28 | "logs:CreateLogStream", 29 | "logs:PutLogEvents", 30 | ] 31 | 32 | resources = [ 33 | "arn:aws:logs:${var.aws_region}:${data.aws_caller_identity.current.account_id}:log-group:/aws/codebuild/${var.project_name}-terraform-pr*", 34 | ] 35 | } 36 | 37 | statement { 38 | sid = "S3Policy" 39 | 40 | actions = [ 41 | "s3:GetObject", 42 | "s3:GetObjectVersion", 43 | "s3:PutObject", 44 | ] 45 | 46 | resources = [ 47 | "${aws_s3_bucket.bucket.arn}*", 48 | ] 49 | } 50 | 51 | statement { 52 | sid = "SSMAccess" 53 | 54 | actions = [ 55 | "ssm:GetParameters", 56 | ] 57 | 58 | resources = [ 59 | "arn:aws:ssm:${var.aws_region}:${data.aws_caller_identity.current.account_id}:parameter/${var.project_name}-terraform-pr*", 60 | ] 61 | } 62 | } 63 | 64 | resource "aws_iam_policy" "codebuild_service" { 65 | name = "${aws_iam_role.codebuild.name}-policy" 66 | policy = "${data.aws_iam_policy_document.codebuild_service.json}" 67 | } 68 | 69 | resource "aws_iam_role_policy_attachment" "codebuild_service" { 70 | role = "${aws_iam_role.codebuild.name}" 71 | policy_arn = "${aws_iam_policy.codebuild_service.arn}" 72 | } 73 | -------------------------------------------------------------------------------- /terraform-pr-pipeline/iam-codepipeline.tf: -------------------------------------------------------------------------------- 1 | // Allows IAM roles to be assumed by lambda 2 | data "aws_iam_policy_document" "codepipeline_assume_role" { 3 | statement { 4 | actions = ["sts:AssumeRole"] 5 | 6 | principals { 7 | type = "Service" 8 | 9 | identifiers = [ 10 | "codepipeline.amazonaws.com", 11 | ] 12 | } 13 | } 14 | } 15 | 16 | // AWS IAM role for codepipeline 17 | resource "aws_iam_role" "codepipeline" { 18 | name = "${var.project_name}-codepipeline" 19 | assume_role_policy = "${data.aws_iam_policy_document.codepipeline_assume_role.json}" 20 | } 21 | 22 | data "aws_iam_policy_document" "codepipeline_service" { 23 | statement { 24 | sid = "BuildPolicy" 25 | 26 | actions = [ 27 | "codebuild:StartBuild", 28 | "codebuild:StopBuild", 29 | "codebuild:BatchGetBuilds", 30 | ] 31 | 32 | resources = [ 33 | "arn:aws:codebuild:${var.aws_region}:${data.aws_caller_identity.current.account_id}:project/${var.project_name}-terraform-pr*", 34 | ] 35 | } 36 | 37 | statement { 38 | sid = "S3Policy" 39 | 40 | actions = [ 41 | "s3:Get*", 42 | "s3:Put*", 43 | ] 44 | 45 | resources = [ 46 | "${aws_s3_bucket.bucket.arn}*", 47 | ] 48 | } 49 | } 50 | 51 | resource "aws_iam_policy" "codepipeline_service" { 52 | name = "${aws_iam_role.codepipeline.name}-policy" 53 | policy = "${data.aws_iam_policy_document.codepipeline_service.json}" 54 | } 55 | 56 | resource "aws_iam_role_policy_attachment" "codepipeline_service" { 57 | role = "${aws_iam_role.codepipeline.name}" 58 | policy_arn = "${aws_iam_policy.codepipeline_service.arn}" 59 | } 60 | -------------------------------------------------------------------------------- /terraform-pr-pipeline/kms.tf: -------------------------------------------------------------------------------- 1 | resource "aws_kms_key" "pipeline_key" { 2 | description = "KMS key for ${var.project_name} terraform pr pipeline" 3 | deletion_window_in_days = 30 4 | policy = "${data.aws_iam_policy_document.pipeline_key.json}" 5 | enable_key_rotation = true 6 | } 7 | 8 | resource "aws_kms_alias" "pipeline_key" { 9 | name = "alias/${var.project_name}-terraform-pr" 10 | target_key_id = "${aws_kms_key.pipeline_key.key_id}" 11 | } 12 | 13 | data "aws_iam_policy_document" "pipeline_key" { 14 | statement { 15 | sid = "Key Administrators" 16 | 17 | actions = [ 18 | "kms:Create*", 19 | "kms:Describe*", 20 | "kms:Enable*", 21 | "kms:List*", 22 | "kms:Put*", 23 | "kms:Update*", 24 | "kms:Revoke*", 25 | "kms:Disable*", 26 | "kms:Get*", 27 | "kms:Delete*", 28 | "kms:TagResource", 29 | "kms:UntagResource", 30 | "kms:ScheduleKeyDeletion", 31 | "kms:CancelKeyDeletion", 32 | ] 33 | 34 | resources = ["*"] 35 | 36 | principals { 37 | type = "AWS" 38 | 39 | identifiers = [ 40 | "${data.aws_caller_identity.current.arn}", 41 | ] 42 | } 43 | } 44 | 45 | statement { 46 | sid = "Allows usage of key" 47 | 48 | actions = [ 49 | "kms:Encrypt", 50 | "kms:Decrypt", 51 | "kms:ReEncrypt*", 52 | "kms:GenerateDataKey*", 53 | "kms:DescribeKey", 54 | "kms:List*", 55 | "kms:Get*", 56 | ] 57 | 58 | resources = ["*"] 59 | 60 | principals { 61 | type = "AWS" 62 | 63 | identifiers = [ 64 | "${data.aws_caller_identity.current.arn}", 65 | "${aws_iam_role.codebuild.arn}", 66 | "${aws_iam_role.codepipeline.arn}", 67 | "${aws_iam_role.poller_create.arn}", 68 | "${aws_iam_role.poller_delete.arn}", 69 | "${aws_iam_role.pipeline_create.arn}", 70 | ] 71 | } 72 | } 73 | 74 | statement { 75 | sid = "Allows attachment of persistent resources" 76 | 77 | actions = [ 78 | "kms:CreateGrant", 79 | "kms:ListGrants", 80 | "kms:RevokeGrant", 81 | ] 82 | 83 | resources = ["*"] 84 | 85 | principals { 86 | type = "AWS" 87 | 88 | identifiers = [ 89 | "${data.aws_caller_identity.current.arn}", 90 | ] 91 | } 92 | } 93 | } 94 | -------------------------------------------------------------------------------- /terraform-pr-pipeline/main.tf: -------------------------------------------------------------------------------- 1 | /** 2 | # AWS terraform pull request pipeline 3 | Provisions CI/CD pipeline for terraform pull request reviews. A new pipeline (AWS CodePipeline) is created for each new pull request in the given GitHub repository. The solution uses AWS lambda to sync contents of PRs with S3 and to create the pipeline. CodeBuild is used to check for terraform fmt, runs terrascan for static code analysis, and comments results into the PR. 4 | 5 | The pipeline will only perform tests on directories that contain modified files only. Testing that terraform plan/apply exesutes without errors will be done on /tests directories assuming that such directory exists on each template directory. Here's an example directory tree: 6 | ``` 7 | . 8 | |-- main.tf 9 | |-- tests 10 | | `-- main.tf 11 | |-- tf_templates 12 | | |-- main.tf 13 | | `-- tests 14 | | `-- main.tf 15 | |-- tf_another_dir 16 | | |-- main.tf 17 | | `-- tests 18 | | `-- main.tf 19 | `-- tf_yet_another_dir 20 | |-- main.tf 21 | `-- tests 22 | `-- main.tf 23 | ``` 24 | 25 | Prior to running the terraform templates for the first time. Execute setup.sh to prepopulate required zip files. Before the pipeline gets executed place your GitHub Personal Access Token (PAT) in the SSM parameter store (using a KMS key to encrypt it) at ${PROJECT_NAME}-terraform-pr-pat. Here's an AWS CLI example for PROJECT_NAME=therasec: `aws ssm put-parameter --name therasec-terraform-pr-pat --value THE_TOKEN --type SecureString --key-id THE_KMS_KEY_ID`. 26 | 27 | ## poller-create lambda 28 | Funtion that is triggered by default every 5 minutes to poll the repository for open pull requets. A zip file of latest commit for each pull request is saved into an S3 bucket. 29 | 30 | ## poller-delete lambda 31 | Function that is triggered by default every 60 minutes to remove zip files from S3 corresponding to pull requests no longer open. 32 | 33 | ## pipeline-create lambda 34 | Triggered each time there's a zip file uploaded to S3. This function creates an AWS CodePipeline pipeline if one doesn't exists yet for that pull request. 35 | 36 | ## pipeline-delete lambda 37 | Triggered each time there's a zip file deleted from S3. This function deletes the pipeline for closed PRs. 38 | */ 39 | 40 | provider aws { 41 | profile = "${var.aws_profile}" 42 | region = "${var.aws_region}" 43 | } 44 | 45 | // Used to get AWS IAM account number 46 | data "aws_caller_identity" "current" {} 47 | 48 | // Stores PR zip files and terraform statefiles 49 | resource "aws_s3_bucket" "bucket" { 50 | bucket = "${var.project_name}-terraform-pr-pipeline" 51 | acl = "private" 52 | 53 | versioning { 54 | enabled = true 55 | } 56 | 57 | lifecycle_rule = { 58 | id = "deletein30days" 59 | prefix = "/" 60 | enabled = true 61 | 62 | expiration { 63 | days = "30" 64 | } 65 | } 66 | 67 | force_destroy = "True" 68 | } 69 | 70 | // Allows IAM roles to be assumed by lambda 71 | data "aws_iam_policy_document" "lambda_assume_role" { 72 | statement { 73 | actions = ["sts:AssumeRole"] 74 | 75 | principals { 76 | type = "Service" 77 | 78 | identifiers = [ 79 | "lambda.amazonaws.com", 80 | ] 81 | } 82 | } 83 | } 84 | 85 | // S3 trigger event config 86 | resource "aws_s3_bucket_notification" "pipeline_create_delete" { 87 | bucket = "${aws_s3_bucket.bucket.id}" 88 | 89 | lambda_function { 90 | lambda_function_arn = "${aws_lambda_function.pipeline_create.arn}" 91 | events = ["s3:ObjectCreated:*"] 92 | filter_suffix = "repo.zip" 93 | } 94 | 95 | lambda_function { 96 | lambda_function_arn = "${aws_lambda_function.pipeline_delete.arn}" 97 | events = ["s3:ObjectRemoved:*"] 98 | filter_suffix = "repo.zip" 99 | } 100 | } 101 | -------------------------------------------------------------------------------- /terraform-pr-pipeline/outputs.tf: -------------------------------------------------------------------------------- 1 | // IAM role used by codebuild 2 | output "iam_role_codebuild" { 3 | value = "${aws_iam_role.codebuild.arn}" 4 | } 5 | 6 | // IAM role used by codepipeline 7 | output "iam_role_codepipeline" { 8 | value = "${aws_iam_role.codepipeline.arn}" 9 | } 10 | 11 | // Pipeline KMS Key ID 12 | output "kms_key_id" { 13 | value = "${aws_kms_key.pipeline_key.id}" 14 | } 15 | 16 | // Pipeline KMS Key ARN 17 | output "kms_key_arn" { 18 | value = "${aws_kms_key.pipeline_key.arn}" 19 | } 20 | 21 | // Name of the s3 bucket used for storage of PRs and pipeline artifacts 22 | output "s3_bucket_name" { 23 | value = "${aws_s3_bucket.bucket.id}" 24 | } 25 | 26 | // ARN of the s3 bucket used for storage of PRs and pipeline artifacts 27 | output "s3_bucket_arn" { 28 | value = "${aws_s3_bucket.bucket.arn}" 29 | } 30 | 31 | // ARN for pipeline-create lambda function 32 | output "pipeline_create_lambda" { 33 | value = "${aws_lambda_function.pipeline_create.arn}" 34 | } 35 | 36 | // ARN of the pipeline-create IAM role 37 | output "pipeline_create_iam_role_arn" { 38 | value = "${aws_iam_role.pipeline_create.arn}" 39 | } 40 | 41 | // ARN for pipeline-delete lambda function 42 | output "pipeline_delete_lambda" { 43 | value = "${aws_lambda_function.pipeline_delete.arn}" 44 | } 45 | 46 | // ARN of the pipeline-delete IAM role 47 | output "pipeline_delete_iam_role_arn" { 48 | value = "${aws_iam_role.pipeline_delete.arn}" 49 | } 50 | 51 | // ARN for poller-create lambda function 52 | output "poller_create_lambda" { 53 | value = "${aws_lambda_function.poller_create.arn}" 54 | } 55 | 56 | // ARN of the poller-create IAM role 57 | output "poller_create_iam_role_arn" { 58 | value = "${aws_iam_role.poller_create.arn}" 59 | } 60 | 61 | // ARN for poller-delete lambda function 62 | output "poller_delete_lambda" { 63 | value = "${aws_lambda_function.poller_delete.arn}" 64 | } 65 | 66 | // ARN of the poller-delete IAM role 67 | output "poller_delete_iam_role_arn" { 68 | value = "${aws_iam_role.poller_delete.arn}" 69 | } 70 | -------------------------------------------------------------------------------- /terraform-pr-pipeline/pipeline-create.tf: -------------------------------------------------------------------------------- 1 | /** 2 | Resources for the pipeline-create lambda function 3 | */ 4 | 5 | resource "aws_lambda_function" "pipeline_create" { 6 | depends_on = ["null_resource.pipeline_create_dependencies"] 7 | filename = ".lambda-zip/pipeline-create.zip" 8 | function_name = "${var.project_name}-pipeline-create" 9 | role = "${aws_iam_role.pipeline_create.arn}" 10 | handler = "pipeline-create.lambda_handler" 11 | source_code_hash = "${base64sha256(file("${path.module}/.lambda-zip/pipeline-create.zip"))}" 12 | runtime = "python3.6" 13 | kms_key_arn = "${aws_kms_key.pipeline_key.arn}" 14 | 15 | tags { 16 | Name = "${var.project_name}-pipeline-create" 17 | } 18 | 19 | environment { 20 | variables = { 21 | GITHUB_API_URL = "${var.github_api_url}" 22 | GITHUB_REPO_NAME = "${var.github_repo_name}" 23 | BUCKET_NAME = "${aws_s3_bucket.bucket.id}" 24 | PROJECT_NAME = "${var.project_name}" 25 | CODE_BUILD_IMAGE = "${var.code_build_image}" 26 | TERRAFORM_DOWNLOAD_URL = "${var.terraform_download_url}" 27 | CODEBUILD_SERVICE_ROLE = "${aws_iam_role.codebuild.arn}" 28 | CODEPIPELINE_SERVICE_ROLE = "${aws_iam_role.codepipeline.arn}" 29 | KMS_KEY = "${aws_kms_key.pipeline_key.arn}" 30 | } 31 | } 32 | } 33 | 34 | // Allows s3 to trigger the function 35 | resource "aws_lambda_permission" "pipeline_create" { 36 | statement_id = "schedule" 37 | action = "lambda:InvokeFunction" 38 | function_name = "${aws_lambda_function.pipeline_create.function_name}" 39 | principal = "s3.amazonaws.com" 40 | source_arn = "${aws_s3_bucket.bucket.arn}" 41 | } 42 | 43 | // Creates zip file with dependencies every time pipeline-create.py is updated 44 | resource "null_resource" "pipeline_create_dependencies" { 45 | triggers { 46 | lambda_function = "${file("${path.module}/pipeline-create/pipeline-create.py")}" 47 | } 48 | 49 | provisioner "local-exec" { 50 | command = "rm -rf ${path.module}/.lambda-zip/pipeline-create-resources" 51 | } 52 | 53 | provisioner "local-exec" { 54 | command = "mkdir -p ${path.module}/.lambda-zip/pipeline-create-resources" 55 | } 56 | 57 | provisioner "local-exec" { 58 | command = "pip install --target=${path.module}/.lambda-zip/pipeline-create-resources requests" 59 | } 60 | 61 | provisioner "local-exec" { 62 | command = "cp -R ${path.module}/pipeline-create/pipeline-create.py ${path.module}/.lambda-zip/pipeline-create-resources/." 63 | } 64 | 65 | provisioner "local-exec" { 66 | command = "cp -R ${path.module}/pipeline-create/*.yml ${path.module}/.lambda-zip/pipeline-create-resources/." 67 | } 68 | 69 | provisioner "local-exec" { 70 | command = "cd ${path.module}/.lambda-zip/pipeline-create-resources/ && zip -r ../pipeline-create.zip ." 71 | } 72 | } 73 | 74 | // AWS IAM role for pipeline-create function 75 | resource "aws_iam_role" "pipeline_create" { 76 | name = "${var.project_name}-pipeline-create-lambda" 77 | assume_role_policy = "${data.aws_iam_policy_document.lambda_assume_role.json}" 78 | } 79 | 80 | data "aws_iam_policy_document" "pipeline_create" { 81 | statement { 82 | sid = "CreateLogs" 83 | 84 | actions = [ 85 | "logs:CreateLogGroup", 86 | "logs:CreateLogStream", 87 | "logs:PutLogEvents", 88 | ] 89 | 90 | resources = [ 91 | "arn:aws:logs:${var.aws_region}:${data.aws_caller_identity.current.account_id}:log-group:/aws/lambda/${var.project_name}-pipeline-create:*", 92 | ] 93 | } 94 | 95 | statement { 96 | sid = "CodeBuildAccess" 97 | 98 | actions = [ 99 | "codebuild:CreateProject", 100 | "codebuild:BatchGetProjects", 101 | ] 102 | 103 | resources = [ 104 | "arn:aws:codebuild:${var.aws_region}:${data.aws_caller_identity.current.account_id}:project/${var.project_name}*", 105 | ] 106 | } 107 | 108 | statement { 109 | sid = "CodePipelineAccess" 110 | 111 | actions = [ 112 | "codepipeline:GetPipeline", 113 | "codepipeline:CreatePipeline", 114 | ] 115 | 116 | resources = [ 117 | "arn:aws:codepipeline:${var.aws_region}:${data.aws_caller_identity.current.account_id}:${var.project_name}*", 118 | ] 119 | } 120 | 121 | statement { 122 | sid = "iam" 123 | 124 | actions = [ 125 | "iam:PassRole", 126 | ] 127 | 128 | resources = [ 129 | "${aws_iam_role.codebuild.arn}", 130 | "${aws_iam_role.codepipeline.arn}", 131 | ] 132 | } 133 | } 134 | 135 | resource "aws_iam_policy" "pipeline_create" { 136 | name = "${aws_iam_role.pipeline_create.name}-policy" 137 | policy = "${data.aws_iam_policy_document.pipeline_create.json}" 138 | } 139 | 140 | resource "aws_iam_role_policy_attachment" "pipeline_create" { 141 | role = "${aws_iam_role.pipeline_create.name}" 142 | policy_arn = "${aws_iam_policy.pipeline_create.arn}" 143 | } 144 | -------------------------------------------------------------------------------- /terraform-pr-pipeline/pipeline-create/buildspec-terraform-fmt.yml: -------------------------------------------------------------------------------- 1 | version: 0.2 2 | 3 | phases: 4 | install: 5 | commands: 6 | - curl -o terraform.zip ${TERRAFORM_DOWNLOAD_URL} && unzip terraform.zip 7 | - mv terraform /usr/local/bin 8 | - apt-get -qq update 9 | - apt-get -qq install jq 10 | pre_build: 11 | commands: 12 | - export BUILD_URL="https://console.aws.amazon.com/codebuild/home?region=${AWS_REGION}#/builds/${CODEBUILD_BUILD_ID}/view/new" 13 | - export SHA=$( aws s3api get-object-tagging --bucket $S3_BUCKET --key $PR_NUMBER/repo.zip | jq '.TagSet[] | select(.Key=="latest_commit")' | jq -r '.Value' ) 14 | - "jq -n -r --arg url \"$BUILD_URL\" '{ state: \"pending\", target_url: $url, description: \"Checks that terraform templates are formatted\", context: \"terraform-fmt\"}' > data.json" 15 | - "curl -d \"@data.json\" -H \"Content-Type: application/json\" -H \"Authorization: token ${GITHUB_PAT}\" -X POST \"${GITHUB_API_URL}/repos/${REPO_OWNER}/${REPO}/statuses/${SHA}\"" 16 | build: 17 | commands: 18 | - repo=$( ls | grep ${REPO_NAME}) 19 | - cd $repo 20 | - echo $MODIFIED_DIRS 21 | - for dir in $MODIFIED_DIRS; do terraform fmt $dir | tee -a results.log; done 22 | - if [ -s results.log ]; then echo "Unformatted templates detected" && exit 1; fi 23 | post_build: 24 | commands: 25 | - if [ $CODEBUILD_BUILD_SUCCEEDING -eq 1 ]; then export STATE="success"; else export STATE="failure"; fi 26 | - "jq -n -r --arg state \"${STATE}\" --arg url \"$BUILD_URL\" '{ state: $state, target_url: $url, description: \"Checks that terraform templates are formatted\", context: \"terraform-fmt\"}' > data.json" 27 | - "curl -d \"@data.json\" -H \"Content-Type: application/json\" -H \"Authorization: token ${GITHUB_PAT}\" -X POST $GITHUB_API_URL/repos/$REPO_OWNER/$REPO/statuses/$SHA" 28 | - echo "# terraform-fmt results for $SHA\nThe following files need to be formatted:\n" > results.txt && cat results.log >> results.txt 29 | - "jq -n -r --arg body \"$(cat results.txt)\" '{ body: $body }' > data.json" 30 | - "if [ $CODEBUILD_BUILD_SUCCEEDING -eq 0 ]; then curl -d \"@data.json\" -H \"Content-Type: application/json\" -H \"Authorization: token ${GITHUB_PAT}\" -X POST $GITHUB_API_URL/repos/$REPO_OWNER/$REPO/issues/$PR_NUMBER/comments ; fi" 31 | -------------------------------------------------------------------------------- /terraform-pr-pipeline/pipeline-create/buildspec-terraform-plan.yml: -------------------------------------------------------------------------------- 1 | version: 0.2 2 | 3 | phases: 4 | install: 5 | commands: 6 | - curl -o terraform.zip ${TERRAFORM_DOWNLOAD_URL} && unzip terraform.zip 7 | - mv terraform /usr/local/bin 8 | - apt-get -qq update 9 | - apt-get -qq install jq 10 | pre_build: 11 | commands: 12 | - export BUILD_URL="https://console.aws.amazon.com/codebuild/home?region=${AWS_REGION}#/builds/${CODEBUILD_BUILD_ID}/view/new" 13 | - export SHA=$( aws s3api get-object-tagging --bucket $S3_BUCKET --key $PR_NUMBER/repo.zip | jq '.TagSet[] | select(.Key=="latest_commit")' | jq -r '.Value' ) 14 | - "jq -n -r --arg url \"$BUILD_URL\" '{ state: \"pending\", target_url: $url, description: \"Checks that terraform plan executes without errors\", context: \"terraform-plan\"}' > data.json" 15 | - "curl -d \"@data.json\" -H \"Content-Type: application/json\" -H \"Authorization: token ${GITHUB_PAT}\" -X POST \"${GITHUB_API_URL}/repos/${REPO_OWNER}/${REPO}/statuses/${SHA}\"" 16 | build: 17 | commands: 18 | - repo=$( ls | grep ${REPO_NAME}) 19 | - cd $repo 20 | - export REPO_DIR=$(pwd) 21 | - echo $MODIFIED_TEST_DIRS 22 | - for dir in $MODIFIED_TEST_DIRS; do echo "\n## Running in - $dir\n### Terraform init\n" | tee -a results-plan.log && cd $dir && terraform init -input=false -no-color | tee -a $REPO_DIR/results-plan.log && echo "\n### Terraform plan\n" | tee -a $REPO_DIR/results-plan.log && terraform plan -input=false -no-color -out=tfplan 2>&1 | tee -a $REPO_DIR/results-plan.log && echo "\n### Terraform apply\n" | tee -a $REPO_DIR/results-plan.log && terraform apply -input=false -no-color tfplan 2>&1 | tee -a $REPO_DIR/results-plan.log && echo "\n### Terraform destroy\n" | tee -a $REPO_DIR/results-plan.log && terraform destroy -input=false -no-color -force 2>&1 | tee -a $REPO_DIR/results-plan.log && cd $REPO_DIR; done 23 | - if [ $(cat results-plan.log | grep "Error" | wc -l) -gt 0 ]; then echo "Error in terraform plan" && exit 1; fi 24 | post_build: 25 | commands: 26 | - if [ $CODEBUILD_BUILD_SUCCEEDING -eq 1 ]; then export STATE="success"; else export STATE="failure"; fi 27 | - "jq -n -r --arg state \"${STATE}\" --arg url \"$BUILD_URL\" '{ state: $state, target_url: $url, description: \"Checks that terraform plan executes without errors\", context: \"terraform-plan\"}' > data.json" 28 | - "curl -d \"@data.json\" -H \"Content-Type: application/json\" -H \"Authorization: token ${GITHUB_PAT}\" -X POST $GITHUB_API_URL/repos/$REPO_OWNER/$REPO/statuses/$SHA" 29 | - echo "# terraform-plan results results for ${SHA}\n" > results.txt && cat results-plan.log >> results.txt 30 | - "jq -n -r --arg body \"$(cat results.txt)\" '{ body: $body }' > data.json" 31 | - "curl -d \"@data.json\" -H \"Content-Type: application/json\" -H \"Authorization: token ${GITHUB_PAT}\" -X POST $GITHUB_API_URL/repos/$REPO_OWNER/$REPO/issues/$PR_NUMBER/comments" 32 | -------------------------------------------------------------------------------- /terraform-pr-pipeline/pipeline-create/buildspec-terrascan.yml: -------------------------------------------------------------------------------- 1 | version: 0.2 2 | 3 | phases: 4 | install: 5 | commands: 6 | - curl -o terraform.zip ${TERRAFORM_DOWNLOAD_URL} && unzip terraform.zip 7 | - mv terraform /usr/local/bin 8 | - apt-get -qq update 9 | - apt-get -qq install python3 python3-pip jq 10 | - pip3 install terrascan -q 11 | pre_build: 12 | commands: 13 | - export BUILD_URL="https://console.aws.amazon.com/codebuild/home?region=${AWS_REGION}#/builds/${CODEBUILD_BUILD_ID}/view/new" 14 | - export SHA=$( aws s3api get-object-tagging --bucket $S3_BUCKET --key $PR_NUMBER/repo.zip | jq '.TagSet[] | select(.Key=="latest_commit")' | jq -r '.Value' ) 15 | - "jq -n -r --arg url \"$BUILD_URL\" '{ state: \"pending\", target_url: $url, description: \"Checks that terrascan runs without errors\", context: \"terrascan\"}' > data.json" 16 | - "curl -d \"@data.json\" -H \"Content-Type: application/json\" -H \"Authorization: token ${GITHUB_PAT}\" -X POST \"${GITHUB_API_URL}/repos/${REPO_OWNER}/${REPO}/statuses/${SHA}\"" 17 | build: 18 | commands: 19 | - repo=$( ls | grep ${REPO_NAME}) 20 | - cd $repo 21 | - echo $MODIFIED_DIRS 22 | - for dir in $MODIFIED_DIRS; do echo "## Scanning $dir\n" >> results.log && terrascan -l $dir 2>&1 | tee -a results.log; done 23 | - if [ $(cat results.log | grep "FAILED" | wc -l) -gt 0 ]; then echo "terrascan test failed" && exit 1; fi 24 | post_build: 25 | commands: 26 | - if [ $CODEBUILD_BUILD_SUCCEEDING -eq 1 ]; then export STATE="success"; else export STATE="failure"; fi 27 | - "jq -n -r --arg state \"${STATE}\" --arg url \"$BUILD_URL\" '{ state: $state, target_url: $url, description: \"Checks that terrascan runs without errors\", context: \"terrascan\"}' > data.json" 28 | - "curl -d \"@data.json\" -H \"Content-Type: application/json\" -H \"Authorization: token ${GITHUB_PAT}\" -X POST $GITHUB_API_URL/repos/$REPO_OWNER/$REPO/statuses/$SHA" 29 | - echo "# terrascan results for $SHA\n" > results.txt && cat results.log >> results.txt 30 | - "jq -n -r --arg body \"$(cat results.txt)\" '{ body: $body }' > data.json" 31 | - "curl -d \"@data.json\" -H \"Content-Type: application/json\" -H \"Authorization: token ${GITHUB_PAT}\" -X POST $GITHUB_API_URL/repos/$REPO_OWNER/$REPO/issues/$PR_NUMBER/comments" 32 | -------------------------------------------------------------------------------- /terraform-pr-pipeline/pipeline-create/pipeline-create.py: -------------------------------------------------------------------------------- 1 | import os 2 | import logging 3 | import re 4 | import boto3 5 | import requests 6 | 7 | # Configuring logger 8 | logger = logging.getLogger() 9 | logger.setLevel(logging.INFO) 10 | logger.handlers[0].setFormatter(logging.Formatter( 11 | '%(asctime)s - %(name)s - %(levelname)s - %(message)s\n' 12 | )) 13 | logging.getLogger('boto3').setLevel(logging.ERROR) 14 | logging.getLogger('botocore').setLevel(logging.ERROR) 15 | 16 | # Boto clients 17 | codepipeline = boto3.client('codepipeline') 18 | codebuild = boto3.client('codebuild') 19 | 20 | # Global vars 21 | github_api_url = os.environ['GITHUB_API_URL'] 22 | repo = os.environ['GITHUB_REPO_NAME'] 23 | bucket = os.environ['BUCKET_NAME'] 24 | project_name = os.environ['PROJECT_NAME'] 25 | code_build_image = os.environ['CODE_BUILD_IMAGE'] 26 | terraform_download_url = os.environ['TERRAFORM_DOWNLOAD_URL'] 27 | codebuild_service_role = os.environ['CODEBUILD_SERVICE_ROLE'] 28 | codepipeline_service_role = os.environ['CODEPIPELINE_SERVICE_ROLE'] 29 | kms_key = os.environ['KMS_KEY'] 30 | 31 | 32 | def pipeline_exists(pr_number): 33 | """Returns True if AWS CodePipeline pipeline exists""" 34 | try: 35 | codepipeline.get_pipeline(name='{}-terraform-pr-{}'.format( 36 | project_name, 37 | pr_number)) 38 | except codepipeline.exceptions.PipelineNotFoundException: 39 | return False 40 | return True 41 | 42 | 43 | def codebuild_project_exists(pr_number, name): 44 | """ 45 | Returns True if AWS CodeBuild project exists 46 | """ 47 | results = codebuild.batch_get_projects(names=[ 48 | '{}-terraform-pr-{}-{}'.format(project_name, name, pr_number) 49 | ]) 50 | if results['projectsNotFound'] == []: 51 | return True 52 | return False 53 | 54 | 55 | def remove_list_duplicates(input_list): 56 | """Removes duplicates entries from list""" 57 | return list(dict.fromkeys(input_list)) 58 | 59 | 60 | def get_modified_directories(pr_number): 61 | """ 62 | Returns a dict containing paths to the modified directories from the PR 63 | """ 64 | # Get pull request info 65 | response = requests.get('https://api.github.com/repos/{}/pulls/{}'.format( 66 | repo, 67 | pr_number 68 | )) 69 | # Get Diffs 70 | diff_url = response.json()['diff_url'] 71 | response = requests.get(diff_url) 72 | 73 | # Determine all counts of files modified 74 | expression = r'( b\/[^\s][^\\]*)+' 75 | files_modified = [] 76 | for obj in re.finditer(expression, '{}'.format(response.content)): 77 | if '.tf' in obj.group(1): 78 | files_modified.append(obj.group(1)[3:]) 79 | files_modified = remove_list_duplicates(files_modified) 80 | 81 | # Get DIRs (for terraform fmt) 82 | dirs = [] 83 | test_dirs = [] 84 | for file in files_modified: 85 | dir_only = file.split('/')[:-1] 86 | if dir_only == []: 87 | dirs.append('.') 88 | test_dirs.append('tests') 89 | continue 90 | dirs.append('/'.join(dir_only)) 91 | if 'tests' in file: 92 | test_dirs.append('/'.join(dir_only)) 93 | else: 94 | test_dirs.append( 95 | '{}/tests'.format( 96 | '/'.join(dir_only) 97 | )) 98 | dirs = remove_list_duplicates(dirs) 99 | test_dirs = remove_list_duplicates(test_dirs) 100 | 101 | return { 102 | 'dirs': dirs, 103 | 'test_dirs': test_dirs 104 | } 105 | 106 | 107 | def create_pipeline(pr_number): 108 | """ 109 | Creates pipeline and codebuild resources 110 | """ 111 | modified_dirs = get_modified_directories(pr_number) 112 | dirs = modified_dirs['dirs'] 113 | test_dirs = modified_dirs['test_dirs'] 114 | if not codebuild_project_exists(pr_number, 'fmt'): 115 | logger.info('Creating terraform-fmt codebuild project') 116 | with open('buildspec-terraform-fmt.yml', 'r') as buildspecfile: 117 | buildspec = buildspecfile.read() 118 | codebuild.create_project( 119 | name='{}-terraform-pr-fmt-{}'.format(project_name, pr_number), 120 | description='Checks if code is formatted', 121 | source={ 122 | 'type': 'CODEPIPELINE', 123 | 'buildspec': buildspec, 124 | }, 125 | artifacts={ 126 | 'type': 'CODEPIPELINE', 127 | }, 128 | environment={ 129 | 'type': 'LINUX_CONTAINER', 130 | 'image': code_build_image, 131 | 'computeType': 'BUILD_GENERAL1_SMALL', 132 | 'environmentVariables': [ 133 | { 134 | 'name': 'GITHUB_API_URL', 135 | 'value': github_api_url, 136 | 'type': 'PLAINTEXT' 137 | }, 138 | { 139 | 'name': 'GITHUB_PAT', 140 | 'value': '{}-terraform-pr-pat'.format(project_name), 141 | 'type': 'PARAMETER_STORE' 142 | }, 143 | { 144 | 'name': 'MODIFIED_DIRS', 145 | 'value': ' '.join(dirs), 146 | 'type': 'PLAINTEXT' 147 | }, 148 | { 149 | 'name': 'MODIFIED_TEST_DIRS', 150 | 'value': ' '.join(test_dirs), 151 | 'type': 'PLAINTEXT' 152 | }, 153 | { 154 | 'name': 'PR_NUMBER', 155 | 'value': pr_number, 156 | 'type': 'PLAINTEXT' 157 | }, 158 | { 159 | 'name': 'REPO', 160 | 'value': repo.split('/')[1], 161 | 'type': 'PLAINTEXT' 162 | }, 163 | { 164 | 'name': 'REPO_NAME', 165 | 'value': repo.replace('/', '-'), 166 | 'type': 'PLAINTEXT' 167 | }, 168 | { 169 | 'name': 'REPO_OWNER', 170 | 'value': repo.split('/')[0], 171 | 'type': 'PLAINTEXT' 172 | }, 173 | { 174 | 'name': 'S3_BUCKET', 175 | 'value': bucket, 176 | 'type': 'PLAINTEXT' 177 | }, 178 | { 179 | 'name': 'TERRAFORM_DOWNLOAD_URL', 180 | 'value': terraform_download_url, 181 | 'type': 'PLAINTEXT' 182 | }, 183 | { 184 | 'name': 'TF_IN_AUTOMATION', 185 | 'value': 'True', 186 | 'type': 'PLAINTEXT' 187 | }, 188 | ] 189 | }, 190 | serviceRole=codebuild_service_role, 191 | timeoutInMinutes=5, 192 | encryptionKey=kms_key, 193 | ) 194 | 195 | if not codebuild_project_exists(pr_number, 'terrascan'): 196 | logger.info('Creating terrascan codebuild project') 197 | with open('buildspec-terrascan.yml', 'r') as buildspecfile: 198 | buildspec = buildspecfile.read() 199 | codebuild.create_project( 200 | name='{}-terraform-pr-terrascan-{}'.format( 201 | project_name, pr_number), 202 | description='Runs terrascan against PR', 203 | source={ 204 | 'type': 'CODEPIPELINE', 205 | 'buildspec': buildspec, 206 | }, 207 | artifacts={ 208 | 'type': 'CODEPIPELINE', 209 | }, 210 | environment={ 211 | 'type': 'LINUX_CONTAINER', 212 | 'image': code_build_image, 213 | 'computeType': 'BUILD_GENERAL1_SMALL', 214 | 'environmentVariables': [ 215 | { 216 | 'name': 'GITHUB_API_URL', 217 | 'value': github_api_url, 218 | 'type': 'PLAINTEXT' 219 | }, 220 | { 221 | 'name': 'GITHUB_PAT', 222 | 'value': '{}-terraform-pr-pat'.format(project_name), 223 | 'type': 'PARAMETER_STORE' 224 | }, 225 | { 226 | 'name': 'MODIFIED_DIRS', 227 | 'value': ' '.join(dirs), 228 | 'type': 'PLAINTEXT' 229 | }, 230 | { 231 | 'name': 'MODIFIED_TEST_DIRS', 232 | 'value': ' '.join(test_dirs), 233 | 'type': 'PLAINTEXT' 234 | }, 235 | { 236 | 'name': 'PR_NUMBER', 237 | 'value': pr_number, 238 | 'type': 'PLAINTEXT' 239 | }, 240 | { 241 | 'name': 'REPO', 242 | 'value': repo.split('/')[1], 243 | 'type': 'PLAINTEXT' 244 | }, 245 | { 246 | 'name': 'REPO_NAME', 247 | 'value': repo.replace('/', '-'), 248 | 'type': 'PLAINTEXT' 249 | }, 250 | { 251 | 'name': 'REPO_OWNER', 252 | 'value': repo.split('/')[0], 253 | 'type': 'PLAINTEXT' 254 | }, 255 | { 256 | 'name': 'S3_BUCKET', 257 | 'value': bucket, 258 | 'type': 'PLAINTEXT' 259 | }, 260 | { 261 | 'name': 'TERRAFORM_DOWNLOAD_URL', 262 | 'value': terraform_download_url, 263 | 'type': 'PLAINTEXT' 264 | }, 265 | { 266 | 'name': 'TF_IN_AUTOMATION', 267 | 'value': 'True', 268 | 'type': 'PLAINTEXT' 269 | }, 270 | ] 271 | }, 272 | serviceRole=codebuild_service_role, 273 | timeoutInMinutes=5, 274 | encryptionKey=kms_key, 275 | ) 276 | 277 | if not codebuild_project_exists(pr_number, 'plan'): 278 | logger.info('Creating tfplan codebuild project') 279 | with open('buildspec-terraform-plan.yml', 'r') as buildspecfile: 280 | buildspec = buildspecfile.read() 281 | codebuild.create_project( 282 | name='{}-terraform-pr-plan-{}'.format(project_name, pr_number), 283 | description='Runs terraform plan against PR', 284 | source={ 285 | 'type': 'CODEPIPELINE', 286 | 'buildspec': buildspec, 287 | }, 288 | artifacts={ 289 | 'type': 'CODEPIPELINE', 290 | }, 291 | environment={ 292 | 'type': 'LINUX_CONTAINER', 293 | 'image': code_build_image, 294 | 'computeType': 'BUILD_GENERAL1_SMALL', 295 | 'environmentVariables': [ 296 | { 297 | 'name': 'GITHUB_API_URL', 298 | 'value': github_api_url, 299 | 'type': 'PLAINTEXT' 300 | }, 301 | { 302 | 'name': 'GITHUB_PAT', 303 | 'value': '{}-terraform-pr-pat'.format(project_name), 304 | 'type': 'PARAMETER_STORE' 305 | }, 306 | { 307 | 'name': 'MODIFIED_DIRS', 308 | 'value': ' '.join(dirs), 309 | 'type': 'PLAINTEXT' 310 | }, 311 | { 312 | 'name': 'MODIFIED_TEST_DIRS', 313 | 'value': ' '.join(test_dirs), 314 | 'type': 'PLAINTEXT' 315 | }, 316 | { 317 | 'name': 'PR_NUMBER', 318 | 'value': pr_number, 319 | 'type': 'PLAINTEXT' 320 | }, 321 | { 322 | 'name': 'REPO', 323 | 'value': repo.split('/')[1], 324 | 'type': 'PLAINTEXT' 325 | }, 326 | { 327 | 'name': 'REPO_NAME', 328 | 'value': repo.replace('/', '-'), 329 | 'type': 'PLAINTEXT' 330 | }, 331 | { 332 | 'name': 'REPO_OWNER', 333 | 'value': repo.split('/')[0], 334 | 'type': 'PLAINTEXT' 335 | }, 336 | { 337 | 'name': 'S3_BUCKET', 338 | 'value': bucket, 339 | 'type': 'PLAINTEXT' 340 | }, 341 | { 342 | 'name': 'TERRAFORM_DOWNLOAD_URL', 343 | 'value': terraform_download_url, 344 | 'type': 'PLAINTEXT' 345 | }, 346 | { 347 | 'name': 'TF_IN_AUTOMATION', 348 | 'value': 'True', 349 | 'type': 'PLAINTEXT' 350 | }, 351 | ] 352 | }, 353 | serviceRole=codebuild_service_role, 354 | timeoutInMinutes=10, 355 | encryptionKey=kms_key, 356 | ) 357 | 358 | logger.info('Creating pipeline') 359 | codepipeline.create_pipeline( 360 | pipeline={ 361 | 'name': '{}-terraform-pr-pipeline-{}'.format( 362 | project_name, 363 | pr_number), 364 | 'roleArn': codepipeline_service_role, 365 | 'artifactStore': { 366 | 'type': 'S3', 367 | 'location': bucket, 368 | 'encryptionKey': { 369 | 'id': kms_key, 370 | 'type': 'KMS' 371 | } 372 | }, 373 | 'stages': [ 374 | { 375 | 'name': 'receive-pr-source', 376 | 'actions': [ 377 | { 378 | 'name': 'pr-repo', 379 | 'actionTypeId': { 380 | 'category': 'Source', 381 | 'owner': 'AWS', 382 | 'provider': 'S3', 383 | 'version': '1' 384 | }, 385 | 'configuration': { 386 | 'S3Bucket': bucket, 387 | 'PollForSourceChanges': 'True', 388 | 'S3ObjectKey': '{}/repo.zip'.format( 389 | pr_number) 390 | }, 391 | 'outputArtifacts': [ 392 | { 393 | 'name': 'source_zip' 394 | }, 395 | ] 396 | }, 397 | ] 398 | }, 399 | { 400 | 'name': 'pull-request-tests', 401 | 'actions': [ 402 | { 403 | 'name': 'terraform-fmt', 404 | 'actionTypeId': { 405 | 'category': 'Test', 406 | 'owner': 'AWS', 407 | 'provider': 'CodeBuild', 408 | 'version': '1' 409 | }, 410 | 'configuration': { 411 | 'ProjectName': '{}-terraform-pr-fmt-{}'.format( 412 | project_name, pr_number) 413 | }, 414 | 'inputArtifacts': [ 415 | { 416 | 'name': 'source_zip' 417 | }, 418 | ] 419 | }, 420 | { 421 | 'name': 'terrascan', 422 | 'actionTypeId': { 423 | 'category': 'Test', 424 | 'owner': 'AWS', 425 | 'provider': 'CodeBuild', 426 | 'version': '1' 427 | }, 428 | 'configuration': { 429 | 'ProjectName': 430 | '{}-terraform-pr-terrascan-{}'.format( 431 | project_name, pr_number) 432 | }, 433 | 'inputArtifacts': [ 434 | { 435 | 'name': 'source_zip' 436 | }, 437 | ] 438 | }, 439 | { 440 | 'name': 'terraform-plan', 441 | 'actionTypeId': { 442 | 'category': 'Build', 443 | 'owner': 'AWS', 444 | 'provider': 'CodeBuild', 445 | 'version': '1' 446 | }, 447 | 'configuration': { 448 | 'ProjectName': 449 | '{}-terraform-pr-plan-{}'.format( 450 | project_name, pr_number) 451 | }, 452 | 'inputArtifacts': [ 453 | { 454 | 'name': 'source_zip' 455 | }, 456 | ] 457 | } 458 | ] 459 | } 460 | ] 461 | } 462 | ) 463 | 464 | 465 | def lambda_handler(event, context): 466 | """ 467 | Creates pipeline when a new repo is uploaded to s3 468 | """ 469 | pr_number = event['Records'][0]['s3']['object']['key'].split('/')[0] 470 | logger.info('Changes detected on PR #{}'.format(pr_number)) 471 | 472 | if not pipeline_exists(pr_number): 473 | create_pipeline(pr_number) 474 | -------------------------------------------------------------------------------- /terraform-pr-pipeline/pipeline-create/requirements.txt: -------------------------------------------------------------------------------- 1 | requests==2.18.4 2 | boto3==1.5.6 3 | -------------------------------------------------------------------------------- /terraform-pr-pipeline/pipeline-delete.tf: -------------------------------------------------------------------------------- 1 | /** 2 | Resources for the pipeline-delete lambda function 3 | */ 4 | 5 | resource "aws_lambda_function" "pipeline_delete" { 6 | depends_on = ["null_resource.pipeline_delete_dependencies"] 7 | filename = ".lambda-zip/pipeline-delete.zip" 8 | function_name = "${var.project_name}-pipeline-delete" 9 | role = "${aws_iam_role.pipeline_delete.arn}" 10 | handler = "pipeline-delete.lambda_handler" 11 | source_code_hash = "${base64sha256(file("${path.module}/.lambda-zip/pipeline-delete.zip"))}" 12 | runtime = "python3.6" 13 | kms_key_arn = "${aws_kms_key.pipeline_key.arn}" 14 | 15 | tags { 16 | Name = "${var.project_name}-pipeline-delete" 17 | } 18 | 19 | environment { 20 | variables = { 21 | BUCKET_NAME = "${aws_s3_bucket.bucket.id}" 22 | PROJECT_NAME = "${var.project_name}" 23 | } 24 | } 25 | } 26 | 27 | // Allows s3 to trigger the function 28 | resource "aws_lambda_permission" "pipeline_delete" { 29 | statement_id = "schedule" 30 | action = "lambda:InvokeFunction" 31 | function_name = "${aws_lambda_function.pipeline_delete.function_name}" 32 | principal = "s3.amazonaws.com" 33 | source_arn = "${aws_s3_bucket.bucket.arn}" 34 | } 35 | 36 | // Creates zip file with dependencies every time pipeline-delete.py is updated 37 | resource "null_resource" "pipeline_delete_dependencies" { 38 | triggers { 39 | lambda_function = "${file("${path.module}/pipeline-delete/pipeline-delete.py")}" 40 | } 41 | 42 | provisioner "local-exec" { 43 | command = "rm -rf ${path.module}/.lambda-zip/pipeline-delete-resources" 44 | } 45 | 46 | provisioner "local-exec" { 47 | command = "mkdir -p ${path.module}/.lambda-zip/pipeline-delete-resources" 48 | } 49 | 50 | provisioner "local-exec" { 51 | command = "pip install --target=${path.module}/.lambda-zip/pipeline-delete-resources requests" 52 | } 53 | 54 | provisioner "local-exec" { 55 | command = "cp -R ${path.module}/pipeline-delete/pipeline-delete.py ${path.module}/.lambda-zip/pipeline-delete-resources/." 56 | } 57 | 58 | provisioner "local-exec" { 59 | command = "cd ${path.module}/.lambda-zip/pipeline-delete-resources/ && zip -r ../pipeline-delete.zip ." 60 | } 61 | } 62 | 63 | // AWS IAM role for pipeline-delete function 64 | resource "aws_iam_role" "pipeline_delete" { 65 | name = "${var.project_name}-pipeline-delete-lambda" 66 | assume_role_policy = "${data.aws_iam_policy_document.lambda_assume_role.json}" 67 | } 68 | 69 | data "aws_iam_policy_document" "pipeline_delete" { 70 | statement { 71 | sid = "CreateLogs" 72 | 73 | actions = [ 74 | "logs:CreateLogGroup", 75 | "logs:CreateLogStream", 76 | "logs:PutLogEvents", 77 | ] 78 | 79 | resources = [ 80 | "arn:aws:logs:${var.aws_region}:${data.aws_caller_identity.current.account_id}:log-group:/aws/lambda/${var.project_name}-pipeline-delete:*", 81 | ] 82 | } 83 | 84 | statement { 85 | sid = "CodeBuildAccess" 86 | 87 | actions = [ 88 | "codebuild:DeleteProject", 89 | ] 90 | 91 | resources = [ 92 | "arn:aws:codebuild:${var.aws_region}:${data.aws_caller_identity.current.account_id}:project/${var.project_name}*", 93 | ] 94 | } 95 | 96 | statement { 97 | sid = "CodePipelineAccess" 98 | 99 | actions = [ 100 | "codepipeline:DeletePipeline", 101 | ] 102 | 103 | resources = [ 104 | "arn:aws:codepipeline:${var.aws_region}:${data.aws_caller_identity.current.account_id}:${var.project_name}*", 105 | ] 106 | } 107 | 108 | statement { 109 | sid = "s3" 110 | 111 | actions = [ 112 | "s3:GetObject", 113 | ] 114 | 115 | resources = [ 116 | "${aws_s3_bucket.bucket.arn}*", 117 | ] 118 | } 119 | } 120 | 121 | resource "aws_iam_policy" "pipeline_delete" { 122 | name = "${aws_iam_role.pipeline_delete.name}-policy" 123 | policy = "${data.aws_iam_policy_document.pipeline_delete.json}" 124 | } 125 | 126 | resource "aws_iam_role_policy_attachment" "pipeline_delete" { 127 | role = "${aws_iam_role.pipeline_delete.name}" 128 | policy_arn = "${aws_iam_policy.pipeline_delete.arn}" 129 | } 130 | -------------------------------------------------------------------------------- /terraform-pr-pipeline/pipeline-delete/pipeline-delete.py: -------------------------------------------------------------------------------- 1 | import os 2 | import logging 3 | import boto3 4 | 5 | # Configuring logger 6 | logger = logging.getLogger() 7 | logger.setLevel(logging.INFO) 8 | logger.handlers[0].setFormatter(logging.Formatter( 9 | '%(asctime)s - %(name)s - %(levelname)s - %(message)s\n' 10 | )) 11 | logging.getLogger('boto3').setLevel(logging.ERROR) 12 | logging.getLogger('botocore').setLevel(logging.ERROR) 13 | 14 | # Boto clients 15 | codepipeline = boto3.client('codepipeline') 16 | codebuild = boto3.client('codebuild') 17 | s3 = boto3.client('s3') 18 | 19 | # Global vars 20 | bucket = os.environ['BUCKET_NAME'] 21 | project_name = os.environ['PROJECT_NAME'] 22 | 23 | 24 | def object_exists(key): 25 | """ 26 | Determines if objet exists in S3 bucket 27 | """ 28 | try: 29 | s3.get_object(Bucket=bucket, Key=key) 30 | except (s3.exceptions.NoSuchKey, s3.exceptions.ClientError): 31 | return False 32 | return True 33 | 34 | 35 | def delete_pipeline(pipeline_name): 36 | """ 37 | Deletes the specified AWS CodePipeline pipeline 38 | """ 39 | logger.info('Deleting pipeline: {}'.format(pipeline_name)) 40 | codepipeline.delete_pipeline(name=pipeline_name) 41 | 42 | 43 | def delete_codebuild_project(project_name): 44 | """ 45 | Deletes the specified AWS CodeBuild project 46 | """ 47 | logger.info('Deleting codebuild project: {}'.format( 48 | project_name)) 49 | codebuild.delete_project(name=project_name) 50 | 51 | 52 | def lambda_handler(event, context): 53 | """ 54 | Deletes pipeline resources associated to S3 object that no longer exists 55 | """ 56 | repo_object = event['Records'][0]['s3']['object']['key'] 57 | 58 | if 'repo.zip' in repo_object and not object_exists(repo_object): 59 | logger.info('Object no longer exists at: {}'.format(repo_object)) 60 | pr_number = repo_object.split('/')[0] 61 | 62 | delete_pipeline('{}-terraform-pr-pipeline-{}'.format( 63 | project_name, pr_number)) 64 | 65 | delete_codebuild_project('{}-terraform-pr-fmt-{}'.format( 66 | project_name, pr_number)) 67 | 68 | delete_codebuild_project('{}-terraform-pr-terrascan-{}'.format( 69 | project_name, pr_number)) 70 | 71 | delete_codebuild_project('{}-terraform-pr-plan-{}'.format( 72 | project_name, pr_number)) 73 | -------------------------------------------------------------------------------- /terraform-pr-pipeline/pipeline-delete/requirements.txt: -------------------------------------------------------------------------------- 1 | boto3==1.5.6 2 | -------------------------------------------------------------------------------- /terraform-pr-pipeline/poller-create.tf: -------------------------------------------------------------------------------- 1 | /** 2 | Resources for the poller-create lambda function 3 | */ 4 | 5 | resource "aws_lambda_function" "poller_create" { 6 | depends_on = ["null_resource.poller_create_dependencies"] 7 | filename = ".lambda-zip/poller-create.zip" 8 | function_name = "${var.project_name}-poller-create" 9 | role = "${aws_iam_role.poller_create.arn}" 10 | handler = "poller-create.lambda_handler" 11 | source_code_hash = "${base64sha256(file("${path.module}/.lambda-zip/poller-create.zip"))}" 12 | runtime = "python3.6" 13 | kms_key_arn = "${aws_kms_key.pipeline_key.arn}" 14 | 15 | tags { 16 | Name = "${var.project_name}-poller-create" 17 | } 18 | 19 | environment { 20 | variables = { 21 | BUCKET_NAME = "${aws_s3_bucket.bucket.id}" 22 | GITHUB_API_URL = "${var.github_api_url}" 23 | GITHUB_REPO_NAME = "${var.github_repo_name}" 24 | KMS_KEY_ID = "${aws_kms_key.pipeline_key.key_id}" 25 | } 26 | } 27 | } 28 | 29 | // Allows cloudwatch to trigger the function 30 | resource "aws_lambda_permission" "poller_create" { 31 | statement_id = "schedule" 32 | action = "lambda:InvokeFunction" 33 | function_name = "${aws_lambda_function.poller_create.function_name}" 34 | principal = "events.amazonaws.com" 35 | source_arn = "${aws_cloudwatch_event_rule.poller_create_schedule.arn}" 36 | } 37 | 38 | // Creates zip file with dependencies every time poller-create.py is updated 39 | resource "null_resource" "poller_create_dependencies" { 40 | triggers { 41 | lambda_function = "${file("${path.module}/poller-create/poller-create.py")}" 42 | } 43 | 44 | provisioner "local-exec" { 45 | command = "rm -rf ${path.module}/.lambda-zip/poller-create-resources" 46 | } 47 | 48 | provisioner "local-exec" { 49 | command = "mkdir -p ${path.module}/.lambda-zip/poller-create-resources" 50 | } 51 | 52 | provisioner "local-exec" { 53 | command = "pip install --target=${path.module}/.lambda-zip/poller-create-resources requests" 54 | } 55 | 56 | provisioner "local-exec" { 57 | command = "cp -R ${path.module}/poller-create/poller-create.py ${path.module}/.lambda-zip/poller-create-resources/." 58 | } 59 | 60 | provisioner "local-exec" { 61 | command = "cd ${path.module}/.lambda-zip/poller-create-resources/ && zip -r ../poller-create.zip ." 62 | } 63 | } 64 | 65 | // Triggers lambda function for polling based on the given time in minutes 66 | resource "aws_cloudwatch_event_rule" "poller_create_schedule" { 67 | name = "${var.project_name}-poller-create-schedule" 68 | description = "Periodically triggers poller-create lambda" 69 | schedule_expression = "rate(${var.poller_create_rate} minutes)" 70 | } 71 | 72 | resource "aws_cloudwatch_event_target" "poller_create_target" { 73 | rule = "${aws_cloudwatch_event_rule.poller_create_schedule.name}" 74 | arn = "${aws_lambda_function.poller_create.arn}" 75 | } 76 | 77 | // AWS IAM role for poller-create function 78 | resource "aws_iam_role" "poller_create" { 79 | name = "${var.project_name}-poller-create-lambda" 80 | assume_role_policy = "${data.aws_iam_policy_document.lambda_assume_role.json}" 81 | } 82 | 83 | data "aws_iam_policy_document" "poller_create" { 84 | statement { 85 | sid = "CreateLogs" 86 | 87 | actions = [ 88 | "logs:CreateLogGroup", 89 | "logs:CreateLogStream", 90 | "logs:PutLogEvents", 91 | ] 92 | 93 | resources = [ 94 | "arn:aws:logs:*:*:log-group:/aws/lambda/${var.project_name}-poller-create:*", 95 | ] 96 | } 97 | 98 | statement { 99 | sid = "s3" 100 | 101 | actions = [ 102 | "s3:List*", 103 | "s3:GetObjectTagging", 104 | "s3:PutObject", 105 | "s3:PutObjectTagging", 106 | ] 107 | 108 | resources = [ 109 | "${aws_s3_bucket.bucket.arn}*", 110 | ] 111 | } 112 | } 113 | 114 | resource "aws_iam_policy" "poller_create" { 115 | name = "${aws_iam_role.poller_create.name}-policy" 116 | policy = "${data.aws_iam_policy_document.poller_create.json}" 117 | } 118 | 119 | resource "aws_iam_role_policy_attachment" "poller_create" { 120 | role = "${aws_iam_role.poller_create.name}" 121 | policy_arn = "${aws_iam_policy.poller_create.arn}" 122 | } 123 | -------------------------------------------------------------------------------- /terraform-pr-pipeline/poller-create/poller-create.py: -------------------------------------------------------------------------------- 1 | import os 2 | import logging 3 | import requests 4 | import boto3 5 | 6 | # Configuring logger 7 | logger = logging.getLogger() 8 | logger.setLevel(logging.INFO) 9 | logger.handlers[0].setFormatter(logging.Formatter( 10 | '%(asctime)s - %(name)s - %(levelname)s - %(message)s\n' 11 | )) 12 | logging.getLogger('boto3').setLevel(logging.ERROR) 13 | logging.getLogger('botocore').setLevel(logging.ERROR) 14 | 15 | # Boto clients 16 | s3 = boto3.client('s3') 17 | 18 | # Global vars 19 | github_api_url = os.environ['GITHUB_API_URL'] 20 | repo = os.environ['GITHUB_REPO_NAME'] 21 | bucket = os.environ['BUCKET_NAME'] 22 | kms_key_id = os.environ['KMS_KEY_ID'] 23 | 24 | 25 | def get_open_pull_requests(): 26 | """ 27 | Returns JSON of open PRs on given repo 28 | """ 29 | pulls_url = '{}/repos/{}/pulls'.format( 30 | github_api_url, 31 | repo) 32 | logger.info('Loading open pull requests from: {}'.format(pulls_url)) 33 | r = requests.get(pulls_url) 34 | 35 | if r.status_code != 200: 36 | logger.error('GH pull URL status code error: {}'.format( 37 | r.status_code)) 38 | raise Exception('GH pull URL status code != 200') 39 | 40 | return r.json() 41 | 42 | 43 | def object_exists(key): 44 | """ 45 | Determines if objet exists in S3 bucket 46 | """ 47 | try: 48 | s3.get_object(Bucket=bucket, Key=key) 49 | except s3.exceptions.NoSuchKey: 50 | return False 51 | return True 52 | 53 | 54 | def is_pr_synced(pr_number, sha): 55 | """ 56 | Checks if there's a corresponding S3 object for the given pr_number and sha 57 | Assumes that zip files are at pr_number/repo.zip and that they are 58 | tagged with "Key":"latest_commit","Value":sha 59 | """ 60 | response = s3.list_objects_v2(Bucket=bucket) 61 | repo_zip = '{}/repo.zip'.format(pr_number) 62 | try: 63 | for obj in response['Contents']: 64 | if repo_zip == obj['Key']: 65 | tags = s3.get_object_tagging( 66 | Bucket=bucket, Key=obj['Key'])['TagSet'] 67 | for tag in tags: 68 | if tag['Key'] == 'latest_commit' and tag['Value'] == sha: 69 | logger.debug( 70 | 'is_pr_synced({},{}) returned True'.format( 71 | pr_number, 72 | sha 73 | )) 74 | return True 75 | except Exception as e: 76 | logger.error( 77 | 'is_pr_synced({},{}) Exception: {}'.format( 78 | pr_number, 79 | sha, 80 | e 81 | )) 82 | return False 83 | logger.debug( 84 | 'is_pr_synced({},{}) returning False'.format( 85 | pr_number, 86 | sha 87 | )) 88 | return False 89 | 90 | 91 | def lambda_handler(event, context): 92 | """ 93 | Checks if repo for PRs and syncs open PRs (and commits) into an S3 bucket 94 | """ 95 | open_pr_json = get_open_pull_requests() 96 | 97 | synced_prs = [] 98 | for pr in open_pr_json: 99 | """ 100 | Check if in S3 101 | If not in s3 get zip and place it there 102 | """ 103 | logger.debug('Checking PR: {}'.format(pr['number'])) 104 | branch_name = pr['head']['ref'].replace('refs/heads/', '') 105 | archive_url = pr['head']['repo']['archive_url'].replace( 106 | '{archive_format}', 107 | 'zipball').replace( 108 | '{/ref}', '/' + branch_name 109 | ) 110 | headers = {} 111 | if not is_pr_synced(pr['number'], pr['head']['sha']): 112 | r = requests.get(archive_url, headers=headers) 113 | archive_name = '/tmp/repo.zip' 114 | s3_object_key = '{}/repo.zip'.format(pr['number']) 115 | with open(archive_name, 'wb') as f: 116 | f.write(r.content) 117 | s3.upload_file( 118 | archive_name, 119 | bucket, 120 | s3_object_key, 121 | ExtraArgs={ 122 | "ServerSideEncryption": "aws:kms", 123 | "SSEKMSKeyId": kms_key_id 124 | } 125 | ) 126 | logger.debug('Copied zip file to s3') 127 | s3.put_object_tagging( 128 | Bucket=bucket, 129 | Key=s3_object_key, 130 | Tagging={ 131 | 'TagSet': [ 132 | { 133 | 'Key': 'latest_commit', 134 | 'Value': pr['head']['sha'] 135 | }, 136 | ] 137 | } 138 | ) 139 | synced_prs.append({ 140 | 'number': pr['number'], 141 | 'title': pr['title'], 142 | 'submitted_by': pr['user']['login'], 143 | 'url': pr['url'], 144 | 'html_url': pr['html_url'], 145 | 'pr_repo': pr['head']['repo']['full_name'], 146 | 'archive_url': pr['head']['repo']['archive_url'].replace( 147 | '{archive_format}', 148 | 'zipball').replace( 149 | '{/ref}', '/' + branch_name) 150 | }) 151 | if synced_prs == []: 152 | logger.info('No updates') 153 | else: 154 | logger.info('The following PRs where updated in S3: {}'.format( 155 | synced_prs)) 156 | -------------------------------------------------------------------------------- /terraform-pr-pipeline/poller-create/requirements.txt: -------------------------------------------------------------------------------- 1 | requests==2.18.4 2 | boto3==1.5.6 3 | -------------------------------------------------------------------------------- /terraform-pr-pipeline/poller-delete.tf: -------------------------------------------------------------------------------- 1 | /** 2 | Resources for the poller-delete lambda function 3 | */ 4 | 5 | resource "aws_lambda_function" "poller_delete" { 6 | depends_on = ["null_resource.poller_delete_dependencies"] 7 | filename = ".lambda-zip/poller-delete.zip" 8 | function_name = "${var.project_name}-poller-delete" 9 | role = "${aws_iam_role.poller_delete.arn}" 10 | handler = "poller-delete.lambda_handler" 11 | source_code_hash = "${base64sha256(file("${path.module}/.lambda-zip/poller-delete.zip"))}" 12 | runtime = "python3.6" 13 | kms_key_arn = "${aws_kms_key.pipeline_key.arn}" 14 | 15 | tags { 16 | Name = "${var.project_name}-poller-delete" 17 | } 18 | 19 | environment { 20 | variables = { 21 | BUCKET_NAME = "${aws_s3_bucket.bucket.id}" 22 | GITHUB_API_URL = "${var.github_api_url}" 23 | GITHUB_REPO_NAME = "${var.github_repo_name}" 24 | } 25 | } 26 | } 27 | 28 | // Allows cloudwatch to trigger the function 29 | resource "aws_lambda_permission" "poller_delete" { 30 | statement_id = "schedule" 31 | action = "lambda:InvokeFunction" 32 | function_name = "${aws_lambda_function.poller_delete.function_name}" 33 | principal = "events.amazonaws.com" 34 | source_arn = "${aws_cloudwatch_event_rule.poller_delete_schedule.arn}" 35 | } 36 | 37 | // Creates zip file with dependencies every time poller-delete.py is updated 38 | resource "null_resource" "poller_delete_dependencies" { 39 | triggers { 40 | lambda_function = "${file("${path.module}/poller-delete/poller-delete.py")}" 41 | } 42 | 43 | provisioner "local-exec" { 44 | command = "rm -rf ${path.module}/.lambda-zip/poller-delete-resources" 45 | } 46 | 47 | provisioner "local-exec" { 48 | command = "mkdir -p ${path.module}/.lambda-zip/poller-delete-resources" 49 | } 50 | 51 | provisioner "local-exec" { 52 | command = "pip install --target=${path.module}/.lambda-zip/poller-delete-resources requests" 53 | } 54 | 55 | provisioner "local-exec" { 56 | command = "cp -R ${path.module}/poller-delete/poller-delete.py ${path.module}/.lambda-zip/poller-delete-resources/." 57 | } 58 | 59 | provisioner "local-exec" { 60 | command = "cd ${path.module}/.lambda-zip/poller-delete-resources/ && zip -r ../poller-delete.zip ." 61 | } 62 | } 63 | 64 | // Triggers lambda function for polling based on the given time in minutes 65 | resource "aws_cloudwatch_event_rule" "poller_delete_schedule" { 66 | name = "${var.project_name}-poller-delete-schedule" 67 | description = "Periodically triggers poller-delete lambda" 68 | schedule_expression = "rate(${var.poller_delete_rate} minutes)" 69 | } 70 | 71 | resource "aws_cloudwatch_event_target" "poller_delete_target" { 72 | rule = "${aws_cloudwatch_event_rule.poller_delete_schedule.name}" 73 | arn = "${aws_lambda_function.poller_delete.arn}" 74 | } 75 | 76 | // AWS IAM role for poller-delete function 77 | resource "aws_iam_role" "poller_delete" { 78 | name = "${var.project_name}-poller-delete-lambda" 79 | assume_role_policy = "${data.aws_iam_policy_document.lambda_assume_role.json}" 80 | } 81 | 82 | data "aws_iam_policy_document" "poller_delete" { 83 | statement { 84 | sid = "CreateLogs" 85 | 86 | actions = [ 87 | "logs:CreateLogGroup", 88 | "logs:CreateLogStream", 89 | "logs:PutLogEvents", 90 | ] 91 | 92 | resources = [ 93 | "arn:aws:logs:*:*:log-group:/aws/lambda/${var.project_name}-poller-delete:*", 94 | ] 95 | } 96 | 97 | statement { 98 | sid = "s3" 99 | 100 | actions = [ 101 | "s3:List*", 102 | "s3:DeleteObject", 103 | "s3:GetObject*", 104 | ] 105 | 106 | resources = [ 107 | "${aws_s3_bucket.bucket.arn}*", 108 | ] 109 | } 110 | } 111 | 112 | resource "aws_iam_policy" "poller_delete" { 113 | name = "${aws_iam_role.poller_delete.name}-policy" 114 | policy = "${data.aws_iam_policy_document.poller_delete.json}" 115 | } 116 | 117 | resource "aws_iam_role_policy_attachment" "poller_delete" { 118 | role = "${aws_iam_role.poller_delete.name}" 119 | policy_arn = "${aws_iam_policy.poller_delete.arn}" 120 | } 121 | -------------------------------------------------------------------------------- /terraform-pr-pipeline/poller-delete/poller-delete.py: -------------------------------------------------------------------------------- 1 | import os 2 | import logging 3 | import requests 4 | import boto3 5 | 6 | # Configuring logger 7 | logger = logging.getLogger() 8 | logger.setLevel(logging.INFO) 9 | logger.handlers[0].setFormatter(logging.Formatter( 10 | '%(asctime)s - %(name)s - %(levelname)s - %(message)s\n' 11 | )) 12 | logging.getLogger('boto3').setLevel(logging.ERROR) 13 | logging.getLogger('botocore').setLevel(logging.ERROR) 14 | 15 | # Boto clients 16 | s3 = boto3.client('s3') 17 | 18 | # Global vars 19 | github_api_url = os.environ['GITHUB_API_URL'] 20 | repo = os.environ['GITHUB_REPO_NAME'] 21 | bucket = os.environ['BUCKET_NAME'] 22 | 23 | 24 | def is_pr_open(pr_number): 25 | """ 26 | Returns True if the pull request's status is open 27 | """ 28 | pr_url = '{}/repos/{}/pulls/{}'.format( 29 | github_api_url, 30 | repo, 31 | pr_number) 32 | logger.debug('Checking if PR is open in this URL: {}'.format(pr_url)) 33 | if requests.get(pr_url).json()['state'] == 'open': 34 | logger.debug('PR open True') 35 | return True 36 | logger.debug('PR open False') 37 | return False 38 | 39 | 40 | def lambda_handler(event, context): 41 | """ 42 | Deletes objects in S3 for PRs that are no longer open 43 | """ 44 | logger.debug('Removing resources for PRs no longer open') 45 | s3_resources = s3.list_objects_v2(Bucket=bucket)['Contents'] 46 | objects_to_delete = [] 47 | for resource in s3_resources: 48 | if 'terraform' in resource['Key']: 49 | continue 50 | if not is_pr_open(resource['Key'].split('/')[0]): 51 | objects_to_delete.append({ 52 | 'Key': resource['Key'] 53 | }) 54 | 55 | if objects_to_delete == []: 56 | logger.info('All PRs still open') 57 | else: 58 | s3.delete_objects( 59 | Bucket=bucket, 60 | Delete={ 61 | 'Objects': objects_to_delete, 62 | 'Quiet': False 63 | } 64 | ) 65 | logger.info('The following objects have been deleted: {}'.format( 66 | objects_to_delete)) 67 | -------------------------------------------------------------------------------- /terraform-pr-pipeline/poller-delete/requirements.txt: -------------------------------------------------------------------------------- 1 | requests==2.18.4 2 | boto3==1.5.6 3 | -------------------------------------------------------------------------------- /terraform-pr-pipeline/setup.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | set -ex 4 | 5 | echo "Empties directory" 6 | rm -rf .lambda-zip/ 7 | 8 | echo "Makes new directories for lambda resources" 9 | mkdir -p .lambda-zip/pipeline-create-resources 10 | mkdir -p .lambda-zip/pipeline-delete-resources 11 | mkdir -p .lambda-zip/poller-create-resources 12 | mkdir -p .lambda-zip/poller-delete-resources 13 | 14 | echo "Installing dependencies" 15 | pip install --target=.lambda-zip/pipeline-create-resources -r pipeline-create/requirements.txt 16 | #pip install --target=.lambda-zip/pipeline-delete-resources -r pipeline-delete/requirements.txt 17 | pip install --target=.lambda-zip/poller-create-resources -r poller-create/requirements.txt 18 | pip install --target=.lambda-zip/poller-delete-resources -r poller-delete/requirements.txt 19 | 20 | echo "Copying function" 21 | cp -R pipeline-create/pipeline-create.py .lambda-zip/pipeline-create-resources/. 22 | #cp -R pipeline-delete/pipeline-delete.py .lambda-zip/pipeline-delete-resources/. 23 | cp -R poller-create/poller-create.py .lambda-zip/poller-create-resources/. 24 | cp -R poller-delete/poller-delete.py .lambda-zip/poller-delete-resources/. 25 | 26 | echo "Creating zip files" 27 | pushd .lambda-zip/pipeline-create-resources/ && zip -r ../pipeline-create.zip . 28 | popd 29 | #pushd .lambda-zip/pipeline-delete-resources/ && zip -r ../pipeline-delete.zip . 30 | #popd 31 | pushd .lambda-zip/poller-create-resources/ && zip -r ../poller-create.zip . 32 | popd 33 | pushd .lambda-zip/poller-delete-resources/ && zip -r ../poller-delete.zip . 34 | popd 35 | -------------------------------------------------------------------------------- /terraform-pr-pipeline/variables.tf: -------------------------------------------------------------------------------- 1 | variable "aws_region" { 2 | description = "AWS region where resources are provisioned" 3 | } 4 | 5 | variable "aws_profile" { 6 | description = "AWS credentials profile to use" 7 | } 8 | 9 | variable "project_name" { 10 | description = "All resources will be prepended with this name" 11 | } 12 | 13 | variable "github_api_url" { 14 | description = "API URL for GitHub" 15 | default = "https://api.github.com" 16 | } 17 | 18 | variable "github_repo_name" { 19 | description = "Name of the repository to track pull requests in org/repo format (e.g. cesar-rodriguez/test-repo)" 20 | } 21 | 22 | variable "poller_create_rate" { 23 | description = "Rate in minutes for polling the GitHub repository for open pull requests" 24 | default = 5 25 | } 26 | 27 | variable "poller_delete_rate" { 28 | description = "Rate in minutes for polling the GitHub repository to check if PRs are still open" 29 | default = 60 30 | } 31 | 32 | variable "code_build_image" { 33 | description = "Docker image to use for CodeBuild container - Use http://amzn.to/2mjCI91 for reference" 34 | default = "aws/codebuild/ubuntu-base:14.04" 35 | } 36 | 37 | variable "terraform_download_url" { 38 | description = "URL for terraform version to be used for builds" 39 | default = "https://releases.hashicorp.com/terraform/0.11.1/terraform_0.11.1_linux_amd64.zip" 40 | } 41 | --------------------------------------------------------------------------------