├── .gitignore ├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── LICENSE ├── README.md ├── Setup.yaml ├── TemplatePipeline.yaml └── buildspec ├── CDAction.yaml └── CIAction.yaml /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | *$py.class 5 | 6 | # C extensions 7 | *.so 8 | 9 | # Distribution / packaging 10 | .Python 11 | build/ 12 | develop-eggs/ 13 | dist/ 14 | downloads/ 15 | eggs/ 16 | .eggs/ 17 | lib/ 18 | lib64/ 19 | parts/ 20 | sdist/ 21 | var/ 22 | wheels/ 23 | share/python-wheels/ 24 | *.egg-info/ 25 | .installed.cfg 26 | *.egg 27 | MANIFEST 28 | 29 | # PyInstaller 30 | # Usually these files are written by a python script from a template 31 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 32 | *.manifest 33 | *.spec 34 | 35 | # Installer logs 36 | pip-log.txt 37 | pip-delete-this-directory.txt 38 | 39 | # Unit test / coverage reports 40 | htmlcov/ 41 | .tox/ 42 | .nox/ 43 | .coverage 44 | .coverage.* 45 | .cache 46 | nosetests.xml 47 | coverage.xml 48 | *.cover 49 | .hypothesis/ 50 | .pytest_cache/ 51 | 52 | # Translations 53 | *.mo 54 | *.pot 55 | 56 | # Django stuff: 57 | *.log 58 | local_settings.py 59 | db.sqlite3 60 | 61 | # Flask stuff: 62 | instance/ 63 | .webassets-cache 64 | 65 | # Scrapy stuff: 66 | .scrapy 67 | 68 | # Sphinx documentation 69 | docs/_build/ 70 | 71 | # PyBuilder 72 | target/ 73 | 74 | # Jupyter Notebook 75 | .ipynb_checkpoints 76 | 77 | # IPython 78 | profile_default/ 79 | ipython_config.py 80 | 81 | # pyenv 82 | .python-version 83 | 84 | # celery beat schedule file 85 | celerybeat-schedule 86 | 87 | # SageMath parsed files 88 | *.sage.py 89 | 90 | # Environments 91 | .env 92 | .venv 93 | env/ 94 | venv/ 95 | ENV/ 96 | env.bak/ 97 | venv.bak/ 98 | 99 | # Spyder project settings 100 | .spyderproject 101 | .spyproject 102 | 103 | # Rope project settings 104 | .ropeproject 105 | 106 | # mkdocs documentation 107 | /site 108 | 109 | # mypy 110 | .mypy_cache/ 111 | .dmypy.json 112 | dmypy.json 113 | 114 | # Pyre type checker 115 | .pyre/ 116 | 117 | # Visual Studio Code 118 | .vscode/ 119 | 120 | # JetBrains 121 | .idea/ 122 | 123 | # MacOS 124 | .DS_Store 125 | 126 | # Files generated by AWS Cloudformation package 127 | output/ 128 | 129 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | ## Code of Conduct 2 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 3 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 4 | opensource-codeofconduct@amazon.com with any additional questions or comments. 5 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing Guidelines 2 | 3 | Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional 4 | documentation, we greatly value feedback and contributions from our community. 5 | 6 | Please read through this document before submitting any issues or pull requests to ensure we have all the necessary 7 | information to effectively respond to your bug report or contribution. 8 | 9 | 10 | ## Reporting Bugs/Feature Requests 11 | 12 | We welcome you to use the GitHub issue tracker to report bugs or suggest features. 13 | 14 | When filing an issue, please check existing open, or recently closed, issues to make sure somebody else hasn't already 15 | reported the issue. Please try to include as much information as you can. Details like these are incredibly useful: 16 | 17 | * A reproducible test case or series of steps 18 | * The version of our code being used 19 | * Any modifications you've made relevant to the bug 20 | * Anything unusual about your environment or deployment 21 | 22 | 23 | ## Contributing via Pull Requests 24 | Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that: 25 | 26 | 1. You are working against the latest source on the *master* branch. 27 | 2. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already. 28 | 3. You open an issue to discuss any significant work - we would hate for your time to be wasted. 29 | 30 | To send us a pull request, please: 31 | 32 | 1. Fork the repository. 33 | 2. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change. 34 | 3. Ensure local tests pass. 35 | 4. Commit to your fork using clear commit messages. 36 | 5. Send us a pull request, answering any default questions in the pull request interface. 37 | 6. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation. 38 | 39 | GitHub provides additional document on [forking a repository](https://help.github.com/articles/fork-a-repo/) and 40 | [creating a pull request](https://help.github.com/articles/creating-a-pull-request/). 41 | 42 | 43 | ## Finding contributions to work on 44 | Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels (enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any 'help wanted' issues is a great place to start. 45 | 46 | 47 | ## Code of Conduct 48 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 49 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 50 | opensource-codeofconduct@amazon.com with any additional questions or comments. 51 | 52 | 53 | ## Security issue notifications 54 | If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github issue. 55 | 56 | 57 | ## Licensing 58 | 59 | See the [LICENSE](LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution. 60 | 61 | We may ask you to sign a [Contributor License Agreement (CLA)](http://en.wikipedia.org/wiki/Contributor_License_Agreement) for larger changes. 62 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining a copy of 4 | this software and associated documentation files (the "Software"), to deal in 5 | the Software without restriction, including without limitation the rights to 6 | use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of 7 | the Software, and to permit persons to whom the Software is furnished to do so. 8 | 9 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 10 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS 11 | FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 12 | COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER 13 | IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN 14 | CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 15 | 16 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Multi-branch CodePipeline strategy with event-driven architecture 2 | 3 | *Henrique Bueno, DevOps Consultant, Professional Services* 4 | 5 | This blog post presents a solution for automated pipelines creation in AWS CodePipeline when a new branch is created in an AWS CodeCommit repository. A use case for this solution is when a GitFlow approach using CodePipeline is required. The strategy presented here is used by AWS customers to enable the use of GitFlow using only AWS tools. 6 | 7 | CodePipeline is a fully managed continuous delivery service that helps you automate your release pipelines for fast and reliable application and infrastructure updates. CodePipeline automates the build, test, and deploy phases of your release process every time there is a code change, based on the release model you define. 8 | 9 | CodeCommit is a fully managed source control service that hosts secure Git-based repositories. It makes it easy for teams to collaborate on code in a secure and highly scalable ecosystem. 10 | 11 | GitFlow is a branching model designed around the project release. This provides a robust framework for managing larger projects. Gitflow is ideally suited for projects that have a scheduled release cycle. 12 | 13 | 14 | 15 | ## Applicability 16 | 17 | When using CodePipeline to orchestrate pipelines and CodeCommit as a code source, in addition to setting a repository, you must also set which branch will trigger the pipeline. This configuration works perfectly for the trunk-based strategy, in which you have only one main branch and all the developers interact with this single branch. However, when you need to work with a multi-branching strategy like GitFlow, the requirement to set a pipeline for each branch brings additional challenges. 18 | 19 | It’s important to note that trunk-based is, by far, the best strategy for taking full advantage of a DevOps approach; this is the branching strategy that AWS recommends to its customers. On the other hand, many customers like to work with multiple branches and believe it justifies the effort and complexity in dealing with branching merges. This solution is for these customers. 20 | 21 | ## Solution Overview 22 | 23 | One of the great benefits of working with Infrastructure as Code is the ability to create multiple identical environments through a single template. This example uses AWS CloudFormation templates to provision pipelines and other necessary resources, as shown in the following diagram. 24 | 25 | The template is hosted in an Amazon S3 bucket. An AWS Lambda function deploys a new AWS CloudFormation stack based on this template. This Lambda function is trigged for an Amazon CloudWatch Events rule that looks for events at the CodePipeline repository. 26 | 27 | ### The CreatePipeline Events rule 28 | 29 | The AWS CloudFormation snippet that creates the Events rule follows. The Events rule monitors create and delete branches events in all repositories, triggering the CreatePipeline Lambda function. 30 | 31 | ````` 32 | #----------------------------------------------------------------------# 33 | # EventRule to trigger LambdaPipeline lambda 34 | #----------------------------------------------------------------------# 35 | CreatePipelineRule: 36 | Type: AWS::Events::Rule 37 | Properties: 38 | Description: "EventRule" 39 | EventPattern: 40 | source: 41 | - aws.codecommit 42 | detail-type: 43 | - 'CodeCommit Repository State Change' 44 | detail: 45 | event: 46 | - referenceDeleted 47 | - referenceCreated 48 | referenceType: 49 | - branch 50 | State: ENABLED 51 | Targets: 52 | - Arn: !GetAtt CreatePipeline.Arn 53 | Id: CreatePipeline 54 | ````` 55 | 56 | ### The CreatePipeline Lambda function 57 | 58 | The Lambda function receives the event details, parses the variables, and executes the appropriate actions. If the event is referenceCreated, then the stack is created; otherwise the stack is deleted. The stack name created or deleted is the junction of the repository name plus the new branch name. This is a very simple function. 59 | 60 | ````` 61 | #----------------------------------------------------------------------# 62 | # Lambda for Stack Creation 63 | #----------------------------------------------------------------------# 64 | import boto3 65 | def lambda_handler(event, context): 66 | Region = event['region'] 67 | Account = event['account'] 68 | RepositoryName = event['detail']['repositoryName'] 69 | NewBranch = event['detail']['referenceName'] 70 | Event = event['detail']['event'] 71 | if NewBranch == "master": 72 | quit() 73 | if Event == "referenceCreated": 74 | cf_client = boto3.client('cloudformation') 75 | cf_client.create_stack( 76 | StackName=f'Pipeline-{RepositoryName}-{NewBranch}', 77 | TemplateURL=f'https://s3.amazonaws.com/{Account}-templates/TemplatePipeline.yaml', 78 | Parameters=[ 79 | { 80 | 'ParameterKey': 'RepositoryName', 81 | 'ParameterValue': RepositoryName, 82 | 'UsePreviousValue': False 83 | }, 84 | { 85 | 'ParameterKey': 'BranchName', 86 | 'ParameterValue': NewBranch, 87 | 'UsePreviousValue': False 88 | } 89 | ], 90 | OnFailure='ROLLBACK', 91 | Capabilities=['CAPABILITY_NAMED_IAM'] 92 | ) 93 | else: 94 | cf_client = boto3.client('cloudformation') 95 | cf_client.delete_stack( 96 | StackName=f'Pipeline-{RepositoryName}-{NewBranch}' 97 | ) 98 | ````` 99 | 100 | The logic for creating only the CI or the CI+CD is on the AWS CloudFormation template. The Conditions section of AWS CloudFormation analyzes the new branch name. 101 | 102 | ````` 103 | Conditions: 104 | BranchMaster: !Equals [ !Ref BranchName, "master" ] 105 | BranchDevelop: !Equals [ !Ref BranchName, "develop"] 106 | Setup: !Equals [ !Ref Setup, true ] 107 | ````` 108 | 109 | * If the new branch is named master, then a stack will be created containing CI+CD pipelines, with deploy stages in the homologation and production environments. 110 | * If the new branch is named develop, then a stack will be created containing CI+CD pipelines, with a deploy stage in the Dev environment. 111 | * If the new branch has any other name, then the stack will be created with only a CI pipeline. 112 | 113 | *NOTE: Since the purpose of this blog post is to present only a sample of automated pipelines creation, the pipelines used here are for examples only: they don't deploy to any environment.* 114 | 115 | 116 | 117 | ## Applicability 118 | 119 | This event-driven strategy permits pipelines to be created or deleted along with the branches. Since the entire environment is created using Infrastructure as Code and the template is the same for all pipelines, there is no possibility of different configuration issues between environments and pipeline stages. 120 | 121 | A GitFlow simulation could resemble that shown in the following diagram: 122 | 123 | 124 | 1. First, a CodeCommit repository is created, along with the master branch and its respective pipeline (CI+CD). 125 | 2. The developer creates a branch called develop based on the master branch. The pipeline (CI+CD at Dev) is automatically created. 126 | The developer creates a feature-branch called feature-a based on the develop branch. The CI pipeline for this branch is automatically created. 127 | 3. The developer creates a Pull Request from the feature-a branch to the develop branch. As soon as the Pull Request is accepted and merged and the feature-a branch is deleted, its pipeline is automatically deleted. 128 | 4. The same process can be followed for the release branch and hotfix branch. Once the branch is created, a new pipeline is created for it which follows its branch lifecycle. 129 | 130 | 131 | 132 | ## Implementation 133 | 134 | Before you start, make sure that the AWS CLI is installed and configured on your machine by following these steps: 135 | 136 | 1. Clone the repository. 137 | 2. Create the prerequisites stack. 138 | 3. Copy the AWS CloudFormation template to Amazon S3. 139 | 4. Copy the seed.zip file to the Amazon S3 bucket. 140 | 5. Create the first repository and its pipeline. 141 | 6. Create the develop branch. 142 | 7. Create the first feature branch. 143 | 8. Create the first Pull Request. 144 | 9. Execute the Pull Request approval. 145 | 10. Cleanup. 146 | 147 | 148 | ### 1. Clone the repository 149 | 150 | Clone the repository with the sample code. 151 | 152 | The main files are: 153 | * Setup.yaml: an AWS CloudFormation template for creating pipeline prerequisites. 154 | * TemplatePipeline.yaml: an AWS CloudFormation template for pipeline creation. 155 | * seed/buildspec/CIAction.yaml: a configuration file for an AWS CodeBuild project at the CI stage. 156 | * seed/buildspec/CDAction.yaml: a configuration file for a CodeBuild project at the CD stage. 157 | 158 | 159 | ````` 160 | # Command to clone the repository 161 | git clone https://github.com/aws-samples/aws-codepipeline-multi-branch-strategy.git 162 | cd aws-codepipeline-multi-branch-strategy 163 | ````` 164 | 165 | 166 | ### 2. Create the prerequisite stack 167 | 168 | The Setup stack creates the resources that are prerequisites for pipeline creation, as shown in the following chart. 169 | 170 | These resources are created only once and they fit all the pipelines created in this example. 171 | 172 | 173 | ````` 174 | # Command to create Setup stack 175 | aws cloudformation deploy --stack-name Setup-Pipeline \ 176 | --template-file Setup.yaml --region us-east-1 --capabilities CAPABILITY_NAMED_IAM 177 | ````` 178 | 179 | 180 | ### 3. Copy the AWS CloudFormation template to Amazon S3 181 | 182 | For the Lambda function to deploy a new pipeline stack, it needs to get the AWS CloudFormation template from somewhere. To enable it to do so, you need to save the template inside the Amazon S3 bucket that you just created at the Setup stack. 183 | 184 | ````` 185 | # Command that copy Template to S3 Bucket 186 | aws s3 cp TemplatePipeline.yaml s3://"$(aws sts get-caller-identity --query Account --output text)"-templates/ --acl private 187 | ````` 188 | 189 | 190 | ### 4. Copy the seed.zip file to the Amazon S3 bucket 191 | 192 | CodeCommit permits you to populate a repository at the moment of its creation as a first commit. The content of this first commit can be saved in a .zip file in an Amazon S3 bucket. Use this CodeCommit option to populate your repository with BuildSpec files for CodeBuild. 193 | 194 | ````` 195 | 196 | # Command to create zip file with the Buildspec folder content. 197 | zip -r seed.zip buildspec 198 | 199 | # Command that copy seed.zip file to S3 Bucket. 200 | aws s3 cp seed.zip s3://"$(aws sts get-caller-identity --query Account --output text)"-templates/ --acl private 201 | ````` 202 | 203 | 204 | ### 5. Create the first repository and its pipeline 205 | 206 | Now that the Setup stack is created and the seed file is stored in an Amazon S3 bucket, create the first CodeCommit repository. Every time that you want to create a new repository, execute the command below to create a new stack. 207 | 208 | ````` 209 | # Command to create the stack with the CodeCommit repository, 210 | # CodeBuild Projects and the Pipeline for the master branch. 211 | # Note: Change "myapp" by the name you want. 212 | 213 | RepoName="myapp" 214 | aws cloudformation deploy --stack-name Repo-$RepoName --template-file TemplatePipeline.yaml \ 215 | --parameter-overrides RepositoryName=$RepoName Setup=true \ 216 | --region us-east-1 --capabilities CAPABILITY_NAMED_IAM 217 | ````` 218 | 219 | When the stack is created, in addition to the CodeCommit repository, the CodeBuild projects and the master branch pipeline are also created. By default, a CodeCommit repository is created empty, with no branch. When the repository is populated with the seed.zip file, the master branch is created. 220 | 221 | 222 | Access the CodeCommit repository to see the seed files at the master branch. Access the CodePipeline console to see that there's a new pipeline with the name as the repository. This pipeline contains the CI+CD stages (homolog and prod). 223 | 224 | 225 | 226 | ### 6. Create the develop branch 227 | 228 | To simulate a real development scenario, create a new branch called develop based on the master branch. In the GitFlow concept these two (master and develop) branches are fixed and never deleted. 229 | 230 | When this new branch is created, the Events rule identifies that there's a change on this repository and triggers the CreatePipeline Lambda function to create a new pipeline for this branch. Access the CodePipeline console to see that there's a new pipeline with the name of the repository plus the branch name. This pipeline contains the CI+CD stages (Dev). 231 | 232 | ````` 233 | # Configure Git Credentials using AWS CLI Credential Helper 234 | mkdir myapp 235 | cd myapp 236 | git config --global credential.helper '!aws codecommit credential-helper $@' 237 | git config --global credential.UseHttpPath true 238 | 239 | # Clone the CodeCommit repository 240 | # You can get the URL in the CodeCommit Console 241 | git clone https://git-codecommit.us-east-1.amazonaws.com/v1/repos/myapp . 242 | 243 | # Create the develop branch 244 | # For more details: https://docs.aws.amazon.com/codecommit/latest/userguide/how-to-create-branch.html 245 | git checkout -b develop 246 | git push origin develop 247 | ````` 248 | 249 | 250 | ### 7. Create the first feature branch 251 | 252 | Now that there are two main and fixed branches (master and develop), you can create a feature branch. In the GitFlow concept, feature branches have a short lifetime and are frequently merged to the develop branch. This type of branch only exists during the development period. When the feature development is finished, it is merged to the develop branch and the feature branch is deleted. 253 | 254 | ````` 255 | # Create the feature-branch branch 256 | # make sure that you are at develop branch 257 | git checkout -b feature-abc 258 | git push origin feature-abc 259 | ````` 260 | 261 | Just as with the develop branch, when this new branch is created, the Events rule triggers the CreatePipeline Lambda function to create a new pipeline for this branch. Access the CodePipeline console to see that there's a new pipeline with the name of the repository plus the branch name. This pipeline contains only the CI stage, without a CD stage. 262 | 263 | 264 | 265 | ### 8. Create the first Pull Request 266 | 267 | It’s possible to simulate the end of a feature development, when the branch is ready to be merged with the develop branch. Keep in mind that the feature branch has a short lifecycle:  when the merge is done, the feature branch is deleted along with its pipeline. 268 | 269 | ````` 270 | # Create the Pull Request 271 | aws codecommit create-pull-request --title "My Pull Request" \ 272 | --description "Please review these changes by Tuesday" \ 273 | --targets repositoryName=$RepoName,sourceReference=feature-abc,destinationReference=develop \ 274 | --region us-east-1 275 | ````` 276 | 277 | 278 | ### 9. Execute the Pull Request approval 279 | 280 | To merge the feature branch to the develop branch, the Pull Request needs to be approved. In a real scenario, a teammate should do a peer review before approval. 281 | 282 | ````` 283 | # Accept the Pull Request 284 | # You can get the Pull-Request-ID in the json output of the create-pull-request command 285 | aws codecommit merge-pull-request-by-fast-forward --pull-request-id \ 286 | --repository-name $RepoName --region us-east-1 287 | 288 | # Delete the feature-branch 289 | aws codecommit delete-branch --repository-name $RepoName --branch-name feature-abc --region us-east-1 290 | ````` 291 | 292 | The new code is integrated with the develop branch. If there's a conflict, it needs to be solved. After that, the featurebranch is deleted together with its pipeline. The Event Rule triggers the CreatePipeline Lambda function to delete the pipeline for its branch. Access the CodePipeline console to see that the pipeline for the feature branch is deleted. 293 | 294 | 295 | ### 10. Cleanup 296 | 297 | To remove the resources created as part of this blog post, follow these steps: 298 | 299 | #### Delete Pipeline Stacks 300 | ````` 301 | # Delete all the pipeline Stacks created by CreatePipeline Lambda 302 | Pipelines=$(aws cloudformation list-stacks --stack-status-filter --region us-east-1 --query 'StackSummaries[? StackStatus==`CREATE_COMPLETE` && starts_with(StackName, `Pipeline`) == `true`].[StackName]' --output text) 303 | while read -r Pipeline rest; do aws cloudformation delete-stack --stack-name $Pipeline --region us-east-1 ; done <<< $Pipelines 304 | ````` 305 | 306 | #### Delete Repository Stacks 307 | ````` 308 | # Delete all the Repository stacks Stacks created by Step 5. 309 | Repos=$(aws cloudformation list-stacks --stack-status-filter --region us-east-1 --query 'StackSummaries[? StackStatus==`CREATE_COMPLETE` && starts_with(StackName, `Repo-`) == `true`].[StackName]' --output text) 310 | while read -r Repo rest; do aws cloudformation delete-stack --stack-name $Repo --region us-east-1 ; done <<< $Repos 311 | ````` 312 | 313 | #### Delete Setup-Pipeline Stack 314 | ````` 315 | # Cleaning Bucket before Stack deletion 316 | aws s3 rm s3://"$(aws sts get-caller-identity --query Account --output text)"-templates --recursive 317 | 318 | # Delete Setup Stack 319 | aws cloudformation delete-stack --stack-name Setup-Pipeline --region us-east-1 320 | ````` 321 | 322 | 323 | 324 | ## Conclusion 325 | 326 | This blog post discussed how you can work with event-driven strategy and Infrastructure as Code to implement a multi-branch pipeline flow using CodePipeline. It demonstrated how an Events rule and Lambda function can be used to fully orchestrate the creation and deletion of pipelines. 327 | 328 | 329 | 330 | ## License 331 | 332 | This library is licensed under the MIT-0 License. See the LICENSE file. 333 | -------------------------------------------------------------------------------- /Setup.yaml: -------------------------------------------------------------------------------- 1 | # Copyright 2019 Amazon.com, Inc. and its affiliates. All Rights Reserved. 2 | # 3 | # Licensed under the Amazon Software License (the 'License'). 4 | # You may not use this file except in compliance with the License. 5 | # A copy of the License is located at 6 | # 7 | # http://aws.amazon.com/asl/ 8 | # 9 | # or in the 'license' file accompanying this file. This file is distributed 10 | # on an 'AS IS' BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 11 | # express or implied. See the License for the specific language governing 12 | # permissions and limitations under the License. 13 | 14 | AWSTemplateFormatVersion: "2010-09-09" 15 | Description: "Create Lambda Function" 16 | 17 | Resources: 18 | 19 | #----------------------------------------------------------------------# 20 | # EventRule to trigger CreatePipeline lambda 21 | #----------------------------------------------------------------------# 22 | CreatePipelineRule: 23 | Type: AWS::Events::Rule 24 | Properties: 25 | Description: "EventRule" 26 | EventPattern: 27 | source: 28 | - aws.codecommit 29 | detail-type: 30 | - 'CodeCommit Repository State Change' 31 | detail: 32 | event: 33 | - referenceDeleted 34 | - referenceCreated 35 | referenceType: 36 | - branch 37 | State: ENABLED 38 | Targets: 39 | - Arn: !GetAtt CreatePipeline.Arn 40 | Id: CreatePipeline 41 | 42 | LambdaInvokePermission: 43 | Type: AWS::Lambda::Permission 44 | Properties: 45 | FunctionName: !Ref CreatePipeline 46 | Action: 'lambda:InvokeFunction' 47 | Principal: events.amazonaws.com 48 | SourceArn: 49 | Fn::GetAtt: 50 | - CreatePipelineRule 51 | - Arn 52 | 53 | 54 | #----------------------------------------------------------------------# 55 | # Role for lambda execution 56 | #----------------------------------------------------------------------# 57 | LambdaRole: 58 | Type: AWS::IAM::Role 59 | Properties: 60 | RoleName: LambdaRole 61 | AssumeRolePolicyDocument: 62 | Version: 2012-10-17 63 | Statement: 64 | - 65 | Effect: Allow 66 | Principal: 67 | Service: 68 | - lambda.amazonaws.com 69 | Action: 70 | - sts:AssumeRole 71 | ManagedPolicyArns: 72 | - arn:aws:iam::aws:policy/AWSLambdaFullAccess 73 | - arn:aws:iam::aws:policy/AWSCloudFormationFullAccess 74 | - arn:aws:iam::aws:policy/AWSCodePipelineFullAccess 75 | Path: / 76 | 77 | 78 | #----------------------------------------------------------------------# 79 | # Role for Pipeline Execution 80 | #----------------------------------------------------------------------# 81 | CodePipelineRole: 82 | Type: AWS::IAM::Role 83 | Properties: 84 | RoleName: CodePipelineRole 85 | ManagedPolicyArns: 86 | - arn:aws:iam::aws:policy/AWSCodePipelineFullAccess 87 | - arn:aws:iam::aws:policy/AWSCodeCommitFullAccess 88 | - arn:aws:iam::aws:policy/AWSCodeBuildDeveloperAccess 89 | - arn:aws:iam::aws:policy/AmazonS3FullAccess 90 | Path: / 91 | AssumeRolePolicyDocument: 92 | Version: 2012-10-17 93 | Statement: 94 | - Effect: Allow 95 | Principal: 96 | Service: 97 | - codepipeline.amazonaws.com 98 | Action: 99 | - 'sts:AssumeRole' 100 | 101 | 102 | 103 | #----------------------------------------------------------------------# 104 | # Role for CodeBuild Execution 105 | #----------------------------------------------------------------------# 106 | CodeBuildRole: 107 | Type: AWS::IAM::Role 108 | Properties: 109 | RoleName: CodeBuildRole 110 | ManagedPolicyArns: 111 | - arn:aws:iam::aws:policy/AdministratorAccess 112 | Path: / 113 | AssumeRolePolicyDocument: 114 | Version: 2012-10-17 115 | Statement: 116 | - Effect: Allow 117 | Principal: 118 | Service: 119 | - codebuild.amazonaws.com 120 | Action: 121 | - 'sts:AssumeRole' 122 | 123 | 124 | #----------------------------------------------------------------------# 125 | # S3 Bucket to store template 126 | #----------------------------------------------------------------------# 127 | TemplateBucket: 128 | Type: 'AWS::S3::Bucket' 129 | Properties: 130 | BucketName: !Sub ${AWS::AccountId}-templates 131 | 132 | TemplateBucketPolicy: 133 | Type: AWS::S3::BucketPolicy 134 | Properties: 135 | Bucket: !Ref TemplateBucket 136 | PolicyDocument: 137 | Statement: 138 | - 139 | Action: 140 | - s3:* 141 | Effect: Allow 142 | Resource: 143 | - !Sub arn:aws:s3:::${TemplateBucket} 144 | - !Sub arn:aws:s3:::${TemplateBucket}/* 145 | Principal: 146 | AWS: 147 | - !Sub arn:aws:iam::${AWS::AccountId}:root 148 | 149 | 150 | 151 | #----------------------------------------------------------------------# 152 | # Lambda for Stack Creation 153 | #----------------------------------------------------------------------# 154 | CreatePipeline: 155 | DependsOn: LambdaRole 156 | Type: "AWS::Lambda::Function" 157 | Properties: 158 | FunctionName: CreatePipeline 159 | Handler: "index.lambda_handler" 160 | Role: !GetAtt LambdaRole.Arn 161 | Runtime: "python3.6" 162 | Timeout: 25 163 | Code: 164 | ZipFile: | 165 | import boto3 166 | def lambda_handler(event, context): 167 | Region=event['region'] 168 | Account = event['account'] 169 | RepositoryName = event['detail']['repositoryName'] 170 | NewBranch = event['detail']['referenceName'] 171 | Event = event['detail']['event'] 172 | if NewBranch == "master": 173 | quit() 174 | if Event == "referenceCreated": 175 | cf_client = boto3.client('cloudformation') 176 | cf_client.create_stack( 177 | StackName= f'Pipeline-{RepositoryName}-{NewBranch}', 178 | TemplateURL= f'https://s3.amazonaws.com/{Account}-templates/TemplatePipeline.yaml', 179 | Parameters=[ 180 | { 181 | 'ParameterKey': 'RepositoryName', 182 | 'ParameterValue': RepositoryName, 183 | 'UsePreviousValue': False 184 | }, 185 | { 186 | 'ParameterKey': 'BranchName', 187 | 'ParameterValue': NewBranch, 188 | 'UsePreviousValue': False 189 | } 190 | ], 191 | OnFailure='ROLLBACK', 192 | Capabilities=['CAPABILITY_NAMED_IAM'] 193 | ) 194 | else: 195 | cf_client = boto3.client('cloudformation') 196 | cf_client.delete_stack( 197 | StackName= f'Pipeline-{RepositoryName}-{NewBranch}' 198 | ) 199 | -------------------------------------------------------------------------------- /TemplatePipeline.yaml: -------------------------------------------------------------------------------- 1 | # Copyright 2019 Amazon.com, Inc. and its affiliates. All Rights Reserved. 2 | # 3 | # Licensed under the Amazon Software License (the 'License'). 4 | # You may not use this file except in compliance with the License. 5 | # A copy of the License is located at 6 | # 7 | # http://aws.amazon.com/asl/ 8 | # 9 | # or in the 'license' file accompanying this file. This file is distributed 10 | # on an 'AS IS' BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either 11 | # express or implied. See the License for the specific language governing 12 | # permissions and limitations under the License. 13 | 14 | 15 | AWSTemplateFormatVersion: "2010-09-09" 16 | Description: "Pipeline Template" 17 | 18 | Parameters: 19 | 20 | RepositoryName: 21 | Type: String 22 | BranchName: 23 | Type: String 24 | Default: "master" 25 | Setup: 26 | Type: String 27 | Default: false 28 | 29 | 30 | Conditions: 31 | BranchMaster: !Equals [ !Ref BranchName, "master" ] 32 | BranchDevelop: !Equals [ !Ref BranchName, "develop"] 33 | Setup: !Equals [ !Ref Setup, true ] 34 | 35 | 36 | Resources: 37 | 38 | 39 | #----------------------------------------------------------------------# 40 | # Create CodeCommit Repository 41 | # Condition: Only at Microservice Setup 42 | #----------------------------------------------------------------------# 43 | Repository: 44 | Type: AWS::CodeCommit::Repository 45 | Condition: Setup 46 | Properties: 47 | RepositoryName: !Ref RepositoryName 48 | Code: 49 | S3: 50 | Bucket: !Sub '${AWS::AccountId}-templates' 51 | Key: seed.zip 52 | 53 | 54 | #----------------------------------------------------------------------# 55 | # Resource CodePipeline 56 | #----------------------------------------------------------------------# 57 | Pipeline: 58 | Type: "AWS::CodePipeline::Pipeline" 59 | Properties: 60 | Name: !Join ['-', [!Ref RepositoryName, !Ref BranchName]] 61 | RoleArn: !Sub 'arn:aws:iam::${AWS::AccountId}:role/CodePipelineRole' 62 | ArtifactStore: 63 | Type: S3 64 | Location: !Sub '${AWS::AccountId}-templates' 65 | Stages: 66 | - Name: Source 67 | Actions: 68 | - Name: App 69 | ActionTypeId: 70 | Category: Source 71 | Owner: AWS 72 | Version: "1" 73 | Provider: CodeCommit 74 | Configuration: 75 | RepositoryName: !Ref RepositoryName 76 | BranchName: !Ref BranchName 77 | OutputArtifacts: 78 | - Name: Source 79 | RunOrder: 1 80 | 81 | 82 | 83 | #----------------------------------------------------------------------# 84 | # Regardless of branch type, the CI Stage will always be created. 85 | #----------------------------------------------------------------------# 86 | - Name: Continuous-Integration 87 | Actions: 88 | - Name: CI-Action 89 | ActionTypeId: 90 | Category: Build 91 | Owner: AWS 92 | Version: "1" 93 | Provider: CodeBuild 94 | OutputArtifacts: 95 | - Name: CIAction 96 | InputArtifacts: 97 | - Name: Source 98 | Configuration: 99 | ProjectName: !Join ['-', [!Ref 'RepositoryName', 'CIAction' ]] 100 | RunOrder: 1 101 | 102 | 103 | 104 | #----------------------------------------------------------------------# 105 | # If BranchName=Develop, then create the CD Stage to deploy to Dev 106 | #----------------------------------------------------------------------# 107 | - !If 108 | - BranchDevelop 109 | - Name: Deploy-Dev 110 | Actions: 111 | - Name: CDActionDev 112 | ActionTypeId: 113 | Category: Build 114 | Owner: AWS 115 | Version: "1" 116 | Provider: CodeBuild 117 | InputArtifacts: 118 | - Name: Source 119 | OutputArtifacts: 120 | - Name: CDActionDev 121 | Configuration: 122 | ProjectName: !Join ['-', [!Ref 'RepositoryName', 'CDActionDev']] 123 | RunOrder: 1 124 | - !Ref AWS::NoValue 125 | 126 | 127 | 128 | #----------------------------------------------------------------------# 129 | # If BranchName=Master, then create the CD Stage to deploy to Homolog 130 | #----------------------------------------------------------------------# 131 | - !If 132 | - BranchMaster 133 | - Name: Deploy-Homolog 134 | Actions: 135 | - Name: CDActionHomolog 136 | ActionTypeId: 137 | Category: Build 138 | Owner: AWS 139 | Version: "1" 140 | Provider: CodeBuild 141 | InputArtifacts: 142 | - Name: Source 143 | OutputArtifacts: 144 | - Name: CDActionHomolog 145 | Configuration: 146 | ProjectName: !Join ['-', [!Ref 'RepositoryName', 'CDActionHomolog']] 147 | RunOrder: 1 148 | - !Ref AWS::NoValue 149 | 150 | 151 | 152 | #----------------------------------------------------------------------# 153 | # If BranchName=Master, then create CD Stage to deploy to Prod 154 | #----------------------------------------------------------------------# 155 | - !If 156 | - BranchMaster 157 | - Name: Deploy-Prod 158 | Actions: 159 | - Name: CDActionProd 160 | ActionTypeId: 161 | Category: Build 162 | Owner: AWS 163 | Version: "1" 164 | Provider: CodeBuild 165 | InputArtifacts: 166 | - Name: Source 167 | OutputArtifacts: 168 | - Name: CDActionProd 169 | Configuration: 170 | ProjectName: !Join ['-', [!Ref 'RepositoryName', 'CDActionProd']] 171 | RunOrder: 2 172 | - !Ref AWS::NoValue 173 | 174 | 175 | 176 | #----------------------------------------------------------------------# 177 | # CodeBuild Projects 178 | #----------------------------------------------------------------------# 179 | CIAction: 180 | Condition: Setup 181 | Type: AWS::CodeBuild::Project 182 | Properties: 183 | Name: !Join ['-', [!Ref 'RepositoryName', 'CIAction' ]] 184 | Source: 185 | Type: CODEPIPELINE 186 | BuildSpec: 'buildspec/CIAction.yaml' 187 | Environment: 188 | Type: LINUX_CONTAINER 189 | ComputeType: BUILD_GENERAL1_SMALL 190 | Image: aws/codebuild/standard:2.0 191 | EnvironmentVariables: 192 | - Name: BranchName 193 | Value: !Ref BranchName 194 | Artifacts: 195 | Type: CODEPIPELINE 196 | ServiceRole: !Sub 'arn:aws:iam::${AWS::AccountId}:role/CodeBuildRole' 197 | TimeoutInMinutes: 10 198 | 199 | CDActionDev: 200 | Condition: Setup 201 | Type: AWS::CodeBuild::Project 202 | Properties: 203 | Name: !Join ['-', [!Ref 'RepositoryName', 'CDActionDev' ]] 204 | Source: 205 | Type: CODEPIPELINE 206 | BuildSpec: 'buildspec/CDAction.yaml' 207 | Environment: 208 | Type: LINUX_CONTAINER 209 | ComputeType: BUILD_GENERAL1_SMALL 210 | Image: aws/codebuild/standard:2.0 211 | EnvironmentVariables: 212 | - Name: pipeline_environment 213 | Value: DEV 214 | - Name: BranchName 215 | Value: !Ref BranchName 216 | Artifacts: 217 | Type: CODEPIPELINE 218 | ServiceRole: !Sub 'arn:aws:iam::${AWS::AccountId}:role/CodeBuildRole' 219 | TimeoutInMinutes: 10 220 | 221 | CDActionHomolog: 222 | Condition: Setup 223 | Type: AWS::CodeBuild::Project 224 | Properties: 225 | Name: !Join ['-', [!Ref 'RepositoryName', 'CDActionHomolog']] 226 | Source: 227 | Type: CODEPIPELINE 228 | BuildSpec: 'buildspec/CDAction.yaml' 229 | Environment: 230 | Type: LINUX_CONTAINER 231 | ComputeType: BUILD_GENERAL1_SMALL 232 | Image: aws/codebuild/standard:2.0 233 | EnvironmentVariables: 234 | - Name: pipeline_environment 235 | Value: HOMOLOG 236 | - Name: BranchName 237 | Value: !Ref BranchName 238 | Artifacts: 239 | Type: CODEPIPELINE 240 | ServiceRole: !Sub 'arn:aws:iam::${AWS::AccountId}:role/CodeBuildRole' 241 | TimeoutInMinutes: 10 242 | 243 | CDActionProd: 244 | Condition: Setup 245 | Type: AWS::CodeBuild::Project 246 | Properties: 247 | Name: !Join ['-', [!Ref 'RepositoryName', 'CDActionProd']] 248 | Source: 249 | Type: CODEPIPELINE 250 | BuildSpec: 'buildspec/CDAction.yaml' 251 | Environment: 252 | Type: LINUX_CONTAINER 253 | ComputeType: BUILD_GENERAL1_SMALL 254 | Image: aws/codebuild/standard:2.0 255 | EnvironmentVariables: 256 | - Name: pipeline_environment 257 | Value: PROD 258 | - Name: BranchName 259 | Value: !Ref BranchName 260 | Artifacts: 261 | Type: CODEPIPELINE 262 | ServiceRole: !Sub 'arn:aws:iam::${AWS::AccountId}:role/CodeBuildRole' 263 | TimeoutInMinutes: 10 264 | -------------------------------------------------------------------------------- /buildspec/CDAction.yaml: -------------------------------------------------------------------------------- 1 | version: 0.2 2 | 3 | phases: 4 | install: 5 | runtime-versions: 6 | python: 3.7 7 | 8 | build: 9 | commands: 10 | - echo "Running CD Stage..." 11 | - echo $pipeline_environment 12 | - echo $BranchName 13 | 14 | -------------------------------------------------------------------------------- /buildspec/CIAction.yaml: -------------------------------------------------------------------------------- 1 | version: 0.2 2 | 3 | phases: 4 | install: 5 | runtime-versions: 6 | python: 3.7 7 | 8 | build: 9 | commands: 10 | - echo "CI Stage" 11 | - echo $BranchName 12 | 13 | --------------------------------------------------------------------------------