├── ConfigInfrastructure.md ├── HighAvail.md ├── Incident.md ├── Misc.md ├── MonitoringLogging.md ├── Policies.md ├── README.md └── SDLC.md /ConfigInfrastructure.md: -------------------------------------------------------------------------------- 1 | # Configuration Management and Infrastructure as Code 2 | ## Table of Contents 3 | - [Cloudformation](#Cloudformation) 4 | - [Key Terms](#Key-Terms) 5 | - [Anatomy of a Template](#Anatomy-of-a-Template) 6 | - [Intrinsic Functions](#Intrinsic-Functions) 7 | - [Fn::Base64](#FnBase64) 8 | - [Fn::Cidr](#FnCidr) 9 | - [Fn::FindInMap](#FnFindInMap) 10 | - [Fn::GetAtt](#FnGetAtt) 11 | - [Fn::GetAZs](#FnGetAZs) 12 | - [Fn::ImportValue](#FnImportValue) 13 | - [Fn::Join](#FnJoin) 14 | - [Fn::Select](#FnSelect) 15 | - [Fn::Split](#FnSplit) 16 | - [Fn::Sub](#FnSub) 17 | - [Fn::Transform](#FnTransform) 18 | - [Ref](#Ref) 19 | - [Condition Functions](#Condition-Functions) 20 | - [Condition Intrinsic Functions](#Condition-Intrinsic-Functions) 21 | - [Fn::And](#FnAnd) 22 | - [Fn::Equals](#FnEquals) 23 | - [Fn::If](#FnIf) 24 | - [Fn::Not](#FnNot) 25 | - [Fn::Or](#FnOr) 26 | - [Wait Conditions/Creation Policy](#Wait-Conditons/Creation-Policy) 27 | - [Nested Stack](#Nested-Stack) 28 | - [Resource Deletion Policies](#Resource-Deletion-Policies) 29 | - [Stack Updates](#Stack-Updates) 30 | - [Custom Resources](#Custom-Resources) 31 | - [CloudFormation Best Practices](#CloudFormation-Best-Practices) 32 | - [Elastic Beanstalk](#Elastic-Beanstalk) 33 | - [ebextentions](#ebextentions) 34 | - [AWS Config](#AWS-config) 35 | - [ECS](#ECS) 36 | - [Lambda Step Functions](#Lambda-Step-Functions) 37 | - [OpsWorks](#OpsWorks) 38 | - [Chef](#Chef) 39 | 40 | # Cloudformation 41 | ### Key Terms 42 | 1. Stack - manage related resources as a single unit called a stack. Create, update, and delete a collection of resources by creating, updating, and deleting stacks. All the resources in a stack are defined by the stack's AWS CloudFormation template. 43 | 2. Template - JSON or YAML formatted text file. AWS CloudFormation uses these templates as blueprints for building your AWS resources. 44 | 3. Stack Policy - IAM style policy statement which governs what can be changed and who can change it. 45 | 46 | ### Anatomy of a Template 47 | - Format Version (optional) 48 | - The AWS CloudFormation template version that the template conforms to. The template format version is not the same as the API or WSDL version. 49 | - Description (optional) 50 | - A text string that describes the template. This section must always follow the template format version section. 51 | - Metadata (optional) 52 | - Objects that provide additional information about the template. 53 | - **Parameters** (optional) 54 | - Values to pass to your template at runtime (when you create or update a stack). 55 | - **Mappings** (optional) 56 | - A mapping of keys and associated values that you can use to specify conditional parameter values, similar to a lookup table. You can match a key to a corresponding value by using the Fn::FindInMap intrinsic function in the Resources and Outputs sections. 57 | - Conditions (optional) 58 | - Conditions that control whether certain resources are created or whether certain resource properties are assigned a value during stack creation or update. For example, you could conditionally create a resource that depends on whether the stack is for a production or test environment. 59 | - Transform (optional) 60 | - For serverless applications (also referred to as Lambda-based applications), specifies the version of the AWS Serverless Application Model (AWS SAM) to use. When you specify a transform, you can use AWS SAM syntax to declare resources in your template. The model defines the syntax that you can use and how it is processed. 61 | - You can also use AWS::Include transforms to work with template snippets that are stored separately from the main AWS CloudFormation template. You can store your snippet files in an Amazon S3 bucket and then reuse the functions across multiple templates. 62 | - **Resources** (required) 63 | - Specifies the stack resources and their properties, such as an Amazon Elastic Compute Cloud instance or an Amazon Simple Storage Service bucket. You can refer to resources in the Resources and Outputs sections of the template. 64 | - **Outputs** (optional) 65 | - Describes the values that are returned whenever you view your stack's properties. For example, you can declare an output for an S3 bucket name and then call the aws cloudformation describe-stacks AWS CLI command to view the name. 66 | 67 | ## Intrinsic Functions 68 | ### Fn::Base64 69 | - Returns the Base64 representation of the input string. This function is typically used to pass encoded data to Amazon EC2 instances by way of the UserData property. 70 | 71 | JSON 72 | ``` 73 | { "Fn::Base64" : valueToEncode } 74 | ``` 75 | YAML 76 | ``` 77 | !Base64 valueToEncode 78 | ``` 79 | 80 | ### Fn::Cidr 81 | - Returns an array of CIDR address blocks. The number of CIDR blocks returned is dependent on the count parameter. 82 | 83 | JSON 84 | ``` 85 | { "Fn::Cidr" : [ipBlock, count, cidrBits]} 86 | ``` 87 | YAML 88 | ``` 89 | !Cidr [ ipBlock, count, cidrBits ] 90 | ``` 91 | - **ipBlock**: 92 | The user-specified CIDR address block to be split into smaller CIDR blocks. 93 | - **count**: 94 | The number of CIDRs to generate. Valid range is between 1 and 256. 95 | - **cidrBits**: 96 | The number of subnet bits for the CIDR. For example, specifying a value "8" for this parameter will create a CIDR with a mask of "/24". 97 | 98 | ### Fn::FindInMap 99 | - Returns the value corresponding to keys in a two-level map that is declared in the Mappings section 100 | JSON 101 | ``` 102 | { "Fn::FindInMap" : [ "MapName", "TopLevelKey", "SecondLevelKey"] } 103 | ``` 104 | YAML 105 | ``` 106 | !FindInMap [ MapName, TopLevelKey, SecondLevelKey ] 107 | ``` 108 | - **MapName**: 109 | - The logical name of a mapping declared in the Mappings section that contains the keys and values. 110 | - **TopLevelKey**: 111 | The top-level key name. Its value is a list of key-value pairs. 112 | - **SecondLevelKey**: 113 | The second-level key name, which is set to one of the keys from the list assigned to TopLevelKey. 114 | 115 | ### Fn::GetAtt 116 | - Returns the value of an attribute from a resource in the template. 117 | JSON 118 | ``` 119 | { "Fn::GetAtt" : [ "logicalNameOfResource", "attributeName" ] } 120 | ``` 121 | YAML 122 | ``` 123 | !GetAtt logicalNameOfResource.attributeName 124 | ``` 125 | 126 | ### Fn::GetAZs 127 | - Returns an array that lists Availability Zones for a specified region. Prevents hard-coding a full list of Availability Zones for a specified region. 128 | JSON 129 | ``` 130 | { "Fn::GetAZs" : "region" } 131 | ``` 132 | YAML 133 | ``` 134 | !GetAZs region 135 | ``` 136 | 137 | ### Fn::ImportValue 138 | - Returns the value of an output exported by another stack. You typically use this function to create cross-stack references. 139 | Note: 140 | - For each AWS account, Export names must be unique within a region. 141 | - You can't create cross-stack references across regions. 142 | - For outputs, the value of the Name property of an Export can't use Ref or GetAtt functions that depend on a resource. 143 | - Similarly, the ImportValue function can't include Ref or GetAtt functions that depend on a resource. 144 | - You can't delete a stack if another stack references one of its outputs. 145 | - You can't modify or remove an output value that is referenced by another stack. 146 | JSON 147 | ``` 148 | { "Fn::ImportValue" : sharedValueToImport } 149 | ``` 150 | YAML 151 | ``` 152 | !ImportValue sharedValueToImport 153 | ``` 154 | - Note: 155 | You can't use the short form of !ImportValue when it contains a !Sub. 156 | 157 | Example: 158 | ``` 159 | Fn::ImportValue: 160 | !Sub "${NetworkStack}-SubnetID" 161 | ``` 162 | ### Fn::Join 163 | - Appends a set of values into a single value, separated by the specified delimiter. If a delimiter is the empty string, the set of values are concatenated with no delimiter. 164 | JSON 165 | ``` 166 | { "Fn::Join" : [ "delimiter", [ comma-delimited list of values ] ] } 167 | "Fn::Join" : [ ":", [ "a", "b", "c" ] ] 168 | ``` 169 | YAML 170 | ``` 171 | !Join [ delimiter, [ comma-delimited list of values ] ] 172 | !Join [ ":", [ a, b, c ] ] 173 | !Join 174 | - '' 175 | - - 'arn:' 176 | - !Ref Partition 177 | - ':s3:::elasticbeanstalk-*-' 178 | - !Ref 'AWS::AccountId' 179 | ``` 180 | 181 | ### Fn::Select 182 | - Returns a single object from a list of objects by index. 183 | 184 | Note: 185 | - Fn::Select does not check for null values or if the index is out of bounds of the array. Both conditions will result in a stack error, so you should be certain that the index you choose is valid, and that the list contains non-null values. 186 | 187 | JSON 188 | ``` 189 | { "Fn::Select" : [ index, listOfObjects ] } 190 | { "Fn::Select" : [ "1", [ "apples", "grapes", "oranges", "mangoes" ] ] } 191 | ``` 192 | YAML 193 | ``` 194 | !Select [ index, listOfObjects ] 195 | !Select [ "1", [ "apples", "grapes", "oranges", "mangoes" ] ] 196 | ``` 197 | 198 | ### Fn::Split 199 | - Split a string into a list of string values so that you can select an element from the resulting string list. Specify the location of splits with a delimiter, such as , (a comma). After you split a string, use the Fn::Select function to pick a specific element. 200 | JSON 201 | ``` 202 | { "Fn::Split" : [ "|" , "a|b|c" ] } 203 | { "Fn::Select" : [ "2", { "Fn::Split": [",", {"Fn::ImportValue": "AccountSubnetIDs"}]}] } 204 | ``` 205 | YAML 206 | ``` 207 | !Split [ "|" , "a|b|c" ] 208 | !Select [2, !Split [",", !ImportValue AccountSubnetIDs]] 209 | ``` 210 | ### Fn::Sub 211 | - Substitutes variables in an input string with values that you specify 212 | JSON 213 | ``` 214 | { "Fn::Sub" : [ String, { Var1Name: Var1Value, Var2Name: Var2Value } ] } 215 | { "Fn::Sub": [ "www.${Domain}", { "Domain": {"Ref" : "RootDomainName" }} ]} 216 | ``` 217 | YAML 218 | ``` 219 | !Sub 220 | - String 221 | - { Var1Name: Var1Value, Var2Name: Var2Value } 222 | Name: !Sub 223 | - www.${Domain} 224 | - { Domain: !Ref RootDomainName } 225 | ``` 226 | 227 | ### Fn::Transform 228 | - Specifies a macro to perform custom processing on part of a stack template. Macros enable you to perform custom processing on templates, from simple actions like find-and-replace operations to extensive transformations of entire templates. 229 | - You can also use Fn::Transform to call the AWS::Include Transform transform, which is a macro hosted by AWS CloudFormation. 230 | JSON 231 | ``` 232 | { "Fn::Transform" : { "Name" : macro name, "Parameters" : {key : value, ... } } } 233 | ``` 234 | YAML 235 | ``` 236 | !Transform { "Name" : macro name, "Parameters" : {key : value, ... } } 237 | ``` 238 | ### Ref 239 | - Returns the value of the specified parameter or resource. 240 | JSON 241 | ``` 242 | { "Ref" : "logicalName" } 243 | ``` 244 | YAML 245 | ``` 246 | !Ref logicalName 247 | ``` 248 | 249 | ### Condition Functions 250 | - Use intrinsic functions to conditionally create stack resources. Conditions are evaluated based on input parameters that you declare when you create or update a stack. 251 | - Define all conditions in the Conditions section of a template except for Fn::If 252 | 253 | #### Condition Intrinsic Functions 254 | - ##### Fn::And 255 | - `Fn::And: [condition]` 256 | - `!And [condition]` 257 | - Examples: 258 | ``` 259 | "MyAndCondition": { 260 | "Fn::And": [ 261 | {"Fn::Equals": ["sg-mysggroup", {"Ref": "ASecurityGroup"}]}, 262 | {"Condition": "SomeOtherCondition"} 263 | ] 264 | } 265 | ``` 266 | ``` 267 | MyAndCondition: !And 268 | - !Equals ["sg-mysggroup", !Ref ASecurityGroup] 269 | - !Condition SomeOtherCondition 270 | 271 | ``` 272 | - ##### Fn::Equals 273 | - `"Fn::Equals" : ["value_1", "value_2"]` 274 | - `!Equals [value_1, value_2]` 275 | - Examples: 276 | ``` 277 | "UseProdCondition" : { 278 | "Fn::Equals": [ 279 | {"Ref": "EnvironmentType"}, 280 | "prod" 281 | ] 282 | } 283 | ``` 284 | ``` 285 | UseProdCondition: 286 | !Equals [!Ref EnvironmentType, prod] 287 | ``` 288 | - ##### Fn::If 289 | - `"Fn::If": [condition_name, value_if_true, value_if_false]` 290 | - `!If [condition_name, value_if_true, value_if_false]` 291 | - Examples: 292 | ``` 293 | "SecurityGroups" : [{ 294 | "Fn::If" : [ 295 | "CreateNewSecurityGroup", 296 | {"Ref" : "NewSecurityGroup"}, 297 | {"Ref" : "ExistingSecurityGroup"} 298 | ] 299 | }] 300 | ``` 301 | ``` 302 | SecurityGroups: 303 | - !If [CreateNewSecurityGroup, !Ref NewSecurityGroup, !Ref ExistingSecurityGroup] 304 | ``` 305 | - ##### Fn::Not 306 | - `"Fn::Not": [{condition}]` 307 | - `!Not [condition]` 308 | - Examples: 309 | ``` 310 | "MyNotCondition" : { 311 | "Fn::Not" : [{ 312 | "Fn::Equals" : [ 313 | {"Ref" : "EnvironmentType"}, 314 | "prod" 315 | ] 316 | }] 317 | } 318 | ``` 319 | ``` 320 | MyNotCondition: 321 | !Not [!Equals [!Ref EnvironmentType, prod]] 322 | ``` 323 | - ##### Fn::Or 324 | - `"Fn::Or": [{condition}, {...}]` 325 | - `!Or [condition, ...]` 326 | - Examples: 327 | ``` 328 | "MyOrCondition" : { 329 | "Fn::Or" : [ 330 | {"Fn::Equals" : ["sg-mysggroup", {"Ref" : "ASecurityGroup"}]}, 331 | {"Condition" : "SomeOtherCondition"} 332 | ] 333 | } 334 | ``` 335 | ``` 336 | MyOrCondition: 337 | !Or [!Equals [sg-mysggroup, !Ref ASecurityGroup], Condition: SomeOtherCondition] 338 | ``` 339 | ## Wait Conditons/Creation Policy 340 | 341 | **CreationPolicy** attribute prevents resource's status from reaching create complete until AWS CloudFormation receives a specified number of success signals or the timeout period is exceeded. 342 | - Use the cfn-signal helper script or SignalResource API to send signal 343 | - Only available for following resources 344 | - AWS::AutoScaling::AutoScalingGroup 345 | - AWS::EC2::Instance 346 | - AWS::CloudFormation::WaitCondition 347 | 348 | ##### Example: 349 | In template: 350 | ``` 351 | CreationPolicy: 352 | AutoScalingCreationPolicy: 353 | MinSuccessfulInstancesPercent: Integer 354 | ResourceSignal: 355 | Count: Integer 356 | Timeout: String 357 | MyInstance: 358 | Type: AWS::EC2::Instance 359 | Properties: 360 | UserData: !Base64 361 | 'Fn::Join': 362 | - '' 363 | - - | 364 | #!/bin/bash -x 365 | - | 366 | # Install the files and packages from the metadata 367 | - '/opt/aws/bin/cfn-init -v ' 368 | - ' --stack ' 369 | - !Ref 'AWS::StackName' 370 | - ' --resource MyInstance ' 371 | - ' --region ' 372 | - !Ref 'AWS::Region' 373 | - |+ 374 | 375 | - | 376 | # Signal the status from cfn-init 377 | - '/opt/aws/bin/cfn-signal -e $? ' 378 | - ' --stack ' 379 | - !Ref 'AWS::StackName' 380 | - ' --resource MyInstance ' 381 | - ' --region ' 382 | - !Ref 'AWS::Region' 383 | - |+ 384 | ``` 385 | In most conditions, CreationPolicy is preferable to **WaitConditon**. Use a wait condition for situations like the following: 386 | 387 | - To coordinate stack resource creation with configuration actions that are external to the stack creation 388 | 389 | - To track the status of a configuration process 390 | 391 | ##### Example 392 | ``` 393 | WebServerGroup: 394 | Type: AWS::AutoScaling::AutoScalingGroup 395 | Properties: 396 | AvailabilityZones: 397 | Fn::GetAZs: "" 398 | LaunchConfigurationName: 399 | Ref: "LaunchConfig" 400 | MinSize: "1" 401 | MaxSize: "5" 402 | DesiredCapacity: 403 | Ref: "WebServerCapacity" 404 | LoadBalancerNames: 405 | - 406 | Ref: "ElasticLoadBalancer" 407 | WaitHandle: 408 | Type: AWS::CloudFormation::WaitConditionHandle 409 | WaitCondition: 410 | Type: AWS::CloudFormation::WaitCondition 411 | DependsOn: "WebServerGroup" 412 | Properties: 413 | Handle: 414 | Ref: "WaitHandle" 415 | Timeout: "300" 416 | Count: 417 | Ref: "WebServerCapacity" 418 | ``` 419 | 420 | ## Nested Stack 421 | Stack is a resource which has following benefits 422 | - Overcome limits of CloudFormation 423 | - Split large number of resources over multiple templates 424 | - Reuse common template patterns 425 | 426 | 427 | ## Resource Deletion Policies 428 | Three types of DeletionPolicy for each resource 429 | - Delete (default) 430 | - Retain 431 | - Snapshot (only on a few services) 432 | 433 | ## Stack Updates 434 | Use Stack Policy to control actions 435 | Example: 436 | ``` 437 | { 438 | "Statement" : [ 439 | { 440 | "Effect" : "Allow", 441 | "Action" : "Update:*", 442 | "Principal": "*", 443 | "Resource" : "*" 444 | }, 445 | { 446 | "Effect" : "Deny", 447 | "Action" : "Update:*", 448 | "Principal": "*", 449 | "Resource" : "LogicalResourceId/ProductionDatabase" 450 | } 451 | ] 452 | } 453 | ``` 454 | - No stack policy = allow all updates 455 | - Once a stack policy is applied, can't be deleted 456 | - Once a policy is applied, all resources are protected by default 457 | - `Update:*` is denied 458 | 459 | Updates can impact a resource in 4 ways, depending on resource 460 | 1. No interruption 461 | 2. Some interruption 462 | 3. Replacement 463 | 4. Deletion 464 | 465 | ## Custom Resources 466 | Create resources outside of the available AWS resources. 467 | Involves 3 parties: 468 | 1. Template developer 469 | - Creates a template that includes a custom resource type. The template developer specifies the service token and any input data in the template. 470 | 2. custom resource provider 471 | - Owns the custom resource and determines how to handle and respond to requests from AWS CloudFormation. The custom resource provider must provide a service token that the template developer uses. 472 | 3. AWS CloudFormation 473 | - During a stack operation, sends a request to a service token that is specified in the template, and then waits for a response before proceeding with the stack operation. 474 | 475 | Steps of creating a custom resource 476 | 1. Template developer defines a custom resource in his or her template, which includes a service token and any input data parameters. 477 | - input data might be required; service token is always required. 478 | - service token specifies where AWS CloudFormation sends requests to, such as to an Amazon SNS topic ARN or to an AWS Lambda function ARN 479 | 2. During create, update, or delete a custom resource, AWS CloudFormation sends a request to the specified service token. 480 | - service token must be in the same region in which you are creating the stack. 481 | - CloudFormation includes information such as the request type and a pre-signed Amazon Simple Storage Service URL, where the custom resource sends responses 482 | 3. Custom resource provider processes the AWS CloudFormation request and returns a response of SUCCESS or FAILED to the pre-signed URL. 483 | - Response is in a JSON-formatted file 484 | - Custom resource provider can also include name-value pairs that the template developer can access 485 | 4. After getting a SUCCESS response, AWS CloudFormation proceeds with the stack operation. If a FAILED response or no response is returned, the operation fails. 486 | 487 | 488 | ## CloudFormation Best Practices 489 | - Organize Your Stacks By Lifecycle and Ownership 490 | - Two common frameworks: 491 | 1. **multi-layered architecture** 492 | - Layered architecture organizes stacks into multiple horizontal layers that build on top of one another, where each layer has a dependency on the layer directly below it. Layers can have multiple stacks grouped by owner/lifecycle. 493 | 2. **service-oriented architecture (SOA)** 494 | - A service-oriented architecture organizes problems into manageable parts, each with a clearly defined purpose representing a self-contained unit of functionality. You can map these services to a stack, with its own lifecycle and owners. 495 | - Use Cross-Stack References to Export Shared Resources 496 | - Use cross-stack references to export resources from a stack so that other stacks can use them. Stacks can use the exported resources by calling them using Fn::ImportValue 497 | - Use IAM to Control Access 498 | - Manage users permissions for viewing stack templates, creating stacks, or deleting stacks 499 | - Separate permissions between a user and the AWS CloudFormation service using a service role. AWS CloudFormation uses the service role's policy to make calls instead of the user's policy. 500 | - Verify Quotas for All Resource Types 501 | - for example: you can only launch 200 AWS CloudFormation stacks per region in your AWS account 502 | - Reuse Templates to Replicate Stacks in Multiple Environments 503 | - Use Nested Stacks to Reuse Common Template Patterns 504 | - Do Not Embed Credentials in Your Templates 505 | - Use AWS-Specific Parameter Types 506 | - For example, you can specify a parameter as type AWS::EC2::KeyPair::KeyName, which takes an existing key pair name that is in your AWS account and in the region where you are creating the stack 507 | - Use Parameter Constraints 508 | - Use AWS::CloudFormation::Init to Deploy Software Applications on Amazon EC2 Instances 509 | - Describe the configurations that you want rather than scripting procedural steps. Update configurations without recreating instances. 510 | - Use the Latest Helper Scripts 511 | - Inlcude following command in userdata: `yum install -y aws-cfn-bootstrap` 512 | - Validate Templates Before Using Them 513 | - check for valid json or yaml 514 | - Manage All Stack Resources Through AWS CloudFormation 515 | - do not update resources outside of stack template 516 | - Create Change Sets Before Updating Your Stacks 517 | - See how proposed changes to a stack might impact your running resources before you implement them. 518 | - Use Stack Policies 519 | - prevent unintentional disruptions by protecting important resources 520 | - Use Code Reviews and Revision Controls to Manage Your Templates 521 | - Update Your Amazon EC2 Linux Instances Regularly 522 | - `yum update` 523 | 524 | # Elastic Beanstalk 525 | Upload an application and Elastic Beanstalk automatically handles the deployment details of capacity provisioning, load balancing, auto-scaling, and application health monitoring 526 | - Useful for developers who only want to write code and not maintain infrastructure 527 | - supports many languages as well as Docker, allowing extensibility for languages not natively supported 528 | 529 | ## ebextentions 530 | .ebextensions allow advanced environment customization with YAML or JSON configuration files 531 | - Placed in a folder called `.ebextensions` 532 | - Used for achieving automation in creation Elastic Beanstalk environment 533 | 534 | # AWS Config 535 | Enables you to assess, audit, and evaluate the configurations of your AWS resources 536 | Allows for: 537 | - Continuous monitoring 538 | - Get SNS notifications on changes 539 | - Continuous assessment 540 | - keep resources in line with policies 541 | - Troubleshooting 542 | - compare points in time 543 | - integrates directly with cloudtrail 544 | - Compliance monitoring 545 | - view compliance status across entire enterprise in AWS 546 | - Change management 547 | - Track resource relationships 548 | - Review resource dependencies 549 | 550 | At a high level: 551 | - Define rules that are used to check compliance of resources in your account 552 | - View compliance in console or receive warnings via SNS 553 | 554 | # ECS 555 | Highly scalable, high performance container management service that supports Docker containers and allows you to easily run applications on a managed cluster of Amazon EC2 instances. 556 | - Highly scalable 557 | - Fast 558 | - Manage docker containers on a cluster 559 | - Supports API calls 560 | - Supports scheduling 561 | 562 | # Lambda Step Functions 563 | - Service that allows you to orchestrate your lambda functions 564 | - Reliable way to step through functions in a particular order 565 | - Give graphical view of application components 566 | 567 | Use cases: 568 | - Data processing: consolidate data from multiple databases into unified reports, refine and reduce large data sets into useful formats, or coordinate multi-step analytics and machine learning workflows 569 | - DevOps and IT automation: build tools for continuous integration and continuous deployment, or create event-driven applications that automatically respond to changes in infrastructure 570 | - E-commerce: automate mission-critical business processes, such as order fulfillment and inventory tracking 571 | - Web applications: implement robust user registration processes and sign-on authentication 572 | 573 | 574 | # OpsWorks 575 | There are three OpsWorks services 576 | 1. AWS OpsWorks for Chef Automate 577 | - Fully managed service configuration management service that hosts Chef 578 | 2. AWS Opsworks for Puppet Enterprise 579 | - Fully managed service configuration management service that hosts Puppet 580 | 3. AWS OpsWorks Stacks 581 | - Application and server management service 582 | - Allows application deployment in stacks 583 | - Configuration using Chef Solo 584 | - This is the service that will be in exam 585 | 586 | What OpsWorks does: 587 | - Manage application on AWS and on-premises 588 | - Design layers that perform different functions 589 | - Supports autoscaling and scheduled scaling 590 | - Implements chef solo for configuration 591 | 592 | ### Chef: 593 | OpsWorks Stacks/Chef is a declarative state engine 594 | - State what you want to happen, service handles how 595 | 596 | Opsworks stacks/Chef has resources it can use: 597 | - pacakages to install 598 | - services to control 599 | - configuraton files to update 600 | 601 | Chef Recipes 602 | - Use Chef recipes to automate everything Chef can do, such as creating directories and users, configuring databases, etc 603 | 604 | Berkshelf 605 | - Manage external cookbooks to safely include them in your stacks 606 | 607 | -------------------------------------------------------------------------------- /HighAvail.md: -------------------------------------------------------------------------------- 1 | # High Availability, Fault Tolerance, & Disaster Recovery 2 | 3 | ## Table of Contents 4 | - [Single Sign-On](#Single-Sign-On) 5 | - [CloudFront](#CloudFront) 6 | - [AutoScaling](#AutoScaling) 7 | - [AutoScaling lifecycle](#AutoScaling-Lifecycle) 8 | - [Route 53](#Route-53) 9 | - [RDS](#RDS) 10 | - [Aurora](#Aurora) 11 | - [DynamoDB](#DynamoDB) 12 | 13 | # Single Sign-On 14 | Cloud service that makes it easy to centrally manage single sign on access to multiple AWS accounts and business applications 15 | 16 | Features 17 | - Integrates with AWS Organizations 18 | - SSO access to cloud applications 19 | - Create/Manage users and groups 20 | - Use existing corporate indentities (Active Directory) 21 | 22 | Relevance 23 | - High availability 24 | - Fault tolerant 25 | 26 | # CloudFront 27 | A fast content delivery network (CDN) that securely delivers data, video, applications, and APIs to customers globally 28 | 29 | ## Steps 30 | 1. Specify origin servers, like an Amazon S3 bucket or your own HTTP server 31 | 32 | 2. An origin server stores the original, definitive version of your objects, either in an Amazon S3 bucket or an HTTP server, such as a web server 33 | 34 | 3. Adobe Media Server RTMP protocol is always an Amazon S3 bucket (streaming video) 35 | 36 | 4. Upload files to your origin servers: web pages, images, and media files, or anything that can be served over HTTP or a supported version of Adobe RTMP (Adobe Flash Media Server) 37 | 38 | 5. With S3, you can make the objects in your bucket publicly readable or keep objects private 39 | 40 | 6. Create a CloudFront distribution, which tells CloudFront which origin servers to get your files from when users request the files through your web site or application. Specify details such as whether you want CloudFront to log all requests and whether you want the distribution to be enabled as soon as it's created. 41 | 42 | 7. CloudFront assigns a domain name to your new distribution that you can see in the CloudFront console or via API. 43 | 44 | 8. CloudFront sends your distribution's configuration (but not your content) to all of its edge locations 45 | 46 | # AutoScaling 47 | Scale EC2 instance capacity automatically according to defined conditions 48 | - High availability 49 | - Better fault tolerance 50 | - Cost savings 51 | 52 | ### AutoScaling Lifecycle 53 | - Starts when AutoScaling group launches an instance 54 | - Ends when you terminate an instance 55 | - Ends when AutoScaling group takes the instance out of service and terminates it 56 | 57 | Overview 58 | - Starts to scale out 59 | - **Pending state** 60 | - Hook: EC2_INSTANCE_LAUNCHING 61 | - Pending:Wait 62 | - Pending:Proceed 63 | - In **Service State** 64 | - If instance fails or taken out of service, moves to **Terminating State** 65 | - Hook: EC2_INSTANCE_TERMINATING 66 | - Terminating:Wait 67 | - Terminating:Proceed 68 | - Optional: **Standby state** 69 | - Managed by autoscaling group, but don't receive traffic 70 | - EnteringStandby 71 | - Standby 72 | - When **Standby state** is put back into service, it enters **Pending State** 73 | - Optional: Detach Instance 74 | - Detaching state 75 | - Detached state 76 | - Can be stand-alone EC2 or re-attached to another autoscaling group 77 | 78 | How Lifecycles Work 79 | 1. Autoscaling responds to scale out event by launching an instance 80 | 2. Autoscaling puts the instance into **Pending:Wait** state 81 | 3. Autoscaling sends a message to notification target defined for the hook, along with information and token 82 | 4. Waits until you tell it or timeout ends 83 | 5. You can perform custom action, install software, etc. 84 | 6. By default, instance will wait 1 hour and change state to **Pending:Proceed**, then enter **InService** state 85 | 86 | Notes on Lifecycle Hooks 87 | - You can change heartbeat timeout or define it when you create the hook in the CLI with `heartbeat-timeout` parameter 88 | - You can call the `complete-lifecycle-action` to tell autoscaling group to proceed 89 | - You can call the `record-lifecycle-action-heartbeat` command to add more time to timeout 90 | - 48 hours is the maximum amount you can keep server in wait state 91 | 92 | Cooldowns 93 | - Cooldowns ensure that autoscaling group does not launch or terminate more instances than needed 94 | - Cooldowns start when an instance enter InService state, so if instance is left in Pending:Wait, autoscaling will wait before adding any other servers 95 | 96 | Abandon or Continue 97 | - At conclusion of lifecycle hook, an instance can result in two states: `abandon` or `continue` 98 | - Abandon will cause autoscaling group to terminate instance and, if necessary, launch a new one 99 | - Continue will put instance into service 100 | 101 | Spot Instances 102 | - can use lifecycle hooks with spot instances 103 | - does NOT stop an instance from terminating due to change in Spot Price 104 | - When spot instance terminates, you must complete lifecycle action 105 | 106 | # Route 53 107 | Highly available and scalable cloud Domain Name System (DNS) service 108 | 109 | Features 110 | - Highly Available 111 | - Interface directly with EC2 and S3 112 | - Fault Tolerant 113 | - multiple routing types (e.g. latency based routing, weighted round robin) 114 | 115 | ### Routing Policies 116 | - Failover routing policy 117 | - for active-passive failure 118 | - Geolocation routing policy 119 | - route based on location of users 120 | - Geoproximity routing policy 121 | - route traffic based on location of resources and users 122 | - Latency routing policy 123 | - route traffic to region with best latency 124 | - Multivalue answer routing policy 125 | - route randomly to up to 8 destinations 126 | - Weighted routing policy 127 | - route to different resources using a percentage split 128 | - good for **A/B Testing** 129 | 130 | 131 | #RDS 132 | Amazon Relational Database Service - create, run, scale relational DBs in the cloud 133 | 134 | Benefits 135 | - Fast 136 | - Cost effecient 137 | - Resizable 138 | - Secure 139 | - Highly Available 140 | - Minimal administration 141 | 142 | Supported DB engines 143 | - MySQL 144 | - MariaDB 145 | - Microsoft SQL Server 146 | - PostgreSQL 147 | - Oracle 148 | - Amazon Aurora 149 | 150 | Managemed Administration 151 | - AWS 152 | - Provision infrastructure 153 | - install DB software 154 | - automatic backups 155 | - automatic patching 156 | - synchronous data replication 157 | - automatic failover 158 | - You 159 | - Settings 160 | - Schema 161 | - Performance tuning 162 | 163 | Scaling 164 | - Vertical: 165 | - Change instance type 166 | - discounts for reserved instances 167 | - storage can be scaled live except MSSQL 168 | - Horizontal 169 | - Read replicas 170 | - Read-only copies synched with master db 171 | - can be in different regions 172 | - can be promoted for disaster recovery 173 | - can create a replica of a replica 174 | 175 | # Aurora 176 | MySql and Postgres compatible DB built for the cloud 177 | 178 | Benefits 179 | - fast and reliable 180 | - simple 181 | - cost effective 182 | - 5x throughput of mysql on same hardware 183 | - compatibly with mysql 5.7 184 | - fault tolerant and self healing 185 | - disk failures repaired in background 186 | - detects crashes and restarts 187 | - no crash recovery or cache rebuilding required 188 | - automatic failover to one of up to 15 read replicas 189 | - storage autoscaling 190 | 191 | Backups 192 | - Automatic, continuous, or incremental 193 | - point-in-time restore within second 194 | - up to 35 day retention period 195 | - no impact on db performance 196 | 197 | Snapshots 198 | - user-initiated snapshots stored in s3 199 | - kept until explicitly deleted 200 | - incremental 201 | 202 | Failure and Fault Tolerance 203 | - maintains 6 copies of data across 3 AZs 204 | - point in time/snapshot restore 205 | - data divided into 10 gb segments across many disks 206 | - transparently handles loss 207 | - can lose 2 copies without affecting ability to write 208 | - can lose 3 copies without affecting ability to read 209 | - self-healing 210 | 211 | Replicas 212 | - Amazon Aurora replicas 213 | - share underlying volume with primary instance 214 | - updates made by primary are visible to all replicas 215 | - up to 15 216 | - low performance impact on memory 217 | - replica can be failover target 218 | - MySQL read replicas 219 | - Primary instance data is replayed on replica as transactions 220 | - up to 5 221 | - high performance impact on primary 222 | - replica can be target, but may have minutes of data loss 223 | 224 | Security 225 | - Must be in VPC 226 | - SSL secures data in transit 227 | - can encrypt data with KMS 228 | - encrypted storage at storage/backup/snapshots/replicas 229 | - NOTE: you can't encrypt an unencrypted database 230 | 231 | # DynamoDB 232 | Fully managed NoSQL database that supports key-value and document data structures 233 | 234 | Details 235 | - no servers 236 | - no storage limitations 237 | - fully resilient, highly available 238 | - performance scales in a linear way 239 | - fully intregrated with IAM 240 | 241 | Concepts 242 | - Collection of tables, highest level of structure 243 | - specify performance requirements on table 244 | - WCUs - write capacity units - number of 1 kb writes per second 245 | - RCUs - read capacity units - number of 4 kb reads per second 246 | 247 | DynamoDB Tables 248 | - contain items 249 | - items contain attributes (including partition key, which must be unique) 250 | - attributes can also include sort key 251 | 252 | Attribute Types 253 | - string 254 | - number 255 | - binary 256 | - boolean 257 | - null 258 | - Document (lists, maps) 259 | - Set (array) 260 | 261 | Streams 262 | - Optional feature capturing data modification events 263 | - Data events appear in the stream in near real time and in order 264 | - Each event is represented by a stream record when 265 | - new item is added 266 | - item is update 267 | - item is deleted 268 | - Can trigger lambda when a certain event appears in the stream 269 | -------------------------------------------------------------------------------- /Incident.md: -------------------------------------------------------------------------------- 1 | # Incident and Event Response 2 | 3 | ## Table of Contents 4 | - [GuardDuty](#GuardDuty) 5 | - [Amazon Inspector](#Amazon-Inspector) 6 | - [Amazon Kinesis](#Amazon-Kinesis) 7 | 8 | # GuardDuty 9 | A threat detection service that continuously monitors for malicious or unauthorized behavior 10 | 11 | It does this by monitoring for: 12 | - Unusual API calls 13 | - Unauthorized deployments 14 | - Compromised instances 15 | 16 | It uses: 17 | - Threat intelligence feeds 18 | - Machine Learning 19 | - CloudWatch Events 20 | 21 | How it works: 22 | 1. Enable service 23 | 2. Analyze continuously 24 | 3. Detect Threats using machine learning 25 | 4. Act 26 | - Detailed analysis is available in console 27 | - Can integrate logging, alerting, or trigger lambda 28 | 29 | # Amazon Inspector 30 | Automated service that assess your applications for vulnerabilities and produces a security findings report. Mainly based around protecting EC2 instances. 31 | 32 | - Identify security issuesu 33 | - API driven 34 | - Reduces risk by warning you of risk before it's a problem 35 | - Leverage expertise: experts researching potential security issues 36 | - Define and Enforce standards 37 | 38 | How it works: 39 | - Network assessments (no agent needed) 40 | - network configuration check (see if ports are reachable from outside VPC) 41 | - Host assessment (Inspector Agent on EC2) 42 | - Can be automatically installed via Systems Manager run command 43 | - Can run weekly or once 44 | 45 | # Amazon Kinesis 46 | Collect, process, and analyze video and data streams in real time 47 | 48 | Made of 4 services 49 | - Kinesis Data Analytics 50 | - Kinesis Data Firehose 51 | - Kinesis Data Streams 52 | - Kinesis Video Streams 53 | 54 | ### Kinesis Data Analytics 55 | - Analyze streaming data 56 | - Respond in real time 57 | - Query data using SQL 58 | - completely managed service 59 | - pay as you go (for what you use) 60 | 61 | ### Kinesis Firehose 62 | - Deliver streaming data 63 | - No applications to manage 64 | - Just configure data producer to send data to Firehose 65 | - Firehose send data to: 66 | - S3 67 | - Redshift 68 | - Amazon ElasticSearch 69 | - Splunk 70 | - Accepts records in chunks up to 1000 kb 71 | - Data can be transformed 72 | 73 | ### Kinesis Data Streams 74 | - Collect streaming data 75 | - Massively scalable 76 | - Capture GBs per second 77 | - data is available in milliseconds 78 | - durable, streamed across 3 data centers 79 | - data stored for 7 days 80 | - Elastic, dynamically scale 81 | 82 | ### Kinesis Video Streams 83 | - Collect streaming video 84 | - Handle ingestion from millions of devices 85 | - Enables live and on-demand playback 86 | - Can take advantage of Amazon Rekognition and Machine Learning Frameworks 87 | - Access data through APIs 88 | - Build real-time, video-enabled applications 89 | 90 | 91 | ### Kinesis Overview 92 | 93 | - Kinesis Data Analytics: Analyze streaming data using SQL 94 | - Kinesis Firehose: Deliver streaming data to another AWS service, e.g. S3 95 | - Kinesis Data Streams: Collect streaming data and do things with it (e.g. create dashboard) 96 | - Kinesis Video Streams: Collect streaming video data and do things with it 97 | 98 | Exam hint: Kinesis will be an answer if it has to do with large amounts of data 99 | -------------------------------------------------------------------------------- /Misc.md: -------------------------------------------------------------------------------- 1 | 1. Know Cloudwatch like the back of your hand - logs, events, metrics, triggers. what services integrate with it. 2 | 3 | 2. CodePipeline, CodeBuild, CodeDeploy - know the lifecycle well. Testing, rollbacks. Would at least test out building a pipeline to deploy a lambda function, also an AMI. Just so you understand the process. 4 | 5 | 3. Different deployment types across all services - CodePipeline, EC2, EB, CFN, Lambda, API Gateway. Know how to do a canary, blue-green, and rolling in each. 6 | 7 | 4. Was a few questions about DynamoDB - one in particular where I had to design a sort/primary key and a LSI/GSI. Was only one question but still.. 8 | 9 | 5. AutoScaling/Launch Configs/ELBs. Know them like the back of your hand. Lifecycle hooks will come up for sure. 10 | 11 | 6. Amazon Macie, Quick Sight, Personal Health Dashboard came up a few times as well. Just be aware of they do. 12 | 13 | 7. RDS - DR scenarios, CFN scenarios, cross-region read replicas (which engines support it). 14 | 15 | 8. Elastic Beanstalk - understand ebextensions as well as the deployment types. 16 | 17 | 9. Understand container deployment lifecycle - ECS, ECR, and maybe EB. 18 | 19 | 10. Route 53 comes up a lot. Know how it plays into different scenarios.. DR, B/G deployment, Canary, etc. 20 | 21 | 22 | ALB's vs ELB's 23 | 24 | Git branching 25 | 26 | CloudFormation change sets 27 | 28 | Amazon EC2 dedicated hosts 29 | 30 | AWS Server Migration Service 31 | 32 | VPC flow logs 33 | 34 | Quicksight 35 | 36 | Triggering lambda to make route53 changes 37 | 38 | IPV6 (listed as an assumed knowledge topic) 39 | 40 | Classic loadbalancer 41 | 42 | Tags and cost reports 43 | 44 | Matthew's tips from https://acloud.guru/forums/aws-certified-devops-engineer-professional-2019/discussion/-LaO1SD8JROLZSxJ36U8/newvsolddevopspro_exam 45 | 46 | CloudFormation StackSets 47 | 48 | Beanstalk deployment strategies 49 | 50 | Cloudwatch events from things like CodeDeploy/CodePipeline and automation based on that 51 | 52 | Some things about working with multiple accounts (literally DR accounts of the type where the original account won't be accessible anymore and how to prepare for that) 53 | 54 | Hybrid environment CodeDeploy (how to enable that and how it shows up) 55 | 56 | Various multi-region/DR setup questions 57 | 58 | S3 permissions 59 | 60 | Lambda@Edge 61 | -------------------------------------------------------------------------------- /MonitoringLogging.md: -------------------------------------------------------------------------------- 1 | # Monitoring and Logging 2 | 3 | ## Table of Contents 4 | - [CloudWatch](#CloudWatch) 5 | - [CloudWatch Metrics](#CloudWatch-Metrics) 6 | - [CloudWatch Events](#CloudWatch-Events) 7 | - [CloudWatch Logs](#CloudWatch-Logs) 8 | - [X-ray](#X-ray) 9 | 10 | # CloudWatch 11 | - Metric gathering 12 | - Monitoring / Alerting 13 | - Graphing 14 | - Logging 15 | 16 | **Retention Period** (good to know for exam) 17 | - Data points < 60 seconds are available for 3 hours (high resolution) 18 | - Data points = 60 seconds are available for 15 days 19 | - Data points = 300 seconds (5 minutes) are available for 63 days 20 | - Data points = 3600 (1 hour) are available for 445 days (15 months) 21 | 22 | Note: shorter term data points are averaged to the next tier at the end of their lifecycle. e.g. a 1 second data point becomes a 1 minute average after 3 hours 23 | 24 | ### CloudWatch Metrics 25 | Concepts: 26 | - **Metrics**: A metric represents a time-ordered set of data points that are published to CloudWatch. 27 | - Metrics are uniquely defined by a name, a namespace, and zero or more dimensions. Each data point in a metric has a time stamp, and (optionally) a unit of measure. 28 | - Metrics exist only in the region in which they are created. Metrics cannot be deleted, but they automatically expire after 15 months if no new data is published to them. 29 | - **Namespace**: A namespace is a container for CloudWatch metrics. AWS namespaces use the following naming convention: AWS/service. 30 | - **Dimension**: A dimension is a name/value pair that is part of the identity of a metric. You can assign up to 10 dimensions to a metric. 31 | - For example, you can get statistics for a specific EC2 instance by specifying the InstanceId dimension when you search for metrics. 32 | 33 | ### Cloudwatch Events 34 | Concepts: 35 | - **Events**: An event indicates a change in your AWS environment. AWS resources can generate events when their state changes. 36 | - e.g. Amazon EC2 generates an event when the state of an EC2 instance changes from pending to running 37 | - **Targets**: A target processes events. 38 | - Targets can include: 39 | - Amazon EC2 instances 40 | - AWS Lambda functions 41 | - Kinesis streams 42 | - Amazon ECS tasks 43 | - Step Functions state machines 44 | - Amazon SNS topics 45 | - Amazon SQS queues 46 | - built-in targets 47 | - A target receives events in JSON format. 48 | - **Rules**: A rule matches incoming events and routes them to targets for processing. 49 | - A single rule can route to multiple targets, all of which are processed in parallel. 50 | - Rules are not processed in a particular order. 51 | - A rule can customize the JSON sent to the target, by passing only certain parts or by overwriting it with a constant. 52 | 53 | ### CloudWatch Logs 54 | Centralize the logs from all of your systems, applications, and AWS services that you use, in a single, highly scalable service. 55 | - view logs 56 | - search/query them for specific error codes or patterns 57 | - filter them based on specific fields 58 | - archive them securely for future analysis 59 | - visualize log data in dashboards 60 | 61 | #### Concepts: 62 | - **Log Events**: A log event is a record of some activity recorded by the application or resource being monitored. 63 | - contains two properties: the timestamp of when the event occurred, and the raw event message. Event messages must be UTF-8 encoded. 64 | - **Log Streams**: A log stream is a sequence of log events that share the same source. 65 | - intended to represent the sequence of events coming from the application instance or resource being monitored. 66 | - **Log Groups**: Log groups define groups of log streams that share the same retention, monitoring, and access control settings. 67 | - Each log stream has to belong to one log group. 68 | 69 | # X-Ray 70 | - Collects data about requests that your application serves 71 | - Provides tools you can use to view, filter, and gain insights into that data 72 | - Identify issues and opportunities for optimization 73 | -------------------------------------------------------------------------------- /Policies.md: -------------------------------------------------------------------------------- 1 | # Policies and Standards Automation 2 | 3 | ## Table of Contents 4 | - [Service Catalog](#Service-Catalog) 5 | - [Trusted Advisor](#Trusted-Advisor) 6 | - [Systems Manager](#Systems-Manager) 7 | - [Organizations](#Organizations) 8 | - [Secrets Manager](#Secrets-Manager) 9 | - [Macie](#Macie) 10 | - [Certificate Manager](#Certificate-Manager) 11 | 12 | # Service Catalog 13 | Allows IT administrators to create, manage, and distribute catalogs of approved products to end users, who can then access the products they need in a personalized portal 14 | - Has an API that provides programmatic contol over all end-user actions as an alternative to AWS Console 15 | - Allows you to create custom interfaces and apps 16 | - Allows administrators to create and distribute application stacks called **products** 17 | - Products can be grouped into folders called **portfolios** 18 | - Users can then launch and manage products without requiring access to AWS services/console 19 | - Users can only see products they are supposed to see 20 | 21 | # Trusted Advisor 22 | Service that provides real time guidance to ensure resources are provisioned and managed correctly, following AWS best practices 23 | 24 | Categories: 25 | - Cost optimization 26 | - Performance 27 | - e.g. under-utilizing resources 28 | - Security 29 | - Fault tolerance 30 | - Service Limits 31 | 32 | # Systems Manager 33 | - Collecting software inventory 34 | - Applying OS patches 35 | - Creating system images 36 | - Configuring operation systems 37 | - Manage hybrid cloud systems from a single interface (AWS, on-prem) 38 | - Reducing costs 39 | 40 | Features 41 | - Run command 42 | - remote management of servers 43 | - State manager 44 | - manage all your servers' configuration 45 | - e.g. firewall settings 46 | - e.g. anti-virus definitions 47 | - Inventory 48 | - info on installed applications 49 | - Maintentance Window 50 | - Patch Manager 51 | - e.g. select and deploy software across all EC2 instances 52 | - Automation 53 | - automate repetitive IT tasks 54 | - Parameter Store 55 | 56 | 57 | # Organizations 58 | Policy-based management for multiple AWS accounts. 59 | 60 | Accounts can be split several ways within business: Environment (dev, qa, prod), Projects, or Business Units (Sales, support, dev, etc) 61 | 62 | AWS Organizations allows you to: 63 | - Programmatically create new accounts 64 | - Create and maintain a group of accounts 65 | - Set policies on those groups 66 | - Set single payer, cost tracking 67 | 68 | Overrides IAM policies with an organization policy 69 | 70 | 71 | # Secrets Manager 72 | Service to help protect secrets needed to access applications, service, and IT resources 73 | 74 | - Encrypts secrets at rest using your own encrpytion keys stored in KMS 75 | - Secrets can be database credentials, passwords, 3rd part API keys, any text, etc. 76 | - Store and control access to them with Secrets Manager Console/CLI/API/SDK 77 | - Hardcoded credentials in code is replaced with API call 78 | - Secrets can be rotated automatically according to your own schedule 79 | 80 | # Macie 81 | Security service that uses machine learning to automatically discover, classify, and protect sensitive data in AWS 82 | 83 | Managed service: 84 | - Can recognize any Personally Identifiable Information (PII) 85 | - Provides dashboard showing how data is accessed/moved 86 | - Monitors data access for anomalies 87 | - Generates detailed alerts when it detects risk of unauthorized access or accidental data leaks 88 | - Currently only protects S3, but more is planned 89 | 90 | # Certificate Manager 91 | Easily provision, manage, and deploy SSL/TLS certificates 92 | 93 | - Centrally manage certificates in AWS 94 | - Audit the use of each certificate in CloudTrail logs 95 | - Private Certificate Authority 96 | - AWS integration 97 | - Import 3rd party certificates from other CAs 98 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # DevOps Professional Study Workbook 2 | 3 | ## Topics: 4 | ### 1. [SDLC](/SDLC.md) 5 | - #### [Deployment Types](/SDLC.md#Deployment-Types) 6 | - #### [CodeCommit](/SDLC.md#CodeCommit) 7 | - #### [CodeBuild](/SDLC.md#CodeBuild) 8 | - #### [CodeDeploy](/SDLC.md#CodeDeploy) 9 | ### 2. [Configuration Management and Infrastructure as Code](/ConfigInfrastructure.md) 10 | - #### [Cloudformation](/ConfigInfrastructure.md#Cloudformation) 11 | - #### [Elastic Beanstalk](/ConfigInfrastructure.md#Elastic-Beanstalk) 12 | - #### [AWS Config](/ConfigInfrastructure.md#AWS-config) 13 | - #### [ECS](/ConfigInfrastructure.md#ECS) 14 | - #### [Lambda Step Functions](/ConfigInfrastructure.md#Lambda-Step-Functions) 15 | - #### [OpsWorks](/ConfigInfrastructure.md#OpsWorks) 16 | 17 | ### 3. [Monitoring and Logging](/MonitoringLogging.md) 18 | - #### [CloudWatch](/MonitoringLogging.md#CloudWatch) 19 | - #### [X-Ray](/MonitoringLogging.md#X-Ray) 20 | 21 | ### 4. [Policies and Standards Automation](/Policies.md) 22 | - #### [Service Catalog](/Policies.md#Service-Catalog) 23 | - #### [Trusted Advisor](/Policies.md#Trusted-Advisor) 24 | - #### [Systems Manager](/Policies.md#Systems-Manager) 25 | - #### [Organizations](/Policies.md#Organizations) 26 | - #### [Secrets Manager](/Policies.md#Secrets-Manager) 27 | - #### [Macie](/Policies.md#Macie) 28 | - #### [Certificate Manager](/Policies.md#Certificate-Manager) 29 | 30 | ### 5. [Incident and Event Response](Incident.md) 31 | - #### [GuardDuty](/Incident.md#GuardDuty) 32 | - #### [Amazon Inspector](/Incident.md#Amazon-Inspector) 33 | - #### [Amazon Kinesis](/Incident.md#Amazon-Kinesis) 34 | 35 | ### 6. [High Availability, Fault Tolerance, & Disaster Recovery](/HighAvail.md) 36 | - #### [Single Sign-On](/HighAvail.md#Single-Sign-On) 37 | - #### [CloudFront](/HighAvail.md#CloudFront) 38 | - #### [AutoScaling](/HighAvail.md#AutoScaling) 39 | - #### [Route 53](/HighAvail.md#Route-53) 40 | -------------------------------------------------------------------------------- /SDLC.md: -------------------------------------------------------------------------------- 1 | # SDLC 2 | ## Table of Contents 3 | - [Deployment Types](#Deployment-Types) 4 | - [Single Target](#Single-Target) 5 | - [All-at-once](#All-at-once) 6 | - [Minimum In-service](#Minimum-In-service) 7 | - [Rolling Deployment](#Rolling-Deployment) 8 | - [Blue Green Deployment](#Blue-Green-Deployment) 9 | - [CodeCommit](#CodeCommit) 10 | - [Creating Web Hooks with CodeCommit](#Creating-Web-Hooks-with-CodeCommit) 11 | - [CodeBuild](#CodeBuild) 12 | - [How it Works](#How-it-Works) 13 | - [Steps](#Steps) 14 | - [Sources](#Sources) 15 | - [Programming Frameworks](#Programming-Frameworks) 16 | - [Jenkins Integration](#Jenkins-Integration) 17 | - [Security](#Security) 18 | - [Buildspec.yml](#Buildspec.yml) 19 | - [Build Environment Compute Types](#Build-Environment-Compute-Types) 20 | - [Docker Images](#Docker-Images-Provided-by-CodeBuild) 21 | - [CodeDeploy](#CodeDeploy) 22 | - [Supported Platforms/Deployment Types](#Supported-Platforms/Deployment-Types) 23 | - [CodeDeploy Components](#CodeDeploy-Components-per-Compute-Platform) 24 | - [App Spec File](#App-Spec-File) 25 | - [ECS](#AppSpec-Files-on-ECS) 26 | - [Lambda](#AppSpec-Files-on-Lambda) 27 | - [EC2/On-Premises](#AppSpec-Files-on-EC2/On-Premises) 28 | [](#) 29 | 30 | # Deployment Types: 31 | ### Single Target 32 | - New app version is installed on target server 33 | - Outage occurs during installation 34 | - No secondary servers, testing is limited 35 | - Rollback requires removing new version and installing previous version 36 | ### All-at-once 37 | - Deployment happens in one step 38 | - Destination is multiple targets 39 | - Requires orchestration tooling 40 | - Shares negatives of Single Target 41 | - No ability to test 42 | - Outage 43 | - Less than ideal rollback 44 | ### Minimum In-service 45 | - Happens in multiple stages 46 | - Deployment happens to as many targets as possible while maintaining minimum in-servive targets 47 | - Allows automated testing 48 | - Deployment targets are assessed and tested prior to continuing 49 | - No downtime 50 | ### Rolling Deployment 51 | - Deployment happens in multiple stages, number of targets per stage is user-defined 52 | - Moving parts, orchestration, health checks required 53 | - Overall applicable health is not necessarily maintained 54 | - Can be least effecient deployment type based on time-taken 55 | - Allows automated testing 56 | - Deployment targets are assessed and tested prior to continuing 57 | - Generally no downtime, assuming number of targets will not impact application 58 | - Can be paused, allowing limited multi-version testing 59 | ### Blue Green Deployment 60 | - Requires advanced orchestration tooling 61 | - Costs more: maintain two environments at once 62 | - Deployment is rapid, entire environment is deployed at once 63 | - Cutover and migration is clean and controlled (DNS change) 64 | - Rollback is equally clean (DNS regression) 65 | - Health and performance of entire green environment can be tested prior to cutover 66 | - Using advanced template systems (CloudFormation), entire process can be automated 67 | ### Canary Deployment 68 | - Similar to Blue Green, but blue stays incrementally active 69 | - Can be done with Route53 - weighted Round Robin 70 | - Easily allows for a/b testing 71 | 72 | # CodeCommit 73 | Version control service to privately store and manage assets (such as documents, source code, and binary files) in the cloud 74 | - Fully Managed –AWS CodeCommit eliminates the need to host, maintain, backup, and scale your own source control servers. 75 | - Secure –AWS CodeCommit automatically encrypts your files in transit and at rest. AWS CodeCommit is integrated with AWS Identity and Access Management (IAM), allowing you to assign user-specific permissions to your repositories. 76 | - Highly Available – AWS CodeCommit is built on highly scalable, redundant, and durable AWS services such as Amazon S3 and Amazon DynamoDB. 77 | - Scalable - AWS CodeCommit allows you store any number of files and there are no repository size limits. 78 | - Faster Development Lifecycle - AWS CodeCommit keeps your repositories close to your build, staging, and production environments in the AWS cloud. This allows you to increase the speed and frequency of your development lifecycle. 79 | ### Creating Web Hooks with CodeCommit 80 | In the Amazon Simple Notification Service (SNS) console, you can create a SNS topic with an HTTP endpoint and the desired URL for the webhook. From the AWS CodeCommit console, you can then configure that SNS topic to a repository event using triggers. 81 | 82 | # CodeBuild 83 | Fully managed continuous integration service in the cloud. CodeBuild compiles source code, runs tests, and produces packages that are ready to deploy. 84 | - Eliminates the need to provision, manage, and scale your own build servers 85 | - Automatically scales up and down and processes multiple builds concurrently, so builds don’t have to wait in a queue. 86 | - Use prepackaged build environments or custom build environments to use your own build tools 87 | - Pay by the minute 88 | ### How it Works 89 | As input, you must provide CodeBuild with a build project. A build project defines how CodeBuild runs a build. It includes information such as where to get the source code, the build environment to use, the build commands to run, and where to store the build output. A build environment represents a combination of operating system, programming language runtime, and tools that CodeBuild uses to run a build. 90 | 91 | #### Steps: 92 | 93 | 1. CodeBuild uses the build project to create the build environment. 94 | 95 | 2. CodeBuild downloads the source code into the build environment and then uses the build specification (build spec), as defined in the build project or included directly in the source code. A build spec is a collection of build commands and related settings, in YAML format, that CodeBuild uses to run a build. 96 | 97 | 3. If there is any build output, the build environment uploads its output to an Amazon S3 bucket. The build environment can also perform tasks that you specify in the build spec (for example, sending build notifications to an Amazon SNS topic). 98 | 99 | 4. While the build is running, the build environment sends information to CodeBuild and Amazon CloudWatch Logs. 100 | 101 | 5. While the build is running, you can use the CodeBuild console, AWS CLI, or AWS SDKs, to get summarized build information from CodeBuild and detailed build information from Amazon CloudWatch Logs. If you use AWS CodePipeline to run builds, you can get limited build information from CodePipeline. 102 | ### Sources 103 | CodeBuild can connect to AWS CodeCommit, S3, GitHub, and GitHub Enterprise and Bitbucket to pull source code 104 | ### Programming Frameworks 105 | Java, Ruby, Python, Go, Node.js, Android, .NET Core, PHP, and Docker. Customize an environment by creating a Docker image and uploading it to the Amazon EC2 Container Registry or the Docker Hub registry 106 | ### Jenkins Integration 107 | CodeBuild Plugin for Jenkins can be used to integrate CodeBuild into Jenkins jobs 108 | ### Security 109 | - Specify a key stored in the AWS Key Management Service (AWS KMS) to encrypt your artifacts 110 | - Runs build in fresh environments isolated from other users and discards each build environment upon completion. CodeBuild provides security and separation at the infrastructure and execution levels. 111 | ### Buildspec.yml 112 | A build spec is a collection of build commands and related settings, in YAML format, that CodeBuild uses to run a build. You can include a build spec as part of the source code or you can define a build spec when you create a build project. 113 | 114 | By default, the build spec file must be named buildspec.yml and placed in the root of your source directory. You can override the default build spec file name and location 115 | 116 | The build spec has the following syntax: 117 | ``` 118 | version: 0.2 119 | 120 | run-as: Linux-user-name 121 | 122 | env: 123 | variables: 124 | key: "value" 125 | key: "value" 126 | parameter-store: 127 | key: "value" 128 | key: "value" 129 | 130 | phases: 131 | install: 132 | run-as: Linux-user-name 133 | commands: 134 | - command 135 | - command 136 | finally: 137 | - command 138 | - command 139 | pre_build: 140 | run-as: Linux-user-name 141 | commands: 142 | - command 143 | - command 144 | finally: 145 | - command 146 | - command 147 | build: 148 | run-as: Linux-user-name 149 | commands: 150 | - command 151 | - command 152 | finally: 153 | - command 154 | - command 155 | post_build: 156 | run-as: Linux-user-name 157 | commands: 158 | - command 159 | - command 160 | finally: 161 | - command 162 | - command 163 | artifacts: 164 | files: 165 | - location 166 | - location 167 | name: artifact-name 168 | discard-paths: yes 169 | base-directory: location 170 | secondary-artifacts: 171 | artifactIdentifier: 172 | files: 173 | - location 174 | - location 175 | name: secondary-artifact-name 176 | discard-paths: yes 177 | base-directory: location 178 | artifactIdentifier: 179 | files: 180 | - location 181 | - location 182 | discard-paths: yes 183 | base-directory: location 184 | cache: 185 | paths: 186 | - path 187 | - path 188 | ``` 189 | ### Build Environment Compute Types 190 | Compute type | computeType | value Memory | vCPUs | Disk space | Operating system 191 | | --- | --- | --- | --- | --- | --- | 192 | build.general1.small | BUILD_GENERAL1_SMALL | 3 GB | 2 | 64 GB | Linux | 193 | build.general1.medium | BUILD_GENERAL1_MEDIUM | 7 GB | 4 |128 GB |Linux, Windows 194 | build.general1.large |BUILD_GENERAL1_LARGE |15 GB |8 |128 GB |Linux, Windows 195 | ### Docker Images Provided by CodeBuild 196 | |Platform |Programming language or framework |Image identifier| Definition| 197 | ---|---|---|---| 198 | Ubuntu 18.04 |(Standard image)|aws/codebuild/standard:1.0 |ubuntu/standard/1.0| 199 | Windows Server Core 2016 |(Base Image) |aws/codebuild|windows-base:1.0| N/A| 200 | Standard image of the Ubuntu 18.04 platform contains the following programming languages: 201 | - Ruby 2.x 202 | - Python 3.x 203 | - PHP 7.x 204 | - Node 10.x 205 | - Java 8 206 | - Golang 1.x 207 | - .NET Core 2.x 208 | - Docker 18.x 209 | - Android 28.x 210 | 211 | # CodeDeploy 212 | Service that automates code deployments to any instance, including Amazon EC2 instances and instances running on-premises. 213 | 214 | 215 | ### Supported Platforms/Deployment Types: 216 | 217 | - EC2/On-Premises: In-Place or Blue/Green Deployments 218 | - Describes instances of physical servers that can be Amazon EC2 cloud instances, on-premises servers, or both. Applications created using the EC2/On-Premises compute platform can be composed of executable files, configuration files, images, and more. 219 | - Deployments that use the EC2/On-Premises compute platform manage the way in which traffic is directed to instances by using an in-place or blue/green deployment type. 220 | 221 | - AWS Lambda: Canary, Linear, All-At-Once Deployments 222 | - Applications created using the AWS Lambda compute platform can manage the way in which traffic is directed to the updated Lambda function versions during a deployment by choosing a canary, linear, or all-at-once configuration. 223 | 224 | - Amazon ECS: Blue/Green Deployment 225 | - Used to deploy an Amazon ECS containerized application as a task set. 226 | - CodeDeploy performs a blue/green deployment by installing an updated version of the containerized application as a new replacement task set. CodeDeploy reroutes production traffic from the original application, or task set, to the replacement task set. The original task set is terminated after a successful deployment. 227 | 228 | ### CodeDeploy Components per Compute Platform 229 | |CodeDeploy Component |EC2/On-Premises |AWS Lambda |Amazon ECS| 230 | ---|---|---|---| 231 | Deployment group |Deploys a revision to a set of instances. | Deploys a new version of a serverless Lambda function on a high-availability compute infrastructure. |Specifies the Amazon ECS service with the containerized application to deploy as a task set, a production and optional test listener used to serve traffic to the deployed application, when to reroute traffic and terminate the deployed application's original task set, and optional trigger, alarm, and rollback settings.| 232 | Deployment| Deploys a new revision that consists of an application and AppSpec file. The AppSpec specifies how to deploy the application to the instances in a deployment group. |Shifts production traffic from one version of a Lambda function to a new version of the same function. The AppSpec file specifies which Lambda function version to deploy. |Deploys an updated version of an Amazon ECS containerized application as a new, replacement task set. CodeDeploy reroutes production traffic from the task set with the original version to the new replacement task set with the updated version. When the deployment completes, the original task set is terminated.| 233 | Deployment configuration |Settings that determine the deployment speed and the minimum number of instances that must be healthy at any point during a deployment. |Settings that determine how traffic is shifted to the updated Lambda function versions. |Traffic is always shifted all at once. Custom deployment configuration settings cannot be specified for an Amazon ECS deployment.| 234 | Revision |A combination of an AppSpec file and application files, such as executables, configuration files, and so on. |An AppSpec file that specifies which Lambda function to deploy and Lambda fuctions that can run validation tests during deployment lifecycle event hooks. | An AppSpec file that specifies:| 235 | Application| A collection of deployment groups and revisions. An EC2/On-Premises application uses the EC2/On-Premises compute platform. |A collection of deployment groups and revisions. An application used for an AWS Lambda deployment uses the Amazon ECS compute platform. |A collection of deployment groups and revisions. An application used for an Amazon ECS deployment uses the Amazon ECS compute platform. 236 | 237 | ### App Spec File 238 | The application specification file (AppSpec file) is a YAML-formatted or JSON-formatted file used by CodeDeploy to manage a deployment. 239 | 240 | Note: the name of the AppSpec file for an EC2/On-Premises deployment must be appspec.yml. The name of the AppSpec file for an Amazon ECS or AWS Lambda deployment must be appspec.yaml. 241 | 242 | #### AppSpec Files on ECS 243 | Determine: 244 | - Amazon ECS task definition file. This is specified with its ARN in the TaskDefinition instruction in the AppSpec file. 245 | - The container and port in replacement task set where your Application Load Balancer or Network Load Balancer reroutes traffic during a deployment. This is specified with the LoadBalancerInfo instruction in the AppSpec file. 246 | - Optional information about your Amazon ECS service, such the platform version on which it runs, its subnets, and its security groups. 247 | - Optional Lambda functions to run during hooks that correspond with lifecycle events during an Amazon ECS deployment. For more information, see AppSpec 'hooks' Section for an Amazon ECS Deployment. 248 | 249 | Example appspec.yaml (ECS) 250 | ``` 251 | version: 0.0 252 | Resources: 253 | - TargetService: 254 | Type: AWS::ECS::Service 255 | Properties: 256 | TaskDefinition: "arn:aws:ecs:us-east-1:111222333444:task-definition/my-task-definition-family-name:1" 257 | LoadBalancerInfo: 258 | ContainerName: "SampleApplicationName" 259 | ContainerPort: 80 260 | # Optional properties 261 | PlatformVersion: "LATEST" 262 | NetworkConfiguration: 263 | AwsvpcConfiguration: 264 | Subnets: ["subnet-1234abcd","subnet-5678abcd"] 265 | SecurityGroups: ["sg-12345678"] 266 | AssignPublicIp: "ENABLED" 267 | Hooks: 268 | - BeforeInstall: "LambdaFunctionToValidateBeforeInstall" 269 | - AfterInstall: "LambdaFunctionToValidateAfterTraffic" 270 | - AfterAllowTestTraffic: "LambdaFunctionToValidateAfterTestTrafficStarts" 271 | - BeforeAllowTraffic: "LambdaFunctionToValidateBeforeAllowingProductionTraffic" 272 | - AfterAllowTraffic: "LambdaFunctionToValidateAfterAllowingProductionTraffic" 273 | ``` 274 | 275 | #### AppSpec Files on Lambda 276 | Determine: 277 | 278 | - Which Lambda function version to deploy. 279 | - Which Lambda functions to use as validation tests. 280 | 281 | An AppSpec file can be YAML-formatted or JSON-formatted. You can also enter the contents of an AppSpec file directly into CodeDeploy console when you create a deployment. 282 | 283 | Example appspec.yaml (Lambda) 284 | ``` 285 | version: 0.0 286 | Resources: 287 | - myLambdaFunction: 288 | Type: AWS::Lambda::Function 289 | Properties: 290 | Name: "myLambdaFunction" 291 | Alias: "myLambdaFunctionAlias" 292 | CurrentVersion: "1" 293 | TargetVersion: "2" 294 | Hooks: 295 | - BeforeAllowTraffic: "LambdaFunctionToValidateBeforeTrafficShift" 296 | - AfterAllowTraffic: "LambdaFunctionToValidateAfterTrafficShift" 297 | ``` 298 | 299 | #### AppSpec Files on EC2/On-Premises 300 | Determine: 301 | - What it should install onto your instances from your application revision in Amazon S3 or GitHub. 302 | - Which lifecycle event hooks to run in response to deployment lifecycle events. 303 | 304 | Note: An AppSpec file must be a YAML-formatted file named appspec.yml and it must be placed in the root of the directory structure of an application's source code. Otherwise, deployments fail. 305 | 306 | Steps: 307 | - Complete AppSpec file, bundle it, along with the content to deploy, into an archive file (zip, tar, or compressed tar) 308 | - After you have a bundled archive file (known in CodeDeploy as a revision), you upload it to an Amazon S3 bucket or Git repository. 309 | - Use CodeDeploy to deploy the revision. 310 | - The appspec.yml for an EC2/On-Premises compute platform deployment is saved in the root directory of your revision. 311 | 312 | Example appspec.yml ( EC2/On-Premises) 313 | ``` 314 | version: 0.0 315 | os: linux 316 | files: 317 | - source: Config/config.txt 318 | destination: /webapps/Config 319 | - source: source 320 | destination: /webapps/myApp 321 | hooks: 322 | BeforeInstall: 323 | - location: Scripts/UnzipResourceBundle.sh 324 | - location: Scripts/UnzipDataBundle.sh 325 | AfterInstall: 326 | - location: Scripts/RunResourceTests.sh 327 | timeout: 180 328 | ApplicationStart: 329 | - location: Scripts/RunFunctionalTests.sh 330 | timeout: 3600 331 | ValidateService: 332 | - location: Scripts/MonitorService.sh 333 | timeout: 3600 334 | runas: codedeployuser 335 | ``` 336 | --------------------------------------------------------------------------------