├── API Gateway └── apigateway.md ├── Cloudformation └── cloudformation.md ├── Cognito └── cognito.md ├── Compute & Networking └── compute&networking.md ├── Databases └── databases.md ├── Developer Tools └── developertools.md ├── DynamoDB └── dynamodb.md ├── ECS-Fargate-ECR └── container-registry.md ├── Elastic BeanStalk └── elasticbeanstalk.md ├── IAM └── iam.md ├── KMS └── KMS.md ├── Lambda └── Lambda.md ├── README.md ├── SNS └── sns.md ├── SQS └── sqs.md └── Storage └── storage.md /API Gateway/apigateway.md: -------------------------------------------------------------------------------- 1 | # API Gateway (1) 2 | ![Untitled](https://user-images.githubusercontent.com/53600644/194395907-22070317-cc87-415c-8435-90147b5b4e4e.png) 3 | 4 | 5 | 6 | ## Features 7 | 8 | - Handle API version 9 | - Handle different environments (dev, test, prod...) 10 | - Handle security (Authentication vs Authorization) 11 | - Create API keys, handle request throttling 12 | - Swagger / Open API import to quickly define APIs 13 | - Transform and validate requests and responses 14 | - Generate SDK and API specifications 15 | - Cache API responses 16 | 17 | ## Deployment 18 | 19 | - Making changes in the API Gateway does not mean they're effective 20 | 21 | ⇒ need to make a "deployment" 22 | 23 | - Changes are deployed to "Stages", can use the naming you like for stages (dev, test, prod) 24 | 25 | - Each stage has its own configuration parameters 26 | 27 | - Stage can be rolled back as a history of deployments is kept 28 | 29 | - Stage variables are like environment variables for API Gateway 30 | 31 | 32 | ### Stage Variables & Lambda Aliases 33 | 34 | - create stage variables to indicate the corresponding Lambda alias 35 | 36 | ### Canary Deployment 37 | 38 | 39 | - can enable deployments for any stage 40 | 41 | - choose the % off traffic the canary channel receives 42 | 43 | ⇒ blue / green deployment with Lambda & Gateway 44 | 45 | - metrics & logs are separate ( for better monitoring) 46 | 47 | - can override stage variables for canary 48 | 49 | 50 | 51 | ## Mapping Templates 52 | 53 | - use to modify request / responses 54 | - rename parameters 55 | - modify body content 56 | - add headers 57 | - map JSON to XML 58 | - use Velocity Template Language (VTL) 59 | 60 | ## Swagger / Open API Integrate 61 | 62 | - API deifinition as code 63 | - Import / export existing Swagger / Open API 3.0 spec to API Gateway 64 | - Method 65 | - Method Request 66 | - Integration Request 67 | - Method Response 68 | - AWS extensions for API gateway 69 | - written in YAML, JSON 70 | - can generate SDK for our apps 71 | 72 | ## Caching API responses 73 | 74 | - reduce the number of calls made to the backend 75 | - default time TTL is 300 second (5 minutes) (0s - 3600s) 76 | - define per stage 77 | - can encrypt 78 | - cache capacity 0.5GB - 237GB 79 | - can override cache settings for specific methods 80 | - can flush the entire cache 81 | - client can bypass (invalidate) the cache with header: `Cache-Controll: max-age=0` (with proper IAM authorization) 82 | 83 | ## CORS 84 | 85 | - The OPTIONS pre-flight request must contain the following headers: 86 | - `Access-Control-Allow-Methods` 87 | - `Access-Control-Allow-Headers` 88 | - `Access-Control-Allow-Origin` 89 | - CORS can be enabled through the console 90 | 91 | ## Usage plans & API Keys 92 | 93 | - Usage plans: 94 | - Throttling: set overall capacity and burst capacity 95 | - Quotas: number of requests made per day / week / month 96 | - Associate with desired API Stages 97 | - API Keys: 98 | - Generate one per customer 99 | - Associate with usage plans 100 | - can track usage for API Keys 101 | 102 | ## Authentication + Authorization 103 | 104 | ### IAM 105 | 106 | - good for users / roles already within your AWS account 107 | - handle authentication + authorization 108 | - leverages Sig v4 109 | 110 | ![Untitled 2](https://user-images.githubusercontent.com/53600644/194395957-dbcbae47-0fed-44a4-9c50-069afa325a61.png) 111 | 112 | ### Lambda Authorizer (Custom Authorizer) 113 | 114 | - good for 3rd party tokens (FB, GG...) 115 | - flexible in terms of what IAM policy is returned 116 | - handle authentication + authorization 117 | - pay per lambda invocation 118 | ![Untitled 3](https://user-images.githubusercontent.com/53600644/194396085-323402e5-6023-4685-8f43-5f86465f1f59.png) 119 | 120 | 121 | ### Cognito User Pool 122 | 123 | - manage your own user pool (can be backed by FB, GG...) 124 | - no need to write any custom code 125 | - must implement authorization in the backend (use Federated Identity) 126 | -------------------------------------------------------------------------------- /Cloudformation/cloudformation.md: -------------------------------------------------------------------------------- 1 | # Infrastructure As Code (CloudFormation) 2 | 3 | ## Cloud formation Building Blocks 4 | 5 | ### Stacks 6 | 7 | A stack represents a collection of resources to deploy and manage by AWS Cloudformation when you submit a template the resources you configure are provisioned and then make up the stack itself. 8 | 9 | ### Change Sets 10 | 11 | A change set is a description of the changes that will occur to a stack, should you submit the template. if the changes are acceptable, the change set itself can execute on the stack and implement the proposed modifications. This is especially important in situations where there is a potential for data loss. 12 | 13 | ## Template Structure 14 | 15 | AWS Cloudformation uses specific template syntax in JSON or YAML. 16 | 17 | ```json 18 | { 19 | "AWSTemplateFormatVersion": "2010-09-09", 20 | "Description": "String Description", 21 | "Metadata": { }, 22 | "Parameters": { }, 23 | "Mappings": { }, 24 | "Conditions": { }, 25 | "Transform": { }, 26 | "Resources": { }, 27 | "Outputs": { } 28 | } 29 | ``` 30 | 31 | ### AWSTemplateFormationVersion 32 | 33 | AWSTemplateFormatVersion corresponds to the template version to which this template adheres. Do not confuse this with an API version or the version of the developer template draft. Currently AWS Cloudformation only supports the value “2010-09-09” which you must provide as a literal string. 34 | 35 | ### Description 36 | 37 | The Description section allow you to provide a text explanation of the template purpose or other arbitrary information. The maximum length of the description field is 1024 byte. 38 | 39 | ### MetaData 40 | 41 | The Metadata section of a template allow you to provide structured detail about the template. 42 | 43 | ### Parameters 44 | 45 | You can use Parameteres to provide inputs into your template, which allow for more flexibility in how this template behaves when you deploy it. paramaters values can be set when you create the stack or perform updates. 46 | 47 | ### Mappings 48 | 49 | You can use Mapping section of a template to create a rudimentary lookup tables that you can reference in other sections of your template when you create the stack. for example, mapping usuage is to look up Amazon Ec2 instance AMI ID based on the region and architecture type. 50 | 51 | ### Conditions 52 | 53 | You can use conditions in AWS Cloudformation template to determine when to create a resource or when a property of a resource is defined. Conditional statements make use of intrinsic functions to evaluate multiple inputs against one other. 54 | 55 | ### Transforms 56 | 57 | A templates grow in size and complexity , there may be situaution where you use certian components repeatedly across multiple templates, such as common resources or mappings. Transforms allow you to simplify the template authoring process through a prowerful set of macros you use to reduce the amount of time spent in the authoring process. 58 | 59 | **AWS::Serverless Transform** 60 | 61 | you can use this tranform to convert SAM template to valid AWS Cloudformation template for deployment. 62 | 63 | ## Instrinsic Functions 64 | 65 | ### FN:Base64 66 | 67 | This will convert an input string into Base64 . 68 | 69 | ### FN::CIDR 70 | 71 | This will convert an IP address block , subnet count and size mask into valid CIDR notation. 72 | 73 | ### FN::FindInMap 74 | 75 | This will query information stored within the mapping table. 76 | 77 | ### FN::GetAtt 78 | 79 | With GetAtt you can query in other part of the same template. 80 | 81 | ### FN::Join 82 | 83 | AWS Cloudformation support string concatenation with Fn::Join intrinsic function. 84 | 85 | ### Fn::Select 86 | 87 | if you pass a list of values in your template, there need to be a way to select an item from the list based on the index it is in the list. Fn::Select allow you to choose an item in a list based on the zero-based index. 88 | 89 | ### Fn::Split 90 | 91 | FN::Split to create a list of strings by seperating a single string by known delimiter. 92 | 93 | ### Outputs 94 | 95 | Outputs are values that can be made available to use outside a single stack. 96 | 97 | ## Templates components 98 | 99 | - Resources: AWS resources declared in the template (can use `!Ref` function to refer to others resources in the same YAML file) 100 | - Parameters: the dynamic inputs (use with `!Ref` function) 101 | - Mappings: the static variables (use with `!FindInMap[ MapName, TopLevelKey, SecondLevelKey ]` function) 102 | - Outputs: export to be used by other stacks (other stacks use `!ImportValue` function to reference). Exported output names must be unique within region 103 | - Conditionals: List of conditions to perform resource creation (`!If`, `!Not`, `!Equal`...) 104 | - Metadata 105 | 106 | ### Templates helpers: 107 | 108 | - References 109 | - Functions 110 | 111 | ## Rollbacks 112 | 113 | - Stack Creation Fails: 114 | - Default: everything rolls back (gets deleted) 115 | - Option to disable rollback and troubleshoot what happened 116 | - Stack Update Fails: 117 | - The stack automatically rolls back to the previous known working state 118 | - Ability to see in the log what happened and error messages 119 | 120 | ## Tips: 121 | 122 | - Can I create a dynamic amount of resources? 123 | 124 | - No, you can't. Everything in CloudFormation templates has to be declared. Can't perform code generation here 125 | - Is every AWS Service supported in CloudFormation's resources? 126 | 127 | - Almost. Only a select few niches are not there yet 128 | - You can work around that using AWS Lambda Custom Resources 129 | - Exported output names must be unique within region 130 | 131 | - `!GetAtt` 132 | 133 | ![Untitled](https://user-images.githubusercontent.com/53600644/194138304-0dc664e8-bcfc-456c-9722-7ffe3daaa740.png) 134 | 135 | 136 | - `!Join` 137 | 138 | - want to create "a:b:c" ⇒ `!Join [ ":", [a, b, c] ]` 139 | - `!Sub` 140 | 141 | - with a Mapping 142 | 143 | ```yaml 144 | Name: !Sub 145 | - www.${Domain} 146 | - { Domain: !Ref RootDomainName } 147 | ``` 148 | 149 | - without a Mapping 150 | 151 | ```yaml 152 | !Sub 'arn:aws:ec2:${AWS::Region}:${AWS::AccountId}:vpc/${vpc}' 153 | ``` 154 | -------------------------------------------------------------------------------- /Cognito/cognito.md: -------------------------------------------------------------------------------- 1 | # Cognito 2 | 3 | ## Cognito User Pools (Authentication) 4 | 5 | - For Simple Login function (username/email + password combination) 6 | - create a server less database of user 7 | - can verify emails / phone numbers and add MFA 8 | - can enable Federated Identities (FB, GG, SAML...) 9 | - send back a JWT 10 | - can integrate with API Gateway for authen 11 | 12 | ## Federated Identity Pools (Authorization) 13 | 14 | - provide direct access to AWS Resources from the Client Side 15 | - login to federated identity provider - or remain anonymous 16 | - get temporary AWS credentials back from the Federated Identity Pool - come with a pre-defined IAM policy 17 | 18 | ## Cognito Sync 19 | 20 | - deprecated - use AWS AppSync now 21 | - store preferences, configuration, state of app 22 | - cross device synchronization (any platform - iOS, Android, etc...) 23 | - offline capability (synchronization when back online) 24 | - Require Federated Identity Pool in Cognito (not User Pool) 25 | - Store data in datasets (up to 1MB) 26 | - Up to 20 datasets to synchronise -------------------------------------------------------------------------------- /Compute & Networking/compute&networking.md: -------------------------------------------------------------------------------- 1 | # Compute and Networking 2 | 3 | ## Amazon EC2 4 | 5 | ### Instances Types 6 | 7 | Amazon Ec2 Instances family types are following 8 | 9 | 1. **General Purpose** - A balanced mix of CPU, RAM, and other resources 10 | 2. **Compute Optimized**- A high amount of CPU, such as high-performance web services 11 | 3. **Memory Optimized**- A large amount of RAM, such as in the memory database 12 | 4. **Storage Optimized**- A large amount of storage and Input/Output throughput. 13 | 5. **Accelerated Computing** - use for 3D rendering, and real-time video processing. 14 | 15 | ## EC2 Instance Launch Types 16 | 17 | - **On-Demand:** short workload, predictable pricing 18 | - **Reserved Instances:** long workloads (≥ 1 year) 19 | - **Convertible Reserved Instances**: long workloads with flexible instances 20 | - **Scheduled Reserved Instances:** launch within time window you reserve 21 | - **Spot Instances: short workloads,** for cheap, can lose instances 22 | - **Dedicated Instances**: no other customers will share your hardware 23 | - **Dedicated Hosts:** book an entire physical server, control instance placement 24 | 25 | ### Storage 26 | 27 | Your instance requires storage requires for both the root volume and any additional storage volume that you want to configure. You can create persistent storage volume with the Amazon Elastic Block Store (EBS) service to provide block storage also instance store which is temporary for instance. 28 | 29 | ### Network Interface 30 | 31 | A virtual network interface called elastic network interface provides networking for your Amazon Ec2 Instances. Elastic network interfaces are associated with a software-defined network provided by Amazon VPC. Each Amazon Ec2 instance is assigned a primary network interface that is associate with a subnet within a AWS VPC , by default if you omit the network configurations Amazon EC2 assign the instance to one of the subnet within default VPC 32 | 33 | ### Discovering Instance MetaData 34 | 35 | With the instance metadata service , code running on an Amazon Ec2 instance can discover properties about the instance. you can curl to get the metadata attributes 36 | 37 | ```bash 38 | curl http://169.254.169.254/latest/meta-data/ 39 | ``` 40 | 41 | ## AWS Virtual Provide Cloud 42 | 43 | Amazon Virtual Private Cloud provides logically isolated networking within your AWS account. These networks are software-defined and can span all of the availability zones within a specific region. For each VPC, you have full control over whether the Amazon VPC is connected to the internet, to a premises network, or another Amazon VPC. 44 | 45 | Connections types are following 46 | 47 | 1. **Internet Gateway** - A highly available connection that allows outbound and inbound request to the internet from your VPC 48 | 2. **Egress Only Internet Gateway** - A special type that allows outbound traffic 49 | 3. **Gateway** - Allow you to establish a private connection to your other network. 50 | 4. **Amazon VPC Endpoint**- Allow traffic from your Amazon VPC to go to a specific AWS Service. 51 | 5. **Amazon VPC peering** - Privately route traffic from one VCPC to another VPC 52 | 6. **AWS Transit Gateway** - This allows you to centrally manage the connectivity between many VPCs and an on-premises environment using a single gateway 53 | 54 | ### IP Address 55 | 56 | There are four types of IP Addresses. 57 | 58 | ### Public IP 59 | 60 | Unlike the private IP address, the public IP address is IPv4 which can be reachable from the internet. You cannot manually associate or disassociate public IP addresses from an instance. 61 | 62 | ### Private IP 63 | 64 | Private IP addresses are IPv4 addresses there not reachable from the internet. These addresses are unique within a VPC and use for traffic that is to be routed internally within the VPC for private communication with other networks. 65 | 66 | ### Elastic IP 67 | 68 | An Elastic IP Address is similar to a public IP Address that can be reachable from the internet. You can disassociate or allocate between the instances. 69 | 70 | You can also assign the elastic IP with infrastructure i.e NAT Gateway 71 | 72 | ### IPV6 73 | 74 | When you enable IPv6 in your VPC the network operates in dual-stack mode meaning the IPv4 and IPv6 communicate independently of each other. 75 | 76 | ### Subnets 77 | 78 | Within VPC, subnets are defined. A subnet is associated with a specific Availability Zone within the region containing the Amazon VPC. Each subnet has its block of private IP addresses defined using CIDR notation. for example, a subnet may have assigned the CIDR block range 10.0.0.0/24 which would include addresses in the range 10.0.0.0=10.0.0.255. Out of 256 possible address VPC reserve the first four IP address and the last IP address in the range. leaving 251 IP addresses in the subnet. 79 | 80 | **Public Subnet**- Directly reach from the internet 81 | 82 | **Private Subnet** - Cannot be directly accessible from the internet 83 | 84 | ### Route Tables 85 | 86 | Networking traffic exiting a subnet is controlled with routes that are defined in the routing table. Route defines how the implicit router in the Amazon VPC routes IP traffic from a subnet to destinations outside that subnet. Each routing table includes a rule called a local route. This rule or route is what allows traffic from instances in one subnet within VPC to send traffic to instances in any other subnet within the same VPC. 87 | 88 | ### Security Groups 89 | 90 | Security Groups act as a stateful firewall for your Ec2 instances. When you define a security group rules, you specify the source or destination of network traffic in addition to the protocol and protocol you allowed. 91 | 92 | - should maintain one separate security group for SSH access 93 | - time out error ⇒ security group issue 94 | - connection refused error ⇒ it's an application error or it's not launched 95 | - inbound traffic is blocked by default 96 | - outbound traffic is allowed by default 97 | - can allow inbound traffic from: 98 | - anywhere 99 | 100 | - IP 101 | - other security groups 102 | ![Untitled](https://user-images.githubusercontent.com/53600644/193905378-b644924e-5eee-44de-9c17-386f0686d599.png) 103 | 104 | 105 | ## SSH troubleshooting 106 | 107 | ### There's a connection timeout 108 | 109 | This is a security group issue. Any timeout (not just for SSH) is related to security groups or a firewall. Ensure your security group looks like this and is correctly assigned to your EC2 instance. 110 | 111 | ![Untitled 1](https://user-images.githubusercontent.com/53600644/193905468-3f8d6aa6-2d6c-450d-9a55-b2480f27752c.png) 112 | 113 | 114 | ### NACL 115 | 116 | NaCl allows an administrator to control traffic that enters and leaves a subnet. it consists of inbound and outbound rule that is associated with multiple subnets within a specific VPC. it can have a stateless firewall for traffic to or from a specific subnet. 117 | 118 | ### Network Address translation 119 | 120 | NAT allows for instance a private subnet to make an outbound request to the internet without exposing those instances to inbound connections from internet users. 121 | 122 | ### DHCP Option Set 123 | 124 | DHCP provides a standard for passing configuration information to a host on a TCP/IP network. The options field of a DHCP message contains the configuration parameter. 125 | 126 | ### Shared Responsibility Model 127 | 128 | ![Untitled 2](https://user-images.githubusercontent.com/53600644/193905488-822cc8ca-b480-47c9-9d07-ae7b5b37f206.png) 129 | 130 | 131 | ### Things you Should Know! 132 | 133 | - You can use 10 tags per EC2 Instance. 134 | - AMI is regional 135 | - When you create an AMI, it’s marked private 136 | - stateful meaning that any request allowed in is automatically allowed out. 137 | - Allowed up to 5 VPC per region 138 | - All subnets in default VPC has an internet gateway attached. 139 | - SA subnet cannot mapped to multiple AZ 140 | - You can have 1 IGW per VPC 141 | - NAT Gateway scale up to 10GBPS 142 | - NAT Gateway automatically assigned to public IP 143 | - Highes rule number in NACL is 32766 144 | - NACL rules are stateless means it does not create outbound traffic automatically. 145 | - Subnet per VPC - 200 146 | - Internet per gateway - 5 147 | - Customer Gateway per region 50 148 | - VPN connection per region - 50 149 | - Route table per VPC - 5 150 | - Security group per VPC - 500 151 | - Network interface per instance - 350 152 | - Rules per NACL - 20 153 | - NAT Gateway per AZ - 5 154 | -------------------------------------------------------------------------------- /Databases/databases.md: -------------------------------------------------------------------------------- 1 | # Databases 2 | 3 | | Service | Type | Relation | 4 | | --- | --- | --- | 5 | | Amazon RDS | Relational Database | A Managed Database for MySQL, PostgreSQL , Oracle , SQL Server and MariaDB | 6 | | Amazon DynamoDB | Non-Relational Database | A Serverless managed NoSQL database that delivers consistent single-digit millisecond latency at any scale. | 7 | | Amazon Redshift | Data Warehouse | A Fast Full managed petabyte-scale data warehouse at one-tenth the cost of traditional solution | 8 | | Amazon Elastic Cache | In Memory Data store | To deploy , operate and scale an in memory data store based on Memcached or Redis | 9 | | Amazon Neptune | Graph Database | A Fast , Reliable full managed graph database to store and manage highly connected datasets. | 10 | | Amazon Document DB (MongoDB) | Non Relational | A fast , scalable , highly available and fully managed document database service that supported MongoDB workloads | 11 | | Amazon timestream | Time Series Database | A Fast Scalable, fully managed time service database for IoT and operational applications. | 12 | | Amazon Quantum | Ledger Database | A fully managed ledger database that provides transparent, immutable, and cryptographically verifiable transactions. | 13 | | AWS DMS | Data Migration | Help migrate your databases to AWS easily and inexpensively with minimal downtime. | 14 | 15 | | Applications | Service | 16 | | --- | --- | 17 | | Transactional Applications such as ERP , CRM and ecommerce to log transactions and store structured data. | Amazon RDS | 18 | | Internet scale application such as hospitality , dating and ride sharing to server content and store structure and unstructured data | DynamoDB or DocumentDB | 19 | | Analytics Applications for operational reporting and querying terabyte to exabyte scale data | Redshift | 20 | | Real-time application use cases that require sub millisecond latency such as gaming lead boards , chat messaging , streaming and IoT | AWS Elastic Cache | 21 | | Applications with use cases that require navigation of highly connected data such as social news feeds recommendations and fraud detection. | Neptune | 22 | | Applications that collect data at millions of inserts per second in a time series fashion such as clickstream data | Timestream | 23 | | Application that require an accurate history of their application data | Amazon QLDB | 24 | 25 | ## RedShift 26 | 27 | ### Leader Node 28 | 29 | A Leader node acts as SQL endpoint and receives queries from client application parses the queries and develops query execution plans. 30 | 31 | ### Compute Nodes 32 | 33 | Computer node execute the query execution plan and transmit data among themselves to serve these queries. 34 | 35 | ### Node Slices 36 | 37 | A Computer node is partitioned into slices. Each slice is allocated a portion of the node memory and disk space. 38 | 39 | ## Amazon ElastiCache 40 | 41 | Amazon ElasticCache is a web service that make it easy to deploy , operate and scale an in-memory cache in the AWS Cloud. The service improves the performance of web applications by allowing you to retrieve information from fast, managed , in-memory cache instead of relying entirely on slower disk -based database. 42 | 43 | ### Redis 44 | 45 | Redist is an opensource, key-value store that supports more advanced data structures such as sorted sets , hashes and lists . Unlike Memcached , Redis has disk presistence built in , meaning that you can use it for long-lived data. 46 | 47 | ### MemCached 48 | 49 | MemCached is a widely adopted in-memory key store . it is histoically that gold standard of web caching. Elastic Cache is protocol compliant with memcached and it is designed to work with popular tol that you can use today with existing memcached enviornments. -------------------------------------------------------------------------------- /Developer Tools/developertools.md: -------------------------------------------------------------------------------- 1 | # Developer Tools 2 | 3 | ## Continuous Delivery with AWS Code pipeline 4 | 5 | The AWS Code service lay the foundation to deploy different parts of an enterprise starting from a source repository. You can start with AWS Codepipeline to create a continuous integration/continuous deployment pipeline (CI/CD) that integerates various sources , tests, deployments or other components. AWS Code pipeline implements AWS CodeCommit as a source in that it acts as the intilization point of your deployment process. AWS CodeBuild allow you to pull code and packages from various sources to create publishable build artifacts. AWS code Deploy allow you to deploy compiled artifacts to infrastructure in your enviornment. AWS Code pipeline is not limited to deploying application , it can be use for provision , configure and manage infrastructure. 6 | 7 | ![Untitled](https://user-images.githubusercontent.com/53600644/194136869-9abcb463-959f-4831-8089-d465f76c22c1.png) 8 | 9 | 10 | ## Overview 11 | 12 | - codecommit: source code control 13 | - codebuild: CI 14 | - codedeploy: CD 15 | - codepipeline: combines source control, build, and deployment 16 | 17 | ![Untitled 1](https://user-images.githubusercontent.com/53600644/194136915-c793f79c-5292-46f7-aedc-1e55282f1ac7.png) 18 | 19 | 20 | ## CodeCommit 21 | 22 | ### Security 23 | 24 | - Authentication in Git: 25 | - SSH Keys: can configure SSH keys in IAM Console 26 | - HTTPS: do with AWS CLI Authentication helper or Generating HTTPS credentials 27 | - MFA: can be enable 28 | - Authorization in Git: 29 | - IAM Policies manage user / roles 30 | - Encryption: 31 | - automatically encrypt repositories at rest using KMS 32 | - encrypted in transit (can only use HTTPS or SSH) 33 | - Cross Account access: 34 | - never share SSH keys 35 | - never share AWS credentials 36 | - use IAM Role and use AWS STS (with AssumeRole API) 37 | 38 | ### Notifications 39 | 40 | - can trigger notifications in CodeCommit using AWS SNS, Lambda, CloudWatch Event Rules 41 | - use cases for SNS, Lambda 42 | - deletion of branches 43 | - pushes that happens in master branch 44 | - notify external Build System 45 | - trigger Lambda to perform codebase analysis 46 | - use cases for CloudWatch Event Rules 47 | - pull request updates (created/upload/deleted/commented) 48 | - commit comment events 49 | - CloudWatch Event Rules goes into an SNS topic 50 | 51 | ### CodeCommit vs GitHub: 52 | 53 | - both are git repositories 54 | - both support code reivew (pull request) 55 | - both can be integrated with AWS CodeBuild 56 | - both support HTTPS, SSH 57 | - security 58 | - github: github users 59 | - codecommit: AWS IAM users + roles 60 | - hosted 61 | - github: hosted by github 62 | - github enterprise: selft hosted on client servers 63 | - codecommit: managed + hosted by AWS 64 | - UI 65 | - github UI is fully featured 66 | - codecommit UI: minimal 67 | 68 | ## CodeBuild 69 | 70 | - build instructions can be defined in code by `buildspec.yml` file - must be at the root of the code 71 | - define environment variables: 72 | - plaintext variables 73 | - secure secrets: use SSM Parameter store 74 | - phases (command to run): 75 | - install: install dependencies 76 | - pre build: final commands to execute before build 77 | - build: actual build commands 78 | - post build: finishing touches (eg: zip output) 79 | - artifacts: what to upload to S3 (encrypted with KMS) 80 | - cache: Files to cache (eg: dependencies) to S3 for future build speedup 81 | 82 | ![Untitled 2](https://user-images.githubusercontent.com/53600644/194136943-bec694e4-7cb2-437e-818c-ee87bcd5adfc.png) 83 | 84 | 85 | ## CodeDeploy 86 | 87 | - deploy application automatically to many EC2 instances 88 | - each EC2 instances (or on-premise) must be running the CodeDeploy Agent 89 | - the agent is continuously polling AWS CodeDeploy for work to do 90 | - CodeDeploy sends `appspec.yml` file 91 | - application is pulled from Github or S3 92 | - EC2 will run the deployment instructions 93 | - CodeDeploy Agent will report of success/failure of deployment on the instance 94 | 95 | ![Untitled 3](https://user-images.githubusercontent.com/53600644/194136963-7c67f2c6-9663-4ca4-a32e-7fb65155ab90.png) 96 | 97 | 98 | - EC2 will be grouped by deployment group (dev/test/prod) 99 | - Blue / Green only works with EC2 instances (not on premise) 100 | - can deploy to Lambda 101 | - `appspec.yml` 102 | - file sections: how to source and copy from S3/Github to filesystem 103 | - hooks: set of instructions to do to deploy the new version (hooks can have timeout). The order is: 104 | - ApplicationStop 105 | - DownloadBundle 106 | - BeforeInstall 107 | - AfterInstall 108 | - ApplicationStart 109 | - ValidateService: really important 110 | - configs: 111 | - one a time: one instance at a time, one instance fails ⇒ deployment stops 112 | - half at a time: 50% 113 | - all at once: quick but no healthu host, downtime ⇒ good for dev 114 | - custom: min healthy host = a% 115 | - failures: 116 | - instances stay in "failed state" 117 | - **new deployments will first be deployed to "failed state" instances** 118 | - to rollback: redeploy old deployment or enable automated rollback for failures 119 | - deployment targets: 120 | - **set of EC2 instances with tags** 121 | - **directly to an ASG** 122 | - mix of ASG/Tags 123 | - customization in scripts with DEPLOYMENT_GROUP_NAME env variables 124 | 125 | ## CodePipeline 126 | 127 | - made of stages: 128 | 129 | - each stage can have sequential actions and / or parallel actions 130 | - Stages examples: Source/Build/Deploy/Review, Build/Test/Deploy/Load Test 131 | - manual approval can be defined at any stage 132 | - each stage can create artifacts 133 | 134 | - artifacts are passed and stored in Amazon S3 and passed on to the next stage 135 | 136 | 137 | - each stage can have multiple action group 138 | 139 | - if codepipeline fails a stage, it stops and show information in the console 140 | 141 | - can create SNS notifications for 142 | 143 | - failed pipelines 144 | - cancelled stages 145 | - ... 146 | - if codepipeline can't perform an action, make sure its' IAM Role have enough permissions 147 | 148 | ![Untitled 4](https://user-images.githubusercontent.com/53600644/194137003-054d483f-4b51-46a5-871c-a891b764aed5.png) 149 | 150 | 151 | ## CodeStar 152 | 153 | - integrate solution that regroups: Github, CodeCommit, CodeBuild, CodeDeploy, CloudFormation, CodePipeline, CloudWatch 154 | - one dashboard to view all components 155 | - quickly create CICD-ready projects for EC2, Lambda, Beanstalk 156 | -------------------------------------------------------------------------------- /DynamoDB/dynamodb.md: -------------------------------------------------------------------------------- 1 | # DynamoDB 2 | 3 | ## Primary Keys 4 | 5 | ### Option 1: Partition key only 6 | 7 | - partition key must be unique for each item 8 | 9 | - partition key must be "diverse" so that the data is distributed 10 | 11 | - Example: `user_id` for a users table 12 | 13 | ![Untitled](https://user-images.githubusercontent.com/53600644/194133833-0aff5798-f63a-483c-8cdf-6bf7a8b028dc.png) 14 | 15 | 16 | 17 | ### Option 2: Partition key + Sort Key 18 | 19 | - the combination must be unique 20 | 21 | - data is grouped by partition key 22 | 23 | - sort key == range key 24 | 25 | - Example: users-games table 26 | 27 | - `user_id` for the partition key 28 | 29 | - `game_id` for the sort key 30 | 31 | ![Untitled 1](https://user-images.githubusercontent.com/53600644/194133866-9323a703-e8e1-40bf-84e1-9760f85113c7.png) 32 | 33 | 34 | 35 | ## Provisioned Throughput 36 | 37 | - Read Capacity Units (RCU): throughput for reads 38 | - Write Capacity Units (WCU): throughput for writes 39 | - Throughput can be exceeded temporarily using "burst credit" 40 | - If burst credit are empty, you'll get a "ProvisionedThroughputException" 41 | - exponential back-off retry 42 | 43 | ### Write Capacity Units 44 | 45 | - **1 WCU = 1 write/second for 1 item up to 1 KB** 46 | - Example: 47 | - write 10 objects per seconds of 2 KB each ⇒ 2 * 10 = 20 WCU 48 | - write 6 objects per second of 4.5 KB each ⇒ 6 * 5 = 30 WCU (4.5KB rounded ⇒ 5KB) 49 | - write 120 objects per minute of 2KB each ⇒ 120 / 6 * 2 = 4 WCU 50 | 51 | ### Read Capacity Units 52 | 53 | - By default: DynamoDB uses Eventually Consistent Reads, but GetItem, Query, Scan provide a "ConsistentRead" parameter you can set to True 54 | - **1 RCU = 1 strongly consistent read / second for 1 item up to 4KB 55 | 56 | ``` 57 | = 2 eventually consistent reads / second for 1 item up to 4KB** 58 | ``` 59 | 60 | - Example: 61 | - 10 strongly consistent reads / seconds of 4 KB each ⇒ 10 * 4 / 4 = 10 RCU 62 | - 16 eventually consistent reads / seconds of 12 KB each ⇒ 16 / 2 * 12 / 4 = 24 RCU 63 | - 10 strongly consistent reads / seconds of 6 KB each ⇒ 10 * 8 KB / 4 = 20 RCU (6KB round up 8KB) 64 | 65 | ### Partitions Internal 66 | 67 | - number of partitions: 68 | - by capacity: (TOTAL RCU / 3000) + (TOTAL WCU / 1000) 69 | - by size: Total Size / 10 GB 70 | - total partitions = CEILING(MAX(Capacity, Size)) 71 | - WCU and RCU are spread evenly between partitions 72 | - Exceed RCU or WCU ⇒ **ProvisionedThroughputExceededExceptions** 73 | 74 | ## API 75 | 76 | ### Write API 77 | 78 | - `PutItem`: Write data to DynamoDB (consume WCU) 79 | - `UpdateItem`: Update data in DynamoDB (partial update) 80 | - can use Atomic Counters and increase them 81 | - `BatchWriteItem` 82 | - up to 25 `PutItem` or `DeleteItem` in 1 call 83 | - up to 16 MB of data written 84 | - up to 400 KB of data per item 85 | - decrease latency by reducing the number of API calls 86 | - operations are done in parallel 87 | - it's possible for part of a batch to fail, can retry 88 | - Conditional Writes: Write/Update only if conditions are respected, otherwise reject 89 | - deal with concurrent access to items 90 | - no performance impact 91 | 92 | ### Delete API 93 | 94 | - `DeleteItem`: Delete an individual row 95 | - can perform a conditional delete 96 | - `DeleteTable`: Delete a whole table and all its items 97 | - much quicker deletion than calling `DeleteItem` on all Items 98 | 99 | ### Reading Data 100 | 101 | - `GetItem`: Read based on Primary key (HASH or HASH+RANGE) 102 | - **eventually consistent read by default**, option to use strongly consistent reads 103 | - `ProjectionExpression`: only get certain attributes (save bandwitdth, no change RCU) 104 | - `BatchGetItem`: 105 | - up to 100 items 106 | - up to 16 MB of data 107 | - items are retrieved in parallel 108 | 109 | ### `Query` 110 | 111 | - parameters: 112 | - PartitionKey - required, `=` operator 113 | - SortKey - optional, `=, <, <=, >, >=, Between, Begin` operator 114 | - `FilterExpression` to filter (**client side do this**) 115 | - returns: 116 | - up to 1 MB data 117 | - can use `Limit` option 118 | - can do pagination on the results 119 | 120 | ### `Scan` 121 | 122 | - scan entire table 123 | - return up to 1 MB of data 124 | - **inefficient, consume a lot of RCU** 125 | - limit impact using Limit or reduce the size of the result and pause 126 | - for faster performance, use **parallel scans** 127 | - scan multiple partitions at the same time 128 | - increase the throughput and RCU consumed 129 | - limit the impact like normal Scan 130 | - can use `ProjectionExpression` + `FilterExpression` 131 | 132 | ## Indexes 133 | 134 | ### Local Secondary Index (LSI) 135 | 136 | - alternate range key for table, local to the hash key 137 | - LSI must be defined at table creation time 138 | - up to 5 local secondary index 139 | - throttling: 140 | - use WCU, RCU of the main table 141 | - no special throttling considerations 142 | 143 | ### Global Secondary Index (GSI) 144 | 145 | - GSI = new partition key + optional sort key 146 | - the index is a new "table" and can project attributes on it 147 | - partition key + sort key of the original table are always projected (`KEYS_ONLY`) 148 | - extra attributes to project (`INCLUDE`) 149 | - can use all attrs from main table (`ALL`) 150 | - Must define RCU/WCU 151 | - can add / modify GSI 152 | - throttling: 153 | - if the writes are throttled on the GSI, then the main table will be throttled 154 | - even if WCU on the main tables are fine 155 | - choose your GSI partition key carefully 156 | - assign WCU carefully 157 | 158 | ## Concurrency 159 | 160 | - feature called "Conditional Update / Delete" 161 | - **optimistic locking** 162 | - ensure an item hasn't change before altering it (race condition) 163 | 164 | ## DynamoDB Accelerator - DAX 165 | 166 | - cache for DynamoDB, solve the Hot Key problem 167 | - don't need to rewrite your apps 168 | - micro second latency for cached reads & queries 169 | - 5 minutes TTL for cache by default 170 | - MultiAZ 171 | 172 | ![Untitled 2](https://user-images.githubusercontent.com/53600644/194133900-c5632586-ba8d-42ad-8319-2e1705093aae.png) 173 | 174 | 175 | ## DynamoDB Streams 176 | 177 | 178 | - change in DynamoDB (Create, Update, Delete) can end up in a DynamoDB Stream 179 | - This Stream can be read by AWS Lambda, and can: 180 | - React to changes in real time 181 | - Analytics 182 | - Insert into ElasticSearch 183 | - can implement cross region replication using Stream 184 | - only 24 hours of data retention (≠ Kinesis 1 - 7 days) 185 | 186 | ![Untitled 3](https://user-images.githubusercontent.com/53600644/194133949-59055a01-c813-4fb3-8b25-4316f4c11d1c.png) 187 | 188 | 189 | ## TTL (Time to Live) 190 | 191 | - automatically delete an item after an expiry date / time 192 | - no extra cost, not use WCU / RCU 193 | - enabled per row (define a TTL column, add a date there) 194 | - typically delete expired items within 48 hours of expiration 195 | - delete items due to TTL are also deleted in GSI / LSI 196 | - use DynamoDB Streams to recover expired items 197 | 198 | ## DynamoDB CLI 199 | 200 | - `--projection-expression`: attributes to retrive 201 | - `--filter-expression`: filter results 202 | - pagination for DynamoDB / S3 203 | - Optimization 204 | - `--page-size`: full dataset is still received but API is called many time and request less data (void timeouts) - 3 items but call 3 times and each request receives 1 item 205 | - pagination 206 | - `--max-items`: max number of results returned. Returns `NextToken` - 3 items, call once and receives only 1 item 207 | - `--starting-token`: specify the last received `NextToken` to keep on reading 208 | 209 | ## Transactions 210 | 211 | - like RDBMS transaction 212 | - Ability to Create / Update / Delete multiple rows in different tables at the same time 213 | - do all or nothing 214 | - Write modes: Standard + Transactional 215 | - Read modes: Eventual Consistency, Strong Consistency + Transactional 216 | - consume x2 WCU / RCU 217 | 218 | ## Security & Others 219 | 220 | - Security: 221 | - VPC Endpoints can access DynamoDB 222 | - control access by IAM 223 | - encrypt at rest using KMS 224 | - encrypt in transit using SSL/TLS 225 | - Backup and Restore 226 | - Point in time restore like RDS 227 | - No performance impact 228 | - Global Tables 229 | - Multi region, fully replicated, high performance 230 | - use Amazon DMS to migrate Mongo, Oracle, MySQL, S3... to DynamoDB 231 | - can launch a local DynamoDB for development purposes 232 | -------------------------------------------------------------------------------- /ECS-Fargate-ECR/container-registry.md: -------------------------------------------------------------------------------- 1 | # ECS & ECR & Fargate 2 | 3 | ## ECS 4 | ![D499EA64-19DD-482A-8E85-DA811D4A39D8](https://user-images.githubusercontent.com/53600644/194398458-983c8f9b-a289-4f26-a1cd-fd2dcc20b0ee.jpeg) 5 | 6 | 7 | 8 | - used to run Docker containers and has 3 flavors: 9 | - ECS "Classic": provision EC2 instances to run containers 10 | - Fargate: ECS serverless, no need to provision EC2 11 | - EKS: managed Kubernetes by AWS 12 | 13 | ### ECS Clusters 14 | 15 | - logical grouping of EC2 instances 16 | - EC2 instances run the ECS agent (Docker container) 17 | - The ECS agents registers the instance to the ECS cluster 18 | - The ECS instances run a special AMI, made specifically for ECS 19 | - EC2 instances must enable config `ECS_ENABLE_TASK_IAM_ROLE` in `/etc/ecs/ecs.config` file (automatically setup by ECS) 20 | 21 | ### ECS Task Definitions 22 | 23 | - Tasks definitions are metadata in JSON form to tell ECS how to run a Docker Container 24 | - Information about: 25 | - Image Name 26 | - Port Binding for Container and Host 27 | - Memory and CPU required 28 | - Environment variables 29 | - Networking information 30 | 31 | ### EC2 Service 32 | 33 | - ECS Services help define how many tasks should run and how they should be run 34 | - ensure that the number of tasks desired is running across our fleet of EC2 instances 35 | - can be linked to ELB / NLB / ALB 36 | 37 | ### ECS Service with Load Balancer 38 | 39 | - must not specify a host port (only container port) 40 | - use ALB with dynamic port mapping 41 | - EC2 instance security group must allow traffic from the ALB on all ports 42 | 43 | ## ECR 44 | 45 | - private Docker images repository 46 | - access is controlled through IAM 47 | - commands: 48 | - `$(aws ecr get-login --no-include-email --region eu-west-1)` 49 | - `docker push[123456789.dkr.ecr.eu-west-1.amazoneaws.com/demo:latest](http://123456789.dkr.ecr.eu-west-1.amazoneaws.com/demo:latest)` 50 | - `[docker pull 123456789.dkr.ecr.eu-west-1.amazoneaws.com/demo:latest](http://123456789.dkr.ecr.eu-west-1.amazoneaws.com/demo:latest)` 51 | 52 | ## Fargate 53 | 54 | - serverless 55 | - no need provision EC2 instances 56 | - provision the container spec (CPU/RAM) 57 | - just create task definitions, AWS will run our containers 58 | - to scale, just increase the task number 59 | 60 | ## ECS & X-Ray 61 | 62 | - `AWS_XRAY_DAEMON_ADDRESS`: Use this variable if you have configured the daemon to listen on a different port or if it is running on a different host. 63 | 64 | ### ECS Cluster: X-Ray Container as a Daemon 65 | 66 | ![Untitled](https://user-images.githubusercontent.com/53600644/194398484-6d34a3e7-6a5d-48d6-a360-0ed14e292ee3.png) 67 | 68 | 69 | ### ECS Cluster: X-Ray Container as a "Side Car" 70 | 71 | ![Untitled 1](https://user-images.githubusercontent.com/53600644/194398602-48fa1adf-0ec5-4fcc-96a6-4168eab14e0d.png) 72 | 73 | 74 | ### Fargate Cluster: X-Ray Container as a "Side Car" 75 | 76 | ![Untitled 2](https://user-images.githubusercontent.com/53600644/194398627-e008a0d9-5185-4189-8599-f61c64d14032.png) 77 | 78 | ## Elastic Beanstalk + ECS 79 | 80 | - can run Elastic Beanstalk in Single & Multi Docker Container mode 81 | - Multi Docker helps run multiple containers per EC2 instance in EB 82 | - This will create: 83 | - ECS Cluster 84 | - EC2 instances 85 | - Load Balancer 86 | - Task definitions and execution 87 | - Require `Dockerrun.aws.json` file at the root of source code 88 | - Docker images must be pre-built and stored in ECR 89 | -------------------------------------------------------------------------------- /Elastic BeanStalk/elasticbeanstalk.md: -------------------------------------------------------------------------------- 1 | # Beanstalk 2 | 3 | ## Phases of Release Lifecycle 4 | 5 | ### Source 6 | 7 | During the source phase, developers check changes into a source code repository. Many teams requires peer feedback on code changes before delivery code to target environment. 8 | 9 | ### Build 10 | 11 | During the build process, application source code is built, quality of the code is tested on the build machine. 12 | 13 | ### Test 14 | 15 | The test phase is to perform a test that cannot be done during the build phase and that requires the software to be deployed to the target environment. These test includes testing integration with other live systems, load testing, UI testing, and penetrations testing. 16 | 17 | ### Deploy 18 | 19 | In the deployment phase, the code is deployed to the target environment. 20 | 21 | ### Monitor 22 | 23 | Monitor phases, and check the application to detect unusual activities and errors quickly. 24 | 25 | ## Deploying Highly Available and Scalable Applications 26 | 27 | Load balancing is a integral part to directing and managing traffic among your instances. 28 | 29 | 1. **Application Load Balancer** : This load balancer operates at level (Layer 7 ) to route HTTP/HTTPS traffic to its targets. 30 | 2. **Network Load Balancer** : This loadbalancer operates at the connection level (Layer 4) t route TCP traffic to targe . 31 | 3. **Classic Load balancer :** This loadbalancer operates at both request level and the connection level. 32 | 33 | ### Deploy and Maintain Services 34 | 35 | 1. **AWS Beanstalk -** With AWS BeanStalk you do not need to worry about managing the infrastructure for your application. You deploy your application such as Python application and elastic beanstalk takes care of scaling and managing it. 36 | 2. **AWS OpsWork-** AWS OpsWorks is a configuration and deployment management tool for your chef or puppet resources stack. 37 | 3. **AWS Cloudformation-** is infrastructure as code. The service helps you model and set up AWS resources so that you can spend less time managing them. 38 | 39 | ## Elastic BeanStalk Overview 40 | 41 | - managed service 42 | - instance configuration / OS is handled by beanstalk 43 | - deployment strategy is configurable but performed by BeanStalk 44 | - just the application code is the responsibility of the developer 45 | - three architecture models: 46 | - single instance deployment: good for dev 47 | - LB + ASG: great for production or pre-production web applications 48 | - ASG only: great for non-web apps in production (workers, etc...) 49 | - include 3 components: 50 | - application 51 | - application version: each deployment gets assigned a version 52 | - environment name (dev, test, prod...): free naming 53 | - deploy application version to environments and can promote application versions to the next environment 54 | - rollback to previous application version 55 | - full control over lifecycle of environments 56 | 57 | ## Deployment Strategy 58 | 59 | ### All at once 60 | 61 | - fastest deployment 62 | - application has downtime 63 | - great for quick iterations in development environment 64 | - no additional cost 65 | 66 | ![0905BCEB-5ADF-41CD-95F6-F58C36C53549](https://user-images.githubusercontent.com/53600644/194135776-7becdd8b-d1a6-4565-9f0d-6544161dd6d7.jpeg) 67 | 68 | 69 | ### Rolling 70 | 71 | - application is running below capacity 72 | - can set the bucket size 73 | - application is running both versions simultaneously 74 | - no additional cost 75 | - long deployment 76 | ![C33A92C5-DFFF-441B-B15C-6F69FA061ABC](https://user-images.githubusercontent.com/53600644/194135836-c670a67e-3758-42eb-bb2a-b8693acad46c.jpeg) 77 | 78 | 79 | 80 | ### Rolling with additional batches 81 | 82 | - application is running at capacity 83 | - can set the bucket size 84 | - application is running both versions simultaneously 85 | - small additional cost 86 | - additional batch is removed at the end of the deployment 87 | - longer deployment 88 | - good for prod 89 | 90 | ![EB49BCFD-1629-49EB-BA0E-B127A2A4A817](https://user-images.githubusercontent.com/53600644/194135861-b40674bb-da8b-46da-aff9-d5940edd8f60.jpeg) 91 | 92 | 93 | ### Immutable 94 | 95 | - zero downtime 96 | - new code is deployed to new instances on a temporary ASG 97 | - high cost, double capacity 98 | - longest deployment 99 | - quick rollback in case of failures (just terminate new ASG) 100 | - great for prod 101 | 102 | ![3D42B2F6-31E0-4F2D-A39A-9E0C86F201FF](https://user-images.githubusercontent.com/53600644/194135884-cf1d27ec-f40f-4072-9f40-672d0448a732.jpeg) 103 | 104 | 105 | ### Blue / Green 106 | 107 | - not a "direct feature" of Beanstalk 108 | - zero downtime and release facility 109 | - create a new "stage" environment and deploy v2 there 110 | - the new environment (green) can be validated independently and roll back if issues 111 | - Route53 can be setup using weighted policies to redirect a little bit of traffic to the stage environment 112 | - Using Beanstalk, "swap URLs" when done with the environment test 113 | 114 | ![13EA311F-FF99-429F-9C57-903ED3723699](https://user-images.githubusercontent.com/53600644/194135918-9347f106-c0ef-4053-96a5-263e263be5ef.jpeg) 115 | 116 | ### Comparison Table 117 | 118 | |Method |Impact of failed deployment |Deploy time|Zero downtime|No DNS change|Rollback process |Code deployed to | 119 | |-----------------------------|--------------------------------------------------------------------------------------------------|-----------|-------------|-------------|-----------------------|-----------------------------------| 120 | |All at once |Downtime |🕒 |☓ |✓ |Manual Redeploy |Existing instances | 121 | |Rolling |Single batch out of service; any successful batches before failure running new application version|🕒🕒☨ |✓ |✓ |Manual Redeploy |Existing instances | 122 | |Rolling with additional batch|Minimal if first batch fails; otherwise, similar to Rolling |🕒🕒🕒☨ |✓ |✓ |Manual Redeploy |Redeploy New and existing instances| 123 | |Immutable |Minimal |🕒🕒🕒🕒 |✓ |✓ |Terminate New Instances|New instances | 124 | |Blue/green |Minimal |🕒🕒🕒🕒 |✓ |☓ |Swap URL |New instances | 125 | 126 | 127 | ## Elastic Beanstalk Extensions 128 | 129 | - all the parameters set in the UI can be configured with code using files 130 | - requirements: 131 | - in the `.ebextensions/` directory in the root of source code 132 | - YAML/JSON format 133 | - `*.config` extensions 134 | - able to modify some default settings using: option_settings 135 | - ability to add resources such as RDS, ElastiCache, DynamoDB... 136 | - Resources managed by `.ebextensions` get deleted if the environment goes away 137 | 138 | ## Beanstalk Lifecycle Policy 139 | 140 | - Beanstalk can store at most 1000 application versions 141 | - if not remove old versions ⇒ can't deploy anymore 142 | - 2 types of lifecycle policy: 143 | - based on time (old versions are removed) 144 | - based on space (when you have too many versions) 145 | - versions that are currently used won't be deleted 146 | - option not to delete the source bundle in S3 to prevent data loss 147 | 148 | ## Web server vs Worker Environment 149 | 150 | - application performs tasks that are long to complete, offload ⇒ use **worker environment** 151 | - decoupling application into 1 tiers is common 152 | - Eg: processing a video, generating a zip file... 153 | - can define periodic tasks in a file cron.yaml 154 | 155 | ![Untitled](https://user-images.githubusercontent.com/53600644/194136196-d9a8d4fc-519e-48a8-859e-311f9f3c9872.png) 156 | 157 | 158 | ## RDS with Elastic Beanstalk 159 | 160 | - RDS provisioned with Beanstalk is great for dev/test 161 | - use separate RDS for production 162 | - migrate from RDS coupled with EB to standalone RDS: 163 | - take RDS db snapshot 164 | - enable deletion protection in RDS 165 | - create a new environment without an RDS, point to existing old RDS 166 | - perform blue/green deployment and swap new and old environments 167 | - terminate the old environment (RDS won't get deleted thanks to protection) 168 | - delete CloudFormation stack (will be in DELETE_FAILED state) 169 | 170 | ## Tips 171 | 172 | - under the hood, Beanstalk relies on CloudFormation 173 | - package code as zip file ⇒ zip file is uploaded to each EC2 machine ⇒ each EC2 machine resolves dependencies ⇒ slow 174 | - optimization: package dependencies with source code to improve deployment performance and speed 175 | - Beanstalk with HTTPS 176 | - load the SSL certificate onto the Load Balancer 177 | - can be done from Console (EB console, load balancer configuration) 178 | - can be done from the code `.ebextension/securelistener-alb.config` 179 | - SSL certificate can be provisioned using ACM (AWS certificate manager) or CLI 180 | - Must configure a security group rule to allow incoming port 443 (HTTPS port) 181 | - Beanstalk redirect HTTP to HTTPS 182 | - configure instances to redirect HTTP to HTTPS 183 | - or configure the Application Load Balancer (only) with a rule 184 | - make sure health checks are not redirected 185 | -------------------------------------------------------------------------------- /IAM/iam.md: -------------------------------------------------------------------------------- 1 | # API Credentials and AWS Identity and Access Management 2 | 3 | ## AWS Software Development Kits 4 | 5 | AWS SDKs are available in many popular languages such as Java, .NET, JavaScript, PHP, Python, Ruby , Go, and C++. The Python SDK for AWS is called AWS SDK for Python (Boto).It allows developers to access AWS from JavaScript code that runs directly in the browser, and it includes access to AWS components like Amazon S3, Amazon SNS, Amazon SQS, DynamoDB, and more. 6 | 7 | ## AWS CLI 8 | 9 | The AWS CLI is an open-source tool built on top of the AWS SDK for Python(Boto) that provides commands for interacting with AWS Services. With Minimal configurations, you can start using all of the functionality provided by AWS Console from your favorite terminal program. 10 | 11 | - use aws configure to set your configurations 12 | - use `—dry-run` to check cli command have permission 13 | - `aws sts decode-authorization-message --encoded-message ` to show entire message 14 | - exponential backoff 15 | - any API that fails because of too many calls needs to be retried with exponential backoff 16 | - apply to rate limited API 17 | - also apply to SDK call api 18 | - `aws configure --profile ` config multiple aws profiles 19 | - `aws s3 ls —-profile ` use aws cli with specific profile 20 | - Can create Access Keys/ Secret Access Keys to allow IAM users (or service accounts) to be used with AWS CLI or API calls 21 | 22 | ## Working With Regions 23 | 24 | Each AWS region is located in a separate geographic area and maintains its own, isolated copies of AWS Services. Each AWS Regions contain multiple data centers, grouped together to form Availability Zones. Regions are composed of multiple Availability zones, which allows AWS to provide highly available services in a way that differentiates them from traditional architecture with single or multiple data centers. Availability zones are physically separated from each other and designed to operate independently in case of a fault or natural disaster. 25 | 26 | ## Identity Access and Management (IAM) 27 | 28 | ## Users 29 | 30 | - A user can represent a real person who requires access to operate and maintain your AWS Environment. 31 | - A user can be used by an application that requires permissions to access your AWS resources programmatically 32 | - Permissions can be assigned to the user or inherited from a group 33 | 34 | ![User_(1)](https://user-images.githubusercontent.com/53600644/193903172-f657b6db-eb8c-4b1a-a092-8eba574a137e.png) 35 | 36 | ## Groups 37 | 38 | - IAM Groups are objects like user objects 39 | - Groups are not used in the authentication process 40 | - They are used to authorize access through AWS Policies 41 | - IAM Group contains Users and has IAM Policies Associated. 42 | - AWS Has a default maximum limit of hundred groups 43 | - A user can only be associated with 10 groups 44 | 45 | ![Untitled](https://user-images.githubusercontent.com/53600644/193903219-678f4cbc-28c4-4e24-b066-853a78c1766f.png) 46 | 47 | 48 | ## Roles 49 | 50 | IAM Roles allow you to adopt a set of temporary IAM Permissions 51 | 52 | There are currently four different types of Roles 53 | 54 | 1. AWS Service Role 55 | 2. AWS Service - Linked Role 56 | 3. A Role for Cross Accounts Access 57 | 4. A Role for Identity Provider Access 58 | 59 | ![Untitled 1](https://user-images.githubusercontent.com/53600644/193903285-de101e3d-cec6-4d26-8eb9-155c443c24ed.png) 60 | 61 | 62 | ## Advanced IAM - Authorization Model 63 | 64 | - **AWS Managed Policy** 65 | - maintained by AWS 66 | - good for power users and administrators 67 | - updated in case of new services/ new api 68 | - **customer Managed Policy** 69 | - best practice, re-usable, can be applied to many principals 70 | - version controlled + rollback, central change management 71 | - **Inline** 72 | - strict one-to-one relationship between policy and principal 73 | - policy is deleted if you delete the IAM principal 74 | 75 | ### AWS Policy Structure 76 | 77 | ```json 78 | "Version": "2012-10-17", 79 | "Statement": [{ 80 | "Sid": "AllowDeleteForSpecifiedLexicon", 81 | "Effect": "Allow", 82 | "Action": [ 83 | "polly:DeleteLexicon"], 84 | "Resource": "arn:aws:polly:us-west-2:123456789012:lexicon/awsLexicon" 85 | } 86 | ] 87 | 88 | ``` 89 | 90 | *Effect* –Allow or Deny access to the resource is decided by Effect (Allow/Deny) 91 | 92 | *Action* — A set of service-specific parameters (like “iam: CreateUser”). 93 | 94 | *Resource* — Resource names (like “arn:aws:s3:::conf-* “) 95 | 96 | *Condition* (Optional) — Grant conditions (like “aws: RequestedRegion”: “ap-south-1”) 97 | 98 | ## Permissions Evaluations Structure 99 | ![Untitled 2](https://user-images.githubusercontent.com/53600644/193903420-341e8f9a-65bd-498f-843c-7bc68a8d184b.png) 100 | 101 | 102 | ### IAM Policies & S3 Bucket Policies 103 | 104 | - IAM policies are attached to users, roles, groups 105 | - S3 Bucket Policies are attached to buckets 106 | 107 | - Example 108 | 109 | - IAM Role attached to an EC2 instance, Read Write permission provided to bucket" + no S3 Bucket policy attached 110 | 111 | ⇒ EC2 instance can read, write to the bucket 112 | 113 | - IAM Role attached to an EC2 instance, RW permission to bucket + S3 Bucket Policy deny to the IAM Role 114 | 115 | ⇒ EC2 instance cannot read, write to "bucket" 116 | 117 | - IAM Role attached to an EC2 instance, no S3 bucket permission + S3 Bucket Policy attached, RW permission to the IAM Role 118 | 119 | ⇒ EC2 instance can read, and write to "bucket" 120 | 121 | - IAM Role attached to an EC2 instance, deny S3 bucket permission + S3 Bucket Policy attached, allow RW permission to the IAM Role 122 | 123 | ⇒ EC2 instance cannot read, write to "bucket" 124 | 125 | ![Untitled 3](https://user-images.githubusercontent.com/53600644/193903450-7a395f6c-19a1-4e5e-82f1-11ba5725346e.png) 126 | 127 | 128 | ### Dynamic Policies 129 | 130 | - assign each user a `/home/` folder in S3 131 | - create only one dynamic policy with IAM 132 | - use the special policy variable `${aws:username}` 133 | 134 | ```json 135 | { 136 | "Sid": "AllowAllS3ActionsInUserFolder", 137 | "Action": ["s3:*"], 138 | "Effect": "Allow", 139 | "Resource": ["arn:aws:s3:::my-company/home/${aws:username}/*"] 140 | } 141 | ``` 142 | 143 | ### STS 144 | 145 | - Grants users limited and temporary access to AWS resources. 146 | - 3 sources: 147 | 1. Federation (often Active Directory) 148 | - Uses SAML 149 | - SSO allows users to log in to AWS Console without assigning IAM credentials 150 | 2. Federation with mobile app 151 | - Use Facebook/Amazon/Google or other openID provider 152 | 3. Cross account access 153 | - Lets users from one AWS account access resources in another 154 | 155 | ![Untitled 4](https://user-images.githubusercontent.com/53600644/193903552-860c46c7-cfbf-471d-be9a-e83be93ebb21.png) 156 | 157 | 158 | ## Things to Know 159 | 160 | - IAM is global 161 | - IAM has many predefined "managed policies" by Amazon 162 | - Can integrate users with IAM, so the employee can log in to AWS using company credentials by using **Identity Federation** (Eg: company use Active Directory, SAML standard) 163 | - use MFA (Multi-Factor Authentication) 164 | - give users the minimal amount of permissions they need to perform their job 165 | - 1 IAM User per 1 Physical Person 166 | - 1 IAM Role per Application 167 | - never share IAM credentials 168 | - never write IAM credential in code or commit to git 169 | - only use the ROOT account for initial setup 170 | - never put personal credentials on EC2 ⇒ use IAM roles instead 171 | - on-premise server, the best practice is to call STS to obtain temporary security credentials 172 | - cross-account access 173 | - define an IAM Role for another account to access 174 | - define which accounts can access this IAM Role 175 | - use AWS STS (Security Token Service) to retrieve credentials and impersonate the IAM Role you have access to (`AssumeRole` API) 176 | - temporary credentials can be valid between 15 minutes to 1 hour 177 | 178 | ### Resources 179 | 180 | * AWS Whitepapers (Introduction to AWS): 181 | [https://aws.amazon.com/whitepapers/](https://aws.amazon.com/whitepapers/) 182 | * AWS Training and Certification: 183 | [https://aws.amazon.com/training](https://aws.amazon.com/training) 184 | * AWS Events and Webinars: 185 | [https://aws.amazon.com/about-aws/events](https://aws.amazon.com/about-aws/events) 186 | -------------------------------------------------------------------------------- /KMS/KMS.md: -------------------------------------------------------------------------------- 1 | # KMS 2 | 3 | ## Centralized Key Management System 4 | 5 | AWS KMS provides you with a centralized view of your encryption keys. you can create a customer master key (CMK) to control access to your data encryptions data and encrypt an encrypt your data. AWS Key Management System Use AES-256 Bit mode to encrypt and secure your data. 6 | 7 | ### Custom Key Store 8 | 9 | You can create your own custom key store in a CloudHSM cluster that you control enabling you to store your AWS KMS keys in a single tenant environment instead of the default multi-tenant environment of AWS KSM. 10 | 11 | ### CloudHSM 12 | 13 | AWS CloudHSM offer third party, validated FIPS 140-2 level three hardware security modules in the AWS Cloud. The hardware security module is a computing device that provide a dedicated infrastructure to support cryptographic operations. 14 | 15 | ## Things you Should know! 16 | 17 | - never store your secrets key in plaintext ⇒ can store encrypted secret key in code / environment variables 18 | - KMS can only help in encrypting up to 4KB of data per call 19 | - if data > 4KB, use envelope encryption (client side encrypt) 20 | - give access to KMS: 21 | - make sure the Key Policy allows the user 22 | - make sure the IAM Policy allows call to KMS API 23 | - manage the keys & policies: 24 | - create 25 | - rotation policies 26 | - disable 27 | - enable 28 | - can audit key usage (using CloudTrail) 29 | - 2 types of CMK: 30 | - AWS Managed Service Default CMK: free 31 | - User Keys created in KMS: 1$/month 32 | - User Keys imported (must be 256-bit symmetric key): 1$/month 33 | - pay for API call to KMS (0.03$ / 10000 calls) 34 | 35 | ## AWS Encryption SDK 36 | 37 | - encrypt over 4KB using KMS 38 | - different from the S3 Encryption SDK 39 | - Encryption SDK = Envelope Encryption = GenerateDataKey API 40 | 41 | ## AWS Parameter Store 42 | 43 | - Systems Manager / Parameter Store UI 44 | - secure storage for configuration and secrets 45 | - encrypt using KMS 46 | - serverless, scalable, durable, easy SDK, free 47 | - version tracking 48 | - manage configuration (who can see, decrypt...) by path & IAM 49 | - notifications with CloudWatch Events 50 | - integration with CloudFormation -------------------------------------------------------------------------------- /Lambda/Lambda.md: -------------------------------------------------------------------------------- 1 | # Lambda 2 | 3 | ## Overview 4 | 5 | - pay per request and compute time 6 | 7 | ## Configuration 8 | 9 | - timeout: default 3 seconds, maximum 15 minutes 10 | - Environment variables 11 | - allocated memory (128MB to 3G) 12 | - ability to deploy within a VPC + assign security groups 13 | - IAM execution role must be attached to the Lambda function 14 | - tracing with AWS X-Ray ⇒ enable active tracing will automatically adds X-Ray permission 15 | 16 | ## Concurrency and Throttling 17 | 18 | - concurrency up to 1000 executions (can increase through ticket) (standard concurrency) 19 | 20 | - can set a "reserved concurrency" at the function level (maximum concurrency that function can reach) 21 | 22 | - can set "provisioned concurrency" 23 | 24 | - each invocation over the concurrency limit will trigger a "Throttle" 25 | 26 | - Throttle behavior: 27 | 28 | - if synchronous invocation ⇒ return ThrottleError - 429 (eg: http request) 29 | 30 | - If asynchronous invocation ⇒ retry automatically twice and after all retries, then go to DLQ (eg: S3 trigger) 31 | 32 | - DLQ can be a SNS topic or SQS queue 33 | 34 | ⇒ debug in production 35 | 36 | 37 | ## Limitation 38 | 39 | - Execution: 40 | - Memory allocation: 128MB - 3008MB 41 | - Maximum execution time: 15 minutes 42 | - Disk capacity (in `/tmp`): 512MB 43 | - Concurrency limits: 1000 44 | - Deployment: 45 | - install the dependencies packages and zip with code 46 | - libraries need to be compiled on Amazon Linux 47 | - Lambda function deployment size (compressed): 50MB 48 | - if > 50MB, use S3 49 | - Uncompressed deployment: 250MB 50 | - can use `/tmp` directory to load other file at startup 51 | - size of environment variables: 4KB 52 | 53 | ## Versions vs Aliases 54 | 55 | ### Version 56 | 57 | - publish a Lambda function ⇒ create a version 58 | - versions are **immutable** 59 | - versions have their own ARN 60 | - version = code + configuration 61 | 62 | ### Alias 63 | 64 | - aliases are **pointers** to Lambda function versions 65 | - can define a "dev", "test", "prod" aliases and point to different lambda versions 66 | - aliases are **mutable** 67 | - enable Blue / Green deployment by assigning weights to versions 68 | - aliases have their own ARNs 69 | 70 | ## Using with CloudFormation 71 | 72 | - store the Lambda zip in S3 73 | - refer S3 zip location in the CloudFormation code 74 | 75 | ## `/tmp` space 76 | 77 | - use cases: 78 | - if your Lambda function needs to download a big file to work 79 | - if your Lambda function needs disk space to perform operations 80 | - max size is 512MB 81 | - The directory content remains when the execution context is frozen, providing transient cache that can be used for multiple invocations 82 | - For persistence of object (non temporary), use S3 83 | 84 | ## Best practices 85 | 86 | - perform heavy-duty work outside of your function handler, function will reuse it 87 | - connect to databases 88 | - initialize the AWS SDK 89 | - pull in dependencies or datasets 90 | - use environment variables 91 | - database connection string, S3 bucket... 92 | - password, sensitive value... encrypt using KMS 93 | - minimize deployment package size 94 | - break down the function 95 | - the Lambda zip limit 96 | - avoid using recursive code, never have a Lambda function call itself ⇒ the bill will increase 97 | - don't put Lambda function in VPC unless you have to ⇒ initialize speed will increase 98 | 99 | Read more: [AWS Serverless Lambda functions best practice](https://www.notion.so/AWS-Serverless-Lambda-functions-best-practice-82bc188d2b634120985ecf18bb99ba91) 100 | 101 | ## Lambda@Edge 102 | 103 | - use Lambda@Edge deploy Lambda functions alongside your CloudFront CDN 104 | 105 | ⇒ deploy Lambda globally 106 | 107 | - can use Lambda to change CloudFront request and responses: 108 | 109 | - after CloudFront receives a request from a viewer (viewer request) 110 | 111 | - before CloudFront forwards the request to the origin (origin request) 112 | 113 | - after CloudFront receives the response from the origin (origin response) 114 | 115 | - before CloudFront forwards the response the the viewer (viewer response 116 | 117 | ![Untitled](https://user-images.githubusercontent.com/53600644/194392446-6b80a72a-f156-4ad6-a2a5-bb56b6d3b131.png) 118 | 119 | 120 | - can generate responses to viewers without ever sending the request to the origin 121 | 122 | - global application 123 | 124 | ![Untitled 1](https://user-images.githubusercontent.com/53600644/194392518-4fcf8ff6-88ed-4e0a-b7d8-2cf4882ead15.png) 125 | 126 | 127 | - Use cases: 128 | 129 | - Website Security and Privacy 130 | - Dynamic Web App at the Edge 131 | - Search Engine Optimization (SEO) 132 | - Intelligently Route Across Origins and Data Centers 133 | - Bot Mitigation at the Edge 134 | - Real-time Image Transformation 135 | - A/B Testing 136 | - User Authentication, Authorization 137 | - User Prioritization 138 | - User Tracking and Analytics 139 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ![aws-certified-developer-associate](https://user-images.githubusercontent.com/53600644/194399324-6a84e92d-a9c7-4caf-b08e-4d79076d3d91.png) 2 | 3 | ## Table of contents 4 | 5 | - [AWS Developer Associate Notes](#project-name) 6 | - [API Credential & IAM ](https://github.com/adilshehzad786/AWS-Developer-Associate-Notes/blob/main/IAM/iam.md) 7 | - [Compute and Networking ](https://github.com/adilshehzad786/AWS-Developer-Associate-Notes/blob/main/Compute%20&%20Networking/compute&networking.md) 8 | - [Storage](https://github.com/adilshehzad786/AWS-Developer-Associate-Notes/blob/main/Storage/storage.md) 9 | - [Databases](https://github.com/adilshehzad786/AWS-Developer-Associate-Notes/blob/main/Databases/databases.md) 10 | - [DynamoDB](https://github.com/adilshehzad786/AWS-Developer-Associate-Notes/blob/main/DynamoDB/dynamodb.md) 11 | - [KMS](https://github.com/adilshehzad786/AWS-Developer-Associate-Notes/blob/main/KMS/KMS.md) 12 | - [Elastic BeanStalk](https://github.com/adilshehzad786/AWS-Developer-Associate-Notes/blob/main/Elastic%20BeanStalk/elasticbeanstalk.md) 13 | - [Developer Tools](https://github.com/adilshehzad786/AWS-Developer-Associate-Notes/blob/main/Developer%20Tools/developertools.md) 14 | - [Cloudformation](https://github.com/adilshehzad786/AWS-Developer-Associate-Notes/blob/main/Cloudformation/cloudformation.md) 15 | - [Cognito](https://github.com/adilshehzad786/AWS-Developer-Associate-Notes/blob/main/Cognito/cognito.md) 16 | - [SQS](https://github.com/adilshehzad786/AWS-Developer-Associate-Notes/blob/main/SQS/sqs.md) 17 | - [SNS](https://github.com/adilshehzad786/AWS-Developer-Associate-Notes/blob/main/SNS/sns.md) 18 | - [Lambda](https://github.com/adilshehzad786/AWS-Developer-Associate-Notes/blob/main/Lambda/Lambda.md) 19 | - [API Gateway](https://github.com/adilshehzad786/AWS-Developer-Associate-Notes/blob/main/API%20Gateway/apigateway.md) 20 | - [Containers & ECR](https://github.com/adilshehzad786/AWS-Developer-Associate-Notes/blob/main/ECS-Fargate-ECR/container-registry.md) 21 | 22 | # Other Notes 23 | 24 | - [AWS Developer Notes 1](https://github.com/mransbro/aws-developer-notes) 25 | - [AWS Developer Notes 2](http://clusterfrak.com/notes/certs/aws_deva_notes/) 26 | - [AWS Developer Notes 3](https://www.notion.so/AWS-Certified-Developer-Associate-Notes-fd46d61073764ab8b48eb4a6b0f597df) 27 | 28 | 29 | # Recommended Courses 30 | 31 | - [AWS Developer Associate by Stephane Maarek](https://www.udemy.com/course/aws-certified-developer-associate-dva-c01/) 32 | - [AWS Certified Developer - Associate by Faye](https://acloudguru.com/course/aws-certified-developer-associate) 33 | 34 | # Recommended E-Book/Book 35 | 36 | - [AWS Certified Developer Official Study Guide](https://www.amazon.com/Certified-Developer-Official-Study-Guide/dp/1119508193) 37 | 38 | # Recommended Practice Exam 39 | - [AWS CERTIFIED DEVELOPER ASSOCIATE PRACTICE EXAMS](https://tutorialsdojo.com/courses/aws-certified-developer-associate-practice-exams/) 40 | 41 | 42 | # Tips 43 | 44 | * Makes Notes, and clear your cloud concepts 45 | * Practice Daily on AWS Free Tier Account 46 | * Do Hands-on Labs for better Understanding 47 | * Don't Use Dumps! 48 | * Try to invest in courses, which can help you to build more robust concepts. 49 | 50 | _Feel Free to Open a pull request for contributions :)_ 51 | -------------------------------------------------------------------------------- /SNS/sns.md: -------------------------------------------------------------------------------- 1 | # SNS 2 | 3 | - direct integration 4 | ![3136F4BC-C5AD-4832-B81C-4121EC29717C](https://user-images.githubusercontent.com/53600644/194388828-7d97ff2b-7c4c-473f-984f-93349087571a.jpeg) 5 | 6 | 7 | - Pub / Sub 8 | ![984354AD-2A53-4CA8-AB45-4758A02A8640](https://user-images.githubusercontent.com/53600644/194388862-7546a47a-9aa1-4642-bb94-e98c23eac7f1.jpeg) 9 | 10 | 11 | - "event producer" only sends message to one SNS topic 12 | 13 | - each subscriber to the topic will get all the messages (new feature to filter messages) 14 | 15 | - up to 10,000,000 subscriptions per topic 16 | 17 | - 100,000 topics limit 18 | 19 | - subscribers can be: 20 | 21 | - SQS 22 | - HTTP/HTTPS (with delivery retries - how many times) 23 | - Lambda 24 | - Emails 25 | - SMS messages 26 | - Mobile Notifications 27 | - integrate with other AWS services: 28 | 29 | - CloudWatch (for alarms) 30 | - ASG notifications 31 | - S3 (on bucket event) 32 | - CloudFormation (state change ⇒ failed to build) 33 | - ... 34 | - Topic Publish (within AWS server - using the SDK) 35 | 36 | - create a topic 37 | - create a subscription 38 | - publish to the topic 39 | - direct publish (for mobile apps SDK) 40 | 41 | - create a platform app 42 | - create a platform endpoint 43 | - publish to platform endpoint 44 | 45 | ## SNS + SQS: Fan Out 46 | 47 | - push once in SNS, receive in many SQS 48 | - No data loss 49 | - Ability to add receivers of data later (when number of messages increase, can add consumer to handle) 50 | - SQS allows for delayed processing 51 | - SQS allows for retries of work 52 | -------------------------------------------------------------------------------- /SQS/sqs.md: -------------------------------------------------------------------------------- 1 | # SQS 2 | 3 | ![E3A330F4-9B42-409B-8248-839DC5113426](https://user-images.githubusercontent.com/53600644/194141048-7a374665-13aa-4455-a2a8-2b09cd833c9e.jpeg) 4 | 5 | 6 | ## Application communication 7 | 8 | - synchronous communications 9 | ![0B8C1B7A-0867-446E-A46E-8489681F86CE](https://user-images.githubusercontent.com/53600644/194141101-81dfeb5b-4f42-491b-a80c-dbb15dfdeb0a.jpeg) 10 | 11 | 12 | ⇒ can be problematic if there are sudden spikes of traffic 13 | 14 | - asynchronous / event based 15 | ![9E25FC29-91BC-49D0-9746-B534E73F0BFF](https://user-images.githubusercontent.com/53600644/194141142-43b3b6f3-0636-4a0c-bd85-6b8dfa32f405.jpeg) 16 | 17 | 18 | ⇒ scalable 19 | 20 | 21 | ## Producing messages 22 | 23 | - define body 24 | - add message attributes (metadata - optional) 25 | - delay delivery (optional) 26 | - get back 27 | - message identifier 28 | - MD5 hash of body 29 | ![C4AE4E3D-C9CE-4CBB-A7E8-29D591156D7B](https://user-images.githubusercontent.com/53600644/194141497-f8611f0e-3812-4269-a20a-f6518ec63ad8.png) 30 | 31 | 32 | ## Consuming messages 33 | 34 | - poll message from SQS (maximum 10 messages at a time) 35 | - process the message within the visibility timeout 36 | - delete the message using message ID & receipt handle 37 | 38 | ![3B962D11-7210-4CD8-9C60-76DF91E522F8](https://user-images.githubusercontent.com/53600644/194141174-5369f4cf-e2af-4b0e-962b-3663347d4db6.jpeg) 39 | 40 | 41 | ### visibility timeout 42 | 43 | - when a consumer polls a message from a queue ⇒ the message is invisible to other consumers for a defined period ⇒ **Visibility Timeout** 44 | - 0 second - 12 hours (default 30 seconds) 45 | - when consumer fails to process the message, the message is still invisible until **Visibility Timeout** finish ⇒ be carefull to set high Visibility Timeout (eg 15 minutes) 46 | - if too low (30 senconds) and consumer needs time to process the message (2 minutes) ⇒ another consumer will receive and process the message more than once 47 | - **ChangeMessageVisibility** API to change the visibility 48 | - **DeleteMessage** API to tell SQS the message was successfully processed 49 | 50 | ### Long Polling 51 | 52 | - when a consumer requests message from the queue, it can optionally "wait" for messages to arrive if there are none in the queue 53 | 54 | ⇒ decrease the number of API call 55 | 56 | - wait time between 1 second - 20 second (20 is preferable) 57 | 58 | - Long Polling > Short Polling 59 | 60 | - enable at queue level or **WaitTimeSeconds** API 61 | 62 | - Short polling 63 | 64 | ![38EDDEFD-FFC3-4DA5-96DA-11134B733336](https://user-images.githubusercontent.com/53600644/194141198-1ebf1019-1af2-4fe4-8e45-88b7ef71b43e.jpeg) 65 | 66 | 67 | 68 | - long polling 69 | 70 | ![B46EC6B8-E519-4F05-BC0A-55B9BF4C8245](https://user-images.githubusercontent.com/53600644/194141224-39dc48f0-1bf4-421e-b70a-a8a39987923e.png) 71 | 72 | 73 | ## Standard Queue 74 | 75 | ![E6E72E5A-C02B-45F4-83ED-E60EE47F331E](https://user-images.githubusercontent.com/53600644/194141264-726d9981-5703-4685-abc8-b342881379f0.jpeg) 76 | 77 | 78 | - **unlimited** throughput 79 | - retention 4 days by default, maximum 14 days 80 | - no limit number of messages in the queue 81 | - low latency <10ms 82 | - horizontal scaling in terms of number of consumers 83 | - can have **duplicate** messages (at least once delivery) 84 | - can have **out of order** messages (best-effort ordering) 85 | - limit 256KB per message 86 | 87 | ### Delay Queue 88 | 89 | - delay a message up to 15 minutes (consumers don't see it immediately) 90 | - default is 0 seconds 91 | - can set a default at queue level (per message delay + per queue delay) 92 | - can override the default using the **DelaySeconds** parameter 93 | 94 | ### Dead Letter Queue 95 | 96 | - set threshold of how many times a message can go back to the queue (because of failure process) ⇒ **redrive policy** 97 | - by default DLQ is not enable, messages don't go back the the queue 98 | - after the threshold is exceeded, the message goes into a DLQ 99 | 100 | ![E6485E57-AEE1-487C-BE0D-89F1AAC5B0D1](https://user-images.githubusercontent.com/53600644/194141283-afff32e9-9932-40a8-95d5-85ca3ac6c648.jpeg) 101 | 102 | 103 | ## FIFO Queue 104 | 105 | ![7BF7D8EB-09A6-4418-806D-F66A303C57BF](https://user-images.githubusercontent.com/53600644/194141313-c197f1b7-d24f-41cf-90fe-6d718dcbf512.jpeg) 106 | 107 | 108 | - high throughput (300/s without batching, 3000/s with batching) 109 | - process messages **in order** by the consumer (**sequencing**) 110 | - specify **MessageGroupId** to ensure ordering 111 | - messages with different Group ID may be received out of order 112 | - messages with the same Group ID are delivered to one consumer at a time 113 | - sent messages exactly **once (deduplication)** 114 | - provide **MessageDeduplicationId** with message 115 | - content based duplication: **MessageDeduplicationId** is generated as the SHA-256 of the message body (not attributes) 116 | - deduplication interval: 5 minutes 117 | - no per message delay (only per **queue delay**) 118 | 119 | ## SQS Extended Client 120 | 121 | - to send messages larger than 256KB ⇒ use the SQS Extended Client 122 | 123 | ![D6A6F41D-FD62-4F1E-B749-96519C6AB669](https://user-images.githubusercontent.com/53600644/194141368-6f9783a4-1ce2-4949-8060-2870c2dae261.jpeg) 124 | 125 | 126 | ## Security 127 | 128 | - encrypt on fly using HTTPS endpoint 129 | - enable SSE using KMS 130 | - can set CMK (Customer Key) 131 | - can set encrypt key reuse period (1 minute - 24 hours) 132 | - only encrypt the body 133 | - SQS queue access policy 134 | - control over IP 135 | - control over the time the request come in 136 | 137 | API 138 | 139 | - CreateQueue, DeleteQueue 140 | - PurgeQueue: delete all messages in queue 141 | - SendMessage, ReceiveMessage, DeleteMessage 142 | - ChangeMessageVisibility: change visibility timeout 143 | - WaitTimeSeconds 144 | - DelaySeconds 145 | - Batch [SendMessage/DeleteMessage/ChangeMessageVisibility] 146 | -------------------------------------------------------------------------------- /Storage/storage.md: -------------------------------------------------------------------------------- 1 | # Storage 2 | 3 | ## Amazon Elastic Block Storage (EBS) 4 | 5 | ### EBS Types 6 | 7 | - General Purpose SSD Volume (gps-gp3) 8 | - Provisioned IOPS SSD (Io1/Io2) 9 | - Throughput Optimized HDD(st1) 10 | - Cold HDD(sc1) 11 | 12 | ![Untitled](https://user-images.githubusercontent.com/53600644/193906441-6bf60aaf-9479-4171-ab0c-b0eb7f7db2e8.png) 13 | 14 | 15 | ## S3 16 | 17 | Even though each bucket is created in specific region, S3 names must be **unique** globally 18 | 19 | ### **S3's Object is consists of the following:** 20 | 21 | - Key - full path: `/my_folder/another_folder/my_file.txt` 22 | - Value - content of the body 23 | - Version ID 24 | - Metadata 25 | - Subresources (Access Control Lists, Torrent) 26 | 27 | ### **Data consistency work for S3:** 28 | 29 | - Read after Write consistency for PUTS of new Objects 30 | - Eventual Consistency for overwrite PUTS and DELETES 31 | 32 | ### **Features:** 33 | 34 | - Tiered Storage Available 35 | - Lifecycle Management 36 | - Versioning 37 | - Encryption 38 | - AES-256 (SSE-S3): use SSE with Amazon S3-Managed Key 39 | - AWS-KMS (SSE-KMS): use SSE with Amazon KMS-Managed Key 40 | - SSE-C: you manage your own encryption keys 41 | - Client Side Encryption 42 | - MFA Delete 43 | - signed URLs: URLs that are valid only for a limited time 44 | - restrict access to S3 by **Access Control Lists** or **Bucket Policies** 45 | - can create up to **100 buckets** per account by default 46 | - object's size is from **0 bytes** up to **5TB** 47 | - upload object > 100MB => **should** use multipart upload 48 | - upload object > 5GB => **must** use multipart upload 49 | 50 | ### **S3 Storage Classes (tier)** 51 | 52 | - S3 Standard (**99.99% availability** 99.999999999% durability ) 53 | - S3 - IA (Infrequently Accessed) 54 | - **99.9% availability** 99.999999999% durability 55 | - same low latency and high throughput performance of S3 Standard 56 | - minimum billable object storage size: 128KB 57 | - S3 One Zone - IA 58 | - **99.5% availability** 99.999999999% durability 59 | - Same low latency and high throughput performance of S3 Standard 60 | - minimum billable object storage size: 128KB 61 | - S3 - Intelligent Tiering 62 | - S3 Glacier 63 | - Minutes to hours of restore time 64 | - **99.99% availability** 99.999999999% durability 65 | - minimum billable object storage size: 40KB 66 | - S3 Glacier Deep Archive 67 | - 12 hours can acceptable restore time 68 | - **99.99% availability** 99.999999999% durability 69 | - minimum billable object storage size: 40KB 70 | 71 | ### **charged ways:** 72 | 73 | - Storage 74 | - Requests 75 | - Storage Management Pricing (tier) 76 | - Data Transfer Pricing 77 | - Transfer Acceleration 78 | - Cross Region Replication Pricing 79 | 80 | **Tip: RRS (reduce redundancy storage) ⇒ S3 One Zone - IA** 81 | 82 | ### **S3 LifeCycle Management can:** 83 | 84 | - Automates moving your objects between the different storage tiers 85 | - can be used in conjunction with versioning 86 | - can be applied to current versions and previous versions 87 | 88 | ### **Cross Region Replication** 89 | 90 | - Versioning must be enabled on both the source and destination buckets 91 | - Regions must be unique 92 | - Files in an existing bucket are not replicated automatically (only when add a new file that new file was automatically replicated) 93 | - All subsequent updated files will be replicated automatically 94 | - Delete markers are not replicated 95 | - Deleting individual versions or delete markers will not be replicated 96 | 97 | ### **S3 Transfer Acceleration** 98 | 99 | Instead of uploading directly to S3 bucket, user can use a distinct URL to upload directly to an edge location which will then transfer that file to S3. 100 | ![Untitled 1](https://user-images.githubusercontent.com/53600644/193906506-78a6406f-b8dd-4f80-98b8-b4fdab933bb5.png) 101 | 102 | 103 | ### Access Point 104 | 105 | - managing data access at scale for applications using shared data sets on S3 106 | - bucket can have hundreds of access points 107 | - can create an access point for your S3 bucket that grants access for groups of users or applications for your data lake. An Access Point could support a single user or application, or groups of users or applications, allowing separate management of each access point 108 | 109 | ### Exams Tip: 110 | 111 | - protected against inadvertent or intentional deletion ⇒ versioning (not cross-region replication) 112 | 113 | ## Encryption in rest 114 | 115 | ### SSE-S3 116 | 117 | - encryption using keys handled & managed by S3 118 | - object is encrypted server side 119 | - AES-256 encryption type 120 | - set header `"x-amz-server-side-encryption": "AES256"` 121 | 122 | ![Untitled 2](https://user-images.githubusercontent.com/53600644/193906628-378bca22-f86f-48df-8567-4a31ba7b8a5f.png) 123 | 124 | 125 | ### SSE-KMS 126 | 127 | - encryption using keys handled & managed by KMS 128 | - KMS Advantages: user control + audit trail 129 | - object is encrypted server side 130 | - set header `"x-amz-server-side-encryption": "aws:kms"` 131 | ![Untitled 3](https://user-images.githubusercontent.com/53600644/193906690-2f4f0de6-d2d6-467e-8167-c79f2ac73686.png) 132 | 133 | 134 | 135 | ### SSE-C 136 | 137 | - encryption using keys handled & managed by client 138 | - object is encrypted server side (S3 use key to encrypt then through key - don't store any things) 139 | - HTTPS must be used 140 | - encryption key must provided in HTTP headers, for every HTTP request made 141 | - set header 142 | - `x-amz-server-side​-encryption​-customer-algorithm` 143 | - `x-amz-server-side-encryption-customer-key` 144 | - `x-amz-server-side-encryption-customer-key-MD5` 145 | 146 | ![Untitled 4](https://user-images.githubusercontent.com/53600644/193906725-805e0a64-da7c-40a6-aec0-96f64fd2e060.png) 147 | 148 | 149 | ### Client side encryption 150 | 151 | - client encrypt data before sending to S3 152 | - client decrypt data after retrieving from S3 153 | - client fully manages the keys and encryption cycle 154 | ![Untitled 5](https://user-images.githubusercontent.com/53600644/193906741-4edc242e-fc4f-4566-a0e0-ec8caf577617.png) 155 | 156 | 157 | 158 | ## Encryption in transit (in fly) 159 | 160 | use SSL/TLS (https) to access to S3 endpoint 161 | 162 | ## Bucket policies 163 | 164 | - json based: 165 | - resources: buckets and objects 166 | - Actions: set of API to Allow or Deny 167 | - Effect: Allow/ Deny 168 | - Principal: The account or user to apply the policy to 169 | - use policy to: 170 | - Grant public access to the bucket 171 | - force objects to be encrypted at upload 172 | 173 | ## S3 CORS 174 | 175 | - if you host a static website on S3, and you . request data from another S3 bucket, you need enable CORS 176 | - CORS allow you to limit the number of websites that can request your file in S3 177 | 178 | 179 | ## S3 Performance 180 | 181 | - it was recommend to have random characters in front of your key name to optimize performance 182 | - /**5r4d**_my_folder/my_file1.txt 183 | - /**a91e**_my_folder/my_file2.txt 184 | - is was recommend to never to use dates to prefix keys: 185 | - ~~/2018_09_09_my_folder/my_file1.txt~~ 186 | - ~~/2018_09_10_my_folder/my_file2.txt~~ 187 | - from 2018/07, up to 3500 RPS for PUT, and 5500 RPS for GET ⇒ don't need to add random characters 188 | - use CloudFront for read 189 | - S3 Transfer Acceleration for write 190 | - If using SSE-KMS encryption, you may be limited to your AWS limits for KMS usage ⇒ increase the KMS limits 191 | 192 | ## S3 + Glacier Select 193 | 194 | - only retrieve some parts of data 195 | - `select * from s3object s where s.\"Country (Name)\" like '%United States%'` 196 | - save cost up to 80% and increase performance up to 400% 197 | - work with files in CSV, JSON, Parquet format 198 | - can be compressed with GZIP or BZIP2 199 | - no sub queries or Join queries 200 | 201 | ## Kinesis 202 | 203 | ### data is automatically replicated to 3 AZ 204 | 205 | - **Kinesis Streams**: low latency streaming ingest at scale 206 | - **Kinesis Analytics**: perform real-time analytics on streams using SQL 207 | - **Kinesis Firehose:** load streams into S3, Redshift, ElasticSearch 208 | 209 | ![Untitled 6](https://user-images.githubusercontent.com/53600644/193906835-6bd4038b-9272-4bca-9d3a-8704b0e5712d.png) 210 | 211 | 212 | ## Kinesis Streams 213 | 214 | - streams are divided in ordered Shards / Partitions 215 | - retentions: 1 (default) - 7 days 216 | - ability to reprocess/replay data 217 | - multiple apps can consume the same stream 218 | - real-time processing with scale of throughput 219 | - one data is inserted in Kinesis, it can't be deleted (immutability) 220 | 221 | ### Shard 222 | 223 | ![Untitled 7](https://user-images.githubusercontent.com/53600644/193906868-88e486f3-cddd-4770-9982-2263582e9b9b.png) 224 | 225 | 226 | - one stream is made of many different shards 227 | - 1MB/s or 1000 messages/s at write PER SHARD 228 | - 2MB/s at read PER SHARD 229 | - billing is per shard provisioned 230 | - batching available or per message calls 231 | - number of shards can change over time (reshard/merge) 232 | - records are ordered **per shard** 233 | 234 | ### Put records 235 | 236 | - data + partition key (message key) ⇒ PutRecord 237 | - partition key to determine shard id 238 | - messages sent get a "sequence number" 239 | - choose a partition key that is highly distributed 240 | - user_id if many users 241 | - **Not** country_id if 90% of the users are in one country 242 | - **Not** date 243 | - use Batching with PutRecords to reduce costs and increase throughput 244 | - **ProvisionedThroughputExceeded** if go over the limits 245 | 246 | ### Consumers 247 | 248 | - can use normal consumer 249 | 250 | - can use Kinesis Client Library 251 | 252 | - read record from a Kinesis Streams with distributed apps sharing the read workload (run on EC2, Elastic Beanstalk, on-premise) 253 | 254 | - each shard is read by only 1 KCL instance 255 | 256 | - means 4 shards = max 4 KCL instances, means 6 shards = max 6 KCL instances 257 | 258 | - progress is check pointed into DynamoDB (need IAM access) 259 | 260 | ![Untitled 8](https://user-images.githubusercontent.com/53600644/193906878-712da3de-39c6-4de5-ae7c-a19ec5389b95.png) 261 | 262 | 263 | 264 | ### Exceptions 265 | 266 | - ProvisionedThroughputExceeded: over the limits of any shard 267 | - make sure don't have a hot shard 268 | - solutions: 269 | - retries with backoff 270 | - increase shards 271 | - ensure good partition key 272 | 273 | ## Kinesis Analytics 274 | 275 | - perform real-time analytics on Kinesis Streams using SQL 276 | - auto scaling 277 | - don't need provision 278 | - can create streams out of the real-time queries (windowed average) 279 | - pay for actual consumption rate 280 | 281 | ## Kinesis Firehose 282 | 283 | - near real time (60s latency) 284 | - load data into Redshift / S3 / ElasticSearch / Splunk 285 | - auto scaling 286 | - don't need provision 287 | - pay for the amount of data going through Firehose 288 | 289 | ## Security 290 | 291 | - control access / authorization using IAM policies 292 | - Encrypt in flight using HTTPS 293 | - Encrypt at rest using KMS 294 | - possible to encrypt / decrypt data client side 295 | - VPC Endpoints available for Kinesis 296 | 297 | ## SQS vs SNS vs Kinesis 298 | 299 | - consumer "pull data" 300 | 301 | - data is deleted after being consumed 302 | 303 | - can have as many workers (consumers) as we want 304 | 305 | - no need to provision throughput 306 | 307 | - no ordering guarantee (except FIFO queues) 308 | 309 | - individual message delay capability 310 | 311 | - push data to many subscribers 312 | 313 | - up to 10,000,000 subscribers 314 | 315 | - data is not persisted (lost if not delivered) 316 | 317 | - Pub/Sub 318 | 319 | - Up to 100,000 topics 320 | 321 | - No need to provision throughput 322 | 323 | - Integrates with SQS for fan-out architecture pattern 324 | 325 | - consumers " pull data" 326 | 327 | - as many consumers as we want (1 shard - 1 consumer) 328 | 329 | - possible to replay data 330 | 331 | - meant for real-time big data, analytics and ETL (IOT) 332 | 333 | - ordering at the shard level 334 | 335 | - data expires after X days 336 | 337 | - must provision throughput 338 | --------------------------------------------------------------------------------