├── EC2 ├── templates │ ├── healthcheck.html │ ├── Build-A-Serverless-Website │ │ ├── error.html │ │ ├── hellocloudgurus.py │ │ └── index.html │ ├── Using-Polly-To-Help-Lab │ │ ├── pollyassets │ │ │ ├── mappings.json │ │ │ ├── sample.json │ │ │ ├── bucketpolicypermissions.json │ │ │ ├── getposts.py │ │ │ ├── lambdapolicy.json │ │ │ ├── newposts.py │ │ │ ├── styles.css │ │ │ ├── scripts.js │ │ │ ├── convertoaudio.py │ │ │ └── index.html │ │ └── Change-to-Lambda-Console.txt │ ├── bootstrap.sh │ ├── index.html │ └── bootstrapscript.sh ├── Exam-tips │ └── README.md └── README.md ├── Databases ├── scripts │ ├── rds.sh │ └── connect.txt ├── Exam-tips │ └── README.md └── README.md ├── Route-53 ├── Exam-Tips │ └── README.md └── README.md ├── LICENSE ├── Application Services ├── Exam Tips │ └── README.md ├── README.md └── kinesis-data-vis-sample-app.template ├── VPC ├── Exam tips │ └── README.md └── README.md ├── Object-Storage-and-CDN-S3-Glacier-Cloudfront ├── CDN-Cloudfront │ └── README.md ├── Snowball │ └── README.md ├── Exam-tips │ └── README.md ├── Storage-Gateway │ └── README.md └── S3-Glacier │ └── README.md ├── IAM └── README.md ├── README.md ├── 1000-ft-overview └── README.md ├── Additional-Exam-Tips └── README.md └── The-Well-Architected-Framework └── README.md /EC2/templates/healthcheck.html: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /EC2/templates/Build-A-Serverless-Website/error.html: -------------------------------------------------------------------------------- 1 |

There has been an error!

-------------------------------------------------------------------------------- /EC2/templates/Using-Polly-To-Help-Lab/pollyassets/mappings.json: -------------------------------------------------------------------------------- 1 | { 2 | "postId" : "$input.params('postId')" 3 | } -------------------------------------------------------------------------------- /EC2/templates/Using-Polly-To-Help-Lab/pollyassets/sample.json: -------------------------------------------------------------------------------- 1 | { 2 | "voice" : "Joanna", 3 | "text" : "Hello Cloud Gurus!" 4 | } -------------------------------------------------------------------------------- /EC2/templates/bootstrap.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | yum install httpd -y 3 | yum update -y 4 | aws s3 cp s3://my-default-bucket-blah/healthcheck.html /var/www/html 5 | service httpd start 6 | chkconfig httpd on -------------------------------------------------------------------------------- /EC2/templates/Using-Polly-To-Help-Lab/Change-to-Lambda-Console.txt: -------------------------------------------------------------------------------- 1 | Change to Console 2 | The Lambda console has changed slightly 3 | 4 | Instead of "Blank Project" select "Author from scratch" 5 | 6 | -------------------------------------------------------------------------------- /Databases/scripts/rds.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | yum install httpd php php-mysql -y 3 | yum update -y 4 | chkconfig httpd on 5 | service httpd start 6 | echo "" > /var/www/html/index.php 7 | cd /var/www/html 8 | wget https://s3.eu-west-2.amazonaws.com/acloudguru-example/connect.php 9 | -------------------------------------------------------------------------------- /EC2/templates/Build-A-Serverless-Website/hellocloudgurus.py: -------------------------------------------------------------------------------- 1 | def lambda_handler(event, context): 2 | print("In lambda handler") 3 | 4 | resp = { 5 | "statusCode": 200, 6 | "headers": { 7 | "Access-Control-Allow-Origin": "*", 8 | }, 9 | "body": "Ryan Kroonenburg" 10 | } 11 | 12 | return resp -------------------------------------------------------------------------------- /EC2/templates/index.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | Hello Cloud Gurus 8 | 9 | 10 |

Hello Cloud Gurus!

11 | 12 | -------------------------------------------------------------------------------- /EC2/templates/Using-Polly-To-Help-Lab/pollyassets/bucketpolicypermissions.json: -------------------------------------------------------------------------------- 1 | { 2 | "Version": "2012-10-17", 3 | "Statement": [ 4 | { 5 | "Sid": "PublicReadGetObject", 6 | "Effect": "Allow", 7 | "Principal": "*", 8 | "Action": [ 9 | "s3:GetObject" 10 | ], 11 | "Resource": [ 12 | "arn:aws:s3:::BUCKET_NAME/*" 13 | ] 14 | } 15 | ] 16 | } -------------------------------------------------------------------------------- /Route-53/Exam-Tips/README.md: -------------------------------------------------------------------------------- 1 | # Exam Tips 2 | 3 | ## DNS 4 | 5 | - ELB's do not have pre-defined IPv4 addresses, you resolve to them using a DNS name 6 | - Understand the difference between an Alias Record and a CNAME. 7 | - Given the choice, always choose and Alias Record over a CNAME. 8 | 9 | Remember the different routing policies and their use cases. 10 | 11 | - Simple 12 | - Weighted 13 | - Latency 14 | - Failover 15 | - Geolocation 16 | -------------------------------------------------------------------------------- /EC2/templates/bootstrapscript.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | yum install httpd php php-mysql -y 3 | cd /var/www/html 4 | echo "healthy" > healthy.html 5 | wget https://wordpress.org/latest.tar.gz 6 | tar -xzf latest.tar.gz 7 | cp -r wordpress/* /var/www/html/ 8 | rm -rf wordpress 9 | rm -rf latest.tar.gz 10 | chmod -R 755 wp-content 11 | chown -R apache:apache wp-content 12 | wget https://s3.amazonaws.com/bucketforwordpresslab-donotdelete/htaccess.txt 13 | mv htaccess.txt .htaccess 14 | chkconfig httpd on -------------------------------------------------------------------------------- /Databases/scripts/connect.txt: -------------------------------------------------------------------------------- 1 | "; 10 | $selected = mysql_select_db("$dbname",$dbhandle) or die("Unable to connect to MySQL DB - check the database name and try again."); 11 | ?> -------------------------------------------------------------------------------- /EC2/templates/Using-Polly-To-Help-Lab/pollyassets/getposts.py: -------------------------------------------------------------------------------- 1 | import boto3 2 | import os 3 | from boto3.dynamodb.conditions import Key, Attr 4 | 5 | def lambda_handler(event, context): 6 | 7 | postId = event["postId"] 8 | 9 | dynamodb = boto3.resource('dynamodb') 10 | table = dynamodb.Table(os.environ['DB_TABLE_NAME']) 11 | 12 | if postId=="*": 13 | items = table.scan() 14 | else: 15 | items = table.query( 16 | KeyConditionExpression=Key('id').eq(postId) 17 | ) 18 | 19 | return items["Items"] -------------------------------------------------------------------------------- /EC2/templates/Using-Polly-To-Help-Lab/pollyassets/lambdapolicy.json: -------------------------------------------------------------------------------- 1 | { 2 | "Version": "2012-10-17", 3 | "Statement": [ 4 | { 5 | "Effect": "Allow", 6 | "Action": [ 7 | "polly:SynthesizeSpeech", 8 | "dynamodb:Query", 9 | "dynamodb:Scan", 10 | "dynamodb:PutItem", 11 | "dynamodb:UpdateItem", 12 | "sns:Publish", 13 | "s3:PutObject", 14 | "s3:PutObjectAcl", 15 | "s3:GetBucketLocation", 16 | "logs:CreateLogGroup", 17 | "logs:CreateLogStream", 18 | "logs:PutLogEvents" 19 | ], 20 | "Resource": [ 21 | "*" 22 | ] 23 | } 24 | ] 25 | } -------------------------------------------------------------------------------- /EC2/templates/Build-A-Serverless-Website/index.html: -------------------------------------------------------------------------------- 1 | 2 | 18 |

19 |

Hello Cloud Gurus!

20 |
21 |

22 | 23 | 24 | -------------------------------------------------------------------------------- /Databases/Exam-tips/README.md: -------------------------------------------------------------------------------- 1 | # Exam Tips 2 | 3 | ## ElastiCache 4 | 5 | You will be given a scenario where a particular database is under a lot of stress/load. You may be asked which service you should use to alleviate this. 6 | 7 | ElastiCache is a good choice if your database is particularly read heavy and not prone to frequent changing. 8 | 9 | Redshift is a good answer if the reason your database is feeling stress is because management keep running OLAP transactions on it etc. 10 | 11 | ## Summary 12 | 13 | ### Types 14 | 15 | - RDS - OLTP 16 | - SQL 17 | - MySQL 18 | - PostgreSQL 19 | - Oracle 20 | - Aurora 21 | - MariaDB 22 | - DynamoDB - NoSQL 23 | - Redshift - OLAP 24 | - Elasticache - In Memory Caching 25 | - Memacached 26 | - Redis 27 | 28 | --- 29 | 30 | #### READ FAQ RDS SECTION IN DOCUMENTATION!! 31 | 32 | [https://aws.amazon.com/rds/faqs/](https://aws.amazon.com/rds/faqs/) -------------------------------------------------------------------------------- /EC2/templates/Using-Polly-To-Help-Lab/pollyassets/newposts.py: -------------------------------------------------------------------------------- 1 | import boto3 2 | import os 3 | import uuid 4 | 5 | def lambda_handler(event, context): 6 | 7 | recordId = str(uuid.uuid4()) 8 | voice = event["voice"] 9 | text = event["text"] 10 | 11 | print('Generating new DynamoDB record, with ID: ' + recordId) 12 | print('Input Text: ' + text) 13 | print('Selected voice: ' + voice) 14 | 15 | #Creating new record in DynamoDB table 16 | dynamodb = boto3.resource('dynamodb') 17 | table = dynamodb.Table(os.environ['DB_TABLE_NAME']) 18 | table.put_item( 19 | Item={ 20 | 'id' : recordId, 21 | 'text' : text, 22 | 'voice' : voice, 23 | 'status' : 'PROCESSING' 24 | } 25 | ) 26 | 27 | #Sending notification about new post to SNS 28 | client = boto3.client('sns') 29 | client.publish( 30 | TopicArn = os.environ['SNS_TOPIC'], 31 | Message = recordId 32 | ) 33 | 34 | return recordId 35 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2018 Nigel Earle 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /EC2/templates/Using-Polly-To-Help-Lab/pollyassets/styles.css: -------------------------------------------------------------------------------- 1 | .buttons { 2 | border : solid 0px #e6b215; 3 | border-radius : 8px; 4 | moz-border-radius : 8px; 5 | font-size : 16px; 6 | color : #ffffff; 7 | padding : 5px 18px; 8 | background-color : #FF9900; 9 | cursor:pointer; 10 | } 11 | 12 | .buttons:hover { 13 | background-color:#ffc477; 14 | } 15 | 16 | .buttons:active { 17 | position:relative; 18 | top:1px; 19 | } 20 | 21 | #newPost { 22 | margin: 0 auto; 23 | width: 90%; 24 | } 25 | 26 | #charCounter { float:right } 27 | 28 | textarea { 29 | width: 100%; 30 | height: 10em; 31 | } 32 | 33 | #content { 34 | width: 90% ; 35 | margin-left: auto ; 36 | margin-right: auto; 37 | margin-bottom: 10px; 38 | font-family:verdana, sans-serif; 39 | word-spacing:4pt; 40 | font-size:14px; 41 | } 42 | 43 | #posts { 44 | font-weight:normal; 45 | color:#000000; 46 | word-spacing:4pt; 47 | font-size:10px; 48 | text-align:left; 49 | font-family:verdana, sans-serif; 50 | width: 90%; 51 | margin: 0 auto; 52 | } 53 | 54 | #posts th { 55 | background-color: #FF9900; 56 | color: white; 57 | padding: 8px; 58 | border-bottom: 1px solid #ddd; 59 | } 60 | 61 | #posts td { 62 | padding: 8px; 63 | border-color: #666666; 64 | background-color: #ffffff; 65 | border-bottom: 1px solid #ddd; 66 | } 67 | -------------------------------------------------------------------------------- /Application Services/Exam Tips/README.md: -------------------------------------------------------------------------------- 1 | # Exam Tips 2 | 3 | ## SQS 4 | 5 | - SQS is a distributed message queuing system 6 | - Allows you to decouple the components of an application so that they are independent 7 | - Pull-based, not push- based 8 | - Standard queues (default) - best effort ordering; message delivered at least once 9 | - FIFO Queues (First In First Out) - ordering strictly preserved, message delivered once, no duplicates. eg. good for banking transactions which need to happen in strict order. 10 | 11 | **NOTE:** Read FAQ section of SQS to help with exam 12 | 13 | ## SNS 14 | 15 | **Subscribers:** 16 | 17 | - HTTP 18 | - HTTPS 19 | - Email 20 | - Email-JSON 21 | - SQS 22 | - Application 23 | - Lambda 24 | 25 | ## API Gateway 26 | 27 | - Remember what API Gateway is at a high level 28 | - API Gateway has caching capabilities to increase performance 29 | - API Gateway is low cost and scales automatically 30 | - You can throttle API Gateway to prevent attacks 31 | - You can log results to CloudWatch 32 | - If you are using JS/AJAX that uses multiple domains with API Gateway, ensure that you have CORS enabled on API Gateway 33 | 34 | ## Kinesis 35 | 36 | - Know the difference between Kinesis Streams and Kinesis Firehose. You will be given scenario questions and you must choose the most relevant service 37 | 38 | - High level understanding of Kinesis Analytics -------------------------------------------------------------------------------- /VPC/Exam tips/README.md: -------------------------------------------------------------------------------- 1 | # Exam Tips 2 | 3 | ## VPC Intro 4 | 5 | - Think of a VPC as a logical data center in AWS 6 | - Consists of IGW(or Virtual Private Gateways), route tables, network access control lists (NACL), subnets, security groups 7 | - 1 subnet = 1 AZ 8 | - Security groups are Stateful; NACLs are Stateless 9 | - Must open both inbound and outbound ports for NACLs 10 | - NO TRANSITIVE PEERING!! 11 | - Allowed 5 VPC's in each AWS Region by default 12 | 13 | ## Flow Logs 14 | 15 | - You cannot enable flow logs for VPCs that are peered with your VPC unless the peer VPC is in your account. 16 | - Cannot tag a flow log 17 | - After you've created a flow log, you cannot change its configuration; for example, you cant associate a different IAM role with the flow log 18 | 19 | **Not all IP traffic is monitored** 20 | 21 | - Traffic generated by instances when they contact the Amazon DNS server. If you use your own DNS server, then all traffic to the DNS server is logged. 22 | - Traffic generated by a Windows instance for Amazon Windows license activation. 23 | - Traffic to and from 169.254.169.254 for instance metadata 24 | - DHCP traffic 25 | - Traffic to the reserved IP address for the default VPC router 26 | 27 | ## NAT vs Bastion 28 | 29 | - A NAT is used to provide internet traffic to EC2 instances in private subnets 30 | - A Bastion is used to securely administer EC2 instances using SSH or RDP 31 | 32 | `bastion host` -> `private server` 33 | -------------------------------------------------------------------------------- /Object-Storage-and-CDN-S3-Glacier-Cloudfront/CDN-Cloudfront/README.md: -------------------------------------------------------------------------------- 1 | # CDN (Content Delivery Network) 2 | 3 | ## What's a CDN? 4 | 5 | A system of distributed servers that deliver webpages and other content to a user based on the geographic locations of that user, the origin of the webpage and a content delivery server 6 | 7 | ## CloudFront 8 | 9 | CloudFront can be used to deliver your entire website, including dynamic content, static, streaming and interactive content using a global network of edge locations. 10 | 11 | Requests for your content are automically routed to the nearest Edge Location, so content is delivered with the best possible performance. 12 | 13 | CloudFront is optimized to work with other Amazon Web Services like S3, EC2, Elastic Load Balancing and Route 53. CloudFront also works seamlessly with any non-AWS origin server which stores the original,definitive versions of your files. 14 | 15 | ## Key Terminology 16 | 17 | - **Edge Location** - Location where content will be cached. Separate to and AWS region (See [1000-ft-overview/Edge-locations](https://github.com/NigelEarle/AWS-CSA-Notes-2018/tree/master/1000-ft-overview#edge-locations)) 18 | - **Origin** - This is the origin of all the files that the CDN will distribute. Can be S3 bucket, EC2 instance, Elastic Load Balancer or Route 53. 19 | - **Distribution** - Given name of CDN which consists of a collection of Edge Locations 20 | - **Web Distribution** - Typically used for websites 21 | - **RTMP (Real Time Message Protocol)** - Used for media streaming 22 | 23 | ## Links 24 | 25 | - [https://aws.amazon.com/cloudfront/](https://aws.amazon.com/cloudfront/) 26 | - [https://aws.amazon.com/cloudfront/details/](https://aws.amazon.com/cloudfront/details/) 27 | -------------------------------------------------------------------------------- /IAM/README.md: -------------------------------------------------------------------------------- 1 | # IAM - Identity Access Management 2 | 3 | ## What is IAM? 4 | 5 | Allow you to manage users and their level of access management to the AWS console. Tested for exam and co. aws account in real life. IAM is globally available and not specified to region 6 | 7 | ## What can you do with IAM? 8 | 9 | - Centralized control of your AWS account 10 | - Shared Access to your AWS account 11 | - Granular permissions 12 | - Identity Federation 13 | - Access to 3rd party service, Active Directory, Facebook, LinkedIn 14 | - Multifactor Authentication (MFA) 15 | - Provide temporary access for users/devices and services where necessary 16 | - Set up and manage password rotation 17 | - Integrates with many different AWS services 18 | - Supports PCI, DSS compliance 19 | 20 | ## Terminology 21 | 22 | - **Users** - End users (people) 23 | - **Groups** - Collection of users under one set of permissions 24 | - **Roles** - Permissions defined for AWS resources (i.e. EC2 etc.) 25 | - **Policy Documents** - Document that defines one or more permissions - JSON format 26 | - **Root account** - user used to sign into AWS account 27 | 28 | ## General Notes 29 | 30 | - Universal. Does not apply to regions at this time. 31 | - Attach permissions to users as well as groups 32 | - New users have NO permissions when first created 33 | - New users are assigned and Access Key ID and Secret Key when first created 34 | - Keys are not the same as passwords 35 | - Must regenerate keys if lost 36 | - ALWAYS setup multifactor auth on root account 37 | - Customize password rotation policies 38 | - Unable to set billing alarm in cloud watch because of new account 39 | 40 | ## Links 41 | 42 | - [https://aws.amazon.com/iam/faqs/](https://aws.amazon.com/iam/faqs/) 43 | - [https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html) 44 | -------------------------------------------------------------------------------- /EC2/templates/Using-Polly-To-Help-Lab/pollyassets/scripts.js: -------------------------------------------------------------------------------- 1 | var API_ENDPOINT = " https://env69m0wnj.execute-api.us-west-1.amazonaws.com/prod" 2 | 3 | document.getElementById("sayButton").onclick = function(){ 4 | 5 | var inputData = { 6 | "voice": $('#voiceSelected option:selected').val(), 7 | "text" : $('#postText').val() 8 | }; 9 | 10 | $.ajax({ 11 | url: API_ENDPOINT, 12 | type: 'POST', 13 | data: JSON.stringify(inputData) , 14 | contentType: 'application/json; charset=utf-8', 15 | success: function (response) { 16 | document.getElementById("postIDreturned").textContent="Post ID: " + response; 17 | }, 18 | error: function () { 19 | alert("error"); 20 | } 21 | }); 22 | } 23 | 24 | 25 | document.getElementById("searchButton").onclick = function(){ 26 | 27 | var postId = $('#postId').val(); 28 | 29 | 30 | $.ajax({ 31 | url: API_ENDPOINT + '?postId='+postId, 32 | type: 'GET', 33 | success: function (response) { 34 | 35 | $('#posts tr').slice(1).remove(); 36 | 37 | jQuery.each(response, function(i,data) { 38 | 39 | var player = "

" 40 | 41 | if (typeof data['url'] === "undefined") { 42 | var player = "" 43 | } 44 | 45 | $("#posts").append(" \ 46 | " + data['id'] + " \ 47 | " + data['voice'] + " \ 48 | " + data['text'] + " \ 49 | " + data['status'] + " \ 50 | " + player + " \ 51 | "); 52 | }); 53 | }, 54 | error: function () { 55 | alert("error"); 56 | } 57 | }); 58 | } 59 | 60 | document.getElementById("postText").onkeyup = function(){ 61 | var length = $(postText).val().length; 62 | document.getElementById("charCounter").textContent="Characters: " + length; 63 | } 64 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # AWS Certified Solutions Architect - Associate Notes 2018 2 | 3 | Curated documentation/study notes on going through the [Udemy, Certified Solutions Archictect - Associate 2018](https://www.udemy.com/aws-certified-solutions-architect-associate/) course. These notes are to help myself, as well as, anyone else going through this same course study and prepare for the exam. Thanks! 4 | 5 | ## My Study Practice 6 | 7 | While planning out my study schedule, I felt it would be necessary to watch one section a week, starting with Section 2: 1000 ft overview, followed up by 6 days of review (1 - 2 hours a day) solidifying everything I learned. This seems to work for me but might be either too slow or fast for someone else. That's OK! Learn at your own pace until you feel comfortable with the concepts, practices and information given! 8 | 9 | ## The Exam Blueprint 10 | 11 | AWS has recently released the updated/new version of the AWS Certified Solutions Architect - Associate exam (released February 2018). The old associate exam will no longer be available starting **August 12, 2018**. 12 | 13 | Let's have a look at the details of the exam... 14 | 15 | ### New Exam 16 | 17 | Generally easier than previous exam. Across 5 different domains. 18 | 19 | | Objective | Weighting | 20 | | ------------- |:-----------------------------------:| 21 | | Design Resilient Architectures | 34% | 22 | | Define Performant Architectures | 24% | 23 | | Specify Secure Applications and Architectures | 26% | 24 | | Design Cost-Optimized Architectures | 10% | 25 | | Define Operationally-Excellent Architectures | 6% | 26 | 27 | _Details about this exam:_ 28 | 29 | - 130 minutes in length 30 | - 65 questions 31 | - $150 USD 32 | - Multiple choice 33 | - Pass mass based on bell curve 34 | - Aim for 70% 35 | - Qualification is valid for 2 years 36 | - Scenario based questions 37 | 38 | Have a look at the [Certified Solutions Architect - Associate homepage](https://aws.amazon.com/certification/certified-solutions-architect-associate/) to get an in-depth look at what to expect for your exam! 39 | -------------------------------------------------------------------------------- /Object-Storage-and-CDN-S3-Glacier-Cloudfront/Snowball/README.md: -------------------------------------------------------------------------------- 1 | # Snowballs 2 | 3 | ## What are the different types of Snowballs available? 4 | 5 | ### Snowball 6 | 7 | Petabyte-scale data transport solution that uses secure appliances to transfer large amounts of data into and out of AWS. 8 | 9 | Using Snowball addresses common challenges with large-scale data transfers including high network costs, long transfer times, and security concerns. 10 | 11 | Transferring data with Snowball is simple, fast, secure and can be as little as 1/5 the cost of high-speed internet. 12 | 13 | 80TB Snowballs are available in all regions. Snowball uses multiple layers of security designed to protect your data including tamper-resistant enclosures, 256-bit encryption, and an industry-standard Trusted Platform Module (TPM) designed to ensure both security and full chain-of-custody of your data. 14 | 15 | Once the data transfer job has been processed and verified, AWS performs a software erase of the Snowball appliance 16 | 17 | ### Snowball Edge 18 | 19 | Snowball Edge is a 100TB data transfer device with on-board storage and compute capabilities. You can use Snowball Edge to move large amounts of data into and out of AWS, as a temporary storage tier for large datasets, or to support local workloads in remote or offline locations. 20 | 21 | Snowball Edge connects to your existing applications and infrastructure using standard storage interfaces, streamlining the data transfer process and minimizing setup and integration. 22 | 23 | Snowball Edge can cluster together to form a local storage tier and process your data on-premises, helping ensure your applications continue to run even when they are not able to access the cloud. 24 | 25 | ### Snowmobile 26 | 27 | Snowmobile is an Exabyte-scale data transfer service used to move EXTREMELY large amounts of data to AWS. 28 | 29 | You can transfer up to 100PB per Snowmobile, a 45ft long ruggedized shipping container, pulled by a semi-truck. 30 | 31 | Snowmobile makes it easy to move massive volumes of data to the cloud, including video libraries, image repositories, or even a complete data center migration. Transferring data with Snowmobile is secure, fast and cost effective. 32 | 33 | ## Links 34 | 35 | - [https://aws.amazon.com/snowball/](https://aws.amazon.com/snowball/) 36 | -------------------------------------------------------------------------------- /EC2/templates/Using-Polly-To-Help-Lab/pollyassets/convertoaudio.py: -------------------------------------------------------------------------------- 1 | import boto3 2 | import os 3 | from contextlib import closing 4 | from boto3.dynamodb.conditions import Key, Attr 5 | 6 | def lambda_handler(event, context): 7 | 8 | postId = event["Records"][0]["Sns"]["Message"] 9 | 10 | print "Text to Speech function. Post ID in DynamoDB: " + postId 11 | 12 | #Retrieving information about the post from DynamoDB table 13 | dynamodb = boto3.resource('dynamodb') 14 | table = dynamodb.Table(os.environ['DB_TABLE_NAME']) 15 | postItem = table.query( 16 | KeyConditionExpression=Key('id').eq(postId) 17 | ) 18 | 19 | 20 | text = postItem["Items"][0]["text"] 21 | voice = postItem["Items"][0]["voice"] 22 | 23 | rest = text 24 | 25 | #Because single invocation of the polly synthesize_speech api can 26 | # transform text with about 1,500 characters, we are dividing the 27 | # post into blocks of approximately 1,000 characters. 28 | textBlocks = [] 29 | while (len(rest) > 1100): 30 | begin = 0 31 | end = rest.find(".", 1000) 32 | 33 | if (end == -1): 34 | end = rest.find(" ", 1000) 35 | 36 | textBlock = rest[begin:end] 37 | rest = rest[end:] 38 | textBlocks.append(textBlock) 39 | textBlocks.append(rest) 40 | 41 | #For each block, invoke Polly API, which will transform text into audio 42 | polly = boto3.client('polly') 43 | for textBlock in textBlocks: 44 | response = polly.synthesize_speech( 45 | OutputFormat='mp3', 46 | Text = textBlock, 47 | VoiceId = voice 48 | ) 49 | 50 | #Save the audio stream returned by Amazon Polly on Lambda's temp 51 | # directory. If there are multiple text blocks, the audio stream 52 | # will be combined into a single file. 53 | if "AudioStream" in response: 54 | with closing(response["AudioStream"]) as stream: 55 | output = os.path.join("/tmp/", postId) 56 | with open(output, "a") as file: 57 | file.write(stream.read()) 58 | 59 | 60 | 61 | s3 = boto3.client('s3') 62 | s3.upload_file('/tmp/' + postId, 63 | os.environ['BUCKET_NAME'], 64 | postId + ".mp3") 65 | s3.put_object_acl(ACL='public-read', 66 | Bucket=os.environ['BUCKET_NAME'], 67 | Key= postId + ".mp3") 68 | 69 | location = s3.get_bucket_location(Bucket=os.environ['BUCKET_NAME']) 70 | region = location['LocationConstraint'] 71 | 72 | if region is None: 73 | url_begining = "https://s3.amazonaws.com/" 74 | else: 75 | url_begining = "https://s3-" + str(region) + ".amazonaws.com/" \ 76 | 77 | url = url_begining \ 78 | + str(os.environ['BUCKET_NAME']) \ 79 | + "/" \ 80 | + str(postId) \ 81 | + ".mp3" 82 | 83 | #Updating the item in DynamoDB 84 | response = table.update_item( 85 | Key={'id':postId}, 86 | UpdateExpression= 87 | "SET #statusAtt = :statusValue, #urlAtt = :urlValue", 88 | ExpressionAttributeValues= 89 | {':statusValue': 'UPDATED', ':urlValue': url}, 90 | ExpressionAttributeNames= 91 | {'#statusAtt': 'status', '#urlAtt': 'url'}, 92 | ) 93 | 94 | return 95 | -------------------------------------------------------------------------------- /Object-Storage-and-CDN-S3-Glacier-Cloudfront/Exam-tips/README.md: -------------------------------------------------------------------------------- 1 | # Exam Tips 2 | 3 | ## S3, Glacier 4 | 5 | ### General 6 | 7 | - S3 is object based, allows you to upload files 8 | - Files can be 0B up to 5TB 9 | - Unlimited storage 10 | - Files are stored in Buckets (folder) 11 | - S3 uses universal namespace. bucket names must unique 12 | - Control access to buckets using either a bucket ACL routing Bucket Policies 13 | - By default, **BUCKETS ARE PRIVATE AND ALL OBJECTS STORED INSIDE THEM ARE PRIVATE** 14 | 15 | ### Reads and Writes 16 | 17 | - Read after Write consistency for PUTS of new objects 18 | - Eventual consistency of overwrite PUTS and DELETES (can take time to propagate) 19 | 20 | ### Storage Class Tiers 21 | 22 | - S3 (normal) - durable, immediately available, frequently used 23 | - S3 IA (infrequent access) - like normal S3 tier but infrequently accessed 24 | - S3 Reduced Redundancy Storage (RRS) - data storage that is easily reproducible, such as thumb nails etc 25 | - Glacier (separate product from S3) - Used to archive data. Low and slow retrieval 26 | 27 | ### Core fundamentals of S3 Object 28 | 29 | - key (name) 30 | - value (data) 31 | - version id 32 | - metadata 33 | - subresources 34 | - ACL 35 | - Torrent 36 | - Object based storage only 37 | - Not installable on apps, DB or OS 38 | - Success uploads will generate HTTP 200 status code 39 | - Read S3 FAQ before taking the exam. it comes up a lot 40 | 41 | ### Encryption 42 | 43 | - Client side encryption 44 | - Server side encryption 45 | - encryption with amazon s3 managed keys (SSE-S3) 46 | - encryption with KMS (SSE-KMS) 47 | - encryption with Customer Provided Keys (SSE-C) 48 | 49 | ### Versioning 50 | 51 | - Stores all version of an object (all writes/updates and even if you delete the object). Must manually delete object if you wish to delete a version 52 | - Great back up tool 53 | - Once enabled, cannot be disabled, only suspended 54 | - Integrates with Lifecycle rules 55 | - Versioning MFA Delete capability, uses mulit-factor authentication, can be used to provide an additional layer of security 56 | 57 | ### Cross Region Replication 58 | 59 | - Versioning must be enabled on source and destination buckets 60 | - Regions must be unique, Cannot cross region to same region 61 | - Files are not replicated automatically. All subsequent updated files will be replicated automatically. 62 | - You cannot replicate to multiple buckets - daisy chaining (currently). 63 | - Delete markers are replicated 64 | - Deleting individual versions or delete markers will not be replicated 65 | - Understand what CRR at high level 66 | 67 | ### Lifecycle management 68 | 69 | - Can be used with or without versioning 70 | - Can be applied to current version as well as previous versions 71 | - Acceptable actions 72 | - Transition to Standard - IA Storage Class (128kb and 30 days after creation date) 73 | - Archive to Glacier - 30 days after IA Storage if relevant 74 | - Permanently delete 75 | - Understand at high level 76 | 77 | ## CDN Cloudfront 78 | 79 | - Edge Location - Location where content will be cached - separate from AWS Region 80 | - Origin - Origin of all files the the CDN will distribute. Can be S3, EC2, Elastic Load Balancer, Route 53 or your own custom server. 81 | - Distribution - Name given to the CDN which consists of a collection of Edge Locations 82 | - Web Distribution - Typically used for websites 83 | - RTMP - Used for media streaming 84 | - Edge Locations are not just for READ only, you can write (PUT) too! 85 | - Object are cached for life of TTL (Time To Live) 86 | - Can clear cached objects, but you will be charged 87 | 88 | ## Storage Gateway 89 | 90 | - File Gateway - For flat files, stored directly on S3. 91 | - Volume Gateway: 92 | - Stored Volumes - Entire dataset is stored on site and is asynchronously backed up to S3 93 | - Cached Volumes - Entire dataset is stored on S3 and the most frequent accessed data is cached on site. 94 | - Gateway Virtual Tape Library 95 | - Used for backup and uses popular backup applications like NetBackup, Backup Exec, Veeam etc. 96 | 97 | ## Snowballs 98 | 99 | - Understand what a Snowball is 100 | - Understand what Import Export is 101 | - Snowball can 102 | - Import to S3 103 | - Export from S3 -------------------------------------------------------------------------------- /EC2/templates/Using-Polly-To-Help-Lab/pollyassets/index.html: -------------------------------------------------------------------------------- 1 | 2 | A Cloud Guru - Polly Study Notes Generator 3 | 4 | 5 | 6 |

Hello Cloud Gurus!

7 | 8 |

9 | 10 | Voice: 11 | 12 | 59 | 60 | 61 | 62 | 63 |

64 | 65 | 66 | Characters: 0 67 | 68 |

69 | Provide post ID which you want to retrieve: 70 |
71 |

72 | 73 | 74 | 75 | 76 | 77 | 78 | 79 | 80 | 81 | 82 | 83 | 84 | 85 | 86 | 87 | 88 | 89 | 90 | 91 | 92 |

Post ID	Voice	Post	Status	Player

93 | 94 | 95 | 96 | 97 | 98 | 99 | -------------------------------------------------------------------------------- /Object-Storage-and-CDN-S3-Glacier-Cloudfront/Storage-Gateway/README.md: -------------------------------------------------------------------------------- 1 | # Storage Gateway 2 | 3 | **_Understand at theoretical level_** 4 | 5 | ## What is Storage Gateway? 6 | 7 | A Service that connects an on-premise software appliance with cloud based storage to provide seamless and secure integration between an organization's on-premise IT environment and AWS's storage infrastructure. 8 | 9 | The service enables you to securely store data to AWS cloud for scalable and cost-effective storage. Replicates your data to specifically S3 bucket. 10 | 11 | Downloaded as virtual machine (VM) that you install on a host in your datacenter. Storage Gateway supports either VMware ESXi or MS Hyper-V. Once you've installed your gateway and associate with AWS account through activation process, you can use the AWS Management Console to create the storage gateway option this is right for you. 12 | 13 | ## Four Types of Gateway Storage 14 | 15 | ### File Gateway (NFS) 16 | 17 | Store flat files in S3 through a Network File System (NFS) mount point. Ownership, permissions, and timestamps are durably stored in S3 in the user-metadata of the object associated with the file. 18 | 19 | Once objects are transferred to S3, they can be managed as native S3 objects, and bucket policies such as versioning, lifecycle management, and cross-region replication apply directly to objects stored in your bucket. 20 | 21 | ### Volumes Gateway (iSCSI) 22 | 23 | The volume interface presents your applications with disk volumes using the iSCSI block protocol. 24 | 25 | Data written to these volumes can be asynchronously backed up as point-in-time snapshots of your volumes, and stored in the cloud as AWS EBS (Elastic Block Store - VM) snapshots. 26 | 27 | Snapshots are incremental backups that capture only the changed blocks. All snapshot storage is also compressed to minimize your storage charges. 28 | 29 | _NOTE: iSCSI is block based storage. Store OS, DB's. Think of as virtual hard disk_ 30 | 31 | #### Stored Volumes 32 | 33 | Stored volumes let you store your primary data locally, while asynchronously backing up that data to AWS. Stored volumes provide your on-premise applications with low-latency access to their entire datasets, while providing durable, off-site backups. 34 | 35 | You can create storage volumes and mount them as iSCSI devices from your on-premises application servers. Data written to your stored volumes is stored on your on-premises storage hardware. 36 | 37 | This data is asynchronously backed up to S3 in the form of AWS EBS (Elastic Block Store) snapshots 1 GB - 16 TB in size for Stored Volumes. 38 | 39 | #### Cached Volumes 40 | 41 | Cached volumes let you use S3 as your primary data storage while retaining frequently accessed data locally in your storage gateway. 42 | 43 | Cached volumes minimize the need to scale your on-premise storage infrastructure, while still providing your applications with low-latency access to their frequently accessed data. 44 | 45 | You can create storage volumes up to 32Tb in size and attach to them as iSCSI devices from your on-premises application servers. Your gateway stores data that you write to these volumes in S3 and retains recently read data in your on-premises storage gateways cache and upload buffer storage. 1 GB - 32 TB size for cached volumes. 46 | 47 | **_TLDR;Volume Gateway takes virtual hard disks that are on premise and back them up to AWS_** 48 | 49 | ### Tape Gateway (VTL) 50 | 51 | Offers a durable, cost-effective solution to archive your data in AWS cloud. The VTL interface it provides lets you leverage your existing tape-based backup application infrastructure to store data on virtual tape cartridges that you create on your tape gateway. 52 | 53 | Each tape gateway is preconfigured with a media changer and tape drivers, which are available to your existing client backup applications as iSCSI devices. You add tape cartridges as you need to archive your data. Supported by Netbackup, Backup Exec, Veeam etc. 54 | 55 | ## Tips (Summary) 56 | 57 | - File Gateway - For flat files, stored directly to S3 58 | - Volume Gateway: 59 | - Stored Volumes - Entire dataset is stored on site and is asynchronously backed up to S3 60 | - Cached Volumes - Entire dataset is stored on S3 and the most frequently accessed data is cached on site. 61 | - Gateway Virtual Tape Library (VTL) 62 | - Used for backup and uses popular backup applications like NetBackup, Backup Exec, Veeam etc. 63 | 64 | ## Links 65 | 66 | - [https://aws.amazon.com/storagegateway/faqs/](https://aws.amazon.com/storagegateway/faqs/) 67 | - [https://aws.amazon.com/blogs/aws/the-aws-storage-gateway-integrate-your-existing-on-premises-applications-with-aws-cloud-storage/](https://aws.amazon.com/blogs/aws/the-aws-storage-gateway-integrate-your-existing-on-premises-applications-with-aws-cloud-storage/) 68 | -------------------------------------------------------------------------------- /VPC/README.md: -------------------------------------------------------------------------------- 1 | # VPC - Virtual Private Cloud 2 | 3 | Think of VPC as virtual data center in the cloud! 4 | 5 | ## VPC Definition 6 | 7 | Amazon Virtual Private Cloud (Amazon VPC) lets you provision a logically isolated section of the AWS Cloud where you can launch AWS resources in a virtual network that you define. 8 | 9 | You have complete control over you virtual network environment, including selection of your own IP address range, creation of subnets and config of route tables and network gateways. 10 | 11 | You can easily customize the network config for your VPC. For example, you can create a public facing subnet of your webservers that has access to the internet, and place your backend systems such as databases or application servers in a private-facing subnet with no internet access. 12 | 13 | You can leverage multiple layers of security, including security groups and network access control lists, to help control access to EC2 instances on each subnet. 14 | 15 | Additionally, you can create a Hardware Virtual Private Network (VPN) connection between your corporate datacenter and your VPC and leverage the AWS cloud as an extension of your corporate datacenter. 16 | 17 | **NOTE:** Private and public subnets within a VPC can only have one subnet per AZ 18 | 19 | Use (cidr.xyz)[https://cidr.xyz/] to figure out subnet ranges within a VPC 20 | 21 | ### What can you do with a VPC? 22 | 23 | - Launch instances into a subnet of your choosing 24 | - Assign custom IP address ranges in each subnet 25 | - Configure route tables between subnets 26 | - Create single internet gateway and attach it to our VPC 27 | - Much better security control over your AWS resources 28 | - Instance security groups 29 | - Subnet network access control lists (ACLS) 30 | 31 | ### Default VPC vs Private VPC 32 | 33 | - Default VPC is user friendly, allowing you to immediately deploy instances 34 | - All subnets have a route to internet 35 | - No private subnets in default VPC 36 | - EC2 instance has both a public and private IP address. 37 | 38 | ### VPC Peering 39 | 40 | - Allows you to connect one VPC with another via a direct network route using private IP addresses 41 | - Instances behave as if they are on the same private network. 42 | - You can peer VPCs with other AWS accounts as well as with other VPCs in the same account 43 | - Peering is in a star config: ie 1 central VPC peers with 4 others. NO TRANSITIVE PEERING!! 44 | 45 | ## NAT - Network Address Translation 46 | 47 | ### NAT Instances 48 | 49 | - When creating a NAT instance, Disable Source/Destination Check on the instance. 50 | - NAT instances must be in a public subnet 51 | - There must be a route out of the private subnet to the NAT, in order for this to work. 52 | - The amount of traffic that NAT instances can support depends on the instance size. If you are bottlenecking, increase the instance size. 53 | - You can create high availability using Autoscaling Groups, multiple subnets in different AZs, and a script to automate failover. 54 | - Behind a Security Group. 55 | 56 | ### NAT Gateways 57 | 58 | - Preferred by the enterprise 59 | - Scale automatically up to 10G 60 | - No need to patch OS 61 | - Not associated with security groups 62 | - Automatically assigned public IP 63 | - Must update root tables and point them to NAT Gateway 64 | - Having one NAT Gateway in one AZ is not good enough, must me redundant in multiple AZs 65 | - No need to disable Source/Destination Checks 66 | - More Secure than NAT Instance 67 | 68 | ## NACL - Network Access Control Lists 69 | 70 | - Can only associate **1** subnet to a Network ACLs 71 | - Your VPC automatically comes with a default NACL, and by default it allows all inbound and outbound and traffic. 72 | - You can create custom NACLs. By default each custom NACL denies all inbound and outbound traffic until you add rules 73 | - Each subnet in your VPC must be associated with a network ACL. If you don't explicitly associate a subnet with a network ACL, the subnet is automatically associated with the default NACL. 74 | - You can associate a NACL with multiple subnets; however, a subnet can be associated with only one network ACL at a time. When you associate a NACL with a subnet, the previous association is removed. 75 | - NACLs contain a numbered list of rules that is evaluated in order, starting with the lowest numbered rule 76 | - NACL have separate inbound and outbound rules, and each rule can either allow or deny traffic. 77 | - NACL are stateless; responses to allowed inbound traffic are subject to the rules for outbund traffic 78 | 79 | ## ALB 80 | 81 | - You need at least 2 public subnets in order to deeply and application load balancer 82 | 83 | ## VPC Flow Logs 84 | 85 | VPC Flow Logs is a feature that enables you to capture info about the IP traffic going to and from network interfaces in your VPC. Flow log data is stored using Amazon Cloudwatch Logs. 86 | 87 | After you've created a flow log, you can view and retrieve its data in Amazon CloudWatch Logs. 88 | 89 | Flow logs can be create at 3 levels: 90 | 91 | - VPC 92 | - Subnet 93 | - Network Interface Level 94 | -------------------------------------------------------------------------------- /Object-Storage-and-CDN-S3-Glacier-Cloudfront/S3-Glacier/README.md: -------------------------------------------------------------------------------- 1 | # S3 - HEAVY EXAM TOPIC 2 | 3 | S3 is a safe place to store your static files being one the oldest services of AWS. It is an object based storage where your data is spread across multiple devices. 4 | 5 | S3 allows you to upload, where files can be from 0 bytes to 5TB. If an upload is successful, you will receive an HTTP status code of `200`. 6 | It is capable of unlimited storage. All files are stored into 'Buckets' which is basically an S3 term for folders. 7 | 8 | S3 uses a universal namespace meaning all names must be **_globally_** unique. 9 | 10 | _Example S3 URL:_ 11 | 12 | **`https://s3-eu-west-1.amazonaws.com/[bucket-name]`** 13 | 14 | ## Data Consistency 15 | 16 | S3 maintains **_Read After Write_** consistency for PUTS of new objects. Meaning, as soon a new object is uploaded or written, it is available to read/view. 17 | 18 | When performing overwrite PUTS and DELETES, these updated and/or deleted objects can take time to propagate because, also known as **_Eventual Consistency_**. These type of updates are known as **_Atomic_** - fetching these resources could be old or new. 19 | 20 | ## S3 Object - Key, Value Store 21 | 22 | - Key - Name of object to be stored 23 | - Value - Data being stored - made up of a sequence of bytes 24 | - Version ID - Version signifier 25 | - Metadata - Data about the data you are storing - date stored, size, 26 | - Subresource 27 | - Access Control Lists 28 | - Torrents 29 | 30 | ## S3 Basics 31 | 32 | - Built for 99.99% availability for the S3 platform 33 | - Amazon guarantee 99.9% availability - always available 34 | - Amazon guarantees 99.99999999999% (11, 9’s) durability for S3 information 35 | - Tiered storage 36 | - Lifecycle management 37 | - Versioning 38 | - Encryption 39 | - Secure data using Access Control Lists bucket policies 40 | 41 | ### Storage Tiers 42 | 43 | - **S3 (Normal)** 44 | - 99.99% availability, 99.(11 9’s ) 45 | - durable, reliable - stored redundantly across multiple devices in multiple facilities and is designed to sustain the loss of 2 facilities concurrently 46 | 47 | - **S3 IA (Infrequent Access)** 48 | - Used for data that is accessed less frequently but requires rapid access when needed 49 | - Lower fee than S3 but, are charged a retrieval fee 50 | 51 | - **S3 Reduces Redundancy Storage (RRS)** 52 | - Designed to provide 99.99% durability and 99.99% availability of objects over a given year. 53 | 54 | - **Glacier (Separate product from S3)** 55 | - Very cost effective but used for data archival only 56 | - Generally takes 3 - 5 hours to restore from glacier 57 | - Stores data for as low as .01G a month 58 | - Optimized for data that is infrequently accessed and for which retrieval times of 3 to 5 hours are suitable (slow retrieval). 59 | 60 | ### S3 Charges 61 | 62 | - Storage 63 | - Requests 64 | - Storage Management Pricing 65 | - Data transfer pricing 66 | - Transfer Acceleration 67 | 68 | #### Transfer Acceleration 69 | 70 | - Enables fast, easy and secure transfers of files over long distances between you and your end users and an S3 bucket. 71 | - Takes advantage of AWS CloudFront global, distributed edge locations. 72 | - When data arrives at an edge location, it is then routed to Amazon S3 over an optimized network path. 73 | 74 | ## S3 Encryption and Security 75 | 76 | By default all newly created buckets are **PRIVATE**. You need to manually change permissions to access resources. 77 | 78 | You can set policies and permissions using either Access Control Lists or Bucket Policies. 79 | 80 | You have the ability to make a bucket private but all certain objects in that bucket to be public. 81 | 82 | ### Logging 83 | 84 | S3 buckets can be configured to create access logs which log all requests made to that bucket. This can be done to another bucket through cross account access. 85 | 86 | ### Encryption 87 | 88 | **4** different methods and **2** types of encryption for S3 buckets. 89 | 90 | 1. **In Transit** - from client uploading to S3 bucket. 91 | - Using SSL/TLS encryption. HTTPS 92 | 93 | 2. **At Rest** 94 | - Server Side Encryption 95 | - **SSE-S3** - S3 Managed key. Each object is encrypted with a unique key employing strong multi-factor encryption with rotating master key (AES-256 encryption). 96 | - **SSE KMS** - AWS Key Management Service, Managed Keys. Similar to SSE-S3. Separate permissions for envelope key - key that protects data encryption key. Audit trail - when keys were used and who were using. 97 | - **SSE-C** - Server Side Encryption with Customer Provided Keys. You manage encryption key. 98 | - Client Side Encryption 99 | - Encrypt data on client side and upload to S3 100 | 101 | ## Links 102 | 103 | - [https://aws.amazon.com/s3/](https://aws.amazon.com/s3/) 104 | - [https://docs.aws.amazon.com/AmazonS3/latest/dev/Welcome.html](https://docs.aws.amazon.com/AmazonS3/latest/dev/Welcome.html) 105 | - [https://aws.amazon.com/s3/faqs/](https://aws.amazon.com/s3/faqs/) 106 | - [https://aws.amazon.com/s3/storage-classes/](https://aws.amazon.com/s3/storage-classes/) 107 | - [https://aws.amazon.com/glacier/faqs/](https://aws.amazon.com/glacier/faqs/) -------------------------------------------------------------------------------- /Route-53/README.md: -------------------------------------------------------------------------------- 1 | # Route 53 2 | 3 | ## DNS 4 | 5 | ### What is DNS? (Domain Name Service) 6 | 7 | If you've used the internet, you've used DNS. DNS is used to convert human friendly domain names `(http://acloud.guru)` into an Internet Protocol (IP) address `(http://92.123.92.1)` 8 | 9 | IP addresses are used by computers to identify eachother on the network. IP addresses commonly come in 2 different forms, **IPv4** and **IPv6** 10 | 11 | ### IPv4 vs IPv6 12 | 13 | The IPv4 space is 32 bit field and has over 4 billion different addresses (4,294,967,296) 14 | 15 | IPv6 was created to solve this the depletion issue and has an address space of 128 bits - which is in theory **340,282,366,920,938,463,463,374,607,431,768,211,456** different addresses! _340 undecillion addresses_ 16 | 17 | ### Top Level Domains 18 | 19 | If common domain names such as google.com, bbc.co.uk etc. you'll notice a string of characters separated but a `.`. The last work in the domain name represents the 'Top Level Domain'. The second word in the domain, known as the 'Second Level Domain' is optional 20 | 21 | **_Example Top Level and Second Level Domains:_** 22 | 23 | ``` 24 | .com 25 | .edu 26 | .gov 27 | .org 28 | .co 29 | .co.uk 30 | .gov.au 31 | ``` 32 | 33 | These top level domains are controlled by the Internet Assigned Numbers Authority (IANA) in a root zone database _(database of all available top level domains)_. You can view this database by going to https://www.iana.org/domains/root/db 34 | 35 | ### Domain Registrars 36 | 37 | Because all the names in a given domain have to be unique there needs to be a way to organize all of this so that domains are duplicated - hence **Domain Registrars**. 38 | 39 | A registrar is an authority that can assign domain names directly under one or more top level domains. Domains are registered with InterNIC, a service of ICANN, which enforces uniqueness of domain names across the Internet. Each domain name becomes registered in a central database known as the WhoIS database. 40 | 41 | ### SOA Records 42 | 43 | SOA Records store information related to a domain about: 44 | 45 | - The name of the server that supplied data for that zone. 46 | - The admin of that zone. 47 | - The current version of the datafile. 48 | - The number of seconds a secondary name server should wait before checking for updates. 49 | - The number of seconds a secondary name server should wait before retrying a failed zone transfer. 50 | - The maximum number of seconds that secondary name server can use data before it must either be refreshed or expire. 51 | - The default number of seconds for the TTL file on resource records. 52 | 53 | ### NS Records 54 | 55 | NS stands for Name Server records and are used by top level domain servers to direct traffic to the Content DNS server which contains the authoritative DNS records. 56 | 57 | ### A Records 58 | 59 | An A Record is the fundamental type of DNS record and the 'A' in A record stands for 'Address'. The A Record is used by the computer to translate the name of the domain to the IP address. For example `https://google.com` -> `https://92.123.12.1` 60 | 61 | ### TTL 62 | 63 | The length that a DNS record is cached on eitherthe Resolving Server o the users own local PC is equal to the value of the 'Time To Live' _(TTL)_ in seconds. The lower the time to live, the faster changes to DNS records take to propagate throughout the internet. 64 | 65 | ### CNAMES 66 | 67 | A Canonical Name (CName) can be used to resolve one domain name to another. For example, you may have a mobile website with a domain name `http://m.acloud.guru` that is used for when users browse to your domain name on their mobile devices. You may also want the name `http://mobile.acloud.guru` to resolve to this same address. 68 | 69 | ### Alias Records 70 | 71 | Alias resource record sets can save you time because AWS Route 53 automatically recognizes changes in the record sets that the alias resource record set refers to. 72 | 73 | For example, suppose an alias resource record set for example.com points to an ELB load balancer at lb1-1234.us-west-1.elb.amazonaws.com. If the IP address of the load balancer change, AWS Route 53 will automatically reflect those changes in DNS answers for example.com whout any changes to the hosted zone that contains resource record sets for example.com 74 | 75 | ## Routing Policies 76 | 77 | ### Simple 78 | 79 | This is the default routing policy when you create a new record set. This is the most commonly used when you have a single resource that performs a given function for your domain, for example, one web server that serves content for the `http://acloud.guru` website. 80 | 81 | ### Weighted 82 | 83 | Weighted Routing Policies let you split your traffic based on different weights assigned. 84 | For example you can set 10% of your traffic to go to US-EAST-1 and 90% to go to EU-WEST-1 85 | 86 | ### Latency 87 | 88 | Latency based routing allows you to route your traffic based on the lowest network latency for your end user (ie which region will give them the fastest response time) 89 | 90 | To use latency-based routing you create a latency resource record set for the EC2 (or ELB) resource in each region that hosts your website. When Route 53 receives a query for your site, it selects the latency resource record set for the region that gives the user the lowest latency. Route 53 then responds with the value associated with that resource record set 91 | 92 | ### Failover 93 | 94 | Failover routing policies are used when you want to create an active/passive set up. For example you may want your primary site to be in EU-WEST-2 and your secondary DR site in AP-SOUTHEAST-2 95 | 96 | Route 53 will monitor the health of your primary site using a health check. 97 | 98 | A health check monitors the health of your endpoints. 99 | 100 | ### Geolocation 101 | 102 | Geolocation routing lets you choose where your traffic will be sent based on the geographic location of your users (ie the location from which DNS queries originate). 103 | 104 | For example, you might want all queries from Europe to be routed to a fleet of EC2 instances that are specifically configured for your European customers. These servers may have the local language of your European customers and all prices are displayed in Euros. 105 | -------------------------------------------------------------------------------- /1000-ft-overview/README.md: -------------------------------------------------------------------------------- 1 | # Section 2: 1,000 ft Overview 2 | 3 | ## Part 1. Regions, Availability Zones (AZ), Edge Locations 4 | 5 | ### Regions 6 | 7 | **AWS Region** is a physical, geographical area or location, consisting of 2 or more Availability Zones. 8 | 9 | **_Current regions across the world:_** 10 | 11 | - US East (N. Virginia) - `us-east-1` 12 | - US East (Ohio) - `us-east-2` 13 | - US West (Northern California) - `us-west-1` 14 | - US West (Oregon) - `us-west-2` 15 | - Canada (Central) - `ca-central-1` 16 | - EU (Frankfurt) - `eu-central-1` 17 | - EU (Ireland) - `eu-west-1` 18 | - EU (London) - `eu-west-2` 19 | - EU (Paris) - `eu-west-3` 20 | - Asia Pacific (Tokyo) - `ap-northeast-1` 21 | - Asia Pacific (Seoul) - `ap-northeast-2` 22 | - Asia Pacific (Osaka-Local) - `ap-northeast-3` 23 | - Asia Pacific (Singapore) - `ap-southeast-1` 24 | - Asia Pacific (Sydney) - `ap-southeast-2` 25 | - Asia Pacific (Mumbai) - `ap-south-1` 26 | - South America (Sao Paulo) - `sa-east-1` 27 | 28 | ### Availability Zones (AZ) 29 | 30 | **AWS Availability Zones** are one or more discrete data centers, each with redundant power, networking and connectivity housed in separate facilities. Deploying your application across multiple Availability Zones is useful for redundancy, low latency and fault tolerance. 31 | 32 | **_Regions with multiple Availability Zones:_** 33 | 34 | - US East 35 | - Ohio (3) 36 | - North Virginia (6) 37 | - US West 38 | - Oregon (3) 39 | - Northern California (3) 40 | - Canada 41 | - Central (3) 42 | - South America 43 | - Sao Paulo (3) 44 | - Europe 45 | - Ireland (3) 46 | - Frankfurt (3) 47 | - London (3) 48 | - Paris (3) 49 | - Asia Pacific 50 | - Singapore (3) 51 | - Seoul (2) 52 | - Tokyo (4) 53 | - Mumbai (2) 54 | - Sydney (3) 55 | - Beijing (2) 56 | - Ningxia (2) 57 | 58 | ### Edge Locations 59 | 60 | **AWS Edge Locations** are locations around the world meant for caching content, enhancing the user experience, reducing latency. Edge locations are specifically used by AWS Cloudfront and AWS CDN. Every Region is has its own set Availability Zone's and Edge Locations. 61 | 62 | ## Part 2. AWS Services Overview 63 | 64 | Compute: 65 | EC2 - elastic compute cloud 66 | EC2 Container Services - containerization docker 67 | Elastic Beanstalk - plug and play - for developers 68 | Lambda (server less) - code/functions uploaded to the cloud to run at different points 69 | Lightsail - plug and play 70 | Batch - batch computing in the cloud 71 | 72 | Storage: 73 | S3 - simple storage service - object based storage - buckets 74 | EFS - elastic file system 75 | Glacier - data archival 76 | Snowball - large amounts of data to aws data center 77 | Storage gateway - VM installed in datacenter or office - replicate info to S3 78 | 79 | Databases: 80 | RDS - relation database service - postgres, mysql, oracle 81 | DynamoDB - non relational db 82 | Elasticache - cache things from db 83 | Redshift - data warehousing business intelligence, complex queries 84 | 85 | Migration: 86 | AWS Migration Hub - tracking service for moving to aws 87 | Application Discover Service - track applications and dependency 88 | Database Migration Service - migrate db from on premise to AWS 89 | Server Migration Service - migrate server to AWS cloud 90 | Snowball - in between storage and migration 91 | 92 | Networking and Content Delivery: 93 | VPC (highlight) - Amazon virtual private cloud - virtual datacenter - configure avail zones, firewall, network acl etc. 94 | Cloudfront - AWS content delivery network, store assets specific regions around the world 95 | Route 53 - AWS DNS service - lookup ip to get ipv4 and ipv6 address 96 | API Gateway - Serverless way of creating own api 97 | Direct Connect - Dedicated line from office directly into amazon, connects to VPC 98 | 99 | Developer Tools: 100 | Codestart - project management, CI toolchain, collaborate 101 | Codecommit - store code, like github 102 | Codebuild - compile and run tests, produce package 103 | Code deploy - deployment service to ec2 instance 104 | Codepipeline - automate and visualize steps to release software 105 | X-ray - debug and analyze server less application 106 | Cloud9 - IDE environment in browser 107 | 108 | ## Part 3. AWS Services Overview (Continued) 109 | 110 | Management tools: 111 | Cloudwatch - Monitoring service 112 | Cloudformation - solutions architect specific - scripting infrastructure - turn infrastructure to code 113 | Cloudtrail - log changes to aws environment 114 | Config - monitors config of aws environment 115 | Opswork - similar to elastic beanstalk - chef and puppet to automate environments 116 | Service Catalog - manage a catalog of IT services 117 | Systems manager - interface for managing aws resources - group resources 118 | Trusted Advisor - advice around security, advice for aws services and resources, accountant like 119 | Managed Services - manage service for aws cloud 120 | 121 | ** Recap for exam - cloudformation, cloudtrail, cloudtrail, trusted advisor 122 | 123 | Media Services: 124 | Elastic transcoder - takes media and resizes on different devices 125 | Media convert - file based video transcoding with broadcast grade features 126 | Media live - broadcast grade live video processing service. tv internet connected multiscreen 127 | Media Package - protect content over internet 128 | Media Store - media storage, optimized for media 129 | Media Tailor - target advertising into video streams with out harming broadcast 130 | 131 | Machine Learning: 132 | Sage maker - easy for deep learning when coding for environment 133 | Comprehend - sentiment analysis on products. good or bad? 134 | Deep lens - computer vision on camera, recognition, physical piece of hardware 135 | Lex - powers alexa, AI 136 | Machine Learning - throw dataset to AWS cloud and predict outcome 137 | Polly - text to speech, voices sound real, accents 138 | Rekognition - upload file, tells you what is in the file 139 | Amazon translate - translate to other langs 140 | Amazon transcribe - hard of hearing, speech recognition, speech to text 141 | 142 | Analytics: 143 | Athena - SQL queries ins S3 buckets, serverless 144 | EMR - elastic map reduce - processing large amounts of data, chops data up for analysis 145 | Cloudsearch - search service 146 | Elastic Search service - search service 147 | Kinesis - solutions architect highlight, ingesting large amounts data 148 | Kinesis Video streams - ingesting streams and analyze 149 | Quicksight - business intelligence tool 150 | Datapipeline - moving data between different services 151 | Glue - ETL (extract transform load) 152 | 153 | ## Part 4. AWS Services Overview (Continued) 154 | 155 | Security Identity and Compliance: 156 | IAM - identity access management 157 | Cognito - device authentication, oath, after authenticated, use aws services 158 | Guard Duty - monitor for malicious activity 159 | Inspector - install on vm or instances, test against it, schedule 160 | Macie - Scan s3 buckets and looks for sensitive info and alert 161 | Certificate Manager - ssl cert for free, manage ssl cert 162 | Cloud HSM - cloud hardware security module - dedicate bits of hardware to store keys to authenticated 163 | Directory Service - integration ms active service to aws services 164 | WAF - web application firewall - at application layer to stop attacks, XSS, sql injection 165 | Shield - by default for cloud front - ddos mitigation, prevent ddos attacks 166 | Artifact - portal to download aws client reports, manage agreements 167 | 168 | **Key security services for exam: IAM, inspector, cloudHMS, directory services, waf, shield, cert manager 169 | 170 | Mobile Services: 171 | Mobile hub - management console for mobile app for aws services 172 | AWS Pinpoint - targeted push notifications 173 | AWS Appsync - atomically updates data in web or mobile in real time 174 | Device Farm - test apps on real device, iOS, android 175 | Mobile Analytics - analytics service for mobile 176 | 177 | AR/VR: 178 | Sumerian - tools to create environment, super new 179 | 180 | Application Integration: 181 | Step functions - manage lambda functions and ways to go through it 182 | Amazon MQ - message queue 183 | SNS - notification services 184 | SQS - decouple infrastructure, queue 185 | SWF - workflow job creation 186 | 187 | Customer Engagement: 188 | Connect - contact center as a service, call center 189 | Simple Email Service - email service, send grid, mailchimp 190 | 191 | Business Productivity: 192 | Alexa for business - manager for business needs 193 | Amazon chime - google hangouts like 194 | Work Docs - dropbox for AWS 195 | Work Mail - Office 365 like 196 | 197 | Desktop and App streaming: 198 | Workspaces - VDI solution, run OS in aws cloud 199 | App stream 2.0 - streaming application to desktop of device 200 | 201 | IOT: 202 | iOT - devices sending sensor information 203 | iOT Device Management - device management 204 | Amazon FreeRTOS - OS for microcontrollers 205 | Greengrass - ?? 206 | 207 | Game Development: 208 | Gamelift - service to develop games 209 | 210 | ## What Services Will Be Tested On The Exam?? 211 | 212 | Analytics 213 | Management Tools 214 | Migration 215 | Compute 216 | AWS Global infrastructure 217 | Storage 218 | Databases 219 | Network and Content delivery 220 | Security and Identity compliance 221 | Application Integration 222 | Desktop and App streaming 223 | 224 | ## Links 225 | 226 | - [https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html) 227 | 228 | - [https://www.linuxnix.com/amazon-aws-regions-vs-availability-zones-vs-edge-locations-vs-data-centers/](https://www.linuxnix.com/amazon-aws-regions-vs-availability-zones-vs-edge-locations-vs-data-centers/) -------------------------------------------------------------------------------- /EC2/Exam-tips/README.md: -------------------------------------------------------------------------------- 1 | # Exam Tips 2 | 3 | ## EC2 Instance Run Down 4 | 5 | - **On Demand** - allows you to pay a fixed rate by the hour (or second) with not commitment 6 | 7 | - **Reserved** - provides you with the capacity reservation, and offer a significant discount on the hourly charge for an instance. 1 year or 3 year terms 8 | 9 | - **Spot** - Enables you to bid whatever price you want for instant capacity, providing for even greater savings if your applications have flexible start and end times 10 | 11 | - **Dedicated Hosts** - Physical EC2 server dedicated for your use. Dedicated Hosts can help reduce costs by allowing you to use your existing server-bound software license 12 | 13 | **_Important Note!!_** 14 | 15 | If a Spot instance is terminated by Amazon EC2, you will not be charged for a partial hour of usage. However, if you terminate the instance yourself, you will be charged for the complete hour in which the instance ran. 16 | 17 | ## Instance Types 18 | 19 | - **F.** - FGPA 20 | - **I.** - IOPS 21 | - **G.** - Graphics 22 | - **H.** - High Disk Throughput 23 | - **T.** - Cheap General Purpose (think T2 Micro) 24 | - **D.** - Density 25 | - **R.** - Ram 26 | - **M.** - Main choice for general purpose applications 27 | - **C.** - Compute 28 | - **P.** - Graphics(Pics) 29 | - **X.** - Extreme Memory 30 | 31 | ## Volume Types 32 | 33 | ### SSD 34 | 35 | - **General Purpose (SSD)** - balances price and perf. for a wide variety of workloads 36 | 37 | - **Provisioned IOPS (SSD)** - Highest perf. SSD volume for mission-critical low-latency or high-throughput workloads 38 | 39 | ### Magnetic 40 | 41 | - **Throughput Optimized HDD** - Low cost HDD volume designed for frequently accessed, throughput-intensive workloads 42 | 43 | - **Cold HDD** - Lowest cost HDD volume designed for less frequently accessed workloads 44 | 45 | - **Magnetic** - Previous Generation. Can be a boot volume. 46 | 47 | ## Upgrading EBS Volume Types - Lab 48 | 49 | ### Volumes & Snapshots 50 | 51 | - Volumes exist on EBS 52 | - Virtual Hard Disk 53 | - Snapshots exist on S3 54 | - Snapshots are a point in time copies of Volumes 55 | - Snapshots are incremental - this means that only the blocks that have changed since your last snapshot are moved to S3. Only recording the changes 56 | - If it's 1st snapshot, takes time to create 57 | 58 | ### Snapshots of Root Device Volumes 59 | 60 | - To create a snapshot of Amazon EBS volumes that serve as root devices, you should stop the instance before taking the snapshot, however you can take a snapshot while instance is running. 61 | - However you can take a snap while the instance is running. 62 | - You can create AMI's from EBS-backed Instances and Snapshots. 63 | - You can change EBS volume sizes on the fly, including changing the size and storage type. 64 | - Volumes will **ALWAYS** be in the same availability zone as the EC2 instance. 65 | - To move and EC2 volume from one AZ/Region to another, take a snap or an image of it, then copy it to the new AZ/Region. 66 | 67 | ### Volumes vs Snapshots - Security 68 | 69 | - Snapshots of encrypted volumes are encrypted automatically. 70 | - Volumes restored from encrypted snapshots are encrypted automatically. 71 | - You can share snapshots, but only if they are unencrypted. 72 | - These snapshots can be shared with other AWS accounts or made public. 73 | 74 | ### EBS vs. Instance Store 75 | 76 | - Instance store volumes are sometimes called _Ephemeral Storage_. 77 | - Instance store volumes cannot be stopped. If the underlying host fails, you will lose all your data. 78 | - EBS backed instances can be stopped. You will not los the data on this instance if it is stopped. 79 | - You can reboot both, you will not lose your data. 80 | - By default, both ROOT volumes will be deleted on termination, however with EBS volumes, you can tell AWS to keep the root device volume. 81 | 82 | ## Load Balancers 83 | 84 | - 3 Types of Load Balancers 85 | - Application Load Balancers 86 | - Network Load Balancers 87 | - Classic Load Balancers 88 | 89 | 90 | - 504 Error means the gateway has timed out. Application is not responding within the idle timeout period 91 | - Trouble shoot the application. Web Server or Database Server? 92 | 93 | 94 | - If you need IPv4 address of your end user, look fro the X-Forwarded-For header. 95 | - Instances are monitored but ELB are reported as `InService` or `OutofService`. 96 | - Health Checks check the instance health by talking to it. 97 | - ELB's have their own DNS name. You are **never** given an IP address 98 | - Read the ELB FAQ for Classic Load Balancers 99 | 100 | _Note: ELB's do not have IP Addresses, only found by DNS namespace_ 101 | 102 | ## CloudWatch 103 | 104 | - Standard Monitoring - 5 minutes 105 | - Detailed Monitoring - 1 minute 106 | 107 | ### What can you do with CloudWatch? (Not to be confused with CloudTrail) 108 | 109 | - **Dashboards** - Creates awesome dashboards to see/monitor what is happening with your AWS environment. 110 | - **Alarms** - Allows you to set Alarms that notify you when a particular thresholds are hit. 111 | - **Events** - Helps you to respond to state changes in your AWS resources. 112 | - **Logs** - Helps you to aggregate, monitor and store logs. 113 | 114 | ## Placement Groups 115 | 116 | - A Clustered Placement Group can not span multiple Availability Zones. 117 | - A Spread Placement Group can. 118 | - The name you specify for a placement group must be unique within your aws account. 119 | - Only certain types of instances can be launched in a placement group (Compute Optimized, GPU, Memory Optimized, Storage Optimized) 120 | - AWS recommend homogeneous instances within placement groups. 121 | - You cant merge placement groups 122 | - You cant move an existing instance into a placement group. You can create an AMI from your existing instance, and then launch a new instance from the AMI into a placement group. 123 | 124 | ## Lambda 125 | 126 | - Lambda scales horizontally (not vertically) automatically. Redundancy 127 | - Lambda functions are independent, 1 event = 1 function 128 | - Lambda is serverless 129 | - Know what services are serverless! 130 | - S3 131 | - API Gateway 132 | - DynamoDB 133 | - Lambda functions can trigger other lambda functions, 1 event can = x functions if functions trigger other functions. 134 | - Architectures can get extremely complicated, AWS X-ray allows you to debug what is happening 135 | - Lambda can do things globally, you can use it to back up S3 buckets to other S3 buckets etc. 136 | - Know your triggers - connecting AWS services 137 | 138 | ## Summary (TLDR;) 139 | 140 | - Know the differences between EC2 instances 141 | - On Demand 142 | - Spot 143 | - Reserved 144 | - Dedicated hosts 145 | 146 | **_Remember with Spot Instances_** 147 | 148 | - If you terminate the instance, you pay for the hour 149 | - If AWS terminates the instance, you get the hour it was terminated for free. 150 | 151 | ### EC2 Instance Types 152 | 153 | **F.I.G.H.T.D.R.M.C.P.X** (Use Reference) 154 | 155 | ### EBS (Elastic Block Storage) 156 | 157 | **Consists of:** 158 | 159 | - SSD, General Purpose - GP2 - Up to 10,000 IOPS 160 | - SSD, Provisioned IOPS - IO1 - More than 10,000 IOPS 161 | - HDD, Throughput Optimized - ST1 - frequently accessed workloads 162 | - HDD, Cold - SC1 - Less frequently accessed data 163 | - HDD, Magnetic - Standard - Cheap, Infrequently accessed storage. 164 | 165 | **IMPORTANT NOTE:** You cannot mount 1 EBS volume to multiple EC2 instances; Instead use EFS (Elastic File Storage) 166 | 167 | ### Lab Tips! 168 | 169 | - Termination Protection is turned off by default, you must turn this on! 170 | - On a EBS-backed instance, the default action is for the root EBS volume to be deleted when the instance is terminated. 171 | - EBS backed Root volumes can now be encrypted using AWS API or console, or you can use a third party tool (bitlocker etc.) to encrypt the root volume. 172 | - Additional volumes can be encrypted 173 | 174 | ### Volumes vs. Snapshots 175 | 176 | - Volumes exist on EBS; Virtual Hard Disks 177 | - Snapshots exist on S3 178 | - You can take a snapshot of a volume, this will store that volume on S3 179 | - Snapshots are point-in-time copies of volumes 180 | - Snapshots are incremental. This means that only the blocks that have changed since your last snapshot are moved to S3 181 | - If taking your first snapshot, may take some time 182 | 183 | **Security** 184 | 185 | - Snapshots of encrypted volumes are encrypted automatically 186 | - Volumes restored from encrypted snapshots are encrypted automatically 187 | - You can share snapshots, but only if they are unencrypted 188 | - These snapshots can be shared with other AWS accounts or made public 189 | 190 | **Snapshots or Root Device Volumes** 191 | 192 | - To create a snapshot for EBS volumes that serve as root devices, you should stop the instance before taking the snapshot. 193 | 194 | ### EBS vs Instance Store 195 | 196 | - Instance Store Volumes are sometimes called Ephemeral Storage 197 | - Instance Store Volumes cannot be stopped. If the underlying host fails, you will lose your data. 198 | - EBS backed instances can be stopped. You will not lose the data on this instance if it is stopped. 199 | - You can reboot both, you will not lose your data 200 | - By default, both ROOT volumes will be deleted on termination. However, with EBS volumes, you can tell AWS to keep the root device volume. 201 | 202 | ### How can you take a snapshot of a RAID Array? 203 | 204 | **Problem** - Take a snapshot, the snapshots excludes data held in the cache by applications and the OS. This tends not to matter on a single volume. However, using multiple volumes in a RAID array, this can be a problem due to interdependencies of the array. 205 | 206 | **Solution** - Take an application consistent snapshot. 207 | 208 | - Stop the application from writing to disk 209 | - Flush all caches to the disk. 210 | 211 | How is this accomplised? 212 | 213 | - Freeze the file system 214 | - Unmount the RAID array 215 | - Shutting down the associated EC2 instance. 216 | 217 | ### AMI (Amazon Machine Image) 218 | 219 | AMIs are regional. You can only launch an AMI from the region in which its stored. However you can copy AMIs to other regions using the console, command line, or the Amazon EC2 API 220 | 221 | - Standard monitoring - 5 min 222 | - Detailed monitoring - 1 min 223 | 224 | - Cloudwatch is for **performance monitoring** 225 | - Cloudtrail is for **auditing** 226 | 227 | ### Cloudtrail 228 | 229 | - **Dashboards** - Cloudwatch creates awesome dashboards to see what is happening with your AWS envrionment 230 | - **Alarms** - Allows you to set alarms when particular thresholds are hit. 231 | - **Events** - Helps you to respond to state changes in your AWS resources. 232 | - **Logs** - Helps you to aggregate, monitor, and store logs 233 | 234 | ### Roles 235 | 236 | - Roles are more secure than storing your access key and secret access key on individual instances. 237 | - Roles are easier to manage 238 | - Roles can be assigned to an EC2 instance after it has been provisioned using both the command line and the AWS console 239 | - Roles are universal - they can be used in any region 240 | 241 | ### Instance Metadata 242 | 243 | - Used to get information about an instance (public IP, DNS etc) 244 | - `curl http://169.254.169.254/latest/meta-data` 245 | - `curl http://169.254.169.254/latest/user-data` 246 | 247 | ### EFS (Elastic File System) 248 | 249 | - Supports the Network File System version 4 (NFSv4) protocol 250 | - You only pay for the storage you use (no pre-provisioning required) 251 | - Can scale up to petabytes 252 | - Can support thousands of concurrent NFS connections 253 | - Data is stored accross multiple AZs within a region 254 | - Read after Write consistency 255 | 256 | ### Lambda 257 | 258 | - Lambda is a compute service where you can upload you code and create a Lambda function. 259 | - Takes care of provisioning and managing servers that you use to run your code. 260 | - Need not worry about OS, patching, scaling etc. 261 | 262 | **_Use Lambda as:_** 263 | 264 | - Event driven compute service where Lambda runs your code in response to events. These events could be changes in an S3 bucket or Dynamo DB table. 265 | - A compute service to run your code in response to HTTP requests using API Gateway or API calls made using AWS SDKs 266 | 267 | ### Placement Groups 268 | 269 | **Know the differences between and why you would use...** 270 | 271 | - Clustered Placement Groups 272 | - Spread Placement Groups -------------------------------------------------------------------------------- /Application Services/README.md: -------------------------------------------------------------------------------- 1 | # Application Services 2 | 3 | ## SQS - Simple Queue Service 4 | 5 | First **EVER** AWS Service! 6 | 7 | Amazon SQS is a web service that gives you access to a message queue that can be used to store messages while waiting for a computer to process them. 8 | 9 | Amazon SQS is a distributed queue sustem that enables web service applications to quickly and reliably queue messages that one component in the application generates to be consumed by another component. A queue is a temporary repository for messages that are awaiting processing. 10 | 11 | ### SQS Breakdown 12 | 13 | Using Amazon SQS, you can decouple the components of an application so they run independentlym easing message management between components 14 | 15 | Any component of a distributed application can store messages in the queue. Messages can contain up to 256Kb of text in any format. Any component can later retrieve the messages programatically using the SQS API 16 | 17 | ### What do you mean by "Queue"? 18 | 19 | The queue acts as a buffer between the component producing and saving data, and the component receiving the data for processing. This means the queue resolves issues that arise if the producer is producing faster than the consumer can process it, of if the producer or consumer are only intermittently connected to the network. 20 | 21 | ### Queue Types 22 | 23 | ### Standard Queue (default) 24 | 25 | Amazon SQS offers standard as the default queue type. A standard queue lets you have a nearly-unlimited number of transactions per second. Standard queues guarantee that a message is delivered at least once. However, because of the highly distributed architecture that allows high throughput, more than one copy of a message might be delivered out of order. Standard queues provide best effort ordering which ensures that messages are generally delivered in the same order as they are sent. 26 | 27 | ### FIFO Queues (First In, First Out) 28 | 29 | The FIFO queue complements the standard queue. The most important features of this queue type are FIFO delivery and exactly one processing: The order in which messages are sent and received is strictly preserved and a message is delivered once and remains available until a consumer processes and deletes it; duplicates are not introduced into the queue. FIFO queues also support message groups that allow multiple ordered message groups within a single queue. FIFO queues are limited to 300 transactions per second, but have all the capabilities of standard queues 30 | 31 | ```|_5_| ---> |_4_| ---> |_3_| ---> |_2_| ---> |_1_|``` 32 | 33 | ### Key Facts 34 | 35 | - SQS is pull-based, not pushed based 36 | - Messages are 256Kb in size 37 | - Messages can be kept in the queue from 1 minute to 14 days 38 | - Default retention period is 4 days 39 | - SQS guarantees that your messages will be processed at least once. 40 | 41 | ### Visibility Timeout 42 | 43 | - The Visibility Timeout is the amount of time that the message is invisible in the SQS queue after the reader picks up that message. Provided the job is processed before the visibility timeout expires, the message will then be deleted from the queue. If the job is not processed within that time, the message will become visible again and another reader/worker will process it. This could result in the same message delivered twice 44 | - Default visibility timeout is 30 seconds 45 | - Increase it if your task takes >30 seconds 46 | - Maximum is 12 hours 47 | 48 | ### Long Polling 49 | 50 | - Amazon SQS long polling is a way to retrieve messages from your Amazon SQS queues 51 | - While the regular short polling returns immediately (even if the message queue being polled is empty), long polling doesn't return a repsonse until a message arrives in the message queue, or the long poll times out. 52 | - Waits til message is in the queue. 53 | - As such, long polling saves you money. 54 | 55 | ## SWF - Simple Workflow Service 56 | 57 | Amazon Simple Workflow Service is a web service that makes it easy to coordinate work across distributed application components. Amazon SWF enables applications for a range of use cases, including media processing, web application back-ends, business process workflows, and analytics pipelines, to be designed as a coordination of tasks. 58 | 59 | Tasks represent invocations of various processing steps in an application which can be performed by executable code, web service calls, human actions, scripts. 60 | 61 | ### Starters 62 | 63 | An application that can initiate a workflow. Could be your e-commerce website when placing an order or a mobile app searching for bus times 64 | 65 | ### Workers 66 | 67 | Workers are programs that interact with Amazon SWF to get tasks, process received tasks and return results. 68 | 69 | ### Deciders 70 | 71 | The decider is a program that controls the coordination of tasks, ie their ordering, concurrency and scheduling according to the application logic. 72 | 73 | 74 | ### Workers and Deciders Interaction 75 | 76 | The workers and the decider can run on cloud infrastructure, such as Amazon EC2, or on machines behind firewalls, Amazon SWF brokers the interactions between workers and the decider. It allows the decider to get consistent views into the progress of tasks and to initiate new tasks in an ongoing manner. 77 | 78 | At the same time, Amazon SWF stores tasks, assigns them to workers when they are ready and monitors their progress. It ensures that a task is assigned **ONLY ONCE** and is **NEVER DUPLICATED** (key difference from SQS). 79 | 80 | Since Amazon SWF maintains the applications state durably, workers and deciders dont have to keep track of execution state. They can run independently, and scale quickly. 81 | 82 | ### SWF Domains 83 | 84 | Your workflow and activity types and the workflow execution itself are all scoped to a domain. Domains isolate a set of types, executions, and task lists from others within the same account. 85 | 86 | You can register a domain by using the AWS Management Console or by using the Register Domain action inthe Amazon SWF API. 87 | 88 | Maximum workflow can be 1 year and the value is always measured in seconds 89 | 90 | _JSON Domain Registration Example_ 91 | 92 | ```JSON 93 | { 94 | "name": "92034", 95 | "description": "music", 96 | "workflowExecutionRetentionPeriodInDays": "60" 97 | } 98 | ``` 99 | 100 | ### SWF vs. SQF 101 | 102 | - Amazon SWF has a retention period of 1 year vs SQS's 14 days retention 103 | - Amazon SWF presents a task-oriented API, whereas Amazon SQS offers a message-oriented API 104 | - Amazon SWF ensures that a task is assigned **ONLY ONCE** and is **NEVER DUPLICATED**. With SQS, you need to handle duplicated messages and may also need to ensure that a message is processed only once. 105 | - Amazon SWF keeps track of all the tasks and events in an application. With SQS, you need to implement your own application level tracking, especially if your application uses multiple queues. 106 | 107 | ## SNS - Simple Notification Service 108 | 109 | SNS is a web service that makes it easy to set up, operate and send notifications from the cloud. It provides developers with a highly scalable, flexible and cost-effective capability to publish messages from an application and immediately deliver them to subscribers or their applications 110 | 111 | May push notifications to Apple, Google, Fire OS and Windows devices as well as Android devices in China with Baidu Cloud Push. 112 | 113 | Besides pushing cloud notifications directly to mobile devices, SNS can also deliver notifications by SMS text message or email, to SQS queues, or to any HTTP endpoint. 114 | 115 | SNS notifications can also trigger Lambda functions. When a messge is published to and SNS topic that has a Lambda function subscribed to it, the Lambda function is invoked with the payload of the published message. The Lambda function receives the message payload as an input parameter and can manipulate the information in the message, publish the message to other SNS topics, or send the message to other AWS services. 116 | 117 | ### SNS Structure 118 | 119 | SNS allows you to group multiple recipients using topics. A topic is an "access point" for allowing recipients to dynamically subscribe for identical copies of the same notification. 120 | 121 | One topic can support deliveries to multiple endpoint types - for example, you can group together iOS, Android and SMS recipients. When you publish once to a topic, SNS delivers appropriately formatted copies of your message to each subscriber. 122 | 123 | To prevent messages from being lost, all messages published to SNS are stored redundantly across multiple availability zones. 124 | 125 | ### Subscribers - Who may subscribe to notifications? 126 | 127 | - HTTP 128 | - HTTPS 129 | - Email 130 | - Email-JSON 131 | - SQS 132 | - Application 133 | - Lambda 134 | 135 | 136 | ### SNS Benefits 137 | 138 | - Instantaneous, push-based delivery (no polling) 139 | - Simple APIs and easy integration with applications 140 | - Flexible message delivery over multiple transport protocols 141 | - Inexpensive, pay-as-you-go model with no up-front costs 142 | - Web-based AWS Management Console offers the simplicity of a point-and-click interface 143 | 144 | ### SNS vs SQS 145 | 146 | - Both messaging services in AWS 147 | - SNS = push; SQS = polls (pulls) 148 | 149 | ### Pricing 150 | 151 | - User pays $0.50 per 1 million SNS Requests 152 | - $0.06 per 100,000 notification deliveries over HTTP 153 | - $0.75 per 100 notifications deliveries over SMS 154 | - $2.00 per 100,000 notification deliveries over email 155 | 156 | ## Elastic Transcoder 157 | 158 | - Media Transcoder in the cloud. 159 | - Convert media files from their original source format in to different formats that will play on smarphones, tablets, PCs etc. 160 | - Provides transcoding presets for popular output formats, which means that you don't need to guess about which settings work bets on particular devices. 161 | - Pay based on the minutes that you transcode and the resolution at which you transcode. 162 | 163 | ## API Gateway 164 | 165 | API Gatewayis a fully managed service that makes it easy for developers to publish, maintain, monitor and secure APIs at any scale. With a few clicks in the AWS Management Console, you can create and API that acts as a "front door" for applications to access data, business logic, or functionality from you back-end services, such as applications running on EC2, code running on Lambda or any web application. 166 | 167 | ### Caching 168 | 169 | You can enable API caching in API Gateway to cache your endpoints response. With caching, you can reduce the number of calls made to your endpoint and also improve the latency of the requests to your API. 170 | 171 | When you enable caching for a stage, API Gateway caches responses from your endpoint for a specified TTL period, in seconds. API Gateway then responds to the request by looking up the endpoint response from the cache instead of making a request to your endpoint. 172 | 173 | - Low cost & efficient 174 | - Scales effortlessly 175 | - You can throttle requests to prevent attacks 176 | - Connect to Cloudwatch to log all requests 177 | 178 | ## Kinesis 179 | 180 | ### What is streaming data? 181 | 182 | Streaming data is data that is generated continuously by thousands of data sources, which typically send in the data records simultaneously, and in small sizes (order of KB) 183 | 184 | **Examples of usage:** 185 | 186 | - Purchases from online stores 187 | - Stock prices 188 | - Game data 189 | - Social network data 190 | - Geospatial data - uber, google maps 191 | - iOT data 192 | 193 | ### What is Kinesis? 194 | 195 | AWS Kinesis is a platform on AWS to send your streaming data to. Kinesis makes it easy to load and analyze streaming data, and also providing the ability for you to build your own custom applications for your business needs. 196 | 197 | ### Core Kinesis Services? 198 | 199 | #### Kinesis Streams 200 | 201 | - Streams consist of shards 202 | - 5 transactions per second for reads, up to a maximum total data read rate of 2Mb per second and up to 1,000 records per second for writes, up to a maximum total data write rate of 1 Mb per second (including partition keys). 203 | - The data capacity of your stream is a function of the number of shards that you specify for the stream. The total capacity of the stream is the sum of the capacities of its shards. 204 | 205 | #### Kinesis Firehose 206 | 207 | - Handles stream data automatically, no need to specify shards. 208 | 209 | #### Kinesis Analytics 210 | 211 | - Allows you to run SQL queries, analyzing the data and store said data in to another storage service like S3 212 | 213 | 214 | -------------------------------------------------------------------------------- /Application Services/kinesis-data-vis-sample-app.template: -------------------------------------------------------------------------------- 1 | { 2 | "AWSTemplateFormatVersion" : "2010-09-09", 3 | 4 | "Description" : "The Amazon Kinesis Data Visualization Sample Application", 5 | 6 | "Parameters" : { 7 | "InstanceType" : { 8 | "Description" : "EC2 instance type", 9 | "Type" : "String", 10 | "Default" : "t2.micro", 11 | "AllowedValues" : [ "t2.micro", "t2.small", "t2.medium", "m3.medium", "m3.large", "m3.xlarge", "m3.2xlarge", "c3.large", "c3.xlarge", "c3.2xlarge", "c3.4xlarge", "c3.8xlarge" ], 12 | "ConstraintDescription" : "must be a supported EC2 instance type for this template." 13 | }, 14 | 15 | "KeyName" : { 16 | "Description" : "(Optional) Name of an existing EC2 KeyPair to enable SSH access to the instance. If this is not provided you will not be able to SSH on to the EC2 instance.", 17 | "Type" : "String", 18 | "Default" : "", 19 | "MinLength" : "0", 20 | "MaxLength" : "255", 21 | "AllowedPattern" : "[\\x20-\\x7E]*", 22 | "ConstraintDescription" : "can contain only ASCII characters." 23 | }, 24 | 25 | "SSHLocation" : { 26 | "Description" : "The IP address range that can be used to SSH to the EC2 instances", 27 | "Type" : "String", 28 | "MinLength" : "9", 29 | "MaxLength" : "18", 30 | "Default" : "0.0.0.0/0", 31 | "AllowedPattern" : "(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\\d{1,2})", 32 | "ConstraintDescription" : "must be a valid IP CIDR range of the form x.x.x.x/x." 33 | }, 34 | 35 | "ApplicationArchive" : { 36 | "Description" : "A publicly accessible URL to the sample application archive as produced by 'mvn package'", 37 | "Type" : "String", 38 | "MinLength" : "7", 39 | "MaxLength" : "255", 40 | "Default" : "https://github.com/awslabs/amazon-kinesis-data-visualization-sample/releases/download/v1.1.1/amazon-kinesis-data-visualization-sample-1.1.1-assembly.zip" 41 | } 42 | }, 43 | 44 | "Conditions": { 45 | "UseEC2KeyName": {"Fn::Not": [{"Fn::Equals" : [{"Ref" : "KeyName"}, ""]}]} 46 | }, 47 | 48 | "Mappings" : { 49 | "AWSInstanceType2Arch" : { 50 | "t2.micro" : { "Arch" : "64" }, 51 | "t2.small" : { "Arch" : "64" }, 52 | "t2.medium" : { "Arch" : "64" }, 53 | "m3.medium" : { "Arch" : "64" }, 54 | "m3.large" : { "Arch" : "64" }, 55 | "m3.xlarge" : { "Arch" : "64" }, 56 | "m3.2xlarge" : { "Arch" : "64" }, 57 | "c3.large" : { "Arch" : "64" }, 58 | "c3.xlarge" : { "Arch" : "64" }, 59 | "c3.2xlarge" : { "Arch" : "64" }, 60 | "c3.4xlarge" : { "Arch" : "64" }, 61 | "c3.8xlarge" : { "Arch" : "64" } 62 | }, 63 | 64 | "AWSRegionArch2AMI" : { 65 | "us-east-1" : { "64" : "ami-76817c1e" }, 66 | "us-west-2" : { "64" : "ami-d13845e1" }, 67 | "eu-west-1" : { "64" : "ami-892fe1fe" }, 68 | "ap-southeast-1" : { "64" : "ami-a6b6eaf4" }, 69 | "ap-southeast-2" : { "64" : "ami-d9fe9be3" }, 70 | "ap-northeast-1" : { "64" : "ami-29dc9228" } 71 | } 72 | }, 73 | 74 | "Resources" : { 75 | "KinesisStream" : { 76 | "Type" : "AWS::Kinesis::Stream", 77 | "Properties" : { 78 | "ShardCount" : "2" 79 | } 80 | }, 81 | 82 | "KCLDynamoDBTable" : { 83 | "Type" : "AWS::DynamoDB::Table", 84 | "Properties" : { 85 | "AttributeDefinitions" : [ 86 | { 87 | "AttributeName" : "leaseKey", 88 | "AttributeType" : "S" 89 | } 90 | ], 91 | "KeySchema" : [ 92 | { 93 | "AttributeName" : "leaseKey", 94 | "KeyType" : "HASH" 95 | } 96 | ], 97 | "ProvisionedThroughput" : { 98 | "ReadCapacityUnits" : "10", 99 | "WriteCapacityUnits" : "5" 100 | } 101 | } 102 | }, 103 | 104 | "CountsDynamoDBTable" : { 105 | "Type" : "AWS::DynamoDB::Table", 106 | "Properties" : { 107 | "AttributeDefinitions" : [ 108 | { 109 | "AttributeName" : "resource", 110 | "AttributeType" : "S" 111 | }, 112 | { 113 | "AttributeName" : "timestamp", 114 | "AttributeType" : "S" 115 | } 116 | ], 117 | "KeySchema" : [ 118 | { 119 | "AttributeName" : "resource", 120 | "KeyType" : "HASH" 121 | }, 122 | { 123 | "AttributeName" : "timestamp", 124 | "KeyType" : "RANGE" 125 | } 126 | ], 127 | "ProvisionedThroughput" : { 128 | "ReadCapacityUnits" : "10", 129 | "WriteCapacityUnits" : "5" 130 | } 131 | } 132 | }, 133 | 134 | "Ec2SecurityGroup" : { 135 | "Type" : "AWS::EC2::SecurityGroup", 136 | "Properties" : { 137 | "GroupDescription" : "Enable SSH access and HTTP access on the inbound port", 138 | "SecurityGroupIngress" : 139 | [{ "IpProtocol" : "tcp", "FromPort" : "22", "ToPort" : "22", "CidrIp" : { "Ref" : "SSHLocation"} }, 140 | { "IpProtocol" : "tcp", "FromPort" : "80", "ToPort" : "80", "CidrIp" : "0.0.0.0/0"}] 141 | } 142 | }, 143 | 144 | "EIP" : { 145 | "Type" : "AWS::EC2::EIP", 146 | "Properties" : { 147 | "InstanceId" : { "Ref" : "Ec2Instance" } 148 | } 149 | }, 150 | 151 | "RootRole": { 152 | "Type" : "AWS::IAM::Role", 153 | "Properties" : { 154 | "AssumeRolePolicyDocument": { 155 | "Version" : "2012-10-17", 156 | "Statement" : [ { 157 | "Effect" : "Allow", 158 | "Principal" : { 159 | "Service" : [ "ec2.amazonaws.com" ] 160 | }, 161 | "Action" : [ "sts:AssumeRole" ] 162 | } ] 163 | }, 164 | "Path" : "/" 165 | } 166 | }, 167 | 168 | "RolePolicies" : { 169 | "Type" : "AWS::IAM::Policy", 170 | "Properties" : { 171 | "PolicyName" : "root", 172 | "PolicyDocument" : { 173 | "Version" : "2012-10-17", 174 | "Statement" : [ { 175 | "Effect" : "Allow", 176 | "Action" : "kinesis:*", 177 | "Resource" : { "Fn::Join" : [ "", [ "arn:aws:kinesis:", { "Ref" : "AWS::Region" }, ":", { "Ref" : "AWS::AccountId" }, ":stream/", { "Ref" : "KinesisStream" } ]]} 178 | }, { 179 | "Effect" : "Allow", 180 | "Action" : "dynamodb:*", 181 | "Resource" : { "Fn::Join" : [ "", [ "arn:aws:dynamodb:", { "Ref" : "AWS::Region" }, ":", { "Ref" : "AWS::AccountId" }, ":table/", { "Ref" : "KCLDynamoDBTable" } ]]} 182 | }, { 183 | "Effect" : "Allow", 184 | "Action" : "dynamodb:*", 185 | "Resource" : { "Fn::Join" : [ "", [ "arn:aws:dynamodb:", { "Ref" : "AWS::Region" }, ":", { "Ref" : "AWS::AccountId" }, ":table/", { "Ref" : "CountsDynamoDBTable" } ]]} 186 | }, { 187 | "Effect" : "Allow", 188 | "Action" : "cloudwatch:*", 189 | "Resource" : "*" 190 | } ] 191 | }, 192 | "Roles" : [ { "Ref": "RootRole" } ] 193 | } 194 | }, 195 | 196 | "RootInstanceProfile" : { 197 | "Type" : "AWS::IAM::InstanceProfile", 198 | "Properties" : { 199 | "Path" : "/", 200 | "Roles" : [ { "Ref": "RootRole" } ] 201 | } 202 | }, 203 | 204 | "Ec2Instance": { 205 | "Type" : "AWS::EC2::Instance", 206 | "Metadata" : { 207 | "AWS::CloudFormation::Init" : { 208 | "config" : { 209 | "packages" : { 210 | "yum" : { 211 | "java-1.7.0-openjdk" : [] 212 | } 213 | }, 214 | "files" : { 215 | "/var/kinesis-data-vis-sample-app/watchdog.sh" : { 216 | "content" : {"Fn::Join" : ["", [ 217 | "#!/bin/bash\n", 218 | "if ! ps aux | grep HttpReferrerCounterApplication | grep -v grep ; then\n", 219 | " # Launch the Kinesis application for counting HTTP referrer pairs\n", 220 | " java -cp /var/kinesis-data-vis-sample-app/lib/\\* com.amazonaws.services.kinesis.samples.datavis.HttpReferrerCounterApplication ", { "Ref" : "KCLDynamoDBTable" }, " ", { "Ref" : "KinesisStream" }, " ", { "Ref" : "CountsDynamoDBTable" }, " ", { "Ref" : "AWS::Region" }, " &>> /home/ec2-user/kinesis-data-vis-sample-app-kcl.log &\n", 221 | "fi\n", 222 | "if ! ps aux | grep HttpReferrerStreamWriter | grep -v grep ; then\n", 223 | " # Launch our Kinesis stream writer to fill our stream with generated HTTP (resource, referrer) pairs.\n", 224 | " # This will create a writer with 5 threads to send records indefinitely.\n", 225 | " java -cp /var/kinesis-data-vis-sample-app/lib/\\* com.amazonaws.services.kinesis.samples.datavis.HttpReferrerStreamWriter 5 ", { "Ref" : "KinesisStream" }, " ", { "Ref" : "AWS::Region" }, " &>> /home/ec2-user/kinesis-data-vis-sample-app-publisher.log &\n", 226 | "fi\n", 227 | "if ! ps aux | grep WebServer | grep -v grep ; then\n", 228 | " # Launch the webserver\n", 229 | " java -cp /var/kinesis-data-vis-sample-app/lib/\\* com.amazonaws.services.kinesis.samples.datavis.WebServer 80 /var/kinesis-data-vis-sample-app/wwwroot ", { "Ref" : "CountsDynamoDBTable" }, " ", { "Ref" : "AWS::Region" }, " &>> /home/ec2-user/kinesis-data-vis-sample-app-www.log &\n", 230 | "fi\n" 231 | ]]}, 232 | "mode" : "000755", 233 | "owner" : "ec2-user", 234 | "group" : "ec2-user" 235 | }, 236 | "/var/kinesis-data-vis-sample-app/crontask" : { 237 | "content" : {"Fn::Join" : ["", [ 238 | "* * * * * bash /var/kinesis-data-vis-sample-app/watchdog.sh\n" 239 | ]]}, 240 | "mode" : "000644", 241 | "owner" : "ec2-user", 242 | "group" : "ec2-user" 243 | } 244 | }, 245 | "sources": { 246 | "/var/kinesis-data-vis-sample-app" : { "Ref" : "ApplicationArchive" } 247 | } 248 | } 249 | } 250 | }, 251 | 252 | "Properties" : { 253 | "KeyName" : { "Fn::If" : [ "UseEC2KeyName", { "Ref" : "KeyName" }, { "Ref" : "AWS::NoValue" } ]}, 254 | "ImageId" : { "Fn::FindInMap" : [ "AWSRegionArch2AMI", { "Ref" : "AWS::Region" }, 255 | { "Fn::FindInMap" : [ "AWSInstanceType2Arch", { "Ref" : "InstanceType" }, 256 | "Arch" ] } ] }, 257 | "InstanceType" : { "Ref" : "InstanceType" }, 258 | "SecurityGroups" : [{ "Ref" : "Ec2SecurityGroup" }], 259 | "IamInstanceProfile": { "Ref": "RootInstanceProfile" }, 260 | "UserData" : { "Fn::Base64" : { "Fn::Join" : ["", [ 261 | "#!/bin/bash\n", 262 | "yum update -y aws-cfn-bootstrap\n", 263 | 264 | "/opt/aws/bin/cfn-init -s ", { "Ref" : "AWS::StackId" }, " -r Ec2Instance ", 265 | " --region ", { "Ref" : "AWS::Region" }, "\n", 266 | 267 | "# Register watchdog script with cron\n", 268 | "crontab /var/kinesis-data-vis-sample-app/crontask\n", 269 | 270 | "# Launch watchdog script immediately so if it fails this stack fails to start\n", 271 | "/var/kinesis-data-vis-sample-app/watchdog.sh\n", 272 | 273 | "/opt/aws/bin/cfn-signal -e $? '", { "Ref" : "WaitHandle" }, "'\n" 274 | ]]}} 275 | } 276 | }, 277 | 278 | "WaitHandle" : { 279 | "Type" : "AWS::CloudFormation::WaitConditionHandle" 280 | }, 281 | 282 | "WaitCondition" : { 283 | "Type" : "AWS::CloudFormation::WaitCondition", 284 | "DependsOn" : "Ec2Instance", 285 | "Properties" : { 286 | "Handle" : {"Ref" : "WaitHandle"}, 287 | "Timeout" : "600" 288 | } 289 | } 290 | }, 291 | "Outputs" : { 292 | "URL" : { 293 | "Description" : "URL to the sample application's visualization", 294 | "Value" : { "Fn::Join" : [ "", [ "http://", { "Fn::GetAtt" : [ "Ec2Instance", "PublicDnsName" ] }]]} 295 | }, 296 | "InstanceId" : { 297 | "Description" : "InstanceId of the newly created EC2 instance", 298 | "Value" : { "Ref" : "Ec2Instance" } 299 | }, 300 | "AZ" : { 301 | "Description" : "Availability Zone of the newly created EC2 instance", 302 | "Value" : { "Fn::GetAtt" : [ "Ec2Instance", "AvailabilityZone" ] } 303 | }, 304 | "StreamName" : { 305 | "Description" : "The name of the Kinesis Stream. This was autogenerated by the Kinesis Resource named 'KinesisStream'", 306 | "Value" : { "Ref" : "KinesisStream" } 307 | }, 308 | "ApplicationName" : { 309 | "Description" : "The name of the Kinesis Client Application. This was autogenerated by the DynamoDB Resource named 'KCLDynamoDBTable'", 310 | "Value" : { "Ref" : "KCLDynamoDBTable" } 311 | }, 312 | "CountsTable" : { 313 | "Description" : "The name of the DynamoDB table where counts are persisted. This was autogenerated by the DynamoDB Resource named 'CountsDynamoDBTable'", 314 | "Value" : { "Ref" : "CountsDynamoDBTable" } 315 | } 316 | } 317 | } 318 | -------------------------------------------------------------------------------- /Databases/README.md: -------------------------------------------------------------------------------- 1 | # Databases 2 | 3 | ## Types of Databases 4 | 5 | ### Relational Databases 6 | 7 | Relational databases are what most of us are all used to. They have been around since the 70's and you can think about them like spreadsheets! 8 | 9 | - Database 10 | - Tables 11 | - Columns 12 | - Rows 13 | 14 | | id | name | age | location | 15 | | --------- |:-------:| :-----:| :--------: | 16 | | 1 | nigel | 30 | San Diego | 17 | | 2 | jim | 28 | NYC | 18 | | 3 | betty | 31 | San Francisco| 19 | 20 | **_Relational Databases Examples_** 21 | 22 | - SQL Server 23 | - Oracle 24 | - MySQL 25 | - PostgreSQL 26 | - Aurora 27 | - MariaDB 28 | 29 | ### Non-Relational (NoSQL) 30 | 31 | - Database 32 | - Collection => Table 33 | - Document => Row 34 | - Key, Value Pairs => Columns 35 | 36 | **_Non Relational Databases Examples_** 37 | 38 | ```json 39 | { 40 | "_id": "394ejojaj903091881dnna", 41 | "name": "nigel", 42 | "age": 30, 43 | "location": "San Diego" 44 | } 45 | ``` 46 | 47 | ### Data Warehousing 48 | 49 | Used for business intelligence. Tools like Cognos, Jaspersoft, SQL Server, Reporting Services, Oracle Hyperion, SAP NetWeaver. 50 | 51 | Used to pull in very large and complex data sets. Usually used by management to do queries on data (such as current performance vs targets etc). 52 | 53 | ### OLTP (Online Transaction Processing) vs. OLAP (Online Analytics Processing) 54 | 55 | OTLP differs from OLAP in terms of the types of queries you will run. 56 | 57 | **_OLTP Example_** 58 | 59 | Used for transactional type queries. 60 | 61 | ``` 62 | Order number: 2120121 63 | 64 | Pulls up a row of data such as Name, Date, Address to Deliver to, Delivery Status etc. 65 | ``` 66 | 67 | **_OLAP Example_** 68 | 69 | Used for business logic type queries. 70 | 71 | ``` 72 | Net Profit of given product or device 73 | Pulls in large number of records 74 | 75 | Sum of products sold in region 76 | Sum of products sold in continent 77 | Unit cost of product in each region 78 | Sales price of each product 79 | Sales price - unit cost 80 | ``` 81 | 82 | Data Warehousing databases use different type of architecture both from a database perspective and infrastructure layer. 83 | 84 | ### Elasticache 85 | 86 | ElastiCache is a web service that makes it easy to deploy, operate and scale an in-memory cache in the cloud. The service improves the performance of web applications by allowing you to retrieve information from fast, managed, in-memory caches, instead of relying entirely on slower disk-based databases. 87 | 88 | ElasticCache supports two open-source in-memory caching engines... 89 | 90 | 1. Memcached 91 | 2. Redis 92 | 93 | ## Backups, Multi-AZ & Read Replicas 94 | 95 | ### Automated Backups 96 | 97 | Automated Backups allow you to recover your database to any point in time within a 'retention period'. The retention period can be between one and 35 days. 98 | 99 | Automated Backups will take a full daily snapshot and will also store transaction logs throughout the day. 100 | 101 | When you do a recovery, AWS will first choose the most recent daily backup, and then apply transaction logs relevant to that day. This allows you to do a point in time recovery down to a second, within a retention period. 102 | 103 | ### Database Snapshots 104 | 105 | DB Snapshots are done manually (ie they are user initiated) They are stored even after you delete the original RDS instance, unlike automated backups. 106 | 107 | ### Restoring Backups 108 | 109 | Whenever you restore either an Automatic Backup or a manual Snapshot, the restored version of the database will be a new RDS instance with a new DNS endpoint 110 | 111 | `original.us-west-1.rds.amazonaws.com` -> `restored.eu-west-1.rds.amazonaws.com` 112 | 113 | ### Encyrption 114 | 115 | Encryption at rest is supported for MySQL, Oracle, SQL Server, PostgreSQL, MariaDB & Aurora. 116 | 117 | Encryption is done using the AWS Key Management System (KMS) service. Once your RDS instance is encrypted, the data stored at rest in the underlying storage is encrypted, as are its automated backups, read replicas and snapshots. 118 | 119 | At the present time, encrypting an existing DB Instance is not supported. To use RDS encryption for an existing database, you must first create a snapshot, make a copy of that snapshot and encrypt the copy. 120 | 121 | ### Multi-AZ 122 | 123 | Multi-AZ allows you to have an exact copy of your production database in another Availability Zone. AWS handles the replication for you, so when your production database is written to, this write will automatically be synchronized to the stand by database. 124 | 125 | In the event of planned database maintenance, DB instance failure, or AZ failure, RDS will automatically failover to the standby so that database operations can resume quickly without admin intervention. 126 | 127 | **NOTE:** It is not primarily used for improving performance, really only **disaster recovery**. For performance improvement, you need **Read Replicas** 128 | 129 | **Multi-AZ Available DBs** 130 | 131 | - SQL Server 132 | - Oracle 133 | - MySQL Server 134 | - PostgreSQL 135 | - MariaDB 136 | 137 | ### Read Replicas 138 | 139 | Read replicas allow you to have a read-only copy of your production database. This is achieved by using async replication from the primay RDS instance to the Read Replica. You use Read Replicas primarily for very read-heavy database workloads. 140 | 141 | - Used for scaling, not disaster control! 142 | - Must have auto backups turned on in order to deploy a Read Replica 143 | - You can have up to 5 Read Replica copies of any database. 144 | - You can have Read Replicas of Read Replicas _(inception)_ - mindful of latency 145 | - Each Read Replica will have its own DNS end point. 146 | - You can have Read Replicas that have Multi-AZ 147 | - You can create Read Replicas of Mulit-AZ source databases 148 | - Read Replicas can be promoted to be their own databases. This breaks the replication. 149 | - You can have a Read Replica in a second region. 150 | 151 | **Read Replica Available DBs** 152 | 153 | - MySQL Server 154 | - PostgreSQL 155 | - MariaDB 156 | - Aurora 157 | 158 | ## DynamoDB 159 | 160 | DynamoDB is a fast and flexible NoSQL database service for all applications that need consistent, single-digit millisecond latency at any scale. It is a fully managed db nd supports both document and key-value data models. Its flexible data model and reliable performance make it a great fit for mobile, web, gaming, ad-tech, IoT etc. 161 | 162 | - Stored on SSD Storage 163 | - Spread Across **3** geographically distinct data centers 164 | 165 | - Eventual Consistent Read (Default) 166 | - Consistency across all copies of data is usually reached within a second. Repeating a read after a short amount of time should return the updated data. (Best Read Perf.) 167 | 168 | 169 | - Strongly Consistent Reads 170 | - A stronly consistent read returns a result that reflects all writes that received a successful response prior to the read. 171 | 172 | **NOTE:** Super easy to scale! Push button scaling 173 | 174 | ### Pricing 175 | 176 | Pricing is based on provision throughput capacity 177 | 178 | - Write Throughput $0.0065 per hour for every 10 units 179 | - Read Throughput $0.0065 per hour for every 50 units 180 | - Storage costs of $0.25G per month 181 | 182 | _Pricing Example:_ 183 | 184 | ``` 185 | Constraint: 1 million WRITEs and 1 million READs per day, while storing 3G of data. 186 | 187 | First, calculate how many writes and reads per second you need. 188 | 189 | 1 million evenly spread writes per day is equivalent to 1,000,000 (writes) /24 (hours) / 60 (minutes) / 60 (seconds) = 11.6 writes per second. 190 | 191 | -- BREAKDOWN -- 192 | 193 | DynamoDB WRITE Capacity Unit - 1 per second = 12 194 | DynamoDB READ Capacity Unit - 1 per second = 12 195 | 196 | READ Capacity Units - billed in blocks of 50 197 | WRITE Capacity Units - billed in blocks of 10 198 | 199 | Calc WRITE Capacity Units = (0.0065 / 10) x 12 x 24 = $0.1872 200 | Calc READ Capacity Units = (0.0065 / 10) x 12 x 24 = $0.0374 201 | ``` 202 | 203 | ## Redshift 204 | 205 | Amazon Redshift is a fast and powerful, fully managed petabyte-scale data warehouse service in the cloud. 206 | 207 | Customers can start small for just $0.25 per hour with no commitments or upfront costs and scale to a petabyte or more for $1,000 per terabyte per year, less than 1/10 of most data warehousing solutions. 208 | 209 | ### Configuration 210 | 211 | - Single Node (160Gb) 212 | - Multi-Node 213 | - Leader Node _(manages client connections and receives queries)_ 214 | - Compute Node _(store data and perform queries and computations)_ - Up to 128 Compute Nodes 215 | 216 | ### Columns 217 | 218 | **Columnar Data Storage** - Instead of storing data as rows, Redshift organizes the data by column. 219 | 220 | Unlike row-based systems, which are ideal for transaction processing, column-based systems are ideal for data warehousing and analytics, where queries often involve aggregates performed over large data sets. 221 | 222 | Since only the columns involved in the queries are processing and columnar data is stored sequentially on the storage media, column-based systems require far fewer I/Os, greatly improving query performance. 223 | 224 | ### Compression 225 | 226 | **Advanced Compression** - Columnar data storescan be compressed much more than row-based data stores because similar data is stored sequentially on disk. 227 | 228 | Redshift employs multiple compression techniques and can often achieve significant compression relative to traditional relational data stores. In addition, Redshift doesn't require indexes or materialized views and so uses less space than traditional relational database systems. 229 | 230 | When loading data into an empty table, Redshift automatically samples you data and selects the most appropriate compression scheme. 231 | 232 | ### MPP 233 | 234 | **Massive Parallel Processing (MPP)** - Redshift automatically distributes data and query load across all nodes. Redshift makes it easy to add nodes to your data warehouse and enables you to maintain fast query performance as your data warehouse grows. 235 | 236 | ### Pricing 237 | 238 | **How is Redshift priced?** 239 | 240 | - Compute Node Hours 241 | - Total number of hours you run across all your compute nodes for the billing period 242 | - Billed for 1 unit per node per hour, so a 3 - node data warehouse cluster running persistently for an entire month would incur 2,160 instance hours. 243 | - You will **not** be charged for leader node hours; only compute nodes will incur charges 244 | 245 | - Backups 246 | - Data transfers (Only within a VPC, not outside of it) 247 | 248 | ### Security 249 | 250 | - Encrypted in transit using SSL 251 | - Encrypted at rest using AES-256 encryption 252 | - By default Redshift takes care of key management 253 | - Manages your keys through HSM (Hardware Security Module) 254 | - AWS Key Management Service (KMS) 255 | 256 | ### Availability 257 | 258 | - Currently only available in 1 AZ - Realistically only for business logic 259 | - Can restore snapshots to new AZ's in the event of outage. 260 | 261 | ## Elasticache 262 | 263 | ElastiCache is a web service that makes it easy to deploy, operate and scale an in-memory cache in the cloud. The service improves the performance of web applications by allowing you to retrieve information from fast, mananged, in-memory caches, instead of relying entirely on slower disk-based databases. 264 | 265 | ### Why ElastiCache? 266 | 267 | ElastiCache can be used to significantly improve latency and throughput for many read-heavy application workloads - ie networking, gaming, media sharing and Q&A portals or compute intensive workloads. 268 | 269 | Caching improves application performance by storing critical pieces of data in memory for low-latency access. Cached information may include the results of I/O intensive database queries or the results of computational intensive calculations 270 | 271 | ### Types of Elasticache 272 | 273 | - Memcached 274 | - A widely adopted memory object caching system. Elasticache is protocol compliant with Memcached, so popular tools that you use today with existing Memcached environments will work seamlessly with the service. 275 | 276 | - Redis 277 | - A popular open-source in-memory key-value store that supports data structures such as sorted sets and lists. ElastiCache supports Master/Slave replcation and Multi-AZ which can be used to achieve cross AZ redundancy. 278 | 279 | ## Aurora 280 | 281 | ### What is Aurora? 282 | 283 | Aurora is a MySQL-compatible,relational database engine that combines the speed and availability of high-end commercial databases with the simplicity and cost effectiveness of open source databases. Aurora provides up to 5x better performance than MySQL at a price point of 1/10 that of a commercial database while delivering similar performance and availability 284 | 285 | ### Scaling 286 | 287 | - Start with 10G, Scles in 10G increments to 64 TB (Storage Autoscaling) 288 | - Compute resource can scale up to 32vCPUs and 244G of Memory. 289 | - 2 copies of your data is contained in each availability zone, with minimum of 3 AZ -> 6 copies of your data! Highly redundant 290 | - Designed to transparently handle the loss of up to 2 copies of data without affecting database write availability and up to 3 copies without affecting read availability. 291 | - Aurora storage is also self-healing. Data blocks and disks are continuously scanned for errors and repaired automatically. 292 | 293 | ### Aurora Replicas 294 | 295 | - 2 Types of Replicas are available 296 | - Aurora Replicas - Up to 15 replicas currently 297 | - MySQL Replicas - Up to 5 replicas currently -------------------------------------------------------------------------------- /EC2/README.md: -------------------------------------------------------------------------------- 1 | # EC2 (Elastic Cloud Compute) 2 | 3 | AWS EC2 is a web service that provides re-sizable compute capacity in the cloud. EC2 reduces the time required to obtain and boot new server instances to minutes, allowing you to quickly scale capacity, both up and down, as your computing requirements change. 4 | 5 | EC2 has changed the economics of cloud computing by allowing you to pay only for capacity that your actually use. EC2 provides developers the tools to build failure resistant applications and isolate themselves from common failure scenarios. 6 | 7 | ## Pricing Options 8 | 9 | ### On Demand 10 | 11 | Allows you to pay a fixed rate by the hour (or by the second) with no commitment. 12 | 13 | **_Use Cases_** 14 | 15 | - Perfect for users that want the low cost and flexibility of EC2 without any of the up front payment or long term commitment 16 | - Applications with short term, spiky or unpredictable workloads that cannot be interrupted 17 | - Applications being developed or tested on EC2 for the first time 18 | 19 | ### Reserved 20 | 21 | Provides you with a capacity reservation, and offer a significant discount on the hourly charge for an instance. 1 year or 3 year terms. 22 | 23 | **_Use Cases_** 24 | 25 | - Applications with steady state or predictable usage 26 | - Applications that require reserved capacity 27 | - Users can make up front payments to reduce their total computing costs even further 28 | - Standard RIs (Up to 75% off on-demand) 29 | - Convertible RIs (Up to 54% off on-demand) feature the capability to change the attributes of the RI as long as the exchange results in the creation of Reserved Instances of equal or greater value. Ability to go from CPU intensive instance to Memory intensive. 30 | - Scheduled RIs are available to launch within the time window you reserve. This option allows you to match your capacity reservation to predictable recurring schedule that only requires a fraction of a day, a week, or a month. 31 | 32 | ### Spot 33 | 34 | Enables you to bid whatever price you want for an instance capacity, providing for even greater savings if your applications have flexible start and end times. 35 | 36 | **Use Cases** 37 | 38 | - Applications that have flexible start and end times 39 | - Applications that are only feasible at very low compute prices 40 | - Used for single compute instances to save on costs compared to 9-5 during the week. 41 | - Users with an urgent need for a large amount of additional computing capacity. 42 | 43 | ### Dedicated Hosts 44 | 45 | Physical EC2 server dedicated for your use. Dedicated Hosts can help you reduce costs by allowing you to use your existing server-bound software licenses. 46 | 47 | **Use Cases** 48 | 49 | - Useful for regulatory requirements that may not support multi-tenant virtualization. 50 | - Great for licensing which does not support multi-tenancy or cloud deployments 51 | - Can be purchased On-Demand (hourly). 52 | - Can be purchased as a Reservation for up to 70% off the On-Demand price. 53 | 54 | ## EC2 Instance Types 55 | 56 | **_No need to memorize for associate exams_** 57 | 58 | | Family | Specialty | Use Cases | 59 | | :------:|:-----------------------------:| :------------------------------:| 60 | | F1 | Field Programmable Gate Array | Genomics research, financial analytics, real-time video processing, big data etc| 61 | | I3 | High Speed Storage | NoSQL DBs, Datawarehousing | 62 | | G3 | Graphics Intensive | Video Encoding / 3D Application Streaming| 63 | | H1 | High Disk Throughput | MapReduce-based workloads, distributed file systems such as HDFS and MapR-FS | 64 | | T2 | Lowest Cost General Purpose | Web Servers / Small DBs | 65 | | D2 | Dense Storage | Fileservers / Data Warehousing / Hadoop | 66 | | R4 | Memory Optimization | Memory Intensive Apps/DBs | 67 | | M5 | General Purpose | Application Servers | 68 | | C5 | Compute Optimized | CPU Intensive Apps / DBs | 69 | | P3 | Graphics / General Purpose GPU | Machine Learning, Bit Coin Mining etc | 70 | | X1 | Memory Optimized | SAP HANA / Apache Spark | 71 | 72 | 73 | **How to remember EC2 instance types F.I.G.H.T.D.R.M.C.P.X (after 2017 reinvent):** 74 | - **_F_** - FGPA 75 | - **_I_** - IOPS 76 | - **_G_** - Graphics 77 | - **_H_** - High Disk Throughput 78 | - **_T_** - Cheap General Purpose (think T2 Micro) 79 | - **_D_** - Density 80 | - **_R_** - Ram 81 | - **_M_** - Main choice for general purpose applications 82 | - **_C_** - Compute 83 | - **_P_** - Graphics(Pics) 84 | - **_X_** - Extreme Memory 85 | 86 | ## EBS - Elastic Block Storage 87 | 88 | Amazon EBS allows you to create storage volumes and attach them Amazon EC2 instances. Once attached, you can create a file system on top of theses volumes, run a database, or use them in any other way you would use a block device. EBS volumes are placed in a specific Availability Zone, where they are automatically replicated to protect you from the failure of a single component. 89 | 90 | _TLDR; A disk in the cloud that you attach to your EC2 instances_ 91 | 92 | ### EBS Volume Types 93 | 94 | - General Purpose SSD (GP2) 95 | - General purpose, balances both price and performance. 96 | - Ratio of 3 IOPS per GB with up to 10,000 IOPS and the ability to burst up to 3000 IOPS for extended periods of time for volumes at 3334 GB and above 97 | - Provisioned IOPS SSD (IO1) 98 | - Designed for I/O intensive applications such as large relational or NoSQL databases. 99 | - Use if you need more than 10,000 IOPS 100 | - Provision up to 20,000 IOPS per volume 101 | - Super high performance 102 | - Throughput Optimized HDD (ST1) 103 | - Big Data 104 | - Data warehouses 105 | - Log processing 106 | - Cannot be a boot volume 107 | - Cold HDD (SC1) 108 | - Lowest cost storage for infrequently accessed workloads 109 | - File server 110 | - Cannot be a boot volume 111 | - Magnetic (Standard) 112 | - Lowest cost per GB of all EBS volume types that is bootable. Magnetic volumes are ideal for workloads where data is accessed infrequently, and applications where the lowest storage cost is important 113 | 114 | ## Let's get our hands dirty! Launch an EC2 instance lab! 115 | 116 | ### Summary 117 | 118 | - Termination protection is turned off by default, you **MUST** turn it on. 119 | - On an EBS-backed instance, the default action is for the root EBS volume to be deleted when the instance is terminated 120 | - EBS Root Volume of you DEFAULT AMI's cannot be encrypted. You can also use a third party tool (such as bit locker) to encrypt the root volume, or this can be done when creating AMI's (future lab) in the AWS console or using the API. 121 | - Additional volumes can be encrypted. 122 | 123 | ## Security Groups 124 | 125 | ### What is a Security Group? 126 | 127 | A security group is a virtual firewall that's controlling traffic to your EC2 instance. When you first launch as EC2 instance you associate it to 1 or more instances. You have the ability to add rules to these security groups that allows traffic to or from these instances. 128 | 129 | ### Security Groups - General 130 | 131 | 1. Any security group rules apply immediately 132 | 2. Security groups are **_STATEFUL_**. Inbound rules automatically add outbound rules 133 | 3. All traffic is blocked by default and included through the rules. Whitelist 134 | 4. All outbound traffic is allowed 135 | 5. You can have multiple EC2 instances within a security group. 136 | 6. You can have multiple security groups attached to EC2 instances. 137 | 7. You cannot block specific IP addresses using Security Groups, use Network Access Control Lists. 138 | 8. You can specify allow rules, but not deny rules. 139 | 140 | ## RAID, Volumes & Snapshots 141 | 142 | ### RAID - Redundant Array of Independent Disks 143 | 144 | - RAID 0 - Striped, no redundancy, good performance. If one fails, you lose all 145 | - RAID 1 - Mirrored, redundant. If one fails, others available 146 | - RAID 5 - Good for reads, bad for writes, AWS does not recommend ever putting RAID 5's on EBS. Strongly discouraged. 147 | - RAID 10 - Striping & Mirrored, good redundancy, good performance. 148 | 149 | #### How can I take a Snapshot of a RAID Array? 150 | 151 | - **Problem** - Taking a snapshot excludes the data held in cache by applications and the OS. This doesn't really matter on single volume, however when using multiple volumes in a RAID Array, this can be a problem due to interdependencies of the array. 152 | 153 | - **Solution** - Take an application specific snapshot. 154 | - Stop application from writing to disk. 155 | - Flush all caches to the disk. 156 | - How can we do this? 157 | - Freeze the file system 158 | - Unmount the RAID Array 159 | - Shutting down the associated EC2 instance. 160 | 161 | ## Create an AMI lab - Volumes vs. Snapshots 162 | 163 | ### Snapshots of Root Device Volumes 164 | 165 | - To create a snapshot for Amazon EBS volumes that server as root devices, you should stop the instance before taking the snapshot. 166 | 167 | ### Security 168 | 169 | - Snapshots of encrypted volumes are encrypted automatically 170 | - Volumes restored from encrypted snapshots are encrypted automatically. 171 | - You can share snapshots, but only if they are unencrypted. 172 | - Said snapshots can be shared with other AWS accounts of made public 173 | 174 | ## AMI Types 175 | 176 | ### What should you select your AMI based on? 177 | 178 | - Region 179 | - OS 180 | - Architecture 181 | - Launch Permissions 182 | - Storage for the Root Device (Root Device Volume) 183 | - Instance Store (Ephemeral Store) 184 | - EBS Backed Volumes 185 | 186 | ### EBS vs. Instance Store 187 | 188 | All AMIs are categorized as either backed by Amazon EBS or backed by instance store. 189 | 190 | **_For EBS Volumes:_** 191 | 192 | The root device for an instance launched from the AMI is an Amazon EBS volume created from an Amazon EBS snapshot. 193 | 194 | **_For Instance Store Volumes:_** 195 | 196 | The root device for an instance launched from the AMI is an instance store volume created from a template stored in Amazon S3. 197 | 198 | ## Elastic Load Balancers 199 | 200 | ### What is a load balancer? 201 | 202 | A virtual appliance that balances the load of HTTP traffic etc. of your web application/web servers. 203 | 204 | ### Types of Load Balancers 205 | 206 | - Application Load Balancers 207 | - Network Load Balancers 208 | - Classic Load Balancers 209 | 210 | ### Application Load Balancer _(Intelligent)_ 211 | 212 | Best suited for load balancing of HTTP(S) traffic. They operate at Layer 7 (OSI) and are application aware. The are intelligent, and you can create advanced request routing, sending specified requests to specific web servers. 213 | 214 | ### Network Load Balancer _(Performance)_ 215 | 216 | Best suited for load balancing of TCP traffic where extreme performance is required. Operating at the connection level (Layer 4), Network Load Balancers are capable of handling millions of requests per second, while maintaining ultra-low latencies. 217 | 218 | ### Classic Load Balancer _(OG, Legacy Load Balancer)_ 219 | 220 | Used to load balance HTTP(S) applications and use Layer 7-specific features, such as X-Forwarded and stick-sessions. You can use strict Layer 4 load balancing for applications that rely purely on the TCP protocol. 221 | 222 | ### 504 Error 223 | 224 | - If no response or timeout, the ELB (Elastic Load Balancer) responds with status code 504. 225 | - Internal Server Error type - DB Layer or Web Server Layer. 226 | - Solution: Identify issue where failing and scale up or out where possible. 227 | 228 | ## Placement Groups (Exam MUST KNOW!!) 229 | 230 | ### Two Types of Placement Groups 231 | 232 | **Clustered Placement Group** 233 | 234 | A cluster placement group is a grouping of instances within a **single** Availability Zone. Placement groups are recommended for applications that need low network latency, high network throughput, or both. 235 | 236 | _NOTE: Only a certain number instances can be launched in to a Clustered Placement Group._ 237 | 238 | **Spread Placement Group** 239 | 240 | Opposite of a Clustered Placement Group. A Spread Placement Group is a group of instances that are each placed on distinct underlying hardware. Spread Placement Groups are recommended for applications that have a small number of critical instances that should be kept separate from each other. 241 | 242 | ## EFS (Elastic File System) 243 | 244 | AWS EFS is file storage service for AWS EC2 instances. Amazon EFS is easy to use and provides a simple interface that allows you to create and configure file systems quickly and easily. With AWS EFS, storage capacity is elastic, growing and shrinking automatically as you add and remove files, so your applications have the storage they need, when they need it. 245 | 246 | ### EFS Features 247 | 248 | - Supports the Network File System version 4 (NFSv4) protocol 249 | - You only pay for the storage you use (no pre-provisioning required) 250 | - Can scale up to the petabytes 251 | - Can support thousands of concurrent NFS connections 252 | - Data is stored across multiple AZ's within a region 253 | - Read After Write Consistency 254 | 255 | ## Lambda 256 | 257 | ### What is Lambda? 258 | 259 | AWS Lambda is a compute service where you can upload your code and create Lambda function. AWS Lambda takes care of provisioning and managing the servers that you use to run the code. Worry free from OS, patching, scaling, etc. 260 | 261 | **Use Cases** 262 | 263 | - As an event-driven compute service where AWS Lambda runs your code in response to events. These events could be changes to data in an Amazon S3 bucket or an Amazon DynamoDB table. 264 | 265 | - As a compute service to run your code in response to HTTP requests using Amazon API Gateway or API calls made using AWS SDKs. 266 | 267 | **Encapsulation of the following:** 268 | 269 | - Data Centers 270 | - Hardware 271 | - Assembly Code/Protocols 272 | - High Level languages 273 | - Operation Systems 274 | - Application Layer/AWS API's 275 | - AWS Lambda 276 | 277 | ### Compatible Languages: 278 | 279 | - C# 280 | - Java 281 | - Node.js 282 | - Python 283 | 284 | ### How is Lambda priced? 285 | 286 | - Number of requests 287 | - First 1m requests are free. $0.20 per 1m requests thereafter. 288 | 289 | - Duration 290 | - Duration is calculated from the time your code begins execution until it returns or otherwise terminates, rounded up to the nearest 100ms. The price depends on the amount of memory you allocate to your function. You are charged $0.00001667 for every GB-second used. 291 | 292 | 293 | ### Why is Lambda cool? 294 | 295 | - No SERVERS!! 296 | - Continuous Scaling 297 | - Super super super cheap -------------------------------------------------------------------------------- /Additional-Exam-Tips/README.md: -------------------------------------------------------------------------------- 1 | # Additional Exam Tips 2 | 3 | ## Based on Student Feedback... 4 | 5 | ### Kinesis 6 | 7 | - Used to consume Big Data 8 | - Stream large amounts of social media, news feeds, logs etc to the cloud 9 | - Think Kinesis when approached with big data questions 10 | 11 | - Process large amounts of data; 12 | - Redshift for business intelligence 13 | - Elastic Map Reduce for Big Data Processing 14 | 15 | 16 | ### EC2 - EBS Backed Volumes vs Instance Store Volumes 17 | 18 | - EBS backed volumes are persistent 19 | - Instance Store backed volumes are not persistent (ephemeral) 20 | - EBS Volumes can be detached and reattached to other EC2 instances 21 | 22 | - Instance store volumes cannot be detached and reattached to other instances - they exist only for the life of that instance. 23 | - EBS volumes can be stopped; data will persist 24 | 25 | - Instance store volumes cannot be stopped - if you do this the data will be wiped 26 | 27 | - EBS Backed = Store Data Long Term 28 | - Instance Store = Shouldn't be used for long-term data storage 29 | 30 | ### OpsWork 31 | 32 | - Orchestration Service that uses Chef 33 | - Chef consists of recipes to maintain a consistent state 34 | - Look for the term "chef" or "recipes" or "cook books" and think OpsWorks 35 | 36 | ### Elastic Transcoder 37 | 38 | - Media Transcoder in the cloud 39 | - Convert media files from their original source format in to different formats that will play on smartphones, tablets, PC's, etc 40 | - Provides transcoding presets for popular output formats, which means that you dont need to guess about which settings work best on particular devices 41 | - Pay based on the minutes that you transcode and the resolution at which you transcode 42 | 43 | ### SWF Actors 44 | 45 | - **Workflow Starter** - An application that can initiate (start) a workflow. Could be your e-commerce website when placing an order or a mobile app searching for bus times. 46 | - **Deciders** - Control the flow of activity tasks in a workflow execution. If something has finished in a workflow (or fails) a Decider decides what to do next. 47 | - **Activity Workers** - Carry out the activity tasks 48 | 49 | ### EC2 - Get Public IP Address 50 | 51 | - Query the instances metadata: 52 | - `curl http://169.254.169.254/latest/meta-data` 53 | - `wget http://169.254.169.254/latest/meta-data` 54 | - Key thing to remember is that its an instances META-DATA, not user data 55 | 56 | 57 | ## Consolidated Billing 58 | 59 | ### AWS Organizations 60 | 61 | AWS Organizations is an account management service that enables you to consolidate multiple AWS accounts into an organization that you create and centrally manage. 62 | 63 | Available in 2 feature sets 64 | - Consolidated Billing 65 | - All features 66 | 67 | **General Rules** 68 | 69 | - Paying account is independent 70 | - Cannot access resources of other accounts 71 | - All linked accounts are independent 72 | - Currently a limit of 20 linked accounts - can add more 73 | 74 | **Advantages** 75 | 76 | - One bill per AWS account 77 | - Very esy to track charges and allocate costs 78 | - Volume Pricing 79 | 80 | ### Best Practices 81 | 82 | - Always enable MFA on root account 83 | - Always use a strong and complex password on root account 84 | - Paying account should be used for billing purposes only. Do not deploy resources in to paying account 85 | 86 | ### Things to note 87 | 88 | - Billing Alerts 89 | - When monitoring is enabled on the paying account the billing data for all linked accounts is included 90 | - You can still create billing alerts per individual account 91 | 92 | - CloudTrail 93 | - Per AWS Account and is enabled per region 94 | 95 | - Can consolidate logs using an S3 bucket 96 | 1. Turn on CloudTrail in the paying account 97 | 2. Create a bucket policy that allows cross account access 98 | 3. Turn on CloudTrail in the other accounts and use the bucket in the paying account 99 | 100 | ### Tips 101 | 102 | - Consolidate billing allows you to get volume discounts on all your accounts. 103 | - Unused reserved instances for EC2 are applied across the group 104 | - CloudTrail is on a per account and per region basis but can be aggregated in to a single bucket in the paying account. 105 | 106 | ## Cross Account Access 107 | 108 | Many AWS customers use separate AWS accounts for their development and production resources. This separation allows them to cleanly separate different types of resources and can also provide some security benefits. 109 | 110 | Cross account access makes it easier for you to work productively within a multi-account (or multi-role) AWS environment by making it easy for you to switch roles within the AWS Management Console. 111 | 112 | You can sign in to the console using you IAM user name then switch the console using your IAM user name then switch the console to manage another account without having to enter (or remember) another user name and password 113 | 114 | ## Resource Groups & Tags 115 | 116 | Key Value Pairs attached to AWS resources 117 | 118 | Metadata (data about data) 119 | 120 | Tags can sometimes be inherited 121 | 122 | - Autoscaling, CloudFormation and Elastic Beanstalk can create other resources 123 | 124 | Resource groups make it easy to groups your resources using the tags that are assigned to them. You can group that share one or more tags. 125 | 126 | **Note: Container for resources** 127 | 128 | Resource groups contain information such as: 129 | - Region 130 | - Name 131 | - Health Checks 132 | 133 | Specific Information: 134 | - For EC2 - Public and Private IP Addresses 135 | - For ELB - Port Configurations 136 | - For RDS - Database Engine etc. 137 | 138 | ## VPC Peering 139 | 140 | **Note: Generally not tested in Associate exams, only in Professional exams** 141 | 142 | ### What is VPC Peering? 143 | 144 | VPC Peering is simply a connection between 2 VPCs that enables you to route traffic between them using private IP addresses. 145 | 146 | Instances in either VPC can communicate with each other as if they are within the same network. You can create a VPC peering connection between your own VPCs, or with a VPC in another AWS account within a **SINGLE REGION** 147 | 148 | AWS uses the existing infrastructure of a VPC to create a VPC peering connection; it is neither a gateway nor a VPN connection, and does not rely on separate piece of physical hardware. There is no single point of failure for communication or bandwidth bottleneck. 149 | 150 | ### VPC Peering Limitations 151 | 152 | 1. You cannot create a VPC peering connection between VPCs that have overlapping CIDR blocks ie. `10.0.0.0/16 -- X --> 10.0.0.0/24` 153 | 2. You cannot create a VPC peering connection between VPCs in different regions 154 | 3. VPC peering does not support transitive peering relationships. 155 | 156 | ## Direct Connect 157 | 158 | AWS Direct Connect makes it easier to establish a dedicated network connection from your premises to AWS. 159 | 160 | Using AWS Direct Connect, you can establish private connectivity between AWS and your datacenter, office or colocation environment, which in many cases can reduce your network costs, increase bandwidth throughput, and provide a more consistent network experience than internet-based connections. 161 | 162 | ### Main Benefits 163 | 164 | - Reduce costs when using large volumes of traffic 165 | - Increase reliability 166 | - Increase bandwidth 167 | 168 | ### How is Direct Connect different from a VPN? 169 | 170 | VPN Connections can be configured in minutes and are a good solution if you have and immediate need, have low to moderate bandwidth requirements, and can tolerate the inherent variability in Internet-based connectivity. 171 | 172 | AWS Direct Connect does not involve the Internet - instead, it uses dedicated, private network connections between your intranet and AWS VPC 173 | 174 | ### Direct Connect Connections 175 | 176 | Available in: 177 | - 10Gbs 178 | - 1Gbbs 179 | 180 | - Sub 1 Gbps can be purchased through AWS Direct Connect Partners 181 | - Uses Ethernet VLAN trunking (802.1Q) 182 | 183 | ## STS - Security Token Service 184 | 185 | Grants users limited and temporary access to AWS resources. Users can come from 3 sources 186 | 187 | ### Federation (typically Active Directory) 188 | 189 | - Uses Security Assertion Markup Language (SAML) 190 | - Grants temp access based off the users Active Directory credentials. Does not need to be a user in IAM 191 | - Single sign on allows users to log in to AWS console without assigning IAM credentials 192 | - Federation with mobile apps - Facebook/AWS/Google/OpenID providers 193 | - Cross Account Access - Access resources from one account to another 194 | 195 | ### Key Terms 196 | 197 | - Federation 198 | - Combining or joining a list of users in one domain (such as IAM) with a list of users in another domain (such as Active Directory, Facebook etc) 199 | - Identity Broker 200 | - A service that allows you to take an identity from point A and join it (federate it) with point B 201 | - Identity Store 202 | - Services like Active Directory, Facebook, Google etc 203 | - Identities 204 | - A user of a service like Facebook etc. 205 | 206 | **SCENARIO!** 207 | 208 | ``` 209 | You are hosting a company website on some EC2 web servers in your VPC. Users of the website must log in to the site which authenticates against the companies active directory servers which are based on site at the companies head quarters 210 | 211 | Your VPC is connected to your company HQ via a secure IPSEC VPN. Once logged in the user can only have access to their own S3 bucket. How do you set this up? 212 | ``` 213 | 214 | **SOLUTION!** 215 | 216 | 1. Users enter credentials (username and password) 217 | 2. Application calls identity broker - broker captures username and passwords 218 | 3. Broker checks with LDAP directory server - validates credentials 219 | 4. Call to STS (security token service) - getFederationToken function using IAM credentials 220 | 5. STS confirms policy and gives permission to create new tokens - returns 4 values 221 | - Access Key 222 | - Secret Access Key 223 | - Token 224 | - Duration (lifetime of token) 225 | 6. 4 values are sent back to application via broker 226 | 7. Application makes call to S3 227 | 8. S3 uses IAM to validate credentials 228 | 9. Credentials validated via IAM 229 | 230 | **In The Exam!** 231 | 232 | 1. Develop and Identity Broker to communicate with LDAP and AWS STS. 233 | 2. Identity Broker alway authenticates with LDAP first, THEN with AWS STS 234 | 3. Application then gets temp access to AWS resources 235 | 236 | ## Active Directory Tips 237 | 238 | ### Exam Questions 239 | 240 | **QUESITON: _Can you authenticate with Active Directory?_** 241 | 242 | **ANSWER: Yes. Using SAML** 243 | 244 | **QUESITON: _In what order do you authenticate to get the security credentials to log into Active Directory?_** 245 | 246 | **ANSWER: Authenticate with Active Directory first and then you are assigned the temp security credentials.** 247 | 248 | ## Workspaces 249 | 250 | It's basically a VDI (virtual development infrastructure). A Workspace is a cloud-based replacement for a traditional desktop. 251 | 252 | A Workspace is available as a bundle of compute resources, storage space, and software application access that allow a user to perform day-to-day tasks just like using a traditional desktop. 253 | 254 | A user can connect to a Workspace from any supported device (PC, Mac, Chromebook, iPad, KindleFire or Android Tablets) using free Amazon Workspaces client application and credentials set up by an administrator, or their existing Active Directory credentials if Amazon Workspaces is integrated with an existing Active Directory domain. 255 | 256 | ### Quick Facts 257 | 258 | - Windows 7 experience, provided by Windows Server 2008 R2 259 | - By default, users can personalize their workspaces. This can be locked down by an admin however 260 | - By default, you will be given local admin access, so you can install your own applications 261 | - Workspaces are persistent 262 | - All data on the D:\ is backed up every 12 hours 263 | - You do not need an AWS account to login into workspaces 264 | 265 | ## ECS 266 | 267 | - ECS is a regional service that you can use in one or more AZs across a new or existing, VPC to schedule the placement of containers across your cluster based on your resource needs, isolation policies, and availability requirements 268 | 269 | - ECS eliminates the need for you to operate your own cluster management and config management systems, or to worry about scaling your management infrastructure. 270 | 271 | - ECS can also be used to create a consistent deployment and build experience, manage and scale batch and ETL workloads, and build sophisticated application architectures on a microservice level. 272 | 273 | ### ECR (Elastic Container Registery) 274 | 275 | - Managed AWS Docker registery service that is secure, scalable and reliable. 276 | - Supports private Docker repos with resource based permissions using AWS IAM so that specific users or EC2 instances can access repos and images. 277 | - Developers can use the Docker CLI to push, pull and manage images. 278 | 279 | ### ECS Task Definitions 280 | 281 | - A Task Definition is required to run Docker containers in ECS. 282 | - Task Definitions are text files in JSON format that describe one or more containers that form your application. 283 | - Some of the params you can specify in a task definition include: 284 | - Which Docker images to yse with the containers in your task 285 | - How much CPU and memory to use with each container 286 | - Whether containers are linked together in a task 287 | - The Docker networking mode to use for the containers in your task 288 | - What (if any) ports from the container are mapped to the host container service 289 | - Whether the task should continue to run if the container finishes or fails 290 | - The command the container should run when it is started 291 | - What (if any) env variables should be passed to the container when it starts. 292 | - Any data volumes that should be used with containers in the task 293 | - What (if any) IAM role your tasks should use for permissions 294 | 295 | ### ECS Services 296 | 297 | - An ECS service allows you to run and maintain a specified number (or, the "desired count") of instances of a task definition simultaneously in and ECS cluster 298 | - Think of services like AutoScaling groups for ECS 299 | - If a task should fail or stop, the ECS service scheduler launches another instance of your task definition to replace it and maintain the desired count of tasks in the service. 300 | 301 | ### ECS Clusters 302 | 303 | - An ECS cluster is a logical grouping of container instances that you can place tasks on. 304 | - When you first use the Amazon ECS service, a default cluster is created for you, but you can create multiple clusters in an account to keep your resources separate. 305 | - **Concepts:** 306 | - Clusters can contain multiple different container instance types 307 | - Clusters are region-specifc 308 | - Container instances can only be part of one cluster at a time. 309 | - You can create IAM policies for your clusters to allow or restrict users' access to specific clusters 310 | 311 | ### ECS Scheduling 312 | 313 | - **Service Scheduler:** 314 | - Ensures that the specific number of tasks are constantly running and reschedules tasks when a task fails (for example, if the underlying container instance fails for some reason) 315 | - Can ensure tasks are registered against and ELB 316 | - **Custon Scheduler:** 317 | - You can create your own schedulers that meet your business needs. 318 | - Leverage 3rd party schedulers such as Blox 319 | - The ECS schedulers leverage the same cluster state information provided by the ECS API to make appropriate placement decisions 320 | 321 | ### ECS Container Agent 322 | 323 | ECS Container Agent allows container instances to connect to your cluster. ECS Container Agent is included in the ECS optimized AMI, but you can also install it on any EC2 instance that supports ECS specs. ECS Container Agent is only supported on EC2 instances. 324 | 325 | - Pre installed on special ECS AMIs 326 | - Linux based: 327 | - Works with AWS Linux, Ubuntu, Redhat, CentOS, etc. 328 | - Will **not** work with Windows 329 | 330 | ### ECS Security 331 | 332 | - IAM Roles: 333 | - EC2 instances use an IAM role to access ECS 334 | - ECS tasks use an IAM role to access services and resources 335 | - Security Groups attach at the instance-level (i.e. the host - not the task or container) 336 | - You can access and configure the OS of the EC2 instances in your ECS cluster 337 | 338 | ### ECS Limits 339 | 340 | - Soft Limits: 341 | - Clusters per Region (default = 100) 342 | - Instances per Cluster (default = 100) 343 | - Services per Cluster (default = 100) 344 | - Hard Limits: 345 | - One Load Balancer per Service 346 | - 1000 Tasks per Service ("desired") 347 | - Max 10 Containers per Task Defintion 348 | - Max 10 Tasks per Instance (host) 349 | -------------------------------------------------------------------------------- /The-Well-Architected-Framework/README.md: -------------------------------------------------------------------------------- 1 | # The Well Architected Framework 2 | 3 | This section aggregates the well architected framework white paper 4 | 5 | https://aws.amazon.com/architecture/well-architected/ 6 | https://d0.awsstatic.com/whitepapers/AWS_Cloud_Best_Practices.pdf 7 | 8 | ## Best Practices 9 | 10 | ### Business Benefits of the Cloud 11 | 12 | - Almost zero upfront infrastructure investment 13 | - Just-in-time infrastructure 14 | - More efficient resource utilization 15 | - Usage-based costing 16 | - Reduced time to market 17 | 18 | ### Technical Benefits of the Cloud 19 | 20 | - Automation - "Scriptable Infrastructure" 21 | - Auto-Scaling 22 | - Proactive Scaling 23 | - More Efficient Development lifecycle 24 | - Improved Testability 25 | - Disaster Recovery and Business Continuity 26 | - "Overflow" the traffic to the cloud 27 | 28 | ### Design For Failure 29 | 30 | **Rule of thumb:** 31 | 32 | Be a pessimist when designing architectures in the cloud - assume things will fail. In other words, always design, implement and deploy for automated recovery from failure. 33 | 34 | **Assume that...** 35 | 36 | - your hardware _will_ fail 37 | - disaster _will_ strike your application 38 | - you _will_ slammed with more than the expected number of requests per second some day. 39 | - with time your application software _will_ fail too. 40 | 41 | Being a pessimist, you end up thinking about recovery strategies during design time, which helps in designing overall system better. 42 | 43 | ### Decouple Your Components 44 | 45 | The key is to build components that do not have tight dependencies on each other, so that if once component were to die(fail), sleep(not respond) or remain busy(slow to respond) for some reason, the other components in the system are built so as to continue to work as if no failure is happening. 46 | 47 | In essence, loose coupling isolates the various layers and components of you application so that each component interacts async with the others and treats them as a "black box". 48 | 49 | **For Example...** 50 | 51 | In the case of web application architecture, you can isolate the app server from the web server and from the db. The app server does not know about your web server and vice versa, this gives decoupling between these layers and there are not dependencies code wise or functional perspectives. 52 | 53 | In the case of batch processing architecture, you can create async components that are independent of each other. 54 | 55 | ### Implement Elasticity 56 | 57 | The cloud brings a new concept of elasticity in your applications. Elasticity can be implemented in 3 ways.. 58 | 59 | 1. **Proactive Cyclic Scaling:** Periodic scaling that occurs at a fixed interval (daily, weekly, monthly, quarterly) 60 | 2. **Proactive Event-base Scaling:** Scaling just when you are expecting a big surge of traffic requests due to a scheduled business event (new product launch, marketing campaigns) 61 | 3. **Auto-scaling based on demain:** By using monitoring service, you system can send triggers to take appropriate actions so that if scales up or down based on metrics (utilization of servers or network I/O) 62 | 63 | ## The Well Architected Framework 64 | 65 | ### What is the well architected framework? 66 | 67 | This has been developed by the Solutions Architecture team based on their experience with helping AWS customers. The well architected framework is a set of questions that you can use to evaluate how well your architecture is aligned to AWS best practices. 68 | 69 | ### 5 Pillars of the Well Architected Framework 70 | 71 | - Security 72 | - Reliability 73 | - Performance Efficiency 74 | - Cost Optimization 75 | - Operation Excellence 76 | 77 | ### Structure of each pillar 78 | 79 | - Design Principles 80 | - Definition 81 | - Best Practices 82 | - Key AWS Services 83 | - Resources 84 | 85 | ### General Design Principles 86 | 87 | - Stop guessing your capacity needs 88 | - Test systems at production scale 89 | - Automate to make architectural experimentation easier 90 | - Allow for evolutionary architectures 91 | - Data-driven architectures 92 | - Improve through game days 93 | 94 | ## Pillar 1 - Security 95 | 96 | ### Design Principles 97 | 98 | - Apply security at all layers! 99 | - Enable traceability 100 | - Automate responses to security events 101 | - Focus on securing your system 102 | - Automate security best practices 103 | 104 | ### Definitions 105 | 106 | Security in the cloud consists of 4 areas... 107 | 108 | ### Data Protection 109 | 110 | Before you begin to architect security practices across your environment, **basic data classification should be in place**. You should organize and classify your data in to segments such as publicly available, available to only members of your organization, available to only certain members of your organization, available only to the board etc. 111 | 112 | You should also implement a least privilege access system so that people are only able to access what they need. However most importantly, you should encrypt everything where possible, whether it be at rest or in transit. 113 | 114 | **In AWS the following practices help to protect your data...** 115 | 116 | - AWS customers maintain full control over their data 117 | - AWS makes it easier for you to encrypt your data and manage keys, including regular key rotation, which can be easily automated natively by AWS or maintained by a customer. 118 | - Detailed logging is available that contains important content, such as file access and changes. 119 | - AWS has designed storage systems for exceptional resiliency. As an example, Amazon S3 is designed for 11, 9's of durability. (if you store 10,000 objects with AWS S3, you can on average expect to incur a loss of a single object once every 10,000,000 years) 120 | - Versioning, which can be part of a larger data lifecycle-management process, can protect against accidental overwrites, deletes and similar harm 121 | - AWS never initiates the movement of data between regions. Content placed in a region will remain in that region unless the customer explicitly enable a feature or leverages a service that provides that functionality 122 | 123 | **What questions should you be asking yourself?** 124 | 125 | - How are you encrypting your data at rest? 126 | - How are you encrypting your data in transit (SSL)? 127 | 128 | ### Privilege Management 129 | 130 | Privilege Management ensures that only authorized and authenticated users are able to access your resources, and only in a manner that is intended. 131 | 132 | **This can include** 133 | 134 | - Access Control Lists (ACLs) 135 | - Role Based Access Controls 136 | - Password Management (such as password rotation policies) 137 | 138 | **What questions should you be asking yourself?** 139 | 140 | - How are you protecting access to and use the AWS root account credentials? 141 | - How are you defining roles and responsibilities of system users to control human access to the AWS Management Console and APIs? 142 | - How are you limiting automated access (such as from applications, scripts, or 3rd party tools or services) to AWS resources? 143 | - How are you managing keys and credentials? 144 | 145 | ### Infrastructure Protection 146 | 147 | Outside of Cloud, this is how you protect your data center. RFID controls, security, lockable cabinets, CCTV etc. Within AWS they handle this so Infrastructure Protection exists at a VPC level. 148 | 149 | **What questions should you be asking yourself?** 150 | 151 | - How are you enforcing network and host-level boundary protection? 152 | - How are you enforcing AWS service level protection? 153 | - How are you protecting the integrity of the OS on your AWS EC2 instances? 154 | 155 | ### Detective Controls 156 | 157 | You can use detective controls to detect or identify a security breach. AWS Services to achieve this include 158 | 159 | - AWS Cloudtrail 160 | - AWS CloudWatch 161 | - AWS Config 162 | - AWS S3 163 | - AWS Glacier 164 | 165 | **What questions should you be asking yourself?** 166 | 167 | - How are you capturing and analyzing your logs? 168 | 169 | ### Key AWS Services 170 | 171 | 1. Data Protection 172 | - Encrypt both in transit and at rest using - ELB, EBS, S3 and RDS 173 | 2. Privilege Management 174 | - IAM, MFA 175 | 3. Infrastructure Protection 176 | - VPC 177 | 4. Detective Controls 178 | - AWS Cloud Trail, AWS Config, AWS Cloud Watch 179 | 180 | ## Pillar 2 - Reliability 181 | 182 | The reliability pillar covers the ability of a system to recover from service or infrastructure outages/disruptions as well as the ability to dynamically acquire computing resources to meet demand. 183 | 184 | - Test recovery procedures 185 | - Automatically recover from failure - Netflix SimianArmy 186 | - Scale horizontally increase aggregate system availability 187 | - Stop guessing capacity 188 | 189 | ### Definition 190 | 191 | Reliability in the cloud consists of 3 areas... 192 | 193 | 1. Foundations 194 | 2. Change Management 195 | 3. Failure Management 196 | 197 | ### Foundations 198 | 199 | With AWS, they handle most of the foundations for you. The cloud is designed to be essentially limitless meaning that AWS handle the networking and compute requirements themselves. However they do set the service limits to stop customers from accidentally over-provisioning resource 200 | 201 | https://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html 202 | 203 | **What questions should you be asking yourself?** 204 | 205 | - How are you managing AWS service limits for your account? 206 | - How are you planning your network topology on AWS? 207 | - Do you have an escalation path to deal with technical issues? 208 | 209 | ### Change Management 210 | 211 | You need to be aware of how change affects a system so that you can plan provocatively around it. Monitoring allows you to detect any changes to your environment and react. In traditional systems, change control is done manually and are carefully co-ordinated with auditing. 212 | 213 | With AWS things are a lot easier, you can use CloudWatch to monitor your environment and services such as autoscaling to automate change in response on your production environment 214 | 215 | **What questions should you be asking yourself?** 216 | 217 | - How does your system adapt to changes in demand? 218 | - How are you monitoring AWS resources? 219 | - How are you executing change management? 220 | 221 | ### Failure Management 222 | 223 | With cloud, you should always architect your systems with the assumptions that failure will occur. You should become aware of these failures, how they occurred, how to respond to them and then plan on how to prevent these from happening again. 224 | 225 | **What questions should you be asking yourself?** 226 | 227 | - How are you backing up your data? 228 | - How does your system withstand component failures? 229 | - How are you planning for recovery? 230 | 231 | ### Key AWS Services 232 | 233 | 1. Foundations 234 | - IAM, VPC 235 | 2. Change Management 236 | - AWS CloudTrail 237 | 3. Failure Management 238 | - AWS CloudFormation 239 | 240 | ## Pillar 3 - Performance Efficiency 241 | 242 | The Performance Efficiency pillar focuses on how to use computing resources efficiency to meet your requirements and how to maintain that efficiency as demand and technology evolves. 243 | 244 | ### Design Principles 245 | 246 | - Democratize advanced technologies 247 | - Go global in minutes 248 | - Use server-less architectures 249 | - Experiment more often 250 | 251 | ### Definition 252 | 253 | **Performance Efficiency in the cloud consists of 4 areas...** 254 | 255 | ### Compute 256 | 257 | When architecting your system it is important to choose the right kind of server!! 258 | 259 | Some applications require heavy CPU utilization, some require heavy memory utilization etc. 260 | 261 | With AWS servers are virtualized and at the click of a button (or API call) you can change the type of server in which your environment is running on. You can even switch to running with no servers at all and use AWS Lambda. 262 | 263 | **What questions should you be asking yourself?** 264 | 265 | - How do you select the appropriate instance type for your system? 266 | - How do you ensure that you continue to have the most appropriate instance type as new instance types and features are introduced? 267 | - How do you monitor your instances post launch to ensure they are performing as expected? 268 | - How do you ensure that the quantity of your instances match demand? 269 | 270 | ### Storage 271 | 272 | The optimal storage solutions for your environment depends on a number of factors 273 | 274 | - Access Methods - Block, File or Object 275 | - Patterns of Access - Random or Sequential 276 | - Throughput Required 277 | - Frequency of Access - Online, Offline or Archival 278 | - Frequency of Update - Worm, Dynamic 279 | - Availability Constraints 280 | - Durability Constraints 281 | 282 | At AWS the storage is virtualized. With S3 you can have 11 x 9's durability, Cross Region Replication etc. With EBS you can choose between storage mediums (SSD, Magnetic, PIOPS etc). 283 | You can also easily move volumes between the different types of storage mediums. 284 | 285 | **What questions should you be asking yourself?** 286 | 287 | - How do you select the appropriate storage solution for your system? 288 | - How do you ensure that you continue to have the most appropriate storage solution as new storage solution features are launched? 289 | - How do you monitor your storage solution to ensure it is performing as expected? 290 | - How do you ensure that the capacity and throughput of your storage solutions matches demand? 291 | 292 | ### Database 293 | 294 | The optimal database solution depends on a number of factors. Do you need database consistency, do you need high availability, do you need No-SQL, do you need DR etc. 295 | 296 | With AWS you get a LOT of options. RDS, DynamoDB, Redshift etc. 297 | 298 | **What questions should you be asking yourself?** 299 | 300 | - How do you select the appropriate database solution for your system? 301 | - How do you ensure that you continue to have the most appropriate database solution and features as new database solution and features are launched? 302 | - How do you monitor your databases to ensure performance is as expected? 303 | - How do you ensure the capacity and throughput of your databases matches demand? 304 | 305 | ### Space-time trade-off 306 | 307 | Using AWS you can use services such as RDS to add read replicas, reducing the load on your database and creating multiple copies of the database. This helps to lower latency. 308 | 309 | You can use the global infrastructure to have multiple copies of your environment, in regions that is closest to our customer base. You can also use caching services such as ElastiCache or CloudFront to reduce latency. 310 | 311 | **What questions should you be asking yourself?** 312 | 313 | - How do you select the appropriate proximity and caching solutions for your system? 314 | - How do you ensure that you continue to have the most appropriate proximity and caching solutions as new solutions are launched? 315 | - How do you monitor your proximity and caching solutions to ensure performance is as expected? 316 | - How do you ensure that the proximity and caching solutions you have matches demand? 317 | 318 | ### Key AWS Services 319 | 320 | 1. Compute 321 | - Autoscaling 322 | 2. Storage 323 | - EBS, S3, Glacier 324 | 3. Database 325 | - RDS, DynamoDB, Redshift 326 | 4. Space-time Trade-Off 327 | - Cloudfront, Elasticache, Direct Connect, RDS Read Replicas etc. 328 | 329 | ## Pillar 4 - Cost Optimization 330 | 331 | Use the Cost Optimization pillar to reduce your costs to a minimum and use those savings for other parts of your business. A cost-optimized system allows you to pay the lowest price possible while sitll achieving your business objectives. 332 | 333 | ### Design Principles 334 | 335 | - Transparently attribute expenditure 336 | - Use managed services to reduce cost of ownership 337 | - Trade capital expense for operating expense 338 | - Benefit from economies of scale 339 | - Stop spending money on data center operations 340 | 341 | ### Definition 342 | 343 | **Cost optimization in the cloud consists of 4 areas...** 344 | 345 | ### Matched supply and demand 346 | 347 | Try to optimally align supply with demand. Don't over provision or under provision, instead as demand grows, so should your supply of compute resources. Think of things like Autoscaling which scale with demand. 348 | 349 | Similarly, in a server-less context, use services such as Lambda that only execute when a request comes in. 350 | 351 | Services such as CloudWatch can also help you keep track as to what your demand is. 352 | 353 | **What questions should you be asking yourself?** 354 | 355 | - How do you make sure your capacity matches but does not substantially exceed what you need? 356 | - How are you optimizing your usage of AWS services? 357 | 358 | ### Cost-effective resources 359 | 360 | Using the correct instance type can be key to cost savings. For example you might have a reporting process that is running on a t2-Micro and it takes 7 hours to complete. That same process could be run on a an m4.2xlarge in a manner of minutes. The result remains the same but the t2.micro is more expensive because it ran for longer. 361 | 362 | A well architected system will use the most cost efficient resources to reach the end business goal 363 | 364 | **What questions should you be asking yourself?** 365 | 366 | - Have you selected the appropriate resource types to meet your cost targets? 367 | - Have you selected the appropriate pricing model to meet your cost targets? 368 | - Are there managed services (higher level services that Amazon EC2, Amazon EBS) that you can use improve your ROI? 369 | 370 | ### Expenditure Awareness 371 | 372 | With cloud you no longer have to go out and get quotes on physical servers, choose a supplier, have those resources delivered, installed, made available etc. You can provision things within seconds, however this comes with its own issues. 373 | 374 | Many organizations have different teams, each with their own AWS accounts. Being aware of what each team is spending and where is crucial to any well architected system. 375 | 376 | You can use cost allocation tags to track this, billing alerts as well as consolidated billing. 377 | 378 | **What questions should you be asking yourself?** 379 | 380 | - What access control and procedures do you have in place to govern AWS costs? 381 | - How are you monitoring usage and spending? 382 | - How do you decommission resources that you no longer need, or stop resources that are temporarily not needed? 383 | - How do you consider data-transfer charges when designing your architecture? 384 | 385 | ### Optimizing Over Time 386 | 387 | AWS moves FAST! There are hundreds of new services (and potentially 1000 new services this year). A service that you chose yesterday may not be the best service to be using today. 388 | 389 | For example, consider MySQL RDS, Aurora was launched at re:invent 2014 and is now out of preview. Aurora may be a better option now for your business because of its performance and redundancy. 390 | 391 | You should keep track of the changes made to AWS and constantly re-evaluate your existing architecture. You can do this by subscribing to AWS blog nd by using services such as Trusted Advisor. 392 | 393 | **What questions should you be asking yourself?** 394 | 395 | - How do you manage and/or consider the adoption of new services? 396 | 397 | ### Key AWS Services 398 | 399 | 1. Matched Supply and Demand 400 | - Autoscaling 401 | 2. Cost-effective resources 402 | - EC2 (reserved instances), AWS Trusted Advisor 403 | 3. Expenditure Awareness 404 | - CloudWatch Alarms, SNS 405 | 4. Optimizing Over Time 406 | - AWS Blog, AWS Trusted Advisor 407 | 408 | ## Pillar 5 - Operational Excellence 409 | 410 | The Operational Excellence pillar includes operational practices and procedures used to manage production workloads 411 | 412 | This includes how planned changes are executed, as well as responses to unexpected operational events. 413 | 414 | Change execution and responses should be automated. All processes and procedures of operational excellence should be documented, tested and regularly reviewed 415 | 416 | ### Design Principles 417 | 418 | - Perform operations with code 419 | - Align operations processes to business objectives 420 | - Make regular, small, incremental changes 421 | - Test for responses to unexpected events 422 | - Learn from operational events and failures 423 | - Keep operations procedures current 424 | 425 | ### Definition 426 | 427 | **There are 3 best practice areas of Operational Excellence in the cloud...** 428 | 429 | ### Preparation 430 | 431 | Effective preparation is required to drive operational excellence. Operations checklists will ensure that workloads are ready for production operation, and prevent unintentional production promotion without effective preparation. 432 | 433 | Workloads should have... 434 | 435 | **Runbooks** - operations guidance that operations teams can refer to so they can perform normal daily tasks. 436 | 437 | **Playbooks** - guidance for responding to unexpected operational events. Playbooks should include response plans, as well as escalation paths and stakeholder notifications. 438 | 439 | In AWS there are several methods, services and features that can be used to support operational readiness and the ability to prepare for normal day-to-day operations as well as unexpected operational events. 440 | 441 | **CloudFormation** can be used to ensure that environments contain all required resources when deployed to prod and the configuration of the environment is based on tested best practices, which reduces the opportunity for human error. 442 | 443 | **Autoscaling** or other automated scaling mechanisms will allow workloads to automatically respond when business-related events affect operational needs. 444 | 445 | **AWS Config** with the AWS Config rules feature create mechanisms to automatically track and respond to changes in your AWS workloads and environments 446 | 447 | It is also important to use features like **tagging** to make sure all resources in a workload can be easily identified when needed during operations and responses. 448 | 449 | **What preparation questions should you ask yourself for operational excellence?** 450 | 451 | - What best practices for cloud operations are your using? 452 | - How are you doing configuration management for your workload? 453 | 454 | Be sure that documentation doesn't become stale or out of date! Documentation should be thorough! 455 | 456 | Without application designs, environment configs, resource configs, response plans, and mitgation plans documentation is not complete. 457 | 458 | If documentation is not updated and tested regularly, it will not be useful when unexpected operational events occur. If workloads are not reviewed before production, operations will be affected when undetected issues occur. 459 | 460 | If resources are not documented, when operational events occur, determining how to respond will be more difficult while the correct resources are identified. 461 | 462 | ### Operation 463 | 464 | Operations should be standardized and manageable on a routine basis. The focus should be on automation, small frequent changes, regular QA testing, and defined mechanisms to track, audit, roll back and review changes. 465 | 466 | Changes should not be large and infrequent, they should not require scheduled downtime, and they should not require manual execution. A wide range of logs and metrics that are based on key operational indicators for a workload should be collected and reviewed to ensure continuous operations. 467 | 468 | **What questions should you be asking yourself for operational excellence?** 469 | 470 | - How are you evolving your workload while minimizing the impact of change? 471 | - How do you monitor your workload to ensure it is operating as expected? 472 | 473 | Routine operations, as well as responses to unplanned events, should be automated. Manual processes for deployments, release management, changes and rollbacks should **avoided**. 474 | 475 | Releases should **not** be large batches that are done infrequently. 476 | 477 | Rollbacks are more difficult in large changes, and failing ot have a rollback plan or the ability to mitigate failure impacts will prevent continuity of operations. 478 | 479 | Align monitoring to business needs, so that the responses are effective at maintaining business continuity. Monitoring that is ad hoc and not centralized, with responses that are manual, will cause more impact to operations during unexpected events. 480 | 481 | ### Response 482 | 483 | Responses to unexpected operational events should be automated. This is not just for alerting but also for mitigation, remediation, rollback and recovery. 484 | 485 | Alerts should be timely and should invoke escalations when response are not adequate to mitigate the impact of operational events. 486 | 487 | QA mechanisms should be in place to automatically roll back failed deployments. 488 | 489 | Responses should follow a pre-defined playbook that includes stakeholders, the escalation process and procedures. Escalation paths should be defined and include both functional and hierarchical escalation capabilities. Hierarchical escalation should be automated and escalated priority should result in stakeholder notifications. 490 | 491 | **What questions should you be asking yourself?** 492 | 493 | - How do respond to unplanned operational events? 494 | - How is escalation managed when responding to unplanned operational events? 495 | 496 | ### Key AWS Services 497 | 498 | 1. **Preparation** 499 | AWS Config provides a detailed inventory of your AWS resources and configuration, and continuously records configuration changes. AWS Service Catalog helps to create a standardized set of service offerings that are aligned to best practices. Designing workloads that use automation with services like AutoScaling, AWS SQS are good methods to ensure continuous operations in the event of unexpected operational events. 500 | 2. **Operations** 501 | AWS CodeCommit, AWS CodeDeploy and AWS CodePipeline can be used to manage and automate code changes to AWS workloads. Use AWS SDKs or 3rd party libs to automate operational changes. Use AWS CloudTrail to audit and track changes made to AWS environments 502 | 3. **Responses** 503 | Take advantage of all of the AWS CloudWatch service features for effective and automated responses. CloudWatch alarms can be used to set thresholds for alerting and notification and CloudWatch events can trigger notifications and automated responses. 504 | 505 | 506 | --------------------------------------------------------------------------------