├── 1000-ft-overview └── README.md ├── Additional-Exam-Tips └── README.md ├── Application Services ├── Exam Tips │ └── README.md ├── README.md └── kinesis-data-vis-sample-app.template ├── Databases ├── Exam-tips │ └── README.md ├── README.md └── scripts │ ├── connect.txt │ └── rds.sh ├── EC2 ├── Exam-tips │ └── README.md ├── README.md └── templates │ ├── Build-A-Serverless-Website │ ├── error.html │ ├── hellocloudgurus.py │ └── index.html │ ├── Using-Polly-To-Help-Lab │ ├── Change-to-Lambda-Console.txt │ └── pollyassets │ │ ├── bucketpolicypermissions.json │ │ ├── convertoaudio.py │ │ ├── getposts.py │ │ ├── index.html │ │ ├── lambdapolicy.json │ │ ├── mappings.json │ │ ├── newposts.py │ │ ├── sample.json │ │ ├── scripts.js │ │ └── styles.css │ ├── bootstrap.sh │ ├── bootstrapscript.sh │ ├── healthcheck.html │ └── index.html ├── IAM └── README.md ├── LICENSE ├── Object-Storage-and-CDN-S3-Glacier-Cloudfront ├── CDN-Cloudfront │ └── README.md ├── Exam-tips │ └── README.md ├── S3-Glacier │ └── README.md ├── Snowball │ └── README.md └── Storage-Gateway │ └── README.md ├── README.md ├── Route-53 ├── Exam-Tips │ └── README.md └── README.md ├── The-Well-Architected-Framework └── README.md └── VPC ├── Exam tips └── README.md └── README.md /1000-ft-overview/README.md: -------------------------------------------------------------------------------- 1 | # Section 2: 1,000 ft Overview 2 | 3 | ## Part 1. Regions, Availability Zones (AZ), Edge Locations 4 | 5 | ### Regions 6 | 7 | **AWS Region** is a physical, geographical area or location, consisting of 2 or more Availability Zones. 8 | 9 | **_Current regions across the world:_** 10 | 11 | - US East (N. Virginia) - `us-east-1` 12 | - US East (Ohio) - `us-east-2` 13 | - US West (Northern California) - `us-west-1` 14 | - US West (Oregon) - `us-west-2` 15 | - Canada (Central) - `ca-central-1` 16 | - EU (Frankfurt) - `eu-central-1` 17 | - EU (Ireland) - `eu-west-1` 18 | - EU (London) - `eu-west-2` 19 | - EU (Paris) - `eu-west-3` 20 | - Asia Pacific (Tokyo) - `ap-northeast-1` 21 | - Asia Pacific (Seoul) - `ap-northeast-2` 22 | - Asia Pacific (Osaka-Local) - `ap-northeast-3` 23 | - Asia Pacific (Singapore) - `ap-southeast-1` 24 | - Asia Pacific (Sydney) - `ap-southeast-2` 25 | - Asia Pacific (Mumbai) - `ap-south-1` 26 | - South America (Sao Paulo) - `sa-east-1` 27 | 28 | ### Availability Zones (AZ) 29 | 30 | **AWS Availability Zones** are one or more discrete data centers, each with redundant power, networking and connectivity housed in separate facilities. Deploying your application across multiple Availability Zones is useful for redundancy, low latency and fault tolerance. 31 | 32 | **_Regions with multiple Availability Zones:_** 33 | 34 | - US East 35 | - Ohio (3) 36 | - North Virginia (6) 37 | - US West 38 | - Oregon (3) 39 | - Northern California (3) 40 | - Canada 41 | - Central (3) 42 | - South America 43 | - Sao Paulo (3) 44 | - Europe 45 | - Ireland (3) 46 | - Frankfurt (3) 47 | - London (3) 48 | - Paris (3) 49 | - Asia Pacific 50 | - Singapore (3) 51 | - Seoul (2) 52 | - Tokyo (4) 53 | - Mumbai (2) 54 | - Sydney (3) 55 | - Beijing (2) 56 | - Ningxia (2) 57 | 58 | ### Edge Locations 59 | 60 | **AWS Edge Locations** are locations around the world meant for caching content, enhancing the user experience, reducing latency. Edge locations are specifically used by AWS Cloudfront and AWS CDN. Every Region is has its own set Availability Zone's and Edge Locations. 61 | 62 | ## Part 2. AWS Services Overview 63 | 64 | Compute: 65 | EC2 - elastic compute cloud 66 | EC2 Container Services - containerization docker 67 | Elastic Beanstalk - plug and play - for developers 68 | Lambda (server less) - code/functions uploaded to the cloud to run at different points 69 | Lightsail - plug and play 70 | Batch - batch computing in the cloud 71 | 72 | Storage: 73 | S3 - simple storage service - object based storage - buckets 74 | EFS - elastic file system 75 | Glacier - data archival 76 | Snowball - large amounts of data to aws data center 77 | Storage gateway - VM installed in datacenter or office - replicate info to S3 78 | 79 | Databases: 80 | RDS - relation database service - postgres, mysql, oracle 81 | DynamoDB - non relational db 82 | Elasticache - cache things from db 83 | Redshift - data warehousing business intelligence, complex queries 84 | 85 | Migration: 86 | AWS Migration Hub - tracking service for moving to aws 87 | Application Discover Service - track applications and dependency 88 | Database Migration Service - migrate db from on premise to AWS 89 | Server Migration Service - migrate server to AWS cloud 90 | Snowball - in between storage and migration 91 | 92 | Networking and Content Delivery: 93 | VPC (highlight) - Amazon virtual private cloud - virtual datacenter - configure avail zones, firewall, network acl etc. 94 | Cloudfront - AWS content delivery network, store assets specific regions around the world 95 | Route 53 - AWS DNS service - lookup ip to get ipv4 and ipv6 address 96 | API Gateway - Serverless way of creating own api 97 | Direct Connect - Dedicated line from office directly into amazon, connects to VPC 98 | 99 | Developer Tools: 100 | Codestart - project management, CI toolchain, collaborate 101 | Codecommit - store code, like github 102 | Codebuild - compile and run tests, produce package 103 | Code deploy - deployment service to ec2 instance 104 | Codepipeline - automate and visualize steps to release software 105 | X-ray - debug and analyze server less application 106 | Cloud9 - IDE environment in browser 107 | 108 | ## Part 3. AWS Services Overview (Continued) 109 | 110 | Management tools: 111 | Cloudwatch - Monitoring service 112 | Cloudformation - solutions architect specific - scripting infrastructure - turn infrastructure to code 113 | Cloudtrail - log changes to aws environment 114 | Config - monitors config of aws environment 115 | Opswork - similar to elastic beanstalk - chef and puppet to automate environments 116 | Service Catalog - manage a catalog of IT services 117 | Systems manager - interface for managing aws resources - group resources 118 | Trusted Advisor - advice around security, advice for aws services and resources, accountant like 119 | Managed Services - manage service for aws cloud 120 | 121 | ** Recap for exam - cloudformation, cloudtrail, cloudtrail, trusted advisor 122 | 123 | Media Services: 124 | Elastic transcoder - takes media and resizes on different devices 125 | Media convert - file based video transcoding with broadcast grade features 126 | Media live - broadcast grade live video processing service. tv internet connected multiscreen 127 | Media Package - protect content over internet 128 | Media Store - media storage, optimized for media 129 | Media Tailor - target advertising into video streams with out harming broadcast 130 | 131 | Machine Learning: 132 | Sage maker - easy for deep learning when coding for environment 133 | Comprehend - sentiment analysis on products. good or bad? 134 | Deep lens - computer vision on camera, recognition, physical piece of hardware 135 | Lex - powers alexa, AI 136 | Machine Learning - throw dataset to AWS cloud and predict outcome 137 | Polly - text to speech, voices sound real, accents 138 | Rekognition - upload file, tells you what is in the file 139 | Amazon translate - translate to other langs 140 | Amazon transcribe - hard of hearing, speech recognition, speech to text 141 | 142 | Analytics: 143 | Athena - SQL queries ins S3 buckets, serverless 144 | EMR - elastic map reduce - processing large amounts of data, chops data up for analysis 145 | Cloudsearch - search service 146 | Elastic Search service - search service 147 | Kinesis - solutions architect highlight, ingesting large amounts data 148 | Kinesis Video streams - ingesting streams and analyze 149 | Quicksight - business intelligence tool 150 | Datapipeline - moving data between different services 151 | Glue - ETL (extract transform load) 152 | 153 | ## Part 4. AWS Services Overview (Continued) 154 | 155 | Security Identity and Compliance: 156 | IAM - identity access management 157 | Cognito - device authentication, oath, after authenticated, use aws services 158 | Guard Duty - monitor for malicious activity 159 | Inspector - install on vm or instances, test against it, schedule 160 | Macie - Scan s3 buckets and looks for sensitive info and alert 161 | Certificate Manager - ssl cert for free, manage ssl cert 162 | Cloud HSM - cloud hardware security module - dedicate bits of hardware to store keys to authenticated 163 | Directory Service - integration ms active service to aws services 164 | WAF - web application firewall - at application layer to stop attacks, XSS, sql injection 165 | Shield - by default for cloud front - ddos mitigation, prevent ddos attacks 166 | Artifact - portal to download aws client reports, manage agreements 167 | 168 | **Key security services for exam: IAM, inspector, cloudHMS, directory services, waf, shield, cert manager 169 | 170 | Mobile Services: 171 | Mobile hub - management console for mobile app for aws services 172 | AWS Pinpoint - targeted push notifications 173 | AWS Appsync - atomically updates data in web or mobile in real time 174 | Device Farm - test apps on real device, iOS, android 175 | Mobile Analytics - analytics service for mobile 176 | 177 | AR/VR: 178 | Sumerian - tools to create environment, super new 179 | 180 | Application Integration: 181 | Step functions - manage lambda functions and ways to go through it 182 | Amazon MQ - message queue 183 | SNS - notification services 184 | SQS - decouple infrastructure, queue 185 | SWF - workflow job creation 186 | 187 | Customer Engagement: 188 | Connect - contact center as a service, call center 189 | Simple Email Service - email service, send grid, mailchimp 190 | 191 | Business Productivity: 192 | Alexa for business - manager for business needs 193 | Amazon chime - google hangouts like 194 | Work Docs - dropbox for AWS 195 | Work Mail - Office 365 like 196 | 197 | Desktop and App streaming: 198 | Workspaces - VDI solution, run OS in aws cloud 199 | App stream 2.0 - streaming application to desktop of device 200 | 201 | IOT: 202 | iOT - devices sending sensor information 203 | iOT Device Management - device management 204 | Amazon FreeRTOS - OS for microcontrollers 205 | Greengrass - ?? 206 | 207 | Game Development: 208 | Gamelift - service to develop games 209 | 210 | ## What Services Will Be Tested On The Exam?? 211 | 212 | Analytics 213 | Management Tools 214 | Migration 215 | Compute 216 | AWS Global infrastructure 217 | Storage 218 | Databases 219 | Network and Content delivery 220 | Security and Identity compliance 221 | Application Integration 222 | Desktop and App streaming 223 | 224 | ## Links 225 | 226 | - [https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html) 227 | 228 | - [https://www.linuxnix.com/amazon-aws-regions-vs-availability-zones-vs-edge-locations-vs-data-centers/](https://www.linuxnix.com/amazon-aws-regions-vs-availability-zones-vs-edge-locations-vs-data-centers/) -------------------------------------------------------------------------------- /Additional-Exam-Tips/README.md: -------------------------------------------------------------------------------- 1 | # Additional Exam Tips 2 | 3 | ## Based on Student Feedback... 4 | 5 | ### Kinesis 6 | 7 | - Used to consume Big Data 8 | - Stream large amounts of social media, news feeds, logs etc to the cloud 9 | - Think Kinesis when approached with big data questions 10 | 11 | - Process large amounts of data; 12 | - Redshift for business intelligence 13 | - Elastic Map Reduce for Big Data Processing 14 | 15 | 16 | ### EC2 - EBS Backed Volumes vs Instance Store Volumes 17 | 18 | - EBS backed volumes are persistent 19 | - Instance Store backed volumes are not persistent (ephemeral) 20 | - EBS Volumes can be detached and reattached to other EC2 instances 21 | 22 | - Instance store volumes cannot be detached and reattached to other instances - they exist only for the life of that instance. 23 | - EBS volumes can be stopped; data will persist 24 | 25 | - Instance store volumes cannot be stopped - if you do this the data will be wiped 26 | 27 | - EBS Backed = Store Data Long Term 28 | - Instance Store = Shouldn't be used for long-term data storage 29 | 30 | ### OpsWork 31 | 32 | - Orchestration Service that uses Chef 33 | - Chef consists of recipes to maintain a consistent state 34 | - Look for the term "chef" or "recipes" or "cook books" and think OpsWorks 35 | 36 | ### Elastic Transcoder 37 | 38 | - Media Transcoder in the cloud 39 | - Convert media files from their original source format in to different formats that will play on smartphones, tablets, PC's, etc 40 | - Provides transcoding presets for popular output formats, which means that you dont need to guess about which settings work best on particular devices 41 | - Pay based on the minutes that you transcode and the resolution at which you transcode 42 | 43 | ### SWF Actors 44 | 45 | - **Workflow Starter** - An application that can initiate (start) a workflow. Could be your e-commerce website when placing an order or a mobile app searching for bus times. 46 | - **Deciders** - Control the flow of activity tasks in a workflow execution. If something has finished in a workflow (or fails) a Decider decides what to do next. 47 | - **Activity Workers** - Carry out the activity tasks 48 | 49 | ### EC2 - Get Public IP Address 50 | 51 | - Query the instances metadata: 52 | - `curl http://169.254.169.254/latest/meta-data` 53 | - `wget http://169.254.169.254/latest/meta-data` 54 | - Key thing to remember is that its an instances META-DATA, not user data 55 | 56 | 57 | ## Consolidated Billing 58 | 59 | ### AWS Organizations 60 | 61 | AWS Organizations is an account management service that enables you to consolidate multiple AWS accounts into an organization that you create and centrally manage. 62 | 63 | Available in 2 feature sets 64 | - Consolidated Billing 65 | - All features 66 | 67 | **General Rules** 68 | 69 | - Paying account is independent 70 | - Cannot access resources of other accounts 71 | - All linked accounts are independent 72 | - Currently a limit of 20 linked accounts - can add more 73 | 74 | **Advantages** 75 | 76 | - One bill per AWS account 77 | - Very esy to track charges and allocate costs 78 | - Volume Pricing 79 | 80 | ### Best Practices 81 | 82 | - Always enable MFA on root account 83 | - Always use a strong and complex password on root account 84 | - Paying account should be used for billing purposes only. Do not deploy resources in to paying account 85 | 86 | ### Things to note 87 | 88 | - Billing Alerts 89 | - When monitoring is enabled on the paying account the billing data for all linked accounts is included 90 | - You can still create billing alerts per individual account 91 | 92 | - CloudTrail 93 | - Per AWS Account and is enabled per region 94 | 95 | - Can consolidate logs using an S3 bucket 96 | 1. Turn on CloudTrail in the paying account 97 | 2. Create a bucket policy that allows cross account access 98 | 3. Turn on CloudTrail in the other accounts and use the bucket in the paying account 99 | 100 | ### Tips 101 | 102 | - Consolidate billing allows you to get volume discounts on all your accounts. 103 | - Unused reserved instances for EC2 are applied across the group 104 | - CloudTrail is on a per account and per region basis but can be aggregated in to a single bucket in the paying account. 105 | 106 | ## Cross Account Access 107 | 108 | Many AWS customers use separate AWS accounts for their development and production resources. This separation allows them to cleanly separate different types of resources and can also provide some security benefits. 109 | 110 | Cross account access makes it easier for you to work productively within a multi-account (or multi-role) AWS environment by making it easy for you to switch roles within the AWS Management Console. 111 | 112 | You can sign in to the console using you IAM user name then switch the console using your IAM user name then switch the console to manage another account without having to enter (or remember) another user name and password 113 | 114 | ## Resource Groups & Tags 115 | 116 | Key Value Pairs attached to AWS resources 117 | 118 | Metadata (data about data) 119 | 120 | Tags can sometimes be inherited 121 | 122 | - Autoscaling, CloudFormation and Elastic Beanstalk can create other resources 123 | 124 | Resource groups make it easy to groups your resources using the tags that are assigned to them. You can group that share one or more tags. 125 | 126 | **Note: Container for resources** 127 | 128 | Resource groups contain information such as: 129 | - Region 130 | - Name 131 | - Health Checks 132 | 133 | Specific Information: 134 | - For EC2 - Public and Private IP Addresses 135 | - For ELB - Port Configurations 136 | - For RDS - Database Engine etc. 137 | 138 | ## VPC Peering 139 | 140 | **Note: Generally not tested in Associate exams, only in Professional exams** 141 | 142 | ### What is VPC Peering? 143 | 144 | VPC Peering is simply a connection between 2 VPCs that enables you to route traffic between them using private IP addresses. 145 | 146 | Instances in either VPC can communicate with each other as if they are within the same network. You can create a VPC peering connection between your own VPCs, or with a VPC in another AWS account within a **SINGLE REGION** 147 | 148 | AWS uses the existing infrastructure of a VPC to create a VPC peering connection; it is neither a gateway nor a VPN connection, and does not rely on separate piece of physical hardware. There is no single point of failure for communication or bandwidth bottleneck. 149 | 150 | ### VPC Peering Limitations 151 | 152 | 1. You cannot create a VPC peering connection between VPCs that have overlapping CIDR blocks ie. `10.0.0.0/16 -- X --> 10.0.0.0/24` 153 | 2. You cannot create a VPC peering connection between VPCs in different regions 154 | 3. VPC peering does not support transitive peering relationships. 155 | 156 | ## Direct Connect 157 | 158 | AWS Direct Connect makes it easier to establish a dedicated network connection from your premises to AWS. 159 | 160 | Using AWS Direct Connect, you can establish private connectivity between AWS and your datacenter, office or colocation environment, which in many cases can reduce your network costs, increase bandwidth throughput, and provide a more consistent network experience than internet-based connections. 161 | 162 | ### Main Benefits 163 | 164 | - Reduce costs when using large volumes of traffic 165 | - Increase reliability 166 | - Increase bandwidth 167 | 168 | ### How is Direct Connect different from a VPN? 169 | 170 | VPN Connections can be configured in minutes and are a good solution if you have and immediate need, have low to moderate bandwidth requirements, and can tolerate the inherent variability in Internet-based connectivity. 171 | 172 | AWS Direct Connect does not involve the Internet - instead, it uses dedicated, private network connections between your intranet and AWS VPC 173 | 174 | ### Direct Connect Connections 175 | 176 | Available in: 177 | - 10Gbs 178 | - 1Gbbs 179 | 180 | - Sub 1 Gbps can be purchased through AWS Direct Connect Partners 181 | - Uses Ethernet VLAN trunking (802.1Q) 182 | 183 | ## STS - Security Token Service 184 | 185 | Grants users limited and temporary access to AWS resources. Users can come from 3 sources 186 | 187 | ### Federation (typically Active Directory) 188 | 189 | - Uses Security Assertion Markup Language (SAML) 190 | - Grants temp access based off the users Active Directory credentials. Does not need to be a user in IAM 191 | - Single sign on allows users to log in to AWS console without assigning IAM credentials 192 | - Federation with mobile apps - Facebook/AWS/Google/OpenID providers 193 | - Cross Account Access - Access resources from one account to another 194 | 195 | ### Key Terms 196 | 197 | - Federation 198 | - Combining or joining a list of users in one domain (such as IAM) with a list of users in another domain (such as Active Directory, Facebook etc) 199 | - Identity Broker 200 | - A service that allows you to take an identity from point A and join it (federate it) with point B 201 | - Identity Store 202 | - Services like Active Directory, Facebook, Google etc 203 | - Identities 204 | - A user of a service like Facebook etc. 205 | 206 | **SCENARIO!** 207 | 208 | ``` 209 | You are hosting a company website on some EC2 web servers in your VPC. Users of the website must log in to the site which authenticates against the companies active directory servers which are based on site at the companies head quarters 210 | 211 | Your VPC is connected to your company HQ via a secure IPSEC VPN. Once logged in the user can only have access to their own S3 bucket. How do you set this up? 212 | ``` 213 | 214 | **SOLUTION!** 215 | 216 | 1. Users enter credentials (username and password) 217 | 2. Application calls identity broker - broker captures username and passwords 218 | 3. Broker checks with LDAP directory server - validates credentials 219 | 4. Call to STS (security token service) - getFederationToken function using IAM credentials 220 | 5. STS confirms policy and gives permission to create new tokens - returns 4 values 221 | - Access Key 222 | - Secret Access Key 223 | - Token 224 | - Duration (lifetime of token) 225 | 6. 4 values are sent back to application via broker 226 | 7. Application makes call to S3 227 | 8. S3 uses IAM to validate credentials 228 | 9. Credentials validated via IAM 229 | 230 | **In The Exam!** 231 | 232 | 1. Develop and Identity Broker to communicate with LDAP and AWS STS. 233 | 2. Identity Broker alway authenticates with LDAP first, THEN with AWS STS 234 | 3. Application then gets temp access to AWS resources 235 | 236 | ## Active Directory Tips 237 | 238 | ### Exam Questions 239 | 240 | **QUESITON: _Can you authenticate with Active Directory?_** 241 | 242 | **ANSWER: Yes. Using SAML** 243 | 244 | **QUESITON: _In what order do you authenticate to get the security credentials to log into Active Directory?_** 245 | 246 | **ANSWER: Authenticate with Active Directory first and then you are assigned the temp security credentials.** 247 | 248 | ## Workspaces 249 | 250 | It's basically a VDI (virtual development infrastructure). A Workspace is a cloud-based replacement for a traditional desktop. 251 | 252 | A Workspace is available as a bundle of compute resources, storage space, and software application access that allow a user to perform day-to-day tasks just like using a traditional desktop. 253 | 254 | A user can connect to a Workspace from any supported device (PC, Mac, Chromebook, iPad, KindleFire or Android Tablets) using free Amazon Workspaces client application and credentials set up by an administrator, or their existing Active Directory credentials if Amazon Workspaces is integrated with an existing Active Directory domain. 255 | 256 | ### Quick Facts 257 | 258 | - Windows 7 experience, provided by Windows Server 2008 R2 259 | - By default, users can personalize their workspaces. This can be locked down by an admin however 260 | - By default, you will be given local admin access, so you can install your own applications 261 | - Workspaces are persistent 262 | - All data on the D:\ is backed up every 12 hours 263 | - You do not need an AWS account to login into workspaces 264 | 265 | ## ECS 266 | 267 | - ECS is a regional service that you can use in one or more AZs across a new or existing, VPC to schedule the placement of containers across your cluster based on your resource needs, isolation policies, and availability requirements 268 | 269 | - ECS eliminates the need for you to operate your own cluster management and config management systems, or to worry about scaling your management infrastructure. 270 | 271 | - ECS can also be used to create a consistent deployment and build experience, manage and scale batch and ETL workloads, and build sophisticated application architectures on a microservice level. 272 | 273 | ### ECR (Elastic Container Registery) 274 | 275 | - Managed AWS Docker registery service that is secure, scalable and reliable. 276 | - Supports private Docker repos with resource based permissions using AWS IAM so that specific users or EC2 instances can access repos and images. 277 | - Developers can use the Docker CLI to push, pull and manage images. 278 | 279 | ### ECS Task Definitions 280 | 281 | - A Task Definition is required to run Docker containers in ECS. 282 | - Task Definitions are text files in JSON format that describe one or more containers that form your application. 283 | - Some of the params you can specify in a task definition include: 284 | - Which Docker images to yse with the containers in your task 285 | - How much CPU and memory to use with each container 286 | - Whether containers are linked together in a task 287 | - The Docker networking mode to use for the containers in your task 288 | - What (if any) ports from the container are mapped to the host container service 289 | - Whether the task should continue to run if the container finishes or fails 290 | - The command the container should run when it is started 291 | - What (if any) env variables should be passed to the container when it starts. 292 | - Any data volumes that should be used with containers in the task 293 | - What (if any) IAM role your tasks should use for permissions 294 | 295 | ### ECS Services 296 | 297 | - An ECS service allows you to run and maintain a specified number (or, the "desired count") of instances of a task definition simultaneously in and ECS cluster 298 | - Think of services like AutoScaling groups for ECS 299 | - If a task should fail or stop, the ECS service scheduler launches another instance of your task definition to replace it and maintain the desired count of tasks in the service. 300 | 301 | ### ECS Clusters 302 | 303 | - An ECS cluster is a logical grouping of container instances that you can place tasks on. 304 | - When you first use the Amazon ECS service, a default cluster is created for you, but you can create multiple clusters in an account to keep your resources separate. 305 | - **Concepts:** 306 | - Clusters can contain multiple different container instance types 307 | - Clusters are region-specifc 308 | - Container instances can only be part of one cluster at a time. 309 | - You can create IAM policies for your clusters to allow or restrict users' access to specific clusters 310 | 311 | ### ECS Scheduling 312 | 313 | - **Service Scheduler:** 314 | - Ensures that the specific number of tasks are constantly running and reschedules tasks when a task fails (for example, if the underlying container instance fails for some reason) 315 | - Can ensure tasks are registered against and ELB 316 | - **Custon Scheduler:** 317 | - You can create your own schedulers that meet your business needs. 318 | - Leverage 3rd party schedulers such as Blox 319 | - The ECS schedulers leverage the same cluster state information provided by the ECS API to make appropriate placement decisions 320 | 321 | ### ECS Container Agent 322 | 323 | ECS Container Agent allows container instances to connect to your cluster. ECS Container Agent is included in the ECS optimized AMI, but you can also install it on any EC2 instance that supports ECS specs. ECS Container Agent is only supported on EC2 instances. 324 | 325 | - Pre installed on special ECS AMIs 326 | - Linux based: 327 | - Works with AWS Linux, Ubuntu, Redhat, CentOS, etc. 328 | - Will **not** work with Windows 329 | 330 | ### ECS Security 331 | 332 | - IAM Roles: 333 | - EC2 instances use an IAM role to access ECS 334 | - ECS tasks use an IAM role to access services and resources 335 | - Security Groups attach at the instance-level (i.e. the host - not the task or container) 336 | - You can access and configure the OS of the EC2 instances in your ECS cluster 337 | 338 | ### ECS Limits 339 | 340 | - Soft Limits: 341 | - Clusters per Region (default = 100) 342 | - Instances per Cluster (default = 100) 343 | - Services per Cluster (default = 100) 344 | - Hard Limits: 345 | - One Load Balancer per Service 346 | - 1000 Tasks per Service ("desired") 347 | - Max 10 Containers per Task Defintion 348 | - Max 10 Tasks per Instance (host) 349 | -------------------------------------------------------------------------------- /Application Services/Exam Tips/README.md: -------------------------------------------------------------------------------- 1 | # Exam Tips 2 | 3 | ## SQS 4 | 5 | - SQS is a distributed message queuing system 6 | - Allows you to decouple the components of an application so that they are independent 7 | - Pull-based, not push- based 8 | - Standard queues (default) - best effort ordering; message delivered at least once 9 | - FIFO Queues (First In First Out) - ordering strictly preserved, message delivered once, no duplicates. eg. good for banking transactions which need to happen in strict order. 10 | 11 | **NOTE:** Read FAQ section of SQS to help with exam 12 | 13 | ## SNS 14 | 15 | **Subscribers:** 16 | 17 | - HTTP 18 | - HTTPS 19 | - Email 20 | - Email-JSON 21 | - SQS 22 | - Application 23 | - Lambda 24 | 25 | ## API Gateway 26 | 27 | - Remember what API Gateway is at a high level 28 | - API Gateway has caching capabilities to increase performance 29 | - API Gateway is low cost and scales automatically 30 | - You can throttle API Gateway to prevent attacks 31 | - You can log results to CloudWatch 32 | - If you are using JS/AJAX that uses multiple domains with API Gateway, ensure that you have CORS enabled on API Gateway 33 | 34 | ## Kinesis 35 | 36 | - Know the difference between Kinesis Streams and Kinesis Firehose. You will be given scenario questions and you must choose the most relevant service 37 | 38 | - High level understanding of Kinesis Analytics -------------------------------------------------------------------------------- /Application Services/README.md: -------------------------------------------------------------------------------- 1 | # Application Services 2 | 3 | ## SQS - Simple Queue Service 4 | 5 | First **EVER** AWS Service! 6 | 7 | Amazon SQS is a web service that gives you access to a message queue that can be used to store messages while waiting for a computer to process them. 8 | 9 | Amazon SQS is a distributed queue sustem that enables web service applications to quickly and reliably queue messages that one component in the application generates to be consumed by another component. A queue is a temporary repository for messages that are awaiting processing. 10 | 11 | ### SQS Breakdown 12 | 13 | Using Amazon SQS, you can decouple the components of an application so they run independentlym easing message management between components 14 | 15 | Any component of a distributed application can store messages in the queue. Messages can contain up to 256Kb of text in any format. Any component can later retrieve the messages programatically using the SQS API 16 | 17 | ### What do you mean by "Queue"? 18 | 19 | The queue acts as a buffer between the component producing and saving data, and the component receiving the data for processing. This means the queue resolves issues that arise if the producer is producing faster than the consumer can process it, of if the producer or consumer are only intermittently connected to the network. 20 | 21 | ### Queue Types 22 | 23 | ### Standard Queue (default) 24 | 25 | Amazon SQS offers standard as the default queue type. A standard queue lets you have a nearly-unlimited number of transactions per second. Standard queues guarantee that a message is delivered at least once. However, because of the highly distributed architecture that allows high throughput, more than one copy of a message might be delivered out of order. Standard queues provide best effort ordering which ensures that messages are generally delivered in the same order as they are sent. 26 | 27 | ### FIFO Queues (First In, First Out) 28 | 29 | The FIFO queue complements the standard queue. The most important features of this queue type are FIFO delivery and exactly one processing: The order in which messages are sent and received is strictly preserved and a message is delivered once and remains available until a consumer processes and deletes it; duplicates are not introduced into the queue. FIFO queues also support message groups that allow multiple ordered message groups within a single queue. FIFO queues are limited to 300 transactions per second, but have all the capabilities of standard queues 30 | 31 | ```|_5_| ---> |_4_| ---> |_3_| ---> |_2_| ---> |_1_|``` 32 | 33 | ### Key Facts 34 | 35 | - SQS is pull-based, not pushed based 36 | - Messages are 256Kb in size 37 | - Messages can be kept in the queue from 1 minute to 14 days 38 | - Default retention period is 4 days 39 | - SQS guarantees that your messages will be processed at least once. 40 | 41 | ### Visibility Timeout 42 | 43 | - The Visibility Timeout is the amount of time that the message is invisible in the SQS queue after the reader picks up that message. Provided the job is processed before the visibility timeout expires, the message will then be deleted from the queue. If the job is not processed within that time, the message will become visible again and another reader/worker will process it. This could result in the same message delivered twice 44 | - Default visibility timeout is 30 seconds 45 | - Increase it if your task takes >30 seconds 46 | - Maximum is 12 hours 47 | 48 | ### Long Polling 49 | 50 | - Amazon SQS long polling is a way to retrieve messages from your Amazon SQS queues 51 | - While the regular short polling returns immediately (even if the message queue being polled is empty), long polling doesn't return a repsonse until a message arrives in the message queue, or the long poll times out. 52 | - Waits til message is in the queue. 53 | - As such, long polling saves you money. 54 | 55 | ## SWF - Simple Workflow Service 56 | 57 | Amazon Simple Workflow Service is a web service that makes it easy to coordinate work across distributed application components. Amazon SWF enables applications for a range of use cases, including media processing, web application back-ends, business process workflows, and analytics pipelines, to be designed as a coordination of tasks. 58 | 59 | Tasks represent invocations of various processing steps in an application which can be performed by executable code, web service calls, human actions, scripts. 60 | 61 | ### Starters 62 | 63 | An application that can initiate a workflow. Could be your e-commerce website when placing an order or a mobile app searching for bus times 64 | 65 | ### Workers 66 | 67 | Workers are programs that interact with Amazon SWF to get tasks, process received tasks and return results. 68 | 69 | ### Deciders 70 | 71 | The decider is a program that controls the coordination of tasks, ie their ordering, concurrency and scheduling according to the application logic. 72 | 73 | 74 | ### Workers and Deciders Interaction 75 | 76 | The workers and the decider can run on cloud infrastructure, such as Amazon EC2, or on machines behind firewalls, Amazon SWF brokers the interactions between workers and the decider. It allows the decider to get consistent views into the progress of tasks and to initiate new tasks in an ongoing manner. 77 | 78 | At the same time, Amazon SWF stores tasks, assigns them to workers when they are ready and monitors their progress. It ensures that a task is assigned **ONLY ONCE** and is **NEVER DUPLICATED** (key difference from SQS). 79 | 80 | Since Amazon SWF maintains the applications state durably, workers and deciders dont have to keep track of execution state. They can run independently, and scale quickly. 81 | 82 | ### SWF Domains 83 | 84 | Your workflow and activity types and the workflow execution itself are all scoped to a domain. Domains isolate a set of types, executions, and task lists from others within the same account. 85 | 86 | You can register a domain by using the AWS Management Console or by using the Register Domain action inthe Amazon SWF API. 87 | 88 | Maximum workflow can be 1 year and the value is always measured in seconds 89 | 90 | _JSON Domain Registration Example_ 91 | 92 | ```JSON 93 | { 94 | "name": "92034", 95 | "description": "music", 96 | "workflowExecutionRetentionPeriodInDays": "60" 97 | } 98 | ``` 99 | 100 | ### SWF vs. SQF 101 | 102 | - Amazon SWF has a retention period of 1 year vs SQS's 14 days retention 103 | - Amazon SWF presents a task-oriented API, whereas Amazon SQS offers a message-oriented API 104 | - Amazon SWF ensures that a task is assigned **ONLY ONCE** and is **NEVER DUPLICATED**. With SQS, you need to handle duplicated messages and may also need to ensure that a message is processed only once. 105 | - Amazon SWF keeps track of all the tasks and events in an application. With SQS, you need to implement your own application level tracking, especially if your application uses multiple queues. 106 | 107 | ## SNS - Simple Notification Service 108 | 109 | SNS is a web service that makes it easy to set up, operate and send notifications from the cloud. It provides developers with a highly scalable, flexible and cost-effective capability to publish messages from an application and immediately deliver them to subscribers or their applications 110 | 111 | May push notifications to Apple, Google, Fire OS and Windows devices as well as Android devices in China with Baidu Cloud Push. 112 | 113 | Besides pushing cloud notifications directly to mobile devices, SNS can also deliver notifications by SMS text message or email, to SQS queues, or to any HTTP endpoint. 114 | 115 | SNS notifications can also trigger Lambda functions. When a messge is published to and SNS topic that has a Lambda function subscribed to it, the Lambda function is invoked with the payload of the published message. The Lambda function receives the message payload as an input parameter and can manipulate the information in the message, publish the message to other SNS topics, or send the message to other AWS services. 116 | 117 | ### SNS Structure 118 | 119 | SNS allows you to group multiple recipients using topics. A topic is an "access point" for allowing recipients to dynamically subscribe for identical copies of the same notification. 120 | 121 | One topic can support deliveries to multiple endpoint types - for example, you can group together iOS, Android and SMS recipients. When you publish once to a topic, SNS delivers appropriately formatted copies of your message to each subscriber. 122 | 123 | To prevent messages from being lost, all messages published to SNS are stored redundantly across multiple availability zones. 124 | 125 | ### Subscribers - Who may subscribe to notifications? 126 | 127 | - HTTP 128 | - HTTPS 129 | - Email 130 | - Email-JSON 131 | - SQS 132 | - Application 133 | - Lambda 134 | 135 | 136 | ### SNS Benefits 137 | 138 | - Instantaneous, push-based delivery (no polling) 139 | - Simple APIs and easy integration with applications 140 | - Flexible message delivery over multiple transport protocols 141 | - Inexpensive, pay-as-you-go model with no up-front costs 142 | - Web-based AWS Management Console offers the simplicity of a point-and-click interface 143 | 144 | ### SNS vs SQS 145 | 146 | - Both messaging services in AWS 147 | - SNS = push; SQS = polls (pulls) 148 | 149 | ### Pricing 150 | 151 | - User pays $0.50 per 1 million SNS Requests 152 | - $0.06 per 100,000 notification deliveries over HTTP 153 | - $0.75 per 100 notifications deliveries over SMS 154 | - $2.00 per 100,000 notification deliveries over email 155 | 156 | ## Elastic Transcoder 157 | 158 | - Media Transcoder in the cloud. 159 | - Convert media files from their original source format in to different formats that will play on smarphones, tablets, PCs etc. 160 | - Provides transcoding presets for popular output formats, which means that you don't need to guess about which settings work bets on particular devices. 161 | - Pay based on the minutes that you transcode and the resolution at which you transcode. 162 | 163 | ## API Gateway 164 | 165 | API Gatewayis a fully managed service that makes it easy for developers to publish, maintain, monitor and secure APIs at any scale. With a few clicks in the AWS Management Console, you can create and API that acts as a "front door" for applications to access data, business logic, or functionality from you back-end services, such as applications running on EC2, code running on Lambda or any web application. 166 | 167 | ### Caching 168 | 169 | You can enable API caching in API Gateway to cache your endpoints response. With caching, you can reduce the number of calls made to your endpoint and also improve the latency of the requests to your API. 170 | 171 | When you enable caching for a stage, API Gateway caches responses from your endpoint for a specified TTL period, in seconds. API Gateway then responds to the request by looking up the endpoint response from the cache instead of making a request to your endpoint. 172 | 173 | - Low cost & efficient 174 | - Scales effortlessly 175 | - You can throttle requests to prevent attacks 176 | - Connect to Cloudwatch to log all requests 177 | 178 | ## Kinesis 179 | 180 | ### What is streaming data? 181 | 182 | Streaming data is data that is generated continuously by thousands of data sources, which typically send in the data records simultaneously, and in small sizes (order of KB) 183 | 184 | **Examples of usage:** 185 | 186 | - Purchases from online stores 187 | - Stock prices 188 | - Game data 189 | - Social network data 190 | - Geospatial data - uber, google maps 191 | - iOT data 192 | 193 | ### What is Kinesis? 194 | 195 | AWS Kinesis is a platform on AWS to send your streaming data to. Kinesis makes it easy to load and analyze streaming data, and also providing the ability for you to build your own custom applications for your business needs. 196 | 197 | ### Core Kinesis Services? 198 | 199 | #### Kinesis Streams 200 | 201 | - Streams consist of shards 202 | - 5 transactions per second for reads, up to a maximum total data read rate of 2Mb per second and up to 1,000 records per second for writes, up to a maximum total data write rate of 1 Mb per second (including partition keys). 203 | - The data capacity of your stream is a function of the number of shards that you specify for the stream. The total capacity of the stream is the sum of the capacities of its shards. 204 | 205 | #### Kinesis Firehose 206 | 207 | - Handles stream data automatically, no need to specify shards. 208 | 209 | #### Kinesis Analytics 210 | 211 | - Allows you to run SQL queries, analyzing the data and store said data in to another storage service like S3 212 | 213 | 214 | -------------------------------------------------------------------------------- /Application Services/kinesis-data-vis-sample-app.template: -------------------------------------------------------------------------------- 1 | { 2 | "AWSTemplateFormatVersion" : "2010-09-09", 3 | 4 | "Description" : "The Amazon Kinesis Data Visualization Sample Application", 5 | 6 | "Parameters" : { 7 | "InstanceType" : { 8 | "Description" : "EC2 instance type", 9 | "Type" : "String", 10 | "Default" : "t2.micro", 11 | "AllowedValues" : [ "t2.micro", "t2.small", "t2.medium", "m3.medium", "m3.large", "m3.xlarge", "m3.2xlarge", "c3.large", "c3.xlarge", "c3.2xlarge", "c3.4xlarge", "c3.8xlarge" ], 12 | "ConstraintDescription" : "must be a supported EC2 instance type for this template." 13 | }, 14 | 15 | "KeyName" : { 16 | "Description" : "(Optional) Name of an existing EC2 KeyPair to enable SSH access to the instance. If this is not provided you will not be able to SSH on to the EC2 instance.", 17 | "Type" : "String", 18 | "Default" : "", 19 | "MinLength" : "0", 20 | "MaxLength" : "255", 21 | "AllowedPattern" : "[\\x20-\\x7E]*", 22 | "ConstraintDescription" : "can contain only ASCII characters." 23 | }, 24 | 25 | "SSHLocation" : { 26 | "Description" : "The IP address range that can be used to SSH to the EC2 instances", 27 | "Type" : "String", 28 | "MinLength" : "9", 29 | "MaxLength" : "18", 30 | "Default" : "0.0.0.0/0", 31 | "AllowedPattern" : "(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\\d{1,2})", 32 | "ConstraintDescription" : "must be a valid IP CIDR range of the form x.x.x.x/x." 33 | }, 34 | 35 | "ApplicationArchive" : { 36 | "Description" : "A publicly accessible URL to the sample application archive as produced by 'mvn package'", 37 | "Type" : "String", 38 | "MinLength" : "7", 39 | "MaxLength" : "255", 40 | "Default" : "https://github.com/awslabs/amazon-kinesis-data-visualization-sample/releases/download/v1.1.1/amazon-kinesis-data-visualization-sample-1.1.1-assembly.zip" 41 | } 42 | }, 43 | 44 | "Conditions": { 45 | "UseEC2KeyName": {"Fn::Not": [{"Fn::Equals" : [{"Ref" : "KeyName"}, ""]}]} 46 | }, 47 | 48 | "Mappings" : { 49 | "AWSInstanceType2Arch" : { 50 | "t2.micro" : { "Arch" : "64" }, 51 | "t2.small" : { "Arch" : "64" }, 52 | "t2.medium" : { "Arch" : "64" }, 53 | "m3.medium" : { "Arch" : "64" }, 54 | "m3.large" : { "Arch" : "64" }, 55 | "m3.xlarge" : { "Arch" : "64" }, 56 | "m3.2xlarge" : { "Arch" : "64" }, 57 | "c3.large" : { "Arch" : "64" }, 58 | "c3.xlarge" : { "Arch" : "64" }, 59 | "c3.2xlarge" : { "Arch" : "64" }, 60 | "c3.4xlarge" : { "Arch" : "64" }, 61 | "c3.8xlarge" : { "Arch" : "64" } 62 | }, 63 | 64 | "AWSRegionArch2AMI" : { 65 | "us-east-1" : { "64" : "ami-76817c1e" }, 66 | "us-west-2" : { "64" : "ami-d13845e1" }, 67 | "eu-west-1" : { "64" : "ami-892fe1fe" }, 68 | "ap-southeast-1" : { "64" : "ami-a6b6eaf4" }, 69 | "ap-southeast-2" : { "64" : "ami-d9fe9be3" }, 70 | "ap-northeast-1" : { "64" : "ami-29dc9228" } 71 | } 72 | }, 73 | 74 | "Resources" : { 75 | "KinesisStream" : { 76 | "Type" : "AWS::Kinesis::Stream", 77 | "Properties" : { 78 | "ShardCount" : "2" 79 | } 80 | }, 81 | 82 | "KCLDynamoDBTable" : { 83 | "Type" : "AWS::DynamoDB::Table", 84 | "Properties" : { 85 | "AttributeDefinitions" : [ 86 | { 87 | "AttributeName" : "leaseKey", 88 | "AttributeType" : "S" 89 | } 90 | ], 91 | "KeySchema" : [ 92 | { 93 | "AttributeName" : "leaseKey", 94 | "KeyType" : "HASH" 95 | } 96 | ], 97 | "ProvisionedThroughput" : { 98 | "ReadCapacityUnits" : "10", 99 | "WriteCapacityUnits" : "5" 100 | } 101 | } 102 | }, 103 | 104 | "CountsDynamoDBTable" : { 105 | "Type" : "AWS::DynamoDB::Table", 106 | "Properties" : { 107 | "AttributeDefinitions" : [ 108 | { 109 | "AttributeName" : "resource", 110 | "AttributeType" : "S" 111 | }, 112 | { 113 | "AttributeName" : "timestamp", 114 | "AttributeType" : "S" 115 | } 116 | ], 117 | "KeySchema" : [ 118 | { 119 | "AttributeName" : "resource", 120 | "KeyType" : "HASH" 121 | }, 122 | { 123 | "AttributeName" : "timestamp", 124 | "KeyType" : "RANGE" 125 | } 126 | ], 127 | "ProvisionedThroughput" : { 128 | "ReadCapacityUnits" : "10", 129 | "WriteCapacityUnits" : "5" 130 | } 131 | } 132 | }, 133 | 134 | "Ec2SecurityGroup" : { 135 | "Type" : "AWS::EC2::SecurityGroup", 136 | "Properties" : { 137 | "GroupDescription" : "Enable SSH access and HTTP access on the inbound port", 138 | "SecurityGroupIngress" : 139 | [{ "IpProtocol" : "tcp", "FromPort" : "22", "ToPort" : "22", "CidrIp" : { "Ref" : "SSHLocation"} }, 140 | { "IpProtocol" : "tcp", "FromPort" : "80", "ToPort" : "80", "CidrIp" : "0.0.0.0/0"}] 141 | } 142 | }, 143 | 144 | "EIP" : { 145 | "Type" : "AWS::EC2::EIP", 146 | "Properties" : { 147 | "InstanceId" : { "Ref" : "Ec2Instance" } 148 | } 149 | }, 150 | 151 | "RootRole": { 152 | "Type" : "AWS::IAM::Role", 153 | "Properties" : { 154 | "AssumeRolePolicyDocument": { 155 | "Version" : "2012-10-17", 156 | "Statement" : [ { 157 | "Effect" : "Allow", 158 | "Principal" : { 159 | "Service" : [ "ec2.amazonaws.com" ] 160 | }, 161 | "Action" : [ "sts:AssumeRole" ] 162 | } ] 163 | }, 164 | "Path" : "/" 165 | } 166 | }, 167 | 168 | "RolePolicies" : { 169 | "Type" : "AWS::IAM::Policy", 170 | "Properties" : { 171 | "PolicyName" : "root", 172 | "PolicyDocument" : { 173 | "Version" : "2012-10-17", 174 | "Statement" : [ { 175 | "Effect" : "Allow", 176 | "Action" : "kinesis:*", 177 | "Resource" : { "Fn::Join" : [ "", [ "arn:aws:kinesis:", { "Ref" : "AWS::Region" }, ":", { "Ref" : "AWS::AccountId" }, ":stream/", { "Ref" : "KinesisStream" } ]]} 178 | }, { 179 | "Effect" : "Allow", 180 | "Action" : "dynamodb:*", 181 | "Resource" : { "Fn::Join" : [ "", [ "arn:aws:dynamodb:", { "Ref" : "AWS::Region" }, ":", { "Ref" : "AWS::AccountId" }, ":table/", { "Ref" : "KCLDynamoDBTable" } ]]} 182 | }, { 183 | "Effect" : "Allow", 184 | "Action" : "dynamodb:*", 185 | "Resource" : { "Fn::Join" : [ "", [ "arn:aws:dynamodb:", { "Ref" : "AWS::Region" }, ":", { "Ref" : "AWS::AccountId" }, ":table/", { "Ref" : "CountsDynamoDBTable" } ]]} 186 | }, { 187 | "Effect" : "Allow", 188 | "Action" : "cloudwatch:*", 189 | "Resource" : "*" 190 | } ] 191 | }, 192 | "Roles" : [ { "Ref": "RootRole" } ] 193 | } 194 | }, 195 | 196 | "RootInstanceProfile" : { 197 | "Type" : "AWS::IAM::InstanceProfile", 198 | "Properties" : { 199 | "Path" : "/", 200 | "Roles" : [ { "Ref": "RootRole" } ] 201 | } 202 | }, 203 | 204 | "Ec2Instance": { 205 | "Type" : "AWS::EC2::Instance", 206 | "Metadata" : { 207 | "AWS::CloudFormation::Init" : { 208 | "config" : { 209 | "packages" : { 210 | "yum" : { 211 | "java-1.7.0-openjdk" : [] 212 | } 213 | }, 214 | "files" : { 215 | "/var/kinesis-data-vis-sample-app/watchdog.sh" : { 216 | "content" : {"Fn::Join" : ["", [ 217 | "#!/bin/bash\n", 218 | "if ! ps aux | grep HttpReferrerCounterApplication | grep -v grep ; then\n", 219 | " # Launch the Kinesis application for counting HTTP referrer pairs\n", 220 | " java -cp /var/kinesis-data-vis-sample-app/lib/\\* com.amazonaws.services.kinesis.samples.datavis.HttpReferrerCounterApplication ", { "Ref" : "KCLDynamoDBTable" }, " ", { "Ref" : "KinesisStream" }, " ", { "Ref" : "CountsDynamoDBTable" }, " ", { "Ref" : "AWS::Region" }, " &>> /home/ec2-user/kinesis-data-vis-sample-app-kcl.log &\n", 221 | "fi\n", 222 | "if ! ps aux | grep HttpReferrerStreamWriter | grep -v grep ; then\n", 223 | " # Launch our Kinesis stream writer to fill our stream with generated HTTP (resource, referrer) pairs.\n", 224 | " # This will create a writer with 5 threads to send records indefinitely.\n", 225 | " java -cp /var/kinesis-data-vis-sample-app/lib/\\* com.amazonaws.services.kinesis.samples.datavis.HttpReferrerStreamWriter 5 ", { "Ref" : "KinesisStream" }, " ", { "Ref" : "AWS::Region" }, " &>> /home/ec2-user/kinesis-data-vis-sample-app-publisher.log &\n", 226 | "fi\n", 227 | "if ! ps aux | grep WebServer | grep -v grep ; then\n", 228 | " # Launch the webserver\n", 229 | " java -cp /var/kinesis-data-vis-sample-app/lib/\\* com.amazonaws.services.kinesis.samples.datavis.WebServer 80 /var/kinesis-data-vis-sample-app/wwwroot ", { "Ref" : "CountsDynamoDBTable" }, " ", { "Ref" : "AWS::Region" }, " &>> /home/ec2-user/kinesis-data-vis-sample-app-www.log &\n", 230 | "fi\n" 231 | ]]}, 232 | "mode" : "000755", 233 | "owner" : "ec2-user", 234 | "group" : "ec2-user" 235 | }, 236 | "/var/kinesis-data-vis-sample-app/crontask" : { 237 | "content" : {"Fn::Join" : ["", [ 238 | "* * * * * bash /var/kinesis-data-vis-sample-app/watchdog.sh\n" 239 | ]]}, 240 | "mode" : "000644", 241 | "owner" : "ec2-user", 242 | "group" : "ec2-user" 243 | } 244 | }, 245 | "sources": { 246 | "/var/kinesis-data-vis-sample-app" : { "Ref" : "ApplicationArchive" } 247 | } 248 | } 249 | } 250 | }, 251 | 252 | "Properties" : { 253 | "KeyName" : { "Fn::If" : [ "UseEC2KeyName", { "Ref" : "KeyName" }, { "Ref" : "AWS::NoValue" } ]}, 254 | "ImageId" : { "Fn::FindInMap" : [ "AWSRegionArch2AMI", { "Ref" : "AWS::Region" }, 255 | { "Fn::FindInMap" : [ "AWSInstanceType2Arch", { "Ref" : "InstanceType" }, 256 | "Arch" ] } ] }, 257 | "InstanceType" : { "Ref" : "InstanceType" }, 258 | "SecurityGroups" : [{ "Ref" : "Ec2SecurityGroup" }], 259 | "IamInstanceProfile": { "Ref": "RootInstanceProfile" }, 260 | "UserData" : { "Fn::Base64" : { "Fn::Join" : ["", [ 261 | "#!/bin/bash\n", 262 | "yum update -y aws-cfn-bootstrap\n", 263 | 264 | "/opt/aws/bin/cfn-init -s ", { "Ref" : "AWS::StackId" }, " -r Ec2Instance ", 265 | " --region ", { "Ref" : "AWS::Region" }, "\n", 266 | 267 | "# Register watchdog script with cron\n", 268 | "crontab /var/kinesis-data-vis-sample-app/crontask\n", 269 | 270 | "# Launch watchdog script immediately so if it fails this stack fails to start\n", 271 | "/var/kinesis-data-vis-sample-app/watchdog.sh\n", 272 | 273 | "/opt/aws/bin/cfn-signal -e $? '", { "Ref" : "WaitHandle" }, "'\n" 274 | ]]}} 275 | } 276 | }, 277 | 278 | "WaitHandle" : { 279 | "Type" : "AWS::CloudFormation::WaitConditionHandle" 280 | }, 281 | 282 | "WaitCondition" : { 283 | "Type" : "AWS::CloudFormation::WaitCondition", 284 | "DependsOn" : "Ec2Instance", 285 | "Properties" : { 286 | "Handle" : {"Ref" : "WaitHandle"}, 287 | "Timeout" : "600" 288 | } 289 | } 290 | }, 291 | "Outputs" : { 292 | "URL" : { 293 | "Description" : "URL to the sample application's visualization", 294 | "Value" : { "Fn::Join" : [ "", [ "http://", { "Fn::GetAtt" : [ "Ec2Instance", "PublicDnsName" ] }]]} 295 | }, 296 | "InstanceId" : { 297 | "Description" : "InstanceId of the newly created EC2 instance", 298 | "Value" : { "Ref" : "Ec2Instance" } 299 | }, 300 | "AZ" : { 301 | "Description" : "Availability Zone of the newly created EC2 instance", 302 | "Value" : { "Fn::GetAtt" : [ "Ec2Instance", "AvailabilityZone" ] } 303 | }, 304 | "StreamName" : { 305 | "Description" : "The name of the Kinesis Stream. This was autogenerated by the Kinesis Resource named 'KinesisStream'", 306 | "Value" : { "Ref" : "KinesisStream" } 307 | }, 308 | "ApplicationName" : { 309 | "Description" : "The name of the Kinesis Client Application. This was autogenerated by the DynamoDB Resource named 'KCLDynamoDBTable'", 310 | "Value" : { "Ref" : "KCLDynamoDBTable" } 311 | }, 312 | "CountsTable" : { 313 | "Description" : "The name of the DynamoDB table where counts are persisted. This was autogenerated by the DynamoDB Resource named 'CountsDynamoDBTable'", 314 | "Value" : { "Ref" : "CountsDynamoDBTable" } 315 | } 316 | } 317 | } 318 | -------------------------------------------------------------------------------- /Databases/Exam-tips/README.md: -------------------------------------------------------------------------------- 1 | # Exam Tips 2 | 3 | ## ElastiCache 4 | 5 | You will be given a scenario where a particular database is under a lot of stress/load. You may be asked which service you should use to alleviate this. 6 | 7 | ElastiCache is a good choice if your database is particularly read heavy and not prone to frequent changing. 8 | 9 | Redshift is a good answer if the reason your database is feeling stress is because management keep running OLAP transactions on it etc. 10 | 11 | ## Summary 12 | 13 | ### Types 14 | 15 | - RDS - OLTP 16 | - SQL 17 | - MySQL 18 | - PostgreSQL 19 | - Oracle 20 | - Aurora 21 | - MariaDB 22 | - DynamoDB - NoSQL 23 | - Redshift - OLAP 24 | - Elasticache - In Memory Caching 25 | - Memacached 26 | - Redis 27 | 28 | --- 29 | 30 | #### READ FAQ RDS SECTION IN DOCUMENTATION!! 31 | 32 | [https://aws.amazon.com/rds/faqs/](https://aws.amazon.com/rds/faqs/) -------------------------------------------------------------------------------- /Databases/README.md: -------------------------------------------------------------------------------- 1 | # Databases 2 | 3 | ## Types of Databases 4 | 5 | ### Relational Databases 6 | 7 | Relational databases are what most of us are all used to. They have been around since the 70's and you can think about them like spreadsheets! 8 | 9 | - Database 10 | - Tables 11 | - Columns 12 | - Rows 13 | 14 | | id | name | age | location | 15 | | --------- |:-------:| :-----:| :--------: | 16 | | 1 | nigel | 30 | San Diego | 17 | | 2 | jim | 28 | NYC | 18 | | 3 | betty | 31 | San Francisco| 19 | 20 | **_Relational Databases Examples_** 21 | 22 | - SQL Server 23 | - Oracle 24 | - MySQL 25 | - PostgreSQL 26 | - Aurora 27 | - MariaDB 28 | 29 | ### Non-Relational (NoSQL) 30 | 31 | - Database 32 | - Collection => Table 33 | - Document => Row 34 | - Key, Value Pairs => Columns 35 | 36 | **_Non Relational Databases Examples_** 37 | 38 | ```json 39 | { 40 | "_id": "394ejojaj903091881dnna", 41 | "name": "nigel", 42 | "age": 30, 43 | "location": "San Diego" 44 | } 45 | ``` 46 | 47 | ### Data Warehousing 48 | 49 | Used for business intelligence. Tools like Cognos, Jaspersoft, SQL Server, Reporting Services, Oracle Hyperion, SAP NetWeaver. 50 | 51 | Used to pull in very large and complex data sets. Usually used by management to do queries on data (such as current performance vs targets etc). 52 | 53 | ### OLTP (Online Transaction Processing) vs. OLAP (Online Analytics Processing) 54 | 55 | OTLP differs from OLAP in terms of the types of queries you will run. 56 | 57 | **_OLTP Example_** 58 | 59 | Used for transactional type queries. 60 | 61 | ``` 62 | Order number: 2120121 63 | 64 | Pulls up a row of data such as Name, Date, Address to Deliver to, Delivery Status etc. 65 | ``` 66 | 67 | **_OLAP Example_** 68 | 69 | Used for business logic type queries. 70 | 71 | ``` 72 | Net Profit of given product or device 73 | Pulls in large number of records 74 | 75 | Sum of products sold in region 76 | Sum of products sold in continent 77 | Unit cost of product in each region 78 | Sales price of each product 79 | Sales price - unit cost 80 | ``` 81 | 82 | Data Warehousing databases use different type of architecture both from a database perspective and infrastructure layer. 83 | 84 | ### Elasticache 85 | 86 | ElastiCache is a web service that makes it easy to deploy, operate and scale an in-memory cache in the cloud. The service improves the performance of web applications by allowing you to retrieve information from fast, managed, in-memory caches, instead of relying entirely on slower disk-based databases. 87 | 88 | ElasticCache supports two open-source in-memory caching engines... 89 | 90 | 1. Memcached 91 | 2. Redis 92 | 93 | ## Backups, Multi-AZ & Read Replicas 94 | 95 | ### Automated Backups 96 | 97 | Automated Backups allow you to recover your database to any point in time within a 'retention period'. The retention period can be between one and 35 days. 98 | 99 | Automated Backups will take a full daily snapshot and will also store transaction logs throughout the day. 100 | 101 | When you do a recovery, AWS will first choose the most recent daily backup, and then apply transaction logs relevant to that day. This allows you to do a point in time recovery down to a second, within a retention period. 102 | 103 | ### Database Snapshots 104 | 105 | DB Snapshots are done manually (ie they are user initiated) They are stored even after you delete the original RDS instance, unlike automated backups. 106 | 107 | ### Restoring Backups 108 | 109 | Whenever you restore either an Automatic Backup or a manual Snapshot, the restored version of the database will be a new RDS instance with a new DNS endpoint 110 | 111 | `original.us-west-1.rds.amazonaws.com` -> `restored.eu-west-1.rds.amazonaws.com` 112 | 113 | ### Encyrption 114 | 115 | Encryption at rest is supported for MySQL, Oracle, SQL Server, PostgreSQL, MariaDB & Aurora. 116 | 117 | Encryption is done using the AWS Key Management System (KMS) service. Once your RDS instance is encrypted, the data stored at rest in the underlying storage is encrypted, as are its automated backups, read replicas and snapshots. 118 | 119 | At the present time, encrypting an existing DB Instance is not supported. To use RDS encryption for an existing database, you must first create a snapshot, make a copy of that snapshot and encrypt the copy. 120 | 121 | ### Multi-AZ 122 | 123 | Multi-AZ allows you to have an exact copy of your production database in another Availability Zone. AWS handles the replication for you, so when your production database is written to, this write will automatically be synchronized to the stand by database. 124 | 125 | In the event of planned database maintenance, DB instance failure, or AZ failure, RDS will automatically failover to the standby so that database operations can resume quickly without admin intervention. 126 | 127 | **NOTE:** It is not primarily used for improving performance, really only **disaster recovery**. For performance improvement, you need **Read Replicas** 128 | 129 | **Multi-AZ Available DBs** 130 | 131 | - SQL Server 132 | - Oracle 133 | - MySQL Server 134 | - PostgreSQL 135 | - MariaDB 136 | 137 | ### Read Replicas 138 | 139 | Read replicas allow you to have a read-only copy of your production database. This is achieved by using async replication from the primay RDS instance to the Read Replica. You use Read Replicas primarily for very read-heavy database workloads. 140 | 141 | - Used for scaling, not disaster control! 142 | - Must have auto backups turned on in order to deploy a Read Replica 143 | - You can have up to 5 Read Replica copies of any database. 144 | - You can have Read Replicas of Read Replicas _(inception)_ - mindful of latency 145 | - Each Read Replica will have its own DNS end point. 146 | - You can have Read Replicas that have Multi-AZ 147 | - You can create Read Replicas of Mulit-AZ source databases 148 | - Read Replicas can be promoted to be their own databases. This breaks the replication. 149 | - You can have a Read Replica in a second region. 150 | 151 | **Read Replica Available DBs** 152 | 153 | - MySQL Server 154 | - PostgreSQL 155 | - MariaDB 156 | - Aurora 157 | 158 | ## DynamoDB 159 | 160 | DynamoDB is a fast and flexible NoSQL database service for all applications that need consistent, single-digit millisecond latency at any scale. It is a fully managed db nd supports both document and key-value data models. Its flexible data model and reliable performance make it a great fit for mobile, web, gaming, ad-tech, IoT etc. 161 | 162 | - Stored on SSD Storage 163 | - Spread Across **3** geographically distinct data centers 164 | 165 | - Eventual Consistent Read (Default) 166 | - Consistency across all copies of data is usually reached within a second. Repeating a read after a short amount of time should return the updated data. (Best Read Perf.) 167 | 168 | 169 | - Strongly Consistent Reads 170 | - A stronly consistent read returns a result that reflects all writes that received a successful response prior to the read. 171 | 172 | **NOTE:** Super easy to scale! Push button scaling 173 | 174 | ### Pricing 175 | 176 | Pricing is based on provision throughput capacity 177 | 178 | - Write Throughput $0.0065 per hour for every 10 units 179 | - Read Throughput $0.0065 per hour for every 50 units 180 | - Storage costs of $0.25G per month 181 | 182 | _Pricing Example:_ 183 | 184 | ``` 185 | Constraint: 1 million WRITEs and 1 million READs per day, while storing 3G of data. 186 | 187 | First, calculate how many writes and reads per second you need. 188 | 189 | 1 million evenly spread writes per day is equivalent to 1,000,000 (writes) /24 (hours) / 60 (minutes) / 60 (seconds) = 11.6 writes per second. 190 | 191 | -- BREAKDOWN -- 192 | 193 | DynamoDB WRITE Capacity Unit - 1 per second = 12 194 | DynamoDB READ Capacity Unit - 1 per second = 12 195 | 196 | READ Capacity Units - billed in blocks of 50 197 | WRITE Capacity Units - billed in blocks of 10 198 | 199 | Calc WRITE Capacity Units = (0.0065 / 10) x 12 x 24 = $0.1872 200 | Calc READ Capacity Units = (0.0065 / 10) x 12 x 24 = $0.0374 201 | ``` 202 | 203 | ## Redshift 204 | 205 | Amazon Redshift is a fast and powerful, fully managed petabyte-scale data warehouse service in the cloud. 206 | 207 | Customers can start small for just $0.25 per hour with no commitments or upfront costs and scale to a petabyte or more for $1,000 per terabyte per year, less than 1/10 of most data warehousing solutions. 208 | 209 | ### Configuration 210 | 211 | - Single Node (160Gb) 212 | - Multi-Node 213 | - Leader Node _(manages client connections and receives queries)_ 214 | - Compute Node _(store data and perform queries and computations)_ - Up to 128 Compute Nodes 215 | 216 | ### Columns 217 | 218 | **Columnar Data Storage** - Instead of storing data as rows, Redshift organizes the data by column. 219 | 220 | Unlike row-based systems, which are ideal for transaction processing, column-based systems are ideal for data warehousing and analytics, where queries often involve aggregates performed over large data sets. 221 | 222 | Since only the columns involved in the queries are processing and columnar data is stored sequentially on the storage media, column-based systems require far fewer I/Os, greatly improving query performance. 223 | 224 | ### Compression 225 | 226 | **Advanced Compression** - Columnar data storescan be compressed much more than row-based data stores because similar data is stored sequentially on disk. 227 | 228 | Redshift employs multiple compression techniques and can often achieve significant compression relative to traditional relational data stores. In addition, Redshift doesn't require indexes or materialized views and so uses less space than traditional relational database systems. 229 | 230 | When loading data into an empty table, Redshift automatically samples you data and selects the most appropriate compression scheme. 231 | 232 | ### MPP 233 | 234 | **Massive Parallel Processing (MPP)** - Redshift automatically distributes data and query load across all nodes. Redshift makes it easy to add nodes to your data warehouse and enables you to maintain fast query performance as your data warehouse grows. 235 | 236 | ### Pricing 237 | 238 | **How is Redshift priced?** 239 | 240 | - Compute Node Hours 241 | - Total number of hours you run across all your compute nodes for the billing period 242 | - Billed for 1 unit per node per hour, so a 3 - node data warehouse cluster running persistently for an entire month would incur 2,160 instance hours. 243 | - You will **not** be charged for leader node hours; only compute nodes will incur charges 244 | 245 | - Backups 246 | - Data transfers (Only within a VPC, not outside of it) 247 | 248 | ### Security 249 | 250 | - Encrypted in transit using SSL 251 | - Encrypted at rest using AES-256 encryption 252 | - By default Redshift takes care of key management 253 | - Manages your keys through HSM (Hardware Security Module) 254 | - AWS Key Management Service (KMS) 255 | 256 | ### Availability 257 | 258 | - Currently only available in 1 AZ - Realistically only for business logic 259 | - Can restore snapshots to new AZ's in the event of outage. 260 | 261 | ## Elasticache 262 | 263 | ElastiCache is a web service that makes it easy to deploy, operate and scale an in-memory cache in the cloud. The service improves the performance of web applications by allowing you to retrieve information from fast, mananged, in-memory caches, instead of relying entirely on slower disk-based databases. 264 | 265 | ### Why ElastiCache? 266 | 267 | ElastiCache can be used to significantly improve latency and throughput for many read-heavy application workloads - ie networking, gaming, media sharing and Q&A portals or compute intensive workloads. 268 | 269 | Caching improves application performance by storing critical pieces of data in memory for low-latency access. Cached information may include the results of I/O intensive database queries or the results of computational intensive calculations 270 | 271 | ### Types of Elasticache 272 | 273 | - Memcached 274 | - A widely adopted memory object caching system. Elasticache is protocol compliant with Memcached, so popular tools that you use today with existing Memcached environments will work seamlessly with the service. 275 | 276 | - Redis 277 | - A popular open-source in-memory key-value store that supports data structures such as sorted sets and lists. ElastiCache supports Master/Slave replcation and Multi-AZ which can be used to achieve cross AZ redundancy. 278 | 279 | ## Aurora 280 | 281 | ### What is Aurora? 282 | 283 | Aurora is a MySQL-compatible,relational database engine that combines the speed and availability of high-end commercial databases with the simplicity and cost effectiveness of open source databases. Aurora provides up to 5x better performance than MySQL at a price point of 1/10 that of a commercial database while delivering similar performance and availability 284 | 285 | ### Scaling 286 | 287 | - Start with 10G, Scles in 10G increments to 64 TB (Storage Autoscaling) 288 | - Compute resource can scale up to 32vCPUs and 244G of Memory. 289 | - 2 copies of your data is contained in each availability zone, with minimum of 3 AZ -> 6 copies of your data! Highly redundant 290 | - Designed to transparently handle the loss of up to 2 copies of data without affecting database write availability and up to 3 copies without affecting read availability. 291 | - Aurora storage is also self-healing. Data blocks and disks are continuously scanned for errors and repaired automatically. 292 | 293 | ### Aurora Replicas 294 | 295 | - 2 Types of Replicas are available 296 | - Aurora Replicas - Up to 15 replicas currently 297 | - MySQL Replicas - Up to 5 replicas currently -------------------------------------------------------------------------------- /Databases/scripts/connect.txt: -------------------------------------------------------------------------------- 1 | "; 10 | $selected = mysql_select_db("$dbname",$dbhandle) or die("Unable to connect to MySQL DB - check the database name and try again."); 11 | ?> -------------------------------------------------------------------------------- /Databases/scripts/rds.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | yum install httpd php php-mysql -y 3 | yum update -y 4 | chkconfig httpd on 5 | service httpd start 6 | echo "" > /var/www/html/index.php 7 | cd /var/www/html 8 | wget https://s3.eu-west-2.amazonaws.com/acloudguru-example/connect.php 9 | -------------------------------------------------------------------------------- /EC2/Exam-tips/README.md: -------------------------------------------------------------------------------- 1 | # Exam Tips 2 | 3 | ## EC2 Instance Run Down 4 | 5 | - **On Demand** - allows you to pay a fixed rate by the hour (or second) with not commitment 6 | 7 | - **Reserved** - provides you with the capacity reservation, and offer a significant discount on the hourly charge for an instance. 1 year or 3 year terms 8 | 9 | - **Spot** - Enables you to bid whatever price you want for instant capacity, providing for even greater savings if your applications have flexible start and end times 10 | 11 | - **Dedicated Hosts** - Physical EC2 server dedicated for your use. Dedicated Hosts can help reduce costs by allowing you to use your existing server-bound software license 12 | 13 | **_Important Note!!_** 14 | 15 | If a Spot instance is terminated by Amazon EC2, you will not be charged for a partial hour of usage. However, if you terminate the instance yourself, you will be charged for the complete hour in which the instance ran. 16 | 17 | ## Instance Types 18 | 19 | - **F.** - FGPA 20 | - **I.** - IOPS 21 | - **G.** - Graphics 22 | - **H.** - High Disk Throughput 23 | - **T.** - Cheap General Purpose (think T2 Micro) 24 | - **D.** - Density 25 | - **R.** - Ram 26 | - **M.** - Main choice for general purpose applications 27 | - **C.** - Compute 28 | - **P.** - Graphics(Pics) 29 | - **X.** - Extreme Memory 30 | 31 | ## Volume Types 32 | 33 | ### SSD 34 | 35 | - **General Purpose (SSD)** - balances price and perf. for a wide variety of workloads 36 | 37 | - **Provisioned IOPS (SSD)** - Highest perf. SSD volume for mission-critical low-latency or high-throughput workloads 38 | 39 | ### Magnetic 40 | 41 | - **Throughput Optimized HDD** - Low cost HDD volume designed for frequently accessed, throughput-intensive workloads 42 | 43 | - **Cold HDD** - Lowest cost HDD volume designed for less frequently accessed workloads 44 | 45 | - **Magnetic** - Previous Generation. Can be a boot volume. 46 | 47 | ## Upgrading EBS Volume Types - Lab 48 | 49 | ### Volumes & Snapshots 50 | 51 | - Volumes exist on EBS 52 | - Virtual Hard Disk 53 | - Snapshots exist on S3 54 | - Snapshots are a point in time copies of Volumes 55 | - Snapshots are incremental - this means that only the blocks that have changed since your last snapshot are moved to S3. Only recording the changes 56 | - If it's 1st snapshot, takes time to create 57 | 58 | ### Snapshots of Root Device Volumes 59 | 60 | - To create a snapshot of Amazon EBS volumes that serve as root devices, you should stop the instance before taking the snapshot, however you can take a snapshot while instance is running. 61 | - However you can take a snap while the instance is running. 62 | - You can create AMI's from EBS-backed Instances and Snapshots. 63 | - You can change EBS volume sizes on the fly, including changing the size and storage type. 64 | - Volumes will **ALWAYS** be in the same availability zone as the EC2 instance. 65 | - To move and EC2 volume from one AZ/Region to another, take a snap or an image of it, then copy it to the new AZ/Region. 66 | 67 | ### Volumes vs Snapshots - Security 68 | 69 | - Snapshots of encrypted volumes are encrypted automatically. 70 | - Volumes restored from encrypted snapshots are encrypted automatically. 71 | - You can share snapshots, but only if they are unencrypted. 72 | - These snapshots can be shared with other AWS accounts or made public. 73 | 74 | ### EBS vs. Instance Store 75 | 76 | - Instance store volumes are sometimes called _Ephemeral Storage_. 77 | - Instance store volumes cannot be stopped. If the underlying host fails, you will lose all your data. 78 | - EBS backed instances can be stopped. You will not los the data on this instance if it is stopped. 79 | - You can reboot both, you will not lose your data. 80 | - By default, both ROOT volumes will be deleted on termination, however with EBS volumes, you can tell AWS to keep the root device volume. 81 | 82 | ## Load Balancers 83 | 84 | - 3 Types of Load Balancers 85 | - Application Load Balancers 86 | - Network Load Balancers 87 | - Classic Load Balancers 88 | 89 | 90 | - 504 Error means the gateway has timed out. Application is not responding within the idle timeout period 91 | - Trouble shoot the application. Web Server or Database Server? 92 | 93 | 94 | - If you need IPv4 address of your end user, look fro the X-Forwarded-For header. 95 | - Instances are monitored but ELB are reported as `InService` or `OutofService`. 96 | - Health Checks check the instance health by talking to it. 97 | - ELB's have their own DNS name. You are **never** given an IP address 98 | - Read the ELB FAQ for Classic Load Balancers 99 | 100 | _Note: ELB's do not have IP Addresses, only found by DNS namespace_ 101 | 102 | ## CloudWatch 103 | 104 | - Standard Monitoring - 5 minutes 105 | - Detailed Monitoring - 1 minute 106 | 107 | ### What can you do with CloudWatch? (Not to be confused with CloudTrail) 108 | 109 | - **Dashboards** - Creates awesome dashboards to see/monitor what is happening with your AWS environment. 110 | - **Alarms** - Allows you to set Alarms that notify you when a particular thresholds are hit. 111 | - **Events** - Helps you to respond to state changes in your AWS resources. 112 | - **Logs** - Helps you to aggregate, monitor and store logs. 113 | 114 | ## Placement Groups 115 | 116 | - A Clustered Placement Group can not span multiple Availability Zones. 117 | - A Spread Placement Group can. 118 | - The name you specify for a placement group must be unique within your aws account. 119 | - Only certain types of instances can be launched in a placement group (Compute Optimized, GPU, Memory Optimized, Storage Optimized) 120 | - AWS recommend homogeneous instances within placement groups. 121 | - You cant merge placement groups 122 | - You cant move an existing instance into a placement group. You can create an AMI from your existing instance, and then launch a new instance from the AMI into a placement group. 123 | 124 | ## Lambda 125 | 126 | - Lambda scales horizontally (not vertically) automatically. Redundancy 127 | - Lambda functions are independent, 1 event = 1 function 128 | - Lambda is serverless 129 | - Know what services are serverless! 130 | - S3 131 | - API Gateway 132 | - DynamoDB 133 | - Lambda functions can trigger other lambda functions, 1 event can = x functions if functions trigger other functions. 134 | - Architectures can get extremely complicated, AWS X-ray allows you to debug what is happening 135 | - Lambda can do things globally, you can use it to back up S3 buckets to other S3 buckets etc. 136 | - Know your triggers - connecting AWS services 137 | 138 | ## Summary (TLDR;) 139 | 140 | - Know the differences between EC2 instances 141 | - On Demand 142 | - Spot 143 | - Reserved 144 | - Dedicated hosts 145 | 146 | **_Remember with Spot Instances_** 147 | 148 | - If you terminate the instance, you pay for the hour 149 | - If AWS terminates the instance, you get the hour it was terminated for free. 150 | 151 | ### EC2 Instance Types 152 | 153 | **F.I.G.H.T.D.R.M.C.P.X** (Use Reference) 154 | 155 | ### EBS (Elastic Block Storage) 156 | 157 | **Consists of:** 158 | 159 | - SSD, General Purpose - GP2 - Up to 10,000 IOPS 160 | - SSD, Provisioned IOPS - IO1 - More than 10,000 IOPS 161 | - HDD, Throughput Optimized - ST1 - frequently accessed workloads 162 | - HDD, Cold - SC1 - Less frequently accessed data 163 | - HDD, Magnetic - Standard - Cheap, Infrequently accessed storage. 164 | 165 | **IMPORTANT NOTE:** You cannot mount 1 EBS volume to multiple EC2 instances; Instead use EFS (Elastic File Storage) 166 | 167 | ### Lab Tips! 168 | 169 | - Termination Protection is turned off by default, you must turn this on! 170 | - On a EBS-backed instance, the default action is for the root EBS volume to be deleted when the instance is terminated. 171 | - EBS backed Root volumes can now be encrypted using AWS API or console, or you can use a third party tool (bitlocker etc.) to encrypt the root volume. 172 | - Additional volumes can be encrypted 173 | 174 | ### Volumes vs. Snapshots 175 | 176 | - Volumes exist on EBS; Virtual Hard Disks 177 | - Snapshots exist on S3 178 | - You can take a snapshot of a volume, this will store that volume on S3 179 | - Snapshots are point-in-time copies of volumes 180 | - Snapshots are incremental. This means that only the blocks that have changed since your last snapshot are moved to S3 181 | - If taking your first snapshot, may take some time 182 | 183 | **Security** 184 | 185 | - Snapshots of encrypted volumes are encrypted automatically 186 | - Volumes restored from encrypted snapshots are encrypted automatically 187 | - You can share snapshots, but only if they are unencrypted 188 | - These snapshots can be shared with other AWS accounts or made public 189 | 190 | **Snapshots or Root Device Volumes** 191 | 192 | - To create a snapshot for EBS volumes that serve as root devices, you should stop the instance before taking the snapshot. 193 | 194 | ### EBS vs Instance Store 195 | 196 | - Instance Store Volumes are sometimes called Ephemeral Storage 197 | - Instance Store Volumes cannot be stopped. If the underlying host fails, you will lose your data. 198 | - EBS backed instances can be stopped. You will not lose the data on this instance if it is stopped. 199 | - You can reboot both, you will not lose your data 200 | - By default, both ROOT volumes will be deleted on termination. However, with EBS volumes, you can tell AWS to keep the root device volume. 201 | 202 | ### How can you take a snapshot of a RAID Array? 203 | 204 | **Problem** - Take a snapshot, the snapshots excludes data held in the cache by applications and the OS. This tends not to matter on a single volume. However, using multiple volumes in a RAID array, this can be a problem due to interdependencies of the array. 205 | 206 | **Solution** - Take an application consistent snapshot. 207 | 208 | - Stop the application from writing to disk 209 | - Flush all caches to the disk. 210 | 211 | How is this accomplised? 212 | 213 | - Freeze the file system 214 | - Unmount the RAID array 215 | - Shutting down the associated EC2 instance. 216 | 217 | ### AMI (Amazon Machine Image) 218 | 219 | AMIs are regional. You can only launch an AMI from the region in which its stored. However you can copy AMIs to other regions using the console, command line, or the Amazon EC2 API 220 | 221 | - Standard monitoring - 5 min 222 | - Detailed monitoring - 1 min 223 | 224 | - Cloudwatch is for **performance monitoring** 225 | - Cloudtrail is for **auditing** 226 | 227 | ### Cloudtrail 228 | 229 | - **Dashboards** - Cloudwatch creates awesome dashboards to see what is happening with your AWS envrionment 230 | - **Alarms** - Allows you to set alarms when particular thresholds are hit. 231 | - **Events** - Helps you to respond to state changes in your AWS resources. 232 | - **Logs** - Helps you to aggregate, monitor, and store logs 233 | 234 | ### Roles 235 | 236 | - Roles are more secure than storing your access key and secret access key on individual instances. 237 | - Roles are easier to manage 238 | - Roles can be assigned to an EC2 instance after it has been provisioned using both the command line and the AWS console 239 | - Roles are universal - they can be used in any region 240 | 241 | ### Instance Metadata 242 | 243 | - Used to get information about an instance (public IP, DNS etc) 244 | - `curl http://169.254.169.254/latest/meta-data` 245 | - `curl http://169.254.169.254/latest/user-data` 246 | 247 | ### EFS (Elastic File System) 248 | 249 | - Supports the Network File System version 4 (NFSv4) protocol 250 | - You only pay for the storage you use (no pre-provisioning required) 251 | - Can scale up to petabytes 252 | - Can support thousands of concurrent NFS connections 253 | - Data is stored accross multiple AZs within a region 254 | - Read after Write consistency 255 | 256 | ### Lambda 257 | 258 | - Lambda is a compute service where you can upload you code and create a Lambda function. 259 | - Takes care of provisioning and managing servers that you use to run your code. 260 | - Need not worry about OS, patching, scaling etc. 261 | 262 | **_Use Lambda as:_** 263 | 264 | - Event driven compute service where Lambda runs your code in response to events. These events could be changes in an S3 bucket or Dynamo DB table. 265 | - A compute service to run your code in response to HTTP requests using API Gateway or API calls made using AWS SDKs 266 | 267 | ### Placement Groups 268 | 269 | **Know the differences between and why you would use...** 270 | 271 | - Clustered Placement Groups 272 | - Spread Placement Groups -------------------------------------------------------------------------------- /EC2/README.md: -------------------------------------------------------------------------------- 1 | # EC2 (Elastic Cloud Compute) 2 | 3 | AWS EC2 is a web service that provides re-sizable compute capacity in the cloud. EC2 reduces the time required to obtain and boot new server instances to minutes, allowing you to quickly scale capacity, both up and down, as your computing requirements change. 4 | 5 | EC2 has changed the economics of cloud computing by allowing you to pay only for capacity that your actually use. EC2 provides developers the tools to build failure resistant applications and isolate themselves from common failure scenarios. 6 | 7 | ## Pricing Options 8 | 9 | ### On Demand 10 | 11 | Allows you to pay a fixed rate by the hour (or by the second) with no commitment. 12 | 13 | **_Use Cases_** 14 | 15 | - Perfect for users that want the low cost and flexibility of EC2 without any of the up front payment or long term commitment 16 | - Applications with short term, spiky or unpredictable workloads that cannot be interrupted 17 | - Applications being developed or tested on EC2 for the first time 18 | 19 | ### Reserved 20 | 21 | Provides you with a capacity reservation, and offer a significant discount on the hourly charge for an instance. 1 year or 3 year terms. 22 | 23 | **_Use Cases_** 24 | 25 | - Applications with steady state or predictable usage 26 | - Applications that require reserved capacity 27 | - Users can make up front payments to reduce their total computing costs even further 28 | - Standard RIs (Up to 75% off on-demand) 29 | - Convertible RIs (Up to 54% off on-demand) feature the capability to change the attributes of the RI as long as the exchange results in the creation of Reserved Instances of equal or greater value. Ability to go from CPU intensive instance to Memory intensive. 30 | - Scheduled RIs are available to launch within the time window you reserve. This option allows you to match your capacity reservation to predictable recurring schedule that only requires a fraction of a day, a week, or a month. 31 | 32 | ### Spot 33 | 34 | Enables you to bid whatever price you want for an instance capacity, providing for even greater savings if your applications have flexible start and end times. 35 | 36 | **Use Cases** 37 | 38 | - Applications that have flexible start and end times 39 | - Applications that are only feasible at very low compute prices 40 | - Used for single compute instances to save on costs compared to 9-5 during the week. 41 | - Users with an urgent need for a large amount of additional computing capacity. 42 | 43 | ### Dedicated Hosts 44 | 45 | Physical EC2 server dedicated for your use. Dedicated Hosts can help you reduce costs by allowing you to use your existing server-bound software licenses. 46 | 47 | **Use Cases** 48 | 49 | - Useful for regulatory requirements that may not support multi-tenant virtualization. 50 | - Great for licensing which does not support multi-tenancy or cloud deployments 51 | - Can be purchased On-Demand (hourly). 52 | - Can be purchased as a Reservation for up to 70% off the On-Demand price. 53 | 54 | ## EC2 Instance Types 55 | 56 | **_No need to memorize for associate exams_** 57 | 58 | | Family | Specialty | Use Cases | 59 | | :------:|:-----------------------------:| :------------------------------:| 60 | | F1 | Field Programmable Gate Array | Genomics research, financial analytics, real-time video processing, big data etc| 61 | | I3 | High Speed Storage | NoSQL DBs, Datawarehousing | 62 | | G3 | Graphics Intensive | Video Encoding / 3D Application Streaming| 63 | | H1 | High Disk Throughput | MapReduce-based workloads, distributed file systems such as HDFS and MapR-FS | 64 | | T2 | Lowest Cost General Purpose | Web Servers / Small DBs | 65 | | D2 | Dense Storage | Fileservers / Data Warehousing / Hadoop | 66 | | R4 | Memory Optimization | Memory Intensive Apps/DBs | 67 | | M5 | General Purpose | Application Servers | 68 | | C5 | Compute Optimized | CPU Intensive Apps / DBs | 69 | | P3 | Graphics / General Purpose GPU | Machine Learning, Bit Coin Mining etc | 70 | | X1 | Memory Optimized | SAP HANA / Apache Spark | 71 | 72 | 73 | **How to remember EC2 instance types F.I.G.H.T.D.R.M.C.P.X (after 2017 reinvent):** 74 | - **_F_** - FGPA 75 | - **_I_** - IOPS 76 | - **_G_** - Graphics 77 | - **_H_** - High Disk Throughput 78 | - **_T_** - Cheap General Purpose (think T2 Micro) 79 | - **_D_** - Density 80 | - **_R_** - Ram 81 | - **_M_** - Main choice for general purpose applications 82 | - **_C_** - Compute 83 | - **_P_** - Graphics(Pics) 84 | - **_X_** - Extreme Memory 85 | 86 | ## EBS - Elastic Block Storage 87 | 88 | Amazon EBS allows you to create storage volumes and attach them Amazon EC2 instances. Once attached, you can create a file system on top of theses volumes, run a database, or use them in any other way you would use a block device. EBS volumes are placed in a specific Availability Zone, where they are automatically replicated to protect you from the failure of a single component. 89 | 90 | _TLDR; A disk in the cloud that you attach to your EC2 instances_ 91 | 92 | ### EBS Volume Types 93 | 94 | - General Purpose SSD (GP2) 95 | - General purpose, balances both price and performance. 96 | - Ratio of 3 IOPS per GB with up to 10,000 IOPS and the ability to burst up to 3000 IOPS for extended periods of time for volumes at 3334 GB and above 97 | - Provisioned IOPS SSD (IO1) 98 | - Designed for I/O intensive applications such as large relational or NoSQL databases. 99 | - Use if you need more than 10,000 IOPS 100 | - Provision up to 20,000 IOPS per volume 101 | - Super high performance 102 | - Throughput Optimized HDD (ST1) 103 | - Big Data 104 | - Data warehouses 105 | - Log processing 106 | - Cannot be a boot volume 107 | - Cold HDD (SC1) 108 | - Lowest cost storage for infrequently accessed workloads 109 | - File server 110 | - Cannot be a boot volume 111 | - Magnetic (Standard) 112 | - Lowest cost per GB of all EBS volume types that is bootable. Magnetic volumes are ideal for workloads where data is accessed infrequently, and applications where the lowest storage cost is important 113 | 114 | ## Let's get our hands dirty! Launch an EC2 instance lab! 115 | 116 | ### Summary 117 | 118 | - Termination protection is turned off by default, you **MUST** turn it on. 119 | - On an EBS-backed instance, the default action is for the root EBS volume to be deleted when the instance is terminated 120 | - EBS Root Volume of you DEFAULT AMI's cannot be encrypted. You can also use a third party tool (such as bit locker) to encrypt the root volume, or this can be done when creating AMI's (future lab) in the AWS console or using the API. 121 | - Additional volumes can be encrypted. 122 | 123 | ## Security Groups 124 | 125 | ### What is a Security Group? 126 | 127 | A security group is a virtual firewall that's controlling traffic to your EC2 instance. When you first launch as EC2 instance you associate it to 1 or more instances. You have the ability to add rules to these security groups that allows traffic to or from these instances. 128 | 129 | ### Security Groups - General 130 | 131 | 1. Any security group rules apply immediately 132 | 2. Security groups are **_STATEFUL_**. Inbound rules automatically add outbound rules 133 | 3. All traffic is blocked by default and included through the rules. Whitelist 134 | 4. All outbound traffic is allowed 135 | 5. You can have multiple EC2 instances within a security group. 136 | 6. You can have multiple security groups attached to EC2 instances. 137 | 7. You cannot block specific IP addresses using Security Groups, use Network Access Control Lists. 138 | 8. You can specify allow rules, but not deny rules. 139 | 140 | ## RAID, Volumes & Snapshots 141 | 142 | ### RAID - Redundant Array of Independent Disks 143 | 144 | - RAID 0 - Striped, no redundancy, good performance. If one fails, you lose all 145 | - RAID 1 - Mirrored, redundant. If one fails, others available 146 | - RAID 5 - Good for reads, bad for writes, AWS does not recommend ever putting RAID 5's on EBS. Strongly discouraged. 147 | - RAID 10 - Striping & Mirrored, good redundancy, good performance. 148 | 149 | #### How can I take a Snapshot of a RAID Array? 150 | 151 | - **Problem** - Taking a snapshot excludes the data held in cache by applications and the OS. This doesn't really matter on single volume, however when using multiple volumes in a RAID Array, this can be a problem due to interdependencies of the array. 152 | 153 | - **Solution** - Take an application specific snapshot. 154 | - Stop application from writing to disk. 155 | - Flush all caches to the disk. 156 | - How can we do this? 157 | - Freeze the file system 158 | - Unmount the RAID Array 159 | - Shutting down the associated EC2 instance. 160 | 161 | ## Create an AMI lab - Volumes vs. Snapshots 162 | 163 | ### Snapshots of Root Device Volumes 164 | 165 | - To create a snapshot for Amazon EBS volumes that server as root devices, you should stop the instance before taking the snapshot. 166 | 167 | ### Security 168 | 169 | - Snapshots of encrypted volumes are encrypted automatically 170 | - Volumes restored from encrypted snapshots are encrypted automatically. 171 | - You can share snapshots, but only if they are unencrypted. 172 | - Said snapshots can be shared with other AWS accounts of made public 173 | 174 | ## AMI Types 175 | 176 | ### What should you select your AMI based on? 177 | 178 | - Region 179 | - OS 180 | - Architecture 181 | - Launch Permissions 182 | - Storage for the Root Device (Root Device Volume) 183 | - Instance Store (Ephemeral Store) 184 | - EBS Backed Volumes 185 | 186 | ### EBS vs. Instance Store 187 | 188 | All AMIs are categorized as either backed by Amazon EBS or backed by instance store. 189 | 190 | **_For EBS Volumes:_** 191 | 192 | The root device for an instance launched from the AMI is an Amazon EBS volume created from an Amazon EBS snapshot. 193 | 194 | **_For Instance Store Volumes:_** 195 | 196 | The root device for an instance launched from the AMI is an instance store volume created from a template stored in Amazon S3. 197 | 198 | ## Elastic Load Balancers 199 | 200 | ### What is a load balancer? 201 | 202 | A virtual appliance that balances the load of HTTP traffic etc. of your web application/web servers. 203 | 204 | ### Types of Load Balancers 205 | 206 | - Application Load Balancers 207 | - Network Load Balancers 208 | - Classic Load Balancers 209 | 210 | ### Application Load Balancer _(Intelligent)_ 211 | 212 | Best suited for load balancing of HTTP(S) traffic. They operate at Layer 7 (OSI) and are application aware. The are intelligent, and you can create advanced request routing, sending specified requests to specific web servers. 213 | 214 | ### Network Load Balancer _(Performance)_ 215 | 216 | Best suited for load balancing of TCP traffic where extreme performance is required. Operating at the connection level (Layer 4), Network Load Balancers are capable of handling millions of requests per second, while maintaining ultra-low latencies. 217 | 218 | ### Classic Load Balancer _(OG, Legacy Load Balancer)_ 219 | 220 | Used to load balance HTTP(S) applications and use Layer 7-specific features, such as X-Forwarded and stick-sessions. You can use strict Layer 4 load balancing for applications that rely purely on the TCP protocol. 221 | 222 | ### 504 Error 223 | 224 | - If no response or timeout, the ELB (Elastic Load Balancer) responds with status code 504. 225 | - Internal Server Error type - DB Layer or Web Server Layer. 226 | - Solution: Identify issue where failing and scale up or out where possible. 227 | 228 | ## Placement Groups (Exam MUST KNOW!!) 229 | 230 | ### Two Types of Placement Groups 231 | 232 | **Clustered Placement Group** 233 | 234 | A cluster placement group is a grouping of instances within a **single** Availability Zone. Placement groups are recommended for applications that need low network latency, high network throughput, or both. 235 | 236 | _NOTE: Only a certain number instances can be launched in to a Clustered Placement Group._ 237 | 238 | **Spread Placement Group** 239 | 240 | Opposite of a Clustered Placement Group. A Spread Placement Group is a group of instances that are each placed on distinct underlying hardware. Spread Placement Groups are recommended for applications that have a small number of critical instances that should be kept separate from each other. 241 | 242 | ## EFS (Elastic File System) 243 | 244 | AWS EFS is file storage service for AWS EC2 instances. Amazon EFS is easy to use and provides a simple interface that allows you to create and configure file systems quickly and easily. With AWS EFS, storage capacity is elastic, growing and shrinking automatically as you add and remove files, so your applications have the storage they need, when they need it. 245 | 246 | ### EFS Features 247 | 248 | - Supports the Network File System version 4 (NFSv4) protocol 249 | - You only pay for the storage you use (no pre-provisioning required) 250 | - Can scale up to the petabytes 251 | - Can support thousands of concurrent NFS connections 252 | - Data is stored across multiple AZ's within a region 253 | - Read After Write Consistency 254 | 255 | ## Lambda 256 | 257 | ### What is Lambda? 258 | 259 | AWS Lambda is a compute service where you can upload your code and create Lambda function. AWS Lambda takes care of provisioning and managing the servers that you use to run the code. Worry free from OS, patching, scaling, etc. 260 | 261 | **Use Cases** 262 | 263 | - As an event-driven compute service where AWS Lambda runs your code in response to events. These events could be changes to data in an Amazon S3 bucket or an Amazon DynamoDB table. 264 | 265 | - As a compute service to run your code in response to HTTP requests using Amazon API Gateway or API calls made using AWS SDKs. 266 | 267 | **Encapsulation of the following:** 268 | 269 | - Data Centers 270 | - Hardware 271 | - Assembly Code/Protocols 272 | - High Level languages 273 | - Operation Systems 274 | - Application Layer/AWS API's 275 | - AWS Lambda 276 | 277 | ### Compatible Languages: 278 | 279 | - C# 280 | - Java 281 | - Node.js 282 | - Python 283 | 284 | ### How is Lambda priced? 285 | 286 | - Number of requests 287 | - First 1m requests are free. $0.20 per 1m requests thereafter. 288 | 289 | - Duration 290 | - Duration is calculated from the time your code begins execution until it returns or otherwise terminates, rounded up to the nearest 100ms. The price depends on the amount of memory you allocate to your function. You are charged $0.00001667 for every GB-second used. 291 | 292 | 293 | ### Why is Lambda cool? 294 | 295 | - No SERVERS!! 296 | - Continuous Scaling 297 | - Super super super cheap -------------------------------------------------------------------------------- /EC2/templates/Build-A-Serverless-Website/error.html: -------------------------------------------------------------------------------- 1 |
Post ID | 86 |Voice | 87 |Post | 88 |Status | 89 |Player | 90 |
---|