├── Chapter01 └── README.md ├── Chapter02 └── README.md ├── Chapter03 ├── CSVtoParquetLambda.py ├── DataEngLambdaS3CWGluePolicy.json ├── README.md └── test.csv ├── Chapter04 └── README.md ├── Chapter05 ├── Data-Engineering-Completed-Whiteboard.drawio ├── Data-Engineering-Whiteboard-Completed-Notes.drawio ├── Data-Engineering-Whiteboard-Template.drawio └── README.md ├── Chapter06 └── README.md ├── Chapter07 └── README.md ├── Chapter08 └── README.md ├── Chapter09 └── README.md ├── Chapter10 ├── ProcessFileStateMachine.json ├── README.md ├── dataeng-check-file-ext.py └── dataeng-random-failure-generator.py ├── Chapter11 └── README.md ├── Chapter12 └── README.md ├── Chapter13 ├── README.md └── website-reviews-analysis-role.py ├── Chapter14 └── README.md ├── LICENSE └── README.md /Chapter01/README.md: -------------------------------------------------------------------------------- 1 | # Chapter 1 - An Introduction to Data Engineering 2 | In this chapter we reviewed how data is becoming an increasingly important asset for organizations, and looked at some of the core challenges of working with big data. We then reviewed some of the data related roles that are commonly seen in organizations today. 3 | 4 | ## Links 5 | - **AWS Data Lake Defition:** https://aws.amazon.com/big-data/datalakes-and-analytics/what-is-a-data-lake/ 6 | 7 | ## Hands-on Activity 8 | 9 | In this ***hands-on activity section*** of this chapter you were given step-by-step instructions to guide you in creating a new AWS account. There was no coding or policies configured in this chapter. 10 | 11 | ### Links 12 | - AWS documentation on monitoring your usage and costs: https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/monitoring-costs.html 13 | - Google Voice for creating a virtual phone number for use with your account: https://voice.google.com/ 14 | - What to do if you don’t receive a confirmation email within 24 hours: https://aws.amazon.com/premiumsupport/knowledge-center/create-and-activate-aws-account/ 15 | - Best practices for securing your account - root user: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_root-user.html 16 | - Best practices for securing your account - Multi Factor Authentication (MFA): https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_mfa_enable_virtual.html 17 | -------------------------------------------------------------------------------- /Chapter02/README.md: -------------------------------------------------------------------------------- 1 | # Chapter 2 - Data Management Architectures for Analytics 2 | In this chapter, we learned about the foundational architectural concepts that are typically applied when designing real-life analytics data management and processing solutions. We also discussed three analytics data management architectures that you would find most commonly used across organizations today: data warehouse, data lake, and data lakehouse. 3 | 4 | ## Hands-on Activity 5 | In the ***hands-on activity*** section of this chapter we installed the AWS Command Line Interface (CLI) tool, and used the AWS CLI to create a new Amazon S3 bucket. 6 | 7 | ### Links 8 | - **AWS CLI download:** https://aws.amazon.com/cli/ 9 | - **Learn more about S3 bucket naming rules:** https://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html 10 | 11 | ### Commands 12 | #### Create a new Amazon S3 bucket 13 | The following command creates a new bucket called *dataeng-test-bucket-123*. If a bucket with this name already exists the command will fail, so you need to ensure you provide a globally unique name. 14 | 15 | ``` 16 | aws s3 mb s3://dataeng-test-bucket-123 17 | ``` 18 | -------------------------------------------------------------------------------- /Chapter03/CSVtoParquetLambda.py: -------------------------------------------------------------------------------- 1 | import boto3 2 | import awswrangler as wr 3 | from urllib.parse import unquote_plus 4 | 5 | def lambda_handler(event, context): 6 | # Get the source bucket and object name as passed to the Lambda function 7 | for record in event['Records']: 8 | bucket = record['s3']['bucket']['name'] 9 | key = unquote_plus(record['s3']['object']['key']) 10 | 11 | # We will set the DB and table name based on the last two elements of 12 | # the path prior to the file name. If key = 'dms/sakila/film/LOAD01.csv', 13 | # then the following lines will set db to sakila and table_name to 'film' 14 | key_list = key.split("/") 15 | print(f'key_list: {key_list}') 16 | db_name = key_list[len(key_list)-3] 17 | table_name = key_list[len(key_list)-2] 18 | 19 | print(f'Bucket: {bucket}') 20 | print(f'Key: {key}') 21 | print(f'DB Name: {db_name}') 22 | print(f'Table Name: {table_name}') 23 | 24 | input_path = f"s3://{bucket}/{key}" 25 | print(f'Input_Path: {input_path}') 26 | output_path = f"s3://dataeng-clean-zone-INITIALS/{db_name}/{table_name}" 27 | print(f'Output_Path: {output_path}') 28 | 29 | input_df = wr.s3.read_csv([input_path]) 30 | 31 | current_databases = wr.catalog.databases() 32 | wr.catalog.databases() 33 | if db_name not in current_databases.values: 34 | print(f'- Database {db_name} does not exist ... creating') 35 | wr.catalog.create_database(db_name) 36 | else: 37 | print(f'- Database {db_name} already exists') 38 | 39 | result = wr.s3.to_parquet( 40 | df=input_df, 41 | path=output_path, 42 | dataset=True, 43 | database=db_name, 44 | table=table_name, 45 | mode="append") 46 | 47 | print("RESULT: ") 48 | print(f'{result}') 49 | 50 | return result 51 | -------------------------------------------------------------------------------- /Chapter03/DataEngLambdaS3CWGluePolicy.json: -------------------------------------------------------------------------------- 1 | { 2 | "Version": "2012-10-17", 3 | "Statement": [ 4 | { 5 | "Effect": "Allow", 6 | "Action": [ 7 | "logs:PutLogEvents", 8 | "logs:CreateLogGroup", 9 | "logs:CreateLogStream" 10 | ], 11 | "Resource": "arn:aws:logs:*:*:*" 12 | }, 13 | { 14 | "Effect": "Allow", 15 | "Action": [ 16 | "s3:*" 17 | ], 18 | "Resource": [ 19 | "arn:aws:s3:::dataeng-landing-zone-INITIALS/*", 20 | "arn:aws:s3:::dataeng-landing-zone-INITIALS", 21 | "arn:aws:s3:::dataeng-clean-zone-INITIALS/*", 22 | "arn:aws:s3:::dataeng-clean-zone-INITIALS" 23 | ] 24 | }, 25 | { 26 | "Effect": "Allow", 27 | "Action": [ 28 | "glue:*" 29 | ], 30 | "Resource": "*" 31 | } 32 | ] 33 | } 34 | -------------------------------------------------------------------------------- /Chapter03/README.md: -------------------------------------------------------------------------------- 1 | # Chapter 3 - The AWS Data Engineers Toolkit 2 | In this chapter we reviewed a range of AWS services at a high level, including services for ingesting data from a variety of sources, services for transforming data, and services for consuming and working with data. 3 | 4 | ## Hands-on Activity 5 | In the ***hands-on activity*** section of this chapter, we configured an S3 bucket to automatically trigger a Lambda function whenever a new CSV file was written to the bucket. In the Lambda function, we used an open-source library to convert the CSV file into Parquet format, and wrote the file out to a new zone of our data lake. 6 | 7 | #### Creating a Lambda layer containing the AWS Data Wrangler library 8 | - AWS Data Wrangler GitHub site: https://github.com/awslabs/aws-data-wrangler 9 | 10 | - AWS Data Wrangler v2.10 ZIP file for Python 3.8: https://github.com/awslabs/aws-data-wrangler/releases/download/2.10.0/awswrangler-layer-2.10.0-py3.8.zip 11 | - AWS Management Console - Lambda Layers: https://console.aws.amazon.com/lambda/home#/layers 12 | 13 | #### Creating new Amazon S3 buckets 14 | - AWS Management Console - S3: https://s3.console.aws.amazon.com/s3/home 15 | 16 | #### Creating an IAM policy and role for your Lambda function 17 | - AWS Management Console - IAM Policies: https://console.aws.amazon.com/iamv2/home?#/policies 18 | - Policy JSON for `DataEngLambdaS3CWGluePolicy`: [DataEngLambdaS3CWGluePolicy](DataEngLambdaS3CWGluePolicy.json) 19 | [Ensure you replace INITIALS in the policy statements to reflect the name of the S3 buckets you created] 20 | 21 | #### Creating a Lambda function 22 | - AWS Management Console - Lambda Functions: https://console.aws.amazon.com/lambda/home#/functions 23 | - `CSVtoParquetLambda` function code: [CSVtoParquetLambda.py](CSVtoParquetLambda.py) 24 | **Note:** Make sure you don't miss the step about increasing the Lambda function timeout to 1 minute (Step 10 on Page 93 of the printed book). If using a larger CSV file than the file provided here as a sample (test.csv) then consider also increasing the memory allocation. 25 | 26 | #### Configuring our Lambda function to be triggered by an S3 upload 27 | - Sample CSV file: [test.csv](test.csv) 28 | 29 | ##### Command to upload file to S3: 30 | **Note:** In the original version of this exercise (as shown in the print edition of the book on Page 95), the path below was `dataeng-landing-zone-INITIALS/testdb/csvaprquet/test.csv`. However, in Chapter 4 when querying the database/table that gets created by this step, we refer to the database as `cleanzonedb`. The path of the upload determines the name of the database/table, which is why you should use `cleanzonedb` as shown below, rather than `testdb` as in the original version of this exercise. *Many thanks to the reader that noticed this and let us know about the mismatch!* 31 | 32 | ###### Ensure you replace INITIALS below to reflect the name of the bucket you previously created 33 | 34 | ``` 35 | aws s3 cp test.csv s3://dataeng-landing-zone-INITIALS/cleanzonedb/csvparquet/test.csv 36 | ``` 37 | ##### Command to list the newly created Parquet files in the clean-zone bucket: 38 | ###### Ensure you replace INITIALS below to reflect the name of the bucket you previously created 39 | 40 | ``` 41 | aws s3 ls s3://dataeng-clean-zone-INITIALS/cleanzonedb/csvparquet/ 42 | ``` 43 | 44 | 45 | -------------------------------------------------------------------------------- /Chapter03/test.csv: -------------------------------------------------------------------------------- 1 | Name,favorite_num 2 | Vrinda,22 3 | Tracy,28 4 | Gareth,23 5 | Chris,16 6 | Emma,14 7 | Carlos,7 8 | Cooper,11 9 | Praful,4 10 | David,33 11 | Shilpa,2 12 | Gary,18 13 | Sean,20 14 | Ha-yoon,9 15 | Elizabeth,8 16 | Mary,1 17 | Chen,15 18 | Janet,22 19 | Mariusz,25 20 | Romain,11 21 | Matt,25 22 | Brendan,19 23 | Roger,2 24 | Jack,7 25 | Sachin,17 26 | Francisco,5 27 | -------------------------------------------------------------------------------- /Chapter04/README.md: -------------------------------------------------------------------------------- 1 | # Chapter 4 - Data Cataloging, Security, and Governance 2 | 3 | In this chapter, we reviewed important concepts around data security and governance, including how a data catalog can be used to help prevent your data lake from becoming a data swamp. 4 | 5 | ## Hands-on Activity 6 | In the ***hands-on activity*** section of this chapter, we created a new data lake user and assigned them permissions using AWS Identitty and Access Management (IAM). Once we verified their permissions, we then transitioned data authorization over to use Lake Formation for fine-grained access control (including the ability to control permissions at the column level). 7 | 8 | #### Creating a new user with IAM permissions 9 | - AWS Management Console - IAM Policies: https://console.aws.amazon.com/iamv2/home?#/policies 10 | 11 | - Resource section of policy that is updated to limit access to just the Glue `cleanzonedb` database and tables in that database 12 | ``` 13 | "Resource": [ 14 | "arn:aws:glue:*:*:catalog", 15 | "arn:aws:glue:*:*:database/cleanzonedb", 16 | "arn:aws:glue:*:*:database/cleanzonedb*", 17 | "arn:aws:glue:*:*:table/cleanzonedb/*" 18 | ] 19 | ``` 20 | 21 | - New section of policy that enables access to the underlying S3 storage for the `cleanzonedb` database. Ensure that you modify INITIALS below to reflect the correct name for your CleanZoneDB bucket 22 | ``` 23 | { 24 | "Effect": "Allow", 25 | "Action": [ 26 | "s3:GetBucketLocation", 27 | "s3:GetObject", 28 | "s3:ListBucket", 29 | "s3:ListBucketMultipartUploads", 30 | "s3:ListMultipartUploadParts", 31 | "s3:AbortMultipartUpload", 32 | "s3:PutObject" 33 | ], 34 | "Resource": [ 35 | "arn:aws:s3:::dataeng-clean-zone-INITIALS/*" 36 | ] 37 | }, 38 | ``` 39 | 40 | - Athena query to validate that IAM permissions are correct for the datalake-user: 41 | `select * from cleanzonedb.csvparquet` 42 | 43 | #### Transitioning to managing fine-grained permissions with AWS Lake Formation 44 | 45 | - AWS Management Console - Lake Formation: https://console.aws.amazon.com/lakeformation/home 46 | 47 | When asked to exclude the column 'Age', use 'fav_num' instead. 48 | -------------------------------------------------------------------------------- /Chapter05/Data-Engineering-Completed-Whiteboard.drawio: -------------------------------------------------------------------------------- 1 | 7L3XluTWsS36NRrj3AdpJDzwCO9tIhPmRQMJ7xPefP3Fqu6WxK6WRJ1DUtTWLpJVyAUksBBmxoyIBfAPCNvu4hi9C71P0uYP8C3Z/4Bwf4BhGCLx6w8YOb6MQDeM+DKSj2XydeyvA/fyTL8d+HV0KZN0+smBc983c/n+6WDcd10azz8Zi8ax3356WNY3P73qO8rTTwP3OGo+j3plMhffZne7/XWHlJZ58fXS6LcdbfTt4K8DUxEl/fY3Qwj/B4Qd+37+stXubNoA6X2Ty5fvCX9n718mNqbd/HO+gJfLn98wzyXhezzudde8ZOmP8JezrFGzfL3hr5Odj28SGPulS1JwktsfEGYryjm9v6MY7N0upV9jxdw21yfo2vw8qa/zXNNxTve/Gfo6STHt23Qej+uQr3v/iGNfJfbVZv6IYNSfsC9D2191gFJfDyt+Iv6vBhd91Xv+l/P/VTTXxlfp/AuSQj5Jiovm6Bpxx6ibsn68FF723SfxXXc9/1RG0zz2dcr2TT9eI13fXUcyWdk03w1FTZl318f4kmV6jTNAhuVlmvTXHW2ZJOAyP1TKT9WW9d381blg9JfSE/y9nhDkk5Zg/Ada+vbFX1xJ6D83536Zm7K7xP8NL76J55v0/wAj1z8CuCqTj1FSpn/d9wNlgcMJAoLwv6fZJJqKv+jhmwq16JU2Vj+VHzaDcK9+nvv2Bzqe+/ePTOFvzOlvVAvB3z5/vV9wyWh6f7nRrNzBPJh3X4Kz8Ot1sumbTRbRG3yh3XMA5H+Ktgn902uJ63T+81bOxZ/7V3WdZfq1TAe6oX+CqL/9+WRJGPbZkDDiT/CvZErYPzelNLlCxdeP/TgXfd53UcP/dfQ7J/zrMVoP1Poh+Sqd5+Or/qJl7v++cvF/JPypX8Y4/Qf38xUX52jM0/kfHAd99WFwc/9Ql2PaXIi3/jRO/uJqwD+pQYu6pOzyazAEDva7x9t/rLd/xWkg+FsY/AviQp/D4m+Jt8S/BW8xGrkx2P88vM0vKf45uVjFn+PrV9Pnv5DhwBj6ndmg5A/g9U8I9dl08F/LdMi/x6fYr/f+3+PYyOXYOPKdiuDPKvpNXZv6Xyr1H0GlUPx3T6W+5fL/Y7gU9K0S8U/JFPG7IlPf5v03imCbNOr++7gUiv3uuBT0M2ox/1legvxcL4F+X17yudTzSRHfohlXjheWfwk8HZA2EPr/S1gE9PaGkRcd+BQbIR7GYf6TE157so+f32OA/GEwHNMvtiPHYD7M9fHL1ndsOO3SMWp+IYfHYOq7MPkDd/8B//029stb2f/Wqv4zCBZy+/0TrM/FKnYZo/kS4X9dZIep319k/1zE+g+P7MTPjezY7yuy/4x61f9G9v+wyI4Q3+Pzvzmyoz+DP/7mTVUIxb8DRfz2g6Yq9CNgRG/EryWqn0GCfmtRkZ/Cx7cKy79RTp8D/NdqKR3H6TT9/uP7L9Z1hlDoO/X8oOsMIb9p1/lzfP9Wy+67aWnT8b9IQdhn//mBgn7bLtXPqD/+5oBMfR+2EOxzHgFhv+kaF+JzffDfLygS+jmC+jEe/2qC+lyr++rx9w/+81/k73+kfs4yoN/W3z8TMLnL0+nL+qyb2/fNf5OCiO87RL+lgpqjeEmIHtpwMOMZSQ3z/uefs0wrv+Ty/vl3/5cFoNHr2xlu/1gqFPpdmIJw5E/oJ7n8aJEhhf1KcvkZ8Ps39vlPs8BPhtmAI5korvMPq/tRnvnjzPGLK3xbAAv/KFO93cibcPuSGH5Z53r7Pm39PmUs+4n4U3kpbvoTWPXws0oU/9Ce/rlP/BOL/yVWO/xwhp/jxd3WQLhIx2vW1wZ8+1Yl+j1jEvRjTPp/VMpXDyR+sMb3t0Slz1GDXabLkz7oO94ATbwuVeE52LLGPllAkfnTHiedl7H7wQ5zTH54ql9B7U2azb9zpX9LF34am+DPqfZvagM/Z6nCNxgr249nB5gfQPJP0fOfAvXPwOWPi9HfinA/rMh9nQ9XzDN4OIIGkoCFOOngD5DNyku945/i64qw8IG3sPABvtffy5TSP35Zwz59yEDAQX9KgG9//uOf2fvzjxBM/und5T8wjO9M7hdhlPh3ofkHy9UI9LNVfBv75a3iR4H5ixuX37yY3y+g7aLm2w4g3b94OFDGzUujufgA+y+5wn+k1/9f5G/Ud9r8d/s4/HnpoR6NdTp/WVV8v9yqvEQ6zf+x8fgXKNrACPGjZ25+03j8GYu/FT0vPzv+u/UDwT8q3/+2+vn7RZAYNL7+yz3o3+8/P6MP9f/a0byyaE6A/2bfj9qmP2/5/v/UbuewlHE9fdH3L9MBud0+oTWJ/sja/jL4k6roX0Z/eYv7XNj5igjPclouhZxfn5G8/R8XFGqi5boM3Ubnx5gNBHUHE/3//ntw4yK736kS+u0W/v9YiT/jybf/gSkQsKcuB5/+uPbNH2H0j9mYpiAHQr8mSPPxTv8IYb9tLvQDZ/9xoRK7/ekHKdFfR395O/nc9aSTP0p9DAjAR3HLXtKxBL2Q2//5i5PTVwrUAUToQSI0lW3ZRON/k79DyHcOTxGfdAnBv6nD/9pP8V37UJqBKfpf4wkQhHMY81/DE6YLutqo/pjCLxJYvlvwjP0g7SZ/QA/IX8vMPj9RpgOUAO04ABGXJ441GPkrI7hfItGBSP6bAIL66ZOAP1rN9pv26n5GU+pXb9YROPxdDPyZzQLqF1hK+uOS4Ofk9+eB5q8NWT9AIOSjvfaKpvTPbXmNgov8eUrHtYy/9xEA4zyBMOQPYPybS/1fdgb+ZU/4zHG+hc6fMJxfTcOf0+e/ABP3VZ7Xpv5Non/40s0DMv1PBav/V8V9a+T9uyHrZ/jmr7++4PsmBkz+aBnpDzg79WtF4M+JnfaD5+z/FSv96PsTFId/y+D+BsM+mez3IPcX0/2ZIPc9GfshvE6XB5Rd7n48E/FH/DNLFT5+fgCU17wu/vX+099778I/NrV/aenBj1b1Yb+WN/wgTbtkVADs8tLXH/5mDcKHNdyEK9X9NZZH/ft60j9fR1/3Ej99XwaM/un2tz/Q58hE/Jb49rl/9b9+/Nv58be9P124AH0G99/Uyz8nV/9rE7+5TXy3EvYHje7f1Cagz0DxySJ+PzlKfU1jKqc/X4aZRu302RoFjMQQ9JdPTP5lNX+3ZukHjwd9I40/CQfwr6Xmz76vfpHlNUjn4O5//wkJ/Kv6JfpdDZ343GMhf6SyXy1D+T2sgCbw71tP5Od14T8CLBL54fPvv7xhf05Y/pKB/9XChXJMi376n5Z1/8srKYmfgtKPdfRvW1f5G/QY/nctwj/tMXyLsR8vGcy+Oc6vYZT/Ls6D/qxH/H9tZCW+X8D68zJG4tdamAH/jKVA3/f0f4mW/d/xhh+27X/S2f/WyP/yXAj95SMsgLY7zJZPxnS2myrmPX39GPdHwT/ya0sir19g5S1LB9cWJ67OzQWH0L5xd24yPU5ojNvgyKLx+KZ1HzdaBd+kc5qh+YimH+ADA35ZOf/6289Ozl+X079+lq9/kzdN23/Z736cB/yw3scfsEsJvu438+sz93X8+szm1yX5vxwn/QFmcnr72xHyY1PVrCjawFaz3YXmBPfCX1cSdpkJL3MR7uDOr1MqDi88UlqaM6/dwmcCIQvhj2Tm52jk2Sqt3O28Zowi3+iwVK7LeXvF8ajwTAe0U2JlFN/T2wrON/losmO7zkyVyrDLVvMY1fhxYI/34wgdVdr8cIPtU5zy252uDFlmRbt22AjXmheSwbhPUOQGvUcCsdYsui5EhI4kx9ZwbcZUmOLQmnndOmdBSBBji1zsVDi0y20F4nYS90bzrs30NubTbL8usHPSV9Kw9+OAhRldosA5PbdPh0Z6LTYTToPvCT1tQcrrrPnIckTlhQ++aqbFJShXDW4LrB4TWhAkIl9wb0MwphvIpiHo2+lOq0hYLFPpkTLkenqXft9hQ8w85/CWXRzxErG1c6GWyiequQZqK1VyjTYaATlgwtUdWf38BUOrD1Gvaw/FtbxWGhNeme+t5qQn5fvvzgpwTq6LrqTyhoG8TZBvPiLLTkm+z37OuOnFwc4LfXbMVm/dTELTQ47zabwZkrI/7vxhO9itrq8r9rZGO/1LLsdIZZUq8MRFLs1XYOarYal8AJa7OMPqLsMuqfQDDVf8VgmbGTjVYomMTiYMz4vXQcGObvH1N9LNxGYqhpO9ir6TWF93b5hVS+BFnPhQCGYjsWh8gzUwZE2Uzb6lfJ4G4kBQj34ycA++STJjw95EvxPsNfKmw98jvtJLPjrFtC3bSyy0AGfqWPveyuMozmbk49QkLnj0J8Zxz/ptX8fUgstFw4JnG2TvBQfMInwj1VifXSy3bicukw9PTP4g6Du+lI18CzBfolmea+t8CCkbp8RIlenwldlrIphKG0XSndGvU1c1OUeCnkkMeLThmq1/2bX1eEA0preD2IFjGIHaaLn3rwOWUF6rXoRabCH5mQroTmlzD4XbeL8l+Wu3x5W5TfK82uSRPqe3w4ychSKm6KeqbR2t4/i7t/MFum3pYtCReF+vC+j1ILAqH9tUFLYEhRW3vSDLO/7IJ1/FmBrZhLvJ1938MOV7R7B6ubyGpK0vUwjtghFhcZRDhqfXyWlx/gRLl0xzHKw5qDc6B3qfmszjSOJM6etyQH/Xf3NRhXc7eCosCiV183gUccG3lHHUVqjaUUmPr724N3NKNxRuFh7vYCvH3DKeC3pV6rqz5PgT1ccXLmX8OOzNBNXhELYK82bcpOtvmew06OM8Z6p6XHSciaxnfgoHM5pgVRXfrKMy8gXwJfmNQpp1bZxaiLCGwue+hkx1FrtbWESuHpgrX7LCFIZi/WYVltv2y1IZYzYijbgPawsrqak8KSlE8BOenCumMbdILrLhwkGGg0vSeOiBfQW9VyNwreBHeGX0Jk7lnTp1pxAtFOkJNssRN5O2HUFXhs0ejC1PNHJiK0/TTbE5g8LgilzHE+xePNVC8/VLhKrgj+3zVkBiTZVms9kiziYork+QbvKvwDIx2kXvqeDHD2cZASx3ukTtx3zfSB11bH1pjCfXDwP71mEeakbPLgl6fzs1SQvjGhmtegPfiu0i3W51YG8Vq+MPbHyUg932ontn4vJV3j1mE9bVjauDeGl0JPTudefzZdmWjL5t2oJXuaOt8SCwfAmjk04nC+LVh/w24bnW7FCUZnrCXM7pCvE4/NPwQwDQZ7T0OQ9XeEvTa8ZlnaNgTf6kZNnN9VxnqXrFnap5lAE9xU3tI6p2W8fcnfZbBQcTDee+s+CK0kh9TnmYGSj1Eq8KTMs8gEleEaUzKGk1iFVP7Oe1aPdqUB3zVB1MXFF3uSZhccvZ3lkzuz+WW6peJ/QBnhInfs/kSQnQDnHa4LgndGGqQc/YeiMMihReQqEPJ5kCx8lx1TMDdZzMKxicA+sLT3omk03QjOC9b43UQnJYQhdhizLbgbVWynVU01ndR72bfr8uJ2n9kazVIPfyy/DKecOglbUOSsXPtdZrvQvv6Rx4tnUrZfGg+HZ16CjdivxNbveXRD91EOUG1+W3GwcLdT3jDxojogcbprMq+tGDRsWnUuUvLLY3LGvxRrMlVs8Up4Bu4W3zhA6/JWSw5F7LsDX7Gu5FoN09rq8zpPBfo5IL2XirAYQjSm766Iu1mpp5vBFTvXli+Kgi9IEG1ywE/Zl7jCL4fT03JtJ4Nw0zfbeH00qxi3HX87DcjUXIhm6mbz1KiXWgB7455K4HTj87UCNfEfWBYvfu7E2Sqe7gqjwbooa1H6olS5U87Np1Lb71ec8lANLTOKYRTUGXzLut33LxXAI0WAq2UvytUq8jJkNlarQ9M7EehqPWFup9GQmDFnbj7pAZbXYi7UjGe5wjaM8kV+V4DfAJ5l/kISgYSRfvaiY57bGPjuASN1QxUjeOH7t8ul790GEQoMHKT5NJlscTvvkBvwXGc0tleKRy1a4/0JGpdAKS2Nd+5WKrdDPnwfX4emovVsA86jWwZO4Fk2wIjvZD7K4EHj+7IueGUmYMRDLrCs/aiZWoUlISkwiginmTB5cV+W0KM+Ku7ChunUNrNy1hlL6dQGOgFGScVJ0CK0HQPl5QXfaYiHshhwj0Q/cH7eCzSczdqnrCTKGNPSjiC6XEbiysE6yZwkVDxiyw1f7gBeAiz2ysGZdrQLzs0wotWZkMbUEZLjS3sZAmKNr3+921LiBPFVF4HZcrO7q6KhD+Kvj5YDYMbQW3u871FrSGNni25LRaky3ZUFZdEvmdwNl2S9h7GkftoUthxcZBZau4g9rh4eg7/Z7Fewu7gO5FZFqqGakal1+yfKV2jMDpaT8+W7msCeKN6KZcYkiiQ0kO8SVZvSKXy4ZtOP1H2NBjed2IsQcktVKOU/Ybw+my31/7dVuJdqpf9+nudF50IbLBjNMpwIEgSzo23m6P5n4HpqtW9HZFfP6DcrHvMn1rxhX8BhU7wouKafv7jqF6P0T800AdIsZN3gYuM8kE21uy5jIDJ5VIFTS1md/vt11618zi3ARvkbpLtc7yQmM7jMWEX2Ec0vhHS3t6rXlugTv3gBU2srBbwyjmisFDee+I+xNP9bK1z3i5jARTQuwMiEbeKVuuJTVHlFJ6Yfwo0S9gb8viOHQMiJPw1hEapltDQzPatZgeEIkn7XFKZqIQxb4yD1oUd/VbqbKVOq6lwaSxlm+8YmToSTxyzKw8iw0ghpyWGjjaSb9lpXNm2UIM7VVHTztZHC0N+lU27KTOpMaVEJrtTVj02tyyJaa/SLdOG+Wa+Hx7dhhCX8iVKyvE7Wmnv+nAUAIlyHiTrALq4NlzbXBiqnkr4vN4iJL5igNRbDdw0fdq23J3EK8vKcl6MYvOSxAfjdjoRC63DyIPIopcDqgyeHWYHyoLbwWfD1i3i9L7lmt4lHv8KwM+XkyYdx9ddrb7oMQMNMrjRiRqoW0kdUMWhN/Mwt/5iexizWNDh5GsQEE5Baq50HNaJmHDp4hYMnQ4DtdQd9eHedGHR/Kx7otUMZWrLu7d9hViXj64xgXBch1SqBJaxuL0XDI92+31sUb8XvsDrjucVz+pfuHbEUvZMy2wi/gJKE+Q4sXgSTpTl+V8+MHgdCgDMZQWB4k10f1h1vY92GI9ATxbpLwggKW3eTwa49BsT7kifqEkQkA84SIeX2utOVGiMCjupxJNIUE1p4+qFP0SkTnTq+PBgNO2Yhm2yOO6IM6CC0VDYOs6uqjPYaP9idZL2ziL5ngy4+cT6kkNTmJwPHZ0fpNtiDHzhqYcRh807AgqPcDy+7PB+JOyw1vFy/SrR7w6cR5cTk16aYkiQNzwVqQTo8Z3nBJ8dRKn8HnBF8Oq0xxmARVdXBgA7/PSugVvTIyz6pGSFUh+dy0eG2aoKJURXJXErugDPHng6HBs9axhwkeGBIaD4AGZ4J5S7EpPvmnUkMrMYtCKWSuazzrs1cokZAoaMSlXSvDsTrlwxFc5csIJmWNAGRJwsE7DLcgJK6w9g+1BJztJCPWhOzxJhn2uyxQ6D7ZEJZqneUgnreg8pzdiHu/zAUwXT4gIghJrJLL7nnjw00o1/gIQoert5UWL5VsAvCpfbC94aKJwa6GiDnrdNtoU3tdM2oYEdUp7B+Lw+8iin9eXn+lz1y98m98Z9A6YA6+lydYV1i4A3Lc4XQTSHhlo8EB17ZQddwmHQmdVVewfUzRV8uEJbIXTkcM5BHkXsoBW7cKqIMOvKlOrEHLw4Y5JpteVlvhXou4TSJwsUBZVM6y9/Ob1csbQGLE5e8Kv0YP0I7NmzJBOIuMqDBpdYMzQ9WvEb9CEXpPKXtjM+NGdRsyU4h5SyT5peq79oA1aW7I/wNhbSIutQTJAb2UIvp26e+6O68aqQtMCgvgO5yvcH2MlZ66f0erG6kcCewX/0LlJkEwaGT+IVRXP4iQ8Y63xl+CiAyBtpMrnTSdo95UA1UARqJQlNw9D8RiDqcx/wR0K8rvp3dypsYphClm8l1dK1kJIb0BZ5gpLlsvNqhWHrA7emeuuc5AbJ0OQpJ1tOKjag/DeS6oeo3GbSfaQmEAa0+HmHJzXGFXBr4g0nqgjIeCmp7NA49MvrdQkSBP4QKGYPkUqG92/n/Dx0OP0oZZIeo9YRIga1hkSNUZrNzdOHyHECsmkGbAlBBOlDZwgx/RCLdzd3RGmW3Hzab/LMboJaaIRBqKB24YRguS8faRGvy4Br0/h9gl86yOrJaZoXBEXZMDVoHHevGi3LX5i8Tx1gQrqPTMoLlaMWQiq68gVE+9XgAuxHXJmptttMSiIENAEgS9lqKo7XmafGUhtEcmWNiTywCTwsaF61Se847Ivfb7P6saA1MaV1hO18XW+ovtIuLZJwPFOx0RFO7mXSnUIbezlYU6SEhn7gmcCyRdQG2KydOVhN7C9rgx9Cqbmp3EcCEZQfYqsGzFbs76TJAdu0et6g+HqGWPUvmjfnY6iTHoyIp3S2OOCI5Cf7jxXZ606eU6XxWJXiUXFSuXo3+GbnFFrZ7PTQ3WLSiNbZ2PE1PpSQkAUh8RhK4zXO0JsXWPNZJCZeAWYJF7louQkMEZIUoRTXRzW0ZhDqP4qdfZ2QenbpzxqTVUiGUZQ6IIMd32FwMSY2ZYXfqBADSwLYeYeSOMOShDgxiH3CmC2QQ4G8TLAwSJbWHQyPJTLJJTXkDdBkF3gKSi1g8eQ7RPLlU8qKkOazK1wz0F4XySPwW4D+ZCBDXF4iCM9jURoJVq5ZfkRjGRICjwZcaXzDC83x+cCmMiiBmMKzi3dAsu4t213u/ybzbjm8TxvY6rIG2uxx/1yGMgMbJ7Kb915WRSOiBdCMHCPIC4F2IY3ePTq5peuPNzHtDUjEQd4HWd0TUgdzxH3jVwKZCyqCEoaeEE5xqYet7yM92M6qgzetCszOubjih9GSJ8yL3vkU6c11BuO9qy5LhBGSZ0Lhe+2DTLpY0zZXYXeerQHVbI/ilEHlTOxZyS4voTOXER8h042942TO2oV771kMPF2wS7+pphchsaZD7+BembqnQJN3EYiCX2C4pw8iVlg5uISZMtSn/lz1CPz3anEjJiQc6B7r5nhIS/vS8VX4n6iBIteSccAFMXUC0mByLC3jG/SxfQkYlG5ka1H08EFDd32JrR5fHAnRaYE6gnjFSC9OaprKYIPSM03+33OxzwY97gvVrqcM0Nznupj2M4lRKp3JqEy4CP4mz8O2Rxg9nRu+42xhsUqYz/079wkmmSJ5D0osq54WR2wmSDzDBgVaeBwlqZeMt1mmHqPWVsI04tA48dGP8O6mTS3FwcOjky0JyUm0yQTNaqJfUmuUUuIZ76uk677887F3qWcITa1KUdpnysvCDiRYIZnMDlq8tboGLFGrw8vSvjunU8Xh6ER+kRBWWshhLzCqJcKXtkvELE1o0Br2XPsYipkHmiZ4E+mx4mqaUQ7mwUe2G9FAla5OCCkUrbkLiYbvyBqB0/vTbvtl49xxfcWmZ8eFUdRegXvdLVkxt1kb6GfruGKrAxKrB/VJIGrX2IKsyhJuxWykKo4tkKLNafE7w7JiCJLvZIFg1icSIt5qdf7iE03kWBWY+4h5Ip40thxsZEyYbRGFn+wloL3T6+YEvqiagCrEs1EoNJNYmZlLXQFlZx9BCEs8GLltV0RWgNj6ABeZycQ3jpH4dYWYiJLtrhJqemWsh7Q5IDM5HVAzZocSdgzqzkVWcpua9JivaJ5Zev4EeMLevOl2meplDGKCWi/jWukP3tx87lw2/wk11jtQDH82lUUOdJU2ZoOni47/VMcA1Tr5fGYk6eLUTMnAT2BALhsiUKNwHiChKDCa9oXV54/ulUwszbIfHZpnPCFVDinbth8+fbEuAVgVlr+KN8LHQYORl3BhLRlDAXlwZug41zO3PVHGhvyc/SPUcm2O7JqMajJCzs1+JbVVNfJjcalcvRJdj6bPtPd7JtiVP2Z1sYnlb2kFKFvUsvKIAgyfAlIVhragGN7LcuPbbIPUIhMFwtpTqvrrQ8oTWkdy2nwFI5QJ6YFvgSv6ayqREEk5LGhoAMwUhu1t1k0Or6OUEJ1GDQkvBU7wEEGHGo8ALh7nPEGMYmD0QKBL/7T6rvT041FRpvUZq7IM19/CMIi1ycsgMdQGQpBvWkGl7imWCvnempGGz1oJ50U2zBmAJ31MMnQBO1ifa8CFEHrCRmwZuNTkXgklDA83acCPfc5oKuhvmUePN4yc/aWHSyoEE57QXD75bzA068v+AOMKfW98HBqOQ7P4S0f9lQ1xPR6AxgoFqWFH9b5LmOl3HCzSdD5aTGezbkOCgetKL8EzeNrXLhP7OVILKtA+ZWp+dmkYDnOGTmuqrlSmQ/85gU3ZqpHFLWkshZfxN29O2KpoY7inmsqLE/XxU9D4lCUr8ijWivOxfYxIRDkzaMgLhpnbrrxOl8cdXkiKy0PwAr1vjD4needB9+vT4+/TURD0T4y94r0rsgnyE38S8esiR1TgJ62goe5EufIwwGxTincia5aTA1hajvP4sBdgZZZh6Lx9CEVFW0+ZxN7RWu5+LOb7V2QU7gSHVMSvgChVW0mRQ69RHlgNIXFETaxtvolXP9dRN6rP9uOV8sbz1bnsqg7A2UvGsmKDJA00O16JvPLTakYsE6Gfdon8IlIhPoVv9f6Dug//lG/xamb01/8BLgfh4kXow7YvJtpqscv72I1b3z6GFuQV/rnUWUQbKHLLX5em0cxDY/yAYznFofnFe3JzfRAf5vB3sDul5dBRGD6hgoRsbd6UTfWMo5kW/AGsM5rjs2wAJ8m8bS/+Pd0TBAJmgyovmXRQVCnCy0q+2CASZWv1AxeKe3qOhv4nYKyGeghGe6cxdki4pgFYoH3yoqgfYgSktNr8drZht1tkqmL1/R0leFe3GssIbyzGjjAJvxgqo1pdFVkGFlGi8u2Et/4O8JbttyeqbPBt6WP7FPXL/4g7PIzBJ1Mpngwe3caE7Jm0BqxDeAldwQwWfdAUuRDDIixEb6MUEdCaI9Kgjpi3C0AbB7Ya1xgCvejNSHEOibpxVuThd4OBKJb7GmTtScdzPOtjG8IiEhz15HDOFk+MlbAwMtihchL9vJ937HkEESnHdttT7YdNLWZV8iaNYr4wRvvVjSEkrSSrjg/XWhqNFry5oct0oSlYo0Foi8YCFJJf1rcNF3h2Xo3O+6KIybHK3RlCoulATmhEbVliy9NIfJYW0uX1UmOBKtNEqXyoWL+6LY+31TvZsRt4JYxpTRdgkFzU4B2NDZAHEKPR0zHll7ULDjn1gNpdH61gr9cFLq7v4kzqgm7wA2DUWPpQdDsQ5Kg3QTcaZCX4SAIxZUeS0zi+hMqdzYkEqqjWaezjxMUjvsyklvQ69PT+nEEppuDgD3yt0faykc9CSL8rg8Gfllhlj8Iv3sN3ilxGDRXFACEe5c5sI6WdzMa0Pe8QlzVLzUa+iQP4vjXUCQUIFVcLZ6XL8piIc8GvHkCgdQtj7zMqSMWa43oCCJA9eNNTpgMgTwBGciDyIipvKH0wmg36imlG0pxw2irAhapJP3oqShA0EpCDL9rAkPjKqbY7uQbOFYIF5vZLvxCq+kjrUwQvEy3Im+7vuAaU+PllWZcByr65aVZQeh6ZhV3cFdZQVVGtvYXPzCwZmXUDDwXy4L6+pXlCreqAAH8dj7QfbpYmOcBix5A20laQx1IX+06F8oRlSKz0aPYW0orwCt51qG7E+TcVwZJXmSQKm86LbCX+GzmGAr+WD6Ky5C7sLN0Ugqmay6U+j5qROgiQ4745jmWgkaLKjnkWY1SEKIPTwcRti+t44aGeBla65Q+moEeRPoO3fG61jo76znUtu1XpzP0mmVQQ6QRKF+nKJj8C0xengPzrQjNS45e3j3Agpozx4io/PV0Cd9AuoStYTkArw8QgoQ0HoGEk3iclTExELv29OOhoAG5YmQmANzxETMphixsuhdKK90z+iIEfGpciY8BL0uLv5TlBsDPuj+PBSRI1jkpw7BfJGdvQD3cNbJWPHh9urKlcjuEJ5pc1KvL1wDUtWIpyoIDO1oI4fV7fBRiZm/pg7UkWr3LDAiem+pk3MLPRW/PmaQyx/vWn4iAFJq+MQaD4ZR7w1XidE2ZOokcRsaX668ZT0tvi30m5jmrR4r1cHPRmbDaT0m7dVWe7+tHuT/hA92yQAQJyRXGhMJ9gVxugF8oJRIHutFe1eYSz1pyr5AmlqW7BlAvmjb6FY3nRyLDBxcDUT/yIl4B/MvLLecOHTfhMc6WXOouafj6492RpBUbrQgCLoJuoF4+VDcb4t9bHQr8POH4kMzAeODTK1AWWqpm0LFIs+hFo4VnRlQyIcxKyPHhsg0iukg5Qumq7bEv+8Hhb1EIVpMiUnKPayYlwHKKQ48pi7ZO/PlOLUksvnnyU+Stznl52dSZjR0873OXtCIrsA+YxosxB6BPon3f+qH6gYcQSsGGRHRFuN4xamCEG5HuKJaWxcdrDjnCLcrE5aOn3Lj23dE4VfWSouHXQKV9PF02d9s7zjZ0Cq56I5jr4dkGwJ8H1tm4LQ7AS+4YC3dfWXPl+gNMOGh8epugPZp3Wvm9TOMzneoKlVZdhwZm6llbywYZKPK/2AWRuoexTeAG41cjM5TAZpoN+P8UbZRrb89asWVZfe3vffQVKKkqUFp+Gh+9LMGHK0JCByYG+kM24lD9Co8ZH1fqCgT3h5PrJJ+u6l0Ak6ZQigcimgaFLzSIKw79PtWxK5xy+xKxhTXp22JL5pHbWQcYEvnWYbvomCSN3HwemkpXiXxTiaXeMx4dAJKPuCXNz0zotf1eR3In0bpCINOr7Wa1YfUnjeTZjuU81fcsPQ2cMsQJc8b35D1Zuq7YHLDcyVLF7sTp4gwRXAFIyBSkmGUuUhHAzDOd3DkLDdQtVszlfj+g4oU1UEbxTJQGoaoxZtZu9FYV5wQMfcquVNN/Z4yIKzitP5mB9PA+f5ePcIQeRmD2x80kCcHWngN8wPDdfHgRK4nMqt3Q1rSkqRYf2pDGJIlqWUPqpF8hzS1NBKkA2ebTxNTC9OVMNa14sFOhRVipu1kgwRvV5HBJfYpf2TLQ3Sp1vYKIcrZUxIl41noBwzEh6PTM9x01MqYLJ5ujSyRWP3zQFwiUQO9WCWmWJ9mkYpZkwWKhiudRaHO+IrO08n7J+ASw6JpPNQuSxN8C6e5URq/eCpxinZ7uKtpDgjtWvOu+T2y+NZ5RXnoF4fPnZI1lYUhWVmmmhkxIvw5VURJJgd/GYICMiffZpyR8LDPLAFWxYKqC1uWlgAQFlxOzAgvFMn8zAJk7HaoFf9OzxIOLTdGdDNl4ActzCLUL69tEtW3vR6YmiRP0coZTFgHjj0KiNS25E1Mh0B/pegbf9ppeH7lNVvRwawLioPt1bXmvnvp4lpzw0atKST9x45WrrnY3Nmeg694fgOjeWAVsHm8MEgHGfmriiWFVwmOls6N3ejbGxJevAyXcykjEMKk38oUklE8iG6+0vg/30AwCCRDWLRFg7Jz3W5Ag+tjVmJqIHMOo75rAAFvaYUSu4mKj4OVC6q0B3oIo8TWhQdH08GBeizIcySO0DplYWOVKCKkjj1ErhVmIWUAmztweZTm8Qjd+QrqCvDEMFDTkECzS02lvyaMHZZ60q+RYJgeIrp+H78KcAqfZ5RtV+FGNypudZOFnTUpLJesPuvfsY0illEwRH/9ye+nFoPTsHUtSlbTEinGcda5BymFuaW7y+75e+bBpwmJRbZAlEyuMJjySSS+6eGaUg6Hkm7KkXAiOzhcn9xTg18roE3v3+taQClbTGIazjtgZxmAqHzACaxN7M7mYW+YrGwNEBwm9ImOsYRAkTOUvlkct9UtvNd93jf5IkJ3TnWDX19x4AJR3LY1vSPx2Ujs2MwiFV11CCz2xb3OErqY85DtY7ck84wy31lANwqo7ke2j/h6rHGYZRPLl1g0gZhIgZvx0AOuguTy8nycYPhebxhYGVAVuvl/FVWwbw33eHJSSFPJV+gBI0DjBjVO0qxPPLRDYvTsVSswcaDIMnkK4IhULKIG8zaNzLwjUz6EXOh3xmhZs5E5yN/ZM/9wulUjulFUbgwlgNUXGGmeTh4IS3eiqLiPavvmLWu+3x10wMKRU0lTR3tT8JeHCnlICAiyzp8xLVR4GSldZSqrD6wwYVKrq42XNH7GPfIM1YBJXgeg3d8h0JV285CCrEde8RMPcbQWzVoQVRbJygBAPIZ4fq1CfVxLG4bzZP9ZMDolGc/0Qe60gB5SXTfAFycVS0mpbeYmxdGDwhxikL83oX6pgiMBlLtLn89N5iM4dhRxFLw8uEXt9uQc9nZ+3G73vQeWkTpnkYZkWeyy+3PbpWhl66h2HvA92OhcPzP9mRlIW0TLNCRaBvxMfBlBENsCgB7whEKSy2ou3tL17j6V0edK3AKrJW0VzpIFIJGQhyA21dABKULbhdx4oefF8c0pwQBAx2Jp5EGbI0kKUy+UCkr51zRTyAODEGrygjgFV8Fdw4YeFmRdNmMagN8TZsMyphr2tTwStQDh5sctdpv0L/D4U5IAkM7mytA5Wav2Nq5j3woMQoi9gtqRTQYUEVfDUme+YDuCKvGmrVi5OhlVNPuiTar8laXLZW3vFNzmrrjzrIng1ZmRC19VWdfJlwvfdendSKjVn547jF72od4ZrwysogSR09BrGbnYKLulAJSgpuPXv0JgwOSiChA4UBfS0TPBkiMDFi+NJqPA+TtgB5m3q47MP9KSU1GdasQni95gLlRgDx9OpXSmNfBsAe8ziN20/EA/UtwolE6f6Jd3hYIXfbCNsnrRJ6PzmM3YGDXSzQ+OKnihXHSQ940hK4ZH6VfUlz6dE/HzYNImJ7uOilYT9QgK+lLsrkejjyjRX0HYsUN2s2yqS+rR8FHcsP6+MAHMnVpLaK41/rViYrVHyShdUTUCN0XyxkRT1bxbdmCINbSJspzFqYEjUAo/ntsg7+yTw9MpOh6YYQQupBEvdsIeTMqada0o+KztBSPEzxKIH62D0sJAAUECvcYbQVDRf1Np30olrwDQbk3s1L1dB+yjYhWiuqYOZDZp1WTmFu1xiEKWpzouTSsCjb7MfQ5sgEExUqTbtPnnebDGlDmFc9iWbz5r2OUoisa5Hi8wvAsHwNTQQBBe0EX1Qztzvr1QCufwyKwjHJA9Hyr3kpKctpcPXXumNMVJVU9yYC+IwVVrPQH1C7qBH0UM5zm7gscAbZcOunvYAF6UxnKSiVxtHM7teKRdP0lFctOKTkPI+FPCblOj2SwtVEnNe5TwTyJUXA14A2oecgyGVrIGVnutKstUtJLRm0TnZSWTn8Ryeji4rT3MtRh5ndCqHozmvS323aO1QZnFERsYC9HWkRwTENDwhXOIFXH2TdFpHKhyjNKwLeFf1XzXf9Zj9Ae2bWhhwEBw0qNNkGGX6DrV3+FbmKrTco0ImFFCjAlNM7kmB2FXRjRY4MVP1ntacLJmknhLiiiGa9EbkuaKT8nMk4oB4zoOhUfXs3smgPTaXw1ZZ9Fx9FXt2mjVZEsJcjLI0fZV2MMgSsAHkacD7CihFiFBEMr7mYQH1OgLjFq3KEDcGR71ls1QIm/G00aSbFVGccOidCgjPUceVruQRfU/AzDZaHubwOHrJc50rEolCZhOMQ6KFQyc8WZYVc6tz07sSoGch3R7sHXRNcUv1ARgzNQA/1t0Be6MuFFFci4rv72DwWn4RSQZbBjGjOD0QUNYTNTntO6SroD3JgYRQXFLX+x4Y4lqiB33XKU/KNuHo7jy935WNJltmniy6QsU5W9WFovoMvLiHSbKYI31D25gE4AlBWg+YsFo+hej8LW6Mz7VkxTxDlcEq3qEwvdSmRNCTQ6/ihmfywAyOuNVAMDtn6j1I53LhAdYzpeTADx0z6L3lmxZVWnHReVnBtN3wPJfSjtcbdHGO0RXyvTzSipMpSV5XgSfMMhS5nqtiBrLVp0xhNQlgHrOBoOy3LL06lJ4Ao7ApMc/pBqfPXkyH8u7v+ZjzdlYn+TxSZ+JfbPSggG3AjWVdXurPG+ASVWoCAjjUxBFvcg8/yEluy1bVkWl52wYrQUNngPfACLn8dJaDQGFajCaxsOW7rmCHBN/tWDk23lT3++Ip95HQNe+y5HzKXt79nmbEi6WrzYwySY7CqnbxJsXlfG8pTzGTKR5hhNmYpZtHz5AS0q787daTMEB6mQzi6ZbSSXJMqIe8nsiJZOt5hzBC5NAIjU1u3fKEBXSndFHi3YwcflDFfdOUnh7pm6SgL3/2Hp64X/pTVj7HREiROq86Gqn13eANI7bjaNe9NFDTrm3MNtIVur1rqhtTneVjotMk3Gh7wnSVTYLbja9b0A7ViC1PNb0Kvag5p9VBn2Za0qX+9ghBJKxaobm7OhdllWh015Fp5Qnra4yoaEsWzEqR9kUNM7Hz27mK3JXefyxKmAHH346UPJePgrR75hg8nXOBUD5TIGiH6I1dCJc7PJ2krPjkdlgSXcLESuVCi7+Y/GY8FaiWDJyTrzTixlwhA4/PZ1hUr1mP+CAhAlmipVmKC5bE0kXcjkF5u3BvVnnYlEZQA7/X3cTMlY9EyRweeuu9TF+9QNc5hZwd+4si4tKVc70noKTtga7cF84KNQgUs0kidUieWKsM2vRC+UoQd4ozVwJ1QHR4koBXL9hptpI4yNgG6gmck5ahlDMFgZgOw5ynAxtezAy4TJ86U+QbYRTxScdR9J72oyafSdnjMjRfGCO8MZFxcPSJ28qdf4YXodoz20gRGoQZFH1qW3Lz1F6oWpiYx8Bf9zE7eHHlEKz56AhqSY5aQKmKGxub6xw5dWr8JeUQwIJvbTqO+GiVof5Ic8ieBfkjJUlzSVLpHgPupFxJmpECKyRc8BV4tTo6BjyWsGbLItkrT6vk014VW+F5PX8NuKg2OSjX1dIYy3xGqRITPMnQZWO6vOV02n48eyS4TC9bh12YdBfJIBmAA5ALPIw3Q/dqLaYbhjyy/Qo2zkZ7LmbJd7ewDiU+aO9i2OGCqUI6jZNKr8ub0Kgst3rYirNOE9cD9ZhUYXX0yk4jBL3sPAYVstQnXvmTdNEzZeJzUd/gWZOLNREKhfeWRKoVIrrH60SpwboLj0d275gKwYkGLYAZ2I1ykudRXlOC3Y8a8EPWdY+r3dnVJMl8bi+XjRjbqZ/Be9K2aazgpu/UN6OPehXxGHJNQ84ps5obuAMr39LtkXIZMWBZRuKcP3cVCNgXCGKo4sJ653d8GdAf1cDMh9VDRmtIYiwMhRQfQdksOBE7gyjgW3x+KV8kroiqg1odn34Yu5KsVHKlroIQb/dxEZeLh17uds6cQG1Q/MgiSsmt3Hus7nG5iUjFQ16oVnY22Fvw0pwKHi5v9IrtKDkIf5OO5fzD0XiL0itauiGekkc+djZNFN0TLtQTdmP11+w1M2hMCEQ3ki9/vNAZmvHjo7x7zA41cxQG2Vf2e+kydIPJjn2amB5n9QCvSReQMotI8FZgYWyr0ieo4YSC5CDmU+d9irzLTsZ8pOPgfIQd+ccLQ1wCH6yu1/VMWwhy2eiLsj68ZduSCQvKvMAly77F6RPkH0yOQ/kSEDob5f6TiXt9KpznjEt1BEKzXDjN079/LHh61BwHMkr/eStLTH0EaV2rN1AN6HFFtyIavUUdzT/zulX8+2YNnohJgXhMzxRLi9fE4mIGOldC1b0mzl9LQuL2fp6dOV21NaE09UlNT8I9QAmr4WgcD7Xb4RjCgy4XrEiG8BHM0kisiokjHSnUMD4A7FFeDtdtVi1fuNcVAbYn4mXRmAljVke668cjPrvGQdRBXXHsxpNIK8Pr7jugwF88CbJIz3RUQksel7OIoDcbzdX74wkWV7baPBmnlwva7VMk6wTL3QOaUPrV9GblCsYYLuqS6KaaO+11DxYyP3NUfJOpf175rOwIz8tJXs2JuxRuzEs0gtQfhx9RaqAgNE93iRu9D7mw9usNq8C/8kFElSnN4fzNXPxfGl6Am1XPx1lMPQgaegdYZX+Tb2+ueJ3bpld4PtljDnApDTKVbCDNQuiQUNSYC/sgPnmUhcJcdRhPxdgBzShdsaLNbMwZXx+NwyhsD0EdulOJ0x+7914ZxHs0TeZUtlQlbqynZSYJCtwOqGVV4P91yeDGuhBdQiajRt3tpbt48C7H5gwSR1oOSSvaQwtlp7R8swdbOMojTZiBLLlABmnGgWFHb3VPdSH2jFeDGrCqwWMVc5Ynxq5FOtGP413KG9NkYtS91q6iOLGd3joDqYbDnGX9jKypq74mLeKrb0jabZ0xdKONjngSDx0572Im5gfeyq3uwWqn0nDH27S+voR7NZFZAY5AdDJlPtbeB3hBRN3qZ0hZSy4SrMecxXeQu+rbPfjSkLhDjFtW1BBs7XOKjShYsj7jOPRp38X4jaC8Y+Cg8OJUj3SV+ZeSQjVUT004y7XGaAab1k5HB7A3UbIcouFCpPo8yGGBe2/e9rL78RAnhsb0Qt+6iBkxZlZi1ysdTUDyFP8oIvsUEr+mTPB9VCIWPTONKqUcMrvGtn6bU5rGyHAR4NljLDGZigaS4DiZ4yLfxzJumYHWs2DvlBctu2xL6jhTYiw8WWX/iClTLNwsM9kHkzl3wi217FnSykXPXz7+YgGRMfg3TAheeyBq2nRtm2MH2yGo0L+XixdqenFfhpueIbrQ6DtVvszbbH2ROZtMKUK5IPcGgE0R04qYKwL3nXuKINttHrSuGTwJq4LFa0bqSMD/iJ0yTJ/SXCebLom38kX//ZKyh+HGGTTlr3xxUUFzCcFaKwN0fMNJf49b+LEel2AQ7uJZxUWf8NsUfTzGlZB4qZJv4Xj6uZ0EBOTUVborUJYJY/FoXTqLtBMSruy9hHDmRUrlBOJkU2IAVaCO6BpQpWGFfUXfc6eZtPhIjvvDUrF5thmlfXBy7qSv0Jib3W6TNCpt2tLIs1mwFz97Juyj2/OF7zwnSXrp5+3jLXdYeA8J2bZR6/WQWw++ICe7PyY0diAzdPy9BwH3HlhVh9S78PaPyh2H9wX+D4Q1tVZVfIwUSHp8PFRRhYhmgDyBuF9Q/0pEAtadFsookNFZYYpgyCsAiPTEuyS11jSZVo0inoSFyDx/U0P8UQLs13gNcBVLrm6Iuqx9Q9mvle4r1XEUb9VYwMK0DtUgdUEiYJz3GoKlu0Bqyc0SSt6Ye7zBhD7N2gOyMuDpQrEs0/21BQGMvax+sWWDeVtLST7CrKMbyr/ycLtKBMtkivrxVvamZNz9Sr5ep2KDGk+oBfso2wrQ7lwrtUkESvMsXJedooTfkiSUl5qRiYWr3tT68WwGKCRSc5itNJzVJfJmpvljvPOoYl7TjXZRiY75IDR327GZyhBilZ7k2HiNXV7oHOjF1kTVYPd1Nr/YMhNfQWsBHHnNquqy2iN5+wFnUqSDWA3BUFP25cDQfdmguiLUoT2qQKjv1Q4a6kFIVl9ViWe/sYtTXOMBz7LhrmPkG2mi5b6TA7DV26kcTXkGl09I7xZX4l4Z1dwwFKWyhRHyHAOZuGgjHb2yPjhxp10ZC/ii/3UCHwtb4LIFoeNpjR6YmFlShRJ/vJug58UyKmvy0GK+sMpFubc9nigNrPkOW2eSLtxJgNBRic1FjmYzRBAl+fG09kiTQOnpldPNpfGK3SsdXXfHSuPGfi5P33zWxiWLqj7lPTjgpiMNVyRHGsIT/CkuKeRggbjMYwSW4/fRFllpGXiZKRlUd1f8ixNVoBGPOqt1EhOoDJJtRRiPg/6gViuBMYeJo0y2yv4LvFRCwJMapfNNuChHFaOhtKIjUJYqmsqgQ1iCjvfOLxoxBOVmpwaZmHDsmTkO+YqbfhrxYzi3pBm/k/MQ2iW7V1esFxuplt1Fq5EVROHJlo4kZMq8Y+pbsy1JMdKsbyAjIEqCMk3PNoft9Q4mHFwx0HsQJyzF8Ukg092aG4o4jiyCZ4KKNW5GGavyNVJ+oxKWgTJzeQoYqeykCYwiEnwySoSSimwhoeUDybMvhsUUF6Rn0lNL8StRxTukHdZ7Kc0ELeVyGKcuHSDr1Mqol71X9SVz9yvARlLOvldKQBb2wJm56Ok3rRaiyr5Zg4Ya2cmb6ej///aubLtRZNl+zX28XszDIyDmQYBACF7OQohJTBIIMXz9JVXlqmrJ1e3utqt9+9jl5TIyQpCxc8eQGRHQrtO30YHud3ppqwvjlDncT6udHsDESWkP1bXDDB9nM+AAAWjcopsj3X+5sclLCNViEgAN4rhDs8XQqnuK7sE+HHabQayYrXC9YvDtJnY4fdp5CSMtM5yOQqQHhiUDHK8+2F5AmF+RxWQOwSdQ7vmwn+sZphhsQ3euFq9iowCUzBtVAW8u21kNmkyzGzkQRS4TNV48dZHgETHU1X3e4BzwOWbjiOXsdi36jlEy1TaOLL5ZseAzNYVvKqwuiWIzEGq9OIrFHGyyEbvWcHK9rkNZGcxrv1Nzktq4AFotvMD2rI1ehFBdV0YXft2AgAqIt8uuYRYTQPAEO4XubOTkWFT0oBt6xtFbj688rZEroi9O6zANo6EJ4tUyb/aadT0AaoELt9lk5UUzsVCUC5SgYE/v3fXijXhDXOQrq2Dbtg3mnTPg2YCIlqCshUQwPesgHtOQ5piITeBk4NAvshHAll6BktYkhR60PjEjh3S+LDeygUS0UHshhFkAhhUJUfMeGqKYc7+ob/Ywz4uqcNbKuFmtkBpYsejZzr9m94fpBFHnaVOJRHOMLJsZy7E7hfQaE5XtYtelMqlgVtbS4o7XT6KqNOsLuOpMuoZF4LSp4ASgdCbBKnjgIrAcNeyFObw0RynHtLkkXP9wUgplbWBSqtaHA51kNEiiAJcJS4RagV/OtOfsF7OTcoh5v/ONiMXR3CyRI3O1ccFv2WKkMyogRLwcJXiVE4uhRshXfkpDhrcoPeT8oIO4PEhDKxxkXqFQELBlx24fT60mzC0HDytFwYJosX7qcFAHEWuaONMV3qpLZG95c3UF7n/FqOykKUIb4VLD1vRW1YbLpolzY8dgNNgwwTpaCgzL1Dj60N5V+Y7Bmf0exFJIUui39SFnd1preftsr3CTnQ/sLsvXq0vOu4DWNjlzbqrFUtqgeiFfKchmCWZ7AFuOAPmPVCra8yZkNvPqElMUCGcflwsbvZTE0iVghVZR1DBAG95nkzlR8cZrFkZBN77VOYpokyLiEmt7YNYHJXDSAQsPo8xNUxZO5HjzaMxbVhAkJb1KgO2tA+tf1SvMuXArw6dYb0sm7kUiKrGArR0XuLTEdIuiMHS2/SI6oENXV6lqTFSCNzXa6z6ur6rjLrvWfXYYJOUYiag49N5yJ8Etv1vmxW21NRRM8peZs99c6iw401wWeLxeJczKLU7QQWCG6BqIjFEcB+y0wyLitokHjMwX0Aglyl/pErUAqQtgpyQJ1IJk9945R6q6dDtljNKNbPXnYcZpBl44bxhq4EL4rUZyZdpMBjO7ttnjuS5z68t23ECu6IIuCWxqWVWtmoZ+tNbjdl58R2mWdHIvSbAz9xyLkZNxC3Zg5yqHsymYwIbYwNsTl21KhHh0XpUOtbJuqJ4P7Tk24oXCY7+HuSYFc0Wtm0ZT9La9mdQzsU3IvTFJTLCwXWEPqsPyoyT6bOUiGLmBpI6b1UkwyNWxwU2lDFDjkDJba4tK0wpcgjLMy+q2vX9CoL3Uxj0VXnnXQJrYRvWACph5m8EulC+O80ZTsKOjkIfNGSyGeaVCiGizampPN0DEOxk1nLQv6DglHF4xbeq5LuuxOklzCmuSkJ9Fl5XEcPN5tDF2YRkqxw+5p0j7Hqb0uAOOuJcN7FrfLe7SGtsl19s8XPe7reOFbtjPHDqfikl3dN1anJxK1IwgZ9DRH+ZN30MD4mtrjeglO5f9YsXP5BEo1A4xZteNmIT25CNCzqe0crCM2s24pVsaBG3P1rbTB4wyV3pySCHIWEM+W2/RWlc2YP0VyIyxA96gWOPYcI5pYSvEZmPn0Cu3sZNoNDWQC+YAjRJkax5vaM5aWwyXCYQB0qancQuqRrLFrRjNxqllnvV1JXEu1ipF292qUA10+Rax0aC3UCanOO7TQoYiJ8aHmJaIJEW++M1QBBhusa2U8n461K6V8alUk9jJ9XuzOEgsXZ0W3a61TmZ3602AKZ7WEcL+xNEx5pnMUWQGvDfL8XJFL8YlvoIZiXJHnDRIXKLhRKJ3FCfKhiIf1WlUmZlEeK4TgsZPKz3NrXi1J/gSEkY02PHQysVZm5wciGyGLa/na6Bts2JfHffSVhhSmQ3J88o20XL2myMjNTaDZMbaOaXCeZy5jRsLIRrzqmc1BKXsr25dLzqoq3O8AUr05J561jGEVivtyXb3c8Mr1aWRfTTm2BohMm6xBzurBcR6VekKTRLchQKIuF78pDdDfH8N1pKc4wRTODRsFv5qpR5Nu+jOGhHzFLdKR9i8Rvt0DdNeta+mmj/GottX6d6sigslnJXIVpw9hXf9LbeI12h1J509by6LoNcpLtL8gIJ0uj1c+V2/MzLe02yIv1XxmYe1Ktq6fTQ49uKTNr0AIYTDrl8lER8zhNhfU0Wp2NmJZiF1mqM17IzFIuNtfwtoqSO2JTq1wwGochqsH8KgixHVTHBB5V95dDtXR7M/dubOV2BX3bAktIYpp2b5LIH5UpBMIZjPbc+k81CVDiMLxBYdC2Y1o9245XxITyPdPqRVkltFoG1ZGwR8LN4y2GnVIE7BFaYzgPl9FM77ziTKNYjFXc26FuLRoAKR24zRbJmV4azXy5gWbrRyqapOAW8D3yns+Z3ehaiVn9wjz+3AHkP5UjNMlKt5MubOGJrcNsXAVMvgYh4TrwJhI045RzzczIzGXtzT4s1MmYBB3nbhTciiwMpGQTGRbUhVrQP3ZxPpbqROs80sjvFAykNEbeaevanWkr7UZi1bNZrM9LnVdlWnQXLAsMw5Ze3rSSbivpsrPG6AvxZlrZ/mejMvNhXl0LEp2yBa3x/RuBIpjLEBrTg2w9RxEDcT76KxcoVFk1EbdtuSoY1KNaCCIgJqvA6VlZT4sLgmUnqlz1E+cGdUyiejRCQJOOhCWF/JzRQi6ogKzkWMpROmXCze3+I8GxTofhE3K9BCbh6Lwq3W9AWHmCNrkaGgs6gra72P0IuXpo4G1O6BHQXGDw17it+e7GGQiYQZxARNq7hCUzsiMpE5hNl8vZaRx57zseUDej2kRYcHtVeNnDrlxUHPcZ5rp5Wgtq2n7eLrWWES3jv7uXpoYpdZG5O2hbMw3u3awMTXcrG+nsWjTbfGlcyVlF2BPVdCxjIZkkWpgimcM3ggDhSNxxTyDmbFjhGO9qfTsKpG9Eyc4JEU90Q/WuLyoAURbq5RmWFiPwSVvCMZURm7MaaaLXfYhwarr/yreR1KpGPXwKiymzkdYyYx8fpqFhsHRhWh2SDlddhlYwzMltA/euHEK1fHNtJ9O5xiEZKJXaXMfIbXBJaNFDOv7DPLh0pmjRjEx7u+tkOZbZR9exiDougrqQk0DoMpqZHT5X4aylcCN2DW5Q4PLTZXLQkW2R1cjqfJEtuKDRgRP8uOw8LSlmYICN7RDYHKSamLBKrqNMsS7tE6b5E4VKOoP5TGsMYIX2W6614uSuUkVMZKG7Z4zIz4lZa8wEGpXer5FudtGwGLGNExNlUUmddug6ke0UHNJddMExN9eIClYnJWNXAuGAdhtm47+MujS2EK+ddt4ERdyTPFDuPRge8MsiFN7yLPTda5lZ9SgbJevHaz9E5Vq0F7VecrWORYaGXqKFw6B+x6wGBRDRy6ZchjV8j7ieqVRiTk9iyyuo/IY73Yv8AKY/l56tXaKo6p5kUt6qw5ZDTaKmykYeN2onXoY0fl5VHYLOOz63OTm7aicYHG3qdguassxbQ45RK7i37iNP1oWIUs1jpvV6sLokXBHmKjtX0+qod1l16251A+OAdeIaA6TkKCEt3zyhW5+Lg77+0Gjqhj0KS15e5gZrJwupLOyrrcj2cY1O5hqVomM3vxqxcE4wWOWKW29hTmfDgbAMGj3ZShftCqXXqLvLiX8iQ4Sl9V4jo9BhmJEtNhKODAhd2ymSZBbgjIQOyyGpkRyTF+dC+5kqWQFSpGETaRKEJnOFhU5EXBfcaS4Un1tWSnYtyWkhPG28k401KW6xrlqSSXSVxRtNoaUOIFeZJFR0del3l5GHjF8Jt2lawGAQRHC2Yu4545OetUkI+3TSGrmeRo6+JZlr3x4D3AjyxpBnMxWO6k0ot/otGKQu+MLkMNWR3kfDRFpSIZBdqaYbnWmpPcDadNTms7uEh4wpo4M+428sjhi2Wsq3nAzWZ/Wbv4AGx3Ic1SRm5nSwpogpnPYivg1BAk/VAvKksQYNCi1Z8XJBcYg6TIQZa9Q8gsStRqzMbhjg5PVjymp9oY+JjJ+IHkA2N7VDz+YJrXC9Rr2o51PHbDBmdr8HB+ANGyrRQjqlTYh0nrgt1FqWcgV91tQO0a4KztKd5d0yXZphs8V9Znpuby82kfX/K4AWMnTgGW+Z0q+LpQ8Senu6z3UKjjzSx0lNL68J4Th/Wo+fuTYhqFY8Qby005mHBJJRgdiN107KnhgYUFfLcrmXWEuD8fRsdAQIErVkwjWFZG3Dp5Ic0EzizDaIJrInyaykhVQEBhpTSEfCmQ1tPhUA2UFRpq7uSclKxBmt7cLN7fwPPtGlU2Tteur5InbAvkqtyq+FCBpwbbhbDjkj4NwNuxvWGTragQQqANtYpl7bpCVSac2M2eO/JYWJqcWO1lathjAzvoOWvPvCilWq6BLd5IYmIheJyDiqhM7o47trsE0TpI81OV1OfS4bQ0wjeelCwuaJDsYmsr+LISGTkSL6YiHoQwmR+jjlEhCMqngPALa+OdlcTfUvudrMkKhjHoYJ4slRBdfYyIjR23FykiwjE46pUBSpxsNV8ND/qc06KEmVcZBPSiOaBPDA9f1HXp1b7tdRUcjlMDG0rUgZhqcDQWM0a8Nqw2oEaeUZS1zeIITc9TdUoMV0entZ90+ghCPGx6xWNOJeVbLQ/nHOf8DoPYxQoI+qy68Gi1OXkId/AN5tS2R06D2bM6uTCVcToudsFc22mgmxp1Lf1OTwt84C130zOVHUjhJV98IzHrmlOU94BKSkWp4X5DOkcNmxVDDod1aKe3kmR+mayqRmw6TvQtlhcXnNszSxCoIxz3lhMzAhUbHtPZ3nm1ixDVMJjgFtg5lZ0r3bw0XeyucmihO6SHSUzmAhNGOFXZFGK23iKt3wcXzbeOpe+T48VA8XznT2Yse5IrEMdaqw3MgPQVgm6AI1+Vs6QwcHPFM3tEPCimeGYdcyunplS+lsbDZSiTi1YnyVxAUjyriRXjbtEodH7mNR/JlPlMjjrI/hLbsxZdHTXa2hC82dpnZPDHxVwez6GKmWePR3fqZYI3IVWCybNuTTgVkXWt8doeeONjtk2jFKuRcyMxbmNB6zqsd9mVazja5w7qzh6gcyPAJ4P0pJXBIIFwNH1buXaKtM52Cn0uEzM6qqyUFSxpZIfyZl9vRe0UmyfVJxHDVlYCQsiaE3N4Zx+JOVnpUbQ5ulvi6DCDk18JTJuhXXY6Vhx8yQLbHc3YKtUing7RRqqz/MiBcWOPp5gAorYq+sgVjki0M5PhaeWlx9ydZkUlCIzLOGwxlRt7NlP4AIU7KyPkIeSpodwUlp7OHq+hbF9MGbQJCiULotNeWJw5FlHjBNZFRN0JG6guLeKAYWOpaoa6Ua9nmdSlQHc2OTdlVdUpwa7UY2t3ijObtc/jxkuuuhRR6YBTFpqFl/WmTAB3sKsjMtGBcMjs4KTykkMlVtnivNe7uAk8WfRkuNMEz6dcwfZGKW9BXpQgrCRsxc1r8aSqfXQuvQpegaIuJacf/V5OeKgiWxu6YrWaZfgBT9apI3urRTVQruEWroRijsae7AwbDkEqkiEqOzV0rN31gRKpU9me9LWzTpiZZY11Xsr4eIpQWZ+qCIW13OM8FYcr9+yU1vEsOkiojy26LTFkvvQX3lktmmGokHMnMkWXnROHOxCLD9M1naOZGJWlouGClX2BrVqTiYgY6Q5naEutbb+gDZBiGDZMHHJ0zmTsQbY002jtDYI0I72Iiu5suRu567V2qdQWzGspBlgTLUb3xT5l8ZgCIB9Z9SKGJ+ADTRnvOEkgmqES7VT+emUZW+43ujM1nX8UgRsWDjo3Xvw8oxt/X59RbruysdUpa3c8rPRXkJGEItfrnkpTwe7quWMuEhGygRgLbV3iOITo5x0ZEXvqOMeVcN5tkMKjpsXjk7CtckHraxzBm0i4eIHB2H1bC8Jam1loXjz0rI8wEJIxnSbmbTWI/TlF3HOWzu3aH+b52JG9dkh2J70YGxJWVG2Fg8cabP5gBW4dJHOVnnf8lTvDlV/meKkmGHqUwjDs4XQdOmbKCemWA8FBFTh87GqjH04dYWBzcIAWg1mkRg7oSa21kcSNoOuFrSMruuYlAhywAuFoDmyo71nZopEOY8CG0WGxyM5lS4yZCKtgDfeqeni4TYaEuuU6d3xElElri8RuPRQZWChs+wj46+sywBAroBSdqwfslnmd9zW9y/rEdEYwKTZHe/EhS83jymJv93vlGjUHcapP6nWnNUqE14ieyMxMulMAC3njnvjuVr8b3rg2u+WbdKMniKQcNWJ1LC6my8UrodI6rlp0iConh/yE7mWfEW92Whebo4nH8W1LCA7WgLLpOlwHllpMC3RDQ+Xt2qDytovzbaGkaQrKn4PvVxfUJ39Sc/7nbaceeyv+0vr6z105f+zJ4S0u760l5vIfP0ZZWKf/tm4cr5fT15L/+GN/2l/aI+65jfPvVfyP6wPTts0AxrYMuy6Pft6ZAgxWPOaXHRjIJxL/euh/PRX8vhq/DvLtYPp68NM+C1/6VfzOI3ztTnMJ2zT+PeE89xaJD2n8Fj0p2rhczMRr/Ju7fUlAXz/BbHLQYulb9wcKue/8id5J+cujf33fd0E/XuqhkcS3JqLPl/oyOg+XumHm26P/DRg99pb62zD664h4lvSPkCDy/j8nhF8dglM7bYq63MvScze8j4II/LlF2Lf21fRfRQR+13fwf7/18fkDRCwiCqcfTjuBE7rfu+mHT/pKVq+/t6+tzL9j8stNvClCn2X0/6PJXRlW+0P4n6SvvzYmevsmd3++4zz5PGTfJP0Mzx97b7+kvd6tU/Erms6/Ge38sYJBPhadUOg9ndxzwF9XMDCF/FIFgyMfSsEgr1Qw2MdCBIGSd2LEkac7Qb5exdx3Lodx+ulupr83Kl7Rr+qvW6/Uope+m6/LMfa7Fiw4MOM2Xx4KcPzrrdoXkfPsBfwh6zz37PooGHs0a9+Kc/C7K703uF7RfPPXUc6zmP/fUQ55r4QWtxD9oUs6SvxV/qHJJ/qeguAnHEIgmqBwcrFOYPju4u8NGeKPIXPfP+8HtLxsPL5Fg73HznkvN9dbjrPL5bSM1td0nOhQY0951NRJXh/i9ilqwArDrd0eIoDXwY69QxP11YLo7tZzV8CBevz24n8WyzX+z2l56Hb5mcb/ixCL/qSeQI++304Q4o3sVJy+hxwOPWu5HxubUo+ox97NUn1Fv0kwZZ87eoI2pE3a1GHJf3/1Lkj1/RytufU6Bxg6xpfL9HVEw/7SvBcf4dQreebPEcifdz+pO5MGIX/jTT6+A7k3i+/e8Vv/8zXvp36x1fOKmN0zy0R9W05su5ADUBovRj5/gMd3fL0ElsMPfXIXfNz65L7FfCXQO78SJl+ar/RLnuV7xUWJV4QLHij0Bta45a+AC//nm3t/uNkHD7GEPwpxl3d8/xDX/rtBiLuQAgTBFLV6KSIx9238BGj8nQibQMinux7jMEa+AAH023m/Ie13A8ErSPsTBG8EAhTFn7B7HniEAEw/kdgLRPA7iuXvYeCRbM2wPfcxUAnCMhzdlye/3ASwyBWBQJ4IdABqTMgWNfoAmY+3xPVmMsTg51n7TTnC6KMMl3n8kgzfbR6/YnXicx6/FQbgDziLyU91/usQgNDI00Og8SOA4BWrBZ8geEd1jj8uF/1yDCCfGPgHVcFHQMAr1gw+EfCequAjgOAVsf1PELyjKnjeEvBPYgD/xMA/qAo+AgJesVzziYD3VAUfAAT0pz3w60CAUcQTTv/w9eAmPu5b/+WAeLQNPiN+PxXoMqvvmB153h31D0b86E/d/gsnNYHfR30/xDz+1O6/EAMY+oR/RDL/XML7QNr9OU3knwTEKzZPfALiHTXDR8DA53rgP6wZPgAIYOhzTfADqQaE/ACI+Fwg/Gd1w4cAAfIJgn9WOXwIFDxGBQ/gQxEIxCyE2w53FH4AxscL9CBvJSkEfaLQO0kRj2qcQp4I9AVJvVekB4Ze4eb/Azvrv6eRPdc9+JJE9lwR4Y1zyH4yNq8tefB3s3yQhyTwV+6J/9P7/ZFnzH37JOIP0s0f7+2373iffPNvY/9v2aWPouTTfWYf9gJTfz/tF83/f1k2BIiof8hxfk2Jj38lz/5hyuUvY1kEeoLgxRh6/rrnXOR9OBdD7pd4vmZG/JxzkQfOJe8g+Ls5VsDaeKhpg9093k8yRd+MweFXBAje1vC/s8txCPx7B4fgxUInDxbovTfQJEkexU99F7fd7efbMB4BP92njMME9bxc+GP2D/YS433nwbfnPPgVAYFPBLwHAvBvsv5jABDvCIBXBAM+AfAeAEAo+PUIeE8K+FBVIOCXKs+8eCb6wcqCPBgEz3lhf7buw0P2NXF3oXfOvobhV+wteTNEfBDx4dS9TwJjiy0KYTRJQCSB4/hfE+a3smw/LVH17tJ8TfjoXybN+zKDyF8twQKStO95+0Ebv7sEP15pDRAeX2TfTruvf7wdfHFMv9REBcffXdPb0d+vhPrNXv1j7YC/EpB/lzfwR96gnsgfyv9Ab8Uc3+rmvrH3e6+4vtYr+tl93Z8OI785/52CjfDblvFFsB8BDD3ROP1bEGPLpPt9FN+O3i6Ujb62/BXyoZgWhM0eonnUX6Vb+L6w0YMR9e5c+7Z1fpHfRPD+RPzubwDptcz3qwDyQBgE/FfhQT1c6hcbU8grImb/KDw+jtSxJ+oHJfjM39+VzBP0426Zt4PE/QLBu0PiMYT2vJC+/EWAnn/coeS/bCEde8yD+uUL6c9z9Xcl9Tif/8sk9cLmlF8vqc/NKXe5hST8RCB3knoh++DXS+oxevjfzX4vSuqFggC/XlKPUb3/bvZ7UVLIe274Wg7bprn8aICARQe9OcTgjP8D -------------------------------------------------------------------------------- /Chapter05/Data-Engineering-Whiteboard-Completed-Notes.drawio: -------------------------------------------------------------------------------- 1 | 7VrfU9s4EP5rMgMPyThxEuAxBGh7VwpHmGGmLzeKrcQaHMuV5ITw19/uSv6R2CmBQoe2x0NwZGm12v1291tNWv548fBBsTS6lCGPWz0vfGj5Z61er9c9HsI/HFnbka43OLIjcyVCN1YOTMQjzye60UyEXG9MNFLGRqSbg4FMEh6YjTGmlFxtTpvJeHPXlM15bWASsLg+eidCE+XaeV754iMX88ht3c9fLFg+2Q3oiIVyVRnyz1v+WElp7NPiYcxjtF5uF7vuYsfbQjHFE7PPgqHI/k1752fh11StJ/dJPP30sd2zUpYsztyBnbJmnVtAySwJOQrxWv7pKhKGT1IW4NsVOB3GIrOI4VsXHutKOT2XXBn+UBlySn7gcsGNWsMU97Y99I7tGoeZtj887gzs0Kr0QT+HSFQ1/9Ewn8qc5+fFDqVx4MHZ5xm28mu2OmOGwcitYomeSQUuFzKpGRDObTatpI2S93wsY6lgJJEJzDydiTjeGmKxmCfwNQBrchg/RSsKAOfIvViIMMRtGt2y6biZTIwLr17/lTw18AfbnurV/NQbNvgpD4pXd1K//w4R3T3eC9F+/ycjuj/cBemxTHS24Er/QWjuev4eaPYKd/wUPB957xDPJ7lWFTzX0TzoNaL5rQxVL2UOyxOZqYD/SUg+6df804DkJv+8GY5PThpwPIzJAXCa4dzQ2e3IVG2PwJ71aemGP4ffMqRSZNC2JouOYEJ3kD6UL7dF4OSXS2m30F5O53xwBN6OOLy541NEH1fgOXj4LOfweSFipLOjysGmFaXYAhGSTDX+u1ByAUugonmF0BUJ1SRUd5r2n6y14bjuapUgOEfbYvfZ+guETixwwxs2FYlciiBq3M3FWGWvJ2VfMnXPjUjmnV2aFUs/JXOuiVTtJ3osM+D3PS/TqPrfIuFaaDTfnJOfjcSPnK7lA39Nrr4g5pOQTEv/6EUp4EIoHknNO5VnbGsk5hUEtQgd+fMOMg1nQ4ezxTRE48yyJMCXh8UmhQo63+qaqW8ZN9SiIJHsUBG2p1kpSCK4G2wcFMcIYs5wv0fMSb2xTRpG4EYcV00xI4QMPw/W8IdhC0e+CMPDzncDzcZEbTj9NcIPATlleu8gG2faSMt00IbXSoZZYNy3G24ylbgvVyqkaR65efLP52p0g45Q2bxzLAipEoSO85C8sXeUPqnr1Yrj1pfBOGZL/lrxOGExofiWs0UtJK+lgdARLI7XRVyNFuyRbHB2OcnRqHgaQwUknNJAMWni42u2KqDqiaQZ8DvT1K+NyDvOTEQgIefs55TzBwBSwug2xbpUE43BFLJkImbTGJ+Xgl5lUx0oke5E2w9m6brA5ybmUUXnmS1ro7tJbhIIlYcgYiCzhRcnuHEaA2fqNGVRTH2U+GLJwtI8InkKdquIoqcwX+dZCHs18qihlGOB8M8G5bdbCVY6a/d2EXgJWWYW0/VRBNwSDvIdHgmZ6JV45KC/xSNPvDrP7w0beH7e8b46kex6TR3Rb8kkixBEOKc8gCwstNm3sN2xkvLYvD1FZsK1ttECthZAiyilHCyFzkD6I8t5Cl6y5quXgmMYZWnbyHZoczzwUG1ZCYMstYbuB8UesLAdgBYM2yAsCjOuFItdAQ1klko7HvJFaoN5TMhhZsHICZFc2QMzCPalMOucyQCA5njHLAIK/cBq2qEiPVNcR3lhZnhq4EYa/0eQNKlyASUQxJdHFXUNVDwSDWmxrHBUCzFBsMyqN81MkZescm6ho3+JpK2gJOPOArpCEIJ7XaWUcciI0tWA00/Ouvp5+ec9A9UlcTLt/gC94RrgoIWrChKtEyiAgrUxMAqpkIg5zixgKuQXTU5yQUEcDGiEEtbgOfkrFq2K2uvKROA49ZRbfCJyuC0jAgFQ+hvWISBWbkHCuRVY9l8oDtx9D5nf4iGmFm8mLJ+qSm8A5W/n/kkgkCruD4Av1qSQMyIZFO5kQWBzFIVNWNB5ck3pz9y/G+6oOADhcGtdGGQKq2CFwUJ1VfeXnylltFPFF6LYwTpfViI3oewXxDILbSLhYR7CBBNms8OaPu9RvE7tyTL0V8hjAdqtC50DGQP/kCpvGC3y4VhCkYggEsRXAC4qsUBPXVfSRJKbO+iXNXZ/IsnpHm1fzh/XL8t8v+lq/uStbsu6XvdNSM4LU8Fw71RwQ4T7K4FgrxzwdMYuOktbBGy3WfB6DPM7S+zzvCuoN7FBqsR8bpv2pj6CZKUAOnsZ5Db8hkyIiEcQccjuhWrG7oN9hy51Gbt7mK+2y2i4hNm8fXm1y5c3d+bGyfYknTvzsTOcrbV0kUX52Blx8zLLYzND6+sXcK2CorEfcurWRV4gk5mYZ4oXd3/5pVuTevv4+HtcpGYQ1xJvtKu5DXaiclxILvR5AnHvDV+ZcmH9soSRmxEqBV4hQfuCwtAsPFEisMzrYCVM1JRX0vJm69Cuyil6pU8CAFPDwaAckVev667nohBMl63NlAWbKir0yP5Ir52hcvh/zX7qhwcne9Ts4qc4GzXbf37Nhq/lz3ToXeXXTv75fw== -------------------------------------------------------------------------------- /Chapter05/Data-Engineering-Whiteboard-Template.drawio: -------------------------------------------------------------------------------- 1 | 7VrbbuI6FP0aHoviOBd4LJeZqdQjHYmRRpoXZBKTeBrsyHEKnK8fh9iFxIFmdICGoSDReNtx7LX2rdvpwfFq85WjNP6HhTjp2Va46cFJz7ZtMPDkn0KyLSXAcv1SEnESKtleMCP/YT1QSXMS4qwyUDCWCJJWhQGjFAeiIkOcs3V12JIl1aemKMKGYBagxJT+IKGI9eosa9/xDZMoVo92dMcK6cFKkMUoZOsDEZz24JgzJsqr1WaMkwI9jUt535cjvW8L45iKNjd4JJ+n9nQS/kz5dvZCk8XTtwe7nOUVJbnasFqs2GoEOMtpiItJrB4crWMi8CxFQdG7lqRLWSxWiWwBeWkuSq3zFXOBNwcitcivmK2w4Fs5RPU+eK5CTOnMA3SHfbcUrfccOEM1LK7ArxQOKd6jt/n30MgLhc4fIAUNpCZIICn5zhHNloxLwgmjBnxy16KKUSY4e8FjljAuJZRROXK0JElSE6GERFQ2A4kllvJRgSGRqvmoOlYkDIvHNJJSpW3JqFDGZTvn4smu8wShwZLtNbCkbzw7Sc776sxykRAq4df+QsOj0e/ZUH6/FE8dRRyFBO/7Gsgqhvs+AN4xZkOUxW88aAqf0QIn/7KM7HQGThZMCLZq4FiwtEkVDtTpgFpg67bab/FIlKXlRpdkU6xjlDJSzDJ9lZNlWidjlBY3rDZR4cj7aJ05/UUevGAxXxMRz9nil5wlu5TqABv0wfDwY2iS65qK5Pp9+0Kq5L6vSjiUoUI1GRcxixhFyXQvrRnhfswzK2jdIf8LC7FV/KFcsOPkeqfAz1jOA3xiP8ovCsQjLE6MA8qGi82d5JLjRHq812qcPDsNnkHDM6IhoZEU/iwMrPP+9jRvf2I0wNZhUJuNMzDD4jX9rf8h/tZ9hNbI/fv8bSRRnIcyq5gH8idh0ZkUx3adWqBu0BvX7cOhqTrepVRncCyfGqu9349hQ2nYHqxRZH+waQ8/U6mbSKUcr/OplP5f/q/JpYCuRLybTPmdSqb0ug+IGCcY0fvLpRy3c7kUaFGLuS0rgW2tBHTLSsxSj0GEjmYTwqUvLwMPLdAuQP8/YbFIby13INMBIzaCqe3ZU8MIZc9y9+ligGwMhhyXuvMUFOsZyWZ5VcuGMcUcJWcyeNceVs29Kb9qyH+17Pxa9lmruo0EC1rdT7DMYtU450hICO8ustvD7kV2s4h145HdbxvZ3W5F9hb1qs/IfmORHfo1//zRkd1pkT9e/VAVOF4VJehZDYeqoMkxOpZ/KahaJEHXhmpQDx/Nx89HkPL00PNjZQZ5VTF9DAKcZd2P8Wc7eQYOqFHUcPIM4FVPns0Yr+vZjGb5CvM7Isg1baiBoOueVLWoQV7dKQ/roUu6mtaO5lJAmTXCjwdqADoIlFmvUxY/2+VAd2TvD8M2rwJd197NJOyJRjgr39GyvjOW3BNBfv2U6IIEyeb+rcZd38HLoXD6Gw== -------------------------------------------------------------------------------- /Chapter05/README.md: -------------------------------------------------------------------------------- 1 | # Chapter 5 - Architecting Data Engineering Pipelines 2 | 3 | In this chapter, we reviewed an approach to developing data engineering pipelines by 4 | identifying a limited-scope project, and then whiteboarding a high-level architecture 5 | diagram. We looked at how we could have a workshop, in conjunction with relevant 6 | stakeholders in the organization, to discuss requirements and plan the initial architecture. 7 | 8 | ## Hands-on Activity 9 | In the ***hands-on activity*** section of this chapter, we read through some fictional notes of a meeting to discuss a new project that had specific data requirements. As we read through the notes, we sketched out a high-level whiteboard architecture showing data consumers, data ingestion sources, and transformations. 10 | 11 | - Link to diagrams.net - an online architecture design tool: https://www.diagrams.net/. 12 | 13 | **NOTE:** The files linked to below can be downloaded from here (in .drawio format) and then opened in diagrams.net and modified. 14 | 15 | - Generic Data Architecture Whiteboard Template (drawio format): [Data-Engineering-Whiteboard-Template.drawio](Data-Engineering-Whiteboard-Template.drawio) 16 | 17 | - Completed Data Architecture Whiteboard Diagram (drawio format): [Data-Engineering-Completed-Whiteboard.drawio](Data-Engineering-Completed-Whiteboard.drawio) 18 | 19 | - Completed Data Architecture Whiteboard Notes (drawio format): [Data-Engineering-Whiteboard-Completed-Notes.drawio](Data-Engineering-Whiteboard-Completed-Notes.drawio) 20 | 21 | 22 | -------------------------------------------------------------------------------- /Chapter06/README.md: -------------------------------------------------------------------------------- 1 | # Chapter 6 - Ingesting Batch and Streaming Data 2 | 3 | In this chapter, we discussed the 5 V's of data (volume, velocity, variety, validity, and value). 4 | We then reviewed a few different approaches for ignesting data from databases, and from streaming data sources. 5 | 6 | ## Hands-on Activity 7 | In the ***hands-on activity*** section of this chapter, we firstly used Amazon Database Migration Service (DMS) to ingest data from a MySQL database, and then we configured Amazon Kinesis Data Firehose to ingest streaming data that we generated using Amazon Kinesis Data Generator (KDG). 8 | 9 | ### Ingesting data with Amazon DMS 10 | 11 | #### Creating a MySQL database instance 12 | - AWS Management Console - RDS: https://console.aws.amazon.com/rds/home 13 | 14 | #### Loading the demo database using an Amazon EC2 instance 15 | 16 | - User data that is specified for the EC2 instance, to load data to the RDS database. **Change** the *HOST* and *PASSWORD* below to match your RDS instance host name and the password you set when creating the RDS instance. 17 | 18 | ``` 19 | #!/bin/bash 20 | yum install -y mariadb 21 | curl https://downloads.mysql.com/docs/sakila-db.zip -o sakila.zip 22 | unzip sakila.zip 23 | cd sakila-db 24 | mysql --host=HOST --user=admin --password=PASSWORD -f < sakila-schema.sql 25 | mysql --host=HOST --user=admin --password=PASSWORD -f < sakila-data.sql 26 | ``` 27 | 28 | #### Creating an IAM policy and role for DMS 29 | 30 | - AWS Management Console - IAM Policies: https://console.aws.amazon.com/iamv2/home?#/policies 31 | 32 | - AWS IAM policy for `DataEngDMSLandingS3BucketPolicy`. **Change** INITIALS in the policy below to match the name of the landing zone bucket that you previously created. 33 | ``` 34 | { 35 | "Version": "2012-10-17", 36 | "Statement": [ 37 | { 38 | "Effect": "Allow", 39 | "Action": [ 40 | "s3:*" 41 | ], 42 | "Resource": [ 43 | "arn:aws:s3:::dataeng-landing-zone-INITIALS", 44 | "arn:aws:s3:::dataeng-landing-zone-INITIALS/*" 45 | ] 46 | } 47 | ] 48 | } 49 | ``` 50 | 51 | #### Querying data with Amazon Athena 52 | - Athena query to validate that data has been successfully ingested using DMS 53 | `select * from film limit 20;` 54 | 55 | - Further reading: [Using Amazon S3 as a target for AWS Database Migration Service](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.S3.html) 56 | 57 | - Further reading: [Using a MySQL-compatible database as a source for AWS DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.MySQL.html) 58 | 59 | ### Ingesting Streaming Data 60 | 61 | #### Configuring Kinesis Data Firehose for streaming delivery to Amazon S3 62 | - *S3 Bucket Prefix* for Kinesis Data Firehose: `streaming/!{timestamp:yyyy/MM/}` 63 | - *S3 Bucket Error Output Prefix* for Kinesis Data Firehose: `!{firehose:error-output-type}/!{timestamp:yyyy/MM/}` 64 | 65 | #### Configuring Amazon Kinesis Data Generator (KDG) 66 | 67 | - Record template for Kinesis Data Generator 68 | ``` 69 | { 70 | "timestamp":"{{date.now}}", 71 | "eventType":"{{random.weightedArrayElement( 72 | { 73 | "weights": [0.3,0.1,0.6], 74 | "data": ["rent","buy","trailer"] 75 | } 76 | )}}", 77 | "film_id":{{random.number( 78 | { 79 | "min":1, 80 | "max":1000 81 | } 82 | )}}, 83 | "distributor":"{{random.arrayElement( 84 | ["amazon prime", "google play", "apple itunes","vudo", "fandango now", "microsoft", "youtube"] 85 | )}}", 86 | "platform":"{{random.arrayElement( 87 | ["ios", "android", "xbox", "playstation", "smarttv", "other"] 88 | )}}", 89 | "state":"{{address.state}}" 90 | } 91 | ``` 92 | 93 | #### Querying data with Amazon Athena 94 | - Athena query to validate that data has been successfully ingested using DMS 95 | `select * from streaming limit 20;` 96 | 97 | 98 | 99 | 100 | -------------------------------------------------------------------------------- /Chapter07/README.md: -------------------------------------------------------------------------------- 1 | # Chapter 7 - Transforming Data to Optimize for Analytics 2 | 3 | In this chapter, we reviewed a number of common transformations that can be applied 4 | to raw datasets, covering both generic transformations used to optimize data for analytics, 5 | and business transforms used to enrich and denormalize datasets. 6 | 7 | ## Hands-on Activity 8 | In the ***hands-on activity*** section of this chapter, we joined various datasets that we had previously ingested in order to denormalize the underlying datasets. We then joined data we had ingested from a database with data that had been streamed into the data lake. 9 | 10 | #### Creating a new data lake zone – the curated zone 11 | - AWS Management Console - S3: - AWS Management Console - S3: https://s3.console.aws.amazon.com/s3/home 12 | 13 | #### Creating a new IAM role for the Glue job 14 | - AWS Management Console - IAM Policies: https://console.aws.amazon.com/iamv2/home?#/policies 15 | 16 | - AWS IAM policy for `DataEngGlueCWS3CuratedZoneWrite`. Change *INITIALS* in the policy below to match the name of the relevant bucket that you previously created. 17 | 18 | ``` 19 | { 20 | "Version": "2012-10-17", 21 | "Statement": [ 22 | { 23 | "Effect": "Allow", 24 | "Action": [ 25 | "s3:GetObject" 26 | ], 27 | "Resource": [ 28 | "arn:aws:s3:::dataeng-landing-zone-INITIALS/*", 29 | "arn:aws:s3:::dataeng-clean-zone-INITIALS/*" 30 | ] 31 | }, 32 | { 33 | "Effect": "Allow", 34 | "Action": [ 35 | "s3:*" 36 | ], 37 | "Resource": "arn:aws:s3:::dataeng-curated-zone-INITIALS/*" 38 | } 39 | ] 40 | } 41 | ``` 42 | 43 | 44 | -------------------------------------------------------------------------------- /Chapter08/README.md: -------------------------------------------------------------------------------- 1 | # Chapter 8 - Identifying and Enabling Data Consumers 2 | 3 | In this chapter, we explored a variety of data consumers that you are likely to find in most 4 | organizations, including business users, data analysts, and data scientists. We briefly 5 | examined their roles, and then looked at the types of AWS services that each of them is 6 | likely to use to work with data. 7 | 8 | ## Hands-on Activity 9 | In the hands-on section of this chapter, we took on the role of a data analyst, tasked 10 | with creating a mailing list for the marketing department. We used data that had been 11 | imported from a MySQL database into S3 in a previous chapter, joined two of the tables 12 | from that database, and transformed the data in some of the columns. Then, we wrote the 13 | newly transformed dataset out to Amazon S3 as a CSV file. 14 | 15 | #### Configuring new datasets for AWS Glue DataBrew 16 | - AWS Management Console - Glue DataBrew: https://console.aws.amazon.com/databrew 17 | 18 | 19 | -------------------------------------------------------------------------------- /Chapter09/README.md: -------------------------------------------------------------------------------- 1 | # Chapter 9 - Loading Data into a Data Mart 2 | 3 | In this chapter, we learned how a cloud data warehouse can be used to store hot data to 4 | optimize performance and manage costs. We reviewed some common "anti-patterns" 5 | for data warehouse usage before diving deep into the Redshift architecture to learn more 6 | about how Redshift optimizes data storage across nodes. 7 | We then reviewed some of the important design decisions that need to be made when 8 | creating an optimized schema in Redshift, before reviewing ingested unloaded from 9 | Redshift. 10 | 11 | ## Hands-on Activity 12 | In the hands-on section of this chapter we created a new Redshift cluster, 13 | configured Redshift Spectrum to query data from Amazon S3, and then loaded a 14 | subset of data from S3 into Redshift. We then ran some complex queries to calculate the 15 | distance between two points before creating a materialized view with the results of our 16 | complex query. 17 | 18 | #### Uploading our sample data to Amazon S3 19 | In this exercise, we use open-source data from an organization called **Inside Airbnb** that provides data that quantifies the impact of short-term rentals on housing and residential communities. To learn more about the organization, see http://insideairbnb.com/index.html. 20 | 21 | The following links are to download data from [*Inside Airbnb*](http://insideairbnb.com/index.html), who have licensed this data under [Creative Commons Attribution 4.0 International License](http://creativecommons.org/licenses/by/4.0/). 22 | 23 | Download the ***listings.csv*** file for Jersey City, New Jersey, and for New York City, New York, using the following links. *Make sure to download the CSV version of the file, and not the csv.gz version*. Rename each file when downloading so you can identify which city it is for (such as jc-listings.csv and ny-listings.csv). 24 | 25 | - Jersey City Listings: [Access here](http://insideairbnb.com/get-the-data.html#:~:text=Jersey%20City%2C%20New%20Jersey%2C%20United%20States) 26 | - New York Listings: [Access here](http://insideairbnb.com/get-the-data.html#:~:text=New%20York%20City%2C%20New%20York%2C%20United%20States) 27 | 28 | **Commands to upload files to Amazon S3** 29 | 30 | - Use the following commands to upload the data to your S3 Landing Zone bucket 31 | ``` 32 | aws s3 cp jc-listings.csv s3://dataeng-landing-zone-INITIALS/listings/city=jersey_city/jc-listings.csv 33 | ``` 34 | 35 | ``` 36 | aws s3 cp ny-listings.csv s3://dataeng-landing-zone-INITIALS/listings/city=new_york_city/ny-listings.csv 37 | ``` 38 | 39 | #### IAM Roles for Redshift 40 | 41 | - AWS Management Console - IAM Roles: https://console.aws.amazon.com/iamv2/home?#/roles 42 | 43 | #### Creating a Redshift cluster 44 | 45 | - AWS Management Console - Redshift: https://console.aws.amazon.com/redshiftv2/ 46 | 47 | #### Creating eternal tables for querying data in S3 48 | 49 | - The following query can be run in the Redshift Query Editor to create an external schema. Make sure to specify the ARN for the new role you created in place of the ***iam_role*** listed below. 50 | 51 | ``` 52 | create external schema spectrum_schema 53 | from data catalog 54 | database 'accommodation' 55 | iam_role 'arn:aws:iam::1234567890:role/AmazonRedshiftSpectrumRole' 56 | create external database if not exists; 57 | ``` 58 | 59 | - The following query can be run to create a new external table. Make sure to replace ***INITIALS*** in the query below with the correct identifier for your Landing Zone bucket. 60 | 61 | ``` 62 | CREATE EXTERNAL TABLE spectrum_schema.listings( 63 | listing_id INTEGER, 64 | name VARCHAR(100), 65 | host_id INT, 66 | host_name VARCHAR(100), 67 | neighbourhood_group VARCHAR(100), 68 | neighbourhood VARCHAR(100), 69 | latitude Decimal(8,6), 70 | longitudes Decimal(9,6), 71 | room_type VARCHAR(100), 72 | price SMALLINT, 73 | minimum_nights SMALLINT, 74 | number_of_reviews SMALLINT, 75 | last_review DATE, 76 | reviews_per_month NUMERIC(8,2), 77 | calculated_host_listings_count SMALLINT, 78 | availability_365 SMALLINT) 79 | partitioned by(city varchar(100)) 80 | row format delimited 81 | fields terminated by ',' 82 | stored as textfile 83 | location 's3://dataeng-landing-zone-INITIALS/listings/'; 84 | ``` 85 | 86 | - The following two queries create partitions for our Jersey City and New York City data. *Make sure to run each of these queries separately, and to replace **INITIALS** with the correct identifier for your Landing Zone bucket.* 87 | 88 | **Query 1:** 89 | ``` 90 | alter table spectrum_schema.listings add 91 | partition(city='jersey_city') 92 | location 's3://dataeng-landing-zone-INITIALS/listings/city=jersey_city/' 93 | ``` 94 | **Query 2:** 95 | ``` 96 | alter table spectrum_schema.listings add 97 | partition(city='new_york_city') 98 | location 's3://dataeng-landing-zone-INITIALS/listings/city=new_york_city/' 99 | ``` 100 | 101 | - Validate that the data has been loaded and defined correctly by running queries using both Redshift and Amazon Athena. 102 | 103 | **Redshift Query:** 104 | ``` 105 | select * from spectrum_schema.listings limit 100; 106 | ``` 107 | 108 | **Amazon Athena Query:** 109 | ``` 110 | select * from accommodation.listings limit 100; 111 | ``` 112 | 113 | #### Creating a schema for a local Redshift table 114 | 115 | - Create new local (not external) Redshift schema 116 | 117 | ``` 118 | create schema if not exists accommodation_local; 119 | ``` 120 | 121 | - Create new local listings table 122 | 123 | ``` 124 | CREATE TABLE dev.accommodation_local.listings( 125 | listing_id INTEGER, 126 | name VARCHAR(100), 127 | neighbourhood_group VARCHAR(100), 128 | neighbourhood VARCHAR(100), 129 | latitude Decimal(8,6), 130 | longitudes Decimal(9,6), 131 | room_type VARCHAR(100), 132 | price SMALLINT, 133 | minimum_nights SMALLINT, 134 | city VARCHAR(40)) 135 | distkey(listing_id) 136 | sortkey(price); 137 | ``` 138 | 139 | - Load data from our Redshift Spectrum (external) table into the new local table 140 | 141 | ``` 142 | INSERT into accommodation_local.listings 143 | (SELECT listing_id, 144 | name, 145 | neighbourhood_group, 146 | neighbourhood, 147 | latitude, 148 | longitudes, 149 | room_type, 150 | price, 151 | minimum_nights 152 | FROM spectrum_schema.listings); 153 | ``` 154 | 155 | #### Running complex SQL queries against our data 156 | In this section we create a complex query, but do so in steps in order to better understand how the query works. 157 | 158 | **Query 1** 159 | ``` 160 | WITH touristspots_raw(name,lon,lat) AS ( 161 | (SELECT 'Freedom Tower', -74.013382,40.712742) UNION 162 | (SELECT 'Empire State Building', -73.985428, 40.748817)), 163 | touristspots (name,location) AS (SELECT name, 164 | ST_Point(lon, lat) FROM touristspots_raw) 165 | select name, location from touristspots 166 | ``` 167 | 168 | **Query 2** 169 | ``` 170 | WITH accommodation(listing_id, name, room_type, location) AS (SELECT listing_id, name, room_type, ST_Point(longitudes, latitude) from accommodation_local.listings) 171 | select listing_id, name, room_type, location from accommodation 172 | ``` 173 | 174 | **Query 3** 175 | ``` 176 | WITH touristspots_raw(name,lon,lat) AS ( 177 | (SELECT 'Freedom Tower', -74.013382,40.712742) UNION 178 | (SELECT 'Empire State Building', -73.985428, 40.748817) 179 | ), 180 | touristspots(name,location) AS ( 181 | SELECT name, ST_Point(lon, lat) 182 | FROM touristspots_raw), 183 | accommodation(listing_id, name, room_type, price, 184 | location) AS 185 | ( 186 | SELECT listing_id, name, room_type, price, 187 | ST_Point(longitudes, latitude) 188 | FROM accommodation_local.listings) 189 | SELECT 190 | touristspots.name as tourist_spot, 191 | accommodation.listing_id as listing_id, 192 | accommodation.name as location_name, 193 | (ST_DistanceSphere(touristspots.location, 194 | accommodation.location) / 1000)::decimal(10,2) AS 195 | distance_in_km, 196 | accommodation.price AS price, 197 | accommodation.room_type as room_type 198 | FROM touristspots, accommodation 199 | WHERE tourist_spot like 'Empire%' 200 | ORDER BY distance_in_km 201 | LIMIT 100; 202 | ``` 203 | 204 | **Query 4** 205 | ``` 206 | CREATE MATERIALIZED VIEW listings_touristspot_distance_view AS 207 | WITH touristspots_raw(name, lon, lat) AS ( 208 | (SELECT 'Freedom Tower', -74.013382,40.712742) UNION 209 | (SELECT 'Empire State Building', -73.985428, 40.748817) 210 | ), 211 | touristspots(name,location) AS ( 212 | SELECT name, ST_Point(lon, lat) 213 | FROM touristspots_raw), 214 | accommodation(listing_id, name, room_type, price,location) AS 215 | ( 216 | SELECT listing_id, name, room_type, price, ST_Point(longitudes, latitude) 217 | FROM accommodation_local.listings) 218 | SELECT 219 | touristspots.name as tourist_spot, 220 | accommodation.listing_id as listing_id, 221 | accommodation.name as location_name, 222 | (ST_DistanceSphere(touristspots.location,accommodation.location) / 1000)::decimal(10,2) AS distance_in_km, 223 | accommodation.price AS price, 224 | accommodation.room_type as room_type 225 | FROM touristspots, accommodation 226 | ``` 227 | 228 | **Query 5** 229 | ``` 230 | select * from listings_touristspot_distance_view 231 | where tourist_spot like 'Empire%' 232 | order by distance_in_km 233 | limit 100 234 | ``` 235 | 236 | 237 | 238 | -------------------------------------------------------------------------------- /Chapter10/ProcessFileStateMachine.json: -------------------------------------------------------------------------------- 1 | { 2 | "Comment": "A description of my state machine", 3 | "StartAt": "Check File Extension", 4 | "States": { 5 | "Check File Extension": { 6 | "Type": "Task", 7 | "Resource": "arn:aws:states:::lambda:invoke", 8 | "OutputPath": "$.Payload", 9 | "Parameters": { 10 | "Payload.$": "$", 11 | "FunctionName": "arn:aws:lambda:us-east-2:123456789:function:dataeng-check-file-ext:$LATEST" 12 | }, 13 | "Retry": [ 14 | { 15 | "ErrorEquals": [ 16 | "Lambda.ServiceException", 17 | "Lambda.AWSLambdaException", 18 | "Lambda.SdkClientException" 19 | ], 20 | "IntervalSeconds": 2, 21 | "MaxAttempts": 6, 22 | "BackoffRate": 2 23 | } 24 | ], 25 | "Next": "Choice" 26 | }, 27 | "Choice": { 28 | "Type": "Choice", 29 | "Choices": [ 30 | { 31 | "Variable": "$.file_extension", 32 | "StringMatches": ".csv", 33 | "Next": "Process CSV" 34 | } 35 | ], 36 | "Default": "Pass - Invalid File Ext" 37 | }, 38 | "Process CSV": { 39 | "Type": "Task", 40 | "Resource": "arn:aws:states:::lambda:invoke", 41 | "OutputPath": "$.Payload", 42 | "Parameters": { 43 | "Payload.$": "$", 44 | "FunctionName": "arn:aws:lambda:us-east-2:123456789:function:dataeng-random-failure-generator:$LATEST" 45 | }, 46 | "Retry": [ 47 | { 48 | "ErrorEquals": [ 49 | "Lambda.ServiceException", 50 | "Lambda.AWSLambdaException", 51 | "Lambda.SdkClientException" 52 | ], 53 | "IntervalSeconds": 2, 54 | "MaxAttempts": 6, 55 | "BackoffRate": 2 56 | } 57 | ], 58 | "Catch": [ 59 | { 60 | "ErrorEquals": [ 61 | "States.ALL" 62 | ], 63 | "Next": "SNS Publish", 64 | "ResultPath": "$.Payload" 65 | } 66 | ], 67 | "Next": "Success" 68 | }, 69 | "Success": { 70 | "Type": "Succeed" 71 | }, 72 | "Pass - Invalid File Ext": { 73 | "Type": "Pass", 74 | "Result": { 75 | "Error": "InvalidFileFormat" 76 | }, 77 | "Next": "SNS Publish", 78 | "ResultPath": "$.Payload" 79 | }, 80 | "SNS Publish": { 81 | "Type": "Task", 82 | "Resource": "arn:aws:states:::sns:publish", 83 | "Parameters": { 84 | "TopicArn": "arn:aws:sns:us-east-2:123456789:dataeng-failure-notification", 85 | "Message.$": "$" 86 | }, 87 | "Next": "Fail" 88 | }, 89 | "Fail": { 90 | "Type": "Fail" 91 | } 92 | } 93 | } 94 | -------------------------------------------------------------------------------- /Chapter10/README.md: -------------------------------------------------------------------------------- 1 | # Chapter 10 - Orchestrating the Data Pipeline 2 | 3 | In this chapter, we looked at a critical part of a data engineers' job: designing and 4 | orchestrating data pipelines. First, we examined some of the core concepts around data 5 | pipelines, such as scheduled and event-based pipelines, and how to handle failures and 6 | retries. 7 | We then looked at four different AWS services that can be used for creating and 8 | orchestrating data pipelines. This included Amazon Data Pipeline, AWS Glue Workflows, 9 | Amazon Managed Workflows for Apache Airflow (MWAA), and AWS Step functions. 10 | We discussed some of the use cases for each of these services, as well as the pros and cons 11 | of them. 12 | 13 | ## Hands-on Activity 14 | In the hands-on section of this chapter, we built an event-driven pipeline. We used 15 | two AWS Lambda functions for processing and an Amazon SNS topic for sending out 16 | notifications about failure. Then, we put these pieces of our data pipeline together into 17 | a state machine orchestrated by AWS Step functions 18 | 19 | #### Lambda function to determine the file extension 20 | 21 | - AWS Management Console - Lambda Functions: https://console.aws.amazon.com/lambda/home 22 | 23 | - Code for Lambda function to check file extension: [dataeng-check-file-ext.py](dataeng-check-file-ext.py) 24 | 25 | #### Lambda function to randomly generate failures 26 | 27 | - Code for Lambda function to generate random failures: [dataeng-random-failure-generator.py](dataeng-random-failure-generator.py) 28 | 29 | #### Creating an SNS topic and subscribing to an email address 30 | 31 | - AWS Management Console - SNS: https://us-east-2.console.aws.amazon.com/sns/v3/home 32 | 33 | #### Creating a new Step function state machine 34 | 35 | - AWS Management Console - Step Functions: https://console.aws.amazon.com/states/home 36 | 37 | - Example of Step Function JSON for completed state machine: [ProcessFileStateMachine.json](ProcessFileStateMachine.json) 38 | [Note that the ARN references in this state machine are not valid, and would need to be updated to reflect your AWS account number] 39 | 40 | #### Configuring Amazon S3 data events 41 | 42 | - AWS Management Console - CloudTrail: https://console.aws.amazon.com/cloudtrail/home 43 | 44 | #### Create an EventBridge rule for triggering our Step functions state machine 45 | 46 | - AWS Management Console - EventBridge: https://console.aws.amazon.com/events/home 47 | 48 | #### Testing out event-driven data orchestration pipeline 49 | 50 | - AWS Management Console - Amazon S3: https://s3.console.aws.amazon.com/s3 51 | 52 | -------------------------------------------------------------------------------- /Chapter10/dataeng-check-file-ext.py: -------------------------------------------------------------------------------- 1 | import urllib.parse 2 | import os 3 | import json 4 | print('Loading function') 5 | 6 | def lambda_handler(event, context): 7 | print("Received event: " + json.dumps(event, indent=2)) 8 | # Get the object from the event and show its content type 9 | bucket = event['detail']['requestParameters']['bucketName'] 10 | key = urllib.parse.unquote_plus(event['detail']['requestParameters']['key'], encoding='utf-8') 11 | filename, file_extension = os.path.splitext(key) 12 | print(f'File extension is: {file_extension}') 13 | payload = { 14 | "file_extension": file_extension, 15 | "bucket": bucket, 16 | "key": key 17 | } 18 | return payload 19 | -------------------------------------------------------------------------------- /Chapter10/dataeng-random-failure-generator.py: -------------------------------------------------------------------------------- 1 | from random import randint 2 | 3 | def lambda_handler(event, context): 4 | print('Processing') 5 | #Our ETL code would go here 6 | value = randint(0, 2) 7 | # We now divide 10 by our random number. 8 | # If the random numebr is 0, our function will fail 9 | newval = 10 / value 10 | print(f'New Value is: {newval}') 11 | return(newval) 12 | -------------------------------------------------------------------------------- /Chapter11/README.md: -------------------------------------------------------------------------------- 1 | # Chapter 11 - Ad-hoc queries with Amazon Athena 2 | 3 | Amazon Athena is a serverless, fully managed service that lets you use SQL to directly query data in the data lake, as well as query various other databases. It requires no setup, and the cost is based purely on the amount of data that is scanned to complete the query. 4 | 5 | In this chapter, we did a deep dive into Athena, examining how Athena can be used to query data directly in the data lake, query data from other data sources with Query Federation, and how Athena provides workgroup functionality to help with governance and cost management. 6 | 7 | ## Hands-on Activity 8 | 9 | In the hands-on activity section of this chapter, we're going to create and configure a new Athena Workgroup and learn more about how Workgroups can help separate groups of users. 10 | 11 | #### Creating an Amazon Athena workgroup and configuring Athena settings 12 | 13 | - AWS Management Console - Amazon Athena: https://console.aws.amazon.com/athena 14 | 15 | #### Switching Workgroups and running queries 16 | 17 | - AWS Documentation on IAM Policies for Accessing Workgroups: https://docs.aws.amazon.com/athena/latest/ug/workgroups-iampolicy.html 18 | 19 | - Query to determine most popular category of films (Step 3) 20 | ``` 21 | SELECT category_name, count(category_name) streams 22 | FROM streaming_films 23 | GROUP BY category_name 24 | ORDER BY streams DESC 25 | ``` 26 | 27 | - Query to determine which State streamed the most films (Step 6) 28 | ``` 29 | SELECT state, count(state) count 30 | FROM streaming_films 31 | GROUP BY state 32 | ORDER BY count desc 33 | ``` 34 | 35 | 36 | 37 | 38 | 39 | 40 | -------------------------------------------------------------------------------- /Chapter12/README.md: -------------------------------------------------------------------------------- 1 | # Chapter 12 - Visualizing Data with Amazon QuickSight 2 | 3 | In this chapter we discussed the power of visually representing data, and then explored core Amazon 4 | QuickSight concepts. We looked at how various data sources can be used with QuickSight, 5 | how data can optionally be imported into the SPICE storage engine, and how you can 6 | perform some data preparation tasks using QuickSight. 7 | 8 | We then did a deeper dive into the concepts of analyses (where new visuals are authored) 9 | and dashboards (published analyses that can be shared with data consumers). As part 10 | of this, we also examined some of the common types of visualizations available in 11 | QuickSight. 12 | 13 | We then looked at some of the advanced features available in QuickSight, including ML 14 | Insights (which uses machine learning to detect outliers in data and forecast future data 15 | trends), as well as embedded dashboards (which enable you to embed either the full 16 | QuickSight console or dashboards directly into your websites and applications). 17 | 18 | ## Hands-on Activity 19 | 20 | #### Setting up a new QuickSight account and loading a dataset 21 | In this section, we create a visualization using data from [SimpleMaps.com](https://simplemaps.com). The basic map data is distributed under the [Creative Commons Attribution 4.0 International (CC BY 4.0) license](https://creativecommons.org/licenses/by/4.0/). 22 | 23 | - Link to download the Simple Maps basic data: https://simplemaps.com/static/data/world-cities/basic/simplemaps_worldcities_basicv1.74.zip 24 | [Extract the ZIP file to access the underlying CSV file] 25 | 26 | - Link to the Amazon QuickSight service: https://quicksight.aws.amazon.com/ 27 | 28 | 29 | 30 | 31 | -------------------------------------------------------------------------------- /Chapter13/README.md: -------------------------------------------------------------------------------- 1 | # Chapter 13 - Enabling Artificial Intelligence and Machine Learning 2 | 3 | In this chapter, you learned more about the broad range of AWS ML and AI services 4 | and had the opportunity to get hands-on with Amazon Comprehend, an AI service for 5 | extracting insights from written text. 6 | 7 | We discussed how ML and AI services can apply to a broad range of use cases, both 8 | specialized (such as detecting cancer early) and general (business forecasting or 9 | personalization). 10 | 11 | We examined different AWS services related to ML and AI. We looked at how different 12 | Amazon SageMaker capabilities can be used to prepare data for ML, build models, train 13 | and fine-tune models, and deploy and manage models. SageMaker makes building custom 14 | ML models much more accessible to developers without existing expertise in ML. 15 | 16 | We then looked at a range of AWS AI services that provide prebuilt and trained models for 17 | common use cases. We looked at services for transcribing text from audio files (Amazon 18 | Transcribe), for extracting text from forms and handwritten documents (Amazon 19 | Textract), for recognizing images (Amazon Rekognition), and for extracting insights from 20 | text (Amazon Comprehend). We also briefly discussed other business-focused AI services, 21 | such as Amazon Forecast and Amazon Personalize. 22 | 23 | ## Hands-on Activity 24 | In the hands-on acitvity section of this chapter we looked at how we can use the ***Amazon Comprehend*** service to gain insight into the sentiment of reviews posted to a website. We configured an SQS queue to receive the details of newly posted reviewes, and had a Lambda function configured to pass the review text to Amazon Comprehend to gain insight into the review sentiment (postivie, negative, or netural). 25 | 26 | #### Setting up a new Amazon SQS message queue 27 | 28 | - Amazon Management Console - SQS: https://console.aws.amazon.com/sqs/v2/. 29 | 30 | #### Creating a Lambda function for calling Amazon Comprehend 31 | 32 | - Amazon Management Console - Lambda: https://console.aws.amazon.com/lambda/ 33 | 34 | - Lambda function code for calling Amazon Comprehend for sentiment analysis: [website-reviews-analysis-role.py](website-reviews-analysis-role.py) 35 | [Make sure to set region on Line 4 to the correct region that you are using for the hands-on activities in this book] 36 | 37 | #### Testing the solution with Amazon Comprehend 38 | 39 | - Example of a postivie review 40 | 41 | ``` 42 | I recently stayed at the Kensington Hotel in downtown Cape Town and was very impressed. 43 | The hotel is beautiful, the service from the staff is amazing, and the sea views cannot be beaten. 44 | If you have the time, stop by Mary's Kitchen, a coffee shop not far from the hotel, 45 | to get a coffee and try some of their delicious cakes and baked goods. 46 | ``` 47 | 48 | 49 | 50 | 51 | 52 | 53 | -------------------------------------------------------------------------------- /Chapter13/website-reviews-analysis-role.py: -------------------------------------------------------------------------------- 1 | import boto3 2 | import json 3 | comprehend = boto3.client(service_name='comprehend', 4 | region_name='us-east-2') 5 | 6 | def lambda_handler(event, context): 7 | for record in event['Records']: 8 | payload = record["body"] 9 | print(str(payload)) 10 | 11 | print('Calling DetectSentiment') 12 | response = comprehend.detect_sentiment(Text=payload, 13 | LanguageCode='en') 14 | sentiment = response['Sentiment'] 15 | sentiment_score = response['SentimentScore'] 16 | print(f'SENTIMENT: {sentiment}') 17 | print(f'SENTIMENT SCORE: {sentiment_score}') 18 | 19 | print('Calling DetectEntities') 20 | response = comprehend.detect_entities(Text=payload, 21 | LanguageCode='en') 22 | #print(response['Entities']) 23 | for entity in response['Entities']: 24 | entity_text = entity['Text'] 25 | entity_type = entity['Type'] 26 | print( 27 | f'ENTITY: {entity_text}, ' 28 | f'ENTITY TYPE: {entity_type}' 29 | ) 30 | return 31 | -------------------------------------------------------------------------------- /Chapter14/README.md: -------------------------------------------------------------------------------- 1 | # Chapter 14 - Wrapping Up the First Part of Your Learning Journey 2 | 3 | In this chapter we introduced some important real-world concepts to help 4 | manage the data infrastructure/pipeline development process, had a look at some 5 | examples of real-world data pipelines, and then discussed some emerging trends in the field. 6 | 7 | ## Hands-on Activity 8 | In the hands-on activity we looked at how you can review your AWS spend in the billing console, and then how you could optionally close your AWS account. 9 | 10 | #### Reviewing AWS Billing to identify the resources being charged for 11 | 12 | - AWS Management Console - Billing Console: https://console.aws.amazon.com/billing/home 13 | 14 | - AWS Documentation on how to create a billing alarm to monitor estimated charges: [Documentation Link](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/monitor_estimated_charges_with_cloudwatch.html) 15 | 16 | #### Closing your AWS account 17 | 18 | - AWS Documentation on considerations before closing your AWS account: [Documentation Link](https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/close-account.html) 19 | 20 | - AWS Management Console - Login: https://console.aws.amazon.com 21 | [Make sure to log in with the email address you used when creating the account] 22 | 23 | - AWS Billing Console: https://console.aws.amazon.com/billing/home 24 | 25 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2021 Packt 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | # Data Engineering with AWS 5 | 6 | Data Engineering with AWS 7 | 8 | This is the code repository for [Data Engineering with AWS](https://www.packtpub.com/product/data-engineering-with-aws/9781800560413), published by Packt. 9 | 10 | **Learn how to design and build cloud-based data transformation pipelines using AWS** 11 | 12 | ## What is this book about? 13 | 14 | Knowing how to architect and implement complex data pipelines is a highly sought-after skill. Data engineers are responsible for building these pipelines that ingest, transform, and join raw datasets - creating new value from the data in the process. 15 | 16 | Amazon Web Services (AWS) offers a range of tools to simplify a data engineer's job, making it the preferred platform for performing data engineering tasks. 17 | This book will take you through the services and the skills you need to architect and implement data pipelines on AWS. You'll begin by reviewing important data engineering concepts and some of the core AWS services that form a part of the data engineer's toolkit. You'll then architect a data pipeline, review raw data sources, transform the data, and learn how the transformed data is used by various data consumers. The book also teaches you about populating data marts and data warehouses along with how a data lakehouse fits into the picture. Later, you'll be introduced to AWS tools for analyzing data, including those for ad-hoc SQL queries and creating visualizations. In the final chapters, you'll understand how the power of machine learning and artificial intelligence can be used to draw new insights from data. 18 | 19 | By the end of this AWS book, you'll be able to carry out data engineering tasks and implement a data pipeline on AWS independently. 20 | 21 | This book covers the following exciting features: 22 | * Understand data engineering concepts and emerging technologies 23 | * Ingest streaming data with Amazon Kinesis Data Firehose 24 | * Optimize, denormalize, and join datasets with AWS Glue Studio 25 | * Use Amazon S3 events to trigger a Lambda process to transform a file 26 | * Run complex SQL queries on data lake data using Amazon Athena 27 | * Load data into a Redshift data warehouse and run queries 28 | * Create a visualization of your data using Amazon QuickSight 29 | * Extract sentiment data from a dataset using Amazon Comprehend 30 | 31 | If you feel this book is for you, get your [copy](https://www.amazon.in/Data-Engineering-AWS-cloud-based-transformation/dp/1800560419/ref=sr_1_3?keywords=Data+Engineering+with+AWS&qid=1638757232&sr=8-3) today! 32 | 33 | https://www.packtpub.com/ 34 | 35 | ## Instructions and Navigations 36 | All of the code is organized into folders. 37 | 38 | The code will look like the following: 39 | ``` 40 | import boto3 41 | import awswrangler as wr 42 | from urllib.parse import unquote_plus 43 | ``` 44 | **Following is what you need for this book:** 45 | This book is for data engineers, data analysts, and data architects who are new to AWS and looking to extend their skills to the AWS cloud. Anyone who is new to data engineering and wants to learn about the foundational concepts while gaining practical experience with common data engineering services on AWS will also find this book useful. 46 | A basic understanding of big data-related topics and Python coding will help you get the most out of this book but is not needed. Familiarity with the AWS console and core services is also useful but not necessary. 47 | 48 | With the following software and hardware list you can run all code files present in the book (Chapter 1-14). 49 | 50 | ### Software and Hardware List 51 | 52 | | Chapter | Software required | OS required | 53 | | -------- | -------------------------------------------------------------------------------------| -----------------------------------| 54 | | 1-14 | AWS Web Services(AWS) with a recent version of a modern web browser(Chrome, Edge, etc.) | Any OS | 55 | 56 | We also provide a PDF file that has color images of the screenshots/diagrams used in this book. [Click here to download it](https://static.packt-cdn.com/downloads/9781800560413_ColorImages.pdf). 57 | 58 | ### Related products 59 | * Serverless Analytics with Amazon Athena [[Packt]](https://www.packtpub.com/product/serverless-analytics-with-amazon-athena/9781800562349) [[Amazon]](https://www.amazon.in/Serverless-Analytics-Amazon-Athena-semi-structured/dp/1800562349/ref=sr_1_1?keywords=Serverless+Analytics+with+Amazon+Athena&qid=1638757768&sr=8-1) 60 | 61 | * Scalable Data Streaming with Amazon Kinesis [[Packt]](https://www.packtpub.com/product/scalable-data-streaming-with-amazon-kinesis/9781800565401) [[Amazon]](https://www.amazon.in/Scalable-Data-Streaming-Amazon-Kinesis/dp/1800565402/ref=sr_1_1?keywords=Scalable+Data+Streaming+with+Amazon+Kinesis&qid=1638757818&sr=8-1) 62 | 63 | ## Get to Know the Author 64 | **Gareth Eagar** has worked in the IT industry for over 25 years, starting in South Africa, then working in the United Kingdom, and now based in the United States. In 2017, he started working at Amazon Web Services (AWS) as a solution architect, working with enterprise customers in the NYC metro area. Gareth has become a recognized subject matter expert for building data lakes on AWS, and in 2019 he launched the Data Lake Day educational event at the AWS Lofts in NYC and San Francisco. He has also delivered a number of public talks and webinars on topics relating to big data, and in 2020 Gareth transitioned to the AWS Professional Services organization as a senior data architect, helping customers architect and build complex data pipelines. 65 | 66 | **Note from the author:** 67 | 68 | You can use the resources provided in this GitHub repo as you work through the hands-on activities includes in each chapter of the book. This repo is laid out with resources matched to each chapter of the book - such as the JSON used to define IAM policies, sample files, relevant links, etc. 69 | ### Download a free PDF 70 | 71 | If you have already purchased a print or Kindle version of this book, you can get a DRM-free PDF version at no cost.
Simply click on the link to claim your free PDF. 72 |

https://packt.link/free-ebook/9781800560413

--------------------------------------------------------------------------------