├── .github └── PULL_REQUEST_TEMPLATE.md ├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── LICENSE ├── LICENSE-SAMPLECODE ├── LICENSE-SUMMARY ├── README.md ├── assets ├── cfn1.png ├── cfn2.png ├── cfn3.png ├── cfn4.png └── cfn5.png ├── lab1-TaxiDataMigration ├── README.md └── assets │ ├── cfn-lab1.png │ ├── cfn6.png │ ├── cloud9-1.png │ ├── cloud9-2.png │ ├── ddb-start-task.png │ ├── ddb.png │ ├── dms-task-2-5.png │ ├── dms-task1-1.png │ ├── dms-task1-2.png │ ├── dms-task1-3.png │ ├── dms-task1-4.png │ ├── dms-task2-1.png │ ├── dms-task2-2.png │ ├── dms-task2-3.png │ ├── dms-task2-4.png │ ├── dms-task2-6.png │ ├── dms1.png │ ├── dms2.png │ ├── dms3.png │ ├── dms4.png │ ├── dms5.png │ ├── dms6.png │ ├── dms7.png │ ├── lab1-arch.png │ ├── oracle-taxi-schema.txt │ └── taxi-data-model.png ├── lab2-TaxiBookingAndPayments ├── README.md └── assets │ ├── architecture.png │ └── asset0.txt ├── lab3-AthenaFederatedQuery ├── README.md └── assets │ ├── athenaqryloc.png │ ├── catalog.png │ ├── datasource.png │ ├── ddb.png │ ├── ddbconn.png │ ├── lambda.png │ ├── lambdafn.png │ ├── pgconn.png │ ├── pgdatasource.png │ ├── s3bucketloc.png │ ├── sam.png │ ├── switch.png │ └── workgroup.png └── src ├── cloudformation.template ├── create_taxi_schema.sql ├── create_taxi_schema_oracle.sql ├── ddb-python-script ├── driver-accept-trip.py ├── driver-complete-trip.py ├── rider-book-trip.py └── util.py ├── ddb-stream-processor ├── lambda_function.py └── template.yaml ├── drop_taxi_schema.sql ├── taxi-ride-workflow ├── dependencies │ └── util.zip ├── driver-accept-trip │ └── driver-accept-trip.py ├── driver-complete-trip │ └── driver-complete-trip.py ├── rider-book-trip │ └── rider-book-trip.py └── template.yaml └── taxi.dmp /.github/PULL_REQUEST_TEMPLATE.md: -------------------------------------------------------------------------------- 1 | *Issue #, if available:* 2 | 3 | *Description of changes:* 4 | 5 | 6 | By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice. 7 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | ## Code of Conduct 2 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 3 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 4 | opensource-codeofconduct@amazon.com with any additional questions or comments. 5 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Guidelines for contributing 2 | 3 | Thank you for your interest in contributing to AWS documentation! We greatly value feedback and contributions from our community. 4 | 5 | Please read through this document before you submit any pull requests or issues. It will help us work together more effectively. 6 | 7 | ## What to expect when you contribute 8 | 9 | When you submit a pull request, our team is notified and will respond as quickly as we can. We'll do our best to work with you to ensure that your pull request adheres to our style and standards. If we merge your pull request, we might make additional edits later for style or clarity. 10 | 11 | The AWS documentation source files on GitHub aren't published directly to the official documentation website. If we merge your pull request, we'll publish your changes to the documentation website as soon as we can, but they won't appear immediately or automatically. 12 | 13 | We look forward to receiving your pull requests for: 14 | 15 | * New content you'd like to contribute (such as new code samples or tutorials) 16 | * Inaccuracies in the content 17 | * Information gaps in the content that need more detail to be complete 18 | * Typos or grammatical errors 19 | * Suggested rewrites that improve clarity and reduce confusion 20 | 21 | **Note:** We all write differently, and you might not like how we've written or organized something currently. We want that feedback. But please be sure that your request for a rewrite is supported by the previous criteria. If it isn't, we might decline to merge it. 22 | 23 | ## How to contribute 24 | 25 | To contribute, send us a pull request. For small changes, such as fixing a typo or adding a link, you can use the [GitHub Edit Button](https://blog.github.com/2011-04-26-forking-with-the-edit-button/). For larger changes: 26 | 27 | 1. [Fork the repository](https://help.github.com/articles/fork-a-repo/). 28 | 2. In your fork, make your change in a branch that's based on this repo's **master** branch. 29 | 3. Commit the change to your fork, using a clear and descriptive commit message. 30 | 4. [Create a pull request](https://help.github.com/articles/creating-a-pull-request-from-a-fork/), answering any questions in the pull request form. 31 | 32 | Before you send us a pull request, please be sure that: 33 | 34 | 1. You're working from the latest source on the **master** branch. 35 | 2. You check [existing open](https://github.com/awsdocs/amazon-rds-purpose-built-workshop/pulls), and [recently closed](https://github.com/awsdocs/amazon-rds-purpose-built-workshop/pulls?q=is%3Apr+is%3Aclosed), pull requests to be sure that someone else hasn't already addressed the problem. 36 | 3. You [create an issue](https://github.com/awsdocs/amazon-rds-purpose-built-workshop/issues/new) before working on a contribution that will take a significant amount of your time. 37 | 38 | For contributions that will take a significant amount of time, [open a new issue](https://github.com/awsdocs/amazon-rds-purpose-built-workshop/issues/new) to pitch your idea before you get started. Explain the problem and describe the content you want to see added to the documentation. Let us know if you'll write it yourself or if you'd like us to help. We'll discuss your proposal with you and let you know whether we're likely to accept it. We don't want you to spend a lot of time on a contribution that might be outside the scope of the documentation or that's already in the works. 39 | 40 | ## Finding contributions to work on 41 | 42 | If you'd like to contribute, but don't have a project in mind, look at the [open issues](https://github.com/awsdocs/amazon-rds-purpose-built-workshop/issues) in this repository for some ideas. Any issues with the [help wanted](https://github.com/awsdocs/amazon-rds-purpose-built-workshop/labels/help%20wanted) or [enhancement](https://github.com/awsdocs/amazon-rds-purpose-built-workshop/labels/enhancement) labels are a great place to start. 43 | 44 | In addition to written content, we really appreciate new examples and code samples for our documentation, such as examples for different platforms or environments, and code samples in additional languages. 45 | 46 | ## Code of conduct 47 | 48 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). For more information, see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact [opensource-codeofconduct@amazon.com](mailto:opensource-codeofconduct@amazon.com) with any additional questions or comments. 49 | 50 | ## Security issue notifications 51 | 52 | If you discover a potential security issue, please notify AWS Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public issue on GitHub. 53 | 54 | ## Licensing 55 | 56 | See the [LICENSE](https://github.com/awsdocs/amazon-rds-purpose-built-workshop/blob/master/LICENSE) file for this project's licensing. We will ask you to confirm the licensing of your contribution. We may ask you to sign a [Contributor License Agreement (CLA)](http://en.wikipedia.org/wiki/Contributor_License_Agreement) for larger changes. 57 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Creative Commons Attribution-ShareAlike 4.0 International Public License 2 | 3 | By exercising the Licensed Rights (defined below), You accept and agree to be bound by the terms and conditions of this Creative Commons Attribution-ShareAlike 4.0 International Public License ("Public License"). To the extent this Public License may be interpreted as a contract, You are granted the Licensed Rights in consideration of Your acceptance of these terms and conditions, and the Licensor grants You such rights in consideration of benefits the Licensor receives from making the Licensed Material available under these terms and conditions. 4 | 5 | Section 1 – Definitions. 6 | 7 | a. Adapted Material means material subject to Copyright and Similar Rights that is derived from or based upon the Licensed Material and in which the Licensed Material is translated, altered, arranged, transformed, or otherwise modified in a manner requiring permission under the Copyright and Similar Rights held by the Licensor. For purposes of this Public License, where the Licensed Material is a musical work, performance, or sound recording, Adapted Material is always produced where the Licensed Material is synched in timed relation with a moving image. 8 | 9 | b. Adapter's License means the license You apply to Your Copyright and Similar Rights in Your contributions to Adapted Material in accordance with the terms and conditions of this Public License. 10 | 11 | c. BY-SA Compatible License means a license listed at creativecommons.org/compatiblelicenses, approved by Creative Commons as essentially the equivalent of this Public License. 12 | 13 | d. Copyright and Similar Rights means copyright and/or similar rights closely related to copyright including, without limitation, performance, broadcast, sound recording, and Sui Generis Database Rights, without regard to how the rights are labeled or categorized. For purposes of this Public License, the rights specified in Section 2(b)(1)-(2) are not Copyright and Similar Rights. 14 | 15 | e. Effective Technological Measures means those measures that, in the absence of proper authority, may not be circumvented under laws fulfilling obligations under Article 11 of the WIPO Copyright Treaty adopted on December 20, 1996, and/or similar international agreements. 16 | 17 | f. Exceptions and Limitations means fair use, fair dealing, and/or any other exception or limitation to Copyright and Similar Rights that applies to Your use of the Licensed Material. 18 | 19 | g. License Elements means the license attributes listed in the name of a Creative Commons Public License. The License Elements of this Public License are Attribution and ShareAlike. 20 | 21 | h. Licensed Material means the artistic or literary work, database, or other material to which the Licensor applied this Public License. 22 | 23 | i. Licensed Rights means the rights granted to You subject to the terms and conditions of this Public License, which are limited to all Copyright and Similar Rights that apply to Your use of the Licensed Material and that the Licensor has authority to license. 24 | 25 | j. Licensor means the individual(s) or entity(ies) granting rights under this Public License. 26 | 27 | k. Share means to provide material to the public by any means or process that requires permission under the Licensed Rights, such as reproduction, public display, public performance, distribution, dissemination, communication, or importation, and to make material available to the public including in ways that members of the public may access the material from a place and at a time individually chosen by them. 28 | 29 | l. Sui Generis Database Rights means rights other than copyright resulting from Directive 96/9/EC of the European Parliament and of the Council of 11 March 1996 on the legal protection of databases, as amended and/or succeeded, as well as other essentially equivalent rights anywhere in the world. 30 | 31 | m. You means the individual or entity exercising the Licensed Rights under this Public License. Your has a corresponding meaning. 32 | 33 | Section 2 – Scope. 34 | 35 | a. License grant. 36 | 37 | 1. Subject to the terms and conditions of this Public License, the Licensor hereby grants You a worldwide, royalty-free, non-sublicensable, non-exclusive, irrevocable license to exercise the Licensed Rights in the Licensed Material to: 38 | 39 | A. reproduce and Share the Licensed Material, in whole or in part; and 40 | 41 | B. produce, reproduce, and Share Adapted Material. 42 | 43 | 2. Exceptions and Limitations. For the avoidance of doubt, where Exceptions and Limitations apply to Your use, this Public License does not apply, and You do not need to comply with its terms and conditions. 44 | 45 | 3. Term. The term of this Public License is specified in Section 6(a). 46 | 47 | 4. Media and formats; technical modifications allowed. The Licensor authorizes You to exercise the Licensed Rights in all media and formats whether now known or hereafter created, and to make technical modifications necessary to do so. The Licensor waives and/or agrees not to assert any right or authority to forbid You from making technical modifications necessary to exercise the Licensed Rights, including technical modifications necessary to circumvent Effective Technological Measures. For purposes of this Public License, simply making modifications authorized by this Section 2(a)(4) never produces Adapted Material. 48 | 49 | 5. Downstream recipients. 50 | 51 | A. Offer from the Licensor – Licensed Material. Every recipient of the Licensed Material automatically receives an offer from the Licensor to exercise the Licensed Rights under the terms and conditions of this Public License. 52 | 53 | B. Additional offer from the Licensor – Adapted Material. Every recipient of Adapted Material from You automatically receives an offer from the Licensor to exercise the Licensed Rights in the Adapted Material under the conditions of the Adapter’s License You apply. 54 | 55 | C. No downstream restrictions. You may not offer or impose any additional or different terms or conditions on, or apply any Effective Technological Measures to, the Licensed Material if doing so restricts exercise of the Licensed Rights by any recipient of the Licensed Material. 56 | 57 | 6. No endorsement. Nothing in this Public License constitutes or may be construed as permission to assert or imply that You are, or that Your use of the Licensed Material is, connected with, or sponsored, endorsed, or granted official status by, the Licensor or others designated to receive attribution as provided in Section 3(a)(1)(A)(i). 58 | 59 | b. Other rights. 60 | 61 | 1. Moral rights, such as the right of integrity, are not licensed under this Public License, nor are publicity, privacy, and/or other similar personality rights; however, to the extent possible, the Licensor waives and/or agrees not to assert any such rights held by the Licensor to the limited extent necessary to allow You to exercise the Licensed Rights, but not otherwise. 62 | 63 | 2. Patent and trademark rights are not licensed under this Public License. 64 | 65 | 3. To the extent possible, the Licensor waives any right to collect royalties from You for the exercise of the Licensed Rights, whether directly or through a collecting society under any voluntary or waivable statutory or compulsory licensing scheme. In all other cases the Licensor expressly reserves any right to collect such royalties. 66 | 67 | Section 3 – License Conditions. 68 | 69 | Your exercise of the Licensed Rights is expressly made subject to the following conditions. 70 | 71 | a. Attribution. 72 | 73 | 1. If You Share the Licensed Material (including in modified form), You must: 74 | 75 | A. retain the following if it is supplied by the Licensor with the Licensed Material: 76 | 77 | i. identification of the creator(s) of the Licensed Material and any others designated to receive attribution, in any reasonable manner requested by the Licensor (including by pseudonym if designated); 78 | 79 | ii. a copyright notice; 80 | 81 | iii. a notice that refers to this Public License; 82 | 83 | iv. a notice that refers to the disclaimer of warranties; 84 | 85 | v. a URI or hyperlink to the Licensed Material to the extent reasonably practicable; 86 | 87 | B. indicate if You modified the Licensed Material and retain an indication of any previous modifications; and 88 | 89 | C. indicate the Licensed Material is licensed under this Public License, and include the text of, or the URI or hyperlink to, this Public License. 90 | 91 | 2. You may satisfy the conditions in Section 3(a)(1) in any reasonable manner based on the medium, means, and context in which You Share the Licensed Material. For example, it may be reasonable to satisfy the conditions by providing a URI or hyperlink to a resource that includes the required information. 92 | 93 | 3. If requested by the Licensor, You must remove any of the information required by Section 3(a)(1)(A) to the extent reasonably practicable. 94 | 95 | b. ShareAlike.In addition to the conditions in Section 3(a), if You Share Adapted Material You produce, the following conditions also apply. 96 | 97 | 1. The Adapter’s License You apply must be a Creative Commons license with the same License Elements, this version or later, or a BY-SA Compatible License. 98 | 99 | 2. You must include the text of, or the URI or hyperlink to, the Adapter's License You apply. You may satisfy this condition in any reasonable manner based on the medium, means, and context in which You Share Adapted Material. 100 | 101 | 3. You may not offer or impose any additional or different terms or conditions on, or apply any Effective Technological Measures to, Adapted Material that restrict exercise of the rights granted under the Adapter's License You apply. 102 | 103 | Section 4 – Sui Generis Database Rights. 104 | 105 | Where the Licensed Rights include Sui Generis Database Rights that apply to Your use of the Licensed Material: 106 | 107 | a. for the avoidance of doubt, Section 2(a)(1) grants You the right to extract, reuse, reproduce, and Share all or a substantial portion of the contents of the database; 108 | 109 | b. if You include all or a substantial portion of the database contents in a database in which You have Sui Generis Database Rights, then the database in which You have Sui Generis Database Rights (but not its individual contents) is Adapted Material, including for purposes of Section 3(b); and 110 | 111 | c. You must comply with the conditions in Section 3(a) if You Share all or a substantial portion of the contents of the database. 112 | For the avoidance of doubt, this Section 4 supplements and does not replace Your obligations under this Public License where the Licensed Rights include other Copyright and Similar Rights. 113 | 114 | Section 5 – Disclaimer of Warranties and Limitation of Liability. 115 | 116 | a. Unless otherwise separately undertaken by the Licensor, to the extent possible, the Licensor offers the Licensed Material as-is and as-available, and makes no representations or warranties of any kind concerning the Licensed Material, whether express, implied, statutory, or other. This includes, without limitation, warranties of title, merchantability, fitness for a particular purpose, non-infringement, absence of latent or other defects, accuracy, or the presence or absence of errors, whether or not known or discoverable. Where disclaimers of warranties are not allowed in full or in part, this disclaimer may not apply to You. 117 | 118 | b. To the extent possible, in no event will the Licensor be liable to You on any legal theory (including, without limitation, negligence) or otherwise for any direct, special, indirect, incidental, consequential, punitive, exemplary, or other losses, costs, expenses, or damages arising out of this Public License or use of the Licensed Material, even if the Licensor has been advised of the possibility of such losses, costs, expenses, or damages. Where a limitation of liability is not allowed in full or in part, this limitation may not apply to You. 119 | 120 | c. The disclaimer of warranties and limitation of liability provided above shall be interpreted in a manner that, to the extent possible, most closely approximates an absolute disclaimer and waiver of all liability. 121 | 122 | Section 6 – Term and Termination. 123 | 124 | a. This Public License applies for the term of the Copyright and Similar Rights licensed here. However, if You fail to comply with this Public License, then Your rights under this Public License terminate automatically. 125 | 126 | b. Where Your right to use the Licensed Material has terminated under Section 6(a), it reinstates: 127 | 128 | 1. automatically as of the date the violation is cured, provided it is cured within 30 days of Your discovery of the violation; or 129 | 130 | 2. upon express reinstatement by the Licensor. 131 | 132 | c. For the avoidance of doubt, this Section 6(b) does not affect any right the Licensor may have to seek remedies for Your violations of this Public License. 133 | 134 | d. For the avoidance of doubt, the Licensor may also offer the Licensed Material under separate terms or conditions or stop distributing the Licensed Material at any time; however, doing so will not terminate this Public License. 135 | 136 | e. Sections 1, 5, 6, 7, and 8 survive termination of this Public License. 137 | 138 | Section 7 – Other Terms and Conditions. 139 | 140 | a. The Licensor shall not be bound by any additional or different terms or conditions communicated by You unless expressly agreed. 141 | 142 | b. Any arrangements, understandings, or agreements regarding the Licensed Material not stated herein are separate from and independent of the terms and conditions of this Public License. 143 | 144 | Section 8 – Interpretation. 145 | 146 | a. For the avoidance of doubt, this Public License does not, and shall not be interpreted to, reduce, limit, restrict, or impose conditions on any use of the Licensed Material that could lawfully be made without permission under this Public License. 147 | 148 | b. To the extent possible, if any provision of this Public License is deemed unenforceable, it shall be automatically reformed to the minimum extent necessary to make it enforceable. If the provision cannot be reformed, it shall be severed from this Public License without affecting the enforceability of the remaining terms and conditions. 149 | 150 | c. No term or condition of this Public License will be waived and no failure to comply consented to unless expressly agreed to by the Licensor. 151 | 152 | d. Nothing in this Public License constitutes or may be interpreted as a limitation upon, or waiver of, any privileges and immunities that apply to the Licensor or You, including from the legal processes of any jurisdiction or authority. 153 | -------------------------------------------------------------------------------- /LICENSE-SAMPLECODE: -------------------------------------------------------------------------------- 1 | Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining a copy of this 4 | software and associated documentation files (the "Software"), to deal in the Software 5 | without restriction, including without limitation the rights to use, copy, modify, 6 | merge, publish, distribute, sublicense, and/or sell copies of the Software, and to 7 | permit persons to whom the Software is furnished to do so. 8 | 9 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, 10 | INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A 11 | PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT 12 | HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION 13 | OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE 14 | SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 15 | -------------------------------------------------------------------------------- /LICENSE-SUMMARY: -------------------------------------------------------------------------------- 1 | Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | 3 | The documentation is made available under the Creative Commons Attribution-ShareAlike 4.0 International License. See the LICENSE file. 4 | 5 | The sample code within this documentation is made available under the MIT-0 license. See the LICENSE-SAMPLECODE file. 6 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Database Modernization Hands On Workshop 2 | 3 | A tutorial for developers, DBAs and data engineers to get hands-on experience on how to migrate relational data to AWS purpose-built databases such as Amazon DynamoDB and Amazon Aurora using AWS DMS and build data processing applications on top of it. 4 | 5 | # Overview: Working with AWS Purpose-Built databases 6 | 7 | The days of one-size-fits-all, monolithic databases are behind us. Database design and management requires a different mindset in AWS compared to traditional relational database management system (RDBMS) design. In this workshop, we will demonstrate how to leverage both relational (Amazon Aurora) and non-relational databases (Amazon DynamoDB) that are purpose-built to handle the specific needs of an application. For illustrative purpose, we will leverage a relational schema used by a taxi application. 8 | 9 | ## Pre-requisites 10 | 11 | If you are doing this lab as part of a workshop conducted by AWS, all the CloudFormation templates required by this lab will be pre-installed in the provided AWS account. 12 | 13 | If you are doing this lab on your own, you need to have an AWS account with IAM administrator privileges. You can launch the [lab CloudFormation template](./src/cloudformation.template) in [us-east-1](https://console.aws.amazon.com/console/home?region=us-east-1) AWS region to setup the required resources for this lab. 14 | 15 | We will leverage the following AWS services as part of this workshop. 16 | 17 | - [VPC](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Scenario2.html) with Public and Private subnets, NAT Gateway and Route tables 18 | - [Oracle RDS](https://aws.amazon.com/rds/oracle/) instance launched from a snapshot preloaded with a sample taxi schema . This will be used as a source for our migration. 19 | - [Amazon Aurora PostgreSQL](https://aws.amazon.com/rds/aurora/postgresql-features/) as a target for relational data 20 | - [Amazon DynamoDB](https://aws.amazon.com/dynamodb/) as a target for NoSQL data 21 | - [AWS DMS](https://aws.amazon.com/dms/) for migrating database from source to target 22 | - [AWS Lambda](https://aws.amazon.com/lambda/) for event driven data processing 23 | - [AWS Cloud9](https://aws.amazon.com/cloud9) IDE for running scripts and deploying code 24 | - [IAM Roles](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html) for permissions required by AWS DMS 25 | 26 | :warning: **You will be billed for the AWS resource usage if you running this lab on your own AWS Account. Make sure to delete the CloudFormation stacks and AWS resources created by this Lab after you are done to avoid incurring additional charges.** 27 | 28 | ## (Optional) SQL Client Installation 29 | Download and install a SQL Client in your laptop which can connect to both Oracle and PostgreSQL if you don't already have one. Suggested open source tools which can work with both Oracle and PostgreSQL are provided below. Please note that these tools require [PostgreSQL JDBC](https://jdbc.postgresql.org/) and [Oracle JDBC](https://www.oracle.com/technetwork/database/features/jdbc/jdbc-drivers-12c-download-1958347.html) drivers for connectivity. 30 | 31 | 32 | - [dbeaver Community Edition](https://dbeaver.io/download/) 33 | 34 | 35 | - [SQL-Workbench](https://www.sql-workbench.eu/downloads.html) 36 | 37 | 38 | For step-by-step instructions on how to configure SQL-Workbench to connect to Oracle/PostgreSQL instances, please refer to AWS Documentation ([PostgreSQL](https://aws.amazon.com/getting-started/tutorials/create-connect-postgresql-db/) and [Oracle](https://docs.aws.amazon.com/dms/latest/sbs/CHAP_RDSOracle2Aurora.Steps.ConnectOracle.html)). 39 | 40 | > **_NOTE:_** For doing these Labs, you don't need an Oracle client or any GUI based client. An Oracle client is required only if you want to explore the sample data in the source Oracle database. For working with Aurora PostgreSQL database, you can leverage [psql](https://www.postgresql.org/docs/9.5/app-psql.html) command line utility. We will install this as part of Lab 1. 41 | 42 | 43 | ## Workshop Details 44 | 45 | **Lab 1**: In this lab, you will be performing a migration of sample taxi data from RDS Oracle to Amazon DynamoDB and Amazon Aurora PostgreSQL databases using AWS DMS. 46 | 47 | **Lab 2**: In this lab, you will simulate taxi trip booking by a rider and acceptance by a driver followed by billing and payment using Python scripts and SQL commands. You will utilize DynamoDB streams and AWS lambda functions to insert completed trip data from DynamoDB to Aurora PostgreSQL. 48 | 49 | **Lab 3**: In this lab, we will leverage Athena federated query feature (**_in preview_**) to query both DynamoDB and Aurora trip data using a single SQL query. You will also utilize this feature to query DynamoDB and data stored in Amazon S3. 50 | 51 | 52 | |Lab|Name|Estimated Completion Time| 53 | |---|----|----| 54 | |Lab 1|[Taxi Data Migration using AWS DMS](./lab1-TaxiDataMigration)|50 minutes| 55 | |Lab 2|[Taxi Booking, Billing and Payments](./lab2-TaxiBookingAndPayments)|40 minutes| 56 | |Lab 3|[Query multiple data stores using Athena Federated Query](./lab3-AthenaFederatedQuery)|30 minutes| 57 | 58 | 59 | ## License Summary 60 | 61 | The documentation is made available under the Creative Commons Attribution-ShareAlike 4.0 International License. See the LICENSE file. 62 | 63 | The sample code within this documentation is made available under the MIT-0 license. See the LICENSE-SAMPLECODE file. 64 | -------------------------------------------------------------------------------- /assets/cfn1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-rds-purpose-built-workshop/be574ce7c00e5badd8d96e457d17900bb58a0897/assets/cfn1.png -------------------------------------------------------------------------------- /assets/cfn2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-rds-purpose-built-workshop/be574ce7c00e5badd8d96e457d17900bb58a0897/assets/cfn2.png -------------------------------------------------------------------------------- /assets/cfn3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-rds-purpose-built-workshop/be574ce7c00e5badd8d96e457d17900bb58a0897/assets/cfn3.png -------------------------------------------------------------------------------- /assets/cfn4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-rds-purpose-built-workshop/be574ce7c00e5badd8d96e457d17900bb58a0897/assets/cfn4.png -------------------------------------------------------------------------------- /assets/cfn5.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-rds-purpose-built-workshop/be574ce7c00e5badd8d96e457d17900bb58a0897/assets/cfn5.png -------------------------------------------------------------------------------- /lab1-TaxiDataMigration/README.md: -------------------------------------------------------------------------------- 1 | # Lab 1: Migrating Taxi data to Amazon DynamoDB and Amazon Aurora using AWS DMS 2 | 3 | - [Overview](#overview) 4 | - [High Level Architecture](#high-level-architecture) 5 | - [Preparing the Environment](#preparing-the-environment) 6 | - [Creating Endpoints for Source and Target databases](#creating-endpoints-for-source-and-target-databases) 7 | * [Create a Source endpoint for Oracle RDS](#create-a-source-endpoint-for-oracle-rds) 8 | * [Create a Target endpoint for Aurora PostgreSQL](#create-a-target-endpoint-for-aurora-postgresql) 9 | * [Create a Target endpoint for Amazon DynamoDB](#create-a-target-endpoint-for-amazon-dynamodb) 10 | - [Migrate data from Oracle source to DynamoDB target](#migrate-data-from-oracle-source-to-dynamodb-target) 11 | * [Creating Replication Task for DynamoDB Migration](#creating-replication-task-for-dynamodb-migration) 12 | * [Modify Replication Task for DynamoDB](#modify-replication-task-for-dynamodb) 13 | * [Monitoring Replication Task for DynamoDB](#monitoring-replication-task-for-dynamodb) 14 | - [Migrate data from Oracle source to Aurora PostgreSQL target](#migrate-data-from-oracle-source-to-aurora-postgresql-target) 15 | * [Creating Replication Task for Aurora Migration](#creating-replication-task-for-aurora-migration) 16 | * [Monitoring Replication Task for Aurora PostgreSQL](#monitoring-replication-task-for-aurora-postgresql) 17 | - [Final Validation of DMS Tasks](#final-validation-of-dms-tasks) 18 | 19 | ## Overview 20 | 21 | [AWS DMS](https://aws.amazon.com/dms/) (Database Migration Service) helps you migrate databases to AWS quickly and securely. 22 | In this lab, you will be performing a migration of sample taxi application data from RDS Oracle to Amazon DynamoDB and Amazon Aurora PostgreSQL databases. For this lab, we have created a typical relational schema with foreign key dependencies between tables. We have used sample trip data (green taxi-Jan 2016) from [AWS Open dataset registry](https://registry.opendata.aws/nyc-tlc-trip-records-pds/) to populate trips table. 23 | 24 | 25 | ## High Level Architecture 26 | 27 | As part of this lab, we will migrate the **Trips** table which is used by trips booking and management application to DynamoDB. The application will store the data as a key-value schema and leverage the automatic scaling, flexible schema, serverless characteristics of DynamoDB for better scalability, availability and performance. 28 | 29 | In DynamoDB you work with tables, items, and attributes that are the core components. A table is a collection of items and each item is a collection of attributes. DynamoDB uses primary keys called partition keys to uniquely identify each item in a [table](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.CoreComponents.html). You can also use [secondary indexes](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/SecondaryIndexes.html) to provide more querying flexibility. For this lab, we have created a DynamoDB table namely [aws-db-workshop-trips](./assets/ddb.png) with a partition\_key (_riderid_) and sort\_key (_tripinfo_) which uniquely identifies the trip data. We will also create a secondary index on _driverid_ so that the application can query the table using both _riderid_ as well as _driverid_. 30 | 31 | 32 | For billing and payment use cases, we will migrate the **Billing**, **Riders**, **Drivers** and **Payment** tables to Aurora PostgreSQL. Amazon Aurora combines the performance and availability of traditional enterprise databases with the simplicity and cost-effectiveness of open source databases. The billing and payment application will leverage the ACID, transactional and analytics capabilities of [PostgreSQL](https://aws.amazon.com/rds/aurora/postgresql-features/). Using Aurora, we can scale the read traffic by adding additional replicas as needed. The figure below depicts the high level deployment architecture. 33 | 34 | 35 | ![](./assets/lab1-arch.png) 36 | 37 | ## Preparing the Environment 38 | 39 | 1. Check if the CloudFormation Stack has successfully created the AWS resources. Go to [CloudFormation](https://us-east-1.console.aws.amazon.com/cloudformation/home?region=us-east-1#). You will see two Stacks created as shown below. Click on the **Stack** that is created with the name **mod-**. This is a parent stack. The Stack with the name **aws-cloud9-xx** is a child stack and is used for launching Cloud9 environment. Click the parent stack and look at the **Outputs** section. We suggest to copy paste all the output values in a notepad. These details will be used in the subsequent steps as well as in Lab 2 and Lab 3. 40 | 41 | ![](./assets/cfn-lab1.png) 42 | 43 | ![](./assets/cfn6.png) 44 | 45 | 46 | 47 | 2. **Optional Step** Test the connectivity to RDS Oracle from your laptop using SQL client. You may want to explore the source Oracle schema by running some sample queries using SQL client of your choice. Alternatively, you can also explore the source relational schema [data model](./assets/taxi-data-model.png) and [sample output](./assets/oracle-taxi-schema.txt). Sample query output from source Oracle database schema are given below for your reference. 48 | 49 | Use the following information to connect to the RDS Oracle database. 50 | 51 | |Parameter|Description| 52 | |-------------|--------------| 53 | |Server name | Enter the Oracle RDS DNS value shown in parent CloudFormation stack output| 54 | |Port | 1521| 55 | |Username | dbadmin| 56 | |Password| oraadmin123| 57 | |SID| ORCL| 58 | 59 | ```sql 60 | SELECT OWNER,OBJECT_TYPE, Count(*) FROM DBA_OBJECTS WHERE OWNER IN ('TAXI') GROUP BY OWNER,OBJECT_TYPE; 61 | OWNER OBJECT_TYPE COUNT(*) 62 | TAXI TRIGGER 3 63 | TAXI LOB 2 64 | TAXI TABLE 11 65 | TAXI SEQUENCE 5 66 | TAXI INDEX 17 67 | 68 | Select count(*) from taxi.trips; 69 | Count(*) 70 | 128714 71 | ``` 72 | 73 | 3. We will leverage [AWS Cloud9](https://aws.amazon.com/cloud9/) IDE throughout this workshop for running scripts and deploying code, etc. 74 | 75 | 4. Open [Cloud9](https://us-east-1.console.aws.amazon.com/cloud9/home?region=us-east-1#) development environment which is created as part of the CloudFormation stack. Click on __Open IDE__. 76 | 77 | ![](./assets/cloud9-1.png) 78 | 79 | 5. Open a terminal window in the AWS Cloud9 IDE by clicking on __Window__ from the menu bar on the top and select __New Terminal__. In the Cloud9 terminal, run the below command to clone the github repository. 80 | 81 | ```shell script 82 | cd ~/environment 83 | git clone https://github.com/aws-samples/amazon-rds-purpose-built-workshop.git 84 | ``` 85 | 86 | 6. Install PostgreSQL client and related libraries in the Cloud9 environment. This is required to use the PostgreSQL command line utility psql. 87 | 88 | ```shell script 89 | sudo yum install -y postgresql96 postgresql96-contrib postgresql96-devel 90 | ``` 91 | 92 | 7. Install [JQ](https://stedolan.github.io/jq/) in the Cloud9 environment. We will leverage **JQ** to slice and filter JSON data. 93 | 94 | ```shell script 95 | cd ~/environment 96 | sudo yum -y install jq gettext 97 | ``` 98 | 99 | 8. Connect to target Aurora PostgreSQL using psql command as shown below in the Cloud9 terminal. For more options and commands, refer to [psql](https://www.postgresql.org/docs/9.6/app-psql.html) documentation. 100 | 101 | Use the following information to connect to the Aurora PostgreSQL database. 102 | 103 | |Parameter| Description| 104 | |------|------------- 105 | |Host Name | Enter the AuroraClusterEndpointName value shown in parent CloudFormation stack output| 106 | |Port | 5432| 107 | |Username | auradmin| 108 | |Password| auradmin123| 109 | |Database Name| taxidb| 110 | 111 | ```shell script 112 | psql -h -U -d -p 113 | ``` 114 | 115 | e.g. psql -h xxxxx.us-east-1.rds.amazonaws.com -U auradmin -d taxidb -p 5432 116 | 117 | ```shell script 118 | \l #list the databases in PostgreSQL cluster 119 | ``` 120 | 121 | ```shell script 122 | \dn #list the schemas in the database 123 | ``` 124 | 125 | ```shell script 126 | \d #list the tables in the public schema of the database 127 | ``` 128 | 129 | ```shell script 130 | \q #quit 131 | ``` 132 | 133 | > Note: As you have figured out, there are no tables created in Aurora PostgreSQL yet. 134 | 135 | 9. Please note that before we migrate data from Oracle RDS to Aurora PostgreSQL, we need to setup a target schema. We recommend to leverage [AWS SCT](https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Welcome.html) to migrate schema from Oracle to PostgreSQL. However, for this workshop, we have provided a converted schema to use in the target Aurora PostgreSQL environment. Please execute the following command in the Cloud9 terminal to create the schema. 136 | 137 | ```shell script 138 | cd ~/environment/amazon-rds-purpose-built-workshop/ 139 | psql -h -U auradmin -d taxidb -p 5432 -f ./src/create_taxi_schema.sql 140 | ``` 141 | 142 | You can verify if the tables are created by running the below command after logging via psql. 143 | 144 | ```shell script 145 | \dt #list the tables in the public schema of the database 146 | \d trips #describe a table 147 | ``` 148 | 149 | ![](./assets/cloud9-2.png) 150 | 151 | **Good Job!!** At this point, you have completed all the pre-requisites. Please proceed to the data migration part. 152 | 153 | ## Creating Endpoints for Source and Target databases 154 | 155 | Before we perform data migration using AWS DMS, we need to create endpoints for both source and target databases. This will be required for creating a migration task later in the workshop. 156 | 157 | Open the [AWS DMS console](https://us-east-1.console.aws.amazon.com/dms/home?region=us-east-1), and choose **Endpoints** in the navigation pane. 158 | 159 | 160 | ![](./assets/dms1.png) 161 | 162 | ### Create a Source endpoint for Oracle RDS 163 | 164 | Click **Create endpoint**. Enter the values as follows: 165 | 166 | |Parameter|Description| 167 | |-------------|--------------| 168 | |Endpoint type | Select **Source endpoint**| 169 | |Endpoint Identifier | Type a name, such as **`orasource`**| 170 | |Source Engine | oracle| 171 | |Server name | Enter the Oracle RDS DNS value shown in parent CloudFormation stack output| 172 | |Port | 1521| 173 | |Username | Enter as dbadmin| 174 | |Password| Enter oraadmin123| 175 | |SID| ORCL| 176 | 177 | > **_NOTE:_** You can also choose the corresponding RDS instance by clicking the option "Select RDS DB Instance". 178 | 179 | 180 | ![](./assets/dms2.png) 181 | 182 | Please leave the rest of the settings default. Make sure that the database name, port, and user information are correct. Click **Create endpoint**. 183 | 184 | 185 | After creating the endpoint, you should test the connection. Click on the endpoint and go to Connections Tab. Choose **Test connection** option. 186 | Choose the DMS instance created by the CloudFormation stack and click **Run Test**. The Status should return as successful after some time. 187 | 188 | 189 | ![](./assets/dms3.png) 190 | 191 | ### Create a Target endpoint for Aurora PostgreSQL 192 | 193 | Click **Create endpoint**. Enter the values as follows: 194 | 195 | |Parameter| Description| 196 | |------|------------- 197 | |Endpoint type | Select **Target endpoint**| 198 | |Endpoint Identifier | Type a name, such as **`aurtarget`**| 199 | |Target Engine | aurora-postgresql| 200 | |Server name | Enter the AuroraClusterEndpointName value shown in parent CloudFormation stack output| 201 | |Port | 5432| 202 | |Username | Enter as auradmin| 203 | |Password| Enter auradmin123| 204 | |Database Name| taxidb| 205 | 206 | > **_NOTE:_** You can also choose the corresponding Aurora instance by clicking the option "Select RDS DB Instance". 207 | 208 | ![](./assets/dms4.png) 209 | 210 | Please leave the rest of the settings default. Make sure that the Aurora cluster DNS, database name, port, and user information are correct. Click **Create endpoint**. 211 | 212 | After creating the endpoint, test the connection like you did earlier for the source endpoint. Click on the endpoint and go to Connections Tab. Choose **Test connection** option. 213 | Choose the DMS instance created by the CloudFormation stack and click Run Test. The Status should return as successful after some time. 214 | 215 | ![](./assets/dms5.png) 216 | 217 | ### Create a Target endpoint for Amazon DynamoDB 218 | 219 | Click **Create endpoint**. Enter the values as follows: 220 | 221 | |Parameter| Description| 222 | |------|---------------| 223 | |Endpoint type | Select **Target endpoint**| 224 | |Endpoint Identifier | Type a name, such as **`ddbtarget`**| 225 | |Target Engine | dynamodb| 226 | |Service access role ARN| Enter the IAM Role ARN (Note: Provide the value of DMSDDBRoleARN from CloudFormation Outputs section)| 227 | 228 | ![](./assets/dms6.png) 229 | 230 | Please leave the rest of the settings default. Make sure that the IAM Role ARN information is correct. Click **Create endpoint**. 231 | 232 | After creating the endpoint, you should **test the connection** as shown below. Choose the DMS instance created by the CloudFormation stack. 233 | 234 | ![](./assets/dms7.png) 235 | 236 | ## Migrate data from Oracle source to DynamoDB target 237 | 238 | ### Creating Replication Task for DynamoDB Migration 239 | 240 | Using an AWS DMS task, you can specify which schema to migrate and the type of migration. You can migrate existing data, migrate existing data and replicate ongoing changes, or replicate data changes only. In this lab, we will migrates existing data only. 241 | 242 | AWS DMS uses table-mapping rules to map data from the source to the target DynamoDB table. AWS DMS currently supports map-record-to-record and map-record-to-document as the only valid values for the rule-action parameter. For this lab, we will use map-record-to-record option to migrate trip data from Oracle to DynamoDB. Please refer to [DMS documentation](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.DynamoDB.html) for more details. 243 | 244 | 1. Open the [AWS DMS console](https://us-east-1.console.aws.amazon.com/dms/home?region=us-east-1), and choose **Database migration tasks** in the navigation pane. 245 | 246 | 2. Click **Create task** 247 | 248 | 3. Task creation includes multiple sections. Under Task Configuration, enter below. 249 | 250 | |Parameter| Description| 251 | |------|---------| 252 | |Task Identifier | Type a name, such as **`ora2ddb`**| 253 | |Replication Instance| Choose the DMS instance created by the CloudFormation stack| 254 | |Source database Endpoint| Choose orasource| 255 | |Target database Endpoint | Choose ddbtarget| 256 | |Migration Type| Choose Migrate existing data| 257 | 258 | > **_NOTE:_** Typical production migration involves full load followed by continuous data capture CDC. This can be achieved by using choosing option Migrate existing data and replicate ongoing changes. For this lab, we will go with full load only. 259 | 260 | 4. Uncheck the option **Start task on create**. We will be modifying the task after it's created to speed up the migration. 261 | 262 | ![](./assets/dms-task1-1.png) 263 | 264 | 265 | 5. Under Task settings , enter as below 266 | - Target Table preparation mode - Choose **Do Nothing** 267 | - Include LOB columns in replication - Choose **Limited LOB Mode** 268 | - Select **Enable CloudWatch Logs** 269 | 270 | ![](./assets/dms-task1-2.png) 271 | 272 | 6. Under Table Mapping section, enter as below: 273 | - choose **JSON Editor** and copy & paste the following mapping code. 274 | 275 | ```json 276 | { 277 | "rules": [ 278 | { 279 | "rule-type": "selection", 280 | "rule-id": "1", 281 | "rule-name": "1", 282 | "object-locator": { 283 | "schema-name": "TAXI", 284 | "table-name": "TRIPS" 285 | }, 286 | "rule-action": "include" 287 | }, 288 | { 289 | "rule-type": "object-mapping", 290 | "rule-id": "2", 291 | "rule-name": "2", 292 | "rule-action": "map-record-to-record", 293 | "object-locator": { 294 | "schema-name": "TAXI", 295 | "table-name": "TRIPS" 296 | }, 297 | "target-table-name": "aws-db-workshop-trips", 298 | "mapping-parameters": { 299 | "partition-key-name": "riderid", 300 | "sort-key-name": "tripinfo", 301 | "attribute-mappings": [{ 302 | "target-attribute-name": "riderid", 303 | "attribute-type": "scalar", 304 | "attribute-sub-type": "string", 305 | "value": "${RIDER_EMAIL}" 306 | }, 307 | { 308 | "target-attribute-name": "tripinfo", 309 | "attribute-type": "scalar", 310 | "attribute-sub-type": "string", 311 | "value": "${PICKUP_DATETIME},${ID}" 312 | }, 313 | { 314 | "target-attribute-name": "driverid", 315 | "attribute-type": "scalar", 316 | "attribute-sub-type": "string", 317 | "value": "${DRIVER_EMAIL}" 318 | }, 319 | { 320 | "target-attribute-name": "DriverDetails", 321 | "attribute-type": "scalar", 322 | "attribute-sub-type": "string", 323 | "value": "{\"Name\":\"${DRIVER_NAME}\",\"Vehicle Details\":{\"id\":\"${VEHICLE_ID}\",\"type\":\"${CAB_TYPE_ID}\"}}" 324 | }] 325 | } 326 | } 327 | ] 328 | } 329 | ``` 330 | 331 | 332 | > **_NOTE:_** As part of the migration task, we have created transformation rules to add few extra attributes from the original data. for e.g. RIDER_EMAIL as riderid (partition_key) and ID & PICKUP_DATETIME as tripinfo (sort key). This will ensure that our new design will able to uniquely identify the trip data by riderid. Also, we have combined many vehicle attributes and stored it as DriverDetails. In summary, this table stores all the rider and driver information in a de-normalized format. 333 | 334 | 335 | ![](./assets/dms-task1-3.png) 336 | 337 | 7. Do not modify anything in the Advanced settings. 338 | 339 | 8. Click **Create task**. Wait for the task status to change from *Creating* to *Ready* in the DMS console. 340 | 341 | ### Modify Replication Task for DynamoDB 342 | 343 | We will modify a few DMS level task settings (`ParallelLoadThreads` and `ParallelLoadBufferSize`) to speed up the migration from Oracle source to DynamoDB target and then manually start the task. 344 | 345 | 1. Execute the following command in Cloud9 terminal to set the DMS task ARN in a variable. 346 | 347 | ```shell script 348 | TASK_ARN=$(aws dms describe-replication-tasks --filters Name=replication-task-id,Values=ora2ddb | jq -r '.ReplicationTasks[].ReplicationTaskArn') 349 | ``` 350 | 351 | 2. Modify the DMS task settings by running the following in the Cloud9 terminal. 352 | 353 | ```shell script 354 | aws dms modify-replication-task --replication-task-arn $TASK_ARN --replication-task-settings '{"TargetMetadata":{"ParallelLoadThreads": 8,"ParallelLoadBufferSize": 50}}' 355 | ``` 356 | 357 | 3. Wait for the task status to change from *Modifying* to *Ready* in the DMS Console. 358 | 359 | 4. Start the DMS task by selecting it and choosing **Restart/Resume** from the **Actions** Menu. 360 | 361 | ![](./assets/ddb-start-task.png) 362 | 363 | ### Monitoring Replication Task for DynamoDB 364 | 365 | After task is started, monitor the task by looking at the console as shown below. You can also look at the CloudWatch logs for more information. 366 | 367 | ![](./assets/dms-task1-4.png) 368 | 369 | > **_NOTE:_** This task will take 2 to 3 minutes to complete. After the full load, you will see 128,714 Rows are migrated. 370 | 371 | ## Migrate data from Oracle source to Aurora PostgreSQL target 372 | 373 | ### Creating Replication Task for Aurora Migration 374 | 375 | Now, we will migrate four tables (Riders, Drivers, Payment and Billing) from RDS Oracle to Aurora PostgreSQL. We have already created those tables in Aurora PostgreSQL as part of the environment setup. 376 | 377 | 1. Open the [AWS DMS console](https://us-east-1.console.aws.amazon.com/dms/home?region=us-east-1), and choose **database migration tasks** in the navigation pane. 378 | 379 | 2. Click **Create task**. 380 | 381 | 3. Database Migration Task creation includes multiple sections. Under Task Configuration, enter below. 382 | 383 | |Parameter| Description| 384 | |------|----------------| 385 | |Task Identifier | Type a name, such as **`ora2aurora`**| 386 | |Replication Instance| Choose the DMS instance created by CloudFormation| 387 | |Source database Endpoint| Choose orasource| 388 | |Target database Endpoint | Choose aurtarget| 389 | |Migration Type| Choose Migrate existing data| 390 | 391 | 4. Click **Start task on create** 392 | 393 | ![](./assets/dms-task2-1.png) 394 | 395 | 5. Under Task settings , enter as below 396 | - Target Table preparation mode - Choose **Do Nothing** 397 | - Include LOB columns in replication - Choose **Limited LOB Mode** 398 | - Select **Enable validation** 399 | - Select **Enable CloudWatch Logs** 400 | 401 | ![](./assets/dms-task2-2.png) 402 | 403 | 6. Under Table Mapping section, enter as below: 404 | - choose **JSON Editor** and copy & paste the following mapping code. 405 | 406 | ```json 407 | { 408 | "rules": [ 409 | { 410 | "rule-type": "transformation", 411 | "rule-id": "1", 412 | "rule-name": "1", 413 | "rule-target": "schema", 414 | "object-locator": { 415 | "schema-name": "TAXI", 416 | "table-name": "%" 417 | }, 418 | "rule-action": "convert-lowercase" 419 | }, 420 | { 421 | "rule-type": "transformation", 422 | "rule-id": "2", 423 | "rule-name": "2", 424 | "rule-target": "table", 425 | "object-locator": { 426 | "schema-name": "TAXI", 427 | "table-name": "%" 428 | }, 429 | "rule-action": "convert-lowercase" 430 | }, 431 | { 432 | "rule-type": "transformation", 433 | "rule-id": "3", 434 | "rule-name": "3", 435 | "rule-target": "schema", 436 | "object-locator": { 437 | "schema-name": "TAXI" 438 | }, 439 | "value": "public", 440 | "rule-action": "rename" 441 | }, 442 | { 443 | "rule-type": "selection", 444 | "rule-id": "4", 445 | "rule-name": "4", 446 | "object-locator": { 447 | "schema-name": "TAXI", 448 | "table-name": "DRIVERS" 449 | }, 450 | "rule-action": "include" 451 | }, 452 | { 453 | "rule-type": "selection", 454 | "rule-id": "5", 455 | "rule-name": "5", 456 | "object-locator": { 457 | "schema-name": "TAXI", 458 | "table-name": "RIDERS" 459 | }, 460 | "rule-action": "include" 461 | }, 462 | { 463 | "rule-type": "selection", 464 | "rule-id": "6", 465 | "rule-name": "6", 466 | "object-locator": { 467 | "schema-name": "TAXI", 468 | "table-name": "BILLING" 469 | }, 470 | "rule-action": "include" 471 | }, 472 | { 473 | "rule-type": "selection", 474 | "rule-id": "7", 475 | "rule-name": "7", 476 | "object-locator": { 477 | "schema-name": "TAXI", 478 | "table-name": "PAYMENT" 479 | }, 480 | "rule-action": "include" 481 | }, 482 | { 483 | "rule-type": "transformation", 484 | "rule-id": "8", 485 | "rule-name": "8", 486 | "rule-target": "column", 487 | "object-locator": { 488 | "schema-name": "TAXI", 489 | "table-name": "%", 490 | "column-name": "%" 491 | }, 492 | "rule-action": "convert-lowercase" 493 | } 494 | ] 495 | } 496 | ``` 497 | 498 | > **_NOTE:_** As part of the migration task, we have created the above mapping rule to include tables that are related to Payment and Billing use case only. DMS provides rich set of selection and transformation rules for migration (e.g. selecting specific tables, remove column, define primary key etc.). For this lab, we will convert the source schema, table and column names to lower case, and rename the schema owner from taxi to public in Aurora PostgreSQL database. Please refer to [DMS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.CustomizingTasks.TableMapping.html) documentation to learn more. 499 | 500 | 501 | ![](./assets/dms-task2-3.png) 502 | 503 | 7. Do not modify anything in the Advanced settings. 504 | 505 | 8. Click **Create task**. The task will begin immediately. If the task is not started, start the task manually (Select the task, Click Actions and then Start/Restart task). 506 | 507 | ![](./assets/dms-task2-4.png) 508 | 509 | ### Monitoring Replication Task for Aurora PostgreSQL 510 | 511 | 1. Go to **Data Migration tasks** and Click the task name and look at the **Table Statistics** tab. 512 | 513 | ![](./assets/dms-task-2-5.png) 514 | 515 | 2. You can also review the replication task details in CloudWatch Logs. 516 | 517 | ![](./assets/dms-task2-6.png) 518 | 519 | 520 | ## Final Validation of DMS Tasks 521 | 522 | Please check if both the DMS tasks are completed. You will see the below output. 523 | 524 | 1. **ora2ddb** task status as "Load Completed". TRIPS table full load row count as 128,714 525 | 526 | 2. **ora2aur** task status as "Load Completed". Table and Full load count should be: DRIVERS (Count-100,001), PAYMENT (Count-60,001), BILLING (Count -60,001), RIDERS (Count-100,000) 527 | 528 | **Congrats!!** You have successfully completed Lab 1. Now you can proceed to [Lab 2](../lab2-TaxiBookingAndPayments/). -------------------------------------------------------------------------------- /lab1-TaxiDataMigration/assets/cfn-lab1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-rds-purpose-built-workshop/be574ce7c00e5badd8d96e457d17900bb58a0897/lab1-TaxiDataMigration/assets/cfn-lab1.png -------------------------------------------------------------------------------- /lab1-TaxiDataMigration/assets/cfn6.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-rds-purpose-built-workshop/be574ce7c00e5badd8d96e457d17900bb58a0897/lab1-TaxiDataMigration/assets/cfn6.png -------------------------------------------------------------------------------- /lab1-TaxiDataMigration/assets/cloud9-1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-rds-purpose-built-workshop/be574ce7c00e5badd8d96e457d17900bb58a0897/lab1-TaxiDataMigration/assets/cloud9-1.png -------------------------------------------------------------------------------- /lab1-TaxiDataMigration/assets/cloud9-2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-rds-purpose-built-workshop/be574ce7c00e5badd8d96e457d17900bb58a0897/lab1-TaxiDataMigration/assets/cloud9-2.png -------------------------------------------------------------------------------- /lab1-TaxiDataMigration/assets/ddb-start-task.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-rds-purpose-built-workshop/be574ce7c00e5badd8d96e457d17900bb58a0897/lab1-TaxiDataMigration/assets/ddb-start-task.png -------------------------------------------------------------------------------- /lab1-TaxiDataMigration/assets/ddb.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-rds-purpose-built-workshop/be574ce7c00e5badd8d96e457d17900bb58a0897/lab1-TaxiDataMigration/assets/ddb.png -------------------------------------------------------------------------------- /lab1-TaxiDataMigration/assets/dms-task-2-5.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-rds-purpose-built-workshop/be574ce7c00e5badd8d96e457d17900bb58a0897/lab1-TaxiDataMigration/assets/dms-task-2-5.png -------------------------------------------------------------------------------- /lab1-TaxiDataMigration/assets/dms-task1-1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-rds-purpose-built-workshop/be574ce7c00e5badd8d96e457d17900bb58a0897/lab1-TaxiDataMigration/assets/dms-task1-1.png -------------------------------------------------------------------------------- /lab1-TaxiDataMigration/assets/dms-task1-2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-rds-purpose-built-workshop/be574ce7c00e5badd8d96e457d17900bb58a0897/lab1-TaxiDataMigration/assets/dms-task1-2.png -------------------------------------------------------------------------------- /lab1-TaxiDataMigration/assets/dms-task1-3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-rds-purpose-built-workshop/be574ce7c00e5badd8d96e457d17900bb58a0897/lab1-TaxiDataMigration/assets/dms-task1-3.png -------------------------------------------------------------------------------- /lab1-TaxiDataMigration/assets/dms-task1-4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-rds-purpose-built-workshop/be574ce7c00e5badd8d96e457d17900bb58a0897/lab1-TaxiDataMigration/assets/dms-task1-4.png -------------------------------------------------------------------------------- /lab1-TaxiDataMigration/assets/dms-task2-1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-rds-purpose-built-workshop/be574ce7c00e5badd8d96e457d17900bb58a0897/lab1-TaxiDataMigration/assets/dms-task2-1.png -------------------------------------------------------------------------------- /lab1-TaxiDataMigration/assets/dms-task2-2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-rds-purpose-built-workshop/be574ce7c00e5badd8d96e457d17900bb58a0897/lab1-TaxiDataMigration/assets/dms-task2-2.png -------------------------------------------------------------------------------- /lab1-TaxiDataMigration/assets/dms-task2-3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-rds-purpose-built-workshop/be574ce7c00e5badd8d96e457d17900bb58a0897/lab1-TaxiDataMigration/assets/dms-task2-3.png -------------------------------------------------------------------------------- /lab1-TaxiDataMigration/assets/dms-task2-4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-rds-purpose-built-workshop/be574ce7c00e5badd8d96e457d17900bb58a0897/lab1-TaxiDataMigration/assets/dms-task2-4.png -------------------------------------------------------------------------------- /lab1-TaxiDataMigration/assets/dms-task2-6.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-rds-purpose-built-workshop/be574ce7c00e5badd8d96e457d17900bb58a0897/lab1-TaxiDataMigration/assets/dms-task2-6.png -------------------------------------------------------------------------------- /lab1-TaxiDataMigration/assets/dms1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-rds-purpose-built-workshop/be574ce7c00e5badd8d96e457d17900bb58a0897/lab1-TaxiDataMigration/assets/dms1.png -------------------------------------------------------------------------------- /lab1-TaxiDataMigration/assets/dms2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-rds-purpose-built-workshop/be574ce7c00e5badd8d96e457d17900bb58a0897/lab1-TaxiDataMigration/assets/dms2.png -------------------------------------------------------------------------------- /lab1-TaxiDataMigration/assets/dms3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-rds-purpose-built-workshop/be574ce7c00e5badd8d96e457d17900bb58a0897/lab1-TaxiDataMigration/assets/dms3.png -------------------------------------------------------------------------------- /lab1-TaxiDataMigration/assets/dms4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-rds-purpose-built-workshop/be574ce7c00e5badd8d96e457d17900bb58a0897/lab1-TaxiDataMigration/assets/dms4.png -------------------------------------------------------------------------------- /lab1-TaxiDataMigration/assets/dms5.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-rds-purpose-built-workshop/be574ce7c00e5badd8d96e457d17900bb58a0897/lab1-TaxiDataMigration/assets/dms5.png -------------------------------------------------------------------------------- /lab1-TaxiDataMigration/assets/dms6.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-rds-purpose-built-workshop/be574ce7c00e5badd8d96e457d17900bb58a0897/lab1-TaxiDataMigration/assets/dms6.png -------------------------------------------------------------------------------- /lab1-TaxiDataMigration/assets/dms7.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-rds-purpose-built-workshop/be574ce7c00e5badd8d96e457d17900bb58a0897/lab1-TaxiDataMigration/assets/dms7.png -------------------------------------------------------------------------------- /lab1-TaxiDataMigration/assets/lab1-arch.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-rds-purpose-built-workshop/be574ce7c00e5badd8d96e457d17900bb58a0897/lab1-TaxiDataMigration/assets/lab1-arch.png -------------------------------------------------------------------------------- /lab1-TaxiDataMigration/assets/oracle-taxi-schema.txt: -------------------------------------------------------------------------------- 1 | SELECT OBJECT_TYPE, Count(*) FROM DBA_OBJECTS WHERE OWNER IN ('TAXI') 2 | GROUP BY owner,object_type; 3 | 4 | OBJECT_TYPE COUNT(*) 5 | TRIGGER 3 6 | LOB 2 7 | TABLE 11 8 | SEQUENCE 5 9 | INDEX 17 10 | 11 | 12 | 13 | SELECT table_name,num_rows FROM dba_tables WHERE owner IN ('TAXI'); 14 | TABLE_NAME NUM_ROWS 15 | RIDERS 100000 16 | PAYMENT 60001 17 | CAB_TYPES 4 18 | CAR_MODEL 5 19 | PAYMENT_TYPES 7 20 | TRIP_STATUS 4 21 | RATE_CODES 7 22 | DRIVERS 100001 23 | TRIPS 128714 24 | awsdms_validation_failures_v1 0 25 | BILLING 60001 26 | 27 | 28 | SQL> desc taxi.trips 29 | Name Null? Type 30 | ----------------------------------------- -------- ---------------------------- 31 | ID NOT NULL NUMBER(10) 32 | RIDER_ID NUMBER(19) 33 | DRIVER_ID NUMBER 34 | RIDER_NAME VARCHAR2(100) 35 | RIDER_MOBILE VARCHAR2(100) 36 | RIDER_EMAIL VARCHAR2(100) 37 | DRIVER_NAME VARCHAR2(100) 38 | DRIVER_EMAIL VARCHAR2(100) 39 | DRIVER_MOBILE VARCHAR2(100) 40 | VEHICLE_ID VARCHAR2(50) 41 | CAB_TYPE_ID NUMBER 42 | VENDOR_ID NUMBER 43 | PICKUP_DATETIME TIMESTAMP(6) 44 | DROPOFF_DATETIME TIMESTAMP(6) 45 | STORE_AND_FWD_FLAG VARCHAR2(100) 46 | RATE_CODE_ID NUMBER 47 | PICKUP_LONGITUDE NUMBER 48 | PICKUP_LATITUDE NUMBER 49 | DROPOFF_LONGITUDE NUMBER 50 | DROPOFF_LATITUDE NUMBER 51 | PASSENGER_COUNT NUMBER 52 | TRIP_DISTANCE NUMBER 53 | FARE_AMOUNT NUMBER 54 | EXTRA NUMBER 55 | MTA_TAX NUMBER 56 | TIP_AMOUNT NUMBER 57 | TOLLS_AMOUNT NUMBER 58 | EHAIL_FEE NUMBER 59 | IMPROVEMENT_SURCHARGE NUMBER 60 | TOTAL_AMOUNT NUMBER 61 | PAYMENT_TYPET NUMBER 62 | TRIP_TYPE NUMBER 63 | PICKUP_LOCATION_ID NUMBER 64 | DROPOFF_LOCATION_ID NUMBER 65 | STATUS VARCHAR2(100) 66 | 67 | 68 | 69 | 70 | SQL> desc taxi.billing; 71 | Name Null? Type 72 | ----------------------------------------- -------- ---------------------------- 73 | ID NOT NULL NUMBER(10) 74 | DRIVER_ID NUMBER(10) 75 | BILLING_CYCLE NUMBER(6) 76 | BILLING_START TIMESTAMP(6) 77 | BILLING_END TIMESTAMP(6) 78 | BILLING_DATE TIMESTAMP(6) 79 | BILLING_AMOUNT NUMBER 80 | COMMISSIONS NUMBER 81 | DESCRIPTION VARCHAR2(500) 82 | RIDES_TOTAL NUMBER(6) 83 | BILLING_STATUS VARCHAR2(100) 84 | 85 | SQL> desc taxi.payment; 86 | Name Null? Type 87 | ----------------------------------------- -------- ---------------------------- 88 | ID NOT NULL NUMBER(10) 89 | BILLING_ID NUMBER(10) 90 | DRIVER_ID NUMBER(10) 91 | BILLING_CYCLE NUMBER(6) 92 | PAYMENT_AMOUNT NUMBER 93 | PAYMENT_DATE TIMESTAMP(6) 94 | PAYMENT_ID NUMBER(6) 95 | PAYMENT_STATUS VARCHAR2(100) 96 | DESCRIPTION VARCHAR2(500) 97 | 98 | 99 | 100 | 101 | SQL> desc taxi.riders; 102 | Name Null? Type 103 | ----------------------------------------- -------- ---------------------------- 104 | RIDER_ID NOT NULL NUMBER(10) 105 | RIDER_NAME NOT NULL VARCHAR2(100) 106 | RIDER_EMAIL NOT NULL VARCHAR2(100) 107 | CREATED_ON NOT NULL TIMESTAMP(6) 108 | PAYMENT_ID NOT NULL NUMBER(19) 109 | RIDER_MOBILE NOT NULL VARCHAR2(100) 110 | ADDRESS VARCHAR2(1000) 111 | RATING VARCHAR2(10) 112 | PROFILE VARCHAR2(1000) 113 | 114 | SQL> desc taxi.drivers; 115 | Name Null? Type 116 | ----------------------------------------- -------- ---------------------------- 117 | DRIVER_ID NOT NULL NUMBER(10) 118 | DRIVER_NAME NOT NULL VARCHAR2(100) 119 | VEHICLE_ID NOT NULL VARCHAR2(50) 120 | MAKE NUMBER(19) 121 | DRIVER_EMAIL NOT NULL VARCHAR2(100) 122 | CREATED_ON NOT NULL TIMESTAMP(6) 123 | DRIVER_MOBILE NOT NULL VARCHAR2(100) 124 | PAYMENT_ID NOT NULL NUMBER(10) 125 | ADDRESS VARCHAR2(1000) 126 | RATING VARCHAR2(10) 127 | PROFILE VARCHAR2(1000) 128 | 129 | 130 | -------------------------------------------------------------------------------- /lab1-TaxiDataMigration/assets/taxi-data-model.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-rds-purpose-built-workshop/be574ce7c00e5badd8d96e457d17900bb58a0897/lab1-TaxiDataMigration/assets/taxi-data-model.png -------------------------------------------------------------------------------- /lab2-TaxiBookingAndPayments/README.md: -------------------------------------------------------------------------------- 1 | # Lab 2: Simulate Taxi Booking, Billing and Payments using Amazon DynamoDB, DynamoDB Streams, AWS Lambda and Amazon Aurora PostgreSQL 2 | 3 | - [Overview](#overview) 4 | - [Prerequisites](#prerequisites) 5 | - [Setup the AWS Cloud 9 Environment](#setup-the-aws-cloud-9-environment) 6 | * [Update SAM CLI](#update-sam-cli) 7 | * [Install AWS SDK for Python](#install-aws-sdk-for-python) 8 | * [Save CloudFormation Stack name as a variable](#save-cloudformation-stack-name-as-a-variable) 9 | - [Enable Amazon DynamoDB Streams](#enable-amazon-dynamodb-streams) 10 | - [Deploying AWS Lambda Function](#deploying-aws-lambda-function) 11 | * [Packaging the PG8000 binaries](#packaging-the-pg8000-binaries) 12 | * [Deploy AWS Lambda Function and AWS Lambda Layer using AWS SAM template](#deploy-aws-lambda-function-and-aws-lambda-layer-using-aws-sam-template) 13 | - [Taxi Ride Workflow](#taxi-ride-workflow) 14 | * [Taxi Trip Booking Workflow](#taxi-trip-booking-workflow) 15 | * [Driver Billing and Payments](#driver-billing-and-payments) 16 | 17 | ## Overview 18 | 19 | In this lab, you will simulate taxi trip booking by a rider and acceptance by a driver followed by billing and payment using Python scripts and SQL commands. You will utilize DynamoDB streams and AWS lambda functions to insert completed trip data from DynamoDB to Aurora PostgreSQL. 20 | 21 | ![architecture.png](./assets/architecture.png) 22 | 23 | ## Prerequisites 24 | 25 | 1. You should have completed [Lab1](https://github.com/aws-samples/amazon-rds-purpose-built-workshop/tree/master/lab1-TaxiDataMigration) from the [github repository](https://github.com/aws-samples/amazon-rds-purpose-built-workshop). 26 | 27 | ## Setup the AWS Cloud 9 Environment 28 | 29 | ### Update SAM CLI 30 | 31 | 1. Open the AWS Management Console for [AWS Cloud9](https://us-east-1.console.aws.amazon.com/cloud9/home/account). You will leverage AWS Cloud9 IDE throughout this lab for running scripts, deploying AWS SAM (Serverless Application Model) templates, executing SQL queries etc. 32 | 2. Click on __Open IDE__ for the AWS Cloud9 IDE that was created as part of the Amazon CloudFormation template that was deployed. 33 | 3. Open a terminal window in the AWS Cloud9 IDE by clicking on __Window__ from the menu bar on the top and select __New Terminal__. 34 | 4. Update AWS SAM CLI to the latest version by running the following commands in the Cloud9 terminal window. 35 | 36 | ```shell script 37 | cd ~/environment 38 | pip install --user --upgrade awscli aws-sam-cli 39 | sam --version 40 | ``` 41 | > Note: Ensure that the SAM CLI version is 0.21.0 or above. 42 | 43 | ### Install AWS SDK for Python 44 | 45 | - To install [Boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html?id=docs_gateway) (AWS SDK for Python) copy paste the following commands in the terminal window in the AWS Cloud9 IDE 46 | 47 | ```shell script 48 | cd ~/environment 49 | curl -O https://bootstrap.pypa.io/get-pip.py # Get the install script. 50 | sudo python3 get-pip.py # Install pip. 51 | pip3 install boto3 --user 52 | ``` 53 | 54 | ### Save CloudFormation Stack name as a variable 55 | 56 | 1. Open the AWS Management Console for CloudFormation from [here](https://us-east-1.console.aws.amazon.com/cloudformation/home?region=us-east-1). 57 | 2. In the upper-right corner of the AWS Management Console, confirm you are in the US East (N. Virginia) Region. 58 | 3. Click on __Stacks__ in the right navigation pane. 59 | 4. Under __Stacks__ copy the parent CloudFormation stack name starting with **mod-** (e.g. _mod-aa8afde9acf04c7f_). 60 | 5. Substitute the string (_substitute-name-of-copied-cf-stack-name_) in the command below with the name of the CloudFormation stack and run it in the Cloud9 terminal window. 61 | 62 | ```shell script 63 | AWSDBWORKSHOP_CFSTACK_NAME="substitute-name-of-copied-cf-stack-name" 64 | ``` 65 | 66 | 6. Copy and paste the following commands in the Cloud9 terminal window to set the environment variable _$AWSDBWORKSHOP_CFSTACK_NAME_ . 67 | 68 | ```shell script 69 | echo "export AWSDBWORKSHOP_CFSTACK_NAME=${AWSDBWORKSHOP_CFSTACK_NAME}" >> ~/.bash_profile 70 | . ~/.bash_profile 71 | echo $AWSDBWORKSHOP_CFSTACK_NAME 72 | ``` 73 | 74 | > Note: Ensure that the name of the CloudFormation stack printed in the terminal window (e.g. _mod-aa8afde9acf04c7f_) matches with the name of your parent CloudFormation stack name. 75 | 76 | ## Enable Amazon DynamoDB Streams 77 | In this section, you will enable Amazon DynamoDB stream for the Amazon DynamoDB table named _'aws-db-workshop-trips'_ that was created as part of the CloudFormation stack. 78 | 79 | - Copy and paste the commands below in the Cloud9 terminal window to enable streams for the Amazon DynamoDB Tables named _'aws-db-workshop-trips'_ 80 | ```shell script 81 | STREAM_ID=$(aws dynamodb update-table --table-name aws-db-workshop-trips --stream-specification StreamEnabled=true,StreamViewType=NEW_AND_OLD_IMAGES | jq '.TableDescription.LatestStreamArn' | cut -d'/' -f4) 82 | STREAM_NAME=stream/${STREAM_ID::-1} 83 | echo "export AWSDBWORKSHOP_DDB_STREAM_NAME=${STREAM_NAME}" >> ~/.bash_profile 84 | . ~/.bash_profile 85 | echo $AWSDBWORKSHOP_DDB_STREAM_NAME 86 | ``` 87 | 88 | > Note: The output should be similar to the following 89 | >``` 90 | >stream/2019-09-18T20:18:33.343 91 | >``` 92 | 93 | Now that you have enabled the Amazon DynamoDB stream, the next step is to deploy the AWS Lambda function that will process records from the stream. 94 | 95 | ## Deploying AWS Lambda Function 96 | In this section, you will be using AWS Serverless Application Model ([SAM]((https://aws.amazon.com/serverless/sam/))) CLI to deploy a Lambda Function within the same Amazon Virtual Private Network([VPC](https://aws.amazon.com/vpc/)). The SAM deployment will also include a Python interface to the PostgreSQL database engine as an AWS Lambda Layer. This Lambda function will read the taxi trip information from DynamoDB streams as they are inserted/updated in the DynamoDB table **_aws-db-workshop-trips_**. Only when a trip is complete (denoted by the _STATUS_ attribute in the trip item/record), the Lambda function will insert information into **_trips_** table in Aurora PostgreSQL. 97 | 98 | ### Packaging the PG8000 binaries 99 | 100 | In this section, you will download and package the binaries for [PG8000](https://pypi.org/project/pg8000/) - a Python interface to PostgreSQL. The package will be deployed as an AWS Lambda Layer. 101 | 102 | - Copy and paste the below commands in the Cloud9 terminal window. 103 | 104 | ```shell script 105 | cd ~/environment 106 | mkdir pglayer 107 | virtualenv -p python3 pglayer 108 | cd pglayer 109 | source bin/activate 110 | mkdir -p pg8000-layer/python 111 | pip install pg8000 -t pg8000-layer/python 112 | cd pg8000-layer 113 | zip -r pg8000-layer.zip python 114 | mkdir ~/environment/amazon-rds-purpose-built-workshop/src/ddb-stream-processor/dependencies/ 115 | cp ~/environment/pglayer/pg8000-layer/pg8000-layer.zip ~/environment/amazon-rds-purpose-built-workshop/src/ddb-stream-processor/dependencies/ 116 | ``` 117 | 118 | ### Deploy AWS Lambda Function and AWS Lambda Layer using AWS SAM template 119 | In this section, you will validate the SAM template that contains the configuration for the Lambda function and the Lambda Layer. 120 | 121 | 122 | 1. To validate the SAM template, copy and paste the commands below in the terminal window 123 | 124 | ```shell script 125 | cd ~/environment/amazon-rds-purpose-built-workshop/src/ddb-stream-processor 126 | sam validate 127 | ``` 128 | 129 | > Note: The terminal window output should display the following: 130 | > ``` 131 | > /home/ec2-user/environment/amazon-rds-purpose-built-workshop/src/ddb-stream-processor/template.yaml is a valid SAM Template 132 | >``` 133 | 134 | 2. To package the AWS SAM application, copy and paste the commands below in the Cloud9 terminal window. This will create a _template-out.yaml_ file is the same folder and will upload the packaged binaries to the specified Amazon S3 bucket. 135 | 136 | ```shell script 137 | cd ~/environment/amazon-rds-purpose-built-workshop/src/ddb-stream-processor 138 | S3_BUCKETNAME=$(aws cloudformation describe-stacks --stack-name $AWSDBWORKSHOP_CFSTACK_NAME | jq -r '.Stacks[].Outputs[] | select(.OutputKey=="S3bucketName") | .OutputValue') 139 | echo $S3_BUCKETNAME 140 | sam package --output-template-file template-out.yaml --s3-bucket $S3_BUCKETNAME 141 | ``` 142 | 143 | 3. [_Optional_] Please take some time to review _template-out.yaml_. 144 | 145 | 4. Copy and paste the command below to ensure that the packages have been uploaded successfully to the Amazon S3 bucket. 146 | 147 | ```shell script 148 | aws s3 ls s3://$S3_BUCKETNAME 149 | ``` 150 | 151 | > Sample Output: 152 | > ``` 153 | > 019-09-15 16:39:56 70451 14b63970e9437bf82ea16664d46a929e 154 | > 2019-09-15 16:39:56 71954 d3eec91527b02d78de30ae42198cd0c0 155 | > ``` 156 | 157 | 5. Set the variables from the output of the Amazon CloudFormation template that was deployed. 158 | 159 | > Note: You can copy the entire script block below and paste in the Cloud9 terminal window. Press Enter for the final command to execute. 160 | 161 | ```shell script 162 | cd ~/environment/amazon-rds-purpose-built-workshop/src/ddb-stream-processor 163 | 164 | AURORADB_NAME=$(aws cloudformation describe-stacks --stack-name $AWSDBWORKSHOP_CFSTACK_NAME | jq -r '.Stacks[].Outputs[] | select(.OutputKey=="AuroraDBName") | .OutputValue') 165 | echo $AURORADB_NAME 166 | AURORACLUSTERENDPOINT_NAME=$(aws cloudformation describe-stacks --stack-name $AWSDBWORKSHOP_CFSTACK_NAME | jq -r '.Stacks[].Outputs[] | select(.OutputKey=="AuroraClusterEndpointName") | .OutputValue') 167 | echo $AURORACLUSTERENDPOINT_NAME 168 | AURORADBMASTERUSER_NAME=$(aws cloudformation describe-stacks --stack-name $AWSDBWORKSHOP_CFSTACK_NAME | jq -r '.Stacks[].Outputs[] | select(.OutputKey=="AuroraDBMasterUser") | .OutputValue') 169 | echo $AURORADBMASTERUSER_NAME 170 | LAMBDASECURITYGROUP_ID=$(aws cloudformation describe-stacks --stack-name $AWSDBWORKSHOP_CFSTACK_NAME | jq -r '.Stacks[].Outputs[] | select(.OutputKey=="LambdaSecurityGroupId") | .OutputValue') 171 | echo $LAMBDASECURITYGROUP_ID 172 | LAMBDASUBNET1_ID=$(aws cloudformation describe-stacks --stack-name $AWSDBWORKSHOP_CFSTACK_NAME | jq -r '.Stacks[].Outputs[] | select(.OutputKey=="LambdaSubnet1") | .OutputValue') 173 | LAMBDASUBNET2_ID=$(aws cloudformation describe-stacks --stack-name $AWSDBWORKSHOP_CFSTACK_NAME | jq -r '.Stacks[].Outputs[] | select(.OutputKey=="LambdaSubnet2") | .OutputValue') 174 | echo $LAMBDASUBNET1_ID,$LAMBDASUBNET2_ID 175 | ``` 176 | 177 | 6. Copy and paste the following command to deploy the Lambda Function along with the Lambda Layer. 178 | 179 | ```shell script 180 | sam deploy --template-file template-out.yaml --capabilities CAPABILITY_IAM --stack-name SAM-AWSDBWorkshop2019 --parameter-overrides LambdaLayerNameParameter=aws-db-workshop-pg8000-layer DDBStreamName=$AWSDBWORKSHOP_DDB_STREAM_NAME SecurityGroupIds=$LAMBDASECURITYGROUP_ID VpcSubnetIds=$LAMBDASUBNET1_ID,$LAMBDASUBNET2_ID DatabaseName=$AURORADB_NAME DatabaseHostName=$AURORACLUSTERENDPOINT_NAME DatabaseUserName=$AURORADBMASTERUSER_NAME DatabasePassword=auradmin123 181 | ``` 182 | >Note: This will take a few minutes. Ensure that the SAM template was successfully deployed. Look for the following line in the terminal as output 183 | >``` 184 | >Successfully created/updated stack - SAM-AWSDBWorkshop2019 in None 185 | >``` 186 | 187 | 7. Finally deactivate the virtual environment. 188 | 189 | ```shell script 190 | deactivate 191 | ``` 192 | 193 | Now you have successfully deployed the Lambda function. 194 | 195 | 196 | ## Taxi Ride Workflow 197 | In this section, you will run Python scripts to simulate booking of a taxi trip by a rider followed by acceptance and completion of the trip by a driver. After the trip is complete, you will run back-end SQL queries to process billing and driver payments. 198 | 199 | ### Taxi Trip Booking Workflow 200 | 201 | 1. Copy and paste the following commands to book a new trip as a rider. 202 | 203 | ```shell script 204 | cd ~/environment/amazon-rds-purpose-built-workshop/src/ddb-python-script/ 205 | python3 rider-book-trip.py 206 | ``` 207 | 208 | From the output of the script make a note of the _tripinfo_ value. You will be entering this value when prompted by the subsequent scripts. It's a randomly generated string that uniquely identifies a trip. It will be similar to 209 | 210 | >``` 211 | >"tripinfo": "2020-04-05T18:52:42Z,8987397" 212 | >``` 213 | 214 | 2. Copy and paste the following command as a driver to accept the trip. The script will prompt for the 'tripinfo' value. Enter the value (without double quotes for e.g. 2020-04-05T18:52:42Z,8987397) from the output of the previous Python script you just ran as a rider to book a new trip. 215 | 216 | ```shell script 217 | python3 driver-accept-trip.py 218 | ``` 219 | 220 | 3. Copy and paste the following command as a driver to complete the trip. The script will prompt for the 'tripinfo' value. Enter the same 'tripinfo' value (without double quotes for e.g. 2020-04-05T18:52:42Z,8987397) that you provided as input to the previous Python script you just ran as a driver to accept the trip. 221 | 222 | ```shell script 223 | python3 driver-complete-trip.py 224 | ``` 225 | 226 | >Note: The trip status is **_COMPLETE_** now and so the Lambda function would have picked the record from DynamoDB and inserted into Aurora PostgreSQL. 227 | 228 | ### Driver Billing and Payments 229 | 230 | 1. Copy and paste the commands below to connect to the Aurora PostgreSQL database. Enter the password string for the Aurora database when prompted. 231 | 232 | ```shell script 233 | AURORADB_NAME=$(aws cloudformation describe-stacks --stack-name $AWSDBWORKSHOP_CFSTACK_NAME | jq -r '.Stacks[].Outputs[] | select(.OutputKey=="AuroraDBName") | .OutputValue') 234 | echo $AURORADB_NAME 235 | AURORACLUSTERENDPOINT_NAME=$(aws cloudformation describe-stacks --stack-name $AWSDBWORKSHOP_CFSTACK_NAME | jq -r '.Stacks[].Outputs[] | select(.OutputKey=="AuroraClusterEndpointName") | .OutputValue') 236 | echo $AURORACLUSTERENDPOINT_NAME 237 | AURORADBMASTERUSER_NAME=$(aws cloudformation describe-stacks --stack-name $AWSDBWORKSHOP_CFSTACK_NAME | jq -r '.Stacks[].Outputs[] | select(.OutputKey=="AuroraDBMasterUser") | .OutputValue') 238 | echo $AURORADBMASTERUSER_NAME 239 | 240 | sudo psql -h $AURORACLUSTERENDPOINT_NAME -U $AURORADBMASTERUSER_NAME -d $AURORADB_NAME 241 | ``` 242 | 243 | 2. Execute the query below to review the trip information in the _trips_ table that you just completed in the previous section. 244 | 245 | ```sql 246 | select * from trips; 247 | ``` 248 | 249 | > The output of the query should have at-least 1 row in the trips table, similar to the output below. 250 | >``` 251 | > id | rider_id | driver_id | rider_name | rider_mobile | rider_email | trip_info | driver_name | driver_email | driver_mobile | vehicle_id | cab_type_id | vendor_id | pickup_datetime | dropoff_datetime | store_and_fwd_flag | rate_code_id | pickup_longitude | pickup_latitude | dropoff_longitude | dropoff_latitude | pa 252 | > ssenger_count | trip_distance | fare_amount | extra | mta_tax | tip_amount | tolls_amount | ehail_fee | improvement_surcharge | total_amount | payment_type | trip_type | pickup_location_id | dropoff_location_id | status 253 | > ---------+----------+-----------+-------------+--------------+-------------------------+-------------------------------------+--------------+-----------------------+---------------+------------+-------------+-----------+---------------------+---------------------+--------------------+--------------+------------------+-----------------+-------------------+------------------+--- 254 | > --------------+---------------+-------------+-------+---------+------------+--------------+-----------+-----------------------+--------------+--------------+-----------+--------------------+---------------------+----------- 255 | > 2000001 | 69257 | 528204 | person69257 | +11609467790 | person69257@example.com | 2019-12-18T05:15:33.640038Z,3219350 | driver528204 | driver528204@taxi.com | +11185992795 | PXX248130 | 2 | 2 | 2019-12-18 05:15:33 | 2019-12-18 05:19:10 | N | 4 | -73.496113 | 40.664146 | -73.527485 | 40.665024 | 256 | > 3 | 32 | 142.96 | 0.3 | 0.4 | 4.92 | 4.4 | 0 | 0.3 | 14.18 | 3 | 2 | 0 | 0 | Completed 257 | > 2000002 | 69257 | 507977 | person69257 | +11609467790 | person69257@example.com | 2019-12-18T05:31:13.478619Z,1747531 | driver507977 | driver507977@taxi.com | +11088418780 | XVJ356159 | 2 | 2 | 2019-12-18 05:31:13 | 2019-12-18 05:31:57 | N | 3 | -73.401165 | 40.866392 | -73.065379 | 40.96106 | 258 | > 4 | 8 | 55.39 | 1.0 | 0.4 | 8.57 | 2.25 | 0 | 0.8 | 127.75 | 3 | 2 | 0 | 0 | Completed 259 | > (2 rows) 260 | >``` 261 | 262 | 3. Execute the query below to insert driver billing information for all the drivers, for the current daily billing cycle based on the trip information in the _trips_ table. 263 | 264 | ```sql 265 | insert into billing (driver_id, billing_cycle, billing_start, billing_end, billing_amount, commissions, rides_total, description, billing_status) 266 | select driver_id, 2, current_date, current_date+1, sum(total_amount), 0.8, count(*), 'billing cycle 2', 'completed' from trips 267 | where dropoff_datetime < current_date+1 and dropoff_datetime > current_date 268 | group by driver_id; 269 | ``` 270 | 271 | > The query should insert at-least 1 row into the billing table and the output should be similar to the following. 272 | >``` 273 | > INSERT 0 1 274 | >``` 275 | 276 | 4. Execute the query below to review the billing information that you just inserted. 277 | 278 | ```sql 279 | select * from billing where billing_cycle=2; 280 | ``` 281 | 282 | > The output of the query should retrieve at-least 1 row from the billing table, similar to the output below. 283 | >``` 284 | > id | driver_id | billing_cycle | billing_start | billing_end | billing_date | billing_amount | commissions | description | rides_total | billing_status 285 | > --------+-----------+---------------+---------------------+---------------------+----------------------------+----------------+-------------+-----------------+-------------+---------------- 286 | > 200001 | 510909 | 2 | 2019-09-15 00:00:00 | 2019-09-16 00:00:00 | 2019-09-16 01:59:05.634323 | 42.26 | 0.8 | billing cycle 2 | 1 | completed 287 | > (1 row) 288 | >``` 289 | 290 | 5. Execute the query below to insert driver payment information for all the drivers for the current billing cycle based on the billing information in the _billing_ table. 291 | 292 | ```sql 293 | insert into payment(billing_id,driver_id,billing_cycle,payment_amount,payment_date, payment_id, payment_status,description) select a.id, a.driver_id, a.billing_cycle,sum(a.billing_amount*a.commissions),a.billing_date, b.payment_id, 'completed','Payment cycle Jan 2020' 294 | from billing a, drivers b where a.driver_id=b.driver_id and a.billing_cycle = 2 and a.billing_status = 'completed' group by a.id, a.driver_id,b.payment_id, a.billing_cycle, a.billing_date; 295 | ``` 296 | 297 | > The query should insert at-least 1 row into the payment table and the output should be similar to the output below. 298 | >``` 299 | > INSERT 0 1 300 | >``` 301 | 302 | 6. Execute the query below to review the payment information that you just inserted. 303 | 304 | ```sql 305 | select * from payment where description='Payment cycle Jan 2020'; 306 | ``` 307 | 308 | > The output of the query should retrieve at-least 1 row from the payment table, similar to the output below. 309 | >``` 310 | > id | billing_id | driver_id | billing_cycle | payment_amount | payment_date | payment_id | payment_status | description 311 | > --------+------------+-----------+---------------+----------------+----------------------------+------------+----------------+------------------------ 312 | > 200001 | 200001 | 510909 | 2 | 33.808 | 2019-09-16 01:59:05.634323 | 7 | completed | Payment cycle Jan 2020 313 | > (1 row) 314 | >``` 315 | 316 | 7. Execute the following command to close the database connection. 317 | 318 | ```sql 319 | \q 320 | ``` 321 | 322 | **Congrats!!** You have successfully completed Lab 2. Now you can proceed to [Lab 3](../lab3-AthenaFederatedQuery/). -------------------------------------------------------------------------------- /lab2-TaxiBookingAndPayments/assets/architecture.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-rds-purpose-built-workshop/be574ce7c00e5badd8d96e457d17900bb58a0897/lab2-TaxiBookingAndPayments/assets/architecture.png -------------------------------------------------------------------------------- /lab2-TaxiBookingAndPayments/assets/asset0.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-rds-purpose-built-workshop/be574ce7c00e5badd8d96e457d17900bb58a0897/lab2-TaxiBookingAndPayments/assets/asset0.txt -------------------------------------------------------------------------------- /lab3-AthenaFederatedQuery/README.md: -------------------------------------------------------------------------------- 1 | # Lab 3: Leverage Athena federated query feature (in preview) to query Amazon DynamoDB, Aurora PostgreSQL and AWS S3 2 | 3 | - [Overview](#overview) 4 | - [Prerequisites](#prerequisites) 5 | - [Preparing the Environment](#preparing-the-environment) 6 | * [Setup environment variables on Cloud9 terminal window](#setup-environment-variables-on-cloud9-terminal-window) 7 | * [Create a copy of trips table for federated query](#create-a-copy-of-trips-table-for-federated-query) 8 | * [Create a AmazonAthenaPreviewFunctionality workgroup](#create-a-amazonathenapreviewfunctionality-workgroup) 9 | * [Switch to the AmazonAthenaPreviewFunctionality workgroup](#switch-to-the-amazonathenapreviewfunctionality-workgroup) 10 | - [Setup Athena Connectors and Catalogs](#setup-athena-connectors-and-catalogs) 11 | * [Setting up Amazon DynamoDB Connector](#setting-up-amazon-dynamodb-connector) 12 | * [Setting up catalog for querying DynamoDB](#setting-up-catalog-for-querying-dynamodb) 13 | * [Setting up JDBC connector for Aurora PostgreSQL database](#setting-up-jdbc-connector-for-aurora-postgresql-database) 14 | * [Setting up catalog for querying Aurora PostgreSQL database](#setting-up-catalog-for-querying-aurora-postgresql-database) 15 | - [Query multiple data sources using Athena Federated Query](#query-multiple-data-sources-using-athena-federated-query) 16 | * [Use Case 1 - Querying data from Amazon DynamoDB and Amazon Aurora](#use-case-1---querying-data-from-amazon-dynamodb-and-amazon-aurora) 17 | * [Use Case 2 - Querying data from Amazon DynamoDB and Amazon S3](#use-case-2---querying-data-from-amazon-dynamodb-and-amazon-s3) 18 | * [Use Case 3 - Querying data from Amazon DynamoDB and partitioned data in Amazon S3](#use-case-3---querying-data-from-amazon-dynamodb-and-partitioned-data-in-amazon-s3) 19 | 20 | ## Overview 21 | 22 | **Federated query** (in preview) is a new Amazon Athena feature that enables data analysts, engineers, and data scientists to execute SQL queries across data stored in relational, non-relational, object, and custom data sources. In this lab, we have used purpose built databases to store trip data in Amazon DynamoDB and billing/payment related data in Amazon Aurora PostgreSQL. Running analytics on data spread across various data stores can be complex and time consuming. Typically, to analyze such data across data stores, you need to learn new programming languages, data access constructs, and build complex pipelines to extract, transform and load into a data warehouse before you can easily query the data. Data pipelines introduce delays and require custom processes to validate data accuracy and consistency across systems. Federated queries in Athena eliminate this complexity by allowing customers to query data in-place wherever it resides. Analysts can use familiar SQL constructs to JOIN data across multiple data sources for quick analysis or use scheduled SQL queries to extract and store results in Amazon S3 for subsequent analysis. 23 | 24 | In this lab, we will leverage this feature to query data stored in Amazon DynamoDB, Amazon Aurora PostgreSQL and AWS S3 using a single SQL query. Athena executes federated queries using Data Source Connectors that run on [AWS Lambda](http://aws.amazon.com/lambda). 25 | 26 | :warning: **We don't recommend to onboard your production workload to Athena Federated Query (Preview) yet. Query performance may vary between the preview Workgroup and the other workgroups in your account. Additionally, AWS may add new features and bug fixes to the preview Workgroup that may not be backwards compatible.** 27 | 28 | ## Prerequisites 29 | 30 | 1. Use **Chrome browser** to do this lab as we have noticed some issues with Firefox while doing this Lab. 31 | 2. You should have completed both [Lab1](https://github.com/aws-samples/amazon-rds-purpose-built-workshop/tree/master/lab1-TaxiDataMigration) and [Lab2](https://github.com/aws-samples/amazon-rds-purpose-built-workshop/tree/master/lab2-TaxiBookingAndPayments) from the [github repository.](https://github.com/aws-samples/amazon-rds-purpose-built-workshop) 32 | 33 | ## Preparing the Environment 34 | 35 | ### Setup environment variables on Cloud9 terminal window 36 | 37 | 1) Go to Cloud9 IDE terminal window and set the following environment variables if not done already as part of Lab2. Substitute the string (_substitute-name-of-copied-cf-stack-name_) in the command below with the name of the Amazon CloudFormation parent stack starting with **mod-** (e.g. _mod-aa8afde9acf04c7f_). 38 | 39 | ```shell script 40 | AWSDBWORKSHOP_CFSTACK_NAME="substitute-name-of-copied-cf-stack-name" 41 | AURORADB_NAME=$(aws cloudformation describe-stacks --stack-name $AWSDBWORKSHOP_CFSTACK_NAME | jq -r '.Stacks[].Outputs[] | select(.OutputKey=="AuroraDBName") | .OutputValue') 42 | echo $AURORADB_NAME 43 | AURORACLUSTERENDPOINT_NAME=$(aws cloudformation describe-stacks --stack-name $AWSDBWORKSHOP_CFSTACK_NAME | jq -r '.Stacks[].Outputs[] | select(.OutputKey=="AuroraClusterEndpointName") | .OutputValue') 44 | echo $AURORACLUSTERENDPOINT_NAME 45 | AURORADBMASTERUSER_NAME=$(aws cloudformation describe-stacks --stack-name $AWSDBWORKSHOP_CFSTACK_NAME | jq -r '.Stacks[].Outputs[] | select(.OutputKey=="AuroraDBMasterUser") | .OutputValue') 46 | echo $AURORADBMASTERUSER_NAME 47 | ``` 48 | 49 | 2) Connect to the Aurora PostgreSQL cluster using the below command. Enter _auradmin123_ as password when prompted. 50 | 51 | ```shell script 52 | psql -h $AURORACLUSTERENDPOINT_NAME -U $AURORADBMASTERUSER_NAME -d $AURORADB_NAME 53 | ``` 54 | 55 | ### Create a copy of trips table for federated query 56 | 57 | Run the below SQL Command to create a copy of the trip table in Aurora PostgreSQL without 'pickup_datetime' and 'dropoff_datetime' _timestamp without time zone_ fields. This is to get around a [known bug](https://github.com/awslabs/aws-athena-query-federation/issues/21) associated with JDBC connector used by Athena federated query, which was still open as of authoring this Lab. 58 | 59 | ```shell script 60 | create table trips_query 61 | as select id,rider_id, driver_id, 62 | rider_name, 63 | rider_mobile, 64 | rider_email, 65 | trip_info, 66 | driver_name, 67 | driver_email, 68 | driver_mobile, 69 | vehicle_id, 70 | cab_type_id, 71 | vendor_id, 72 | store_and_fwd_flag, 73 | rate_code_id, 74 | pickup_longitude, 75 | pickup_latitude, 76 | dropoff_longitude, 77 | dropoff_latitude, 78 | passenger_count, 79 | trip_distance, 80 | fare_amount, 81 | extra, 82 | mta_tax, 83 | tip_amount, 84 | tolls_amount, 85 | ehail_fee, 86 | improvement_surcharge, 87 | total_amount, 88 | payment_type, 89 | trip_type, 90 | pickup_location_id, 91 | dropoff_location_id, 92 | status 93 | from trips; 94 | select * from trips_query; 95 | 96 | select rider_email, trip_info from trips_query; 97 | ``` 98 | 99 | ### Create a AmazonAthenaPreviewFunctionality workgroup 100 | 101 | All Athena queries originating from the Workgroup _AmazonAthenaPreviewFunctionality_ will be considered preview test queries. 102 | 103 | 1. Open the[AWS Management Console for Athena](https://console.aws.amazon.com/athena/home?region=us-east-1). 104 | 2. If this is your first time visiting the AWS Management Console for Athena, you will get a Getting Started page. Choose **Get Started** to open the Query Editor. If this isn't your first time, the AthenaQuery Editor opens. 105 | 3. Select **Workgroup:primary** tab and then click **Create workgroup** as shown below. 106 | 4. Provide the following values. Leave the rest to default. 107 | 108 | 1. Specify the Workgroup name as **AmazonAthenaPreviewFunctionality**. 109 | 2. Specify the S3 bucket name with a **s3://** prefix and **/** suffix that was created as part of the CloudFormation Stack (Look for S3bucketName in the parent CloudFormation stack Outputs section). For e.g. _s3://mod-aa8afde9acf04c7f-dbworkshops3bucket-12vvoqlrar5b3/_ 110 | 111 | 5. Click create workgroup. 112 | 113 | ![workgroup.png](./assets/workgroup.png) 114 | 115 | ### Switch to the AmazonAthenaPreviewFunctionality workgroup 116 | 117 | In the **Workgroups** panel, choose the **AmazonAthenaPreviewFunctionality** workgroup and then choose **Switch workgroup**. You will be redirected back to the Athena console. Choose **Get Started** to open the Query Editor again. 118 | 119 | ![switch.png](./assets/switch.png) 120 | 121 | Now, you are ready to deploy the Athena connectors in your AWS account. To learn more about Athena connectors, please see documentation [connect-to-a-data-source-lambda.html](https://docs.aws.amazon.com/athena/latest/ug/connect-to-a-data-source-lambda.html). 122 | 123 | ## Setup Athena Connectors and Catalogs 124 | 125 | ### Setting up Amazon DynamoDB Connector 126 | 127 | This connector enables Amazon Athena to communicate with DynamoDB, making the _trips_ tables accessible via SQL. For more information about the connector usage, parameters and limitations, refer to [athena-dynamodb](https://github.com/awslabs/aws-athena-query-federation/tree/master/athena-dynamodb) documentation. 128 | 129 | 1. Ensure that current AWS region is **US East (N Virginia)** 130 | 2. Click on **data sources** tab 131 | 3. Click **Connect data source** 132 | 4. Select **Query a data source (Beta)** 133 | 5. Under choose a data source option, select **Amazon DynamoDB** 134 | 135 | ![datasource.png](./assets/datasource.png) 136 | 137 | 6. Click **Next** 138 | 7. In the **Connection details: Amazon DynamoDB** panel, Click **Configure new function** 139 | 140 | ![ddb.png](./assets/ddb.png) 141 | 142 | 8. You will be be taken to AWS Lambda home page where the connector will be deployed as a SAM Application. Provide values for the following parameters and leave the rest to the default. 143 | 144 | **SpillBucket**: Specify the S3 bucket name that was created as part of the CloudFormation Stack for e.g. mod-aa8afde9acf04c7f-dbworkshops3bucket-1511cfk17lzed 145 | 146 | **AthenaCatalogName**: taxiddb 147 | 148 | Select the option **_I acknowledge that this app creates custom IAM Roles_** and Click **Deploy**. 149 | 150 | ![ddbconn.png](./assets/ddbconn.png) 151 | 152 | >**Note:** It will take a few minutes to deploy. After successful deployment, you can see the Lambda function deployed in your account as shown below. 153 | 154 | ![lambda.png](./assets/lambda.png) 155 | 156 | ### Setting up catalog for querying DynamoDB 157 | 158 | In this step, we will create a catalog named _ddbcatalog_ as shown below. 159 | 160 | 1. Go Back previous Athena **Data sources** window in your browser and on the **Connection details: Amazon DynamoDB** panel, click the refresh button next to the **Lambda function** input. 161 | 2. Choose the Lambda function **taxiddb** and provide a catalog name as **ddbcatalog**. 162 | 3. (Optional) Type a description for the Connector for e.g. _catalog to query DDB table via SQL_ 163 | 4. Click Connect 164 | 165 | ![catalog.png](./assets/catalog.png) 166 | 167 | Now, we need to repeat the same process to setup Athena JDBC connector for Aurora PostgreSQL. 168 | 169 | ### Setting up JDBC connector for Aurora PostgreSQL database 170 | 171 | This connector enables Amazon Athena to access your Amazon RDS and Amazon Aurora databases using JDBC driver. For more information about the connector usage, parameters and limitations, refer to [athena-jdbc](https://github.com/awslabs/aws-athena-query-federation/tree/master/athena-jdbc) documentation. 172 | 173 | 1. Ensure that current AWS region is **US East (N Virginia)** 174 | 2. Click on **data sources** tab 175 | 3. Click **Connect data source** 176 | 4. Select **Query a data source (Beta)** 177 | 5. Under choose a data source option, select **PostgreSQL** 178 | 179 | ![pgdatasource.png](./assets/pgdatasource.png) 180 | 181 | 6. Click Next 182 | 7. In the **Connection details: PostgreSQL** panel, Click **Configure new function** 183 | 184 | ![pgconn.png](./assets/pgconn.png) 185 | 186 | 8. You will be taken to AWS Lambda home page where the connector will be deployed as a SAM Application. Provide values for the following parameters and leave the rest to the default. 187 | 188 | **SecretNamePrefix** : dbadmin 189 | 190 | **SpillBucket**: Specify the S3 bucket name that was created as part of the CloudFormation Stack for e.g. mod-aa8afde9acf04c7f-dbworkshops3bucket-1511cfk17lzed 191 | 192 | **DefaultConnectionString** : postgres://\?user=auradmin&password=auradmin123 for e.g. postgres://jdbc:postgresql://akolab-auroracluster-qlkwnb51f0ir.cluster-ckxdapvbefiz.us-east-1.rds.amazonaws.com:5432/taxidb?user=auradmin&password=auradmin123 193 | 194 | **LambdaFunctionName** : taxirdb 195 | 196 | **SecurityGroupIds** : specify the LambdaSecurityGroupId from the outputs of CloudFormation stack 197 | 198 | **SubnetIds** : specify the LambdaSubnet1,LambdaSubnet2 (separated by commas) from the output of CloudFormation stack 199 | 200 | Select the option **_I acknowledge that this app creates custom IAM Roles_** and Click **Deploy**. 201 | 202 | >**Note:** The JDBC connector can connect to database using credentials stored in AWS Secrets manager or directly by specifying an userid and password. For this lab, we will specify the userid and password directly in the connection string. We have provided a dummy value as a secretname prefix as this parameter seems to be mandatory. 203 | 204 | ![sam.png](./assets/sam.png) 205 | 206 | >**Note:** It will take a few minutes to deploy. After successful deployment, you can see the Lambda function deployed in your account as shown below. 207 | 208 | ![lambdafn.png](./assets/lambdafn.png) 209 | 210 | ### Setting up catalog for querying Aurora PostgreSQL database 211 | 212 | In this step, we will create a catalog named rdbcatalog as shown below. 213 | 214 | 1. Go Back previous Athena **Data sources** window in your browser and on the **Connection details: PostgreSQL** panel, click the refresh button next to the **Lambda function** input. 215 | 2. Choose the Lambda function **taxirdb** and provide a catalog name as **rdbcatalog**. 216 | 3. (Optional) Type a description for the Connector 217 | 4. Click Connect 218 | 219 | Now, we are ready to query both DynamoDB and Aurora PostgreSQL using Athena federated query. 220 | 221 | ## Query multiple data sources using Athena Federated Query 222 | 223 | ### Use Case 1 - Querying data from Amazon DynamoDB and Amazon Aurora 224 | 225 | In this use case, we will validate data accuracy and consistency for the trip record created as part of Lab 2. 226 | 227 | 1. Click **Query Editor** tab. 228 | 2. In the Athena Query Editor panel, click the banner at the top showing **_set up query result location in Amazon S3_**. 229 | 230 | ![athenaqryloc.png](./assets/athenaqryloc.png) 231 | 232 | 3. Specify the S3 bucket location in the format s3://\/athena/ and click save. For e.g. s://mod-aa8afde9acf04c7f-dbworkshops3bucket-1511cfk17lzed/athena/ 233 | 234 | ![s3bucketloc.png](./assets/s3bucketloc.png) 235 | 236 | 4. Now you can start entering your query in the query pane. Run some sample queries on the Aurora PostgreSQL database. For Aurora, you can directly use the lambda function name as a catalog name. 237 | 238 | 239 | ```shell script 240 | /* query to list the databases */ 241 | show databases in `lambda:taxirdb`; 242 | 243 | /* query to list the tables */ 244 | show tables in `lambda:taxirdb`.public 245 | 246 | /* query to view the trip records */ 247 | select * from "lambda:taxirdb".public.trips_query 248 | ``` 249 | 250 | >**Note:** If the **Run Query** button is disabled, click Query Editor tab again or refresh the window. 251 | 252 | 5. Run some sample queries on DynamoDB either using the catalog name or directly referring the lambda function. 253 | 254 | ```shell script 255 | select * from ddbcatalog.default."aws-db-workshop-trips" where riderid='person69156@example.com' 256 | select * from "lambda:taxiddb".default."aws-db-workshop-trips" where riderid='person69156@example.com' 257 | ``` 258 | 259 | 6. Run the below query which joins (Inner join) the trip data from Aurora PostgreSQL and DynamoDB tables. We have used the _riderid_ attribute from DynamoDB to join with _rider\_email_ in trips_query table. _trips\_info_ field is used as an additional join condition. The purpose of the query is to check data consistency of trip data between the two data stores. 260 | 261 | ```shell script 262 | SELECT ddb.riderid,ddb.tripinfo , ddb.fare_amount "DDB-Fareamount", rdb.fare_amount "RDB-Fareamount", ddb.tolls_amount "DDB-Tollsamount", rdb.tolls_amount "RDB-Tollsamount", ddb.passenger_count "DDB-passenger_count", rdb.passenger_count "RDB-passenger_count", ddb.tip_amount "DDB-Tipamount", rdb.tip_amount "RDB-Tipamount", ddb.total_amount "DDB-Totalamount", rdb.total_amount "RDB-Totalamount" 263 | FROM 264 | ddbcatalog.default."aws-db-workshop-trips" ddb, 265 | "lambda:taxirdb".public.trips_query rdb 266 | where 267 | ddb.riderid=rdb.rider_email 268 | and ddb.tripinfo=rdb.trip_info 269 | ``` 270 | 271 | ### Use Case 2 - Querying data from Amazon DynamoDB and Amazon S3 272 | 273 | In this example, we will perform an adhoc analytics to get the trends on total rides,number of rides per vendor, along with the average fair amount for Green taxi rides. We will use trip data from the [NY taxi public dataset](https://registry.opendata.aws/nyc-tlc-trip-records-pds/) for this illustration. 274 | 275 | 1. Choose the datasource as "**awsdatacatalog**" and create an external table to refer the already saved NY taxi dataset in AWS S3. 276 | 277 | ```shell script 278 | CREATE DATABASE athenataxidb; 279 | 280 | CREATE EXTERNAL TABLE IF NOT EXISTS taxigreen ( 281 | VendorID STRING, 282 | tpep_pickup_datetime TIMESTAMP, 283 | tpep_dropoff_datetime TIMESTAMP, 284 | passenger_count INT, 285 | trip_distance DOUBLE, 286 | pickup_longitude DOUBLE, 287 | pickup_latitude DOUBLE, 288 | RatecodeID INT, 289 | store_and_fwd_flag STRING, 290 | dropoff_longitude DOUBLE, 291 | dropoff_latitude DOUBLE, 292 | payment_type INT, 293 | fare_amount DOUBLE, 294 | extra DOUBLE, 295 | mta_tax DOUBLE, 296 | tip_amount DOUBLE, 297 | tolls_amount DOUBLE, 298 | improvement_surcharge DOUBLE, 299 | total_amount DOUBLE 300 | ) 301 | ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' 302 | STORED AS TEXTFILE 303 | LOCATION 's3://us-east-1.data-analytics/NYC-Pub/green/' 304 | ``` 305 | 306 | 2 . Now you can query both historical as well as current dataset in DynamoDB using federated query. The below query shows the trend on riders and average amount between two data sets for each vendor. 307 | 308 | >**Note:** If you get an internal service error, please run the query again. 309 | 310 | ```shell script 311 | with s3history as (SELECT 312 | CASE vendorid 313 | WHEN '1' THEN 1 314 | WHEN '2' THEN 2 315 | ELSE 2 END AS Vendor, 316 | COUNT(1) as RideCount, 317 | sum(total_amount) as TotalAmount, 318 | avg(total_amount) as AverageAmount 319 | FROM Taxigreen 320 | WHERE total_amount > 0 321 | GROUP BY (1)), 322 | ddbcurrent as ( SELECT 323 | vendor_id, 324 | count(*) as RideCount, 325 | sum(total_amount) as TotalAmount, 326 | avg(total_amount) as AverageAmount 327 | FROM ddbcatalog.default."aws-db-workshop-trips" ddb 328 | where ddb.total_amount > 0 329 | GROUP BY vendor_id) 330 | SELECT s3history.Vendor, 331 | ddbcurrent.TotalAmount, s3history.TotalAmount "Hist-TotalAmount", 332 | ddbcurrent.AverageAmount, s3history.AverageAmount "Hist-AverageAmount", 333 | ddbcurrent.RideCount, s3history.RideCount "Hist-RideCount" 334 | FROM s3history 335 | JOIN ddbcurrent ON s3history.vendor = ddbcurrent.vendor_id; 336 | ``` 337 | 338 | ### Use Case 3 - Querying data from Amazon DynamoDB and partitioned data in Amazon S3 339 | 340 | By partitioning your data, you can restrict the amount of data scanned by each query, thus improving performance and reducing cost. Athena leverages Hive for [partitioning](https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-AlterPartition) data. You can partition your data by any key. A common practice is to partition the data based on time, often leading to a multi-level partitioning scheme. For example, a customer who has data coming in every hour might decide to partition by year, month, date, and hour. Another customer, who has data coming from many different sources but loaded one time per day, may partition by a data source identifier and date. 341 | 342 | 1. In the query pane, paste the following statement to create the NYTaxiRides table, and then choose **Run Query**. 343 | 344 | ```shell script 345 | CREATE EXTERNAL TABLE NYTaxiRides ( 346 | vendorid STRING, 347 | pickup_datetime TIMESTAMP, 348 | dropoff_datetime TIMESTAMP, 349 | ratecode INT, 350 | passenger_count INT, 351 | trip_distance DOUBLE, 352 | fare_amount DOUBLE, 353 | total_amount DOUBLE, 354 | payment_type INT 355 | ) 356 | PARTITIONED BY (YEAR INT, MONTH INT, TYPE string) 357 | STORED AS PARQUET 358 | LOCATION 's3://us-east-1.data-analytics/canonical/NY-Pub'; 359 | 360 | -- Load partitions 361 | 362 | MSCK REPAIR TABLE nytaxirides; 363 | ``` 364 | 365 | This will create a table partitioned by year,month and type. The files are already created and stored in parquet format in AWS S3. 366 | 367 | 2. Now we will perform month-wise comparison of 2016 DynamoDB data with previous year (2015) data stored in AWS S3. This is helpful in a scenario where you wish to perform trend analysis on hot data stored in DynamoDB and cold data stored in AWS S3. 368 | 369 | >**Note:** If you get an internal service error please run the query again. 370 | 371 | ```shell script 372 | with s3history AS 373 | (SELECT YEAR, 374 | month, 375 | CASE vendorid 376 | WHEN '1' THEN 377 | 1 378 | WHEN '2' THEN 379 | 2 380 | ELSE 2 381 | END AS Vendor, COUNT(1) AS RideCount, sum(total_amount) AS TotalAmount, avg(total_amount) AS AverageAmount 382 | FROM NYTaxiRides 383 | WHERE total_amount > 0 384 | GROUP BY (1),(2),(3)), ddbdata AS 385 | (SELECT vendor_id, 386 | extract(year 387 | FROM date_parse(pickup_datetime,'%Y-%m-%dT%H:%i:%sZ')) AS year, extract(month 388 | FROM date_parse(pickup_datetime,'%Y-%m-%dT%H:%i:%sZ')) AS month,total_amount 389 | FROM ddbcatalog.default."aws-db-workshop-trips" ddb 390 | WHERE ddb.total_amount > 0 ), ddbcurrent AS 391 | (SELECT year, 392 | month, 393 | vendor_id,count(1) as ridecount, 394 | sum(total_amount) AS TotalAmount, 395 | avg(total_amount) AS AverageAmount 396 | FROM ddbdata 397 | GROUP BY year,month,vendor_id) 398 | SELECT s3history.YEAR,s3history.month, 399 | s3history.Vendor, 400 | ddbcurrent.TotalAmount, 401 | s3history.TotalAmount "Hist-TotalAmount", 402 | ddbcurrent.AverageAmount, 403 | s3history.AverageAmount "Hist-AverageAmount", 404 | ddbcurrent.RideCount, 405 | s3history.RideCount "Hist-RideCount" 406 | FROM s3history 407 | JOIN ddbcurrent 408 | ON s3history.vendor = ddbcurrent.vendor_id 409 | AND s3history.month=ddbcurrent.month 410 | WHERE s3history.YEAR = 2015 411 | and ddbcurrent.year=2016; 412 | ``` 413 | 414 | 415 | **Congrats!!** You have successfully completed Lab 3. This concludes the workshop Labs. -------------------------------------------------------------------------------- /lab3-AthenaFederatedQuery/assets/athenaqryloc.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-rds-purpose-built-workshop/be574ce7c00e5badd8d96e457d17900bb58a0897/lab3-AthenaFederatedQuery/assets/athenaqryloc.png -------------------------------------------------------------------------------- /lab3-AthenaFederatedQuery/assets/catalog.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-rds-purpose-built-workshop/be574ce7c00e5badd8d96e457d17900bb58a0897/lab3-AthenaFederatedQuery/assets/catalog.png -------------------------------------------------------------------------------- /lab3-AthenaFederatedQuery/assets/datasource.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-rds-purpose-built-workshop/be574ce7c00e5badd8d96e457d17900bb58a0897/lab3-AthenaFederatedQuery/assets/datasource.png -------------------------------------------------------------------------------- /lab3-AthenaFederatedQuery/assets/ddb.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-rds-purpose-built-workshop/be574ce7c00e5badd8d96e457d17900bb58a0897/lab3-AthenaFederatedQuery/assets/ddb.png -------------------------------------------------------------------------------- /lab3-AthenaFederatedQuery/assets/ddbconn.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-rds-purpose-built-workshop/be574ce7c00e5badd8d96e457d17900bb58a0897/lab3-AthenaFederatedQuery/assets/ddbconn.png -------------------------------------------------------------------------------- /lab3-AthenaFederatedQuery/assets/lambda.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-rds-purpose-built-workshop/be574ce7c00e5badd8d96e457d17900bb58a0897/lab3-AthenaFederatedQuery/assets/lambda.png -------------------------------------------------------------------------------- /lab3-AthenaFederatedQuery/assets/lambdafn.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-rds-purpose-built-workshop/be574ce7c00e5badd8d96e457d17900bb58a0897/lab3-AthenaFederatedQuery/assets/lambdafn.png -------------------------------------------------------------------------------- /lab3-AthenaFederatedQuery/assets/pgconn.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-rds-purpose-built-workshop/be574ce7c00e5badd8d96e457d17900bb58a0897/lab3-AthenaFederatedQuery/assets/pgconn.png -------------------------------------------------------------------------------- /lab3-AthenaFederatedQuery/assets/pgdatasource.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-rds-purpose-built-workshop/be574ce7c00e5badd8d96e457d17900bb58a0897/lab3-AthenaFederatedQuery/assets/pgdatasource.png -------------------------------------------------------------------------------- /lab3-AthenaFederatedQuery/assets/s3bucketloc.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-rds-purpose-built-workshop/be574ce7c00e5badd8d96e457d17900bb58a0897/lab3-AthenaFederatedQuery/assets/s3bucketloc.png -------------------------------------------------------------------------------- /lab3-AthenaFederatedQuery/assets/sam.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-rds-purpose-built-workshop/be574ce7c00e5badd8d96e457d17900bb58a0897/lab3-AthenaFederatedQuery/assets/sam.png -------------------------------------------------------------------------------- /lab3-AthenaFederatedQuery/assets/switch.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-rds-purpose-built-workshop/be574ce7c00e5badd8d96e457d17900bb58a0897/lab3-AthenaFederatedQuery/assets/switch.png -------------------------------------------------------------------------------- /lab3-AthenaFederatedQuery/assets/workgroup.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-rds-purpose-built-workshop/be574ce7c00e5badd8d96e457d17900bb58a0897/lab3-AthenaFederatedQuery/assets/workgroup.png -------------------------------------------------------------------------------- /src/cloudformation.template: -------------------------------------------------------------------------------- 1 | { 2 | "AWSTemplateFormatVersion" : "2010-09-09", 3 | 4 | "Description" : "This CloudFormation sample template Oracle_DynamoDB_Aurora_PostgreSQL_DMS creates an Oracle and Aurora PostgreSQL RDS instances in a VPC which can be used to test the database migration using AWS DMS service. It also creates a DynamoDB table and a Cloud9 environment. You will be billed for the AWS resources used if you create a stack from this template", 5 | 6 | "Parameters" : { 7 | 8 | "OracleDBName": { 9 | "Default": "ORCL", 10 | "Description" : "Enter Oracle Database name", 11 | "Type": "String", 12 | "MinLength": "4", 13 | "MaxLength": "63", 14 | "AllowedPattern" : "[a-zA-Z][a-zA-Z0-9]*", 15 | "ConstraintDescription" : "must begin with a letter and contain a minimum of 4 alphanumeric characters." 16 | }, 17 | 18 | "OracleDBPassword": { 19 | "Default": "oraadmin123", 20 | "NoEcho": "true", 21 | "Description" : "Enter password for the oracle admin user: dbadmin", 22 | "Type": "String", 23 | "MinLength": "8", 24 | "MaxLength": "41", 25 | "AllowedPattern" : "[a-zA-Z0-9]*", 26 | "ConstraintDescription" : "must contain only alphanumeric characters with minimum of 8 characters." 27 | }, 28 | 29 | 30 | "OracleDBStorage": { 31 | "Default": "100", 32 | "NoEcho": "false", 33 | "Description" : "Enter storage for Oracle DB in GB", 34 | "Type": "Number", 35 | "MaxValue": "6000", 36 | "MinValue": "100", 37 | "ConstraintDescription" : "must contain only numberic and min 100gb less than 6000 GB." 38 | }, 39 | 40 | 41 | "OracleInstanceType":{ 42 | "Description":"Oracle DB instance type", 43 | "Type":"String", 44 | "Default":"db.t3.medium", 45 | "AllowedValues":[ 46 | "db.t3.medium", 47 | "db.t3.large", 48 | "db.r5.large", 49 | "db.m5.large"], 50 | "ConstraintDescription":"must be a valid Oracle instance type." 51 | }, 52 | 53 | 54 | "AuroraDBName": { 55 | "Default": "taxidb", 56 | "Description" : "Enter Aurora Database name", 57 | "Type": "String", 58 | "MinLength": "4", 59 | "MaxLength": "63", 60 | "AllowedPattern" : "[a-zA-Z][a-zA-Z0-9]*", 61 | "ConstraintDescription" : "must begin with a letter and contain a minimum of 4 alphanumeric characters." 62 | }, 63 | 64 | 65 | "AuroraDBUsername": { 66 | "Default": "auradmin", 67 | "NoEcho": "false", 68 | "Description" : "Enter Master admin username for Aurora RDS", 69 | "Type": "String", 70 | "MinLength": "4", 71 | "MaxLength": "16", 72 | "AllowedPattern" : "[a-zA-Z][a-zA-Z0-9]*", 73 | "ConstraintDescription" : "must begin with a letter and contain a minimum of 4 alphanumeric characters." 74 | }, 75 | 76 | "AuroraDBPassword": { 77 | "Default": "auradmin123", 78 | "NoEcho": "true", 79 | "Description" : "Enter password for Aurora RDS Admin user", 80 | "Type": "String", 81 | "MinLength": "8", 82 | "MaxLength": "41", 83 | "AllowedPattern" : "[a-zA-Z0-9]*", 84 | "ConstraintDescription" : "must contain only alphanumeric characters with minimum of 8 characters." 85 | }, 86 | 87 | 88 | "AuroraInstanceType":{ 89 | "Description":"Aurora PostgreSQL DB instance type", 90 | "Type":"String", 91 | "Default":"db.r5.large", 92 | "AllowedValues":[ 93 | "db.t3.medium", 94 | "db.r5.large", 95 | "db.r5.xlarge" 96 | ], 97 | "ConstraintDescription":"must be a valid Aurora PostgreSQL instance types." 98 | }, 99 | 100 | "AuroraEngineType": { 101 | "Description": "Aurora DB Engine type", 102 | "Type": "String", 103 | "Default": "aurora-postgresql", 104 | "AllowedValues": [ 105 | "aurora-postgresql" 106 | 107 | ], 108 | "ConstraintDescription": "must be a valid Aurora DB engine types." 109 | }, 110 | 111 | 112 | "DMSInstanceType":{ 113 | "Description":"DMS Replication instance type", 114 | "Type":"String", 115 | "Default":"dms.t2.medium", 116 | "AllowedValues":[ 117 | "dms.t2.medium", 118 | "dms.t2.large", 119 | "dms.c4.large", 120 | "dms.c4.xlarge" 121 | ], 122 | "ConstraintDescription":"must be a valid DMS Replication instance types." 123 | }, 124 | 125 | "ExistsDMSRole": { 126 | "Default": "N", 127 | "Description": "check If the dms-vpc-role exists in your account, please enter Y, else enter N", 128 | "Type": "String", 129 | "MinLength": "1", 130 | "MaxLength": "1", 131 | "AllowedPattern" : "[YN]", 132 | "ConstraintDescription" : "Permitted value is Y or N." 133 | }, 134 | 135 | 136 | "ClientIP" : { 137 | "Description" : "The IP address range that can be used to connect to the RDS instances from your local machine.It must be a valid IP CIDR range of the form x.x.x.x/x.Pls get your address using checkip.amazonaws.com or whatsmyip.org", 138 | "Type": "String", 139 | "MinLength": "9", 140 | "MaxLength": "18", 141 | "Default": "0.0.0.0/0", 142 | "AllowedPattern": "(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\\d{1,2})", 143 | "ConstraintDescription": "It must be a valid IP CIDR range of the form x.x.x.x/x. Suggest to enable access to your IP address only. Pls get your address using checkip.amazonaws.com or whatsmyip.org." 144 | } 145 | 146 | }, 147 | 148 | "Metadata" : { 149 | "AWS::CloudFormation::Interface" : { 150 | "ParameterGroups" : [ 151 | { 152 | "Label" : { "default" : "Source Oracle Database Configuration" }, 153 | "Parameters" : [ "OracleDBName", "OracleDBPassword","OracleDBStorage","OracleInstanceType" ] 154 | }, 155 | { 156 | "Label" : { "default":"Target Aurora PostgreSQL Database Configuration" }, 157 | "Parameters" : ["AuroraDBName","AuroraDBUsername", "AuroraDBPassword","AuroraInstanceType", "AuroraEngineType"] 158 | }, 159 | { 160 | "Label" : { "default":"DMS Configuration" }, 161 | "Parameters" : ["DMSInstanceType" ,"ExistsDMSRole"] 162 | }, 163 | { 164 | "Label" : { "default" : "Enter IP address for the DB Security group Configuration" }, 165 | "Parameters" : [ "ClientIP" ] 166 | } 167 | ] 168 | 169 | } 170 | }, 171 | 172 | 173 | "Mappings": 174 | { 175 | 176 | "OracleEngineVersion" : 177 | { 178 | "us-east-1" : {"ver" : "19.0.0.0.ru-2022-10.rur-2022-10.r1"}, 179 | "us-west-2" : {"ver" : "19.0.0.0.ru-2022-10.rur-2022-10.r1"} 180 | }, 181 | 182 | "OracleSnapshotId" : 183 | { 184 | "us-east-1" : {"snapid" : "arn:aws:rds:us-east-1:152097041981:snapshot:aws-db-workshop-oracle-19c-taxidata-sample"}, 185 | "us-west-2" : {"snapid" : "arn:aws:rds:us-west-2:152097041981:snapshot:aws-db-workshop-oracle-19c-taxidata-sample"} 186 | }, 187 | 188 | "AuroraMap": { 189 | 190 | "aurora-postgresql": { 191 | "AuroraEngineVersion": "12.9", 192 | "AuroraPort": "5432", 193 | "AuroraParameterGroupName": "default.aurora-postgresql12", 194 | "AuroraPGFamily": "aurora-postgresql12" 195 | } 196 | } 197 | }, 198 | 199 | "Conditions": { 200 | "NotExistsDMSVPCRole": {"Fn::Equals": [{"Ref": "ExistsDMSRole"}, "N"]} 201 | }, 202 | 203 | "Resources" : { 204 | 205 | "VPC" : { 206 | "Type" : "AWS::EC2::VPC", 207 | "Properties" : { 208 | "CidrBlock" : "10.0.0.0/20", 209 | "EnableDnsSupport" : "true", 210 | "EnableDnsHostnames" : "true", 211 | "Tags" : [ {"Key" : "Application", "Value" : { "Ref" : "AWS::StackId"}}, 212 | {"Key" : "Name", "Value" : { "Ref" : "AWS::StackName"}}] 213 | 214 | } 215 | }, 216 | 217 | "DBSubnet1" : { 218 | "Type" : "AWS::EC2::Subnet", 219 | "Properties" : { 220 | "VpcId" : { "Ref" : "VPC" }, 221 | "CidrBlock" : "10.0.3.0/25", 222 | "AvailabilityZone" : { "Fn::Select" : [ "0", { "Fn::GetAZs" : "" } ] }, 223 | "Tags" : [ {"Key" : "Application", "Value" : { "Ref" : "AWS::StackId"} } ] 224 | } 225 | }, 226 | 227 | "DBSubnet2" : { 228 | "Type" : "AWS::EC2::Subnet", 229 | "Properties" : { 230 | "VpcId" : { "Ref" : "VPC" }, 231 | "CidrBlock" : "10.0.4.0/25", 232 | "AvailabilityZone" : { "Fn::Select" : [ "1", { "Fn::GetAZs" : "" } ] }, 233 | "Tags" : [ {"Key" : "Application", "Value" : { "Ref" : "AWS::StackId"} } ] 234 | } 235 | }, 236 | 237 | 238 | "AppSubnet1" : { 239 | "Type" : "AWS::EC2::Subnet", 240 | "Properties" : { 241 | "VpcId" : { "Ref" : "VPC" }, 242 | "CidrBlock" : "10.0.1.0/25", 243 | "AvailabilityZone" : { "Fn::Select" : [ "0", { "Fn::GetAZs" : "" } ] }, 244 | "Tags" : [ {"Key" : "Application", "Value" : { "Ref" : "AWS::StackId"} } ] 245 | } 246 | }, 247 | 248 | "AppSubnet2" : { 249 | "Type" : "AWS::EC2::Subnet", 250 | "Properties" : { 251 | "VpcId" : { "Ref" : "VPC" }, 252 | "CidrBlock" : "10.0.2.0/25", 253 | "AvailabilityZone" : { "Fn::Select" : [ "1", { "Fn::GetAZs" : "" } ] }, 254 | "Tags" : [ {"Key" : "Application", "Value" : { "Ref" : "AWS::StackId"} } ] 255 | } 256 | }, 257 | 258 | "InternetGateway" : { 259 | "Type" : "AWS::EC2::InternetGateway", 260 | "Properties" : { 261 | "Tags" : [ {"Key" : "Application", "Value" : { "Ref" : "AWS::StackId"} } ] 262 | } 263 | }, 264 | 265 | "AttachGateway" : { 266 | "Type" : "AWS::EC2::VPCGatewayAttachment", 267 | "Properties" : { 268 | "VpcId" : { "Ref" : "VPC" }, 269 | "InternetGatewayId" : { "Ref" : "InternetGateway" } 270 | } 271 | }, 272 | 273 | "RouteTable" : { 274 | "Type" : "AWS::EC2::RouteTable", 275 | "Properties" : { 276 | "VpcId" : {"Ref" : "VPC"}, 277 | "Tags" : [ {"Key" : "Application", "Value" : { "Ref" : "AWS::StackId"} } ] 278 | } 279 | }, 280 | 281 | "Route" : { 282 | "Type" : "AWS::EC2::Route", 283 | "DependsOn" : "AttachGateway", 284 | "Properties" : { 285 | "RouteTableId" : { "Ref" : "RouteTable" }, 286 | "DestinationCidrBlock" : "0.0.0.0/0", 287 | "GatewayId" : { "Ref" : "InternetGateway" } 288 | } 289 | }, 290 | 291 | "SubnetRouteTableAssociation" : { 292 | "Type" : "AWS::EC2::SubnetRouteTableAssociation", 293 | "Properties" : { 294 | "SubnetId" : { "Ref" : "DBSubnet1" }, 295 | "RouteTableId" : { "Ref" : "RouteTable" } 296 | } 297 | }, 298 | 299 | "SubnetRouteTableAssociation1" : { 300 | "Type" : "AWS::EC2::SubnetRouteTableAssociation", 301 | "Properties" : { 302 | "SubnetId" : { "Ref" : "DBSubnet2" }, 303 | "RouteTableId" : { "Ref" : "RouteTable" } 304 | } 305 | }, 306 | 307 | "MyNATGateway": { 308 | "Type": "AWS::EC2::NatGateway", 309 | "DependsOn": "MyNATPublicIP", 310 | "Properties": { 311 | "AllocationId": { 312 | "Fn::GetAtt": [ 313 | "MyNATPublicIP", 314 | "AllocationId" 315 | ] 316 | }, 317 | "SubnetId": { 318 | "Ref": "DBSubnet1" 319 | } 320 | } 321 | }, 322 | "MyNATPublicIP": { 323 | "Type": "AWS::EC2::EIP", 324 | "DependsOn": "VPC", 325 | "Properties": { 326 | "Domain": "vpc" 327 | } 328 | }, 329 | "MyprivateRouteTable": { 330 | "Type": "AWS::EC2::RouteTable", 331 | "Properties": { 332 | "VpcId": { 333 | "Ref" : "VPC" 334 | } 335 | } 336 | }, 337 | "MyprivateRoute": { 338 | "Type": "AWS::EC2::Route", 339 | "Properties": { 340 | "RouteTableId": { 341 | "Ref": "MyprivateRouteTable" 342 | }, 343 | "DestinationCidrBlock": "0.0.0.0/0", 344 | "NatGatewayId": { 345 | "Ref": "MyNATGateway" 346 | } 347 | } 348 | }, 349 | "privateSubnet1RouteTableAssociation": { 350 | "Type": "AWS::EC2::SubnetRouteTableAssociation", 351 | "Properties": { 352 | "SubnetId": { 353 | "Ref": "AppSubnet1" 354 | }, 355 | "RouteTableId": { 356 | "Ref": "MyprivateRouteTable" 357 | } 358 | } 359 | }, 360 | "privateSubnet2RouteTableAssociation": { 361 | "Type": "AWS::EC2::SubnetRouteTableAssociation", 362 | "Properties": { 363 | "SubnetId": { 364 | "Ref": "AppSubnet2" 365 | }, 366 | "RouteTableId": { 367 | "Ref": "MyprivateRouteTable" 368 | } 369 | } 370 | }, 371 | 372 | "MyDBSubnetGroup" : { 373 | "Type" : "AWS::RDS::DBSubnetGroup", 374 | "Properties" : { 375 | "DBSubnetGroupDescription" : "Subnet available for the DMS Demo RDS DB Instance", 376 | "SubnetIds" : [{ "Ref" : "DBSubnet1" },{ "Ref" : "DBSubnet2" }] 377 | } 378 | }, 379 | 380 | "MyDMSReplicationSubnetGroup" : { 381 | "Type" : "AWS::DMS::ReplicationSubnetGroup", 382 | "Properties" : { 383 | "ReplicationSubnetGroupDescription" : "Subnet group for DMS replication instances", 384 | "SubnetIds" : [{ "Ref" : "AppSubnet1" },{ "Ref" : "AppSubnet2" }] 385 | 386 | } 387 | }, 388 | 389 | "OraVPCSecurityGroup" : { 390 | "Type" : "AWS::EC2::SecurityGroup", 391 | "Properties" : 392 | { 393 | "GroupDescription" : "Security group for Oracle Instance.", 394 | "VpcId" : { "Ref" : "VPC" }, 395 | 396 | "SecurityGroupIngress" : [ 397 | 398 | 399 | { 400 | "IpProtocol" : "tcp", 401 | "FromPort" : "1521", 402 | "ToPort" : "1521", 403 | "CidrIp" : { "Ref" : "ClientIP"} 404 | }, 405 | 406 | { 407 | "IpProtocol" : "tcp", 408 | "FromPort" : "1521", 409 | "ToPort" : "1521", 410 | "CidrIp" : "10.0.0.0/20" 411 | } 412 | ] 413 | } 414 | }, 415 | 416 | 417 | "AuroraVPCSecurityGroup" : { 418 | "Type" : "AWS::EC2::SecurityGroup", 419 | "Properties" : 420 | { 421 | "GroupDescription" : "Security group for Aurora PostgreSQL Instances.", 422 | "VpcId" : { "Ref" : "VPC" }, 423 | 424 | "SecurityGroupIngress" : [ 425 | 426 | { 427 | "IpProtocol" : "tcp", 428 | "FromPort" : "5432", 429 | "ToPort" : "5432", 430 | "CidrIp" : { "Ref" : "ClientIP"} 431 | }, 432 | { 433 | "IpProtocol" : "tcp", 434 | "FromPort" : "5432", 435 | "ToPort" : "5432", 436 | "CidrIp" : "10.0.0.0/20" 437 | }] 438 | } 439 | }, 440 | 441 | 442 | "LambdaVPCSecurityGroup" : { 443 | "Type" : "AWS::EC2::SecurityGroup", 444 | "Properties" : 445 | { 446 | "GroupDescription" : "Security group for Lambda and VPC endpoints connectivity.", 447 | "VpcId" : { "Ref" : "VPC" }, 448 | 449 | "SecurityGroupIngress" : [ 450 | { 451 | "IpProtocol" : "tcp", 452 | "FromPort" : "443", 453 | "ToPort" : "443", 454 | "CidrIp" : "10.0.0.0/20" 455 | }] 456 | } 457 | }, 458 | 459 | 460 | 461 | 462 | "OracleDB" : { 463 | "Type" : "AWS::RDS::DBInstance", 464 | "DependsOn" : "AttachGateway", 465 | "DeletionPolicy" : "Delete", 466 | "Properties" : { 467 | "DBName" : { "Ref" : "OracleDBName" }, 468 | "AllocatedStorage" : { "Ref" : "OracleDBStorage" }, 469 | "MasterUserPassword" : { "Ref" : "OracleDBPassword" }, 470 | "DBInstanceClass" : { "Ref" : "OracleInstanceType" }, 471 | "Engine" : "oracle-se2", 472 | "EngineVersion" : {"Fn::FindInMap" : [ "OracleEngineVersion", { "Ref" : "AWS::Region" }, "ver" ]}, 473 | "LicenseModel" : "license-included", 474 | "PubliclyAccessible" : "true", 475 | "AvailabilityZone" : { "Fn::GetAtt" : [ "DBSubnet1", "AvailabilityZone" ] }, 476 | "MultiAZ" : "false", 477 | "DBSubnetGroupName" : { "Ref" : "MyDBSubnetGroup" }, 478 | "VPCSecurityGroups" : [ { "Ref" : "OraVPCSecurityGroup" } ], 479 | "DBSnapshotIdentifier": {"Fn::FindInMap" : [ "OracleSnapshotId", { "Ref" : "AWS::Region" }, "snapid" ]}, 480 | "StorageType" : "gp2", 481 | "Tags" : [ {"Key" : "Application", "Value" : { "Ref" : "AWS::StackId"} } ] 482 | } 483 | }, 484 | 485 | "AuroraCluster" : { 486 | "Type" : "AWS::RDS::DBCluster", 487 | "DependsOn" : "AttachGateway", 488 | "Properties" : { 489 | "MasterUsername" : { "Ref" : "AuroraDBUsername" }, 490 | "MasterUserPassword" : { "Ref" : "AuroraDBPassword" }, 491 | "Engine" : {"Ref": "AuroraEngineType"}, 492 | "EngineVersion": { 493 | "Fn::FindInMap": [ 494 | "AuroraMap", 495 | { 496 | "Ref": "AuroraEngineType" 497 | }, 498 | "AuroraEngineVersion" 499 | ] 500 | }, 501 | "DBClusterParameterGroupName": { 502 | "Fn::FindInMap": [ 503 | "AuroraMap", 504 | { 505 | "Ref": "AuroraEngineType" 506 | }, 507 | "AuroraParameterGroupName" 508 | ] 509 | }, 510 | "DatabaseName" : {"Ref": "AuroraDBName"}, 511 | "Port": "5432", 512 | "DBSubnetGroupName" : { "Ref" : "MyDBSubnetGroup" }, 513 | "VpcSecurityGroupIds" : [{ "Ref" : "AuroraVPCSecurityGroup" }] 514 | 515 | } 516 | }, 517 | "AuroraDBParameterGroup": { 518 | "Type": "AWS::RDS::DBParameterGroup", 519 | "Properties" : { 520 | "Description" : "PostgreSQL Parameter Group for DMS Lab", 521 | "Family" : { 522 | "Fn::FindInMap": [ 523 | "AuroraMap", 524 | { 525 | "Ref": "AuroraEngineType" 526 | }, 527 | "AuroraPGFamily" 528 | ] 529 | }, 530 | "Parameters" : { 531 | "session_replication_role": "replica" 532 | } 533 | } 534 | }, 535 | 536 | "AuroraDB" : { 537 | "Type" : "AWS::RDS::DBInstance", 538 | "DependsOn" : "AttachGateway", 539 | "Properties" : { 540 | "DBSubnetGroupName" : { "Ref" : "MyDBSubnetGroup" }, 541 | "Engine" : {"Ref": "AuroraEngineType"}, 542 | "MultiAZ" : "false", 543 | "DBClusterIdentifier" : {"Ref" : "AuroraCluster" }, 544 | "DBParameterGroupName" : {"Ref":"AuroraDBParameterGroup"}, 545 | "PubliclyAccessible" : "true", 546 | "AvailabilityZone" : { "Fn::GetAtt" : [ "DBSubnet1", "AvailabilityZone" ] }, 547 | "DBInstanceClass" : { "Ref" : "AuroraInstanceType" }, 548 | "Tags" : [ {"Key" : "Application", "Value" : { "Ref" : "AWS::StackId"} } ] 549 | } 550 | }, 551 | 552 | "Cloud9Environment": { 553 | "Type": "AWS::Cloud9::EnvironmentEC2", 554 | "Properties": { 555 | "AutomaticStopTimeMinutes": 60, 556 | "InstanceType": "t3.medium", 557 | "Name": { 558 | "Fn::Sub": "Project-${AWS::StackName}" 559 | }, 560 | "SubnetId": { 561 | "Ref": "DBSubnet1" 562 | } 563 | } 564 | }, 565 | "myDynamoDBTable" : { 566 | "Type" : "AWS::DynamoDB::Table", 567 | "Properties" : { 568 | "AttributeDefinitions": [ 569 | { 570 | "AttributeName": "riderid", 571 | "AttributeType": "S" 572 | }, 573 | { 574 | "AttributeName": "tripinfo", 575 | "AttributeType": "S" 576 | }, 577 | { 578 | "AttributeName": "DROPOFF_DATETIME", 579 | "AttributeType": "S" 580 | }, 581 | { 582 | "AttributeName": "DRIVER_EMAIL", 583 | "AttributeType": "S" 584 | } 585 | ], 586 | 587 | "KeySchema" : [ 588 | { 589 | "AttributeName" : "riderid", 590 | "KeyType" : "HASH" 591 | }, 592 | { 593 | "AttributeName" : "tripinfo", 594 | "KeyType" : "RANGE" 595 | } 596 | ], 597 | "BillingMode" : "PAY_PER_REQUEST", 598 | "TableName" : "aws-db-workshop-trips", 599 | "GlobalSecondaryIndexes" : [{ 600 | "IndexName" : "trips-driver-GSI1", 601 | "KeySchema" : [ 602 | { 603 | "AttributeName" : "DRIVER_EMAIL", 604 | "KeyType" : "HASH" 605 | }, 606 | { 607 | "AttributeName" : "DROPOFF_DATETIME", 608 | "KeyType" : "RANGE" 609 | } 610 | ], 611 | "Projection": { 612 | "ProjectionType": "KEYS_ONLY" 613 | } 614 | 615 | }] 616 | 617 | } 618 | }, 619 | 620 | "Cloud9DevIAMRole" : { 621 | "Type" : "AWS::IAM::Role", 622 | "Properties" : { 623 | "ManagedPolicyArns" : ["arn:aws:iam::aws:policy/AmazonDynamoDBFullAccess","arn:aws:iam::aws:policy/AmazonS3FullAccess","arn:aws:iam::aws:policy/AWSLambda_FullAccess","arn:aws:iam::aws:policy/SecretsManagerReadWrite","arn:aws:iam::aws:policy/AWSCloudFormationFullAccess"], 624 | "AssumeRolePolicyDocument" : { 625 | "Version" : "2012-10-17", 626 | "Statement" : [{ 627 | "Effect" : "Allow", 628 | "Principal" : { "Service" : "ec2.amazonaws.com" }, 629 | "Action" : "sts:AssumeRole" 630 | }] 631 | }, 632 | "Path": "/" 633 | }}, 634 | 635 | "DBWorkshopS3Bucket": { 636 | "Type": "AWS::S3::Bucket" 637 | }, 638 | 639 | 640 | "DMSIAMRole1": { 641 | "Type": "AWS::IAM::Role", 642 | "Condition" : "NotExistsDMSVPCRole", 643 | "Properties": { 644 | "ManagedPolicyArns": [ "arn:aws:iam::aws:policy/service-role/AmazonDMSVPCManagementRole" ], 645 | "Path": "/", 646 | "RoleName": "dms-vpc-role", 647 | "AssumeRolePolicyDocument" : { 648 | "Version" : "2012-10-17", 649 | "Statement" : [{ 650 | "Effect" : "Allow", 651 | "Principal" : { "Service" : "dms.amazonaws.com" }, 652 | "Action" : "sts:AssumeRole" 653 | }] 654 | } 655 | } 656 | }, 657 | 658 | "DMSIAMRole2": { 659 | "Type": "AWS::IAM::Role", 660 | "Condition" : "NotExistsDMSVPCRole", 661 | "Properties": { 662 | "ManagedPolicyArns": ["arn:aws:iam::aws:policy/service-role/AmazonDMSCloudWatchLogsRole"], 663 | "Path": "/", 664 | "RoleName": "dms-cloudwatch-logs-role", 665 | "AssumeRolePolicyDocument" : { 666 | "Version" : "2012-10-17", 667 | "Statement" : [{ 668 | "Effect" : "Allow", 669 | "Principal" : { "Service" : "dms.amazonaws.com" }, 670 | "Action" : "sts:AssumeRole" 671 | }] 672 | } 673 | } 674 | }, 675 | 676 | 677 | "DMSMigratetoDDBRole" : { 678 | "Type" : "AWS::IAM::Role", 679 | "Properties" : { 680 | "ManagedPolicyArns" : ["arn:aws:iam::aws:policy/AmazonDynamoDBFullAccess"], 681 | "AssumeRolePolicyDocument" : { 682 | "Version" : "2012-10-17", 683 | "Statement" : [{ 684 | "Effect" : "Allow", 685 | "Principal" : { "Service" : "dms.amazonaws.com" }, 686 | "Action" : "sts:AssumeRole" 687 | }] 688 | } 689 | } 690 | }, 691 | "DMSWorkshopInstance": { 692 | "Type": "AWS::DMS::ReplicationInstance", 693 | "Properties": { 694 | "AllocatedStorage": 100, 695 | "PubliclyAccessible": "false", 696 | "EngineVersion": "3.4.6", 697 | "ReplicationInstanceClass": { "Ref" : "DMSInstanceType" }, 698 | "ReplicationSubnetGroupIdentifier": { "Ref" : "MyDMSReplicationSubnetGroup" }, 699 | "Tags" : [ {"Key" : "Application", "Value" : { "Ref" : "AWS::StackId"} } ] 700 | } 701 | } 702 | }, 703 | "Outputs" : { 704 | "StackName" : { "Value" : { "Ref" : "AWS::StackName" } }, 705 | "Regionname" : { "Value" : { "Ref" : "AWS::Region" } }, 706 | "VPCid" : { "Value" : { 707 | "Fn::GetAtt" : [ "AppSubnet1" , "VpcId" ] }}, 708 | 709 | "LambdaSecurityGroupId" : { "Value" : { 710 | "Fn::GetAtt" : [ "LambdaVPCSecurityGroup" , "GroupId" ]}}, 711 | 712 | "LambdaSubnet1" : { "Value" : { "Ref" : "AppSubnet1" } }, 713 | "LambdaSubnet2" : { "Value" : { "Ref" : "AppSubnet2" } }, 714 | 715 | "S3bucketName": { 716 | "Description": "S3 bucket used for this workshop", 717 | "Value": { 718 | "Ref": "DBWorkshopS3Bucket" 719 | } 720 | }, 721 | 722 | 723 | "Cloud9Env": { 724 | "Value": { 725 | "Fn::Join": [ 726 | "", 727 | [ 728 | "https://", 729 | { 730 | "Ref": "AWS::Region" 731 | }, 732 | ".console.aws.amazon.com/cloud9/home/", 733 | 734 | "?region=", 735 | { 736 | "Ref": "AWS::Region" 737 | } 738 | ] 739 | ] 740 | } 741 | }, 742 | 743 | 744 | "DDBTableName" : { "Value" : { "Ref" : "myDynamoDBTable" }}, 745 | 746 | "DMSInstanceName" : { "Value" : { "Ref" : "DMSWorkshopInstance" }}, 747 | 748 | "DMSDDBRoleARN" : { "Value" : { "Fn::GetAtt": [ "DMSMigratetoDDBRole", "Arn"] }}, 749 | 750 | "OracleDBMasterUser":{ "Value" :{ "Fn::Select" : [ "0", [ "dbadmin", "auradmin"] ] }}, 751 | 752 | "OracleRDSDetails": { 753 | "Description" : "Oracle RDS DNS:port:DBName", 754 | "Value" : { "Fn::Join": [ "", [ { "Fn::GetAtt": [ "OracleDB", "Endpoint.Address" ] }, 755 | ":", 756 | { "Fn::GetAtt": [ "OracleDB", "Endpoint.Port" ] }, 757 | ":", 758 | { "Ref": "OracleDBName" }]]} 759 | }, 760 | 761 | "OracleJDBCConnectionString": { 762 | "Description" : "JDBC string for Oracle database", 763 | "Value" : { "Fn::Join": [ "", [ "jdbc:oracle:thin:@", 764 | { "Fn::GetAtt": [ "OracleDB", "Endpoint.Address" ] }, 765 | ":", 766 | { "Fn::GetAtt": [ "OracleDB", "Endpoint.Port" ] }, 767 | ":", 768 | { "Ref": "OracleDBName" }]]} 769 | }, 770 | 771 | "AuroraClusterEndpointName": { 772 | "Description" : "Aurora cluster Name", 773 | "Value" : { "Fn::Join": [ "", [ { "Fn::GetAtt": [ "AuroraCluster", "Endpoint.Address" ]} ]] } 774 | }, 775 | 776 | "AuroraDBName": { 777 | "Description" : "Aurora DB Name", 778 | "Value" : { "Ref": "AuroraDBName" } 779 | }, 780 | 781 | "AuroraDBMasterUser": { "Value" : { "Ref" : "AuroraDBUsername" } }, 782 | 783 | "AuroraJDBCConnectionString": { 784 | "Description" : "JDBC string for Aurora database", 785 | "Value" : { "Fn::Join": [ "", [ "jdbc:postgresql://", 786 | { "Fn::GetAtt": [ "AuroraCluster", "Endpoint.Address" ] }, 787 | ":", 788 | { "Fn::GetAtt": [ "AuroraCluster", "Endpoint.Port" ] }, "/", { "Ref": "AuroraDBName" } 789 | ]]} 790 | 791 | } 792 | } 793 | } 794 | -------------------------------------------------------------------------------- /src/create_taxi_schema.sql: -------------------------------------------------------------------------------- 1 | set search_path=public; 2 | \set AUTOCOMMIT off 3 | 4 | /* This file contains DDL for creating relational schema in PostgreSQL. We will also populate the reference tables with values. for some of the tables that uses serial data type, the serial data 5 | is adjusted to higher value so that it doesn't conflict with the migrated data from source */ 6 | 7 | CREATE TABLE cab_types ( 8 | id serial primary key, 9 | type text 10 | ); 11 | 12 | INSERT INTO cab_types (type) SELECT 'yellow'; 13 | INSERT INTO cab_types (type) SELECT 'green'; 14 | INSERT INTO cab_types (type) SELECT 'uber'; 15 | INSERT INTO cab_types (type) SELECT 'lyft'; 16 | 17 | create table car_model(id serial primary key,make varchar not null, model varchar); 18 | insert into car_model(make,model) values('Toyota','SUV 2016'); 19 | insert into car_model(make,model) values('Honda','SEDAN 2016'); 20 | insert into car_model(make,model) values('Nissan','SEDAN 2016'); 21 | insert into car_model(make,model) values('Ford','TRUCK 2016'); 22 | insert into car_model(make,model) values('Subaru','SUV 2016'); 23 | 24 | create table payment_types (payment_id bigint primary key, name varchar(100) unique not null, description varchar(100)); 25 | insert into payment_types values (1,'Credit Card',null); 26 | insert into payment_types values (2,'Cash',null); 27 | insert into payment_types values (3,'No Charge',null); 28 | insert into payment_types values (4,'Dispute',null); 29 | insert into payment_types values (5,'Unknown',null); 30 | insert into payment_types values (6,'Cancelled',null); 31 | insert into payment_types values (7,'Bank Deposit',null); 32 | 33 | create table trip_status (status varchar(100) unique not null,description varchar(100)); 34 | insert into trip_status(status) values ('Completed'); 35 | insert into trip_status(status) values ('Cancelled'); 36 | insert into trip_status(status) values ('Booked'); 37 | insert into trip_status(status) values ('In-progress'); 38 | 39 | create table rate_codes (rate_id bigint primary key, name varchar(100) unique not null); 40 | insert into rate_codes values (1,'Standard rate'); 41 | insert into rate_codes values (2,'JFK'); 42 | insert into rate_codes values (3,'Newark'); 43 | insert into rate_codes values (4,'Nassau or Westchester'); 44 | insert into rate_codes values (5,'Negotiated fare'); 45 | insert into rate_codes values (6,'Group ride'); 46 | insert into rate_codes values (99,'Peak rate'); 47 | 48 | 49 | CREATE TABLE drivers( 50 | driver_id bigint PRIMARY KEY, 51 | driver_name VARCHAR (100) NOT NULL, 52 | vehicle_id VARCHAR (50) UNIQUE NOT NULL, 53 | make bigint references car_model(id), 54 | driver_email VARCHAR (100) UNIQUE NOT NULL, 55 | created_on TIMESTAMP NOT NULL, 56 | driver_mobile varchar(100) NOT NULL, 57 | payment_id bigint not null references payment_types(payment_id), 58 | address varchar(1000), 59 | rating varchar(10), 60 | profile varchar(1000) 61 | ); 62 | 63 | CREATE TABLE riders( 64 | rider_id bigint PRIMARY KEY, 65 | rider_name VARCHAR (100) NOT NULL, 66 | rider_email VARCHAR (100) UNIQUE NOT NULL, 67 | created_on TIMESTAMP NOT NULL, 68 | payment_id bigint not null references payment_types(payment_id), 69 | rider_mobile varchar(100) NOT NULL, 70 | address varchar(1000), 71 | rating varchar(10), 72 | profile varchar(1000) 73 | ); 74 | 75 | CREATE TABLE billing( 76 | id serial primary key, 77 | driver_id bigint references drivers(driver_id), 78 | billing_cycle double precision default 1, 79 | billing_start timestamp without time zone, 80 | billing_end timestamp without time zone, 81 | billing_date timestamp without time zone DEFAULT CURRENT_TIMESTAMP, 82 | billing_amount numeric(10,6), 83 | commissions numeric(10,6) default 0.2, 84 | description varchar(500), 85 | rides_total bigint, 86 | billing_status varchar(100) 87 | ); 88 | 89 | 90 | 91 | CREATE TABLE payment( 92 | id serial primary key, 93 | billing_id bigint references billing(id), 94 | driver_id bigint references drivers(driver_id), 95 | billing_cycle double precision default 1, 96 | payment_amount numeric(10,6), 97 | payment_date timestamp without time zone DEFAULT CURRENT_TIMESTAMP, 98 | payment_id bigint default 7 references payment_types(payment_id), 99 | payment_status varchar(100), 100 | description varchar(500) 101 | ); 102 | 103 | 104 | 105 | create table trips( 106 | id serial primary key, 107 | rider_id bigint references riders(rider_id), 108 | driver_id bigint references drivers(driver_id), 109 | rider_name varchar(100), 110 | rider_mobile varchar(100), 111 | rider_email varchar(100) not null, 112 | trip_info varchar(100) not null, 113 | driver_name varchar(100), 114 | driver_email varchar(100), 115 | driver_mobile varchar(100), 116 | vehicle_id varchar(50), 117 | cab_type_id integer references cab_types(id), 118 | vendor_id integer, 119 | pickup_datetime timestamp without time zone, 120 | dropoff_datetime timestamp without time zone, 121 | store_and_fwd_flag varchar(100), 122 | rate_code_id bigint references rate_codes(rate_id), 123 | pickup_longitude DOUBLE PRECISION, 124 | pickup_latitude DOUBLE PRECISION, 125 | dropoff_longitude DOUBLE PRECISION, 126 | dropoff_latitude DOUBLE PRECISION, 127 | passenger_count DOUBLE PRECISION, 128 | trip_distance DOUBLE PRECISION, 129 | fare_amount DOUBLE PRECISION, 130 | extra DOUBLE PRECISION, 131 | mta_tax DOUBLE PRECISION, 132 | tip_amount DOUBLE PRECISION, 133 | tolls_amount DOUBLE PRECISION, 134 | ehail_fee DOUBLE PRECISION, 135 | improvement_surcharge DOUBLE PRECISION, 136 | total_amount DOUBLE PRECISION, 137 | payment_type bigint references payment_types (payment_id), 138 | trip_type integer, 139 | pickup_location_id integer, 140 | dropoff_location_id integer, 141 | status varchar(100) references trip_status(status), 142 | unique (rider_email,trip_info) 143 | ); 144 | SELECT pg_catalog.setval(pg_get_serial_sequence('trips', 'id'),max(2000000)); 145 | SELECT pg_catalog.setval(pg_get_serial_sequence('billing', 'id'),max(200000)); 146 | SELECT pg_catalog.setval(pg_get_serial_sequence('payment', 'id'),max(200000)); 147 | 148 | create sequence billing_cycle_seq start with 2; 149 | 150 | CREATE or REPLACE PROCEDURE billingandpayments() 151 | LANGUAGE plpgsql 152 | as $$ 153 | DECLARE 154 | t RECORD; 155 | l_sequence integer; 156 | l_count integer; 157 | l_rcount integer; 158 | BEGIN 159 | 160 | select count(*) into l_count from trips where dropoff_datetime < current_date+1 and dropoff_datetime > current_date and status = 'Completed'; 161 | 162 | RAISE NOTICE 'Found % trip(s) record to be processed for billing', l_count; 163 | 164 | if (l_count > 0) then 165 | 166 | l_sequence := (SELECT nextval('billing_cycle_seq')); 167 | 168 | RAISE NOTICE 'Running Billing Cycle # %', l_sequence; 169 | 170 | insert into billing (driver_id, billing_cycle, billing_start, billing_end, billing_amount, commissions, rides_total, description, billing_status) 171 | select driver_id, l_sequence, current_date, current_date+1, sum(total_amount), 0.8, count(*), 'billing cycle ' || l_sequence, 'Completed' 172 | from trips 173 | where dropoff_datetime < current_date+1 and dropoff_datetime > current_date and status = 'Completed' 174 | group by driver_id; 175 | 176 | GET DIAGNOSTICS l_rcount = ROW_COUNT; 177 | 178 | RAISE NOTICE 'Inserted % record(s) into billing table', l_rcount; 179 | 180 | insert into payment(billing_id,driver_id,billing_cycle,payment_amount,payment_date, payment_id, payment_status,description) 181 | select a.id, a.driver_id, a.billing_cycle,sum(a.billing_amount*a.commissions),a.billing_date, b.payment_id, 'Completed','Payment Cycle ' || to_char(transaction_timestamp(),'FMMonth YYYY') 182 | from billing a, drivers b 183 | where a.driver_id=b.driver_id and a.billing_cycle = l_sequence and a.billing_status = 'Completed' 184 | group by a.id, a.driver_id, a.billing_cycle, a.billing_date, b.payment_id; 185 | 186 | GET DIAGNOSTICS l_rcount = ROW_COUNT; 187 | 188 | RAISE NOTICE 'Inserted % record(s) into payment table', l_rcount; 189 | 190 | update trips set status = 'Processed' 191 | where dropoff_datetime < current_date+1 and dropoff_datetime > current_date and status = 'Completed'; 192 | 193 | GET DIAGNOSTICS l_rcount = ROW_COUNT; 194 | 195 | RAISE NOTICE 'Updated % record(s) in trips table and marked status as Processed', l_rcount; 196 | 197 | update billing set billing_status = 'Processed' 198 | where billing_cycle = l_sequence and billing_status = 'Completed'; 199 | 200 | GET DIAGNOSTICS l_rcount = ROW_COUNT; 201 | 202 | RAISE NOTICE 'Updated % record(s) in billing table and marked status as Processed', l_rcount; 203 | 204 | RAISE NOTICE 'Billing Cycle # % Completed Successfully', l_sequence; 205 | 206 | else 207 | 208 | RAISE NOTICE 'No records found to be processed for billing. Exiting...'; 209 | 210 | end if; 211 | 212 | exception when others then 213 | raise; 214 | 215 | END; 216 | $$; 217 | 218 | commit; 219 | \dt 220 | 221 | -------------------------------------------------------------------------------- /src/create_taxi_schema_oracle.sql: -------------------------------------------------------------------------------- 1 | DROP USER "TAXI" CASCADE; 2 | 3 | CREATE USER "TAXI"; 4 | 5 | ALTER USER "TAXI" 6 | DEFAULT TABLESPACE "USERS" 7 | TEMPORARY TABLESPACE "TEMP" 8 | ACCOUNT UNLOCK ; 9 | 10 | GRANT CREATE SESSION, RESOURCE TO TAXI; 11 | ALTER USER TAXI QUOTA 100M ON users; 12 | 13 | GRANT "CONNECT" TO "TAXI" WITH ADMIN OPTION; 14 | GRANT "RESOURCE" TO "TAXI" WITH ADMIN OPTION; 15 | GRANT "DBA" TO "TAXI" WITH ADMIN OPTION; 16 | GRANT "SELECT_CATALOG_ROLE" TO "TAXI" WITH ADMIN OPTION; 17 | GRANT "EXECUTE_CATALOG_ROLE" TO "TAXI" WITH ADMIN OPTION; 18 | GRANT "CAPTURE_ADMIN" TO "TAXI" WITH ADMIN OPTION; 19 | GRANT "EXP_FULL_DATABASE" TO "TAXI" WITH ADMIN OPTION; 20 | GRANT "IMP_FULL_DATABASE" TO "TAXI" WITH ADMIN OPTION; 21 | GRANT "AQ_ADMINISTRATOR_ROLE" TO "TAXI" WITH ADMIN OPTION; 22 | GRANT "AQ_USER_ROLE" TO "TAXI" WITH ADMIN OPTION; 23 | GRANT "DATAPUMP_EXP_FULL_DATABASE" TO "TAXI" WITH ADMIN OPTION; 24 | GRANT "DATAPUMP_IMP_FULL_DATABASE" TO "TAXI" WITH ADMIN OPTION; 25 | GRANT "CTXAPP" TO "TAXI" WITH ADMIN OPTION; 26 | GRANT "EM_EXPRESS_ALL" TO "TAXI" WITH ADMIN OPTION; 27 | GRANT "SCHEDULER_ADMIN" TO "TAXI" WITH ADMIN OPTION; 28 | GRANT "RECOVERY_CATALOG_OWNER" TO "TAXI" WITH ADMIN OPTION; 29 | GRANT "GATHER_SYSTEM_STATISTICS" TO "TAXI" WITH ADMIN OPTION; 30 | GRANT "HS_ADMIN_EXECUTE_ROLE" TO "TAXI" WITH ADMIN OPTION; 31 | GRANT "SELECT_CATALOG_ROLE" TO "TAXI" WITH ADMIN OPTION; 32 | GRANT "SODA_APP" TO "TAXI" WITH ADMIN OPTION; 33 | GRANT "OEM_MONITOR" TO "TAXI" WITH ADMIN OPTION; 34 | GRANT "XDB_SET_INVOKER" TO "TAXI" WITH ADMIN OPTION; 35 | GRANT "EM_EXPRESS_BASIC" TO "TAXI" WITH ADMIN OPTION; 36 | GRANT "OEM_ADVISOR" TO "TAXI" WITH ADMIN OPTION; 37 | GRANT "XDBADMIN" TO "TAXI" WITH ADMIN OPTION; 38 | GRANT "HS_ADMIN_SELECT_ROLE" TO "TAXI" WITH ADMIN OPTION; 39 | 40 | 41 | 42 | 43 | -------------------------------------------------------------------------------- /src/ddb-python-script/driver-accept-trip.py: -------------------------------------------------------------------------------- 1 | import json 2 | import random 3 | from decimal import getcontext, Decimal 4 | from random import randint 5 | 6 | from util import updateTripInfo, getTripInfo, DecimalEncoder 7 | 8 | driver_details = [ 9 | {"driver_email" : "driver556550@taxi.com", "driver_id": "556550", "driver_name": "driver556550", "driver_mobile": "+11720912154", "vehicle_details": {"id": "QSY353471", "type" : 2}}, 10 | {"driver_email" : "driver541829@taxi.com", "driver_id": "541829", "driver_name": "driver541829", "driver_mobile": "+11264112049", "vehicle_details": {"id": "GTR508161", "type" : 2}}, 11 | {"driver_email" : "driver507977@taxi.com", "driver_id": "507977", "driver_name": "driver507977", "driver_mobile": "+11088418780", "vehicle_details": {"id": "XVJ356159", "type" : 2}}, 12 | {"driver_email" : "driver551153@taxi.com", "driver_id": "551153", "driver_name": "driver551153", "driver_mobile": "+11240868167", "vehicle_details": {"id": "CPX160101", "type" : 2}}, 13 | {"driver_email" : "driver520045@taxi.com", "driver_id": "520045", "driver_name": "driver520045", "driver_mobile": "+11751510159", "vehicle_details": {"id": "HHR298952", "type" : 2}}, 14 | {"driver_email" : "driver514040@taxi.com", "driver_id": "514040", "driver_name": "driver514040", "driver_mobile": "+11661484862", "vehicle_details": {"id": "TLA210480", "type" : 2}}, 15 | {"driver_email" : "driver527336@taxi.com", "driver_id": "527336", "driver_name": "driver527336", "driver_mobile": "+11564984764", "vehicle_details": {"id": "OVY229214", "type" : 2}}, 16 | {"driver_email" : "driver510909@taxi.com", "driver_id": "510909", "driver_name": "driver510909", "driver_mobile": "+11261783124", "vehicle_details": {"id": "UDT200764", "type" : 2}}, 17 | {"driver_email" : "driver549736@taxi.com", "driver_id": "549736", "driver_name": "driver549736", "driver_mobile": "+11561755450", "vehicle_details": {"id": "ORX460076", "type" : 2}}, 18 | {"driver_email" : "driver528204@taxi.com", "driver_id": "528204", "driver_name": "driver528204", "driver_mobile": "+11185992795", "vehicle_details": {"id": "PXX248130", "type" : 2}} 19 | ] 20 | 21 | 22 | 23 | 24 | rider_id = "person69257@example.com" 25 | vendor_id = 2 26 | 27 | pickup_longitude = str(round(random.uniform(-74,-73),6)) 28 | pickup_latitude = str(round(random.uniform(40,41),6)) 29 | driver_info = driver_details[randint(0, 9)] 30 | 31 | trip_info = input("Enter your tripinfo : ") 32 | 33 | tripAcceptInfo = { 34 | "riderid" : rider_id, 35 | "tripinfo" : trip_info, 36 | "VENDOR_ID" : vendor_id, 37 | "PICKUP_LONGITUDE" : pickup_longitude, 38 | "PICKUP_LATITUDE" : pickup_latitude, 39 | "TRIP_TYPE" : 2, 40 | "STORE_AND_FWD_FLAG" : "N", 41 | "CAB_TYPE_ID" : driver_info['vehicle_details']['type'], 42 | "DRIVER_NAME" : driver_info['driver_name'], 43 | "VEHICLE_ID" : driver_info['vehicle_details']['id'], 44 | "DRIVER_ID" : driver_info['driver_id'], 45 | "DRIVER_EMAIL" : driver_info['driver_email'], 46 | "DRIVER_MOBILE" : driver_info['driver_mobile'], 47 | "DriverDetails" : { 48 | "Name" : driver_info['driver_name'], 49 | "Vehicle Details" : { 50 | "id" : driver_info['vehicle_details']['id'], 51 | "type": driver_info['vehicle_details']['type'] 52 | } 53 | }, 54 | "Status" : "InProgress" 55 | } 56 | 57 | print("Trip accept information ="+ json.dumps(tripAcceptInfo, indent=2)) 58 | 59 | response = updateTripInfo(tripAcceptInfo, "Booked") 60 | print("Trip accept information has been updated to Trips table") 61 | 62 | print("Driver Accept Trip Informaiton =" + json.dumps(response['Attributes'], indent = 4, cls=DecimalEncoder) ) -------------------------------------------------------------------------------- /src/ddb-python-script/driver-complete-trip.py: -------------------------------------------------------------------------------- 1 | import datetime 2 | import json 3 | import random 4 | 5 | from random import randint 6 | 7 | from util import updateTripInfo, getTripInfo, DecimalEncoder 8 | 9 | rate_codes = [1, 2, 3, 4, 5, 6, 99] 10 | 11 | rider_id = "person69257@example.com" 12 | dropoff_longitude = round(random.uniform(-74,-73),6) 13 | dropoff_latitude = round(random.uniform(40,41),6) 14 | 15 | trip_info = input("Enter your tripinfo : ") 16 | 17 | tripCompletedInfo = { 18 | "riderid" : rider_id, 19 | "tripinfo" : trip_info, 20 | "DROPOFF_LATITUDE" : str(dropoff_latitude), 21 | "RATE_CODE_ID" : rate_codes[randint(0, 6)], 22 | "TOLLS_AMOUNT" : str(round(random.uniform(0,5),2)), 23 | "IMPROVEMENT_SURCHARGE" : str(round(random.uniform(0,1),1)), 24 | "TIP_AMOUNT" : str(round(random.uniform(0,10),2)), 25 | "DROPOFF_DATETIME" : datetime.datetime.utcnow().strftime("%Y-%m-%dT%H:%M:%SZ"), 26 | "TRIP_DISTANCE" : randint(1, 50), 27 | "TOTAL_AMOUNT" : str(round(random.uniform(5,150),2)), 28 | "MTA_TAX" : str(round(random.uniform(0,1),1)), 29 | "DROPOFF_LONGITUDE" : str(dropoff_longitude), 30 | "PAYMENT_TYPET" : randint(1, 7), 31 | "EXTRA" : str(round(random.uniform(0,1),1)), 32 | "FARE_AMOUNT" : str(round(random.uniform(5,150),2)), 33 | "PASSENGER_COUNT": randint(1, 7), 34 | "Status" : "Completed" 35 | } 36 | 37 | print ("Trip complettion information = ", json.dumps(tripCompletedInfo, indent=2)) 38 | 39 | response = updateTripInfo(tripCompletedInfo, "InProgress") 40 | print("Trip completion information has been updated to Trips table") 41 | 42 | print("Driver trip completion informaiton =" + json.dumps(response['Attributes'], indent = 4, cls=DecimalEncoder) ) 43 | -------------------------------------------------------------------------------- /src/ddb-python-script/rider-book-trip.py: -------------------------------------------------------------------------------- 1 | import datetime 2 | import json 3 | 4 | from random import randint 5 | from util import updateTripInfo, getTripInfo, DecimalEncoder 6 | 7 | rider_id = 69257 8 | 9 | rider_name = "person" + str(rider_id) 10 | print("Rider Name=" + rider_name) 11 | 12 | riderid = rider_name + "@example.com" 13 | rider_email = riderid 14 | print("Rider ID= " + riderid) 15 | 16 | rider_mobile = "+11609467790" 17 | print("Rider Mobile = " + rider_mobile) 18 | 19 | pickUpDateTime = datetime.datetime.utcnow().strftime("%Y-%m-%dT%H:%M:%SZ") 20 | id = ''.join(["%s" % randint(0, 9) for num in range(0, 7)]) 21 | tripinfo = pickUpDateTime +"," + id 22 | print("Trip Info= " + tripinfo) 23 | 24 | status = "Booked" 25 | print("Status=" + status) 26 | 27 | tripInfo = { 28 | "riderid" : riderid, 29 | "tripinfo" : tripinfo, 30 | "RIDER_ID" : rider_id, 31 | "RIDER_MOBILE" : rider_mobile, 32 | "PICKUP_DATETIME" : pickUpDateTime, 33 | "RIDER_NAME" : rider_name, 34 | "RIDER_EMAIL" : rider_email, 35 | "Status" : status 36 | } 37 | 38 | print("Trip Information =" + json.dumps(tripInfo, indent=2)) 39 | 40 | response = updateTripInfo(tripInfo) 41 | print("Trip information has been updated to Trips table") 42 | 43 | print("Rider Booking Trip Informaiton =" + json.dumps(response['Attributes'], indent = 4, cls=DecimalEncoder) ) -------------------------------------------------------------------------------- /src/ddb-python-script/util.py: -------------------------------------------------------------------------------- 1 | import boto3 2 | import decimal 3 | import json 4 | from botocore.exceptions import ClientError 5 | 6 | # Helper class to convert a DynamoDB item to JSON. 7 | class DecimalEncoder(json.JSONEncoder): 8 | def default(self, o): 9 | if isinstance(o, decimal.Decimal): 10 | if o % 1 > 0: 11 | return float(o) 12 | else: 13 | return int(o) 14 | return super(DecimalEncoder, self).default(o) 15 | 16 | # Get the service resource. 17 | dynamodb = boto3.resource('dynamodb') 18 | 19 | table = dynamodb.Table('aws-db-workshop-trips') 20 | 21 | 22 | def getTripInfo(infoReq): 23 | 24 | try: 25 | response = table.get_item( 26 | Key={ 27 | "riderid": infoReq['riderid'], 28 | "tripinfo": infoReq['tripinfo'] 29 | } 30 | ) 31 | 32 | try: 33 | print("GetItem succeeded...Response" + json.dumps(response, indent=4)) 34 | except TypeError as te: 35 | print("!Warning! = " + str(te)) 36 | print("GetItem succeeded...Response =") 37 | print(response) 38 | 39 | if 'Item' in response: 40 | return response['Item'] 41 | else: 42 | return "Item not found in table" 43 | 44 | except ClientError as e: 45 | print("Put Item Error=" + e.response['Error']['Message']) 46 | 47 | 48 | def updateTripInfo(newInfo, expectedTripStatus = None): 49 | 50 | 51 | key = { 52 | 'riderid': newInfo['riderid'], 53 | 'tripinfo':newInfo['tripinfo'], 54 | } 55 | 56 | updateExp = "SET " 57 | 58 | expAttrNames={ 59 | "#st": "STATUS" 60 | } 61 | 62 | updateExp += "#st = :s, " 63 | newItem = { 64 | ':s': newInfo['Status'] 65 | } 66 | 67 | if(expectedTripStatus is None): 68 | conditionalExp = "attribute_not_exists(#st)" 69 | else: 70 | newItem.update({":ps": expectedTripStatus}) 71 | conditionalExp = "#st=:ps" 72 | 73 | if "RIDER_ID" in newInfo: 74 | updateExp += "RIDER_ID = :rid, " 75 | newItem.update({":rid": newInfo['RIDER_ID']}) 76 | if 'RIDER_MOBILE' in newInfo: 77 | updateExp += "RIDER_MOBILE = :rm, " 78 | newItem.update({":rm": newInfo['RIDER_MOBILE']}) 79 | if 'PICKUP_DATETIME' in newInfo: 80 | updateExp += "PICKUP_DATETIME = :pdt, " 81 | newItem.update({":pdt": newInfo['PICKUP_DATETIME']}) 82 | if 'RIDER_NAME' in newInfo: 83 | updateExp += "RIDER_NAME = :rn, " 84 | newItem.update({":rn" : newInfo['RIDER_NAME']}) 85 | if 'VENDOR_ID' in newInfo: 86 | updateExp += "VENDOR_ID = :vid, " 87 | newItem.update({":vid" : newInfo['VENDOR_ID']}) 88 | if 'PICKUP_LONGITUDE' in newInfo: 89 | updateExp += "PICKUP_LONGITUDE = :pickLong, " 90 | newItem.update({":pickLong" : decimal.Decimal(newInfo['PICKUP_LONGITUDE'])}) 91 | if 'TRIP_TYPE' in newInfo: 92 | updateExp += "TRIP_TYPE = :tt, " 93 | newItem.update({":tt" : newInfo['TRIP_TYPE']}) 94 | if 'STORE_AND_FWD_FLAG' in newInfo: 95 | updateExp += "STORE_AND_FWD_FLAG = :saf, " 96 | newItem.update({":saf" : newInfo['STORE_AND_FWD_FLAG']}) 97 | if 'DROPOFF_LATITUDE' in newInfo: 98 | updateExp += "DROPOFF_LATITUDE = :dropLat, " 99 | newItem.update({":dropLat" : decimal.Decimal(newInfo['DROPOFF_LATITUDE'])}) 100 | if 'RATE_CODE_ID' in newInfo: 101 | updateExp += "RATE_CODE_ID = :rcid, " 102 | newItem.update({":rcid" : newInfo['RATE_CODE_ID']}) 103 | if 'TOLLS_AMOUNT' in newInfo: 104 | updateExp += "TOLLS_AMOUNT = :tllamt, " 105 | newItem.update({":tllamt" : decimal.Decimal(newInfo['TOLLS_AMOUNT'])}) 106 | if 'IMPROVEMENT_SURCHARGE' in newInfo: 107 | updateExp += "IMPROVEMENT_SURCHARGE = :isc, " 108 | newItem.update({":isc" : decimal.Decimal(newInfo['IMPROVEMENT_SURCHARGE'])}) 109 | if 'TIP_AMOUNT' in newInfo: 110 | updateExp += "TIP_AMOUNT = :tpamt, " 111 | newItem.update({":tpamt" : decimal.Decimal(newInfo['TIP_AMOUNT'])}) 112 | if 'DROPOFF_DATETIME' in newInfo: 113 | updateExp += "DROPOFF_DATETIME = :drpdt, " 114 | newItem.update({":drpdt" : newInfo['DROPOFF_DATETIME']}) 115 | if 'CAB_TYPE_ID' in newInfo: 116 | updateExp += "CAB_TYPE_ID = :ctid, " 117 | newItem.update({":ctid" : newInfo['CAB_TYPE_ID']}) 118 | if 'DRIVER_NAME' in newInfo: 119 | updateExp += "DRIVER_NAME = :dn, " 120 | newItem.update({":dn" : newInfo['DRIVER_NAME']}) 121 | if 'PICKUP_LATITUDE' in newInfo: 122 | updateExp += "PICKUP_LATITUDE = :pickLat, " 123 | newItem.update({":pickLat" : decimal.Decimal(newInfo['PICKUP_LATITUDE'])}) 124 | if 'TRIP_DISTANCE' in newInfo: 125 | updateExp += "TRIP_DISTANCE = :trpDist, " 126 | newItem.update({":trpDist" : decimal.Decimal(newInfo['TRIP_DISTANCE'])}) 127 | if 'VEHICLE_ID' in newInfo: 128 | updateExp += "VEHICLE_ID = :vhid, " 129 | newItem.update({":vhid" : newInfo['VEHICLE_ID']}) 130 | if 'TOTAL_AMOUNT' in newInfo: 131 | updateExp += "TOTAL_AMOUNT = :tamt, " 132 | newItem.update({":tamt" : decimal.Decimal(newInfo['TOTAL_AMOUNT'])}) 133 | if 'MTA_TAX' in newInfo: 134 | updateExp += "MTA_TAX = :mtax, " 135 | newItem.update({":mtax" : decimal.Decimal(newInfo['MTA_TAX'])}) 136 | if 'DROPOFF_LONGITUDE' in newInfo: 137 | updateExp += "DROPOFF_LONGITUDE = :dropLong, " 138 | newItem.update({":dropLong" : decimal.Decimal(newInfo['DROPOFF_LONGITUDE'])}) 139 | if 'PAYMENT_TYPET' in newInfo: 140 | updateExp += "PAYMENT_TYPET = :pmtt, " 141 | newItem.update({":pmtt" : newInfo['PAYMENT_TYPET']}) 142 | if 'DRIVER_ID' in newInfo: 143 | updateExp += "DRIVER_ID = :did, " 144 | newItem.update({":did" : int(newInfo['DRIVER_ID'])}) 145 | if 'DRIVER_EMAIL' in newInfo: 146 | updateExp += "DRIVER_EMAIL = :de, " 147 | newItem.update({":de" : newInfo['DRIVER_EMAIL']}) 148 | updateExp += "driverid = :drid, " 149 | newItem.update({":drid" : newInfo['DRIVER_EMAIL']}) 150 | if 'EXTRA' in newInfo: 151 | updateExp += "EXTRA = :ext, " 152 | newItem.update({":ext" : decimal.Decimal(newInfo['EXTRA'])}) 153 | if 'FARE_AMOUNT' in newInfo: 154 | updateExp += "FARE_AMOUNT = :famt, " 155 | newItem.update({":famt" : decimal.Decimal(newInfo['FARE_AMOUNT'])}) 156 | if 'PASSENGER_COUNT' in newInfo: 157 | updateExp += "PASSENGER_COUNT = :psgCnt, " 158 | newItem.update({":psgCnt" : newInfo['PASSENGER_COUNT']}) 159 | if 'RIDER_EMAIL' in newInfo: 160 | updateExp+= "RIDER_EMAIL = :re, " 161 | newItem.update({":re" : newInfo['RIDER_EMAIL']}) 162 | if 'DRIVER_MOBILE' in newInfo: 163 | updateExp+= "DRIVER_MOBILE = :dm, " 164 | newItem.update({":dm" : newInfo['DRIVER_MOBILE']}) 165 | if 'tripinfo' in newInfo: 166 | updateExp += "ID = :id, " 167 | newItem.update({":id" : int(newInfo['tripinfo'].split(',')[1])}) 168 | if 'DriverDetails' in newInfo: 169 | updateExp += "DriverDetails = :ddet, " 170 | newItem.update({":ddet" : json.dumps(newInfo['DriverDetails'], indent=2, cls=DecimalEncoder)}) 171 | 172 | # Remove last 2 charectors , and space from the update expression 173 | updateExp = updateExp[:-2] 174 | 175 | print("Key= " + json.dumps(key, indent=2)) 176 | print("UpdateExpression= " + updateExp) 177 | print ("ConditionExpression = " + conditionalExp) 178 | print ("ExpressionAttributeValues= " + json.dumps(newItem, indent=2, cls=DecimalEncoder)) 179 | 180 | try: 181 | response = table.update_item( 182 | Key = key, 183 | UpdateExpression = updateExp, 184 | ExpressionAttributeValues = newItem, 185 | ExpressionAttributeNames = expAttrNames, 186 | ConditionExpression = conditionalExp, 187 | ReturnValues = "ALL_NEW" 188 | ) 189 | print("UpdateItem succeeded...Response = " + json.dumps(response, indent=4, cls=DecimalEncoder)) 190 | return response 191 | except ClientError as e: 192 | if e.response['Error']['Code'] == "ConditionalCheckFailedException": 193 | print("UpdateItem Item Error. Conditional check failed =" + e.response['Error']['Message']) 194 | raise 195 | 196 | 197 | 198 | -------------------------------------------------------------------------------- /src/ddb-stream-processor/lambda_function.py: -------------------------------------------------------------------------------- 1 | import json 2 | import os 3 | import pg8000 4 | 5 | from datetime import datetime 6 | 7 | print('Loading function') 8 | 9 | PGDATABASE = os.environ['PG_DATABASE'] 10 | PGHOST = os.environ['PG_HOST'] 11 | PGPORT = int(os.environ['PG_PORT'].strip()) 12 | PGUSER = os.environ['PG_USER'] 13 | PGPASSWORD = os.environ['PG_PASSWORD'] 14 | 15 | print ("DB Connection. DataBase= " + PGDATABASE + ", Host= " + PGHOST + ", PORT= " + str(PGPORT) + ", User= " + PGUSER) 16 | 17 | # This function filters out the records from the DynamoDB stream that represent trip completion i.e. STATUS == Completed 18 | def getTripCompletionRecords(ddbRecords): 19 | 20 | completedTripList = [] 21 | 22 | for record in ddbRecords: 23 | if (record['eventName'] == "MODIFY"): #The ride completion event is always an update an existed item for the ride entry 24 | newInfo = record["dynamodb"]["NewImage"] 25 | 26 | print(newInfo) 27 | 28 | if('STATUS' in newInfo and newInfo['STATUS']['S'] == "Completed"): 29 | 30 | completedtripInfo = { 31 | "rider_mobile": "" if 'RIDER_MOBILE' not in newInfo else newInfo['RIDER_MOBILE']['S'], 32 | "pickup_datetime": "" if 'PICKUP_DATETIME' not in newInfo else datetime.strptime(newInfo['PICKUP_DATETIME']['S'], "%Y-%m-%dT%H:%M:%S%z").strftime('%Y-%m-%d %H:%M:%S'), 33 | "rider_name": "" if 'RIDER_NAME' not in newInfo else newInfo['RIDER_NAME']['S'], 34 | "vendor_id": "" if 'VENDOR_ID' not in newInfo else newInfo['VENDOR_ID']['N'], 35 | "pickup_longitude": "" if 'PICKUP_LONGITUDE' not in newInfo else newInfo['PICKUP_LONGITUDE']['N'], 36 | "trip_type": "" if 'TRIP_TYPE' not in newInfo else newInfo['TRIP_TYPE']['N'], 37 | "store_and_fwd_flag": "" if 'STORE_AND_FWD_FLAG' not in newInfo else newInfo['STORE_AND_FWD_FLAG']['S'], 38 | "dropoff_latitude": "" if 'DROPOFF_LATITUDE' not in newInfo else newInfo['DROPOFF_LATITUDE']['N'], 39 | "rate_code_id": "" if 'RATE_CODE_ID' not in newInfo else newInfo['RATE_CODE_ID']['N'], 40 | "tolls_amount": "" if 'TOLLS_AMOUNT' not in newInfo else newInfo['TOLLS_AMOUNT']['N'], 41 | "improvement_surcharge": "" if 'IMPROVEMENT_SURCHARGE' not in newInfo else newInfo['IMPROVEMENT_SURCHARGE']['N'], 42 | "tip_amount": "" if 'TIP_AMOUNT' not in newInfo else newInfo['TIP_AMOUNT']['N'], 43 | "dropoff_datetime": "" if 'DROPOFF_DATETIME' not in newInfo else datetime.strptime(newInfo['DROPOFF_DATETIME']['S'], "%Y-%m-%dT%H:%M:%S%z").strftime('%Y-%m-%d %H:%M:%S'), 44 | "cab_type_id": "" if 'CAB_TYPE_ID' not in newInfo else newInfo['CAB_TYPE_ID']['N'], 45 | "driver_name": "" if 'DRIVER_NAME' not in newInfo else newInfo['DRIVER_NAME']['S'], 46 | "pickup_latitude": "" if 'PICKUP_LATITUDE' not in newInfo else newInfo['PICKUP_LATITUDE']['N'], 47 | "trip_distance": "" if 'TRIP_DISTANCE' not in newInfo else newInfo['TRIP_DISTANCE']['N'], 48 | "vehicle_id": "" if 'VEHICLE_ID' not in newInfo else newInfo['VEHICLE_ID']['S'], 49 | "total_amount": "" if 'TOTAL_AMOUNT' not in newInfo else newInfo['TOTAL_AMOUNT']['N'], 50 | "mta_tax": "" if 'MTA_TAX' not in newInfo else newInfo['MTA_TAX']['N'], 51 | "dropoff_longitude": "" if 'DROPOFF_LONGITUDE' not in newInfo else newInfo['DROPOFF_LONGITUDE']['N'], 52 | "payment_type": "" if 'PAYMENT_TYPET' not in newInfo else newInfo['PAYMENT_TYPET']['N'], 53 | "driver_id": "" if 'DRIVER_ID' not in newInfo else newInfo['DRIVER_ID']['N'], 54 | "driver_email": "" if 'DRIVER_EMAIL' not in newInfo else newInfo['DRIVER_EMAIL']['S'], 55 | "trip_Info": "" if 'tripinfo' not in newInfo else newInfo['tripinfo']['S'], 56 | "riderId": "" if 'RIDER_ID' not in newInfo else newInfo['RIDER_ID']['N'], 57 | "extra": "" if 'EXTRA' not in newInfo else newInfo['EXTRA']['N'], 58 | "fare_amount": "" if 'FARE_AMOUNT' not in newInfo else newInfo['FARE_AMOUNT']['N'], 59 | "passenger_count": "" if 'PASSENGER_COUNT' not in newInfo else newInfo['PASSENGER_COUNT']['N'], 60 | "rider_email": "" if 'RIDER_EMAIL' not in newInfo else newInfo['RIDER_EMAIL']['S'], 61 | "driver_mobile": "" if 'DRIVER_MOBILE' not in newInfo else newInfo['DRIVER_MOBILE']['S'], 62 | "status": "Completed" 63 | } 64 | # "<>": "" if 'driverid' not in newInfo else newInfo['driverid']['S'], 65 | # "<>": "" if 'DriverDetails' not in newInfo else newInfo['DriverDetails']['S'], 66 | # "<>": "" if 'Vehicle Details' not in newInfo else newInfo['Vehicle Details']['S'], 67 | # "<>": "" if 'ID' not in newInfo else newInfo['ID']['S'], 68 | 69 | print("Completed Trip Information = " + json.dumps(completedtripInfo, indent = 2)) 70 | 71 | completedTripList.append(completedtripInfo) 72 | 73 | return completedTripList 74 | 75 | def createInsertSQLQueries(completedTripList): 76 | insertSQLQuries = [] 77 | 78 | queryStringBase = """INSERT INTO public.trips ( 79 | rider_id, driver_id, rider_name, rider_mobile, rider_email, trip_info, driver_name, 80 | driver_email, driver_mobile, vehicle_id, cab_type_id, vendor_id, pickup_datetime, 81 | dropoff_datetime, store_and_fwd_flag, rate_code_id, pickup_longitude, pickup_latitude, 82 | dropoff_longitude, dropoff_latitude, passenger_count, trip_distance, fare_amount, extra, 83 | mta_tax, tip_amount, tolls_amount, ehail_fee, improvement_surcharge, total_amount, payment_type, 84 | trip_type, pickup_location_id, dropoff_location_id, status)""" 85 | 86 | for tripInfo in completedTripList: 87 | queryString = queryStringBase + " VALUES(" 88 | queryString += tripInfo['riderId'] + "," + tripInfo['driver_id'] + ",'" + tripInfo['rider_name'] + "','" + tripInfo['rider_mobile'] + "','" + tripInfo['rider_email'] + "','" + tripInfo['trip_Info'] + "','" 89 | queryString += tripInfo['driver_name'] + "','" + tripInfo['driver_email'] + "','" + tripInfo['driver_mobile'] + "','" + tripInfo['vehicle_id'] + "'," + tripInfo['cab_type_id'] + "," 90 | queryString += tripInfo['vendor_id'] + ",'" + tripInfo['pickup_datetime'] + "','" + tripInfo['dropoff_datetime'] + "','" + tripInfo['store_and_fwd_flag'] + "'," + tripInfo['rate_code_id'] + "," 91 | queryString += tripInfo['pickup_longitude'] + "," + tripInfo['pickup_latitude'] + "," + tripInfo['dropoff_longitude'] + "," + tripInfo['dropoff_latitude'] + "," + tripInfo['passenger_count'] + "," 92 | queryString += tripInfo['trip_distance'] + "," + tripInfo['fare_amount'] + "," + tripInfo['extra'] + "," + tripInfo['mta_tax'] + "," + tripInfo['tip_amount'] + "," 93 | queryString += tripInfo['tolls_amount'] + "," + "0" + "," + tripInfo['improvement_surcharge'] + "," + tripInfo['total_amount'] + "," + tripInfo['payment_type'] + "," 94 | queryString += tripInfo['trip_type'] + "," + "0" + "," + "0" + ",'" + tripInfo['status'] + "');" 95 | 96 | print("Trip information =" + json.dumps(tripInfo, indent = 2)) 97 | print ("Insert query string =" + queryString) 98 | 99 | insertSQLQuries.append(queryString) 100 | 101 | return insertSQLQuries 102 | 103 | def pusblishTripCompletionInfo(sqlCmds): 104 | 105 | conn = pg8000.connect(database=PGDATABASE, host=PGHOST, port=PGPORT, user=PGUSER, password=PGPASSWORD) 106 | cursor = conn.cursor() 107 | 108 | for cmd in sqlCmds: 109 | print("Executing Query:" + cmd) 110 | cursor.execute(cmd) 111 | print("Completed Executing Query:" + cmd) 112 | 113 | #queryOuts = cursor.fetchall() 114 | cursor.close() 115 | conn.commit() 116 | 117 | #for out in queryOuts: 118 | # print(out) 119 | 120 | return 121 | 122 | def lambda_handler(event, context): 123 | print("Received event: " + json.dumps(event, indent=2)) 124 | 125 | print("Filtering trip completion records ... InProgress") 126 | completedTripInfoList = getTripCompletionRecords(event['Records']) 127 | print("Filtering trip completion records ... Completed") 128 | 129 | if(len(completedTripInfoList) > 0 ): 130 | print("Create insert SQL commands ... InProgress") 131 | insertSQLQueries = createInsertSQLQueries(completedTripInfoList) 132 | print("Create insert SQL commands ... Completed") 133 | 134 | print("Publish completed trip information ... InProgress") 135 | pusblishTripCompletionInfo(insertSQLQueries) 136 | print("Publish completed trip information ... Completed") 137 | else: 138 | print("No Trip Completion Records found.") 139 | 140 | return 'Successfully processed {} records.'.format(len(event['Records'])) -------------------------------------------------------------------------------- /src/ddb-stream-processor/template.yaml: -------------------------------------------------------------------------------- 1 | AWSTemplateFormatVersion: '2010-09-09' 2 | Transform: 'AWS::Serverless-2016-10-31' 3 | Description: Sam application to deploy lambda. 4 | Parameters: 5 | FunctionNameParameter: 6 | Type: String 7 | Default: aws-db-workshop-ddb-stream-processor 8 | Description: Name of the Lambda function. Default is aws-db-workshop-ddb-stream-processor 9 | LambdaLayerNameParameter: 10 | Type: String 11 | Description: Name of the Lambda Layer. 12 | DatabaseName: 13 | Type: String 14 | Description: Name of the Amazon Aurora database containing the trips table 15 | DatabaseHostName: 16 | Type: String 17 | Description: Name of the host of the Amazon Aurora database containing the trips table 18 | DatabasePort: 19 | Type: Number 20 | Default: 5432 21 | Description: Port number of the Amazon Aurora database containing the trips table. Default is 5432 22 | DatabaseUserName: 23 | Type: String 24 | Description: User name of the Amazon Aurora database containing the trips table 25 | DatabasePassword: 26 | Type: String 27 | Description: Password of the Amazon Aurora database containing the trips table 28 | DDBTableName: 29 | Type: String 30 | Description: Name of the Amazon DynamoDB table. Default name is 'aws-db-workshop-trips' 31 | Default: aws-db-workshop-trips 32 | DDBStreamName: 33 | Type: String 34 | Description: Name of the Amazon DynamoDB stream 35 | SecurityGroupIds: 36 | Type: List 37 | Description: Security Group IDs that Lambda will use 38 | VpcSubnetIds: 39 | Type: List 40 | Description: VPC Subnet IDs that Lambda will use (min 2 for HA) 41 | Resources: 42 | PGConnectionLayer: 43 | Type: AWS::Serverless::LayerVersion 44 | Properties: 45 | LayerName: !Ref LambdaLayerNameParameter 46 | Description: Python PostgreSQL Connection Library 47 | ContentUri: dependencies/pg8000-layer.zip 48 | CompatibleRuntimes: 49 | - python3.7 50 | LicenseInfo: 'BSD' 51 | RetentionPolicy: Delete 52 | TripsDDBStreamProcessorFunction: 53 | Type: 'AWS::Serverless::Function' 54 | Properties: 55 | FunctionName: !Ref FunctionNameParameter 56 | Handler: lambda_function.lambda_handler 57 | Runtime: python3.7 58 | CodeUri: . 59 | Description: Process completed taxi trip information from Amazon DynamoDB Streams and publishes the information to the trips table in Amazon Aurora database 60 | MemorySize: 3008 61 | Timeout: 900 62 | Layers: 63 | - !Ref PGConnectionLayer 64 | Events: 65 | Stream: 66 | Type: DynamoDB 67 | Properties: 68 | Stream: 69 | !Sub 70 | - 'arn:${AWS::Partition}:dynamodb:${AWS::Region}:${AWS::AccountId}:table/${tableName}/${streamName}' 71 | - {tableName: !Ref DDBTableName, streamName: !Ref DDBStreamName } 72 | StartingPosition: LATEST 73 | BatchSize: 10 74 | Enabled: True 75 | VpcConfig: 76 | SecurityGroupIds: !Ref SecurityGroupIds 77 | SubnetIds : !Ref VpcSubnetIds 78 | Policies: 79 | - VPCAccessPolicy: {} 80 | - DynamoDBStreamReadPolicy: 81 | TableName: !Ref DDBTableName 82 | StreamName: !Ref DDBStreamName 83 | Environment: 84 | Variables: 85 | PG_DATABASE: !Ref DatabaseName 86 | PG_HOST: !Ref DatabaseHostName 87 | PG_PORT: !Ref DatabasePort 88 | PG_USER: !Ref DatabaseUserName 89 | PG_PASSWORD : !Ref DatabasePassword 90 | Tags: 91 | Name: !Ref FunctionNameParameter 92 | Purpose: 'aws-db-workshop' -------------------------------------------------------------------------------- /src/drop_taxi_schema.sql: -------------------------------------------------------------------------------- 1 | set search_path=public; 2 | \set AUTOCOMMIT off 3 | 4 | /* This file contains DDL for dropping relational schema in PostgreSQL */ 5 | 6 | 7 | drop table trips cascade; 8 | drop table payment cascade; 9 | drop table billing cascade; 10 | drop table riders cascade; 11 | drop table drivers cascade; 12 | drop table rate_codes cascade; 13 | drop table trip_status cascade; 14 | drop table payment_types cascade; 15 | drop table car_model cascade; 16 | drop table cab_types cascade; 17 | 18 | drop sequence billing_cycle_seq; 19 | 20 | drop procedure billingandpayments(); 21 | 22 | commit; 23 | -------------------------------------------------------------------------------- /src/taxi-ride-workflow/dependencies/util.zip: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-rds-purpose-built-workshop/be574ce7c00e5badd8d96e457d17900bb58a0897/src/taxi-ride-workflow/dependencies/util.zip -------------------------------------------------------------------------------- /src/taxi-ride-workflow/driver-accept-trip/driver-accept-trip.py: -------------------------------------------------------------------------------- 1 | import json 2 | import random 3 | from decimal import getcontext, Decimal 4 | from random import randint 5 | 6 | from util import updateTripInfo, getTripInfo, DecimalEncoder 7 | 8 | 9 | def lambda_handler(event, context): 10 | 11 | driver_details = [ 12 | {"driver_email" : "driver556550@taxi.com", "driver_id": "556550", "driver_name": "driver556550", "driver_mobile": "+11720912154", "vehicle_details": {"id": "QSY353471", "type" : 2}}, 13 | {"driver_email" : "driver541829@taxi.com", "driver_id": "541829", "driver_name": "driver541829", "driver_mobile": "+11264112049", "vehicle_details": {"id": "GTR508161", "type" : 2}}, 14 | {"driver_email" : "driver507977@taxi.com", "driver_id": "507977", "driver_name": "driver507977", "driver_mobile": "+11088418780", "vehicle_details": {"id": "XVJ356159", "type" : 2}}, 15 | {"driver_email" : "driver551153@taxi.com", "driver_id": "551153", "driver_name": "driver551153", "driver_mobile": "+11240868167", "vehicle_details": {"id": "CPX160101", "type" : 2}}, 16 | {"driver_email" : "driver520045@taxi.com", "driver_id": "520045", "driver_name": "driver520045", "driver_mobile": "+11751510159", "vehicle_details": {"id": "HHR298952", "type" : 2}}, 17 | {"driver_email" : "driver514040@taxi.com", "driver_id": "514040", "driver_name": "driver514040", "driver_mobile": "+11661484862", "vehicle_details": {"id": "TLA210480", "type" : 2}}, 18 | {"driver_email" : "driver527336@taxi.com", "driver_id": "527336", "driver_name": "driver527336", "driver_mobile": "+11564984764", "vehicle_details": {"id": "OVY229214", "type" : 2}}, 19 | {"driver_email" : "driver510909@taxi.com", "driver_id": "510909", "driver_name": "driver510909", "driver_mobile": "+11261783124", "vehicle_details": {"id": "UDT200764", "type" : 2}}, 20 | {"driver_email" : "driver549736@taxi.com", "driver_id": "549736", "driver_name": "driver549736", "driver_mobile": "+11561755450", "vehicle_details": {"id": "ORX460076", "type" : 2}}, 21 | {"driver_email" : "driver528204@taxi.com", "driver_id": "528204", "driver_name": "driver528204", "driver_mobile": "+11185992795", "vehicle_details": {"id": "PXX248130", "type" : 2}} 22 | ] 23 | 24 | #rider_id = "person69257@example.com" 25 | 26 | #trip_info = input("Enter your tripinfo : ") 27 | #trip_info = event["key1"] 28 | #trip_info = '2021-10-12T01:35:17Z,9150417' 29 | 30 | i_rider_id= int(event['queryStringParameters']['rider_id']) 31 | trip_info = event['queryStringParameters']['trip_info'] 32 | 33 | rider_name = "person" + str(i_rider_id) 34 | rider_id = rider_name + "@example.com" 35 | 36 | vendor_id = 2 37 | 38 | pickup_longitude = str(round(random.uniform(-74,-73),6)) 39 | pickup_latitude = str(round(random.uniform(40,41),6)) 40 | driver_info = driver_details[randint(0, 9)] 41 | 42 | tripAcceptInfo = { 43 | "riderid" : rider_id, 44 | "tripinfo" : trip_info, 45 | "VENDOR_ID" : vendor_id, 46 | "PICKUP_LONGITUDE" : pickup_longitude, 47 | "PICKUP_LATITUDE" : pickup_latitude, 48 | "TRIP_TYPE" : 2, 49 | "STORE_AND_FWD_FLAG" : "N", 50 | "CAB_TYPE_ID" : driver_info['vehicle_details']['type'], 51 | "DRIVER_NAME" : driver_info['driver_name'], 52 | "VEHICLE_ID" : driver_info['vehicle_details']['id'], 53 | "DRIVER_ID" : driver_info['driver_id'], 54 | "DRIVER_EMAIL" : driver_info['driver_email'], 55 | "DRIVER_MOBILE" : driver_info['driver_mobile'], 56 | "DriverDetails" : { 57 | "Name" : driver_info['driver_name'], 58 | "Vehicle Details" : { 59 | "id" : driver_info['vehicle_details']['id'], 60 | "type": driver_info['vehicle_details']['type'] 61 | } 62 | }, 63 | "Status" : "InProgress" 64 | } 65 | 66 | print("Trip accept information ="+ json.dumps(tripAcceptInfo, indent=2)) 67 | 68 | response = updateTripInfo(tripAcceptInfo, "Booked") 69 | print("Trip accept information has been updated to Trips table") 70 | 71 | print("Driver Accept Trip Information =" + json.dumps(response['Attributes'], indent = 4, cls=DecimalEncoder)) 72 | 73 | responseObjects = {} 74 | responseObjects['statusCode'] = 200 75 | responseObjects['headers'] = {} 76 | responseObjects['headers']['Content-Type'] = 'application/json' 77 | responseObjects['body'] = json.dumps(response['Attributes'], indent = 4, cls=DecimalEncoder) 78 | 79 | return responseObjects -------------------------------------------------------------------------------- /src/taxi-ride-workflow/driver-complete-trip/driver-complete-trip.py: -------------------------------------------------------------------------------- 1 | import datetime 2 | import json 3 | import random 4 | from random import randint 5 | 6 | from util import updateTripInfo, getTripInfo, DecimalEncoder 7 | 8 | def lambda_handler(event, context): 9 | 10 | rate_codes = [1, 2, 3, 4, 5, 6, 99] 11 | 12 | #rider_id = "person69257@example.com" 13 | #trip_info = input("Enter your tripinfo : ") 14 | #trip_info = event["key1"] 15 | 16 | i_rider_id= int(event['queryStringParameters']['rider_id']) 17 | trip_info = event['queryStringParameters']['trip_info'] 18 | 19 | rider_name = "person" + str(i_rider_id) 20 | rider_id = rider_name + "@example.com" 21 | 22 | dropoff_longitude = round(random.uniform(-74,-73),6) 23 | dropoff_latitude = round(random.uniform(40,41),6) 24 | 25 | tripCompletedInfo = { 26 | "riderid" : rider_id, 27 | "tripinfo" : trip_info, 28 | "DROPOFF_LATITUDE" : str(dropoff_latitude), 29 | "RATE_CODE_ID" : rate_codes[randint(0, 6)], 30 | "TOLLS_AMOUNT" : str(round(random.uniform(0,5),2)), 31 | "IMPROVEMENT_SURCHARGE" : str(round(random.uniform(0,1),1)), 32 | "TIP_AMOUNT" : str(round(random.uniform(0,10),2)), 33 | "DROPOFF_DATETIME" : datetime.datetime.utcnow().strftime("%Y-%m-%dT%H:%M:%SZ"), 34 | "TRIP_DISTANCE" : randint(1, 50), 35 | "TOTAL_AMOUNT" : str(round(random.uniform(5,150),2)), 36 | "MTA_TAX" : str(round(random.uniform(0,1),1)), 37 | "DROPOFF_LONGITUDE" : str(dropoff_longitude), 38 | "PAYMENT_TYPET" : randint(1, 7), 39 | "EXTRA" : str(round(random.uniform(0,1),1)), 40 | "FARE_AMOUNT" : str(round(random.uniform(5,150),2)), 41 | "PASSENGER_COUNT": randint(1, 7), 42 | "Status" : "Completed" 43 | } 44 | 45 | print ("Trip completion information = ", json.dumps(tripCompletedInfo, indent=2)) 46 | 47 | response = updateTripInfo(tripCompletedInfo, "InProgress") 48 | print("Trip completion information has been updated to Trips table") 49 | 50 | print("Driver trip completion information =" + json.dumps(response['Attributes'], indent = 4, cls=DecimalEncoder)) 51 | 52 | responseObjects = {} 53 | responseObjects['statusCode'] = 200 54 | responseObjects['headers'] = {} 55 | responseObjects['headers']['Content-Type'] = 'application/json' 56 | responseObjects['body'] = json.dumps(response['Attributes'], indent = 4, cls=DecimalEncoder) 57 | 58 | return responseObjects -------------------------------------------------------------------------------- /src/taxi-ride-workflow/rider-book-trip/rider-book-trip.py: -------------------------------------------------------------------------------- 1 | import datetime 2 | import json 3 | from random import randint 4 | 5 | from util import updateTripInfo, getTripInfo, DecimalEncoder 6 | 7 | def lambda_handler(event, context): 8 | 9 | #print(event) 10 | 11 | #rider_id = 69257 12 | #rider_mobile = "+11609467790" 13 | 14 | rider_id = int(event['queryStringParameters']['rider_id']) 15 | rider_mobile = event['queryStringParameters']['rider_mobile'] 16 | 17 | rider_name = "person" + str(rider_id) 18 | print("Rider Name=" + rider_name) 19 | 20 | riderid = rider_name + "@example.com" 21 | rider_email = riderid 22 | print("Rider ID= " + riderid) 23 | 24 | print("Rider Mobile = " + rider_mobile) 25 | 26 | pickUpDateTime = datetime.datetime.utcnow().strftime("%Y-%m-%dT%H:%M:%SZ") 27 | id = ''.join(["%s" % randint(0, 9) for num in range(0, 7)]) 28 | tripinfo = pickUpDateTime +"," + id 29 | print("Trip Info= " + tripinfo) 30 | 31 | status = "Booked" 32 | print("Status=" + status) 33 | 34 | tripInfo = { 35 | "riderid" : riderid, 36 | "tripinfo" : tripinfo, 37 | "RIDER_ID" : rider_id, 38 | "RIDER_MOBILE" : rider_mobile, 39 | "PICKUP_DATETIME" : pickUpDateTime, 40 | "RIDER_NAME" : rider_name, 41 | "RIDER_EMAIL" : rider_email, 42 | "Status" : status 43 | } 44 | 45 | print("Trip Information =" + json.dumps(tripInfo, indent=2)) 46 | 47 | response = updateTripInfo(tripInfo) 48 | print("Trip information has been updated to Trips table") 49 | 50 | print("Rider Booking Trip Information =" + json.dumps(response['Attributes'], indent = 4, cls=DecimalEncoder)) 51 | 52 | responseObjects = {} 53 | responseObjects['statusCode'] = 200 54 | responseObjects['headers'] = {} 55 | responseObjects['headers']['Content-Type'] = 'application/json' 56 | responseObjects['body'] = json.dumps(response['Attributes'], indent = 4, cls=DecimalEncoder) 57 | 58 | return responseObjects -------------------------------------------------------------------------------- /src/taxi-ride-workflow/template.yaml: -------------------------------------------------------------------------------- 1 | AWSTemplateFormatVersion: '2010-09-09' 2 | Transform: 'AWS::Serverless-2016-10-31' 3 | Description: Sam application to deploy lambda. 4 | Parameters: 5 | TripBookingFnName: 6 | Type: String 7 | Default: rider-book-trip 8 | Description: Name of the Lambda function for Taxi Trip Booking by Rider 9 | TripAcceptFnName: 10 | Type: String 11 | Default: driver-accept-trip 12 | Description: Name of the Lambda function for Taxi Trip Acceptance by Driver 13 | TripCompleteFnName: 14 | Type: String 15 | Default: driver-complete-trip 16 | Description: Name of the Lambda function for Taxi Trip Completion by Driver 17 | LambdaLayerNameParameter: 18 | Type: String 19 | Default: util 20 | Description: Name of the Lambda Layer containing Python utility functions 21 | DDBTableName: 22 | Type: String 23 | Description: Name of the Amazon Dynamodb table. Default name is 'aws-db-workshop-trips' 24 | Default: aws-db-workshop-trips 25 | SecurityGroupIds: 26 | Type: List 27 | Description: Security Group IDs that Lambda will use 28 | VpcSubnetIds: 29 | Type: List 30 | Description: VPC Subnet IDs that Lambda will use (min 2 for HA) 31 | Resources: 32 | PythonUtilFnsLayer: 33 | Type: AWS::Serverless::LayerVersion 34 | Properties: 35 | LayerName: !Ref LambdaLayerNameParameter 36 | Description: Python utility functions for Taxi trip workflow 37 | ContentUri: dependencies/util.zip 38 | CompatibleRuntimes: 39 | - python3.7 40 | LicenseInfo: 'MIT-0' 41 | RetentionPolicy: Delete 42 | RiderBookTripFunction: 43 | Type: 'AWS::Serverless::Function' 44 | Properties: 45 | FunctionName: !Ref TripBookingFnName 46 | Handler: rider-book-trip.lambda_handler 47 | Runtime: python3.7 48 | CodeUri: ./rider-book-trip 49 | Description: Takes rider_id and rider_mobile as input and books a trip for the rider by updating ws-db-workshop-trips table in DynamoDB 50 | MemorySize: 3008 51 | Timeout: 900 52 | Layers: 53 | - !Ref PythonUtilFnsLayer 54 | VpcConfig: 55 | SecurityGroupIds: !Ref SecurityGroupIds 56 | SubnetIds : !Ref VpcSubnetIds 57 | Policies: 58 | - VPCAccessPolicy: {} 59 | - DynamoDBCrudPolicy: 60 | TableName: !Ref DDBTableName 61 | Tags: 62 | Name: !Ref TripBookingFnName 63 | Purpose: 'aws-db-workshop' 64 | DriverAcceptTripFunction: 65 | Type: 'AWS::Serverless::Function' 66 | Properties: 67 | FunctionName: !Ref TripAcceptFnName 68 | Handler: driver-accept-trip.lambda_handler 69 | Runtime: python3.7 70 | CodeUri: ./driver-accept-trip 71 | Description: Takes rider_id and trip_info as input and accepts a trip by the driver by updating ws-db-workshop-trips table in DynamoDB 72 | MemorySize: 3008 73 | Timeout: 900 74 | Layers: 75 | - !Ref PythonUtilFnsLayer 76 | VpcConfig: 77 | SecurityGroupIds: !Ref SecurityGroupIds 78 | SubnetIds : !Ref VpcSubnetIds 79 | Policies: 80 | - VPCAccessPolicy: {} 81 | - DynamoDBCrudPolicy: 82 | TableName: !Ref DDBTableName 83 | Tags: 84 | Name: !Ref TripAcceptFnName 85 | Purpose: 'aws-db-workshop' 86 | DriverCompleteTripFunction: 87 | Type: 'AWS::Serverless::Function' 88 | Properties: 89 | FunctionName: !Ref TripCompleteFnName 90 | Handler: driver-complete-trip.lambda_handler 91 | Runtime: python3.7 92 | CodeUri: ./driver-complete-trip 93 | Description: Takes rider_id and trip_info as input and completes a trip by the driver by updating ws-db-workshop-trips table in DynamoDB 94 | MemorySize: 3008 95 | Timeout: 900 96 | Layers: 97 | - !Ref PythonUtilFnsLayer 98 | VpcConfig: 99 | SecurityGroupIds: !Ref SecurityGroupIds 100 | SubnetIds : !Ref VpcSubnetIds 101 | Policies: 102 | - VPCAccessPolicy: {} 103 | - DynamoDBCrudPolicy: 104 | TableName: !Ref DDBTableName 105 | Tags: 106 | Name: !Ref TripCompleteFnName 107 | Purpose: 'aws-db-workshop' -------------------------------------------------------------------------------- /src/taxi.dmp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-rds-purpose-built-workshop/be574ce7c00e5badd8d96e457d17900bb58a0897/src/taxi.dmp --------------------------------------------------------------------------------