├── .gitignore ├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── LICENSE ├── README.md └── workshops ├── nfs-migration ├── EN │ ├── README.md │ ├── images │ │ ├── fullarch.png │ │ ├── mod1arch.png │ │ ├── mod1cli1.png │ │ ├── mod1connect.png │ │ ├── mod1output1.png │ │ ├── mod1output2.png │ │ ├── mod1ssh1.png │ │ ├── mod2arch.png │ │ ├── mod2ds1.png │ │ ├── mod2ds2.png │ │ ├── mod2ds3.png │ │ ├── mod2ds4.png │ │ ├── mod2ds5.png │ │ ├── mod2ds6.png │ │ ├── mod2ds7.png │ │ ├── mod2ds8.png │ │ ├── mod2ds9.png │ │ ├── mod2validate.png │ │ ├── mod3cli1.png │ │ ├── mod3fgw1.png │ │ ├── mod3fgw2.png │ │ ├── mod4cli1.png │ │ ├── mod4cli2.png │ │ ├── mod4ds1.png │ │ ├── mod4ds2.png │ │ ├── mod4s31.png │ │ ├── mod5arch.png │ │ ├── mod5s31.png │ │ └── module1.png │ ├── module1 │ │ └── README.md │ ├── module2 │ │ └── README.md │ ├── module3 │ │ └── README.md │ ├── module4 │ │ └── README.md │ ├── module5 │ │ └── README.md │ └── templates │ │ ├── data-migration-workshop-in-cloud.yaml │ │ └── data-migration-workshop-on-prem.yaml ├── JP │ ├── README.md │ ├── images │ │ ├── fullarch.png │ │ ├── mod1arch.png │ │ ├── mod1cli1.png │ │ ├── mod1connect.png │ │ ├── mod1output1.png │ │ ├── mod1output2.png │ │ ├── mod1ssh1.png │ │ ├── mod2arch.png │ │ ├── mod2ds1.png │ │ ├── mod2ds2.png │ │ ├── mod2ds3.png │ │ ├── mod2ds4.png │ │ ├── mod2ds5.png │ │ ├── mod2ds6.png │ │ ├── mod2ds7.png │ │ ├── mod2ds8.png │ │ ├── mod2ds9.png │ │ ├── mod2validate.png │ │ ├── mod3cli1.png │ │ ├── mod3fgw1.png │ │ ├── mod3fgw2.png │ │ ├── mod4cli1.png │ │ ├── mod4cli2.png │ │ ├── mod4ds1.png │ │ ├── mod4ds2.png │ │ ├── mod4s31.png │ │ ├── mod5arch.png │ │ ├── mod5s31.png │ │ └── module1.png │ ├── module1 │ │ └── README.md │ ├── module2 │ │ └── README.md │ ├── module3 │ │ └── README.md │ ├── module4 │ │ └── README.md │ ├── module5 │ │ └── README.md │ └── templates │ │ ├── data-migration-workshop-in-cloud.yaml │ │ └── data-migration-workshop-on-prem.yaml └── README.md ├── nfs-million-files ├── README.md ├── images │ ├── fullarch.png │ ├── mod1output1.png │ ├── mod1output2.png │ ├── mod2agentsList.png │ ├── mod2ds1.png │ ├── mod2ds2.png │ ├── mod4ds1.png │ ├── mod4ds2.png │ ├── mod4ds3.png │ ├── mod4ds4.png │ ├── mod4ds5.png │ ├── mod5ds1.png │ ├── mod6ds1.png │ ├── mod6ds2.png │ └── mod6ds3.png ├── module1 │ └── README.md ├── module2 │ └── README.md ├── module3 │ └── README.md ├── module4 │ └── README.md ├── module5 │ └── README.md ├── module6 │ └── README.md ├── module7 │ └── README.md └── templates │ ├── nfs-million-files-in-cloud.yaml │ └── nfs-million-files-on-prem.yaml └── scality-object-migration ├── EN ├── README.md ├── images │ ├── fullarch.png │ ├── mod1arch.png │ ├── mod1cli1.png │ ├── mod1connect.png │ ├── mod1output1.png │ ├── mod1output2.png │ ├── mod1ssh1.png │ ├── mod2arch.png │ ├── mod2ds1.png │ ├── mod2ds2.png │ ├── mod2ds3.png │ ├── mod2ds4.png │ ├── mod2ds5.png │ ├── mod2ds6.png │ ├── mod2ds7.png │ ├── mod2ds8.png │ ├── mod2ds9.png │ ├── mod2validate.png │ ├── mod3s1.png │ ├── mod3s2.png │ ├── mod4cli1.png │ ├── mod4ds1.png │ ├── mod4ds2.png │ ├── mod4ds3.png │ └── mod5s31.png ├── module1 │ └── README.md ├── module2 │ └── README.md ├── module3 │ └── README.md ├── module4 │ └── README.md ├── module5 │ └── README.md └── templates │ ├── data-migration-workshop-in-cloud.yaml │ └── data-migration-workshop-on-prem.yaml └── README.md /.gitignore: -------------------------------------------------------------------------------- 1 | .DS_Store 2 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | ## Code of Conduct 2 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 3 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 4 | opensource-codeofconduct@amazon.com with any additional questions or comments. 5 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing Guidelines 2 | 3 | Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional 4 | documentation, we greatly value feedback and contributions from our community. 5 | 6 | Please read through this document before submitting any issues or pull requests to ensure we have all the necessary 7 | information to effectively respond to your bug report or contribution. 8 | 9 | 10 | ## Reporting Bugs/Feature Requests 11 | 12 | We welcome you to use the GitHub issue tracker to report bugs or suggest features. 13 | 14 | When filing an issue, please check existing open, or recently closed, issues to make sure somebody else hasn't already 15 | reported the issue. Please try to include as much information as you can. Details like these are incredibly useful: 16 | 17 | * A reproducible test case or series of steps 18 | * The version of our code being used 19 | * Any modifications you've made relevant to the bug 20 | * Anything unusual about your environment or deployment 21 | 22 | 23 | ## Contributing via Pull Requests 24 | Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that: 25 | 26 | 1. You are working against the latest source on the *master* branch. 27 | 2. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already. 28 | 3. You open an issue to discuss any significant work - we would hate for your time to be wasted. 29 | 30 | To send us a pull request, please: 31 | 32 | 1. Fork the repository. 33 | 2. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change. 34 | 3. Ensure local tests pass. 35 | 4. Commit to your fork using clear commit messages. 36 | 5. Send us a pull request, answering any default questions in the pull request interface. 37 | 6. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation. 38 | 39 | GitHub provides additional document on [forking a repository](https://help.github.com/articles/fork-a-repo/) and 40 | [creating a pull request](https://help.github.com/articles/creating-a-pull-request/). 41 | 42 | 43 | ## Finding contributions to work on 44 | Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels (enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any 'help wanted' issues is a great place to start. 45 | 46 | 47 | ## Code of Conduct 48 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 49 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 50 | opensource-codeofconduct@amazon.com with any additional questions or comments. 51 | 52 | 53 | ## Security issue notifications 54 | If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github issue. 55 | 56 | 57 | ## Licensing 58 | 59 | See the [LICENSE](LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution. 60 | 61 | We may ask you to sign a [Contributor License Agreement (CLA)](http://en.wikipedia.org/wiki/Contributor_License_Agreement) for larger changes. 62 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining a copy of 4 | this software and associated documentation files (the "Software"), to deal in 5 | the Software without restriction, including without limitation the rights to 6 | use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of 7 | the Software, and to permit persons to whom the Software is furnished to do so. 8 | 9 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 10 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS 11 | FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 12 | COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER 13 | IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN 14 | CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 15 | 16 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # **AWS DataSync** 2 | 3 | ### Migration workshops 4 | 5 | © 2019 Amazon Web Services, Inc. and its affiliates. All rights reserved. 6 | This sample code is made available under the MIT-0 license. See the LICENSE file. 7 | 8 | Errors or corrections? Contact [jeffbart@amazon.com](mailto:jeffbart@amazon.com). 9 | 10 | --- 11 | 12 | In this repository you'll find a number of workshops and other resources designed to give you hands-on experience with the AWS DataSync service. The workshops can be done in any order and have no dependencies on one another. 13 | 14 | To run through each workshop, you will need an AWS Account, preferably with Admin privileges. Although the workshops are designed to be self-contained - using CloudFormation templates for easy deployment and cleanup - they should __not__ be run in production accounts. 15 | 16 | ### Workshops 17 | #### The following workshops are now being maintained in AWS [workshop studio](https://workshops.aws/) 18 | 19 | - [NFS server migration using AWS DataSync and Storage Gateway](https://catalog.us-east-1.prod.workshops.aws/datasync-nfs-server-migration-with-storage-gateway/en-US) 20 | - [Migrate millions of files using AWS DataSync](https://catalog.us-east-1.prod.workshops.aws/datasync-migrate-millions-of-files/en-US) 21 | - [Migrate to FSx Windows File Server using AWS DataSync](https://catalog.us-east-1.prod.workshops.aws/datasync-fsx-windows-migration/en-US) 22 | 23 | 24 | ### Related Workshops 25 | 26 | - [Get hands-on with online data migration options to simplify & accelerate your journey to AWS](https://github.com/aws-samples/aws-online-data-migration-workshop) -------------------------------------------------------------------------------- /workshops/nfs-migration/EN/README.md: -------------------------------------------------------------------------------- 1 | # **AWS DataSync** 2 | 3 | ### NFS server migration using AWS DataSync and AWS Storage Gateway 4 | 5 | © 2019 Amazon Web Services, Inc. and its affiliates. All rights reserved. 6 | This sample code is made available under the MIT-0 license. See the LICENSE file. 7 | 8 | Errors or corrections? Contact [jeffbart@amazon.com](mailto:jeffbart@amazon.com). 9 | 10 | --- 11 | 12 | This workshop is now being maintained in AWS [workshop studio](https://workshops.aws/) 13 | 14 | - [NFS server migration using AWS DataSync and Storage Gateway](https://catalog.us-east-1.prod.workshops.aws/datasync-nfs-server-migration-with-storage-gateway/en-US) 15 | 16 | ---- 17 | 18 | ## Workshop scenario 19 | 20 | In your data center, you have an NFS server that is starting to age out. Most of the data on the server is several years old and is only accessed for reading occasionally. There are new files being written to the server but not very often. To reduce your data center footprint and to free up resources, you would like to move the data on the NFS server into the cloud. However, you cannot yet move your application servers that access the NFS data – those need to stay on-premises to minimize latency for your users. 21 | 22 | After doing some research, you have realized that you can use [AWS DataSync](https://aws.amazon.com/datasync/) to migrate the data from your on-premises NFS server to Amazon S3, and you can use [AWS Storage Gateway](https://aws.amazon.com/storagegateway) to provide NFS access on-premises to the data once it is in S3. 23 | 24 | This workshop will walk you through this scenario, using CloudFormation templates to deploy resources and the AWS Management console to configure those resources accordingly. As shown in the architecture diagram below, an NFS server, an Application server, a DataSync agent, and a File Gateway appliance will be deployed in an AWS region simulating the on-premises environment. An S3 bucket will be created in an AWS region, simulating the AWS cloud region to which the NFS server's data will be migrated. 25 | 26 | ![](images/fullarch.png) 27 | 28 | ## Topics covered 29 | 30 | - Deploying resources using CloudFormation 31 | - Configuring a Linux NFS server 32 | - Configure CloudWatch logging for DataSync 33 | - Planning a migration of millions of files using DataSync 34 | - Using DataSync with multiple tasks running in parallel 35 | 36 | ## Prerequisites 37 | 38 | #### AWS Account 39 | 40 | In order to complete this workshop, you will need an AWS account with rights to create AWS IAM roles, EC2 instances, AWS DataSync, AWS Storage Gateway and CloudFormation stacks in the AWS regions you select. 41 | 42 | #### Software 43 | 44 | - **Internet Browser** – It is recommended that you use the latest version of Chrome or Firefox for this workshop. 45 | 46 | ## Cost 47 | 48 | It will cost approximately **3.00 USD** to run this workshop. It is recommended that you follow the cleanup instructions once you have completed the workshop to remove all deployed resources and limit ongoing costs to your AWS account. 49 | 50 | ## Related workshops 51 | 52 | - [Migrate millions of files using AWS DataSync](https://github.com/aws-samples/aws-datasync-migration-workshop/blob/master/workshops/nfs-million-files) 53 | - [Migrate to FSx Windows File Server using AWS DataSync](https://github.com/aws-samples/aws-datasync-fsx-windows-migration) 54 | - [Get hands-on with online data migration options to simplify & accelerate your journey to AWS](https://github.com/aws-samples/aws-online-data-migration-workshop) 55 | 56 | ## Workshop Modules 57 | 58 | This workshop consists of the following five modules: 59 | 60 | - [Module 1](module1/) - Deploy resources in the on-premises and in-cloud regions 61 | - [Module 2](module2/) - Initial file copy to S3 using DataSync 62 | - [Module 3](module3/) - Access S3 bucket on-premises using File Gateway 63 | - [Module 4](module4/) - One last incremental copy before cutover 64 | - [Module 5](module5/) - Cutover to File Gateway and shutdown the NFS server 65 | 66 | To get started, go to [Module 1](module1/). 67 | -------------------------------------------------------------------------------- /workshops/nfs-migration/EN/images/fullarch.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/EN/images/fullarch.png -------------------------------------------------------------------------------- /workshops/nfs-migration/EN/images/mod1arch.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/EN/images/mod1arch.png -------------------------------------------------------------------------------- /workshops/nfs-migration/EN/images/mod1cli1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/EN/images/mod1cli1.png -------------------------------------------------------------------------------- /workshops/nfs-migration/EN/images/mod1connect.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/EN/images/mod1connect.png -------------------------------------------------------------------------------- /workshops/nfs-migration/EN/images/mod1output1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/EN/images/mod1output1.png -------------------------------------------------------------------------------- /workshops/nfs-migration/EN/images/mod1output2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/EN/images/mod1output2.png -------------------------------------------------------------------------------- /workshops/nfs-migration/EN/images/mod1ssh1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/EN/images/mod1ssh1.png -------------------------------------------------------------------------------- /workshops/nfs-migration/EN/images/mod2arch.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/EN/images/mod2arch.png -------------------------------------------------------------------------------- /workshops/nfs-migration/EN/images/mod2ds1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/EN/images/mod2ds1.png -------------------------------------------------------------------------------- /workshops/nfs-migration/EN/images/mod2ds2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/EN/images/mod2ds2.png -------------------------------------------------------------------------------- /workshops/nfs-migration/EN/images/mod2ds3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/EN/images/mod2ds3.png -------------------------------------------------------------------------------- /workshops/nfs-migration/EN/images/mod2ds4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/EN/images/mod2ds4.png -------------------------------------------------------------------------------- /workshops/nfs-migration/EN/images/mod2ds5.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/EN/images/mod2ds5.png -------------------------------------------------------------------------------- /workshops/nfs-migration/EN/images/mod2ds6.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/EN/images/mod2ds6.png -------------------------------------------------------------------------------- /workshops/nfs-migration/EN/images/mod2ds7.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/EN/images/mod2ds7.png -------------------------------------------------------------------------------- /workshops/nfs-migration/EN/images/mod2ds8.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/EN/images/mod2ds8.png -------------------------------------------------------------------------------- /workshops/nfs-migration/EN/images/mod2ds9.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/EN/images/mod2ds9.png -------------------------------------------------------------------------------- /workshops/nfs-migration/EN/images/mod2validate.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/EN/images/mod2validate.png -------------------------------------------------------------------------------- /workshops/nfs-migration/EN/images/mod3cli1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/EN/images/mod3cli1.png -------------------------------------------------------------------------------- /workshops/nfs-migration/EN/images/mod3fgw1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/EN/images/mod3fgw1.png -------------------------------------------------------------------------------- /workshops/nfs-migration/EN/images/mod3fgw2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/EN/images/mod3fgw2.png -------------------------------------------------------------------------------- /workshops/nfs-migration/EN/images/mod4cli1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/EN/images/mod4cli1.png -------------------------------------------------------------------------------- /workshops/nfs-migration/EN/images/mod4cli2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/EN/images/mod4cli2.png -------------------------------------------------------------------------------- /workshops/nfs-migration/EN/images/mod4ds1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/EN/images/mod4ds1.png -------------------------------------------------------------------------------- /workshops/nfs-migration/EN/images/mod4ds2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/EN/images/mod4ds2.png -------------------------------------------------------------------------------- /workshops/nfs-migration/EN/images/mod4s31.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/EN/images/mod4s31.png -------------------------------------------------------------------------------- /workshops/nfs-migration/EN/images/mod5arch.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/EN/images/mod5arch.png -------------------------------------------------------------------------------- /workshops/nfs-migration/EN/images/mod5s31.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/EN/images/mod5s31.png -------------------------------------------------------------------------------- /workshops/nfs-migration/EN/images/module1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/EN/images/module1.png -------------------------------------------------------------------------------- /workshops/nfs-migration/EN/module1/README.md: -------------------------------------------------------------------------------- 1 | # **AWS DataSync** 2 | 3 | ### NFS server migration using AWS DataSync and AWS Storage Gateway 4 | 5 | © 2019 Amazon Web Services, Inc. and its affiliates. All rights reserved. 6 | This sample code is made available under the MIT-0 license. See the LICENSE file. 7 | 8 | Errors or corrections? Contact [jeffbart@amazon.com](mailto:jeffbart@amazon.com). 9 | 10 | --- 11 | 12 | # Module 1 13 | ## Deploy resources in the on-premises and in-cloud regions 14 | 15 | In this module, you will use CloudFormation scripts to deploy resources in two AWS regions: one that represents the on-premises environment, and one for the in-cloud environment. Once all resources have been deployed, you will mount an export from the NFS server on the Application server and verify existing files. 16 | 17 | ![](../images/mod1arch.png) 18 | 19 | ## Module Steps 20 | 21 | #### 1. Deploy AWS resources for the on-premises region 22 | 23 | 1. Click one of the launch links in the table below to deploy the **on-premises** resources using CloudFormation. To avoid errors during deployment, select a region in which you have previously created AWS resources. 24 | 25 | | **Region Code** | **Region Name** | **Launch** | 26 | | --- | --- | --- | 27 | | us-west-1 | US West (N. California) | [Launch in us-west-1](https://console.aws.amazon.com/cloudformation/home?region=us-west-1#/stacks/new?stackName=DataMigrationWorkshop-onPremResources&templateURL=https://aws-datasync-samples.s3-us-west-2.amazonaws.com/workshops/nfs-migration/data-migration-workshop-on-prem.yaml) | 28 | | us-west-2 | US West (Oregon) | [Launch in us-west-2](https://console.aws.amazon.com/cloudformation/home?region=us-west-2#/stacks/new?stackName=DataMigrationWorkshop-onPremResources&templateURL=https://aws-datasync-samples.s3-us-west-2.amazonaws.com/workshops/nfs-migration/data-migration-workshop-on-prem.yaml) | 29 | | us-east-1 | US East (N. Virginia) | [Launch in us-east-1](https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?stackName=DataMigrationWorkshop-onPremResources&templateURL=https://aws-datasync-samples.s3-us-west-2.amazonaws.com/workshops/nfs-migration/data-migration-workshop-on-prem.yaml) | 30 | | us-east-2 | US East (Ohio) | [Launch in us-east-2](https://console.aws.amazon.com/cloudformation/home?region=us-east-2#/stacks/new?stackName=DataMigrationWorkshop-onPremResources&templateURL=https://aws-datasync-samples.s3-us-west-2.amazonaws.com/workshops/nfs-migration/data-migration-workshop-on-prem.yaml) | 31 | | eu-west-1 | Ireland | [Launch in eu-west-1](https://console.aws.amazon.com/cloudformation/home?region=eu-west-1#/stacks/new?stackName=DataMigrationWorkshop-onPremResources&templateURL=https://aws-datasync-samples.s3-us-west-2.amazonaws.com/workshops/nfs-migration/data-migration-workshop-on-prem.yaml) | 32 | | eu-central-1 | Frankfurt | [Launch in eu-central-1](https://console.aws.amazon.com/cloudformation/home?region=eu-central-1#/stacks/new?stackName=DataMigrationWorkshop-onPremResources&templateURL=https://aws-datasync-samples.s3-us-west-2.amazonaws.com/workshops/nfs-migration/data-migration-workshop-on-prem.yaml) | 33 | 34 | 2. Click **Next** on the Create Stack page. 35 | 3. If you want to access EC2 instances created in this workshop using your local SSH client then enter the name of an EC2 key pair in your selected region. Otherwise, you can leave the key pair blank. 36 | 3. Click **Next**. 37 | 4. Click **Next**. 38 | 5. Click **Next** again. (skipping the Options and Advanced options sections) 39 | 6. On the Review page, scroll to the bottom and check the box to acknowledge that CloudFormation will create IAM resources, then click **Create stack**. 40 | 41 | **Note:** Instances that are launched as part of this CloudFormation template may be in the initializing state for few minutes. 42 | 43 | While the CloudFormation deployment progresses in the on-premises region, you can proceed to deploy resources for the in-cloud region. 44 | 45 | #### 2. Deploy AWS resources for the in-cloud region 46 | 47 | 1. Click one of the launch links in the table below to deploy the **in-cloud** resources using CloudFormation. Use a different region from the on-premises region. To avoid errors in deployment, select a region in which you have previously created AWS resources. 48 | 49 | | **Region Code** | **Region Name** | **Launch** | 50 | | --- | --- | --- | 51 | | us-west-1 | US West (N. California) | [Launch in us-west-1](https://console.aws.amazon.com/cloudformation/home?region=us-west-1#/stacks/new?stackName=DataMigrationWorkshop-inCloudResources&templateURL=https://aws-datasync-samples.s3-us-west-2.amazonaws.com/workshops/nfs-migration/data-migration-workshop-in-cloud.yaml) | 52 | | us-west-2 | US West (Oregon) | [Launch in us-west-2](https://console.aws.amazon.com/cloudformation/home?region=us-west-2#/stacks/new?stackName=DataMigrationWorkshop-inCloudResources&templateURL=https://aws-datasync-samples.s3-us-west-2.amazonaws.com/workshops/nfs-migration/data-migration-workshop-in-cloud.yaml) | 53 | | us-east-1 | US East (N. Virginia) | [Launch in us-east-1](https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?stackName=DataMigrationWorkshop-inCloudResources&templateURL=https://aws-datasync-samples.s3-us-west-2.amazonaws.com/workshops/nfs-migration/data-migration-workshop-in-cloud.yaml) | 54 | | us-east-2 | US East (Ohio) | [Launch in us-east-2](https://console.aws.amazon.com/cloudformation/home?region=us-east-2#/stacks/new?stackName=DataMigrationWorkshop-inCloudResources&templateURL=https://aws-datasync-samples.s3-us-west-2.amazonaws.com/workshops/nfs-migration/data-migration-workshop-in-cloud.yaml) | 55 | | eu-west-1 | Ireland | [Launch in eu-west-1](https://console.aws.amazon.com/cloudformation/home?region=eu-west-1#/stacks/new?stackName=DataMigrationWorkshop-inCloudResources&templateURL=https://aws-datasync-samples.s3-us-west-2.amazonaws.com/workshops/nfs-migration/data-migration-workshop-in-cloud.yaml) | 56 | | eu-central-1 | Frankfurt | [Launch in eu-central-1](https://console.aws.amazon.com/cloudformation/home?region=eu-central-1#/stacks/new?stackName=DataMigrationWorkshop-inCloudResources&templateURL=https://aws-datasync-samples.s3-us-west-2.amazonaws.com/workshops/nfs-migration/data-migration-workshop-in-cloud.yaml) | 57 | 58 | 2. Click **Next** on the Create stack page. 59 | 3. Click **Next** (there are no stack parameters) **.** 60 | 4. Click **Next** again. (skipping the Options and Advanced options sections) 61 | 5. On the Review page, scroll to the bottom and check the box to acknowledge that CloudFormation will create IAM resources, then click **Create stack**. 62 | 63 | Wait for the CloudFormation stacks in each region to reach the CREATE\_COMPLETE state before proceeding to the next steps. It should take about 10 minutes for both CloudFormation stacks to complete. 64 | 65 | **NOTE:** If a stack fails to deploy because an EC2 instance type is not available in a particular availability zone, delete the stack and retry in the same region or in a different region. 66 | 67 | #### 3. Stack Outputs 68 | 69 | Upon completion, each CloudFormation stack will have a list of "Outputs". These are values such as IP addresses and resource names that will be used throughout the workshop. You can either copy these values elsewhere or keep the page open in your browser and refer to them as you go through the workshop. 70 | 71 | On the CloudFormation page in the **on-premises** region, click on the **Outputs** tab, as shown in the image below. You should see four values listed: 72 | 73 | - **appServerPrivateIP** – This is the private IP address of the Application Server. You will use this when creating the File Gateway file share to limit access to the NFS export. 74 | - **dataSyncAgentPublicIP** – This is the public IP address of the EC2 instance running the DataSync agent. You will use this when activating the DataSync agent. 75 | - **fileGatewayPublicIP** – This is the public IP address of the EC2 instance running the File Gateway. You will use this when activating the File Gateway. 76 | - **nfsServerPrivateIP** – This is the private IP address of the NFS server. You will use this both on the Application Server and when creating a location for DataSync. 77 | 78 | ![](../images/mod1output1.png) 79 | 80 | On the CloudFormation page in the **in-cloud** region, click on the **Outputs** tab as shown in the image below. You should see two values listed: 81 | 82 | - **bucketName** – This is the name of the S3 bucket where the data will be copied to. You will use this when creating a file share on the File Gateway. 83 | - **bucketRoleForDataSync** – This is the role that will be used by the DataSync agent to write files to the S3 bucket. You will use this when creating the S3 location for DataSync. 84 | 85 | ![](../images/mod1output2.png) 86 | 87 | #### 4. Connect to the Application server 88 | 89 | 1. From the AWS console in the **on-premises** region, click **Services** and select **EC2.** 90 | 2. Select **Instances** from the menu on the left. 91 | 3. Right-click on the **ApplicationServer** instance and select **Connect** from the menu. 92 | 93 | If you configured an EC2 Key Pair when you deployed the on-premises CloudFormation template, then you can connect using an SSH client from your computer: 94 | 95 | ![](../images/mod1ssh1.png) 96 | 97 | Otherwise you can connect to the instanct directly through your browser using either EC2 Instance Connect or Session Manager: 98 | 99 | ![](../images/mod1connect.png) 100 | 101 | If using EC2 Instance Connect or Session Manager, click **Connect**. A new tab will be opened in your browser with a command line interface (CLI) to the Application server. Keep this tab open - you will use the command line on the Application server throughout this workshop. 102 | 103 | ## Validation Step 104 | 105 | In the CLI for the Application server, run the following commands to mount the NFS export and verify the files on the NFS server. Use the **nfsServerPrivateIP** value from the **on-premises** CloudFormation stack output. 106 | 107 | $ sudo mount -t nfs :/media/data /mnt/data 108 | $ ls /mnt/data/images 109 | 110 | In the /mnt/data/images folder you should see 200 image files, as shown below. These image files represent the NFS server data that will be migrated to the S3 bucket in the in-cloud region. 111 | 112 | ![](../images/mod1cli1.png) 113 | 114 | ## Module Summary 115 | 116 | In this module, you deployed all of the resources necessary to complete this workshop in both the on-premises region and the in-cloud region. You have also mounted the NFS export on the Application server and verified your data set. 117 | 118 | In the next module, you will create a DataSync task to do an initial copy of files from the NFS server to the S3 bucket. 119 | 120 | Go to [Module 2](../module2/). 121 | -------------------------------------------------------------------------------- /workshops/nfs-migration/EN/module2/README.md: -------------------------------------------------------------------------------- 1 | # **AWS DataSync** 2 | 3 | ### NFS server migration using AWS DataSync and AWS Storage Gateway 4 | 5 | © 2019 Amazon Web Services, Inc. and its affiliates. All rights reserved. 6 | This sample code is made available under the MIT-0 license. See the LICENSE file. 7 | 8 | Errors or corrections? Contact [jeffbart@amazon.com](mailto:jeffbart@amazon.com). 9 | 10 | --- 11 | 12 | # Module 2 13 | ## Initial file copy to S3 using DataSync 14 | 15 | In this module, you will activate the DataSync agent deployed in the on-premises region, create DataSync locations, and then create a DataSync task to copy data from the source location to the destination location. 16 | 17 | DataSync tasks perform the job of copying data and require two "locations" – a source and a destination. With DataSync, a location is an endpoint where files reside or will be copied to. A location can be an NFS export, an SMB share, an Amazon S3 bucket, or an Amazon EFS file system. Location objects are independent of tasks and a single location can be used with multiple tasks. 18 | 19 | ![](../images/mod2arch.png) 20 | 21 | ## Module Steps 22 | 23 | #### 1. Activate the DataSync agent 24 | 25 | Although the agent instance was created in the previous module, before it can be used it needs to be activated in the **in-cloud** region. Follow the steps below to active the agent. 26 | 27 | 1. Go to the AWS Management console page in the **in-cloud** region and click **Services** then select **DataSync.** 28 | 2. If no DataSync agents exist, click the **Get started** button, otherwise click the **Create agent** button. 29 | 3. The EC2 instance that will run the DataSync agent has already been deployed in the **on-premises** region. 30 | 4. Leave the Service endpoint as **"Public service endpoints"**. 31 | 32 | 5. Under the **Activation key** section, enter the **Public IP address** of the DataSync agent instance running in the on-premises region. You can get this IP address from the CloudFormation outputs in the on-premises region. You use the public IP address here because the agent needs to be accessible by your web browser for activation. Enter the IP address of the agent as shown below, then click **Get key.** 33 | 34 | ![](../images/mod2ds1.png) 35 | 36 | 6. Once the activation is successful, you will be shown the activation key and will be prompted for further information. 37 | 38 | ![](../images/mod2ds2.png) 39 | 40 | 7. Enter an Agent name if desired, then click **Create agent**. 41 | 42 | #### 2. Create NFS location 43 | 44 | 1. On the left-hand side of the DataSync service page, click on **Locations** and then click on **Create location**. 45 | 46 | 2. Create a location for the on-premises NFS server. Select **Network File System (NFS)** from the _Location type_ drop-down. 47 | 3. From the _Agents_ drop-down, select the DataSync agent that was created in the previous step. 48 | 4. Enter the **Private IP address** of the NFS server, per the CloudFormation outputs in the **on-premises** region. This was the same IP address used to mount the NFS export on the Application server, in the previous module. The is the IP address that the DataSync agent will use to mount the NFS export. 49 | 5. Under _Mount path_, enter "/media/data". 50 | 51 | ![](../images/mod2ds3.png) 52 | 53 | 6. Click **Create location**. 54 | 55 | #### 3. Create S3 location 56 | 57 | 1. On the left-hand side of the DataSync service page, click on **Locations** and then click on **Create location**. 58 | 59 | 2. Create a location for the S3 bucket. Select **Amazon S3 bucket** from the _Location type_ drop-down. 60 | 3. From the _S3 bucket_ drop-down, select the S3 bucket that starts with **data-migration-workshop** and is followed by a long GUID. 61 | 4. Keep the S3 storage class as **Standard**. 62 | 4. Under _Folder_, enter "/". This will copy all files to the top-level of the bucket. 63 | 5. Under _IAM role_, select the S3 bucket IAM role that starts with **DataMigrationWorkshop-inCloud**. The full name of the role can be found in the outputs for the in-cloud CloudFormation stack. 64 | 65 | ![](../images/mod2ds4.png) 66 | 67 | 6. Click **Create location**. 68 | 69 | On the left-side of the page, click **Locations** again. You should now have two locations listed. One for the NFS server and one for the S3 bucket. 70 | 71 | ![](../images/mod2ds5.png) 72 | 73 | #### 4. Create a task 74 | 75 | 1. On the left-hand side of the DataSync service page, click on **Tasks** and then click on **Create task**. 76 | 77 | 2. Under _Source location options_, select **Choose an existing location**. 78 | 3. Under the _Existing locations_ drop-down, select the NFS server location you created previously. 79 | 4. Click **Next.** 80 | 81 | ![](../images/mod2ds6.png) 82 | 83 | 5. Under _Destination location options_, select **Choose an existing location**. 84 | 6. Under the _Existing locations_ drop-down, select the S3 bucket location you created previously. 85 | 7. Click **Next.** 86 | 8. Under the **Verify data** drop-down, select **Verify only the data transferred**. Keep all other options as-is and then click **Next.** 87 | 9. Click **Create task.** 88 | 89 | #### 5. Run the task 90 | 91 | 1. Wait for the **Task status** to report "Available" (you may need to refresh the page). 92 | 93 | ![](../images/mod2ds7.png) 94 | 95 | 2. To run the task, click the **Start** button, verify the settings, and then click **Start**. 96 | 3. The task will immediately go into the "Running" state. 97 | 4. Under the **History** tab, click on the task execution object in the list. 98 | 99 | ![](../images/mod2ds8.png) 100 | 101 | 5. As the task runs, the execution status will progress from "Launching" to "Preparing" to "Transferring" to "Verifying" and finally to "Success". The task execution will report statistics on the job, as shown below. It will take a few minutes for the task to complete. Once the task has finished, notice that 202 files were transferred. This is the 200 files in the data set along with the two folders in the path that we specified. 102 | 103 | ![](../images/mod2ds9.png) 104 | 105 | ## Validation Step 106 | 107 | From the in-cloud region management console, select **Services** then **S3.** In the bucket list, click on the **data-migration-workshop** bucket. You should see a top-level folder named "images". Inside this folder should be the 200 .jpg files from the NFS server. 108 | 109 | ![](../images/mod2validate.png) 110 | 111 | ## Module Summary 112 | 113 | In this module you successfully activated the DataSync agent and created a task to copy files from the on-premises NFS server into the S3 bucket. You then verified that the files were copied successfully. 114 | 115 | In the next module, you will configure the on-premises File Gateway to connect to the S3 bucket, providing access to the in-cloud files via NFS. 116 | 117 | Go to [Module 3](../module3/). 118 | -------------------------------------------------------------------------------- /workshops/nfs-migration/EN/module3/README.md: -------------------------------------------------------------------------------- 1 | # **AWS DataSync** 2 | 3 | ### NFS server migration using AWS DataSync and AWS Storage Gateway 4 | 5 | © 2019 Amazon Web Services, Inc. and its affiliates. All rights reserved. 6 | This sample code is made available under the MIT-0 license. See the LICENSE file. 7 | 8 | Errors or corrections? Contact [jeffbart@amazon.com](mailto:jeffbart@amazon.com). 9 | 10 | --- 11 | 12 | # Module 3 13 | ## Access S3 bucket on-premises using File Gateway 14 | 15 | You now have the files from the NFS server copied to your S3 bucket. In this module, you will configure the File Gateway in the on-premises region to connect to your S3 bucket and provide access to the files in the bucket through an NFS share. You will mount the File Gateway share on the Application server to validate access to the files. 16 | 17 | ![](../images/fullarch.png) 18 | 19 | ## Module Steps 20 | 21 | #### 1. Activate the File Gateway 22 | 23 | Just as you activated the DataSync agent in the previous module, you need to perform a similar step for the File Gateway, activating it in the **in-cloud** region. Follow the steps below to activate the gateway. 24 | 25 | 1. Go to the AWS Management console page in the **in-cloud** region and click **Services** then select **Storage Gateway.** 26 | 27 | 2. If no gateways exist, click the **Get started** button, otherwise click the **Create gateway** button. 28 | 3. Select the **File gateway** type and click **Next.** 29 | 4. Select **Amazon EC2** as the host platform, then click **Next**. 30 | 5. Select the **Public** endpoint type, then click **Next**. 31 | 6. Enter the **Public IP address** of the File Gateway instance that was created in the first module using CloudFormation. Click **Connect to gateway**. 32 | 7. Name the gateway "DataMigrationGateway" then click **Activate gateway**. 33 | 8. The gateway will be activated and then it will spend a minute or so preparing the local disk devices. Allocate the **300 GiB /dev/sdc** device to **Cache.** This is the local disk on the gateway that will be used to cache frequently accessed files. 34 | 9. Click **Configure logging.** 35 | 10. Leave the setting at _Disable Logging_ then click **Save and continue.** 36 | 11. From the main Storage Gateway page, you will see your gateway listed. 37 | 38 | ![](../images/mod3fgw1.png) 39 | 40 | #### 2. Create an NFS share 41 | 42 | 1. Click on the **Create file share** button 43 | 2. For the **Amazon S3 bucket name** , enter the name of the S3 bucket that DataSync copied the files to. You can find the bucket name in the outputs of the CloudFormation stack in the **in-cloud** region. Leave the **S3 prefix name** blank. 44 | 3. Select **NFS** as the access method and make sure your gateway from the previous step is selected. Leave the **Automated cache refresh** box unchecked. 45 | 4. Click **Next**. 46 | 5. Keep the default settings, then click **Next** 47 | 6. Under the **Allowed clients** section, click **Edit** and change "0.0.0.0/0" to the **Private IP Address** of the Application server, followed by "/32". This will only allow the Application server to access the NFS file share on the gateway. Click the **Close** button. 48 | 7. Under the **Mount options** section, change the **Squash level** to "No root squash". Click the **Close** button. 49 | 8. Click **Create file share**. 50 | 9. Select the check box next to the new file share and note the mount instructions. 51 | 52 | ![](../images/mod3fgw2.png) 53 | 54 | #### 3. Mount the NFS share on the Application server 55 | 56 | 1. Return to the CLI for the Application server and run the following command to create a new mount point for the File Gateway share: 57 | 58 | $ sudo mkdir /mnt/fgw 59 | 60 | 1. Copy the Linux mount command from the Storage Gateway file share page and replace "[MountPath]" with "/mnt/fgw". **You must run the command as sudo.** 61 | 2. You should now have two NFS mount points on your Application server: one for the on-premises NFS server (mounted at /mnt/data) and one for the File Gateway (mounted at /mnt/fgw). 62 | 63 | ![](../images/mod3cli1.png) 64 | 65 | ## Validation Step 66 | 67 | Run the following command to verify that the same set of files exist on both NFS shares. 68 | 69 | $ diff -qr /mnt/data /mnt/fgw 70 | 71 | You should see only one extra file in /mnt/fgw: .aws-datasync-metadata. This file was created by DataSync in the S3 bucket when the task was executed. All other files are the same, indicating that our data was fully transferred by DataSync without errors. 72 | 73 | ## Module Summary 74 | 75 | In this module you successfully activated the File Gateway and created an NFS file share on the gateway. You then mounted the share on the Application server and verified that the files from the on-premises NFS server were copied correctly to the S3 bucket. 76 | 77 | Remember that our ultimate goal in this workshop is to shut off the on-premises NFS server and free up storage resources. In a production environment, this would typically involve a "cutover point", where there is momentary downtime as the Application server changes over to the new storage, which in this workshop is the File Gateway NFS share. However, there are usually new files being created while a migration occurs, or shortly after, requiring another incremental file copy before cutover. 78 | 79 | In the next module, you'll do one more incremental copy before the final cutover to the File Gateway share. 80 | 81 | Go to [Module 4](../module4/). 82 | -------------------------------------------------------------------------------- /workshops/nfs-migration/EN/module4/README.md: -------------------------------------------------------------------------------- 1 | # **AWS DataSync** 2 | 3 | ### NFS server migration using AWS DataSync and AWS Storage Gateway 4 | 5 | © 2019 Amazon Web Services, Inc. and its affiliates. All rights reserved. 6 | This sample code is made available under the MIT-0 license. See the LICENSE file. 7 | 8 | Errors or corrections? Contact [jeffbart@amazon.com](mailto:jeffbart@amazon.com). 9 | 10 | --- 11 | 12 | # Module 4 13 | ## One last incremental copy before cutover 14 | 15 | In this module, you will perform an incremental data transfer using DataSync. This will get any new files that may have been created after the initial data copy. Once you have verified all files from the on-premises NFS server have been copied, you can proceed to cutover. 16 | 17 | ![](../images/fullarch.png) 18 | 19 | ## Module Steps 20 | 21 | #### 1. Create a new file on the NFS server 22 | 23 | 1. From the CLI for the Application server, run the following command to create a new file on the NFS server: 24 | 25 | $ sudo cp /mnt/data/images/00001.jpg /mnt/data/new-image.jpg 26 | 27 | #### 2. Copy the new file to the S3 bucket 28 | 29 | You have already created a DataSync task to copy files from the NFS server to the S3 bucket. To copy the new file, you will just re-run the task. DataSync will only copy files that have changed between the source and the destination. 30 | 31 | 1. Return to the in-cloud region AWS management console and go to the **DataSync** service. 32 | 2. Select the task created previously and click the **Start** button. 33 | 3. Use the default settings and then click **Start**. 34 | 4. Go to the History tab and select the newest task execution from the list. 35 | 36 | ![](../images/mod4ds1.png) 37 | 38 | It will take a few minutes for the task to complete. When the task completes, take a look at the stats. Although you ran the exact same task as last time, only 2 files were copied (the new file and the changes on the folder that contains the new file). 39 | 40 | ![](../images/mod4ds2.png) 41 | 42 | If you take a look at the S3 bucket, you see that the new file is there, just as expected: 43 | 44 | ![](../images/mod4s31.png) 45 | 46 | ## Validation Step 47 | 48 | With the new file in the S3 bucket, you should be able to see it through the File Gateway share on the Application server, right? Let's take a look: 49 | 50 | ![](../images/mod4cli1.png) 51 | 52 | Hmm. You copied the file from the NFS server to S3 using DataSync. And the File Gateway is connected to the S3 bucket. So what's going on? Why can't you see the file on the File Gateway share on the Application server? 53 | 54 | In this case, the file was written to the S3 bucket via DataSync, **not** through the File Gateway share itself. File Gateway is not aware that there are new objects in the bucket. In order to see the new file on the Application server, you need to refresh the metadata cache on the File Gateway. 55 | 56 | Go to the in-cloud region management console and go to the **Storage Gateway** service. On the left side of the page, click on **File shares** and select the NFS share from the list. Click on the **Actions** button and select **Refresh cache** then click **Start**. 57 | 58 | In this case, you have a bucket with only a few hundred objects, so the refresh will be quick. Note that on large buckets with many objects, a cache refresh can take quite a bit of time. To reduce the scope of the refresh, you can use the API or CLI and limit the refresh to a specific directory. You can also use CloudWatch events to monitor when a cache refresh completes. 59 | 60 | Head back to the CLI for the Application server and repeat the "ls /mnt/fgw" command. You should now see the new file. 61 | 62 | ![](../images/mod4cli2.png) 63 | 64 | ## Module Summary 65 | 66 | In this module you added a new file to the NFS server prior to cutover. You then ran the DataSync task a second time to pick up any file changes and copy them to S3. Finally, you used the Refresh Cache method to update the metadata on the File Gateway to see the new files in S3. 67 | 68 | With all of the data copied from the NFS server to S3, you are now ready to perform the cutover. 69 | 70 | Go to [Module 5](../module5/). 71 | -------------------------------------------------------------------------------- /workshops/nfs-migration/EN/module5/README.md: -------------------------------------------------------------------------------- 1 | # **AWS DataSync** 2 | 3 | ### NFS server migration using AWS DataSync and AWS Storage Gateway 4 | 5 | © 2019 Amazon Web Services, Inc. and its affiliates. All rights reserved. 6 | This sample code is made available under the MIT-0 license. See the LICENSE file. 7 | 8 | Errors or corrections? Contact [jeffbart@amazon.com](mailto:jeffbart@amazon.com). 9 | 10 | --- 11 | 12 | # Module 5 13 | ## Cutover to File Gateway and shutdown the NFS server 14 | 15 | With all of the data in the S3 bucket, you are now ready to shut down your NFS server and move exclusively to using the File Gateway. In this module, you will unmount the NFS server and clean up your DataSync resources. You will then write some test files through File Gateway, verifying they end up in the S3 bucket. 16 | 17 | ![](../images/mod5arch.png) 18 | 19 | ## Module Steps 20 | 21 | #### 1. Unmount the NFS server 22 | 23 | 1. From the CLI for the Application server, run the following command to unmount the NFS server: 24 | 25 | $ sudo umount /mnt/data 26 | 27 | #### 2. Clean up DataSync resources 28 | 29 | You're done with DataSync so you can go ahead and clean up resources. 30 | 31 | 1. Go to the **in-cloud** region AWS management console and go to the **DataSync** service. 32 | 33 | 2. Select **Tasks** and delete the task you created previously 34 | 3. Select **Locations** and delete the locations you created previously 35 | 4. Select **Agents** and delete the agent you activated previously. Note that this **will not** delete the actual DataSync agent EC2 instance. That will get deleted later, when the CloudFormation stack is deleted. 36 | 37 | ## Validation Step 38 | 39 | From the CLI for the Application server, run the following command to create another new file in the S3 bucket through the File Gateway: 40 | 41 | sudo cp /mnt/fgw/images/00002.jpg /mnt/fgw/new-image2.jpg 42 | 43 | Go back to the in-cloud region management console and go to **S3**. Select the **data-migration-workshop** bucket. You should see the new-image2.jpg file in the bucket. 44 | 45 | ![](../images/mod5s31.png) 46 | 47 | Your Application server has completed cutover! You can now read all of the files that used to be on the NFS server using the File Gateway share. And any new files written to the share will automatically be uploaded to the S3 bucket. You can now shutdown and decommission your NFS server! 48 | 49 | One of the benefits of using File Gateway is that it stores files as complete, wholly accessible objects in S3. With your data is in S3, you can now use services such as Amazon Athena, Amazon SageMaker, Amazon EMR, and many other AWS services to gain even greater value and insight from your data. 50 | 51 | ## Workshop Cleanup 52 | 53 | To make sure all resources are deleted after this workshop scenario make sure you execute the steps in the order outlined below (you do not need to wait for CloudFormation to finish deleting before moving to the next step): 54 | 55 | 1. Unmount the File Gateway NFS share on the Application server by running the following command: 56 | 57 | sudo umount /mnt/fgw 58 | 59 | 2. Close the browser window running the CLI. 60 | 3. Go to the Storage Gateway page in the in-cloud region and delete the File Gateway **NFS file share** in the in-cloud region 61 | 4. Delete the File Gateway in the in-cloud region named **DataMigrationGateway**. Note this will not delete the gateway EC2 instance. The instance will get deleted when the CloudFormation in the on-premises region is deleted. 62 | 5. Delete all objects in the **data-migration-workshop** S3 bucket in the in-cloud region. The bucket must be empty before it can be deleted by CloudFormation in the next step. 63 | 6. Go to the CloudFormation page in the in-cloud region and delete the stack named "DataMigrationWorkshop-inCloudResources" 64 | 7. Go to the CloudFormation page in the on-premises region and delete the stack named "DataMigrationWorkshop-onPremResources" 65 | 66 | To make sure that all CloudFormation templates have been deleted correctly, confirm that all EC2 instances created in this workshop in the on-premises region are in the **terminated** state. 67 | -------------------------------------------------------------------------------- /workshops/nfs-migration/EN/templates/data-migration-workshop-in-cloud.yaml: -------------------------------------------------------------------------------- 1 | AWSTemplateFormatVersion: '2010-09-09' 2 | Description: AWS DataSync Workshop - NFS Migration - In-Cloud Region 3 | Metadata: 4 | License: 5 | Description: | 6 | Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. 7 | 8 | Permission is hereby granted, free of charge, to any person obtaining a copy of this 9 | software and associated documentation files (the "Software"), to deal in the Software 10 | without restriction, including without limitation the rights to use, copy, modify, 11 | merge, publish, distribute, sublicense, and/or sell copies of the Software, and to 12 | permit persons to whom the Software is furnished to do so. 13 | 14 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, 15 | INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A 16 | PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT 17 | HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION 18 | OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE 19 | SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 20 | 21 | Resources: 22 | 23 | # We use the GUID from the ARN of the stack ID to generate 24 | # a unique bucket name 25 | s3Bucket: 26 | Type: AWS::S3::Bucket 27 | Properties: 28 | PublicAccessBlockConfiguration: 29 | BlockPublicAcls: True 30 | BlockPublicPolicy: True 31 | IgnorePublicAcls: True 32 | RestrictPublicBuckets: True 33 | BucketName: !Join 34 | - "-" 35 | - - "data-migration-workshop" 36 | - !Select 37 | - 2 38 | - !Split 39 | - "/" 40 | - !Ref "AWS::StackId" 41 | 42 | # Give the role a friendly name as the workshop user will need to 43 | # reference it when creating DataSync tasks. 44 | s3BucketIamRole: 45 | Type: AWS::IAM::Role 46 | Properties: 47 | AssumeRolePolicyDocument: 48 | Statement: 49 | - Action: 50 | - sts:AssumeRole 51 | Effect: Allow 52 | Principal: 53 | Service: 54 | - datasync.amazonaws.com 55 | Version: '2012-10-17' 56 | s3BucketRolePolicy: 57 | Type: AWS::IAM::Policy 58 | DependsOn: s3Bucket 59 | Properties: 60 | PolicyDocument: 61 | Statement: 62 | - Effect: Allow 63 | Resource: !GetAtt s3Bucket.Arn 64 | Action: 65 | - s3:GetBucketLocation 66 | - s3:ListBucket 67 | - s3:ListBucketMultipartUploads 68 | - s3:HeadBucket 69 | - Effect: Allow 70 | Resource: !Join [ "/", [ !GetAtt s3Bucket.Arn, "*" ] ] 71 | Action: 72 | - s3:AbortMultipartUpload 73 | - s3:DeleteObject 74 | - s3:GetObject 75 | - s3:ListMultipartUploadParts 76 | - s3:PutObject 77 | Version: '2012-10-17' 78 | PolicyName: policy 79 | Roles: 80 | - !Ref 's3BucketIamRole' 81 | 82 | Outputs: 83 | bucketName: 84 | Description: S3 Bucket Name 85 | Value: !Ref 's3Bucket' 86 | bucketRoleForDataSync: 87 | Description: S3 Bucket Role for DataSync 88 | Value: !Ref 's3BucketIamRole' 89 | -------------------------------------------------------------------------------- /workshops/nfs-migration/JP/README.md: -------------------------------------------------------------------------------- 1 | # **AWS DataSync** 2 | 3 | ### AWS DataSyncとAWS Storage Gatewayを使ったNFSサーバーマイグレーション 4 | 5 | © 2019 Amazon Web Services, Inc. and its affiliates. All rights reserved. 6 | This sample code is made available under the MIT-0 license. See the LICENSE file. 7 | 8 | Errors or corrections? Contact [jeffbart@amazon.com](mailto:jeffbart@amazon.com). 9 | 10 | --- 11 | 12 | ## このワークショップのシナリオ 13 | 14 | あなたのデータセンターには更新時期が迫ったNFSサーバーが存在します。ほとんどのデータは数年前に作成されたもので、時折、読み出しが必要な時だけアクセスされます。新規のファイル書き込みも有りますが、それほど頻繁では有りません。データセンターのフットプリントを削減し、リソースを解放するため、あなたはNFSサーバーのデータをクラウドへ移行させようとしています。しかし、アプリケーションサーバーはNFSのデータを使用しており、まだ移行する事は出来ません。それらはユーザーへの遅延の影響を最小化するために、オンプレミスに置いておく必要が有ります。 15 | 16 | 色々と調べたところ、あなたはオンプレミスのNFSサーバーからAmazon S3へのデータマイグレーションに[AWS DataSync](https://aws.amazon.com/datasync/)が使える事を知り、更に[AWS Storage Gateway](https://aws.amazon.com/storagegateway)により、S3へ移行したデータへのオンプレミスからのNFSアクセスを提供できる事を知りました。 17 | 18 | このワークショップでは、クラウドフォーメーションテンプレートを使用してリソースをデプロイし、デプロイしたリソースをAWSマネージメントコンソールから操作しながら、このシナリオの流れを学習します。以下の構成図のように、NFSサーバー、アプリケーションサーバー、DataSyncエージェント、File Gatewayアプライアンスが、オンプレミス環境を擬似したリージョンにデプロイされます。1つのS3バケットがNFSサーバーのデータのマイグレート先としてAWSリージョンに作成されます。 19 | 20 | ![](images/fullarch.png) 21 | 22 | ## このワークショップでカバーされる内容 23 | 24 | - クラウドフォーメーションを使ったリソースのデプロイ 25 | - Linux NFSサーバーの設定 26 | - DataSyncのCloudWatchログの設定 27 | - DataSyncを使った数百万ファイルのマイグレーションの流れ 28 | - 複数のDataSyncタスクによる並列処理 29 | 30 | ## 前提条件 31 | 32 | #### AWSアカウント 33 | 34 | このワークショップを完了させるには、ワークショップ内で登場するリージョンで、EC2インスタンス、AWS DataSync、AWS Storage Gateway、CloudFormationスタックを作成する権限を持ったAWS IAMロールを持ったAWSアカウントが必要です。 35 | 36 | #### ソフトウェア 37 | 38 | - **インターネットブラウザ** – このワークショップには最新バージョンのChrome又はFirefoxを推奨します 39 | 40 | ## コスト 41 | 42 | このワークショップを完了させるにはおよそ**3.00 USD**のコストがかかります。ワークショップ完了後は全てのデプロイしたリソースを削除し、無駄なコストを生じさせないように、モジュール最後に有るクリーンアップのガイダンスに従う事を推奨します。 43 | 44 | ## 関連するワークショップ 45 | 46 | - [Migrate millions of files using AWS DataSync](https://github.com/aws-samples/aws-datasync-migration-workshop/blob/master/workshops/nfs-million-files) 47 | - [Migrate to FSx Windows File Server using AWS DataSync](https://github.com/aws-samples/aws-datasync-fsx-windows-migration) 48 | - [Get hands-on with online data migration options to simplify & accelerate your journey to AWS](https://github.com/aws-samples/aws-online-data-migration-workshop) 49 | 50 | ## ワークショップモジュール 51 | 52 | このワークショップは以下の5つのモジュールで構成されます 53 | 54 | - [モジュール 1](module1/) - オンプレミスとin-cloudリージョンへのリソースのデプロイ 55 | - [モジュール 2](module2/) - DataSyncを使用したS3への初回ファイル同期 56 | - [モジュール 3](module3/) - File Gatewayを使用したオンプレミスからのS3バケットへのアクセス 57 | - [モジュール 4](module4/) - カットオーバー前の最後の差分同期 58 | - [モジュール 5](module5/) - File Gatewayへの完全移行とNFSサーバーのシャットダウン 59 | 60 | 始めるには [モジュール 1](module1/)へ 61 | -------------------------------------------------------------------------------- /workshops/nfs-migration/JP/images/fullarch.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/JP/images/fullarch.png -------------------------------------------------------------------------------- /workshops/nfs-migration/JP/images/mod1arch.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/JP/images/mod1arch.png -------------------------------------------------------------------------------- /workshops/nfs-migration/JP/images/mod1cli1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/JP/images/mod1cli1.png -------------------------------------------------------------------------------- /workshops/nfs-migration/JP/images/mod1connect.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/JP/images/mod1connect.png -------------------------------------------------------------------------------- /workshops/nfs-migration/JP/images/mod1output1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/JP/images/mod1output1.png -------------------------------------------------------------------------------- /workshops/nfs-migration/JP/images/mod1output2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/JP/images/mod1output2.png -------------------------------------------------------------------------------- /workshops/nfs-migration/JP/images/mod1ssh1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/JP/images/mod1ssh1.png -------------------------------------------------------------------------------- /workshops/nfs-migration/JP/images/mod2arch.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/JP/images/mod2arch.png -------------------------------------------------------------------------------- /workshops/nfs-migration/JP/images/mod2ds1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/JP/images/mod2ds1.png -------------------------------------------------------------------------------- /workshops/nfs-migration/JP/images/mod2ds2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/JP/images/mod2ds2.png -------------------------------------------------------------------------------- /workshops/nfs-migration/JP/images/mod2ds3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/JP/images/mod2ds3.png -------------------------------------------------------------------------------- /workshops/nfs-migration/JP/images/mod2ds4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/JP/images/mod2ds4.png -------------------------------------------------------------------------------- /workshops/nfs-migration/JP/images/mod2ds5.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/JP/images/mod2ds5.png -------------------------------------------------------------------------------- /workshops/nfs-migration/JP/images/mod2ds6.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/JP/images/mod2ds6.png -------------------------------------------------------------------------------- /workshops/nfs-migration/JP/images/mod2ds7.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/JP/images/mod2ds7.png -------------------------------------------------------------------------------- /workshops/nfs-migration/JP/images/mod2ds8.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/JP/images/mod2ds8.png -------------------------------------------------------------------------------- /workshops/nfs-migration/JP/images/mod2ds9.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/JP/images/mod2ds9.png -------------------------------------------------------------------------------- /workshops/nfs-migration/JP/images/mod2validate.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/JP/images/mod2validate.png -------------------------------------------------------------------------------- /workshops/nfs-migration/JP/images/mod3cli1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/JP/images/mod3cli1.png -------------------------------------------------------------------------------- /workshops/nfs-migration/JP/images/mod3fgw1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/JP/images/mod3fgw1.png -------------------------------------------------------------------------------- /workshops/nfs-migration/JP/images/mod3fgw2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/JP/images/mod3fgw2.png -------------------------------------------------------------------------------- /workshops/nfs-migration/JP/images/mod4cli1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/JP/images/mod4cli1.png -------------------------------------------------------------------------------- /workshops/nfs-migration/JP/images/mod4cli2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/JP/images/mod4cli2.png -------------------------------------------------------------------------------- /workshops/nfs-migration/JP/images/mod4ds1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/JP/images/mod4ds1.png -------------------------------------------------------------------------------- /workshops/nfs-migration/JP/images/mod4ds2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/JP/images/mod4ds2.png -------------------------------------------------------------------------------- /workshops/nfs-migration/JP/images/mod4s31.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/JP/images/mod4s31.png -------------------------------------------------------------------------------- /workshops/nfs-migration/JP/images/mod5arch.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/JP/images/mod5arch.png -------------------------------------------------------------------------------- /workshops/nfs-migration/JP/images/mod5s31.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/JP/images/mod5s31.png -------------------------------------------------------------------------------- /workshops/nfs-migration/JP/images/module1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-migration/JP/images/module1.png -------------------------------------------------------------------------------- /workshops/nfs-migration/JP/module1/README.md: -------------------------------------------------------------------------------- 1 | # **AWS DataSync** 2 | 3 | ### AWS DataSyncとAWS Storage Gatewayを使ったNFSサーバーマイグレーション 4 | 5 | © 2019 Amazon Web Services, Inc. and its affiliates. All rights reserved. 6 | This sample code is made available under the MIT-0 license. See the LICENSE file. 7 | 8 | Errors or corrections? Contact [jeffbart@amazon.com](mailto:jeffbart@amazon.com). 9 | 10 | --- 11 | 12 | # モジュール 1 13 | ## オンプレミスとin-cloudリージョンへのリソースのデプロイ 14 | 15 | このモジュールでは、２つのAWSリージョンにリソースをデプロイするためにクラウドフォーメーションスクリプトを使用します。１つはオンプレミス環境を擬似したリージョン、もう１つはin-cloudリージョンです。全てのリソースのデプロイが完了すると、アプリケーションサーバーからNFSのExportをマウントし、その中のファイルを確認する事が出来ます。 16 | 17 | ![](../images/mod1arch.png) 18 | 19 | ## このモジュールの手順 20 | 21 | #### 1. オンプレミスリージョンへのリソースのデプロイ 22 | 23 | 1. クラウドフォーメーションを使用して**on-premises**リソースをデプロイするために、以下の中からリンクを選択し、クリックして下さい。 デプロイ中のエラーを回避するために、以前にリソースを作成した事の有るリージョンの選択を推奨します。 24 | 25 | | **リージョンコード** | **リージョン名** | **起動リンク** | 26 | | --- | --- | --- | 27 | | us-west-1 | US West (N. California) | [us-west-1へデプロイ](https://console.aws.amazon.com/cloudformation/home?region=us-west-1#/stacks/new?stackName=DataMigrationWorkshop-onPremResources&templateURL=https://aws-datasync-samples.s3-us-west-2.amazonaws.com/workshops/nfs-migration/data-migration-workshop-on-prem.yaml) | 28 | | us-west-2 | US West (Oregon) | [us-west-2へデプロイ](https://console.aws.amazon.com/cloudformation/home?region=us-west-2#/stacks/new?stackName=DataMigrationWorkshop-onPremResources&templateURL=https://aws-datasync-samples.s3-us-west-2.amazonaws.com/workshops/nfs-migration/data-migration-workshop-on-prem.yaml) | 29 | | us-east-1 | US East (N. Virginia) | [us-east-1へデプロイ](https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?stackName=DataMigrationWorkshop-onPremResources&templateURL=https://aws-datasync-samples.s3-us-west-2.amazonaws.com/workshops/nfs-migration/data-migration-workshop-on-prem.yaml) | 30 | | us-east-2 | US East (Ohio) | [us-east-2へデプロイ](https://console.aws.amazon.com/cloudformation/home?region=us-east-2#/stacks/new?stackName=DataMigrationWorkshop-onPremResources&templateURL=https://aws-datasync-samples.s3-us-west-2.amazonaws.com/workshops/nfs-migration/data-migration-workshop-on-prem.yaml) | 31 | | eu-west-1 | Ireland | [eu-west-1へデプロイ](https://console.aws.amazon.com/cloudformation/home?region=eu-west-1#/stacks/new?stackName=DataMigrationWorkshop-onPremResources&templateURL=https://aws-datasync-samples.s3-us-west-2.amazonaws.com/workshops/nfs-migration/data-migration-workshop-on-prem.yaml) | 32 | | eu-central-1 | Frankfurt | [eu-central-1へデプロイ](https://console.aws.amazon.com/cloudformation/home?region=eu-central-1#/stacks/new?stackName=DataMigrationWorkshop-onPremResources&templateURL=https://aws-datasync-samples.s3-us-west-2.amazonaws.com/workshops/nfs-migration/data-migration-workshop-on-prem.yaml) | 33 | 34 | 2. Create Stackページで**Next**をクリック。 35 | 3. **Next**をクリック (変更する必要の有るスタックパラメータは有りません)。 36 | 4. **Next**をクリック。 37 | 5. 再度**Next**をクリック(オプションとアドバンストオプションのセクションはスキップ)。 38 | 6. レビューページで最下部までスクロールし、クラウドフォーメーションがIAMリソースを作成する事を許可するチェックボックスにチェックを入れ、**Create stack**をクリック。 39 | 40 | **注意:** このクラウドフォーメーションテンプレートで起動したインスタンスは数分間initializingステータスになるので、起動までお待ちください。 41 | 42 | オンプレミスリージョンへのクラウドフォーメーションのデプロイが進行中の状態でも、次のin-cloudリージョンのリソースのデプロイに進む事が出来ます。 43 | 44 | #### 2. in-cloudリージョンへのリソースのデプロイ 45 | 46 | 1. クラウドフォーメーションを使用して**in-cloud**リソースをデプロイするために、以下の中からリンクを選択し、クリックして下さい。 先程作成したオンプレミスリージョンとは異なるリージョンを選択して下さい。 デプロイ中のエラーを回避するために、以前にリソースを作成した事の有るリージョンの選択を推奨します。 47 | 48 | | **リージョンコード** | **リージョン名** | **起動リンク** | 49 | | --- | --- | --- | 50 | | us-west-1 | US West (N. California) | [us-west-1へデプロイ](https://console.aws.amazon.com/cloudformation/home?region=us-west-1#/stacks/new?stackName=DataMigrationWorkshop-inCloudResources&templateURL=https://aws-datasync-samples.s3-us-west-2.amazonaws.com/workshops/nfs-migration/data-migration-workshop-in-cloud.yaml) | 51 | | us-west-2 | US West (Oregon) | [us-west-2へデプロイ](https://console.aws.amazon.com/cloudformation/home?region=us-west-2#/stacks/new?stackName=DataMigrationWorkshop-inCloudResources&templateURL=https://aws-datasync-samples.s3-us-west-2.amazonaws.com/workshops/nfs-migration/data-migration-workshop-in-cloud.yaml) | 52 | | us-east-1 | US East (N. Virginia) | [us-east-1へデプロイ](https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?stackName=DataMigrationWorkshop-inCloudResources&templateURL=https://aws-datasync-samples.s3-us-west-2.amazonaws.com/workshops/nfs-migration/data-migration-workshop-in-cloud.yaml) | 53 | | us-east-2 | US East (Ohio) | [us-east-2へデプロイ](https://console.aws.amazon.com/cloudformation/home?region=us-east-2#/stacks/new?stackName=DataMigrationWorkshop-inCloudResources&templateURL=https://aws-datasync-samples.s3-us-west-2.amazonaws.com/workshops/nfs-migration/data-migration-workshop-in-cloud.yaml) | 54 | | eu-west-1 | Ireland | [eu-west-1へデプロイ](https://console.aws.amazon.com/cloudformation/home?region=eu-west-1#/stacks/new?stackName=DataMigrationWorkshop-inCloudResources&templateURL=https://aws-datasync-samples.s3-us-west-2.amazonaws.com/workshops/nfs-migration/data-migration-workshop-in-cloud.yaml) | 55 | | eu-central-1 | Frankfurt | [eu-central-1へデプロイ](https://console.aws.amazon.com/cloudformation/home?region=eu-central-1#/stacks/new?stackName=DataMigrationWorkshop-inCloudResources&templateURL=https://aws-datasync-samples.s3-us-west-2.amazonaws.com/workshops/nfs-migration/data-migration-workshop-in-cloud.yaml) | 56 | 57 | 2. Create stackページで**Next**をクリック。 58 | 3. **Next**をクリック (スタックパラメータは有りません)。 59 | 4. 再度**Next**をクリック(オプションとアドバンストオプションのセクションはスキップ)。 60 | 5. レビューページで最下部までスクロールし、クラウドフォーメーションがIAMリソースを作成する事を許可するチェックボックスにチェックを入れ、**Create stack**をクリック。 61 | 62 | 次のステップに進む前にそれぞれのリージョンのクラウドフォーメーションスタックのステータスがCREATE\_COMPLETEになるまでお待ち下さい。 完了まで10分程度かかります。 63 | 64 | **注意:** もし特定のアベイラビリティゾーンでEC2インスタンスタイプが対応しておらず、スタックのデプロイに失敗する場合は、もう一度 65 | 試す、または別のリージョンでデプロイして下さい。 66 | 67 | #### 3. スタックアウトプット 68 | 69 | 完了後、それぞれのクラウドフォーメーションスタックは実行結果の"アウトプット（出力）"のリストを表示します。IPアドレスやリソース名等のこれらの情報はワークショップを通して使用します。これらの情報をコピーしておくか、このページをブラウザで表示したままワークショップを進めると、スムーズに進行出来ます。 70 | 71 | **オンプレミス**リージョンのクラウドフォーメーションページで**アウトプット（出力）**をクリックすると以下のような4つの項目を確認出来ます。 72 | 73 | - **appServerPrivateIP** – アプリケーションサーバーのプライベートIPアドレスです。File Gatewayにおいてファイル共有を作成する際、NFS Exportへのアクセスを制限するために使用します。 74 | - **dataSyncAgentPublicIP** – DataSyncエージェントが動作するEC2インスタンスのパブリックIPアドレスです。DataSyncエージェントをアクティベートする時に使用します。 75 | - **fileGatewayPublicIP** – File Gatewayが動作するEC2インスタンスのパブリックIPアドレスです。File Gatewayをアクティベートする時に使用します。 76 | - **nfsServerPrivateIP** – NFSサーバーのプライベートIPアドレスです。アプリケーションサーバーからのマウント及びDataSyncのロケーション作成時に使用します。 77 | 78 | ![](../images/mod1output1.png) 79 | 80 | **in-cloud**リージョンのクラウドフォーメーションページで**アウトプット（出力）**をクリックすると以下のような2つの項目を確認出来ます。 81 | 82 | - **bucketName** – データコピー先のS3バケット名です。File Gatewayでファイル共有を作成する時に使用します。 83 | - **bucketRoleForDataSync** – DataSyncエージェントがS3バケットへファイルを書き込む際のIAMロールです。DataSyncでS3ロケーションを作成する時に使用します。 84 | 85 | ![](../images/mod1output2.png) 86 | 87 | #### 4. アプリケーションサーバーに接続します 88 | 89 | 1. AWSコンソールから**オンプレミス** リージョンに移動し、**Services**の中から**EC2**をクリックします。 90 | 2. 左側のメニューから**インスタンス**をクリックします。 91 | 3. **ApplicationServer**インスタンスを右クリックし、メニューから**Connect**を選択します。 92 | オンプレミスクラウドフォーメーションテンプレートのデプロイ時にEC2キーペアを設定した場合は、御使用のコンピューターからSSHクライアントを使って接続する事も出来ます: 93 | 94 | ![](../images/mod1ssh1.png) 95 | 96 | また、EC2インスタンスコネクト又はセッションマネージャーを使って、ブラウザからインスタンスへ直接接続する事も可能です: 97 | 98 | ![](../images/mod1connect.png) 99 | 100 | EC2インスタンスコネクト又はセッションマネージャーを使用する場合は、**Connect**をクリックして下さい。新しいブラウザウィンドウが開き、アプリケーションサーバーのCLIが表示されます。この画面は開いたままにしておいて下さい。後ほどこのワークショップで使用します。 101 | 102 | ## 最後に確認 103 | 104 | アプリケーションサーバーのCLIで、NFS exportをマウントしてNFSサーバー上のファイルを確認するために、以下のコマンドを実行して下さい。 **オンプレミス**クラウドフォーメーションスタックのアウトプット（出力）に含まれる**nfsServerPrivateIP**の値を使用してて下さい。 105 | 106 | $ sudo mount -t nfs :/media/data /mnt/data 107 | $ ls /mnt/data/images 108 | 109 | 以下のように/mnt/data/imagesフォルダーの配下に200個のイメージファイルが確認できるはずです。これらのイメージデータはin-cloudリージョンのS3にマイグレーションするNFSサーバーのデータとして使用します。 110 | 111 | ![](../images/mod1cli1.png) 112 | 113 | ## このモジュールのまとめ 114 | 115 | このモジュールでは、オンプレミスリージョンとin-cloudリージョンに、このワークショップで必要となる全てのリソースをデプロイしました。また、アプリケーションサーバー上でNFS exportをマウントし、データセットを確認しました。 116 | 117 | 次のモジュールではNFSサーバーからS3への初回データ同期を行うためのDataSyncタスクを作成します。 118 | 119 | [モジュール 2](../module2/)へ 120 | -------------------------------------------------------------------------------- /workshops/nfs-migration/JP/module2/README.md: -------------------------------------------------------------------------------- 1 | # **AWS DataSync** 2 | 3 | ### AWS DataSyncとAWS Storage Gatewayを使ったNFSサーバーマイグレーション 4 | 5 | © 2019 Amazon Web Services, Inc. and its affiliates. All rights reserved. 6 | This sample code is made available under the MIT-0 license. See the LICENSE file. 7 | 8 | Errors or corrections? Contact [jeffbart@amazon.com](mailto:jeffbart@amazon.com). 9 | 10 | --- 11 | 12 | # モジュール 2 13 | ## DataSyncを使用したS3への初回ファイル同期 14 | 15 | このモジュールでは、オンプレミスリージョンにデプロイしたDataSyncエージェントをアクティベートし、DataSyncロケーションを作成、その後ソースロケーションから宛先ロケーションへデータをコピーするDataSyncタスクを作成します。 16 | 17 | DataSyncタスクはデータコピージョブを実行するため、ソースと宛先の2つの "ロケーション"が必要です。DataSyncにおいてロケーションとは、ファイルが置かれている又はコピーの宛先となるエンドポイントです。ロケーションにはNFS export、SMB共有、Amazon S3バケット又はAmazon EFSファイルシステムが含まれます。ロケーションオブジェクトはタスク毎に固有で、1つのロケーションを複数のタスクで使用する事が可能です。 18 | 19 | ![](../images/mod2arch.png) 20 | 21 | ## このモジュールの手順 22 | 23 | #### 1. DataSyncエージェントのアクティベート 24 | 25 | 前のモジュールでエージェント用のインスタンスを作成しましたが、実際に使用出来る状態にするには**in-cloud**リージョンでのアクティベートが必要です。エージェントをアクティベートするには以下の手順に従って下さい。 26 | 27 | 1. AWSコンソールから**in-cloud** リージョンに移動し、**Services**の中から**DataSync**をクリックします。 28 | 2. もしDataSyncエージェントが存在しない場合、**Get started** ボタンをクリックして下さい。その他の場合、**Create agent** ボタンをクリックして下さい。 29 | 3. DataSyncエージェントとして動作するEC2インスタンスは既に **オンプレミス** リージョンにデプロイされています。 30 | 4. サービスエンドポイントの値は、そのまま**"Public service endpoints"**にして下さい。 31 | 5. **Activation key** セクションの中に、オンプレミスリージョンのDataSyncエージェントインスタンスの **Public IP address** を入力して下さい。この値はオンプレミスリージョンのクラウドフォーメーションのアウトプット（出力）から入手出来ます。 アクティベーションのためにブラウザからアクセス出来る必要が有るため、ここではパブリックIPアドレスを使用します。以下のようにエージェントのIPアドレスを入力して、**Get key**をクリックして下さい。 32 | 33 | ![](../images/mod2ds1.png) 34 | 35 | 6. アクティベーションが成功すると、アクティベーションキーが表示され、その他諸処の情報が表示されます。 36 | 37 | ![](../images/mod2ds2.png) 38 | 39 | 7. 必要に応じてエージェント名を入力して**Create agent**をクリックして下さい。 40 | 41 | #### 2. NFSロケーションの作成 42 | 43 | 1. DataSyncサービスのページの左側で、**Locations**をクリックし、**Create location**をクリックして下さい。 44 | 2. オンプレミスNFSサーバーのロケーションを作成します。Location typeのドロップダウンメニューから**Network File System (NFS)**を選択して下さい。 45 | 3. Agentsドロップダウンメニューから前の手順で作成したDataSyncエージェントを選択して下さい。 46 | 4. オンプレミスリージョンのクラウドフォーメーションのアウトプット（出力）を参照して、NFSサーバーの**Private IP address**を入力して下さい。これは前のモジュールでアプリケーションサーバーからNFS exportをマウントした時と同じIPアドレスで、DataSyncエージェントがNFS exportをマウントする時に使用します。 47 | 5. Mount pathの所には"/media/data"を入力して下さい。 48 | 49 | ![](../images/mod2ds3.png) 50 | 51 | 6. **Create location**をクリックして下さい。 52 | 53 | #### 3. S3ロケーションの作成 54 | 55 | 1. DataSyncサービスのページの左側で、**Locations**をクリックし、**Create location**をクリックして下さい。 56 | 2. S3バケットのロケーションを作成します。Location typeドロップダウンメニューから**Amazon S3 bucket**を選択して下さい。 57 | 3. S3 bucketドロップダウンメニューから**data-migration-workshop**で始まるバケット名を選択して下さい。 58 | 4. S3ストレージクラスは**Standard**のままにして下さい。 59 | 4. Folderの所には"/"を入力して下さい。これにより全てのファイルがバケットのトップレベルにコピーされます。 60 | 5. IAM roleの所には**DataMigrationWorkshop-inCloud**で始まるS3バケットIAMロールを選択して下さい。このフルネームはin-cloudリージョンのクラウドフォーメーションのアウトプット（出力）から入手出来ます。 61 | 62 | ![](../images/mod2ds4.png) 63 | 64 | 6. **Create location**をクリックして下さい。 65 | 66 | ページの左側で再び**Locations**をクリックして下さい。2つのロケーションが表示されます。1つはNFSサーバー、もう1つはS3バケットです。 67 | ![](../images/mod2ds5.png) 68 | 69 | #### 4. タスクの作成 70 | 71 | 1. DataSyncサービスのページの左側で、**Tasks**をクリックし、**Create task**をクリックして下さい。 72 | 2. Source location optionsの所で**Choose an existing location**を選択して下さい。 73 | 3. Existing locationsドロップダウンメニューで、前の手順で作成したNFSサーバーのロケーションを選択して下さい。 74 | 4. **Next**をクリックして下さい。 75 | 76 | ![](../images/mod2ds6.png) 77 | 78 | 5. Destination location optionsの所で**Choose an existing location**を選択して下さい。 79 | 6. Existing locationsドロップダウンメニューで、 前の手順で作成したS3バケットのロケーションを選択して下さい。d 80 | 7. **Next**をクリックして下さい。 81 | 8. **Verify data**ドロップダウンメニューで、**Verify only the data transferred**を選択して下さい。その他のオプションはデフォルトのままで**Next**をクリックして下さい。 82 | 9. **Create task**をクリックして下さい。 83 | 84 | #### 5. タスクの起動 85 | 86 | 1. **Task status**が"Available"になるまでお待ち下さい。(ページのリフレッシュが必要な場合が有ります) 87 | 88 | ![](../images/mod2ds7.png) 89 | 90 | 2. タスクの起動のため、**Start**ボタンをクリックし、設定確認後、**Start**をクリックして下さい。 91 | 3. タスクは直ぐに"Running"ステータスへと変わります。 92 | 4. **History**タブの所で、実行中のタスクをクリックして下さい。 93 | 94 | ![](../images/mod2ds8.png) 95 | 96 | 5. タスクが進行すると、ステータスが"Launching"から"Preparing"、"Transferring"、"Verifying"へと変化し、最後に"Success"へと変わります。以下のようにタスクの実行状態が表示されます。タスクが完了するまで数分間かかります。タスクが完了すると、202個のファイルが転送された事が示されます。これは200個のファイルと2つのフォルダの合計です。 97 | 98 | ![](../images/mod2ds9.png) 99 | 100 | ## 最後に確認 101 | 102 | in-cloudリージョンのマネジメントコンソールで、**Services**、**S3**の順に選択します。バケットのリストの中から**data-migration-workshop**バケットをクリックします。"images"という名前のトップレベルフォルダが見えます。 このフォルダの中にはNFSサーバーからコピーされた200個のjpgファイルが存在します。 103 | 104 | ![](../images/mod2validate.png) 105 | 106 | ## このモジュールのまとめ 107 | 108 | このモジュールでは、DataSyncエージェントをアクティベートし、オンプレミスのNFSサーバーからS3バケットへファイルをコピーするタスクを作成しました。その後、ファイルがコピーされている事を確認しました。 109 | 110 | 次のモジュールでは、オンプレミスのFile GatewayをS3につなげるように設定し、NFS経由のクラウドアクセスを提供します。 111 | 112 | [モジュール 3](../module3/)へ 113 | -------------------------------------------------------------------------------- /workshops/nfs-migration/JP/module3/README.md: -------------------------------------------------------------------------------- 1 | # **AWS DataSync** 2 | 3 | ### AWS DataSyncとAWS Storage Gatewayを使ったNFSサーバーマイグレーション 4 | 5 | © 2019 Amazon Web Services, Inc. and its affiliates. All rights reserved. 6 | This sample code is made available under the MIT-0 license. See the LICENSE file. 7 | 8 | Errors or corrections? Contact [jeffbart@amazon.com](mailto:jeffbart@amazon.com). 9 | 10 | --- 11 | 12 | # モジュール 3 13 | ## File Gatewayを使用したオンプレミスからのS3バケットへのアクセス 14 | 15 | ここまでの手順でDataSyncを使ってNFSサーバーからS3へファイルをコピーしました。このモジュールでは、オンプレミスリージョンのFile GatewayをS3バケットへ接続し、NFS共有を経由してS3バケットのファイルへのアクセスを提供する設定を行います。ファイルへのアクセスを確認するために、アプリケーションサーバーからFile GatewayのNFS共有をマウントします。 16 | 17 | ![](../images/fullarch.png) 18 | 19 | ## このモジュールの手順 20 | 21 | #### 1. File Gatewayのアクティベート 22 | 23 | 前のモジュールでDataSyncエージェントをアクティベートしたのと同様、File Gatewayを**in-cloud**リージョンでアクティベートする手順を行います。以下の手順に従って下さい。 24 | 25 | 1. AWSコンソールから**in-cloud** リージョンに移動し、**Services**の中から**Storage Gateway**をクリックします。 26 | 2. もしゲートウェイが存在しない場合、**Get started** ボタンをクリックして下さい。その他の場合、**Create gateway** ボタンをクリックして下さい。 27 | 3. **File gateway**タイプを選択し、**Next**をクリックして下さい。 28 | 4. ホストプラットフォームとして**Amazon EC2**を選択し、**Next**をクリックして下さい。 29 | 5. **Public**エンドポイントタイプを選択し、**Next**をクリックして下さい。 30 | 6. モジュール１のクラウドフォーメーションで作成したFile Gatewayインスタンスの**Public IP address**を入力し、**Connect to gateway**をクリックして下さい。 31 | 7. ゲートウェイ名として"DataMigrationGateway"を入力し、**Activate gateway**をクリックして下さい。 32 | 8. ゲートウェイがアクティベートされ、その後ローカルディスクデバイスの準備のため、数分時間がかかります。**300 GiB /dev/sdc**デバイスを**Cache**に割り当てて下さい。これは頻繁にアクセスされるファイルをキャッシュするためのゲートウェイのローカルディスク領域になります。 33 | 9. **Configure logging**をクリックして下さい。 34 | 10. Disable Loggingの設定はそのままにして**Save and continue**をクリックして下さい。 35 | 11. Storage Gatewayのメインページに、作成したゲートウェイが表示されます。 36 | 37 | ![](../images/mod3fgw1.png) 38 | 39 | #### 2. NFS共有の作成 40 | 41 | 1. **Create file share**ボタンをクリックして下さい。 42 | 2. **Amazon S3 bucket name**の所に、DataSyncのデータコピー先のバケット名を入力して下さい。これはin-cloudリージョンのクラウドフォーメーションのアウトプット（出力）からも取得出来ます。**S3 prefix name**のところは空欄のままにして下さい。 43 | 3. アクセス方式として**NFS**を選択し、前の手順で作成したゲートウェイが選択されている事を確認して下さい。**Automated cache refresh**のチェックボックスは外したままにして下さい。 44 | 4. **Next**をクリックして下さい。 45 | 5. デフォルト設定のまま**Next**をクリックして下さい。 46 | 6. **Allowed clients**セクションで、**Edit**をクリックし、"0.0.0.0/0"からアプリケーションサーバーの**Private IP Address**に変更し、末尾に"/32"を加えて下さい。これにより、アプリケーションサーバーだけがゲートウェイのNFS共有にアクセス出来るようになります。**Close**ボタンををクリックして下さい。 47 | 7. **Mount options**セクションで、**Squash level**を"No root squash"に変更して下さい。**Close**ボタンををクリックして下さい。 48 | 8. **Create file share**をクリックして下さい。 49 | 9. 作成したファイル共有のチェックボックスにチェックを入れ、マウント手順を確認して下さい。 50 | 51 | ![](../images/mod3fgw2.png) 52 | 53 | #### 3. アプリケーションサーバーからNFS共有をマウント 54 | 55 | 1. アプリケーションサーバーのCLIに戻り、File Gateway共有へのマウントポイントを作成するために、以下のコマンドを実行して下さい。 56 | 57 | $ sudo mkdir /mnt/fgw 58 | 59 | 1. Storage Gatewayのファイル共有ページのLinuxコマンドをコピーし、"[MountPath]"部分を"/mnt/fgw"に入れ替えて実行して下さい。 **必ずsudoで実行する必要が有ります。** 60 | 2. これでアプリケーションサーバーに2つのNFSマウントポイントが作成されました。一つはオンプレミスのNFSサーバー（/mnt/dataにマウント）、もう1つはFile Gateway（/mnt/fgwにマウント）です。 61 | 62 | ![](../images/mod3cli1.png) 63 | 64 | ## 最後に確認 65 | 66 | 2つのNFS共有に同じファイルが存在する事を確認するため、以下のコマンドを実行して下さい。 67 | 68 | $ diff -qr /mnt/data /mnt/fgw 69 | 70 | /mnt/fgwの中に1つだけ追加のファイルが見えます（.aws-datasync-metadata）。このファイルはDataSyncがタスクの完了の際にS3バケットの中に作り出したものです。他の全てのファイルは一致しており、これはDataSyncによってエラーなく転送された事を示します。 71 | 72 | ## このモジュールのまとめ 73 | 74 | このモジュールでは、File Gatewayをアクティベートし、NFSファイル共有をゲートウェイ上に作成しました。その後、アプリケーションサーバーからファイル共有をマウントし、オンプレミスのNFSサーバーと、DataSyncでS3に転送したデータを比較検証しました。 75 | 76 | このワークショップの完全なゴールはオンプレミスのNFSサーバーをシャットダウンしてストレージリソースを解放する事である事を思い出して下さい。本番環境では通常、アプリケーションサーバーの接続先が完全にFile Gatewayに切り替わる"カットオーバーポイント"が存在します。しかし実際はマイグレーション後に新たな書き込みが発生し、カットオーバー前にデータの差分が生じます。 77 | 78 | 次のモジュールでは、File Gatewayへの最後のカットオーバーの前に、もう一度差分データ同期を行います。 79 | 80 | [モジュール 4](../module4/)へ 81 | -------------------------------------------------------------------------------- /workshops/nfs-migration/JP/module4/README.md: -------------------------------------------------------------------------------- 1 | # **AWS DataSync** 2 | 3 | ### AWS DataSyncとAWS Storage Gatewayを使ったNFSサーバーマイグレーション 4 | 5 | © 2019 Amazon Web Services, Inc. and its affiliates. All rights reserved. 6 | This sample code is made available under the MIT-0 license. See the LICENSE file. 7 | 8 | Errors or corrections? Contact [jeffbart@amazon.com](mailto:jeffbart@amazon.com). 9 | 10 | --- 11 | 12 | # モジュール 4 13 | ## カットオーバー前の最後の差分同期 14 | 15 | このモジュールでは、DataSyncを使って差分データのコピーを行います。これにより、初期同期後に作成されたファイルを再同期する事が出来ます。全てのファイルがオンプレミスのNFSサーバーからコピーされた事が確認出来れば、カットオーバーに進む事が出来ます。 16 | 17 | ![](../images/fullarch.png) 18 | 19 | ## このモジュールの手順 20 | 21 | #### 1. NFSサーバーに新しいファイルを作成 22 | 23 | 1. 新しいファイルをNFSサーバー上に作成するため、アプリケーションサーバーのCLIから以下のコマンドを実行します。 24 | 25 | $ sudo cp /mnt/data/images/00001.jpg /mnt/data/new-image.jpg 26 | 27 | #### 2. 新しいファイルをS3バケットへコピー 28 | 29 | NFSサーバーからS3バケットへファイルをコピーするDataSyncタスクは既に作成済みです。新しいファイルをコピーするのに必要な事は、単にもう一度タスクを起動するだけです。DataSyncはソースと宛先の間で、変化の有ったファイルだけをコピーします。 30 | 31 | 1. in-cloudリージョンのAWSマネジメントコンソールに戻り、**DataSync**サービスをクリック。 32 | 2. 作成済みのタスクを選択し、**Start**ボタンをクリック。 33 | 3. デフォルト設定を使用し、**Start**をクリック。 34 | 4. Historyタブ上で、リストの中から最新のタスクの実行を選択 35 | 36 | ![](../images/mod4ds1.png) 37 | 38 | タスクの完了まで数分かかります。タスクが完了したら転送済ファイルの数を見て下さい。同じタスクを実行したにも関わらず、2つのファイルしか転送されていません。(新しいファイルそのものと、それを格納しているフォルダーです） 39 | 40 | ![](../images/mod4ds2.png) 41 | 42 | S3バケットの中身を見ると、期待通りファイルが転送されている事が確認出来ます。 43 | 44 | ![](../images/mod4s31.png) 45 | 46 | ## 最後に確認 47 | 48 | S3に新しいファイルがコピーされたので、アプリケーションサーバから見ればFile Gateway共有にもファイルが見えるはず？実際に見てみましょう。 49 | 50 | ![](../images/mod4cli1.png) 51 | 52 | んー。。。 確かにDataSyncを使ってNFSサーバーからS3へコピーしたはずです。そしてFile GatewayはS3バケットに繋がっているはずです。何が起きているんでしょう？何故アプリケーションサーバからFile Gateway共有上に新しいファイルが見えないんでしょう。 53 | 54 | このケースでは新しいファイルはFile Gateway共有経由**ではなく**DataSync経由でS3へ書き込まれました。File Gatewayはバケットに新しいオブジェクトが有る事を知りません。新しいファイルをアプリケーションサーバーに見せるために、File Gatewayのメタデータキャッシュをリフレッシュする必要が有ります。 55 | 56 | in-cloudリージョンのAWSマネジメントコンソールに戻り、**Storage Gateway**サービスをクリック。ページの左側で**File shares**をクリックし、リストからNFS共有を選択。**Actions**ボタンをクリックし、**Refresh cache**を選択。次に**Start**をクリック。 57 | 58 | このケースではバケットの中には数百のオブジェクトしか無いため、リフレッシュは直ぐに完了します。多くのオブジェクトを抱えた大きなバケットでは、キャッシュのリフレッシュに時間がかかる可能性が有るので注意して下さい。リフレッシュのスコープを削減してこの問題を改善するために、API又はCLIを使用して、特定のディレクトリだけのリフレッシュをするように制限をかける事が出来ます。CloudWatchイベントで、キャッシュのリフレッシュの完了をモニタする事も出来ます。 59 | 60 | アプリケーションサーバーのCLIに戻り、再度"ls /mnt/fgw"コマンドを実行して下さい。新しいファイルを見る事が出来ます。 61 | 62 | ![](../images/mod4cli2.png) 63 | 64 | ## このモジュールのまとめ 65 | 66 | このモジュールでは、カットオーバー前にNFSサーバーへ新しいファイルを追加しました。次にDataSyncタスクを再度実行し、NFSサーバーとS3間の差分データ転送を行いました。最後にFile Gatewayのメタデータをアップデートするため、キャッシュのリフレッシュを行いました。 67 | 68 | NFSサーバーからS3への全てのデータの転送が完了したため、いよいよカットオーバーに進む事が出来ます。 69 | 70 | [モジュール 5](../module5/)へ 71 | -------------------------------------------------------------------------------- /workshops/nfs-migration/JP/module5/README.md: -------------------------------------------------------------------------------- 1 | # **AWS DataSync** 2 | 3 | ### AWS DataSyncとAWS Storage Gatewayを使ったNFSサーバーマイグレーション 4 | 5 | © 2019 Amazon Web Services, Inc. and its affiliates. All rights reserved. 6 | This sample code is made available under the MIT-0 license. See the LICENSE file. 7 | 8 | Errors or corrections? Contact [jeffbart@amazon.com](mailto:jeffbart@amazon.com). 9 | 10 | --- 11 | 12 | # モジュール 5 13 | ## File Gatewayへの完全移行とNFSサーバーのシャットダウン 14 | 15 | S3バケットへの全てのデータの転送が完了した事で、オンプレミスのNFSサーバーをシャットダウンし、File Gatewayへと移行する準備が出来ました。このモジュールでは、NFSサーバーをアンマウントし、DataSyncのリソースをクリーンアップします。最後にいくつかのテストファイルをFile Gatewayに書き込み、S3バケットに格納されている事を確認し、カットオーバー完了です。 16 | 17 | ![](../images/mod5arch.png) 18 | 19 | ## このモジュールの手順 20 | 21 | #### 1. NFSサーバーをアンマウント 22 | 23 | 1. アプリケーションサーバーのCLIからNFSサーバーをアンマウントするために以下のコマンドを実行します。 24 | 25 | $ sudo umount /mnt/data 26 | 27 | #### 2. DataSyncリソースのクリーンアップ 28 | 29 | DataSyncの活用は終わったので、リソースをクリーンアップします。 30 | 31 | 1. **in-cloud**リージョンのAWSマネジメントコンソールで**DataSync**サービスをクリックして下さい。 32 | 2. **Tasks**を選択し、作成済みのタスクを削除して下さい。 33 | 3. **Locations**を選択し、 作成済みのロケーションを削除して下さい。 34 | 4. **Agents**を選択し、作成済みのエージェントを削除して下さい。これはDataSyncエージェントが動いていたEC2インスタンス自体を削除する訳では**ない**ので注意して下さい。この後のクラウドフォーメーションスタックの削除で削除されます。 35 | 36 | ## 最後に確認 37 | 38 | File Gatewayを通してS3に追加のファイルを作成するため、アプリケーションサーバーのCLIから以下のコマンドを実行して下さい。 39 | 40 | sudo cp /mnt/fgw/images/00002.jpg /mnt/fgw/new-image2.jpg 41 | 42 | in-cloudリージョンのマネジメントコンソールに戻り、**S3**をクリックして下さい。そして**data-migration-workshop**バケットを選択して下さい。バケット内にnew-image2.jpgファイルが見えます。 43 | 44 | ![](../images/mod5s31.png) 45 | 46 | これでアプリケーションサーバーのカットオーバーが完了しました！NFSサーバーに格納していたファイルを全てFile Gateway共有経由で見る事が出来ます。また、共有に書き込まれた全ての新しいファイルは、自動的にS3バケットへアップロードされます。これでNFSサーバーをシャットダウンして撤去する事が出来ます。 47 | 48 | File Gatewayを使用するメリットの1つは、完全でアクセス可能なオブジェクトとしてS3に格納出来る事です。S3にデータが有れば、Amazon Athena、Amazon SageMaker、Amazon EMR、その他多くのAWSサービスを活用し、よりデータの価値を活用する事が出来ます。 49 | 50 | ## ワークショップクリーンアップ 51 | 52 | このワークショップで使用した全てのリソースを削除するために、以下の手順にしたがって実行して下さい。 (次のステップに進むためにクラウドフォーメーションの完了を待つ必要はありません） 53 | 54 | 1. アプリケーションサーバーのCLIで以下のコマンドを実行し、File Gateway NFS共有をアンマウントして下さい。 55 | 56 | sudo umount /mnt/fgw 57 | 58 | 2. アプリケーションサーバーCLIのウィンドウを閉じて下さい。 59 | 3. in-cloudリージョンのStorage Gatewayのページで**NFS file share**を削除して下さい。 60 | 4. in-cloudリージョンの**DataMigrationGateway**という名前のFile Gatewayを削除して下さい。これによりゲートウェイのEC2インスタンスは削除されない点に注意してください。インスタンスの削除はオンプレミスリージョンのクラウドフォーメーションの削除で行われます。 61 | 5. in-cloudリージョンの**data-migration-workshop**S3バケット内の全てのオブジェクトを削除してください。次のクラウドフォーメーションによる削除を実行する前に、バケットをからの状態にして置く必要があります。 62 | 6. in-cloudリージョンのクラウドフォーメーションのページで、 "DataMigrationWorkshop-inCloudResources"という名前のスタックを削除して下さい。 63 | 7. on-premisesリージョンのクラウドフォーメーションのページで、"DataMigrationWorkshop-onPremResources"という名前のスタックを削除して下さい。 64 | 65 | 全てのクラウドフォーメーションテンプレートが正しく削除された事を確認するために、オンプレミスリージョンの全てのEC2インスタンスが**terminated**ステータスになっている事を確認して下さい。 66 | -------------------------------------------------------------------------------- /workshops/nfs-migration/JP/templates/data-migration-workshop-in-cloud.yaml: -------------------------------------------------------------------------------- 1 | AWSTemplateFormatVersion: '2010-09-09' 2 | Description: AWS DataSync Workshop - NFS Migration - In-Cloud Region 3 | Metadata: 4 | License: 5 | Description: | 6 | Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. 7 | 8 | Permission is hereby granted, free of charge, to any person obtaining a copy of this 9 | software and associated documentation files (the "Software"), to deal in the Software 10 | without restriction, including without limitation the rights to use, copy, modify, 11 | merge, publish, distribute, sublicense, and/or sell copies of the Software, and to 12 | permit persons to whom the Software is furnished to do so. 13 | 14 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, 15 | INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A 16 | PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT 17 | HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION 18 | OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE 19 | SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 20 | 21 | Resources: 22 | 23 | # We use the GUID from the ARN of the stack ID to generate 24 | # a unique bucket name 25 | s3Bucket: 26 | Type: AWS::S3::Bucket 27 | Properties: 28 | PublicAccessBlockConfiguration: 29 | BlockPublicAcls: True 30 | BlockPublicPolicy: True 31 | IgnorePublicAcls: True 32 | RestrictPublicBuckets: True 33 | BucketName: !Join 34 | - "-" 35 | - - "data-migration-workshop" 36 | - !Select 37 | - 2 38 | - !Split 39 | - "/" 40 | - !Ref "AWS::StackId" 41 | 42 | # Give the role a friendly name as the workshop user will need to 43 | # reference it when creating DataSync tasks. 44 | s3BucketIamRole: 45 | Type: AWS::IAM::Role 46 | Properties: 47 | AssumeRolePolicyDocument: 48 | Statement: 49 | - Action: 50 | - sts:AssumeRole 51 | Effect: Allow 52 | Principal: 53 | Service: 54 | - datasync.amazonaws.com 55 | Version: '2012-10-17' 56 | s3BucketRolePolicy: 57 | Type: AWS::IAM::Policy 58 | DependsOn: s3Bucket 59 | Properties: 60 | PolicyDocument: 61 | Statement: 62 | - Effect: Allow 63 | Resource: !GetAtt s3Bucket.Arn 64 | Action: 65 | - s3:GetBucketLocation 66 | - s3:ListBucket 67 | - s3:ListBucketMultipartUploads 68 | - s3:HeadBucket 69 | - Effect: Allow 70 | Resource: !Join [ "/", [ !GetAtt s3Bucket.Arn, "*" ] ] 71 | Action: 72 | - s3:AbortMultipartUpload 73 | - s3:DeleteObject 74 | - s3:GetObject 75 | - s3:ListMultipartUploadParts 76 | - s3:PutObject 77 | Version: '2012-10-17' 78 | PolicyName: policy 79 | Roles: 80 | - !Ref 's3BucketIamRole' 81 | 82 | Outputs: 83 | bucketName: 84 | Description: S3 Bucket Name 85 | Value: !Ref 's3Bucket' 86 | bucketRoleForDataSync: 87 | Description: S3 Bucket Role for DataSync 88 | Value: !Ref 's3BucketIamRole' 89 | -------------------------------------------------------------------------------- /workshops/nfs-migration/README.md: -------------------------------------------------------------------------------- 1 | # **AWS DataSync** 2 | 3 | ### NFS server migration using AWS DataSync and AWS Storage Gateway 4 | 5 | © 2019 Amazon Web Services, Inc. and its affiliates. All rights reserved. 6 | This sample code is made available under the MIT-0 license. See the LICENSE file. 7 | 8 | Errors or corrections? Contact [jeffbart@amazon.com](mailto:jeffbart@amazon.com). 9 | 10 | --- 11 | 12 | ## Supported Languages 13 | 14 | This workshop is available in the following languages. Click a link below to begin the workshop. 15 | 16 | - [English](EN/) 17 | - [Japanese](JP/) 18 | -------------------------------------------------------------------------------- /workshops/nfs-million-files/README.md: -------------------------------------------------------------------------------- 1 | # **AWS DataSync** 2 | 3 | ### Migrate millions of files using AWS DataSync 4 | 5 | © 2020 Amazon Web Services, Inc. and its affiliates. All rights reserved. 6 | This sample code is made available under the MIT-0 license. See the LICENSE file. 7 | 8 | Errors or corrections? Contact [jeffbart@amazon.com](mailto:jeffbart@amazon.com). 9 | 10 | --- 11 | 12 | **This workshop is now being maintained in AWS [workshop studio](https://workshops.aws/)** 13 | 14 | - [Migrate millions of files using AWS DataSync](https://catalog.us-east-1.prod.workshops.aws/datasync-migrate-millions-of-files/en-US) 15 | 16 | --- 17 | 18 | ## Workshop scenario 19 | 20 | This is an advanced-level workshop on using [AWS DataSync](https://aws.amazon.com/datasync/) 21 | to transfer millions of files from an on-premises NFS server to Amazon S3. The workshop 22 | will walk you through practical steps and advanced concepts to consider when transferring large 23 | datasets using AWS DataSync. 24 | 25 | In this workshop, you will use CloudFormation templates to deploy resources in two 26 | AWS regions, representing an on-premises environment and an in-cloud region. As shown 27 | in the architecture diagram below, an NFS server and two DataSync agents will be deployed 28 | in the on-premises environment. An S3 bucket will be created in an AWS region, simulating 29 | the AWS cloud region to which the NFS server's data will be transferred. 30 | 31 | Once the resources have been deployed, you will then use AWS DataSync to transfer 32 | two million files from the NFS server to the S3 bucket. You will use Amazon CloudWatch 33 | and other tools and services to monitor performance and transfer results. 34 | 35 | ![](images/fullarch.png) 36 | 37 | ## Topics covered 38 | 39 | - Deploying resources using CloudFormation 40 | - Copying data to Amazon S3 using DataSync 41 | - Performing incremental data transfers with DataSync 42 | - Accessing data stored in an S3 bucket using File Gateway 43 | - Using the Storage Gateway RefreshCache API 44 | 45 | ## Prerequisites 46 | 47 | #### AWS Account 48 | 49 | Because this is an advanced-level workshop, it is expected that you are familiar with using the AWS CLI tools and AWS Management console. 50 | 51 | In order to complete this workshop, you will need an AWS account with rights to create AWS IAM roles, EC2 instances, AWS DataSync objects, and CloudFormation stacks in the AWS regions you select. 52 | 53 | #### Software 54 | 55 | - **Internet Browser** – It is recommended that you use the latest version of Chrome or Firefox 56 | - **AWS CLI** - You should have the latest version of the AWS CLI deployed in your working environment 57 | 58 | ## Cost 59 | 60 | It will cost approximately **15.00 USD** to run this workshop. It is recommended that you follow the cleanup instructions once you have completed the workshop to remove all deployed resources and limit ongoing costs to your AWS account. 61 | 62 | ## Related workshops 63 | 64 | - [NFS server migration using AWS DataSync and Storage Gateway](https://github.com/aws-samples/aws-datasync-migration-workshop/blob/master/workshops/nfs-migration) 65 | - [Migrate to FSx Windows File Server using AWS DataSync](https://github.com/aws-samples/aws-datasync-fsx-windows-migration) 66 | - [Get hands-on with online data migration options to simplify & accelerate your journey to AWS](https://github.com/aws-samples/aws-online-data-migration-workshop) 67 | 68 | ## Workshop Modules 69 | 70 | This workshop consists of the following modules: 71 | 72 | - [Module 1](/workshops/nfs-million-files/module1) - Deploy resources in the on-premises and in-cloud regions 73 | - [Module 2](/workshops/nfs-million-files/module2) - Configure the NFS server 74 | - [Module 3](/workshops/nfs-million-files/module3) - Setup CloudWatch and activate DataSync agents 75 | - [Module 4](/workshops/nfs-million-files/module4) - Run a test transfer to validate network and performance 76 | - [Module 5](/workshops/nfs-million-files/module5) - Run the full transfer 77 | - [Module 6](/workshops/nfs-million-files/module6) - Perform an incremental transfer 78 | - [Module 7](/workshops/nfs-million-files/module7) - Workshop clean-up 79 | 80 | To get started, go to [Module 1](/workshops/nfs-million-files/module1). 81 | -------------------------------------------------------------------------------- /workshops/nfs-million-files/images/fullarch.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-million-files/images/fullarch.png -------------------------------------------------------------------------------- /workshops/nfs-million-files/images/mod1output1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-million-files/images/mod1output1.png -------------------------------------------------------------------------------- /workshops/nfs-million-files/images/mod1output2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-million-files/images/mod1output2.png -------------------------------------------------------------------------------- /workshops/nfs-million-files/images/mod2agentsList.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-million-files/images/mod2agentsList.png -------------------------------------------------------------------------------- /workshops/nfs-million-files/images/mod2ds1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-million-files/images/mod2ds1.png -------------------------------------------------------------------------------- /workshops/nfs-million-files/images/mod2ds2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-million-files/images/mod2ds2.png -------------------------------------------------------------------------------- /workshops/nfs-million-files/images/mod4ds1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-million-files/images/mod4ds1.png -------------------------------------------------------------------------------- /workshops/nfs-million-files/images/mod4ds2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-million-files/images/mod4ds2.png -------------------------------------------------------------------------------- /workshops/nfs-million-files/images/mod4ds3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-million-files/images/mod4ds3.png -------------------------------------------------------------------------------- /workshops/nfs-million-files/images/mod4ds4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-million-files/images/mod4ds4.png -------------------------------------------------------------------------------- /workshops/nfs-million-files/images/mod4ds5.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-million-files/images/mod4ds5.png -------------------------------------------------------------------------------- /workshops/nfs-million-files/images/mod5ds1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-million-files/images/mod5ds1.png -------------------------------------------------------------------------------- /workshops/nfs-million-files/images/mod6ds1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-million-files/images/mod6ds1.png -------------------------------------------------------------------------------- /workshops/nfs-million-files/images/mod6ds2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-million-files/images/mod6ds2.png -------------------------------------------------------------------------------- /workshops/nfs-million-files/images/mod6ds3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/nfs-million-files/images/mod6ds3.png -------------------------------------------------------------------------------- /workshops/nfs-million-files/module1/README.md: -------------------------------------------------------------------------------- 1 | # **AWS DataSync** 2 | 3 | ### Migrate millions of files using AWS DataSync 4 | 5 | © 2020 Amazon Web Services, Inc. and its affiliates. All rights reserved. 6 | This sample code is made available under the MIT-0 license. See the LICENSE file. 7 | 8 | Errors or corrections? Contact [jeffbart@amazon.com](mailto:jeffbart@amazon.com). 9 | 10 | --- 11 | 12 | # Module 1 13 | ## Deploy resources in the on-premises and in-cloud regions 14 | 15 | In this module, you will use CloudFormation scripts to deploy resources in two AWS regions: one that represents your on-premises environment, and one for the in-cloud environment. 16 | 17 | ## Module Steps 18 | 19 | #### 1. Deploy AWS resources for the ON-PREMISES region 20 | 21 | 1. Click one of the launch links in the table below to deploy the **ON-PREMISES** resources using CloudFormation. To avoid errors during deployment, select a region in which you have previously created AWS resources. 22 | 23 | | **Region Code** | **Region Name** | **Launch** | 24 | | --- | --- | --- | 25 | | us-west-1 | US West (N. California) | [Launch in us-west-1](https://console.aws.amazon.com/cloudformation/home?region=us-west-1#/stacks/new?stackName=MillionFiles-OnPrem&templateURL=https://aws-datasync-samples.s3-us-west-2.amazonaws.com/workshops/nfs-million-files/nfs-million-files-on-prem.yaml) | 26 | | us-west-2 | US West (Oregon) | [Launch in us-west-2](https://console.aws.amazon.com/cloudformation/home?region=us-west-2#/stacks/new?stackName=MillionFiles-OnPrem&templateURL=https://aws-datasync-samples.s3-us-west-2.amazonaws.com/workshops/nfs-million-files/nfs-million-files-on-prem.yaml) | 27 | | us-east-1 | US East (N. Virginia) | [Launch in us-east-1](https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?stackName=MillionFiles-OnPrem&templateURL=https://aws-datasync-samples.s3-us-west-2.amazonaws.com/workshops/nfs-million-files/nfs-million-files-on-prem.yaml) | 28 | | us-east-2 | US East (Ohio) | [Launch in us-east-2](https://console.aws.amazon.com/cloudformation/home?region=us-east-2#/stacks/new?stackName=MillionFiles-OnPrem&templateURL=https://aws-datasync-samples.s3-us-west-2.amazonaws.com/workshops/nfs-million-files/nfs-million-files-on-prem.yaml) | 29 | | eu-west-1 | Ireland | [Launch in eu-west-1](https://console.aws.amazon.com/cloudformation/home?region=eu-west-1#/stacks/new?stackName=MillionFiles-OnPrem&templateURL=https://aws-datasync-samples.s3-us-west-2.amazonaws.com/workshops/nfs-million-files/nfs-million-files-on-prem.yaml) | 30 | | eu-central-1 | Frankfurt | [Launch in eu-central-1](https://console.aws.amazon.com/cloudformation/home?region=eu-central-1#/stacks/new?stackName=MillionFiles-OnPrem&templateURL=https://aws-datasync-samples.s3-us-west-2.amazonaws.com/workshops/nfs-million-files/nfs-million-files-on-prem.yaml) | 31 | 32 | 2. Click **Next** on the Create Stack page. 33 | 3. Under the **Parameters** section, select an EC2 key pair to allow for SSH login to instances created by this stack. Do not edit the values for AMI IDs. Click **Next**. 34 | 5. Click **Next** again (skipping the Options and Advanced options sections). 35 | 6. On the Review page, scroll to the bottom and check the box to acknowledge that CloudFormation will create IAM resources, then click **Create stack**. 36 | 37 | While the CloudFormation deployment progresses in the ON-PREMISES region, you can proceed to deploy resources for the IN-CLOUD region. 38 | 39 | #### 2. Deploy AWS resources for the IN-CLOUD region 40 | 41 | 1. Click one of the launch links in the table below to deploy the **IN-CLOUD** resources using CloudFormation. Use a **different region** from the ON-PREMISES region. To avoid errors in deployment, select a region in which you have previously created AWS resources. 42 | 43 | | **Region Code** | **Region Name** | **Launch** | 44 | | --- | --- | --- | 45 | | us-west-1 | US West (N. California) | [Launch in us-west-1](https://console.aws.amazon.com/cloudformation/home?region=us-west-1#/stacks/new?stackName=MillionFiles-InCloud&templateURL=https://aws-datasync-samples.s3-us-west-2.amazonaws.com/workshops/nfs-million-files/nfs-million-files-in-cloud.yaml) | 46 | | us-west-2 | US West (Oregon) | [Launch in us-west-2](https://console.aws.amazon.com/cloudformation/home?region=us-west-2#/stacks/new?stackName=MillionFiles-InCloud&templateURL=https://aws-datasync-samples.s3-us-west-2.amazonaws.com/workshops/nfs-million-files/nfs-million-files-in-cloud.yaml) | 47 | | us-east-1 | US East (N. Virginia) | [Launch in us-east-1](https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?stackName=MillionFiles-InCloud&templateURL=https://aws-datasync-samples.s3-us-west-2.amazonaws.com/workshops/nfs-million-files/nfs-million-files-in-cloud.yaml) | 48 | | us-east-2 | US East (Ohio) | [Launch in us-east-2](https://console.aws.amazon.com/cloudformation/home?region=us-east-2#/stacks/new?stackName=MillionFiles-InCloud&templateURL=https://aws-datasync-samples.s3-us-west-2.amazonaws.com/workshops/nfs-million-files/nfs-million-files-in-cloud.yaml) | 49 | | eu-west-1 | Ireland | [Launch in eu-west-1](https://console.aws.amazon.com/cloudformation/home?region=eu-west-1#/stacks/new?stackName=MillionFiles-InCloud&templateURL=https://aws-datasync-samples.s3-us-west-2.amazonaws.com/workshops/nfs-million-files/nfs-million-files-in-cloud.yaml) | 50 | | eu-central-1 | Frankfurt | [Launch in eu-central-1](https://console.aws.amazon.com/cloudformation/home?region=eu-central-1#/stacks/new?stackName=MillionFiles-InCloud&templateURL=https://aws-datasync-samples.s3-us-west-2.amazonaws.com/workshops/nfs-million-files/nfs-million-files-in-cloud.yaml) | 51 | 52 | 2. Click **Next** on the Create stack page. 53 | 3. Click **Next** (there are no stack parameters). 54 | 4. Click **Next** again (skipping the Options and Advanced options sections). 55 | 5. On the Review page, scroll to the bottom and check the box to acknowledge that CloudFormation will create IAM resources, then click **Create stack**. 56 | 57 | Wait for the CloudFormation stacks in each region to reach the CREATE\_COMPLETE state before proceeding to the next steps. It should take about 3 to 5 minutes for both CloudFormation stacks to complete. 58 | 59 | **NOTE:** If a stack fails to deploy because an EC2 instance type is not available in a particular availability zone, delete the stack and retry in the same region or in a different region. 60 | 61 | #### 3. Stack Outputs 62 | 63 | Upon completion, each CloudFormation stack will have a list of "Outputs". These are values such as IP addresses and resource names that will be used throughout the workshop. You can either copy these values elsewhere or keep the page open in your browser and refer to them as you go through the workshop. 64 | 65 | On the CloudFormation page in the **ON-PREMISES** region, click on the **Outputs** tab, as shown in the image below. You should see the following values listed: 66 | 67 | - **dataSyncAgent1PrivateIP** – This is the private IP address of the EC2 instance running the first DataSync agent. You will use this when configuring exports on the NFS server. 68 | - **dataSyncAgent1PublicIP** – This is the public IP address of the EC2 instance running the first DataSync agent. You will use this when activating this particular DataSync agent. 69 | - **dataSyncAgent2PrivateIP** – This is the private IP address of the EC2 instance running the second DataSync agent. You will use this when configuring exports on the NFS server. 70 | - **dataSyncAgent2PublicIP** – This is the public IP address of the EC2 instance running the second DataSync agent. You will use this when activating this particular DataSync agent. 71 | - **nfsServerPrivateIP** – This is the private IP address of the NFS server. You will use this when creating a location for DataSync. 72 | 73 | ![](../images/mod1output1.png) 74 | 75 | On the CloudFormation page in the **IN-CLOUD** region, click on the **Outputs** tab as shown in the image below. You should see the following values listed: 76 | 77 | - **bucketName** – This is the name of the S3 bucket where the data will be copied to. 78 | - **bucketRoleForDataSync** – This is the role that will be used by the DataSync agent to write files to the S3 bucket. You will use this when creating the S3 location for DataSync. 79 | 80 | ![](../images/mod1output2.png) 81 | 82 | ## Module Summary 83 | 84 | In this module, you deployed all of the resources necessary to complete this workshop in both the on-premises region and the in-cloud region. At this time, datasets are being initialized on the NFS server. That process will take approximately 10-15 minutes to complete. 85 | 86 | In the next module, you will verify that the datasets have been initialized, then you will configure the NFS server and prepare it to be used by DataSync for transferring files to S3. 87 | 88 | Go to [Module 2](/workshops/nfs-million-files/module2). 89 | -------------------------------------------------------------------------------- /workshops/nfs-million-files/module2/README.md: -------------------------------------------------------------------------------- 1 | # **AWS DataSync** 2 | 3 | ### Migrate millions of files using AWS DataSync 4 | 5 | © 2020 Amazon Web Services, Inc. and its affiliates. All rights reserved. 6 | This sample code is made available under the MIT-0 license. See the LICENSE file. 7 | 8 | Errors or corrections? Contact [jeffbart@amazon.com](mailto:jeffbart@amazon.com). 9 | 10 | --- 11 | 12 | # Module 2 13 | ## Configure the NFS server 14 | 15 | Understanding how much data needs to be copied and how that data is organized is a key part of planning when using AWS DataSync. Three file systems have been created on the NFS server and are being pre-populated with randomly generated datasets. 16 | 17 | In this module, you will browse the NFS server that was created in the previous module, verify that the datasets have been fully initialized, and then configure the NFS server with the proper export settings so it can be used by AWS DataSync. 18 | 19 | ## Module Steps 20 | 21 | #### 1. Login to the NFS server 22 | 23 | 1. Go to the AWS Management console page in the **ON-PREMISES** region and click **Services** then select **EC2**. 24 | 2. Click on the list of instances and then select the NFS Server. Click on the **Connect** button and follow the instructions to connect to Session Manager or to create an SSH connection to the NFS server. 25 | 26 | #### 2. Browse the file systems 27 | 28 | There are three 200 GiB EBS volumes attached to the NFS server. Each volume has been formatted with an XFS file system and pre-populated with data from the previous step. 29 | 30 | 1. On the NFS server, run the following command to list the three file systems: 31 | 32 | [ec2-user@ ~]$ mount | grep /mnt 33 | 34 | /dev/nvme1n1 on /mnt/fs1 type xfs (rw,relatime,attr2,inode64,noquota) 35 | /dev/nvme2n1 on /mnt/fs2 type xfs (rw,relatime,attr2,inode64,noquota) 36 | /dev/nvme3n1 on /mnt/fs3 type xfs (rw,relatime,attr2,inode64,noquota) 37 | 38 | 2. The three file systems are mounted under the **/mnt** directory. Run the following command to get the amount of data in each file system: 39 | 40 | [ec2-user@ ~]$ df -h | grep /mnt 41 | 42 | /dev/nvme1n1 200G 12G 189G 6% /mnt/fs1 43 | /dev/nvme2n1 200G 12G 189G 6% /mnt/fs2 44 | /dev/nvme3n1 200G 22G 179G 11% /mnt/fs3 45 | 46 | From the command output, you can see that fs1 and fs2 have **12 GiB** of data each while fs3 has **22 GiB** of data. 47 | 48 | 3. When using AWS DataSync it's important to know not only how much data will be transferred, but how many files will be transferred as well. Run the following commands to determine the number of files and directories in each file system: 49 | 50 | [ec2-user@ ~]$ cd /mnt 51 | [ec2-user@ ~]$ ls -R fs1/ | wc -l 52 | [ec2-user@ ~]$ ls -R fs2/ | wc -l 53 | [ec2-user@ ~]$ ls -R fs3/ | wc -l 54 | 55 | You should see **505,151** files on fs1 and fs2, and **1,010,301** files on fs3. If you take the amount of data on each file system and divide it by the number of files, you get an average file size of around **24 KiB**. This means you are working primarily with small files and that a significant part of the I/O load on the NFS server will be metadata operations as DataSync processes millions of small files. 56 | 57 | 4. Go ahead and browse the directory structure a bit further. Run the following commands to see how the files are organized: 58 | 59 | [ec2-user@ ~]$ ls fs1 60 | [ec2-user@ ~]$ ls fs1/d0001 61 | [ec2-user@ ~]$ ls -a fs1/d0001/dir0001 62 | 63 | These commands show there are 50 folders in fs1, each containing 20 sub-folders. Each sub-folder contains 500 files along with three extra files: **.htaccess, index.html, manifest.lst.** The other file systems are similar, except that fs3 has 100 top-level folders, rather than 50. For this workshop, you will copy all files and directories to your S3 bucket except for .htaccess and index.html. 64 | 65 | #### 3. Configure the NFS exports 66 | 67 | You want to transfer data from all three file systems using DataSync. To do so, you will need to create three NFS exports (aka file shares), one per file system. On this version of Linux, you do this by modifying the **/etc/exports** file. 68 | 69 | 1. On the NFS server, **add the following three lines** to the /etc/exports file. You will need to modify the file as root, so use the sudo command when starting your editor. 70 | 71 | /mnt/fs1 10.12.14.99(ro,no_root_squash) 10.12.14.16(ro,no_root_squash) 72 | /mnt/fs2 10.12.14.99(ro,no_root_squash) 10.12.14.16(ro,no_root_squash) 73 | /mnt/fs3 10.12.14.99(ro,no_root_squash) 10.12.14.16(ro,no_root_squash) 74 | 75 | Replace the IP addresses above with the corresponding **private** IP address assigned to the EC2 instances for each DataSync agent, as reported in the CloudFormation Outputs for the **ON-PREMISES** region. 76 | 77 | 2. Run the following command to restart the NFS server and apply the export settings: 78 | 79 | [ec2-user@ ~]$ sudo systemctl restart nfs 80 | 81 | ## Validation Step 82 | 83 | Run the following command to verify that the three file systems are being exported only to the DataSync agents: 84 | 85 | [ec2-user@ ~]$ showmount -e 86 | 87 | /mnt/fs3 10.12.14.16,10.12.14.99 88 | /mnt/fs2 10.12.14.16,10.12.14.99 89 | /mnt/fs1 10.12.14.16,10.12.14.99 90 | 91 | ## Module Summary 92 | 93 | In this module you dove deep into the file systems on the NFS server to understand how much data and how many files will be transferred. You also explored the directory layout which you will use in the next module to perform a test transfer. 94 | 95 | To allow the DataSync agents to access the NFS server, you created NFS export entries for the three file systems. For this workshop, the agents have no need to write anything to the file system, so you configured each export to be read-only using the **'ro'** flag in the /etc/exports file. Additionally, the DataSync agents will mount the NFS exports as the root user so you also specified the **'no_root_squash'** flag to make sure the agent has full access to all data on the file system. Finally, you only exported the file systems to the DataSync agents, to lock down access. 96 | 97 | In the next module, you will configure CloudWatch for logging by the DataSync service and you will activate your DataSync agents. 98 | 99 | Go to [Module 3](/workshops/nfs-million-files/module3). 100 | -------------------------------------------------------------------------------- /workshops/nfs-million-files/module3/README.md: -------------------------------------------------------------------------------- 1 | # **AWS DataSync** 2 | 3 | ### Migrate millions of files using AWS DataSync 4 | 5 | © 2020 Amazon Web Services, Inc. and its affiliates. All rights reserved. 6 | This sample code is made available under the MIT-0 license. See the LICENSE file. 7 | 8 | Errors or corrections? Contact [jeffbart@amazon.com](mailto:jeffbart@amazon.com). 9 | 10 | --- 11 | 12 | # Module 3 13 | ## Configure CloudWatch and activate DataSync agents 14 | 15 | With the NFS server configured you can now proceed to setup CloudWatch so it can accept logs from DataSync. After that, you will activate the DataSync agents in the IN-CLOUD region, in preparation for creating tasks to transfer data. 16 | 17 | ## Module Steps 18 | 19 | #### 1. Configure CloudWatch to allow DataSync logging 20 | 21 | When you create a DataSync task, it can be configured to write logs to a CloudWatch log group. If there are any issues during the transfer then a log entry will be created for each problematic file, allowing you to investigate once the DataSync task completes. However, before DataSync can write logs to CloudWatch, it must first be configured with the proper resource policy. 22 | 23 | 1. Copy the text for the policy below into an editor and save it as a file on your local machine: 24 | 25 | { 26 | "Statement": [ 27 | { 28 | "Sid": "DataSyncLogsToCloudWatchLogs", 29 | "Effect": "Allow", 30 | "Action": [ 31 | "logs:PutLogEvents", 32 | "logs:CreateLogStream" 33 | ], 34 | "Principal": { 35 | "Service": "datasync.amazonaws.com" 36 | }, 37 | "Resource": "*" 38 | } 39 | ], 40 | "Version": "2012-10-17" 41 | } 42 | 43 | 2. From your command-line terminal, run the following command to apply the policy to CloudWatch: 44 | 45 | aws logs put-resource-policy --region --policy-name trustDataSync --policy-document file:// 46 | 47 | where \ is the file on your local machine that was created in step #1 above and \ is the AWS region you are using for your IN-CLOUD region. 48 | 49 | #### 2. Activate the DataSync agents 50 | 51 | Although the EC2 instances running the DataSync agents were deployed in the first module, before they can be used they first need to be activated in the **IN-CLOUD** region. Follow the steps below to active the agents. 52 | 53 | 1. Go to the AWS Management console page in the **IN-CLOUD** region and click **Services** then select **DataSync.** 54 | 2. If no DataSync agents exist, click the **Get started** button, otherwise click the **Create agent** button. 55 | 3. Leave the Service endpoint as **"Public service endpoints"**. 56 | 4. Under the **Activation key** section, enter the **Public IP address** of the first DataSync agent instance running in the ON-PREMISES region. You can get this IP address from the CloudFormation outputs in the on-premises region. You use the public IP address here because the agent needs to be accessible by your web browser for activation. Enter the IP address of the agent as shown below, then click **Get key.** 57 | 58 | ![](../images/mod2ds1.png) 59 | 60 | 5. Once the activation is successful, you will be shown the activation key and will be prompted for further information. 61 | 62 | ![](../images/mod2ds2.png) 63 | 64 | 6. Enter an Agent name ("Agent 1" or "Agent 2") and apply any tags if desired, then click **Create agent**. 65 | 7. Repeat the above steps for the second agent. 66 | 8. With both agents created, you should see two agents listed in the DataSync console, as shown below: 67 | 68 | ![](../images/mod2agentsList.png) 69 | 70 | ## Module Summary 71 | 72 | In this module you applied a resource policy to CloudWatch allowing the DataSync service to create log streams and to log events. After that, you successfully activated the DataSync agents in the IN-CLOUD region. 73 | 74 | In the next module, you will create a small DataSync task to run a simple transfer. This test run will validate that everything is working as expected, prior to starting the complete transfer. 75 | 76 | Go to [Module 4](/workshops/nfs-million-files/module4). 77 | -------------------------------------------------------------------------------- /workshops/nfs-million-files/module4/README.md: -------------------------------------------------------------------------------- 1 | # **AWS DataSync** 2 | 3 | ### Migrate millions of files using AWS DataSync 4 | 5 | © 2020 Amazon Web Services, Inc. and its affiliates. All rights reserved. 6 | This sample code is made available under the MIT-0 license. See the LICENSE file. 7 | 8 | Errors or corrections? Contact [jeffbart@amazon.com](mailto:jeffbart@amazon.com). 9 | 10 | --- 11 | 12 | # Module 4 13 | ## Run a test transfer to validate network and performance 14 | 15 | Before performing a full transfer of your data, it's always a good idea to do a test run to validate assumptions about performance and to verify DataSync settings. A test run will also validate proper network connectivity, ensuring your DataSync agent(s) can communicate properly with the DataSync service in the cloud. 16 | 17 | In this module, you will run a simple task to copy a portion of the data from the **fs1** file system to the S3 bucket. You will use both agents to validate network connectivity and functionality and you will use exclude filters to avoid copying the .htaccess and index.html files in each directory. Once the task has completed the transfer, you will use various tools to explore the task results. 18 | 19 | ## Module Steps 20 | 21 | #### 1. Create a new DataSync task 22 | 23 | 1. Go to the DataSync console in the **IN-CLOUD** region, click Tasks on the left side of the page and then click the **Create task** button. 24 | 2. You will be prompted to create a source location, which in this particular case, will be the **fs1** file system on your NFS server. Make sure **Create a new location** is selected and for _Location type_ select **Network File System (NFS)** from the drop-down. 25 | 3. You will use both agents for this task, so select **Agent 1** from the drop-down, then select **Agent 2**. Both agents should be listed as selected. 26 | 4. Enter the **Private IP address** of the NFS server, per the CloudFormation outputs in the **ON-PREMISES** region. This is the IP address that the DataSync agent will use to mount the NFS server. 27 | 5. Under _Mount path_, enter **/mnt/fs1/d0001**. This will copy files only from the 'd0001' directory on fs1. 28 | 29 | ![](../images/mod4ds1.png) 30 | 31 | 6. Click **Next**. 32 | 7. Now you will create the destination location, which is the S3 bucket. Make sure **Create a new location** is selected and for _Location type_ select **Amazon S3 bucket** from the drop-down. 33 | 8. From the _S3 bucket_ drop-down, select the bucket that was created as part of the CloudFormation stack in the **IN-CLOUD** region. The bucket name is prefixed with _million_files_nfs_. 34 | 9. For _S3 storage class_, select **Standard**. 35 | 10. Under _Folder_, enter **/fs1/d0001**. This will put all of the files transferred into this folder on the bucket and provides a place for future files to be copied. 36 | 11. Under _IAM role_, select the S3 bucket IAM role that starts with **MillionFiles-inCloud**. The full name of the role can be found in the outputs for the CloudFormation stack from the IN-CLOUD region. This role gives the DataSync service the rights necessary to write data to the S3 bucket. We could have also allowed the DataSync service to Autogenerate a role for us. 37 | 38 | ![](../images/mod4ds2.png) 39 | 40 | 12. Click **Next**. 41 | 42 | 13. Scroll up and name the task **Test Task**. 43 | 14. In the _Options_ section, under _Verify data_, select **Verify only the data transferred**. This will verify that every byte of every file transferred is written successfully to the destination. 44 | 15. Keep all other task options as-is. 45 | 16. Under the _Filtering configuration_ section, for _Exclude patterns_, enter **\*/.htaccess**. Click the _Add Pattern_ button and enter **\*/index.html**. This will prevent the task from copying these two files from any directory on the source file system. 46 | 47 | ![](../images/mod4ds3.png) 48 | 49 | 17. Scroll down to the _Task logging_ section and select the **DataSyncLogs-MillionFiles-InCloud** log group from the drop-down. 50 | 9. Click **Next** and verify all task settings are correct. 51 | 10. Click **Create task**. 52 | 53 | #### 2. Run the task 54 | 55 | 1. Wait for the **Task status** to transition from "Creating" to "Available". 56 | 2. To run the task, click the **Start** button, verify the settings one more time, and then click **Start**. 57 | 3. The task will immediately go into the "Running" state. 58 | 4. Under the **History** tab, click on the task execution object in the list. 59 | 5. As the task runs, the execution status will progress from "Launching" to "Preparing" to "Transferring" to "Verifying" and finally to "Success". The task execution will report statistics on the job, as shown below. It will take a few minutes for the task to complete. Once the task has finished, notice that **10,041** files were transferred. This includes the 20 sub-folders under d0001, containing 500 files each, plus the manifest.lst file in each sub-folder, along with an object in S3 for the folders themselves, including the d0001 folder itself. 60 | 61 | ![](../images/mod4ds4.png) 62 | 63 | ## Validation Step 64 | 65 | From the AWS management console, select **Services** then **S3.** In the bucket list, click on the bucket starting with **million-files-nfs**. You should see a top-level folder **fs1**, inside of which is a folder **d0001**. Inside this folder will be the 20 sub-folders containing the files that were copied, including the manifest.lst file for each sub-folder. Verify that the .htaccess and index.html files were _not_ copied. 66 | 67 | ![](../images/mod4ds5.png) 68 | 69 | To get more information on the results of the task execution, run the following commands, replacing the **execution ID** with the execution ID at the top of the DataSync console page (as shown above). Also, replace the **--region** parameter with your IN-CLOUD region: 70 | 71 | Use the "list-task-executions" command to get the full ARN of the task execution, which is then passed to the "describe-task-executions" command. 72 | 73 | ~$ aws datasync list-task-executions --region us-east-2 | grep exec-087e8a24374165fa2 74 | "TaskExecutionArn": "arn:aws:datasync:us-east-2:--:task/task-0409cb63be00594d2/execution/exec-087e8a24374165fa2", 75 | 76 | ~$ aws datasync describe-task-execution --region us-east-2 --task-execution-arn arn:aws:datasync:us-east-2:--:task/task-0409cb63be00594d2/execution/exec-087e8a24374165fa2 77 | 78 | The output from the "describe-task-executions" command is shown below: 79 | 80 | { 81 | "TaskExecutionArn": "arn:aws:datasync:us-east-2:--:task/task-0409cb63be00594d2/execution/exec-087e8a24374165fa2", 82 | "Status": "SUCCESS", 83 | "Options": { 84 | "VerifyMode": "ONLY_FILES_TRANSFERRED", 85 | "OverwriteMode": "ALWAYS", 86 | "Atime": "BEST_EFFORT", 87 | "Mtime": "PRESERVE", 88 | "Uid": "INT_VALUE", 89 | "Gid": "INT_VALUE", 90 | "PreserveDeletedFiles": "PRESERVE", 91 | "PreserveDevices": "NONE", 92 | "PosixPermissions": "PRESERVE", 93 | "BytesPerSecond": -1, 94 | "TaskQueueing": "ENABLED" 95 | }, 96 | "Excludes": [ 97 | { 98 | "FilterType": "SIMPLE_PATTERN", 99 | "Value": "*/.htaccess|*/index.html" 100 | } 101 | ], 102 | "Includes": [], 103 | "StartTime": 1578092083.501, 104 | "EstimatedFilesToTransfer": 10041, 105 | "EstimatedBytesToTransfer": 204188279, 106 | "FilesTransferred": 10041, 107 | "BytesWritten": 204188279, 108 | "BytesTransferred": 204188279, 109 | "Result": { 110 | "PrepareDuration": 3447, 111 | "PrepareStatus": "SUCCESS", 112 | "TotalDuration": 91639, 113 | "TransferDuration": 77060, 114 | "TransferStatus": "SUCCESS", 115 | "VerifyDuration": 13459, 116 | "VerifyStatus": "SUCCESS" 117 | } 118 | } 119 | 120 | The key takeaway from the command output is the time the task spent in the prepare, transfer and verify stages (PrepareDuration, TransferDuration, and VerifyDuration, respectively). The numbers are in milliseconds, so you can see from the above output that the task spent **3.4** seconds in the prepare phse, **77** seconds in the transfer phase, and **13.4** seconds in the verify phase. 121 | 122 | ## Module Summary 123 | 124 | In this module you successfully created a task to copy some of the files from fs1 on the on-premises NFS server into the S3 bucket. You then verified that the expected files were copied successfully. Because the command executed successfully, you have verified that both agents can communicate properly with the AWS DataSync service and you have verified that the exclude filter settings are configured properly. 125 | 126 | In the next module, you will create three tasks to copy files from the three file systems: fs1, fs2, and fs3. You will then execute the task and verify it executed successfully. 127 | 128 | Go to [Module 5](/workshops/nfs-million-files/module5). 129 | -------------------------------------------------------------------------------- /workshops/nfs-million-files/module5/README.md: -------------------------------------------------------------------------------- 1 | # **AWS DataSync** 2 | 3 | ### Migrate millions of files using AWS DataSync 4 | 5 | © 2020 Amazon Web Services, Inc. and its affiliates. All rights reserved. 6 | This sample code is made available under the MIT-0 license. See the LICENSE file. 7 | 8 | Errors or corrections? Contact [jeffbart@amazon.com](mailto:jeffbart@amazon.com). 9 | 10 | --- 11 | 12 | # Module 5 13 | ## Run the full transfer 14 | 15 | In the previous module, you created and ran a simple task to copy a portion of the files from fs1 to the S3 bucket. You did this to validate network connectivity of both agents and to verify that the exclude filters were configured properly. You are now ready to run the full transfer. 16 | 17 | In this module, you will create three DataSync tasks - one per file system on the NFS server. Because you only have two agents, you will copy the smaller file systems (fs1 and fs2) using one agent, and the other agent will be used to copy the larger file system (fs3). The task for fs2 will be queued while the task for fs1 executes. When the task for fs1 completes, the queued task will be executed automatically. 18 | 19 | ## Module Steps 20 | 21 | #### 1. Create the task for fs1 22 | 23 | 1. Go to the DataSync console in the **IN-CLOUD** region, click Tasks on the left side of the page and then click the **Create task** button. 24 | 2. Use the following settings for the source location: 25 | - Location type: **Network File System (NFS)** 26 | - Agents: use **Agent 1** only 27 | - NFS Server: use the **Private IP address** of the NFS server 28 | - Mount path: **/mnt/fs1** 29 | 3. Click **Next**. 30 | 4. Use the following settings for the destination location: 31 | - Location type: **Amazon S3 bucket** 32 | - S3 bucket: use the bucket name prefixed with **million_files_nfs** 33 | - S3 storage class: **Standard** 34 | - Folder: **/fs1** 35 | - IAM role: select the S3 bucket IAM role that starts with **MillionFiles-inCloud** 36 | 5. Click **Next**. 37 | 6. Name the task **Copy FS1**. 38 | 7. Set the _Verify data_ option to **Verify only the data transferred**. 39 | 8. Add two exclude patterns: **\*/.htaccess** and **\*/index.html** 40 | 9. For _Task logging_, select the **DataSyncLogs-MillionFiles-InCloud** log group from the drop-down. 41 | 10. Click **Next** and verify all task settings are correct. 42 | 11. Click **Create task**. 43 | 44 | #### 2. Create the task for fs2 45 | 46 | 1. Select Tasks and then click the **Create task** button. 47 | 2. Use the following settings for the source location: 48 | - Location type: **Network File System (NFS)** 49 | - Agents: use **Agent 1** only 50 | - NFS Server: use the **Private IP address** of the NFS server 51 | - Mount path: **/mnt/fs2** 52 | 3. Click **Next**. 53 | 4. Use the following settings for the destination location: 54 | - Location type: **Amazon S3 bucket** 55 | - S3 bucket: use the bucket name is prefixed with **million_files_nfs** 56 | - S3 storage class: **Standard** 57 | - Folder: **/fs2** 58 | - IAM role: select the S3 bucket IAM role that starts with **MillionFiles-inCloud** 59 | 5. Click **Next**. 60 | 6. Name the task **Copy FS2**. 61 | 7. Set the _Verify data_ option to **Verify only the data transferred**. 62 | 8. Make sure _Queueing_ is set to **Enabled**. 63 | 8. Add two exclude patterns: **\*/.htaccess** and **\*/index.html** 64 | 17. For _Task logging_, select the **DataSyncLogs-MillionFiles-InCloud** log group from the drop-down. 65 | 9. Click **Next** and verify all task settings are correct. 66 | 10. Click **Create task**. 67 | 68 | #### 3. Create the task for fs3 69 | 70 | 1. Select Tasks and then click the **Create task** button. 71 | 2. Use the following settings for the source location: 72 | - Location type: **Network File System (NFS)** 73 | - Agents: use **Agent 2** only 74 | - NFS Server: use the **Private IP address** of the NFS server 75 | - Mount path: **/mnt/fs3** 76 | 3. Click **Next**. 77 | 4. Use the following settings for the destination location: 78 | - Location type: **Amazon S3 bucket** 79 | - S3 bucket: use the bucket name is prefixed with **million_files_nfs** 80 | - S3 storage class: **Standard** 81 | - Folder: **/fs3** 82 | - IAM role: select the S3 bucket IAM role that starts with **MillionFiles-inCloud** 83 | 5. Click **Next**. 84 | 6. Name the task **Copy FS3**. 85 | 7. Set the _Verify data_ option to **Verify only the data transferred**. 86 | 8. Add two exclude patterns: **\*/.htaccess** and **\*/index.html** 87 | 17. For _Task logging_, select the **DataSyncLogs-MillionFiles-InCloud** log group from the drop-down. 88 | 9. Click **Next** and verify all task settings are correct. 89 | 10. Click **Create task**. 90 | 91 | You should now have four tasks total - the test task from the previous module and the three tasks you just created. 92 | 93 | #### 4. Start the tasks 94 | 95 | 1. Wait for the three tasks to transition from "Creating" to "Available". 96 | 2. Start the task for **fs1** first, followed by the task for **fs2**, and finally the task for **fs3**. 97 | 3. Notice that the task for **fs2** goes immediately to a _Queued_ status. This is because the tasks for fs1 and fs2 use the same agent (Agent 1) and a DataSync agent can only execute one task at a time. 98 | 99 | ![](../images/mod5ds1.png) 100 | 101 | Once the task for fs1 completes, the task for fs2 will be run automatically. 102 | 103 | #### 5. Monitor your tasks 104 | 105 | In a production environment, it's important to monitor all resources to be sure that DataSync transfers proceed as expected. This includes monitoring not only the DataSync tasks themslves, but the NFS server and network as well. Poor performance on the source storage, or network issues such as dropped packets or TCP retransmits can significantly degrade DataSync performance. 106 | 107 | While the three tasks execute, use CloudWatch to explore the various metrics available for both agents and tasks. 108 | 109 | It will take approximately 20 minutes for all tasks to complete. 110 | 111 | ## Validation Step 112 | 113 | Once all tasks have executed successfully, check the S3 bucket. You should now see three top-level folders: fs1, fs2, and fs3 (you may need to refresh the bucket). 114 | 115 | Compare the results of the task execution for fs1 with fs2. For the fs1 task, you should see **492,010** files transferred, while for the fs2 task, you should see **502,051** files transferred. Both file systems have the same number of files, so why the discrepency? 116 | 117 | Remember that when you ran the test task in module 4, you copied one top-level directory from fs1 - **10,041** files and directories total. Because those files had already been copied once, DataSync knew they did not have to be copied again and they were skipped during the task for fs1. DataSync made this determination during the Prepare phase of the task, when it compared files in the source to files in the destination and found some files already in the bucket. 118 | 119 | In total, across the four tasks, you transferred _and verified_ **2,008,203** files in about 25 minutes. 120 | 121 | ## Module Summary 122 | 123 | In this module you successfully created three tasks to copy files from the three file systems into the S3 bucket. You distrubuted your tasks across multiple agents and you saw how multiple tasks can be queued and executed automatically on a single agent. 124 | 125 | In the next module, you will perform two more transfers, diving deeper into how DataSync handles incremental transfers of data. 126 | 127 | Go to [Module 6](/workshops/nfs-million-files/module6). 128 | -------------------------------------------------------------------------------- /workshops/nfs-million-files/module6/README.md: -------------------------------------------------------------------------------- 1 | # **AWS DataSync** 2 | 3 | ### Migrate millions of files using AWS DataSync 4 | 5 | © 2020 Amazon Web Services, Inc. and its affiliates. All rights reserved. 6 | This sample code is made available under the MIT-0 license. See the LICENSE file. 7 | 8 | Errors or corrections? Contact [jeffbart@amazon.com](mailto:jeffbart@amazon.com). 9 | 10 | --- 11 | 12 | # Module 6 13 | ## Perform incremental copies of data 14 | 15 | In the previous module, you copied 2+ million files from your on-premises NFS server to your S3 bucket. 16 | 17 | In many cases, data transfers occur on file systems that are actively in production. Files may continue to be added and modified, even after the initial transfer has been completed. DataSync supports incremental transfers to make sure the destination location is in sync with the source. 18 | 19 | In this module you will perform two additional task executions to demonstrate how DataSync handles incremental transfers of data. 20 | 21 | ## Module Steps 22 | 23 | #### 1. Modify files on fs2 24 | 25 | From your SSH console to the NFS server, run the following command to list the three file systems: 26 | 27 | [ec2-user@ ~]$ cd /mnt/fs2/d0001/dir0001 28 | [ec2-user@ ~]$ dd if=/dev/urandom of=newfile1 bs=1M count=1 29 | [ec2-user@ ~]$ echo "newfile1" >> manifest.lst 30 | 31 | The above commands will create a new file on **fs2** named "newfile1" and append an entry for the new file to the end of the manifest.lst file in the local directory. This means you now have one new file on fs2 and one file that has been modified. 32 | 33 | #### 2. Re-run the FS2 task to get the changes 34 | 35 | To update your S3 bucket with the changes you made to fs2, you only need to simply re-run the task named **Copy FS2** that you created in the previous module. The source and destination locations are the same and you will keep the same task settings as you used previously. 36 | 37 | 1. Go to the DataSync console in the **IN-CLOUD** region, click Tasks on the left side of the page and then select the existing task with the name **Copy FS2**. 38 | 2. Click the **Start** button to launch a new execution of the task. 39 | 3. Verify the task settings, and then click **Start**. 40 | 4. Click on the **History** tab - you should now see two task executions, one for the task executed in the previous module and the task you just launched: 41 | 42 | ![](../images/mod6ds1.png) 43 | 44 | 5. Select the newest task execution from the list to monitor its progress. It will take a few minutes for the task to complete. 45 | 6. When the task completes, you should see that **3 files were transferred** and that the total data transferred was **1 MiB**. The three files are the new file create above (which was 1 MiB in size), the manifest.lst file that was modified, and then the object in the S3 bucket representing the dir0001 folder, which needed to be updated because a new file was added to it. 46 | 7. Go to your S3 bucket and verify that you see the new file along with an updated timestamp on the manifest.lst file. 47 | 48 | #### 3. Compare execution times 49 | 50 | Although this task executed faster than the previous run, you may have noticed that the task spent a few minutes in the Preparing phase. Run the following command to get the time spent preparing (replacing the region and execution ID with your own values): 51 | 52 | ~$ aws datasync list-task-executions --region us-east-2 | grep exec-07e667d3c41fa8341 53 | "TaskExecutionArn": "arn:aws:datasync:us-east-2:--:task/task-00007d923f4357143/execution/exec-07e667d3c41fa8341", 54 | 55 | ~$ aws datasync describe-task-execution --region us-east-2 --task-execution-arn arn:aws:datasync:us-east-2:--:task/task-00007d923f4357143/execution/exec-07e667d3c41fa8341 56 | { 57 | "TaskExecutionArn": "arn:aws:datasync:us-east-2:--:task/task-00007d923f4357143/execution/exec-07e667d3c41fa8341", 58 | "Status": "SUCCESS", 59 | "Options": { 60 | "VerifyMode": "ONLY_FILES_TRANSFERRED", 61 | "OverwriteMode": "ALWAYS", 62 | "Atime": "BEST_EFFORT", 63 | "Mtime": "PRESERVE", 64 | "Uid": "INT_VALUE", 65 | "Gid": "INT_VALUE", 66 | "PreserveDeletedFiles": "PRESERVE", 67 | "PreserveDevices": "NONE", 68 | "PosixPermissions": "PRESERVE", 69 | "BytesPerSecond": -1, 70 | "TaskQueueing": "ENABLED" 71 | }, 72 | "Excludes": [ 73 | { 74 | "FilterType": "SIMPLE_PATTERN", 75 | "Value": "*/.htaccess|*/index.html" 76 | } 77 | ], 78 | "Includes": [], 79 | "StartTime": 1578611450.582, 80 | "EstimatedFilesToTransfer": 3, 81 | "EstimatedBytesToTransfer": 1053085, 82 | "FilesTransferred": 3, 83 | "BytesWritten": 1053085, 84 | "BytesTransferred": 1053085, 85 | "Result": { 86 | "PrepareDuration": 126943, 87 | "PrepareStatus": "SUCCESS", 88 | "TotalDuration": 199632, 89 | "TransferDuration": 71423, 90 | "TransferStatus": "SUCCESS", 91 | "VerifyDuration": 2467, 92 | "VerifyStatus": "SUCCESS" 93 | } 94 | } 95 | 96 | 97 | In the above, you can see that the task spent about **127 seconds** in the prepare phase. Repeat the above commands but use the ID for the previous task execution. How much time did your first task for **fs2** spend in the prepare phase? 98 | 99 | You will likely see a significant difference in times spent in the prepare phase. While the first execution spent only a few seconds preparing, the second execution spent several minutes. Why the difference? 100 | 101 | When you executed the task to copy **fs2** for the first time, there were no objects in the S3 bucket under the /fs2 folder, so there was nothing to compare between the source and the destination. However, with the second execution, there were now 500,000+ objects in the S3 bucket for **fs2** that needed to be compared with the source. Comparing all of those files can take a while - in this case, several minutes. 102 | 103 | Is there a way to reduce this time? 104 | 105 | #### 3. One more incremental run 106 | 107 | One way to reduce the amount of time required for preparing files is to limit the scope of the files to be compared. You could do this by creating a new task with a more precise source location, but that would mean copying all of your settings to a new task, and risk missing a configuration setting. Another option is to use task-level include filters. 108 | 109 | Before doing another run, let's create more changes in the dir0001 directory fs2. Run the following commands: 110 | 111 | [ec2-user@ ~]$ cd /mnt/fs2/d0001/dir0001 112 | [ec2-user@ ~]$ dd if=/dev/urandom of=newfile2 bs=1M count=1 113 | [ec2-user@ ~]$ echo "newfile2" >> manifest.lst 114 | 115 | Now re-run the same task again. However, when prompted for task settings, scroll to the bottom of the dialog box and expand the section that says **Filtering configuration (optional)**. 116 | 117 | ![](../images/mod6ds2.png) 118 | 119 | Here you have the option to provide one or more **include** patterns, similar to the patterns you specified previously to exclude the .htaccess and index.html files. But in this case, you are telling the DataSync task which files and folders to explicitly include, ignoring everything else. 120 | 121 | In the edit box, enter the pattern **/d0001/dir0001/\*** and then click the **Start** button. 122 | 123 | ![](../images/mod6ds3.png) 124 | 125 | Remember that all filters (include and exclude) are relative to the path in the source location, which in this case is **/mnt/fs2/**. 126 | 127 | ## Validation Step 128 | 129 | Once the task completes execution, go back to your S3 bucket and verify you see the latest new file, **newfile2**, as well as an updated timestamp for the manifest.lst file. 130 | 131 | Run the **describe-task-execution** command again, with the latest task execution id, to get the duration for the prepare phase. How does this compare to the previous execution? 132 | 133 | You should see the prepare time has been reduced significantly. This is because the DataSync task only scanned files in the /d0001/dir0001/ directory, which is only about 500 files. Using include filters, you were able to significantly reduce the scope of the DataSync task, without having to create a new task. 134 | 135 | ## Module Summary 136 | 137 | In this module, you re-ran the copy task for **fs2** to get incremental changes to the file system. You also saw how include filters can be used to limit the scope of an existing task and thus reduce overall processing times. 138 | 139 | In the next module, you will go through the steps to clean up resources from this workshop. 140 | 141 | Go to [Module 7](/workshops/nfs-million-files/module7). 142 | -------------------------------------------------------------------------------- /workshops/nfs-million-files/module7/README.md: -------------------------------------------------------------------------------- 1 | # **AWS DataSync** 2 | 3 | ### Migrate millions of files using AWS DataSync 4 | 5 | © 2020 Amazon Web Services, Inc. and its affiliates. All rights reserved. 6 | This sample code is made available under the MIT-0 license. See the LICENSE file. 7 | 8 | Errors or corrections? Contact [jeffbart@amazon.com](mailto:jeffbart@amazon.com). 9 | 10 | --- 11 | 12 | # Module 7 13 | ## Workshop clean-up 14 | 15 | To make sure all resources are deleted after this workshop scenario make sure you execute the steps in the order outlined below (you do not need to wait for CloudFormation to finish deleting before moving to the next step): 16 | 17 | 1. Go to the **IN-CLOUD** region AWS management console and go to the **DataSync** service. 18 | 2. Select **Tasks** and delete the tasks you created during this workshop 19 | 3. Select **Locations** and delete the locations you created during this workshop 20 | 4. Select **Agents** and delete the agents you activated during this workshop. Note that this **will not** delete the actual DataSync agent EC2 instances. Those will get deleted when the associated on-premises CloudFormation stack is deleted. 21 | 22 | 5. Go to the CloudFormation page in the **ON-PREMISES** region and delete the stack named "MillionFiles-OnPrem". 23 | 24 | 6. Delete all objects in the S3 bucket in the in-cloud region. The bucket must be empty before it can be deleted by CloudFormation in the next step. **NOTE:** due to the large number of files in the bucket, this step will take approximately 1.5 hours to complete. 25 | 7. Once the bucket is empty, go to the CloudFormation page in the **IN-CLOUD** region and delete the stack named "MillionFiles-InCloud". 26 | 27 | 28 | To make sure that all CloudFormation templates have been deleted correctly, confirm that all EC2 instances created in this workshop in the on-premises region are in the **terminated** state. 29 | -------------------------------------------------------------------------------- /workshops/nfs-million-files/templates/nfs-million-files-in-cloud.yaml: -------------------------------------------------------------------------------- 1 | AWSTemplateFormatVersion: '2010-09-09' 2 | Description: AWS DataSync Workshop - NFS Small Files Migration - In-Cloud Region 3 | Metadata: 4 | License: 5 | Description: | 6 | Copyright 2019-2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. 7 | 8 | Permission is hereby granted, free of charge, to any person obtaining a copy of this 9 | software and associated documentation files (the "Software"), to deal in the Software 10 | without restriction, including without limitation the rights to use, copy, modify, 11 | merge, publish, distribute, sublicense, and/or sell copies of the Software, and to 12 | permit persons to whom the Software is furnished to do so. 13 | 14 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, 15 | INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A 16 | PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT 17 | HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION 18 | OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE 19 | SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 20 | 21 | Resources: 22 | 23 | # We use the GUID from the ARN of the stack ID to generate 24 | # a unique bucket name 25 | s3Bucket: 26 | Type: AWS::S3::Bucket 27 | Properties: 28 | PublicAccessBlockConfiguration: 29 | BlockPublicAcls: True 30 | BlockPublicPolicy: True 31 | IgnorePublicAcls: True 32 | RestrictPublicBuckets: True 33 | BucketName: !Join 34 | - "-" 35 | - - "million-files-nfs" 36 | - !Select 37 | - 2 38 | - !Split 39 | - "/" 40 | - !Ref "AWS::StackId" 41 | 42 | # Give the role a friendly name as the workshop user will need to 43 | # reference it when creating DataSync tasks. 44 | s3BucketIamRole: 45 | Type: AWS::IAM::Role 46 | Properties: 47 | AssumeRolePolicyDocument: 48 | Statement: 49 | - Action: 50 | - sts:AssumeRole 51 | Effect: Allow 52 | Principal: 53 | Service: 54 | - datasync.amazonaws.com 55 | Version: '2012-10-17' 56 | s3BucketRolePolicy: 57 | Type: AWS::IAM::Policy 58 | DependsOn: s3Bucket 59 | Properties: 60 | PolicyDocument: 61 | Statement: 62 | - Effect: Allow 63 | Resource: !GetAtt s3Bucket.Arn 64 | Action: 65 | - s3:GetBucketLocation 66 | - s3:ListBucket 67 | - s3:ListBucketMultipartUploads 68 | - s3:HeadBucket 69 | - Effect: Allow 70 | Resource: !Join [ "/", [ !GetAtt s3Bucket.Arn, "*" ] ] 71 | Action: 72 | - s3:AbortMultipartUpload 73 | - s3:DeleteObject 74 | - s3:GetObject 75 | - s3:ListMultipartUploadParts 76 | - s3:PutObject 77 | Version: '2012-10-17' 78 | PolicyName: policy 79 | Roles: 80 | - !Ref 's3BucketIamRole' 81 | 82 | dataSyncLogs: 83 | Type: AWS::Logs::LogGroup 84 | Properties: 85 | RetentionInDays: 3 86 | LogGroupName: !Join 87 | - '' 88 | - - DataSyncLogs- 89 | - !Ref 'AWS::StackName' 90 | 91 | Outputs: 92 | bucketName: 93 | Description: S3 Bucket Name 94 | Value: !Ref 's3Bucket' 95 | bucketRoleForDataSync: 96 | Description: S3 Bucket Role for DataSync 97 | Value: !Ref 's3BucketIamRole' 98 | -------------------------------------------------------------------------------- /workshops/nfs-million-files/templates/nfs-million-files-on-prem.yaml: -------------------------------------------------------------------------------- 1 | AWSTemplateFormatVersion: '2010-09-09' 2 | Description: AWS DataSync Workshop - NFS Small Files Migration - On-Premises Region 3 | Metadata: 4 | License: 5 | Description: | 6 | Copyright 2019-2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. 7 | 8 | Permission is hereby granted, free of charge, to any person obtaining a copy of this 9 | software and associated documentation files (the "Software"), to deal in the Software 10 | without restriction, including without limitation the rights to use, copy, modify, 11 | merge, publish, distribute, sublicense, and/or sell copies of the Software, and to 12 | permit persons to whom the Software is furnished to do so. 13 | 14 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, 15 | INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A 16 | PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT 17 | HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION 18 | OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE 19 | SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 20 | 21 | AWS::CloudFormation::Interface: 22 | ParameterGroups: 23 | - Label: 24 | default: Security 25 | Parameters: 26 | - kp 27 | - Label: 28 | default: AMI IDs (do not edit) 29 | Parameters: 30 | - linuxAmi 31 | - dsAmi 32 | ParameterLabels: 33 | kp: 34 | default: 'Access Key Pair:' 35 | linuxAmi: 36 | default: 'Linux' 37 | dsAmi: 38 | default: 'DataSync' 39 | 40 | Parameters: 41 | kp: 42 | Description: '(NOTE: If you don''t see a key in the list you will need to create 43 | one from the EC2 console in this region)' 44 | Type: AWS::EC2::KeyPair::KeyName 45 | linuxAmi: 46 | Type : 'AWS::SSM::Parameter::Value' 47 | Default: '/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2' 48 | dsAmi: 49 | Type : 'AWS::SSM::Parameter::Value' 50 | Default: '/aws/service/datasync/ami' 51 | 52 | Resources: 53 | 54 | # Create a dedicated VPC with internet connectivity 55 | dmVPC: 56 | Type: AWS::EC2::VPC 57 | Properties: 58 | CidrBlock: 10.12.14.0/24 59 | EnableDnsSupport: 'true' 60 | EnableDnsHostnames: 'true' 61 | InstanceTenancy: default 62 | Tags: 63 | - Key: Name 64 | Value: DsSmallFilesNfsWorkshopVPC 65 | dmSubnet1: 66 | Type: AWS::EC2::Subnet 67 | Properties: 68 | VpcId: !Ref 'dmVPC' 69 | CidrBlock: 10.12.14.0/24 70 | MapPublicIpOnLaunch: 'True' 71 | Tags: 72 | - Key: Name 73 | Value: DsSmallFilesNfsWorkshopSubnet1 74 | dmInternetGateway: 75 | Type: AWS::EC2::InternetGateway 76 | Properties: 77 | Tags: 78 | - Key: Name 79 | Value: DsSmallFilesNfsWorkshopIGW 80 | dmAttachGateway: 81 | Type: AWS::EC2::VPCGatewayAttachment 82 | Properties: 83 | VpcId: !Ref 'dmVPC' 84 | InternetGatewayId: !Ref 'dmInternetGateway' 85 | dmRouteTable: 86 | Type: AWS::EC2::RouteTable 87 | Properties: 88 | VpcId: !Ref 'dmVPC' 89 | Tags: 90 | - Key: Name 91 | Value: DsSmallFilesNfsWorkshopRouteTable 92 | dmSubnet1RouteAssociaton: 93 | Type: AWS::EC2::SubnetRouteTableAssociation 94 | Properties: 95 | SubnetId: !Ref 'dmSubnet1' 96 | RouteTableId: !Ref 'dmRouteTable' 97 | dmRoutetoInternet: 98 | Type: AWS::EC2::Route 99 | DependsOn: dmInternetGateway 100 | Properties: 101 | RouteTableId: !Ref 'dmRouteTable' 102 | DestinationCidrBlock: 0.0.0.0/0 103 | GatewayId: !Ref 'dmInternetGateway' 104 | 105 | # We use the same security group for all four resources. Technically port 80 106 | # isn't needed for the NFS server and the application server, but nothing is 107 | # listening on those ports on those servers. 108 | dmSecurityGroup: 109 | Type: AWS::EC2::SecurityGroup 110 | Properties: 111 | GroupDescription: DataSync Small Files NFS Workshop - Security Group for all resources 112 | VpcId: !Ref 'dmVPC' 113 | SecurityGroupIngress: 114 | - IpProtocol: tcp 115 | FromPort: '22' 116 | ToPort: '22' 117 | CidrIp: '0.0.0.0/0' 118 | - IpProtocol: tcp 119 | FromPort: '80' 120 | ToPort: '80' 121 | CidrIp: '0.0.0.0/0' 122 | 123 | # We use this so we can limit access on this port to the SG 124 | dmSecurityGroupIngress: 125 | Type: AWS::EC2::SecurityGroupIngress 126 | Properties: 127 | GroupId: !Ref 'dmSecurityGroup' 128 | IpProtocol: tcp 129 | ToPort: '2049' 130 | FromPort: '2049' 131 | SourceSecurityGroupId: !Ref 'dmSecurityGroup' 132 | DependsOn: dmSecurityGroup 133 | 134 | nfsServerInstanceProfile: 135 | Type: AWS::IAM::InstanceProfile 136 | Properties: 137 | Path: / 138 | Roles: 139 | - !Ref 'nfsServerIamRole' 140 | nfsServerIamRole: 141 | Type: AWS::IAM::Role 142 | Properties: 143 | AssumeRolePolicyDocument: 144 | Statement: 145 | - Action: 146 | - sts:AssumeRole 147 | Effect: Allow 148 | Principal: 149 | Service: 150 | - ec2.amazonaws.com 151 | Version: '2012-10-17' 152 | ManagedPolicyArns: 153 | - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore 154 | nfsServerRolePolicy: 155 | Type: AWS::IAM::Policy 156 | Properties: 157 | PolicyDocument: 158 | Statement: 159 | - Effect: Allow 160 | Action: 161 | - s3:GetObject 162 | Resource: 163 | - arn:aws:s3:::aws-datasync-samples/files/fs1m-small.tar 164 | - arn:aws:s3:::aws-datasync-samples/files/fs500k-small.tar 165 | Version: '2012-10-17' 166 | PolicyName: policy 167 | Roles: 168 | - !Ref 'nfsServerIamRole' 169 | nfsServer: 170 | Type: AWS::EC2::Instance 171 | CreationPolicy: 172 | ResourceSignal: 173 | Timeout: PT30M 174 | Count: 1 175 | Metadata: 176 | AWS::CloudFormation::Init: 177 | config: 178 | files: 179 | /opt/aws/cfn/initdata.sh: 180 | content: !Sub | 181 | #!/bin/bash 182 | mkdir -p /mnt/fs1 /mnt/fs2 /mnt/fs3 183 | devs=`lsblk | grep 200G | awk '{ print $1 }'` 184 | fsnum=1 185 | for d in $devs 186 | do 187 | mkfs.xfs /dev/$d 188 | mount /dev/$d /mnt/fs$fsnum 189 | fsnum=$(( fsnum + 1 )) 190 | done 191 | chmod 777 /mnt/fs* 192 | aws s3 cp s3://aws-datasync-samples/files/fs500k-small.tar /mnt/fs1/fs500k-small.tar >> /dev/null 193 | aws s3 cp s3://aws-datasync-samples/files/fs500k-small.tar /mnt/fs2/fs500k-small.tar >> /dev/null 194 | aws s3 cp s3://aws-datasync-samples/files/fs1m-small.tar /mnt/fs3/fs1m-small.tar >> /dev/null 195 | cd /mnt/fs1/ && tar -xf fs500k-small.tar & 196 | cd /mnt/fs2/ && tar -xf fs500k-small.tar & 197 | cd /mnt/fs3/ && tar -xf fs1m-small.tar & 198 | wait 199 | rm -f /mnt/fs1/fs500k-small.tar /mnt/fs2/fs500k-small.tar /mnt/fs3/fs1m-small.tar 200 | echo READY > /home/ec2-user/datasets_ready 201 | commands: 202 | 1-initData: 203 | command: "bash /opt/aws/cfn/initdata.sh" 204 | 2-finishUp: 205 | command: !Sub "/opt/aws/bin/cfn-signal -e 0 --stack ${AWS::StackId} --resource nfsServer --region ${AWS::Region}" 206 | Properties: 207 | ImageId: !Ref linuxAmi 208 | InstanceType: c5.4xlarge 209 | IamInstanceProfile: !Ref 'nfsServerInstanceProfile' 210 | Tags: 211 | - Key: Name 212 | Value: !Join 213 | - '' 214 | - - NfsServer- 215 | - !Ref 'AWS::StackName' 216 | KeyName: !Ref 'kp' 217 | InstanceInitiatedShutdownBehavior: terminate 218 | BlockDeviceMappings: 219 | - DeviceName: /dev/xvda 220 | Ebs: 221 | VolumeSize: '8' 222 | DeleteOnTermination: 'true' 223 | VolumeType: gp2 224 | - DeviceName: /dev/xvdb 225 | Ebs: 226 | VolumeSize: '200' 227 | DeleteOnTermination: 'true' 228 | VolumeType: gp2 229 | - DeviceName: /dev/xvdc 230 | Ebs: 231 | VolumeSize: '200' 232 | DeleteOnTermination: 'true' 233 | VolumeType: gp2 234 | - DeviceName: /dev/xvdd 235 | Ebs: 236 | VolumeSize: '200' 237 | DeleteOnTermination: 'true' 238 | VolumeType: gp2 239 | NetworkInterfaces: 240 | - AssociatePublicIpAddress: 'true' 241 | DeviceIndex: '0' 242 | GroupSet: 243 | - !Ref 'dmSecurityGroup' 244 | SubnetId: !Ref 'dmSubnet1' 245 | UserData: 246 | Fn::Base64: 247 | !Sub | 248 | #!/bin/bash 249 | /opt/aws/bin/cfn-init -v -s ${AWS::StackId} -r nfsServer --region ${AWS::Region} 250 | 251 | dataSyncAgentInstanceProfile: 252 | Type: AWS::IAM::InstanceProfile 253 | Properties: 254 | Path: / 255 | Roles: 256 | - !Ref 'dataSyncAgentIamRole' 257 | dataSyncAgentIamRole: 258 | Type: AWS::IAM::Role 259 | Properties: 260 | AssumeRolePolicyDocument: 261 | Statement: 262 | - Action: 263 | - sts:AssumeRole 264 | Effect: Allow 265 | Principal: 266 | Service: 267 | - ec2.amazonaws.com 268 | Version: '2012-10-17' 269 | dataSyncAgentRolePolicy: 270 | Type: AWS::IAM::Policy 271 | Properties: 272 | PolicyDocument: 273 | Statement: 274 | - Effect: Allow 275 | Action: 276 | - datasync:* 277 | Resource: 278 | - '*' 279 | - Effect: Allow 280 | Action: 281 | - iam:PassRole 282 | Resource: 283 | - '*' 284 | Version: '2012-10-17' 285 | PolicyName: policy 286 | Roles: 287 | - !Ref 'dataSyncAgentIamRole' 288 | 289 | dataSyncAgent1: 290 | Type: AWS::EC2::Instance 291 | Properties: 292 | ImageId: !Ref dsAmi 293 | InstanceType: m5.2xlarge 294 | IamInstanceProfile: !Ref 'dataSyncAgentInstanceProfile' 295 | Tags: 296 | - Key: Name 297 | Value: !Join 298 | - '' 299 | - - DataSyncAgent1- 300 | - !Ref 'AWS::StackName' 301 | KeyName: !Ref 'kp' 302 | InstanceInitiatedShutdownBehavior: stop 303 | BlockDeviceMappings: 304 | - DeviceName: /dev/xvda 305 | Ebs: 306 | VolumeSize: '80' 307 | DeleteOnTermination: 'true' 308 | VolumeType: gp2 309 | NetworkInterfaces: 310 | - AssociatePublicIpAddress: 'true' 311 | DeviceIndex: '0' 312 | GroupSet: 313 | - !Ref 'dmSecurityGroup' 314 | SubnetId: !Ref 'dmSubnet1' 315 | 316 | dataSyncAgent2: 317 | Type: AWS::EC2::Instance 318 | Properties: 319 | ImageId: !Ref dsAmi 320 | InstanceType: m5.2xlarge 321 | IamInstanceProfile: !Ref 'dataSyncAgentInstanceProfile' 322 | Tags: 323 | - Key: Name 324 | Value: !Join 325 | - '' 326 | - - DataSyncAgent2- 327 | - !Ref 'AWS::StackName' 328 | KeyName: !Ref 'kp' 329 | InstanceInitiatedShutdownBehavior: stop 330 | BlockDeviceMappings: 331 | - DeviceName: /dev/xvda 332 | Ebs: 333 | VolumeSize: '80' 334 | DeleteOnTermination: 'true' 335 | VolumeType: gp2 336 | NetworkInterfaces: 337 | - AssociatePublicIpAddress: 'true' 338 | DeviceIndex: '0' 339 | GroupSet: 340 | - !Ref 'dmSecurityGroup' 341 | SubnetId: !Ref 'dmSubnet1' 342 | 343 | Outputs: 344 | nfsServerPrivateIP: 345 | Description: NFS Server Private IP Address 346 | Value: !GetAtt nfsServer.PrivateIp 347 | dataSyncAgent1PublicIP: 348 | Description: DataSync Agent 1 - Public IP Address 349 | Value: !GetAtt dataSyncAgent1.PublicIp 350 | dataSyncAgent1PrivateIP: 351 | Description: DataSync Agent 1 - Private IP Address 352 | Value: !GetAtt dataSyncAgent1.PrivateIp 353 | dataSyncAgent2PublicIP: 354 | Description: DataSync Agent 2 - Public IP Address 355 | Value: !GetAtt dataSyncAgent2.PublicIp 356 | dataSyncAgent2PrivateIP: 357 | Description: DataSync Agent 2 - Private IP Address 358 | Value: !GetAtt dataSyncAgent2.PrivateIp 359 | -------------------------------------------------------------------------------- /workshops/scality-object-migration/EN/README.md: -------------------------------------------------------------------------------- 1 | # **AWS DataSync** 2 | 3 | ### Object store migration using AWS DataSync from Scality to Amazon S3 4 | 5 | © 2019 Amazon Web Services, Inc. and its affiliates. All rights reserved. 6 | This sample code is made available under the MIT-0 license. See the LICENSE file. 7 | 8 | Errors or corrections? Contact [owolabip@amazon.com](mailto:owolabip@amazon.com). 9 | 10 | --- 11 | 12 | ## Workshop scenario 13 | 14 | In your data center, you have an Object store that is starting to age out. Over the years the data stored has continued to grow exponentially adding a significant management overhead. Your applications and service owners are making copies and replicas of the data stored in the object store to other locations such as AWS S3. To reduce your data center footprint and to free up resources, you would like to move the data on the object store into the cloud. Also, you want to ensure that the data is transfered in the native format and accessible to the applications and users already building solutions in the cloud. 15 | 16 | After doing some research, you have realized that you can use [AWS DataSync](https://aws.amazon.com/datasync/) to migrate the data from your on-premises object store to Amazon S3. 17 | 18 | This workshop will walk you through this scenario, using CloudFormation templates to deploy resources and the AWS Management console to configure those resources accordingly. As shown in the architecture diagram below, an object store (Scality - Zenko), an Application server, and a DataSync agent will be deployed in an AWS region simulating the on-premises environment. An S3 bucket and an Application server will be created in an AWS region, simulating the AWS cloud region to which the object store data will be migrated. 19 | 20 | ![](images/fullarch.png) 21 | 22 | ## Topics covered 23 | 24 | - Deploying resources using CloudFormation 25 | - Registering and configuring DataSync agent 26 | - Configure CloudWatch logging for DataSync 27 | - Planning a migration of object using DataSync 28 | - Using DataSync to perform incremental updates 29 | 30 | ## Prerequisites 31 | 32 | #### AWS Account 33 | 34 | In order to complete this workshop, you will need an AWS account with rights to create AWS IAM roles, EC2 instances, AWS DataSync and CloudFormation stacks in the AWS regions you select. 35 | 36 | #### Software 37 | 38 | - **Internet Browser** – It is recommended that you use the latest version of Chrome or Firefox for this workshop. 39 | 40 | ## Cost 41 | 42 | It will cost approximately **3.00 USD** to run this workshop. It is recommended that you follow the cleanup instructions once you have completed the workshop to remove all deployed resources and limit ongoing costs to your AWS account. 43 | 44 | ## Related workshops 45 | 46 | - [NFS server migration using AWS DataSync and AWS Storage Gateway](https://github.com/aws-samples/aws-datasync-migration-workshop/tree/master/workshops/nfs-migration) 47 | - [Migrate millions of files using AWS DataSync](https://github.com/aws-samples/aws-datasync-migration-workshop/blob/master/workshops/nfs-million-files) 48 | - [Migrate to FSx Windows File Server using AWS DataSync](https://github.com/aws-samples/aws-datasync-fsx-windows-migration) 49 | - [Get hands-on with online data migration options to simplify & accelerate your journey to AWS](https://github.com/aws-samples/aws-online-data-migration-workshop) 50 | 51 | ## Workshop Modules 52 | 53 | This workshop consists of the following five modules: 54 | 55 | - [Module 1](module1/) - Deploy resources in the on-premises and in-cloud regions 56 | - [Module 2](module2/) - Initial copy from Scality to S3 using DataSync 57 | - [Module 3](module3/) - Securely access data on in-cloud region S3 bucket 58 | - [Module 4](module4/) - One last incremental copy before cutover 59 | - [Module 5](module5/) - Cutover to AWS S3 and clean up resources 60 | 61 | To get started, go to [Module 1](module1/). 62 | -------------------------------------------------------------------------------- /workshops/scality-object-migration/EN/images/fullarch.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/scality-object-migration/EN/images/fullarch.png -------------------------------------------------------------------------------- /workshops/scality-object-migration/EN/images/mod1arch.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/scality-object-migration/EN/images/mod1arch.png -------------------------------------------------------------------------------- /workshops/scality-object-migration/EN/images/mod1cli1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/scality-object-migration/EN/images/mod1cli1.png -------------------------------------------------------------------------------- /workshops/scality-object-migration/EN/images/mod1connect.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/scality-object-migration/EN/images/mod1connect.png -------------------------------------------------------------------------------- /workshops/scality-object-migration/EN/images/mod1output1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/scality-object-migration/EN/images/mod1output1.png -------------------------------------------------------------------------------- /workshops/scality-object-migration/EN/images/mod1output2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/scality-object-migration/EN/images/mod1output2.png -------------------------------------------------------------------------------- /workshops/scality-object-migration/EN/images/mod1ssh1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/scality-object-migration/EN/images/mod1ssh1.png -------------------------------------------------------------------------------- /workshops/scality-object-migration/EN/images/mod2arch.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/scality-object-migration/EN/images/mod2arch.png -------------------------------------------------------------------------------- /workshops/scality-object-migration/EN/images/mod2ds1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/scality-object-migration/EN/images/mod2ds1.png -------------------------------------------------------------------------------- /workshops/scality-object-migration/EN/images/mod2ds2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/scality-object-migration/EN/images/mod2ds2.png -------------------------------------------------------------------------------- /workshops/scality-object-migration/EN/images/mod2ds3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/scality-object-migration/EN/images/mod2ds3.png -------------------------------------------------------------------------------- /workshops/scality-object-migration/EN/images/mod2ds4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/scality-object-migration/EN/images/mod2ds4.png -------------------------------------------------------------------------------- /workshops/scality-object-migration/EN/images/mod2ds5.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/scality-object-migration/EN/images/mod2ds5.png -------------------------------------------------------------------------------- /workshops/scality-object-migration/EN/images/mod2ds6.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/scality-object-migration/EN/images/mod2ds6.png -------------------------------------------------------------------------------- /workshops/scality-object-migration/EN/images/mod2ds7.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/scality-object-migration/EN/images/mod2ds7.png -------------------------------------------------------------------------------- /workshops/scality-object-migration/EN/images/mod2ds8.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/scality-object-migration/EN/images/mod2ds8.png -------------------------------------------------------------------------------- /workshops/scality-object-migration/EN/images/mod2ds9.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/scality-object-migration/EN/images/mod2ds9.png -------------------------------------------------------------------------------- /workshops/scality-object-migration/EN/images/mod2validate.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/scality-object-migration/EN/images/mod2validate.png -------------------------------------------------------------------------------- /workshops/scality-object-migration/EN/images/mod3s1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/scality-object-migration/EN/images/mod3s1.png -------------------------------------------------------------------------------- /workshops/scality-object-migration/EN/images/mod3s2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/scality-object-migration/EN/images/mod3s2.png -------------------------------------------------------------------------------- /workshops/scality-object-migration/EN/images/mod4cli1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/scality-object-migration/EN/images/mod4cli1.png -------------------------------------------------------------------------------- /workshops/scality-object-migration/EN/images/mod4ds1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/scality-object-migration/EN/images/mod4ds1.png -------------------------------------------------------------------------------- /workshops/scality-object-migration/EN/images/mod4ds2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/scality-object-migration/EN/images/mod4ds2.png -------------------------------------------------------------------------------- /workshops/scality-object-migration/EN/images/mod4ds3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/scality-object-migration/EN/images/mod4ds3.png -------------------------------------------------------------------------------- /workshops/scality-object-migration/EN/images/mod5s31.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-datasync-migration-workshop/aceaf575e95b36863e6c38d618329b9d5a855a81/workshops/scality-object-migration/EN/images/mod5s31.png -------------------------------------------------------------------------------- /workshops/scality-object-migration/EN/module1/README.md: -------------------------------------------------------------------------------- 1 | # **AWS DataSync** 2 | 3 | ### Object storage migration using AWS DataSync from Scality to Amazon S3 4 | 5 | © 2019 Amazon Web Services, Inc. and its affiliates. All rights reserved. 6 | This sample code is made available under the MIT-0 license. See the LICENSE file. 7 | 8 | Errors or corrections? Contact [owolabip@amazon.com](mailto:owolabip@amazon.com). 9 | 10 | --- 11 | 12 | # Module 1 13 | ## Deploy resources in the on-premises and in-cloud regions 14 | 15 | In this module, you will use CloudFormation scripts to deploy resources in two AWS regions: one that represents the on-premises environment, and one for the in-cloud environment. Once all resources have been deployed, you will access the data on Scality from the Application server and verify existing objects. 16 | 17 | ![](../images/mod1arch.png) 18 | 19 | ## Module Steps 20 | 21 | #### 1. Deploy AWS resources for the on-premises region 22 | 23 | 1. Click one of the launch links in the table below to deploy the **on-premises** resources using CloudFormation. To avoid errors during deployment, select a region in which you have previously created AWS resources. 24 | 25 | | **Region Code** | **Region Name** | **Launch** | 26 | | --- | --- | --- | 27 | | us-west-1 | US West (N. California) | [Launch in us-west-1](https://console.aws.amazon.com/cloudformation/home?region=us-west-1#/stacks/new?stackName=DataMigrationWorkshop-onPremResources&templateURL=https://aws-datasync-samples.s3-us-west-2.amazonaws.com/workshops/object-migration/data-migration-workshop-on-prem.yaml) | 28 | | us-west-2 | US West (Oregon) | [Launch in us-west-2](https://console.aws.amazon.com/cloudformation/home?region=us-west-2#/stacks/new?stackName=DataMigrationWorkshop-onPremResources&templateURL=https://aws-datasync-samples.s3-us-west-2.amazonaws.com/workshops/object-migration/data-migration-workshop-on-prem.yaml) | 29 | | us-east-1 | US East (N. Virginia) | [Launch in us-east-1](https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?stackName=DataMigrationWorkshop-onPremResources&templateURL=https://aws-datasync-samples.s3-us-west-2.amazonaws.com/workshops/object-migration/data-migration-workshop-on-prem.yaml) | 30 | | us-east-2 | US East (Ohio) | [Launch in us-east-2](https://console.aws.amazon.com/cloudformation/home?region=us-east-2#/stacks/new?stackName=DataMigrationWorkshop-onPremResources&templateURL=https://aws-datasync-samples.s3-us-west-2.amazonaws.com/workshops/object-migration/data-migration-workshop-on-prem.yaml) | 31 | | eu-west-1 | Ireland | [Launch in eu-west-1](https://console.aws.amazon.com/cloudformation/home?region=eu-west-1#/stacks/new?stackName=DataMigrationWorkshop-onPremResources&templateURL=https://aws-datasync-samples.s3-us-west-2.amazonaws.com/workshops/object-migration/data-migration-workshop-on-prem.yaml) | 32 | | eu-central-1 | Frankfurt | [Launch in eu-central-1](https://console.aws.amazon.com/cloudformation/home?region=eu-central-1#/stacks/new?stackName=DataMigrationWorkshop-onPremResources&templateURL=https://aws-datasync-samples.s3-us-west-2.amazonaws.com/workshops/object-migration/data-migration-workshop-on-prem.yaml) | 33 | 34 | 2. Click **Next** on the Create Stack page. 35 | 3. If you want to access EC2 instances created in this workshop using your local SSH client then enter the name of an EC2 key pair in your selected region. Otherwise, you can leave the key pair blank. 36 | 3. Click **Next**. 37 | 4. Click **Next**. 38 | 5. Click **Next** again. (skipping the Options and Advanced options sections) 39 | 6. On the Review page, scroll to the bottom and check the box to acknowledge that CloudFormation will create IAM resources, then click **Create stack**. 40 | 41 | **Note:** Instances that are launched as part of this CloudFormation template may be in the initializing state for few minutes. 42 | 43 | While the CloudFormation deployment progresses in the on-premises region, you can proceed to deploy resources for the in-cloud region. 44 | 45 | #### 2. Deploy AWS resources for the in-cloud region 46 | 47 | 1. Click one of the launch links in the table below to deploy the **in-cloud** resources using CloudFormation. Use a different region from the on-premises region. To avoid errors in deployment, select a region in which you have previously created AWS resources. 48 | 49 | | **Region Code** | **Region Name** | **Launch** | 50 | | --- | --- | --- | 51 | | us-west-1 | US West (N. California) | [Launch in us-west-1](https://console.aws.amazon.com/cloudformation/home?region=us-west-1#/stacks/new?stackName=DataMigrationWorkshop-inCloudResources&templateURL=https://aws-datasync-samples.s3-us-west-2.amazonaws.com/workshops/object-migration/data-migration-workshop-in-cloud.yaml) | 52 | | us-west-2 | US West (Oregon) | [Launch in us-west-2](https://console.aws.amazon.com/cloudformation/home?region=us-west-2#/stacks/new?stackName=DataMigrationWorkshop-inCloudResources&templateURL=https://aws-datasync-samples.s3-us-west-2.amazonaws.com/workshops/object-migration/data-migration-workshop-in-cloud.yaml) | 53 | | us-east-1 | US East (N. Virginia) | [Launch in us-east-1](https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?stackName=DataMigrationWorkshop-inCloudResources&templateURL=https://aws-datasync-samples.s3-us-west-2.amazonaws.com/workshops/object-migration/data-migration-workshop-in-cloud.yaml) | 54 | | us-east-2 | US East (Ohio) | [Launch in us-east-2](https://console.aws.amazon.com/cloudformation/home?region=us-east-2#/stacks/new?stackName=DataMigrationWorkshop-inCloudResources&templateURL=https://aws-datasync-samples.s3-us-west-2.amazonaws.com/workshops/object-migration/data-migration-workshop-in-cloud.yaml) | 55 | | eu-west-1 | Ireland | [Launch in eu-west-1](https://console.aws.amazon.com/cloudformation/home?region=eu-west-1#/stacks/new?stackName=DataMigrationWorkshop-inCloudResources&templateURL=https://aws-datasync-samples.s3-us-west-2.amazonaws.com/workshops/object-migration/data-migration-workshop-in-cloud.yaml) | 56 | | eu-central-1 | Frankfurt | [Launch in eu-central-1](https://console.aws.amazon.com/cloudformation/home?region=eu-central-1#/stacks/new?stackName=DataMigrationWorkshop-inCloudResources&templateURL=https://aws-datasync-samples.s3-us-west-2.amazonaws.com/workshops/object-migration/data-migration-workshop-in-cloud.yaml) | 57 | 58 | 2. Click **Next** on the Create stack page. 59 | 3. Click **Next** (there are no stack parameters) **.** 60 | 4. Click **Next** again. (skipping the Options and Advanced options sections) 61 | 5. On the Review page, scroll to the bottom and check the box to acknowledge that CloudFormation will create IAM resources, then click **Create stack**. 62 | 63 | Wait for the CloudFormation stacks in each region to reach the CREATE\_COMPLETE state before proceeding to the next steps. It should take about 10 minutes for both CloudFormation stacks to complete. 64 | 65 | **NOTE:** If a stack fails to deploy because an EC2 instance type is not available in a particular availability zone, delete the stack and retry in the same region or in a different region. 66 | 67 | #### 3. Stack Outputs 68 | 69 | Upon completion, each CloudFormation stack will have a list of "Outputs". These are values such as IP addresses and resource names that will be used throughout the workshop. You can either copy these values elsewhere or keep the page open in your browser and refer to them as you go through the workshop. 70 | 71 | On the CloudFormation page in the **on-premises** region, click on the **Outputs** tab, as shown in the image below. You should see four values listed: 72 | 73 | - **appServerPrivateIP** – This is the private IP address of the Application Server. You will use this to demonstrate access to the objects on both on-premises and in-cloud environments. 74 | - **dataSyncAgentPublicIP** – This is the public IP address of the EC2 instance running the DataSync agent. You will use this when activating the DataSync agent. 75 | - **S3ServerPrivateIP** – This is the public IP address of the EC2 instance running the Scality/Zenko object store. You will use this when accessing the objects. 76 | 77 | ![](../images/mod1output1.png) 78 | 79 | On the CloudFormation page in the **in-cloud** region, click on the **Outputs** tab as shown in the image below. You should see two values listed: 80 | 81 | - **bucketName** – This is the name of the S3 bucket where the data will be copied to. You will use this when creating a file share on the File Gateway. 82 | - **bucketRoleForDataSync** – This is the role that will be used by the DataSync agent to write files to the S3 bucket. You will use this when creating the S3 location for DataSync. 83 | 84 | ![](../images/mod1output2.png) 85 | 86 | #### 4. Connect to the Application server 87 | 88 | 1. From the AWS console in the **on-premises** region, click **Services** and select **EC2.** 89 | 2. Select **Instances** from the menu on the left. 90 | 3. Right-click on the **ApplicationServer** instance and select **Connect** from the menu. 91 | 92 | If you configured an EC2 Key Pair when you deployed the on-premises CloudFormation template, then you can connect using an SSH client from your computer: 93 | 94 | ![](../images/mod1ssh1.png) 95 | 96 | Otherwise you can connect to the instanct directly through your browser using either EC2 Instance Connect or Session Manager: 97 | 98 | ![](../images/mod1connect.png) 99 | 100 | If using EC2 Instance Connect or Session Manager, click **Connect**. A new tab will be opened in your browser with a command line interface (CLI) to the Application server. Keep this tab open - you will use the command line on the Application server throughout this workshop. 101 | 102 | ## Validation Step 103 | 104 | In the CLI for the Application server, run the following commands to verify the files on the object store. Use the **S3ServerPrivateIP** value from the **on-premises** CloudFormation stack output. 105 | 106 | $ aws configure set aws_access_key_id AccessKey --profile scality 107 | $ aws configure set aws_secret_access_key SecretKey --profile scality 108 | $ aws s3 ls s3://data/images/ --endpoint-url http://:8000/ --profile scality 109 | $ aws s3 ls s3://data/images/ --endpoint-url http://:8000/ --profile scality | wc -l 110 | 111 | In the bucket s3://data images folder you should see 200 image objects, as shown below. These images represent the object store data that will be migrated to the S3 bucket in the in-cloud region. 112 | 113 | ![](../images/mod1cli1.png) 114 | 115 | ## Module Summary 116 | 117 | In this module, you deployed all of the resources necessary to complete this workshop in both the on-premises region and the in-cloud region. You have also accessed the on-premises object store from the Application server and verified your data set. 118 | 119 | In the next module, you will create a DataSync task to do an initial copy of files from the Scality to the AWS S3 bucket. 120 | 121 | Go to [Module 2](../module2/). 122 | -------------------------------------------------------------------------------- /workshops/scality-object-migration/EN/module2/README.md: -------------------------------------------------------------------------------- 1 | # **AWS DataSync** 2 | 3 | ### Object store migration using AWS DataSync from Scality to Amazon S3 4 | 5 | © 2019 Amazon Web Services, Inc. and its affiliates. All rights reserved. 6 | This sample code is made available under the MIT-0 license. See the LICENSE file. 7 | 8 | Errors or corrections? Contact [owolabip@amazon.com](mailto:owolabip@amazon.com). 9 | 10 | --- 11 | 12 | # Module 2 13 | ## Initial copy from Scality to S3 using DataSync 14 | 15 | In this module, you will activate the DataSync agent deployed in the on-premises region, create DataSync locations, and then create a DataSync task to copy data from the source location to the destination location. 16 | 17 | DataSync tasks perform the job of copying data and require two "locations" – a source and a destination. With DataSync, a location is an endpoint where files/objects reside or will be copied to. A location can be an NFS export, an SMB share, an object storage bucket, an Amazon S3 bucket, or an Amazon EFS file system. Location objects are independent of tasks and a single location can be used with multiple tasks. 18 | 19 | ![](../images/mod2arch.png) 20 | 21 | ## Module Steps 22 | 23 | #### 1. Activate the DataSync agent 24 | 25 | Although the agent instance was created in the previous module, before it can be used it needs to be activated in the **in-cloud** region. Follow the steps below to active the agent. 26 | 27 | 1. Go to the AWS Management console page in the **in-cloud** region and click **Services** then select **DataSync.** 28 | 2. If no DataSync agents exist, click the **Get started** button, otherwise click the **Create agent** button. 29 | 3. The EC2 instance that will run the DataSync agent has already been deployed in the **on-premises** region. 30 | 4. Leave the Service endpoint as **"Public service endpoints"**. 31 | 32 | 5. Under the **Activation key** section, enter the **Public IP address** of the DataSync agent instance running in the on-premises region. You can get this IP address from the CloudFormation outputs in the on-premises region. You use the public IP address here because the agent needs to be accessible by your web browser for activation. Enter the IP address of the agent as shown below, then click **Get key.** 33 | 34 | ![](../images/mod2ds1.png) 35 | 36 | 6. Once the activation is successful, you will be shown the activation key and will be prompted for further information. 37 | 38 | ![](../images/mod2ds2.png) 39 | 40 | 7. Enter an Agent name if desired, then click **Create agent**. 41 | 42 | #### 2. Create object storage location 43 | 44 | 1. On the left-hand side of the DataSync service page, click on **Locations** and then click on **Create location**. 45 | 46 | 2. Create a location for the on-premises object storage bucket. Select **Object storage** from the _Location type_ drop-down. 47 | 3. From the _Agents_ drop-down, select the DataSync agent that was created in the previous step. 48 | 4. For _Server_ enter the **Private IP address** of the object storage **S3Server**, per the CloudFormation outputs in the **on-premises** region. This was the same IP address used to access the objects from the Application server, in the previous module. 49 | 5. Under _Bucket name_, enter "data". 50 | 6. Under _Folder_, enter "/". 51 | 7. Expand _Additional settings_ and select "HTTP". Under _Server port_ enter "8000". 52 | 8. For _Access Key_ and _Secret key_ enter "AccessKey" and "SecretKey". 53 | 54 | ![](../images/mod2ds3.png) 55 | 56 | 9. Click **Create location**. 57 | 58 | #### 3. Create S3 location 59 | 60 | 1. On the left-hand side of the DataSync service page, click on **Locations** and then click on **Create location**. 61 | 62 | 2. Create a location for the S3 bucket. Select **Amazon S3 bucket** from the _Location type_ drop-down. 63 | 3. From the _S3 bucket_ drop-down, select the S3 bucket that starts with **data-migration-workshop** and is followed by a long GUID. 64 | 4. Keep the S3 storage class as **Standard**. 65 | 5. Under _Folder_, enter "/". 66 | 6. Under _IAM role_, select the S3 bucket IAM role that starts with **DataMigrationWorkshop-inCloud**. The full name of the role can be found in the outputs for the in-cloud CloudFormation stack. 67 | 68 | ![](../images/mod2ds4.png) 69 | 70 | 7. Click **Create location**. 71 | 72 | On the left-side of the page, click **Locations** again. You should now have two locations listed. One for the object storage and one for the S3 bucket. 73 | 74 | ![](../images/mod2ds5.png) 75 | 76 | #### 4. Create a task 77 | 78 | 1. On the left-hand side of the DataSync service page, click on **Tasks** and then click on **Create task**. 79 | 80 | 2. Under _Source location options_, select **Choose an existing location**. 81 | 3. Under the _Existing locations_ drop-down, select the object storage location you created previously. 82 | 4. Click **Next.** 83 | 84 | ![](../images/mod2ds6.png) 85 | 86 | 5. Under _Destination location options_, select **Choose an existing location**. 87 | 6. Under the _Existing locations_ drop-down, select the S3 bucket location you created previously. 88 | 7. Click **Next.** 89 | 8. Under the **Verify data** drop-down, select **Verify only the data transferred**. 90 | 9. Under _Task Logging_ and _CloudWatch log group_ select **dmLogGroup** from the drop-down list. Keep all other options as-is and then click **Next.** 91 | 10. Click **Create task.** 92 | 93 | #### 5. Run the task 94 | 95 | 1. Wait for the **Task status** to report "Available" (you may need to refresh the page). 96 | 97 | ![](../images/mod2ds7.png) 98 | 99 | 2. To run the task, click the **Start** button, verify the settings, and then click **Start**. 100 | 3. The task will immediately go into the "Running" state. 101 | 4. Under the **History** tab, click on the task execution object in the list. 102 | 103 | ![](../images/mod2ds8.png) 104 | 105 | 5. As the task runs, the execution status will progress from "Launching" to "Preparing" to "Transferring" to "Verifying" and finally to "Success". The task execution will report statistics on the job, as shown below. It will take a few minutes for the task to complete. Once the task has finished, notice that 200 files were transferred. This is the 200 files in the data set along with the two folders in the path that we specified. 106 | 107 | ![](../images/mod2ds9.png) 108 | 109 | ## Validation Step 110 | 111 | From the in-cloud region management console, select **Services** then **S3.** In the bucket list, click on the **data-migration-workshop** bucket. You should see a top-level folder named "images". Inside this folder should be the 200 .jpg files from the on-premises object storage. 112 | 113 | ![](../images/mod2validate.png) 114 | 115 | ## Module Summary 116 | 117 | In this module you successfully activated the DataSync agent and created a task to copy files from the on-premises object storage into the AWS S3 bucket. You then verified that the files were copied successfully. 118 | 119 | In the next module, you will configure the in-cloud Application server to connect to the S3 bucket, providing access to the in-cloud object via S3. 120 | 121 | Go to [Module 3](../module3/). 122 | -------------------------------------------------------------------------------- /workshops/scality-object-migration/EN/module3/README.md: -------------------------------------------------------------------------------- 1 | # **AWS DataSync** 2 | 3 | ### Object store migration using AWS DataSync from Scality to Amazon S3 4 | 5 | © 2019 Amazon Web Services, Inc. and its affiliates. All rights reserved. 6 | This sample code is made available under the MIT-0 license. See the LICENSE file. 7 | 8 | Errors or corrections? Contact [owolabip@amazon.com](mailto:owolabip@amazon.com). 9 | 10 | --- 11 | 12 | # Module 3 13 | ## Securely access data on in-cloud region S3 bucket 14 | 15 | You now have the files from the object storage copied to your Amazon S3 bucket. In this module, you will configure the Application server in the in-cloud region to connect to your S3 bucket and provide access to the objects in the bucket. 16 | 17 | ![](../images/fullarch.png) 18 | 19 | ## Module Steps 20 | 21 | #### 1. Verifying access to the Amazon S3 bucket 22 | 23 | In AWS access to resources is administered by the Identity and Management service (IAM). By default, all S3 resources are private: only the resource owner can access them. S3 offers resource-based and user policies or a combination of them to manage permissions to your Amazon S3 resources. We will grant the required S3 access to our Application server in this section. 24 | 25 | 1. Go to the AWS Management console page in the **in-cloud** region and click **Services** then select **EC2.** 26 | 27 | 2. If no gateways exist, click the **Get started** button, otherwise click the **Create gateway** button. 28 | 3. Select the **Instances** under the **Resources** section. 29 | 4. Select the **ApplicationServer** instnace from the list. 30 | 5. Connect to the Application server via SSH or with the Console, click on **Console** on the top right. Then click **Connect** to go to the CLI. If connecting with SSH use the key-pair provided to cloud formation and the Application servers public IP address. 31 | 6. From CLI for the Application server run the following command: 32 | 33 | $ aws s3 ls 34 | 35 | ![](../images/mod3s1.png) 36 | 37 | 7. This will fail and return: 38 | 39 | ```An error occurred (AccessDenied) when calling the ListBuckets operation: Access Denied``` 40 | 41 | 8. Go back to the **EC2** service and select the Application server. Under the details section go to **IAM Role** and click on the role "_stack-name_-s3BucketIamRole-..." 42 | 9. Under the **Permissions** tab, expand the _Policy_ and click on _{}JSON_ to view the policy document: 43 | 44 | ```{ 45 | "Version": "2012-10-17", 46 | "Statement": [ 47 | { 48 | "Action": [ 49 | "s3:GetBucketLocation", 50 | "s3:ListBucket", 51 | "s3:ListBucketMultipartUploads", 52 | "s3:HeadBucket" 53 | ], 54 | "Resource": "arn:aws:s3:::data-migration-workshop-e5819550-1dd0-11eb-9c20-065bc1dc5e74", 55 | "Effect": "Allow" 56 | }, 57 | { 58 | "Action": [ 59 | "s3:AbortMultipartUpload", 60 | "s3:DeleteObject", 61 | "s3:GetObject", 62 | "s3:ListMultipartUploadParts", 63 | "s3:PutObject" 64 | ], 65 | "Resource": "arn:aws:s3:::data-migration-workshop-e5819550-1dd0-11eb-9c20-065bc1dc5e74/*", 66 | "Effect": "Allow" 67 | } 68 | ] 69 | } 70 | ``` 71 | The policy limits access to only the "data-migration-workshop" bucket. This will give the application server just enough access to your S3 resources. 72 | 73 | #### 2. Accessing the data in the Amazon S3 bucket 74 | 75 | 1. Connect to the Application server via SSH or with the Console, click on **Console** on the top right. 76 | 2. From CLI for the Application server run the following command: 77 | 78 | $ aws s3 ls s3://data-migration-workshop-e5819550-1dd0-11eb-9c20-065bc1dc5e74/ 79 | $ aws s3 ls s3://data-migration-workshop-e5819550-1dd0-11eb-9c20-065bc1dc5e74/images/ 80 | 81 | ![](../images/mod3s2.png) 82 | 83 | ## Validation Step 84 | 85 | Run the following command to verify that all the objects were transferred 86 | 87 | $ aws s3 ls s3://data-migration-workshop-e5819550-1dd0-11eb-9c20-065bc1dc5e74/images/ | wc -l 88 | 89 | You should see the same number of object in the AWS S3 bucket. 90 | 91 | ## Module Summary 92 | 93 | In this module you successfully verified the permission granted to the Application server and accessed the data migration S3 bucket. 94 | 95 | Remember that our ultimate goal in this workshop is to shut off the on-premises Object storage and free up storage resources. In a production environment, this would typically involve a "cutover point", where there is momentary downtime to change over to the in-cloud Application server and the new storage, which in this workshop is the Amazon S3 bucket. However, there are usually new object being created while a migration occurs, or shortly after, requiring another incremental transfer copy before cutover. 96 | 97 | In the next module, you'll do one more incremental copy before the final cutover to the Amazon S3. 98 | 99 | Go to [Module 4](../module4/). 100 | -------------------------------------------------------------------------------- /workshops/scality-object-migration/EN/module4/README.md: -------------------------------------------------------------------------------- 1 | # **AWS DataSync** 2 | 3 | ### Object store migration using AWS DataSync from Scality to Amazon S3 4 | 5 | © 2019 Amazon Web Services, Inc. and its affiliates. All rights reserved. 6 | This sample code is made available under the MIT-0 license. See the LICENSE file. 7 | 8 | Errors or corrections? Contact [owolabip@amazon.com](mailto:owolabip@amazon.com). 9 | 10 | --- 11 | 12 | # Module 4 13 | ## One last incremental copy before cutover 14 | 15 | In this module, you will perform an incremental data transfer using DataSync. This will get any new objects that may have been created after the initial data copy. Once you have verified all objects from the on-premises object storage have been copied, you can proceed to cutover. 16 | 17 | ![](../images/fullarch.png) 18 | 19 | ## Module Steps 20 | 21 | #### 1. Create some new file on the on-premises bucket 22 | 23 | 1. In the on-premises region, from the CLI for the **S3Server**, run the following command to create some new object in the Scality bucket: 24 | 25 | $ aws configure set aws_access_key_id AccessKey --profile scality 26 | $ aws configure set aws_secret_access_key SecretKey --profile scality 27 | $ for i in {00201..00210}; do aws s3 cp /media/data/images/$i.jpg s3://data/images/ --endpoint-url http://:8000/ --profile scality; done 28 | 29 | #### 2. Copy the new file to the S3 bucket 30 | 31 | You have already created a DataSync task to copy files from the object store to the S3 bucket. To copy the new file, you will just re-run the task. DataSync will only copy files that have changed between the source and the destination. 32 | 33 | 1. Return to the in-cloud region AWS management console and go to the **DataSync** service. 34 | 2. Select the task created previously and click the **Start** button. 35 | 3. Use the default settings and then click **Start**. 36 | 4. Go to the History tab and select the newest task execution from the list. 37 | 38 | ![](../images/mod4ds1.png) 39 | 40 | It will take a few minutes for the task to complete. When the task completes, take a look at the stats. Although you ran the exact same task as last time, only 10 objects were copied. 41 | 42 | ![](../images/mod4ds2.png) 43 | 44 | If you take a look at the S3 bucket, you see that the new objects is there, just as expected: 45 | 46 | ![](../images/mod4ds3.png) 47 | 48 | ## Module Summary 49 | 50 | In this module you added some new objects to the object storage bucket prior to cutover. You then ran the DataSync task a second time to pick up any object changes and copy them to S3. Finally, you checked the S3 bucket to confirm that the new files were transfered. 51 | 52 | With all of the data copied from the Scality object storage to S3, you are now ready to perform the cutover. 53 | 54 | Go to [Module 5](../module5/). 55 | -------------------------------------------------------------------------------- /workshops/scality-object-migration/EN/module5/README.md: -------------------------------------------------------------------------------- 1 | # **AWS DataSync** 2 | 3 | ### Object store migration using AWS DataSync from Scality to Amazon S3 4 | 5 | © 2019 Amazon Web Services, Inc. and its affiliates. All rights reserved. 6 | This sample code is made available under the MIT-0 license. See the LICENSE file. 7 | 8 | Errors or corrections? Contact [owolabip@amazon.com](mailto:owolabip@amazon.com). 9 | 10 | --- 11 | 12 | # Module 5 13 | ## Cutover to Amazon S3 and shutdown the Scality Object Storage 14 | 15 | With all of the data in the S3 bucket, you are now ready to shut down your object storage and move exclusively to using S3. In this module you will simply clean up your DataSync resources. You will then write some test files from the in-cloud Application server to the S3 bucket. 16 | 17 | ![](../images/mod5arch.png) 18 | 19 | ## Module Steps 20 | 21 | #### 1. Clean up DataSync resources 22 | 23 | You're done with DataSync so you can go ahead and clean up resources. 24 | 25 | 1. Go to the **in-cloud** region AWS management console and go to the **DataSync** service. 26 | 27 | 2. Select **Tasks** and delete the task you created previously 28 | 3. Select **Locations** and delete the locations you created previously 29 | 4. Select **Agents** and delete the agent you activated previously. Note that this **will not** delete the actual DataSync agent EC2 instance. That will get deleted later, when the CloudFormation stack is deleted. 30 | 31 | ## Validation Step 32 | 33 | From the CLI for the Application server, run the following command to create another new file in the S3 bucket: 34 | 35 | $ echo "test data" >> new.txt 36 | $ aws s3 cp new.txt s3:// 37 | 38 | Go back to the in-cloud region management console and go to **S3**. Select the **data-migration-workshop** bucket. You should see the new.txt file in the bucket. 39 | 40 | ![](../images/mod5s31.png) 41 | 42 | Your Application server has completed cutover! You can now read all of the objects that used to be on the on-premises object storage. 43 | 44 | ## Workshop Cleanup 45 | 46 | To make sure all resources are deleted after this workshop scenario make sure you execute the steps in the order outlined below (you do not need to wait for CloudFormation to finish deleting before moving to the next step): 47 | 48 | 1. Delete all objects in the **data-migration-workshop** S3 bucket in the in-cloud region. The bucket must be empty before it can be deleted by CloudFormation in the next step. 49 | 6. Go to the CloudFormation page in the in-cloud region and delete the stack named "DataMigrationWorkshop-inCloudResources" 50 | 7. Go to the CloudFormation page in the on-premises region and delete the stack named "DataMigrationWorkshop-onPremResources" 51 | 52 | To make sure that all CloudFormation templates have been deleted correctly, confirm that all EC2 instances created in this workshop in the on-premises region are in the **terminated** state. 53 | -------------------------------------------------------------------------------- /workshops/scality-object-migration/EN/templates/data-migration-workshop-in-cloud.yaml: -------------------------------------------------------------------------------- 1 | AWSTemplateFormatVersion: '2010-09-09' 2 | Description: AWS DataSync Workshop - Scality Object Migration - In-Cloud Region 3 | Metadata: 4 | License: 5 | Description: | 6 | Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. 7 | 8 | Permission is hereby granted, free of charge, to any person obtaining a copy of this 9 | software and associated documentation files (the "Software"), to deal in the Software 10 | without restriction, including without limitation the rights to use, copy, modify, 11 | merge, publish, distribute, sublicense, and/or sell copies of the Software, and to 12 | permit persons to whom the Software is furnished to do so. 13 | 14 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, 15 | INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A 16 | PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT 17 | HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION 18 | OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE 19 | SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 20 | 21 | 'AWS::CloudFormation::Interface': 22 | ParameterGroups: 23 | - Parameters: 24 | - kp 25 | - Label: 26 | default: AMI IDs (do not edit) 27 | Parameters: 28 | - linuxAmi 29 | ParameterLabels: 30 | kp: 31 | default: 'EC2 Key Pair Name (optional):' 32 | linuxAmi: 33 | default: Linux 34 | Parameters: 35 | kp: 36 | Description: | 37 | A key pair is optional. If not specified, you can access instances 38 | through the EC2 console using Session Manager or EC2 Instance Connect. 39 | #Type: AWS::EC2::KeyPair::KeyName 40 | Type: String 41 | linuxAmi: 42 | Type: 'AWS::SSM::Parameter::Value' 43 | Default: /aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2 44 | AllowedValues: 45 | - /aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2 46 | Conditions: 47 | UseEc2Keypair: !Not 48 | - !Equals 49 | - '' 50 | - !Ref kp 51 | 52 | Resources: 53 | # We use the GUID from the ARN of the stack ID to generate 54 | # a unique bucket name 55 | s3Bucket: 56 | Type: AWS::S3::Bucket 57 | Properties: 58 | PublicAccessBlockConfiguration: 59 | BlockPublicAcls: True 60 | BlockPublicPolicy: True 61 | IgnorePublicAcls: True 62 | RestrictPublicBuckets: True 63 | BucketName: !Join 64 | - "-" 65 | - - "data-migration-workshop" 66 | - !Select 67 | - 2 68 | - !Split 69 | - "/" 70 | - !Ref "AWS::StackId" 71 | 72 | # Give the role a friendly name as the workshop user will need to 73 | # reference it when creating DataSync tasks. 74 | s3BucketIamRole: 75 | Type: AWS::IAM::Role 76 | Properties: 77 | AssumeRolePolicyDocument: 78 | Statement: 79 | - Action: 80 | - sts:AssumeRole 81 | Effect: Allow 82 | Principal: 83 | Service: 84 | - datasync.amazonaws.com 85 | - ec2.amazonaws.com 86 | Version: '2012-10-17' 87 | s3BucketRolePolicy: 88 | Type: AWS::IAM::Policy 89 | DependsOn: s3Bucket 90 | Properties: 91 | PolicyDocument: 92 | Statement: 93 | - Effect: Allow 94 | Resource: !GetAtt s3Bucket.Arn 95 | Action: 96 | - s3:GetBucketLocation 97 | - s3:ListBucket 98 | - s3:ListBucketMultipartUploads 99 | - s3:HeadBucket 100 | - Effect: Allow 101 | Resource: !Join [ "/", [ !GetAtt s3Bucket.Arn, "*" ] ] 102 | Action: 103 | - s3:AbortMultipartUpload 104 | - s3:DeleteObject 105 | - s3:GetObject 106 | - s3:ListMultipartUploadParts 107 | - s3:PutObject 108 | - s3:PutObjectTagging 109 | - s3:GetObjectTagging 110 | Version: '2012-10-17' 111 | PolicyName: policy 112 | Roles: 113 | - !Ref 's3BucketIamRole' 114 | 115 | dmVPC: 116 | Type: 'AWS::EC2::VPC' 117 | Properties: 118 | CidrBlock: 10.11.12.0/24 119 | EnableDnsSupport: 'true' 120 | EnableDnsHostnames: 'true' 121 | InstanceTenancy: default 122 | Tags: 123 | - Key: Name 124 | Value: DataMigrationWorkshopVPC 125 | dmSubnet1: 126 | Type: 'AWS::EC2::Subnet' 127 | Properties: 128 | VpcId: !Ref dmVPC 129 | CidrBlock: 10.11.12.0/24 130 | MapPublicIpOnLaunch: 'True' 131 | Tags: 132 | - Key: Name 133 | Value: DataMigrationWorkshopSubnet1 134 | dmInternetGateway: 135 | Type: 'AWS::EC2::InternetGateway' 136 | Properties: 137 | Tags: 138 | - Key: Name 139 | Value: DataMigrationWorkshopIGW 140 | dmAttachGateway: 141 | Type: 'AWS::EC2::VPCGatewayAttachment' 142 | Properties: 143 | VpcId: !Ref dmVPC 144 | InternetGatewayId: !Ref dmInternetGateway 145 | dmRouteTable: 146 | Type: 'AWS::EC2::RouteTable' 147 | Properties: 148 | VpcId: !Ref dmVPC 149 | Tags: 150 | - Key: Name 151 | Value: DataMigrationWorkshopRouteTable 152 | dmSubnet1RouteAssociaton: 153 | Type: 'AWS::EC2::SubnetRouteTableAssociation' 154 | Properties: 155 | SubnetId: !Ref dmSubnet1 156 | RouteTableId: !Ref dmRouteTable 157 | dmRoutetoInternet: 158 | Type: 'AWS::EC2::Route' 159 | DependsOn: dmInternetGateway 160 | Properties: 161 | RouteTableId: !Ref dmRouteTable 162 | DestinationCidrBlock: 0.0.0.0/0 163 | GatewayId: !Ref dmInternetGateway 164 | dmSecurityGroup: 165 | Type: 'AWS::EC2::SecurityGroup' 166 | Properties: 167 | GroupDescription: Data Migration Workshop - Security Group for all resources 168 | VpcId: !Ref dmVPC 169 | SecurityGroupIngress: 170 | - IpProtocol: tcp 171 | FromPort: '22' 172 | ToPort: '22' 173 | CidrIp: 0.0.0.0/0 174 | 175 | appServerInstanceProfile: 176 | Type: 'AWS::IAM::InstanceProfile' 177 | Properties: 178 | Path: / 179 | Roles: 180 | - !Ref s3BucketIamRole 181 | 182 | appServer: 183 | Type: 'AWS::EC2::Instance' 184 | Properties: 185 | ImageId: !Ref linuxAmi 186 | InstanceType: t2.micro 187 | IamInstanceProfile: !Ref appServerInstanceProfile 188 | Tags: 189 | - Key: Name 190 | Value: !Join 191 | - '' 192 | - - ApplicationServer ( 193 | - !Ref 'AWS::StackName' 194 | - ) 195 | KeyName: !If 196 | - UseEc2Keypair 197 | - !Ref kp 198 | - !Ref 'AWS::NoValue' 199 | InstanceInitiatedShutdownBehavior: terminate 200 | BlockDeviceMappings: 201 | - DeviceName: /dev/xvda 202 | Ebs: 203 | VolumeSize: '8' 204 | DeleteOnTermination: 'true' 205 | VolumeType: gp2 206 | NetworkInterfaces: 207 | - AssociatePublicIpAddress: 'true' 208 | DeviceIndex: '0' 209 | GroupSet: 210 | - !Ref dmSecurityGroup 211 | SubnetId: !Ref dmSubnet1 212 | UserData: !Base64 213 | Fn::Join: 214 | - '' 215 | - - "#!/bin/bash\n" 216 | - "sudo mkdir /mnt/data\n" 217 | dmLogGroup: 218 | Type: AWS::Logs::LogGroup 219 | Properties: 220 | LogGroupName: dmLogGroup 221 | 222 | Outputs: 223 | bucketName: 224 | Description: S3 Bucket Name 225 | Value: !Ref 's3Bucket' 226 | bucketRoleForDataSync: 227 | Description: S3 Bucket Role for DataSync 228 | Value: !Ref 's3BucketIamRole' 229 | appServerPrivateIP: 230 | Description: Application Server Private IP Address 231 | Value: !GetAtt 232 | - appServer 233 | - PrivateIp 234 | dataSyncLogGroupName: 235 | Description: Log group for DataSync tasks 236 | Value: !Ref dmLogGroup 237 | -------------------------------------------------------------------------------- /workshops/scality-object-migration/EN/templates/data-migration-workshop-on-prem.yaml: -------------------------------------------------------------------------------- 1 | AWSTemplateFormatVersion: 2010-09-09 2 | Description: AWS DataSync Workshop - Scality Object Migration - On-Premises Region 3 | Metadata: 4 | License: 5 | Description: | 6 | Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. 7 | 8 | Permission is hereby granted, free of charge, to any person obtaining a 9 | copy of this software and associated documentation files (the "Software"), to deal in 10 | the Software without restriction, including without limitation the rights to use, copy, 11 | modify, merge, publish, distribute, sublicense, and/or sell copies of the 12 | Software, and to permit persons to whom the Software is furnished to do so. 13 | 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS 17 | FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 18 | COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN 19 | ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH 20 | THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 21 | 22 | 'AWS::CloudFormation::Interface': 23 | ParameterGroups: 24 | - Parameters: 25 | - kp 26 | - Label: 27 | default: AMI IDs (do not edit) 28 | Parameters: 29 | - linuxAmi 30 | - dsAmi 31 | ParameterLabels: 32 | kp: 33 | default: 'EC2 Key Pair Name (optional):' 34 | linuxAmi: 35 | default: Linux 36 | dsAmi: 37 | default: DataSync 38 | Parameters: 39 | kp: 40 | Description: | 41 | A key pair is optional. If not specified, you can access instances 42 | through the EC2 console using Session Manager or EC2 Instance Connect. 43 | #Type: AWS::EC2::KeyPair::KeyName 44 | Type: String 45 | linuxAmi: 46 | Type: 'AWS::SSM::Parameter::Value' 47 | Default: /aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2 48 | AllowedValues: 49 | - /aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2 50 | dsAmi: 51 | Type: 'AWS::SSM::Parameter::Value' 52 | Default: /aws/service/datasync/ami 53 | AllowedValues: 54 | - /aws/service/datasync/ami 55 | Conditions: 56 | UseEc2Keypair: !Not 57 | - !Equals 58 | - '' 59 | - !Ref kp 60 | Resources: 61 | dmVPC: 62 | Type: 'AWS::EC2::VPC' 63 | Properties: 64 | CidrBlock: 10.11.12.0/24 65 | EnableDnsSupport: 'true' 66 | EnableDnsHostnames: 'true' 67 | InstanceTenancy: default 68 | Tags: 69 | - Key: Name 70 | Value: DataMigrationWorkshopVPC 71 | dmSubnet1: 72 | Type: 'AWS::EC2::Subnet' 73 | Properties: 74 | VpcId: !Ref dmVPC 75 | CidrBlock: 10.11.12.0/24 76 | MapPublicIpOnLaunch: 'True' 77 | Tags: 78 | - Key: Name 79 | Value: DataMigrationWorkshopSubnet1 80 | dmInternetGateway: 81 | Type: 'AWS::EC2::InternetGateway' 82 | Properties: 83 | Tags: 84 | - Key: Name 85 | Value: DataMigrationWorkshopIGW 86 | dmAttachGateway: 87 | Type: 'AWS::EC2::VPCGatewayAttachment' 88 | Properties: 89 | VpcId: !Ref dmVPC 90 | InternetGatewayId: !Ref dmInternetGateway 91 | dmRouteTable: 92 | Type: 'AWS::EC2::RouteTable' 93 | Properties: 94 | VpcId: !Ref dmVPC 95 | Tags: 96 | - Key: Name 97 | Value: DataMigrationWorkshopRouteTable 98 | dmSubnet1RouteAssociaton: 99 | Type: 'AWS::EC2::SubnetRouteTableAssociation' 100 | Properties: 101 | SubnetId: !Ref dmSubnet1 102 | RouteTableId: !Ref dmRouteTable 103 | dmRoutetoInternet: 104 | Type: 'AWS::EC2::Route' 105 | DependsOn: dmInternetGateway 106 | Properties: 107 | RouteTableId: !Ref dmRouteTable 108 | DestinationCidrBlock: 0.0.0.0/0 109 | GatewayId: !Ref dmInternetGateway 110 | dmSecurityGroup: 111 | Type: 'AWS::EC2::SecurityGroup' 112 | Properties: 113 | GroupDescription: Data Migration Workshop - Security Group for all resources 114 | VpcId: !Ref dmVPC 115 | SecurityGroupIngress: 116 | - IpProtocol: tcp 117 | FromPort: '22' 118 | ToPort: '22' 119 | CidrIp: 0.0.0.0/0 120 | - IpProtocol: tcp 121 | FromPort: '80' 122 | ToPort: '80' 123 | CidrIp: 0.0.0.0/0 124 | - IpProtocol: tcp 125 | FromPort: '8000' 126 | ToPort: '8000' 127 | CidrIp: 0.0.0.0/0 128 | dmSecurityGroupIngress: 129 | Type: 'AWS::EC2::SecurityGroupIngress' 130 | DependsOn: dmSecurityGroup 131 | Properties: 132 | GroupId: !Ref dmSecurityGroup 133 | IpProtocol: tcp 134 | ToPort: '2049' 135 | FromPort: '2049' 136 | SourceSecurityGroupId: !Ref dmSecurityGroup 137 | S3ServerInstanceProfile: 138 | Type: 'AWS::IAM::InstanceProfile' 139 | Properties: 140 | Path: / 141 | Roles: 142 | - !Ref S3ServerIamRole 143 | S3ServerIamRole: 144 | Type: 'AWS::IAM::Role' 145 | Properties: 146 | AssumeRolePolicyDocument: 147 | Statement: 148 | - Action: 149 | - 'sts:AssumeRole' 150 | Effect: Allow 151 | Principal: 152 | Service: 153 | - ec2.amazonaws.com 154 | Version: 2012-10-17 155 | ManagedPolicyArns: 156 | - 'arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore' 157 | S3ServerRolePolicy: 158 | Type: 'AWS::IAM::Policy' 159 | Properties: 160 | PolicyDocument: 161 | Statement: 162 | - Effect: Allow 163 | Action: 164 | - 's3:ListBucket' 165 | Resource: 166 | - 'arn:aws:s3:::aft-vbi-pds' 167 | - Effect: Allow 168 | Action: 169 | - 's3:GetObject' 170 | Resource: 171 | - 'arn:aws:s3:::aft-vbi-pds/*' 172 | Version: 2012-10-17 173 | PolicyName: policy 174 | Roles: 175 | - !Ref S3ServerIamRole 176 | S3Server: 177 | Type: 'AWS::EC2::Instance' 178 | CreationPolicy: 179 | ResourceSignal: 180 | Timeout: PT30M 181 | Count: 1 182 | Metadata: 183 | 'AWS::CloudFormation::Init': 184 | config: 185 | files: 186 | /opt/aws/cfn/initdata.sh: 187 | content: !Sub | 188 | #!/bin/bash 189 | yum update -y 190 | amazon-linux-extras install docker 191 | service docker start 192 | usermod -a -G docker ec2-user 193 | docker run -d --name cloudserver -p 8000:8000 -e SCALITY_ACCESS_KEY_ID=AccessKey -e SCALITY_SECRET_ACCESS_KEY=SecretKey -e REMOTE_MANAGEMENT_DISABLE=1 zenko/cloudserver 194 | mkdir /home/ec2-user/.aws 195 | aws configure set aws_access_key_id AccessKey --profile scality 196 | aws configure set aws_secret_access_key SecretKey --profile scality 197 | aws --profile scality s3 mb s3://data --endpoint-url http://127.0.0.1:8000/ 198 | mkdir -p /media/data/images 199 | for i in {00001..00210}; do 200 | aws s3 cp s3://aft-vbi-pds/bin-images/$i.jpg /media/data/images/ >> /tmp/sync 201 | done 202 | for i in {00001..00200}; do 203 | aws --profile scality s3 cp /media/data/images/$i.jpg s3://data/images/ --endpoint-url http://127.0.0.1:8000/ >> /tmp/sync 204 | done 205 | chown -R ec2-user:ec2-user /home/ec2-user/.aws 206 | commands: 207 | 1-initData: 208 | command: "bash /opt/aws/cfn/initdata.sh" 209 | 2-finishUp: 210 | command: !Sub "/opt/aws/bin/cfn-signal -e 0 --stack ${AWS::StackId} --resource S3Server --region ${AWS::Region}" 211 | Properties: 212 | ImageId: !Ref linuxAmi 213 | InstanceType: m5.xlarge 214 | IamInstanceProfile: !Ref S3ServerInstanceProfile 215 | Tags: 216 | - Key: Name 217 | Value: !Join 218 | - '' 219 | - - S3Server ( 220 | - !Ref 'AWS::StackName' 221 | - ) 222 | KeyName: !If 223 | - UseEc2Keypair 224 | - !Ref kp 225 | - !Ref 'AWS::NoValue' 226 | InstanceInitiatedShutdownBehavior: terminate 227 | BlockDeviceMappings: 228 | - DeviceName: /dev/xvda 229 | Ebs: 230 | VolumeSize: '8' 231 | DeleteOnTermination: 'true' 232 | VolumeType: gp2 233 | NetworkInterfaces: 234 | - AssociatePublicIpAddress: 'true' 235 | DeviceIndex: '0' 236 | GroupSet: 237 | - !Ref dmSecurityGroup 238 | SubnetId: !Ref dmSubnet1 239 | UserData: 240 | Fn::Base64: 241 | !Sub | 242 | #!/bin/bash 243 | /opt/aws/bin/cfn-init -v -s ${AWS::StackId} -r S3Server --region ${AWS::Region} 244 | 245 | appServer: 246 | Type: 'AWS::EC2::Instance' 247 | Properties: 248 | ImageId: !Ref linuxAmi 249 | InstanceType: t2.micro 250 | IamInstanceProfile: !Ref S3ServerInstanceProfile 251 | Tags: 252 | - Key: Name 253 | Value: !Join 254 | - '' 255 | - - ApplicationServer ( 256 | - !Ref 'AWS::StackName' 257 | - ) 258 | KeyName: !If 259 | - UseEc2Keypair 260 | - !Ref kp 261 | - !Ref 'AWS::NoValue' 262 | InstanceInitiatedShutdownBehavior: terminate 263 | BlockDeviceMappings: 264 | - DeviceName: /dev/xvda 265 | Ebs: 266 | VolumeSize: '8' 267 | DeleteOnTermination: 'true' 268 | VolumeType: gp2 269 | NetworkInterfaces: 270 | - AssociatePublicIpAddress: 'true' 271 | DeviceIndex: '0' 272 | GroupSet: 273 | - !Ref dmSecurityGroup 274 | SubnetId: !Ref dmSubnet1 275 | UserData: !Base64 276 | Fn::Join: 277 | - '' 278 | - - "#!/bin/bash\n" 279 | - "sudo mkdir /mnt/data\n" 280 | 281 | 282 | dataSyncAgentInstanceProfile: 283 | Type: 'AWS::IAM::InstanceProfile' 284 | Properties: 285 | Path: / 286 | Roles: 287 | - !Ref dataSyncAgentIamRole 288 | dataSyncAgentIamRole: 289 | Type: 'AWS::IAM::Role' 290 | Properties: 291 | AssumeRolePolicyDocument: 292 | Statement: 293 | - Action: 294 | - 'sts:AssumeRole' 295 | Effect: Allow 296 | Principal: 297 | Service: 298 | - ec2.amazonaws.com 299 | Version: 2012-10-17 300 | dataSyncAgentRolePolicy: 301 | Type: 'AWS::IAM::Policy' 302 | Properties: 303 | PolicyDocument: 304 | Statement: 305 | - Effect: Allow 306 | Action: 307 | - 'datasync:*' 308 | Resource: 309 | - '*' 310 | - Effect: Allow 311 | Action: 312 | - 'iam:PassRole' 313 | Resource: 314 | - '*' 315 | Version: 2012-10-17 316 | PolicyName: policy 317 | Roles: 318 | - !Ref dataSyncAgentIamRole 319 | dataSyncAgent: 320 | Type: 'AWS::EC2::Instance' 321 | Properties: 322 | ImageId: !Ref dsAmi 323 | InstanceType: m5.2xlarge 324 | IamInstanceProfile: !Ref dataSyncAgentInstanceProfile 325 | Tags: 326 | - Key: Name 327 | Value: !Join 328 | - '' 329 | - - DataSyncAgent ( 330 | - !Ref 'AWS::StackName' 331 | - ) 332 | KeyName: !If 333 | - UseEc2Keypair 334 | - !Ref kp 335 | - !Ref 'AWS::NoValue' 336 | InstanceInitiatedShutdownBehavior: stop 337 | BlockDeviceMappings: 338 | - DeviceName: /dev/xvda 339 | Ebs: 340 | VolumeSize: '80' 341 | DeleteOnTermination: 'true' 342 | VolumeType: gp2 343 | NetworkInterfaces: 344 | - AssociatePublicIpAddress: 'true' 345 | DeviceIndex: '0' 346 | GroupSet: 347 | - !Ref dmSecurityGroup 348 | SubnetId: !Ref dmSubnet1 349 | 350 | Outputs: 351 | S3ServerPrivateIP: 352 | Description: S3 Server Private IP Address 353 | Value: !GetAtt 354 | - S3Server 355 | - PrivateIp 356 | appServerPrivateIP: 357 | Description: Application Server Private IP Address 358 | Value: !GetAtt 359 | - appServer 360 | - PrivateIp 361 | dataSyncAgentPublicIP: 362 | Description: DataSync Agent Public IP Address 363 | Value: !GetAtt 364 | - dataSyncAgent 365 | - PublicIp 366 | -------------------------------------------------------------------------------- /workshops/scality-object-migration/README.md: -------------------------------------------------------------------------------- 1 | # **AWS DataSync** 2 | 3 | ### Object storage migration using AWS DataSync from Scality to Amazon S3 4 | 5 | © 2019 Amazon Web Services, Inc. and its affiliates. All rights reserved. 6 | This sample code is made available under the MIT-0 license. See the LICENSE file. 7 | 8 | Errors or corrections? Contact [owolabip@amazon.com](mailto:owolabip@amazon.com). 9 | 10 | --- 11 | 12 | ## Supported Languages 13 | 14 | This workshop is available in the following languages. Click a link below to begin the workshop. 15 | 16 | - [English](EN/) 17 | --------------------------------------------------------------------------------