├── .github └── PULL_REQUEST_TEMPLATE.md ├── .gitignore ├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── LICENSE ├── NOTICE ├── README-Abaqus.rst ├── README-OpenFOAM.rst ├── README-WRF.rst ├── README.md ├── _files ├── abaqus │ └── abaqus_results.jpg └── wrf │ └── namelist.input └── _images ├── OpenFOAM_output.png └── ncview_wrf.png /.github/PULL_REQUEST_TEMPLATE.md: -------------------------------------------------------------------------------- 1 | *Issue #, if available:* 2 | 3 | *Description of changes:* 4 | 5 | 6 | By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. 7 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | *$py.class 5 | 6 | # C extensions 7 | *.so 8 | 9 | # Distribution / packaging 10 | .Python 11 | env/ 12 | build/ 13 | develop-eggs/ 14 | dist/ 15 | downloads/ 16 | eggs/ 17 | .eggs/ 18 | lib/ 19 | lib64/ 20 | parts/ 21 | sdist/ 22 | var/ 23 | wheels/ 24 | *.egg-info/ 25 | .installed.cfg 26 | *.egg 27 | 28 | # PyInstaller 29 | # Usually these files are written by a python script from a template 30 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 31 | *.manifest 32 | *.spec 33 | 34 | # Installer logs 35 | pip-log.txt 36 | pip-delete-this-directory.txt 37 | 38 | # Unit test / coverage reports 39 | htmlcov/ 40 | .tox/ 41 | .coverage 42 | .coverage.* 43 | .cache 44 | nosetests.xml 45 | coverage.xml 46 | *.cover 47 | .hypothesis/ 48 | 49 | # Translations 50 | *.mo 51 | *.pot 52 | 53 | # Django stuff: 54 | *.log 55 | local_settings.py 56 | 57 | # Flask stuff: 58 | instance/ 59 | .webassets-cache 60 | 61 | # Scrapy stuff: 62 | .scrapy 63 | 64 | # Sphinx documentation 65 | docs/_build/ 66 | 67 | # PyBuilder 68 | target/ 69 | 70 | # Jupyter Notebook 71 | .ipynb_checkpoints 72 | 73 | # pyenv 74 | .python-version 75 | 76 | # celery beat schedule file 77 | celerybeat-schedule 78 | 79 | # SageMath parsed files 80 | *.sage.py 81 | 82 | # dotenv 83 | .env 84 | 85 | # virtualenv 86 | .venv 87 | venv/ 88 | ENV/ 89 | 90 | # Spyder project settings 91 | .spyderproject 92 | .spyproject 93 | 94 | # Rope project settings 95 | .ropeproject 96 | 97 | # mkdocs documentation 98 | /site 99 | 100 | # mypy 101 | .mypy_cache/ 102 | 103 | # IDE and Editor artifacts # 104 | *.bbprojectd 105 | .idea 106 | *.iml 107 | 108 | # Temporary Files # 109 | tmp_* 110 | cfg.tmp.json 111 | 112 | # OS generated files # 113 | .DS_Store 114 | .DS_Store? 115 | 116 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | ## Code of Conduct 2 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 3 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 4 | opensource-codeofconduct@amazon.com with any additional questions or comments. 5 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing Guidelines 2 | 3 | Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional 4 | documentation, we greatly value feedback and contributions from our community. 5 | 6 | Please read through this document before submitting any issues or pull requests to ensure we have all the necessary 7 | information to effectively respond to your bug report or contribution. 8 | 9 | 10 | ## Reporting Bugs/Feature Requests 11 | 12 | We welcome you to use the GitHub issue tracker to report bugs or suggest features. 13 | 14 | When filing an issue, please check [existing open](https://github.com/aws-samples/aws-hpc-workshops/issues), or [recently closed](https://github.com/aws-samples/aws-hpc-workshops/issues?utf8=%E2%9C%93&q=is%3Aissue%20is%3Aclosed%20), issues to make sure somebody else hasn't already 15 | reported the issue. Please try to include as much information as you can. Details like these are incredibly useful: 16 | 17 | * A reproducible test case or series of steps 18 | * The version of our code being used 19 | * Any modifications you've made relevant to the bug 20 | * Anything unusual about your environment or deployment 21 | 22 | 23 | ## Contributing via Pull Requests 24 | Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that: 25 | 26 | 1. You are working against the latest source on the *master* branch. 27 | 2. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already. 28 | 3. You open an issue to discuss any significant work - we would hate for your time to be wasted. 29 | 30 | To send us a pull request, please: 31 | 32 | 1. Fork the repository. 33 | 2. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change. 34 | 3. Ensure local tests pass. 35 | 4. Commit to your fork using clear commit messages. 36 | 5. Send us a pull request, answering any default questions in the pull request interface. 37 | 6. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation. 38 | 39 | GitHub provides additional document on [forking a repository](https://help.github.com/articles/fork-a-repo/) and 40 | [creating a pull request](https://help.github.com/articles/creating-a-pull-request/). 41 | 42 | 43 | ## Finding contributions to work on 44 | Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels ((enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any ['help wanted'](https://github.com/aws-samples/aws-hpc-workshops/labels/help%20wanted) issues is a great place to start. 45 | 46 | 47 | ## Code of Conduct 48 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 49 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 50 | opensource-codeofconduct@amazon.com with any additional questions or comments. 51 | 52 | 53 | ## Security issue notifications 54 | If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github issue. 55 | 56 | 57 | ## Licensing 58 | 59 | See the [LICENSE](https://github.com/aws-samples/aws-hpc-workshops/blob/master/LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution. 60 | 61 | We may ask you to sign a [Contributor License Agreement (CLA)](http://en.wikipedia.org/wiki/Contributor_License_Agreement) for larger changes. 62 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | 2 | Apache License 3 | Version 2.0, January 2004 4 | http://www.apache.org/licenses/ 5 | 6 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 7 | 8 | 1. Definitions. 9 | 10 | "License" shall mean the terms and conditions for use, reproduction, 11 | and distribution as defined by Sections 1 through 9 of this document. 12 | 13 | "Licensor" shall mean the copyright owner or entity authorized by 14 | the copyright owner that is granting the License. 15 | 16 | "Legal Entity" shall mean the union of the acting entity and all 17 | other entities that control, are controlled by, or are under common 18 | control with that entity. For the purposes of this definition, 19 | "control" means (i) the power, direct or indirect, to cause the 20 | direction or management of such entity, whether by contract or 21 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 22 | outstanding shares, or (iii) beneficial ownership of such entity. 23 | 24 | "You" (or "Your") shall mean an individual or Legal Entity 25 | exercising permissions granted by this License. 26 | 27 | "Source" form shall mean the preferred form for making modifications, 28 | including but not limited to software source code, documentation 29 | source, and configuration files. 30 | 31 | "Object" form shall mean any form resulting from mechanical 32 | transformation or translation of a Source form, including but 33 | not limited to compiled object code, generated documentation, 34 | and conversions to other media types. 35 | 36 | "Work" shall mean the work of authorship, whether in Source or 37 | Object form, made available under the License, as indicated by a 38 | copyright notice that is included in or attached to the work 39 | (an example is provided in the Appendix below). 40 | 41 | "Derivative Works" shall mean any work, whether in Source or Object 42 | form, that is based on (or derived from) the Work and for which the 43 | editorial revisions, annotations, elaborations, or other modifications 44 | represent, as a whole, an original work of authorship. For the purposes 45 | of this License, Derivative Works shall not include works that remain 46 | separable from, or merely link (or bind by name) to the interfaces of, 47 | the Work and Derivative Works thereof. 48 | 49 | "Contribution" shall mean any work of authorship, including 50 | the original version of the Work and any modifications or additions 51 | to that Work or Derivative Works thereof, that is intentionally 52 | submitted to Licensor for inclusion in the Work by the copyright owner 53 | or by an individual or Legal Entity authorized to submit on behalf of 54 | the copyright owner. For the purposes of this definition, "submitted" 55 | means any form of electronic, verbal, or written communication sent 56 | to the Licensor or its representatives, including but not limited to 57 | communication on electronic mailing lists, source code control systems, 58 | and issue tracking systems that are managed by, or on behalf of, the 59 | Licensor for the purpose of discussing and improving the Work, but 60 | excluding communication that is conspicuously marked or otherwise 61 | designated in writing by the copyright owner as "Not a Contribution." 62 | 63 | "Contributor" shall mean Licensor and any individual or Legal Entity 64 | on behalf of whom a Contribution has been received by Licensor and 65 | subsequently incorporated within the Work. 66 | 67 | 2. Grant of Copyright License. Subject to the terms and conditions of 68 | this License, each Contributor hereby grants to You a perpetual, 69 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 70 | copyright license to reproduce, prepare Derivative Works of, 71 | publicly display, publicly perform, sublicense, and distribute the 72 | Work and such Derivative Works in Source or Object form. 73 | 74 | 3. Grant of Patent License. Subject to the terms and conditions of 75 | this License, each Contributor hereby grants to You a perpetual, 76 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 77 | (except as stated in this section) patent license to make, have made, 78 | use, offer to sell, sell, import, and otherwise transfer the Work, 79 | where such license applies only to those patent claims licensable 80 | by such Contributor that are necessarily infringed by their 81 | Contribution(s) alone or by combination of their Contribution(s) 82 | with the Work to which such Contribution(s) was submitted. If You 83 | institute patent litigation against any entity (including a 84 | cross-claim or counterclaim in a lawsuit) alleging that the Work 85 | or a Contribution incorporated within the Work constitutes direct 86 | or contributory patent infringement, then any patent licenses 87 | granted to You under this License for that Work shall terminate 88 | as of the date such litigation is filed. 89 | 90 | 4. Redistribution. You may reproduce and distribute copies of the 91 | Work or Derivative Works thereof in any medium, with or without 92 | modifications, and in Source or Object form, provided that You 93 | meet the following conditions: 94 | 95 | (a) You must give any other recipients of the Work or 96 | Derivative Works a copy of this License; and 97 | 98 | (b) You must cause any modified files to carry prominent notices 99 | stating that You changed the files; and 100 | 101 | (c) You must retain, in the Source form of any Derivative Works 102 | that You distribute, all copyright, patent, trademark, and 103 | attribution notices from the Source form of the Work, 104 | excluding those notices that do not pertain to any part of 105 | the Derivative Works; and 106 | 107 | (d) If the Work includes a "NOTICE" text file as part of its 108 | distribution, then any Derivative Works that You distribute must 109 | include a readable copy of the attribution notices contained 110 | within such NOTICE file, excluding those notices that do not 111 | pertain to any part of the Derivative Works, in at least one 112 | of the following places: within a NOTICE text file distributed 113 | as part of the Derivative Works; within the Source form or 114 | documentation, if provided along with the Derivative Works; or, 115 | within a display generated by the Derivative Works, if and 116 | wherever such third-party notices normally appear. The contents 117 | of the NOTICE file are for informational purposes only and 118 | do not modify the License. You may add Your own attribution 119 | notices within Derivative Works that You distribute, alongside 120 | or as an addendum to the NOTICE text from the Work, provided 121 | that such additional attribution notices cannot be construed 122 | as modifying the License. 123 | 124 | You may add Your own copyright statement to Your modifications and 125 | may provide additional or different license terms and conditions 126 | for use, reproduction, or distribution of Your modifications, or 127 | for any such Derivative Works as a whole, provided Your use, 128 | reproduction, and distribution of the Work otherwise complies with 129 | the conditions stated in this License. 130 | 131 | 5. Submission of Contributions. Unless You explicitly state otherwise, 132 | any Contribution intentionally submitted for inclusion in the Work 133 | by You to the Licensor shall be under the terms and conditions of 134 | this License, without any additional terms or conditions. 135 | Notwithstanding the above, nothing herein shall supersede or modify 136 | the terms of any separate license agreement you may have executed 137 | with Licensor regarding such Contributions. 138 | 139 | 6. Trademarks. This License does not grant permission to use the trade 140 | names, trademarks, service marks, or product names of the Licensor, 141 | except as required for reasonable and customary use in describing the 142 | origin of the Work and reproducing the content of the NOTICE file. 143 | 144 | 7. Disclaimer of Warranty. Unless required by applicable law or 145 | agreed to in writing, Licensor provides the Work (and each 146 | Contributor provides its Contributions) on an "AS IS" BASIS, 147 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 148 | implied, including, without limitation, any warranties or conditions 149 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 150 | PARTICULAR PURPOSE. You are solely responsible for determining the 151 | appropriateness of using or redistributing the Work and assume any 152 | risks associated with Your exercise of permissions under this License. 153 | 154 | 8. Limitation of Liability. In no event and under no legal theory, 155 | whether in tort (including negligence), contract, or otherwise, 156 | unless required by applicable law (such as deliberate and grossly 157 | negligent acts) or agreed to in writing, shall any Contributor be 158 | liable to You for damages, including any direct, indirect, special, 159 | incidental, or consequential damages of any character arising as a 160 | result of this License or out of the use or inability to use the 161 | Work (including but not limited to damages for loss of goodwill, 162 | work stoppage, computer failure or malfunction, or any and all 163 | other commercial damages or losses), even if such Contributor 164 | has been advised of the possibility of such damages. 165 | 166 | 9. Accepting Warranty or Additional Liability. While redistributing 167 | the Work or Derivative Works thereof, You may choose to offer, 168 | and charge a fee for, acceptance of support, warranty, indemnity, 169 | or other liability obligations and/or rights consistent with this 170 | License. However, in accepting such obligations, You may act only 171 | on Your own behalf and on Your sole responsibility, not on behalf 172 | of any other Contributor, and only if You agree to indemnify, 173 | defend, and hold each Contributor harmless for any liability 174 | incurred by, or claims asserted against, such Contributor by reason 175 | of your accepting any such warranty or additional liability. 176 | 177 | END OF TERMS AND CONDITIONS 178 | 179 | APPENDIX: How to apply the Apache License to your work. 180 | 181 | To apply the Apache License to your work, attach the following 182 | boilerplate notice, with the fields enclosed by brackets "[]" 183 | replaced with your own identifying information. (Don't include 184 | the brackets!) The text should be enclosed in the appropriate 185 | comment syntax for the file format. We also recommend that a 186 | file or class name and description of purpose be included on the 187 | same "printed page" as the copyright notice for easier 188 | identification within third-party archives. 189 | 190 | Copyright [yyyy] [name of copyright owner] 191 | 192 | Licensed under the Apache License, Version 2.0 (the "License"); 193 | you may not use this file except in compliance with the License. 194 | You may obtain a copy of the License at 195 | 196 | http://www.apache.org/licenses/LICENSE-2.0 197 | 198 | Unless required by applicable law or agreed to in writing, software 199 | distributed under the License is distributed on an "AS IS" BASIS, 200 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 201 | See the License for the specific language governing permissions and 202 | limitations under the License. 203 | -------------------------------------------------------------------------------- /NOTICE: -------------------------------------------------------------------------------- 1 | AWS HPC Workshops 2 | Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved. 3 | -------------------------------------------------------------------------------- /README-Abaqus.rst: -------------------------------------------------------------------------------- 1 | ############# 2 | Abaqus on AWS 3 | ############# 4 | 5 | .. contents:: 6 | :backlinks: none 7 | :depth: 2 8 | 9 | 10 | Information 11 | =========== 12 | Abaqus is an FEA application used in manufacturing. 13 | 14 | TL;DR 15 | ===== 16 | * Use an OS with the 3.10 kernel 17 | * Disable Hyperthreading 18 | * Use Platform MPI or Intel MPI 19 | * Set number of MPI threads to match core count 20 | * Enable pinning of MPI to cores 21 | * Set Clock Source to TSC 22 | 23 | Scaling Behavior 24 | ================ 25 | Abaqus has a few different running modes with different hardware requirements. 26 | 27 | I have a customer who's testing Abaqus on 72 cores using c4.8xlarge instances with the guidance below and is seing performance within ~15% of their on premise infiniband backed Haswell cluster 28 | 29 | Performance Guidance 30 | ==================== 31 | Abaqus seems to be especially sensitive to the Linux kernel version. For one customer, moving from a 2.6.32 Kernel to 3.10 saw the following performance improvements. 32 | 33 | UC5 & UC4 represent the customer's model names. Moving from a 2.6 kernel to 3.10 took the run time from 103 minutes to 78 minutes. 34 | 35 | .. image:: _files/abaqus/abaqus_results.jpg 36 | 37 | The customer was also able to set the number of MPI threads to match that of the number of cores. When doing this, they took the run time even further down to 65 minutes. 38 | 39 | Clock Source 40 | ------------ 41 | When monitoring the application, you can monitor the application to see calls it's making to the kernel 42 | 43 | .. code-block:: none 44 | 45 | strace -c -p 46 | 47 | Let that run for ~15 seconds and hit Ctrl+C. 48 | 49 | You'll likely see a number of calls related to gettimeofday. 50 | 51 | Try setting the clock source to TSC and rerunning the application. 52 | 53 | .. code-block:: none 54 | 55 | echo "tsc" > /sys/devices/system/clocksource/clocksource0/current_clocksource 56 | 57 | Or, set edit grub.conf and add the following to the kernel line and reboot. 58 | 59 | .. code-block:: none 60 | 61 | clocksource=tsc tsc=reliable 62 | 63 | Abaqus Support Docs 64 | =================== 65 | Here's a few tidbits from Abaqus support 66 | 67 | The Abaqus mp_process_binding parameter can be used to control the internal CPU binding option. 68 | Setting this parameter to ON will force to Abaqus to utilize the internal binding option, when the analysis is not running on all the cores. Careful attention must be paid to ensure that no other processes running on the same machine are also utilizing process binding. In particular multiple Abaqus analyses cannot be running on the same machine with this option enabled, or .severe performance degradation will result. Multiple Abaqus analyses running on the same machine will bind and compete for the exact same cores leaving other core free. 69 | 70 | mp_process_binding=ON 71 | 72 | https://kb.dsxclient.3ds.com/mashup-ui/page/document?q=docid:QA00000020963 73 | 74 | 75 | I have set the above in the custom_v6.env on AWS cluster. Please retest. 76 | If the performance issues persist, please append the following to your Abaqus 2016 ~/site/custom_v6.env environment file and run the s2a benchmark job across multiple nodes. Please zip and send any job files 77 | 78 | .. code-block:: none 79 | 80 | import os os.environ['ABA_RESOURCE_MONITOR']='yes' 81 | os.environ['ABA_RESOURCE_USEMALLINFO’]=”1” 82 | os.environ['ABA_GETMEMORYVALUE']='1' 83 | mp_mpirun_options="-v -d -T -prot" 84 | verbose=3 85 | 86 | Implicit vs Explicit 87 | ==================== 88 | Abaqus/Explicit: 89 | - CPU intensive (higher clock speed & more cores are better) 90 | - I/O Intensive 91 | - MPI limited 92 | - Scales well up to hundreds of cores (< 500 cores) 93 | 94 | Abaqus/Implicit: 95 | - Memory Intensive (more RAM the better) 96 | - Less I/O Intensive 97 | - MPI limited 98 | - Scales well up to 64-96 cores (< 100 cores) 99 | 100 | 101 | Abaqus Performance Data and Benchmarks 102 | ====================================== 103 | https://www.3ds.com/support/hardware-and-software/simulia-system-information/abaqus-614/performance-data/ 104 | -------------------------------------------------------------------------------- /README-OpenFOAM.rst: -------------------------------------------------------------------------------- 1 | ############### 2 | OpenFOAM on AWS 3 | ############### 4 | 5 | .. contents:: 6 | :backlinks: none 7 | :depth: 2 8 | 9 | **************************** 10 | CFD Direct AMI (Marketplace) 11 | **************************** 12 | 13 | On Marketplace, **CFD Direct** has AMI with OpenFOAM installed and setup 14 | 15 | Links 16 | http://cfd.direct/cloud 17 | 18 | https://aws.amazon.com/marketplace/pp/B017AHYO16/ 19 | 20 | 21 | 22 | **** 23 | Demo 24 | **** 25 | 26 | 27 | Create the demo cluster 28 | ======================= 29 | 30 | **Demo Config file** 31 | 32 | .. code-block:: none 33 | 34 | [aws] 35 | aws_region_name = us-west-2 36 | aws_access_key_id = 37 | aws_secret_access_key = 38 | 39 | [cluster default] 40 | vpc_settings = cluster-vpc 41 | key_name = 42 | scheduler = sge 43 | base_os = centos6 44 | maintain_initial_size = true 45 | master_instance_type = c4.8xlarge 46 | compute_instance_type = c4.8xlarge 47 | initial_queue_size = 2 48 | min_queue_size = 2 49 | max_queue_size = 5 50 | encrypted_ephemeral = true 51 | pre_install = https://s3-us-west-2.amazonaws.com/cfncluster-public-scripts/disable_hyperthreading_preinstall 52 | post_install = https://s3-us-west-2.amazonaws.com/cfncluster-public-scripts/openfoamtutorial_centos.sh 53 | ebs_settings = custom 54 | 55 | [ebs custom] 56 | ebs_snapshot_id = snap-d7efa994 57 | 58 | [vpc cluster-vpc] 59 | vpc_id = 60 | master_subnet_id = 61 | 62 | [global] 63 | update_check = true 64 | sanity_check = true 65 | cluster_template = default 66 | 67 | 68 | **Create the cluster** 69 | 70 | .. code-block:: none 71 | 72 | cfncluster -c config_openfoam_demo create demo1 73 | 74 | 75 | Setup environment 76 | ================= 77 | 78 | .. code-block:: none 79 | 80 | sudo chown -R centos:centos /shared/ 81 | 82 | .. code-block:: none 83 | 84 | export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/shared/OpenFOAM/mesa/lib 85 | 86 | 87 | Submit OpenFOAM job 88 | =================== 89 | 90 | Once the cluster has been created, login to the Master node and you should see two nodes available with qhost: 91 | 92 | .. code-block:: none 93 | 94 | $ qhost 95 | HOSTNAME ARCH NCPU NSOC NCOR NTHR LOAD MEMTOT MEMUSE SWAPTO SWAPUS 96 | ---------------------------------------------------------------------------------------------- 97 | global - - - - - - - - - - 98 | ip-172-31-44-135 lx-amd64 18 2 18 18 0.01 58.8G 603.1M 0.0 0.0 99 | ip-172-31-44-136 lx-amd64 18 2 18 18 0.01 58.8G 604.2M 0.0 0.0 100 | 101 | 102 | Submit the job: 103 | 104 | .. code-block:: none 105 | 106 | qsub ParallelBike.job 107 | 108 | Wait for the job to complete, you should see ``Job Complete`` at the end of the output file. 109 | 110 | Run visualization script 111 | ======================== 112 | 113 | .. code-block:: none 114 | 115 | ./RunVisualization.sh 116 | 117 | You should see something similar to this at the bottom of the output: 118 | 119 | .. code-block:: none 120 | 121 | View output at http://ec2-11-22-33-44.us-west-2.compute.amazonaws.com/output.png 122 | 123 | Example output image: 124 | 125 | .. image:: _images/OpenFOAM_output.png 126 | :width: 300px 127 | 128 | | 129 | 130 | 131 | -------------------------------------------------------------------------------- /README-WRF.rst: -------------------------------------------------------------------------------- 1 | ########## 2 | WRF on AWS 3 | ########## 4 | 5 | **Weather Research and Forecasting Model** 6 | 7 | .. contents:: 8 | :backlinks: none 9 | :depth: 2 10 | 11 | 12 | This guide documents the process of running WRF on AWS, including compiling WRF using the Intel compiler. 13 | 14 | .. note:: This guide uses the trial version of the Intel compiler, if you plan on running this in production, please contact Intel for a license. 15 | 16 | TL;DR 17 | 18 | #. Setup and launch cluster with CfnCluster 19 | #. NetCDF: Download, build with Intel compiler, and install 20 | #. WRF: Download, build with Intel compiler, and install 21 | #. Run CONUS 2.5k benchmark 22 | #. View result with ncview 23 | 24 | 25 | ************ 26 | Introduction 27 | ************ 28 | 29 | **WRF is used for both research and operation forecasting.** 30 | 31 | Main Web Site: 32 | http://www.wrf-model.org 33 | 34 | V3 User Guide: 35 | http://www2.mmm.ucar.edu/wrf/users/docs/user_guide_V3/contents.html 36 | 37 | Introduction Presentation: 38 | WRF Modeling System Overview - http://www2.mmm.ucar.edu/wrf/users/tutorial/201601/overview.pdf 39 | 40 | Tutorials 41 | ========= 42 | 43 | Conference Tutorials: 44 | Many WRF tutorials can be found on Tutorials Presented at user conference: 45 | 46 | http://www2.mmm.ucar.edu/wrf/users/supports/tutorial.html 47 | 48 | OnLine WRF Tutorial 49 | This online tutorial is a great place to start. It walks through some of the basic WRF information, compiling and usage. 50 | 51 | http://www2.mmm.ucar.edu/wrf/OnLineTutorial/index.htm 52 | 53 | 54 | ***************************** 55 | Cluster Setup with CfnCluster 56 | ***************************** 57 | 58 | CfnCluster ( https://aws.amazon.com/hpc/cfncluster/ ) will be used to setup the cluster. 59 | 60 | If this is your first time using CfnCluster, you should familiarize yourself with the usage. Documentation can be found here: https://aws.amazon.com/hpc/cfncluster/ . You should create a small test cluster before attempting to create the cluster you will use with your application. 61 | 62 | You can use **Getting Started with CfnCluster** ( http://cfncluster.readthedocs.io/en/latest/getting_started.html ) to do the initial configuration. 63 | 64 | 65 | Create a Post Install file specific to WRF 66 | ========================================== 67 | 68 | As part of the cluster setup, you will need to create a post install file that will be used when each instance starts. This post install script performs these operations on all instances (including the Master instance): 69 | 70 | - Installs additional packages 71 | - Disables HyperThreading 72 | - Sets the clocksource to "tsc" 73 | - Sets TCP values 74 | - Sets ulimit values - limits need to be setup properly to run (e.g. stack size: ``ulimit -s``) 75 | 76 | ***Appendix C - Post Install Script** has the script. Using this script, create a file in a S3 bucket similar to ``s3://bucket-id1-cfncluster/cfncluster_postinstall.sh``. Use that file for the ``post_install`` argument in the CfnCluster config file, and you will also need to add the bucket to the ``s3_read_write_resource`` option: 77 | 78 | For example: 79 | 80 | .. code-block:: bash 81 | 82 | post_install = s3://bucket-id1-cfncluster/cfncluster_postinstall.sh 83 | s3_read_write_resource = arn:aws:s3:::bucket-id1-cfncluster/* 84 | 85 | 86 | Once you have CfnCluster installed, create the cluster with the additional options below. These options are added or replace options to the previously created ``~/.cfncluster/config`` file. Many of the CfnCluster settings can use the default values (i.e. don't need to be included in the config file). These are cluster settings that have yielded positive results for WRF. The instance type chosen should not be considered the only one that works, but for this guide the ``c4.8xlarge`` instance type will be used. 87 | 88 | .. warning:: Several of these settings will result in higher cost. Please review the `EC2 costs `__ prior to cluster creation. 89 | 90 | .. code-block:: bash 91 | 92 | [cluster wrf] 93 | compute_instance_type = c4.8xlarge 94 | master_instance_type = c4.8xlarge 95 | master_root_volume_size = 100 96 | cluster_type = ondemand 97 | placement = cluster 98 | placement_group = DYNAMIC 99 | base_os = alinux 100 | extra_json = { "cfncluster" : { "cfn_scheduler_slots" : "cores" } } 101 | s3_read_write_resource = arn:aws:s3:::bucket-id1-cfncluster/* 102 | post_install = s3://bucket-id1-cfncluster/cfncluster_postinstall.sh 103 | ebs_settings = wrf-ebs 104 | 105 | [ebs wrf-ebs] ## Used for the NFS mounted file system 106 | volume_type = io1 107 | volume_size = 250 108 | volume_iops = 5000 109 | 110 | 111 | Create the cluster 112 | ================== 113 | 114 | After creating the post install script, and setting options in the CfnCluster config file specific to your application, create the cluster. 115 | 116 | 117 | .. note:: The remaining steps assume that you have created a cluster, and you can login to the Master instance. 118 | 119 | *************************************** 120 | Download and install the Intel compiler 121 | *************************************** 122 | 123 | Before building WRF or other related packages, the Intel compiler will need to be installed to achieve expected performance. You can download the compiler here: 124 | 125 | https://software.intel.com/en-us/intel-parallel-studio-xe 126 | 127 | After the cluster has been created, login to the Master instance. The Intel compiler needs to be installed in ``/shared`` on the **Master Instance**. 128 | 129 | 130 | 131 | ************************* 132 | Download and build NetCDF 133 | ************************* 134 | 135 | It is **strongly recommended** that you use NetCDF version 4.1.3 from the WRF compile link. 136 | 137 | `Appendix B - Build NetCDF with the Intel compiler`_ has the link to NetCDF and the build instructions using the Intel compiler. 138 | 139 | 140 | ***************************** 141 | Download WRF and weather data 142 | ***************************** 143 | 144 | Download WRF - Requires Account 145 | =============================== 146 | 147 | To be able to download WRF you will need an account on the WRF site. 148 | 149 | #. Go to this page: 150 | 151 | http://www2.mmm.ucar.edu/wrf/users/download/get_source.html 152 | 153 | #. Click on **New Users** (or **Retuning Users** if you already have an account) 154 | 155 | #. Complete the registration or just enter in your email address 156 | 157 | #. You should land on the download page, download **WRF-ARW**, the file should look something like this: http://www2.mmm.ucar.edu/wrf/src/WRFV3.8.1.TAR.gz 158 | 159 | #. Optionally download **WPS** 160 | 161 | 162 | Download weather data (as needed) 163 | ================================= 164 | 165 | Data download - NCAR's RDA ( http://www2.mmm.ucar.edu/wrf/users/ ) (Research Data Archive) 166 | 167 | 168 | *********** 169 | Compile WRF 170 | *********** 171 | 172 | These steps should be done on the **Master Instance**. 173 | 174 | These steps summarize the official steps, and use Intel compiler options. Although, the official WRF guide uses the GNU compiler, you should see better performance with the Intel compiler. Here is the official **Compile Tutorial** http://www2.mmm.ucar.edu/wrf/OnLineTutorial/compilation_tutorial.php, and here is a **Compile WRF & WPS** http://www2.mmm.ucar.edu/wrf/users/tutorial/201601/compiling.pdf presentation. 175 | 176 | .. note:: Building WRF using the Intel compiler 177 | 178 | Setup directories and download WRF 179 | ================================== 180 | 181 | Links to WRF code may be different, check the WRF site. This assumes that you already have a WRF account. 182 | 183 | .. code-block:: none 184 | 185 | $ cd /shared 186 | $ mkdir WRF 187 | $ cd WRF 188 | $ wget http://www2.mmm.ucar.edu/wrf/src/WRFV3.8.1.TAR.gz 189 | $ tar xvf WRFV3.8.1.TAR.gz 190 | $ cd WRFV3 191 | 192 | 193 | Setup build env 194 | =============== 195 | 196 | .. code-block:: none 197 | 198 | . /shared/intel/bin/compilervars.sh intel64 199 | export NETCDF=/shared/netcdf 200 | export WRFIO_NCD_LARGE_FILE_SUPPORT=1 201 | 202 | 203 | Run "configure" 204 | =============== 205 | 206 | Choose option "21" (SNB with AVX mods), and then option "1" for nesting: 207 | 208 | 209 | .. code-block:: none 210 | 211 | $ ./configure 212 | checking for perl5... no 213 | checking for perl... found /usr/bin/perl (perl) 214 | Will use NETCDF in dir: /shared/netcdf 215 | HDF5 not set in environment. Will configure WRF for use without. 216 | PHDF5 not set in environment. Will configure WRF for use without. 217 | Will use 'time' to report timing information 218 | $JASPERLIB or $JASPERINC not found in environment, configuring to build without grib2 I/O... 219 | ------------------------------------------------------------------------ 220 | Please select from among the following Linux x86_64 options: 221 | 222 | 1. (serial) 2. (smpar) 3. (dmpar) 4. (dm+sm) PGI (pgf90/gcc) 223 | 5. (serial) 6. (smpar) 7. (dmpar) 8. (dm+sm) PGI (pgf90/pgcc): SGI MPT 224 | 9. (serial) 10. (smpar) 11. (dmpar) 12. (dm+sm) PGI (pgf90/gcc): PGI accelerator 225 | 13. (serial) 14. (smpar) 15. (dmpar) 16. (dm+sm) INTEL (ifort/icc) 226 | 17. (dm+sm) INTEL (ifort/icc): Xeon Phi (MIC architecture) 227 | 18. (serial) 19. (smpar) 20. (dmpar) 21. (dm+sm) INTEL (ifort/icc): Xeon (SNB with AVX mods) 228 | 22. (serial) 23. (smpar) 24. (dmpar) 25. (dm+sm) INTEL (ifort/icc): SGI MPT 229 | 26. (serial) 27. (smpar) 28. (dmpar) 29. (dm+sm) INTEL (ifort/icc): IBM POE 230 | 30. (serial) 31. (dmpar) PATHSCALE (pathf90/pathcc) 231 | 32. (serial) 33. (smpar) 34. (dmpar) 35. (dm+sm) GNU (gfortran/gcc) 232 | 36. (serial) 37. (smpar) 38. (dmpar) 39. (dm+sm) IBM (xlf90_r/cc_r) 233 | 40. (serial) 41. (smpar) 42. (dmpar) 43. (dm+sm) PGI (ftn/gcc): Cray XC CLE 234 | 44. (serial) 45. (smpar) 46. (dmpar) 47. (dm+sm) CRAY CCE (ftn/cc): Cray XE and XC 235 | 48. (serial) 49. (smpar) 50. (dmpar) 51. (dm+sm) INTEL (ftn/icc): Cray XC 236 | 52. (serial) 53. (smpar) 54. (dmpar) 55. (dm+sm) PGI (pgf90/pgcc) 237 | 56. (serial) 57. (smpar) 58. (dmpar) 59. (dm+sm) PGI (pgf90/gcc): -f90=pgf90 238 | 60. (serial) 61. (smpar) 62. (dmpar) 63. (dm+sm) PGI (pgf90/pgcc): -f90=pgf90 239 | 64. (serial) 65. (smpar) 66. (dmpar) 67. (dm+sm) INTEL (ifort/icc): HSW/BDW 240 | 68. (serial) 69. (smpar) 70. (dmpar) 71. (dm+sm) INTEL (ifort/icc): KNL MIC 241 | 242 | Enter selection [1-71] : 21 243 | 244 | ------------------------------------------------------------------------ 245 | Compile for nesting? (1=basic, 2=preset moves, 3=vortex following) [default 1]: 1 246 | 247 | Configuration successful! 248 | ------------------------------------------------------------------------ 249 | testing for MPI_Comm_f2c and MPI_Comm_c2f 250 | MPI_Comm_f2c and MPI_Comm_c2f are supported 251 | testing for MPI_Init_thread 252 | MPI_Init_thread is supported 253 | testing for fseeko and fseeko64 254 | fseeko64 is supported 255 | ------------------------------------------------------------------------ 256 | 257 | ... snip ... 258 | 259 | Testing for NetCDF, C and Fortran compiler 260 | 261 | This installation of NetCDF is 64-bit 262 | C compiler is 64-bit 263 | Fortran compiler is 64-bit 264 | It will build in 64-bit 265 | 266 | 267 | Edit configure.wrf 268 | ================== 269 | 270 | Make these changes: 271 | 272 | .. code-block:: none 273 | 274 | DM_FC = mpiifort 275 | ... 276 | OPTAVX = -xHost 277 | CFLAGS_LOCAL = -w -O3 $(OPTAVX) -qopenmp 278 | ... 279 | FCOPTIM = -O3 $(OPTAVX) -qopenmp 280 | 281 | 282 | Compile 283 | ======= 284 | 285 | The compile time varies, but should take less than an hour. 286 | 287 | .. code-block:: none 288 | 289 | $ ./compile em_real 2>&1 | tee compile.log 290 | 291 | You should see four binaries successfully built, the output will also show the time taken for the compile: 292 | 293 | .. code-block:: none 294 | 295 | ========================================================================== 296 | build started: Fri Nov 18 18:31:37 UTC 2016 297 | build completed: Fri Nov 18 19:10:59 UTC 2016 298 | 299 | ---> Executables successfully built <--- 300 | 301 | -rwxrwxr-x 1 ec2-user ec2-user 46732186 Nov 18 19:10 main/ndown.exe 302 | -rwxrwxr-x 1 ec2-user ec2-user 46722859 Nov 18 19:10 main/real.exe 303 | -rwxrwxr-x 1 ec2-user ec2-user 45994883 Nov 18 19:10 main/tc.exe 304 | -rwxrwxr-x 1 ec2-user ec2-user 51485118 Nov 18 19:09 main/wrf.exe 305 | 306 | ========================================================================== 307 | 308 | 309 | Run wrf.exe to verify build 310 | =========================== 311 | 312 | Set limits 313 | ---------- 314 | 315 | The ``post_install`` script included in the appendix and mentioned above will set the necessary limits. If you don't set the ``stack size`` to ``unlimited``, you will receive an error similar to this: 316 | 317 | .. code-block:: none 318 | 319 | forrtl: severe (174): SIGSEGV, segmentation fault occurred 320 | 321 | You can just set stack size, but you should set all limits with the instructions mentioned above as part of the cluster creation. 322 | 323 | Set the stack size with this command: 324 | 325 | .. code-block:: none 326 | 327 | ulimit -s unlimited 328 | 329 | 330 | Run wrf.exe 331 | ----------- 332 | 333 | Run ``wrf.exe`` at the command line, you will need to export the path of NetCDF and the Intel libraries: 334 | 335 | .. code-block:: none 336 | 337 | $ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/shared/netcdf/lib:/shared/intel/lib 338 | $ ./wrf.exe 339 | starting wrf task 0 of 1 340 | 341 | Check ``rsl.out.0000`` and ``rsl.err.0000`` for errors. 342 | 343 | 344 | 345 | **************************** 346 | Run WRF CONUS 2.5k Benchmark 347 | **************************** 348 | 349 | .. note:: This Benchmark run is specific to the c4.8xlarge instance type. If you use another instance type, you will need to adjust the mpirun command and OpenMP threads. 350 | 351 | The WRF CONUS 2.5k Benchmark is here: 352 | 353 | http://www2.mmm.ucar.edu/wrf/WG2/benchv3/#_Toc212961289 354 | 355 | Benchmark description from http://www2.mmm.ucar.edu/wrf/WG2/benchv3/#_Toc212961289 : 356 | 357 | "Single domain, large size. 2.5 km CONUS, June 4, 2005" 358 | 359 | "Description: Latter 3 hours of a 9-hour, 2.5km resolution case covering the Continental U.S. (CONUS) domain June 4, 2005 with a 15 second time step. The benchmark period is hours 6-9 (3 hours), starting from a restart file from the end of the initial 6 hour period. As an alternative, the model may be run 9 hours from cold start. " 360 | 361 | 362 | Setup Benchmark 363 | =============== 364 | 365 | Create the benchmark directory 366 | ------------------------------ 367 | 368 | It's important to use the ``-a`` flag with the copy command, it preserves all of the symbolic links to the WRF binaries. 369 | 370 | .. code-block:: none 371 | 372 | $ cd WRFV3/test 373 | $ cp -a em_real em_real_2.5k_CONUS 374 | 375 | Copy benchmark files 376 | -------------------- 377 | 378 | The three files that were created above (restart file, boundary file, ``namelist.input`` file), need to be copied in to the ``WRFV3/test/em_real_2.5k_CONUS`` directory. 379 | 380 | 381 | Files need for benchmark 382 | ======================== 383 | 384 | Before downloading or editing ``namelist.input``, move the original out of the way: 385 | 386 | .. code-block:: none 387 | 388 | $ mv namelist.input namelist.input.dist 389 | 390 | You will need three files to run the benchmark: 391 | 392 | - Restart file (e.g. ``wrfrst_d01_2005-06-04_06_00_00``) 393 | - Boundary file (e.g. ``wrfbdy_d01``) 394 | - ``namelist.input`` file 395 | 396 | 397 | Option 1: Download files already prepared 398 | ----------------------------------------- 399 | 400 | If you prefer, all three of these files have been created and can be downloaded here: 401 | 402 | - https://s3.amazonaws.com/duff-public/wrf/2.5k_bench/namelist.input 403 | - https://s3.amazonaws.com/duff-public/wrf/2.5k_bench/wrfbdy_d01 (284MB) 404 | - https://s3.amazonaws.com/duff-public/wrf/2.5k_bench/wrfrst_d01_2005-06-04_06_00_00 (17GB) 405 | 406 | 407 | Option 2: Follow steps on WRF site to construct files 408 | ----------------------------------------------------- 409 | 410 | Go to the WRF CONUS 2.5k Benchmark site ( http://www2.mmm.ucar.edu/wrf/WG2/benchv3/#_Toc212961289 ), and follow the steps to download and construct the files needed. If you do manually download and construct the files you will need to make these changes to the **namelist.input** file. 411 | 412 | **Edit the namelist.input file:** 413 | 414 | - Remove pNetCDF usage: 415 | 416 | The version of WRF used in this guide does not include using pNetCDF, so you will need to edit the ``namelist.input`` to reflect that. In other words, "change the io_form_* settings in the time_control section of the namelist.input file from 11 to 2". 417 | 418 | - Add ``use_baseparam_fr_nml = .t.`` to the ``&dynamics`` section: 419 | 420 | It should look like this: 421 | 422 | .. code-block:: none 423 | 424 | &dynamics 425 | w_damping = 1, 426 | diff_opt = 1, 427 | km_opt = 4, 428 | khdif = 0, 429 | kvdif = 0, 430 | non_hydrostatic = .true., 431 | use_baseparam_fr_nml = .t., 432 | / 433 | 434 | Otherwise, you will see this error: 435 | 436 | .. code-block:: none 437 | 438 | -------------- FATAL CALLED --------------- 439 | FATAL CALLED FROM FILE: start_em.b LINE: 551 440 | start_em: did not find base state parameters in wrfinput. Add use_baseparam_fr_nml = .t. in &dynamics and rerun 441 | 442 | 443 | 444 | Run at the command line 445 | ======================= 446 | 447 | This assumes that the instances have two processors, each with 9 cores, running one task per processor each with 9 threads. 448 | 449 | This shows a run with 1440 threads (OMP_NUM_THREADS * np, or for this case 9 * 160) 450 | 451 | .. code-block:: none 452 | 453 | $ . /shared/intel/bin/compilervars.sh intel64 454 | $ export LD_LIBRARY_PATH=/shared/netcdf/lib:$LD_LIBRARY_PATH 455 | $ export OMP_NUM_THREADS=9 456 | $ export KMP_STACKSIZE=128M 457 | $ export KMP_AFFINITY=granularity=fine,compact,1,0 458 | $ qhost | grep ip- | awk {'print $1'} > ~/hostfile.80 459 | $ mpirun -hostfile ~/hostfile.80 -np 160 -ppn 2 ./wrf.exe 460 | 461 | The KMP_AFFINITY variable is explained in detail here, in the *"permute and offset combinations"* section: https://software.intel.com/en-us/node/522691#PERMUTE_AND_OFFSET_COMBINATIONS_WITH_TYPE 462 | 463 | Short Description: 464 | "The OpenMP* thread n+1 is bound to a thread context as close as possible to OpenMP* thread n, but on a different core. Once each core has been assigned one OpenMP* thread, the subsequent OpenMP* threads are assigned to the available cores in the same order, but they are assigned on different thread contexts." 465 | 466 | You should see this at the command line (for example): 467 | 468 | .. code-block:: none 469 | 470 | $ mpirun -hostfile hosts.2 -np 4 -ppn 2 ./wrf.exe 471 | starting wrf task 1 of 4 472 | starting wrf task 2 of 4 473 | starting wrf task 0 of 4 474 | starting wrf task 3 of 4 475 | 476 | Check the progress in the ``rsl.error.0000`` file: 477 | 478 | .. code-block:: none 479 | 480 | $ tail -1000f rsl.error.0000 481 | 482 | 483 | Verify wrf run 484 | ============== 485 | 486 | You should see **SUCCESS COMPLETE WRF** at the bottom of the rsl.out.0000 file or on STDOUT (for serial): 487 | 488 | .. code-block:: none 489 | 490 | d01 2001-10-25_03:00:00 wrf: SUCCESS COMPLETE WRF 491 | 492 | 493 | Check timing with stats.awk 494 | =========================== 495 | 496 | Download the ``stats.awk`` file from the WRF site: 497 | 498 | http://www2.mmm.ucar.edu/wrf/WG2/benchv3/stats.awk 499 | 500 | Then use this command to gather the timing information: 501 | 502 | .. code-block:: none 503 | 504 | $ grep 'Timing for main' rsl.error.0000 | tail -149 | awk '{print $9}' | awk -f stats.awk 505 | 506 | Example output: 507 | 508 | .. code-block:: none 509 | 510 | $ grep 'Timing for main' rsl.error.0000 | tail -149 | awk '{print $9}' | awk -f stats.awk 511 | --- 512 | items: 149 513 | max: 0.567450 514 | min: 0.154650 515 | sum: 27.599280 516 | mean: 0.185230 517 | mean/max: 0.326425 518 | 519 | 520 | Viewing results with ncview 521 | =========================== 522 | 523 | Install XWindows software on local machine. For OSX, this is XQuartz ( https://www.xquartz.org/ ) 524 | 525 | Install ncview, xterm, and xauth on the **Master Instance** 526 | 527 | .. code-block:: none 528 | 529 | sudo yum install ncview xterm xauth 530 | 531 | Reconnect to the **Master Instance** with X forwarding and test with ``xterm``, and you should see it display on your desktop: 532 | 533 | .. code-block:: none 534 | 535 | $ ssh -X -A -i key_1.pem ec2-user@ 536 | 537 | [ec2-user@ip-address]$ xterm 538 | 539 | Run ncview on the wrfout* file: 540 | 541 | .. code-block:: none 542 | 543 | [ec2-user@ip-address]$ cd WRFV3/test/em_real_2.5k_CONUS 544 | [ec2-user@ip-address]$ ncview wrfout_d01_2005-06-04_09_00_00 545 | 546 | 547 | You see the main application panel, and if you select **2d Vars -> UST** and click on the image, you should see something like this: 548 | 549 | .. image:: _images/ncview_wrf.png 550 | :width: 800px 551 | 552 | 553 | 554 | 555 | 556 | 557 | 558 | ************************************************* 559 | Appendix A - Build NetCDF with the Intel compiler 560 | ************************************************* 561 | 562 | .. note:: Building NetCDF with the Intel compiler 563 | 564 | The steps here summarize Intel's instructions from thier build notes: http://tinyurl.com/zvg7478 565 | 566 | Download NetCDF 4.1.3 from the WRF Compile Tutorial site: 567 | http://www2.mmm.ucar.edu/wrf/OnLineTutorial/compile_tutorial/tar_files/netcdf-4.1.3.tar.gz 568 | 569 | 570 | Setup env vars 571 | ============== 572 | 573 | Assumes compiler install in ``/shared/intel``: 574 | 575 | .. code-block:: none 576 | 577 | $ cat netcdf.intel.env 578 | export PATH=$PATH:/shared/intel/bin 579 | export CC=icc 580 | export CXX=icpc 581 | export CFLAGS='-O3 -xHost -ip -no-prec-div -static-intel' 582 | export CXXFLAGS='-O3 -xHost -ip -no-prec-div -static-intel' 583 | export F77=ifort 584 | export FC=ifort 585 | export F90=ifort 586 | export FFLAGS='-O3 -xHost -ip -no-prec-div -static-intel' 587 | export CPP='icc -E' 588 | export CXXCPP='icpc -E' 589 | 590 | 591 | Build 592 | ===== 593 | 594 | .. code-block:: none 595 | 596 | $ . netcdf.intel.env 597 | $ tar xf netcdf-4.1.3.tar.gz 598 | $ cd netcdf-4.1.3 599 | $ . /shared/intel/bin/compilervars.sh ia32 600 | $ . /shared/intel/bin/compilervars.sh intel64 601 | $ ./configure --prefix=/shared/netcdf --disable-netcdf-4 --disable-dap 602 | $ make 603 | 604 | 605 | 606 | Run "make check" 607 | ================ 608 | 609 | .. code-block:: none 610 | 611 | $ make check 2>&1 | tee make.check.out 612 | 613 | You should see several ``All N tests passed`` messages in the ``make.check.out`` file: 614 | 615 | .. code-block:: none 616 | 617 | $ grep passed make.check.out 618 | All 3 tests passed 619 | All 9 tests passed 620 | 1 test passed 621 | 1 test passed 622 | *** All tests of ncgen and ncdump using test0.cdl passed! 623 | *** All ncgen and ncdump with 64-bit offset format tests passed! 624 | *** All ncgen and ncdump test output for classic format passed! 625 | *** All ncgen and ncdump test output for 64-bit offset format passed! 626 | *** All ncdump test output for -t option with CF calendar atts passed! 627 | *** All utf8 tests of ncgen and ncdump passed! 628 | *** All nccopy tests passed! 629 | All 11 tests passed 630 | All 5 tests passed 631 | *** All tests of C++ API test output passed! 632 | All 6 tests passed 633 | All 2 tests passed 634 | All 7 tests passed 635 | All 7 tests passed 636 | All 7 tests passed 637 | 638 | 639 | 640 | Run "make install" 641 | ================== 642 | 643 | .. code-block:: none 644 | 645 | $ make install 646 | 647 | ... snip ... 648 | 649 | +-------------------------------------------------------------+ 650 | | Congratulations! You have successfully installed netCDF! | 651 | | | 652 | | You can use script "nc-config" to find out the relevant | 653 | | compiler options to build your application. Enter | 654 | | | 655 | | nc-config --help | 656 | | | 657 | | for additional information. | 658 | | | 659 | | CAUTION: | 660 | | | 661 | | If you have not already run "make check", then we strongly | 662 | | recommend you do so. It does not take very long. | 663 | | | 664 | | Before using netCDF to store important data, test your | 665 | | build with "make check". | 666 | | | 667 | | NetCDF is tested nightly on many platforms at Unidata | 668 | | but your platform is probably different in some ways. | 669 | | | 670 | | If any tests fail, please see the netCDF web site: | 671 | | http://www.unidata.ucar.edu/software/netcdf/ | 672 | | | 673 | | NetCDF is developed and maintained at the Unidata Program | 674 | | Center. Unidata provides a broad array of data and software | 675 | | tools for use in geoscience education and research. | 676 | | http://www.unidata.ucar.edu | 677 | +-------------------------------------------------------------+ 678 | 679 | 680 | ******************************** 681 | Appendix B - Post Install Script 682 | ******************************** 683 | 684 | .. code-block:: none 685 | 686 | #!/bin/bash 687 | 688 | USER=ec2-user 689 | 690 | # extra packages 691 | yum -y install screen dstat htop strace perf pdsh 692 | 693 | # Download and install hyperthread disabling script 694 | wget -O /etc/init.d/disable_hyperthreading https://cfncluster-public-scripts.s3.amazonaws.com/disable_hyperthreading 695 | chmod a+x /etc/init.d/disable_hyperthreading 696 | chkconfig --add /etc/init.d/disable_hyperthreading 697 | chkconfig --level 2345 disable_hyperthreading on 698 | /etc/init.d/disable_hyperthreading start 699 | 700 | # Switch the clock source to TSC 701 | echo "tsc" > /sys/devices/system/clocksource/clocksource0/current_clocksource 702 | 703 | # Set TCP windows 704 | cat >>/etc/sysctl.conf << EOF 705 | net.core.netdev_max_backlog = 1000000 706 | 707 | net.core.rmem_default = 124928 708 | net.core.rmem_max = 67108864 709 | net.core.wmem_default = 124928 710 | net.core.wmem_max = 67108864 711 | 712 | net.ipv4.tcp_keepalive_time = 1800 713 | net.ipv4.tcp_mem = 12184608 16246144 24369216 714 | net.ipv4.tcp_rmem = 4194304 8388608 67108864 715 | net.ipv4.tcp_syn_retries = 5 716 | net.ipv4.tcp_wmem = 4194304 8388608 67108864 717 | EOF 718 | 719 | sysctl -p 720 | 721 | # Set ulimits 722 | cat >>/etc/security/limits.conf << EOF 723 | # core file size (blocks, -c) 0 724 | * hard core 0 725 | * soft core 0 726 | 727 | # data seg size (kbytes, -d) unlimited 728 | * hard data unlimited 729 | * soft data unlimited 730 | 731 | # scheduling priority (-e) 0 732 | * hard priority 0 733 | * soft priority 0 734 | 735 | # file size (blocks, -f) unlimited 736 | * hard fsize unlimited 737 | * soft fsize unlimited 738 | 739 | # pending signals (-i) 256273 740 | * hard sigpending 1015390 741 | * soft sigpending 1015390 742 | 743 | # max locked memory (kbytes, -l) unlimited 744 | * hard memlock unlimited 745 | * soft memlock unlimited 746 | 747 | # open files (-n) 1024 748 | * hard nofile 65536 749 | * soft nofile 65536 750 | 751 | # POSIX message queues (bytes, -q) 819200 752 | * hard msgqueue 819200 753 | * soft msgqueue 819200 754 | 755 | # real-time priority (-r) 0 756 | * hard rtprio 0 757 | * soft rtprio 0 758 | 759 | # stack size (kbytes, -s) unlimited 760 | * hard stack unlimited 761 | * soft stack unlimited 762 | 763 | # cpu time (seconds, -t) unlimited 764 | * hard cpu unlimited 765 | * soft cpu unlimited 766 | 767 | # max user processes (-u) 1024 768 | * soft nproc 16384 769 | * hard nproc 16384 770 | 771 | # file locks (-x) unlimited 772 | * hard locks unlimited 773 | * soft locks unlimited 774 | EOF 775 | 776 | 777 | 778 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ## AWS HPC Workshops 2 | 3 | This is a repo of HPC workshops that will be used to facilitate on-site engagements, or be used at conferences and summits. This is not a code repo, but rather instructions and guides for enabling and validating HPC workloads on AWS. 4 | 5 | ## Current Workshops 6 | 7 | * [WRF on AWS](README-WRF.rst) 8 | * [OpenFOAM on AWS](README-OpenFOAM.rst) 9 | * [Abaqus on AWS](README-Abaqus.rst) 10 | 11 | ## Submitting a workshop 12 | 13 | 1. Issue a pull request 14 | 1. Push the new workshop content 15 | 1. The workshop will be verified by three AWS Solutions Architects 16 | 1. It will be checked for any IP or copyright restrictions 17 | 1. Published 18 | 19 | ## License 20 | 21 | This library is licensed under the Apache 2.0 License. 22 | -------------------------------------------------------------------------------- /_files/abaqus/abaqus_results.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-hpc-workshops/0030623f8110dcc037291579bc248674d73fea68/_files/abaqus/abaqus_results.jpg -------------------------------------------------------------------------------- /_files/wrf/namelist.input: -------------------------------------------------------------------------------- 1 | &time_control 2 | run_days = 0, 3 | run_hours = 3, 4 | run_minutes = 0, 5 | run_seconds = 0, 6 | start_year = 2005, 7 | start_month = 06, 8 | start_day = 04, 9 | start_hour = 06, 10 | start_minute = 00, 11 | start_second = 00, 12 | end_year = 2005, 13 | end_month = 06, 14 | end_day = 04, 15 | end_hour = 12, 16 | end_minute = 00, 17 | end_second = 00, 18 | interval_seconds = 10800 19 | input_from_file = .true., 20 | history_interval = 180, 21 | frames_per_outfile = 1, 22 | nocolons = .true., 23 | restart = .true., 24 | restart_interval = 360, 25 | io_form_history = 2, 26 | io_form_restart = 2, 27 | io_form_input = 2, 28 | io_form_boundary = 2, 29 | io_form_auxhist2 = 2, 30 | debug_level = 0 31 | / 32 | 33 | &domains 34 | time_step = 15, 35 | time_step_fract_num = 0, 36 | time_step_fract_den = 1, 37 | max_dom = 1, 38 | s_we = 1, 39 | e_we = 1501, 40 | s_sn = 1, 41 | e_sn = 1201, 42 | s_vert = 1, 43 | e_vert = 35, 44 | dx = 2500, 45 | dy = 2500, 46 | grid_id = 1, 47 | parent_id = 0, 48 | i_parent_start = 0, 49 | j_parent_start = 0, 50 | parent_grid_ratio = 1, 51 | parent_time_step_ratio = 1, 52 | feedback = 1, 53 | smooth_option = 0 54 | lagrange_order = 2 55 | interp_type = 2 56 | extrap_type = 2 57 | t_extrap_type = 2 58 | use_surface = .true. 59 | use_levels_below_ground = .true. 60 | num_metgrid_levels = 40, 61 | eta_levels = 1.00000 , 0.99258 , 0.98275 , 0.96996 , 0.95372 , 62 | 0.93357 , 0.90913 , 0.87957 , 0.84531 , 0.80683 , 63 | 0.76467 , 0.71940 , 0.67163 , 0.62198 , 0.57108 , 64 | 0.51956 , 0.46803 , 0.42030 , 0.37613 , 0.33532 , 65 | 0.29764 , 0.26290 , 0.23092 , 0.20152 , 0.17452 , 66 | 0.14978 , 0.12714 , 0.10646 , 0.08761 , 0.07045 , 67 | 0.05466 , 0.03981 , 0.02580 , 0.01258 , 0.00000 68 | / 69 | 70 | &physics 71 | mp_physics = 4, 72 | mp_zero_out = 2, 73 | mp_zero_out_thresh = 1.e-9, 74 | ra_lw_physics = 1, 75 | ra_sw_physics = 1, 76 | radt = 10, 77 | sf_sfclay_physics = 1, 78 | sf_surface_physics = 2, 79 | bl_pbl_physics = 1, 80 | bldt = 0, 81 | cu_physics = 0, 82 | cudt = 5, 83 | isfflx = 1, 84 | ifsnow = 0, 85 | icloud = 1, 86 | surface_input_source = 1, 87 | num_soil_layers = 4, 88 | maxiens = 1, 89 | maxens = 1, 90 | maxens2 = 1, 91 | maxens3 = 1, 92 | ensdim = 1, 93 | / 94 | 95 | &dynamics 96 | w_damping = 1, 97 | diff_opt = 1, 98 | km_opt = 4, 99 | khdif = 0, 100 | kvdif = 0, 101 | non_hydrostatic = .true., 102 | use_baseparam_fr_nml = .t., 103 | / 104 | 105 | &bdy_control 106 | spec_bdy_width = 5, 107 | spec_zone = 1, 108 | relax_zone = 4, 109 | specified = .true., 110 | nested = .false., 111 | / 112 | 113 | &namelist_quilt 114 | nio_tasks_per_group = 0, 115 | nio_groups = 1, 116 | / 117 | -------------------------------------------------------------------------------- /_images/OpenFOAM_output.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-hpc-workshops/0030623f8110dcc037291579bc248674d73fea68/_images/OpenFOAM_output.png -------------------------------------------------------------------------------- /_images/ncview_wrf.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/aws-hpc-workshops/0030623f8110dcc037291579bc248674d73fea68/_images/ncview_wrf.png --------------------------------------------------------------------------------