├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── LICENSE ├── README.md ├── pc_setup_scripts ├── git_download_and_run.sh ├── pcluster-config.template └── pcluster_install_spack.sh ├── pictures ├── ParallelClusterArchitecture.png ├── QVAPOR.JPG ├── SoilLevelTemperature.JPG └── SoillevelTemperature_over_time.JPG ├── scripts ├── get_noaa_grib_files.sh ├── prepare_config.sh ├── preprocess.sh ├── run_wrf.sh └── slurm_run_wrf.sh ├── setup.sh └── wrf_setup_scripts ├── build_dir.sh ├── compile_and_install_using_spack.sh ├── download_and_install_geog_data.sh ├── install_wrf.sh ├── update_packages_yaml.py └── wgrib_makefile.patch /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | ## Code of Conduct 2 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 3 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 4 | opensource-codeofconduct@amazon.com with any additional questions or comments. 5 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing Guidelines 2 | 3 | Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional 4 | documentation, we greatly value feedback and contributions from our community. 5 | 6 | Please read through this document before submitting any issues or pull requests to ensure we have all the necessary 7 | information to effectively respond to your bug report or contribution. 8 | 9 | 10 | ## Reporting Bugs/Feature Requests 11 | 12 | We welcome you to use the GitHub issue tracker to report bugs or suggest features. 13 | 14 | When filing an issue, please check existing open, or recently closed, issues to make sure somebody else hasn't already 15 | reported the issue. Please try to include as much information as you can. Details like these are incredibly useful: 16 | 17 | * A reproducible test case or series of steps 18 | * The version of our code being used 19 | * Any modifications you've made relevant to the bug 20 | * Anything unusual about your environment or deployment 21 | 22 | 23 | ## Contributing via Pull Requests 24 | Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that: 25 | 26 | 1. You are working against the latest source on the *master* branch. 27 | 2. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already. 28 | 3. You open an issue to discuss any significant work - we would hate for your time to be wasted. 29 | 30 | To send us a pull request, please: 31 | 32 | 1. Fork the repository. 33 | 2. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change. 34 | 3. Ensure local tests pass. 35 | 4. Commit to your fork using clear commit messages. 36 | 5. Send us a pull request, answering any default questions in the pull request interface. 37 | 6. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation. 38 | 39 | GitHub provides additional document on [forking a repository](https://help.github.com/articles/fork-a-repo/) and 40 | [creating a pull request](https://help.github.com/articles/creating-a-pull-request/). 41 | 42 | 43 | ## Finding contributions to work on 44 | Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels (enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any 'help wanted' issues is a great place to start. 45 | 46 | 47 | ## Code of Conduct 48 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 49 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 50 | opensource-codeofconduct@amazon.com with any additional questions or comments. 51 | 52 | 53 | ## Security issue notifications 54 | If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github issue. 55 | 56 | 57 | ## Licensing 58 | 59 | See the [LICENSE](LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution. 60 | 61 | We may ask you to sign a [Contributor License Agreement (CLA)](http://en.wikipedia.org/wiki/Contributor_License_Agreement) for larger changes. 62 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining a copy of 4 | this software and associated documentation files (the "Software"), to deal in 5 | the Software without restriction, including without limitation the rights to 6 | use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of 7 | the Software, and to permit persons to whom the Software is furnished to do so. 8 | 9 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 10 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS 11 | FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 12 | COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER 13 | IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN 14 | CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 15 | 16 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # HPC WORKSHOP WRF ON AWS with Graviton2 leveraging SPACK 2 | 3 | ## Introduction 4 | 5 | The **Weather Research and Forecasting (WRF)** Model is a next-generation mesoscale numerical weather prediction system designed for both atmospheric research and operational forecasting applications. 6 | 7 | If you are looking for additional information, please refer to: 8 | * Main Web Site: http://www.wrf-model.org 9 | * V4 User Guide: https://www2.mmm.ucar.edu/wrf/users/docs/user_guide_v4/contents.html 10 | * WRF Modeling System Toutorial - https://www2.mmm.ucar.edu/wrf/users/tutorial/tutorial.html 11 | * Benchmark results running large scale Weather Forecast models on AWS: https://aws.amazon.com/it/solutions/case-studies/maxar-case-study/ 12 | 13 | In this workshop we will set-up a Elastic HPC cluster, leveraging *AWS ParallelCluster*, 14 | that can immediately be used to start running weather forecasts. 15 | Scripts included in this repo are released under MIT license and perform all required tasks 16 | to download and compile required dependencies and tools needed to be able to run 17 | a full weather forecast from initialization data download to visualization of outcomes. 18 | 19 | **Spack** is a package manager for supercomputers, Linux, and macOS. It makes installing scientific software easy. Spack isn’t tied to a particular language; you can build a software stack in Python or R, link to libraries written in C, C++, or Fortran, and easily swap compilers or target specific microarchitectures. 20 | Learn more [here](https://spack.io/about/) 21 | 22 | ## Security 23 | 24 | See [CONTRIBUTING](CONTRIBUTING.md#security-issue-notifications) for more information. 25 | 26 | ## License 27 | 28 | This library is licensed under the MIT-0 License. See the LICENSE file. 29 | 30 | ## Cluster Setup with AWS ParallelCluster 31 | 32 | This workshop has been prepared for AWS ParallelCluster version 3. 33 | The first step is to install AWS ParallelCluster console according to the following guide: https://docs.aws.amazon.com/parallelcluster/latest/ug/install-v3-virtual-environment.html 34 | 35 | Once you have AWS ParallelCluster installed (and activated if you are installing it in a Python Virtual Env), the next step is to 36 | move to the directory generated by git when cloning the repo: 37 | 38 | ```bash 39 | cd 40 | ``` 41 | 42 | Now it is time to configure AWS ParallelCluster according to our needs. 43 | This is done by editing the template file provided in this repo: *pc_setup_scripts/pcluster-config.template* and adjusting parameters related to AWS account, region and VPC. 44 | In this template file you will find cluster settings that provides good results with WRF and also for many other tightly coupled compute intensive algorithms, 45 | key configuration parameters are: 46 | * using c6gn.16xlarge instances for compute nodes. Those instances have a low memory to CPU ratio but they have Graviton 2 cpu to optimize simulation cost and reduce carbon footprint and 100 Gbps network interfaces with EFA 47 | * enabling EFA to take advantage of low latency networking for distributed computing 48 | * configuring all compute nodes in the same placement group to further reduce latency due to physical distance among hosts 49 | * enabling [DCV](https://docs.aws.amazon.com/dcv/latest/adminguide/what-is-dcv.html) in order to be able to visualize computational results directly from the Head node 50 | * limiting maximum number of nodes to 6 (to avoid generating unexpected large clusters) 51 | * configuring 0 compute node at rest (all compute nodes are shut down whene there are no job submitted to the scheduler) 52 | 53 | For remaining AWS ParallelCluster configuration parameters we will use the default values (do not need to specify them in the config file). 54 | 55 | ### Warning 56 | * Several of these settings will result in higher cost. Please review [EC2 costs](https://aws.amazon.com/ec2/pricing/) prior creation. 57 | * Region has to be selected according to [Amazon Elastic Compute Cloud (EC2) C6gn Instances availability](https://aws.amazon.com/it/about-aws/global-infrastructure/regional-product-services). 58 | 59 | 60 | ## Create the cluster 61 | We are now ready to use AWS ParallelCluster to spin up our new cluster for running weather forecasts by tiping: 62 | ```bash 63 | mkdir -p $HOME/.parallelcluster 64 | cp ./pc_setup_scripts/pcluster-config.template $HOME/.parallelcluster/config 65 | pcluster create-cluster -r -n wrf-workshop -c $HOME/.parallelcluster/config 66 | ``` 67 | 68 | AWS ParallelCluster will create the components highlighted in the following picture, by leveraging AWS CloudFormation: 69 | ![AWS ParallelCluster architecture overview](./pictures/ParallelClusterArchitecture.png) 70 | Cluster spin-up will require approximately 30 minutes due to download and install spack. 71 | 72 | 73 | ## Log into the cluster using DCV 74 | Once the cluster is up and running we can now log onto it leveraging DCV. 75 | DCV allows a logging onto the Head node using a web browser and a signed url returned by following command: 76 | ```bash 77 | pcluster dcv-connect -r -n wrf-workshop --key-path 78 | ``` 79 | 80 | ## Install WRF 81 | Open a terminal within DCV, go to the filesystem shared across all nodes and run the install script 82 | ```bash 83 | cd /shared/hpc-workshop-wrf/wrf_setup_scripts/ 84 | bash install_wrf.sh 85 | ``` 86 | This script leverages spack to install all required software: wps, wgrib, wrf and ncview. It will take about 50 minutes to complete the download, compile and install steps. 87 | 88 | ## Run a weather forecast simulation 89 | 90 | ### Setup Environment variables 91 | Load environment variables related to WRF set-up: 92 | ```bash 93 | source /shared/setup_env.sh 94 | ``` 95 | This step has to be executed for each and every new window/session we open on the Head node. 96 | You can avoud it by appending that command to ~./bashrc 97 | ```bash 98 | echo "source /shared/setup_env.sh" >> ~/.bashrc 99 | echo "echo 'WRF ENVIRONMENT INITIALIZED'" >> ~/.bashrc 100 | 101 | ``` 102 | 103 | ### Download reference data 104 | In order to be able to run a weather forecast on a specific region we need: 105 | * Static Geographical Data: How soil is in that region (i.e. lakes, forests, cities, hills, snow, mountains,...). Since those data do not vary frequently they are downloaded by install_wrf script. 106 | * Gridded Meteorological Data coming from a large scale, gross grained, forecasting system that will set-up our starting condition. 107 | 108 | #### Download Gridded Meteorological Data. 109 | The Global Forecast System (GFS) is a weather forecast model produced by the National Centers for Environmental Prediction (NCEP). 110 | The entire globe is covered by the GFS at a base horizontal resolution of 18 miles (28 kilometers) between grid points, which is used by the operational forecasters who predict weather out to 16 days in the future. Horizontal resolution drops to 44 miles (70 kilometers) between grid point for forecasts between one week and two weeks. 111 | Gridded data are available for download through the NOAA National Operational Model Archive and Distribution System (NOMADS). 112 | As with most works of the U.S. government, GFS data is not copyrighted and is available for free in the public domain under provisions of U.S. law. 113 | Because of this, the model serves as the basis for the forecasts of numerous private, commercial, and foreign weather companies. 114 | 115 | GFS data can be downloaded using the following script and are related to the date when this script is run. 116 | This script should run every time we want to make a new forecast for the next coming days. 117 | ```bash 118 | get_noaa_grib_files.sh 119 | ``` 120 | Downloaded data are saved in: /shared/FORECAST/download/ 121 | 122 | ### Configure specific geographic area to cover with the forecast 123 | The next step is to set-up program configuration data (i.e. namelist.wps and namelist.input) 124 | according to the area and the timeframe for which we want to run our forecast. 125 | The following script automatically generates required configuration files 126 | for a map centered on the Mediterranean sea (latitude 40, longitude 14) covering central and south Europe and north Africa. 127 | ```bash 128 | prepare_config.sh 129 | ``` 130 | 131 | ### Generate reference data for selected domain 132 | In this step, we extract relevant data to us. 133 | We first set-up some environment variables: 134 | ```bash 135 | 136 | ulimit -s unlimited 137 | 138 | day=$(date +%Y%m%d) 139 | 140 | WPSWORK=${TARGET_DIR}/preproc 141 | WRFWORK=${TARGET_DIR}/run 142 | DIRGFS=${SHARED_DIR}/FORECAST/download/$day 143 | 144 | cd $WPSWORK 145 | #Cleanup data related to previous run (in case they exists) 146 | rm -f FILE* 147 | rm -f PFILE* 148 | rm -f met_em* 149 | ``` 150 | geogrid extracts relevant soil data from global dataset 151 | ```bash 152 | ./geogrid.exe 2>&1 |tee geogrid.$day.log 153 | 154 | ``` 155 | 156 | GFS data are then copied to preprocessing directory in order to be filtered for our region on interest 157 | ```bash 158 | cd $DIRGFS 159 | cp -f GRIBFILE* $WPSWORK 160 | 161 | cd $WPSWORK 162 | 163 | ./ungrib.exe 2>&1 |tee ungrib.$day.log 164 | 165 | ``` 166 | and mix them with soil related data 167 | ```bash 168 | ./metgrid.exe 2>&1 |tee metgrid.$day.log 169 | 170 | mv met_em* $WRFWORK 171 | 172 | ``` 173 | 174 | ### Run Weather Forecast 175 | Last data preparation step is performed by real.exe that prepares Initial and Boundary Conditions files for later processing. 176 | ```bash 177 | cd $WRFWORK 178 | ./real.exe 2>&1 |tee real.$day.log 179 | ``` 180 | 181 | We are now ready to submit a WRF job using pre-installed scheduler (i.e. SLURM) 182 | ```bash 183 | sbatch ${SCRIPTDIR}/slurm_run_wrf.sh 184 | ``` 185 | 186 | With this statement we are asking to the scheduler to run the job, job details are defined within the slurm_run_wrf script. 187 | Hereafter you can see the content of that script. 188 | ```bash 189 | #!/bin/bash 190 | #SBATCH --error=job.err 191 | #SBATCH --output=job.out 192 | #SBATCH --time=24:00:00 193 | #SBATCH --job-name=wrf 194 | #SBATCH --nodes=4 195 | #SBATCH --ntasks-per-node=64 196 | #SBATCH --cpus-per-task=1 197 | 198 | cd /shared/FORECAST/domains/test/run 199 | mpirun ./wrf.exe 200 | ``` 201 | 202 | This script requires the scheduler to run a job on 4 nodes, starting 64 process for each node (one per processor). 203 | You can also try changing those parameter to see how forecast performances are affected. 204 | The integration between scheduler and AWS ParallelCluster checks if thre are enough resoruces to accomodate this job and, if not, spin up a number of new instances according to job needs. 205 | Output and log files are saved under $WRFWORK directory. 206 | 207 | We can run the same forecast using a different number of cores in order to be able to understand how WRF scales on AWS. 208 | 209 | Further more, AWS ParallelCluster allows two different usage scenario: 210 | - Single AWS ParallelCluster used to run multiple jobs in parallel leveraging Job schedulers 211 | - Multiple AWS ParallelCluster running jobs indipendently 212 | 213 | this, together with virtually unlimited resources available on AWS, allows running multiple forecasts with different configuration parameters in order to be able 214 | to better evaluate if the forecast is stable (different set-ups converge to the similar results) 215 | or not (small changes in config parameters led to completely different results). 216 | 217 | 218 | ### Explore results 219 | After the forecast is completed we can have a look at WRF's output files using ncview, a graphical application allowing visualization of WRF output. 220 | 221 | 222 | ```bash 223 | cd $WRFWORK 224 | ncview wrfout* 225 | ``` 226 | 227 | Hereafter a few screenshots: 228 | Humidiy (QVapor) 229 | ![QVAPOR](./pictures/QVAPOR.JPG) 230 | 231 | 232 | Soil Level Temperature (SLT) 233 | ![Soil Level Temperature](./pictures/SoilLevelTemperature.JPG) 234 | 235 | 236 | Soil Level Temperature (SLT) over time on a specific map point 237 | ![Soil Level Temperature for a selected poin over Time](./pictures/SoillevelTemperature_over_time.JPG) 238 | 239 | 240 | 241 | The following table shows 3 different tests for the same forecast, involving different level of parallelism and using gcc. 242 | 243 | | Number of Cores | WRF Elapsed Time (gcc) | 244 | |----------------------|:----------------------:| 245 | | 128 (2 nodes) | 6000 sec. | 246 | | 256 (4 nodes) | 3500 sec. | 247 | | 216 (6 nodes) | 2600 sec. | 248 | 249 | 250 | 251 | ### Cluster Cleanup 252 | To remove the cluster we can use the following command issued from the parallel cluster console 253 | ```bash 254 | pcluster delete-cluster -n wrf-workshop -r 255 | ``` 256 | 257 | This command deletes the Head node and you will loose any forecast data. 258 | The Head node can also be simply switched off when not used and switched on again 259 | to start processing a new forecast. 260 | -------------------------------------------------------------------------------- /pc_setup_scripts/git_download_and_run.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | #Load Parallelcluster environment variables 3 | . /etc/parallelcluster/cfnconfig 4 | 5 | github_repo=$(echo ${cfn_postinstall_args}| cut -d ',' -f 1 ) 6 | setup_command=$(echo ${cfn_postinstall_args}| cut -d ',' -f 2 ) 7 | shared_folder=$(echo $cfn_ebs_shared_dirs | cut -d ',' -f 1 ) 8 | 9 | echo "ARUMENTS $cfn_postinstall_args" 10 | echo "REPO: ${github_repo}" 11 | echo "SETUP COMMAND: ${setup_command}" 12 | echo "SHARED FOLDER: ${shared_folder}" 13 | 14 | dir_name=$(basename -s .git ${github_repo}) 15 | 16 | case ${cfn_node_type} in 17 | HeadNode) 18 | echo "I am the HeadNode node" 19 | cd ${shared_folder} 20 | git clone ${github_repo} 21 | REPO_FOLDER=$(basename ${github_repo} | cut -f 1 -d '.') 22 | 23 | ;; 24 | ComputeFleet) 25 | echo "I am a Compute node" 26 | ;; 27 | esac 28 | 29 | cd ${shared_folder}/${dir_name} 30 | bash -x ${setup_command} >/tmp/setup.log 2>&1 31 | exit $? -------------------------------------------------------------------------------- /pc_setup_scripts/pcluster-config.template: -------------------------------------------------------------------------------- 1 | Region: ${AWS_REGION} 2 | Image: 3 | Os: alinux2 4 | HeadNode: 5 | InstanceType: ${HEADNODE_INSTANCE:-c6gn.2xlarge} 6 | #DisableSimultaneousMultithreading: true 7 | Networking: 8 | SubnetId: ${SUBNET_ID} 9 | LocalStorage: 10 | RootVolume: 11 | Size: 100 12 | VolumeType: gp3 13 | Ssh: 14 | KeyName: ${SSH_KEY} 15 | Dcv: 16 | Enabled: true 17 | 18 | CustomActions: 19 | OnNodeConfigured: 20 | Script: https://raw.githubusercontent.com/aws-samples/hpc-workshop-wrf/master/pc_setup_scripts/git_download_and_run.sh 21 | Args: 22 | - https://github.com/aws-samples/hpc-workshop-wrf.git,setup.sh 23 | 24 | SharedStorage: 25 | - MountDir: /shared 26 | Name: shared 27 | StorageType: Ebs 28 | EbsSettings: 29 | VolumeType: gp3 30 | Size: 100 31 | 32 | Scheduling: 33 | Scheduler: slurm 34 | SlurmQueues: 35 | - Name: c6gn 36 | ComputeResources: 37 | - Name: c6gn 38 | InstanceType: ${COMPUTE_INSTANCE:-c6gn.16xlarge} 39 | MinCount: 0 40 | MaxCount: 6 41 | Efa: 42 | Enabled: true 43 | Networking: 44 | SubnetIds: 45 | - ${SUBNET_ID} 46 | PlacementGroup: 47 | Enabled: true 48 | -------------------------------------------------------------------------------- /pc_setup_scripts/pcluster_install_spack.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | set -e 3 | 4 | ## 5 | # Modified version of: https://raw.githubusercontent.com/spack/spack-configs/main/AWS/parallelcluster/postinstall.sh 6 | # Removed gcc compiler install to reduce cluster start-up time 7 | # 8 | 9 | ############################################################################################## 10 | # # This script will setup Spack and best practices for a few applications. # 11 | # # Use as postinstall in AWS ParallelCluster (https://docs.aws.amazon.com/parallelcluster/) # 12 | ############################################################################################## 13 | 14 | install_in_foreground=false 15 | while [ $# -gt 0 ]; do 16 | case $1 in 17 | -v ) 18 | set -v 19 | shift 20 | ;; 21 | -fg ) 22 | install_in_foreground=true 23 | shift 24 | ;; 25 | -nointel ) 26 | export NO_INTEL_COMPILER=1 27 | shift 28 | ;; 29 | * ) 30 | echo "Unknown argument: $1" 31 | exit 1 32 | ;; 33 | esac 34 | done 35 | 36 | setup_variables() { 37 | # Install onto first shared storage device 38 | cluster_config="/opt/parallelcluster/shared/cluster-config.yaml" 39 | if [ -f "${cluster_config}" ]; then 40 | pip3 install pyyaml 41 | os=$(python3 << EOF 42 | #/usr/bin/env python 43 | import yaml 44 | with open("${cluster_config}", 'r') as s: 45 | print(yaml.safe_load(s)["Image"]["Os"]) 46 | EOF 47 | ) 48 | 49 | case "${os}" in 50 | alinux*) 51 | cfn_cluster_user="ec2-user" 52 | ;; 53 | centos*) 54 | cfn_cluster_user="centos" 55 | ;; 56 | ubuntu*) 57 | cfn_cluster_user="ubuntu" 58 | ;; 59 | *) 60 | cfn_cluster_user="" 61 | esac 62 | 63 | cfn_ebs_shared_dirs=$(python3 << EOF 64 | #/usr/bin/env python 65 | import yaml 66 | with open("${cluster_config}", 'r') as s: 67 | print(yaml.safe_load(s)["SharedStorage"][0]["MountDir"]) 68 | EOF 69 | ) 70 | scheduler=$(python3 << EOF 71 | #/usr/bin/env python 72 | import yaml 73 | with open("${cluster_config}", 'r') as s: 74 | print(yaml.safe_load(s)["Scheduling"]["Scheduler"]) 75 | EOF 76 | ) 77 | elif [ -f /etc/parallelcluster/cfnconfig ]; then 78 | . /etc/parallelcluster/cfnconfig 79 | else 80 | echo "Cannot find ParallelCluster configs" 81 | cfn_ebs_shared_dirs="" 82 | fi 83 | 84 | # If we cannot find any shared directory, use $HOME of standard user 85 | if [ -z "${cfn_ebs_shared_dirs}" ]; then 86 | for cfn_cluster_user in ec2-user centos ubuntu; do 87 | [ -d "/home/${cfn_cluster_user}" ] && break 88 | done 89 | cfn_ebs_shared_dirs="/home/${cfn_cluster_user}" 90 | fi 91 | 92 | install_path=${SPACK_ROOT:-"${cfn_ebs_shared_dirs}/spack"} 93 | echo "Installing Spack into ${install_path}." 94 | spack_branch="develop" 95 | 96 | scriptdir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" 97 | } 98 | 99 | major_version() { 100 | pcluster_version=$(grep -oE '[0-9]*\.[0-9]*\.[0-9]*' /opt/parallelcluster/.bootstrapped) 101 | echo "${pcluster_version/\.*}" 102 | } 103 | 104 | # Make first user owner of Spack installation when script exits. 105 | fix_owner() { 106 | rc=$? 107 | if [ ${downloaded} -eq 0 ] 108 | then 109 | chown -R ${cfn_cluster_user}:${cfn_cluster_user} "${install_path}" 110 | fi 111 | exit $rc 112 | } 113 | 114 | download_spack() { 115 | if [ -z "${SPACK_ROOT}" ] 116 | then 117 | [ -d ${install_path} ] || \ 118 | if [ -n "${spack_branch}" ] 119 | then 120 | git clone https://github.com/spack/spack -b ${spack_branch} ${install_path} 121 | elif [ -n "${spack_commit}" ] 122 | then 123 | git clone https://github.com/spack/spack ${install_path} 124 | cd ${install_path} && git checkout ${spack_commit} 125 | fi 126 | return 0 127 | else 128 | # Let the script know we did not download spack, so the owner will not be fixed on exit. 129 | return 1 130 | fi 131 | } 132 | 133 | architecture() { 134 | lscpu | grep "Architecture:" | awk '{print $2}' 135 | } 136 | 137 | # zen3 EC2 instances (e.g. hpc6a) is misidentified as zen2 so zen3 packages are found under packages-zen2.yaml. 138 | target() { 139 | ( 140 | . ${install_path}/share/spack/setup-env.sh 141 | spack arch -t 142 | ) 143 | } 144 | 145 | download_packages_yaml() { 146 | # $1: spack target 147 | . ${install_path}/share/spack/setup-env.sh 148 | target="${1}" 149 | curl -Ls https://raw.githubusercontent.com/spack/spack-configs/main/AWS/parallelcluster/packages-"${target}".yaml -o /tmp/packages.yaml 150 | if [ "$(cat /tmp/packages.yaml)" = "404: Not Found" ]; then 151 | # Pick up parent if current generation is not available 152 | for target in $(spack-python -c 'print(" ".join(spack.platforms.host().target("'"${target}"'").microarchitecture.to_dict()["parents"]))'); do 153 | if [ -z "${target}" ] ; then 154 | echo "Cannot find suitable packages.yaml" 155 | exit 1 156 | fi 157 | download_packages_yaml "${target}" 158 | done 159 | else 160 | # Exit "for target in ..." loop. 161 | break &>/dev/null 162 | fi 163 | } 164 | 165 | set_pcluster_defaults() { 166 | # Set versions of pre-installed software in packages.yaml 167 | [ -z "${SLURM_ROOT}" ] && SLURM_ROOT="/opt/slurm" 168 | [ -z "${SLURM_VERSION}" ] && SLURM_VERSION=$(strings /opt/slurm/lib/libslurm.so | grep -e '^VERSION' | awk '{print $2}' | sed -e 's?"??g') 169 | [ -z "${LIBFABRIC_MODULE_VERSION}" ] && LIBFABRIC_MODULE_VERSION=$(grep 'Version:' /opt/amazon/efa/lib64/pkgconfig/libfabric.pc | awk '{print $2}') 170 | [ -z "${LIBFABRIC_MODULE}" ] && LIBFABRIC_MODULE="libfabric-aws/${LIBFABRIC_MODULE_VERSION}" 171 | [ -z "${LIBFABRIC_VERSION}" ] && LIBFABRIC_VERSION=${LIBFABRIC_MODULE_VERSION//amzn*} 172 | [ -z "${GCC_VERSION}" ] && GCC_VERSION=$(gcc -v 2>&1 |tail -n 1| awk '{print $3}' ) 173 | 174 | # Write the above as actual yaml file and only parse the \$. 175 | mkdir -p ${install_path}/etc/spack 176 | 177 | # Find suitable packages.yaml. If not for this architecture then for its parents. 178 | ( download_packages_yaml "$(target)" ) 179 | eval "echo \"$(cat /tmp/packages.yaml)\"" > ${install_path}/etc/spack/packages.yaml 180 | 181 | for f in modules config; do 182 | curl -Ls https://raw.githubusercontent.com/spack/spack-configs/main/AWS/parallelcluster/${f}.yaml -o ${install_path}/etc/spack/${f}.yaml 183 | done 184 | } 185 | 186 | setup_spack() { 187 | cd "${install_path}" 188 | 189 | # Load spack at login 190 | if [ -z "${SPACK_ROOT}" ] 191 | then 192 | case "${scheduler}" in 193 | slurm) 194 | echo -e "\n# Spack setup from Github repo spack-configs" >> /opt/slurm/etc/slurm.sh 195 | echo -e "\n# Spack setup from Github repo spack-configs" >> /opt/slurm/etc/slurm.csh 196 | echo ". ${install_path}/share/spack/setup-env.sh &>/dev/null || true" >> /opt/slurm/etc/slurm.sh 197 | echo ". ${install_path}/share/spack/setup-env.csh &>/dev/null || true" >> /opt/slurm/etc/slurm.csh 198 | ;; 199 | *) 200 | echo "WARNING: Spack will need to be loaded manually when ssh-ing to compute instances." 201 | echo ". ${install_path}/share/spack/setup-env.sh" > /etc/profile.d/spack.sh 202 | echo ". ${install_path}/share/spack/setup-env.csh" > /etc/profile.d/spack.csh 203 | esac 204 | fi 205 | 206 | . "${install_path}/share/spack/setup-env.sh" 207 | spack compiler add --scope site 208 | spack external find --scope site 209 | 210 | # Remove all autotools/buildtools packages. These versions need to be managed by spack or it will 211 | # eventually end up in a version mismatch (e.g. when compiling gmp). 212 | spack tags build-tools | xargs -I {} spack config --scope site rm packages:{} 213 | [ -z "${CI_PROJECT_DIR}" ] && spack mirror add --scope site "aws-pcluster" "https://binaries.spack.io/develop/aws-pcluster-$(target | sed -e 's?_avx512??1')" 214 | spack buildcache keys --install --trust 215 | } 216 | 217 | install_packages() { 218 | if [ -n "${SPACK_ROOT}" ]; then 219 | [ -f /opt/slurm/etc/slurm.sh ] && . /opt/slurm/etc/slurm.sh || . /etc/profile.d/spack.sh 220 | fi 221 | 222 | # Compiler needed for all kinds of codes. It makes no sense not to install it. 223 | # Get gcc from buildcache 224 | # AVOID GCC INSTALL TO REDUCE SET-UP TIME 225 | #spack install gcc 226 | #( 227 | # spack load gcc 228 | # spack compiler add --scope site 229 | #) 230 | 231 | if [ -z "${NO_INTEL_COMPILER}" ] && [ "x86_64" == "$(architecture)" ] 232 | then 233 | # Add oneapi@latest & intel@latest 234 | spack install intel-oneapi-compilers-classic 235 | bash -c ". "$(spack location -i intel-oneapi-compilers)/setvars.sh"; spack compiler add --scope site" 236 | fi 237 | 238 | # Install any specs provided to the script. 239 | for spec in "$@" 240 | do 241 | [ -z "${spec}" ] || spack install -U "${spec}" 242 | done 243 | } 244 | 245 | if [ "3" != "$(major_version)" ]; then 246 | echo "ParallelCluster version $(major_version) not supported." 247 | exit 1 248 | fi 249 | 250 | tmpfile=$(mktemp) 251 | echo "$(declare -pf) 252 | trap \"fix_owner\" SIGINT EXIT 253 | setup_variables 254 | download_spack | true 255 | downloaded=\${PIPESTATUS[0]} 256 | set_pcluster_defaults 257 | setup_spack 258 | install_packages \"$@\" 259 | echo \"*** Spack setup completed ***\" 260 | rm -f ${tmpfile} 261 | " > ${tmpfile} 262 | 263 | bash ${tmpfile} 264 | -------------------------------------------------------------------------------- /pictures/ParallelClusterArchitecture.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/hpc-workshop-wrf/18ce82a1063694cf88116dccf6c24638f333eb6f/pictures/ParallelClusterArchitecture.png -------------------------------------------------------------------------------- /pictures/QVAPOR.JPG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/hpc-workshop-wrf/18ce82a1063694cf88116dccf6c24638f333eb6f/pictures/QVAPOR.JPG -------------------------------------------------------------------------------- /pictures/SoilLevelTemperature.JPG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/hpc-workshop-wrf/18ce82a1063694cf88116dccf6c24638f333eb6f/pictures/SoilLevelTemperature.JPG -------------------------------------------------------------------------------- /pictures/SoillevelTemperature_over_time.JPG: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/hpc-workshop-wrf/18ce82a1063694cf88116dccf6c24638f333eb6f/pictures/SoillevelTemperature_over_time.JPG -------------------------------------------------------------------------------- /scripts/get_noaa_grib_files.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | source /shared/setup_env.sh 4 | 5 | cd ${SHARED_DIR}/FORECAST/download 6 | 7 | # Script for fast downloading of GFS data in grib2 format 8 | # Select levels, variables and subregion of interest 9 | # 10 | ## D A T E ################################################################# 11 | ## 12 | if [ "$1" = help ] ; then 13 | echo "usage: $0 [HHHHMMGG] date of GFS run to be downloaded (last month only)" 14 | return 15 | fi 16 | if [ "$1" != "" ] ; then 17 | day=$1 18 | else 19 | day=$(date +%Y%m%d) 20 | fi 21 | 22 | echo "INPUT DATA: " $day $1 23 | 24 | ############################################################################# 25 | # Definition of working directory 26 | dirwork=$(pwd) 27 | 28 | # Definition of local directory where files are stored 29 | dirgfs=$dirwork/$day 30 | mkdir $dirgfs 31 | 32 | # Definition of forecast cycle and forecast hours 33 | FCY='00' 34 | fhrs='000 003 006 009 012 015 018 021 024 027 030 033 036 039 042 045 048 051 054 057 060 063 066 069 072 075 078 081 084 087 090 093 096 099 102 105 108 111 114 117 120' 35 | 36 | rm -f $dirgfs/gfs_ok 37 | cd $dirgfs 38 | rm -f GRIBFILE* grib2file* full_grib_file 39 | 40 | declare -A seq_to_sort=( ["000"]="AAA" ["003"]="AAB" ["006"]="AAC" ["009"]="AAD" ["012"]="AAE" ["015"]="AAF" ["018"]="AAG" ["021"]="AAH" ["024"]="AAI" ["027"]="AAJ" ["030"]="AAK" ["033"]="AAL" ["036"]="AAM" ["039"]="AAN" ["042"]="AAO" ["045"]="AAP" ["048"]="AAQ" ["051"]="AAR" ["054"]="AAS" ["057"]="AAT" ["060"]="AAU" ["063"]="AAV" ["066"]="AAW" ["069"]="AAX" ["072"]="AAY" ["075"]="AAZ" ["078"]="ABA" ["081"]="ABB" ["084"]="ABC" ["087"]="ABD" ["090"]="ABE" ["093"]="ABF" ["096"]="ABG" ["099"]="ABH" ["102"]="ABI" ["105"]="ABJ" ["108"]="ABK" ["111"]="ABL" ["114"]="ABM" ["117"]="ABN" ["120"]="ABO" ) 41 | 42 | for fhr in $fhrs 43 | do 44 | 45 | #20210630 Change in NOAA Attributes LANDN vs LAND and file path 46 | #wget --quiet --no-check-certificate "https://nomads.ncep.noaa.gov/cgi-bin/filter_gfs_0p25.pl?file=gfs.t"$FCY"z.pgrb2.0p25.f"$fhr"&lev_0-0.1_m_below_ground=on&lev_0.1-0.4_m_below_ground=on&lev_0.4-1_m_below_ground=on&lev_1000_mb=on&lev_100_m_above_ground=on&lev_100_mb=on&lev_10_m_above_ground=on&lev_10_mb=on&lev_1-2_m_below_ground=on&lev_150_mb=on&lev_200_mb=on&lev_20_mb=on&lev_250_mb=on&lev_2_m_above_ground=on&lev_300_mb=on&lev_30-0_mb_above_ground=on&lev_30_mb=on&lev_350_mb=on&lev_400_mb=on&lev_450_mb=on&lev_500_mb=on&lev_50_mb=on&lev_550_mb=on&lev_600_mb=on&lev_650_mb=on&lev_700_mb=on&lev_70_mb=on&lev_750_mb=on&lev_800_mb=on&lev_80_m_above_ground=on&lev_850_mb=on&lev_900_mb=on&lev_925_mb=on&lev_950_mb=on&lev_975_mb=on&lev_mean_sea_level=on&lev_surface=on&var_ACPCP=on&var_APCP=on&var_DLWRF=on&var_DSWRF=on&var_HGT=on&var_LANDN=on&var_PRES=on&var_PRMSL=on&var_RH=on&var_SOILW=on&var_TMP=on&var_TSOIL=on&var_UGRD=on&var_ULWRF=on&var_USWRF=on&var_VGRD=on&var_WEASD=on&subregion=&leftlon=-25&rightlon=50&toplat=65&bottomlat=20&dir=%2Fgfs."$day"%2F"$FCY"" -O $dirgfs/GRIBFILE.${seq_to_sort[${fhr}]} & 47 | wget --quiet --no-check-certificate "https://nomads.ncep.noaa.gov/cgi-bin/filter_gfs_0p25.pl?file=gfs.t"$FCY"z.pgrb2.0p25.f"$fhr"&lev_0-0.1_m_below_ground=on&lev_0.1-0.4_m_below_ground=on&lev_0.4-1_m_below_ground=on&lev_1000_mb=on&lev_100_m_above_ground=on&lev_100_mb=on&lev_10_m_above_ground=on&lev_10_mb=on&lev_1-2_m_below_ground=on&lev_150_mb=on&lev_200_mb=on&lev_20_mb=on&lev_250_mb=on&lev_2_m_above_ground=on&lev_300_mb=on&lev_30-0_mb_above_ground=on&lev_30_mb=on&lev_350_mb=on&lev_400_mb=on&lev_450_mb=on&lev_500_mb=on&lev_50_mb=on&lev_550_mb=on&lev_600_mb=on&lev_650_mb=on&lev_700_mb=on&lev_70_mb=on&lev_750_mb=on&lev_800_mb=on&lev_80_m_above_ground=on&lev_850_mb=on&lev_900_mb=on&lev_925_mb=on&lev_950_mb=on&lev_975_mb=on&lev_mean_sea_level=on&lev_surface=on&var_ACPCP=on&var_APCP=on&var_DLWRF=on&var_DSWRF=on&var_HGT=on&var_LAND=on&var_PRES=on&var_PRMSL=on&var_RH=on&var_SOILW=on&var_TMP=on&var_TSOIL=on&var_UGRD=on&var_ULWRF=on&var_USWRF=on&var_VGRD=on&var_WEASD=on&subregion=&leftlon=-25&rightlon=50&toplat=65&bottomlat=20&dir=%2Fgfs."$day"%2F"$FCY"%2Fatmos" -O $dirgfs/GRIBFILE.${seq_to_sort[${fhr}]} & 48 | sleep 1 49 | 50 | done 51 | 52 | wait 53 | 54 | ######################################### 55 | 56 | cat GRIBFILE* > full_grib_file 57 | length=$(wgrib2 full_grib_file|tail -1|awk -F":" '{ print $1}') 58 | echo $length 59 | 60 | if [ "$length" != "6839" ] ; then 61 | echo "Download failed" 62 | cd $dirwork 63 | exit 1 64 | else 65 | cd $dirgfs 66 | echo 2 > gfs_ok 67 | rm -f full_grib_file 68 | fi 69 | 70 | echo "End of Download" 71 | exit 0 72 | -------------------------------------------------------------------------------- /scripts/prepare_config.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | source /shared/setup_env.sh 3 | 4 | WPSWORK=${TARGET_DIR}/preproc 5 | WRFWORK=${TARGET_DIR}/run 6 | GEOG_DIR=${TARGET_DIR}/../geog 7 | 8 | cd ${SHARED_DIR}/FORECAST/download 9 | 10 | ulimit -s unlimited 11 | 12 | 13 | #Set-up current date as processing date variables 14 | DATA=$(date +%Y%m%d)'00' 15 | DATINA=$(date +%Y%m%d) 16 | 17 | 18 | run='00' 19 | dt='1' 20 | ndom='1' 21 | 22 | #Models Parameters 23 | #Simulation Elapsed time 24 | days1=2 25 | hrs1=0 26 | #Hours 27 | f1=48 28 | #Simulation Start hours 29 | sh1=00 30 | #Simulation End hours 31 | eh1=00 32 | #BC Intervall 33 | bc1=10800 34 | #Timestep 35 | t1=10 36 | #Timestep to grids ratio 37 | rt=3 38 | #Feedback (1-way =0 ; 2-ways =1 ) 39 | fdbck=0 40 | #Smoothing 41 | smooth=0 42 | 43 | 44 | #Grid Definition 460 x 270 x 35 45 | nx1=460 46 | ny1=270 47 | nz1=35 48 | dx1=10000 49 | 50 | #Metgrid Levels and VTable 51 | metlev=27 52 | Vtab=$WPS_DIR'/ungrib/Variable_Tables/Vtable.GFS' 53 | 54 | 55 | #Compute derived variables 56 | sdate=$(date -d "$DATINA" +%Y%m%d) 57 | sy=$(echo $sdate | awk '{print substr($1,1,4)}') 58 | sm=$(echo $sdate | awk '{print substr($1,5,2)}') 59 | sd=$(echo $sdate | awk '{print substr($1,7,2)}') 60 | 61 | fdate=$(date -d "$DATINA $days1 days" +%Y%m%d) 62 | 63 | ey=$(echo $fdate | awk '{print substr($1,1,4)}') 64 | em=$(echo $fdate | awk '{print substr($1,5,2)}') 65 | ed=$(echo $fdate | awk '{print substr($1,7,2)}') 66 | 67 | datenest=$(date -d "$DATINA" +%Y)'-'$(date -d "$DATINA" +%m)'-'$(date -d "$DATINA" +%d)'_'$sh1':00:00' 68 | dateend=$(date -d "$fdate" +%Y)'-'$(date -d "$fdate" +%m)'-'$(date -d "$fdate" +%d)'_'$eh1':00:00' 69 | 70 | ctlname1='WRF_FAT_'$DATA'.ctl' 71 | grbname1='WRF_FAT_'$DATA'.grb' 72 | 73 | 74 | 75 | #Preparing WRF and WPS config files 76 | 77 | cd $WRFWORK 78 | 79 | rm -f wrfout* 80 | rm -f rsl.* 81 | 82 | cat<namelist.input 83 | &time_control 84 | run_days = $days1, 85 | run_hours = $hrs1, 86 | run_minutes = 0, 87 | run_seconds = 0, 88 | start_year = $sy, 89 | start_month = $sm, 90 | start_day = $sd, 91 | start_hour = $sh1, 92 | start_minute = 00, 93 | start_second = 00, 94 | end_year = $ey, 95 | end_month = $em, 96 | end_day = $ed, 97 | end_hour = $eh1, 98 | end_minute = 00, 99 | end_second = 00, 100 | interval_seconds = $bc1, 101 | input_from_file = .true., 102 | fine_input_stream = 2, 103 | history_interval = 60, 104 | frames_per_outfile = 1, 105 | restart = .false., 106 | restart_interval = 15000, 107 | io_form_history = 2, 108 | io_form_restart = 2, 109 | io_form_input = 2, 110 | io_form_boundary = 2, 111 | io_form_auxinput2 = 2, 112 | debug_level = 0 113 | / 114 | 115 | &domains 116 | time_step = $t1, 117 | time_step_fract_num = 0, 118 | time_step_fract_den = 1, 119 | max_dom = $ndom, 120 | e_we = $nx1, 121 | e_sn = $ny1, 122 | e_vert = $nz1, 123 | p_top_requested = 5000, 124 | num_metgrid_levels = $metlev, 125 | num_metgrid_soil_levels = 4, 126 | dx = $dx1, 127 | dy = $dx1, 128 | grid_id = 1, 129 | parent_id = 1, 130 | i_parent_start = 1, 131 | j_parent_start = 1, 132 | parent_grid_ratio = 1, 133 | parent_time_step_ratio = 1, 134 | use_adaptive_time_step = .false., 135 | step_to_output_time = .true., 136 | target_cfl = 1.2, 137 | max_step_increase_pct = 10, 138 | starting_time_step = -1, 139 | max_time_step = 120, 140 | min_time_step = -1, 141 | adaptation_domain = 3, 142 | feedback = $fdbck, 143 | numtiles = 8, 144 | smooth_option = $smooth, 145 | sfcp_to_sfcp = .true., 146 | / 147 | 148 | &physics 149 | mp_physics = 4, 150 | ra_lw_physics = 1, 151 | ra_sw_physics = 2, 152 | radt = 10, 153 | sf_sfclay_physics = 2, 154 | sf_surface_physics = 2, 155 | bl_pbl_physics = 2, 156 | bldt = 0, 157 | cu_physics = 5, 158 | cudt = 0, 159 | isfflx = 1, 160 | ifsnow = 0, 161 | icloud = 1, 162 | surface_input_source = 1, 163 | num_soil_layers = 4, 164 | maxiens = 1, 165 | maxens = 3, 166 | maxens2 = 3, 167 | maxens3 = 16, 168 | ensdim = 144, 169 | ghg_input = 1, 170 | / 171 | 172 | &fdda 173 | / 174 | 175 | &dynamics 176 | w_damping = 0, 177 | diff_opt = 1, 178 | km_opt = 4, 179 | diff_6th_opt = 0, 180 | diff_6th_factor = 0.12, 181 | base_temp = 290., 182 | damp_opt = 0, 183 | zdamp = 5000., 184 | dampcoef = 0.2, 185 | khdif = 0, 186 | kvdif = 0, 187 | non_hydrostatic = .true., 188 | moist_adv_opt = 1, 189 | scalar_adv_opt = 1, 190 | / 191 | 192 | &bdy_control 193 | spec_bdy_width = 5, 194 | spec_zone = 1, 195 | relax_zone = 4, 196 | specified = .true., 197 | periodic_x = .false., 198 | symmetric_xs = .false., 199 | symmetric_xe = .false., 200 | open_xs = .false., 201 | open_xe = .false., 202 | periodic_y = .false., 203 | symmetric_ys = .false., 204 | symmetric_ye = .false., 205 | open_ys = .false., 206 | open_ye = .false., 207 | nested = .false., 208 | / 209 | 210 | &grib2 211 | / 212 | 213 | &namelist_quilt 214 | nio_tasks_per_group = 0, 215 | nio_groups = 1, 216 | / 217 | EOF 218 | 219 | 220 | cd $WPSWORK 221 | rm -f GRIBFILE* 222 | 223 | cat<namelist.wps 224 | &share 225 | wrf_core = 'ARW', 226 | max_dom = $ndom, 227 | start_date = '$datenest', 228 | end_date = '$dateend', 229 | interval_seconds = $bc1, 230 | io_form_geogrid = 2, 231 | opt_output_from_geogrid_path = '$WPSWORK/', 232 | debug_level = 0, 233 | / 234 | 235 | &geogrid 236 | parent_id = 1, 237 | parent_grid_ratio = 1, 238 | i_parent_start = 1, 239 | j_parent_start = 1, 240 | e_we = $nx1, 241 | e_sn = $ny1, 242 | geog_data_res = '5m', 243 | dx = $dx1, 244 | dy = $dx1, 245 | map_proj = 'lambert', 246 | ref_lat = 40, 247 | ref_lon = 14, 248 | truelat1 = 40, 249 | truelat2 = 40, 250 | stand_lon = 14, 251 | geog_data_path = '$GEOG_DIR/', 252 | opt_geogrid_tbl_path = '$WPSWORK/' 253 | / 254 | 255 | &ungrib 256 | out_format = 'WPS' 257 | prefix = 'FILE' 258 | / 259 | 260 | &metgrid 261 | fg_name = './FILE', 262 | io_form_metgrid = 2, 263 | opt_output_from_metgrid_path = '$WPSWORK/', 264 | opt_metgrid_tbl_path = '$WPSWORK/', 265 | / 266 | 267 | 268 | &mod_levs 269 | press_pa = 201300 , 200100 , 100000 , 270 | 95000 , 90000 , 271 | 85000 , 80000 , 272 | 75000 , 70000 , 273 | 65000 , 60000 , 274 | 55000 , 50000 , 275 | 45000 , 40000 , 276 | 35000 , 30000 , 277 | 25000 , 20000 , 278 | 15000 , 10000 , 279 | 5000 , 1000 280 | / 281 | 282 | &domain_wizard 283 | grib_data_path = 'null', 284 | grib_vtable = 'null', 285 | dwiz_name =euro12 286 | dwiz_desc = 287 | dwiz_user_rect_x1 =915 288 | dwiz_user_rect_y1 =156 289 | dwiz_show_political =true 290 | dwiz_center_over_gmt =true 291 | dwiz_latlon_space_in_deg =10 292 | dwiz_latlon_linecolor =-8355712 293 | dwiz_map_scale_pct =25 294 | dwiz_map_vert_scrollbar_pos =0 295 | dwiz_map_horiz_scrollbar_pos =743 296 | dwiz_gridpt_dist_km =12.0 297 | dwiz_mpi_command = 298 | dwiz_tcvitals =null 299 | / 300 | EOF 301 | -------------------------------------------------------------------------------- /scripts/preprocess.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | ################################################################################### 3 | # Setup Environmen Variable 4 | ################################################################################### 5 | source /shared/setup_env.sh 6 | 7 | ulimit -s unlimited 8 | 9 | NP=$(ls /sys/class/cpuid/ | wc -l) 10 | 11 | NP=$(( $NP / 2 )) 12 | 13 | 14 | day=$(date +%Y%m%d) 15 | 16 | WPSWORK=${TARGET_DIR}/preproc 17 | WRFWORK=${TARGET_DIR}/run 18 | 19 | DIRGFS=${SHARED_DIR}/FORECAST/download/$day 20 | ################################################################################### 21 | # Print log function 22 | ################################################################################### 23 | log () 24 | { 25 | timestamp=`date "+%Y.%m.%d-%H:%M:%S %Z"` 26 | echo "$timestamp $*" 27 | } 28 | 29 | ################################################################################### 30 | # Start Preprocessing 31 | ################################################################################### 32 | cd $WPSWORK 33 | log "INFO - Starting geogrid.exe" 34 | ./geogrid.exe > geogrid.$day.log 2>&1 35 | if [ $? -ne 0 ] 36 | then 37 | log "CRIT - geogrid.exe Completed with errors." 38 | exit 1 39 | fi 40 | log "INFO - geogrid.exe Completed" 41 | 42 | cd $DIRGFS 43 | cp -f GRIBFILE* $WPSWORK 44 | cd $WPSWORK 45 | 46 | 47 | rm -f FILE* 48 | rm -f PFILE* 49 | 50 | 51 | log "INFO - Starting ungrib.exe" 52 | ./ungrib.exe > ungrib.$day.log 2>&1 53 | if [ $? -ne 0 ] 54 | then 55 | log "CRIT - ungrib.exe Completed with errors." 56 | exit 1 57 | fi 58 | 59 | log "INFO - ungrib.exe Completed" 60 | 61 | rm -f met_em* 62 | 63 | log "INFO - Starting metgrid.exe" 64 | #mpirun -np $NP ./metgrid.exe > metgrid.$day.log 2>&1 65 | ./metgrid.exe > metgrid.$day.log 2>&1 66 | if [ $? -ne 0 ] 67 | then 68 | log "CRIT - metgrid.exe Completed with errors." 69 | exit 1 70 | fi 71 | log "INFO - metgrid.exe Completed" 72 | 73 | 74 | mv met_em* $WRFWORK 75 | 76 | ################################################################################### 77 | # Start WRF Processing 78 | ################################################################################### 79 | 80 | cd $WRFWORK 81 | 82 | rm -f rsl.* 83 | rm -f wrfinput* 84 | rm -f wrfbdy* 85 | rm -f wrfout* 86 | 87 | 88 | log "INFO - Starting real.exe" 89 | #mpirun -np $NP ./real.exe >real.$day.log 2>&1 90 | ./real.exe >real.$day.log 2>&1 91 | if [ $? -ne 0 ] 92 | then 93 | log "CRIT - real.exe Completed with errors." 94 | exit 1 95 | fi 96 | log "INFO - real.exe Completed" 97 | 98 | 99 | 100 | exit 0 101 | -------------------------------------------------------------------------------- /scripts/run_wrf.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | source /shared/setup_env.sh 3 | cd /shared/FORECAST/domains/test/run 4 | echo "#################################" 5 | env 6 | echo "#################################" 7 | cat $PE_HOSTFILE 8 | echo "#################################" 9 | for host in $(cat $PE_HOSTFILE | awk '{print $1}') 10 | do 11 | hostlist="$hostlist,$host" 12 | done 13 | echo $hostlist 14 | echo "#################################" 15 | 16 | #Each process uses 2 threads and only phisical cores 17 | let PROCESS=NSLOTS/2 18 | PPN=18 #c5n.18xlarge 19 | unset PE_HOSTFILE 20 | export OMP_NUM_THREADS=2 21 | echo "Processes $PROCESS" 22 | time mpirun -np $PROCESS -host $hostlist -ppn 18 ./wrf.exe 23 | -------------------------------------------------------------------------------- /scripts/slurm_run_wrf.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | #SBATCH --error=job.err 3 | #SBATCH --output=job.out 4 | #SBATCH --time=24:00:00 5 | #SBATCH --job-name=wrf 6 | #SBATCH --nodes=4 7 | #SBATCH --ntasks-per-node=63 8 | #SBATCH --cpus-per-task=1 9 | 10 | cd /shared/FORECAST/domains/test/run 11 | mpirun ./wrf.exe 12 | -------------------------------------------------------------------------------- /setup.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash -i 2 | . /etc/parallelcluster/cfnconfig 3 | 4 | shared_folder=$(echo $cfn_ebs_shared_dirs | cut -d ',' -f 1 ) 5 | 6 | 7 | function create_env_file { 8 | echo "Create Env" 9 | cat <<@EOF >${shared_folder}/gcc_setup_env.sh 10 | #!/bin/bash 11 | 12 | if [ ! -z \$WRF_ENV ] 13 | then 14 | echo "ENV already loadad" 15 | return 16 | fi 17 | 18 | 19 | export SHARED_DIR=${shared_folder} 20 | export SETUP_DIR=${shared_folder}/hpc-workshop-wrf 21 | export BUILDDIR=${shared_folder}/build/WRF 22 | export DIR=${shared_folder}/WRF 23 | export SCRIPTDIR=${shared_folder}/WRF/bin 24 | export CC=gcc 25 | export CXX=g++ 26 | export FC=gfortran 27 | export F77=gfortran 28 | export PATH=${shared_folder}/WRF/bin:\$PATH 29 | export TARGET_DIR=\${SHARED_DIR}/FORECAST/domains/test/ 30 | export GEOG_BASE_DIR=\${SHARED_DIR}/FORECAST/domains/ 31 | 32 | export OMP_NUM_THREADS=2 33 | 34 | 35 | @EOF 36 | 37 | chmod 777 ${shared_folder} 38 | chmod 777 ${shared_folder}/gcc_setup_env.sh 39 | rm -f ${shared_folder}/setup_env.sh 40 | ln -s ${shared_folder}/gcc_setup_env.sh ${shared_folder}/setup_env.sh 41 | 42 | mkdir ${shared_folder}/download 43 | chmod 777 ${shared_folder}/download 44 | mkdir -p ${shared_folder}/WRF/bin 45 | chmod -R 777 ${shared_folder}/WRF 46 | } 47 | 48 | echo "NODE TYPE: ${cfn_node_type}" 49 | 50 | case ${cfn_node_type} in 51 | HeadNode) 52 | echo "I am the HeadNode node" 53 | create_env_file 54 | source ${shared_folder}/setup_env.sh 55 | cd ${shared_folder} 56 | bash ${shared_folder}/hpc-workshop-wrf/pc_setup_scripts/pcluster_install_spack.sh 57 | 58 | ;; 59 | ComputeFleet) 60 | echo "I am a Compute node" 61 | ;; 62 | esac 63 | 64 | # Set ulimits according to WRF needs 65 | cat >>/tmp/limits.conf << EOF 66 | # core file size (blocks, -c) 0 67 | * hard core 0 68 | * soft core 0 69 | 70 | # data seg size (kbytes, -d) unlimited 71 | * hard data unlimited 72 | * soft data unlimited 73 | 74 | # scheduling priority (-e) 0 75 | * hard priority 0 76 | * soft priority 0 77 | 78 | # file size (blocks, -f) unlimited 79 | * hard fsize unlimited 80 | * soft fsize unlimited 81 | 82 | # pending signals (-i) 256273 83 | * hard sigpending 1015390 84 | * soft sigpending 1015390 85 | 86 | # max locked memory (kbytes, -l) unlimited 87 | * hard memlock unlimited 88 | * soft memlock unlimited 89 | 90 | # open files (-n) 1024 91 | * hard nofile 65536 92 | * soft nofile 65536 93 | 94 | # POSIX message queues (bytes, -q) 819200 95 | * hard msgqueue 819200 96 | * soft msgqueue 819200 97 | 98 | # real-time priority (-r) 0 99 | * hard rtprio 0 100 | * soft rtprio 0 101 | 102 | # stack size (kbytes, -s) unlimited 103 | * hard stack unlimited 104 | * soft stack unlimited 105 | 106 | # cpu time (seconds, -t) unlimited 107 | * hard cpu unlimited 108 | * soft cpu unlimited 109 | 110 | # max user processes (-u) 1024 111 | * soft nproc 16384 112 | * hard nproc 16384 113 | 114 | # file locks (-x) unlimited 115 | * hard locks unlimited 116 | * soft locks unlimited 117 | EOF 118 | 119 | sudo bash -c 'cat /tmp/limits.conf > /etc/security/limits.conf' 120 | exit 0 -------------------------------------------------------------------------------- /wrf_setup_scripts/build_dir.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | source /shared/setup_env.sh 3 | 4 | #Copy WRF Files and links 5 | mkdir -p ${TARGET_DIR} 6 | mkdir -p ${TARGET_DIR}/log 7 | 8 | cp -a ${WRF_DIR}/run ${TARGET_DIR} 9 | cd ${TARGET_DIR}/run 10 | rm ndown.exe real.exe tc.exe wrf.exe MPTABLE.TBL 11 | ln -s ${WRF_DIR}/main/ndown.exe ndown.exe 12 | ln -s ${WRF_DIR}/main/real.exe real.exe 13 | ln -s ${WRF_DIR}/main/tc.exe tc.exe 14 | ln -s ${WRF_DIR}/main/wrf.exe wrf.exe 15 | ln -s ${WRF_DIR}/phys/noahmp/parameters/MPTABLE.TBL MPTABLE.TBL 16 | 17 | #Copy WPS Files and links 18 | mkdir ${TARGET_DIR}/preproc 19 | cd ${TARGET_DIR}/preproc 20 | ln -s ${WPS_DIR}/geogrid.exe geogrid.exe 21 | ln -s ${WPS_DIR}/geogrid/GEOGRID.TBL GEOGRID.TBL 22 | ln -s ${WPS_DIR}/metgrid.exe metgrid.exe 23 | ln -s ${WPS_DIR}/metgrid/METGRID.TBL METGRID.TBL 24 | ln -s ${WPS_DIR}/ungrib.exe ungrib.exe 25 | ln -s ${WPS_DIR}/ungrib/Variable_Tables/Vtable.GFS Vtable 26 | 27 | mkdir -p ${SHARED_DIR}/FORECAST/download 28 | 29 | cp -R ${SETUP_DIR}/scripts/* ${DIR}/bin 30 | chmod -R 775 ${DIR}/bin 31 | chmod -R 777 ${SHARED_DIR}/FORECAST 32 | 33 | -------------------------------------------------------------------------------- /wrf_setup_scripts/compile_and_install_using_spack.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | 4 | source /shared/setup_env.sh 5 | 6 | 7 | ARCH=$(uname -m) 8 | #Build packages 9 | if [ $ARCH = "x86_64" ] 10 | then 11 | time spack install wps %intel ^intel-oneapi-mpi #This is going to install WPS and WRF 12 | time spack install ncview%intel 13 | time spack install wgrib2%intel 14 | ARCH_CONFIG=spack load wgrib2%intel 15 | elif [ $ARCH = "aarch64" ] 16 | then 17 | #Remove mirrors.yaml because of errors during build 18 | mv ${SHARED_DIR}/spack/etc/spack/mirrors.yaml ${SHARED_DIR}/spack/etc/spack/mirrors.yaml.old 19 | touch ${SHARED_DIR}/spack/etc/spack/mirrors.yaml 20 | 21 | #Update packages.yaml to avoid rebuildling already installed componentis 22 | sudo pip3 install pyyaml 23 | mv ${SHARED_DIR}/spack/etc/spack/packages.yaml ${SHARED_DIR}/spack/etc/spack/packages.yaml.old 24 | python3 ${SHARED_DIR}/hpc-workshop-wrf/wrf_setup_scripts/update_packages_yaml.py ${SHARED_DIR}/spack/etc/spack/packages.yaml.old > ${SHARED_DIR}/spack/etc/spack/packages.yaml 25 | 26 | spack install wps #This is going to install WPS and WRF 27 | spack install ncview 28 | 29 | #spack install wgrib2 30 | #Manual install wgrib2 because of spack open issue on Graviton2 31 | mkdir -p ${BUILDDIR} 32 | cd ${SHARED_DIR}/download 33 | wget ftp://ftp.cpc.ncep.noaa.gov/wd51we/wgrib2/wgrib2.tgz.v3.1.0 -O wgrib2.tgz 34 | ################## COMPILE wgrib2 ################################################ 35 | cd $BUILDDIR 36 | tar -zxvf /shared/download/wgrib2.tgz 37 | cd grib2 38 | bash <<@EOF 39 | export FC=gfortran 40 | export COMP_SYS=gnu_linux 41 | export NETCDF=\$(spack location -i netcdf-c) 42 | export JASPERDIR=\$(spack location -i jasper) 43 | export CPPFLAGS="-I\$NETCDF/include/ -I\$JASPERDIR/include -L\$NETCDF/lib/ -L\$JASPERDIR/lib64/" 44 | echo "makefile < ${SETUP_DIR}/wgrib_makefile.patch" 45 | patch --fuzz 3 makefile < ${SETUP_DIR}/wrf_setup_scripts/wgrib_makefile.patch 46 | make 47 | cp wgrib2/wgrib2 $DIR/bin 48 | ARCH_CONFIG="" 49 | @EOF 50 | else 51 | echo "Unsupported Architecture $ARCH" 52 | exit 1 53 | fi 54 | 55 | 56 | cat <<@EOF >> /shared/setup_env.sh 57 | spack load wrf 58 | spack load wps 59 | spack load ncview 60 | 61 | $ARCH_CONFIG 62 | 63 | export WRF_DIR=\$(spack location -i wrf) 64 | export WPS_DIR=\$(spack location -i wps) 65 | export NETCDF=\$(spack location -i netcdf-c) 66 | export JASPER=\$(spack location -i jasper) 67 | 68 | export NETCDFRLIB=\$NETCDF/lib 69 | export NETCDFINC=\$NETCDF/include 70 | export JASPERLIB=\$JASPER/lib64 71 | export JASPERINC=\$JASPER/include 72 | export LD_LIBRARY_PATH=\${NETCDFRLIB}:\${JASPERLIB}:\$LD_LIBRARY_PATH 73 | 74 | export WRF_ENV=true 75 | @EOF 76 | 77 | -------------------------------------------------------------------------------- /wrf_setup_scripts/download_and_install_geog_data.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | source /shared/setup_env.sh 3 | 4 | cd ${SHARED_DIR}/download 5 | wget --no-check-certificate https://www2.mmm.ucar.edu/wrf/src/wps_files/geog_complete.tar.gz -q 6 | wget --no-check-certificate https://www2.mmm.ucar.edu/wrf/src/wps_files/albedo_modis.tar.bz2 -q 7 | wget --no-check-certificate https://www2.mmm.ucar.edu/wrf/src/wps_files/maxsnowalb_modis.tar.bz2 -q 8 | 9 | #Copy geog data 10 | mkdir -p ${GEOG_BASE_DIR} 11 | cd ${GEOG_BASE_DIR} 12 | tar -zxf ${SHARED_DIR}/download/geog_complete.tar.gz 13 | cd ${GEOG_BASE_DIR}/geog 14 | bzip2 -dc ${SHARED_DIR}/download/albedo_modis.tar.bz2 | tar -xf - 15 | bzip2 -dc ${SHARED_DIR}/download/maxsnowalb_modis.tar.bz2 | tar -xf - -------------------------------------------------------------------------------- /wrf_setup_scripts/install_wrf.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | source /shared/setup_env.sh 3 | 4 | cd ${SHARED_DIR}/hpc-workshop-wrf/wrf_setup_scripts 5 | 6 | bash download_and_install_geog_data.sh & 7 | 8 | bash compile_and_install_using_spack.sh 9 | 10 | bash build_dir.sh 11 | -------------------------------------------------------------------------------- /wrf_setup_scripts/update_packages_yaml.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python3 2 | import sys 3 | import yaml 4 | from yaml import Loader 5 | 6 | def main(argv): 7 | if len(argv) != 1: 8 | print ("Usage\n update_packages_yaml.py input-file") 9 | exit (1) 10 | 11 | input_file_name=argv[0] 12 | with open (input_file_name,"r") as F: 13 | data = yaml.safe_load(F.read()) 14 | for package in data['packages']: 15 | if package != 'all': 16 | if 'externals' in data['packages'][package]: 17 | data['packages'][package]['externals'][0]['buildable']= False 18 | 19 | #Add openMPI, zlib and perl to already installed packages to reduce compile time 20 | data['packages']['openmpi']={'externals':[{'spec':'openmpi@4.1.4', 'prefix': '/opt/amazon/openmpi','buildable': False}]} 21 | data['packages']['perl']={'externals':[{'spec':'perl@5.16.3', 'prefix': '/usr','buildable': False}]} 22 | data['packages']['bzip2']={'externals':[{'spec':'bzip2@1.0.6', 'prefix': '/usr','buildable': False}]} 23 | data['packages']['zlib']={'externals':[{'spec':'zlib@1.2.7', 'prefix': '/usr','buildable': False}]} 24 | data['packages']['xz']={'externals':[{'spec':'xz@5.2.2', 'prefix': '/usr','buildable': False}]} 25 | data['packages']['krb5']={'externals':[{'spec':'krb5@1.15.1', 'prefix': '/usr','buildable': False}]} 26 | data['packages']['bison']={'externals':[{'spec':'bison@3.0.4', 'prefix': '/usr','buildable': False}]} 27 | data['packages']['tcsh']={'externals':[{'spec':'tcsh@6.18.1', 'prefix': '/usr','buildable': False}]} 28 | data['packages']['pkgconf']={'externals':[{'spec':'pkgconf@0.27.1', 'prefix': '/usr','buildable': False}]} 29 | data['packages']['time']={'externals':[{'spec':'time@1.7.45', 'prefix': '/usr','buildable': False}]} 30 | data['packages']['libpng']={'externals':[{'spec':'libpng@1.2.50', 'prefix': '/usr','buildable': False}]} 31 | data['packages']['ncurses']={'externals':[{'spec':'ncurses@6.0.8', 'prefix': '/usr','buildable': False}]} 32 | data['packages']['m4']={'externals':[{'spec':'m4@1.4.16', 'prefix': '/usr','buildable': False}]} 33 | data['packages']['libxml']={'externals':[{'spec':'libxml@2.9.1', 'prefix': '/usr','buildable': False}]} 34 | data['packages']['cmake']={'externals':[{'spec':'cmake@2.8.12', 'prefix': '/usr','buildable': False}]} 35 | data['packages']['bison']={'externals':[{'spec':'bison@3.0.4', 'prefix': '/usr','buildable': False}]} 36 | data['packages']['libtool']={'externals':[{'spec':'libtool@2.4.2', 'prefix': '/usr','buildable': False}]} 37 | #data['packages']['gettext']={'externals':[{'spec':'gettext@0.19.8', 'prefix': '/usr','buildable': False}]} 38 | 39 | 40 | print (yaml.dump(data)) 41 | 42 | if __name__ == "__main__": 43 | main(sys.argv[1:]) 44 | -------------------------------------------------------------------------------- /wrf_setup_scripts/wgrib_makefile.patch: -------------------------------------------------------------------------------- 1 | --- makefile.orig 2022-05-08 14:26:27.769486628 +0000 2 | +++ makefile 2022-05-08 14:27:57.117228480 +0000 3 | @@ -829,7 +829,7 @@ 4 | all: ${netcdf4src} ${hdf5src} ${prog} aux_progs/gmerge aux_progs/smallest_grib2 aux_progs/smallest_4 5 | 6 | 7 | -${prog}: $w/*.c $w/*.h ${jlib} ${aeclib} ${netcdf3lib} ${pnglib} ${hdf5lib} ${g2clib} ${netcdf4lib} ${iplib} ${spectrallib} ${gctpclib} ${proj4lib} ${ojlib} 8 | +${prog}: $w/*.c $w/*.h ${aeclib} ${pnglib} ${g2clib} ${iplib} ${spectrallib} ${gctpclib} ${proj4lib} ${ojlib} 9 | cd "$w" && export LDFLAGS="${wLDFLAGS}" && export CPPFLAGS="${wCPPFLAGS}" && ${MAKE} 10 | 11 | fast: $w/*.c $w/*.h ${jlib} ${aeclib} ${netcdf3lib} ${pnglib} ${hdf5lib} ${g2clib} ${netcdf4lib} ${iplib} ${spectrallib} ${gctpclib} ${proj4lib} ${ojlib} --------------------------------------------------------------------------------