├── 00_Pre Setup.md ├── 01_OpenHPC PBS Pro Setup.md ├── 01_OpenHPC Slurm Setup.md ├── 02_OpenHPC Setting.md ├── 03_GPU Node Provision.md ├── 04_Operation Guide for OpenHPC Cluster.md ├── 05_PBS configuration.md ├── 05_Slurm configuration.md ├── 06_After Startup or Reboot.md ├── 07_Add Hard disk.md ├── 08_LDAP server.md ├── 09_OpenHPC Slurm Setup for Single Node.md ├── 10_AMBER20.md ├── 11_AMBER22.md ├── 12_CSDS-2023.2.md ├── 13_GROMACS-2023.2 ├── 14_AMBER24_RC4_test.md ├── 15_AMBER24.md ├── 16_GROMACS-2024.2.md ├── 17_AMBER25.md ├── README.md └── _config.yml /00_Pre Setup.md: -------------------------------------------------------------------------------- 1 | Please check MAC address for public and internal cards 2 | 3 |
4 | 5 | ## Information 6 |
7 | 8 | **Master**\ 9 | Hostname: master\ 10 | enp0s3: private(internal) network 192.168.1.254\ 11 | enp0s8: public network 192.168.11.6 12 | 13 | **Compute1**\ 14 | Hostname: c1\ 15 | enp0s3: private network 192.168.1.253 MAC 08:00:27:99:b3:4f 16 | 17 |
18 | 19 | ### Add master host 20 | ``` 21 | # vi /etc/hosts 22 | 23 | 192.168.1.254 master 24 | ``` 25 | 26 | ``` 27 | hostnamectl set-hostname master 28 | ``` 29 | 30 | ### Disable firewall 31 | ``` 32 | # systemctl disable firewalld 33 | # systemctl stop firewalld 34 | ``` 35 | 36 | ### Disable selinux 37 | ``` 38 | # vi /etc/selinux/config 39 | 40 | SELINUX=disabled 41 | ``` 42 | 43 | ### Reboot master node 44 | ``` 45 | # reboot 46 | ``` 47 | 48 | ### Update CentOS 49 | ``` 50 | # yum -y update 51 | ``` 52 | -------------------------------------------------------------------------------- /01_OpenHPC PBS Pro Setup.md: -------------------------------------------------------------------------------- 1 | Credit: Atip Peethong 2 | 3 | ### Install OpenHPC Repository 4 | ``` 5 | # yum -y install http://build.openhpc.community/OpenHPC:/1.3/CentOS_7/x86_64/ohpc-release-1.3-1.el7.x86_64.rpm 6 | ``` 7 | 8 | ### Install basic package for OpenHPC 9 | ``` 10 | # yum -y install ohpc-base 11 | # yum -y install ohpc-warewulf 12 | # yum -y install chrony 13 | ``` 14 | 15 | ### Setup Time Server 16 | ``` 17 | # systemctl enable ntpd.service 18 | # systemctl enable chronyd 19 | # echo "allow all" >> /etc/chrony.conf 20 | # systemctl restart chronyd 21 | ``` 22 | 23 | ### Install PBSPro 24 | ``` 25 | # yum -y install pbspro-server-ohpc 26 | ``` 27 | 28 | ### Detemine enp0s3 for internal interface 29 | ``` 30 | # perl -pi -e "s/device = eth1/device = enp0s3/" /etc/warewulf/provision.conf 31 | ``` 32 | 33 | ### Enable tftp service 34 | ``` 35 | # perl -pi -e "s/^\s+disable\s+= yes/ disable = no/" /etc/xinetd.d/tftp 36 | ``` 37 | 38 | ### Determine ip for enp0s3 39 | ``` 40 | # ifconfig enp0s3 192.168.1.254 netmask 255.255.255.0 up 41 | ``` 42 | 43 | ### Restart ans enable services 44 | ``` 45 | # systemctl restart xinetd 46 | # systemctl enable mariadb.service 47 | # systemctl restart mariadb 48 | # systemctl enable httpd.service 49 | # systemctl restart httpd 50 | # systemctl enable dhcpd.service 51 | ``` 52 | 53 | ### Determine image for compute node 54 | ``` 55 | # export CHROOT=/opt/ohpc/admin/images/centos7.5 56 | # wwmkchroot centos-7 $CHROOT 57 | ``` 58 | 59 | ### Install OpenHPC for compute node 60 | ``` 61 | # yum -y --installroot=$CHROOT install ohpc-base-compute 62 | ``` 63 | 64 | ### Copy resolv.conf to compute node 65 | ``` 66 | # cp -p /etc/resolv.conf $CHROOT/etc/resolv.conf 67 | ``` 68 | 69 | ### Add master host node to compute node 70 | ``` 71 | # vi $CHROOT/etc/hosts 72 | 73 | master 192.168.1.254 74 | ``` 75 | 76 | ### Install PBSPro for compute node 77 | ``` 78 | # yum -y --installroot=$CHROOT install pbspro-execution-ohpc 79 | # perl -pi -e "s/PBS_SERVER=\S+/PBS_SERVER=master/" $CHROOT/etc/pbs.conf 80 | # echo "PBS_LEAF_NAME=master" >> /etc/pbs.conf 81 | # perl -pi -e "s/\$clienthost \S+/\$clienthost master/" $CHROOT/var/spool/pbs/mom_priv/config 82 | # chroot $CHROOT opt/pbs/libexec/pbs_habitat 83 | # echo "\$usecp *:/home /home" >> $CHROOT/var/spool/pbs/mom_priv/config 84 | # chroot $CHROOT systemctl enable pbs 85 | ``` 86 | 87 | ### Install NTP for compute node 88 | ``` 89 | # yum -y --installroot=$CHROOT install ntp 90 | # yum -y --installroot=$CHROOT install chrony 91 | ``` 92 | 93 | ### Install kernel for compute node 94 | ``` 95 | # yum -y --installroot=$CHROOT install kernel 96 | ``` 97 | 98 | ### Install modules user enviroment for compute node and master node 99 | ``` 100 | # yum -y --installroot=$CHROOT install lmod-ohpc 101 | # yum -y install lmod-ohpc 102 | ``` 103 | 104 | ### Create basic values for OpenHPC 105 | ``` 106 | # wwinit database 107 | # wwinit ssh_keys 108 | ``` 109 | 110 | ### Create NFS Cilent (by mount /home and /opt/ohpc/pub from master node) 111 | ``` 112 | # echo "192.168.1.254:/home /home nfs nfsvers=3,nodev,nosuid,noatime 0 0" >> $CHROOT/etc/fstab 113 | # echo "192.168.1.254:/opt/ohpc/pub /opt/ohpc/pub nfs nfsvers=3,nodev,noatime 0 0" >> $CHROOT/etc/fstab 114 | # echo "192.168.1.254:/share /share nfs nfsvers=3,nodev,nosuid,noatime 0 0" >> $CHROOT/etc/fstab 115 | ``` 116 | 117 | ### Create NFS Server 118 | ``` 119 | # echo "/home *(rw,no_subtree_check,fsid=10,no_root_squash)" >> /etc/exports 120 | # echo "/opt/ohpc/pub *(ro,no_subtree_check,fsid=11)" >> /etc/exports 121 | # exportfs -a 122 | # systemctl restart nfs-server 123 | # systemctl enable nfs-server 124 | ``` 125 | 126 | ### Determine kernel setting for compute node 127 | ``` 128 | # chroot $CHROOT systemctl enable ntpd 129 | # chroot $CHROOT systemctl enable chrony 130 | # echo "server 192.168.1.254 iburst" >> $CHROOT/etc/ntp.conf 131 | # echo "server 192.168.1.254 iburst" >> $CHROOT/etc/chrony.conf 132 | ``` 133 | 134 | ### Determine memlock values 135 | ``` 136 | # perl -pi -e 's/# End of file/\* soft memlock unlimited\n$&/s' /etc/security/limits.conf 137 | # perl -pi -e 's/# End of file/\* hard memlock unlimited\n$&/s' /etc/security/limits.conf 138 | # perl -pi -e 's/# End of file/\* soft memlock unlimited\n$&/s' $CHROOT/etc/security/limits.conf 139 | # perl -pi -e 's/# End of file/\* hard memlock unlimited\n$&/s' $CHROOT/etc/security/limits.conf 140 | ``` 141 | 142 | ### Determine rsyslog for compute node by point to master node 143 | ``` 144 | # perl -pi -e "s/\\#\\\$ModLoad imudp/\\\$ModLoad imudp/" /etc/rsyslog.conf 145 | # perl -pi -e "s/\\#\\\$UDPServerRun 514/\\\$UDPServerRun 514/" /etc/rsyslog.conf 146 | # systemctl restart rsyslog 147 | # echo "*.* @192.168.1.254:514" >> $CHROOT/etc/rsyslog.conf 148 | # perl -pi -e "s/^\*\.info/\\#\*\.info/" $CHROOT/etc/rsyslog.conf 149 | # perl -pi -e "s/^authpriv/\\#authpriv/" $CHROOT/etc/rsyslog.conf 150 | # perl -pi -e "s/^mail/\\#mail/" $CHROOT/etc/rsyslog.conf 151 | # perl -pi -e "s/^cron/\\#cron/" $CHROOT/etc/rsyslog.conf 152 | # perl -pi -e "s/^uucp/\\#uucp/" $CHROOT/etc/rsyslog.conf 153 | ``` 154 | 155 | ### Install Ganggila for monitoring OpenHPC 156 | ``` 157 | # yum -y install ohpc-ganglia 158 | # yum -y --installroot=$CHROOT install ganglia-gmond-ohpc 159 | # cp /opt/ohpc/pub/examples/ganglia/gmond.conf /etc/ganglia/gmond.conf 160 | # perl -pi -e "s//master/" /etc/ganglia/gmond.conf 161 | # cp /etc/ganglia/gmond.conf $CHROOT/etc/ganglia/gmond.conf 162 | # echo "gridname MySite.." >> /etc/ganglia/gmetad.conf 163 | # systemctl enable gmond 164 | # systemctl enable gmetad 165 | # systemctl start gmond 166 | # systemctl start gmetad 167 | # chroot $CHROOT systemctl enable gmond 168 | # systemctl try-restart httpd 169 | ``` 170 | 171 | ### Install Clustershell 172 | (by adm: master ansd compute:${compute_prefix}[1-${num_computes}] by compute_prefix = c and num_computes =2) 173 | ``` 174 | # yum -y install clustershell-ohpc 175 | # cd /etc/clustershell/groups.d 176 | # mv local.cfg local.cfg.orig 177 | # echo "adm: master" > local.cfg 178 | # echo "compute: c[1-2]" >> local.cfg 179 | # echo "all: @adm,@compute" >> local.cfg 180 | ``` 181 | 182 | ### Import files using in OpenHPC 183 | ``` 184 | # wwsh file import /etc/passwd 185 | # wwsh file import /etc/group 186 | # wwsh file import /etc/shadow 187 | # wwsh file list 188 | ``` 189 | 190 | ### Determine bootstrap image 191 | ``` 192 | # export WW_CONF=/etc/warewulf/bootstrap.conf 193 | # echo "drivers += updates/kernel/" >> $WW_CONF 194 | # echo "drivers += overlay" >> $WW_CONF 195 | ``` 196 | 197 | ### Setup bootstrap image 198 | ``` 199 | # wwbootstrap `uname -r` 200 | ``` 201 | 202 | ### Create Virtual Node File System (VNFS) image 203 | ``` 204 | # wwvnfs --chroot $CHROOT 205 | ``` 206 | 207 | ### Determine compute node by MAC Address 208 | (by GATEWAYDEV=enp0s8 is Public interface // using in internal lan group) 209 | ``` 210 | # echo "GATEWAYDEV=enp0s8" > /tmp/network.$$ 211 | # wwsh -y file import /tmp/network.$$ --name network 212 | # wwsh -y file set network --path /etc/sysconfig/network --mode=0644 --uid=0 213 | # wwsh file list 214 | ``` 215 | 216 | We have 2 compute nodes 217 | ``` 218 | # wwsh -y node new c1 --ipaddr=192.168.1.253 --hwaddr=08:00:27:99:B3:4F -D enp0s8 219 | # wwsh -y node new c2 --ipaddr=192.168.1.252 --hwaddr=08:00:27:99:B3:5F -D enp0s8 220 | # wwsh node list 221 | ``` 222 | 223 | ### Determine VNFS for compute node 224 | ``` 225 | # wwsh -y provision set "c1" --vnfs=centos7.5 --bootstrap=`uname -r` --files=dynamic_hosts,passwd,group,shadow,network 226 | # wwsh -y provision set "c2" --vnfs=centos7.5 --bootstrap=`uname -r` --files=dynamic_hosts,passwd,group,shadow,network 227 | # wwsh provision list 228 | ``` 229 | 230 | ### Restart ganglia services 231 | ``` 232 | # systemctl restart gmond 233 | # systemctl restart gmetad 234 | # systemctl restart dhcpd 235 | # wwsh pxe update 236 | ``` 237 | 238 | ### Determine resource for pbs system 239 | ``` 240 | # systemctl enable pbs 241 | # systemctl start pbs 242 | # . /etc/profile.d/pbs.sh 243 | # qmgr -c "set server default_qsub_arguments= -V" 244 | # qmgr -c "set server resources_default.place=scatter" 245 | # qmgr -c "set server job_history_enable=True" 246 | # qmgr -c "create node c1" 247 | # qmgr -c "create node c2" 248 | # pbsnodes -a 249 | ``` 250 | 251 | ### Sync all file to compute node 252 | ``` 253 | # wwsh file resync 254 | ``` 255 | 256 | ### Compute node installation (by booting from network with MAC Address) 257 | Open compute node and waiting for finish and you can check by: 258 | ``` 259 | # pdsh -w c1 uptime 260 | # pdsh -w c[1-2] uptime 261 | ``` 262 | -------------------------------------------------------------------------------- /01_OpenHPC Slurm Setup.md: -------------------------------------------------------------------------------- 1 | ### Install OpenHPC Repository 2 | ``` 3 | # yum -y install http://build.openhpc.community/OpenHPC:/1.3/CentOS_7/x86_64/ohpc-release-1.3-1.el7.x86_64.rpm 4 | ``` 5 | 6 | ### Install basic package for OpenHPC 7 | ``` 8 | # yum -y install ohpc-base 9 | # yum -y install ohpc-warewulf 10 | # yum -y install chrony 11 | ``` 12 | 13 | ### Setup Time Server 14 | ``` 15 | # systemctl enable ntpd.service 16 | # systemctl enable chronyd 17 | # echo "allow all" >> /etc/chrony.conf 18 | # systemctl restart chronyd 19 | ``` 20 | 21 | ### Install Slurm 22 | ``` 23 | # yum -y install ohpc-slurm-server 24 | # perl -pi -e "s/ControlMachine=\S+/ControlMachine=master/" /etc/slurm/slurm.conf 25 | ``` 26 | 27 | ### Determine enp0s3 for internal interface 28 | ``` 29 | # perl -pi -e "s/device = eth1/device = enp0s3/" /etc/warewulf/provision.conf 30 | ``` 31 | 32 | ### Enable tftp service 33 | ``` 34 | # perl -pi -e "s/^\s+disable\s+= yes/ disable = no/" /etc/xinetd.d/tftp 35 | ``` 36 | 37 | ### Determine ip for enp0s3 38 | ``` 39 | # ifconfig enp0s3 192.168.1.254 netmask 255.255.255.0 up 40 | ``` 41 | 42 | ### Restart ans enable services 43 | ``` 44 | # systemctl restart xinetd 45 | # systemctl enable mariadb.service # systemctl restart mariadb 46 | # systemctl enable httpd.service 47 | # systemctl restart httpd 48 | # systemctl enable dhcpd.service 49 | ``` 50 | 51 | ### Determine image for compute node 52 | ``` 53 | # export CHROOT=/opt/ohpc/admin/images/centos7.5 54 | # wwmkchroot centos-7 $CHROOT 55 | ``` 56 | 57 | ### Install OpenHPC for compute node 58 | ``` 59 | # yum -y --installroot=$CHROOT install ohpc-base-compute 60 | ``` 61 | 62 | ### Copy resolv.conf to compute node 63 | ``` 64 | # cp -p /etc/resolv.conf $CHROOT/etc/resolv.conf 65 | ``` 66 | 67 | ### Install slurm client in image compute node and master node 68 | ``` 69 | # yum -y --installroot=$CHROOT install ohpc-slurm-client 70 | # yum -y install ohpc-slurm-client 71 | ``` 72 | 73 | ### Add master host node to compute node 74 | ``` 75 | # vi $CHROOT/etc/hosts 76 | 77 | master 192.168.1.254 78 | ``` 79 | 80 | ### Install NTP for compute node 81 | ``` 82 | # yum -y --installroot=$CHROOT install ntp 83 | # yum -y --installroot=$CHROOT install chrony 84 | ``` 85 | 86 | ### Install kernel for compute node 87 | ``` 88 | # yum -y --installroot=$CHROOT install kernel 89 | ``` 90 | 91 | ### Install modules user enviroment for compute node and master node 92 | ``` 93 | # yum -y --installroot=$CHROOT install lmod-ohpc 94 | # yum -y install lmod-ohpc 95 | ``` 96 | 97 | ### Create basic values for OpenHPC 98 | ``` 99 | # wwinit database 100 | # wwinit ssh_keys 101 | ``` 102 | 103 | ### Create NFS Cilent 104 | (by mount /home and /opt/ohpc/pub from master node) 105 | ``` 106 | # echo "192.168.1.254:/home /home nfs nfsvers=3,nodev,nosuid,noatime 0 0" >> $CHROOT/etc/fstab 107 | # echo "192.168.1.254:/opt/ohpc/pub /opt/ohpc/pub nfs nfsvers=3,nodev,noatime 0 0" >> $CHROOT/etc/fstab 108 | # echo "192.168.1.254:/share /share nfs nfsvers=3,nodev,nosuid,noatime 0 0" >> $CHROOT/etc/fstab 109 | ``` 110 | 111 | ### Create NFS Server 112 | ``` 113 | # echo "/home *(rw,no_subtree_check,fsid=10,no_root_squash)" >> /etc/exports 114 | # echo "/opt/ohpc/pub *(ro,no_subtree_check,fsid=11)" >> /etc/exports 115 | # exportfs -a 116 | # systemctl restart nfs-server 117 | # systemctl enable nfs-server 118 | ``` 119 | 120 | ### Determine kernel setting for compute node 121 | ``` 122 | # chroot $CHROOT systemctl enable ntpd 123 | # chroot $CHROOT systemctl enable chrony 124 | # echo "server 192.168.1.254 iburst" >> $CHROOT/etc/ntp.conf 125 | # echo "server 192.168.1.254 iburst" >> $CHROOT/etc/chrony.conf 126 | ``` 127 | 128 | ### Update basic slurm configuration 129 | ``` 130 | # perl -pi -e "s/^NodeName=(\S+)/NodeName=c[1-2]/" /etc/slurm/slurm.conf 131 | # perl -pi -e "s/^PartitionName=normal Nodes=(\S+)/PartitionName=normal Nodes=c[1-2]/" /etc/slurm/slurm.conf 132 | # perl -pi -e "s/^Sockets=(\S+)/Sockets=1/" /etc/slurm/slurm.conf 133 | # perl -pi -e "s/^CoresPerSocket=(\S+)/CoresPerSocket=2/" /etc/slurm/slurm.conf 134 | # perl -pi -e "s/^ThreadsPerCore=(\S+)/ThreadsPerCore=1/" /etc/slurm/slurm.conf 135 | ``` 136 | 137 | ClusterName=galaxy.cluster\ 138 | SlurmctldHost=master 139 | 140 | ``` 141 | # perl -pi -e "s/^NodeName=(\S+)/NodeName=c[1-2]/" $CHROOT/etc/slurm/slurm.conf 142 | # perl -pi -e "s/^PartitionName=normal Nodes=(\S+)/PartitionName=normal Nodes=c[1-2]/" $CHROOT/etc/slurm/slurm.conf 143 | # perl -pi -e "s/^Sockets=(\S+)/Sockets=1/" $CHROOT/etc/slurm/slurm.conf 144 | # perl -pi -e "s/^CoresPerSocket=(\S+)/CoresPerSocket=2/" $CHROOT/etc/slurm/slurm.conf 145 | # perl -pi -e "s/^ThreadsPerCore=(\S+)/ThreadsPerCore=1/" $CHROOT/etc/slurm/slurm.conf 146 | ``` 147 | ClusterName=galaxy.cluster\ 148 | SlurmctldHost=master 149 | 150 | ``` 151 | # systemctl enable munge 152 | # systemctl enable slurmctld 153 | # systemctl start munge 154 | # systemctl start slurmctld 155 | # chroot $CHROOT systemctl enable slurmd 156 | ``` 157 | 158 | ### Determine memlock values 159 | ``` 160 | # perl -pi -e 's/# End of file/\* soft memlock unlimited\n$&/s' /etc/security/limits.conf 161 | # perl -pi -e 's/# End of file/\* hard memlock unlimited\n$&/s' /etc/security/limits.conf 162 | # perl -pi -e 's/# End of file/\* soft memlock unlimited\n$&/s' $CHROOT/etc/security/limits.conf 163 | # perl -pi -e 's/# End of file/\* hard memlock unlimited\n$&/s' $CHROOT/etc/security/limits.conf 164 | ``` 165 | 166 | ### Determine rsyslog for compute node by point to master node 167 | ``` 168 | # perl -pi -e "s/\\#\\\$ModLoad imudp/\\\$ModLoad imudp/" /etc/rsyslog.conf 169 | # perl -pi -e "s/\\#\\\$UDPServerRun 514/\\\$UDPServerRun 514/" /etc/rsyslog.conf 170 | # systemctl restart rsyslog 171 | # echo "*.* @192.168.1.254:514" >> $CHROOT/etc/rsyslog.conf 172 | # perl -pi -e "s/^\*\.info/\\#\*\.info/" $CHROOT/etc/rsyslog.conf 173 | # perl -pi -e "s/^authpriv/\\#authpriv/" $CHROOT/etc/rsyslog.conf 174 | # perl -pi -e "s/^mail/\\#mail/" $CHROOT/etc/rsyslog.conf 175 | # perl -pi -e "s/^cron/\\#cron/" $CHROOT/etc/rsyslog.conf 176 | # perl -pi -e "s/^uucp/\\#uucp/" $CHROOT/etc/rsyslog.conf 177 | ``` 178 | 179 | ### Install Ganggila for monitoring OpenHPC 180 | ``` 181 | # yum -y install ohpc-ganglia 182 | # yum -y --installroot=$CHROOT install ganglia-gmond-ohpc 183 | # cp /opt/ohpc/pub/examples/ganglia/gmond.conf /etc/ganglia/gmond.conf 184 | # perl -pi -e "s//master/" /etc/ganglia/gmond.conf 185 | # cp /etc/ganglia/gmond.conf $CHROOT/etc/ganglia/gmond.conf 186 | # echo "gridname MySite.." >> /etc/ganglia/gmetad.conf 187 | # systemctl enable gmond 188 | # systemctl enable gmetad 189 | # systemctl start gmond 190 | # systemctl start gmetad 191 | # chroot $CHROOT systemctl enable gmond 192 | # systemctl try-restart httpd 193 | ``` 194 | 195 | ### Install Clustershell 196 | (by adm: master ansd compute:${compute_prefix}[1-${num_computes}] by compute_prefix = c and num_computes =2) 197 | ``` 198 | # yum -y install clustershell-ohpc 199 | # cd /etc/clustershell/groups.d 200 | # mv local.cfg local.cfg.orig 201 | # echo "adm: master" > local.cfg 202 | # echo "compute: c[1-2]" >> local.cfg 203 | # echo "all: @adm,@compute" >> local.cfg 204 | ``` 205 | 206 | ### Import files using in OpenHPC 207 | ``` 208 | # wwsh file import /etc/passwd 209 | # wwsh file import /etc/group 210 | # wwsh file import /etc/shadow 211 | # wwsh file import /etc/slurm/slurm.conf 212 | # wwsh file import /etc/munge/munge.key 213 | # wwsh file list 214 | ``` 215 | 216 | ### Determine bootstrap image 217 | ``` 218 | # export WW_CONF=/etc/warewulf/bootstrap.conf 219 | # echo "drivers += updates/kernel/" >> $WW_CONF 220 | # echo "drivers += overlay" >> $WW_CONF 221 | ``` 222 | 223 | ### Setup bootstrap image 224 | ``` 225 | # wwbootstrap `uname -r` 226 | ``` 227 | 228 | ### Create Virtual Node File System (VNFS) image 229 | ``` 230 | # wwvnfs --chroot $CHROOT 231 | ``` 232 | 233 | ### Determine compute node by MAC Address 234 | (by GATEWAYDEV=enp0s8 is Public interface // using in internal lan group) 235 | ``` 236 | # echo "GATEWAYDEV=enp0s8" > /tmp/network.$$ 237 | # wwsh -y file import /tmp/network.$$ --name network 238 | # wwsh -y file set network --path /etc/sysconfig/network --mode=0644 --uid=0 239 | # wwsh file list 240 | ``` 241 | 242 | We have 2 compute nodes 243 | ``` 244 | # wwsh -y node new c1 --ipaddr=192.168.1.253 --hwaddr=08:00:27:99:B3:4F -D enp0s8 245 | # wwsh -y node new c2 --ipaddr=192.168.1.252 --hwaddr=08:00:27:99:B3:5F -D enp0s8 246 | # wwsh node list 247 | ``` 248 | 249 | ### Determine VNFS for compute node 250 | ``` 251 | # wwsh -y provision set "c1" --vnfs=centos7.5 --bootstrap=`uname -r` --files=dynamic_hosts,passwd,group,shadow,slurm.conf,munge.key,network 252 | # wwsh -y provision set "c2" --vnfs=centos7.5 --bootstrap=`uname -r` --files=dynamic_hosts,passwd,group,shadow,slurm.conf,munge.key,network 253 | # wwsh provision list 254 | ``` 255 | 256 | ### Restart ganglia services 257 | ``` 258 | # systemctl restart gmond 259 | # systemctl restart gmetad 260 | # systemctl restart dhcpd 261 | # wwsh pxe update 262 | ``` 263 | 264 | ### Sync all file to compute node 265 | ``` 266 | # wwsh file resync 267 | ``` 268 | 269 | ### Compute node installation (by booting from network with MAC Address) 270 | Open compute node and waiting for finish and you can check by: 271 | ``` 272 | # pdsh -w c1 uptime 273 | # pdsh -w c[1-2] uptime 274 | ``` 275 | 276 | ### Resource Manager Startup 277 | ``` 278 | # systemctl restart munge 279 | # systemctl restart slurmctld 280 | # pdsh -w c[1-2] systemctl restart slurmd 281 | ``` 282 | 283 | ### Test mung by 284 | ``` 285 | # munge -n | unmunge 286 | # munge -n | ssh c1 unmunge 287 | # munge -n | ssh c2 unmunge 288 | ``` 289 | 290 | ### Test slurm by 291 | ``` 292 | # systemctl status slurmctld 293 | # ssh c1 systemctl status slurmd 294 | # ssh c2 systemctl status slurmd 295 | ``` 296 | 297 | ### Test resource by 298 | ``` 299 | # scontrol show nodes 300 | ``` 301 | -------------------------------------------------------------------------------- /02_OpenHPC Setting.md: -------------------------------------------------------------------------------- 1 | ### Installation essential modules/softwares 2 | ``` 3 | # yum -y install openmpi3-gnu7-ohpc mpich-gnu7-ohpc lmod-defaults-gnu7-openmpi3-ohpc 4 | # yum -y install adios-gnu7-openmpi3-ohpc boost-gnu7-openmpi3-ohpc netcdf-gnu7-openmpi3-ohpc phdf5-gnu7-openmpi3-ohpc fftw-gnu7-openmpi3-ohpc hypre-gnu7-openmpi3-ohpc imp-gnu7-openmpi3-ohpc mfem-gnu7-openmpi3-ohpc mpiP-gnu7-openmpi3-ohpc mumps-gnu7-openmpi3-ohpc netcdf-cxx-gnu7-openmpi3-ohpc netcdf-fortran-gnu7-openmpi3-ohpc netcdf-gnu7-openmpi3-ohpc petsc-gnu7-openmpi3-ohpc pnetcdf-gnu7-openmpi3-ohpc scalapack-gnu7-openmpi3-ohpc scalasca-gnu7-openmpi3-ohpc scorep-gnu7-openmpi3-ohpc slepc-gnu7-openmpi3-ohpc superlu_dist-gnu7-openmpi3-ohpc tau-gnu7-openmpi3-ohpc trilinos-gnu7-openmpi3-ohpc singularity-ohpc hwloc-ohpc pmix-ohpc r-gnu7-ohpc hdf5-gnu7-ohpc mvapich2-gnu7-ohpc plasma-gnu7-ohpc scotch-gnu7-ohpc 5 | gnu8-compilers-ohpc intel-mpi-devel-ohpc openmpi3-gnu8-ohpc openmpi3-intel-ohpc python-mpi4py-gnu7-openmpi3-ohpc python-mpi4py-gnu8-openmpi3-ohpc python-mpi4py-intel-openmpi3-ohpc python34-mpi4py-gnu7-openmpi3-ohpc python34-mpi4py-gnu8-openmpi3-ohpc python34-mpi4py-intel-openmpi3-ohpc mpich-gnu8-ohpc mpich-intel-ohpc 6 | # yum install --installroot=$CHROOT lmod-defaults-gnu7-openmpi3-ohpc 7 | # yum -y install *gnu7-mpich* --exclude=lmod* 8 | # yum -y install *intel-*-ohpc --exclude=lmod* 9 | # yum -y install *-impi-* --exclude=lmod* 10 | # yum -y install *python*-ohpc 11 | ``` 12 | 13 | ### Copy lmod from compute node to master node 14 | ``` 15 | # cp -r /opt/ohpc/admin/images/centos7.5/opt/ohpc/admin/lmod/ /opt/ohpc/admin/ 16 | ``` 17 | 18 | ### Copy lmod.sh from $CHROOT/etc/profile.d to /etc/profile.d 19 | ``` 20 | # cp -r $CHROOT/etc/profile.d/lmod* /etc/profile.d 21 | ``` 22 | 23 | ### Change vnfs.conf for copy locale folder to compute node by add "#" 24 | ``` 25 | # vi /etc/warewulf/vnfs.conf 26 | 27 | #hybridize += /usr/share/locale 28 | #hybridize += /usr/lib/locale 29 | #hybridize += /usr/lib64/locale 30 | ``` 31 | 32 | ### Share library to compute nodes 33 | ``` 34 | # vi $CHROOT/etc/fstab 35 | 36 | 192.168.1.254:/usr/lib /usr/lib nfs defaults 0 0 37 | 192.168.1.254:/usr/lib64 /usr/lib64 nfs defaults 0 0 38 | 192.168.1.254:/etc/slurm /etc/slurm nfs defaults 0 0 39 | ``` 40 | ``` 41 | # vi /etc/exports 42 | 43 | #WWEXPORT:/usr/lib:192.168.1.0/255.255.255.0 44 | /usr/lib 192.168.1.0/255.255.255.0(ro,no_root_squash) 45 | 46 | #WWEXPORT:/usr/lib64:192.168.1.0/255.255.255.0 47 | /usr/lib64 192.168.1.0/255.255.255.0(ro,no_root_squash) 48 | 49 | #WWEXPORT:/etc/slurm:192.168.1.0/255.255.255.0 50 | /etc/slurm 192.168.1.0/255.255.255.0(ro,no_root_squash) 51 | ``` 52 | ``` 53 | # exportfs -a 54 | # systemctl restart nfs-server 55 | # systemctl enable nfs-server 56 | ``` 57 | -------------------------------------------------------------------------------- /03_GPU Node Provision.md: -------------------------------------------------------------------------------- 1 | Credit: https://github.com/dasandata/Open_HPC/blob/master/Provisioning/GPU%20Node%20Provisioning%20of%20OpenHPC%20Cluster.md 2 | 3 | ### Create and source ohpc_variable.sh 4 | ``` 5 | # vi ohpc_variable.sh 6 | 7 | export CHROOT=/opt/ohpc/admin/images/centos7.5 8 | ``` 9 | 10 | ### Install CUDA repository on master node 11 | ``` 12 | # curl -L -o cuda-repo-rhel7-10.0.130-1.x86_64.rpm http://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/cuda-repo-rhel7-10.0.130-1.x86_64.rpm 13 | # yum -y install cuda-repo-rhel7-10.0.130-1.x86_64.rpm 14 | ``` 15 | 16 | ### Copy epel-nvidia.repo to compute node 17 | ``` 18 | # yum -y install --installroot ${CHROOT} cuda-repo-rhel7-10.0.130-1.x86_64.rpm 19 | ``` 20 | 21 | ### Install libGLU.so libX11.so libXi.so libXmu.so to master node 22 | ``` 23 | # yum -y install libXi-devel mesa-libGLU-devel libXmu-devel libX11-devel freeglut-devel libXm* openmotif* 24 | ``` 25 | 26 | ### Install libGLU.so libX11.so libXi.so libXmu.so to compute node 27 | ``` 28 | # yum -y install --installroot=$CHROOT libXi-devel mesa-libGLU-devel libXmu-devel libX11-devel freeglut-devel libXm* openmotif* 29 | ``` 30 | 31 | ### Install nvidia-driver and cuda toolkit to master node 32 | ``` 33 | # yum -y install nvidia-driver nvidia-settings cuda nvidia-driver-cuda 34 | ``` 35 | 36 | ### Install nvidia-driver and cuda toolkit to compute node 37 | ``` 38 | # yum -y install --installroot=$CHROOT nvidia-driver nvidia-settings cuda nvidia-driver-cuda 39 | ``` 40 | 41 | ### Install cuda-cudnn 42 | First register here: https://developer.nvidia.com/developer-program/signup and Download cuda-cudnn tar.xz files from https://developer.nvidia.com/rdp/cudnn-archive 43 | ``` 44 | # tar -xvf cudnn-linux-x86_64-*****.tar.xz 45 | # mv cudnn-linux-x86_64-***** cuda 46 | # cp -P cuda/include/cudnn.h /usr/local/cuda-11.8/include 47 | # cp -P cuda/lib/libcudnn* /usr/local/cuda-11.8/lib64/ 48 | # chmod a+r /usr/local/cuda-11.8/lib64/libcudnn* 49 | ``` 50 | Nvidia device enable on boot (/dev/nvidia*) 51 | 52 | ``` 53 | # chroot $CHROOT 54 | ``` 55 | 56 | ``` 57 | # vi /etc/init.d/nvidia 58 | 59 | #!/bin/bash 60 | # 61 | # nvidia Set up NVIDIA GPU Compute Accelerators 62 | # 63 | # chkconfig: 2345 55 25 64 | # description: NVIDIA GPUs provide additional compute capability. \ 65 | # This service sets the GPUs into the desired state. 66 | # 67 | # config: /etc/sysconfig/nvidia 68 | 69 | ### BEGIN INIT INFO 70 | # Provides: nvidia 71 | # Required-Start: $local_fs $network $syslog 72 | # Required-Stop: $local_fs $syslog 73 | # Should-Start: $syslog 74 | # Should-Stop: $network $syslog 75 | # Default-Start: 2 3 4 5 76 | # Default-Stop: 0 1 6 77 | # Short-Description: Set GPUs into the desired state 78 | # Description: NVIDIA GPUs provide additional compute capability. 79 | # This service sets the GPUs into the desired state. 80 | ### END INIT INFO 81 | 82 | 83 | ################################################################################ 84 | ######################## Microway Cluster Management Software (MCMS) for OpenHPC 85 | ################################################################################ 86 | # 87 | # Copyright (c) 2015-2016 by Microway, Inc. 88 | # 89 | # This file is part of Microway Cluster Management Software (MCMS) for OpenHPC. 90 | # 91 | # MCMS for OpenHPC is free software: you can redistribute it and/or modify 92 | # it under the terms of the GNU General Public License as published by 93 | # the Free Software Foundation, either version 3 of the License, or 94 | # (at your option) any later version. 95 | # 96 | # MCMS for OpenHPC is distributed in the hope that it will be useful, 97 | # but WITHOUT ANY WARRANTY; without even the implied warranty of 98 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 99 | # GNU General Public License for more details. 100 | # 101 | # You should have received a copy of the GNU General Public License 102 | # along with MCMS. If not, see 103 | # 104 | ################################################################################ 105 | 106 | 107 | # source function library 108 | . /etc/rc.d/init.d/functions 109 | 110 | # Some definitions to make the below more readable 111 | NVSMI=/usr/bin/nvidia-smi 112 | NVCONFIG=/etc/sysconfig/nvidia 113 | prog="nvidia" 114 | 115 | # default settings 116 | NVIDIA_ACCOUNTING=1 117 | NVIDIA_PERSISTENCE_MODE=0 118 | NVIDIA_COMPUTE_MODE=1 119 | NVIDIA_CLOCK_SPEEDS=max 120 | # pull in sysconfig settings 121 | [ -f $NVCONFIG ] && . $NVCONFIG 122 | 123 | RETVAL=0 124 | 125 | 126 | # Determine the maximum graphics and memory clock speeds for each GPU. 127 | # Create an array of clock speed pairs (memory,graphics) to be passed to nvidia-smi 128 | declare -a MAX_CLOCK_SPEEDS 129 | get_max_clocks() 130 | { 131 | GPU_QUERY="$NVSMI --query-gpu=clocks.max.memory,clocks.max.graphics --format=csv,noheader,nounits" 132 | 133 | MAX_CLOCK_SPEEDS=( $($GPU_QUERY | awk '{print $1 $2}') ) 134 | } 135 | 136 | 137 | start() 138 | { 139 | /sbin/lspci | grep -qi nvidia 140 | if [ $? -ne 0 ] ; then 141 | echo -n $"No NVIDIA GPUs present. Skipping NVIDIA GPU tuning." 142 | warning 143 | echo 144 | exit 0 145 | fi 146 | 147 | echo -n $"Starting $prog: " 148 | 149 | # If the nvidia-smi utility is missing, this script can't do its job 150 | [ -x $NVSMI ] || exit 5 151 | 152 | # A configuration file is not required 153 | if [ ! -f $NVCONFIG ] ; then 154 | echo -n $"No GPU config file present ($NVCONFIG) - using defaults" 155 | echo 156 | fi 157 | 158 | # Set persistence mode first to speed things up 159 | echo -n "persistence" 160 | $NVSMI --persistence-mode=$NVIDIA_PERSISTENCE_MODE 1> /dev/null 161 | RETVAL=$? 162 | 163 | if [ ! $RETVAL -gt 0 ]; then 164 | echo -n " accounting" 165 | $NVSMI --accounting-mode=$NVIDIA_ACCOUNTING 1> /dev/null 166 | RETVAL=$? 167 | fi 168 | 169 | if [ ! $RETVAL -gt 0 ]; then 170 | echo -n " compute" 171 | $NVSMI --compute-mode=$NVIDIA_COMPUTE_MODE 1> /dev/null 172 | RETVAL=$? 173 | fi 174 | 175 | 176 | if [ ! $RETVAL -gt 0 ]; then 177 | echo -n " clocks" 178 | if [ -n "$NVIDIA_CLOCK_SPEEDS" ]; then 179 | # If the requested clock speed value is "max", 180 | # work through each GPU and set to max speed. 181 | if [ "$NVIDIA_CLOCK_SPEEDS" == "max" ]; then 182 | get_max_clocks 183 | 184 | GPU_COUNTER=0 185 | GPUS_SKIPPED=0 186 | while [ "$GPU_COUNTER" -lt ${#MAX_CLOCK_SPEEDS[*]} ] && [ ! $RETVAL -gt 0 ]; do 187 | if [[ ${MAX_CLOCK_SPEEDS[$GPU_COUNTER]} =~ Supported ]] ; then 188 | if [ $GPUS_SKIPPED -eq 0 ] ; then 189 | echo 190 | GPUS_SKIPPED=1 191 | fi 192 | echo "Skipping non-boostable GPU" 193 | else 194 | $NVSMI -i $GPU_COUNTER --applications-clocks=${MAX_CLOCK_SPEEDS[$GPU_COUNTER]} 1> /dev/null 195 | fi 196 | RETVAL=$? 197 | 198 | GPU_COUNTER=$(( $GPU_COUNTER + 1 )) 199 | done 200 | else 201 | # This sets all GPUs to the same clock speeds (which only works 202 | # if the GPUs in this system are all the same). 203 | $NVSMI --applications-clocks=$NVIDIA_CLOCK_SPEEDS 1> /dev/null 204 | fi 205 | else 206 | $NVSMI --reset-applications-clocks 1> /dev/null 207 | fi 208 | RETVAL=$? 209 | fi 210 | 211 | if [ ! $RETVAL -gt 0 ]; then 212 | if [ -n "$NVIDIA_POWER_LIMIT" ]; then 213 | echo -n " power-limit" 214 | $NVSMI --power-limit=$NVIDIA_POWER_LIMIT 1> /dev/null 215 | RETVAL=$? 216 | fi 217 | fi 218 | 219 | if [ ! $RETVAL -gt 0 ]; then 220 | success 221 | else 222 | failure 223 | fi 224 | echo 225 | return $RETVAL 226 | } 227 | 228 | stop() 229 | { 230 | /sbin/lspci | grep -qi nvidia 231 | if [ $? -ne 0 ] ; then 232 | echo -n $"No NVIDIA GPUs present. Skipping NVIDIA GPU tuning." 233 | warning 234 | echo 235 | exit 0 236 | fi 237 | 238 | echo -n $"Stopping $prog: " 239 | [ -x $NVSMI ] || exit 5 240 | 241 | $NVSMI --persistence-mode=0 1> /dev/null && success || failure 242 | RETVAL=$? 243 | echo 244 | return $RETVAL 245 | } 246 | 247 | restart() { 248 | stop 249 | start 250 | } 251 | 252 | force_reload() { 253 | restart 254 | } 255 | 256 | status() { 257 | $NVSMI 258 | } 259 | 260 | case "$1" in 261 | start) 262 | start 263 | ;; 264 | stop) 265 | stop 266 | ;; 267 | restart) 268 | restart 269 | ;; 270 | force-reload) 271 | force_reload 272 | ;; 273 | status) 274 | status 275 | RETVAL=$? 276 | ;; 277 | *) 278 | echo $"Usage: $0 {start|stop|restart|force-reload|status}" 279 | RETVAL=2 280 | esac 281 | exit $RETVAL 282 | ``` 283 | 284 | ``` 285 | # chmod +x /etc/init.d/nvidia 286 | # exit 287 | ``` 288 | 289 | ``` 290 | # vi /lib/systemd/system/nvidia-gpu.service 291 | 292 | [Unit] 293 | Description=NVIDIA GPU Initialization 294 | After=remote-fs.target 295 | 296 | [Service] 297 | Type=oneshot 298 | RemainAfterExit=yes 299 | ExecStart=/etc/init.d/nvidia start 300 | ExecStop=/etc/init.d/nvidia stop 301 | 302 | [Install] 303 | WantedBy=multi-user.target 304 | ``` 305 | 306 | ``` 307 | # chroot $CHROOT 308 | # systemctl enable nvidia-gpu.service 309 | ``` 310 | 311 | ### Update to Node nvfs image 312 | ``` 313 | # wwvnfs --chroot ${CHROOT} 314 | # wwsh vnfs list 315 | ``` 316 | 317 | ### Apply update imgae to nodes (rebooting) 318 | ``` 319 | # ssh node1 reboot 320 | ``` 321 | 322 | ### Download and install cuda-8.0, cuda-9.0, cuda-10.0 toolkits 323 | ``` 324 | # cd /root 325 | # wget https://developer.nvidia.com/compute/cuda/8.0/Prod2/local_installers/cuda_8.0.61_375.26_linux-run 326 | # mv cuda_8.0.61_375.26_linux-run cuda_8.0.61_375.26_linux.run 327 | # wget https://developer.nvidia.com/compute/cuda/9.0/Prod/local_installers/cuda_9.0.176_384.81_linux-run 328 | # mv cuda_9.0.176_384.81_linux-run cuda_9.0.176_384.81_linux.run 329 | # wget https://developer.nvidia.com/compute/cuda/10.0/Prod/local_installers/cuda_10.0.130_410.48_linux 330 | # mv cuda_10.0.130_410.48_linux cuda_10.0.130_410.48_linux.run 331 | # ./cuda_8.0.61_375.26_linux.run --silent --toolkit 332 | # ./cuda_9.0.176_384.81_linux.run --silent --toolkit 333 | # ./cuda_10.0.130_410.48.linux.run --silent --toolkit 334 | ``` 335 | 336 | ### Add Multiple Cuda Module for GPU Node 337 | ``` 338 | # cd /root 339 | # git clone https://github.com/dasandata/open_hpc 340 | # cd /root/open_hpc 341 | # git pull 342 | # mkdir -p /opt/ohpc/pub/modulefiles/cuda 343 | # cd 344 | ``` 345 | 346 | ### Add CUDA Module File by each version 347 | ``` 348 | # export GIT_CLONE_DIR=/root/open_hpc 349 | # export MODULES_DIR=/opt/ohpc/pub/modulefiles 350 | # for CUDA_VERSION in 8.0 9.0 10.0 ; do 351 | cp -a ${GIT_CLONE_DIR}/Module_Template/cuda.lua ${MODULES_DIR}/cuda/${CUDA_VERSION}.lua ; 352 | sed -i "s/{version}/${CUDA_VERSION}/" ${MODULES_DIR}/cuda/${CUDA_VERSION}.lua ; 353 | done 354 | ``` 355 | 356 | ### Refresh modules 357 | ``` 358 | # rm -rf ~/.lmod.d/.cache 359 | # module av 360 | ``` 361 | -------------------------------------------------------------------------------- /04_Operation Guide for OpenHPC Cluster.md: -------------------------------------------------------------------------------- 1 | Credit: https://github.com/dasandata/Open_HPC/blob/master/Opertation%20Guide%20for%20OpenHPC%20Cluster.md 2 | 3 | ### 1. Add user 4 | ``` 5 | # wwuseradd testuser 6 | # passwd testuser 7 | # chage -d 0 testuser 8 | # wwsh file resync 9 | # pdsh -w c[1-5] /warewulf/bin/wwgetfiles 10 | ``` 11 | 12 | ### 2. Delete user 13 | ``` 14 | # userdel testuser 15 | # rm -rf /home/testuser/ 16 | ``` 17 | 18 | ### 3. Install software on nodes 19 | 20 | 3.1 Install with direct to compute nodes 21 | ``` 22 | # pdsh -w node[1-2] 'yum install -y git' 23 | ``` 24 | 25 | 3.2 Install to compute node image 26 | ``` 27 | # yum --installroot=/opt/ohpc/admin/images/centos7.5 install -y git 28 | ``` 29 | 30 | 3.3 Mount compute node image 31 | ``` 32 | # wwvnfs --chroot /opt/ohpc/admin/images/centos7.5 33 | ``` 34 | 35 | ### 4. Add NFS Mount on nodes 36 | ``` 37 | # cat /etc/exports 38 | # echo "/DATA1 *(rw,no_subtree_check,no_root_squash)" >> /etc/exports 39 | # exportfs -a 40 | # pdsh -w node[1-2] "df -hT | grep nfs" 41 | # systemctl restart nfs-server.service 42 | # cat /etc/fstab 43 | # echo "master:/DATA1 /DATA1 nfs nfsvers=3 0 0" >> /etc/fstab 44 | # wwvnfs --chroot /opt/ohpc/admin/images/centos7.5 45 | # pdsh -w node[1-2] "reboot" 46 | # pdsh -w node[1-2] "df -hT | grep nfs" 47 | ``` 48 | 49 | ### 5. Module commands 50 | ``` 51 | # module --version 52 | # ml --version 53 | # module list 54 | # ml list 55 | # ml av 56 | ``` 57 | 58 | ### 6. Install Python 3.5.4 59 | ``` 60 | # yum -y install zlib-devel bzip2-devel sqlite sqlite-devel openssl-devel libffi-devel 61 | # cd /root 62 | # PYTHON_VERSION=3.5.4 63 | # wget https://www.python.org/ftp/python/${PYTHON_VERSION}/Python-${PYTHON_VERSION}.tgz 64 | # tar xvzf Python-${PYTHON_VERSION}.tgz 65 | # cd ~/Python-${PYTHON_VERSION} 66 | # ./configure --enable-optimizations --with-ensurepip=install --enable-shared --prefix=/opt/ohpc/pub/apps/python3/${PYTHON_VERSION} 67 | # make -j$(nproc) ; make install 68 | ``` 69 | 70 | ### 7. Add Python Module 71 | ``` 72 | # cd /root 73 | # PYTHON_VERSION=3.5.4 74 | # git clone https://github.com/dasandata/Open_HPC 75 | # cd /root/Open_HPC 76 | # git pull 77 | # cat /root/Open_HPC/Module_Template/python3.txt 78 | # echo ${PYTHON_VERSION} 79 | # mkdir /opt/ohpc/pub/modulefiles/python3 80 | # cp -a /root/Open_HPC/Module_Template/python3.txt /opt/ohpc/pub/modulefiles/python3/${PYTHON_VERSION} 81 | # sed -i "s/{VERSION}/${PYTHON_VERSION}/" /opt/ohpc/pub/modulefiles/python3/${PYTHON_VERSION} 82 | # cat /opt/ohpc/pub/modulefiles/python3/${PYTHON_VERSION} 83 | # rm -rf ~/.lmod.d/.cache 84 | # ml list 85 | # ml av 86 | # ml load python3/${PYTHON_VERSION} 87 | # ml list 88 | ``` 89 | 90 | ### 8. Add compute node 91 | ``` 92 | # wwsh -y node new c2 --ipaddr=192.168.1.253 --hwaddr=08:00:27:99:B3:4F -D enp0s8 93 | # wwsh node list 94 | # wwsh -y provision set "c2" --vnfs=centos7.5 --bootstrap=`uname -r` --files=dynamic_hosts,passwd,group,shadow,network 95 | # systemctl restart gmond 96 | # systemctl restart gmetad 97 | # systemctl restart dhcpd 98 | # wwsh pxe update 99 | ``` 100 | 101 | ``` 102 | # systemctl restart pbs 103 | # wwsh file resync 104 | # pdsh -w c2 uptime 105 | # cd /etc/clustershell/groups.d 106 | # vi local.cfg 107 | compute: c[1-2] 108 | ``` 109 | -------------------------------------------------------------------------------- /05_PBS configuration.md: -------------------------------------------------------------------------------- 1 | ### Add holiday times to /var/spool/pbs/sched_priv/holidays 2 | ``` 3 | # vi /var/spool/pbs/sched_priv/holidays 4 | 5 | * 6 | YEAR 2019 7 | * 8 | * Prime/Nonprime Table 9 | * 10 | * Prime Non-Prime 11 | * Day Start Start 12 | * 13 | weekday 0800 1730 14 | saturday none all 15 | sunday none all 16 | * 17 | * Day of Calendar Company 18 | * Year Date Holiday 19 | * 20 | 21 | * if a holiday falls on a saturday, it is observed on the friday before 22 | * if a holiday falls on a sunday, it is observed on the monday after 23 | 24 | * Jan 1 25 | 1 Jan 1 New Year's Day 26 | * Jan 2 27 | 2 Jan 2 Substitution Day for New Year's Day 28 | * Feb 19 29 | 50 Feb 19 Makha Bucha Day 30 | * Apr 8 31 | 98 Apr 8 Substitution Day for Chakri Memorial Day 32 | * Apr 15 33 | 105 Apr 15 Songkran Festival Day 34 | * May 20 35 | 140 May 20 Substitution Day for Visakha Bucha Day 36 | * Jul 16 37 | 184 Jul 16 Asalha Bucha Day 38 | * Jul 29 39 | 201 Jul 29 Substitution Day for His Majesty the King's Birthday 40 | * Aug 12 41 | 224 Aug 12 Her Majesty the Queen Mother's Birthday 42 | * Oct 14 43 | 287 Oct 14 Substitution Day for King Bhumibol Adulyadej Memorial Day 44 | * Oct 23 45 | 296 Oct 23 Chulalongkorn Memorial Day 46 | * Dec 5 47 | 339 Dec 5 His Majesty the late King Bhumibol Adulyadej's Birthday 48 | * Dec 25 49 | 359 Dec 25 Christmas Day 50 | * Dec 26 51 | 360 Dec 26 Year-End Holiday 52 | * Dec 27 53 | 361 Dec 27 Year-End Holiday 54 | * Dec 30 55 | 364 Dec 30 Year-End Holiday 56 | * Dec 31 57 | 365 Dec 31 Year-End Holiday 58 | ``` 59 | 60 | ### Add GPU resource 61 | ``` 62 | # qmgr 63 | create resource ngpus type=long, flag=nh 64 | create resource gpu_id type=string, flag=h 65 | exit 66 | ``` 67 | 68 | ### Edit /sched_config to add ngpus to the list of scheduling resources: 69 | ``` 70 | # vi /var/spool/pbs/sched_priv/sched_config 71 | resources: "ncpus, mem, arch, host, vnode, aoe, eoe, ngpus, gpu_id" 72 | ``` 73 | 74 | ``` 75 | # service pbs restart 76 | ``` 77 | 78 | ### Add vnode for cpu and gpu execution 79 | ``` 80 | # qmgr 81 | set node master resources_available.ncpus=0, resources_available.mem=0, resources_available.ngpus=0 82 | set node c1 resources_available.ncpus=0, resources_available.mem=0, resources_available.ngpus=0 83 | set node c2 resources_available.ncpus=0, resources_available.mem=0, resources_available.ngpus=0 84 | set node c3 resources_available.ncpus=0, resources_available.mem=0, resources_available.ngpus=0 85 | exit 86 | ``` 87 | 88 | ### Create vnode file for each vnode and scp to each host and ssh to each host 89 | ``` 90 | # ssh host 91 | # pbs_mom -s insert vnodefile vnodefile 92 | # service pbs restart 93 | # exit 94 | ``` 95 | 96 | **vnodefile-cpu** 97 | ``` 98 | $configversion 2 99 | master-cpu: Mom=master 100 | master-cpu: resources_available.ncpus=8 101 | master-cpu: resources_available.ngpus=0 102 | master-cpu: resources_available.host=master 103 | master-cpu: resources_available.vnode=master-cpu 104 | master-cpu: sharing=default_shared 105 | master-cpu: Priority=100 106 | ``` 107 | 108 | **vnodefile-gpu** 109 | ``` 110 | $configversion 2 111 | master-gpu0: Mom=master 112 | master-gpu0: resources_available.ncpus=1 113 | master-gpu0: resources_available.ngpus=1 114 | master-gpu0: resources_available.host=master 115 | master-gpu0: resources_available.vnode=master-gpu0 116 | master-gpu0: resources_available.gpu_id=gpu0 117 | master-gpu0: sharing=default_excl 118 | master-gpu0: Priority=10 119 | ``` 120 | 121 | ``` 122 | # service pbs restart 123 | ``` 124 | 125 | ### Add compute node to PBS pro 126 | ``` 127 | # qmgr 128 | create node master 129 | exit 130 | # service pbs restart 131 | ``` 132 | 133 | ### Add cpu queue 134 | ``` 135 | # qmgr 136 | create queue qcpu 137 | set queue qcpu queue_type = Execution 138 | set queue qcpu resources_default.nodect = 1 139 | set queue qcpu resources_default.nodes = 1:ppn=1 140 | set queue qcpu resources_default.ncpus = 1 141 | set queue qcpu resources_default.ngpus = 0 142 | set queue qcpu enabled = True 143 | set queue qcpu started = True 144 | exit 145 | # service pbs restart 146 | ``` 147 | 148 | ### Add gpu queue 149 | ``` 150 | # qmgr 151 | create queue qgpu 152 | set queue qgpu queue_type = Execution 153 | set queue qgpu resources_default.nodect = 1 154 | set queue qgpu resources_default.nodes = 1:ppn=1 155 | set queue qgpu resources_default.ngpus = 1 156 | set queue qgpu resources_default.ncpus = 1 157 | set queue qgpu enabled = True 158 | set queue qgpu started = True 159 | exit 160 | # service pbs restart 161 | ``` 162 | -------------------------------------------------------------------------------- /05_Slurm configuration.md: -------------------------------------------------------------------------------- 1 | ### Edit configuration file 2 | ``` 3 | # vi /etc/slurm/slurm.conf 4 | 5 | https://slurm.schedmd.com/configurator.html 6 | 7 | # 8 | # Example slurm.conf file. Please run configurator.html 9 | # (in doc/html) to build a configuration file customized 10 | # for your environment. 11 | # 12 | # 13 | # slurm.conf file generated by configurator.html. 14 | # 15 | # See the slurm.conf man page for more information. 16 | # 17 | ClusterName=galaxy.cluster 18 | SlurmctldHost=master 19 | #ControlMachine=master 20 | #ControlAddr=192.168.1.254 21 | #BackupController= 22 | #BackupAddr= 23 | # 24 | SlurmUser=slurm 25 | SlurmdUser=root 26 | SlurmctldPort=6817 27 | SlurmdPort=6818 28 | AuthType=auth/munge 29 | #JobCredentialPrivateKey= 30 | #JobCredentialPublicCertificate= 31 | StateSaveLocation=/var/spool/slurm/ctld 32 | SlurmdSpoolDir=/var/spool/slurm/d 33 | SwitchType=switch/none 34 | MpiDefault=none 35 | SlurmctldPidFile=/var/run/slurmctld.pid 36 | SlurmdPidFile=/var/run/slurmd.pid 37 | ProctrackType=proctrack/pgid 38 | #PluginDir= 39 | #FirstJobId= 40 | #MaxJobCount= 41 | #PlugStackConfig= 42 | #PropagatePrioProcess= 43 | #PropagateResourceLimits= 44 | #PropagateResourceLimitsExcept= 45 | #Prolog= 46 | #Epilog= 47 | #SrunProlog= 48 | #SrunEpilog= 49 | #TaskProlog= 50 | #TaskEpilog= 51 | TaskPlugin=task/none 52 | #TrackWCKey=no 53 | #TreeWidth=50 54 | #TmpFS= 55 | #UsePAM= 56 | # 57 | # TIMERS 58 | #BatchStartTimeout=10 59 | #CompleteWait=0 60 | #EpilogMsgTime=2000 61 | #GetEnvTimeout=2 62 | #HealthCheckInterval=0 63 | #HealthCheckProgram= 64 | InactiveLimit=0 65 | KillWait=30 66 | #MessageTimeout=10 67 | #ResvOverRun=0 68 | MinJobAge=300 69 | #OverTimeLimit=0 70 | SlurmctldTimeout=120 71 | SlurmdTimeout=300 72 | #UnkillableStepTimeout=60 73 | #VSizeFactor=0 74 | Waittime=0 75 | # 76 | # SCHEDULING 77 | SchedulerType=sched/backfill 78 | #SchedulerAuth= 79 | #SelectType=select/linear 80 | FastSchedule=1 81 | #SchedulerPort=7321 82 | SelectType=select/cons_res 83 | SelectTypeParameters=CR_CPU 84 | #PriorityType=priority/multifactor 85 | #PriorityDecayHalfLife=14-0 86 | #PriorityUsageResetPeriod=14-0 87 | #PriorityWeightFairshare=100000 88 | #PriorityWeightAge=1000 89 | #PriorityWeightPartition=10000 90 | #PriorityWeightJobSize=1000 91 | #PriorityMaxAge=1-0 92 | # 93 | # LOGGING 94 | SlurmctldDebug=3 95 | SlurmctldLogFile=/var/log/slurmctld.log 96 | SlurmdDebug=3 97 | SlurmdLogFile=/var/log/slurmd.log 98 | JobCompType=jobcomp/none 99 | #JobCompLoc= 100 | # 101 | # ACCOUNTING 102 | JobAcctGatherType=jobacct_gather/none 103 | #JobAcctGatherFrequency=30 104 | # 105 | AccountingStorageType=accounting_storage/slurmdbd 106 | AccountingStorageHost=master 107 | AccountingStoragePort=6819 108 | AccountingStoreJobComment=YES 109 | AccountingStorageLoc=slurm_acct_db 110 | AccountingStoragePass=/var/run/munge/munge.socket.2 111 | AccountingStorageUser=slurm 112 | # 113 | JobCompHost=master 114 | JobCompLoc=slurm_acct_db 115 | JobCompPass=slurm@1234 116 | #JobCompPort= 117 | #JobCompType=jobcomp/slurmdbdJobCompUser=slurm 118 | #JobContainerPlugin=job_container/none 119 | # 120 | # COMPUTE NODES 121 | # OpenHPC default configuration 122 | PropagateResourceLimitsExcept=MEMLOCK 123 | #Epilog=/etc/slurm/slurm.epilog.clean 124 | # 125 | GresTypes=gpu 126 | 127 | #NODES 128 | NodeName=master NodeAddr=master NodeHostName=master Gres=gpu:2 CPUs=8 CoresPerSocket=8 ThreadsPerCore=1 RealMemory=64000 State=UNKNOWN 129 | NodeName=c1 NodeAddr=c1 NodeHostName=c1 Gres=gpu:2 CPUs=12 CoresPerSocket=12 ThreadsPerCore=1 RealMemory=64000 State=UNKNOWN 130 | NodeName=c2 NodeAddr=c2 NodeHostName=c2 CPUs=16 CoresPerSocket=16 ThreadsPerCore=1 RealMemory=16000 State=UNKNOWN 131 | NodeName=c3 NodeAddr=c3 NodeHostName=c3 Gres=gpu:2 CPUs=12 CoresPerSocket=12 ThreadsPerCore=1 RealMemory=64000 State=UNKNOWN 132 | NodeName=c4 NodeAddr=c4 NodeHostName=c4 Gres=gpu:1 CPUs=16 CoresPerSocket=16 ThreadsPerCore=1 RealMemory=32000 State=UNKNOWN 133 | NodeName=c5 NodeAddr=c5 NodeHostName=c5 CPUs=16 CoresPerSocket=16 ThreadsPerCore=1 RealMemory=16000 State=UNKNOWN 134 | 135 | #PARTITIONS 136 | PartitionName=qcpu Nodes=c[1-5],master Default=YES State=UP MaxTime=120:00:00 DefaultTime=0:30:0 DefMemPerCPU=512 Shared=NO 137 | PartitionName=qgpu_gtx1070ti Nodes=master,c1 State=UP MaxTime=72:00:00 DefaultTime=0:30:0 DefMemPerCPU=512 Shared=NO 138 | PartitionName=qgpu_rtx2070 Nodes=c3 State=UP MaxTime=72:00:00 DefaultTime=0:30:0 DefMemPerCPU=512 Shared=NO 139 | PartitionName=qgpu_rtx2080 Nodes=c4 State=UP MaxTime=72:00:00 DefaultTime=0:30:0 DefMemPerCPU=512 Shared=NO 140 | 141 | ReturnToService=0 142 | ``` 143 | 144 | ### Create file gres.conf 145 | ``` 146 | # vi /etc/slurm/gres.conf 147 | NodeName=master Name=gpu File=/dev/nvidia[0-1] 148 | NodeName=c1 Name=gpu File=/dev/nvidia[0-1] 149 | NodeName=c2 Name=gpu File=/dev/nvidia0 150 | ``` 151 | 152 | ``` 153 | systemctl restart slurmctld 154 | systemctl restart slurmd 155 | systemctl restart munge 156 | pdsh -w c[1-2] systemctl restart slurmd 157 | pdsh -w c[1-2] systemctl restart munge 158 | scontrol show nodes 159 | ``` 160 | 161 | ### Change state of node 162 | ``` 163 | # scontrol for compute node after reboot 164 | scontrol update NodeName=c[1-2] State=RESUME 165 | ``` 166 | 167 | ### Create slurmdbd.conf 168 | ``` 169 | # 170 | # Example slurmdbd.conf file. 171 | # 172 | # See the slurmdbd.conf man page for more information. 173 | # 174 | # Archive info 175 | ArchiveEvents=yes 176 | ArchiveJobs=yes 177 | ArchiveResvs=yes 178 | ArchiveSteps=no 179 | ArchiveSuspend=no 180 | ArchiveTXN=no 181 | ArchiveUsage=no 182 | #ArchiveDir="/tmp" 183 | ArchiveSteps=yes 184 | #ArchiveScript= 185 | #JobPurge=12 186 | #StepPurge=1 187 | # 188 | # Authentication info 189 | AuthType=auth/munge 190 | AuthInfo=/var/run/munge/munge.socket.2 191 | # 192 | # slurmDBD info 193 | DbdAddr=master 194 | DbdHost=master 195 | DbdPort=6819 196 | SlurmUser=slurm 197 | #MessageTimeout=300 198 | DebugLevel=4 199 | #DefaultQOS=normal,standby 200 | LogFile=/var/log/slurm/slurmdbd.log 201 | PidFile=/var/run/slurmdbd.pid 202 | #PluginDir=/usr/lib/slurm 203 | #PrivateData=accounts,users,usage,jobs 204 | #TrackWCKey=yes 205 | # 206 | # Database info 207 | StorageType=accounting_storage/mysql 208 | StorageHost=localhost 209 | StoragePort=3306 210 | StoragePass=slurm@1234 211 | StorageUser=slurm 212 | StorageLoc=slurm_acct_db 213 | 214 | PurgeEventAfter=12month 215 | PurgeJobAfter=12month 216 | PurgeResvAfter=2month 217 | PurgeStepAfter=2month 218 | PurgeSuspendAfter=1month 219 | PurgeTXNAfter=12month 220 | PurgeUsageAfter=12month 221 | MaxQueryTimeRange=60-0 222 | ``` 223 | 224 | ### Setup MySQL 225 | ``` 226 | # mysql -p 227 | mysql> grant all on slurm_acct_db.* TO 'slurm'@'localhost' identified by 'slurm@1234' with grant option; 228 | mysql> create database slurm_acct_db; 229 | mysql> quit; 230 | ``` 231 | 232 | ``` 233 | systemctl start slurmdbd 234 | systemctl restart slurmctld 235 | sacctmgr add cluster galaxy.cluster 236 | ``` 237 | 238 | 239 | ### Firewalld for Slurm daemons 240 | SlurmctldPort=6817 241 | SlurmdPort=6818 242 | SchedulerPort=7321 243 | ``` 244 | yum install firewalld firewall-config 245 | systemctl start firewalld 246 | systemctl enable firewalld 247 | firewall-cmd --permanent --zone=public --add-port=6817/tcp 248 | firewall-cmd --permanent --direct --add-rule ipv4 filter INPUT_direct 0 -s 192.168.0.0/16 -j ACCEPT 249 | firewall-cmd --permanent --zone=public --add-port=6819/tcp 250 | firewall-cmd --reload 251 | ``` 252 | 253 | ### MySQL configuration 254 | Create a new file /etc/my.cnf.d/innodb.cnf containing: 255 | ``` 256 | [mysqld] 257 | innodb_buffer_pool_size=1024M 258 | innodb_log_file_size=64M 259 | innodb_lock_wait_timeout=900 260 | ``` 261 | To implement this change you have to shut down the database and move/remove logfiles: 262 | ``` 263 | systemctl stop mariadb 264 | mv /var/lib/mysql/ib_logfile? /tmp/ 265 | systemctl start mariadb 266 | ``` 267 | -------------------------------------------------------------------------------- /06_After Startup or Reboot.md: -------------------------------------------------------------------------------- 1 | ### Restart Gangila 2 | ``` 3 | pdsh -w c[1-2] systemctl restart gmond 4 | pdsh -w c[1-2] systemctl restart gmetad 5 | systemctl restart gmond 6 | systemctl restart gmetad 7 | ``` 8 | 9 | ### Restart Slurm 10 | ``` 11 | pdsh -w c[1-2] systemctl restart slurmd 12 | systemctl restart slurmd 13 | systemctl restart slurmdbd 14 | systemctl restart slurmctld 15 | ``` 16 | 17 | ### Check Slurm status 18 | ``` 19 | scontrol show node 20 | sinfo 21 | ``` 22 | 23 | ### How to “undrain” slurm nodes in drain state 24 | 25 | **Example:** 26 | Using sinfo it shows 3 nodes are in drain state, 27 | 28 | PARTITION AVAIL TIMELIMIT NODES STATE NODELIST\ 29 | all* up infinite 3 drain node[10,11,12] 30 | 31 | ``` 32 | scontrol: update NodeName=node10 State=DOWN Reason="undraining" 33 | scontrol: update NodeName=node10 State=RESUME 34 | scontrol: show node node10 35 | ``` 36 | 37 | **If no jobs are currently running on the node:** 38 | ``` 39 | scontrol update nodename=node10 state=idle 40 | ``` 41 | 42 | **If jobs are running on the node:** 43 | ``` 44 | scontrol update nodename=node10 state=resume 45 | ``` 46 | 47 | -------------------------------------------------------------------------------- /07_Add Hard disk.md: -------------------------------------------------------------------------------- 1 | ### Check hard disk 2 | ``` 3 | # fdisk -l 4 | ``` 5 | 6 | ### Add hard disk 7 | ``` 8 | # parted /dev/sdb1 9 | # mklabel gpt 10 | # unit TB 11 | # mkpart primary 0.00TB 3.00TB 12 | # print 13 | # quit 14 | ``` 15 | 16 | ### Use the mkfs.ext4 command to format the file system 17 | ``` 18 | # mkfs.ext4 /dev/sdb1 19 | ``` 20 | 21 | ### Type the following commands to mount /dev/sdb1 22 | ``` 23 | # mkdir /data1 24 | # chmod 777 /data1 25 | # mount /dev/sdb1 /data1 26 | # df -h 27 | ``` 28 | 29 | ### Add part to permanent mount 30 | ``` 31 | # vi /etc/fstab 32 | /dev/sdb1 /data1 auto nosuid,nodev,nofail 0 0 33 | ``` 34 | -------------------------------------------------------------------------------- /08_LDAP server.md: -------------------------------------------------------------------------------- 1 | Ref: https://www.itzgeek.com/how-tos/linux/centos-how-tos/step-step-openldap-server-configuration-centos-7-rhel-7.html 2 | https://www.itzgeek.com/how-tos/linux/centos-how-tos/migrate-local-users-ldap-accounts.html 3 | 4 | ### Install the following LDAP RPM packages on LDAP server 5 | ``` 6 | # yum -y install openldap compat-openldap openldap-clients openldap-servers openldap-servers-sql openldap-devel 7 | ``` 8 | 9 | ### Start the LDAP service and enable it for the auto start of service on system boot. 10 | ``` 11 | # systemctl start slapd 12 | # systemctl enable slapd 13 | ``` 14 | 15 | ### Verify the LDAP 16 | ``` 17 | # netstat -antup | grep -i 389 18 | ``` 19 | 20 | ### Setup LDAP admin password, replace ldppassword with your password 21 | ``` 22 | # slappasswd -h {SSHA} -s ldppassword 23 | ``` 24 | 25 | ### Create a .ldif file 26 | ``` 27 | # cd /etc/openldap/slapd.d/ 28 | # vi db.ldif 29 | 30 | dn: olcDatabase={2}hdb,cn=config 31 | changetype: modify 32 | replace: olcSuffix 33 | olcSuffix: dc=galaxy,dc=local 34 | 35 | dn: olcDatabase={2}hdb,cn=config 36 | changetype: modify 37 | replace: olcRootDN 38 | olcRootDN: cn=ldapadm,dc=galaxy,dc=local 39 | 40 | dn: olcDatabase={2}hdb,cn=config 41 | changetype: modify 42 | replace: olcRootPW 43 | olcRootPW: {SSHA}d/thexcQUuSfe3rx3gRaEhHpNJ52N8D3 // Replace the encrypted password ({SSHA}d/thexcQUuSfe3rx3gRaEhHpNJ52N8D3) with the password you generated in the previous step. 44 | 45 | # ldapmodify -Y EXTERNAL -H ldapi:/// -f db.ldif 46 | ``` 47 | 48 | ### Create monitor.ldif file 49 | ``` 50 | # vi monitor.ldif 51 | 52 | dn: olcDatabase={1}monitor,cn=config 53 | changetype: modify 54 | replace: olcAccess 55 | olcAccess: {0}to * by dn.base="gidNumber=0+uidNumber=0,cn=peercred,cn=external, cn=auth" read by dn.base="cn=ldapadm,dc=galaxy,dc=local" read by * none 56 | 57 | # ldapmodify -Y EXTERNAL -H ldapi:/// -f monitor.ldif 58 | ``` 59 | 60 | ### Set up LDAP database 61 | ``` 62 | # cp /usr/share/openldap-servers/DB_CONFIG.example /var/lib/ldap/DB_CONFIG 63 | # chown ldap:ldap /var/lib/ldap/* 64 | # ldapadd -Y EXTERNAL -H ldapi:/// -f /etc/openldap/schema/cosine.ldif 65 | # ldapadd -Y EXTERNAL -H ldapi:/// -f /etc/openldap/schema/nis.ldif 66 | # ldapadd -Y EXTERNAL -H ldapi:/// -f /etc/openldap/schema/inetorgperson.ldif 67 | # vi base.ldif 68 | 69 | dn: dc=galaxy,dc=local 70 | dc: galaxy 71 | objectClass: top 72 | objectClass: domain 73 | 74 | dn: cn=ldapadm ,dc=galaxy,dc=local 75 | objectClass: organizationalRole 76 | cn: ldapadm 77 | description: LDAP Manager 78 | 79 | dn: ou=People,dc=galaxy,dc=local 80 | objectClass: organizationalUnit 81 | ou: People 82 | 83 | dn: ou=Group,dc=galaxy,dc=local 84 | objectClass: organizationalUnit 85 | ou: Group 86 | 87 | # ldapadd -x -W -D "cn=ldapadm,dc=itzgeek,dc=local" -f base.ldif 88 | ``` 89 | 90 | ### Migrate local users to LDAP accounts 91 | ``` 92 | # yum -y install migrationtools 93 | # grep "ldpuser" /etc/passwd >> /root/users 94 | # grep "ldpuser" /etc/group >> /root/groups 95 | # vi /usr/share/migrationtools/migrate_common.ph 96 | 97 | $DEFAULT_MAIL_DOMAIN = "galaxy.local"; 98 | $DEFAULT_BASE = "dc=galaxy,dc=local"; 99 | $EXTENDED_SCHEMA = 1; 100 | 101 | # /usr/share/migrationtools/migrate_passwd.pl /root/users /root/users.ldif 102 | # /usr/share/migrationtools/migrate_group.pl /root/groups /root/groups.ldif 103 | # ldapadd -x -W -D "cn=ldapadm,dc=galaxy,dc=local" -f /root/users.ldif 104 | # ldapadd -x -W -D "cn=ldapadm,dc=galaxy,dc=local" -f /root/groups.ldif 105 | # ldapsearch -x cn=ldpuser1 -b dc=galaxy,dc=local 106 | # ldapdelete -v -D "cn=ldapadm,dc=galaxy,dc=local" -W "uid=xxxxxxxx,ou=People,dc=galaxy,dc=local" 107 | # ldapdelete -v -D "cn=ldapadm,dc=galaxy,dc=local" -W "cn=xxxxxxxx,ou=Group,dc=galaxy,dc=local" 108 | ``` 109 | -------------------------------------------------------------------------------- /09_OpenHPC Slurm Setup for Single Node.md: -------------------------------------------------------------------------------- 1 | ## Pre-setup 2 | \ 3 | **Master and Compute**\ 4 | Hostname: sirius\ 5 | enp0s8: public network 10.204.100.10 (depend on your ip address) 6 | 7 |
8 | 9 | ### Add host 10 | ``` 11 | # vi /etc/hosts 12 | 13 | 10.204.100.10 sirius 14 | ``` 15 | 16 | ``` 17 | hostnamectl set-hostname sirius 18 | ``` 19 | 20 | ### Disable firewall 21 | ``` 22 | # systemctl disable firewalld 23 | # systemctl stop firewalld 24 | ``` 25 | 26 | ### Disable selinux 27 | ``` 28 | # vi /etc/selinux/config 29 | 30 | SELINUX=disabled 31 | ``` 32 | 33 | ### Reboot node 34 | ``` 35 | # reboot 36 | ``` 37 | 38 | ### Update CentOS 39 | ``` 40 | # yum -y update 41 | ``` 42 |
43 | 44 | ## OpenHPC and Slurm Setup 45 | 46 |
47 | 48 | ### Install OpenHPC Repository 49 | ``` 50 | # yum -y install http://repos.openhpc.community/OpenHPC/2/CentOS_8/x86_64/ohpc-release-2-1.el8.x86_64.rpm 51 | ``` 52 | 53 | ### Install basic package for OpenHPC 54 | ``` 55 | # yum -y install ohpc-base 56 | # yum -y install ohpc-warewulf 57 | ``` 58 | 59 | ### Install Slurm 60 | ``` 61 | # yum -y install ohpc-slurm-server 62 | # yum -y install ohpc-slurm-client 63 | # cp /etc/slurm/slurm.conf.ohpc /etc/slurm/slurm.conf 64 | ``` 65 | 66 | ### Restart and enable services 67 | ``` 68 | # systemctl enable mariadb.service 69 | # systemctl restart mariadb 70 | # systemctl enable httpd.service 71 | # systemctl restart httpd 72 | # systemctl enable dhcpd.service 73 | ``` 74 | 75 | ### Install OpenHPC for compute node 76 | ``` 77 | # yum -y install ohpc-base-compute 78 | ``` 79 | 80 | ### Install modules user enviroment for compute node and master node 81 | ``` 82 | # yum -y install lmod-ohpc 83 | ``` 84 | 85 | ### Create basic values for OpenHPC 86 | ``` 87 | # wwinit database 88 | # wwinit ssh_keys 89 | ``` 90 | 91 | ### Update basic slurm configuration by vim 92 | Please adjust configuration based on your computer. BTW, you can adjust by using configuration tool at https://slurm.schedmd.com/configurator.html 93 | ``` 94 | # vi /etc/slurm/slurm.conf 95 | 96 | # 97 | # Example slurm.conf file. Please run configurator.html 98 | # (in doc/html) to build a configuration file customized 99 | # for your environment. 100 | # 101 | # 102 | # slurm.conf file generated by configurator.html. 103 | # 104 | # See the slurm.conf man page for more information. 105 | # 106 | ClusterName=sirius.cluster 107 | ControlMachine=sirius 108 | #ControlAddr= 109 | #BackupController= 110 | #BackupAddr= 111 | # 112 | SlurmUser=slurm 113 | #SlurmdUser=root 114 | SlurmctldPort=6817 115 | SlurmdPort=6818 116 | AuthType=auth/munge 117 | #JobCredentialPrivateKey= 118 | #JobCredentialPublicCertificate= 119 | StateSaveLocation=/var/spool/slurm/ctld 120 | SlurmdSpoolDir=/var/spool/slurm/d 121 | SwitchType=switch/none 122 | MpiDefault=none 123 | SlurmctldPidFile=/var/run/slurmctld.pid 124 | SlurmdPidFile=/var/run/slurmd.pid 125 | ProctrackType=proctrack/pgid 126 | #PluginDir= 127 | #FirstJobId= 128 | #MaxJobCount= 129 | #PlugStackConfig= 130 | #PropagatePrioProcess= 131 | #PropagateResourceLimits= 132 | #PropagateResourceLimitsExcept= 133 | #Prolog= 134 | #Epilog= 135 | #SrunProlog= 136 | #SrunEpilog= 137 | #TaskProlog= 138 | #TaskEpilog= 139 | #TaskPlugin= 140 | #TrackWCKey=no 141 | #TreeWidth=50 142 | #TmpFS= 143 | #UsePAM= 144 | # 145 | # TIMERS 146 | SlurmctldTimeout=300 147 | SlurmdTimeout=300 148 | InactiveLimit=0 149 | MinJobAge=300 150 | KillWait=30 151 | Waittime=0 152 | # 153 | # SCHEDULING 154 | SchedulerType=sched/backfill 155 | #SchedulerAuth= 156 | SelectType=select/cons_res 157 | SelectTypeParameters=CR_CPU_Memory 158 | #PriorityType=priority/multifactor 159 | #PriorityDecayHalfLife=14-0 160 | #PriorityUsageResetPeriod=14-0 161 | #PriorityWeightFairshare=100000 162 | #PriorityWeightAge=1000 163 | #PriorityWeightPartition=10000 164 | #PriorityWeightJobSize=1000 165 | #PriorityMaxAge=1-0 166 | # 167 | # LOGGING 168 | SlurmctldDebug=info 169 | SlurmctldLogFile=/var/log/slurmctld.log 170 | SlurmdDebug=info 171 | SlurmdLogFile=/var/log/slurmd.log 172 | JobCompType=jobcomp/none 173 | #JobCompLoc= 174 | # 175 | # ACCOUNTING 176 | #JobAcctGatherType=jobacct_gather/linux 177 | #JobAcctGatherFrequency=30 178 | # 179 | #AccountingStorageType=accounting_storage/slurmdbd 180 | #AccountingStorageHost= 181 | #AccountingStorageLoc= 182 | #AccountingStoragePass= 183 | #AccountingStorageUser= 184 | # 185 | # COMPUTE NODES 186 | # OpenHPC default configuration 187 | PropagateResourceLimitsExcept=MEMLOCK 188 | #AccountingStorageType=accounting_storage/filetxt 189 | #Epilog=/etc/slurm/slurm.epilog.clean 190 | GresTypes=gpu 191 | # 192 | # NODES 193 | NodeName=sirius NodeAddr=sirius NodeHostName=sirius Gres=gpu:2 CPUs=12 CoresPerSocket=12 ThreadsPerCore=1 RealMemory=31000 St 194 | ate=UNKNOWN 195 | # 196 | # PARTITIONS 197 | PartitionName=gpu Nodes=sirius DefMemPerCPU=512 Default=YES Shared=NO State=UP MaxTime=INFINITE 198 | PartitionName=cpu Nodes=sirius DefMemPerCPU=512 Default=YES Shared=NO State=UP MaxTime=INFINITE 199 | # 200 | ReturnToService=0 201 | 202 | ``` 203 | 204 | ### Add gres.conf for GPU allocation 205 | Please check number of GPU, e.g. nvidia[0-1] is 2 GPUs 206 | ``` 207 | # vi /etc/slurm/gres.conf 208 | 209 | NodeName=sirius Name=gpu File=/dev/nvidia[0-1] 210 | ``` 211 | 212 | ### Restart Slurm and Munge services 213 | ``` 214 | # systemctl enable munge 215 | # systemctl enable slurmctld 216 | # systemctl enable slurmd 217 | # systemctl start munge 218 | # systemctl start slurmctld 219 | # systemctl start slurmd 220 | ``` 221 | 222 | ### Determine memlock values 223 | ``` 224 | # perl -pi -e 's/# End of file/\* soft memlock unlimited\n$&/s' /etc/security/limits.conf 225 | # perl -pi -e 's/# End of file/\* hard memlock unlimited\n$&/s' /etc/security/limits.conf 226 | # perl -pi -e 's/# End of file/\* soft memlock unlimited\n$&/s' $CHROOT/etc/security/limits.conf 227 | # perl -pi -e 's/# End of file/\* hard memlock unlimited\n$&/s' $CHROOT/etc/security/limits.conf 228 | ``` 229 | 230 | ### Test resource by 231 | ``` 232 | # scontrol show nodes 233 | ``` 234 | 235 | ### Installation essential modules/softwares 236 |
237 | 238 | #### Install Intel Complier (Intel oneAPI Base) 239 | Ref: https://software.intel.com/content/www/us/en/develop/tools/oneapi/base-toolkit/download.html?operatingsystem=linux&distributions=yumpackagemanager 240 | ``` 241 | sudo bash -c 'cat << EOF > /etc/yum.repos.d/oneAPI.repo 242 | [oneAPI] 243 | name=Intel(R) oneAPI repository 244 | baseurl=https://yum.repos.intel.com/oneapi 245 | enabled=1 246 | gpgcheck=1 247 | repo_gpgcheck=1 248 | gpgkey=https://yum.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB 249 | EOF' 250 | ``` 251 | ``` 252 | sudo yum upgrade intel-basekit 253 | ``` 254 |
255 | 256 | #### Install OHPC basic additional packages 257 | You can check additional packages from http://repos.openhpc.community/OpenHPC/2/CentOS_8/x86_64/ 258 | ``` 259 | # yum install -y EasyBuild-ohpc autoconf-ohpc automake-ohpc cmake-ohpc gnu9-compilers-ohpc hwloc-ohpc libtool-ohpc python3-Cython-ohpc singularity-ohpc 260 | ``` 261 |
262 | 263 | #### Install OHPC GNU9 basic additional packages 264 | You can check additional packages from http://repos.openhpc.community/OpenHPC/2/CentOS_8/x86_64/ 265 | ``` 266 | # yum install -y ohpc-gnu9* R-gnu9-ohpc adios-gnu9* boost-gnu9* fftw-gnu9* hdf5-gnu9* mpich-ofi-gnu9* mpich-ucx-gnu9* mvapich2-gnu9* netcdf-gnu9* openmpi4-gnu9* pdtoolkit-gnu9* phdf5-gnu9* pnetcdf-gnu9* python3-mpi4py-gnu9* sionlib-gnu9* 267 | ``` 268 |
269 | 270 | #### Install OHPC Intel basic additional packages 271 | You can check additional packages from http://repos.openhpc.community/OpenHPC/2/CentOS_8/x86_64/ 272 | ``` 273 | # yum install -y ohpc-intel* intel-compilers-devel-ohpc intel-mpi-devel-ohpc adios-intel* boost-intel* hdf5-intel* mpich-ofi-intel* mpich-ucx-intel* mvapich2-intel* netcdf-intel* openmpi4-intel* pdtoolkit-intel* phdf5-intel* pnetcdf-intel* python3-mpi4py-intel* sionlib-intel* 274 | ``` 275 | -------------------------------------------------------------------------------- /10_AMBER20.md: -------------------------------------------------------------------------------- 1 | Amber is a suite of biomolecular simulation programs. It began in the late 1970's, and is maintained by an active development community. 2 | 3 | The term "Amber" refers to two things. First, it is a set of molecular mechanical force fields for the simulation of biomolecules (these force fields are in the public domain, and are used in a variety of simulation programs). Second, it is a package of molecular simulation programs which includes source code and demos. 4 | 5 | Amber is distributed in two parts: AmberTools20 and Amber20. You can use AmberTools20 without Amber20, but not vice versa. 6 | 7 | Ref: https://ambermd.org/ 8 | 9 | 10 | ### Pre-installation 11 | 12 | System: CentOS Linux release 7.6.1810 with CMake 3.20.1, CUDA 11.1.1, GNU10 v10.3.0, OpenMPI4 v4.1.1 operated by Lmod 13 | 14 | Refs: \ 15 | https://www.hull1.com/linux/2020/08/21/complie-amber20.html \ 16 | https://ambermd.org/doc12/Amber20.pdf 17 | 18 | #### Install Dependencies 19 | ``` 20 | yum -y install tcsh make \ 21 | gcc gcc-gfortran gcc-c++ \ 22 | which flex bison patch bc \ 23 | libXt-devel libXext-devel \ 24 | perl perl-ExtUtils-MakeMaker util-linux wget \ 25 | bzip2 bzip2-devel zlib-devel tar 26 | ``` 27 | 28 | #### Extract amber 29 | ``` 30 | tar xvfj AmberTools20.tar.bz2 31 | tar xvfj Amber20.tar.bz2 32 | ``` 33 | 34 | #### Create amber20 folder at destination path (e.g. /apps/amber20) 35 | ``` 36 | mkdir /apps/amber20 37 | ``` 38 | 39 | #### Upgrade and update Amber 40 | ``` 41 | cd amber20_src 42 | ./update_amber --upgrade 43 | ./update_amber --update 44 | ``` 45 | 46 | ### Compile Serial CPU 47 | 48 | #### Compile and install 49 | ``` 50 | cd amber20_src 51 | cd build 52 | cmake .. -DCMAKE_INSTALL_PREFIX=/apps/amber20 -DCOMPILER=GNU -DCUDA=FALSE -DINSTALL_TESTS=FALSE -DDOWNLOAD_MINICONDA=TRUE -DMINICONDA_USE_PY3=TRUE 53 | make install -j8 54 | ``` 55 | 56 | #### Test 57 | ``` 58 | cd /apps/amber20 59 | source amber.sh 60 | make test.serial 61 | ``` 62 | 63 | ### Compile for Serial GPU 64 | 65 | Ref: https://ambermd.org/GPUHardware.php 66 | 67 | Amber tries to support all CUDA SDK versions up to 11.x. In the past, they have recommended CUDA 9.1 or 9.2 for the best speed of the resulting executables, but this needs to be revisited. Here are the minimum requirements for different tiers of hardware: 68 | * Ampere (SM_80) based cards require CUDA 11.0 or later (RTX-3060, RTX-3070, RTX-3080, RTX-3090, RTX-A6000, A100). 69 | * Turing (SM_75) based cards require CUDA 9.2 or later (RTX-2080Ti, RTX-2080, RTX-2070, Quadro RTX6000/5000/4000). 70 | * Volta (SM_70) based cards require CUDA 9.0 or later (Titan-V, V100, Quadro GV100). 71 | * Pascal (SM_61) based cards require CUDA 8.0 or later. (Titan-XP [aka Pascal Titan-X], GTX-1080TI / 1080 / 1070 / 1060, Quadro P6000 / P5000, P4 / P40). 72 | 73 | * GTX-1080 cards require NVIDIA Driver version >= 367.27 for reliable numerical results. 74 | * GTX-Titan and GTX-780 cards require NVIDIA Driver version >= 319.60 for correct numerical results. 75 | * GTX-780Ti cards require a modified Bios from Exxact Corp to give correct numerical results. 76 | * GTX-Titan-Black Edition cards require NVIDIA Driver version >= 337.09 or 331.79 or later for correct numerical results. 77 | 78 | #### Compile and install 79 | ``` 80 | cd amber20_src 81 | cd build 82 | make clean 83 | cmake .. -DCMAKE_INSTALL_PREFIX=/apps/amber20 -DCOMPILER=GNU -DCUDA=TRUE -DCUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda-9.2 -DINSTALL_TESTS=FALSE -DDOWNLOAD_MINICONDA=TRUE -DMINICONDA_USE_PY3=TRUE 84 | make install -j8 85 | ``` 86 | 87 | #### Test 88 | ``` 89 | cd /apps/amber20 90 | source amber.sh 91 | make test.cuda 92 | ``` 93 | 94 | ### Compile for Parallel CPU 95 | 96 | #### Compile and install 97 | ``` 98 | cd amber20_src 99 | cd build 100 | make clean 101 | cmake .. -DCMAKE_INSTALL_PREFIX=/apps/amber20 -DCOMPILER=GNU -DCUDA=FALSE -DMPI=TRUE -DINSTALL_TESTS=FALSE -DDOWNLOAD_MINICONDA=TRUE -DMINICONDA_USE_PY3=TRUE 102 | make install -j8 103 | ``` 104 | 105 | #### Test 106 | ``` 107 | cd /apps/amber20 108 | export DO_PARALLEL='mpirun -np 2' 109 | test -f amber.sh && source amber.sh 110 | export DO_PARALLEL='mpirun -np 2' 111 | make test.parallel 112 | ``` 113 | 114 | ### Compile for Parallel GPU 115 | 116 | Ref: https://ambermd.org/GPUHardware.php 117 | 118 | Amber tries to support all CUDA SDK versions up to 11.x. In the past, they have recommended CUDA 9.1 or 9.2 for the best speed of the resulting executables, but this needs to be revisited. Here are the minimum requirements for different tiers of hardware: 119 | * Ampere (SM_80) based cards require CUDA 11.0 or later (RTX-3060, RTX-3070, RTX-3080, RTX-3090, RTX-A6000, A100). 120 | * Turing (SM_75) based cards require CUDA 9.2 or later (RTX-2080Ti, RTX-2080, RTX-2070, Quadro RTX6000/5000/4000). 121 | * Volta (SM_70) based cards require CUDA 9.0 or later (Titan-V, V100, Quadro GV100). 122 | * Pascal (SM_61) based cards require CUDA 8.0 or later. (Titan-XP [aka Pascal Titan-X], GTX-1080TI / 1080 / 1070 / 1060, Quadro P6000 / P5000, P4 / P40). 123 | 124 | * GTX-1080 cards require NVIDIA Driver version >= 367.27 for reliable numerical results. 125 | * GTX-Titan and GTX-780 cards require NVIDIA Driver version >= 319.60 for correct numerical results. 126 | * GTX-780Ti cards require a modified Bios from Exxact Corp to give correct numerical results. 127 | * GTX-Titan-Black Edition cards require NVIDIA Driver version >= 337.09 or 331.79 or later for correct numerical results. 128 | 129 | #### Compile and install 130 | ``` 131 | cd amber20_src 132 | cd build 133 | make clean 134 | cmake .. -DCMAKE_INSTALL_PREFIX=/apps/amber20 -DCOMPILER=GNU -DMPI=TRUE -DCUDA=TRUE -DCUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda-9.2 -DINSTALL_TESTS=FALSE -DDOWNLOAD_MINICONDA=TRUE -DMINICONDA_USE_PY3=TRUE 135 | make install -j8 136 | ``` 137 | 138 | #### Test 139 | ``` 140 | cd /apps/amber20 141 | export DO_PARALLEL='mpirun -np 2' 142 | test -f amber.sh && source amber.sh 143 | export DO_PARALLEL='mpirun -np 2' 144 | make test.cuda.parallel 145 | ``` 146 | 147 | ### Compile for Parallel GPU with NCCL 148 | 149 | Ref: https://ambermd.org/GPUHardware.php 150 | 151 | Amber tries to support all CUDA SDK versions up to 11.x. In the past, they have recommended CUDA 9.1 or 9.2 for the best speed of the resulting executables, but this needs to be revisited. Here are the minimum requirements for different tiers of hardware: 152 | * Ampere (SM_80) based cards require CUDA 11.0 or later (RTX-3060, RTX-3070, RTX-3080, RTX-3090, RTX-A6000, A100). 153 | * Turing (SM_75) based cards require CUDA 9.2 or later (RTX-2080Ti, RTX-2080, RTX-2070, Quadro RTX6000/5000/4000). 154 | * Volta (SM_70) based cards require CUDA 9.0 or later (Titan-V, V100, Quadro GV100). 155 | * Pascal (SM_61) based cards require CUDA 8.0 or later. (Titan-XP [aka Pascal Titan-X], GTX-1080TI / 1080 / 1070 / 1060, Quadro P6000 / P5000, P4 / P40). 156 | 157 | * GTX-1080 cards require NVIDIA Driver version >= 367.27 for reliable numerical results. 158 | * GTX-Titan and GTX-780 cards require NVIDIA Driver version >= 319.60 for correct numerical results. 159 | * GTX-780Ti cards require a modified Bios from Exxact Corp to give correct numerical results. 160 | * GTX-Titan-Black Edition cards require NVIDIA Driver version >= 337.09 or 331.79 or later for correct numerical results. 161 | 162 | #### Download NVIDIA NCCL from https://developer.nvidia.com/nccl 163 | 164 | #### Install PnetCDF 165 | 166 | * Download PnetCDF from https://parallel-netcdf.github.io/wiki/Download.html 167 | ``` 168 | autoreconf -i 169 | ./configure --prefix=/apps/PnetCDF 170 | make -j8 171 | make install 172 | ``` 173 | 174 | #### Compile and install 175 | ``` 176 | cd amber20_src 177 | export NCCL_HOME="~/apps/lib/nccl_2.6.4-1+cuda10.0_x86_64" 178 | cd build 179 | make clean 180 | cmake .. -DCMAKE_INSTALL_PREFIX=/apps/amber20 -DCOMPILER=GNU -DMPI=TRUE -DCUDA=TRUE -DCUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda-9.2 -DNCCL=TRUE -DPnetCDF_C_LIBRARY=~/apps/pnetcdf/lib/ -DPnetCDF_C_INCLUDE_DIR=~/apps/pnetcdf/include/ -DINSTALL_TESTS=FALSE -DDOWNLOAD_MINICONDA=TRUE -DMINICONDA_USE_PY3=TRUE 181 | make install -j8 182 | ``` 183 | 184 | #### Test 185 | ``` 186 | cd /apps/amber20 187 | export DO_PARALLEL='mpirun -np 2' 188 | test -f amber.sh && source amber.sh 189 | export DO_PARALLEL='mpirun -np 2' 190 | make test.cuda.parallel 191 | ``` 192 | -------------------------------------------------------------------------------- /11_AMBER22.md: -------------------------------------------------------------------------------- 1 | Amber is a suite of biomolecular simulation programs. It began in the late 1970's, and is maintained by an active development community. 2 | 3 | The term "Amber" refers to two things. First, it is a set of molecular mechanical force fields for the simulation of biomolecules (these force fields are in the public domain, and are used in a variety of simulation programs). Second, it is a package of molecular simulation programs which includes source code and demos. 4 | 5 | Amber is distributed in two parts: AmberTools22 and Amber22. You can use AmberTools22 without Amber22, but not vice versa. 6 | 7 | Ref: https://ambermd.org/ 8 | 9 | 10 | ### Pre-installation 11 | 12 | System: CentOS Linux release 7.6.1810 with CMake 3.27.6, CUDA 11.8, GNU11 v11.2.1, and OpenMPI4 v4.1.6 operated by Lmod 13 | 14 | Refs: \ 15 | https://ambermd.org/doc12/Amber22.pdf 16 | 17 | #### Install Dependencies 18 | ``` 19 | yum -y install tcsh make \ 20 | gcc gcc-gfortran gcc-c++ \ 21 | which flex bison patch bc \ 22 | libXt-devel libXext-devel \ 23 | perl perl-ExtUtils-MakeMaker util-linux wget \ 24 | bzip2 bzip2-devel zlib-devel tar 25 | ``` 26 | 27 | #### Extract amber 28 | ``` 29 | tar xvfj AmberTools23.tar.bz2 30 | tar xvfj Amber22.tar.bz2 31 | ``` 32 | 33 | #### Upgrade and update Amber 34 | ``` 35 | cd amber22_src 36 | ./update_amber --update 37 | ``` 38 | if have any error due to "https", please modify the code in 4 py files (downloader.py, main.py, patch.py, and test_updateutils.py) in amber22_src/updateutils folder from "https" to "http". You can easily modify by using the command in VIM with ":%s/https/http". 39 | 40 | 41 | ### Compile Serial CPU 42 | 43 | #### Complie and install 44 | ``` 45 | cd amber22_src 46 | cd build 47 | cmake .. -DCMAKE_INSTALL_PREFIX=/apps/amber22 -DCOMPILER=GNU -DCUDA=FALSE -DMPI=FALSE -DINSTALL_TESTS=FALSE -DDOWNLOAD_MINICONDA=TRUE -DMINICONDA_USE_PY3=TRUE -DMINICONDA_VERSION=py38_4.12.0 48 | make install -j8 49 | ``` 50 | 51 | #### Test 52 | ``` 53 | cd /apps/amber22 54 | source amber.sh 55 | make test.serial 56 | ``` 57 | 58 | ### Compile for Serial GPU 59 | 60 | Ref: https://ambermd.org/GPUHardware.php 61 | 62 | #### Complie and install 63 | ``` 64 | cd amber22_src 65 | cd build 66 | make clean 67 | cmake .. -DCMAKE_INSTALL_PREFIX=/apps/amber22 -DCOMPILER=GNU -DMPI=FALSE -DCUDA=TRUE -DCUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda-11.8 -DINSTALL_TESTS=FALSE -DDOWNLOAD_MINICONDA=TRUE -DMINICONDA_USE_PY3=TRUE -DMINICONDA_VERSION=py38_4.12.0 68 | make install -j8 69 | ``` 70 | 71 | #### Test 72 | ``` 73 | cd /apps/amber22 74 | source amber.sh 75 | make test.cuda.serial 76 | ``` 77 | 78 | ### Compile for Parallel CPU 79 | 80 | #### Complie and install 81 | ``` 82 | cd amber22_src 83 | cd build 84 | make clean 85 | cmake .. -DCMAKE_INSTALL_PREFIX=/apps/amber22 -DCOMPILER=GNU -DCUDA=FALSE -DMPI=TRUE -DINSTALL_TESTS=FALSE -DDOWNLOAD_MINICONDA=TRUE -DMINICONDA_USE_PY3=TRUE -DMINICONDA_VERSION=py38_4.12.0 86 | make install -j8 87 | ``` 88 | 89 | #### Test 90 | ``` 91 | cd /apps/amber22 92 | export DO_PARALLEL='mpirun -np 2' 93 | test -f amber.sh && source amber.sh 94 | export DO_PARALLEL='mpirun -np 2' 95 | make test.parallel 96 | ``` 97 | 98 | ### Compile for Parallel GPU 99 | 100 | Ref: https://ambermd.org/GPUHardware.php 101 | 102 | #### Compile and install 103 | ``` 104 | cd amber22_src 105 | cd build 106 | make clean 107 | cmake .. -DCMAKE_INSTALL_PREFIX=/apps/amber22 -DCOMPILER=GNU -DMPI=TRUE -DCUDA=TRUE -DCUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda-11.8 -DINSTALL_TESTS=FALSE -DDOWNLOAD_MINICONDA=TRUE -DMINICONDA_USE_PY3=TRUE -DMINICONDA_VERSION=py38_4.12.0 108 | make install -j8 109 | ``` 110 | 111 | #### Test 112 | ``` 113 | cd /apps/amber22 114 | export DO_PARALLEL='mpirun -np 2' 115 | test -f amber.sh && source amber.sh 116 | export DO_PARALLEL='mpirun -np 2' 117 | make test.cuda.parallel 118 | ``` 119 | 120 | ### Compile for Parallel GPU with NCCL 121 | 122 | Ref: https://ambermd.org/GPUHardware.php 123 | 124 | #### Download NVIDIA NCCL from https://developer.nvidia.com/nccl 125 | 126 | #### Install PnetCDF 127 | 128 | * Download PnetCDF from https://parallel-netcdf.github.io/wiki/Download.html 129 | ``` 130 | autoreconf -i 131 | ./configure --prefix=/apps/PnetCDF 132 | make -j8 133 | make install 134 | ``` 135 | 136 | #### Compile and install 137 | ``` 138 | cd amber22_src 139 | export NCCL_HOME="~/apps/lib/nccl_2.6.4-1+cuda10.0_x86_64" 140 | cd build 141 | make clean 142 | cmake .. -DCMAKE_INSTALL_PREFIX=/apps/amber22 -DCOMPILER=GNU -DMPI=TRUE -DCUDA=TRUE -DCUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda-11.8 -DNCCL=TRUE -DPnetCDF_C_LIBRARY=~/apps/pnetcdf/lib/ -DPnetCDF_C_INCLUDE_DIR=~/apps/pnetcdf/include/ -DINSTALL_TESTS=FALSE -DDOWNLOAD_MINICONDA=TRUE -DMINICONDA_USE_PY3=TRUE -DMINICONDA_VERSION=py38_4.12.0 143 | make install -j8 144 | ``` 145 | 146 | #### Test 147 | ``` 148 | cd /apps/amber22 149 | export DO_PARALLEL='mpirun -np 2' 150 | test -f amber.sh && source amber.sh 151 | export DO_PARALLEL='mpirun -np 2' 152 | make test.cuda.parallel 153 | ``` 154 | -------------------------------------------------------------------------------- /12_CSDS-2023.2.md: -------------------------------------------------------------------------------- 1 | ``` 2 | # chmod a+x CSDInstallerOffline-2023.2.1-linux 3 | # ./CSDInstallerOffline-2023.2.1-linux --root "/apps/CCDC-2023.2" -c --al in uk.ac.cam.ccdc.csd 4 | ``` 5 | 6 | ### Online Registration 7 | ``` 8 | # cd /apps/CCDC-2023.2/ccdc-utilities/software-activation/bin 9 | # ./ccdc_activator -A -a -k XXXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX 10 | ``` 11 | 12 | ### Licence server setup (New Installation) 13 | ``` 14 | chmod a+x ccdc_licence_server-v2-linux-x64-installer.run 15 | ./ccdc_licence_server-v2-linux-x64-installer.run (Enter Server License Key) 16 | ``` 17 | 18 | Going to CCDCLicServer folder 19 | ``` 20 | # cd /apps/CCDC-2023.2/CCDCLicServer 21 | ``` 22 | 23 | Making file config.yml 24 | ``` 25 | server: 26 | # Port server should listen to 27 | port: XXXX 28 | # License key to activate the server 29 | licenseKey: XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX 30 | # Product file path 31 | productFilePath: ./ccdc.dat 32 | # Determines how long a license lease should last. The time is in seconds. 33 | leaseDuration: 3700 34 | # Allows for a time lag (in secs) between server and client machines. 35 | allowedClockOffset: 60 36 | # Blocked IP addresses 37 | blockedIps: [] 38 | 39 | cryptlexHost: https://license-api.ccdc.cam.ac.uk 40 | 41 | auth: 42 | # API key to access the following web API endpoints: 43 | # - GET /api/server/stats?apiKey=xxx 44 | # - GET /api/floating-licenses?apiKey=xxx 45 | apiKey: XXXXXXXXX 46 | # List of admin users who can access the dashboard 47 | admins: 48 | - username: XXXXXXXXXX 49 | password: XXXXXXXXXX 50 | # Instead of password you can also provide a SHA256 hash of the password - https://xorbin.com/tools/sha256-hash-calculator 51 | passwordHash: 52 | 53 | logging: 54 | # Allowed log levels: "0" - Debug, "1" - Information, "2" - Warning, "3" - Errors 55 | logLevel: 1 56 | console: 57 | # Enable console logs 58 | enabled: true 59 | # Disable colored console logs 60 | noColor: true 61 | file: 62 | # Disable file logs, they will be managed by journald 63 | enabled: true 64 | # Maximum size of each log file in MBs 65 | maxSize: 1 66 | # Maximum backups to retain 67 | maxBackups: 10 68 | # Logs directory 69 | directory: "./logs" 70 | ``` 71 | 72 | Activating local license service 73 | ``` 74 | # ./CCDCFloatServer -a --license-key=XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX-XXXXXX -config=./config.yml --product-file=./ccdc.dat 75 | ``` 76 | 77 | Installing as a daemon on Linux 78 | ``` 79 | # ./CCDCFloatServer -i -p ./ccdc.dat -c ./config.yml --service-name ccdcfloatserver 80 | ``` 81 | 82 | Editing a daemon file 83 | ``` 84 | # vi /etc/systemd/system/ccdcfloatserver.service 85 | 86 | ExecStart=/share/apps/CCDC-2023.2/CCDCLicServer/CCDCFloatServer "-a" "-c" "/apps/CCDC-2023.2/CCDCLicServer/config.yml" "-p" "/apps/CCDC-2023.2/CCDCLicServer/ccdc.dat" "-s" 87 | ``` 88 | 89 | Automatic starting service 90 | ``` 91 | $ systemctl start ccdcfloatserver 92 | $ systemctl enable ccdcfloatserver 93 | ``` 94 | 95 | Activate License for all nodes 96 | ``` 97 | # cd /apps/CCDC-2023.2/ccdc-utilities/software-activation/bin 98 | # ./ccdc_activator -A -s http://ip:port 99 | ``` 100 | 101 | Uninstalling the Licence Server service on Linux 102 | ``` 103 | # ./CCDCFloatServer -d 104 | # ./CCDCFloatServer -u --service-name ccdcfloatserver 105 | ``` 106 | -------------------------------------------------------------------------------- /13_GROMACS-2023.2: -------------------------------------------------------------------------------- 1 | Ref: https://manual.gromacs.org/documentation/2023.2/install-guide/index.html 2 | 3 | ``` 4 | # ml swap gnu7 gnu11 5 | # ml cmake/3.27.6 cuda/11.8 openmpi4 6 | ``` 7 | 8 | ``` 9 | # tar xfz gromacs-2023.2.tar.gz 10 | # cd gromacs-2023.2 11 | # mkdir build 12 | # cd build 13 | # cmake .. -DGMX_BUILD_OWN_FFTW=ON -DREGRESSIONTEST_DOWNLOAD=OFF -DGMX_MPI=on -DGMX_BUILD_MDRUN_ONLY=off -DGMX_GPU=CUDA -DGMX_SIMD=AVX2_256 -DCMAKE_INSTALL_PREFIX=/md0/apps2/gromacs-2023.2 14 | # make -j16 15 | # make check 16 | # sudo make install 17 | # source /md0/apps2/gromacs-2023.2/bin/GMXRC 18 | ``` 19 | -------------------------------------------------------------------------------- /14_AMBER24_RC4_test.md: -------------------------------------------------------------------------------- 1 | ## AMBER24_RC4_test on NVIDIA GeForce GTX 1070 Ti 8GB with Intel(R) Core(TM) i7-7700K CPU @ 4.20GHz 2 | 3 | OS System: CentOS Linux release 7.6.1810 with CMake 3.27.6, CUDA 11.8, GCC 7.3, and OpenMPI 4.1.6 operated by Lmod \ 4 | Computer System (Frontend node): 2 x NVIDIA GeForce GTX 1070 Ti 8GB with Intel(R) Core(TM) i7-7700K CPU @ 4.20GHz 5 | 6 | ### Load modules 7 | ``` 8 | ml cmake 9 | ml openmpi4 10 | ml cuda/11.8 11 | ml cudnn/11.8-8.9.2 12 | ``` 13 | 14 | #### Extract AMBER 15 | ``` 16 | tar xvfj AmberTools24_rc4.tar.bz2 17 | tar xvfj Amber24_rc4.tar.bz2 18 | ``` 19 | 20 | ### Compile Serial CPU 21 | 22 | #### Compile and install 23 | ``` 24 | cd amber24_src 25 | cd build 26 | cmake .. -DCMAKE_INSTALL_PREFIX=/md0/apps2/amber24 -DCOMPILER=GNU -DMPI=FALSE -DCUDA=FALSE -DINSTALL_TESTS=TRUE -DDOWNLOAD_MINICONDA=TRUE 27 | make install -j8 28 | ``` 29 | 30 | #### Test 31 | ``` 32 | cd /md0/apps2/amber24 33 | source amber.sh 34 | make test.serial 35 | ``` 36 | 37 | ### Compile for Serial GPU 38 | 39 | #### Compile and install 40 | ``` 41 | cd amber24_src 42 | cd build 43 | make clean 44 | cmake .. -DCMAKE_INSTALL_PREFIX=/md0/apps2/amber24 -DCOMPILER=GNU -DMPI=FALSE -DCUDA=TRUE -DCUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda-11.8 -DINSTALL_TESTS=TRUE -DDOWNLOAD_MINICONDA=TRUE 45 | make install -j8 46 | ``` 47 | 48 | #### Test 49 | ``` 50 | cd /md0/apps2/amber24 51 | source amber.sh 52 | make test.cuda.serial 53 | ``` 54 | 55 | ### Compile for Parallel CPU 56 | 57 | #### Compile and install 58 | ``` 59 | cd amber24_src 60 | cd build 61 | make clean 62 | cmake .. -DCMAKE_INSTALL_PREFIX=/md0/apps2/amber24 -DCOMPILER=GNU -DCUDA=FALSE -DMPI=TRUE -DINSTALL_TESTS=TRUE -DDOWNLOAD_MINICONDA=TRUE 63 | make install -j16 64 | ``` 65 | 66 | #### Test for 2 MPI threads 67 | ``` 68 | cd /md0/apps2/amber24 69 | source amber.sh 70 | export DO_PARALLEL='mpirun -np 2' 71 | make test.parallel 72 | ``` 73 | 74 | #### Test for 4 MPI threads 75 | ``` 76 | cd /md0/apps2/amber24 77 | source amber.sh 78 | export DO_PARALLEL='mpirun -np 4' 79 | make test.parallel 80 | ``` 81 | 82 | ### Compile for Parallel GPU 83 | 84 | #### Compile and install 85 | ``` 86 | cd amber24_src 87 | cd build 88 | make clean 89 | cmake .. -DCMAKE_INSTALL_PREFIX=/md0/apps2/amber24 -DCOMPILER=GNU -DMPI=TRUE -DCUDA=TRUE -DCUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda-11.8 -DINSTALL_TESTS=TRUE -DDOWNLOAD_MINICONDA=TRUE 90 | make install -j16 91 | ``` 92 | 93 | #### Test for 2 MPI threads 94 | ``` 95 | cd /md0/apps2/amber24 96 | source amber.sh 97 | export DO_PARALLEL='mpirun -np 2' 98 | make test.cuda.parallel 99 | ``` 100 | 101 | ## AMBER24_RC4_test on Tesla V100 SXM2 32GB with Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz 102 | 103 | OS System: CentOS Linux release 7.6.1810 with CMake 3.20.1, CUDA 11.1.1, GCC 10.3.0, and OpenMPI 4.1.1 operated by Lmod 104 | Computer System: 4 x Tesla V100 SXM2 32GB with Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz 105 | 106 | ### Load modules 107 | ``` 108 | ml CMake/3.20.1-GCCcore-10.3.0 109 | ml OpenMPI/4.1.1-GCC-10.3.0 110 | ml CUDAcore/11.1.1 111 | ``` 112 | 113 | #### Extract AMBER 114 | ``` 115 | tar xvfj AmberTools24_rc4.tar.bz2 116 | tar xvfj Amber24_rc4.tar.bz2 117 | ``` 118 | 119 | ### Compile Serial CPU 120 | 121 | #### Compile and install 122 | ``` 123 | cd amber24_src 124 | cd build 125 | cmake .. -DCMAKE_INSTALL_PREFIX=/ist/users/bunditb/bunditb_bak2/apps/amber24 -DCOMPILER=GNU -DMPI=FALSE -DCUDA=FALSE -DINSTALL_TESTS=TRUE -DDOWNLOAD_MINICONDA=TRUE 126 | make install -j16 127 | ``` 128 | 129 | #### Test 130 | ``` 131 | sinteractive -p gpu-cluster --mem=32GB --cpus-per-task=1 --time=1-0:0:0 132 | cd /ist/users/bunditb/bunditb_bak2/apps/amber24 133 | source amber.sh 134 | make test.serial 135 | ``` 136 | 137 | ### Compile for Serial GPU 138 | 139 | #### Complie and install 140 | ``` 141 | cd amber24_src 142 | cd build 143 | make clean 144 | cmake .. -DCMAKE_INSTALL_PREFIX=/ist/users/bunditb/bunditb_bak2/apps/amber24 -DCOMPILER=GNU -DMPI=FALSE -DCUDA=TRUE -DCUDA_TOOLKIT_ROOT_DIR=/ist/apps/modules/software/CUDAcore/11.1.1 -DINSTALL_TESTS=TRUE -DDOWNLOAD_MINICONDA=TRUE 145 | make install -j16 146 | ``` 147 | 148 | #### Test 149 | ``` 150 | sinteractive -p gpu-cluster --nodes=1 --gres=gpu:1 --mem=32GB --cpus-per-gpu=1 --time=1-0:0:0 151 | cd /ist/users/bunditb/bunditb_bak2/apps/amber24 152 | source amber.sh 153 | make test.cuda.serial 154 | ``` 155 | 156 | ### Compile for Parallel CPU 157 | 158 | #### Complie and install 159 | ``` 160 | cd amber24_src 161 | cd build 162 | make clean 163 | cmake .. -DCMAKE_INSTALL_PREFIX=/ist/users/bunditb/bunditb_bak2/apps/amber24 -DCOMPILER=GNU -DCUDA=FALSE -DMPI=TRUE -DINSTALL_TESTS=TRUE -DDOWNLOAD_MINICONDA=TRUE 164 | make install -j16 165 | ``` 166 | 167 | #### Test for 2 MPI threads 168 | ``` 169 | sinteractive -p gpu-cluster --mem=32GB --ntasks=2 --time=1-0:0:0 170 | cd /ist/users/bunditb/bunditb_bak2/apps/amber24 171 | source amber.sh 172 | export DO_PARALLEL='mpirun -np 2' 173 | make test.parallel 174 | ``` 175 | 176 | #### Test for 4 MPI threads 177 | ``` 178 | sinteractive -p gpu-cluster --mem=32GB --ntasks=4 --time=1-0:0:0 179 | cd /ist/users/bunditb/bunditb_bak2/apps/amber24 180 | source amber.sh 181 | export DO_PARALLEL='mpirun -np 4' 182 | make test.parallel 183 | ``` 184 | 185 | ### Compile for Parallel GPU 186 | 187 | #### Complie and install 188 | ``` 189 | cd amber24_src 190 | cd build 191 | make clean 192 | cmake .. -DCMAKE_INSTALL_PREFIX=/ist/users/bunditb/bunditb_bak2/apps/amber24 -DCOMPILER=GNU -DMPI=TRUE -DCUDA=TRUE -DCUDA_TOOLKIT_ROOT_DIR=/ist/apps/modules/software/CUDAcore/11.1.1 -DINSTALL_TESTS=TRUE -DDOWNLOAD_MINICONDA=TRUE 193 | make install -j16 194 | ``` 195 | 196 | #### Test for 2 MPI threads 197 | ``` 198 | sinteractive -p gpu-cluster --nodes=1 --gres=gpu:1 --mem=32GB --ntasks=2 --time=1-0:0:0 199 | cd /ist/users/bunditb/bunditb_bak2/apps/amber24 200 | source amber.sh 201 | export DO_PARALLEL='mpirun -np 2' 202 | make test.cuda.parallel 203 | ``` 204 | 205 | #### Test for 4 MPI threads 206 | ``` 207 | sinteractive -p gpu-cluster --nodes=1 --gres=gpu:1 --mem=32GB --ntasks=4 --time=1-0:0:0 208 | cd /ist/users/bunditb/bunditb_bak2/apps/amber24 209 | source amber.sh 210 | export DO_PARALLEL='mpirun -np 4' 211 | make test.cuda.parallel 212 | ``` 213 | -------------------------------------------------------------------------------- /15_AMBER24.md: -------------------------------------------------------------------------------- 1 | Amber is a suite of biomolecular simulation programs. It began in the late 1970's, and is maintained by an active development community. 2 | 3 | The term "Amber" refers to two things. First, it is a set of molecular mechanical force fields for the simulation of biomolecules (these force fields are in the public domain, and are used in a variety of simulation programs). Second, it is a package of molecular simulation programs which includes source code and demos. 4 | 5 | Amber is distributed in two parts: AmberTools24 and Amber24. You can use AmberTools24 without Amber24, but not vice versa. 6 | 7 | Ref: https://ambermd.org/ 8 | 9 | 10 | ### Pre-installation 11 | 12 | OS System: CentOS Linux release 7.6.1810 with CMake 3.27.6, CUDA 11.8, GCC 7.3, and OpenMPI 4.1.6 operated by Lmod \ 13 | Computer System (Frontend node): 2 x NVIDIA GeForce GTX 1070 Ti 8GB with Intel(R) Core(TM) i7-7700K CPU @ 4.20GHz 14 | 15 | Refs: https://ambermd.org/doc12/Amber24.pdf 16 | 17 | #### Install Dependencies (For CentOS 7) 18 | ``` 19 | yum -y install tcsh make \ 20 | gcc gcc-gfortran gcc-c++ \ 21 | which flex bison patch bc \ 22 | libXt-devel libXext-devel \ 23 | perl perl-ExtUtils-MakeMaker util-linux wget \ 24 | bzip2 bzip2-devel zlib-devel tar 25 | ``` 26 | Other OS, please visit: https://ambermd.org/Installation.php 27 | 28 | #### Extract AMBER 29 | ``` 30 | tar xvfj AmberTools24.tar.bz2 31 | tar xvfj Amber24.tar.bz2 32 | ``` 33 | 34 | #### Upgrade and update Amber 35 | ``` 36 | cd amber24_src 37 | ./update_amber --update 38 | ./update_amber --upgrade 39 | ``` 40 | 41 | ### Compile Serial CPU 42 | 43 | #### Compile and install 44 | ``` 45 | cd amber24_src 46 | cd build 47 | cmake .. -DCMAKE_INSTALL_PREFIX=/XXX/amber24 -DCOMPILER=GNU -DMPI=FALSE -DCUDA=FALSE -DINSTALL_TESTS=TRUE -DDOWNLOAD_MINICONDA=TRUE 48 | make -j16 && make install 49 | ``` 50 | 51 | ### Compile for Serial GPU 52 | 53 | #### Compile and install 54 | ``` 55 | cd amber24_src 56 | cd build 57 | make clean 58 | cmake .. -DCMAKE_INSTALL_PREFIX=/XXX/amber24 -DCOMPILER=GNU -DMPI=FALSE -DCUDA=TRUE -DCUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda-11.8 -DINSTALL_TESTS=TRUE -DDOWNLOAD_MINICONDA=TRUE 59 | make -j16 && make install 60 | ``` 61 | 62 | ### Compile for Parallel CPU 63 | 64 | #### Compile and install 65 | ``` 66 | cd amber24_src 67 | cd build 68 | make clean 69 | cmake .. -DCMAKE_INSTALL_PREFIX=/XXX/amber24 -DCOMPILER=GNU -DCUDA=FALSE -DMPI=TRUE -DINSTALL_TESTS=TRUE -DDOWNLOAD_MINICONDA=TRUE 70 | make -j16 && make install 71 | ``` 72 | 73 | ### Compile for Parallel GPU 74 | 75 | #### Compile and install 76 | ``` 77 | cd amber24_src 78 | cd build 79 | make clean 80 | cmake .. -DCMAKE_INSTALL_PREFIX=/XXX/amber24 -DCOMPILER=GNU -DMPI=TRUE -DCUDA=TRUE -DCUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda-11.8 -DINSTALL_TESTS=TRUE -DDOWNLOAD_MINICONDA=TRUE 81 | make -j16 && make install 82 | ``` 83 | -------------------------------------------------------------------------------- /16_GROMACS-2024.2.md: -------------------------------------------------------------------------------- 1 | GROMACS (GROningen MAchine for Chemical Simulations) is a high-performance molecular dynamics simulation software. It is primarily designed for simulating the interactions between molecules in complex systems, such as biological macromolecules, polymers, and non-biological systems. 2 | 3 | GROMACS was initially developed in the early 1990s at the University of Groningen in the Netherlands by the research group led by Herman Berendsen. The primary motivation was to create a software package for simulating molecular dynamics, particularly for biomolecular systems. The first versions of GROMACS were focused on biochemical molecules and were written in the Fortran programming language. The software aimed to efficiently simulate molecular systems using the GROMOS force field. 4 | 5 | Ref: https://www.gromacs.org and https://www.gromacs.org/development.html 6 | 7 | ### Overview process of installation 8 | - Get the latest version of your C and C++ compilers. (Minimum version: GNU (gcc/libstdc++) 9 or LLVM (clang/libc++) 7 or Microsoft (MSVC) 2019) 9 | - Check that you have CMake version 3.18.4 or later. 10 | - Get and unpack the latest version of the GROMACS tarball. 11 | - Make a separate build directory and change to it. 12 | - Run cmake with the path to the source as an argument 13 | - Run make, and make install 14 | - Source GMXRC to get access to GROMACS 15 | 16 | ### The System 17 | 18 | OS System: CentOS Linux release 7.6.1810 with CMake 3.27.6, CUDA 11.8, GCC 11.2.1, OpenMPI 4.1.6, and PMIX 2.2.2 operated by Lmod \ 19 | Computer System (Frontend node): 2 x NVIDIA GeForce GTX 1070 Ti 8GB with Intel(R) Core(TM) i7-7700K CPU @ 4.20GHz 20 | 21 | Refs: https://manual.gromacs.org/2024.2/install-guide/index.html 22 | 23 | ### Download GROMACS 2024.2 and Extract the tarball file 24 | ``` 25 | wget https://ftp.gromacs.org/gromacs/gromacs-2024.2.tar.gz 26 | tar xfz gromacs-2024.2.tar.gz 27 | ``` 28 | 29 | ### Compile and Install (with OpenMPI parallel and CUDA GPU) 30 | ``` 31 | cd gromacs-2024.2 32 | mkdir build 33 | cd build 34 | cmake .. -DGMX_BUILD_OWN_FFTW=ON -DREGRESSIONTEST_DOWNLOAD=OFF -DGMX_MPI=on -DGMX_BUILD_MDRUN_ONLY=off -DGMX_GPU=CUDA -DGMX_SIMD=AVX2_256 -DCMAKE_INSTALL_PREFIX=/xxx/gromacs-2024.2 35 | make install -j8 36 | sudo make install 37 | ``` 38 | 39 | ### Sample SLURM script for job submission 40 | ``` 41 | #!/bin/bash 42 | 43 | ################# Slurm Head ######################### 44 | 45 | # Set the number of nodes 46 | #SBATCH --nodes=1 47 | 48 | # Set the number of task(s) 49 | #SBATCH --ntasks=1 50 | 51 | # Set the number of task(s) per CPU core 52 | #SBATCH --ntasks-per-core=1 53 | 54 | # Set the number of GPU 55 | #SBATCH --gres=gpu:1 56 | 57 | # Set memory (RAM) for use 58 | #SBATCH --mem=8GB 59 | 60 | # Set wall time 61 | #SBATCH --time=3-00:00:00 62 | 63 | # Set partition 64 | #SBATCH --partition=qgpu_gtx1070ti 65 | 66 | # Set the name of a job 67 | #SBATCH --job-name=gromacs_mdrun 68 | 69 | ###################################################### 70 | 71 | # Set module that is used in this script. Check by command 'ml av' 72 | ml swap gnu7 gnu11 73 | ml cmake/3.27.6 cuda/11.8 openmpi4 pmix/2.2.2 74 | 75 | source /apps/gromacs-2024.2/bin/GMXRC 76 | 77 | # Minimization 78 | # In the case that there is a problem during minimization using a single precision of GROMACS, please try to use 79 | # a double precision of GROMACS only for the minimization step. 80 | mpirun -np 1 gmx_mpi grompp -f step4.0_minimization.mdp -o step4.0_minimization.tpr -c step3_input.gro -r step3_input.gro -p topol.top -n index.ndx -maxwarn +50 81 | mpirun -np 1 gmx_mpi mdrun -v -deffnm step4.0_minimization 82 | 83 | # Equilibration 84 | mpirun -np 1 gmx_mpi grompp -f step4.1_equilibration.mdp -o step4.1_equilibration.tpr -c step4.0_minimization.gro -r step4.0_minimization.gro -p topol.top -n index.ndx -maxwarn +50 85 | mpirun -np 1 gmx_mpi mdrun -v -deffnm step4.1_equilibration 86 | 87 | # Production 88 | mpirun -np 1 gmx_mpi grompp -f step5_production.mdp -o step5_production.tpr -c step4.1_equilibration.gro -p topol.top -n index.ndx -maxwarn +50 89 | mpirun -np 1 gmx_mpi mdrun -v -deffnm step5_production 90 | 91 | ``` 92 | -------------------------------------------------------------------------------- /17_AMBER25.md: -------------------------------------------------------------------------------- 1 | Amber is a suite of biomolecular simulation programs. It began in the late 1970's, and is maintained by an active development community. 2 | 3 | The term "Amber" refers to two things. First, it is a set of molecular mechanical force fields for the simulation of biomolecules (these force fields are in the public domain, and are used in a variety of simulation programs). Second, it is a package of molecular simulation programs which includes source code and demos. 4 | 5 | Cover of Manual for AMBER25 6 | 7 | Starting in 2025, Amber and AmberTools are separate releases, residing in different folders. Each can be compiled on its own, without the other. One improvement is that compilation of just the pmemd program (say on a high-performance computer system) is now much simpler in the past. A second advantage for many users will be this: AmberTools is available as a pre-compiled conda package (see ambermd.org/GetAmber.php) which can be installed via “conda install”. Users who choose this option only need to download and compile pmemd24.tar.bz2. 8 | 9 | **Ref:** https://ambermd.org/ and https://ambermd.org/doc12/Amber25.pdf \ 10 | **Download AmberTools25 and Amber24:** https://ambermd.org/GetAmber.php 11 | 12 | ### Pre-installation 13 | 14 | OS System: CentOS Linux release 7.6.1810 with CMake 3.20.1, CUDA 11.1.1, GCC 10.3.0, and OpenMPI 4.1.1 operated by Lmod Computer \ 15 | System: 4 x Tesla V100 SXM2 32GB with Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz 16 | 17 | #### Install Dependencies (For CentOS 7) 18 | ``` 19 | yum -y install tcsh make \ 20 | gcc gcc-gfortran gcc-c++ \ 21 | which flex bison patch bc \ 22 | libXt-devel libXext-devel \ 23 | perl perl-ExtUtils-MakeMaker util-linux wget \ 24 | bzip2 bzip2-devel zlib-devel tar 25 | ``` 26 | Other OS, please visit: https://ambermd.org/Installation.php 27 | 28 | #### Load modules 29 | ``` 30 | ml CMake/3.20.1-GCCcore-10.3.0 31 | ml OpenMPI/4.1.1-GCC-10.3.0 32 | ml CUDAcore/11.1.1 33 | ``` 34 | 35 | #### Extract AMBER 36 | ``` 37 | tar xvfj ambertools25.tar.bz2 38 | tar xvfj pmemd24.tar.bz2 39 | ``` 40 | 41 | ### AmberTools25 Installation 42 | **Note: please define destination folder for installation by -DCMAKE_INSTALL_PREFIX flag** 43 | 44 | #### Upgrade and update Amber 45 | ``` 46 | cd ambertools25_src 47 | ./update_amber --update 48 | ./update_amber --upgrade 49 | ``` 50 | 51 | #### Compile and install with Serial CPU 52 | ``` 53 | cd ambertools25_src 54 | cd build 55 | cmake .. -DCMAKE_INSTALL_PREFIX=/XXX/amber24_abt25 -DCOMPILER=GNU -DMPI=FALSE -DCUDA=FALSE -DINSTALL_TESTS=TRUE -DDOWNLOAD_MINICONDA=TRUE 56 | make -j32 && make install 57 | ``` 58 | 59 | #### Compile and install with Serial GPU 60 | ``` 61 | cd ambertools25_src 62 | cd build 63 | make clean 64 | cmake .. -DCMAKE_INSTALL_PREFIX=/XXX/amber24_abt25 -DCOMPILER=GNU -DMPI=FALSE -DCUDA=TRUE -DINSTALL_TESTS=TRUE -DDOWNLOAD_MINICONDA=TRUE 65 | make -j32 && make install 66 | ``` 67 | 68 | #### Compile and install with Parallel CPU 69 | ``` 70 | cd ambertools25_src 71 | cd build 72 | make clean 73 | cmake .. -DCMAKE_INSTALL_PREFIX=/XXX/amber24_abt25 -DCOMPILER=GNU -DCUDA=FALSE -DMPI=TRUE -DINSTALL_TESTS=TRUE -DDOWNLOAD_MINICONDA=TRUE 74 | make -j32 && make install 75 | ``` 76 | 77 | #### Compile and install with Parallel GPU 78 | ``` 79 | cd ambertools25_src 80 | cd build 81 | make clean 82 | cmake .. -DCMAKE_INSTALL_PREFIX=/ist/users/bunditb/bunditb_bak2/apps/amber24_abt25 -DCOMPILER=GNU -DMPI=FALSE -DCUDA=FALSE -DINSTALL_TESTS=TRUE -DDOWNLOAD_MINICONDA=FALSE -DBUILD_PYTHON=FALSE -DBUILD_PERL=FALSE -DBUILD_GUI=FALSE -DPMEMD_ONLY=TRUE -DCHECK_UPDATES=FALSE 83 | make -j32 && make install 84 | ``` 85 | 86 | ### Amber24 (pmemd24) Installation 87 | 88 | #### Upgrade and update Amber 89 | ``` 90 | cd pmemd24_src 91 | ./update_amber --update 92 | ./update_amber --upgrade 93 | ``` 94 | 95 | #### Compile and install with Serial CPU 96 | ``` 97 | cd pmemd24_src 98 | cd build 99 | cmake .. -DCMAKE_INSTALL_PREFIX=/XXX/amber24_abt25 -DCOMPILER=GNU -DMPI=FALSE -DCUDA=FALSE -DINSTALL_TESTS=TRUE -DDOWNLOAD_MINICONDA=FALSE -DBUILD_PYTHON=FALSE -DBUILD_PERL=FALSE -DBUILD_GUI=FALSE -DPMEMD_ONLY=TRUE -DCHECK_UPDATES=FALSE 100 | make -j32 && make install 101 | ``` 102 | 103 | #### Compile and install with Serial GPU 104 | ``` 105 | cd pmemd24_src 106 | cd build 107 | make clean 108 | cmake .. -DCMAKE_INSTALL_PREFIX=/XXX/amber24_abt25 -DCOMPILER=GNU -DMPI=FALSE -DCUDA=TRUE -DINSTALL_TESTS=TRUE -DDOWNLOAD_MINICONDA=FALSE -DBUILD_PYTHON=FALSE -DBUILD_PERL=FALSE -DBUILD_GUI=FALSE -DPMEMD_ONLY=TRUE -DCHECK_UPDATES=FALSE 109 | make -j32 && make install 110 | ``` 111 | 112 | #### Compile and install with Parallel CPU 113 | ``` 114 | cd pmemd24_src 115 | cd build 116 | make clean 117 | cmake .. -DCMAKE_INSTALL_PREFIX=/XXX/amber24_abt25 -DCOMPILER=GNU -DCOMPILER=GNU -DMPI=TRUE -DCUDA=FALSE -DINSTALL_TESTS=TRUE -DDOWNLOAD_MINICONDA=FALSE -DBUILD_PYTHON=FALSE -DBUILD_PERL=FALSE -DBUILD_GUI=FALSE -DPMEMD_ONLY=TRUE -DCHECK_UPDATES=FALSE 118 | make -j32 && make install 119 | ``` 120 | 121 | #### Compile and install with Parallel GPU 122 | ``` 123 | cd pmemd24_src 124 | cd build 125 | make clean 126 | cmake .. -DCMAKE_INSTALL_PREFIX=/XXX/amber24_abt25 -DCOMPILER=GNU -DMPI=TRUE -DCUDA=TRUE -DINSTALL_TESTS=TRUE -DDOWNLOAD_MINICONDA=FALSE -DBUILD_PYTHON=FALSE -DBUILD_PERL=FALSE -DBUILD_GUI=FALSE -DPMEMD_ONLY=TRUE -DCHECK_UPDATES=FALSE 127 | make -j32 && make install 128 | ``` 129 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ![OpenHPC Banner](https://repository-images.githubusercontent.com/156569976/556be300-746a-11e9-976f-abad06787c0f) 2 | 3 | ## OpenHPC Installation with PBS-Pro & Slurm job scheduler 4 | 5 | This guide is a method for OpenHPC 1.3.5 installtion which is supported for CentOS 7.5.1804 with diskless compute nodes. 6 | 7 | For more information please see at: https://openhpc.community/downloads/ 8 | 9 | If you have any question, pleace contact to me: bundit.b@hotmail.com 10 | 11 | Enjoys!!! 12 | 13 | ``` 14 | Credits: 15 | 1. Atip Peethong 16 | 17 | **OpenHPC with PBS Pro** 18 | Guide: https://drive.google.com/file/d/1du_daXvQIHRGQ9RFUaCRtQyUk2BRrJ3A/view 19 | 20 | How to Install OpenHPC (part 1/3)(2018): https://www.youtube.com/watch?v=jTBEgdZLmYM 21 | How to Install OpenHPC (part 2/3)(2018): https://www.youtube.com/watch?v=S3ab6iNdXag 22 | How to Install OpenHPC (part 2/3)(2018): https://www.youtube.com/watch?v=otHHpdpv7Ps 23 | 24 | **OpenHPC with Slurm** 25 | Guide: https://drive.google.com/file/d/1sUguvFtAVk62KYnMsivHepslxaDuyrzo/view 26 | 27 | How to Install OpenHPC Slurm (part 1/3)(2018): https://www.youtube.com/watch?v=7Am1WAqQl7M 28 | How to Install OpenHPC Slurm (part 2/3)(2018): https://www.youtube.com/watch?v=UfWeZ6k0KXM 29 | How to Install OpenHPC Slurm (part 3/3)(2018): https://www.youtube.com/watch?v=EwbSBq23RRk 30 | 31 | 2. Sombat Ketrat 32 | 3. https://github.com/dasandata/Open_HPC 33 | ``` 34 | -------------------------------------------------------------------------------- /_config.yml: -------------------------------------------------------------------------------- 1 | theme: jekyll-theme-architect --------------------------------------------------------------------------------