├── docker ├── .dockerignore ├── .gitignore ├── ssh_config ├── Dockerfile-0.5.1-GPU-IntelMPI-xenial ├── Dockerfile-0.5-GPU-OpenMPI-xenial ├── Dockerfile-0.4.2-GPU-OpenMPI-xenial ├── Dockerfile-0.5.1-GPU-OpenMPI-xenial ├── Dockerfile-0.5.3-GPU-OpenMPI-focal ├── Dockerfile-0.5-GPU-OpenMPI-xenial-devel ├── Dockerfile-0.4.2-GPU-OpenMPI-xenial-devel ├── Dockerfile-0.5.1-GPU-OpenMPI-xenial-devel ├── Dockerfile-0.5.3-GPU-OpenMPI-focal-devel ├── Dockerfile-0.5.1-GPU-IntelMPI-xenial-devel ├── Dockerfile-0.5.4-HPCX207-CUDA102-bionic ├── README.md └── Dockerfile-0.5.2-GPU-OpenMPI-centos7 ├── singularity ├── Singularity.0.5-GPU-OpenMPI-xenial ├── Singularity.0.4.2-GPU-OpenMPI-xenial ├── Singularity.0.5.1-GPU-OpenMPI-xenial ├── Singularity.0.5.3-GPU-OpenMPI-focal ├── Singularity.0.5.2-GPU-OpenMPI-centos7 ├── README.md └── Singularity.0.5.4-HPCX207-CUDA102-bionic ├── README.md └── LICENSE /docker/.dockerignore: -------------------------------------------------------------------------------- 1 | README* 2 | Dockerfile* 3 | -------------------------------------------------------------------------------- /docker/.gitignore: -------------------------------------------------------------------------------- 1 | # Ignore Intel MPI Library tarball 2 | l_mpi_2017.2.174.tgz 3 | -------------------------------------------------------------------------------- /docker/ssh_config: -------------------------------------------------------------------------------- 1 | Host 10.* 2 | Port 23 3 | StrictHostKeyChecking no 4 | UserKnownHostsFile /dev/null 5 | -------------------------------------------------------------------------------- /singularity/Singularity.0.5-GPU-OpenMPI-xenial: -------------------------------------------------------------------------------- 1 | Bootstrap: docker 2 | From: barbagroup/petibm:0.5-GPU-OpenMPI-xenial 3 | IncludeCmd: yes 4 | 5 | %labels 6 | AUTHOR mesnardo@gwu.edu 7 | -------------------------------------------------------------------------------- /singularity/Singularity.0.4.2-GPU-OpenMPI-xenial: -------------------------------------------------------------------------------- 1 | Bootstrap: docker 2 | From: barbagroup/petibm:0.4.2-GPU-OpenMPI-xenial 3 | IncludeCmd: yes 4 | 5 | %labels 6 | AUTHOR mesnardo@gwu.edu 7 | -------------------------------------------------------------------------------- /singularity/Singularity.0.5.1-GPU-OpenMPI-xenial: -------------------------------------------------------------------------------- 1 | Bootstrap: docker 2 | From: barbagroup/petibm:0.5.1-GPU-OpenMPI-xenial 3 | IncludeCmd: yes 4 | 5 | %labels 6 | AUTHOR mesnardo@gwu.edu 7 | -------------------------------------------------------------------------------- /singularity/Singularity.0.5.3-GPU-OpenMPI-focal: -------------------------------------------------------------------------------- 1 | Bootstrap: docker 2 | From: barbagroup/petibm:0.5.3-GPU-OpenMPI-focal 3 | IncludeCmd: yes 4 | 5 | %labels 6 | AUTHOR mesnardo@gwu.edu 7 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Docker and Singularity recipes for PetIBM 2 | 3 | [![License](https://img.shields.io/badge/License-BSD%203--Clause-blue.svg)](https://github.com/barbagroup/petibm-recipes/raw/master/LICENSE) 4 | [![Docker Hub](https://img.shields.io/badge/hosted-docker--hub-informational.svg)](https://cloud.docker.com/u/barbagroup/repository/docker/barbagroup/petibm) 5 | [![Singularity Hub](https://www.singularity-hub.org/static/img/hosted-singularity--hub-%23e32929.svg)](https://singularity-hub.org/collections/3692) 6 | 7 | This repository contains Dockerfiles and Singularity recipes used to build and share images of the runtime environment required for [PetIBM](https://github.com/barbagroup/PetIBM). 8 | Dockerfiles are located in the `docker` folder; Singularity recipes are in the `singularity` folder. 9 | 10 | Docker and Singularity images are available for: 11 | 12 | * PetIBM (0.4.2, 0.5, 0.5.1, 0.5.2, 0.5.3, 0.5.4) 13 | 14 | ## Contact 15 | 16 | To report bugs, submit questions, or offer suggestions, please use the GitHub issue tracking system. 17 | We also welcome pull-requests. 18 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | BSD 3-Clause License 2 | 3 | Copyright (c) 2019, Olivier Mesnard 4 | Copyright (c) 2020-2022, Olivier Mesnard, Pi-Yueh Chuang 5 | All rights reserved. 6 | 7 | Redistribution and use in source and binary forms, with or without 8 | modification, are permitted provided that the following conditions are met: 9 | 10 | * Redistributions of source code must retain the above copyright notice, this 11 | list of conditions and the following disclaimer. 12 | 13 | * Redistributions in binary form must reproduce the above copyright notice, 14 | this list of conditions and the following disclaimer in the documentation 15 | and/or other materials provided with the distribution. 16 | 17 | * Neither the name of the copyright holder nor the names of its 18 | contributors may be used to endorse or promote products derived from 19 | this software without specific prior written permission. 20 | 21 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 22 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 23 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 24 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 25 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 26 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 27 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 28 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 29 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 30 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 31 | -------------------------------------------------------------------------------- /singularity/Singularity.0.5.2-GPU-OpenMPI-centos7: -------------------------------------------------------------------------------- 1 | # vim:ft=singularity 2 | 3 | Bootstrap: docker 4 | From: barbagroup/petibm:0.5.2-GPU-OpenMPI-centos7 5 | 6 | # default to shell for "run" command; overwrite the Docker image's SSHD cmd 7 | %runscript 8 | exec /bin/bash 9 | 10 | %apprun poisson 11 | poisson $@ 12 | 13 | %apprun solveFromFile 14 | solveFromFile $@ 15 | 16 | %apprun createxdmf 17 | petibm-createxdmf $@ 18 | 19 | %apprun decoupledibpm 20 | petibm-decoupledibpm $@ 21 | 22 | %apprun ibpm 23 | petibm-ibpm $@ 24 | 25 | %apprun navierstokes 26 | petibm-navierstokes $@ 27 | 28 | %apprun vorticity 29 | petibm-vorticity $@ 30 | 31 | %apprun writemesh 32 | petibm-writemesh $@ 33 | 34 | %labels 35 | Author Pi-Yueh Chuang 36 | PetIBM v0.5.2 37 | PETSc v3.12.5 38 | OpenMPI v4.0.5 39 | CUDA v10.1 40 | AmgX 919fb92 41 | AmgXWrapper 1a93865 42 | 43 | %help 44 | 45 | $ singularity run 46 | 47 | This will start a shell in the container. 48 | 49 | 50 | $ singularity run --app [app arguments] 51 | 52 | apps: 53 | poisson: from the Poisson example in AmgXWrapper 54 | solveFromFile: from the solveFromFile example in AmgXWrapper 55 | createxdmf: PetIBM application code. 56 | decoupledibpm: PetIBM application code. 57 | ibpm: PetIBM application code. 58 | navierstokes: PetIBM application code. 59 | vorticity: PetIBM application code. 60 | writemesh: PetIBM application code. 61 | 62 | $ singularity exec 63 | 64 | This will execute the specified program in container. 65 | -------------------------------------------------------------------------------- /docker/Dockerfile-0.5.1-GPU-IntelMPI-xenial: -------------------------------------------------------------------------------- 1 | # Dockerfile for PetIBM-0.5.1 for use with Batch Shipyard on Azure Batch 2 | 3 | FROM barbagroup/petibm:0.5.1-GPU-IntelMPI-xenial-devel as build 4 | FROM nvidia/cuda:10.1-runtime-ubuntu16.04 5 | MAINTAINER Olivier Mesnard 6 | 7 | # Install base system. 8 | COPY ssh_config /root/.ssh/config 9 | RUN apt-get update && \ 10 | # Infiniband/RDMA 11 | apt-get install -y --no-install-recommends \ 12 | cpio \ 13 | libmlx4-1 \ 14 | libmlx5-1 \ 15 | librdmacm1 \ 16 | libibverbs1 \ 17 | libmthca1 \ 18 | libdapl2 \ 19 | dapl2-utils && \ 20 | # Batch-Shipyard dependencies 21 | apt-get install -y --no-install-recommends \ 22 | openssh-server \ 23 | openssh-client && \ 24 | rm -rf /var/lib/apt/lists/* && \ 25 | # Configure ssh server and keys 26 | mkdir /var/run/sshd && \ 27 | ssh-keygen -A && \ 28 | sed -i 's/PermitRootLogin without-password/PermitRootLogin yes/' /etc/ssh/sshd_config && \ 29 | sed 's@session\s*required\s*pam_loginuid.so@session optional pam_loginuid.so@g' -i /etc/pam.d/sshd && \ 30 | ssh-keygen -f /root/.ssh/id_rsa -t rsa -N '' && \ 31 | chmod 600 /root/.ssh/config && \ 32 | chmod 700 /root/.ssh && \ 33 | cp /root/.ssh/id_rsa.pub /root/.ssh/authorized_keys 34 | 35 | ENV INTEL_DIR=/opt/intel2 36 | ENV PETSC_VERSION=3.12.5 37 | ENV PETSC_DIR=/usr/local/petsc-${PETSC_VERSION} 38 | ENV AMGX_VERSION=2.1 39 | ENV AMGX_DIR=/usr/local/amgx-${AMGX_VERSION} 40 | ENV PETIBM_VERSION=0.5.1 41 | ENV PETIBM_DIR=/usr/local/petibm-${PETIBM_VERSION} 42 | ENV PATH=${PETIBM_DIR}/bin:${PATH} 43 | COPY --from=build ${INTEL_DIR} ${INTEL_DIR} 44 | COPY --from=build ${PETSC_DIR} ${PETSC_DIR} 45 | COPY --from=build ${AMGX_DIR} ${AMGX_DIR} 46 | COPY --from=build ${PETIBM_DIR} ${PETIBM_DIR} 47 | 48 | # Set sshd command. 49 | EXPOSE 23 50 | CMD ["/usr/sbin/sshd", "-D", "-p", "23"] 51 | -------------------------------------------------------------------------------- /docker/Dockerfile-0.5-GPU-OpenMPI-xenial: -------------------------------------------------------------------------------- 1 | # Dockerfile for PetIBM-0.5 2 | 3 | FROM barbagroup/petibm:0.5-GPU-OpenMPI-xenial-devel as build 4 | FROM nvidia/cuda:10.1-runtime-ubuntu16.04 5 | MAINTAINER Olivier Mesnard 6 | 7 | COPY ssh_config /root/.ssh/config 8 | RUN apt-get update && \ 9 | apt-get install -y --no-install-recommends \ 10 | gfortran g++ && \ 11 | apt-get install -y --no-install-recommends \ 12 | libibverbs1 libibverbs-dev ibverbs-utils librdmacm1 rdmacm-utils \ 13 | ibutils ibacm libibcm1 libibmad5 libibumad3 opensm libopensm5a \ 14 | srptools perftest infiniband-diags ibsim-utils \ 15 | libmthca1 libmlx4-1 libmlx5-1 mstflint \ 16 | libdapl2 dapl2-utils && \ 17 | apt-get install -y --no-install-recommends \ 18 | openssh-server \ 19 | openssh-client && \ 20 | rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* && \ 21 | mkdir /var/run/sshd && \ 22 | ssh-keygen -A && \ 23 | sed -i 's/PermitRootLogin without-password/PermitRootLogin yes/' /etc/ssh/sshd_config && \ 24 | sed 's@session\s*required\s*pam_loginuid.so@session optional pam_loginuid.so@g' -i /etc/pam.d/sshd && \ 25 | ssh-keygen -f /root/.ssh/id_rsa -t rsa -N '' && \ 26 | chmod 600 /root/.ssh/config && \ 27 | chmod 700 /root/.ssh && \ 28 | cp /root/.ssh/id_rsa.pub /root/.ssh/authorized_keys 29 | 30 | ENV OMPI_VERSION=3.1.4 31 | ENV OMPI_DIR=/usr/local/openmpi-${OMPI_VERSION} 32 | ENV PETSC_VERSION=3.12.2 33 | ENV PETSC_DIR=/usr/local/petsc-${PETSC_VERSION} 34 | ENV AMGX_VERSION=2.1 35 | ENV AMGX_DIR=/usr/local/amgx-${AMGX_VERSION} 36 | ENV PETIBM_VERSION=0.5 37 | ENV PETIBM_DIR=/usr/local/petibm-${PETIBM_VERSION} 38 | ENV PATH=${PETIBM_DIR}/bin:${OMPI_DIR}/bin:${PATH} 39 | COPY --from=build ${OMPI_DIR} ${OMPI_DIR} 40 | COPY --from=build ${PETSC_DIR} ${PETSC_DIR} 41 | COPY --from=build ${AMGX_DIR} ${AMGX_DIR} 42 | COPY --from=build ${PETIBM_DIR} ${PETIBM_DIR} 43 | 44 | EXPOSE 23 45 | CMD ["/usr/sbin/sshd", "-D", "-p", "23"] 46 | -------------------------------------------------------------------------------- /docker/Dockerfile-0.4.2-GPU-OpenMPI-xenial: -------------------------------------------------------------------------------- 1 | # Dockerfile for PetIBM-0.4.2 2 | 3 | FROM barbagroup/petibm:0.4.2-GPU-OpenMPI-xenial-devel as build 4 | FROM nvidia/cuda:10.1-runtime-ubuntu16.04 5 | MAINTAINER Olivier Mesnard 6 | 7 | COPY ssh_config /root/.ssh/config 8 | RUN apt-get update && \ 9 | apt-get install -y --no-install-recommends \ 10 | gfortran g++ && \ 11 | apt-get install -y --no-install-recommends \ 12 | libibverbs1 libibverbs-dev ibverbs-utils librdmacm1 rdmacm-utils \ 13 | ibutils ibacm libibcm1 libibmad5 libibumad3 opensm libopensm5a \ 14 | srptools perftest infiniband-diags ibsim-utils \ 15 | libmthca1 libmlx4-1 libmlx5-1 mstflint \ 16 | libdapl2 dapl2-utils && \ 17 | apt-get install -y --no-install-recommends \ 18 | openssh-server \ 19 | openssh-client && \ 20 | rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* && \ 21 | mkdir /var/run/sshd && \ 22 | ssh-keygen -A && \ 23 | sed -i 's/PermitRootLogin without-password/PermitRootLogin yes/' /etc/ssh/sshd_config && \ 24 | sed 's@session\s*required\s*pam_loginuid.so@session optional pam_loginuid.so@g' -i /etc/pam.d/sshd && \ 25 | ssh-keygen -f /root/.ssh/id_rsa -t rsa -N '' && \ 26 | chmod 600 /root/.ssh/config && \ 27 | chmod 700 /root/.ssh && \ 28 | cp /root/.ssh/id_rsa.pub /root/.ssh/authorized_keys 29 | 30 | ENV OMPI_VERSION=3.1.4 31 | ENV OMPI_DIR=/usr/local/openmpi-${OMPI_VERSION} 32 | ENV PETSC_VERSION=3.11.4 33 | ENV PETSC_DIR=/usr/local/petsc-${PETSC_VERSION} 34 | ENV AMGX_VERSION=2.1 35 | ENV AMGX_DIR=/usr/local/amgx-${AMGX_VERSION} 36 | ENV PETIBM_VERSION=0.4.2 37 | ENV PETIBM_DIR=/usr/local/petibm-${PETIBM_VERSION} 38 | ENV PATH=${PETIBM_DIR}/bin:${OMPI_DIR}/bin:${PATH} 39 | COPY --from=build ${OMPI_DIR} ${OMPI_DIR} 40 | COPY --from=build ${PETSC_DIR} ${PETSC_DIR} 41 | COPY --from=build ${AMGX_DIR} ${AMGX_DIR} 42 | COPY --from=build ${PETIBM_DIR} ${PETIBM_DIR} 43 | 44 | EXPOSE 23 45 | CMD ["/usr/sbin/sshd", "-D", "-p", "23"] 46 | -------------------------------------------------------------------------------- /docker/Dockerfile-0.5.1-GPU-OpenMPI-xenial: -------------------------------------------------------------------------------- 1 | # Dockerfile for PetIBM-0.5.1 2 | 3 | FROM barbagroup/petibm:0.5.1-GPU-OpenMPI-xenial-devel as build 4 | FROM nvidia/cuda:10.1-runtime-ubuntu16.04 5 | MAINTAINER Olivier Mesnard 6 | 7 | COPY ssh_config /root/.ssh/config 8 | RUN apt-get update && \ 9 | apt-get install -y --no-install-recommends \ 10 | gfortran g++ && \ 11 | apt-get install -y --no-install-recommends \ 12 | libibverbs1 libibverbs-dev ibverbs-utils librdmacm1 rdmacm-utils \ 13 | ibutils ibacm libibcm1 libibmad5 libibumad3 opensm libopensm5a \ 14 | srptools perftest infiniband-diags ibsim-utils \ 15 | libmthca1 libmlx4-1 libmlx5-1 mstflint \ 16 | libdapl2 dapl2-utils && \ 17 | apt-get install -y --no-install-recommends \ 18 | openssh-server \ 19 | openssh-client && \ 20 | rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* && \ 21 | mkdir /var/run/sshd && \ 22 | ssh-keygen -A && \ 23 | sed -i 's/PermitRootLogin without-password/PermitRootLogin yes/' /etc/ssh/sshd_config && \ 24 | sed 's@session\s*required\s*pam_loginuid.so@session optional pam_loginuid.so@g' -i /etc/pam.d/sshd && \ 25 | ssh-keygen -f /root/.ssh/id_rsa -t rsa -N '' && \ 26 | chmod 600 /root/.ssh/config && \ 27 | chmod 700 /root/.ssh && \ 28 | cp /root/.ssh/id_rsa.pub /root/.ssh/authorized_keys 29 | 30 | ENV OMPI_VERSION=3.1.4 31 | ENV OMPI_DIR=/usr/local/openmpi-${OMPI_VERSION} 32 | ENV PETSC_VERSION=3.12.2 33 | ENV PETSC_DIR=/usr/local/petsc-${PETSC_VERSION} 34 | ENV AMGX_VERSION=2.1 35 | ENV AMGX_DIR=/usr/local/amgx-${AMGX_VERSION} 36 | ENV PETIBM_VERSION=0.5.1 37 | ENV PETIBM_DIR=/usr/local/petibm-${PETIBM_VERSION} 38 | ENV PATH=${PETIBM_DIR}/bin:${OMPI_DIR}/bin:${PATH} 39 | COPY --from=build ${OMPI_DIR} ${OMPI_DIR} 40 | COPY --from=build ${PETSC_DIR} ${PETSC_DIR} 41 | COPY --from=build ${AMGX_DIR} ${AMGX_DIR} 42 | COPY --from=build ${PETIBM_DIR} ${PETIBM_DIR} 43 | 44 | EXPOSE 23 45 | CMD ["/usr/sbin/sshd", "-D", "-p", "23"] 46 | -------------------------------------------------------------------------------- /docker/Dockerfile-0.5.3-GPU-OpenMPI-focal: -------------------------------------------------------------------------------- 1 | # Dockerfile for PetIBM-0.5.3 2 | 3 | FROM barbagroup/petibm:0.5.3-GPU-OpenMPI-focal-devel as build 4 | FROM nvidia/cuda:11.6.0-runtime-ubuntu20.04 5 | MAINTAINER Olivier Mesnard 6 | 7 | COPY ssh_config /root/.ssh/config 8 | ENV DEBIAN_FRONTEND=noninteractive 9 | RUN apt-get update && \ 10 | apt-get install -y --no-install-recommends \ 11 | gfortran g++ && \ 12 | apt-get install -y --no-install-recommends \ 13 | libibverbs1 libibverbs-dev ibverbs-utils librdmacm1 rdmacm-utils \ 14 | ibutils ibacm libibmad5 libibumad3 opensm libopensm8 \ 15 | srptools perftest infiniband-diags ibsim-utils \ 16 | libmthca1 libmlx4-1 libmlx5-1 mstflint \ 17 | libdapl2 dapl2-utils && \ 18 | apt-get install -y --no-install-recommends \ 19 | openssh-server \ 20 | openssh-client && \ 21 | rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* && \ 22 | mkdir /var/run/sshd && \ 23 | ssh-keygen -A && \ 24 | sed -i 's/PermitRootLogin without-password/PermitRootLogin yes/' /etc/ssh/sshd_config && \ 25 | sed 's@session\s*required\s*pam_loginuid.so@session optional pam_loginuid.so@g' -i /etc/pam.d/sshd && \ 26 | ssh-keygen -f /root/.ssh/id_rsa -t rsa -N '' && \ 27 | chmod 600 /root/.ssh/config && \ 28 | chmod 700 /root/.ssh && \ 29 | cp /root/.ssh/id_rsa.pub /root/.ssh/authorized_keys 30 | 31 | ENV OMPI_VERSION=3.1.4 32 | ENV OMPI_DIR=/usr/local/openmpi-${OMPI_VERSION} 33 | ENV PETSC_VERSION=3.16.5 34 | ENV PETSC_DIR=/usr/local/petsc-${PETSC_VERSION} 35 | ENV AMGX_VERSION=2.2.0 36 | ENV AMGX_DIR=/usr/local/amgx-${AMGX_VERSION} 37 | ENV PETIBM_VERSION=0.5.3 38 | ENV PETIBM_DIR=/usr/local/petibm-${PETIBM_VERSION} 39 | ENV PATH=${PETIBM_DIR}/bin:${OMPI_DIR}/bin:${PATH} 40 | COPY --from=build ${OMPI_DIR} ${OMPI_DIR} 41 | COPY --from=build ${PETSC_DIR} ${PETSC_DIR} 42 | COPY --from=build ${AMGX_DIR} ${AMGX_DIR} 43 | COPY --from=build ${PETIBM_DIR} ${PETIBM_DIR} 44 | 45 | EXPOSE 23 46 | CMD ["/usr/sbin/sshd", "-D", "-p", "23"] 47 | -------------------------------------------------------------------------------- /singularity/README.md: -------------------------------------------------------------------------------- 1 | # Singularity recipes for PetIBM 2 | 3 | The present directory contains the Singularity recipes used to create Singularity images for PetIBM on [Singularity Hub](https://singularity-hub.org/) (in [this collection](https://singularity-hub.org/collections/3692)). 4 | 5 | ## `0.5.4-hpcx207-cuda102` 6 | 7 | Not added to the collection as Singularity Hub has been archived. 8 | This image is based on the Docker image `barbagroup/petibm:0.5.4-hpcx207-cuda102`. 9 | The basic way to build this Apptainer (the new name of Singularity; works exactly the same as Singularity) image locally is: 10 | 11 | ``` 12 | $ sudo apptainer build petibm-0.5.4-hpcx207-cuda102.sif ./Singularity.0.5.4-HPCX207-CUDA102-bionic 13 | ``` 14 | 15 | By default, Apptainer uses RAM memory for the temporary directory. 16 | If you encounter out-of-memory, you may need to use a disk as the temporary folder by setting the environment variable: 17 | 18 | ``` 19 | $ export APPTAINER_TMPDIR= 20 | ``` 21 | 22 | If you don't have the root privilege but Apptainer is configured properly, you may be able to use fakeroot: 23 | 24 | ``` 25 | $ apptainer build --fakeroot petibm-0.5.4-hpcx207-cuda102.sif ./Singularity.0.5.4-HPCX207-CUDA102-bionic 26 | ``` 27 | 28 | ## `0.5.3-gpu-openmpi-focal` 29 | 30 | Not added to the collection as Singularity Hub has been Archived. 31 | 32 | ## `0.5.2-gpu-openmpi-centos7` 33 | 34 | To pull the Singularity image from the collection: 35 | 36 | ```shell 37 | singularity pull --name petibm-0.5.2_centos7.sif shub://barbagroup/petibm-recipes:0.5.2-gpu-openmpi-centos7 38 | ``` 39 | 40 | ## `0.5.1-gpu-openmpi-xenial` 41 | 42 | To pull the Singularity image from the collection: 43 | 44 | ```shell 45 | singularity pull --name petibm-0.5.1_xenial.sif shub://barbagroup/petibm-recipes:0.5.1-gpu-openmpi-xenial 46 | ``` 47 | 48 | ## `0.5-gpu-openmpi-xenial` 49 | 50 | To pull the Singularity image from the collection: 51 | 52 | ```shell 53 | singularity pull --name petibm-0.5_xenial.sif shub://barbagroup/petibm-recipes:0.5-gpu-openmpi-xenial 54 | ``` 55 | 56 | ## `0.4.2-gpu-openmpi-xenial` 57 | 58 | To pull the Singularity image from the collection: 59 | 60 | ```shell 61 | singularity pull --name petibm-0.4.2_xenial.sif shub://barbagroup/petibm-recipes:0.4.2-gpu-openmpi-xenial 62 | ``` 63 | -------------------------------------------------------------------------------- /singularity/Singularity.0.5.4-HPCX207-CUDA102-bionic: -------------------------------------------------------------------------------- 1 | # vim:ft=singularity 2 | Bootstrap: docker 3 | From: barbagroup/petibm:0.5.4-hpcx207-cuda102 4 | 5 | 6 | # environment variables 7 | %environment 8 | export HPCX_DIR=/hpcx 9 | export HPCX_UCX_DIR=${HPCX_DIR}/ucx 10 | export HPCX_HMC_DIR=${HPCX_DIR}/hmc 11 | export HPCX_HCOLL_DIR=${HPCX_DIR}/hcoll 12 | export HPCX_MPI_DIR=${HPCX_DIR}/ompi 13 | export HPCX_OSHMEM_DIR=${HPCX_MPI_DIR} 14 | export HPCX_MPI_TESTS_DIR=${HPCX_MPI_DIR}/tests 15 | export HPCX_OSU_DIR=${HPCX_MPI_TESTS_DIR}/osu-micro-benchmarks-5.6.2 16 | export HPCX_OSU_CUDA_DIR=${HPCX_MPI_TESTS_DIR}/osu-micro-benchmarks-5.6.2-cuda 17 | export HPCX_IPM_DIR=${HPCX_MPI_TESTS_DIR}/ipm-2.0.6 18 | export HPCX_IPM_LIB=${HPCX_IPM_DIR}/lib/libipm.so 19 | export HPCX_CLUSTERKIT_DIR=${HPCX_DIR}/clusterkit 20 | export OMPI_HOME=${HPCX_MPI_DIR} 21 | export MPI_HOME=${HPCX_MPI_DIR} 22 | export OSHMEM_HOME=${HPCX_MPI_DIR} 23 | export SHMEM_HOME=${HPCX_MPI_DIR} 24 | 25 | export OPAL_PREFIX=${HPCX_MPI_DIR} 26 | export PATH=${HPCX_MPI_DIR}/bin:${PATH} 27 | export PATH=${HPCX_UCX_DIR}/bin:${PATH} 28 | export PATH=${HPCX_HCOLL_DIR}/bin:${PATH} 29 | export PATH=${HPCX_CLUSTERKIT_DIR}/bin:${PATH} 30 | 31 | export LD_LIBRARY_PATH=${HPCX_MPI_DIR}/lib:${LD_LIBRARY_PATH} 32 | export LD_LIBRARY_PATH=${HPCX_HCOLL_DIR}/lib:${LD_LIBRARY_PATH} 33 | export LD_LIBRARY_PATH=${HPCX_SHARP_DIR}/lib:${LD_LIBRARY_PATH} 34 | export LD_LIBRARY_PATH=${HPCX_UCX_DIR}/lib:${LD_LIBRARY_PATH} 35 | export LD_LIBRARY_PATH=${HPCX_UCX_DIR}/lib/ucx:${LD_LIBRARY_PATH} 36 | export LD_LIBRARY_PATH=${HPCX_HMC_DIR}/lib:${LD_LIBRARY_PATH} 37 | export LD_LIBRARY_PATH=${HPCX_NCCL_RDMA_SHARP_PLUGIN_DIR}/lib:${LD_LIBRARY_PATH} 38 | 39 | export LIBRARY_PATH=${HPCX_MPI_DIR}/lib:${LIBRARY_PATH} 40 | export LIBRARY_PATH=${HPCX_HCOLL_DIR}/lib:${LIBRARY_PATH} 41 | export LIBRARY_PATH=${HPCX_UCX_DIR}/lib:${LIBRARY_PATH} 42 | export LIBRARY_PATH=${HPCX_HMC_DIR}/lib:${LIBRARY_PATH} 43 | export LIBRARY_PATH=${HPCX_SHARP_DIR}/lib:${LIBRARY_PATH} 44 | export LIBRARY_PATH=${HPCX_MPI_DIR}/lib:${LIBRARY_PATH} 45 | export LIBRARY_PATH=${HPCX_NCCL_RDMA_SHARP_PLUGIN_DIR}/lib:${LIBRARY_PATH} 46 | 47 | export CPATH=${HPCX_HCOLL_DIR}/include:${CPATH} 48 | export CPATH=${HPCX_SHARP_DIR}/include:${CPATH} 49 | export CPATH=${HPCX_UCX_DIR}/include:${CPATH} 50 | export CPATH=${HPCX_HMC_DIR}/include:${CPATH} 51 | export CPATH=${HPCX_MPI_DIR}/include:${CPATH} 52 | 53 | export PKG_CONFIG_PATH=${HPCX_MPI_DIR}/lib/pkgconfig:${PKG_CONFIG_PATH} 54 | export PKG_CONFIG_PATH=${HPCX_UCX_DIR}/lib/pkgconfig:${PKG_CONFIG_PATH} 55 | export PKG_CONFIG_PATH=${HPCX_SHARP_DIR}/lib/pkgconfig:${PKG_CONFIG_PATH} 56 | export PKG_CONFIG_PATH=${HPCX_HCOLL_DIR}/lib/pkgconfig:${PKG_CONFIG_PATH} 57 | 58 | # default to shell for "run" command; overwrite the Docker image's SSHD cmd 59 | %runscript 60 | exec /bin/bash "$@" 61 | 62 | %labels 63 | Author Pi-Yueh Chuang 64 | CUDA 10.2 65 | HPCX 2.07.4 66 | PETSc 3.16.6 67 | AmgX 6f1401 68 | AmgXWrapper 1.6.2 69 | PetIBM 0.5.4 70 | -------------------------------------------------------------------------------- /docker/Dockerfile-0.5-GPU-OpenMPI-xenial-devel: -------------------------------------------------------------------------------- 1 | # Dockerfile for PetIBM-0.5 2 | 3 | FROM nvidia/cuda:10.1-devel-ubuntu16.04 as build 4 | MAINTAINER Olivier Mesnard 5 | 6 | ENV CUDA_VERSION=10.1 7 | ENV CUDA_DIR=/usr/local/cuda-${CUDA_VERSION} 8 | 9 | # Install base system. 10 | COPY ssh_config /root/.ssh/config 11 | RUN apt-get update && \ 12 | apt-get install -y --no-install-recommends \ 13 | autotools-dev \ 14 | bison \ 15 | build-essential \ 16 | ca-certificates \ 17 | cmake \ 18 | curl \ 19 | flex \ 20 | gfortran \ 21 | git \ 22 | pkg-config \ 23 | python-dev \ 24 | unzip \ 25 | wget && \ 26 | apt-get install -y --no-install-recommends \ 27 | libibverbs1 libibverbs-dev ibverbs-utils librdmacm1 rdmacm-utils \ 28 | ibutils ibacm libibcm1 libibmad5 libibumad3 opensm libopensm5a \ 29 | srptools perftest infiniband-diags ibsim-utils \ 30 | libmthca1 libmlx4-1 libmlx5-1 mstflint \ 31 | libdapl2 dapl2-utils && \ 32 | apt-get install -y --no-install-recommends \ 33 | openssh-server \ 34 | openssh-client && \ 35 | rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* && \ 36 | mkdir /var/run/sshd && \ 37 | ssh-keygen -A && \ 38 | sed -i 's/PermitRootLogin without-password/PermitRootLogin yes/' /etc/ssh/sshd_config && \ 39 | sed 's@session\s*required\s*pam_loginuid.so@session optional pam_loginuid.so@g' -i /etc/pam.d/sshd && \ 40 | ssh-keygen -f /root/.ssh/id_rsa -t rsa -N '' && \ 41 | chmod 600 /root/.ssh/config && \ 42 | chmod 700 /root/.ssh && \ 43 | cp /root/.ssh/id_rsa.pub /root/.ssh/authorized_keys 44 | 45 | # Build OpenMPI. 46 | ENV OMPI_SERIES=3.1 47 | ENV OMPI_VERSION=3.1.4 48 | RUN TARBALL=openmpi-${OMPI_VERSION}.tar.gz && \ 49 | URL=https://download.open-mpi.org/release/open-mpi/v${OMPI_SERIES}/${TARBALL} && \ 50 | wget ${URL} -P /tmp && \ 51 | OMPI_DIR=/opt/openmpi/${OMPI_VERSION} && \ 52 | mkdir -p ${OMPI_DIR}/build && \ 53 | tar -xzf /tmp/${TARBALL} -C ${OMPI_DIR} --strip-components=1 && \ 54 | cd ${OMPI_DIR}/build && \ 55 | ${OMPI_DIR}/configure --prefix=/usr/local/openmpi-${OMPI_VERSION} && \ 56 | make && make install && \ 57 | rm -f /tmp/${TARBALL} 58 | ENV OMPI_DIR=/usr/local/openmpi-${OMPI_VERSION} 59 | 60 | ENV PATH=${OMPI_DIR}/bin:${PATH} 61 | ENV CPATH=${OMPI_DIR}/include:${CPATH} 62 | 63 | # Build PETSc in optimized mode. 64 | ENV PETSC_VERSION=3.12.2 65 | RUN TARBALL=petsc-lite-${PETSC_VERSION}.tar.gz && \ 66 | URL=http://ftp.mcs.anl.gov/pub/petsc/release-snapshots/${TARBALL} && \ 67 | wget ${URL} -P /tmp && \ 68 | PETSC_DIR=/opt/petsc/${PETSC_VERSION} && \ 69 | PETSC_ARCH=linux-gnu-openmpi-opt && \ 70 | mkdir -p ${PETSC_DIR} && \ 71 | tar -xzf /tmp/${TARBALL} -C ${PETSC_DIR} --strip-components=1 && \ 72 | cd ${PETSC_DIR} && \ 73 | ./configure \ 74 | --prefix=/usr/local/petsc-${PETSC_VERSION} \ 75 | PETSC_DIR=${PETSC_DIR} \ 76 | PETSC_ARCH=${PETSC_ARCH} \ 77 | COPTFLAGS="-O3" \ 78 | CXXOPTFLAGS="-O3" \ 79 | --with-fc=0 \ 80 | --with-shared-libraries=1 \ 81 | --with-pic=1 \ 82 | --with-cxx-dialect="C++11" \ 83 | --with-debugging=0 \ 84 | --with-mpi-dir=${OMPI_DIR} \ 85 | --download-hdf5 \ 86 | --download-f2cblaslapack \ 87 | --download-hypre \ 88 | --download-ptscotch \ 89 | --download-metis \ 90 | --download-parmetis \ 91 | --download-superlu_dist && \ 92 | make PETSC_DIR=${PETSC_DIR} PETSC_ARCH=${PETSC_ARCH} all && \ 93 | make PETSC_DIR=${PETSC_DIR} PETSC_ARCH=${PETSC_ARCH} install && \ 94 | rm -f /tmp/${TARBALL} 95 | ENV PETSC_DIR=/usr/local/petsc-${PETSC_VERSION} 96 | 97 | # Build AmgX. 98 | ENV AMGX_VERSION=2.1 99 | ENV AMGX_SHA=cf285c118726f5d1e8eb740d4936dd0bdaaf9b48 100 | RUN REPO=https://github.com/NVIDIA/AMGX && \ 101 | AMGX_DIR=/opt/amgx/${AMGX_VERSION} && \ 102 | AMGX_ARCH=linux-gnu-openmpi-opt && \ 103 | git clone ${REPO} ${AMGX_DIR} && \ 104 | cd ${AMGX_DIR} && \ 105 | git checkout -b use4petibm ${AMGX_SHA} && \ 106 | BUILDDIR=${AMGX_DIR}/${AMGX_ARCH}/build && \ 107 | mkdir -p ${BUILDDIR} && \ 108 | cd ${BUILDDIR} && \ 109 | cmake ${AMGX_DIR} \ 110 | -DCMAKE_BUILD_TYPE="Release" \ 111 | -DCMAKE_INSTALL_PREFIX=/usr/local/amgx-${AMGX_VERSION} \ 112 | -DCMAKE_C_COMPILER=mpicc \ 113 | -DCMAKE_C_FLAGS_PROFILE="-O3 -DNDEBUG" \ 114 | -DCMAKE_CXX_COMPILER=mpicxx \ 115 | -DCMAKE_CXX_FLAGS_PROFILE="-O3 -DNDEBUG" \ 116 | -DMPI_CXX_COMPILER=mpicxx \ 117 | -DMPI_C_COMPILER=mpicc \ 118 | -DCUDA_ARCH="35 37 60 70" \ 119 | -DCUDA_HOST_COMPILER=/usr/bin/gcc-5 && \ 120 | make -j"$(nproc)" all && \ 121 | make install 122 | ENV AMGX_DIR=/usr/local/amgx-${AMGX_VERSION} 123 | 124 | # Build and install PetIBM. 125 | ENV PETIBM_VERSION=0.5 126 | RUN TARBALL=v${PETIBM_VERSION}.tar.gz && \ 127 | URL=https://github.com/barbagroup/PetIBM/archive/${TARBALL} && \ 128 | wget ${URL} -P /tmp && \ 129 | PETIBM_DIR=/opt/petibm/${PETIBM_VERSION} && \ 130 | PETIBM_ARCH=linux-gnu-openmpi-opt && \ 131 | BUILDDIR=${PETIBM_DIR}/${PETIBM_ARCH}/build && \ 132 | mkdir -p ${BUILDDIR} && \ 133 | tar -xzf /tmp/${TARBALL} -C ${PETIBM_DIR} --strip-components=1 && \ 134 | cd ${BUILDDIR} && \ 135 | ${PETIBM_DIR}/configure --prefix=/usr/local/petibm-${PETIBM_VERSION} \ 136 | CC=mpicc \ 137 | CFLAGS="-w -O3 -std=c++14" \ 138 | CXX=mpicxx \ 139 | CXXFLAGS="-w -O3 -std=c++14" \ 140 | --enable-static=no \ 141 | --with-petsc-dir=${PETSC_DIR} \ 142 | --with-petsc-arch="" \ 143 | --with-cuda-dir=${CUDA_DIR} \ 144 | --with-amgx-dir=${AMGX_DIR} \ 145 | --enable-amgxwrapper \ 146 | --enable-yamlcpp \ 147 | --enable-gtest && \ 148 | make -j"$(nproc)" all && \ 149 | make check && \ 150 | make install && \ 151 | rm -f /tmp/${TARBALL} 152 | ENV PETIBM_DIR=/usr/local/petibm-${PETIBM_VERSION} 153 | 154 | EXPOSE 23 155 | CMD ["/usr/sbin/sshd", "-D", "-p", "23"] 156 | -------------------------------------------------------------------------------- /docker/Dockerfile-0.4.2-GPU-OpenMPI-xenial-devel: -------------------------------------------------------------------------------- 1 | # Dockerfile for PetIBM-0.4.2 2 | 3 | FROM nvidia/cuda:10.1-devel-ubuntu16.04 as build 4 | MAINTAINER Olivier Mesnard 5 | 6 | ENV CUDA_VERSION=10.1 7 | ENV CUDA_DIR=/usr/local/cuda-${CUDA_VERSION} 8 | 9 | # Install base system. 10 | COPY ssh_config /root/.ssh/config 11 | RUN apt-get update && \ 12 | apt-get install -y --no-install-recommends \ 13 | autotools-dev \ 14 | bison \ 15 | build-essential \ 16 | ca-certificates \ 17 | cmake \ 18 | curl \ 19 | flex \ 20 | gfortran \ 21 | git \ 22 | pkg-config \ 23 | python-dev \ 24 | unzip \ 25 | wget && \ 26 | apt-get install -y --no-install-recommends \ 27 | libibverbs1 libibverbs-dev ibverbs-utils librdmacm1 rdmacm-utils \ 28 | ibutils ibacm libibcm1 libibmad5 libibumad3 opensm libopensm5a \ 29 | srptools perftest infiniband-diags ibsim-utils \ 30 | libmthca1 libmlx4-1 libmlx5-1 mstflint \ 31 | libdapl2 dapl2-utils && \ 32 | apt-get install -y --no-install-recommends \ 33 | openssh-server \ 34 | openssh-client && \ 35 | rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* && \ 36 | mkdir /var/run/sshd && \ 37 | ssh-keygen -A && \ 38 | sed -i 's/PermitRootLogin without-password/PermitRootLogin yes/' /etc/ssh/sshd_config && \ 39 | sed 's@session\s*required\s*pam_loginuid.so@session optional pam_loginuid.so@g' -i /etc/pam.d/sshd && \ 40 | ssh-keygen -f /root/.ssh/id_rsa -t rsa -N '' && \ 41 | chmod 600 /root/.ssh/config && \ 42 | chmod 700 /root/.ssh && \ 43 | cp /root/.ssh/id_rsa.pub /root/.ssh/authorized_keys 44 | 45 | # Build OpenMPI. 46 | ENV OMPI_SERIES=3.1 47 | ENV OMPI_VERSION=3.1.4 48 | RUN TARBALL=openmpi-${OMPI_VERSION}.tar.gz && \ 49 | URL=https://download.open-mpi.org/release/open-mpi/v${OMPI_SERIES}/${TARBALL} && \ 50 | wget ${URL} -P /tmp && \ 51 | OMPI_DIR=/opt/openmpi/${OMPI_VERSION} && \ 52 | mkdir -p ${OMPI_DIR}/build && \ 53 | tar -xzf /tmp/${TARBALL} -C ${OMPI_DIR} --strip-components=1 && \ 54 | cd ${OMPI_DIR}/build && \ 55 | ${OMPI_DIR}/configure --prefix=/usr/local/openmpi-${OMPI_VERSION} && \ 56 | make && make install && \ 57 | rm -f /tmp/${TARBALL} 58 | ENV OMPI_DIR=/usr/local/openmpi-${OMPI_VERSION} 59 | 60 | ENV PATH=${OMPI_DIR}/bin:${PATH} 61 | ENV CPATH=${OMPI_DIR}/include:${CPATH} 62 | 63 | # Build PETSc in optimized mode. 64 | ENV PETSC_VERSION=3.11.4 65 | RUN TARBALL=petsc-lite-${PETSC_VERSION}.tar.gz && \ 66 | URL=http://ftp.mcs.anl.gov/pub/petsc/release-snapshots/${TARBALL} && \ 67 | wget ${URL} -P /tmp && \ 68 | PETSC_DIR=/opt/petsc/${PETSC_VERSION} && \ 69 | PETSC_ARCH=linux-gnu-openmpi-opt && \ 70 | mkdir -p ${PETSC_DIR} && \ 71 | tar -xzf /tmp/${TARBALL} -C ${PETSC_DIR} --strip-components=1 && \ 72 | cd ${PETSC_DIR} && \ 73 | ./configure \ 74 | --prefix=/usr/local/petsc-${PETSC_VERSION} \ 75 | PETSC_DIR=${PETSC_DIR} \ 76 | PETSC_ARCH=${PETSC_ARCH} \ 77 | COPTFLAGS="-O3" \ 78 | CXXFLAGS="-O3" \ 79 | --with-fc=0 \ 80 | --with-shared-libraries=1 \ 81 | --with-pic=1 \ 82 | --with-cxx-dialect="C++11" \ 83 | --with-debugging=0 \ 84 | --with-mpi-dir=${OMPI_DIR} \ 85 | --download-hdf5 \ 86 | --download-f2cblaslapack \ 87 | --download-hypre \ 88 | --download-ptscotch \ 89 | --download-metis \ 90 | --download-parmetis \ 91 | --download-superlu_dist && \ 92 | make PETSC_DIR=${PETSC_DIR} PETSC_ARCH=${PETSC_ARCH} all && \ 93 | make PETSC_DIR=${PETSC_DIR} PETSC_ARCH=${PETSC_ARCH} install && \ 94 | rm -f /tmp/${TARBALL} 95 | ENV PETSC_DIR=/usr/local/petsc-${PETSC_VERSION} 96 | 97 | # Build AmgX. 98 | ENV AMGX_VERSION=2.1 99 | ENV AMGX_SHA=cf285c118726f5d1e8eb740d4936dd0bdaaf9b48 100 | RUN REPO=https://github.com/NVIDIA/AMGX && \ 101 | AMGX_DIR=/opt/amgx/${AMGX_VERSION} && \ 102 | AMGX_ARCH=linux-gnu-openmpi-opt && \ 103 | git clone ${REPO} ${AMGX_DIR} && \ 104 | cd ${AMGX_DIR} && \ 105 | git checkout -b use4petibm ${AMGX_SHA} && \ 106 | BUILDDIR=${AMGX_DIR}/${AMGX_ARCH}/build && \ 107 | mkdir -p ${BUILDDIR} && \ 108 | cd ${BUILDDIR} && \ 109 | cmake ${AMGX_DIR} \ 110 | -DCMAKE_BUILD_TYPE="Release" \ 111 | -DCMAKE_INSTALL_PREFIX=/usr/local/amgx-${AMGX_VERSION} \ 112 | -DCMAKE_C_COMPILER=mpicc \ 113 | -DCMAKE_C_FLAGS_PROFILE="-O3 -DNDEBUG" \ 114 | -DCMAKE_CXX_COMPILER=mpicxx \ 115 | -DCMAKE_CXX_FLAGS_PROFILE="-O3 -DNDEBUG" \ 116 | -DMPI_CXX_COMPILER=mpicxx \ 117 | -DMPI_C_COMPILER=mpicc \ 118 | -DCUDA_ARCH="35 37 60 70" \ 119 | -DCUDA_HOST_COMPILER=/usr/bin/gcc-5 && \ 120 | make -j"$(nproc)" all && \ 121 | make install 122 | ENV AMGX_DIR=/usr/local/amgx-${AMGX_VERSION} 123 | 124 | # Build and install PetIBM. 125 | ENV PETIBM_VERSION=0.4.2 126 | RUN TARBALL=v${PETIBM_VERSION}.tar.gz && \ 127 | URL=https://github.com/barbagroup/PetIBM/archive/${TARBALL} && \ 128 | wget ${URL} -P /tmp && \ 129 | PETIBM_DIR=/opt/petibm/${PETIBM_VERSION} && \ 130 | PETIBM_ARCH=linux-gnu-openmpi-opt && \ 131 | BUILDDIR=${PETIBM_DIR}/${PETIBM_ARCH}/build && \ 132 | mkdir -p ${BUILDDIR} && \ 133 | tar -xzf /tmp/${TARBALL} -C ${PETIBM_DIR} --strip-components=1 && \ 134 | cd ${BUILDDIR} && \ 135 | ${PETIBM_DIR}/configure --prefix=/usr/local/petibm-${PETIBM_VERSION} \ 136 | CC=mpicc \ 137 | CFLAGS="-w -O3 -std=c++14" \ 138 | CXX=mpicxx \ 139 | CXXFLAGS="-w -O3 -std=c++14" \ 140 | --enable-static=no \ 141 | --with-petsc-dir=${PETSC_DIR} \ 142 | --with-petsc-arch="" \ 143 | --with-cuda-dir=${CUDA_DIR} \ 144 | --with-amgx-dir=${AMGX_DIR} \ 145 | --enable-amgxwrapper \ 146 | --enable-yamlcpp \ 147 | --enable-gtest && \ 148 | make -j"$(nproc)" all && \ 149 | make check && \ 150 | make install && \ 151 | rm -f /tmp/${TARBALL} 152 | ENV PETIBM_DIR=/usr/local/petibm-${PETIBM_VERSION} 153 | 154 | EXPOSE 23 155 | CMD ["/usr/sbin/sshd", "-D", "-p", "23"] 156 | -------------------------------------------------------------------------------- /docker/Dockerfile-0.5.1-GPU-OpenMPI-xenial-devel: -------------------------------------------------------------------------------- 1 | # Dockerfile for PetIBM-0.5.1 2 | 3 | FROM nvidia/cuda:10.1-devel-ubuntu16.04 as build 4 | MAINTAINER Olivier Mesnard 5 | 6 | ENV CUDA_VERSION=10.1 7 | ENV CUDA_DIR=/usr/local/cuda-${CUDA_VERSION} 8 | 9 | # Install base system. 10 | COPY ssh_config /root/.ssh/config 11 | RUN apt-get update && \ 12 | apt-get install -y --no-install-recommends \ 13 | autotools-dev \ 14 | bison \ 15 | build-essential \ 16 | ca-certificates \ 17 | cmake \ 18 | curl \ 19 | flex \ 20 | gfortran \ 21 | git \ 22 | pkg-config \ 23 | python-dev \ 24 | unzip \ 25 | wget && \ 26 | apt-get install -y --no-install-recommends \ 27 | libibverbs1 libibverbs-dev ibverbs-utils librdmacm1 rdmacm-utils \ 28 | ibutils ibacm libibcm1 libibmad5 libibumad3 opensm libopensm5a \ 29 | srptools perftest infiniband-diags ibsim-utils \ 30 | libmthca1 libmlx4-1 libmlx5-1 mstflint \ 31 | libdapl2 dapl2-utils && \ 32 | apt-get install -y --no-install-recommends \ 33 | openssh-server \ 34 | openssh-client && \ 35 | rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* && \ 36 | mkdir /var/run/sshd && \ 37 | ssh-keygen -A && \ 38 | sed -i 's/PermitRootLogin without-password/PermitRootLogin yes/' /etc/ssh/sshd_config && \ 39 | sed 's@session\s*required\s*pam_loginuid.so@session optional pam_loginuid.so@g' -i /etc/pam.d/sshd && \ 40 | ssh-keygen -f /root/.ssh/id_rsa -t rsa -N '' && \ 41 | chmod 600 /root/.ssh/config && \ 42 | chmod 700 /root/.ssh && \ 43 | cp /root/.ssh/id_rsa.pub /root/.ssh/authorized_keys 44 | 45 | # Build OpenMPI. 46 | ENV OMPI_SERIES=3.1 47 | ENV OMPI_VERSION=3.1.4 48 | RUN TARBALL=openmpi-${OMPI_VERSION}.tar.gz && \ 49 | URL=https://download.open-mpi.org/release/open-mpi/v${OMPI_SERIES}/${TARBALL} && \ 50 | wget ${URL} -P /tmp && \ 51 | OMPI_DIR=/opt/openmpi/${OMPI_VERSION} && \ 52 | mkdir -p ${OMPI_DIR}/build && \ 53 | tar -xzf /tmp/${TARBALL} -C ${OMPI_DIR} --strip-components=1 && \ 54 | cd ${OMPI_DIR}/build && \ 55 | ${OMPI_DIR}/configure --prefix=/usr/local/openmpi-${OMPI_VERSION} && \ 56 | make && make install && \ 57 | rm -f /tmp/${TARBALL} 58 | ENV OMPI_DIR=/usr/local/openmpi-${OMPI_VERSION} 59 | 60 | ENV PATH=${OMPI_DIR}/bin:${PATH} 61 | ENV CPATH=${OMPI_DIR}/include:${CPATH} 62 | 63 | # Build PETSc in optimized mode. 64 | ENV PETSC_VERSION=3.12.2 65 | RUN TARBALL=petsc-lite-${PETSC_VERSION}.tar.gz && \ 66 | URL=http://ftp.mcs.anl.gov/pub/petsc/release-snapshots/${TARBALL} && \ 67 | wget ${URL} -P /tmp && \ 68 | PETSC_DIR=/opt/petsc/${PETSC_VERSION} && \ 69 | PETSC_ARCH=linux-gnu-openmpi-opt && \ 70 | mkdir -p ${PETSC_DIR} && \ 71 | tar -xzf /tmp/${TARBALL} -C ${PETSC_DIR} --strip-components=1 && \ 72 | cd ${PETSC_DIR} && \ 73 | ./configure \ 74 | --prefix=/usr/local/petsc-${PETSC_VERSION} \ 75 | PETSC_DIR=${PETSC_DIR} \ 76 | PETSC_ARCH=${PETSC_ARCH} \ 77 | COPTFLAGS="-O3" \ 78 | CXXOPTFLAGS="-O3" \ 79 | --with-fc=0 \ 80 | --with-shared-libraries=1 \ 81 | --with-pic=1 \ 82 | --with-cxx-dialect="C++11" \ 83 | --with-debugging=0 \ 84 | --with-mpi-dir=${OMPI_DIR} \ 85 | --download-hdf5 \ 86 | --download-f2cblaslapack \ 87 | --download-hypre \ 88 | --download-ptscotch \ 89 | --download-metis \ 90 | --download-parmetis \ 91 | --download-superlu_dist && \ 92 | make PETSC_DIR=${PETSC_DIR} PETSC_ARCH=${PETSC_ARCH} all && \ 93 | make PETSC_DIR=${PETSC_DIR} PETSC_ARCH=${PETSC_ARCH} install && \ 94 | rm -f /tmp/${TARBALL} 95 | ENV PETSC_DIR=/usr/local/petsc-${PETSC_VERSION} 96 | 97 | # Build AmgX. 98 | ENV AMGX_VERSION=2.1 99 | ENV AMGX_SHA=cf285c118726f5d1e8eb740d4936dd0bdaaf9b48 100 | RUN REPO=https://github.com/NVIDIA/AMGX && \ 101 | AMGX_DIR=/opt/amgx/${AMGX_VERSION} && \ 102 | AMGX_ARCH=linux-gnu-openmpi-opt && \ 103 | git clone ${REPO} ${AMGX_DIR} && \ 104 | cd ${AMGX_DIR} && \ 105 | git checkout -b use4petibm ${AMGX_SHA} && \ 106 | BUILDDIR=${AMGX_DIR}/${AMGX_ARCH}/build && \ 107 | mkdir -p ${BUILDDIR} && \ 108 | cd ${BUILDDIR} && \ 109 | cmake ${AMGX_DIR} \ 110 | -DCMAKE_BUILD_TYPE="Release" \ 111 | -DCMAKE_INSTALL_PREFIX=/usr/local/amgx-${AMGX_VERSION} \ 112 | -DCMAKE_C_COMPILER=mpicc \ 113 | -DCMAKE_C_FLAGS_PROFILE="-O3 -DNDEBUG" \ 114 | -DCMAKE_CXX_COMPILER=mpicxx \ 115 | -DCMAKE_CXX_FLAGS_PROFILE="-O3 -DNDEBUG" \ 116 | -DMPI_CXX_COMPILER=mpicxx \ 117 | -DMPI_C_COMPILER=mpicc \ 118 | -DCUDA_ARCH="35 37 60 70" \ 119 | -DCUDA_HOST_COMPILER=/usr/bin/gcc-5 && \ 120 | make -j"$(nproc)" all && \ 121 | make install 122 | ENV AMGX_DIR=/usr/local/amgx-${AMGX_VERSION} 123 | 124 | # Build and install PetIBM. 125 | ENV PETIBM_VERSION=0.5.1 126 | RUN TARBALL=v${PETIBM_VERSION}.tar.gz && \ 127 | URL=https://github.com/barbagroup/PetIBM/archive/${TARBALL} && \ 128 | wget ${URL} -P /tmp && \ 129 | PETIBM_DIR=/opt/petibm/${PETIBM_VERSION} && \ 130 | PETIBM_ARCH=linux-gnu-openmpi-opt && \ 131 | BUILDDIR=${PETIBM_DIR}/${PETIBM_ARCH}/build && \ 132 | mkdir -p ${BUILDDIR} && \ 133 | tar -xzf /tmp/${TARBALL} -C ${PETIBM_DIR} --strip-components=1 && \ 134 | cd ${BUILDDIR} && \ 135 | ${PETIBM_DIR}/configure --prefix=/usr/local/petibm-${PETIBM_VERSION} \ 136 | CC=mpicc \ 137 | CFLAGS="-w -O3 -std=c++14" \ 138 | CXX=mpicxx \ 139 | CXXFLAGS="-w -O3 -std=c++14" \ 140 | --enable-static=no \ 141 | --with-petsc-dir=${PETSC_DIR} \ 142 | --with-petsc-arch="" \ 143 | --with-cuda-dir=${CUDA_DIR} \ 144 | --with-amgx-dir=${AMGX_DIR} \ 145 | --enable-amgxwrapper \ 146 | --enable-yamlcpp \ 147 | --enable-gtest && \ 148 | make -j"$(nproc)" all && \ 149 | make check && \ 150 | make install && \ 151 | rm -f /tmp/${TARBALL} 152 | ENV PETIBM_DIR=/usr/local/petibm-${PETIBM_VERSION} 153 | 154 | EXPOSE 23 155 | CMD ["/usr/sbin/sshd", "-D", "-p", "23"] 156 | -------------------------------------------------------------------------------- /docker/Dockerfile-0.5.3-GPU-OpenMPI-focal-devel: -------------------------------------------------------------------------------- 1 | # Dockerfile for PetIBM-0.5.3 2 | 3 | FROM nvidia/cuda:11.6.0-devel-ubuntu20.04 as build 4 | MAINTAINER Olivier Mesnard 5 | 6 | ENV CUDA_VERSION=11.6 7 | ENV CUDA_DIR=/usr/local/cuda-${CUDA_VERSION} 8 | 9 | # Install base system. 10 | COPY ssh_config /root/.ssh/config 11 | ENV DEBIAN_FRONTEND=noninteractive 12 | RUN apt-get update && \ 13 | apt-get install -y --no-install-recommends \ 14 | autotools-dev \ 15 | bison \ 16 | build-essential \ 17 | ca-certificates \ 18 | cmake \ 19 | curl \ 20 | flex \ 21 | gfortran \ 22 | git \ 23 | pkg-config \ 24 | python-dev \ 25 | unzip \ 26 | wget && \ 27 | apt-get install -y --no-install-recommends \ 28 | libibverbs1 libibverbs-dev ibverbs-utils librdmacm1 rdmacm-utils \ 29 | ibutils ibacm libibmad5 libibumad3 opensm libopensm8 \ 30 | srptools perftest infiniband-diags ibsim-utils \ 31 | libmthca1 libmlx4-1 libmlx5-1 mstflint \ 32 | libdapl2 dapl2-utils && \ 33 | apt-get install -y --no-install-recommends \ 34 | openssh-server \ 35 | openssh-client && \ 36 | rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* && \ 37 | mkdir /var/run/sshd && \ 38 | ssh-keygen -A && \ 39 | sed -i 's/PermitRootLogin without-password/PermitRootLogin yes/' /etc/ssh/sshd_config && \ 40 | sed 's@session\s*required\s*pam_loginuid.so@session optional pam_loginuid.so@g' -i /etc/pam.d/sshd && \ 41 | ssh-keygen -f /root/.ssh/id_rsa -t rsa -N '' && \ 42 | chmod 600 /root/.ssh/config && \ 43 | chmod 700 /root/.ssh && \ 44 | cp /root/.ssh/id_rsa.pub /root/.ssh/authorized_keys 45 | 46 | # Build OpenMPI. 47 | ENV OMPI_SERIES=3.1 48 | ENV OMPI_VERSION=3.1.4 49 | RUN TARBALL=openmpi-${OMPI_VERSION}.tar.gz && \ 50 | URL=https://download.open-mpi.org/release/open-mpi/v${OMPI_SERIES}/${TARBALL} && \ 51 | wget ${URL} -P /tmp && \ 52 | OMPI_DIR=/opt/openmpi/${OMPI_VERSION} && \ 53 | mkdir -p ${OMPI_DIR}/build && \ 54 | tar -xzf /tmp/${TARBALL} -C ${OMPI_DIR} --strip-components=1 && \ 55 | cd ${OMPI_DIR}/build && \ 56 | ${OMPI_DIR}/configure --prefix=/usr/local/openmpi-${OMPI_VERSION} && \ 57 | make && make install && \ 58 | rm -f /tmp/${TARBALL} 59 | ENV OMPI_DIR=/usr/local/openmpi-${OMPI_VERSION} 60 | 61 | ENV PATH=${OMPI_DIR}/bin:${PATH} 62 | ENV CPATH=${OMPI_DIR}/include:${CPATH} 63 | 64 | # Build PETSc in optimized mode. 65 | ENV PETSC_VERSION=3.16.5 66 | RUN TARBALL=petsc-lite-${PETSC_VERSION}.tar.gz && \ 67 | URL=http://ftp.mcs.anl.gov/pub/petsc/release-snapshots/${TARBALL} && \ 68 | wget ${URL} -P /tmp && \ 69 | PETSC_DIR=/opt/petsc/${PETSC_VERSION} && \ 70 | PETSC_ARCH=linux-gnu-openmpi-opt && \ 71 | mkdir -p ${PETSC_DIR} && \ 72 | tar -xzf /tmp/${TARBALL} -C ${PETSC_DIR} --strip-components=1 && \ 73 | cd ${PETSC_DIR} && \ 74 | ./configure \ 75 | --prefix=/usr/local/petsc-${PETSC_VERSION} \ 76 | PETSC_DIR=${PETSC_DIR} \ 77 | PETSC_ARCH=${PETSC_ARCH} \ 78 | COPTFLAGS="-O3" \ 79 | CXXOPTFLAGS="-O3" \ 80 | --with-fc=0 \ 81 | --with-shared-libraries=1 \ 82 | --with-pic=1 \ 83 | --with-cxx-dialect="C++11" \ 84 | --with-debugging=0 \ 85 | --with-mpi-dir=${OMPI_DIR} \ 86 | --download-cmake \ 87 | --download-hdf5 \ 88 | --download-f2cblaslapack \ 89 | --download-hypre \ 90 | --download-ptscotch \ 91 | --download-metis \ 92 | --download-parmetis \ 93 | --download-superlu_dist && \ 94 | make PETSC_DIR=${PETSC_DIR} PETSC_ARCH=${PETSC_ARCH} all && \ 95 | make PETSC_DIR=${PETSC_DIR} PETSC_ARCH=${PETSC_ARCH} install && \ 96 | rm -f /tmp/${TARBALL} 97 | ENV PETSC_DIR=/usr/local/petsc-${PETSC_VERSION} 98 | 99 | # Build AmgX. 100 | ENV AMGX_VERSION=2.2.0 101 | RUN TARBALL=v${AMGX_VERSION}.tar.gz && \ 102 | URL=https://github.com/NVIDIA/AMGX/archive/refs/tags/${TARBALL} && \ 103 | wget ${URL} -P /tmp && \ 104 | AMGX_DIR=/opt/amgx/${AMGX_VERSION} && \ 105 | AMGX_ARCH=linux-gnu-openmpi-opt && \ 106 | BUILDDIR=${AMGX_DIR}/${AMGX_ARCH}/build && \ 107 | mkdir -p ${BUILDDIR} && \ 108 | tar -xzf /tmp/${TARBALL} -C ${AMGX_DIR} --strip-components=1 && \ 109 | cd ${BUILDDIR} && \ 110 | cmake ${AMGX_DIR} \ 111 | -DCMAKE_BUILD_TYPE="Release" \ 112 | -DCMAKE_INSTALL_PREFIX=/usr/local/amgx-${AMGX_VERSION} \ 113 | -DCMAKE_C_COMPILER=mpicc \ 114 | -DCMAKE_C_FLAGS_PROFILE="-O3 -DNDEBUG" \ 115 | -DCMAKE_CXX_COMPILER=mpicxx \ 116 | -DCMAKE_CXX_FLAGS_PROFILE="-O3 -DNDEBUG" \ 117 | -DMPI_CXX_COMPILER=mpicxx \ 118 | -DMPI_C_COMPILER=mpicc \ 119 | -DCUDA_ARCH="60 70" \ 120 | -DCUDA_HOST_COMPILER=/usr/bin/gcc-9 && \ 121 | make all && \ 122 | make install && \ 123 | rm -f /tmp/${TARBALL} 124 | ENV AMGX_DIR=/usr/local/amgx-${AMGX_VERSION} 125 | 126 | # Build and install PetIBM. 127 | ENV PETIBM_VERSION=0.5.3 128 | RUN TARBALL=v${PETIBM_VERSION}.tar.gz && \ 129 | URL=https://github.com/barbagroup/PetIBM/archive/${TARBALL} && \ 130 | wget ${URL} -P /tmp && \ 131 | PETIBM_DIR=/opt/petibm/${PETIBM_VERSION} && \ 132 | PETIBM_ARCH=linux-gnu-openmpi-opt && \ 133 | BUILDDIR=${PETIBM_DIR}/${PETIBM_ARCH}/build && \ 134 | mkdir -p ${BUILDDIR} && \ 135 | tar -xzf /tmp/${TARBALL} -C ${PETIBM_DIR} --strip-components=1 && \ 136 | cd ${BUILDDIR} && \ 137 | ${PETIBM_DIR}/configure --prefix=/usr/local/petibm-${PETIBM_VERSION} \ 138 | CC=mpicc \ 139 | CFLAGS="-w -O3 -std=c++14" \ 140 | CXX=mpicxx \ 141 | CXXFLAGS="-w -O3 -std=c++14" \ 142 | --enable-static=no \ 143 | --with-petsc-dir=${PETSC_DIR} \ 144 | --with-petsc-arch="" \ 145 | --with-cuda-dir=${CUDA_DIR} \ 146 | --with-amgx-dir=${AMGX_DIR} \ 147 | --enable-amgxwrapper \ 148 | --enable-yamlcpp \ 149 | --enable-gtest && \ 150 | make -j"$(nproc)" all && \ 151 | make check && \ 152 | make install && \ 153 | rm -f /tmp/${TARBALL} 154 | ENV PETIBM_DIR=/usr/local/petibm-${PETIBM_VERSION} 155 | 156 | EXPOSE 23 157 | CMD ["/usr/sbin/sshd", "-D", "-p", "23"] 158 | -------------------------------------------------------------------------------- /docker/Dockerfile-0.5.1-GPU-IntelMPI-xenial-devel: -------------------------------------------------------------------------------- 1 | # Dockerfile for PetIBM-0.5.1 for use with Batch Shipyard on Azure Batch 2 | 3 | FROM nvidia/cuda:10.1-devel-ubuntu16.04 as build 4 | MAINTAINER Olivier Mesnard 5 | 6 | ENV CUDA_VERSION=10.1 7 | ENV CUDA_DIR=/usr/local/cuda-${CUDA_VERSION} 8 | 9 | # Install base system. 10 | COPY ssh_config /root/.ssh/config 11 | RUN apt-get update && \ 12 | apt-get install -y --no-install-recommends \ 13 | autotools-dev \ 14 | bison \ 15 | build-essential \ 16 | ca-certificates \ 17 | cmake \ 18 | curl \ 19 | flex \ 20 | gfortran \ 21 | git \ 22 | pkg-config \ 23 | python-dev \ 24 | unzip \ 25 | wget && \ 26 | # Infiniband/RDMA 27 | apt-get install -y --no-install-recommends \ 28 | cpio \ 29 | libmlx4-1 \ 30 | libmlx5-1 \ 31 | librdmacm1 \ 32 | libibverbs1 \ 33 | libmthca1 \ 34 | libdapl2 \ 35 | dapl2-utils && \ 36 | # Batch-Shipyard dependencies 37 | apt-get install -y --no-install-recommends \ 38 | openssh-server \ 39 | openssh-client && \ 40 | rm -rf /var/lib/apt/lists/* && \ 41 | # Configure ssh server and keys 42 | mkdir /var/run/sshd && \ 43 | ssh-keygen -A && \ 44 | sed -i 's/PermitRootLogin without-password/PermitRootLogin yes/' /etc/ssh/sshd_config && \ 45 | sed 's@session\s*required\s*pam_loginuid.so@session optional pam_loginuid.so@g' -i /etc/pam.d/sshd && \ 46 | ssh-keygen -f /root/.ssh/id_rsa -t rsa -N '' && \ 47 | chmod 600 /root/.ssh/config && \ 48 | chmod 700 /root/.ssh && \ 49 | cp /root/.ssh/id_rsa.pub /root/.ssh/authorized_keys 50 | 51 | # Build Intel MPI library. 52 | ENV INTEL_DIR=/opt/intel2 53 | ENV MANPATH=/usr/share/man:/usr/local/man \ 54 | COMPILERVARS_ARCHITECTURE=intel64 \ 55 | COMPILERVARS_PLATFORM=linux \ 56 | INTEL_COMPILERVARS=${INTEL_DIR}/bin/compilervars.sh \ 57 | INTEL_MPI_DIR=${INTEL_DIR}/compilers_and_libraries/linux/mpi 58 | ADD l_mpi_2017.2.174.tgz /tmp 59 | RUN cd /tmp/l_mpi_2017.2.174 && \ 60 | sed -i -e 's/^ACCEPT_EULA=decline/ACCEPT_EULA=accept/g' silent.cfg && \ 61 | sed -i -e 's,^PSET_INSTALL_DIR=.*,PSET_INSTALL_DIR=/opt/intel2,g' silent.cfg && \ 62 | sed -i -e 's,^ARCH_SELECTED=.*,ARCH_SELECTED=INTEL64,g' silent.cfg && \ 63 | ./install.sh -s silent.cfg && \ 64 | cd .. && \ 65 | rm -rf l_mpi_2017.2.174 66 | 67 | # Build PETSc-3.12.5 in optimized mode. 68 | ENV PETSC_VERSION=3.12.5 69 | RUN . ${INTEL_COMPILERVARS} \ 70 | -arch ${COMPILERVARS_ARCHITECTURE} \ 71 | -platform ${COMPILERVARS_PLATFORM} && \ 72 | . ${INTEL_MPI_DIR}/${COMPILERVARS_ARCHITECTURE}/bin/mpivars.sh && \ 73 | TARBALL=petsc-lite-${PETSC_VERSION}.tar.gz && \ 74 | URL=http://ftp.mcs.anl.gov/pub/petsc/release-snapshots/${TARBALL} && \ 75 | wget ${URL} -P /tmp && \ 76 | PETSC_DIR=/opt/petsc/${PETSC_VERSION} && \ 77 | PETSC_ARCH=linux-gnu-intelmpi-opt && \ 78 | mkdir -p ${PETSC_DIR} && \ 79 | tar -xzf /tmp/${TARBALL} -C ${PETSC_DIR} --strip-components=1 && \ 80 | cd ${PETSC_DIR} && \ 81 | ./configure \ 82 | --prefix=/usr/local/petsc-${PETSC_VERSION} \ 83 | PETSC_DIR=${PETSC_DIR} \ 84 | PETSC_ARCH=${PETSC_ARCH} \ 85 | COPTFLAGS="-O3" \ 86 | CXXOPTFLAGS="-O3" \ 87 | --with-fc=0 \ 88 | --with-shared-libraries=1 \ 89 | --with-pic=1 \ 90 | --with-cxx-dialect="C++11" \ 91 | --with-debugging=0 \ 92 | --download-hdf5 \ 93 | --download-f2cblaslapack \ 94 | --download-hypre \ 95 | --download-ptscotch \ 96 | --download-metis \ 97 | --download-parmetis \ 98 | --download-superlu_dist && \ 99 | make PETSC_DIR=${PETSC_DIR} PETSC_ARCH=${PETSC_ARCH} all && \ 100 | make PETSC_DIR=${PETSC_DIR} PETSC_ARCH=${PETSC_ARCH} install && \ 101 | rm -f /tmp/${TARBALL} 102 | ENV PETSC_DIR=/usr/local/petsc-${PETSC_VERSION} 103 | 104 | # Build AmgX-2.1. 105 | ENV AMGX_VERSION=2.1 106 | ENV AMGX_SHA=cf285c118726f5d1e8eb740d4936dd0bdaaf9b48 107 | RUN . ${INTEL_COMPILERVARS} \ 108 | -arch ${COMPILERVARS_ARCHITECTURE} \ 109 | -platform ${COMPILERVARS_PLATFORM} && \ 110 | . ${INTEL_MPI_DIR}/${COMPILERVARS_ARCHITECTURE}/bin/mpivars.sh && \ 111 | REPO=https://github.com/NVIDIA/AMGX && \ 112 | AMGX_DIR=/opt/amgx/${AMGX_VERSION} && \ 113 | AMGX_ARCH=linux-gnu-intelmpi-opt && \ 114 | git clone ${REPO} ${AMGX_DIR} && \ 115 | cd ${AMGX_DIR} && \ 116 | git checkout -b use4petibm ${AMGX_SHA} && \ 117 | BUILDDIR=${AMGX_DIR}/${AMGX_ARCH}/build && \ 118 | mkdir -p ${BUILDDIR} && \ 119 | cd ${BUILDDIR} && \ 120 | cmake ${AMGX_DIR} \ 121 | -DCMAKE_BUILD_TYPE="Release" \ 122 | -DCMAKE_INSTALL_PREFIX=/usr/local/amgx-${AMGX_VERSION} \ 123 | -DCMAKE_C_COMPILER=mpicc \ 124 | -DCMAKE_C_FLAGS_PROFILE="-O3 -DNDEBUG" \ 125 | -DCMAKE_CXX_COMPILER=mpicxx \ 126 | -DCMAKE_CXX_FLAGS_PROFILE="-O3 -DNDEBUG" \ 127 | -DMPI_CXX_COMPILER=mpicxx \ 128 | -DMPI_C_COMPILER=mpicc \ 129 | -DCUDA_ARCH="35 37 60 70" \ 130 | -DCUDA_HOST_COMPILER=${INTEL_MPI_DIR}/${COMPILERVARS_ARCHITECTURE}/bin/mpicc && \ 131 | make -j"$(nproc)" all && \ 132 | make install 133 | ENV AMGX_DIR=/usr/local/amgx-${AMGX_VERSION} 134 | 135 | # Build PetIBM-0.5.1. 136 | ENV PETIBM_VERSION=0.5.1 137 | RUN . ${INTEL_COMPILERVARS} \ 138 | -arch ${COMPILERVARS_ARCHITECTURE} \ 139 | -platform ${COMPILERVARS_PLATFORM} && \ 140 | . ${INTEL_MPI_DIR}/${COMPILERVARS_ARCHITECTURE}/bin/mpivars.sh && \ 141 | TARBALL=v${PETIBM_VERSION}.tar.gz && \ 142 | URL=https://github.com/barbagroup/PetIBM/archive/${TARBALL} && \ 143 | wget ${URL} -P /tmp && \ 144 | PETIBM_DIR=/opt/petibm/${PETIBM_VERSION} && \ 145 | PETIBM_ARCH=linux-gnu-intelmpi-opt && \ 146 | BUILDDIR=${PETIBM_DIR}/${PETIBM_ARCH}/build && \ 147 | mkdir -p ${BUILDDIR} && \ 148 | tar -xzf /tmp/${TARBALL} -C ${PETIBM_DIR} --strip-components=1 && \ 149 | cd ${BUILDDIR} && \ 150 | ${PETIBM_DIR}/configure --prefix=/usr/local/petibm-${PETIBM_VERSION} \ 151 | CC=mpicc \ 152 | CFLAGS="-w -O3 -std=c++14" \ 153 | CXX=mpicxx \ 154 | CXXFLAGS="-w -O3 -std=c++14" \ 155 | --enable-static=no \ 156 | --with-petsc-dir=${PETSC_DIR} \ 157 | --with-petsc-arch="" \ 158 | --with-cuda-dir=${CUDA_DIR} \ 159 | --with-amgx-dir=${AMGX_DIR} \ 160 | --enable-amgxwrapper \ 161 | --enable-yamlcpp \ 162 | --enable-gtest && \ 163 | make -j"$(nproc)" all && \ 164 | make check && \ 165 | make install && \ 166 | rm -f /tmp/${TARBALL} 167 | ENV PETIBM_DIR=/usr/local/petibm-${PETIBM_VERSION} 168 | 169 | # Set sshd command. 170 | EXPOSE 23 171 | CMD ["/usr/sbin/sshd", "-D", "-p", "23"] 172 | -------------------------------------------------------------------------------- /docker/Dockerfile-0.5.4-HPCX207-CUDA102-bionic: -------------------------------------------------------------------------------- 1 | # vim:ft=dockerfile 2 | # Packages: 3 | # - CUDA 10.2 4 | # - HPCX 2.7.4 5 | # - AmgX 6f1401 6 | # - PETSc 3.16.6 7 | # - AmgXWrapper 1.6.2 8 | # - PetIBM 0.5.4rc2 9 | # - SymEngine 0.9.0 10 | 11 | # Multi-stage build stage 0: build 12 | # ================================= 13 | FROM nvidia/cuda:10.2-devel-ubuntu18.04 AS builder 14 | 15 | # basic 16 | # ----------------------------------------------------------------------------- 17 | 18 | # configure the environment 19 | RUN export DEBIAN_FRONTEND=noninteractive \ 20 | && ln -sf /bin/bash /bin/sh \ 21 | && apt-get -y update \ 22 | && apt-get -y install tzdata locales \ 23 | && ln -sf /usr/share/zoneinfo/US/Eastern /etc/localtime \ 24 | && dpkg-reconfigure --frontend noninteractive tzdata \ 25 | && sed -i "s/\#\ en_US\.UTF-8\ UTF-8/en_US\.UTF-8\ UTF-8/g" /etc/locale.gen \ 26 | && dpkg-reconfigure locales \ 27 | && echo 'LANG="en_US.UTF-8"' >> /etc/default/locale \ 28 | && echo 'LANG="en_US.UTF-8"' >> /etc/locale.conf 29 | 30 | # essential dependencies 31 | RUN apt-get update \ 32 | && apt-get install -y --no-install-recommends \ 33 | curl build-essential libtool autoconf automake pkg-config \ 34 | numactl libnuma-dev \ 35 | python3 python3-pip wget rsync zlib1g-dev libgmp-dev 36 | 37 | # cmake 38 | # ----------------------------------------------------------------------------- 39 | RUN curl -L https://github.com/Kitware/CMake/releases/download/v3.23.1/cmake-3.23.1-linux-x86_64.sh -o /cmake-installer.sh \ 40 | && sh /cmake-installer.sh --prefix=/usr --exclude-subdir --skip-license 41 | 42 | # HPC-X 43 | # ----------------------------------------------------------------------------- 44 | RUN curl -L https://content.mellanox.com/hpc/hpc-x/v2.7.4/hpcx-v2.7.4-gcc-inbox-ubuntu18.04-x86_64.tbz -o /hpcx.tbz \ 45 | && mkdir /hpcx \ 46 | && tar -xf /hpcx.tbz -C /hpcx --strip-component=1 47 | 48 | # SymEngine 49 | # ----------------------------------------------------------------------------- 50 | ENV SYMENGINE_VERSION="0.9.0" 51 | ENV SYMENGINE_DIR="/usr/local/symengine-${SYMENGINE_VERSION}" 52 | 53 | RUN curl -L https://github.com/symengine/symengine/tarball/v${SYMENGINE_VERSION} -o /symengine.tar.gz 54 | 55 | RUN mkdir symengine \ 56 | && tar -xf /symengine.tar.gz -C /symengine --strip-component=1 \ 57 | && mkdir symengine/build \ 58 | && cd symengine/build \ 59 | && cmake \ 60 | -DCMAKE_INSTALL_PREFIX=${SYMENGINE_DIR} \ 61 | -DBUILD_TESTS=OFF \ 62 | -DBUILD_BENCHMARKS=OFF \ 63 | -DBUILD_SHARED_LIBS=ON \ 64 | -DCMAKE_INSTALL_RPATH_USE_LINK_PATH=ON \ 65 | ../ \ 66 | && make all -j $(nproc) \ 67 | && make install \ 68 | && echo "${SYMENGINE_DIR}/lib" > /etc/ld.so.conf.d/symengine-${PETSC_VERSION}.conf && ldconfig 69 | 70 | # PETSc 71 | # ----------------------------------------------------------------------------- 72 | ENV PETSC_VERSION="3.16.6" 73 | ENV PETSC_DIR="/usr/local/petsc-${PETSC_VERSION}" \ 74 | PETSC_ARCH="" 75 | 76 | RUN curl -L http://ftp.mcs.anl.gov/pub/petsc/release-snapshots/petsc-lite-${PETSC_VERSION}.tar.gz -o /petsc.tar.gz 77 | 78 | RUN source /hpcx/hpcx-init.sh && hpcx_load \ 79 | && export PETSC_INSTALL_PATH=${PETSC_DIR} && unset PETSC_ARCH && unset PETSC_DIR \ 80 | && mkdir petsc && tar -xf petsc.tar.gz -C petsc --strip-component=1 && cd petsc \ 81 | && ./configure \ 82 | --prefix="${PETSC_INSTALL_PATH}" \ 83 | --with-default-arch=0 \ 84 | --with-fortran-bindings=0 \ 85 | --with-precision=double \ 86 | --with-clanguage=C \ 87 | --with-shared-libraries=1 \ 88 | --with-cc=mpicc \ 89 | --with-cxx=mpicxx \ 90 | --with-fc=0 \ 91 | --with-pic=1 \ 92 | --with-cxx-dialect=C++11 \ 93 | --with-debugging=0 \ 94 | --COPTFLAGS="-O3" \ 95 | --CXXOPTFLAGS="-O3" \ 96 | --with-gcov=0 \ 97 | --with-mpi=1 \ 98 | --download-f2cblaslapack=1 \ 99 | --download-hypre=1 \ 100 | --download-superlu_dist=1 \ 101 | --download-hdf5=1 \ 102 | && make PETSC_DIR=${PWD} PETSC_ARCH=arch-linux-c-opt all \ 103 | && make PETSC_DIR=${PWD} PETSC_ARCH=arch-linux-c-opt install \ 104 | && cd .. && rm -rf petsc petsc.tar.gz \ 105 | && export PETSC_DIR=${PETSC_INSTALL_PATH} \ 106 | && echo "${PETSC_DIR}/lib" > /etc/ld.so.conf.d/petsc-${PETSC_VERSION}.conf && ldconfig 107 | 108 | # AmgX 109 | # ----------------------------------------------------------------------------- 110 | ENV AMGX_VERSION="6f14016e420617b4d57580f77b5a2ab4c28a162d" 111 | ENV AMGX_DIR="/usr/local/amgx-${AMGX_VERSION}" 112 | 113 | RUN curl -L https://github.com/NVIDIA/AMGX/tarball/${AMGX_VERSION} -o /amgx.tar.gz 114 | 115 | # build and install amgx 116 | RUN source /hpcx/hpcx-init.sh && hpcx_load \ 117 | && mkdir amgx && tar -xf /amgx.tar.gz -C amgx --strip-component=1 \ 118 | && mkdir amgx/build && cd amgx/build \ 119 | && cmake \ 120 | -DCMAKE_C_COMPILER=gcc \ 121 | -DCMAKE_CXX_COMPILER=g++ \ 122 | -DCMAKE_BUILD_TYPE=Release \ 123 | -DCMAKE_INSTALL_PREFIX=${AMGX_DIR} \ 124 | -DCMAKE_VERBOSE_MAKEFILE=OFF \ 125 | -DCUDA_ARCH="35;70" \ 126 | -DCMAKE_NO_MPI=OFF \ 127 | -DAMGX_NO_RPATH=OFF \ 128 | -DCUDA_NVCC_FLAGS_RELEASE="" \ 129 | .. \ 130 | && make -j $(nproc) all && make install \ 131 | && cd ../.. && rm -rf amgx amgx.tar.gz \ 132 | && echo "${AMGX_DIR}/lib" > /etc/ld.so.conf.d/amgx-${AMGX_VERSION}.conf && ldconfig 133 | 134 | # AmgXWrapper 135 | # ----------------------------------------------------------------------------- 136 | ENV AMGXWRAPPER_VERSION="1.6.2" 137 | ENV AMGXWRAPPER_DIR="/usr/local/amgxwrapper-${AMGXWRAPPER_VERSION}" 138 | 139 | RUN curl -L https://github.com/barbagroup/AmgXWrapper/tarball/v${AMGXWRAPPER_VERSION} -o /amgxwrapper.tar.gz 140 | 141 | # build and install AmgXWrapper 142 | RUN source /hpcx/hpcx-init.sh && hpcx_load \ 143 | && export OMPI_MCA_plm_rsh_agent=sh \ 144 | && mkdir amgxwrapper \ 145 | && tar -xf amgxwrapper.tar.gz -C amgxwrapper --strip-component=1 \ 146 | && mkdir amgxwrapper/build && cd amgxwrapper/build \ 147 | && cmake \ 148 | -DCMAKE_INSTALL_PREFIX=${AMGXWRAPPER_DIR} \ 149 | -DCMAKE_C_COMPILER=gcc \ 150 | -DCMAKE_CXX_COMPILER=g++ \ 151 | -DCMAKE_CUDA_FLAGS_RELEASE="" \ 152 | -DCMAKE_CUDA_ARCHITECTURES="all" \ 153 | -DCMAKE_BUILD_TYPE=Release \ 154 | -DPETSC_DIR=${PETSC_DIR} \ 155 | -DPETSC_ARCH="" \ 156 | -DCUDA_DIR=/usr/local/cuda \ 157 | -DAMGX_DIR=${AMGX_DIR} \ 158 | -DCMAKE_VERBOSE_MAKEFILE=1 \ 159 | .. \ 160 | && make all -j $(nproc) \ 161 | && make install \ 162 | && echo "${AMGXWRAPPER_DIR}/lib" > /etc/ld.so.conf.d/amgxwrapper-${AMGXWRAPPER_VERSION}.conf && ldconfig 163 | 164 | # PetIBM 165 | # ----------------------------------------------------------------------------- 166 | ENV PETIBM_VERSION="0.5.4" 167 | ENV PETIBM_DIR="/usr/local/petibm-${PETIBM_VERSION}" 168 | 169 | RUN curl -L https://github.com/barbagroup/PetIBM/tarball/v${PETIBM_VERSION} -o /petibm.tar.gz 170 | 171 | # build and install PetIBM 172 | RUN source /hpcx/hpcx-init.sh && hpcx_load \ 173 | && export OMPI_MCA_plm_rsh_agent=sh \ 174 | && apt install -y git \ 175 | && mkdir petibm && tar xf petibm.tar.gz -C petibm --strip-component=1 \ 176 | && cd petibm \ 177 | && mkdir build && cd build \ 178 | && cmake \ 179 | -DCMAKE_INSTALL_PREFIX=${PETIBM_DIR} \ 180 | -DCMAKE_CXX_COMPILER=g++ \ 181 | -DCMAKE_CUDA_ARCHITECTURES="all" \ 182 | -DCMAKE_BUILD_TYPE=Release \ 183 | -DPETSC_DIR=${PETSC_DIR} \ 184 | -DPETSC_ARCH="" \ 185 | -DAMGXWRAPPER_DIR=${AMGXWRAPPER_DIR} \ 186 | -DPETIBM_ENABLE_TESTS=ON \ 187 | -DPETIBM_USE_AMGX=ON \ 188 | -DPETIBM_BUILD_YAMLCPP=ON \ 189 | ../ \ 190 | && make all -j $(nproc) \ 191 | && make check \ 192 | && make install \ 193 | && echo "${PETIBM_DIR}/lib" > /etc/ld.so.conf.d/petibm-${PETIBM_VERSION}.conf && ldconfig 194 | 195 | # create a scrip to initialize HPCX all the time 196 | RUN echo "#!/bin/bash" > /entrypoint.sh \ 197 | && echo "set -e" >> /entrypoint.sh \ 198 | && echo ". /hpcx/hpcx-init.sh" >> /entrypoint.sh \ 199 | && echo "hpcx_load" >> /entrypoint.sh \ 200 | && echo "exec \"\$@\"" >> /entrypoint.sh \ 201 | && chmod a+x /entrypoint.sh 202 | 203 | 204 | # Multi-stage build stage 1: runtime image 205 | # ========================================= 206 | FROM nvidia/cuda:10.2-runtime-ubuntu18.04 AS production 207 | LABEL maintainer="Pi-Yueh Chuang " 208 | 209 | ENV SYMENGINE_VERSION="0.9.0" \ 210 | PETSC_VERSION="3.16.6" \ 211 | AMGX_VERSION="6f14016e420617b4d57580f77b5a2ab4c28a162d" \ 212 | AMGXWRAPPER_VERSION="1.6.2" \ 213 | PETIBM_VERSION="0.5.4" 214 | 215 | ENV SYMENGINE_DIR="/usr/local/symengine-${SYMENGINE_VERSION}" \ 216 | PETSC_DIR="/usr/local/petsc-${PETSC_VERSION}" \ 217 | PETSC_ARCH="" \ 218 | AMGX_DIR="/usr/local/amgx-${AMGX_VERSION}" \ 219 | AMGXWRAPPER_DIR="/usr/local/amgxwrapper-${AMGXWRAPPER_VERSION}" \ 220 | PETIBM_DIR="/usr/local/petibm-${PETIBM_VERSION}" 221 | 222 | # configure the environment 223 | RUN export DEBIAN_FRONTEND=noninteractive \ 224 | && ln -sf /bin/bash /bin/sh \ 225 | && apt-get -y update \ 226 | && apt-get -y install tzdata locales \ 227 | && ln -sf /usr/share/zoneinfo/US/Eastern /etc/localtime \ 228 | && dpkg-reconfigure --frontend noninteractive tzdata \ 229 | && sed -i "s/\#\ en_US\.UTF-8\ UTF-8/en_US\.UTF-8\ UTF-8/g" /etc/locale.gen \ 230 | && dpkg-reconfigure locales \ 231 | && echo 'LANG="en_US.UTF-8"' >> /etc/default/locale \ 232 | && echo 'LANG="en_US.UTF-8"' >> /etc/locale.conf \ 233 | && apt-get install -y --no-install-recommends \ 234 | python3 python3-pip numactl libnuma-dev zlib1g-dev \ 235 | openssh-client openssh-server libgmp-dev \ 236 | && apt-get -y autoclean \ 237 | && rm -rf /var/lib/apt/lists/* 238 | 239 | # copy installed programs 240 | COPY --from=builder /hpcx /hpcx 241 | COPY --from=builder ${SYMENGINE_DIR} ${SYMENGINE_DIR} 242 | COPY --from=builder ${PETSC_DIR} ${PETSC_DIR} 243 | COPY --from=builder ${AMGX_DIR} ${AMGX_DIR} 244 | COPY --from=builder ${AMGXWRAPPER_DIR} ${AMGXWRAPPER_DIR} 245 | COPY --from=builder ${PETIBM_DIR} ${PETIBM_DIR} 246 | COPY --from=builder /entrypoint.sh /entrypoint.sh 247 | 248 | # ldconfig 249 | RUN source /hpcx/hpcx-init.sh && hpcx_load \ 250 | && echo "${SYMENGINE_DIR}/lib" > /etc/ld.so.conf.d/petibm-${PETIBM_VERSION}.conf \ 251 | && echo "${PETSC_DIR}/lib" > /etc/ld.so.conf.d/petsc-${PETSC_VERSION}.conf \ 252 | && echo "${AMGX_DIR}/lib" > /etc/ld.so.conf.d/amgx-${AMGX_VERSION}.conf \ 253 | && echo "${AMGXWRAPPER_DIR}/lib" > /etc/ld.so.conf.d/amgxwrapper-${AMGXWRAPPER_VERSION}.conf \ 254 | && echo "${PETIBM_DIR}/lib" > /etc/ld.so.conf.d/petibm-${PETIBM_VERSION}.conf \ 255 | && ldconfig 256 | 257 | # set up ssh for root for MPI applications 258 | RUN mkdir /var/run/sshd \ 259 | && ssh-keygen -A \ 260 | && sed -i "s/#PermitRootLogin\ yes/PermitRootLogin\ yes/" /etc/ssh/sshd_config \ 261 | && sed -i "s/session\s*required\s*pam_loginuid.so/session\ optional\ pam_loginuid.so/g" /etc/pam.d/sshd \ 262 | && ssh-keygen -f /root/.ssh/id_rsa -t rsa -N '' \ 263 | && echo "Host 10.*" > /root/.ssh/config \ 264 | && echo " Port 23" >> /root/.ssh/config \ 265 | && echo " StrictHostKeyChecking no" >> /root/.ssh/config \ 266 | && echo " UserKnownHostsFile /dev/null" >> /root/.ssh/config \ 267 | && chmod 600 /root/.ssh/config && chmod 700 /root/.ssh \ 268 | && cp /root/.ssh/id_rsa.pub /root/.ssh/authorized_keys 269 | 270 | # add paths 271 | ENV PATH="${AMGX_DIR}/bin:${PATH}" 272 | ENV PATH="${PETSC_DIR}/bin:${PATH}" 273 | ENV PATH="${AMGXWRAPPER_DIR}/bin:${PATH}" 274 | ENV PATH="${PETIBM_DIR}/bin:${PATH}" 275 | 276 | # using port 23 for Azure cloud, see 277 | # https://batch-shipyard.readthedocs.io/en/3.0.1/80-batch-shipyard-multi-instance-tasks 278 | EXPOSE 23 279 | ENTRYPOINT ["/entrypoint.sh"] 280 | CMD ["/usr/sbin/sshd", "-D", "-p", "23"] 281 | -------------------------------------------------------------------------------- /docker/README.md: -------------------------------------------------------------------------------- 1 | # Dockerfile for PetIBM 2 | 3 | The present directory contains the `Dockerfile` files used to create Docker images for PetIBM. 4 | Once built, the Docker images are pushed and shared on the DockerHub repository `barbagroup/petibm`. 5 | 6 | ## `barbagroup/petibm:0.5.4-hpcx207-cuda102` 7 | 8 | To use this image, pull it from DockerHub through command-line `docker pull barbagroup/petibm:0.5.4-hpcx207-cuda102` 9 | 10 | PetIBM (0.5.4) was installed along with its dependencies: 11 | 12 | * CUDA 10.2 13 | * OpenMPI v4.0.4rc3 (through NVIDIA's HPC-X v2.07) 14 | * AmgX 6f1401 15 | * PETSc 3.16.6 16 | * AmgXWrapper 1.6.2 17 | 18 | The base image of this version is Ubuntu Bionic. 19 | 20 | Alternative to the image on DockerHub, users can build the image locally. 21 | The Docker file corresponding to this image is `Dockerfile-0.5.4-HPCX207-CUDA102-bionic`. 22 | 23 | 1. To build the developer-version image: 24 | 25 | ```shell 26 | docker build --target builder --tag barbagroup/petibm:petibm:0.5.4-hpcx207-cuda102-dev --file Dockerfile-0.5.4-HPCX207-CUDA102-bionic . 27 | ``` 28 | 29 | 2. To build the production-version image: 30 | 31 | ```shell 32 | docker build --target production --tag barbagroup/petibm:petibm:0.5.4-hpcx207-cuda102 --file Dockerfile-0.5.4-HPCX207-CUDA102-bionic . 33 | ``` 34 | 35 | To launch a container with the GPU support and get a Bash shell: 36 | 37 | ```shell 38 | docker run --gpus --rm --name test -it barbagroup/petibm:petibm:0.5.4-hpcx207-cuda102 /bin/bash 39 | ``` 40 | 41 | If the `/bin/bash` in Docker's `run` command is absent, by default, the PetIBM image will start a SSH server daemon listening to port 23. 42 | 43 | ## `barbagroup/petibm:0.5.3-GPU-OpenMPI-focal-devel` 44 | 45 | Image based on Ubuntu 20.04 (Focal). 46 | To pull the image from DockerHub: 47 | 48 | ```shell 49 | docker pull barbagroup/petibm:0.5.3-GPU-OpenMPI-focal-devel 50 | ``` 51 | 52 | PetIBM (0.5.3) was installed along with its dependencies: 53 | 54 | * CUDA Toolkit 11.6 (requires CUDA Driver Version >= 418.39) 55 | * OpenMPI 3.1.4 56 | * PETSc 3.16.5 57 | * AmgX 2.2.0 58 | 59 | To launch a container: 60 | 61 | ```shell 62 | docker run -it barbagroup/petibm:0.5.3-GPU-OpenMPI-focal-devel /bin/bash 63 | ``` 64 | 65 | To launch a container with the GPU support: 66 | 67 | ```shell 68 | docker run --gpus -it barbagroup/petibm:0.5.3-GPU-OpenMPI-focal-devel /bin/bash 69 | ``` 70 | 71 | (More details on how to specify `` can be found [here](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/user-guide.html#gpu-enumeration).) 72 | 73 | To re-build the image: 74 | 75 | ```shell 76 | docker build --tag= --file=Dockerfile-0.5.3-GPU-OpenMPI-focal-devel . 77 | ``` 78 | 79 | ## `barbagroup/petibm:0.5.3-GPU-OpenMPI-focal` 80 | 81 | This image is the same as `barbagroup/petibm:0.5.3-GPU-OpenMPI-focal-devel`, except it does not contain the source files and build directories. 82 | This is intended to make the Docker image as light as possible. 83 | 84 | To pull the image from DockerHub: 85 | 86 | ```shell 87 | docker pull barbagroup/petibm:0.5.3-GPU-OpenMPI-focal 88 | ``` 89 | 90 | To launch a container: 91 | 92 | ```shell 93 | docker run -it barbagroup/petibm:0.5.3-GPU-OpenMPI-focal /bin/bash 94 | ``` 95 | 96 | To launch a container with the GPU support: 97 | 98 | ```shell 99 | docker run --gpus -it barbagroup/petibm:0.5.3-GPU-OpenMPI-focal /bin/bash 100 | ``` 101 | 102 | (More details on how to specify `` can be found [here](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/user-guide.html#gpu-enumeration).) 103 | 104 | ## `barbagroup/petibm:0.5.2-GPU-OpenMPI-centos7` 105 | 106 | PetIBM (0.5.2) was installed along with its dependencies: 107 | 108 | * CUDA 10.1 109 | * OpenMPI v4.0.5 (with Infiniband, UCX, Slurm, PMI, CUDA enabled) 110 | * AmgX 919fb92 111 | * PETSc 3.12.5 112 | * AmgXWrapper 1a93865 113 | 114 | The base image of this version has been changed to CentOS 7 to align with the 115 | CentOS/Red Hat-based clusters at many HPC centers. 116 | 117 | To build the developer-version image: 118 | 119 | ```shell 120 | docker build --target builder --tag barbagroup/petibm:0.5.2-GPU-OpenMPI-centos7-devel --file Dockerfile-0.5.2-GPU-OpenMPI-centos7 . 121 | ``` 122 | 123 | To build the production-version image: 124 | 125 | ```shell 126 | docker build --target production --tag barbagroup/petibm:0.5.2-GPU-OpenMPI-centos7 --file Dockerfile-0.5.2-GPU-OpenMPI-centos7 . 127 | ``` 128 | 129 | To pull the built images from DockerHub: 130 | 131 | ```shell 132 | docker pull barbagroup/petibm:0.5.2-GPU-OpenMPI-centos7 133 | ``` 134 | 135 | or 136 | 137 | ```shell 138 | docker pull barbagroup/petibm:0.5.2-GPU-OpenMPI-centos7-devel 139 | ``` 140 | 141 | To launch a container with the GPU support and get a Bash shell: 142 | 143 | ```shell 144 | docker run --gpus --rm --name test -it barbagroup/petibm:0.5.2-GPU-OpenMPI-centos7 /bin/bash 145 | ``` 146 | For specifying the GPU devices in `--gpus` flags, see the [documentation here](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/user-guide.html#gpu-enumeration) 147 | 148 | If the `/bin/bash` in Docker's `run` command is absent, by default, the PetIBM image 149 | will start a SSH server daemon listening to port 23. 150 | 151 | ## `barbagroup/petibm:0.5.1-GPU-OpenMPI-xenial-devel` 152 | 153 | Image based on Ubuntu 16.04 (Xenial). 154 | To pull the image from DockerHub: 155 | 156 | ```shell 157 | docker pull barbagroup/petibm:0.5.1-GPU-OpenMPI-xenial-devel 158 | ``` 159 | 160 | PetIBM (0.5.1) was installed along with its dependencies: 161 | 162 | * CUDA Toolkit 10.1 (requires CUDA Driver Version >= 418.39) 163 | * OpenMPI 3.1.4 164 | * PETSc 3.12.2 165 | * AmgX 2.1 (commit [cf285c1](https://github.com/NVIDIA/AMGX/tree/cf285c118726f5d1e8eb740d4936dd0bdaaf9b48)) 166 | 167 | Docker containers based on this image can run PetIBM applications on CPU and GPU. 168 | Note that if you want to run a PetIBM application using GPUs, you have to create the container using the utility [`nvidia-docker v2`](https://github.com/NVIDIA/nvidia-docker) (instead of `docker`): 169 | 170 | ```shell 171 | nvidia-docker run -it barbagroup/petibm:0.5.1-GPU-OpenMPI-xenial-devel /bin/bash 172 | ``` 173 | 174 | To re-build the image locally: 175 | 176 | ```shell 177 | docker build --tag=mypetibm:mytag --file=Dockerfile-0.5.1-GPU-OpenMPI-xenial-devel . 178 | ``` 179 | 180 | ## `barbagroup/petibm:0.5.1-GPU-OpenMPI-xenial` 181 | 182 | This image is the same as `barbagroup/petibm:0.5.1-GPU-OpenMPI-xenial-devel`, except it does not contain the source files and build directories. 183 | This is intended to make the Docker image as light as possible. 184 | 185 | To pull the image from DockerHub: 186 | 187 | ```shell 188 | docker pull barbagroup/petibm:0.5.1-GPU-OpenMPI-xenial 189 | ``` 190 | 191 | To run a container: 192 | 193 | ```shell 194 | nvidia-docker run -it barbagroup/petibm:0.5.1-GPU-OpenMPI-xenial /bin/bash 195 | ``` 196 | 197 | ## `barbagroup/petibm:0.5.1-GPU-IntelMPI-xenial-devel` 198 | 199 | Image based on Ubuntu 16.04 (Xenial). 200 | To pull the image from DockerHub: 201 | 202 | ```shell 203 | docker pull barbagroup/petibm:0.5.1-GPU-IntelMPI-xenial-devel 204 | ``` 205 | 206 | PetIBM (0.5.1) was installed along with its dependencies: 207 | 208 | * CUDA Toolkit 10.1 (requires CUDA Driver Version >= 418.39) 209 | * Intel MPI library for Linux (2017, Update 2) 210 | * PETSc 3.12.5 211 | * AmgX 2.1 (commit [cf285c1](https://github.com/NVIDIA/AMGX/tree/cf285c118726f5d1e8eb740d4936dd0bdaaf9b48)) 212 | 213 | Docker containers based on this image can run PetIBM applications on CPU and GPU. 214 | Note that if you want to run a PetIBM application using GPUs, you have to create the container using the utility [`nvidia-docker v2`](https://github.com/NVIDIA/nvidia-docker) (instead of `docker`): 215 | 216 | ```shell 217 | nvidia-docker run -it barbagroup/petibm:0.5.1-GPU-IntelMPI-xenial-devel /bin/bash 218 | ``` 219 | 220 | To re-build the image locally: 221 | 222 | ```shell 223 | docker build --tag=mypetibm:mytag --file=Dockerfile-0.5.1-GPU-IntelMPI-xenial-devel . 224 | ``` 225 | 226 | ## `barbagroup/petibm:0.5.1-GPU-IntelMPI-xenial` 227 | 228 | This image is the same as `barbagroup/petibm:0.5.1-GPU-IntelMPI-xenial-devel`, except it does not contain the source files and build directories. 229 | This is intended to make the Docker image as light as possible. 230 | 231 | To pull the image from DockerHub: 232 | 233 | ```shell 234 | docker pull barbagroup/petibm:0.5.1-GPU-IntelMPI-xenial 235 | ``` 236 | 237 | To run a container: 238 | 239 | ```shell 240 | nvidia-docker run -it barbagroup/petibm:0.5.1-GPU-IntelMPI-xenial /bin/bash 241 | ``` 242 | 243 | ## `barbagroup/petibm:0.5-GPU-OpenMPI-xenial-devel` 244 | 245 | Image based on Ubuntu 16.04 (Xenial). 246 | To pull the image from DockerHub: 247 | 248 | ```shell 249 | docker pull barbagroup/petibm:0.5-GPU-OpenMPI-xenial-devel 250 | ``` 251 | 252 | PetIBM (0.5) was installed along with its dependencies: 253 | 254 | * CUDA Toolkit 10.1 (requires CUDA Driver Version >= 418.39) 255 | * OpenMPI 3.1.4 256 | * PETSc 3.12.2 257 | * AmgX 2.1 (commit [cf285c1](https://github.com/NVIDIA/AMGX/tree/cf285c118726f5d1e8eb740d4936dd0bdaaf9b48)) 258 | 259 | Docker containers based on this image can run PetIBM applications on CPU and GPU. 260 | Note that if you want to run a PetIBM application using GPUs, you have to create the container using the utility [`nvidia-docker v2`](https://github.com/NVIDIA/nvidia-docker) (instead of `docker`): 261 | 262 | ```shell 263 | nvidia-docker run -it barbagroup/petibm:0.5-GPU-OpenMPI-xenial-devel /bin/bash 264 | ``` 265 | 266 | To re-build the image locally: 267 | 268 | ```shell 269 | docker build --tag=mypetibm:mytag --file=Dockerfile-0.5-GPU-OpenMPI-xenial-devel . 270 | ``` 271 | 272 | ## `barbagroup/petibm:0.5-GPU-OpenMPI-xenial` 273 | 274 | This image is the same as `barbagroup/petibm:0.5-GPU-OpenMPI-xenial-devel`, except it does not contain the source files and build directories. 275 | This is intended to make the Docker image as light as possible. 276 | 277 | To pull the image from DockerHub: 278 | 279 | ```shell 280 | docker pull barbagroup/petibm:0.5-GPU-OpenMPI-xenial 281 | ``` 282 | 283 | To run a container: 284 | 285 | ```shell 286 | nvidia-docker run -it barbagroup/petibm:0.5-GPU-OpenMPI-xenial /bin/bash 287 | ``` 288 | 289 | ## `barbagroup/petibm:0.4.2-GPU-OpenMPI-xenial-devel` 290 | 291 | Image based on Ubuntu 16.04 (Xenial). 292 | To pull the image from DockerHub: 293 | 294 | ```shell 295 | docker pull barbagroup/petibm:0.4.2-GPU-OpenMPI-xenial-devel 296 | ``` 297 | 298 | PetIBM (0.4.2) was installed along with its dependencies: 299 | 300 | * CUDA Toolkit 10.1 (requires CUDA Driver Version >= 418.39) 301 | * OpenMPI 3.1.4 302 | * PETSc 3.11.4 303 | * AmgX 2.1 (commit [cf285c1](https://github.com/NVIDIA/AMGX/tree/cf285c118726f5d1e8eb740d4936dd0bdaaf9b48)) 304 | 305 | Docker containers based on this image can run PetIBM applications on CPU and GPU. 306 | Note that if you want to run a PetIBM application using GPUs, you have to create the container using the utility [`nvidia-docker v2`](https://github.com/NVIDIA/nvidia-docker) (instead of `docker`): 307 | 308 | ```shell 309 | nvidia-docker run -it barbagroup/petibm:0.4.2-GPU-OpenMPI-xenial-devel /bin/bash 310 | ``` 311 | 312 | To re-build the image locally: 313 | 314 | ```shell 315 | docker build --tag=mypetibm:mytag --file=Dockerfile-0.4.2-GPU-OpenMPI-xenial-devel . 316 | ``` 317 | 318 | ## `barbagroup/petibm:0.4.2-GPU-OpenMPI-xenial` 319 | 320 | This image is the same as `barbagroup/petibm:0.4.2-GPU-OpenMPI-xenial-devel`, except it does not contain the source files and build directories. 321 | This is intended to make the Docker image as light as possible. 322 | 323 | To pull the image from DockerHub: 324 | 325 | ```shell 326 | docker pull barbagroup/petibm:0.4.2-GPU-OpenMPI-xenial 327 | ``` 328 | 329 | To run a container: 330 | 331 | ```shell 332 | nvidia-docker run -it barbagroup/petibm:0.4.2-GPU-OpenMPI-xenial /bin/bash 333 | ``` 334 | -------------------------------------------------------------------------------- /docker/Dockerfile-0.5.2-GPU-OpenMPI-centos7: -------------------------------------------------------------------------------- 1 | # vim:ft=dockerfile 2 | # Packages: 3 | # - CUDA 10.1 4 | # - OpenMPI v4.0.5 (with Infiniband, UCX, Slurm, PMI, CUDA enabled) 5 | # - AmgX 919fb92 6 | # - PETSc 3.12.5 7 | # - AmgXWrapper 1a93865 8 | # - PetIBM v0.5.2 9 | 10 | 11 | # use ARGs as a hack for global variables 12 | ARG OMPI_SERIES="4.0" 13 | ARG OMPI_VERSION="4.0.5" 14 | ARG AMGX_VERSION="919fb92a2c02ea3d8d3d0f0660e3b507d84705bc" 15 | ARG PETSC_VERSION="3.12.5" 16 | ARG AMGXWRAPPER_VERSION="1a93865fe885b3a3530e4022aacc00845beb1177" 17 | ARG PETIBM_VERSION="0.5.2" 18 | 19 | # Multi-stage build stage 0: build 20 | # ================================= 21 | FROM nvidia/cuda:10.1-devel-centos7 as builder 22 | 23 | # re-include the global variables 24 | ARG OMPI_SERIES 25 | ARG OMPI_VERSION 26 | ARG AMGX_VERSION 27 | ARG PETSC_VERSION 28 | ARG AMGXWRAPPER_VERSION 29 | ARG PETIBM_VERSION 30 | 31 | # installation paths 32 | ENV OMPI_DIR="/usr/local/openmpi-${OMPI_VERSION}" \ 33 | AMGX_DIR="/usr/local/amgx-${AMGX_VERSION}" \ 34 | PETSC_DIR="/usr/local/petsc-${PETSC_VERSION}" \ 35 | PETSC_ARCH="" \ 36 | AMGXWRAPPER_DIR="/usr/local/amgxwrapper-${AMGXWRAPPER_VERSION}" \ 37 | PETIBM_DIR="/usr/local/petibm-${PETIBM_VERSION}" 38 | 39 | # add paths and headers' path 40 | ENV PATH="${OMPI_DIR}/bin:${PATH}" CPATH="${OMPI_DIR}/include:${CPATH}" 41 | ENV PATH="${AMGX_DIR}/bin:${PATH}" CPATH="${AMGX_DIR}/include:${CPATH}" 42 | ENV PATH="${PETSC_DIR}/bin:${PATH}" CPATH="${PETSC_DIR}/include:${CPATH}" 43 | ENV PATH="${AMGXWRAPPER_DIR}/bin:${PATH}" CPATH="${AMGXWRAPPER_DIR}/include:${CPATH}" 44 | ENV PATH="${PETIBM_DIR}/bin:${PATH}" CPATH="${PETIBM_DIR}/include:${CPATH}" 45 | 46 | # install required packages 47 | RUN yum -y group install --setopt=tsflags=nodocs "Development Tools" \ 48 | && yum -y install --setopt=tsflags=nodocs wget openssl-devel \ 49 | && yum -y install centos-release-scl \ 50 | && yum -y install --setopt=tsflags=nodocs devtoolset-8 rh-python36 \ 51 | && yum -y install --setopt=tsflags=nodocs zlib zlib-devel \ 52 | && yum -y group install --setopt=tsflags=nodocs "Infiniband Support" \ 53 | && yum -y install --setopt=tsflags=nodocs libverbs libfabric \ 54 | && yum -y install --setopt=tsflags=nodocs rdma-core-devel numactl-devel libfabric-devel 55 | 56 | # newer version of cmake 57 | RUN source scl_source enable devtoolset-8 rh-python36 \ 58 | && curl -LO https://cmake.org/files/v3.18/cmake-3.18.2.tar.gz \ 59 | && tar xf cmake-3.18.2.tar.gz && cd cmake-3.18.2 \ 60 | && ./bootstrap --prefix=/usr --parallel=6 \ 61 | && make -j 6 && make install \ 62 | && cd .. && rm -rf cmake-3.18.2 cmake-3.18.2.tar.gz 63 | 64 | # newer version of ucx 65 | RUN source scl_source enable devtoolset-8 rh-python36 \ 66 | && curl -LO https://github.com/openucx/ucx/releases/download/v1.8.1/ucx-1.8.1.tar.gz \ 67 | && tar xf ucx-1.8.1.tar.gz && cd ucx-1.8.1 && mkdir build && cd build \ 68 | && ../contrib/configure-release \ 69 | --prefix=/usr/local/ucx-1.8.1 \ 70 | --disable-doxygen-doc \ 71 | --with-cuda=/usr/local/cuda-10.1 \ 72 | --with-verbs \ 73 | --with-rc \ 74 | --with-ud \ 75 | --with-dc \ 76 | --with-mlx5-dv \ 77 | --with-ib-hw-tm \ 78 | --with-dm \ 79 | --with-rdmacm \ 80 | && make all -j && make install \ 81 | && cd ../.. && rm -rf ucx-1.8.1 ucx-1.8.1.tar.gz \ 82 | && echo "/usr/local/ucx-1.8.1/lib" > /etc/ld.so.conf.d/uxc-1.8.1.conf \ 83 | && echo "/usr/local/ucx-1.8.1/lib/ucx" >> /etc/ld.so.conf.d/uxc-1.8.1.conf \ 84 | && ldconfig 85 | 86 | # build OpenMPI 87 | RUN source scl_source enable devtoolset-8 rh-python36 \ 88 | && TARBALL=openmpi-${OMPI_VERSION}.tar.gz \ 89 | && curl -LO https://download.open-mpi.org/release/open-mpi/v${OMPI_SERIES}/${TARBALL} \ 90 | && tar xf ${TARBALL} \ 91 | && cd openmpi-${OMPI_VERSION} && mkdir build && cd build \ 92 | && ../configure \ 93 | --prefix=${OMPI_DIR} \ 94 | --with-hwloc \ 95 | --with-ucx=/usr/local/ucx-1.8.1 \ 96 | --with-ofi \ 97 | --with-slurm \ 98 | --with-pmix \ 99 | --with-zlib \ 100 | --with-cuda \ 101 | --with-verbs \ 102 | --with-verbs-libdir=/usr/lib64 \ 103 | --with-mpi-param-check=runtime \ 104 | --enable-oshmem=yes \ 105 | --enable-debug=no \ 106 | --enable-mem-debug=no \ 107 | --enable-mem-profile=no \ 108 | --enable-memchecker=no \ 109 | --enable-picky=no \ 110 | --enable-heterogeneous=no \ 111 | CXXFLAGS="-O3" \ 112 | CCASFLAGS="-O3" \ 113 | FCFLAGS="-O3" \ 114 | CFLAGS="-O3" \ 115 | && make all -j && make install \ 116 | && cd ../.. && rm -rf openmpi-${OMPI_VERSION} ${TARBALL} \ 117 | && echo "${OMPI_DIR}/lib" > /etc/ld.so.conf.d/openmpi-${OMPI_VERSION}.conf && ldconfig 118 | 119 | # build amgx 120 | RUN source scl_source enable devtoolset-8 rh-python36 \ 121 | && curl -L https://github.com/NVIDIA/AMGX/tarball/${AMGX_VERSION} -o amgx.tar.gz \ 122 | && mkdir amgx && tar -xf amgx.tar.gz -C amgx --strip-component=1 \ 123 | && export CC=mpicc CXX=mpicxx && mkdir amgx/build && cd amgx/build \ 124 | && cmake \ 125 | -DCMAKE_BUILD_TYPE=Release \ 126 | -DCMAKE_INSTALL_PREFIX=${AMGX_DIR} \ 127 | -DCMAKE_VERBOSE_MAKEFILE=OFF \ 128 | -DCUDA_ARCH="35 52 60 70" \ 129 | -DCMAKE_NO_MPI=OFF \ 130 | -DAMGX_NO_RPATH=OFF \ 131 | .. \ 132 | && make -j 6 all && make install \ 133 | && cd ../.. && rm -rf amgx amgx.tar.gz \ 134 | && echo "${AMGX_DIR}/lib" > /etc/ld.so.conf.d/amgx-${AMGX_VERSION}.conf && ldconfig 135 | 136 | # build PETSc 137 | RUN source scl_source enable devtoolset-8 rh-python36 \ 138 | && unset PETSC_ARCH && unset PETSC_DIR \ 139 | && curl -L http://ftp.mcs.anl.gov/pub/petsc/release-snapshots/petsc-lite-${PETSC_VERSION}.tar.gz -o petsc.tar.gz \ 140 | && mkdir petsc && tar -xf petsc.tar.gz -C petsc --strip-component=1 && cd petsc \ 141 | && ./configure \ 142 | --prefix="/usr/local/petsc-${PETSC_VERSION}" \ 143 | --with-default-arch=0 \ 144 | --with-fortran-bindings=0 \ 145 | --with-precision=double \ 146 | --with-clanguage=C \ 147 | --with-shared-libraries=1 \ 148 | --with-cc=mpicc \ 149 | --with-cxx=mpicxx \ 150 | --with-fc=0 \ 151 | --with-pic=1 \ 152 | --with-cxx-dialect=C++11 \ 153 | --with-debugging=0 \ 154 | --COPTFLAGS="-O3" \ 155 | --CXXOPTFLAGS="-O3" \ 156 | --with-gcov=0 \ 157 | --with-mpi=1 \ 158 | --download-f2cblaslapack=1 \ 159 | --download-hypre=1 \ 160 | --download-superlu_dist=1 \ 161 | --download-hdf5=1 \ 162 | && make PETSC_DIR=${PWD} PETSC_ARCH=arch-linux-c-opt all \ 163 | && make PETSC_DIR=${PWD} PETSC_ARCH=arch-linux-c-opt install \ 164 | && cd .. && rm -rf petsc petsc.tar.gz \ 165 | && echo "${PETSC_DIR}/lib" > /etc/ld.so.conf.d/petsc-${PETSC_VERSION}.conf && ldconfig 166 | 167 | # build AmgXWrapper 168 | RUN source scl_source enable devtoolset-8 rh-python36 \ 169 | && curl -L https://github.com/barbagroup/AmgXWrapper/tarball/${AMGXWRAPPER_VERSION} -o amgxwrapper.tar.gz \ 170 | && mkdir amgxwrapper && tar -xf amgxwrapper.tar.gz -C amgxwrapper --strip-component=1 \ 171 | && cd amgxwrapper && mkdir build && cd build \ 172 | && export CC=mpicc && export CXX=mpicxx \ 173 | && cmake -DCUDA_DIR=/usr/local/cuda-10.1 -DCMAKE_INSTALL_PREFIX=${AMGXWRAPPER_DIR} .. \ 174 | && make all -j && make install \ 175 | && cd .. && mkdir build_poisson && cd build_poisson \ 176 | && cmake -DCUDA_DIR=/usr/local/cuda-10.1 ../example/poisson && make all -j \ 177 | && mkdir ${AMGXWRAPPER_DIR}/bin && cp bin/poisson ${AMGXWRAPPER_DIR}/bin \ 178 | && cp -r configs ${AMGXWRAPPER_DIR} \ 179 | && cd .. && mkdir build_solveFromFiles && cd build_solveFromFiles \ 180 | && cmake -DCUDA_DIR=/usr/local/cuda-10.1 ../example/solveFromFiles && make all -j \ 181 | && cp bin/solveFromFiles ${AMGXWRAPPER_DIR}/bin \ 182 | && cd ../.. && rm -rf amgxwrapper amgxwrapper.tar.gz \ 183 | && echo "${AMGXWRAPPER_DIR}/lib64" > /etc/ld.so.conf.d/amgxwrapper-${AMGXWRAPPER_VERSION}.conf && ldconfig 184 | 185 | # build PetIBM 186 | RUN source scl_source enable devtoolset-8 rh-python36 \ 187 | && curl -L https://github.com/barbagroup/PetIBM/tarball/v${PETIBM_VERSION} -o petibm.tar.gz \ 188 | && mkdir petibm && tar xf petibm.tar.gz -C petibm --strip-component=1 \ 189 | && cd petibm && mkdir build && cd build \ 190 | && ../configure \ 191 | CXX=mpicxx \ 192 | CXXFLAGS="-O2 -std=c++14" \ 193 | --prefix=${PETIBM_DIR} \ 194 | --with-petsc-dir=${PETSC_DIR} \ 195 | --with-petsc-arch=${PETSC_ARCH} \ 196 | --with-cuda-dir=/usr/local/cuda-10.1 \ 197 | --with-amgx-dir=${AMGX_DIR} \ 198 | --with-amgxwrapper-dir=${AMGXWRAPPER_DIR} \ 199 | --enable-yamlcpp \ 200 | --enable-gtest \ 201 | && make all -j && make check && make install \ 202 | && cd ../.. && rm -rf petibm petibm.tar.gz \ 203 | && echo "${PETIBM_DIR}/lib" > /etc/ld.so.conf.d/petibm-${PETIBM_VERSION}.conf && ldconfig 204 | 205 | 206 | # Multi-stage build stage 1: runtime image 207 | # ========================================= 208 | FROM nvidia/cuda:10.1-runtime-centos7 as production 209 | LABEL maintainer="Pi-Yueh Chuang " 210 | 211 | # re-include the global variables 212 | ARG OMPI_SERIES 213 | ARG OMPI_VERSION 214 | ARG AMGX_VERSION 215 | ARG PETSC_VERSION 216 | ARG AMGXWRAPPER_VERSION 217 | ARG PETIBM_VERSION 218 | 219 | ENV OMPI_DIR="/usr/local/openmpi-${OMPI_VERSION}" \ 220 | AMGX_DIR="/usr/local/amgx-${AMGX_VERSION}" \ 221 | PETSC_DIR="/usr/local/petsc-${PETSC_VERSION}" \ 222 | PETSC_ARCH="" \ 223 | AMGXWRAPPER_DIR="/usr/local/amgxwrapper-${AMGXWRAPPER_VERSION}" \ 224 | PETIBM_DIR="/usr/local/petibm-${PETIBM_VERSION}" 225 | 226 | # add paths 227 | ENV PATH="${OMPI_DIR}/bin:${PATH}" 228 | ENV PATH="${AMGX_DIR}/bin:${PATH}" 229 | ENV PATH="${PETSC_DIR}/bin:${PATH}" 230 | ENV PATH="${AMGXWRAPPER_DIR}/bin:${PATH}" 231 | ENV PATH="${PETIBM_DIR}/bin:${PATH}" 232 | 233 | # install required packages and settings 234 | RUN localedef -c -i en_US -f UTF-8 en_US.UTF-8 \ 235 | && ln -snf /usr/share/zoneinfo/UTC /etc/localtime \ 236 | && echo UTC > /etc/timezone \ 237 | && yum -y install --setopt=tsflags=nodocs zlib libquadmath \ 238 | && yum -y group install --setopt=tsflags=nodocs "Infiniband Support" \ 239 | && yum -y install --setopt=tsflags=nodocs libverbs libfabric \ 240 | && yum -y install --setopt=tsflags=nodocs openssh-clients openssh-server \ 241 | && yum -y clean all 242 | 243 | # copy installed programs 244 | COPY --from=builder /usr/local/ucx-1.8.1 /usr/local/ucx-1.8.1 245 | COPY --from=builder ${OMPI_DIR} ${OMPI_DIR} 246 | COPY --from=builder ${AMGX_DIR} ${AMGX_DIR} 247 | COPY --from=builder ${PETSC_DIR} ${PETSC_DIR} 248 | COPY --from=builder ${AMGXWRAPPER_DIR} ${AMGXWRAPPER_DIR} 249 | COPY --from=builder ${PETIBM_DIR} ${PETIBM_DIR} 250 | 251 | # ldconfig 252 | RUN echo "/usr/local/ucx-1.8.1/lib" > /etc/ld.so.conf.d/uxc-1.8.1.conf \ 253 | && echo "/usr/local/ucx-1.8.1/lib/ucx" >> /etc/ld.so.conf.d/uxc-1.8.1.conf \ 254 | && echo "${OMPI_DIR}/lib" > /etc/ld.so.conf.d/openmpi-${OMPI_VERSION}.conf \ 255 | && echo "${AMGX_DIR}/lib" > /etc/ld.so.conf.d/amgx-${AMGX_VERSION}.conf \ 256 | && echo "${PETSC_DIR}/lib" > /etc/ld.so.conf.d/petsc-${PETSC_VERSION}.conf \ 257 | && echo "${AMGXWRAPPER_DIR}/lib64" > /etc/ld.so.conf.d/amgxwrapper-${AMGXWRAPPER_VERSION}.conf \ 258 | && echo "${PETIBM_DIR}/lib64" > /etc/ld.so.conf.d/petibm-${PETIBM_VERSION}.conf \ 259 | && ldconfig 260 | 261 | # set up ssh for root for MPI applications 262 | RUN mkdir /var/run/sshd \ 263 | && ssh-keygen -A \ 264 | && sed -i "s/#PermitRootLogin\ yes/PermitRootLogin\ yes/" /etc/ssh/sshd_config \ 265 | && sed -i "s/session\s*required\s*pam_loginuid.so/session\ optional\ pam_loginuid.so/g" /etc/pam.d/sshd \ 266 | && ssh-keygen -f /root/.ssh/id_rsa -t rsa -N '' \ 267 | && echo "Host 10.*" > /root/.ssh/config \ 268 | && echo " Port 23" >> /root/.ssh/config \ 269 | && echo " StrictHostKeyChecking no" >> /root/.ssh/config \ 270 | && echo " UserKnownHostsFile /dev/null" >> /root/.ssh/config \ 271 | && chmod 600 /root/.ssh/config && chmod 700 /root/.ssh \ 272 | && cp /root/.ssh/id_rsa.pub /root/.ssh/authorized_keys 273 | 274 | # using port 23 for Azure cloud, see 275 | # https://batch-shipyard.readthedocs.io/en/3.0.1/80-batch-shipyard-multi-instance-tasks 276 | EXPOSE 23 277 | CMD ["/usr/sbin/sshd", "-D", "-p", "23"] 278 | --------------------------------------------------------------------------------