├── .travis.yml ├── Dockerfile ├── Readme.md ├── default-mca-params.conf ├── docker-compose.yml ├── makefile ├── mpi4py_benchmarks ├── all_tests.py ├── matrix_vector_product.py ├── osu_bibw.py ├── osu_bw.py └── osu_latency.py └── ssh ├── config ├── id_rsa.mpi └── id_rsa.mpi.pub /.travis.yml: -------------------------------------------------------------------------------- 1 | sudo: 2 | required 3 | 4 | language: 5 | c 6 | 7 | services: 8 | - docker 9 | 10 | env: 11 | - NNODES=2 12 | 13 | before_install: 14 | # # update Docker 15 | - sudo apt-get update 16 | - sudo apt-get -y -o Dpkg::Options::="--force-confnew" install docker-engine wget 17 | - wget https://github.com/docker/compose/releases/download/1.7.1/docker-compose-`uname -s`-`uname -m` 18 | - sudo mv docker-compose-`uname -s`-`uname -m` /usr/local/bin/docker-compose 19 | - sudo chmod +x /usr/local/bin/docker-compose 20 | 21 | script: 22 | - make main -------------------------------------------------------------------------------- /Dockerfile: -------------------------------------------------------------------------------- 1 | # Build this image: docker build -t mpi . 2 | # 3 | 4 | FROM ubuntu:18.04 5 | # FROM phusion/baseimage 6 | 7 | MAINTAINER Ole Weidner 8 | 9 | ENV USER mpirun 10 | 11 | ENV DEBIAN_FRONTEND=noninteractive \ 12 | HOME=/home/${USER} 13 | 14 | 15 | RUN apt-get update -y && \ 16 | apt-get install -y --no-install-recommends sudo apt-utils && \ 17 | apt-get install -y --no-install-recommends openssh-server \ 18 | python-dev python-numpy python-pip python-virtualenv python-scipy \ 19 | gcc gfortran libopenmpi-dev openmpi-bin openmpi-common openmpi-doc binutils && \ 20 | apt-get clean && apt-get purge && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* 21 | 22 | RUN mkdir /var/run/sshd 23 | RUN echo 'root:${USER}' | chpasswd 24 | RUN sed -i 's/PermitRootLogin without-password/PermitRootLogin yes/' /etc/ssh/sshd_config 25 | 26 | # SSH login fix. Otherwise user is kicked off after login 27 | RUN sed 's@session\s*required\s*pam_loginuid.so@session optional pam_loginuid.so@g' -i /etc/pam.d/sshd 28 | 29 | ENV NOTVISIBLE "in users profile" 30 | RUN echo "export VISIBLE=now" >> /etc/profile 31 | 32 | # ------------------------------------------------------------ 33 | # Add an 'mpirun' user 34 | # ------------------------------------------------------------ 35 | 36 | RUN adduser --disabled-password --gecos "" ${USER} && \ 37 | echo "${USER} ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers 38 | 39 | # ------------------------------------------------------------ 40 | # Set-Up SSH with our Github deploy key 41 | # ------------------------------------------------------------ 42 | 43 | ENV SSHDIR ${HOME}/.ssh/ 44 | 45 | RUN mkdir -p ${SSHDIR} 46 | 47 | ADD ssh/config ${SSHDIR}/config 48 | ADD ssh/id_rsa.mpi ${SSHDIR}/id_rsa 49 | ADD ssh/id_rsa.mpi.pub ${SSHDIR}/id_rsa.pub 50 | ADD ssh/id_rsa.mpi.pub ${SSHDIR}/authorized_keys 51 | 52 | RUN chmod -R 600 ${SSHDIR}* && \ 53 | chown -R ${USER}:${USER} ${SSHDIR} 54 | 55 | RUN pip install --upgrade pip 56 | 57 | USER ${USER} 58 | RUN pip install --user -U setuptools \ 59 | && pip install --user mpi4py 60 | 61 | # ------------------------------------------------------------ 62 | # Configure OpenMPI 63 | # ------------------------------------------------------------ 64 | 65 | USER root 66 | 67 | RUN rm -fr ${HOME}/.openmpi && mkdir -p ${HOME}/.openmpi 68 | ADD default-mca-params.conf ${HOME}/.openmpi/mca-params.conf 69 | RUN chown -R ${USER}:${USER} ${HOME}/.openmpi 70 | 71 | # ------------------------------------------------------------ 72 | # Copy MPI4PY example scripts 73 | # ------------------------------------------------------------ 74 | 75 | ENV TRIGGER 1 76 | 77 | ADD mpi4py_benchmarks ${HOME}/mpi4py_benchmarks 78 | RUN chown -R ${USER}:${USER} ${HOME}/mpi4py_benchmarks 79 | 80 | EXPOSE 22 81 | CMD ["/usr/sbin/sshd", "-D"] 82 | -------------------------------------------------------------------------------- /Readme.md: -------------------------------------------------------------------------------- 1 | ## docker.openmpi 2 | 3 | Travis CI: [![Build Status](https://travis-ci.org/ocramz/docker.openmpi.svg?branch=master)](https://travis-ci.org/ocramz/docker.openmpi) 4 | 5 | With the code in this repository, you can build a Docker container that provides 6 | the OpenMPI runtime and tools along with various supporting libaries, 7 | including the MPI4Py Python bindings. The container also runs an OpenSSH server 8 | so that multiple containers can be linked together and used via `mpirun`. 9 | 10 | 11 | ## MPI Container Cluster with `docker-compose` 12 | 13 | While containers can in principle be started manually via `docker run`, we suggest that your use 14 | [Docker Compose](https://docs.docker.com/compose/), a simple command-line tool 15 | to define and run multi-container applications. We provide a sample `docker-compose.yml` file in the repository: 16 | 17 | ``` 18 | mpi_head: 19 | image: openmpi 20 | ports: 21 | - "22" 22 | links: 23 | - mpi_node 24 | 25 | mpi_node: 26 | image: openmpi 27 | 28 | ``` 29 | (Note: the above is docker-compose API version 1) 30 | 31 | The file defines an `mpi_head` and an `mpi_node`. Both containers run the same `openmpi` image. 32 | The only difference is, that the `mpi_head` container exposes its SSH server to 33 | the host system, so you can log into it to start your MPI applications. 34 | 35 | 36 | ## Usage 37 | 38 | The following command, run from the repository's directory, will start one `mpi_head` container and three `mpi_node` containers: 39 | 40 | ``` 41 | $> docker-compose scale mpi_head=1 mpi_node=3 42 | ``` 43 | Once all containers are running, you can login into the `mpi_head` node and start MPI jobs with `mpirun`. Alternatively, you can execute a one-shot command on that container with the `docker-compose exec` syntax, as follows: 44 | 45 | docker-compose exec --user mpirun --privileged mpi_head mpirun -n 2 python /home/mpirun/mpi4py_benchmarks/all_tests.py 46 | ------------------------------------------------------- ----------- -------------------------------------------------- 47 | 1. 2. 3. 48 | 49 | Breaking the above command down: 50 | 51 | 1. Execute command on node `mpi_head` 52 | 2. Run on 2 MPI ranks 53 | 3. Command to run (NB: the Python script needs to import MPI bindings) 54 | 55 | ## Testing 56 | 57 | You can spin up a docker-compose cluster, run a battery of MPI4py tests and remove the cluster using a recipe provided in the included Makefile (handy for development): 58 | 59 | make main 60 | 61 | 62 | ## Credits 63 | 64 | This repository draws from work on https://github.com/dispel4py/ by O. Weidner and R. Filgueira 65 | -------------------------------------------------------------------------------- /default-mca-params.conf: -------------------------------------------------------------------------------- 1 | btl=tcp,self 2 | btl_tcp_if_include=eth0 3 | plm_rsh_no_tree_spawn=1 4 | -------------------------------------------------------------------------------- /docker-compose.yml: -------------------------------------------------------------------------------- 1 | mpi_head: 2 | build: . 3 | # image: openmpi 4 | ports: 5 | - "22" 6 | links: 7 | - mpi_node 8 | 9 | mpi_node: 10 | build: . 11 | # image: openmpi 12 | 13 | 14 | # version: "2" 15 | 16 | # services: 17 | # mpi_head: 18 | # build: . 19 | # # image: openmpi 20 | # ports: 21 | # - "22" 22 | # links: 23 | # - mpi_node 24 | # networks: 25 | # - net 26 | 27 | # mpi_node: 28 | # build: . 29 | # # image: openmpi 30 | # networks: 31 | # - net 32 | 33 | # networks: 34 | # net: 35 | # driver: bridge -------------------------------------------------------------------------------- /makefile: -------------------------------------------------------------------------------- 1 | AUTH=ocramz 2 | NAME=docker-openmpi 3 | TAG=${AUTH}/${NAME} 4 | 5 | export NNODES=4 6 | 7 | .DEFAULT_GOAL := help 8 | 9 | help: 10 | @echo "Use \`make \` where is one of" 11 | @echo " help display this help message" 12 | @echo " build build from Dockerfile" 13 | @echo " rebuild rebuild from Dockerfile (ignores cached layers)" 14 | @echo " main build and docker-compose the whole thing" 15 | 16 | build: 17 | docker build -t $(TAG) . 18 | 19 | rebuild: 20 | docker build --no-cache -t $(TAG) . 21 | 22 | main: 23 | # 1 worker node 24 | docker-compose scale mpi_head=1 mpi_node=1 25 | docker-compose exec --privileged mpi_head mpirun -n 1 python /home/mpirun/mpi4py_benchmarks/all_tests.py 26 | docker-compose down 27 | 28 | # 2 worker nodes 29 | docker-compose scale mpi_head=1 mpi_node=2 30 | docker-compose exec --privileged mpi_head mpirun -n 2 python /home/mpirun/mpi4py_benchmarks/all_tests.py 31 | docker-compose down 32 | 33 | # ${NNODES} worker nodes 34 | docker-compose scale mpi_head=1 mpi_node=${NNODES} 35 | docker-compose exec --privileged mpi_head mpirun -n ${NNODES} python /home/mpirun/mpi4py_benchmarks/all_tests.py 36 | docker-compose down 37 | -------------------------------------------------------------------------------- /mpi4py_benchmarks/all_tests.py: -------------------------------------------------------------------------------- 1 | """ 2 | MVP : Demonstrating a MPI parallel Matrix-Vector Multiplication. 3 | This code will run *iter* iterations of 4 | v(t+1) = M * v(t) 5 | where v is a vector of length *size* and M a dense size*size 6 | matrix. *size* must be an integer multiple of comm.size. 7 | v is initialized to be zero except of v[0] = 1.0 8 | M is a "off-by-one" diagonal matrix M[i, i+1] = 1.0 9 | In effect, after *iter* iterations, the vector v should look like 10 | v[iter] = 1. (all others zero). 11 | In this example every MPI process is responsible for calculating a 12 | different portion of v. Every process only knows the stripe of M, that 13 | is relevant for it's calculation. At the end of every iteration, 14 | Allgather is used to distribute the partial vectors v to all other 15 | processes. 16 | """ 17 | 18 | from __future__ import division 19 | 20 | import numpy as np 21 | # from numpy.fft import fft2, ifft2 22 | from math import ceil, fabs 23 | from mpi4py import MPI 24 | 25 | #============================================================================= 26 | # I/O Utilities 27 | 28 | def pprint(str="", end="\n", comm=MPI.COMM_WORLD): 29 | """Print for MPI parallel programs: Only rank 0 prints *str*.""" 30 | if comm.rank == 0: 31 | print str+end, 32 | 33 | #============================================================================= 34 | # Main 35 | 36 | def mvp_main(BENCHMARH="MPI Matrix action on a vector", 37 | size=10000, 38 | iter=200): 39 | # size = 10000 # length of vector v 40 | # iter = 200 # number of iterations to run 41 | 42 | counter = 0 43 | 44 | comm = MPI.COMM_WORLD 45 | myid = comm.Get_rank() 46 | 47 | pprint("============================================================================") 48 | pprint(" Running %d parallel MPI processes" % comm.size) 49 | 50 | 51 | my_size = size // comm.size # Every process computes a vector of lenth *my_size* 52 | size = comm.size*my_size # size must be integer multiple of comm.size 53 | my_offset = comm.rank*my_size 54 | 55 | bs = 20 # batch size 56 | 57 | if myid == 0: 58 | print ('# %s, %d iterations of size %d' % (BENCHMARH, bs, size)) 59 | print ('# %-8s%20s' % ("Duration [s]", "Throughput [#/s]")) 60 | 61 | # pprint(" %d iterations of size %d " % (bs, size)) 62 | 63 | 64 | 65 | # This is the complete vector 66 | vec = np.zeros(size) # Every element zero... 67 | vec[0] = 1.0 # ... besides vec[0] 68 | 69 | # Create my (local) slice of the matrix 70 | my_M = np.zeros((my_size, size)) 71 | for i in xrange(my_size): 72 | j = (my_offset+i-1) % size 73 | my_M[i,j] = 1.0 74 | 75 | 76 | while counter < iter: 77 | comm.Barrier() ### Start stopwatch ### 78 | t_start = MPI.Wtime() 79 | 80 | for t in xrange(20): 81 | my_new_vec = np.inner(my_M, vec) 82 | 83 | comm.Allgather( 84 | [my_new_vec, MPI.DOUBLE], 85 | [vec, MPI.DOUBLE] 86 | ) 87 | 88 | comm.Barrier() 89 | t_diff = MPI.Wtime() - t_start ### Stop stopwatch ### 90 | 91 | # if fabs(vec[iter]-1.0) > 0.01: 92 | # pprint("!! Error: Wrong result!") 93 | 94 | # pprint(" %d iterations of size %d in %5.2fs: %5.2f iterations per second" % 95 | # (bs, size, t_diff, bs/t_diff) 96 | # ) 97 | if myid == 0: 98 | print ('%-10.3f%20.2f' % (t_diff, bs/t_diff)) 99 | 100 | counter += bs 101 | 102 | 103 | 104 | 105 | def osu_latency( 106 | BENCHMARH = "MPI Latency Test", 107 | skip = 1000, 108 | loop = 10000, 109 | skip_large = 10, 110 | loop_large = 100, 111 | large_message_size = 8192, 112 | MAX_MSG_SIZE = 1<<22, 113 | ): 114 | 115 | comm = MPI.COMM_WORLD 116 | myid = comm.Get_rank() 117 | numprocs = comm.Get_size() 118 | 119 | s_buf = allocate(MAX_MSG_SIZE) 120 | r_buf = allocate(MAX_MSG_SIZE) 121 | 122 | if myid == 0: 123 | print ('# %s' % (BENCHMARH,)) 124 | print ('# %-8s%20s' % ("Size [B]", "Latency [us]")) 125 | 126 | message_sizes = [0] + [2**i for i in range(30)] 127 | for size in message_sizes: 128 | if size > MAX_MSG_SIZE: 129 | break 130 | if size > large_message_size: 131 | skip = skip_large 132 | loop = loop_large 133 | iterations = list(range(loop+skip)) 134 | s_msg = [s_buf, size, MPI.BYTE] 135 | r_msg = [r_buf, size, MPI.BYTE] 136 | 137 | comm.Barrier() 138 | if myid == 0: 139 | for i in iterations: 140 | if i == skip: 141 | t_start = MPI.Wtime() 142 | comm.Send(s_msg, 1, 1) 143 | comm.Recv(r_msg, 1, 1) 144 | t_end = MPI.Wtime() 145 | elif myid == 1: 146 | for i in iterations: 147 | comm.Recv(r_msg, 0, 1) 148 | comm.Send(s_msg, 0, 1) 149 | 150 | if myid == 0: 151 | latency = (t_end - t_start) * 1e6 / (2 * loop) 152 | print ('%-10d%20.2f' % (size, latency)) 153 | 154 | def osu_bibw( 155 | BENCHMARH = "MPI Bi-Directional Bandwidth Test", 156 | skip = 10, 157 | loop = 100, 158 | window_size = 64, 159 | skip_large = 2, 160 | loop_large = 20, 161 | window_size_large = 64, 162 | large_message_size = 8192, 163 | MAX_MSG_SIZE = 1<<22, 164 | ): 165 | 166 | comm = MPI.COMM_WORLD 167 | myid = comm.Get_rank() 168 | numprocs = comm.Get_size() 169 | 170 | s_buf = allocate(MAX_MSG_SIZE) 171 | r_buf = allocate(MAX_MSG_SIZE) 172 | 173 | if myid == 0: 174 | print ('# %s' % (BENCHMARH,)) 175 | print ('# %-8s%20s' % ("Size [B]", "Bandwidth [MB/s]")) 176 | 177 | message_sizes = [2**i for i in range(30)] 178 | for size in message_sizes: 179 | if size > MAX_MSG_SIZE: 180 | break 181 | if size > large_message_size: 182 | skip = skip_large 183 | loop = loop_large 184 | window_size = window_size_large 185 | 186 | iterations = list(range(loop+skip)) 187 | window_sizes = list(range(window_size)) 188 | s_msg = [s_buf, size, MPI.BYTE] 189 | r_msg = [r_buf, size, MPI.BYTE] 190 | send_request = [MPI.REQUEST_NULL] * window_size 191 | recv_request = [MPI.REQUEST_NULL] * window_size 192 | # 193 | comm.Barrier() 194 | if myid == 0: 195 | for i in iterations: 196 | if i == skip: 197 | t_start = MPI.Wtime() 198 | for j in window_sizes: 199 | recv_request[j] = comm.Irecv(r_msg, 1, 10) 200 | for j in window_sizes: 201 | send_request[j] = comm.Isend(s_msg, 1, 100) 202 | MPI.Request.Waitall(send_request) 203 | MPI.Request.Waitall(recv_request) 204 | t_end = MPI.Wtime() 205 | elif myid == 1: 206 | for i in iterations: 207 | for j in window_sizes: 208 | recv_request[j] = comm.Irecv(r_msg, 0, 100) 209 | for j in window_sizes: 210 | send_request[j] = comm.Isend(s_msg, 0, 10) 211 | MPI.Request.Waitall(send_request) 212 | MPI.Request.Waitall(recv_request) 213 | # 214 | if myid == 0: 215 | MB = size / 1e6 * loop * window_size 216 | s = t_end - t_start 217 | print ('%-10d%20.2f' % (size, MB/s)) 218 | 219 | 220 | def osu_bw( 221 | BENCHMARH = "MPI Bandwidth Test", 222 | skip = 10, 223 | loop = 100, 224 | window_size = 64, 225 | skip_large = 2, 226 | loop_large = 20, 227 | window_size_large = 64, 228 | large_message_size = 8192, 229 | MAX_MSG_SIZE = 1<<22, 230 | ): 231 | 232 | comm = MPI.COMM_WORLD 233 | myid = comm.Get_rank() 234 | numprocs = comm.Get_size() 235 | 236 | s_buf = allocate(MAX_MSG_SIZE) 237 | r_buf = allocate(MAX_MSG_SIZE) 238 | 239 | if myid == 0: 240 | print ('# %s' % (BENCHMARH,)) 241 | print ('# %-8s%20s' % ("Size [B]", "Bandwidth [MB/s]")) 242 | 243 | message_sizes = [2**i for i in range(30)] 244 | for size in message_sizes: 245 | if size > MAX_MSG_SIZE: 246 | break 247 | if size > large_message_size: 248 | skip = skip_large 249 | loop = loop_large 250 | window_size = window_size_large 251 | 252 | iterations = list(range(loop+skip)) 253 | window_sizes = list(range(window_size)) 254 | requests = [MPI.REQUEST_NULL] * window_size 255 | # 256 | comm.Barrier() 257 | if myid == 0: 258 | s_msg = [s_buf, size, MPI.BYTE] 259 | r_msg = [r_buf, 4, MPI.BYTE] 260 | for i in iterations: 261 | if i == skip: 262 | t_start = MPI.Wtime() 263 | for j in window_sizes: 264 | requests[j] = comm.Isend(s_msg, 1, 100) 265 | MPI.Request.Waitall(requests) 266 | comm.Recv(r_msg, 1, 101) 267 | t_end = MPI.Wtime() 268 | elif myid == 1: 269 | s_msg = [s_buf, 4, MPI.BYTE] 270 | r_msg = [r_buf, size, MPI.BYTE] 271 | for i in iterations: 272 | for j in window_sizes: 273 | requests[j] = comm.Irecv(r_msg, 0, 100) 274 | MPI.Request.Waitall(requests) 275 | comm.Send(s_msg, 0, 101) 276 | # 277 | if myid == 0: 278 | MB = size / 1e6 * loop * window_size 279 | s = t_end - t_start 280 | print ('%-10d%20.2f' % (size, MB/s)) 281 | 282 | 283 | 284 | def allocate(n): 285 | try: 286 | import mmap 287 | return mmap.mmap(-1, n) 288 | except (ImportError, EnvironmentError): 289 | try: 290 | from numpy import zeros 291 | return zeros(n, 'B') 292 | except ImportError: 293 | from array import array 294 | return array('B', [0]) * n 295 | 296 | 297 | if __name__ == '__main__': 298 | mvp_main() 299 | comm = MPI.COMM_WORLD 300 | myid = comm.Get_rank() 301 | numprocs = comm.Get_size() 302 | if numprocs==2 : 303 | osu_latency() 304 | osu_bw() 305 | osu_bibw() 306 | else: 307 | if myid==0: 308 | print ("# Warning ! OSU examples require MPI rank size == 2. Not running since rank size = %d" % (numprocs)) 309 | -------------------------------------------------------------------------------- /mpi4py_benchmarks/matrix_vector_product.py: -------------------------------------------------------------------------------- 1 | """ 2 | Demonstrating a MPI parallel Matrix-Vector Multiplication. 3 | This code will run *iter* iterations of 4 | v(t+1) = M * v(t) 5 | where v is a vector of length *size* and M a dense size*size 6 | matrix. *size* must be an integer multiple of comm.size. 7 | v is initialized to be zero except of v[0] = 1.0 8 | M is a "off-by-one" diagonal matrix M[i, i+1] = 1.0 9 | In effect, after *iter* iterations, the vector v should look like 10 | v[iter] = 1. (all others zero). 11 | In this example every MPI process is responsible for calculating a 12 | different portion of v. Every process only knows the stripe of M, that 13 | is relevant for it's calculation. At the end of every iteration, 14 | Allgather is used to distribute the partial vectors v to all other 15 | processes. 16 | """ 17 | 18 | from __future__ import division 19 | 20 | import numpy as np 21 | # from numpy.fft import fft2, ifft2 22 | from math import ceil, fabs 23 | from mpi4py import MPI 24 | 25 | #============================================================================= 26 | # I/O Utilities 27 | 28 | def pprint(str="", end="\n", comm=MPI.COMM_WORLD): 29 | """Print for MPI parallel programs: Only rank 0 prints *str*.""" 30 | if comm.rank == 0: 31 | print str+end, 32 | 33 | #============================================================================= 34 | # Main 35 | 36 | size = 10000 # length of vector v 37 | iter = 2000 # number of iterations to run 38 | 39 | counter = 0 40 | 41 | comm = MPI.COMM_WORLD 42 | 43 | pprint("============================================================================") 44 | pprint(" Running %d parallel MPI processes" % comm.size) 45 | 46 | my_size = size // comm.size # Every process computes a vector of lenth *my_size* 47 | size = comm.size*my_size # Make sure size is a integer multiple of comm.size 48 | my_offset = comm.rank*my_size 49 | 50 | # This is the complete vector 51 | vec = np.zeros(size) # Every element zero... 52 | vec[0] = 1.0 # ... besides vec[0] 53 | 54 | # Create my (local) slice of the matrix 55 | my_M = np.zeros((my_size, size)) 56 | for i in xrange(my_size): 57 | j = (my_offset+i-1) % size 58 | my_M[i,j] = 1.0 59 | 60 | 61 | while counter < iter: 62 | comm.Barrier() ### Start stopwatch ### 63 | t_start = MPI.Wtime() 64 | 65 | for t in xrange(20): 66 | my_new_vec = np.inner(my_M, vec) 67 | 68 | comm.Allgather( 69 | [my_new_vec, MPI.DOUBLE], 70 | [vec, MPI.DOUBLE] 71 | ) 72 | 73 | comm.Barrier() 74 | t_diff = MPI.Wtime() - t_start ### Stop stopwatch ### 75 | 76 | # if fabs(vec[iter]-1.0) > 0.01: 77 | # pprint("!! Error: Wrong result!") 78 | 79 | pprint(" %d iterations of size %d in %5.2fs: %5.2f iterations per second" % 80 | (20, size, t_diff, 20/t_diff) 81 | ) 82 | 83 | counter += 20 84 | -------------------------------------------------------------------------------- /mpi4py_benchmarks/osu_bibw.py: -------------------------------------------------------------------------------- 1 | # http://mvapich.cse.ohio-state.edu/benchmarks/ 2 | 3 | from mpi4py import MPI 4 | 5 | def osu_bw( 6 | BENCHMARH = "MPI Bi-Directional Bandwidth Test", 7 | skip = 10, 8 | loop = 100, 9 | window_size = 64, 10 | skip_large = 2, 11 | loop_large = 20, 12 | window_size_large = 64, 13 | large_message_size = 8192, 14 | MAX_MSG_SIZE = 1<<22, 15 | ): 16 | 17 | comm = MPI.COMM_WORLD 18 | myid = comm.Get_rank() 19 | numprocs = comm.Get_size() 20 | 21 | if numprocs != 2: 22 | if myid == 0: 23 | errmsg = "This test requires exactly two processes" 24 | else: 25 | errmsg = None 26 | raise SystemExit(errmsg) 27 | 28 | s_buf = allocate(MAX_MSG_SIZE) 29 | r_buf = allocate(MAX_MSG_SIZE) 30 | 31 | if myid == 0: 32 | print ('# %s' % (BENCHMARH,)) 33 | if myid == 0: 34 | print ('# %-8s%20s' % ("Size [B]", "Bandwidth [MB/s]")) 35 | 36 | message_sizes = [2**i for i in range(30)] 37 | for size in message_sizes: 38 | if size > MAX_MSG_SIZE: 39 | break 40 | if size > large_message_size: 41 | skip = skip_large 42 | loop = loop_large 43 | window_size = window_size_large 44 | 45 | iterations = list(range(loop+skip)) 46 | window_sizes = list(range(window_size)) 47 | s_msg = [s_buf, size, MPI.BYTE] 48 | r_msg = [r_buf, size, MPI.BYTE] 49 | send_request = [MPI.REQUEST_NULL] * window_size 50 | recv_request = [MPI.REQUEST_NULL] * window_size 51 | # 52 | comm.Barrier() 53 | if myid == 0: 54 | for i in iterations: 55 | if i == skip: 56 | t_start = MPI.Wtime() 57 | for j in window_sizes: 58 | recv_request[j] = comm.Irecv(r_msg, 1, 10) 59 | for j in window_sizes: 60 | send_request[j] = comm.Isend(s_msg, 1, 100) 61 | MPI.Request.Waitall(send_request) 62 | MPI.Request.Waitall(recv_request) 63 | t_end = MPI.Wtime() 64 | elif myid == 1: 65 | for i in iterations: 66 | for j in window_sizes: 67 | recv_request[j] = comm.Irecv(r_msg, 0, 100) 68 | for j in window_sizes: 69 | send_request[j] = comm.Isend(s_msg, 0, 10) 70 | MPI.Request.Waitall(send_request) 71 | MPI.Request.Waitall(recv_request) 72 | # 73 | if myid == 0: 74 | MB = size / 1e6 * loop * window_size 75 | s = t_end - t_start 76 | print ('%-10d%20.2f' % (size, MB/s)) 77 | 78 | 79 | def allocate(n): 80 | try: 81 | import mmap 82 | return mmap.mmap(-1, n) 83 | except (ImportError, EnvironmentError): 84 | try: 85 | from numpy import zeros 86 | return zeros(n, 'B') 87 | except ImportError: 88 | from array import array 89 | return array('B', [0]) * n 90 | 91 | 92 | if __name__ == '__main__': 93 | osu_bw() 94 | -------------------------------------------------------------------------------- /mpi4py_benchmarks/osu_bw.py: -------------------------------------------------------------------------------- 1 | # http://mvapich.cse.ohio-state.edu/benchmarks/ 2 | 3 | from mpi4py import MPI 4 | 5 | def osu_bw( 6 | BENCHMARH = "MPI Bandwidth Test", 7 | skip = 10, 8 | loop = 100, 9 | window_size = 64, 10 | skip_large = 2, 11 | loop_large = 20, 12 | window_size_large = 64, 13 | large_message_size = 8192, 14 | MAX_MSG_SIZE = 1<<22, 15 | ): 16 | 17 | comm = MPI.COMM_WORLD 18 | myid = comm.Get_rank() 19 | numprocs = comm.Get_size() 20 | 21 | if numprocs != 2: 22 | if myid == 0: 23 | errmsg = "This test requires exactly two processes" 24 | else: 25 | errmsg = None 26 | raise SystemExit(errmsg) 27 | 28 | s_buf = allocate(MAX_MSG_SIZE) 29 | r_buf = allocate(MAX_MSG_SIZE) 30 | 31 | if myid == 0: 32 | print ('# %s' % (BENCHMARH,)) 33 | if myid == 0: 34 | print ('# %-8s%20s' % ("Size [B]", "Bandwidth [MB/s]")) 35 | 36 | message_sizes = [2**i for i in range(30)] 37 | for size in message_sizes: 38 | if size > MAX_MSG_SIZE: 39 | break 40 | if size > large_message_size: 41 | skip = skip_large 42 | loop = loop_large 43 | window_size = window_size_large 44 | 45 | iterations = list(range(loop+skip)) 46 | window_sizes = list(range(window_size)) 47 | requests = [MPI.REQUEST_NULL] * window_size 48 | # 49 | comm.Barrier() 50 | if myid == 0: 51 | s_msg = [s_buf, size, MPI.BYTE] 52 | r_msg = [r_buf, 4, MPI.BYTE] 53 | for i in iterations: 54 | if i == skip: 55 | t_start = MPI.Wtime() 56 | for j in window_sizes: 57 | requests[j] = comm.Isend(s_msg, 1, 100) 58 | MPI.Request.Waitall(requests) 59 | comm.Recv(r_msg, 1, 101) 60 | t_end = MPI.Wtime() 61 | elif myid == 1: 62 | s_msg = [s_buf, 4, MPI.BYTE] 63 | r_msg = [r_buf, size, MPI.BYTE] 64 | for i in iterations: 65 | for j in window_sizes: 66 | requests[j] = comm.Irecv(r_msg, 0, 100) 67 | MPI.Request.Waitall(requests) 68 | comm.Send(s_msg, 0, 101) 69 | # 70 | if myid == 0: 71 | MB = size / 1e6 * loop * window_size 72 | s = t_end - t_start 73 | print ('%-10d%20.2f' % (size, MB/s)) 74 | 75 | 76 | def allocate(n): 77 | try: 78 | import mmap 79 | return mmap.mmap(-1, n) 80 | except (ImportError, EnvironmentError): 81 | try: 82 | from numpy import zeros 83 | return zeros(n, 'B') 84 | except ImportError: 85 | from array import array 86 | return array('B', [0]) * n 87 | 88 | 89 | if __name__ == '__main__': 90 | osu_bw() 91 | -------------------------------------------------------------------------------- /mpi4py_benchmarks/osu_latency.py: -------------------------------------------------------------------------------- 1 | # http://mvapich.cse.ohio-state.edu/benchmarks/ 2 | 3 | from mpi4py import MPI 4 | 5 | def osu_latency( 6 | BENCHMARH = "MPI Latency Test", 7 | skip = 1000, 8 | loop = 10000, 9 | skip_large = 10, 10 | loop_large = 100, 11 | large_message_size = 8192, 12 | MAX_MSG_SIZE = 1<<22, 13 | ): 14 | 15 | comm = MPI.COMM_WORLD 16 | myid = comm.Get_rank() 17 | numprocs = comm.Get_size() 18 | 19 | if numprocs != 2: 20 | if myid == 0: 21 | errmsg = "This test requires exactly two processes" 22 | else: 23 | errmsg = None 24 | raise SystemExit(errmsg) 25 | 26 | s_buf = allocate(MAX_MSG_SIZE) 27 | r_buf = allocate(MAX_MSG_SIZE) 28 | 29 | if myid == 0: 30 | print ('# %s' % (BENCHMARH,)) 31 | if myid == 0: 32 | print ('# %-8s%20s' % ("Size [B]", "Latency [us]")) 33 | 34 | message_sizes = [0] + [2**i for i in range(30)] 35 | for size in message_sizes: 36 | if size > MAX_MSG_SIZE: 37 | break 38 | if size > large_message_size: 39 | skip = skip_large 40 | loop = loop_large 41 | iterations = list(range(loop+skip)) 42 | s_msg = [s_buf, size, MPI.BYTE] 43 | r_msg = [r_buf, size, MPI.BYTE] 44 | # 45 | comm.Barrier() 46 | if myid == 0: 47 | for i in iterations: 48 | if i == skip: 49 | t_start = MPI.Wtime() 50 | comm.Send(s_msg, 1, 1) 51 | comm.Recv(r_msg, 1, 1) 52 | t_end = MPI.Wtime() 53 | elif myid == 1: 54 | for i in iterations: 55 | comm.Recv(r_msg, 0, 1) 56 | comm.Send(s_msg, 0, 1) 57 | # 58 | if myid == 0: 59 | latency = (t_end - t_start) * 1e6 / (2 * loop) 60 | print ('%-10d%20.2f' % (size, latency)) 61 | 62 | 63 | def allocate(n): 64 | try: 65 | import mmap 66 | return mmap.mmap(-1, n) 67 | except (ImportError, EnvironmentError): 68 | try: 69 | from numpy import zeros 70 | return zeros(n, 'B') 71 | except ImportError: 72 | from array import array 73 | return array('B', [0]) * n 74 | 75 | 76 | if __name__ == '__main__': 77 | osu_latency() 78 | -------------------------------------------------------------------------------- /ssh/config: -------------------------------------------------------------------------------- 1 | StrictHostKeyChecking no 2 | -------------------------------------------------------------------------------- /ssh/id_rsa.mpi: -------------------------------------------------------------------------------- 1 | -----BEGIN RSA PRIVATE KEY----- 2 | MIIEogIBAAKCAQEA7PWLZmgdJ508dD15T6+xqGDvL9Ehzo9SgsnN6xJ+qpUvvOi4 3 | 1axW0AqR4MnPTg/uuvk+x4tUpuufOW4w22UTGjsdvmIVWa9ujLtcRiN3YPY+SU+Y 4 | O5FfqKg7r/hBn+/GMcSoffwSs7vVgmhBBnp/mJh2O1cOAFZEe98/47mbg3/kHBAk 5 | 36NOQktaU3l48B38EhBTnjWfcEGm1HcTRPFxXV5Wiko6ZhKFEuHcTVKng4ROtUqE 6 | mgHyI0aB7TAxg4na0ejItsYWEPWGeDOw6ms/4MwylxNosWzHFPW9p4zgLCLNr+b6 7 | bDDfYKjXZflAuTQtQhLmJUwD9uuYLAijpSE2fQIDAQABAoIBADgcgRET8Gt0CV/B 8 | OtvKz/f+VEVvcWD3gWNlJDTZIVOFllNWjIZUlA4ZoqenQkbK8Q4nfV1FOht4yjCQ 9 | TlN1oMtiWk297i5Zo4UBzPzy4w774I39oh/g8dT/WXr2/5s+7SDV38xNh6Q2A34o 10 | 79T35wUcfUrZ93/O7dKjb/6d8hx2FMha0wVKqY4lmG1lQE3bbx3kakec0PdvU5kO 11 | YHKlpqj3pMR7CpMa+4yL/iXFwWYmnK+uu+zw7JR7PwvH1CzrnvW438wjQ1QmYbSx 12 | mHHOE89X67Lsl5hn81qYWBhpwAlBwi1qscsE0cV9GcFyKqWFqZsj5coM9u3CRfvy 13 | lrWe1OUCgYEA+LBUFEd3Hxs4sFiYElJ8R9SAs1udaqPvAl01hTEijJLfYlMMVs/y 14 | rgNN7j22zjDak2f8QdyMJZX7EZdRmdYcHO0csYOwbYvalzcnwk+U3mxmdD3r4xSo 15 | DSvkJ70fogAqUlcVIg2re6fCmZVJQTvMQYTVEM8zQomJRt/Lb2esSfsCgYEA8+zv 16 | 44aToe8uqiDs4w8guRW7LCDkTw4z4IVo9JUibIaPjaAs5bZEBXSB43EEywXCR75H 17 | fML0rU1PVvKh1rqcvZdVzm+XMWVr3asPk0sapaiHaTcmyZvJRDxxqbLFp0zRP1T6 18 | cCtXNFdHWU4KiuKrUi6cDyOKchpfkSZa4seiT+cCgYB+n4FgBfdQPlMB70oW4irn 19 | g/q32CjxuGCk6oKqu5bkzo+xB6obtavSEFqouIGQwO056tNVUY+GP7Rjg5GH663K 20 | yKw4cl3tmS0Gm43B8TVSfw03mKO3rrfWZQe5eCFYIg9qd26KNT2gK435FzsCXQkm 21 | PxUhhu6JrW/ZR2/U3Iur6wKBgADrWLAb1ryagSuE+j+U1AO+kDkHWrTtkcZ72jxp 22 | v3p3O11GSEUJXdJDcSXhTCpTuDq6/dv7hB6PFwh126RKicKxKlKf2wsFndV1Cpb8 23 | hnovW2tLGOtTmfuW2rrQAKyzvmolsNfxYd/BoHQ2thV16z1hDZeFA8WQUeHjKh6G 24 | sBbrAoGATdtQlaUxx4izua6k02ihkxx/cRYwDl2N8UDvDBHokS7vJFMX8b8NpsGg 25 | zMElnqSpu/pe/0UG7N2MtPF6uyMcX8AZzzcsRkiMkDvWJzYt8Jpf+Eyd/uryF+Yv 26 | yrXaOEY83tm6x/fny5ZaZmk8lNth7bfWywuTMkZLX3fYpWtIeE4= 27 | -----END RSA PRIVATE KEY----- 28 | -------------------------------------------------------------------------------- /ssh/id_rsa.mpi.pub: -------------------------------------------------------------------------------- 1 | ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDs9YtmaB0nnTx0PXlPr7GoYO8v0SHOj1KCyc3rEn6qlS+86LjVrFbQCpHgyc9OD+66+T7Hi1Sm6585bjDbZRMaOx2+YhVZr26Mu1xGI3dg9j5JT5g7kV+oqDuv+EGf78YxxKh9/BKzu9WCaEEGen+YmHY7Vw4AVkR73z/juZuDf+QcECTfo05CS1pTeXjwHfwSEFOeNZ9wQabUdxNE8XFdXlaKSjpmEoUS4dxNUqeDhE61SoSaAfIjRoHtMDGDidrR6Mi2xhYQ9YZ4M7Dqaz/gzDKXE2ixbMcU9b2njOAsIs2v5vpsMN9gqNdl+UC5NC1CEuYlTAP265gsCKOlITZ9 oweidner@peahi 2 | --------------------------------------------------------------------------------