MVAPICH
MVAPICH is a MPI (Message Passing Interface) library which is developed and distributed by Network-Based Computing Laboratory of Ohio State University.
Module
| Module name |
Version |
| mvapich |
3.0-gcc8.5.0 |
| 3.0-intel2023.2 |
Refer to the following page for the usage of modules:
Module usage
Usage
Set up (module commands)
To compile with GNU compiler
$ module load gcc
$ module load mvapich/3.0-gcc8.5.0
|
To compile with Intel compiler
$ module load intel
$ module load mvapich/3.0-intel2023.2
|
Compile
Examples of basic compilation are as follows.
Please add optimization options (such as
-O3), OpenMP enabling option (
-fopenmp), etc, as needed.
Example of compiling C program
Example of compiling Fortran program
$ mpifort test.f90 -o test
|
Job script
Example of a job script for a flat MPI program compiled by GNU compiler
(8 nodes, 120 processes per node)
#!/bin/bash
#PJM -L rscgrp=a-batch
#PJM -L node=8
#PJM -L elapse=00:20:00
#PJM -j
#PJM -S
module load gcc
module load mvapich/3.0-gcc8.5.0
mpiexec -np 960 -ppn 120 -machinefile ${PJM_O_NODEINF} -launcher ssh -launcher-exec /bin/pjrsh ./a.out
|
Example of a job script for a flat MPI program compiled by Intel compiler
(8 nodes, 120 processes per node)
#!/bin/bash
#PJM -L rscgrp=a-batch
#PJM -L node=8
#PJM -L elapse=00:20:00
#PJM -j
#PJM -S
module load intel
module load mvapich/3.0-intel2023.2
mpiexec -np 960 -ppn 120 -machinefile ${PJM_O_NODEINF} -launcher ssh -launcher-exec /bin/pjrsh ./a.out
|
Example of a job script for a hybrid MPI program compiled by GNU compiler
(8 nodes, 15 processes per node, 8 threads per process)
#!/bin/bash
#PJM -L rscgrp=a-batch
#PJM -L node=8
#PJM -L elapse=00:20:00
#PJM -j
#PJM -S
module load gcc
module load mvapich/3.0-gcc8.5.0
export OMP_NUM_THREADS=8
mpiexec -np 120 -ppn 15 -machinefile ${PJM_O_NODEINF} -launcher ssh -launcher-exec /bin/pjrsh ./a.out
|
Example of a job script for a hybrid MPI program compiled by Intel compiler
(8 nodes, 15 processes per node, 8 threads per process)
#!/bin/bash
#PJM -L rscgrp=a-batch
#PJM -L node=8
#PJM -L elapse=00:20:00
#PJM -j
#PJM -S
module load intel
module load mvapich/3.0-intel2023.2
export OMP_NUM_THREADS=8
mpiexec -np 120 -ppn 15 -machinefile ${PJM_O_NODEINF} -launcher ssh -launcher-exec /bin/pjrsh ./a.out
|
Note
Warning message at runtime
If you let mpiexec command to launch the same number of processes as the available number of cores,
the following warning will appear.
In this case, try reducing the number of processes per node by two or more to see if it improves the performance.
[][mvp_generate_implicit_cpu_mapping] WARNING: You appear to be running at full subscription for this j
ob. UCX spawns an additional thread for each process which may result in oversubscribed cores and poor
performance. Please consider reserving at least 2 cores per node for the additional threads, enabling S
MT, or setting MVP_THREADS_PER_PROCESS=2 to ensure that sufficient resources are available.
|
