GPU Sharing Across Multiple Processes and MPS

Last updated: June 4, 2025

---

CUDA and OpenACC allow multiple processes to share a single GPU.
When using a resource group with multiple GPUs (sub-GPUs), specify the GPUs to use via the environment variable CUDA_VISIBLE_DEVICES.
When using a resource group with only a single GPU (sub-GPU), no special configuration is required.

When sharing a GPU across multiple processes, using Multi-Process Service (MPS) can potentially improve performance.
For more details, see NVIDIA's website.

At Genkai, the environment variables CUDA_MPS_PIPE_DIRECTORY and CUDA_MPS_LOG_DIRECTORY are set automatically, so there is no risk of interference with other jobs.

Furthermore, GPU Compute Mode can be modified via job script options (cannot be changed when using MIG).
Set it with PJM -x PJM_GPU_COMPUTE_MODE=N, where N can be 0, 2, or 3:

• 0 : DEFAULT mode. Multiple processes can use the same GPU simultaneously.
• 2 : PROHIBITED mode. Process assignment to the GPU is prohibited.
• 3 : EXCLUSIVE_PROCESS mode. Only one process can use the GPU, but multiple threads within that process can utilize it.

Below is an example job script that sets Compute Mode and MPS:

#!/bin/bash
#PJM -L rscgrp=b-batch
#PJM -L elapse=10:00
#PJM -L gpu=1
#PJM -x PJM_GPU_COMPUTE_MODE=0
#PJM -j
module load cuda
# Start MPS
nvidia-cuda-mps-control -d
# Limit GPU resources
export CUDA_MPS_ACTIVE_THREAD_PERCENTAGE=50
# Run programs and wait for completion
./a1.out &
./a2.out &
wait
# Stop MPS
echo quit | nvidia-cuda-mps-control