Python

Last Updated: December 6, 2024

Overview

Multiple versions of Python are installed in Genkai. You can use them without loading a module.


The main ones available are as follows.

version Python command/pip command
Python 3.6 python3/pip3
python3.6/pip3.6
Python 3.8 python3.8/pip3.8
Python 3.9 python3.9/pip3.9
Python 3.11 python3.11/pip3.11

In addition, Intel Python is also available. To use Intel Python, load the intel module (module load intel) and then load the intelpython module (module load intelpython).


Examples of Python commands available for login nodes:

[ku40000105@genkai0002 ~]$ python (Interpolate commands with the Tab key)
python                    python2-debug-config      python3.6m-config         python3.9
python2                   python3                   python3.6m-x86_64-config  python3.9-config
python2.7                 python3.11                python3.8                 python3.9-x86_64-config
python2.7-config          python3.11-config         python3.8-config          python3-config
python2.7-debug           python3.11-x86_64-config  python3.8d                python-debug
python2.7-debug-config    python3.6                 python3.8d-config         python-debug-config
python2-config            python3.6-config          python3.8d-x86_64-config
python2-debug             python3.6m                python3.8-x86_64-config
[ku40000105@genkai0002 ~]$ python

Adding packages and modules at the user level

You can add packages and modules to Python by using the pip command (pip3, etc.), but since Genkai is a shared environment, please add the --user option. This allows packages to be installed under the home directory of each user. (If you do not add the --user option, it will try to install to the system directory and give an error.)


Example of a command to install a package or module into a user directory:

[ku40000105@genkai0002 ~]$ pip insall package name --user

If you use the virtual environment described below, you do not need to add the --user option because it will run/update Python installed in the user directory instead of the system directory.


Using Virtual Environments

You may want to use a specific version of Python or switch between Python versions or Python module installations to match the software you are using. In such cases, consider using venv, pyenv, conda (miniconda), etc.

When executing programs or modules maintained by the virtual environment in a batch job, be sure to activate the virtual environment in the batch job script as well.


How to use venv

venv is a Python virtual environment. Using a virtual environment allows you to prepare packages and modules for each virtual environment, so it is convenient to prepare a separate virtual environment for each project or application and switch between them.

How to create a virtual environment (example using Python 3.6)

Create a virtual environment with the command python3.6 -mvenv virtual_environment_name. When the command is executed, the directory specified by the virtual environment name is created on the spot. When activating it, you will access the files in it, so please be careful about the directory relationship when creating and activating the virtual environment.

How to activate/terminate use of a virtual environment

The source virtual_environment_name/bin/activate command activates the virtual environment, and the deactivate command terminates its use.

Example of use

# Create virtual environment
# (in this example, the test1 environment is created in the venv subdirectory under the home directory)
[ku40000105@genkai0001 ~]$ python3.6 -mvenv venv/test1
# Enable virtual environment
[ku40000105@genkai0001 ~]$ source venv/test1/bin/activate
# The display "(test1)" is added to the beginning of the line because test1 is activated
(test1) [ku40000105@genkai0001 ~]$
# If you use commands such as python or pip in this state,
# the ones in the virtual environment will take precedence
(test1) [ku40000105@genkai0001 ~]$ which python
~/venv/test1/bin/python
(test1) [ku40000105@genkai0001 ~]$ which pip
~/venv/test1/bin/pip
(test1) [ku40000105@genkai0001 ~]$ python --version
Python 3.6.8
(test1) [ku40000105@genkai0001 ~]$ pip --version
pip 9.0.3 from /home/pj24001603/ku40000105/venv/test1/lib64/python3.6/site-packages (python 3.6)
# You do not need to add `--user` to the pip install command when virtual environment is enabled
# (it will be installed under the virtual environment directory)
(test1) [ku40000105@genkai0001 ~]$ pip install pandas
Collecting pandas
(snip)
Installing collected packages: pytz, numpy, six, python-dateutil, pandas
Successfully installed numpy-1.19.5 pandas-1.1.5 python-dateutil-2.9.0.post0 pytz-2024.1 six-1.16.0
You are using pip version 9.0.3, however version 24.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
# When you finish using the virtual environment,
# (test1) will disappear and commands such as python, pip, etc.
# will return to the state where the system ones take precedence
(test1) [ku40000105@genkai0001 ~]$ deactivate
[ku40000105@genkai0001 ~]$ which python
/usr/bin/python
[ku40000105@genkai0001 ~]$ which pip
/usr/bin/which: no pip in (snip)
[ku40000105@genkai0001 ~]$

How to use pyenv

pyenv allows you to coexist (switch between) multiple Python versions. Consider using pyenv when you want to use a Python version other than the one already installed.

Installation

To install pyenv, execute the following command. This will place the necessary files in the ~/.pyenv directory.

[ku40000105@genkai0002 ~]$ curl https://pyenv.run | bash

Add the following to the end of the .bashrc so that pyenv is invoked at login. (If you are using a shell other than BASH, please adapt accordingly.)

export PYENV_ROOT="$HOME/.pyenv"
[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

When you log in again (start the shell again) with the .bashrc modified, the pyenv command will be available. At this point, the Python command will switch to the Python included in pyenv.

[ku40000105@genkai0002 ~]$ which python
~/.pyenv/shims/python
[ku40000105@genkai0002 ~]$ python --version
Python 3.6.8
[ku40000105@genkai0002 ~]$

Get a list of Python versions available for installation

You can get a list of Python versions available for installation by running pyenv install --list. (Many versions will be displayed.)

[ku40000105@genkai0002 ~]$ pyenv install --list
Available versions:
  2.1.3
  2.2.3
(snip)
  stackless-3.5.4
  stackless-3.7.5
[ku40000105@genkai0002 ~]$

Installing/Uninstalling a specified version of Python

You can install a specific version of Python by running pyenv install target version. (For the target version, choose the desired one from those listed with pyenv install --list.)

[ku40000105@genkai0002 ~]$ pyenv install 3.12.0
Downloading Python-3.12.0.tar.xz...
-> https://www.python.org/ftp/python/3.12.0/Python-3.12.0.tar.xz
Installing Python-3.12.0...
Installed Python-3.12.0 to /home/pj24001603/ku40000105/.pyenv/versions/3.12.0
[ku40000105@genkai0002 ~]$

You can uninstall the installed Python by running pyenv uninstall target_version.

Get a list of installed Python versions

You can get a list of installed Python versions by running pyenv versions.

  • Immediately after installing pyenv
[ku40000105@genkai0002 ~]$ pyenv versions
* system (set by /home/pj24001603/ku40000105/.pyenv/version)
[ku40000105@genkai0002 ~]$
  • After installing 3.12.0
[ku40000105@genkai0002 ~]$ pyenv versions
* system (set by /home/pj24001603/ku40000105/.pyenv/version)
  3.12.0
[ku40000105@genkai0002 ~]$

The one with an asterisk ( * ) at the beginning is the Python you are currently using.

Switching which Python to use

You can switch which Python to use by running pyenv local environment_name or pyenv global environment_name. Use local if you want to use the specified Python only in a specific directory, and use global otherwise (not limited to a specific directory). Python information will be inherited even if you log in again. (The Python information is stored in ~/.pyenv/version for global and in the .python-version file for local.)

[ku40000105@genkai0001 ~]$ which python
~/.pyenv/shims/python
[ku40000105@genkai0001 ~]$ python --version
Python 3.6.8
[ku40000105@genkai0001 ~]$ pyenv versions
* system (set by /home/pj24001603/ku40000105/.pyenv/version)
  3.12.0
# pyenv global to switch Python
[ku40000105@genkai0001 ~]$ pyenv global 3.12.0
[ku40000105@genkai0001 ~]$ which python
~/.pyenv/shims/python
[ku40000105@genkai0001 ~]$ python --version
Python 3.12.0
[ku40000105@genkai0001 ~]$ pyenv versions
  system
* 3.12.0 (set by /home/pj24001603/ku40000105/.pyenv/version)
# Python path has not changed, but the version of Python running has changed to 3.12.0
# (pyenv versions will show * for the currently active Python)
[ku40000105@genkai0001 ~]$ logout
# Logged out of Genkai.

# Loged back in to Genkai.
$ ssh genkai
[ku40000105@genkai0001 ~]$ python --version
Python 3.12.0
[ku40000105@genkai0001 ~]$ pyenv versions
  system
* 3.12.0 (set by /home/pj24001603/ku40000105/.pyenv/version)
# Python was still 3.12.0 when logged back in
# To revert back, enable system
[ku40000105@genkai0001 ~]$ pyenv global system
[ku40000105@genkai0001 ~]$ python --version
Python 3.6.8
[ku40000105@genkai0001 ~]$ pyenv versions
* system (set by /home/pj24001603/ku40000105/.pyenv/version)
  3.12.0
[ku40000105@genkai0001 ~]$

Create and use a new Python environment.

You can create a new Python environment by running pyenv virtualenv installed_Python_version_number environment_name. You can use pyenv activate environment_name to switch to the new environment. You can exit using pyenv deactivate. You can use pyenv deactivate if you want to use the same Python version but with different package configurations. (Please Use pyenv and venv as you like.)

# Create a new environment based on 3.12.0
[ku40000105@genkai0002 ~]$ pyenv virtualenv 3.12.0 test_3.12.0
# Activate the created environment
[ku40000105@genkai0001 ~]$ pyenv activate test_3.12.0
# actiavted environment name is displayed at the beginning of the line
(test_3.12.0) [ku40000105@genkai0001 ~]$
# deactivate to end use
(test_3.12.0) [ku40000105@genkai0001 ~]$ pyenv deactivate
[ku40000105@genkai0001 ~]$

How to use conda (Miniconda)

conda is a tool (package management software, environment management system) for managing software applications that include Python. There are two versions of conda: Anaconda, which includes many software packages, and Miniconda, which is limited to the major ones. Here we will show you how to use Miniconda.

Installing Miniconda

Download and run the installation script. (The -b -u -p ~/miniconda3 option has been added so that interactive confirmations such as where to install are omitted.)

# Python included in conda is ready to use when conda is ready to use
(base) [ku40000377@genkai0002 ~]$ which python
~/miniconda3/bin/python
(base) [ku40000377@genkai0002 ~]$ python --version
Python 3.12.4
(base) [ku40000377@genkai0002 ~]$

# change the channel to solve the issue of Anaconda's commercial use
# ("-c conda-forge --override-channels" options are added to solve this issue in the following commands)
(base) [ku40000377@genkai0002 ~]$ conda config --add channels conda-forge
(base) [ku40000377@genkai0002 ~]$ conda config --remove channels defaults
(base) [ku40000377@genkai0002 ~]$ conda config --show channels
channels:
  - conda-forge
(base) [ku40000377@genkai0002 ~]$

# Update conda itself
# if you don't add -y, it will check if the contents are correct
(base) [ku40000377@genkai0002 ~]$ conda update -n base conda -y -c conda-forge --override-channels
Channels:
 - conda-forge
Platform: linux-64

(snip)

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
(base) [ku40000377@genkai0002 ~]$

# Create environment
(base) [ku40000377@genkai0002 ~]$ conda create -n test_3.12.0 python=3.12.0 pip -y -c conda-forge --override-channels
Channels:
 - conda-forge
Platform: linux-64

(snip)

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate test_3.12.0
#
# To deactivate an active environment, use
#
#     $ conda deactivate
(base) [ku40000377@genkai0002 ~]$

# Activate the created environment
(base) [ku40000377@genkai0002 ~]$ conda activate test_3.12.0
# The name of the activated environment will appear at the beginning of the line,
# and you will be able to use the Python and pip you have installed
(test_3.12.0) [ku40000377@genkai0002 ~]$ which python
~/miniconda3/envs/test_3.12.0/bin/python
(test_3.12.0) [ku40000377@genkai0002 ~]$ python --version
Python 3.12.0
# Check the installed modules
(test_3.12.0) [ku40000377@genkai0002 ~]$ pip list
Package    Version
---------- -------
pip        24.2
setuptools 72.1.0
wheel      0.44.0
# Install pandas module (--user option is not necessary since it is installed in the virtual environment)
(test_3.12.0) [ku40000377@genkai0002 ~]$ pip install pandas
Collecting pandas
  Downloading pandas-2.2.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (19 kB)
Collecting numpy>=1.26.0 (from pandas)
  Downloading numpy-2.1.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (60 kB)
Collecting python-dateutil>=2.8.2 (from pandas)
  Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl.metadata (8.4 kB)
Collecting pytz>=2020.1 (from pandas)
  Downloading pytz-2024.1-py2.py3-none-any.whl.metadata (22 kB)
Collecting tzdata>=2022.7 (from pandas)
  Downloading tzdata-2024.1-py2.py3-none-any.whl.metadata (1.4 kB)
Collecting six>=1.5 (from python-dateutil>=2.8.2->pandas)
  Downloading six-1.16.0-py2.py3-none-any.whl.metadata (1.8 kB)
Downloading pandas-2.2.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.7 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.7/12.7 MB 87.1 MB/s eta 0:00:00
Downloading numpy-2.1.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (16.0 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 16.0/16.0 MB 156.7 MB/s eta 0:00:00
Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB)
Downloading pytz-2024.1-py2.py3-none-any.whl (505 kB)
Downloading tzdata-2024.1-py2.py3-none-any.whl (345 kB)
Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: pytz, tzdata, six, numpy, python-dateutil, pandas
Successfully installed numpy-2.1.0 pandas-2.2.2 python-dateutil-2.9.0.post0 pytz-2024.1 six-1.16.0 tzdata-2024.1
# Checking the installed modules, there are more new modules such as pandas.
(test_3.12.0) [ku40000377@genkai0002 ~]$ pip list
Package         Version
--------------- -----------
numpy           2.1.0
pandas          2.2.2
pip             24.2
python-dateutil 2.9.0.post0
pytz            2024.1
setuptools      72.1.0
six             1.16.0
tzdata          2024.1
wheel           0.44.0
# Disable the enabled environment
(test_3.12.0) [ku40000377@genkai0002 ~]$ conda deactivate
# Disable the conda itself
(base) [ku40000377@genkai0002 ~]$ conda deactivate
[ku40000377@genkai0002 ~]$ which python
/usr/bin/python
[ku40000377@genkai0002 ~]$