How to use Jupyter Notebook

Jupyter Notebook

Jupyter Notebook, which allows you to execute Python commands from a web browser, is installed.
Jupyter Notebook can be used on both login nodes and compute nodes, but the available compute core and memory limits depend on the situation where the jupyter command is executed, so please use it appropriately according to the resources you need.

(If your process requires a lot of compute cores or memory, please use compute nodes instead of login nodes.)

Basic usage of Jupyter Notebook

You can use it by loading the jupyter_notebook/7.2.1 module, launching the process with the jupyter command, and accessing it with a web browser.

[ku40000105@genkai0002 ~]$ module load jupyter_notebook/7.2.1
[ku40000105@genkai0002 ~]$ jupyter notebook --no-browser --ip=0.0.0.0
中略
    To access the server, open this file in a browser:
        file:///home/pj24001603/ku40000105/.local/share/jupyter/runtime/jpserver-801304-open.html
    Or copy and paste one of these URLs:
        http://genkai0002:8888/tree?token=927512b3ef9387a2ea5c70811e842be0ca3f861ef470eab0
        http://127.0.0.1:8888/tree?token=927512b3ef9387a2ea5c70811e842be0ca3f861ef470eab0

Lines beginning with http:// are the addresses you should access with your web browser.
However, these addresses are not addresses to which you can connect directly from the outside.
Please use SSH port forwarding or start a web browser on Genkai to access them.

Since firefox can be started by firefox command on the login node of Genkai, you can use Jupyter Notebook by entering the above address
(http://genkai0002:8888/tree?token=82132e56f7bef97717ff1ed7a3f50ac46873ccfb3faa2f2a in the above example) in the URL input field after starting.
If you use this method, please use a connection method such as MobaXterm that can use X-forwarding.

X-forwarding is easy to use if your connection software supports it, but it is often not very fast (slow to respond).
It is often more comfortable to use SSH port forwarding.

The following shows how to use Jupyter Notebook with SSH port forwarding.

When you make an SSH connection with the SSH command, specify the port to forward and the destination information with the -L option.
For example, for the above usage example, connect as follows.
ssh -i private_key -L source_port:genkai0002:8888 supercomputer_account@genkai.hpc.kyushu-u.ac.jp
In this state, start a web browser on your PC and open
http://127.0.0.1:source_port/tree?token=927512b3ef9387a2ea5c70811e842be0ca3f861ef470eab0
to use Jupyter Notebook. Jupyter Notebook can be used.

If you connect with MoboXterm, you can use the above SSH command from the Local Terminal if openssh is installed in the Packages (MobApt).
You can also use MobaSSHTunnel, which can be started from the Tools menu or the Tunneling icon. The configuration procedure is as follows
Push the New SSH tunnel button, configure the forwarding settings as shown in the image below, and select the play button in the Start/stop section to start port forwarding.
In this state, start a web browser on your PC and open
http://127.0.0.1:source_port/tree?token=927512b3ef9387a2ea5c70811e842be0ca3f861ef470eab0
to use Jupyter Notebook. Jupyter Notebook can be used.

port forwarding by MobaXterm

To exit Jupyter Notebook, do one of the following

select File-Shut Down on your web browser.
execute Ctrl+c on the terminal where you ran jupyter notebook and type y for Shut down this Jupyter server (y/[n])? and press Enter.

[I 2024-07-02 16:37:09.353 ServerApp] interrupted
[I 2024-07-02 16:37:09.354 ServerApp] Serving notebooks from local directory: /home/pj24001603/ku40000105
    0 active kernels
    Jupyter Server 2.14.1 is running at:
（snip）
Shut down this Jupyter server (y/[n])? y
[C 2024-07-02 16:37:11.385 ServerApp] Shutdown confirmed
[I 2024-07-02 16:37:11.386 ServerApp] Shutting down 5 extensions
[ku40000105@genkai0001 ~]$

Load the module on the login node and run the jupyter command to see the connection information.

[ku40000105@genkai0001 ~]$ module load jupyter_notebook/7.2.1
[ku40000105@genkai0001 ~]$ jupyter notebook --no-browser --ip=0.0.0.0
[I 2024-07-02 16:21:07.119 ServerApp] jupyter_lsp | extension was successfully linked.
[I 2024-07-02 16:21:07.123 ServerApp] jupyter_server_terminals | extension was successfully linked.
[I 2024-07-02 16:21:07.127 ServerApp] jupyterlab | extension was successfully linked.
[I 2024-07-02 16:21:07.131 ServerApp] notebook | extension was successfully linked.
[I 2024-07-02 16:21:07.741 ServerApp] notebook_shim | extension was successfully linked.
[I 2024-07-02 16:21:07.808 ServerApp] notebook_shim | extension was successfully loaded.
[I 2024-07-02 16:21:07.810 ServerApp] jupyter_lsp | extension was successfully loaded.
[I 2024-07-02 16:21:07.811 ServerApp] jupyter_server_terminals | extension was successfully loaded.
[I 2024-07-02 16:21:07.820 LabApp] JupyterLab extension loaded from /home/app/jupyter_notebook/7.2.1/lib/python3.11/site-packages/jupyterlab
[I 2024-07-02 16:21:07.820 LabApp] JupyterLab application directory is /home/app/jupyter_notebook/7.2.1/share/jupyter/lab
[I 2024-07-02 16:21:07.821 LabApp] Extension Manager is 'pypi'.
[I 2024-07-02 16:21:07.830 ServerApp] jupyterlab | extension was successfully loaded.
[I 2024-07-02 16:21:07.836 ServerApp] notebook | extension was successfully loaded.
[I 2024-07-02 16:21:07.836 ServerApp] Serving notebooks from local directory: /home/pj24001603/ku40000105
[I 2024-07-02 16:21:07.836 ServerApp] Jupyter Server 2.14.1 is running at:
[I 2024-07-02 16:21:07.836 ServerApp] http://genkai0001:8888/tree?token=2b9c7f0b6810fb6a8944e32f35afeaaecb3277490af9b4d9
[I 2024-07-02 16:21:07.836 ServerApp]     http://127.0.0.1:8888/tree?token=2b9c7f0b6810fb6a8944e32f35afeaaecb3277490af9b4d9
[I 2024-07-02 16:21:07.836 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 2024-07-02 16:21:07.842 ServerApp]

    To access the server, open this file in a browser:
        file:///home/pj24001603/ku40000105/.local/share/jupyter/runtime/jpserver-1694794-open.html
    Or copy and paste one of these URLs:
        http://genkai0001:8888/tree?token=2b9c7f0b6810fb6a8944e32f35afeaaecb3277490af9b4d9
        http://127.0.0.1:8888/tree?token=2b9c7f0b6810fb6a8944e32f35afeaaecb3277490af9b4d9
[I 2024-07-02 16:21:08.173 ServerApp] Skipped non-installed server(s): bash-language-server, dockerfile-language-server-nodejs, javascript-typescript-langserver, jedi-language-server, julia-language-server, pyright, python-language-server, python-lsp-server, r-languageserver, sql-language-server, texlab, typescript-language-server, unified-language-server, vscode-css-languageserver-bin, vscode-html-languageserver-bin, vscode-json-languageserver-bin, yaml-language-server

In this state, you can use Jupyter Notebook by connecting from a web browser with SSH port forwarding.

Example of SSH port forwarding:

$ ssh -i private_key -L 8888:genkai0001:8888 supercomputer_account@genkai.hpc.kyushu-u.ac.jp
[ku40000105@genkai0002 ~]$ (logged into Genkai)

Connect to http://127.0.0.1:8888/tree?token=2b9c7f0b6810fb6a8944e32f35afeaaecb3277490af9b4d9 with a web browser and you will see the Jupyter Notebook screen.

Example of use on compute node

The following is an example of connecting with an interactive job. The same connection can be made with a batch job.

Start an interactive job and execute the module load and jupyter commands to display the connection information.

[ku40000105@genkai0001 ~]$ pjsub --interact -L rscgrp=b-inter,elapse=10:00,gpu=1
[INFO] PJM 0000 pjsub Job 82900 submitted.
[INFO] PJM 0081 .connected.
[INFO] PJM 0082 pjsub Interactive job 82900 started.
[ku40000105@b0031 ~]$ module load jupyter_notebook/7.2.1
[ku40000105@b0031 ~]$ jupyter notebook --no-browser --ip=0.0.0.0
[I 2024-07-02 16:31:27.878 ServerApp] jupyter_lsp | extension was successfully linked.
[I 2024-07-02 16:31:27.882 ServerApp] jupyter_server_terminals | extension was successfully linked.
[I 2024-07-02 16:31:27.886 ServerApp] jupyterlab | extension was successfully linked.
[I 2024-07-02 16:31:27.890 ServerApp] notebook | extension was successfully linked.
[I 2024-07-02 16:31:28.516 ServerApp] notebook_shim | extension was successfully linked.
[I 2024-07-02 16:31:28.586 ServerApp] notebook_shim | extension was successfully loaded.
[I 2024-07-02 16:31:28.588 ServerApp] jupyter_lsp | extension was successfully loaded.
[I 2024-07-02 16:31:28.589 ServerApp] jupyter_server_terminals | extension was successfully loaded.
[I 2024-07-02 16:31:28.597 LabApp] JupyterLab extension loaded from /home/app/jupyter_notebook/7.2.1/lib/python3.11/site-packages/jupyterlab
[I 2024-07-02 16:31:28.597 LabApp] JupyterLab application directory is /home/app/jupyter_notebook/7.2.1/share/jupyter/lab
[I 2024-07-02 16:31:28.598 LabApp] Extension Manager is 'pypi'.
[I 2024-07-02 16:31:28.608 ServerApp] jupyterlab | extension was successfully loaded.
[I 2024-07-02 16:31:28.613 ServerApp] notebook | extension was successfully loaded.
[I 2024-07-02 16:31:28.614 ServerApp] Serving notebooks from local directory: /home/pj24001603/ku40000105
[I 2024-07-02 16:31:28.614 ServerApp] Jupyter Server 2.14.1 is running at:
[I 2024-07-02 16:31:28.614 ServerApp] http://b0031:8888/tree?token=6394023d0b3ebadd9a9bb565b0d7bd77dc28c385b0db63e1
[I 2024-07-02 16:31:28.614 ServerApp]     http://127.0.0.1:8888/tree?token=6394023d0b3ebadd9a9bb565b0d7bd77dc28c385b0db63e1
[I 2024-07-02 16:31:28.614 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 2024-07-02 16:31:28.619 ServerApp]

    To access the server, open this file in a browser:
        file:///home/pj24001603/ku40000105/.local/share/jupyter/runtime/jpserver-143-open.html
    Or copy and paste one of these URLs:
        http://b0031:8888/tree?token=6394023d0b3ebadd9a9bb565b0d7bd77dc28c385b0db63e1
        http://127.0.0.1:8888/tree?token=6394023d0b3ebadd9a9bb565b0d7bd77dc28c385b0db63e1
[I 2024-07-02 16:31:30.846 ServerApp] Skipped non-installed server(s): bash-language-server, dockerfile-language-server-nodejs, javascript-typescript-langserver, jedi-language-server, julia-language-server, pyright, python-language-server, python-lsp-server, r-languageserver, sql-language-server, texlab, typescript-language-server, unified-language-server, vscode-css-languageserver-bin, vscode-html-languageserver-bin, vscode-json-languageserver-bin, yaml-language-server