Storage system

• Storage Types and Usage
• File backup and file migration for system updates
• File transfer
• Usage status and limits of storage systems
• Sharing files among users
• Performance of storage

Type and usage of storage systems

On Genkai, the following storage systems are available.

Large Storage

Large Storage is used as the storage area for home directories and intragroup shared directories. This area can be accessed from all compute nodes and login nodes of Genkai by using the following path.

• Home directory

  /home/{Group name}/{User name}

• Intragroup shared directory
  

  /home/{Group name}/share

In addition, directories to be published externally with NextCloud will also be placed on this area.

Available capacity of the Large Storage is limited on a group basis. 1TB is provided as the available capacity for a group at the basic fee. The available capacity can be further increased by applying for additional capacity. Application for additional capacity should be submitted from the portal. Application Portal.

When creating a new file or appending to an existing file in the Large Storage, if the total file usage of all users in the group exceeds the contracted capacity of the group’s Large Storage at that time, the process will be terminated during the process.

Files in the Large Storage will be deleted at the end of the user account contract period. Files of users who have applied for continued used in the next fiscal year can be used across fiscal years.

Fast Storage

Fast Storage can be used for programs that require high-speed file access. This can be expected to improve performance, especially for applications that access files frequently. If you wish to use this area, please apply at the portal.
Application Portal.

If the application has been approved, files in this area can be accessed from all compute nodes and login nodes of Genkai by using the following path.

/fast/{Group name}

This directory can also be specified by the variable ${GENKAI_FAST_DIR}. This directory can be read and written by all users within the group.

When creating a new file or appending to an existing file in the Fast Storage, if the total file usage of all users in the group exceeds the contracted capacity of the group’s Fast Storage at that time, the process will be terminated during the process.

Files in the Fast Storage will be deleted at the end of the user account contract period. Files of users who have applied for continued used in the next fiscal year can be used across fiscal years.

Local SSD

Storage space on SSDs on each compute node in node group B and node group C of Genkai. These SSDs have two areas: a free access area that can read and write freely from a job, and a read-only cache area where the Genkai file system temporarily stores files.

Usage of the free access area

In node group B and node group C, a directory /ssd/${PJM_JOBID} is created as a free access area on the local SSD of each node used by the job when the job execution starts. Files can be freely created in this directory while the job is running. This directory can also be specified by the variable ${PJM_SSD_DIR}.

For example, if there are files that are frequently read and written to in a job, copying the files to a free access area in advance or creating the files in a free access area in a job can greatly improve the read/write speed compared to placing the files in a large storage area. Depending on the size and frequency of file accesses, it may also be faster than accessing the files placed on the Fast Storage.

No prior application is required to use the free access area. There is no additional fee for using the free access area.

The maximum capacity of the frea access area of each node group is as follows:

Notes for using the free access area

• Files in this free access area are automatically deleted at the end of the job. Necessary files should be copied to the home directory during the job.
• Directories and files in the free access area cannot be referenced by the login node. In addition, since the free access area is created independently on each node, it cannot be shared among nodes in the same job.
• An error will occur if you try to create a file in the free access area beyond the above capacity.
• When executing a job with less than one node, other jobs in the same node may also use the free access area. In such a case, an error will occur if the total size of files created in the free access area by all jobs in the node exceeds the above capacity.

Usage of the cache area

A portion of the SSD on each node in Node Group B and Node Group C is used by the Genkai file system as a read-only cache area. When each node reads a file placed under /home/cache/{Group name}/{User name} in mass storage, it temporarily copies the file to the cache area. This greatly speeds up reads from that file while it is in the cache area. This directory can also be specified by the variable ${GENKAI_CACHE_DIR}.

Notes for using the cache area

• When the cache area overflows, the files in the cache area that have not been referenced the longest are deleted.
• Files placed in directories other than the /home/cache/groupname/username directory will not be copied to the cache area. Files to be copied to the cache area must be placed in this directory before submitting a job, and must also be read from this directory in the job.
• Writing to a file does not count as copying to the cache area. If a write occurs to a file copied to the cache area, the file will be deleted from the cache area to maintain coherence.
• The capacity of files placed in the /home/cache/{Group name}/{User name} directory will be added to the capacity of the mass storage. That is, the total capacity of files placed in /home/{Group name}, /home/cache/{Group name}, and /home/nextcloud/{Group name} will be the total capacity used by that group in the Large Storage.

File backup and file migration for system updates

• Files are not backed up. Deleted files cannot be restored. Please back up important files yourself.
• Similar to the file migration from ITO to GENKAI, users are requested to migrate files from GENKAI to the next system by themselves.
• Normally, files on the storage cannot be browsed during the GENKAI maintenance period, and files and directories published via NextCloud are also inaccessible.

File transfer

Preparation

Before transferring files, refer to the following page and complete key pair generation and public key registration
Usage startup procedures

WinSCP (Windows PC)

File transfer between Windows PC and Genkai can be done by using WinSCP,a file transfer software available for free.

Obtaining and Using WinSCP

Download a file from the following site, and follow the instructions to install it. Also refer to this site for instructions on using WinSCP.

WinSCP

Connect to Genkai with WinSCP

Enter the following information in the “new connection” of WinSCP:

• Hostname: genkai.hpc.kyushu-u.ac.jp
• Port number: 22
• User Name: Genkai User ID

scp command (Mac, Linux)

File transfer between Mac or Linux and Genkai can be done by using scp command.

Usage of the scp command

Open a terminal and run the following command to tranfer files:

$ scp -i private key file path source path destination path 
Enter passphrase for key 'private key': ← Enter passphrase (set during key generation) (not displayed)

Example of scp command

Example 1

Upload the local.txt file on your local (Mac or Linux) to the /home/aaaaa/abc1234/work/ directory in Genkai. (If your Genkai user name is abc1234 and your local private key is in .ssh/genkai_rsa in your local home area)

$ scp -i ~/.ssh/genkai_rsa . /local.txt abc1234@genkai.hpc.kyushu-u.ac.jp:/home/aaaa/abc1234/work/

Example 2

Download /home/aaaaa/abc1234/work/remote.txt of Genkai to your local current directory. (If your Genkai username is abc1234 and your local private key is in .ssh/genkai_rsa in your local home area)

$ scp -i ~/.ssh/genkai_rsa abc1234@genkai.hpc.kyushu-u.ac.jp:/home/aaaa/abc1234/remote.txt . /

NextCloud (Web browser, etc.)

NextCloud is software that enables access to the directories and files in the storage area prepared for NextCloud in Genkai via web browsers or other tools.

Refer to the following page for the usege of NextCloud:

NextCloud Usage

Usage status and limits of storage systems

Both Large Storage and Fast Storage have limits on the total amount of files and the number of files that can be created. Usage status of the storage systems on Genkai can be checked by the following command:

$ show_quota

With the command, the following information will be displayed:

Example output of show_quota

$ show_quota 
Filesystem UsedSize[GB] LimitSize[GB] UsedFiles LimitFiles 
/home 3283 11264 32362 9900000 
/fast 140 2048 2 34000000

In addition to that, both Large Storage and Fast Storage have limits on the number of files that can be created in one directory. This limitation depends on the length of the filename, with a minimum of about 4000000 and a maximum of about 12000000. If you try to create a file beyond the limit, you will get the error “No space left on device”.

Sharing files among users

intragroup shared directory

Users within the same group can share files in the following directory:

/home/groupname/share

If you want to grant access to a specific file to users in other groups, or only to specific users in the same group, use the following setfacl command to control the permissions of each file.

setfacl command

The setfacl command sets access control information for certain users or groups on the specified files. For example, with this command, you can grant access to the file to even the users in other groups.

Basic Usage

The usage of the setfacl command is as follows:

$ setfacl options File or directory path ...

To grant access to a certain user, specify the following as the option: (To grant access to a certain group, change u:User_Name to g:Group_Name.):

 -m u:User_Name:Access_Permission 

As the access permission, r (read), w (write) and x (execute) can be specified. For example if rw are specified, setfacl grants read and write permission to the file. The execute permission means represents the permission to execute the file as a command if the target is a file, or the permission to go into the directory if the target is a directory.

If the target is a directory, by adding option -R, permissions of all files and directories under it can be set.

In addition to that, specifying -d option for a directory sets its default access permissions. After setting the default access permissions, the permissions of all files and directories created in the directory will be set to the default. Note that the default access permission and the access permission of the directory itself are set seperately.

Current access permission of a file or a directory can be checked by the following command:

$ getfacl Path_to_File_or_Directory

Major options of `setfacl`` command

Major options of setfacl command are as follows:

Examples

Example 1
Allow read and write access to the file test.txt for the user abcd1234

$ setfacl -m u:abcd1234:rw test.txt

Example 2
Allow read and execute (or go into) access to the directory work and all files and directories under it for the group aaaa.

$ setfacl -R -m g:aaaa:rx work

Example 3
Set read and write access permission as the default access permission of the directory work for the user abcd1234.

$ setfacl -d -m u:abcd1234:rw work

Note on granting access to other users

Please note the following when using the setfacl command to grant access to files and directories to other users in GENKAI.

• Before granting any access to other users, only the owner of the home directory (/home/Group_name/User_name) can enter the directory. In this situation, even if other users are granted read access to files in the home directory, they will not be able to see the files because they will not be able to enter the home directory. Threfore, if you want to allow other users to access files or directories in the home directory, you must first grant them execute permissions on the home directory using the setfacl command.
  For example, to give the user abcd1234 execute permissions on the home directory, run the following commend (${HOME} is a variable that holds the path of the user's home directory).

  $ setfacl -m u:abcd1234:x ${HOME}

• Permissions to a group shared directory (/home/Group_name/share) cannot be changed by a user. If you want to grant other group users to access files or directories in the group shared directory, apply for changing the permissions of the directory to the contact (Access/Contact)
• Allowing other users write access to a directory is not recommended. Allowing it will result in a mix of files with different owners in the same directory. So if the user who owns these files is deleted, for example because the contract term of the project the user belongs to expires, other users may no longer be able to reference or edit these files. For this reason, GENKAI recommends granting only read and execute permissions to directories.

Reference

• online-manual

  $ man setfacl

• RedHat Enterprise Linux 9 Documentation 11.2 Managing Access Control Lists

Performance of storage

Storage structure of Genkai

Both Large Storage and Fast Storage of Genkai are constructed by multiple disk units. If not specified the files are distributed and stored on multiple disk units on a file-by-file basis. Therefore, when multiple files are accessed simultaneously, multiple disk units are used for fast read/write operations.

Striping files

In Genkai, you can also choose “stripe” as the layout of a file. This layout divides a file into multiple parts and stores them separately on the disk units. When reading or writing a single large file, access performance may be improved by storing it in this layout.

Setting stripe layout

The following command sets the stripe layout to the specfied file or directory.

$ lfs setstripe options directory name or file name

If a directory is specified, the files created after that will be applied the stripe layout according to the options.

The major options of lfs setstripe are as follows:

Check the current setting of stripe layout

The following command displays the current stripe layout setting for the file or directory.

$ lfs getstripe directory or file name

The main contents displayed are as follows:

Reference

• online-manual

  $ man lfs-setstripe

• Configuring Lustre File Striping