File transfer from ITO to Genkai

Files stored in ITO will not be accessible after 11 Sep. Please download the necessary files to your PC or transfer them to Genkai yourself using the following information.

Preparation for the file transfer

Check the Source Server and Destination Server for file tranfer

Since may users will be transferring files simultaneously, multiple servers are used for both the source and destination of file transfers in order to prevent load concentration on any one server. They are associated with the users according to the location of their home directories as shown in the following table.

Location of the home directory means its location in ITO. You can check the location by running the following command on ITO.

$ pwd
 /home/usr6/a0123456g

This example shows /home/usr6/a0123456g as this user’s home directory. Therefore, this user should use users associated to /home/usr6 in the table.

Note that two addresses are written for both Source and Destination servers. You can either of them. If the transfer speed is too slow, depending on congestion conditions, changing the Source or Destination server address to the other one may improve the transfer rate.

Retister a public key

Register the same public key registered in Application Portal to ITO also. This allows you to omit user authentication during file transfer using a feature called “agent forwarding”, which will be introduce in the next section. Note that if you are using the same pair of private and public keys, you can skip to the next “Setting agent forwarding”.

To register a Genkai public key with ITO, first login to Genkai, select and copy the contents of the public key displayed by the following command with the mouse.

$ cat ~/.ssh/authorized_keys

Next, login to ITO, open ~/.ssh/authorized_keys with a text editor, and paste the copied contents at the end of the file. Or, if you are not famliar with the text editor in Linux, you can log in to the ITO Portal, go to “File Operations”, check “Show hidden files” at the bottom of the window, open .ssh/ authorized_keys in your home directory and paste the copied contents.

Setting up agent forwarding

When transferring files from ITO to Genkai, first log in to the Source Server identified above, and then transfer files to the Destination Server. Normally, user authentication is required when logging in to the Source Server and when transferring files to the Destination Server. However, if the same public key is registered on both the Source and Destination Servers, and the user is authenticated using the public key when logging in to the Source Server, the agent forwarding facility can be used to omit user authentication when transferring files to the Destination Server.

Since file transfers to the Destination are performed many times while checking the file to be transferred, we recommend using this agent forwarding facility to improve work efficiency.

Agent forwarding on Windows

We recommend using MobaXterm to login to the Source Server from Windows. This is because MobaXterm can save the agent forwarding information once configured and reuse it from the next time onward. TeraTerm and PuTTY can also use the agent forwarding by using the tool called Pageant, but the configuration is a little complicated. If you wish to use these terminal emulator, please find information about Pagent with TeraTerm or PuTTY on the Web.

In MobaXterm, the following procedure is used to register the private key to be used for authentication.

1. Start MobaXterm.
2. Check “Use internal SSH agent MobAgent” in “Settings” -> “SSH” -> “SSH agents”.
3. Click “+” on the right side, select the private key you want to use and click “OK”.
4. MobaXterm will restart, and enter the passphrase for the private key. From now on, you will be asked to enter your passphrase to start MobaXterm.

After that, register the information of the Source Server in MobaXterm as follows.

1. Click “Session” -> “SSH” and enter the IP address of the Source Server in “Remote Host”.
2. Check the box on the left of “Specify username” and enter the username in the box on the right. (If you are using MobaXterm v22.2 or later, click “Advanced SSH settings”, click “Expert SSH setting”, check “Allow agent forwarding”, and click OK. )
3. Click “OK” to log in to the Source Server. From now on, simply click on Sessions (star icon) in MobaXterm and click on the Source Server, will let you log in to the server with the agent forwarding feature enabled.

Agent forwarding on MacOS

On MacOS, before logging in to the Source Server, register the private key by the following command.

$ ssh-add -K Path_of_Private_Key

You can confirm the information of the registered key as follows:

$ ssh-add -l

Then, login to the Source Server with -A option as follows:

$ ssh -A ITO_User_Name@IP_Address_of_Source_Server

Agent forwarding on Linux

On Linux, before logging in to the Source Server, start ssh-agent by the following command.

$ eval $(ssh-agent)

Then, register the private key by the following command.

$ ssh-add Path_of_Private_Key

You can confirm the information of the registered key as follows:

$ ssh-add -L

After that, login to the Source Server with -A option as follows:

$ ssh -A ITO_User_Name@IP_Address_of_Source_Server

File transfer

rsynccommand

We recommend using the rsync command for file transfers from ITO to Genkai. If you stop the file transfer in the middle of the transfer, if you are using the rsync command, you can resume the file transfer from where you stopped by starting the file transfer again with the same command. On the other hand, the scp command can also transfer a file from ITO to Genkai, but if you stop the transfer, you will have to start it again from the beginning.

The basic usage of the rsync command is as follows.

$ rsync -avh -e ssh Path_of_Source_Directory Genkai_User_Name@IP_Address_of_Destination_Server:Path_of_Destination_Directory

Major options of rsync command are as follows:

An example is shown below. This example transfers the directory ./test to the home directory of the Sestination Server 172.31.1.1

[xxxxxx@ncm0154 ~]$ rsync -avh -e ssh ./test ku9999999@172.31.1.1:./

In this case, a directory test is created in the home directory of Genkai, and files and directories in the test directory of ITO are transferred into it.

On the other hand, if you append / at the end of the source directory as shown below, the test directory is not created in the Genkai side, and the files and directories in the test directory of ITO are transferred directly under the home directory.

[xxxxxx@ncm0154 ~]$ rsync -avh -e ssh ./test/ ku9999999@172.31.1.1:./

Please refer to the Web sites like following one for the usage of rsync command:

• How to Use Rsync Command in Linux: 16 Practical Examples

Points to note when transferring files

If the file size is large or the number of files is large, file transfers can take a long time. Also, if the source or destination directory is incorrectly specified, the files in the destination may be accidentally overwritten. The following are precautions and recommendations for file transfers.

• Try file transfers several times first with a directory that has a small number of files and total size to make sure that the files are transferred correctly and that the time required for the transfer is reasonable.
• Do not transfer unnecessary files to save time. However, be careful not to delete files on the ITO side inadvertently, as they cannot be restored if they are deleted.
• When transferring a directory containing a large number of files, it may be possible to save time by converting them into one file in zip or tar+gz format and transfer the converted file.
• If most of the files in the directory are text files such as source programs, you may be able to save time by adding the -z option to rsync.
• If you temporarily create a directory such as fromito in your home directory and specify that directory as the destination of rsync, it is less likely that a file transfer will accidentally overwrite a newly created file in Genkai.
• The total file usage in the /home directory of the ITO can be checked for each group by the following command. If you belong to more than one group in the ITO, check for each group. (For example, in the following example, the total usage of this group is 9200609276KB.)

  $ lfs quota -g Group_Name /home Disk quotas for group Group_Name: Filesystem kbytes quota limit grace files quota limit grace /home 9200690276 107374182400 107374182400 - 22958385 80000000 100000000

• The total capacity and the number of files in a directory can be checked with the following command. However, if the number of files is large, usage of these commands should be kept to a minimum, as it places a heavy load on the file system and affects the file transfer rate for all users.

  $ du -sk Path_to_Directory

  $ find Path_to_Directory -type f | wc -l