Genkai Hardware¶
Last Update: April 6, 2026
Table of Contents¶
- Overview
- Node Group A
- Node Group B
- Node Group C
- Login Nodes
- Shared Storage
- Network
- Differences between ITO and Genkai
Overview¶
Outline¶
GENKAI is a system consisting of multiple node groups with different characteristics, login nodes, two types of shared storage, and a group of peripheral devices connected by a high-speed network. Compared to the previous system (ITO), the number of nodes has been reduced to approximately half, but overall performance has been greatly improved.
Overview image¶
Overall performance¶
| Amount of nodes | 1,064 nodes |
|---|---|
| Amount of racks | 32 racks |
| Theoretical performance | CPU FP64 7.76 PFLOPS GPU FP64 5.63 PFLOPS GPU FP16,BF16 TC (Deep Learning) 166.22 PFLOPS |
| Total memory amount | Main memory 566,912 GiB Device memory 15,568 GiB |
| Interconnect | InfiniBand NDR 200/400 Gbps |
| Shared storage capacity | HDD 55.2 PB SSD 737.28 TB |
Node Group A¶
Outline¶
Node Group A consists of CPU-only compute nodes.
It offers high application performance with the latest-generation CPUs, and the large number of nodes (1024) is suitable for large-scale distributed-memory parallel computing.
Hardware configuration¶
| Model Number | FUJITSU Server PRIMERGY CX2550 M7 | |
|---|---|---|
| Compute node | CPU | Intel Xeon Platinum 8490H (Sapphire Rapids, 60 cores, 1.90–3.50 GHz) × 2 sockets |
| Memory | DDR5 4800 MHz, 512 GiB (32GiB×8 ×2 sockets) | |
| Theoretical performance | FP64 7,296 GFLOPS (3,648 GFLOPS×2 sockets) | |
| Memory bandwidth | 614.2 GB/s (4800MHz×8Byte×8 channels×2 sockets) | |
| Amount of nodes and cores | 1,024 nodes, 122,880 cores | |
| Total theoretical performance | FP64 7.47 PFLOPS (3,648 GF ×2 sockets ×1,024 nodes) | |
| Total memory | 512 TiB | |
| Total memory bandwidth | 629 TB/s | |
| User local storage | - | |
| Interconnect | InfiniBand NDR (200Gbps) × 1 port/node | |
| Cooling | Water | |
Compute Node Diagram¶
Driver Updates¶
- To be published
Node Group B¶
Outline¶
Node Group B consists of CPU+GPU compute nodes.
It is equipped with high-speed H100 GPUs and supports MIG, making it suitable for AI, data science, and numerical simulation workloads.
Hardware configuration¶
| Model Number | FUJITSU Server PRIMERGY GX2560 M7 | |
|---|---|---|
| Compute node | CPU | Intel Xeon Platinum 8490H (Sapphire Rapids, 60 cores, 1.90–3.50 GHz) ×2 sockets |
| GPU | NVIDIA H100 (Hopper) ×4 sockets | |
| Memory | DDR5 4800 MHz, 1024 GiB (64GiB×8 ×2 sockets), HBM2e 376 GiB (94GiB×4 sockets) | |
| Theoretical performance (CPU) | FP64 7,296 GFLOPS (3,648 GFLOPS×2 sockets) | |
| Theoretical performance (GPU) | FP64: 134.0 TFLOPS (33.5 TFLOPS×4 sockets) FP64 (TC): 267.6 TFLOPS FP16, BF16 (TC): 3,957.6 TFLOPS | |
| Memory bandwidth | CPU 614.4 GB/s GPU 9,584 GB/s | |
| GPU-GPU connection | NVLink ×18 (25 GB/s per lane, bi-directional 900 GB/s) | |
| CPU-GPU connection | PCIe Gen5 x16, 128 GB/s bi-directional | |
| Amount of nodes and cores | 38 nodes, 4,560 CPU cores + 1,284,096 FP64 GPU cores | |
| Total theoretical performance | CPU FP64: 277.25 TFLOPS GPU FP64: 5.09 PFLOPS GPU FP16,BF16 (TC): 150.39 PFLOPS | |
| Total memory | Host memory 38.91 TiB, Device memory 14.29 TiB | |
| Total memory bandwidth | Host memory 23.35 TB/s, Device memory 364.19 TB/s | |
| User local storage | NVMe SSD 12.8 TB/node | |
| Interconnect | InfiniBand NDR (400Gbps) ×2 ports/node | |
| Cooling | Water | |
Compute Node Diagram¶
Driver Updates¶
- 2026.04.03: GPU driver from 580.105.08 to 595.58.03
Node Group C¶
Outline¶
Node Group C consists of nodes with more GPUs and memory than Node Group B.
Although there are only 2 nodes, they may be effective for programs that are difficult to run or accelerate on other nodes.
Hardware configuration¶
| Model Number | Supermicro GPU SuperServer SYS-821GE-TNHR | |
|---|---|---|
| Compute node | CPU | Intel Xeon Platinum 8480+ (Sapphire Rapids, 56 cores, 2.00–3.80 GHz) ×2 sockets |
| GPU | NVIDIA H100 (Hopper) ×8 sockets | |
| Memory | DDR5 4400 MHz, 8 TiB (256GiB×16 ×2 sockets), HBM3 640 GiB (80GiB×8 sockets) | |
| Theoretical performance (CPU) | FP64 7,168 GFLOPS (3,584 GFLOPS×2 sockets) | |
| Theoretical performance (GPU) | FP64: 268 TFLOPS FP64 (TC): 535.2 TFLOPS FP16, BF16 (TC): 7,915.2 TFLOPS | |
| Memory bandwidth | CPU 563.2 GB/s GPU 26,816 GB/s | |
| GPU-GPU connection | NVLink ×18 (25 GB/s per lane, bi-directional 900 GB/s) | |
| CPU-GPU connection | PCIe Gen5 x16, 128 GB/s bi-directional | |
| Amount of nodes and cores | 2 nodes, 240 CPU cores + 135,168 FP64 GPU cores | |
| Total theoretical performance | CPU FP64: 14.34 TFLOPS GPU FP64: 536 TFLOPS GPU FP16, BF16 (TC): 15.83 PFLOPS | |
| Total memory | Host memory 16 TiB, Device memory 1,280 GiB | |
| Total memory bandwidth | Host memory 1.23 TB/s, Device memory 53.63 TB/s | |
| User local storage | NVMe SSD 15.3 TB/node | |
| Interconnect | InfiniBand NDR (400Gbps) ×4 ports/node | |
| Cooling | Water | |
Compute Node Diagram¶
Driver Updates¶
- 2026.04.03: GPU driver from 580.105.08 to 595.58.03
Login Nodes¶
Outline¶
Two login nodes are provided, each equipped with only CPUs.
SSH connections are assigned to one of the nodes via DNS round-robin.
Hardware configuration¶
| Model Number | FUJITSU Server PRIMERGY CX2530 M7 | |
|---|---|---|
| Compute node | CPU | Intel Xeon Platinum 8490H (Sapphire Rapids, 60 cores, 1.90–3.50 GHz) ×2 sockets |
| Memory | DDR5 4800 MHz, 1024 GiB (64GiB ×8 ×2 sockets) | |
| Amount of nodes and cores | 2 nodes, 240 cores | |
| Total memory | 2048 GiB | |
| User local storage | - | |
| Interconnect | InfiniBand NDR (200Gbps) ×1 port/node | |
| Cooling | Air | |
Shared Storage¶
Outline¶
GENKAI provides large-capacity storage (HDD) and fast storage (SSD).
RAID protects data, but no backup is provided. Important data should be backed up independently to prevent loss in case of accidents or natural disasters.
Hardware configuration¶
| MDS/MDT (Large storage) | DDN ES400NVX2 ×2 | |
|---|---|---|
| Per 1 set | MDS | 4 |
| MDT | 4 | |
| Drive | 1.92 TB NVMe SSD ×20 (+1 spare) | |
| RAID | RAID6 (8D +2P) | |
| Inodes | ≈11 billion | |
| MDS/MDT (Fast storage) | DDN ES400NVX2 ×1 | |
| Per 1 set | MDS | 4 |
| MDT | 8 | |
| Drive | 3.84 TB NVMe SSD ×20 (+1 spare) | |
| RAID | RAID6 (8D +2P) | |
| Inodes | ≈22.5 billion | |
| OSS/OST (Large storage) | DDN ES400NVX2 ×6 (shared) | |
|---|---|---|
| Per 1 set | OSS | 4 |
| OST | 32 | |
| HDD Capacity | 9.21 TB | |
| RAID | RAID6 (8D +2P) | |
| Write Performance | 60 GB/s | |
| Read Performance | 70 GB/s | |
| OSS/OST (Fast storage) | DDN ES400NVX2 ×6 (shared) | |
| Per 1 set | OSS | 4 |
| OST | 8 | |
| SSD Capacity | 122.88 TB | |
| RAID | RAID6 (8D +2P) | |
| Write Performance | 60 GB/s | |
| Read Performance | 80 GB/s | |
| Metadata backup server | 1 | |
|---|---|---|
| S3 access server | 2 | |
| NFS access server | 1 | |
| Monitoring server | 1 | |
Network¶
Outline¶
All GENKAI nodes and devices are connected via InfiniBand and Ethernet.
Compute nodes, shared storage, and login nodes are connected via a full-bisection bandwidth Fat Tree topology over InfiniBand.
Data communication between nodes and storage access is done over InfiniBand. External network access goes through Ethernet via login nodes.
Network Diagram¶
