Joint Institute for Nuclear Research (JINR)
"Govorun" is the irst hyper-converged and 100% ‘hot water’ liquid cooled supercomputer in the world.
The project aims to accelerate complex theoretical and experimental science research in high energy physics, nuclear physics and physics of condensed environments at Joint Institute for Nuclear Research, including NICA mega-project.
Hyperconvergent architecture
Hyperconvergent systems combine computing and storage resources on each node. Each node is a part of both computing and distributed storage subsystems (compute/store workloads). This enables linear scalability of system resources with the increase of the number of nodes.
Unlike classic HPC approach where computing and storage resources are scaled separately, in hyperconvergent system you can increase both computing power and distributed storage capacity/speed as you increase the number of nodes.
Supercomputer software based on RSC BasIS supports configuration and management of such systems allocating computing and storage resources for each user workload.
Distributed On-Demand Storage
Hyper-converged approach resulted in unique high-speed storage system RSC On-Demand Storage for “Govorun” with leading characteristics - total parallel file system speed is up to 300 GB/s, which is a record not only for Russia / CIS, but also for Europe.
The system is ranked 22nd in the current edition of IO500 list (November 2020), the new industrial rating for HPC-class storage systems.
RSC Tornado servers as compute resource and distributed storage system
RSC Tornado servers TDN511, TDN511S and TDN551 AFS with direct liquid cooling are well balanced for various computing tasks and as a part of RSC Storage-on-Demand system supporting different file systems.
Distributed storage is a centrally managed system with multiple data layers (for “hot”, “very hot” and "warm" data).
Very hot data storage is based on four blade servers RSC Tornado TDN511S. Each server has 12 high-speed low latency Intel® Optane™ SSD DC P4801X 375GB M.2 Series with Intel® Memory Drive Technology (IMDT) providing 4,2 TB for very hot data on each server.
Hot data storage has static and dynamic storage. Static storage is based on 14 blade servers RSC Tornado TDN511S and parallel file system Lustre. Dynamic storage RSC Storage-on-Demand is based on 84 blade servers RSC Tornado TDN511. Parallel file system Lustre and distributed asynchronous object storage DAOS are supported. Low latency Intel® Optane™ SSD DC P4801X 375GB M.2 Series drives are used for fast access to Lustre file system metadata, for DAOS metadata Intel® Optane™ Persistent Memory is used. Intel® SSD DC P4511 (NVMe, M.2) drives are used to store user data.
Warm data storage is based on RSC Tornado TDN551 AFS server with up to 32 solid state drives of EDSFF E1.L form-factor - Intel® SSD DC P4326 with 15.36 TB capacity each. This allows to have 491.5 Terabytes of data storage in just 1U of rack space. As higher density solid state drives become available on the market, the capacity will increase to 1 Petabyte per 1U and more.
High-speed interconnect Intel Omni-Path
High-speed data transfer between JINR’s supercomputer nodes uses innovative Intel® Omni-Path Architecture communication technology providing up to 100 Gbps non-blocking switching rate. It is based on 48-port Intel® Omni-Path Edge Switch 100 Series with 100% liquid cooling to provide high efficiency of the cooling system in ‘hot water’ mode and the lowest TCO of the system. Intel® Omni-Path Architecture meets current requirements of resource-intensive user applications and provides sufficient throughput reserve for the future.
Energy efficient solution
Using RSC Liquid Cooling technology and cooling distribution unit RSC CDU-HM 400 in hot water’ mode enabled deployment of free cooling solution (24x7x365) using only dry coolers running at up to +50°C ambient temperature and complete elimination of freon circuit and chillers. As the result, average power efficiency factor (PUE) is less than 1.06. The system uses less than 6% of consumed power for cooling, which is an outstanding result for HPC industry.
