Scientists and researchers at the U.S. Department of Energy’s National Renewable Energy Laboratory (NREL) are constantly innovating, integrating novel technologies, and “walking the talk.” Since 1982, NREL has won 52 R&D 100 Awards — known in the research and development community as “the Oscars of Innovation” — for its groundbreaking work.
When it came time for the lab to build its own high performance computing (HPC) data center, the NREL team knew it would have to be made up of firsts: The first HPC data center dedicated solely to advancing energy systems integration, renewable energy research, and energy efficiency technologies. The HPC data center ranked first in the world when it comes to energy efficiency. The first petascale HPC to use warm-water liquid cooling and reach an annualized average power usage effectiveness (PUE) rating of 1.06 or better.
To accomplish this, NREL worked closely with industry leaders to track rapid technology advances and to develop a holistic approach to data center sustainability in the lab’s new Energy Systems Integration Facility (ESIF).
“We took an integrated approach to the HPC system, the data center, and the building as part of the ESIF project,” NREL’s Computational Science Center Director Steve Hammond said. “First, we wanted an energy-efficient HPC system appropriate for our workload. This is being supplied by HP and Intel. A new component-level liquid cooling system, developed by HP, will be used to keep computer components within safe operating range, reducing the number of fans in the backs of the racks.”
Next, the NREL team, which included the design firms SmithGroupJJR and the Integral Group, created the most energy-efficient data center it could to house and provide power and cooling to the HPC system. High-voltage (480 VAC) electricity is supplied directly to the racks rather than the typical 208 V, which saves on power electronics equipment, power conversions, and losses. Energy-efficient pumps largely replace noisy, less-efficient fans.
“Last but not least, we wanted to capture and use the heat generated by the HPC system,” Hammond said. “Most data centers simply throw away the heat generated by the computers. An important part of the ESIF is that we will capture as much of the heat as possible that is generated by the HPC system in the data center and reuse that as the primary heat source for the ESIF office space and laboratories. These three things manifest themselves in an integrated ‘chips-to-bricks’ approach.”
Like NREL’s Research Support Facility, the ESIF HPC data center did not cost more to build than the average facility of its kind. It actually cost less to construct than comparable data centers and will be much cheaper to operate. NREL’s approach was to minimize the energy needed, supply it as efficiently as possible, and then capture and reuse the heat generated.
“Compared to a typical data center, we may save $800,000 of operating expenses per year,” Hammond said. “Because we are capturing and using waste heat, we may save another $200,000 that would otherwise be used to heat the building. So, we are looking at saving almost $1 million per year in operation costs for a data center that cost less to build than a typical data center.”
Warm-Water Cooling Boosts Data Center Efficiency
The ultra-efficient HPC system in NREL’s new data center has been designed in collaboration with HP and Intel. The HPC system will be deployed in two phases that will include scalable HP ProLiant SL230s and SL250s Generation 8 (Gen8) servers based on eight-core Intel Xeon E5-2670 processors as well as the next generation of servers using future 22nm Ivy Bridge architecture-based Intel Xeon processors and Intel Many Integrated Core architecture-based Intel Xeon Phi coprocessors. The first phase of the HPC installation began in November 2012, and the system will reach petascale capacity in the summer of 2013.
In the spirit of overall energy efficiency, the Intel Xeon Phi coprocessor delivers on several fronts. According to Intel, it can easily port complete applications in a short time, so software engineers won’t need specialized tools or new languages to support significant software packages. “Intel coprocessors also increase the efficiency of computer resource usage,” said Stephen Wheat, general manager of high performance computing at Intel. “The methods of code optimization for Xeon Phi are identical to what one does to make the most of Xeon processors. Finely tuned optimizations for Xeon Phi almost always result in a better-performing source code for Xeon processors. As the optimized and tuned application is run in production, the achieved performance per watt on both Xeon Phi and Xeon processors allows achieving the results with the lowest energy use.”
While some of the NREL HPC components may be off the shelf, the team is taking a different approach in cooling this supercomputer.
“In traditional computer systems, you have a mechanical chiller outside that delivers cold water into the data center, where air-conditioning units blow cold air under a raised floor to try to keep computer components from overheating,” Hammond said. “From a data center perspective, that’s not very efficient; it’s like putting your beverage on your kitchen table and then going outside to turn up the air conditioner to get your drink cold.”
“NREL’s ultimate HPC system is currently under development and will be a new, warm-water cooled high-performance system,” said Ed Turkel, group manager of HPC marketing at HP. “It will be a next-generation HPC solution that’s specifically designed for high power efficiency and extreme density, as well as high performance — things that NREL requires.”