Researchers break million-core supercomputer barrier

Jan 28, 2013 by Andrew Myers
This is a floor view of the newly installed Sequoia supercomputer at the Lawrence Livermore National Laboratories. Credit: Photo: Courtesy of Lawrence Livermore National Laboratories

Stanford Engineering's Center for Turbulence Research (CTR) has set a new record in computational science by successfully using a supercomputer with more than one million computing cores to solve a complex fluid dynamics problem—the prediction of noise generated by a supersonic jet engine.

Joseph Nichols, a research associate in the center, worked on the newly installed Sequoia IBM /Q system at Lawrence Livermore National Laboratories (LLNL) funded by the Advanced Simulation and Computing (ASC) Program of the (NNSA). Sequoia once topped list of the world's most powerful supercomputers, boasting 1,572,864 compute cores (processors) and 1.6 petabytes of memory connected by a high-speed five-dimensional torus interconnect.

Because of Sequoia's impressive numbers of cores, Nichols was able to show for the first time that million-core fluid dynamics simulations are possible—and also to contribute to research aimed at designing quieter aircraft engines.

THE PHYSICS OF NOISE

The exhausts of high-performance aircraft at takeoff and landing are among the most powerful human-made sources of noise. For ground crews, even for those wearing the most advanced hearing protection available, this creates an acoustically hazardous environment. To the communities surrounding airports, such noise is a major annoyance and a drag on property values.

Understandably, engineers are keen to design new and better aircraft engines that are quieter than their predecessors. New nozzle shapes, for instance, can reduce jet noise at its source, resulting in quieter aircraft.

Predictive simulations—advanced computer models—aid in such designs. These complex simulations allow scientists to peer inside and measure processes occurring within the harsh exhaust environment that is otherwise inaccessible to experimental equipment. The data gleaned from these simulations are driving computation-based as researchers uncover the physics of noise.

This is an image from the jet noise simulation. A new design for an engine nozzle is shown in gray at left. Exhaust temperatures are in red/orange. The sound field is blue/cyan. Chevrons along the nozzle rim enhance turbulent mixing to reduce noise. Credit: Courtesy of the Center for Turbulence Research, Stanford University

MORE CORES, MORE CHALLENGES

"Computational fluid dynamics (CFD) simulations, like the one Nichols solved, are incredibly complex. Only recently, with the advent of massive supercomputers boasting hundreds of thousands of computing cores, have engineers been able to model jet engines and the noise they produce with accuracy and speed," said Parviz Moin, the Franklin M. and Caroline P. Johnson Professor in the School of Engineering and Director of CTR.

CFD simulations test all aspects of a supercomputer. The waves propagating throughout the simulation require a carefully orchestrated balance between computation, memory and communication. Supercomputers like Sequoia divvy up the complex math into smaller parts so they can be computed simultaneously. The more cores you have, the faster and more complex the calculations can be.

And yet, despite the additional computing horsepower, the difficulty of the calculations only becomes more challenging with more cores. At the one-million-core level, previously innocuous parts of the computer code can suddenly become bottlenecks.

IRONING OUT THE WRINKLES

Over the past few weeks, Stanford researchers and LLNL computing staff have been working closely to iron out these last few wrinkles. This week, they were glued to their terminals during the first "full-system scaling" to see whether initial runs would achieve stable run-time performance. They watched eagerly as the first CFD simulation passed through initialization then thrilled as the code performance continued to scale up to and beyond the all-important one-million-core threshold, and as the time-to-solution declined dramatically.

"These runs represent at least an order-of-magnitude increase in computational power over the largest simulations performed at the Center for Turbulence Research previously," said Nichols "The implications for predictive science are mind-boggling."

A HOMECOMING

The current simulations were a homecoming of sorts for Nichols. He was inspired to pursue a career in supercomputing as a high-school student when he attended a two-week summer program at Lawrence Livermore computing facility in 1994 sponsored by the Department of Energy. Back then he worked on the Cray Y-MP, one of the fastest supercomputers of its time.

"Sequoia is approximately 10 million times more powerful than that machine," Nichols noted.

The Stanford ties go deeper still. The computer code used in this study is named CharLES and was developed by former Stanford senior research associate, Frank Ham. This code utilizes unstructured meshes to simulate turbulent flow in the presence of complicated geometry.

In addition to jet noise simulations, Stanford researchers in the Predictive Science Academic Alliance Program (PSAAP), sponsored by the Department of Energy, are using the CharLES code to investigate advanced-concept scramjet propulsion systems used in hypersonic flight (with video)—flight at many times the speed of sound—and to simulate the turbulent flow over an entire airplane wing.

Explore further: How to secure the cloud

Related Stories

Predictive simulation successes on Dawn supercomputer

Sep 30, 2009

(PhysOrg.com) -- The 500-teraFLOPS Advanced Simulation and Computing program's Sequoia Initial Delivery System (Dawn), an IBM machine of the same lineage as BlueGene/L, has immediately proved itself useful ...

Titan is also a green powerhouse

Nov 14, 2012

Not only is Oak Ridge National Laboratory's Titan the world's most powerful supercomputer, it is also one of the most energy-efficient.

NRL researchers study ways to reduce jet aircraft noise

Feb 15, 2011

Advanced military jet aircraft have engines that provide the needed speed and maneuverability. However, with this greater power there is significant noise during takeoff and landing. The noise can impact the ...

Recommended for you

How to secure the cloud

19 hours ago

For many of us, the primary reason we use "the cloud" is for storage—whether it's storing email through services like Gmail and Yahoo!, photos on Flickr, or personal documents on Dropbox. Many organizations ...

Berkeley team explores sound for indoor localization

23 hours ago

The global positioning system, or GPS, has its limitations—namely, it cannot work indoors. Potential solutions for indoor positioning continue to fire up the imaginations of scientists. The latest news ...

Taking great ideas from the lab to the fab

Jul 31, 2014

A "valley of death" is well-known to entrepreneurs—the lull between government funding for research and industry support for prototypes and products. To confront this problem, in 2013 the National Science ...

User comments : 12

Adjust slider to filter visible comments by rank

Display comments: newest first

Whydening Gyre
1.7 / 5 (6) Jan 28, 2013
This methodology can be applied to ANY problem.
gwrede
4.3 / 5 (6) Jan 28, 2013
in 1994 ... he worked on the Cray Y-MP, one of the fastest supercomputers of its time. Sequoia is approximately 10 million times more powerful than that machine
Few of us can really fathom what the next 19 years' worth of CS will give us. I hope to see the day. And I hope no political turmoil or killer meteorite destroys it all.
El_Nose
5 / 5 (3) Jan 28, 2013
This isn't a CS contribution but an electrical engineering one. I am a CS grad and would love to take credit - but this is an engineering feat more than an algorithmic acheivement.
rkolter
5 / 5 (4) Jan 28, 2013
Sequoia once topped list of the world's most powerful supercomputers, boasting 1,572,864 compute cores (processors) and 1.6 petabytes of memory connected by a high-speed five-dimensional torus interconnect.


I like to think I'm fairly adept in CS. I have no idea how to fathom what a five-dimensional torus interconnect is. I'd love to see the schematics of this thing.
DrWayne
5 / 5 (2) Jan 28, 2013
Scientists are going to continue building machines with more and more cores. The problem that will inevitably arise is how to use these powerful machines to solve problems instead of just model them.
I think genetic algorithms are going to play a big role, since core count results in a larger population with a greater range of mutation. This often leads to the evolution of better results. How we solve problems might boil down to this:
1st: We notice a problem, gather data on the problem and then model the problem.
2nd: We then evolve a solution to the problem based on the model data and wait until a significant improvement is found.
Nature has been doing this for millions of years ... it works pretty good.
wiyosaya
not rated yet Jan 28, 2013
Sequoia once topped list of the world's most powerful supercomputers, boasting 1,572,864 compute cores (processors) and 1.6 petabytes of memory connected by a high-speed five-dimensional torus interconnect.


I like to think I'm fairly adept in CS. I have no idea how to fathom what a five-dimensional torus interconnect is. I'd love to see the schematics of this thing.

en.wikipedia.org/wiki/Torus_interconnect
Argiod
1 / 5 (2) Jan 29, 2013
I like to think I'm fairly adept in CS. I have no idea how to fathom what a five-dimensional torus interconnect is. I'd love to see the schematics of this thing.


Check this out:
en.wikipedia.org/wiki/Torus_interconnect
Sonhouse
not rated yet Jan 29, 2013
Anyone hear what the petaflop rating of this computer is?
djr
5 / 5 (1) Jan 29, 2013
I bet they need really really big loud speakers to run this simulation!!!!!!
chromodynamics
not rated yet Jan 29, 2013
Sonhouse, Sequoia is 20 petaflop, behind Titan at 27

http://www.top500...2012/11/
duckduckgo
not rated yet Jan 30, 2013
...
2nd: We then evolve a solution to the problem based on the model data and wait until a significant improvement is found.
Nature has been doing this for millions of years ... it works pretty good.


Will the best solution also take millions of years to be computed? Will the answer be 42?
Sonhouse
not rated yet Feb 02, 2013
Sonhouse, Sequoia is 20 petaflop, behind Titan at 27

http://www.top500...2012/11/


Thanks for that. I did a quick calculation that seemed to put it at the 10 petaflop area. The thing about that is, Titan has far less core CPU's but has better performance. I wonder if they will be able to crank Sequoia to better Titan. All things being equal, it seems right now Sequoia is running at about 1% of capacity. It might be just a matter of optimization. I guess time will tell.