Team sets records in simulating seismic wave propagation across the Earth

Nov 25, 2008
To learn more about the inner sanctum of the earth’s core, seismologists take advantage of one of nature’s most destructive forces: earthquakes. Somewhat like the way a CAT scan images the brain, seismologists track seismic wave patterns from earthquakes to model the structure of the earth’s core. One of the great challenges is to capture the propagation of high-frequency waves, with periods of 1 to 2 seconds, as they travel across the globe. To simulate this activity, seismologists employ a spectral-element application called SPECFEM3D_GLOBE that uses a fine mesh of hexahedral finite elements, pictured here, and high-performance computers. Credit: D. Komatitsch, Université de Pau; L. Carrington, San Diego Supercomputer Center at UC San Diego

A team led by researchers at the San Diego Supercomputer Center at UC San Diego has successfully completed record-setting, petascale-level simulations of the earth's inner structure, paving the way for seismologists to model seismic wave propagations at frequencies of just over one second – the same frequencies that occur in nature.

Results of these latest seismic wave simulations were announced at SC08, the international conference for high-performance computing, where the research by the SDSC-led team was a finalist for Gordon Bell prize, awarded annually for outstanding achievement in high-performance computing applications. The SDSC team includes researchers from the California Institute of Technology; Université de Pau and INRIA, France; and the Institut Universitaire de France, France.

The record runs were completed on the 'Jaguar' system at Oak Ridge National Laboratory (ORNL). The record run broke the two-second barrier by achieving a shortest period of 1.15 seconds and 161 teraflops, using 149,784 cores.

That is the shortest wave period ever obtained in seismic wave propagation, as well as the highest level of parallelism and the first sustained performance of seismic wave propagation exceeding 160 TFlops. (One teraflop equals one trillion calculations per second. To put that in perspective, it would take a person operating a hand-held calculator more than 30,000 years to complete one trillion calculations.)

The latest supercomputer simulations, led by SDSC researcher Laura Carrington, focused on overcoming a key challenge in global seismology: modeling the propagation of seismic waves with frequencies as small as one second. The team broke the previous record set in 2003, which achieved frequencies of 3.5 seconds.

"In breaking the two-second barrier, we were able to model wave propagation clear through the Earth to more accurately predict its structure," said Carrington, who is with SDSC's Performance Modeling and Characterization (PMaC) laboratory. "More significantly, by achieving a frequency just above one second, we were able to duplicate the same frequencies that occur in nature, providing an unprecedented level of resolution that will greatly enhance our understanding of the physics and chemistry of the Earth's interior."

Researchers used a spectral finite-element application called SPECFEM3D_GLOBE for the latest and largest simulations ever done in this area of research. Waves at frequencies of between one and two seconds, generated when major earthquakes with magnitudes measuring 6.5 or more occur, help reveal new information about the 3D structure of the Earth because they are the highest frequency signals that can propagate, or travel, all the way through the Earth, particularly near the core-mantle boundary (CMB), the inner core boundary (ICB), and the enigmatic inner core, which is comprised of solid iron.

There is no need to simulate wave periods of less than one second for seismographic comparisons, because such frequencies signals do not propagate across the entire globe, according to the researchers.

Moreover, the team's research is crucial in helping seismologists better understand the dramatic differences in the complex structure of the Earth's inner core, which appears to be anisotropic, or having unequal physical properties along different axes, with dramatic differences between its eastern and western hemispheres.

Earlier, the research team conducted simulations using a wide range of resources provided by the TeraGrid, the nation's largest open scientific discovery infrastructure linking compute resources among 11 partner sites across the U.S. Runs were conducted using approximately 32,000 cores on the 'Ranger' supercomputer at the Texas Advanced Computing Center (TACC) at The University of Texas in Austin. That run achieved a seismic period length of 1.84 seconds and at a sustained 28.7 TFlops.

Prior to that, the team successfully completed several simulations using TeraGrid supercomputer resources, including 29,000 cores on the 'Jaguar' system at ORNL's National Center for Computational Sciences, achieving 1.94 seconds at 35.7 TFlops. Runs were also conducted on the Cray XT4 'Franklin' system at the National Energy Research Scientific Computing Center (NERSC) at the Lawrence Berkeley National Laboratory in Berkeley, Calif.; (12,150 cores with a shortest seismic wave of 3 seconds at 24 TFlops), and on the Cray XT4 'Kraken' system at the University of Tennessee-Knoxville (17,496 cores at 2.52 seconds at 22.4T Flops. The TFlops number in these and subsequent runs were measures using PSiNSlight, a performance measurement and tracing tool developed by SDSC's PMaC Lab.

The researchers also made a number of radical structure and memory management changes to the SPECFEM3D_GLOBAL application to enable what researchers call "peta-scale-ability", or using processors numbering in the hundreds of thousands. Efforts focused on the application to overlapping communications using non blocking MPI calls, optimizations to reduce cache misses, optimization to reduce I/O, large restructurings to reducing memory consumption, and work to reduce memory fragmentation.

In addition to Carrington, SDSC researchers on the team include Allan Snavely, director of the PMaC Lab; and researchers Michael Laurenzano and Mustafa Tikir. Additional team members include Dimitri Komatitsch, David Michéa, and Nicolas Le Goff of the Université de Pau, CNRS and INRIA Magique-3D, Laboratoire de Modélisation et d'Imagerie en Géosciences, Pau, France; and Jeroen Tromp, formerly a professor of geophysics with the California Institute of Technology's Seismological Laboratory. Tromp recently joined Princeton University's geosciences and mathematics departments. Komatitsch is also affiliated with the Institut Universitaire de France, in Paris.

Komatitsch and Tromp developed SPECFEM3D_GLOBE over 10 years ago, in order to do 3D whole Earth models that account for geological variations in and under the Earth's crust. These variations have a dramatic effect on how earthquakes propagate. Komatitsch's work on the application was vital in the collaboration that accomplished this record breaking run.

About the run, Tromp and Komatitsch had this to say: "One of the long-term goals of seismology is to be able to routinely simulate 3D seismic wave propagation in the whole Earth following earthquakes at frequencies of about 1 Hz, the highest frequency signal that can be seen clear across the planet. Very large numerical simulations performed on the new Cray XT5 system at Oak Ridge will enable us to get increasingly closer to this lofty goal."

Source: University of California - San Diego

Explore further: Ocean bacteria get 'pumped up': Team discovers new factors impacting fate of sinking carbon

Related Stories

Can sound help us detect 'earthquakes' on Venus?

Apr 23, 2015

Detecting an "earthquake" on Venus would seem to be an impossible task. The planet's surface is a hostile zone of crushing pressure and scorching temperatures—about 874 degrees F, hot enough to melt lead—that ...

SDSC researchers win NVIDIA's 2015 Global Impact Award

Mar 17, 2015

Researchers at the San Diego Supercomputer Center (SDSC) at the University of California, San Diego, are the recipient of NVIDIA's 2015 Global Impact Award for their collaborative work in developing an accelerated ...

Seismic study aims to map Earth's interior in 3-D

Mar 16, 2015

When a 7.9-magnitude earthquake struck central China's Sichuan province in 2008, seismic waves rippled through the region, toppling apartment houses in the city of Chengdu and swaying office buildings 1,000 ...

Florentine basilica gets high-tech physical

Feb 26, 2015

Late last year, two University of California, San Diego students set out for Florence, Italy, to diagnose a patient that had no prior medical record, couldn't be poked or prodded in any way, and hadn't been ...

Recommended for you

Did Kathmandu shift? Questions and Answers

5 hours ago

The tremor which struck Nepal on Saturday, killing more than 3,500 people, may have caused a land area around the capital Kathmandu to budge by several metres, experts say.

More than 2,200 confirmed dead in Nepal earthquake

Apr 26, 2015

A powerful aftershock shook Nepal on Sunday, making buildings sway and sending panicked Kathmandu residents running into the streets a day after a massive earthquake left more than 2,200 people dead.

User comments : 0

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.