Team sets records in simulating seismic wave propagation across the Earth

Nov 25, 2008
To learn more about the inner sanctum of the earth’s core, seismologists take advantage of one of nature’s most destructive forces: earthquakes. Somewhat like the way a CAT scan images the brain, seismologists track seismic wave patterns from earthquakes to model the structure of the earth’s core. One of the great challenges is to capture the propagation of high-frequency waves, with periods of 1 to 2 seconds, as they travel across the globe. To simulate this activity, seismologists employ a spectral-element application called SPECFEM3D_GLOBE that uses a fine mesh of hexahedral finite elements, pictured here, and high-performance computers. Credit: D. Komatitsch, Université de Pau; L. Carrington, San Diego Supercomputer Center at UC San Diego

A team led by researchers at the San Diego Supercomputer Center at UC San Diego has successfully completed record-setting, petascale-level simulations of the earth's inner structure, paving the way for seismologists to model seismic wave propagations at frequencies of just over one second – the same frequencies that occur in nature.

Results of these latest seismic wave simulations were announced at SC08, the international conference for high-performance computing, where the research by the SDSC-led team was a finalist for Gordon Bell prize, awarded annually for outstanding achievement in high-performance computing applications. The SDSC team includes researchers from the California Institute of Technology; Université de Pau and INRIA, France; and the Institut Universitaire de France, France.

The record runs were completed on the 'Jaguar' system at Oak Ridge National Laboratory (ORNL). The record run broke the two-second barrier by achieving a shortest period of 1.15 seconds and 161 teraflops, using 149,784 cores.

That is the shortest wave period ever obtained in seismic wave propagation, as well as the highest level of parallelism and the first sustained performance of seismic wave propagation exceeding 160 TFlops. (One teraflop equals one trillion calculations per second. To put that in perspective, it would take a person operating a hand-held calculator more than 30,000 years to complete one trillion calculations.)

The latest supercomputer simulations, led by SDSC researcher Laura Carrington, focused on overcoming a key challenge in global seismology: modeling the propagation of seismic waves with frequencies as small as one second. The team broke the previous record set in 2003, which achieved frequencies of 3.5 seconds.

"In breaking the two-second barrier, we were able to model wave propagation clear through the Earth to more accurately predict its structure," said Carrington, who is with SDSC's Performance Modeling and Characterization (PMaC) laboratory. "More significantly, by achieving a frequency just above one second, we were able to duplicate the same frequencies that occur in nature, providing an unprecedented level of resolution that will greatly enhance our understanding of the physics and chemistry of the Earth's interior."

Researchers used a spectral finite-element application called SPECFEM3D_GLOBE for the latest and largest simulations ever done in this area of research. Waves at frequencies of between one and two seconds, generated when major earthquakes with magnitudes measuring 6.5 or more occur, help reveal new information about the 3D structure of the Earth because they are the highest frequency signals that can propagate, or travel, all the way through the Earth, particularly near the core-mantle boundary (CMB), the inner core boundary (ICB), and the enigmatic inner core, which is comprised of solid iron.

There is no need to simulate wave periods of less than one second for seismographic comparisons, because such frequencies signals do not propagate across the entire globe, according to the researchers.

Moreover, the team's research is crucial in helping seismologists better understand the dramatic differences in the complex structure of the Earth's inner core, which appears to be anisotropic, or having unequal physical properties along different axes, with dramatic differences between its eastern and western hemispheres.

Earlier, the research team conducted simulations using a wide range of resources provided by the TeraGrid, the nation's largest open scientific discovery infrastructure linking compute resources among 11 partner sites across the U.S. Runs were conducted using approximately 32,000 cores on the 'Ranger' supercomputer at the Texas Advanced Computing Center (TACC) at The University of Texas in Austin. That run achieved a seismic period length of 1.84 seconds and at a sustained 28.7 TFlops.

Prior to that, the team successfully completed several simulations using TeraGrid supercomputer resources, including 29,000 cores on the 'Jaguar' system at ORNL's National Center for Computational Sciences, achieving 1.94 seconds at 35.7 TFlops. Runs were also conducted on the Cray XT4 'Franklin' system at the National Energy Research Scientific Computing Center (NERSC) at the Lawrence Berkeley National Laboratory in Berkeley, Calif.; (12,150 cores with a shortest seismic wave of 3 seconds at 24 TFlops), and on the Cray XT4 'Kraken' system at the University of Tennessee-Knoxville (17,496 cores at 2.52 seconds at 22.4T Flops. The TFlops number in these and subsequent runs were measures using PSiNSlight, a performance measurement and tracing tool developed by SDSC's PMaC Lab.

The researchers also made a number of radical structure and memory management changes to the SPECFEM3D_GLOBAL application to enable what researchers call "peta-scale-ability", or using processors numbering in the hundreds of thousands. Efforts focused on the application to overlapping communications using non blocking MPI calls, optimizations to reduce cache misses, optimization to reduce I/O, large restructurings to reducing memory consumption, and work to reduce memory fragmentation.

In addition to Carrington, SDSC researchers on the team include Allan Snavely, director of the PMaC Lab; and researchers Michael Laurenzano and Mustafa Tikir. Additional team members include Dimitri Komatitsch, David Michéa, and Nicolas Le Goff of the Université de Pau, CNRS and INRIA Magique-3D, Laboratoire de Modélisation et d'Imagerie en Géosciences, Pau, France; and Jeroen Tromp, formerly a professor of geophysics with the California Institute of Technology's Seismological Laboratory. Tromp recently joined Princeton University's geosciences and mathematics departments. Komatitsch is also affiliated with the Institut Universitaire de France, in Paris.

Komatitsch and Tromp developed SPECFEM3D_GLOBE over 10 years ago, in order to do 3D whole Earth models that account for geological variations in and under the Earth's crust. These variations have a dramatic effect on how earthquakes propagate. Komatitsch's work on the application was vital in the collaboration that accomplished this record breaking run.

About the run, Tromp and Komatitsch had this to say: "One of the long-term goals of seismology is to be able to routinely simulate 3D seismic wave propagation in the whole Earth following earthquakes at frequencies of about 1 Hz, the highest frequency signal that can be seen clear across the planet. Very large numerical simulations performed on the new Cray XT5 system at Oak Ridge will enable us to get increasingly closer to this lofty goal."

Source: University of California - San Diego

Explore further: Deciphering clues to prehistoric climate changes locked in cave deposits

Related Stories

Dedication of Advanced LIGO

May 19, 2015

The Advanced LIGO Project, a major upgrade that will increase the sensitivity of the Laser Interferometer Gravitational-wave Observatories instruments by a factor of 10 and provide a 1,000-fold increase in the number of astrophysical ...

Earthquakes reveal deep secrets beneath East Asia

May 14, 2015

A new work based on 3-D supercomputer simulations of earthquake data has found hidden rock structures deep under East Asia. Researchers from China, Canada, and the U.S. worked together to publish their results ...

Nepal quake could have been much worse: Here's why

May 01, 2015

The structural engineer strides through Kathmandu's old city, past buildings reduced to rubble, buildings whose facades are cracked in dozens of places, like the fractured shell of a hardboiled egg. But it's ...

Can sound help us detect 'earthquakes' on Venus?

Apr 23, 2015

Detecting an "earthquake" on Venus would seem to be an impossible task. The planet's surface is a hostile zone of crushing pressure and scorching temperatures—about 874 degrees F, hot enough to melt lead—that ...

Source of Earth's ringing? French team views ocean waves

Apr 20, 2015

Three researchers in France have authored "How ocean waves rock the Earth: Two mechanisms explain microseisms with periods 3 to 300 s," published in Geophysical Research Letters, a journal of the Americ ...

Recommended for you

Image: Cambodian rivers from orbit

May 22, 2015

A flooded landscape in Cambodia between the Mekong River (right) and Tonlé Sap river (left) is pictured by Japan's ALOS satellite. The centre of this image is about 30 km north of the centre of the country's ...

User comments : 0

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.