Team sets records in simulating seismic wave propagation across the Earth

Nov 25, 2008
To learn more about the inner sanctum of the earth’s core, seismologists take advantage of one of nature’s most destructive forces: earthquakes. Somewhat like the way a CAT scan images the brain, seismologists track seismic wave patterns from earthquakes to model the structure of the earth’s core. One of the great challenges is to capture the propagation of high-frequency waves, with periods of 1 to 2 seconds, as they travel across the globe. To simulate this activity, seismologists employ a spectral-element application called SPECFEM3D_GLOBE that uses a fine mesh of hexahedral finite elements, pictured here, and high-performance computers. Credit: D. Komatitsch, Université de Pau; L. Carrington, San Diego Supercomputer Center at UC San Diego

A team led by researchers at the San Diego Supercomputer Center at UC San Diego has successfully completed record-setting, petascale-level simulations of the earth's inner structure, paving the way for seismologists to model seismic wave propagations at frequencies of just over one second – the same frequencies that occur in nature.

Results of these latest seismic wave simulations were announced at SC08, the international conference for high-performance computing, where the research by the SDSC-led team was a finalist for Gordon Bell prize, awarded annually for outstanding achievement in high-performance computing applications. The SDSC team includes researchers from the California Institute of Technology; Université de Pau and INRIA, France; and the Institut Universitaire de France, France.

The record runs were completed on the 'Jaguar' system at Oak Ridge National Laboratory (ORNL). The record run broke the two-second barrier by achieving a shortest period of 1.15 seconds and 161 teraflops, using 149,784 cores.

That is the shortest wave period ever obtained in seismic wave propagation, as well as the highest level of parallelism and the first sustained performance of seismic wave propagation exceeding 160 TFlops. (One teraflop equals one trillion calculations per second. To put that in perspective, it would take a person operating a hand-held calculator more than 30,000 years to complete one trillion calculations.)

The latest supercomputer simulations, led by SDSC researcher Laura Carrington, focused on overcoming a key challenge in global seismology: modeling the propagation of seismic waves with frequencies as small as one second. The team broke the previous record set in 2003, which achieved frequencies of 3.5 seconds.

"In breaking the two-second barrier, we were able to model wave propagation clear through the Earth to more accurately predict its structure," said Carrington, who is with SDSC's Performance Modeling and Characterization (PMaC) laboratory. "More significantly, by achieving a frequency just above one second, we were able to duplicate the same frequencies that occur in nature, providing an unprecedented level of resolution that will greatly enhance our understanding of the physics and chemistry of the Earth's interior."

Researchers used a spectral finite-element application called SPECFEM3D_GLOBE for the latest and largest simulations ever done in this area of research. Waves at frequencies of between one and two seconds, generated when major earthquakes with magnitudes measuring 6.5 or more occur, help reveal new information about the 3D structure of the Earth because they are the highest frequency signals that can propagate, or travel, all the way through the Earth, particularly near the core-mantle boundary (CMB), the inner core boundary (ICB), and the enigmatic inner core, which is comprised of solid iron.

There is no need to simulate wave periods of less than one second for seismographic comparisons, because such frequencies signals do not propagate across the entire globe, according to the researchers.

Moreover, the team's research is crucial in helping seismologists better understand the dramatic differences in the complex structure of the Earth's inner core, which appears to be anisotropic, or having unequal physical properties along different axes, with dramatic differences between its eastern and western hemispheres.

Earlier, the research team conducted simulations using a wide range of resources provided by the TeraGrid, the nation's largest open scientific discovery infrastructure linking compute resources among 11 partner sites across the U.S. Runs were conducted using approximately 32,000 cores on the 'Ranger' supercomputer at the Texas Advanced Computing Center (TACC) at The University of Texas in Austin. That run achieved a seismic period length of 1.84 seconds and at a sustained 28.7 TFlops.

Prior to that, the team successfully completed several simulations using TeraGrid supercomputer resources, including 29,000 cores on the 'Jaguar' system at ORNL's National Center for Computational Sciences, achieving 1.94 seconds at 35.7 TFlops. Runs were also conducted on the Cray XT4 'Franklin' system at the National Energy Research Scientific Computing Center (NERSC) at the Lawrence Berkeley National Laboratory in Berkeley, Calif.; (12,150 cores with a shortest seismic wave of 3 seconds at 24 TFlops), and on the Cray XT4 'Kraken' system at the University of Tennessee-Knoxville (17,496 cores at 2.52 seconds at 22.4T Flops. The TFlops number in these and subsequent runs were measures using PSiNSlight, a performance measurement and tracing tool developed by SDSC's PMaC Lab.

The researchers also made a number of radical structure and memory management changes to the SPECFEM3D_GLOBAL application to enable what researchers call "peta-scale-ability", or using processors numbering in the hundreds of thousands. Efforts focused on the application to overlapping communications using non blocking MPI calls, optimizations to reduce cache misses, optimization to reduce I/O, large restructurings to reducing memory consumption, and work to reduce memory fragmentation.

In addition to Carrington, SDSC researchers on the team include Allan Snavely, director of the PMaC Lab; and researchers Michael Laurenzano and Mustafa Tikir. Additional team members include Dimitri Komatitsch, David Michéa, and Nicolas Le Goff of the Université de Pau, CNRS and INRIA Magique-3D, Laboratoire de Modélisation et d'Imagerie en Géosciences, Pau, France; and Jeroen Tromp, formerly a professor of geophysics with the California Institute of Technology's Seismological Laboratory. Tromp recently joined Princeton University's geosciences and mathematics departments. Komatitsch is also affiliated with the Institut Universitaire de France, in Paris.

Komatitsch and Tromp developed SPECFEM3D_GLOBE over 10 years ago, in order to do 3D whole Earth models that account for geological variations in and under the Earth's crust. These variations have a dramatic effect on how earthquakes propagate. Komatitsch's work on the application was vital in the collaboration that accomplished this record breaking run.

About the run, Tromp and Komatitsch had this to say: "One of the long-term goals of seismology is to be able to routinely simulate 3D seismic wave propagation in the whole Earth following earthquakes at frequencies of about 1 Hz, the highest frequency signal that can be seen clear across the planet. Very large numerical simulations performed on the new Cray XT5 system at Oak Ridge will enable us to get increasingly closer to this lofty goal."


Source: University of California - San Diego

Explore further: Lava creeps toward road on Hawaii's Big Island

add to favorites email to friend print save as pdf

Related Stories

Scientific drilling project underway on Alpine Fault

Oct 06, 2014

The ambitious project near Whataroa, north of Franz Josef Glacier, is expected to take about two months to complete. It will enable scientists to install monitoring equipment deep inside the fault to record small earthquakes ...

Exoplanet measured with remarkable precision

Aug 19, 2014

Barely 30 years ago, the only planets astronomers had found were located right here in our own solar system. The Milky Way is chock-full of stars, millions of them similar to our own sun. Yet the tally ...

2010 Chilean earthquake causes icequakes in Antarctica

Aug 10, 2014

Seismic events aren't rare occurrences on Antarctica, where sections of the frozen desert can experience hundreds of micro-earthquakes an hour due to ice deformation. Some scientists call them icequakes. ...

Liners can protect pipelines during earthquakes

Aug 07, 2014

Cornell civil engineers report that retrofitting pipelines with "cured-in-place pipe" (CIPP) linings – flexible tubular membranes saturated with thermosetting resin – would prevent earthquake damage to ...

The most precise measurement of an alien world's size

Jul 24, 2014

Thanks to NASA's Kepler and Spitzer Space Telescopes, scientists have made the most precise measurement ever of the radius of a planet outside our solar system. The size of the exoplanet, dubbed Kepler-93b, ...

Recommended for you

Icelandic volcano sits on massive magma hot spot

Oct 24, 2014

Spectacular eruptions at Bárðarbunga volcano in central Iceland have been spewing lava continuously since Aug. 31. Massive amounts of erupting lava are connected to the destruction of supercontinents and ...

NASA sees Tropical Storm Ana still vigorous

Oct 24, 2014

NASA's TRMM satellite saw that Tropical Storm Ana was still generating moderate rainfall is it pulled away from Hawaii. The next day, NASA's Aqua satellite saw that wind shear was having an effect on the ...

User comments : 0