Climate scientists compute in concert

February 28, 2012 By Eric Gedenk
The SEACISM project will improve the fidelity of ice-sheet models. The simulated (right) versus observed (left) flow velocities for Greenland ice sheets match well. Simulation data from Steve Price, Los Alamos National Laboratory; observational data from Jonathan Bamber and colleagues in Journal of Glaciology, 46, 2000.

Researchers at Oak Ridge National Laboratory (ORNL) are sharing computational resources and expertise to improve the detail and performance of a scientific application code that is the product of one of the world's largest collaborations of climate researchers. The Community Earth System Model (CESM) is a mega-model that couples components of atmosphere, land, ocean, and ice to reflect their complex interactions. By continuing to improve science representations and numerical methods in simulations, and exploiting modern computer architectures, researchers expect to further improve the CESM's accuracy in predicting climate changes. Achieving that goal requires teamwork and coordination rarely seen outside a symphony orchestra.

" is a complex system. We're not solving one problem, but a collection of problems coupled together," said ORNL computational Earth scientist Kate Evans. Of all the components contributing to climate, ice sheets such as those covering Greenland and Antarctica are particularly difficult to model—so much so that the Intergovernmental Panel on (IPCC) could not make any strong claim about the future of large ice sheets in its 2007 Assessment Report, the most recent to date.

Evans and her team began the Scalable, Efficient, and Accurate Community Ice Sheet Model (SEACISM) project in 2010 in an effort to fully incorporate a three-dimensional, thermomechanical ice sheet model called Glimmer-CISM into the greater CESM. The research is funded by the Department of Energy's (DOE's) Office of Advanced Scientific Computing Research (ASCR). Once fully integrated, Glimmer-CISM will be able to send information back and forth among other CESM codes, making it the first fully coupled ice sheet model in the CESM.

Through the ASCR Leadership Computing Challenge, the team of computational climate experts at multiple national labs and universities received allocations of processor hours on the Oak Ridge Leadership Computing Facility's (OLCF's) Jaguar supercomputer, which is capable of 2.3 petaflops, or 2.3 thousand trillion calculations per second.

Evans said the team is on track to have the code running massively parallel by October of 2012. Currently, simulations of a small test problem have employed 1,600 of Jaguar's 224,000 processors. Evans said the team expects that number to expand substantially in the near future when they begin simulating larger problems with greater realism.

The CESM began as the Community Climate Model in 1983 at the National Center for Atmospheric Research (NCAR) as a means to model the atmosphere computationally. In 1994, NCAR scientists pitched to the National Science Foundation (NSF) the idea of expanding their model to include realistic simulations of other components of the climate system. The result was the Climate System Model—adding land, ocean, and sea ice component models—which was renamed the Community Climate System Model (CCSM) to recognize the many contributors to the project. Development of the CCSM also benefitted from DOE and National Aeronautics and Space Administration (NASA) expertise and resources.

The CCSM developed into the CESM as its complexity increased. Today the model is a computational collection of the Earth's oceans, atmosphere, and land, as well as ice covering land and sea. Its components calculate in chorus. "The model is about getting a higher level of detail, improving our accuracy, and decreasing the uncertainty in our estimates of future changes," said Los Alamos National Laboratory climate scientist Phil Jones, who leads that laboratory's Climate, Ocean, and Sea Ice Modeling group. The group develops the firstprinciples ocean, sea ice, and ice sheet components of CESM. Its members are interested in sea-level rise, high-latitude climate, and changes in ocean thermohaline circulation—aqueous transport of heat and minerals around the globe.

The team is changing its ocean code, the Parallel Ocean Program, to become the Model for Prediction Across Scales-Ocean (MPAS-Ocean) code. Unlike the Parallel Ocean Program, MPAS-Ocean is a variable-resolution, unstructured grid model. It will allow researchers to sharpen resolution on a regional scale when they want to look at climate impacts in particular localities.

The CESM has continually grown in intricacy, enabling researchers to calculate more detail over larger spatial scales and longer time scales. Further, researchers are able to introduce more complex physics variables and simulate in greater detail Earth's biogeochemical components—chemical and ecosystem impacts on the climate. But these advances do not come without a price.

"At any given time when we're integrating the model forward, it's very [computationally] expensive—very time consuming and using large amounts of memory," Evans said. "It all has to work in concert to generate huge amounts of data that we then need to analyze afterward."

Conducting climate codes

Researchers and code developers for the CESM are scattered around the United States. One of their biggest challenges is tying together separate climate code components created in different places on different computer architectures. What's more, climate researchers usually focus on a specific aspect of the climate, such as ocean or atmosphere. An atmosphere scientist, for example, needs the ability to raise resolution in the atmosphere but may want to lower the resolution used in the ocean to minimize the computational cost of the simulation. ORNL computer scientist Patrick Worley helps researchers optimize their codes. That makes him one in a small group in the CESM community who conduct the climate code orchestra.

"There are many scientific issues with getting simulations right, and the computer scientists are involved with helping the scientists test and optimize them," said Worley. He serves as a co-chair in the CESM software engineering working group, which is dedicated to solving the unique challenges climate research imposes on computing resources.

A number of computational issues distinguish climate science from other scientific disciplines that make heavy use of simulations, Worley said. First, if researchers want to change their problem size, they must rework the simulations' new physical processes in correspondingly increased or decreased resolution. Second, many researchers focus only on particular areas of the Earth during their simulations, meaning they need only a particular part of the CESM framework running in high resolution. Finally, climate simulations run at varying time scales, sometimes spanning several thousand years. The required time to solution and available computing resources can force researchers to choose between high spatial resolution in their models or an extended observation period.

According to Evans, computer scientists like Worley help climate scientists deal with that complexity in the coupled model. "Pat does bridge across many components. He can look at an ice code, which solves very different equations and is structured in a very different way than an atmospheric model, and be able to help us run both systems not only individually at their maximum ability but coupled together," she said. "There are few in the climate community that have an understanding of how all of it works."

A hybrid horizon

With software improvements and increasingly more powerful supercomputers, resolution and realism in climate simulations have reached new heights. But there is still work to be done. "What we can't do yet in the current model is get down to regional spatial scales so we can tell people what specific impacts are going to happen locally," said Jones. "The current IPCC simulations are at coarse enough resolution that we can only give people general trends."

Climate research that concluded in 2010 makes up the final pieces of information that will go into the next IPCC Assessment Report, due for release in 2013. Meanwhile, are preparing for the future. "The climate research community doesn't have a single climate center where they run all of their climate simulations," Worley said. "They run wherever there is supercomputing time available. So it's important for the codes to run on as many different platforms as possible." Currently, US climate codes are running on National Oceanic and Atmospheric Administration, NASA, DOE, and NSF supercomputing resources.

One of the biggest challenges facing climate research, according to Worley, is writing the computational score for hybrid architectures. Next-generation supercomputers, such as the OLCF's Titan, a 10-20 petaflop machine, will use both central processing units and graphics processing units to share the computational workload. This novel approach will require closer attention to all levels of parallelism and will alter the approach to computing climate. "There are a number of people that want to make sure we are not surprised by the new machines," Worley said. If all goes well, CESM researchers may hear calls for an encore

Explore further: Scientists give boost to climate change predictions

Related Stories

Scientists give boost to climate change predictions

July 29, 2005

Researchers from MIT, NASA's Goddard Space Flight Center and several other government and academic institutions have created four new supercomputer simulations that for the first time combine mathematical computer models ...

Precision climate modeling forecast

September 7, 2006

Climate modeling of tomorrow will feature precision and scale only imagined just a few years ago, say researchers David Erickson and John Drake of Oak Ridge National Laboratory's Computer Science and Mathematics Division.

Researchers perform multi-century

April 1, 2008

Using state-of-the-art supercomputers, Lawrence Livermore National Laboratory climate scientists have performed a 400-year high-resolution global ocean-atmosphere simulation with results that are more similar to actual observations ...

Using supercomputers to explore ice sheet dynamics

February 28, 2011

Recently, Rhode Island-sized chunks of ice have separated from Greenland and Antarctica, garnering worldwide attention. But is this calving due to typical seasonal variations or a long-term warmer world? Climate scientists ...

Recommended for you

Auto, aerospace industries warm to 3D printing

August 25, 2016

New 3D printing technology unveiled this week sharply increases the size of objects that can be produced, offering new possibilities to remake manufacturing in the auto, aerospace and other major industries.


Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.