Researchers work toward systematic assessment of climate models

Researchers work toward systematic assessment of climate models
In assessing climate models, experts typically evaluate across a range of criteria to arrive at an overall evaluation of the model's fidelity. They use their knowledge of the physical system and scientific goals to assess the relative importance of different aspects of models in the presence of trade-offs. Burrows et al. (2018) show that climate scientists adjust the importance they assign to different aspects of a simulation depending on the science question the model will be used to address. Their research also shows that expert consensus on importance differs across model variables. Credit: Advances in Atmospheric Sciences

A research team based at the Pacific Northwest National Laboratory in Richland, Washington, has published the results of an international survey designed to assess the relative importance climate scientists assign to variables when analyzing a climate model's ability to simulate real-world climate. The results, which have serious implications for studies using the models, were published as a cover article in Advances in Atmospheric Sciences on June 22, 2018.

"Climate modelers spend a lot of effort on calibrating certain model parameters to find a model version that does a credible job of simulating the Earth's observed ," said Susannah Burrows, first author on the paper and a scientist at the Pacific Northwest National Laboratory who specializes in Earth systems analysis and modeling.

However, Burrows noted, there is little systematic study on how experts prioritize such variables as cloud cover or sea ice when judging the performance of climate models.

"Different people might come to slightly different assessments of how 'good' a particular model is, depending to large extent on which aspects they assign the most importance to," Burrows said.

One model, for example, may better simulate sea ice while another model excels in cloud simulation. Each scientist must strike a balance between their competing priorities and goals—a difficult thing to capture systematically in data analysis tools.

"In other words, there isn't a single, completely objective definition of what makes a 'good' climate model, and this fact is an obstacle to developing more systematic approaches and tools to assist in model evaluations and comparisons," Burrows said.

The researchers found, from a survey of 96 participants representing the climate modelling community, that experts took specific scientific objectives into consideration when rating variable importance. They found a high degree of consensus that certain variables are important in certain studies, such as rainfall and evaporation in the assessment of the Amazonian water cycle. That agreement falters on other variables, such as how important it is to accurately simulate surface winds when studying the water cycle in Asia.

Understanding these discrepancies and developing more systematic approaches to model assessment is important, according to Burrows, since each new version of a climate model must undergo significant evaluation, and calibration by multiple developers and users. The labor-intensive process can take more than a year.

The tuning, while designed to maintain a rigorous standard, requires experts to make trade-offs between competing priorities. A model may be calibrated at the expense of one scientific objective in order to achieve another.

Burrows is a member of an interdisciplinary research team at PNNL working to develop a more systematic solution to this assessment problem. The team includes Aritra Dasgupta, Lisa Bramer, and Sarah Reehl, experts in data science and visualization, and Yun Qian, Po-Lun Ma, and Phil Rasch, climate science experts.

To help climate modelers understand these trade-offs more clearly and efficiently, the visualization researchers are building interactive, intuitive visual interfaces that allow modelers to summarize and explore complex information about different aspects of model performance.

The data scientists are working to characterize climate assessment in greater detail, building on the findings from the initial survey. Eventually, the researchers aim to blend a combination of metrics with human expertise to assess how well-suited are for specific science objectives, as well as to predict how frequently experts will agree or disagree with that assessment.

"[We plan] to combine the best of both worlds, using computing to reduce manual effort and allowing scientists to more efficiently apply their human insight and judgment where it is most needed," Burrows said.


Explore further

A cheaper way to explore distant relations in climate models

More information: Susannah M. Burrows et al, Characterizing the Relative Importance Assigned to Physical Variables by Climate Scientists when Assessing Atmospheric Climate Model Fidelity, Advances in Atmospheric Sciences (2018). DOI: 10.1007/s00376-018-7300-x
Citation: Researchers work toward systematic assessment of climate models (2018, June 26) retrieved 21 February 2020 from https://phys.org/news/2018-06-systematic-climate.html
This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no part may be reproduced without the written permission. The content is provided for information purposes only.
13 shares

Feedback to editors

User comments