Efficient method for revealing key uncertainties proves viable in regional integrated assessment analysis

In a direct evaluation of the uncertainty characterization process being applied as part of the Platform for Regional Integrated Modeling and Analysis (PRIMA) framework, Pacific Northwest National Laboratory scientists examined building energy policy under multiple energy-efficiency settings and several distinct uncertainty scenarios using a sub-regional version of the Global Change Assessment Model known as GCAM-USA. Their goal was to develop a decision-focused (i.e., relevant to a particular stakeholder) sensitivity analysis that identified factors that most influence stakeholder decisions. In an effort to generate an option that falls between ad hoc one-variable-at-a-time sensitivity assessments (for large models) or Monte Carlo analysis of all potential uncertainties (for small models), PNNL scientists used a middle-course, fractional-factorial analysis method to test the building energy policies. Their work showed that the number of factors required for modeling a given GCAM outcome actually was small, greatly reducing the modeling and computational burden in the presence of uncertainty.

By using a fractional-factorial method, the researchers were able to determine that sensitivity patterns exist to assist decision makers in making impactful mitigation and adaptation decisions. Moreover, the method reduced the number of model runs needed to identify and compare key sensitivities that determine which decisions are important to investigate with formal Monte Carlo simulations. In most cases, it would be computationally impossible to fully analyze every potential source of uncertainty in a large, data-driven model such as GCAM-USA. This option provides a viable computational model for uncertainty characterization with significant time and cost benefits while still revealing important drivers for decisions and more detailed uncertainty analysis.

The fractional-factorial analysis method employs a specifically selected subset (or fraction) of decisions, decision variables, and uncertainties and applies it to an integrated assessment model (GCAM-USA) to identify and assess key sensitivities of the model output to socioeconomic and Earth system uncertainties involving a specific energy and climate policy question. In this case, they examined sub-national energy and climate policy related to the energy efficiency of buildings. Drawing from information culled from stakeholder groups concerned about the regional effects of building energy codes and equipment energy efficiency, they used three residential and commercial building energy efficiency scenarios: no change (fixed at 2005 levels), energy-efficiency improvements to reduce energy use per unit of floor space by 50% (by 2050), and aggressive energy-efficiency improvements to cut energy use per unit of floor space by 80% (by 2050). Then, they created plausible future scenarios for uncertainties relevant in almost any region: population, wealth, carbon policy, and climate change.

The full-factorial analysis of all possible combinations required 162 GCAM runs, so they used an incremental fractional-factorial strategy that initially reduced the overall number of GCAM model runs from 162 to 54. Based on their comparison of the full (162) and fractional-factorial (54) model outcomes, the statistically sensitive variables remained the same. It actually took only seven combinations out of the 54 to explain more than 98 percent of variability in energy use, carbon dioxide emissions, and cost. The specificity of the decisions, variables, and uncertainties in this experiment demonstrated how sensitivity analysis can allow analysts to determine the uncertainties that matter in a given decision context versus those that do not. This allows decision makers to concentrate on resolving the uncertainties that truly matter with confidence in the outcomes.

As this work succeeded in demonstrating a viable empirical sensitivity analysis, the next step will involve engaging this method to assist regional decision makers in making energy and climate policy decisions under uncertainty.