Weather forecasting is one of the most computationally challenging problems in science. Forecasts are produced by simulating a numerical model of the Earth's weather patterns and then predicting its future behaviours. For such simulations to correctly reflect actual weather conditions with high accuracy, it is important to update the model with the latest satellite data. This is an extremely challenging problem, where billions of model components that can affect the weather are involved. Fortunately, there are advanced mathematical structures which can be used to simplify this task.
Computational power available today can process several hundred simulations at one time. According to classical mathematical theory, this would not be sufficient to accurately simulate the complexities of the Earth model. Interestingly, weather forecasts made today can be quite accurate. This is made possible by using an algorithm known as the ensemble Kalman filter (EnKF), which performs well for this task. Such stark contrast between theory and reality is mystifying. Prof TONG Xin from the Department of Mathematics, NUS in collaboration with Prof Andrew MAJDA from New York University, are studying this phenomenon and have shown that the predictive power of EnKF may come from a systematic low effective dimensionality and localisation structure.
The ensemble Kalman filter (EnKF) (see Figure 1) is a popular computational method for weather forecasting. Existing theoretical studies of the EnKF consider mostly the low dimension-large sample size scenario, in which there is a large number of samples with few model components. Unfortunately, the situation is often reversed in real-life applications. This makes existing theory difficult to improve the performance of existing EnKF algorithms for practical scenarios. In order to resolve this issue, two model structures are investigated: a low effective dimension and a localisation covariance (see Figure 2). While these structures were identified heuristically before, a rigorous justification requires knowledge of advanced statistics, dynamical system and control theory.
This research lays the mathematical foundation for the use of EnKF in high dimensions for selected applications. It provides the fundamental conditions in which EnKF can operate effectively. Armed with these latest findings, new EnKF algorithms with improved accuracy can be developed for selected applications. For example, one important issue for EnKF application is the optimisation of an algorithmic parameter, known as the localisation radius. This problem was studied previously only by trial and error. The results by Prof Tong provide a new theoretical perspective for this problem, enabling users to develop guidelines for determining the localisation parameters.
Despite the fact that EnKF was derived in a linear setting, it works very well with nonlinear models in practice. The current research outcomes from this work are conducted only for linear models and would need to be expanded to cover nonlinear models so as to be able to understand and tap the full potential of EnKF.
Prof Tong said, "Linear and nonlinear problems are vastly different in theory. The former can be efficiently solved even in high dimensions, while the latter is the opposite. However, we see in practice that many nonlinear problems can be solved using linear methods. EnKF is one good example. The reasons behind await our further exploration."
Explore further: Real time forecast of Hurricane Sandy had track and intensity accuracy
More information: Xin T Tong et al. Nonlinear stability and ergodicity of ensemble based Kalman filters, Nonlinearity (2016). DOI: 10.1088/0951-7715/29/2/657
David Kelly et al. Concrete ensemble Kalman filters with rigorous catastrophic filter divergence, Proceedings of the National Academy of Sciences (2015). DOI: 10.1073/pnas.1511063112
Majda AJ; Tong XT, "Performance of Ensemble Kalman filters in large dimensions" COMMUNICATIONS ON PURE AND APPLIED MATHEMATICS Accepted: 2017.