Blended particle methods with adaptive subspaces for filtering turbulent dynamical systems

Research output: Contribution to journalArticle

Abstract

It is a major challenge throughout science and engineering to improve uncertain model predictions by utilizing noisy data sets from nature. Hybrid methods combining the advantages of traditional particle filters and the Kalman filter offer a promising direction for filtering or data assimilation in high dimensional turbulent dynamical systems. In this paper, blended particle filtering methods that exploit the physical structure of turbulent dynamical systems are developed. Non-Gaussian features of the dynamical system are captured adaptively in an evolving-in-time low dimensional subspace through particle methods, while at the same time statistics in the remaining portion of the phase space are amended by conditional Gaussian mixtures interacting with the particles. The importance of both using the adaptively evolving subspace and introducing conditional Gaussian statistics in the orthogonal part is illustrated here by simple examples. For practical implementation of the algorithms, finding the most probable distributions that characterize the statistics in the phase space as well as effective resampling strategies is discussed to handle realizability and stability issues. To test the performance of the blended algorithms, the forty dimensional Lorenz 96 system is utilized with a five dimensional subspace to run particles. The filters are tested extensively in various turbulent regimes with distinct statistics and with changing observation time frequency and both dense and sparse spatial observations. In real applications perfect dynamical models are always inaccessible considering the complexities in both modeling and computation of high dimensional turbulent system. The effects of model errors from imperfect modeling of the systems are also checked for these methods. The blended methods show uniformly high skill in both capturing non-Gaussian statistics and achieving accurate filtering results in various dynamical regimes with and without model errors.

Original languageEnglish (US)
Pages (from-to)21-41
Number of pages21
JournalPhysica D: Nonlinear Phenomena
Volume298-299
DOIs
StatePublished - Apr 1 2015

Fingerprint

dynamical systems
statistics
filters
assimilation
Kalman filters
engineering
predictions

Keywords

  • Curse of dimensionality
  • Data assimilation
  • Hybrid filters
  • Turbulent dynamical systems

ASJC Scopus subject areas

  • Condensed Matter Physics
  • Statistical and Nonlinear Physics

Cite this

Blended particle methods with adaptive subspaces for filtering turbulent dynamical systems. / Qi, Di; Majda, Andrew J.

In: Physica D: Nonlinear Phenomena, Vol. 298-299, 01.04.2015, p. 21-41.

Research output: Contribution to journalArticle

@article{02273f88d02b4b4b8fffdc06185aec62,
title = "Blended particle methods with adaptive subspaces for filtering turbulent dynamical systems",
abstract = "It is a major challenge throughout science and engineering to improve uncertain model predictions by utilizing noisy data sets from nature. Hybrid methods combining the advantages of traditional particle filters and the Kalman filter offer a promising direction for filtering or data assimilation in high dimensional turbulent dynamical systems. In this paper, blended particle filtering methods that exploit the physical structure of turbulent dynamical systems are developed. Non-Gaussian features of the dynamical system are captured adaptively in an evolving-in-time low dimensional subspace through particle methods, while at the same time statistics in the remaining portion of the phase space are amended by conditional Gaussian mixtures interacting with the particles. The importance of both using the adaptively evolving subspace and introducing conditional Gaussian statistics in the orthogonal part is illustrated here by simple examples. For practical implementation of the algorithms, finding the most probable distributions that characterize the statistics in the phase space as well as effective resampling strategies is discussed to handle realizability and stability issues. To test the performance of the blended algorithms, the forty dimensional Lorenz 96 system is utilized with a five dimensional subspace to run particles. The filters are tested extensively in various turbulent regimes with distinct statistics and with changing observation time frequency and both dense and sparse spatial observations. In real applications perfect dynamical models are always inaccessible considering the complexities in both modeling and computation of high dimensional turbulent system. The effects of model errors from imperfect modeling of the systems are also checked for these methods. The blended methods show uniformly high skill in both capturing non-Gaussian statistics and achieving accurate filtering results in various dynamical regimes with and without model errors.",
keywords = "Curse of dimensionality, Data assimilation, Hybrid filters, Turbulent dynamical systems",
author = "Di Qi and Majda, {Andrew J.}",
year = "2015",
month = "4",
day = "1",
doi = "10.1016/j.physd.2015.02.002",
language = "English (US)",
volume = "298-299",
pages = "21--41",
journal = "Physica D: Nonlinear Phenomena",
issn = "0167-2789",
publisher = "Elsevier",

}

TY - JOUR

T1 - Blended particle methods with adaptive subspaces for filtering turbulent dynamical systems

AU - Qi, Di

AU - Majda, Andrew J.

PY - 2015/4/1

Y1 - 2015/4/1

N2 - It is a major challenge throughout science and engineering to improve uncertain model predictions by utilizing noisy data sets from nature. Hybrid methods combining the advantages of traditional particle filters and the Kalman filter offer a promising direction for filtering or data assimilation in high dimensional turbulent dynamical systems. In this paper, blended particle filtering methods that exploit the physical structure of turbulent dynamical systems are developed. Non-Gaussian features of the dynamical system are captured adaptively in an evolving-in-time low dimensional subspace through particle methods, while at the same time statistics in the remaining portion of the phase space are amended by conditional Gaussian mixtures interacting with the particles. The importance of both using the adaptively evolving subspace and introducing conditional Gaussian statistics in the orthogonal part is illustrated here by simple examples. For practical implementation of the algorithms, finding the most probable distributions that characterize the statistics in the phase space as well as effective resampling strategies is discussed to handle realizability and stability issues. To test the performance of the blended algorithms, the forty dimensional Lorenz 96 system is utilized with a five dimensional subspace to run particles. The filters are tested extensively in various turbulent regimes with distinct statistics and with changing observation time frequency and both dense and sparse spatial observations. In real applications perfect dynamical models are always inaccessible considering the complexities in both modeling and computation of high dimensional turbulent system. The effects of model errors from imperfect modeling of the systems are also checked for these methods. The blended methods show uniformly high skill in both capturing non-Gaussian statistics and achieving accurate filtering results in various dynamical regimes with and without model errors.

AB - It is a major challenge throughout science and engineering to improve uncertain model predictions by utilizing noisy data sets from nature. Hybrid methods combining the advantages of traditional particle filters and the Kalman filter offer a promising direction for filtering or data assimilation in high dimensional turbulent dynamical systems. In this paper, blended particle filtering methods that exploit the physical structure of turbulent dynamical systems are developed. Non-Gaussian features of the dynamical system are captured adaptively in an evolving-in-time low dimensional subspace through particle methods, while at the same time statistics in the remaining portion of the phase space are amended by conditional Gaussian mixtures interacting with the particles. The importance of both using the adaptively evolving subspace and introducing conditional Gaussian statistics in the orthogonal part is illustrated here by simple examples. For practical implementation of the algorithms, finding the most probable distributions that characterize the statistics in the phase space as well as effective resampling strategies is discussed to handle realizability and stability issues. To test the performance of the blended algorithms, the forty dimensional Lorenz 96 system is utilized with a five dimensional subspace to run particles. The filters are tested extensively in various turbulent regimes with distinct statistics and with changing observation time frequency and both dense and sparse spatial observations. In real applications perfect dynamical models are always inaccessible considering the complexities in both modeling and computation of high dimensional turbulent system. The effects of model errors from imperfect modeling of the systems are also checked for these methods. The blended methods show uniformly high skill in both capturing non-Gaussian statistics and achieving accurate filtering results in various dynamical regimes with and without model errors.

KW - Curse of dimensionality

KW - Data assimilation

KW - Hybrid filters

KW - Turbulent dynamical systems

UR - http://www.scopus.com/inward/record.url?scp=84923538363&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84923538363&partnerID=8YFLogxK

U2 - 10.1016/j.physd.2015.02.002

DO - 10.1016/j.physd.2015.02.002

M3 - Article

AN - SCOPUS:84923538363

VL - 298-299

SP - 21

EP - 41

JO - Physica D: Nonlinear Phenomena

JF - Physica D: Nonlinear Phenomena

SN - 0167-2789

ER -