Mathematical test criteria for filtering complex systems

Plentiful observations

E. Castronovo, J. Harlim, A. J. Majda

Research output: Contribution to journalArticle

Abstract

An important emerging scientific issue is the real time filtering through observations of noisy turbulent signals for complex systems as well as the statistical accuracy of spatio-temporal discretizations for such systems. These issues are addressed here in detail for the setting with plentiful observations for a scalar field through explicit mathematical test criteria utilizing a recent theory [A.J. Majda, M.J. Grote, Explicit off-line criteria for stable accurate time filtering of strongly unstable spatially extended systems, Proceedings of the National Academy of Sciences 104 (4) (2007) 1124-1129]. For plentiful observations, the number of observations equals the number of mesh points. These test criteria involve much simpler decoupled complex scalar filtering test problems with explicit formulas and elementary numerical experiments which are developed here as guidelines for filter performance. The theory includes information criteria to avoid filter divergence with large model errors, asymptotic Kalman gain, filter stability, and accurate filtering with small ensemble size as well as rigorous results delineating the role of various turbulent spectra for filtering under mesh refinement. These guidelines are also applied to discrete approximations for filtering the stochastically forced dissipative advection equation with very turbulent and noisy signals with either an equipartition of energy or -5/3 turbulent spectrum with infrequent observations as severe test problems. The theory and companion simulations demonstrate accurate statistical filtering in this context with implicit schemes with large time step with very small ensemble sizes and even with unstable explicit schemes under appropriate circumstances provided the filtering strategies are guided by the off-line theoretical criteria. The surprising failure of other strongly stable filtering strategies is also explained through these off-line criteria.

Original languageEnglish (US)
Pages (from-to)3678-3714
Number of pages37
JournalJournal of Computational Physics
Volume227
Issue number7
DOIs
StatePublished - Mar 20 2008

Fingerprint

complex systems
Large scale systems
filters
Information theory
Advection
scalars
information theory
advection
mesh
congressional reports
emerging
divergence
Experiments
approximation

Keywords

  • Data assimilation
  • Finite difference
  • Kalman filter
  • Turbulence

ASJC Scopus subject areas

  • Computer Science Applications
  • Physics and Astronomy(all)

Cite this

Mathematical test criteria for filtering complex systems : Plentiful observations. / Castronovo, E.; Harlim, J.; Majda, A. J.

In: Journal of Computational Physics, Vol. 227, No. 7, 20.03.2008, p. 3678-3714.

Research output: Contribution to journalArticle

Castronovo, E. ; Harlim, J. ; Majda, A. J. / Mathematical test criteria for filtering complex systems : Plentiful observations. In: Journal of Computational Physics. 2008 ; Vol. 227, No. 7. pp. 3678-3714.
@article{ca1f93703af8412ea5e843dc26ff6719,
title = "Mathematical test criteria for filtering complex systems: Plentiful observations",
abstract = "An important emerging scientific issue is the real time filtering through observations of noisy turbulent signals for complex systems as well as the statistical accuracy of spatio-temporal discretizations for such systems. These issues are addressed here in detail for the setting with plentiful observations for a scalar field through explicit mathematical test criteria utilizing a recent theory [A.J. Majda, M.J. Grote, Explicit off-line criteria for stable accurate time filtering of strongly unstable spatially extended systems, Proceedings of the National Academy of Sciences 104 (4) (2007) 1124-1129]. For plentiful observations, the number of observations equals the number of mesh points. These test criteria involve much simpler decoupled complex scalar filtering test problems with explicit formulas and elementary numerical experiments which are developed here as guidelines for filter performance. The theory includes information criteria to avoid filter divergence with large model errors, asymptotic Kalman gain, filter stability, and accurate filtering with small ensemble size as well as rigorous results delineating the role of various turbulent spectra for filtering under mesh refinement. These guidelines are also applied to discrete approximations for filtering the stochastically forced dissipative advection equation with very turbulent and noisy signals with either an equipartition of energy or -5/3 turbulent spectrum with infrequent observations as severe test problems. The theory and companion simulations demonstrate accurate statistical filtering in this context with implicit schemes with large time step with very small ensemble sizes and even with unstable explicit schemes under appropriate circumstances provided the filtering strategies are guided by the off-line theoretical criteria. The surprising failure of other strongly stable filtering strategies is also explained through these off-line criteria.",
keywords = "Data assimilation, Finite difference, Kalman filter, Turbulence",
author = "E. Castronovo and J. Harlim and Majda, {A. J.}",
year = "2008",
month = "3",
day = "20",
doi = "10.1016/j.jcp.2007.12.016",
language = "English (US)",
volume = "227",
pages = "3678--3714",
journal = "Journal of Computational Physics",
issn = "0021-9991",
publisher = "Academic Press Inc.",
number = "7",

}

TY - JOUR

T1 - Mathematical test criteria for filtering complex systems

T2 - Plentiful observations

AU - Castronovo, E.

AU - Harlim, J.

AU - Majda, A. J.

PY - 2008/3/20

Y1 - 2008/3/20

N2 - An important emerging scientific issue is the real time filtering through observations of noisy turbulent signals for complex systems as well as the statistical accuracy of spatio-temporal discretizations for such systems. These issues are addressed here in detail for the setting with plentiful observations for a scalar field through explicit mathematical test criteria utilizing a recent theory [A.J. Majda, M.J. Grote, Explicit off-line criteria for stable accurate time filtering of strongly unstable spatially extended systems, Proceedings of the National Academy of Sciences 104 (4) (2007) 1124-1129]. For plentiful observations, the number of observations equals the number of mesh points. These test criteria involve much simpler decoupled complex scalar filtering test problems with explicit formulas and elementary numerical experiments which are developed here as guidelines for filter performance. The theory includes information criteria to avoid filter divergence with large model errors, asymptotic Kalman gain, filter stability, and accurate filtering with small ensemble size as well as rigorous results delineating the role of various turbulent spectra for filtering under mesh refinement. These guidelines are also applied to discrete approximations for filtering the stochastically forced dissipative advection equation with very turbulent and noisy signals with either an equipartition of energy or -5/3 turbulent spectrum with infrequent observations as severe test problems. The theory and companion simulations demonstrate accurate statistical filtering in this context with implicit schemes with large time step with very small ensemble sizes and even with unstable explicit schemes under appropriate circumstances provided the filtering strategies are guided by the off-line theoretical criteria. The surprising failure of other strongly stable filtering strategies is also explained through these off-line criteria.

AB - An important emerging scientific issue is the real time filtering through observations of noisy turbulent signals for complex systems as well as the statistical accuracy of spatio-temporal discretizations for such systems. These issues are addressed here in detail for the setting with plentiful observations for a scalar field through explicit mathematical test criteria utilizing a recent theory [A.J. Majda, M.J. Grote, Explicit off-line criteria for stable accurate time filtering of strongly unstable spatially extended systems, Proceedings of the National Academy of Sciences 104 (4) (2007) 1124-1129]. For plentiful observations, the number of observations equals the number of mesh points. These test criteria involve much simpler decoupled complex scalar filtering test problems with explicit formulas and elementary numerical experiments which are developed here as guidelines for filter performance. The theory includes information criteria to avoid filter divergence with large model errors, asymptotic Kalman gain, filter stability, and accurate filtering with small ensemble size as well as rigorous results delineating the role of various turbulent spectra for filtering under mesh refinement. These guidelines are also applied to discrete approximations for filtering the stochastically forced dissipative advection equation with very turbulent and noisy signals with either an equipartition of energy or -5/3 turbulent spectrum with infrequent observations as severe test problems. The theory and companion simulations demonstrate accurate statistical filtering in this context with implicit schemes with large time step with very small ensemble sizes and even with unstable explicit schemes under appropriate circumstances provided the filtering strategies are guided by the off-line theoretical criteria. The surprising failure of other strongly stable filtering strategies is also explained through these off-line criteria.

KW - Data assimilation

KW - Finite difference

KW - Kalman filter

KW - Turbulence

UR - http://www.scopus.com/inward/record.url?scp=39649123724&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=39649123724&partnerID=8YFLogxK

U2 - 10.1016/j.jcp.2007.12.016

DO - 10.1016/j.jcp.2007.12.016

M3 - Article

VL - 227

SP - 3678

EP - 3714

JO - Journal of Computational Physics

JF - Journal of Computational Physics

SN - 0021-9991

IS - 7

ER -