Empirical dynamic modeling

Empirical dynamic modeling (EDM) is a framework for analysis and prediction of nonlinear dynamical systems. Applications include population dynamics,^[1]^[2]^[3]^[4]^[5]^[6] ecosystem service,^[7] medicine,^[8] neuroscience,^[9]^[10]^[11] dynamical systems,^[12]^[13]^[14] geophysics,^[15]^[16]^[17] and human-computer interaction.^[18] EDM was originally developed by Robert May and George Sugihara. It can be considered a methodology for data modeling, predictive analytics, dynamical system analysis, machine learning and time series analysis.

Description

Mathematical models have tremendous power to describe observations of real-world systems. They are routinely used to test hypothesis, explain mechanisms and predict future outcomes. However, real-world systems are often nonlinear and multidimensional, in some instances rendering explicit equation-based modeling problematic. Empirical models, which infer patterns and associations from the data instead of using hypothesized equations, represent a natural and flexible framework for modeling complex dynamics.

Donald DeAngelis and Simeon Yurek illustrated that canonical statistical models are ill-posed when applied to nonlinear dynamical systems.^[19] A hallmark of nonlinear dynamics is state-dependence: system states are related to previous states governing transition from one state to another. EDM operates in this space, the multidimensional state-space of system dynamics rather than on one-dimensional observational time series. EDM does not presume relationships among states, for example, a functional dependence, but projects future states from localised, neighboring states. EDM is thus a state-space, nearest-neighbors paradigm where system dynamics are inferred from states derived from observational time series. This provides a model-free representation of the system naturally encompassing nonlinear dynamics.

A cornerstone of EDM is recognition that time series observed from a dynamical system can be transformed into higher-dimensional state-spaces by time-delay embedding with Takens's theorem. The state-space models are evaluated based on in-sample fidelity to observations, conventionally with Pearson correlation between predictions and observations.

Methods

EDM is continuing to evolve. As of 2022, the main algorithms are Simplex projection,^[20] Sequential locally weighted global linear maps (S-Map) projection,^[21] Multivariate embedding in Simplex or S-Map,^[1] Convergent cross mapping (CCM),^[22] and Multiview Embeding,^[23] described below.

Nomenclature
Parameter	Description
$E$	embedding dimension
$k$	number of nearest neighbors
$T_{p}$	prediction interval
$X\in \mathbb {R}$	observed time series
$y\in \mathbb {R} ^{E}$	vector of lagged observations
$\theta \geq 0$	S-Map localization
$X_{t}^{E}=(X_{t},X_{t-1},\dots ,X_{t-E+1})\in \mathbb {R} ^{E}$	lagged embedding vectors
$\\|v\\|$	norm of v
$N=\{N_{1},\dots ,N_{k}\}$	list of nearest neighbors

Nearest neighbors are found according to: ${\text{NN}}(y,X,k)=\|X_{N_{i}}^{E}-y\|\leq \|X_{N_{j}}^{E}-y\|{\text{ if }}1\leq i\leq j\leq k$

Simplex

Simplex projection^[20]^[24]^[25]^[26] is a nearest neighbor projection. It locates the $k$ nearest neighbors to the location in the state-space from which a prediction is desired. To minimize the number of free parameters $k$ is typically set to $E+1$ defining an $E+1$ dimensional simplex in the state-space. The prediction is computed as the average of the weighted phase-space simplex projected $Tp$ points ahead. Each neighbor is weighted proportional to their distance to the projection origin vector in the state-space.

Find $k$ nearest neighbor: $N_{k}\gets {\text{NN}}(y,X,k)$
Define the distance scale: $d\gets \|X_{N_{1}}^{E}-y\|$
Compute weights: For{ $i=1,\dots ,k$ } : $w_{i}\gets \exp(-\|X_{N_{i}}^{E}-y\|/d)$
Average of state-space simplex: ${\hat {y}}\gets \sum _{i=1}^{k}\left(w_{i}X_{N_{i}+T_{p}}\right)/\sum _{i=1}^{k}w_{i}$

S-Map

S-Map^[21] extends the state-space prediction in Simplex from an average of the $E+1$ nearest neighbors to a linear regression fit to all neighbors, but localised with an exponential decay kernel. The exponential localisation function is $F(\theta )={\text{exp}}(-\theta d/D)$ , where $d$ is the neighbor distance and $D$ the mean distance. In this way, depending on the value of $\theta$ , neighbors close to the prediction origin point have a higher weight than those further from it, such that a local linear approximation to the nonlinear system is reasonable. This localisation ability allows one to identify an optimal local scale, in-effect quantifying the degree of state dependence, and hence nonlinearity of the system.

Another feature of S-Map is that for a properly fit model, the regression coefficients between variables have been shown to approximate the gradient (directional derivative) of variables along the manifold.^[27] These Jacobians represent the time-varying interaction strengths between system variables.

Find $k$ nearest neighbor: $N\gets {\text{NN}}(y,X,k)$
Sum of distances: $D\gets {\frac {1}{k}}\sum _{i=1}^{k}\|X_{N_{i}}^{E}-y\|$
Compute weights: For{ $i=1,\dots ,k$ } : $w_{i}\gets \exp(-\theta \|X_{N_{i}}^{E}-y\|/D)$
Reweighting matrix: $W\gets {\text{diag}}(w_{i})$
Design matrix: $A\gets {\begin{bmatrix}1&X_{N_{1}}&X_{N_{1}-1}&\dots &X_{N_{1}-E+1}\\1&X_{N_{2}}&X_{N_{2}-1}&\dots &X_{N_{2}-E+1}\\\vdots &\vdots &\vdots &\ddots &\vdots \\1&X_{N_{k}}&X_{N_{k}-1}&\dots &X_{N_{k}-E+1}\end{bmatrix}}$
Weighted design matrix: $A\gets WA$
Response vector at $Tp$ : $b\gets {\begin{bmatrix}X_{N_{1}+T_{p}}\\X_{N_{2}+T_{p}}\\\vdots \\X_{N_{k}+T_{p}}\end{bmatrix}}$
Weighted response vector: $b\gets Wb$
Least squares solution (SVD): ${\hat {c}}\gets {\text{argmin}}_{c}\|Ac-b\|_{2}^{2}$
Local linear model ${\hat {c}}$ is prediction: ${\hat {y}}\gets {\hat {c}}_{0}+\sum _{i=1}^{E}{\hat {c}}_{i}y_{i}$

Multivariate Embedding

Multivariate Embedding^[1]^[12]^[28] recognizes that time-delay embeddings are not the only valid state-space construction. In Simplex and S-Map one can generate a state-space from observational vectors, or time-delay embeddings of a single observational time series, or both.

Convergent Cross Mapping

Convergent cross mapping (CCM)^[22] leverages a corollary to the Generalized Takens Theorem^[12] that it should be possible to cross predict or cross map between variables observed from the same system. Suppose that in some dynamical system involving variables $X$ and $Y$ , $X$ causes $Y$ . Since $X$ and $Y$ belong to the same dynamical system, their reconstructions (via embeddings) $M_{x}$ , and $M_{y}$ , also map to the same system.

The causal variable $X$ leaves a signature on the affected variable $Y$ , and consequently, the reconstructed states based on $Y$ can be used to cross predict values of $X$ . CCM leverages this property to infer causality by predicting $X$ using the $M_{y}$ library of points (or vice versa for the other direction of causality), while assessing improvements in cross map predictability as larger and larger random samplings of $M_{y}$ are used. If the prediction skill of $X$ increases and saturates as the entire $M_{y}$ is used, this provides evidence that $X$ is casually influencing $Y$ .

Multiview Embedding

Multiview Embedding^[23] is a Dimensionality reduction technique where a large number of state-space time series vectors are combitorially assessed towards maximal model predictability.

Extensions

Extensions to EDM techniques include:

Generalized Theorems for Nonlinear State Space Reconstruction^[12]
Extended Convergent Cross Mapping^[13]
Dynamic stability^[4]
S-Map regularization^[29]
Visual analytics with EDM^[30]
Convergent Cross Sorting^[31]
Expert system with EDM hybrid^[32]
Sliding windows based on the extended convergent cross-mapping^[33]
Empirical Mode Modeling^[17]
Variable step sizes with bundle embedding^[34]
Multiview distance regularised S-map^[35]

References

^ ^a ^b ^c [1]Dixon, P. A., et al. 1999. Episodic fluctuations in larval supply. Science 283:1528–1530
^ [2]Hao Ye, Richard J. Beamish, Sarah M. Glaser, et al. 2015. Equation-free mechanistic ecosystem forecasting using empirical dynamic modeling. Proceedings of the National Academy of Sciences Mar 2015, 112 (13) E1569-E1576; DOI: 10.1073/pnas.1417063112
^ [3]Ethan R. Deyle, Michael Fogarty, Chih-hao Hsieh, et al. 2013. Proceedings of the National Academy of Sciences Apr 2013, 110 (16) 6430-6435; DOI: 10.1073/pnas.1215506110
^ ^a ^b [4]Ushio, M., Hsieh, Ch., Masuda, R. et al., 2018. Fluctuating interaction network and time-varying stability of a natural fish community. Nature 554, 360–363
^ [5]Deyle E.R., et al. 2016. Tracking and forecasting ecosystem interactions in real time. Proc. R. Soc. B 283: 20152258
^ [6]Tanya L. Rogers, Stephan B. Munch, Simon D. Stewart, Eric P. Palkovacs, Alfredo Giron-Nava, Shin-ichiro S. Matsuzaki, Celia C. Symons. Ecology Letters, 23 (8) August 2020, 1287-1297
^ [7]Park J., et al. 2021. Dynamics of Florida milk production and total phosphate in Lake Okeechobee. PLoS ONE 16(8): e0248910. doi:10.1371/journal.pone.0248910
^ [8]George Sugihara, Walter Allan, Daniel Sobel, and Kenneth D. Allan, 1996. Nonlinear control of heart rate variability in human infants. Proc. Natl. Acad. Sci. USA. Vol. 93, pp. 2608-2613, March 1996. Medical Sciences
^ [9]McBride, J. C., et al. Sugihara causality analysis of scalp EEG for detection of early Alzheimer's disease. Neuroimage-Clinical 7:258–265 (2015)
^ [10]Tajima S, Yanagawa T, Fujii N, Toyoizumi T (2015) Untangling Brain-Wide Dynamics in Consciousness by Cross-Embedding. PLoS Comput Biol 11(11): e1004537. https://doi.org/10.1371/journal.pcbi.1004537
^ [11]W. Watanakeesuntorn et al., "Massively Parallel Causal Inference of Whole Brain Dynamics at Single Neuron Resolution," 2020 IEEE 26th International Conference on Parallel and Distributed Systems (ICPADS), 2020, pp. 196-205, doi: 10.1109/ICPADS51040.2020.00035
^ ^a ^b ^c ^d [12] Deyle ER, Sugihara G (2011) Generalized Theorems for Nonlinear State Space Reconstruction. PLoS ONE 6(3): e18295. doi:10.1371/journal.pone.0018295
^ ^a ^b [13]Ye, H., Deyle, E., Gilarranz, L. et al., 2015. Distinguishing time-delayed causal interactions using convergent cross mapping. Sci Rep 5, 14750 (2015). doi:10.1038/srep14750
^ [14]Cenci, S., Saavedra, S. Non-parametric estimation of the structural stability of non-equilibrium community dynamics. Nat Ecol Evol 3, 912–918 (2019). https://doi.org/10.1038/s41559-019-0879-1
^ [15]Tsonis A. A., et al. Dynamical evidence for causality between galactic cosmic rays and interannual variation in global temperature. Proc Natl Acad Sci 112(11):3253–3256 (2015).
^ [16]Nes EH Van, et al. Causal feedbacks in climate change. Nat Clim Chang 5(5):445–448 (2015)
^ ^a ^b [17]Park, J., et al. Empirical mode modeling. Nonlinear Dyn (2022). https://doi.org/10.1007/s11071-022-07311-y
^ van Berkel, Niels; Dennis, Simon; Zyphur, Michael; Li, Jinjing; Heathcote, Andrew; Kostakos, Vassilis (2021-07-04). "Modeling interaction as a complex system". Human–Computer Interaction. 36 (4): 279–305. doi:10.1080/07370024.2020.1715221. hdl:11343/247884. ISSN 0737-0024. S2CID 211267275.
^ [18]Donald L. DeAngelis, Simeon Yurek, 2015, Equation-free modeling unravels the behavior of complex ecological systems. Proceedings of the National Academy of Sciences Mar 2015, 112 (13) 3856-3857; DOI: 10.1073/pnas.1503154112
^ ^a ^b [19] Sugihara G. and May R., 1990. Nonlinear forecasting as a way of distinguishing chaos from measurement error in time series. Nature, 344:734–741
^ ^a ^b [20] Sugihara G., 1994. Nonlinear forecasting for the classification of natural time series. Philosophical Transactions: Physical Sciences and Engineering, 348 (1688) : 477–495
^ ^a ^b [21] Sugihara G., May R., Ye H., et al. 2012. Detecting Causality in Complex Ecosystems. Science 338:496-500
^ ^a ^b [22] Ye H., and G. Sugihara, 2016. Information leverage in interconnected ecosystems: Overcoming the curse of dimensionality. Science 353:922–925
^ [23] Takens, F. (1981). Detecting strange attractors in turbulence. In D. A. Rand & L. S. Young (Eds.), Dynamical Systems and Turbulence (pp. 366–381). Springer.
^ [24] Casdagli, M. (1989). Nonlinear prediction of chaotic time series. Physica D: Nonlinear Phenomena, 35(3), 335–356.
^ [25] Judd, K., & Mees, A. (1998). Embedding as a modeling problem. Physica D: Nonlinear Phenomena, 120(3), 273–286.
^ [26]Deyle ER. et al. 2016. Tracking and forecasting ecosystem interactions in real time. Proc. R. Soc. B 283: 20152258
^ [27] Sauer, T., Yorke, J. A., & Casdagli, M. (1991). Embedology. Journal of Statistical Physics, 65(3), 579–616
^ [28]Cenci S, Sugihara G, Saavedra S, 2019. Regularized S-map for inference and forecasting with noisy ecological time series, METHODS IN ECOLOGY AND EVOLUTION, 10 (5), 650-660
^ [29] Hiroaki Natsukawa, et al. 2021. A Visual Analytics Approach for Ecosystem Dynamics based on Empirical Dynamic Modeling. IEEE Transactions on Visualization and Computer Graphics. Feb. 2021, 506-516, vol. 27 DOI: 10.1109/TVCG.2020.3028956
^ [30] Breston, L., Leonardis, E.J., Quinn, L.K. et al. 2021. Convergent cross sorting for estimating dynamic coupling. Sci Rep 11, 20374 (2021). doi:10.1038/s41598-021-98864-2
^ [31] Deyle E. R. et al. A hybrid empirical and parametric approach for managing ecosystem complexity: Water quality in Lake Geneva under nonstationary futures. PNAS Vol. 119, No. 26 (2022).
^ [32] Ge, X., Lin, A. Dynamic causality analysis using overlapped sliding windows based on the extended convergent cross-mapping. Nonlinear Dyn 104, 1753–1765 (2021). https://doi.org/10.1007/s11071-021-06362-x
^ [33] Bethany Johnson, Stephan B. Munch. 2022. An empirical dynamic modeling framework for missing or irregular samples. Ecological Modelling, Volume 468, June 2022, 109948.
^ [34] Chang, C.-W., Miki, T., Ushio, M., et al. (2021) Reconstructing large interaction networks from empirical time series data. Ecology Letters, 24, 2763– 2774. https://doi.org/10.1111/ele.13897

External links

Animations

Online books or lecture notes

EDM Introduction. Introduction with video, examples and references.
Geometrical theory of dynamical systems. Nils Berglund's lecture notes for a course at ETH at the advanced undergraduate level.
Arxiv preprint server has daily submissions of (non-refereed) manuscripts in dynamical systems.

Research groups

Sugihara Lab, Scripps Institution of Oceanography, University of California San Diego.

[Episodic_fluctuations_in_larval_supply,_Science_1999-1] [1]Dixon, P. A., et al. 1999. Episodic fluctuations in larval supply. Science 283:1528–1530

[2] [2]Hao Ye, Richard J. Beamish, Sarah M. Glaser, et al. 2015. Equation-free mechanistic ecosystem forecasting using empirical dynamic modeling. Proceedings of the National Academy of Sciences Mar 2015, 112 (13) E1569-E1576; DOI: 10.1073/pnas.1417063112

[3] [3]Ethan R. Deyle, Michael Fogarty, Chih-hao Hsieh, et al. 2013. Proceedings of the National Academy of Sciences Apr 2013, 110 (16) 6430-6435; DOI: 10.1073/pnas.1215506110

[Fluctuating_interaction_network,_Nature_2018-4] [4]Ushio, M., Hsieh, Ch., Masuda, R. et al., 2018. Fluctuating interaction network and time-varying stability of a natural fish community. Nature 554, 360–363

[5] [5]Deyle E.R., et al. 2016. Tracking and forecasting ecosystem interactions in real time. Proc. R. Soc. B 283: 20152258

[6] [6]Tanya L. Rogers, Stephan B. Munch, Simon D. Stewart, Eric P. Palkovacs, Alfredo Giron-Nava, Shin-ichiro S. Matsuzaki, Celia C. Symons. Ecology Letters, 23 (8) August 2020, 1287-1297

[7] [7]Park J., et al. 2021. Dynamics of Florida milk production and total phosphate in Lake Okeechobee. PLoS ONE 16(8): e0248910. doi:10.1371/journal.pone.0248910

[8] [8]George Sugihara, Walter Allan, Daniel Sobel, and Kenneth D. Allan, 1996. Nonlinear control of heart rate variability in human infants. Proc. Natl. Acad. Sci. USA. Vol. 93, pp. 2608-2613, March 1996. Medical Sciences

[McBride_et_al_EEG_Alzheimers_2015-9] [9]McBride, J. C., et al. Sugihara causality analysis of scalp EEG for detection of early Alzheimer's disease. Neuroimage-Clinical 7:258–265 (2015)

[Tajima_et_al_Brain-wide_dynamics_2015-10] [10]Tajima S, Yanagawa T, Fujii N, Toyoizumi T (2015) Untangling Brain-Wide Dynamics in Consciousness by Cross-Embedding. PLoS Comput Biol 11(11): e1004537. https://doi.org/10.1371/journal.pcbi.1004537

[Watanakeesuntorn_et_al_Massively_Parallel_Causal_Inference_of_Whole_Brain_Dynamics_2020-11] [11]W. Watanakeesuntorn et al., "Massively Parallel Causal Inference of Whole Brain Dynamics at Single Neuron Resolution," 2020 IEEE 26th International Conference on Parallel and Distributed Systems (ICPADS), 2020, pp. 196-205, doi: 10.1109/ICPADS51040.2020.00035

[Generalized_Theorems_for_Nonlinear_State_Space_Reconstruction._PLoS_ONE,_2011-12] [12] Deyle ER, Sugihara G (2011) Generalized Theorems for Nonlinear State Space Reconstruction. PLoS ONE 6(3): e18295. doi:10.1371/journal.pone.0018295

[Convergent_cross_mapping_Science_2015-13] [13]Ye, H., Deyle, E., Gilarranz, L. et al., 2015. Distinguishing time-delayed causal interactions using convergent cross mapping. Sci Rep 5, 14750 (2015). doi:10.1038/srep14750

[14] [14]Cenci, S., Saavedra, S. Non-parametric estimation of the structural stability of non-equilibrium community dynamics. Nat Ecol Evol 3, 912–918 (2019). https://doi.org/10.1038/s41559-019-0879-1

[Tsonis_et_al_2015-15] [15]Tsonis A. A., et al. Dynamical evidence for causality between galactic cosmic rays and interannual variation in global temperature. Proc Natl Acad Sci 112(11):3253–3256 (2015).

[Nes_EH_Van_et_al_2015-16] [16]Nes EH Van, et al. Causal feedbacks in climate change. Nat Clim Chang 5(5):445–448 (2015)

[Park_et_al_Nonlinear_Dynamics_2022-17] [17]Park, J., et al. Empirical mode modeling. Nonlinear Dyn (2022). https://doi.org/10.1007/s11071-022-07311-y

[18] van Berkel, Niels; Dennis, Simon; Zyphur, Michael; Li, Jinjing; Heathcote, Andrew; Kostakos, Vassilis (2021-07-04). "Modeling interaction as a complex system". Human–Computer Interaction. 36 (4): 279–305. doi:10.1080/07370024.2020.1715221. hdl:11343/247884. ISSN 0737-0024. S2CID 211267275.

[19] [18]Donald L. DeAngelis, Simeon Yurek, 2015, Equation-free modeling unravels the behavior of complex ecological systems. Proceedings of the National Academy of Sciences Mar 2015, 112 (13) 3856-3857; DOI: 10.1073/pnas.1503154112

[Nonlinear_forecasting_Nature_1990-20] [19] Sugihara G. and May R., 1990. Nonlinear forecasting as a way of distinguishing chaos from measurement error in time series. Nature, 344:734–741

[royalsocietypublishing.org-21] [20] Sugihara G., 1994. Nonlinear forecasting for the classification of natural time series. Philosophical Transactions: Physical Sciences and Engineering, 348 (1688) : 477–495

[Detecting_Causality_Science_2012-22] [21] Sugihara G., May R., Ye H., et al. 2012. Detecting Causality in Complex Ecosystems. Science 338:496-500

[science.sciencemag.org-23] [22] Ye H., and G. Sugihara, 2016. Information leverage in interconnected ecosystems: Overcoming the curse of dimensionality. Science 353:922–925

[24] [23] Takens, F. (1981). Detecting strange attractors in turbulence. In D. A. Rand & L. S. Young (Eds.), Dynamical Systems and Turbulence (pp. 366–381). Springer.

[25] [24] Casdagli, M. (1989). Nonlinear prediction of chaotic time series. Physica D: Nonlinear Phenomena, 35(3), 335–356.

[26] [25] Judd, K., & Mees, A. (1998). Embedding as a modeling problem. Physica D: Nonlinear Phenomena, 120(3), 273–286.

[27] [26]Deyle ER. et al. 2016. Tracking and forecasting ecosystem interactions in real time. Proc. R. Soc. B 283: 20152258

[28] [27] Sauer, T., Yorke, J. A., & Casdagli, M. (1991). Embedology. Journal of Statistical Physics, 65(3), 579–616

[29] [28]Cenci S, Sugihara G, Saavedra S, 2019. Regularized S-map for inference and forecasting with noisy ecological time series, METHODS IN ECOLOGY AND EVOLUTION, 10 (5), 650-660

[30] [29] Hiroaki Natsukawa, et al. 2021. A Visual Analytics Approach for Ecosystem Dynamics based on Empirical Dynamic Modeling. IEEE Transactions on Visualization and Computer Graphics. Feb. 2021, 506-516, vol. 27 DOI: 10.1109/TVCG.2020.3028956

[31] [30] Breston, L., Leonardis, E.J., Quinn, L.K. et al. 2021. Convergent cross sorting for estimating dynamic coupling. Sci Rep 11, 20374 (2021). doi:10.1038/s41598-021-98864-2

[A_hybrid_empirical_and_parametric_approach_for_managing_ecosystem_complexity,_PNAS_2022-32] [31] Deyle E. R. et al. A hybrid empirical and parametric approach for managing ecosystem complexity: Water quality in Lake Geneva under nonstationary futures. PNAS Vol. 119, No. 26 (2022).

[33] [32] Ge, X., Lin, A. Dynamic causality analysis using overlapped sliding windows based on the extended convergent cross-mapping. Nonlinear Dyn 104, 1753–1765 (2021). https://doi.org/10.1007/s11071-021-06362-x

[34] [33] Bethany Johnson, Stephan B. Munch. 2022. An empirical dynamic modeling framework for missing or irregular samples. Ecological Modelling, Volume 468, June 2022, 109948.

[35] [34] Chang, C.-W., Miki, T., Ushio, M., et al. (2021) Reconstructing large interaction networks from empirical time series data. Ecology Letters, 24, 2763– 2774. https://doi.org/10.1111/ele.13897

[1]

[2]

[3]

[4]

[5]

[6]

[7]

[8]

[9]

[10]

[11]

[12]

[13]

[14]

[15]

[16]

[17]

[18]

[19]

[20]

[21]

[22]

[23]

[24]

[25]

[26]

[27]

[28]

[29]

[30]

[31]

[32]

[33]

[34]

[35]