Curated list of neural differential equations for time series analysis and the IJCAI-25 review paper.

Oh, Y., Kam, S., Lee, J., Lim, D.-Y., Kim, S., & Bui, A. A. T. (2025). Comprehensive Review of Neural Differential Equations for Time Series Analysis. In J. Kwok (Ed.), Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25 (pp. 10621–10631). International Joint Conferences on Artificial Intelligence Organization. https://doi.org/10.24963/ijcai.2025/1179 [paper]

2026

1. NCDEs

   Neural Controlled Differential Equation and Its Application in Pharmacokinetics and Pharmacodynamics

   Zhisong Wu, Pingyao Luo, Rong Chen, and 3 more authors

   CPT: Pharmacometrics & Systems Pharmacology, 2026

   Abs DOI Paper

   With the recent advances in machine learning (ML) and artificial intelligence (AI), data-driven modeling approaches for pharmacokinetics (PK) and pharmacodynamics (PD) have gained popularity due to their versatility in diverse settings and reduced reliance on prior assumptions. However, most of the ML methods ignore the hidden dynamics behind the data, lacking interpretability. This study investigated the applicability of neural controlled differential equation (NCDE), a novel ML method that is suitable for data-driven modeling of PK and PD profiles, especially in the setting of multiple dosing. We demonstrated that NCDE was capable of combining differential-equation-based dynamics with data-driven characteristics, flexibly incorporating various types of inputs, and embedding discontinuous dynamics. Moreover, a direct correspondence was identified between the learned dynamics of NCDE and the dynamics behind the data, which highlights the intrinsic interpretability of NCDE. Additionally, the influence of important hyperparameters was systematically investigated, and it was found that L1 regularization and the AdaMax optimizer were useful for stabilizing the training process and leading to a generalizable NCDE model. Together, these findings demonstrate the accuracy, generalizability, and interpretability of NCDE, indicating that NCDE is a reliable method for further application. In the future, NCDE may further facilitate PK and PD prediction in general.

2. NCDEs

   FlowPath: Learning Data-Driven Manifolds with Invertible Flows for Robust Irregularly-sampled Time Series Classification

   YongKyung Oh, Dong-Young Lim, and Sungil Kim

   Proceedings of the AAAI Conference on Artificial Intelligence, 2026

   Abs DOI Paper arXiv

   Modeling continuous-time dynamics from sparse and irregularly-sampled time series remains a fundamental challenge. Neural controlled differential equations provide a principled framework for such tasks, yet their performance is highly sensitive to the choice of control path constructed from discrete observations. Existing methods commonly employ fixed interpolation schemes, which impose simplistic geometric assumptions that often misrepresent the underlying data manifold, particularly under high missingness. We propose FlowPath, a novel approach that learns the geometry of the control path via an invertible neural flow. Rather than merely connecting observations, FlowPath constructs a continuous and data-adaptive manifold, guided by invertibility constraints that enforce information-preserving and well-behaved transformations. This inductive bias distinguishes FlowPath from prior unconstrained learnable path models. Empirical evaluations on 18 benchmark datasets and a real-world case study demonstrate that FlowPath consistently achieves statistically significant improvements in classification accuracy over baselines using fixed interpolants or non-invertible architectures. These results highlight the importance of modeling not only the dynamics along the path but also the geometry of the path itself, offering a robust and generalizable solution for learning from irregular time series.

3. NDEs

   Continuum Dropout for Neural Differential Equations

   Jonghun Lee, YongKyung Oh, Sungil Kim, and 1 more author

   Proceedings of the AAAI Conference on Artificial Intelligence, 2026

   Abs DOI Paper arXiv

   Neural Differential Equations (NDEs) excel at modeling continuous-time dynamics, effectively handling challenges such as irregular observations, missing values, and noise. Despite their advantages, NDEs face a fundamental challenge in adopting dropout, a cornerstone of deep learning regularization, making them susceptible to overfitting. To address this research gap, we introduce Continuum Dropout, a universally applicable regularization technique for NDEs built upon the theory of alternating renewal processes. Continuum Dropout formulates the on-off mechanism of dropout as a stochastic process that alternates between active (evolution) and inactive (paused) states in continuous time. This provides a principled approach to prevent overfitting and enhance the generalization capabilities of NDEs. Moreover, Continuum Dropout offers a structured framework to quantify predictive uncertainty via Monte Carlo sampling at test time. Through extensive experiments, we demonstrate that Continuum Dropout outperforms existing regularization methods for NDEs, achieving superior performance on various time series and image classification tasks. It also yields better-calibrated and more trustworthy probability estimates, highlighting its effectiveness for uncertainty-aware modeling.

2025

1. NSDEs

   A Generalized Degradation Model Based on Semi-Physics-Informed Neural Stochastic Differential Equation

   Zirong Wang, Zhen Chen, Tangbin Xia, and 1 more author

   IEEE Transactions on Reliability, 2025

   Abs DOI Paper

   Accurate degradation modeling is a prerequisite for reliable prognostics. When historical data are scarce and operating conditions vary over time, conventional approaches struggle to balance accuracy, adaptability, and interpretability, and lack robustness to changing environments, especially when encoding the effects of operating conditions directly in the model. To overcome these limitations, this paper proposes a novel generalized degradation model based on a semi-physics-informed neural stochastic differential equation, where neural stochastic differential equation (NSDE) is utilized to describe the degradation dynamics. In contrast, the effect of operating conditions on degradation rate is injected in a plug-and-play prior without being locked into the NSDE structure. A variational inference-based generative training procedure jointly estimates the parameters of the NSDE and the prior, mitigating the adverse effect of imperfect physics and requiring only modest historical data. Then, an approximate closed-form distribution for the remaining useful lifetime (RUL) is derived. Thus, an approach for RUL prognostics of in-service products under dynamic operating conditions is established, leveraging the knowledge of degradation from historical data. Comprehensive studies on simulated and battery degradation data demonstrate the robustness and effectiveness of the proposed model.

2. NCDEs

   DualDynamics: Synergizing Implicit and Explicit Methods for Robust Irregular Time Series Analysis

   YongKyung Oh, Dong-Young Lim, and Sungil Kim

   Proceedings of the AAAI Conference on Artificial Intelligence, 2025

   Abs DOI Paper arXiv GitHub

   Real-world time series analysis faces significant challenges when dealing with irregular and incomplete data. While Neural Differential Equation (NDE) based methods have shown promise, they struggle with limited expressiveness, scalability issues, and stability concerns. Conversely, Neural Flows offer stability but falter with irregular data. We introduce ’DualDynamics’, a novel framework that synergistically combines NDE-based method and Neural Flow-based method. This approach enhances expressive power while balancing computational demands, addressing critical limitations of existing techniques. We demonstrate DualDynamics’ effectiveness across diverse tasks: classification of robustness to dataset shift, irregularly-sampled series analysis, interpolation of missing data, and forecasting with partial observations. Our results show consistent outperformance over state-of-the-art methods, indicating DualDynamics’ potential to advance irregular time series analysis significantly.

3. Review

   Neural Differential Equations for Continuous-Time Analysis

   Yongkyung Oh, Dongyoung Lim, and Sungil Kim

   In Proceedings of the 34th ACM International Conference on Information and Knowledge Management, 2025

   Abs DOI Paper

   Modeling complex, irregular time series is a critical challenge in knowledge discovery and data mining. This tutorial introduces Neural Differential Equations (NDEs)–a powerful paradigm for continuous-time deep learning that intrinsically handles the non-uniform sampling and missing values where traditional models falter. We provide a comprehensive review of the theory and practical application of the entire NDE family: Neural Ordinary (NODEs), Controlled (NCDEs), and Stochastic (NSDEs) Differential Equations. The tutorial emphasizes robustness and stability and culminates in a hands-on session where participants will use key open-source libraries to solve real-world tasks like interpolation and classification. Designed for AI researchers and practitioners, this tutorial equips attendees with essential tools for time series analysis.

4. Review

   Graph ODEs and Beyond: A Comprehensive Survey on Integrating Differential Equations with Graph Neural Networks

   Zewen Liu, Xiaoda Wang, Bohan Wang, and 3 more authors

   In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2, 2025

   Abs DOI Paper

   Graph Neural Networks (GNNs) and differential equations (DEs) are two rapidly advancing areas of research that have shown remarkable synergy in recent years. GNNs have emerged as powerful tools for learning on graph-structured data, while differential equations provide a principled framework for modeling continuous dynamics across time and space. The intersection of these fields has led to innovative approaches that leverage the strengths of both, enabling applications in physics-informed learning, spatiotemporal modeling, and scientific computing. This survey aims to provide a comprehensive overview of the burgeoning research at the intersection of GNNs and DEs. We will categorize existing methods, discuss their underlying principles, and highlight their applications across domains such as molecular modeling, traffic prediction, and epidemic spreading. Furthermore, we identify open challenges and outline future research directions to advance this interdisciplinary field. A comprehensive paper list is provided at https://github.com/Emory-Melody/Awesome-Graph-NDEs.

5. NODEs

   Neural fractional order differential equations

   S M Sivalingam and V. Govindaraj

   Expert Systems with Applications, 2025

   Abs DOI Paper

   Neural ordinary differential equations are architectures that connect the neural network theory with dynamical system, providing scope for continuous propagation of the layers. Meanwhile, fractional-order differential equations offer an approach to model systems with the scope to retain long-term memory. In this study, we use the idea of fractional derivative in the neural differential equations and develop the fractional version named the neural fractional differential equations. These networks can retain long-term memory and is trained using the ensemble Kalman inversion algorithm. The detailed derivation of the network architecture and the existence of a solution using Schauder’s fixed point theorem are given in detail. The forward propagation of the system is executed utilizing the L1 predictor–corrector. The issue in obtaining the gradient is explained in detail. The ensemble Kalman inversion is used to train the network memory efficiently. Numerical results for studying the dynamics of linear and nonlinear systems are provided to demonstrate the effectiveness of the network with the proposed learning method. Also, the model’s capacity to fit with real-world data is shown in the HIV, monkey-pox, Delhi temperature, and Netflix stock price datasets. The method has shown excellent performance in fewer iterations compared to neural ordinary differential equations, classical recurrent neural network, gated recurrent unit, long short-term memory networks and Bi-directional long short-term memory neural network.

6. NDEs

   TANDEM: Temporal Attention-guided Neural Differential Equations for Missingness in Time Series Classification

   Yongkyung Oh, Dongyoung Lim, Sungil Kim, and 1 more author

   In Proceedings of the 34th ACM International Conference on Information and Knowledge Management, 2025

   Abs DOI Paper arXiv

   Handling missing data in time series classification remains a significant challenge in various domains. Traditional methods often rely on imputation, which may introduce bias or fail to capture the underlying temporal dynamics. In this paper, we propose TANDEM (Temporal Attention-guided Neural Differential Equations for Missingness), an attention-guided neural differential equation framework that effectively classifies time series data with missing values. Our approach integrates raw observation, interpolated control path, and continuous latent dynamics through a novel attention mechanism, allowing the model to focus on the most informative aspects of the data. We evaluate TANDEM on 30 benchmark datasets and a real-world medical dataset, demonstrating its superiority over existing state-of-the-art methods. Our framework not only improves classification accuracy but also provides insights into the handling of missing data, making it a valuable tool in practice.

7. Review

   Comprehensive Review of Neural Differential Equations for Time Series Analysis

   YongKyung Oh, Seungsu Kam, Jonghun Lee, and 3 more authors

   In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25, 2025

   Abs DOI Paper arXiv

   Neural Differential Equations (NDEs) have emerged as a powerful paradigm for modeling complex dynamic systems, particularly in the context of time series analysis. This comprehensive review provides an in-depth exploration of the various NDE architectures, including Neural Ordinary Differential Equations (NODEs), Neural Controlled Differential Equations (NCDEs), and Neural Stochastic Differential Equations (NSDEs), and their applications in handling irregularly sampled, noisy, and high-dimensional time series data. We discuss the theoretical foundations, computational challenges, and practical considerations for implementing and training NDE-based models. Furthermore, we highlight recent advancements, open problems, and future research directions in this rapidly evolving field, aiming to provide a valuable resource for researchers and practitioners interested in leveraging NDEs for advanced time series analysis.

8. NSDEs

   Modeling Irregular Astronomical Time Series with Neural Stochastic Delay Differential Equations

   Yongkyung Oh, Seungsu Kam, Dongyoung Lim, and 1 more author

   In Proceedings of the 34th ACM International Conference on Information and Knowledge Management, 2025

   Abs DOI Paper

   Astronomical time series are often irregularly sampled due to observational constraints, making them challenging to model with standard methods. While neural ordinary differential equations (NODEs) have shown promise in handling irregular time series, they struggle to capture the stochastic nature of astronomical data, such as quasar light curves. In this paper, we propose a neural stochastic delay differential equation (NSDE) framework that extends NODEs to model irregular time series with stochastic dynamics. Our approach incorporates stochastic noise into the latent dynamics and uses a delay mechanism to capture long-term dependencies. We evaluate our framework on three astronomical datasets, demonstrating its superior performance in capturing the underlying dynamics and improving prediction accuracy compared to existing methods.

9. NODEs

   Dynamic Spatio-Temporal Attention-Based Graph Neural Network Using Ordinary Differential Equation and Multi-Scale Semantics for Traffic Prediction

   Yihang Li, Weimin Peng, Jing Chen, and 1 more author

   IEEE Transactions on Intelligent Transportation Systems, 2025

   Abs DOI Paper

   In the fields of traffic flow prediction and other intelligent forecasting applications, the technology of multivariate time series forecasting using graph neural networks (GNNs) is receiving growing attention. Although various GNN models based on dynamic graph structures have introduced promising approaches, they still face challenges in adequately capturing spatio-temporal information for traffic forecasting. To address this issue, this paper proposes a novel dynamic spatio-temporal attention-based GNN model using ordinary differential equations (ODEs), termed the dynamic graph ordinary differential equation (DGODE) model. Building upon the prior graph ordinary differential equation (GODE) framework, DGODE employs ODEs to mitigate the over-smoothing problem in GNNs and further incorporates dynamic graph structures to enhance the graph information extraction capabilities of GODE. These enhancements enable DGODE to optimize the integration of ODEs within GNNs, alleviate overfitting, and deepen the mining of spatio-temporal dependencies. The related experimental results demonstrate that, compared with existing baseline models, the proposed DGODE model achieves superior performance in traffic flow prediction with a reduced prediction error rate.

2024

1. NCDEs

   Bridging Dynamic Factor Models and Neural Controlled Differential Equations for Nowcasting GDP

   Seonkyu Lim, Jeongwhan Choi, Noseong Park, and 4 more authors

   In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, 2024

   Abs DOI Paper

   Gross domestic product (GDP) nowcasting is crucial for policy-making as GDP growth is a key indicator of economic conditions. Dynamic factor models (DFMs) have been widely adopted by government agencies for GDP nowcasting due to their ability to handle irregular or missing macroeconomic indicators and their interpretability. However, DFMs face two main challenges: i) the lack of capturing economic uncertainties such as sudden recessions or booms, and ii) the limitation of capturing irregular dynamics from mixed-frequency data. To address these challenges, we introduce NCDENow, a novel GDP nowcasting framework that integrates neural controlled differential equations (NCDEs) with DFMs. This integration effectively handles the dynamics of irregular time series.NCDENow consists of 3 main modules: i) factor extraction leveraging DFM, ii) dynamic modeling using NCDE, and iii) GDP growth prediction through regression. We evaluate NCDENow against 6 baselines on 2 real-world GDP datasets from South Korea and the United Kingdom, demonstrating its enhanced predictive capability. Our empirical results favor our method, highlighting the significant potential of integrating NCDE into nowcasting models. Our code and dataset are available at https://github.com/sklim84/NCDENow_CIKM2024.

2. NODEs

   Adaptive Decision Spatio-temporal neural ODE for traffic flow forecasting with Multi-Kernel Temporal Dynamic Dilation Convolution

   Zihao Chu, Wenming Ma, Mingqi Li, and 1 more author

   Neural Networks, 2024

   Abs DOI Paper

   Traffic flow prediction is crucial for efficient traffic management. It involves predicting vehicle movement patterns to reduce congestion and enhance traffic flow. However, the highly non-linear and complex patterns commonly observed in traffic flow pose significant challenges for this task. Current Graph Neural Network (GNN) models often construct shallow networks, which limits their ability to extract deeper spatio-temporal representations. Neural ordinary differential equations for traffic prediction address over-smoothing but require significant computational resources, leading to inefficiencies, and sometimes deeper networks may lead to poorer predictions for complex traffic information. In this study, we propose an Adaptive Decision spatio-temporal Neural Ordinary Differential Network, which can adaptively determine the number of layers of ODE according to the complexity of traffic information. It can solve the over-smoothing problem better, improving overall efficiency and prediction accuracy. In addition, traditional temporal convolution methods make it difficult to deal with complex and variable traffic time information with a large time span. Therefore, we introduce a multi-kernel temporal dynamic expansive convolution to handle the traffic time information. Multi-kernel temporal dynamic expansive convolution employs a dynamic dilation strategy, dynamically adjusting the network’s receptive field across levels, effectively capturing temporal dependencies, and can better adapt to the changing time data of traffic information. Additionally, multi-kernel temporal dynamic expansive convolution integrates multi-scale convolution kernels, enabling the model to learn features across diverse temporal scales. We evaluated our proposed method on several real-world traffic datasets. Experimental results show that our method outperformed state-of-the-art benchmarks.

3. NCDEs

   Log Neural Controlled Differential Equations: The Lie Brackets Make A Difference

   Benjamin Walker, Andrew Donald Mcleod, Tiexin Qin, and 3 more authors

   In Proceedings of the 41st International Conference on Machine Learning, 2024

   Abs Paper

   The vector field of a controlled differential equation (CDE) describes the relationship between a control path and the evolution of a solution path. Neural CDEs (NCDEs) treat time series data as observations from a control path, parameterise a CDE’s vector field using a neural network, and use the solution path as a continuously evolving hidden state. As their formulation makes them robust to irregular sampling rates, NCDEs are a powerful approach for modelling real-world data. Building on neural rough differential equations (NRDEs), we introduce Log-NCDEs, a novel, effective, and efficient method for training NCDEs. The core component of Log-NCDEs is the Log-ODE method, a tool from the study of rough paths for approximating a CDE’s solution. Log-NCDEs are shown to outperform NCDEs, NRDEs, the linear recurrent unit, S5, and MAMBA on a range of multivariate time series datasets with up to 50,000 observations.

4. NSDEs

   Stable Neural Stochastic Differential Equations in Analyzing Irregular Time Series Data.

   YongKyung Oh, Dongyoung Lim, and Sungil Kim

   In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, 2024

   Abs Paper arXiv GitHub

   Irregular sampling intervals and missing values in real-world time series data present challenges for conventional methods that assume consistent intervals and complete data. Neural Ordinary Differential Equations (Neural ODEs) offer an alternative approach, utilizing neural networks combined with ODE solvers to learn continuous latent representations through parameterized vector fields. Neural Stochastic Differential Equations (Neural SDEs) extend Neural ODEs by incorporating a diffusion term, although this addition is not trivial, particularly when addressing irregular intervals and missing values. Consequently, careful design of drift and diffusion functions is crucial for maintaining stability and enhancing performance, while incautious choices can result in adverse properties such as the absence of strong solutions, stochastic destabilization, or unstable Euler discretizations, significantly affecting Neural SDEs’ performance. In this study, we propose three stable classes of Neural SDEs: Langevin-type SDE, Linear Noise SDE, and Geometric SDE. Then, we rigorously demonstrate their robustness in maintaining excellent performance under distribution shift, while effectively preventing overfitting. To assess the effectiveness of our approach, we conduct extensive experiments on four benchmark datasets for interpolation, forecasting, and classification tasks, and analyze the robustness of our methods with 30 public datasets under different missing rates. Our results demonstrate the efficacy of the proposed method in handling real-world irregular time series data.

5. NCDEs

   Attentive neural controlled differential equations for time-series classification and forecasting

   Sheo Yon Jhin, Heejoo Shin, Sujie Kim, and 7 more authors

   Knowledge and Information Systems, Mar 2024

   Abs DOI Paper arXiv GitHub

   Neural Controlled Differential Equations (NCDEs) have shown promising results in modeling irregularly sampled time series. However, their ability to capture long-range dependencies and focus on relevant information within the time series can be limited. We propose Attentive Neural Controlled Differential Equations (Attentive NCDEs), a novel framework that integrates attention mechanisms into NCDEs. This allows the model to dynamically weigh the importance of different time points and features, enhancing its ability to capture salient information for time-series classification and forecasting. We demonstrate the effectiveness of Attentive NCDEs on various real-world datasets, showing significant improvements over state-of-the-art NCDE models, especially in complex scenarios with varying data quality and length.

6. NSDEs

   Neural Jump-Diffusion Temporal Point Processes

   Shuai Zhang, Chuan Zhou, Yang Aron Liu, and 3 more authors

   In Proceedings of the 41st International Conference on Machine Learning, Mar 2024

   Abs Paper GitHub

   We present a novel perspective on temporal point processes (TPPs) by reformulating their intensity processes as solutions to stochastic differential equations (SDEs). In particular, we first prove the equivalent SDE formulations of several classical TPPs, including Poisson processes, Hawkes processes, and self-correcting processes. Based on these proofs, we introduce a unified TPP framework called Neural Jump-Diffusion Temporal Point Process (NJDTPP), whose intensity process is governed by a neural jump-diffusion SDE (NJDSDE) where the drift, diffusion, and jump coefficient functions are parameterized by neural networks. Compared to previous works, NJDTPP exhibits model flexibility in capturing intensity dynamics without relying on any specific functional form, and provides theoretical guarantees regarding the existence and uniqueness of the solution to the proposed NJDSDE. Experiments on both synthetic and real-world datasets demonstrate that NJDTPP is capable of capturing the dynamics of intensity processes in different scenarios and significantly outperforms the state-of-the-art TPP models in prediction tasks.

7. Related

   Mamba: Linear-Time Sequence Modeling with Selective State Spaces

   Albert Gu and Tri Dao

   May 2024

   Abs DOI Paper arXiv

   Foundation models, now powering most of the exciting applications in deep learning, are almost universally based on the Transformer architecture and its core attention module. Many subquadratic-time architectures such as linear attention, gated convolution and recurrent models, and structured state space models (SSMs) have been developed to address Transformers’ computational inefficiency on long sequences, but they have not performed as well as attention on important modalities such as language. We identify that a key weakness of such models is their inability to perform content-based reasoning, and make several improvements. First, simply letting the SSM parameters be functions of the input addresses their weakness with discrete modalities, allowing the model to selectively propagate or forget information along the sequence length dimension depending on the current token. Second, even though this change prevents the use of efficient convolutions, we design a hardware-aware parallel algorithm in recurrent mode. We integrate these selective SSMs into a simplified end-to-end neural network architecture without attention or even MLP blocks (Mamba). Mamba enjoys fast inference (5

8. NSDEs

   Latent Stochastic Differential Equations for Modeling Quasar Variability and Inferring Black Hole Properties

   Joshua Fagin, Ji Won Park, Henry Best, and 6 more authors

   The Astrophysical Journal, May 2024

   Abs DOI Paper

   Quasars are bright and unobscured active galactic nuclei (AGN) thought to be powered by the accretion of matter around supermassive black holes at the centers of galaxies. The temporal variability of a quasar’s brightness contains valuable information about its physical properties. The UV/optical variability is thought to be a stochastic process, often represented as a damped random walk described by a stochastic differential equation (SDE). Upcoming wide-field telescopes such as the Rubin Observatory Legacy Survey of Space and Time (LSST) are expected to observe tens of millions of AGN in multiple filters over a ten year period, so there is a need for efficient and automated modeling techniques that can handle the large volume of data. Latent SDEs are machine learning models well suited for modeling quasar variability, as they can explicitly capture the underlying stochastic dynamics. In this work, we adapt latent SDEs to jointly reconstruct multivariate quasar light curves and infer their physical properties such as the black hole mass, inclination angle, and temperature slope. Our model is trained on realistic simulations of LSST ten year quasar light curves, and we demonstrate its ability to reconstruct quasar light curves even in the presence of long seasonal gaps and irregular sampling across different bands, outperforming a multioutput Gaussian process regression baseline. Our method has the potential to provide a deeper understanding of the physical properties of quasars and is applicable to a wide range of other multivariate time series with missing data and irregular sampling.

9. NODEs

   ClimODE: Climate and Weather Forecasting with Physics-informed Neural ODEs.

   Yogesh Verma, Markus Heinonen, and Vikas Garg

   In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, May 2024

   Paper

2023

1. NODEs

   Neural ODE Control for Classification, Approximation, and Transport

   Domènec Ruiz-Balet and Enrique Zuazua

   SIAM Review, May 2023

   Abs DOI Paper

   We analyze neural ordinary differential equations (NODEs) from a control theoretical perspective to address some of the main properties and paradigms of deep learning (DL), in particular, data classification and universal approximation. These objectives are tackled and achieved from the perspective of the simultaneous control of systems of NODEs. For instance, in the context of classification, each item to be classified corresponds to a different initial datum for the control problem of the NODE, to be classified, all of them by the same common control, to the location (a subdomain of the Euclidean space) associated to each label. Our proofs are genuinely nonlinear and constructive, allowing us to estimate the complexity of the control strategies we develop. The nonlinear nature of the activation functions governing the dynamics of NODEs under consideration plays a key role in our proofs, since it allows deforming half of the phase space while the other half remains invariant, a property that classical models in mechanics do not fulfill. This very property allows us to build elementary controls inducing specific dynamics and transformations whose concatenation, along with properly chosen hyperplanes, allows us to achieve our goals in finitely many steps. The nonlinearity of the dynamics is assumed to be Lipschitz. Therefore, our results apply also in the particular case of the ReLU activation function. We also present the counterparts in the context of the control of neural transport equations, establishing a link between optimal transport and deep neural networks.

2. NODEs

   Stabilized neural ordinary differential equations for long-time forecasting of dynamical systems

   Alec J. Linot, Joshua W. Burby, Qi Tang, and 3 more authors

   Journal of Computational Physics, May 2023

   Abs DOI Paper

   In data-driven modeling of spatiotemporal phenomena careful consideration is needed in capturing the dynamics of the high wavenumbers. This problem becomes especially challenging when the system of interest exhibits shocks or chaotic dynamics. We present a data-driven modeling method that accurately captures shocks and chaotic dynamics by proposing a new architecture, stabilized neural ordinary differential equation (ODE). In our proposed architecture, we learn the right-hand-side (RHS) of an ODE by adding the outputs of two NN together where one learns a linear term and the other a nonlinear term. Specifically, we implement this by training a sparse linear convolutional NN to learn the linear term and a dense fully-connected nonlinear NN to learn the nonlinear term. This contrasts with the standard neural ODE which involves training a single NN for the RHS. We apply this setup to the viscous Burgers equation, which exhibits shocked behavior, and show stabilized neural ODEs provide better short-time tracking, prediction of the energy spectrum, and robustness to noisy initial conditions than standard neural ODEs. We also apply this method to chaotic trajectories of the Kuramoto-Sivashinsky equation. In this case, stabilized neural ODEs keep long-time trajectories on the attractor, and are highly robust to noisy initial conditions, while standard neural ODEs fail at achieving either of these results. We conclude by demonstrating how stabilizing neural ODEs provide a natural extension for use in reduced-order modeling by projecting the dynamics onto the eigenvectors of the learned linear term.

3. NODEs

   ContiFormer: Continuous-Time Transformer for Irregular Time Series Modeling

   Yuqi Chen, Kan Ren, Yansen Wang, and 3 more authors

   In Advances in Neural Information Processing Systems, May 2023

   Abs Paper

   Modeling continuous-time dynamics on irregular time series is critical to account for data evolution and correlations that occur continuously. Traditional methods including recurrent neural networks or Transformer models leverage inductive bias via powerful neural architectures to capture complex patterns. However, due to their discrete characteristic, they have limitations in generalizing to continuous-time data paradigms. Though neural ordinary differential equations (Neural ODEs) and their variants have shown promising results in dealing with irregular time series, they often fail to capture the intricate correlations within these sequences. It is challenging yet demanding to concurrently model the relationship between input data points and capture the dynamic changes of the continuous-time system. To tackle this problem, we propose ContiFormer that extends the relation modeling of vanilla Transformer to the continuous-time domain, which explicitly incorporates the modeling abilities of continuous dynamics of Neural ODEs with the attention mechanism of Transformers. We mathematically characterize the expressive power of ContiFormer and illustrate that, by curated designs of function hypothesis, many Transformer variants specialized in irregular time series modeling can be covered as a special case of ContiFormer. A wide range of experiments on both synthetic and real-world datasets have illustrated the superior modeling capacities and prediction performance of ContiFormer on irregular time series data. The project link is https://seqml.github.io/contiformer/.

4. NCDEs

   Learnable Path in Neural Controlled Differential Equations

   Sheo Yon Jhin, Minju Jo, Seungji Kook, and 1 more author

   Proceedings of the AAAI Conference on Artificial Intelligence, Jun 2023

   Abs DOI Paper arXiv GitHub

   Neural Controlled Differential Equations (NCDEs) have shown great promise in modeling irregularly sampled time series data. A key component of NCDEs is the path, which typically relies on linear or cubic spline interpolation of the input data. However, these fixed interpolation schemes may not be optimal for capturing complex underlying dynamics. In this paper, we propose a novel approach that introduces a ’learnable path’ within NCDEs. Our method allows the model to dynamically learn the most suitable interpolation scheme for the given data, thereby enhancing its expressiveness and adaptability. We demonstrate the effectiveness of our learnable path NCDE on various real-world datasets, showing improved performance in tasks such as time-series classification and forecasting, especially in scenarios with highly irregular observations.

5. NODEs

   Neural Lad: A Neural Latent Dynamics Framework for Times Series Modeling

   ting li, Jianguo Li, and Zhanxing Zhu

   In Advances in Neural Information Processing Systems, Jun 2023

   Abs Paper

   Modeling complex time series data, especially those with irregular sampling and missing values, remains a significant challenge. Traditional methods often struggle with capturing underlying continuous dynamics and long-term dependencies. We propose Neural LAD (Neural Latent Dynamics), a novel framework that leverages latent ordinary differential equations (ODEs) to model the continuous evolution of time series in a low-dimensional latent space. Neural LAD learns a flexible, data-driven latent dynamics model that can handle irregular observations and infer missing values. We demonstrate the effectiveness of Neural LAD on various real-world datasets, including medical records and human activity recognition, showing superior performance in tasks such as imputation, forecasting, and classification compared to state-of-the-art methods.

6. NODEs

   Long-term Time Series Forecasting based on Decomposition and Neural Ordinary Differential Equations

   S. Lim, J. Park, S. Kim, and 5 more authors

   In 2023 IEEE International Conference on Big Data (BigData), Dec 2023

   Abs DOI Paper arXiv

   Long-term time series forecasting (LTSF) is a challenging task that has been investigated in various domains such as finance investment, health care, traffic, and weather forecasting. In recent years, Linear-based LTSF models showed better performance, pointing out the problem of Transformer-based approaches causing temporal information loss. However, Linear-based approach has also limitations that the model is too simple to comprehensively exploit the characteristics of the dataset. To solve these limitations, we propose LTSF-DNODE, which applies a model based on linear ordinary differential equations (ODEs) and a time series decomposition method according to data statistical characteristics. We show that LTSF-DNODE outperforms the baselines on various real-world datasets. In addition, for each dataset, we explore the impacts of regularization in the neural ordinary differential equation (NODE) framework.

2022

1. NODEs

   Closed-form continuous-time neural networks

   Ramin Hasani, Mathias Lechner, Alexander Amini, and 5 more authors

   Nature Machine Intelligence, Dec 2022

   Abs DOI Paper

   Continuous-time neural networks are a class of machine learning systems that can tackle representation learning on spatiotemporal decision-making tasks. These models are typically represented by continuous differential equations. However, their expressive power when they are deployed on computers is bottlenecked by numerical differential equation solvers. This limitation has notably slowed down the scaling and understanding of numerous natural physical phenomena such as the dynamics of nervous systems. Ideally, we would circumvent this bottleneck by solving the given dynamical system in closed form. This is known to be intractable in general. Here, we show that it is possible to closely approximate the interaction between neurons and synapses—the building blocks of natural and artificial neural networks—constructed by liquid time-constant networks efficiently in closed form. To this end, we compute a tightly bounded approximation of the solution of an integral appearing in liquid time-constant dynamics that has had no known closed-form solution so far. This closed-form solution impacts the design of continuous-time and continuous-depth neural models. For instance, since time appears explicitly in closed form, the formulation relaxes the need for complex numerical solvers. Consequently, we obtain models that are between one and five orders of magnitude faster in training and inference compared with differential equation-based counterparts. More importantly, in contrast to ordinary differential equation-based continuous networks, closed-form networks can scale remarkably well compared with other deep learning instances. Lastly, as these models are derived from liquid networks, they show good performance in time-series modelling compared with advanced recurrent neural network models.

2. NSDEs

   Neural Laplace: Learning diverse classes of differential equations in the Laplace domain

   Samuel I Holt, Zhaozhi Qian, and Mihaela Schaar

   In Proceedings of the 39th International Conference on Machine Learning, Dec 2022

   Abs Paper

   Neural Ordinary Differential Equations model dynamical systems with ODEs learned by neural networks. However, ODEs are fundamentally inadequate to model systems with long-range dependencies or discontinuities, which are common in engineering and biological systems. Broader classes of differential equations (DE) have been proposed as remedies, including delay differential equations and integro-differential equations. Furthermore, Neural ODE suffers from numerical instability when modelling stiff ODEs and ODEs with piecewise forcing functions. In this work, we propose Neural Laplace, a unifying framework for learning diverse classes of DEs including all the aforementioned ones. Instead of modelling the dynamics in the time domain, we model it in the Laplace domain, where the history-dependencies and discontinuities in time can be represented as summations of complex exponentials. To make learning more efficient, we use the geometrical stereographic map of a Riemann sphere to induce more smoothness in the Laplace domain. In the experiments, Neural Laplace shows superior performance in modelling and extrapolating the trajectories of diverse classes of DEs, including the ones with complex history dependency and abrupt changes.

3. NSDEs

   Modeling Irregular Time Series with Continuous Recurrent Units

   Mona Schirmer, Mazin Eltayeb, Stefan Lessmann, and 1 more author

   In Proceedings of the 39th International Conference on Machine Learning, Dec 2022

   Abs Paper

   Recurrent neural networks (RNNs) are a popular choice for modeling sequential data. Modern RNN architectures assume constant time-intervals between observations. However, in many datasets (e.g. medical records) observation times are irregular and can carry important information. To address this challenge, we propose continuous recurrent units (CRUs) – a neural architecture that can naturally handle irregular intervals between observations. The CRU assumes a hidden state, which evolves according to a linear stochastic differential equation and is integrated into an encoder-decoder framework. The recursive computations of the CRU can be derived using the continuous-discrete Kalman filter and are in closed form. The resulting recurrent architecture has temporal continuity between hidden states and a gating mechanism that can optimally integrate noisy observations. We derive an efficient parameterization scheme for the CRU that leads to a fast implementation f-CRU. We empirically study the CRU on a number of challenging datasets and find that it can interpolate irregular time series better than methods based on neural ordinary differential equations.

4. NCDEs

   EXIT: Extrapolation and Interpolation-based Neural Controlled Differential Equations for Time-series Classification and Forecasting.

   Sheo Yon Jhin, Jaehoon Lee, Minju Jo, and 5 more authors

   In WWW ’22: The ACM Web Conference 2022, Virtual Event, Lyon, France, April 25 - 29, 2022, Dec 2022

   Abs DOI Paper arXiv GitHub

   Neural Controlled Differential Equations (NCDEs) have emerged as a powerful tool for modeling irregularly sampled time series data. However, existing NCDE models often struggle with extrapolation and interpolation tasks, which are crucial for real-world applications like time-series classification and forecasting. To address these limitations, we propose EXIT (Extrapolation and Interpolation-based Neural Controlled Differential Equations), a novel framework that enhances NCDEs’ capabilities in handling both extrapolation and interpolation. EXIT introduces a dual-path architecture that explicitly models both the observed data points and the underlying continuous dynamics, allowing for more robust and accurate predictions. We demonstrate the effectiveness of EXIT on various real-world datasets, showing significant improvements over state-of-the-art NCDE models in time-series classification and forecasting tasks, especially in scenarios with high irregularity and missing data.

5. NCDEs

   On the Choice of Interpolation Scheme for Neural CDEs

   James Morrill, Patrick Kidger, Lingyi Yang, and 1 more author

   Transactions on Machine Learning Research, Dec 2022

   Abs Paper

   Neural Controlled Differential Equations (Neural CDEs) are a powerful model for irregularly-sampled time series. A key component of Neural CDEs is the choice of interpolation scheme for the input data. Common choices include linear interpolation and cubic spline interpolation. We investigate the impact of different interpolation schemes on the performance of Neural CDEs. We show that the choice of interpolation scheme can significantly affect the model’s ability to capture complex dynamics and generalize to unseen data. We provide theoretical insights and empirical evidence on various real-world datasets, demonstrating that a carefully chosen interpolation scheme can lead to substantial performance improvements.

6. Review

   On Neural Differential Equations

   Patrick Kidger

   Feb 2022

   Abs DOI Paper arXiv

   The conjoining of dynamical systems and deep learning has become a topic of great interest. In particular, neural differential equations (NDEs) demonstrate that neural networks and differential equation are two sides of the same coin. Traditional parameterised differential equations are a special case. Many popular neural network architectures, such as residual networks and recurrent networks, are discretisations. NDEs are suitable for tackling generative problems, dynamical systems, and time series (particularly in physics, finance, ...) and are thus of interest to both modern machine learning and traditional mathematical modelling. NDEs offer high-capacity function approximation, strong priors on model space, the ability to handle irregular data, memory efficiency, and a wealth of available theory on both sides. This doctoral thesis provides an in-depth survey of the field. Topics include: neural ordinary differential equations (e.g. for hybrid neural/mechanistic modelling of physical systems); neural controlled differential equations (e.g. for learning functions of irregular time series); and neural stochastic differential equations (e.g. to produce generative models capable of representing complex stochastic dynamics, or sampling from complex high-dimensional distributions). Further topics include: numerical methods for NDEs (e.g. reversible differential equations solvers, backpropagation through differential equations, Brownian reconstruction); symbolic regression for dynamical systems (e.g. via regularised evolution); and deep implicit models (e.g. deep equilibrium models, differentiable optimisation). We anticipate this thesis will be of interest to anyone interested in the marriage of deep learning with dynamical systems, and hope it will provide a useful reference for the current state of the art.

7. NCDEs

   Graph Neural Controlled Differential Equations for Traffic Forecasting

   Jeongwhan Choi, Hwangyong Choi, Jeehyun Hwang, and 1 more author

   Proceedings of the AAAI Conference on Artificial Intelligence, Jun 2022

   Abs DOI Paper arXiv GitHub

   Traffic forecasting is one of the most popular spatio-temporal tasks in the field of machine learning. A prevalent approach in the field is to combine graph convolutional networks and recurrent neural networks for the spatio-temporal processing. There has been fierce competition and many novel methods have been proposed. In this paper, we present the method of spatio-temporal graph neural controlled differential equation (STG-NCDE). Neural controlled differential equations (NCDEs) are a breakthrough concept for processing sequential data. We extend the concept and design two NCDEs: one for the temporal processing and the other for the spatial processing. After that, we combine them into a single framework. We conduct experiments with 6 benchmark datasets and 20 baselines. STG-NCDE shows the best accuracy in all cases, outperforming all those 20 baselines by non-trivial margins.

2021

1. NODEs

   Liquid Time-constant Networks

   Ramin Hasani, Mathias Lechner, Alexander Amini, and 2 more authors

   Proceedings of the AAAI Conference on Artificial Intelligence, Jun 2021

   Abs DOI Paper

   We introduce a new class of time-continuous recurrent neural network models. Instead of declaring a learning system’s dynamics by implicit nonlinearities, we construct networks of linear first-order dynamical systems modulated via nonlinear interlinked gates. The resulting models represent dynamical systems with varying (i.e., liquid) time-constants coupled to their hidden state, with outputs being computed by numerical differential equation solvers. These neural networks exhibit stable and bounded behavior, yield superior expressivity within the family of neural ordinary differential equations, and give rise to improved performance on time-series prediction tasks. To demonstrate these properties, we first take a theoretical approach to find bounds over their dynamics, and compute their expressive power by the trajectory length measure in a latent trajectory space. We then conduct a series of time-series prediction experiments to manifest the approximation capability of Liquid Time-Constant Networks (LTCs) compared to classical and modern RNNs.

2. NCDEs

   Neural Rough Differential Equations for Long Time Series.

   James Morrill, Cristopher Salvi, Patrick Kidger, and 1 more author

   In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event., Jun 2021

   Abs Paper arXiv GitHub

   Neural Controlled Differential Equations (CDEs) are a powerful model for irregularly-sampled time series. However, their computational cost scales quadratically with the length of the time series, making them impractical for long sequences. We propose Neural Rough Differential Equations (RDEs), a new model that leverages the theory of rough paths to reduce the computational cost of CDEs to linear in the length of the time series. Our model retains the expressiveness of CDEs while being significantly more efficient. We demonstrate the effectiveness of our model on several real-world datasets, including human motion capture data and medical records. Our model outperforms existing methods for long irregularly-sampled time series.

3. Related

   Score-Based Generative Modeling through Stochastic Differential Equations.

   Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, and 3 more authors

   In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, Jun 2021

   Abs Paper arXiv GitHub

   We propose a unified framework for generative modeling that leverages stochastic differential equations (SDEs) to transform a simple prior distribution into a complex data distribution. Our framework encompasses various existing generative models, including score-based generative models and denoising diffusion probabilistic models, as special cases. By defining a continuous-time SDE that gradually perturbs data towards noise, and its reverse-time SDE that transforms noise back into data, we can generate samples by solving the reverse SDE. We demonstrate the effectiveness of our framework on various tasks, including image generation and unconditional sampling, showing competitive performance with state-of-the-art generative models.

4. Related

   Universal Differential Equations for Scientific Machine Learning

   Christopher Rackauckas, Yingbo Ma, Julius Martensen, and 6 more authors

   Nov 2021

   Abs DOI Paper arXiv

   In the context of science, the well-known adage "a picture is worth a thousand words" might well be "a model is worth a thousand datasets." In this manuscript we introduce the SciML software ecosystem as a tool for mixing the information of physical laws and scientific models with data-driven machine learning approaches. We describe a mathematical object, which we denote universal differential equations (UDEs), as the unifying framework connecting the ecosystem. We show how a wide variety of applications, from automatically discovering biological mechanisms to solving high-dimensional Hamilton-Jacobi-Bellman equations, can be phrased and efficiently handled through the UDE formalism and its tooling. We demonstrate the generality of the software tooling to handle stochasticity, delays, and implicit constraints. This funnels the wide variety of SciML applications into a core set of training mechanisms which are highly optimized, stabilized for stiff equations, and compatible with distributed parallelism and GPU accelerators.

5. Related

   Neural Delay Differential Equations.

   Qunxi Zhu, Yao Guo, and Wei Lin

   In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, Nov 2021

   Abs Paper arXiv

   Neural Ordinary Differential Equations (NODEs), a framework of continuous-depth neural networks, have been widely applied, showing exceptional efficacy in coping with some representative datasets. Recently, an augmented framework has been successfully developed for conquering some limitations emergent in application of the original framework. Here we propose a new class of continuous-depth neural networks with delay, named as Neural Delay Differential Equations (NDDEs), and, for computing the corresponding gradients, we use the adjoint sensitivity method to obtain the delayed dynamics of the adjoint. Since the differential equations with delays are usually seen as dynamical systems of infinite dimension possessing more fruitful dynamics, the NDDEs, compared to the NODEs, own a stronger capacity of nonlinear representations. Indeed, we analytically validate that the NDDEs are of universal approximators, and further articulate an extension of the NDDEs, where the initial function of the NDDEs is supposed to satisfy ODEs. More importantly, we use several illustrative examples to demonstrate the outstanding capacities of the NDDEs and the NDDEs with ODEs’ initial value. More precisely, (1) we successfully model the delayed dynamics where the trajectories in the lower-dimensional phase space could be mutually intersected, while the traditional NODEs without any argumentation are not directly applicable for such modeling, and (2) we achieve lower loss and higher accuracy not only for the data produced synthetically by complex models but also for the real-world image datasets, i.e., CIFAR10, MNIST and SVHN. Our results on the NDDEs reveal that appropriately articulating the elements of dynamical systems into the network design is truly beneficial to promoting the network performance.

6. Related

   Neural Flows: Efficient Alternative to Neural ODEs

   Marin Biloš, Johanna Sommer, Syama Sundar Rangapuram, and 2 more authors

   In Advances in Neural Information Processing Systems, Nov 2021

   Abs Paper arXiv GitHub

   Neural ordinary differential equations describe how values change in time. This is the reason why they gained importance in modeling sequential data, especially when the observations are made at irregular intervals. In this paper we propose an alternative by directly modeling the solution curves - the flow of an ODE - with a neural network. This immediately eliminates the need for expensive numerical solvers while still maintaining the modeling capability of neural ODEs. We propose several flow architectures suitable for different applications by establishing precise conditions on when a function defines a valid flow. Apart from computational efficiency, we also provide empirical evidence of favorable generalization performance via applications in time series modeling, forecasting, and density estimation.

7. Related

   Normalizing Flows for Probabilistic Modeling and Inference

   George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, and 2 more authors

   Journal of Machine Learning Research, Nov 2021

   Abs Paper arXiv

   Normalizing flows provide a general mechanism for defining expressive probability distributions, only requiring the specification of a (usually simple) base distribution and a series of bijective transformations. There has been much recent work on normalizing flows, ranging from improving their expressive power to expanding their application. We believe the field has now matured and is in need of a unified perspective. In this review, we attempt to provide such a perspective by describing flows through the lens of probabilistic modeling and inference. We place special emphasis on the fundamental principles of flow design, and discuss foundational topics such as expressive power and computational trade-offs. We also broaden the conceptual framing of flows by relating them to more general probability transformations. Lastly, we summarize the use of flows for tasks such as generative modeling, approximate inference, and supervised learning.

2020

1. NODEs

   How to Train Your Neural ODE: the World of Jacobian and Kinetic Regularization

   Chris Finlay, Joern-Henrik Jacobsen, Levon Nurbekyan, and 1 more author

   In Proceedings of the 37th International Conference on Machine Learning, Nov 2020

   Abs Paper

   Training neural ODEs on large datasets has not been tractable due to the necessity of allowing the adaptive numerical ODE solver to refine its step size to very small values. In practice this leads to dynamics equivalent to many hundreds or even thousands of layers. In this paper, we overcome this apparent difficulty by introducing a theoretically-grounded combination of both optimal transport and stability regularizations which encourage neural ODEs to prefer simpler dynamics out of all the dynamics that solve a problem well. Simpler dynamics lead to faster convergence and to fewer discretizations of the solver, considerably decreasing wall-clock time without loss in performance. Our approach allows us to train neural ODE-based generative models to the same performance as the unregularized dynamics, with significant reductions in training time. This brings neural ODEs closer to practical relevance in large-scale applications.

2. NCDEs

   Neural Controlled Differential Equations for Irregular Time Series.

   Patrick Kidger, James Morrill, James Foster, and 1 more author

   In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual., Nov 2020

   Abs Paper arXiv GitHub

   Many real-world time series are irregularly sampled, with observations arriving at arbitrary time points. Standard recurrent neural networks are not well-suited to this setting, as they assume fixed time intervals between observations. We propose a new model for irregularly-sampled time series based on Neural Controlled Differential Equations (CDEs). Our model learns the continuous-time dynamics of the underlying system, and can be evaluated at arbitrary time points. We demonstrate the effectiveness of our model on several real-world datasets, including medical records and human motion capture data. Our model outperforms state-of-the-art methods for irregularly-sampled time series, and is competitive with methods for regularly-sampled data.

3. NSDEs

   Scalable Gradients for Stochastic Differential Equations

   Xuechen Li, Ting-Kam Leonard Wong, Ricky T. Q. Chen, and 1 more author

   In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, Aug 2020

   Abs Paper arXiv GitHub

   Stochastic differential equations (SDEs) are a powerful tool for modeling systems with inherent noise. However, training models that involve SDEs can be challenging due to the difficulty of computing gradients through stochastic processes. We propose a new method for computing gradients of expectations with respect to SDEs that is both scalable and unbiased. Our method leverages the adjoint method, which allows us to compute gradients by solving a backward SDE. We demonstrate the effectiveness of our method on several tasks, including training generative models and learning latent SDEs from data. Our method outperforms existing approaches in terms of both accuracy and computational efficiency.

4. NODEs

   Learning Long-Term Dependencies in Irregularly-Sampled Time Series

   Mathias Lechner and Ramin Hasani

   Dec 2020

   Abs DOI Paper arXiv GitHub

   Many real-world time series are irregularly sampled, with observations arriving at arbitrary time points. Standard recurrent neural networks struggle with such data due to their reliance on fixed time steps. We propose a novel approach for learning long-term dependencies in irregularly-sampled time series by combining the strengths of recurrent neural networks with continuous-time models. Our method leverages a continuous-time hidden state that evolves according to a learned ordinary differential equation, allowing it to capture complex temporal dynamics regardless of sampling irregularity. We demonstrate the effectiveness of our model on several real-world datasets, including medical records and human motion capture data, showing improved performance in tasks such as classification and forecasting.

5. NODEs

   STEER : Simple Temporal Regularization For Neural ODE

   Arnab Ghosh, Harkirat Behl, Emilien Dupont, and 2 more authors

   In Advances in Neural Information Processing Systems, Dec 2020

   Abs Paper arXiv

   Neural Ordinary Differential Equations (Neural ODEs) have emerged as a powerful tool for modeling continuous-time dynamics. However, training Neural ODEs can be challenging due to their sensitivity to initial conditions and the lack of explicit regularization for temporal smoothness. We propose STEER (Simple Temporal Regularization For Neural ODE), a novel regularization technique that encourages temporal smoothness in the learned dynamics of Neural ODEs. STEER adds a penalty term to the loss function that regularizes the magnitude of the derivative of the hidden state, promoting more stable and interpretable dynamics. We demonstrate the effectiveness of STEER on various tasks, including time series modeling and density estimation, showing improved performance and robustness compared to standard Neural ODEs.

6. NODEs

   Adaptive Checkpoint Adjoint Method for Gradient Estimation in Neural ODE

   Juntang Zhuang, Nicha Dvornek, Xiaoxiao Li, and 3 more authors

   In Proceedings of the 37th International Conference on Machine Learning, Jul 2020

   Abs Paper arXiv

   Neural Ordinary Differential Equations (Neural ODEs) offer a memory-efficient way to model continuous-time dynamics by backpropagating through an ODE solver using the adjoint method. However, the adjoint method can be computationally expensive, especially for long integration times, as it requires solving an additional backward ODE. We propose the Adaptive Checkpoint Adjoint Method (ACAM), a novel approach that combines checkpointing with the adjoint method to reduce memory and computational costs. ACAM adaptively selects checkpoints during the forward pass, allowing for efficient gradient computation without storing the entire trajectory. We demonstrate the effectiveness of ACAM on various tasks, showing significant reductions in memory and computation while maintaining accuracy compared to standard adjoint methods.

7. NODEs

   Approximation Capabilities of Neural ODEs and Invertible Residual Networks

   Han Zhang, Xi Gao, Jacob Unterman, and 1 more author

   In Proceedings of the 37th International Conference on Machine Learning, Jul 2020

   Abs Paper arXiv

   Neural Ordinary Differential Equations (Neural ODEs) and Invertible Residual Networks (i-ResNets) are two recent architectures that offer memory-efficient training and exact likelihood computation, respectively. Both models are based on continuous-depth transformations. We investigate the approximation capabilities of Neural ODEs and i-ResNets, showing that they are universal approximators of continuous functions. We provide theoretical guarantees on their ability to approximate arbitrary continuous functions and demonstrate their practical performance on various tasks, including density estimation and image classification. Our results shed light on the expressive power of these continuous-depth models and their potential for deep learning applications.

8. NODEs

   Universal Approximation Property of Neural Ordinary Differential Equations

   Takeshi Teshima, Koichi Tojo, Masahiro Ikeda, and 2 more authors

   Dec 2020

   Abs DOI Paper arXiv

   Neural Ordinary Differential Equations (Neural ODEs) have recently emerged as a new class of deep learning models that parameterize the derivative of the hidden state using a neural network. This allows them to model continuous-time dynamics and offer memory-efficient training. We prove that Neural ODEs possess the universal approximation property, meaning they can approximate any continuous function to arbitrary accuracy. Our proof provides theoretical guarantees on the expressive power of Neural ODEs and sheds light on their ability to learn complex mappings. This result further solidifies the theoretical foundation of Neural ODEs and supports their wide applicability in various domains.

9. NODEs

   Neural Manifold Ordinary Differential Equations

   Aaron Lou, Derek Lim, Isay Katsman, and 4 more authors

   In Advances in Neural Information Processing Systems, Dec 2020

   Abs Paper arXiv GitHub

   Neural Ordinary Differential Equations (Neural ODEs) have shown promise in modeling continuous-time dynamics. However, they operate in Euclidean space, which may not be suitable for data residing on non-Euclidean manifolds. We introduce Neural Manifold Ordinary Differential Equations (Neural Manifold ODEs), a novel framework that extends Neural ODEs to learn dynamics on Riemannian manifolds. Our approach leverages geometric deep learning techniques to define ODEs on curved spaces, allowing for more natural and expressive modeling of data with underlying manifold structures. We demonstrate the effectiveness of Neural Manifold ODEs on tasks such as learning dynamics on spheres and hyperbolic spaces, showing improved performance and interpretability compared to Euclidean counterparts.

10. NODEs

    Dissecting Neural ODEs

    Stefano Massaroli, Michael Poli, Jinkyoo Park, and 2 more authors

    In Advances in Neural Information Processing Systems, Dec 2020

    Abs Paper arXiv

    Neural Ordinary Differential Equations (Neural ODEs) have gained significant attention for their ability to model continuous-time dynamics and offer memory-efficient training. However, their internal workings and the factors influencing their performance are not yet fully understood. We conduct a thorough dissection of Neural ODEs, analyzing various aspects such as their approximation capabilities, sensitivity to hyperparameters, and the role of the ODE solver. Through extensive empirical studies and theoretical insights, we provide a deeper understanding of why Neural ODEs work and how to effectively train them. Our findings offer practical guidelines for designing and applying Neural ODEs in various machine learning tasks.

11. Related

    Learning Differential Equations that are Easy to Solve

    Jacob Kelly, Jesse Bettencourt, Matthew J Johnson, and 1 more author

    In Advances in Neural Information Processing Systems, Dec 2020

    Abs Paper arXiv GitHub

    Differential equations parameterized by neural networks become expensive to solve numerically as training progresses. We propose a remedy that encourages learned dynamics to be easier to solve. Specifically, we introduce a differentiable surrogate for the time cost of standard numerical solvers, using higher-order derivatives of solution trajectories. These derivatives are efficient to compute with Taylor-mode automatic differentiation. Optimizing this additional objective trades model performance against the time cost of solving the learned dynamics. We demonstrate our approach by training substantially faster, while nearly as accurate, models in supervised classification, density estimation, and time-series modelling tasks.

2019

1. NODEs

   Latent Ordinary Differential Equations for Irregularly-Sampled Time Series.

   Yulia Rubanova, Tian Qi Chen, and David Duvenaud

   In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada., Dec 2019

   Abs Paper arXiv GitHub

   Many real-world time series are irregularly-sampled, with observations arriving at arbitrary time points. Standard recurrent neural networks are not well-suited to this setting, as they assume fixed time intervals between observations. We propose a new model for irregularly-sampled time series based on Neural Ordinary Differential Equations (ODEs). Our model learns the continuous-time dynamics of the underlying system, and can be evaluated at arbitrary time points. We demonstrate the effectiveness of our model on several real-world datasets, including medical records and human motion capture data. Our model outperforms state-of-the-art methods for irregularly-sampled time series, and is competitive with methods for regularly-sampled data.

2. NODEs

   Augmented Neural ODEs.

   Emilien Dupont, Arnaud Doucet, and Yee Whye Teh

   In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada., Dec 2019

   Abs Paper arXiv GitHub

   Neural Ordinary Differential Equations (Neural ODEs) parameterize the derivative of the hidden state using a neural network. This allows them to model continuous-time dynamics and handle irregularly sampled data. However, Neural ODEs are limited by the fact that the hidden state must have the same dimension as the input. We propose Augmented Neural ODEs (ANODEs), which augment the hidden state with additional dimensions. This allows ANODEs to learn more complex dynamics and improve performance on tasks such as density estimation and time series modeling. We demonstrate the effectiveness of ANODEs on several real-world datasets, showing significant improvements over standard Neural ODEs.

3. NODEs

   GRU-ODE-Bayes: Continuous Modeling of Sporadically-Observed Time Series.

   Edward De Brouwer, Jaak Simm, Adam Arany, and 1 more author

   In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada., Dec 2019

   Abs Paper arXiv GitHub

   Many real-world time series are sporadically observed, meaning that observations arrive at irregular time intervals and some values may be missing. Standard recurrent neural networks struggle with such data due to their reliance on fixed time steps. We propose GRU-ODE-Bayes, a novel model that combines the strengths of Gated Recurrent Units (GRUs) with Ordinary Differential Equations (ODEs) and Bayesian inference to handle sporadically observed time series. Our model learns continuous-time dynamics and provides uncertainty estimates for predictions. We demonstrate the effectiveness of GRU-ODE-Bayes on several real-world datasets, including medical records and climate data, showing significant improvements over state-of-the-art methods for irregularly sampled time series.

4. NSDEs

   Neural Stochastic Differential Equations: Deep Latent Gaussian Models in the Diffusion Limit

   Belinda Tzen and Maxim Raginsky

   Oct 2019

   Abs DOI Paper arXiv

   We introduce Neural Stochastic Differential Equations (Neural SDEs), a new class of deep generative models that extend Neural Ordinary Differential Equations (Neural ODEs) by incorporating a stochastic diffusion term. Neural SDEs model continuous-time dynamics with inherent noise, allowing for more robust and expressive representations of complex data. We show that Neural SDEs can be viewed as deep latent Gaussian models in the diffusion limit, providing a theoretical foundation for their generative capabilities. We demonstrate the effectiveness of Neural SDEs on tasks such as density estimation and time series modeling, showing competitive performance with state-of-the-art methods while offering improved flexibility and interpretability.

5. NODEs

   ODE2VAE: Deep generative second order ODEs with Bayesian neural networks

   Cagatay Yildiz, Markus Heinonen, and Harri Lahdesmaki

   In Advances in Neural Information Processing Systems, Oct 2019

   Abs Paper arXiv GitHub

   Deep generative models have achieved remarkable success in various domains, but modeling complex temporal dynamics, especially with irregularly sampled data, remains a challenge. We propose ODE2VAE, a novel deep generative model that combines second-order Ordinary Differential Equations (ODEs) with Variational Autoencoders (VAEs) and Bayesian neural networks. Our model learns continuous-time dynamics in the latent space, allowing it to handle irregular observations and capture complex temporal dependencies. The use of second-order ODEs enables modeling of acceleration and momentum, providing richer dynamics. We demonstrate the effectiveness of ODE2VAE on several real-world datasets, including medical records and motion capture data, showing improved performance in tasks such as imputation, forecasting, and anomaly detection.

6. NSDEs

   Neural SDE: Stabilizing Neural ODE Networks with Stochastic Noise

   Xuanqing Liu, Tesi Xiao, Si Si, and 3 more authors

   Oct 2019

   Abs DOI Paper arXiv

   Neural Ordinary Differential Equations (Neural ODEs) have gained popularity for modeling continuous-time dynamics. However, they can be sensitive to noise and prone to instability, especially in high-dimensional settings. We propose Neural SDE, a novel approach that stabilizes Neural ODE networks by introducing stochastic noise into the dynamics. By augmenting the deterministic ODE with a learnable diffusion term, Neural SDE can better capture uncertainties and improve robustness. We demonstrate the effectiveness of Neural SDE on various tasks, including density estimation and time series modeling, showing improved stability and performance compared to standard Neural ODEs, particularly in the presence of noisy data.

7. NSDEs

   Neural Jump Stochastic Differential Equations

   Junteng Jia and Austin R Benson

   In Advances in Neural Information Processing Systems, Oct 2019

   Abs Paper arXiv

   Stochastic differential equations (SDEs) are widely used to model systems with continuous-time random fluctuations. However, many real-world phenomena exhibit sudden, discrete changes or jumps, which are not captured by standard SDEs. We introduce Neural Jump Stochastic Differential Equations (Neural JSDEs), a novel class of models that extend Neural SDEs by incorporating jump processes. Neural JSDEs can model both continuous diffusion and discrete jumps, providing a more comprehensive framework for complex temporal dynamics. We demonstrate the effectiveness of Neural JSDEs on various tasks, including financial modeling and anomaly detection, showing improved performance in scenarios with abrupt changes.

8. Related

   Invertible Residual Networks

   Jens Behrmann, Will Grathwohl, Ricky T. Q. Chen, and 2 more authors

   In Proceedings of the 36th International Conference on Machine Learning, Oct 2019

   Abs Paper arXiv GitHub

   We introduce Invertible Residual Networks (i-ResNets), a new class of deep neural networks that are invertible by construction. This property allows for exact likelihood computation and memory-efficient training, as activations do not need to be stored for backpropagation. i-ResNets are built upon the idea of residual blocks, where each block is designed to be invertible. We demonstrate the effectiveness of i-ResNets on various tasks, including image classification and density estimation, showing competitive performance with state-of-the-art models while offering significant memory savings. Our work opens up new possibilities for designing deep architectures with desirable properties.

9. Related

   FFJORD: Free-Form Continuous Dynamics for Scalable Reversible Generative Models.

   Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, and 2 more authors

   In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, Oct 2019

   Abs Paper arXiv GitHub

   Normalizing flows are a powerful class of generative models that allow for exact likelihood computation. However, existing normalizing flows often suffer from limitations in expressiveness or scalability. We introduce FFJORD (Free-Form Jacobian of Reversible Dynamics), a novel normalizing flow model based on Neural Ordinary Differential Equations (Neural ODEs). FFJORD parameterizes the transformation as a continuous-time dynamic system, allowing for flexible and expressive transformations while maintaining invertibility and exact likelihood computation. We demonstrate the scalability and effectiveness of FFJORD on various tasks, including density estimation and image generation, showing competitive performance with state-of-the-art generative models.

10. NODEs

    ANODE: Unconditionally Accurate Memory-Efficient Gradients for Neural ODEs.

    Amir Gholaminejad, Kurt Keutzer, and George Biros

    In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, Oct 2019

    Abs DOI Paper arXiv

    Neural Ordinary Differential Equations (Neural ODEs) offer a memory-efficient way to compute gradients using the adjoint method. However, the adjoint method can be computationally expensive and numerically unstable for long integration times. We propose ANODE (Unconditionally Accurate Memory-Efficient Gradients for Neural ODEs), a novel method for computing gradients in Neural ODEs that is both memory-efficient and numerically stable. ANODE leverages a new adjoint formulation that avoids the need for backward integration, leading to significant computational savings and improved stability. We demonstrate the effectiveness of ANODE on various tasks, showing competitive performance with existing methods while offering superior efficiency and robustness.

2018

1. NODEs

   Neural Ordinary Differential Equations.

   Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and 1 more author

   In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada., Oct 2018

   Abs Paper arXiv GitHub

   We introduce a new family of deep neural network models. Instead of specifying a discrete sequence of hidden layers, we parameterize the derivative of the hidden state using a neural network. The output of the network is computed by an off-the-shelf ODE solver. These ’Neural ODEs’ can model continuous-time dynamics, and can be trained with backpropagation through the ODE solver. This allows for memory-efficient training, as the adjoint method can be used to compute gradients without storing the entire forward pass. We demonstrate the effectiveness of Neural ODEs on various tasks, including density estimation, time series modeling, and image classification, showing competitive performance with state-of-the-art models while offering significant memory savings.

2. Related

   Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations

   Yiping Lu, Aoxiao Zhong, Quanzheng Li, and 1 more author

   In Proceedings of the 35th International Conference on Machine Learning, Oct 2018

   Abs Paper arXiv GitHub

   Deep neural networks have become the state-of-the-art models in numerous machine learning tasks. However, general guidance to network architecture design is still missing. In our work, we bridge deep neural network design with numerical differential equations. We show that many effective networks, such as ResNet, PolyNet, FractalNet and RevNet, can be interpreted as different numerical discretizations of differential equations. This finding brings us a brand new perspective on the design of effective deep architectures. We can take advantage of the rich knowledge in numerical analysis to guide us in designing new and potentially more effective deep networks. As an example, we propose a linear multi-step architecture (LM-architecture) which is inspired by the linear multi-step method solving ordinary differential equations. The LM-architecture is an effective structure that can be used on any ResNet-like networks. In particular, we demonstrate that LM-ResNet and LM-ResNeXt (i.e. the networks obtained by applying the LM-architecture on ResNet and ResNeXt respectively) can achieve noticeably higher accuracy than ResNet and ResNeXt on both CIFAR and ImageNet with comparable numbers of trainable parameters. In particular, on both CIFAR and ImageNet, LM-ResNet/LM-ResNeXt can significantly compress (>50%) the original networks while maintaining a similar performance. This can be explained mathematically using the concept of modified equation from numerical analysis. Last but not least, we also establish a connection between stochastic control and noise injection in the training process which helps to improve generalization of the networks. Furthermore, by relating stochastic training strategy with stochastic dynamic system, we can easily apply stochastic training to the networks with the LM-architecture. As an example, we introduced stochastic depth to LM-ResNet and achieve significant improvement over the original LM-ResNet on CIFAR10.

3. Related

   Stable architectures for deep neural networks

   Eldad Haber and Lars Ruthotto

   Inverse Problems, Jan 2018

   Abs DOI Paper arXiv

   Deep neural networks have become invaluable tools for supervised machine learning, e.g. classification of text or images. While often offering superior results over traditional techniques and successfully expressing complicated patterns in data, deep architectures are known to be challenging to design and train such that they generalize well to new data. Critical issues with deep architectures are numerical instabilities in derivative-based learning algorithms commonly called exploding or vanishing gradients. In this paper, we propose new forward propagation techniques inspired by systems of ordinary differential equations (ODE) that overcome this challenge and lead to well-posed learning problems for arbitrarily deep networks. The backbone of our approach is our interpretation of deep learning as a parameter estimation problem of nonlinear dynamical systems. Given this formulation, we analyze stability and well-posedness of deep learning and use this new understanding to develop new network architectures. We relate the exploding and vanishing gradient phenomenon to the stability of the discrete ODE and present several strategies for stabilizing deep learning for very deep networks. While our new architectures restrict the solution space, several numerical experiments show their competitiveness with state-of-the-art networks.

2017

1. Related

   Deep Learning-Based Numerical Methods for High-Dimensional Parabolic Partial Differential Equations and Backward Stochastic Differential Equations

   Weinan E, Jiequn Han, and Arnulf Jentzen

   Communications in Mathematics and Statistics, Dec 2017

   Abs DOI Paper arXiv

   We propose a deep learning-based numerical method for solving high-dimensional parabolic partial differential equations (PDEs) and backward stochastic differential equations (BSDEs). The method is based on approximating the unknown function by a deep neural network and training the network by minimizing a loss function derived from the PDE or BSDE. We demonstrate the effectiveness of our method on several high-dimensional examples, including the Black-Scholes equation and a high-dimensional Hamilton-Jacobi-Bellman equation. Our method outperforms traditional numerical methods in terms of accuracy and computational efficiency for high-dimensional problems.

2. Related

   Solving differential equations of fractional order using an optimization technique based on training artificial neural network

   M. Pakdaman, A. Ahmadian, S. Effati, and 2 more authors

   Applied Mathematics and Computation, Jan 2017

   Abs DOI Paper

   The current study aims to approximate the solution of fractional differential equations (FDEs) by using the fundamental properties of artificial neural networks (ANNs) for function approximation. In the first step, we derive an approximate solution of fractional differential equation (FDE) by using ANNs. In the second step, an optimization approach is exploited to adjust the weights of ANNs such that the approximated solution satisfies the FDE. Different types of FDEs including linear and nonlinear terms are solved to illustrate the ability of the method. In addition, the present scheme is compared with the analytical solution and a number of existing numerical techniques to show the efficiency of ANNs with high accuracy, fast convergence and low use of memory for solving the FDEs.