Papers

2025

1. NCDEs

   DualDynamics: Synergizing Implicit and Explicit Methods for Robust Irregular Time Series Analysis

   YongKyung Oh, Dong-Young Lim, and Sungil Kim

   In AAAI-25, Sponsored by the Association for the Advancement of Artificial Intelligence, February 25 - March 4, 2025, Philadelphia, PA, USA, 2025

   Abs DOI Paper arXiv GitHub

   Real-world time series analysis faces significant challenges when dealing with irregular and incomplete data. While Neural Differential Equation (NDE) based methods have shown promise, they struggle with limited expressiveness, scalability issues, and stability concerns. Conversely, Neural Flows offer stability but falter with irregular data. We introduce ’DualDynamics’, a novel framework that synergistically combines NDE-based method and Neural Flow-based method. This approach enhances expressive power while balancing computational demands, addressing critical limitations of existing techniques. We demonstrate DualDynamics’ effectiveness across diverse tasks: classification of robustness to dataset shift, irregularly-sampled series analysis, interpolation of missing data, and forecasting with partial observations. Our results show consistent outperformance over state-of-the-art methods, indicating DualDynamics’ potential to advance irregular time series analysis significantly.

2. Review

   Comprehensive Review of Neural Differential Equations for Time Series Analysis

   YongKyung Oh, Seungsu Kam, Jonghun Lee, and 3 more authors

   2025

   Abs DOI Paper arXiv

   Neural Differential Equations (NDEs) have emerged as a powerful paradigm for modeling complex dynamic systems, particularly in the context of time series analysis. This comprehensive review provides an in-depth exploration of the various NDE architectures, including Neural Ordinary Differential Equations (NODEs), Neural Controlled Differential Equations (NCDEs), and Neural Stochastic Differential Equations (NSDEs), and their applications in handling irregularly sampled, noisy, and high-dimensional time series data. We discuss the theoretical foundations, computational challenges, and practical considerations for implementing and training NDE-based models. Furthermore, we highlight recent advancements, open problems, and future research directions in this rapidly evolving field, aiming to provide a valuable resource for researchers and practitioners interested in leveraging NDEs for advanced time series analysis.

2024

1. NSDEs

   Stable Neural Stochastic Differential Equations in Analyzing Irregular Time Series Data

   YongKyung Oh, Dongyoung Lim, and Sungil Kim

   In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, 2024

   Abs Paper arXiv GitHub

   Irregular sampling intervals and missing values in real-world time series data present challenges for conventional methods that assume consistent intervals and complete data. Neural Ordinary Differential Equations (Neural ODEs) offer an alternative approach, utilizing neural networks combined with ODE solvers to learn continuous latent representations through parameterized vector fields. Neural Stochastic Differential Equations (Neural SDEs) extend Neural ODEs by incorporating a diffusion term, although this addition is not trivial, particularly when addressing irregular intervals and missing values. Consequently, careful design of drift and diffusion functions is crucial for maintaining stability and enhancing performance, while incautious choices can result in adverse properties such as the absence of strong solutions, stochastic destabilization, or unstable Euler discretizations, significantly affecting Neural SDEs’ performance. In this study, we propose three stable classes of Neural SDEs: Langevin-type SDE, Linear Noise SDE, and Geometric SDE. Then, we rigorously demonstrate their robustness in maintaining excellent performance under distribution shift, while effectively preventing overfitting. To assess the effectiveness of our approach, we conduct extensive experiments on four benchmark datasets for interpolation, forecasting, and classification tasks, and analyze the robustness of our methods with 30 public datasets under different missing rates. Our results demonstrate the efficacy of the proposed method in handling real-world irregular time series data.

2. NCDEs

   Attentive neural controlled differential equations for time-series classification and forecasting

   Sheo Yon Jhin, Heejoo Shin, Sujie Kim, and 7 more authors

   Knowledge and Information Systems, Mar 2024

   Abs DOI Paper arXiv GitHub

   Neural Controlled Differential Equations (NCDEs) have shown promising results in modeling irregularly sampled time series. However, their ability to capture long-range dependencies and focus on relevant information within the time series can be limited. We propose Attentive Neural Controlled Differential Equations (Attentive NCDEs), a novel framework that integrates attention mechanisms into NCDEs. This allows the model to dynamically weigh the importance of different time points and features, enhancing its ability to capture salient information for time-series classification and forecasting. We demonstrate the effectiveness of Attentive NCDEs on various real-world datasets, showing significant improvements over state-of-the-art NCDE models, especially in complex scenarios with varying data quality and length.

3. NSDEs

   Neural Jump-Diffusion Temporal Point Processes

   Shuai Zhang, Chuan Zhou, Yang Aron Liu, and 3 more authors

   In Proceedings of the 41st International Conference on Machine Learning, Mar 2024

   Abs Paper GitHub

   We present a novel perspective on temporal point processes (TPPs) by reformulating their intensity processes as solutions to stochastic differential equations (SDEs). In particular, we first prove the equivalent SDE formulations of several classical TPPs, including Poisson processes, Hawkes processes, and self-correcting processes. Based on these proofs, we introduce a unified TPP framework called Neural Jump-Diffusion Temporal Point Process (NJDTPP), whose intensity process is governed by a neural jump-diffusion SDE (NJDSDE) where the drift, diffusion, and jump coefficient functions are parameterized by neural networks. Compared to previous works, NJDTPP exhibits model flexibility in capturing intensity dynamics without relying on any specific functional form, and provides theoretical guarantees regarding the existence and uniqueness of the solution to the proposed NJDSDE. Experiments on both synthetic and real-world datasets demonstrate that NJDTPP is capable of capturing the dynamics of intensity processes in different scenarios and significantly outperforms the state-of-the-art TPP models in prediction tasks.

4. Related

   Mamba: Linear-Time Sequence Modeling with Selective State Spaces

   Albert Gu and Tri Dao

   May 2024

   Abs DOI Paper arXiv

   Foundation models, now powering most of the exciting applications in deep learning, are almost universally based on the Transformer architecture and its core attention module. Many subquadratic-time architectures such as linear attention, gated convolution and recurrent models, and structured state space models (SSMs) have been developed to address Transformers’ computational inefficiency on long sequences, but they have not performed as well as attention on important modalities such as language. We identify that a key weakness of such models is their inability to perform content-based reasoning, and make several improvements. First, simply letting the SSM parameters be functions of the input addresses their weakness with discrete modalities, allowing the model to selectively propagate or forget information along the sequence length dimension depending on the current token. Second, even though this change prevents the use of efficient convolutions, we design a hardware-aware parallel algorithm in recurrent mode. We integrate these selective SSMs into a simplified end-to-end neural network architecture without attention or even MLP blocks (Mamba). Mamba enjoys fast inference (5

2023

1. NCDEs

   Learnable Path in Neural Controlled Differential Equations

   Sheo Yon Jhin, Minju Jo, Seungji Kook, and 1 more author

   Proceedings of the AAAI Conference on Artificial Intelligence, Jun 2023

   Abs DOI Paper arXiv GitHub

   Neural Controlled Differential Equations (NCDEs) have shown great promise in modeling irregularly sampled time series data. A key component of NCDEs is the path, which typically relies on linear or cubic spline interpolation of the input data. However, these fixed interpolation schemes may not be optimal for capturing complex underlying dynamics. In this paper, we propose a novel approach that introduces a ’learnable path’ within NCDEs. Our method allows the model to dynamically learn the most suitable interpolation scheme for the given data, thereby enhancing its expressiveness and adaptability. We demonstrate the effectiveness of our learnable path NCDE on various real-world datasets, showing improved performance in tasks such as time-series classification and forecasting, especially in scenarios with highly irregular observations.

2. NCDEs

   Neural Lad: A Neural Latent Dynamics Framework for Times Series Modeling

   ting li, Jianguo Li, and Zhanxing Zhu

   In Advances in Neural Information Processing Systems, Jun 2023

   Abs Paper

   Modeling complex time series data, especially those with irregular sampling and missing values, remains a significant challenge. Traditional methods often struggle with capturing underlying continuous dynamics and long-term dependencies. We propose Neural LAD (Neural Latent Dynamics), a novel framework that leverages latent ordinary differential equations (ODEs) to model the continuous evolution of time series in a low-dimensional latent space. Neural LAD learns a flexible, data-driven latent dynamics model that can handle irregular observations and infer missing values. We demonstrate the effectiveness of Neural LAD on various real-world datasets, including medical records and human activity recognition, showing superior performance in tasks such as imputation, forecasting, and classification compared to state-of-the-art methods.

3. NODEs

   Long-term Time Series Forecasting based on Decomposition and Neural Ordinary Differential Equations

   S. Lim, J. Park, S. Kim, and 5 more authors

   In 2023 IEEE International Conference on Big Data (BigData), Dec 2023

   Abs DOI Paper arXiv

   Long-term time series forecasting (LTSF) is a challenging task that has been investigated in various domains such as finance investment, health care, traffic, and weather forecasting. In recent years, Linear-based LTSF models showed better performance, pointing out the problem of Transformer-based approaches causing temporal information loss. However, Linear-based approach has also limitations that the model is too simple to comprehensively exploit the characteristics of the dataset. To solve these limitations, we propose LTSF-DNODE, which applies a model based on linear ordinary differential equations (ODEs) and a time series decomposition method according to data statistical characteristics. We show that LTSF-DNODE outperforms the baselines on various real-world datasets. In addition, for each dataset, we explore the impacts of regularization in the neural ordinary differential equation (NODE) framework.

2022

1. NCDEs

   EXIT: Extrapolation and Interpolation-based Neural Controlled Differential Equations for Time-series Classification and Forecasting.

   Sheo Yon Jhin, Jaehoon Lee, Minju Jo, and 5 more authors

   In WWW ’22: The ACM Web Conference 2022, Virtual Event, Lyon, France, April 25 - 29, 2022, Dec 2022

   Abs DOI Paper arXiv GitHub

   Neural Controlled Differential Equations (NCDEs) have emerged as a powerful tool for modeling irregularly sampled time series data. However, existing NCDE models often struggle with extrapolation and interpolation tasks, which are crucial for real-world applications like time-series classification and forecasting. To address these limitations, we propose EXIT (Extrapolation and Interpolation-based Neural Controlled Differential Equations), a novel framework that enhances NCDEs’ capabilities in handling both extrapolation and interpolation. EXIT introduces a dual-path architecture that explicitly models both the observed data points and the underlying continuous dynamics, allowing for more robust and accurate predictions. We demonstrate the effectiveness of EXIT on various real-world datasets, showing significant improvements over state-of-the-art NCDE models in time-series classification and forecasting tasks, especially in scenarios with high irregularity and missing data.

2. NCDEs

   On the Choice of Interpolation Scheme for Neural CDEs

   James Morrill, Patrick Kidger, Lingyi Yang, and 1 more author

   Transactions on Machine Learning Research, Dec 2022

   Abs Paper

   Neural Controlled Differential Equations (Neural CDEs) are a powerful model for irregularly-sampled time series. A key component of Neural CDEs is the choice of interpolation scheme for the input data. Common choices include linear interpolation and cubic spline interpolation. We investigate the impact of different interpolation schemes on the performance of Neural CDEs. We show that the choice of interpolation scheme can significantly affect the model’s ability to capture complex dynamics and generalize to unseen data. We provide theoretical insights and empirical evidence on various real-world datasets, demonstrating that a carefully chosen interpolation scheme can lead to substantial performance improvements.

3. Review

   On Neural Differential Equations

   Patrick Kidger

   Feb 2022

   Abs DOI Paper arXiv

   The conjoining of dynamical systems and deep learning has become a topic of great interest. In particular, neural differential equations (NDEs) demonstrate that neural networks and differential equation are two sides of the same coin. Traditional parameterised differential equations are a special case. Many popular neural network architectures, such as residual networks and recurrent networks, are discretisations. NDEs are suitable for tackling generative problems, dynamical systems, and time series (particularly in physics, finance, ...) and are thus of interest to both modern machine learning and traditional mathematical modelling. NDEs offer high-capacity function approximation, strong priors on model space, the ability to handle irregular data, memory efficiency, and a wealth of available theory on both sides. This doctoral thesis provides an in-depth survey of the field. Topics include: neural ordinary differential equations (e.g. for hybrid neural/mechanistic modelling of physical systems); neural controlled differential equations (e.g. for learning functions of irregular time series); and neural stochastic differential equations (e.g. to produce generative models capable of representing complex stochastic dynamics, or sampling from complex high-dimensional distributions). Further topics include: numerical methods for NDEs (e.g. reversible differential equations solvers, backpropagation through differential equations, Brownian reconstruction); symbolic regression for dynamical systems (e.g. via regularised evolution); and deep implicit models (e.g. deep equilibrium models, differentiable optimisation). We anticipate this thesis will be of interest to anyone interested in the marriage of deep learning with dynamical systems, and hope it will provide a useful reference for the current state of the art.

4. NCDEs

   Graph Neural Controlled Differential Equations for Traffic Forecasting

   Jeongwhan Choi, Hwangyong Choi, Jeehyun Hwang, and 1 more author

   Proceedings of the AAAI Conference on Artificial Intelligence, Jun 2022

   Abs DOI Paper arXiv GitHub

   Traffic forecasting is one of the most popular spatio-temporal tasks in the field of machine learning. A prevalent approach in the field is to combine graph convolutional networks and recurrent neural networks for the spatio-temporal processing. There has been fierce competition and many novel methods have been proposed. In this paper, we present the method of spatio-temporal graph neural controlled differential equation (STG-NCDE). Neural controlled differential equations (NCDEs) are a breakthrough concept for processing sequential data. We extend the concept and design two NCDEs: one for the temporal processing and the other for the spatial processing. After that, we combine them into a single framework. We conduct experiments with 6 benchmark datasets and 20 baselines. STG-NCDE shows the best accuracy in all cases, outperforming all those 20 baselines by non-trivial margins.

2021

1. NCDEs

   Neural Rough Differential Equations for Long Time Series.

   James Morrill, Cristopher Salvi, Patrick Kidger, and 1 more author

   In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event., Jun 2021

   Abs Paper arXiv GitHub

   Neural Controlled Differential Equations (CDEs) are a powerful model for irregularly-sampled time series. However, their computational cost scales quadratically with the length of the time series, making them impractical for long sequences. We propose Neural Rough Differential Equations (RDEs), a new model that leverages the theory of rough paths to reduce the computational cost of CDEs to linear in the length of the time series. Our model retains the expressiveness of CDEs while being significantly more efficient. We demonstrate the effectiveness of our model on several real-world datasets, including human motion capture data and medical records. Our model outperforms existing methods for long irregularly-sampled time series.

2. Related

   Score-Based Generative Modeling through Stochastic Differential Equations.

   Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, and 3 more authors

   In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, Jun 2021

   Abs Paper arXiv GitHub

   We propose a unified framework for generative modeling that leverages stochastic differential equations (SDEs) to transform a simple prior distribution into a complex data distribution. Our framework encompasses various existing generative models, including score-based generative models and denoising diffusion probabilistic models, as special cases. By defining a continuous-time SDE that gradually perturbs data towards noise, and its reverse-time SDE that transforms noise back into data, we can generate samples by solving the reverse SDE. We demonstrate the effectiveness of our framework on various tasks, including image generation and unconditional sampling, showing competitive performance with state-of-the-art generative models.

3. Related

   Universal Differential Equations for Scientific Machine Learning

   Christopher Rackauckas, Yingbo Ma, Julius Martensen, and 6 more authors

   Nov 2021

   Abs DOI Paper arXiv

   In the context of science, the well-known adage "a picture is worth a thousand words" might well be "a model is worth a thousand datasets." In this manuscript we introduce the SciML software ecosystem as a tool for mixing the information of physical laws and scientific models with data-driven machine learning approaches. We describe a mathematical object, which we denote universal differential equations (UDEs), as the unifying framework connecting the ecosystem. We show how a wide variety of applications, from automatically discovering biological mechanisms to solving high-dimensional Hamilton-Jacobi-Bellman equations, can be phrased and efficiently handled through the UDE formalism and its tooling. We demonstrate the generality of the software tooling to handle stochasticity, delays, and implicit constraints. This funnels the wide variety of SciML applications into a core set of training mechanisms which are highly optimized, stabilized for stiff equations, and compatible with distributed parallelism and GPU accelerators.

4. Related

   Neural Delay Differential Equations.

   Qunxi Zhu, Yao Guo, and Wei Lin

   In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, Nov 2021

   Abs Paper arXiv

   Neural Ordinary Differential Equations (NODEs), a framework of continuous-depth neural networks, have been widely applied, showing exceptional efficacy in coping with some representative datasets. Recently, an augmented framework has been successfully developed for conquering some limitations emergent in application of the original framework. Here we propose a new class of continuous-depth neural networks with delay, named as Neural Delay Differential Equations (NDDEs), and, for computing the corresponding gradients, we use the adjoint sensitivity method to obtain the delayed dynamics of the adjoint. Since the differential equations with delays are usually seen as dynamical systems of infinite dimension possessing more fruitful dynamics, the NDDEs, compared to the NODEs, own a stronger capacity of nonlinear representations. Indeed, we analytically validate that the NDDEs are of universal approximators, and further articulate an extension of the NDDEs, where the initial function of the NDDEs is supposed to satisfy ODEs. More importantly, we use several illustrative examples to demonstrate the outstanding capacities of the NDDEs and the NDDEs with ODEs’ initial value. More precisely, (1) we successfully model the delayed dynamics where the trajectories in the lower-dimensional phase space could be mutually intersected, while the traditional NODEs without any argumentation are not directly applicable for such modeling, and (2) we achieve lower loss and higher accuracy not only for the data produced synthetically by complex models but also for the real-world image datasets, i.e., CIFAR10, MNIST and SVHN. Our results on the NDDEs reveal that appropriately articulating the elements of dynamical systems into the network design is truly beneficial to promoting the network performance.

5. Related

   Neural Flows: Efficient Alternative to Neural ODEs

   Marin Biloš, Johanna Sommer, Syama Sundar Rangapuram, and 2 more authors

   In Advances in Neural Information Processing Systems, Nov 2021

   Abs Paper arXiv GitHub

   Neural ordinary differential equations describe how values change in time. This is the reason why they gained importance in modeling sequential data, especially when the observations are made at irregular intervals. In this paper we propose an alternative by directly modeling the solution curves - the flow of an ODE - with a neural network. This immediately eliminates the need for expensive numerical solvers while still maintaining the modeling capability of neural ODEs. We propose several flow architectures suitable for different applications by establishing precise conditions on when a function defines a valid flow. Apart from computational efficiency, we also provide empirical evidence of favorable generalization performance via applications in time series modeling, forecasting, and density estimation.

6. Related

   Normalizing Flows for Probabilistic Modeling and Inference

   George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, and 2 more authors

   Journal of Machine Learning Research, Nov 2021

   Abs Paper arXiv

   Normalizing flows provide a general mechanism for defining expressive probability distributions, only requiring the specification of a (usually simple) base distribution and a series of bijective transformations. There has been much recent work on normalizing flows, ranging from improving their expressive power to expanding their application. We believe the field has now matured and is in need of a unified perspective. In this review, we attempt to provide such a perspective by describing flows through the lens of probabilistic modeling and inference. We place special emphasis on the fundamental principles of flow design, and discuss foundational topics such as expressive power and computational trade-offs. We also broaden the conceptual framing of flows by relating them to more general probability transformations. Lastly, we summarize the use of flows for tasks such as generative modeling, approximate inference, and supervised learning.

2020

1. NCDEs

   Neural Controlled Differential Equations for Irregular Time Series.

   Patrick Kidger, James Morrill, James Foster, and 1 more author

   In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual., Nov 2020

   Abs Paper arXiv GitHub

   Many real-world time series are irregularly sampled, with observations arriving at arbitrary time points. Standard recurrent neural networks are not well-suited to this setting, as they assume fixed time intervals between observations. We propose a new model for irregularly-sampled time series based on Neural Controlled Differential Equations (CDEs). Our model learns the continuous-time dynamics of the underlying system, and can be evaluated at arbitrary time points. We demonstrate the effectiveness of our model on several real-world datasets, including medical records and human motion capture data. Our model outperforms state-of-the-art methods for irregularly-sampled time series, and is competitive with methods for regularly-sampled data.

2. NSDEs

   Scalable Gradients for Stochastic Differential Equations

   Xuechen Li, Ting-Kam Leonard Wong, Ricky T. Q. Chen, and 1 more author

   In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, Aug 2020

   Abs Paper arXiv GitHub

   Stochastic differential equations (SDEs) are a powerful tool for modeling systems with inherent noise. However, training models that involve SDEs can be challenging due to the difficulty of computing gradients through stochastic processes. We propose a new method for computing gradients of expectations with respect to SDEs that is both scalable and unbiased. Our method leverages the adjoint method, which allows us to compute gradients by solving a backward SDE. We demonstrate the effectiveness of our method on several tasks, including training generative models and learning latent SDEs from data. Our method outperforms existing approaches in terms of both accuracy and computational efficiency.

3. NODEs

   Learning Long-Term Dependencies in Irregularly-Sampled Time Series

   Mathias Lechner and Ramin Hasani

   Dec 2020

   Abs DOI Paper arXiv GitHub

   Many real-world time series are irregularly sampled, with observations arriving at arbitrary time points. Standard recurrent neural networks struggle with such data due to their reliance on fixed time steps. We propose a novel approach for learning long-term dependencies in irregularly-sampled time series by combining the strengths of recurrent neural networks with continuous-time models. Our method leverages a continuous-time hidden state that evolves according to a learned ordinary differential equation, allowing it to capture complex temporal dynamics regardless of sampling irregularity. We demonstrate the effectiveness of our model on several real-world datasets, including medical records and human motion capture data, showing improved performance in tasks such as classification and forecasting.

4. NODEs

   STEER : Simple Temporal Regularization For Neural ODE

   Arnab Ghosh, Harkirat Behl, Emilien Dupont, and 2 more authors

   In Advances in Neural Information Processing Systems, Dec 2020

   Abs Paper arXiv

   Neural Ordinary Differential Equations (Neural ODEs) have emerged as a powerful tool for modeling continuous-time dynamics. However, training Neural ODEs can be challenging due to their sensitivity to initial conditions and the lack of explicit regularization for temporal smoothness. We propose STEER (Simple Temporal Regularization For Neural ODE), a novel regularization technique that encourages temporal smoothness in the learned dynamics of Neural ODEs. STEER adds a penalty term to the loss function that regularizes the magnitude of the derivative of the hidden state, promoting more stable and interpretable dynamics. We demonstrate the effectiveness of STEER on various tasks, including time series modeling and density estimation, showing improved performance and robustness compared to standard Neural ODEs.

5. NODEs

   Adaptive Checkpoint Adjoint Method for Gradient Estimation in Neural ODE

   Juntang Zhuang, Nicha Dvornek, Xiaoxiao Li, and 3 more authors

   In Proceedings of the 37th International Conference on Machine Learning, Jul 2020

   Abs Paper arXiv

   Neural Ordinary Differential Equations (Neural ODEs) offer a memory-efficient way to model continuous-time dynamics by backpropagating through an ODE solver using the adjoint method. However, the adjoint method can be computationally expensive, especially for long integration times, as it requires solving an additional backward ODE. We propose the Adaptive Checkpoint Adjoint Method (ACAM), a novel approach that combines checkpointing with the adjoint method to reduce memory and computational costs. ACAM adaptively selects checkpoints during the forward pass, allowing for efficient gradient computation without storing the entire trajectory. We demonstrate the effectiveness of ACAM on various tasks, showing significant reductions in memory and computation while maintaining accuracy compared to standard adjoint methods.

6. NODEs

   Approximation Capabilities of Neural ODEs and Invertible Residual Networks

   Han Zhang, Xi Gao, Jacob Unterman, and 1 more author

   In Proceedings of the 37th International Conference on Machine Learning, Jul 2020

   Abs Paper arXiv

   Neural Ordinary Differential Equations (Neural ODEs) and Invertible Residual Networks (i-ResNets) are two recent architectures that offer memory-efficient training and exact likelihood computation, respectively. Both models are based on continuous-depth transformations. We investigate the approximation capabilities of Neural ODEs and i-ResNets, showing that they are universal approximators of continuous functions. We provide theoretical guarantees on their ability to approximate arbitrary continuous functions and demonstrate their practical performance on various tasks, including density estimation and image classification. Our results shed light on the expressive power of these continuous-depth models and their potential for deep learning applications.

7. NODEs

   Universal Approximation Property of Neural Ordinary Differential Equations

   Takeshi Teshima, Koichi Tojo, Masahiro Ikeda, and 2 more authors

   Dec 2020

   Abs DOI Paper arXiv

   Neural Ordinary Differential Equations (Neural ODEs) have recently emerged as a new class of deep learning models that parameterize the derivative of the hidden state using a neural network. This allows them to model continuous-time dynamics and offer memory-efficient training. We prove that Neural ODEs possess the universal approximation property, meaning they can approximate any continuous function to arbitrary accuracy. Our proof provides theoretical guarantees on the expressive power of Neural ODEs and sheds light on their ability to learn complex mappings. This result further solidifies the theoretical foundation of Neural ODEs and supports their wide applicability in various domains.

8. NODEs

   Neural Manifold Ordinary Differential Equations

   Aaron Lou, Derek Lim, Isay Katsman, and 4 more authors

   In Advances in Neural Information Processing Systems, Dec 2020

   Abs Paper arXiv GitHub

   Neural Ordinary Differential Equations (Neural ODEs) have shown promise in modeling continuous-time dynamics. However, they operate in Euclidean space, which may not be suitable for data residing on non-Euclidean manifolds. We introduce Neural Manifold Ordinary Differential Equations (Neural Manifold ODEs), a novel framework that extends Neural ODEs to learn dynamics on Riemannian manifolds. Our approach leverages geometric deep learning techniques to define ODEs on curved spaces, allowing for more natural and expressive modeling of data with underlying manifold structures. We demonstrate the effectiveness of Neural Manifold ODEs on tasks such as learning dynamics on spheres and hyperbolic spaces, showing improved performance and interpretability compared to Euclidean counterparts.

9. NODEs

   Dissecting Neural ODEs

   Stefano Massaroli, Michael Poli, Jinkyoo Park, and 2 more authors

   In Advances in Neural Information Processing Systems, Dec 2020

   Abs Paper arXiv

   Neural Ordinary Differential Equations (Neural ODEs) have gained significant attention for their ability to model continuous-time dynamics and offer memory-efficient training. However, their internal workings and the factors influencing their performance are not yet fully understood. We conduct a thorough dissection of Neural ODEs, analyzing various aspects such as their approximation capabilities, sensitivity to hyperparameters, and the role of the ODE solver. Through extensive empirical studies and theoretical insights, we provide a deeper understanding of why Neural ODEs work and how to effectively train them. Our findings offer practical guidelines for designing and applying Neural ODEs in various machine learning tasks.

10. Related

    Learning Differential Equations that are Easy to Solve

    Jacob Kelly, Jesse Bettencourt, Matthew J Johnson, and 1 more author

    In Advances in Neural Information Processing Systems, Dec 2020

    Abs Paper arXiv GitHub

    Differential equations parameterized by neural networks become expensive to solve numerically as training progresses. We propose a remedy that encourages learned dynamics to be easier to solve. Specifically, we introduce a differentiable surrogate for the time cost of standard numerical solvers, using higher-order derivatives of solution trajectories. These derivatives are efficient to compute with Taylor-mode automatic differentiation. Optimizing this additional objective trades model performance against the time cost of solving the learned dynamics. We demonstrate our approach by training substantially faster, while nearly as accurate, models in supervised classification, density estimation, and time-series modelling tasks.

2019

1. NODEs

   Latent Ordinary Differential Equations for Irregularly-Sampled Time Series.

   Yulia Rubanova, Tian Qi Chen, and David Duvenaud

   In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada., Dec 2019

   Abs Paper arXiv GitHub

   Many real-world time series are irregularly-sampled, with observations arriving at arbitrary time points. Standard recurrent neural networks are not well-suited to this setting, as they assume fixed time intervals between observations. We propose a new model for irregularly-sampled time series based on Neural Ordinary Differential Equations (ODEs). Our model learns the continuous-time dynamics of the underlying system, and can be evaluated at arbitrary time points. We demonstrate the effectiveness of our model on several real-world datasets, including medical records and human motion capture data. Our model outperforms state-of-the-art methods for irregularly-sampled time series, and is competitive with methods for regularly-sampled data.

2. NODEs

   Augmented Neural ODEs.

   Emilien Dupont, Arnaud Doucet, and Yee Whye Teh

   In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada., Dec 2019

   Abs Paper arXiv GitHub

   Neural Ordinary Differential Equations (Neural ODEs) parameterize the derivative of the hidden state using a neural network. This allows them to model continuous-time dynamics and handle irregularly sampled data. However, Neural ODEs are limited by the fact that the hidden state must have the same dimension as the input. We propose Augmented Neural ODEs (ANODEs), which augment the hidden state with additional dimensions. This allows ANODEs to learn more complex dynamics and improve performance on tasks such as density estimation and time series modeling. We demonstrate the effectiveness of ANODEs on several real-world datasets, showing significant improvements over standard Neural ODEs.

3. NODEs

   GRU-ODE-Bayes: Continuous Modeling of Sporadically-Observed Time Series.

   Edward De Brouwer, Jaak Simm, Adam Arany, and 1 more author

   In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada., Dec 2019

   Abs Paper arXiv GitHub

   Many real-world time series are sporadically observed, meaning that observations arrive at irregular time intervals and some values may be missing. Standard recurrent neural networks struggle with such data due to their reliance on fixed time steps. We propose GRU-ODE-Bayes, a novel model that combines the strengths of Gated Recurrent Units (GRUs) with Ordinary Differential Equations (ODEs) and Bayesian inference to handle sporadically observed time series. Our model learns continuous-time dynamics and provides uncertainty estimates for predictions. We demonstrate the effectiveness of GRU-ODE-Bayes on several real-world datasets, including medical records and climate data, showing significant improvements over state-of-the-art methods for irregularly sampled time series.

4. NSDEs

   Neural Stochastic Differential Equations: Deep Latent Gaussian Models in the Diffusion Limit

   Belinda Tzen and Maxim Raginsky

   Oct 2019

   Abs DOI Paper arXiv

   We introduce Neural Stochastic Differential Equations (Neural SDEs), a new class of deep generative models that extend Neural Ordinary Differential Equations (Neural ODEs) by incorporating a stochastic diffusion term. Neural SDEs model continuous-time dynamics with inherent noise, allowing for more robust and expressive representations of complex data. We show that Neural SDEs can be viewed as deep latent Gaussian models in the diffusion limit, providing a theoretical foundation for their generative capabilities. We demonstrate the effectiveness of Neural SDEs on tasks such as density estimation and time series modeling, showing competitive performance with state-of-the-art methods while offering improved flexibility and interpretability.

5. NODEs

   ODE2VAE: Deep generative second order ODEs with Bayesian neural networks

   Cagatay Yildiz, Markus Heinonen, and Harri Lahdesmaki

   In Advances in Neural Information Processing Systems, Oct 2019

   Abs Paper arXiv GitHub

   Deep generative models have achieved remarkable success in various domains, but modeling complex temporal dynamics, especially with irregularly sampled data, remains a challenge. We propose ODE2VAE, a novel deep generative model that combines second-order Ordinary Differential Equations (ODEs) with Variational Autoencoders (VAEs) and Bayesian neural networks. Our model learns continuous-time dynamics in the latent space, allowing it to handle irregular observations and capture complex temporal dependencies. The use of second-order ODEs enables modeling of acceleration and momentum, providing richer dynamics. We demonstrate the effectiveness of ODE2VAE on several real-world datasets, including medical records and motion capture data, showing improved performance in tasks such as imputation, forecasting, and anomaly detection.

6. NSDEs

   Neural SDE: Stabilizing Neural ODE Networks with Stochastic Noise

   Xuanqing Liu, Tesi Xiao, Si Si, and 3 more authors

   Oct 2019

   Abs DOI Paper arXiv

   Neural Ordinary Differential Equations (Neural ODEs) have gained popularity for modeling continuous-time dynamics. However, they can be sensitive to noise and prone to instability, especially in high-dimensional settings. We propose Neural SDE, a novel approach that stabilizes Neural ODE networks by introducing stochastic noise into the dynamics. By augmenting the deterministic ODE with a learnable diffusion term, Neural SDE can better capture uncertainties and improve robustness. We demonstrate the effectiveness of Neural SDE on various tasks, including density estimation and time series modeling, showing improved stability and performance compared to standard Neural ODEs, particularly in the presence of noisy data.

7. NSDEs

   Neural Jump Stochastic Differential Equations

   Junteng Jia and Austin R Benson

   In Advances in Neural Information Processing Systems, Oct 2019

   Abs Paper arXiv

   Stochastic differential equations (SDEs) are widely used to model systems with continuous-time random fluctuations. However, many real-world phenomena exhibit sudden, discrete changes or jumps, which are not captured by standard SDEs. We introduce Neural Jump Stochastic Differential Equations (Neural JSDEs), a novel class of models that extend Neural SDEs by incorporating jump processes. Neural JSDEs can model both continuous diffusion and discrete jumps, providing a more comprehensive framework for complex temporal dynamics. We demonstrate the effectiveness of Neural JSDEs on various tasks, including financial modeling and anomaly detection, showing improved performance in scenarios with abrupt changes.

8. Related

   Invertible Residual Networks

   Jens Behrmann, Will Grathwohl, Ricky T. Q. Chen, and 2 more authors

   In Proceedings of the 36th International Conference on Machine Learning, Oct 2019

   Abs Paper arXiv GitHub

   We introduce Invertible Residual Networks (i-ResNets), a new class of deep neural networks that are invertible by construction. This property allows for exact likelihood computation and memory-efficient training, as activations do not need to be stored for backpropagation. i-ResNets are built upon the idea of residual blocks, where each block is designed to be invertible. We demonstrate the effectiveness of i-ResNets on various tasks, including image classification and density estimation, showing competitive performance with state-of-the-art models while offering significant memory savings. Our work opens up new possibilities for designing deep architectures with desirable properties.

9. Related

   FFJORD: Free-Form Continuous Dynamics for Scalable Reversible Generative Models.

   Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, and 2 more authors

   In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, Oct 2019

   Abs Paper arXiv GitHub

   Normalizing flows are a powerful class of generative models that allow for exact likelihood computation. However, existing normalizing flows often suffer from limitations in expressiveness or scalability. We introduce FFJORD (Free-Form Jacobian of Reversible Dynamics), a novel normalizing flow model based on Neural Ordinary Differential Equations (Neural ODEs). FFJORD parameterizes the transformation as a continuous-time dynamic system, allowing for flexible and expressive transformations while maintaining invertibility and exact likelihood computation. We demonstrate the scalability and effectiveness of FFJORD on various tasks, including density estimation and image generation, showing competitive performance with state-of-the-art generative models.

10. NODEs

    ANODE: Unconditionally Accurate Memory-Efficient Gradients for Neural ODEs.

    Amir Gholaminejad, Kurt Keutzer, and George Biros

    In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, Oct 2019

    Abs DOI Paper arXiv

    Neural Ordinary Differential Equations (Neural ODEs) offer a memory-efficient way to compute gradients using the adjoint method. However, the adjoint method can be computationally expensive and numerically unstable for long integration times. We propose ANODE (Unconditionally Accurate Memory-Efficient Gradients for Neural ODEs), a novel method for computing gradients in Neural ODEs that is both memory-efficient and numerically stable. ANODE leverages a new adjoint formulation that avoids the need for backward integration, leading to significant computational savings and improved stability. We demonstrate the effectiveness of ANODE on various tasks, showing competitive performance with existing methods while offering superior efficiency and robustness.

2018

1. NODEs

   Neural Ordinary Differential Equations.

   Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and 1 more author

   In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada., Oct 2018

   Abs Paper arXiv GitHub

   We introduce a new family of deep neural network models. Instead of specifying a discrete sequence of hidden layers, we parameterize the derivative of the hidden state using a neural network. The output of the network is computed by an off-the-shelf ODE solver. These ’Neural ODEs’ can model continuous-time dynamics, and can be trained with backpropagation through the ODE solver. This allows for memory-efficient training, as the adjoint method can be used to compute gradients without storing the entire forward pass. We demonstrate the effectiveness of Neural ODEs on various tasks, including density estimation, time series modeling, and image classification, showing competitive performance with state-of-the-art models while offering significant memory savings.

2. Related

   Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations

   Yiping Lu, Aoxiao Zhong, Quanzheng Li, and 1 more author

   In Proceedings of the 35th International Conference on Machine Learning, Oct 2018

   Abs Paper arXiv GitHub

   Deep neural networks have become the state-of-the-art models in numerous machine learning tasks. However, general guidance to network architecture design is still missing. In our work, we bridge deep neural network design with numerical differential equations. We show that many effective networks, such as ResNet, PolyNet, FractalNet and RevNet, can be interpreted as different numerical discretizations of differential equations. This finding brings us a brand new perspective on the design of effective deep architectures. We can take advantage of the rich knowledge in numerical analysis to guide us in designing new and potentially more effective deep networks. As an example, we propose a linear multi-step architecture (LM-architecture) which is inspired by the linear multi-step method solving ordinary differential equations. The LM-architecture is an effective structure that can be used on any ResNet-like networks. In particular, we demonstrate that LM-ResNet and LM-ResNeXt (i.e. the networks obtained by applying the LM-architecture on ResNet and ResNeXt respectively) can achieve noticeably higher accuracy than ResNet and ResNeXt on both CIFAR and ImageNet with comparable numbers of trainable parameters. In particular, on both CIFAR and ImageNet, LM-ResNet/LM-ResNeXt can significantly compress (>50%) the original networks while maintaining a similar performance. This can be explained mathematically using the concept of modified equation from numerical analysis. Last but not least, we also establish a connection between stochastic control and noise injection in the training process which helps to improve generalization of the networks. Furthermore, by relating stochastic training strategy with stochastic dynamic system, we can easily apply stochastic training to the networks with the LM-architecture. As an example, we introduced stochastic depth to LM-ResNet and achieve significant improvement over the original LM-ResNet on CIFAR10.

3. Related

   Stable architectures for deep neural networks

   Eldad Haber and Lars Ruthotto

   Inverse Problems, Jan 2018

   Abs DOI Paper arXiv

   Deep neural networks have become invaluable tools for supervised machine learning, e.g. classification of text or images. While often offering superior results over traditional techniques and successfully expressing complicated patterns in data, deep architectures are known to be challenging to design and train such that they generalize well to new data. Critical issues with deep architectures are numerical instabilities in derivative-based learning algorithms commonly called exploding or vanishing gradients. In this paper, we propose new forward propagation techniques inspired by systems of ordinary differential equations (ODE) that overcome this challenge and lead to well-posed learning problems for arbitrarily deep networks. The backbone of our approach is our interpretation of deep learning as a parameter estimation problem of nonlinear dynamical systems. Given this formulation, we analyze stability and well-posedness of deep learning and use this new understanding to develop new network architectures. We relate the exploding and vanishing gradient phenomenon to the stability of the discrete ODE and present several strategies for stabilizing deep learning for very deep networks. While our new architectures restrict the solution space, several numerical experiments show their competitiveness with state-of-the-art networks.

2017

1. Related

   Deep Learning-Based Numerical Methods for High-Dimensional Parabolic Partial Differential Equations and Backward Stochastic Differential Equations

   Weinan E, Jiequn Han, and Arnulf Jentzen

   Communications in Mathematics and Statistics, Dec 2017

   Abs DOI Paper arXiv

   We propose a deep learning-based numerical method for solving high-dimensional parabolic partial differential equations (PDEs) and backward stochastic differential equations (BSDEs). The method is based on approximating the unknown function by a deep neural network and training the network by minimizing a loss function derived from the PDE or BSDE. We demonstrate the effectiveness of our method on several high-dimensional examples, including the Black-Scholes equation and a high-dimensional Hamilton-Jacobi-Bellman equation. Our method outperforms traditional numerical methods in terms of accuracy and computational efficiency for high-dimensional problems.

2. Related

   Solving differential equations of fractional order using an optimization technique based on training artificial neural network

   M. Pakdaman, A. Ahmadian, S. Effati, and 2 more authors

   Applied Mathematics and Computation, Jan 2017

   Abs DOI Paper

   The current study aims to approximate the solution of fractional differential equations (FDEs) by using the fundamental properties of artificial neural networks (ANNs) for function approximation. In the first step, we derive an approximate solution of fractional differential equation (FDE) by using ANNs. In the second step, an optimization approach is exploited to adjust the weights of ANNs such that the approximated solution satisfies the FDE. Different types of FDEs including linear and nonlinear terms are solved to illustrate the ability of the method. In addition, the present scheme is compared with the analytical solution and a number of existing numerical techniques to show the efficiency of ANNs with high accuracy, fast convergence and low use of memory for solving the FDEs.