Neural Ordinary Differential Equations 🐰🦊 πŸ”—

Presented by Christabella Irwanto

Available as slides

Neural ODE πŸ”—

ODE? πŸ˜• πŸ”—

β€œIn the 300 years since Newton, mankind has come to realize that the laws of physics are always expressed in the language of differential equations.”

  • Steven Strogatz
  • E.g. radioactive decay, kinematic systems, or drug concentration in a body, over time

Bunny example πŸ‡ πŸ”—

Visualize ODE πŸ”—

Slope field of derivative for each \((B, t)\)

Solve ODE πŸ”—

  • Infinitely many solution, but generally only one satisfying initial conditions

Numerical ODE solver πŸ”—

Enter neural networks πŸ”—

\(\mathbf{h}_{t+1} = f(\mathbf{h}_t)\)

Resnet πŸ”—

Neural ODE πŸ”—

Resnet vs neural ODE πŸ”—

Forward pass πŸ”—

Numerical ODE solver πŸ”—

Advantages πŸ”—

Backward pass πŸ”—

  • How do we train the function in the ODE?

\begin{equation} L(\mathbf{z}(t_1)) = L\left( \mathbf{z}(t_0) + \int_{t_0}^{t_1} f(\mathbf{z}(t), t, \theta)dt \right) = L(\textrm{ODESolve}(\mathbf{z}(t_0), f, t_0, t_1, \theta)) \end{equation}

How to train? πŸ”—

  • Backpropagate through the ODE solver layer

Adjoint sensitivity πŸ”—

Results of ODE-Net vs Resnet πŸ”—

Generative latent time-series model πŸ”—

  • Standard VAE algorithm with ODESolve as decoder

  • Using ODEs as a generative model allows us to make predictions for arbitrary time points \(t_1 \ldots t_M\) on a continuous timeline

Results of experiments πŸ”—

Advantages πŸ”—

Normalizing flows πŸ”—

\begin{align} z_0 &\sim Normal(0, I) \\
z_1 &= f_0(z_0) \\
… \\
x &= f_t(z_t) \end{align}

Results πŸ”—

Pros and cons πŸ”—

Summary πŸ”—

References πŸ”—

Code: πŸ”—

Blogposts πŸ”—

For adjoint sensitivity: πŸ”—

Original content from author(s) πŸ”—


Chen, R. T. Q., Rubanova, Y., Bettencourt, J., & Duvenaud, D., Neural Ordinary Differential Equations, , (), (2018). ↩

Grathwohl, W., Chen, R. T. Q., Bettencourt, J., Sutskever, I., & Duvenaud, D., Ffjord: free-form continuous dynamics for scalable reversible generative models, , (), (2018). ↩