← blog

An Exploration of PINNs

11.10.2024

This post summarizes my self-study of physics-informed neural networks (PINNs), a method that elegantly blends differential equations with deep learning. PINNs offer a mesh-free alternative to classical solvers for PDEs, encoding physical laws directly into a neural network's training objective.

Traditional methods for solving PDEs: finite difference, finite element, or spectral methods—require discretizing the domain. These techniques trade off accuracy, complexity, and resolution. In contrast, PINNs aim to model the solution function itself using a continuous, differentiable neural network.

Consider, for example, the Navier–Stokes equations in incompressible fluid dynamics:

\( \nabla \cdot \mathbf{u} = 0 \quad \text{and} \quad \frac{\partial \mathbf{u}}{\partial t} + (\mathbf{u} \cdot \nabla)\mathbf{u} = -\frac{1}{\rho}\nabla p + \nu \nabla^2\mathbf{u} + \mathbf{f} \)

When the velocity field \( \mathbf{u} \) is turbulent, its behavior is chaotic, multi-scale, and notoriously hard to resolve. PINNs shine in such scenarios by using neural networks to approximate the field while incorporating the PDE directly into the loss function.

But what exactly is a physics-informed neural network? Let's build from first principles. A neural network is a parameterized function mapping inputs \( \mathbf{x} \) to outputs \( \mathbf{y} \), denoted:

\( NN(\mathbf{x}, \boldsymbol{\theta}) \approx \mathbf{y}(\mathbf{x}) \)

During training, we tune parameters \( \boldsymbol{\theta} \) by minimizing a loss function, which measures the difference between predicted and ground-truth outputs:

\( \mathcal{L}(\boldsymbol{\theta}) = \frac{1}{N} \sum_{i=1}^N \left( NN(\mathbf{x}_i, \boldsymbol{\theta}) - \mathbf{y}_i \right)^2 \)

Neural networks are known as universal function approximators: they can represent any smooth function to arbitrary precision given enough capacity. In PINNs, we leverage this expressiveness not just to fit data, but to solve differential equations.

To illustrate how PINNs work, consider a simpler problem from undergraduate physics: the damped harmonic oscillator with mass \( m \), spring constant \( k \), and damping coefficient \( \mu \). Its governing ODE is:

\( m \ddot{u} + \mu \dot{u} + ku = 0 \)

Suppose we’re given initial conditions \( u(0) = 1 \) and \( \dot{u}(0) = 0 \). While the solution is analytic, we treat this as a test case for applying a PINN. We model the solution \( u(t) \) with a neural network:

\( \hat{u}(t) = NN(t, \boldsymbol{\theta}) \)

The total loss combines two components: one to enforce initial conditions and another to enforce the ODE. Specifically, we define:

\( \mathcal{L}_b = \lambda_1 \left( NN(0, \boldsymbol{\theta}) - 1 \right)^2 + \lambda_2 \left( \dot{NN}(0, \boldsymbol{\theta}) - 0 \right)^2 \)

\( \mathcal{L}_p = \frac{1}{N} \sum_{i=1}^N \left( m\frac{d^2}{dt^2} NN(t_i, \boldsymbol{\theta}) + \mu \frac{d}{dt} NN(t_i, \boldsymbol{\theta}) + k NN(t_i, \boldsymbol{\theta}) \right)^2 \)

We minimize the total loss \( \mathcal{L} = \mathcal{L}_b + \mathcal{L}_p \), ensuring the neural network satisfies both the physical dynamics and initial conditions. For interpretability, we can also write the equation in terms of damping ratio \( \zeta \) and natural frequency \( \omega \):

\( \ddot{u} + 2\zeta\omega \dot{u} + \omega^2 u = 0 \)

where \( \zeta = \frac{\mu}{2\sqrt{mk}} \) and \( \omega = \sqrt{\frac{k}{m}} \). The trained network \( \hat{u}(t) \) can then be compared to the closed-form solution \( u(t) \), validating the effectiveness of the PINN framework.

The full implementation of this example is available on my GitHub repository.