The Hamilton-Jacobi Theory of Deep Learning

Researchers have recently discovered that training a neural network is mathematically identical to solving a specific type of physics equation called a Hamilton–Jacobi (HJ) equation. This means that different AI architectures like Transformers and ResNets are actually just different ways of solving the same underlying physical process.

Traditionally, we treat neural networks as black-box function approximators. We design architectures and optimize weights using heuristics like backpropagation (the method of calculating gradients to update weights). We often treat the connection between architecture math and the physics of the loss landscape as purely metaphorical. Before this work, the question of "what equation does a neural network actually solve?" was largely unanswered. We relied on empirical scaling laws and trial-and-error tuning of hyperparameters like softmax temperature.

This paper provides the answer. It proves that a trained network is not just approximating a solution. It is an exact algebraic instantiation of a viscous Hamilton–Jacobi equation.

The Problem

The status quo in deep learning relies heavily on empirical observation. We know that increasing model width $N$ or decreasing temperature $\epsilon$ changes how a model generalizes. It also affects how robust it is to adversarial attacks. However, we lack a unified theory to explain why these disparate knobs move in tandem. Currently, we treat "softmax temperature," "PDE viscosity" (a parameter representing fluid friction or smoothing), and "convex regularization strength" as separate concepts.

Furthermore, our understanding of scaling laws is mostly descriptive. We observe power laws. But we lack a formal mechanism linking the scaling exponent $\alpha$ directly to the geometry of the data. This leaves engineers guessing at the optimal "operating point" for hyperparameters. This often results in either over-smoothed predictions or brittle, high-curvature decision boundaries.

How It Works

The authors unify four distinct fields—neural networks, tropical algebra (a math system using max and plus instead of sum and product), viscous PDEs, and convex optimization—using a single deformation parameter, $\epsilon$. The core of the theory rests on the Log-Sum-Exp (LSE) activation function.

1. The Algebraic Link: The authors use "Maslov dequantization" to show a transition. As $\epsilon \to 0$, the smooth LSE layer collapses into a "tropical" max-plus algebra. In this limit, the network behaves like a max-affine spline operator (MASO). It effectively acts like a decision tree that partitions input space into polyhedral regions.
2. The PDE Identity: For $\epsilon > 0$, the authors prove a rigorous identity (Theorem 4.1). They show that an LSE layer is the exact Hopf–Cole solution to a viscous HJ equation. In this mapping, the network weights encode the initial data of the PDE. The architecture defines the Hamiltonian (the function describing the system's total energy). The input $x$ is the spatial point where we evaluate the solution.
3. Architectural Mapping: The framework extends beyond simple feedforward layers. The authors demonstrate that ResNets and Recurrent architectures (RNNs, LSTMs, SSMs) are discretizations of the ODE characteristics (the paths followed by particles in a flow) of these same HJ equations. Even Transformer attention is revealed to be an expected value calculation under a Gibbs measure (a probability distribution used in statistical mechanics).
4. The Unified Parameter: The parameter $\epsilon$ serves as the master dial. It is simultaneously the softmax temperature, the PDE viscosity, and the regularization strength. As seen in, varying $\epsilon$ moves the system from a "particle" regime (sharp, localized attribution) to a "wave" regime (diffusive, smooth attribution).

Numbers

The authors provide several quantitative proofs. Most notably, they derive a theoretical basis for scaling laws. The paper reports that the scaling exponent $\alpha$ is the inverse of the data's intrinsic dimension ($d_{eff}$). This means $\alpha = 1/d_{eff}$. This is verified in .

Trained LSE networks across dimensions $d=1, 2, 4$ show RMSE loss scaling that matches the predicted $N^{-1/d}$ rates.

Regarding robustness, the authors provide a closed-form bound on the Hessian (the matrix of second-order derivatives). They show the spectral norm of the Hessian is bounded by $|W|_{2,\infty}^2/\epsilon$. This is verified in for both synthetic and trained MNIST/CIFAR-10 networks. Increasing $\epsilon$ (temperature) directly suppresses curvature. This expands the certified adversarial radius (the distance within which a model is guaranteed to be stable).

Finally, they report that the identity between LSE layers and the Hopf–Cole solution holds to machine precision ($\sim 10^{-16}$) [Table 2]. Transformer attention identities hold to exact zero error [Table 3].

What's Missing

While the mathematical rigor is high, there are practical gaps for a production engineer:

• Activation Constraints: The "exact" correspondence is strictly limited to the quadratic Hamiltonian class (LSE/softmax layers). For practitioners using GELU, SiLU, or other common non-LSE activations, this theory remains a structural correspondence rather than an exact algebraic identity.
• The Optimization Gap: The paper characterizes what a trained network computes. However, it does not fully characterize how Stochastic Gradient Descent (SGD) selects specific local minima in finite-$N$ regimes. It describes the "search" as a Wasserstein gradient flow (optimization in the space of probability distributions) in the mean-field limit.
• Dimensionality Limits: The theory acknowledges that the "curse of dimensionality" persists. The error in the quadrature approximation scales as $O(N^{-1/d})$. For very high-dimensional raw data, the number of neurons $N$ required to maintain a specific error bound might be computationally prohibitive.

Should You Prototype This

Depends on your research focus.

If you are building standard production models using off-the-shelf libraries, this paper won't change your daily workflow. However, if you are designing new architectures or studying why models behave erratically under adversarial pressure, this is a goldmine. Specifically, the insight that $\epsilon^* \approx N^{-1/d}$ provides a principled way to set softmax temperatures. You can base this on your model width and data dimensionality.

The ability to certify robustness via $\epsilon$ is a tangible lever. If you need a more stable model, the math suggests you can trade off some expressiveness for smoothness. You can achieve this by adjusting the temperature. It is a highly rigorous framework. However, until we see implementations for non-LSE activations, treat the "exactness" as a guiding principle rather than a literal guarantee for your existing stack.

Figures from the paper