Spectral Audit of In-Context Operator Networks

Spectral Audit: A New Diagnostic for In-Context Operator Learning Fidelity

When AI learns to solve physics equations from examples, it might get the answer right but use the wrong logic. This paper introduces a way to check if the AI's internal "math engine" actually matches the real physics by looking at how it responds to tiny changes.

In scientific machine learning, we often use In-Context Operator Networks (ICONs). These models receive a "prompt" consisting of input-output pairs. This prompt represents a specific physical regime. The model must then predict the response to a new query. The goal is to learn the mapping between function spaces, such as the solution operator of a partial differential equation (PDE). Traditionally, we evaluate these models using standard $L^2$ prediction error (a measure of the distance between predicted and true functions).

The problem is that accurate output prediction does not guarantee the correct local dynamical structure. A model can achieve low pointwise error while still exhibiting incorrect sensitivities or distorted frequency responses. This is a critical failure mode for rollouts (repeatedly applying the model to its own output) or optimization. In these cases, the model's sensitivity to perturbations—its "tangent behavior"—is what drives the system.

The Problem

The status quo relies almost exclusively on prediction error metrics. As the authors demonstrate, this is a risky way to validate a physical surrogate. A model might successfully interpolate between known solutions but fail to capture the underlying physics.

For example, in the advection task (moving a signal through space), a model might predict the correct position of a wave. However, it might fail to realize that the operator should be purely transport-based. If the model introduces artificial smoothing or incorrect phase shifts, it has not truly learned the advection operator. It has merely learned a sophisticated interpolator. The authors note that prediction accuracy and local operator fidelity are distinct properties. High accuracy in the training range does not imply that the model has captured the "mechanisms" of the PDE. These mechanisms include how viscosity damps high frequencies or how nonlinearity couples different modes.

How It Works

Instead of looking at the output, the authors look at the Jacobian (a matrix of first-order partial derivatives). They treat this Jacobian as a "learned tangent operator." The audit follows a structured process:

1. Differentiation: For a fixed context (the prompt), the model differentiates its prediction with respect to the query function $u_q$. This yields the Jacobian $J_{ICON}^C(u_q)$. This represents how a small perturbation in the input changes the output.
2. Spectral Projection: They project this Jacobian onto a Fourier basis (a set of sine and cosine functions). This makes the data physically interpretable. By applying the Jacobian to these functions, they extract "gains." Gains measure how much the model amplifies or damps specific frequencies.
3. Structural Analysis: They use several specialized metrics to inspect the resulting spectrum:
4. Fourier-mode gains: Measuring frequency-dependent damping or amplification.
5. Accumulated relative spectral error: A cumulative metric to track how discrepancies build up across frequencies.
6. Fourier-projected feature maps: Projecting the Jacobian into a matrix $\Psi^\top J \Phi$. Each entry shows how one input Fourier mode contributes to one output mode. This detects "off-diagonal" mass. Off-diagonal mass signifies spurious mode coupling (information leaking between frequencies).

By comparing this learned tangent map to the true PDE tangent map, the audit identifies if the model captured the algebraic structure of the physics. An example is the block-diagonal rotation required for advection .

Numbers

The authors evaluate their framework using a Transformer-based ICON. They tested three PDE benchmarks: Advection, Burgers, and Allen–Cahn. Training involved 2,000 hidden operators with 5 demonstrations per prompt.

The primary finding is that the spectral audit catches failures that $L^2$ metrics miss. In the advection experiment, prediction error remains low within the training range .

However, the audit reveals that the model's ability to maintain correct phase rotation depends on the training bandwidth. Specifically, a model trained on only 8 Fourier modes fails to generalize the correct rotation pattern to higher frequencies. This is a structural failure that is hard to see in a simple error curve .

In the Burgers equation (which includes nonlinearity and viscosity), the audit separates two distinct physical effects. It distinguishes viscosity-dependent damping in the gain curves from nonlinear mode coupling in the feature maps . Furthermore, the authors use a "negative control" to test prompt consistency. When prompts are internally inconsistent (e.g., the context uses a different viscosity than the query), the spectral audit detects a significant structural mismatch . This occurs even if the model maintains somewhat acceptable pointwise predictions.

What's Missing

The audit has clear boundaries. First, the methodology is optimized for 1D periodic domains. The Fourier basis is naturally suited to these problems. Moving to higher dimensions or irregular geometries would require replacing the Fourier basis. One might use Laplacian eigenfunctions or localized wave packets instead. For practitioners working on complex 3D fluid simulations, this specific implementation requires adaptation.

Second, this is a posterior diagnostic. It tells you that your model is broken. It does not tell you how to fix it. Unlike physics-informed loss functions, this audit is an inspection tool. You can identify that a model lacks high-frequency phase accuracy. However, you must still decide whether to increase training bandwidth or change the architecture.

Finally, the paper focuses on deterministic base states for the audit. The authors include auxiliary checks to show these states are not poorly predicted . However, more rigorous stress tests against highly stochastic regimes would be needed to prove reliability in high-entropy environments.

Should You Prototype This

Yes, specifically as a validation step in your CI/CD pipeline for scientific models. If you build neural operators for high-stakes engineering, do not trust $L^2$ error alone. This applies if the model is used in a closed-loop controller or a long-term rollout. The risk of a "right answer, wrong physics" failure is too high.

Do not implement this to improve your training loss. Implement it to verify your deployment. If your model passes the $L^2$ check but fails the spectral audit's phase or damping requirements, it is not ready for production. Start with 1D periodic toy problems to calibrate your expectations. Then, look into adapting the Jacobian projection for your specific geometry.

Figures from the paper