Do Neural Operators Forget Geometry? The Forgetting Hypothesis in Deep Operator Learning

Abstract: Neural operators perform well on structured domains, yet their behaviour on irregular geometries remains poorly understood. We show that this limitation is not merely an encoding issue, but a depth-wise failure mode inherent to deep operator architectures. We formalise the Geometric Forgetting Hypothesis: due to the Markovian structure of operator layers and their reliance on global mixing mechanisms, neural operators progressively lose access to domain geometry as depth increases. Using layer-wise geometric probing, we demonstrate that both spectral and attention-based operators systematically lose geometric fidelity. We show that this geometric forgetting degrades accuracy, stability, and generalisation. To counteract it, we introduce a lightweight geometry memory injection mechanism that restores geometric constraints at intermediate depths with minimal architectural overhead. This simple intervention consistently mitigates forgetting and exposes a geometric shortcut instability in transformer-based operators, revealing that geometric retention is a structural requirement rather than a design choice.

• The paper demonstrates that as neural operator depth increases, geometric information fades due to the Markovian nature of global-mixing architectures.
• It introduces geometry memory injection methods, such as FiLM layers and additive injection, to effectively restore and maintain geometric fidelity.
• Empirical results on tasks like fluid flow and airfoil simulations confirm that memory injection significantly reduces error rates and enhances generalization.

Geometric Forgetting in Deep Operator Learning

Introduction and Problem Statement

The paper "Do Neural Operators Forget Geometry? The Forgetting Hypothesis in Deep Operator Learning" (2605.05862) systematically analyzes the information flow of geometric features in neural operator architectures for parametric PDE surrogacy. The authors propose the Geometric Forgetting Hypothesis: as depth increases in neural operators, especially those built on global mixing (Fourier transform, self-attention), the network’s access to geometry is progressively eroded unless geometric information is explicitly re-introduced at multiple layers.

Most existing works in neural operator learning focus on geometry encoding at the input stage. However, for real-world applications involving irregular geometries or varying domains, the persistence of geometric information across layers becomes crucial for both accuracy and generalization. This work formalizes geometric forgetting in an information-theoretic setting, presents empirical probes to quantify layerwise geometric fidelity, and introduces a minimal intervention—geometry memory injection—to combat forgetting and stabilize geometric generalization.

The Geometric Forgetting Hypothesis: Theory and Diagnostics

Neural operators are defined as deep architectures mapping between function spaces, typically $a(x)$ (PDE coefficients, boundary conditions) to $u(x)$ (solution fields) over a domain $D$. Geometry is encoded via masks, signed distance functions, or coordinates, and injected into the initial network input.

Due to the Markovian property of sequential operator layers, the data processing inequality guarantees that the mutual information $I(G; V_l)$ between the hidden state at layer $l$ and the geometry encoding $G$ is non-increasing with depth. In global-mixing architectures (FNO/transformers), geometric information decays especially quickly, often approaching statistical independence at the deepest layers.

Figure 1: Geometric forgetting as an architectural information-flow phenomenon. Geometry is only provided at the input in standard neural operators; hidden states form a Markov chain, leading to irreversible loss of geometric information with depth.

To render geometric forgetting measurable, the paper introduces two diagnostic probes:

1. Layer-wise geometric fidelity: An auxiliary decoder attempts to reconstruct the geometry mask from hidden states. Increasing reconstruction error confirms macroscopic geometric loss.
2. Spectral power analysis: Tracking the preservation of mode energy related to geometric boundaries allows one to detect boundary-information loss.

Empirical results confirm the hypothesis: both Fourier and attention-based operators show monotonic decay in geometric information unless intervened upon.

Geometry Memory Injection: Mechanism and Variants

To counteract geometric forgetting, the authors propose geometry memory injection: explicit architectural pathways that reintegrate geometric signals into the network at arbitrary depths, thus breaking the strict Markovian flow.

Memory is injected through diverse mechanisms:

• Affine modulation (FiLM layers),
• Additive injection (learned projections of $G$),
• Concatenation of geometric and hidden features.

Injection policies can be early (only in initial layers), late (final layers), single-layer, or full (every layer), allowing for isolation of depth-dependent behaviors. The design is architecture-agnostic and minimally intrusive.

Experimental Results: Forgetting, Recovery, and Shortcut Instabilities

Dataset and Evaluation Protocol

Benchmarks include steady-state and multiphysics flows in the FlowBench dataset (Navier-Stokes, NSHT), aerodynamic surrogacy over NACA airfoils (AirfRANS), and Darcy diffusion in changing polygonal domains. All tasks are framed to test zero-shot domain generalization: models are forced to extrapolate to unseen geometries.

Evidence for Forgetting and Restoration

Through geometric probes, the study observes rapid loss of geometry information in both FNO and Transolver architectures. Memory injection consistently restores fidelity. In FNO, spectral boundary features are preserved only with memory; in Transolver, macroscopic mask information is retained.

Figure 2: Impact of Memory Injection (LDC-NSHT). Without memory, FNO loses flow dynamics and Transolver ignores the obstacle. Injecting memory at all layers corrects these failure modes.

Strong numerical improvements are obtained. For instance, on the LDC-NSHT task, FNO with geometry memory injection reduces relative $L^2$ error from $4.11\mathrm{e}{-1}$ (no memory) to $2.39\mathrm{e}{-1}$ (all layers injected), a 41.8% reduction. Transolver shows even more dramatic gains, particularly under early or full injection strategies.

Robustness and Geometric Shortcut

A nuanced finding emerges from ablations on injection site. In FNO, performance gains are robust to memory injection depth; in attention-based operators (Transolver), late/final-layer injection causes catastrophic degradation (Geometric Shortcut). When geometry is first exposed at the final layer, the optimizer sidesteps the physics backbone, relying entirely on geometric signals—a form of shortcut learning leading to gradient collapse upstream.

Figure 3: FNO Layer-wise Gradient Ratios (LDC-NS). Meaningful gradients propagate through all layers when memory is injected, indicating stable learning.

Figure 4: Evidence of the Geometric Shortcut in Transolver. Final-layer injection results in all optimization focusing on the last layer, with earlier layers ignored—the optimizer exploits injected geometry to bypass the physics backbone.

Intrinsic vs. Extrinsic Memory: The Case of LNO

Laplace Neural Operator (LNO), by construction, encodes geometry through its pole-residue boundary treatment (non-periodic basis). Empirical results indicate negligible gains in LNO from memory injection—its intrinsic architecture retains geometry, confirming that memory injection targets a specific structural limitation unique to global-mixing architectures rather than acting as a generic booster.

Ablations and Encoder Mechanisms

Memory injection remains effective regardless of the specific geometry encoder (U-Net, DeepONet), highlighting that the critical factor is restoring access to geometric information, not the richness of the geometric representation. However, attention-based operators are sensitive to the injection type: FiLM is more stable than additive/concatenation, particularly under late injection.

Implications and Future Directions

The results establish that geometric memory is a persistent constraint, not simply a capacity or depth-related design preference. Explicit maintenance of geometric information is necessary for robust out-of-domain generalization in operator surrogates, especially as depth and complexity increase.

Practically, this finding changes the architectural priorities for deep operator learning—emphasizing principled geometry propagation over input-stage encoding or network size scaling. Theoretically, it raises open questions regarding the precise quantification of geometric forgetting, its interplay with optimization and expressivity, and possible connections to information bottleneck theory for operator surrogates.

Future work should address:

• Extension to time-dependent and non-steady-state problems,
• More sophisticated geometric injection strategies and regularizers,
• Theoretical bounds on forgetting and memory restoration,
• Application to architectures with irregular meshes and graph-based operators.

Conclusion

This paper formalizes and empirically validates the Geometric Forgetting Hypothesis for deep neural operators, identifies the failure mode through both macroscopic and boundary information loss, and provides a corrective architecture-agnostic solution via geometry memory injection. The results clarify that robust generalization in neural operator surrogacy is predicated not only on sophisticated geometric encodings but on principled information preservation throughout depth, demanding a new generation of operator architectures with actively managed geometric memory (2605.05862).