Interpretability and Generalization Bounds for Learning Spatial Physics (2506.15199v1)

Abstract: While there are many applications of ML to scientific problems that look promising, visuals can be deceiving. For scientific applications, actual quantitative accuracy is crucial. This work applies the rigor of numerical analysis for differential equations to machine learning by specifically quantifying the accuracy of applying different ML techniques to the elementary 1D Poisson differential equation. Beyond the quantity and discretization of data, we identify that the function space of the data is critical to the generalization of the model. We prove generalization bounds and convergence rates under finite data discretizations and restricted training data subspaces by analyzing the training dynamics and deriving optimal parameters for both a white-box differential equation discovery method and a black-box linear model. The analytically derived generalization bounds are replicated empirically. Similar lack of generalization is empirically demonstrated for deep linear models, shallow neural networks, and physics-specific DeepONets and Neural Operators. We theoretically and empirically demonstrate that generalization to the true physical equation is not guaranteed in each explored case. Surprisingly, we find that different classes of models can exhibit opposing generalization behaviors. Based on our theoretical analysis, we also demonstrate a new mechanistic interpretability lens on scientific models whereby Green's function representations can be extracted from the weights of black-box models. Our results inform a new cross-validation technique for measuring generalization in physical systems. We propose applying it to the Poisson equation as an evaluation benchmark of future methods.

• The paper establishes novel generalization bounds and convergence rates for various ML models applied to the 1D Poisson equation.
• It proves that finite difference parameter learning errors converge at rates determined by grid spacing, finite difference order, and the training function’s polynomial basis order.
• A new method for extracting Green’s function representations from black-box models is introduced, offering mechanistic insights for improved data-driven physics modeling.

Interpretability and Generalization Bounds for Learning Spatial Physics

This paper presents an analytical and empirical exploration of ML approaches applied to solving the 1D Poisson differential equation, focusing on the generalization capabilities and interpretability of these methods. The authors aim to establish theoretical foundations analogous to those in classical numerical analysis for ML models applied to spatial physics problems, emphasizing the implications of data discretization and the underlying function spaces on model generalization.

Summary of Methodology and Findings

The paper addresses both "white-box" models with known physical equations and "black-box" models, such as neural networks, aiming to generalize learned solution operators of the Poisson equation. The canonical Poisson problem serves as a test case due to its foundational role in computational physics. The authors derive generalization bounds and assess convergence rates by analyzing ML models' training dynamics for learning the Poisson equation. They demonstrate that naive training data sampling can lead to learned models failing to generalize beyond known data distributions. The research reveals that various ML models, including deep linear models, shallow neural networks, DeepONets, and Neural Operators, exhibit varying and often contradicting generalization behaviors under different conditions.

Key theoretical results include:

1. Finite Difference Parameter Learning: A proof is provided showing that the error in learned parameters for underparameterized models converges at rates dictated by grid spacing, finite difference order, and the training function's polynomial basis order.
2. Linear Model Generalization: Theoretically, it is shown that linear models learn the projection of the true operator onto the subspace formed by the training data, emphasizing that infinite data quantity does not guarantee operator learning unless the function basis spans the necessary subspace.
3. Deep Models and Function Generalization: The deep nonlinear models do not consistently generalize across different data distributions, confirming that increased model complexity does not inherently solve the generalization problem.
4. Mechanistic Interpretability: A novel approach is introduced for extracting Green's function representations from the weights of black-box models, providing insights into the mechanistic learning of scientific models.

Implications and Future Directions

The implications of this research are multifaceted, motivating new approaches to both modeling and dataset construction for ML in scientific applications. Practically, this suggests that the data collection processes should be strategically designed to enhance model generalization by representing a broad function space. Theoretically, the results encourage further exploration of methods to discover underlying function spaces from data, emphasizing the necessity to improve ML models' theoretical underpinning in scientific contexts.

The paper highlights a pressing challenge in data-driven scientific modeling: the inherent limitations of real-world data collection in achieving full generalization. This limitation calls for continued investigation into dynamically adaptable ML frameworks and robust validation techniques to address potential shortcomings in unseen conditions.

Looking ahead, future work is recommended to expand beyond the Poisson equation to more complex spatial physics equations and to real-world data applications, bridging the gap between theoretical insights and practical implementations. This paper lays the groundwork for a rigorous evaluation benchmark in perceptual learning systems for physical equations, prompting further innovation in ML approaches that can learn and generalize complex scientific phenomena effectively.