Orthogonal Discrepancy Kernels for Learning with Partial Physics

Abstract: We introduce a semi-parametric framework for nonlinear system identification, which decouples discrepancy functions from physics-based components. Orthogonal Gaussian process regression balances sparse parameter selection (the white box) with discrepancy learning (the black box) to produce interpretable models from incomplete physics.

• The paper introduces an orthogonal discrepancy kernel that separates physics-driven basis functions from a data-driven Gaussian Process to ensure model interpretability.
• It demonstrates how enforcing orthogonality improves predictive accuracy and parameter identifiability, even when the basis function dictionary is incomplete.
• Numerical experiments on a nonlinear oscillator show lower NMSE and effective decomposition between physical dynamics and unmodeled discrepancies.

Orthogonal Discrepancy Kernels for Nonlinear System Identification with Partial Physics

Introduction and Motivation

The paper "Orthogonal Discrepancy Kernels for Learning with Partial Physics" (2606.21199) interrogates the persistent challenge in nonlinear dynamical system identification: inferring interpretable, physically-consistent models from data when prior knowledge of the underlying physics is incomplete. The paradigm integrates a sparse regression approach—anchored in physics-driven basis functions (SINDy)—with nonparametric Gaussian Process (GP) modeling to capture discrepancies arising from missing or imperfect prior structure. This orthogonal decomposition enables explicit enforcement of model interpretability and parsimony, while flexibly accommodating unmodeled dynamics.

Related Methods and Shortcomings

Traditional SINDy methods address the identification of ODEs by selecting a parsimonious subset of candidate basis expansions via sparse regression, assuming the candidate dictionary encompasses all influential latent dynamics. The critical limitation emerges when the dictionary is incomplete: basis omission induces bias, as the model cannot fit dynamics absent from its expressivity. While grey-box approaches augment the parametric model with a flexible black-box component (such as a GP), these typically lack incentives for the machine-learner to respect or defer to physically-informed constituents, often resulting in diminished interpretability and excessive absorption of the signal by the black box.

Orthogonality-Constrained Discrepancy Modeling

The core innovation is the introduction of orthogonal discrepancy kernels in the GP component. Given a parametric component $\Phi(\mathbf{x})^\top \boldsymbol{\theta}$ (with basis functions $\Phi$ and sparse coefficients $\boldsymbol{\theta}$), a GP prior is placed on the residual $\tilde{f}(\mathbf{x})$ and the kernel is projected onto the orthogonal complement of $\mathrm{span}(\Phi)$. Formally, for a chosen base kernel $k(\mathbf{x}, \mathbf{x'})$ and dictionary Gram matrix $\mathbf{H}$:

$$k_\perp(\mathbf{x}, \mathbf{x}') = k(\mathbf{x}, \mathbf{x}') - \Phi(\mathbf{x})^\top \mathbf{H}^{-1} \Phi(\mathbf{x}')$$

This constraint enforces that the GP can only account for discrepancies unexplainable by the dictionary, securing identifiability and interpretability of the parametric coefficients $\boldsymbol{\theta}$.

Numerical Demonstration: Nonlinear Oscillator

A canonical testbed is constructed using a forced oscillator governed by $\ddot{y} + c\dot{y}^3 + k y^3 = 0$, with nonlinear damping and cubic stiffness. The state-space is constructed as $\Phi$0 and synthetic acceleration data $\Phi$1 is generated numerically.

Figure 1: The simulation shows acceleration $\Phi$2 with pronounced nonlinear oscillations, a setting sensitive to dictionary completeness.

Two experimental settings are explored: (i) a complete dictionary including all relevant nonlinear basis terms (e.g., $\Phi$3), and (ii) a partial dictionary omitting key terms (e.g., removing $\Phi$4 to simulate partial physics).

Complete Dictionary: Interpretability Versus Greediness

Application of the non-orthogonal GP (standard SE kernel) shows that the black-box component greedily appropriates the cubic term, even when this is present as a basis function, resulting in diminished significance of $\Phi$5 and reduced interpretability.

Figure 2: Non-orthogonal GP with complete dictionary; right: the GP greedily learns the cubic term, overshadowing dictionary-based identification.

The orthogonal GP, in contrast, suppresses this redundancy by enforcing strict complementarity. Consequently, the dictionary coefficients accurately activate physical terms, and the GP residual is minimal.

Figure 3: Orthogonal GP with complete dictionary; right: the discrepancy process is forced to zero in regions well-captured by the dictionary, ensuring parameter interpretability.

Quantitative results (NMSE: $\Phi$6 for orthogonal GP versus $\Phi$7 for non-orthogonal GP) reflect the improved decomposition between physical and nonphysical structure and more interpretable parameter estimation.

Partial Dictionary: Compensation for Missing Physics

To interrogate robustness under incomplete prior knowledge, the cubic term is omitted from the dictionary. Both orthogonal and non-orthogonal models flexibly allocate the missing nonlinear dynamics to the GP.

Figure 4: Orthogonal GP with partial dictionary; the GP discrepancy compensates for the omitted $\Phi$8 term while dictionary coefficients reflect their best proxy activation.

Figure 5: Non-orthogonal GP with partial dictionary; again, the GP models the missing structure, but identifiability for the present bases is degraded.

In this setting, the orthogonal GP still activates proxy basis functions in the dictionary but assigns the missing structure exclusively to the GP. NMSE further decreases (orthogonal: $\Phi$9, non-orthogonal: $\boldsymbol{\theta}$0), confirming strong predictive alignment.

Practical and Theoretical Implications

The proposed orthogonal discrepancy GP framework explicitly regularizes the division of labor between physically-motivated bases and black-box machine learners, critical for credible system identification in scientific domains where interpretability, extrapolability, and faithfulness to physics are non-negotiable. The orthogonality constraint guards against signal overfitting by the data-driven component—an issue exacerbated with highly expressive learners—making it suitable for hybrid discovery in contexts such as scientific machine learning, digital twins, and model-based control. This improvement in interpretability comes with no loss in predictive accuracy and, in cases of an incomplete dictionary, robustly enables discovery of missing dynamics.

Theoretically, this work generalizes physics-informed learning by formalizing the requirement that the black-box hypothesis space be the orthogonal complement of the span of known physics. This formal separation, implementable with efficient kernel projection via the dictionary's Gram matrix, holds promise for any scenario coupling parametric priors with kernel-based nonparametric inference.

Conclusion

The orthogonal discrepancy kernel framework systematically disentangles physical and non-physical structure in nonlinear system identification, simultaneously enhancing the interpretability and fidelity of hybrid grey-box models. By enforcing orthogonality of GP discrepancies to the physics-informed dictionary, the method addresses the critical challenge of model selection under partial-physics priors and curtails the common pathologies of greedy machine learning components. Future directions include extending the orthogonalization principle to distributed systems (PDEs), integrating learned dictionaries, and scaling to high-dimensional dynamical systems.