Multi-Fidelity Residual Learning

• Multi-fidelity residual learning is a paradigm that decomposes high-fidelity outputs into a low-fidelity approximation and a learned residual correction to capture discrepancies.
• It employs diverse architectures such as neural networks and Gaussian processes, with physics-informed penalties and regularization to enhance model accuracy.
• Leveraging scarce high-fidelity data for residual calibration, the approach achieves low error rates and robust uncertainty quantification across applications like PDE solving and robotic state estimation.

Multi-fidelity residual learning is a machine learning paradigm designed to fuse information from simulation or measurement sources with varying levels of accuracy (“fidelity”) in order to produce accurate, data-efficient, and uncertainty-aware surrogate models. Instead of modeling the mapping from inputs to high-fidelity outputs directly, these schemes decompose the prediction into an initial approximation given by a low-fidelity model and a residual correction that captures the discrepancy between low- and high-fidelity information. The residual is typically parameterized and learned from scarce high-fidelity data, with rigorous approaches to regularization, physics-informed penalty embedding, and uncertainty quantification. Multi-fidelity residual learning has been adopted in domains such as simulation-based scientific inference, physical surrogate modeling, machine-learned interatomic potentials, and robotic state estimation, and is now central to scalable multi-fidelity surrogate model construction in both deterministic and probabilistic settings.

1. Fundamental Principles, Mathematical Formulation, and Motivation

The defining principle of multi-fidelity residual learning is the additive or corrective ansatz: for a given input $x$,

$f_\text{HF}(x) = f_\text{LF}(x) + r(x) \,,$

where $f_\text{LF}(x)$ is the output of a low-fidelity model (e.g., fast simulation, coarse measurement, reduced-order model), $f_\text{HF}(x)$ is the desired high-fidelity output, and $r(x)$ is the residual function mapping the input (often including $f_\text{LF}(x)$ itself) to the discrepancy between HV and LF models. This structure appears in classical Gaussian process AR(1) co-kriging (Raissi et al., 2016), GP-based additive residual models (Xing et al., 2021), neural-process surrogates (Niu et al., 2024), distribution-conditioned foundation models (Yu et al., 29 Jan 2026), as well as deep neural network architectures (Chen, 2023, Davis et al., 2023, Saadat, 2023, Yi et al., 2024, Imanov, 1 Feb 2026).

The rationale for residual learning is error–complexity tradeoff: the residual $r(x)$ can be much “simpler” (in terms of norm or model complexity) than $f_\text{HF}$ itself, and thus can be learned efficiently from a much smaller set of high-fidelity data (Davis et al., 2023). This is supported by error–complexity bounds for ReLU networks: when the residual is uniformly small, approximate learning requires a smaller and more data-efficient network.

Table 1 organizes characteristic residual-learning frameworks by model type:

Framework Class	LF Model	Residual Model	Notable Features
Additive GP (ResGP, AR(1))	GP	GP	Closed-form posterior, error bounds, modular training (Xing et al., 2021, Raissi et al., 2016)
Deep NNs (MFNN, RMFNN)	DNN/ROM	DNN/ResNet	Nonlinear/coupled, learns nonadditive or high-dim residuals (Davis et al., 2023, Chen, 2023)
Neural Processes (MFRNP)	NP	NP (residual branch)	Scalable to high dimension, explicit decoder aggregation (Niu et al., 2024)
Foundation Model Surrogates	Tabular FM	TFM (in context, Bayes)	Full distributional summaries, training-free (Yu et al., 29 Jan 2026)

2. Model Architectures and Residual Parameterizations

In multi-fidelity residual learning, the model is architected so that the low-fidelity stage provides an initial estimate, and a distinct model (often neural network or GP-based) is trained to map a suitably augmented input (which can include raw input, LF prediction, gradients, or other summaries) to the residual.

• Neural networks: Standard multifidelity NN schemes use two networks: a LF network $f_L(x)$ trained on abundant simulation or proxy data, and a residual/correction network $r(x)$ trained on limited high-fidelity observations. In many modern settings, $r(x)$ is implemented as a deep, possibly physics-regularized network that receives as input either $(x, f_L(x))$ or $(x, f_L(x), \nabla_x f_L(x))$ (Chen, 2023, Saadat, 2023). Committee models (ensemble of residuals under perturbed physical assumptions) further improve generalization (Chen, 2023).
• Residual Gaussian Processes: Additive multi-level structure $f_F(x) = f_1(x) + \sum_{i=2}^F \delta_i(x)$, where each $\delta_i(x)$ is a GP for the difference between subsequent fidelities, yields modular inference and provably calibrated uncertainty (Xing et al., 2021).
• Neural Processes & NP Ensembles: The residual is represented via latent neural process models, often taking as input aggregates of the decoded lower-fidelity surrogates (Niu et al., 2024, Hunter et al., 11 Nov 2025).
• Hybrid and Bayesian Variants: Adaptive residual learning architectures mix linear and nonlinear residual branches with input-dependent gating (learned or physics-prescribed) for maximum modeling flexibility (Imanov, 1 Feb 2026).
• Foundation Models: FIRE introduces a two-stage, distribution-conditioned, in-context residual learning pipeline, using the mean, variance, and quantiles of the pre-trained LF TFM as features for the zero-shot HF correction, providing strong heteroscedastic error handling without retraining (Yu et al., 29 Jan 2026).
• Graph Neural Networks & Physical Surrogates: In interatomic potentials and related molecular modeling, the residual is parameterized implicitly via per-fidelity parameter blocks or output heads within a shared equivariant GNN (Kim et al., 2024).

3. Training Protocols, Regularization, and Physics Constraints

Training a multi-fidelity residual model involves solving first the low-fidelity approximation problem on large simulated datasets, then calibrating the residual/correction model using scarce high-fidelity data.

Key stages:

• Pre-train LF model to high accuracy on synthetic or reduced-solver data; freeze or selectively fine-tune its weights in subsequent stages.
• Train the residual model on high-fidelity (or real-world) data, often with only a handful of calibration points (Chen, 2023).
• Alternate optimization between simulation-derived (mid-level) data and true high-fidelity data when practical.
• Embed domain knowledge or physical constraints in the residual/correction loss: e.g., stress-strain monotonicity, nonnegative hardening, conservation of physical quantities, or constitutive response limits (Chen, 2023, Imanov, 1 Feb 2026).
• Regularization (weight decay, dropout, early stopping, L1 penalty) is used to prevent overfitting in the high-fidelity-scarce regime.
• Stage-wise or adaptive gating (with explicit input-dependent mixing between linear and nonlinear residual branches) further stabilizes the mapping between fidelities (Imanov, 1 Feb 2026).

Uncertainty quantification is often handled by Bayesian treatments of the residual (e.g., GP, BNN, variational NPs), conformal prediction, or explicit propagation of predictive variance from the LF model through the residual mapping (Xing et al., 2021, Yi et al., 2024, Yu et al., 29 Jan 2026, Niu et al., 2024, Hunter et al., 11 Nov 2025).

4. Applications and Empirical Outcomes

Multi-fidelity residual learning has been successfully applied in a wide range of scientific and engineering domains:

• Physics-driven inverse problems: Optical imprint to stress-strain mapping in elastoplastic materials leverages a simulator-trained NN and a calibrated residual correction, achieving relative errors ≲3–4% with only three real calibration shots (Chen, 2023).
• Parametric PDE surrogates: Adaptive residual frameworks with nonlinear gating and Bayesian UQ deliver high-fidelity PDE solutions with sample efficiency and rigorous uncertainty estimates (Imanov, 1 Feb 2026).
• Robotic state estimation: Online multi-fidelity residual neural-processes outperform deep Kalman filter baselines in real-time, safety-critical settings (lower RMSE and calibrated uncertainty bounds) (Hunter et al., 11 Nov 2025).
• High-dimensional surrogate modeling: MFRNP achieves ≈90% lower nRMSE than previous deep multi-fidelity methods across PDE and climate-modeling benchmarks by explicitly aggregating decoded lower-fidelity outputs before residual estimation (Niu et al., 2024).
• Interatomic potentials: A single GNN with per-fidelity branches or bias/scale layers captures both LF and HF energy-force-stress predictions; significant accuracy and transferability gains are observed relative to transfer learning and $\Delta$-learning in molecular/solid-state applications (Kim et al., 2024).
• Reduced-order modeling: DeepONet-based multifidelity residual learning provides 3–5× error reduction over pure ROM or DeepONet surrogates in Navier-Stokes and parametric algebraic benchmark problems (Demo et al., 2023).
• Tabular regression and foundation models: FIRE achieves best-in-class normalized RMSE and NLL across 31 real-world and synthetic MF regression tasks, demonstrating stable performance even under extreme HF-scarcity (Yu et al., 29 Jan 2026).

5. Model Selection, Practical Considerations, and Limitations

Multi-fidelity residual learning frameworks must be adapted to the specifics of the application domain, data regime, and modeling objectives.

Best practices and guidelines include:

• Ensure LF model is maximally accurate and physically faithful on synthetic/surrogate data before calibration.
• Select residual model capacity (GP, NP, DNN) commensurate with discrepancy magnitude and data volume.
• When possible, use domain-informed features and physics constraints in the residual loss to enforce admissible corrections (monotonicity, physical bounds).
• Use Bayesian or conformal methods for uncertainty quantification, especially when predictive intervals or coverage guarantees are required.
• In high-dimensional or out-of-distribution regimes, include fidelity-specific decoder blocks and/or attention to prevent cross-fidelity information leakage (Niu et al., 2024).
• Empirically, three-shot (i.e., three calibration examples) was found optimal in sim-to-real parameter identification; more can risk overfitting, fewer can under-correct (Chen, 2023).

Common limitations and potential pitfalls:

• The additive residual ansatz may fail if the LF/HF relation is strongly non-additive or the discrepancy is not regular (e.g., highly non-smooth or context-dependent).
• Cross-fidelity residuals can be underdetermined without sufficiently informative LF features.
• For classical ResGP, interaction across residuals is ignored; for neural/NP-based methods, OOD generalization can fail if lower-fidelity decoders are not trained to interpolate at HF inputs (Niu et al., 2024).
• In tabular foundation models, performance depends on the meta-prior and context size; large-scale in-context residual learning may demand significant computational resources (Yu et al., 29 Jan 2026).

6. Extensions, Hybrid Schemes, and Current Research Directions

Recent developments have broadened the scope of multi-fidelity residual learning to hybrid and hierarchical ensembles, heterogeneous tasks, and adaptive physical embedding.

• Hybrid Bayesian-Deterministic Models: Three-stage schemes pairing deterministic surrogates with Bayesian residual models combine interpretability, uncertainty quantification, and data efficiency (e.g., KRR–linear–GPR or DNN–linear–BNN) (Yi et al., 2024).
• Hierarchical multi-task multi-fidelity GPs exploit shared residual structure across related but distinct tasks, using fidelity-dependent heteroscedastic noise models and hierarchical Bayesian parameter estimation to jointly reduce data requirements and suppress fidelity-dependent noise (Mehta et al., 10 Mar 2026).
• Gradient-enhanced multifidelity residuals employ automatic differentiation to inject local slope information from the LF model, markedly improving accuracy in problems with phase-shifted or nonlinearly transformed discrepancy (Saadat, 2023).
• Unified GNN-based multifidelity surrogates with per-fidelity weights enable seamless extension to >2 fidelity levels and transfer to much more expensive ab initio methods (e.g., CCSD(T)), outperforming baseline and transfer learning approaches in ab initio molecular dynamics and crystal property prediction (Kim et al., 2024).
• Operator learning with DeepONet and similar architectures extends multifidelity residual learning to the mapping between function spaces or trajectories as required in nonlinear physics problems (Demo et al., 2023).

A plausible implication is that multi-fidelity residual learning will continue to expand in both theoretical and applied scope due to its modularity, scalability, and compatibility with both probabilistic and deep learning approaches, as well as its provable efficiency when high-fidelity data is limited or prioritizes uncertainty quantification.