Autoregressive Deep-Learning Surrogates

• Autoregressive deep-learning surrogates are neural models that iteratively reproduce the evolution of complex dynamical systems using recursive, data-driven predictions.
• They employ architectures like MLPs, U-Nets, and Fourier Neural Operators to capture both spatial and temporal patterns for accurate multi-step forecasting.
• Practical deployment involves mitigating error accumulation through scheduled training, residual parameterizations, and ensemble strategies for robust uncertainty quantification.

Autoregressive deep-learning surrogates are neural models designed to emulate the time evolution of complex dynamical systems by recursively predicting future states from past predictions. In these frameworks, the surrogate is trained to forecast system states at the next time step, then used in an autoregressive (recursive) fashion to roll out trajectories many steps ahead, serving as a scalable, data-driven replacement for computationally intensive numerical solvers of systems governed by partial differential equations (PDEs), dynamical simulators, or physics-based models.

1. Mathematical Formulation and Recurrence Structure

The defining property of autoregressive deep-learning surrogates is their use of their own predictions as inputs when iterating forward. Consider a dynamical system evolving via an operator $\mathcal{M}$ (e.g., a simulator or a discrete-time PDE integrator): $x_{t+1} = \mathcal{M}(x_t, \theta),$ where $x_t$ is the system state, and $\theta$ may include control or parameter fields.

A typical deep surrogate $f_\phi$ is trained to approximate this map: $\hat{x}_{t+1} = f_\phi(\hat{x}_t, \theta),$ and is recursively rolled out: $$\hat{x}_{t+1} = f_\phi(\hat{x}_t, \theta), \quad \hat{x}_{t+2} = f_\phi(\hat{x}_{t+1}, \theta), \ldots$$ As in (Chen et al., 2019), this recurrence may include richer inputs such as a sequence history $[x_{t-k}, \ldots, x_t]$ (to capture temporal context) or parameter embeddings. For systems with spatial fields (e.g., $u(x,y,t)$), the input and output are tensors (e.g., $u^{t-k:t}\in\mathbb{R}^{k+1 \times N_x \times N_y}$).

Table: Core Recurrence Patterns in Deep Autoregressive Surrogates

Surrogate Type	Recurrence Equation	Input Features
DRN/MLP (Chen et al., 2019)	$\hat{z}_t = H(\theta, \hat{z}_{t-1})$	Parameters, last output
U-Net/Conv (Nguyen et al., 13 Jan 2026)	$\Delta u^{n+1}=f_{\theta}(u^{n-L+1},\ldots,u^{n})$	Sequence of past $L$ frames
FNO/Operator (Sun et al., 22 Aug 2025)	$\hat{x}_{t+1} = f_\theta(\hat{x}_t, \kappa)$	Last field, parameter map

This structure is universal across surrogates for time-dependent scientific systems (Ji et al., 5 Nov 2025, Khurjekar et al., 5 Jul 2025, Vlachas et al., 2023), with specific choices of history length, parameterization, and field representation reflecting the underlying physics.

2. Model Architectures and Training Methodologies

Surrogate architectures span a range of deep-learning models, including feed-forward MLPs, convolutional encoder–decoders (e.g., U-Net), operator-learning networks (FNOs), temporal/self-attention backbones, and recurrent schemes (ConvRNN, LSTM). Key architectural strategies are:

• Feed-forward DRN: For parameterized simulations (e.g., Bayesian surrogate inversion (Chen et al., 2019)), a compact DNN takes $(\theta, \hat{z}_{t-1})$ as input and predicts $\hat{z}_t$, sharing weights across time.
• Fully convolutional (FCN/U-Net): For forecasting 2D/3D fields, convolutional U-Net backbones with skip connections enforce spatial locality and multi-scale context (Nguyen et al., 13 Jan 2026, Ji et al., 5 Nov 2025).
• Operator-learning: Fourier Neural Operators (FNO) and AFNO/Transformers embed global receptive fields and fast spectral convolution, suitable for high-resolution physical fields (Sun et al., 22 Aug 2025).
• Temporal attention: Deep ensembles with temporal self-attention blocks and residual connections can enhance temporal context aggregation and long-term feature propagation (Khurjekar et al., 5 Jul 2025).
• Recurrent Networks: ConvRNN, ConvLSTM, and convolutional autoencoder–RNN hybrids treat each frame as input, propagating hidden representations across time, often regularized by scheduled autoregressive BPTT (Vlachas et al., 2023).

Training regimes universally optimize for one-step or multi-step mean-squared error (MSE) losses, often augmented with spectral or perceptual terms. For spatiotemporal surrogates, physics-based regularization (e.g., mass/energy penalties) and history-augmentation (e.g., random noise injection or mixup) are used to improve roll-out stability (Ji et al., 5 Nov 2025, Nguyen et al., 13 Jan 2026). In convolutional surrogates, explicit domain boundary conditions (e.g., periodic padding) are enforced at architecture level (Nguyen et al., 13 Jan 2026).

3. Challenges in Autoregressive Inference: Error Accumulation, Generalization, and Exposure Bias

A central challenge is the "exposure bias" or iterative error accumulation: small inaccuracies at each autoregressive step can propagate and amplify, leading to deviation from physically plausible trajectories. Standard teacher-forcing training, where only true states are fed as inputs, fails to regularize the surrogate for the distribution of its own predictions encountered during long autoregressive rollouts (Vlachas et al., 2023).

One effective mitigation is scheduled autoregressive BPTT (BPTT-SA), which gradually replaces ground-truth inputs with model predictions during training, annealing from pure teacher forcing to fully autoregressive supervision. The loss is a convex combination of one-step and recursive errors, and the annealing schedule (e.g., inverse-sigmoid) controls the rate of transition (Vlachas et al., 2023).

Error propagation can be further controlled via architectural and training strategies:

• Residual parameterization: Predicting increments ($\Delta u_{t+1}$) instead of full states focuses the model on learning physically plausible corrections, aiding stability across long rollouts (Nguyen et al., 13 Jan 2026).
• Physics-informed losses: Inclusion of soft constraints on conserved quantities or symmetry (e.g., total mass, four-fold symmetry error) sharpen long-term fidelity (Ji et al., 5 Nov 2025, Amarel et al., 18 Aug 2025).
• Input noise augmentation: Training with Gaussian noise on inputs can make the surrogate robust to its own imperfect rollouts (Ji et al., 5 Nov 2025).

Quantitative metrics for diagnosing regime include "trust horizon" (maximum prediction horizon before metrics degrade), field-space RMSE, spectral similarity, and metrics for physics violation (mass, energy, symmetry) (Nguyen et al., 13 Jan 2026, Ji et al., 5 Nov 2025, Amarel et al., 18 Aug 2025).

4. Ensembling, Uncertainty Quantification, and Sensitivity

Aggregation of multiple independently trained surrogate models via ensembling significantly mitigates error accumulation and quantifies epistemic uncertainty (Khurjekar et al., 5 Jul 2025, Sun et al., 22 Aug 2025). In ensemble strategies:

• Independent surrogates $\{\mathcal{M}_i\}$, trained with different random initializations or hyperparameters, are rolled out in parallel.
• At each step, predictions are aggregated via arithmetic mean: $$\overline{u}^{t+1} = \frac{1}{N}\sum_{i=1}^{N} \mathcal{M}_i(u^{t-k:t}; \theta_i)$$.
• Ensemble outputs exhibit reduced error variance due to cancellation of uncorrelated errors, with the MSE ideally reducing as $1/N$ in the uncorrelated limit (Khurjekar et al., 5 Jul 2025).

This framework not only improves forward predictive fidelity but enables adjoint-mode computation of parametric sensitivities and provides uncertainty estimates for both function values and derivatives (Sun et al., 22 Aug 2025). Ensemble methods require no modification to base architectures or training routines, but demand increased computational/storage resources.

Table: Empirical Performance of Ensemble Strategies (Khurjekar et al., 5 Jul 2025, Sun et al., 22 Aug 2025)

System	Single-Model Error	Ensemble Error	Speed-up
J$_2$-plasticity field	RLE: 0.120–0.137	RLE: 0.0987	%%%%18$L$19%%%%
Gray–Scott PDE	MAE: 0.095–0.12	0.085	%%%%20$x_t$21%%%%
Ocean modeling (FNO)	RMSE: 0.0011–0.049	0.0004–0.039	OOM faster

5. Quantitative Evaluation and Generalization Diagnostics

Standard evaluation of autoregressive deep surrogates proceeds on several axes:

• Field-space RMSE: Pixel- or pointwise deviation from ground-truth, averaged over test trajectories (Nguyen et al., 13 Jan 2026, Ji et al., 5 Nov 2025).
• Spectral fidelity: Cosine similarity between 2D Fourier power spectra of predicted and ground-truth fields, diagnosing resolution of multi-scale features (Nguyen et al., 13 Jan 2026).
• Physics-aware errors: Deviation in conserved or derived quantities (mass, energy, morphology metrics, tip-selection constants, symmetry) (Ji et al., 5 Nov 2025, Nguyen et al., 13 Jan 2026).
• Trust horizon: Maximum length $T^*$ such that two-time gradient influence or physics metrics remain in regime before breakdown (Amarel et al., 18 Aug 2025).
• Generalization to OOD conditions: Testing robustness to novel initializations or parameter regimes (e.g., sparse nucleation, new pattern morphology) (Nguyen et al., 13 Jan 2026).

For example, me-UNet (Nguyen et al., 13 Jan 2026) achieves RMSEs as low as 0.0004 on canonical 2D advection–diffusion datasets, matching or outperforming transformer/operator surrogates while being more robust in small-data and OOD settings. The ADS (Ji et al., 5 Nov 2025) demonstrates $>$100$\times$ speed-up for dendrite growth and $\sim$5% accuracy in physically determined tip-selection constants relative to detailed phase-field benchmarks.

6. Scaling, Data Efficiency, and Inductive Bias Analysis

Scalability is addressed in two ways:

• Spatial scaling: Surrogates with translation-equivariant architectures (e.g., fully convolutional SI-ConvNeXt (Ji et al., 5 Nov 2025)) extend directly to arbitrarily large domains with minimal accuracy loss, supporting zero-shot large-scale generalization.
• Small-data regime: Models with strong inductive biases toward locality and domain topology (e.g., periodic padding in U-Nets) require only $\sim$20 training simulations to approach full-data accuracy (Nguyen et al., 13 Jan 2026).
• Multi-scale learning: Deep encoder–decoder paths, residual outputs, and explicit inclusion of PDE symmetries/physics features improve data efficiency and generalization to OOD conditions (Nguyen et al., 13 Jan 2026, Amarel et al., 18 Aug 2025).

Grad-CAM analyses confirm that, in well-designed convolutional surrogates, shallow layers focus on local field features (fronts, patterns), while deeper layers encode global or semantic contexts, paralleling the multi-scale nature of physical transport and reaction mechanisms (Nguyen et al., 13 Jan 2026).

7. Best Practices and Open Challenges

Effective deployment of autoregressive deep-learning surrogates relies on:

• Exposure-bias mitigation: Use scheduled BPTT or noise-injection/mixup to regularize for self-generated state distributions (Vlachas et al., 2023, Ji et al., 5 Nov 2025).
• Ensemble strategies: Apply ensemble aggregation to reduce error growth and enable credible uncertainty quantification (Khurjekar et al., 5 Jul 2025, Sun et al., 22 Aug 2025).
• Physics-aware objectives: Incorporate loss terms or constraints reflecting system invariants to prevent long-term forecast drift (Amarel et al., 18 Aug 2025, Ji et al., 5 Nov 2025).
• Inductive-bias alignment: Choose architectures respecting problem symmetries and boundary conditions (e.g., periodicity, locality) to increase data efficiency and OOD robustness (Nguyen et al., 13 Jan 2026).
• Quantitative monitoring: Evaluate multi-step, OOD, and physics-based metrics, not just one-step RMSE, to faithfully diagnose surrogate reliability and trustworthiness (Amarel et al., 18 Aug 2025, Nguyen et al., 13 Jan 2026).

Open issues include diversity and scalability of ensemble approaches, online adaptation to new data regimes, and systematic quantification of trust horizons. Approaches leveraging influence function diagnostics and temporal coherence analysis offer actionable frameworks for model selection and improvement (Amarel et al., 18 Aug 2025).

---

Key references: (Chen et al., 2019, Vlachas et al., 2023, Khurjekar et al., 5 Jul 2025, Sun et al., 22 Aug 2025, Amarel et al., 18 Aug 2025, Ji et al., 5 Nov 2025, Nguyen et al., 13 Jan 2026).