Neural Active Manifold (NeurAM)

• NeurAM identifies one-dimensional nonlinear active manifolds via neural autoencoding to capture essential model output variability for surrogate modeling.
• It balances manifold reconstruction, surrogate fidelity, and latent consistency losses to enable sensitivity analysis and significant multifidelity variance reduction.
• Empirical results demonstrate reduced surrogate MAE and enhanced Monte Carlo efficiency in complex cases like magnetohydrodynamics and cardiac electrophysiology.

Neural Active Manifold (NeurAM) refers to a class of methodologies that perform nonlinear, task-aware dimensionality reduction for uncertainty quantification and surrogate modeling in the context of computationally expensive scalar-output models. The approach leverages neural network-based autoencoding structures or task-supervised encoders to discover low-dimensional, typically one-dimensional, "active manifolds" constituting the principal directions of model output variability. By learning these manifolds and fitting efficient surrogates on them, NeurAM enables tractable sensitivity analysis, uncertainty propagation, and multifidelity variance reduction, with provable statistical guarantees and efficiently addresses sample complexity in high-dimensional settings (Zanoni et al., 2024, Cheng et al., 26 Jun 2025).

1. Mathematical Foundations and Problem Statement

Let $f:\mathbb{R}^d\to\mathbb{R}$ denote a computationally expensive scalar-valued model. NeurAM seeks to identify a nonlinear low-dimensional representation $e_\psi:\mathbb{R}^d\to\mathbb{R}$ (the encoder, typically with bottleneck dimension one), a decoder $g_\theta:\mathbb{R}\to\mathbb{R}^d$ mapping points in latent space to the input domain, and a surrogate $h_\phi:\mathbb{R}\to\mathbb{R}$ that approximates $f$ restricted to the learned active manifold.

The composite objective balances three desiderata:

• Manifold reconstruction:

$$L_{\mathrm{rec}}(\psi,\theta) = \mathbb{E}_{x\sim\mu} \|x - g_\theta(e_\psi(x))\|^2$$

• Surrogate fidelity:

$$L_{\mathrm{sur}}(\psi,\phi) = \mathbb{E}_{x\sim\mu} |f(x) - h_\phi(e_\psi(x))|^2$$

• Latent consistency:

$$L_{\mathrm{coh}}(\psi,\theta) = \mathbb{E}_{x\sim\mu} |e_\psi(g_\theta(e_\psi(x)))-e_\psi(x)|^2$$

The aggregate minimization is:

$$\mathcal{L}(\psi, \theta, \phi) = L_{\mathrm{rec}} + L_{\mathrm{sur}} + L_{\mathrm{coh}}$$

Expectations are replaced in practice via Monte Carlo on $N$ pilot samples $e_\psi:\mathbb{R}^d\to\mathbb{R}$0 and optimization is performed over the neural network parameters $e_\psi:\mathbb{R}^d\to\mathbb{R}$1, $e_\psi:\mathbb{R}^d\to\mathbb{R}$2, $e_\psi:\mathbb{R}^d\to\mathbb{R}$3 using stochastic gradient descent (e.g., Adam and L-BFGS) (Zanoni et al., 2024, Cheng et al., 26 Jun 2025).

2. Network Architectures and Training Algorithms

NeurAM is typically instantiated with the following architectural pattern:

• Encoder ($e_\psi:\mathbb{R}^d\to\mathbb{R}$4): Input $e_\psi:\mathbb{R}^d\to\mathbb{R}$5 two hidden ReLU layers (width 8–32) $e_\psi:\mathbb{R}^d\to\mathbb{R}$6 linear bottleneck $e_\psi:\mathbb{R}^d\to\mathbb{R}$7.
• Decoder ($e_\psi:\mathbb{R}^d\to\mathbb{R}$8): $e_\psi:\mathbb{R}^d\to\mathbb{R}$9 two hidden ReLU layers (width 8–32) $g_\theta:\mathbb{R}\to\mathbb{R}^d$0 linear output $g_\theta:\mathbb{R}\to\mathbb{R}^d$1.
• Surrogate ($g_\theta:\mathbb{R}\to\mathbb{R}^d$2): $g_\theta:\mathbb{R}\to\mathbb{R}^d$3 two hidden ReLU layers (width 8–32) $g_\theta:\mathbb{R}\to\mathbb{R}^d$4 linear output $g_\theta:\mathbb{R}\to\mathbb{R}^d$5.

Variations in architectures include deeper or wider networks (2–4 layers, widths 8–32), ReLU or tanh activations (the latter for bounded manifolds), and optional regularization (e.g., weight decay or dropout). Surrogate loss underfitting can be mitigated by increasing the surrogate loss weight.

Typical training uses Adam or L-BFGS optimizers, learning rates $g_\theta:\mathbb{R}\to\mathbb{R}^d$6–$g_\theta:\mathbb{R}\to\mathbb{R}^d$7, batch sizes 32–128, and early stopping on a 20% validation split. Hyperparameter selection can employ random or Bayesian searches, e.g., Optuna (Zanoni et al., 2024, Cheng et al., 26 Jun 2025).

3. Theoretical Guarantees and Statistical Properties

Theoretical results establish the existence of global minimizers for the NeurAM loss $g_\theta:\mathbb{R}\to\mathbb{R}^d$8 over measurable networks, by construction of right-inverses under mild functional assumptions. In the idealized regime (i.e., perfect manifold and surrogate recovery), NeurAM's shared one-dimensional latent variable across high- and low-fidelity models enables the reparametrization of multifidelity estimators to maximize Pearson correlation, provably yielding a reduction in the variance of Monte Carlo estimators.

More precisely, replacing the standard control-variates low-fidelity evaluation $g_\theta:\mathbb{R}\to\mathbb{R}^d$9 by a matched sample using the shared manifold,

$h_\phi:\mathbb{R}\to\mathbb{R}$0

the induced modified correlation $h_\phi:\mathbb{R}\to\mathbb{R}$1 obeys $h_\phi:\mathbb{R}\to\mathbb{R}$2 (with means and variances preserved), leading to statistical efficiency improvement in multifidelity Monte Carlo (Zanoni et al., 2024). Proofs employ rearrangement inequalities and explicit covariance analysis.

4. Multifidelity Sampling and Outer-Loop Applications

NeurAM systematically supports multifidelity Monte Carlo (MFMC) estimators. Classically, the MFMC control-variates estimator for $h_\phi:\mathbb{R}\to\mathbb{R}$3 combines high- and low-fidelity samples:

$h_\phi:\mathbb{R}\to\mathbb{R}$4

with optimal weights given in closed form and estimator variance explicitly characterized in terms of the correlation coefficient $h_\phi:\mathbb{R}\to\mathbb{R}$5.

By joint learning of the latent coordinate for high- and low-fidelity models and coupling sampling via the inverse-CDF mapping in the shared latent space, NeurAM increases $h_\phi:\mathbb{R}\to\mathbb{R}$6 to $h_\phi:\mathbb{R}\to\mathbb{R}$7 and thus enables variance reductions by orders of magnitude in test problems (e.g., $h_\phi:\mathbb{R}\to\mathbb{R}$8 yields a 10$h_\phi:\mathbb{R}\to\mathbb{R}$9 variance reduction in $f$0) (Zanoni et al., 2024). This paradigm is operationally realized via mapping new samples in $f$1 to the latent, resampling uniformly in latent space, and generating paired points via decoding for each fidelity.

5. Empirical Validation and Comparative Performance

NeurAM exhibits robust empirical advantages in nonlinear dimensionality reduction, surrogate construction, and multifidelity variance reduction:

• Analytical low-dimensional tests: On functions $f$2 and $f$3, NeurAM identifies the analytically correct manifold $f$4, achieving projection and surrogate MAE $f$5 for $f$6–$f$7 samples. Compared to Active Subspaces (AS) and Active Manifolds (AM), NeurAM attains lower test errors and does not require gradients (Zanoni et al., 2024).
• Multifidelity tests: Highly nonlinear two-dimensional functions with minimal standard correlation ($f$8) are transformed by NeurAM pairing to achieve $f$9, producing 10$$L_{\mathrm{rec}}(\psi,\theta) = \mathbb{E}_{x\sim\mu} \|x - g_\theta(e_\psi(x))\|^2$$0 variance reduction.
• Hartmann magnetohydrodynamics (d=5): For quantities of interest (QoI) such as velocity $$L_{\mathrm{rec}}(\psi,\theta) = \mathbb{E}_{x\sim\mu} \|x - g_\theta(e_\psi(x))\|^2$$1 and field $$L_{\mathrm{rec}}(\psi,\theta) = \mathbb{E}_{x\sim\mu} \|x - g_\theta(e_\psi(x))\|^2$$2, NeurAM achieves MAE $$L_{\mathrm{rec}}(\psi,\theta) = \mathbb{E}_{x\sim\mu} \|x - g_\theta(e_\psi(x))\|^2$$3 for $$L_{\mathrm{rec}}(\psi,\theta) = \mathbb{E}_{x\sim\mu} \|x - g_\theta(e_\psi(x))\|^2$$4 and $$L_{\mathrm{rec}}(\psi,\theta) = \mathbb{E}_{x\sim\mu} \|x - g_\theta(e_\psi(x))\|^2$$5 for $$L_{\mathrm{rec}}(\psi,\theta) = \mathbb{E}_{x\sim\mu} \|x - g_\theta(e_\psi(x))\|^2$$6, outperforming AS/AM.
• Cardiac electrophysiology (d=4, bifurcation): In complex bifurcating models with bimodal output, class-specific NeurAM models are fitted post-classification (97% accuracy), achieving per-class surrogate MAE $$L_{\mathrm{rec}}(\psi,\theta) = \mathbb{E}_{x\sim\mu} \|x - g_\theta(e_\psi(x))\|^2$$7 and aggregate MAE $$L_{\mathrm{rec}}(\psi,\theta) = \mathbb{E}_{x\sim\mu} \|x - g_\theta(e_\psi(x))\|^2$$8 (Zanoni et al., 2024).

In all regimes, NeurAM's ability to find 1D nonlinear embeddings is robust and practical with $$L_{\mathrm{rec}}(\psi,\theta) = \mathbb{E}_{x\sim\mu} \|x - g_\theta(e_\psi(x))\|^2$$9–$$L_{\mathrm{sur}}(\psi,\phi) = \mathbb{E}_{x\sim\mu} |f(x) - h_\phi(e_\psi(x))|^2$$0 samples for $$L_{\mathrm{sur}}(\psi,\phi) = \mathbb{E}_{x\sim\mu} |f(x) - h_\phi(e_\psi(x))|^2$$1. For higher-dimensional models, loss of manifold accuracy may signal the need for multidimensional latent spaces.

6. Extensions and Active Learning Formulation

Generalizations of NeurAM include multidimensional latent embeddings, addition of gradient penalties or orthogonality constraints for manifold uniqueness, and closed-loop integration with Bayesian inference. Further, the methodology can be extended to active learning settings, where the manifold encoder is learned jointly with a Gaussian process (GP) regressor in the latent space. This is achieved by minimizing the negative log marginal likelihood (NLML) of the GP over the mapped data, optionally coupled with regularization or reconstruction terms (Cheng et al., 26 Jun 2025).

Active sample selection uses the Active Learning Cohn (ALC) criterion, maximizing expected variance reduction in a reference set, with batch and candidate screening heuristics to control computational cost. This approach yields rapid RMSE decay compared to random selection, even in high dimensions (e.g., in an 8D borehole simulator, active NeurAM reached RMSE $$L_{\mathrm{sur}}(\psi,\phi) = \mathbb{E}_{x\sim\mu} |f(x) - h_\phi(e_\psi(x))|^2$$2 vs. random $$L_{\mathrm{sur}}(\psi,\phi) = \mathbb{E}_{x\sim\mu} |f(x) - h_\phi(e_\psi(x))|^2$$3 at $$L_{\mathrm{sur}}(\psi,\phi) = \mathbb{E}_{x\sim\mu} |f(x) - h_\phi(e_\psi(x))|^2$$4) (Cheng et al., 26 Jun 2025).

7. Recommendations for Practice and Outlook

Empirical and theoretical evidence suggests the following recommendations:

• For input dimension $$L_{\mathrm{sur}}(\psi,\phi) = \mathbb{E}_{x\sim\mu} |f(x) - h_\phi(e_\psi(x))|^2$$5, train NeurAM with $$L_{\mathrm{sur}}(\psi,\phi) = \mathbb{E}_{x\sim\mu} |f(x) - h_\phi(e_\psi(x))|^2$$6–$$L_{\mathrm{sur}}(\psi,\phi) = \mathbb{E}_{x\sim\mu} |f(x) - h_\phi(e_\psi(x))|^2$$7 samples for robust 1D manifold and surrogate accuracy (MAE $$L_{\mathrm{sur}}(\psi,\phi) = \mathbb{E}_{x\sim\mu} |f(x) - h_\phi(e_\psi(x))|^2$$8 1%).
• Start with 2–4 hidden layers of width 8–32, ReLU activations, linear final layers. Equal loss weighting is typical; increase surrogate weight if underfitting observed.
• Monitor manifold quality via projection and surrogate MAE, holdout validation, and parity plots.
• In multifidelity settings, train high- and low-fidelity encoders/decoders and sample in the latent space for well-correlated, variance-reducing pairings.
• Extensions include higher-dimensional latent spaces, penalized or regularized loss terms, and integration within active learning loops as in closed-loop Gaussian process regression (Zanoni et al., 2024, Cheng et al., 26 Jun 2025).

NeurAM provides a scalable, flexible nonlinear dimensionality reduction and uncertainty quantification tool for computational science, supporting both surrogate modeling and efficient outer-loop sampling across single- and multifidelity models. Its combination of theoretical guarantees, empirical efficiency, and extensibility positions it as a central methodology in modern computational UQ.