Neural Surrogates for Anisotropic Covariance

• The paper introduces neural surrogates that parameterize local anisotropic covariance in Gaussian processes using direct parameter regression and hyperparameter mapping.
• It leverages CNNs and deep feed-forward networks to output local length-scales and noise variances, enabling nonstationary adaptation in complex spatial domains.
• Empirical results demonstrate significant computational speedups and enhanced prediction accuracy compared to traditional maximum likelihood methods.

Neural surrogates for anisotropic covariance refer to the use of neural networks to learn or parameterize spatially or input-dependent, directionally varying covariance structures in Gaussian processes (GPs) and geostatistical models. These approaches address the limitations of stationary isotropic kernels by allowing local adaptation to the effective geometry of the data, thereby improving uncertainty quantification and prediction in heterogeneously distributed or geometrically complex input domains.

1. Foundations of Anisotropic Covariance in Spatial Models

Anisotropy in covariance functions denotes direction-dependent correlation decay, contrasting with isotropic models where dependence is only a function of Euclidean distance. Mathematically, geometric anisotropy is introduced by replacing $\| \mathbf{h} \|$ with the effective lag $$d(\mathbf{h}) = [\mathbf{h}^\top \mathbf{\Omega} \mathbf{h}]^{1/2}$$, where $\mathbf{\Omega}$ encodes axis scaling and rotation: $$\mathbf{\Omega} = P(\alpha)^\top D(\lambda) P(\alpha)$$ with $P(\alpha)$ a planar rotation (angle $\alpha$) and $D(\lambda) = \mathrm{diag}(1, \lambda)$ determining the axis ratio $\lambda$ ($0 < \lambda < 1$).

A commonly used anisotropic model is the Matérn kernel with geometric deformation: $$C(\mathbf{h}) = \sigma^2 \varphi\bigl(d(\mathbf{h});\, \theta \bigr)$$ where $$d(\mathbf{h}) = [\mathbf{h}^\top \mathbf{\Omega} \mathbf{h}]^{1/2}$$0 is the Matérn correlation function parameterized by smoothness $$d(\mathbf{h}) = [\mathbf{h}^\top \mathbf{\Omega} \mathbf{h}]^{1/2}$$1 and range $$d(\mathbf{h}) = [\mathbf{h}^\top \mathbf{\Omega} \mathbf{h}]^{1/2}$$2 (Villazón et al., 2024).

2. Neural Architectures for Surrogate Covariance Modeling

Neural surrogates emerge via two primary classes:

• Direct parameter regression: CNNs trained to estimate covariance parameters $$d(\mathbf{h}) = [\mathbf{h}^\top \mathbf{\Omega} \mathbf{h}]^{1/2}$$3 from field data or empirical variograms (Villazón et al., 2024).
• Hyperparameter mapping: Deep feed-forward networks $$d(\mathbf{h}) = [\mathbf{h}^\top \mathbf{\Omega} \mathbf{h}]^{1/2}$$4 outputting local length-scales and noise variances for GP kernels. This enables per-point or per-pair local hyperparameters, parameterizing fully nonstationary anisotropic kernels (Cremanns et al., 2017).

A canonical architecture for the latter (“Deep Gaussian Covariance Network”, Editor’s term: DGCN) involves:

• Input: standardized $$d(\mathbf{h}) = [\mathbf{h}^\top \mathbf{\Omega} \mathbf{h}]^{1/2}$$5
• Hidden layers: $$d(\mathbf{h}) = [\mathbf{h}^\top \mathbf{\Omega} \mathbf{h}]^{1/2}$$6, width $$d(\mathbf{h}) = [\mathbf{h}^\top \mathbf{\Omega} \mathbf{h}]^{1/2}$$7, activation ReLU/ELU
• Output: $$d(\mathbf{h}) = [\mathbf{h}^\top \mathbf{\Omega} \mathbf{h}]^{1/2}$$8 length-scales $$d(\mathbf{h}) = [\mathbf{h}^\top \mathbf{\Omega} \mathbf{h}]^{1/2}$$9, noise variance $\mathbf{\Omega}$0 (all via softplus)

Convolutional surrogates for geostatistical estimation use 2D CNNs on either gridded field data or variogram images, terminating in fully connected layers for scalar parameter estimation. Typical CNN depths/profiles are:

• NF (“field-based”): $\mathbf{\Omega}$1M parameters, multi-stage convolution/pooling [16×16x1] → [1,1,1024]
• NV (“variogram-based”): $\mathbf{\Omega}$2M parameters, similar reduction from 13×13x1

3. Covariance Kernel Parameterization via Neural Networks

Neural surrogates operationalize anisotropy through adaptive kernel hyperparameters. For DGCNs, the nonstationary anisotropic Gibbs kernel is: $\mathbf{\Omega}$3 where $\mathbf{\Omega}$4 and $\mathbf{\Omega}$5 produces all local hyperparameters (Cremanns et al., 2017).

For multi-kernel surrogates, the network outputs mixture weights $\mathbf{\Omega}$6 over $\mathbf{\Omega}$7 kernel families (e.g., RBF, Matérn, periodic), combined as: $\mathbf{\Omega}$8 This allows neural surrogates to represent highly nonstationary, heteroscedastic, and directionally adaptive covariance landscapes.

4. Training Methodologies and Scalability

Parameter estimation networks (CNN surrogates) (Villazón et al., 2024):

• Data generation: Simulate up to 450,000 unique parameter triplets $\mathbf{\Omega}$9, each with a single field realization on $$\mathbf{\Omega} = P(\alpha)^\top D(\lambda) P(\alpha)$$0 grids, leveraging GPU vectorization.
• Data augmentation: Rotation of spatial fields by 180° to expand the data set and improve angular robustness.
• Loss and optimization: Predict standardized parameters, optimize mean absolute error (MAE) using AdamW (learning rate 0.01, batch size 500), with early convergence (50–100 epochs, $$\mathbf{\Omega} = P(\alpha)^\top D(\lambda) P(\alpha)$$1 hour).
• Validation: 10% held-out grid; no explicit regularization beyond weight decay.

DGCN and neural kernel surrogates (Cremanns et al., 2017):

• Objective: Maximize log marginal likelihood of $$\mathbf{\Omega} = P(\alpha)^\top D(\lambda) P(\alpha)$$2 given the neural kernel matrix $$\mathbf{\Omega} = P(\alpha)^\top D(\lambda) P(\alpha)$$3.
• Gradients: Backpropagate through both $$\mathbf{\Omega} = P(\alpha)^\top D(\lambda) P(\alpha)$$4 and neural network by chain rule; leverage automatic differentiation frameworks.
• Scalability: Employ mini-batching, stochastic gradient updates of the likelihood, and inducing-point or SVGP extensions to reduce GP complexity from $$\mathbf{\Omega} = P(\alpha)^\top D(\lambda) P(\alpha)$$5 to $$\mathbf{\Omega} = P(\alpha)^\top D(\lambda) P(\alpha)$$6. The network adapts local length-scale mappings continuously under streaming updates.

5. Empirical Performance and Practical Considerations

Empirical studies (Villazón et al., 2024):

• Accuracy: Boxplots and bias/SD tables demonstrate that both neural surrogates (NF, NV) outperform maximum likelihood (ML) for the principal angle $$\mathbf{\Omega} = P(\alpha)^\top D(\lambda) P(\alpha)$$7, axis ratio $$\mathbf{\Omega} = P(\alpha)^\top D(\lambda) P(\alpha)$$8, and especially range $$\mathbf{\Omega} = P(\alpha)^\top D(\lambda) P(\alpha)$$9, particularly when ML fails for long-range scenarios.
• Directional precision: Rose diagrams reveal improved angular estimates, especially for directionally extreme cases.
• Computational efficiency: NN surrogates yield $P(\alpha)$0 speedup—full ML analysis of 115,600 fields requires 10.5 hours (Cholesky) versus $P(\alpha)$1 seconds for NN inference.
• Real-world data: In sea surface temperature datasets (27,761 patches), ML requires 3 hours, NF 41 seconds, NV 11 seconds; NN-based estimates align with ML but present reduced incidence of outlier predictions.

Practical guidelines (Villazón et al., 2024):

• Use nonuniform parameter grids to capture relevant covariance regimes.
• Standardize all regression targets.
• Apply data augmentation to improve angular estimation robustness.
• Monitor parameter boundaries (e.g., $P(\alpha)$2 or $P(\alpha)$3) to prevent out-of-domain NN outputs.
• Anticipate needed retraining for architectural extensions (e.g., addition of a nugget, higher dimensions, or spatio-temporal covariances).

6. Extensions and Limitations

Neural surrogates are directly extensible to more complex anisotropy, including spatio-temporal models and higher-dimensional (3D) covariances. For DGCNs, the kernel parameterization can be adapted for various base kernels and more sophisticated mixture models. However, individualized retraining is required when extending the parameterization grid, feature representation, or to accommodate different covariance families.

A notable limitation arises near parameter boundaries, especially under limited training coverage. Out-of-bound predictions for $P(\alpha)$4 or degenerate behavior for small/large $P(\alpha)$5 can occur if the surrogate is not explicitly trained in those regimes (Villazón et al., 2024). For DGCNs, computation of $P(\alpha)$6 and $P(\alpha)$7 via softplus guarantees positivity but does not enforce plausible cross-region smoothness without architectural regularization (Cremanns et al., 2017). In all cases, full robustness for extrapolation to unseen spatial geometries or parameter regimes requires careful validation and, when necessary, expanded training coverage.

7. Relationship to Broader Gaussian Process and Surrogate Modeling Literature

Neural surrogates for anisotropic covariance represent a bridge between flexible, scalable deep learning and the principled uncertainty quantification of Gaussian process theory. By learning either local hyperparameters or direct parameter maps, these methods overcome classical bottlenecks in stationary kernel selection, hyperparameter inference, and computational scaling. They enable both fine-grained spatial adaptation and efficient estimation in high-volume or high-dimensional domains, providing effective alternatives to maximum likelihood and manual kernel engineering (Cremanns et al., 2017, Villazón et al., 2024). Their usage is increasingly prevalent in spatial statistics, emulation of physical systems, and large-scale surrogate modeling.