Gradient Mean Squared Error

• Gradient Mean Squared Error (GMSE) is a metric that weights pixel errors by local gradient magnitudes, enhancing fidelity in generative models and convergence analysis in optimization.
• It employs a pipeline of gradient extraction, Gaussian blurring, gamma correction, and normalization to prioritize critical features and reduce spurious artifacts.
• Empirical results show up to 82% error reduction and a faster convergence rate compared to traditional MSE, demonstrating its robustness in noisy environments.

Gradient Mean Squared Error (GMSE) is a family of metrics and algorithmic tools with two distinct but convergent roles in contemporary machine learning research: (i) as a weighted loss function for enhancing the fidelity of generative models in structured data regimes such as computational fluid dynamics (CFD); and (ii) as a convergence metric and analytic tool for stochastic optimization methods such as nonlinear stochastic gradient descent (SGD) in the presence of irregular, heavy-tailed noise. Both usages are united by the principle of incorporating local or instantaneous gradient information into either the error metric or its analytical control. GMSE provides improved convergence, heightened sensitivity to critical features, and robustness to nonclassical noise structures (Armacki et al., 2024, Cooper-Baldock et al., 2024).

1. Mathematical Formulation: Loss and Metric Variants

The two principal GMSE paradigms are instantiated as follows:

A. Weighted Loss for Generative Models

Given a ground-truth field $I_R\in\mathbb{R}^{h\times w}$ and generated field $\hat{I}_G\in\mathbb{R}^{h\times w}$ (e.g., CFD velocity magnitude distributions), the per-instance Mean Squared Error (MSE) is

$$\mathrm{MSE} = \frac{1}{n}\sum_{i=1}^n \left[ \frac{1}{hw}\sum_{j,k} \big(I_R^{(i)}(j,k) - \hat{I}_G^{(i)}(j,k)\big)^2\right] \tag{1}$$

GMSE introduces a per-pixel importance weighting $W_i(j,k) \in [C_o, 1]$ based on the gradient magnitude of $I_R$. The GMSE loss becomes

$$\mathrm{GMSE} = \frac{1}{n}\sum_{i=1}^n \left[ \frac{1}{hw}\sum_{j,k} W_i(j,k)\, \big(I_R^{(i)}(j,k) - \hat{I}_G^{(i)}(j,k)\big)^2\right] \tag{2}$$

where $W_i(j,k)$ is computed through a pipeline of gradient extraction, Gaussian blurring, gamma correction, and min–max normalization with an additive offset $C_o$:

• $D_x(j,k) = I_R(j,k) - I_R(j,k-1)$, $D_y(j,k) = I_R(j,k) - I_R(j-1,k)$
• $\hat{I}_G\in\mathbb{R}^{h\times w}$0
• $\hat{I}_G\in\mathbb{R}^{h\times w}$1 with $\hat{I}_G\in\mathbb{R}^{h\times w}$2 Gaussian kernel
• $\hat{I}_G\in\mathbb{R}^{h\times w}$3
• $\hat{I}_G\in\mathbb{R}^{h\times w}$4
• $\hat{I}_G\in\mathbb{R}^{h\times w}$5

B. Stochastic Optimization Metric

In stochastic gradient methods under heavy-tailed, symmetric noise,

• NonconVex: GMSE metric is $\hat{I}_G\in\mathbb{R}^{h\times w}$6
• Strongly Convex: GMSE tracks $\hat{I}_G\in\mathbb{R}^{h\times w}$7

Analysis focuses on rate and deviation behavior of $\hat{I}_G\in\mathbb{R}^{h\times w}$8 and $\hat{I}_G\in\mathbb{R}^{h\times w}$9 as $$\mathrm{MSE} = \frac{1}{n}\sum_{i=1}^n \left[ \frac{1}{hw}\sum_{j,k} \big(I_R^{(i)}(j,k) - \hat{I}_G^{(i)}(j,k)\big)^2\right] \tag{1}$$0 (Armacki et al., 2024).

2. Algorithmic Workflow and Implementation

For generative architectures, such as controlled cGANs applied to CFD surrogate modeling, the GMSE loss function is integrated as follows:

1. Compute per-instance gradient maps of $$\mathrm{MSE} = \frac{1}{n}\sum_{i=1}^n \left[ \frac{1}{hw}\sum_{j,k} \big(I_R^{(i)}(j,k) - \hat{I}_G^{(i)}(j,k)\big)^2\right] \tag{1}$$1 (see above).
2. Apply spatial Gaussian filter, gamma correction, and normalization.
3. Form $$\mathrm{MSE} = \frac{1}{n}\sum_{i=1}^n \left[ \frac{1}{hw}\sum_{j,k} \big(I_R^{(i)}(j,k) - \hat{I}_G^{(i)}(j,k)\big)^2\right] \tag{1}$$2 and apply it in the squared error between $$\mathrm{MSE} = \frac{1}{n}\sum_{i=1}^n \left[ \frac{1}{hw}\sum_{j,k} \big(I_R^{(i)}(j,k) - \hat{I}_G^{(i)}(j,k)\big)^2\right] \tag{1}$$3 and $$\mathrm{MSE} = \frac{1}{n}\sum_{i=1}^n \left[ \frac{1}{hw}\sum_{j,k} \big(I_R^{(i)}(j,k) - \hat{I}_G^{(i)}(j,k)\big)^2\right] \tag{1}$$4.
4. Average over pixels and batch, yielding the final GMSE value used for generator loss backpropagation.
5. In DGMSE, the hyperparameters $$\mathrm{MSE} = \frac{1}{n}\sum_{i=1}^n \left[ \frac{1}{hw}\sum_{j,k} \big(I_R^{(i)}(j,k) - \hat{I}_G^{(i)}(j,k)\big)^2\right] \tag{1}$$5 are adaptively scheduled by epoch to sharpen or broaden importance masks as the generator improves.

For stochastic optimization, the GMSE metric governs large deviation and convergence rate analyses as a function of step-size $$\mathrm{MSE} = \frac{1}{n}\sum_{i=1}^n \left[ \frac{1}{hw}\sum_{j,k} \big(I_R^{(i)}(j,k) - \hat{I}_G^{(i)}(j,k)\big)^2\right] \tag{1}$$6 (with $$\mathrm{MSE} = \frac{1}{n}\sum_{i=1}^n \left[ \frac{1}{hw}\sum_{j,k} \big(I_R^{(i)}(j,k) - \hat{I}_G^{(i)}(j,k)\big)^2\right] \tag{1}$$7), bounded/bias-free nonlinearity $$\mathrm{MSE} = \frac{1}{n}\sum_{i=1}^n \left[ \frac{1}{hw}\sum_{j,k} \big(I_R^{(i)}(j,k) - \hat{I}_G^{(i)}(j,k)\big)^2\right] \tag{1}$$8, and the denoising-inducing symmetry structure of the noise (Armacki et al., 2024).

3. Theoretical Properties and Analytical Guarantees

Weighted Loss for Generative Models:

By upweighting errors in regions of high physical relevance (e.g., vortex sheets, boundary layers), GMSE facilitates accelerated convergence and substantially improves structural fidelity in generated fields. All pixels contribute, but low-gradient/freestream errors are downweighted; spurious artifacts are suppressed more efficiently than with uniform MSE (Cooper-Baldock et al., 2024).

SGD Analysis:

Key guarantees established for nonlinear SGD with GMSE metrics under heavy-tailed, symmetric noise distributions:

• For nonconvex $$\mathrm{MSE} = \frac{1}{n}\sum_{i=1}^n \left[ \frac{1}{hw}\sum_{j,k} \big(I_R^{(i)}(j,k) - \hat{I}_G^{(i)}(j,k)\big)^2\right] \tag{1}$$9, $W_i(j,k) \in [C_o, 1]$0 with $W_i(j,k) \in [C_o, 1]$1.
• For strongly convex $W_i(j,k) \in [C_o, 1]$2, $W_i(j,k) \in [C_o, 1]$3, with rates arbitrarily close to optimal $W_i(j,k) \in [C_o, 1]$4.
• Large deviation bounds: $W_i(j,k) \in [C_o, 1]$5 for the gradient norm metric, with explicit rate functions dependent on optimizer, nonlinearity, and noise symmetry. The theoretical sharpness and uniformity owe to "positive alignment" enforced by distributional symmetry, sub-Gaussian error bounds enabled by bounded nonlinearity, and smoothness-plus-alignment descent inequalities (Armacki et al., 2024).

4. Empirical Evaluation and Comparative Performance

In CFD generative modeling:

• GMSE and its dynamic variant DGMSE achieve markedly higher Structural Similarity Index (SSIM) at all epochs compared to vanilla MSE. At epoch 300, GMSE and DGMSE achieve $W_i(j,k) \in [C_o, 1]$6, compared to $W_i(j,k) \in [C_o, 1]$7 for MSE.
• Final structural-dissimilarity error is reduced by roughly $W_i(j,k) \in [C_o, 1]$8 for GMSE and $W_i(j,k) \in [C_o, 1]$9 for DGMSE over MSE.
• GMSE-trained networks reach high-quality SSIM ($I_R$0) in $I_R$1 epochs versus $I_R$2 for MSE, indicating a $I_R$3 reduction in effective training time.
• The maximum gradient (loss rate) of the GMSE loss curve is up to $I_R$4 higher for DGMSE, reflecting faster learning.
• Discriminators are more frequently "fooled"—that is, assign "real" with higher confidence—to GMSE/DGMSE-generated images, reflecting improved visual and structural plausibility.

Method	SSIM (Epoch 300)	Max Normalized Loss-Rate	Error Reduction vs. MSE
MSE	0.933	0.107	–
GMSE	0.988	0.143	82.1%
DGMSE	0.989	0.189	83.6%

Quantitative results are robust to variations in hyperparameters $I_R$5, but DGMSE's schedule accelerates convergence most efficiently. Qualitative output also demonstrates correction of spurious artifacts and preservation of essential high-gradient flow features (Cooper-Baldock et al., 2024).

5. Hyperparameters, Scheduling, and Practical Considerations

Crucial GMSE hyperparameters:

• Gaussian blur width $I_R$6, controlling the locality of gradient magnitude.
• Gamma $I_R$7, tuning the nonlinear emphasis on strong gradients.
• Offset $I_R$8, setting the minimal contribution of low-gradient regions.

Sensitivity studies, with $I_R$9, $$\mathrm{GMSE} = \frac{1}{n}\sum_{i=1}^n \left[ \frac{1}{hw}\sum_{j,k} W_i(j,k)\, \big(I_R^{(i)}(j,k) - \hat{I}_G^{(i)}(j,k)\big)^2\right] \tag{2}$$0, and $$\mathrm{GMSE} = \frac{1}{n}\sum_{i=1}^n \left[ \frac{1}{hw}\sum_{j,k} W_i(j,k)\, \big(I_R^{(i)}(j,k) - \hat{I}_G^{(i)}(j,k)\big)^2\right] \tag{2}$$1, are conducted via cross-validation for best SSIM and convergence rate (Cooper-Baldock et al., 2024). DGMSE dynamically schedules these parameters—initially selecting broad, flat masks (large $$\mathrm{GMSE} = \frac{1}{n}\sum_{i=1}^n \left[ \frac{1}{hw}\sum_{j,k} W_i(j,k)\, \big(I_R^{(i)}(j,k) - \hat{I}_G^{(i)}(j,k)\big)^2\right] \tag{2}$$2, small $$\mathrm{GMSE} = \frac{1}{n}\sum_{i=1}^n \left[ \frac{1}{hw}\sum_{j,k} W_i(j,k)\, \big(I_R^{(i)}(j,k) - \hat{I}_G^{(i)}(j,k)\big)^2\right] \tag{2}$$3) and sharpening over training. This "coarse-to-fine" adaptation matches the learning progression of the generator network.

For stochastic optimization, step-size schedules are critical. $$\mathrm{GMSE} = \frac{1}{n}\sum_{i=1}^n \left[ \frac{1}{hw}\sum_{j,k} W_i(j,k)\, \big(I_R^{(i)}(j,k) - \hat{I}_G^{(i)}(j,k)\big)^2\right] \tag{2}$$4 is optimal for nonconvex MSE rates, while $$\mathrm{GMSE} = \frac{1}{n}\sum_{i=1}^n \left[ \frac{1}{hw}\sum_{j,k} W_i(j,k)\, \big(I_R^{(i)}(j,k) - \hat{I}_G^{(i)}(j,k)\big)^2\right] \tag{2}$$5 near $$\mathrm{GMSE} = \frac{1}{n}\sum_{i=1}^n \left[ \frac{1}{hw}\sum_{j,k} W_i(j,k)\, \big(I_R^{(i)}(j,k) - \hat{I}_G^{(i)}(j,k)\big)^2\right] \tag{2}$$6 recovers near-optimal strongly convex rates. Performance is guaranteed irrespective of noise moment bounds, relying only on symmetry and local regularity conditions (Armacki et al., 2024).

6. Broader Significance and Relationships

GMSE, both as a loss and as a convergence metric, offers a paradigm for integrating structural priors or local signal importance into error assessment or optimizer analysis:

• In generative modeling for scientific data, GMSE ensures that rare or crucial structured information is preserved, overcoming the "pixel-level democracy" limitation of uniform MSE.
• In non-standard stochastic optimization, GMSE-type metrics provide mathematically robust performance characterization under heavy-tailed, potentially infinite-variance noise, leveraging densified symmetry and bounded nonlinearity for convergence that matches light-tailed classical guarantees.

A plausible implication is that similar strategies may generalize to other domains where spatial or topological signal disparities challenge uniform error-based objectives, and to optimization contexts with heteroscedastic or non-Gaussian noise profiles.

• "Large Deviation Upper Bounds and Improved MSE Rates of Nonlinear SGD: Heavy-tailed Noise and Power of Symmetry" (Armacki et al., 2024)
• "A generalised novel loss function for computational fluid dynamics" (Cooper-Baldock et al., 2024)