Normalized Mean Square Error (NMSE)

• NMSE is a dimensionless metric that measures the fidelity of an estimator by normalizing the mean square error with the signal power or variance.
• It is widely used in signal processing, statistical estimation, and communications, including applications like generalized LASSO, channel estimation, and metasurface design.
• Analytical and empirical studies of NMSE guide system optimization, highlighting trade-offs in power settings, hardware impairments, and signal structure.

Normalized mean square error (NMSE) is a dimensionless performance metric that quantifies the relative fidelity of an estimator, system, or algorithm by measuring mean square deviation and normalizing it by an appropriate power or variance term. It is prevalent in signal processing, statistical estimation, compressed sensing, machine learning, and physical layer design for communication systems.

1. Formal Definition and Variants

NMSE is universally defined as the ratio of mean-squared error (MSE) between the true quantity and its estimate to a normalization term, typically the power or variance of the reference/target signal. Given a target $g$ and its estimate $\hat{g}$, the general form is: $$\mathrm{NMSE} = \frac{\mathbb{E}[\|g - \hat{g}\|^2]}{\mathbb{E}[\|g\|^2]}$$ In dB, this is often reported as $10\log_{10}(\mathrm{NMSE})$.

In system identification and communications, NMSE also appears as

$$\mathrm{NMSE}_\ell = \frac{\mathbb{E}[|y_\ell - y_{o,\ell}|^2]}{\mathbb{E}[|y_{o,\ell}|^2]}$$

where $y_\ell$ is the system's output and $y_{o,\ell}$ the ideal reference (Händel et al., 2019).

In statistics and learning theory, the normalized square error (NSE) is sometimes used, especially in LASSO and similar estimators, where normalization is performed w.r.t. the noise variance and potentially the number of measurements,

$$\mathrm{NMSE} = \frac{\mathbb{E}[\|x^* - x_0\|^2]}{m \sigma^2}$$

with $x_0$ as the true signal, $x^*$ as the estimate, $\hat{g}$0 the measurement dimension, and $\hat{g}$1 the noise variance (1311.0830).

2. NMSE in Estimation and Detection Algorithms

NMSE is employed to evaluate the quality of estimators in high-dimensional statistics and communications.

• Generalized LASSO: For $\hat{g}$2, with $\hat{g}$3 and Gaussian noise $\hat{g}$4, the NMSE of an estimator $\hat{g}$5 is $\hat{g}$6 (1311.0830).
• Channel Estimation: In MIMO and related systems, channel estimation algorithms such as LS, MMSE, and data-aided MMSE compare NMSE to benchmark estimation accuracy under different scenarios. For a true channel $\hat{g}$7 and estimator $\hat{g}$8,

$\hat{g}$9

is used directly for comparison (Liu et al., 2018).

NMSE provides a fair metric for algorithms operating at different power regimes or using disparate reference signals, and enables meaningful performance benchmarking across scenarios.

3. Analytical Expressions and Model Dependencies

Analytical NMSE expressions often capture rich dependencies on noise, signal structure, system nonlinearity, and interference.

• MIMO Transmitter with Crosstalk:

In the context of nonlinear MIMO transmitters with backward crosstalk (Händel et al., 2019), the per-branch NMSE for distorted output $$\mathrm{NMSE} = \frac{\mathbb{E}[\|g - \hat{g}\|^2]}{\mathbb{E}[\|g\|^2]}$$0 and ideal output $$\mathrm{NMSE} = \frac{\mathbb{E}[\|g - \hat{g}\|^2]}{\mathbb{E}[\|g\|^2]}$$1 is expressed as

$$\mathrm{NMSE} = \frac{\mathbb{E}[\|g - \hat{g}\|^2]}{\mathbb{E}[\|g\|^2]}$$2

with $$\mathrm{NMSE} = \frac{\mathbb{E}[\|g - \hat{g}\|^2]}{\mathbb{E}[\|g\|^2]}$$3 the diagonal of error covariance $$\mathrm{NMSE} = \frac{\mathbb{E}[\|g - \hat{g}\|^2]}{\mathbb{E}[\|g\|^2]}$$4. Explicit formulas for $$\mathrm{NMSE} = \frac{\mathbb{E}[\|g - \hat{g}\|^2]}{\mathbb{E}[\|g\|^2]}$$5 show NMSE as convex, third-order polynomials in input power, encapsulating effects of PA nonlinearity, crosstalk, and noise.

• Generalized LASSO NMSE:

For estimators defined via

$$\mathrm{NMSE} = \frac{\mathbb{E}[\|g - \hat{g}\|^2]}{\mathbb{E}[\|g\|^2]}$$6

the asymptotic NMSE is

$$\mathrm{NMSE} = \frac{\mathbb{E}[\|g - \hat{g}\|^2]}{\mathbb{E}[\|g\|^2]}$$7

where $$\mathrm{NMSE} = \frac{\mathbb{E}[\|g - \hat{g}\|^2]}{\mathbb{E}[\|g\|^2]}$$8 encapsulates signal structure via expected squared Gaussian distance to the scaled subdifferential of the regularizer (1311.0830).

• Data-aided MMSE Channel Estimation:

For DA-MMSE, the NMSE is characterized (in dB) as

$$\mathrm{NMSE} = \frac{\mathbb{E}[\|g - \hat{g}\|^2]}{\mathbb{E}[\|g\|^2]}$$9

where $10\log_{10}(\mathrm{NMSE})$0 aggregates contributions from pilot SNR, data energy, and symbol BER, providing a direct link between estimator fidelity and system configuration (Liu et al., 2018).

• SIM-assisted Channel Estimation in Rician Fading:

For a multi-user system with stacked intelligent metasurfaces (SIM) (Papazafeiropoulos et al., 18 Feb 2025), the closed-form per-user NMSE is:

$10\log_{10}(\mathrm{NMSE})$1

where key parameters are the Rician K-factor $10\log_{10}(\mathrm{NMSE})$2, large-scale fading $10\log_{10}(\mathrm{NMSE})$3, and meta-surface transformations $10\log_{10}(\mathrm{NMSE})$4.

4. NMSE Optimization and Theoretical Properties

Closed-form optimization of NMSE is tractable in several key scenarios.

• Power Back-off for Distortion Minimization: For MIMO transmitters affected by nonlinearities and crosstalk, minimal worst-case NMSE across branches is achieved by solving for the input power $10\log_{10}(\mathrm{NMSE})$5 that minimizes $10\log_{10}(\mathrm{NMSE})$6. This is found via roots of convex, third-order polynomials in $10\log_{10}(\mathrm{NMSE})$7 (Händel et al., 2019).
• Phase-Shift Design in Metasurfaces: In SIM-assisted channel estimation, the phase shifts of each metasurface layer are optimized to minimize the average NMSE across users, using projected gradient-descent, exploiting the fact that NMSE is differentiable with respect to these parameters (Papazafeiropoulos et al., 18 Feb 2025).
• Sample Complexity and Geometry: In generalized LASSO, NMSE is governed by geometric summary parameters (e.g., statistical dimension $10\log_{10}(\mathrm{NMSE})$8) capturing the "effective size" of regularizer descent cones. Smaller $10\log_{10}(\mathrm{NMSE})$9 permits recovery at lower dimensions $$\mathrm{NMSE}_\ell = \frac{\mathbb{E}[|y_\ell - y_{o,\ell}|^2]}{\mathbb{E}[|y_{o,\ell}|^2]}$$0 and incurs lower NMSE (1311.0830). The key theoretical property is concentration: the high-dimensional NMSE converges to its analytical prediction as ($$\mathrm{NMSE}_\ell = \frac{\mathbb{E}[|y_\ell - y_{o,\ell}|^2]}{\mathbb{E}[|y_{o,\ell}|^2]}$$1).

5. Empirical Trends and Practical Implications

Empirical studies show consistent NMSE trends across application domains.

• Effect of System Nonidealities: Even modest backward crosstalk in MIMO transmitters induces a non-trivial NMSE optimum shifted below the classical compression-only optimum, requiring practitioners to judiciously select transmit power for balanced NMSE and spectral efficiency (Händel et al., 2019).
• Role of Training Resources and SNR: Increasing pilot or data power, length, or reducing BER monotonically reduces NMSE in channel estimation. DA-MMSE outperforms both LS and pilot-only MMSE, particularly at low pilot power and high data reliability (Liu et al., 2018). For SIMs, more metasurface layers yield diminishing NMSE improvement after $$\mathrm{NMSE}_\ell = \frac{\mathbb{E}[|y_\ell - y_{o,\ell}|^2]}{\mathbb{E}[|y_{o,\ell}|^2]}$$2, and LoS presence (higher K-factor) consistently lowers NMSE (Papazafeiropoulos et al., 18 Feb 2025).
• Estimator Comparison: Table 1 summarizes the NMSE expressions for typical estimators in communications:

Estimator	NMSE Expression	Key Parameterization
LS (pilot only)	$$\mathrm{NMSE}_\ell = \frac{\mathbb{E}[|y_\ell - y_{o,\ell}|^2]}{\mathbb{E}[|y_{o,\ell}|^2]}$$3	Pilot SNR $$\mathrm{NMSE}_\ell = \frac{\mathbb{E}[|y_\ell - y_{o,\ell}|^2]}{\mathbb{E}[|y_{o,\ell}|^2]}$$4
MMSE (pilot only)	$$\mathrm{NMSE}_\ell = \frac{\mathbb{E}[|y_\ell - y_{o,\ell}|^2]}{\mathbb{E}[|y_{o,\ell}|^2]}$$5	Pilot SNR $$\mathrm{NMSE}_\ell = \frac{\mathbb{E}[|y_\ell - y_{o,\ell}|^2]}{\mathbb{E}[|y_{o,\ell}|^2]}$$6
DA-MMSE	$$\mathrm{NMSE}_\ell = \frac{\mathbb{E}[|y_\ell - y_{o,\ell}|^2]}{\mathbb{E}[|y_{o,\ell}|^2]}$$7	SNR-like $$\mathrm{NMSE}_\ell = \frac{\mathbb{E}[|y_\ell - y_{o,\ell}|^2]}{\mathbb{E}[|y_{o,\ell}|^2]}$$8 incl. data/BER
SIM-MMSE	$$\mathrm{NMSE}_\ell = \frac{\mathbb{E}[|y_\ell - y_{o,\ell}|^2]}{\mathbb{E}[|y_{o,\ell}|^2]}$$9	SIM responses, Rician factor, SNR

6. Key Applications and Advanced Use Cases

NMSE is a central metric in several advanced research areas.

• Hardware-Impaired MIMO Links: Used to optimize power back-off and predistortion strategies where PA nonlinearity and isolation constraints interplay (Händel et al., 2019).
• Sparse Signal Recovery: Asymptotic NMSE formulations guide the design and benchmarking of convex recovery algorithms under structured priors (1311.0830).
• Data-Driven Channel Acquisition: NMSE expressions incorporating decoded data error rates set operational guidelines for scheduling, resource allocation, and reliability in heterogeneous networks (Liu et al., 2018).
• Metasurface-Aided Massive MIMO: The NMSE metric drives the design and deployment of intelligent surface architectures, phase control algorithms, and per-layer resource allocation under composite fading models (Papazafeiropoulos et al., 18 Feb 2025).

These uses highlight NMSE's centrality as a link between analytical tractability, hardware design, and practical trade-off analysis in communication and statistical estimation.

7. Limitations and Interpretative Considerations

While NMSE is widely used due to its normalization and interpretability, certain caveats apply:

• In high SNR or low-error regimes, NMSE differences across algorithms may become marginal and numerically sensitive.
• In the presence of strongly structured signals, the interpretation of normalization must account for intrinsic signal power variability.
• NMSE as a single-number summary may obscure performance variability over feature subspaces or system dimensions.
• In channel and system design, NMSE minimization should be contextualized within broader requirements, e.g., spectral efficiency, robustness, and hardware overhead.

Nevertheless, NMSE remains a critical and theoretically grounded metric for algorithmic comparison and system optimization across high-dimensional signal processing and communications research.