Truncated Mean Squared Error

• Truncated Mean Squared Error is a robust loss function that caps squared residuals to limit the influence of extreme outliers.
• It replaces the classic quadratic loss with a truncated norm, enabling efficient 1D robust mean estimation and improved matrix denoising.
• The method allows precise tuning of the cutoff parameter, balancing robustness against inlier bias for optimal signal recovery.

Truncated mean squared error (TMSE) refers to the use of a quadratic loss that is capped at a fixed threshold, conferring robustness to outliers and non-Gaussian noise in estimation and signal recovery problems. TMSE arises from replacing the standard squared loss with a truncated quadratic error norm, most prominently in robust estimation and matrix denoising applications. Its essential feature is the selective penalization of residuals: residuals below a cutoff are penalized quadratically, while residuals above the cutoff only incur a constant loss.

1. Definition and Formal Properties

For a set of one-dimensional samples $x_1, \dots, x_n \in \mathbb{R}$ and a candidate mean $\mu \in \mathbb{R}$, the truncated quadratic error norm is defined as

$\rho(r) = \min(r^2, c^2), \qquad r \in \mathbb{R},$

for a cutoff $c > 0$. The total truncated mean squared error is then

$$\mathcal{E}(\mu) = \sum_{k=1}^n \rho(x_k - \mu) = \sum_{k=1}^n \min((x_k - \mu)^2, c^2).$$

This cost function reduces to traditional least squares when $c = \infty$. For finite $c$, contributions from samples with $|x_k - \mu| > c$ are constant ($c^2$), imposing a hard cap on their influence. The minimizer $\mu^* = \arg\min_\mu \mathcal{E}(\mu)$ yields a robust mean estimate (Jonsson et al., 2016).

2. Motivation for the Truncated Quadratic

The primary motivation for using TMSE is robustness to outliers. In standard mean estimation, a single arbitrarily distant outlier can disproportionately impact the estimated mean, due to quadratic penalization of large residuals. TMSE ameliorates this by "clipping" the loss for residuals larger than $\mu \in \mathbb{R}$0, resulting in:

• Limited influence of outliers: Samples with $\mu \in \mathbb{R}$1 no longer drive the mean arbitrarily far.
• M-estimator framework: TMSE is a special case of M-estimators, robust alternatives to least squares, where minimizing $\mu \in \mathbb{R}$2 with a nonconvex, saturating norm $\mu \in \mathbb{R}$3 yields robust statistics.
• Nonconvex but saturating behavior: While $\mu \in \mathbb{R}$4 is strictly convex for $\mu \in \mathbb{R}$5, it becomes flat for $\mu \in \mathbb{R}$6, differentiating it from strictly convex (quadratic) loss functions and conferring statistical robustness (Jonsson et al., 2016).

3. Efficient 1D Robust Mean Estimation Under TMSE

For one-dimensional data, efficient algorithms exist to minimize $\mu \in \mathbb{R}$7. The method in (Jonsson et al., 2016) exploits the observation that, at the minimizer, the set of inliers (samples within $\mu \in \mathbb{R}$8) constitutes a contiguous window in the sorted sample list. The procedure involves:

• Sorting: $\mu \in \mathbb{R}$9 time to sort $\rho(r) = \min(r^2, c^2), \qquad r \in \mathbb{R},$0.
• Window sweep: Performing a single pass to consider every maximal interval $\rho(r) = \min(r^2, c^2), \qquad r \in \mathbb{R},$1 where $\rho(r) = \min(r^2, c^2), \qquad r \in \mathbb{R},$2, as well as the two boundary intervals. For each window:
  • Compute the window mean $\rho(r) = \min(r^2, c^2), \qquad r \in \mathbb{R},$3,
  • Compute the sum of squared deviations $\rho(r) = \min(r^2, c^2), \qquad r \in \mathbb{R},$4,
  • Calculate the TMSE upper bound $\rho(r) = \min(r^2, c^2), \qquad r \in \mathbb{R},$5, where $\rho(r) = \min(r^2, c^2), \qquad r \in \mathbb{R},$6 is the count of samples outside $\rho(r) = \min(r^2, c^2), \qquad r \in \mathbb{R},$7.
  • Update if a lower $\rho(r) = \min(r^2, c^2), \qquad r \in \mathbb{R},$8 is found at $\rho(r) = \min(r^2, c^2), \qquad r \in \mathbb{R},$9.
• Update rules: Maintain running sums $c > 0$0 and $c > 0$1 to compute window statistics in $c > 0$2 per window movement.

This method yields a global minimizer in $c > 0$3 time (dominated by sorting), with essentially $c > 0$4 amortized time for sliding-window computations (Jonsson et al., 2016).

4. Parameter Tuning and Statistical Behavior

The threshold $c > 0$5 is the principal hyperparameter governing the robustness–efficiency trade-off. In practice, it is often set in relation to the inlier scale, such as $c > 0$6–$c > 0$7 for data with expected (effective) noise standard deviation $c > 0$8. A value of $c > 0$9 that is too small results in excessive trimming of inliers and bias; a $$\mathcal{E}(\mu) = \sum_{k=1}^n \rho(x_k - \mu) = \sum_{k=1}^n \min((x_k - \mu)^2, c^2).$$0 that is too large offers little robustness compared to the mean. Selection criteria include cross-validation or targeting a pre-specified fraction of inliers. The hard clipping mechanism ensures that, for single outlier scenarios, the robust mean remains unaffected by the outlier location if it lies beyond $$\mathcal{E}(\mu) = \sum_{k=1}^n \rho(x_k - \mu) = \sum_{k=1}^n \min((x_k - \mu)^2, c^2).$$1, in contrast to smooth but systematically biased alternatives such as channel averaging (Jonsson et al., 2016).

5. Experimental Performance and Comparison

Empirical evaluation on synthetic and real data demonstrates the following:

• Outlier resistance: When a single outlier is present, the robust mean computed via TMSE is invariant to the outlier's location as long as it lies outside the $$\mathcal{E}(\mu) = \sum_{k=1}^n \rho(x_k - \mu) = \sum_{k=1}^n \min((x_k - \mu)^2, c^2).$$2 band.
• No grid artifacts: In two-point symmetric tests and image smoothing applications, the TMSE estimator yields grid-free, exact results, unlike soft-histogram or channel-averaging methods, which incur periodic bias depending on grid alignment.
• Image denoising: Applied to image data (e.g., $$\mathcal{E}(\mu) = \sum_{k=1}^n \rho(x_k - \mu) = \sum_{k=1}^n \min((x_k - \mu)^2, c^2).$$3 windows on the $$\mathcal{E}(\mu) = \sum_{k=1}^n \rho(x_k - \mu) = \sum_{k=1}^n \min((x_k - \mu)^2, c^2).$$4 “Lenna” image; $$\mathcal{E}(\mu) = \sum_{k=1}^n \rho(x_k - \mu) = \sum_{k=1}^n \min((x_k - \mu)^2, c^2).$$5), TMSE-based smoothing preserves edges while reducing noise and small artifacts, outperforming standard mean filters in robustness to intensity discontinuities.
• Algorithmic efficiency: TMSE minimization is computationally efficient; after sorting, the cost per window is $$\mathcal{E}(\mu) = \sum_{k=1}^n \rho(x_k - \mu) = \sum_{k=1}^n \min((x_k - \mu)^2, c^2).$$6, suitable for sliding-window applications in image processing (Jonsson et al., 2016).

Method	Grid artifacts	Robust to outliers	Complexity
Truncated quadratic (TMSE)	No	Yes	$$\mathcal{E}(\mu) = \sum_{k=1}^n \rho(x_k - \mu) = \sum_{k=1}^n \min((x_k - \mu)^2, c^2).$$7 (1D case)
Channel averaging	Yes	Partial	Dependent on channel width

6. TMSE in Matrix Denoising via Truncated SVD

Beyond point estimation, TMSE appears in matrix denoising contexts as the mean squared error achieved by truncated singular value decomposition (TSVD). In this scenario, the original matrix $$\mathcal{E}(\mu) = \sum_{k=1}^n \rho(x_k - \mu) = \sum_{k=1}^n \min((x_k - \mu)^2, c^2).$$8 (rank $$\mathcal{E}(\mu) = \sum_{k=1}^n \rho(x_k - \mu) = \sum_{k=1}^n \min((x_k - \mu)^2, c^2).$$9) is observed with Gaussian noise: $c = \infty$0, $c = \infty$1 i.i.d. $c = \infty$2. The TSVD estimator

$c = \infty$3

is the best rank-$c = \infty$4 approximation to $c = \infty$5. The mean squared error (MSE) of this estimator, for noise level $c = \infty$6, is

$c = \infty$7

at first order, and, including second-order corrections,

$c = \infty$8

where $c = \infty$9 is the Moore–Penrose pseudoinverse of $c$0. This analysis quantifies the trade-off between rank truncation (robustness to noise) and denoising bias. Explicit non-asymptotic upper bounds on the MSE hold universally, even for finite noise and arbitrary magnitude perturbations, and depend only on the least singular value of $c$1 and the norm of the perturbation matrix. This analysis relies on exact rank and known noise statistics, emphasizing the applicability of truncated error norms in low-rank signal processing (Vu et al., 2020).

7. Comparative Analysis and Applications

TMSE, via its robust influence function, enables principled suppression of outlier contamination in both univariate and structured (matrix) data regimes. In mean estimation, it admits efficient, exact, and grid-free algorithms. In matrix denoising (TSVD), it underpins rigorous error bounds and optimality under Frobenius-norm approximations. Compared with competing robust approaches such as channel averaging, TMSE consistently yields lower bias, higher accuracy in the presence of contamination, and favorable computational properties (Jonsson et al., 2016, Vu et al., 2020).