Selective Denoising Loss (SDDLM)

• Selective Denoising Loss (SDDLM) is a framework for low-rank matrix estimation that uses weighted loss functions to prioritize denoising accuracy in user-specified submatrices and features.
• It employs optimal spectral denoising within the spiked-model setting and explicit weight constructions to achieve focused signal recovery under heteroscedastic noise and missing data scenarios.
• Empirical results show that SDDLM consistently outperforms unweighted shrinkage methods across applications like image restoration, synthetic datasets, and matrix completion.

Selective Denoising Loss (SDDLM) is a framework for low-rank matrix estimation with prioritized accuracy in user-specified regions, submatrices, or features. Formulated to address denoising in contexts with submatrix interest, heteroscedastic noise, and missing data, SDDLM leverages weighted loss functions and optimal spectral denoising in the canonical spiked-model setting to enable regionally selective signal recovery. The framework provides explicit constructions for weights, closed-form optimal estimators, localization algorithms, and empirical validation of gains over unweighted shrinkage methods (Leeb, 2019).

1. Weighted Loss Formulation in the Spiked-Model Setting

The SDDLM approach operates within the standard spiked-model, observing

$Y = X + G,$

where $X \in \mathbb{R}^{p \times n}$ is low-rank ($\rank(X) = r$), and $G$ is a noise matrix (typically Gaussian or whitened-Gaussian).

A nonnegative weight matrix $W \in \mathbb{R}^{p \times n}$ encodes selective emphasis on entries of $X$. The weighted Frobenius-norm loss between an estimate $\widehat{X}$ and the ground truth $X$ is

$$L_W(X, \widehat{X}) = \left\| W \circ (\widehat{X} - X) \right\|_F^2 = \sum_{i=1}^p \sum_{j=1}^n W_{ij}^2 (\widehat{X}_{ij} - X_{ij})^2,$$

where $\circ$ denotes entrywise multiplication.

Three canonical instantiations for $X \in \mathbb{R}^{p \times n}$0 are:

• Submatrix denoising: $X \in \mathbb{R}^{p \times n}$1 for $X \in \mathbb{R}^{p \times n}$2 in the region of interest $X \in \mathbb{R}^{p \times n}$3, 0 otherwise.
• Heteroscedastic noise: $X \in \mathbb{R}^{p \times n}$4 when $X \in \mathbb{R}^{p \times n}$5.
• Missing data (matrix completion): $X \in \mathbb{R}^{p \times n}$6 if unobserved; for observed entries with sampling probability $X \in \mathbb{R}^{p \times n}$7, commonly $X \in \mathbb{R}^{p \times n}$8 is used.

Frequently, SDDLM uses a pair-of-weights formalism, factoring $X \in \mathbb{R}^{p \times n}$9 with $\rank(X) = r$0 and $\rank(X) = r$1 diagonal. This enables representing the loss as

$\rank(X) = r$2

2. Optimal Spectral Denoiser with Selective Weights

SDDLM restricts denoising estimators to the spectral family: $\rank(X) = r$3

Under spiked asymptotics ($\rank(X) = r$4), optimal choices for $\rank(X) = r$5 can be derived. Critical limits are the (possibly weighted) inner products of empirical singular vectors $\rank(X) = r$6 of $\rank(X) = r$7 with their population counterparts $\rank(X) = r$8: $\rank(X) = r$9 with analogous definitions for the right singular vectors.

Define the matrices

$G$0

The optimal estimator is given by

$G$1

where $G$2 is the Moore–Penrose pseudoinverse of $G$3, and hence

$G$4

In the case of diagonal $G$5 and $G$6 ("weighted-orthogonality"), this yields explicit componentwise shrinkage: $G$7 with

• $G$8: row/column-weight projections of population singular vectors,
• $G$9,
• $W \in \mathbb{R}^{p \times n}$0: unweighted cosines and sines.

The proof proceeds by expanding the weighted error in singular subspaces, employing spiked-model asymptotics for all cross-terms, and minimizing the quadratic form in $W \in \mathbb{R}^{p \times n}$1 (Leeb, 2019).

3. Localized Denoising via Combinations of Selective Weights

SDDLM supports localized denoising through orthogonal projections partitioning the $W \in \mathbb{R}^{p \times n}$2-index space. Consider projections $W \in \mathbb{R}^{p \times n}$3 (row) and $W \in \mathbb{R}^{p \times n}$4 (column), decomposing the identity: $W \in \mathbb{R}^{p \times n}$5, $W \in \mathbb{R}^{p \times n}$6.

The localized denoiser is constructed as follows:

$\widehat{X}$1

Decomposing the error into region-wise blocks, SDDLM denoises each block optimally for its local weight and aggregates the resulting estimates. This approach yields total unweighted error no larger than global singular-value shrinkage, with strict improvement when singular vectors are heterogeneously distributed.

4. SDDLM as Weight Selection for Targeted Denoising

The term "Selective Denoising Loss (SDDLM)" refers to the construction of $W \in \mathbb{R}^{p \times n}$7 for user-specified prioritization:

• Large $W \in \mathbb{R}^{p \times n}$8: high-priority denoising for entry $W \in \mathbb{R}^{p \times n}$9.
• Small or zero $X$0: deprioritized or ignored entries.

Examples recapitulated:

• Submatrix focus: $X$1 equals the submatrix indicator.
• Heteroscedastic whitening: $X$2.
• Missing data: $X$3 for unobserved, $X$4 for observed.

A global "temperature" or focus parameter $X$5 is sometimes introduced: $X$6 Tuning $X$7 allows balancing fidelity within versus outside the focus region. Cross-validation on a held-out validation set of entries can be employed for parameter selection.

5. Empirical Performance and Applicability

Empirical studies demonstrate that SDDLM confers substantial improvements relative to unweighted singular-value shrinkage:

Application	Standard method rel. error	SDDLM/localized rel. error	Notes
MIT-logo image (with corners focused)	~0.125	~0.074	15x30 grid, heterogeneity in corners
Checkerboard synthetic (rank-2)	~0.19	~0.14	70% energy in "light" squares (800x800)
Submatrix denoising (rank 1)	-	Improved	SDDLM outperforms global and submatrix-only cases
Doubly-heteroscedastic noise	-	Up to 30% better	Outperforms OptShrink for condition number κ=16
Missing–values case	-	Improved	Outperforms nuclear-norm completion (200x400)

In all scenarios, SDDLM does not underperform unweighted shrinkage in asymptotic regimes and, even with additional parameter overhead, more than compensates with focused accuracy in practical sample sizes.

6. Implementation and Theoretical Guarantees

The core algorithms and explicit formulas are detailed in (Leeb, 2019), with:

• Algorithm 1: OptimalSpectralDenoiser (for general $X$8 or pairs $X$9).
• Algorithm 2: Localized denoising (for decompositions into projection blocks).
• Application-specific derivations for submatrix, heteroscedastic, and missing data settings (see Section 6).
• Asymptotic theory and proofs in the Appendix.

The theoretical foundation ensures that SDDLM asymptotically achieves the optimal selective tradeoff prescribed by $\widehat{X}$0, yielding nontrivial improvements in heterogeneous or region-focused denoising scenarios. All steps are justified by rigorous spiked-model asymptotics and random matrix theory, with turnkey algorithms provided for direct research implementation (Leeb, 2019).