---
title: Selective Denoising Loss (SDDLM)
url: https://www.emergentmind.com/topics/selective-denoising-loss-sddlm
type: topic
---

# Selective Denoising Loss (SDDLM)

Selective Denoising Loss (SDDLM) is a framework for low-rank matrix estimation with prioritized accuracy in user-specified regions, submatrices, or features. Formulated to address denoising in contexts with submatrix interest, heteroscedastic noise, and missing data, SDDLM leverages weighted loss functions and optimal spectral denoising in the canonical spiked-model setting to enable regionally selective signal recovery. The framework provides explicit constructions for weights, closed-form optimal estimators, localization algorithms, and empirical validation of gains over unweighted shrinkage methods [1902.09474].

## 1. Weighted Loss Formulation in the Spiked-Model Setting

The SDDLM approach operates within the standard spiked-model, observing
\[
Y = X + G,
\]
where \( X \in \mathbb{R}^{p \times n} \) is low-rank (\( \rank(X) = r \)), and \( G \) is a noise matrix (typically Gaussian or whitened-Gaussian).

A nonnegative weight matrix \( W \in \mathbb{R}^{p \times n} \) encodes selective emphasis on entries of \( X \). The weighted Frobenius-norm loss between an estimate \( \widehat{X} \) and the ground truth \( X \) is
\[
L_W(X, \widehat{X}) = \left\| W \circ (\widehat{X} - X) \right\|_F^2 = \sum_{i=1}^p \sum_{j=1}^n W_{ij}^2 (\widehat{X}_{ij} - X_{ij})^2,
\]
where \( \circ \) denotes entrywise multiplication.

Three canonical instantiations for \( W \) are:
- **Submatrix denoising:** \( W_{ij} = 1 \) for \( (i,j) \) in the region of interest \( \mathcal{S} \), 0 otherwise.
- **Heteroscedastic noise:** \( W_{ij} = 1/\sigma_{ij} \) when \( \operatorname{Var}(G_{ij}) = \sigma_{ij}^2 \).
- **Missing data (matrix completion):** \( W_{ij} = 0 \) if unobserved; for observed entries with sampling probability \( p_{ij} \), commonly \( W_{ij} = 1/\sqrt{p_{ij}} \) is used.

Frequently, SDDLM uses a pair-of-weights formalism, factoring \( W = \Omega \circ \Pi \) with \( \Omega \) and \( \Pi \) diagonal. This enables representing the loss as
\[
L_W(X, \widehat{X}) = \| \Omega(\widehat{X} - X)\Pi \|_F^2.
\]

## 2. Optimal Spectral Denoiser with Selective Weights

SDDLM restricts denoising estimators to the spectral family:
\[
\mathcal{S} = \left\{ \widehat{X} : \widehat{X} = UBU^T, \, (U, \operatorname{diag}(\sigma), V) \text{ top-} r \, \text{SVD of } Y,\, B \in \mathbb{R}^{r \times r} \right\}.
\]

Under spiked asymptotics (\( \min(p,n) \to \infty \)), optimal choices for \( B \) can be derived. Critical limits are the (possibly weighted) inner products of empirical singular vectors \( (\hat{u}_j, \hat{v}_j) \) of \( Y \) with their population counterparts \( (u_k, v_k) \):
\[
\begin{aligned}
& c_{jk} := \lim_{\substack{p, n \to \infty}} \langle \hat{u}_j, u_k \rangle, \qquad \tilde{c}_{jk} := \lim_{\substack{p, n \to \infty}} \langle \hat{v}_j, v_k \rangle, \\
& d_{jk} := \lim_{\substack{p, n \to \infty}} \langle \Omega \hat{u}_j, \Omega \hat{u}_k \rangle, \qquad c^{\omega}_{jk} := \lim_{\substack{p, n \to \infty}} \langle \Omega \hat{u}_j, \Omega u_k \rangle,
\end{aligned}
\]
with analogous definitions for the right singular vectors.

Define the matrices
\[
D = (d_{jk}), \quad C = (c_{jk}^{\omega}), \quad \widetilde{D}, \quad \widetilde{C}, \quad T = \operatorname{diag}(t_1, \dots, t_r).
\]

The optimal estimator is given by
\[
B^{\text{opt}} = D^+ C T C^T D^+,
\]
where \( D^+ \) is the Moore–Penrose pseudoinverse of \( D \), and hence
\[
\widehat{X} = U D^+ C T C^T D^+ V^T.
\]
In the case of diagonal \( D \) and \( C \) ("weighted-orthogonality"), this yields explicit componentwise shrinkage:
\[
f_k = t_k c_k \tilde{c}_k
  \frac{\alpha_k}{c_k^2 \alpha_k + s_k^2 \mu}
  \frac{\beta_k}{\tilde{c}_k^2 \beta_k + \tilde{s}_k^2 \nu},
\]
with
- \( \alpha_k, \beta_k \): row/column-weight projections of population singular vectors,
- \( \mu = \frac{1}{p} \operatorname{tr}(\Omega^2), \nu = \frac{1}{n} \operatorname{tr}(\Pi^2) \),
- \( c_k, s_k, \tilde{c}_k, \tilde{s}_k \): unweighted cosines and sines.

The proof proceeds by expanding the weighted error in singular subspaces, employing spiked-model asymptotics for all cross-terms, and minimizing the quadratic form in \( B \) [1902.09474].

## 3. Localized Denoising via Combinations of Selective Weights

SDDLM supports localized denoising through orthogonal projections partitioning the \( (i, j) \)-index space. Consider projections \( \{\Omega_i\}_{i=1}^I \) (row) and \( \{\Pi_j\}_{j=1}^J \) (column), decomposing the identity: \( \sum_{i=1}^I \Omega_i = I_p \), \( \sum_{j=1}^J \Pi_j = I_n \).

The localized denoiser is constructed as follows:

```
Algorithm 2: Localized denoising for unweighted loss
Input:
    Y
    {Ω_i}_{i=1..I}, {Π_j}_{j=1..J} (pairwise-orthogonal projections)
Output:
    X_loc  (the localized estimate of X)
for i=1..I, j=1..J do
    (1) Construct the weighted estimator:
        X^(i,j) = OptimalSpectralDenoiser(Y, Ω_i, Π_j)
    (2) Record its (estimated) AMSE: AMSE^(i,j)
end
X_loc = sum_{i=1}^I sum_{j=1}^J   Ω_i * X^(i,j) * Π_j
AMSE_loc = sum_{i=1}^I sum_{j=1}^J AMSE^(i,j)
```

Decomposing the error into region-wise blocks, SDDLM denoises each block optimally for its local weight and aggregates the resulting estimates. This approach yields total unweighted error no larger than global singular-value shrinkage, with strict improvement when singular vectors are heterogeneously distributed.

## 4. SDDLM as Weight Selection for Targeted Denoising

The term "Selective Denoising Loss (SDDLM)" refers to the construction of \( W \) for user-specified prioritization:
- Large \( W_{ij} \): high-priority denoising for entry \( (i, j) \).
- Small or zero \( W_{ij} \): deprioritized or ignored entries.

Examples recapitulated:
- Submatrix focus: \( W \) equals the submatrix indicator.
- Heteroscedastic whitening: \( W_{ij} = 1/\sigma_{ij} \).
- Missing data: \( W_{ij} = 0 \) for unobserved, \( W_{ij} = 1/\sqrt{p_{ij}} \) for observed.

A global "temperature" or focus parameter \( \lambda > 0 \) is sometimes introduced:
\[
W \gets \lambda W, \qquad L_W(X, \widehat{X}) \propto \lambda^2 L_W(X, \widehat{X}).
\]
Tuning \( \lambda \) allows balancing fidelity within versus outside the focus region. Cross-validation on a held-out validation set of entries can be employed for parameter selection.

## 5. Empirical Performance and Applicability

Empirical studies demonstrate that SDDLM confers substantial improvements relative to unweighted singular-value shrinkage:

| Application                          | Standard method rel. error | SDDLM/localized rel. error | Notes                                             |
|---------------------------------------|---------------------------|----------------------------|---------------------------------------------------|
| MIT-logo image (with corners focused) | ~0.125                    | ~0.074                     | 15x30 grid, heterogeneity in corners              |
| Checkerboard synthetic (rank-2)       | ~0.19                     | ~0.14                      | 70% energy in "light" squares (800x800)           |
| Submatrix denoising (rank 1)          | -                         | Improved                   | SDDLM outperforms global and submatrix-only cases |
| Doubly-heteroscedastic noise          | -                         | Up to 30% better           | Outperforms OptShrink for condition number κ=16   |
| Missing–values case                   | -                         | Improved                   | Outperforms nuclear-norm completion (200x400)     |

In all scenarios, SDDLM does not underperform unweighted shrinkage in asymptotic regimes and, even with additional parameter overhead, more than compensates with focused accuracy in practical sample sizes.

## 6. Implementation and Theoretical Guarantees

The core algorithms and explicit formulas are detailed in [1902.09474], with:
- Algorithm 1: OptimalSpectralDenoiser (for general \( W \) or pairs \( \Omega, \Pi \)).
- Algorithm 2: Localized denoising (for decompositions into projection blocks).
- Application-specific derivations for submatrix, heteroscedastic, and missing data settings (see Section 6).
- Asymptotic theory and proofs in the Appendix.

The theoretical foundation ensures that SDDLM asymptotically achieves the optimal selective tradeoff prescribed by \( W \), yielding nontrivial improvements in heterogeneous or region-focused denoising scenarios. All steps are justified by rigorous spiked-model asymptotics and random matrix theory, with turnkey algorithms provided for direct research implementation [1902.09474].

Source: https://www.emergentmind.com/topics/selective-denoising-loss-sddlm