Unsupervised Degradation Embeddings

• Unsupervised degradation embeddings are learned feature descriptors that quantify the severity and type of signal degradation without relying on manual annotations.
• They leverage methods like triplet loss in audio (NOMAD) and probabilistic noise injection in image GANs to distill perceptual degradation cues into robust representations.
• Applications include reference-free quality ranking, enhancing restoration losses, and generating pseudo-labels, resulting in improved metrics such as PESQ, PSNR, and LPIPS.

Unsupervised degradation embeddings refer to learned feature representations that capture the characteristics and severity of signal or perceptual quality degradations in an unsupervised setting. These embeddings serve as compact, task-agnostic quantitative descriptors of how much and in what manner an input (such as audio or an image) has been degraded, without relying on ground-truth subjective ratings or explicit human annotation. Recent research demonstrates the power and versatility of such embeddings across domains, particularly in audio quality assessment and image super-resolution, by exploiting self-supervised objectives, perceptual surrogates, and stochastic generative models (Ragano et al., 2023, Lee et al., 2022).

1. Concept and Motivation

Unsupervised degradation embeddings are designed to summarize the perceptual difference between a clean reference and its degraded variants using data-driven, automated approaches instead of costly human studies. The core idea is to learn a mapping from signals to an embedding space in which distances or structures reflect the perceptual severity and sometimes the type of degradation. This is crucial in domains where degradations are diverse and labeled data is expensive to acquire, such as real-world audio enhancement or image super-resolution. The motivation is twofold: enabling reference-free or non-matching-reference analysis, and providing meaningful perceptual constraints for generative or restoration tasks (Ragano et al., 2023, Lee et al., 2022).

2. Learning Degradation Embeddings in Audio: The NOMAD Approach

The NOMAD (Non-Matching Audio Distance) framework exemplifies unsupervised learning of degradation embeddings for speech signals (Ragano et al., 2023). NOMAD fine-tunes a pre-trained audio embedder (wav2vec 2.0 BASE) using a triplet loss guided by an objective perceptual similarity metric (NSIM), without requiring human labels.

Model Architecture and Loss

• A frozen convolutional encoder and fine-tuned transformer stack yield frame-level features, followed by a linear head and L₂ normalization to obtain final 256-D embeddings.
• For a clean utterance $x$ and its degraded versions $x_a$, $x_p$, $x_n$, the loss ensures that degraded signals closer to the clean reference in NSIM space also lie closer in embedding space:

$$Q(x, x_a) > Q(x, x_p) > Q(x, x_n) \implies \|f(x) - f(x_a)\|_2 < \|f(x) - f(x_p)\|_2 < \|f(x) - f(x_n)\|_2.$$

• Triplet mining is unsupervised and leverages NSIM [see section 2.1–2.4, (Ragano et al., 2023)].

Embedding Space Properties and Inference

The resulting embedding space disentangles speech content from degradation, as all triplets share the same clean anchor. At inference, distances between an arbitrary degraded sample and multiple non-matching clean references are averaged to provide robust perceptual scores, enabling both reference-based and non-matching-reference quality assessment.

3. Generative Embeddings for Image Degradation: Probabilistic GANs

Lee et al. (Lee et al., 2022) present a complementary approach for the image domain, where unsupervised degradation embeddings are formed implicitly within probabilistic hierarchical latent-variable GANs constructed as degradation generators.

Probabilistic Degradation Generator

• Instead of deterministically mapping an HR image $y$ to an LR $x$, the generator $G_\theta$ injects Gaussian noise at multiple latent layers:

$$p_\theta(x|y) = \int p_\theta(x|z_1) p_\theta(z_T|y) \prod_{i=2}^T p_\theta(z_{i-1}|z_i) dz_{1:T}$$

• This architecture, with per-channel noise injection akin to StyleGAN, spans the diverse and complex real-world degradation space, improving mode coverage relative to a deterministic generator.

Training and Use

• Training uses unpaired adversarial and cycle-consistency losses, with no explicit recognition network or ELBO.
• Multiple generators (with differing architectures) are collaboratively trained, and their synthetic degraded outputs are used to create pseudo-paired data for supervised restoration tasks.

A plausible implication is that the learned latent variables $z$ within $G_\theta$ serve as an implicit degradation embedding, helping bridge the gap between synthetic and real-world degradations and facilitating more robust downstream restoration.

4. Applications

Unsupervised degradation embeddings have demonstrated impact in three major application areas:

• Quality Assessment and Ranking: NOMAD embeddings enable accurate, non-matching-reference perceptual quality measurement, including ranking unseen degradation types (e.g., noise, compression, clipping, reverberation), with strong monotonicity and correlation with ground-truth intensity and MOS (Ragano et al., 2023).
• Perceptual Losses for Enhancement: Embeddings serve as differentiable perceptual losses for improving generative models. For instance, NOMAD-based losses yield subjective and objective gains in speech enhancement tasks (e.g., improved PESQ and listener preference over DEMUCS baseline) (Ragano et al., 2023).
• Pseudo-Label Generation for Restoration: Probabilistic image degradation generators use latent embeddings to synthesize diverse, realistic LR–HR pairs, crucial for unsupervised image super-resolution. Collaborative learning among multiple generators further distills these embeddings into robust restoration models (Lee et al., 2022).

5. Implementation Protocols, Ablations, and Practical Considerations

Deployment and training of unsupervised degradation embedding models follow distinct, empirically validated procedures:

• Audio Embeddings (NOMAD):
  • Training leverages NSIM-computed triplets generated from clean–degraded pairs across parameterized degradation types.
  • Hyperparameters include a triplet margin of $m=0.2$, batch size selected for convergent learning, and frozen lower layers to enforce disentanglement of content and degradation (Ragano et al., 2023).
  • At inference, averaging embedding distances over a set of $>50$ clean, non-matching references mitigates variance inherent to reference selection.
  • Limitations include decreased performance on pure clipping and ongoing reference-induced variance.
• Image Degradation Generators:
  • Each generator is a noise-injected ResNet GAN, trained with cycle-GAN style objectives.
  • Multiple generators and collaborative learning (including pseudo-labeling for real LR images) improve both diversity and robustness.
  • Empirical results confirm that probabilistic noise-injection and collaborative distillation yield higher PSNR/SSIM, lower LPIPS, and more graceful degradation under increased input noise when compared to deterministic approaches (Lee et al., 2022).

6. Empirical Results and Benchmarks

Experimental validation demonstrates that unsupervised degradation embeddings approach and sometimes surpass the performance of traditional, label-heavy or full-reference metrics.

• NOMAD:
  • Achieves Spearman rank-correlations with degradation intensity from $-0.74$ (Noise) to $+0.89$ (Clip/Reverb), consistently outperforming raw wav2vec 2.0 and NORESQA except in clipping.
  • Correlations with human MOS on four databases are competitive with or exceed full-reference metrics (e.g., $r=-0.85$, $\rho=-0.88$ on P23 EXP1) (Ragano et al., 2023).
  • As a loss for speech enhancement, NOMAD variants yield both slight objective gains and substantially improved MUSHRA scores (MT NOMAD: median $72$ vs. baseline $58$ at SNR $2.5$ dB).
• Probabilistic Degradation GANs:
  • On NTIRE2020, two probabilistic generators with collaborative learning achieve PSNR/SSIM up to $27.25$/0.758, substantially improving over deterministic baselines.
  • LPIPS drops from $0.049$ (deterministic) to $0.026$ (probabilistic).
  • Robustness to unseen noise and qualitative diversity in degradations are confirmed via ablation studies (Lee et al., 2022).

A summary of key experimental benchmarks:

Method / Metric	Audio: NOMAD (Ragano et al., 2023)	Image: MSSR (Lee et al., 2022)
Best Pearson $r$ (MOS, P23 EXP1)	$-0.85$	–
Best Spearman $\rho$	$-0.88$	–
PSNR (SR, NTIRE2020, 4×)	–	$27.25$ (probabilistic + collab)
LPIPS (LR-GT comparison)	–	$0.026$ (probabilistic G)

7. Limitations, Practical Deployment, and Outlook

Despite their advantages, unsupervised degradation embeddings present several practical considerations. The need for clean–degraded pairs (albeit unlabeled) for triplet-guided learning remains a bottleneck for domains without synthetic degradation pipelines. Performance can degrade for specific distortion types (e.g., clipping in NOMAD), and non-matching reference approaches introduce variance that must be mitigated by aggregation. In image restoration, the latent-space coverage of degradation generators is crucial; in practice, multiple stochastic generators and collaborative distillation are required to span the diversity of real-world degradations (Ragano et al., 2023, Lee et al., 2022).

Research suggests that further progress in unsupervised degradation embeddings may come from advances in robust perceptual metrics, improved disentanglement of content and degradation features, and more principled generative architectures capable of modeling the full support of real-world degradation manifolds.