Your U-Net Dereverberation Model is Secretly an RIR Encoder

Abstract: In this work, we analyze the ability of NCSN++ U-Net based audio dereverberation models to capture global room characteristics in their intermediate representations. Through an empirical study of both a state-of-the-art diffusion-based model and a discriminative counterpart, we show that deeper layers encode structured room impulse response (RIR)-dependent embeddings. Moreover, the discriminative ability of this implicit room representation correlates with dereverberation performance across objective metrics. Motivated by this observation, we propose a training strategy that explicitly conditions the network on pre-trained RIR embeddings, obtained via self-supervised contrastive learning. Incorporating RIR conditioning improves representation quality, accelerates convergence, and enhances dereverberation performance, while significantly reducing the number of reverse diffusion steps required by the diffusion-based model during inference.

• The paper demonstrates that U-Net dereverberation models inherently encode room impulse responses (RIRs) in intermediate layers, correlating with enhanced dereverberation performance.
• It introduces a self-supervised contrastive learning framework to condition the U-Net backbone with dedicated RIR embeddings, leading to faster convergence and improved quality metrics.
• Quantitative results show that RIR-conditioned models require fewer diffusion steps, reducing computational load while maintaining robust speech enhancement.

Analysis of RIR Representation Learning in U-Net-based Audio Dereverberation Models

Context and Motivation

The paper examines the implicit capability of NCSN++ U-Net–based dereverberation models, both diffusion-based and discriminative, to encode structured representations of Room Impulse Responses (RIRs) within their intermediate activations. Dereverberation, critical for speech enhancement, seeks to remove the effects of reverberation caused by reflections in enclosed environments, commonly modeled as a convolution between clean speech and RIR. Classical approaches rely on statistical modeling or explicit RIR estimation—with significant limitations for practical deployment. In contrast, modern deep learning methods, including diffusion models, focus on data-driven mappings, typically in a blind setting where the RIR is unknown.

The deterministic nature of reverberation challenges the probabilistic intuition underpinning diffusion models, traditionally framed for additive noise processes. This paper interrogates whether generative models genuinely capture the reverberation artifacts or if a latent RIR-encoding property is responsible for their empirical performance.

Implicit RIR Encoding in U-Net Dereverberation Models

Through unsupervised clustering (t-SNE) of embeddings extracted from the attention blocks of trained NCSN++ models, the authors demonstrate that both diffusion-based and discriminative U-Net architectures learn intermediate representations highly structured by the RIR (Figure 1). These clusters emerge strongly in mid-level bottleneck and attention layers, largely independent of the model's loss or training regime.

Figure 1: t-SNE visualizations indicating strong RIR-dependent latent structure in intermediate U-Net features for both discriminative and diffusion-based dereverberation models.

The phenomenon implies a hybrid behavior: despite their intended probabilistic mapping, diffusion models encode deterministic RIR-specific information during inference, particularly when reverberant speech is present at each diffusion step as a concatenated input. This suggest that the models' dereverberation capabilities arise from implicit knowledge localization of the acoustic degradation operator, rather than purely probabilistic denoising.

Explicit Conditioning with Contrastive RIR Embeddings

To leverage and regularize this capability, the paper introduces a self-supervised contrastive learning framework to train dedicated RIR encoders (ResNet, Conformer variants), which produce embeddings invariant to speech content but sensitive to RIR characteristics. These embeddings are used to condition the U-Net backbone through FiLM-based modulation in each residual block, providing direct room-acoustic context to all layers.

Comparative t-SNE plots for the RIR encoders and conditioned models confirm improved cluster separation, highlighting enhanced representational quality and discriminative ability relative to their unconditioned counterparts (Figure 1, e–f). The conditioning accelerates convergence and stabilizes training dynamics.

Numerical Results and Performance Gains

Objective metrics (PESQ, DNSMOS) reveal statistically significant gains in dereverberation quality and perceptual speech assessment across varying numbers of reverse diffusion inference steps (N), particularly for Conformer-conditioned models (Figure 2). Conditioning leads to PESQ improvements ranging from 0.17 to 0.28, with similar trends in DNSMOS, reducing the computational load for real-time deployment by requiring fewer iterative denoising steps.

Figure 2: PESQ and DNSMOS improvements under varied diffusion steps, demonstrating that RIR-conditioned models achieve superior quality with fewer steps.

Training curves further illustrate accelerated convergence in validation PESQ, especially when excluding exponential moving average (EMA) from the optimizer in RIR-conditioned models, overcoming optimization stability issues observed in the baseline (Figure 3).

Figure 3: Validation PESQ trajectories indicating faster and more stable convergence for RIR-conditioned diffusion models.

Theoretical and Practical Implications

The analysis substantiates that state-of-the-art deep dereverberation models, including diffusion-based ones, are not merely generative but also act as implicit RIR encoders. The observed correspondence between the strength of latent RIR clusters and dereverberation metrics validates the necessity for explicit conditioning and knowledge localization. Practically, explicit RIR embedding conditioning enables significant inference speedup and quality gains, critical for real-time speech enhancement scenarios.

The approach also addresses convergence bottlenecks inherent to current diffusion models, suggesting that architectural modifications that inject acoustic context can mitigate reliance on EMA heuristics and extensive diffusion steps, streamlining deployment without sacrificing robustness.

Future Directions

Further exploration of hybrid architectures that utilize explicit contrastive embeddings for broader acoustic parameter estimation (beyond RIRs) could generalize the method to multi-modal restoration tasks. The disentanglement of speech and environment representations within end-to-end models may yield greater adaptability across variable acoustic and linguistic domains. Integrating learned RIR embeddings with generative augmentations and uncertainty quantification may enhance robustness in out-of-distribution and mismatched acoustic settings.

Conclusion

This paper provides empirical and theoretical evidence that U-Net dereverberation models, especially diffusion-based variants, inherently encode RIR structure within their intermediate activations. The strength of this RIR-dependent representation is highly correlated with dereverberation performance. Conditioning the backbone with explicit, contrastively trained RIR embeddings via FiLM modulation enhances both convergence and output quality, significantly reducing the computational requirements. This insight paves the way for knowledge-localized architectures in speech enhancement, with promising implications for both model interpretability and real-time robustness.