CA-Dense U-Net for Speech Enhancement

• The paper demonstrates that CA-Dense U-Net achieves state-of-the-art multichannel speech enhancement by integrating complex ratio masking with a novel channel-attention mechanism.
• Dense connectivity paired with attention-based feature fusion in a U-Net structure enables effective phase-aware processing and non-linear spatial filtering.
• Empirical evaluations on CHiME-3 highlight that the model outperforms traditional beamforming and prior deep learning approaches in both SDR and PESQ.

The Channel-Attention Dense U-Net (CA-Dense U-Net) is a supervised deep learning architecture for multichannel speech enhancement that integrates complex ratio masking and a novel channel-attention mechanism. The model enables end-to-end, non-linear spatial filtering by embedding attention-based feature fusion directly within the latent space of a densely connected U-Net. This approach facilitates phase-aware enhancement and outperforms classical and prior deep learning methods on benchmark datasets such as CHiME-3 (Tolooshams et al., 2020).

1. Multichannel Speech Enhancement: Background and Motivation

Multichannel speech enhancement seeks to estimate the clean speech signal at a designated reference microphone from $C$ observed noisy time-domain recordings, where $y^{(c)}[n] = s^{(c)}[n] + n^{(c)}[n]$ for channel $c$ and sample $n$. Traditional methods such as minimum variance distortionless response (MVDR) beamformers compute per-frequency linear filters $w_f \in \mathbb{C}^C$ to produce a spatially filtered output $\hat{S}_f = Y_f w_f^H$, where $Y_f \in \mathbb{C}^{T \times C}$ stacks the short-time Fourier transform (STFT) of each channel for $T$ frames.

Single-channel deep learning methods typically estimate a real-valued time-frequency mask $M(f, t)$ applied to the magnitude of the noisy mixture, neglecting phase and spatial cues. Some "neural beamforming" approaches leverage these masks within traditional MVDR pipelines to enable mask-informed spatial filtering but restrict the model to linear, frequency-wise operations.

Conventional multichannel deep learning techniques may encode inter-channel phase differences (IPD), inter-level differences (ILD), and related features as additional network inputs, yet they frequently estimate only real-valued masks and reuse the noisy phase, thereby limiting the ability to jointly enhance magnitude and phase or learn higher-order spatial relations. These limitations underscore the need for architectures explicitly designed for full-rank multichannel feature fusion and end-to-end phase enhancement (Tolooshams et al., 2020).

2. CA-Dense U-Net: Architectural Overview

The architecture of CA-Dense U-Net consists of three principal stages:

1. Encoder: Fixed STFT and inverse-STFT (iSTFT) layers transform time-domain signals $y \in \mathbb{R}^{N \times C}$ into a stacked complex-valued spectrogram $y^{(c)}[n] = s^{(c)}[n] + n^{(c)}[n]$0, where $y^{(c)}[n] = s^{(c)}[n] + n^{(c)}[n]$1 (with one frequency bin dropped) and $y^{(c)}[n] = s^{(c)}[n] + n^{(c)}[n]$2.
2. Mask-Estimation Core: The architecture is structurally a U-Net, incorporating $y^{(c)}[n] = s^{(c)}[n] + n^{(c)}[n]$3 down-sampling and $y^{(c)}[n] = s^{(c)}[n] + n^{(c)}[n]$4 up-sampling blocks, where $y^{(c)}[n] = s^{(c)}[n] + n^{(c)}[n]$5 (as used in experiments). Each block contains a DenseNet-style dense-block, in which the input to each convolutional layer is the concatenation of all preceding feature maps within that block. This design promotes feature re-use and enhances gradient flow stability.
3. Decoder: The output of the mask estimation network generates complex ratio masks, which are then applied to recover the clean speech and noise in the complex spectrogram domain; the enhanced waveforms are reconstructed via iSTFT.

Dense connectivity within each block increases feature-map dimensionality, which is subsequently reduced by pooling (encoder) or up-sampling (decoder). Channel-attention units are inserted after every dense-block, including after the initial block operating on the raw input feature maps. Skip connections are used to preserve fine-grained spectral and temporal information throughout the network flow (Tolooshams et al., 2020).

3. Channel-Attention Mechanism: Formulation and Role

The channel-attention (CA) unit is devised to implement a non-linear, frequency-dependent "beamformer" operating within the network's deep feature space. For each frequency bin, the CA unit recalibrates microphone features by constructing attention weights that allow the network to learn complex spatial filtering at each layer.

• Each CA unit projects the input feature map tensor $y^{(c)}[n] = s^{(c)}[n] + n^{(c)}[n]$6 to key, query, and value representations using $y^{(c)}[n] = s^{(c)}[n] + n^{(c)}[n]$7 convolutions followed by exponential activation:
  • $y^{(c)}[n] = s^{(c)}[n] + n^{(c)}[n]$8, where $y^{(c)}[n] = s^{(c)}[n] + n^{(c)}[n]$9.
  • $c$0.
• For each frequency $c$1, the similarity matrix $c$2 is computed.
• Attention weights $c$3 are derived via a column-wise softmax over magnitudes (phases are preserved): $c$4; $c$5.
• The value representation is aggregated as $c$6 and processed by skipping back into the U-Net flow via concatenation of real and imaginary parts.

By repeating the CA operation at every layer, the network constructs a cascade of non-linear, frequency-wise spatial filters. This paradigm enables robust, data-driven beamforming beyond the linear, per-frequency constraints of classical approaches and improves spatial discrimination in adverse acoustic environments (Tolooshams et al., 2020).

4. Complex Ratio Masking and Training Strategy

CA-Dense U-Net employs complex ratio masking (cRM) to jointly enhance both magnitude and phase components of the speech signal:

• The mask for speech is computed as $c$7 for each frequency-time bin, where $c$8 and $c$9 are clean and noisy complex STFTs, respectively. Real and imaginary parts are concatenated as $n$0.
• The corresponding noise mask is obtained via $n$1.
• Estimated speech and noise are recovered by element-wise complex multiplication: $n$2.

The loss function combines time-domain $n$3 error and $n$4 error between magnitude spectrograms:

$n$5

where $n$6 is chosen such that the time-domain term has twice the initial weight of the magnitude-spectrogram difference (Tolooshams et al., 2020).

Training employs the CHiME-3 simulated 6-microphone dataset, using segments of length 19,200 samples with randomized noise attenuation in $n$7 dB. The STFT uses a Hanning window of length 1,024 and hop 256, with the network depth set to $n$8 and a channel cap of 256 per layer. Optimization is performed using ADAM with learning rate $n$9 and batch size 8.

5. Empirical Evaluation and Performance

The efficacy of CA-Dense U-Net is demonstrated on CHiME-3 using standard speech enhancement metrics:

• Signal-to-Distortion Ratio (SDR): calculated with the BSS-Eval library
• Perceptual Evaluation of Speech Quality (PESQ): using the wideband ITU-T P.862.2 standard

The main evaluated baselines are listed in the table below.

Method	SDR (Dev/Test, dB)	PESQ (Dev/Test)
Noisy Channel-5	5.79 / 6.50	1.27 / 1.27
U-Net (Real, mag. mask)	14.65 / 15.97	2.105 / 2.176
Dense U-Net (Real)	14.90 / 16.86	2.242 / 2.378
Dense U-Net (Complex)	16.96 / 18.40	2.330 / 2.404
CA Dense U-Net (Complex)	17.17 / 18.64	2.368 / 2.436

CA-Dense U-Net (Complex) delivers the highest performance among evaluated methods on both development and test sets. For comparison, neural beamforming [Erdogan et al.] achieves SDR = 15.12 dB and $w_f \in \mathbb{C}^C$0PESQ = 1.02; NMF-informed beamforming yields SDR = 16.16 dB and $w_f \in \mathbb{C}^C$1PESQ = 0.52. CA-Dense U-Net achieves SDR = 18.64 dB and $w_f \in \mathbb{C}^C$2PESQ = 1.16 (Tolooshams et al., 2020).

6. Architectural Insights and Implications

CA-Dense U-Net demonstrates that complex ratio masking is effective in joint magnitude–phase enhancement, resulting in improved SDR and PESQ compared to magnitude masking alone. The CA units serve as learned non-linear beamformers, capable of dynamically reweighting microphone features in latent space at each network layer. DenseNet-based dense connectivity fosters feature re-use and stable training, with skip connections preserving spectro-temporal detail essential for high-resolution signal reconstruction.

Empirically, channel-attention weights correlate with the signal-to-noise ratio of channels and place emphasis on low-frequency bands, corresponding to regions with concentrated speech energy. The results indicate that deep, end-to-end trainable architectures featuring cascaded spatial fusion units can surpass classical linear beamforming and shallow neural approaches, particularly when spatial information must be fused in highly non-stationary or reverberant settings (Tolooshams et al., 2020).

7. Comparative Context and Conclusion

The development of CA-Dense U-Net addresses key limitations of both traditional beamforming and prior deep learning models by enabling non-linear, frequency-dependent spatial filtering directly within the network architecture. This approach achieves state-of-the-art performance on standard benchmarks and demonstrates that integrating attention-based spatial fusion mechanisms deeply within end-to-end learning frameworks can obviate the need for treating spatial filtering as a separable post-processing step.

All formulas, hyperparameters, and architectural innovations described above support reproducibility and provide a foundation for further development in multichannel speech enhancement (Tolooshams et al., 2020).