---
title: Complex Ratio Masks (CRM) in Audio Processing
url: https://www.emergentmind.com/topics/complex-ratio-masks-crm
type: topic
---

# Complex Ratio Masks (CRM) in Audio Processing

A Complex Ratio Mask (CRM) is a time–frequency-domain mask defined on the complex short-time Fourier transform (STFT) of a signal that enables simultaneous magnitude and phase estimation. CRMs generalize conventional magnitude masks by directly predicting both real and imaginary components, allowing explicit correction of phase artifacts that limit the performance of magnitude-only approaches in speech enhancement, music source separation, and multi-channel spatial filtering. Recent research demonstrates that CRMs yield significant improvements in objective and perceptual metrics, notably signal-to-distortion ratio (SDR) and perceptual evaluation of speech quality (PESQ), when integrated into modern deep learning frameworks for audio source separation and enhancement tasks [2103.12864, 2010.14122, 1901.00295, 2104.12359].

## 1. Mathematical Formulation and Interpretation

Let $X(f,t)\in\mathbb{C}$ denote the complex STFT of a mixture signal, and $S(f,t)\in\mathbb{C}$ the STFT of a target source. The CRM at each time–frequency (T–F) bin is defined as:
\[
M_c(f,t) = \frac{S(f,t)}{X(f,t)}
\]
Expressed via real and imaginary parts:
\[
\begin{aligned}
X &= X_r + i X_i \\
S &= S_r + i S_i \\
M_c &= M_r + i M_i \\
M_r &= \frac{S_r X_r + S_i X_i}{X_r^2 + X_i^2} \\
M_i &= \frac{S_i X_r - S_r X_i}{X_r^2 + X_i^2}
\end{aligned}
\]
Thus, $M_r$ mixes and rescales the input’s real and imaginary parts, while $M_i$ facilitates phase rotation.

When a CRM $M_c(f,t)$ is applied to the mixture,
\[
\hat S(f,t) = M_c(f,t) \cdot X(f,t)
\]
or, in polar form if $X(f,t) = |X| e^{i \angle X}$ and $M_c = |M_c| e^{i\angle M_c}$,
\[
\hat S = |M_c||X| e^{i(\angle X + \angle M_c)},
\]
so the CRM scales the magnitude and rotates the phase to approximate the target.

CRMs can be constrained by applying pointwise $\tanh$ nonlinearities to avoid unbounded mask values, such that $|M_c| = \tanh(|\bar O|)$ and $\angle M_c = \angle \bar O$ for a neural output $\bar O(f,t)$ [2103.12864, 2010.14122].

## 2. Network Architectures and Training Paradigms

CRMs are typically estimated by neural networks operating in the T–F domain. Input representations include either the magnitude $|X(f,t)|$ (for magnitude-only models) or explicit real and imaginary components $\{\operatorname{Re} X(f,t), \operatorname{Im} X(f,t)\}$ (for complex models).

Common architectural choices:
- **U-Net**: Six-layer encoder–decoder with skip connections and real-valued convolutions. Input fed as two real channels (re/imag), outputting two real-valued masks which are recombined to form $M_c$ [2103.12864].
- **Complex U-Net/BLSTM**: All convolutions and batch norms generalized to the complex domain; complex arithmetic (using paired real convolutions), complex activations [2104.12359].
- **Complex-valued LSTM (RCLSTM)**: Sequential models processing complex-valued inputs using paired real LSTMs recombined as
  \[
  \begin{aligned}
  H_r[t] &= f_1(Z_r[1:t]) - f_2(Z_i[1:t]) \\
  H_i[t] &= f_2(Z_r[1:t]) + f_1(Z_i[1:t])
  \end{aligned}
  \]
  [2010.14122].

Architectures may incorporate spatial features (e.g., inter-channel phase differences, directional features) for multi-channel scenarios, stacking these as complex-valued feature tensors [2104.12359].

## 3. Supervised Learning Objectives

CRM estimation is driven by composite losses reflecting both spectral and time-domain criteria:

- **Spectral Losses**: $L_\mathrm{mag} = \sum_{f,t} ||\hat S(f,t)| - |S(f,t)||_1$ penalizes deviations in estimated versus reference magnitudes [2103.12864]. Fully complex MSE losses, $L_\mathrm{CplxMSE} = \sum_{f,t} |\hat S(f,t)-S(f,t)|^2$, can be used to jointly optimize real and imaginary components [2103.12864, 2010.14122].
- **Time-domain Losses**: Scale-invariant source-to-distortion ratio (SI-SDR) is a common criterion, optimizing the negative cosine similarity between estimated and reference time-domain signals recovered via inverse STFT [2103.12864, 2104.12359]:
  \[
  L_\mathrm{SDR} = -\frac{y \cdot \hat y}{\|y\|\|\hat y\|}
  \]
- **Hybrid Losses**: Networks may be trained with a sum of spectral and SI-SDR losses for best convergence and generalization [2103.12864].
- **Consistent Spectrogram Masking**: Losses can be computed after inverse STFT to enforce that the estimated spectrogram corresponds to an actual time-domain signal, reducing artifacts and improving convergence [1901.00295].

Bounding the CRM’s real and imaginary parts by $\tanh$ nonlinearities mitigates instability due to small $|X|$ magnitudes [2010.14122].

## 4. Advantages Over Magnitude-Only Masking

Magnitude masking estimates only $|\hat S|$ and reuses the input mixture phase, leading to inherent phase artifacts, notably "phase smearing" and musical noise. In contrast, CRMs enable:

- **Phase Correction**: Rotates the input phase to match the target, reducing residual artifacts [2103.12864].
- **Interference Cancellation**: Imaginary component of the CRM can cancel interference in phase, impossible for real-valued masks [2103.12864].
- **Sharper Transients**: Improved reconstruction of percussive onsets when compared to magnitude-only separation [2103.12864].

Quantitative gains are consistent across tasks and datasets:
- SDR improvements of up to $\sim$1 dB on bass and drums, and 0.3–0.7 dB for vocals and "other" [2103.12864].
- Perceptual gains in PESQ (e.g., RCLSTM achieves PESQ = 2.62, a 4.3% improvement over real-valued counterparts) [2010.14122].
- In multi-channel separation, SI-SDR gains of $\sim$1.1 dB (12.1% relative) and up to 33.1% relative reduction in word error rate versus baseline real-mask separation [2104.12359].

Subjective evaluation confirms perceptible reductions in artifacts and increased preference for CRM-based outputs [2103.12864].

## 5. Extensions and Consistency Constraints

Application of an arbitrary complex mask can move a spectrogram out of the *consistent* subspace, yielding artifacts through STFT/ISTFT inconsistency [1901.00295]. Consistent Spectrogram Masking (CSM) addresses this by defining the loss over the reconstructed waveform after ISTFT, guaranteeing consistency and bringing two benefits:
1. Reduced solution space (faster training convergence).
2. Fewer artifacts due to frame misalignment.

CSM yields faster training (10–20% fewer epochs) and measurable PESQ/SNR improvements over standard CRM masking [1901.00295].

In multi-channel scenarios, estimated CRMs can be integrated with Minimum Variance Distortionless Response (MVDR) beamforming. Here, the CRM output informs spatial correlation matrix estimation and spatial filtering, further boosting SI-SDR and reducing recognition errors in ASR tasks [2104.12359].

## 6. Experimental Benchmarks

Empirical evaluations corroborate the superiority of CRM approaches:

| Task/Dataset             | Metric      | Magnitude Mask | CRM (Best)    |
|-------------------------|-------------|---------------|---------------|
| MUSDB18 (vocals)        | SDR (dB)    | 3.6           | 3.9           |
| MUSDB18 (bass)          | SDR (dB)    | 6.0           | 6.4           |
| MUSDB18 (drums)         | SDR (dB)    | 4.6           | 5.7           |
| VoiceBank+DEMAND        | PESQ        | 2.51–2.55     | 2.62          |
| Sim. multi-ch (speech)  | SI-SDR (dB) | 10.6–11.3     | 12.0*         |
|                         | WER (%)     | 25.96–18.04   | 17.03*        |

*With MVDR post-filter integration. Source: [2103.12864, 2104.12359, 2010.14122].

Largest CRM gains are observed for low-frequency sources or high-SNR regimes, and in settings with significant phase-related interference.

## 7. Limitations and Future Directions

Although CRMs address key deficiencies of magnitude-only masking, challenges remain:

- Mask inconsistency can introduce artifacts if not constrained; CSM approaches offer a promising mitigation [1901.00295].
- CRM estimation is more demanding computationally and architecturally, motivating research into efficient complex-valued networks and deeper integration of complex arithmetic [2104.12359, 2010.14122].
- Further advances are anticipated in fully complex-gated recurrent architectures, application to direct time-domain enhancement, and extension to more general spatial scenarios.

A plausible implication is that CRMs, combined with consistency constraints and complex architectures, will define new baselines for both single- and multi-channel audio enhancement and separation tasks.

---

**References:**  
- [2103.12864] Learned complex masks for multi-instrument source separation  
- [2010.14122] Phase Aware Speech Enhancement using Realisation of Complex-valued LSTM  
- [1901.00295] End-to-End Model for Speech Enhancement by Consistent Spectrogram Masking  
- [2104.12359] Complex Neural Spatial Filter: Enhancing Multi-channel Target Speech Separation in Complex Domain

Source: https://www.emergentmind.com/topics/complex-ratio-masks-crm