---
title: Conformer-based CSS Architectures
url: https://www.emergentmind.com/topics/conformer-based-css-architectures
type: topic
---

# Conformer-based CSS Architectures

Conformer-based Conditional Speaker Separation (CSS) architectures constitute a class of time-domain neural models for extractive speech separation, exploiting both local convolutional context and long-range self-attention. In the context of target speaker extraction, these architectures integrate a learned speaker embedding into the separator network, enabling extraction of a desired speaker from a single-channel input mixture. The dominant approaches—Conformer-FFN and TCN-Conformer—jointly train a speaker embedder and separator, and are characterized by the systematic combination of Conformer blocks (comprising multi-head self-attention, convolution, and feed-forward layers) with either additional feed-forward modules or temporal convolutional network (TCN) blocks to maximize separation performance in diverse and challenging audio conditions [2205.13851].

## 1. Architecture and System Overview

Conformer-based CSS systems operate within a three-stage, time-domain extraction pipeline. The initial **encoder** converts waveform frames $y(t)$ into latent vectors $Z \in \mathbb{R}^{T \times D}$. A **speaker embedder**—a ResNet-based network—outputs a fixed embedding $E_{spk} \in \mathbb{R}^{256}$ for the target, which is tile-repeated across the sequence dimension. The **separator** network, parameterized by either Conformer-FFN or TCN-Conformer design, receives the concatenated input $[Z\,\|\,E_{spk}]$ and estimates a mask $M \in [0,1]^{T \times D}$. The **decoder**, realized by a transposed convolution, reconstructs the target time-domain signal $\hat{s}(t)$ as
$$
\hat{S} = \text{Decoder}( \hat{M} \odot Z )
$$

### Conformer-FFN Separator

This design consists of $K$ repeated stacks, each containing:
- A Conformer block as originally formulated in Gulati et al. (2020).
- An external feed-forward network (FFN) comprising two linear layers, Swish activation, and dropout.
- At each step, the output is concatenated with $E_{spk}$ before input to the next Conformer block.

**Block diagram for $K=3$:**
$$
[Z \| E_{spk}] \rightarrow C_1 \rightarrow FF_1 \rightarrow (\,) \| E_{spk} \rightarrow C_2 \rightarrow FF_2 \rightarrow (\,) \| E_{spk} \rightarrow C_3 \rightarrow FF_3 \rightarrow \text{mask-head} \rightarrow M
$$

### TCN-Conformer Separator

This architecture stacks $K$ blocks of the following sequence:
- Temporal Convolutional Network (TCN) block: dilated 1D convolution, PReLU and group LayerNorm (gLN), depthwise separable structure.
- Conformer block as above.
- Concatenation with $E_{spk}$ before next TCN block.

**Block diagram for $K=3$:**
$$
[Z \| E_{spk}] \rightarrow T_1 \rightarrow C_1 \rightarrow (\,) \| E_{spk} \rightarrow T_2 \rightarrow C_2 \rightarrow (\,) \| E_{spk} \rightarrow T_3 \rightarrow C_3 \rightarrow \text{mask-head} \rightarrow M
$$

## 2. Conformer Block Formulation

Each Conformer block combines four key submodules:

- **Feed-Forward Network (FFN) Pre-MHSA**:
  $$
  \text{FFN}(X) = W_2 \cdot \text{Swish}(W_1 X + b_1) + b_2
  $$
  First FFN employs residual scaling by 0.5.

- **Multi-Head Self-Attention (MHSA)**:
  $$
  Q^h = X W_Q^h,\quad K^h = X W_K^h,\quad V^h = X W_V^h, \quad d_k = D/H
  $$
  $$
  A^h = \text{softmax}\left( \frac{Q^h {K^h}^\top}{\sqrt{d_k}} \right)
  $$
  $$
  O^h = A^h V^h
  $$
  Heads are concatenated and linearly projected, followed by dropout and residual addition.

- **Convolutional Module**:
  Involves pointwise convolution (expanding to 2D), gated linear unit (GLU) activation, 1D depthwise separable convolution (kernel size $K_c$), batch normalization, Swish nonlinearity, second pointwise convolution, and dropout.

- **FFN Post-Conv**:
  Another instance of the FFN as above, again with 0.5 residual scaling.

**Full forward pass:**
1. $X_0 \leftarrow$ Pre-FFN($X$)
2. $X_1 \leftarrow X_0 + \text{Dropout}(\text{MHSA}(\text{LayerNorm}(X_0)))$
3. $X_2 \leftarrow X_1 + \text{Dropout}(\text{ConvModule}(\text{LayerNorm}(X_1)))$
4. $X_3 \leftarrow X_2 + 0.5 \cdot \text{Dropout}(\text{FFN}(\text{LayerNorm}(X_2)))$
5. Output = $X_3$

## 3. Mask Estimation and Reconstruction

Both separator types parameterize a mask estimator $f_{sep}$ computing
$$
\hat{M} = f_{sep}(Z, E_{spk}) \in [0,1]^{T \times D}
$$
This mask is element-wise applied to the encoded mixture $Z$, and the masked representation is inverted by the decoder to yield the separated time-domain waveform.

## 4. Training Objective and Optimization

Training is end-to-end and supervised, jointly optimizing the speaker embedder and separator via a multi-task loss:
$$
L = L_{sep} + \lambda \cdot L_{emb}
$$
with
- **Separator loss**: Negative multi-scale SI-SNR over three encoder scales,
$$
L_{sep} = -\sum_{\ell=1}^3 w_\ell \cdot \text{SI-SNR}(s_\ell, \hat{s}_\ell)
$$
$w_\ell$ are typically uniform.
  - SI-SNR is computed as
    $$
    \text{SI-SNR}(s, \hat{s})=10 \log_{10} \frac{\|\alpha s\|^2}{\|\alpha s-\hat{s}\|^2}, \quad \alpha = \frac{\langle \hat{s}, s\rangle}{\|s\|^2}
    $$
- **Embedder loss**: Cross-entropy loss over $N$ training speakers,
$$
L_{emb} = -\sum_{i=1}^N y_i \log p_i(E_{spk})
$$

ADAM optimization is used for 150 epochs on 4s audio segments, employing early stopping (6 epochs patience).

## 5. Empirical Evaluation and Results

Table: SI-SDR gain (dB) relative to the input mixture, for systems trained on 2-mix only ($K$=3):

| System                | 2-mix | 3-mix | noisy-mix |
|-----------------------|------:|------:|----------:|
| Mixture (input)       |  2.51 | -1.27 |    -3.21  |
| TCN baseline [20]     | 16.15 |  4.18 |    -2.30  |
| Conformer-FFN         | 15.60 |  4.08 |    -3.64  |
| TCN-Conformer         | 16.85 |  4.56 |    -0.24  |

In the joint test scenario (2/3/noisy-mix):

| System           | K | 2-mix | 3-mix | noisy-mix |
|------------------|---|-------|-------|-----------|
| TCN baseline     | – | 14.87 |  8.43 |    7.92   |
| Conformer-FFN    | 4 | 14.07 |  7.67 |    7.56   |
| TCN-Conformer    | 4 | 17.51 | 10.70 |    9.32   |

TCN-Conformer with $K=4$ achieves absolute SI-SDR improvements over the TCN baseline of +2.64 dB (2-mix), +2.27 dB (3-mix), and +1.40 dB (noisy-mix). Conformer-FFN gives modest gains up to $K=4$ but does not surpass the TCN baseline.

## 6. Ablation Studies and Hyperparameter Configurations

Ablations varying $K$ (number of stacks) demonstrate that Conformer-FFN yields limited improvement, while TCN-Conformer shows monotonically increasing SI-SDR as $K$ grows from 1 to 3 to 4. Core hyperparameters include:

- Encoder/decoder filter lengths: [2.5 ms, 10 ms, 20 ms].
- Separator dimension: $D=512$ (after concat); external FFN output size: 256.
- Conformer: $H=8$ attention heads, $d_k=64$, Conv kernel $K_c=31$, FFN expansion factor: 4.
- TCN block: two $1\times1$ conv, PReLU+gLN, depthwise separable CNN (kernel=3, dilation=$2^\ell$).
- External FFN: two linear layers, Swish, dropout $p=0.1$.
- $K \in \{1,3,4\}$ stacks.

This suggests that deeper (larger $K$) TCN-Conformer architectures, through alternating convolutional and Conformer modules, more effectively capture both short-term and long-range dependencies required for robust CSS.

## 7. Context and Significance

The Conformer-based CSS architectures interleave convolutional and self-attention mechanisms, allowing simultaneous modeling of local structure (via TCN/dilated CNN) and global dependencies (via MHSA). The TCN-Conformer design, by alternating TCN and Conformer blocks, consistently outperforms baselines in all tested conditions. These findings establish the utility of Conformer-based designs for speaker-conditioned separation, highlighting the importance of architectural hybridization for tackling complex audio mixtures [2205.13851].

Source: https://www.emergentmind.com/topics/conformer-based-css-architectures