---
title: Kaldi-Based ASR Protocol
url: https://www.emergentmind.com/topics/kaldi-based-asr-protocol
type: topic
---

# Kaldi-Based ASR Protocol

A Kaldi-based ASR protocol refers to end-to-end speech processing systems that leverage protocol components or modeling strategies akin to the Kaldi toolkit, with a particular focus on architectures, workflows, and system evaluation typical of state-of-the-art research. In the context of target speaker extraction (TSE), recent approaches integrate conformer-based and temporal convolutional network (TCN) separators, speaker conditioning, and multi-scale encoders for robust performance in adverse speech mixtures. Two principal separator architectures—the Conformer-FFN and the TCN-Conformer—demonstrate how such protocols combine deep learning blocks with established practices in the Kaldi ecosystem to enable accurate, speaker-aware speech separation and robust automatic speech recognition in multi-speaker and noisy environments [2205.13851].

## 1. Conformer Layer: Structure and Computation

Each Conformer block comprises four sub-layers: (1) position-wise feed-forward (FFN), (2) multi-head self-attention (MHSA), (3) depthwise convolutional module, and (4) a second FFN. All sub-layers deploy pre-normalization and residual connections.

Given input $X \in \mathbb{R}^{T \times D}$:

- **FFN₁:** $Y_1 = \text{LayerNorm}(X)$, $\text{FFN}(X) = \text{Dropout}(\text{Swish}(XW_1 + b_1))W_2 + b_2$, $W_1 \in \mathbb{R}^{D \times 4D}$, $W_2 \in \mathbb{R}^{4D \times D}$, and $Z_1 = X + \frac{1}{2}\text{FFN}(Y_1)$.
- **MHSA:** LayerNorm normalizes $Z_1$; queries, keys, and values are projected into $H=8$ heads, $d_k = D / H$, with standard scaled dot-product attention and output dimension $D$; $Z_2 = Z_1 + \text{Dropout}(\text{MHSA}(Y_2))$.
- **Convolutional Module:** Includes point-wise convolution (expansion factor $C=3$), gated linear unit (GLU), depthwise convolution (kernel size=31, padding=15, groups=D), batch normalization, Swish activation, another point-wise convolution, dropout, and residual link; output $Z_3 = Z_2 + \text{Dropout}(F)$.
- **FFN₂:** Second FFN and residual connection as in FFN₁ yields $Z_4 = Z_3 + \frac{1}{2}\text{FFN}(Y_4)$.

LayerNorm and Dropout occur within each residual block, following best practices for stability and regularization.

## 2. Separator Architectures: Conformer-FFN and TCN-Conformer

Both separators process features derived from a multi-scale encoder. For frame lengths $L \in \{2.5\,\mathrm{ms},\,10\,\mathrm{ms},\,20\,\mathrm{ms}\}$, the encoder outputs $X^{(s)} \in \mathbb{R}^{T_s \times 512}$ per scale $s$. A ResNet-based speaker embedder extracts an embedding $e \in \mathbb{R}^{256}$ from a reference utterance, tiled across time and concatenated to $X^{(s)}$.

**Conformer-FFN Network Workflow:**
- **Input projection:** $U_0 = \text{concat}(X^{(s)}, e) \in \mathbb{R}^{T_s \times 768}$, projected to $512$ dimensions.
- **K Conformer stacks:** For $k = 1..K$ ($K \in \{1,3,4\}$), $C_k = \text{ConformerBlock}(U_{k-1}’)$ (input $512$), followed by an ExternalFFN ($\mathbb{R}^{512} \to \mathbb{R}^{256}$), then concatenation with $e$.
- **Mask estimation:** $M^{(s)} = \text{Sigmoid}(U_K’W_{mask} + b_{mask}),\,W_{mask} \in \mathbb{R}^{512 \times 512}$.
- **Reconstruction:** Masking $X^{(s)}$, multi-scale decoder reconstructs $\hat{y}(t)$.

**TCN-Conformer Network Workflow:**
- **Initialization:** $U_0 = \text{concat}(X^{(s)}, e) \in \mathbb{R}^{T_s \times 768}$. Projection to match TCN input, typically $512$ channels.
- **K block stacks:** For $k = 1..K$, apply TCNBlock (per Lüo & Mesgarani, 2019) followed by ConformerBlock with concatenated $e$. Mask generation and decoder are identical to the Conformer-FFN pathway.

The TCNBlock consists of one 1×1 convolution followed by PReLU and Group LayerNorm (GLN), a depthwise convolution (kernel size 3, configurable dilation), and another 1×1 convolution, aggregated via residual connection.

## 3. End-to-End Time-Domain Target Speaker Extraction Workflow

The architecture forms a pipeline comprising:
- **Multi-scale encoder:** Applied to mixture $x(t)$, yielding $\{X^{(s)}\}$.
- **Speaker embedder:** Processes reference $r(t)$ to produce $e \in \mathbb{R}^{256}$.
- **Separator (Conformer-FFN or TCN-Conformer):** Accepts $\{X^{(s)}, e\}$, outputs masks $M^{(s)}$.
- **Masking and decoding:** $M^{(s)} \cdot X^{(s)}$ reconstructed to waveform $\hat{y}(t)$.

Speaker conditioning is performed by constant concatenation of the tiled embedding $e$ to every frame, with no feature-wise affine transform (e.g., FiLM) applied, although replacement is possible.

## 4. Training Procedures, Datasets, and Evaluation

**Dataset:** WSJ0 + WHAM, encompassing 2-speaker (“2-mix”), 3-speaker (“3-mix”), and noisy 2-speaker (“noisy-mix”) mixtures. Interferer SNR uniformly sampled from [0, 5] dB. Approximately $48$k training, $13$k development, and $7.5$k test utterances per condition.

**Training regime:** Separator and speaker embedder jointly trained using a multi-task loss:
- $\mathcal{L} = \mathcal{L}_{\rm SI\textrm{-}SNR,\;multiscale} + \mathcal{L}_{\rm CE,speakerID}$
- SI-SNR loss applies scale weights $\{0.5, 0.5, 1.0\}$.
- Cross-entropy used for speaker identification.
- 4-second input segments, Adam optimizer (learning rate $1 \times 10^{-3}$ decaying per schedule), 150 epochs, early stopping patience 6.

**Evaluation metric:** Scale-invariant signal-to-distortion ratio (SI-SDR) measured in decibels.

## 5. Comparative Results and Significance

Experimental results using the SI-SDR metric highlight the superior performance of the TCN-Conformer separator over the Conformer-FFN and pure-TCN baselines. For $K=3$ (2-mix-trained):

| Separator            | 2-mix SI-SDR | 3-mix SI-SDR | Noisy-mix SI-SDR |
|----------------------|--------------|--------------|------------------|
| Baseline TCN         | 16.15 dB     | 4.18 dB      | –2.30 dB         |
| Conformer-FFN        | 15.60 dB     | 4.08 dB      | –3.64 dB         |
| TCN-Conformer        | **16.85 dB** | **4.56 dB**  | **–0.24 dB**     |

When trained on the joint dataset:

| Separator            | 2-mix SI-SDR | 3-mix SI-SDR | Noisy-mix SI-SDR |
|----------------------|--------------|--------------|------------------|
| Baseline TCN         | 14.87 dB     | 8.43 dB      | 7.92 dB          |
| Conformer-FFN        | 14.07 dB     | 7.67 dB      | 7.56 dB          |
| TCN-Conformer        | **17.51 dB** | **10.70 dB** | **9.32 dB**      |

The TCN-Conformer yields approximately +2–3 dB improvement over both alternatives in all evaluated settings [2205.13851].

## 6. Block-Level Implementation and Pseudocode

The forward pass of the TCN-Conformer separator can be summarized as:

```python
function Separator_TCN_Conformer(X: [T×D], e: [E], K):
    U = Conv1x1( concat(X, tile(e, T)) )  # → [T×D]
    for k in 1..K do
        # TCN block with residual
        R = GLN(Conv1x1(U))
        R = PReLU(R)
        R = GLN(DepthwiseConv(R, kernel=3, dilation=d_k, padding=1))
        R = PReLU(R)
        R = Conv1x1(R)
        U_tcn = U + R

        # Conformer block
        C = ConformerBlock( concat(U_tcn, tile(e, T)) )         

        # Prepare for next iteration
        U = concat(C, tile(e, T))
    end for

    # Mask estimation and apply
    M = Sigmoid( Conv1x1(U) )
    Y_hat = Decoder( M * X )
    return Y_hat
end function
```

All network depths, kernel sizes, activation functions, and dropout rates precisely mirror the configuration reported in [2205.13851].

## 7. Architectural Implications and Research Context

These separator designs exemplify the integration of conformer architectures and speaker conditioning mechanisms in end-to-end TSE systems. The demonstrated efficacy of the TCN-Conformer underscores the merit of combining temporal convolutional receptive field expansion with self-attention and convolutional feature transformations. The use of multi-scale encoding, residual pre-norm layers, and concatenative speaker conditioning offers a robust template for future ASR protocols targeting challenging acoustic scenarios. Substituting speaker conditioning mechanisms (e.g., using FiLM) remains a viable direction for further study, as noted in [2205.13851].

Source: https://www.emergentmind.com/topics/kaldi-based-asr-protocol