---
title: Joint CTC–USDM Decoding Framework
url: https://www.emergentmind.com/topics/joint-ctc-usdm-decoding-framework
type: topic
---

# Joint CTC–USDM Decoding Framework

The Joint CTC–USDM decoding framework is a hybrid speech recognition decoding method that integrates Connectionist Temporal Classification (CTC) acoustic models and Uniform-State Diffusion Models (USDMs) to produce hypotheses benefiting from both strong framewise acoustic information and contextual sequence-level language modeling. This method constructs candidate hypotheses by fusing the CTC-derived framewise probability distributions with the labelwise probability outputs of USDM at each step of the reverse diffusion process, resulting in improved recognition accuracy relative to either subsystem alone [2604.14001].

## 1. Constituent Models: CTC and USDM

Connectionist Temporal Classification (CTC) defines, for every acoustic frame $t$, a probability distribution $P_{\mathrm{CTC},t}(v \mid X)$ over an alphabet $V$ augmented with the blank symbol $\varnothing$, where $X = x_{1:T}$ denotes the sequence of input acoustic features. The probability of a label sequence $y_{1:S}$ (post-collapse) is given by summing over all alignments $\pi_{1:T} \in (V \cup \{\varnothing\})^T$ such that $\mathrm{Collapse}(\pi) = y$:
\[
P_{\mathrm{CTC}}(y \mid X) = \sum_{\pi:\,\mathrm{Collapse}(\pi)=y} \prod_{t=1}^T P_{\mathrm{CTC},t}(\pi_t\mid X)
\]

The Uniform-State Diffusion Model (USDM) is a discrete diffusion-based language model. The forward corruption process at each step $t$ randomly replaces each token $w$ with a token drawn from the uniform distribution $\pi = \tfrac{1}{|V|}\mathbf{1}$ with probability $1-\alpha_t$, and retains it with probability $\alpha_t$. The transition distribution is
\[
q(z_t \mid w) = \mathrm{Cat}\left(z_t;\,\alpha_t\,\mathbf{1}_w+(1-\alpha_t)\,\boldsymbol{\pi}\right)
\]
During the denoising reverse process, a transformer $f_\theta$ predicts, for each position $i$ in the sequence $z_t$, a categorical distribution over $V$:
\[
P_\theta\bigl(v_j\mid z_t,\,t\bigr)
\]

## 2. Construction of the Joint Scoring Function

The framework aligns the CTC encoder’s predictions to token positions and fuses these distributions with the per-position USDM predictions during the denoising process, as follows.

- A greedy CTC pass produces a collapsed label sequence $\hat y = \mathrm{Collapse}\bigl(\arg\max_\pi \prod_t P_{\mathrm{CTC},t}(\pi_t \mid X)\bigr)$.
- Each token $\hat y_i$ is assigned its first corresponding frame index $\tau_i$ from the CTC alignment path.
- The relevant CTC framewise distribution (renormalized) is $P_{\mathrm{CTC},\tau_i}(v_j\mid X)/\sum_{v\in V}P_{\mathrm{CTC},\tau_i}(v\mid X)$ for $v_j \in V$.

At each USDM denoising step $\ell$ with noised input $z_{t_\ell}$, the joint log-probability for position $i$ is formed as a weighted sum:
\[
\log P_{\mathrm{comb},\,i}(v_j) = \lambda_{\mathrm{CTC}}\,\log P_{\mathrm{CTC},\tau_i}(v_j\mid X) + \lambda_{\mathrm{DiffLM}}\,\log P_{\theta,i}(v_j\mid z_{t_\ell})
\]
The interpolation weights $\lambda_{\mathrm{CTC}}$ and $\lambda_{\mathrm{DiffLM}} = 1 - \lambda_{\mathrm{CTC}}$ balance the acoustic and language model contributions. This fused distribution is used to resample each sequence element for the next denoising step:
\[
z_{t_{\ell-1},i}\sim\mathrm{Cat}\bigl(z;\,P_{\mathrm{comb},\,i}(\cdot)\bigr)
\]

## 3. Decoding Procedure

The complete decoding method initializes from a greedy CTC sequence at a specified noise level and iteratively denoises using the hybrid probability fusion until a clean sequence is produced. The loop for each denoising step synthesizes both acoustic and contextual probabilities at the token level.

```pseudo
Algorithm: Joint CTC–USDM Decoding

Input: X, trained CTC and USDM, noise schedule {t_0 > ... > t_L=0}, λ_CTC, λ_DiffLM, ℓ_start
Output: decoded token sequence y^*

1.  π* ← argmax_π ∏ P_CTC,t(π_t | X)
2.  ŷ ← Collapse(π*)
3.  For each i, τ_i ← CTC alignment for ŷ_i
4.  z_{t_{ℓ_start}} ← ŷ
5.  For ℓ from ℓ_start down to 1:
6.    P_θ(·|z_{t_ℓ}) ← f_θ(z_{t_ℓ}, t_ℓ)
7.    For i, j: log P_comb,i(v_j) ← λ_CTC · log P_CTC,τ_i(v_j|X) + λ_DiffLM · log P_θ,i(v_j|z_{t_ℓ})
8.    Normalize P_comb,i
9.    Sample z_{t_{ℓ-1}, i} ∼ Cat(P_comb,i)
10. y^* ← z_{t_0}
return y^*
```
Key implementation details include ancestral sampling at each denoising step, and the option to generate multiple chains for selection based on the final CTC score or to approximate beam search effects.

## 4. Hyperparameters and Tuning Strategies

Optimal performance is highly dependent on three main hyperparameters:
- $\lambda_{\mathrm{CTC}}$, $\lambda_{\mathrm{DiffLM}}$ (acoustic-language balance): Best results were achieved with $\lambda_{\mathrm{DiffLM}}=0.3$ ($\lambda_{\mathrm{CTC}}=0.7$).
- $\ell_{\mathrm{start}}$ (initial noise index): Controls initial corruption. A $t_{\mathrm{start}}=0.3$ value accelerates convergence without sacrificing accuracy.
- Number of denoising steps $K$ (equivalently $L$): Experimentation covered $K\in\{8,16,32,48,64\}$, with diminishing returns beyond $K\approx48$.

The recommended tuning procedure involves:
1. Fixing the USDM checkpoint;
2. Grid search over $\lambda_{\mathrm{DiffLM}}$ (dev set, $K\approx32$);
3. Sweeping $t_{\mathrm{start}}$;
4. Varying $K$ for speed-accuracy trade-off.

## 5. Computational Complexity and Practical Optimizations

The per-step complexity is $O(S|V|)$ for the computation and normalization of joint distributions, with $O(S)$ additional for sampling. The total decoding cost is $O(KS|V|)$. Typical problems feature $S\ll T$ (sequence length shorter than frame count) but large vocabularies ($|V|$ in the range of 10,000).

Optimization techniques include:
- Restricting candidate tokens per position to the top-$N$ ($N=50$) from the CTC distribution,
- Caching static per-position CTC probabilities,
- Employing half-precision (FP16) inference for the USDM transformer,
- Early termination if the sequence remains unchanged over several denoising steps.

## 6. Empirical Outcomes and Comparative Analysis

Evaluation on Librispeech dev-other gives the following word error rates (WER):

| System                    | WER (%)          |
|---------------------------|------------------|
| CTC greedy (no LM)        | 5.08             |
| USDM rescoring (K=256)    | 4.82             |
| Joint CTC+USDM decoding   | 4.71–4.77        |
| MDLM rescoring            | 4.52             |
| AR LM joint decoding      | 3.86             |

For joint CTC–USDM decoding with $\lambda_{\mathrm{DiffLM}}=0.3$, $t_{\mathrm{start}}=0.3$ and with training extended to 25 epochs ($K=64$), the best obtained WER was 4.71%. The absolute reduction in WER relative to CTC-only and USDM-rescoring baselines is 0.31%.

This fusion approach consistently surpasses static USDM rescoring. Per-step joint fusion enables the decoder to recover from certain CTC errors. Most performance gain is captured with 32–48 denoising steps; longer chains yield little additional benefit. Although state-of-the-art autoregressive LM joint decoding still achieves lower WER, the performance gap narrows as USDM is trained longer and leveraged in joint decoding. The findings confirm that stepwise fusion of CTC and USDM probabilistic information yields superior recognition hypotheses [2604.14001].

Source: https://www.emergentmind.com/topics/joint-ctc-usdm-decoding-framework