--- title: 'UniEnc-CASSNAT: Efficient Non-Autoregressive ASR' url: https://www.emergentmind.com/topics/unienc-cassnat type: topic --- # UniEnc-CASSNAT: Efficient Non-Autoregressive ASR UniEnc-CASSNAT is an encoder-only non-autoregressive automatic speech recognition (ASR) framework designed to efficiently leverage speech self-supervised learning (SSL) models as foundational components. It integrates and extends the connectionist temporal classification (CTC) and CTC alignment-based single-step non-autoregressive transformer (CASS-NAT) paradigms, enabling efficient, dependency-aware sequence modeling with reduced parameter count and fast inference [2402.08898]. ## 1. Architectural Overview UniEnc-CASSNAT comprises three principal modules: - A front-end convolutional encoder mapping raw acoustic signals to a frame-level hidden sequence $H^0 \in \mathbb{R}^{T \times d}$, as in HuBERT’s configuration. - A contextual encoder $f_{\mathrm{ctx}}(\cdot)$ instantiating $L$ Transformer layers (e.g., $L=12$), optionally initialized from a speech SSL model such as HuBERT. - A token-level acoustic extractor ("TAE-E") employing a compact self-attention mechanism to pool $U$ token-level embeddings $E \in \mathbb{R}^{U \times d}$ based on frame-token alignments. UniEnc-CASSNAT discards the classic sequence decoder. Instead, the contextual encoder $f_{\mathrm{ctx}}$ is reused in two consecutive forward passes: 1. The first pass processes acoustic features alone: $O^0 = f_{\mathrm{ctx}}(H^0) \in \mathbb{R}^{T \times d}$. 2. Token-level embeddings are extracted from $O^0$ using TAE-E, then concatenated back to the frame sequence: $H^1 = \textrm{concat}(H^0, E) \in \mathbb{R}^{(T+U) \times d}$. 3. The second pass computes $O^1 = f_{\mathrm{ctx}}(H^1) \in \mathbb{R}^{(T+U) \times d}$. The final $U$ vectors correspond to token-level “decoder” outputs. The initial $T$ vectors provide optional auxiliary CTC supervision. This design enables a single encoder module to encode both acoustic and token contexts, reducing parameter count and facilitating transfer from SSL pretraining. ## 2. Mathematical Formalism Key notation: - $X = \{x_1, \ldots, x_T\}$: acoustic feature sequence after convolutional encoding ($X \in \mathbb{R}^{T \times d}$). - $Y = \{y_1, \ldots, y_U\}$: ground-truth output token sequence. - $W_{\mathrm{ctc}}, W_{\mathrm{dec}} \in \mathbb{R}^{|V| \times d}$: output-classification matrices for CTC and "decoder" projections. **First Pass (Encoder-side)** - $H^0 = X$ - $O^0 = f_{\mathrm{ctx}}(H^0)$ - Per-frame CTC distribution: $P_{\mathrm{ctc}}^1(\pi_t \mid O^0_t) = \textrm{Softmax}(W_{\mathrm{ctc}} O^0_t)$ - Frame-to-token alignment $Z = \{t_0 = 0 < t_1 \leq \cdots \leq t_U \leq T\}$, typically found via Viterbi decoding or sampling. - Token-level embedding extraction: $E = \mathrm{TAE\text{-}E}(O^0, Z)$, with $E_u$ formed by pooling (or attending) over $O^0_{t_{u-1}+1:t_u}$. **Second Pass (Joint Frame/Token Context)** - $H^1 = \textrm{concat}(H^0, E)$ - $O^1 = f_{\mathrm{ctx}}(H^1)$ - Frame outputs: $O^1_{1:T}$ (auxiliary CTC loss) - Token outputs: $O^1_{T+1:T+U}$ (final classification) - Decoder-side distribution: $P_{\mathrm{dec}}(y_u \mid X) = \textrm{Softmax}(W_{\mathrm{dec}} O^1_{T+u})$ - Auxiliary second-pass CTC: $P_{\mathrm{ctc}}^2(\pi_t \mid O^1_t) = \textrm{Softmax}(W_{\mathrm{ctc}} O^1_t)$ ## 3. Training Objectives The objective unifies multiple losses: - **First-pass CTC loss:** $L_{\mathrm{CTC}\text{-}1} = -\log \sum_{\pi \in B^{-1}(Y)} \prod_{t=1}^T P_{\mathrm{ctc}}^1(\pi_t \mid O^0_t)$ - **Second-pass CTC loss:** $L_{\mathrm{CTC}\text{-}2} = -\log \sum_{\pi \in B^{-1}(Y)} \prod_{t=1}^T P_{\mathrm{ctc}}^2(\pi_t \mid O^1_t)$ - **Decoder-side cross-entropy:** $L_{\mathrm{dec}} = -\sum_{u=1}^U \log P_{\mathrm{dec}}(y_u \mid X)$ The full training objective is $L = L_{\mathrm{dec}} + \lambda_1 L_{\mathrm{CTC}\text{-}1} + \lambda_2 L_{\mathrm{CTC}\text{-}2}$ with hyperparameters $\lambda_1, \lambda_2$ (empirically, $\lambda_1 = \lambda_2 = 1$ yields optimal WER). ## 4. Inference and Decoding Workflow UniEnc-CASSNAT achieves non-autoregressive, single-step generation of $U$ tokens while capturing token dependencies through iterative refinement called error-based sampled alignments (ESA): - **Iteration 0:** Run first pass, obtain $O^0$ and greedy CTC. Identify low-confidence frames ($\max P_{\mathrm{ctc}}^1 < \textrm{threshold}$). For each of $S_0$ samples, resample low-confidence labels, extract alternative alignments and TAEs. - **Iteration 1:** For each sampled alignment from iteration 0, run the second pass with concatenated features, compute $O^{1,(s)}$, and output token predictions. Compute updated low-confidence frame sets, sample $S_1$ additional alignments per first-pass sample; extract new TAEs; optionally further iterate. - **Candidate Ranking:** Score each candidate hypothesis $Y^{(s,k)}$ by summing log-probabilities from decoder cross-entropy and/or second-pass CTC losses; return the top-ranked sequence. Despite a single non-autoregressive step, token-token and token-frame interactions arise from self-attention in pass two and iterative TAE extraction. ## 5. Empirical Results UniEnc-CASSNAT was benchmarked on Librispeech (100h, 1024 word-pieces), MyST (240h, 500 word-pieces), and Aishell1 (170h, 4230 chars), always fine-tuning a HuBERT-base encoder (12 Transformer layers) and never using an external language model. Model sizes, word error rates (WER), and real-time factors (RTF) are summarized as follows: | Model | Params (M) | Librispeech WER (dev-clean/other) | MyST WER (dev/test) | Aishell1 (dev/test) | RTF | |--------------------|------------|------------------------------------|---------------------|---------------------|-------| | AT-w/o SSL | 85.1 | 6.6 / 18.2 | — | — | 0.325 | | AT-w/ SSL | 121.6 | 4.8 / 11.0 | 11.4/13.1 | 4.0/4.3 | 0.486 | | CTC | 95.7 | 6.1 / 13.8 | 12.9/14.5 | 4.5/4.9 | 0.005 | | CASS-NAT | 130.5 | 4.7 / 11.4 | 11.9/13.5 | 4.0/4.3 | 0.014 | | UniEnc-CASSNAT | 99.3 | 4.9 / 11.0 | 11.8/13.5 | 4.2/4.5 | 0.093 | UniEnc-CASSNAT matches or outperforms CASS-NAT while using approximately 30% fewer parameters. It approaches the performance of autoregressive SSL-initialized ASR systems, offering 3–5× faster inference. ## 6. Design Ablations and Analysis Extensive ablations elucidate several critical findings: - **Multi-pass CTC (MP-CTC):** SP-CTC (only $L_{\mathrm{CTC}\text{-}1} + L_{\mathrm{dec}}$) underperforms relative to CASS-NAT. Introducing the second pass ($\lambda_2 > 0$) without sampling does not close the gap. MP-CTC plus two-iteration decoding ($S_0 = 25$, $S_1 = 2$) surpasses CASS-NAT (WER reaches 4.9/11.0 on dev, 4.8/11.0 test). - **TAE-E module size:** Evaluating feed-forward dimensions $d_{256}, d_{512}, d_{768}$ in TAE-E shows marginal WER improvement from $d_{768}$ at a substantial parameter increase (+5M); $d_{512}$ represents an optimal trade-off. - **Sampling Strategy:** Splitting a fixed sample budget (e.g., 50) across two ESA refinement rounds (25×2) improves performance beyond single-round (50) sampling, supporting the benefit of iterative TAE extraction. ## 7. Significance and Implications UniEnc-CASSNAT establishes that a single Transformer encoder, repurposed across encoding and decoding via two forward passes, can learn token interdependencies comparably to encoder-decoder architectures but with considerable parameter and inference efficiency benefits. The multi-pass CTC loss, coupled with ESA-based iterative decoding, closes much of the typical accuracy gap to autoregressive SSL-initialized models while preserving substantial speed advantages. This approach demonstrates a scalable strategy for integrating speech foundation models within non-autoregressive ASR frameworks [2402.08898]. Source: https://www.emergentmind.com/topics/unienc-cassnat