---
title: Transformer-Transducers for Code-Switched ASR
url: https://www.emergentmind.com/topics/transformer-transducers-for-code-switched-speech-recognition
type: topic
---

# Transformer-Transducers for Code-Switched ASR

Transformer-Transducers for code-switched speech recognition constitute a class of end-to-end automatic speech recognition (ASR) systems designed to handle utterances containing multiple languages, including within a single sentence. These systems aim to address the pervasive phenomenon of code-switching—where bilingual or multilingual speakers alternate languages at utterance or intra-sentential boundaries—by deploying Transformer-based architectures in the Transducer framework, leveraging both code-switched and large-scale monolingual corpora while employing tailored training objectives and label manipulations to improve robustness and accuracy in low-resource scenarios [2011.15023].

## 1. Transformer-Transducer Framework for Code-Switched Speech

The core of the system is a Transformer-Transducer model constructed to model the posterior $P(y|x)$ of token sequences $y=(y_1,\ldots,y_L)$ given audio feature sequences $x=(x_1,\ldots,x_T)$, where $y_l \in \mathcal{V}$ and $x_t \in \mathbb{R}^d$. The model sums over all alignments $z$ that interleave emitted output tokens and “emit‐blank” symbols ($\phi$):

$$
P(y \mid x) = \sum_{z \in \mathcal{Z}(x,y)} \prod_{i=1}^{T+L} P \bigl(z_i \mid x, y_{1:u(i)-1} \bigr)
$$

where $u(i)$ represents the number of non-blank tokens in $z_1,\ldots,z_{i-1}$.

The model factorizes this probability through three specialized subnetworks:

- **Audio encoder (Enc):** A front-end stack consisting of strided convolutional subsampling followed by 12 Transformer blocks, mapping $x_{1:T}$ to $h^{\mathrm{enc}}_{1:T}$.
- **Label encoder (Predictor, Pred):** A 4-layer Transformer stack processing previous predicted tokens (with <sos> prefix), producing $h^{\mathrm{pred}}_u$ for each output position.
- **Joint network:** Combines $h^{\mathrm{enc}}_t$ and $h^{\mathrm{pred}}_u$ via a feedforward tanh-activated network:
  $$
  h^\mathrm{joint}_{t,u} = \tanh(W_\mathrm{enc} h^{\mathrm{enc}}_t + W_\mathrm{pred} h^{\mathrm{pred}}_u + b)
  $$
  This is transformed by a softmax classifier over $\mathcal{V} \cup \{\phi\}$, producing emission probabilities.

The primary training objective is the negative log-likelihood of the correct token sequence under this transducer marginalization.

## 2. Auxiliary Objectives for Data-Sparse Code-Switching

To counteract the low-resource nature of code-switched (CS) data, two auxiliary losses are introduced:

- **Connectionist Temporal Classification (CTC) Loss** on the audio encoder:
  $$
  \mathcal{F}_{\mathrm{CTC}}(x, y) = -\ln \sum_{\pi \in \mathrm{CTCAlign}(y)} \prod_{t=1}^T P_{\mathrm{CTC}}(\pi_t \mid x)
  $$
  This term assists frame-wise learning and regularization of the audio encoder.

- **Language Model (LM) Loss** on the label encoder:
  $$
  \mathcal{F}_{\mathrm{LM}}(y) = -\sum_{l=1}^L \ln P_{\mathrm{LM}}(y_l \mid y_{1:l-1})
  $$
  This next-token cross-entropy loss encourages the predictor network to better model the target distribution.

These are integrated with tuned weights ($\lambda_\mathrm{CTC}=0.5$, $\lambda_\mathrm{LM}=0.4$) into the final objective:

$$
\mathcal{F}_\mathrm{obj} = \mathcal{F}_\mathrm{Transducer}(x, y) + \lambda_\mathrm{CTC} \, \mathcal{F}_\mathrm{CTC} + \lambda_\mathrm{LM} \, \mathcal{F}_\mathrm{LM}
$$

Empirical evidence on the SEAME corpus demonstrates a consistent $\sim$0.7% absolute Mixed Error Rate (MER) gain from the CTC loss; the LM loss alone may slightly degrade MER but accelerates convergence and tolerates a doubled learning rate [2011.15023].

## 3. Label Masking and LID-Aware Training for Intra-Sentential Code-Switching

To equip the label encoder for intra-sentential language shifts and disfluencies, two key label sequence transformations are introduced prior to passage through the predictor stack:

**A. Random Masking:** Each ground-truth token $y_l$ is independently masked (with probability $p=0.4$), replacing $y_l$ with <mask>:

$$
\tilde{y}_l = \begin{cases}
\texttt{<mask>}, & \text{w.p. } 0.4 \\
y_l, & \text{otherwise}
\end{cases}
$$

**B. Language-ID Tags:** Language segmentation is performed using gold transcripts, inserting <en> or <man> tokens at detected boundaries where $\mathrm{lang}(y_l) \neq \mathrm{lang}(y_{l+1})$. The final input to the predictor is:

$$
(\texttt{<sos>}, \text{LID tags}, \tilde{y}_1, \ldots, \tilde{y}_L, \texttt{<eos>})
$$

This dual modification improves the ability of the network to attend to both acoustic evidence and language context, empirically yielding roughly 1.6% absolute dev MER gain when both masked training and LID tags are employed [2011.15023].

## 4. Leveraging Monolingual Corpora with Multi-Label and Multi-Audio Encoders

Because acquisition of large-scale CS corpora is costly, the system incorporates a multi-label/multi-audio encoder structure to combine abundant monolingual resources with CS data:

- Two independent audio encoders, $\mathrm{Enc}^\mathrm{man}$ and $\mathrm{Enc}^\mathrm{en}$ (and analogously for the predictor), maintain separate parameterizations for Mandarin and English.
- For each minibatch (CS or monolingual), both $(h_t^\mathrm{man}, h_t^\mathrm{en})$ are computed and then fused by a learned gating mechanism:
  $$
  \alpha_\mathrm{enc} = \sigma\left( w_\alpha^{T} \tanh \left( W_\mathrm{man} h_t^\mathrm{man} + W_\mathrm{en} h_t^\mathrm{en} \right)\right)
  $$
  $$
  h_t = \alpha_\mathrm{enc} h_t^\mathrm{man} + (1-\alpha_\mathrm{enc}) h_t^\mathrm{en}
  $$
- For purely monolingual utterances, $\alpha_\mathrm{enc}$ is clamped (1 for Mandarin, 0 for English), ensuring modality-specific updates; for CS utterances, $\alpha_\mathrm{enc}$ remains learnable.

End-to-end training proceeds on pooled CS and monolingual batches, with refinement using only CS batches in later training iterations.

## 5. Experimental Setup and Performance

Experiments are conducted on the SEAME Mandarin-English CS corpus (∼100h), with additional monolingual data from AISHELL-1 (150h, Mandarin) and TEDLIUMv2 (211h, English). Feature extraction employs 83-dimensional log-Mel plus pitch at 16 kHz, with global CMVN, SpecAugment (SS policy), and 3-way speed perturbation. The audio encoder is composed of two convolutional subsampling layers followed by 12 Transformer blocks (512-dim attention, 1k FFN, 8 heads). The label encoder is a 4-layer Transformer stack with elevated dropout.

Vocabulary consists of 2.5k Mandarin characters and English subwords (2k BPE merges for mixed, 4k for monolingual, to equate to ∼4k tokens per language). Mixed Error Rate (MER)—the sum of character errors (Mandarin) and word errors (English)—serves as the principal metric.

Final results on SEAME are summarized as follows:

| Model                            | test_man | test_sge |
|-----------------------------------|----------|----------|
| RNN-Transducer (Zhang ’20)        | 33.3     | 44.9     |
| Att-EncDec (Zhou ’20)             | 18.9     | 26.2     |
| Transformer-Transducer            | 20.2     | 27.7     |
| + Monolingual (multi-label)       | 18.5     | 26.3     |

Ablation studies (on dev MER) provide the following insights:

- **Vanilla T-T:** 25.6%
- **+ CTC:** 24.9%
- **+ CTC + LM:** 25.1% (faster convergence)
- **+ Masked training:** 24.0%
- **+ Masked + LID tags:** 23.4%
- **+ Multi-audio encoder:** 23.1%
- **+ Multi-label encoder:** 22.2%

Key ablation findings establish CTC as a reliable regularizer, masking plus LID tags as critical for intra-sentential CS, and multi-label encoding as the most effective strategy for integrating monolingual data in this 200h/resource regime [2011.15023].

## 6. Discussion and Future Work

The Transformer-Transducer architecture surpasses prior RNN-T baselines by 13–17% absolute MER on SEAME, without the aid of external language models or hand-crafted phone merging. Auxiliary objectives (CTC, LM) and LID-aware masking provide steady gains on data-sparse CS tasks; the multi-label encoder leverages monolingual data within a unified model.

Limitations include: restricted gain from audio-side gating at moderate resource scale (∼200h per language), dependence on gold LID segmentation and transcript-based masking during training, and absence of explicit pronunciation modeling—mispronunciations arising from accent drift in CS speech remain inadequately addressed.

Future directions involve:

- Unsupervised or weakly supervised pre-training (e.g., wav2vec2.0) to exploit unlabeled audio.
- Joint modeling of acoustic and pronunciation adaptation to better address phone-shifting in CS utterances.
- Real-time, online extensions with integrated lightweight LID detectors.
- Curriculum learning and domain-invariant gating for large-scale monolingual integration [2011.15023].

Source: https://www.emergentmind.com/topics/transformer-transducers-for-code-switched-speech-recognition