Transformer-Transducers for Code-Switched ASR

• The paper introduces a Transformer-Transducer framework that integrates CTC and LM losses to robustly handle code-switching by regularizing both audio and label encoders.
• It employs label masking and LID tags to boost performance in intra-sentential language switches, achieving absolute MER gains on the SEAME dataset.
• The multi-label and multi-audio encoder design effectively fuses monolingual and code-switched data, demonstrating significant accuracy improvements over prior RNN-T baselines.

Transformer-Transducers for code-switched speech recognition constitute a class of end-to-end automatic speech recognition (ASR) systems designed to handle utterances containing multiple languages, including within a single sentence. These systems aim to address the pervasive phenomenon of code-switching—where bilingual or multilingual speakers alternate languages at utterance or intra-sentential boundaries—by deploying Transformer-based architectures in the Transducer framework, leveraging both code-switched and large-scale monolingual corpora while employing tailored training objectives and label manipulations to improve robustness and accuracy in low-resource scenarios (Dalmia et al., 2020).

1. Transformer-Transducer Framework for Code-Switched Speech

The core of the system is a Transformer-Transducer model constructed to model the posterior $P(y|x)$ of token sequences $y=(y_1,\ldots,y_L)$ given audio feature sequences $x=(x_1,\ldots,x_T)$, where $y_l \in \mathcal{V}$ and $x_t \in \mathbb{R}^d$. The model sums over all alignments $z$ that interleave emitted output tokens and “emit‐blank” symbols ($\phi$):

$$P(y \mid x) = \sum_{z \in \mathcal{Z}(x,y)} \prod_{i=1}^{T+L} P \bigl(z_i \mid x, y_{1:u(i)-1} \bigr)$$

where $u(i)$ represents the number of non-blank tokens in $z_1,\ldots,z_{i-1}$.

The model factorizes this probability through three specialized subnetworks:

• Audio encoder (Enc): A front-end stack consisting of strided convolutional subsampling followed by 12 Transformer blocks, mapping $y=(y_1,\ldots,y_L)$0 to $y=(y_1,\ldots,y_L)$1.
• Label encoder (Predictor, Pred): A 4-layer Transformer stack processing previous predicted tokens (with <sos> prefix), producing $y=(y_1,\ldots,y_L)$2 for each output position.
• Joint network: Combines $y=(y_1,\ldots,y_L)$3 and $y=(y_1,\ldots,y_L)$4 via a feedforward tanh-activated network:

$y=(y_1,\ldots,y_L)$5

This is transformed by a softmax classifier over $y=(y_1,\ldots,y_L)$6, producing emission probabilities.

The primary training objective is the negative log-likelihood of the correct token sequence under this transducer marginalization.

2. Auxiliary Objectives for Data-Sparse Code-Switching

To counteract the low-resource nature of code-switched (CS) data, two auxiliary losses are introduced:

• Connectionist Temporal Classification (CTC) Loss on the audio encoder:

$y=(y_1,\ldots,y_L)$7

This term assists frame-wise learning and regularization of the audio encoder.

• LLM (LM) Loss on the label encoder:

$y=(y_1,\ldots,y_L)$8

This next-token cross-entropy loss encourages the predictor network to better model the target distribution.

These are integrated with tuned weights ($y=(y_1,\ldots,y_L)$9, $x=(x_1,\ldots,x_T)$0) into the final objective:

$x=(x_1,\ldots,x_T)$1

Empirical evidence on the SEAME corpus demonstrates a consistent $x=(x_1,\ldots,x_T)$20.7% absolute Mixed Error Rate (MER) gain from the CTC loss; the LM loss alone may slightly degrade MER but accelerates convergence and tolerates a doubled learning rate (Dalmia et al., 2020).

3. Label Masking and LID-Aware Training for Intra-Sentential Code-Switching

To equip the label encoder for intra-sentential language shifts and disfluencies, two key label sequence transformations are introduced prior to passage through the predictor stack:

A. Random Masking: Each ground-truth token $x=(x_1,\ldots,x_T)$3 is independently masked (with probability $x=(x_1,\ldots,x_T)$4), replacing $x=(x_1,\ldots,x_T)$5 with <mask>:

$x=(x_1,\ldots,x_T)$6

B. Language-ID Tags: Language segmentation is performed using gold transcripts, inserting <en> or <man> tokens at detected boundaries where $x=(x_1,\ldots,x_T)$7. The final input to the predictor is:

$x=(x_1,\ldots,x_T)$8

This dual modification improves the ability of the network to attend to both acoustic evidence and language context, empirically yielding roughly 1.6% absolute dev MER gain when both masked training and LID tags are employed (Dalmia et al., 2020).

4. Leveraging Monolingual Corpora with Multi-Label and Multi-Audio Encoders

Because acquisition of large-scale CS corpora is costly, the system incorporates a multi-label/multi-audio encoder structure to combine abundant monolingual resources with CS data:

• Two independent audio encoders, $x=(x_1,\ldots,x_T)$9 and $y_l \in \mathcal{V}$0 (and analogously for the predictor), maintain separate parameterizations for Mandarin and English.
• For each minibatch (CS or monolingual), both $y_l \in \mathcal{V}$1 are computed and then fused by a learned gating mechanism:

$y_l \in \mathcal{V}$2

$y_l \in \mathcal{V}$3

• For purely monolingual utterances, $y_l \in \mathcal{V}$4 is clamped (1 for Mandarin, 0 for English), ensuring modality-specific updates; for CS utterances, $y_l \in \mathcal{V}$5 remains learnable.

End-to-end training proceeds on pooled CS and monolingual batches, with refinement using only CS batches in later training iterations.

5. Experimental Setup and Performance

Experiments are conducted on the SEAME Mandarin-English CS corpus (∼100h), with additional monolingual data from AISHELL-1 (150h, Mandarin) and TEDLIUMv2 (211h, English). Feature extraction employs 83-dimensional log-Mel plus pitch at 16 kHz, with global CMVN, SpecAugment (SS policy), and 3-way speed perturbation. The audio encoder is composed of two convolutional subsampling layers followed by 12 Transformer blocks (512-dim attention, 1k FFN, 8 heads). The label encoder is a 4-layer Transformer stack with elevated dropout.

Vocabulary consists of 2.5k Mandarin characters and English subwords (2k BPE merges for mixed, 4k for monolingual, to equate to ∼4k tokens per language). Mixed Error Rate (MER)—the sum of character errors (Mandarin) and word errors (English)—serves as the principal metric.

Final results on SEAME are summarized as follows:

Model	test_man	test_sge
RNN-Transducer (Zhang ’20)	33.3	44.9
Att-EncDec (Zhou ’20)	18.9	26.2
Transformer-Transducer	20.2	27.7
+ Monolingual (multi-label)	18.5	26.3

Ablation studies (on dev MER) provide the following insights:

• Vanilla T-T: 25.6%
• + CTC: 24.9%
• + CTC + LM: 25.1% (faster convergence)
• + Masked training: 24.0%
• + Masked + LID tags: 23.4%
• + Multi-audio encoder: 23.1%
• + Multi-label encoder: 22.2%

Key ablation findings establish CTC as a reliable regularizer, masking plus LID tags as critical for intra-sentential CS, and multi-label encoding as the most effective strategy for integrating monolingual data in this 200h/resource regime (Dalmia et al., 2020).

6. Discussion and Future Work

The Transformer-Transducer architecture surpasses prior RNN-T baselines by 13–17% absolute MER on SEAME, without the aid of external LLMs or hand-crafted phone merging. Auxiliary objectives (CTC, LM) and LID-aware masking provide steady gains on data-sparse CS tasks; the multi-label encoder leverages monolingual data within a unified model.

Limitations include: restricted gain from audio-side gating at moderate resource scale (∼200h per language), dependence on gold LID segmentation and transcript-based masking during training, and absence of explicit pronunciation modeling—mispronunciations arising from accent drift in CS speech remain inadequately addressed.

Future directions involve:

• Unsupervised or weakly supervised pre-training (e.g., wav2vec2.0) to exploit unlabeled audio.
• Joint modeling of acoustic and pronunciation adaptation to better address phone-shifting in CS utterances.
• Real-time, online extensions with integrated lightweight LID detectors.
• Curriculum learning and domain-invariant gating for large-scale monolingual integration (Dalmia et al., 2020).