Masked Audio Text Encoder

• Masked Audio Text Encoder is a neural architecture that applies masked modeling to both audio and text streams, producing semantically rich representations.
• It fuses acoustic features and textual data using techniques like frame masking, reconstruction loss, and joint attention to enhance ASR and captioning performance.
• Empirical results demonstrate significant error rate reductions and improved robustness in low-resource and domain-transfer settings on benchmarks like WSJ and LibriSpeech.

A masked audio text encoder is a neural architecture that leverages masked modeling objectives, originally introduced for text (e.g., BERT), and extends them to settings where both audio and text representations are integrated. Across speech recognition, multi-modal rescoring, and audio captioning, such encoders utilize masking and reconstruction or masked prediction tasks—possibly alongside contrastive or cross-modal objectives—to acquire context-aware, robust, and semantically rich representations from both labeled and massive unlabeled data. Key implementations include the Masked Pre-trained Encoder (MPE) with Joint CTC-Transformer (JCT) for semi-supervised ASR (Liu et al., 2020), Masked Audio Text Encoder (MATE) for multi-modal ASR rescoring (Cai et al., 2023), and masked audio–text encoding within EnCLAP for audio captioning (Kim et al., 2024). These approaches demonstrate significant improvements in downstream tasks, especially in low-resource and domain-transfer scenarios.

1. Architectural Paradigms

Across implementations, masked audio text encoders combine neural self-supervised modeling, masking strategies, and fusion of acoustic and textual streams.

• MPE+JCT (ASR, (Liu et al., 2020)):

The architecture consists of a Masked Pre-trained Encoder (MPE) that receives log-Mel filterbank inputs enriched with deltas, passes them through convolutional layers for temporal downsampling, and projects these to high-dimensional embeddings. A deep bidirectional Transformer stack (12 layers, $d_\mathrm{model}=512$) with stochastic BERT-inspired frame masking processes the sequence. Downstream, the encoder is incorporated into a Joint CTC-Transformer (JCT) attention-based model with both CTC and attention losses, allowing for fine-tuning under low-resource labeling.

• MATE (Multi-Modal Rescoring, (Cai et al., 2023)):

Three main components: a pre-trained speech encoder (WavLM-Base+), a cross-modal adaptation module (CNN subsampling + adapter), and a BERT-Base MLM. Frame-level audio representations are mapped to BERT space and concatenated with tokenized text, forming a single sequence processed by the Transformer layers for cross-modal interaction. This architecture fuses audio and text along the token dimension, enabling direct attention-based integration.

• EnCLAP (Audio Captioning, (Kim et al., 2024)):

Audio is processed by two pretrained models: EnCodec (to produce discrete neural codec tokens) and CLAP (to provide a global semantic embedding), both mapped into BART’s embedding space. Discrete codec tokens and the global CLAP embedding are input to the BART encoder with masking applied to spans of the codec sequence. The BART decoder generates captions autoregressively, conditioned on these multi-granular audio representations.

2. Masking and Training Objectives

Masking mechanisms are central for context integration and regularization.

• Frame/Token Masking:

In MPE, 15% of time frames are stochastically masked (zeroed, replaced by another frame, or left unchanged in an 80/10/10 schedule). In MATE, 15% of text tokens are masked as in standard BERT MLM. EnCLAP randomly masks 15% of codec token positions in contiguous spans of length 10.

• Reconstruction and Prediction Losses:
  • MPE: L1 reconstruction loss is applied only on masked frames to force integration of past/future context:

  $$L_\mathrm{MPE} = \sum_{t \in \text{masked}} \bigl\|h_t - e_t\bigr\|_1$$ - MATE: Standard MLM cross-entropy loss over masked text tokens, conditioned on both observed text and audio, plus an InfoNCE contrastive loss to align utterance-level audio and text representations. The joint objective is:

  $$\mathcal{L} = \mathcal{L}_\mathrm{MLM} + \alpha \mathcal{L}_\mathrm{CTR}$$

  with $\alpha = 1.0$. - EnCLAP: Cross-entropy on masked codec tokens (with head per codebook, weighted by codebook rank), plus standard captioning loss and a weight $\lambda$ governing the masked codec loss contribution:

  $$L_\mathrm{total} = L_\mathrm{caption} + \lambda L_\mathrm{mcm}$$

3. Fusion of Audio and Text Modalities

Masked audio text encoders enable explicit audio-text integration at the representation and optimization levels.

• Concatenation and Self-Attention:

In MATE, audio representations from a speech encoder (via CNN + adapter) and text embeddings are concatenated, allowing unified self-attention for cross-modal modeling. No explicit cross-attention or gating is required (Cai et al., 2023).

• Joint Embedding Prefix:

EnCLAP prepends a global CLAP-derived embedding to the sequence of codec-token embeddings, giving BART’s encoder access to both local and global audio semantics. The caption decoder can thus condition on both fine-grained and global acoustic representations (Kim et al., 2024).

• Attention-Based Sequence Modeling:

In MPE+JCT, the same encoder is shared by attention-based and CTC branches. Input features are exclusively acoustic, but CTC and attention heads encourage both monotonic and flexible alignment, benefiting audio-text alignment in decoding (Liu et al., 2020).

4. Training Protocols and Hyperparameters

Optimization details and data regimens strongly influence the effectiveness of masked audio text encoders.

Model	Data Regimen	Key Hyperparameters
MPE+JCT	Unlabeled: 81h–960h	Adam, lr=1e-4, 12-layer encoder, mask ratio 15%
MATE	10,000h real+synthetic	Adam, peak lr=5e-5, 32 batch, full finetuning, mask 15%
EnCLAP	AudioCaps/Clotho	AdamW, lr=2.5e-5–6.5e-5, 15 epoch, mask 15%, span=10, $\lambda=0.7$

MATE and EnCLAP rely on large pretraining/fine-tuning datasets and full end-to-end updates for best performance. Parameter-efficient tuning yields inferior results in MATE. Masking schedules consistently use 15% masking but span length (EnCLAP) and masking target (text/audio) vary.

5. Empirical Results and Comparative Performance

Masked audio text encoders achieve state-of-the-art performance or significant improvements over baselines in diverse settings.

• MPE+JCT: On WSJ eval92, two-stage training with MPE (using only 30% of labeled data) delivers a 17% WER reduction over a fully supervised JCT with 50% labeled data. Scaling unlabeled data from 81h (WSJ) to 960h (LibriSpeech) yields ≈22% further WER reduction, with 3.48% WER when fine-tuned on all labels and pre-trained on LibriSpeech, a 36% reduction relative to the supervised baseline. On LibriSpeech test-clean, joint training with 100h labeled achieves WER 9.68% (vs baseline 12.17%), and with 360h: 7.83% (vs 9.70%) (Liu et al., 2020).
• MATE: Achieves 4–16% relative WER reduction in-domain, and 3–7% out-of-domain (zero-shot), compared to text-only BERT. With only 0.8h of in-domain data, an 8–23% WER reduction is obtained over the first-pass baseline. Ablative analysis shows contrastive loss is essential: removing it causes performance to revert to text-only baselines (Cai et al., 2023).
• EnCLAP: On AudioCaps, EnCLAP-large achieves METEOR=0.255, CIDEr=0.803, SPIDEr=0.495. On Clotho, METEOR=0.182, CIDEr=0.426, SPIDEr=0.278. Removing masked codec modeling (setting $\lambda=0$) drops CIDEr by 0.021 on Clotho, confirming the contribution of the masking objective for audio-aware captioning (Kim et al., 2024).

6. Limitations and Prospective Directions

Notable limitations across implementations include:

• MPE+JCT: Framewise masking is agnostic to phonetic/semantic boundaries, and objectives focus on reconstruction of low-level representations, not linguistic units. Downsampling must balance context integration with loss of fine alignment (Liu et al., 2020).
• MATE: Performance is sensitive to the strength of cross-modal alignment; end-to-end fine-tuning outperforms adapter-based schemes. The approach is agnostic to the first-pass ASR, but efficacy depends on high-quality audio representations and ample paired data (Cai et al., 2023).
• EnCLAP: Masking hyperparameters (ratio, span length) may require dataset/domain tuning. Overfitting is a risk on smaller captioning datasets for larger models. Continuous features alone are outperformed by discrete codec tokens combined with joint embeddings (Kim et al., 2024).

A plausible implication is that further advances may involve adaptive/phonetic span masking, auxiliary contrastive or discriminative objectives (as in CPC), and tighter integration of global and local audio representations. The masked audio–text paradigm is applicable beyond ASR and captioning, including music processing and multimodal understanding.

7. Related Techniques and Comparative Analysis

Across the literature, masked audio text encoders are compared to established self-supervised and multi-modal learning approaches:

• wav2vec (Ye et al., 2017), DeCoAR (Liu et al., 2019):

MPE+JCT outperforms both wav2vec and DeCoAR on dev93 in WSJ but slightly underperforms DeCoAR on WSJ eval92 (Liu et al., 2020).

• Contrastive Audio–Text Models (CLAP, etc.):

EnCLAP demonstrates that adding a masked codec modeling head to joint contrastive CLAP embeddings yields additional gains, compared to using continuous features or text-only BART.

• Hybrid CTC/Attention:

The joint CTC-attention objective in MPE+JCT, with a recommended mix ratio $\alpha\approx0.3$, combines monotonic alignment (CTC) with flexible sequence modeling (attention), yielding improved robustness and accuracy (Liu et al., 2020).

These comparisons emphasize that masked modeling over acoustic and joint audio–text spaces constitutes a unifying principle for robust, data-efficient neural sequence modeling across spoken, transcribed, and captioned audio modalities.