---
title: Text Denoising Adapter in ASR
url: https://www.emergentmind.com/topics/text-denoising-adapter-tda
type: topic
---

# Text Denoising Adapter in ASR

The Text Denoising Adapter (TDA) is a lightweight method for text-only domain adaptation in automatic speech recognition (ASR) systems that use large language models (LLMs). TDA leverages synthetic text noise to emulate the distribution shift introduced by the speech projector in multimodal ASR. By training the LLM to recover clean transcripts from noisy text inputs that mimic audio-projected representations, TDA preserves cross-modal alignment and improves domain transfer, all without additional architectural changes or parameters [2601.20900].

## 1. Problem Formulation and Limitations of Prior Approaches

In LLM-based ASR architectures, a pretrained speech encoder feeds acoustic representations through a learnable speech projector into the LLM’s embedding space. The projector’s output, which can be regarded as “noisy text,” must be denoised by the LLM to yield accurate transcriptions. When only target-domain text is available, standard text-only fine-tuning of the LLM disrupts the projector-to-LLM alignment, leading to degraded ASR performance as the LLM forgets how to interpret the outputs of the speech projector.

Prior text-only adaptation methods attempted to preserve this alignment using early stopping based on perplexity monitoring (Fang et al.) or soft prompt embeddings (Ma et al.), both introducing additional monitoring and/or trainable modules with new hyperparameters. Such techniques require careful calibration and introduce complexity [2601.20900].

## 2. Text Denoising as Proxy for Audio-Induced Noise

TDA reframes text-only adaptation as a denoising problem, mimicking the corruption introduced in the audio-to-text mapping. Let $x$ be a clean transcript. The method defines a noisy-input generation function $N(\cdot)$ so that $\tilde x = N(x)$ simulates the type of corruption produced by the speech projector and discretization.

The adaptation objective maximizes the log-likelihood of recovering $x$ from $\tilde x$ under the LLM’s parameters $\theta$:
$$
L_{\mathrm{denoise}}(\theta) = -\sum_{t=1}^{|x|} \log p_\theta(x_t \mid \tilde x)
$$
The synthetic corruption $N(\cdot)$ consists of random character substitution—selecting 15% of words, replacing 30% of their characters—and random character duplication, with a 10% chance for remaining characters to be repeated 1–3 times. This process reproduces the modalities’ misalignment as seen in multimodal ASR pipelines [2601.20900].

## 3. Alignment Preservation Through Synthetic Noise

The central analogy of TDA is to treat $N(x)$ as a proxy for the output of the speech projector: in standard multimodal ASR, audio $a$ is encoded as embeddings $sp(a)$, which behave as noisy text-like sequences. The LLM, during standard ASR training, is conditioned to reconstruct clean transcripts from these noisy sequences. By fine-tuning the LLM on pairs $(N(x), x)$, TDA ensures the LLM remains matched to the speech projector’s distribution, while simultaneously adapting to target-domain linguistic content. This preserves critical cross-modal alignment and avoids catastrophic forgetting that otherwise occurs in naive adaptation [2601.20900].

## 4. Model Architecture and Training Dynamics

The TDA framework consists of a frozen speech encoder (e.g., WavLM-Large), a learnable speech projector (two linear layers with ReLU), and a LLM decoder (Llama 3B Instruct). The speech projector is trained exclusively on source audio-text pairs, while the LLM is adapted using mixed batches involving both real and synthetically noised data.

### Batch Mixing Proportions

Each adaptation batch is a mixture of four example types:
- $\sigma_a:$ Source audio-text pairs $(a, t)$
- $\sigma_{ta}:$ Source audio projected through speech projector and discretization, paired with transcript $(\text{noise}_a(t), t)$
- $\sigma_t:$ Synthetic noise applied to source transcripts $(\text{noise}(t), t)$
- $\tau_t:$ Synthetic noise applied to target-domain transcripts $(\text{noise}(t), t)$

Proportions are set as $\tau = |D_{\text{tgt}}| / |D_{\text{src}}|$ and the remainder equally split among $\sigma_a$, $\sigma_{ta}$, and $\sigma_t$:
$$
\sigma_a = \sigma_{ta} = \sigma_t = (1-\tau)/3
$$

### Hyperparameters
- Projector training: learning rate $10^{-4}$, warmup $1000$ steps, batch size $4$, $4$ epochs.
- TDA fine-tuning: learning rate $10^{-4}$, warmup $1000$ steps, batch size $4$, approximately $4$ epochs (varied by domain).
- Synthetic noise uses nlpaug defaults except with reduced word-edit rate $0.15$ [2601.20900].

No new modules or parameters are introduced relative to the pretrained ASR model, in contrast to soft-prompt methods.

## 5. Empirical Evaluation and Results

### Benchmarked Datasets

- **DefinedAI**: 125h of conversational audio, split between source domains (Banking, Insurance, Healthcare) and target-only text domains (Banking or Insurance).
- **SlideSpeech**: Audio-visual YouTube dataset; source domains (Life, Talent, English), target-only text in Agriculture, Animation, and Musical Instruments.

### Baselines

- Base Model: Projector trained, LLM frozen.
- Audio adaptation: LLM fine-tuned on target-domain audio/text using LoRA on self-attention layers.
- Recent text-only methods: perplexity-monitored early stopping (Fang et al.), soft-prompt embeddings (Ma et al.).

### Results

| Domain                      | Base WER | Audio Adapt | Fang et al. | Ma et al. | TDA (Ours)      |
|-----------------------------|----------|-------------|-------------|-----------|-----------------|
| DefinedAI → Banking         | 12.98%   | 9.92% (-23.6%) | 10.92% (-15.9%) | 10.63% (-18.1%) | 10.11% (-22.1%) |
| DefinedAI → Insurance       | -        | -           | -           | -         | -17.9% rel. gain|
| SlideSpeech (out-of-domain) | -        | -           | -           | -         | 4–8% rel. gain  |
| SlideSpeech (cross-domain)  | -        | -           | -           | -         | 10–15% rel. gain|

Relative improvement is defined as $\Delta=(\mathrm{WER}_{\text{base}} - \mathrm{WER}_{\text{method}})/\mathrm{WER}_{\text{base}}$.

TDA demonstrates up to 22.1% relative improvement in WER, approaching the upper bound achieved with audio-supervised fine-tuning and outperforming competing text-only methods.

## 6. Ablation Studies and Analysis

Batch composition ablations reveal that omitting the real audio component $\sigma_a$ leads to catastrophic forgetting (WER $>$ 73%). Models using all three source modalities $\{\sigma_a, \sigma_{ta}, \sigma_t\}$ achieve optimal results. Substituting or omitting the synthetic noise process for target text reduces gains (e.g., echoing prompt yields -20.6% vs -22.1%, empty prompts -18.6%, no prompt -18.9%), underscoring the necessity of explicit denoising.

## 7. Summary of TDA Training Workflow

A high-level pseudocode summary is provided for reference:

```python
# Input: D_src = {(a_i, t_i)}
#        D_src_text = {t_i}
#        D_tgt_text = {u_j}
#        τ = |D_tgt_text| / |D_src|
#        σ = (1−τ)/3
#        N(·) = synthetic_noise_fn

# Initialize:
#   Freeze LLM parameters θ_LLM
#   Train speech projector sp(·) on D_src for E₁ epochs

for epoch in 1…E₂:
    for each batch:
        B_a     ← sample σ·B examples from D_src
        B_ta    ← sample σ·B from D_src; x' = discretize(sp(audio_of(x)))
        B_srcN  ← sample σ·B from D_src_text; x' = N(x)
        B_tgtN  ← sample τ·B   from D_tgt_text;  x' = N(x)
        Inputs = concatenate([ (input_tokens(x'), x) for each set ])
        Loss = −∑ log p_θ_LLM(x|x')
        Backpropagate Loss w.r.t. θ_LLM

# Output: Adapted LLM parameters θ_LLM*
```

This approach ensures robust cross-modal alignment while imparting target-domain specialization, achieving strong gains without model expansion or additional hyperparameters [2601.20900].

Source: https://www.emergentmind.com/topics/text-denoising-adapter-tda