---
title: 'Denoising GER: Noise-Robust Multi-Modal ASR Correction'
url: https://www.emergentmind.com/topics/denoising-ger
type: topic
---

# Denoising GER: Noise-Robust Multi-Modal ASR Correction

Searching arXiv for the specified paper and closely related GER work to ground the article in current literature.
Denoising GER is a noise-robust multi-modal generative error correction framework for automatic speech recognition post-processing that targets the failure modes of large language model based GER in complex noisy environments. It is designed to improve adaptability to different noisy scenarios, increase utilization of multi-modal information, and align training with word error rate rather than only token-level cross-entropy. The framework combines a noise-adaptive acoustic encoder, a heterogeneous feature compensation dynamic fusion mechanism, and a reinforcement learning objective on top of a pretrained LLM backbone, and is reported to improve accuracy, robustness, and generalization in both seen and unseen noisy conditions while preserving clean-speech accuracy [2509.04392].

## 1. Position within noise-robust GER research

Generative error correction treats ASR post-processing as a conditional generation problem: an LLM receives ASR hypotheses and generates a corrected transcription. Earlier work on noisy GER had already shown that directly incorporating noise embeddings from an audio encoder could harm LLM tuning due to a cross-modality gap, and proposed instead a language-space noise embedding extracted from the N-best list to represent the noise conditions of source speech [2401.10446].

Denoising GER addresses a related but distinct problem formulation. Rather than remaining purely in language space, it introduces a noise-robust multi-modal GER framework in which acoustic and textual signals are both used, but are coupled through an explicit mechanism intended to mitigate heterogeneous representation mismatch. The paper’s central claim is that LLM-based GER in noisy conditions is limited by poor adaptability and low information utilization, and that these limitations can be addressed through three coordinated interventions: a light-weight acoustic adapter, a dynamic cross-modal fusion method, and reinforcement learning aligned with WER [2509.04392].

This suggests a transition within noise-robust GER research from text-only denoising cues toward controlled multi-modal integration. The paper’s design does not assume that simply adding speech features is sufficient; instead, it treats the speech-text gap as a structural problem that must be compensated during representation fusion.

## 2. Noise-adaptive acoustic encoding

The acoustic front end is a Noise-Adaptive Acoustic Encoder, or NAAE. Its base ASR encoder is Whisper-Large v2, which is kept mostly frozen, while a small U-Net-style adapter is inserted in front of the frozen encoder to denoise Mel-spectrograms. The adapter consists of 3 down-sampling (conv+ReLU+pool) layers and 3 up-sampling (transposed-conv+ReLU) layers, with approximately \(7.5\) M trainable parameters [2509.04392].

The adapter acts on noisy Mel-filterbank features \(X_{\rm in}\in\mathbb{R}^{T\times F}\) through residual adaptation:
\[
X_{\rm in}' \;=\; f_{\theta}(X_{\rm in})
\;=\; X_{\rm in} \;+\; \mathrm{Adapter}_\theta(X_{\rm in}).
\]

The adapted representation is then processed by the ASR encoder and decoder:
\[
\begin{cases}
X_{\rm audio} = \mathrm{Encoder}_{\rm ASR}\bigl(X_{\rm in}^{\prime}\bigr)\\
\widehat{Y}_{\rm hyp} = \mathrm{Decoder}_{\rm ASR}\bigl(X_{\rm audio}\bigr).
\end{cases}
\]

Training of this stage uses a combined denoising and cross-entropy objective for dynamic n-best generation:
\[
\mathcal{L}_{\rm ASR}
= \min_{\theta}\;\Bigl[\,
\lambda\,\mathcal{L}_{\rm CE}(X_{\rm in}',Y)
\;+\;(1-\lambda)\,\mathcal{L}_{1}(X_{\rm in}',X_{\rm in})
\Bigr].
\]

Architecturally, NAAE is a constrained adaptation strategy rather than full encoder retraining. In the reported ablations on DNS_LS100h and CHiME4, the NAAE setting outperforms frozen, full fine-tune, and LoRA-only acoustic encoder variants: for DNS, \(6.05/8.58\) WER versus \(7.16/10.69\), \(6.35/9.24\), and \(6.22/8.82\); for CHiME4, \(4.77/6.21\) versus \(5.92/8.12\), \(5.01/6.58\), and \(4.89/6.32\) [2509.04392].

The significance of this design is not merely parameter efficiency. The paper explicitly frames NAAE as a means of improving adaptability to different noisy scenarios. A plausible implication is that the frozen Whisper encoder preserves strong pretrained acoustic priors, while the small adapter absorbs the domain shift introduced by noise.

## 3. Heterogeneous feature compensation dynamic fusion

The second core component is Heterogeneous Feature Compensation Dynamic Fusion, abbreviated HFCDF. Its motivation is explicit: speech and text embeddings differ in space, and naïve concatenation leads to redundancy or misinterpretation by the LLM [2509.04392].

The mechanism begins by computing cross-modal difference vectors between frame-level speech embeddings \(X_{\rm tok}\) and 1-best text embeddings \(Y_{\rm tok}\):
\[
\begin{aligned}
\Delta_x &= X_{\rm tok}\;-\;Y_{\rm tok},\\
\Delta_y &= Y_{\rm tok}\;-\;X_{\rm tok}.
\end{aligned}
\]

These are used for cross-modal compensation:
\[
\begin{cases}
X'_{\rm tok} = X_{\rm tok}+k\,\Delta_y,\\
Y'_{\rm tok} = Y_{\rm tok}+(1-k)\,\Delta_x,
\end{cases}
\quad k\in[0,1].
\]

Dynamic weighting is then computed through cosine similarity with \(y\):
\[
\begin{aligned}
R_a &= \cos\bigl(X'_{\rm tok},\,y\bigr),\quad
R_t = \cos\bigl(Y'_{\rm tok},\,y\bigr),
\end{aligned}
\]
\[
\mu \;=\;\frac{\exp(R_a)}{\exp(R_a)+\exp(R_t)}.
\]

The compensated multi-modal representation is
\[
X'_{\rm mmc} \;=\;\mathrm{Concat}\bigl(\,
\mu\,X'_{\rm tok},\;\,(1-\mu)\,Y'_{\rm tok\,-1}
\bigr),
\]
where \(Y'_{\rm tok\,-1}\) denotes the top-1 text embedding. The LLM then predicts
\[
P(Y)\;=\;\prod_{t=1}^T P\bigl(y_t\,\bigm|\,\mathrm{Concat}(X'_{\rm mmc}, Y'_{\rm tok}),\,\theta_{\rm LLM}\bigr).
\]

In the fusion ablation where only the LLM is fine-tuned and the ASR is frozen, HFCDF yields the best results among the compared fusion strategies. On DNS_LS100h it achieves \(6.17/8.61\) versus linguistic-only \(6.48/9.39\), acoustic-only \(7.16/9.88\), Add \(6.41/8.93\), Concat \(6.35/8.81\), and Transformer \(6.28/8.77\). On CHiME4 it achieves \(4.53/6.08\) versus \(5.23/6.42\), \(5.86/6.93\), \(4.90/6.25\), \(4.78/6.12\), and \(4.65/6.13\) [2509.04392].

A common misconception in multi-modal GER is that stronger performance follows directly from adding more modalities. The formulation of HFCDF argues against that view: the issue is not only modality availability, but compensation for heterogeneous feature geometry and dynamic weighting of their relevance.

## 4. Reinforcement learning and metric alignment

The third component is a reinforcement learning training strategy motivated by the observation that cross-entropy loss does not align with the sentence-level WER metric [2509.04392]. The reward design is based on the negative WER difference per hypothesis. With
\[
w(y_i,y^*) = \mathrm{WER}(y_i,y^*), \qquad
\bar w = \tfrac1N\sum_i w(y_i,y^*),
\]
the policy-gradient-style loss over the n-best list is
\[
\mathcal{L}_{\rm RL}
= \frac1N \sum_{i=1}^N \widehat{p}(y_i|x)\;\bigl(w(y_i,y^*)-\bar w\bigr).
\]

The total training objective is
\[
\mathcal{L}_{\rm total}
= \mathcal{L}_{\rm LLM}
+ \alpha\,\mathcal{L}_{\rm ASR}
+ \beta\,\mathcal{L}_{\rm RL},
\]
where \(\mathcal{L}_{\rm LLM}\) is standard cross-entropy over generated tokens and \(\alpha,\beta\) are tunable hyper-parameters set to \(0.2\) in experiments.

The paper treats RL not as an isolated optimization trick but as a correction to objective mismatch. In the component-wise ablation on DNS_LS100h and CHiME4, the RL-only variant improves over the base system without NAAE, HFCDF, or RL: base \(6.32/8.87\) and \(5.03/6.62\), versus \(+{\rm RL}\) \(5.92/8.72\) and \(4.58/4.92\) [2509.04392].

This suggests that sentence-level optimization contributes independently of architectural denoising. More cautiously, the ablation indicates that WER-aware training can supply gains even when the acoustic encoder and fusion mechanism are not simultaneously upgraded.

## 5. Experimental configuration and empirical performance

The experimental protocol spans synthetic noisy, real noisy, out-of-domain noisy, and clean conditions. The synthetic noisy setting is LibriSpeech 100 h (train-clean-100) mixed with DNS-Challenge noise at SNR \(= 5\ldots20\) dB to form DNS_LS100h, evaluated on DNS_LS100h test-clean and test-other. The real noisy setting is CHiME-4 dev-real and test-real with single-channel real recordings from bus, cafe, pedestrian, and street. The out-of-domain noisy setting is MUSAN mixed with LibriSpeech100h at SNR \(= 5\ldots20\) dB to form MUSAN_LS100h. The clean setting is LibriSpeech test-clean and test-other. The evaluation metric is WER in percent, lower being better [2509.04392].

Model settings are specified as follows: acoustic model Whisper-Large v2 with adapter U-Net approximately \(7.5\) M parameters; LLM Qwen-7B with LoRA \((r=8)\) on Q/K/V/O layers, yielding \(26\) M trainable parameters, or \(0.32\%\) of the LLM; beam size \(=5\); max sequence length \(=1024\); Adam learning rate \(=2\times 10^{-4}\); \(5\) epochs; batch size \(=32\) [2509.04392].

In the main comparison, Denoising GER is reported as best overall. Its WERs are \(5.82/8.24\) on DNS_LS100h, \(4.28/5.32\) on CHiME4, \(6.18/8.16\) on MUSAN_LS100h, and \(2.08/4.32\) on clean LibriSpeech. The compared systems include the ASR baseline, \(\mathrm{LLM}_{\rm Rank}\), GER (Hyporadise), robustGER, Qwen-Audio, and MMGER [2509.04392].

The out-of-domain analysis is especially important. On MUSAN_LS100h test-clean and test-other, Denoising GER achieves \(6.18/8.16\%\) WER versus the best prior \(6.44/8.55\%\). On clean speech, performance is \(2.08/4.32\%\), and the paper states that clean speech is unaffected and that the system is more accurate than the baseline and other GERs. The case study in Table 5 shows Denoising GER correcting all substitutions and insertions to match ground truth [2509.04392].

## 6. Ablation logic, interpretation, and scope

The component-wise ablation presents the framework as an additive composition of three modules. Starting from the base system with no NAAE, no HFCDF, and no RL, the paper reports \(6.32/8.87\) on DNS_LS100h and \(5.03/6.62\) on CHiME4. The \(+{\rm NAAE}\) variant reaches \(6.05/8.58\) and \(4.77/6.21\); \(+{\rm HFCDF}\) reaches \(6.17/8.61\) and \(4.53/6.08\); \(+{\rm RL}\) reaches \(5.92/8.72\) and \(4.58/4.92\); \(+{\rm NAAE+HFCDF}\) reaches \(5.87/8.49\) and \(4.35/4.75\); and the full system reaches \(5.82/8.24\) and \(4.28/5.32\) [2509.04392].

The paper summarizes these ablations as confirming three functional roles: NAAE adapts to unseen noise, HFCDF balances modalities, and RL-loss directly optimizes WER. This interpretation is consistent with the formal structure of the model: NAAE modifies acoustic inputs before frozen ASR encoding, HFCDF corrects and reweights the heterogeneous speech-text interface, and RL shifts optimization pressure from token prediction toward sentence-level recognition quality.

Two technical issues recur throughout discussion of the method. First, the work explicitly rejects naïve concatenation as a sufficient fusion strategy because speech and text embeddings differ in space and may be misinterpreted by the LLM. Second, it rejects pure cross-entropy as a sufficient objective because cross-entropy loss does not align with sentence-level WER. These are not ancillary details; they define the paper’s account of why noisy GER remains difficult even when powerful pretrained backbones are available [2509.04392].

In broader context, Denoising GER can be understood as a multi-modal counterpart to earlier noise-robust GER work that stayed in language space to avoid a cross-modality gap [2401.10446]. The present framework instead attempts to recover the benefits of acoustic evidence without relinquishing LLM-centered generation. A plausible implication is that future GER systems may continue to move toward controlled multi-modal coupling rather than either pure text-only correction or raw audio-feature injection.

Source: https://www.emergentmind.com/topics/denoising-ger