Denoising GER: Noise-Robust Multi-Modal ASR Correction

• The paper introduces a noise-robust multi-modal GER framework that fuses acoustic and textual features using a noise-adaptive encoder, dynamic fusion, and reinforcement learning to optimize WER.
• The noise-adaptive acoustic encoder employs a lightweight U-Net adapter to denoise Mel-spectrograms while preserving pretrained Whisper priors for effective adaptation in noisy environments.
• The heterogeneous feature compensation dynamic fusion computes cross-modal difference vectors to mitigate modality mismatches, enabling improved accuracy and robustness in ASR post-processing.

Searching arXiv for the specified paper and closely related GER work to ground the article in current literature. Denoising GER is a noise-robust multi-modal generative error correction framework for automatic speech recognition post-processing that targets the failure modes of LLM based GER in complex noisy environments. It is designed to improve adaptability to different noisy scenarios, increase utilization of multi-modal information, and align training with word error rate rather than only token-level cross-entropy. The framework combines a noise-adaptive acoustic encoder, a heterogeneous feature compensation dynamic fusion mechanism, and a reinforcement learning objective on top of a pretrained LLM backbone, and is reported to improve accuracy, robustness, and generalization in both seen and unseen noisy conditions while preserving clean-speech accuracy (Liu et al., 4 Sep 2025).

1. Position within noise-robust GER research

Generative error correction treats ASR post-processing as a conditional generation problem: an LLM receives ASR hypotheses and generates a corrected transcription. Earlier work on noisy GER had already shown that directly incorporating noise embeddings from an audio encoder could harm LLM tuning due to a cross-modality gap, and proposed instead a language-space noise embedding extracted from the N-best list to represent the noise conditions of source speech (Hu et al., 2024).

Denoising GER addresses a related but distinct problem formulation. Rather than remaining purely in language space, it introduces a noise-robust multi-modal GER framework in which acoustic and textual signals are both used, but are coupled through an explicit mechanism intended to mitigate heterogeneous representation mismatch. The paper’s central claim is that LLM-based GER in noisy conditions is limited by poor adaptability and low information utilization, and that these limitations can be addressed through three coordinated interventions: a light-weight acoustic adapter, a dynamic cross-modal fusion method, and reinforcement learning aligned with WER (Liu et al., 4 Sep 2025).

This suggests a transition within noise-robust GER research from text-only denoising cues toward controlled multi-modal integration. The paper’s design does not assume that simply adding speech features is sufficient; instead, it treats the speech-text gap as a structural problem that must be compensated during representation fusion.

2. Noise-adaptive acoustic encoding

The acoustic front end is a Noise-Adaptive Acoustic Encoder, or NAAE. Its base ASR encoder is Whisper-Large v2, which is kept mostly frozen, while a small U-Net-style adapter is inserted in front of the frozen encoder to denoise Mel-spectrograms. The adapter consists of 3 down-sampling (conv+ReLU+pool) layers and 3 up-sampling (transposed-conv+ReLU) layers, with approximately $7.5$ M trainable parameters (Liu et al., 4 Sep 2025).

The adapter acts on noisy Mel-filterbank features $X_{\rm in}\in\mathbb{R}^{T\times F}$ through residual adaptation: $$X_{\rm in}' \;=\; f_{\theta}(X_{\rm in}) \;=\; X_{\rm in} \;+\; \mathrm{Adapter}_\theta(X_{\rm in}).$$

The adapted representation is then processed by the ASR encoder and decoder: $$\begin{cases} X_{\rm audio} = \mathrm{Encoder}_{\rm ASR}\bigl(X_{\rm in}^{\prime}\bigr)\ \widehat{Y}_{\rm hyp} = \mathrm{Decoder}_{\rm ASR}\bigl(X_{\rm audio}\bigr). \end{cases}$$

Training of this stage uses a combined denoising and cross-entropy objective for dynamic n-best generation: $$\mathcal{L}_{\rm ASR} = \min_{\theta}\;\Bigl[\, \lambda\,\mathcal{L}_{\rm CE}(X_{\rm in}',Y) \;+\;(1-\lambda)\,\mathcal{L}_{1}(X_{\rm in}',X_{\rm in}) \Bigr].$$

Architecturally, NAAE is a constrained adaptation strategy rather than full encoder retraining. In the reported ablations on DNS_LS100h and CHiME4, the NAAE setting outperforms frozen, full fine-tune, and LoRA-only acoustic encoder variants: for DNS, $6.05/8.58$ WER versus $7.16/10.69$, $6.35/9.24$, and $6.22/8.82$; for CHiME4, $4.77/6.21$ versus $X_{\rm in}\in\mathbb{R}^{T\times F}$0, $X_{\rm in}\in\mathbb{R}^{T\times F}$1, and $X_{\rm in}\in\mathbb{R}^{T\times F}$2 (Liu et al., 4 Sep 2025).

The significance of this design is not merely parameter efficiency. The paper explicitly frames NAAE as a means of improving adaptability to different noisy scenarios. A plausible implication is that the frozen Whisper encoder preserves strong pretrained acoustic priors, while the small adapter absorbs the domain shift introduced by noise.

3. Heterogeneous feature compensation dynamic fusion

The second core component is Heterogeneous Feature Compensation Dynamic Fusion, abbreviated HFCDF. Its motivation is explicit: speech and text embeddings differ in space, and naïve concatenation leads to redundancy or misinterpretation by the LLM (Liu et al., 4 Sep 2025).

The mechanism begins by computing cross-modal difference vectors between frame-level speech embeddings $X_{\rm in}\in\mathbb{R}^{T\times F}$3 and 1-best text embeddings $X_{\rm in}\in\mathbb{R}^{T\times F}$4: $X_{\rm in}\in\mathbb{R}^{T\times F}$5

These are used for cross-modal compensation: $X_{\rm in}\in\mathbb{R}^{T\times F}$6

Dynamic weighting is then computed through cosine similarity with $X_{\rm in}\in\mathbb{R}^{T\times F}$7: $X_{\rm in}\in\mathbb{R}^{T\times F}$8

$X_{\rm in}\in\mathbb{R}^{T\times F}$9

The compensated multi-modal representation is

$$X_{\rm in}' \;=\; f_{\theta}(X_{\rm in}) \;=\; X_{\rm in} \;+\; \mathrm{Adapter}_\theta(X_{\rm in}).$$0

where $$X_{\rm in}' \;=\; f_{\theta}(X_{\rm in}) \;=\; X_{\rm in} \;+\; \mathrm{Adapter}_\theta(X_{\rm in}).$$1 denotes the top-1 text embedding. The LLM then predicts

$$X_{\rm in}' \;=\; f_{\theta}(X_{\rm in}) \;=\; X_{\rm in} \;+\; \mathrm{Adapter}_\theta(X_{\rm in}).$$2

In the fusion ablation where only the LLM is fine-tuned and the ASR is frozen, HFCDF yields the best results among the compared fusion strategies. On DNS_LS100h it achieves $$X_{\rm in}' \;=\; f_{\theta}(X_{\rm in}) \;=\; X_{\rm in} \;+\; \mathrm{Adapter}_\theta(X_{\rm in}).$$3 versus linguistic-only $$X_{\rm in}' \;=\; f_{\theta}(X_{\rm in}) \;=\; X_{\rm in} \;+\; \mathrm{Adapter}_\theta(X_{\rm in}).$$4, acoustic-only $$X_{\rm in}' \;=\; f_{\theta}(X_{\rm in}) \;=\; X_{\rm in} \;+\; \mathrm{Adapter}_\theta(X_{\rm in}).$$5, Add $$X_{\rm in}' \;=\; f_{\theta}(X_{\rm in}) \;=\; X_{\rm in} \;+\; \mathrm{Adapter}_\theta(X_{\rm in}).$$6, Concat $$X_{\rm in}' \;=\; f_{\theta}(X_{\rm in}) \;=\; X_{\rm in} \;+\; \mathrm{Adapter}_\theta(X_{\rm in}).$$7, and Transformer $$X_{\rm in}' \;=\; f_{\theta}(X_{\rm in}) \;=\; X_{\rm in} \;+\; \mathrm{Adapter}_\theta(X_{\rm in}).$$8. On CHiME4 it achieves $$X_{\rm in}' \;=\; f_{\theta}(X_{\rm in}) \;=\; X_{\rm in} \;+\; \mathrm{Adapter}_\theta(X_{\rm in}).$$9 versus $$\begin{cases} X_{\rm audio} = \mathrm{Encoder}_{\rm ASR}\bigl(X_{\rm in}^{\prime}\bigr)\ \widehat{Y}_{\rm hyp} = \mathrm{Decoder}_{\rm ASR}\bigl(X_{\rm audio}\bigr). \end{cases}$$0, $$\begin{cases} X_{\rm audio} = \mathrm{Encoder}_{\rm ASR}\bigl(X_{\rm in}^{\prime}\bigr)\ \widehat{Y}_{\rm hyp} = \mathrm{Decoder}_{\rm ASR}\bigl(X_{\rm audio}\bigr). \end{cases}$$1, $$\begin{cases} X_{\rm audio} = \mathrm{Encoder}_{\rm ASR}\bigl(X_{\rm in}^{\prime}\bigr)\ \widehat{Y}_{\rm hyp} = \mathrm{Decoder}_{\rm ASR}\bigl(X_{\rm audio}\bigr). \end{cases}$$2, $$\begin{cases} X_{\rm audio} = \mathrm{Encoder}_{\rm ASR}\bigl(X_{\rm in}^{\prime}\bigr)\ \widehat{Y}_{\rm hyp} = \mathrm{Decoder}_{\rm ASR}\bigl(X_{\rm audio}\bigr). \end{cases}$$3, and $$\begin{cases} X_{\rm audio} = \mathrm{Encoder}_{\rm ASR}\bigl(X_{\rm in}^{\prime}\bigr)\ \widehat{Y}_{\rm hyp} = \mathrm{Decoder}_{\rm ASR}\bigl(X_{\rm audio}\bigr). \end{cases}$$4 (Liu et al., 4 Sep 2025).

A common misconception in multi-modal GER is that stronger performance follows directly from adding more modalities. The formulation of HFCDF argues against that view: the issue is not only modality availability, but compensation for heterogeneous feature geometry and dynamic weighting of their relevance.

4. Reinforcement learning and metric alignment

The third component is a reinforcement learning training strategy motivated by the observation that cross-entropy loss does not align with the sentence-level WER metric (Liu et al., 4 Sep 2025). The reward design is based on the negative WER difference per hypothesis. With

$$\begin{cases} X_{\rm audio} = \mathrm{Encoder}_{\rm ASR}\bigl(X_{\rm in}^{\prime}\bigr)\ \widehat{Y}_{\rm hyp} = \mathrm{Decoder}_{\rm ASR}\bigl(X_{\rm audio}\bigr). \end{cases}$$5

the policy-gradient-style loss over the n-best list is

$$\begin{cases} X_{\rm audio} = \mathrm{Encoder}_{\rm ASR}\bigl(X_{\rm in}^{\prime}\bigr)\ \widehat{Y}_{\rm hyp} = \mathrm{Decoder}_{\rm ASR}\bigl(X_{\rm audio}\bigr). \end{cases}$$6

The total training objective is

$$\begin{cases} X_{\rm audio} = \mathrm{Encoder}_{\rm ASR}\bigl(X_{\rm in}^{\prime}\bigr)\ \widehat{Y}_{\rm hyp} = \mathrm{Decoder}_{\rm ASR}\bigl(X_{\rm audio}\bigr). \end{cases}$$7

where $$\begin{cases} X_{\rm audio} = \mathrm{Encoder}_{\rm ASR}\bigl(X_{\rm in}^{\prime}\bigr)\ \widehat{Y}_{\rm hyp} = \mathrm{Decoder}_{\rm ASR}\bigl(X_{\rm audio}\bigr). \end{cases}$$8 is standard cross-entropy over generated tokens and $$\begin{cases} X_{\rm audio} = \mathrm{Encoder}_{\rm ASR}\bigl(X_{\rm in}^{\prime}\bigr)\ \widehat{Y}_{\rm hyp} = \mathrm{Decoder}_{\rm ASR}\bigl(X_{\rm audio}\bigr). \end{cases}$$9 are tunable hyper-parameters set to $$\mathcal{L}_{\rm ASR} = \min_{\theta}\;\Bigl[\, \lambda\,\mathcal{L}_{\rm CE}(X_{\rm in}',Y) \;+\;(1-\lambda)\,\mathcal{L}_{1}(X_{\rm in}',X_{\rm in}) \Bigr].$$0 in experiments.

The paper treats RL not as an isolated optimization trick but as a correction to objective mismatch. In the component-wise ablation on DNS_LS100h and CHiME4, the RL-only variant improves over the base system without NAAE, HFCDF, or RL: base $$\mathcal{L}_{\rm ASR} = \min_{\theta}\;\Bigl[\, \lambda\,\mathcal{L}_{\rm CE}(X_{\rm in}',Y) \;+\;(1-\lambda)\,\mathcal{L}_{1}(X_{\rm in}',X_{\rm in}) \Bigr].$$1 and $$\mathcal{L}_{\rm ASR} = \min_{\theta}\;\Bigl[\, \lambda\,\mathcal{L}_{\rm CE}(X_{\rm in}',Y) \;+\;(1-\lambda)\,\mathcal{L}_{1}(X_{\rm in}',X_{\rm in}) \Bigr].$$2, versus $$\mathcal{L}_{\rm ASR} = \min_{\theta}\;\Bigl[\, \lambda\,\mathcal{L}_{\rm CE}(X_{\rm in}',Y) \;+\;(1-\lambda)\,\mathcal{L}_{1}(X_{\rm in}',X_{\rm in}) \Bigr].$$3 $$\mathcal{L}_{\rm ASR} = \min_{\theta}\;\Bigl[\, \lambda\,\mathcal{L}_{\rm CE}(X_{\rm in}',Y) \;+\;(1-\lambda)\,\mathcal{L}_{1}(X_{\rm in}',X_{\rm in}) \Bigr].$$4 and $$\mathcal{L}_{\rm ASR} = \min_{\theta}\;\Bigl[\, \lambda\,\mathcal{L}_{\rm CE}(X_{\rm in}',Y) \;+\;(1-\lambda)\,\mathcal{L}_{1}(X_{\rm in}',X_{\rm in}) \Bigr].$$5 (Liu et al., 4 Sep 2025).

This suggests that sentence-level optimization contributes independently of architectural denoising. More cautiously, the ablation indicates that WER-aware training can supply gains even when the acoustic encoder and fusion mechanism are not simultaneously upgraded.

5. Experimental configuration and empirical performance

The experimental protocol spans synthetic noisy, real noisy, out-of-domain noisy, and clean conditions. The synthetic noisy setting is LibriSpeech 100 h (train-clean-100) mixed with DNS-Challenge noise at SNR $$\mathcal{L}_{\rm ASR} = \min_{\theta}\;\Bigl[\, \lambda\,\mathcal{L}_{\rm CE}(X_{\rm in}',Y) \;+\;(1-\lambda)\,\mathcal{L}_{1}(X_{\rm in}',X_{\rm in}) \Bigr].$$6 dB to form DNS_LS100h, evaluated on DNS_LS100h test-clean and test-other. The real noisy setting is CHiME-4 dev-real and test-real with single-channel real recordings from bus, cafe, pedestrian, and street. The out-of-domain noisy setting is MUSAN mixed with LibriSpeech100h at SNR $$\mathcal{L}_{\rm ASR} = \min_{\theta}\;\Bigl[\, \lambda\,\mathcal{L}_{\rm CE}(X_{\rm in}',Y) \;+\;(1-\lambda)\,\mathcal{L}_{1}(X_{\rm in}',X_{\rm in}) \Bigr].$$7 dB to form MUSAN_LS100h. The clean setting is LibriSpeech test-clean and test-other. The evaluation metric is WER in percent, lower being better (Liu et al., 4 Sep 2025).

Model settings are specified as follows: acoustic model Whisper-Large v2 with adapter U-Net approximately $$\mathcal{L}_{\rm ASR} = \min_{\theta}\;\Bigl[\, \lambda\,\mathcal{L}_{\rm CE}(X_{\rm in}',Y) \;+\;(1-\lambda)\,\mathcal{L}_{1}(X_{\rm in}',X_{\rm in}) \Bigr].$$8 M parameters; LLM Qwen-7B with LoRA $$\mathcal{L}_{\rm ASR} = \min_{\theta}\;\Bigl[\, \lambda\,\mathcal{L}_{\rm CE}(X_{\rm in}',Y) \;+\;(1-\lambda)\,\mathcal{L}_{1}(X_{\rm in}',X_{\rm in}) \Bigr].$$9 on Q/K/V/O layers, yielding $6.05/8.58$0 M trainable parameters, or $6.05/8.58$1 of the LLM; beam size $6.05/8.58$2; max sequence length $6.05/8.58$3; Adam learning rate $6.05/8.58$4; $6.05/8.58$5 epochs; batch size $6.05/8.58$6 (Liu et al., 4 Sep 2025).

In the main comparison, Denoising GER is reported as best overall. Its WERs are $6.05/8.58$7 on DNS_LS100h, $6.05/8.58$8 on CHiME4, $6.05/8.58$9 on MUSAN_LS100h, and $7.16/10.69$0 on clean LibriSpeech. The compared systems include the ASR baseline, $7.16/10.69$1, GER (Hyporadise), robustGER, Qwen-Audio, and MMGER (Liu et al., 4 Sep 2025).

The out-of-domain analysis is especially important. On MUSAN_LS100h test-clean and test-other, Denoising GER achieves $7.16/10.69$2 WER versus the best prior $7.16/10.69$3. On clean speech, performance is $7.16/10.69$4, and the paper states that clean speech is unaffected and that the system is more accurate than the baseline and other GERs. The case study in Table 5 shows Denoising GER correcting all substitutions and insertions to match ground truth (Liu et al., 4 Sep 2025).

6. Ablation logic, interpretation, and scope

The component-wise ablation presents the framework as an additive composition of three modules. Starting from the base system with no NAAE, no HFCDF, and no RL, the paper reports $7.16/10.69$5 on DNS_LS100h and $7.16/10.69$6 on CHiME4. The $7.16/10.69$7 variant reaches $7.16/10.69$8 and $7.16/10.69$9; $6.35/9.24$0 reaches $6.35/9.24$1 and $6.35/9.24$2; $6.35/9.24$3 reaches $6.35/9.24$4 and $6.35/9.24$5; $6.35/9.24$6 reaches $6.35/9.24$7 and $6.35/9.24$8; and the full system reaches $6.35/9.24$9 and $6.22/8.82$0 (Liu et al., 4 Sep 2025).

The paper summarizes these ablations as confirming three functional roles: NAAE adapts to unseen noise, HFCDF balances modalities, and RL-loss directly optimizes WER. This interpretation is consistent with the formal structure of the model: NAAE modifies acoustic inputs before frozen ASR encoding, HFCDF corrects and reweights the heterogeneous speech-text interface, and RL shifts optimization pressure from token prediction toward sentence-level recognition quality.

Two technical issues recur throughout discussion of the method. First, the work explicitly rejects naïve concatenation as a sufficient fusion strategy because speech and text embeddings differ in space and may be misinterpreted by the LLM. Second, it rejects pure cross-entropy as a sufficient objective because cross-entropy loss does not align with sentence-level WER. These are not ancillary details; they define the paper’s account of why noisy GER remains difficult even when powerful pretrained backbones are available (Liu et al., 4 Sep 2025).

In broader context, Denoising GER can be understood as a multi-modal counterpart to earlier noise-robust GER work that stayed in language space to avoid a cross-modality gap (Hu et al., 2024). The present framework instead attempts to recover the benefits of acoustic evidence without relinquishing LLM-centered generation. A plausible implication is that future GER systems may continue to move toward controlled multi-modal coupling rather than either pure text-only correction or raw audio-feature injection.