Generative Error Correction with LLMs

• GER is a generative paradigm that leverages LLMs to synthesize accurate transcriptions from noisy ASR outputs by recombining multiple candidate hypotheses.
• It utilizes a two-stage pipeline where an upstream ASR generates N-best hypotheses and the LLM, guided by prompt engineering, corrects errors through instruction-driven sequence generation.
• The approach extends to multimodal integration, accent conditioning, and retrieval augmentation, achieving significant improvements in word error rate and named entity recall.

Generative Error Correction (GER) with LLMs

Generative Error Correction (GER) denotes a paradigm in which LLMs are used to post-process, denoise, or correct outputs from upstream systems—most prominently automatic speech recognition (ASR)—using a generative, rather than merely discriminative or rescoring, approach. GER maps sets of candidate outputs (e.g., ASR N-best hypotheses) directly to high-fidelity textual transcriptions or other target forms, leveraging the compositional, contextual, and world knowledge encoded in LLMs. Unlike classical rescoring or error-detection methods, GER operates by treating error correction as an instruction-driven sequence generation problem. Recent work further demonstrates the modality expansion of GER (e.g., audio-visual inputs) and its application to a spectrum of error types, noise scenarios, and linguistic domains.

1. Core Principles and Methodological Foundations

At its foundation, GER for ASR operates as a two-stage pipeline: (1) an upstream recognizer (typically off-the-shelf ASR) produces a set of N-best hypotheses for each input utterance, and (2) a LLM, possibly multimodal, generates the corrected transcription conditioned on these hypotheses (and, in advanced cases, additional sensory or context information). This process can be mathematically formalized as:

$$\hat{y} = \arg\max_{y} \; P_\theta(y \mid \mathcal{H}, \mathcal{C})$$

where $\mathcal{H}$ is the set of N-best textual hypotheses, $\mathcal{C}$ represents optional conditioning such as multimodal embeddings (e.g., visual, acoustic, or noise information), and $P_\theta$ denotes the conditional distribution defined by the LLM with parameters $\theta$.

The generative nature is crucial: the LLM need not simply choose among or lightly edit the given hypotheses, but may synthesize new outputs by flexibly recombining tokens, repairing gaps across candidates, and introducing tokens not present in any single hypothesis (though bounded in practice by the information content of $\mathcal{H}$ and $\mathcal{C}$) (Ma et al., 2023, Yang et al., 2023, Mu et al., 2024, Ghosh et al., 2024, Ghosh et al., 2024, Yamashita et al., 23 May 2025).

Prompt design and instruction conditioning are foundational—for example, systems such as HyPoradise and LipGER employ canonical prompts that explicitly enumerate the "best" and "other" hypotheses and instruct the LLM to generate the "true transcription" using only the evidence contained within the input candidate set (Ghosh et al., 2024). Task-activating prompting, chain-of-thought reasoning, and in-context demonstration further improve stability and factual faithfulness (Yang et al., 2023, Sachdev et al., 2024).

Parameter-efficient adaptation via LoRA, adapters, or prefix-tuning is widely adopted; only the lightweight augmentation layers and possible auxiliary encoders are updated, while the LLM backbone remains frozen. This enables data efficiency and rapid adaptation to new domains or modalities (Chen et al., 2023, Ghosh et al., 2024, Ghosh et al., 2024, Yamashita et al., 23 May 2025).

2. Modalities and Conditioning Mechanisms

GER has rapidly extended from purely text-based N-best reasoning to leverage multimodal cues:

Text-Only GER: In the basic case, the LLM conditions solely on the N-best hypotheses from a speech-only recognizer. Both zero/few-shot and instruction-tuned approaches are effective; in some settings, flexible prompt engineering with in-domain examples yields competitive or superior performance relative to supervised baselines (Ma et al., 2023, Yang et al., 2023, Yang et al., 2024).

Audio-Visual GER: LipGER (Ghosh et al., 2024) and AVGER (Liu et al., 3 Jan 2025) demonstrate robust gains by augmenting GER with visual features, particularly lip motion. Instead of end-to-end audio-visual modeling, they extract fixed-length lip encodings via face tracking, 3D convolution, and temporal convolutional networks, and integrate these encodings into the LLM's input stream using adapter modules or multimodal prompt constructs. AVGER further introduces a synchronous multibranch encoder (audio and visual) and multi-level consistency constraints (cross-entropy, WER, and central moment discrepancy) to regularize LLM reasoning over joint modalities, achieving up to 24% WERR improvement on LRS3.

Accent and Pronunciation Conditioning: MMGER (Mu et al., 2024) and HDMoLE (Mu et al., 12 Jul 2025) integrate accent embeddings and multi-granularity features (frame, token, and utterance level), fusing acoustic representations with alignment to tokenized hypotheses and explicit accent recognizer outputs to specialize GER for multi-accent speech. Phonetic hints, e.g., simplified or IPA-based phoneme sequences, help preserve rare and domain-specific pronunciations (Yamashita et al., 23 May 2025), mitigating semantic overcorrection and improving out-of-vocabulary recall.

Noise and Environmental Awareness: RobustGER (Hu et al., 2024) extracts sentence- and token-level language-space noise embeddings from N-best hypotheses using SBERT/embedding models, then transfers audio noise information into these embeddings via mutual information estimation (MINE-based knowledge distillation). Denoising GER (Liu et al., 4 Sep 2025) employs a noise-adaptive acoustic encoder (U-Net residual) and a dynamic heterogeneous fusion module (HFCDF) to efficiently combine denoised acoustic features and textual hypotheses, with reinforcement learning loss directly minimizing WER under variable noise conditions.

Retrieval and Named Entity Augmentation: DARAG (Ghosh et al., 2024) explicitly retrieves named entities from a corpus using embedding similarity (SentenceBERT), appending top candidates as explicit input to help GER models correct rare or OOD entity strings—a significant bottleneck in baseline GER.

Cloze-Test/Multiple-Choice Conditioned GER: ClozeGER (Hu et al., 2024) frames the correction step as a (multimodal) cloze test: only positions that differ across N-best hypotheses are presented as blanks, with possible options enumerated per span. This efficiently reduces prompt length and focuses LLM computation on error localization. A logits calibration procedure corrects LLM bias toward always selecting the first candidate, further improving accuracy.

3. Training Protocols, Objectives, and Losses

The canonical GER learning objective is sequence-level autoregressive cross-entropy between generated tokens and ground-truth transcriptions:

$$\mathcal{L}_{\mathrm{CE}} = -\sum_{t=1}^T \log p_{\theta}(y_t^* \mid y_{<t}^*, \text{prompt}, \text{cond})$$

No explicit CTC or alignment loss is typically required; teacher forcing with paired (input, reference) examples suffices. For multimodal/adapter-based architectures, only the injected adapters and auxiliary encoders are updated (parameter efficiency).

Several extensions augment this recipe:

• Multi-objective consistency (AVGER): Regularizes model outputs at the logits, utterance (WER), and embedding (CMD) levels simultaneously (Liu et al., 3 Jan 2025).
• Reinforcement learning (Denoising GER): Directly aligns LLM output policy with WER minimization under sampled noisy hypotheses. The RL loss, often based on a negative WER reward, is combined with cross-entropy and reconstruction/adaptation objectives to stabilize learning (Liu et al., 4 Sep 2025).
• Rule-Based RL for GEC: On grammatical error correction tasks, rule-informed RL rewards ensure both format compliance and edit quality, balancing precision and recall for LLM GEC (Li et al., 26 Aug 2025).
• Retrieval-augmented entity losses (DARAG): Named entities from retrieved datastores are injected during training and fine-tuning, boosting F1 micro scores for NE correction (Ghosh et al., 2024).

Synthetic data augmentation is a widely used method to expose the GER model to rare words, unseen errors, or OOD domains—leveraging TTS, LLM-based utterance generation, and beam search over multi-speaker audio (Ghosh et al., 2024, Yamashita et al., 23 May 2025).

4. Evaluation Metrics, Benchmarks, and Comparative Results

Primary metrics: WER (word error rate), CER (character error rate), MER (mixed error rate in CS), F0.5 (edit-weighted GEC), and task-specific metrics such as entity F1.

GER consistently reduces WER relative to both:

• Upstream ASR baselines
• N-best oracles (by composition and flexible correction)
• Traditional LM rescoring and discriminative correction approaches.

Quantitative highlights:

• LipGER: WER reductions of 1.1–49.2% across LRS2, LRS3, Facestar, EasyCom (Ghosh et al., 2024).
• Text-only GER (ChatGPT, T5, LLaMA2): Gains of 4–10% relative WERR on LibriSpeech, TED-LIUM3, and Artie testbeds (Ma et al., 2023). One-shot in-context LLMs match supervised T5 with strong beam diversity.
• DARAG: 8–30% relative WER improvement ID, 10–33% OOD; named entity F1 increase of 2–5 points over baseline GEC (Ghosh et al., 2024).
• Rare Word–aware GER: Rare-word recall improves from <30% to >80% (MedTxt/Japanese domain) and correspondingly large WER/CER reductions when using appropriate synthetic data and phonetic hints (Yamashita et al., 23 May 2025).
• Audio-visual GER: AVGER achieves WER of 1.10% (24% WERR over AV-HuBERT) on LRS3 clean; DualHyp achieves up to 57.7% error-rate gain under severe noise/occlusion (Liu et al., 3 Jan 2025, Kim et al., 15 Oct 2025).
• Accent-aware GER: HDMoLE reduces WER by 67.35% vs. Whisper-large-v3 baseline by blending mono-accent LoRA experts dynamically (Mu et al., 12 Jul 2025).
• Noise-robust GER: RobustGER and Denoising GER attain up to 53.9% and 3.61pp absolute WER reduction, respectively, under varied and unseen noise (Hu et al., 2024, Liu et al., 4 Sep 2025).
• ClozeGER: Outperforms N-best oracle on several corpora by efficient slot-focused multimodal correction and calibration (Hu et al., 2024).

For non-ASR GEC, LLMs with tailored prompting/overcorrection and post-correction via sLMs (PoCO) achieve new single-model recall records while maintaining or exceeding F0.5 precision vs. state-of-the-art supervised baselines (Park et al., 25 Sep 2025).

5. Practical Considerations, Limitations, and Best Practices

GER with LLMs is sensitive to:

Prompt engineering: Explicit task description, prompt structure (tagged hypotheses, in-context demonstrations), and avoidance of prompt bias ("always select A") are essential. Evolutionary search for optimal prompts further improves WER (Sachdev et al., 2024).

Information bottleneck: GER cannot hallucinate missing tokens not present in $\mathcal{H}$. If all N-best hypotheses omit crucial words, correction fails. Adding more diverse hypotheses, multimodal evidence, or synthetic training examples mitigates this limit (Ghosh et al., 2024, Hu et al., 2024).

Scalability/data efficiency: LoRA fine-tuning or adapter-tuning enables adaptation with modest data (e.g., <20 h of parallel pairs or <10k examples); synthetic augmentation is critical for rare word and OOD robustness (Chen et al., 2023, Yamashita et al., 23 May 2025, Ghosh et al., 2024).

Error composition and hallucination: Unconstrained generation can hallucinate or over-correct; constrained or closest-mapping approaches (e.g., mapping generation back to an actual hypothesis via edit distance) balance creativity and fidelity (Ma et al., 2023, Mu et al., 2024).

Modality and domain transfer: Multimodal and domain-specific adapters dramatically increase robustness. Cross-modal gap must be addressed via embedding projection, dynamic fusion, and/or knowledge distillation for noisy or accented speech (Ghosh et al., 2024, Mu et al., 2024, Liu et al., 4 Sep 2025, Mu et al., 12 Jul 2025).

Computational cost: For large LLMs, context window and decoding latency are practical bottlenecks; prompt compaction via cloze-tests and post-processing stages address this (Hu et al., 2024).

6. Extensions and Related Tasks

GER has demonstrated generality beyond core ASR, with LLM-based GER (with or without multimodal adapters) now extended to:

• Speaker-attribution correction, emotion recognition, etc.: As in the GenSEC challenge, LLMs generate corrected transcriptions and associated labels from possibly erroneous ASR outputs—paving the way for next-generation language-agent pipelines (Yang et al., 2024).
• Code-switching ASR: GER performs H2T mapping for complex mixed-language hypotheses, leveraging LoRA for efficient cross-lingual adaptation (Chen et al., 2023).
• Grammatical error correction: LLMs (including open-source) can achieve minimal-edit correction with tailored prompting, role conditioning, and overcorrection+post-processing. RL-based and compositional voters further boost recall and F0.5 (Davis et al., 2024, Park et al., 25 Sep 2025, Li et al., 26 Aug 2025).
• Real-time caption editing, punctuation restoration, domain transfer: GER frameworks are being investigated for online correction and auxiliary generation tasks by expanding their input condition space (gesture, scene context) and output targets.

7. Open Problems and Future Directions

Key research frontiers include:

• Bayesian and joint modeling: Moving beyond N-best lists to full recognition lattices and hierarchical or streaming correction (requiring efficient context management and inference).
• Integrated multimodal LLMs: Bridging the cross-modal gap with end-to-end multimodal transformers, continuous acoustic unit modeling, or cross-modal attention/fusion (Liu et al., 3 Jan 2025, Ghosh et al., 2024, Hu et al., 2024).
• Adaptive and retrieval-based correction: Expanding entity retrieval, external knowledge bases, or dynamic error simulation for domain transfer (Ghosh et al., 2024).
• Prompt optimization: Automating prompt engineering, ordering of in-context demonstrations, and inferring optimal task activators for few-/zero-shot adaptation (Yang et al., 2023, Sachdev et al., 2024).
• Low-resource and real-time deployment: Investigating on-device LoRA adapters, prompt compaction (as in ClozeGER), and streaming architectures for latency-sensitive applications.
• Analysis of failure modes: Diagnosing systematic overcorrection, hallucination, or insensitivity to paralinguistic cues; exploring hybrid rescoring/generation models to guarantee bounded risk.
• Task expansion: Joint ASR correction with diarization, emotion, punctuation, document-level GEC, and multilingual support (Yang et al., 2024, Ghosh et al., 2024, Liu et al., 3 Jan 2025).

This rapidly evolving paradigm continues to reframe both ASR error correction and broader post-processing tasks as language-space generative problems, leveraging the inherent compositionality, multimodal reasoning, and factual knowledge of LLMs, with demonstrated state-of-the-art performance across a wide range of domains and scenarios (Ghosh et al., 2024, Ma et al., 2023, Ghosh et al., 2024, Hu et al., 2024, Liu et al., 3 Jan 2025, Mu et al., 2024, Mu et al., 12 Jul 2025, Kim et al., 15 Oct 2025, 2520.10025, Chen et al., 2023, Hu et al., 2024, Liu et al., 4 Sep 2025).