Whispering LLaMA: Cross-Modal ASR Correction

Updated 13 July 2026

Whispering LLaMA is a cross-modal framework that integrates Whisper’s acoustic encodings with LLaMA’s linguistic representations for generative ASR error correction.
It replaces traditional n-best rescoring by using lightweight adapters and identity-initialized projection layers to generate refined transcriptions.
Evaluations report up to 37.66% relative word error rate reduction, evidencing its effectiveness across diverse ASR datasets.

Whispering LLaMA is a cross-modal generative error correction framework for automatic speech recognition (ASR) that couples Whisper-derived acoustic representations with LLaMA-derived linguistic representations in a unified second-pass model (Radhakrishnan et al., 2023). Introduced as an alternative to ranking-based rescoring over n-best hypotheses, it performs generative correction rather than candidate selection, using distinct initialization techniques and parameter-efficient adaptation of pre-trained speech and text models. In evaluation across diverse ASR datasets, the framework reported improved word error rate relative (WERR) performance of 37.66% relative to n-best hypotheses (Radhakrishnan et al., 2023).

1. Problem setting and conceptual scope

Whispering LLaMA was proposed for ASR error correction in a setting where first-pass decoding already produces candidate transcriptions, but residual errors remain. The framework targets the limitations of conventional rescoring pipelines, which typically use a LLM only to rank n-best hypotheses after ASR decoding. In contrast, Whispering LLaMA uses both acoustic information and external linguistic representations to generate corrected transcription contexts, thereby moving from ranking to generation (Radhakrishnan et al., 2023).

Its basic input configuration is explicit. Each sample consists of Whisper-encoded audio together with a set of 15 n-best textual hypotheses generated by Whisper-Tiny. These hypotheses are provided to LLaMA through an "Instruction-Input-Response" template from Alpaca, while acoustic features are fused into the LLM’s internal computation (Radhakrishnan et al., 2023). A recurrent misconception is to treat the method as another rescoring model; the formulation instead generates new corrected hypotheses and is not limited to selecting one item from the original candidate list.

The framework therefore occupies a specific place in the ASR pipeline. It is neither a first-pass end-to-end recognizer nor a pure text-only post-processor. Its defining characteristic is cross-modal correction conditioned simultaneously on the original speech signal and on textual hypotheses produced by an upstream recognizer (Radhakrishnan et al., 2023).

The core architectural move in Whispering LLaMA is to inject acoustic features from Whisper into the self-attention mechanism of LLaMA at every transformer layer. The model uses token-level fusion throughout the LLaMA stack rather than a shallow late-fusion stage, which makes the LLM’s generative process directly sensitive to both modalities (Radhakrishnan et al., 2023).

Two residual adapter modules are inserted after self-attention in every LLaMA layer. The adapter $A_{L}^{i}$ learns to fine-tune LLaMA representations via attention, while $A_{W}^{i}$ fuses cross-modal audio features into the LLM. The formulation is summarized in the paper as follows (Radhakrishnan et al., 2023):

$A_{L}^{i} = \text{Softmax}\left(\frac{Q_{llama}^{i}(H_L^i) \cdot K_{llama}^{i}(M_{\theta}^i)^T}{\sqrt{d_k}}\right) V_{llama}^{i}(M_{\theta}^i)$

$A_{W}^{i}(x) = \text{SiLU}(x M_{down}^i) M_{up}^i$

$\hat{K}_{whisper}^{i} = A_{W}^{i}(K_{whisper}^{i}(H_\text{audio})), \qquad \hat{V}_{whisper}^{i} = A_{W}^{i}(V_{whisper}^{i}(H_\text{audio}))$

$SA_{\mathcal{WL}^i} = SA_L^i + \lambda_L A_L^i + \lambda_W SA_W^i$

This construction gives LLaMA access to linguistic priors from its pre-training while grounding correction in Whisper encodings that retain environmental and pronunciation details. The learnable gating scalars $\lambda_L$ and $\lambda_W$ regulate the relative contribution of text-side and audio-side adaptation (Radhakrishnan et al., 2023). Within the paper’s framing, this fusion is the mechanism that enables generative correction unavailable to either a standalone speech model or a text-only LLM.

3. Initialization and parameter-efficient adaptation

Whispering LLaMA is explicitly designed as a parameter-efficient system. Both LLaMA and Whisper backbones remain frozen, and only adapters and trainable projection components are optimized. The paper reports 4.89M trainable parameters for the smallest model and 7.97M for the best-performing model, despite relying on substantially larger frozen backbones (Radhakrishnan et al., 2023).

A central technical issue is dimensional compatibility between Whisper and LLaMA latent spaces. The framework uses key/value reshaping and padding to match the two models’ latent dimensionalities, with a careful initialization procedure that preserves Whisper’s latent structure. Projection layers in adapters are initialized as identity matrices, and key/value paddings are placed on principal diagonals to avoid information loss in early training (Radhakrishnan et al., 2023). The paper treats this as necessary for convergence rather than an implementation detail.

The ablation evidence reinforces that point. Removal of initialization leads to convergence failure, while removing cross-modal components or masking causes large performance degradation (Radhakrishnan et al., 2023). This suggests that the method’s contribution is not merely the presence of two frozen foundation models, but the specific way in which their representations are aligned and adapted.

4. Empirical performance and ablation evidence

The reported experiments cover ATIS and multiple GigaSpeech domains. The strongest model, denoted $\mathcal{WL}_M$ , achieved an average WER of 13.48% and a relative WERR of 37.66% against the Oracle baseline, which had an average WER of 21.64% (Radhakrishnan et al., 2023).

Method	Params	Avg WER / WERR
Oracle (baseline)	-	21.64 / -
$\mathcal{WL}_M$	7.97M	13.48 / 37.66%

The paper also reports representative dataset-specific values for this comparison: Oracle gives 13.76 on ATIS and 28.22 on GS(E), whereas $A_{W}^{i}$ 0 gives 1.77 on ATIS and 21.61 on GS(E) (Radhakrishnan et al., 2023). These values are presented as evidence that the generative fusion strategy improves over the n-best hypothesis baseline across heterogeneous ASR conditions.

The ablation table is as important as the headline result. Without masking, WERR drops by more than 15%. Without audio input, WERR becomes massively negative at $A_{W}^{i}$ 1. Without the initialization strategy, the model fails to converge and yields $A_{W}^{i}$ 2 WERR. Without the cross-modal adapter, WERR drops to 28.8% from 37.7% (Radhakrishnan et al., 2023). Collectively, these results establish that acoustic grounding, initialization, and cross-modal adaptation are all load-bearing components of the framework.

The project was released with open-source code and pre-trained models at https://github.com/Srijith-rkr/Whispering-LLaMA, which the paper presents as a reproducibility measure intended to support subsequent research (Radhakrishnan et al., 2023).

5. Relation to later Whisper–LLaMA speech systems

Subsequent work broadened the design space of Whisper–LLaMA integration beyond n-best ASR correction. AudioChatLlama extends Llama-2-chat with end-to-end general-purpose speech processing and reasoning abilities while keeping the LLM frozen and training only the audio encoder side. Its key innovation is a modality-invariant objective that uses abundant ASR data and synthetic LLM responses instead of curated paired audio-text-response data, enabling spoken question answering, speech translation, audio summarization, and modality switching within a conversation (Fathullah et al., 2023). In relation to Whispering LLaMA, the important contrast is functional: AudioChatLlama is a conversational speech-input LLM, not a second-pass correction model.

WHISMA combines the speech encoder from Whisper with Llama-3-8B-Instruct through a trainable modality aligner and LoRA-based parameter-efficient fine-tuning for spoken language understanding. It was tailored for zero-shot SLU and reported a relative gain of 26.6% on zero-shot slot filling on SLURP, as well as a relative gain of 33.0% over Qwen-Audio on the SLU-GLUE benchmark (Li et al., 2024). The architecture remains within the Whisper-plus-LLM family, but the task emphasis shifts from transcript correction to intent classification, slot filling, and task-agnostic spoken reasoning.

A further extension appears in multi-talker settings. MT-LLM combines a Whisper encoder and WavLM with Llama-2-7B-chat, using LoRA and trainable adapters to support instruction-following ASR in cocktail party scenarios. It handles multi-talker ASR, target-talker ASR, keyword-tracing, sex-specific, order-specific, and target-lingual transcription within one unified model (Meng et al., 2024). A plausible implication is that the original Whispering LLaMA paper helped crystallize a broader architectural pattern in which Whisper-derived acoustic representations are fused with LLaMA-family decoders, although later systems target tasks substantially different from generative n-best correction.

6. Long-context, efficiency, and deliberation extensions

Later research also explored what happens when the Whisper–LLaMA connection is used to inject global linguistic context back into ASR. "Whispering Context: Distilling Syntax and Semantics for Long Speech Transcripts" distills contextual knowledge from LLaMA models into Whisper with two strategies: token-level distillation with optimal transport and representation loss minimization between sentence embeddings. On Spoken Wikipedia, the fine-tuned Whisper baseline achieves WER 0.26, while the distilled model reduces WER to 0.20 and, with the best configuration at context 1024, to 18.5%; NER F1 rises from 0.61 to as high as 0.89 (Altinok, 18 Aug 2025). Unlike the original Whispering LLaMA, which corrects errors by cross-modal generation over n-best hypotheses, this line of work distills syntax and semantics directly into the recognizer for long-form, fully formatted transcription.

Efficiency-oriented work addressed a different bottleneck: KV-cache growth in Whisper itself. Whisper-MLA adapts Multi-Head Latent Attention to the Whisper architecture despite Whisper’s use of absolute positional embeddings, and shows that applying MLA only to decoder self-attention yields the best trade-off between memory savings and recognition accuracy. On LibriSpeech, Whisper-MLA reduces KV cache size by up to 87.5% while maintaining competitive accuracy; in the DSO configuration with uniform sampling it reports 81.25% KV reduction and average WER 11.12 versus 10.95 for the fine-tuned Whisper baseline. For batch size 64 and sequence length 2048, standard Whisper overflows a 24 GB GPU, whereas Whisper-MLA remains at approximately 15.4 GB (Zhang et al., 28 Feb 2026). Although this paper is not a Whisper–LLaMA fusion method, it addresses a systems issue that directly affects long-context speech pipelines in which Whisper-like encoders or decoders are paired with LLMs.

Diffusion-based deliberation pushed the post-correction idea closer to a hybrid decoding stack. "Audio-Conditioned Diffusion LLMs for ASR and Deliberation Processing" uses LLaDA as an external deliberation module for Whisper-LLaMA transcripts. With audio-conditioned embeddings, the best cascade system on LibriSpeech achieves 2.25% WER on test-clean and 4.94% on test-other, corresponding to a 12.3% relative improvement over the Whisper-LLaMA baseline on test-other. By contrast, plain-text LLaDA without acoustic features degrades performance to 3.89% and 6.91%, respectively (Wang et al., 20 Sep 2025). This result reinforces a theme already present in the original framework: linguistic refinement alone is insufficient, and acoustic conditioning remains essential when the goal is reliable ASR correction.

Across these extensions, Whispering LLaMA remains most precisely identified with the 2023 cross-modal generative error correction framework (Radhakrishnan et al., 2023). Its broader significance lies in establishing a concrete recipe for combining frozen speech and language backbones through lightweight adapters, careful initialization, and multimodal attention, thereby influencing later work on conversational speech prompting, zero-shot SLU, long-form contextual transcription, and deliberative post-processing.