Speech LLaMA: Audio-Text Integration

• Speech LLaMA is an architectural paradigm that integrates speech representations into a LLaMA decoder, enabling tasks such as multilingual ASR, translation, and speech synthesis.
• Design strategies leverage techniques like explicit compression, prefix-conditioning, and cross-modal fusion to effectively map high-dimensional audio features into LLaMA’s embedding space.
• Parameter-efficient adaptations, including LoRA and task-specific training objectives, reduce trainable parameters while enhancing performance and mitigating issues like catastrophic forgetting.

Searching arXiv for recent and foundational papers on Speech LLaMA. Speech LLaMA denotes a line of architectures that attach speech representations to a LLaMA or LLaMA-derived decoder-only LLM so that the model can condition on audio, generate text, and, in some variants, generate speech tokens or waveforms. In the literature, the term does not identify a single canonical model. Instead, it covers a family of systems that differ in how speech is compressed, projected, fused, or tokenized before reaching the LLM, and in which downstream tasks are targeted: multilingual ASR, speech translation, spoken language understanding, audio reasoning, error correction, dialogue, emotion captioning, and text-to-speech (Fathullah et al., 2023, Wu et al., 2023, Li et al., 2024, Ye et al., 6 Feb 2025).

1. Origins and scope of the term

Two 2023 papers established the core pattern now commonly associated with Speech-LLaMA. "On decoder-only architecture for speech-to-text and LLM integration" used a frozen LLaMA-7B with a CTC compressor and a small audio encoder for multilingual speech translation (Wu et al., 2023). "Prompting LLMs with Speech Recognition Abilities" attached a conformer audio encoder to LLaMA-7B for multilingual ASR by prepending projected audio embeddings to text token embeddings (Fathullah et al., 2023). These works framed speech not as a separate encoder-decoder problem, but as a prefix-conditioning problem for a decoder-only LLM.

Subsequent work broadened the label well beyond ASR. WHISMA targeted zero-shot spoken language understanding by combining Whisper-large-v2 with Llama-3-8B-Instruct (Li et al., 2024). LTU-AS coupled Whisper and LLaMA for joint audio and speech understanding (Gong et al., 2023). AudioChatLlama extended Llama-2-chat-7B to accept audio prompts in open-domain conversation (Fathullah et al., 2023). Whispering LLaMA used cross-modal fusion for generative ASR error correction (Radhakrishnan et al., 2023). On the generation side, TTS-LLaMA, MoLE-LLaMA, Llasa, Llasa+, and Llama-VITS used LLaMA backbones or LLaMA embeddings for speech synthesis and multimodal speech-output systems (Shen et al., 2024, Ye et al., 6 Feb 2025, Tian et al., 8 Aug 2025, Feng et al., 2024).

System	Bridge to LLaMA	Primary use
Speech LLaMA	Conformer audio encoder + projection	Multilingual ASR
Speech-LLaMA	CTC compressor + audio encoder	Multilingual speech translation
WHISMA	Whisper encoder + modality aligner + LoRA	Zero-shot SLU
LTU-AS	Whisper + TLTR + LoRA	Joint audio and speech understanding
AudioChatLlama	Conformer-CTC frontend + projection	General-purpose speech interaction
Whispering LLaMA	Layerwise cross-modal adapters	Generative ASR error correction
Llasa	Codec tokens in a LLaMA Transformer	Speech synthesis

This distribution of systems suggests that "Speech LLaMA" is best understood as an architectural paradigm rather than a benchmark-specific model family.

2. Architectural patterns

The simplest Speech-LLaMA design projects speech features into the same embedding space as text tokens and then concatenates them before decoding. In Speech LLaMA for ASR, raw waveforms are converted to 80-dimensional log-Mel filterbank features, processed by a conformer encoder, stacked to reduce sequence length, and projected into LLaMA’s 4096-dimensional token space. The resulting joint input is

$$X = [\,s'_1+p_1,\dots,s'_{T'_a}+p_{T'_a};\; t_1+p_{T'_a+1},\dots,t_{T_t}+p_{T'_a+T_t}\,],$$

which is then passed through the standard decoder layers without modifying the core LLaMA architecture (Fathullah et al., 2023).

A second recurring pattern is explicit speech compression before projection. The multilingual speech-translation Speech-LLaMA used a frozen CTC compressor with blank-removal or frame-averaging, followed by a 4-layer audio encoder that maps compressed 512-dimensional features into LLaMA’s 4096-dimensional space (Wu et al., 2023). WHISMA used Whisper-large-v2 as speech encoder and a modality aligner composed of two 1-D CNN layers, a bottleneck adaptor with hidden dimension 320, and a linear projection to Llama-3’s embedding dimension; after Whisper’s $8\times$ down-sampling and the aligner’s additional $2\times$ reduction, the system produces 375 vectors per utterance (Li et al., 2024). These designs address the principal systems problem of Speech-LLaMA: raw audio sequences are too long to feed directly into a large decoder.

Other systems replace simple prefixing with more structured fusion. LTU-AS formed continuous audio tokens $A$ from Whisper encoder states via a Time-and-Layer-Wise Transformer, discrete speech tokens $S$ from the Whisper decoder, and question tokens $Q$, then concatenated $[A,S,Q]$ before a frozen LLaMA with LoRA on self-attention key and query projections (Gong et al., 2023). Whispering LLaMA injected Whisper acoustic features into LLaMA layer by layer through two adapters per layer and a gated fusion term, reusing LLaMA’s self-attention machinery for cross-modal error correction (Radhakrishnan et al., 2023). SECap adopted an encoder-bridge-decoder design in which HuBERT supplies frame-level speech features, a Q-Former Bridge-Net compresses them into emotion-related embeddings, and a frozen LLaMA decoder generates captions (Xu et al., 2023).

On the synthesis side, the architectural direction is inverted. TTS-LLaMA extends the Llama 3-8B-Instruct vocabulary with 4,096 discrete audio-semantic tokens and couples it to a small acoustic transformer and an RVQ vocoder (Shen et al., 2024). Llasa replaces text-only output with X-codec2 speech tokens under a single Transformer aligned to LLaMA (Ye et al., 6 Feb 2025). Llama-VITS does not use LLaMA as the decoder, but instead injects frozen Llama 2 semantic embeddings into VITS through linear projection and either addition or attention, showing a parallel line in which LLaMA serves as a semantic conditioner for TTS (Feng et al., 2024).

3. Training objectives and adaptation strategies

Most Speech-LLaMA systems retain standard autoregressive next-token prediction as the final optimization target. Speech LLaMA for ASR pretrains the audio encoder with the CTC loss and then jointly fine-tunes the audio encoder and LLaMA interface with cross-entropy, while optionally keeping the LLM frozen or updating only LoRA adapters (Fathullah et al., 2023). The multilingual speech-translation variant similarly pretrains the CTC compressor, trains the audio encoder alone, and then adds LoRA adapters of rank $r=2$ to the four attention matrices per LLaMA layer for a second stage of fine-tuning (Wu et al., 2023).

Parameter-efficient adaptation is a defining motif. WHISMA inserted LoRA into every self-attention weight matrix $W\in\mathbb{R}^{d\times d}$ in all 32 Llama-3 layers, using

$$W' = W + \frac{\alpha}{r}BA,\quad B\in\mathbb{R}^{d\times r},\;A\in\mathbb{R}^{r\times d},$$

with rank $8\times$0 and scaling $8\times$1 (Li et al., 2024). This combination of frozen backbone and lightweight adaptation recurs across the literature because it reduces trainable parameters, mitigates catastrophic forgetting, and preserves the base LLM’s text behavior. A plausible implication is that Speech-LLaMA is less a matter of retraining an LLM from scratch than of learning a narrow alignment interface between pretrained speech and text modules.

The literature also introduced task-specific training schemes that go beyond plain speech-to-text supervision. AudioChatLlama used a modality-invariant objective: for each transcript, the frozen Llama-2-chat model first generates a text response, and the speech frontend is then trained so that the same response is produced from the spoken version of that prompt, without curated task-specific paired speech-text-response data (Fathullah et al., 2023). WHISMA used multi-task fine-tuning on approximately 2000 hours spanning ASR, intent classification, slot filling, spoken QA, spoken query instruction tuning, and spoken instruction tuning, together with prompt diversity and three SLU-chain modes: SLU-alone, speech chain-of-thought, and multi-round prompting (Li et al., 2024). SECap trained its Q-Former Bridge-Net with mutual-information minimization between transcript and acoustic embeddings and contrastive learning against caption embeddings so that LLaMA receives emotion-related rather than content-leaking features (Xu et al., 2023).

Sequence-level optimization and inference-aware training also appear. Faster Speech-LLaMA extends the decoder to predict $8\times$2 tokens in parallel under the approximation

$8\times$3

and supplements cross-entropy with minimum word error rate fine-tuning using prefix-based beam search (Raj et al., 2024). On the synthesis side, Llasa scales inference-time compute through verifier-guided search, while Llasa+ adds plug-and-play multi-token prediction modules with a verification algorithm that uses the frozen backbone as an oracle (Ye et al., 6 Feb 2025, Tian et al., 8 Aug 2025).

4. Recognition, translation, and spoken language understanding

The first clear empirical success of Speech-LLaMA was multilingual ASR. On Multilingual LibriSpeech dev-test, Speech LLaMA with stride $8\times$4 ms and LoRA rank $8\times$5 achieved an average WER of $8\times$6, compared with $8\times$7 for the best monolingual no-LM baseline, corresponding to an approximately $8\times$8 relative reduction (Fathullah et al., 2023). The same study showed that multilingual ASR remained possible even with LLaMA completely frozen, where $8\times$9 yielded $2\times$0 average WER, and that increasing the LoRA rank to $2\times$1 reduced average WER to $2\times$2 (Fathullah et al., 2023). The multilingual speech-translation Speech-LLaMA reached $2\times$3 average BLEU on CoVoST-2 13$2\times$4EN, a $2\times$5 absolute improvement over the strong seq2seq baseline, while using only $2\times$6 million learned parameters (Wu et al., 2023).

WHISMA moved Speech-LLaMA into zero-shot SLU. On the SLURP zero-shot split, WHISMA with multi-round prompting achieved SLU-F1 $2\times$7, compared with $2\times$8 for ZS-Whisper-SLU, giving the reported $2\times$9 relative gain (Li et al., 2024). On FSC, WHISMA with speech chain-of-thought or multi-round prompting reached $A$0 intent accuracy, above the $A$1 of ZS-Whisper-SLU (Li et al., 2024). On the task-agnostic SLU-GLUE benchmark, WHISMA with multi-round prompting achieved $A$2 average binary accuracy, versus $A$3 for Qwen-Audio + MR, a $A$4 relative gain (Li et al., 2024). Its ablation on Spoken-Alpaca is particularly notable: removing 44k Spoken-Alpaca examples, only $A$5 of training data, dropped UTUC average from $A$6 to $A$7 under MR, indicating that diverse instruction tuning materially affects zero-shot generalization (Li et al., 2024).

Whispering LLaMA shows that the Speech-LLaMA pattern is not limited to direct transcription. In generative ASR error correction over n-best hypotheses, the best medium-sized model $A$8 used only $A$9 million trainable parameters yet reduced average WER from $S$0 for the oracle n-best baseline to $S$1, corresponding to $S$2 WERR (Radhakrishnan et al., 2023). The same paper reported that removing masking dropped WERR to $S$3, removing the audio pathway caused collapse, and removing the initialization caused failure to converge (Radhakrishnan et al., 2023). This indicates that in Speech-LLaMA-style correction systems, alignment and initialization are not peripheral implementation details but central determinants of stability.

5. General audio reasoning, dialogue, and speech generation

Speech LLaMA has also been used for audio reasoning beyond ASR. LTU-AS integrated Whisper as a perception module and LLaMA as a reasoning module, allowing simultaneous treatment of spoken text, speech paralinguistics, and non-speech audio events (Gong et al., 2023). On closed-ended benchmarks it reported $S$4 accuracy on ESC-50, $S$5 SPICE on AudioCaps, $S$6 WER on LibriSpeech test-clean, $S$7 accuracy on IEMOCAP, $S$8 macro-F1 on VoxCeleb2 gender, $S$9 MAE on speaker age, and $Q$0 accuracy on GTZAN (Gong et al., 2023). The model’s design, which jointly feeds continuous audio tokens and discrete speech tokens into LLaMA, directly contradicts the misconception that speech-capable LLMs are merely ASR front ends attached to text-only reasoning modules.

AudioChatLlama pursued general-purpose speech interaction with a completely frozen Llama-2-chat-7B and a trainable Conformer-CTC frontend (Fathullah et al., 2023). On response perplexity under the oracle text-prompted answer, AudioChatLlama with a 36-layer Conformer achieved $Q$1 PPL on MLS and $Q$2 on TriviaQA-TTS, compared with $Q$3 and $Q$4 for a cascaded ASR $Q$5 LLM system using a $Q$6 WER ASR checkpoint (Fathullah et al., 2023). The same work emphasized modal interchange: audio and text prompts can be mixed across turns while preserving dialogue context (Fathullah et al., 2023).

SECap extended the pattern to emotion captioning. Using HuBERT, Q-Former, and a frozen Chinese-finetuned LLaMA, SECap improved over HTSAT-BART on all objective evaluations; for example, SIM$Q$7 increased from $Q$8 to $Q$9, BLEU$[A,S,Q]$0 from $[A,S,Q]$1 to $[A,S,Q]$2, METEOR from $[A,S,Q]$3 to $[A,S,Q]$4, and CIDEr from $[A,S,Q]$5 to $[A,S,Q]$6 (Xu et al., 2023). In subjective evaluation, Human Caption achieved MOS $[A,S,Q]$7, SECap (Q-Emb+T-Emb) MOS $[A,S,Q]$8, SECap (Q-Emb only) MOS $[A,S,Q]$9, and HTSAT-BART MOS $r=2$0 (Xu et al., 2023). Here LLaMA contributes primarily as a fluent caption generator once emotion-salient features have been disentangled upstream.

On speech generation, several strands emerged. TTS-LLaMA fine-tuned Llama 3-8B-Instruct to emit semantic speech tokens and reported zero-shot MOS scores of $r=2$1 for likeness and $r=2$2 for quality, while MoLE-LLaMA used a mixture-of-LoRA-experts with late fusion to mitigate catastrophic forgetting and recovered text QA performance to $r=2$3 on 5-shot MMLU, $r=2$4 on zero-shot GPQA, and $r=2$5 on ARC Challenge after TTS adaptation (Shen et al., 2024). Llasa unified text and speech tokens under a single LLaMA-aligned Transformer and showed that scaling from $r=2$6B at $r=2$7k hours to $r=2$8B at $r=2$9k hours reduced LibriSpeech-clean WER from $W\in\mathbb{R}^{d\times d}$0 to $W\in\mathbb{R}^{d\times d}$1, while hybrid verifier-guided search at $W\in\mathbb{R}^{d\times d}$2B reached WER $W\in\mathbb{R}^{d\times d}$3, SIM-o $W\in\mathbb{R}^{d\times d}$4, and SIM-r $W\in\mathbb{R}^{d\times d}$5 on LibriSpeech (Ye et al., 6 Feb 2025). Llasa+ then added multi-token prediction and verification to obtain a $W\in\mathbb{R}^{d\times d}$6 speedup without sacrificing generation quality, whereas removing verification raised WER to $W\in\mathbb{R}^{d\times d}$7 (Tian et al., 8 Aug 2025). Llama-VITS, finally, showed that even frozen Llama 2 embeddings can enhance VITS-style TTS: on EmoV_DB_bea_sem, Llama-VITS with attention over text-sequence embeddings reached ESMOS $W\in\mathbb{R}^{d\times d}$8, above ORI-VITS at $W\in\mathbb{R}^{d\times d}$9 (Feng et al., 2024).

6. Efficiency, misconceptions, and open directions

Efficiency is a first-order concern because a Speech-LLaMA decoder must process long multimodal prefixes. Several papers attack this bottleneck directly. Speech LLaMA for ASR showed that stacking up to 12 frames, corresponding to 960 ms stride, reduces the audio-embedding sequence length by a factor of 96 with only modest WER degradation from $$W' = W + \frac{\alpha}{r}BA,\quad B\in\mathbb{R}^{d\times r},\;A\in\mathbb{R}^{r\times d},$$0 to $$W' = W + \frac{\alpha}{r}BA,\quad B\in\mathbb{R}^{d\times r},\;A\in\mathbb{R}^{r\times d},$$1 (Fathullah et al., 2023). Faster Speech-LLaMA used four latent multi-token heads and threshold-based decoding to reduce decoder calls by approximately $$W' = W + \frac{\alpha}{r}BA,\quad B\in\mathbb{R}^{d\times r},\;A\in\mathbb{R}^{r\times d},$$2, with decoder RTF on LibriSpeech test-other dropping from $$W' = W + \frac{\alpha}{r}BA,\quad B\in\mathbb{R}^{d\times r},\;A\in\mathbb{R}^{r\times d},$$3 to $$W' = W + \frac{\alpha}{r}BA,\quad B\in\mathbb{R}^{d\times r},\;A\in\mathbb{R}^{r\times d},$$4 while improving WER from $$W' = W + \frac{\alpha}{r}BA,\quad B\in\mathbb{R}^{d\times r},\;A\in\mathbb{R}^{r\times d},$$5 to $$W' = W + \frac{\alpha}{r}BA,\quad B\in\mathbb{R}^{d\times r},\;A\in\mathbb{R}^{r\times d},$$6 (Raj et al., 2024). In AVSR, Llama-SMoP replaced a single projector with a Sparse Mixture of Projectors; on LRS3, SMoP-3 DEDR improved WER from approximately $$W' = W + \frac{\alpha}{r}BA,\quad B\in\mathbb{R}^{d\times r},\;A\in\mathbb{R}^{r\times d},$$7 to $$W' = W + \frac{\alpha}{r}BA,\quad B\in\mathbb{R}^{d\times r},\;A\in\mathbb{R}^{r\times d},$$8 for $$W' = W + \frac{\alpha}{r}BA,\quad B\in\mathbb{R}^{d\times r},\;A\in\mathbb{R}^{r\times d},$$9B, $8\times$00 to $8\times$01 for $8\times$02B, and $8\times$03 to $8\times$04 for $8\times$05B, while also improving noise robustness at SNR $8\times$06 dB from $8\times$07 to $8\times$08 in AVSR (Cappellazzo et al., 20 May 2025).

A common misconception is that Speech LLaMA is synonymous with Whisper plus LLaMA. The literature is broader: conformers, HuBERT, WavLM, AV-HuBERT, Q-Former, TLTR, CTC compressors, codec tokenizers, and mixture-of-projector modules all appear as speech interfaces (Fathullah et al., 2023, Xu et al., 2023, Cappellazzo et al., 20 May 2025, Ye et al., 6 Feb 2025). Another misconception is that freezing the LLM is always sufficient. Some papers show strong results with frozen backbones, but others identify residual failure modes: AudioChatLlama reports hallucinations on acoustically similar rare words; LTU-AS notes that its $8\times$09 ASR WER is still above Whisper’s $8\times$10; SECap reports that freezing Q-Former in stage 2 drops SIM$8\times$11 to $8\times$12; and monolithic TTS fine-tuning in TTS-LLaMA severely degrades text QA, motivating MoLE-LLaMA’s late-fusion routing (Fathullah et al., 2023, Gong et al., 2023, Xu et al., 2023, Shen et al., 2024).

The open directions stated across the literature are consistent. They include direct alignment objectives between audio and text, semi- or self-supervised pretraining for audio encoders, longer-context and streaming variants, multimodal extensions with video, and lighter adaptation mechanisms for latency-constrained settings (Fathullah et al., 2023, Wu et al., 2023, Gong et al., 2023). On the synthesis side, direct end-to-end speech QA remains open in MoLE-LLaMA, and broader accent, style, and multilingual adaptation remain active problems for LLaMA-based TTS systems (Shen et al., 2024, Feng et al., 2024). This suggests that Speech LLaMA has evolved from a narrow method for turning LLaMA into an ASR decoder into a general recipe for aligning pretrained speech and language modules, but that compression, grounding, and generation control remain the principal unresolved technical challenges.