Projector-based LLM-ASR Framework
- Projector-based LLM-ASR is a framework that interfaces pretrained speech encoders with large language models via a learnable projector for dimensional matching and temporal compression.
- It bridges the gap between noisy acoustic features and text generation by aligning dense representations to an LLM embedding space with minimal trainable parameters.
- Advanced variants incorporate mixtures of experts, prompt projection, and dynamic downsampling to improve multilingual, multi-domain, and low-resource ASR performance.
Projector-based LLM-ASR denotes a class of automatic speech recognition systems in which a pretrained speech encoder is connected to a pretrained LLM through a learnable projector that maps acoustic representations into the LLM input space. In this paradigm, the projector is the central speech-language interface: it performs dimensional matching, usually performs temporal compression, and determines how much of the encoder and LLM can remain frozen while still enabling transcription. The framework appears in minimal forms such as SLAM-ASR, where only the projector is trained, and in more elaborate multilingual, multi-domain, and task-specific systems that augment the projector with mixtures of experts, dynamic downsampling, prompt-side projection, or auxiliary supervision (Ma et al., 2024, Lin et al., 9 Jun 2026, Fong et al., 7 Aug 2025).
1. Architectural definition and consolidation
The canonical projector-based LLM-ASR pipeline has three blocks: a speech encoder, a projector, and a decoder-only or autoregressive LLM. In SLAM-ASR, this was made explicit as a frozen speech foundation encoder, a small trainable projector, and a frozen decoder-only LLM, with only the projector updated during ASR training (Ma et al., 2024). Closely related formulations recur across later work: a frozen Whisper-large-v3-turbo encoder plus a lightweight projector and a frozen multilingual LLM in low-resource ASR (Fong et al., 7 Aug 2025); a pretrained Whisper encoder, projector, and Qwen-2.5 7B with the encoder and LLM frozen in multilingual ASR (Lin et al., 9 Jun 2026); and Chinese encoder-projector-LLM stacks using Whisper or HuBERT, a projector, and Atom-7B or Baichuan2-7B-Chat (Geng et al., 2024).
At the level of sequence modeling, the projector outputs are treated as token-like embeddings that condition autoregressive generation. Training is typically standard next-token cross-entropy on transcripts conditioned on projected speech embeddings and, often, textual prompts. This design collapses the classical acoustic-model/language-model split into a single interface problem: the speech encoder supplies dense acoustic states, while the projector learns how to place those states into regions of the LLM embedding space that support accurate text generation (Ma et al., 2024, Geng et al., 2024).
Two design tendencies emerged. One treats the projector as the only trainable module, relying on frozen foundation models for both acoustics and language. The other keeps the basic encoder-projector-LLM decomposition but adds targeted trainable components such as LoRA inside the LLM, prompt projectors, or expert routing inside the projector. The contrast between these tendencies defines much of the subsequent literature: simplicity works surprisingly well in monolingual settings, whereas multilingual, multi-domain, and atypical-speech settings often require richer projector behavior (Ma et al., 2024, Burdisso et al., 28 Jan 2026, Lin et al., 9 Jun 2026).
2. Core mechanics of projection and modality alignment
A standard formalization writes the speech encoder output as
followed by a projector that maps acoustic features into the LLM space,
after which the LLM models transcript tokens autoregressively conditioned on and a prompt. In SLAM-ASR, the encoder output at about 50 Hz is downsampled by concatenating every consecutive frames, and a two-layer MLP with a ReLU hidden layer maps the resulting vectors to the LLM embedding dimension (Ma et al., 2024). A closely related low-resource formulation uses the same pattern: frozen Whisper-large-v3-turbo, downsampling by a factor , and a two-layer MLP projector with ReLU (Fong et al., 7 Aug 2025).
Other works vary the projector while preserving the same interface role. A Chinese LLM-ASR study compared a Q-former projector and a 4-layer Transformer projector, with a final linear layer mapping projector outputs to the LLM embedding space; the Transformer projector performed markedly better on Mandarin benchmarks (Geng et al., 2024). Another adaptation-oriented framework used a single linear projector on top of downsampled WavLM-Large features, explicitly framing the projector as the bridge from acoustic space to the LLM embedding space (Bañeras-Roux et al., 7 Apr 2026). A prompt-sensitivity study retained the speech projector as a two-layer MLP but added a second projector on the prompt side, learning to map prompt embeddings themselves to more effective regions of the LLM input space (Burdisso et al., 28 Jan 2026).
Temporal compression is a recurrent difficulty because speech encoders emit long frame sequences whereas LLMs operate on much shorter token sequences. Fixed downsampling is the simplest answer, but not the only one. In multilingual Qwen-2.5-based ASR, a baseline projector used two convolutional layers and two linear layers with fixed downsampling factor 4, while the proposed projector replaced the linear layers with Mixture-of-Experts layers and replaced fixed downsampling with a Continuous Integrate-and-Fire mechanism. There, the CIF predictor was pretrained with
and a modified target length was used to avoid overcompression (Lin et al., 9 Jun 2026).
This body of work suggests that a projector has three technically distinct obligations. It must compress time, map feature dimensions, and regularize the geometry of the speech representation so that the frozen or lightly adapted LLM can interpret it as a meaningful prefix. Systems differ in which of these obligations they assign to fixed downsampling, convolution, explicit routing, or prompt-side compensation, but the underlying interface problem remains the same (Ma et al., 2024, Lin et al., 9 Jun 2026, Burdisso et al., 28 Jan 2026).
3. Adaptation, prompts, and the speech-text gap
A central issue in projector-based LLM-ASR is the modality gap between clean text embeddings and the noisy or off-manifold representations produced by the speech projector. Text-only adaptation papers make this explicit: naive LLM fine-tuning on target-domain text improves pure text modeling but disrupts the alignment that the projector previously learned, because the LLM is no longer optimized to interpret projector outputs as noisy text-like inputs (Burdisso et al., 28 Jan 2026). To address this, one method recasts adaptation as text denoising, training the LLM to recover clean transcripts from noisy inputs that imitate projector outputs; this yields up to 22.1% relative improvement and outperforms recent text-only adaptation baselines while requiring no architectural changes or additional parameters (Burdisso et al., 28 Jan 2026).
A related study asked whether small amounts of target-domain speech can repair this mismatch more efficiently than conventional ASR fine-tuning. It compared text-only adaptation, paired speech-text adaptation, and mixed batching. The main empirical finding is that mixed batching using only 10% of the target-domain speech, less than 4 hours, achieves word error rates comparable to, or better than, conventional ASR fine-tuning with the full dataset, indicating that small amounts of speech provide a strong modality-alignment signal (Bañeras-Roux et al., 7 Apr 2026). This directly reframes projector-based adaptation: the scarce resource is not only paired labels, but exposure to the projector’s speech-conditioned embedding distribution.
Prompting is another nontrivial component of the framework. A comprehensive prompt study showed that prompt choice significantly affects ASR performance and introduces instability, with no single prompt performing best across all cases (Burdisso et al., 28 Jan 2026). The same work proposed a prompt projector, architecturally matched to the speech projector, that transforms prompt token embeddings before they enter the frozen LLM. Across four datasets, the prompt projector consistently improved performance, reduced variability, and outperformed the best manually selected prompts (Burdisso et al., 28 Jan 2026). This is a conceptual extension of the projector idea: once ASR is viewed as geometry in LLM input space, both speech embeddings and prompt embeddings become objects that can be learned as projections rather than fixed tokens.
Taken together, these results counter two recurrent misconceptions. First, projector-based ASR is not merely a matter of attaching a speech front-end to a text model; cross-modal alignment can be degraded by seemingly benign text-only adaptation (Burdisso et al., 28 Jan 2026, Bañeras-Roux et al., 7 Apr 2026). Second, prompting is not a superficial implementation detail; prompt embeddings participate in the same alignment problem as speech embeddings and can require their own learned projection (Burdisso et al., 28 Jan 2026).
4. Multilingual and multi-domain expert projectors
Multilingual ASR exposed a limitation of the monolithic projector: a single mapping struggles to capture both shared and language-specific acoustic-to-semantic correspondences. One multilingual Qwen-based system therefore introduced a projector-internal Mixture-of-Experts with experts, matching the number of training languages, and paired it with CIF-based dynamic downsampling. On 1500 hours of training data, the baseline LLM-ASR system scored 23.26% WER on MLCSLM-dev, 19.57 on CommonVoice, and 13.05 on FLEURS; adding the MoE projector reduced MLCSLM-dev to 16.10, and the full proposed system with modified CIF further improved results to 15.27, 13.87, and 10.46, respectively (Lin et al., 9 Jun 2026).
The same pattern appears in other multilingual studies but with different stabilization strategies. SMEAR-MoE, built on Whisper large-v3 and Gemma-2-9B, replaced a single projector with a shared convolutional downsampler plus multiple MLP experts and stabilized routing through soft parameter merging. Across four Indic languages, it delivered up to a 7.6% relative WER reduction over the single-projector baseline while maintaining comparable runtime efficiency, and routing analysis showed linguistically meaningful specialization in which related languages shared experts (Pandey et al., 27 Jan 2026). MOSA pursued a closely related objective with mixtures of simple adapters rather than one complex adapter; MOSA-Base achieved a 15.4% relative reduction in average WER compared to Baseline-Base, consistently outperformed it across all languages, and surpassed Baseline-Base even when trained with only 60% of its parameters (Li et al., 26 Aug 2025).
Projector specialization is not restricted to multilinguality. In multi-accent Mandarin ASR, HDMoLE applied hierarchical routing and dynamic thresholds to LoRA experts inserted into a 4-layer Transformer projector, keeping HuBERT and Baichuan2 frozen. Applied to the projector module, HDMoLE achieved similar performance to full fine-tuning in the target multi-accent domains while using only 9.6% of the trainable parameters required for full fine-tuning and with minimal degradation in the source general domain (Mu et al., 2024). In child-adult ASR, a Classifier-based Domain Router with Mixture-of-Projectors and Mixture-of-LoRAs used coarse-to-fine domain labels and entropy-aware routing to handle both adult and child speech; the framework improved child ASR while preserving adult performance (Shi et al., 9 Jun 2026).
These systems collectively suggest that, once the projector becomes the primary trainable interface, it is a natural location for domain decomposition. The shared encoder can capture universal acoustics, while the projector carries domain-, language-, or accent-specific specialization. This suggests that multilingual robustness in projector-based LLM-ASR depends less on ever larger monolithic adapters than on controlled mixtures of small, interpretable ones (Lin et al., 9 Jun 2026, Pandey et al., 27 Jan 2026, Li et al., 26 Aug 2025, Mu et al., 2024, Shi et al., 9 Jun 2026).
5. Specialized variants and neighboring interfaces
Projector-based LLM-ASR has also been extended to settings where the speech-language interface itself becomes part of the task definition. In multi-talker ASR, one line of work applies Serialized Output Training, in which transcriptions from multiple speakers are concatenated according to the emission times of their speech. An LLM-based SOT approach for multi-talker ASR, leveraging a pretrained speech encoder and LLM, surpasses traditional AED-based methods on LibriMix and achieves state-of-the-art performance on the evaluation set of AMI, outperforming an AED model trained with 1000 times more supervised data in previous works (Shi et al., 2024). A related encoder-only system adapts an LLM to multi-talker conditioning during training and distills its semantic guidance into an encoder with serialized CTC and talker-count routing; on LibriMix it achieves comparable performance to LLM-based systems in the two-talker condition and significant improvements in the three-talker condition with small real-time factor (Shi et al., 11 Mar 2026).
Low-resource ASR clarifies the data requirements of the projector itself. In a SLAM-ASR study with Whisper-large-v3-turbo and multilingual LLMs, training the projector from scratch on Italian required around 200 hours of labeled speech to beat Whisper-large-v3-turbo on in-domain Common Voice Italian, while out-of-domain FLEURS Italian remained substantially harder (Fong et al., 7 Aug 2025). The same study showed that mono- or multilingual projector pretraining on high-resource languages substantially reduced data scarcity effects; for Galician with only 10 hours of fine-tuning data, a multilingual pretrained projector reduced FLEURS WER from 43.0% to 19.4 (Fong et al., 7 Aug 2025). By contrast, a direct comparison of interfaces found that on low-resource Tatar a phoneme-based interface substantially outperformed the vanilla projector, with 17.44 test WER for the phoneme-based system versus 33.58 for the projector-based system, while a phoneme-informed hybrid projector improved over the vanilla projector in English (Li et al., 10 Apr 2026). This suggests that discrete interfaces can be more data-efficient when continuous acoustic-to-LLM alignment is weakly supervised.
Task-specific projector variants further expand the framework. For stuttered Mandarin speech, an LLM-driven ASR-SED architecture used a speech projector, CTC-generated soft prompts, and a stutter-embedding projector from the SED branch to the LLM, reducing CER to 5.45 and achieving an average SED F1-score of 73.63 (Huang et al., 28 May 2025). For privacy-constrained training with synthetic speech, a SLAM-ASR study localized real-versus-synthetic separability to early and middle LLM layers and showed that combining a layer-selection module with room impulse response augmentation matched a fully real-data baseline using only 25% of the real speech, 13.6 hours, and surpassed it at all higher proportions (Labrak et al., 27 Jun 2026).
These variants indicate that “projector-based LLM-ASR” is not a single architecture but a design family. The common invariant is the learnable speech-language interface; the differences arise from whether that interface must encode multilingual structure, multiple speakers, speech pathology, synthetic-real domain shifts, or low-resource transfer.
6. Empirical picture, recurring misconceptions, and open problems
The empirical record is bifurcated. In monolingual benchmark settings, very simple projector-based systems are already competitive with or superior to more elaborate alternatives. SLAM-ASR, using a frozen speech encoder, a frozen decoder-only LLM, and only a trainable projector, achieves 1.9/3.8 WER on LibriSpeech test-clean/test-other and outperforms the latest LLM-based audio-universal model trained on massive pair data (Ma et al., 2024). A Chinese encoder-projector-LLM study likewise shows that a carefully chosen encoder, Transformer projector, and three-stage training schedule can yield state-of-the-art performance on AISHELL-1, Test_Net, and Test_Meeting (Geng et al., 2024). These results directly rebut the claim that projector-based LLM-ASR requires elaborate cross-attention connectors or extensive LLM fine-tuning in every setting.
Yet multilinguality, domain shift, and adaptation expose the opposite failure mode: the simple monolithic projector becomes a bottleneck. Multilingual MoE projectors, mixtures of simple adapters, hierarchical LoRA experts inside projectors, and domain-routed mixtures all outperform single-projector baselines precisely because one global mapping cannot easily represent shared and domain-specific structure simultaneously (Lin et al., 9 Jun 2026, Pandey et al., 27 Jan 2026, Li et al., 26 Aug 2025, Mu et al., 2024, Shi et al., 9 Jun 2026). This counters a second misconception, namely that projector design is secondary once encoder and LLM are strong enough.
Several limitations recur across the literature. Projectors can be data-hungry in low-resource conditions and remain weaker than strong Whisper baselines out of domain when trained from scratch (Fong et al., 7 Aug 2025). Modality mismatch remains a persistent adaptation hazard, so text-only tuning without explicit alignment preservation is brittle (Burdisso et al., 28 Jan 2026, Bañeras-Roux et al., 7 Apr 2026). Prompt sensitivity is real enough that learned prompt projection can outperform manual prompt engineering (Burdisso et al., 28 Jan 2026). In synthetic-speech training, reducing representation-level separability between real and synthetic speech helps but does not directly predict downstream ASR gains, so geometry-based analyses must still be validated by WER (Labrak et al., 27 Jun 2026). Multilingual expert systems introduce their own fragilities, including routing uncertainty, expert collapse, domain-label dependence, and sensitivity to downsampling strategy (Lin et al., 9 Jun 2026, Shi et al., 9 Jun 2026).
A plausible implication is that the projector is gradually becoming the principal research object in LLM-based ASR. Early work treated it as a thin dimensional adapter; later work treats it as the site of routing, temporal alignment, prompt control, and domain adaptation. The resulting picture is not that projector-based LLM-ASR converges on one universally optimal architecture, but that the projector formalizes where speech-specific structure enters an otherwise text-pretrained system. The framework’s future, accordingly, is likely to depend on better controlled mixtures, stronger alignment objectives under scarce speech, and interfaces that remain simple enough to preserve the modularity that made the paradigm attractive in the first place (Ma et al., 2024, Bañeras-Roux et al., 7 Apr 2026, Labrak et al., 27 Jun 2026).