Speech Encoder Fusion for LLM-based Automatic Speech Recognition

Abstract: Speech-aware LLMs can incorporate speech through pre-trained acoustic encoders that project speech features into the LLM embedding space. While the choice of the speech encoder critically influences performance, different encoders often exhibit complementary strengths, motivating their combination. In this work, we investigate whether fusing multiple pre-trained speech encoders can enhance speech-aware LLMs for automatic speech recognition (ASR). We explore several fusion strategies beyond simple feature concatenation, including learned combinations and Transformer-based fusion architectures, and evaluate them across mono- and multilingual ASR settings as well as diarized speech recognition. Our results indicate that carefully fusing multiple parallel speech encoders improves downstream performance in all scenarios with limited computational overhead.

• The paper introduces fusion architectures that combine multiple pre-trained speech encoders to leverage their complementary strengths for LLM-based ASR.
• The study evaluates methods such as sigmoid gating, multi-head attention, and transformer-based fusion, demonstrating significant improvements in WER and speaker metrics.
• Experimental results show that encoder fusion enhances robustness and accuracy in monolingual, multilingual, and diarized speech scenarios.

Speech Encoder Fusion for LLM-based Automatic Speech Recognition

Introduction and Motivation

The integration of speech modalities into LLMs has become a major focus in speech technology, enabling LLMs to directly process acoustic signals for tasks such as automatic speech recognition (ASR). Conventional speech-LLM architectures employ a single, pre-trained acoustic encoder—often a resource-intensive model trained on broad data distributions—that projects speech features into the LLM's embedding space, with only lightweight adaptors and LoRA-style parameter-efficient finetuning. However, different pre-trained speech encoders, whether large multilingual models or smaller, domain-specific monolingual models, exhibit complementary strengths and weaknesses. Motivated by the observed non-overlapping error patterns of such models, this work systematically studies fusion strategies for leveraging multiple pre-trained speech encoders within a speech-aware LLM, with the hypothesis that careful fusion can exploit their complementary inductive biases, improving robustness and recognition accuracy across monolingual, multilingual, and diarized speech scenarios.

Figure 1: Speech-LLM with encoder fusion of multiple pre-trained speech encoders.

Methods: Encoder Fusion Architectures

The study evaluates several encoder fusion architectures, all of which aggregate the outputs of multiple pre-trained speech encoders before projection into the LLM:

• Feature Concatenation: Baseline method, temporally aligning and concatenating frame-level features from each encoder.
• Sigmoid Gate: Learns a dynamic, frame-wise weighting between encoder streams via a sigmoid gate applied to projected features, facilitating adaptive information flow based on input characteristics.
• Multi-Head Gate: Employs multi-head attention over encoder streams per time step, allowing multiple heads to specialize and dynamically weight encoder contributions depending on acoustic or linguistic context.
• Positional Transformer: Concatenates features, applies linear reduction, and processes with a Transformer to integrate contextual cues across the sequence, learning joint representations.
• Temporal Transformer: Interleaves features from encoders along the temporal axis, applies a Transformer encoder, and downsamples post-attention, emphasizing temporal feature interaction without early fusion.

All fusion architectures ensure the LLM context window length remains tractable and are extensible to more than two encoders.

Experimental Setup

• Encoders: Whisper-large-v3 is used universally; domain-specific encoders include NeLF (Dutch), Wav2Vec2-FT (English), and ECAPA2 (speaker embedding).
• Downsampling: Temporal alignment at 16.7 Hz via stacking or averaging.
• LLMs: Tweety-7B (Dutch) and Llama-3.1-8B (English, multilingual); quantized and finetuned with QLoRA, rank 4, for efficiency.
• Datasets: Dutch: 240h CGN, English: 960h Librispeech, Multilingual: Dutch + LS 360h. Diarized data from Dutch CGN utterances with multiple speakers.
• Evaluation: Reports word error rates (WER), speaker attributed WER (SA-WER), and speaker confusion (Spk-Conf), with tight confidence intervals due to dataset sizes.

Results

Monolingual ASR

Fusion of Whisper (multilingual) with NeLF (Dutch), or with Wav2Vec2-FT (English), consistently outperforms single-encoder baselines. In Dutch ASR, the temporal transformer fusion yields the best WERs (6.8%/8.3%, clean/other), clearly outperforming either constituent model (Whisper: 8.3%/11.5%; NeLF: 7.5%/9.0%). In English, sigmoid gating provides the largest relative improvement among fusions (clean: 2.8%, compared to Whisper: 3.2%), underlining that fusing mature, highly optimized encoders provides compound gains only with sufficiently adaptive fusion mechanisms.

Multilingual ASR

Joint training on Dutch and English demonstrates that fusion layers—especially multi-head gating—enable strong language-dependent adaptation, outperforming both constituent encoders for their respective target language. Whisper+NeLF with multi-head gating achieves 6.5% (NL) and 2.5% (EN) WER, a marked improvement over Whisper alone in Dutch (8.4%) and NeLF alone in English (10.9%).

Diarized Speech Recognition

Combining a speech encoder (NeLF) with a pretrained speaker encoder (ECAPA2) using transformer-based fusion reduces speaker confusion (Spk-Conf) and improves SA-WER. Temporal transformer achieves best performance (SA-WER: 18.1%, WER: 14.5%, Spk-Conf: 3.6%), demonstrating that effective fusion can leverage speaker identity embeddings for superior diarized transcription, even when the underlying ASR encoder captures some speaker-change information on its own.

Inclusion of Decoder Hypotheses

Augmenting the LLM prompt with initial decoder hypotheses from the component ASR models further improves recognition, often exceeding the performance of any speech-only configuration. In Dutch, fused encoders with all decoder predictions achieve a WER of 5.6% (clean), outperforming both NeLF (6.8%) and Whisper (11.3%) standalones. In English, text-only LLMs perform exceptionally well owing to dataset overlap with LLM pretraining, but fusion with decoder predictions still reduces WER compared to speech-only LLMs.

Analysis and Implications

Empirical evidence highlights that fusion architectures outperform naive concatenation, and the choice of fusion method must be matched to the degree of complementarity and maturity of encoders. Transformer-based fusion methods are especially effective in structured settings (e.g., diarization), while gating methods suffice for well-aligned ASR encoders in monolingual settings. The gains are most pronounced when the base encoders possess distinct domains, languages, or training distributions.

Practical benefits include improved deployment flexibility and robustness without significant computational overhead, since encoders operate in parallel. The approaches support generalization across monolingual, multilingual, and diarization tasks, implying potential impact for low-resource or domain-adapted ASR and multi-speaker environments. The results also suggest that speech-LLMs, while still trailing dedicated ASR in some metrics, can be substantially enhanced using encoder fusion, bringing them closer to task-specific systems.

Theoretically, this work establishes that the modality-bridging capacity of LLMs is not limited by the weakest encoder, and that compositional fusion can be systematically designed rather than ad-hoc.

Future Directions

Potential advancements include the integration of more than two encoders or heterogeneous modality encoders (e.g., phoneme-level, prosodic, noise-robust), scaling up with larger LLMs or unquantized settings, and further exploration of fusion mechanisms (e.g., prompt-dependent dynamic mixtures, deeper adaptive attention stacks). Extending the approach to longer context, real-time (streaming) recognition, and broader multimodal understanding is also promising.

Conclusion

Fusing multiple pre-trained speech encoders via learned combination strategies robustly enhances LLM-based ASR performance across languages and tasks. These findings validate careful encoder fusion as a critical design axis for improving the flexibility, adaptability, and accuracy of speech-augmented LLMs, enabling effective transfer of heterogeneous acoustic modeling knowledge and better integration with LLMs for speech understanding.