Interactive Language Model Fusion

• Interactive language model fusion is a dynamic integration technique that adaptively combines outputs from heterogeneous models based on user input and context.
• It leverages query-driven gating, cross-attention mechanisms, and user-driven feedback to enhance performance in tasks like dialogue systems, VQA, and speech recognition.
• Empirical results demonstrate significant improvements in metrics such as AUC, BLEU, and WER, showcasing its potential in speech, vision, language, and neural interface applications.

Interactive LLM fusion refers to methodologies that combine the outputs, internal representations, or decisions of multiple LLMs—or LLMs alongside other modalities—in a dynamic, context-driven, or user-interactive manner. This paradigm extends beyond classical fusion techniques (e.g., shallow fusion in speech recognition) by enabling the fusion mechanism itself to adapt to downstream tasks, explicit instructions, or ongoing interaction history. The goal is to leverage complementary strengths of heterogeneous models or modalities for continuous, robust, and context-aware multimodal reasoning and generation across domains such as dialogue systems, machine translation, VQA, neural decoding, and autonomous control.

1. Foundational Principles and Taxonomy

Interactive LLM fusion is characterized by several foundational properties:

• Dynamic Modality Integration: Fusion decisions and weighting are determined dynamically at inference, often as a function of external queries or interaction context, rather than statically combining model predictions.
• Heterogeneous Model Fusion: The constituent models can differ in architecture (causal vs. encoder-decoder), modality (text, audio, vision, neural), or vocabulary/tokenization.
• User/Task Guidance: Inputs such as user instructions, questions, or interaction history modulate the fusion process itself, dictating which modalities or model outputs are emphasized for a given task or user query.
• Granularity of Coupling: Fusion may occur at feature, token, segment, or sequence level, and may involve logic ranging from simple ensembling through voting/scoring to joint latent-space embedding and cross-attention.

A high-level taxonomy emerges:

Fusion Granularity	Adaptivity Level	Modalities	Example
Token/Segment	Fixed/stateless	Homogeneous	Cool-Fusion (Liu et al., 2024)
Embedding/Hidden	Context-driven	Multimodal	MMDrive (Hou et al., 15 Dec 2025)
Autoregressive Latent	Instruction-tuned	Cross-modal/Neural	CorText (Bosch et al., 28 Sep 2025)

2. Model Architectures and Algorithms

Several representative methodologies exemplify interactive LLM fusion:

2.1 Late-Fusion with Task-Driven Adaptation

Wang et al. (Wang et al., 2024) propose a late-fusion framework for spoken dialogue state prediction. Here, a frozen acoustic model (HuBERT) and a LLM (GPT2 or RedPajama) each encode separate modalities into fixed-dimensional embeddings, which are concatenated and passed through a learned classifier. Multi-task instruction fine-tuning reframes the output as binary detection per class (turn-taking, backchannel, or continuing) and supplies explicit instructions as LLM prompts, such that the LLM's encoding is dynamically modulated by both the partial utterance and the task description (see §3). Dialogue history can be injected as part of the prompt for richer context sensitivity.

2.2 Query-Driven Multimodal Reasoning

In MMDrive (Hou et al., 15 Dec 2025), the fusion mechanism itself is dynamically modulated by the natural-language question. The Text-oriented Multimodal Modulator (TMM) computes per-modality fusion weights α based on an embedding of the query, then applies cross-attention between the image and other modality features (LiDAR, occupancy grid, text). The Cross-Modal Abstractor (CMA) further distills the fused context into learnable “abstract tokens,” yielding a highly compact and query-adaptive summary injected directly into the LLM decoder. This architecture is explicitly interactive: the fusion pipeline is tightly coupled to the semantics of the current user query rather than being fixed a priori.

2.3 Iterative, Zero-Shot, and Feedback-Driven Fusion

CorText (Bosch et al., 28 Sep 2025) embeds fMRI-derived neural tokens directly as a prefix to a frozen LLM's token sequence. User instructions modulate the downstream generative task (captioning, VQA) and the model permits direct in-silico experimentation, such as brain stimulation, by manipulating the neural data input and observing changes in generated answers. This approach dissolves the barrier between neural decoding and natural language reasoning, enabling open-ended, context-dependent query-answering based exclusively on neural input.

2.4 Segment-Level Ensembling without Training

Cool-Fusion (Liu et al., 2024) implements a fully training-free fusion of arbitrary causal LLMs (heterogeneous in vocabulary and architecture) by aligning on common word boundaries, generating proposals from each model at each step, and selecting the candidate with lowest average perplexity as judged by all models. This mechanism can operate in live, interactive settings, fusing outputs in response to changing context without retraining or fine-tuning any constituent model.

2.5 Real-Time Full-Duplex Speech-Language Fusion

LSLM (Ma et al., 2024) introduces an architecture for simultaneous speech generation and streaming audio input (full-duplex), enabling interruption and turn-taking. The core innovation is middle fusion: listening and speaking embeddings are fused inside every transformer block, allowing real-time external audio to modulate ongoing TTS generation. Special IRQ tokens mark immediate cessation of speech in response to input, with empirical results showing extremely low WER and high F1 for interruption detection in noisy, online scenarios.

3. Mathematical Formalisms and Training Objectives

Mathematical formulation depends on application context, with general principles:

• Feature-level fusion: Given two modality encoders $f_A, f_L$ producing $h^A, h^L$, late fusion computes $h^{fusion} = [h^A ; h^L]$, with classification $$P(Y|X^A, X^L) = \mathrm{softmax}(W_f h^{fusion} + b_f)$$ (Wang et al., 2024).
• Query-driven gating: For features $F_Q, F_M$, compute fusion weights $\alpha = \mathrm{softmax}(W f_q + b)$ (where $f_q$ is a pooled query embedding), and fuse via $$F_{enh} = F_I + \mathrm{LayerNorm}(\sum_m \alpha_m E_m)$$, with $E_m$ modality-specific attended features (Hou et al., 15 Dec 2025).
• Token-sequence fusion: For interfacing neural tokens $z_1,...,z_N$ and language tokens $h^A, h^L$0, the concatenated sequence $h^A, h^L$1 is fed into a frozen LLM backbone. Optimization is via standard cross-entropy over output sequences for both captioning and QA, with the fusion weights either fixed or adaptively modulated via LoRA adapters (Bosch et al., 28 Sep 2025).
• In training-free ensembling, the optimal next segment is chosen at each step via

$h^A, h^L$2

where $h^A, h^L$3 is perplexity under model $h^A, h^L$4 for candidate $h^A, h^L$5 (Liu et al., 2024).

4. Practical Implementations and Use Cases

• Conversational AI and Speech Agents: Late-fusion and instruction-tuned approaches (Wang et al., 2024, Ma et al., 2024) yield superior turn-taking, backchanneling, and real-time interruption detection in spoken dialogue, leveraging both prosodic and semantic cues.
• Autonomous Driving and VQA: Interactive fusion frameworks with query-adaptive gating achieve robust scene reasoning with multiple modalities (image, LiDAR, occupancy, textual) and significantly outperform traditional vision-LLMs in both language generation and multiple-choice QA (Hou et al., 15 Dec 2025).
• Neural Decoding and Brain-Computer Interfaces: Fusion methods embedding neural activations into LLM latent manifolds permit open-ended, user-guided querying and counterfactual experimentation with neural recordings, achieving zero-shot generalization on held-out semantic categories (Bosch et al., 28 Sep 2025).
• Machine Translation: Attention-based dynamic fusion between NMT models and external LLMs enhances fluency and robustness over conventional fusion schemes, particularly aiding in grammatical adaptation and noise resilience (Kurosawa et al., 2019).
• Model Ensembling and Diversity: Training-free, segment-wise ensembling methods enable on-the-fly fusion of LLM outputs, offering significant accuracy gains in math and QA tasks without requiring aligned vocabularies or finetuning (Liu et al., 2024).

5. Evaluation Metrics and Experimental Insights

Interactive LLM fusion approaches are empirically validated by:

• Task-specific metrics: AUC, EER, and balanced accuracy for dialogue tasks (Wang et al., 2024); BLEU, METEOR, ROUGE-L, and CIDEr for VQA (Hou et al., 15 Dec 2025); CLIPScore for neural decoding (Bosch et al., 28 Sep 2025); WER and F1 for full-duplex speech models (Ma et al., 2024).
• Comparative ablations: Demonstrate that interactive/learnable fusion strategies and instruction-based prompts yield large relative gains over single-modality or static-fusion baselines (+3–22% AUC, +17 pp QA accuracy, or +5–16 pp math task accuracy depending on task). MMDrive's TMM and CMA modules are individually responsible for 4–5 BLEU point improvement over simpler fusion (Hou et al., 15 Dec 2025).
• Robustness analyses: Middle fusion in LSLM preserves generation fidelity and interactive accuracy under varying noise conditions; delayed fusion reduces latency and computation in ASR while maintaining or improving WER (Ma et al., 2024, Hori et al., 16 Jan 2025).

6. Limitations and Future Prospects

Challenges and open research directions highlighted across the literature include:

• Scalability and Efficiency: Segment-level and training-free approaches incur $h^A, h^L$6 inference complexity for $h^A, h^L$7 fused models, though efficient batched and parallelized implementations mitigate runtime (Liu et al., 2024).
• Vocabulary and Tokenization Alignment: Handling heterogeneous vocabularies remains an infrastructural challenge, addressed variously by word-alignment constraints (Liu et al., 2024) or on-the-fly re-tokenization (Hori et al., 16 Jan 2025).
• Generalization and Extension: Modular fusion APIs, learnable fusion modules (e.g., TMM, CMA), and explicit cross-modal attention mechanisms generalize beyond current domains to robotics, biomedicine, and interactive edge applications (Hou et al., 15 Dec 2025).
• Human-in-the-Loop and Multiround Interactivity: Future architectures enabling multi-turn, query-dependent, or feedback-driven adjustment of both model outputs and fusion logic promise richer interactive capabilities (Hou et al., 15 Dec 2025).

Interactive LLM fusion thus provides a principled methodological foundation for dynamically and adaptively combining multiple models or modalities in open-ended tasks, with demonstrated utility in speech, vision, language, and neural interface domains. This paradigm underpins advances in flexible, interpretable, and high-performance AI systems across emerging real-world application scenarios.