Multi-LLM Knowledge Fusion Techniques

Updated 3 July 2026

Multi-LLM knowledge fusion is a set of methods that combine multiple pre-trained or fine-tuned language models into one system, overcoming architectural and vocabulary differences.
Fusion strategies employ techniques like distribution alignment, embedding fusion, parameter-space merging, and test-time weighting to balance efficiency, accuracy, and specialization.
Empirical results show that these methods can significantly boost reasoning, code generation, and safety metrics compared to standalone models or traditional ensembles.

Multi-LLM knowledge fusion is a suite of methodologies enabling the aggregation of capabilities, knowledge, or alignment traits from multiple pre-trained or fine-tuned LLMs into a single target system. Fusion strategies operate at different abstraction levels—ranging from output distribution alignment, embedding/intermediate representation fusion, parameter-space operations, to ensemble routing—addressing the heterogeneity of model architectures, vocabularies, and functional domains. Research in this area targets improvements over individual models and standard ensembles, showing measurable gains in reasoning, code generation, retrieval, honesty, safety, and more, by leveraging LLM complementarity (Dai et al., 2024, Wan et al., 2024, Kong et al., 28 May 2025, Wan et al., 2024, Wan et al., 2024, Gao et al., 4 Jun 2025, Yan et al., 6 Jan 2025, Mavromatis et al., 2024, Tekin et al., 2024, Zhang et al., 2024).

1. Formalizations and Motivations

The primary goal of multi-LLM knowledge fusion is to combine specialized or generalist models (possibly with heterogeneous architectures and tokenizers) into a single deployable system that exceeds the abilities of any constituent part. Formally, given $K$ source LLMs $\{M^S_j: \theta_j\}_{j=1}^K$ and a target LLM $M^T: \theta^T$ , fusion aims to update $\theta^T$ such that $M^T$ internalizes the generative, retrieval, or alignment behaviors of the sources. Diverse motivations include:

Overcoming Architectural Incompatibility: Direct parameter merging is typically infeasible for models with different structures or vocabularies; fusion at the distributional, embedding, or output level is thus emphasized (Wan et al., 2024, Kong et al., 28 May 2025, Wan et al., 2024).
Efficiency: Ensembles incur $O(K)$ resource overhead at inference, motivating fusion methods that yield a single target (Kong et al., 28 May 2025, Wan et al., 2024).
Task Generality and Specialization: Fused models seek to aggregate domain- or alignment-specific strengths, addressing knowledge redundancy and task interference present in naive merges (Tekin et al., 2024, Kong et al., 28 May 2025).

2. Distributional and Representation-Level Fusion

A dominant paradigm involves aligning and fusing the output probability distributions or high-dimensional embedding representations from diverse LLMs.

Distribution Fusion: FuseLLM (Wan et al., 2024) aligns token sequences and vocabularies (via minimum-edit distance) across heterogeneous sources, then defines a fusion objective—either MinCE (minimum cross-entropy selection per token) or AvgCE (weighted averaging)—to guide continual training of the target via a convex combination of standard language modeling and distribution-matching losses.
Adaptive Fusion: Fusion-X (Kong et al., 28 May 2025) extends this by using a neural selector (ASN) to adaptively choose and weight source models per input, incorporating a feedback-driven loss to prevent expert selection collapse.

Method	Distribution Alignment	Fusion Rule	Weighting
FuseLLM	MinED vocab/token alignment	MinCE / AvgCE	Static, per-distrib.
Fusion-X	MinED + ASN selector	Weighted sum	Adaptive (ASN/softmax)

Embedding/Similarity Fusion: In retrieval, LLM-KnowSimFuser (Dai et al., 2024) fuses similarity matrices from multiple embedding models, after normalization, via weighted linear combination calibrated on validation data.

3. Parameter-Space and Post-Hoc Merging

Pairwise distillation followed by parameter merging has proven effective for both task and domain aggregation.

FuseChat/FusionChat (Wan et al., 2024, Wan et al., 2024): First distill each source into a pivot architecture via output distribution fusion, then merge resulting models in parameter space. Merging uses “Variation Ratio Merge” (VaRM) or SCE (Select/Calculate/Erase) matrix-wise schemes, assigning per-matrix weights proportional to parameter update magnitude and applying sign-consistency rules to limit destructive interference.
InfiFusion (Yan et al., 6 Jan 2025): Proposes two strategies:
- Pairwise Fusion: Sequentially distill each source into the pivot, then merge via RSAF (skewness- and variation-adaptive selection of top-K parameter differences).
- Unified Fusion: Aggregate source logits via uncertainty-weighted convex combinations, distilling all sources simultaneously.
H³Fusion (Tekin et al., 2024): Utilizes mixture-of-experts (MoE) layers in transformer FFNs. Each expert is an individually aligned FFN (helpful, harmless, honest), and a sparse-but-adaptive router dynamically selects experts per token. Freezing core parameters and only tuning routers (plus light FFN regularization) enables effective behavioral fusion even on highly aligned datasets.

4. Test-Time and Retrieval Fusion

Patterned toward in-situ, “plug-and-play” fusions, test-time methods avoid retraining and yield dynamic expert weighting:

PackLLM (Mavromatis et al., 2024): At inference, combine arbitrary LLMs by assigning softmax or sequentially optimized weights proportional to negative perplexity on input prompts. Empirically, this surpasses uniform ensembles and static retrieval-based selectors.
Retrieval Fusion: LLM-KnowSimFuser (Dai et al., 2024) constructs per-model similarity matrices (e.g., via normalized inner product/cosine), followed by normalization and weighted combination, ranking candidates by fused scores. Grid search is used to determine static fusion weights and ablations confirm modularity gains.

5. Data Generation, Domain Balancing, and Knowledge Editing

Recent approaches automate domain discovery, data allocation, and address catastrophic forgetting in fusion.

Bohdi (Gao et al., 4 Jun 2025): Introduces a synthetic-data-only fusion process over a hierarchical domain tree. A multi-armed bandit (with DynaBranches) adaptively samples/explores domains, harvesting QA data via collaborative prompting and leader selection among sources. Introspection-Rebirth (IR) uses sliding-window likelihood ratio testing to reset bandit posteriors when target capabilities drift. This induces balanced knowledge coverage and high data efficiency (<2% of data volume of SFT baselines) and closes performance gaps across domains.
Sequential Fusion (Zhang et al., 2024): Decomposes multi-LLM fusion into (1) relation extraction by a general LLM for structured knowledge acquisition, and (2) prompt- or in-context-based knowledge editing (IKE) for a specialized domain LLM. This method achieves substantial accuracy gains over LoRA, RAG, and memory-based alternatives without catastrophic forgetting.

6. Empirical Results and Performance Gains

Empirical validation demonstrates consistent advantages over baselines, with fusion strategies outperforming single models, static ensembles, and traditional distillation:

Retrieval: In the KDD 2024 AQA Challenge, LLM-KnowSimFuser achieved a score of 0.20726, +0.018 over the best single model (Dai et al., 2024).
Reasoning and Code: FuseLLM reported 41.8% BBH accuracy (compared to 40.4% for classical LM), 64.56% (CS) and 15.56% (MultiPL-E), with up to +6.3% relative improvements (Wan et al., 2024).
Flexible Selection: Fusion-X improved average accuracy over BBH and MMLU by 5.3% and 6.4% versus continued training, halved knowledge interference, and reduced training steps by 50% (Kong et al., 28 May 2025).
Chat Fusion: FuseChat-7B achieved MT-Bench scores of 7.38 (vs 6.88 base), on par with Mixtral-8x7B-MoE and approaching GPT-3.5-Turbo (Wan et al., 2024). VaRM outperformed SLERP, TIES, and linear fusion in merging.
Synthetic Data Fusion: Bohdi obtained +8.83 and +7.60 pp average accuracy improvements on Llama3.2-3B and Gemma2-9B-IT, respectively, versus SFT or past fusion frameworks, with near-perfect domain balance using ~1.8k synthetic QA pairs (Gao et al., 4 Jun 2025).
Safety and Alignment: H³Fusion-MoE outperformed the best single alignment model by +11.37% and the best ensemble by +13.77% on helpfulness, with additional gains in safety and honesty (Tekin et al., 2024).
Test-time Gains: PackLLM-opt achieved up to +1.89 pp and +3.92–11.94 pp improvements over uniform ensemble and learned fusers, respectively, across 100+ LLMs (Mavromatis et al., 2024).

7. Limitations, Open Problems, and Future Directions

Several technical and practical limits remain:

Token/Vocabulary Alignment: Robust, efficient alignment remains challenging; methods rely on minimum-edit-distance or statistics-based token mapping, which may not cover semantic drift or new subword splits (Wan et al., 2024, Wan et al., 2024).
Computational Cost: Storing and computing many embedding or output spaces incurs significant memory and disk costs (noted for similarity-fusion and parameter-merge approaches) (Dai et al., 2024).
Manual Tuning and Static Weights: Most frameworks use grid search, manual thresholding, or static adaptation; there is active research on query- or token-adaptive weighting and gating networks (ASN, MoE, feedback loss) (Kong et al., 28 May 2025, Tekin et al., 2024).
Interference and Generalization: Fusion can degrade performance if weak sources are included or domain/task discrepancy is high. Handling catastrophic forgetting and ensuring cross-task stability remains nontrivial (Wan et al., 2024, Kong et al., 28 May 2025).
Scalability and Extensibility: Merging more than a handful of experts, scaling to cross-modal or sparse-expert architectures, and dynamically selecting candidate pools under hardware constraints are unsolved (Kong et al., 28 May 2025, Wan et al., 2024).
Data Generation and Hallucination: Reliance on synthetic QA, knowledge extraction, or leader-picking can induce domain bias or hallucinations, especially for poorly represented areas (Gao et al., 4 Jun 2025, Zhang et al., 2024).

Open directions include dynamic or per-input fusion weight learning, end-to-end differentiable objective integration, enhanced intermediate feature alignment (beyond output logit fusion), automated prompt/tag optimization, and integration with external retrieval systems for grounded generation or knowledge update.

References:

(Dai et al., 2024) Advancing Academic Knowledge Retrieval via LLM-enhanced Representation Similarity Fusion
(Wan et al., 2024) Knowledge Fusion of LLMs
(Kong et al., 28 May 2025) Enabling Flexible Multi-LLM Integration for Scalable Knowledge Aggregation
(Wan et al., 2024) FuseChat: Knowledge Fusion of Chat Models
(Wan et al., 2024) Knowledge Fusion of Chat LLMs: A Preliminary Technical Report
(Gao et al., 4 Jun 2025) Bohdi: Heterogeneous LLM Fusion with Automatic Data Exploration
(Yan et al., 6 Jan 2025) InfiFusion: A Unified Framework for Enhanced Cross-Model Reasoning via LLM Fusion
(Mavromatis et al., 2024) Pack of LLMs: Model Fusion at Test-Time via Perplexity Optimization
(Tekin et al., 2024) $H^3$ Fusion: Helpful, Harmless, Honest Fusion of Aligned LLMs
(Zhang et al., 2024) General LLMs as Instructors for Domain-Specific LLMs: A Sequential Fusion Method to Integrate Extraction and Editing