---
title: 'FuseChat-3.0: Efficient LLM Fusion'
url: https://www.emergentmind.com/topics/fusechat-3-0
type: topic
---

# FuseChat-3.0: Efficient LLM Fusion

FuseChat-3.0 is a suite of large language models (LLMs) constructed by compactly integrating the expertise of multiple heterogeneous source models into smaller, more efficient targets. Through a dual-stage training methodology that leverages high-quality multi-source data and advanced preference-guided optimization, FuseChat-3.0 achieves state-of-the-art results across a range of benchmarks encompassing instruction following, general knowledge, mathematics, and coding. The approach aims to eliminate the overhead of multi-model inference by “implicitly” fusing the capabilities of large, complementary pretrained models into a single, deployable architecture while retaining most of their diverse strengths [2503.04222].

## 1. Model Selection and Fusion Protocol

FuseChat-3.0 utilizes diverse, high-capacity source LLMs, each excelling in distinct domains:

- **Sources (27–123B parameters):**
  - Gemma-2-27B-it
  - Mistral-Large-Instruct-2407 (≈24B)
  - Qwen-2.5-72B-Instruct
  - Llama-3.1-70B-Instruct

- **Targets (1–9B parameters):**
  - Llama-3.1-8B-Instruct
  - Gemma-2-9B-it
  - Qwen-2.5-7B-Instruct
  - Llama-3.2-3B-Instruct
  - Llama-3.2-1B-Instruct

The core fusion strategy involves:
- **Generating multi-source outputs:** For a curated set of prompts, each source model produces multiple candidate responses.
- **Supervised Fine-Tuning (SFT):** The target LLM is fine-tuned using the highest-reward response from the ensemble for each prompt.
- **Direct Preference Optimization (DPO):** Targets are further refined using intra-model best/worst response pairs, applying preference constraints to distill reward-aligned behaviors observed within each source.

This protocol achieves an “implicit” fusion—transferring complementary expertise of the sources (e.g., translation, code generation) into a single, compact model, thereby preventing inference-time redundancy and computational overhead [2503.04222].

## 2. Data Construction and Preference Scheme

A meticulous data pipeline tailors construction to cover a broad distribution of tasks and languages:

- **Prompt Coverage (total ≈ 160K):**
  - Instruction following: 80,907 prompts (from UltraFeedback, Magpie-Pro-DPO, HelpSteer2, filtering code/math cases)
  - Mathematics: ≈52K (from OpenMathInstruct-2)
  - Coding: 16,005 (from LeetCode and Self-Oss-Instruct-SC2, validated by test cases)
  - Chinese: ≈10K (from Alpaca-GPT4-Zh and Magpie-Qwen2-Pro-Zh, excluding code/math)

- **Response Sampling:**
  - For each prompt-source pair: 5 samples (instruction/math), 8 samples (coding), and Qwen-2.5-72B-Instruct specifically for Chinese.
  - Sampling in vLLM: temperature 0.7–0.8, top-p 0.8–0.95, repetition penalty 1.05.

- **Preference Pair Construction:**
  - Each response is scored by ArmoRM-LLaMA3-8B-v0.1 (instruction) or by rule-based correctness plus reward model for math/coding.
  - SFT: The highest-scoring response per prompt across all sources.
  - DPO: For each source, best/worst intra-model pairs with RM-score gap (0.01–0.1).
  - Final dataset: 94,539 SFT samples, 64,128 DPO pairs.

This controlled, reward-guided data regime supports robust knowledge transfer and suppresses negative reward-style bias [2503.04222].

## 3. Training Mechanism and Optimization

The pipeline comprises two major stages:

#### 3.1. Supervised Fine-Tuning (SFT)
Target models are first optimized toward high-quality outputs with a token-level causal LM objective:
\[
\mathcal{L}_{\mathrm{SFT}}(\theta) = -\mathbb{E}_{(x, y) \sim \mathcal{D}_{\mathrm{SFT}}} \left[ \sum_{t=1}^{|y|} \log p_\theta(y_t \mid y_{<t}, x) \right]
\]
Key hyperparameters (all targets): 3 epochs, batch size 128, max sequence length 2048, optimizer with cosine LR decay and 10% warmup, model-specific learning rates.

#### 3.2. Direct Preference Optimization (DPO)
Preference optimization is applied using a Bradley–Terry policy loss, shaping behaviors through best/worst pairs:
\[
\mathcal{L}_{\mathrm{DPO}}(\theta) = -\mathbb{E}_{(x, y^+, y^-) \sim \mathcal{D}_{\mathrm{DPO}}} \left[ \log \sigma \left( s_\theta(x, y^+) - s_\theta(x, y^-) \right) \right]
\]
where \( s_\theta(x, y) = \beta \left[ \log \pi_\theta(y \mid x) - \log \pi_{\mathrm{ref}}(y \mid x) \right] \).

A length-normalized DPO variant further corrects verbosity-induced biases:
\[
\mathcal{L}_{\mathrm{LN\text{-}DPO}}(\theta) = -\mathbb{E}_{(x, y^+, y^-)} \left[ \log \sigma \left( \frac{\beta}{|y^+|}\Delta^+ - \frac{\beta}{|y^-|}\Delta^- \right) \right]
\]
with \(\Delta^\pm = \log \pi_\theta(y^\pm \mid x) - \log \pi_{\mathrm{ref}}(y^\pm \mid x)\).

DPO uses a single epoch, batch size 128, maintaining the post-SFT model as reference and per-model β, LR settings [2503.04222].

## 4. Benchmarking and Empirical Analysis

Evaluation spans fourteen standard benchmarks:

| Benchmark        | Base | +SFT | FuseChat-3.0 | Gain vs Base |
|------------------|------|------|--------------|--------------|
| AlpacaEval-2     | 28.3 | 41.3 | 65.4         | +37.1        |
| Arena-Hard       | 28.1 | 38.7 | 58.2         | +30.1        |
| GSM8K (CoT)      | 85.9 | 87.0 | 88.0         | +2.1         |
| HumanEval        | 69.5 | 69.5 | 71.3         | +1.8         |
| Average (14)     | 40.5 | 43.2 | 47.3         | +6.8         |

FuseChat-3.0, when instantiated on Llama-3.1-8B-Instruct, exhibits an average improvement of 6.8 points across all benchmarks, with pronounced gains in instruction-following (37.1 on AlpacaEval-2, 30.1 on Arena-Hard) over the base architecture. Compared to Tülu 3 (Llama-3.1-8B), FuseChat-3.0 achieves 47.3 vs 40.2 (+7.1 overall), especially excelling on instruction-following tasks (+11.4). Length-normalized DPO further yields +1.1 average improvement versus vanilla DPO [2503.04222].

## 5. Architectural and Implementation Considerations

- FuseChat-3.0 preserves the original transformer architectures of its target models, introducing no new layers or modules.
- Parameter counts coincide with targets: 8B, 9B, 7B, 3B, 1B.
- SFT leverages Llama-Factory; DPO employs the alignment-handbook, both via HuggingFace.
- Inference is performed using vLLM.
- Training was conducted on 8–16 A100 GPUs, with each stage requiring approximately one day.

All code, datasets, and model weights are accessible at https://github.com/SLIT-AI/FuseChat-3.0 [2503.04222].

## 6. Analysis, Limitations, and Future Directions

The effectiveness of FuseChat-3.0 is attributed to:
- **SFT-based implicit fusion:** Alignment to high-quality, multi-source outputs.
- **Reward-model-guided DPO:** Intra-model preference pairs reduce style-conformance bias/variance versus inter-model pairings.
- **Length normalization:** Corrects verbosity bias typical in DPO for long outputs.

Limitations and challenges include:
- Minor regression in code-generation for ultra-compact targets, likely due to lower coding data representation.
- Absence of a DPO phase for Chinese language tasks—future work may develop specialized reward models for improved multilingual preference optimization.
- Open questions pertain to scalability across larger numbers of sources, robust multilingual fusion, and generalized multi-task scenarios.

Plausible extensions comprise:
- Incorporation of human-annotated preferences, potentially hybridizing with RLHF protocols.
- Extension of the fusion paradigm to multi-modal domains, including vision and speech.
- Dynamic, reward-weighted source sampling in DPO to refine knowledge transfer efficacy.

A plausible implication is that this two-stage fusion protocol, leveraging carefully constructed multi-source data and intra-model preference feedback, provides a generalizable blueprint for scalable, efficient, and high-performing LLM distillation and fusion [2503.04222].

Source: https://www.emergentmind.com/topics/fusechat-3-0