FusionBench: Unified Fusion Benchmark Framework

• FusionBench is a suite of benchmarks and frameworks designed to standardize evaluation of fusion techniques across deep learning, fusion energy, and data fusion domains.
• It provides unified datasets, interfaces, and evaluation metrics to ensure reproducible, fair comparisons and effective performance assessments.
• FusionBench supports diverse applications from model fusion in deep learning and LLM routing to computational simulations in fusion energy and multi-source video fusion.

FusionBench is a term denoting a family of benchmarks and frameworks designed for systematic evaluation and comparison of fusion methodologies across multiple disciplines, notably in deep learning (model fusion), fusion energy physics (computational simulation), information retrieval (data fusion), and multi-modal data processing. In each context, FusionBench provides standardized datasets, interfaces, evaluation metrics, and in some cases modular infrastructures, enabling fair and reproducible assessment of fusion techniques, their robustness, and performance trade-offs. This article surveys the key variants of FusionBench, detailing their structure, methodological foundations, application domains, and their significance for benchmarking and advancing state-of-the-art fusion strategies.

1. Definition and Core Objectives

FusionBench is conceived as a comprehensive benchmark or framework that systematically evaluates diverse fusion methodologies within a target domain. Its core aims are:

• To provide a unified evaluation platform for fusion approaches, including but not limited to model fusion in machine learning, code or physics module coupling in fusion energy research, routing and capability aggregation in LLMs, and temporal fusion in video analysis.
• To address the absence of consistency and comparability in prior fusion evaluations by offering standardized tasks, well-defined metrics, and documented interfaces.
• To facilitate benchmarking under a spectrum of scenarios, including in-distribution and distribution-shift conditions, and to support assessment across model sizes, scales, and problem complexities.

The term “FusionBench” now refers to several realized benchmarks tailored to their respective domains, offering datasets, experimental protocols, and, in many cases, extensible codebases or documentation for reproducibility and ongoing expansion (2406.03280, 2507.10540).

2. Deep Model Fusion: FusionBench in Multi-Task Learning

The “FusionBench” of (2406.03280) is the first benchmark dedicated to deep model fusion in artificial intelligence, addressing issues related to inconsistent and inadequate evaluation of model fusion methods.

• Task Coverage: Includes 26 tasks in open-vocabulary image classification, scene understanding (segmentation, depth estimation), text classification, and text-to-text generation (e.g., GLUE benchmark tasks).
• Model Pool: Features 74 fine-tuned models of diverse architectures (e.g., CLIP-ViT for images, ResNet-50, GPT-2, Flan-T5 for NLP), with both full and LoRA fine-tuning.
• Fusion Techniques: Categorizes techniques as follows:

| Category | Example Methods | |--------------------------|---------------------------------------------------------------------------------------------| | Model Ensemble | Simple Ensemble, Weighted Ensemble, Max-Model Predictor | | Model Merging | Simple Average/ModelSoups, Weighted Average, Fisher Merging, RegMean, Task Arithmetic | | Model Mixing | Depth Upscaling, MoE-based Upscaling/Merging, Model Recombination |

• Methodological Features: Each fusion method is formally specified (e.g., merging parameters θ via θ = A(θ₁, θ₂, …, θ_N; w)), and evaluation metrics are domain-appropriate (accuracy, mIoU, RMSE, and Spearman’s ρ).
• Robustness and Scalability: The benchmark enables systematic analysis of robustness under distribution shift, cost-efficiency, and task transferability.
• Documentation and Expansion: Accompanied by detailed documentation, a modular CLI, YAML configuration files, and a public codebase for extensibility and reproducibility.

FusionBench thus establishes standards for comparison and scientific progress in deep model fusion research (2406.03280).

3. LLM Capability Fusion and Routing: FusionBench and FusionFactory

FusionBench as defined in (2507.10540) targets multi-model LLM capability fusion, focusing on leveraging routing data from diverse model deployments.

• Dataset Characterization: 14 tasks covering Math, Code, Commonsense Reasoning, World Knowledge, Reading Comprehension, and a “Popular” challenge set; responses from 20 open-source LLMs (8B–671B parameters); 103M tokens in total.
• Fusion Levels in FusionFactory Framework:

1. Query-level Fusion: Trained routers dynamically select the optimal LLM per query, integrating considerations of performance, token cost, and LLM-judge (secondary evaluation) scores through an explicit optimization:

   $$\phi^* = \arg\max_\phi \mathbb{E}_{(q,t)\sim\mathcal{D},\,m\in\mathcal{M}} \left[\mathrm{Reward}(f_\phi(q,t,m))\right]$$

2. Thought-level Fusion: Aggregation and summarization of top-k LLM responses for reusable “thought templates,” which guide new answer generation via few-shot prompting, enhancing reasoning consistency.
3. Model-level Fusion: Knowledge distillation or supervised fine-tuning, where a base model is trained on high-quality responses from multiple LLMs, although gains are often more modest due to distillation challenges.

• Findings: Thought-level fusion produced the most robust performance improvements, while query-level fusion offered an effective trade-off between efficiency and accuracy. Model-level fusion was more variable in benefit. FusionFactory consistently outperformed the best individual LLM across all tasks.
• Annotation and Analysis: Routing data is meticulously labeled with performance, cost, and LLM-judge ratings, illuminating complementary model strengths and guiding both fusion and future LLM system design (2507.10540).

4. FusionBench in Scientific Computing and Fusion Energy Simulation

Within computational fusion research, “FusionBench” frequently appears as shorthand for benchmarking platforms or canonical test suites for fusion device simulations and submodule coupling.

• Framework Integration: The FACETS framework (1004.1611) is emblematic, providing:
  • Standardized component interfaces (for core, edge, wall, etc.), enabling low-latency, tightly synchronized MPI-based coupling of heterogeneous physics modules.
  • Hierarchical, dynamically composed simulation “wiring” via XML-based input files for flexible processor allocation and component interaction.
  • Emphasis on reproducibility, test suites, standardized output formats (e.g., VizSchema/HDF5), and modular support for legacy and new codes.
• Application in FusionBench: Features and practices in FACETS are cited as foundational for high-fidelity benchmarking (in a “FusionBench setting”), including cross-checks of accuracy and scalability for coupled plasma transport, edge interactions, and wall physics.
• Monte Carlo Fusion Reactivity Evaluation: In reaction physics, FusionBench denotes Monte Carlo evaluation frameworks capable of computing fusion reactivity for arbitrary ion velocity distributions via six-dimensional integrals (2302.09753). Methods implemented include:
  • Direct sampling (O(N) scaling)
  • Complete pairings for small N (O(N²))
  • Importance sampling with weight correction
  • Rigorous benchmarking against analytical cases demonstrates reliability at under 1% error for N~10⁴ samples.
• Deterministic Neutronics Modeling: Recent work (2411.16369) develops deterministic, discontinuous-Galerkin neutron transport solvers, reporting benchmarking results against analytical solutions and Monte Carlo simulations, with direct relevance for reactor optimization under “FusionBench” protocols.
• Nuclear Data Benchmarks: The FENDL library (2311.10063) represents the state of the art in evaluated nuclear data for fusion, underpinning neutronics benchmarks and enabling trustworthy FusionBench assessments of shielding, activation, and breeding performance.

5. Video and Data Fusion: Benchmarking Temporal and Multi-Source Integration

Recent extensions of the FusionBench concept appear in video fusion and information retrieval:

• Video Fusion Benchmark (VF-Bench): (2505.19858) introduces VF-Bench for multi-frame, temporally coherent video fusion, spanning multi-exposure, multi-focus, infrared-visible, and medical fusion tasks. The benchmark provides curated, well-aligned video data and unified protocols quantifying both spatial quality (VIF, SSIM, MI) and temporal consistency (Bi-directional Self-Warp Error, MS2R metric).
• Information Retrieval Fusion: Data fusion benchmarks such as ProbFuse (1409.8518) formalize probabilistic aggregation of multiple system outputs, with systematic training and empirical MAP improvements over traditional rank aggregation schemes (for example, CombMNZ). Though not named FusionBench, these methodologies are analogous in their benchmarking rigor.

6. Impact, Expansion, and Community Resources

FusionBench benchmarks—across domains—advance research by promoting:

• Reproducibility and comparability via standardized datasets, evaluation scripts, and public leaderboards.
• Extensibility, enabling addition of tasks, models, and fusion strategies through open-source codebases and modular APIs (e.g., CLI and YAML for deep model fusion (2406.03280), REST APIs for SLAM benchmarking (1808.06820), project homepages such as https://tanganke.github.io/fusion_bench/ and https://vfbench.github.io).
• Rigorous error analysis, robust metrics, and empirical validation against analytic and real-world standards.
• Transparent reporting of resource requirements, trade-offs (e.g., parameter merging efficiency vs. ensemble accuracy; performance vs. inference cost in LLM routing), and the identification of open areas such as uncertainty quantification and trustworthiness in model fusion.

7. Future Directions and Challenges

Current and planned developments in FusionBench environments include:

• Broadening task and domain coverage (e.g., multi-modal, reinforcement learning, and preference alignment tasks in deep learning).
• Incorporating more granular trust and bias metrics, especially for fused LLM outputs (2507.10540).
• Enhanced support for uncertainty quantification (covariance data in nuclear simulations), real-time optimization, and dynamic load balancing in coupled simulation frameworks.
• Automated benchmarking workflows and richer visualization/support tools for interpretability and meta-benchmarking.

A plausible implication is that FusionBench, as a paradigm, enables a coherent and cumulative foundation for evaluating fusion strategies—whether in integrating neural predictors, simulating plasma behavior, or aggregating multimodal sensor data—thereby systematically catalyzing progress in multi-system and multi-model research.