FusionEnsemble-Net: Adaptive Fusion Patterns

• FusionEnsemble-Net is a design pattern that adapts multiple expert networks through learned feature integration rather than static averaging.
• It combines diverse modalities like RGB and radar using techniques such as attention-based fusion and meta-learning to improve accuracy.
• Empirical results across applications, including sign language recognition and hyperspectral unmixing, show enhanced performance and robustness despite computational challenges.

Searching arXiv for the named method and closely related papers. FusionEnsemble-Net denotes a family of fusion-centric ensemble architectures in which multiple experts, modalities, or trained networks are combined through learned feature integration rather than through static late averaging alone. The exact name appears in an attention-based ensemble for multimodal isolated sign language recognition, where RGB video and range–Doppler map radar streams are fused inside four spatiotemporal channels and then aggregated by an ensemble head (Islam et al., 12 Aug 2025). Several adjacent works explicitly state that they do not use the name, but describe architectures or fusion procedures that can be instantiated under it, including adaptive feature fusion with meta-learning (Mungoli, 2023), latent-space fusion of heterogeneous CNN and RNN branches with Bayesian uncertainty estimation (Araz et al., 2021), and model-level fusion by neuron transplantation or partial optimal transport (Öz et al., 7 Feb 2025, Morelli et al., 21 May 2026).

1. Terminology and scope

The name is not used uniformly across the literature. One line of work uses FusionEnsemble-Net as the actual model name for multimodal sign language recognition; another uses it as a practical instantiation label for a broader adaptive ensemble-learning framework; others present closely related methods under names such as FusionNet, Ensemble Neural Network, or partial fusion. This makes the term best understood as a design pattern centered on three recurring ideas: diversity of experts, learned fusion, and a unified prediction mechanism.

Work	Relation to the term	Core mechanism
(Islam et al., 12 Aug 2025)	Exact name used	Attention-based multimodal spatiotemporal ensemble
(Mungoli, 2023)	Explicit instantiation under the name	Adaptive feature fusion with meta-learning
(Araz et al., 2021)	Explicitly described as consistent with the term	Latent-space CNN–RNN fusion with Bayesian heads
(Ratnayake et al., 2024)	Mapped directly to the architecture named FusionNet	Endmember fusion with spatial context
(Shi et al., 27 Apr 2025)	Described as FusionNet or “FusionEnsemble-Net”	Parallel multi-model linear fusion
(Öz et al., 7 Feb 2025) / (Morelli et al., 21 May 2026)	Practical capacity-preserving or partial model fusion	Neuron transplantation or partial fusion

A common misconception is that FusionEnsemble-Net refers to a single canonical architecture. The literature does not support that interpretation. The exact label is specific in (Islam et al., 12 Aug 2025), whereas several other works explicitly frame it as an apt name for related fusion-ensemble constructions rather than as their original nomenclature (Mungoli, 2023, Araz et al., 2021, Ratnayake et al., 2024, Shi et al., 27 Apr 2025).

2. Architectural principles and mathematical formulation

Across these works, a recurring formulation begins with a set of experts $\{E_i\}_{i=1}^N$ producing feature streams $x_i \in \mathbb{R}^{d_i}$. A projection layer maps these to a shared embedding space,

$h_i = P_i(x_i) = W_i x_i + b_i \in \mathbb{R}^d,$

after which attention, gating, or stacking computes a fused representation. In one generic formulation, attention scores and weights are

$$s_i = v^\top \tanh(W_h h_i + b_h), \qquad w_i = \mathrm{softmax}(s)_i,$$

and the fused feature is

$z = \sum_i w_i h_i.$

An alternative is concatenation-plus-MLP,

$$h_{\mathrm{cat}} = [h_1; \ldots; h_N], \qquad z = \phi(h_{\mathrm{cat}}),$$

with optional bilinear pooling, mixture-of-experts gating, or a stacked meta-learner on expert predictions (Mungoli, 2023).

The named FusionEnsemble-Net for sign language recognition instantiates this pattern with four distinct spatiotemporal backbones—3D ResNet-18, MC3-18, R(2+1)D-18, and Swin-B—each duplicated across RGB and radar branches. For backbone $i$,

$$F_{\mathrm{RGB},i} = \Phi_{i,\mathrm{RGB}}(V_{\mathrm{RGB}}), \qquad F_{\mathrm{RDM},i} = \frac{1}{A}\sum_{a=1}^{A}\Phi_{i,\mathrm{RDM}}(V_{\mathrm{RDM},a}),$$

followed by modality-specific temporal modeling,

$$H_{\mathrm{RGB},i} = \mathcal{T}_{i,\mathrm{RGB}}(F_{\mathrm{RGB},i}), \qquad H_{\mathrm{RDM},i} = \mathcal{T}_{i,\mathrm{RDM}}(F_{\mathrm{RDM},i}),$$

concatenation,

$$H_{\mathrm{concat},i} = [H_{\mathrm{RGB},i}; H_{\mathrm{RDM},i}],$$

and scaled dot-product self-attention,

$x_i \in \mathbb{R}^{d_i}$0

$x_i \in \mathbb{R}^{d_i}$1

yielding $x_i \in \mathbb{R}^{d_i}$2 before per-stream classification and final probability averaging (Islam et al., 12 Aug 2025).

A historically earlier but conceptually related formulation is deep fusion of intermediate representations. In Deeply-Fused Nets, the fused activation at stage $x_i \in \mathbb{R}^{d_i}$3 is

$x_i \in \mathbb{R}^{d_i}$4

and the fused output is fed into the next stage of every base network. This repeatedly fuses intermediate activations rather than only final features or final decisions (Wang et al., 2016).

3. Training regimes, objectives, and uncertainty handling

Training protocols vary with the fusion granularity. In the generic adaptive-ensemble formulation, experts may be trained independently via bagging, boosting-like reweighting, or stacking, after which the fusion projections, gating module, and task head are jointly optimized. A representative total objective is

$x_i \in \mathbb{R}^{d_i}$5

where $x_i \in \mathbb{R}^{d_i}$6 is the supervised loss on the fused prediction, $x_i \in \mathbb{R}^{d_i}$7 aggregates expert-level losses, $x_i \in \mathbb{R}^{d_i}$8 is a diversity regularizer, and $x_i \in \mathbb{R}^{d_i}$9 includes weight decay or norm constraints. Optional extensions include uncertainty-based weighting, distillation, and MAML-style updates for fusion parameters (Mungoli, 2023).

The sign-language FusionEnsemble-Net is trained end to end with summed cross-entropy across the four per-stream heads,

$h_i = P_i(x_i) = W_i x_i + b_i \in \mathbb{R}^d,$0

using AdamW with cosine-annealed learning rate. The backbones are pre-trained—3D CNNs on Kinetics-400 and Swin-B on ImageNet—and the reported training configuration uses batch size 4, 25 epochs, two NVIDIA A6000 GPUs, and approximately 44 hours total training time (Islam et al., 12 Aug 2025).

A separate but influential variant couples latent-feature fusion with Bayesian output modeling. In the Bayesian ENN formulation, the predictive distribution is approximated by Monte Carlo samples from an approximate posterior,

$h_i = P_i(x_i) = W_i x_i + b_i \in \mathbb{R}^d,$1

with predictive mean and variance

$h_i = P_i(x_i) = W_i x_i + b_i \in \mathbb{R}^d,$2

For binary classification, the uncertainty decomposition is

$h_i = P_i(x_i) = W_i x_i + b_i \in \mathbb{R}^d,$3

and the hypothesis entropy is

$h_i = P_i(x_i) = W_i x_i + b_i \in \mathbb{R}^d,$4

This formulation uses TensorFlow Probability DenseFlipout layers and variational inference to reduce epistemic uncertainty and the entropy of the hypothesis (Araz et al., 2021).

4. Domain-specific instantiations

In multimodal isolated sign language recognition, FusionEnsemble-Net is a synchronous RGB–radar architecture for the MultiMeDaLIS dataset, which contains 126 isolated Italian Sign Language classes in medical communication scenarios. Its distinctive properties are mid-level attention-based multimodal fusion inside each backbone and hierarchical ensembling across the four fused streams. Radar contributes motion dynamics and privacy-preserving sensing, while RGB contributes handshape, facial expression, and posture; the attention block dynamically re-weights their joint contribution (Islam et al., 12 Aug 2025).

In event reconstruction for boosted hadronic top-quark jet discrimination, the closest analogue is the Ensemble Neural Network formed from a CNN branch on jet images and an RNN/LSTM branch on clustering sequences. The penultimate latent features $h_i = P_i(x_i) = W_i x_i + b_i \in \mathbb{R}^d,$5 and $h_i = P_i(x_i) = W_i x_i + b_i \in \mathbb{R}^d,$6 are concatenated into $h_i = P_i(x_i) = W_i x_i + b_i \in \mathbb{R}^d,$7 and passed through a fusion head with a Dense(96, ReLU, L2 kernel regularization $h_i = P_i(x_i) = W_i x_i + b_i \in \mathbb{R}^d,$8) layer and dropout(0.25) before and after. This design goes beyond simple prediction combining by linking different sources of error correlations, hence improving the representation between data and hypothesis (Araz et al., 2021).

In hyperspectral unmixing, FusionNet is explicitly mapped to FusionEnsemble-Net. Its three-part architecture consists of a Pixel Contextualizer, an Abundance Predictor, and a Signature Predictor. The Pixel Contextualizer uses the center pixel as query and an arbitrary neighborhood as keys and values; the Signature Predictor uses single-head attention per endmember to fuse multiple endmember candidates from several extraction algorithms. The method targets two bottlenecks at once: initialization bias from a single endmember extraction algorithm and underuse of spatial context (Ratnayake et al., 2024).

In low-light image enhancement, FusionNet is a fully parallel three-branch framework combining Retinexformer in sRGB, ESDNet in sRGB, and CIDNet in HVI space, with the final output given by a convex combination

$h_i = P_i(x_i) = W_i x_i + b_i \in \mathbb{R}^d,$9

The fusion weights are selected by minimizing expected $$s_i = v^\top \tanh(W_h h_i + b_h), \qquad w_i = \mathrm{softmax}(s)_i,$$0 risk in a Hilbert space, and the framework is presented as a way to capture global and local features across diverse color spaces while avoiding feature misalignment and extra fusion-head parameters (Shi et al., 27 Apr 2025).

A related ensemble-fusion logic also appears in multi-exposure image fusion. EMEF uses a UNet-based imitator network with Style Control Blocks to imitate four MEF contributors and then tunes a continuous style code at runtime by minimizing

$$s_i = v^\top \tanh(W_h h_i + b_h), \qquad w_i = \mathrm{softmax}(s)_i,$$1

This suggests a broader interpretation of FusionEnsemble-Net as a network that internalizes several expert behaviors and then adapts its fusion behavior per input pair (Liu et al., 2023).

5. Empirical behavior and reported performance

The named FusionEnsemble-Net reports 99.44% test accuracy and 99.37% validation accuracy on MultiMeDaLIS. Among its four single fused streams, MC3-18 is strongest at 99.06% test and 98.96% validation, while the full ensemble improves over that best single stream by +0.38% absolute. The paper further reports that radar-only baselines peak at 93.6%, and strong RGB-only baselines reach approximately 98%, which is presented as evidence of modality complementarity (Islam et al., 12 Aug 2025).

The Bayesian ENN literature emphasizes not only accuracy improvement but also uncertainty reduction. The reported standard deviation of entropy is >0.025 for the CNN, approximately 0.015 for the RNN, and $$s_i = v^\top \tanh(W_h h_i + b_h), \qquad w_i = \mathrm{softmax}(s)_i,$$2 for the ENN. The fraction of events with mean entropy below 0.5 rises from 71.92% for the RNN and 75.22% for the CNN to 78.05% for a one-dense-layer ENN and 79.55% for a two-dense-layer ENN. Predictive standard deviation for mean-of-predictions reaches up to approximately 0.01, while the ENN limits it below approximately 0.004 (Araz et al., 2021).

In hyperspectral unmixing, FusionNet reports the best average abundance RMSE on both datasets studied: 0.0333 on Samson and 0.0854 on Jasper-Ridge. For average SAD, it reports 0.0250 on Samson, which is best, and 0.0870 on Jasper-Ridge, which is second best. The same work reports that using all three endmember extraction algorithms—VCA, N-FINDR, and ATGP—yields the best average RMSE and average SAD on Samson, and that removing the Pixel Contextualizer raises the Samson average RMSE from 0.0333 to 0.0489 (Ratnayake et al., 2024).

In low-light image enhancement, FusionNet achieved 1st place in the CVPR2025 NTIRE Low Light Enhancement Challenge with PSNR 26.24, SSIM 0.861, LPIPS 0.128, and NIQE 10.95. On LOL benchmarks it reports 25.17 / 0.857 / 0.103 on LOLv1, 24.44 / 0.875 / 0.110 on LOLv2-real, and 26.50 / 0.945 / 0.042 on LOLv2-synthetic. The reported ablation on LOLv1 shows linear fusion outperforming serial, multi-stage serial, and concatenation-plus-new-network fusion across PSNR, SSIM, and LPIPS (Shi et al., 27 Apr 2025).

The broader adaptive ensemble-learning framework reports consistent improvements over baseline models and traditional feature fusion techniques across image classification, object detection, sentiment analysis, and graph-based learning, but the text does not provide exact metric values. This is important methodologically: the empirical claim is qualitative rather than numerically specified (Mungoli, 2023).

At the model-fusion level, neuron transplantation and partial fusion show that “fusion” can also mean collapsing an ensemble into a single model or interpolating between aggregation and an ensemble. Neuron Transplantation reports, for example, that VGG11 on CIFAR100 improves from 36.0±18.88 post-fusion to 60.54±2.38 after fine-tuning, compared with a best individual score of 52.02±0.3 and an ensemble score of 57.25±1.07. Partial fusion reports that with $$s_i = v^\top \tanh(W_h h_i + b_h), \qquad w_i = \mathrm{softmax}(s)_i,$$3 accuracy approached the ensemble while using only approximately 1.45× parameters versus 2× for the ensemble (Öz et al., 7 Feb 2025, Morelli et al., 21 May 2026). This suggests that the term spans both feature-level and parameter-level fusion regimes.

6. Misconceptions, limitations, and future directions

A second misconception is that fusion necessarily means averaging final predictions. The literature covers at least four distinct regimes: repeated fusion of intermediate representations, latent-feature fusion before the decision head, fusion of multiple initialization candidates or endmembers, and parameter-space fusion of entire trained models (Wang et al., 2016, Araz et al., 2021, Ratnayake et al., 2024, Öz et al., 7 Feb 2025, Morelli et al., 21 May 2026). The common thread is not a single operator but the deliberate use of complementary representations or neurons to reduce redundancy and improve robustness.

Compute and deployment costs remain a recurring limitation. The sign-LLM reports a training footprint of two NVIDIA A6000 GPUs for 25 epochs and approximately 44 hours total, with batch size 4, while the Bayesian ENN requires dual-modality preprocessing and Monte Carlo sampling at prediction time (Islam et al., 12 Aug 2025, Araz et al., 2021). The model-fusion literature presents this as one motivation for capacity-preserving fusion, but those methods have their own constraints: neuron transplantation requires identical architectures and layer sizes across ensemble members, and partial fusion experiments are reported for the two-network case (Öz et al., 7 Feb 2025, Morelli et al., 21 May 2026).

Robustness claims also differ sharply by application. The sign-LLM argues that radar improves robustness to lighting and occlusion, but explicit modality-drop evaluation is not reported. The Bayesian ENN is less susceptible to limited training sample size, yet its authors note that aleatoric noise may still dominate. EMEF focuses on extreme static multi-exposure fusion and does not directly address dynamic scenes with ghosting. Hyperspectral FusionNet demonstrates resilience to additive noise, but its neighborhood selection remains user-defined rather than learned (Islam et al., 12 Aug 2025, Araz et al., 2021, Liu et al., 2023, Ratnayake et al., 2024).

Future directions reported across these works are correspondingly heterogeneous. For sign language recognition, the stated directions are extension to continuous signing and model compression through knowledge distillation or TinyML (Islam et al., 12 Aug 2025). For hyperspectral unmixing, plausible extensions include learned neighborhood policies, deeper transformer stacks, and alternative geometric priors, although these are presented as possible improvements rather than reported results (Ratnayake et al., 2024). For parameter-level fusion, open questions include principled selection of $$s_i = v^\top \tanh(W_h h_i + b_h), \qquad w_i = \mathrm{softmax}(s)_i,$$4, layer-wise $$s_i = v^\top \tanh(W_h h_i + b_h), \qquad w_i = \mathrm{softmax}(s)_i,$$5 schedules, scaling to larger models, improved similarity metrics, and stronger guarantees on preserving ensemble diversity (Morelli et al., 21 May 2026).

In that sense, FusionEnsemble-Net is best understood not as a single fixed network but as a technical lineage organized around adaptive fusion of complementary experts. The exact named model in multimodal sign language recognition is one concrete realization; the surrounding literature shows that the same principle extends to vision, language, graphs, hyperspectral analysis, image enhancement, event reconstruction, and model compression, with the particular form of fusion—attention, gating, stacking, convex blending, or neuron transplantation—determined by the task and the level at which complementarity is exploited.