---
title: 'CentralNet: Adaptive Multimodal Fusion'
url: https://www.emergentmind.com/topics/centralnet
type: topic
---

# CentralNet: Adaptive Multimodal Fusion

CentralNet is a deep multimodal fusion architecture that introduces a trainable, multilayer central network responsible for adaptive integration of hidden representations from multiple modality-specific subnetworks. Its design provides a continuous interpolation between early and late fusion strategies, driven by learnable fusion weights at each network depth and regularized via multi-task learning objectives. CentralNet has been validated across diverse multimodal benchmarks, consistently yielding improved accuracy relative to state-of-the-art baselines by automatically selecting optimal fusion depths for each task [1808.07275][1811.02447].

## 1. Motivation and Overview

The challenge in multimodal deep learning lies in how and when to fuse independent modality streams—early (at feature level), late (at decision/score level), or mid-level. No universal best practice exists, as optimal fusion depth is highly task-dependent. Traditional approaches tend toward architectural locality, either concatenating features early on or aggregating predicted scores late. CentralNet proposes a central fusion module that interfaces with each unimodal network at every layer, combining their representations via learnable scalar weights. This realizes a multi-level fusion framework in which the network dynamically learns "how much" and "where" to fuse across modalities [1808.07275][1811.02447].

## 2. Architecture and Mathematical Formulation

Consider $K$ modality-specific networks $M^k$ ($k=1,\ldots,K$), each generating hidden representations $h^k_i\in\mathbb{R}^{d_i}$ at depth $i$ ($i=0,\ldots,L$), with $h^k_0$ as the input feature and $h^k_L$ as the logit vector. The central network $C$ is constructed with the same depth $L$, maintaining its hidden state $c_i\in\mathbb{R}^{d_i}$, initialized with $c_0=0$ or by a weighted sum of $h^k_0$.

At every layer $i$, the central network fuses all $h^k_i$ and its own previous state $c_i$ by a learned weighted sum as follows:
\[
u_{i+1} = \alpha^C_i\,c_i + \sum_{k=1}^K \alpha^k_i\,h^k_i + b_i
\]
\[
c_{i+1} = \rho\left(W^C_i u_{i+1}\right)
\]
where $\alpha^k_i,\,\alpha^C_i\in\mathbb{R}_{\geq 0}$ are trainable, scalar fusion weights associated respectively with the $k$-th modality and the central net self-connection, $b_i$ is bias, $W^C_i$ is central layer weight matrix, and $\rho$ is a nonlinearity (usually ReLU).

At the output, the logits of all unimodal branches $h_L^k$ and the central branch $c_L$ can be fused by another weighted sum before the final softmax or sigmoid, thus realizing generalized late fusion as a special case.

## 3. Multi-Task Learning Objective and Regularization

CentralNet employs a multi-objective loss to ensure the central fusion improves over, but does not degrade, unimodal branch performance. Let $\hat{y}^C$ denote the softmax output of the central net, and $\hat{y}^k$ the unimodal outputs. With ground-truth $y$, central and unimodal cross-entropy losses are given as
\[
L_C = -\sum_{j} y_j\log\hat{y}^C_j,\qquad L_k = -\sum_{j} y_j\log\hat{y}^k_j
\]
The overall loss takes the form:
\[
L_{\mathrm{total}} = L_C + \sum_{k=1}^K \beta_k L_k
\]
In all reported experiments, $\beta_k=1$, so the fusion and unimodal objectives are optimized jointly, regularizing the entire architecture and stabilizing training. For multilabel setups (MM-IMDb), a weighted binary cross-entropy with positive class weight $=2$ is applied for the central and unimodal outputs [1808.07275][1811.02447].

## 4. Implementation and Training Protocol

CentralNet is designed to wrap arbitrary modality-specific deep networks. In each branch, best-practice modules (e.g., LeNet-5 CNN for image, MLPs for audio/text) are used. Dense and convolutional layers are regularized via batch normalization and dropout (typically $50\%$ on dense layers).

The recommended training protocol is:

1. (Optional) Pre-train each unimodal subnetwork independently using its own data and loss $L_k$.
2. Initialize the central net’s parameters and all fusion weights.
3. Jointly fine-tune the entire architecture (central and all modality-specific branches) using Adam optimizer (default $\beta_1=0.9,\ \beta_2=0.999$), optimizing $L_{\mathrm{total}}$ on all samples.
4. Use dataset-specific learning rates and schedule, apply early stopping where appropriate; apply batch-wise dropout and batch normalization to combat overfitting.

Typical hyperparameters include a batch size of $128$ (or $42$ for Montalbano), learning rates in $[0.001,0.05]$, and 100 epochs (unless specified) [1808.07275][1811.02447].

## 5. Empirical Evaluation Across Multimodal Tasks

CentralNet was validated on four diverse benchmarks:

- **Multimodal MNIST** (two artificial views): Each branch is LeNet-5; CentralNet achieves a 5% relative error reduction over best baselines (53% errors vs. 56–68% for late/early fusions, ModDrop, GMU).
- **Audiovisual MNIST** (corrupted images + audio spectrograms): CentralNet yields 95.0% accuracy ($\pm0.12$), outperforming the 94.8% baseline.
- **ChaLearn Montalbano** (4-modal gesture recognition): Macro-accuracy for CentralNet is 98.27% ($\pm0.03$)—a statistically significant improvement over ModDrop (98.19%), GMU (97.98%) and other fusions.
- **MM-IMDb** (poster+plot genre multilabel): CentralNet attains a micro-F1 of 0.639, surpassing GMU (0.630) and all other listed multimodal baselines.

Results are consistent across all metrics, with CentralNet outperforming the best single-modality, early-fusion, late-fusion, ModDrop, and GMU benchmarks. Experimental runs are averaged over 64 trials, providing strong confidence in statistical significance.

| Dataset           | CentralNet | Best Baseline Type     | Best Baseline Value   |
|-------------------|------------|------------------------|-----------------------|
| Multimodal MNIST  | 53% err    | Early/Late/ModDrop/GMU | 56–68% accuracy       |
| Audiovisual MNIST | 95.0%      | Early/Late/ModDrop/GMU | 94.8%                 |
| Montalbano        | 98.27%     | ModDrop                | 98.19%                |
| MM-IMDb           | 0.639 F1   | GMU                    | 0.630                 |

## 6. Analysis of Fusion Weights and Fusion Schedules

Inspection of learned fusion weights $\alpha^k_i$ (modalities) and $\alpha^C_i$ (central) enables interpretation of preferred fusion strategies per task:

- **Early fusion** emerges when $\alpha^k_0$ is strong, favoring low-level features.
- **Late fusion** is expressed via dominant $\alpha^C_i$ at deeper layers.
- **Hybrid** schedules, with weights spread across depth, are observed on tasks like Montalbano.
- For MM-IMDb, early fusion dominates (low-level fusion of posters/text), with deeper layers shifting slightly toward textual inputs.
- AFEW prefers face-dominated early fusion, but central fusion gains prominence in the last layer.
- Audioset exhibits negligible central influence in early conv layers, increasing toward the classifier output.

This demonstrates that CentralNet adapts the modality-combination schedule contingent on intrinsic task requirements [1811.02447].

## 7. Significance and Comparative Context

CentralNet establishes a unified, flexible mechanism for multi-level fusion in multimodal deep neural networks. Its add-on central module does not mandate specific architectural restrictions on the unimodal branches, allowing plug-in use with recent advances in each modality domain. The empirical results indicate consistent, statistically significant improvements over state-of-the-art multimodal fusion techniques such as ModDrop and Gated Multimodal Units (GMU). The multi-objective regularizer is critical not only for maintaining unimodal performance but also for accelerating convergence during training. A plausible implication is that adaptive, learnable fusion depth can be leveraged broadly in multimodal or multi-sensor systems to automatically discover fusion schedules optimal for diverse domains [1808.07275][1811.02447].

Source: https://www.emergentmind.com/topics/centralnet