---
title: 'FusionMAE: Multimodal Masked Autoencoder'
url: https://www.emergentmind.com/topics/fusion-masked-auto-encoder-fusionmae
type: topic
---

# FusionMAE: Multimodal Masked Autoencoder

Searching arXiv for recent papers directly relevant to FusionMAE and closely related multimodal masked-autoencoder fusion methods.
Fusion Masked Auto-Encoder (FusionMAE) denotes a family of masked-autoencoding architectures in which masking and reconstruction are combined with multimodal, multi-view, or cross-domain fusion, and it also names a specific large-scale foundation model for tokamak diagnostics and control [2509.12945]. In the broader literature, this label covers substantially different designs: some methods perform genuine joint multimodal masking and reconstruction in a shared latent space, as in SAR-optical remote sensing, camera-LiDAR driving models, and multi-sequence brain MRI pretraining [2401.02764][2405.07573][2509.11442], whereas others reuse a pretrained MAE encoder as a feature extractor and add fusion modules or downstream attention heads afterward, as in infrared-visible image fusion, facial expression recognition, and dynamic emotion recognition [2404.11016][2403.13039][2404.18327]. Across these variants, the recurrent objective is to use masked reconstruction, shared latent structure, or both to make fused representations more semantically informative, more robust to missing inputs, or more transferable to downstream tasks.

## 1. Conceptual scope

The literature contains at least two clearly distinguishable FusionMAE patterns. In the first, masking and fusion are integrated into one pretraining pipeline. Fus-MAE fuses SAR and multispectral optical tokens through cross-attention inside the MAE architecture, MaskFuser masks a joint multimodal token sequence for camera-LiDAR driving, UniM$^2$AE projects image and LiDAR features into a unified 3D volume before reconstructing both modalities, and MultiMAE for Brain MRIs uses modality-specific adapters with a shared encoder and per-modality decoders [2401.02764][2405.07573][2308.10421][2509.11442]. In the second, a pretrained MAE contributes a latent space or initialization, but the actual fusion logic is added later. MaeFuse reuses a pretrained MAE-Large encoder for infrared-visible image fusion and adapts lightweight fusion modules to that latent space, while FER systems such as the dual-view MAE-Face pipeline and MultiMAE-DER introduce fusion only after MAE-style pretraining or downstream fine-tuning [2404.11016][2403.13039][2404.18327].

This distinction is technically consequential. A canonical FusionMAE learns fusion-relevant structure during masked reconstruction itself; a weaker MAE-backed fusion model imports priors from MAE pretraining but leaves multimodal interaction to later supervised or unsupervised modules. The difference explains why papers that all appear “FusionMAE-like” can nevertheless optimize different losses, expose the model to different missing-modality conditions, and make different claims about robustness or transfer.

## 2. Architectural patterns

Architecturally, FusionMAE-like systems differ mainly in where fusion is inserted relative to masking and reconstruction. MaeFuse uses a unified pretrained encoder, \(\Phi_V=\mathbb{E}(V)\) and \(\Phi_I=\mathbb{E}(I)\), then applies the Comparative Fusion Module and Merging Fusion Module before a 4-layer ViT-block decoder [2404.11016]. Fus-MAE replaces the first encoder block with an XAttnEncoder and inserts XAttnDecoder before lightweight modality-specific decoders, so early fusion and feature-level fusion are both explicit [2401.02764]. MaskFuser performs hybrid fusion: Monotonic-to-BEV Translation attention as geometry-aware early fusion, followed by joint tokenization into a unified sequence of 174 tokens with shared transformer encoding [2405.07573]. UniM$^2$AE instead lifts image and LiDAR features into a unified 3D volume and applies the Multi-modal 3D Interaction Module before projecting back to modality-specific decoders [2308.10421]. In medical imaging, MultiMAE for Brain MRIs patchifies each MRI sequence separately, projects them with modality-specific linear adapters into a shared token space, and reconstructs each modality with a dedicated decoder [2509.11442]. The plasma model titled FusionMAE uses an \(88\times10\) diagnostic-signal window, maps each channel patch to 64 dimensions, processes the resulting \(88\times64\) sequence with Transformer blocks, compresses to a 256-dimensional plasma status embedding, and reconstructs all 88 signals through a mirrored decoder [2509.12945].

| System | Fusion locus | Reconstruction or primary output |
|---|---|---|
| MaeFuse | Shared pretrained MAE encoder plus CFM/MFM | Fused infrared-visible luminance image |
| Fus-MAE | XAttnEncoder and XAttnDecoder | SAR and optical modality reconstruction |
| MaskFuser | MBT early fusion plus unified multimodal tokens | Image/LiDAR reconstruction and driving representation |
| UniM$^2$AE | Unified 3D volume plus MMIM | Masked image and voxel reconstruction |
| MultiMAE for Brain MRIs | Shared encoder with modality-specific decoders | Per-sequence MRI reconstruction |
| FusionMAE | Diagnostic-channel Transformer bottleneck | 88-signal reconstruction and 256-d embedding |

These architectures show that “fusion” is not a single operation. It may be cross-attention between modality-specific tokens, token concatenation into a common sequence, lifting to a geometry-aware shared space, or adaptation of lightweight fusion layers to a fixed pretrained latent manifold. What remains stable is the attempt to force complementary modalities or channels into a representation that is reconstructable, transferable, or both.

## 3. Masking, objectives, and optimization

Masking policy is the point at which these systems diverge most sharply. Fus-MAE follows MAE convention with 75% masking and studies independent versus consistent masking across SAR and optical streams, computing mean squared error over reconstructed tokens only [2401.02764]. MaskFuser masks 75% of a joint cross-modal token sequence and reconstructs original image and LiDAR sensory inputs, while also using auxiliary depth, segmentation, BEV map, and detection tasks during pretraining [2405.07573]. UniM$^2$AE randomly masks both image patches and LiDAR voxels, with best-performing ratios of 75% for camera and 70% for LiDAR, and optimizes image MSE together with LiDAR Chamfer and occupancy losses [2308.10421]. MultiMAE for Brain MRIs fixes a global masking ratio of 75% but samples per-modality masking from a Dirichlet distribution with \(\alpha=1\), allowing one MRI sequence to be masked up to 100% while preserving the expected global rate, and optimizes reconstruction MSE over all tokens [2509.11442].

MaeFuse is different because masked reconstruction is not the training signal for the fusion task itself. Instead, the method treats the pretrained MAE latent space as structured and introduces alignment losses,
\[
\mathcal{L}_{\text{CFM-align}}=\left(\frac{\Phi_I+\Phi_V}{2}-\Phi_D\right)^2,\qquad
\mathcal{L}_{\text{MFM-align}}=\left(\frac{\Phi_I+\Phi_V}{2}-\Phi_F\right)^2,
\]
before switching to unsupervised image-level intensity and gradient objectives [2404.11016]. The plasma FusionMAE likewise centers training on reconstruction, but with a domain-specific missing-signal regime: 25% of valid diagnostic channels are randomly masked, and the weighted reconstruction uses \(W_i=1\) for intentionally masked valid channels, \(W_i=0.2\) for valid unmasked channels, and \(W_i=0\) for invalid channels [2509.12945].

The resulting optimization regimes range from pure masked reconstruction to curriculum-style latent alignment, and from symmetric modality reconstruction to asymmetric conditional reconstruction. That heterogeneity is central to the concept: FusionMAE is better understood as a design principle for combining masking with fusion than as one fixed objective.

## 4. Representative application domains

The domain spread of FusionMAE-like designs is unusually wide. In infrared-visible imaging, MaeFuse uses MAE-derived “omni features” to preserve low-level detail and high-level salience without downstream supervision, while MAFS uses a masked reconstruction stage only as the first step of a later fusion-segmentation multitask framework rather than as the final deployed model [2404.11016][2509.11817]. In remote sensing, Fus-MAE targets the large domain gap between SAR and multispectral optical data and reports 87.9 mAP on S1+S2 with full labels and 68.7 mAP in the 1% label setting for the XAED variant on BigEarthNet-MM [2401.02764]. In autonomous driving, MaskFuser reaches a driving score of 49.05 and route completion of 92.85% on CARLA LongSet6, while UniM$^2$AE improves downstream 3D object detection and BEV map segmentation by 1.2% NDS and 6.5% mIoU, respectively, on nuScenes [2405.07573][2308.10421]. Camera-conditioned LiDAR reconstruction is represented by MaskedFusion360, which raises test MSSIM from 0.6410 without camera features to 0.9612 with camera features [2306.07087]. In medical imaging, MultiMAE for Brain MRIs reports absolute improvement of 10.1 overall Dice score and 0.46 MCC over MAE-ViT baselines with missing input sequences [2509.11442].

These examples make clear that FusionMAE is not tied to one signal geometry. Some systems operate on images, some on voxel grids or BEV-aligned features, some on time-series channels, and some on 3D medical volumes. A plausible implication is that the transferability of the paradigm depends less on any specific modality than on whether there exists a meaningful shared latent space in which masking can force cross-source inference.

## 5. The plasma foundation model explicitly named FusionMAE

The literal model named “FusionMAE” is a self-supervised, Transformer-based masked autoencoder for magnetic-fusion diagnostics and control on HL-3 [2509.12945]. It operates on 88 diagnostic signals from 12 systems, each represented over a 10 ms window sampled at 1 kHz, so the raw input is \(88\times10\). Each channel is projected by an MLP to 64 dimensions, sinusoidal positional encoding is added over channel index, the resulting \(88\times64\) sequence passes through Transformer blocks with 8 heads, and the flattened 5632-dimensional representation is compressed to a 256-dimensional plasma status embedding before a mirrored decoder reconstructs the original signals [2509.12945].

The model is trained on 2,445 HL-3 plasma discharges from four campaigns spanning 2022–2025, split into 1,950 development shots and 495 test shots [2509.12945]. Its two explicit mechanisms are compression-reduction and missing-signal reconstruction. The information bottleneck is the 256-dimensional latent embedding, while robustness is induced by random masking of 25% of valid channels during training. The reported reconstruction quality is 98.6% PCC for full compression-and-restoration across all 88 channels, and 96.7% PCC for masked-channel reconstruction, which the paper interprets as the reliability of “virtual backup diagnosis” [2509.12945].

The downstream interpretation is broader than signal reconstruction. The pretrained embedding is reused for disruption prediction, equilibrium fitting surrogate modeling, and plasma evolution prediction, and the paper describes three emergent capabilities: automatic data analysis, universal control-diagnosis interface, and enhancement of control performance on multiple tasks [2509.12945]. The embedding also organizes test-set slices by physically meaningful operating conditions including \(B_t\), \(I_p\), \(n_e\), and \(\beta_N\) in UMAP space, and the paper defines a diagnostic contribution score,
\[
\text{contribution}(i)=\frac{1-PCC^*(i)}{\sum_{j=1}^{88}(1-PCC^*(j))},
\]
to quantify how strongly masking a particular channel perturbs the plasma-status embedding [2509.12945]. In this narrow sense, FusionMAE is not a multimodal image-fusion model at all, but a foundation model for fusing heterogeneous diagnostic channels into a reusable plasma-state representation.

## 6. Misconceptions, limitations, and research directions

A recurrent misconception is that every FusionMAE system is a jointly pretrained multimodal autoencoder. The dual-view FER system built on MAE-Face is better described as MAE-pretrained downstream attention fusion, because fusion is introduced after branch-wise fine-tuning rather than inside masked autoencoding [2403.13039]. MultiMAE-DER similarly uses a pretrained VideoMAE-style encoder and supervised fine-tuning over six multimodal packing strategies, without a multimodal masking or reconstruction stage in the proposed training procedure [2404.18327]. MAFS contains a genuine masked reconstruction stage, but confines it to the first stage of a broader fusion-segmentation multitask framework [2509.11817].

Implementation under-specification remains pervasive. MaeFuse does not spell out the micro-architecture of CFM and MFM, exact decoder width, or whether the MAE backbone is fully frozen or fine-tuned [2404.11016]. Fus-MAE leaves patch size \(P\), embedding dimension, number of heads, and exact decoder depth unspecified, and also notes that models still rely mainly on optical data in the SAR+optical setting [2401.02764]. MaskFuser improves driving stability under damaged sensory inputs, but at 50% masking its route completion is slightly lower than TransFuser even though driving score is higher [2405.07573]. The plasma FusionMAE exhibits a nonlinear loss escalation above roughly 50% masking and defaults to training-set averages at 100% masking [2509.12945]. This suggests that the central technical problem in FusionMAE is not merely multimodal feature concatenation but the design of a latent space and masking regime that remain informative under severe domain gap, modality imbalance, and missing-sensor conditions.

Source: https://www.emergentmind.com/topics/fusion-masked-auto-encoder-fusionmae