FusionMAE: Multimodal Masked Autoencoder

• The paper demonstrates FusionMAE's unique integration of masked reconstruction with joint multimodal fusion, yielding robust representations for diverse tasks.
• FusionMAE architectures vary in fusion loci and masking strategies, impacting performance metrics across areas like remote sensing, autonomous driving, and medical imaging.
• FusionMAE enhances resilience to missing inputs and modality gaps, enabling improved downstream applications such as tokamak diagnostics and disruption prediction.

Searching arXiv for papers directly relevant to FusionMAE and closely related multimodal masked-autoencoder fusion methods. Fusion Masked Auto-Encoder (FusionMAE) denotes a family of masked-autoencoding architectures in which masking and reconstruction are combined with multimodal, multi-view, or cross-domain fusion, and it also names a specific large-scale foundation model for tokamak diagnostics and control (Yang et al., 16 Sep 2025). In the broader literature, this label covers substantially different designs: some methods perform genuine joint multimodal masking and reconstruction in a shared latent space, as in SAR-optical remote sensing, camera-LiDAR driving models, and multi-sequence brain MRI pretraining (Chan-To-Hing et al., 2024, Duan et al., 2024, Erdur et al., 14 Sep 2025), whereas others reuse a pretrained MAE encoder as a feature extractor and add fusion modules or downstream attention heads afterward, as in infrared-visible image fusion, facial expression recognition, and dynamic emotion recognition (Li et al., 2024, Nguyen-Xuan et al., 2024, Xiang et al., 2024). Across these variants, the recurrent objective is to use masked reconstruction, shared latent structure, or both to make fused representations more semantically informative, more robust to missing inputs, or more transferable to downstream tasks.

1. Conceptual scope

The literature contains at least two clearly distinguishable FusionMAE patterns. In the first, masking and fusion are integrated into one pretraining pipeline. Fus-MAE fuses SAR and multispectral optical tokens through cross-attention inside the MAE architecture, MaskFuser masks a joint multimodal token sequence for camera-LiDAR driving, UniM$^2$AE projects image and LiDAR features into a unified 3D volume before reconstructing both modalities, and MultiMAE for Brain MRIs uses modality-specific adapters with a shared encoder and per-modality decoders (Chan-To-Hing et al., 2024, Duan et al., 2024, Zou et al., 2023, Erdur et al., 14 Sep 2025). In the second, a pretrained MAE contributes a latent space or initialization, but the actual fusion logic is added later. MaeFuse reuses a pretrained MAE-Large encoder for infrared-visible image fusion and adapts lightweight fusion modules to that latent space, while FER systems such as the dual-view MAE-Face pipeline and MultiMAE-DER introduce fusion only after MAE-style pretraining or downstream fine-tuning (Li et al., 2024, Nguyen-Xuan et al., 2024, Xiang et al., 2024).

This distinction is technically consequential. A canonical FusionMAE learns fusion-relevant structure during masked reconstruction itself; a weaker MAE-backed fusion model imports priors from MAE pretraining but leaves multimodal interaction to later supervised or unsupervised modules. The difference explains why papers that all appear “FusionMAE-like” can nevertheless optimize different losses, expose the model to different missing-modality conditions, and make different claims about robustness or transfer.

2. Architectural patterns

Architecturally, FusionMAE-like systems differ mainly in where fusion is inserted relative to masking and reconstruction. MaeFuse uses a unified pretrained encoder, $\Phi_V=\mathbb{E}(V)$ and $\Phi_I=\mathbb{E}(I)$, then applies the Comparative Fusion Module and Merging Fusion Module before a 4-layer ViT-block decoder (Li et al., 2024). Fus-MAE replaces the first encoder block with an XAttnEncoder and inserts XAttnDecoder before lightweight modality-specific decoders, so early fusion and feature-level fusion are both explicit (Chan-To-Hing et al., 2024). MaskFuser performs hybrid fusion: Monotonic-to-BEV Translation attention as geometry-aware early fusion, followed by joint tokenization into a unified sequence of 174 tokens with shared transformer encoding (Duan et al., 2024). UniM$^2$AE instead lifts image and LiDAR features into a unified 3D volume and applies the Multi-modal 3D Interaction Module before projecting back to modality-specific decoders (Zou et al., 2023). In medical imaging, MultiMAE for Brain MRIs patchifies each MRI sequence separately, projects them with modality-specific linear adapters into a shared token space, and reconstructs each modality with a dedicated decoder (Erdur et al., 14 Sep 2025). The plasma model titled FusionMAE uses an $88\times10$ diagnostic-signal window, maps each channel patch to 64 dimensions, processes the resulting $88\times64$ sequence with Transformer blocks, compresses to a 256-dimensional plasma status embedding, and reconstructs all 88 signals through a mirrored decoder (Yang et al., 16 Sep 2025).

System	Fusion locus	Reconstruction or primary output
MaeFuse	Shared pretrained MAE encoder plus CFM/MFM	Fused infrared-visible luminance image
Fus-MAE	XAttnEncoder and XAttnDecoder	SAR and optical modality reconstruction
MaskFuser	MBT early fusion plus unified multimodal tokens	Image/LiDAR reconstruction and driving representation
UniM$^2$AE	Unified 3D volume plus MMIM	Masked image and voxel reconstruction
MultiMAE for Brain MRIs	Shared encoder with modality-specific decoders	Per-sequence MRI reconstruction
FusionMAE	Diagnostic-channel Transformer bottleneck	88-signal reconstruction and 256-d embedding

These architectures show that “fusion” is not a single operation. It may be cross-attention between modality-specific tokens, token concatenation into a common sequence, lifting to a geometry-aware shared space, or adaptation of lightweight fusion layers to a fixed pretrained latent manifold. What remains stable is the attempt to force complementary modalities or channels into a representation that is reconstructable, transferable, or both.

3. Masking, objectives, and optimization

Masking policy is the point at which these systems diverge most sharply. Fus-MAE follows MAE convention with 75% masking and studies independent versus consistent masking across SAR and optical streams, computing mean squared error over reconstructed tokens only (Chan-To-Hing et al., 2024). MaskFuser masks 75% of a joint cross-modal token sequence and reconstructs original image and LiDAR sensory inputs, while also using auxiliary depth, segmentation, BEV map, and detection tasks during pretraining (Duan et al., 2024). UniM$^2$AE randomly masks both image patches and LiDAR voxels, with best-performing ratios of 75% for camera and 70% for LiDAR, and optimizes image MSE together with LiDAR Chamfer and occupancy losses (Zou et al., 2023). MultiMAE for Brain MRIs fixes a global masking ratio of 75% but samples per-modality masking from a Dirichlet distribution with $\alpha=1$, allowing one MRI sequence to be masked up to 100% while preserving the expected global rate, and optimizes reconstruction MSE over all tokens (Erdur et al., 14 Sep 2025).

MaeFuse is different because masked reconstruction is not the training signal for the fusion task itself. Instead, the method treats the pretrained MAE latent space as structured and introduces alignment losses,

$$\mathcal{L}_{\text{CFM-align}}=\left(\frac{\Phi_I+\Phi_V}{2}-\Phi_D\right)^2,\qquad \mathcal{L}_{\text{MFM-align}}=\left(\frac{\Phi_I+\Phi_V}{2}-\Phi_F\right)^2,$$

before switching to unsupervised image-level intensity and gradient objectives (Li et al., 2024). The plasma FusionMAE likewise centers training on reconstruction, but with a domain-specific missing-signal regime: 25% of valid diagnostic channels are randomly masked, and the weighted reconstruction uses $\Phi_V=\mathbb{E}(V)$0 for intentionally masked valid channels, $\Phi_V=\mathbb{E}(V)$1 for valid unmasked channels, and $\Phi_V=\mathbb{E}(V)$2 for invalid channels (Yang et al., 16 Sep 2025).

The resulting optimization regimes range from pure masked reconstruction to curriculum-style latent alignment, and from symmetric modality reconstruction to asymmetric conditional reconstruction. That heterogeneity is central to the concept: FusionMAE is better understood as a design principle for combining masking with fusion than as one fixed objective.

4. Representative application domains

The domain spread of FusionMAE-like designs is unusually wide. In infrared-visible imaging, MaeFuse uses MAE-derived “omni features” to preserve low-level detail and high-level salience without downstream supervision, while MAFS uses a masked reconstruction stage only as the first step of a later fusion-segmentation multitask framework rather than as the final deployed model (Li et al., 2024, Wang et al., 15 Sep 2025). In remote sensing, Fus-MAE targets the large domain gap between SAR and multispectral optical data and reports 87.9 mAP on S1+S2 with full labels and 68.7 mAP in the 1% label setting for the XAED variant on BigEarthNet-MM (Chan-To-Hing et al., 2024). In autonomous driving, MaskFuser reaches a driving score of 49.05 and route completion of 92.85% on CARLA LongSet6, while UniM$\Phi_V=\mathbb{E}(V)$3AE improves downstream 3D object detection and BEV map segmentation by 1.2% NDS and 6.5% mIoU, respectively, on nuScenes (Duan et al., 2024, Zou et al., 2023). Camera-conditioned LiDAR reconstruction is represented by MaskedFusion360, which raises test MSSIM from 0.6410 without camera features to 0.9612 with camera features (Wagner et al., 2023). In medical imaging, MultiMAE for Brain MRIs reports absolute improvement of 10.1 overall Dice score and 0.46 MCC over MAE-ViT baselines with missing input sequences (Erdur et al., 14 Sep 2025).

These examples make clear that FusionMAE is not tied to one signal geometry. Some systems operate on images, some on voxel grids or BEV-aligned features, some on time-series channels, and some on 3D medical volumes. A plausible implication is that the transferability of the paradigm depends less on any specific modality than on whether there exists a meaningful shared latent space in which masking can force cross-source inference.

5. The plasma foundation model explicitly named FusionMAE

The literal model named “FusionMAE” is a self-supervised, Transformer-based masked autoencoder for magnetic-fusion diagnostics and control on HL-3 (Yang et al., 16 Sep 2025). It operates on 88 diagnostic signals from 12 systems, each represented over a 10 ms window sampled at 1 kHz, so the raw input is $\Phi_V=\mathbb{E}(V)$4. Each channel is projected by an MLP to 64 dimensions, sinusoidal positional encoding is added over channel index, the resulting $\Phi_V=\mathbb{E}(V)$5 sequence passes through Transformer blocks with 8 heads, and the flattened 5632-dimensional representation is compressed to a 256-dimensional plasma status embedding before a mirrored decoder reconstructs the original signals (Yang et al., 16 Sep 2025).

The model is trained on 2,445 HL-3 plasma discharges from four campaigns spanning 2022–2025, split into 1,950 development shots and 495 test shots (Yang et al., 16 Sep 2025). Its two explicit mechanisms are compression-reduction and missing-signal reconstruction. The information bottleneck is the 256-dimensional latent embedding, while robustness is induced by random masking of 25% of valid channels during training. The reported reconstruction quality is 98.6% PCC for full compression-and-restoration across all 88 channels, and 96.7% PCC for masked-channel reconstruction, which the paper interprets as the reliability of “virtual backup diagnosis” (Yang et al., 16 Sep 2025).

The downstream interpretation is broader than signal reconstruction. The pretrained embedding is reused for disruption prediction, equilibrium fitting surrogate modeling, and plasma evolution prediction, and the paper describes three emergent capabilities: automatic data analysis, universal control-diagnosis interface, and enhancement of control performance on multiple tasks (Yang et al., 16 Sep 2025). The embedding also organizes test-set slices by physically meaningful operating conditions including $\Phi_V=\mathbb{E}(V)$6, $\Phi_V=\mathbb{E}(V)$7, $\Phi_V=\mathbb{E}(V)$8, and $\Phi_V=\mathbb{E}(V)$9 in UMAP space, and the paper defines a diagnostic contribution score,

$\Phi_I=\mathbb{E}(I)$0

to quantify how strongly masking a particular channel perturbs the plasma-status embedding (Yang et al., 16 Sep 2025). In this narrow sense, FusionMAE is not a multimodal image-fusion model at all, but a foundation model for fusing heterogeneous diagnostic channels into a reusable plasma-state representation.

6. Misconceptions, limitations, and research directions

A recurrent misconception is that every FusionMAE system is a jointly pretrained multimodal autoencoder. The dual-view FER system built on MAE-Face is better described as MAE-pretrained downstream attention fusion, because fusion is introduced after branch-wise fine-tuning rather than inside masked autoencoding (Nguyen-Xuan et al., 2024). MultiMAE-DER similarly uses a pretrained VideoMAE-style encoder and supervised fine-tuning over six multimodal packing strategies, without a multimodal masking or reconstruction stage in the proposed training procedure (Xiang et al., 2024). MAFS contains a genuine masked reconstruction stage, but confines it to the first stage of a broader fusion-segmentation multitask framework (Wang et al., 15 Sep 2025).

Implementation under-specification remains pervasive. MaeFuse does not spell out the micro-architecture of CFM and MFM, exact decoder width, or whether the MAE backbone is fully frozen or fine-tuned (Li et al., 2024). Fus-MAE leaves patch size $\Phi_I=\mathbb{E}(I)$1, embedding dimension, number of heads, and exact decoder depth unspecified, and also notes that models still rely mainly on optical data in the SAR+optical setting (Chan-To-Hing et al., 2024). MaskFuser improves driving stability under damaged sensory inputs, but at 50% masking its route completion is slightly lower than TransFuser even though driving score is higher (Duan et al., 2024). The plasma FusionMAE exhibits a nonlinear loss escalation above roughly 50% masking and defaults to training-set averages at 100% masking (Yang et al., 16 Sep 2025). This suggests that the central technical problem in FusionMAE is not merely multimodal feature concatenation but the design of a latent space and masking regime that remain informative under severe domain gap, modality imbalance, and missing-sensor conditions.