Masked Layer Fusion Attention (MLFA)

• Masked Layer Fusion Attention (MLFA) is a multimodal fusion mechanism that integrates SAR and optical data using cross-attention within Vision Transformers.
• It employs dedicated XAttnEncoder and XAttnDecoder modules to perform early and feature-level fusion, enhancing masked autoencoder performance.
• By using independent and consistent masking strategies, MLFA efficiently aligns heterogeneous features, achieving competitive results against contrastive methods.

Masked Layer Fusion Attention (MLFA) is a multimodal data fusion mechanism developed for Masked Autoencoder (MAE)-style Vision Transformers, enabling early and feature-level information merging between heterogeneous sources such as synthetic aperture radar (SAR) and multispectral optical imagery. MLFA integrates cross-attention operations both at the input and decoder stages to facilitate interaction between modalities with non-trivial domain gaps. This framework underlies the Fus-MAE architecture, which is designed for self-supervised representation learning in remote sensing and achieves competitive or superior performance relative to contrastive and other masked modeling approaches (Chan-To-Hing et al., 2024).

1. Architectural Components and Data Flow

MLFA operates on paired modality inputs $I_1 \in \mathbb{R}^{H \times W \times C_1}$ (SAR) and $I_2 \in \mathbb{R}^{H \times W \times C_2}$ (optical). Each input is separately patch embedded using a 2D convolution with kernel and stride $P$, yielding tokens $z_{0,1}$ and $z_{0,2}$ of shape $(H/P)^2 \times d$, where $d$ is the latent dimension.

The fusion process incorporates cross-attention mechanisms at specific layers:

• Early fusion occurs in the first encoder block, replacing the standard self-attention+MLP pair with an XAttnEncoder that combines the patch-embedded tokens from both modalities.
• Feature-level fusion is conducted in the decoder just before per-modality decoding, introducing additional cross-attention via XAttnDecoder to enhance intra- and inter-modality representations.

Table 1 summarizes the key architectural stages.

Stage	Operation	Token Shape
Patch Embedding	Conv2d per modality, add $E_{emb}$	$(H/P)^2 \times d$
Early Fusion	XAttnEncoder (replaces block 1)	$(H/P)^2 \times d$
Encoder	$I_2 \in \mathbb{R}^{H \times W \times C_2}$0 ViT blocks (self-attention+MLP)	$I_2 \in \mathbb{R}^{H \times W \times C_2}$1
Decoder Fusion	XAttnDecoder pre-decoder per modality	$I_2 \in \mathbb{R}^{H \times W \times C_2}$2

2. Cross-Attention Mechanism and Fusion Operators

The core of MLFA is its cross-attention (CA) subroutine, instantiated as:

$I_2 \in \mathbb{R}^{H \times W \times C_2}$3

where $I_2 \in \mathbb{R}^{H \times W \times C_2}$4, $I_2 \in \mathbb{R}^{H \times W \times C_2}$5, and $I_2 \in \mathbb{R}^{H \times W \times C_2}$6 denote learned linear projections per head (with $I_2 \in \mathbb{R}^{H \times W \times C_2}$7, $I_2 \in \mathbb{R}^{H \times W \times C_2}$8 for $I_2 \in \mathbb{R}^{H \times W \times C_2}$9 heads). This bi-directional mechanism enables each stream to query and integrate features from the other.

For fusion, the operator is defined as:

$P$0

The XAttnEncoder combines both the direct concatenation of input streams and their mutual cross-attention outputs. This aggregated token stream is refined with a two-layer MLP using GELU activations.

In the decoder, feature-level fusion augments each modality's latent with cross-attended features from its counterpart:

$P$1

A separate XAttnDecoder refines these cross-attended latents before passing them to lightweight modality-specific decoders.

3. Masking Strategies and Data Handling

MLFA employs a masking ratio of $P$2 on the patch tokens, occurring before patch embedding. Two masking strategies are used:

• Independent masking: Randomly sample $P$3 of tokens to drop independently in each modality, encouraging discovery of both intra- and inter-modal correlations.
• Consistent masking: Apply the same mask indices across both modalities to facilitate cross-modal alignment.

Only the unmasked tokens are transformed via $P$4, $P$5, $P$6 projections and participate in cross-attention; masked tokens are omitted from encoder input as in classical MAE frameworks.

4. Layer-Wise Fusion Protocol

MLFA restricts fusion to two specific locations in the network:

• Early fusion: Performed solely in the first encoder block via XAttnEncoder; no fusion occurs in intermediate layers, which use standard ViT self-attention and MLP blocks.
• Decoder fusion: Executed once pre-decoder per modality using XAttnDecoder, followed by an MLP for further token refinement.

Post-fusion, a learned $P$7 token is appended at the encoder output.

5. Training Objective and Optimization Details

The objective is a per-modality masked reconstruction loss:

$P$8

where $P$9 denotes the per-modality lightweight transformer decoder, and $z_{0,1}$0 is the number of masked patches in modality $z_{0,1}$1. The total loss is the sum $z_{0,1}$2, with no additional weighting or contrastive terms.

Optimization follows AdamW with a batch size of 200, a peak learning rate of $z_{0,1}$3 (warmed up over 10 epochs), cosine decay, and standard ViT/MAE weight decay (e.g. 0.05). Pretraining is performed for 100 epochs on 354k BigEarthNet-MM images, optionally initializing from ImageNet-pretrained weights (Chan-To-Hing et al., 2024).

6. Implementation Considerations

MLFA adapts the MAE framework as follows:

• Backbone: Utilizes standard ViT-Base defaults, e.g., $z_{0,1}$4, $z_{0,1}$5, $z_{0,1}$6.
• Patch size: Commonly $z_{0,1}$7, yielding $z_{0,1}$8 tokens.
• Decoder: Lightweight, per-modality transformer, depth $z_{0,1}$9, hidden dim $z_{0,2}$0, $z_{0,2}$1 heads.
• Fusion point frequency: Fusion is explicitly non-repetitive in the encoder, occurring only in the first block and once per decoder.

This architectural and procedural approach is tailored to multimodal data with substantial domain differences, such as SAR and optical satellite imagery.

7. Significance and Comparative Performance

Empirical results demonstrate that MLFA within the Fus-MAE framework achieves performance on par with or exceeding contrastive learning-based fusion strategies when applied to SAR-optical paired data, while addressing the limitations of augmentation-heavy protocols. The cross-attention-based fusion of MLFA offers an alternative to contrastive alignment, enabling masked modeling to bridge disparate sensor domains efficiently. The framework is shown to outperform other masked autoencoder formulations even when trained on larger datasets (Chan-To-Hing et al., 2024).