Synchronized Class-Token Fusion for Remote Sensing

• SCTF is an architectural framework that fuses high-level class tokens across modality-specific transformer encoders to improve multi-label remote sensing classification.
• Its iterative synchronization mechanism retains modality-specific features while aggregating complementary signals, overcoming early fusion limitations.
• Experimental results on BigEarthNet-MM demonstrate enhanced performance metrics compared to single-modality and early fusion approaches.

Synchronized Class-Token Fusion (SCTF) is an architectural framework designed to advance multi-modal multi-label classification (MLC) of remote sensing (RS) imagery. The core idea is to enable modality-specific transformer encoders to iteratively exchange distilled, high-level representations by synchronizing their "class tokens" after each transformer block. This process permits specialized processing per modality while effectively aggregating cross-modal information, resulting in representations that are both robust and semantically rich for complex MLC tasks (Hoffmann et al., 2023).

1. Problem Setting and Motivation

The SCTF paradigm targets the classification of $N$ co-located RS images (modalities), $\{x^1, \ldots, x^N\}$, where each observation $x^j$ may differ in resolution, channel count, or sensor characteristics. The goal is to predict an $l$-dimensional multi-hot label $y \in \{0,1\}^l$. Traditional methods approach this task via either single-modality networks, which fail to leverage complementary signals, or early-fusion networks, which forcibly concatenate modalities—potentially conflating heterogeneous data distributions and losing modality-specific inductive biases. SCTF seeks to balance these extremes, offering a structured mechanism to fuse high-level information through synchronized class tokens that summarize per-modality evidence at each network depth (Hoffmann et al., 2023).

2. Architecture: Modality-Specific Transformers and Class-Token Synchronization

Each modality $j$ is processed by a dedicated Vision Transformer (ViT) encoder. The workflow for each image $x^j$ involves:

• Partitioning into $k_{p_j}$ non-overlapping patches of size $p_j \times p_j$.
• Linear embedding of each patch into a shared $d_e$-dimensional space: $\{x^1, \ldots, x^N\}$0.
• Prepending a learnable class token $\{x^1, \ldots, x^N\}$1 to form an augmented sequence: $\{x^1, \ldots, x^N\}$2.
• Passing $\{x^1, \ldots, x^N\}$3 through $\{x^1, \ldots, x^N\}$4 stacked transformer encoder blocks. Within each block, the class token participates as a peer, aggregating global modality-specific features via self-attention.

After each transformer block $\{x^1, \ldots, x^N\}$5, the architecture synchronizes class tokens across modalities according to a precise mathematical protocol.

3. Mathematical Mechanism and Synchronization Process

The synchronization at block $\{x^1, \ldots, x^N\}$6 operates as follows:

1. Class Token Extraction and Concatenation: Each updated class token $\{x^1, \ldots, x^N\}$7 (for $\{x^1, \ldots, x^N\}$8) is concatenated to form

$\{x^1, \ldots, x^N\}$9

2. Trainable Fusion Transformation: The fused class token is generated via a learned linear projection $x^j$0:

$x^j$1

where $x^j$2 and $x^j$3 are trainable parameters.

3. Token Redistribution: The fused token $x^j$4 replaces the class token in each modality’s token sequence for the next block:

$x^j$5

At the final transformer depth $x^j$6, $x^j$7 is fed to a classification head $x^j$8 (a linear layer plus sigmoid activation) to produce multi-label predictions $x^j$9. End-to-end training employs the standard multi-label binary cross-entropy loss: $l$0

4. Comparison to Single-Modality and Early Fusion Approaches

SCTF stands in contrast to two canonical alternatives:

• Single-modality ViT: Each encoder processes only its assigned bands or channels, inherently limiting cross-modal context. While effective for unimodal patterns, this approach ignores valuable complementary information.
• Early fusion: All bands/channels are concatenated and processed jointly from the input layer. This can dilute modality-specific feature extraction and induce learning difficulties rooted in divergent spectral or spatial statistics, often leading to reduced generalization.

SCTF, by contrast, orchestrates a repeated exchange of class tokens at each depth. This mechanism allows every encoder to retain modality-specific inductive biases yet incrementally aggregate a distilled summary of all modalities' high-level cues. Empirical results indicate that this iterative “round-robin” exchange yields sharper, more robust feature representations, particularly on heterogeneous and multi-source RS datasets (Hoffmann et al., 2023).

5. Implementation Details and Experimental Evaluation

SCTF was evaluated on the BigEarthNet-MM dataset (590,326 Sentinel-1 SAR and Sentinel-2 MSI pairs, 19 land-cover classes) using the following configuration:

• ViT Backbone per Modality: patch size $l$1, embedding dimension $l$2, depth $l$3, 8 attention heads.
• Initialization: Patch embeddings and class tokens via Gaussian distribution (small $l$4), fusion layer $l$5 via Xavier initialization.
• Training setup: Adam optimizer ($l$6), 60 epochs, data augmentations including RandomSensorDrop (randomly dropping a modality), random flip, random crop, and stochastic depth 0.25.
• Performance metrics: micro-average AP, macro-average AP, macro-F$l$7.

The following table summarizes key results:

Architecture	Micro-AP	Macro-AP	Macro-F$l$8
Sentinel-1 ViT	0.8128	0.6796	0.5581
Sentinel-2 ViT	0.8910	0.8159	0.7125
Early fusion ViT	0.8956	0.8227	0.7253
SCT Fusion	0.9015	0.8343	0.7369

SCTF provided gains of +1.16 percentage points in macro-AP and +1.6 percentage points in macro-F$l$9 over early fusion, and a +15 point macro-AP increase compared to single-modality SAR. The computational complexity scales roughly by a factor of $y \in \{0,1\}^l$0 (number of modalities) compared to a single ViT but is tunable via hyperparameters such as $y \in \{0,1\}^l$1, network depth, and patch size.

6. Architectural Flexibility and Theoretical Implications

SCTF’s design enables learning both modality-specialized and cross-modal shared feature spaces without entangling low-level sensor signals. The synchronized class token acts as a compact, learned summary of modality evidence at each block, and the fusion layer adaptively emphasizes the most discriminative cross-modal cues for the downstream MLC objective. The process can be interpreted as dynamically reinforcing the most salient, task-relevant features iteratively at every depth, rather than imposing rigid early or late fusion strategies.

A plausible implication is that SCTF may generalize to other contexts involving heterogeneous modalities with distinct noise profiles or spatial/spectral statistics, provided that the class token carries meaningful semantic information per modality and that the fusion transformation is sufficiently expressive (Hoffmann et al., 2023).

7. Limitations and Considerations

SCTF incurs approximately $y \in \{0,1\}^l$2 times the compute cost of a single-modality ViT, owing to parallel encoder processing for each modality. This cost can be mitigated by reducing embedding dimensions, network depth, or by customizing patch sizes per modality. Furthermore, while the method strictly relies on cross-entropy classification loss without auxiliary supervision on class tokens, the observed empirical gains indicate robust cross-modal feature integration for the challenging RS MLC benchmark. No evidence of auxiliary loss benefits was reported, and further research could assess the extensibility of SCTF under alternative supervision schemes or diverse multi-modal collections.

---

Reference

• "Transformer-based Multi-Modal Learning for Multi Label Remote Sensing Image Classification" (Hoffmann et al., 2023)