M-SpecGene: RGBT Multispectral Foundation Model

• The paper introduces a self-supervised Siamese masked autoencoder that replaces task-specific fusion pipelines with a single modality-invariant backbone.
• It employs a novel Cross-Modality Structural Sparsity (CMSS) metric and a progressive masking curriculum to enhance object-centric and cross-modal feature learning.
• Extensive evaluations demonstrate improved detection, segmentation, feature matching, and saliency performance across diverse RGBT datasets.

M-SpecGene designates a generalized foundation model for RGB–thermal multispectral vision that replaces task-specific fusion pipelines with a single self-supervised pretrained backbone for multiple downstream tasks. It is formulated as a Siamese masked autoencoding framework with a shared-weight ViT encoder, modality-specific decoders, and a masking curriculum driven by a new Cross-Modality Structural Sparsity metric. The model is presented as the first generalized RGBT multispectral foundation model, is pretrained on the curated RGBT550K corpus, and is evaluated on four downstream RGBT tasks across eleven datasets, where it is adapted with simple generic heads rather than bespoke fusion modules (Zhou et al., 22 Jul 2025).

1. Problem formulation and conceptual context

RGBT multispectral vision uses paired visible-spectrum RGB imagery and thermal imagery to obtain complementary perception under adverse conditions. In the formulation used by M-SpecGene, RGB contributes texture, color, and fine detail under favorable illumination, whereas thermal is robust to low light, fog, smog, and illumination changes but is lower-texture and often lower-resolution. The paper identifies four structural obstacles in existing RGBT research: modality bias, information imbalance, data bottleneck, and artificial inductive bias induced by case-by-case model design (Zhou et al., 22 Jul 2025).

“Modality bias” denotes the asymmetry created when RGB-pretrained backbones are reused for thermal imagery, causing filters to reflect RGB statistics more strongly than thermal statistics. “Information imbalance” has both spatial and modal forms: foreground objects are often small and non-object-centric, and the information density of RGB and thermal varies strongly with scene conditions. “Data bottleneck” refers to the cost of collecting pixel-aligned RGB–thermal data at scale. “Artificial inductive bias” refers to the proliferation of task-specific fusion modules—attention blocks, graph structures, or dedicated transformers—whose design assumptions are typically narrow and not naturally reusable across tasks (Zhou et al., 22 Jul 2025).

This positioning is best understood against earlier RGBT literature. MANet introduced explicit modality-shared, modality-specific, and instance-aware adapters for tracking (Li et al., 2019). M$^5$L added hard-sample-aware metric structuring and modality-attention fusion (Tu et al., 2020). Challenge-aware RGBT tracking used modality-shared and modality-specific challenge branches with cross-modal guidance (Li et al., 2020). CMM addressed modality bias causally through mode-dependent removal of thermal direct effects (Kim et al., 2024). MS2Fusion later formalized shared and complementary cross-modal interactions through state-space fusion (Shen et al., 19 Jul 2025). M-SpecGene departs from all of these by making the backbone itself the primary locus of multispectral generalization rather than attaching highly specialized fusion logic to each downstream task (Zhou et al., 22 Jul 2025).

A related issue arises from the broader foundation-model literature in remote sensing: masked autoencoding is not automatically well aligned with all downstream multispectral tasks, and its effectiveness depends on pretext-task alignment (Xie et al., 2024). This suggests that M-SpecGene’s central innovation is not masked autoencoding alone, but masked autoencoding coupled to an object-centric, modality-aware masking policy tailored to RGBT information structure.

2. Architecture and pretraining framework

The pretraining model is a Siamese masked autoencoder built on a ViT backbone. Paired RGB and thermal images are patchified into tokens, complementary masking is applied to the two modalities, and both streams are processed by a single shared-weight encoder $E_{\theta}$. If RGB patch embeddings are denoted $A_{rgb}=\{a_i\}_{i=1}^{p\times p}$ and thermal embeddings $B_t=\{b_i\}_{i=1}^{p\times p}$, with $a_i,b_i\in\mathbb{R}^{768}$, then after masking with $M_{rgb}$ and $M_t$, the visible patches are encoded as

$z_i^{rgb}=E_{\theta}(a_i), \quad i\notin M_{rgb}$

and

$z_i^{t}=E_{\theta}(b_i), \quad i\notin M_t.$

Weight sharing is the key architectural device for encouraging modality-invariant representation learning (Zhou et al., 22 Jul 2025).

A cross-attention layer is inserted after the encoder to exchange complementary information between latent RGB and thermal tokens. In schematic form,

$$\tilde{Z}^{rgb}=\mathrm{Attn}(Q=Z^{rgb},K=Z^t,V=Z^t),\qquad \tilde{Z}^{t}=\mathrm{Attn}(Q=Z^{t},K=Z^{rgb},V=Z^{rgb}).$$

The reconstruction stage uses two independent ViT decoders, $E_{\theta}$0 and $E_{\theta}$1, which reconstruct masked RGB and thermal content separately from the fused latent states. The total self-supervised objective is the sum of modality-wise masked reconstruction losses,

$E_{\theta}$2

with each term computed over masked patches only (Zhou et al., 22 Jul 2025).

Fine-tuning reuses only the pretrained encoder. RGB and thermal feature tensors $E_{\theta}$3 are concatenated along the batch dimension, encoded jointly, and then reshaped into $E_{\theta}$4, yielding a simple channel-concatenation fusion without handcrafted attention or gating modules. Downstream heads are standard: ViTDet for object detection, UperNet for semantic segmentation and salient object detection, and a LoFTR-style matcher for cross-modality feature matching. Fine-tuning is full-parameter rather than frozen-backbone adaptation (Zhou et al., 22 Jul 2025).

The architecture was compared against several alternative RGBT pretraining layouts. On FLIR detection, a vanilla MAE that mixes RGB and thermal at input achieved $E_{\theta}$5, channel concatenation achieved $E_{\theta}$6, an auxiliary-branch variant achieved $E_{\theta}$7, and the Siamese design achieved $E_{\theta}$8; with the full GMM-CMSS masking strategy, the final system reached $E_{\theta}$9 (Zhou et al., 22 Jul 2025). This establishes the shared-encoder, dual-decoder symmetry as the preferred representation geometry within the paper’s design space.

3. Cross-Modality Structural Sparsity and progressive masking

The core methodological contribution is the Cross-Modality Structural Sparsity metric, CMSS, which quantifies how structurally informative and cross-modally consistent a patch is. For an RGB patch embedding $A_{rgb}=\{a_i\}_{i=1}^{p\times p}$0 and a thermal patch embedding $A_{rgb}=\{a_i\}_{i=1}^{p\times p}$1, CMSS is defined as

$A_{rgb}=\{a_i\}_{i=1}^{p\times p}$2

The cosine-similarity term captures cross-modal agreement, while the variance term reflects structural richness within each modality. High CMSS therefore corresponds to object-like, salient, information-dense, cross-modally consistent regions; low CMSS corresponds to flat regions, noisy regions, or modality-disagreeing regions (Zhou et al., 22 Jul 2025).

The CMSS scores over patches are modeled with a Gaussian mixture

$A_{rgb}=\{a_i\}_{i=1}^{p\times p}$3

with $A_{rgb}=\{a_i\}_{i=1}^{p\times p}$4 by default. The GMM is updated dynamically during training rather than estimated once offline, because encoder embeddings evolve over epochs. After fitting the CMSS distribution, patch sampling is driven by the bias-shifted sampling function

$A_{rgb}=\{a_i\}_{i=1}^{p\times p}$5

where $A_{rgb}=\{a_i\}_{i=1}^{p\times p}$6 changes over training to move the distribution from easy, high-CMSS regions toward harder, low-CMSS regions (Zhou et al., 22 Jul 2025).

This yields an information-aware masking curriculum. Early in training, visible patches are sampled from high-CMSS regions, concentrating learning on object-centric and strongly aligned RGB–thermal content. Later in training, the bias shifts toward lower-CMSS regions, forcing the model to handle background, weak modalities, and harder contexts. The paper characterizes this as a flexible, easy-to-hard, object-centric pretraining process (Zhou et al., 22 Jul 2025).

The masking strategy is empirically important. On FLIR detection with ViT-B, random masking reached $A_{rgb}=\{a_i\}_{i=1}^{p\times p}$7; masking driven only by low-CMSS Gaussian sampling achieved $A_{rgb}=\{a_i\}_{i=1}^{p\times p}$8; masking driven only by high-CMSS Gaussian sampling achieved $A_{rgb}=\{a_i\}_{i=1}^{p\times p}$9; GMM-CMSS progressive masking achieved $B_t=\{b_i\}_{i=1}^{p\times p}$0 (Zhou et al., 22 Jul 2025). Decoder depth $B_t=\{b_i\}_{i=1}^{p\times p}$1 was best among $B_t=\{b_i\}_{i=1}^{p\times p}$2, and a masking ratio of $B_t=\{b_i\}_{i=1}^{p\times p}$3 was best among $B_t=\{b_i\}_{i=1}^{p\times p}$4 (Zhou et al., 22 Jul 2025). A plausible implication is that the masking curriculum acts as a form of pretext-task alignment for RGBT data, compensating for weaknesses of uniform MAE masking that have been observed in other multispectral settings (Xie et al., 2024).

4. Data curation and optimization pipeline

The pretraining corpus begins from RGBT3M, described as roughly 3 million RGB–thermal pairs collected from 41 datasets and 10 tasks. Because this source pool contains dataset imbalance, temporal redundancy, and low-quality pairs, the final RGBT550K corpus is produced through three filtering steps: dataset balancing, temporal sampling, and image-quality filtering using SSIM. Only aligned pairs with $B_t=\{b_i\}_{i=1}^{p\times p}$5 are retained, yielding 548,238 high-quality RGB–thermal pairs with diverse scenes, tasks, lighting conditions, resolutions, and object categories (Zhou et al., 22 Jul 2025).

The optimization pipeline has two phases. First, encoder and decoders are pretrained unimodally with MAE-style self-supervision on ImageNet for RGB and on single-modality thermal datasets. Second, joint RGBT self-supervised pretraining is performed on RGBT550K with the full M-SpecGene pipeline (Zhou et al., 22 Jul 2025). Data augmentation uses random cropping with scale between $B_t=\{b_i\}_{i=1}^{p\times p}$6 and $B_t=\{b_i\}_{i=1}^{p\times p}$7 and random horizontal flipping with probability $B_t=\{b_i\}_{i=1}^{p\times p}$8. The default masking ratio is $B_t=\{b_i\}_{i=1}^{p\times p}$9 for both modalities. Optimization uses AdamW with base learning rate $a_i,b_i\in\mathbb{R}^{768}$0 and half-cycle cosine decay on $a_i,b_i\in\mathbb{R}^{768}$1 NVIDIA RTX 4090 GPUs (Zhou et al., 22 Jul 2025).

The reliance on high-quality aligned pairs is a defining assumption of the pretraining corpus. By contrast, later RGBT video detection work has emphasized alignment-free settings and raw sensor mismatch as a central obstacle (Wang et al., 2024). This suggests that M-SpecGene’s current data regime is optimized for representation quality under alignment rather than robustness to large geometric misregistration.

5. Downstream adaptation and empirical generalization

M-SpecGene is evaluated on four downstream tasks spanning eleven datasets: multispectral object detection, multispectral semantic segmentation, RGBT cross-modality feature matching, and multispectral salient object detection (Zhou et al., 22 Jul 2025).

Task	Datasets / head	Representative result
Detection	KAIST, FLIR, LLVIP / ViTDet	FLIR ViT-B: mAP $a_i,b_i\in\mathbb{R}^{768}$2, $a_i,b_i\in\mathbb{R}^{768}$3
Segmentation	SemanticRT, MVSeg, FMB / UperNet	MVSeg ViT-B: mIoU $a_i,b_i\in\mathbb{R}^{768}$4
Feature matching	LLVIP / LoFTR-style	ViT-S: $a_i,b_i\in\mathbb{R}^{768}$5 AUC at $a_i,b_i\in\mathbb{R}^{768}$6
Saliency detection	VT821, VT1000, VT5000, VI-RGBT1500 / UperNet	VI-RGBT1500 ViT-B: $a_i,b_i\in\mathbb{R}^{768}$7, $a_i,b_i\in\mathbb{R}^{768}$8, $a_i,b_i\in\mathbb{R}^{768}$9, MAE $M_{rgb}$0

For detection, M-SpecGene(ViT-B) achieved $M_{rgb}$1 on KAIST, $M_{rgb}$2 and $M_{rgb}$3 on FLIR, and $M_{rgb}$4 with $M_{rgb}$5 on LLVIP (Zhou et al., 22 Jul 2025). For segmentation, it achieved $M_{rgb}$6 mIoU on SemanticRT, $M_{rgb}$7 on MVSeg, and $M_{rgb}$8 on FMB (Zhou et al., 22 Jul 2025). For cross-modality matching on LLVIP, M-SpecGene(ViT-S) achieved $M_{rgb}$9, $M_t$0, and $M_t$1 AUC at $M_t$2, $M_t$3, and $M_t$4, respectively (Zhou et al., 22 Jul 2025). For salient object detection, the ViT-B model outperformed prior methods across VT821, VT1000, VT5000, and VI-RGBT1500, with the latter yielding $M_t$5, $M_t$6, $M_t$7, and MAE $M_t$8 (Zhou et al., 22 Jul 2025).

The pretraining effect is unusually clear in the FLIR/LLVIP ablation. From scratch, the model obtained $M_t$9 on FLIR and $z_i^{rgb}=E_{\theta}(a_i), \quad i\notin M_{rgb}$0 at $z_i^{rgb}=E_{\theta}(a_i), \quad i\notin M_{rgb}$1 on LLVIP matching. Supervised ImageNet-1K pretraining increased detection to $z_i^{rgb}=E_{\theta}(a_i), \quad i\notin M_{rgb}$2 but reduced matching to $z_i^{rgb}=E_{\theta}(a_i), \quad i\notin M_{rgb}$3. MAE pretraining on ImageNet increased detection further to $z_i^{rgb}=E_{\theta}(a_i), \quad i\notin M_{rgb}$4 but reduced matching to $z_i^{rgb}=E_{\theta}(a_i), \quad i\notin M_{rgb}$5. M-SpecGene pretraining achieved $z_i^{rgb}=E_{\theta}(a_i), \quad i\notin M_{rgb}$6 on FLIR and $z_i^{rgb}=E_{\theta}(a_i), \quad i\notin M_{rgb}$7 on LLVIP matching, improving both detection and cross-modal correspondence simultaneously (Zhou et al., 22 Jul 2025). The paper interprets this as evidence that conventional RGB pretraining improves unimodal pattern learning but can damage cross-modality symmetry, whereas joint RGBT pretraining yields modality-invariant features.

Feature analysis reinforces this interpretation. Object–background Wasserstein distance was largest for M-SpecGene features among the compared pretraining schemes, and t-SNE plots showed more compact and better-separated object clusters (Zhou et al., 22 Jul 2025). This is consistent with the claim that GMM-CMSS induces object-centric representation learning even though the raw RGBT corpus is not naturally object-centric.

6. Relation to adjacent multispectral foundation-model designs

M-SpecGene belongs to a broader transition from handcrafted multimodal fusion to reusable spectral backbones. In the RGBT literature, earlier architectures typically factorized the problem into modality-shared and modality-specific branches, hard-example metric shaping, or challenge-conditioned adapters (Li et al., 2019, Tu et al., 2020, Li et al., 2020). M-SpecGene preserves the importance of symmetry and shared latent space but relocates those concerns into self-supervised backbone pretraining rather than downstream fusion engineering (Zhou et al., 22 Jul 2025).

In parallel spectral-foundation work, a general-purpose spectral MAE for proximal and remote sensing used wavelength-aware spectral encoding, channel-wise patching, and spatial–spectral masking to support arbitrary band counts (Laprade et al., 3 Mar 2025). SpectralX later showed that frozen RGB-centric RSFMs can be adapted to diverse spectral inputs with HyperT, AoMoA, and Are-adapter, updating only $z_i^{rgb}=E_{\theta}(a_i), \quad i\notin M_{rgb}$8M of $z_i^{rgb}=E_{\theta}(a_i), \quad i\notin M_{rgb}$9M parameters (Zhang et al., 3 Aug 2025). These works are not RGBT-specific, but they establish two complementary trajectories: direct multispectral pretraining, as in M-SpecGene, and parameter-efficient adaptation of frozen optical backbones, as in SpectralX.

A plausible synthesis is that M-SpecGene and spectral PEFT frameworks solve different parts of the same problem. M-SpecGene provides a native RGBT pretraining recipe with modality-aware masking and cross-attention. SpectralX suggests a route for extending such recipes to heterogeneous multispectral settings with frozen backbones and attribute-aware adapters (Zhang et al., 3 Aug 2025). Likewise, causal debiasing in CMM indicates that modality-invariant pretraining does not by itself eliminate spurious modality shortcuts, especially in skewed day/night regimes (Kim et al., 2024). This suggests that future generalized RGBT foundation models may need both strong self-supervised symmetry and explicit debiasing or routing mechanisms.

7. Limitations and open directions

The paper identifies several limitations directly. RGBT550K, although large and diverse, still inherits the biases of existing RGB–thermal datasets. Thermal images remain low-texture and sensor-limited, which constrains reconstruction quality. GMM-CMSS adds nontrivial bookkeeping through per-patch CMSS estimation and dynamic GMM updates. The current design is limited to RGB+thermal rather than broader spectral sets, and it evaluates only four downstream tasks (Zhou et al., 22 Jul 2025).

Several directions follow naturally. Extending CMSS and GMM-CMSS to more than two modalities is explicitly suggested in the paper, as is combining CMSS-based masking with other pretext tasks such as cross-modal contrastive learning or temporal prediction (Zhou et al., 22 Jul 2025). A plausible implication is that alignment-free RGBT scenarios, now emphasized in unaligned video detection benchmarks, would require extending the present aligned-pair assumption with geometric robustness mechanisms (Wang et al., 2024). Another plausible implication is that parameter-efficient adaptation schemes like HyperT, AoMoA, and Are-adapter could become relevant if M-SpecGene-scale backbones are transferred across sensors or domains where full-parameter tuning is undesirable (Zhang et al., 3 Aug 2025).

Within the paper’s stated scope, however, the central claim is narrower and more specific: carefully designed self-supervised pretraining with a symmetric shared encoder, dual decoders, and CMSS-driven progressive masking is sufficient to produce a single RGBT backbone that transfers across detection, segmentation, feature matching, and saliency with simple generic heads rather than bespoke fusion modules (Zhou et al., 22 Jul 2025).