GMM-CMSS: Progressive Masking in RGBT Pre-Training

• GMM-CMSS is a progressive masking strategy for RGBT pre-training that uses a 3-component Gaussian mixture model and a time-varying schedule to shift focus from foreground to background.
• It employs Cross-Modality Structural Sparsity (CMSS) to quantify patch-level alignment between RGB and thermal data, ensuring modality-invariant representation learning.
• Empirical results show that its dynamic masking schedule outperforms fixed and random schemes, leading to significant gains in object detection and cross-modality matching across datasets.

Searching arXiv for the specified paper and closely related work on masking strategies in MAE and graph/vision pretraining. The GMM-CMSS progressive masking strategy is a pre-training mechanism introduced within M-SpecGene, a generalized RGBT multispectral foundation model for self-supervised learning on RGB-thermal data. It combines a cross-modality patch statistic, Cross-Modality Structural Sparsity (CMSS), with a 3-component Gaussian mixture model (GMM) and a time-varying masking schedule that shifts from easy to hard, object-centric reconstruction. The method is designed for RGBT data, where the two modalities exhibit information imbalance and where uniform random masking can miss jointly informative regions. In M-SpecGene, GMM-CMSS is presented as a flexible masking procedure that supports modality-invariant representation learning and contributes to generalizability across eleven datasets and four RGBT downstream tasks (Zhou et al., 22 Jul 2025).

1. Conceptual role within RGBT masked pre-training

GMM-CMSS addresses a specific limitation of random masking in multispectral masked autoencoding. In RGB-thermal settings, objects are often small and sparse, thermal imagery lacks texture, and RGB imagery lacks heat cues. Under these conditions, a uniform mask may cover only background or only modality-sparse regions, which degrades cross-modality alignment. The strategy therefore replaces uniform patch selection with a score-driven sampling process derived from the joint structure of the two modalities.

Its operation is explicitly progressive. Early in training, masking is biased toward high-CMSS regions, described as foreground or “easy”; later, the bias migrates toward low-CMSS regions, described as background or “hard.” This constructs an easy-to-hard curriculum that first emphasizes jointly strong signals and then expands toward less informative context. The intended outcome is a pre-training process that is simultaneously flexible, object-centric, and sensitive to cross-modality information density.

The method is not a generic masking heuristic detached from model architecture. In the pseudocode, patch embeddings are obtained through a ViT linear head, the selected patches are masked, and the masked batch is passed through an MAE encoder-decoder with a reconstruction loss. GMM-CMSS thus functions as the sampling layer that determines which portions of the RGBT input are hidden at each stage of pre-training.

2. Cross-Modality Structural Sparsity

CMSS is the statistic used to quantify the information density shared by RGB and thermal patches. An RGBT image pair is partitioned into $p \times p$ non-overlapping patches. For patch index $i$, the RGB and thermal patch embeddings are denoted by $a_i \in \mathbb{R}^d$ and $b_i \in \mathbb{R}^d$, with Euclidean norms $\|a_i\|$ and $\|b_i\|$, and elementwise variances $\sigma_{a_i}^2$ and $\sigma_{b_i}^2$. The CMSS score is defined as

$$m_i = \mathrm{CMSS}(a_i,b_i) = \frac{1 + \bigl\langle a_i / \|a_i\|,\; b_i / \|b_i\|\bigr\rangle}{2\,\sigma_{a_i}^2\,\sigma_{b_i}^2}\;.$$

The interpretation given for this quantity is direct: a high $m_i$ indicates that the two modalities are both strong and well aligned. The cosine term in the numerator measures directional agreement between normalized RGB and thermal embeddings, while the denominator uses the product of variances. In the description accompanying the method, high-CMSS patches correspond to jointly informative regions, typically associated with foreground structure.

This score is central because the masking policy does not operate on raw intensity or on single-modality saliency. Instead, it is parameterized by a cross-modality statistic intended to capture whether the two modalities provide mutually reinforcing evidence at patch level. This suggests that the masking schedule is aligned not merely with patch importance, but with the reliability of RGB-thermal correspondence.

3. Gaussian mixture modeling and the progressive masking schedule

After CMSS scores are computed, their empirical distribution over the pre-training corpus is modeled with a $i$0-component Gaussian mixture. In practice, the method sets $i$1. The mixture density is

$i$2

with $i$3. For a minibatch of $i$4 images, containing $i$5 patch scores, the E-step computes responsibilities

$i$6

The M-step then updates the mixture means, covariances, and mixing coefficients from the weighted batch statistics. One E-M update is run every training iteration, and the mixture parameters are reported to stabilize after $i$7 early in training.

The progressive component is introduced through a time variable $i$8, defined as the fraction of training completed. A shifted sampling density is constructed as

$i$9

with a simple linear bias schedule

$a_i \in \mathbb{R}^d$0

The stated role of $a_i \in \mathbb{R}^d$1 is to ensure that at $a_i \in \mathbb{R}^d$2 the Gaussians concentrate on the largest-$a_i \in \mathbb{R}^d$3 component, corresponding to foreground, and that as $a_i \in \mathbb{R}^d$4 they move toward the smallest-$a_i \in \mathbb{R}^d$5 component, corresponding to background. Given a new batch, the model computes CMSS scores $a_i \in \mathbb{R}^d$6, evaluates $a_i \in \mathbb{R}^d$7 for each patch, normalizes the resulting values to sum to $a_i \in \mathbb{R}^d$8, and samples a fixed fraction $a_i \in \mathbb{R}^d$9 of patches to be masked by treating $b_i \in \mathbb{R}^d$0 as the masking probability.

A distinctive property of this schedule is that it does not statically privilege either high- or low-CMSS regions. Instead, it shifts between them over the course of optimization. This temporal migration is the core of the easy-to-hard formulation.

4. Training procedure and implementation parameters

The high-level training loop consists of seven recurrent operations: compute patch embeddings, compute CMSS scores, perform the E-step, perform the M-step, build the progressive sampling density, sample a fixed fraction of patches to mask, and apply MAE reconstruction on the masked batch. GMM parameters are initialized by a quick pass over the data, and then refined online.

The reported implementation choices are specific. The mask ratio is fixed at $b_i \in \mathbb{R}^d$1 throughout training. The number of GMM components is $b_i \in \mathbb{R}^d$2, and the mixture is updated every batch by EM. The bias term $b_i \in \mathbb{R}^d$3 is ramped linearly from $b_i \in \mathbb{R}^d$4 to $b_i \in \mathbb{R}^d$5 over $b_i \in \mathbb{R}^d$6 epochs. Optimization uses AdamW with base learning rate $b_i \in \mathbb{R}^d$7, cosine decay, and $b_i \in \mathbb{R}^d$8 GPUs. Pre-training runs for 1,600 epochs, following MAE defaults.

These parameters indicate that GMM-CMSS is intended to operate under a high-mask-ratio masked autoencoding regime rather than under a lightly corrupted input setting. The online EM update also means that the masking distribution is not precomputed once and frozen; it co-evolves with the training dynamics as patch embeddings change.

The pseudocode further shows that masking is sampled from normalized scores over all patches in the current batch, rather than by thresholding CMSS independently per patch. This detail matters because the method implements a probabilistic allocation of a fixed masking budget.

5. Empirical effects, ablations, and interpretive scope

The ablation reported for Table 7(c) isolates the role of the masking policy on $b_i \in \mathbb{R}^d$9:

Masking strategy	$\|a_i\|$0
Random masking	83.8 %
Mask only high-CMSS	83.4 %
Mask only low-CMSS	83.6 %
GMM-CMSS progressive	84.8 %

The ablation is significant because it excludes the trivial interpretation that performance gains arise merely from preferring one CMSS regime. Masking only high-CMSS patches underperforms the progressive schedule, and masking only low-CMSS patches also underperforms it. The reported outcome therefore supports the specific value of the temporal progression, not just the underlying CMSS score.

Quantitative gains are reported on several downstream settings. On FLIR object detection, the MAE(IN1K) baseline reaches $\|a_i\|$1 $\|a_i\|$2, whereas M-SpecGene with GMM-CMSS reaches $\|a_i\|$3, a gain of $\|a_i\|$4 points. On KAIST pedestrian detection, the values given are $\|a_i\|$5 for random masking and $\|a_i\|$6 for GMM-CMSS in $\|a_i\|$7. On cross-modality matching for LLVIP measured by AUC@$\|a_i\|$8, from-scratch and Sup(IN1K) are reported to drop to approximately $\|a_i\|$9, while M-SpecGene reaches $\|b_i\|$0.

The accompanying visual analyses, described as t-SNE and Wasserstein-distance histograms, are said to show tighter object-cluster alignment and larger object-background separations in feature space. This suggests that the masking strategy alters representation geometry in a way consistent with the object-centric rationale advanced for the method.

A common misconception is that any non-random density-biased masking should be sufficient. The ablation contradicts that view: neither always-high-CMSS nor always-low-CMSS masking matches the progressive variant. The empirical record presented for GMM-CMSS is therefore specifically about a time-varying masking distribution.

6. Relation to structure-guided progressive masking in other domains

GMM-CMSS belongs to a broader class of progressive masking strategies that replace uniform random masking with structure-aware selection. A closely related example appears in graph masked autoencoders: StructMAE extends standard GMAE by replacing uniform random masking with a two-stage, structure-guided, progressive masking strategy composed of Structure-Based Scoring and Structure-Guided Masking. There, each node receives a structural-importance score, using either PageRank or a learnable scorer, and masking moves from random behavior toward higher-score nodes over training (Liu et al., 2024).

The two methods are defined on different data structures and different notions of structure. In StructMAE, the prior is graph topology, and the easy-to-hard schedule is implemented through node ranking and a progressively increasing top-$\|b_i\|$1 guided set. In GMM-CMSS, the prior is cross-modality patch structure, and the schedule is implemented through a 3-component GMM whose mean bias shifts from high- to low-CMSS regions. The commonality is therefore architectural at the level of training philosophy rather than scoring mechanics.

This suggests a useful taxonomy. StructMAE can be understood as topology-guided progressive masking for graph reconstruction, whereas GMM-CMSS is cross-modality density-guided progressive masking for RGBT masked autoencoding. In both cases, the masking policy is treated as a learnable or data-adaptive component of pre-training rather than as a neutral corruption operator. That shared perspective helps situate GMM-CMSS within the broader development of structure-guided self-supervision, while preserving its domain-specific design for multispectral fusion.