ColorMAE: Frequency-Guided Masking for MAE

• The paper introduces ColorMAE, a data-independent masking strategy using filtered random noise to impose frequency-domain priors in MAE.
• It details a mask generation process with four distinct filters (Red, Blue, Green, Purple), with the band-pass (Green) masks yielding optimal performance.
• ColorMAE maintains baseline efficiency by preserving model complexity while significantly improving downstream tasks like semantic segmentation and object detection.

ColorMAE is a data-independent masking strategy for Masked AutoEncoders (MAE) designed to enhance visual representation learning without introducing extra model complexity or data-adaptive computations. It leverages filtered random noise, inspired by color noise in image processing, to impose frequency-domain priors on the masking patterns, affecting both the spatial distribution and semantic content of the masked regions. ColorMAE demonstrates superior downstream performance compared to random masking schemes, particularly in semantic segmentation tasks, while maintaining the simplicity and efficiency of baseline MAE architectures (Hinojosa et al., 2024).

1. Mask Generation via Filtered Random Noise

ColorMAE constructs binary mask patterns by filtering a sampled white-noise image $W \in \mathbb{R}^{H \times W}$ using several linear filters $F_i$ associated with different frequency characteristics. The process consists of four steps:

1. Noise Filtering: The white noise $W$ is convolved with a filter $F_i$ to generate the noise map $N$.
2. Normalization: $N$ is normalized to zero mean and unit variance.
3. Cropping: A random crop is taken to align with the number of patches $P$ used by ViT models.
4. Thresholding: The mask $M \in \{0,1\}^P$ is generated by thresholding the highest $(1-r)P$ elements of $N$, where $r$ is the mask ratio.

Four types of filters establish different spatial and frequency priors:

Filter Name	Operation	Mask Characteristics
Red (Low-pass)	$N_{\text{red}} = G_\sigma * W$	Large contiguous (global context)
Blue (High-pass)	$N_{\text{blue}} = W - G_\sigma * W$	Fine, spatially uniform speckles
Green (Band-pass)	$$N_{\text{green}} = G_{\sigma_1} * W - G_{\sigma_2} * W$$	Medium-sized “islands” (object parts)
Purple (Band-stop)	$N_{\text{purple}} = W - N_{\text{green}}$	Mix of large blobs and fine speckles

Thresholded masks are defined by: $$M_i = \begin{cases} 0,& \text{if } N_i \text{ is in TopK (visible)} \ 1,& \text{otherwise (masked)} \end{cases}$$ This mechanism produces masks that impose specific reconstructive challenges, guided by frequency-domain properties rather than image content.

2. Integration into Masked Autoencoder Framework

In ColorMAE, patch-based input tokenization and the masking workflow are retained from standard MAE. For an input image $x \in \mathbb{R}^{H \times W \times 3}$:

• It is split into $P$ non-overlapping patches and projected linearly to tokens ($x \in \mathbb{R}^{P \times d}$).
• Mask application is performed as follows:

$$x_{\text{vis}} = (1-M) \odot x,\quad x_{\text{mask}} = M \odot x = \{ \texttt{[MASK]} \}$$

• The encoder $\mathcal{E}$ processes only visible patches ($x_{\text{vis}}$), while the decoder $\mathcal{D}$ reconstructs the original input ($\hat{x}$) from latent features $z$ and mask tokens.
• The reconstruction objective is a mean-squared error computed over only masked patches:

$$\mathcal{L} = \frac{1}{\|M\|_1} \sum_{i=1}^{P} M_i \left\lVert x_i - \hat{x}_i \right\rVert_2^2$$

Notably, no auxiliary predictions or adversarial/teacher-guided modules are introduced.

3. Computational Complexity and Resource Requirements

ColorMAE preserves the computational footprint of the baseline MAE framework. Key considerations include:

• No increase in trainable parameters or per-step FLOPs; e.g., ViT-B/16 MAE utilizes 111.91M parameters and 16.87G FLOPs per 224×224 image.
• The only overhead is the storage and use of pre-computed filtered noise tensors, which add approximately 2.8% to the total GPU memory consumption.
• Mask generation incurs negligible CPU/GPU computation.
• In contrast, data-adaptive mask schemes such as HPM and CAE necessitate additional predictors or attention branches, incurring roughly 1.1–1.3× greater training time or parameter counts.

This design choice ensures that increases in pretext task difficulty do not necessitate corresponding increases in model or training complexity.

4. Empirical Evaluation and Performance

ColorMAE was evaluated with MAE pretraining on ImageNet-1K (224×224 resolution, patch size 16, $P=196$) at a mask ratio $r=0.75$ over 100–1600 epochs. Downstream evaluation utilized the following protocols:

• ImageNet classification: 100-epoch fine-tuning, Top-1 accuracy reported.
• ADE20K semantic segmentation: UperNet head, 160K iterations, mIoU as metric.
• COCO detection and instance segmentation: ViTDet backbone, 768×768 input, reporting AP\textsuperscript{bbox}/AP\textsuperscript{mask}.

Key results at 800 pretrain epochs (ViT-B/16):

Task	Random Masking	Green Masking	Absolute Improvement
ImageNet Top-1 (%)	83.17	83.57	+0.40
ADE20K mIoU	46.46	49.18	+2.72
COCO AP\textsuperscript{bbox}	49.15	49.50	+0.35

Among the filter types, only band-pass ("Green") masks consistently yielded improvements over random masking across all tasks. Analysis revealed:

• "Blue" (High-pass): Easiest pretext (lowest reconstruction loss); weakest downstream representations.
• "Red" (Low-pass): Most difficult pretext; poor downstream generalization.
• "Purple" (Band-stop): Intermediate performance.
• "Green" (Band-pass): Balanced difficulty/randomness, optimal downstream results.

5. Interpretation, Limitations, and Prospects

Imposing frequency-domain priors through filtered noise masks introduces data-independent “inductive bias” into masked image modeling, favoring the learning of representations sensitive to specific spatial and semantic structures. In particular, the "Green" (band-pass) masks, by segmenting the image into mid-frequency regions, prompt MAE to reconstruct semantically salient object components without reference to actual input content.

No extra learnable parameters or auxiliary losses are required, preserving the lightweight and modular character of MAE. Limitations include a modest increase (~3%) in GPU memory due to pre-filtered noise banks and the current restriction to simple linear filter types.

Future research directions include dynamic per-image mixing of noise filters, extension to alternative ViT architectures (e.g., Swin, pyramid ViTs), and exploration of hybrid schemes combining data-independent color priors with lightweight data-adaptive elements (Hinojosa et al., 2024). The generation of more sophisticated masking patterns (such as optimal green noise via halftoning) is also a prospective avenue.

This suggests that introducing structured, content-agnostic frequency priors through masking can effectively enhance the challenge and transferability of the MAE pretext task without substantial additional resource requirements or design modifications.