---
title: Wavelet Domain Masked Image Modeling
url: https://www.emergentmind.com/topics/wavelet-domain-masked-image-modeling-w-mim
type: topic
---

# Wavelet Domain Masked Image Modeling

Searching arXiv for the cited papers and closely related MIM baselines.
Wavelet-domain Masked Image Modeling (W-MIM) denotes a class of masked reconstruction methods in which the supervisory signal, the masking operation, or both are defined in a wavelet representation rather than directly in pixel space. In the image-representation setting of "Wavelet-Driven Masked Image Modeling: A Path to Efficient Visual Representation" [2503.00782], wavelet coefficients at multiple decomposition levels are used as compact reconstruction targets for masked image modeling. In the HDR video setting of "Wavelet-Domain Masked Image Modeling for Color-Consistent HDR Video Reconstruction" [2602.07393], masking is applied directly to wavelet subbands to pre-train an encoder for color-consistent reconstruction. Across both formulations, the central premise is that pixel-based MIM overemphasizes redundant fine detail, whereas wavelet decomposition supplies frequency-aware, spatially localized, and multi-level structure that can be aligned with hierarchical neural architectures.

## 1. Conceptual definition and scope

Masked Image Modeling (MIM) is presented as a self-supervised paradigm for learning scalable visual representations, but the wavelet-domain variants argue that images inherently contain abundant redundant information, causing pixel-based reconstruction to focus excessively on finer details such as textures and thereby prolong training times unnecessarily [2503.00782]. W-MIM addresses this by shifting reconstruction from raw pixels to a compact representation of features in the wavelet domain.

The key distinction from Fourier-based frequency analysis is explicit in the underlying formulation: wavelet transform not only offers frequency information but also preserves spatial characteristics and multi-level features of the image [2503.00782]. In the video-reconstruction formulation, this property is extended to color and detail restoration, where wavelet-domain masking is used to reconstruct brightness, color, and fine structure from low dynamic range inputs [2602.07393].

Two concrete instantiations define the current scope of W-MIM. The first is **WaMIM**, an image pre-training framework that assigns different wavelet bands to different encoder depths and optimizes a multi-scale reconstruction loss [2503.00782]. The second is **WMNet**, an HDR video reconstruction network in which W-MIM is the Phase I self-reconstruction pre-training strategy preceding temporal modeling with T-MoE and DMM [2602.07393]. This suggests that W-MIM is better understood as a design pattern—wavelet-structured masking and reconstruction—than as a single fixed architecture.

## 2. Wavelet decomposition as the representational substrate

In WaMIM, a 2D image \(x\in\mathbb{R}^{M\times N}\) undergoes a one-level discrete wavelet transform (DWT) that produces one low-frequency approximation coefficient map \(w^0_{\phi}\) and three high-frequency detail maps \(w^0_{\psi,i}\), \(i\in\{H,V,D\}\) [2503.00782]. In practice, DWT is implemented by separable filter banks, for example Haar, and recursively applied to the low-frequency branch. After \(J\) levels, the decomposition is written as
\[
[\,w^0_{\phi};\; w^0_{\psi},\,w^1_{\psi},\,\dots,\,w^{J}_{\psi}\;] = f_w(x),
\]
where each \(w^j_{\psi}\) bundles the three detail maps \(\{w^j_{\psi,H},w^j_{\psi,V},w^j_{\psi,D}\}\) [2503.00782].

In WMNet, the corresponding frame-level formulation uses a discrete 2-D Haar wavelet transform \(W(\cdot)\) at \(N\) levels for an LDR frame \(x\in\mathbb{R}^{H\times W\times 3}\). At each level \(n=1\ldots N\), the transform yields four subbands: one approximation \(LL^{(n)}\) and three detail bands \(LH^{(n)}, HL^{(n)}, HH^{(n)}\). With separable filters
\[
\alpha^\top = \frac{1}{\sqrt{2}}[1,1], \qquad \beta^\top = \frac{1}{\sqrt{2}}[-1,1],
\]
the single-level transform is defined by
\[
LL^{(1)} = (\alpha\otimes\alpha) * x,\qquad LH^{(1)} = (\alpha\otimes\beta) * x,
\]
\[
HL^{(1)} = (\beta\otimes\alpha) * x,\qquad HH^{(1)} = (\beta\otimes\beta) * x,
\]
with higher levels recursively applied to the preceding \(LL\) branch [2602.07393].

A common misconception is that frequency-domain modeling necessarily discards localization. The wavelet formulations used here do not: the image paper explicitly contrasts wavelets with the commonly used Fourier transform on the grounds that wavelets preserve spatial characteristics and multi-level features, while the video paper operationalizes that claim by reconstructing masked wavelet coefficients back into an image through inverse DWT before encoding [2503.00782; 2602.07393].

## 3. Multi-scale target assignment and loss design in WaMIM

WaMIM selects \(K\) encoder layers \(l_1<l_2<\cdots<l_K\le L\) and sets \(K=J+1\). The assignment rule is depth-to-frequency aligned: shallow layers reconstruct high-frequency details, while deep layers reconstruct low-frequency content. For layer \(l_k\), the target level is
\[
j = J+1-k.
\]
The target generator is defined as
\[
f_t^{l_k}(x) =
\begin{cases}
w^{J+1-k}_{\psi}, & k\neq K,\\
[\,w^{J+1-k}_{\phi},\,w^{J+1-k}_{\psi}\,], & k=K,
\end{cases}
\]
where \([\cdot,\cdot]\) denotes spatial-channel concatenation [2503.00782].

The masking operator is \(f_m\), the encoder is \(f_{EN}\), and the layer-wise decoder is \(f_{DE}^{l_k}\). If
\[
\hat y^{l_k}=f_{DE}^{l_k}\bigl(f_{EN}(f_m(x))\bigr),
\]
and \(\mathcal{D}(a,b)=\|a-b\|_2^2\), the total loss is
\[
\mathcal{L}
= \sum_{k=1}^{K-1}
c^{l_k}\,\bigl\|\hat y^{l_k}-w^{J+1-k}_{\psi}\bigr\|_2^2
+
c^{l_K}\,\bigl\|\hat y^{l_K}-[\,w^0_{\phi},\,w^0_{\psi}\,]\bigr\|_2^2.
\]
An equivalent form is
\[
\mathcal{L}=\sum_{j=0}^J\alpha_j\|\hat w^j-w^j\|^2,
\]
where \(\alpha_j=c^{l_{K-j}}\) [2503.00782].

Architecturally, the framework plugs in any ViT or Swin backbone, masks patches at the input with mask ratio \(75\%\), extracts hidden features \(\{z^{l_k}\}_{k=1}^K\) from selected layers, and attaches to each chosen layer a light-weight decoder consisting of one Transformer block plus a small linear head [2503.00782]. The multi-scale targets are fed independently to each decoder; there is no pooling or mixing across scales. This clean separation is explicitly intended to let each decoder focus on one frequency band while fusing spatial structure with a frequency-domain target.

## 4. Training protocol, ablations, and reported performance of WaMIM

The reported WaMIM training setup uses ImageNet-1K at resolution \(224\times224\), block-wise random masking at \(75\%\) of patches, Haar wavelet DWT with \(5\) decomposition levels, and selected levels \(\{2,3,4,5\}\) as targets for \(K=4\) [2503.00782]. The optimizer is AdamW with weight decay \(0.05\), \(\beta_1=0.9\), \(\beta_2=0.95\), batch size \(2048\), cosine learning-rate schedule, and warmup of \(10\)–\(40\) epochs depending on total pre-train epochs. The base learning rate is \(2\times10^{-4}\) for ViT and \(1\times10^{-4}\) for Swin. Pre-training is reported for \(100/300/400\) epochs with fixed loss weights \(c^{l_k}=\{0.8,0.9,1.1,1.2\}\) and no curriculum on weights [2503.00782].

The ablation results identify several operating points. The best mask ratio is \(75\%\); too low is described as under-constrained, and too high (\(90\%\)) as under-learned. Testing \(3\), \(4\), and \(5\) decomposition levels shows \(5\) to be best because it covers a broader range of scales and frequencies. Light-weight decoders with \(128\)–\(256\) dimensions and \(4\)–\(8\) heads suffice, matching heavier decoders’ accuracy. Uniform spacing of feature locations, exemplified by \(\{3,6,9,12\}\) in ViT-B and \(\{2,4,22,24\}\) in Swin-B, outperforms other heuristics. Emphasizing low-frequency, high-level targets moderately with \(\{0.8,0.9,1.1,1.2\}\) yields the best results, whereas reversing that ordering degrades accuracy [2503.00782].

| Setting | Baseline | WaMIM |
|---|---|---|
| ImageNet-1K, ViT-B | MAE, 1600 epochs, 752 GPU hrs, 83.6 Top-1 | 400 epochs, 96 GPU hrs, 83.8 Top-1 |
| ImageNet-1K, ViT-S | MAE, 300 epochs, 96 GPU hrs, 80.9 Top-1 | 300 epochs, 45 GPU hrs, 82.0 Top-1 |
| ImageNet-1K, Swin-B | SimMIM, 800 epochs, 656 GPU hrs, 84.0 Top-1 | 400 epochs, 160 GPU hrs, 84.1 Top-1 |

For downstream transfer, COCO object detection and instance segmentation with Mask R-CNN and Swin-B report AP\(^b\)=\(50.9\) and AP\(^m\)=\(45.1\) for WaMIM, compared with \(50.4\) and \(44.4\) for SimMIM, at \(160\) versus \(656\) pre-train hours. ADE20K semantic segmentation with UperNet and ViT-B reports mIoU \(48.7\) for WaMIM versus \(48.1\) for MAE, at \(96\) versus \(752\) pre-train hours [2503.00782]. The paper summarizes these findings by stating that WaMIM matches or exceeds pixel-based MIM accuracy at a small fraction (\(10\)–\(20\%\)) of the GPU time.

## 5. Wavelet-domain masking and two-phase training in WMNet

In WMNet, W-MIM is not a generic representation-learning objective but the pre-training phase of an HDR video reconstruction pipeline [2602.07393]. The masking strategy introduces two binary masks. A random mask \(\Omega_r\in\{0,1\}^{h\times w\times 3}\) is applied to the deepest \(LL^{(N)}\), covering a proportion \(\rho_{\text{low}}\) of its coefficients. A full-zero mask \(\Omega_z=0\) is applied to all high-frequency bands \(LH^{(n)}, HL^{(n)}, HH^{(n)}\), forcing the network to reconstruct edges, textures, and chromatic variations from the remaining approximation. Formally,
\[
\hat L^{(N)} = \Omega_r \circ LL^{(N)}, \qquad
\hat H^{(n)} = \Omega_z \circ H^{(n)},
\]
where \(H^{(n)}=(LH^{(n)},HL^{(n)},HH^{(n)})\), and the masked coefficients are
\[
M(W(x)) \coloneqq \{\hat L^{(N)},\hat H^{(n)} \mid n=1\ldots N\}.
\]
The masked frame is then
\[
x' = W^{-1}(M(W(x))).
\]

Phase I trains an asymmetric encoder–decoder \(E\)–\(D\) to reconstruct the original LDR frame from \(x'\) with the \(\ell_1\) loss
\[
L_{\text{pretrain}} = \frac{1}{H\cdot W\cdot 3}\|D(E(x')) - x\|_1.
\]
The encoder is specified as \(15\) ResBlocks of width \(64\), and the decoder is lightweight [2602.07393]. Difficulty is increased gradually through curriculum learning:
\[
\rho_{\text{low}}(t)=\left(\frac{t}{T_1}\right)\rho_{\max}, \qquad \rho_{\max}=0.5,
\]
so the low-frequency mask ratio rises from zero to \(0.5\) over the first \(T_1\) iterations or epochs.

Phase II initializes the video encoder \(M_{ENC}=E\) with the pre-trained weights and adds a full-sized video decoder \(M_{DEC}\), the Temporal Mixture of Experts (T-MoE), and the Dynamic Memory Module (DMM). For an input clip \(x_1,\ldots,x_T\), the network computes \(z_t=M_{ENC}(x_t)\), conditions the features temporally, and reconstructs HDR output as
\[
\hat y_t = M_{DEC}(\hat z_t).
\]
The reconstruction loss is
\[
L_{\text{recon}} =
\frac{1}{T}\sum_{t=1}^T |\hat y_t-y_t|_1
+
\lambda\cdot(1-\mathrm{SSIM}(\hat y_t,y_t)),
\]
with \(\lambda=1\), and no extra regularizers beyond weight decay [2602.07393].

## 6. Architectural integration, empirical effects, and interpretive implications

WMNet inserts W-MIM before any 2-D convolutional layer: each input frame is wavelet-decomposed, masked, inverse-transformed to produce \(x'\), and then sent into a shared encoder defined as \(\text{conv}(3\rightarrow64)\rightarrow\text{ReLU}\rightarrow[\text{ResBlock}\times15]\) with hidden dimension \(64\) [2602.07393]. The decoder uses upsampling by PixelShuffle and convolution layers to reconstruct spatial resolution. T-MoE splits the encoder’s \(15\) ResBlocks into \(D=3\) groups, applies a small expert network \(\text{Conv3D}\rightarrow\text{Softmax}\) over a temporal window of three adjacent frames, and fuses group outputs as
\[
\bar z_t = \sum_{d=1}^D (z^d_t \circ w^d_t), \qquad
\hat z_t^{(T)} = \text{Conv3D}(\bar z_t)+\bar z_t.
\]
DMM maintains a per-scene queue \(M_s\) of up to \(l=2\) memory features, computes cross-attention between current features and scene memory, and updates memory through convolutional refinement [2602.07393].

The ablation on HDRTV4K-Scene isolates the contribution of W-MIM. The baseline without W-MIM reports PSNR \(35.15\) dB, SSIM \(0.9612\), and \(\Delta E_{ITP}=12.11\). Adding W-MIM only yields PSNR \(36.00\) dB, SSIM \(0.9627\), and \(\Delta E_{ITP}=11.05\). A low-frequency-only variant reports PSNR \(36.19\) and \(+1.04\) dB, while a high-frequency-only variant reports \(\Delta E_{ITP}=10.77\), described as best color. A spatial-domain MIM with the same overall mask rate yields PSNR \(36.12\) and \(\Delta E_{ITP}=10.97\). With T-MoE and DMM added, PSNR reaches \(36.23\) dB and \(\Delta E_{ITP}\) drops to \(10.78\), with state-of-the-art across seven metrics: PSNR, SSIM, SR-SIM, \(\Delta E_{ITP}\), HDR-VDP3, LPIPS, and \(E_{warp}\) [2602.07393].

| Configuration | Reported outcome | Interpretation |
|---|---|---|
| No W-MIM | PSNR 35.15, SSIM 0.9612, \(\Delta E_{ITP}=12.11\) | Reference point |
| + W-MIM only | PSNR 36.00, SSIM 0.9627, \(\Delta E_{ITP}=11.05\) | Better fidelity and color |
| + W-MIM + T-MoE + DMM | PSNR 36.23, \(\Delta E_{ITP}=10.78\) | Best full-system result |

Taken together, the image and video formulations support a unified interpretation. WaMIM uses multi-scale wavelet coefficients as compact, frequency-aware reconstruction targets and aligns encoder depth with wavelet scale [2503.00782]. WMNet instead uses wavelet-domain masking to pre-train color and detail restoration before temporal reconstruction [2602.07393]. A plausible implication is that W-MIM is most effective when the decomposition hierarchy is explicitly coupled to the architecture or task structure rather than treated as a purely alternative signal representation.

Source: https://www.emergentmind.com/topics/wavelet-domain-masked-image-modeling-w-mim