Masked Feature Modeling (MFM)

• Masked Feature Modeling (MFM) is a modality-agnostic approach that predicts masked input features from the unmasked context using an encoder–decoder structure.
• It spans diverse applications including masked image modeling, video-language tasks, and structured data processing, each with tailored target spaces and loss functions.
• MFM encourages middle-order interactions by focusing on context aggregation, thereby improving downstream tasks such as recognition, segmentation, and retrieval.

Searching arXiv for recent and foundational papers on Masked Feature Modeling across modalities. Masked Feature Modeling (MFM) denotes a family of masked prediction methods in which a model observes only an unmasked subset of an input’s features, encodes the visible context, and predicts targets at the masked positions. In the most general formulation, an input $x$ is mapped by a tokenization or featureization operator $P$ to features $z=P(x)$, a random index set $M$ is masked, and an encoder–decoder pair predicts targets $t_i$ for $i\in M$ from $z_{\bar M}$. Within this umbrella, Masked Image Modeling (MIM) is the image-specific case, while later work extends the same idea to skeleton sequences, video-LLMs, graphs, point clouds, semantic segmentation, recommendation systems, and multimodal token spaces (Li et al., 2022, Sun et al., 3 Sep 2025, Wang et al., 2022, Mizrahi et al., 2023). At the same time, the acronym “MFM” is not uniform across the literature: in some papers it means “Masked Feature Modeling,” in others “Masked Frequency Modeling,” and in older video-language work it may coexist with “masked frame modeling” (Xie et al., 2022, Fu et al., 2021).

1. Definition and terminological scope

The broadest explicit definition treats MFM as a modality-agnostic masked modeling procedure. Let $x$ be an input, let $P$ produce a sequence or grid of features $z=P(x)$, let $P$0 denote masked positions, and let the model reconstruct targets $P$1 at $P$2 from the unmasked context $P$3. In this sense, MIM is simply the vision instantiation of MFM, where the features are image patches or feature maps (Li et al., 2022). Later work on skeletons uses essentially the same idea but replaces raw reconstruction with hierarchical feature prediction, and multimodal work generalizes the notion further by treating text, images, geometry, semantics, and network feature maps as discrete token spaces to be masked and predicted under a single transformer (Sun et al., 3 Sep 2025, Mizrahi et al., 2023).

A source of recurrent confusion is that “MFM” is also used as a paper-specific acronym for different tasks. In VIOLET, prior “MFM” refers to masked frame modeling, while the paper’s own ablated MFM baseline means patch-level regression of continuous CNN features from masked video patches (Fu et al., 2021). In visual pre-training and medical imaging, “MFM” can instead mean “Masked Frequency Modeling,” i.e., masking Fourier-domain components rather than masking features in an intermediate latent space (Xie et al., 2022, Iqbal et al., 2023). This terminological overlap matters because methods sharing the acronym may differ in target space, masking locus, decoder design, and optimization objective.

Usage of “MFM”	Representative papers	Meaning
General masked feature modeling	(Li et al., 2022, Sun et al., 3 Sep 2025, Zhou et al., 17 Sep 2025)	Predict masked latent or semantic features
Video-language MFM baseline	(Fu et al., 2021)	Regress continuous CNN patch features
Masked frequency modeling	(Xie et al., 2022, Iqbal et al., 2023)	Mask Fourier components and predict missing spectrum
Feature-mask training in recommendation	(Zhu et al., 27 Jan 2025)	Mask feature embeddings and supervise masked forwards

2. Canonical formulation and target spaces

A canonical MFM objective is

$P$4

Here $P$5 is the encoder, $P$6 is the decoder or prediction head, and $P$7 depends on the target space. The decisive design choice is therefore not merely what is masked, but what the model is asked to predict (Li et al., 2022).

The target space varies substantially across the literature. In image models, targets include raw RGB pixels, HOG descriptors, and perceptual features such as DINO features; in video-LLMs they include continuous ResNet-50 patch features or discrete dVAE visual tokens; in masked video distillation they are frozen image-teacher or video-teacher features; in skeleton modeling they are online-generated hierarchical features spanning local motion, mid-term dynamics, and global action semantics; and in multimodal masked modeling they are modality-specific discrete tokens for RGB, captions, depth, surface normals, semantic segmentation, bounding boxes, and CLIP feature maps (Li et al., 2022, Fu et al., 2021, Wang et al., 2022, Sun et al., 3 Sep 2025, Mizrahi et al., 2023).

This diversity of targets produces a corresponding diversity of losses. Pixel and feature regression typically use $P$8, Smooth L1, or masked MSE; token prediction uses cross-entropy over codebooks or vocabularies; graph and point-cloud variants may align assignments or feature distributions rather than exact values. PointGAC, for example, does not reconstruct masked coordinates or features directly, but aligns student assignments to codebook cluster centers obtained from teacher features. CORE goes further by rewriting masked feature reconstruction as a contrastive objective on masked nodes, where the positive pair is the original versus reconstructed feature of a masked node and other masked nodes serve as negatives (Li et al., 7 Jul 2025, Bo et al., 15 Dec 2025).

A further distinction concerns where masking occurs. Some methods mask the input itself, some insert mask tokens into intermediate feature maps, and some drop masked tokens entirely from the encoder and reconstruct only in the decoder. A$P$9MIM explicitly separates “where to mask” from “what to reconstruct,” using RGB-mean filling at the image input and additive mask tokens at intermediate features. MFM for UDA segmentation instead masks the final-stage target-domain feature map and reconstructs it with a lightweight Rebuilder before applying the standard segmentation decoder, while AdaF$z=P(x)$0M$z=P(x)$1 masks feature embeddings with per-feature default mask embeddings and keeps the downstream supervised objective unchanged (Li et al., 2022, Zhou et al., 17 Sep 2025, Zhu et al., 27 Jan 2025).

3. Mechanistic interpretations

A$z=P(x)$2MIM provides one of the clearest mechanistic accounts of why masked modeling can work. Its central claim is that the benefit of masked modeling is not primarily high-fidelity reconstruction, but the induction of “middle-order interactions” among patches: interactions richer than purely local texture cues, yet not requiring fully global scene semantics. The paper formalizes interaction order with game-theoretic measures $z=P(x)$3 and a normalized distribution $z=P(x)$4, and reports that masked modeling shifts $z=P(x)$5 toward middle orders, roughly $z=P(x)$6–$z=P(x)$7 depending on architecture, with diagnostics also reporting approximately $z=P(x)$8–$z=P(x)$9 (Li et al., 2022). This is a direct challenge to the common reduction of masked modeling to pixel inpainting.

Other domains propose closely related but differently phrased mechanisms. GFP for skeletons argues that reconstructing raw coordinates creates computational redundancy and limited semantic representation, whereas predicting a smaller set of hierarchical high-level features aligns pretraining with recognition and retrieval. MVD for video argues that low-level RGB targets are noisy and redundant, and that teacher features provide semantically richer supervision; it explicitly reports that image teachers strengthen spatial representations while video teachers strengthen temporal ones. VIOLET reaches a similar conclusion from the opposite direction: its continuous-feature MFM baseline yields smaller gains than masked visual-token modeling, which the paper attributes to imperfect supervision from external backbones and the difficulty of regressing high-dimensional continuous features (Sun et al., 3 Sep 2025, Wang et al., 2022, Fu et al., 2021).

Frequency-aware work adds another mechanistic lens. Visual Masked Frequency Modeling argues that spatial redundancy makes spatial masking inefficient, and that predicting masked low- or high-frequency components reveals global statistics and structural detail more directly. A$M$0MIM’s Fourier-domain auxiliary loss makes a related point from inside spatial MIM: it weights under-modeled frequency bands, empirically emphasizing mid-frequency structure and complementing ViTs’ slight low-pass bias and CNNs’ high-pass tendencies (Xie et al., 2022, Li et al., 2022). This suggests that many apparently different MFM variants are organizing the same underlying question—how much context is needed, and at what representational level, to force nontrivial structure learning.

4. Architectural realizations across modalities

In image pre-training, A$M$1MIM is built around two design choices: input masking with the image’s per-channel RGB mean, and additive mask tokens at an intermediate layer or stage. For CNNs, the default mask-token insertion point is stage-3; for ViTs, mid-layer placement such as layer-5 in ViT-S ablations is effective. The decoder is lightweight, typically linear, and the loss combines masked spatial reconstruction with an adaptive Fourier-domain loss (Li et al., 2022). By contrast, frequency-domain MFM applies FFT per channel, masks either low- or high-frequency components using a circular mask, reconstructs the image with a linear decoder, and computes loss only on the masked spectrum coordinates, without mask tokens or Siamese structure (Xie et al., 2022).

Video work splits into two lines. VIOLET uses Video Swin Transformer with explicit spatiotemporal attention and compares a masked feature regression baseline against Masked Visual-token Modeling, where masked patches are predicted as discrete dVAE tokens under blockwise and attended masking. MVD instead implements a two-stage masked feature modeling pipeline: image and video teachers are first pretrained by pixel reconstruction, then a student video transformer predicts teacher features at masked spatiotemporal patches, optionally with spatial–temporal co-teaching via separate image and video decoders (Fu et al., 2021, Wang et al., 2022).

Structured-data variants shift the masking unit from patches to joints, nodes, or geometric regions. GFP for skeletons uses a transformer encoder on visible tokens, a cascade of shallow predictor heads at temporal spans $M$2, $M$3, and $M$4, and an online Target Generation Network regularized by variance and covariance constraints inspired by VICReg. PointGAC partitions a point cloud into geometry-aware non-overlapping patches, builds an online codebook via teacher features and online k-means, and aligns masked student predictions to teacher codebook assignments rather than exact coordinates. CORE on graphs keeps the masked-feature-reconstruction setting of GraphMAE-style methods but replaces pointwise reconstruction with a masked-only InfoNCE loss on reconstructed versus original node features (Sun et al., 3 Sep 2025, Li et al., 7 Jul 2025, Bo et al., 15 Dec 2025).

Several works adapt MFM to downstream-task structure rather than pre-training alone. In UDA segmentation, MFM masks final-stage target features, reconstructs them with a lightweight Rebuilder, fuses reconstructed and unmasked features, and then classifies them with the same decoder used for segmentation; the Rebuilder is discarded at inference. In 4M, all modalities are tokenized into discrete vocabularies and a single encoder–decoder predicts masked target tokens from a small visible subset sampled by modality-wise Dirichlet budgets. In industrial recommendation, AdaF$M$5M$M$6 masks feature embeddings with feature-specific $M$7 embeddings, performs supervised auxiliary masked forwards, and combines this with a state-aware adapter that weights features by user and item state (Zhou et al., 17 Sep 2025, Mizrahi et al., 2023, Zhu et al., 27 Jan 2025).

5. Empirical behavior and transfer

Image-scale results show that MFM-style objectives can be competitive across both transformers and convolutional networks when the masking and target design match the architecture. On ImageNet-1K, A$M$8MIM reports 84.2% top-1 with ViT-B using RGB targets and up to 84.4% with A$M$9MIM+ and HOG targets; on CNNs, ResNet-50 reaches 80.4% top-1. On downstream transfer with ViT-B, A$t_i$0MIM reports COCO Mask R-CNN $t_i$1 and $t_i$2, and ADE20K $t_i$3 (Li et al., 2022). Frequency-domain MFM reports 83.1% top-1 with ViT-B/16 on ImageNet-1K and 48.6 mIoU on ADE20K, while also reporting strong robustness on FGSM, PGD, ImageNet-C, ImageNet-A, ImageNet-R, and ImageNet-Sketch (Xie et al., 2022).

In video, the empirical picture is more differentiated. VIOLET finds that adding its continuous-feature MFM baseline to VTM+MLM gives smaller gains than discrete masked visual-token modeling on TGIF, MSRVTT, and DiDeMo, which is consistent with its critique of continuous feature regression (Fu et al., 2021). MVD, however, shows that feature prediction becomes substantially stronger when the targets come from frozen image and video teachers: with ViT-L, it reports 86.4% Top-1 on Kinetics-400 and 76.7% Top-1 on Something-Something-v2, and with ViT-H it reports 77.3% on Something-Something-v2 and 41.1 mAP on AVA v2.2 (Wang et al., 2022).

Skeleton and structured-geometry results emphasize efficiency and semantic alignment. GFP reports 6.2$t_i$4 faster training than standard masked skeleton modeling methods, with NTU RGB+D 60 linear-probe accuracy of 85.9 on x-sub and 92.0 on x-view, together with retrieval scores of 70.9 and 87.1. PointGAC reports 94.3% fine-tuning accuracy on ModelNet40 and strong results on ScanObjectNN, S3DIS, ShapeNetPart, and point-cloud completion. CORE reports improvements over GraphMAE and GraphMAE2 by up to 2.80% and 3.72% on node classification tasks, and by up to 3.82% and 3.76% on graph classification tasks (Sun et al., 3 Sep 2025, Li et al., 7 Jul 2025, Bo et al., 15 Dec 2025).

Task-aligned feature-space MFM also yields clear gains in adaptation and production systems. In UDA segmentation on GTA5$t_i$5Cityscapes, MFM raises DACS from 52.1 to 55.9, DAFormer from 68.3 to 70.3, HRDA from 73.8 to 75.4, and MIC from 75.9 to 77.5; on SYNTHIA$t_i$6Cityscapes it raises DACS from 48.3 to 55.8 and also improves DAFormer, HRDA, and MIC (Zhou et al., 17 Sep 2025). AdaF$t_i$7M$t_i$8 reports online A/B-test improvements of +1.37% cumulative user active days and +1.89% cumulative app duration, and has been deployed on retrieval and ranking tasks in multiple Douyin Group applications (Zhu et al., 27 Jan 2025).

6. Limitations, controversies, and research directions

A first limitation is terminological rather than algorithmic. The same acronym can denote masked feature modeling, masked frequency modeling, or masked frame modeling, and papers with identical three-letter names may therefore not be directly comparable (Xie et al., 2022, Fu et al., 2021, Iqbal et al., 2023). This suggests that comparisons labeled “MFM” should be checked at the level of masking domain, target space, decoder role, and whether the objective is regression, classification, assignment alignment, or contrastive alignment.

A second limitation is sensitivity. A$t_i$9MIM reports dependence on mask-token placement, masking ratio, and patch size, with mid-layer or mid-stage insertion and around 60% masking as robust defaults; it also notes that longer pretraining helps ViTs more than CNNs and that ResNets beyond about 300 epochs may show diminishing or negative returns (Li et al., 2022). UDA segmentation MFM reports that high mask ratios and large Rebuilder capacity hurt performance, and recommends $i\in M$0, $i\in M$1, $i\in M$2, and $i\in M$3 transformer blocks (Zhou et al., 17 Sep 2025). PointGAC requires explicit codebook maintenance to avoid dead vectors, and CORE’s empirical performance depends on temperature and the number of masked-node negatives (Li et al., 7 Jul 2025, Bo et al., 15 Dec 2025).

A third limitation concerns dependence on auxiliary machinery. Some variants rely on tokenizers, teachers, detectors, pseudo-labels, or target networks. VIOLET’s critique of continuous feature regression is precisely that it depends on a fixed supervised backbone that may be imperfectly aligned with the video-language domain (Fu et al., 2021). MVD inherits teacher-quality dependence by design, although it turns that dependence into a controllable choice between spatial and temporal teachers (Wang et al., 2022). Graph-based video-event MFM depends on object detection and on a pretrained vector-quantized tokenizer, and 4M’s generation and transfer quality are bounded by tokenizer quality and pseudo-label quality (Daskalakis et al., 2023, Mizrahi et al., 2023).

The forward-looking agenda is correspondingly broad. A$i\in M$4MIM explicitly sketches extensions to audio, language, and multimodal masking at intermediate layers (Li et al., 2022). UDA segmentation work identifies adaptive masking, stronger but stable Rebuilders, and improved pseudo-labeling as open directions (Zhou et al., 17 Sep 2025). PointGAC proposes multi-scale codebooks and hierarchical clustering, while 4M points toward any-to-any conditional generation from a shared masked-token space (Li et al., 7 Jul 2025, Mizrahi et al., 2023). Across these strands, the recurrent pattern is that MFM is moving away from a single recipe and toward a design principle: choose a masking locus, target space, and objective that force context aggregation at the level of structure most relevant to the downstream problem.