Curriculum by Masking (CBM)

• Curriculum by Masking (CBM) is a technique that converts input masking into a curriculum, guiding models from easier to harder prediction tasks.
• CBM dynamically adjusts masking based on factors like saliency, gradient magnitude, and task-specific metrics across modalities such as vision and language.
• Empirical results show that CBM enhances convergence speed, accuracy, and efficiency in various applications including reinforcement learning and point cloud analysis.

Curriculum by Masking (CBM) refers to a spectrum of techniques in deep learning that convert masking patterns—the selective removal or occlusion of input tokens, patches, or goals—into an explicit curriculum learning schedule. Unlike static or random masking, CBM methods dynamically adjust which parts of the input are masked, and how, in order to control task difficulty throughout pretraining or finetuning. This process structures learning to proceed from “easy” masked prediction tasks to progressively more difficult ones, yielding accelerated convergence, improved generalization, and robust representation learning across multiple modalities, including vision, language, reinforcement learning, and point cloud analysis (Lee et al., 2022, Jarca et al., 2024, Madan et al., 2023, Lin et al., 2024, Tang et al., 2024, Yin et al., 18 Sep 2025, Son et al., 2023, Roy et al., 2024, Eppe et al., 2018).

1. Core Principles and Taxonomy

A central observation underlying CBM is that the difficulty of masked prediction tasks can be modulated with respect to properties of the input, the mask, or the model’s own state. CBM strategies generally fall into four archetypes:

CBM Variant	Difficulty Schedule	Masking Target
Data-driven (Saliency)	From salient to global	Patches/Tokens
Learnable/Adversarial	From easy to hard masks	Patches/Tokens
Curriculum via Goal Mask	By subgoal achievement	Subgoals/Coords
Prototype/Sample Curriculum	From prototypical to diverse	Samples

Techniques include masking spatially or semantically salient regions, adaptively expanding the mask ratio, employing learnable masking modules, or prioritizing subgoals of intermediate difficulty. Masking difficulty may be posed as an explicit function of graph connectivity (in language), gradient magnitude (vision), goal success rates (RL), or sample prototypicality (MIM/MAE).

2. Methodological Realizations

2.1. Vision: Patch Masking and Saliency

In vision, masking curriculum can be achieved by occluding an increasing fraction of image patches, typically selected for saliency based on local image-gradient magnitude. The mask ratio $r_k$ is annealed according to a schedule (linear, exponential, logarithmic), moving from minimally to maximally occluded examples (Jarca et al., 2024). At each stage, patches are sampled for masking by their gradient strength to ensure discriminative regions are prioritized. This approach is universally applicable to CNNs, vision transformers (ViTs), and detection pipelines, requiring only input-level modifications.

2.2. Self-supervised and Adversarial Masking Agents

CL-MAE (Madan et al., 2023) introduces a jointly trained masking module that transitions from “assisting” to “challenging” the main encoder through a curriculum weight $\lambda_{CL}^{(t)}$ that linearly decays from positive (minimizing reconstruction loss; easy) to negative (maximizing loss; hard). The masking network leverages a transformer to output soft mask scores, which are binarized per epoch and regulated to maintain a fixed mask ratio via a Kullback–Leibler regularizer. This supplies a continually shifting easy-to-hard sequence of masked prediction tasks.

2.3. Language and Genomic Sequence Modeling

In masked language modeling and genomic sequence tasks, CBM assigns a difficulty metric to tokens or spans: concept degree in a knowledge graph (linguistically, high-degree words are easier) (Lee et al., 2022), or normalized PMI scores for k-mers in genes (Roy et al., 2024). The curriculum then organizes masking targets from frequent/high-connectivity or low-PMI units to more specialized or co-occurring “difficult” spans. Masking transitions over multiple stages, expanding the masked set via graph traversal or triggered by plateaus in validation perplexity.

2.4. Prototype-to-General Curricula

Masked image modeling is subject to sample-level curricula: models are exposed first to prototypical, easy-to-reconstruct sample clusters (identified by embedding with k-means in a pretrained feature space), then, via a temperature-annealing schedule, to the full diversity of the data distribution (Lin et al., 2024). The mask ratio per sample is held constant; only the training sample distribution is annealed.

2.5. Reinforcement Learning: Masked Prediction and Goal Masking

For sequence modeling in RL, CBM employs masking schemes parameterized by block size and mask ratio. An adaptive multi-armed bandit (EXP3) selects masking schemes during pretraining, guided by the magnitude of loss improvement on a validation set (Tang et al., 2024). In goal-conditioned RL, CBM can operate at the level of subgoal masking, dynamically estimating subgoal success probabilities and sampling masks that match a “Goldilocks” level of difficulty—neither too easy nor too hard (Eppe et al., 2018).

3. Algorithmic Components

3.1. Masking Schedule and Curriculum Control

Masking schedules are typically defined by a parameter $r_k$ or $\alpha(t)$, controlling fraction masked, which is updated according to a curriculum schedule:

• Linear: $r_k = r_N \frac{k}{N}$
• Logarithmic: $r_k = r_N \log_2\left(1+\frac{k}{N}\right)$
• Exponential: $r_k = r_N \exp\left(\frac{k-N}{N}\right)$

In learnable-masking approaches, a curriculum weight $\lambda_{CL}^{(t)}$ is annealed, modulating the masking adversary from helpful to challenging (Madan et al., 2023).

3.2. Saliency and Difficulty Computation

• Gradient-based saliency: patches with high local gradient magnitude are masked preferentially (Jarca et al., 2024).
• Graph degree or PMI: concept connectivity or PMI quantifies token or k-mer prediction difficulty (Lee et al., 2022, Roy et al., 2024).
• Attention-driven semantic clustering: semantic components emerge via clustering transformer attention features in point clouds, with masking progression moving from geometric grid masking to semantic component masking (Yin et al., 18 Sep 2025).

3.3. Automated Difficulty Adaptation

EXP3 bandit algorithms or validation-based perplexity plateaus are used to dynamically pick which masking regime to sample at each training step, removing the need for fixed, hand-tuned curriculum sequences (Tang et al., 2024, Roy et al., 2024).

4. Quantitative Impact and Empirical Results

CBM consistently yields improved efficiency and accuracy across diverse settings:

Setting	Efficiency/Accuracy Gains
BERT w/ CCM (Lee et al., 2022)	GLUE +1.9 pts, ~2× faster convergence
ResNet-18 CBM (Jarca et al., 2024)	+1–2.6% accuracy, [email protected] +1.5% on PASCAL VOC
MAE/ViT CL-MAE (Madan et al., 2023)	+2.9% k-NN accuracy; strong downstream transfer
GeneMask CM-GEMS (Roy et al., 2024)	Matches SoTA with 10× fewer pretraining steps
Reinforcement CBM (Eppe et al., 2018, Tang et al., 2024)	36%+ speed up and superior zero-shot skill prompting
Point Cloud CBM (Yin et al., 18 Sep 2025)	+0.3–2% across SVM/rotation scenarios

Ablation studies repeatedly highlight that adaptive or learnable masking, coupled with a progressive easy-to-hard schedule, outperforms static or randomly assigned masks.

5. Practical Implementation and Recommendations

CBM is frequently implemented as a data- or input-level modification, requiring either no or minimal changes to model backbone and incurring low computational overhead. Key best practices include:

• Saliency-based patch selection with a patch grid (vision): $4 \times 4$ grid, mask ratio up to 0.4 (CIFAR) or 0.6 (ImageNet), linear repeat schedules (Jarca et al., 2024).
• Curriculum length/schedule tuning: 4-stage curricula or annealed temperature schedules provide stable gains (Lee et al., 2022, Lin et al., 2024).
• Whole-concept/token/patch masking—always mask semantically atomic units together.
• Automated curricula (bandit, validation-driven) are preferred over hand-tuned ramp or staged approaches for versatility and adaptability (Tang et al., 2024, Roy et al., 2024).
• In distillation, guiding teacher supervision by student attention both lowers computational cost and implements CBM automatically (Son et al., 2023).

6. Limitations and Open Problems

Identified limitations include:

• Masking difficulty metrics (e.g., concept degree, PMI) may not transfer across domains or capture all relevant forms of prediction complexity (Lee et al., 2022, Roy et al., 2024).
• Learnable masking modules introduce additional hyperparameters and require careful schedule design to prevent instability in the adversarial regime (Madan et al., 2023).
• Independence assumptions in subgoal estimation (RL) may fail in hierarchically structured tasks (Eppe et al., 2018).
• CBM schedules may need extension or new abstractions for non-sequential, highly multimodal tasks, or more complex segmentation/unit discovery (Yin et al., 18 Sep 2025).
• The effect of CBM on interpretability and failure modes is not yet fully understood.

7. Cross-Modal Extensions and Future Directions

CBM has seen successful translation into point cloud processing (dual-stream grid and semantic masking) (Yin et al., 18 Sep 2025), knowledge distillation for ViTs (Son et al., 2023), and genomic transformers (Roy et al., 2024). Potential research directions include:

• Generalized, hierarchical curricula combining sample selection, mask selection, and difficulty estimation (Tang et al., 2024).
• Self-adaptive nonlinear curriculum schedules and feature-space adversarial objectives (Madan et al., 2023).
• Broadening CBM to video, audio, and cross-modal pretraining pipelines.
• Integration with model selection and resource allocation strategies for large foundation models.

Curriculum by Masking thus constitutes a principled, domain-general framework for scaffolding the acquisition of complex behaviors and representations via structured input occlusion and adaptive difficulty progression. Its empirical benefits—including compute savings, improved accuracy, and faster convergence—make it a foundational technique across the current spectrum of masked prediction and pretraining paradigms.